3rd party DLL plugin returns wrong values, was: Chart not getting updated with live data from plugin

Plug-ins
#1

I'm having an issue with data plugin where historical data is loaded and charted OK, but real time data isn't.

The first time GetQuotesEx is called (lastValid = -1, size = 3000) plugin gets all 3000 ticks and stores them into quotes array passed to GetQuotesEx (starting at index 0, incrementing lastValid on the way up to 2999) and displays the chart correctly.
The only thing suspicious I'm noticing at this point is that timestamps displayed on chart are offset by -1 second, so instead of displaying the timestamp obtained from the API (seen in log bellow) "21.3.2018 3:15:00" it instead displays "21.3.2018 3:14:59" (this feels strange already and I don't know why it happens...could be the source of issue?).

Being offset by 1s doesn't bother me that much, but the problem is when AmiBroker requests next (real time) tick...the API returns a correct timestamp (as seen in log bellow) and inserts the quotation into quotes at index 2999 (lastValid: 2998, size: 3000), but for some reason chart doesn't display this last tick (or any new tick from here on)...instead it seems that it only overwrites the last candle charted instead of adding a new one at "21.3.2018 3:29:59".
To elaborate, last 2 candles displayed on chart after 2nd call to GetQuotesEx have timestamps "21.3.2018 2:59:59" (correct) and "21.3.2018 3:00:00" (wrong - latest tick obtained via API has a timestamp "21.3.2018 3:15:00" and should be charted as "21.3.2018 3:14:59"...and I think as a consequence last 2 candles at "21.3.2018 3:14:59" and "21.3.2018 2:29:59" are now missing.

Would anyone know why AmiBroker would behave this way?
Am I filling the quotes wrong?
Is the source of problem timestamps being offset by 1s?

I already tried by starting to fill quotes at index 1 (instead of 0) but that doesn't solve the problem.

Here is the log produced by plugin (all seems OK here):

21.3.2018 4:23:43: periodicity: FifteenMinutes, lastValid: -1, size: 3000, lastDateTime:1.1.1970 0:00:00
21.3.2018 4:23:43: 3000 ticks of data requested (async)...
21.3.2018 4:23:43: 0: 17.2.2018 21:30:00,10776,10824,10776,10817,211.5846
21.3.2018 4:23:43: 1: 17.2.2018 21:45:00,10817,10825,10799.55,10808,206.3401
...
21.3.2018 4:24:12: 2998: 21.3.2018 3:00:00,8938.2,8960,8928.8,8938.3,228.258
21.3.2018 4:24:12: 2999: 21.3.2018 3:15:00,8936.5,8957.3,8920.6,8947,247.9846
21.3.2018 4:24:12: Downloaded next 1000 tick(s) (3000/3000 imported).
21.3.2018 4:24:12: periodicity: FifteenMinutes, lastValid: -1, size: 3000, lastDateTime:1.1.1970 0:00:00
21.3.2018 4:24:12: Imported 3000/3000 tick(s). Last valid quote at index 2999.
21.3.2018 4:30:01: periodicity: FifteenMinutes, lastValid: 2998, size: 3000, lastDateTime:21.3.2018 3:00:00
21.3.2018 4:30:01: 1 ticks of data requested (sync)...
21.3.2018 4:30:01: 0: 21.3.2018 3:30:00,8957.3,8959.7,8954.4,8957,2.355983
21.3.2018 4:30:01: Downloaded next 1 tick(s) (1/1 imported).
21.3.2018 4:30:01: Imported 1/1 tick(s). Last valid quote at index 2999.

Btw, I'm using a separate thread (async) to obtain ticks if there are many (+ notifying AmiBroker main window via SendMessage) and for getting 1 tick I obtain the data inside UI thread (that's why GetQuotesEx is called 2x when obtaining historical data - seen in log above).

1 Like
#2

LIVE DATA SOLVED: After rewriting most of the plugin and even adding code to handle ticks missing from the API I figured out that the problem is that AmiBrokers doesn't shift Quotation array for us (last valid entry I've added was at "lastValid + 1" index), so the plugin has to do the shift and add last live tick only after those ticks.

OFFSET STILL PRESENT: Not sure why, but 1 second offset is still present and displayed by AmiBroker, although timestamps of ticks are rounded on minutes. As said above it's not a big deal, just would be nicer to have accurate ones displayed.

#3

You don't seem to get the idea of nLastValid. The plugin RECEIVES already existing data, so it DOES NOT need to copy data that are already existing. In fact plugin MUST NOT copy data over and over again. Instead it should look at nLastValid quote and compare with what it has and only fill MISSING bars. Ideally plugin updates/adds just ONE bar in the array.

AmiBroker does not change the timestamps. It displays what you place in the timestamp so apparently you are not filling timestamps correctly. Compile example plugins in the ADK and you will see that they work OK.

1 Like
#4

Right, that's what I thought...but actual behavior confuses me.

When AmiBroker requests data for the 1st time I just fill 3000 (size=3000) ticks (lastValid = -1 and is incremented to 2999) - all OK.

On next tick AmiBroker calls plugin again, this time lastValid is pointing to 2998, so I would assume this position is holding the most recent tick/timestamp, however, if I compare this last tick timestamp to the one that was on position 2999 with previous call, the timestamps are off for 1 tick...so it looks like that last valid tick is actually still at 2999 and AmiBroker didn't shift the array for 1 tick for me (what I would expect since it change lastValid index). Could you please explain why? Because array is not shifted I assumed plugin needs to do that before adding new data. So what's the actual position plugin needs to fill? 2999 (according to lastValid index pointer) or 3000 (last quotation without value)?

I'm not sure what's going on with 1s offset, but the timestamps obtained from API are rounded, so that shouldn't happen. This is the implementation I use: https://github.com/kriasoft/amibroker/blob/master/Plugin/Models/AmiDate.cs#L42

Don't see any reasons for that 1s delay... Maybe it could be this: The API I'm using is returning data with 1 tick delay, so plugin always imports one tick before the last full candle (last full candle and the forming candle are imported with delay). Could this be the issue for AmiBroker or is it not sensitive to this?

#5

AmiBroker does NOT modify the data that you passed to it. The Quote array that you fill is NOT modified, so next time GetQuotesEx is called you will get EXACTLY THE SAME array and nLastValid will be 2999 (not 2998). If you need to shift bars, you should shift them in your plugin.

Again, plugin TOTALLY controls the data and AmiBroker does NOT touch the data in any way.

It is PLUGIN responsibility to manage the data.

1 Like
#6

Good to know that no modification is done to data, makes it clearer now, thanks.

Right, I get the same array, but for some reason lastValid integer passed to plugin is really decremented by 1 at next tick (I initially thought that's done while array is shifted for 1 which would then make sense, but as you say that's not the case). When returning lastValid from plugin the first time I just keep it's value so it points to last valid entry that the plugin has added, which is 2999....but I get 2998 instead. Any idea why lastValid integer would be set to 2998 then?

Here is the log from my plugin, and as you can see with 2nd call AmiBroker is pointing to last valid index at 2998 (although the real last valid index is at 2999).

1ST CALL
04/10/2018 11:30:22: ticker: Bitfinex:BCH/USD, periodicity: FiveMinutes, lastValid: -1, size: 3000
04/10/2018 11:30:23: Starting to download 3000 ticks to 30.3.2018 23:25:00...
04/10/2018 11:30:23: stepStartTs: 1523052300 (6.4.2018 22:05:00)
04/10/2018 11:30:23: stepEndTs: 1523352300 (10.4.2018 9:25:00)
04/10/2018 11:30:25: https://api.bitfinex.com/v2/candles/trade:5m:tBCHUSD/hist?start=1523052300001&end=1523352300000&limit=1000
04/10/2018 11:30:25: 0: 10.4.2018 9:25:00,635,635,632.57,633.73,32.91133
04/10/2018 11:30:25: 1: 10.4.2018 9:20:00,635.57,637.93,635,635,91.97565
MORE TICKS HERE...
04/10/2018 11:30:33: 2999: ImportedDate: 10.4.2018 9:25:00
04/10/2018 11:30:33: 3000 tick(s) available. Last valid quote at index 2999.
2ND CALL
04/10/2018 11:35:01: ticker: Bitfinex:BCH/USD, periodicity: FiveMinutes, lastValid: 2998, size: 3000
04/10/2018 11:35:01: Last valid quote date is 10.4.2018 9:25:00
04/10/2018 11:35:01: stepStartTs: 1523352300 (10.4.2018 9:25:00)
04/10/2018 11:35:01: stepEndTs: 1523352600 (10.4.2018 9:30:00)
04/10/2018 11:35:01: https://api.bitfinex.com/v2/candles/trade:5m:tBCHUSD/hist?start=1523352300001&end=1523352600000&limit=1000
04/10/2018 11:35:01: 0: 10.4.2018 9:30:00,634.19,635.82,633.86,634.26,84.79739
04/10/2018 11:35:01: Downloaded next 1 tick(s) (1/1).

In the log above one can see that first call to plugin imports all 3000 ticks (lastValid = -1), the problem only appears on 2nd call of plugin (row with lastValid is logged right after GetQuotesEx is called, so I'm 99% sure that plugin is not changing it - at least not in this method).

Regarding timestamps and 1 second offset displayed on chart....the actual timestamps the plugin uses are also visible in the log above (they are also converted to DateTime) and there is no offset, they are all rounded to 00 , 15, 30, or 45 minutes (no seconds).

And yes, I think I should shift it to make room for the next tick to be added, although I would prefer to hold all historical ticks (if I don't remove/shift AmiBrokers array, i gets full and it complains that more than 3000/size items were added). I guess that it is always limited by size option (user defined when creating DB).

#7

Nothing is decremented.

Your plugin has a mistake, since you are simply returning wrong value from GetQuotes(Ex) function.

You should return the SIZE of the actual data set from GetQuotes/GetQuotesEx. You need to follow the EXAMPLES included in the ADK as starting point. Indices in C/C++ are numbered from ZERO. So if array has 3000 valid data bars (indices 0...2999), you should return 3000 from GetQuotes/GetQuotesEx. nLastValid = 2999 means that you have 3000 valid elements first one has index 0, and valid data size is 3000. And that (3000) is the value to be returned.

ADK requires C/C++ programming knowledge, including things how arrays are represented in memory in C++. For DLL development it is advised to hire somebody with 10+ year professional experience at least.

To all plugin developers: the API/ADK was created back in 2003 and since then (15 years) zero errors were found in it, simply because the are no errors in AmiBroker API. We are using it for all our plugins and the programming interface did not change since 2003 simply because it just works. So if your plugin does not work, look for mistakes in your code.

4 Likes