IB plugin incorrectly backfills Open prices on ASX

Plug-ins
1

I was hoping that someone can help with the incorrect backfill problem using IB plugin.

My database with the base interval of 5min contains about 70 ASX stock symbols. ASX exchange trading starts at 10am and all stocks make their first trade of the day sometime between 10:00am and 10:10am.

For example, IGO first trade today was at 10:04 with the Open price of 10.71 - please see the attached Amibroker Quotations Editor screenshot.

For some reason IB-Plugin creates dummy entries for all 1 minute intervals between 10.00 am and 10:03 am with the Open price equal to the previous day Close and zero volume. This makes Amibroker charts and any indicators use the incorrect open price ( 10.85 instead of 10.71 in the IGO example).

The following day, IB plugin then deletes any dummy quotes (those between 10:00am and 10:03a with the wrong Open price and zero volume) and only shows the actual first trade with the correct open price of 10.71 - So all the previous days have correct Open prices and only the current day shows incorrect Open until the next day when the problem is corrected by the backfill.

I noticed this problem over the past week or so, but it is possible that the problem was there before. I am not sure if anything has changed in the way IB plugin works with Amibroker that I am not aware of that is causing this rather annoying issue.

I would greatly appreciate any help or guidance from experienced users of TWS and IB plugin.

Quotations Editor Screen
Quotations Editor Screen880×663 308 KB

2

IB plug-in transparently passes whatever data is sent by TWS. Complain to IB. Or if you want to workaround, as far as daily records are considered, you can choose the time that is treated as open, see

3

Thank you Tomasz,

Your suggestion makes sense. It seems that TWS prices are correctly displayed in Amibroker - see the Realtime Quote window below. The IB-plugin backfill however creates dummy records with the incorrect Open, while the Realtime Quote window shows the correct Open.

I have tried the workaround to choose open time, but regrettably I cannot know the exact time for all individual stocks since the first trade can happen at any time during the first 10 minutes of the ASX.

I would somehow need to exclude the zero volume trades from Amibroker charts, but that in itself is a task that is beyond what I believe is feasible.

In any case, thank you for your response. I'll contact IB but doubt I would get much joy from them.

Realtime Quote
Realtime Quote919×473 245 KB

4

@Ziza

That's because you request last trade prices.

When there is no volume, IB will not return None/Null/ empty value. Instead, IB will return the last prices with more than 0 volume to you.

In your first image,
if you look at the range (25/10/2023 4:10:00 PM ~ 26/10/2023 10:04:00 AM), all the last prices with 0 volume will be replaced by the last prices ($10.85) with 68321 volume until there is a new last trade price with more than 80750 volume ($10.71) coming out.

5

Chris,
Thank you for this extract from your extended response you generated on the IB support messaging platform.

I copy your full response below, for the benefit of other Amibroker users, including @Tomasz who I guess would be very concerned if all Amibroker users using IB plugin and TWS could not use Amibroker with TWS due to incorrect open prices.

I respectfully disagree with your statement that "This is neither IB issue nor Amibroker issue" and offer my reasons:

The open prices must be correct regardless of anything else...
When Amibroker generates higher interval candles based on the data received from TWS, it does so without changing TWS data in any way.

According to your response, TWS passes the previous Open price where there is no real trade (i.e non existent trade during the first few 1 minute intervals). So in case IGO it passes $10.85 (the previous day Close price) as the Open price for all dummy records, inserting zero as the volume, until the real trade occurs several minutes later.

As a result any Amibroker user trading 10 minutes candles or any other interval, will then have $10.85 as the Open price for the first 10min with the closing price of $10.80 (see the timestamp 10:10:00 in the picture above) . Clearly that is wrong, because the Open price for the first 10 minute interval is $10.71 and Close is $10.80.

TWS does not have such a problem when it generates 10 minutes candles on its own charts - since it does not pass to its own charts anything else but records of real trades (not the dummy ones that TWS passes to Amibroker).

The following day however when backfilling data to Amibroker, TWS removes the dummy records all together and only passes data from real trades. This then corrects the problem for the previous day. I only wish I could trade using the previous data - but I have to trade with what TWS has given to Amibroker and regrettably the passed data are incorrect and totally unusable.

Btw, everything starts ok in the morning until I either re-connect to the TWS or log out and then log back in later on. At that point TWS generates the initial dummy records and Amibroker simply displays what TWS sent.

I repeat, this problem only started to occur recently and certainly not over the past 15 or more years I've been using Amibroker and TWS with IB-plugin. For some reason TWS finds it necessary to generate dummy records if there are no real trades and that has never occurred before to the best of my knowledge.

@Tomasz may have a better view of the IB-plugin interaction with the TWS, but one thing is certain, something has changed in the most recent releases of TWS so that it creates dummy, zero volume records, when the Exchange opens and until the first real trade occurs.

I believe that this is an issue that would affect all Amibroker users using TWS.

Please consider the above as I, and other Amibroker users, may not be able to continue to use TWS and IB in conjuction with Amibroker.

Extended Response from Interactive Brokers @ChriX

Dear Mr. Brkic,

I just read the question that you asked to Tomasz in Amibroker forum. I understand the reason now.

As you requested the last trade prices, only last trade price will be returned.

If the stock has 0 volume, IB server will not return None/ Null values.
Instead, IB will return the last updated trade prices with more than 0 volume to you.

Your attached image IGO-ASX-S-AUD explains this concept.

Date: Open: Volume:
26/10/2023 10:04:00 AM 10.71 80750
26/10/2023 10:03:00 AM 10.85 0
26/10/2023 10:02:00 AM 10.85 0
26/10/2023 10:01:00 AM 10.85 0
26/10/2023 10:00:00 AM 10.85 0
26/10/2023 09:59:00 AM 10.85 0
25/10/2023 4:11:00 PM 10.85 0
25/10/2023 4:10:00 PM 10.85 68321

From 25/10/2023 4:11:00 PM to 26/10/2023 10:03:00 AM, as there is no traded volume, all the open prices will be replaced by the last updated trade price ($10.85).

When the time is 26/10/2023 10:04:00 AM, as there is 80750 volume, the open price will be updated to $10.71.

This is neither IB issue nor Amibroker issue.

Kind regards,
Christophe P
IBKR API Support Team, Technical Assistance Center

6

@Ziza

When you consider the meaning of last trade price, you will think this mechanism makes sense.

You never know the actual market value of a product with no volume.

Let me take an example: NFT

Now, no one is willing to buy NFT, what can you do to measure its actual market value?

What you can try to do for measuring its value is to:

  1. Look at its historical price (last trade price)
  2. See who is the willing buyer/ seller ( Midpoint/ bid ask price)

Also, I think you can write a function to delete the last trade price with 0 volume, so that you could get your expected output.

7

@ChriX

Christopher, I assume you are responding in this forum as a private individual and not as a representative of Interactive Brokers and if so, I am responding to you in that capacity.

Your response is a digression and it does not have any relevance to the actual problem.

TWS is generating dummy records and sending incorrect data to the IB-Plugin while at the same time for its own charts IB uses the correct Open prices and not those from the previous day.

Please pause for a second and read the above again.

No other reputable broker, either full service or on-line only, anywhere in the world uses the previous day Close as an Open price for the following day for asset class like stocks. The many technical indicators, like opening gaps etc, could never work because the gaps will never be shown.

The Market Makers do not use the previous Close as their Open price, but open the market based on what they believe the opening price should be depending on the demand and supply for a particular stock. All brokers then report that Open price determined by the Market Makers and not the one from the previous day.

Since you are responding here in your private capacity, I will contact Interactive Brokers again and request a senior technical support person with detailed knowledge of IB Plugin functionality to confirm why the discrepancies only started to occur in the last few weeks and why IB own charts in TWS do not have the same problem.

The simple matter of fact is the TWS backfill creates dummy records before the first actual trade of the day happens, using previous day Closing price as an Open price for the following day. That is wrong and no other broker in the world does that.

I am not sure what your trading or software technical experience is, but based on your latest response referencing "actual market value of NFT" etc. makes me think that a better understanding of the technical issue I reported is needed to resolve the problem that was not present in the past.

I hope that @Tomasz will have the time to review the issue as all Amibroker users using TWS will have the same problem.

Thank you.

8

Yes. Please consider me as a private individual Amibroker user (ex-Amibroker Professional Editon User) despite I haven't used Amibroker since 2022. And I could not tell you much information here as I may violate company communication policy.

I come here just because want to see how Tomasz responds. He is the only guy who knows the Amibroker the most.

But you are right. It is suggested that you should create a ticket in IB.

9

It should be noted that the open price can be determined:
a) by first reported trade after the open
b) by opening price auction (if there was a result)

Each exchange effectively has its own "opening price auction" due to them having their own separate order books, so sometimes a) will happen before b).

In years gone by, various stocks were opened by specialists, and that could occur several minutes after the opening bell.

Furthermore, just to complicate things, ASX opens its stocks in a staggered group fashion. For example stocks with a symbol starting with S-Z open at 10:09:00 +- 15 (random) seconds. A-B stocks open at 10:00:00 +- 15 (random) seconds. The case above shows "IGO" on ASX which opens at 10:04:30 +- 15 seconds..

The staggered open is fairly silly in my opinion - it was developed when the original data feeds were X.25 protocol at 32 or 64 Kbps. ASX uses the Nasdaq Genium platform which opens 3000 listings plus all of the secondary trading of other securities listed at other exchanges simultanously. Norgate Data has provided feedback to the ASX about this but we're just a data vendor - not a large broking house with larger influence. Redevelopment of legacy systems appears to be difficult. This is probably why newer competitors (Chi-X, now called Cboe Australia) opens everyting at 10:00:00 and their closing price action of everything finishes at 16:11:00.

Decent data providers have an optional "padding" setting whereby the previous bar's close is repeated into the current bar, with volume=0 indicating no trades. It does NOT repeat OHLCV. Such an optional padding setting can be applied to intraday bars too, which appears to be what the IB is reporting (albeit not optional).

In any of these scenarios you need to consider various corner cases, such as:
Consider a stock that doesn't have a valid trade until 10:30
Consider a stock that is halted until 15:35
Consider a stock that has nobody willing to cross a trade on its listed exchange at the open, but plenty of trading at other venues
Consider a stock that is suspended for 2 days
Consider a stock that has had a stock split or reverse split, but nobody wants to trade it for a few hours after open.
(there's plenty more corner cases)

This is perhaps a more complex issue than most people understand.

I hope my contribution sheds some further light on this.

Best regards,
Richard Dale
Chief Information Officer
Norgate Data

7 Likes
10

@NorgateData

Richard, thank you very much indeed for this explanation.
As you rightly say the open price is either determined by the 1st reported trade or by the opening price auction.

But the Opening price for the current day is certainly not determined by copying the previous day Closing price.

For some reason, I suspect due to a software bug introduced by the most recent releases of the IB TWS software backfill function, all Open prices of all stocks are filled with the previous day Close.

That is simply wrong, making Amibroker and any other trading software using IB and IB backfill unusable for trading.

I guess if the IB Plugin was able to delete dummy records (those created by IB with zero volume prior to any real trade) then the charts and all technical indicators would work well. The feasibility of this is however best determined by @Tomasz

No other broker in the world is asking traders to trade using previous day Closing prices as the Opening prices for the current trading day.

I hope that IB and Amibroker via IB plugin will eventually resolve this problem.

11

First many thanks to Richard @NorgateData for the insight regarding local ASX peculiarities.

In addition to what was said already in this thread I would like to point out two things:

  1. the Interactive Brokers plugin does not have any special handling for ASX. It works for ASX exactly the same as it works for any other exchange and it transparently passes data received from Interactive Brokers.

  2. It is important to distinguish between real-time streaming data and historical data (i.e. backfill) as they come from different endpoints of TWS API

Real time streaming data

For what it is worth, the "Open" column in Real-Time Quote window shows what plugin reports as day's open price. The Interactive Brokers data plugin requests TWS to send
market updates by making reqMktData call
https://interactivebrokers.github.io/tws-api/md_request.html

In the response to this request TWS sends updates in the form of records that they call "ticks". There are several tick types that TWS sends:
https://interactivebrokers.github.io/tws-api/tick_types.html

In fact all data in Real Time quote window come directly from real-time stream sent in response to reqMktData call.

As for "Open" price the plugin uses TickID=14 that is described in the TWS docs as follows:

Current session's opening price. Before open will refer to previous day. The official opening price requires a market data subscription to the native exchange of the instrument.

So the data displayed in "Open" column in RT quote window come directly from TWS without any processing and as such Interactive Brokers have total control over what is displayed there.

You just might use TWS API sample application to display exactly same data and see that all that data come from IB.

Our plugins are meant to be 100% transparent. They are all built on principle of just doing verbatim copy of data received from the source. No calculations, transformations or changes are done. What comes from the source is transparently passed to the display.

Any complaints about data that come from TWS should be sent to Interactive Brokers as we can't fix their data.

Backfill (historical) data

It should be noted that BACKFILL is another story. BACKFILL data are result of different request. To get historical data (backfill) the plugin makes different request as described in TWS API documentation

https://interactivebrokers.github.io/tws-api/historical_data.html

Historical data are built by the Interactive Brokers. Again the plugin passes data received from TWS.

Now it is important to remember that if you backfill 1-minute data, the plugin asks IB for 1-minute data using reqHistoricalData call:

https://interactivebrokers.github.io/tws-api/historical_bars.html

What you will get is the 1-minute history. Now if you display DAILY bars, the "open" price will be the result of time compression of 1-minute data to EOD bars. That compression uses session start and end times as defined by the user. Minute to daily bar compression is explained in the Knowledge Base: AmiBroker Knowledge Base » How does the intraday-to-daily compression work?

Because backfill data (that what you see on chart) come from different request than streaming data (that you see in RT quote window) they might be different especially when session start time set by you does not align with actual market open or when delays like explained by @NorgateData happen.

Typically time-compressed data from 1-minute input will differ slightly from officially reported "daily" bars.
To get those officially reported "daily bars" you would need a data source that delivers such EOD data such as NorgateData for example. Note that such data are available after the session.

2 Likes
12

@Tomasz

Tomasz, thank you. I do appreciate and am very grateful for your excellent explanation of the technical side of the problem.

The big question is why this problem only started very recently when there was no problem before (for at least 15 years that I know of) . Of course, the easiest answer to this question would be to put pressure on IB to change how they construct data. But as they say "we got play with the cards that we are dealt and with the ones we wish we had".

So at the technical level I fully understand the issue. From a practical trading point of view and actually risking real money, it is much more difficult to understand.

My own dilemma now is: how can I with a right mind, risk any money by relying on the functionality of the TWS / Amibroker interface that is not delivering the right data regardless of the source / cause of the problem.

For me, this problem remain unresolved and I have to find a way to retain Amibroker in conjunction with IB TWS because I invested too much time in both. Perhaps I need to invest time to somehow delete dummy records with zero volume created before the very first real trade occurring at the start of the trading day. That way my charts and indicators will not be "wrong". I don't know.

It seems that this problem could also negatively impact usability of Amibroker in conjunction with TWS for intraday trading regardless of why the problem exists. But that is more of a business problem for you rather than a technical issue.

Once again, thank you for your great help in understanding the technical side of the issue.

13

Please explain what do you expect from me? What kind of "solution" do you envision from me?

14

Tomasz,

I could not be more grateful for what you have already done - thank you again.

Your excellent explanation of the reasons IB-pugin brings over the non-existent trades generated by IB to Amibroker and the contribution from Richard @NorgateData explaining the complexities of the ASX market, enabled me to create a not so elegant, but usable workaround I described below.

I've used Amibroker for a very long time and still believe that it is the best software for a swing / momentum trader like myself. But the issue is that an end-user or trader, experiences the Open prices problem at the Amibroker end of the interface and not necessarily at the TWS end, despite the fact that neither IB-plugin nor Amibroker software itself are the cause of the problem.

My concern is that this situation does not reflect well on Amibroker because that is where the trader interacts with charts and indicators.

So I am not sure if I am in position to expect anything from you other than for you to continue your outstanding work, but I do have an idea. (I consider any idea as good idea when brainstorming).

Perhaps a simple solution to avoid TWS making Ambroker looking bad, would be for Amibroker (IB-plugin) to exclude dummy trades from TWS with zero volume on the fly and not store them in the database. Perhaps introduce a configuration param that would allow for dummy trades to be either included or excluded.

I of course realise that this "simple" solution could be very complex to implement but it could be worth considering so that TWS does not force Amibroker users to trade relying on incorrect Open prices.

The not so elegant workaround I implemented today is to execute a java script outside Amibroker removing dummy trades with the incorrect Open price and zero volume, every time after connecting and re-connecting IB plugin to TWS. Since I only have 70 or so stocks in my database, executing this script on the current trading day enables correct AB charts and indicators. It also does not take great amount of time. Yes it is not so elegant workaround, but I do this in preference to risking money by using charts displaying wrong Open prices, regardless of the reasons.

I hope you do not mind my extended explanation of my thinking about the rather important practical issue with TWS / Amibroker interface.

1 Like
15

This real trading problem remains unresolved, despite all programming superiority of both IB TWS and IB Plugin.

IB continues to use previous day Close as Open price for the current trading day and IB Plugin simply propagates and carries over the wrong Open to my Amibroker charts. Surely, no professional trader \would be willing to just accept this situation and keep trading.

Trading is risky, but trading with wrong Open prices is totally ridiculous in my honest opinion, no matter what the reason is for wrong data.

Btw, I am not aware of any other reputable trading platform that has this problem, so It seems that my suggested simple and real solution above is starting to make lot of sense.

I'd greatly appreciate if any experienced trader on this forum could offer a better solution to what I suggested above.