Annoying AmiBroker problem - Does not save all files (sometimes)

AmiBroker
#1

I have been using Amibroker for almost 10 years and love it for its features.

But .....

One problem seems to have been carried over in all Amibroker versions since 2008 and on all Windows versions since Windows 7 to Windows 10, both 32bit and 64bit.

The problem is that occasionally Amibroker does not save some data files. This happens at the time of program exit only.

Once I stop IB plugin, I usually click on File then exit. Amibroker then displays a small window with a message "Saving ... symbol file name" and after a while it simply exits out. I know every time when the problem occurs as Amibroker takes longer than normal to "save" the database. However up to 10 files may not be saved saved at all - not the same files, but any data files in the database. The database is still there, with all the files, but some of the files have no data at all showing file sizes of 2K - i.e total loss of data.

My standard mode of operation is as follows: I run IB TWS on my computer (i.e not the web version). I start Amibroker and connect to IB using IB plugin for data updates to about 50 symbols. I only use Amibroker for charting and not for placing or monitoring orders.

I also run Amiquote to automatically download and import data for the remaining 150 ASX200 stocks into the same database every 5 minutes.

The computer running Amibroker x 64 bit, runs Win10, has more than plenty of resources in terms of memory, processing capacity and disk space (Amibroker almost never exceeds 1% to 2% of CPU time for example). The system is only used for trading during normal hours and no other applications are active during the trading day.

Searching the web, I found the some other Amibroker users experienced the same problem in the past but I could not find a solution from anyone. I also reported this problem to Amibroker Support in 2009 but no resolution has been found.

By experimenting, I found a way to preserve data by opening Database Settings and clicking on "Flush Cache". By flushing, all data get written to the database folders at that time. I then, before exiting Amibroker, manually copy the complete database to another location while Amibroker is still running. That way, if any data is lost when I exit Amibroker, I can copy data back to the database and restart. I do this every time now, as I am scared of losing data that it took me so long to collect.

But, having spent almost 40 years in the computer industry at a professional level, I know that problems like this are annoying to say the least and should not exist after many software and hardware updates to my trading environment.

I hope that someone in Amibroker community has discovered a solution to this problem that has bugged me (no pun intended) for the past 10 years.

Thank you !

1 Like
#2

This has never happened to me personally over 20+ years. The database save is totally simple it is just fopen/fwrite/fclose. There is no "magic", no extra "manipulation", no "processing", no nothing. It is the simplest possible block write to file. So it "just works" provided there are no problems elsewhere, with the computer (like 3rd party ANTIVIRUS blocking access to files) and if the disk has free space. Flushing cache is NO different than just save (uses very same fopen/fwrite/fclose).

One thing to check though: make sure that you don't have VirtualStore folder like this:

C:\Users\*current logged in user*\AppData\Local\VirtualStore\Program Files (x86)\AmiBroker

If such folder exists, it means that your AmiBroker was NOT properly installed in first place or 3rd party antivirus it playing games with you. Existence of such folder means that writes are redirected by Windows to wrong place (VirtualStore) and they may cause all kinds of troubles including inability to save new content.

To fix that you need to:

  1. Delete Virtual Store folder
    C:\Users\*current logged in user*\AppData\Local\VirtualStore\Program Files (x86)\AmiBroker

  2. Run FULL SETUP of AmiBroker

1 Like
#3

I had this same problem 1 year ago. Unfortunately I could not find a solution either and stopped the IQ account. It was random, some files just dit not update and some did update oke. I could not find any logical explanation. I tried IQ to download 10 years of 5 min data and databases with upto 2000 tickers and thought that might be too much for Amibroker. I tried some things adviced by support but still the problem persisted. I used the latest version of IQ and Amibroker at that time and the computer had 16 GB of memory and no obvious problems with any other software.
I hope the reason of this problems will be discovered

#4

Thank you Tomasz for your suggestions.
Amibroker is not in my Virtual Store folder, but your comment regarding Antivirus is one possibility as I use Kaspersky Internet Security.

I suspect that Kaspersky antivirus hooks into fwrite operation and compares the data being written against virus patterns known to Kaspersky in order to classify it as virus or safe. So while Amibroker access to files is not blocked by Kaspersky, the antivirus checking process slows writing of many files to the disk that need to be written at the time of Amibroker exit.

So I would suspect Amibroker does not finish writing some files, while waiting on Kaspersky to release control back to Amibroker after checking for antivirus patterns.

I cannot explain however, as to why Amibroker finishes writing files without any problems most of the time. Perhaps no need for Kaspersky to check the files at that time and Amibroker simply fclose the file because it was written before the exit.

If my suspicions above are correct, then simply excluding Amibroker files from Kaspersky monitoring, while not desirable, should remove the reasons for Amibroker not finishing writing files.

So based on your suggestion and my suspicion of what is happening, I will exclude the complete Amibroker file folder storing data and observe what is happening.

This should remove any reason for Amibroker not to complete writing of some files, but I will report back my findings in a few days time.

Thanks again

#5

You should always exclude from antivirus also because portfolio
backtests will be very slow. But in my case 1 year ago I also did
have AV on so I am curious about your progress.

#6

PWFores,
thank you for suggestion. I do not use backtests and cannot report how fast or slow it would work - I guess you could say I am abusing my computer by making it work harder than it should because it has plenty of resources.

The problem with Amibroker not writing some files has been with me for that past 10 years. I changed computer hardware and several Windows operating systems, but the problem persisted.

Most of the time no problems though, but once a week or so, I lose the content of some files (different on each occasion). For example I may have 10 years of data for CBA stock and once I exit Amibroker, the CBA file ends up having no data at all next time I start Amibroker. So I copy the CBA file from a backup and start again. Next time it could be some other file.

I do not remember if I have ever excluded files from being checked by Kaspersky. So you could say, yes I am guilty of making my computer working harder (slower).

Applications including Amibroker may run slower as they have to wait for antivurs to complete it's checking, BUT they should not loose data under any circumstances.

I will let you know how I go with excluding Amibroker data files from Kaspersky.

#7

I have been using IB-TWS with ASX data for two years and have always had this problem too.
The only method I have found to guarantee all files are updated, is this:

  • List all symbols in the RT quote window.
  • Run a 1 day backfill on all symbols in the RT quote window
  • Click on the first symbol in the symbol list in the AB symbol list window on the left hand side
  • Hold down the down arrow so each symbol is briefly displayed as a chart as you scroll through the symbols
  • Then save the database.

I use a simple 5 min chart with no indicators for this process and watch the date/time on the x-axis to make sure each symbol has updated. It's pretty quick at cycling through the 80 or so symbols I watch.

I checked this situation with Marcin about 18 months ago and he confirmed that this was an acceptable procedure or to run a scan or exploration using all symbols.

#8

For TWS/IB make sure that "request data on save" is UNCHECKED.

image

3 Likes
Data wiped intermittently
#9

Tomasz,

Wow, thank you for this. I have exited Amibroker now dozen of times after I switched off "request data on save" and not a single time did I experience the problem.

Come to think of it, I do not know why I had this "request for data save" checked on in the first place. I've been using Amibroker since version 4.xx and always had this thing switched on.

In fact I think, I only ever had problems with symbols that use IB/TWS as the source of data (and not Amiquote that updates the same database), but because I was shutting the plugin off before I was exiting, I thought this setting would not impact the database.

In any case, switching scanning of files by Kaspersky, did not help, but switching the "Request data on save" seems to have resolved the problem.

I do not expect the problem again after this, but will let you how I go over the next few weeks.

For now THANK YOU again.

Cheers, Ziza

3 Likes
#10

Dave, thank your for your suggestion. I followed your procedure to the letter, but it looks that "request data on save" with IB/TWS is causing the problem.

It looks like Tomasz's suggestion to switch this setting off has solved the problem.

I wonder if you have "request data on save" off in your Amibroker and if this would help you in any way.

Kind Regards,
Ziza

#11

Hi Ziza, yes, I have "Request data on save (external RT sources only) unchecked.
However this alone did not solve the problem for me. I still have to follow the procedure described above to consistently get all symbols to save properly.

#12

Thanks Dave,
I just realised that the reason I am getting all the quotes from IB/TWS is because I automatically run an exploration every 3 minutes which then triggers a backfill from IB/TWS for all symbols that had a transaction since the last exploration.

Your solution is however still worth performing as I could exit before I run the last exploration, in which case I may not get the most recent transactions that occurred since the last run of my exploration.

So, I'll start using your procedure in conjunction with switching off "request data on save". That will ensure that I get all transactions from IB and when Amibroker exits, it won't request data from IB, ensuring that everything is saved.

Thanks again for your help, it is much appreciated.
Kind Regards,
Ziza

#13

Hi Ziza, may I ask what the data source is for these symbols accessed via Amiquote? I am looking for an additional data source for ASX symbols.

#14

Hi Dave,

The short answer to your question is : I use Amiquote and Google Intraday as the source for about 150 stocks.

The longer answer including explanation on how and why I do this, is as follows:

I focus on ASX200 stocks only. I have divided them in 2 groups:

For the smaller group of 50 stocks, a broad shortlist if you will, I use IB Plugin and obtain data from IB. These are stocks that I monitor daily.

The other group are those 150 or so stocks that I am not likely to trade but use their data, together with the 1st group, for automatically calculating ASX200 volume (not normally supplied by data vendors) and some other market breadth type of indicators.

For the second group I use Amiquote with Google Intraday data as the source and 5min Interval. Amiquote obtains data from Google Finance every 3 minutes and automatically imports data into Amibroker Database. These are obviously delayed data, but since I do not trade these stocks, a delay of 15 minutes or so is not relevant to me as I only use the Amiquote data for market breadth indicators.

If at some stage I determine that a stock, that is normally downloaded by Amiquote, is worth trading, I remove that stock from Amiquote and change the source of data to be IB Plugin.

Hope all of the above makes some sense, but you are welcome to ask for any additional info if it is going to be of help.

Here is a screen shot of Amiquote
image

Cheers, Ziza

#15

Thanks Ziza, that's great info. Sounds like we are doing similar things - calculating ASX200 volume etc. Except I'm only using IB data at the moment. Will look into Google

#16

Are you able to add data with IB for existing history that you had from google intraday? IB has few history.

If at some stage I determine that a stock, that is normally downloaded by Amiquote, is worth trading, I remove that stock from Amiquote and change the source of data to be IB Plugin.

#17

Hi PWFores
The simple answer to your question is Yes.

The longer explanation is:

I only use Amiquote for stocks that I am not likely to trade but i keep their data into the same Amibroker Database used by IB.

The database is updated by both IB plugin and Amiquote simultaneously - i.e both running in parallel with each other, but each populating their own set of stock symbols.

To achieve that, I use Amibroker information window and set "Use only local database" to "Yes" for all stock symbols that are populated by Amiquote.

For example I do not trade Blackmores Limited so I set it as "local database" as per the image here:
image

So BKL symbol receives data from Google at the same time as Amibroker interacts via IB plugin to obtain data for all other symbols where the "Use only local database" is set to "No" i.e those stocks that I am actively monitoring and am likely to trade .

Say that I determine that I'd like to trade BKL at some point in time. At that point in time, I switch the "Use only database" back to "No". This causes IB plugin to obtain BKL data from IB/TWS. At the same time, I remove BKL from Amiquote download list in order to avoid duplication of data.

So at any point in time, I have historical data for all stocks. Amiquote populates data from Google and IB plugin does it by using IB/TWS. I initially obtain history from any source I can find, like Google or Alphavantage for example. If I am to trade to stock in the near future, I use IB plugin, otherwise I set the above parameter to "Yes" and include the symbol in the Amiquote download list.

One could ask a question about accuracy of historical data obtained from Google or similar sources. And yes, sometimes accuracy is not 100%, but it is good enough for historical prices.

My reasoning is that If you are a very short term trader, historical prices have little meaning to you in any case. If you are a inter-medium swing trader, or longer term trader, then a small difference is of little if any importance to you. In any case, you are obtaining the most recent accurate enough data from IB/TWS and IB provides long enough backfill if you missed a few days or a month of data.

So, there you have it... A long answer that I hope is more helpful than it is confusing.

Hope this helps.
Ziza

#18

Zizza,

Thank you so much for your reply. This is a very good description.
Please let this forum know if you still have this problem (not saving data
in some tickers) or if it really fixed it forgood.

#19

PWFores,

thank you. I am confident that the reason Amibroker was not saving all the files was because it was set to "request data on save" .

So Amibroker tried to talk to IB/TWS when it was exiting and failed to save data for some symbols while waiting on IB/TWS to respond.

I thanked Tomasz, the Amibroker author and owner, for suggesting that I should switch "request data on save" off, which was the main cause of the problem.

Good luck trading.

Ziza

1 Like
#20

Hi,
This problem I have only when filling via RT Quote window.
When I backfill quotes by double click on symbol name (one by one) it is OK.
Problem is 32-bit and 64-bit version.
Przemek