50 charts using TWS as data source

Hi Hari. I was wondering if you had found a way to run 50 charts.

Personally, the issue is not screen space but CPU load. During market hours with quotes fed from Trader Workstation, I can have about 25-30 small bare-minimum charts with the same Chart ID before CPU is at 100% continuous load. I have an i7-8850H at 2.6 GHz. I tried disabling all plots but it didn't make a difference.

Running the equivalent from Exploration would be good but the smallest refresh rate currently available is 1 s whereas for charts it's tick-by-tick. I'm still looking at the best way to harness Amibroker's huge power.

Hey Hari and everyone. I'd like to add the code check says my formula should run in 0.055 sec. I set the backfill to be 5 days and my charts are hourly.

My broker is Interactive Brokers so I tried getting around calling GetTradingInterface every time it runs but could not find a way. Was anyone able to?

Am I doing something wrong? Am I missing anything? Any suggestion is appreciated. Thanks!

Dear Amibroker Forum members, I installed the 64 bit version instead of 32 and saw a huge improvement when running my formula from Exploration. There is still however an important impact on fill quality of running every second with Exploration vs every 0.3 sec from charts. Any suggestion is welcome. Any and all suggestions are welcome. Thanks!

I would suggest you build yourself an AMD Epyc or Threadripper CPU build, with 128 GB+ of memory and fast Nvme or raid storage, and run a version of Windows 10 (64) for Workstations.

1 Like

Also, as far as non hardware related issues...

  • Optimize your code
  • Pre-cache symbols with a quick exploration
  • Try not to use DDE
  • Find another way other than writing to disk
  • Do you actually need to display all these charts?

Sean, thanks so much! There's a lot in there for me to try. Everything in there is new to me other than better code and faster CPU.

To answer your question, I use charts so my formula runs every tick in order to get the best possible fills and hence best profits but I don't need charts per say.

No worries. It's hard to gauge what your code is doing and what parts of it need to be done and when. For example, symbol filtering/dynamic selection, then a tactical trade setup, and then an execution algorithm may not have to be all calculated every tick. Keep in mind that orders, fills, cancel and replace messages might not be even optimal with IB or whomever your trading with, not to mention internet speeds may make tick by tick trading very tricky. You want to look for any backlog you could be generating on your side or with IB. Perhaps you can accomplish much the same using 5 second bars, but using the incoming last print as your tick et al.

With respect to Cores vs Clock speed, best of both worlds might be something like a 32/64 core latest Threadripper and experiment with disabling Hyperthreading.

Amazing, thanks Sean! I trade with IB. Can you elaborate a bit more on the backlog aspect?

Well, for example if you were to actually see an unfiltered, or unconsolidated S&P E-mini tick file, I think you would be astonished at the size of the file for just 1 day. But even using a less of a "fire - hydrant" incoming quotes, they can be very substantial.

So what I mean by backlog, is simply not being able to keep-up with the incoming quotes and firing orders per tick and/or sending messages/orders to IB, before IB can even process the previous order etc.

Then there is the speed at which IB reports back to you on fills, positions, cash, etc. and which are not instantaneous, all account metrics updated sometimes in different chronological order and can even be further delayed in fast markets...
https://forum.amibroker.com/t/quantity-of-modifyorder-partial-filled/36874/5

This make high frequncy trading problematic. Multiply this by 50 charts that have to be updated might even slow things down further. Perhaps you can minimize them.

I would ask others that trade in your time frame, or similar, or ask the "MAN" Tomasz what he thinks.

Thanks a lot for the clarification Sean. I will factor all that in including asking Tomasz.

FYI, AB can run 50 charts but remember if your autotrading code runs in a chart, and the chart window gets closed, your code is interrupted mid-execution. It's better to use a repeating exploration.

Also remember that AB has one GUI thread, which also runs the data plugin. Although your CPU may support many threads, the performance bottleneck will always be the GUI thread and if your code takes too long to execute in a worker thread, it will delay the GUI thread.

2 Likes

Thanks Peter! Repeating Exploration is great except I have to avoid the open since it's too volatile. I'm testing the rest of the day with limit orders to see how it looks.

Do you think there's a way to separate/view the times when my code is the issue from when it's not?

Not sure what you mean. You could certainly add rules about the time of day to only allow trades during certain times, using TimeNum(). You could set it to only allow entry between 10:00 and 3:30. Maybe another rule to allow exits between 9:30 and 4:00.

Nonsense. Worker threads don't block GUI. Code execution in worker thread does NOT DELAY GUI.

Also pretty much every application in Windows has one GUI thread because Windows API discourages multi-thread access to UI controls (as GUI controls are not multi-threading friendly) - some beginner discussion as to "why" is here

Properly designed application, like AmiBroker, does THE WORK in WORKER (non-UI) threads, main UI thread is left just for handling USER input. Human input is not as frequent and single thread is perfectly capable of handling user input with small CPU usage.

Really user input can be handled with 7MHz cpu (yes 7 MHz, 0.007 GHz).
AmiBroker was originally written on Amiga and it had 7MHz CPU MC 68000 with 1 MB (megabyte) of RAM and it was perfectly capable of using one thread for GUI.

So stop spreading nonsense.

As to "100% usage". The CPU is meant to be used. You want your formulas to be executed with maximum speed, not to have CPU sitting idle. AmiBroker automatically scales frequency of refreshes to maximize processor use to refresh real time charts as frequently as your FORMULA allows.
If your formula takes 0.05 seconds to execute and you have 50 charts and say 10 cores then 100% usage just means that your charts refresh every 0.05 * 50 / 10 = 0.25 seconds.
100% CPU use on multi-core CPU actually means PERFECT multi-threading.

EVERYONE SHOULD JUST DO THE MATH.

Don't spread blue sky opinions, without any evidence or actual real world experience.

Hi Hari. I was wondering if you had found a way to run 50 charts.

50 charts can be run with ease. Question is, what for? Do you have 50 eyes? Can you really watch 50 charts simultaneously?? I doubt that. Notice that changing from one symbol to another is instant in AmiBroker. As others probably mentioned, exploration is way better way to programmatically detect interesting things than staring 8 hours a day on 50 charts. How many days are you able to work that way? Information overload is guaranteed. Not to mention huge eye strain.

Also original poster, as always did NOT include ANY information. ANY screenshots, ANY formulas, NOTHING at all, except that he has some model of CPU.
Really CPU doesn't matter. What matters is your setup, YOUR FORMULAS, YOUR SETTINGS. Screenshots, details, etc. Please follow this advice: How to ask a good question

All general questions like "am I able to do something" make absolutely NO SENSE at all.
The only answer that can be given without providing ALL necessary details is I don't know about you, but yes sure I am able to do this, whenever you are able solely depends on YOU, not me nor the program itself. It is like asking "am I able to use a hammer to make La Pietà di Michelangelo". I don't know about you, but Michelangelo apparently was able.

Recommended reading:
https://www.amibroker.com/guide/x_performance.html
https://www.amibroker.com/guide/h_multithreading.html

As long as your data source is capable of timely delivering data for as many symbols as you wish, AmiBroker is certainly able to display charts for those.

Tomasz, thank you for correcting my misunderstanding.

But when I run an exploration that requires a lot of calculations, my GUI slows down until the exploration is complete. And if I have multiple explorations running, it gets even slower. When I stop the explorations, the GUI returns to its normal responsiveness. Is this normal behavior, or is something wrong with my configuration?

It depends on your exploration (formula). If your exploration produces lots of output, this output must be presented ON SCREEN. Formula execution happens in WORKER thread. But you don't see that - it happens behind the screen. Anything that is displayed on screen is the UI. And it is NOT done in worker thread. Worker thread executes your formula and produces output that is IN RAM in digital form (not human readable). It means that after your formula was executed, the results are passed to GUI thread that is responsible for presentation of output data and is the only thread that in Windows can talk to GUI. If you have thousands of exploration lines it means that all those output data must be formatted as human readable letters and added as items to LIST VIEW. LIST VIEW is Microsoft code. It is not performance king. I am using "virtual data" and "owner draw" that makes Microsoft list view about 100x FASTER than "normal" listview, still with lots of data to be displayed ON SCREEN (in list view control), it obviously takes time.
Also your computer is not made of elastic gum. It has limits. The RAM has certain bandwidth. No matter how many cores your CPU has, it still has one and only one shared RAM. Multiple threads trying to read/write from RAM would fight for that shared bandwidth. And RAM bandwidth is actually so low that single thread can fully saturate it.
Now note that you might have say 16 worker threads that do the job and only 1 UI thread that outputs results. If you are producing lots of output it means that your UI thread will get busy.

Try copying result list from AmiBroker exploration (Ctrl+C) and paste it to Excel. Notice how much time it will take in Excel.

That's because you are deliberately overloading the system. A SINGLE exploration, by default, uses ALL CPU capacity to run it as fast as possible. If you want to cause less load and you have to tell it to the system.

You might slow down things. Add

#pragma maxthreads 1

then less output will be produced in a given amount of time, since only one worker thread will be producing results.

Again, without specifics, formula, screenshots showing exploration output etc, it is impossible to give any serious advice. There is nothign like "one size fits all". Everybody's case is different, there is no "generic" advice that fits everybody.

1 Like

This is very helpful. I was already using #pragma but I hadn't realized how the GUI thread handles the output, and one of my explorations is a long list with lots of custom formatting. This explains a lot. Thank you!

Actually, text formatting, memory allocation and pretty much everything connected with preparing human-readable text for exploration is done in worker thread(s). The GUI thread only receives ready-to-use array of pointers that point to individual rows of already prepared text to display in list view in 'virtual data' mode (no data copying occurs). This is absolutely ultimate solution speed wise. AmiBroker can output easily millions of rows per second in exploration into ListView, something that is absolutely impossible without all that magic that happens behind the scenes and nobody is aware of.

3 Likes

Tomasz, I really appreciate you taking all this time to provide this valuable information. I look forward to using all that to improve my formula and set up. To answer your question, the 50 charts is not to look at but to run my formula more often that 1 s compared to Exploration. 50 charts was also my reply to Hari. I personally aim to use more than that. It's part of my strategy. Going back to charts vs Exploration, I honestly did not provide a screenshot because from my own experience in trading and working on a trading floor, I know the open is very volatile. It just was obvious but I confirmed by observing my fills over a few days. Here's a screenshot of a recent day. The fills are generally wider at 9:30 than the rest of the day. So if Exploration could be run more often than each second, the fills would possibly be better in my humble opinion. Don't get me wrong, Amibroker is a freaking rocket already and using what you said about charts, I'll be able to sort my formula out to run many charts. I certainly intend to understand AND use all the math discussed here.

Hi Jase,

I perhaps don't really understand what it is you are trying to do, but I think having 50+ charts open simultaneously trading, straight off the chart, is probably a recipie for disaster. It would be diffiocult to keep a track of, but you could do it but I'd really suggest spending a lot of time, having a clear and precise understanding of exactly what is happening with your code execution in the background.

I'd suggest you make sure that you code in some alerts so that you know what trades your system is making and I think you'd need to find a way to ensure each trade triggered only once given a certain scenario, not repeatedly until condition no longer true.

To my understanding, charts continually refresh and if you have a condition which triggers to launch a trade, that will continue to repeatedly execute while said condition is true. It would be very easy to lose control very quickly if you had multiple simultaneous trades going and that could get very stressful real quick.

Im just trying to think of something which might be helpful to you. I reckon you might benefit, if not already, in learning how to use the batch functionality in Amibroker which will help you extract data from your explorations to launch your trades.

If you have a programmer on your trading floor, it should be straightforward enough for them to write you a small programme to launch your trades at the broker via API