I am happy to announce that after a deep educational journey, a first test release of a data plugin that will allow users to simplify a data connection with any Vendor or Broker that provides Ticker data via APIs or Websocket.
There is a technical element involved which will require users to build their Client side application according to the Vendor-client Library and format the data to meet the plug-ins specifications.
The design would be a general Relay server that listens to a port and Amibroker connects to it as a receiving or consuming client.
The user application would just have to send the data to the specified websocket port.
Alot of thanks and wishes to Amibroker and Dr. Tomasz for all the help and guide. Interacting with him has been insightful and to get approval for the source code so that it is compliant.
Because this is initially a closed-beta group, you can DM me if you are interested in such a solution for your needs.
Soon, I can release the Documentation for public viewing as I don't know the volume of queries that might come along.
@nsm51, congratulations: it definitely wasn't an easy task!
I think that in your documentation, before going into technical details you should include a high level overview with one or more diagrams to better understand how the system should be configured, also listing examples of exisiting (free or commercial) data sources for the clients (although it is up to the user to write the client code and the connection to any specific data source)
Even better if you use sequence diagrams with the "actors" and example scenarios of exchanged "messages".
Yes, i also realized that later. Original plan was to bundle a zip file, but i forgot about that from last month. done it now.
There was a small issue with how github renders the MD file vs dillinger. Alignment is not the same.
Yes, certainly very hard especially when you haven't used a strongly typed language in decades back java/C# was very popular.
Your feedback is noted to get diagrams but actually this an extension of the ADK. So maybe i can make some references to the ADK.
Regarding vendors, I still need to survey but here almost all our local brokers have begun providing websockets/api. Users from other markets could share their doc links here.
So every such Broker has their well-written documentation, and all have sample working code on github.
Secondly, they have their libraries in multiple programming languages.
Python, Node.js, C# etc
Basically, the struggle for a normal user is to write a data plugin You need thorough understanding of data plugin design of AB. in C++
Now if you see, the Brokers' developers have made the Client side very easy so now the user would require minimal effort to import vendor-specific client library and write a small wrapper that builds bars. (this is very important for now)
Then, we just relay the same data on 127.0.0.1 to where the client is connected.
I have the RELAY server also ready, with source code available so yet one more step less.
Even those users complaining about slow ODBC plugin can give this design a shot.
Relational DB and RTD+AB are worlds apart. We are talking a few thousand concurrent ticks per second
Either you use super-fast AB DB directly, or maybe something like ArcticDB which has genuine performance benchmarks for "Time-series data". Its only drawback is it has Python interface. Also, GIL does not become a real problem where we have high i/o throughput.
With some shortcuts, the plugin would be closer to 1MB. Packing it under 150KB requires alot of clever choices and that is what the other libraries i used do. They all work on C++98/C++11. smiling...
From a user perspective, my understanding is that sample_server.py is the RELAY server in the diagram above? If so, my understanding is that handler() initiates recv() via an asyncio task and recv refers to messages coming from the .dll and that
In recv()...
if 'cmd' in jo:
if 'arg' in jo:
if jo['cmd']=='bfall':
print( f"bfall cmd in {mr}")
elif jo['cmd'] in ['bfauto', 'bffull']:
print( f"bfauto cmd in {mr}")
sym = jo['arg'] if ' ' not in jo['arg'] else (jo['arg'].split(' '))[0]
jo['arg'] = f"y {sym} 2" if jo['cmd']=='bfauto' else f"y {sym} 5"
"cmd", "arg", "bfall" and "bfauto" are all strings that I can simply ignore?
From a user perspective, I simply want to clarify something.
Am I;
a) writing a function that calls data from an endpoint, parses the data and then returns it to a function (??handler()) , and if so, where would be the best place to call it?)
or
b) writing a python server that publishes parsed data into localhost:10102
or
The Sample_server.py is the Scenario 2 object in the diagram.
If a user chooses this model, he should implement the websocket server and the vendor WS specs.
I am yet to upload the Relay_Server and a Sample_Client, I might as well do it and let users patch it if it is buggy
They both work, but all edge cases are not tested.
For a user, even in sample_server, you just need to replace 2 parts, 1st is RTD Generator and 2nd is Backfill Generator
Hopefully over time, users will contribute enough Client code that wraps the vendors specs.
Congrats for your efforts.Do you have any broker specific activity that needs to be done on the part of users. Any documentation in this effort for non technical users. Like Dummies Guide on How to use ??
I have just added 2 more py source codes. 1 is the Relay server.
One is an outline of the Client. For now my focus is on releasing the plugin so I don't have vendor library specific code.
The hope is for a community effort to wrap as much vendor specific code into the codebase in as many languages as possible.
The client sample needs to include 2 vendor specific codes, on for RTD and another for backfill that meets their specs.
@nms I'm reporting a minor issue in the plugin's UI.
Today I was testing the "relay" and "client" code. I launched them both on port 10101 (by default).
So I launched AB with the database configured for port 10102.
Obviously the plugin didn't connect.
I then changed the plugin configuration to port 10101.
The connection still doesn't work.
I selected "ABF reset" and the RT data comes in (false), but the plug-in status indicator in AB was still "red". The "Connect" option remained disabled (grayed out).
I had to press "Shutdown" and then "Connect" to make it turn "green" (by the way I suggest using a lighter shade of green like the other plugins do).
I hope you are able to replicate the situation, but in any case it is not a big deal since I assume restarting AB after changing the config would have worked anyway.
You need to check DebugView.
So basically there are 2 scenarios:
critical memory alloc size change
Max Quote Size if changed, DebugView will say changes take effect "only after AB restart"
Some settings take effect only after shutdown/connect. ie. reconnect
so port cant be changed while running. If plugin detects a change in such settings, it ill log and ask for reconnect.
Dark green is a status, that tells the user that they need to click "RETRIEVE" in Configure to import some new symbols detected.
Then it will go to light green.
This in already in the docs.
The reconnect attempt for WS is 15 secs, like QT plugin. That is when it stays red, with 3 reconnect attempts.
@beppe Thanks. I realize user is not going to check DebugLog so I have added a non-intrusive Message Box that informs user changes will take effect when AB Restarts ( if certain scenario is matched) or Plugin reconnect is required.
This is only 1 condition, when MAX_Size_Sym_Quote is changed and plugin DB has memory allocated.
The idea was not to interfere with the smooth AB UI, so this message box is conditional and after the ok Configure is clicked. It is similar to AB MesageBox, when base time interval is not supported.
( Code is added ), Thanks
@SpandexMan
You can check the client.py file, now the Websocket Receive() is truly async. (git update)
Websocket buffers should not be blocked. So at computing pace, the requests are pushed into a Queue and processed asynchronously.
This means Vendor/Remote server RATE Limits are also respected.
back java/C# was very popular.
