Data Plugin - Design discussions

Thanks, I got it. As I said in first post, that's what I am trying to refine, and all should be done in the plugin.

@Tomasz
This is 2 part Q for you:
In my quest to find a robust backfill design, I have certainly walked into your IP rights for AFL DateTime struct ( I codenamed it DT32 ) for now.
I will have to share details in private channel, deriving from the open sourced AmiDate ( DT64).

The other part is for AFL DateTime32, from usage in formula, "similar to something I came across" its bits max out at the end of 2026 so the _DT() will give the same message "Accepts string upto 2027-01-01".
So is this currently fixed to its epoch from 1995 in the software or would it dynamically change or similar logic. Referring to old builds like ver 6.0x64 etc. For newer versions obviously flexibility remains.
Thanks

Version 6.00 will be removed from the web site pretty soon. It is almost 10 years old and obsolete and not supported any more.

Single precision float packaging of date time is completely internal (opaque) and subject to change in each version simply because there is not enough bits. The plugin interface does NOT USE float packed date times.

Just use 64-bit AmiDate that has enough bits to represent dates upto year 4095 with microsecond precision and it is there to stay without changes upto year 4095 :slight_smile:

1 Like

I am aware but you have called this IP :smiley: hence I am asking.
Is there a consent to use your logic because in future one cannot say if code is redistributed in some form.

The DT64 is primarily used but just for plugin internal purpose DT32 sits well in the float fields of Struct Quotation.

Forward packing from epoch is simple, 5 bits year and MS-bit as 0 and rest of the fields as they are.

The only difference in packing was pre epoch "which has to be" reverse chronological with MS-bit set 1.
All fields, for example month, I packed as (12 - nMonth) with 4 bits whereas your packing is with highest bitfield value which seems more elegant. ( I'm still hesitant to elaborate without consent :wink: ) Thanks

No. Quotation structure DOES NOT use float datetime. And it does NOT use anything what you are writing about.

Again use AmiDate structure it is well documented, not really any IP involved as this is just plain 64 bit integer.

The whole "problem" is not existing really.

The DB design i used is a non-persistent internal plugin DB to hold realtime quotes which is a jagged 2D-like array of Struct Quotation.
My plugin DB is: (originally std::vector but changed to CArray)

typedef CArray< struct Quotation, struct Quotation > CQuoteArray;
// Single Array of Quotations <class TYPE, class ARG_TYPE>

typedef CArray< CQuoteArray*, CQuoteArray* > DB_rtd;					
// Array of Array(Quotations) for each symbol like Jagged Array

// And a Map< (String) Ticker, (Int) Index > SymMap;

At element index 0, I'm using fields to store that ticker "meta-data". So plugin can use any field for another purpose. (I have 1 DT64 and 8 floats) mostly for index/timestamp of some events like GetQuotesEx() call, BlendQuotesArray(), Backfill() etc.
From [1] to nArraySize new RT quotes(bars) stored then it acts like ring-buffer.
Like in your case DT32 is internal, similar internal tracking need for the day.
Then its all pointers, ie. ptr->ptr->data or pt->data etc field(s)

Once other parts/bits are added I'll send you the code :slight_smile: still some moving parts right now
For you, Full view is not clear yet so I am causing some confusion, but in summary i wanted to use your DT32 logic for storing timestamp. Thanks

My intention is to build something that is currently working for me but to expand AB functionality for others who may not be able to write themselves. Hence, this communication also not private and thankful for your support.

I advise you to NOT use 32-bit timestamp. It is obsolete already.

1 Like

Thumb rule to follow Windows/MFC > STL > Boost etc
So by switching from std::vector to MFC CArray, using memcpy() is the fastest way in the non-overlapping universe.

in your experience, would this impact performance because some manual tests dont really show any degradation.

// 40-bytes 8-byte aligned
struct Quotation {
	union AmiDate DateTime;			 // 8 byte
	float   Price;
	float   Open;
	float   High;
	float   Low;
	float   Volume;
	float	OpenInterest;
	union {								// Anonymous union	8 
		DATE_TIME_INT	  DateTime2;
		struct PackedDate PackDate2;
		
		struct {						// Anonymous struct 4+4
			float	AuxData1;
			float	AuxData2;
		};
	};
};

With Anonymous, except "visual complexity", flow seems to be as usual.


Everything remains the same, as Author, one should be aware as to what is being coded.
Nothing is disturbed with regards to Memory layout or alignment, even if some more fields extend same functionality.

I am not exactly sure what you are trying to achieve with that second union at the end.

For a few reserve elements at the "beginning" of the array, they would be my metadata fields for that ticker as explained in previous post instead of maintaining a separate data structure and tracking it.
Even a ptr nowadays is 64 bit.

I would store and copy real quotes only from index[2] to the arraysize and memcpy() into pQuotes when GetQuotesEx() calls.
But the reserved elements would store 64-bit timestamp over there. The first few reserved elements only for metadata, not real quotes and for plugin only use.
Now I have option of 2 DT64 as required.

This is in response to abandon the idea of 32-bit timestamp as you advised.

I don't see any advantage of doing this instead of just keeping

struct MyInfoThatHoldsWhateverINeed
{
   __int64 whatever;
   __int64 whateverelse;

   struct Quotation *pQuotes; // if you prefer pointer
}

or

struct MyInfoThatHoldsWhateverINeed
{
   __int64 whatever;
   __int64 whateverelse;

   struct Quotation aQuotes[1]; // dynamically sized array (having at least one bar to keep compiler happy)
}

I don't see any reason to pollute irrelevant data into Quotation structure

1 Like

Thanks, I see your point.

Writing something generic is N times harder :stuck_out_tongue: (one size fits all). Back to the drawing board.