Persistent logger

The code below implements a simple persistent logging framework written in pure AFL. I would be interested to hear ideas to improve the efficiency and robustness.

// This code is a persistent logger with the capability to control logging severity. Multiple
// threads can write to the same log file.
//
// The implementation requires each thread (stock) to separately open and close the log file.
//
// Example Usage
// -------------
// LogOpen("D:\\Trading\\Logs\\%Y-%m-%d.log");
// LogSetLevel(LOG_INFO);
// LogMessage(LOG_DEBUG, "Debug message 1");  // This will not be logged because the logging severity is INFO
// LogMessage(LOG_INFO, "Something noteworthy happened");
// LogMessage(LOG_WARNING, "Stocknum=" + Status("stocknum") + " Name=" + Name() + " warning condition");
// LogClose();


LOG_DEBUG = 1;    // Highest level of detail
LOG_INFO = 2;     // Log noteworthy normal events, warnings, and errors
LOG_WARNING = 3;  // Log warnings and errors
LOG_ERROR = 4;    // Only log errors
LOG_NONE = 5;     // Disable all logging

logLevel = LOG_DEBUG;
logFileHandle = 0;  // Global log file handle
criticalSectionName = "";  // Global critical section name


//////////////////////////////////////////////////////////////////////////
// Enter critical section
//
function LogTryEnterCriticalSection()
{
  global criticalSectionName;

  locked = false;

  // try obtaining semaphore for 1000 ms
  for (i = 0; i < 2000; i++)
    if (StaticVarCompareExchange(criticalSectionName, 1, 0) == 0)
    {
      locked = true;
      break;
    }
    else
      ThreadSleep(1); //sleep one millisecond

  return locked;
}


//////////////////////////////////////////////////////////////////////////
// Leave critical section
//
function LogLeaveCriticalSection()
{
  global criticalSectionName;
  StaticVarSet(criticalSectionName, 0);
}


//////////////////////////////////////////////////////////////////////////
// Open log file, appending if the file is already existing.
//
// Parameters
// ----------
// logPathPattern : string
//    Log file path with optional user-defined datetime pattern.
//    The directory must already exist.
//
// Examples
// --------
// LogOpen("D:\\Trading\\Logs\\%Y-%m-%d.log");
//
// See Also
// --------
// https://www.amibroker.com/guide/afl/datetimeformat.html
//
procedure LogOpen(logPathPattern)
{
  global logFileHandle;
  global criticalSectionName;

  logFilePath = DateTimeFormat(logPathPattern, Now(5));
  logFileHandle = fopen(logFilePath, "a");
  criticalSectionName = "Logger" +  GetFormulaPath() + logFilePath;
}


//////////////////////////////////////////////////////////////////////////
// Close log file.
//
procedure LogClose()
{
  global logFileHandle;

  if (logFileHandle != 0)
    fclose(logFileHandle);
}


//////////////////////////////////////////////////////////////////////////
// Set the logging severity level.
//
// Parameters
// ----------
// level : int
//    Minimum Level of messages which will be logged. One of LOG_DEBUG, LOG_INFO, LOG_WARNING, LOG_ERROR.
//
procedure LogSetLevel(level)
{
  global logLevel;
  logLevel = level;
}


//////////////////////////////////////////////////////////////////////////
// Convert integer log level to string.
//
// Parameters
// ----------
// level : int
//    Level of this message. One of LOG_DEBUG, LOG_INFO, LOG_WARNING, LOG_ERROR.
function LogLevelStr(level)
{
  switch(level)
  {
    case LOG_DEBUG: result = "Debug"; break;
    case LOG_INFO: result = "Info"; break;
    case LOG_WARNING: result = "Warning"; break;
    case LOG_ERROR: result = "Error"; break;
    default: result = ""; break;
  }
  return result;
}


//////////////////////////////////////////////////////////////////////////
// Print message to log file if the requested serverity level is at least
// as high as the currently configured logging level.
//
// Parameters
// ----------
// level : int
//    Serverity level of this message. One of LOG_DEBUG, LOG_INFO, LOG_WARNING, LOG_ERROR.
// text : string
//    Message to write to the log file.
//
procedure LogMessage(level, text)
{
  global logLevel;

  if (level >= logLevel && logFileHandle != 0 && LogTryEnterCriticalSection())
  {
    timestamp = DateTimeFormat("%Y-%m-%d %H:%M:%S", Now(5));
    levelStr = LogLevelStr(level);
    fputs(timestamp + "\t" + levelStr + "\t" + text + "\n", logFileHandle);

    LogLeaveCriticalSection();
  }
}



1 Like

I am not quite sure why don't you just use built-in file sharing mechanism that works without critical section as described in Knowledge Base:
http://www.amibroker.com/kb/2016/01/27/how-to-write-to-single-shared-file-in-multi-threaded-scenario/

@Tomasz thanks for the suggestion. I tried using fopen(shared). It is simple to use and worked fine with a small watchlist, but when I scaled up to thousands of stocks, I experienced numerous fopen failures and slow afl execution.

I also tried to share a single file handle across all threads by storing the handle in a static variable, but that approach completely failed for multiple reasons.

I’m not entirely happy with my implementation because of the frequent file open and close operations, but the performance is better than I expected. I’m sure you could implement something far superior using techniques that are not available in afl. (Not a feature request by the way, I’m sure you have more important development activities.)

Not directly related to your topic but when there is a lot of file access, one can definitley try a RAM Drive
Many tools/applications are available, and RAM access is many times faster than other storage devices.

Yes shared fopen is the solution if you restructure the code so it is not opened and closed with every line of log output.

Also you can use _TRACE instead. It can produce output to AmiBroker's Log window (that you can later copy-paste and save to file) or it can produce output in DebugView where it can be logged to file automatically, see:

image

image

3 Likes

Didn’t know DebugView could log to a file. Thanks for the tip.

Anyway I think I will add native functionality for easy high-performance logging directly to file.

5 Likes

Steve, DebugView++ is very useful and by the way nobody prohibits using it in a less obvious way. For example I've been using it as a " Tape recorder :wink: " to display aggregated (in AFL) Bid/Ask transactions, hidden orders (UK/US in the example below) etc. I intercept such data (and save in Static variables) using AmiBroker, during the session, on some less heavily traded issues (where that kind of data is unavailable in any other way) and it works well. When the session is over I save it and have a long history of such files which I can "replay" whenever needed. It is just an add-on to the AFL visualizing such statistics on the chart.

Bid-Ask

7 Likes

Forgive my intrusion to the discussion here. Good share Milosz!

Curiosity provokes me to ask. When you say:

You mean using the buit-in File IO functions, right? But in that case, isn't the frequent file read/write a bit costly? How do you avoid that?

Thank you

No, I simply use _TRACE() to output live (processed and aggregated in AFL) data to DebugView++ (and I save the file only once a day in *.dblog file) and at the same time StaticVarSet() or StaticVarAdd() to store raw intercepted data in AmiBroker. So it's a two way split.

I only make sure that Permanent Static variables are saved periodically not to lose data in case of any failure. For instance:

SetOption( "StaticVarAutoSave", 120 );

1 Like

Your recent edit clarifying the use of Permanent Static variables answered my next question. Actually, apart from debugging, I never thought that the _Trace() function could be used this way.

Very smart trick indeed! Good one!

Thank you

2 Likes

Hi

May I know where do you intercept bid/ask data from? is it from the online trading platform or from Amibroker data plugin provider?

Thanks.

@newbietrader,

Saving some wait time of yours before Milosz replies. :slightly_smiling_face:

Actually he has already answered this question of yours, since he mentioned:

As he is using native AmiBroker _TRACE() function to output, there is no question of any other external source to intercept data from, hence, it is a Plugin.

1 Like

Hi

Does that mean, the data plugin must already have bid/ask data in column (OpenInt/Aux1/Aux2)? currently my data plugin provider only provide OHLCV and OpenInt (Value). It is EOD data that is being updated every 10 minutes.

Thanks in advance.

Yes!

Not necessarily that the Bid/Ask Price would be streamed as Aux1/Aux2 arrays only. It totally depends on the 3rd Party data provider on how they facilitate. Alternately, one might be required to use GetRTData() function in order to capture such data source:

For e.g.:

"Bid = "+GetRTData("Bid"); 
"Ask = "+GetRTData("Ask"); 
"Last = "+GetRTData("Last"); 
"Vol = "+GetRTData("TradeVolume"); 

"EPS = "+GetRTData("EPS"); 
"52week high = "+GetRTData("52weekhigh");

(Code Source: GetRTData Function guide)

2 Likes

My plugin doesn't provide any Bid/Ask data and we can't access AmiBroker's Time & Sales Window's statistics (from AFL), so yes (as @Cougar wrote) I'm extracting Bid/Ask transactions analysing data coming from GetRTData() functions. Yes I know it's not a perfect solution - it's only an approximation because of a snapshot nature of such data (in my case acquired up to 10 times per second), but I'm happy with it.

If you are asking about my trading platform - I use it to extract and import to AmiBroker information about aggregated (up to the 5th line/level) resting buy/sell orders. It lets me analyse how resting orders imbalance was changing during the session - no need to stare at the screen all day. An example of such data stored in a permanent static variable:

Delta%20Resting%20Orders

Sometimes the conclusions can be very interesting. Hidden or repetitive orders, substantial difference between accumulated buy/sell resting orders, diverging Bid/Ask Delta line and similar things help me better read true intentions of the market participants ...

6 Likes

To be precise, my plugin doesn't provide historical Bid/Ask data, but gives me access to current Ask, AskSize, Bid, BidSize, Last price, last TradeVolume, ChangeTime etc. so during market hours one of my AFLs is constantly storing these values and when a trade happens, it compares current plugin readings (acquired via GetRtData()) with the previously stored values assigning each trade to Bid or Ask. I also implemented several correcting mechanisms taking into account possible delays and other problems that might happen. I store such Bid/Ask data in permanent static variables and use in many different ways in other AFLs.

2 Likes

Sometime back I attempted kind of similar process to determine Buy/Sell Volume (however not using your _trace() method) by running a tick-chart (without any plot) side-by-side to store the tick data onto permanent StaticVars, then retrieve the same onto a Higher Timeframe Chart for Plot visuals.

Obviously, it was rudimentary and neither I day-trade everyday nor own a Cloud VPS. Needless to mention, on days when I did not login or somehow got late from work to miss the night session open of few Indices, there would be historical gaps which evidently became a bit of awe to my sensitive eyes. Then, there came a day when I totally gave up this idea!

Moreover, it was technically not possible for me to capture the same for more than 10-15 symbols at the same time. So, in a middle of a session if I would intend to check a new Symbol (not enlisted beforehand), then there would be no historical Bid/Ask data available for that specific Symbol.

Hope my weekend babble makes any sense.:laughing:

Do you face such issues?

2 Likes

@Cougar that's interesting what you wrote - I agree. This method is demanding because you need to collect data on your own during session - there's no backfill option. So people who have access to sources providing historical Bid/Ask data are in better position. On the other hand if such data is widely available and many people make use of it - it looses its advantage. Information is really important when it gives you an edge over other competitors. On my market historical Bid/Ask data is generally not available for individual investors.

Is it worth the effort? If you witness a situations in which a futures contract is trading flat all day - so at first glance nothing interesting is happening, in comments you read about another boring session without history, but thanks to collecting additional data you are able to see Bid/Ask cumulative line declining all day, impressive buy resting orders making false impression as if there was much more real buying power and identical (or similar) sell orders repeating every x seconds for many hours, than it's worth it because you know that something bad is going to happen and you can adjust your position accordingly. You can clearly see a big player building (in a smart way) a short position all day. Of course sometimes this additional data doesn't provide useful insights.

Because collecting such data is inconvenient I do it only for selected futures contracts and a few stocks which are currently on the spotlight. It is just another tool helping me analyse issues which cought my attention for some other reason. In general I'm news or event driven stock/futures trader. Not technical analysis, but news flow coming from the company (or its surrounding) and fundamental changes are the most important for me. When it comes to TA, I pay special attention to volume - I try to dissect it in all possible ways. I don't buy stocks because of moving averages crossovers or some other indicator readings :wink:

2 Likes

Thank you very much Milosz for sharing your thoughts! Had the impression that I am alone with this problem. You said it all very meticulously. Totally agree to every letter that you have posted and on the same boat with you. Tried using other platforms who "claim" to provide such graphics but that's just their "claim" only, as usual, reality remains different. I believe, everybody's needs are different - there might be some commonalities in between but the structural framework to solve a problem would diversify. So, expression in any terms would differ. And honestly AFL is a boon to Humanity (not an emotional outburst, I mean it, although just surfing the surface of it :smile:), and under no circumstances I can afford to think otherwise.

As of today, I am seriously considering to own a VPS. Since Cloud security would be one of my major concerns (social media posts like this scare the __ out of me). Years ago, a close one to me suffered identity theft (long story). So, vulnerability needs to be checked first.

For obvious reasons, being a simple commoner like me cannot afford to buy leased connection to a Pro-Server directly from the Exchange like the Big Boys. Neither do I have the brain nor money to pull-off such a thing. The only other alternative would be to go for a VPS such that things could go-on 24/7. Considering Security, computational Speed, Software rights, it's going to be expensive but hopefully would be able to afford it in sometime.

Anyways, Market and time has taught a lesson, to stay selective while short-term trading rather than trying to find trades in everything. So, on that ground, I totally agree with you to stick to fixed set of Indices or Stocks or Fx pairs only for observation and strike at opportune moments only, instead of beating all that ticks. Still peace of mind is required when it comes to data, especially, when it is something as important as Historical Bid/Ask - a lot concrete conclusions can be drawn out of it and Makers know that very well like you pointed out.

Sometime back my Broker got irritated off my frequent phone-calls on how things work and gave the privilege to look around their prop. trading desk along with some other curious souls like me and the Traders were real. A Trader spoke exactly the same what you have written.

BTW it got me little frustrated as several meanings can be drawn from the same thing which is similar to other indicators too and I could not draw anything conclusive with any certainty. I mean, let's say, when the price is making Lower-Lows or consolidating at Previous Low but the Cum. Delta line is rising, intuitively, I would go Long but the Trader explained that I could be wrong too, as Makers might make it look like that. And on some other day, the theory might hold true - the Market might actually go up. Hence, I dropped the idea because I could not get hold of the "subjectivity". May be I lack intuition!

3 Likes