There is a limit in the Amibroker built in MC analysis of 100 Percent of Equity per position at the moment, even for systems with margin enabled. So if Safe-F happens to be above 100 percent, you won’t be able to calculate it or CAR25 without custom code.

# Safe-f parameters: a check of the resuts

**howardbandy**#23

Greetings –

Please be aware of:

- In-sample results have no value in estimating future risk or profitability. One of the drawbacks of using individual decision tree models is their propensity to overfit.
- Using the percentage change over a full trade will underestimate risk and overestimate profit. Rather, I recommend use of state signals and mark-to-market daily. This is particularly important with issues that have high volatility. And also with trades that are held a long time (more than a week).

Best regards, Howard

**howardbandy**#24

And – importantly – data values drawn from a sample set into a Monte Carlo simulation must be independent and identically distributed – iid, in the jargon. This condition is usually met when working with a single issue. It is definitely not met when applying the technique to a portfolio of issues traded by the same model, and the results will be misleading.

Best, Howard

**alp.dogu**#25

Dear Howard,

Thanks for this detailed explanation as I find it most interesting. Since we trade utilizing Amibroker as a tool, I am sure this is a good platform to find traders who appreciate importance of most advanced technology in their trading routine.

In regards to ML, NN and AI - would you please name a “**trading premise**” or two that could be run utilizing this latest technology ?

As other trading premises such as momentum (cross sectional or time series), mean reversion, pairs trading, divergence, volatility trading etc etc are not necessary dependent on this latest technology and I am keen to see a trading premise that would require utilizing this latest advancement.

Many thanks,

Kind regards,

AD

**howardbandy**#26

Hi AD, and all –

One of the difficulties some traders have when moving from traditional platform development to machine learning is that the concept of the trading system itself changes. From “compute an indicator, then see what happens after” to “identify something important, then see what happened before.”

When developing trading systems in AmiBroker, we begin by computing an indicator, such as RSI(2), and associating a rule with that indicator, such as Buy = Cross (30,RSI(2)). That is, we compute an indicator, define a rule that gives Buy signals, then evaluate the result of the trades that follow.

For machine learning, we begin by defining the target. Think about my questions to myself – “What would I like to know about tomorrow? What would I do if I knew that?” I know it is correct to think about tomorrow, because I am using state signals and mark-to-market management. I do not create targets that overlap or that are more than one mark-to-market period in the future. That is tomorrow.

The target part comes from “what I would like to know about tomorrow?” A reasonable question to begin with is “Will tomorrow’s close be higher than today’s close?” If I knew it would be, I would buy, or continue to hold, from today’s close to tomorrow’s close – and then re-evaluate – always for a one-day lookahead. Since I am using state signals, the state of the target associated with that question is either beLong or beFlat. In afl, beLong = Ref(C,1) > C.

Next, we add some predictors. Each day (or bar) has its own row. Each predictor has its own column. If we think RSI(2) is important, compute it, and give it its own column. We plan to let the model identify which patterns that precede the target are important. If we want the model to consider a falling RSI, the data for each day’s row must contain both today’s value and yesterday’s value. Let’s call today’s value RSI and yesterday’s value RSIY1. If it turns out that the RSI falling through 30 is a signal (in the “signal and noise” sense), the model will notice that signal as a day when RSIY1 was greater than 30 and RSI was less than 30. Importantly, we do not choose the value of 30 – we let the model fitting procedure find that by itself. Add other predictor variables we think might be useful, possibly including some history of each. And so forth.

The point is that the trading system has disappeared. There is no longer a set of rules defined by the developer resulting in Buy and Sell signals. Rather, there is a target defined by the developer and some number of possibly useful predictors.

Concepts of trend following, mean reversion, divergence, flag, trend line, any of the many indicators you are already familiar with, and whatever other chart or technical analysis techniques have been used in the past must be encoded in some way into one or more predictor variables.

It is difficult to compare a traditional system that buys when RSI(2) falls through 30 with a machine learning system because the machine learning model fitting process identifies whatever corresponds to the “falls through 30” part itself.

Best regards, Howard

**howardbandy**#27

To AD –

Not require. If the trading systems developed using AmiBroker are satisfactory, there is no requirement to change to machine learning.

The benefits of systems developed using AmiBroker are significant – no new learning curve, ease of understanding, great support, …

Best, Howard

**cesare**#28

Mr Jani, I wish to start with my grateful thanks for your work, your effort and your generosity to share your code with the forum .

I haven’t fully understood the choises you made with the code, specially the issue about trade or daily change: but giving that I am now studying portfolio systems with few items (up to seven/nine at most) and a fixed monthly (or weekly) lenght , I believe your work fits perfectly to my purposes. I only added minor changes with a parameter input table, and a custon output that is more similar to my habits.

And of course I’ll very eagerly appreciate the possibility of a new code for a computation based on the daily equity variations, more similar to Dr Bandy tought. A code like this is beyond my capabilities, but I belive to be not the only one interested.

I am specially grateful for the explainations offered to use the AFL : after all, I was using the results of pag 139, in the ‘prospector’ area, so perhaps part of the difference between my results and those of the book is explained by the trade/daily equity issue … I’ll try for it.

many many thanks again

Cesare

**cesare**#29

Thanks Mr Greeyleaf: your suggestion is very appreciated :

I was using ‘non-Total-Return’ data , as you argued, and therefore the difference is perhaps due to dividends … I haven’t tought of it

thanks for your kind help

Cesare

**cesare**#30

Dr Bandy, thanks for your very deep toughts about the statistic involved in the backtest of a system (I specially appreciate the analogy of the rivet gun, as being myself a modeler)

and is very interesting the machine-learning development: not in my possibilities for now, but I had a glimpse to a new and poteliaaly revolutionary area of development

Thanks for share with us your clear and deep reflections

Regards

Cesare

**howardbandy**#31

The Monte Carlo simulation relies on the sample set from which the individual data points are drawn – such as daily percentage gained – are independent and identically distributed. As you read modeling, simulation, and statistical articles and texts, these two conditions are abbreviated as “iid.”

If these two conditions are not met, the analysis will not accurately represent the forecasts made by the simulation. The analysis will be biased. When analysis has a bias, it is always unfavorable.

Independence ----

Daily percentage gained data is independent when the trading system being analyzed is a single ticker. I recommend beginning with long/flat, or short/flat. Conditions that accurately predict good long trades are different than those that predict short trades, so it is easier to code, test, and validate two separate systems. Although the program will be more complex, trades resulting from a system that trades a single issue long/flat/short will also be independent.

Portfolios consisting of several issues, whether the holding is a single issue at a time or multiple issues, result in dependencies that are very hard to analyze. Results will not be accurate.

Similarly, time forecasts that overlap are not independent.

State signals that are always one evaluation period are independent. (If you find they are not, your system will be better if you can redesign the rules to take advantage of the dependency.)

Identically distributed ----

In order to be identically distributed, the distribution must be stationary. That is, data points from any period, say a year, must have the same distribution as data points from all other periods that contribute to the set being analyzed.

The stationarity we care about is with respect to the metric – pattern or trade – that is being analyzed. Not necessarily mean-stationary or trend stationary – in fact neither of these will be satisfied for stock prices or price changes, whether raw or when differences are used.

In order for a trading system to be profitable in live trading, it must be profitable over a period long enough to determine the parameters in-sample, then those patterns must continue out-of-sample. The in-sample data is always overfit. We have no way of knowing what the risk and profit potential are until we make an out-of-sample run and analyze those results.

Successful systems rely on “The future must resemble the past.”

The system must be stationary over a period that is as long as the in-sample time period plus the out-of-sample time period. This “period of stationarity” will change. The signs of change are that out-of-sample or live trading results are poor. Take this as a warning that the model and data may be falling out of synchronization, and reduce position size.

Best regards,

Howard

**howardbandy**#33

Refer to the AmiBroker reference.

Ref (C,1) uses the value of the Close of the bar that is more recent than the current bar. It refers to a future bar. This is true whether in backtest of historical data or live trading. At today's close, you will not yet know what tomorrow's close will be.