ValueWhen() and Null values

I noticed some interesting behavior with ValueWhen() today, and wondered if it is intentional and/or documented anywhere. The code snippet below fills an array named data with values from BarIndex(), and then overwrites some of those values with Null. It creates a second array named select which is true for any bar that is evenly divisible by 10. The exploration outputs these two values as well as the result of calling ValueWhen() with the two arrays as inputs.

data = BarIndex();
select = BarIndex() % 10 == 0;

fbir = LastValue(ValueWhen(Status("FirstBarInRange") , BarIndex()));
for (i = 1; i <= 20; ++i) 
{
	bar = fbir + 20 + i;
	data[bar] = null;
}

Filter = True;
AddColumn(data, "Data", 1.0);
AddColumn(select, "Select?", 1.0);
AddColumn(ValueWhen(select, data), "Selected Data", 1.0);

My expectation was that the ValueWhen call would always return the value of the data array when the select array is True, i.e. the value from every bar that is evenly divisible by 10. In other words, I would expect the array returned by ValueWhen to have some Nulls in it because my source array (data) also had some nulls. However, this does not appear to be the case. ValueWhen() returns the last value from the data array where the select array was true and the data array was not null.

Run the code snippet above against a single symbol and a date range of 50 bars or more to clearly illustrate the issue when running AB v6.30.5, 64-bit.

1 Like

(First of all, (as aside) code can be written loopless.)
Secondly, AFAICS it has been always been that way with every version of AB. E.g. run code in AB 4.8 or other versions earlier than 6.30.
Thirdly, Null in AFL has special meaning in AFL -> Empty. It is not zero.
So it is not recognized as number but it (Null) is recognized by e.g. IsNull() function check (or if converting Null to number then ->Nz()). ValueWhen has its functionality in its name already -> "return value when something is true".

Anyway to get result as desired I did like this

  1. converting Null in data to number (in example to -1)
  2. then applying ValueWhen(select, data)
  3. And then converting -1 back to Null
data = BarIndex();
select = data % 10 == 0;

fbir = LastValue(ValueWhen(Status("FirstBarInRange") , data));
data = IIf(data > fbir + 20 AND data < fbir + 41, Null, data);
data = Nz(data, -1);

data_select = ValueWhen(select, data);
data_select = IIf(data_select == -1, Null, data_select);

Filter = True;
AddColumn(data, "Data", 1.0);
AddColumn(select, "Select?", 1.0);
AddColumn(data_select, "Selected Data", 1.0);

Results in

5

6 Likes

Thanks @fxshrat. I am aware that Null has special meaning in AFL, and I am already using a workaround similar to the one you described. As for the function name ValueWhen() implying that Nulls are ignored and that only actual values are returned, I think that's a bit of a leap. If the function is indeed working as intended (and knowing @Tomasz, it probably is), then a small addition to the ValueWhen() documentation to describe this behavior would probably be a good idea.

It is not a leap.
Null simply is not standard number. It is not treated as value. As said its meaning is being empty. So it is skipped by ValueWhen.

You can not check array for being Null in this way:
data == Null

It will result in Null.

You can only check via special helper detectors such as IsNull():
IsNull(data)

Think of Null like of Vacuum (no matter in there).


data = BarIndex();

fbir = LastValue(ValueWhen(Status("FirstBarInRange") , data));
data = IIf(data > fbir + 20 AND data < fbir + 41, Null, data);

Filter = True;
AddColumn(data, "Data", 1.0);
AddColumn(data == null, "'Data is null' check (WRONG)", 1.0);
AddColumn(IsNull(data), "'Data is null' check (CORRECT)", 1.0);

5

1 Like

For what it is worth: for any AFL function, Nulls are only skipped at the BEGINNING of series (array). Once first non-null value is present in the array, Nulls appearing after that, in the middle of the array, are NOT skipped anymore.

ValueWhen is somewhat special in this regard as it handles Nulls in the middle in both input arrays. It has specific code that skips Null values all the way thru the array. Why does it do that? To ensure that it does NOT produce the output that has Nulls in the middle (because other functions don't expect that).

The semantics of Null is "having no known value at all" or "having UNKNOWN value". Null does not represent any specific value. It rather means the state when we simply don't know what the value is.

Also any operation involving Null produces a Null. So 5 + Null is Null. This also applies to comparisons. If you compare anything with UNKNOWN you simply don't know the answer. So if you compare number to Null, you won't get True or False. You will get NULL (which means "I don't know"):

// use that in commentary window
WriteVal( 5 == Null ); // will output {EMPTY}

The same if you compare one UNKNOWN value to second UNKNOWN value you just don't know what the result is, because Null is NOT really a number but the state when we say "we don't know anything about the value".

WriteVal( Null == Null );  // will output {EMPTY} too since Null means "I don't know"

So Nulls generally propagate through statements.

As written previously to test for Nulls you need to use IsNull() function

WriteVal( IsNull( Null ) ); // will produce 1 (true)

The general intention of Null use is plain fact that many indicators are UNKNOWN for given number of bars at the BEGINNING of the array. For example, 20-bar moving average value is unknown for the first 19 bars. So the output of MA for that initial bars is NULL ("no known value"). If you are then doing nested MA from MA : MA( MA( C, 20 ), 30 ); then second MA would skip first 19 bars with Nulls and start taking input form first non-null value.
So the nested MA would start producing non-Null values at bar 59.

This way you can nest any function as any function expects to skip Nulls at the beginning.

That is the primary use of Null.

Secondary use of Null is to mark bars when you don't want to Plot anything or output anything (in say Exploration cell).

Except of that use one should avoid using Nulls in the middle of the array (after first non-null value).

7 Likes

Adding to Tomasz explanation, Nulls do indeed propagate through most operations, but not all. Boolean operators are special. See the examples below. The result of each expression is found in the inline comment.

ARRAY_ZERO = BarIndex() && 0;
ARRAY_ONE = !ARRAY_ZERO;
ARRAY_NULL = ARRAY_ZERO * Null;

// Unary operations
unary_op_not = !ARRAY_NULL;        // Array Null

// Boolean operators mixed operands
mix_op_and_1 = 1 && ARRAY_NULL;   // Array 0
mix_op_and_0 = 0 && ARRAY_NULL;   // Scalar 0
mix_op_or_1 = 1 || ARRAY_NULL;    // Scalar 1
mix_op_or_0 = 0 || ARRAY_NULL;    // Array 0

// Boolean operators array operands
array_op_and_1 = ARRAY_ONE && ARRAY_NULL;   // Array 0
array_op_and_0 = ARRAY_ZERO && ARRAY_NULL;  // Array 0
array_op_or_1 = ARRAY_ONE || ARRAY_NULL;    // Array 1
array_op_or_0 = ARRAY_ZERO || ARRAY_NULL;   // Array 0