RFC: InternetPostRequest

Hi Tomasz,

I previously asked about support for POST operations with the Internet functions. You mentioned this might come after the 6.30 release. I'm wondering if you could possibly include this in the next beta release? I desperately need it, as I'm currently using OLE to do posts and the performance is horrible, doesn't run multithreaded, etc...

Anything called remotely (Internet) is subject to ping times that are pretty much in >10ms milliseconds range, so OLE function call speed should be irrelevant considering that internet responds much slower, no?

Hi Tomasz,

I'm querying a remote API function and when running backtests / optimizations, it's executing multiple simultaneous API queries. Even though each query runs in its own thread, it seems the queries are executing sequentially, rather than in parallel, thus defeating the multithreading. When I analyze the trace results using the thread IDs, I can see that each OLE request runs and downloads the data, then the next thread ID executes, etc... Thus, I'm guessing that each OLE request is blocking the other threads from executing until it's complete.

To use a specific example, let's say I'm running an optimization in eight threads, that requires 1,000 steps to complete. Each API request takes about 5 seconds to complete. Running in parallel, it would be 125 steps per thread, or about 625 seconds to run the optimization. Running sequentially, it takes eight times longer, since only one request can execute at a time.

1 Like

Yes that is true that OLE automation in Windows is essentially single-threaded.

The workaround is pretty easy though - you can run local web server running php and just accepting GET requests and forwarding them as POST to remote server.

FWIW: HTTP Post will be available in next version

2 Likes

That's great to hear. Thank you so much!

By the way, I appreciate your suggestion regarding setting up a local GET->POST proxy, but it won't work for my use case, because I'm posting way more data than can be encoded in the query string. Therefore, I'm very much looking forward to the next version.

The API for the HTTP post will look something like this:

InternetPostRequest( url, data, flags )

where url and flags params will be the same as for InternetOpenURL and
data will be URL-encoded string with data to be posted (like "param1=somethign&param2=somethingelse&param3=otherdata") Of course there would be no fixed limit on data string length.

Please comment if that would work for your application.

Yes, that looks good.

I do have a question regarding timeouts, as some of my queries run for a long time. Do your Internet functions use the system timeout settings specified in the Windows registry? It would be nice if there was a parameter for timeout seconds.

These functions are based on Windows Internet (WINET) subsystem and generally use same settings as MS Internet Explorer. The default timeouts are pretty long (like 20 seconds on connection and 60 seconds on receive) and I never had any reason to increase them, but yes as anything in Windows it can also be controlled by registry (HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings)

It is available now in 6.30.4

1 Like

Thank you very mcuh, Tomasz!

Usage example, AFL formula:

// You can even mix GET parameters (in URL) with POST request
ih = InternetPostRequest( "http://server_name_here.com/testpost.php?param3=7&m4=8", /*POST DATA*/ "param1=9&m2=12" );

if( ih )
{
    while( ( str = InternetReadString( ih ) ) != "" )
    {
       printf( "%s", str );
    }
   
    InternetClose( ih );
}

Testing code (server side ) in PHP:

<?php

echo "Post variables: \r\n";

foreach ($_POST as $key => $value){
   echo "{$key} = {$value} \r\n";
}

echo "Get variables: \r\n";

foreach ($_GET as $key => $value){
   echo "{$key} = {$value} \r\n";
}

?>
1 Like

Tomasz,

Can you please double-check this function? In my testing, I'm unable to get it to post any data. It sends the page request and it retrieves whatever response I send back from the server, but nothing comes through in the $_POST array or anything at all, the body is completely empty.

I'm including the POST data in the second parameter as a URL-encoded query string, just like in your example above. On the server side, I have a PHP script, similar to the one in your example. It simply converts the $_POST array to a string using print_r($_POST, true) then writes the string to a file, but the resulting file is always empty. I also tried it using your foreach approach in the above example, and I also tried using php://input to capture the raw data, but it always writes an empty file.

I'm testing in AB version 6.30.5 (64-bit). When I use the objXmlHttpMain OLE object, it works fine, the script correctly writes the POST data, so there's definitely something going on with the InternetPostRequest function.

I tested it. I always test before release. I have given you working code in PHP and in AFL (above). Please use exactly the AFL code and PHP code I provided above. It was tested and works.

In your AFL code you may be encoding POST data incorrectly and this may also be source of problems.

If you believe otherwise, send your code.

1 Like

After further testing, I discovered the problem is it doesn't support HTTPS. It works fine with HTTP, but with HTTPS, nothing gets posted. It's not a certificate problem, because:

A. I'm using a valid certificate.
B. It makes the request and receives the response from the server, it just doesn't send any POST data in the request body.
C. When I use the objXmlHttpMain OLE object, it works fine with HTTPS.

I checked the code given in my post from March with HTTPS and it works perfectly fine.
You can check yourself:

// You can even mix GET parameters (in URL) with POST request
ih = InternetPostRequest( "https://www.amibroker.org/post/post.php?param3=7&m4=8", /*POST DATA*/ "param1=9&m2=12" );

if( ih )
{
    while( ( str = InternetReadString( ih ) ) != "" )
    {
       printf( "%s", str );
    }
   
    InternetClose( ih );
}

It produces the output:
image

1 Like

Yeah, you're right. I tested it on another server and it worked, so obviously there's some SSL problem on the server I was testing with. I apologize for wasting your time with this stupidity.

1 Like

No problem. Glad to have it solved.

1 Like

Tomasz,

I'm very sorry to have to re-open this discussion, but I couldn't understand why it works fine on some servers and not others, so I used a packet analyzer to view the request at the protocol level.

I discovered that InternetPostRequest ALWAYS sends as HTTP, regardless of whether you specify HTTP or HTTPS in the URL. Therefore, if the server accepts HTTP requests, HTTPS will appear to be working fine, but the requests will actually be sent over HTTP. In my case, my server is configured for HTTPS-only, HTTP is blocked, so it doesn't work. However, your server accepts both HTTP and HTTPS, so it APPEARS to be working.

In this first screen shot, I'm showing the code using the COM object, followed by a screen shot from my packet analyzer, showing the request being sent over HTTPS:

comObjectCode
comObjectAnalysis

Next, take a look at the code and analysis for InternetPostRequest:

IPRCode
IPRAnalysis

As you can see, it's being sent over HTTP, even though the URL specified HTTPS.

1 Like

That is rather strange because InternetOpenURL opens up a tunnel at 443 just fine and InternetPostRequest isn't even though they both use pretty much same WinInet code.