Access Text File on FTP - problem

FTP is tricky.

FTP is old protocol that requires:

  • outbound connection on ports other than 80, usually 21 (ports other than 80 are often disabled by firewall)
  • inbound connections (again often disabled by firewall), unless passive mode is configured
  • for non-anonymous logins - passing user/password credentials usually in plain text (with myriad of various methods depending on target server) - as @awilson noted certain configs may disallow plain text transmission

Note that Windows Firewall is configured on per-application basis.

Depending on your local firewall configuration it may work on machine A (that allows above) and may not work on machine B (that does not allow that).

For these reasons you should avoid FTP as possibly problematic.

2 Likes

@trongart, as an alternative, you can use a ShellExecute invoking a utility like "curl" (that offers a lot more control on what to do...)

ftpaddress = "ftp://shortstock:%20@ftp3.interactivebrokers.com";
filename = "usa.txt";
exportPath = "C:\\Temp";

remotefile = ftpaddress + "/" + filename;
localfile  = exportPath + "\\" + filename;
_TRACE( "Remote file [" + remotefile + "]" );
_TRACE( "Local file  [" + localfile  + "]" );
parameters = remotefile + " -o " + localfile; // CURL syntax to download and save to file
_TRACE( "Parameters  [" + parameters + "]" );
ShellExecute( "CURL.EXE", parameters, "", 1 );

// Read the downloaded file if it exists...
fh = fopen( localfile, "r" );

if( fh )
{
    while( ! feof( fh ) )
    {
        _TRACE( fgets( fh ) );
    }

    fclose( fh );
}
else
{
    _TRACE( "File " + localfile + " not found..." );
}

The most difficult part seems to get correct the curl binary download to install in Windows!
(Note that my example assumes that curl.exe is in a folder included in the Windows path).

3 Likes

@beppe thank you for taking your time and preparing this (ready to use) code :+1:

Because there are so many versions available, I used Curl Download wizard to choose and download the proper, executable version.

I remember, that some time ago Tomasz was recommending another program for downloading files from the Internet - Wget

I don't know which one is better, but your solution is working properly :slight_smile:

2 Likes

... by the way, I was wondering - if we retrieve (directly from the AFL) content of some web page using InternetOpenURL(), InternetReadString() - are we recognized by the server as AmiBroker or Internet Explorer (which components are used by AmiBroker) ? If it is "AmiBroker", is it possible to "hide" and be recognized/identified as a regular web browser ( like it is possible in some specialized programs - for example iMacros?

1 Like

https://www.whatismybrowser.com/ reports as:



Whoa! We can't figure out what browser you're using!

We're working hard to write detection code for all the different types of web browsers, but it looks like we haven't figured yours out yet.

And occasionally, either because of a problem or a changed configuration, sometimes web browsers don't provide the necessary information for us to detect exactly what you're using.

Hopefully soon we can detect your web browser, until then; check out your User Agent string:

AmiBroker

Hopefully that can give you a hint about what exactly you're using.

2 Likes

Thanks for the info @awilson !

... so it looks like we are identified as AmiBroker (not I.E.)

I was asking, because for several months I've been using AFL Internet functions for web scraping / data extraction. Usually it all works properly and I am happy with this solution, but recently I’ve been experiencing problem with just one web page which was working fine for a long time. When I try to connect, AFL's GetLastOSError() returns --> The connection with the server has been reset. I know, that there could be lots of reasons for that (it has been already discussed on the forum --> Web scraping / data extraction problem - and I have resolved issues discussed in that thread ) but the same code on another computer in my room (same Internet connection) is working fine. The problem occurs only on the machine which has been used for web scraping for a couple of months. For this reason I thought, that maybe this computer was enlisted on a server's black list and I am rejected when I try to connect. But the question is how is my PC (when using AmiBroker for web scraping) identified by the server? Is it by:

  1. Cookies - rather not, because I don't use a web browser for that
  2. Browser fingerprinting - rather not, because I don't use a web browser for that
  3. IP address - rather not, because same code on another PC sharing the same IP address, works properly.
  4. MAC address?
  5. .... ?

... and what can I do about it :wink: ?

How web servers identify a client

1 Like

@Milosz - clear cookies in IE. Wininet (windows os component responsible for Internet functions) shares cookies with IE.

1 Like

Thank you Tomasz.

I have cleared all cookies in IE but the problem still persists. When I try to connect to this site, InternetOpenURL() returns 0 (zero) and GetLastOSError() returns --> No connection. The connection with the server was reset. So I'm rejected right away. On the other PC - no problems with this site.

I could use the second PC for web scraping, but I'm sure that after some time, the situation is going to repeat and I will run out of PC's :wink: so I will be examining another ways of solving this problem ...

Regards

1 Like

I'm sorry for all the concern - I think I was wrong. My issue has probably nothing to do with one of my PC's being enlisted on some server's black list, but just with my outdated Internet Explorer. I've noticed, that currently I can open this problematic web page only in FireFox and Chrome, but not in I.E. So probably something in this web page's code has recently changed (maybe some HTML5 or https issues) and it is not working properly in my I.E. Personally I never use this web browser and that's why I haven't noticed any problems before - but AmiBroker uses it's components... I think this should help:

@Tomasz if I may ask one additional question. Can I edit some value in the registry to be recognized (when using Internet functions) from the outside as Internet Explorer not AmiBroker (User Agent string: AmiBroker)?

Regards

1 Like

This key should be already in place. AmiBroker creates such key automatically at start.

But you guys mix up several things. InternetOpenURL is a Winnet function, NOT Internet explorer function. WinInet is part (subsystem) of Operating System. IE uses WinInet, not the other way round. Also OTHER browsers use WinInet as well. And AmiBroker uses WinInet too.

https://msdn.microsoft.com/en-us/library/windows/desktop/aa385098(v=vs.85).aspx

BTW: "No connection/ conneciton with the server was reset" is typical when you have NO connection at all. Either your firewall blocks you or remote server blocks you. It is external problem.

1 Like

Tomasz, thank you for the additional info.

Yes, I understand that, but I was wondering if I have any possibility of changing this default name or User agent string - "AmiBroker" and be recognized/identified by servers as a regular web browser - like it is possible for example in iMacros (data extraction / web scraping utility):

iMacros

I thought that maybe it is possible via editing some values/keys in the registry, similar to editing "SenderName" in case of SMTP:

Sender%20name

Sender%20name%202

... or in any other way, but I am far from being proficient in this field (excuse my lack of knowledge - I surely don't grasp all the necessary details) and I don't want to abuse your kindness. If you don't reply, I will assume that it is (by any reason) impossible, but if such possibility exists I would be very grateful for that info.

Regards.

2 Likes

You can't change agent string with current version. Agent string is set to the WinInet default, which is application name. I can provide such feature in the future versions, if you wish.

2 Likes

Tomasz - thank you very much!

If it doesn't require much work from you to implement this feature, I will be very grateful for that :slight_smile:

2 Likes