Yahoo EOD is Unreliable

[bkonia]

With Yahoo EOD, I always have to manually retry two or three times before it finally downloads the quotes. The problem seems to be with establishing the initial connection. Once it's established and the downloads begin, they go fast.

The problem is specific to Yahoo; all the other data providers work fine and there's nothing wrong with my Internet connection. I only ever see this message with Yahoo.

Rather than giving up right away, it would be nice if AmiQuote would automatically retry a user-specified number of times.

As you can see, it's configured to retry 10 times, but I guess this only works after a connection has been established. I'm suggesting there should be a separate retry setting for establishing the initial connection. I tried increasing the Timeout Connect setting to 300 seconds, but that seems to make no difference. It always gives up after about a minute, regardless of this setting.

I run an automatic process every night to download quotes from various data providers, but the process never completes, because it always gets stuck on Yahoo.

[Dionysos]

Honestely, yahoo does weird things. It's a great starting point, but for live operations there is no way around a real data provider. Norgate's new NDU is very neat. CSI DATA is very good too.

It's difficult to complain about "freebies".

[bkonia]

I'm not complaining about Yahoo. I'm complaining about AmiQuote. AmiQuote is a commercial product, not a "freebie". The point is, AmiQuote is not interfacing with Yahoo in an optimal manner. It would be simple to make it retry a few times on the initial connection, rather than just immediately giving up and displaying a failure dialog, requiring the user to manually intervene.

[Dionysos]

Based on experience, AmiQuote works just fine. 99% of all cases when you encounter an error like yours, it's the yahoo server's fault.
You could try quandl as an alternative. Or for live operations, a paid service.

[bkonia]

Quandl does not provide quotes for equities in their free database. Therefore, I use Quandl for Futures and Finam for Forex and Yahoo for equities and cryptos. You can argue that I should pay for commercial data and perhaps at some point I will, but the point of AmiQuote is to pay for the software rather than paying for the data.

The problem may or may not be with Yahoo. However, as you pointed out, Yahoo is a free service, so there's not much we can do about that. AmiQuote on the other hand is a commercial product. It does work fine in most cases, but certainly not 99%. Considering that AmiQuote features nine data providers and assuming the other eight work fine, we can say that AmiQuote works just fine in 89% of cases.

Regardless of whether the problem is with Yahoo or not, the solution is for AmiQuote to retry the initial connection, rather than simply giving up after one attempt. I'm sure Tomasz would agree that when designing software that interfaces with external systems, it's important to anticipate possible points of failure and code the software such that it handles those failure gracefully.

[Tomasz]

@bkonia - blaming AmiQuote is absurd. It seems that you have zero knowledge about internet programming. AmiQuote is just a very simplistic web browser (browser without rendering engine). There is nothing "magical" in it. It uses Windows to connect to the internet. There is no "interfacing" of any kind. It just call InternetOpenURL function from Windows and that's it.
The error you are getting is coming from Windows. Specifically from Windows Internet subsystem (WINET). It is generic error that means your connectivity to Yahoo is broken.

It has NOTHING to do with AmiQuote.

See Microsoft WINET error list:
https://support.microsoft.com/en-us/help/193625/info-wininet-error-codes-12001-through-12156

The reason for this message is either:

1. your internet connection is broken
2. firewall or antivirus blocks connections
3. (MOST LIKELY) Yahoo BLOCKS YOU because you downloaded too much, too quickly. Yahoo has a limit of 2000 symbols PER HOUR. If you download more too quickly they will block your IP.

Yahoo does this to prevent from denial of service attacks.

Download less or slower and you will be fine. You can go to the Settings and DECREASE number of simultaneous downloads back to 1 ( the DEFAULT value ) and Delay between requests to 2000 ms or more.

So the solution to your problem is to: download LESS and/or download at SLOWER pace.
And the functionality is ALREADY THERE - see the SETTINGS page.

And as far as "brilliant" ideas of "graceful" resolution are considered - there is no "graceful" resolution to Yahoo blocking your IP. If server blocked your IP, the only thing you can is to
wait for unblock on the server side (which may or may not happen) or change IP.

And as far as percentages are given AmiQuote works in 100% of cases if only remote server responds. This works like telephone, you can call somebody and your phone just works, the remote side 'rings', but somebody on the other side has to pick up the phone. Blaming your phone for the fact that somebody did not answer your call is what you are doing here.

[bkonia]

Considering I've developed extensively in multiple programming languages and developed a CURL-based content-filtering reverse proxy in PHP, it's a little silly to say that I have ZERO knowledge about Internet programming. I actually have some knowledge on the subject and you seem to be missing my point.

I fully acknowledged that the problem may be with Yahoo, but since AmiQuote is a commercial application, I feel that it should anticipate connectivity issues and recover more gracefully. In other words, while the problem may be with Yahoo, AmiQuote should be smart enough to handle it properly. In my view, throwing up a modal dialog after the FIRST connection attempt is not handling the issue; it's throwing it back to the user saying, "YOU deal with it".

And no, Tomasz, my IP is certainly not blocked by Yahoo. This happens on the very first connection attempt of the day. After receiving the failure dialog, if I immediately retry, it usually works on the second attempt, without changing any settings. Yeah, it's set to 6 simultaneous downloads and this works fine on the second attempt. I just tried reducing it to 1 simultaneous connection and it did the exact same thing. So, so much for that theory!

Anyway, clearly you don't want to fix this and you just want to be insulting and abrasive, so that's fine, forget I mentioned it, lol.

[Tomasz]

CUrl and PHP is NOT Windows Internet programming. It is WEB programming. I specifically wrote about WINET (Windows Internet subsystem). If you can't see the difference, then my point is even better proved.

It is hilarious to see users making assumptions like this

throwing up a modal dialog after the FIRST connection attempt

which are of course false. Did you see AmiQuote source code ? I do not remember you coming to my office and me showing you the source? Did I miss something? I don't think so. For your information - 5 retries are made.

The lack of understanding is shown also by your assumption that the behavior you are describing NOW (not earlier) is proving that you are not blocked.
If you were Internet expert you just claimed to be, you would know that services like Yahoo use content delivery networks and load balancing (i.e. multiple servers in multiple locations and request one may be served by one server and second by other server). Blocks are per CDN server, so one server might have block on you and the other not.
AmiQuote ALREADY includes measures to RETRY (see again SETTINGS screen). But if retry count is hit you get the error message.

If you think you can do better - write your own application and sell it. I gave you even technical details how to do that on Windows. If you have all that experience and knowledge - go for it. It is so easy, 5 minute work, isn't it?

[bkonia]

LOL I know the difference between Windows and PHP and I have no "lack of understanding". Why do you always have to be so insulting, man? I've been your customer for years and have never once insulted or disrespected you. Moreover, I evangelize for AmiBroker all the time. I think it's the greatest TA platform in the world.

Yes, I know what a CDN is and obviously Yahoo has many servers, undoubtedly using some sort of round-robin and/or geo-based DNS routing. Nevertheless, when you consider DNS caching, it's probable that making two connection attempts within a minute or so would route to the same server each time. Regardless, my point is that it's the same behavior every time. First connection attempt fails, second one goes right through. Even if it's connecting to a different server on the second attempt, it's using the exact same settings, so if the first server blocks the connection, why would the second server allow it? In my view, this points to some sort of issue with AmiQuote's Yahoo interface, not with Yahoo itself.

Regarding the retries. as you can see in my screen shot, the Connect timeout is set to 120 seconds. Yet, it fails and displays the dialog after only one minute. If it's indeed retrying five times, as you claim, it should take TEN minutes to fail, not one minute.

Anyway, enough of this ridiculous discussion. Like I said, you don't want to fix it, nor do you even want to acknowledge the possibility of a problem. All you want to do is throw out insults. I never claimed to be a superior programmer, only that I have some knowledge on the subject and trying to point out an issue which clearly you don't want to even consider. I have no idea why you'd want to treat a customer with such contempt, but I've seen you do it a thousand times before on the forum to others, so I don't take it personally.

[Tomasz]

I don't understand why do you consider providing technical details as insults.

The "lack of understanding" is not an insult. For example worlds' top physicists and inventors including for example genius Tesla did not understand special relativity theory for years.

On a side note: if anybody insulted anyone, it is you insulting me in first place, because you assumed that I am incompetent and did not do retries in connection attempts or otherwise wrote program badly.

I actually would be extremely HAPPY if ANY programmer from say Microsoft, Adobe or whatever answered my questions EVER in my life, instead of serving canned responses.

I would be extremely HAPPY if programmer of given product explained me inner workings of the code and technical details NOT known to anyone else. I would consider having received such a knowledge as BIG positive.

I would gladly accept the fact that I lack understanding of word processor programming. I would happily say myself that I have zero knowledge about word processor programming as I never wrote MS Word kind of app. And I did not see MS Word source code, so I know nothing about it. These are just plain facts. I would not approach Microsoft chief programmer with "hey I wrote million lines of C++ code so I know....". I would rather be curious and listened to what they had to say.

It comes as infinite surprise to me that people seem to actually prefer typical "customer service" blah blah talk without any substance just because it makes them feel good.

I am sorry but you are getting response from (ugly) programmer who only understands 0's and 1's, not some nice and pretty young lady fluent in courteous english. Programmers are technical and are known to have social skills of thermonuclear device. I gave you technical details, not some pre-written "nice to hear" canned propaganda that you would get from typical "customer service". I thought that you might find having first-hand tech info useful. I was wrong.

Apparently you don't appreciate having programmer answering your questions and don't appreciate that I ACTUALLY improve product ALL the time.
I never claimed that there is no room for improvement. Just the OPPOSITE. My acts not words speak for themselves. Endless stream of updates http://www.amibroker.com/devlog/ prove that I am DOING improvements, not talking.

But next time I will respond with typical customer service talk "thank you for opinion, we appreciate your input and it is very valuable and of course we will pass it to the other department, thank you for your time". And you will be happy.

Back to technicals:

Firstly, timeout setting (and all other settings) applies to DATA downloads, not to the initial Yahoo cookie query. Initial query is always done with WINET default 5 retries and default WINET timeout. So you are again making false assumptions. Advice: if you did not see source code, don't make assumptions on it.

Secondly I don't see the behavior you described. AmiQuote does NOT display ANY error message with Yahoo Historical on my end.

And no-one else actually seem to be seeing this as well. If they did, I would receive 1000+ emails in single day about that (I know because that is exactly what happened when Yahoo changed their servers and stopped working back in 2017). So as things stand now it is your local problem. First thing to do is to clear cookies and browser cache in the Internet Explorer as explained here (WINET subsystem uses Internet Explorer cache regardless of what browser you are using - again this is only tech info) and check local firewall/router. I have seen broadband routers that had firmware errors causing bizarre behavior of only few web pages, others working normally.

Without ability to reproduce the behavior you described it is impossible to "fix" anything. It is not about "wanting" or "not wanting". First step to resolution is reproducing the issue.

On a side note: I recently visited Barcelona to see La Sagrada Familia. Gaudi devoted 40 years of his life to design this masterpiece. Seeing this was very humbling experience. Of course in current shape it is work of lots of people but really that is THE Opus Magnum and will stay there for centuries to come.
In the light of this I consider 23 years that I spent on designing and developing Amibroker and AmiQuote almost wasted as all that work will all disappear sooner or later in the information noise. Still I spent 23 years of my life on that work and it is not "nice" to be blamed for no reason.

It would be quite interesting to see Gaudi's face and response when he was 20 years into his work if you claimed he designed something badly.

[bkonia]

Tomasz,

I think you often feel unappreciated on this forum. I want you to know first of all that you are very much appreciated, not only by myself, but by many others as well. AmiBroker is an incredible piece of software and your concept of array processing is a brilliant innovation. When I first began trading, more than 20 years ago, I tried pretty much every TA software on the market, many of them costing thousands of dollars. I wasted so much time and money on software, it was ridiculous. Then one day, I stumbled across AmiBroker. I don't remember how I found your website, but it changed my life. Here was this software that only cost $250, yet it outperformed everything else I'd tried, both in terms of speed and features. I was amazed and felt immensely grateful for having discovered it. Since then, I've never looked back. I'm on all the trading mailing lists, so I get bombarded with offers for new software, or new versions of old software, etc... Honestly, I'm never even tempted. I haven't looked at any other TA software in years, because I simply have no reason to. AmiBroker does everything I could possibly want and it does it faster than anything else I've tried. Plus, you continue to add new features that match or exceed anything else being offered by your competitors. So I don't know where you got the idea that I don't appreciate that you improve the product all the time. Of course I appreciate that greatly and I always use the latest beta builds, so I'm well aware of all the work you're doing.

Regarding customer service and support. You're absolutely right that your support is awesome and that's another reason to use AmiBroker. When you get support on this forum or via email, you're getting it directly from the developer, not from some idiotic customer service robot. The only problem is, the developer has the social skills of a thermonuclear device, lol. Except that's not really true. The fact that you'd take the time to write such a detailed, heartfelt response to my last message speaks volumes about your humanity and your ability to empathize. And I've seen other forum postings in which you've written passionately about the genius of great composers and about other non-technical subjects. So I don't buy this self-depracating idea that you only think in zeros and ones.

Nobody wants to deal with a tier-1 support tech, a/k/a/ robot, and you're right that their nice, polite customer service skills are no substitute for actual technical knowledge. No one is expecting you to give pleasant feel-good responses, like "Have a nice day, I hope your customer service experience has been excellent, is there anything else I can do to better serve you today, blah, blah blah..." No one expects that or wants that from you. However, I don't think it's too much to ask for you to be civil and respectful and not hurl insults at people, telling them how stupid or clueless they are. If someone intentionally provokes you or insults you first, then sure go for it. I would do the same! But here you have someone who thinks the world of you and is a loyal AmiBroker customer for I don't know how many years, and tells everyone he knows how great AmiBroker is. In other words, I'm not the enemy, I'm on your side!

I think part of the problem in this case was I had some back and forth with another user before you jumped into the fray. Sometimes when I ask a technical question and other users reply, I become frustrated because they offer workarounds, like "don't use Yahoo", since of course they don't have the technical knowledge to address my specific question. Perhaps you took my responses to him as being insulting to you. That was certainly not my intention.

I was confused because I had the timeout interval set to 120 secs and wrongly assumed this pertained to the initial connection, not the data downloads. Hopefully you can appreciate the confusion, as the box is labeled "Connect Timeout", which in this context is a bit ambiguous, since it doesn't specify that it pertains to the data connection, not the initial connection. Nevertheless, it wasn't necessary for you to come at me so aggressively, declaring I have ZERO knowledge of Internet programming and various other insults, lol. You were making an assumption about me, based on limited information, just as I made an assumption about how AmiQuote works, without having seen your source code.

Tomasz, it doesn't cost you anything to be nice and respectful and it doesn't take any additional time to reply. It fact, it would probably save you time, since it requires additional typing to add insults to your reply .You can have great technical knowledge and also be nice. The two aren't mutually exclusive and being nice doesn't make you a customer service robot; it makes you a human being.

Getting back to this specific technical issue, I appreciate the information you provided. I will do some more testing on my end and try your suggestions regarding IE, etc... I agree that if you're not seeing the same problem and no one else is reporting it, it's probably something on my end. As always, thank you for your time and rest assured, you are very much appreciated!

[QuantTrader]

for what is worth I am having problems too. I immediately get you have reached yahoo download limit after few symbols. I tried from three different IP address with the same result. Two of the IP addresses has never been used to download data from Yahoo. It seems something is going on with Yahoo.

[Dionysos]

I use Yahoo data extensively for preliminary tests and "scouting".
Yet it brings me back to the point where I started off:
For live trading, use a paid service that does its due-diligence.

You get what you pay. Norgate has really great data. Very well worth the $. Saves time and trouble.

[bkonia]

I don't understand why you keep harping on this criticism of Yahoo. Yahoo may or may not be a good data provider, but that is NOT the topic of this particular discussion. This is a discussion about the technical aspect of AmiQuote downloading Yahoo data. Whether or not Yahoo is a good data provider is completely irrelevant.

[Tomasz]

I immediately get you have reached yahoo download limit after few symbols.

@QuantTrader this is fundamentally different from what @bkonia sees.

If you are getting "you have reached download limit of 2000 symbols per hour" auto-continue message after successfully downloading some data already, it means that Yahoo responded, delivered data first and later responded with "invalid cookie". So Yahoo server actually responded. The thing that you see the message sooner than expected may suggest that Yahoo is starting to tighten the rules for data downloads. The limit of 2000 symbols per hour is nowhere documented by Yahoo and it is just totally heuristic guess based on some testing, but they may be doing changes even daily to the way how they balance the load of their servers.

You might need to tell me exactly the symbol count that you tried because on my end it works fine. You also need to tell me if allowed auto-continue to wait and continue downloads. This dialog should not be dismissed. I can download way, way more than 2000+ symbols before getting the "auto-continue". The thing is NOT to use too many simultaneous downloads. There is "Fail safe" preset available in the Settings that you might try AND please clear IE cache/cookies too.

UPDATE: I was checking this all the time today and I am able to download 6630 symbols (NYSE+Nasdaq+AMEX stocks) without hitting any limits using these settings:

What @bkonia described happens when Yahoo (or some other piece of puzzle like router/firewall) does NOT even allow you to make initial connection and there is NO response from their side (timeout). This is fundamentally different issue, not related to the other.

Also @Dionysos has some point here - since Verizon takeover Yahoo does not seem to be interested in their finance.yahoo.com web site the way they used to be and one needs to expect that sooner or later they will shut this web site down entirely or limit downloads even more. No matter how much trickery is put on the client side, if they decide that they do not want to give data, they won't. There are alternatives like Tiingo, Google that work great.

[xiaozhou]

I see the same issue. Yahoo download stops at less then 100 symbols( not always the same ) , The dialog said it exceed 2000 symbol per hour even I waited more than a few hours.
The setting is same as Tomasz's setting.
I try google but I got open and close price reversed for yesterday, which could be a google data problem.
I also found it the symbol is not in capital letter, google download returns error. Could we convert to capital when fetch google data?
For Yahoo download, the first time I hit the error is because I tried to download a lot of days(maybe 5 years data). I think maybe there is something in the cookie? I unchecked Allow cookies checkbox but still see the same thing.
I use chrome as browser, Does amiquote use IE cookie only?
Thanks

[bkonia]

Hi Tomasz,

I received your reply in my email, but for some reason it didn't show up in the forum. Anyway, I want to say that I agree with your point about being nice, no matter what. In my previous reply, I did mention that if someone insults you, you shouldn't just take it, you should give it back to them. I'm more of an "eye for an eye" person than a "turn the other cheek" person.

When I said you should be nice, I meant up until the point where it no longer serves you. I have to deal with customer service in my business and I'm always very nice to my customers until they give me a reason to stop being nice. If a customer seems stupid or misinformed, or clueless, I explain the facts objectively and dispassionately, not making judgements or comments about his lack of intelligence or knowledge. However, if someone gives me attitude, I turn off and I give it right back to them. You're absolutely right when you talk about customer service expectations in Western society. The false mantra of "the customer is always right" has become so deeply entrenched in our society that many customers believe it gives them license to abuse the very people who are trying to help them.

The problem is customers are so used to dealing with support robots that when they encounter an actual human being, they don't know how to act. It's really the fault of the Western business gurus who made themselves rich over the past 30-40 years, by training an entire generation of corporate drones on the "customer is king" philosophy. Years ago, when you called a business, they would answer the phone by stating the company name, sometimes followed by "How may I help you?" Nowadays, you have to listen to a whole long speech like "Thank you for calling such and such company, it's a beautiful day here in such and such place, how can we make your customer service experience amazing today?

Anyway, I just want you to know that we're of the same mind and I DO believe you're fundamentally a nice person. However, I also think you can be oversensitive toward criticism of your products. AmiBroker is a masterful work and anyone who's used it for any length of time is well-aware of that. No one, least of all me, is trying to diminish that accomplishment. But, if someone reports a "bug", you shouldn't take that as a poke in the eye and there's no need for a fiery response telling the customer how stupid or clueless he is for thinking he found a bug. He didn't work on AmiBroker for the past 23 years, so he can't possibly relate to your experience or understand how much pride you take in your work. He's just a guy who bought a piece of software and is trying to use it to solve a problem. I'd suggest you point out his factual error, refer him to the appropriate manual page and move on. Why get yourself all worked up over someone's poor choice of words?

[Tomasz]

@xiaozhou:

1. Yahoo now requires cookies. It won't work if cookies are disabled.
2. The less data you request (for example 1 year worth instead of 5+ years) you are less likely to hit Yahoo limits. As I wrote today I downloaded 6200 symbols 1 year worth without hitting any limit.

@bkonia

thank you and I agree