Downloading fundamental data extra from Yahoo Finance

I would like to describe a problem that I have had with the downloading of fundamental data extra from Yahoo Finance website.
As I am a stock picker of exponential growth stocks in US stock market, I am used to download not only data of prices and volumes of listed companies (about 10.0000) but also fundamental data extra. Amiquote require only 1 hour about to download data of prices and volumes for these stocks. However, when I need to download fundamental data extra, Amiqote needs about 72 hours of time of uninterrupted download. In fact, if I try to use it at the same speed that I use for the other group of data, I receive an error phrase by Amiquore like the following: "you have exceded the number of downloadable simbols data per minute, retry in...". I need to download fundamental data one time per mounth at least. I would like to know if there is some trick to increase the speed of this process. 1 hour of time about, would be a fantastic target. This issue is very important for me and for many other stock pickers who need updated fundamental data to make their own choices. I hope that Mr. Tomasz may help me and this comunity in solving this problem. I have installed the last version of Amiquote 4.10. Many thanks.

1 Like

There is no trick to speed this up. There is big difference between the way how historical quotes are downloaded and fundamental data are.
Historical quotes are downloaded using "Download" link present in the Historical data page, for example:

Yahoo created a special link for downloads, they are in CSV (easily importable) format and Yahoo responds to download requests for historical data quickly.

Fundamental data however are NOT available as CSV download from Yahoo. Data have to be extracted from HTML from this Statistics web page:

That page (Statistics) is meant to be read by humans, includes advertising and lots of Javascript and other stuff. This page is naturally slow to load because it is large and is meant to be accessed by human (so you are not likely to request more than one page per second - probably way less frequently than that).
You should in fact be overjoyed by the fact that you CAN download fundamental data for 10000 symbols in only 72 hours for free. If you were to do all the job manually copy pasting figures from web page it would take weeks of human work to copy-paste data for 10000 symbols.

There are paid sources that offer the same data for huge $$$ and then you can download in minutes.

3 Likes

Many thanks for your polite answer.

I wonder if a connection with a VPN line that hides my IP may be a pratical solution to this problem.

Otherwise, it would be interesting to find such data from another website respect Yahoo Finance.

I think that the solution to this problem, if there is any, may be of great help for stock picker traders.

Think about that. Thanks.

Hi Tomasz,

I think there may be a way to get what you need in less than 1/100th the time.
I've been using this method to get yahoo fundamentals data it allows you to bypass the parsing of java script etc.

The result is in the form of JSON response instead of HTML and is easily parsed.

Example query:

https://query2.finance.yahoo.com/v10/finance/quoteSummary/baba?formatted=true&lang=en-US&region=US&modules=upgradeDowngradeHistory,recommendationTrend,financialData,earningsHistory,earningsTrend,industryTrend&corsDomain=finance.yahoo.com

Replace or add modules to what you need; can add all modules but sill still be fast.

# list of Modules for query:
#     [
#        'assetProfile',
#        'summaryProfile',

#        'summaryDetail',
#        'esgScores',
#        'price',
#        'incomeStatementHistory',
#        'incomeStatementHistoryQuarterly',
#        'balanceSheetHistory',
#        'balanceSheetHistoryQuarterly',
#        'cashflowStatementHistory',
#        'cashflowStatementHistoryQuarterly',
#        'defaultKeyStatistics',
#        'financialData',
#        'calendarEvents',
#        'secFilings',
#       'recommendationTrend',
#        'upgradeDowngradeHistory',
#        'institutionOwnership',
#        'fundOwnership',
#        'majorDirectHolders',
#        'majorHoldersBreakdown',
#        'insiderTransactions',
#        'insiderHolders',
#        'netSharePurchaseActivity',
#        'earnings',
#        'earningsHistory',
#        'earningsTrend',
#        'industryTrend',
#        'indexTrend',
#        'sectorTrend']
4 Likes

Thanks, it was my plan to eventually remove all "built-in" sources at all and leave only user-definable ones. That way you could quickly change the way how data are retrieved if data source changes their layout and/or API.
So, if that proves to be complete replacement for old method, then built-in fundamental downloader will be removed and user-definable data source will be added for that.

I am sorry but I do not understand how I can solve my problem to download fundamental data of more than 5000 listed stocks. Explain me in a simple way. How can I use that program code inserted above? Many thank for a clarification. Fredrick

1 Like

Hi Federico,

This will require a change to AmiQuote to use the new method of fetching Yahoo data. I don't think from the above that it will be implemented. Alternatively if you're techsavvy, you can write a program in JAVA, Python, etc. and fetch the fundamental data for a list of stocks and load it into the AmiBroker DB.

No, it does not require change in AmiQuote because AmiQuote 4.10 features user-definable data sources WITH Javascript parser. So you can define your own data source that will use URLs provided in this thread.

1 Like

If I have well understood, you are telling me that in the new release of Amiquote, I will be able to define the fundamental data that I am willing to download for all the stocks I am used to work with. Then I will download these fundamental data quickly (for instance 2-3 hours). Is this correct? I suggest you to insert all the fundamental data already present in the last version of Amiquote. A trader may choose to download them all or only a part of them. I ask you for a confirmation of my reasoning. Thanks.

Yes, you can DEFINE any data source in AmiQuote 4.10.
I took time to prepare a new Yahoo Fundamental user-definable data source.

The ready-to-use file is here:
[OBSOLETE - Removed. Download AmiQuote 4.14 instead]

You can download it, save to disk, then use Data Sources->Import menu to import it into AmiQuote. Once you do that a new data source will appear at the end of the list called "Yahoo Fundamental Extra New".
Now you can use it instead of old Yahoo Fundamental.

image

17 Likes

Thank you @Tomasz for taking the time to do that!

1 Like

Can be displayed on Amibroker Charts or in information ?, for example 'insiderTransactions' . Thanks

fa show

I just tried to run this and kept getting this window to pop up and had to shut it down....

Screen Shot 2022-01-29 at 12.13.33 AM

I just tried on WL DJI with no problems

@MCassICT Again and again, ANY error displayed by AMiQuote is not coming from AmiQuote but from YAHOO (or any other provider). In case of error returned by YAHOO AmiQuote just passes it "as is" and displays to the user. One cannot do anything if error occurs on server side except REMOVING symbols that are NOT present on YAHOO.

As for Javascript error, it is because Yahoo for that particular symbol misses several fields and script attempts to read certain object by name and found none.

I have modified the script to handle such situation:
[OBSOLETE - Removed. Download AmiQuote 4.13 instead]

8 Likes

Thank you Mr. Thomazs for the update, this new menu is very helpful. Gbu

Sorry, I automatically typed Mr. Tomasz name

@Tomasz,
Please help add ONE_YEAR_TARGET to csv and "FD/targetMeanPrice" to field.
Is there a way to disable the pop-up windows? (please ignore this. The later script has already fixed the problem.)
Thanks.

Thank You Tomasz,,

but now I need to use "Symbol Translation" for my symbols
how can I do that now?

all my symbols are same but I have to add at the end of any symbol ====> .SR

1 Like

Very helpful @Tomasz .....thank you