Downloading fundamental data extra from Yahoo Finance

AFL Programming
21

Very helpful @Tomasz .....thank you

22

@sahir001, I suggest to take a look at this article.

In any case, before attempting this operation, be sure to have a BACKUP copy of your database.

1 Like
23

Can you please help explain the fundamental function of extra data from Yahoo Finance, and how to display it. Thanks very much :pray:

26

Everything is (as always) documented and explained in the users guide: Using Fundamental data
Entire tutorial Tutorial is a must-read for every newbie.

2 Likes
27

Many thanks Mr Tomas. The new upgrade of Fundamental extra new download sw works very well. I am downloading all the fundamental data in a short period of time. I have defined the "initial delay between requests" in 1000 millisec and I am downloading the fundamental data for one symbol at a time. This should be the correct and it does not interfere with Yahoo permissions.I think to perform all the download of 6000 symbols fundamental data in less than 2 hours. Thank you so much!

28

It turns out that the extra fundamental data update function from Yahoo Finance is to speed up downloads in large quantities at once. I thought before to add a fundamental data variable like this Downloading fundamental data extra from Yahoo Finance - #4 by justintime

29

Thank-you for the New fundamental Extra !

30

Thank you Tomasz, very useful. Can you offer how I can augment the ads script to also include Sector and Industry from the Yahoo Profile page?

31

Sector and industry info are downloadable already via Tools->Update US symbol lists and categories (in AmiBroker). That data are not downloadable directly from Yahoo.

2 Likes
32

I want to get these values for any stock which is accessible on Yahoo Financial. This is a Powershell script that quickly gets the Sector and Industry for a Canadian (TSX) stock from the Profile page. Can ads scripts incorporate Powershell instead of Javascript? Alternatively I can load the results using Import ASCII, I suppose.

$ticker = 'npi.to'
$modules= 'assetProfile'
$request = -join @("https://query2.finance.yahoo.com/v10/finance/quoteSummary/$ticker",
			"?formatted=true&lang=en-US&region=US&modules=$modules&corsDomain=finance.yahoo.com&fmt=json")
$cont = Invoke-WebRequest $request | select -expand Content
$res1 = ($cont -split "," | Select-String -Pattern "Sector|Industry" )
$res1 = $res1 -replace '"',''
$res1 = $ticker, ($res1 -join ',') -join ','
# the next line removes the field name from the string, so check the value of $res above before determining the field order in the next string value
$res2 = $res1 -replace ",[a-z]+:", ","
33

This is a full script to generate the Industry & Sector for an entire .lst file

$modules= 'assetProfile'
rm outFile.txt

# load all tickers from the Amiquote symbols list file
$tickerList = Get-Content 'C:\Program Files\Amibroker\Amiquote\FullEODdata.tls'
#$tickerList = Get-Content 'C:\Program Files (x86)\Amibroker\Amiquote\FullEODdata.tls'
foreach ($ticker in $tickerList) {
	$request = -join @("https://query2.finance.yahoo.com/v10/finance/quoteSummary/$ticker",
				"?formatted=true&lang=en-US&region=US&modules=$modules&corsDomain=finance.yahoo.com&fmt=json")
	try { 
		$cont = Invoke-WebRequest $request | select -expand Content
		Write-Host $ticker "" -noNewLine
	}
	catch { Write-Host $ticker "(N/A) " -noNewLine }
	finally { $error.Clear() }
		
	$res1 = ($cont -split "," | Select-String -Pattern '"Sector"|"Industry"' )
	$res1 = $res1 -replace '"',''
	$res1 = $ticker, ($res1 -join ',') -join ','
	# the next line removes the field name from the string, so check the value of $res above 
	# before confirming the field order in the next string value
	$res2 = $res1 -replace ",[a-z]+:", ","
	Add-Content "outFile.txt" $res2
}
echo "Finished"

# to run:  .\getSectorIndustry.ps1
2 Likes
34

This is an improved version, avoiding symbols in the output that are problematic, plus adding final processing stats:

$modules= 'assetProfile' # keyword for finance.yahoo.com 'Profile' subpage 

rm outFile.txt # start with a clean slate

# load all tickers from the Amiquote symbols list file
$tickerList = Get-Content 'C:\Program Files\Amibroker\Amiquote\FullEODdata.tls' # for 64bit AB location
#$tickerList = Get-Content 'C:\Program Files (x86)\Amibroker\Amiquote\FullEODdata.tls' # for 32bit AB location

$ok = 0 # tickers correctly handled
$nf = 0 # tickers not having Sector/Industry field on Profile page
$err= 0 # tickers not found on finance.yahoo.com

foreach ($ticker in $tickerList) {
	#$ticker = 'all-pb' # for testing
	$request = -join @("https://query2.finance.yahoo.com/v10/finance/quoteSummary/$ticker",
				"?formatted=true&lang=en-US&region=US&modules=$modules&corsDomain=finance.yahoo.com&fmt=json")
	try { 
		$cont = Invoke-WebRequest $request | select -expand Content
		$res0 = ($cont -split "," | Select-String -Pattern '"Sector"|"Industry"' )
		$res1 = $res0 -replace '"',''
		$res1 = $ticker, ($res1 -join ',') -join ','
		# the next line removes the field name from the string, so check the value of $res above 
		# before confirming the field order in the resulting string value
		$res2 = $res1 -replace ",[a-z]+:", ","
		
		if($res2 -like '*,') { # either didn't find a Sector field, or field has no value (typical for ETFs)
			Write-Host $ticker "(N/F) " -noNewLine # flag a Not Found error to console
			$nf++
		} else { 
			Add-Content "outFile.txt" $res2
			Write-Host $ticker "" -noNewLine
			$ok++
		}
	}
	catch { 
		Write-Host $ticker "(ERR) " -noNewLine # flag a Invoke-WebRequest error to console
		$error.Clear()
		$err++
	}
}
echo "Finished"
echo "      Tickers Read:" $tickerList.count
echo " Correctly Handled:" $ok
echo "No Sector/Industry:" $nf
echo "  Missing on Yahoo:" $err

# to run:  .\getSectorIndustry.ps1
1 Like
35

I have installed update (x2) and still getting the message:
image

36

The message is informational only. It is ALWAYS displayed with AmiQuote 4.10 so people don't report "problem" with fundamentals when solution already exists. If you installed data source it is OK. The program does not check what user-definable data source you have installed. The message will only disappear with NEW RELEASE that would have updated data source already built-in out of the box.

1 Like
38
39

Yahoo has changed their API now and attempt to use this data source yields error message boxes.

This is modified Data source definition that displays error codes in-line instead of popup message boxes, but this does not change the fact that Yahoo requires now authorization and probably will require paying for the fundamental data.

[Yahoo Fundamental Extra New.ads|attachment]
[OBSOLETE - Removed. Download AmiQuote 4.14 instead]

UPDATE: I found out that they just added requirement for crub and support for that will be added in next AmiQuote version.

2 Likes
40

I have upload this new data source. In fact, it does not work. What can we do now to fix this problem? Have we only to wait for an answer by Yahoo? Any thought?

41

Did you read the last sentence from previous post (i.e. "UPDATE" line)?

42

I don't know but apparently ability to read is diminishing in the population, so let me re-iterate what I wrote:

UPDATE: I found out that they just added requirement for crub and support for that will be added in next AmiQuote version.

And the update promised yesterday is POSTED TODAY:

IMPORTANT: READ ENTIRE THREAD about AMiQuote 4.14 including INSTRUCTIONS that are in second post!!!!

5 Likes
44