-
Notifications
You must be signed in to change notification settings - Fork 681
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Issues with the data reader fetching yahoo finance #315
Comments
Can you provide a sample that replicates your issue? This works for me:
|
On Thu, May 11, 2017 at 9:49 PM, Rob Kimball ***@***.***> wrote:
start = datetime(2016, 12, 31) end = datetime.now() INPX =
data.DataReader('INPX', 'yahoo', start, end)
I'm not sure what the problem was. Perhaps using 'end = datetime.today()'?
You're code worked. Thank you for your help!
|
From my requirements.txt:
and in python shell:
with error:
I have a feeling yahoo updated their endpoint to be something else. I get a 502 when I try curl too. The link at |
Looks like there's also a cookie that needs to be sent in order to avoid a |
…stock still failing (pydata#315)
Not sure if the icharts failure is a temporary problem, but I submitted a WIP PR (above) to replace the request structure. Even if icharts does come back online, may be a good idea to implement a backup. |
I don't have the time to fix this in the library, but, essentially, there is another API endpoint that one can use. It's It can be useful to someone who needs to create a URL that they want to query automatically. You can also get a valid cookie / crumb combination from the Chrome dev tools in the Network tab. |
They've confirmed icharts isn't coming back, so @rgkimball's patch should certainly go in. https://forums.yahoo.net/t5/Yahoo-Finance-help/Is-Yahoo-Finance-API-broken/td-p/250503/page/3 |
@rgkimball @IvanTrendafilov , can the fix be pip install upgrade for the ease for beginner? |
Yahoo has change the URL, and the way the use date. Now date are Unixtime. period1 or period2 is date in (unixtime = (Human time - 25568) * 86400) but you must check your timezone, for example my place is Europe, then I have UTC+2 and I must sustract 7200 seconds. So my formula is (Human time - 25568) * 86400) -7200; where Human time is the time (d/mm/yyyy), 25568 is the number of days from 01/01/1900 till 01/01/1970 (This is because i do it in Excel and this is the minimun date), 86400 are the seconds in a day and 7200 are the number of seconds in my 2 hours difference with UTC Interval is day, week or month I'm using this for update my data in Excel and it works and now I don't need to wait until morning to get the historical data because it's available at less an hour after the market close (I´m talking about american markets) I apologyze for not to be fluent in english. I hope this help |
@Franlodo DO you try do download using the new link to download like 1000 stocks, will it get blocked? The old API seems has no limitation but I am curious if the new one still allow that. Hope you can try on it to validate |
The link is to get the csv file in the web, it must run for 2 or for 2000; In my Excel file I have nearly 200 and run properly. You can get 1000 of csv files and import from pandas, it will be the same made it saving files or "in the air" Anyway, I will try and comment. |
@bkcollection This will be available once the bugs are ironed out and the pull request is merged into the main repository. |
@rgkimball how optimistic the bugs can be fixed? |
…stock still failing (pydata#315)
@bkcollection The latest commits of #331 about wraps it up. It's a little frustrating that the new API drops out periodically, but you can now pull any historical price range, splits and dividends. I haven't found a new interface for Yahoo Options - this may be permanently removed, but I'm happy to implement if someone finds the endpoint. Same thing for index constituents. All of the failing tests on my PR are due to Eurostat or Yahoo's API sporadically failing. Notice that there is inconsistency across different tests for which ones are failing - appears to be random. Hoping now some other people find time pull down the code and give it a thorough review before the maintainers make a decision on merging it in. |
How to get the implement the fix into my current module? How is the drop
out happen? Can you explain more?
|
For a temporary fix (until this PR gets merged), try: $ git clone https://github.com/rgkimball/pandas-datareader
$ cd pandas-datareader
$ git checkout fix-yahoo
$ pip install -e . On Python: import pandas_datareader as pdr
print(pdr.__version__) # Make sure it is '0.4.1'. I originally wrote this as an answer to this Stackoverflow question |
@gusutabopb |
@Hmz123 |
seems like still have some errors. Not sure when will be a fixed release 0.4.1. Hopefully in this week. |
@bkcollection what do you mean by some errors? |
@arose13 you have blocked for using too many symbols? How many is it? |
@javadba a python 2.7-compatible pull request was merged 5 days ago. FYI. |
rgkimball's version 0.4.1 is still working for me. Is there a reason why it isn't being merged? |
it needs to pass tests and respond to comments |
I believe comments have been covered - I will try to work on tests next week. Did I miss anything?
Rob Kimball
Sent from my mobile device.
… On Jun 28, 2017, at 2:29 PM, Jeff Reback ***@***.***> wrote:
it needs to pass tests and respond to comments
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
@rgkimball if you can provide me instructions on how to install your version, i will run some tests and provide feedback and results. i'm currently using the fix-yahoo_finance developed by ranaroussi. |
…stock still failing (#315) Restores change necessary for Google to function Fixes yahoo-actions per API endpoint update Update regex pattern for crumbs, per heyuhere's review 'v' is no longer a valid interval value Fixes Yahoo intervals and cases where the Yahoo cookie could not be extracted. Implements multi-stock queries to Yahoo API Adds a pause multiplier for subsequent requests from Yahoo, error handling for empty data requests, and updates some test logic for pandas 0.20.x (notably ix deprecation) Check object type before checking contents Replacement regex logic for additional Yahoo cookie token structures, per chris-b1 Improved error handling and refactoring test to best practices, per jreback review. closes #315
* Replaces ichart API for single-stock price exports from Yahoo, multi-stock still failing (#315) Restores change necessary for Google to function Fixes yahoo-actions per API endpoint update Update regex pattern for crumbs, per heyuhere's review 'v' is no longer a valid interval value Fixes Yahoo intervals and cases where the Yahoo cookie could not be extracted. Implements multi-stock queries to Yahoo API Adds a pause multiplier for subsequent requests from Yahoo, error handling for empty data requests, and updates some test logic for pandas 0.20.x (notably ix deprecation) Check object type before checking contents Replacement regex logic for additional Yahoo cookie token structures, per chris-b1 Improved error handling and refactoring test to best practices, per jreback review. closes #315 * better error handling after get_response * docs for 0.5.0 * remove deprecation warnings: ix usage -> loc/iloc remove deprecation warnings: sortlevel usage -> sort_index * more resource cleaning * update changelog * skip enigma tests locally if no api key * fixturize test_yahoo_options * add in test.sh script * CI: use trusty dist
@Harrymon12 is a paid troll that spams about "MarketXLS" all over the internet. Here some examples of his "work":
Alias he uses to help find this post quickly: Harrison Delfino |
@m3nu Ahw. Oh really? I am just giving the facts about MarketXLS. |
@Harrymon12 This is a pandas site. If you have some tips about helping out on PANDAS and specifically this issue please feel free to do so. Otherwise your posts ARE spam. |
@javadba I understand. Thank you. :) |
Another small package I wrote to overcome the 401 issues |
@javadba My replacement for the old yahoo-finance module, YahooFinancials, can get all of the historical price data needed by pandas' users. YF can return daily, weekly, and monthly historical price and volume JSON data for all stocks, ETFs, indices, cryptocurrencies, currencies, and commodity futures available on Yahoo Finance. Most Stackoverflow questions I have encountered regarding the module seem to revolve around getting it to work with Pandas. As long as Yahoo Finance is running on it's new React setup (why they killed the old API late last year, they got a new web app), my module will get the financial data. Usage Example:
Returns
Anyway I'd be happy to fork a branch and build the price data from YahooFinancials into the panda-datareader's get_data_yahoo() method if you all want. I'd also be happy to work with one of your contributors to do so as well. Just let me know and I'd be happy to help! More details at: |
Hi, Yahoo Finance is working for me. I am confused as to why you guys saying it is deprecated? |
Maybe you are out of the great wall. Maybe others.
发件人: Sarah Chen [mailto:[email protected]]
发送时间: 2019年4月2日 6:44
收件人: pydata/pandas-datareader <[email protected]>
抄送: Subscribed <[email protected]>
主题: Re: [pydata/pandas-datareader] Issues with the data reader fetching yahoo finance (#315)
Hi, Yahoo Finance is working for me. I am confused as to why you guys saying it is deprecated?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<#315 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AnffVte7oyUokQHvkUflX6sJAlZ0Ln5xks5vcouwgaJpZM4NUwZN>.
|
It was deprecated, and then it was restored using a new implementation.
…On Fri, Apr 5, 2019, 14:59 GRECH ***@***.***> wrote:
Maybe you are out of the great wall. Maybe others.
发件人: Sarah Chen ***@***.***
发送时间: 2019年4月2日 6:44
收件人: pydata/pandas-datareader ***@***.***>
抄送: Subscribed ***@***.***>
主题: Re: [pydata/pandas-datareader] Issues with the data reader fetching
yahoo finance (#315)
Hi, Yahoo Finance is working for me. I am confused as to why you guys
saying it is deprecated?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<
#315 (comment)>,
or mute the thread<
https://github.com/notifications/unsubscribe-auth/AnffVte7oyUokQHvkUflX6sJAlZ0Ln5xks5vcouwgaJpZM4NUwZN>.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#315 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFU5RehtuRgFg_NuXpIZ7wzdv6OHRmfUks5vd1aygaJpZM4NUwZN>
.
|
Then it would be good to stay in the official documentation for version
0.70. Otherwise people get scared of using it or confused:)
On Fri, Apr 5, 2019 at 11:54 AM Kevin Sheppard <[email protected]>
wrote:
… It was deprecated, and then it was restored using a new implementation.
On Fri, Apr 5, 2019, 14:59 GRECH ***@***.***> wrote:
> Maybe you are out of the great wall. Maybe others.
>
> 发件人: Sarah Chen ***@***.***
> 发送时间: 2019年4月2日 6:44
> 收件人: pydata/pandas-datareader ***@***.***>
> 抄送: Subscribed ***@***.***>
> 主题: Re: [pydata/pandas-datareader] Issues with the data reader fetching
> yahoo finance (#315)
>
>
> Hi, Yahoo Finance is working for me. I am confused as to why you guys
> saying it is deprecated?
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub<
>
#315 (comment)
>,
> or mute the thread<
>
https://github.com/notifications/unsubscribe-auth/AnffVte7oyUokQHvkUflX6sJAlZ0Ln5xks5vcouwgaJpZM4NUwZN
>.
>
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <
#315 (comment)
>,
> or mute the thread
> <
https://github.com/notifications/unsubscribe-auth/AFU5RehtuRgFg_NuXpIZ7wzdv6OHRmfUks5vd1aygaJpZM4NUwZN
>
> .
>
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#315 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AVz5ewGbHBkppQ6MB9rxRk2HFtSFd3XKks5vd3HOgaJpZM4NUwZN>
.
|
Can anyone help me fix this error NotImplementedError: data_source=datetime.datetime(2015, 1, 1, 0, 0) is not implemented |
I recently started to learn python for finance. From this thread, I understood at some level the yahoo fin package related issues but while I started to run the following code, 'ticker' not found error is persistent. Could you please check it and guide me further? Here are the libraries used and the code: import tensorflow as tf import numpy as np import matplotlib def load_data(ticker, n_steps=50, scale=True, shuffle=True, lookup_step=1,
load the datadata = load_data(ticker, N_STEPS, lookup_step=LOOKUP_STEP, test_size=TEST_SIZE, feature_columns=FEATURE_COLUMNS) THIS GIVES FOLLOWING ERROR: NameError Traceback (most recent call last) NameError: name 'ticker' is not defined |
The error you've shared doesn't have anything to do with the pandas datareader package. You need to define ticker before passing it into the function. |
Thank you for responding. When I defined load_data as following, it did not ask for defining 'ticker' def load_data(ticker, n_steps=50, scale=True, shuffle=True, lookup_step=1, But when I use ticker below, it complains as "ticker" not defined. As I am novice to this, kindly guide. data = load_data(ticker, N_STEPS, lookup_step=LOOKUP_STEP, test_size=TEST_SIZE, feature_columns=FEATURE_COLUMNS) |
Good morning ... That's not the place for this kind of questions ...
You could look for correct answers in
https://bit.ly/35c8jlI or in shorturl.at/mMW47
If you add ticket = "whatever ticker stock you want" before call the
función, the first part of your problem will be solved.
Best regards
|
Apologies first issue/comment on GitHub. I will review proper protocol. Please correct me if this is not the correct place to put this.
RemoteDataError Traceback (most recent call last)
in ()
4 end = dt.datetime(2017, 5, 8)
5
----> 6 INPX = data.DataReader(INPX ,'yahoo', start, end)
7
8 #Convert Volume from Int to Float
C:\Users\randomname\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas_datareader\data.py in DataReader(name, data_source, start, end, retry_count, pause, session)
92 adjust_price=False, chunksize=25,
93 retry_count=retry_count, pause=pause,
---> 94 session=session).read()
95
96 elif data_source == "yahoo-actions":
C:\Users\randomname\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas_datareader\yahoo\daily.py in read(self)
75 def read(self):
76 """ read one data from specified URL """
---> 77 df = super(YahooDailyReader, self).read()
78 if self.ret_index:
79 df['Ret_Index'] = _calc_return_index(df['Adj Close'])
C:\Users\randomname\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas_datareader\base.py in read(self)
176 df = self._dl_mult_symbols(self.symbols.index)
177 else:
--> 178 df = self._dl_mult_symbols(self.symbols)
179 return df
180
C:\Users\randomname\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas_datareader\base.py in _dl_mult_symbols(self, symbols)
195 if len(passed) == 0:
196 msg = "No data fetched using {0!r}"
--> 197 raise RemoteDataError(msg.format(self.class.name))
198 try:
199 if len(stocks) > 0 and len(failed) > 0 and len(passed) > 0:
RemoteDataError: No data fetched using 'YahooDailyReader'
The text was updated successfully, but these errors were encountered: