ENH: Added method to pandas.data.Options to download all option data for...

[davidastephens]

... a ticker.

Also added a few helper functions. These functions could be applied to refactor some of the other methods.

[cancan101]

Why is call default to True and put to False? That is not intuitive to me.

[davidastephens]

Me neither - I matched it with the other methods (get_forward_data and get_near_stock_price).

[davidastephens]

Also note that the other methods say they return a dict, but actually return either a dataframe (if just one of call or put is chosen) or a list of 2 dataframes with no identifier. I've set up this method to return the dict, but it probably maybe it should be consistent either way..

[davidastephens]

I fixed the issues you pointed out that weren't related to consistency with the rest of the module.

Any thoughts on the consistency issues? I'm happy to refactor the other methods if you think that would be helpful.

[jasonstrimpel]

Hi, I don't know how to contribute changes but I noticed that the Chg column only has positive values. This is because yahoo only provides an class to display an up or down arrow to show the change. I modified the _unpack method to look inside the elements to find the appropriate class name and change the Chg to the appropriate sign.

def _unpack(row, kind):
    els = row.xpath('.//%s' % kind)
    res = []
    for i in els:
        img = i.xpath('.//img')
        if len(img)>0:
            cls=img[0].xpath('.//@class')[0]
            if cls == 'neg_arrow':
                res.append(float(i.text_content())*(-1.0))
            else:
                res.append(i.text_content())
        else:
            res.append(i.text_content())
    return res

[davidastephens]

Nice catch strimp. How about this?

def _unpack(row, kind):
    els = row.xpath('.//%s' % kind)
    return [_parse_row_values(val) for val in els]


def _parse_row_values(val):
    ret = val.text_content()
    img = val.xpath('.//img')
    if img:
        if img[0].xpath('.//@class')[0] == 'neg_arrow':
            ret = float(ret)*(-1.0)
    return ret

[davidastephens]

Or this:

def _unpack(row, kind):
    els = row.xpath('.//%s' % kind)
    return [_parse_row_values(val) for val in els]


def _parse_row_values(val):
    ret = val.text_content()
    if 'neg_arrow' in val.xpath('.//@class'):
        ret = float(ret)*(-1.0)
    return ret

[jasonstrimpel]

That works. Same results as my suggestion but much more compact and likely faster (avoids appending to a list). Thanks.

[jasonstrimpel]

Another issue I was having with lxml. In the _get_option_data and _get_expiry_months methods, I believe your original code uses lxml.html.parse directly on the set url variable. I was having issues with that (potentially proxy although my HTML_PROXY path variable is set).

I simply used urllib to solve in both methods:

    url = 'http://finance.yahoo.com/q/op?s={sym}'.format(sym=self.symbol)

    try:
        from lxml.html import parse
    except ImportError:
        raise ImportError("Please install lxml if you want to use the "
                          "{0!r} class".format(self.__class__.__name__))
    try:
        # i am failing the try when attempting to call parse(url)
        page = urllib.urlopen(url) # added this line
        doc = parse(page) # changed to pass page not url
    except _network_error_classes:
        raise RemoteDataError("Unable to parse expiry months from URL "
                              "{0!r}".format(url))

[davidastephens]

Hmm. I don't know what the problem is there, I didn't write that code, works fine for me. I have a couple fixes for the _get_expiry_months. I'll add them.

[jasonstrimpel]

Probably some proxy issue then.

Sent from mobile

On Dec 11, 2013, at 15:36, dstephens99 [email protected] wrote:

Hmm. I don't know what the problem is there, I didn't write that code,
works fine for me. I have a couple fixes for the _get_expiry_months. I'll
add them.

—
Reply to this email directly or view it on
GitHubhttps://github.com//pull/5602#issuecomment-30366529
.

[mcneary70]

New to git, new to python, new to pandas. For that reason I'm only commenting here and if it is in the wrong place I apologize.

There is an issue in data.py where if you request option data for CUR_MONTH and CUR_YEAR+1 you will only get CUR_MONTH/CUR_YEAR options. This is due to an unnecessary check in _get_option_data() to see if the passed in month is CUR_MONTH and using the URL without additional date parameters if it is. If you just simply pass the month and year parameters in the URL every time this problem goes away.

[jreback]

@mcneary70

why don't you create a new issue for that, and show some code that can reprod the problem?

thanks

[jreback]

@dstephens99 if you can rebase and get this passing, and prob needs some more tests, we can consider for 0.14

[davidastephens]

Sure, I will do that.

@mcneary70 I fixed that issue in my last commit.

[mcneary70]

@dstephens99 That is awesome, thanks!

[jreback]

pls rebase this in current master

need to update the doc strings as well

I am ok with changing the API here where it makes sense
eg best for all of these routines to return dataframes always (possibly multi level so u can hold put/calls) - but should always do the same thing

[davidastephens]

Rebased - I'll work on this some more this week.

[jreback]

gr8

as I say popose a nice pandonic API for this

[jreback]

@dstephens99 update?

[jreback]

@dstephens99 this needs some API work, can you comment on that

[cancan101]

I think you can clean up some of the logic using something like:

    df["rootExp"] = df.Symbol.str[0:-9]
    df["root"] = df.rootExp.str[0:-6]
    df["exp"] = pd.to_datetime(df.rootExp.str[-6:])

You can define has_tick (which would be better is named "is_nonstandard" (meaning the deliverable of the option basket is not 100 shares) to be: df["root"]==symbol

[jreback]

@dstephens99 can you show the API?

[davidastephens]

Sorry for the delay in responding, I've been really busy on a project at work.

What do you think about the API being a data frame if you request just calls or just puts and a dictionary of {'calls': call_df, 'puts' : put_df} if you request both? Pythonic?

[davidastephens]

Never mind, I just saw your prior comment about always returning a dataframe. I will do it that way.

[jreback]

make this line the same length as the title (otherwise the doc generator will complain)

[jreback]

I would make this 'Open_Int' (for consistency with the rest); also don't really like spaces in column names (pls change in doc-tring in all places). FYI, you might want to make 1 doc-string, and then just assign it (e.g. func.__doc__ = doc_string)``

[cancan101]

This provides an alternative source of options data: http://www.cboe.com/delayedquote/QuoteTableDownload.aspx

[cancan101]

This is failing for me:

In [3]:

tsla = pandas.io.data.Options("TSLA", "yahoo")
In [4]:

tsla.get_all_data()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-7b3741a59d2e> in <module>()
----> 1 tsla.get_all_data()

/home/alex/git/pandas/pandas/io/data.py in get_all_data(self, call, put)
   1109                 except AttributeError:
   1110                     meth_name = 'get_{0}_data'.format(name[:-1])
-> 1111                     frame = getattr(self, meth_name)(expiry=month)
   1112 
   1113                 all_data.append(frame)

/home/alex/git/pandas/pandas/io/data.py in get_call_data(self, month, year, expiry)
    774         for the expiry of the options.
    775         """
--> 776         return self._get_option_data(month, year, expiry, 'calls')
    777 
    778     def get_put_data(self, month=None, year=None, expiry=None):

/home/alex/git/pandas/pandas/io/data.py in _get_option_data(self, month, year, expiry, name)
    702             tables = getattr(self, table_name)
    703         except AttributeError:
--> 704             tables = self._get_option_tables(month, year, expiry)
    705 
    706         ntables = len(tables)

/home/alex/git/pandas/pandas/io/data.py in _get_option_tables(self, month, year, expiry)
    685         #Get the time of the quote, note this is actually the time of the underlying price.
    686         quote_time_text = root.xpath('.//*[@class="time_rtq"]')[0].getchildren()[0].text
--> 687         split = quote_time_text.split(",")
    688         timesplit = split[1].strip().split(":")
    689         timestring = split[0] + ", " + timesplit[0].zfill(2) + ":" + timesplit[1]

AttributeError: 'NoneType' object has no attribute 'split'

[davidastephens]

I referenced that bug above - its the difference in Quote_Time format between the weekend and the week. I'll fix that one tonight.

[davidastephens]

The CBOE website has an interesting quote at the bottom re downloading quote information electronically..

[cancan101]

Good call...

[jreback]

pls rebase this

[jreback]

to consistently return a multi-indexed DataFrame

[jreback]

can you rebase on master, and ping when ready

[davidastephens]

@jreback Rebased and green.

[jreback]

@dstephens99 thanks!

pls look at the docs after they are built, may need to adjust (after travis finishes they will update): http://pandas-docs.github.io/pandas-docs-travis/remote_data.html

[jreback]

@dstephens99 in particular, I suspect the appl optinos will now fail (as the 600 strike price is now gone after the split).

[jreback]

docs are built.... @jorisvandenbossche any thoughts?

[jorisvandenbossche]

This should be moved to v0.14.1.txt (or removed if it is already there)

[davidastephens]

Yes, its in v0.14.1.txt, I will remove it here.