Pull request for Sift 47 #6
Conversation
|
Any idea why the build is failing? @estroz |
|
Not exactly sure, everything works fine on my machine. its the requirements installation, which is weird |
| @@ -1,6 +1,18 @@ | |||
| language: python | |||
| python: 2.7 | |||
| # Setup anaconda | |||
| before_install: | |||
There was a problem hiding this comment.
Wow Travis is not happy with us
| return df, feature_names | ||
|
|
||
| # Initialize an LDA model object with 20 topics and 'online' learning method | ||
| def init_and_fit_lda_(dataframe, num_topics=20, rand_state=1, learn_method='online'): |
There was a problem hiding this comment.
Do we need to be explicit about num_topics? How do we choose a good value for this?
There was a problem hiding this comment.
The runtime is O(NKV) where N = #docs, V = #words in vocabulary and K = #topics, so the runtime scales linearly with K and N/V kept relatively static. The more topics, the more granularity we'll see for similarity in meaning of the words in each topic-word set, i.e. the clarity of topic meaning, but longer runtime. I chose 20 because that's what all the examples have pointed to, but there's a good resource I found on choosing K (http://archive.is/KBGwt) which I need to go through.
| print(sentimentResults) | ||
| assert sentimentResults[0]== 45.23809523809524 | ||
| assert sentimentResults[1] == 54.761904761904766 | ||
| assert sentimentResults['pos_pct']== 45.23809523809524 |
| @@ -2,7 +2,7 @@ | |||
| import json | |||
There was a problem hiding this comment.
Can we change the name of this test to something more informative?
| @@ -1,5 +1,5 @@ | |||
| # Project-specific files | |||
There was a problem hiding this comment.
Can we remove the siftnlp folder from root if there isn't anything in it? Maybe replace with a lib folder for functions common to multiple jobs?
EDIT never mind
There was a problem hiding this comment.
Can we also consolidate data/* and test/test_data/*, unless there's a good reason to have both of them
There was a problem hiding this comment.
Is this LDA stuff testable in a useful way?
There was a problem hiding this comment.
Once we have curated data (hand-selected topics for pieces of feedback) I can set up tests for ranges of correctness.
| polarityProp.append(negative_percentage) | ||
| return(polarityProp) | ||
|
|
||
| positive = sum(filter(lambda x: x > 0, polarity_scores)) |
No description provided.