feat: Add initial implementation of data analysis tools

[AnkitRajSri]

I have added the muser-data-app project to the branch of the same name.

The app currently offers the below functionalities:
• The app is now able to extract data from Spotify API, save a raw CSV file with the timestamp, and perform ETL operation to dump the extracted data in SQL Server table.
• In addition to the data extraction functionality, we have incorporated functionality to build a doc2vec NLP model and train it on the data dumped in the SQL table.
• There is one final functionality to expand the muser data, collected from FireBase, with metadata information collected from Spotify. The app first queries Spotify for an exact match with the muser record (artist, track, and album), and in case there is no exact match, the app utilizes the doc2vec model to predict the most similar match in the SQL table.

We are still working to make the UI more interactive.

[CLAassistant]

All committers have signed the CLA.

[barbeau]

Thanks @AnkitRajSri! Some comments are in-line below.

@pradeepsalunke after @AnkitRajSri makes changes to the hard-coded paths, could you please try cloning this project and running it from the command line? See the bottom of the PR for instructions how to do this. I want to make sure that anyone can clone this project and run it on their own machine without needing to edit any files (e.g., fixing hard-coded paths).

Also, the project needs a README in the root of the project that explains what the project does and how to run it. Please see https://github.com/CUTR-at-USF/muser-firebase-export for an example.

All the .pyc files should also be ignored and not included in the Git repository because they are generated by the Python compiler. You can add them to the .gitignore, either in the root directory or another one in the sub-directory.

You can make these changes to your local branch on your computer, commit them, and then push to GitHub and they will show up in this PR.

[barbeau]

This path is hard-coded and won't work on someone else's computer. Could you please change to a relative project path that will work across different computers?

[pradeepsalunke]

Hi Sean, We have closed that pull request as it was not cloned from main  and raised a new and corrected one. Regards,Pradeep Pradeep SalunkeGraduate Student,Masters in Business Analytics and Information SystemUniversity of South Florida

[barbeau]

@AnkitRajSri Either way works - I'd prefer to continue in this pull request since I already left review comments here. So please re-open this and you'll just need to push new commits to the branch on your fork repository at AnkitRajSri to update it - https://github.com/AnkitRajSri/muser-data-analysis/tree/muser-data-app.

[AnkitRajSri]

@barbeau Is it okay to continue working from the forked repository on my profile, as I just created it for testing and the pull request was created from that repository by mistake?

[barbeau]

s it okay to continue working from the forked repository on my profile, as I just created it for testing and the pull request was created from that repository by mistake?

Yes, that's fine, as long as all the changes are merged back into this project via pull requests.

[AnkitRajSri]

Reopening the pull request as per Sean's suggestion.

[AnkitRajSri]

Hey Sean,

I have implemented all the comments, could you review and let me know if it needs any further updates?

Thanks,
Ankit Raj

[barbeau]

Thanks @AnkitRajSri. Another comment in-line.

Also, the .pyc files are still under version control - you'll need to delete them and commit that.

@pradeepsalunke Were you able to clone this repository and run it locally on your machine without changing any code?

[pradeepsalunke]

Hello team, I was having health issues during this week and was resting and Im on the verge of recovery. I will do end to end testing and report any bugs before this weekend that is 20th December.Sorry for the inconvenience caused.Thanks Pradeep Sent from Yahoo Mail for iPhone On Thursday, December 17, 2020, 1:44 AM, Sean Barbeau <[email protected]> wrote: @barbeau requested changes on this pull request. Thanks @AnkitRajSri. Another comment in-line. Also, the .pyc files are still under version control - you'll need to delete them and commit that. @pradeepsalunke Were you able to clone this repository and run it locally on your machine without changing any code? In config.py:

@@ -0,0 +1,30 @@

+class Config: + """ + Use this class to share any default attributes with any subsequent + classes that inherit from Config. + """ + DEBUG = False + TESTING = False + + # Only required when using the session object + # Generated with secrets.token_urlsafe(16) + # You could also use os.urandom(16) + SECRET_KEY = "muser_data_app" Did something change? - I still see the credentials above. They should be removed from the committed code and if necessary read it programmatically from an unversioned file. Need information in the README for how to configure this too. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[AnkitRajSri]

@barbeau I have deleted the .pyc files from the remote branch, hide the secret key, and added instructions for the config.py file in the README file.

@pradeepsalunke Let me know if you face any issues while testing the application on your local system.

[pradeepsalunke]

Hi Team,
I have tested the application locally and its working fine with few errors.

[pradeepsalunke]

1. Previous message doesn't get hidden even after clicking another button
2. @AnkitRajSri is taking muser data in the form of a csv that we provided to expedite, it will be better if we can access the data directly from the muser app database, build it with the spotify features and then save as a SQL table.

[pradeepsalunke]

Hi Team, I have tested application on my local application works as accepted except 3rd funcationlity which is to build the data.Below are the screen shotsPFA Regards,Pradeep Pradeep SalunkeGraduate Student,Masters in Business Analytics and Information SystemUniversity of South Florida [email protected] On Monday, 21 December 2020, 08:35:01 PM IST, Ankit Raj <[email protected]> wrote: @barbeau I have deleted the .pyc files from the remote branch, hide the secret key, and added instructions for the config.py file in the README file. @pradeepsalunke Let me know if you face any issues while testing the application on your local system. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[AnkitRajSri]

@barbeau @pradeepsalunke Sorry for the delayed response, I was traveling back to the US and couldn't look into your comment.
@pradeepsalunke thanks for the update, I am looking into the third functionality and will fix it asap.

[AnkitRajSri]

@barbeau @pradeepsalunke Seems like we were exhausting the Spotify API limit when we were using the muser data builder functionality after harvesting data from Spotify.
I wasn't able to replicate the issue when I tried to run just the data builder functionality.
I, however, have added a sleep condition to handle the API exhaustion, this should take care of the issue, even if we use all the functionalities together.

[barbeau]

My understanding is that all the issues are resolved and @AnkitRajSri and @pradeepsalunke are both able to clone this PR and run from scratch on their own machines, with the exception of #4, which doesn't always happen because it's related to an API quota. So, I'm going to go ahead and merge so we can iterate on the work in this PR.