Update error capturing #309
Conversation
Added an option to retrieve erroneous tweets with their error title. Changes are made in get_tweets() and df_to_json() in order to bind the erroneous tweets to the bound dataframe and make error_-json files in the data_path
Function to access the API v2 Tweets Lookup (https://developer.twitter.com/en/docs/twitter-api/tweets/lookup/introduction) function. This is useful when looking to rehydrate tweets via their tweet IDs. An option to catch errors in the lookup process is added (see changes to utils for implementation)
added total success overview at the end of the loop
minor adjustment to make sure the reporting doesn't break the function when bind_tweets == F ("Retrieved" and "Total Tweets" verbose messages rely on the new_rows object which is only made if tweets are bound)
Revert "update to 0.3.1"
…d bind_errors() as a convenience function to bind errors into a dataframe
|
@TimBMK TODO
Do you know how to write tests? If yes, it would be useful. |
|
I've added the doc files. I'm not familiar with tests other than the basic devtools::check() though. I took a look at your Guide for writing tests, but that only covers handling the dummy API. I can write some tests if you give me some pointers though |
|
I've added some test data and a test for bind_errors(). I've basically copied the bind_tweets() tests, since the functionality is more or less equal. Let me know if that's what you'd expect from the tests! Edit: looks like test coverage is still failing. Could you explain to me why @chainsawriot ? |
|
@TimBMK One of the possible reasons is that many tests assume an envvar academictwitteR/.github/workflows/test-coverage.yaml Lines 17 to 18 in 2809432 If it still doesn't work, I will try to merge it to a secondary branch here and see how it goes. |
|
@TimBMK I can run (and pass) those |
|
Bumping this. Will this now pass given changed we've made to the gh actions? |
|
It would be great if it did, it also seems a little strenuous having to go via @chainsawriot for the actual commits every time. I'm still not sure why the tests fail on my side, I set up the TWITTER_BEARER Secret months ago |
I've added support for the error capturing mechanism to get_all_tweets() and get_user_timeline(). Among other things, this may help with issue #307, as it provides more variety in terms of test data. In practice I also found it useful to capture errors when retrieving timelines or tweets, as it gives useful information on the data retrieved (e.g., when less data than expected is returned). Furthermore, I added a convenience function to bind errors into a dataframe, comparable to bind_tweets.
All functions that rely on make_query() directly rather than get_tweets() currently do not support error capturing, as they do not store their output in .json files. Looking forward, we might consider adding error capturing mechanisms for these functions as well.
This should not interfere with #308, as the naming conventions remain unchanged. bind_errors() merely gathers .json files with the "errors_" prefix. The changes in get_all_tweets() and get_user_timeline() only add the "error" T/F variable to be passed on to get_tweets().