Downloading 5 tweets and rate limit immediately reached

[John-Kaz]

A script using a for loop to run get_user_tweets one user at a time downloaded over a thousand users' 1 million tweets a couple of weeks ago. I updated the package and tried using get_all_tweets, again, one user at a time. This time, however, I get called on hitting the data limit almost immediately and the script keeps waiting for 15 minute intervals. For instance, the first time I tried today, the first user's 5 tweets downloaded just fine but when it attempted downloading the second user, I got hit with a 15 minute wait period. I have a developer Twitter account and bearer, and am nowhere near my 10 million monthly quota. Very befuddled...

[chainsawriot]

As your question is not an issue (e.g. bug report or issue request), I moved this here.

[chainsawriot]

@shmuhammad2004 BTW, "àti 😃" means you want to have tweets containing both àti and 😃 right? If it's the case, please quote all of them, i.e. (àti 😃).

require(academictwitteR)
#> Loading required package: academictwitteR
happy_emoji <- c("(àti 😃)", "(àti 😄)", "(àti 😆)",
                "(àti 😊)", "(àti 😍)", "(àti 😗)")

build_query(happy_emoji)
#> [1] "((àti 😃) OR (àti 😄) OR (àti 😆) OR (àti 😊) OR (àti 😍) OR (àti 😗))"

^{Created on 2021-07-25 by the reprex package (v2.0.0)}

[chainsawriot]

Actually, if you format your query correctly, it should work even without a fix.

[John-Kaz]

It is actually due to the 1 req per 1 sec limit of the fulltext search endpoint.

Usually, a request takes more than 1 sec. But for request with no result, it would take less than 1 sec.

Of course, the easiest way to deal with this is to add 1 sec sleep after an empty request.

A smarter approach is to keep track of the previous request timestamp (t0) and then compare the current time (t1) with the previous request time.

Delta := t1 - t0 < 1,
If Delta < 1, add a sleep time of the Delta plus a small surplus (e.g. 10 microsecs); then make the new request.

Thanks @chainsawriot! I added a 1 second wait time between iterations of the for loop, and this seems to have solved the problem.

[chainsawriot]

I just want to let you know that the current version 70e11bb has patched this.
Please consider updating.

[shmuhammadd]

@chainsawriot Thanks a lot. We appreciate you all for this project.