get_user_timeline inconsistently returns Error in make_query(url = endpoint_url, ...) something went wrong. Status code: 400

[maria-pro]

Please confirm the following

• I have searched the existing issues
• The behaviour of the program is deviated from what is described in the documentation.
• I can reproduce this problem for more than one time.
• This is NOT a 3-digit error -- it does not display an error message like something went wrong. Status code: 400.
• This is a 3-digit error and I have consulted the Understanding API errors vignette and the suggestions do not help.

Describe the bug

Hi
I am trying to use get_user_timeline() with my set of user that i got

data_tidy<-bind_tweets(data_path = "data", output_format = "tidy")
users<-as_tibble(unique(data_tidy$author_id))
users_short<-users$value

The same step runs with no issues one time or produces the error below. The step is:

user_timelines<-get_user_timeline(users_short[1201:1400],
                                  start_tweets = "2018-01-01T00:00:00Z",
                                  end_tweets = "2022-01-09T00:00:00Z",
                                  data_path = "user_timelines_new10__",
                                  n= Inf
)

The error I am getting from occasionally is:

Error in make_query(url = endpoint_url, params = params, bearer_token = bearer_token,  : 
  something went wrong. Status code: 400

No further details on what exactly is wrong...

When i split my batch and run it not in users_short[1201:1400] but it smaller batches it works (at times again with the same intermittent error).

How to fix this?

Thank you

Expected Behavior

to collect data as expected or at least provide more details on the error - i believe it is more likely Twitter API limitations related

Steps To Reproduce

users<-read_csv("https://raw.githubusercontent.com/maria-pro/daniel/main/users.csv")

users_short<-users$value

ser_timelines<-get_user_timeline(users_short[1201:1400],
start_tweets = "2018-01-01T00:00:00Z",
end_tweets = "2022-01-09T00:00:00Z",
data_path = "user_timelines_new10__",
n= Inf
)

Environment

R version 4.2.0 (2022-04-22)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Monterey 12.2.1

Anything else?

No response

[chainsawriot]

@maria-pro ?get_user_timeline suggests that this function is for getting tweets by a single user.

If you want to collect data for many users, I suggest purrr::map or purrr::walk.

But I agree that get_user_timeline should give a more informative error when x is a vector.

[maria-pro]

i tried the following

t<-users_short[1:2]

test<-map(.x=t,
  
  
  function(x)
  
  get_user_timeline(x,
                        start_tweets = "2018-01-01T00:00:00Z",
                        end_tweets = "2022-01-09T00:00:00Z",
                        data_path = "test",
                        n= Inf
))

and getting this now:

Warning messages:
1: Tweets will be bound in local memory as well as stored as JSONs. 
2: Tweets will be bound in local memory as well as stored as JSONs. 
3: Directory already exists. Existing JSON files may be parsed and returned, choose a new path if this is not intended.

I guess this is because i need to specify data_path and in map it is getting overwritten...

1 How can i recover the collected tweets

2 How to fix this in map?

[chainsawriot]

A simple fix:

test<-map(.x=t,
  
  
  function(x)
  
  get_user_timeline(x,
                        start_tweets = "2018-01-01T00:00:00Z",
                        end_tweets = "2022-01-09T00:00:00Z",
                        data_path = paste0("test_", x),
                        n= Inf
))

[chainsawriot]

But actually, I also wanted to report to you that the following works.

require(readr)
#> Loading required package: readr
require(academictwitteR)
#> Loading required package: academictwitteR

users <- readr::read_csv("https://raw.githubusercontent.com/maria-pro/daniel/main/users.csv")
#> Rows: 8744 Columns: 1
#> ── Column specification ────────────────────────────────────────────────────────
#> Delimiter: ","
#> dbl (1): value
#> 
#> ℹ Use `spec()` to retrieve the full column specification for this data.
#> ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

users_short<-users$value

ser_timelines <- get_user_timeline(users_short[1201:1203],
start_tweets = "2018-01-01T00:00:00Z",
end_tweets = "2022-01-09T00:00:00Z",
data_path = academictwitteR:::.gen_random_dir(),
n= Inf
)
#> Warning: Tweets will be bound in local memory as well as stored as JSONs.
#> user:  1.183385e+18 
#> Total pages queried: 1 (tweets captured this page: 0).
#> This is the last page for : finishing collection.
#> user:  1.248843e+18
#> Warning: Directory already exists. Existing JSON files may be parsed and
#> returned, choose a new path if this is not intended.
#> Total pages queried: 1 (tweets captured this page: 99).
#> Total pages queried: 2 (tweets captured this page: 100).
#> Total pages queried: 3 (tweets captured this page: 70).
#> This is the last page for : finishing collection.
#> user:  3051971884
#> Warning: Directory already exists. Existing JSON files may be parsed and
#> returned, choose a new path if this is not intended.
#> Total pages queried: 1 (tweets captured this page: 99).
#> Total pages queried: 2 (tweets captured this page: 98).
#> Total pages queried: 3 (tweets captured this page: 97).
#> Total pages queried: 4 (tweets captured this page: 99).
#> Total pages queried: 5 (tweets captured this page: 100).
#> Total pages queried: 6 (tweets captured this page: 98).
#> Total pages queried: 7 (tweets captured this page: 96).
#> Total pages queried: 8 (tweets captured this page: 100).
#> Total pages queried: 9 (tweets captured this page: 99).
#> Total pages queried: 10 (tweets captured this page: 98).
#> Total pages queried: 11 (tweets captured this page: 98).
#> Total pages queried: 12 (tweets captured this page: 99).
#> Total pages queried: 13 (tweets captured this page: 100).
#> Total pages queried: 14 (tweets captured this page: 98).
#> Total pages queried: 15 (tweets captured this page: 91).
#> Total pages queried: 16 (tweets captured this page: 99).
#> Total pages queried: 17 (tweets captured this page: 97).
#> Total pages queried: 18 (tweets captured this page: 94).
#> Total pages queried: 19 (tweets captured this page: 95).
#> Total pages queried: 20 (tweets captured this page: 50).
#> This is the last page for : finishing collection.

^{Created on 2022-09-21 by the reprex package (v2.0.1)}