Warning: Rate limit exceeded - 88 (Using get_timeline())

[IWeavey]

I have been using some code to output a dataframe containing some tweet information from get_timeline() using an input of a vector of Twitter user_ids (see bottom of the post for the code). Essentially the program checks to make sure I have enough requests left before continuing (17 is needed when n= 3200). I also have a tryCatch() in case some accounts are protected or have no tweets, etc.. It has been working for weeks producing results from dozens of vectors containing 500 user ids without producing a single error.

However, today it started producing this error at random (I checked to see if it was happening around a certain user id, but that is not the case). This type of error I am receiving is identical to the one described in Issue #266 opened a week ago.

Attempt 1: (The top numbers are the number of requests I have left)
...

49
40
Warning: Rate limit exceeded - 88

Error in if (rl$remaining < 20) { : argument is of length zero

This happened 396 users into my vector.

Attempt 2: (The top numbers are the number of requests I have left)
...

454
438
Warning: Rate limit exceeded - 88

Error in if (rl$remaining < 20) { : argument is of length zero

This happened 70 users into my vector.

Moreover as soon as the error popped up, I used (rate_limit('get_timeline')) immediately and there was 422 requests as I expected (using 16 requests is common for one user). My rate limit had clearly not been exceeded, and as shown through my trials the error happens at random times.

# A tibble: 1 x 7
  query                  limit remaining reset            reset_at            timestamp           app            
  <chr>                  <int>     <int> <time>           <dttm>              <dttm>              <chr>          
1 statuses/user_timeline   900       422 6.00734655062358 2018-07-08 18:34:14 2018-07-08 18:28:14 NetworkVoyager2

I guess I just need add some sort of exception for this error, but if anybody has any other suggestions that would be appreciated.

My Code.

#my code
tweet_extractor <- function(users){
  output_list <- list()
  for (i in 1:length(users)) {
    rl <- rate_limit('get_timeline')
    message(rl$remaining)
    if (rl$remaining < 20){
      message("Waiting for ", rl$reset, " minutes for rate reset!")
      Sys.sleep(as.numeric(rl$reset, "secs"))
    }
    output_list[[i]] <- tryCatch(
      {get_timeline(users[i], n = 3200)[,c(1,3,15)]},
      error = function(e){return_nothing()},
      warning = function(w){suppressWarnings(return_nothing())}
    )
  }
  output = do.call(rbind, output_list)
  return(output)
}

Here is the vector of users I was using in case someone really wants to try and replicate it

  [1] "2914070918"          "805207941720768512"  "895100866738487296"  "14150137"            "4060962886"          "462492670"          
  [7] "384761859"           "221478246"           "477268420"           "399156475"           "948378639225249792"  "154129798"          
 [13] "160994023"           "37599351"            "833030483252563968"  "918650935441874944"  "1623709814"          "2434092937"         
 [19] "319387069"           "305706781"           "3007525490"          "31303187"            "130522705"           "5392522"            
 [25] "14604008"            "994540152465821696"  "168911975"           "3318608640"          "143788214"           "845931919879667712" 
 [31] "1042165135"          "828478304"           "885932010"           "1864576153"          "72441434"            "1903916654"         
 [37] "42366225"            "1559377141"          "3291379333"          "24417870"            "807638995937075200"  "67480485"           
 [43] "83039050"            "2432158448"          "850645452366262272"  "3240792759"          "202952121"           "61771187"           
 [49] "313381067"           "720785987740172289"  "222921304"           "541138500"           "28870086"            "3258926170"         
 [55] "827569324219043840"  "2394758311"          "374392774"           "895930458675986432"  "1008503286284992512" "369040310"          
 [61] "1409514882"          "3404580113"          "34104616"            "17506698"            "1211794507"          "740030108371537920" 
 [67] "17400374"            "4385021773"          "2613358347"          "606513809"           "882522278780706816"  "2150278275"         
 [73] "150899875"           "333383304"           "815689902"           "2572939838"          "539954620"           "4168919367"         
 [79] "21024630"            "905244526821773312"  "4863282653"          "897259993371729923"  "4876091717"          "23916856"           
 [85] "242245578"           "977773665910738944"  "397457300"           "2285565848"          "80551951"            "806036195734147072" 
 [91] "4686170306"          "150619905"           "164969445"           "82396141"            "823031755"           "2628390126"         
 [97] "991128366198804485"  "976943729318735872"  "729587402197618688"  "1264567338"          "2967515559"          "745841902797340672" 
[103] "287635335"           "65685839"            "105375099"           "260494836"           "907303680"           "3888856528"         
[109] "867901247596179456"  "907729596161810433"  "280659753"           "4114610705"          "880371080887750658"  "500385733"          
[115] "22461427"            "859732597"           "1729980547"          "4539388937"          "104443458"           "29808890"           
[121] "274119641"           "76235236"            "617759565"           "847459253883289601"  "576241232"           "538447484"          
[127] "129865155"           "737381132673658880"  "819087104995008513"  "39837411"            "440954376"           "16788661"           
[133] "896751120047099904"  "912674310606770176"  "16540829"            "30029450"            "333663029"           "16903190"           
[139] "2496927404"          "19693183"            "3417318741"          "73165790"            "8453682"             "728592182"          
[145] "836305915452272641"  "52000267"            "120962765"           "826508178158387200"  "725818770623762432"  "858083159912710145" 
[151] "743569437043920899"  "94335344"            "58233603"            "762051267623473152"  "347588162"           "34281855"           
[157] "17649829"            "807699274045779970"  "847238528"           "949400438"           "133640329"           "2568058225"         
[163] "184609000"           "3344203452"          "964035774"           "4903543349"          "852340814172868608"  "458406193"          
[169] "2177598939"          "88523228"            "744926451598688256"  "934539692"           "203879364"           "200230420"          
[175] "2820058180"          "315584738"           "419107960"           "27286571"            "862809714"           "787711857305980928" 
[181] "1012587471597547520" "141052723"           "305506502"           "2936009137"          "920160588650516480"  "925590970569232384" 
[187] "78621637"            "4127736912"          "467371318"           "73576826"            "248275105"           "18936161"           
[193] "422822656"           "213928062"           "2572530289"          "919027070"           "27372490"            "936204583"          
[199] "3301213924"          "192504878"           "1604107964"          "988071074"           "601485766"           "2415254493"         
[205] "3374267140"          "25073877"            "889066228752367616"  "115801762"           "444984054"           "715864298"          
[211] "26866148"            "753937513710968832"  "919275226768297985"  "2151669733"          "179756684"           "771710377679814656" 
[217] "825042781584240640"  "46279037"            "2290170218"          "743448473748463622"  "182317200"           "947228869127561216" 
[223] "2677713829"          "744035168835477505"  "375367471"           "733509545830567937"  "3075132115"          "308711195"          
[229] "2280783692"          "315840116"           "115686771"           "774797223066341377"  "1062798174"          "18536385"           
[235] "779614816725893120"  "16818600"            "755739896459898880"  "780777144532119552"  "48029066"            "15524689"           
[241] "859297966581891072"  "883002414726557696"  "223322706"           "75089197"            "123281817"           "1343734051"         
[247] "841807948200329216"  "739655770384400384"  "1117068859"          "894531365823238146"  "78067338"            "312887235"          
[253] "3327720838"          "1740171890"          "3796624395"          "1328222208"          "3767682072"          "17507545"           
[259] "17948810"            "2886819885"          "347108490"           "37251100"            "155264719"           "2687152688"         
[265] "3301902172"          "105570305"           "802846266283749376"  "71379929"            "3184795599"          "2720603757"         
[271] "946829450976661504"  "260979139"           "3300437193"          "727337190427119617"  "949813914"           "924541302"          
[277] "25317426"            "17626692"            "900446878550159360"  "730023720958365697"  "3177171018"          "708877047469314048" 
[283] "152564203"           "176775753"           "109298652"           "132570884"           "2963079614"          "763239870324150273" 
[289] "244252293"           "430613946"           "838053222052945920"  "546527528"           "235747358"           "315575454"          
[295] "860170163097788416"  "734923792082952192"  "2709264608"          "2173009363"          "910398080268115968"  "49161904"           
[301] "782658296465403904"  "843181875267080192"  "214364824"           "3051966975"          "113487739"           "526999980"          
[307] "153769041"           "2316122581"          "236926356"           "808175644903440384"  "25621031"            "1638491166"         
[313] "24947285"            "410033782"           "164474716"           "351704282"           "74701069"            "90056351"           
[319] "807014065080037377"  "909997239073832960"  "782619390986465280"  "804138736908910595"  "142009582"           "965810168"          
[325] "321587900"           "179562358"           "54341363"            "1310226354"          "4040705062"          "826393587264942080" 
[331] "892100880270577664"  "3123451734"          "326905365"           "364807455"           "1172225336"          "72628276"           
[337] "1267798494"          "848489616965595136"  "107234633"           "278473230"           "864053889347330049"  "42420346"           
[343] "915027480"           "788752980019408900"  "1023983360"          "63707233"            "117758095"           "1011648944"         
[349] "1270293871"          "169274239"           "481226258"           "737302248338952193"  "955643807860576256"  "2475139904"         
[355] "3089855932"          "770206624275394560"  "4099890041"          "822215673812119553"  "984261871824920577"  "791482596865609728" 
[361] "220026953"           "221575779"           "116888674"           "2227556974"          "4082553437"          "385132278"          
[367] "67534455"            "212733925"           "1002145785478287361" "1010623047538282496" "1630330410"          "232275603"          
[373] "817143588568109056"  "1104703459"          "2473715683"          "92118192"            "14407302"            "960238034917756929" 
[379] "798207991006855170"  "388612336"           "3040425511"          "594040592"           "24197885"            "208254359"          
[385] "377836074"           "67886297"            "975947918585880576"  "875685446721908736"  "163960831"           "4464742761"         
[391] "854742143130361857"  "994703524943581185"  "19397785"            "389201324"           "506424815"           "1006317872015032320"
[397] "713383487819091968"  "159286187"           "1319187156"          "957538104100999168"  "3118872163"          "371764330"          
[403] "2176057987"          "2988144971"          "336210713"           "16063257"            "261887239"           "78183672"           
[409] "2428678796"          "782225207356379138"  "2385959732"          "850464929698263040"  "282875270"           "397986404"          
[415] "770682888"           "3655070235"          "18967998"            "944791079781482496"  "98795912"            "961182865"          
[421] "46726590"            "774057035130347520"  "946768499732606976"  "2223785116"          "103249668"           "884097932118679553" 
[427] "863766437663391744"  "203119044"           "295785445"           "26285962"            "969555562571714561"  "726859780187394049" 
[433] "789340552382717953"  "2435045539"          "18810387"            "302257627"           "729146150457122816"  "14109485"           
[439] "3145607457"          "2815931773"          "386300590"           "357027057"           "761787197368524800"  "3589323675"         
[445] "743832828547338241"  "594886554"           "2580699308"          "1191169219"          "806158140274249728"  "51280043"           
[451] "124388793"           "717927308229742593"  "2388292507"          "482671259"           "734334955996762112"  "235058712"          
[457] "935934325"           "2915203605"          "817012838329884672"  "56623202"            "4199940134"          "410644296"          
[463] "902406987010289664"  "217901786"           "70193066"            "46178001"            "1858876064"          "363880689"          
[469] "97522873"            "2436191736"          "1111838276"          "1577663222"          "2181053413"          "738777685032194048" 
[475] "262350069"           "84447226"            "629097517"           "92123079"            "3051480362"          "216148052"  

[IWeavey]

The solution here is the same as in Issue #266.

"I have figured out what your problem was. There is a rate limit on the number of times you can use rate_limit(), which is 180. If you call rate_limit() and look at the 11th row you will see it. This is what it looks like for me.

11 application/rate_limit_status 180 178 14.8827226996422 2018-07-10 00:09:04 2018-07-09 23:54:12 NetworkVoyager2