a new way to make searches

[yunusemrecatalcam]

Changed url for search, updated parser to the new page.
After the search you can get tweets from "c.search_tweet_list" as a list

timestamps, fav-rt counts does not return with this page but it is the best option for now

example

c = twint.Config()
c.Search = "yey"
c.Store_object = True
c.Limit = 100
twint.run.Search(c)
tlist = c.search_tweet_list
print("hacked that out")

@pielco11

[Bobpick]

I've edited the files as shown in your changes.
If I want to search for tweets by a user, say, @nbcnews, what would the command line syntax look like?

[Vickycats]

Hi Thank you for the quick changes. I have unistalledand reinstalled Twint using both the 'clone github' approach and a simple pip Install Twint but I am still not able to get Tweets using the code in the example above. All was fine until yesterday. I am relatively new to Twint and Git, but how do I make sure the 'fixed' code is installed on my machine - is it available to install with pip? Thank you!

[essentialols]

I've installed your commit using pip3 install --user --upgrade git+https://github.com/yunusemrecatalcam/twint.git, double-checked that the files were correctly copied onto my computer, and it's unfortunately also not working for me. I don't get an error message but it just runs forever.

[yunusemrecatalcam]

@Vickycats @essentialols
You should fetch the branch from my fork, you can use

git clone --branch=twitter_legacy2 https://github.com/yunusemrecatalcam/twint.git

to fetch and checkout to branch while cloning.

@Bobpick cli will not print output with this but you can easily follow the method above.

[Bobpick]

I'm still getting the same error.

[yunusemrecatalcam]

let's run a "git log" and be sure you're running the right code @Bobpick

[boshez]

I made the changes but how do I use it?

[essentialols]

@Bobpick you'd say
c.Search = "from:nbcnews"

It works for me now. I installed it using pip3 install --user --upgrade git+https://github.com/yunusemrecatalcam/twint.git@twitter_legacy2

Thanks so much yunusemrecatalcam.

[yunusemrecatalcam]

@boshez create a python file on the twint main directory and try that code

import twint
c = twint.Config()
c.Search = "yey"
c.Store_object = True
c.Limit = 100
twint.run.Search(c)
tlist = c.search_tweet_list

[boshez]

@yunusemrecatalcam how did you manage to run it? I am a bit confused.

[boshez]

@boshez create a python file on the twint main directory and try that code

import twint
c = twint.Config()
c.Search = "yey"
c.Store_object = True
c.Limit = 100
twint.run.Search(c)
tlist = c.search_tweet_list

I did but it gave me an idented error on line 62.

[hockeybro12]

It works for me, thanks for the help! Is it possible to get more information about the tweets like before when running twint.storage.panda.Tweets_df? For example, I don't have data about the # of RT's, # of likes, etc. anymore.

[Bobpick]

I ran this:
pip3 install --user --upgrade git+https://github.com/yunusemrecatalcam/twint.git@twitter_legacy2

and I'm running it, it seems to have suppressed the sleeping notifications and nothing is happening. I'll let it time out and let you know. How do I do a git log?

[boshez]

I ran this:
pip3 install --user --upgrade git+https://github.com/yunusemrecatalcam/twint.git@twitter_legacy2

and I'm running it, it seems to have suppressed the sleeping notifications and nothing is happening. I'll let it time out and let you know. How do I do a git log?

How did you run it? I am struggling.

[Bobpick]

I ran this:
pip3 install --user --upgrade git+https://github.com/yunusemrecatalcam/twint.git@twitter_legacy2
and I'm running it, it seems to have suppressed the sleeping notifications and nothing is happening. I'll let it time out and let you know. How do I do a git log?

How did you run it? I am struggling.

I'm struggling to. This is the result of running it:
Traceback (most recent call last):
File "/home/bob/.local/lib/python3.6/site-packages/twint/get.py", line 148, in Response
async with session.get(url, ssl=True, params=params, proxy=httpproxy) as response:
File "/home/bob/.local/lib/python3.6/site-packages/aiohttp/client.py", line 1012, in aenter
self._resp = await self._coro
File "/home/bob/.local/lib/python3.6/site-packages/aiohttp/client.py", line 504, in _request
await resp.start(conn)
File "/home/bob/.local/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 847, in start
message, payload = await self._protocol.read() # type: ignore # noqa
File "/home/bob/.local/lib/python3.6/site-packages/aiohttp/streams.py", line 591, in read
await self._waiter
concurrent.futures._base.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/bob/.local/bin/twint", line 8, in
sys.exit(run_as_command())
File "/home/bob/.local/lib/python3.6/site-packages/twint/cli.py", line 311, in run_as_command
main()
File "/home/bob/.local/lib/python3.6/site-packages/twint/cli.py", line 303, in main
run.Search(c)
File "/home/bob/.local/lib/python3.6/site-packages/twint/run.py", line 424, in Search
run(config, callback)
File "/home/bob/.local/lib/python3.6/site-packages/twint/run.py", line 323, in run
get_event_loop().run_until_complete(Twint(config).main(callback))
File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
return future.result()
File "/home/bob/.local/lib/python3.6/site-packages/twint/run.py", line 249, in main
await task
File "/home/bob/.local/lib/python3.6/site-packages/twint/run.py", line 296, in run
await self.tweets()
File "/home/bob/.local/lib/python3.6/site-packages/twint/run.py", line 194, in tweets
await self.Feed()
File "/home/bob/.local/lib/python3.6/site-packages/twint/run.py", line 47, in Feed
response = await get.RequestUrl(self.config, self.init, headers=[("User-Agent", self.user_agent)])
File "/home/bob/.local/lib/python3.6/site-packages/twint/get.py", line 119, in RequestUrl
response = await Request(_url, params=params, connector=_connector, headers=headers)
File "/home/bob/.local/lib/python3.6/site-packages/twint/get.py", line 143, in Request
return await Response(session, url, params)
File "/home/bob/.local/lib/python3.6/site-packages/twint/get.py", line 149, in Response
return await response.text()
File "/home/bob/.local/lib/python3.6/site-packages/async_timeout/init.py", line 45, in exit
self._do_exit(exc_type)
File "/home/bob/.local/lib/python3.6/site-packages/async_timeout/init.py", line 92, in _do_exit
raise asyncio.TimeoutError
concurrent.futures._base.TimeoutError

[hockeybro12]

I ran this:
pip3 install --user --upgrade git+https://github.com/yunusemrecatalcam/twint.git@twitter_legacy2
and I'm running it, it seems to have suppressed the sleeping notifications and nothing is happening. I'll let it time out and let you know. How do I do a git log?

How did you run it? I am struggling.

I installed it that way and the example in the original post at the top of this thread works for me.

[boshez]

I ran this:
pip3 install --user --upgrade git+https://github.com/yunusemrecatalcam/twint.git@twitter_legacy2
and I'm running it, it seems to have suppressed the sleeping notifications and nothing is happening. I'll let it time out and let you know. How do I do a git log?

How did you run it? I am struggling.

I installed it that way and the example in the original post at the top of this thread works for me.

I am facing identation errors when I run the script.

[Bobpick]

I ran this:
pip3 install --user --upgrade git+https://github.com/yunusemrecatalcam/twint.git@twitter_legacy2
and I'm running it, it seems to have suppressed the sleeping notifications and nothing is happening. I'll let it time out and let you know. How do I do a git log?

How did you run it? I am struggling.

I installed it that way and the example in the original post at the top of this thread works for me.

I am facing identation errors when I run the script.

Go to a good script and look how many spaces there are, and work from that.

[estebanpdl]

I run pip install --user --upgrade git+https://github.com/yunusemrecatalcam/twint.git@twitter_legacy2
It removed the sleeping-loop, but it did not find the account.

twint -u estebanpdl --csv -o test.csv

Threw:

CRITICAL:root:twint.get:User:
Traceback (most recent call last):
  File "c:\anaconda3\envs\bolivia\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\anaconda3\envs\bolivia\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Anaconda3\envs\bolivia\Scripts\twint.exe\__main__.py", line 7, in <module>
  File "C:\Users\Usuario\AppData\Roaming\Python\Python38\site-packages\twint\cli.py", line 311, in run_as_command
    main()
  File "C:\Users\Usuario\AppData\Roaming\Python\Python38\site-packages\twint\cli.py", line 303, in main
    run.Search(c)
  File "C:\Users\Usuario\AppData\Roaming\Python\Python38\site-packages\twint\run.py", line 424, in Search
    run(config, callback)
  File "C:\Users\Usuario\AppData\Roaming\Python\Python38\site-packages\twint\run.py", line 323, in run
    get_event_loop().run_until_complete(Twint(config).main(callback))
  File "c:\anaconda3\envs\bolivia\lib\asyncio\base_events.py", line 616, in run_until_complete
    return future.result()
  File "C:\Users\Usuario\AppData\Roaming\Python\Python38\site-packages\twint\run.py", line 249, in main
    await task
  File "C:\Users\Usuario\AppData\Roaming\Python\Python38\site-packages\twint\run.py", line 266, in run
    raise ValueError("Cannot find twitter account with name = " + self.config.Username)
ValueError: Cannot find twitter account with name = estebanpdl

[Bobpick]

I run pip install --user --upgrade git+https://github.com/yunusemrecatalcam/twint.git@twitter_legacy2
It removed the sleeping-loop, but it did not find the account.

twint -u estebanpdl --csv -o test.csv

Threw:
Use: twint -s "from:@estebanpdl"

[EssbieWGT]

I was able to get it working on Python 3.7.6 using the following:

pip install --upgrade git+https://github.com/yunusemrecatalcam/twint.git@twitter_legacy2

Was able to search for specific username like so:

c = twint.Config() c.Search = "from:@nbcnews" c.Store_object = True c.Limit = 100 twint.run.Search(c) tlist = c.search_tweet_list

Some things I've noticed so far...

• When searching a profile, Tweets do not seem to come back in any particular order (e.g., with NBC news first story is from five hours ago, and second is from earlier than that). Maybe the new "conversation" set up means it's returning things in the order of the most recent update to a conversation? Is unclear given what's returned only includes a date, instead of a full timestamp.
• It doesn't appear that I can search for a specific search string (e.g. ""hurricane sally"") without running into the same timeout/sleep error.

But, regardless... this is a big improvement over having NOTHING to work with!

[boshez]

I hope the developers find a solution to this. I don't know what I am going to do.

[Vickycats]

I am finding that it will not output to a file using c.Output (but one can dump the list output to a csv of course at the end instead), and it will not make use of c.Hide_output=False so as to display results on screen as it goes (it just ignores both parameters).

For me it will accept c.Search="(from:username)" to find tweets from that person and c.Search="@username" to locate mentions/replies.

For me it will accept c.Search="(from:username) max_id:tweet number" as well if one is attempting to use a date range but I have not tested that it is returning correctly using the max_id number (ie it returns tweets and does not freeze). It also does not freeze if one uses c.Since but again I have not checked whether Since is working as it should.

It sometimes returns error "shit: 7s list index out of range" which might be relevant?

[Bobpick]

In what context are you doing the c.Search, is this in a python file?

[estebanpdl]

I run pip install --user --upgrade git+https://github.com/yunusemrecatalcam/twint.git@twitter_legacy2
It removed the sleeping-loop, but it did not find the account.
twint -u estebanpdl --csv -o test.csv
Threw:
Use: twint -s "from:@estebanpdl"

Thanks, @Bobpick
It freezes using twint -s "from:@estebanpdl" --csv -o test.csv

[Bobpick]

I run pip install --user --upgrade git+https://github.com/yunusemrecatalcam/twint.git@twitter_legacy2
It removed the sleeping-loop, but it did not find the account.
twint -u estebanpdl --csv -o test.csv
Threw:
Use: twint -s "from:@estebanpdl"

Thanks, @Bobpick
It freezes using twint -s "from:@estebanpdl" --csv -o test.csv
It's what I'm running into as well.

[Vickycats]

In what context are you doing the c.Search, is this in a python file?

Yes, Its part of a complex program but the relevant part of my test code which seems to work is:

c=twint.Config()
c.Hide_output=False #but it ignores this
c.Store_object=True
c.Store_csv=True
c.Since="2020-09-01"
c.User_full=True
c.Limit=100 #I have tested up to 1000 ok
c.Output="testing.csv" #but it ignores this

srchname="(from:RealDonaldTrump)"
srch=srchname+" max_id:"+str(1307047063197224961)

twint.run.Search(c)
tlist = c.search_tweet_list
print(tlist)

[arceuss]

?

[dansar39]

With what script did I achieve that?

…

________________________________ De: arceuss <[email protected]> Enviado: miércoles, 30 de septiembre de 2020 0:03 Para: twintproject/twint <[email protected]> Cc: dansar39 <[email protected]>; Mention <[email protected]> Asunto: Re: [twintproject/twint] a new way to make searches (#917) ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#917 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/APYWXU6POH4U7SPUODZXNZDSIJYVPANCNFSM4RR7YMPQ>.

[pielco11]

Locking this conversation because it's becoming too much off-topic, in the last few comments about another repo are mentioned and issues about it are being reported.

I may decide to re-open this in a few days and hide (not deleting) comments not related to twint

[Abdullah955]

@Lulua-Ali i found a temp solution

change this line
tweet_dict['tweet'] = tweet.find("div", {"class": "tweet-text"}).find("div", {"class": "dir-ltr"}).text

to

tweet_dict['tweet'] = tweet.find("div", {"class": "tweet-text"}).find("div", {"class": "dir-rtl"}).text

but it will stop the English search