Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KeyError: 'url' #37

Closed
mswsn opened this issue Oct 12, 2022 · 14 comments
Closed

KeyError: 'url' #37

mswsn opened this issue Oct 12, 2022 · 14 comments
Assignees
Labels
in progress Currently being worked on

Comments

@mswsn
Copy link

mswsn commented Oct 12, 2022

I'm occasionally getting crashes while using beets on an rpi4 and the culprit seems to be the beetcamp plugin.

Sometimes the hangup seems to be on particular albums and I'm forced to disable the plugin until that one is cleared. I've also noticed that the error always tends to occur if I have left beets alone for a few minutes.

Traceback (most recent call last):
  File "/home/mswsn/.local/bin/beet", line 8, in <module>
    sys.exit(main())
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/ui/__init__.py", line 1285, in main
    _raw_main(args)
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/ui/__init__.py", line 1272, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/ui/commands.py", line 973, in import_func
    import_files(lib, paths, query)
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/ui/commands.py", line 943, in import_files
    session.run()
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/importer.py", line 340, in run
    pl.run_parallel(QUEUE_SIZE)
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/util/pipeline.py", line 446, in run_parallel
    raise exc_info[1].with_traceback(exc_info[2])
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/util/pipeline.py", line 311, in run
    out = self.coro.send(msg)
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/util/pipeline.py", line 193, in coro
    func(*(args + (task,)))
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/importer.py", line 1376, in lookup_candidates
    task.lookup_candidates()
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/importer.py", line 660, in lookup_candidates
    autotag.tag_album(self.items, search_ids=self.search_ids)
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/autotag/match.py", line 461, in tag_album
    for matched_candidate in hooks.album_candidates(items,
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/plugins.py", line 573, in decorated
    for v in generator(*args, **kwargs):
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/autotag/hooks.py", line 629, in album_candidates
    yield from plugins.candidates(items, artist, album, va_likely,
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/plugins.py", line 384, in candidates
    yield from plugin.candidates(items, artist, album, va_likely,
  File "/home/mswsn/.local/lib/python3.10/site-packages/beetsplug/bandcamp/__init__.py", line 214, in candidates
    for res in filter(truth, map(self.get_album_info, results)):
KeyError: 'url'

Here's what I'm getting when using the -v flag:

Sending event: import_task_created
Sending event: import_task_start
Looking up: /styx/music/swinsian/Hear & Now/Aurora Baleare
Tagging Hear & Now - Aurora Baleare
No album ID found.
Search terms: Hear & Now - Aurora Baleare
Album might be VA: False
Searching for MusicBrainz releases with: {'release': 'aurora baleare', 'artist': 'hear & now', 'tracks': '8'}
Requesting MusicBrainz release df62a53e-6071-49c7-8870-95dca450f142
Sending event: import_task_created
Sending event: import_task_created
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: Hear & Now - Aurora Baleare (df62a53e-6071-49c7-8870-95dca450f142)
Computing track assignment...
...done.
Success. Distance: 0.02
Requesting MusicBrainz release 366b692f-854c-4b8a-9778-7f4fd862e8ad
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: Hear & Now - Milvus (366b692f-854c-4b8a-9778-7f4fd862e8ad)
Computing track assignment...
...done.
Success. Distance: 0.54
Requesting MusicBrainz release f115fd29-f4c9-43e5-8447-77d62ebd539d
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: Hear - 5 Pieces (f115fd29-f4c9-43e5-8447-77d62ebd539d)
Computing track assignment...
...done.
Success. Distance: 0.68
Requesting MusicBrainz release f81b3578-7193-42fa-b220-8d70cd4031e7
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: KMLN & Roderic - Aurora (f81b3578-7193-42fa-b220-8d70cd4031e7)
Computing track assignment...
...done.
Success. Distance: 0.62
Requesting MusicBrainz release 5864ea59-d21f-4522-8068-10ed48c89d29
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: Vintage & Morelli - Aurora (5864ea59-d21f-4522-8068-10ed48c89d29)
Computing track assignment...
...done.
Success. Distance: 0.64
bandcamp: Searching as for Aurora Baleare using {'query': 'Aurora Baleare', 'artist': 'Hear & Now', 'label': '', 'search_type': 'a'}
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: albuminfo_received
Candidate: Hear & Now - Aurora Baleare (https://claremont56.bandcamp.com/album/aurora-baleare)
Computing track assignment...
...done.
Success. Distance: 0.17
Sending event: albuminfo_received
Candidate: Hear & Now - Aurora Baleare (https://claremont56.bandcamp.com/album/aurora-baleare#p2376566435)
Computing track assignment...
...done.
Success. Distance: 0.18
Sending event: import_task_created
Traceback (most recent call last):
  File "/home/mswsn/.local/bin/beet", line 8, in <module>
    sys.exit(main())
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/ui/__init__.py", line 1285, in main
    _raw_main(args)
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/ui/__init__.py", line 1272, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/ui/commands.py", line 973, in import_func
    import_files(lib, paths, query)
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/ui/commands.py", line 943, in import_files
    session.run()
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/importer.py", line 340, in run
    pl.run_parallel(QUEUE_SIZE)
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/util/pipeline.py", line 446, in run_parallel
    raise exc_info[1].with_traceback(exc_info[2])
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/util/pipeline.py", line 311, in run
    out = self.coro.send(msg)
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/util/pipeline.py", line 193, in coro
    func(*(args + (task,)))
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/importer.py", line 1376, in lookup_candidates
    task.lookup_candidates()
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/importer.py", line 660, in lookup_candidates
    autotag.tag_album(self.items, search_ids=self.search_ids)
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/autotag/match.py", line 461, in tag_album
    for matched_candidate in hooks.album_candidates(items,
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/plugins.py", line 573, in decorated
    for v in generator(*args, **kwargs):
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/autotag/hooks.py", line 629, in album_candidates
    yield from plugins.candidates(items, artist, album, va_likely,
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/plugins.py", line 384, in candidates
    yield from plugin.candidates(items, artist, album, va_likely,
  File "/home/mswsn/.local/lib/python3.10/site-packages/beetsplug/bandcamp/__init__.py", line 214, in candidates
    for res in filter(truth, map(self.get_album_info, results)):
KeyError: 'url'
@tomauty
Copy link

tomauty commented Dec 5, 2022

Following because I get this too. Can't find a pattern as to when. Might look into it codewise later

@tomauty
Copy link

tomauty commented Dec 5, 2022

Looks like it crashes when it encounters a recaptcha URL. Which explains why this happens more often the more I keep importing. This was a crashing results object from around line 214

results ['https://greenovamusic.bandcamp.com/album/back-ellin-crack', 'https://www.recaptcha.net/recaptcha/enterprise.js']

@tomauty
Copy link

tomauty commented Dec 5, 2022

@mwassen I was able to temporarily make the crashes stop by setting search_max to 1. Not great as it limits to one possible result, but recaptchas get filtered out.

@mswsn
Copy link
Author

mswsn commented Dec 15, 2022

Cheers, at least good to have a band-aid for now.

@snejus
Copy link
Owner

snejus commented Dec 17, 2022

Sorry this took ages to respond to, currently looking into it!

@snejus snejus added the in progress Currently being worked on label Dec 17, 2022
@snejus snejus self-assigned this Dec 17, 2022
@snejus
Copy link
Owner

snejus commented Dec 17, 2022

@tomauty you are completely right regarding the recaptcha url! See below for a confirmation of your issue (see results number 7 and 18)

image

@snejus
Copy link
Owner

snejus commented Dec 17, 2022

Have fixed it, see the equivalent below

image

Will get it released in a moment.

@snejus
Copy link
Owner

snejus commented Dec 17, 2022

Done, try updating beetcamp to 0.16.1!

@snejus snejus closed this as completed Dec 17, 2022
@marty-oehme
Copy link

I am still constantly getting a url error with beetcamp, have updated to 0.16.1 as far as I know with a direct git clone to the beets environment (is there a cli way of getting the version?)

I am not sure if it is the same issue, due to recaptchas. However, the traceback looks basically identical:

beetcamp stack traceback
Traceback (most recent call last):
  File "/home/marty/.local/bin/beet", line 8, in <module>
    sys.exit(main())
  File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/ui/__init__.py", line 1285, in main
    _raw_main(args)
  File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/ui/__init__.py", line 1272, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/ui/commands.py", line 973, in import_func
    import_files(lib, paths, query)
  File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/ui/commands.py", line 943, in import_files
    session.run()
  File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/importer.py", line 340, in run
    pl.run_parallel(QUEUE_SIZE)
  File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/util/pipeline.py", line 446, in run_parallel
    raise exc_info[1].with_traceback(exc_info[2])
  File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/util/pipeline.py", line 311, in run
    out = self.coro.send(msg)
  File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/util/pipeline.py", line 193, in coro
    func(*(args + (task,)))
  File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/importer.py", line 1376, in lookup_candidates
    task.lookup_candidates()
  File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/importer.py", line 660, in lookup_candidates
    autotag.tag_album(self.items, search_ids=self.search_ids)
  File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/autotag/match.py", line 461, in tag_album
    for matched_candidate in hooks.album_candidates(items,
  File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/plugins.py", line 573, in decorated
    for v in generator(*args, **kwargs):
  File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/autotag/hooks.py", line 629, in album_candidates
    yield from plugins.candidates(items, artist, album, va_likely,
  File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/plugins.py", line 384, in candidates
    yield from plugin.candidates(items, artist, album, va_likely,
  File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beetsplug/bandcamp/__init__.py", line 215, in candidates
    for res in filter(truth, map(self.get_album_info, results)):
KeyError: 'url'

The error happens consistently on this link: https://derleemusic.com/album/off-the-beatn-path-2, while other albums by the same artist worked.

beets verbose output
user configuration: /home/marty/.config/beets/config.yaml
data directory: /home/marty/.config/beets
plugin paths:
lastgenre: Loading canonicalization tree ~/.config/beets/lastgenre_canonicallist.yaml
inline: adding item field multidisc
inline: adding item field artist_differs
Sending event: pluginload
library database: /home/marty/.local/share/beets/library.db
library directory: /home/marty/media/audio/music
Sending event: library_opened
Sending event: import_begin
Sending event: import_task_created
Sending event: import_task_start
Looking up: /home/marty/inbox/music/Derlee/Off The Beat'n Path
Tagging Derlee - Off The Beat'n Path
Searching for discovered album ID: 11684198
Requesting MusicBrainz release 11684198
Invalid MBID (11684198).
Search terms: Derlee - Off The Beat'n Path
Additional search terms: {'year': 2015, 'label': 'Philos Records', 'catalognum': '', 'country': '', 'media': ''}
Album might be VA: False
Searching for MusicBrainz releases with: {'release': "off the beat'n path", 'artist': 'derlee', 'tracks': '16', 'date': '2015', 'label': 'philos records'}
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Requesting MusicBrainz release fbfcd09e-8a2b-4599-ae86-e1389f7bf49a
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: Dave Koz - Off the Beaten Path (fbfcd09e-8a2b-4599-ae86-e1389f7bf49a)
Computing track assignment...
...done.
Success. Distance: 0.77
Requesting MusicBrainz release 0824695f-456e-40fe-8c8e-4295c91ff197
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: The New Breed - Off the Beaten Path (0824695f-456e-40fe-8c8e-4295c91ff197)
Computing track assignment...
...done.
Success. Distance: 0.71
Requesting MusicBrainz release 7d66b04e-386c-4ff4-9659-785c9c29873a
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: Justin Moore - Off the Beaten Path (7d66b04e-386c-4ff4-9659-785c9c29873a)
Computing track assignment...
...done.
Success. Distance: 0.73
Requesting MusicBrainz release 126ab765-595a-409b-8b86-94a45b5d5087
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: Phonique - Beat'n Greet (126ab765-595a-409b-8b86-94a45b5d5087)
Computing track assignment...
...done.
Success. Distance: 0.86
Requesting MusicBrainz release 4601d64b-bd38-47a7-bddf-04e1cbba10de
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: Justin Moore - Off the Beaten Path (4601d64b-bd38-47a7-bddf-04e1cbba10de)
Computing track assignment...
...done.
Success. Distance: 0.73
deezer: Searching Deezer for 'album:"Off The Beat'n Path" artist:"Derlee"'
deezer: Found 1 result(s) from Deezer for 'album:"Off The Beat'n Path" artist:"Derlee"'
deezer: Searching Deezer for album '11684198'
Sending event: albuminfo_received
Candidate: Derlee - Off The Beat'n Path (11684198)
Computing track assignment...
...done.
Success. Distance: 0.33
bandcamp: Searching as for Off The Beat'n Path using {'query': "Off The Beat'n Path", 'artist': 'Derlee', 'label': 'Philos Records', 'search_type': 'a'}

I tried to look around a tiny bit myself but did not get far with any coherent reasoning unfortunately.

@shagr4th
Copy link

shagr4th commented Dec 23, 2022

Same error here. It's actually due to an artist's use of another domain name : For instance if you search for "Colors in the dark", the first bandcamp result will point to https://3six.net/album/colours-in-the-dark, which is not on bandcamp.com, same as your album link (but use the same backend). Then the regex defined in search.py is not matched, and the url empty, leading to this error...

@snejus imho, I would just change this regex to a simpler one with just 'http' and some query like '?from=search', given that all search results are likely to be bandcamp albums...

@snejus
Copy link
Owner

snejus commented Dec 26, 2022

Okay, seeing this is re-appearing I am now testing the search against all album names I have in my beets database (1500+).

I can see that the first 300 searches gave me 80 results with a wrong URL (missing or some irrelevant HTML), each of which would cause the issue you above. This gives more than enough data to work with.

@shagr4th thank you for your input - ?from=search is a great idea - see my comments under the PR. Aiming to get the fix out tonight.

@tomauty
Copy link

tomauty commented Dec 27, 2022

@snejus Sorry to be off track, but what's the table command you're piping to to show the response data?

@snejus
Copy link
Owner

snejus commented Dec 27, 2022

@tomauty, it's a JSON prettifier based on rich. Since it's now in a somewhat of a more shareable state, see rich-tables. Early version is on pypi, so pip install rich-tables should give you the executable. There aren't any docs or usage so far, but see the tests/json folder in the repo and README to see what they give. Then pipe in some data and see what you get 😉

@snejus
Copy link
Owner

snejus commented Dec 28, 2022

This should now be fully fixed in 0.16.2!

@snejus snejus closed this as completed Dec 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
in progress Currently being worked on
Projects
None yet
Development

No branches or pull requests

5 participants