Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regression: Patreon metadata no longer downloads if there are no files #1541

Closed
GiovanH opened this issue May 10, 2021 · 2 comments
Closed
Labels

Comments

@GiovanH
Copy link
Contributor

GiovanH commented May 10, 2021

Using configuration

{
  "extractor": {
    "base-directory": "./meta/",
    "event": "post",
    "postprocessors": [{
      "name": "metadata",
      "mode": "json"
    }]
  }
}

(with the event "post" as instructed) patreon posts that only have text content are skipped entirely as of the latest git revision.

See also #866 (comment)

Program flow goes through _pagination and _process for each post, including the skipped posts, and _process yields the correct attribute dictionary, so this doesn't seem like a problem with the patreon extractor specifically.

Edit 2: With the given configuration the "post" hook callback in job.py doesn't even seem to be firing at all? It may be that "event": "post" is actually broken. For text-only posts (and only text-only posts) [patreon][debug] Active postprocessor modules: [MetadataPP] doesn't even fire.

@GiovanH
Copy link
Contributor Author

GiovanH commented May 10, 2021

After... extensive testing, it seems the configuration must be

{
  "extractor": {
    "base-directory": "./meta/",
    "skip": false,
    "postprocessors": [{
      "name": "metadata",
      "event": "post",
      "mode": "json",
      "filename": "{id}_{title}_{num:>02}.json"
    }]
  }
}

The mode must be specified within the specific extractor (inheritance doesn't work) and you have to supply an explicit filename template or else it uses metadata.json and overwrites itself.

This is... probably not desired.

@mikf
Copy link
Owner

mikf commented Jul 15, 2021

(with the event "post" as instructed) patreon posts that only have text content are skipped entirely as of the latest git revision.

"event": "post" must be specified inside the post processor settings, otherwise it uses "event": "file" by default. The comment you linked has all 3 required settings (name, event, filename) in the same post processor block and it works as promised

The mode must be specified within the specific extractor (inheritance doesn't work)

Yeah, but "mode": "json" is the default and doesn't need to be specified explicitly

you have to supply an explicit filename template

There is no default template for post files yet

@mikf mikf closed this as completed Jul 15, 2021
@mikf mikf added the invalid label Jul 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants