Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Occasional "b2.exception.DestFileNewer: source file is older than destination" error #46

Open
anthonyfok opened this issue May 18, 2021 · 0 comments
Assignees
Labels
Bug Something isn't working Severity: Minor Task

Comments

@anthonyfok
Copy link
Member

Will notified me of two GitHub Actions error as seen at https://github.com/OpenDRR/opendrr-data-store/actions, in particular:

It turns out that the Backblaze B2 tool errors out occasionally when it detected the source files being older than what is already on the B2 bucket:

  File "/usr/lib/python3/dist-packages/b2/sync/policy.py", line 99, in files_are_different
    raise DestFileNewer(
b2.exception.DestFileNewer: source file is older than destination: local://view_outputs/all_indicators/multi_hazard_threat/economic_region/nhsl_hazard_threat_all_indicators_b_1010.zip with a time of 1621298707917 cannot be synced to b2://view_outputs/all_indicators/multi_hazard_threat/economic_region/nhsl_hazard_threat_all_indicators_b_1010.zip with a time of 1621299046514, unless --skipNewer or --replaceNewer is provided
ERROR: source file is older than destination: local://view_outputs/all_indicators/multi_hazard_threat/economic_region/nhsl_hazard_threat_all_indicators_b_1010.zip with a time of 1621298707917 cannot be synced to b2://view_outputs/all_indicators/multi_hazard_threat/economic_region/nhsl_hazard_threat_all_indicators_b_1010.zip with a time of 1621299046514, unless --skipNewer or --replaceNewer is provided

Issues:

  • The "GeoPackage exports" GitHub workflow, which takes 4 hours or so to run, is being triggered on every single push and pull request, and that is probably too frequent.
  • With it being triggered on each push, and each instance taking 3 or 4 hours to run, when there are multiple pushes in close succession, more than one instance might be trying to uploading to B2 bucket at the same time.
  • The error avoids a race condition, so it is "good".
  • The error message can easily be overridden with --skipNewer or --replaceNewer, but there should be a better way to resolve this issue.
@anthonyfok anthonyfok added Bug Something isn't working Severity: Minor Task labels May 18, 2021
@anthonyfok anthonyfok self-assigned this May 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working Severity: Minor Task
Projects
None yet
Development

No branches or pull requests

1 participant