Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(api): fix CSV feature upload [MRXN23-249, MRXN23-246] #1391

Conversation

yulia-bel
Copy link
Contributor

CSV feature upload: insert temporary data in bulk and in chunks, change project sources

Overview

This PR adds small fixes for Id0 feature upload via CSV file:

  1. Saving csv data in temporary storage in (apiDb).features_amounts is fixed to be done in bulk, divided in chunks
  2. After successful insert of all feature data in relevant tables, project sources is changed to 'legacy_import' for correct amount usage during runs
  3. Small test update to check projects.sources change

Designs

Link to the related design prototypes (if applicable)

Testing instructions

Please explain how to test the PR: ID of a dataset, steps to reach the feature, etc.

Feature relevant tickets

Link to the related task manager tickets


Checklist before submitting

  • Meaningful commits and code rebased on develop.
  • If this PR adds feature that should be tested for regressions when
    deploying to staging/production, please add brief testing instructions
    to the deploy checklist (docs/deployment-checklist.md)
  • Update CHANGELOG file

@vercel
Copy link

vercel bot commented Jul 24, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
marxan ✅ Ready (Inspect) Visit Preview 💬 Add feedback Jul 24, 2023 4:08pm

Copy link
Member

@hotzevzl hotzevzl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @yulia-bel.

I suggest to merge this as-is. As we've been discussing, we know we'll need to make this whole workflow work much faster, but that's a separate concern to me - I wouldn't cram more stuff into this PR.

I've created https://vizzuality.atlassian.net/browse/MRXN23-256 to track profiling and performance work on this.

@yulia-bel yulia-bel merged commit d6c419b into develop Jul 26, 2023
@yulia-bel yulia-bel deleted the fix/api/MRXN23-249-chunk-and-fast-insert-for-features-csv-upload branch July 26, 2023 16:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants