Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rewrite WMRC ETL updates #269

Closed
3 tasks done
nathankota opened this issue Nov 30, 2023 · 7 comments
Closed
3 tasks done

Rewrite WMRC ETL updates #269

nathankota opened this issue Nov 30, 2023 · 7 comments
Assignees
@jacobdadams

Comments

@nathankota
Copy link
Contributor

nathankota commented Nov 30, 2023 •
edited by jacobdadams
Loading

Benefit

Improve the ETL processes for the WMRC recycling map/app data from google sheets and salesforce, and to calculate the precision rates more efficiently than Arcade expressions in the app.

Acceptance Criteria

  • google sheet ETL (used oil collection data)
  • salesforce ETL (solid waste data)
  • precision rates calculations

Notes

Another possible task is to open a GitHub repo for the WMRC project overall, mostly to capture and track issues.

Risks

No response

Issue Reference

refs #221
@nathankota nathankota added the status: idea This issue is an idea label Nov 30, 2023
@nathankota nathankota changed the title Rewrite WMRC ETL update Rewrite WMRC ETL updates Nov 30, 2023
@jacobdadams jacobdadams mentioned this issue Dec 5, 2023
Closed
@jacobdadams jacobdadams mentioned this issue Dec 19, 2023
Closed
2 tasks
@jacobdadams jacobdadams removed the status: idea This issue is an idea label Dec 19, 2023
@nathankota nathankota mentioned this issue Jan 2, 2024
Closed
@jacobdadams
Copy link
Member

jacobdadams commented Jan 2, 2024 •
edited
Loading

Sprint 3

  • Get connected to DEQ salesforce sandbox (Radiation data are restricted, which constrains our access to the sandbox. Jake will ding Barry. )
  • Write Salesforce ETL
    • Create feature service for summary stats
    • Calc summary stats
  • Update skid w/Salesforce ETL

@jacobdadams
Copy link
Member

Still waiting on Barry Blackington & Tom to get access to their salesforce org

@jacobdadams jacobdadams added the status: blocked This issue or pull request has an unmet prerequisite label Jan 29, 2024
@steveoh
Copy link
Member

steveoh commented Jan 31, 2024

Still waiting on Barry Blackington & Tom to get access to their salesforce org

Do you know when the last reminder was given to Barry?

@nathankota
Copy link
Contributor Author

Sprint 4

  • ding Barry & Tom

@nathankota nathankota added the carry forward This task was carried forward into the next quarter label Mar 25, 2024
@nathankota
Copy link
Contributor Author

nathankota commented Apr 16, 2024 •
edited
Loading

Sprint 2

  • connect app
  • update palletjack to account for REST limitations
  • validate access to correct data

@nathankota
Copy link
Contributor Author

nathankota commented Apr 30, 2024 •
edited
Loading

Q4 Sprint 4

  • validate report/queries by comparing extracted data with Stevie's report

@nathankota nathankota removed status: blocked This issue or pull request has an unmet prerequisite carry forward This task was carried forward into the next quarter labels May 28, 2024
@nathankota
Copy link
Contributor Author

nathankota commented May 28, 2024 •
edited by jacobdadams
Loading

Sprint 5

  • Finalize ETL component
    • terraform, gcp function, etc.

From meeting with Stevie:

  • Make sure update is adding any new locations found on sheet
  • Add MRF facility category symbology
  • De-dupe facility ids
    • Try to use LastModifiedDate field
    • Report duplicates in status email
  • Add Rachel Boyer to email list

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jacobdadams jacobdadams

Labels
None yet
Projects
UGRC PI Planning Objectives
Status: Done
Milestone
No milestone
Development

No branches or pull requests
3 participants
@steveoh @nathankota @jacobdadams