Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Duplicate syncs & schedules in DB #791

Closed
bastienbeurier opened this issue Jul 18, 2023 · 2 comments · Fixed by #793, #827 or #835
Closed

Duplicate syncs & schedules in DB #791

bastienbeurier opened this issue Jul 18, 2023 · 2 comments · Fixed by #793, #827 or #835
Assignees
@khaliqgant
Labels
bug Something isn't working

Comments

@bastienbeurier
Copy link
Member

bastienbeurier commented Jul 18, 2023 •
edited by khaliqgant
Loading

As discussed this morning with Khaliq, we noticed that the _nango_syncs table has duplicate nango_connection_id/name pairs, each having a corresponding _nang_sync_schedules row (that has RUNNING status).

We should investigate:

  • Why these duplicates are created
  • If that has impact on customers
  • How to prevent future duplicates and mitigate custom impact

Also with this issue dev creds should be encrypted
@bastienbeurier bastienbeurier added the bug Something isn't working label Jul 18, 2023
khaliqgant added a commit that referenced this issue Jul 18, 2023
@khaliqgant
[gh-#791] handle dupe syncs by forcing user to report the error
723cf70
@khaliqgant khaliqgant mentioned this issue Jul 18, 2023
Merged
khaliqgant added a commit that referenced this issue Jul 18, 2023
@khaliqgant
[gh-#791] on delete report it to the activity tab
660ac02
khaliqgant added a commit that referenced this issue Jul 18, 2023
@khaliqgant
[gh-#791] handle dupe syncs by forcing user to report the error (#793)
3e665b2 
* [gh-#791] handle dupe syncs by forcing user to report the error

* [gh-#791] on delete report it to the activity tab
khaliqgant added a commit that referenced this issue Jul 18, 2023
@khaliqgant
[gh-#791] await initiate and return null if sync exists
58c4aca
@khaliqgant
Copy link
Member

khaliqgant commented Jul 18, 2023 •
edited
Loading

Why these duplicates are created

If a pre existing connection id is created again and goes through the oauth flow the sync will be initiated again

If that has impact on customers

No major impact except that sync data is overwritten and their connection sync tab will have duplicates.

How to prevent future duplicates and mitigate customer impact

Prevent further duplicates with #796. We should write a script that removes dupes and deletes the sync and duplicate schedules and then add the migration that enforced uniqueness at the db level

@bastienbeurier

@khaliqgant khaliqgant reopened this Jul 18, 2023
khaliqgant added a commit that referenced this issue Jul 18, 2023
@khaliqgant
[gh-#791] await initiate and return null if sync exists (#796)
e5b5e12
@bastienbeurier
Copy link
Member Author

Thank you for getting to the bottom of this @khaliqgant. The script will also help with having precise BI!

@bastienbeurier bastienbeurier mentioned this issue Jul 19, 2023
Closed
6 tasks
khaliqgant added a commit that referenced this issue Jul 24, 2023
@khaliqgant
[gh-#791] deleted fixes
ade0fc7
khaliqgant added a commit that referenced this issue Jul 24, 2023
@khaliqgant
[gh-#791] update lookup, resolves #753
3a5db25
@khaliqgant khaliqgant mentioned this issue Jul 24, 2023
Merged
khaliqgant added a commit that referenced this issue Jul 24, 2023
@khaliqgant
[gh-#791] Merge branch 'gh-#792-bi-improvements' into gh-#791-db-scri…
a449b7e 
…pt-cleanups
khaliqgant added a commit that referenced this issue Jul 24, 2023
@khaliqgant
[gh-#791] set active to false as well
f072fb0
khaliqgant added a commit that referenced this issue Jul 25, 2023
@khaliqgant
[gh-#791] Merge branch 'gh-#792-bi-improvements' into gh-#791-db-scri…
0b0b953 
…pt-cleanups
khaliqgant added a commit that referenced this issue Jul 25, 2023
@khaliqgant
[gh-#791] Merge branch 'gh-#792-bi-improvements' into gh-#791-db-scri…
77cb161 
…pt-cleanups
khaliqgant added a commit that referenced this issue Jul 25, 2023
@khaliqgant
[gh-#791] bring in latest
42e8cc9
khaliqgant added a commit that referenced this issue Jul 25, 2023
@khaliqgant @bastienbeurier
[Github #791] db script cleanups (#827)
b54a344 
* [gh-#322] start api key auth and validation for the providers.yaml

* [gh-#322] add validation logic

* [gh-#322] later version and relative path

* [gh-#322] full path

* [gh-#322] better regex

* [gh-#322] add more props and set additionalProperties to true

* [gh-#322] valid json

* [gh-#322] oauth -> auth and start auth service work

* [gh-#322] formatting

* [gh-#322] api auth updates with type wrangling

* [gh-#322] add type for api and render fields in connection details

• Kick off sync if connection was upserted

* [gh-#322] add shortcut and update errors

* [gh-#322] consolidate methods

* [gh-#322] update get connection

* [gh-#322] proxy headers and query params + tests

* [gh-#322] Authorization param override

* [gh-#322] hmac support

* [gh-#322] base_api_url -> proxy.base_url

* [gh-#322] getToken sdk for api token

* [gh-#322] indentation

* [gh-#322] don't delete type

* [gh-#322] clean up proxy calls

* [gh-#322] info circle for notice on api auth

* [gh-#322] add logic for auth link and fix zustand

* [gh-#322] add credentials

* [gh-#322] semantic html

* [gh-#322] TODO remove after bumping verion

* [gh-#322] support extra headers

* [gh-#322] restore whitespace

* [gh-#322] base_api_url -> proxy.base_url

* [gh-#322] import connection for api and oauth

* [gh-#322] Resolves #804

* [gh-#322] better check for base_url, report error, and restart on providers.yaml change

* [gh-#322] make key prop dynamic for config.template.proxy.query

* [gh-#322] quote connection creds

* [gh-#322] single quote for consistency

* [gh-#322] consolidate

* [gh-#322] type declarations

* [gh-#322] cleanup

* [gh-#322] better comment

* [gh-#322] handle invalid auth mode

* [gh-#322] reorder

* [gh-#792] add migrations

Resolves #780

* [gh-#322] Docs for API Key

* [gh-#322] Remove type from import connection endpoint

* [gh-#792] soft deletes

* [gh-#322] check for credentials length

* [gh-#792] [gh-#322] check for credentials length

* [gh-#322] fix trailing

* [gh-#322] Remove path from base URL

* [gh-#792] use same keys as expected for the cli

* [gh-#792] change upsert method

* [gh-#792] fix merge markers

* [gh-#792] remove debug

* [gh-#792] remove cast

* [gh-#792] analytics and job tracking

* [gh-#792] deleted on joins

* [gh-#792] add last_fetch_at

* [gh-#791] deleted fixes

* [gh-#791] update lookup, resolves #753

* [gh-#792] fix delete logic

* [gh-#791] set active to false as well

* [gh-#792] filter on deleted false

* [gh-#792] and more deleted clauses

* [gh-#792] updateLastFetched in more instances

---------

Co-authored-by: Bastien Beurier <[email protected]>
khaliqgant added a commit that referenced this issue Jul 25, 2023
@khaliqgant
[gh-#791] remove job script and quick activity page improvements
19d4182
@khaliqgant khaliqgant mentioned this issue Jul 25, 2023
Merged
khaliqgant added a commit that referenced this issue Jul 25, 2023
@khaliqgant
[gh-#791] remove job script and quick activity page improvements (#835)
91a8177
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@khaliqgant khaliqgant

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
2 participants
@khaliqgant @bastienbeurier