Backfill the last_synced_with_source field in the database

[obulat]

Current Situation

Currently, there are 554 237 images that set NULL as last_synced_with_source value.

Suggested Improvement

Set last_synced_with_source to the value of updated_on, if available, or to created_on in the database where it is currently NULL.

Benefit

Data consistency.

Additional context

Part of #244

Implementation

• 🙋 I would be interested in implementing this feature.

[krysal]

I believe this should be done naturally during the data ingestion/refresh process. @stacimc @AetherUnbound what do you think?

[stacimc]

That should be possible! With perhaps a bit of lag for dated DAGs, as it will require the reingestion process to cover old data over time. Reingestion flows are now supported but we need to hook up workflows for a few of our dated DAGs, and turn them on in production.

[AetherUnbound]

I think there may still be some records which will never receive a last_synced_with_source even if all dated DAGs have reingestion workflow and if all non-dated DAGs were certain to re-consume every record they've ever touched. IIRC we have some records from the common crawl dataset which did not come from a provider, and so I would expect those to remain NULL since we're not likely to touch them again.

[stacimc]

IIRC we have some records from the common crawl dataset which did not come from a provider

! I did not know this -- I had it in my head that the crawler only identified potential providers -- but of course it must be true. Very good point 😮