Add tag table SQL migration scripts

[Arkatufus]

• Split scripts into steps
  • 1_Migration_Setup.sql : setup table and functions/procedures, if they didn't exist
  • 2_Migration.sql : the actual conversion process
  • 3_Post_Migration_Cleanup.sql : deletes functions/procedures
  • DROP_Migration.sql : deletes both table and functions/procedures
• Remove any transaction inside the script, scripts are idempotent
• All scripts are designed as idempotent UPSERT operation
• Special script with DROP prefix to restart

[Arkatufus]

Would these be good enough for migration purposes?

[Aaronontheweb]

Left some thoughts

[Aaronontheweb]

Just a general thought here - doing something destructive like dropping the TagTable... It might be better to fail the operation altogether and force the user to decide what to do in the event that they try to run this script twice. We can move the destructive / cleanup bits to a separate script that they can run explicitly that way. Buuuuuut for the sake of getting something into the repo that we can test now? I think this looks like it checks all the boxes.

[Aaronontheweb]

The other way we could go about making these scripts safe to run multiple times is rather than delete / recreate we can try the following idempotent pattern:

• create if not exists (for tables, functions)
• for each item to be added to the tag table, do an upsert (insert if not exist, update row if does exist)

That way running the script multiple times would be safe.

[Aaronontheweb]

Actually instead of doing an upsert, we could just skip the insert since the data is immutable -therefore no change.

[to11mtm]

IMO these should be grabbing batches of items rather than trying to do the whole table. Otherwise for large tables this could run for some unknown time before erroring out with a transaction size issue. (Also in general batching will make it easier to run 'on the fly' without locking too much, especially for something like MSSQL.)

[Aaronontheweb]

Forgive how rusty my SQL experience is here, but you're talking about doing pagination with cursors right?

[to11mtm]

Not cursors... the overall flow I was imagining (at least, for 0 downtime migration) was along the lines of

1. deploy everything in 'hybrid' mode (where we write both old style tags and new tags)
2. Once everything is deployed in that mode, run a script in the background, psuedo-sql:

declare @moveUpTo = Select max(ordering) from journal_table 
where (tags is not null and strlen(tags)>0)
and ordering not in (select ordering from journal_tag_table);

declare @startAtOrdering = 0;

while (@startAtOrdering < @moveUpTo)
begin
     insert into journal tag table (select tags from journal_table where orderingId >= @startAtOrdering && orderingId < @startAtOrdering+1000); --Obviously need more transform here
     @startAtOrdering = @startAtOrdering+1000;
end

3. Re-deploy in 'tag-table-only' mode

4. (optional) run a clean-up script to get rid of the old tags in the rows. but TBH only if you really need the storage back, since in cloud the storage space gained back may or may not be worth the RUs of re-writing everything and indexes getting kershuffled.
   you might need to think about how you do the transactions here... but hopefully gets the point across.

[Aaronontheweb]

deploy everything in 'hybrid' mode (where we write both old style tags and new tags)

That makes sense and I think this can be done easily since we're not getting rid of the old tag column - does that make sense to you @Arkatufus ?

[Aaronontheweb]

Once everything is deployed in that mode, run a script in the background, psuedo-sql:

I see what you mean: breaking up the inserts into transactions with groups of N "tag rows" at a time. I think that should be doable.

[Arkatufus]

Last changes:

• Split scripts into steps
• Drop any transactions inside the script
• All scripts are designed as idempotent UPSERT operation
• Special script with DROP prefix to restart

[Aaronontheweb]

LGTM for now but need to implement @to11mtm 's feedback on subsequent PRs