Dagster job ensure_job_triggered is failing due to invalid usage of > comparator

[aclum]

@sujaypatil96 reported that he submitted a changesheet but it hadn't been applied yet. When I check dagit i see that changesheets hasn't run since ~8 AM this morning and when looking at the activity tab it shows failures b/w 1-2pm for ensure_job_triggered

Please see if you can debug the ensure_job_triggered failures so we can get the changesheets applied. This is blocking workflows that will trigger once part_of is fixed (via the changesheet submission).
cc @Michal-Babins @shreddd

env:prod

[eecavanna]

On Dagit, I see the failing runs of the ensure_job_triggered job that is referenced in the issue description:

[eecavanna]

I clicked "View run" for Run ID 052af738 and saw the following stack trace:

Here's a copy/paste of it:

dagster._core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "maybe_post_jobs":
The above exception was caused by the following exception:
TypeError: '>' not supported between instances of 'datetime.datetime' and 'str'

Stack Trace:
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/utils.py", line 54, in op_execution_error_boundary
    yield
,  File "/usr/local/lib/python3.10/site-packages/dagster/_utils/__init__.py", line 461, in iterate_with_context
    next_output = next(iterator)
,  File "/opt/dagster/lib/nmdc_runtime/site/ops.py", line 409, in maybe_post_jobs
    if ts > candidate_job_object_id_timestamp:

[eecavanna]

The other failing run (Run ID 71cda0c6) has an identical stack trace.

[eecavanna]

Based upon that stack trace, I think the issue is:

• On line 409 of nmdc_runtime/site/ops.py, we are trying to use > to compare a string and a datetime, which is not allowed

nmdc-runtime/nmdc_runtime/site/ops.py

Line 409 in dc668ea

if ts > candidate_job_object_id_timestamp:

Given that the issue involves types, I suspect it stems from the recent update from Pydantic v1 to v2.

CC: @dwinston

[ssarrafan]

Assuming this won't be completed today so will move to the next sprint to complete. @dwinston @eecavanna @PeopleMakeCulture

[PeopleMakeCulture]

Related to: #339

[dwinston]

First will try to reproduce locally.

I see, from the landing page for the dagster job-run in dagit, i.e., https://dagit.microbiomedata.org/runs/052af738-7a04-4fe1-a305-5692455a313b, one of the run tags is dagster/run_key:(('apply-changesheet-1.0.0', 'gfs06r32'),). This means that the changesheet is available at https://api.microbiomedata.org/metadata/stored_files/gfs06r32. Downloaded that locally.

[dwinston]

I am able to reproduce. Submitting the above changesheet via POST /metadata/changesheets:submit, that is successful, but the spawned dagster job fails, with

dagster._core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "maybe_post_jobs":
The above exception was caused by the following exception:
TypeError: '>' not supported between instances of 'datetime.datetime' and 'str'

Stack Trace:
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/utils.py", line 54, in op_execution_error_boundary
    yield
,  File "/usr/local/lib/python3.10/site-packages/dagster/_utils/__init__.py", line 461, in iterate_with_context
    next_output = next(iterator)
,  File "/opt/dagster/lib/nmdc_runtime/site/ops.py", line 409, in maybe_post_jobs
    if ts > candidate_job_object_id_timestamp: