Pydantic AttributeError (on Dagit) after using /metadata/json:submit endpoint

[brynnz22]

When trying to submit a json using the json:submit endpoint, the job failed. The error looks like:

@aclum

[eecavanna]

I found the failing run on Dagit and copy/pasted the full error message below for our future reference:

Full stack trace:

dagster._core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "get_json_in":

  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/execute_plan.py", line 275, in dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/execute_step.py", line 476, in core_dagster_event_sequence_for_step
    for user_event in _step_output_error_checked_user_event_sequence(
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/execute_step.py", line 159, in _step_output_error_checked_user_event_sequence
    for user_event in user_event_sequence:
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/execute_step.py", line 94, in _process_asset_results_to_events
    for user_event in user_event_sequence:
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/compute.py", line 204, in execute_core_compute
    for step_output in _yield_compute_results(step_context, inputs, compute_fn):
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/compute.py", line 173, in _yield_compute_results
    for event in iterate_with_context(
  File "/usr/local/lib/python3.10/site-packages/dagster/_utils/__init__.py", line 459, in iterate_with_context
    with context_fn():
  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/utils.py", line 84, in op_execution_error_boundary
    raise error_cls(

The above exception was caused by the following exception:
AttributeError: 'pydantic_core._pydantic_core.Url' object has no attribute 'startswith'

  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/utils.py", line 54, in op_execution_error_boundary
    yield
  File "/usr/local/lib/python3.10/site-packages/dagster/_utils/__init__.py", line 461, in iterate_with_context
    next_output = next(iterator)
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/compute_generator.py", line 131, in _coerce_op_compute_fn_to_iterator
    result = invoke_compute_fn(
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/compute_generator.py", line 125, in invoke_compute_fn
    return fn(context, **args_to_pass) if context_arg_provided else fn(**args_to_pass)
  File "/opt/dagster/lib/nmdc_runtime/site/ops.py", line 478, in get_json_in
    rv = client.get_object_bytes(object_id)
  File "/opt/dagster/lib/nmdc_runtime/site/resources.py", line 197, in get_object_bytes
    if access.url.startswith(

Looks to me like the get_object_bytes function in the nmdc_runtime/site/resources.py module is trying to call a method on an object that is a pydantic_core._pydantic_core.Url as though that object were instead a str (string).

I think this is a result of the recent update from Pydantic v1 to Pydantic v2.

[eecavanna]

After the fix for this issue was introduced, re-submitting the same JSON file to the same endpoint resulted in a Dagster job failing for a new reason, which is described in issue #378