Process fails without error message when executing merge

[cesar-vermeulen]

Environment

Delta-rs version:
0.16.1
Binding:
Python
Environment:

• Cloud provider: Azure
• OS: Docker image on kubernetes, running Debian GNU/Linux 11 (bullseye)

---

Bug

What happened:

During execution of following code, pod crashes without any error message:

  dt = DeltaTable(full_path)
  merge_definition = dt.merge(
      source=df,
      predicate=construct_merge_statement_predicates(primary_keys),
      source_alias="source",
      target_alias="target",
  )

  if write_method.merge_mode["update_when_matched"]: 
      merge_definition = merge_definition.when_matched_update_all()
  if write_method.merge_mode["insert_when_not_matched"]: 
      merge_definition = merge_definition.when_not_matched_insert_all()
  LOGGER.info("Starting merge")
  merge_execution = merge_definition.execute()
  LOGGER.info("Done merging")

Effectively, the code reaches the merge statement, but then crashes after only 10 seconds without passing any error messages (in python console).
However, this is a table that exists in 2 schemas in our source database, and gets converted to the same pandas schema. This code runs successfully on a daily basis for one table, but fails to be executed for the other, whilst having relatively the same amount of rows (source being ~1.5m rows, target ~20k rows. Both update when matched and insert when not matched are set to true for both scenarios.

It appears to me that the process crashes in the rust backend, but I have no clue how to further debug this.

What you expected to happen:
Successful merge execute statements
How to reproduce it:

/
More details:

• There are no concurrent writes
• First 6 merges (in this case) have succeeded
• Pandas dataframes are being used

[ion-elgreco]

@cesar-vermeulen are you not simply running OOM?

set this env variable, RUST_LOG=debug, and check the logs for something fishy