After ingesting around 100M docs, I started to get such errors:

WARN merge{merge_split_id=01JCZ8FMKQ06XP94SYK95TB9E8 split_ids=["01JCY8PPA9CQ6QGCK29J4RCSG5", "01JCY998YBNJKP9PF9HMG6487S", "01JCY4SFWYDDYBCVRD2ZPVJ026"] typ=Merge}:uploader: quickwit_indexing::actors::uploader: Failed to upload split. Killing! cause=failed uploading key 01JCZ8FMKQ06XP94SYK95TB9E8.split in bucket azure://quickwit/indexes/

Of which the root cause seems to be:

2024-11-18T10:27:37.793408+01:00 merklemap quickwit[876138]: 0: storage error(kind=Service, source=Azure error wrapper(inner=server returned error status which will not be retried: 413))

My first guess is that quickwit is trying to upload parts that are too large? https://learn.microsoft.com/en-us/troubleshoot/azure/azure-storage/blobs/connectivity/request-body-large

Expected behavior
Using azure blob storage should not fail.

Configuration:

1. Quickwit 0.8.2 (x86_64-unknown-linux-gnu 2024-06-17T16:36:47Z 42766b8)

version: 0.8

index_id: data

doc_mapping:
  field_mappings:
    - name: id
      type: u64
      fast: true
      stored: true
    - name: raw_data
      type: bytes
      stored: true
      indexed: false

indexing_settings:
  commit_timeout_secs: 60