At the moment, processing of uploaded BOMs is happening entirely in-memory.

BomUploadProcessingTasks are enqueued to the internal task queue (see https://github.com/DependencyTrack/hyades/blob/main/WTF.md#why), and processed by the EventService thread pool.

The current design has some downsides:

1. When the API server crashes or is stopped, queued tasks are lost. Users have to re-upload their BOM again after the API server was restarted, which may or may not be practical, depending on the users' workflows.
2. Processing can not be shared among multiple instances of the API server.
3. We do not store the original BOM, because it is not practical to store many large documents in a RDBMS.
4. If BOMs are re-uploaded for the same project in close succession, we'll run into race conditions, as BOM ingestion is not executed in a single, large DB transaction.

The proposed enhancement involves storage of uploaded BOMs in a Koala-compatible system (e.g. the CycloneDX BOM Repository Server), and publishing "BOM uploaded" events to Kafka. Consumers (API server or specialized workers) consume from the Kafka topic, and perform the actual ingestion into the database.

• Processing could then happen in a distributed fashion, by one or more API server instances (or even specialized workers).
• Uploaded BOMs would not be lost in case an API server instance is stopped or crashes.
• Original BOMs would be retained, and could be referenced by the DT UI.
  • See BOM Retention Policy dependency-track#877
• Kafka's consumer model would ensure that BOM uploads for the same project are processed serially.
• Would allow for policies on original BOMs.
  • See Policy: Add support for BOM in policy conditions dependency-track#773
• If Koala was able to accept SPDX, but return CycloneDX, we could implement official support for SPDX ingestion.
  • See Add support for SPDX v3 dependency-track#1746
• Aids in achieving Achieve horizontal scalability of API server #375.

sequenceDiagram
    Client->>+API Server: Upload BOM
    API Server->>Koala: Upload BOM
    Koala->>Koala: Validate BOM
    Koala->>API Server: Location of BOM in Koala (URL)
    API Server->>API Server: Generate and persist correlation ID<br/>identifying the upload
    API Server->>Kafka: Publish event to "BOM uploaded" topic
    Note over API Server, Kafka: Key=Project UUID<br/>Value=Koala URL<br/>Header=Correlation ID
    API Server->>Client: Report correlation ID
    loop continuously
        API Server->>Kafka: Consume from "BOM uploaded" topic
        loop for each event
            API Server->>Koala: Fetch BOM
            API Server->>API Server: Process BOM
            alt processing failed
                API Server->>API Server: Update status of upload in DB to "failed"
                API Server->>Kafka: Publish event to "BOM Processing failed" topic
            else processing succeeded
                API Server->>API Server: Update status of upload in DB to "successful"
                API Server->>Kafka: Publish event to "BOM Processed" topic
                API Server->>API Server: Trigger vuln analysis etc.
            end
        end
    end

Warning
Because BOM processing can take up to multiple minutes for huge BOMs, it is not viable to perform processing in Kafka Streams, where short processing times are mandatory (see #529). We either need to offload processing to a separate thread pool, or write some custom logic around the low-level Kafka consumer.

We need to look into proper AuthN / AuthZ for the Koala service. The BOM repo server does not have those built-in.

Focusing on the client-side a little more, existing workflows should still continue to work:

sequenceDiagram
    Client->>+API Server: Upload BOM
    API Server->>Client: Report correlation ID
    loop continuously
        Client->>API Server: Is BOM still being processed?<br/>(Using correlation ID)
        alt processing ongoing
            API Server->>Client: "true"
        else processing completed
            API Server->>Client: "false"
            Client->>Client: Stop polling
        end
    end
    Client->>API Server: Fetch findings
    Client->>API Server: Fetch policy violations