KernelCI API and KCIDB

[gctucker]

tl;dr KernelCI has two APIs and we should have only one

KCIDB provides a way to collate data from multiple independent CI systems and provide a single view of the results.

The KernelCI API includes all the tests run directly by the KernelCI pipeline and can also accept data sent by other users and CI systems.

Plans for the new KernelCI web dashboard and User Experience in general are based on the idea that it should encompass the whole dataset, so realistically we would need to have only one database and one backend API to access it.

Most users are still confused about the difference between the KernelCI API (legacy or new one) and KCIDB. Having a single way of submitting and accessing the KernelCI data would remove this issue.

This discussion is about determining what the best way forward might be for the KernelCI project itself as well as the ecosystem it is part of to address the issues mentioned above.

[gctucker]

Option 1: Intermediate database and client-side bridges

The closest solution to what we currently have would be to keep KCIDB and the KernelCI API as independent entities and have a client-side bridge to forward API results to KCIDB. Then we could have an intermediate database on top of KCIDB/BigQuery tailored for the new web dashboard, similarly to what is currently being done for the Grafana dashboard.

Pros

• no major changes required in KCIDB or KernelCI API
• systems stay decoupled
• intermediate database can be fine-tuned for the web dashboard

Cons

• does not solve the issue of having 2 APIs, in fact this adds yet another one
• extra costs and inefficient data duplication
• only KCIDB schema available, any additional data from API would be left out

[gctucker]

Option 2: KCIDB data managed by KernelCI API

In this scenario, a translation layer is added on top of the API to make it compatible with KCIDB so users can keep submitting and reading data exactly in the same way. However the data is sent directly to the API and stored in its database (MongoDB). Then the web dashboard and emails can be based on it and directly have all the data available in one place. The same data might still be synced periodically in BigQuery for archiving purposes to only keep a rolling time window in MongoDB and to keep full backwards compatibility with the current KCIDB implementation.

Pros

• No unnecessary duplication of data, lower maintenance costs
• Effectively one single API and central KernelCI database, just different tools to access it
• Continuity with current KCIDB

Cons

• Some rework of the KCIDB tools to go via the translation layer
• Increased coupling between API schema and KCIDB
• Impact on KCIDB branding and overall communication

[gctucker]

We had a meeting last week with @spbnick and a few others to discuss this. The outcome was that Option 2 appears as the ideal scenario, in the meantime we need to find a way to make it work.

The first step is to create a hook in KCIDB to forward all incoming data to the KernelCI API as a way to start evaluating how this would work in practice. We can gather metrics in terms of volume, frequency etc. to see if this can scale.

In parallel, we need to compare the current KCIDB data schema with the API ones and make any adjustments if necessary. The Early Access phase is perfect of this kind of experiment as we haven't declared a "Version 1" of the API yet so it's easier to make changes.