Optimize ROC AUC Computation

[czaloom]

Feature Description

ROC AUC computation experienced significant slowdowns in resource-constrained settings as a large query was spawned for each unique label.

The solution was to aggregate the ROC AUC computation into a single query.

Local Performance

before (v0.29.0)

    "number_of_datums": 5000,
    "number_of_unique_labels": 50,
    "number_of_annotations": 10000,
    "ingest_runtime": "17.0 seconds",
    "base_eval_runtime": "8.0 seconds",
    "base+pr_eval_runtime": "14.3 seconds",
    "base+pr+detailed_eval_runtime": "14.2 seconds",
    "del_runtime": "0.3 seconds"

after

        "number_of_datums": 5000,
        "number_of_unique_labels": 50,
        "number_of_annotations": 10000,
        "ingest_runtime": "17.1 seconds",
        "eval_runtime": "2.1 seconds",
        "eval_pr_runtime": "8.7 seconds",
        "eval_pr_detail_runtime": "9.2 seconds",
        "del_runtime": "0.7 seconds"

Resource Constrained Performance (dev01)

before (v0.29.0)

{
    "number_of_datums": 5000,
    "number_of_unique_labels": 50,
    "number_of_annotations": 10000,
    "base_eval_runtime": "201.3 seconds",
    "base+pr_eval_runtime": "226.3 seconds",
    "base+pr+detailed_eval_runtime": "228.5 seconds",
}

after

{
    "number_of_datums": 5000,
    "number_of_unique_labels": 50,
    "number_of_annotations": 10000,
    "eval_base_runtime": "85.1 seconds",
    "eval_pr_runtime": "111.5 seconds",
    "eval_pr_detailed_runtime": "110.5 seconds",
}

[ntlind]

nit: could this error message be made more specific?

[czaloom]

Updated to ROC AUC computation failed as the label '{label}' could not be found.