Skip to content

Commit

Permalink
Challenge documentation transferred to repo
Browse files Browse the repository at this point in the history
  • Loading branch information
@dreavjr
dreavjr committed Jun 6, 2024
1 parent 0e3e8b6 commit 441b7b2
Showing 1 changed file with 14 additions and 241 deletions.
255 changes: 14 additions & 241 deletions index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -554,250 +554,17 @@ <h2>Important Dates</h2>

<div id="challenge" class="container-md section-container">
<h2>BRAVO Challenge</h2>
<p>In conjunction with the BRAVO workshop at ICCV'23, we are organizing a challenge on the robustness of
autonomous driving in the open world. The 2023 BRAVO Challenge aims at benchmarking segmentation models
<p>In conjunction with the <a href="https://uncertainty-cv.github.io/2024/">Workshop on Uncertainty
Quantification for Computer Vision</a>, we are organizing a challenge on the robustness of
autonomous driving in the open world. The 2024 BRAVO Challenge aims at benchmarking segmentation models
on urban scenes undergoing diverse forms of natural degradation and realistic-looking synthetic
corruptions. We offer two tracks for benchmarking segmentation models: (1) trained on a single dataset
and (2) trained on multiple heterogeneous datasets.</p>
corruptions.</p>

<h3>General rules</h3>
<ol>
<li>Models in each track must be trained using only the datasets allowed for that track.</li>
<li>It is strictly forbidden to employ generative models for synthetic data augmentation.</li>
<li>All results must be reproducible. Participants must submit a white paper containing comprehensive
technical details alongside their results. Participants must make models and inference code
accessible.</li>
</ol>

<h3>Track 1 – Single-domain training</h3>
<p>In this track, models must be trained exclusively on the published
<a href="https://www.cityscapes-dataset.com/" target="_blank">Cityscapes dataset</a>. This track
evaluates the
robustness of models trained with limited supervision and geographical diversity when facing unexpected
corruptions observed in real-world scenarios.
</p>

<p>The evaluation will be performed on the 19 semantic classes of Cityscapes.
</p>

<h3>Track 2 – Multi-domain training</h3>
<p>In this track, the models may be trained over a mix of multiple datasets, whose choice is strictly
limited to the list provided below, comprising both real and synthetic domains. This track aims to
assess how fewer constraints on the training data can enhance robustness.</p>

<p>The evaluation will be performed on the 19 semantic classes of
<a href="https://www.cityscapes-dataset.com/" target="_blank">Cityscapes</a>. Participants may choose to
maintain
the label sets of each dataset or remap them to Cityscapes.
</p>

<p>Allowed training datasets for Track 2:</p>
<ul>
<li><a href="https://www.cityscapes-dataset.com/" target="_blank">Cityscapes</a></li>
<li><a href="https://bdd-data.berkeley.edu/" target="_blank">BDD100k</a></li>
<li><a href="https://www.mapillary.com/datasets" target="_blank">Mapillary Vistas</a></li>
<li><a href="https://idd.insaan.iiit.ac.in/" target="_blank">India Driving Dataset</a></li>
<li><a href="https://www.wilddash.cc/" target="_blank">WildDash 2</a></li>
<li><a href="https://download.visinf.tu-darmstadt.de/data/from_games/" target="_blank">GTA5 Dataset</a>
(synthetic)</li>
<li><a href="https://www.vis.xyz/shift/" target="_blank">SHIFT Dataset</a> (synthetic)</li>
</ul>

<h3>BRAVO Dataset</h3>
<p>We created the benchmark dataset with real captured images and realistic-looking augmented images,
repurposing existing datasets and combining them with newly generated data. The benchmark dataset
comprises images from <a href="https://acdc.vision.ee.ethz.ch/" target="_blank">ACDC</a>,
<a href="https://segmentmeifyoucan.com/" target="_blank">SegmentMeIfYouCan</a>,
<a href="https://arxiv.org/abs/2108.00968" target="_blank">Out-of-context Cityscapes</a>, and new
synthetic data.
</p>
<p>Get the full benchmark dataset at the following link:
<a href="https://drive.google.com/drive/u/4/folders/11-dnlbMjm8O_ynq1REuDYKOmHLqEhGYP"
target="_blank">full BRAVO
Dataset download link</a>.
</p>
<p>The dataset includes the following splits (with individual download links):</p>
<p><b>bravo-synobjs:</b> augmented scenes with inpainted synthetic OOD objects. We augmented the
validation images of Cityscapes and generated 656 images with 26 OOD objects. (<a
href="https://drive.google.com/drive/u/4/folders/1KKt_25S69DBf8ZTxhOhELpLgS2gyyGnf"
target="_blank">download
link</a>)</p>
</p>
<p class="image-container">
<img src="images/bravobenchmark/synobjs/cheetah.png" alt="Image 1">
<img src="images/bravobenchmark/synobjs/chimpanzee.png" alt="Image 2">
<img src="images/bravobenchmark/synobjs/lion.png" alt="Image 3">
<img src="images/bravobenchmark/synobjs/panda.png" alt="Image 4">
<img src="images/bravobenchmark/synobjs/penguine.png" alt="Image 5">
</p>
<p><b>bravo-synrain:</b> augmented scenes with synthesized raindrops on the camera lens. We
augmented the validation images of Cityscapes and generated 500 images with raindrops. (<a
href="https://drive.google.com/drive/u/4/folders/1onP6tUVSjV-qKWWLm6wiOZCB9U14_gQ6"
target="_blank">download
link</a>)</p>
<p class="image-container">
<img src="images/bravobenchmark/synrain/rain1.png" alt="Image 1">
<img src="images/bravobenchmark/synrain/rain2.png" alt="Image 2">
<img src="images/bravobenchmark/synrain/rain3.png" alt="Image 3">
<img src="images/bravobenchmark/synrain/rain4.png" alt="Image 4">
<img src="images/bravobenchmark/synrain/rain5.png" alt="Image 5">
</p>
<p><b>bravo-synflare:</b> augmented scenes with synthesized light flares. We augmented the
validation images of Cityscapes and generated 308 images with random light flares. (<a
href="https://drive.google.com/drive/u/4/folders/13EpBXUY8BChoqfMxR5JhiyhqrzqLAO2y"
target="_blank">download
link</a>)</p>
<p class="image-container">
<img src="images/bravobenchmark/synflare/flare1.png" alt="Image 1">
<img src="images/bravobenchmark/synflare/flare2.png" alt="Image 2">
<img src="images/bravobenchmark/synflare/flare3.png" alt="Image 3">
<img src="images/bravobenchmark/synflare/flare4.png" alt="Image 4">
<img src="images/bravobenchmark/synflare/flare5.png" alt="Image 5">
</p>
<p><b>bravo-outofcontext:</b> augmented scenes with random backgrounds. We augmented the
validation images of Cityscapes and generated 329 images with random random backgrounds. (<a
href="https://drive.google.com/drive/u/4/folders/1NoXqTQWxrj_yKMNRKLOd1rnn2TjqIaU5"
target="_blank">download
link</a>)</p>
<p class="image-container">
<img src="images/bravobenchmark/synooc/ooc1.png" alt="Image 1">
<img src="images/bravobenchmark/synooc/ooc2.png" alt="Image 2">
<img src="images/bravobenchmark/synooc/ooc3.png" alt="Image 3">
<img src="images/bravobenchmark/synooc/ooc4.png" alt="Image 4">
<img src="images/bravobenchmark/synooc/ooc5.png" alt="Image 5">
</p>
<p><b>bravo-ACDC:</b> real scenes captured in adverse weather conditions, i.e. fog, night, rain
and snow. (<a href="https://drive.google.com/drive/u/4/folders/1IW6-Tdfk2At6CrIIrA-QJF6CEcHgqqha"
target="_blank">download
link</a> or directly from <a href="https://acdc.vision.ee.ethz.ch/download">ACDC
website</a>)</p>
<p class="image-container">
<img src="images/bravobenchmark/acdc/acdc1.png" alt="Image 1">
<img src="images/bravobenchmark/acdc/acdc2.png" alt="Image 2">
<img src="images/bravobenchmark/acdc/acdc3.png" alt="Image 3">
<img src="images/bravobenchmark/acdc/acdc4.png" alt="Image 4">
<img src="images/bravobenchmark/acdc/acdc5.png" alt="Image 5">
</p>
<p><b>bravo-SMIYC:</b> real scenes featuring out-of-distribution (OOD) objects rarely encountered
on the road. (<a href="https://drive.google.com/drive/u/4/folders/1XnC9_7RzwZCWaDpP3iETbGt7Yvmg0MOg"
target="_blank">download
link</a> or directly from <a href="https://segmentmeifyoucan.com/">SMIYC
website</a>)</p>
<p class="image-container">
<img src="images/bravobenchmark/smiyc/smiyc1.jpg" alt="Image 1">
<img src="images/bravobenchmark/smiyc/smiyc2.jpg" alt="Image 2">
<img src="images/bravobenchmark/smiyc/smiyc3.jpg" alt="Image 3">
<img src="images/bravobenchmark/smiyc/smiyc4.jpg" alt="Image 4">
<img src="images/bravobenchmark/smiyc/smiyc5.jpg" alt="Image 5">
</p>

<h3>Metrics</h3>
<p>For a comprehensive assessment of the robustness of various semantic segmentation models, we adopt the
following metrics:</p>
<ul>
<li><em>mIoU</em>: mean Intersection Over Union, quantifying the degree of overlap between correct
predictions and actual labels against the total number of true positives, false positives, and false
negatives. Evaluated splits: bravo-ACDC, bravo-synrain, bravo-synflare, bravo-outofcontext.</li>

<li><em>ECE</em>: Expected Calibration Error, measuring the expected difference between accuracy and
predicted uncertainty. Evaluated splits: bravo-ACDC, bravo-synrain, bravo-synflare,
bravo-outofcontext.</li>

<li><em>AUPR-Success</em>: Area Under the Precision-Recall curve Error, computing the area under the
Precision-Recall curve using semantic prediction successes as the positive class. Evaluated splits:
bravo-ACDC, bravo-synrain, bravo-synflare, bravo-outofcontext. The evaluation code should resemble:
<code>sklearn.metrics.precision_recall_curve(pred==label, conf)</code>.
</li>

<li><em>AUPR-Error</em>: Area Under the Precision-Recall curve Error, computing the area under the
Precision-Recall curve using semantic prediction failures as the positive class. Evaluated splits:
bravo-ACDC, bravo-synrain, bravo-synflare, bravo-outofcontext. The evaluation code should resemble:
<code>sklearn.metrics.precision_recall_curve(pred!=label, -conf)</code>.
</li>
<p>For more information, please check the <a href="https://github.com/valeoai/bravo_challenge">BRAVO
Challenge Repository</a> and the <a
href="https://benchmarks.elsa-ai.eu/?ch=1&com=introduction">Challenge Task Website at
ELLIS/ELSA</a>.</p>

<li><em>AUROC-ood</em>: Area Under the ROC Curve, a threshold-free metric quantifying the probability
that a randomly chosen certain example will be ranked higher than a randomly chosen uncertain one.
Evaluated splits: bravo-ACDC, bravo-synrain, bravo-synflare, bravo-synobjs, bravo-SMIYC. The
evaluation code should resemble: <code>sklearn.metrics.roc_curve(ood_label, -conf)</code>.
</li>

<li><em>AUPR-ood</em>: Area Under the ROC Curve, a threshold-free metric quantifying the probability
that a randomly chosen certain example will be ranked higher than a randomly chosen uncertain one.
Evaluated splits: bravo-ACDC, bravo-synrain, bravo-synflare, bravo-synobjs, bravo-SMIYC. The
evaluation code should resemble:
<code>sklearn.metrics.precision_recall_curve(ood_label, -conf)</code>.
</li>

<li><em>FPR@95TPR-ood</em>: measuring the False Positive Rate when setting the True Positive Rate to
95%. Evaluated splits: bravo-ACDC, bravo-synrain, bravo-synflare, bravo-synobjs, bravo-SMIYC.
</li>
</ul>

<h3>Benchmark server</h3>
<p>We are excited to unveil the BRAVO Challenge as an initiative within
<a href="https://www.elsa-ai.eu/" target="_blank">ELSA — European Lighthouse on Secure and Safe AI</a>,
a network of excellence funded by the European Union. The BRAVO Challenge is officially featured on the
<a href="https://benchmarks.elsa-ai.eu/" target="_blank">ELSA Benchmarks website</a> as
the Autonomous Driving/Robust Perception task.
</p>

<!-- <p style="color:#FF0000">For the moment, we opt to the manual submission via email. Please directly contact
us via [email protected] or [email protected]. To ensure that your results are available by
the time of the workshop, please submit your entry no later than September 30th.</p>
-->
<p>Please refer to the <a href="https://benchmarks.elsa-ai.eu/?ch=1&com=introduction" target="_blank">task
website</a> for detailed submission information on the submission format and schedule.</p>

<h3>Submission format</h3>
<ul>
<li>A single tar archive with the same structure of the bravo dataset:
<ul class="tree">
<li>
bravo_ACDC
<ul class="tree">
<li>fog</li>
<li>night</li>
<li>rain</li>
<li>snow</li>
</ul>
</li>
<li>
bravo_synrain
<ul class="tree">
<li>frankfurt</li>
<li>lindau</li>
<li>munster</li>
</ul>
</li>
<li>...</li>
</ul>
</li>
<li>For each input image "ori_name.png", we requires two corresponding files: "ori_name_pred.png" for
the semantic prediction and "ori_name_conf.png" for the confidence level regarding the model's
predictions."</li>
<li>Semantic predictions <code>pred</code> must be 8-bit grayscale <code>.png</code> images
(<code>numpy.uint8</code>). Predictions must be encoded in Cityscapes 19-class format, e.g., road
should correspond to ID 0. List of 19 Cityscapes
<code>['road', 'sidewalk', 'building', 'wall', 'fence', 'pole', 'traffic light', 'traffic sign', 'vegetation', 'terrain', 'sky', 'person', 'rider', 'car', 'truck', 'bus', 'train', 'motorcycle','bicycle']</code>.
Please refer to <a href="./files/cityscapes.json" target="_blank">cityscapes.json</a> for the
19-class mapping.
</li>
<li>Confidence maps <code>conf</code> must be 16-bit grayscale <code>.png</code> <s>.webp</s> images
(<code>numpy.uint16</code>), where a value of <code>0.0</code> corresponds to confidence
<code>0</code> and a value of <code>65535</code> <s>100</s>
corresponds to confidence <code>1.0</code>. The same confidence map <code>conf</code> is used for
all metrics.
The python code for saving confidence should resemble
<code>cv2.imwrite(conf_webp_file, conf)</code> <s>[cv2.IMWRITE_WEBP_QUALITY, 100]</s>
</li>
<li>Example <a
href="https://drive.google.com/file/d/14knXF7wIE4hTBOBZllX9QxaL610UF_hn/view?usp=drive_link"
target="_blank">submission</a>.
</li>
</ul>

<h3>Leaderboard</h3>
<p style="color:#FF0000">Coming soon!</p>

<h3>Acknowledgements</h3>
<p>We extend our heartfelt gratitude to the authors of
Expand All @@ -813,6 +580,12 @@ <h3>Acknowledgements</h3>
flares. All those people collectively contributed to creating BRAVO, a unified benchmark
for robustness in autonomous driving.
</p>
<p>We are excited to unveil the BRAVO Challenge as an initiative within
<a href="https://www.elsa-ai.eu/" target="_blank">ELSA — European Lighthouse on Secure and Safe AI</a>,
a network of excellence funded by the European Union. The BRAVO Challenge is officially featured on the
<a href="https://benchmarks.elsa-ai.eu/" target="_blank">ELSA Benchmarks website</a> as
the Autonomous Driving/Robust Perception task.
</p>
</div>

<div id="organizers" class="container-md section-container">
Expand Down

0 comments on commit 441b7b2

Please sign in to comment.