Coopetition

How to organize a coopetition?

A coopetition is a mix of competition and collaboration. To encourage collaboration in a competition, you can make use of public submissions.

See our example of coopetition.. This was run for the Pertersburg hackathon.

This example uses special features of the leaderboard. The ranking is performed by averaging the prediction performances (averaged over 5 datasets) and the so-called coopetition score (the popularity of the shared submissions as measured by the number of downloads of shared code).

This is how the "RESULT" leaderboard looks like:

This is how the "PUBLIC SUBMISSIONS" leaderboard looks like:

We describe the leaderboard columns:

• : This is the average of the numbers in parenthesis in columns "AVG" and column "Coopetition". This is the column used for the final evaluation.
• AVG: This is the average of the scores shown in columns Set1 through Set 5.
• Set i: These are the columns with the scores for each dataset.
• Duration: Execution time of the code on the 5 datasets. sub. no: Number of submissions made by the participant.
• Coopetition: Total number of downloads of code by other participants.
• Diff: Difference in average performance to the best previous submission.
• Detailed Results: Scores with respect to all metrics.

For each column, the rank of the participant obtained by sorting is shown in parenthesis.

To enable coopetitions, you need to configure the leaderboard in a particular way in the YAML file:

leaderboard:
    leaderboards:
        Results: &RESULTS
            label: RESULTS
            rank: 1
    columns:
        set1_score:
            leaderboard: *RESULTS
            label: Set 1
            numeric_format: 3
            rank: 3
        set2_score:
            leaderboard: *RESULTS
            label: Set 2
            numeric_format: 3
            rank: 4
        set3_score:
            leaderboard: *RESULTS
            label: Set 3
            numeric_format: 3
            rank: 5
        set4_score:
            leaderboard: *RESULTS
            label: Set 4
            numeric_format: 3
            rank: 6
        set5_score:
            leaderboard: *RESULTS
            label: Set 5
            numeric_format: 3
            rank: 7
        avg_score:
            leaderboard: *RESULTS
            label: AVG
            numeric_format: 4
            rank: 2
        rank:
            leaderboard: *RESULTS
            label: <Rank>
            numeric_format: 2
            rank: 1
            computed:
                operation: Avg
                fields: avg_score, Ndown   
        Duration:
            leaderboard: *RESULTS
            label: Duration 
            numeric_format: 2
            rank: 8
        Nsub:
            leaderboard: *RESULTS
            label: sub. no 
            numeric_format: 0
            rank: 9
        Ndown:
            leaderboard: *RESULTS
            label: Coopetition
            numeric_format: 0
            rank: 10  
        Diff:   
            leaderboard: *RESULTS
            label: Diff
            numeric_format: 4
            rank: 11  

Notice the special columns:

• rank: averaging avg_score and Ndown
• Nsub: number of submissions.
• Ndown: number of downloads.

The scoring program supplies the other scores.