Create a Leaderboard¶

A leaderboard is backed by a benchmark: a list of metrics and one key metric used for its default ordering.

1. Choose the metrics¶

Check the DVBench website for existing metrics. Record the ID of every metric the leaderboard should require.

from dvbench import Client

client = Client(API_URL)
metric = client.define_measurement(
    "accuracy",
    "Fraction of correct predictions.",
)
metric_id = metric["id"]
client.close()

2. Create the benchmark¶

from dvbench import Client

client = Client(API_URL)

benchmark = client.create_benchmark(
    "Core evaluation",
    "Compare runs on the core evaluation suite.",
    measurement_ids=[CODING_ID, MATH_ID, PERPLEXITY_ID],
    key_measurement_id=CODING_ID,
    sort_direction="desc",
)

client.close()

Use desc when higher is better and asc when lower is better.