Your first evaluation¶

Once you have a verified training (or, actually, even if you don't), you can post metrics for it to our website.

1. Install DVBench¶

If you haven't done this already in the previous step.

pip install "dvbench[torch]"

pip install "dvbench[jax]"

2. Connect a Client¶

Begin by connecting to DVBench.

from dvbench import Client  # import before Jax/Torch
client = Client(API_URL)

2. Load your checkpoint¶

To ensure that the validation is bound to your declared training run, use the same model and checkpoint produced during attested training.

checkpoint = ...
checkpoint.eval()  # if in torch

You can checkpoint and load however you'd like, but ensure its the same structure as what you had during training.

3. Bind the checkpoint¶

You now open the experiment and add some measurements! Use the same name as your training run if you want them to be associated together.

experiment = client.experiment("your-name/your-run")
measurement = experiment.measure(checkpoint.state_dict())

experiment = client.experiment("your-name/your-run")
measurement = experiment.measure(checkpoint)

measure() will bind the local checkpoint against the run.

4. Describe the evaluation¶

Declare how you are going to do the evaluation. Note that your code in these functions will be uploaded for reference.

import torch
from torch.func import functional_call

@measurement.evaluate(params=0, batch=1)
def evaluate(params, batch):
    # tip: use functional_call here to call your model
    # given its state_dict
    outputs = functional_call(model, params, **batch)
    return logits.softmax(dim=-1).max(dim=-1).values

@measurement.aggregate
def aggregate(outputs):
    return torch.cat(outputs).mean().item()

import jax
import jax.numpy as jnp

@measurement.evaluate(params=0, batch=1)
def evaluate(params, batch):
    logits = model.apply({"params": params}, batch)
    return jax.nn.softmax(logits, axis=-1).max(axis=-1)

@measurement.aggregate
def aggregate(outputs):
    return float(jnp.concatenate(outputs).mean())

5. Run and submit¶

Finally, we are ready to submit the measurements to the website. First locate the name of the measurement you are submitting to in /measurements, and then pass that to .submit().

batches = load_your_evaluation_batches()
score = measurement.run(checkpoint, batches)
request_id = measurement.submit("measurement_name", score)
client.close()

Check the website again. Your result appears after validation.

client.define_measurement(
    "mean_confidence",
    "Mean maximum class probability over the evaluation set.",
)

Next: participate in a leaderboard.