Transcript algorithm¶

ZKDV's original goal is to produce a public, signed training trace that supports after-the-fact checks without publishing full private batches, gradients, updates, checkpoints, or exported program bodies. The trainer runs the model update, the prover records compact evidence and performs sampled checks, and a public verifier examines the resulting transcript and released checkpoint.

The intended evidence has four parts:

1. Data evidence: rolling fingerprints represent windows from the private token stream without placing the full batches in the public trace.
2. Checkpoint evidence: orthonormal probes reduce each parameter leaf and a BLAKE3 Merkle root binds the ordered projected state.
3. Computation evidence: a committed, exported update program recomputes the full transition after a sampled challenge.
4. Authenticity evidence: a session key signs an append-only record chain rooted in measured prover identity.

The current implementation realizes this as an ordered CBOR record stream. A root establishes the session, a separate measurement record describes the prover, rolling-hash and update-program commitments fix run-wide inputs, and update records commit the evolving training state.

1. Establish the session¶

The root record binds the protocol version, run name, prover build identity, session public key, time and nonce, floating-point tolerance, and sampling policy. Its prover name is whitebox-jax or whitebox-torch, identifying the frontend that produced the transcript. The next record contains a challenged measurement of the prover. The diagram below captures the original trust-boundary goal; in the current record schema, measurements are a separate payload rather than fields nested directly in the root.

2. Fix the run-wide commitments¶

Before updates are recorded, start() commits the rolling-hash parameters derived for the session. On the first compiled call, ZKDV generates Newton--Schulz-orthonormalized probe rows, records each leaf's shape, probe count, and vector digest, then exports and commits the sampled update program. The run configuration fixes the last-dimension probe ratio (default 0.01) and rolling-hash window.

3. Commit each update¶

Each update contributes an opening header, rolling-window batch members, a full packed-batch digest, a post-update projected BLAKE3 Merkle commitment, an optimizer-state hash, a sampling decision, and a footer. The pre-update parameter commitment and optimizer-state hash connect an update to the state it started from.

The sampling decision is derived after the update commitment. ZKDV periodically copies a complete pre-state into JAX's compiler-addressable host memory (pinned where the backend provides it). At the beginning of a due update, a background host dispatcher invokes one precompiled whole-tree D2H executable. The training thread waits only for the dispatch handoff, not the transfer result. That handoff makes JAX register every source read before the donated training call can reuse the same buffers. A bounded set of complete host destinations is allocated before training and donated into successive copy outputs, avoiding cold pinning and Python leaf-by-leaf copies. Unchecked flow retains but never reads or waits for the copy destination; join() and a sampled check are explicit drain points. This establishes exact liveness and portable ordering, but transfer/training overlap remains a backend scheduling property and must be demonstrated by an offload-inclusive profile. With the default replay_snapshot_interval=12, ZKDV journals the intervening packed batches and bounds the replay suffix at 11 preceding transitions. Set the interval to 1 to snapshot every step and execute exactly the challenged transition.

The training-path snapshot is not trusted merely because it resides on the host. A sampled check restores it only after selection, runs the sealed transition(s), and binds the challenged update's pre- and post-states to the signed parameter projections and optimizer commitments. The optimizer commitment uses eight independent verifier-keyed 32-bit fingerprints per leaf before the BLAKE3 Merkle reduction. The sampled parameter-update coordinate is still compared directly. Corrupt or inconsistent snapshot state therefore fails the same commitment checks as replay from an older checkpoint.

The CheckResult may be appended later, after the update footer, because verification work can overlap subsequent training. Checked replay is allowed to be much slower than an unchecked training step; its full-state device and host work occurs only after sampling.

The training step still computes the full update. Sampling controls the extra verification work; it does not replace the production update with a partial one.

4. Chain and sign records¶

Every record includes the previous record digest. The session key signs a domain-separated digest of the payload and predecessor; the record digest then covers the payload, predecessor, and signature. This makes both content and order detectable during verification.

5. End and verify¶

ZKDV.close() waits for queued commitments and sampled checks, flushes the tape, and reports asynchronous verification failure. The current Rust transcript verifier checks chain links, record signatures, protocol version, update ordering, sampling derivation, challenged indices, and the presence and result of deferred checks.

The larger product goal is to let a public verifier compare the trace's final parameter commitment with a separately released checkpoint and query rolling fingerprints for candidate data. Those end-to-end public-verification workflows are goals from the original specification and should not be read as documented CLI support today.

Terms¶