Pick a saved investigation and press Replay
Download report (JSON) Includes progress recorded so far.
The fix against its control
What to change
Cost
How it got there
Causes, before and after each check
Benchmark on 13 Token Factory jobs with a known cause
9 with a learning rate that is too low, 4 healthy| Method | Cause ranked first | False alarms | Wrong confirmations | Fix success | Per case | Jobs per case |
|---|---|---|---|---|---|---|
| Single Ultra call | 6/9 | 0/4 | 0 | 0/9 | $0.009 | 0.0 |
| ReAct on Super | 9/9 | 0/4 | 0 | 6/9 | $0.205 | 4.9 |
| Checklist, every check | 9/9 | 0/4 | 6 | 6/9 | $0.417 | 12.0 |
| Next Experiment | 9/9 | 0/4 | 0 | 6/9 | $0.144 | 3.7 |
Thresholds were set on 5 dev cases; the 8 held-out cases ran once. Every fault case has the same cause, so this covers one failure family. Case by case
How a run goes
Investigate your own data with Nemotron
Upload four chat JSONL splits and follow Nemotron's hypotheses, the free checks and a fix trained against a control on Token Factory. Testing is free and needs no API key. Live runs share a $20 allowance, and the form shows what is left before you start.
Replays and the label check below never spend the live allowance.
Inspect local training labels
Paste reference and observed unpadded token rows. The reference must use correct answer-only labels. This checks prompt loss, EOS masking and label alignment. It does not train a model or confirm a cause.
On this static site, pasted data stays in your browser. Supply unpadded rows: reference -100 positions must be prompt tokens only. Position 0 is excluded from causal loss. The reference is yours; this audit cannot judge its semantic correctness or prove a cause of model behavior.
Limits: 1–1024 rows per side, 32768 positions per row, 131072 positions across both sides. Token IDs are nonnegative integers; labels may also be -100. Both sides must describe identical token sequences. Optional eos_token_id must be a nonnegative integer or null.
Download audit and inputs (JSON)Computed audit details
double_shift means the supplied labels match one extra shift. other_mismatch means neither alignment pattern matches. not_applicable means the changes do not fit that check. no_eos_targets and inconclusive provide no conclusion for that check. EOS removal is observable; whether padding caused it is not established.
Investigate your own job
BANKING77, JSON extraction or function calling. Paste a standalone InvestigationCase for a free dataset audit, or case and experiment for a paid Token Factory investigation on Qwen3-0.6B. Dataset paths must exist on the API server.