dokimi eval
usage: dokimi eval [-h]
{score,ragas,grounding,judge-validity,checklist,judge-wording,cross-judge,pairwise,panel-score} ...
positional arguments:
{score,ragas,grounding,judge-validity,checklist,judge-wording,cross-judge,pairwise,panel-score}
score Score answers using LLM-as-judge
ragas Run RAGAS meta-evaluation on answers
grounding Reference-free grounding/citation metrics with confidence intervals
judge-validity Measure judge discriminative validity on deterministic degradations (exits
1 on FAIL)
checklist Required-point coverage per arm: is each expected_topic in the answer?
(both polarities)
judge-wording Judge-validity under rewrites of the judge prompt, paired against the
original on the same pairs
cross-judge Each arm's delta under several judges and axes, labelled CONSISTENT /
JUDGE-DEPENDENT / ...
pairwise Every arm vs the baseline, both answers shown together in both orders;
optional escalation
panel-score Score answers using a multi-judge panel
options:
-h, --help show this help message and exit
dokimi eval score
usage: dokimi eval score [-h] --product PRODUCT --answers ANSWERS [--run-id RUN_ID]
[--output OUTPUT] [--judge-model JUDGE_MODEL]
[--judge-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}]
[--concurrency CONCURRENCY] [--judge-temperature JUDGE_TEMPERATURE]
options:
-h, --help show this help message and exit
--product PRODUCT Product name (e.g., oneTBB)
--answers ANSWERS Path to answers JSON file
--run-id RUN_ID Run tag (e.g. gpt4o). Sets output to
results/{product}_{run_id}/eval/{product}.json
--output OUTPUT Output file (default: eval/{product}.json)
--judge-model JUDGE_MODEL
LLM model for judging
--judge-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}
--concurrency CONCURRENCY
Parallel judge calls (default: 5)
--judge-temperature JUDGE_TEMPERATURE
Judge sampling temperature, recorded as judge_temperature in the output
dokimi eval ragas
usage: dokimi eval ragas [-h] --answers ANSWERS [--ragas-model RAGAS_MODEL]
[--ragas-provider {openai,anthropic}]
options:
-h, --help show this help message and exit
--answers ANSWERS Path to answers JSON file
--ragas-model RAGAS_MODEL
LLM for RAGAS judge (default: claude-sonnet-5-5)
--ragas-provider {openai,anthropic}
Provider for RAGAS judge LLM (default: anthropic)
dokimi eval grounding
usage: dokimi eval grounding [-h] --answers ANSWERS [--output OUTPUT] [--threshold THRESHOLD]
options:
-h, --help show this help message and exit
--answers ANSWERS Path to answers JSON file
--output OUTPUT Output JSON report path
--threshold THRESHOLD
Grounding score above which an answer counts as grounded (default: 0.5)
dokimi eval judge-validity
usage: dokimi eval judge-validity [-h] --answers ANSWERS [--product PRODUCT] [--bases BASES]
[--variants VARIANTS] [--seed SEED]
[--min-accuracy MIN_ACCURACY] [--strict] [--sequential]
[--max-bases MAX_BASES] [--mode {absolute,pairwise}]
[--escalate-model ESCALATE_MODEL]
[--escalate-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}]
[--escalate-ties] [--concurrency CONCURRENCY]
[--judge-model JUDGE_MODEL]
[--judge-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}]
[--judge-temperature JUDGE_TEMPERATURE] [--scorer SCORER]
[--typed-max-len TYPED_MAX_LEN] [--typed-threads TYPED_THREADS]
[--output OUTPUT]
options:
-h, --help show this help message and exit
--answers ANSWERS Path to answers JSON file
--product PRODUCT Library name (default: read from answers file)
--bases BASES Number of base answers to degrade (default: 20)
--variants VARIANTS Comma-separated degradations (default: all)
--seed SEED Seed for the degradations (default: 0)
--min-accuracy MIN_ACCURACY
Pairwise accuracy gate, per degradation (default: 0.8); with --sequential
it must lie in 0.03-0.97
--strict Also exit 1 when a PASS is inconclusive — its 95% CI straddles the gate,
so the run cannot tell it from a FAIL
--sequential Add bases in rounds and stop each degradation once an SPRT can call it
PASS or FAIL (ignores --bases; see --max-bases)
--max-bases MAX_BASES
Cap on base answers in --sequential mode (default: 120)
--mode {absolute,pairwise}
absolute: score each member 0-100 (the shipped judge); pairwise: show the
judge both members, in both orders, and ask which is better
--escalate-model ESCALATE_MODEL
With --mode pairwise: re-ask pairs the judge answered inconsistently
across the two orders with this model
--escalate-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}
Provider for --escalate-model (default: --judge-provider)
--escalate-ties Also escalate pairs called TIE in both orders (default: only order-
inconsistent)
--concurrency CONCURRENCY
Parallel judge calls (default: 5)
--judge-model JUDGE_MODEL
LLM model for judging
--judge-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}
--judge-temperature JUDGE_TEMPERATURE
Judge sampling temperature; match arms run --judge-temperature to certify
the same judge
--scorer SCORER Arm to grade: 'judge' (default, the shipped LLM judge) or
'typed:<backend>[:<repo>]' for a local typed-decision baseline arm — needs
the [typed-scorer] extra
--typed-max-len TYPED_MAX_LEN
Token budget for a typed arm's state (default: the checkpoint's own, 512
for laya English — answers are longer than that and get truncated)
--typed-threads TYPED_THREADS
CPU threads for a typed arm (default: torch's own)
--output OUTPUT Output JSON report path
dokimi eval checklist
usage: dokimi eval checklist [-h] --answers ANSWERS --questions QUESTIONS
[--baseline-arm BASELINE_ARM] [--arms ARMS]
[--judge-model JUDGE_MODEL]
[--judge-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}]
[--judge-temperature JUDGE_TEMPERATURE] [--concurrency CONCURRENCY]
[--output OUTPUT]
options:
-h, --help show this help message and exit
--answers ANSWERS Arms output JSON (arms run --out-json)
--questions QUESTIONS
Questions JSON whose expected_topics are the points
--baseline-arm BASELINE_ARM
Arm the others are compared to (default: the run's baseline_arm)
--arms ARMS Comma-separated arms to check (default: all)
--judge-model JUDGE_MODEL
--judge-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}
--judge-temperature JUDGE_TEMPERATURE
Judge sampling temperature, recorded as judge_temperature in the output
--concurrency CONCURRENCY
--output OUTPUT Output JSON (default: <answers>_checklist.json); Markdown written next to
it
dokimi eval judge-wording
usage: dokimi eval judge-wording [-h] --answers ANSWERS [--product PRODUCT] [--bases BASES]
[--variants VARIANTS] [--wordings WORDINGS] [--redraws REDRAWS]
[--seed SEED] [--min-accuracy MIN_ACCURACY] [--strict]
[--judge-model JUDGE_MODEL]
[--judge-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}]
[--judge-temperature JUDGE_TEMPERATURE]
[--concurrency CONCURRENCY] [--output OUTPUT]
options:
-h, --help show this help message and exit
--answers ANSWERS Path to answers JSON file
--product PRODUCT Library name (default: read from answers file)
--bases BASES Base answers to degrade (default: 20)
--variants VARIANTS Comma-separated degradations (default: all)
--wordings WORDINGS Comma-separated rewrites (default: all)
--redraws REDRAWS Re-draws of the original prompt, shown as a noise floor; the FRAGILE flag
is a paired test against the original and does not use them (default: 1)
--seed SEED
--min-accuracy MIN_ACCURACY
--strict Exit 1 when any variant is FRAGILE
--judge-model JUDGE_MODEL
--judge-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}
--judge-temperature JUDGE_TEMPERATURE
Judge sampling temperature, recorded as judge_temperature in the output
--concurrency CONCURRENCY
--output OUTPUT Output JSON report path
dokimi eval cross-judge
usage: dokimi eval cross-judge [-h] --answers ANSWERS [--judges JUDGES] [--checklist CHECKLIST]
[--pairwise PAIRWISE] [--validity VALIDITY] [--require-validated]
[--questions QUESTIONS] [--grounded] [--both] [--no-stored]
[--product PRODUCT] [--baseline-arm BASELINE_ARM]
[--judge-temperature JUDGE_TEMPERATURE] [--concurrency CONCURRENCY]
[--strict] [--output OUTPUT]
options:
-h, --help show this help message and exit
--answers ANSWERS Arms output JSON (arms run --out-json)
--judges JUDGES Comma-separated provider/model judges to re-score with, e.g. amazon-
bedrock/us.anthropic.claude-haiku-5-5,amazon-bedrock/openai.gpt-
oss-120b-1:0
--checklist CHECKLIST
An `eval checklist` JSON to include as a coverage measure (repeatable)
--pairwise PAIRWISE An `eval pairwise` JSON to include as a measure (repeatable)
--validity VALIDITY An `eval judge-validity` JSON; that judge's measures are marked PASS/FAIL
and a FAILed judge does not vote in the verdict (repeatable)
--require-validated Only judges that PASSED a --validity gate vote in the verdict
--questions QUESTIONS
Questions JSON; needed for --grounded
--grounded Re-score against each question's expected_topics as ground truth
--both With --grounded, keep the ungrounded score as a measure too
--no-stored Leave out the scores the arms run itself recorded
--product PRODUCT Library name (default: read from answers file)
--baseline-arm BASELINE_ARM
--judge-temperature JUDGE_TEMPERATURE
Judge sampling temperature, recorded as judge_temperature in the output
--concurrency CONCURRENCY
--strict Exit 1 when any arm is JUDGE-DEPENDENT
--output OUTPUT Output JSON (default: <answers>_crossjudge.json); Markdown written next to
it
dokimi eval pairwise
usage: dokimi eval pairwise [-h] --answers ANSWERS [--product PRODUCT]
[--baseline-arm BASELINE_ARM] [--arms ARMS]
[--judge-model JUDGE_MODEL]
[--judge-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}]
[--judge-temperature JUDGE_TEMPERATURE]
[--escalate-model ESCALATE_MODEL]
[--escalate-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}]
[--escalate-ties] [--concurrency CONCURRENCY] [--output OUTPUT]
options:
-h, --help show this help message and exit
--answers ANSWERS Arms output JSON (arms run --out-json)
--product PRODUCT Library name (default: read from answers file)
--baseline-arm BASELINE_ARM
--arms ARMS Comma-separated arms to compare (default: all)
--judge-model JUDGE_MODEL
--judge-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}
--judge-temperature JUDGE_TEMPERATURE
Judge sampling temperature, recorded as judge_temperature in the output
--escalate-model ESCALATE_MODEL
Re-ask questions the judge left undecided (tie or order-inconsistent) with
this model
--escalate-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}
--escalate-ties Also escalate questions the judge called TIE in both orders (default: only
order-inconsistent ones)
--concurrency CONCURRENCY
--output OUTPUT Output JSON (default: <answers>_pairwise.json); Markdown written next to
it
dokimi eval panel-score
usage: dokimi eval panel-score [-h] --answers ANSWERS --product PRODUCT [--output OUTPUT]
[--model MODEL]
[--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}]
[--roles ROLES] [--judge-configs JUDGE_CONFIGS]
[--concurrency CONCURRENCY] [--limit LIMIT]
[--judge-temperature JUDGE_TEMPERATURE]
options:
-h, --help show this help message and exit
--answers ANSWERS Path to answers JSON file
--product PRODUCT Product name (e.g., oneTBB)
--output OUTPUT Output file (default: eval/{product}_panel.json)
--model MODEL Default LLM model for all judges
--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}
--roles ROLES Comma-separated judge roles (default:
technical_expert,developer_advocate,doc_reviewer)
--judge-configs JUDGE_CONFIGS
Per-role model config (overrides --model/--provider/--roles). Format:
'role:model:provider,...' e.g. 'technical_expert:claude-
sonnet-5-5:anthropic,developer_advocate:claude-
opus-5-5:anthropic,doc_reviewer:gemini-2.0-flash:google'
--concurrency CONCURRENCY
Parallel judge API calls (default: 6)
--limit LIMIT Evaluate only first N answers (useful for testing)
--judge-temperature JUDGE_TEMPERATURE
Sampling temperature for every panel judge, recorded as judge_temperature