dokimi benchmark
usage: dokimi benchmark [-h] {run,preflight,batch} ...
positional arguments:
{run,preflight,batch}
run Run full benchmark for a registered target
preflight Validate resolved benchmark plan without running LLM calls
batch Run pipeline for multiple libraries
options:
-h, --help show this help message and exit
dokimi benchmark run
usage: dokimi benchmark run [-h] --library LIBRARY [--doc-source DOC_SOURCE]
[--output-dir OUTPUT_DIR] [--model MODEL]
[--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}]
[--judge-model JUDGE_MODEL]
[--judge-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}]
[--registry REGISTRY] [--max-tokens MAX_TOKENS]
[--concurrency CONCURRENCY] [--force-regen]
[--questions-from QUESTIONS_FROM] [--questions-root QUESTIONS_ROOT]
[--multi-run N]
options:
-h, --help show this help message and exit
--library, --target LIBRARY
Target key from registry (e.g., onetbb). --library kept as compatibility
alias
--doc-source DOC_SOURCE
Override doc/context source (default: first in registry entry)
--output-dir OUTPUT_DIR
Output directory (default: results/{target}_final)
--model MODEL
--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}
--judge-model JUDGE_MODEL
--judge-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}
--registry REGISTRY Path to custom products.yaml
--max-tokens MAX_TOKENS
Max tokens to retrieve per question from doc source (default: 4000)
--concurrency CONCURRENCY
Parallel API calls for answering and judging (default: 5)
--force-regen Regenerate personas/questions even if cached files exist
--questions-from QUESTIONS_FROM
Reuse questions from another run's output directory or JSON file. Skips
question generation entirely — essential for fair multi-model comparisons
(all models evaluated on the same question set). Example: --questions-from
results/onedal_gpt51
--questions-root QUESTIONS_ROOT
Directory where the golden question set <target>_golden.json is looked up,
for example a checkout of the evaluation-content repository: --questions-
root /path/to/content-repo/questions. Without it the run has no golden
question set.
--multi-run N Run answer generation + evaluation N times (default: 1). N>=3 enables
variance check in the trust gate. Results are averaged; variance is
measured for stability.
dokimi benchmark preflight
usage: dokimi benchmark preflight [-h] --library LIBRARY [--doc-source DOC_SOURCE]
[--output-dir OUTPUT_DIR] [--model MODEL]
[--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}]
[--judge-model JUDGE_MODEL]
[--judge-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}]
[--registry REGISTRY] [--max-tokens MAX_TOKENS]
[--concurrency CONCURRENCY] [--questions-from QUESTIONS_FROM]
[--questions-root QUESTIONS_ROOT]
options:
-h, --help show this help message and exit
--library, --target LIBRARY
Target key from registry (e.g., onetbb)
--doc-source DOC_SOURCE
Override doc/context source
--output-dir OUTPUT_DIR
Output directory (default: results/{target}_final)
--model MODEL
--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}
--judge-model JUDGE_MODEL
--judge-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}
--registry REGISTRY Path to custom products.yaml
--max-tokens MAX_TOKENS
--concurrency CONCURRENCY
--questions-from QUESTIONS_FROM
Question-set source for fair comparison
--questions-root QUESTIONS_ROOT
Directory where <target>_golden.json is looked up (e.g. /path/to/content-
repo/questions)
dokimi benchmark batch
usage: dokimi benchmark batch [-h] [--libraries LIBRARIES] [--all] [--output-dir OUTPUT_DIR]
[--model MODEL]
[--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}]
[--judge-model JUDGE_MODEL]
[--judge-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}]
[--max-tokens MAX_TOKENS] [--force-regen] [--registry REGISTRY]
[--questions-root QUESTIONS_ROOT] [--fail-fast]
options:
-h, --help show this help message and exit
--libraries LIBRARIES
Comma-separated library keys (e.g., onetbb,onemkl). Omit or use --all for
all registered libraries.
--all Run for all libraries in registry
--output-dir OUTPUT_DIR
--model MODEL
--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}
--judge-model JUDGE_MODEL
--judge-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}
--max-tokens MAX_TOKENS
Max tokens to retrieve per question from doc source (default: 4000)
--force-regen Regenerate personas/questions even if cached files exist
--registry REGISTRY Path to custom products.yaml
--questions-root QUESTIONS_ROOT
Directory where each <target>_golden.json is looked up (e.g.
/path/to/content-repo/questions)
--fail-fast Stop on first failure (default: continue all)