dokimi arms
usage: dokimi arms [-h] {run,matrix-run,plugin-delta} ...
positional arguments:
{run,matrix-run,plugin-delta}
run Run an N-arm comparison and (optionally) judge it
matrix-run Run every cell from a matrix.cells descriptor and write a rollup
plugin-delta Compare paired arms artifacts that differ only by plugin set
options:
-h, --help show this help message and exit
dokimi arms run
usage: dokimi arms run [-h] --product PRODUCT --questions QUESTIONS --arms ARMS [--model MODEL]
[--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}]
[--harness HARNESS] [--plugins PLUGINS] [--matrix-cells MATRIX_CELLS]
[--matrix-cell MATRIX_CELL] [--context7-id CONTEXT7_ID]
[--baseline-arm BASELINE_ARM] [--stats-correction {holm,bh}] [--judge]
[--judge-model JUDGE_MODEL]
[--judge-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}]
[--top-k TOP_K] [--rerank-threshold RERANK_THRESHOLD]
[--max-iterations MAX_ITERATIONS] [--max-tokens MAX_TOKENS]
[--grounded-from-topics] [--judge-temperature JUDGE_TEMPERATURE]
[--temperature TEMPERATURE] [--seed SEED] [--judge-panel JUDGE_PANEL]
[--concurrency CONCURRENCY] [--out-json OUT_JSON] [--out-md OUT_MD]
options:
-h, --help show this help message and exit
--product PRODUCT Product name (e.g., oneTBB)
--questions QUESTIONS
Path to questions JSON file
--arms ARMS Comma-separated arm specs. Examples: 'baseline,docs', 'baseline,docs:local
:./docs,profile:data/agent_profiles/concise_expert.md',
'baseline,mcp:http=https://mcp.context7.com/mcp,skill:/path/to/skills-
repo/skills/<name>'
--model MODEL LLM for answering
--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}
--harness HARNESS Execution harness label stamped into output (default: arms-runner)
--plugins PLUGINS Comma-separated plugin refs, e.g. 'plugin:caveman:ultra' or
'plugin:truncate:800'
--matrix-cells MATRIX_CELLS
JSON file containing matrix.cells descriptors
--matrix-cell MATRIX_CELL
Named matrix cell to run from --matrix-cells
--context7-id CONTEXT7_ID
Explicit library id for doc/MCP arms (skips resolution)
--baseline-arm BASELINE_ARM
Arm name used as the delta baseline (default: baseline)
--stats-correction {holm,bh}
Multiple-comparison correction across treatment arms: 'holm' (Holm-
Bonferroni, controls family-wise error, default) or 'bh' (Benjamini-
Hochberg, controls false discovery rate)
--judge Also score each arm with the LLM-as-judge
--judge-model JUDGE_MODEL
--judge-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}
--top-k TOP_K Docs to retrieve before reranking, for doc/MCP arms (default: 5)
--rerank-threshold RERANK_THRESHOLD
--max-iterations MAX_ITERATIONS
Max tool-call rounds for agentic arms (agent:/skill-agent:, default: 6)
--max-tokens MAX_TOKENS
Output-token ceiling per answer. Raise for reasoning models, which can
spend the provider default entirely on reasoning and return empty text
--grounded-from-topics
Let the judge use a question's expected_topics as ground truth when they
are substantive answer statements, enabling the factual_grounding
dimension for curated golden sets. Keyword-style topics are ignored
--judge-temperature JUDGE_TEMPERATURE
Sampling temperature for the JUDGE. Separate from --temperature, which
shapes answer generation; grading is its own sampled call and its own
noise source.
--temperature TEMPERATURE
Sampling temperature for answer generation. Omitted by default (provider
default applies). Pin it to reduce run-to-run score variance.
--seed SEED Sampling seed for answer generation, where the provider honours one. Best-
effort reproducibility.
--judge-panel JUDGE_PANEL
Comma-separated judge roles (e.g.
'technical_expert,developer_advocate,doc_reviewer'). When set, --judge
scores each answer with a panel and records grader spread and agreement
instead of one opinion.
--concurrency CONCURRENCY
--out-json OUT_JSON Output JSON path (default: results/arms/{product}.json)
--out-md OUT_MD Output Markdown report path (default: results/arms/{product}.md)
dokimi arms matrix-run
usage: dokimi arms matrix-run [-h] --product PRODUCT --questions QUESTIONS --arms ARMS
[--model MODEL]
[--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}]
[--harness HARNESS] [--plugins PLUGINS] --matrix-cells MATRIX_CELLS
[--context7-id CONTEXT7_ID] [--baseline-arm BASELINE_ARM]
[--stats-correction {holm,bh}] [--judge] [--judge-model JUDGE_MODEL]
[--judge-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}]
[--top-k TOP_K] [--rerank-threshold RERANK_THRESHOLD]
[--max-iterations MAX_ITERATIONS] [--max-tokens MAX_TOKENS]
[--grounded-from-topics] [--judge-temperature JUDGE_TEMPERATURE]
[--temperature TEMPERATURE] [--seed SEED]
[--judge-panel JUDGE_PANEL] [--concurrency CONCURRENCY]
[--out-dir OUT_DIR] [--out-json OUT_JSON] [--out-md OUT_MD]
[--no-plugin-deltas]
options:
-h, --help show this help message and exit
--product PRODUCT Product name (e.g., oneTBB)
--questions QUESTIONS
Path to questions JSON file
--arms ARMS Comma-separated arm specs. Examples: 'baseline,docs', 'baseline,docs:local
:./docs,profile:data/agent_profiles/concise_expert.md',
'baseline,mcp:http=https://mcp.context7.com/mcp,skill:/path/to/skills-
repo/skills/<name>'
--model MODEL LLM for answering
--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}
--harness HARNESS Execution harness label stamped into output (default: arms-runner)
--plugins PLUGINS Comma-separated plugin refs, e.g. 'plugin:caveman:ultra' or
'plugin:truncate:800'
--matrix-cells MATRIX_CELLS
JSON file containing matrix.cells descriptors
--context7-id CONTEXT7_ID
Explicit library id for doc/MCP arms (skips resolution)
--baseline-arm BASELINE_ARM
Arm name used as the delta baseline (default: baseline)
--stats-correction {holm,bh}
Multiple-comparison correction across treatment arms: 'holm' (Holm-
Bonferroni, controls family-wise error, default) or 'bh' (Benjamini-
Hochberg, controls false discovery rate)
--judge Also score each arm with the LLM-as-judge
--judge-model JUDGE_MODEL
--judge-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}
--top-k TOP_K Docs to retrieve before reranking, for doc/MCP arms (default: 5)
--rerank-threshold RERANK_THRESHOLD
--max-iterations MAX_ITERATIONS
Max tool-call rounds for agentic arms (agent:/skill-agent:, default: 6)
--max-tokens MAX_TOKENS
Output-token ceiling per answer. Raise for reasoning models, which can
spend the provider default entirely on reasoning and return empty text
--grounded-from-topics
Let the judge use a question's expected_topics as ground truth when they
are substantive answer statements, enabling the factual_grounding
dimension for curated golden sets. Keyword-style topics are ignored
--judge-temperature JUDGE_TEMPERATURE
Sampling temperature for the JUDGE. Separate from --temperature, which
shapes answer generation; grading is its own sampled call and its own
noise source.
--temperature TEMPERATURE
Sampling temperature for answer generation. Omitted by default (provider
default applies). Pin it to reduce run-to-run score variance.
--seed SEED Sampling seed for answer generation, where the provider honours one. Best-
effort reproducibility.
--judge-panel JUDGE_PANEL
Comma-separated judge roles (e.g.
'technical_expert,developer_advocate,doc_reviewer'). When set, --judge
scores each answer with a panel and records grader spread and agreement
instead of one opinion.
--concurrency CONCURRENCY
--out-dir OUT_DIR Directory for per-cell artifacts (default: results/arms/{product}-matrix)
--out-json OUT_JSON Matrix rollup JSON path
--out-md OUT_MD Matrix rollup Markdown path
--no-plugin-deltas Do not compute paired plugin_delta artifacts
dokimi arms plugin-delta
usage: dokimi arms plugin-delta [-h] --baseline-json BASELINE_JSON --plugin-json PLUGIN_JSON
[--out-json OUT_JSON] [--out-md OUT_MD]
options:
-h, --help show this help message and exit
--baseline-json BASELINE_JSON
arms.v1 artifact for the baseline/no-plugin cell
--plugin-json PLUGIN_JSON
arms.v1 artifact for the plugin-enabled cell
--out-json OUT_JSON
--out-md OUT_MD