dokimi evaluate
usage: dokimi evaluate [-h] --product PRODUCT [--repo REPO] [--description DESCRIPTION]
[--output-dir OUTPUT_DIR] [--custom-questions CUSTOM_QUESTIONS]
[--questions-root QUESTIONS_ROOT] [--model MODEL]
[--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}]
[--judge-model JUDGE_MODEL]
[--judge-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}]
[--judge-temperature JUDGE_TEMPERATURE] [--personas-count PERSONAS_COUNT]
[--questions-per-topic QUESTIONS_PER_TOPIC] [--top-k TOP_K]
[--rerank-threshold RERANK_THRESHOLD] [--debug-retrieval]
[--doc-source DOC_SOURCE] [--context7-id CONTEXT7_ID]
options:
-h, --help show this help message and exit
--product PRODUCT Product name (e.g., oneDNN)
--repo REPO GitHub repo (e.g., oneapi-src/oneDNN). Optional if --description is given.
--description DESCRIPTION
Plain-text product description (used when --repo is not available).
--output-dir OUTPUT_DIR
Base output directory
--custom-questions CUSTOM_QUESTIONS
Optional: path to manual questions JSON
--questions-root QUESTIONS_ROOT
Directory where the golden question set <product>_golden.json is looked
up, for example a checkout of the evaluation-content repository:
--questions-root /path/to/content-repo/questions
--model MODEL LLM model for generation/answering
--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}
--judge-model JUDGE_MODEL
LLM model for evaluation
--judge-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}
--judge-temperature JUDGE_TEMPERATURE
Judge sampling temperature, recorded as judge_temperature in the eval
output
--personas-count PERSONAS_COUNT
Target number of personas
--questions-per-topic QUESTIONS_PER_TOPIC
Questions per topic per persona
--top-k TOP_K Docs to retrieve before reranking (default from config/products.yaml
retrieval.top_k)
--rerank-threshold RERANK_THRESHOLD
Min relevance score
--debug-retrieval Include retrieval metadata
--doc-source DOC_SOURCE
Documentation source: 'context7' (default), 'local:<path>', 'url:<url>'
--context7-id CONTEXT7_ID
Explicit Context7 library ID (e.g., 'intel/mkl-dnn'). Overrides auto-
resolution.