Skip to content

dokimi evaluate

usage: dokimi evaluate [-h] --product PRODUCT [--repo REPO] [--description DESCRIPTION]
                       [--output-dir OUTPUT_DIR] [--custom-questions CUSTOM_QUESTIONS]
                       [--questions-root QUESTIONS_ROOT] [--model MODEL]
                       [--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}]
                       [--judge-model JUDGE_MODEL]
                       [--judge-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}]
                       [--judge-temperature JUDGE_TEMPERATURE] [--personas-count PERSONAS_COUNT]
                       [--questions-per-topic QUESTIONS_PER_TOPIC] [--top-k TOP_K]
                       [--rerank-threshold RERANK_THRESHOLD] [--debug-retrieval]
                       [--doc-source DOC_SOURCE] [--context7-id CONTEXT7_ID]

options:
  -h, --help            show this help message and exit
  --product PRODUCT     Product name (e.g., oneDNN)
  --repo REPO           GitHub repo (e.g., oneapi-src/oneDNN). Optional if --description is given.
  --description DESCRIPTION
                        Plain-text product description (used when --repo is not available).
  --output-dir OUTPUT_DIR
                        Base output directory
  --custom-questions CUSTOM_QUESTIONS
                        Optional: path to manual questions JSON
  --questions-root QUESTIONS_ROOT
                        Directory where the golden question set <product>_golden.json is looked
                        up, for example a checkout of the evaluation-content repository:
                        --questions-root /path/to/content-repo/questions
  --model MODEL         LLM model for generation/answering
  --provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}
  --judge-model JUDGE_MODEL
                        LLM model for evaluation
  --judge-provider {openai,anthropic,google,google-vertex,openrouter,amazon-bedrock}
  --judge-temperature JUDGE_TEMPERATURE
                        Judge sampling temperature, recorded as judge_temperature in the eval
                        output
  --personas-count PERSONAS_COUNT
                        Target number of personas
  --questions-per-topic QUESTIONS_PER_TOPIC
                        Questions per topic per persona
  --top-k TOP_K         Docs to retrieve before reranking (default from config/products.yaml
                        retrieval.top_k)
  --rerank-threshold RERANK_THRESHOLD
                        Min relevance score
  --debug-retrieval     Include retrieval metadata
  --doc-source DOC_SOURCE
                        Documentation source: 'context7' (default), 'local:<path>', 'url:<url>'
  --context7-id CONTEXT7_ID
                        Explicit Context7 library ID (e.g., 'intel/mkl-dnn'). Overrides auto-
                        resolution.