dokimi report

usage: dokimi report [-h] {eval,generate,model-compare} ...

positional arguments:
  {eval,generate,model-compare}
    eval                Generate quality report from eval JSON (dynamic vs static breakdown)
    generate            Generate comprehensive report from eval results
    model-compare       Compare multiple model runs over a common question set and write a
                        Markdown report

options:
  -h, --help            show this help message and exit

dokimi report eval

usage: dokimi report eval [-h] --product PRODUCT --eval EVAL [--run-id RUN_ID] [--out OUT]

options:
  -h, --help         show this help message and exit
  --product PRODUCT  Product name (e.g., oneDAL)
  --eval EVAL        Path to eval JSON file
  --run-id RUN_ID    Run tag (e.g. gpt4o). Auto-sets output path.
  --out OUT          Output .md file (default: results/{product}/reports/{product}_full.md)

dokimi report generate

usage: dokimi report generate [-h] --product PRODUCT --eval EVAL --questions QUESTIONS
                              [--output OUTPUT] [--format {markdown,json}]

options:
  -h, --help            show this help message and exit
  --product PRODUCT     Product name (e.g., oneTBB)
  --eval EVAL           Path to eval JSON file
  --questions QUESTIONS
                        Path to questions JSON file
  --output OUTPUT       Output file (default: reports/{product}.md)
  --format {markdown,json}
                        Output format

dokimi report model-compare

usage: dokimi report model-compare [-h] [--regular-runs REGULAR_RUNS [REGULAR_RUNS ...]]
                                   [--golden-runs GOLDEN_RUNS [GOLDEN_RUNS ...]] --run-ids RUN_IDS
                                   --out OUT [--treatment-arm TREATMENT_ARM]
                                   [--stats-correction {holm,bh}] [--pool-replicates]

options:
  -h, --help            show this help message and exit
  --regular-runs REGULAR_RUNS [REGULAR_RUNS ...]
                        Paths to regular-question arms JSON files (one per model, same order as
                        --run-ids)
  --golden-runs GOLDEN_RUNS [GOLDEN_RUNS ...]
                        Paths to golden-question arms JSON files (one per model, same order as
                        --run-ids)
  --run-ids RUN_IDS     Comma-separated run labels (e.g. sonnet46,opus48)
  --out OUT             Output Markdown file path
  --treatment-arm TREATMENT_ARM
                        Name of the treatment arm to use as the scored context arm. Required when
                        a run contains more than one non-baseline arm.
  --stats-correction {holm,bh}
                        Multiple-comparison correction applied across the compared runs: holm
                        (family-wise error, default) or bh (false discovery rate).
  --pool-replicates     Average runs that share a configuration (same answer model/provider,
                        judge, sampling, harness, plugin set and arms) into one row per
                        configuration, instead of treating them as rival models. Required when any
                        configuration appears more than once.