dokimi novelty
usage: dokimi novelty [-h] {analyze,atoms,probe,shorten,routing,items,view} ...
positional arguments:
{analyze,atoms,probe,shorten,routing,items,view}
analyze Full pipeline: atoms → probes → categories → attribution → shortening
atoms Extract knowledge atoms and spans only
probe Run paired probes against a saved atom set
shorten Score shortening candidates on tokens/knowledge/value/protection/routing
routing Can the artifact be selected against competitors? (name+description)
items Item analysis across saved probe runs: which atoms separate the models
view Re-render the annotated artifact view
options:
-h, --help show this help message and exit
dokimi novelty analyze
usage: dokimi novelty analyze [-h] --artifact ARTIFACT [--max-atoms MAX_ATOMS] [--all-atoms]
[--budgets BUDGETS [BUDGETS ...]] [--cache-dir CACHE_DIR]
[--seed SEED] [--model MODEL]
[--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}]
[--judge-model JUDGE_MODEL]
[--judge-provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}]
[--extract-model EXTRACT_MODEL]
[--extract-provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}]
[--max-tokens MAX_TOKENS] [--arms ARMS]
[--placebo-text PLACEBO_TEXT] [--concurrency CONCURRENCY]
[--allow-self-judge] [--no-paraphrase] [--no-reversal]
[--max-usd MAX_USD] [--out-dir OUT_DIR] [--out-json OUT_JSON]
[--view-format {html,md}] [--no-classify] [--no-shorten]
[--no-confirm-carriers] [--no-verify]
[--tier {standard,study,triage}] [--estimate-only]
[--include-resources]
[--resource-roles {reference,pattern,eval,script,template,trigger,data,top,other} [{reference,pattern,eval,script,template,trigger,data,top,other} ...]]
[--max-file-words MAX_FILE_WORDS]
[--max-resource-words MAX_RESOURCE_WORDS]
options:
-h, --help show this help message and exit
--artifact ARTIFACT skill:<path to SKILL.md or its directory>, or text:<path>
--max-atoms MAX_ATOMS
Atom cap per artifact (default 120). The measurement is LLM-call bound:
atoms × probes × arms × (answer + grade). Unset by default so that passing
the default value explicitly still overrides a --tier cap
--all-atoms Extract every atom (opt-in: this is the expensive sweep)
--budgets BUDGETS [BUDGETS ...]
Target token fractions for shortening candidates
--cache-dir CACHE_DIR
Disk cache for retention adjudications
--seed SEED Seed for the shuffled control and random selection, so both are
reproducible across runs
--model MODEL The model under measurement — novelty is a property of the (artifact,
model) pair, not of the file
--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}
--judge-model JUDGE_MODEL
Grading model; must differ from --model (default: anthropic/claude-
sonnet-5-5, or openai/gpt-6.1-sol when --model is that judge). A --judge-
model without --judge-provider runs on --provider
--judge-provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}
Provider for the judge. Without --judge-model, the default judge that runs
on this provider; if none does, the run exits asking for --judge-model
--extract-model EXTRACT_MODEL
Extractor/adjudicator model (defaults to --judge-model, so the answering
model never writes its own answer key)
--extract-provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}
--max-tokens MAX_TOKENS
Output-token ceiling. Raise for reasoning models, which can spend the
provider default on reasoning and return empty text
--arms ARMS Arms to run: no_artifact,full[,placebo][,shuffled]
--placebo-text PLACEBO_TEXT
File of domain-mismatched text for the placebo control: it separates 'this
artifact carries knowledge' from 'this model reacts to any long context'
--concurrency CONCURRENCY
--allow-self-judge Let the answering model grade itself. Inflates scores; recorded as a
warning on the artifact
--no-paraphrase
--no-reversal
--max-usd MAX_USD Stop the run once its model calls have cost this much (USD, all roles
together; calls in flight can overshoot it). A call with no known price
also stops it; a reply replayed from DOKIMI_RESPONSE_CACHE counts $0, so a
stopped run can be resumed under the same cap. Default: no cap
--out-dir OUT_DIR
--out-json OUT_JSON
--view-format {html,md}
--no-classify Skip the judged content categories (cheaper; the artifact records the skip
as a warning)
--no-shorten
--no-confirm-carriers
Skip checking each atom against the span cited for it; the span views then
report the extractor's citation as given (0.19 of atoms in the study were
not recoverable from it)
--no-verify Skip the per-atom faithfulness check
--tier {standard,study,triage}
Named cost tier. The measurement is probe-budget bound (~4 model calls per
paired probe), so a tier is mostly a statement about atoms and probe
kinds: triage ~100 calls, study ~1000 on the same 1,700-word skill. Every
tier records the claims it forfeits into run_config and the artifact's
warnings. An explicit flag always wins over the tier
--estimate-only Print the projected model-call bill and exit without spending anything
--include-resources Measure the whole skill bundle, not just SKILL.md. In the 33-skill study
corpus 0.72 of all words live in bundled references/scripts/patterns
(worst skill: 0.99), so an entry-point-only run says little about a
progressive-disclosure skill. Costs proportionally more: the bill is ~4
LLM calls per paired probe
--resource-roles {reference,pattern,eval,script,template,trigger,data,top,other} [{reference,pattern,eval,script,template,trigger,data,top,other} ...]
Restrict bundled files to these roles (default: every role except eval)
--max-file-words MAX_FILE_WORDS
Skip a bundled file larger than this instead of truncating it; a cut
reference would be attributed as if complete
--max-resource-words MAX_RESOURCE_WORDS
Total budget across bundled files, taken in role order (reference,
pattern, eval, rest) and recorded per exclusion
dokimi novelty atoms
usage: dokimi novelty atoms [-h] --artifact ARTIFACT [--max-atoms MAX_ATOMS] [--all-atoms]
[--budgets BUDGETS [BUDGETS ...]] [--cache-dir CACHE_DIR]
[--seed SEED] [--model MODEL] [--extract-model EXTRACT_MODEL]
[--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}]
[--max-tokens MAX_TOKENS] [--no-verify] [--out OUT]
[--max-usd MAX_USD] [--include-resources]
[--resource-roles {reference,pattern,eval,script,template,trigger,data,top,other} [{reference,pattern,eval,script,template,trigger,data,top,other} ...]]
[--max-file-words MAX_FILE_WORDS]
[--max-resource-words MAX_RESOURCE_WORDS]
options:
-h, --help show this help message and exit
--artifact ARTIFACT skill:<path to SKILL.md or its directory>, or text:<path>
--max-atoms MAX_ATOMS
Atom cap per artifact (default 120). The measurement is LLM-call bound:
atoms × probes × arms × (answer + grade). Unset by default so that passing
the default value explicitly still overrides a --tier cap
--all-atoms Extract every atom (opt-in: this is the expensive sweep)
--budgets BUDGETS [BUDGETS ...]
Target token fractions for shortening candidates
--cache-dir CACHE_DIR
Disk cache for retention adjudications
--seed SEED Seed for the shuffled control and random selection, so both are
reproducible across runs
--model MODEL Extractor model when --extract-model is not given (default: claude-
sonnet-5-5, the default judge)
--extract-model EXTRACT_MODEL
--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}
--max-tokens MAX_TOKENS
--no-verify
--out OUT
--max-usd MAX_USD Stop the run once its model calls have cost this much (USD, all roles
together; calls in flight can overshoot it). A call with no known price
also stops it; a reply replayed from DOKIMI_RESPONSE_CACHE counts $0, so a
stopped run can be resumed under the same cap. Default: no cap
--include-resources Measure the whole skill bundle, not just SKILL.md. In the 33-skill study
corpus 0.72 of all words live in bundled references/scripts/patterns
(worst skill: 0.99), so an entry-point-only run says little about a
progressive-disclosure skill. Costs proportionally more: the bill is ~4
LLM calls per paired probe
--resource-roles {reference,pattern,eval,script,template,trigger,data,top,other} [{reference,pattern,eval,script,template,trigger,data,top,other} ...]
Restrict bundled files to these roles (default: every role except eval)
--max-file-words MAX_FILE_WORDS
Skip a bundled file larger than this instead of truncating it; a cut
reference would be attributed as if complete
--max-resource-words MAX_RESOURCE_WORDS
Total budget across bundled files, taken in role order (reference,
pattern, eval, rest) and recorded per exclusion
dokimi novelty probe
usage: dokimi novelty probe [-h] --artifact ARTIFACT [--max-atoms MAX_ATOMS] [--all-atoms]
[--budgets BUDGETS [BUDGETS ...]] [--cache-dir CACHE_DIR]
[--seed SEED] [--model MODEL]
[--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}]
[--judge-model JUDGE_MODEL]
[--judge-provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}]
[--extract-model EXTRACT_MODEL]
[--extract-provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}]
[--max-tokens MAX_TOKENS] [--arms ARMS] [--placebo-text PLACEBO_TEXT]
[--concurrency CONCURRENCY] [--allow-self-judge] [--no-paraphrase]
[--no-reversal] [--max-usd MAX_USD] --atoms ATOMS [--out OUT]
options:
-h, --help show this help message and exit
--artifact ARTIFACT skill:<path to SKILL.md or its directory>, or text:<path>
--max-atoms MAX_ATOMS
Atom cap per artifact (default 120). The measurement is LLM-call bound:
atoms × probes × arms × (answer + grade). Unset by default so that passing
the default value explicitly still overrides a --tier cap
--all-atoms Extract every atom (opt-in: this is the expensive sweep)
--budgets BUDGETS [BUDGETS ...]
Target token fractions for shortening candidates
--cache-dir CACHE_DIR
Disk cache for retention adjudications
--seed SEED Seed for the shuffled control and random selection, so both are
reproducible across runs
--model MODEL The model under measurement — novelty is a property of the (artifact,
model) pair, not of the file
--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}
--judge-model JUDGE_MODEL
Grading model; must differ from --model (default: anthropic/claude-
sonnet-5-5, or openai/gpt-6.1-sol when --model is that judge). A --judge-
model without --judge-provider runs on --provider
--judge-provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}
Provider for the judge. Without --judge-model, the default judge that runs
on this provider; if none does, the run exits asking for --judge-model
--extract-model EXTRACT_MODEL
Extractor/adjudicator model (defaults to --judge-model, so the answering
model never writes its own answer key)
--extract-provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}
--max-tokens MAX_TOKENS
Output-token ceiling. Raise for reasoning models, which can spend the
provider default on reasoning and return empty text
--arms ARMS Arms to run: no_artifact,full[,placebo][,shuffled]
--placebo-text PLACEBO_TEXT
File of domain-mismatched text for the placebo control: it separates 'this
artifact carries knowledge' from 'this model reacts to any long context'
--concurrency CONCURRENCY
--allow-self-judge Let the answering model grade itself. Inflates scores; recorded as a
warning on the artifact
--no-paraphrase
--no-reversal
--max-usd MAX_USD Stop the run once its model calls have cost this much (USD, all roles
together; calls in flight can overshoot it). A call with no known price
also stops it; a reply replayed from DOKIMI_RESPONSE_CACHE counts $0, so a
stopped run can be resumed under the same cap. Default: no cap
--atoms ATOMS atoms.json from `novelty atoms`
--out OUT
dokimi novelty shorten
usage: dokimi novelty shorten [-h] --artifact ARTIFACT [--max-atoms MAX_ATOMS] [--all-atoms]
[--budgets BUDGETS [BUDGETS ...]] [--cache-dir CACHE_DIR]
[--seed SEED] --atoms ATOMS [--model MODEL]
[--extract-model EXTRACT_MODEL]
[--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}]
[--max-tokens MAX_TOKENS] [--lexical-only] [--allow-unsafe]
[--out OUT] [--max-usd MAX_USD]
options:
-h, --help show this help message and exit
--artifact ARTIFACT skill:<path to SKILL.md or its directory>, or text:<path>
--max-atoms MAX_ATOMS
Atom cap per artifact (default 120). The measurement is LLM-call bound:
atoms × probes × arms × (answer + grade). Unset by default so that passing
the default value explicitly still overrides a --tier cap
--all-atoms Extract every atom (opt-in: this is the expensive sweep)
--budgets BUDGETS [BUDGETS ...]
Target token fractions for shortening candidates
--cache-dir CACHE_DIR
Disk cache for retention adjudications
--seed SEED Seed for the shuffled control and random selection, so both are
reproducible across runs
--atoms ATOMS atoms.json from `novelty atoms`
--model MODEL Retention adjudicator model (default: claude-sonnet-5-5, the default
judge)
--extract-model EXTRACT_MODEL
--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}
--max-tokens MAX_TOKENS
--lexical-only Run the retention oracle without an LLM adjudicator. Cheap and
reproducible, but blind to contradictions
--allow-unsafe Allow a best-candidate pick that drops protected spans
--out OUT Write the chosen candidate here
--max-usd MAX_USD Stop the run once its model calls have cost this much (USD, all roles
together; calls in flight can overshoot it). A call with no known price
also stops it; a reply replayed from DOKIMI_RESPONSE_CACHE counts $0, so a
stopped run can be resumed under the same cap. Default: no cap
dokimi novelty routing
usage: dokimi novelty routing [-h] --artifact ARTIFACT
[--competitors COMPETITORS [COMPETITORS ...]] [--catalog CATALOG]
[--queries QUERIES] [--queries-from-headings] [--model MODEL]
[--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}]
[--max-tokens MAX_TOKENS] [--seed SEED] [--out OUT]
[--max-usd MAX_USD]
options:
-h, --help show this help message and exit
--artifact ARTIFACT skill:<path> or text:<path> under test
--competitors COMPETITORS [COMPETITORS ...]
Other artifacts the selector sees. Selection among one candidate is not
routing: the sibling confusion is the failure mode (every `dpnp-*` skill
scored 0.0 lexically)
--catalog CATALOG Directory of skills to take the competitors from: every SKILL.md under it
except the artifact itself, i.e. the siblings a harness lists next to it.
Combines with --competitors
--queries QUERIES JSON list (or one per line) of user-shaped requests that should select
this artifact, written independently of it
--queries-from-headings
Derive queries from the artifact's own headings. Triage only: the artifact
then defines both the problem scope and its own relevance, and the result
is stamped as such
--model MODEL Also ask this model to pick from the same cards. Optional: without it only
the free lexical floor is computed
--provider {openai,anthropic,amazon-bedrock,google-vertex,openrouter,openai-codex}
--max-tokens MAX_TOKENS
--seed SEED Seeds the per-query card permutation, so a position-biased model scores
~1/N and the run is reproducible
--out OUT
--max-usd MAX_USD Stop the run once its model calls have cost this much (USD, all roles
together; calls in flight can overshoot it). A call with no known price
also stops it; a reply replayed from DOKIMI_RESPONSE_CACHE counts $0, so a
stopped run can be resumed under the same cap. Default: no cap
dokimi novelty items
usage: dokimi novelty items [-h] --probes PROBES [PROBES ...] [--arm ARM] [--out OUT]
options:
-h, --help show this help message and exit
--probes PROBES [PROBES ...]
probes.json files from `novelty probe`, one per model. Point-biserial
discrimination needs at least three, since the models are the respondents
--arm ARM Arm to score discrimination and check probe-kind agreement in (default:
the artifact arm)
--out OUT
dokimi novelty view
usage: dokimi novelty view [-h] --novelty NOVELTY --atoms ATOMS [--format {html,md}] [--out OUT]
options:
-h, --help show this help message and exit
--novelty NOVELTY novelty.json from `novelty analyze`
--atoms ATOMS atoms.json from `novelty atoms`
--format {html,md}
--out OUT