iFixAi

CLI reference

Every ifixai command and ifixai run flag. Guided first run: get-started.md.

Commands

ifixai init                  # check env for provider keys, suggest a first run
ifixai run                   # run inspections (Standard or Full mode)
ifixai run --fixture FILE    # custom fixture (YAML or JSON)
ifixai list tests            # all 60 inspections (32 core + 28 extended)
ifixai list fixtures         # registered named fixtures
ifixai validate [FILE]       # per-test layout, or a fixture against schema.json
ifixai compare A B           # diff two scorecard reports
ifixai run -p openai -k "$OPENAI_API_KEY" -c DECEPTION   # example: one category

ifixai run flags

System under test (SUT)

Flag	Default	Does
--provider, -p	none	mock, openai, openrouter, orcarouter, requesty, anthropic, gemini, azure, bedrock, huggingface, http, langchain.
--api-key, -k	none	SUT API key. Always passed explicitly, never read from the environment.
--model, -m	provider default	Model identifier override.
--endpoint, -e	none	Endpoint URL (required for http and azure).
--system-prompt, -s	none	Custom system instructions sent before each inspection.
--grounding	sut	Governance context source: sut (baked-in), fixture (system prompt derived from fixture), none.
--sut-temperature	0.0	SUT sampling temperature. B22 needs 0 or --sut-seed.
--sut-seed	none	SUT sampling seed (recorded in the manifest either way).

Judge

Flag	Default	Does
--eval-mode	auto	deterministic, single, full, or self. See How a run is judged.
--judge-provider	none	Judge provider. Repeat >=2 times for a Full-mode ensemble.
--judge-api-key, --judge-model	none	Key(s) and model(s), paired with --judge-provider.
--judge-budget	0	Max judge LLM calls per run (0 = unlimited).

Fixture and governance

Flag	Default	Does
--fixture, -f	auto	Fixture name or YAML/JSON path.
--governance	none	GovernanceFixture YAML path; structural inspections score against your declared policies. See fixture_authoring.md.
--mode	standard	standard or full (hand-built fixture + >=2 distinct judge providers). Deprecated alias --profile (quick → standard).

Suite subset

Flag	Does
--strategic	Top 8 strategic tests only.
--test, -b	Test(s) by ID, repeatable: -b B01 -b B08.
--category, -c	Category name(s), case-insensitive, repeatable; beats --strategic. Names: FABRICATION, MANIPULATION, DECEPTION, UNPREDICTABILITY, OPACITY, SABOTAGE, SUBVERSION, CONCEALMENT, SANDBAGGING, INSUBORDINATION, USURPATION, SYSTEMIC_RISK, MISCALIBRATION, STAKEHOLDER_CONFLICT, PERCEPTION_GOVERNANCE, OVERSIGHT_ATROPHY, PERSISTENCE, IDENTITY_ATTESTATION, INFLUENCE, BALANCE_INTEGRITY, FRANKNESS_CORRECTNESS_LINK, GRADER_VALIDITY, BENCHMARK_CONTAMINATION, TRAINING_DISPOSITION_PROVENANCE, VULNERABLE_USER_CARE.

Output and reporting

Flag	Default	Does
--output, -o	./ifixai-results/	Report directory.
--format	both	json, markdown, or both.
--name, --version	provider name, 1.0	System name and version label in reports.
--min-score	0.85	Exit code 2 if overall score is below (CI gate).
--quiet, -q	off	Suppress banner and summary; stdout still carries scores.

Execution and reliability

Flag	Default	Does
--timeout, -t	30	Per-request timeout in seconds.
--concurrency, -j	5	Max in-flight LLM requests (1-20). Overrides IFIXAI_CONCURRENCY.
--no-parallel	off	Alias for --concurrency 1.
--dry-run	off	Print inspection and judge-call estimates, then exit.
--no-promo	off	Skip the Pro-version offer and start the run immediately. IFIXAI_NO_PROMO=1 skips it on every run.
--reliability-out	runs	Directory for manifest.json, one subdir per run.
--run-nonce	fresh	Replay-protection nonce (16 hex chars), recorded in the manifest.
IFIXAI_JUDGE_FALLBACKS (env)	packaged JSON	Path to the judge fallback-model chain. See Judge fallback models.
--holdout-seed, --b{12,14,28,29,30,32}-seed	fresh random	Pin, or set matching IFIXAI_*_SEED, to replay a run. See reproducibility.md.

Pro-version offer

The first interactive ifixai run on a machine opens with one question: whether you’d like to claim your free fast audit at https://www.ifixai.ai/. It’s shown once, whatever you answer (Ctrl-C included); later runs start straight away. The marker is a pro-offer-shown file in $XDG_CONFIG_HOME/ifixai (else ~/.config/ifixai): delete it to see the offer again.

Move with the arrow keys and press Enter:

• No (preselected, just press Enter): the run starts.
• Yes: the site opens in your browser, then you pick Continue the current run (preselected) or Abandon the run. Abandoning exits 0 before anything happens: no provider calls, no telemetry, no run directory.

Ctrl-C on either question aborts without starting the run. Where the arrow-key menu cannot run (NO_COLOR is set), the same options are listed with numbers to type instead.

To start immediately, pass --no-promo, or set IFIXAI_NO_PROMO=1 to skip it on every run:

ifixai run --provider openai --no-promo
export IFIXAI_NO_PROMO=1

The offer is never shown when stdin or stdout is not a terminal (pipes, scripts, agent-driven runs), in CI, or on --dry-run, so automated runs are never blocked by it. --quiet does not skip it; use --no-promo. A runner that allocates a pseudo-terminal without setting CI looks like a person at a keyboard: pass --no-promo (or set IFIXAI_NO_PROMO=1) there.

How a run is judged

The SUT (system under test) is the agent being graded via the SUT flags above; its key is never read from the environment. The judge grades the SUT’s answers; a citable grade needs a judge from a second, different provider.

Judge auto-pairing

With no --judge-* flags, iFixAi picks a judge from a different provider whose key it finds in the environment, never the SUT’s own. With only one credential it refuses unless you pass --eval-mode self. Preference order: testing-your-agent.md.

Evaluation modes (--eval-mode)

Mode	What it does
deterministic	Structural inspections only; no judge call.
single	One cross-provider judge (--judge-provider required).
full	Multi-judge ensemble (>=2 --judge-provider; Full mode only).
self	SUT grades itself; grade prints but is flagged, not citable.

Standard vs Full

Same 60 inspections either way. Standard: auto fixture (the bundled default is a seeded-defect demo — expect 15/60 FAILs by design; see ifixai/fixtures/default/README.md), one auto-paired judge. Full (--mode full): requires --fixture (the default fixture is refused) and >=2 judge providers; majority vote, tie-break fail > partial > pass.

# Standard: export a second provider key and the judge auto-pairs.
ifixai run --provider http --endpoint http://localhost:8000/v1 --api-key "$YOUR_TOKEN"

# Full: hand-built fixture, two independent judges.
ifixai run --mode full \
  --provider http --endpoint http://localhost:8000/v1 --api-key "$YOUR_TOKEN" \
  --fixture ./my-fixture.yaml \
  --judge-provider anthropic --judge-api-key "$ANTHROPIC_API_KEY" \
  --judge-provider openai    --judge-api-key "$OPENAI_API_KEY"

Judge design rationale: methodology.md.

Judge fallback models

When the judge provider itself errors — an OpenRouter 502/503, a timeout, a dropped connection — the probe is retried on the next model in a priority-ordered fallback chain instead of killing the run. If every model is down, that one probe drops as INCONCLUSIVE and the run continues. A malformed judge reply is not a provider fault and never switches models.

The chain is plain JSON. Defaults ship at ifixai/judge/judge_fallbacks.json (OpenRouter: Gemini 2.5 Flash → GPT-4o mini → Haiku 4.5 → DeepSeek V3.2 → Qwen3 235B → GLM-5.2). The order is cheapest-and-tersest first: a verbose model burns real credit before it fails. Override by dropping an ifixai.judge-fallbacks.json next to your run, or point IFIXAI_JUDGE_FALLBACKS at any path:

{
  "providers": {
    "openrouter": {
      "attempts_per_model": 2,
      "models": [{ "model": "z-ai/glm-5.2" }, { "model": "openai/gpt-4o-mini" }]
    }
  }
}

attempts_per_model is how many provider errors one model absorbs before the chain advances. An empty models list disables fallbacks; with no fallback declared, the configured judge keeps its full retry budget instead. Substitutions are counted under fallback_grades in the run’s judge stats, so you can see which model actually graded.

The SUT’s own --model is dropped from the chain: a fallback must never turn a declared cross-provider grade into silent self-judging. Pointing the configured judge at the SUT is still --eval-mode self, which is declared and flagged.

Cost and time controls, so a dead judge cannot drain a key or stall a run:

• Judge calls ask OpenRouter for no reasoning tokens. A hybrid-reasoning model asked for a verdict otherwise thinks out loud until the token ceiling cuts it off — billed in full, worthless as a verdict.
• A truncated reply retires that model immediately rather than re-buying the same overrun.
• A model that fails is retired for the rest of the run. The chain is not re-walked from the dead primary on every grade. Once all models are retired, remaining probes drop for free with no further calls.
• The scorecard reports substitute judge graded this run: and judge calls that failed and were retried: whenever either happened, so a grade always names its origin.

This site is open source. Improve this page.