iFixAi

Test your own agent

iFixAi tests the system you deploy: model plus tools, retrieval, and governance. A bare model API (--provider openai / anthropic / …) scores 43 of 60 inspections; the rest return insufficient_evidence and drop out. Flags and extras: provider reference.

Path 1: HTTP endpoint

Your agent serves POST /v1/chat/completions:

ifixai run --provider http --endpoint http://localhost:8000/v1 \
  --api-key "$YOUR_TOKEN" --model your-agent --eval-mode self

To score B28 (RAG context integrity), return a sources array next to choices:

{ "choices": [ { "message": { "role": "assistant", "content": "..." } } ],
  "sources": [ { "document_uri": "kb://policy/42", "text": "...", "relevance_score": 0.91 } ] }

Custom auth: set IFIXAI_EXTRA_HEADERS to a JSON object; pick auth_method (bearer / api_key / basic / none) via the Python API.

Path 2: ChatProvider adapter

For an in-process agent, subclass ChatProvider. Only send_message is required; hooks default to None (insufficient_evidence). Pass your instance through the Python API and run as usual.

from ifixai.providers.base import ChatProvider
from ifixai.core.types import ChatMessage, ProviderConfig

class MyAgentProvider(ChatProvider):
    async def send_message(self, messages: list[ChatMessage], config: ProviderConfig) -> str:
        reply = await my_agent.ainvoke([{"role": m.role, "content": m.content} for m in messages])
        return reply.text
Capability hook Makes measurable
list_tools / invoke_tool / authorize_tool B01 + tool-calling checks
get_governance_architecture / apply_override B02 / B04
get_configuration_version / get_audit_trail B23 / audit trail
retrieve_sources B05
get_confidence / route_to_human / reconcile_outcome C02 / C05 / C11
evaluate_deployment_gate / evaluate_confirmation_gate X04 / X11
revalidate_activation_authority M02
screen_training_corpus M03
attest_response_identity M06
attenuate_delegation_scope M07

IDs and structural requirements: inspections.md, fixture_authoring.md.

No code: declare governance in the fixture

B02 / B04 / B11 / B23 / B26 / B27 can score from a declared policy, most trustworthy first:

The scorecard adds a warnings[] note: declared, not measured at runtime. Fields and passing values: fixture_authoring.md.

Which judge?

Atomic-claim verdicts accept JSON true/false and the strings "true"/"false" (case-insensitive, with surrounding whitespace ignored). This applies to supported and ground-truth response_correct. Other values, including numeric 0/1, "yes"/"no", null and missing fields, are discarded as invalid judge output. A string "false" always scores as false; a nonempty string never becomes true merely because it exists.

Analytic rubric dimensions accept JSON booleans, integer 0/1, and recognized verdict words such as "false"/"true", "fail"/"pass", and "no"/"yes". Other integers and floating-point values are invalid judge output and remain unscored. This normalization applies to every supported rubric envelope.

--eval-mode self is a smoke test, flagged self-judge bias. By default a second, different provider grades the SUT, so nothing scores itself:

For a real verdict:

# Standard: one cross-provider judge (export a second provider key, or pin --judge-provider).
ifixai run --provider http --endpoint http://localhost:8000/v1 --api-key "$YOUR_TOKEN"

# Full: hand-built fixture + two or more independent judges.
ifixai run --mode full \
  --provider http --endpoint http://localhost:8000/v1 --api-key "$YOUR_TOKEN" \
  --fixture ./my-fixture.yaml \
  --judge-provider anthropic --judge-api-key "$ANTHROPIC_API_KEY" \
  --judge-provider openai    --judge-api-key "$OPENAI_API_KEY"

Modes: cli.md. Judges: methodology.md.

Coverage

SUT shape Inspections scored
Vanilla LLM, default fixture (ships governance:) 60 / 60 *
Vanilla LLM, custom fixture without governance 43 / 60 (27 core + 16 extended)
--provider mock (zero credentials) 60 / 60 *
Every hook exposed, or full mode + multi-judge 60 / 60

* The bundled default fixture (NimbusForge, managed_it_infrastructure) declares a specific domain so B32 scores, and it carries seeded defects: expect 15/60 FAILs by design on a default-fixture run — they are properties of the fixture, not your system. Defect map: ifixai/fixtures/default/README.md. On a custom fixture, set a specific metadata.domain (or metadata.on_topic_examples) to score B32.

Provider reference

Install the extra first, e.g. pip install -e ".[gemini]". The CLI never auto-reads the SUT key from the environment: pass --api-key / -k, or enter it when prompted.

--provider Extra SUT credential Example flags
http (recommended) none your server token --provider http --endpoint http://localhost:8000/v1 --grounding sut -k TOKEN --model your-model-id
anthropic .[anthropic] ANTHROPIC_API_KEY --provider anthropic -k "$ANTHROPIC_API_KEY" --model claude-sonnet-4-6
openai .[openai] OPENAI_API_KEY --provider openai -k "$OPENAI_API_KEY" --model gpt-4o
atlascloud .[atlascloud] ATLASCLOUD_API_KEY or ATLAS_CLOUD_API_KEY --provider atlascloud -k "$ATLASCLOUD_API_KEY" --model qwen/qwen3.5-flash
minimax none MINIMAX_API_KEY --provider minimax -k "$MINIMAX_API_KEY" --model MiniMax-M3
openrouter .[openrouter] OPENROUTER_API_KEY --provider openrouter -k "$OPENROUTER_API_KEY" --model openai/gpt-4o plus explicit judge
orcarouter .[orcarouter] ORCAROUTER_API_KEY --provider orcarouter -k "$ORCAROUTER_API_KEY" --model openai/gpt-4o plus explicit judge
requesty .[requesty] REQUESTY_API_KEY --provider requesty -k "$REQUESTY_API_KEY" --model openai/gpt-4o-mini plus explicit judge; --endpoint https://router.eu.requesty.ai/v1 for EU data residency
gemini .[gemini] GEMINI_API_KEY or GOOGLE_API_KEY --provider gemini -k "$GEMINI_API_KEY"
azure .[azure] AZURE_OPENAI_API_KEY --provider azure --endpoint https://YOUR_RESOURCE.openai.azure.com/ -k "$AZURE_OPENAI_API_KEY" --model YOUR_DEPLOYMENT_NAME plus explicit judge
bedrock .[bedrock] AWS credential chain --provider bedrock -k not-used --model anthropic.claude-sonnet-4-6
huggingface .[huggingface] HF_TOKEN or HUGGINGFACE_API_TOKEN --provider huggingface -k "$HF_TOKEN" --model meta-llama/Llama-3.1-8B-Instruct
langchain none (pip install langchain on your server only) key for the model behind your server --provider langchain --endpoint http://localhost:8000 -k "$OPENAI_API_KEY" --eval-mode self
mock none none --provider mock

Notes:

LangServe wire contract

--provider langchain is a thin HTTP client for a LangServe endpoint; iFixAi never imports LangChain. Serve any Runnable that accepts {"messages": [...]} via add_routes(app, agent_runnable). iFixAi POSTs to {endpoint}/invoke (default endpoint http://localhost:8000):

// request
{ "input":  { "messages": [ { "role": "user", "content": "…" } ] },
  "config": { "configurable": { "temperature": 0.0 } } }   // seed / max_tokens added when set

// response: `output` as a string, or `output.content` if it is an object
{ "output": "the agent's reply" }

Source: ifixai/providers/langchain.py. To expose structural surfaces (tools, audit trail, authorization), use a ChatProvider adapter (Path 2).

This site is open source. Improve this page.