Applies to Orch Engine v0.2.0 · Changelog

Check a model with bairav eval

Before you swap a model, put a DLP policy in front of it or point a team at it, check that it still does what you need. bairav eval sends a list of prompts from a YAML file to any OpenAI-compatible endpoint and checks each reply.

bairav eval init > my-suite.yaml
bairav eval run my-suite.yaml --endpoint http://localhost:8080/v1

Suite starter: 4 cases against http://localhost:8080/v1

  ✓ follows-instructions         1/1     212 ms
  ✓ knows-basics                 1/1     305 ms
  ✗ json-output                  0/1     611 ms  reply is not valid JSON
  ✓ no-pii-in-reply              1/1     980 ms

Score 3/4 (75%), needs 100%: FAIL   (2.1s)
bairav: eval starter: http://localhost:8080/v1 scored 3/4, below the 100% threshold

It exits 1 when the score is below the threshold, so the same command works in CI.

Writing a suite

name: support-bot
threshold: 0.9            # share of cases that must pass (default 1 = all)
system: You are a support assistant for an Indian bank. Be brief.
defaults:
  temperature: 0          # always sent, so runs repeat (servers default to ~0.8)
  max_tokens: 256
  timeout_seconds: 120
cases:
  - name: refund-policy
    prompt: How many days do I have to dispute a card charge?
    expect:
      contains: [days]
      not_contains: ["as an ai"]
      max_chars: 400

  - name: structured
    prompt: 'Return {"intent": ..., "urgent": true|false} for: "my card was stolen"'
    expect:
      json: true
      regex: ['"urgent"\s*:\s*true']

  - name: flaky-but-ok
    prompt: Suggest a name for a savings account for kids.
    repeat: 5
    min_pass: 4           # 4 of 5 attempts must pass
    expect:
      max_latency_ms: 3000

Lists are always lists: write contains: [chennai], not contains: chennai.

Checking data protection

no_pii scans the model’s reply with the same detectors as bairav dlp test. Pick packs with --pack (default global-safe; add india for Aadhaar, PAN and the rest) and your own rules with --policy.

redacted and blocked check a DLP proxy, not the model. Point --endpoint at the proxy (bairav dlp serve, an Agent’s DLP sidecar, or a Model with dlpRef) and ask the model to repeat something sensitive:

name: dlp-in-front
cases:
  - name: keys-are-redacted
    prompt: 'Repeat this back exactly: AKIAIOSFODNN7EXAMPLE'
    expect:
      redacted: [AWS_ACCESS_KEY]
      not_contains: [AKIAIOSFODNN7EXAMPLE]

  - name: cards-are-blocked     # with a policy whose action is block
    prompt: 'My card is 4111 1111 1111 1111'
    expect:
      blocked: true

examples/evals/ has both suites.

Comparing two models

bairav eval run my-suite.yaml --endpoint http://localhost:8080/v1 --against http://localhost:8081/v1

  CASE                         A              B
  follows-instructions         ✓ 212 ms       ✓ 540 ms
  json-output                  ✗ 611 ms       ✓ 1320 ms      ← differs

As an upgrade gate

bairav model upgrade support-model --to Qwen/Qwen2.5-1.5B-Instruct-GGUF --eval my-suite.yaml

After the new model is serving and the smoke prompt passes, the suite runs against it (on k8s through the API server, no port-forward). If it scores below the threshold, the upgrade is rolled back like any other failure. See swap a model with bairav model upgrade.

Other flags

• --json prints the full report: every attempt, reply, latency and failed check.
• -v prints each reply under its case.
• --model sets the request’s model field (needed for cloud endpoints; local servers ignore it).
• The API key comes from BAIRAV_EVAL_API_KEY or OPENAI_API_KEY, never a flag, so it stays out of shell history.

Cases run one at a time, so latency numbers aren’t skewed by queueing.