Headless runs in CI

omnicoreagent run runs one instruction and exits. Nobody is there to approve a call or top up a budget, so you say up front how those are answered; the exit code names how the run ended, and the evidence is written to files. By the end of this page you will have an agent that opens a pull request from a CI job, the run’s record read back from its files, and an outcome recorded once the tests have passed. It needs only the core install: no server.

Every output on this page is what the command printed when it was run. The model’s wording, the ids and the token counts will differ on your run.

An agent file

The command loads a Python file that defines an agent variable or a create_agent() function, the same file omniserve run takes. This agent lists changes and may open a pull request, and its policy says a person approves pull requests:

# agent.py
import os

os.environ.setdefault("DATABASE_URL", "sqlite:///./runs.db")

from omnicoreagent import MemoryRouter, OmniCoreAgent, ToolRegistry
from omnicoreagent.governance import PolicyEffect, PolicyRule, build_default_policy

tools = ToolRegistry()

@tools.register_tool("list_changes", idempotent=True)
def list_changes(since: str) -> dict:
    """List the merged changes since a release tag."""
    return {"since": since, "changes": ["Add CSV export", "Fix login timeout", "Faster search"]}

@tools.register_tool("open_pull_request")
def open_pull_request(title: str, body: str) -> dict:
    """Open a pull request."""
    return {"number": 318, "title": title}

policy = build_default_policy("interactive-dev")
policy.rules.ask.append(
    PolicyRule(
        rule_id="ask_before_pull_requests",
        effect=PolicyEffect.ASK,
        capability="tool.local.call",
        target={"tool_name": "open_pull_request"},
        reason="A person approves pull requests.",
    )
)

agent = OmniCoreAgent(
    name="release_notes",
    system_instruction="You write release notes. Answer in plain text, in two sentences at most.",
    model_config={"provider": "openai", "model": "gpt-5.4-mini"},
    local_tools=tools,
    memory_router=MemoryRouter("sql"),
    agent_config={"governance_config": {"enabled": True, "policy": policy}},
)

The SQLite store keeps the run’s record after the command exits, so a later step can read it and record its outcome (SQLite needs pip install "omnicoreagent[postgres]", which brings SQLAlchemy). With the default in-memory store the record is gone when the command ends.

Run it

export LLM_API_KEY=...
omnicoreagent run --agent agent.py \
  --instruction "Write release notes for the changes since v1.2, then open a pull request with them." \
  --tag nightly --provenance environment=ci --provenance build=4471 \
  --output-dir out
echo "exit code: $?"

status=awaiting_approval exit=3 run_id=run_2e2c93f291314a789492554e689b4483
exit code: 3

The model asked to open the pull request, the policy asked for a person, and by default (--approval-mode stop) the command leaves the run waiting and exits 3. out/result.json says what it is waiting for:

"pending": {
  "approvals": [
    {
      "approval_id": "approval_f031f81207eb4bcabe060d3c59fa99f8",
      "tool_call_id": "call_aEoMcrcIxWivLGt8b5AvD7AZ",
      "tool_name": "open_pull_request",
      "capability": "tool.local.call",
      "target": {
        "path": null,
        "host": null,
        "resource": null,
        "tool_name": "open_pull_request",
        "mcp_server": null
      },
      "arguments": {
        "title": "Release notes since v1.2",
        "body": "Release notes since v1.2:\n\n- Added CSV export for easier data portability.\n- Fixed a login timeout issue to improve session reliability.\n- Improved search performance for faster results."
      },
      "command": null,
      "risk_level": "low",
      "reason": "A person approves pull requests.",
      ...

A person can approve it later from any process that shares the store (resolve_approval, then resume: Approvals). In CI, decide instead. An approvals file approves pull requests and denies anything else the policy asks about:

{
  "default": "deny",
  "rules": [
    {"tool_name": "open_pull_request", "decision": "approve", "note": "release PRs are allowed in CI"}
  ]
}

omnicoreagent run --agent agent.py \
  --instruction "Write release notes for the changes since v1.2, then open a pull request with them." \
  --approval-mode scripted --approvals-file approvals.json \
  --timeout 300 --tag nightly --provenance environment=ci --provenance build=4472 \
  --output-dir out2 --json

{
  "schema": "omnicoreagent.headless_result/v1",
  "status": "success",
  "exit_code": 0,
  "run_id": "run_0923cebcb71c4b68a4094b82e465298d",
  "session_id": "omni_core_agent_release_notes_6ad9dc78",
  "response": "Release notes since v1.2: added CSV export, fixed login timeout handling, and improved search performance.  \nOpened pull request #318 titled \u201cRelease notes for changes since v1.2.\u201d",
  "termination_reason": "stop",
  "error": null,
  "trace_ids": [
    "trace_e43c0f0aa5904040840f5fc5c5cb49e6",
    "trace_7b97624dbfe14ecca8772ef74179e737"
  ],
  "approval_mode": "scripted",
  "budget_mode": "stop",
  "cli_decisions": [
    {
      "kind": "approval",
      "approval_id": "approval_7254bc648d1d401c9adc6ff85d31d5ff",
      "tool_name": "open_pull_request",
      "capability": "tool.local.call",
      "decision": "approve",
      "rule": 0,
      "note": "release PRs are allowed in CI"
    }
  ],
  "pending": null,
  "usage": {
    "requests": 3,
    "request_tokens": 4904,
    "response_tokens": 133,
    "total_tokens": 5037
  },
  "duration_seconds": 21.545,
  "omnicoreagent_version": "0.5.0",
  "evidence_error": null
}
status=success exit=0 run_id=run_0923cebcb71c4b68a4094b82e465298d

The run paused at the pull request, the CLI approved it by rule 0, and the run resumed and finished: two traces, one run. The last line is on stderr. usage is the whole run’s, both segments counted (three model calls: two before the pause, one after), the same totals trajectory.json holds.

How it works

Load the agent, run the instruction

The run is tagged headless, approval-mode:<mode> and budget-mode:<mode>, plus your --tags, and recorded as entered from the headless surface. MCP servers the agent names are connected first.

Answer each pause by the mode you chose

When the run pauses for approvals, each one is decided by --approval-mode through the same resolve_approval a person would use, with the approver omnicoreagent-cli and a note naming the mode or rule, and the run is resumed. This repeats up to --max-approval-rounds (20) times. When a budget runs out, --budget-mode deny denies the top-up through deny_budget; the CLI never grants money.

One deadline for all of it

--timeout covers the whole run, pauses and resumes included. When it passes, the run is stopped and recorded as timeout, not cancelled.

Write the evidence, exit with the state

result.json and trajectory.json go to --output-dir; the answer (or with --json, the whole result.json) to stdout; one status line to stderr. Reading the evidence never changes the outcome: if the trajectory cannot be read, evidence_error says why.

Approval modes

`--approval-mode`	Each approval is
`stop` (default)	Left waiting; the command exits 3
`allow`	Approved
`deny`	Denied; the model is told, and carries on
`scripted`	Decided by the first matching rule in `--approvals-file`, else by its `default`

A rule matches on tool_name, capability, or both, and has a decision (approve or deny) and an optional note. The file’s default is approve, deny (when omitted) or stop.

Exit codes

Code	The run
0	succeeded
1	failed, errored, was blocked by the guardrail, or ended because its budget was denied
2	never started: a usage error or a bad agent file
3	is waiting for an approval
4	is waiting for a budget
5	timed out
6	was interrupted: `agent.interrupt`, or Ctrl-C / SIGTERM, which stops the run at its next step and still writes `result.json` and `trajectory.json` (a second Ctrl-C ends the run at once, though the process exits only when a blocking, synchronous tool call in progress returns). A Ctrl-C before the run has started (while the agent loads) exits 6 too, saying nothing ran, with no files to write

What it writes

File	Holds
`result.json`	`schema` (`omnicoreagent.headless_result/v1`), `status`, `exit_code`, `run_id`, `session_id`, `response`, `termination_reason`, `error`, `trace_ids`, the two modes, `cli_decisions`, `pending` (the approvals or budget request it stopped on), `usage` (the whole run’s, every segment counted), `duration_seconds`, `omnicoreagent_version`, `evidence_error`
`trajectory.json`	The whole run across its pauses, as `get_run_trajectory(run_id)` returns it: `segments` (each with its full trajectory), `totals`, `approvals`, `tool_calls`, `usage`

The traces themselves are also in the agent’s telemetry store (./workspace/telemetry/ by default), and the run’s record in its memory store.

Read the run afterwards

trajectory.json needs no agent to read:

# read_run.py
import json

run = json.load(open("out2/trajectory.json"))
cost = run["totals"]["estimated_cost_usd"]  # None when nothing was priced (a run cut off before any answer)
print(run["status"], "| segments:", len(run["segments"]), "| tokens:", run["totals"]["tokens"]["total"],
      "| cost:", round(cost, 6) if cost is not None else "unknown")
for approval in run["approvals"]:
    print("approval:", approval["tool_name"], approval["status"], approval["decision"],
          "by", approval["approver"], "-", approval["note"])
first = run["segments"][0]["trajectory"]
print("tags:", first["tags"])
print("provenance:", {k: v for k, v in first["provenance"].items() if v})
for segment in run["segments"]:
    for step in segment["trajectory"]["steps"]:
        for call in step["tool_calls"]:
            print(f"step {step['step']}:", call["tool_name"], call["outcome"])

completed | segments: 2 | tokens: 5037 | cost: 0.003585
approval: open_pull_request used approve by omnicoreagent-cli - release PRs are allowed in CI
tags: ['nightly', 'headless', 'approval-mode:scripted', 'budget-mode:stop']
provenance: {'source': 'omnicoreagent', 'environment': 'ci', 'external_ids': {'build': '4472'}}
step 1: list_changes success
step 2: open_pull_request awaiting_approval
step 2: open_pull_request success

The approval’s status is used: its decision was applied when the run resumed. decision says which decision it was, approve or deny; a denied approval is used too once the run resumes, and its call was not made. --provenance keys the trace knows (environment, trial_id, case_id, evaluation_id, deployment_id, application_version, adapter, …) are set directly; any other key, like build, goes under external_ids. What every field means is in Read a run.

Record the outcome after the tests

The run said it opened a pull request; whether the change was any good is known when the tests run. A later step records it on the run, with no model key needed:

# record_outcome.py
import asyncio
import json
import sys

from agent import agent

async def main(result_file, passed):
    run_id = json.load(open(result_file))["run_id"]
    outcome = await agent.record_outcome(
        run_id,
        source="ci",
        reward=1.0 if passed else 0.0,
        label="tests_passed" if passed else "tests_failed",
        detail={"build": "4472"},
    )
    print(run_id, outcome["label"], outcome["reward"])
    await agent.cleanup()

asyncio.run(main(sys.argv[1], sys.argv[2] == "passed"))

python record_outcome.py out2/result.json passed

run_0923cebcb71c4b68a4094b82e465298d tests_passed 1.0

The run is now one training record, both segments in it, with its outcome:

# training_record.py
import asyncio
import json
from agent import agent

async def main():
    run_id = json.load(open("out2/result.json"))["run_id"]
    [record] = await agent.training_records(run_id=run_id)
    print("trace_ids:", len(record["trace_ids"]), "| outcomes:", [(o["source"], o["label"]) for o in record["outcomes"]])
    for step in record["steps"]:
        print("segment", step["segment"], "step", step["step"], "resumed" if step.get("resumed") else "model turn",
              [(c["name"], c["outcome"]) for c in step["tool_calls"]])
    await agent.cleanup()

asyncio.run(main())

trace_ids: 2 | outcomes: [('ci', 'tests_passed')]
segment 0 step 1 model turn [('list_changes', 'success')]
segment 0 step 2 model turn [('open_pull_request', 'awaiting_approval')]
segment 1 step 2 resumed [('open_pull_request', 'success')]
segment 1 step 3 model turn []

In a CI job

A GitHub Actions job: run the agent, keep its evidence whatever happened, run the tests, record the outcome. The steps share one machine, so the SQLite file and the workspace carry from step to step; across jobs or machines, point DATABASE_URL at a shared database and keep the workspace with it.

jobs:
  release-notes:
    runs-on: ubuntu-latest
    env:
      LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install "omnicoreagent[postgres]"

      - name: Run the agent
        id: agent
        run: |
          set +e
          omnicoreagent run --agent agent.py \
            --instruction-file .github/release-notes.md \
            --approval-mode scripted --approvals-file approvals.json \
            --budget-mode deny --timeout 900 \
            --tag ci --provenance environment=ci --provenance build=${{ github.run_id }} \
            --output-dir out
          code=$?
          echo "exit_code=$code" >> "$GITHUB_OUTPUT"
          exit $code

      - name: Keep the evidence
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: agent-run
          path: out/

      - name: Tests
        id: tests
        run: pytest

      - name: Record the outcome
        if: always() && steps.agent.outputs.exit_code == '0'
        run: python record_outcome.py out/result.json ${{ steps.tests.outcome == 'success' && 'passed' || 'failed' }}

--instruction-file reads the instruction from a file (or - for stdin), which keeps a long one out of the workflow. With --budget-mode deny, a run that runs out of budget ends (exit 1) instead of waiting (exit 4) for a top-up nobody will give.

Options

Flag	Default	What
`--agent`, `-a`	required	The agent file
`--instruction`, `-i` / `--instruction-file`, `-f`	one is required	The instruction, or a file with it (`-` for stdin)
`--session-id`, `--run-id`	new	Run in this session, with this run ID
`--tag`		A label on the trace; repeatable
`--provenance KEY=VALUE`		Provenance on the trace; repeatable
`--approval-mode`	`stop`	`stop`, `allow`, `deny`, `scripted`
`--approvals-file`		The rules for `scripted`
`--budget-mode`	`stop`	`stop` (exit 4) or `deny` (end the run)
`--timeout`	none	Seconds for the whole run
`--max-approval-rounds`	20	Most pause-and-resume cycles
`--output-dir`, `-o`		Write `result.json` and `trajectory.json` here
`--json`		Print `result.json` to stdout instead of the answer

The command line reference is generated from the code.

When things go wrong

Exit 5: run exceeded its deadline

status=timeout exit=5 run_id=run_40b361d1d5974bb08421e1b1e083550b error=run exceeded its deadline

--timeout 2 against a model call that takes several seconds. The run is recorded as timeout in its trace and its record, and result.json has "error": "run exceeded its deadline". Give it the time a real run needs. The deadline starts once the agent is set up (its tools, MCP servers and model client loaded), so a cold start does not spend it.

Exit 1: the budget was denied

With --budget-mode deny and a request budget of one model call:

This run stopped because the request budget is exhausted: The request budget for model_calls is exhausted: 1 of 1 used, 1 more needed
status=error exit=1 run_id=run_40377197700e44f496e0126339dcf553

termination_reason is budget_exhausted, and cli_decisions holds the denial. Raise the budget (Budgets).

Exit 1: LLM_API_KEY not found

status=error exit=1 run_id=run_be9dcaaf35a34657b075e4a50607eb93 error=ValueError: LLM_API_KEY not found in environment variables

The model key is not in the step’s environment. In GitHub Actions, set it from a secret, as above.

Exit 2: the command did not start

Nothing ran; the message says why:

Error: Give exactly one of --instruction or --instruction-file
Error: The instruction is empty
Error: Agent file not found: missing.py
Error: --approval-mode scripted needs --approvals-file
Error: approval rule 0 needs a decision of approve or deny
Error: Invalid value for '--approval-mode': 'maybe' is not one of 'stop', 'allow', 'deny', 'scripted'.

An agent file that raises on import, or defines neither agent nor create_agent(), is exit 2 as well.

LookupError: No run … when recording the outcome

The later step cannot find the run: the agent keeps its records in memory (the default), or the step points at a different store. Give the agent a durable store, as agent.py does, and run both steps against it.

Read a run

Every field of the trajectory the command writes.

Outcomes and training records

What the outcome is for, and how to export runs.

Observability

Traces, stores and exporters.

Harbor and Terminal-Bench

The same headless run, on benchmark tasks.

Get Started

Make It Safe

Run It

See and Improve

Build

How It Works

Reference

HTTP API (OmniServe)

Releases

Headless runs in CI

Headless runs in CI

An agent file

Run it

How it works

Approval modes

Exit codes

What it writes

Read the run afterwards

Record the outcome after the tests

In a CI job

Options

When things go wrong

Next

Read a run

Outcomes and training records

Observability

Harbor and Terminal-Bench

​Headless runs in CI

​An agent file

​Run it

​How it works

​Approval modes

​Exit codes

​What it writes

​Read the run afterwards

​Record the outcome after the tests

​In a CI job

​Options

​When things go wrong

​Next

Read a run

Outcomes and training records

Observability

Harbor and Terminal-Bench

Headless runs in CI

An agent file

Run it

How it works

Approval modes

Exit codes

What it writes

Read the run afterwards

Record the outcome after the tests

In a CI job

Options

When things go wrong

Next