Changelog
Every release, with its notes, is also on GitHub Releases.0.6.0
October 2026 · Python 3.12+ Proved under load and failure, no new features. A support desk app ran at 100 concurrent users on a 2-core container, soaked for 30 minutes, and was hurt in 11 ways; everything it broke is fixed below (what broke, and why). Most code runs unchanged. What now behaves differently:- OmniServe admits 24 concurrent runs per process. Past that a request waits up to 5 s, then gets
503withRetry-After, nothing started.OMNICOREAGENT_SERVE_MAX_CONCURRENT_RUNS=0admits everything, as before. - OmniServe resumes runs nobody else will: those a dead process left behind, and approved runs no client resumed.
- Budgets move to new storage on first touch. Do not run 0.5.x and 0.6.0 against the same store at once. A custom memory store that implements budgets needs three new methods (below).
- A local trace archive writes segment files, not a file per trace. Archives written by 0.5.x are still read and pruned.
/prometheusrequest durations are a cumulative histogram, without the_avgline.
Fixed
-
The release includes the portable-evidence schema. The sdist listed only
*.pyfiles, and the wheel is built from it, so 0.5.x shipped withoutcore/telemetry/schemas/portable-execution-evidence.schema.json: validating or importing portable evidence from an installed release raisedFileNotFoundError. Every file in the package is now in the release, and a test holds that. Found by running the test suite against the built 0.6.0 wheel. -
A resumed run’s final summary covers the whole run. The
run_summaryon itsfinal_answerevent described the last segment only: a run that took two segments, two model calls and 24 tokens read as one call and 12 tokens. It now also carriesrun_summary["whole_run"]: every segment’s steps, model calls, tokens and cost,duration_ms(the time the segments ran) andelapsed_ms(from the first start, the wait for a person included).run_summaryitself is still the final segment’s. -
A resumed run’s trace no longer asks twice, or reports a pause as an error. The resumed
segment recorded an
approval_request_createdevent again, under a new id, for an approval the record already held; it is now recorded once, in the segment that paused (the decision is still anapproval_resolvedin the resumed one). A call waiting for a person now haserror: nullin the trajectory (its outcome was alreadyawaiting_approval), and an OTLP backend no longer sees a skipped span (waiting for approval, or stopped by a budget) as an error: its status is unset and it carries noerror.type. -
/prometheussays what runs, models, budgets and approvals did. A server under load showed only request counts. It now also counts runs finished by status and reason, runs resumed by trigger and cause (a client, the orphan sweep, a background retry), model calls, errors and retries by model, budget refusals, and approvals asked and decided (omniserve_runs_finished_total,omniserve_runs_resumed_total,omniserve_model_*_total,omniserve_budget_refused_total,omniserve_approvals_*_total). Labels are bounded sets; none carries a run id, session id or tool argument.OmniCoreAgent.resume()takes atrigger(defaultexplicit) that the supervisor sets tobackground_retry. -
/prometheusrequest-duration counts keep growing.omniserve_request_duration_seconds_countand_sumwere worked out from the last 1000 requests, so under load both stopped at 1000 andrate()read zero. They now count since the server started, as a Prometheus histogram:_bucket(withlebounds from 5 ms to 5 minutes),_sumand_count. The non-standard_avgline is gone; use_sum / _count. - The trace archive no longer leaves a lock file for every trace. Local workspace storage kept one lock file per path and never removed it, so the support desk’s archive made two files for each trace. In a server soak (2026-10-08) the Python heap was flat, but the container’s memory still rose about 2 MiB a minute: the rest was the kernel’s cache of the inodes and names of those files, which the container is charged for. A path now maps to one of 256 lock files, so the number of locks is fixed however many files are written. Old lock files are left alone and can be deleted. A write that worked also no longer looks for its temp file afterwards, which had left the kernel a remembered name for every body.
-
A background retry no longer repeats a side effect. A task whose
retry_onincludestimeoutorexceptionstarted a new attempt on a retry, and a new attempt wipes the run’s tool calls: a refund job that timed out after issuing the refund would issue it again. A retry after a timeout now resumes the same run (completed calls are not run again; a cut-off non-idempotent call isunknown). A failed run cannot be resumed, so it is retried from the start only if it made no non-idempotent call; otherwise it endsfailedwith an error saying it was not retried, andrun.metadata["side_effects"]lists the calls for a person to decide. -
A run that times out or fails after side effects says what already happened. In the
provider-hang rounds of the support desk chaos run, a refund was issued and the run then hit its
deadline on the next model call; the client saw only
timeout, and a client that started a new run could refund twice. The run’s record and itserrornow carryside_effects: each non-idempotent call that completed (success) or has anunknownoutcome, by tool name and call id, never its arguments. OmniServe returns it in the504of/run/sync, in a/run/syncanswer withstatus: error, in the SSEerrorandcompleteevents, and inGET /runs/{id}.resume(run_id)now also continues a run that endedtimeout(it did not before), so the documented way on is to resume: completed calls are not repeated. Afailedrun still cannot be resumed, and a new run starts over. -
A call that began and then timed out is
unknown, not a timeout. A plain tool runs in a thread that cannot be stopped; whentool_call_timeoutfired on a non-idempotent call that had already begun, its effect could land afterwards while the record saidtimeoutand the model told a customer “The refund was not issued” (the support desk chaos run). The call is nowunknownin the run record, the trace’stool_errorevent hasmay_still_complete: true, and the model receives anunknown_outcomeerror that says to check before acting again or telling the user it failed. A call that never began, an idempotent tool and anasynctool keeptimeout. -
An approved run no longer waits forever for a resume that never comes. At 100 users the
admission limit answered some
POST /runs/{id}/resumewith503, the client did not retry, and 119 runs sat inawaiting_approvalwith their approval alreadyapproved. The orphan sweep skipped every run waiting for a person. It now also resumes a run inawaiting_approvalorawaiting_budgetwhose every decision is made, that no live owner holds, and that has sat unresumed forOMNICOREAGENT_SERVE_ORPHAN_SWEEP_DECIDED_GRACE_SECONDS(30 by default). The claim is the same version-checked claim, and therun_resumedevent hastriggerorphan_sweepand acauseanddecisionnaming what was decided. A client resume that arrives with the sweep loses cleanly (409) and the tool runs once. A run with a decision still pending is left alone.claim_orphaned_runstakesdecided_grace_seconds. A sweep that fills every slot is followed by the next as soon as a slot frees, so a backlog is not resumed two runs per interval. The support desk load client now waits out a503withRetry-After, reports 503s seen, retried successes and give-ups apart, and gives the sweep its grace period before it judges a refund. -
A crowd of runs on one budget no longer fails. Every budget change was a read, a change in
Python and a versioned write of one JSON row per budget key, holding every in-flight run’s hold.
On the support desk ramp (100 concurrent runs on one application budget) the row conflicted
constantly, the retries ran out, and 46 runs failed with
Could not record the budget change, reported asprovider_errorwitherror: null; ten of them had already issued a refund. Each change is now one atomic operation in the store: a guardedUPDATE ... SET spent = spent + xin SQL (one row per key and meter, one row per hold), a Lua script in Redis, a filteredfind_one_and_updatein MongoDB, a lock-free in-process step in memory. The limit check and the hold are one step, so no more than the limit is ever held or spent. 200 concurrent runs on one key now finish with the exact total and no hold left. -
A charge for work already done never fails the run. If the store cannot take the charge for a
model call or the seconds a sandbox ran, it is tried again for up to 15 seconds and then kept as
unrecorded: on the run record (
unrecorded_charges), as abudget_charge_unrecordedtrace event and asunrecordedinbudget_statusandGET /runs/{run_id}/budget. A hold before a call can still refuse the call. Holds a run could not release are released when it ends. -
Redis waits for a free connection instead of raising
MaxConnectionsErrorwhen more than 20 operations are in flight. -
A provider 500 is retried, and
Retry-Afteris obeyed. Twelve 500s failed twelve runs in the support desk chaos run, and a 429 asking for 3 seconds was retried after about 1. Transient provider errors (429, 500, 502, 503, 504, 529, 408, timeouts, dropped connections) are retried with backoff and jitter;Retry-Afteris a floor (capped at 60 seconds), and a wait that would pass the run’s deadline is not started. Other 4xx errors still fail at once. -
A failed run’s record says why, under the right cause. Failed runs read
error: null, with the cause only in the trace, and an internal error in budget recording was labelledprovider_error. Every run that endsfailednow records itserror(type and message, credentials removed and privacy-redacted) and itstermination_reasononget_runandlist_runs.provider_erroris reserved for the model provider; a fault in the runtime or a component it uses isinternal_error. -
run_resumedsays why. The event carriescause(approval,budget_grant,budget_denied,interrupted,recovered_after_lapsed_lease) andtrigger; a recovered run adds theprevious_owner,lease_expired_atandorphaned_seconds. -
A request is never lost when the memory store fails at the start of a
run. A run whose first memory read failed (Postgres restarting) was
resumed without the user’s message and answered a refund request nobody had
recorded. The request is now on the run’s record from its first save, a
resume puts it back in the history, a record with no request is refused
(
RunRequestLost, the run endsfailed), and a failed memory read is tried again twice before the run fails with the store’s error. - Budget writes in the SQL store survive a dropped connection. With the per-checkout ping gone, a restarted database failed the first budget statement on each pooled connection. Budget reads, the legacy-counter move and budget changes now retry once on a fresh connection. A change is retried only if its transaction had not begun to commit, so a charge that did commit is never applied twice.
-
A message the store could not write is an error. The SQL, Redis and
MongoDB stores logged a failed
store_messageand returned, so a request or an answer that was in no history looked stored. The write now raises and the run fails with the store’s error. The message goes on the run’s record first, so a failed history write does not also lose the answer. A custom memory store should raise fromstore_messagetoo. -
SQLite takes concurrent runs, and
sqlite://works. Fifty runs saving at once on a SQLite file raiseddatabase is locked. File databases now run in WAL mode with a 30 second busy timeout.DatabaseMessageStore("sqlite://")(in memory) failed on pool arguments SQLite’s in-memory pool rejects; it now uses one shared connection that sessions take in turn. -
A slow resume no longer lets a second sweeper claim the same run. A
claimed orphan’s lease started at the claim, but the resume can take longer
than a lease to start (a cold model client took 74 s against 60 s), and a
second sweeper then claimed the run again. The claim now keeps the lease
alive until the run’s own heartbeat takes over. Separately,
resumeremoved the dead attempt’s sandboxes before it saved the run as its own, so a process that lost the claim could remove the winner’s sandbox; it now does so only after winning the save. Newrelease_claim(claim)on the agent for a claim you will not resume. - MongoDB opens one connection, not one per concurrent first call. Fifty calls arriving before the first connection was ready each opened a client, and all but one were leaked. The calls now wait for a single connect.
-
A person’s approval is not lost to the run’s last heartbeat. A run waiting for approval stops
its keep-alive as it returns, but a heartbeat already sent to the database can still land after
it and change the record’s version between the moment the decision reads the record and the
moment it saves it. With 32 refund runs at once on Postgres, one decision in the benchmark failed
with
RunStateConflict: ... changed since version 11, about a record the approver had not touched. Deciding now reads the record again, checks the approval again (a decision another person made meanwhile stands) and saves against the new version, up to ten times. -
A session’s state is released when its run ends. The agent kept a
SessionStatefor every session that ever ran, with the last run’s messages and its loop detector, for the life of the process. In a 30-minute server soak of the support desk (20 users, 2,233 visits) memory rose 2.6 to 2.8 MiB a minute with no plateau. Locally the same flow left about 48 more allocated blocks per visit (81 against 33). The store now drops the session when the run leaves, on every exit: completed, failed, cancelled, suspended. -
A decision that could not be saved is retryable, not a 500.
POST /runs/{run_id}/approvals/{approval_id}answered 500 when the paused run’s own writes kept changing its record pastdecide’s re-reads. Nothing was saved in that case, so it now answers503withRetry-After: 1; send the same decision again.
Changed
-
A local trace archive packs bodies into segment files. It kept one file per finished
trace, about 4.7 a visit for 30 days: millions of files, and kernel inode and name caches
that grew with them. Bodies are now appended to
bodies/segments/<hour>-<writer>.seg, one segment per process per hour, and a trace’s index row records its offset and length. A segment is deleted when no trace points into it. Archives written by 0.5.1 and earlier are still read and pruned; nothing is rewritten. Object storage keeps one object per trace. For a busy deployment, export traces and shortenretention_days. -
A call’s budgets go to the store in one transaction, not one per scope. A model call holds
its cost on the request, session and application counters and settles all three afterwards, and
each was its own thread hop and commit: 27 of the 60 database transactions of one support desk
refund run (a lookup, a refund, an approval and a resume, with three budget scopes). SQL and the
in-memory store now apply every scope of a call together, all or none, locking rows in key order;
a refusal on any scope applies nothing, and the first refusal in scope order (request first) is
the one reported. The desk run now makes 11 budget transactions. Settling a request and
releasing a resumed run’s old holds also take every scope in one call, and the end of a run reads
its counters in one read. Redis and MongoDB keep one call per key: Redis keeps each key under its
own hash tag, so one script cannot span keys on a cluster, and MongoDB’s multi-document
transactions need a replica set; their calls are asynchronous and cheap, and each is already
atomic. A custom store may offer
apply_budget_changes,get_budget_statesand setbatches_budget_changes = True(seeAbstractMemoryStore); without them nothing changes. -
A step boundary is one write, not a read and a write. Each step began by reading the run’s
record to look for steering messages and a request to stop, then saved the new step number. The
save already names the version it read and, finding another writer, merges their inbox and stop
request, so the read told nothing the save did not: three reads per refund run. The boundary is
now the save alone. A run asked to stop keeps its step number, a steering message is delivered
once, and a run another process has taken over still cannot begin a step. The end of a run no
longer reads its own record back to look for workers when it started none, and an ID made by
generate_run_id()is not looked up before its run (nothing can be stored under it yet). -
The SQL store’s database calls run on their own threads. They shared the event loop’s default
executor with every other
asyncio.to_threadin the process, capped atmin(32, CPUs + 4)(6 on a 2-core container) however many connections the pool allowed; on the support desk ramp most active samples were database threads waiting. The store now has one thread for each connection its pool can lend (pool_size + max_overflow), so database waits neither queue behind a tool’s file read nor hold it up. The caller’s context variables still reach the thread. - Token counts and redactions are kept, not worked out again. The support desk ramp profiled the event loop: privacy redaction 14-28% of its time and token counting about 12%, mostly the same content handled again at every step. A text’s token count and a text’s redaction are now kept by a digest of the content (never the text) in bounded caches, so the cache holds integers and already-redacted text. What is counted and redacted is unchanged.
- A message’s privacy-safe digest form is worked out once per trace. Each step digested the whole context again, and each message was redacted, scrubbed and masked three times over (the context events, the model call’s evidence, each message’s own digest): about 19% of the event loop on the support desk ramp. The form is kept for the length of its trace, bounded, and dropped when a credential is registered or the redaction policy changes; a message is also put in canonical form once, not twice, when a model call’s context is described. The digests the trace records are byte for byte what they were. A resumed run still records its run configuration in its own trace: each segment’s trace is read on its own (the trajectory header and the version metadata are per trace), so it is made cheap instead of skipped.
-
Telemetry IDs no longer cost a system call each. An event or span gets an ID (about forty to a
step), and
uuid4()asked the operating system for random bytes every time. A thread returning from a system call has to win the interpreter lock back, which with the database threads busy waits up to the switch interval: sampled on a loaded machine, ID generation was a fifth of the event loop’s busy time. One read of the same source now serves 256 IDs (a forked child starts a new block). IDs are as random as before and keep the version-4 shape. -
A budget transaction sends fewer statements. On a remote database each statement is a round
trip, and a transaction is several. Where the database can answer a
DELETEor anUPDATEwith the row it changed (PostgreSQL, SQLite 3.35 and later), a hold is removed in one statement (not aSELECTand aDELETE), and a meter’s new total comes back with itsUPDATE(not aSELECTafterwards). A refund run’s budget statements went from 72 to 52. Other databases are asked the long way and give the same answers. -
Budget storage layout. SQL stores gain the tables
budget_meters,budget_holdsandbudget_grants; Redis and MongoDB keep counters under new keys (omnicoreagent_budget2:,<collection>_budgets). Counters written by 0.5.x are moved on first touch; do not run 0.5.x and this version against the same store at once. A custom memory store that implements budgets must now implementapply_budget_change,get_budget_grant_historyandlist_budget_holds(seeAbstractMemoryStore); without them, budgets are off for that store, as before. -
OmniServe takes only as many concurrent runs as one process can serve well. On a 2-core
container every request was accepted and all of them slowed down together (chat p50 37 s at 100
users; throughput 3-4.5 requests a second). A process now admits 24 concurrent runs, the knee
measured on a server: one process gave 5.1/s at 24 users and 5.5/s at its peak, then slowed down
(p95 22.8 s at 64), and it used about one core whatever the cap. The limit does not scale with
CPUs; scale out with one replica per core behind a load balancer sharing the store (two 1-CPU
replicas peaked at 9.4/s against 5.5/s for one process, with every refund done exactly once).
apps/support_desk/scale/has the example. A request over the limit waits up to 5 seconds for a slot, then gets503withRetry-Afterbefore any work starts. Only/run,/run/sync,/runs/{run_id}/resumeand/background/tasks/{task_id}/runcount; health,/prometheus, reads and approvals never wait. SetOMNICOREAGENT_SERVE_MAX_CONCURRENT_RUNS(0ornonefor no limit) andOMNICOREAGENT_SERVE_RUN_ADMISSION_WAIT./prometheusgainsomniserve_runs_in_flight,omniserve_runs_waiting,omniserve_runs_limitandomniserve_runs_rejected_total. Callers that sent more than the limit at once now see503and should retry. -
The SQL memory store does less per operation. About 45% of a 30-user profile was its
per-operation work: a
SELECT 1ping on every checkout, aBEGINand aROLLBACKaround every read, and a fixed 20+30 pool. It no longer pings; reads run on their own autocommit pool, so a read sends one statement; the pool is 10 plus up to 22 overflow (OMNICOREAGENT_SQL_POOL_SIZE,OMNICOREAGENT_SQL_MAX_OVERFLOW, orpool_sizeandmax_overflowonDatabaseMessageStore). A connection the server dropped is survived by running the operation once more on a new connection, which loses no message; a write is never repeated once its commit has begun.clear_memoryalso no longer blocks the event loop. -
Resuming a run no longer lists the telemetry store. Closing a resumed run’s dead trace
segments listed every trace of the run through
list_traces, which walks every trace in the store; about 5% of a 30-user profile, growing with the store. It now reads only the segments the run’s record names. The result is the same. -
OmniServe resumes the interactive runs a dead process left behind. Until
now a run whose process died (a killed container, a lost store connection)
stayed
runningwith a lapsed lease until someone calledresume(run_id). The server now sweeps for them on start and every 30 seconds (each wait varies by up to a quarter, so replicas do not sweep together), claims a run through its record’s own version check (two replicas never take one run), and resumes it under the durable rules: a call that did not finish becomesunknown_outcomeunless its tool is idempotent. Runs waiting for a person and background runs are left alone; a run recovered three times and still not finished is endedfailed. The trace says why (run_resumed, see above). To keep the earlier behaviour, setOMNICOREAGENT_SERVE_ORPHAN_SWEEP_ENABLED=false. Without OmniServe,agent.claim_orphaned_runs()andagent.resume_claimed(claim)do the same. -
A server’s memory no longer climbs with every visit from the C allocator. In the 30-minute
soak the desk’s memory rose 2.6 to 2.8 MiB a minute with no plateau. Most of it was glibc: each
thread that allocates gets an arena (up to eight a core), and OmniServe runs many (the database
calls, the telemetry writer, tool calls), so freed memory stayed in arenas. OmniServe now asks
for at most two arenas when it is built.
MALLOC_ARENA_MAX, if you set it, wins, andOMNICOREAGENT_SERVE_MALLOC_ARENASchanges the number (0leaves the allocator alone). It does nothing off glibc.
Known issues
None breaks a core promise. Each is measured, and none is planned to change in a patch.- The first run in a fresh process starts slowly. Loading the model client takes 15–60 s on a
cold or loaded machine: it is
import litellm, which OmniCoreAgent does not control. It runs off the event loop and shows in the trace as amodel.client.loadspan. OmniServe and background workers load it while they start, so only a short script or a headless run pays for it. - The runtime’s own time per step is about 160 ms at p95. Measured on a 2-core container at 20 users, excluding the model’s time; about 100 ms of it is CPU in the model client library and the telemetry write. Against model calls of 1 to 10 seconds it is a few percent of a run. It is also why one process bends at about 24 concurrent runs: run more processes (how many runs one process takes).
- A run can fail after a side effect when the model keeps failing. If the provider still
answers 429 or 500 after every retry, the run fails with that error, after whatever its tools
already did. The work is never repeated, and the run’s record lists it under
side_effects(for exampleissue_refund,success), so the failure is explainable from the record alone. A failed run cannot be resumed; the caller starts a new one.
0.5.1
October 2026 · Python 3.12+ Fixes only, no new features: the known issues from 0.5.0, and three more found running the steward on 0.5.0. Most code runs unchanged. What now behaves differently: a custom strict policy no longer blocks the sandbox’s own file sync;local refuses network-off and image when the agent is built; two agents sharing one telemetry store must agree on retention_days; under capture="full", credentials in command text are redacted.
Fixed
-
Copying many files into a fresh Docker sandbox is one archive. Each file was its own
mkdirandput_archive(about 0.3 s a file on the test machine), so 200 workspace files took 61 s; they now go in as one tar stream, in 0.8 s. The same files are copied, with the same paths and modes, and the sandbox user owns them. One path outside the working directory now refuses the whole copy before anything is written. -
The generated
.dockerignoreis no longer broad and narrow at once. It left out any folder namedworkspace(including a Python package of that name). It now leaves out only the workspace folder you configured (OMNICOREAGENT_WORKSPACE_DIR, default./workspace), anchored at the build context’s root. It also leaves out, in any folder,.env,.env.*,*.pem,*.key,*.p12,id_rsa*,id_ed25519*,.netrc,.npmrc,.pypirc,credentials*.json,.awsand.ssh(before,.env*only). An existing.dockerignoreis still kept as it is. -
The
localprovider refusesnetwork_policy: denyandimagewhen the agent is built. It refused them at the first command, after a person had been asked to approve the call. Asandbox_manifestthat asks for either, or says nothing about the network (the default isdeny), or no manifest at all, now fails at construction with the same message the other providers give. Its other behaviour is unchanged. -
A sandbox command is in the record under
capture="full". Thesandbox_exec_startedevent carried only the program name andargc, andsandbox_exec_completedkept the exit code in its metadata but not besidestdoutandstderr. Underfull, the start event now holds the command text, with credentials in it (--api-key=...,TOKEN=...,Authorization: Bearer ...) redacted; under the default capture it still holds the name andargconly. The outcome’soutputnow carriesexit_codeunder both. -
Two agents sharing one telemetry store must agree on
retention_days. The store is one object per file in a process, so the second agent silently got the first one’s window (a warning was logged and its value ignored). A second, different value is now refused when that agent opens the store (initialize()), with a message naming both values; the same value is fine, andNone(keep every trace) counts as a value. Give such agents the sameretention_days, or astorage_patheach. -
A folder operation under an ask rule asks once. Deleting, moving or clearing a folder that
holds a file covered by an ask rule was refused, and the message said the policy “does not
allow” it. The operation now asks a person once and lists the covered files (the first ten, then
“and N more”; the pending approval carries them as
covered_files). Approving lets the operation run, and the covered files count as approved for that operation only; denying changes nothing. A deny rule on any file inside still refuses the whole operation, without asking. Where nobody can be asked (approval_mode="fail") the message says the call needs a person’s approval. -
An approval’s command text follows the same secret rule as a sandbox command. Under
capture="full"the approval events in the trace kept a shell command’s summary verbatim, soprintf token=...carried its credential. They now keep the text withkey=valuecredentials redacted, as the sandbox command’s own record does; under the default capture the summary is still replaced by its count. The approval shown to the person and the run’s own record are unchanged. -
read_skill_fileno longer reads a sibling skill’s files. The requested path is resolved, symlinks followed, and must sit inside the skill’s own folder; otherwise it is refused with a message naming the skill. -
A strict policy no longer silently denies the sandbox’s own file listing. After a sandbox
command, the runtime lists the sandbox’s files to copy what changed back to the workspace
(
sandbox.workspace.sync). A custommode: strictpolicy with no rule for it denied the listing on every command, and nothing came back. With no rule of its own the listing now follows the decision for its command (allowed, or approved), and the trace says so (follows_command). A rule you write for it still decides, and each file copied back is still decided by the workspace file rules. When a rule refuses the listing, the model is told why instead of “the sandbox was lost”. -
The
tool_callsbudget counts a call once. A call that asked for approval was counted when it asked and again when it ran after approval and resume, so the meter ran ahead of the calls made by the number of approved writes (44 against 41 in one run, 40 against 37 in another). A call is now counted when it is allowed to run: one that is waiting, denied or refused by a rule is not counted, and a call the budget cannot afford still does not run. - An approved call the budget then refuses is not asked about again. When an approved call resumed and the budget had nothing left, the person’s approval was already marked used, so after a grant the same call asked a second time. The approval is now kept until the call runs.
-
After a crash and resume, the dead segment no longer shows
running. When a run resumes, its earlier trace segments that never ended are closed with the new trace statusinterruptedand an end time equal to their last event. Listings filtered bystatus=runningno longer return them, and the run’s own record is unchanged.interruptedis a new value of the tracestatusfilter. -
A worker whose lead crashed no longer stays
running. When a lead run ends (finished after a resume, cancelled or abandoned withabandon_run), the workers it started withspawn_subagentsthat are stillrunningwith a lapsed lease are marked with the new run statusabandoned, and the error names the lead’s run. A worker whose lease is alive is never touched.abandonedis also a value of thestatusfilter oflist_runsandGET /runs. -
A very small budget no longer pauses on every call, once a grant names its amount. Nothing in
the ledger is per call: a grant raises the run’s limit and later calls draw on it (a limit of
one tool call plus a grant of 3 runs four calls). The pause came from the default: a grant with
no
amountis the shortfall of the one call that stopped, so a run that stops one call at a time asked again at each. That default is unchanged (a grant never gives more than was asked), and the budget docs now say to name anamountfor the work still to do.
0.5.0
September 2026 · Python 3.12+ Give your agent real work. Keep control. Every action checked before it runs. Every run survives a crash without silently redoing anything. Every step on the record. From this release an agent is governed unless it says otherwise.Changed: governance is on by default
An agent with no governance settings is governed by thepermissive-dev
profile. Its own tools, workspace, memory, code mode, skills, sub-agents and
background runs are allowed, and nothing pauses for a person unless the agent is
given a sandbox with network (then a person is asked first). Refused unasked:
reading raw secrets, unrestricted host files and network, installing packages,
and shell commands on the host. The MCP servers you configure are
trusted: starting them and calling their tools is allowed by a named rule. There
is no budget until you set one. governance_config={"enabled": False} turns it
off. Enabling governance
yourself without naming a profile still means interactive-dev, as before, so
no one’s policy loosens. See Upgrading, “From 0.4 to 0.5.0”.
New: worker profiles
List the kinds of worker a lead may spawn inworker_profiles, each with its
own model, reasoning effort, tools, MCP servers, steps and deny or ask rules,
and the lead picks one for each worker
(Sub-agents).
A profile only narrows what the lead has. Every worker spends the lead’s
budgets, and each spawn records the profile, model and effort it ran with.
Changed: a simple policy, the sandbox is the boundary
- Command rules are prefix rules. A rule names a command by its first
words (
{"prefix": ["git", "push"]}, or{"program": "rm"}). Only a plain chain of commands is split; anything else (a redirect,$(...), a variable,eval,sh -c…) is one unreadable command, never allowed by a rule and, where the policy has command rules, asked about in every mode (refused instrict).args_any,redirect,envand globs in rules are refused with the new form. This replaces 0.4.4’s parser, which was never published. - A sandbox setting the provider does not enforce is refused when the agent is built, naming the provider and the setting. Examples are a workspace mount on a hosted provider, a sandbox lifetime, or a host allowlist on Daytona. Each provider is listed in What stops harm.
filesystem_policyis removed. Its path lists were approved and then applied by no provider; a manifest naming it is refused with what to use.- Modal and Vercel check a sandbox’s network from inside, as E2B and Daytona do, and refuse one that should be offline but is not.
Changed: OmniServe defaults
OMNICOREAGENT_SERVE_CORS_CREDENTIALS is false by default and never applies
with * origins. X-Forwarded-For is read only from the proxies in
OMNICOREAGENT_SERVE_TRUSTED_PROXIES. /prometheus counts per route, not per
path.
Changed: what a trace keeps by default
Undertelemetry_config={"capture": "default"}, a governed agent’s tool and
delegation arguments are recorded by name, not value. Every agent is now
governed, so under that preset this applies to all of them. The capture you
get without setting one, full, keeps the values (personal data redacted).
Fixed
- A folder delete, move or clear is checked for each file in it: a deny on
deleting
prod/*did not stopdelete_file("prod"). A workspace path through a link is refused. - Spawned workers spend the lead’s budgets; they each had their own.
- OmniServe: error bodies never carry a registered credential, and the token is compared in constant time.
- A governed run’s evidence under full capture was reported incomplete.
Every delegation’s policy request masked its task as
[REDACTED], which the trajectory counted as a capture gap. The task text is left out of that record instead (the delegation call keeps it);task_lengthstays.
Docs
- A new landing page that leads with what is different, and a
comparison with an Agno column and a row for what happens
to an interrupted tool call after a crash, each cell from the project’s own
docs, read 2026-09-29. Pydantic AI’s
cost_limitcell is corrected: it is checked after each response.
Known issues in 0.5.0
These came from the 0.5.0 release testing. None broke a core promise on a normal path. The current list is under 0.6.0.- The first run in a fresh process starts slowly. Loading the model client can take 15–60 s on
a cold or loaded machine. Measured in 0.5.1, it is
import litellm(about 3,000 modules; about 9 s of CPU on a warm bytecode cache and about 20 s on a cold one, on the test machine), which OmniCoreAgent does not control. It is already off the event loop, on a worker thread, and shows in the trace as amodel.client.loadspan; the headless--timeoutstarts after it. What is ours is light:import omnicoreagenttakes about 10 ms, building an agent and initializing it well under a second, withlitellmandopenainot yet loaded. OmniServe and background workers load it while they start, so only a short script or a headless run pays for it on its first run. Still applies: see 0.6.0. - A very small budget pauses on every call. With a limit as low as one tool call, each call pauses on its own, so a person grants each one in turn. Fixed in 0.5.1 for a grant that names its amount.
read_skill_filecan read a sibling skill’s files. Its path check is by prefix, so a sibling whose name starts with the skill’s own name (../my-skill-extra/filefrommy-skill) is not refused. Paths outside the skills folder are. Fixed in 0.5.1.
0.4.4
Not published: tagged and tested, never uploaded. Its fixes ship in 0.5.0, and its command rules were replaced there by prefix rules. Filming OmniCoreAgent as it really runs (an agent killed mid-payment, a command denied, a budget topped up from another process) found two gaps. This release closes them.New
- Rules on shell commands.
Every command reached the policy as
sh, so no rule could namerm -rf. Aprocess.execrule can now carry acommandmatcher (program,prefix,args_any). The command is parsed first, so quoting, paths,$(...),sudo,sh -c '...',xargsandfind -execdo not hide what runs. Any matching deny or ask applies; an allow needs every command proven; a command whose effect cannot be proven (eval,... | sh) is never allowed by one. Rules carryexamples, checked when the policy loads. Tested against the bypasses that defeated text denylists elsewhere, and against a real shell.
Fixed
- An approval for a command now shows the command, and is bound to it. A
person asked to approve a command on the host saw
shand an argument count, and the approval did not name the command: approvinglslooked the same as approvingrm -rf ~. It now lists the commands it would run and applies to that exact text. budget_statusandabandon_runwork from a process that never ran a query.budget_statusanswered[], andabandon_runleft the run’s budget holds on the day’s counter, when called from a second process.- An unknown key in a policy rule is a clear
PolicyLoadErrornaming the rule and the key, not a bareTypeError.
Changed
- New dependencies:
tree-sitterandtree-sitter-bash(MIT, binary wheels), to parse commands.
0.4.3
September 2026 · Python 3.12+ An outside review read these docs and asked the question every durable runtime must answer exactly: what happens when the process dies halfway through a tool that has side effects? Answering it, we found one case where the answer was wrong, and fixed it.Fixed
- A call that finished while another call of the same turn was still running is no longer run again after a crash. Results of a turn’s calls are saved once every call of the turn has finished. A call that completed while a sibling was still running, when the process died, had its state saved but not its result, and recovery ran it again whatever its idempotency: a card could be charged twice. Now it runs again only if its tool is idempotent; otherwise the model is told “This call finished (it reported success), but its result was lost … do not call it again without checking.”
New
- On Python 3.10 or 3.11, installing fails loudly. A guard release, 0.3.10, exists for exactly those Pythons: pip and uv choose it over 0.3.9 (which cannot build an agent) and stop with a message saying how to get Python 3.12. It never shows as the latest release.
Docs
- What is guaranteed:
what a recovered run repeats and what it never does, per call and model-call
state (exactly once, at most once, at least once), one owner, how budgets
count across a crash, sub-agents and background retries. Four places that
claimed more than the code are corrected, and the recovery sample now also
catches
RunStateConflict. - The navigation leads with what is different: Make It Safe, Run It and See and Improve now come before Build.
- Short paths work:
/docs/quickstart,/docs/installand others redirect instead of returning 404. - One tagline: The governed runtime for Python agents you can let act.
0.4.2
September 2026 · Python 3.12+ Every section of these docs was rewritten by running its examples, and then tested by agents that had only the docs and a build of this release: they built a refunds desk, an ops service, and an evaluation loop, and every feature they used worked. What they and the writers found in the code is fixed here. Most of it is fixes; five settings now do what their names say, and those are listed first.Before you upgrade
- A durable memory store needs its URL.
MemoryRouter("sql")withoutDATABASE_URL(rediswithoutREDIS_URL,mongodbwithoutMONGODB_URI) raises aValueErrornaming the variable. It fell back to memory, so runs silently did not survive a restart or resume elsewhere.MemoryRouter("in_memory"), the default, is unchanged. - Sandbox settings no provider keeps are refused when the agent is
built:
denied_hosts(except withhttp),environment.secret_refs, and a host allowlist on Daytona. They were accepted and silently not enforced. cancel_previousinterrupts the run it replaces, within a heartbeat, instead of letting it finish its attempt and then discarding it.misfire_policy: the default is nowrun_once, which behaves as the old default did.skip_missednow skips: nothing runs for a time missed by more than a minute.- Trace retention removes abandoned traces: a trace still
runningwith no activity for longer thanretention_days, whose process died. A trace waiting for a person is never removed. - OmniServe: a timed-out
/run/syncor resume answers504with adetailobject that names the run (message,run_id, …), wheredetailwas a string. A403for a refused request addscapability,reason_codeandmatched_rule_ids.
New
workspace_config["files_dir"]: the file tools work in a directory you already have; the runtime’s own files stay inworkspace_dir. Workspace files- Harbor trials use the task’s own directory for the agent’s files,
whatever the task set (they pointed at the agent’s workspace, and a model
spent steps on “Path not found”).
harbor doctortakes--akasrundoes. Harbor - A denial is a decision in the evidence: a
denywithdenied_by(the person, orsystemfor an approval nobody decided in time), as an approval is anallowwithapproved_by. - The run header says which privacy boundaries redact (
harness.privacy). await memory_router.close()closes MongoDB, Redis and SQL connections.
Fixed
Privacy and governanceprivacy_config["redact_model_io"]now keeps personal data from the model provider. It was accepted but never applied.- An approval past its expiry no longer strands its run: one nobody decided
in time is a refusal on resume, and a decision made in time stands however
late the resume. A person’s denial reaches the model as theirs, note
included; with
approval_mode="fail", the model is told the call was refused. ls("."), andgloborgrepwith no path, list the whole workspace; they were refused as invalid arguments.- Clearer messages: a strict-policy denial says no rule matched; a YAML policy says YAML is not supported.
- A call that crosses a
model_tokensbudget is recorded in full, and the next call is stopped before it is made. The crossing call’s tokens were lost and a resume made the call again.
- A finished trace read back by a new process was marked incomplete; it reads back complete.
- Pruned traces stay pruned (they came back on the next start), and
prune()counts archived traces. - A resumed run’s totals count a step once; a run with nothing priced says
Nonefor its cost everywhere. training_records()carriesfinal_answer, and every step hasresumed.- An exporter’s failure keeps its reason in the trace.
- A timed-out run’s record says
timeout, as its trace does. list_runs()approvals carry the call’sarguments.- The MongoDB store closes its connections; a store built per request leaked a pool each time. SQL stores release their shared pool.
omniserve runwith a Redis or MongoDB task store and no URL is a clean config error.- The prompt-injection guardrail logs through the
omnicoreagent.guardrailslogger instead of printing to stdout.
- A headless run’s
result.jsoncounts the whole run’s usage. harbor doctorfails a development runtime nothing can build into a wheel.
Docs
Build, Make It Safe, Run It and See and Improve rewritten page by page, with new pages for Policies, Approvals, Budgets, Privacy and credentials, Stores and scale, Read a run, and Outcomes and training records; a comparison with other agent frameworks; and the generated reference now shows each method’s whole description.0.4.1
September 2026 · Python 3.12+Changed: finished run records are removed after 30 days
Every run keeps a record in the memory store; until now they were kept forever. From 0.4.1 a finished run’s record (completed, failed, blocked, cancelled, timed out) is removed 30 days after the run started. A run waiting for a person or a resume is never removed. To keep every record, setagent_config["run_retention_days"] = None. Any other number of days
works too; await agent.prune_runs() prunes on demand and reports what it
removed, and the retention status has a runs section.
This is the run’s record (status, approvals, usage), not its traces: the
step-by-step trajectory lives in the traces, still kept 7 days by
telemetry_config["retention_days"]. When a run’s traces are gone and its
record is not, get_run_trajectory now says so (trace_kept: False,
traces_missing) instead of reporting zero tokens and no cost.
How long a run’s evidence is kept
New
base_urlinmodel_config. Point any provider at another endpoint that speaks its API: vLLM, LM Studio, a gateway, a proxy. Sent with each of the agent’s calls; two agents can use two endpoints. Models
Fixed
- Agents sharing one
ToolRegistryrebound each other’s file tools, so a lead could read a child’s workspace (“File not found” on a file it had listed). Each agent adds its tools to its own copy; your registry is not changed. - A file tool that failed (“File not found”, “File already exists”) was recorded as a success; it is the tool’s error now, with the same text.
get_latest_trace(session)returned a delegated child’s trace; it returns the agent’s own.- A
delegate_<name>tool offered the model the child’stags,provenanceand private parameters; it offers the task only. - An approval for a call made inside a
run_codeprogram carried no arguments; the run’s approval record has them. - A partial
memory_config({"mode": ...}) failed withKeyError; it keeps the other defaults.switch_memory_storekeeps the window and summary settings. - A tool parameter named like a secret (
api_key) had its schema redacted, which marked the run’s evidence partial.
Docs
- The Build section rewritten page by page, every example run; navigation by what a reader is trying to do.
- How OmniCoreAgent compares, each cell sourced.
- Skills:
allowed-toolsis read and grants nothing; the policy decides. - Settings explained one by one: the telemetry
record_*switches (which are master switches and which narrow them), the guardrail, archive, offload and timeout settings that shared one description,memory_config’s default window, and which limit to use amongmax_steps,request_limit,total_tokens_limitand budgets. - How long a run’s evidence is kept: traces (7 days) and run records (30 days), side by side.
0.4.0
September 2026 · Python 3.12+ 0.4 rebuilds OmniCoreAgent as a runtime for agents that act: an agent that can run commands, spend money and change things needs a policy over what it may do, a person to ask, a sandbox, a budget, a record of everything it did, and a run that survives a crash. 0.4 has each of them, on by default where that is safe and one setting away where it is not. Coming from 0.3? Upgrading to 0.4 lists, as measured, what runs unchanged, what was removed and what took its place, and the defaults that changed.New
- Governance. A policy — allow, ask, deny — over every capability: each
tool, each MCP server, the sandbox, the network, delegation, background
runs. Built-in profiles;
askpauses the run for a person. Security model - Durable runs. Every run has a record. It pauses for an approval or a
budget top-up and resumes where it stopped (
resolve_approval,resume);interrupt,steerandabandon_run; a run whose process died continues from its checkpoint, and a call interrupted mid-way is reported, never repeated. Durable runs - Budgets in dollars, tokens, calls and sandbox seconds, per request, session, agent or application.
- Sandboxed execution. An
executetool whose commands run in Docker, E2B, Modal, Daytona or Vercel — orlocalwhere a container is already the boundary — with no network unless the policy allows it, and without your process’s keys and tokens. Sandbox providers - Code mode. A
run_codetool: the model writes a short Python program that calls your tools, run in Monty, every call inside it governed and traced. Code mode - Telemetry. One trace per run, readable end to end, kept on local disk
by default; trajectories,
get_run_trajectoryacross a run’s segments, export to OTLP, LangSmith, Opik or JSONL. Outcomes that arrive later (record_outcome) andtraining_records()for evaluators and trainers. Observability - Privacy. Emails, phone numbers, SSNs and card numbers are redacted from the record; the run itself works with the real data.
- Native tool calls for every provider, streamed (
agent.stream(),on_event=), with provider continuation data kept across turns. - Sub-agents spawned under the same policy and budgets, each with a
linked trace;
sub_agents=[...]exposes agents asdelegate_*tools. - Skills (
.agents/skills), AGENTS.md files, workspace files on local disk, S3 or R2. - Background agents rebuilt on a durable task store (memory, SQL, Redis, MongoDB): cron, intervals, leases, retries, recovery after a restart.
- OmniServe gains approvals, budgets, runs, background tasks and traces over REST and SSE. OmniServe
- The
omnicoreagentcommand:runfor one headless run (CI, evaluations);harborfor Terminal-Bench through Harbor, withharbor doctorandharbor results. CLI reference - The reference, generated from the code: every method, setting, command and HTTP route. Reference
Changed
- Python 3.12 or later.
- Defaults:
max_steps50 (was 15),tool_call_timeout180 s (was 30), context management and tool offload on, the prompt-injection guardrail on (guardrail_mode="full"), workspace files on, telemetry on. get_tracealso takestrace_id=andrun_id=.- An
agent_configkey that does not exist is refused with the settings that do, instead of a bareTypeError.
Removed
SequentialAgent,ParallelAgent,RouterAgent,DeepAgent,OmniAgent.EventRouter,event_router=and the event-store methods, replaced by telemetry.BackgroundOmniCoreAgent,BackgroundTaskScheduler,APSchedulerBackend,TaskRegistry, and thebackgroundextra.agent_config["memory_tool_backend"]and thememory_*tools, replaced by workspace files.- The top-level retry and circuit-breaker helpers;
omniserve run --reload.
Fixed
Found by four agents that built applications from these docs alone, before the release:-
A resumed run’s evidence is whole: the call a person approved appears in
its own step, as approved by them; the run’s totals and its training
record include every segment;
get_run()shows what each approval is for. -
An ask on delegation (
delegate_<name>) pauses the lead, as documented; it was a tool error. -
Full capture keeps governed tool arguments, and the calls a
run_codeprogram makes have theirs. -
budget_statuscounts a person’s grant, as enforcement does. - A run’s deadline holds when the work does not stop when told.
- A host allowlist a sandbox provider cannot enforce is refused when the agent is built.
-
0.3.9 was published without
omnicoreagent.core.workspace, soOmniCoreAgent(...)failed on import. A test now fails if any source file of the package is left out of the wheel.