Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs-omnicoreagent.omnirexfloralabs.com/llms.txt

Use this file to discover all available pages before exploring further.

Telemetry Events

OmniCoreAgent records typed telemetry events for user messages, model calls, tool batches, tool calls, observation handling, subagents, final answers, and background run lifecycle changes. Telemetry is available by default. Simple apps use the in-memory telemetry store automatically. Serving integrations can replay stored telemetry and follow new events through Server-Sent Events.

Quick Start

result = await agent.run("Research three AI agent runtimes.", session_id="user_1")

events = await agent.get_telemetry_events_after(
    cursor=None,
    session_id=result["session_id"],
    run_id=result["run_id"],
)

for event in events:
    print(event.event_type, event.model_dump())

exact_trace = await agent.get_trace(result["trace_id"])
latest_session_trace = await agent.get_latest_trace(result["session_id"])
run_correlated_trace = await agent.get_trace(run_id=result["run_id"])
normalized_trace = await agent.get_trace(result["trace_id"], normalize=True)

Common Event Types

EventDescription
user_messageUser input received
model_callModel request started
model_responseModel returned content and usage
model_errorModel call failed
agent_stepReAct loop step started
tool_batch_startParallel tool batch started
tool_callIndividual tool execution started
tool_resultIndividual tool returned successfully
tool_errorTool validation or execution failed
observation_pipeline_endTool output was parsed, cleaned, and formatted
subagent_spawnOne or more subagents were delegated work
subagent_resultSubagent returned output
subagent_errorSubagent execution failed
guardrail_violationGuardrail blocked unsafe input
final_answerAgent produced final response
runtime_errorRuntime failed

Streaming

session_id = "user_1"
run_id = "run_ui_1"
cursor = await agent.get_telemetry_stream_cursor(session_id=session_id)

task = asyncio.create_task(
    agent.run("Use tools if needed.", session_id=session_id, run_id=run_id)
)

async for event in agent.stream_telemetry_after(
    cursor=cursor,
    session_id=session_id,
    run_id=run_id,
):
    print(event.event_type)
    if event.event_type == "final_answer":
        break

await task
OmniServe uses the same telemetry stream:
  • POST /run streams telemetry for the run it starts and finishes with complete.
  • GET /telemetry/events returns stored telemetry events as JSON. Filter by trace_id, run_id, session_id, task_id, or event_type. The default event limit is 200.
  • GET /telemetry/events/stream?session_id=... replays stored telemetry for a session and follows new telemetry. Add run_id to isolate one run.
  • GET /telemetry/traces/{trace_id} returns one exact trace.
  • GET /telemetry/runs/{run_id}/trace returns the latest trace correlated to one run.
  • GET /telemetry/sessions/{session_id}/trace returns the latest trace for a session.
The older /events/{session_id}, /events/{session_id}/list, and /events/{session_id}/trace routes remain as compact session-oriented aliases. Use ?run_id=... on those aliases to isolate one run inside a shared session. Trace detail routes return 404 when the requested exact trace, run trace, or session trace does not exist. The trace list route defaults to limit=100. Every telemetry event includes its trace_id. Runtime-created events also carry correlation metadata such as run_id, session_id, and agent_id when that context exists. Use trace_id for exact trace lookup. Use run_id to filter or correlate one runtime execution inside a session. If several traces share one run_id, the run lookup returns the latest matching trace. Telemetry traces can also be exported. The exporter layer maps normalized OmniCoreAgent traces to OpenTelemetry span records, then sends them through OTLP/HTTP or vendor presets such as LangSmith and Opik.

Background Run Events

Background agents expose run-local lifecycle events through get_run_events(run_id) and OmniServe’s /background/runs/{run_id}/events. These events are emitted into telemetry, stored in the manager cache, and mirrored to workspace events.jsonl when the task workspace policy allows it. Common background lifecycle names include:
Lifecycle EventMeaning
background_task_scheduledA scheduled occurrence created a run.
background_run_queuedA run was created and queued.
background_run_claimedA worker claimed the run with a lease.
background_run_startedAgent execution started.
background_run_heartbeatActive worker refreshed the run lease.
background_run_retryingFailed attempt entered retry state.
background_run_recoveredExpired lease was recovered and requeued.
background_run_completedRun completed successfully.
background_run_failedRun failed terminally.
background_run_timeoutRun exceeded its timeout.
background_run_cancelledRun was cancelled.
background_run_skippedOverlap policy skipped a scheduled run.