> ## Documentation Index
> Fetch the complete documentation index at: https://docs-omnicoreagent.omnirexfloralabs.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Telemetry Events

> Real-time telemetry events for streaming, debugging, and future trace surfaces

# Telemetry Events

OmniCoreAgent records typed telemetry events for user messages, model calls,
tool batches, tool calls, observation handling, subagents, final answers, and
background run lifecycle changes.

Telemetry is available by default. Simple apps use the in-memory telemetry
store automatically. Serving integrations can replay stored telemetry and follow
new events through Server-Sent Events.

***

## Quick Start

```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
result = await agent.run("Research three AI agent runtimes.", session_id="user_1")

events = await agent.get_telemetry_events_after(
    cursor=None,
    session_id=result["session_id"],
    run_id=result["run_id"],
)

for event in events:
    print(event.event_type, event.model_dump())

exact_trace = await agent.get_trace(result["trace_id"])
latest_session_trace = await agent.get_latest_trace(result["session_id"])
run_correlated_trace = await agent.get_trace(run_id=result["run_id"])
normalized_trace = await agent.get_trace(result["trace_id"], normalize=True)
```

***

## Common Event Types

| Event                      | Description                                    |
| -------------------------- | ---------------------------------------------- |
| `user_message`             | User input received                            |
| `model_call`               | Model request started                          |
| `model_response`           | Model returned content and usage               |
| `model_error`              | Model call failed                              |
| `agent_step`               | ReAct loop step started                        |
| `tool_batch_start`         | Parallel tool batch started                    |
| `tool_call`                | Individual tool execution started              |
| `tool_result`              | Individual tool returned successfully          |
| `tool_error`               | Tool validation or execution failed            |
| `observation_pipeline_end` | Tool output was parsed, cleaned, and formatted |
| `subagent_spawn`           | One or more subagents were delegated work      |
| `subagent_result`          | Subagent returned output                       |
| `subagent_error`           | Subagent execution failed                      |
| `guardrail_violation`      | Guardrail blocked unsafe input                 |
| `final_answer`             | Agent produced final response                  |
| `runtime_error`            | Runtime failed                                 |

***

## Streaming

```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
session_id = "user_1"
run_id = "run_ui_1"
cursor = await agent.get_telemetry_stream_cursor(session_id=session_id)

task = asyncio.create_task(
    agent.run("Use tools if needed.", session_id=session_id, run_id=run_id)
)

async for event in agent.stream_telemetry_after(
    cursor=cursor,
    session_id=session_id,
    run_id=run_id,
):
    print(event.event_type)
    if event.event_type == "final_answer":
        break

await task
```

OmniServe uses the same telemetry stream:

* `POST /run` streams telemetry for the run it starts and finishes with
  `complete`.
* `GET /telemetry/events` returns stored telemetry events as JSON. Filter by
  `trace_id`, `run_id`, `session_id`, `task_id`, or `event_type`. The default
  event limit is `200`.
* `GET /telemetry/events/stream?session_id=...` replays stored telemetry for a
  session and follows new telemetry. Add `run_id` to isolate one run.
* `GET /telemetry/traces/{trace_id}` returns one exact trace.
* `GET /telemetry/runs/{run_id}/trace` returns the latest trace correlated to
  one run.
* `GET /telemetry/sessions/{session_id}/trace` returns the latest trace for a
  session.

The older `/events/{session_id}`, `/events/{session_id}/list`, and
`/events/{session_id}/trace` routes remain as compact session-oriented aliases.
Use `?run_id=...` on those aliases to isolate one run inside a shared session.

Trace detail routes return `404` when the requested exact trace, run trace, or
session trace does not exist. The trace list route defaults to `limit=100`.

Every telemetry event includes its `trace_id`. Runtime-created events also carry
correlation metadata such as `run_id`, `session_id`, and `agent_id` when that
context exists.

Use `trace_id` for exact trace lookup. Use `run_id` to filter or correlate one
runtime execution inside a session. If several traces share one `run_id`, the
run lookup returns the latest matching trace.

Telemetry traces can also be exported. The exporter layer maps normalized
OmniCoreAgent traces to OpenTelemetry span records, then sends them through
OTLP/HTTP or vendor presets such as LangSmith and Opik.

***

## Background Run Events

Background agents expose run-local lifecycle events through
`get_run_events(run_id)` and OmniServe's `/background/runs/{run_id}/events`.
These events are emitted into telemetry, stored in the manager cache, and
mirrored to workspace `events.jsonl` when the task workspace policy allows it.

Common background lifecycle names include:

| Lifecycle Event             | Meaning                                   |
| --------------------------- | ----------------------------------------- |
| `background_task_scheduled` | A scheduled occurrence created a run.     |
| `background_run_queued`     | A run was created and queued.             |
| `background_run_claimed`    | A worker claimed the run with a lease.    |
| `background_run_started`    | Agent execution started.                  |
| `background_run_heartbeat`  | Active worker refreshed the run lease.    |
| `background_run_retrying`   | Failed attempt entered retry state.       |
| `background_run_recovered`  | Expired lease was recovered and requeued. |
| `background_run_completed`  | Run completed successfully.               |
| `background_run_failed`     | Run failed terminally.                    |
| `background_run_timeout`    | Run exceeded its timeout.                 |
| `background_run_cancelled`  | Run was cancelled.                        |
| `background_run_skipped`    | Overlap policy skipped a scheduled run.   |