Monitor and troubleshoot¶

Use readiness for admission, metrics for trends, and audit/authority records for accounting evidence. A green process health check alone does not mean requests can be admitted.

Health and signal channels¶

Signal	Endpoint / transport	Use
Process liveness	`GET :9090/healthz`	Is the gateway process serving?
Readiness	`GET :9090/readyz`	Can this profile admit traffic under its current dependencies/gates?
Metrics	`GET :9090/metrics`, Prometheus	Latency, requests, cost estimates, refusals, audit health
Traces	Opt-in `otel` or governed exporters, OTLP HTTP/gRPC	Request/attempt timing and bounded metadata
Governed export	`observability.exporters` + policy, OTLP logs	Audit-record copies and egress content per team/user (details)
Usage windows	Gateway to CP `/v1alpha1/usage`	Control-plane analytics; not OTLP
Audit	Configured per-instance sinks/WAL	Request evidence and exact-byte chain verification

Health and metrics are unauthenticated by design. Restrict the admin network path. The static console is public/data-free; its data APIs require authentication.

Start with the Grafana dashboard. Configure your Prometheus scrape or collector receiver separately from the OTLP trace receiver. The chart does not install a ServiceMonitor.

Signals worth alerting on¶

Signal	Investigate
Sustained readiness failures / 503s	Policy sync, identity fingerprint, DB availability, authority grant/journal, audit WAL pressure
`inferplane_pricing_miss_total`	Missing route rates or incorrect pricing metadata
`inferplane_audit_write_failures_total`	Sink permissions, capacity, downstream availability
`inferplane_audit_buffer_utilization_ratio`	Retained WAL records / 1024; at 1 or above, new inference is blocked. Check `/readyz` for a latched WAL failure even below 1
`inferplane_audit_anchor_failures_total`	Object storage/IAM/retention configuration
`inferplane_audit_auto_mode_unparsed_total`	Claude Code auto mode block calls or server safeguards the console could not read (ADR-057); a rise suggests a client format change
`gen_ai_server_time_to_first_token_seconds`	Upstream or request-path latency
`inferplane_fallback_total` / circuit state	Provider failure or incompatibility
`inferplane_usage_windows_dropped_total`	Analytics delivery loss; distinguish from authority accounting
`process_resident_memory_bytes`, `go_memstats_heap_inuse_bytes`, `go_goroutines`	Memory growth: the gateway holds several copies of every request body being processed (about 5.5× measured), so concurrent large bodies dominate
`inferplane_model_fallback_total`	Clients requesting unrouted model IDs (a typo or a newer alias); the log line `router: model fallback` names the requested ID
`inferplane_inflight_refusals_total`, `inferplane_inflight_request_bytes`	In-flight body budget (`server.max_inflight_request_bytes`) too small for the workload, a burst of large requests, or a slow or stalled request body holding its reservation until it finishes or the connection closes (size the budget at several times `server.max_request_bytes`; `route` is `generation` or `count`)

Choose thresholds from measured traffic and the selected profile. Prometheus spend is observational; integer accounting records and authority commitments determine available balance.

Troubleshooting by symptom¶

Symptom	First checks
401	Virtual key vs provider/admin token; selected database; revocation and identity bindings
403	Model/team access, privacy/region restriction, policy compatibility
402 in authority profiles	Money exhausted or conservative bound exceeds available balance
429	Rate/token quota exhaustion; inspect the enforcing scope
503 before first request	Required sync, shared namespace/binding, initial grant, DB/journal availability
503 with `Retry-After: 2` during a burst of large requests	In-flight body budget (`server.max_inflight_request_bytes`); `inferplane_inflight_refusals_total`
Retryable grant wait	Respect `Retry-After`; do not lower accounting bounds to force admission
HTTP 200 stream ends early	Inspect terminal protocol error, observed usage and retained liability
Count returns zero while generation refuses	Counts remain local/200; check readiness and key-store availability
Missing model in Codex	Explicit tool/Responses capability, native catalog mapping, installed client version
Lower cache hit rate	Protocol conversion, masking, model switching, changed request bytes

Read outcome and cost evidence¶

The gateway Overview, Usage and Logs use observed completion records (ADR-053). They include costless failures previously absent from analytics. The denominator is request_completed, not all received traffic or unique requests/sessions. request_started refusals, unauthenticated 401s, admin events and client-local failures are excluded; retries/fallbacks may have different recording granularity across ingresses. A succeeded record describes transport, not model answer quality or completion of the user's task.

Investigate (failed + partial) / completed with its numerator, denominator and unknown count. Partial takes precedence even with HTTP 200; missing/invalid outcomes remain unknown. With no completions the ratio is —. Unknown outcomes make the observed proportion incomplete, rather than successful. Metrics traffic and this denominator are separate observations.

Read cost state before amount: estimated is a usage/rate-based estimate (zero is valid), uncertain is a recorded amount with incomplete evidence, legacy_unknown lacks usage evidence, unpriced lacks pricing and unavailable has no Cost record. Unpriced/unavailable numeric zeros are not known dollar amounts. Estimated totals and uncertain/legacy recorded totals are separate; the existing Cost-present financial totals remain unchanged. No state is a supplier invoice or permission to refund retained liability.

For Responses streams the bounded reasons are context_length_exceeded, invalid_encrypted_content, request_canceled, upstream_timeout, stream_incomplete, response_incompatible, usage_missing, client_write_failed and upstream_error. Known provider codes require typed provider evidence. Cancellation/deadline and gateway reasons require explicit runtime evidence; raw error text and native provider-owned frames are not parsed to guess a code. Unknown nonempty analytics codes become upstream_error.

Before streaming commits, existing HTTP/retry rules apply. After commit, the response retains HTTP 200 with partial=true and the same bounded code in audit and a gateway-generated SSE error when writable; it is not retried. Client-write failure records client_write_failed without forcing another SSE write. Native success bytes remain intact. Observed usage still follows the existing exactly-once finish/settlement path; missing usage is not fabricated as zero.

Check legacy_projection_records, /admin/analytics/health, queue diagnostics and retained source segments together. Version 0 means source evidence has not been replayed; version 1 can still be legacy_unknown if the original lacked usage evidence. Zero legacy rows does not prove full historical collection. Retention gaps, the local nonblocking 4096-entry queue's drops, collection lag, truncated JSONL tails and requests with only request_started limit coverage. Missing fields from old APIs require an evidence-unavailable notice.

Use the API examples and recovery procedure. This projection adds no metric labels, raw prompts/errors, key/session identifiers or financial authority. Default/local, node-local authority and shared-governance dependency contracts retain their existing profile-specific limits.

Verify records¶

bin/mayu audit verify --file /path/to/instance-audit.jsonl
bin/mayu report --file /path/to/instance-audit.jsonl --by team,model

Keep instance/restart segments distinct. A valid local chain does not prove resistance to a host rewriting its history; configure and verify external anchoring when that evidence is required.

For an issue report, include the commit/image, profile, client version, status/error class, sanitized configuration, and whether the failure occurs before egress or during streaming. Exclude keys, credentials, raw prompts, identity declarations and DSNs. Send security reports through the private security policy.

Audit durability limits¶

The default chart writes audit records to container stdout (a best-effort sink) with a WAL on the data volume and the local analytics index disabled, so nothing accumulates there during successful WAL operation; see Collect audit records. Any other deployment must configure its sinks explicitly.

audit.failure_mode defaults to buffer_then_block, the only supported mode. A required-sink failure retains that record and all later records in the WAL. At 1024 retained records the writer latches unready: authenticated new inference receives a protocol-specific 503 with Retry-After, and /readyz reports 503. WAL append/truncate failures latch unready immediately. Best-effort sink failures do not close admission. Count APIs remain local/200 and model listing remains available. Health checks continue to report process liveness.

The latch does not recover automatically. Already admitted requests, admin events and count requests may still append records, so 1024 is an admission threshold, not a strict record or byte bound. Startup validates and counts retained WAL records; restarting alone cannot clear a full backlog, and a malformed/truncated backlog refuses startup. No automatic sink replay is performed because a previous partial fan-out may already have delivered some records. Preserve and reconcile evidence using the audit recovery procedure before restart. Readiness and local chain verification do not prove complete historical delivery.

Collect audit records¶

examples/otel-collector/audit-filelog.yaml is an OpenTelemetry Collector (contrib) configuration for a node DaemonSet. It tails /var/log/pods/*/inferplane/*.log, keeps only lines that start with {"schema_version" (mayu also prints startup lines), and exports each body verbatim to S3 as JSONL. Checkpoints and the send queue persist in file_storage.

Do not parse and re-serialize record bodies: each prev_hash covers the previous line's exact bytes, and a rewritten line fails mayu audit verify.
stdout is not a required sink, so the container log is the gateway's remaining local copy until the collector durably enqueues the record. The collector's persistent queue holds a separate local copy pending export. Kubelet rotation is size-based (containerLogMaxSize × containerLogMaxFiles); size it for the longest period when the collector cannot ingest (about 1.3 KB per request).
Verify each instance's collected records at the destination, together with external anchoring: a truncated or restarted chain still verifies without an anchor.

Verified on 2026-09-25 with otelcol-contrib 0.119.0 against a CRI-format log that included a split (partial) line: 41 records exported byte-identically and the collected file verified with mayu audit verify. The S3 upload itself was not exercised.

Governed OTLP export¶

mayu can send traces, audit-record copies and egress content directly to OTLP collectors. Two layers decide what goes where:

Local ceiling — observability.exporters in mayu's config declares every collector a policy may select: endpoint, protocol, TLS, header credentials (env/file refs only) and the signals it may carry. Only this file can add a destination.
Policy choice — an observability rule in a GovernancePolicy (local files or the control plane) selects, per team or user, which declared exporter receives which signal.

The control plane is not on this path. It keeps receiving only consumption reports, leases and usage windows on its own channels.

"observability": {
  "service_name": "inferplane",
  "exporters": {
    "corp": {
      "endpoint": "otel-collector.internal:4318",
      "signals": ["traces", "audit"],
      "headers": { "authorization": { "env": "OTEL_COLLECTOR_AUTH" } }
    },
    "vault": {
      "endpoint": "secure-collector.internal:4317",
      "protocol": "grpc",
      "signals": ["content"],
      "data_boundary": "internal",
      "max_content_bytes": 2097152
    }
  }
}

apiVersion: inferplane.dev/v1alpha1
kind: GovernancePolicy
metadata:
  name: engineering-observability
spec:
  subject:
    team: engineering
  rules:
  - name: export
    failurePolicy: FailOpen
    observability:
      traces: { exporter: corp, samplePercent: 20 }
      audit: { exporter: corp }
      content: { exporter: vault }

Signal	What is exported	Notes
`traces`	The request span (GenAI and `inferplane.*` attributes)	`samplePercent` 1–100, default 100
`audit`	Each audit record as an OTLP log (`event_name` `inferplane.audit`)	Body is the canonical chain line; `inferplane.audit.hash` equals the next record's `prev_hash`
`content`	Egress request body and, when buffered, the response (`inferplane.content`)	Same bytes as body capture: after routing and masking, only for requests that reached a provider; streams carry no response

An exported audit record is the same canonical line every audit sink receives, so it also carries the ADR-057 evidence fields a record has: attempts, client (including the key-scoped client.session_ref digest), auto_mode and governance. A collector that receives audit copies can therefore group one key's requests by session, as the audit file can. Export adds no metric label: inferplane_otel_export_records_total keeps exporter, signal and outcome.

Rules a data plane enforces:

An observability rule must be FailOpen. Export is best-effort; the WAL and required audit sinks remain the evidence of record.
A rule that names an undeclared exporter, or a signal the exporter does not allow, is refused. Local policy files fail the boot or reload. Control-plane documents are rejected per document and reported in Rejections. A data plane without an observability block refuses every observability rule.
Keep observability rules in their own documents: a rejected document is removed whole, including any budget rule in it. In required-identity mode one rejection keeps the previous distributed set.
When several policies match a request, traces and audit go to the union of the named exporters (traces at the highest samplePercent per exporter). Content is exported only when every matching rule names the same exporter.
A content exporter must declare data_boundary: internal and use TLS unless its endpoint is loopback. Records larger than max_content_bytes (default 2 MiB) are dropped, never truncated.
The gateway draws the sampling decision: a random draw per request and target, not deterministic on the trace id, so replicas and upstream services do not sample the same traces. A span no exporter selected is not recorded. An incoming traceparent still joins the client's trace id, but its sampled flag neither suppresses nor forces export. The legacy otel block keeps working as a local baseline exporter with these same semantics (its sample_ratio no longer follows a parent's sampled flag); the name otel is reserved.
Each exporter holds at most queue_bytes (default 64 MiB) of queued records while its collector is slow; further records are dropped. The queue's slots take about 1/16 of queue_bytes more, allocated at startup. Batches are sent every second or at 512 records. HTTP 429/502/503/504 and gRPC Unavailable, ResourceExhausted, DeadlineExceeded and Aborted are retried twice. Watch inferplane_otel_export_records_total{exporter,signal,outcome}. withheld_limit (a runtime limit withheld a policy selection) and sampled_out (a trace target's sampling skipped the span) are outcome values of that metric too. inferplane_otel_export_decisions_total{signal,decision} counts what governed export decided (selected, no_rule, content_conflict, withheld_limit); neither family has a team, user or key label.
Changes to observability need a restart; SIGHUP reload does not rebuild exporters. Shutdown drains the audit writer first, then flushes exporters for up to 5 seconds.

Limits: metrics stay Prometheus exposition; there is no OTLP metric export. An operator who controls mayu's host can edit the local ceiling, so export that users cannot turn off requires mayu to run on a host they do not administer.

Observability console¶

Full administrators can open Observability in the gateway console (/admin/ui/). It shows each declared exporter (endpoint, protocol, TLS, data boundary, header names, declared and current signals, queue bytes), records per signal since the process started (sent, dropped, withheld by a limit, sampled out, last success, last error), governed decisions per signal (exported, no rule, rules disagree on content, withheld by a limit) and every observability rule with its sampling after limits. The last error is a status code or a fixed text such as collector unreachable or span export failed, never collector content. The legacy otel baseline is not a policy selection, so a span only it exports counts as no_rule in the decisions. Counters reset when mayu restarts; use inferplane_otel_export_records_total and inferplane_otel_export_decisions_total for history. The legacy otel exporter is listed read-only.

Edits are opt-in in the bootstrap config:

"policies": ["/etc/inferplane/policies", "/var/lib/inferplane/console-observability.yaml"],
"observability": {
  "exporters": {
    "corp": { "endpoint": "otel-collector.internal:4318", "signals": ["traces", "audit"] }
  },
  "console": {
    "state_file": "/var/lib/inferplane/observability-limits.json",
    "policy_file": "/var/lib/inferplane/console-observability.yaml",
    "content_rules": false
  }
}

Runtime limits (state_file): per declared exporter, pause it, keep a subset of its declared signals, lower max_content_bytes, or cap trace sampling with max_sample_percent. A limit only narrows: it cannot add a signal, raise a size, or change an endpoint, protocol, TLS setting or header. Limits withhold export when the decision is made and again when a queued audit or content batch is sent: records queued before a pause are dropped then and counted as withheld_limit, while a batch already being sent, retries included, completes. Spans already handed to the trace SDK's batch processor are not rechecked: a pause stops new spans at the routing decision, and spans already batched may still be sent within the batch delay (about 5 seconds by default). Limits never reject a policy document, so a control-plane document keeps its budget rules. They stay editable with a control plane because they narrow the local ceiling. mayu writes the file (0600, atomically) and loads it at boot; a file that does not parse refuses boot instead of resuming paused export. Entries the bootstrap no longer supports are narrowed and logged. Edit the file by hand only while mayu is stopped.
Console rules (policy_file): GovernancePolicy documents named console-…, one FailOpen observability rule each, written by the console and loaded by the local policy file channel like every other policy file. The file must also be listed in policies, so it cannot be combined with control_plane; with a control plane the console shows rules read-only. Rules in other policy files are read-only too. Selecting content needs content_rules: true; without it the console can keep or remove a content selection, never add or redirect one. mayu creates an empty file on first boot.
Edit flow: every edit is previewed first (dry_run), showing the changed fields, the rules whose export changes, or the decisions per subject. Apply sends the same body with the revision the preview saw (409 if anything changed in between), persists, applies (rules through a policy reload; a failed reload restores the file) and appends admin_observability_changed with the administrator and the SHA-256 of the persisted state before and after. Edits are refused while the audit chain is not accepting records.
Scope: one mayu process. The state file, the console policy file and the counters belong to the mayu process that answered; the view's header names it (instance in the status, the audit chain's instance id). With several replicas (the ADR-046 shared profile allows replicaCount above 1), a pause or a rule edit applies only to the replica that served the request, and a later refresh may be answered by another replica. Keep one replica per edit target, or make the same edit on each replica by reaching its pod directly (for example kubectl port-forward to that pod's admin port) and check the header each time. The files live on the process's data volume, so a pause survives a process restart only on the same volume: with the chart's default emptyDir volume a replaced pod starts without the state file, so the pause lapses, and with an empty console policy file, so console rules lapse too. Set persistence.enabled whenever state_file or policy_file is set; the shared profile then gives each replica its own claim.

Runtime limits and console rules apply without a restart; changes to observability.exporters and observability.console still need one. The API reference lists the request and response shapes.