Observability: Logs, Metrics & Traces

4 · Error tracking with Sentry

Logs scroll past. An error deserves to be caught, grouped with its identical siblings, counted, and surfaced with a full stack trace and the surrounding context. That's error tracking, and Sentry is the standard. Its FastAPI integration captures unhandled exceptions automatically.

app/main.py

import sentry_sdk
from sentry_sdk.integrations.fastapi import FastApiIntegration
from sentry_sdk.integrations.starlette import StarletteIntegration

sentry_sdk.init(
    dsn=settings.SENTRY_DSN,
    environment=settings.ENV,            # "production" / "staging"
    release=settings.GIT_SHA,            # tie errors to a deploy
    traces_sample_rate=0.1,             # sample 10% for performance
    send_default_pii=False,            # don't ship user data by default
    integrations=[StarletteIntegration(), FastApiIntegration()],
)

Unhandled exceptions now appear in Sentry without another line of code. But the gold is context: attach the values that make an error reproducible before you re-raise or capture.

with sentry_sdk.new_scope() as scope:
    scope.set_tag("feature", "ask")
    scope.set_context("retrieval", {"doc_id": doc_id, "k": top_k})
    scope.set_user({"id": user_id})           # grouped, searchable
    try:
        answer = await run_ask(question, doc_id)
    except LLMTimeout as e:
        sentry_sdk.capture_exception(e)        # with all the context above
        raise

5 · Metrics, the RED method & Prometheus

Metrics are cheap numbers over time, and they come in three flavours. Get these straight — it's a common interview gap.

For request-serving systems the RED method tells you which metrics actually matter:

• Rate — requests per second (a counter's rate).
• Errors — the rate of failing requests (a counter labelled by status).
• Duration — the distribution of latencies (a histogram, so you can ask p50/p95/p99).

Expose them on a /metrics endpoint that Prometheus scrapes every few seconds.

app/metrics.py

from prometheus_client import Counter, Histogram, make_asgi_app

REQUESTS = Counter(
    "http_requests_total", "Total HTTP requests",
    ["method", "path", "status"],          # labels = dimensions to slice by
)
LATENCY = Histogram(
    "http_request_duration_seconds", "Request latency",
    ["method", "path"],
)

# mount at /metrics — Prometheus scrapes this
metrics_app = make_asgi_app()

# in the middleware, after timing the request:
REQUESTS.labels(request.method, route, response.status_code).inc()
LATENCY.labels(request.method, route).observe(duration_ms / 1000)

Then Grafana reads Prometheus and draws the dashboard: a graph of request rate, an errors-per-second panel, and a p95-latency line per route. A good DocChat dashboard puts RED for /ask top-left so one glance tells you if the service is healthy.

6 · Distributed tracing with OpenTelemetry

A trace follows one request as a tree of spans. The root span is the whole /ask request; child spans are the steps inside it — the DB query for retrieval, the outbound HTTP call to the LLM, the post-processing. Each span has a start, a duration, and attributes. Seeing them stacked tells you instantly where the 9 seconds went.

OpenTelemetry (OTel) is the vendor-neutral standard, and June 2026's default. The magic is auto-instrumentation: install the integrations and FastAPI, SQLAlchemy, and outbound HTTP get traced with zero application code.

app/tracing.py

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor

def configure_tracing(app, engine) -> None:
    provider = TracerProvider()
    provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
    trace.set_tracer_provider(provider)

    FastAPIInstrumentor.instrument_app(app)        # every route → a root span
    SQLAlchemyInstrumentor().instrument(engine=engine)   # every query → a span
    HTTPXClientInstrumentor().instrument()         # every outbound call → a span

tracer = trace.get_tracer("docchat")

Context propagation is what makes it "distributed": OTel injects a traceparent header on outbound calls so the next service continues the same trace instead of starting a new one. One id links the whole journey across process boundaries.

Auto-instrumentation won't know about your business steps, though — like the LLM call. Wrap those in a manual span and tag it with the attributes you'll want to filter on:

with tracer.start_as_current_span("llm.chat") as span:
    span.set_attribute("llm.model", model)
    span.set_attribute("llm.prompt_tokens", prompt_tokens)
    answer = await client.messages.create(...)   # the slow part, now visible
    span.set_attribute("llm.completion_tokens", answer.usage.output_tokens)

7 · Health, readiness & LLM cost metrics

Orchestrators (Kubernetes, ECS, a load balancer) need to ask your service two different questions, so expose two endpoints:

@app.get("/healthz")            # LIVENESS: is the process alive at all?
async def healthz():
    return {"status": "ok"}     # cheap, no dependencies — never restart on a DB blip

@app.get("/readyz")             # READINESS: can it serve traffic right now?
async def readyz():
    await db.execute(text("SELECT 1"))     # check real dependencies
    return {"status": "ready"}

Liveness failing means "restart me". Readiness failing means "stop sending me traffic until I recover" — distinct actions, which is why they're separate endpoints.

Now the bit unique to AI products: treat LLM cost and tokens as first-class metrics. They're a real line on your AWS-and-Anthropic bill and the thing finance asks about. Record them like any other RED-style metric:

from prometheus_client import Counter, Histogram

LLM_TOKENS = Counter("llm_tokens_total", "Tokens used", ["model", "kind"])
LLM_COST = Counter("llm_cost_usd_total", "LLM spend in USD", ["model"])
LLM_LATENCY = Histogram("llm_call_seconds", "LLM call latency", ["model"])

def record_llm(model, in_tok, out_tok, cost_usd, seconds):
    LLM_TOKENS.labels(model, "input").inc(in_tok)
    LLM_TOKENS.labels(model, "output").inc(out_tok)
    LLM_COST.labels(model).inc(cost_usd)
    LLM_LATENCY.labels(model).observe(seconds)

Graph llm_cost_usd_total in Grafana and you can answer "what does a single /ask cost us on average?" and catch a runaway prompt the day it ships, not when the invoice lands.

8 · Alerting on symptoms, not noise

An alert that fires when nothing is actually wrong trains everyone to ignore alerts — and the night something is wrong, nobody looks. The discipline: alert on user-visible symptoms tied to your SLOs, not on internal causes.

• Good (symptom): "p95 latency on /ask > 5s for 10 minutes" or "error rate > 2% for 5 minutes." The user feels these.
• Noisy (cause): "CPU > 80%." High CPU might be fine. Alerting on it pages you for non-problems.

Anchor thresholds to a Service Level Objective — e.g. "99% of /ask requests succeed within 5s over 30 days." Your alert fires when you're burning that error budget too fast. Use causes (CPU, queue depth) as dashboard signals you check after a symptom alert wakes you, not as pagers themselves.

9 · Build it

Assuming logging, tracing, Sentry, and the metrics are configured at startup (sections 2–7), the route itself ties the pillars together:

app/routes/ask.py

import time
import sentry_sdk
from opentelemetry import trace
from app.logging import log
from app.metrics import record_llm

tracer = trace.get_tracer("docchat")

@router.post("/ask")
async def ask(body: AskIn, user=Depends(current_user)):
    # request_id is already bound by LoggingMiddleware (section 3)
    log.info("ask_started", doc_id=body.doc_id, q_len=len(body.question))

    with tracer.start_as_current_span("ask") as root:
        root.set_attribute("doc_id", body.doc_id)
        try:
            # --- retrieval span (child) ---
            with tracer.start_as_current_span("retrieval") as rspan:
                chunks = await retrieve(body.doc_id, body.question, k=5)
                rspan.set_attribute("chunks", len(chunks))
                log.info("retrieval_done", chunks=len(chunks))

            # --- LLM span (child) with cost + tokens ---
            with tracer.start_as_current_span("llm.chat") as lspan:
                lspan.set_attribute("llm.model", MODEL)
                t0 = time.perf_counter()
                resp = await client.messages.create(
                    model=MODEL,
                    max_tokens=1024,
                    messages=build_prompt(chunks, body.question),
                )
                seconds = time.perf_counter() - t0
                in_tok = resp.usage.input_tokens
                out_tok = resp.usage.output_tokens
                cost = price(MODEL, in_tok, out_tok)   # your pricing table
                lspan.set_attribute("llm.prompt_tokens", in_tok)
                lspan.set_attribute("llm.completion_tokens", out_tok)
                lspan.set_attribute("llm.cost_usd", cost)

            # --- first-class cost telemetry ---
            record_llm(MODEL, in_tok, out_tok, cost, seconds)
            log.info("ask_completed",
                     model=MODEL, prompt_tokens=in_tok,
                     completion_tokens=out_tok, cost_usd=round(cost, 5),
                     llm_seconds=round(seconds, 2))
            return {"answer": resp.content[0].text}

        except Exception as e:
            with sentry_sdk.new_scope() as scope:
                scope.set_tag("feature", "ask")
                scope.set_context("ask", {"doc_id": body.doc_id})
                scope.set_user({"id": user.id})
                sentry_sdk.capture_exception(e)
            root.record_exception(e)            # mark the trace as errored too
            log.exception("ask_failed", doc_id=body.doc_id)
            raise HTTPException(502, "answer generation failed")

Now trace a slow /ask the way you would on the job: the Grafana RED panel shows p95 climbing, you open a slow trace and see the llm.chat span is 8s while retrieval is 40ms, you grep logs by that request_id and the ask_completed line shows prompt_tokens: 31000 — retrieval handed the model too much context. Three pillars, one story, root cause in minutes.

10 · Check yourself

Answer from memory — retrieval is what moves this from "I read it" to "I know it".