功能对比
| Tool | Best for | Core LLM observability features | When to add agent observability |
|---|---|---|---|
| Langfuse | Open-source LLM traces and prompt management | Traces, prompts, datasets, evals, scores, and self-hosting. | When traces need tool calls, agent spans, and workflow outcome tracking. |
| Braintrust | Eval-driven AI product teams | Experiments, scorers, datasets, logs, prompt comparison, and feedback. | When evals need to grade complete agent tasks, not only model outputs. |
| Helicone | Gateway-level logging and spend visibility | Request logs, user metadata, latency, cache, rate limits, and model spend. | When a single request log no longer explains multi-step agent failures. |
| OpenTelemetry | Portable instrumentation | Standard spans, metrics, traces, collectors, and vendor export. | When agent steps must connect to services, queues, databases, and infra traces. |
| Datadog | Enterprise monitoring and incident workflows | Dashboards, logs, traces, alerts, SLOs, and LLM/app monitoring integrations. | When AI incidents need the same on-call process as application incidents. |
| AgentOps or agent-specific tools | Agent run debugging | Sessions, tool calls, retries, agent errors, cost, and run timelines. | Use directly when the product is built around autonomous or semi-autonomous agents. |
Direct Answer
The best LLM observability tools expose prompt versions, model requests, traces, eval scores, latency, token usage, cost, and feedback. Start with request and prompt visibility, then move to agent observability when users depend on multi-step tool-using agents.
LLM Observability Checklist
A mature LLM observability setup should connect quality, reliability, and cost. Logging every prompt is not enough if teams cannot compare prompt versions, reproduce failures, or understand budget burn.
[ ] Prompt version and model version [ ] Request and response metadata [ ] Latency, errors, retries, and cache status [ ] Token usage and estimated cost [ ] Evals tied to datasets and releases [ ] User feedback linked to traces [ ] Redaction and retention policy [ ] Alerts for quality, latency, and spend regressions
LLM Observability Versus Agent Observability
LLM observability is the foundation. Agent observability adds plans, tool calls, approvals, retrieved context, retries, intermediate state, and final task success. If your product is a coding agent, browser agent, MCP workflow, or back-office agent, use the agent-specific checklist too.
Pricing And Retention
Cost comes from event volume, indexed logs, trace retention, eval runs, seats, and storage. Keep sensitive prompt logs on the shortest useful retention path and store summarized tool output when full payloads are not needed.
- Use sampling for low-value success traces, but keep failures and release-eval traces.
- Separate prompt debugging retention from compliance or audit retention.
- Track cost per feature, user, model, and workflow.
- Review self-hosting cost against managed plans before assuming open source is cheaper.
Recommended Adoption Path
Start with simple request logs and cost dashboards, add evals before major prompt changes, then instrument traces once the app has chains, retrieval, tools, or agents.
- Week 1: request logs, prompt version, latency, cost, and redaction.
- Week 2: regression datasets, eval scorers, and release comparison.
- Week 3: trace spans for retrieval, tools, retries, and app boundaries.
- Week 4: alerts, on-call routing, and agent-specific dashboards.
常见问题
What are LLM observability tools?
They monitor LLM-powered applications by collecting prompt versions, requests, responses, traces, evals, latency, errors, cost, and feedback.
What is the difference between LLM observability and evals?
Evals measure output quality against tests or scorers. Observability connects evals with live traces, prompts, latency, cost, errors, and user feedback.
When do I need agent observability?
You need agent observability when your app uses multi-step plans, tools, browser actions, MCP servers, approvals, retries, or autonomous workflows.
Can I use OpenTelemetry for LLM observability?
Yes. OpenTelemetry is useful for portable traces and integration with existing observability backends, but you may still need LLM-specific prompt, eval, and cost features.