AI Observability & Production Monitoring

1. AI Observability#

AI observability refers to the practice, and the collection of tools and techniques, used to gain genuine visibility into how an AI system is actually behaving in production — not just whether it’s technically running without crashing, but what it’s actually doing, step by step, and how well it’s actually doing it.

To understand why this needs to be its own distinct thing, it helps to compare it to traditional software monitoring, which we touched on briefly in the LLMOps explanation. Traditional monitoring tools are very good at answering questions like “is the server up,” “how many requests are coming in,” and “how long is each request taking to process.” These are all still genuinely useful and relevant questions for an AI system too. But AI systems introduce a whole new category of failure that these traditional tools were never built to catch — a system can return a perfectly normal, well-formatted response, in a perfectly normal amount of time, with zero technical errors anywhere in the pipeline, while that response is nonetheless confidently wrong, based on a retrieval step that quietly started pulling irrelevant information, or produced by a prompt change that subtly degraded quality in a way nobody would notice just by glancing at server logs.

Good AI observability specifically layers in visibility into the things that are unique to AI systems: what exact prompt was actually sent to the model, what the model actually returned, which specific tools or retrieval steps were called along the way and what they actually returned, how many tokens were used and what that cost, and, ideally, some genuine signal about whether the actual output was good. This connects very directly back to the AI evaluation concepts from the previous explanation — observability is, in a real sense, what lets you actually apply evaluation-style thinking continuously, to real production traffic, rather than only to a fixed set of test cases you ran once before deployment.

2. LangSmith#

LangSmith is an observability and evaluation platform built by the same company behind LangChain and LangGraph, which we covered in the earlier Agent Frameworks explanation, and its biggest strength comes directly from that connection — it integrates especially deeply and smoothly with LangChain and LangGraph-based applications, since it’s built by the same team and designed to plug in naturally alongside them.

For a team building their agent using LangGraph specifically, LangSmith tends to offer a particularly rich, detailed view into exactly what’s happening — tracing individual steps through an agent’s graph, showing how the internal state changed at each node, and even letting a developer take a past trace and “replay” it against a different, updated model version to see how the behavior would have changed. It also includes a prompt playground, letting a developer open up a specific past trace, tweak the prompt right there, and immediately rerun it to see whether the change actually helps, which is a genuinely convenient way of debugging a poorly performing response.

The main practical tradeoff worth understanding is that LangSmith’s deepest, most seamless value shows up specifically when you’re already using LangChain or LangGraph — while it does support other frameworks too, largely through the OpenTelemetry standard we’ll discuss shortly, teams not using LangChain-family tools sometimes find that other options fit a bit more naturally. It’s generally the strongest, most natural pick specifically for teams already building on LangChain or LangGraph who want the tightest, most seamless integration with the tools they’re already using.

3. LangFuse#

Langfuse is one of the most widely used open-source alternatives in this space, and its defining characteristic, compared to LangSmith, is a strong commitment to being open-source, self-hostable, and genuinely framework-agnostic — meaning it’s built to work equally well regardless of which specific agent framework, or no framework at all, a given team happens to be using.

The “open-source and self-hostable” part matters quite a lot in practice for certain kinds of teams. Since Langfuse can be run entirely on a company’s own servers rather than only being available as someone else’s hosted cloud service, teams with strict data residency requirements, regulatory compliance needs, or a general desire to avoid depending entirely on a third party for something as sensitive as their production AI traffic tend to gravitate toward it specifically for this reason. It also offers a genuinely full-featured hosted cloud option for teams that would rather not deal with the extra work of running their own infrastructure, so the self-hosting is a genuine option rather than a requirement.

Langfuse has also standardized on OpenTelemetry (which we’ll cover as its own topic shortly) as its underlying foundation, which reinforces its framework-agnostic design — since it can ingest traces from essentially any LLM SDK or agent framework through this shared, common standard, rather than being built primarily around one particular company’s own specific framework. This combination of open-source flexibility, genuine data ownership, and broad framework compatibility has made it a particularly popular default choice for teams that specifically want to avoid getting locked into one vendor’s particular ecosystem.

4. Phoenix#

Phoenix is an open-source observability and evaluation tool built by Arize, a company that was actually doing traditional machine learning observability — monitoring more classic, non-LLM machine learning models in production — well before large language models became the dominant focus of the industry. This history matters, because it shows up directly in Phoenix’s particular strengths.

Phoenix’s standout area is specifically around rigorous evaluation, and especially evaluation of RAG systems — this connects very directly back to the whole RAG evaluation explanation we covered earlier in this series, including things like measuring retrieval quality, faithfulness, and catching a kind of subtle, easy-to-miss problem sometimes called “embedding drift,” where the quality of retrieval quietly gets worse over time as data or usage patterns shift, without anyone actually noticing the gradual change until it’s already caused real, accumulated harm. Because Arize’s broader background is genuinely rooted in the more established, rigorous evaluation traditions of traditional machine learning, Phoenix tends to bring a comparably rigorous, statistically-grounded evaluation methodology into the LLM observability world, rather than treating evaluation as a looser, more informal afterthought.

Phoenix is fully open-source and free, distinguishing it from Arize’s separate, full commercial enterprise platform (sometimes called Arize AX), which offers additional managed, enterprise-focused capabilities on top. For teams specifically running RAG-heavy applications who want to catch subtle retrieval quality problems before their users start noticing and complaining, Phoenix tends to be a particularly strong, well-suited choice.

5. Weights & Biases#

Weights & Biases (often shortened to W&B) is a platform that, similar to Arize, has genuine roots that go back well before the current wave of large language models — it was originally, and remains, a widely used and well-respected tool specifically for tracking and managing traditional machine learning experiments, like training and comparing different versions of a model during a research or development process.

W&B’s more recent LLM-focused offering (often referred to as W&B Weave) extends that same underlying experiment-tracking philosophy specifically into the world of LLM applications — letting a team systematically track and compare many different prompt versions, model configurations, and evaluation runs against each other, in a way that draws directly on W&B’s long, well-established history of doing exactly this same kind of comparative experiment tracking for traditional machine learning models.

Because of this heritage, W&B tends to be especially well-suited to research-heavy teams that are running a genuinely large number of structured evaluation experiments and want strong, detailed tooling specifically for comparing many different prompt or configuration variants against each other in an organized, rigorous way. It’s generally considered somewhat less focused on full-scale, deep production tracing of live agent behavior compared to some of the other tools discussed here, and more strongly oriented toward this experimentation and research side of the workflow — making it a particularly natural fit for teams whose day-to-day work looks more like a series of structured experiments than pure production incident debugging.

6. MLflow for LLMs#

MLflow is another platform, like W&B, with genuine roots in the broader, more established world of traditional machine learning operations (MLOps) that existed well before large language models became such a central focus — it was originally built as a general-purpose tool for tracking machine learning experiments, packaging and managing trained models, and managing the overall lifecycle of getting a traditional ML model from initial development into production.

MLflow’s LLM-specific capabilities extend this same well-established, general MLOps foundation specifically to cover language-model-based applications — including tracking prompts and their different versions, running and recording evaluations, and tracing the individual steps involved in more complex LLM-based pipelines, playing a broadly similar role to some of the more LLM-native-first tools we’ve already discussed above.

The genuine appeal of MLflow specifically often comes from continuity, particularly for organizations that already have an existing MLOps practice built around it for their other, more traditional machine learning models. For a team that already uses MLflow to manage their traditional ML models, extending that same familiar, already-adopted tool to also cover their newer LLM-based applications can mean less new tooling to learn, and a more unified, single, consistent place to track and manage all of an organization’s models overall — both the more traditional ones and the newer, LLM-based ones — rather than needing to adopt and separately maintain a completely different, additional tool specifically and only for their LLM-based work.

7. OpenTelemetry for AI#

OpenTelemetry (often abbreviated as OTel) is not itself a specific commercial product or company-branded platform, the way LangSmith, Langfuse, Phoenix, W&B, and MLflow all are — it’s an open, vendor-neutral standard for how software should collect and format observability data (traces, metrics, and logs) in the first place, and it long predates the current focus on LLMs specifically, having originally been developed for traditional software and infrastructure monitoring more broadly.

Why does this general-purpose, pre-existing standard matter so much specifically in the AI observability world? Because it addresses a genuinely real, practical concern: vendor lock-in. If your application’s observability data is captured in some proprietary, one-off format that’s specific to only one particular commercial platform, switching to a different platform later on, if you ever want or need to, becomes a genuinely painful, disruptive undertaking, since you’d essentially have to redo all your instrumentation work again from scratch. By building your AI application’s tracing on top of the shared, open OpenTelemetry standard instead, you keep the flexibility to send that same underlying data to different observability backends, or even switch between different providers entirely later on, without needing to fundamentally rip out and completely redo your core instrumentation work.

This is exactly why we noted that Langfuse and Phoenix have both specifically standardized on OpenTelemetry as their underlying foundation — it directly reinforces their framework-agnostic, avoid-lock-in design philosophy. LangSmith, while more tightly coupled to and optimized around the LangChain ecosystem specifically, has also added support for ingesting OpenTelemetry-formatted data. For a team just starting out, building your instrumentation on top of OpenTelemetry from the very beginning is a genuinely sensible practice precisely because it keeps your options open, giving you real flexibility to evaluate and potentially switch between different specific observability tools later, without needing to redo your foundational underlying tracing work each time you might want to make that kind of change.

8. AI Tracing#

Tracing refers to the specific technique of recording the complete, detailed sequence of steps a request actually went through inside your system — capturing not just the final input and the final output, but everything genuinely relevant that happened in between, on the way from one to the other.

This becomes especially important, and especially valuable, once you’re dealing with agent-based systems specifically, rather than a single, simple call directly to a language model. Recall from the AI Agent Fundamentals explanation that an agent might loop through several rounds of perceiving, reasoning, and acting — calling multiple different tools, running several retrieval steps, and making more than one separate call to the underlying language model along the way, all before it ever actually produces its final answer. Without genuine tracing, all a developer trying to debug a problem can actually see is the original question that went in, and the final answer that eventually came out — with the entire, potentially quite complex sequence of steps that happened in between remaining a complete, invisible mystery.

A proper trace instead captures the entire structure of what actually happened — often represented visually as a tree, since steps can branch off and later reconverge (for example, an agent might call two entirely different tools at once, in parallel, before ultimately combining both of their results back together into one unified next step). Each individual step within that trace, often referred to as a “span,” records genuinely relevant details like exactly what specific input that step received, what specific output it actually produced, how long that individual step took, and how many tokens it used. This kind of genuinely detailed tracing is what actually lets a developer look at a specific, individual case that went wrong, and reliably pinpoint exactly which particular step in a potentially long, complex chain of steps was actually the real, underlying source of the problem — rather than having to guess blindly, based purely on the disconnected input and the disconnected final output alone, with no real visibility whatsoever into everything that genuinely happened in between.

9. AI Metrics#

Metrics refer to the specific, quantifiable numbers you actually track over time to understand how your AI system is genuinely performing in aggregate — across a large volume of many different requests overall, rather than looking closely at just one single specific individual trace at a time, the way tracing (which we just discussed) is more naturally oriented toward.

A comprehensive AI observability practice generally tracks several genuinely different categories of metrics together, because each one specifically reveals a different, distinct dimension of overall system health. Performance metrics cover things like latency (which we discussed at length in the previous LLMOps explanation) and overall system uptime — the more traditional, familiar kinds of metrics that would already be genuinely relevant and tracked in any regular, traditional software system, AI-based or not. Cost metrics track things like total token usage and the resulting actual dollar cost, ideally broken down in enough useful, meaningful detail to understand specifically where that cost is actually coming from (which particular feature, which particular user, or which particular specific model is actually driving it). Quality metrics are the ones genuinely unique and specific to AI systems — things like measured accuracy on an ongoing evaluation set, user feedback signals like explicit thumbs up or thumbs down ratings, or automated scores produced by an AI-model-as-judge evaluation approach (which we discussed back in the RAG evaluation and AI evaluation explanations).

The genuine value of tracking metrics like these consistently over time, rather than only ever looking closely at individual traces one at a time, is that they let a team reliably notice broader trends and gradual, otherwise easy-to-miss changes — for example, noticing that average response quality has been slowly, gradually declining over the past several weeks, even though absolutely no single individual trace, examined entirely on its own in isolation, would obviously and immediately look like some kind of dramatic, sudden, glaring failure.

10. Failure Analysis#

Failure analysis refers to the deliberate, structured process of actually digging into cases where an AI system produced a genuinely poor result, specifically to understand why that failure actually happened, and, ideally, to identify a broader, recurring pattern behind it, rather than just noting and dismissing each individual bad response as a completely isolated, one-off, unrelated occurrence.

This connects directly to both tracing and metrics, discussed just above — metrics are typically what first alerts a team that something’s actually genuinely wrong overall (for example, noticing that quality scores have recently dropped, or that user complaints have noticeably increased over some recent period), while tracing is what then actually lets a team dig into specific individual example cases and genuinely understand exactly what happened in each one. Failure analysis is the broader, more disciplined practice of doing this kind of investigation systematically and rigorously, rather than only ever reactively looking into problems one at a time, purely as isolated individual complaints happen to come in.

Good failure analysis generally involves actively looking for genuine patterns across a whole meaningful collection of different individual failures, rather than treating each one as a completely separate, unrelated, one-off incident — for example, correctly noticing that a large cluster of recent failures all specifically involve a particular type of question the retrieval step keeps consistently handling poorly, or noticing that failures have recently spiked specifically since a particular, specific model or prompt update went out. This is directly analogous to the input-distribution-shift detection capability that some observability tools specifically build in — automatically noticing when a meaningful cluster of new, similar-looking user queries has started appearing that the system clearly wasn’t originally well-prepared to properly handle. Genuinely systematic failure analysis is what actually turns individual, scattered incidents into real, durable, lasting improvements — directly feeding back into and informing the prompt testing, regression testing, and prompt optimization practices we covered in the previous LLMOps explanation, rather than each individual failure simply being fixed once, in isolation, and then quietly, silently forgotten about.

11. Production Monitoring#

Production monitoring is really the ongoing, continuous, always-on operational practice that ties everything we’ve covered throughout this entire section together — the actual, real, sustained discipline of continuously watching how an AI system is genuinely behaving while it’s actually live and being used by real, actual users, and reliably being alerted quickly and promptly when something has actually gone wrong.

This is meaningfully different from the testing and evaluation work we covered in the previous LLMOps explanation, and it’s worth being clear about that distinction. Testing and evaluation are primarily about checking a given system’s behavior before a specific change actually goes out to real users — running it carefully against a defined, curated set of test cases to try to catch problems proactively, in advance. Production monitoring is about continuously watching the system’s genuine, real, live behavior after it’s already out there and running for actual real users, catching things that inevitably weren’t, and often genuinely couldn’t have been, fully and completely anticipated and caught by testing alone, no matter how thorough and careful that original testing effort might have genuinely been.

A well-designed production monitoring setup generally combines several of the pieces we’ve already discussed throughout this whole explanation into one single, cohesive, ongoing practice: tracing to give a team genuine visibility into individual, specific cases when they need to dig deeper into any one of them, metrics tracked continuously over time to reveal broader trends and catch subtle, gradual degradation, clear and well-configured alerting so the right relevant people get notified quickly and promptly when something genuinely important actually changes for the worse, and disciplined, structured failure analysis to turn whatever gets caught into real, lasting, durable improvements going forward. Taken together, this is really what closes the full loop we’ve been building toward across this entire Week 10 section — building a genuinely good AI system isn’t just about the initial work of getting it built and correctly working once; it’s about a whole sustained, ongoing operational discipline that keeps it genuinely working well, reliably, over real, extended time, even as usage patterns naturally shift, as underlying models get updated by their providers, and as genuinely new, previously unanticipated edge cases inevitably continue to show up along the way.