Ontology-Driven Observability: Building an End-to-End Knowledge Graph at Netflix Scale

13 min read2 days ago--Prasanna Vijayanathan, Renzo Sanchez-SilvaWhen Jake Paul and Mike Tyson stepped into the ring in November 2024, we delivered the fight to roughly 65 million concurrent streams. On an ordinary day our infrastructure takes in more than 2 billion requests and handles upwards of 38 million real-time logging events every second. At that scale, the whole picture doesn’t fit into a single engineering team’s head or onto a single dashboard. With more Live content, Podcasts and real-time streams, our need to understand and update our systems as quickly as possible becomes imperative to provide a good user experience. Understanding the system at this scale and within the time constraints we have, often becomes a daunting exercise in finding a needle in a haystack, in a dark jungle, blindfolded.This post is about that search for the needle.The Four hours before a fixConsider a regression of the kind we see in a given week. The home screen where you see the list of lists of movies, starts taking about 1.5 seconds longer to render. An alert fires. And then we go looking for what caused this issue and how do we fix it?As it so happens, a regression on the application could be related to anywhere in the system — a client app regression from a new version deployment or A/B test rollout, a load shedding in the infrastructure, a GraphQL query responding with bad data because of a service rollout 6 hops downstream, are all suspects at this point. So, answering “what caused the issue?” needs insights from across the system, and this involves combing through layers of data in a meticulous manner.The MELT Layer“What caused the issue?”Metrics, events, logs and traces, or MELT, in the industry’s shorthand, are the right raw materials for such an investigation, and each of them answers roughly a quarter of the question. We collect observability data across these four pillars, unified into a single observability data layer. This lets us observe our systems in real time, across what our users are experiencing, how our applications are doing and what our backend infrastructure and services are processing and responding to.Press enter or click to view image in full sizeMELT — Metrics, Events, Logs and Traces, and how they are captured across the system.The investigation that follows may look like this in each layer:Press enter or click to view image in full sizeWhat each layer of MELT tells us and what it leaves outThe timeline of an investigationSo how long does it take to investigate an issue, figure out the root cause and fix it, so our users stop running into this issue again?Press enter or click to view image in full sizeA timeline of an investigation: ~4 hoursIn our chosen example, four hours. Very little of it went into computing anything difficult. Most of it went into reconstructing relationships that our observability systems already knew but we had never written down anywhere a human or machine could simply read and discover.The disagreement underneathThe instinct here could have been, around 2023, to reach for more data or better dashboards, and while both help at the margins, neither addresses what makes these investigations slow.It’s 2026 now, so we might say, “hey, agents”, but without the right context delivered to the agents, AI systems hallucinate, and especially when teams are on-call and need a fix “now”, we cannot let inaccurate agentic analysis lead us down the wrong path. We need the right needle, not any needle.Each of our systems is internally coherent and externally isolated. Atlas, our telemetry platform, knows applications and their time series.

Incident chat carries the reasoning behind an incident. The hypotheses tried and discarded, the mitigation that worked, who knew why, is all context in the incident’s Slack channel and that knowledge evaporates when the channel goes quiet. So a 200-message, 50-person, 15-service incident channel becomes a structure, converting the operational chaos of an incident into ordered triples:Press enter or click to view image in full sizeCreating determinism from non-determinismBuilding this graph means using tools that are inherently stochastic. LLMs summarize discussions. Fuzzy matching resolves service names. A probabilistic engine proposes causal chains. None of these produce the same output twice with certainty, but what comes out at the end of our system is a deterministic graph that answers a SPARQL query the same way every time.Get Netflix Technology Blog’s stories in your inboxJoin Medium for free to get updates from this writer.Remember me for faster sign inThe ontology is the mechanism that creates this determinism through its pipelined approach. We think of the pipeline as a ladder that drives entropy down, where entropy means the proportion of information still trapped in unstructured form. For example, at the start of this pipeline, an LLM could say anything, but it cannot emit a triple that violates the type system. We constrain the output space tightly enough and a stochastic process yields a deterministic contract:# The model may produce any prose. Its output MUST fit these predicates:category IN ["incident", "question", "discussion", "decision"]sentiment IN ["positive", "neutral", "negative", "urgent"]# Result — a deterministic triple:thread:123 slack:category "incident" .thread:123 slack:isIncidentRelated "true"^^xsd:boolean .There are five layers that get us down this ladder. Any single step of this ladder is non-deterministic. But in aggregate, the chain becomes deterministic. That inversion is the whole trick. We describe them below:Press enter or click to view image in full sizeThe five layers to create determinism from non-determinismEnrichers: Agents as entropy reducersWe use what we call, enrichers, to reduce entropy across the knowledge graph. Dozens of enrichers run over the graph. Each one of them takes noisy input and emits typed triples, and each reads what earlier enrichers wrote.