In this talk, we’ll delve into troubleshooting complex systems comprising hundreds or even thousands of services. We’ll address key questions like: What information do we require about each service? How do we structure dashboards for rapid comprehension of the system's current state? We'll also explore the effective utilization of observability signals - metrics, logs, traces, and profiles - discussing their respective applications and the insights they provide. Additionally, we'll examine automation strategies for root cause analysis, considering both complete and partial automation approaches.

Observability is critical to any infrastructure as it enables teams to identify and address issues promptly. Nevertheless, achieving system observability comes with its own set of challenges. It is a time - and resource-intensive process as it necessitates the incorporation of instrumentation into every application.

In this talk, we will delve into the gathering of telemetry data, including metrics, logs, and traces, using eBPF. We will explore tracking various container activities, such as network calls and filesystem operations. Additionally, we will discuss the effective utilization of this telemetry data for troubleshooting.