Observability Basics
Introduction
Observability is a critical aspect of modern software systems. It refers to the ability to understand the internal state of a system by examining its outputs, such as logs, metrics, and traces. By using observability tools and practices, engineers can quickly and accurately diagnose problems, identify performance bottlenecks, and make informed decisions about how to improve the system.
Observability is particularly important in distributed systems, where the complexity of multiple interconnected components can make it difficult to understand the root cause of problems. In these systems, observability helps engineers to identify which component is causing the issue and how it is affecting the rest of the system.
Reading List
- CloudWatch
- Collect your own Premetheus exporter metrics to Grafana Cloud with Grafana agent
- Customize your status page with CloudFlare Worker and Atlassian Status Page
- DataDog
- Dynatrace
- Four Types Metrics in Prometheus
- Grafana
- InfluxDB
- Linux Log Types and Usage
- Logging
- Modern Monitoring System
- Monitoring
- Observability
- Open-Falcon
- Sentry
- Tracing
- Website Health Check with Cloudflare Worker
- What is APM? - Application Performance Monitoring Explained
- Zabbix
- eBPF
There are three main pillars of observability: logs, metrics, and traces.
Logs are records of events that have occurred within a system. They can include information about errors, warnings, and other important events that may indicate a problem with the system. By analyzing logs, engineers can identify patterns and correlations that can help them understand the underlying causes of issues.
Metrics are numerical measurements of a system’s performance and behavior. They can include things like response times, throughput, and resource utilization. By collecting and analyzing metrics, engineers can identify trends and patterns that can help them understand how the system is behaving over time.
Traces are detailed records of a single request or transaction as it flows through a system. They can include information about the components involved, the data being processed, and the time taken to complete the request. Traces can be particularly useful for understanding the root cause of issues in distributed systems, as they allow engineers to see how a request is being handled as it travels through different components.
To effectively use observability, it is important to have a clear understanding of the goals and objectives of the system being monitored. This will help engineers to identify the most relevant logs, metrics, and traces to collect, and to determine what thresholds and patterns to look for.
There are many tools and practices available to help with observability. These can include logging libraries, metrics collectors, and trace analyzers. It is important to choose the right tools for the specific needs of the system being monitored, and to ensure that they are properly configured and integrated into the system.
In summary, observability is a vital aspect of modern software systems. By using logs, metrics, and traces, engineers can understand the internal state of a system and quickly identify and resolve issues. By using the right tools and practices, they can ensure that their systems are reliable, efficient, and performant.
References
- What is Observability? (
https://www.datadoghq.com/blog/observability/) - Observability: The Key to Modern Software (
https://www.datadoghq.com/blog/observability/) - Charity Majors, Liz Fong-Jones, George Miranda - Observability Engineering Achieving Production Excellence (2022)
- Alex Boten - Cloud-Native Observability with OpenTelemetry (2022)
- Alex Pollitt, Manish Sampat - Kubernetes Security and Observability (2021)
https://grafana.com/products/cloud/https://newrelic.com