Monitoring with Prometheus and Grafana: A Complete Guide

Monitoring modern infrastructure and applications is essential for ensuring high performance and reliability. Two popular open-source tools, Prometheus and Grafana, work seamlessly together to help you collect, analyze, and visualize metrics. In this post, we will dive into what these tools are, how they work together, and provide some real-world use cases for monitoring with Prometheus and Grafana.

What is Prometheus?

Prometheus is an open-source time-series database designed specifically for monitoring systems. It pulls metrics from various endpoints using HTTP and stores them efficiently in a format optimized for metrics data. It supports querying using a specialized language called PromQL and has a built-in alerting mechanism via AlertManager. Prometheus excels in scaling, dynamic service discovery, and reliability.

What is Grafana?

Grafana is a data visualization tool that integrates with several data sources, including Prometheus. It provides an intuitive UI for creating dashboards and graphs that allow you to monitor metrics in real time. Grafana also comes with features like alerting, user-defined thresholds, and pre-built dashboards, making it a popular choice for visualizing Prometheus metrics.

Why Use Prometheus with Grafana?

While Prometheus excels at collecting and querying metrics, its native visualization tools are basic. Grafana complements Prometheus by providing rich, customizable dashboards for visualizing the metrics data stored in Prometheus. This combination creates a complete monitoring stack that is scalable, open-source, and flexible for modern applications, particularly in environments like Kubernetes and microservices.

Benefits of Combining Prometheus with Grafana

  • Real-Time Visualization: Get real-time insights into your infrastructure and applications.
  • Alerting: Both Prometheus and Grafana support alerting, but Grafana offers more integration with third-party tools like Slack, PagerDuty, etc.
  • Flexibility: You can pull data from multiple sources and correlate them in Grafana dashboards.
  • Scalability: Whether you’re running on bare metal, virtual machines, or containers, the Prometheus and Grafana stack is designed to scale.

Setting Up Prometheus and Grafana

Step 1: Install Prometheus

Download and install Prometheus from its official site. Configure prometheus.yml to scrape metrics from your application or system, and run it locally or on your server.

Example prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

Step 2: Install Node Exporter (Optional)

For system-level metrics (CPU, memory, disk usage), you can use Node Exporter on each host you want to monitor. This exposes metrics that Prometheus can scrape.

./node_exporter &

Step 3: Install Grafana

Head over to Grafana’s download page and install it on your system. Once installed, you can access the UI via http://localhost:3000.

Step 4: Connect Grafana to Prometheus

  1. In Grafana, navigate to Configuration > Data Sources.
  2. Select Prometheus and enter your Prometheus URL (http://localhost:9090).
  3. Click Save & Test to ensure the connection is successful.

Step 5: Create Dashboards in Grafana

Once the data source is configured, you can start building dashboards. Grafana provides a flexible interface where you can create various types of visualizations like graphs, heatmaps, and gauges.

Example PromQL query to monitor CPU usage:

100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Real-World Use Cases

1. Monitoring Microservices

Microservices architecture involves many small services that scale dynamically. Monitoring each service independently and tracking how they interact can be challenging. With Prometheus’ service discovery and Grafana’s visualization tools, you can monitor individual services, track errors, and ensure smooth operation.

Example: Monitoring HTTP request latency across microservices.

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))

2. Kubernetes Monitoring

Prometheus integrates natively with Kubernetes, making it ideal for monitoring dynamic clusters. With the Kube-State-Metrics and cAdvisor integrations, you can monitor the health of your Kubernetes nodes, pods, and services.

Example dashboard: You can use the official Kubernetes monitoring mixin for a quick start.

3. Database Performance Monitoring

You can use Prometheus to scrape metrics from databases (e.g., PostgreSQL, MySQL) using specific exporters. Combine these metrics in Grafana to monitor query performance, connection pooling, and storage usage.

Real-world example: A financial company used Prometheus and Grafana to replace expensive proprietary tools for monitoring their Oracle databases, improving both cost efficiency and dashboard flexibility.

4. System Health Monitoring

For infrastructure monitoring, you can track key system metrics such as CPU, memory, and disk usage. Using Node Exporter with Prometheus and visualizing with Grafana allows for in-depth system health monitoring. This is ideal for cloud environments where scaling servers up or down dynamically is common.

5. Python API Monitoring

If you’re running Python APIs, you can use Prometheus client libraries to expose custom metrics. Grafana can then visualize these metrics, providing insights into API performance, error rates, and response times.

Example: Monitoring the number of requests processed by a Python API.

from prometheus_client import Counter

requests_total = Counter('api_requests_total', 'Total number of requests received')

@app.route('/api')

def api():
    requests_total.inc()
    return 'API response'

Best Practices for Monitoring with Prometheus and Grafana

  1. Define Metrics: Identify the key metrics you want to monitor and create custom metrics if needed.
  2. Use Labels: Leverage Prometheus labels to add dimensions to your metrics for better querying.
  3. Alerting Rules: Define alerting rules in Prometheus to notify you of critical issues.
  4. Dashboard Design: Create clear, concise dashboards in Grafana that provide actionable insights.
  5. Regular Maintenance: Update Prometheus and Grafana regularly to benefit from new features and bug fixes.
  6. Backup Configuration: Regularly back up your Prometheus configuration to avoid data loss.

Some Useful Resources for Prometheus and Grafana

Conclusion

Prometheus and Grafana together form a powerful and flexible monitoring stack for modern infrastructures, particularly for dynamic environments like Kubernetes and microservices. The ability to easily scale, customize dashboards, and integrate various data sources makes this combo highly effective for real-time monitoring and alerting. Whether you’re tracking application metrics or system health, Prometheus and Grafana can handle it all, providing critical insights to keep your infrastructure running smoothly.