PromQL Cheat Sheet - Complete Reference Guide

Quick reference for Prometheus Query Language (PromQL) with practical examples for monitoring, alerting, and SLO calculations. Covers essential functions, aggregations, and common patterns for effective Observability.

Metric Types

Data Types

Selectors & Filtering

Basic Selectors

Select all time series for a metric named http_requests_total:

1http_requests_total

Label Matchers

Multiple label filters:

1http_requests_total{method="GET", status=~"2.."}

Time Ranges

Range Vector Selector:

1http_requests_total[5m]  # Last 5 minutes of data

Time Units:

• ms - milliseconds
• s - seconds
• m - minutes
• h - hours
• d - days
• w - weeks
• y - years

Offset modifier Compare with data from the past:

1rate(http_requests_total[1h]) <= rate(http_requests_total[1h] offset 1w)

Operators

Arithmetic Operators

Basic math operators: +, -, *, /, %, ^

Example - Converting Bytes:

Comparison Operators

Comparison filters results:

1http_requests_total > 100

Logical Operators

• and - Intersection
• or - Union
• unless - Complement

Combine conditions:

1up{job="prometheus"} == 1 and on(instance) rate(http_requests_total[5m]) > 10

Essential Functions

Rate Functions (for Counters)

rate() - Per-second rate of increase interpolated from the last 5m

1rate(http_requests_total[5m])

Use for counters that always increase. Handles counter resets. Always normalized to per-second.

irate() - Instant rate (last 2 samples within range)

1irate(http_requests_total[5m])

Uses the last two samples within the time range. More sensitive to short-term spikes. Good for CPU metrics. Normalized to per-second.

increase() - Total increase over time range

1increase(http_requests_total[1h])

Extrapolates total increase. Use for counters. Effectively rate() multiplied by the number of seconds in the time range.

Aggregation Functions

Reduces many time series into fewer time series.

Aggregation with BY and WITHOUT

Keep or drop specific labels.

Group by specific labels:

1sum by (job, instance) (rate(http_requests_total[5m]))

Exclude specific labels:

1sum without (pod) (rate(http_requests_total[5m]))

Time & Date Functions

These all (except time()) take a unix timestamp as an optional first argument. Values returned are all in UTC.

• time() - Current Unix timestamp
• minute() - Minute of hour (0-59)
• hour() - Hour of day (0-23)
• day_of_week() - Day of week (0-6, Sunday=0)
• day_of_month() - Day of month (1-31)
• days_in_month() - Number of days in month (28-31)
• month() - Month (1-12)
• year() - Current year

Example - Repeating Test Patterns

 1- alert: TestAlert
 2  annotations:
 3    description: |
 4      This alert fires every 2 hours and resolves after 60 minutes.
 5    runbook_url: https://example.com
 6    summary: Test that alerts are working
 7  expr: |
 8    vector(1)
 9    unless (
10      (hour() % 2 == 0)
11    )
12  labels:
13    severity: none

Example - Alert only during business hours:

1http_errors > 100 and on() hour() >= 9 and on() hour() < 17

Math Functions

Rounding:

• round() - Round to nearest integer, or nearest multiple of second argument that defaults to 1
• ceil() - Round up away from zero
• floor() - Round down toward zero

Other Math:

• abs() - Absolute value
• sqrt() - Square root
• exp() - Exponential
• ln() - Natural logarithm
• log2() - Log base 2
• log10() - Log base 10

Need trigonometric functions?

Quantile Functions

quantile() - quantile aggregation

1quantile(0.95, rate(http_requests_total[5m]))

Get 95th percentile of the containers or pods HTTP rate of requests per second. Used to detect if a pod is much slower or faster than the rest.

histogram_quantile() - Estimate quantile from histogram

1histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Calculate the 95th percentile from a histogram aggregating all pods or containers together. Useful to judge have behavior is anomalous vs expected.

Prediction Functions

predict_linear() - Linear prediction

1predict_linear(node_filesystem_free_bytes[1h], 4 * 3600)

Predict disk space in 4 hours based on last hour’s trend.

deriv() - Derivative normalized to per-second

1deriv(node_memory_active_bytes[10m])

Rate of change over time. Useful for Gauge metrics. Can be negative. This example would show how fast memory is being consumed on a VM.

Sorting Functions

Common Query Patterns

CPU Usage

CPU usage per instance:

Node-Exporter Metrics:

1sum by (instance) (rate(node_cpu_seconds_total{mode!="idle", mode!="iowait", mode!="steal"}[5m]))

Kubernetes Metrics:

1sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))

Memory Usage

Memory usage percentage:

Node-Exporter Metrics:

1100 * (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes)
2/
3node_memory_MemTotal_bytes

Kubernetes Metrics:

1100 * sum by (pod) (container_memory_working_set_bytes)
2/
3sum by (pod) (kube_pod_container_resource_limits{resource="memory"})

Disk Usage

Disk usage percentage:

Node-Exporter Metrics:

1100 * (node_filesystem_size_bytes - node_filesystem_free_bytes)
2/
3node_filesystem_size_bytes

Kubernetes PersistentVolume Metrics:

1100 * (1 - kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes)

HTTP Error Rate

Percentage of 5xx errors:

1100 * sum(rate(http_requests_total{status=~"5.."}[5m]))
2/
3sum(rate(http_requests_total[5m]))

Request Latency (p95)

95th percentile latency:

1histogram_quantile(0.95, sum by(le) (rate(http_request_duration_seconds_bucket[5m])))

Kubernetes Pod Restarts

Pods restarting frequently:

1increase(kube_pod_container_status_restarts_total[30m]) > 1

Available Services

Count of healthy targets:

1sum(up{job="myapp"})

Percentage of healthy targets:

1100 * avg(up{job="myapp"})

Alert Rule Examples

High CPU Throttling Alert

 1- alert: HighCPUThrottles
 2  expr: |
 3    100 * sum(
 4      increase(
 5        container_cpu_cfs_throttled_periods_total{container!=""}[5m]
 6      )
 7    ) by (container, pod, namespace)
 8    /
 9    sum(
10      increase(
11        container_cpu_cfs_periods_total[5m]
12      )
13    ) by (container, pod, namespace)
14    > 25
15  for: 5m
16  labels:
17    severity: warning
18  annotations:
19    summary: "High CPU on {{ $labels.instance }}"
20    description: "CPU slices were throttled {{ $value | humanizePercentage }} over the last 5m"

High Memory Alert

1- alert: HighMemory
2  expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 90
3  for: 5m
4  labels:
5    severity: critical
6  annotations:
7    summary: "High memory usage on {{ $labels.instance }}"
8    description: "Memory usage is {{ $value | humanizePercentage }}"

High Error Rate

1- alert: HighErrorRate
2  expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
3  for: 5m
4  labels:
5    severity: critical
6  annotations:
7    summary: "High 5xx error rate"
8    description: "Error rate is {{ $value | humanizePercentage }}"

Pod Crash Loop

1- alert: PodCrashLooping
2  expr: increase(kube_pod_container_status_restarts_total[15m]) > 1
3  for: 5m
4  labels:
5    severity: warning
6  annotations:
7    summary: "Pod {{ $labels.pod }} is crash looping"
8    description: "Pod in namespace {{ $labels.namespace }} restarting frequently"

Best Practices

Quick Reference

Recording Rule Example

1groups:
2  - name: example
3    interval: 30s
4    rules:
5      - record: instance:node_cpu:avg_rate5m
6        expr: avg by(instance) (rate(node_cpu_seconds_total[5m]))

Template Variables in Annotations

1{{ $labels.instance }}           # Label value
2{{ $value }}                      # Current value
3{{ $value | humanize }}           # Human-readable value (1000 -> 1k)
4{{ $value | humanizePercentage }} # Format as percentage
5{{ $value | humanizeDuration }}   # Format as duration

Further Resources

• Official PromQL Documentation
• PromQL for Humans
• Query Examples
• Best Practices

Generate alerts automatically: Use our Prometheus Alert Generator to create SLO-based alerting rules with multi-window burn rate detection.