One of the key features of Prometheus is its ability to evaluate PromQL expressions, which allow you to define queries that can be evaluated against your metrics.
However, working with timestamps and calculating the total increase of a vector in a calendar month is not directly possible within PromQL.
What is oy-periodic-queries
This is where oy-periodic-queries comes in. It allows you to evaluate expressions and export the results as metrics with monthly boundaries.
It uses calendar months to define the boundaries, which makes it easier to calculate monthly resource usage or any other metric that requires periodic evaluation.
Some of the questions that can be addressed by the tool are:
- How to monitor my monthly resource usage on my servers?
- How to generate a monthly report of my application’s key performance indicators?
- How to measure the monthly increase in the number of items sold in my e-commerce store?
- How to keep track of monthly user signups on my application?
- How to calculate the monthly traffic on my website?
How to use oy-periodic-queries
Using oy-periodic-queries is straightforward. You specify the recording rules and time boundaries in a rule file (similar to Prometheus rule files).
The exporter will query a Prometheus server with the appropriate time boundaries, iterating over each month and providing you with accurate, timely metrics.
You can access a basic templating system in the recording rule and labels. The templating system provides several variables that are replaced with the relevant information when the rule is evaluated such as the range of the query and the start and end times of the query.
Additionally, you can format the time-related variables using Go’s Time.Format to get the desired output.
In a rule file, you define one or more rule groups, each containing a set of rules that are evaluated at the specified time boundary and minimum lookback period (currently only monthly periods are supported).
In each rule, you define the PromQL expression to evaluate, the name of the time series to output to, and any labels to add or overwrite before storing the result.
What is Go Time Format?
Go’s Time.Format function is used to format time and date in variables inside the PromQl expression.
Inside of ${}, write a layout string that consists of
predefined format codes.
For example, the format code ${2006-01-02} would be replaced
by the date at the end of the interval in the format of “year-month-day”.
The default timestamp used in Go is “Jan 2 15:04:05 2006 MST”.
Example rule file
Here’s an example rules file that calculates the monthly network usage of a set of Prometheus nodes:
groups:
- name: prometheus network usage
time_period: monthly
lookback: 365d
include_incomplete_ranges: true
rules:
- expr: |
increase(node_network_receive_bytes_total{instance=~"prometheus.*"}[$RANGE])
and last_over_time(node_network_receive_bytes_total{instance=~"prometheus.*"}[${1_PC_RANGE}] @ ${1_PC_TIMESTAMP})
and last_over_time(node_network_receive_bytes_total{instance=~"prometheus.*"}[${1_PC_RANGE}] @ ${100_PC_TIMESTAMP})
record: prometheus:node_network_receive_bytes_total:monthly
labels:
month: "${2006-01}"
- expr: |
increase(node_network_transmit_bytes_total{instance=~"prometheus.*"}[$RANGE])
and last_over_time(node_network_transmit_bytes_total{instance=~"prometheus.*"}[${1_PC_RANGE}] @ ${1_PC_TIMESTAMP})
and last_over_time(node_network_transmit_bytes_total{instance=~"prometheus.*"}[${1_PC_RANGE}] @ ${100_PC_TIMESTAMP})
record: prometheus:node_network_transmit_bytes_total:monthly
labels:
month: "${2006-01}"
As you can see, the rules file defines a single group called “prometheus network usage” that specifies a monthly time period and a 365-day lookback.
The group contains two rules that calculate the monthly network receive and transmit bytes of Prometheus nodes.
The rules use PromQL expressions to select the relevant metrics and apply the necessary filters. They also specify the name of the time series to output and any labels to add or overwrite before storing the result.
In this case, the labels include the month of the time boundary in the format “2006-01”.
That would generate the following metrics:
# HELP prometheus:node_network_receive_bytes_total:monthly
# TYPE prometheus:node_network_receive_bytes_total:monthly untyped
prometheus:node_network_receive_bytes_total:monthly{device="ens3",environment="o11ylab",instance="prometheus02.example.com",job="node",month="2023-01"} 2.2096172024710458e+11
prometheus:node_network_receive_bytes_total:monthly{device="ens3",environment="o11ylab",instance="prometheus11.example.com",job="node",month="2023-01"} 1.4455463702554254e+11
prometheus:node_network_receive_bytes_total:monthly{device="lo",environment="o11ylab",instance="prometheus02.example.com",job="node",month="2023-01"} 9.635369117702129e+08
prometheus:node_network_receive_bytes_total:monthly{device="lo",environment="o11ylab",instance="prometheus11.example.com",job="node",month="2023-01"} 2.4574080228794675e+09
# HELP prometheus:node_network_transmit_bytes_total:monthly
# TYPE prometheus:node_network_transmit_bytes_total:monthly untyped
prometheus:node_network_transmit_bytes_total:monthly{device="ens3",environment="o11ylab",instance="prometheus02.example.com",job="node",month="2023-01"} 1.8763866363794147e+11
prometheus:node_network_transmit_bytes_total:monthly{device="ens3",environment="o11ylab",instance="prometheus11.example.com",job="node",month="2023-01"} 3.813239344424051e+11
prometheus:node_network_transmit_bytes_total:monthly{device="lo",environment="o11ylab",instance="prometheus02.example.com",job="node",month="2023-01"} 9.635369117702129e+08
prometheus:node_network_transmit_bytes_total:monthly{device="lo",environment="o11ylab",instance="prometheus11.example.com",job="node",month="2023-01"} 2.4574080228794675e+09
Conclusion
In conclusion, oy-periodic-queries is a powerful tool for evaluating Prometheus recording rules and exporting the results as metrics with monthly boundaries. The tool provides a simple and flexible solution to work with timestamps, which is a limitation of PromQL.
