Introducing Promtool PromQL commands for advanced queries editingBlog

Promtool PromQL command

Prometheus 2.46 introduces a set of new commands for Promtool. One of them, promtool promql, brings new possibilities to PromQL query editing and formating.

Labels selectors in Prometheus

In Prometheus, the data model relies heavily on labels, allowing users to select metrics based on label selectors.

For instance, the query node_boot_time_seconds will produce the boot time of all your nodes. You can further refine the results to only show the boot time of machines in your production environment by using a label selector like {environment="production"}. The resulting selector would be node_boot_time_seconds{environment="production"}.

This approach works well for simple queries. However, alerting rules can be more complex an contain multiple selectors.

Consider this more complex query, designed to detect if a system is filling up. Adding the matcher to every selector in the query becomes more complex.

(
            node_filesystem_avail_bytes{fstype!="",job="node"}
          /
            node_filesystem_size_bytes{fstype!="",job="node"}
        *
          100
      <
        40
    and
      predict_linear(node_filesystem_avail_bytes{fstype!="",job="node"}[6h], 24
* 60 * 60) < 0
  and
    node_filesystem_readonly{fstype!="",job="node"} == 0
)

In PromQL, thanks to vector matching, adding the matcher to one of the selector would be enough. You could just replace node_filesystem_avail_bytes{fstype!="",job="node"} by node_filesystem_avail_bytes{fstype!="",job="node", environemnt="production"}.

Adding a matcher to a single selector might seem sufficient, but this forces all other selectors to collect data for all environments, leading to unnecessary resource usage for results that would be discarded.

Adding label matchers with promtool

The new Promtool command promtool promql offers a solution to this problem by enabling users to efficiently add label matchers to multiple selectors. This allows you to customize your query for specific usecases. Here is how this plays out, after writing your query to a file name query:

$ vim query
$ promtool --experimental promql label-matchers set "$(<query)" environment prod
(
            node_filesystem_avail_bytes{environment="prod",fstype!="",job="node"}
          /
            node_filesystem_size_bytes{environment="prod",fstype!="",job="node"}
        *
          100
      <
        40
    and
        predict_linear(
          node_filesystem_avail_bytes{environment="prod",fstype!="",job="node"}[6h],
          24 * 60 * 60
        )
      <
        0
  and
    node_filesystem_readonly{environment="prod",fstype!="",job="node"} == 0
)

Now, every selector in your PromQL query has a new environment="production" matcher, matcher, effectively narrowing down the query to the production environment and making it more efficient.

Additional flags and commands

The promtool promql label-matchers set command offers a -t flag to work with the different types of matchers (=, !=, =~, and !~). Additionally, you can use promtool promql label-matchers delete to remove label matchers.

For more information and to explore other useful Promtool commands, take a look at the command line documentation provided in the official Prometheus documentation.

With promtool promql, you have a new way to optimize and manage your complex Prometheus queries. Although this feature is currently experimental and requires the --experimental flag, we highly value your feedback to further enhance its capabilities.