Prometheus Server and TLSBlog

With the Prometheus 2.24 release, server-side TLS (HTTPS) and basic auth are supported. In this blog we give an introduction to TLS ans Basic Authentication in Prometheus


BY Julien Pivotto / ON Jan 06, 2021

Prometheus supports TLS and basic authentication over its HTTP endpoints.

Scraping target using HTTPS instead of HTTP has been supported for a long time. You could fetch metrics using HTTPS, client-certificate authentication, and basi contentc authentication.

Last year, the Node Exporter became the first ‘official’ exporter to natively be able to expose metrics over HTTPS. Read our previous blog post to get all the details.

This week, we celebrate the 2.24.0 release of Prometheus. TLS is just the latest in a series of exciting changes that happened recently in Prometheus - like backfilling (also in 2.24) or even switching to a modern React UI.

This blog post focuses on two features: the introduction of TLS and Basic Authentication.

The Prometheus Security Model is the place to look at when it comes to Prometheus and security. If you want to report security vulnerabilities or question our implementation, you can find more information about how those topics are managed within the project here.

If you find a critical security vulnerability in Prometheus, an official exporter or library, please follow the procedure and not communicate about it in public, in such a way that we can find the best way to deploy a fix.

Admin and lifecycle API’s

By default, Prometheus will expose API endpoints to query the data. But there are more API endpoints available:

  • Lifecycle endpoints: reload the configuration, quit Prometheus
  • Admin endpoints: Delete metrics, take snapshots

They are not enabled by default. If you enable them, you probably want to secure the Prometheus port with some kind of security / authentication.

Securing Prometheus access

Previously, it was common to put a reverse proxy between your clients and your Prometheus server:

Prometheus behind a reverse proxy

That pattern provides you with more than just access control and encryption. You also get all the extra features of your reverse proxy - they can be: automatic certificate generation, throttling, extra controls, mangling, etc. Realistically, Prometheus is not in the load balancer / reverse proxy business. Therefore, users who need those advanced features would better continue to rely on reverse proxies.

For simple use cases or to secure the traffic between the reverse proxy and Prometheus, there now is a way to secure all the incoming HTTP traffic, built-in.

Prometheus with native security

How to - TLS

Let’s see how that works in practice. This how-to will demonstrate the setup on a Linux box of a Prometheus setup.

Setup a working directory

In this setup, we will work on a dedicated directory:

$ mkdir ~/prometheus_tls_example
$ cd ~/prometheus_tls_example

Generating TLS certificates

The first step is to generate a self-signed TLS certificate that will be used later on.

$ cd ~/prometheus_tls_example
$ openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout prometheus.key -out prometheus.crt -subj "/C=BE/ST=Antwerp/L=Brasschaat/O=Inuits/CN=localhost" -addext "subjectAltName = DNS:localhost"

In this command line, localhost is the hostname of the Prometheus server.

Two files are created: prometheus.crt and prometheus.key.

Prometheus web configuration

Download Prometheus v2.24.0, extract it, and move the previously created certificates:

$ cd ~/prometheus_tls_example
$ wget
$ tar xvf prometheus-2.24.0.linux-amd64.tar.gz
$ cp prometheus.crt prometheus.key prometheus-2.24.0.linux-amd64
$ cd prometheus-2.24.0.linux-amd64

At this point, we need to create a new configuration file. The server TLS and authentication are not configured in the main prometheus.yml file. It enables us to read the file upon every request to directly pick up new users and certificates.

Let’s create a web.yml file with the TLS configuration:

  cert_file: prometheus.crt
  key_file: prometheus.key

Start the Prometheus server, using the --web.config.file command line option:

$ ./prometheus --web.config.file=web.yml
enabled and it cannot be disabled on the fly." http2=true
level=info ts=2021-01-05T13:27:53.677Z caller=tls_config.go:223 component=web
msg="TLS is enabled." http2=true

If you see that message, it means that the Prometheus server is started with TLS support.

Note: All the TLS parameter can be changed on the fly, however it is not possible to disable TLS once it has been enabled without restarting Prometheus.

You can find the complete documentation about that extra configuration file on the Prometheus documentation.

Manual check of the TLS configuration

Check with curl that the TLS configuration is fine. In a new terminal, run some commands to test it:

$ cd ~/prometheus_tls_example
$ curl localhost:9090/metrics
Client sent an HTTP request to an HTTPS server.
$ curl --cacert prometheus.crt https://localhost:9090/metrics

Instead of --cacert prometheus.crt you can pass -k to skip curl certificate validation.

Scrape configuration

Configuring TLS is an all-or-nothing operation. Once enabled, all the endpoints are protected. This means that Prometheus will use TLS to fetch its own metrics.

Therefore, you need to instruct Prometheus to scrape itself using HTTPS.

Adapt the prometheus.yml ‘prometheus’ job:

  scrape_interval:     15s
  evaluation_interval: 15s
  - job_name: 'prometheus'
    scheme: https
      ca_file: prometheus.crt
    - targets: ['localhost:9090']

You need to set both the tls_config part of the configuration and the scheme to https. You can see the full tls_config client options in the Prometheus configuration.

And then, reload the Prometheus configuration:

$ killall -HUP prometheus

If visit locally https://localhost:9090/targets with your browser, you should see https://localhost:9090/metrics in the list of targets.

The target is up? Congratulations, you have successfully set-up the Prometheus server with TLS and metrics are scraped encrypted!

How to - Basic Auth

Let’s go one step further and ask for a username/password. TLS is not mandatory but highly recommended. We will re-use the setup of the previous steps.

Web configuration

Prometheus uses bcrypt for its passwords, a salted and adaptive password hashing algorithm. However, using untrusted websites to generate bcrypt passwords is risky. The O11y Toolkit’s password generator application generates a web.yml file for basic authentication in Prometheus, allowing you to add and remove multiple users and adjust bcrypt cost, all of this happening locally in your browser. For more information, check out our post on passwords for Prometheus.

In this tutorial, the password is inuitsdemo.

Add a user in the Prometheus configuration file web.yml, that we generated from the password generator:

  cert_file: prometheus.crt
  key_file: prometheus.key
  prometheus: $2y$10$EYxs8IOG46m9CtpB/XlPxO1ei7E4BjAen0SUv6di7mD4keR/8JO6m

Note: In this file, prometheus is the username.

If Prometheus is still running, you now have to enter a password to access the web interface at and in the targets page, that the Prometheus target now returns a 401 Unauthorized error.

Prometheus requesting a password

Prometheus configuration

Let’s change the prometheus.yml to scrape with the username / password pair.

  scrape_interval:     15s
  evaluation_interval: 15s
  - job_name: 'prometheus'
    scheme: https
      username: prometheus
      password: inuitsdemo
      ca_file: prometheus.crt
    - targets: ['localhost:9090']

Reload the Prometheus configuration with a SIGHUP signal:

$ killall -HUP prometheus

If all works, Prometheus should be up again in our targets page.

Prometheus successfully scraping itself


Prometheus is shipped with a command line tool, promtool. It gained support to validate web configuration files:

$ ./promtool check web-config web.yml
web.yml SUCCESS

Use that with your favourite automation tool to ensure smooth updates of the web.yml files.


Grafana itself supports all the required features to connect to your Prometheus server. Note that you can either provide a CA (our prometheus.crt) or skip certificate validation.

Grafana data source configuration for secure Prometheus


This is a getting started introduction. In production, you should use a proper CA, and pay attention to more details. It is also possible to use client-based certificates for authentication.

It is worth noting that for security concerns, this feature is marked as experimental. We might indeed change ciphers, supported TLS versions, over time. Therefore the exact behavior of those features might change over time.

The intent is to roll over this kind of HTTPS support across all the official Prometheus exporters in the coming months and the other projects, such as Alertmanager, Pushgateway.

Let’s make monitoring more secure!

Are you having trouble with setting up and scaling Prometheus? Our team of observability experts can support the operations of your Prometheus environments.

Get Prometheus Support


to our newsletter