Docker Compose is used for running multiple containers as one service. If you have an application/ stack requiring different services, docker compose enables creating one file which would start all containers as one service and avoid the need to start them separately. It is also possible to start one service at a time with docker compose. Today, we are going to look at how to run Prometheus and Grafana using docker compose.

To be able to use Docker Compose, you should have both Docker and Docker compose installed on your system. If you are running debian-based system, our guide on Install and use docker compose on Linux Mint should help you install Docker and Docker Compose.

Prometheus is a time series database, making use of exporters to monitor various servers/services while Grafana is one of the most popular monitoring tools. When combined with Prometheus, Grafana offers powerful visualization tool for time series data.

To run Prometheus and Grafana with docker compose, we need to create a docker compose file defining the individual services (Prometheus and Grafana), the images to be used, ports running on and any other thing necessary.

Using persistent volumes with Docker Compose

We need to consider the fact that Grafana requires a configuration file and a datasource file. Promemetheus equally requires a configuration file. For these, we are going to be using mounted volumes (persistent volumes) so that it is easy to make changes to the files whenever need be.

Create a working directory called prometheus-grafana and within it create directories to hold Prometheus and Grafana configuration files

sudo mkdir -p prometheus-grafana/{grafana,prometheus}

Now change to grafana directory and create grafana configuration files. The default Grafana default configuration file is provided in their official Github repository. Create a file called grafana.ini and copy and paste the default grafana configuration to this file and save.

The content for Grafana default configuration can be found in https://github.com/grafana/grafana/blob/master/conf/defaults.ini. Note that you can make changes to the default configuration file according to your needs.

wget https://github.com/grafana/grafana/blob/master/conf/defaults.ini -O prometheus-grafana/grafana/grafana.ini
vim  prometheus-grafana/grafana/grafana.ini

Once done, change into Prometheus folder and create prometheus configuration file to tell Prometheus where to take the metrics. Since I don’t have a separate server to monitor, the metrics to be exposed are for the prometheus server itself (localhost:9090)

tee prometheus-grafana/promethueus/prometheus.yml<<EOF
global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s
alerting:
  alertmanagers:
  - static_configs:
    - targets: []
    scheme: http
    timeout: 10s
    api_version: v1
scrape_configs:
- job_name: prometheus
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - localhost:9090
EOF

You can open and modify the file further to fit your use case

$ vim prometheus-grafana/promethueus/prometheus.yml

At this point, Grafana does not show where to get data from. We need to tell it to get from Prometheus. Create a file called datasource.yml in the grafana directory.

tee prometheus-grafana/grafana/datasource.yml<<EOF
apiVersion: 1
datasources:
- name: Prometheus
  type: prometheus
  url: http://localhost:9090 
  isDefault: true
  access: proxy
  editable: true
EOF

Further modification can be done:

vim prometheus-grafana/grafana/datasource.yml

By now we have all our volumes ready. Let us go ahead to create to create out docker-compose file

vim prometheus-grafana/docker-compose.yml

The contents should look like below. Ensure that you change the path of the configuration files to suit the paths in your system.

version: "3.7"
services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
            - ~/prometheus-grafana/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - 9090:9090
  grafana:
    image: grafana/grafana:latest
    volumes:
            - ~/prometheus-grafana/grafana/grafana.ini:/etc/grafana/grafana.ini
            - ~/prometheus-grafana/grafana/datasource.yml:/etc/grafana/provisioning/datasources/datasource.yaml
    ports:
      - 3000:3000
    links:
            - prometheus

If you need to specify a specific version or Prometheus or Grafana, you can add version number so that it looks as below:

services:
  prometheus:
    image: prom/prometheus:2.52.2
---
grafana:
    image: grafana/grafana:7.4.5-ubuntu

Also the aim of creating Prometheus/Grafana stack is to monitor other services running probably on different servers. Promethues uses different exporters to expose hardware- and kernel-related metrics depending on the service you are monitoring, the most common being node-exporter. Exporters are installed in the hosts to be monitored. Since I don’t have another external server, I am going to install node exporter on local using. My docker-compose file should look like this:

version: "3.7"
services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
            - ~/prometheus-grafana/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - 9090:9090
  grafana:
    image: grafana/grafana:7.4.5-ubuntu
    volumes:
            - ~/prometheus-grafana/grafana/grafana.ini:/etc/grafana/grafana.ini
            - ~/prometheus-grafana/grafana/datasource.yml:/etc/grafana/provisioning/datasources/datasource.yaml
    ports:
      - 3000:3000
    links:
            - prometheus
  node-exporter:
    image: prom/node-exporter:latest
    container_name: monitoring_node_exporter
    restart: unless-stopped
    expose:
      - 9100

I should also update prometheus.yml file to add node-exporter as a target

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s
alerting:
  alertmanagers:
  - static_configs:
    - targets: []
    scheme: http
    timeout: 10s
    api_version: v1
scrape_configs:
- job_name: prometheus
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets: ['localhost:9090','node-exporter:9100']

If you have other exporters from other services, you can add to the array to have something that looks like below:

static_configs:
         - targets: ['localhost:9090','cadvisor:8080','node-exporter:9100', 'nginx-exporter:9113']

Running Containers with docker-compose

We are now all set to run our containers. To start the containers, run the command as shown:

$ docker-compose up -d
Creating network "prometheus-grafana_default" with the default driver
Creating monitoring_node_exporter        ... done
Creating prometheus-grafana_prometheus_1 ... done
Creating prometheus-grafana_grafana_1    ... done

Confirm running containers:

$ docker ps
CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS              PORTS                    NAMES
aaa7b165423e        grafana/grafana:7.4.5-ubuntu   "/run.sh"                32 seconds ago      Up 30 seconds       0.0.0.0:3000->3000/tcp   prometheus-grafana_grafana_1
e2cc533ab639        prom/prometheus:latest         "/bin/prometheus --c…"   33 seconds ago      Up 31 seconds       0.0.0.0:9090->9090/tcp   prometheus-grafana_prometheus_1
fd69b20b92f1        prom/node-exporter:latest      "/bin/node_exporter"     33 seconds ago      Up 31 seconds       9100/tcp                 monitoring_node_exporter

Access Prometheus and grafana from the browser using the default ports. For Prometheus, we use http://localhost:9090. If you go to status-> targets, you should be able to see you targets status as below:

For Grafana, visit http://localhost:3000 and login with the username and password provided in the .ini configuration file. The default credentials are admin:admin

You can now go ahead and create dashboard for your monitoring. One thing I noted is that I could not get metrics to use in generating graphs when ‘access’ in ‘HTTP’ section of data sources was set to ‘server’. I had to set it as ‘browser’ as below:

Stopping and Cleaning up with docker-compose

To stop all running containers, run the below comand:

docker-compose down

To delete all containers, use the command:

docker rm -f $(docker ps -a -q)

Delete all volumes with the command below:

docker volume rm $(docker volume ls -q)

You can restart all the containers again using the command:

docker-compose up -d

That marks the end of our guide on how to run prometheus and Grafana with docker-compose. Using docker-compose eases the work of running many containers as well as improve organization. I hope the guide has been helpful. Check below related guides: