Setup Prometheus and Grafana

Setup Prometheus and Grafana to collect and present FortifiedID Integrity metrics

The solution described below involves two new docker containers, Prometheus resp Grafana, running on the same host as the integrity container.

Overview of monitoring setup.

1. Expose metrics on the integrity platform

Edit the docker yml file for integrity. Export port 7081 and make sure the JVM is started with JVM_CONFIG=prometheus option.

Example extract

ports:
      - "8080:8080"
      - "7081:7081"
environment:
    - "JVM_CONFIG=prometheus"
command:
      - "Integrity"
      - "/opt/fortifiedid/integrity/config/server-config.json"

2. Configure and run Prometheus.

Create a prometheus file, prometheus.yml, and add this info to the file.

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'fid'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    metrics_path: /metrics/prometheus
    static_configs:
      - targets: ['host.docker.internal:7081']

3. Configure and run grafana.

1. Follow this guide, until "Add the first dashboard": https://www.theairtips.com/post/setting-up-grafana-with-docker-compose. Change the prometheus url from http://prometheus:9090 to http://host.docker.internal:9090
2. Add a dashboard by importing the json from grafana_fortifiedid_integrity.json. Modify the dashboard to suite your needs and environment.

3. Login to Grafana, http://<grafana_host>:3000, admin/admin. The dashbord "Integrity" should be visible.

Troubleshooting

1. Browse to http://<integrity_host>:7081/metrics/prometheus. A list of metrics should be presented. Check the config if npt.

2. Browse to http://<prometheus_host>:9090/. Go to Status->Targets. Make sure that the fid target is in state = UP. Check the config if npt.

3. Browse to http://<grafana_host>:3000/ and login with admin/admin. Click the cog wheel on the left panel and select Data Sources. Click on the Prometheus data source and then Save and test. Check the result of the operation. Check the config if the test was unsuccessful.