Metrics

Tracking metrics allows for deep understanding in how the system is operating and creates baseline understanding on what is normal. Consistent tracking of metrics can help in early anomaly detection.

The Integrity application, if configured to do so, generates a number of metrics. Each metric has a name and a type, e.g. Counter, Gauge, or Timer. The name and type determines a number of different time series exposed by a particular metric. These time series can be monitored and analyzed, using supported monitoring backends. Currently, Integrity supports JMX and Prometheus.

The type, naming conventions and number of time series generated from a particular metric may vary depending on the selected monitoring backend.

Monitoring systems

JMX

The JVM runtime can be monitored using standard JMX tools, like JConsole and Java Mission Control. These tools displays, among many other things, information about the metrics generated by Integrity.

To instrument the application to provide JMX metrics, activate the JMX scenario. The following example shows how to start a server that activates JMX metrics and allows a JMX client to connect remotely from 192.168.0.10 to port 7091 on the server host:

docker run -e JVM_CONFIG=jmx,jmx_remote -e RMI_SERVER_HOSTNAME=192.168.0.10 -p 8080:8080 -p 7091:7091 --detach --name integrity fortifiedid.jfrog.io/default-docker-virtual/fortifiedid/integrity:latest Integrity config/config.json

JMX clients can always connect to the JVM from localhost, but if the JMX client is running on a remote host (e.g. if the server is running inside a Docker container and the client on the Docker host), the JMX_REMOTE scenario must be activated.

Prometheus

To instrument the application to provide metrics to the Prometheus monitoring system, activate the PROMETHEUS scenario. The following example shows how to start a server that instruments Prometheus metrics:

docker run -e JVM_CONFIG=prometheus -p 8080:8080 -p 7081:7081 --detach --name integrity fortifiedid.jfrog.io/default-docker-virtual/fortifiedid/integrity:latest Integrity config/config.json

The application activates a HTTP service exposing metrics that can be scraped by the Prometheus application on the following URL:

http://<host>:7081/metrics/prometheus

The table below lists the base names of all available metrics. Each metrics type is exposed by Prometheus as follows:

• Counter: a value that can only be incremented. Exposes the single time series <base_name>_total.

• Gauge: a value that can go up and down. Exposes the single time series <base_name>_max.

• Timer: a value that represents a sampled value like durations and latencies. Exposes the time series <base_name>_count and <base_name>_sum.

Example: the metric ffid_pipes_processed_duration is a Timer, describing time spent processing successful requests. This metric generates the time series

• ffid_pipes_processed_duration_seconds_sum

• ffid_pipes_processed_duration_seconds_count

Using the following Prometheus queries, we can graph the most commonly used statistics about timers.

Average latency:

rate(ffid_valves_processed_duration_seconds_sum[5m])/rate(ffid_valves_processed_duration_seconds_count[5m])

Throughput (requests per second):

rate(ffid_pipes_processed_duration_seconds_count[5m])

Available Metrics

Integrity

Here is a list of fixed metrics exposed by the Integrity application.

Name	Labels	Type	Description
ffid_pipes_pending	id	Gauge	Number of outstanding requests
ffid_pipes_processed	id	Counter	Number of requests returned successfully
ffid_pipes_failed	id	Counter	Number of requests returned failure
ffid_pipes_errors	id	Counter	Number of requests returned error
ffid_pipes_processed_duration	id	Timer	Success processing time in milliseconds
ffid_pipes_failed_duration	id	Timer	Failure processing time in milliseconds
ffid_pipes_errors_duration	id	Timer	Error processing time in milliseconds
ffid_valves_processed	id, class	Counter	Number of requests returned sucessfully
ffid_valves_failed	id, class	Counter	Number of requests returned failure
ffid_valves_processed_duration	id, class	Timer	Success processing time in milliseconds
ffid_valves_failed_duration	id, class	Timer	Failure processing time in milliseconds
fortifiedInternalSessionCount	id	Gauge	Number of active sessions.

Dynamic metrics

Apart from fixed metric names Integrity exposes a number of dynamic metric names. What tis means is that metrics will be exposed based on naming and configuration of the current system setup. The idea behind this is to allow for detailed monitoring if desired.

Dynamic metrics, in general, comes from modules that have outbound relations (http, ldap sql ,smtp etc), Pipes module (valves) & authenticators. Details in regards to metris is found in respective section av the documentation.

Vert.x

The Integrity application is largely built on Eclipse Vert.x. Vert.x generates metrics related to the Vert.x event bus, monitoring low-level activity on the application level.

Metrics generated by Vert.x can also help monitor and analyse incoming/outgoing traffic over various protocols, for example:

• HTTP activity

• TCP activity

• Datagram transmission and reception

Metrics exposed by Eclipse Vert.x are documented here:

https://vertx.io/docs/vertx-micrometer-metrics/java/#_vert_x_core_tools_metrics