Metrics

Tracking metrics gives a deeper understanding of how the system is operating and creates a baseline understanding of what is normal. Consistent tracking of metrics can help in early anomaly detection.

The Integrity application, if configured to do so, generates a number of metrics. Each metric has a name and a type, e.g. Counter, Gauge, or Timer. The name and type determine a number of different time series exposed by a particular metric. These time series can be monitored and analyzed, using supported monitoring backends. Currently, Integrity supports JMX and Prometheus.

The type, naming conventions and number of time series generated from a particular metric may vary depending on the selected monitoring backend.

Enabling metrics

Metrics are configured on the server object configuration in "config.json".

"server":

In the options object, metrics are enabled using JMX and/or Prometheus.

"options":{ "metricsOptions": {
    "enabled": true
}

Monitoring systems

Additionally one or more metrics system must be enabled.

JMX

 "options": {
        "metricsOptions": {
            "enabled": true,
            "jmxMetricsOptions": {
                "enabled": true
            }
        }
    }

The default JMX setup does not allow remote JMX connections. To enable them, startup options must be set:

Add the following JVM arguments when starting your Vert.x application to enable remote JMX access:

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=PORT_NUMBER
-Dcom.sun.management.jmxremote.rmi.port=PORT_NUMBER
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.local.only=false
-Djava.rmi.server.hostname=SERVER_IP_ADDRESS

PORT_NUMBER is the port you want to use for JMX (e.g., 9010).

SERVER_IP_ADDRESS should be set to the IP address where the server is reachable.

Note: If you enable remote access in a production environment, you should set authenticate=true and configure a password file for security, as well as apply firewall rules to restrict access.

This configuration enables JMX metrics in Vert.x and allows remote access through the JMX protocol.

Ensure that the appropriate firewall ports are open and that the connection is properly secured.

Prometheus

Basic setup

{
        "metricsOptions": {
            "enabled": true,
            "prometheusOptions": {
                "enabled": true,
                "startEmbeddedServer": true,
                "embeddedServerOptions": {
                    "host": "0.0.0.0",
                    "port": 9090,
                    "embeddedServerEndpoint": "/metrics"
                }
            }
        }
    }

Using SSL

{
    "metricsOptions": {
        "enabled": true,
        "prometheusOptions": {
            "enabled": true,
            "startEmbeddedServer": true,
            "embeddedServerOptions": {
                "host": "0.0.0.0",
                "port": 9090,
                "ssl": true,
                "keyCertOptions": {
                    "pemKeyCertOptions": {
                        "keyPath": "config/server-key.pem",
                        "certPath": "config/server-cert.pem"
                    }
                }
            },
            "embeddedServerEndpoint": "/metrics"
        }
    }
}

Available Metrics

Here is a list of fixed metrics exposed by the Integrity application.

Name	Labels	Type	Description
`ffid_pipes_pending`	id	Gauge	Number of outstanding requests
`ffid_pipes_processed`	id	Counter	Number of requests returned successfully
`ffid_pipes_failed`	id	Counter	Number of requests returned failure
`ffid_pipes_errors`	id	Counter	Number of requests returned error
`ffid_pipes_processed_duration`	id	Timer	Success processing time in milliseconds
`ffid_pipes_failed_duration`	id	Timer	Failure processing time in milliseconds
`ffid_pipes_errors_duration`	id	Timer	Error processing time in milliseconds
`ffid_valves_processed`	id, class	Counter	Number of requests returned successfully
`ffid_valves_failed`	id, class	Counter	Number of requests returned failure
`ffid_valves_processed_duration`	id, class	Timer	Success processing time in milliseconds
`ffid_valves_failed_duration`	id, class	Timer	Failure processing time in milliseconds
fortifiedInternalSessionCount	id	Gauge	Number of active sessions.

Dynamic metrics

Apart from fixed metric names, Integrity exposes a number of dynamic metric names. This means that metrics will be exposed based on naming and configuration of the current system setup. The idea behind this is to allow for detailed monitoring if desired.

Dynamic metrics generally come from modules that have outbound relations (HTTP, LDAP, SQL, SMTP, etc.), the Pipes module (valves), and authenticators. Details about metrics are found in the respective sections of the documentation.

Vert.x

The Integrity application is largely built on Eclipse Vert.x. Vert.x generates metrics related to the Vert.x event bus, monitoring low-level activity on the application level.

Metrics generated by Vert.x can also help monitor and analyse incoming/outgoing traffic over various protocols, for example:

HTTP activity
TCP activity
Datagram transmission and reception

Metrics exposed by Eclipse Vert.x are documented here:

https://vertx.io/docs/vertx-micrometer-metrics/java/#_vert_x_core_tools_metrics