Logging

FROST-Server uses SLF4J and Logback as logging framework. Each component (HTTP, MQTT and combined) comes with a default logback.xml configuration file. You can override the logback configuration file by specifying a different configuration file in a JVM parameter -Dlogback.configurationFile. Of course this is not trivial in a Docker environment. Therefore there are some environment variables that influence basic logging options.

Environment variables

• logSensitiveData:
  If false, sensitive data like passwords and database connection URLs are not logged when loading settings. Default: false.
• queueLoggingInterval:
  If non-zero, log queue statistics ever x milliseconds. Default: 0 (off)
• FROST_LOG_path: Since 2.2.0, 2.1.1, 2.0.7
  The path to log to. Defaults to ${catalina.base}/logs
• FROST_LOG_filename: Since 2.2.0, 2.1.1, 2.0.7
  The base filename to use for log files. Defaults to FROST-Server.
• FROST_LOG_maxtotalsize: Since 2.2.0, 2.1.1, 2.0.7
  The maximum total size for all log files. See the logback docs. Default: 200MB.
• FROST_LOG_maxfilesize: Since 2.2.0, 2.1.1, 2.0.7
  The maximum individual log file size. See the logback docs. Default: 50MB.
• FROST_LOG_maxfilecount: Since 2.2.0, 2.1.1, 2.0.7
  The maximum number of log file to keep. See the logback docs. Default: 5.
• FROST_LL: Since 2.0.0
  The “root” log level. Default: INFO
• FROST_LL_parser: Since 2.0.0
  The log level for the de.fraunhofer.iosb.ilt.frostserver.parser package. Default: INFO
• FROST_LL_queries: Since 2.0.0
  The log level for the de.fraunhofer.iosb.ilt.frostserver.persistence.pgjooq.QueryBuilder package. Default: INFO
• FROST_LL_requests: Since 2.0.0
  The log level for the request decoder. When set to DEBUG each request with its headers will be logged. Default: INFO
• FROST_LL_service: Since 2.0.0
  The log level for the de.fraunhofer.iosb.ilt.frostserver.service package. When set to DEBUG, 4xx error details and 5xx stack traces will be logged. Default: INFO
• FROST_LL_settings: Since 2.0.0
  The log level for the de.fraunhofer.iosb.ilt.frostserver.settings package. Default: INFO
• FROST_LL_io_moquette: Since 2.0.0
  The log level for the io.moquette package. Default: WARN
• FROST_LL_liquibase: Since 2.0.0
  The log level for the liquibase package. Default: INFO
• FROST_LL_org_jooq: Since 2.0.0
  The log level for the org.jooq package. Default: INFO

Metrics

FROST-Server uses prometheus to gather metrics. For the HTTP component, the metrics end point can be made available on the same port as the normal service by setting the useServlet variable, or on a separate port by setting the useInternalHttpServer variable. If both useServlet and useInternalHttpServer are false, then metrics gathering is disabled.

Environment variables

• metrics.useServlet: Since 2.6.0
  For the HTTP and All-In-One components, use a servlet to expose the metrics. This makes /FROST-Server/metrics available, next to the other end-points. Default: false.
• metrics.useInternalHttpServer: Since 2.6.0
  Use a separate HTTP service, on its own port, to expose the metrics. This makes a /metrics URL available on a configurable port. Default: false.
• metrics.endpointPort: Since 2.6.0
  The port to use for the separate HTTP service that exposes the metrics. Default: 9400.

Gathered metrics

Besides the FROST-Server specific metrics listed below, the JVM metrics are also gathered, as described in the Prometheus Documentation

HTTP Pods

One HTTP request may result in multiple service requests, which in turn may result in may SQL queries.

• http_request_duration_seconds (histogram) HTTP request service time in seconds. Split along the HTTP Method used. (GET, PATCH, POST, …)
  • http_request_duration_seconds_bucket (counter) Cumulative counters for the observation buckets.
  • http_request_duration_seconds_count (counter) Count of events that have been observed for the histogram metric.
  • http_request_duration_seconds_sum (counter) Total sum of all observed values for the histogram metric.
• service_request_duration_seconds (histogram) Service request service time in seconds. Split along the service request type. (read, create, getCapabilities)
  • service_request_duration_seconds_bucket (counter) Cumulative counters for the observation buckets.
  • service_request_duration_seconds_count (counter) Count of events that have been observed for the histogram metric.
  • service_request_duration_seconds_sum (counter) Total sum of all observed values for the histogram metric.
• sql_query_duration_seconds (histogram) SQL query execution time in seconds. Split along the entity type.
  • sql_query_duration_seconds_bucket (counter) Cumulative counters for the observation buckets.
  • sql_query_duration_seconds_count (counter) Count of events that have been observed for the histogram metric.
  • sql_query_duration_seconds_sum (counter) Total sum of all observed values for the histogram metric.

MQTT Pods

The MQTT Manager is the part of the FROST-MQTT component that sits between the Data Model and the MQTT Broker. It tracks which changes are made to the data, and determines which MQTT Topics need to be notified of these changes.

• mqtt_manager_queue_fill (gauge) Fill level of the Queue (0 - 1). Split into Create, Changed.
  • Create: Queue for entities being created over MQTT.
  • Changed: Queue for entities that have changed and may require messages to be sent to users.
• mqtt_manager_queue_fill_max (gauge) Maximum fill level of the Queue since last scrape call (0 - 1). Split into Create, Changed.
• mqtt_manager_queue_overruns_total (gauge) Total number of actions dropped because a queue was full. If this increases, the queue is too small, the worker count too low, or the system is overloaded.
• mqtt_manager_topics (gauge) Number of distinct topics that have subscriptions.
• mqtt_manager_worker_status (gauge) Overview of what workers do Overview of what workers do. Split along two dimensions: Waiting, Working, Dead and Create, Changed.
  • Waiting: Number of workers that are waiting for work.
  • Working: Number of workers that are actively working.
  • Dead: Number of workers that have not responded for several seconds. If this is non-zero ther is a problem.

Moquette is the MQTT broker that distributes messages to the clients that listen to the topics those messages are sent to.

• moquette_open_sessions (gauge) The number of open sessions in the broker.
• moquette_publishes_total (counter) The number of messages publised to the broker.
• moquette_session_messages_total (counter) The number of messages publised by the broker, per Session Queue. Split along two dimensions: The session queue and the QualityOfService level of the message. The number of Session Queues equals the number of CPUs detected.
• moquette_session_queue_fill (gauge) Number of items in each Session Queue.
• moquette_session_queue_fill_max (gauge) Maximum number of items in each Session Queue since the last scrape call.
• moquette_session_queue_overruns_total (counter) Total number of actions dropped because the session queue was full. If this increases, the queue is too small, or the system is overloaded.

Message Bus

• message_bus_queue_fill (gauge) Fill level of the Queue (0 - 1). Split into Send and Receive sub-gauges.
  • Send: The outgoing queue, items being sent over the bus.
  • Receive: The incoming queue, items being received over the bus.
• message_bus_queue_fill_max (gauge) Maximum fill level of the Queue since last scrape call (0 - 1). Also split into Send and Receive.
• message_bus_queue_overruns_total (gauge) Total number of messages that were dropped because the queue was full. If this increases, either the queue is too small, or the number of workers is too low. Also split into Send and Receive.
• message_bus_worker_status (gauge) Overview of what workers do. Split along two dimensions: Waiting, Working, Dead and Send, Receive.
  • Waiting: Number of workers that are waiting for work.
  • Working: Number of workers that are actively working.
  • Dead: Number of workers that have not responded for several seconds. If this is non-zero ther is a problem.