Telemetry

Overview

Each Anapaya appliance exposes a telemetry endpoint that can be used to retrieve telemetry data from the appliance.

Tip

To enable telemetry of the appliance, the telemetry endpoint needs to be configured in the Management section of the appliance configuration.

The telemetry data is exported in the form of Prometheus metrics. Prometheus is an open-source systems monitoring and alerting tool. It collects and stores metrics as time series data alongside optional key-value pairs called labels. A metric is a numeric measurement of a specific event or condition, e.g., the number of packets sent on a specific interface. Recording metrics in time series provides then higher-level insights such as the rate of change of the sent packet counter to calculate the throughput of the interface. Labels add additional dimensions to a metric, e.g., the name of the interface for which the packet count is collected is added as a label.

Each Anapaya appliance internally has several modules that expose some of their internal states as metrics. Each module manages a particular part of the system, such as the SCION control plane, the SCION data plane, or the IP-in-SCION tunneling service. For each module, we list the exposed metrics, their names, the type of the metric, a brief description, and the attached labels. Please refer to the individual sections below for more information.

To access these metrics, a Prometheus server is needed that ingests the metrics from each appliance. How to set up a Prometheus server to collect appliance metrics is outside the scope of this document. Please refer to the Prometheus Getting Started guide for more information. Should you require assistance with integrating appliance metrics in your monitoring setup, please contact Anapaya’s customer support team (customer-support@anapaya.net).

Control Plane Metrics

control_beaconing_originated_beacons_total

Description

Total number of beacons originated.

Type

counter

Labels

egress_interface result

control_beaconing_propagated_beacons_total

Description

Total number of beacons propagated.

Type

counter

Labels

start_isd_as ingress_interface egress_interface result

control_beaconing_received_beacons_total

Description

Total number of beacons received.

Type

counter

Labels

ingress_interface neighbor_isd_as result

control_beaconing_registered_segments_total

Description

Total number of segments registered.

Type

counter

Labels

start_isd_as ingress_interface seg_type result

control_segment_expiration_deficient

Description

Indicates whether the expiration time of the segment is below the configured maximum. This happens when the signer expiration time is lower than the maximum segment expiration time.

Type

gauge

Labels

None

control_segment_lookup_requests_total

Description

Total number of path segments requests received.

Type

counter

Labels

dst_isd seg_type result

control_segment_registry_segments_received_total

Description

Total number of path segments received through registrations.

Type

counter

Labels

src seg_type result

renewal_ca_health_status

Description

Exposes the status of the CA (available, unavailable, starting, stopping), if the host acts as CA and is delegating certificate renewal to the CA service.

Type

gauge

Labels

status

renewal_handled_requests_total

Description

Total number of renewal requests served by each handler type (legacy, in-process, delegating).

Type

counter

Labels

result type

renewal_received_requests_total

Description

Total number of renewal requests served.

Type

counter

Labels

result

renewal_registered_handlers

Description

Exposes which handler type (legacy, in-process, delegating) is registered.

Type

gauge

Labels

type

trustengine_latest_trc_not_after_time_seconds

Description

The not_after time of the latest TRC for the local ISD in seconds since UNIX epoch.

Type

gauge

Labels

None

trustengine_latest_trc_not_before_time_seconds

Description

The not_before time of the latest TRC for the local ISD in seconds since UNIX epoch.

Type

gauge

Labels

None

trustengine_latest_trc_serial_number

Description

The serial number of the latest TRC for the local ISD.

Type

gauge

Labels

None

Data Plane Metrics

router_dropped_pkts_total

Description

Total number of packets dropped.

Type

counter

Labels

interface isd_as neighbor_isd_as type

router_input_bytes_total

Description

Total number of bytes received

Type

counter

Labels

interface isd_as neighbor_isd_as

router_input_pkts_total

Description

Total number of packets received

Type

counter

Labels

interface isd_as neighbor_isd_as

router_interface_up

Description

1 indicates the interface is up, 0 otherwise.

Type

gauge

Labels

interface isd_as link_to neighbor_isd_as

router_output_bytes_total

Description

Total number of bytes sent.

Type

counter

Labels

interface isd_as neighbor_isd_as

router_output_pkts_total

Description

Total number of packets sent.

Type

counter

Labels

interface isd_as neighbor_isd_as

dataplane_control_dataplane_sync_error

Description

Indicates whether the last dataplane sync had an error (1) or not (0).

Type

gauge

Labels

None

IP-in-SCION Tunneling Metrics

gateway_as_certificate_expiration_time_second

Description

The expiration time of the AS certificate.

Type

gauge

Labels

isd_as

gateway_domain_paths_total

Description

The metric indicates the number of paths available for a domain and traffic matcher. The status indicates more details about the paths: ‘total’ indicates the total number of paths available to the domain, ‘eligible’ indicates the number of paths that were accepted by the path policies, ‘monitored’ indicates the number of paths out of the eligible that are being actively monitored, ‘alive’ indicates the number of paths out of the monitored that were recently seen alive.

Type

counter

Labels

domain traffic_matcher status

gateway_domain_traffic_matcher_sessions_total

Description

The number of live sessions per traffic matcher in a domain.

Type

gauge

Labels

domain traffic_matcher

gateway_domain_traffic_redirections_total

Description

The metric is incremented each time some subset of traffic is potentially sent either via a different SCION path or to a different remote gateway instance. The reason indicates the cause of the redirection.

Type

counter

Labels

domain traffic_matcher reason

gateway_flow_exporter_cleanup_run_time

Description

Overall time the flow clean up has been running, in seconds.

Type

gauge

Labels

None

gateway_flow_exporter_export_run_time

Description

Overall time the flow exporting has been running, in seconds.

Type

gauge

Labels

None

gateway_flow_exporter_flows_limit

Description

The upper limit on how many flows can be tracked.

Type

gauge

Labels

None

gateway_flow_exporter_flows_total

Description

Total number of flows that are currently being tracked.

Type

counter

Labels

None

gateway_flow_exporter_last_cleanup_time

Description

The timestamp up until which the finished flows were deleted. Seconds since UNIX epoch.

Type

gauge

Labels

None

gateway_flow_exporter_last_export_time

Description

The timestamp of the last time when the flow metrics were exported, successfully. Measured in seconds since UNIX epoch.

Type

gauge

Labels

None

gateway_flow_exporter_lost

Description

The cumulative duration of time (in seconds) for which there has been flow data lost by the flow exporter.

Type

counter

Labels

None

gateway_info_fetch_errors_total

Description

Total number of errors fetching gateway info.

Type

counter

Labels

isd_as remote_isd_as remote_address

gateway_info_seccom_addresses_fetched

Description

The number of fetched seccom addresses from the remote.

Type

gauge

Labels

isd_as remote_isd_as remote_address

gateway_ippkt_bytes_local_received_total

Description

Total IP packet bytes received from the local network.

Type

counter

Labels

None

gateway_ippkt_bytes_local_sent_total

Description

Total IP packet bytes sent to the local network.

Type

counter

Labels

isd_as

gateway_ippkt_bytes_received_total

Description

Total IP packet bytes received from remote gateways.

Type

counter

Labels

isd_as remote_isd_as

gateway_ippkt_bytes_sent_total

Description

Total IP packet bytes sent to remote gateways.

Type

counter

Labels

isd_as remote_isd_as domain traffic_class path_filter remote_address frame_type

gateway_ippkts_discarded_total

Description

Total number of discarded IP packets received from the local network.

Type

counter

Labels

reason

gateway_ippkts_local_received_total

Description

Total number of IP packets received from the local network.

Type

counter

Labels

None

gateway_ippkts_local_sent_total

Description

Total number of IP packets sent to the local network.

Type

counter

Labels

isd_as

gateway_ippkts_received_total

Description

Total number of IP packets received from remote gateways.

Type

counter

Labels

isd_as remote_isd_as

gateway_ippkts_sent_total

Description

Total number of IP packets sent to remote gateways.

Type

counter

Labels

isd_as remote_isd_as domain traffic_class path_filter remote_address frame_type

gateway_netlink_listener_subscribed

Description

Flag reflecting whether the netlink listener is subscribed route updates.

Type

gauge

Labels

object

gateway_netlink_listener_updates_errors_total

Description

Total number of netlink route updates errors.

Type

counter

Labels

object

gateway_path_fetch_errors_total

Description

Total number of errors fetching paths from the daemon.

Type

counter

Labels

isd_as

gateway_paths_monitored

Description

Total number of paths being monitored by the gateway.

Type

gauge

Labels

isd_as remote_isd_as

gateway_ping_reachability_changes

Description

The number of times the reachability of the gateway changed.

Type

counter

Labels

isd_as remote_isd_as remote_address interface_group

gateway_ping_reachable

Description

Whether the gateway is reachable via a specific SCION interface group.

Type

gauge

Labels

isd_as remote_isd_as remote_address interface_group

gateway_ping_received_total

Description

Total number of probe replies received from remote gateways.

Type

counter

Labels

isd_as remote_isd_as remote_address interface_group

gateway_ping_sent_total

Description

Total number of probes sent to remote gateways.

Type

counter

Labels

isd_as remote_isd_as remote_address interface_group

gateway_prefix_fetch_errors_total

Description

Total number of errors fetching prefixes via SGRP.

Type

counter

Labels

isd_as remote_isd_as remote_address

gateway_prefix_fetch_invalid_total

Description

Total number of invalid prefixes received via SGRP.

Type

gauge

Labels

isd_as remote_isd_as remote_address

gateway_prefixes_advertised

Description

Total number of IP prefixes advertised over SGRP.

Type

gauge

Labels

isd_as remote_isd_as remote_address

gateway_prefixes_fetched

Description

Total number of IP prefixes fetched via SGRP.

Type

gauge

Labels

isd_as remote_isd_as remote_address

gateway_remote_discovery_errors_total

Description

Total number of errors discovering remote gateways.

Type

counter

Labels

isd_as remote_isd_as

gateway_remote_discovery_paths_available

Description

Total number of SCION paths available to the remote gateway discovery.

Type

gauge

Labels

isd_as remote_isd_as status

gateway_remotes

Description

Total number of discovered remote gateways.

Type

gauge

Labels

isd_as remote_isd_as

gateway_remotes_changes

Description

The number of times the remotes number changed.

Type

counter

Labels

isd_as remote_isd_as

gateway_seccom_egress_sa_expiration

Description

The timestamp the current SAs expire. Measured in seconds since UNIX epoch.

Type

gauge

Labels

isd_as remote_isd_as remote_address domain traffic_class

gateway_seccom_egress_sa_last_update

Description

The timestamp the current SAs were created. Measured in seconds since UNIX epoch.

Type

gauge

Labels

isd_as remote_isd_as remote_address domain traffic_class

gateway_seccom_egress_sa_update_errors

Description

Total number of failed updates of the egress SAs.

Type

counter

Labels

isd_as remote_isd_as remote_address domain traffic_class

gateway_seccom_egress_sas

Description

Number of egress SAs that are currently configured.

Type

gauge

Labels

isd_as remote_isd_as remote_address domain traffic_class

gateway_seccom_ingress_request_errors_total

Description

Total number of errors processing incoming security communication requests.

Type

counter

Labels

isd_as remote_isd_as remote_address type reason

gateway_seccom_ingress_requests_total

Description

Total number of incoming security communication requests.

Type

counter

Labels

isd_as remote_isd_as remote_address type

gateway_seccom_ingress_sas

Description

Number of ingress SAs that are currently configured.

Type

gauge

Labels

isd_as remote_isd_as remote_address

gateway_session_is_healthy

Description

Flag reflecting session healthiness.

Type

gauge

Labels

isd_as remote_isd_as remote_address path_filter domain

gateway_session_latest_path_expiration

Description

Latest path expiration per session monitor.

Type

gauge

Labels

isd_as remote_isd_as remote_address path_filter domain

gateway_session_path_changes

Description

Number of path changes per session monitor.

Type

counter

Labels

isd_as remote_isd_as remote_address path_filter domain

gateway_session_paths_available

Description

Total number of paths available per session.

Type

gauge

Labels

isd_as remote_isd_as remote_address path_filter domain status

gateway_session_state_changes

Description

Number of state changes per session monitor.

Type

counter

Labels

isd_as remote_isd_as remote_address path_filter domain

gateway_sgrp_paths_available

Description

Total number of paths available for SGRP per remote gateway.

Type

gauge

Labels

remote_isd_as remote_address status

LAN Monitoring Metrics

mole_gateway_alive

Description

Whether the probes to the given gateway are passing through.

Type

gauge

Labels

gateway

mole_gateway_jitter_milliseconds

Description

The latency jitter to the given gateway.

Type

gauge

Labels

gateway

mole_gateway_latency_milliseconds

Description

The RTT latency to the given gateway.

Type

gauge

Labels

gateway

mole_gateway_probes_received_total

Description

Number of probes received from the given gateway.

Type

counter

Labels

gateway

mole_gateway_probes_sent_total

Description

Number of probes sent to the given gateway.

Type

counter

Labels

gateway

Appliance Cluster Metrics

appliance_controller_enforcer_license_expiry

Description

Time when the current license expires or when the current trial/grace period ends.

Type

gauge

Labels

None

nodesync_topology_fetch_errors_total

Description

The number of errors when fetching topology information from a remote node.

Type

counter

Labels

remote

nodesync_topology_merge_interface_conflicts_total

Description

The number of topology merge conflicts. This indicates a severe misconfiguration of appliances. It means that multiple appliances have the same interfaces configured.

Type

counter

Labels

isd_as interface

nodesync_topology_merge_service_conflicts_total

Description

The number of topology merge conflicts. This indicates a severe misconfiguration of appliances. It means that multiple appliances have services configured with the same configuration.

Type

counter

Labels

service isd_as shard

Installer Metrics

appliance_installer_checksum_consistent

Description

Whether the checksum of the installed package does match the checksum in the package signature file. This may fail if a different package with the same version number was uploaded but it hasn’t been installed.

Type

gauge

Labels

pkgtype

appliance_installer_controller_watchdog_errors_total

Description

Total number of errors encountered by the appliance controller watchdog. If this counter increases, the installer logs should be inspected for more details.

Type

counter

Labels

None

appliance_installer_installed_package_versions

Description

The version of the installed scion and system package.

Type

gauge

Labels

pkgtype version

appliance_installer_metastore_inconsistent

Description

Whether the appliance installer’s metastore is in an inconsistent state. Value is 1 if the metastore is in an inconsistent state, 0 otherwise.

Type

gauge

Labels

None

appliance_installer_rollback_installations_total

Description

Total number of rollback installations. Result label is the result of the installation.

Type

counter

Labels

result

appliance_installer_scion_installations_total

Description

Total number of scion package installations. Result label is the result of the installation.

Type

counter

Labels

result

appliance_installer_system_installations_total

Description

Total number of system package installations. Result label is the result of the installation.

Type

counter

Labels

result

BGP Metrics

BGP metrics are metrics from the BGP daemon (FRR).

frr_bgp_peer_groups_count_total

Description

Number of peer groups configured.

Type

gauge

Labels

vrf afi safi local_as

frr_bgp_peer_groups_memory_bytes

Description

Memory consumed by peer groups.

Type

gauge

Labels

vrf afi safi local_as

frr_bgp_peer_message_received_total

Description

Number of received messages.

Type

counter

Labels

vrf afi safi local_as peer peer_as

frr_bgp_peer_message_sent_total

Description

Number of sent messages.

Type

counter

Labels

vrf afi safi local_as peer peer_as

frr_bgp_peer_prefixes_advertised_count_total

Description

Number of prefixes advertised.

Type

gauge

Labels

vrf afi safi local_as peer peer_as

frr_bgp_peer_prefixes_received_count_total

Description

Number of prefixes received.

Type

gauge

Labels

vrf afi safi local_as peer peer_as

frr_bgp_peer_state

Description

State of the peer (2 = Administratively Down, 1 = Established, 0 = Down).

Type

gauge

Labels

vrf afi safi local_as peer peer_as

frr_bgp_peer_types_up

Description

Total Number of Peer Types that are Up.

Type

gauge

Labels

type afi safi

frr_bgp_peer_uptime_seconds

Description

How long has the peer been up.

Type

gauge

Labels

vrf afi safi local_as peer peer_as

frr_bgp_peers_count_total

Description

Number peers configured.

Type

gauge

Labels

vrf afi safi local_as

frr_bgp_peers_memory_bytes

Description

Memory consumed by peers.

Type

gauge

Labels

vrf afi safi local_as

frr_bgp_rib_count_total

Description

Number of routes in the RIB.

Type

gauge

Labels

vrf afi safi local_as

frr_bgp_rib_memory_bytes

Description

Memory consumbed by the RIB.

Type

gauge

Labels

vrf afi safi local_as

Host Metrics

Host metrics are metrics from the host itself, such as CPU usage, memory consumption or network traffic on the physical network ports.

node_cpu_seconds_total

Description

Seconds the CPU spends in each mode.

Type

counter

Labels

cpu mode

node_load1

Description

1 minute load average.

Type

gauge

Labels

None

node_load5

Description

5 minute load average.

Type

gauge

Labels

None

node_load15

Description

15 minute load average.

Type

gauge

Labels

None

node_memory_MemTotal_bytes

Description

Total amount of memory in the node.

Type

gauge

Labels

None

node_memory_MemAvailable_bytes

Description

Amount of available memory in the node.

Type

gauge

Labels

None

node_filesystem_size_bytes

Description

Filesystem size in bytes.

Type

gauge

Labels

device fstype mountpoint

node_filesystem_avail_bytes

Description

Filesystem available bytes.

Type

gauge

Labels

device fstype mountpoint

node_network_receive_bytes_total

Description

Number of bytes received from the network.

Type

counter

Labels

device

node_network_transmit_bytes_total

Description

Number of bytes transmitted to the network.

Type

counter

Labels

device