Appliance Release v0.39

This page contains the release notes for the v0.39 Anapaya appliance software release. The appliance software release is applicable for the following Anapaya products:

  • Anapaya CORE

  • Anapaya EDGE

  • Anapaya GATE

We recommend always upgrading to the latest available patch release. Please refer to Upgrade Notes (if any) of each release if there are any special steps to be taken when upgrading. For general information on how to upgrade your appliance, please refer to Appliance Update Guide.

Upgrade Notes

Note

This release requires the installation of the anapaya-system package v2.14.0. The installer will automatically reject the installation if the required anapaya-system package is not available.

Warning

After upgrading to v0.39, it is not possible to access secrets (passwords, forwarding key, etc.) that were contained in the appliance configuration in plain text prior to version v0.39.

Please make sure that you have stored a copy of the secrets in a secure location prior to upgrading, if this is not already the case. When upgrading to v0.39, the appliance will dump the secrets contained in the configuration to /root/secrets_dump_v0_38.json. This will allow you to retrieve the secrets and store them in a secure location, provided you have root access to the appliance.

Warning

There are some breaking changes in this release. Please refer to Breaking Changes to check if you are affected and what needs to be done.

v0.39.2 (2025-02-20)

Improvements

  • Add a new BGPUnexportedRoutes alert that reports if there are IP-in-SCION tunneling routes, which are not exported to BGP neighbors along with a runbook that explains how to resolve this issue. To use the new alert you will have to download the latest alert definitions

    curl -O https://dl.cloudsmith.io/<access-token>/anapaya/stable/raw/names/anapaya-alerts-edge/versions/v0.39.2/anapaya-alerts-edge-v0.39.2.yml)

    and import them into your Prometheus instance.

Fixes

  • When calculating the number of buffers per NUMA for VPP the appliance controller now respects the number of NUMA nodes on the system. This ensures that the number of buffers per NUMA node is calculated correctly.

  • The appliance-cli is now able again to set service log levels and enable/disable interface dumps. This was broken in v0.39.0.

v0.39.1 (2025-02-12)

Improvements

  • Write all levels, including the debug, to the FRR logs. This makes it easier to enable the debug logs when needed.

  • The IP-in-SCION tunneling component has improved logging of its monitoring activity. This will help with troubleshooting and monitoring the component’s activity. To not spam the logs, most events are only logged if they constitute a change, i.e., if the same event happens multiple times in a row, then only a single log statement reporting this event will be present in the logs.

Fixes

  • If the username or the password in the basic auth settings for loki are unset, the configuration migration from v0.38 to v0.39 will drop the entire basic auth section for loki. This is done to not trigger validation errors in v0.39.

  • Setting MAC addresses on interfaces with Linux driver now works as expected. Previously, the MAC setting would sometimes not apply, furthermore, interfaces would not be correctly configured after being down.

  • The configuration migration now drops invalid features in the experiments section. Previously, the migration failed because there is a new validation that checks for validity of the features.

  • Reduce Linux route flaps when routing to remote IP-in-SCION endpoints changes.

  • Prevent VRRP VRs going into INTF_DOWN state which required the VR to be restarted or setting the VR interface admin down then up again. The VRRP VRs are now stopped on admin and carrier state down and started again on admin and carrier up.

v0.39.0 (2025-01-27)

Features

Appliance Secret Management

This release includes significant changes to the configuration interface due to a redesigned architecture for secret management. The API is designed so that secrets stored on the appliance cannot be revealed either intentionally or accidentally. As a result, configuration and observability data can be exchanged without the risk of exposure.

Starting with this release, secrets are decoupled from configuration. I.e., secrets are referenced in the appliance configuration rather than being included in plain text. Going forward, it will be safe to share the appliance configuration without any post-processing.

Because secrets are referenced in the appliance configuration, they need to be provisioned first. The appliance API is extended with suitable endpoints for secrets management.

Secret Identifier

Secrets are referenced in the configuration via a secret identifier:

  • scheme: <name>@<version>

  • regex with permitted chars: ^[_a-zA-Z0-9:.-]+@[0-9]+$

The initial version must be 1. Subsequent version must be incremented without any gaps. I.e., if the current highest version is x, the next version must be x+1. If you require a more elaborate versioning scheme, we recommend to encode it in the name portion of the secret identifier.

Secrets Management

The appliance management API contains three new endpoints to manage secrets:

  • GET /v1/secrets: List the existing secret identifiers.

    Example response

    {
  "secrets": [
    {
      "id": "forwarding-key_1-ff00:0:110@1",
    },
    {
      "id": "loki-password@1",
    }
    {
      "id": "loki-password@2",
    }
  ]
}

  • POST /v1/secrets: Add a new secret.

    Example request body

    {
  "secret": {
    "id": "bgp-password@1",
    "value": "myBgpPassword"
  }
}

  • POST /v1/secrets/batch: Add a batch of new secrets.

    Example request body

    {
  "secrets": [
    {
      "id": "bgp-password@1",
      "value": "myBgpPassword"
    },
    {
      "id": "forwarding-key_1-ff00:0:110@1",
      "value": "Zm9yd2FyZGtleQo="
    }
  ]
}
Deployment Workflow

So far, the appliance configuration contained plain text secrets directly. With the move to secret references, the appliances needs to be provisioned with secrets first, before they can be referenced in a configuration. This changes the deployment workflow slightly:

  1. Provision the secrets.

  2. Deploy the appliance configuration with secret references.

For example, to add a BGP password for an existing BGP neighbor, you can add the secret using:

# Interactive mode:
$ appliance-cli secrets add bgp-password@1

# Piping the secret from stdin:
$ echo myBgpPassword > secret.txt
$ cat secret.txt | appliance-cli secrets add bgp-password@1

# Using the management API endpoint:
$ echo '{"id": "bgp-password@1", "value": "myBgpPassword"}' > bgp-password.json
$ appliance-cli post secrets secret: @bgp-password.json

Update the appliance configuration to reference the secret:

$ appliance-cli edit config bgp.neighbors[0].auth_password_ref: bgp-password@1

The appliance configuration validation ensures that the referenced secret exists and has the expected format. For some secret references, a specific format of the referenced plain text secret is required. The API documentation of the secret reference field provides information on the expected format if necessary. For example, the forwarding key must be base64 encoded.

Configuration Migration

When upgrading to this release, the existing appliance configuration is automatically migrated. The appliance automatically chooses secret identifiers for the existing secrets in the appliance configuration.

For example, to find the generated secret identifier for the forwarding key you can use the following command:

$ appliance-cli get config -f body.config.scion.ases[0].forwarding_key_ref

If you desire a different naming scheme, you can re-provision the secrets with an identifier of your choice and update the references in a configuration update.

Note

When upgrading to v0.39, the appliance will dump the secrets contained in the configuration to /root/secrets_dump_v0_38.json. This will allow you to retrieve the secrets and store them in a secure location, provided you have root access to the appliance.

Warning

If you rely on the the migration management API to prepare the configuration for the upgraded appliance, the automatic secrets migration will not be triggered. You need to manually provision the secrets using the new secrets management API. We recommend that you use the auto-migration when upgrading to v0.39.

RADIUS support

This release adds support for RADIUS authentication for the appliance. Together with PAM SSH connections can now be authenticated against a RADIUS server.

To enable RADIUS authentication for SSH connections, both the RADIUS server and PAM service need to be configured. The following example shows how to configure both the RADIUS server and the PAM service. Note that the users need to already exist on the appliance.

{
  "config": {
    "management": {
        "ssh": {
            "enable_password_login": true,
            "radius": {
                "servers": [
                    {
                        "description": "Radius server 1",
                        "secret_id_ref": "radius-secret@1",
                        "address": "10.0.0.1"
                    }
                ]
            }
        },
        "pam": {
            "services": [
                {
                    "auth_modules": ["auth sufficient pam_radius_auth.so conf=/etc/pam_radius_auth.conf"],
                    "description": "SSH login customization",
                    "enabled": true,
                    "service": "sshd"
                }
            ]
        }
    }
}

For more information how to configure RADIUS and PAM refer to the management configuration.

Role-based access control for basic auth users

The appliance API now supports roles for basic auth users. The standard roles "reader" and "writer" that already exit for OAuth2 users are now also available in basic auth. Furthermore, a new role "observer" has been added. The "observer" role has the same capabilities as the "reader" role with the addition of access on the POST /api/v1/debug/scion-tunneling/paths/search and POST /api/v1/tools/scion/* endpoints. This allows observers to have more network insights through additional API endpoints. This new role is available for both basic auth and OAuth2 users. Going forward, the role is required for basic auth users. Existing users are automatically migrated to have the "writer" role to keep the same set of capabilities.

Management API available via a UNIX socket

The appliance management API is now always exposed via a local UNIX socket. This allows local privileged users to interact with the API without the need to authenticate. This is useful as a way to always be able to connect to the management API without any configuration or if other configured listeners are not available. This local UNIX socket can be disabled by setting the management.api.disable_local_unix_socket configuration option to true.

Improvements

  • Introduce an explicit allow_hostname_change query parameter to the PUT /config API endpoint. By default, the appliance API now disallows changing the hostname, except when it is still unset. If the hostname is already set, the API will return a validation error. This is a safety measure to prevent accidental deployment of a configuration meant for a different appliance. If you want to change the hostname after it has been set, you need to set the allow_hostname_change query parameter to true.

  • The Appliance API is extended with a new POST config/diff endpoint which reports the diff between the current active configuration on the appliance, and the provided configuration. The diff is reported in a patch like structure to be easily understandable to operators. We recommend that you use this endpoint before applying a configuration change to sanity check the effects of your change. The endpoint also reports an E-Tag in the response headers, which you can use to ensure that in the subsequent configuration push, the appliance configuration has not changed on the device in the meantime.

  • The GET api/v1/health now supports negative queries to filter out health checks. E.g., you can now pass status=-passing as a query parameter to filter out all health checks with the status passing.

  • The api/v1/debug/bgp/config endpoint now reports the BGP timers in seconds rather than milliseconds.

  • FRR has been updated to version to v10.2.1. This addresses CVE-2024-55553

Fixes

  • Metrics are now always exposed, regardless of whether the management.telemetry.address is configured or not. Metrics have always also been exposed on the management API address, but this was only working if the management.telemetry.address was configured - now they will also be exposed on the management API address even if no telemetry address is configured.

  • The appliance-cli now properly handles --query and --header values that contain a ,. The appliance-cli get health --query status=degraded,failing now works as intended.

  • The appliance-cli info command now deals better with incomplete state when displaying CPPKI information. Previously, the command relied on health checks to show TRC and certificate information. With this change, the /api/v1/config, /api/v1/cppki/trcs, and /api/v1/cppki/certificates endpoints are queried for information, instead of only relying on health checks.

  • Network interfaces with a configured IPv6 address can now be removed, without the need to deconfigure the IPv6 address first.

Breaking Changes

Secrets management

The secrets management has been completely overhauled. Secrets are now stored separately from the appliance configuration. For more information, refer to Appliance Secret Management.

SCION applications directly open an underlay UDP socket

SCION native applications now directly open a UDP socket on the operating system instead of going through the SCION dispatcher. To provide backwards compatibility, there is still a dispatcher component which dispatches traffic sent to port 30041 to the correct application.

This also means that SCION packets do not only use UDP port 30041 anymore, but depend on the L4 port configured for the application. If you run multiple appliances in a cluster setup and have a firewall between them, you need to allow UDP traffic between the appliances on the port configured for the control service (scion.ases[0].control.address) and for the IP-in-SCION tunneling control, data, and probe ports (scion-tunneling.endpoint.control_port, scion-tunneling.endpoint.data_port, scion-tunneling.endpoint.probe_port).

When upgrading to this release, be aware that SCION management API listeners will break as soon as you upgrade the anapaya-system package to the new v2.14.0 version. If you only have a SCION management listener configured, we recommend to also configure at least a localhost listener before upgrading the system package. If you do not have any SCION management listeners configured (the common case), you can safely upgrade to this release.

Finally, SCION management listeners on port 443 are no longer supported. The appliance will automatically migrate them to port 8443 on update. If this port is already used you will need to manually update the configuration before the upgrade.

Secret References

All fields in the appliance configuration that previously contained secrets have been replaced by fields that now refer to the secrets via the identifier.

The following fields have been changed:

  • old: /bgp/neighbors/{neighbor}/auth_password

    new: /bgp/neighbors/{neighbor}/auth_password_ref

  • old: /management/telemetry/logging/loki/basic_auth/password

    new: /management/telemetry/logging/loki/basic_auth/password_ref

  • old: /management/api/oauth/identity_providers/client_secret

    new: /management/api/oauth/identity_providers/client_secret_ref

  • old: /scion/ases/{as}/forwarding_key

    new: /scion/ases/{as}/forwarding_key_ref

  • old: /scion/ases/{as}/ca_service/anapaya_vault/credentials/secret_id

    new: /scion/ases/{as}/ca_service/anapaya_vault/credentials/secret_id_ref

  • old: /scion/ases/{as}/ca_service/external/shared_secret

    new: /scion/ases/{as}/ca_service/external/shared_secret_ref

Appliance UI

In an effort to focus our resources on the most useful features, we decided to remove the appliance UI. Very few users have been using it to configure their appliances. By now, the Anapaya Console offers a much better experience and we want to focus our efforts on that. Alternatively, you can still use the appliance CLI to interact with the appliance from the command line.