Appliance Software Updates

The Anapaya appliance features an installer that can be used to upgrade the software of an Anapaya appliance. The purpose here is to explain how the upgrade process works. For a step by step guide refer to to Appliance Update Guide.

Note

The assumption here is that the Anapaya appliance with installer (i.e., version v0.28.0 or higher) is up and running. Refer to Anapaya Appliance to install an Anapaya appliance initially.

Anapaya ships the appliance software in two packages: anapaya-scion and anapaya-system. The packages and their versioning are described in SCION and System Packages. Appliance Management API gives an overview of the relevant API endpoints for software upgrades. Building on these, the process of upgrading is explained in Upgrade Process.

Note

In the commands below the management API address or the version must be replaced with the actual values. The management API address is configured for the Anapaya appliance. If the commands are run directly via the machine itself, then it can be set to localhost. The version must be set to the anapaya-scion or anapaya-system package that is to be installed.

We recommend setting the variables as environment variables

export APPLIANCE_API=Z.Z.Z.Z
export VERSION_SCION=v0.XX.Y
export VERSION_SYSTEM=vA.B.C

or replace the variables directly in the commands.

By providing the values here, the commands are automatically updated with the correct values.

Version anapaya-scion: $VERSION_SCION v0.37.0 v0.36.4 v0.35.9   Version anapaya-system: $VERSION_SYSTEM v2.13.2 v2.13.0 v2.12.3 v2.12.0   Appliance API Address:

SCION and System Packages

There are two types of packages, anapaya-scion packages and anapaya-system packages. An anapaya-scion package contains Anapaya’s own software and tightly integrated third-party software. An anapaya-system package contains the system packages present on the appliance. (For example, currently, these are the packages included in the ubuntu-minimal 18.04 image and a set of packages that are installed on top.)

The anapaya-scion package is a compressed archive containing the images and binaries that are necessary to run the SCION services. It also contains some files with meta information that can be used for validation of the package or during the installation process.

The anapaya-system package is a compressed archive that contains the desired system packages including all their dependencies in a package format that can be fed into a package manager. Currently, this is the APT package manager.

One of the main reasons for the split into the aforementioned package types are that these packages are on different release cycles.

For both types of packages, semantic versioning in the form of <major>.<minor>.<patch> is used. While for the anapaya-scion packages, any two arbitrary versions v1 and v2 can be installed on top of each other (as far as, v1 and v2 are v0.28 or higher, i.e., have Anapaya installer), this is not entirely true for the anapaya-system packages, due to restrictions imposed by how the system packages are handled. In the following, the semantics behind the versioning schema for anapaya-system packages is explained.

The <major> version indicates the underlying system release version against which the corresponding system package can be installed.

The <minor> version of the system package includes the service pack version. The service packs contain all available updates for every package installed in a major release. The service packs are self-contained with respect to the major release, i.e., it is possible to install service pack 2 and skip service pack 1 for a given major release.

The <patch> releases contain individual updates to packages for a given service pack. The patch releases are on-demand and are usually triggered by security updates that become available for critical security vulnerabilities. The patch releases are self-contained with respect to the currently installed service pack, i.e., it is possible to install patch 2 and skip patch 1 for a given service pack and a given major release.

Appliance Management API

The Anapaya appliance offers an HTTP REST API supporting various operations. In particular, it provides a set of endpoints that can be used to manage and install software packages.

You can refer to Management API Specification, for the complete list of software endpoints that are supported. However, let us have a closer look at some of the endpoints, in particular, the ones needed to upgrade to a new version, as will be explained in Upgrade Process. Additionally, further information regarding signed releases, keys and signatures can be found at Signed Releases.

Authentication

To access the Appliance management API you need to authenticate yourself. The API supports basic authentication. The username and password are configured during the installation of the appliance. When accessing the API using CURL commands you can use the following command line options to authenticate:

curl -u <username>:<password> <url>

scion vs system Endpoints

All endpoints can be classified into either scion or system. As their names suggest, the scion and system endpoints are utilized to handle anapaya-scion and anapaya-system packages, respectively. For example you can use the endpoint GET /software/scion/installed (resp. GET /software/system/installed) to get the version of anapaya-scion (resp. anapaya-system) package currently installed.

keys Endpoint

The keys endpoint allows you to view and update the installed public keys that will be used to verify the integrity and authenticity of the anapaya-scion and anapaya-system packages.

signatures Endpoints

The signatures endpoints allows you to view and update the installed signatures of the anapaya-scion and anapaya-system packages that will be used for package verification. For example, the endpoint GET /software/signatures/scion/$VERSION_SCION can be used to list the installed signatures for the anapaya-scion package of version $VERSION.

Upgrade Process

To upgrade to a new version, you need to download the package for the desired version from Anapaya’s software releases repository and then execute the necessary operations via software endpoints of the appliance management API to complete the installation.

Checking the Installed Version

To check what is the version of the anapaya-scion package which is currently installed, run:

curl -k -u <user>:<password> https://$APPLIANCE_API_PH/api/v1/software/scion/installed
# or using the appliance-cli
appliance-cli get software/scion/installed

For the installed system version, run:

curl -k -u <user>:<password> https://$APPLIANCE_API_PH/api/v1/software/system/installed
# or using the appliance-cli
appliance-cli get software/system/installed

Uploading the Desired Package

Now, you need to ensure that the anapaya-scion/anapaya-system package for the version that you want to install is locally available. Download it from the Anapaya software releases repository (ask your Anapaya Customer Success Engineer if you do not know how to access the repository). Then you can upload the package to the appliance via the management API.

To upload an anapaya-scion package named anapaya-scion-$VERSION_SCION.tar.gz, run:

curl -k -u <user>:<password> https://$APPLIANCE_API_PH/api/v1/software/scion/packages/local -X POST --data-binary "@anapaya-scion-$VERSION_SCION_PH.tar.gz"

Similarly, to upload an anapaya-system package named anapaya-system-$VERSION_SYS.tar.gz, run:

curl -k -u <user>:<password> https://$APPLIANCE_API_PH/api/v1/software/system/packages/local -X POST --data-binary "@anapaya-system-$VERSION_SYS_PH.tar.gz"

To list the version of all anapaya-scion packages which are available locally, run:

curl -k -u <user>:<password> https://$APPLIANCE_API_PH/api/v1/software/scion/packages/local
# or using the appliance-cli
appliance-cli get software/scion/packages/local

Similarly, for anapaya-system packages, run:

curl -k -u <user>:<password> https://$APPLIANCE_API_PH/api/v1/software/system/packages/local
# or using the appliance-cli
appliance-cli get software/system/packages/local

Check that the anapaya-scion or anapaya-system package that you want to install is stored locally.

Upload the Package Signatures

The appliance installer verifies the package integrity and authenticity using the signatures and public keys that are stored in the installer. See Signed Releases for more details.

The installer on an Anapaya appliance already includes the official public keys used by Anapaya to sign the software packages, so there is no further action required to fetch the public keys.

The signatures for the packages can be found on the Anapaya Releases website. Find and download the valid signatures for the package version:

curl https://releases.anapaya.net/anapaya-scion-$VERSION_SCION_PH.tar.gz.signatures.json > signatures.json

Push the latest signatures for this package version to the host using the API:

curl -X POST -k -u <user>:<password> https://$APPLIANCE_API_PH/api/v1/software/signatures/scion/$VERSION_SCION_PH -d @signatures.json
# or using the appliance-cli
appliance-cli post software/signatures/scion/$VERSION_SCION_PH <signatures.json

Triggering the Installation

To trigger the installation of the anapaya-scion package with version $VERSION, run:

curl -k -u <user>:<password> https://$APPLIANCE_API_PH/api/v1/software/scion/install -X POST -d '{"version": "$VERSION_SCION_PH"}'
# or using the appliance-cli
appliance-cli post software/scion/install "version: $VERSION_SCION_PH"

Note

By default, signature verification is enabled. If you want to disable it, you can pass the skip_signature_verification parameter to the request.

curl -k  -u <user>:<password> https://$APPLIANCE_API_PH/api/v1/software/scion/install -X POST -d '{"version": "$VERSION_SCION_PH","skip_signature_verification": true}'
# or using the appliance-cli
appliance-cli post software/scion/install "version: $VERSION_SCION_PH, skip_signature_verification: true"

Note that this should never be used in productive setups.

As a part of the response body of the above API call to trigger the installation, you get an installation ID. You can use this installation ID in the below command to check the progress of the installation:

curl -k -u <user>:<password> https://$APPLIANCE_API_PH/api/v1/software/scion/install/{id}
# or using the appliance-cli
appliance-cli get software/scion/install/{id}

Similarly, to trigger the installation of the anapaya-system package with version $VERSION_SCION, run:

curl -k -u <user>:<password> https://$APPLIANCE_API_PH/api/v1/software/system/install -X POST -d '{"version": "$VERSION_SYS_PH"}'
# or using the appliance-cli
appliance-cli post software/system/install 'version: $VERSION_SYS_PH'

You can again use the returned installation ID in the below command to check the progress of the installation:

curl -k -u <user>:<password> https://$APPLIANCE_API_PH/api/v1/software/system/install/{id}
# or using the appliance-cli
appliance-cli get software/system/install/{id}

The installation status will be “in_progress”, “completed”, or “failed”. In the case of a “failed” installation of an anapaya-scion package, a roll-back to the original version is triggered automatically. For the anapaya-system packages, a roll-back is not supported and manual intervention is required. However, this is very unlikely, should it happen please contact Anapaya Customer Support.

Verifying the Installed Version

Check that the desired version has been successfully installed. For anapaya-scion, run:

curl -k -u <user>:<password> https://$APPLIANCE_API_PH/api/v1/software/scion/installed
# or using the appliance-cli
appliance-cli get software/scion/installed

And for anapaya-system, run:

curl -k -u <user>:<password> https://$APPLIANCE_API_PH/api/v1/software/system/installed
# or using the appliance-cli
appliance-cli get software/system/installed

Rolling Back to a Previous Version

In case of a failed installation of an anapaya-scion package, a rollback to the original version is triggered automatically and no interaction is required. In case you need to manually roll back, the process is the same as the process to install a new version of the package (see ref:core_upgrade_version). In most cases, the package and signatures should already be available on the appliance from a previous installation. If that is not the case, you need to upload the desired package and its signature to the host, and then trigger the installation using the API.

In general, rollbacks are possible for all versions greater than v0.28.0 of the anapaya-scion package. Please refer to the release notes of the newer version to see if any known issues prevent a rollback to the previous version.

For the anapaya-system packages, a roll-back is not supported and manual intervention is required. However, this is very unlikely, should it happen please contact Anapaya Customer Support.

Signed Releases

To verify that the Anapaya software has not been tampered with from the time it is released until the operator installs it on a device, we cryptographically sign our software releases of the anapaya-scion and anapaya-system packages. This enables our own installer as well as third parties to verify the authenticity and integrity of our software releases against a root of trust.

Signing and verification process

At a high level, a signature is a cryptographically verifiable statement that a certain piece of data - in our case, a software package - (1) has been signed by the owner of the private key corresponding to the public key that is used to verify the signature (authenticity) and (2) that the data has not been modified since it has been signed (integrity).

To sign our releases, we generate an ECDSA-P256 key pair. Using the private key, our signing tool computes the signature over the SHA256 hash of an Anapaya software package, which will serve as the signature of the corresponding package. The private key is stored in a secure, access-controlled location to ensure that it is not compromised.

Retrieving public keys and package signatures

To ensure that the verification process is valid, the verifier (operator) needs to have trust in the public key used for verification. To ensure that operators have access to Anapaya’s untampered, valid public keys, we publish them on our website. Specifically, the public keys and the signatures of the Anapaya packages can be found at releases.anapaya.net. In this way, as long as Anapaya manages its own releases, operators can trust that the published keys are valid.

The releases.anapaya.net page contains two sections:

• The Signing keys section contains the keys.json file. This is a JSON file that lists all the valid public keys and metadata (e.g. fingerprint and creation time) corresponding to the private keys that Anapaya uses to sign a release version.

• The Signatures section is split into two subsections for the anapaya-scion and the anapaya-system. For each new release of the anapaya-scion or anapaya-system package, we provide a JSON file that contains the name and checksum of the package, as well as a list of package signatures along with the signature creation time and the public key that can be used to verify the package.

With every release, signatures are added to releases.anapaya.net to reflect the latest state of affairs.

Updating installed public keys

The software installer provided by Anapaya already includes the official public keys used by Anapaya to sign the software packages. Usually, there is no further action required to fetch the public keys. Still, there can be cases where it is required to update the list of public keys on an appliance, for instance in case of a key compromise or periodic key update. To achieve this, you can follow these steps:

1. Fetch the latest public keys from Anapaya’s website.

   curl https://releases.anapaya.net/keys.json > keys.json
   

2. Update the installed keys using the Appliance Installer’s API.

   curl -X POST -k -u <user>:<password> https://$APPLIANCE_API_PH/api/v1/software/keys -d @keys.json
   

3. (Optional): Verify which public keys are currently installed.

   curl -k -u <user>:<password> https://$APPLIANCE_API_PH/api/v1/software/keys
   # or using the appliance-cli
   appliance-cli get software/keys
   

Verifying the signature manually

To manually verify that the signature of the software package is valid, you can use Cosign. Cosign is part of the larger sigstore project - a new standard for signing, verifying and protecting software that is owned by the Linux Foundation and has been developed by a collaboration between Redhat and the Google Security team.

After installing cosign following the official instructions, you can verify a signature by running:

cosign verify-blob --key <path-to-public-key> --signature <signature> <path-to-package>

where:

• <path-to-public-key> is the path to the public key that should be used for signature verification. The public key must be in PEM format.

• <signature> is the signature of the software package in string format

• <path-to-package> is the path to the software package for which you are doing the verification.

  For example, the command could look similar to the following:

  cosign verify-blob --key public_keys/public_key1.json --signature MEQCIBS8KnIP07fRD2Hi1L0HvvZ2Rc4jkCifx8pjR+/0NxTXAiACzQ+f4piwhVisUdCnJj0bpQgi0CsDlCCJd5MgdPznwA== anapaya-scion-$VERSION_SCION_PH.tar.gz
  

In case of key compromise

While Anapaya goes to great lengths to ensure the private signing keys are secure, we have defined a process for the unlikely case of key compromise. If a private key is compromised, Anapaya will inform all affected parties via email. The compromised key will be removed from our webpage (releases.anapaya.net), and our webpage will explicitly state which key was tainted.

You must then follow this process:

1. Update the verification public keys that are installed in all appliances, following the steps in Updating installed public keys.

2. Reinstall the anapaya-scion and anapaya-system packages following the steps in Upgrade Process.

With this process, you can ensure that the installed public keys are up-to-date and that the Anapaya software packages can successfully be verified.

Custom package installation

Warning

The installation of custom packages might interfere with the Anapaya-managed packages. The following configuration changes might lead to issues with the appliance installer when upgrading to a newer software version and cause the system to break. We highly recommend avoiding custom package installation, unless absolutely necessary.

By default, based on the latest software versions, it is not possible to install packages that are not part of the anapaya-system package. However, there might be special cases that require custom package repositories and additional packages to be available. In such cases, you need to follow the next steps:

If the appliance has Internet access

Note

For the following commands, the operator needs to be connected to the appliance as the root user or use sudo.

1. Copy the following section to the file /etc/apt/sources.list.d/jammy.list and save the file.

   deb http://archive.ubuntu.com/ubuntu jammy main restricted
   deb http://archive.ubuntu.com/ubuntu jammy-updates main restricted
   
   deb http://archive.ubuntu.com/ubuntu jammy universe
   deb http://archive.ubuntu.com/ubuntu jammy-updates universe
   
   deb http://archive.ubuntu.com/ubuntu jammy multiverse
   deb http://archive.ubuntu.com/ubuntu jammy-updates multiverse
   
   deb http://security.ubuntu.com/ubuntu jammy-security main restricted
   deb http://security.ubuntu.com/ubuntu jammy-security universe
   deb http://security.ubuntu.com/ubuntu jammy-security multiverse
   

2. Update the package lists on your appliance by running apt update.

3. Install the package running apt install <package>.

4. After installing the custom package, you need to remove the apt sources that were added above. To do this, delete the file /etc/apt/sources.list.d/jammy.list that was added in step 2.

If the appliance does not have Internet access

1. Find the desired package in the official list of Ubuntu Jammy packages.

2. Download the amd64 package version to your local system.

3. Copy it to the appliance device (you can use scp).

4. Install the debian package using dpkg -i <package>.deb