SCION CPPKI Certificate Authority Service¶

The SCION CPPKI requires some ASes to act as Certificate Authorities (CA) in their local ISD. These ASes need to provide a certificate renewal service to their customer ASes. The certificate renewal request is sent via SCION to the Anapaya appliance in the CA AS, and on successful certificate issuance, a valid SCION CPPKI Certificate Chain consisting of the SCION CPPKI AS and CA certificates is returned. This document describes the different deployment models the Anapaya appliance supports for such a CA AS, and goes into details on the configuration and operations of the standard solution.

Deployment Models¶

The Anapaya Appliance supports multiple deployment models for a SCION CPPKI Certificate Authority (CA). The different models are tailored to address different needs and requirements. In the rest of this section, these models are explained in more detail.

External SCION CPPKI Certificate Authority¶

In high-security networks, customers usually require specific policies on the management of the certificates and keys. These customers can provide their own implementation of a SCION CPPKI Certificate Authority that adheres to these policies. For example, a policy could be to store private keys for the SCION CPPKI CA certificate in an HSM, or to have strict identity checks before issuing an initial SCION CPPKI AS certificate.

The Anapaya appliance is capable of using an external SCION CPPKI Certificate Authority implementation via a REST HTTP API. For this deployment type, the Anapaya appliance needs to be configured with a shared-secret such that it can authenticate itself with the external CA service.

Vault-based SCION CPPKI Certificate Authority¶

The standard solution is to use HashiCorp Vault as the SCION CPPKI Certificate Authority backend. The Anapaya appliance internally has an adapter that exposes the REST HTTP API and translates them into calls to the Vault.

The required configuration of Vault is described in Vault-based CA: Configuration.

The following list defines terms that will be used extensively throughout the document:

Secrets Engine: A secrets engine is a term defined by Vault that describes a component that stores, generates, or encrypts data. They are very flexible and define most of Vault’s capabilities. Secrets engines are enabled at a “path” (think URL path). Multiple instances of the same secrets engine can be mounted at different paths. This allows us to mount separate PKI engine instances per SCION AS.

For more details, see https://www.vaultproject.io/docs/secrets.
PKI Engine: Vault defines a secrets engine for handling X.509 certificates called PKI. To avoid confusion, this secrets engine is called PKI engine for the rest of this document.

For more details, see https://www.vaultproject.io/docs/secrets/pki.
Policy: Various degrees of access to secrets and secrets engine functionality can be defined for individual users and/or applications. The rules for access control are defined in policies.

For more details, see https://www.vaultproject.io/docs/concepts/policies.

The following diagram highlights the interactions between different Vault clients and the respective Vault endpoints.

Certificate Authority Adapter

The Certificate Authority (CA) adapter is part of the Anapaya Appliance. It only interacts with a subset of the available secrets engine endpoints. The sign-verbatim endpoint is used by the CA adapter to service the certificate renewal requests. This endpoint issues certificates as specified in a CSR. All the validation steps are done in the CA adapter itself.

To simplify the storage requirements, and ensure consistency between different Anapaya appliances, Vault acts as the storage backend for the necessary TRCs, as well as the blocklist that prevents specific SCION ASes from renewing their certificates.

The CA adapter requires a token to interact with Vault. Tokens in Vault always have a TTL and need to be renewed periodically, the CA adapter takes care of this internally. It authenticates itself periodically with Vault via the AppRole authentication method and fetches the token. The secrets for this need to be provisioned in the Anapaya appliance configuration.
Certificate Authority Provisioning

The Vault needs to be provisioned with the right data in order for the SCION AS to provide the correct SCION CPPKI CA functionality. Because of the short lifetime of the SCION CPPKI CA certificates (recommended is 11 days), this task needs to be automated and executed periodically.
Manual Intervention

Manual intervention is only necessary to adapt the blocklist and to provision new TRCs after a TRC update.

In-process SCION CPPKI Certificate Authority¶

For testing purposes, the Anapaya appliance supports renewing SCION CPPKI AS certificates in-process. In this mode, no external CA service or Vault is required. However, a valid CA private key and the corresponding SCION CPPKI CA certificate are required and must be stored on the appliance itself.

Vault-based CA: Configuration¶

In this section, we describe the necessary configuration of HashiCorp Vault such that it can be used as a SCION CPPKI Certificate Authority backend by the Anapaya appliance. We also list deployment considerations that an operator should be aware of.

Vault cluster¶

The Vault cluster is an integral part of the CA architecture. It should be deployed with support for individual Vault node failure. This can be achieved by deploying multiple Vault instances in a cluster. (see: High Availability Mode)

Vault can be run on top of various different databases that provide the storage layer. However, for simplicity, we suggest that Vault is run using the integrated storage.

Authentication Setup¶

The Anapaya appliance (via the CA adapter) needs to authenticate itself with Vault to gain access to the target endpoints. For this, the AppRole authentication method must be enabled, and the appropriate roles need to be provisioned. Each CA adapter should get its own individual identity and role_id, such that access to individual Anapaya appliances can be revoked without affecting the remaining appliances.

Each CA adapter should get the right policy assigned, such that it can access the correct Vault PKI Engine and the Key/Value Engine.

To take advantage of the policies that are listed in Policies, the Vault identity for the CA adapter needs to have the isd_as with the ISD-AS number in its metadata. To do this, execute the following steps:

Create a Vault identity for the CA adapter, one identity per adapter instance.

The suggested name pattern is of the form ca-adapter_<isd-as>-<instance-id>. The metadata should include the key isd_as with the ISD-AS number as the value and the identity should have the CA Adapter Policy attached to it. Use the /identity/entity endpoint to create the identities.
Create a Vault AppRole for each CA adapter instance.

The suggested name pattern is ca-adapter_<isd-as>-<instance-id>. Use the /auth/approle/role/:role_name endpoint.
Create a Vault entity alias that links the AppRole and the identity together.

Create an alias so that once you log in with the AppRole you are automatically assigned the entity. For this, you will need the following information:
- role ID of the CA adapter. Get this with the /auth/approle/role/:role_name/role-id endpoint. (data.role_id in the response body)
- identity ID of the CA adapter identity. Get this with the /identity/entity/name/:name endpoint. (data.id in the response body)
- accessor ID of the AppRole authentication engine. Get this with the /sys/auth endpoint. (.["approle/"].accessor in the response body)
Then, link the AppRole to the identity by creating an alias using the /identity/entity-alias endpoint. The name is set to the role ID, the canonical_id is set to the identity ID, and the mount_accessor is set to the accessor ID.
Extract role_id and secret_id of the created AppRole(s) and distribute them to provision the Anapaya appliance where the instance of the CA adapter should run. To read the secret_id, you can use the /auth/approle/role/:role_name/secret-id endpoint.

Secrets Engine Setup¶

In this section, we describe the different Vault Secrets Engines that are required for the Vault to act as a CA backend.

PKI Engine¶

Docs: https://www.vaultproject.io/docs/secrets/pki

The PKI engine is the core of the CA backend. It stores the currently active SCION CPPKI CA certificate and the associated private key, and is used to issue the renewed certificates.

Create one active PKI engine per SCION CA AS that is backed by the Vault backend. Each CA adapter only has access to the PKI engine for their respective ISD-AS. The currently active PKI engine is mounted at /ca/<isd-as>/pki. For example, the PKI engine for AS 1-ff00:0:110 is mounted at /ca/1-ff00:0:110/pki. The CA adapter will use the /ca/<isd-as>/sign-verbatim endpoint to request the certificate issuance based on the CSR that it received from the customer AS.

We suggest that each PKI engine is only ever associated with exactly one CA certificate. This allows for a clean history of what SCION CPPKI AS certificates were issued with any given SCION CPPKI CA certificate. To achieve this, we recommend the Rolling CA Provision Strategy that is described below.

Key/Value Engine¶

Docs: https://www.vaultproject.io/docs/secrets/kv/kv-v2

The key/value engine is used to keep the state for our CA adapter. For verification of the request, the CA adapter needs the currently active TRC (and potentially the TRC that is in the grace period currently). Furthermore, the CA adapter needs access to the blocklist that allows operators to manually block certain ASes, and prevent them from renewing their certificate.

Again, mount one instance per SCION CPPKI CA AS at /ca/<isd-as>/kv. For example, the key/value engine for AS 1-ff00:0:110 is mounted at /ca/1-ff00:0:110/kv. When enabling the Vault storage mount at ca/<isd-as>/kv, version 2 must be selected.

Note

We require version 2 of the key/value engine in vault. Follow the vault setup documentation and ensure you are using the right version.

/ca/<isd-as>/kv/data/trcs hosts the TRCs for the local ISD.

The TRCs are keyed by the following pattern ISD<isd>-B<base>-S<serial>. The value must be a PEM encoded TRC. For example, the TRC ISD1-B1-S1 is expected to be located at /ca/<isd-as>/kv/data/trcs/ISD1-B1-S1. Additionally, the latest TRC must be located at /ca/<isd-as>/kv/data/trcs/latest.

The CA adapter uses the TRCs to verify the signatures on the CSRs that are received from the customer AS. It will first look up the latest TRC. If the latest TRC implies that the predecessor TRC is still in the grace period, it will additionally look up that predecessor TRC at the keyed path.
/ca/<isd-as>/kv/data/blocklist hosts the blocklist.

For each AS that is on the blocklist, an entry in the KV story is required. The data of the entry can be empty, the presence is enough to block an AS. For example to put 1-ff00:0:111 into the blocklist, it suffices to post on the path ca/<isd-as>/kv/data/blocklist/1-ff00:0:111. To check if 1-ff00:0:111 is on the blocklist, read from the same path. If the status code is 404 the item is not on the blocklist, if the status code is 200 the item is blocked.

The CA adapter uses this blocklist every time it gets a CSR to decide whether a certificate should be issued, or whether the request should be denied.

Policies¶

Each role gets its individual policy. Each policy should grant the role the minimal set of capabilities for proper operation.

CA Adapter Policy¶

The CA adapter should only get access to the endpoints belonging to its own ISD-AS. We can leverage identity.entity.metadata.isd_as to ensure this. The metadata is set as part of the AppRole creation for the CA adapter. The CA adapter should get the following policy associated with its role:

# The sign-verbatim endpoint is required to sign new AS certificates.
path "ca/{{identity.entity.metadata.isd_as}}/pki/sign-verbatim" {
    capabilities = ["create", "update"]
}

# TRCs are required in the CA adapter to verify requests.
path "ca/{{identity.entity.metadata.isd_as}}/kv/data/trcs/+" {
    capabilities = ["read"]
}

# Blocklist is required in the CA adapter to validate requests.
path "ca/{{identity.entity.metadata.isd_as}}/kv/data/blocklist/+" {
    capabilities = ["read"]
}

# The CA adapter needs to login via approle to get a token for the other endpoints.
path "auth/approle/login" {
    capabilities = ["create", "update"]
}

CA Provisioning Policy¶

The exact policy for the CA provisioning is dependent on its implementation. To support the Rolling CA Provision Strategy, the following should be part of the policy:

# The CA provision task needs to be able to configure the CA certificate.
path "ca/+/+/config/ca" {
    capabilities = ["create", "update"]
}

# The CA provision task needs to be able to create new mounts.
path "sys/mounts/ca/+/+" {
    capabilities = ["create", "update", "delete"]
}

# The CA provision task needs to be able to remount old CAs.
path "sys/remount" {
    capabilities = ["create", "update", "sudo"]
    allowed_parameters = {
        "from" = ["ca/*"]
        "to" = ["ca/*"]
    }
}

# The CA provision task needs to be able to tune the newly create PKI engine
path "sys/mounts/ca/+/+/tune" {
    capabilities = ["create", "update"]
    allowed_parameters = {
        "default_lease_ttl" = []
        "max_lease_ttl" = []
        "force_no_cache" = [false]
        "options" = []
    }
}

Manual intervention¶

The exact policy for manual intervention depends on the capabilities an operator should have in order to intervene. The following is a suggestion with a potentially useful capability set for an operator.

# Provision, list and read TRCs.
patch "ca/+/kv/data/trcs/+" {
    capabilities = ["read", "create", "update"]
}

# Add, list, and delete entities from the blocklist.
patch "ca/+/kv/data/blocklist/+" {
    capabilities = ["read", "create", "update", "delete"]
}

# Manually sign certificates.
path "ca/+/pki/sign-verbatim" {
    capabilities = ["create", "update"]
}

# Manually provision CA certificate.
path "ca/+/pki/config/ca" {
    capabilities = ["create", "update"]
}

# Manually run garbage collection
path "ca/+/pki/tidy" {
    capabilities = ["create", "update"]
}

Vault-based CA: Operations¶

The following section describes common tasks that are done while operating a Vault-based CA backend.

Upload a new TRC¶

To add a new TRC to the Key/Value Engine, you need to use the /ca/<isd-as>/kv/data/trcs/ISD<isd>-B<base>-S<serial> endpoint. The TRC must be PEM encoded.

Note

Make sure that the TRC is PEM encoded. The raw ASN.1 DER encoding is not supported. You can format any TRC to PEM encoding with the following command:

scion-pki trc format <TRC-file>

For example, to upload the TRC ISD1-B1-S4, you could use the following curl command:

curl \
  --header "X-Vault-Token: $VAULT_TOKEN" \
  --request POST \
  --data '{"data": {"trc": "-----BEGIN TRC-----\nMIIJ..."}}' \
  https://<vault-addr>/v1/ca/1-ff00:0:110/kv/data/trcs/ISD1-B1-S4

If the uploaded TRC is the new latest TRC, you also need to upload it to /ca/<isd-as>/kv/data/trcs/latest. This could be done with the following curl command:

curl \
  --header "X-Vault-Token: $VAULT_TOKEN" \
  --request POST \
  --data '{"data": {"trc": "-----BEGIN TRC-----\nMIIJ..."}}' \
  https://<vault-addr>/v1/ca/1-ff00:0:110/kv/data/trcs/latest

Note

In order for the SCION CPPKI AS certificate to be verifiable, the Anapaya appliance should also be provisioned with this TRC. Furthermore, the CA certificate potentially has to be rotated if the issuing Root Certificate has changed in the TRC update.

Add an AS to the blocklist¶

To add an AS to the blocklist, you need to create an entry in the Key/Value Engine with the /ca/<ca-isd-as>/kv/data/blocklist/<blocked-isd-as> endpoint. For example, to block ISD-AS 1-ff00:0:112 in the CA 1-ff00:0:110, you could use the following curl command:

curl \
  --header "X-Vault-Token: $VAULT_TOKEN" \
  --request POST \
  --data '{"data": {"blocked": true}}' \
  https://<vault-addr>/v1/ca/1-ff00:0:110/kv/data/blocklist/1-ff00:0:112

Note

The "blocked": true is for informational purposes only. The existence of the entry blocks the 1-ff00:0:112 from getting a renewed SCION CPPKI AS certificate. You are free to add any additional informational metadata.

Remove an AS from the blocklist¶

To remove an AS from the blocklist, you need to delete the entry in the Key/Value Engine with the /ca/<ca-isd-as>/kv/metadata/blocklist/<blocked-isd-as> endpoint. For example, to remove the ISD-AS 1-ff00:0:112 in the CA 1-ff00:0:110 from the blocklist, you could use the following curl command:

curl \
  --header "X-Vault-Token: $VAULT_TOKEN" \
  --request DELETE \
  https://<vault-addr>/v1/ca/1-ff00:0:110/kv/metadata/blocklist/1-ff00:0:112

Rolling CA Provision Strategy¶

Vault is designed such that only one CA certificate is used per PKI engine. For example, it is not possible to query what CA certificate was used to issue an AS certificate via the /pki/cert endpoint. Together with the short lifetime of the CA certificates, this severely limits the observability and auditability of the system.

To remedy this problem, we suggest a rolling CA certificate provisioning strategy. In this strategy, you do not reconfigure the PKI engine with the updated CA certificate and private key. Instead, the previous PKI engine is archived, and a new PKI engine with the new crypto material is created. This must be transparent to our CA adapter to simplify configuration and business logic. You can achieve this with a sequence of /sys/remount operations:

Create the new PKI engine at ca/<isd-as>/pki-next and provision it with the fresh CA certificate and private key.
Archive the current PKI engine by remounting ca/<isd-as>/pki to ca/<isd-as>/pki_<timestamp>, where the timestamp is the current date in yyyymmddHHMMSS format. E.g., ca/1-ff00:0:110/pki_20210624090000.
Remount the new PKI engine from ca/<isd-as>/pki-next to ca/<isd-as>/pki.

This process can be scripted, and thus should cause a very minimal downtime. The advantage here is, that the CA adapter does not need to decide which PKI engine instance to talk to. It is always reachable under the same path at ca/<isd-as>/pki.

Note

There is a practical limit on the number of mount points of ~14000. If you rotate the CA certificate every 14 days and run a single CA in Vault, you run out of mount points after more than 250 years. If you run more CAs in the same Vault cluster, this number decreases.

https://www.vaultproject.io/docs/internals/limits#mount-point-limits

The current amount of mounts can be monitored via the /sys/mounts endpoint. Before you exhaust the limits, you can drop old PKI engines that are no longer needed for auditing purposes.