Deployment Example

In the following, we describe a typical small deployment scenario, where an ISP (ISP 1) operates two CORE appliances, both connected to a peering SCION ISP. Furthermore, the ISP has two downstream customers that are connected to each of the two CORE appliances.

This redundant access design is a best practice because it increases the redundancy of the customer access and allows the ISP to perform maintenance on one of the CORE appliances without causing a service interruption.

Topology

../../_images/core_deployment.drawio.svg

Our target topology contains the following elements:

  • ISP 1 deploys two CORE appliances, s01.chzrh1.isp1 and s01.chbrn1.isp1, that are connected internally via the internal network 10.0.0.0/24.

  • ISP 1 is peering with ISP 2 (ISD-AS 1-ff00:2:1) at s01.chzrh1.isp1. The local SCION interface ID is 10 and the remote SCION interface ID is 20. The underlay IPv6 network used to establish the link is fe80:1::/64, where fe80:1::1/64 is the local IPv6 address and fe80:1::2/64 is the remote IPv6 address.

  • ISP 1 is peering with ISP 3 (ISD-AS 3-ff00:3:1) at s01.chbrn1.isp1. The local SCION interface ID is 20 and the remote SCION interface ID is 5. The underlay IPv6 network used to establish the link is fe80:2::/64, where fe80:2::1/64 is the local IPv6 address and fe80:2::/64 is the remote IPv6 address.

  • ISP 1 is connecting Customer 1 (ISD-AS 1-ff00:1:10) redundantly:

    • s01.chzrh1.isp1: The local SCION interface ID is 100 and the remote SCION interface ID is 1. The underlay IPv4 network used to establish the link is 169.254.0.0/30, where .1 is the local IPv4 address and .2 is the remote IPv4 address.

    • s01.chbrn1.isp1: The local SCION interface ID is 201 and the

      remote SCION interface ID is 2. The underlay IPv4 network used to establish the link is 169.254.2.0/30, where .1 is the local IPv4 address and .2 is the remote IPv4 address.

  • ISP 1 is connecting Customer 2 (ISD-AS 1-ff00:1:20) redundantly:

    • s01.chzrh1.isp1: The local SCION interface ID is 101 and the remote SCION interface ID is 2. The underlay IPv4 network used to establish the link is 169.254.3.0/30, where .1 is the local IPv4 address and .2 is the remote IPv4 address.

    • s01.chbrn1.isp1: The local SCION interface ID is 200 and the

      remote SCION interface ID is 1. The underlay IPv4 network used to establish the link is 169.254.1.0/30, where .1 is the local IPv4 address and .2 is the remote IPv4 address.

We will now configure the CORE appliances step-by-step to implement the above scenario.

Network Interface Configuration

First, we will look at the configuration for the network interfaces. In the setup, there are three physical network interfaces - one for the internal network lan, one for the peering link to the neighboring SCION ISP ens1, and one for the links to the downstream customer ens2. The links to the downstream customers are configured as VLANs on the ens2 interface.

Note

It is not required that each link is on a separate physical interface. It is also possible to have multiple links on the same physical interface in the same or different VLANs, or multiple virtual functions on top of a single physical interface if the underlying hardware supports it.

In this example, we also configure the lan interface to be the default route.

Network Interface Configuration

s01.chzrh1.isp1

"interfaces": {
  "ethernets": [
    {
      "addresses": ["10.0.0.100/24"],
      "name": "lan",
      "gateway": {
        "ipv4_gateway": "10.0.0.1"
      }
    },
    {
      "addresses": ["fe80:1::1/64"],
      "name": "ens1"
    },
    {
      "name": "ens2"
    }
  ],
  "vlans": [
    {
      "name": "ens2.100",
      "addresses": ["169.254.0.1/30"],
      "link": "ens2",
      "id": 100
    },
    {
      "name": "ens2.200",
      "addresses": ["169.254.3.1/30"],
      "link": "ens2",
      "id": 200
    }
  ]
},

s01.chbrn1.isp1

"interfaces": {
  "ethernets": [
    {
      "addresses": ["10.0.0.200/24"],
      "name": "lan",
      "gateway": {
        "ipv4_gateway": "10.0.0.1"
      }
    },
    {
      "addresses": ["fe80:2::1/64"],
      "name": "ens1"
    },
    {
      "name": "ens2"
    }
  ],
  "vlans": [
    {
      "name": "ens2.100",
      "addresses": ["169.254.1.1/30"],
      "link": "ens2",
      "id": 100
    },
    {
      "name": "ens2.200",
      "addresses": ["169.254.2.1/30"],
      "link": "ens2",
      "id": 200
    }
  ]
},

Please refer to Network Interfaces for the full documentation on network interface configuration and to Network for troubleshooting network configuration issues.

SCION Configuration

Next, we will look at the SCION configuration. The SCION section contains the configuration of the SCION protocol and AS and can be grouped into the general AS, the control plane, and the data plane configuration.

General AS Configuration

Each SCION AS has several general AS configuration options such as the ISD-AS identifier, the AS forwarding secret key, or a human-readable description of the AS. For the full list of the general AS configuration options, please refer to General AS Configuration.

In this example, we configure the following fields:

isd_as

The ISD-AS identifier of the AS. In this example, the ISD-AS identifier is 1-ff00:0:1. This must be the same for all CORE appliances.

forwarding_key

The base64 encoded secret key used to verify the integrity and authenticity of the SCION packet header during forwarding on the data plane. This must be the same for all CORE appliances.

Note

In the example, we use a dummy key. In production, the secret key should be a strong password that is at least 12 characters long and contain uppercase letters, lowercase letters, numbers, and special characters.

core

The core flag indicates whether the AS is a core AS. In this example, ISP 1 is a core AS of ISD 1 and therefore sets this flag to true.

scion_mtu

The SCION MTU is the maximum size of a SCION packet supported in the internal network. Since we are using an IPv4 underlay network, we set this to 1472 bytes. This assumes an Ethernet frame size of 1500 bytes, minus 20 bytes for the IPv4 header and 8 bytes for the UDP header that is used to transport SCION packets. This must be the same for all CORE appliances.

shard_id

CORE appliances are typically deployed in a sharded manner, where each shard can function independently. The shard_id is used to identify the shard. The shard_id must be unique for each CORE appliance in a given AS, as each constitutes its own shard.

General AS Configuration

s01.chzrh1.isp1

"scion": {
  "ases": [
    {
      "isd_as": "1-ff00:0:1",
      "forwarding_key": "Zm9yd2FyZGluZ19rZXk=",
      "core": true,
      "scion_mtu": 1472,
      "shard_id": 1,

s01.chbrn1.isp1

"scion": {
  "ases": [
    {
      "isd_as": "1-ff00:0:1",
      "forwarding_key": "Zm9yd2FyZGluZ19rZXk=",
      "core": true,
      "scion_mtu": 1472,
      "shard_id": 2,

Control Plane Configuration

The SCION control plane configuration consists of the control and cppki sections that configure the network endpoint of the control service and the AS certificate renewal. The full documentation on the SCION control plane configuration can be found in Control Plane Configuration.

The network endpoint of the control service is typically configured to listen on the internal interface. The port can be set to 0 to let the appliance automatically allocate a free port.

The cppki section configures the SCION AS certificate issuers. These are required to periodically renew the AS certificate on the CORE appliances. In this example, we assume that there are two certificate issuers for ISD 1 at 1-ff00:0:100 and 1-ff00:0:200. We want to configure one as the primary issuer and the other as the fallback in case the primary issuer is not reachable.

SCION Control Plane Configuration

s01.chzrh1.isp1

"control": {
  "address": "10.0.0.100:0",
  "enabled": true
},
"cppki": {
  "issuers": [
    {
      "isd_as": "1-ff00:0:100",
      "priority": 0
    },
    {
      "isd_as": "1-ff00:0:200",
      "priority": 1
    }
  ]
},

s01.chbrn1.isp1

"control": {
  "address": "10.0.0.200:0",
  "enabled": true
},
"cppki": {
  "issuers": [
    {
      "isd_as": "1-ff00:0:100",
      "priority": 0
    },
    {
      "isd_as": "1-ff00:0:200",
      "priority": 1
    }
  ]
},

SCION Data Plane Configuration

The SCION data plane configuration consists of the router and the neighbors sections. The router section contains the configuration of the internal SCION interface internal_interface, while the neighbors section defines all the SCION links to other SCION ASes, i.e., the external SCION interfaces. The full documentation on the SCION data plane configuration can be found in Data Plane Configuration.

The internal SCION interface is where the appliance receives SCION packets from AS internal hosts for forwarding. It is defined as a UDP/IP network endpoint on which the SCION packets are received from the internal network. The endpoint needs to be specified as a string in the format <ip>:<port>. If the port is set to 0 it will be automatically assigned.

In our case, we configure the internal SCION interface to listen on the lan interface.

Each entry in the neighbors section contains a remote SCION AS to which one or multiple links are established. Each of the CORE appliances has two neighbors - CORE link to ISP 2/3 and a CHILD link to Customer 1/2. For each neighbor, we need to specify the local and remote interface network address endpoints, the link type (CORE or CHILD), the SCION interface ID, and the SCION MTU supported on this link. The SCION interface ID must be unique for each SCION interface in an AS. We set the SCION MTU to 1472 bytes for the links to the customers that use an IPv4 underlay network and 1452 bytes for the links to the other ISPs that use an IPv6 underlay network.

SCION Data Plane Configuration

s01.chzrh1.isp1

"router": {
  "enabled": true,
  "internal_interface": "10.0.0.100:30100"
},
"neighbors": [
  {
    "neighbor_isd_as": "1-ff00:2:1",
    "relationship": "CORE",
    "interfaces": [
      {
        "address": "[fe80:1::1]:31000",
        "administrative_state": "UP",
        "interface_id": 10,
        "remote": {
          "address": "[fe80:1::2]:31000",
          "interface_id": 20
        },
        "scion_mtu": 1452
      }
    ]
  },
  {
    "neighbor_isd_as": "1-ff00:1:10",
    "relationship": "CHILD",
    "interfaces": [
      {
        "address": "169.254.0.1:31000",
        "administrative_state": "UP",
        "interface_id": 100,
        "remote": {
          "address": "169.254.0.2:31000",
          "interface_id": 1
        },
        "scion_mtu": 1472
      },
      {
        "address": "169.254.3.1:31000",
        "administrative_state": "UP",
        "interface_id": 101,
        "remote": {
          "address": "169.254.2.2:31000",
          "interface_id": 2
        },
        "scion_mtu": 1472
      }
    ]
  }
]

s01.chbrn1.isp1

"router": {
  "enabled": true,
  "internal_interface": "10.0.0.200:30100"
},
"neighbors": [
  {
    "neighbor_isd_as": "3-ff00:3:1",
    "relationship": "CORE",
    "interfaces": [
      {
        "address": "[fe80:2::1]:31000",
        "administrative_state": "UP",
        "interface_id": 20,
        "remote": {
          "address": "[fe80:2::2]:31000",
          "interface_id": 5
        },
        "scion_mtu": 1452
      }
    ]
  },
  {
    "neighbor_isd_as": "1-ff00:1:20",
    "relationship": "CHILD",
    "interfaces": [
      {
        "address": "169.254.1.1:31000",
        "administrative_state": "UP",
        "interface_id": 200,
        "remote": {
          "address": "169.254.1.2:31000",
          "interface_id": 1
        },
        "scion_mtu": 1472
      },
      {
        "address": "169.254.2.1:31000",
        "administrative_state": "UP",
        "interface_id": 201,
        "remote": {
          "address": "169.254.3.2:31000",
          "interface_id": 2
        },
        "scion_mtu": 1472
      }
    ]
  }
]

Please refer to SCION for the full documentation on the SCION configuration and to SCION / CPPKI for troubleshooting SCION configuration issues.

Cluster Configuration

The CORE appliances are deployed in a sharded manner as part of a cluster. To still work as a single unit, the appliances exchange topology and SCION beacon and path information amongst each other.

The cluster configuration includes the local cluster endpoint and the list of peers that are part of the cluster. For CORE deployments we recommend to use automatic topology synchronization (see Cluster for more details).

Cluster Configuration

s01.chzrh1.isp1

"cluster": {
  "synchronization": {
    "address": "10.0.0.100:40000"
  },
  "peers": [
    {
      "name": "s01.chbrn1.isp1",
      "synchronization": {
        "address": "10.0.0.200:40000"
      }
    }
  ]
},

s01.chbrn1.isp1

"cluster": {
  "synchronization": {
    "address": "10.0.0.200:40000"
  },
  "peers": [
    {
      "name": "s01.chzrh1.isp1",
      "synchronization": {
        "address": "10.0.0.100:40000"
      }
    }
  ]
},

Complete Configuration

For completeness, we provide the full configuration of each CORE appliance:

Complete Configuration s01.chzrh1.isp1

{
  "config": {
    "interfaces": {
      "ethernets": [
        {
          "addresses": ["10.0.0.100/24"],
          "name": "lan",
          "gateway": {
            "ipv4_gateway": "10.0.0.1"
          }
        },
        {
          "addresses": ["fe80:1::1/64"],
          "name": "ens1"
        },
        {
          "name": "ens2"
        }
      ],
      "vlans": [
        {
          "name": "ens2.100",
          "addresses": ["169.254.0.1/30"],
          "link": "ens2",
          "id": 100
        },
        {
          "name": "ens2.200",
          "addresses": ["169.254.3.1/30"],
          "link": "ens2",
          "id": 200
        }
      ]
    },
    "scion": {
      "ases": [
        {
          "isd_as": "1-ff00:0:1",
          "forwarding_key": "Zm9yd2FyZGluZ19rZXk=",
          "core": true,
          "scion_mtu": 1472,
          "shard_id": 1,
          "control": {
            "address": "10.0.0.100:0",
            "enabled": true
          },
          "cppki": {
            "issuers": [
              {
                "isd_as": "1-ff00:0:100",
                "priority": 0
              },
              {
                "isd_as": "1-ff00:0:200",
                "priority": 1
              }
            ]
          },
          "router": {
            "enabled": true,
            "internal_interface": "10.0.0.100:30100"
          },
          "neighbors": [
            {
              "neighbor_isd_as": "1-ff00:2:1",
              "relationship": "CORE",
              "interfaces": [
                {
                  "address": "[fe80:1::1]:31000",
                  "administrative_state": "UP",
                  "interface_id": 10,
                  "remote": {
                    "address": "[fe80:1::2]:31000",
                    "interface_id": 20
                  },
                  "scion_mtu": 1452
                }
              ]
            },
            {
              "neighbor_isd_as": "1-ff00:1:10",
              "relationship": "CHILD",
              "interfaces": [
                {
                  "address": "169.254.0.1:31000",
                  "administrative_state": "UP",
                  "interface_id": 100,
                  "remote": {
                    "address": "169.254.0.2:31000",
                    "interface_id": 1
                  },
                  "scion_mtu": 1472
                },
                {
                  "address": "169.254.3.1:31000",
                  "administrative_state": "UP",
                  "interface_id": 101,
                  "remote": {
                    "address": "169.254.2.2:31000",
                    "interface_id": 2
                  },
                  "scion_mtu": 1472
                }
              ]
            }
          ]
        }
      ]
    },
    "cluster": {
      "synchronization": {
        "address": "10.0.0.100:40000"
      },
      "peers": [
        {
          "name": "s01.chbrn1.isp1",
          "synchronization": {
            "address": "10.0.0.200:40000"
          }
        }
      ]
    },
    "management": {
      "api": {
        "address": "0.0.0.0:443"
      },
      "hostname": "s01.chzrh1.isp1"
    }
  }
}

Complete Configuration s01.chbrn1.isp1

{
  "config": {
    "interfaces": {
      "ethernets": [
        {
          "addresses": ["10.0.0.200/24"],
          "name": "lan",
          "gateway": {
            "ipv4_gateway": "10.0.0.1"
          }
        },
        {
          "addresses": ["fe80:2::1/64"],
          "name": "ens1"
        },
        {
          "name": "ens2"
        }
      ],
      "vlans": [
        {
          "name": "ens2.100",
          "addresses": ["169.254.1.1/30"],
          "link": "ens2",
          "id": 100
        },
        {
          "name": "ens2.200",
          "addresses": ["169.254.2.1/30"],
          "link": "ens2",
          "id": 200
        }
      ]
    },
    "scion": {
      "ases": [
        {
          "isd_as": "1-ff00:0:1",
          "forwarding_key": "Zm9yd2FyZGluZ19rZXk=",
          "core": true,
          "scion_mtu": 1472,
          "shard_id": 2,
          "control": {
            "address": "10.0.0.200:0",
            "enabled": true
          },
          "cppki": {
            "issuers": [
              {
                "isd_as": "1-ff00:0:100",
                "priority": 0
              },
              {
                "isd_as": "1-ff00:0:200",
                "priority": 1
              }
            ]
          },
          "router": {
            "enabled": true,
            "internal_interface": "10.0.0.200:30100"
          },
          "neighbors": [
            {
              "neighbor_isd_as": "3-ff00:3:1",
              "relationship": "CORE",
              "interfaces": [
                {
                  "address": "[fe80:2::1]:31000",
                  "administrative_state": "UP",
                  "interface_id": 20,
                  "remote": {
                    "address": "[fe80:2::2]:31000",
                    "interface_id": 5
                  },
                  "scion_mtu": 1452
                }
              ]
            },
            {
              "neighbor_isd_as": "1-ff00:1:20",
              "relationship": "CHILD",
              "interfaces": [
                {
                  "address": "169.254.1.1:31000",
                  "administrative_state": "UP",
                  "interface_id": 200,
                  "remote": {
                    "address": "169.254.1.2:31000",
                    "interface_id": 1
                  },
                  "scion_mtu": 1472
                },
                {
                  "address": "169.254.2.1:31000",
                  "administrative_state": "UP",
                  "interface_id": 201,
                  "remote": {
                    "address": "169.254.3.2:31000",
                    "interface_id": 2
                  },
                  "scion_mtu": 1472
                }
              ]
            }
          ]
        }
      ]
    },
    "cluster": {
      "synchronization": {
        "address": "10.0.0.200:40000"
      },
      "peers": [
        {
          "name": "s01.chzrh1.isp1",
          "synchronization": {
            "address": "10.0.0.100:40000"
          }
        }
      ]
    },
    "management": {
      "api": {
        "address": "0.0.0.0:443"
      },
      "hostname": "s01.chbrn1.isp1"
    }
  }
}