Anapaya CORE¶
The Anapaya CORE implements a SCION core router and is typically deployed by ISPs to provide connectivity to other SCION ISPs and downstream customers.
In the following, we describe a typical small deployment scenario, where an ISP (ISP 1) operates two CORE appliances, both connected to a peering SCION ISP. Furthermore, the ISP has two downstream customers that are connected to each of the two CORE appliances.
This redundant access design is a best practice because it increases the redundancy of the customer access and allows the ISP to perform maintenance on one of the CORE appliances without causing a service interruption.
Topology¶
Our target topology contains the following elements:
ISP 1 deploys two CORE appliances,
s01.chzrh1.isp1
ands01.chbrn1.isp1
, that are connected internally via the internal network10.0.0.0/24
.ISP 1 is peering with ISP 2 (ISD-AS 1-ff00:2:1) at
s01.chzrh1.isp1
. The local SCION interface ID is10
and the remote SCION interface ID is20
. The underlay IPv6 network used to establish the link isfe80:1::/64
, wherefe80:1::1/64
is the local IPv6 address andfe80:1::2/64
is the remote IPv6 address.ISP 1 is peering with ISP 3 (ISD-AS 3-ff00:3:1) at
s01.chbrn1.isp1
. The local SCION interface ID is20
and the remote SCION interface ID is5
. The underlay IPv6 network used to establish the link isfe80:2::/64
, wherefe80:2::1/64
is the local IPv6 address andfe80:2::/64
is the remote IPv6 address.- ISP 1 is connecting Customer 1 (ISD-AS 1-ff00:1:10) redundantly:
s01.chzrh1.isp1
: The local SCION interface ID is100
and the remote SCION interface ID is1
. The underlay IPv4 network used to establish the link is169.254.0.0/30
, where.1
is the local IPv4 address and.2
is the remote IPv4 address.s01.chbrn1.isp1
: The local SCION interface ID is201
and theremote SCION interface ID is
2
. The underlay IPv4 network used to establish the link is169.254.2.0/30
, where.1
is the local IPv4 address and.2
is the remote IPv4 address.
- ISP 1 is connecting Customer 2 (ISD-AS 1-ff00:1:20) redundantly:
s01.chzrh1.isp1
: The local SCION interface ID is101
and the remote SCION interface ID is2
. The underlay IPv4 network used to establish the link is169.254.3.0/30
, where.1
is the local IPv4 address and.2
is the remote IPv4 address.s01.chbrn1.isp1
: The local SCION interface ID is200
and theremote SCION interface ID is
1
. The underlay IPv4 network used to establish the link is169.254.1.0/30
, where.1
is the local IPv4 address and.2
is the remote IPv4 address.
We will now configure the CORE appliances step-by-step to implement the above scenario.
Network Interface Configuration¶
First, we will look at the configuration for the network interfaces. In the
setup, there are three physical network interfaces - one for the internal
network lan
, one for the peering link to the neighboring SCION ISP
ens1
, and one for the links to the downstream customer ens2
. The links
to the downstream customers are configured as VLANs on the ens2
interface.
Note
It is not required that each link is on a separate physical interface. It is also possible to have multiple links on the same physical interface in the same or different VLANs, or multiple virtual functions on top of a single physical interface if the underlying hardware supports it.
In this example, we also configure the lan
interface to be the default
route.
Network Interface Configuration
s01.chzrh1.isp1
"interfaces": { "ethernets": [ { "addresses": ["10.0.0.100/24"], "name": "lan", "gateway": { "ipv4_gateway": "10.0.0.1" } }, { "addresses": ["fe80:1::1/64"], "name": "ens1" }, { "name": "ens2" } ], "vlans": [ { "name": "ens2.100", "addresses": ["169.254.0.1/30"], "link": "ens2", "id": 100 }, { "name": "ens2.200", "addresses": ["169.254.2.1/30"], "link": "ens2", "id": 200 } ] },s01.chbrn1.isp1
"interfaces": { "ethernets": [ { "addresses": ["10.0.0.200/24"], "name": "lan", "gateway": { "ipv4_gateway": "10.0.0.1" } }, { "addresses": ["fe80:2::1/64"], "name": "ens1" }, { "name": "ens2" } ], "vlans": [ { "name": "ens2.100", "addresses": ["169.254.1.1/30"], "link": "ens2", "id": 100 }, { "name": "ens2.200", "addresses": ["169.254.3.1/30"], "link": "ens2", "id": 200 } ] },
Please refer to Network Interfaces for the full documentation on network interface configuration.
SCION Configuration¶
Next, we will look at the SCION configuration. The SCION section contains the configuration of the SCION protocol and AS and can be grouped into the general AS, the control plane, and the data plane configuration.
General AS Configuration¶
Each SCION AS has several general AS configuration options such as the ISD-AS identifier, the AS forwarding secret key, or a human-readable description of the AS. For the full list of the general AS configuration options, please refer to General AS Configuration.
In this example, we configure the following fields:
isd_as
The ISD-AS identifier of the AS. In this example, the ISD-AS identifier is
1-ff00:0:1
. This must be the same for all CORE appliances.forwarding_key
The base64 encoded secret key used to verify the integrity and authenticity of the SCION packet header during forwarding on the data plane. This must be the same for all CORE appliances.
Note
In the example, we use a dummy key. In production, the secret key should be a strong password that is at least 12 characters long and contain uppercase letters, lowercase letters, numbers, and special characters.
core
The
core
flag indicates whether the AS is a core AS. In this example, ISP 1 is a core AS of ISD 1 and therefore sets this flag totrue
.scion_mtu
The SCION MTU is the maximum size of a SCION packet supported in the internal network. Since we are using an IPv4 underlay network, we set this to 1472 bytes. This assumes an Ethernet frame size of 1500 bytes, minus 20 bytes for the IPv4 header and 8 bytes for the UDP header that is used to transport SCION packets. This must be the same for all CORE appliances.
shard_id
CORE appliances are typically deployed in a sharded manner, where each shard can function independently. The
shard_id
is used to identify the shard. Theshard_id
must be unique for each CORE appliance in a given AS, as each constitutes its own shard.
General AS Configuration
s01.chzrh1.isp1
"scion": { "ases": [ { "isd_as": "1-ff00:0:1", "forwarding_key": "Zm9yd2FyZGluZ19rZXk=", "core": true, "scion_mtu": 1472, "shard_id": 1,s01.chbrn1.isp1
"scion": { "ases": [ { "isd_as": "1-ff00:0:1", "forwarding_key": "Zm9yd2FyZGluZ19rZXk=", "core": true, "scion_mtu": 1472, "shard_id": 2,
Control Plane Configuration¶
The SCION control plane configuration consists of the control
and cppki
sections that configure the network endpoint of the control service and the AS
certificate renewal. The full documentation on the SCION control plane
configuration can be found in Control Plane Configuration.
The network endpoint of the control service is typically configured to listen on
the internal interface. The port can be set to 0
to let the appliance
automatically allocate a free port.
The cppki
section configures the SCION AS certificate issuers. These are
required to periodically renew the AS certificate on the CORE appliances. In
this example, we assume that there are two certificate issuers for ISD 1 at
1-ff00:0:100
and 1-ff00:0:200
. We want to configure one as the primary
issuer and the other as the fallback in case the primary issuer is not
reachable.
SCION Control Plane Configuration
s01.chzrh1.isp1
"control": { "address": "10.0.0.100:0", "enabled": true }, "cppki": { "issuers": [ { "isd_as": "1-ff00:0:100", "priority": 0 }, { "isd_as": "1-ff00:0:200", "priority": 1 } ] },s01.chbrn1.isp1
"control": { "address": "10.0.0.200:0", "enabled": true }, "cppki": { "issuers": [ { "isd_as": "1-ff00:0:100", "priority": 0 }, { "isd_as": "1-ff00:0:200", "priority": 1 } ] },
SCION Data Plane Configuration¶
The SCION data plane configuration consists of the router
and the
neighbors
sections. The router
section contains the configuration of the
internal SCION interface internal_interface
, while the neighbors
section defines all the SCION links to other SCION ASes, i.e., the external
SCION interfaces. The full documentation on the SCION data plane configuration
can be found in Data Plane Configuration.
The internal SCION interface is where the appliance receives SCION packets from
AS internal hosts for forwarding. It is defined as a UDP/IP network endpoint on
which the SCION packets are received from the internal network. The endpoint
needs to be specified as a string in the format <ip>:<port>
. If the port is
set to 0 it will be automatically assigned.
In our case, we configure the internal SCION interface to listen on the lan
interface.
Each entry in the neighbors
section contains a remote SCION AS to which one
or multiple links are established. Each of the CORE appliances has two
neighbors - CORE
link to ISP 2/3 and a CHILD
link to Customer 1/2. For
each neighbor, we need to specify the local and remote interface network address
endpoints, the link type (CORE
or CHILD
), the SCION interface ID, and
the SCION MTU supported on this link. The SCION interface ID must be unique for
each SCION interface in an AS. We set the SCION MTU to 1472 bytes for the links
to the customers that use an IPv4 underlay network and 1452 bytes for the links
to the other ISPs that use an IPv6 underlay network.
SCION Data Plane Configuration
s01.chzrh1.isp1
"router": { "enabled": true, "internal_interface": "10.0.0.100:30100" }, "neighbors": [ { "neighbor_isd_as": "1-ff00:2:1", "relationship": "CORE", "interfaces": [ { "address": "[fe80:1::1]:31000", "administrative_state": "UP", "interface_id": 10, "remote": { "address": "[fe80:1::2]:31000", "interface_id": 20 }, "scion_mtu": 1452 } ] }, { "neighbor_isd_as": "1-ff00:1:10", "relationship": "CHILD", "interfaces": [ { "address": "169.254.0.1:31000", "administrative_state": "UP", "interface_id": 100, "remote": { "address": "169.254.0.2:31000", "interface_id": 1 }, "scion_mtu": 1472 }, { "address": "169.254.2.1:31000", "administrative_state": "UP", "interface_id": 101, "remote": { "address": "169.254.2.2:31000", "interface_id": 2 }, "scion_mtu": 1472 } ] } ]s01.chbrn1.isp1
"router": { "enabled": true, "internal_interface": "10.0.0.200:30100" }, "neighbors": [ { "neighbor_isd_as": "3-ff00:3:1", "relationship": "CORE", "interfaces": [ { "address": "[fe80:2::1]:31000", "administrative_state": "UP", "interface_id": 20, "remote": { "address": "[fe80:2::2]:31000", "interface_id": 5 }, "scion_mtu": 1452 } ] }, { "neighbor_isd_as": "1-ff00:1:20", "relationship": "CHILD", "interfaces": [ { "address": "169.254.1.1:31000", "administrative_state": "UP", "interface_id": 200, "remote": { "address": "169.254.1.2:31000", "interface_id": 1 }, "scion_mtu": 1472 }, { "address": "169.254.3.1:31000", "administrative_state": "UP", "interface_id": 201, "remote": { "address": "169.254.3.2:31000", "interface_id": 2 }, "scion_mtu": 1472 } ] } ]
Cluster Configuration¶
The CORE appliances are deployed in a sharded manner as part of a cluster. To still work as a single unit, the appliances exchange topology and SCION beacon and path information amongst each other.
The cluster configuration includes the local cluster endpoint and the list of peers that are part of the cluster. For CORE deployments we recommend to use automatic topology synchronization (see Cluster for more details).
Cluster Configuration
s01.chzrh1.isp1
"cluster": { "synchronization": { "address": "10.0.0.100:40000" }, "peers": [ { "name": "s01.chbrn1.isp1", "synchronization": { "address": "10.0.0.200:40000" } } ] },s01.chbrn1.isp1
"cluster": { "synchronization": { "address": "10.0.0.200:40000" }, "peers": [ { "name": "s01.chzrh1.isp1", "synchronization": { "address": "10.0.0.100:40000" } } ] },
Complete Configuration¶
For completeness, we provide the full configuration of each CORE appliance:
Complete Configuration s01.chzrh1.isp1
{ "config": { "interfaces": { "ethernets": [ { "addresses": ["10.0.0.100/24"], "name": "lan", "gateway": { "ipv4_gateway": "10.0.0.1" } }, { "addresses": ["fe80:1::1/64"], "name": "ens1" }, { "name": "ens2" } ], "vlans": [ { "name": "ens2.100", "addresses": ["169.254.0.1/30"], "link": "ens2", "id": 100 }, { "name": "ens2.200", "addresses": ["169.254.2.1/30"], "link": "ens2", "id": 200 } ] }, "scion": { "ases": [ { "isd_as": "1-ff00:0:1", "forwarding_key": "Zm9yd2FyZGluZ19rZXk=", "core": true, "scion_mtu": 1472, "shard_id": 1, "control": { "address": "10.0.0.100:0", "enabled": true }, "cppki": { "issuers": [ { "isd_as": "1-ff00:0:100", "priority": 0 }, { "isd_as": "1-ff00:0:200", "priority": 1 } ] }, "router": { "enabled": true, "internal_interface": "10.0.0.100:30100" }, "neighbors": [ { "neighbor_isd_as": "1-ff00:2:1", "relationship": "CORE", "interfaces": [ { "address": "[fe80:1::1]:31000", "administrative_state": "UP", "interface_id": 10, "remote": { "address": "[fe80:1::2]:31000", "interface_id": 20 }, "scion_mtu": 1452 } ] }, { "neighbor_isd_as": "1-ff00:1:10", "relationship": "CHILD", "interfaces": [ { "address": "169.254.0.1:31000", "administrative_state": "UP", "interface_id": 100, "remote": { "address": "169.254.0.2:31000", "interface_id": 1 }, "scion_mtu": 1472 }, { "address": "169.254.2.1:31000", "administrative_state": "UP", "interface_id": 101, "remote": { "address": "169.254.2.2:31000", "interface_id": 2 }, "scion_mtu": 1472 } ] } ] } ] }, "cluster": { "synchronization": { "address": "10.0.0.100:40000" }, "peers": [ { "name": "s01.chbrn1.isp1", "synchronization": { "address": "10.0.0.200:40000" } } ] }, "management": { "api": { "address": "0.0.0.0:443" }, "hostname": "s01.chzrh1.isp1" } } }
Complete Configuration s01.chbrn1.isp1
{ "config": { "interfaces": { "ethernets": [ { "addresses": ["10.0.0.200/24"], "name": "lan", "gateway": { "ipv4_gateway": "10.0.0.1" } }, { "addresses": ["fe80:2::1/64"], "name": "ens1" }, { "name": "ens2" } ], "vlans": [ { "name": "ens2.100", "addresses": ["169.254.1.1/30"], "link": "ens2", "id": 100 }, { "name": "ens2.200", "addresses": ["169.254.3.1/30"], "link": "ens2", "id": 200 } ] }, "scion": { "ases": [ { "isd_as": "1-ff00:0:1", "forwarding_key": "Zm9yd2FyZGluZ19rZXk=", "core": true, "scion_mtu": 1472, "shard_id": 2, "control": { "address": "10.0.0.200:0", "enabled": true }, "cppki": { "issuers": [ { "isd_as": "1-ff00:0:100", "priority": 0 }, { "isd_as": "1-ff00:0:200", "priority": 1 } ] }, "router": { "enabled": true, "internal_interface": "10.0.0.200:30100" }, "neighbors": [ { "neighbor_isd_as": "3-ff00:3:1", "relationship": "CORE", "interfaces": [ { "address": "[fe80:2::1]:31000", "administrative_state": "UP", "interface_id": 20, "remote": { "address": "[fe80:2::2]:31000", "interface_id": 5 }, "scion_mtu": 1452 } ] }, { "neighbor_isd_as": "1-ff00:1:20", "relationship": "CHILD", "interfaces": [ { "address": "169.254.1.1:31000", "administrative_state": "UP", "interface_id": 200, "remote": { "address": "169.254.1.2:31000", "interface_id": 1 }, "scion_mtu": 1472 }, { "address": "169.254.3.1:31000", "administrative_state": "UP", "interface_id": 201, "remote": { "address": "169.254.3.2:31000", "interface_id": 2 }, "scion_mtu": 1472 } ] } ] } ] }, "cluster": { "synchronization": { "address": "10.0.0.200:40000" }, "peers": [ { "name": "s01.chzrh1.isp1", "synchronization": { "address": "10.0.0.100:40000" } } ] }, "management": { "api": { "address": "0.0.0.0:443" }, "hostname": "s01.chbrn1.isp1" } } }