TL;DR
The SCION Association built a custom workstation centered on an LGA4677 platform with a 12-core Intel Xeon and three Mellanox/NVIDIA BlueField-2 dual-25G smart NICs to develop an AF_XDP underlay for the open-source SCION border router. The multi-week build cost about CHF 3,741.34 and targets much higher data-plane throughput than the current SCION OSS border router achieves.
What happened
Faced with the need to develop and test a zero-copy AF_XDP underlay for the SCION open-source border router, the team designed and assembled a bespoke workstation. The build uses an LGA4677 socket populated with a 12-core Intel Xeon to supply 64 PCIe Gen5 lanes, and three Mellanox/NVIDIA BlueField-2 DPUs, each offering dual 25 Gbit/s ports. The goal was to reach 25 Gbit/s on a single border-router thread in bench-top topology while keeping acoustic noise low enough for office use. The project was chosen over buying used servers because off-the-shelf options that met PCIe, NIC, and noise requirements were hard to find within the budget. The work spanned multiple weeks; the documented parts and assembly workflow aim to enable development of a higher-performance AF_XDP underlay to improve SCION OSS data-plane throughput.
Why it matters
- SCION OSS's current software-based border router throughput is substantially below what is needed for high-bandwidth use cases, motivating kernel-bypass approaches.
- AF_XDP provides a kernel-supported zero-copy path that can deliver raw frames to user space with much lower overhead than the full Linux networking stack.
- Testing zero-copy AF_XDP requires bare-metal NICs and drivers that expose necessary capabilities; typical cloud VMs do not suffice.
- Improving SCION border-router throughput is relevant for deployments requiring deterministic performance and stronger path controls, such as critical infrastructure.
Key facts
- SCION is an IETF draft-stage inter-AS routing architecture intended as an alternative or complement to BGP.
- Since 2024 Switzerland's banking infrastructure runs on SSFN, a SCION-powered network, using a commercial implementation from Anapaya Systems AG.
- The SCION OSS border router currently reaches about 400k–500k packets per second, roughly 5–6 Gbit/s at a 1500-byte MTU.
- The new workstation is built around an LGA4677 socket with a 12-core Intel Xeon CPU and 64 PCIe Gen5 lanes.
- Three Mellanox/NVIDIA BlueField-2 DPUs with dual 25 Gbit/s ports were acquired to provide the NIC hardware for AF_XDP zero-copy testing.
- The total cost of materials for the build was approximately CHF 3,741.34 (~$4,700 USD).
- AF_XDP was chosen over DPDK primarily because the SCION OSS border router is written in Go and maintainability and operational simplicity were prioritized.
- Common kernel-bypass alternatives include DPDK, AF_XDP, and VPP, each with different trade-offs around performance and manageability.
- Typical VM offerings generally do not expose the capabilities required for AF_XDP XDP_ZEROCOPY mode, necessitating bare-metal hardware for proper testing.
What to watch next
- Whether the AF_XDP underlay achieves the target of 25 Gbit/s single-thread performance in published benchmarks — not confirmed in the source
- If performance gains from the AF_XDP underlay are merged upstream into the SCION OSS repository and adopted by the project — not confirmed in the source
- How the workstation's noise profile performs in sustained tests in the office environment — not confirmed in the source
Quick glossary
- AF_XDP: A Linux kernel mechanism that enables high-performance packet I/O by establishing a shared memory region (UMEM) and rings between NIC drivers and user space for low-overhead packet exchange.
- eBPF / XDP: A programmable in-kernel execution environment (eBPF) used by XDP to run small bytecode programs on incoming packets, enabling actions like redirecting frames to AF_XDP sockets.
- DPDK: A user-space framework designed for high-speed packet processing that bypasses the kernel networking stack, often requiring exclusive control of NIC hardware.
- DPU / Smart NIC: A network interface card that includes on-board processing capabilities to offload networking tasks, sometimes used to accelerate packet processing or provide isolation.
- PCIe Gen5: A version of the PCI Express interface standard providing increased per-lane bandwidth compared with earlier generations, relevant for high-throughput NIC connectivity.
Reader FAQ
How much did the workstation cost?
The materials cost about CHF 3,741.34 (roughly $4,700 USD).
Why was AF_XDP chosen instead of DPDK?
AF_XDP was preferred because the SCION OSS border router is written in Go and the team prioritized maintainability, usability, and operational simplicity.
Did the build achieve 25 Gbit/s per thread?
not confirmed in the source
Is a full parts list provided in the article?
The source states there is a complete list of components at the end, but that list is not included in the provided excerpt — not confirmed in the source.
SCION 25 Gbit/s Workstation This is an LGA4677 socket and it's about to be fitted with a 12-core Intel Xeon CPU to power the 64 PCIe Gen5 lanes for 3x…
Sources
- Building a 25 Gbit/s workstation for the SCION Association
- SCIONLAB: A Next-Generation Internet Testbed
- Building a SCION enabled Home Router – Claude's Blog
- SCinet at Scale: Building the Network Infrastructure …
Related posts
- LLVM: The bad parts — maintainers outline pain points and fixes
- LLVM: The Bad Parts — Lead Maintainer Lists Key Design and Process Issues
- Microsoft abruptly retires long-lived Microsoft Deployment Toolkit (MDT)