93% Less Validator Kernel CPU Time with AF_XDP on an Intel X710

We switched our Firedancer testnet validator from bonded sockets to a standalone Intel X710 using native AF_XDP with zero-copy enabled. Measured validator system CPU fell from 0.35508 to 0.02610 core equivalents: a 92.65% reduction in CPU time spent in the kernel.

The absolute difference was 0.329 core, about 1.1% of our 30-logical-CPU budget. Total validator CPU stayed near 28 cores. Firedancer’s busy-polling design can keep utilization high even when packet handling uses less kernel time; these measurements do not establish why the total stayed unchanged.

A separate 15-minute socket run on the standalone port gave a 92.42% system-CPU reduction against the AF_XDP run. We did not test an otherwise matched AF_XDP copy-mode configuration, so the result belongs to the measured AF_XDP setup; it does not isolate the benefit of zero-copy itself. Both main windows lasted about an hour, using the same machine and binary under ordinary testnet traffic.

The Linux i40e driver and Firedancer build were unmodified. No firmware replacement was needed.

What AF_XDP zero-copy does

AF_XDP gives an application a fast path for sending and receiving network packets. In zero-copy mode, a compatible network card writes incoming packets directly into memory registered for the application. That avoids the extra memory copy used by AF_XDP copy mode. Firedancer can then handle that traffic without the ordinary UDP-socket receive path.

This path can reduce copying, system calls and ordinary networking overhead. Linux still handles parts of the network path. It is different from Firedancer’s native mlx5 implementation.

The runtime proof here was a successful physical AF_XDP bind with flags 0xc (forced zero-copy plus need-wakeup) on the X710, a driver-mode XDP attachment, continued packet processing and independently advancing own votes. Loopback used a separate generic path with flags 8; it does not establish physical zero-copy.

Why this was not just enabled automatically

The answer is specific to Firedancer 26.10.5, commit 6286f99e400c1d5ac6f95b8cf36491555faa3726.

Its default configuration sets the provider, XDP mode, zero-copy and native-bond choices to auto. The pinned auto-selection code lists i40e on Linux 5.19+ for driver-mode XDP, but its separate Zero Copy allowlist contains only mlx5_core on Linux 6.1+. The auto resolver initially sets zero-copy false and enables it only when that feature's checks pass. Consequently, driver-mode XDP support does not imply that this release automatically enables zero-copy on i40e.

Automatic selection is conservative and depends on the exact driver/kernel combination. We explicitly selected driver mode, disabled native bonding for the standalone port and forced zero-copy for this test.

There were practical setup constraints too:

• The existing public interface was an active-backup bond. This release's native XDP bonding supports LACP, so we temporarily used the connected physical port directly. We did not convert the switch or bond to LACP.
• The voting XDP layout needed more hugepage capacity than the socket layout. Contiguous 1 GiB pages were provisioned early in a controlled reboot and retained across backend restarts.
• Queue/RSS/offload preparation and two loopback-only sysctls were required. This was not simply a switch in the Intel driver.
• Redirected validator UDP bypasses normal netfilter processing. The exact address/port selector and GRE behavior were reviewed for this lab. RPC and metrics remained localhost-only, and SSH used independent protected management.

Runbook: the configuration to aim for

The X710 experiment package contains the final TOML examples, measured service excerpts, optional configuration editor, full apply/rollback runbook, collector, summarizer, sanitized measurements and chart. Its machine-readable experiment record lists versions, prerequisites, operation effects and limitations. These links point to the fixed revision used for this article.

Start with a healthy Firedancer testnet validator that is already the sole voter on this host and already uses a dedicated standalone X710 port. These examples describe the final XDP configuration. Moving an identity, converting a bond, preparing the NIC and provisioning hugepages are separate maintenance tasks that must be complete first.

Firedancer TOML

The following is the complete sanitized reference file supplied for this experiment, including the application, resource and network settings we used. It is not an export of every Firedancer default and it is not ready to install unchanged. Omitted options use the pinned release's defaults. Preserve any additional settings in your working configuration.

Use your existing service's configuration path. In the repository runbook, the example path is /etc/firedancer/testnet.toml. Replace the runtime user, storage/key/history paths, interface, public bind address, vote account and trusted snapshot provider with your verified values. 192.0.2.10 is a documentation address. Keep your current identity and newest history in place; changing the backend does not require transferring or replacing either.

# Template: replace operator-specific values before use. Alpenglow TESTNET tuple only.
user = "sol"
name = "testnet-validator"
telemetry = false
[paths]
base = "/var/lib/firedancer/testnet"
identity_key = "/var/lib/firedancer/keys/identity.json"
vote_account = "REPLACE_WITH_PUBLIC_VOTE_ACCOUNT"
vote_history = "/var/lib/firedancer/testnet/vote-history"
[accounts]
max_accounts = 200000000
cache_size_gib = 12
[layout]
affinity = "auto"
enable_block_production = true
enable_snapshot_production = false
net_tile_count = 1
execrp_tile_count = 4
snapdc_tile_count = 1
snapin_tile_count = 1
shred_tile_count = 2
[tiles.gui]
enabled = false
[tiles.rpc]
enabled = true
rpc_listen_address = "127.0.0.1"
rpc_listen_port = 8899
[net]
provider = "xdp"
interface = "ens1f0" # Replace with the verified standalone X710 port.
bind_address = "192.0.2.10" # Documentation address: replace with actual public address.
[hugetlbfs]
mount_path = "/mnt/firedancer"
[snapshots.server]
enabled = false
[tiles.metric]
prometheus_listen_address = "127.0.0.1"
[log]
path = "/var/lib/firedancer/testnet/logs/firedancer.log"
[development]
sandbox = true
core_dump = "disabled"
[consensus]
expected_genesis_hash = "4uhcVJyU9pJkvQyS88uRDiswHXSCkY3zQawwpjk2NsNY"
alpenglow = true
[snapshots.sources]
servers = []
[snapshots.sources.gossip]
allow_any = false
allow_list = ["REPLACE_WITH_TRUSTED_SNAPSHOT_PROVIDER"]

[net.xdp]
xdp_mode = "drv"
native_bond = "false"
xdp_zero_copy = "true"
poll_mode = "softirq"
rss_queue_mode = "simple"

# listen_gre is omitted: pinned default is the string "auto".
# In rss_queue_mode="simple" this field is ignored; omission does not disable GRE.

The backend-specific settings are provider = "xdp", xdp_mode = "drv", native_bond = "false" and xdp_zero_copy = "true". The experiment also used poll_mode = "softirq" and rss_queue_mode = "simple". These quoted values are strings in Firedancer 26.10.5.

listen_gre is deliberately omitted. Its pinned default is "auto", and simple RSS mode ignores it. Adding listen_gre = "false" would not disable the stock selector's GRE branch. Review the loaded selector and intended traffic: redirected UDP bypasses ordinary netfilter processing. Downstream receive validation rejects unconfigured GRE remotes. DoubleZero was not validated.

systemd: the resource drop-in

Keep the existing, working service unit, including its binary/config paths, startup-history checks and security settings. The repository now includes a sanitized record of the measured effective service and its dependencies. It distinguishes root bootstrap from Firedancer’s later runtime privilege drop. Those site-dependent excerpts are records, not ready-to-install service units; the inherited base unit’s earlier authorship is not asserted.

The block below contains the tested resource settings, not a complete service file. An operator-chosen drop-in location for a service named firedancer.service is /etc/systemd/system/firedancer.service.d/60-x710-resources.conf. That filename is an example, not a claim about the lab's original filename. Check existing drop-ins for conflicting values.

[Service]
LimitMEMLOCK=103079215104
MemoryMax=111669149696
CPUAffinity=1-15 17-31
Restart=no

LimitMEMLOCK is 96 GiB and MemoryMax is 104 GiB. The CPU list gives this Ryzen 7950X a fixed 30-logical-CPU budget. These numbers belong to our 128 GB host and tested layout. Recalculate them for another machine; do not copy that CPU list without checking the host topology. Restart=no was the experiment's explicit recovery policy, not a universal production recommendation. Preserve a reviewed recovery path and pause external startup automation during the maintenance window.

The installed binary used in the experiment was /opt/firedancer/v26.10.5/bin/firedancer. Confirm that your existing service runs your verified pinned binary and reads your intended configuration. The repository runbook's clean-build recipe is reconstructed and untested; it does not assert a build-output executable path.

Hugepages and loopback settings

The final resource pool used for both backends was:

The configuration preview required 78 one-GiB plus 2305 two-MiB pages for sockets, versus 79 plus 2222 for XDP. We reserved the maximum of both sets and retained it across restarts. Use firedancer mem --json --testnet --config CONFIG on your complete candidate to determine its actual footprint. Block production changes the requirements.

Under the configured hugetlbfs mount root, the tested mounts had these effective options. This is a record of mount settings, not an fstab file or mount command:

.gigantic: rw,relatime,pagesize=1024M,min_size=84825604096
.huge:     rw,relatime,pagesize=2M,min_size=4833935360

The mount roots were owned by the runtime account with mode 0700. The page pools must already exist; min_size reserves capacity and does not create contiguous physical pages. Our one-GiB pages needed early provisioning in a separately planned reboot.

While XDP is active, the required loopback values in this setup are:

net.ipv4.conf.lo.accept_local = 1
net.ipv4.conf.lo.route_localnet = 1

These are runtime values, not a proposed persistent sysctl file. The experiment recorded their previous values, set them for XDP, and restored them on return to sockets. Leave the physical and management interfaces' corresponding settings unchanged; both remained zero in our test. The repository runbook contains the reversible commands.

Collector prerequisite: confirm all validator tiles are in the selected service cgroup before using the portable collector. It covered the measured X710 layout, but it will undercount a layout that moves tiles into another cgroup.

Apply and verify

• Before stopping: confirm sole-voter ownership, independent management, saved configuration, current history, sufficient page pools and a recovery path. Complete the NIC's queue/RSS/offload preparation separately with saved state and a reviewed restore procedure. Our one-net-tile setup used one combined queue, queue-zero RSS, UDP4 four-tuple hashing and 512/512 RX/TX rings.
• Review the candidate: compare it with the working standalone/socket configuration. Preserve identity, vote account, history, addresses, routes, trusted snapshot policy and protected endpoints. Preview memory with the pinned binary. If adding a resource drop-in, reload systemd and inspect the effective service settings before restarting; do not replace the whole unit.
• Stop and apply: pause external startup automation, stop the service and verify its cgroup is empty. Apply the reviewed backend config and temporary loopback values. Require the pinned binary's configure check hugetlbfs sysctl ethtool-channels ethtool-offloads ethtool-loopback checks to pass. The repository runbook supplies the exact arguments and config-edit commands.
• Start and prove: start through the existing service/history checks. Confirm the physical forced-zero-copy bind (0xc in this run), driver-mode XDP attachment, intended selector and advancing packet traffic. Then verify identity/cluster, catch-up and independently advancing own votes. Loopback's generic bind with flags 8 is not evidence of physical zero-copy.
• Measure: collect the same CPU, RAM, NIC, disk and voting metrics at the same cadence, separately from startup. Do not send synthetic stress to cluster peers. Record Linux drops, driver errors and application counters separately.
• Return to sockets: stop and verify an empty cgroup; restore the saved standalone/socket config and original loopback values while retaining the newest history. Confirm XDP attachments are gone before restart. Investigate any remaining attachment rather than blindly detaching it. Verify own voting, catch-up, endpoints and management before resuming automation. Restoring a bond or NIC preparation uses its separate site procedure.

The commands in the repository runbook automate only the scoped configuration/sysctl swap. They do not migrate hosts, restore old vote history or provision a whole validator.

Measured results

All times below are UTC on 31 October 2026. These are sequential observations of ordinary testnet traffic on the same machine.

The whole-system samples inside those guards span 3,590.064 seconds for A, 900.015 for B and 3,600.064 for C. Rates use those actual sampled durations; the ten-second sampling boundaries explain the small differences from guard duration.

The reductions are calculated from system CPU rates, not unequal raw totals: 100 × (1 − 0.02610255 / 0.35507991) = 92.65%. The topology-control comparison is 92.42%. Average combined NIC packet rates differ by about 0.04% between A and C, although packet mix and transaction/leader work were not controlled.

RX/TX error counters did not increase. Guarded driver and network application drop/stall counters did not increase during the stable runs. Linux RX drops rose at roughly 0.5/s in both standalone phases; their cause is unverified. This is not a losslessness claim. The shorter system-sampling interiors contain 450 and 1,800 drops for B/C, while the complete guard windows contain 453 and 1,802.

Disk: keep the startup burst in view

C included a 216 MiB/s write burst about 12 seconds after measurement began, even though replay was already caught up. Its full-window write-wait average is therefore much higher. The second calculation excludes the first 300 seconds of every phase, a fixed sensitivity check chosen before C finished. Both results are retained:

The later disk rates and waits are similar. There is no convincing storage-performance improvement here. Full read/write IOPS, average queue depth, busy-time counters, memory pressure and per-core CPU are in the sanitized results.

The chart label “Validator kernel CPU” means validator system CPU in core equivalents, the metric used in the headline. The disk panel retains the early write burst.

The numeric chart data contains the same sampled observations. No memory/OOM events were observed during these windows.

Tested configuration

After measurement, the lab was restored to bond/socket and verified caught up with advancing own votes. NIC settings, loopback values, routes, protected endpoints and runtime isolation were checked before the experiment timer was retired. Any later host handover is outside these measured windows.

Limits of this result

This establishes a working stock X710 AF_XDP zero-copy path on the documented configuration, with lower measured validator system CPU and healthy sampled replay/voting. It does not establish maximum throughput, latency percentiles, energy savings, losslessness, long-term reliability, mainnet suitability or parity with native mlx5. Full-window leader-production verification was unavailable. A partial baseline interval showed four assigned and four produced slots, which is not a full-hour skip-rate result.

The runs were sequential. A-to-C changes topology and backend; B-to-C holds the standalone topology but still changes queue/offload/backend behavior, and B is shorter. Average packet rates were close, but packet mix, transaction load and leader work were not controlled. A matched AF_XDP copy-mode arm, repeated/reversed ordering and a controlled workload would help separate the causes.

Validator system CPU is cgroup system CPU seconds divided by elapsed wall time. Host system/softirq percentages use a different denominator and must not be added to it. Shared hugepages and changing cache state complicate RAM comparisons; process RSS can double-count shared mappings. Disk waits are averages, not percentiles, and NVMe busy time is not a general saturation measure. Cgroup io.stat was unavailable, so the write burst cannot be assigned precisely to a service. Thread/tile-regime samples lacked a matched full-hour baseline and do not establish useful work, polling efficiency or cycles per packet.

Evidence and implementation references

The repository data contains the portable collector, resource summarizer, sanitized aggregate data, time series and reference configurations. Private evidence retains complete phase records, runtime bind/selector proof, signed-history verification metadata and restoration checks. Signing keys, private addresses and validator identifiers are excluded from the public package.

Implementation references are pinned: AF_XDP socket binding, XDP tile setup, and receive validation. Runtime evidence establishes the path actually exercised.

Browse the experiment package or download this exact repository revision (ZIP). The longer editing and rollback scripts are retained in the full runbook.