Broadcom P4 zero-copy: a voting Firedancer validator using one fewer CPU core

Our Broadcom BCM57416 validator completed a full hour of voting on Solana testnet with AF_XDP zero-copy. It stayed caught up in all 361 observations, its votes advanced throughout the hour, and it remained the voting node at the end of that measurement.

The complete configuration used 1.052 fewer CPU cores on average: 37.021 down to 35.969. We also changed Firedancer from two net tiles to one, so that reduction belongs to the configuration as a whole. It is not proof that zero-copy alone saved a core.

That is the useful milestone: an experimental P4 driver and a small Firedancer initialization change ran a caught-up voting validator, with lower observed CPU use and slightly more network traffic.

What made it work

The test server used an AMD EPYC 9254 with 48 logical CPUs and a Broadcom BCM57416 (14e4:16d8). Firmware stayed at 220.0.59.0, package 220.0.83.0; the kernel was Ubuntu 6.8.0-138-generic.

Two code changes were required:

• Driver: an experimental derivative of Broadcom 239.1.170.0, reporting 1.10.3-p4xsk-239.1.170.0, to support this P4 AF_XDP path.
• Firedancer: an opt-in initialization change on v26.10.5, base commit 6286f99e400c1d5ac6f95b8cf36491555faa3726. It supplies 1,024 receive buffers before binding the physical AF_XDP socket, verifies actual zero-copy, and keeps physical receive, transmit and loopback buffer ownership separate. It does not publish those physical receive buffers a second time. Loopback remains in COPY mode.

The measured hour used compile-time EXTRAS=p4-prebind. A later same-base candidate uses the default-off runtime option net.xdp.experimental_prebind_fill = true. The performance numbers below belong to the earlier binary and driver, not that later correction. There is no implemented bnxt_zero_copy=true TOML setting.

The repository package contains the source patches, licenses, build instructions, verification scripts and test records. This is experimental software with a manual deployment procedure. Read the recovery limits below before loading it; it is not a turnkey production installer.

The configuration used for the measured hour

The data port was standalone, outside the original active-backup bond. The measured network configuration was:

[net]
provider = "xdp"
interface = "SITE_P4_INTERFACE"

[net.xdp]
xdp_mode = "drv"
xdp_zero_copy = "true"
poll_mode = "softirq"
rss_queue_mode = "dedicated"
native_bond = "false"
listen_gre = "false"
xdp_rx_queue_size = 1024
xdp_tx_queue_size = 1024

The physical socket used queue 0. We configured 48 combined channels, with ordinary RSS traffic on queues 1–47. MTU was 1500. Hardware RX/TX rings were 511; those are distinct from the 1,024-entry software queues above.

The NIC setup to reproduce is:

The voting layout was:

[layout]
mode = "performance"
affinity = "auto"
blocklist_cores = "0h"
shred_tile_count = 4
net_tile_count = 1
enable_block_production = true
enable_snapshot_production = false

[accounts]
max_accounts = 200_000_000
cache_size_gib = 12

[snapshots.sources.gossip]
allow_any = false
allow_list = ["SITE_TRUSTED_SNAPSHOT_IDENTITY"]

The full TOML example includes localhost-only RPC and metrics, data paths and the remaining settings. Replace every site placeholder and reconcile the historical cluster checkpoints. Snapshot production is disabled; downloading and loading snapshots remains enabled.

This voting layout required 87 × 1 GiB plus 2,747 × 2 MiB hugepages for 52 startup tiles. Fixed tiles used CPUs 1–23, 25–28 and 30–40 on this server. Calculate the footprint from your exact binary and final configuration rather than copying the earlier unstaked allocation:

/SITE/bin/firedancer-p4-prebind mem --json --config /SITE/config/voting.toml
sudo /SITE/bin/firedancer-p4-prebind configure check all --config /SITE/config/voting.toml

These are checks, not a replacement for network, driver and resource preparation. Create the configured vote-history directory even for an unstaked trial; its absence caused our first startup failure.

The service used:

[Service]
Restart=no
KillMode=control-group
KillSignal=SIGTERM
SendSIGKILL=no
TimeoutStopSec=120
LimitNOFILE=1048576
LimitMEMLOCK=104G
MemoryMax=112G
UMask=0077

Fixed tiles move into a separate cpuset cgroup, which also needs the 112 GiB memory limit. Monitoring only the systemd service misses those tiles. The complete service record includes required preparation and voting-authority gates; it is documentary, and its site-specific helpers must be supplied and reviewed. Firedancer’s normal privilege drop and sandbox remain in place.

For reproduction, follow the voting runbook: build and verify the pinned sources, prepare an isolated standalone port and independent management, establish the resource layout, then perform a controlled single-voter handover. Stop and fence the old voter before starting the new one, and transfer its newest verified signed Alpenglow history. Never start both with the same identity.

What the hour showed

P4 downloaded fresh snapshots from an explicitly trusted source and caught up before scoring. We measured 16:48:04–17:48:04 UTC on 1 November 2026, excluding startup and snapshot restoration. The collector covered all validator tiles across both cgroups, per-core CPU, RAM, network, disk and voting/replay health.

The previous same-server baseline used stock Ubuntu bnxt_en, kernel 6.8.0-117, generic XDP COPY on an active-backup bond, and two net tiles.

Validator system CPU fell 21.29%, an absolute saving of 0.01813 cores. Host softirq CPU fell about 54.4%, or 0.05141 cores. These are different accounting views; do not add them together or confuse either with the 1.052-core total change.

Firedancer busy-polls, so useful work becoming cheaper need not produce an equivalent fall in total CPU utilization. Here, removing one net tile is a particularly important part of the total-CPU comparison.

The driver recorded 83,776,284 successful XSK receives and 90,396,814 TX submissions and completions each, with no recorded active-port error/drop-counter increases. Replay stayed caught up throughout; independent finalized-head versus local-root lag was two slots at the median and six at the maximum.

Disk writes and mean write latency increased. We did not see a storage improvement. Cgroup memory also fell sharply, from 90.717 to 28.663 GiB, but most of that difference was file/cache memory: anonymous memory remained approximately 1.1 GiB. The results and data preserve these less flattering or easily misread measurements alongside the CPU result.

How this differs from X710’s 93% result

The earlier Intel X710 run measured a 92.65% reduction in validator system CPU, from 0.3551 to 0.02610 cores, comparing bonded sockets with standalone AF_XDP zero-copy. It used a stock driver and unmodified Firedancer; a standalone/socket control gave a similar result.

The Broadcom baseline already used generic XDP COPY and started with much less system CPU overhead. It also ran on different hardware. The two percentages describe different comparisons; they do not rank the NICs.

What restart testing caught afterward

Later starts failed AF_XDP bind with ENOBUFS, including with the earlier known-good build. Tracing caught the RX producer at 525 on a 511-entry ring. The driver was checking a moving producer counter after packet processing had started, so a normal advance could look like an incomplete initial allocation.

The correction records whether the initial RX allocation completed before packet processing can move that counter. It checks that stable result after open. Genuine partial fills still fail, including a partial fill whose producer later reaches 511.

All 27 existing offline suites and the new allocation regressions passed, including sanitizer cases. The corrected module then passed three ordinary unstaked start/traffic/stop cycles with physical zero-copy, without observed ENOBUFS or kernel faults. Each active-service interval was about 60 seconds, including initialization; these were short restart checks, not another performance hour or production certification.

The corrected driver payload was installed and verified inside the rebuilt boot image. The payload was verified; this persistent setup was not reboot-tested.

A subsequent same-base Firedancer candidate uses net.xdp.experimental_prebind_fill = true. A separate standby restored a trusted snapshot and caught up before a controlled live voting handover. X710 demoted first; its newest signed history was verified and transferred; P4 then promoted using its existing local key. Both processes remained running during the identity transition. This required coherent vote-account settings, topology, history and restart roles; live identity switching does not prepare those automatically.

The corrected-source instructions distinguish the measured compile-time build from the later runtime-option build and corrected driver. For the later candidate, add experimental_prebind_fill = true to the existing [net.xdp] section shown above; use the complete runtime-option patch against pristine v26.10.5, not on top of the historical compile-time patches. No repeat performance hour is claimed for the latter.

Limits and recovery

This is one successful voting hour on the recorded hardware and software tuple. Kernel, driver, topology, tile layout, traffic and cache history changed between measurements. We have not isolated a zero-copy-only CPU improvement, measured maximum throughput or established Mellanox parity. Disk await is a mean, not a latency percentile; the baseline’s internal replay-distance metric also differs from the new independent lag measurement.

The persistent boot image’s driver payload was inspected, but the persistent voting setup has not been reboot-tested. Restart and failback require controlled reconciliation of current signed history and exclusive voting ownership. Private preparation, fencing and recovery helpers are not shipped as portable automation, and clean dependency provisioning has not been independently reproduced.

A known steering limitation remains: changing the MAC or disabling ntuple while a pool is attached can remove rules without the backstop detecting it. Exceptional firmware timeout, reset and device-loss DMA cleanup remain unqualified. Independent management helps with network failures but cannot recover every kernel hang. Do not force an unload or reset when DMA ownership is uncertain. DoubleZero was not validated.

At the end of the original hour, P4 was the sole voter and X710 was stopped and fenced. The later live handover left P4 voting and X710 caught up under its separate unstaked identity, with zero-copy intact. These describe the recorded handovers, not a guarantee of current host roles. The private startup guard requires fresh peer-role confirmation and fails closed without it; this is not unattended failover. The next contribution work separates the vendor-driver port from the Firedancer initializer proposal; neither has been submitted upstream.