Our Intel X710 completed an hour of voting on Solana testnet using Firedancer’s experimental native iAVF networking path. Compared with our earlier AF_XDP zero-copy run on the same server, validator system CPU fell 86.67%, from 0.026103 to 0.003480 core equivalents.
The absolute saving was 0.02262 core, or about 81 CPU-seconds over an hour. Total validator CPU stayed essentially flat: 28.00657 versus 28.00863 cores. That is a useful reduction in kernel work, but it is not evidence that we freed a core or increased validator capacity.
Along the way, we found a filter-validation bug that prevented a correct hardware rule from passing setup. We contributed a focused fix and regression test to the Intel networking PR discussion. The fix is available as commit db6637e; it has been submitted for review.
What “native” changes
With AF_XDP zero-copy, the NIC can write received packets directly into application memory, while the Linux driver still participates in queue setup and packet processing. Zero-copy removes a copy; it does not remove the kernel from the whole path.
The native iAVF provider takes a different route. It creates a virtual function, or VF, on the Intel physical port and gives Firedancer access to its queues through VFIO. The physical function, or PF, remains under the normal Linux driver for ordinary host traffic. Hardware filters keep selected validator UDP traffic from also being processed by the PF’s kernel path. The operating system still provides setup, isolation and other services.
This is closer in purpose to Firedancer’s native mlx5 path: directly accessing device queues rather than using AF_XDP for the main packet path. It is a different driver implementation and device interface. We did not compare it against a Mellanox card, and this experiment says nothing about parity in latency or capacity.
The package lives under intel-native/i40e/: i40e identifies the Linux driver for the X710 physical function, while iavf is Firedancer’s native provider for its virtual function. A future E810 experiment can have its own ice/ directory; this article contains no E810 results.
The machine used an X710 with NVM 6.01/API 1.7, a Ryzen 7950X, and Linux 7.0.0-42-generic. We did not install a custom Intel kernel driver or update firmware.
Two builds, two different tests
| Build | What we actually tested |
|---|---|
PR #12540 at 696316b2, plus our readback fix |
About 15 minutes of caught-up unstaked native traffic, followed by ordinary shutdown and restoration to AF_XDP |
Experimental backport onto Firedancer v26.10.5 (6286f99e), including the eight Intel commits and readback fix |
Qualification of that exact binary, an ordinary stop/restart, a controlled voting handover, then the one-hour voting measurement |
The performance table below belongs to the release backport. We have only the shorter unstaked test for the PR-head build. Both are experimental builds.
The operator configuration
These are the relevant network sections for the tested release backport, not a complete validator configuration:
[net]
provider = "iavf"
interface = "YOUR_X710_PF"
bind_address = "YOUR_PUBLIC_IPV4"
[net.iavf]
rx_queue_size = 4096
tx_queue_size = 4096
interface names the physical port; the reviewed setup procedure creates and assigns its VF. Replace the placeholders and retain the rest of your validated identity, history, accounts, snapshot and cluster configuration. Keep RPC and metrics on localhost and preserve Firedancer’s sandbox and privilege drop.
Use an explicit provider. In this derivative, provider = "auto" can select iAVF on supported Intel ports. To retain AF_XDP deliberately, use provider = "xdp" and its reviewed settings.
The measured service used these resource settings:
[Service]
LimitMEMLOCK=96G
MemoryMax=104G
MemorySwapMax=0
Restart=no
This is a resource fragment, not a service unit. Keep your existing startup gates, binary/configuration paths and identity checks. Restart=no describes our supervised experiment; it is not an unattended recovery policy.
The voting layout calculated 39 tiles, 78 one-GiB pages and 2,354 two-MiB pages. We reserved 79 one-GiB pages and 2,354 two-MiB pages, adding 49 two-MiB pages to the earlier reservation. These are this machine’s values. Recalculate memory using the exact binary and complete configuration; the PR-head build has different requirements.
What must be ready before startup
The tested setup used one VF with an exclusive IOMMU group, bound to vfio-pci. IOMMU protection stayed enabled. Spoof checking was disabled only for that VF; VF trust remained disabled. Independent management used another NIC. Do not substitute a no-IOMMU mode or bind an unrelated device merely to make startup succeed.
The release backport installed nine exact destination-address/UDP selectors, including legacy UDP port 9006. The PR-head trial used eight selectors. Verify port ownership and destination scope on your own host. These are intentional PF DROP rules that suppress duplicate kernel delivery, not a general host firewall replacement. A failed rule readback is a setup failure to investigate, not a check to skip.
Before voting, qualify the exact candidate with an isolated unstaked identity. Confirm VFIO ownership, advancing native receive/transmit completions, absence of a physical XDP attachment, independent catch-up and ordinary teardown. Prepare the fallback and current signed vote history before any identity transition. Never allow both nodes to vote with the same identity.
The supporting package separates the settings from the procedure: a configuration example and service resource fragment, followed by an ordered runbook for preparation, native-path checks, voting handover and rollback. Source pins, patch order and measurement definitions give agents the detail they need without turning this article into an installation script. The package also includes the full release backport, its ordered patch series, a public Docker build recipe and machine-readable results. Start with the package README, then use the operator runbook and source/build instructions. These links pin the exact published revision.
What the voting hour showed
The scored interval ran from 23:48:04 UTC on 1 November to 00:48:04 UTC on 2 November 2026. All 361 observations were caught up, public votes advanced, and the collector recorded no errors or process-generation changes. Accounting covered all 36 steady-state processes in a single disjoint service cgroup.
| Measurement | Earlier AF_XDP zero-copy | Native iAVF |
|---|---|---|
| Total validator CPU, cores | 28.006571 | 28.008631 |
| Validator system CPU, cores | 0.026103 | 0.003480 |
| Whole-host softirq CPU, cores | 0.016433 | 0.001000 |
| Hardware RX packets/second | 21,691 | 23,297 |
| Hardware TX packets/second | 24,484 | 25,079 |
| Median cgroup memory, GiB | 97.737 | 101.375 |
| Disk writes, MiB/second | 2.027 | 1.759 |
| Mean disk write await, ms | 15.989 | 1.164 |
Here, a CPU core equivalent is CPU time divided by elapsed wall time. Validator system CPU measures kernel time charged to the validator. Whole-host softirq measures another accounting scope; do not add the two figures.
Whole-host softirq time fell 93.91%, an absolute difference of 0.01543 core. Total validator CPU changed by only about 0.00206 core. Busy polling can keep utilization high even when work gets cheaper, but these measurements do not establish cycles per packet or explain every difference in total CPU.
The native run received 7.40% more packets per second and replayed 13.37% more transactions per second. That describes the traffic that arrived during ordinary testnet operation. We did not offer a controlled load, measure maximum throughput, test a leader burst or measure network or transaction latency. Independent slot lag was two slots at the median and six at the maximum; slot lag is not packet latency.
Native counters recorded 83,844,400 receives and 90,266,276 transmit completions. Captured iAVF drop/failure and hardware error/discard counters did not increase. That is not a zero-loss result: Linux RX-drop counters increased by 1,800 on each of the public PF and management interfaces, and protocol rejection counters also advanced. PF interface traffic counters miss most VF traffic, so compare hardware-port and application counters rather than treating Linux PF bytes as the whole workload.
Memory and disk need different interpretations
RAM did not show a clear saving. Median cgroup charge was 3.639 GiB higher. The native run included a stable 82.598 GiB of hugepage charge, about 0.911 GiB of anonymous memory and growing file cache. On this kernel, memory.current already includes hugepages; adding them again would double-count them. Host available memory stayed similar.
Peak cgroup charge reached 103.230 GiB against a 104 GiB limit: just 0.770 GiB of headroom. No max/OOM/kill event occurred during the measured interval. That is a reason to watch the limit, not evidence that every workload fits.
Disk writes, IOPS and average write await were lower in the native interval. The earlier AF_XDP interval included a startup write burst, and cache/workload state differed. We cannot attribute the disk difference to native networking. Block-device await is not transaction latency.
The readback bug and the narrow fix
The original setup compared entire kernel rule structures byte-for-byte. On our X710/i40e system, reading back a correct IPv4/UDP filter populated 16 extra 0xff bytes in inactive IPv6-mask union storage. The active IPv4 selectors, masks and action matched, but the whole-union comparison rejected it.
Our fix compares the supported rule’s meaning: active IPv4 fields and masks, action, location and meaningful extensions. Unsupported flags, types and extensions still fail. Only inactive storage and reserved padding are ignored. It changes neither VF privileges nor DMA ownership, queue lifecycle or filter scope.
The regression fixture reproduces the captured representation, with 687 negative mutations alongside padding and wildcard cases. Normal and comparator sanitizer tests passed. Those sanitizer results cover the relevant tests or translation units, not an instrumented full validator.
The contribution is available in the PR discussion and focused commit. The native networking implementation belongs to the upstream Intel PR; our contribution is this readback correction and its regression coverage.
What remains experimental
The native layout had 39 configured tiles versus 38 in the historical AF_XDP layout. An iAVF tile and a socket tile replace the net tile; the socket tile was idle during the native hour. Builds, compiler/ISA choices, epochs, packet mix and cache history also differed. This is a sequential same-machine comparison, not an isolated causal benchmark.
The source package reconstructs all 3,661 manifested files of the release backport. Its public Ubuntu Docker recipe built Firedancer and all five unit-test binaries, and the rule comparator passed all 687 rejection cases. Four other tests could not run their cases in the original Docker environment because it lacked NUMA topology; they were not rerun in the fresh public image. The original voting build had passed those tests on supported Linux.
A rebuild is not the measured binary. In the original build environment, 654 of 656 object files and the executable’s code section matched. The two changed objects embed build metadata, and the resulting binary hash differs. The repository records that comparison and the separate public-image build hash. Neither rebuilt artifact has inherited the original binary’s hardware qualification.
The native service was runtime-only with a fail-closed start gate. Automatic boot, unattended restart, exceptional reset/device-loss recovery and production failover are not established. DoubleZero was not validated. Returning to AF_XDP or another voter requires ordinary ownership cleanup and the newest verified signed history; ambiguous DMA ownership is not permission for forced reset or unload.
For operators, the result is a working experimental native path, a specific setup bug fixed, and less observed kernel work. The next useful tests are repeatable workload comparisons and recovery qualification, not a larger claim from the percentage alone.
