We upgraded our Solana validator’s networking and ended up deliberately lowering its link speed. Our BTR validator is now running its 25G-capable Mellanox connection at 10 Gbps. Early results look good. With help from Jump, we have a strong lead involving short bursts during block production, but we have not yet established the exact point of packet loss.
Our previous NIC used SFP+ ports capped at 10 Gbps. It could not put a 20–25 Gbps burst onto this link. Moving to a 25G-capable NIC changed the peak rate at which the server could send traffic, even if its average bandwidth requirements had not changed. That helps explain why this particular problem could emerge after an upgrade.
Good votes, missing blocks
The symptoms were confusing. We were seeing skipped leader slots while voting performance remained strong. During one observation covering four leader slots, mlx5 tile utilization stayed below 5 percent. Our link checks also looked clean. Those observations did not establish that block data was reaching other validators reliably. Voting and distributing a newly produced block put different demands on the network. A low utilization reading can also miss a burst lasting less than a millisecond. Jump’s receiver-side observations gave us a better view.
The first clue looked like a port problem
Our initial native mlx5 configuration used the Mellanox NIC’s np0 port, which had negotiated at 10 Gbps. Jump reported a clean window of 340 blocks produced without a skip. Later, after a firmware update and a move to np1 at 25 Gbps, Jump reported six skipped slots in a window of 284, with the sample still accumulating. Other validators were also missing substantially more Solana FEC sets.
The immediate suspect was the new port or its network path. Firmware and port changes had happened together, so this comparison could not isolate a single cause. We switched back to np0. Jump could still see missing FEC sets, even before another skipped slot appeared. But np0 was now also negotiating at 25 Gbps. Returning to the original port had not returned us to the original link conditions.
A burst can matter more than an average
Jump’s preliminary diagnosis pointed to bursts of approximately 20–25 Gbps lasting around 0.6 milliseconds during block production. The working hypothesis is that those bursts were overwhelming a link or a hardware queue at the other end. The exact bottleneck has not been confirmed.
A network can have plenty of capacity on average and still struggle with a short burst. If packets arrive faster than the next stage can forward or process them, a queue must absorb the difference. If that queue fills, packets can be dropped. An average bandwidth graph can hide that event. Clean physical-link counters also do not prove that every downstream queue handled the traffic successfully.
The previous NIC’s 10G limit imposed a lower ceiling on the rate entering this path. The upgrade allowed a much faster burst, potentially exposing a limitation that the old connection could not exercise in the same way. That does not mean the previous setup was immune to congestion or packet loss.
Testing 10G safely
We moved voting to our FRA validator before changing BTR’s active connection. After setting BTR’s link to 10 Gbps, we verified connectivity and allowed it to catch up before returning voting to BTR. BTR was back voting at approximately 17:07 UTC on November 4. The subsequent link check showed 10G full duplex, Ethernet FEC off, and no new errors, discards or link drops in the checked counters.
Jump’s early feedback was encouraging: delivery looked clean so far, and the issue appeared to be resolved. They were collecting a larger sample before confirming that assessment. That is our status at publication: an encouraging mitigation, with the longer observation window still pending.
Two different meanings of FEC
Ethernet FEC, including RS-FEC, corrects bit errors on a physical link. Solana FEC sets concern groups of data and coding shreds used in block propagation. Jump’s graph measures reception of Solana FEC sets. It is not a graph of Ethernet RS-FEC errors.
Our 25G link reported RS-FEC, while the 10G test reports Ethernet FEC off. Because those operational conditions changed together, the test does not cleanly separate link speed from Ethernet FEC behavior. The presence of RS-FEC alone is not evidence that it malfunctioned.
What we take from this
We are keeping this finding narrow. The evidence points toward a problem associated with our 25G configuration, and the initial 10G result supports Jump’s burst-handling hypothesis. It does not yet identify a defective NIC, cable, switch or specific queue. Nor does it establish that every validator should run at 10G.
For us, the practical lesson is to check what happens during block production at the receivers, not just what the sender’s average utilization and link counters report. A faster interface increases the rate the server can transmit. Reliable block delivery still depends on the entire path keeping up.
