2026-04-01 — Infrastructure Migration
Validator migrated from Intel i5-9600K to AMD Ryzen 5 5500
RESOLVED Duration: ~30 minutes
Impact: Validator offline during migration.
Root cause: Planned hardware migration for SHA-NI support and improved hash computation performance.
Resolution: Full state re-sync on new hardware. Hash compute time improved from 2.4s to 15ms (depth-4 bucket caching). Round time dropped from 2.5s to ~100ms.
Follow-up: rippled moved to separate machine (localai) to eliminate CPU contention.
2026-04-01 — DNS Volume Resolver
Volume resolver container causing resource contention
RESOLVED Duration: ~2 hours (intermittent)
Impact: Incomplete state downloads (8.4M of 18.7M objects). rippled RPC timeouts under concurrent load.
Root cause: Docker volume-resolver container consuming excessive resources on the same machine as rippled, causing RPC request failures during bulk state download.
Resolution: Volume resolver stopped. All machines switched to 8.8.8.8 DNS. Full state download completed successfully.
2026-03-29 — LedgerHashes Sub-page Fix
Hash mismatches caused by incorrect LedgerHashes sub-page key computation
RESOLVED Duration: ~6 hours (debugging)
Impact: Intermittent hash mismatches every ~100 ledgers.
Root cause: LedgerHashes sub-page key divisor was 256 instead of 65536. The sub-page key is computed as SHA512Half(0x0073 || seq/65536), not seq/256.
Resolution: Fixed divisor. Achieved 28,500+ consecutive matches with zero mismatches. Self-healing retry mechanism verified working.
2026-03-29 — Transaction Ordering Bug
State hash divergence from incorrect transaction application order
RESOLVED Duration: ~12 hours (investigation + fix)
Impact: Consistent hash mismatches on every ledger.
Root cause: rippled returns transactions in arbitrary array order, not execution order. Transactions must be sorted by TransactionIndex before applying AffectedNodes. When transactions touch the same objects (create then modify), processing out of order produces incorrect state.
Resolution: Sort transactions by TransactionIndex before processing. Also fixed: FeeSettings singleton, entryNotFound handling, backfill race condition, and response ledger verification.