Merge master commit 100cc3816c59357df488b17972aa5e2846ead831 into overlay V2 by SirTyson · Pull Request #5428 · stellar/stellar-core

SirTyson · 2026-08-24T18:33:47Z

Merges 100cc38 into overlay-v2.

Most changes could be applied without issue. The two "conflicts" that we resolved were porting tx apply support back into overlay v2 (which will be useful for the new "end-to-end" apply load tests). I've also reverted overlay-v2 SimpleTimer metrics back to Timer, since master has picked up a significant perf improvement on the original Timer implementation.

Additionally, I cleaned up our unit tests such that we pass CI. This was mostly removing tests that were intentionally broken and fixing a couple topology issues. There were two legitimate bugs though, fixed in 292eed4.

The first was a use after free, which failed a unit test, but was only called when we receive bad quorum set messages, so it wouldn't have shown up in our experiments. The 2nd bug is a race condition, where an SCP message with two unfetched dependencies (i.e. the sender's quorum set and a tx set) could wedge a node if the quorum set message arrives after the tx set. This one could actually occur in our experiments on startup, since our first message will require transmitting qsets with an empty tx set, such that the race could hit. We've seen instability on startup, this may or may not have been a contributing factor in practice. The good news is both of these were caught by "PendingEnvelopes recvSCPEnvelope".

Checklist

Reviewed the contributing document
Rebased on top of master (no merge commits)
Ran clang-format v8.0.0 (via make format or the Visual Studio extension)
Compiles
Ran all tests
If change impacts performance, include supporting evidence per the performance document

marta-lokhova

thanks so much, all helpful changes!

Copilot

Pull request overview

Merges the referenced master changes into overlay V2 while adapting simulation, load generation, metrics, and tests to the Rust overlay.

Changes:

Adds Rust-overlay topology wiring, mempool handling, and SCP dependency fixes.
Extends apply-load benchmarks with validation-and-consensus timing.
Updates tests, CI selection, metrics, and documentation.

Reviewed changes

Copilot reviewed 38 out of 40 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
`src/simulation/Topologies.cpp`	Wires simulation topologies.
`src/simulation/test/LoadGeneratorTests.cpp`	Expands load-generator coverage.
`src/simulation/Simulation.h`	Declares connection helpers.
`src/simulation/Simulation.cpp`	Implements Rust-overlay peer wiring.
`src/simulation/LoadGenerator.h`	Adds account lifecycle tracking.
`src/simulation/LoadGenerator.cpp`	Delays account reuse until apply.
`src/simulation/CoreTests.cpp`	Re-enables adapted simulation tests.
`src/simulation/ApplyLoad.h`	Declares benchmark timing phases.
`src/simulation/ApplyLoad.cpp`	Adds consensus-based timing.
`src/overlay/test/OverlayIPCTests.cpp`	Updates ports, topology, and tags.
`src/overlay/RustOverlayManager.h`	Extends test-peer and mempool APIs.
`src/overlay/RustOverlayManager.cpp`	Implements peer updates and removals.
`src/overlay/OverlayMetrics.h`	Restores transaction receive timer.
`src/overlay/OverlayMetrics.cpp`	Registers the full timer.
`src/overlay/OverlayIPC.h`	Declares transaction removal.
`src/overlay/OverlayIPC.cpp`	Hardens process shutdown and FD handling.
`src/main/test/CommandHandlerTests.cpp`	Adapts asynchronous submission tests.
`src/main/test/BannedAccountsPersistorTests.cpp`	Removes disabled tests.
`src/main/test/ApplicationUtilsTests.cpp`	Fixes watcher topology and timing.
`src/main/Config.h`	Adds apply-load timing configuration.
`src/main/Config.cpp`	Parses timing mode and restricts manual close.
`src/main/ApplicationImpl.cpp`	Skips overlay startup in standalone mode.
`src/ledger/test/LedgerCloseMetaStreamTests.cpp`	Configures explicit watcher peers.
`src/ledger/LedgerManagerImpl.cpp`	Updates timer documentation.
`src/herder/PendingEnvelopes.cpp`	Fixes quorum-set arrival handling.
`src/herder/HerderSCPDriver.h`	Adds tx-set validation timer.
`src/herder/HerderSCPDriver.cpp`	Instruments tx-set validation.
`src/herder/HerderImpl.cpp`	Updates recovery and Rust mempool nomination.
`src/database/test/DatabaseTests.cpp`	Removes obsolete peer migration checks.
`docs/stellar-core_example.cfg`	Documents manual-close restriction.
`docs/software/commands.md`	Documents apply-load timing modes.
`docs/metrics.md`	Updates timer metric definitions.
`docs/apply-load-max-sac-tps.cfg`	Configures apply-only timing.
`docs/apply-load-ledger-limits.cfg`	Documents timing selection.
`docs/apply-load-for-meta.cfg`	Adds timing configuration.
`docs/apply-load-benchmark-token.cfg`	Adds timing configuration.
`docs/apply-load-benchmark-sac.cfg`	Adds timing configuration.
`ci-build.sh`	Revises test partitioning and selection.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+    // The mempool is fee-ordered and sequence-number-oblivious, so it can
+    // hand us several transactions from one source account (e.g. a chained
+    // pair). A tx set may only contain one tx per source account, so keep
+    // the lowest sequence number per account and let the others wait for a
+    // later ledger.
+    auto onePerSourceAccount = [](TxFrameList& txs) {
+        std::unordered_map<AccountID, size_t> firstBySource;
+        TxFrameList kept;
+        for (auto const& tx : txs)
+        {
+            auto [it, inserted] =
+                firstBySource.emplace(tx->getSourceID(), kept.size());
+            if (inserted)
+            {
+                kept.push_back(tx);
+            }
+            else if (tx->getSeqNum() < kept[it->second]->getSeqNum())
+            {
+                kept[it->second] = tx;
+            }
+        }
+        txs = std::move(kept);
+    };
+    onePerSourceAccount(classicTxs);
+    onePerSourceAccount(sorobanTxs);


+                auto acc = ledgerView.getAccount(tx->getSourceID());
+                if (acc &&
+                    tx->getSeqNum() > acc.current().data.account().seqNum + 1)
+                {
+                    continue;
+                }


+            sim->addNode(key, qSetHere);
+
+            // connect to core nodes (round-robin)
+            curCore = (curCore + 1) % coreNodeIDs.size();


+    auto& herder = mApp.getHerder();
+    auto const& lcl = mApp.getLedgerManager().getLastClosedLedgerHeader();
+
+    uint64_t const closeTime = lcl.header.scpValue.closeTime + 1;


+    , mRecvTransactionTimer(
+          app.getMetrics().NewTimer({"overlay", "recv", "transaction"}))


    // ── Recv timers (aggregate) ──
-    // SimpleTimer: high-frequency TX recv path
-    SimpleTimer& mRecvTransactionTimer;
+    medida::Timer& mRecvTransactionTimer;


 } // anonymous namespace

-TEST_CASE("OverlayIPC connects to Rust overlay", "[overlay-ipc-rust][.]")
+TEST_CASE("OverlayIPC connects to Rust overlay", "[overlay-ipc]")


+    std::optional<uint64> detachedCheckpoint;
+    if (slots.size() >= 2 && slots[1] > slots[0] + 1)
+    {
+        detachedCheckpoint = slots[0];
+    }


+    auto stateArchivalConfigSettinsgEntry =
+        ledgerView.load(configSettingKey(CONFIG_SETTING_STATE_ARCHIVAL));
+    auto& contractExecutionLanesSettings =
+        contractExecutionLanesSettingsEntry.current().data.configSetting();
+    auto& stateArchivalSettings =
+        stateArchivalConfigSettinsgEntry.current().data.configSetting();


+        cfg.ARTIFICIALLY_GENERATE_LOAD_FOR_TESTING = true;
+        cfg.UPDATE_SOROBAN_COSTS_DURING_PROTOCOL_UPGRADE_FOR_TESTING = true;
+        cfg.GENESIS_TEST_ACCOUNT_COUNT = 500;
+        //  Use tight bounds to we can verify storage works properly


SirTyson · 2026-08-27T23:43:47Z

CI was failing due to a race condition, so I did another lap. I ended up porting many of the tests in master back into this branch, and fixing a few small issues along the way. Notably, there was some instability in loadgen, which may be causing simulation failures for the non-overlay only tests.

SirTyson requested a review from marta-lokhova August 25, 2026 17:57

marta-lokhova approved these changes Aug 25, 2026

View reviewed changes

SirTyson added 11 commits August 26, 2026 10:39

Merge master commit 100cc38 into overlay-v2

fe02a3d

Remove stale DB table unit tests

d9b496c

Fix some broken simulation tests

8927b30

Remove support for standalone mode (and tests)

45a6ad1

Remove tests for unsuported tx submission checks

acc71f6

Fix race condition in nomination envelope processing

c7459b3

update CI and make check for parallelism

45ac2a5

Restore previously deleted herder tests that now work

ecbcefb

Reintroduce basic herder tests

dcb3920

Fix SCP checkpoint handling

3f7b6d0

Fix loadgen seqno issus

5fd6b4b

Copilot AI balanced review requested due to automatic review settings August 27, 2026 23:20

SirTyson force-pushed the overlay-v2-aug-20-merge branch from 20bef27 to 5fd6b4b Compare August 27, 2026 23:20

Copilot started reviewing on behalf of SirTyson August 27, 2026 23:20 View session

Copilot AI reviewed Aug 27, 2026

View reviewed changes

marta-lokhova approved these changes Aug 28, 2026

View reviewed changes

SirTyson merged commit b90d48e into stellar:overlay-v2-shared Aug 28, 2026
5 of 6 checks passed

Navigation Menu

Uh oh!

Merge master commit 100cc3816c59357df488b17972aa5e2846ead831 into overlay V2 - #5428

Merge master commit 100cc3816c59357df488b17972aa5e2846ead831 into overlay V2#5428
SirTyson merged 11 commits into
stellar:overlay-v2-sharedfrom
SirTyson:overlay-v2-aug-20-merge

SirTyson commented Aug 24, 2026 •

edited

Loading

Uh oh!

marta-lokhova left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

SirTyson commented Aug 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		, mRecvTransactionTimer(
		app.getMetrics().NewTimer({"overlay", "recv", "transaction"}))

Uh oh!

Conversation

SirTyson commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

marta-lokhova left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

SirTyson commented Aug 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

SirTyson commented Aug 24, 2026 •

edited

Loading