Skip to content

Send BlockComponentProcessed on HTTP block import #8110

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dapplion wants to merge 3 commits into from
Closed

Send BlockComponentProcessed on HTTP block import #8110

dapplion wants to merge 3 commits into from

Conversation

@dapplion
Copy link
Collaborator

@dapplion dapplion commented Sep 24, 2025 •
edited
Loading

Issue Addressed

Now in unstable block import from the HTTP API also insert the block in the da_checker while the block is being execution verified. If lookup sync finds the block in the da_checker in NotValidated state it expects a GossipBlockProcessResult message sometime later. That message is only sent after block import in gossip.

I confirmed in our node's logs for 4/4 cases of stuck lookups are caused by this sequence of events:

  • Receive block through API, insert into da_checker in fn process_block in put_pre_execution_block
  • Create lookup and leave in AwaitingDownload(block in processing cache) state
  • Block from HTTP API finishes importing
  • Lookup is left stuck

Closes #8104

Proposed Changes

This PR fixes the issue as when the HTTP block completes import the lookup continues thanks to the BlockProcessResult event.

The interaction of lookup sync with the da_checker feels very fragile. I reviewed #8045 and didn't realize that we had to send this event. The sync tests do not cover this code either. We only assert that certain events are sent to the beacon processor but we don't execute them. I would like to have a "better" fix at some point, such as storing a handle in the da_checker that lookup sync can await. However, this approach requires way more changes as sync is not async.

@dapplion dapplion requested a review from jxs as a code owner September 24, 2025 13:23
@jimmygchen jimmygchen added the v8.0.0-rc.0 Q3 2025 release for Fusaka on Holesky label Sep 24, 2025
@dapplion dapplion marked this pull request as draft September 24, 2025 14:20
@dapplion dapplion added the bug Something isn't working label Sep 24, 2025
@dapplion dapplion mentioned this pull request Sep 24, 2025
Merged
@dapplion
Copy link
Collaborator Author

dapplion commented Sep 24, 2025

@dapplion dapplion added the work-in-progress PR is a work-in-progress label Sep 24, 2025
@jimmygchen
Copy link
Member

jimmygchen commented Sep 25, 2025

Thanks @dapplion, I agree that #8112 is a better solution 👍

@jimmygchen jimmygchen closed this Sep 25, 2025
mergify bot pushed a commit that referenced this pull request Sep 25, 2025
@dapplion
Only mark block lookups as pending if block is importing from gossip (#…
ffa7b2b 
…8112)

- PR #8045 introduced a regression of how lookup sync interacts with the da_checker.

Now in unstable block import from the HTTP API also insert the block in the da_checker while the block is being execution verified. If lookup sync finds the block in the da_checker in `NotValidated` state it expects a `GossipBlockProcessResult` message sometime later. That message is only sent after block import in gossip.

I confirmed in our node's logs for 4/4 cases of stuck lookups are caused by this sequence of events:
- Receive block through API, insert into da_checker in fn process_block in put_pre_execution_block
- Create lookup and leave in AwaitingDownload(block in processing cache) state
- Block from HTTP API finishes importing
- Lookup is left stuck

Closes #8104


  - #8110 was my initial solution attempt but we can't send the `GossipBlockProcessResult` event from the `http_api` crate without adding new channels, which seems messy.

For a given node it's rare that a lookup is created at the same time that a block is being published. This PR solves #8104 by allowing lookup sync to import the block twice in that case.


Co-Authored-By: dapplion <[email protected]>
pawanjay176 pushed a commit to pawanjay176/lighthouse that referenced this pull request Sep 25, 2025
@dapplion @pawanjay176
Only mark block lookups as pending if block is importing from gossip (s…
3f8998f 
…igp#8112)

- PR sigp#8045 introduced a regression of how lookup sync interacts with the da_checker.

Now in unstable block import from the HTTP API also insert the block in the da_checker while the block is being execution verified. If lookup sync finds the block in the da_checker in `NotValidated` state it expects a `GossipBlockProcessResult` message sometime later. That message is only sent after block import in gossip.

I confirmed in our node's logs for 4/4 cases of stuck lookups are caused by this sequence of events:
- Receive block through API, insert into da_checker in fn process_block in put_pre_execution_block
- Create lookup and leave in AwaitingDownload(block in processing cache) state
- Block from HTTP API finishes importing
- Lookup is left stuck

Closes sigp#8104

  - sigp#8110 was my initial solution attempt but we can't send the `GossipBlockProcessResult` event from the `http_api` crate without adding new channels, which seems messy.

For a given node it's rare that a lookup is created at the same time that a block is being published. This PR solves sigp#8104 by allowing lookup sync to import the block twice in that case.

Co-Authored-By: dapplion <[email protected]>
jimmygchen added a commit to jimmygchen/lighthouse that referenced this pull request Oct 10, 2025
@jimmygchen
Squashed commit of the following (sigp#7946):
a0f88b2 
commit 9b2de09
Author: Pawan Dhananjay <[email protected]>
Date:   Mon Oct 6 17:22:44 2025 -0700

    Rethink peer scoring

commit 5c562c6
Author: Pawan Dhananjay <[email protected]>
Date:   Mon Oct 6 11:17:57 2025 -0700

    Fix some more issues

commit 15df3d2
Merge: c491856 26575c5
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Oct 2 13:15:59 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit c491856
Merge: 826a06e ffa7b2b
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Sep 25 16:23:27 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit 826a06e
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Sep 25 16:21:55 2025 -0700

    Fix variant name

commit 421e954
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Sep 25 16:05:45 2025 -0700

    Revert "Revert type change in UnexpectedRequestId"

    This reverts commit 6ea14016f3d164456bc4c3cae0355ab532fe1a86.

commit 3f8998f
Author: Lion - dapplion <[email protected]>
Date:   Thu Sep 25 05:52:27 2025 +0200

    Only mark block lookups as pending if block is importing from gossip (sigp#8112)

    - PR sigp#8045 introduced a regression of how lookup sync interacts with the da_checker.

    Now in unstable block import from the HTTP API also insert the block in the da_checker while the block is being execution verified. If lookup sync finds the block in the da_checker in `NotValidated` state it expects a `GossipBlockProcessResult` message sometime later. That message is only sent after block import in gossip.

    I confirmed in our node's logs for 4/4 cases of stuck lookups are caused by this sequence of events:
    - Receive block through API, insert into da_checker in fn process_block in put_pre_execution_block
    - Create lookup and leave in AwaitingDownload(block in processing cache) state
    - Block from HTTP API finishes importing
    - Lookup is left stuck

    Closes sigp#8104

      - sigp#8110 was my initial solution attempt but we can't send the `GossipBlockProcessResult` event from the `http_api` crate without adding new channels, which seems messy.

    For a given node it's rare that a lookup is created at the same time that a block is being published. This PR solves sigp#8104 by allowing lookup sync to import the block twice in that case.

    Co-Authored-By: dapplion <[email protected]>

commit d99df0a
Author: Jimmy Chen <[email protected]>
Date:   Thu Sep 25 12:52:07 2025 +1000

    Only send data coumn subnet discovery requests after peerdas is scheduled (sigp#8109)

    sigp#8105 (to be confirmed)

    I noticed a large number of failed discovery requests after deploying latest `unstable` to some of our testnet and mainnet nodes. This is because of a recent PeerDAS change to attempt to maintain sufficient peers across data column subnets - this shouldn't be enabled on network without peerdas scheduled, otherwise it will keep retrying discovery on these subnets and never succeed.

    Also removed some unused files.

    Co-Authored-By: Jimmy Chen <[email protected]>

    Co-Authored-By: Jimmy Chen <[email protected]>

commit cf46d10
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Sep 25 15:54:32 2025 -0700

    Fix issues from review

commit c2aa4ae
Author: dapplion <[email protected]>
Date:   Thu Sep 25 23:58:31 2025 +0200

    Revert type change in UnexpectedRequestId

commit 7488755
Author: dapplion <[email protected]>
Date:   Thu Sep 25 22:09:06 2025 +0200

    De-duplicate data columns by root request type

commit 7650032
Author: dapplion <[email protected]>
Date:   Thu Sep 25 23:52:47 2025 +0200

    Rename DataColumnsFromRange

commit 4b0b655
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Sep 24 12:31:06 2025 -0700

    Fix nits

commit aa6a1bc
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Sep 24 12:16:05 2025 -0700

    Create a custom penalize_sync_peer method for clarity

commit 27195ca
Merge: 4a59d35 af27402
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Sep 24 11:16:27 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit 4a59d35
Merge: 2f35c36 5928407
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Sep 17 15:15:44 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit 2f35c36
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Sep 17 15:13:40 2025 -0700

    Add some metrics

commit b3b3756
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Sep 17 15:01:46 2025 -0700

    Fix tests

commit e3aed89
Author: Pawan Dhananjay <[email protected]>
Date:   Tue Sep 16 17:46:21 2025 -0700

    Remove retry test that we do not use anymore

commit baee27a
Merge: 9db4c30 191570e
Author: Pawan Dhananjay <[email protected]>
Date:   Tue Sep 16 16:39:30 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit 9db4c30
Author: Pawan Dhananjay <[email protected]>
Date:   Tue Sep 16 16:38:19 2025 -0700

    Fix small bug

commit 08bba3f
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Sep 11 18:24:25 2025 -0700

    fmt

commit cffbd34
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Sep 11 18:19:16 2025 -0700

    Reduce code duplication

commit bf09d57
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Sep 11 15:06:07 2025 -0700

    Fix some issues from lion's review

commit 04398ad
Author: Pawan Dhananjay <[email protected]>
Date:   Mon Sep 8 15:37:26 2025 -0700

    Fix some more issues from review

commit 4f62a9c
Merge: e259ecd 8ec2640
Author: Pawan Dhananjay <[email protected]>
Date:   Fri Sep 5 13:00:47 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit e259ecd
Author: Pawan Dhananjay <[email protected]>
Date:   Fri Sep 5 12:58:49 2025 -0700

    More renamings

commit 6a2a33d
Author: Pawan Dhananjay <[email protected]>
Date:   Fri Sep 5 12:48:05 2025 -0700

    Fix some issues from review

commit e0d8f04
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Sep 4 18:09:07 2025 -0700

    Tests compile

commit 7e91eeb
Merge: 29c2f83 9d2f55a
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Sep 4 18:08:52 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit 29c2f83
Author: Pawan Dhananjay <[email protected]>
Date:   Tue Sep 2 13:29:19 2025 -0700

    Cleanup SyncingChain

commit 8458df6
Author: Pawan Dhananjay <[email protected]>
Date:   Mon Sep 1 12:53:01 2025 -0700

    Attempt sending awaitingDownload batches when restarting sync

commit 7a6d0d9
Author: Pawan Dhananjay <[email protected]>
Date:   Sun Aug 31 19:07:39 2025 -0700

    Force processing_target request

commit 4f60e86
Author: Pawan Dhananjay <[email protected]>
Date:   Sun Aug 31 14:54:44 2025 -0700

    Add logs to debug stuck range sync

commit b07bc6d
Author: Pawan Dhananjay <[email protected]>
Date:   Fri Aug 29 16:18:53 2025 -0700

    Force requests if batch buffer is full under certain conditions

commit 19b0a5c
Merge: 8e1337d 438fb65
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Aug 28 23:25:14 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit 8e1337d
Merge: da1aaba a134d43
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Aug 28 18:03:44 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit da1aaba
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 27 17:34:27 2025 -0700

    Revise error tolerance

commit 7331323
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 27 17:30:46 2025 -0700

    AwaitingValidation state only needs block peer

commit 8386bd9
Merge: b4bc7fe d235f2c
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 27 17:08:42 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit b4bc7fe
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 27 14:30:46 2025 -0700

    Remove more debug logs

commit 05adb71
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 27 14:26:28 2025 -0700

    Fix bug with partial column responses before all column requests sent

commit a97cf88
Author: Pawan Dhananjay <[email protected]>
Date:   Tue Aug 26 18:53:51 2025 -0700

    Add docs

commit 27d0b36
Author: Pawan Dhananjay <[email protected]>
Date:   Tue Aug 26 16:06:27 2025 -0700

    Remove debug statements

commit 90d319f
Merge: 7c214f5 3e78034
Author: Pawan Dhananjay <[email protected]>
Date:   Tue Aug 26 15:38:47 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit 7c214f5
Merge: 52762b9 daf1c7c
Author: Pawan Dhananjay <[email protected]>
Date:   Mon Aug 25 11:01:40 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit 52762b9
Author: Pawan Dhananjay <[email protected]>
Date:   Fri Aug 22 11:52:56 2025 -0700

    Handle 0 blobs per epoch case

commit da27441
Merge: 521778b cee30d8
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Aug 21 11:02:22 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit 521778b
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Aug 21 07:03:58 2025 -0700

    Random logs

commit 4540195
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 13 17:33:47 2025 -0700

    Request columns from global peer pool

commit fdce537
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 20 18:37:44 2025 -0700

    Implement responsible peer tracking

commit 17c4e34
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 20 14:18:15 2025 -0700

    Avoid root requests for backfill sync

commit 1a0df30
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 20 14:14:48 2025 -0700

    Also penalize all batch peers for availability check errors

commit 6da924b
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 20 14:14:26 2025 -0700

    Fix bug in initialization code

commit 68cce37
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 20 14:14:11 2025 -0700

    Try to avoid chains failing for rpc errors

commit ca9cfd5
Author: Pawan Dhananjay <[email protected]>
Date:   Tue Aug 19 12:17:59 2025 -0700

    Small fixes

commit 5337e46
Author: Pawan Dhananjay <[email protected]>
Date:   Mon Aug 18 22:59:38 2025 -0700

    Add a function to retry column requests that could not be made

commit 9455153
Author: Pawan Dhananjay <[email protected]>
Date:   Mon Aug 18 14:48:44 2025 -0700

    Without retries

commit 6bd8944
Author: Pawan Dhananjay <[email protected]>
Date:   Fri Aug 15 14:23:48 2025 -0700

    Reduce backfill buffer size

commit 156449c
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Aug 14 09:10:28 2025 -0700

    Increase columns_by_root quota

commit 836f9c6
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 13 16:37:12 2025 -0700

    Priorotize status v2

commit 490b627
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 13 16:36:56 2025 -0700

    Penalize if invalid EL block
lmnzx pushed a commit to lmnzx/lighthouse that referenced this pull request Nov 4, 2025
@dapplion @lmnzx
Only mark block lookups as pending if block is importing from gossip (s…
b7da8b1 
…igp#8112)

- PR sigp#8045 introduced a regression of how lookup sync interacts with the da_checker.

Now in unstable block import from the HTTP API also insert the block in the da_checker while the block is being execution verified. If lookup sync finds the block in the da_checker in `NotValidated` state it expects a `GossipBlockProcessResult` message sometime later. That message is only sent after block import in gossip.

I confirmed in our node's logs for 4/4 cases of stuck lookups are caused by this sequence of events:
- Receive block through API, insert into da_checker in fn process_block in put_pre_execution_block
- Create lookup and leave in AwaitingDownload(block in processing cache) state
- Block from HTTP API finishes importing
- Lookup is left stuck

Closes sigp#8104


  - sigp#8110 was my initial solution attempt but we can't send the `GossipBlockProcessResult` event from the `http_api` crate without adding new channels, which seems messy.

For a given node it's rare that a lookup is created at the same time that a block is being published. This PR solves sigp#8104 by allowing lookup sync to import the block twice in that case.


Co-Authored-By: dapplion <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jxs jxs Awaiting requested review from jxs jxs is a code owner

Assignees

No one assigned

Labels

bug Something isn't working v8.0.0-rc.0 Q3 2025 release for Fusaka on Holesky work-in-progress PR is a work-in-progress

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dapplion @jimmygchen