Feature/voxel annotation

[briossant]

This Draft Pull Request introduces an interactive voxel annotation feature, allowing users to perform manual segmentation by painting directly onto volumetric layers. This implementation is based on the proposal in Issue #851 and incorporates the feedback from @jbms.

Here is a live demo to try the feature, watch out, there is persistent storage, so your annotations will be saved and will override the ones already present: OPEN DEMO VIEWER

Key Changes & Architectural Overview

Following the discussion, this implementation has been significantly revised from the initial prototype:

1. Integration with Existing Layers: Instead of a new vox layer type, the voxel editing functionality is now integrated directly into ImageUserLayer and SegmentationUserLayer via a UserLayerWithVoxelEditingMixin. This mixin adds a new "Draw" tab in the UI.

2. New Tool System: The Brush and Flood Fill tools are implemented as toggleable LayerTools, while the Picker tool is a one-shot tool. All integrate with Neuroglancer's new tool system. The drawing action is bound to Ctrl + Left Click.

3. Optimistic Preview for Compressed Chunks: To provide immediate visual feedback and solve the performance problem with compressed chunks, edits are now rendered through an optimistic preview layer.

• When a user paints, edits are first applied to an InMemoryVolumeChunkSource.
• This preview source is rendered by a second instance of the layer's primary RenderLayer (e.g., ImageRenderLayer or SegmentationRenderLayer). This ensures the preview perfectly matches the user's existing shader and display settings.
• The base data chunk is not modified on the frontend, avoiding the need to decompress/recompress it.

Data-flow

sequenceDiagram
participant User
participant Tool as VoxelBrushTool
participant ControllerFE as VoxelEditController (FE)
participant EditSourceFE as OverlayChunkSource (FE)
participant BaseSourceFE as VolumeChunkSource (FE)
participant ControllerBE as VoxelEditController (BE)
participant BaseSourceBE as VolumeChunkSource (BE)

    User->>Tool: Mouse Down/Drag
    Tool->>ControllerFE: paintBrushWithShape(mouse, ...)
    ControllerFE->>ControllerFE: Calculates affected voxels and chunks

    ControllerFE->>EditSourceFE: applyLocalEdits(chunkKeys, ...)
    activate EditSourceFE
    EditSourceFE->>EditSourceFE: Modifies its own in-memory chunk data
    note over EditSourceFE: This chunk's texture is re-uploaded to the GPU
    deactivate EditSourceFE

    ControllerFE->>ControllerBE: commitEdits(edits, ...) [RPC]

    activate ControllerBE
    ControllerBE->>ControllerBE: Debounces and batches edits
    ControllerBE->>BaseSourceBE: applyEdits(chunkKeys, ...)
    activate BaseSourceBE
    BaseSourceBE-->>ControllerBE: Returns VoxelChange (for undo stack)
    deactivate BaseSourceBE
    ControllerBE->>ControllerFE: callChunkReload(chunkKeys) [RPC]
    activate ControllerFE
    ControllerFE->>BaseSourceFE: invalidateChunks(chunkKeys)
    note over BaseSourceFE: BaseSourceFE re-fetches chunk with the now-permanent edit.
    ControllerFE->>EditSourceFE: clearOptimisticChunk(chunkKeys)
    deactivate ControllerFE

    ControllerBE->>ControllerBE: Pushes change to Undo Stack & enqueues for downsampling
    deactivate ControllerBE

    loop Downsampling & Reload Cascade
        ControllerBE->>ControllerBE: downsampleStep(chunkKeys)
        ControllerBE->>ControllerFE: callChunkReload(chunkKeys) [RPC]
        activate ControllerFE
        ControllerFE->>BaseSourceFE: invalidateChunks(chunkKeys)
        note over BaseSourceFE: BaseSourceFE re-fetches chunk with the now-permanent edit.
        ControllerFE->>EditSourceFE: clearOptimisticChunk(chunkKeys)
        deactivate ControllerFE
    end

Loading

4. Writable source selection Aside every activated volume sub-sources of a writable datasource, an additional checkbox lets the user mark the sub-source as writable, then neuroglancer will try to write in it.

5. Dataset creation To complete Neuroglancer's writing capabilities, a dataset metadata creation/initialization feature was introduced.

The workflow is triggered when a user provides a URL to a data source that does not resolve:

Neuroglancer recognizes the potential intent to create a new dataset and prompts the user:

Finally, the user is able to access dataset creation form:

Data sources & Kvstores

Currently, there is a very limited set of supported data sources and kvstores, which are:

• datasources:
  • zarr v2 and v3 with codecs: raw, gzip, blosc
• kvstores:
  • s3+http(s): can be used with a local s3 bucket (e.g. minio) or with anonymous s3 urls
  • opfs: in-browser storage, also used for local development at some point, the relevancy can be discussed.
  • ssa+https: a kvstore linked to an in development project, which is a stateless (thanks to OAuth 2.0) worker providing signed urls to read/write in s3 stores

Limitations

• Only the rank 3 volumes are supported
• Float32 dataset are not supported
• Unaligned chunk hierarchy/resolutions (e.g. child chunks that may have multiple parents) is not supported

Open Questions & Future Work

This PR focuses on establishing the core architecture. Several larger topics from the original discussion are noted here as future work:

• Efficient Low-Resolution Drawing: As discussed, efficient, multi-resolution drawing with upsampling is a complex challenge that requires a new data format.
• 3D Drawing Tools: As suggested by @fcollman, 3D-specific tools like interpolation between slices are out of scope for this initial PR but could be a valuable direction for future work.

---

Checklist

• Completed the todo list found in src/voxel_annotations/TODOs.md
• [ ] Added support to more (every?) datasources and kvstores
• Signed the CLA.

---

Edits

• updated zarr support
• added "5. Dataset creation" section
• strikethrough what's no longer part of this PR
• added the Limitations section
• added live demo

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[jbms]

Can you complete the CLA?

[jbms]

The brush hover outline (circle where the mouse pointer is) seems to go away in some cases when changing the zoom level.

[briossant]

I need to rewrite history, my commits are signed with the wrong email, I will open a new PR I made a mistake with this force push nevermind its fixed

[briossant]

Hello,

I setup a live demo to let everyone easily try the feature and give feedback on UI, ergonomics, and bugs.

Watch out: There is persistent storage, so your annotations will be saved and will override the ones already present.

OPEN DEMO VIEWER

[chrisj]

@briossant thanks for setting that up! Some UX feedback.

These tools aren't showing the keybind:

It isn't obvious without a large brush size when the brush tool is active. Annotation layers currently indicate an active tool with this entry in the top bar: (the "annotate point") though I don't think that is obvious enough, particularly when you have multiple tools keybind which fill up the top row.

I would suggest trying a bottom bar and you could populate it with brush settings, along with keybinds that only become active while the brush tool is active. This is what our edit toolbars looks like.

[briossant]

@briossant thanks for setting that up! Some UX feedback.

These tools aren't showing the keybind:

It isn't obvious without a large brush size when the brush tool is active. Annotation layers currently indicate an active tool with this entry in the top bar: (the "annotate point") though I don't think that is obvious enough, particularly when you have multiple tools keybind which fill up the top row.

I would suggest trying a bottom bar and you could populate it with brush settings, along with keybinds that only become active while the brush tool is active. This is what our edit toolbars looks like.

Appreciate the feedback! I fixed the missing shortcuts ; and the bottom bar is a good idea, it would cleanup the tab ; do you know where I could find the implementation for the one you shared me?

[chrisj]

Appreciate the feedback! I fixed the missing shortcuts ; and the bottom bar is a good idea, it would cleanup the tab ; do you know where I could find the implementation for the one you shared me?

Check out the activate functions here:

neuroglancer/src/datasource/graphene/frontend.ts

Line 2388 in f0ffc01

activate(activation: ToolActivation<this>) {

neuroglancer/src/datasource/graphene/frontend.ts

Line 2079 in f0ffc01

activate(activation: ToolActivation<this>) {

Also check out MULTICUT_SEGMENTS_INPUT_EVENT_MAP to see dynamic (when tool is active) keybinds

[briossant]

I've updated the demo to the latest commit, this includes:

• addition of a value-specific eraser mode
• a fix of the "double-opacity" artifact and the missing preview for the eraser in the segmentation layer, I used the sentinel strategy, as explained in previous comments. You can see the intentionally added artifact using the paint value 18446744073709551614.
• addition of a bottom bar for the brush and flood fill, with their respective settings moved to it. I did not change the settings binding strategy; as suggested by @chrisj, pre-defined bindings (e.g. shift+wheel for the brush size instead of shift+key+wheel for example) that only become active along the tool would be beneficial for ergonomics, at the cost of configurability; if this is the path we want to take, I would like some feedback on what bindings should we choose, my initial proposal is: shift+wheel for the brush size and the max flood fill area, shift+keys to cycle brush shapes.

[chrisj]

@briossant It feels better!

These controls are draggable right now which prevents dragging the slider

Have you thought about having ctrl+mousedown+shift trigger erasing and then have the erase setting just be a toggle between everything and selected?

[briossant]

@briossant It feels better!

These controls are draggable right now which prevents dragging the slider

Have you thought about having ctrl+mousedown+shift trigger erasing and then have the erase setting just be a toggle between everything and selected?

Yeah erasing feels better like this, I've updated the demo with the refactored erasing.

[briossant]

Hello,

I've cleared my current todo list and am standing by for review, or further feedback.

One question regarding scope: @jbms mentioned keeping new datasources and kvstores separate, but would you recommand to add write support for existing datasources (e.g. precomputed and n5) in this PR?

[chrisj]

I would remove this then because when we later add it, it will be more clear that it that it was a requirement for that functionality.

[chrisj]

I think it is best to use a block statement with if statements that aren't single line

[chrisj]

From some performance testing of the paint tool, it seems you can get a significant speedup for the UI thread if you only check getEnsuredValue if filterValue !== undefined and ignore the v === value check. Though I believe that would cause some unnecessary writes so perhaps that logic could be moved to the backend.

I'm wondering how much of this logic could be moved to the backend, if the only thing the frontend did when receiving input was to forward that user input to the backend and have the backend do all the voxel calculations.

[briossant]

Yes, the v === value check can be removed. This would indeed speed up the function, and I can add the check back in the backend to avoid unnecessary writes. As for moving the whole logic to the backend: for the brush, I believe this would make the preview delay longer, resulting in a less responsive user experience; but for the flood fill it has less downside as the preview is already slow, and it would also fix the slowdown on big regions.

[chrisj]

Performing a large sphere brush stroke is the main performance issue I've noticed