segment color shader

[chrisj]

Rather than transpiling the shader to javascript, I setup a basic shader to render onto an offscreen canvas, and using gl.readPixels to get the value back into javascript. glsl-transpiler assumes a node environment, rather than modify or use polyfills, this seemed like a simpler solution. I also have a prototype for calculating multiple segment colors with one render pass, outputting a pixel per segment, I noted in the code that get automaticLayerBarColors would be a great place to use that.

It doesn't have much purpose yet, I need to re-add my segmentaton property shader additions, but I wanted to get feedback on if this readPixels approach is preferable.

[chrisj]

I've added support for segment properties here: https://github.com/seung-lab/neuroglancer/tree/cj-color-shader-share

It is still rough, some slice view segmentation rendering options are left out for now, but it would be great to have some feedback on the format of the user shader so we could potentially start using this before the implementation is settled.

I will add an example link of h01 with no auth requirement in the following comment. Here is a short link requiring authentication that showcases coloring segments by their tags. https://cj-color-shader-share-dot-neuroglancer-dot-seung-lab.ue.r.appspot.com/#!middleauth+https://global.daf-apis.com/nglstate/api/v1/5748761140658176

I decided to try to get the color computation for all the various output(3d/2d/html) running on as much of the same code as possible. That means I used the approach necessary for slice view, having each draw call have access to all the segment properties. I have one texture that is a map from segment id to segment property index which is used to determine if we have segment property info for that segment as well as used as an index to read from the individual texture for each segment property. For now I have each tag on it's own texture for simplicity but it could be optimized later to use bit flags. Textures only exist if they are actively used by the user shader. My goal is for SegmentColorUserShaderManager to contain all of the coloring logic aside of output specific features such as highlightColor.

One difficulty is the fact that segment properties allow an id to be any string. With the h01 dataset for instance, there are property ids with a forward slash. Annotation properties are restricted to the regular expression /^[a-z][a-zA-Z0-9_]*$/. I wish that were the case for segment properties as well. I currently have a preprocessor to support prop("property_id") and tag("tag_name"). I have left out some error handling that would be necessary to completely support that feature.

The approach to use the same rendering logic between all three allows for the possibility of meshes that combine multiple segments, which could be used in the future for base segment coloring. It can also be used to efficiently calculate automaticLayerBarColors if we expand getShaderSegmentColor to take an array of segments and output to a canvas with a pixel per segment.

I should try to reuse code that generates the segment query property histograms for the invlerp control.
The tests I wrote for getShaderSegmentColor give us coverage on most of the color logic for mesh/skeleton and 2d segmentation. I would like to add tests for the rest as well, especially 2d. Another important issue to address is the UI crowding, particularly bad when you have a skeleton source so there are two editable use shaders within the render tab. Making the tab a scrollable element will help.

[chrisj]

Here is a public link that shows some more functionality of the segment color shader

https://cj-color-shader-share-dot-neuroglancer-dot-seung-lab.ue.r.appspot.com/#!%7B%22dimensions%22:%7B%22x%22:%5B8e-9%2C%22m%22%5D%2C%22y%22:%5B8e-9%2C%22m%22%5D%2C%22z%22:%5B3.3e-8%2C%22m%22%5D%7D%2C%22position%22:%5B358522.71875%2C86918.5%2C2660.74072265625%5D%2C%22crossSectionScale%22:115.89143046777498%2C%22projectionOrientation%22:%5B0.12615253031253815%2C0.009984897449612617%2C-0.0959339365363121%2C0.9873107671737671%5D%2C%22projectionScale%22:95374.09332403584%2C%22layers%22:%5B%7B%22type%22:%22image%22%2C%22source%22:%22precomputed://gs://h01-release/data/20210601/4nm_raw%22%2C%22tab%22:%22rendering%22%2C%22shaderControls%22:%7B%22normalized%22:%7B%22range%22:%5B147%2C255%5D%2C%22window%22:%5B120%2C255%5D%7D%7D%2C%22name%22:%224nm%20EM%22%7D%2C%7B%22type%22:%22segmentation%22%2C%22source%22:%7B%22url%22:%22precomputed://gs://h01-release/data/20210601/c3%22%2C%22subsources%22:%7B%22default%22:true%2C%22bounds%22:true%2C%22properties%22:true%2C%22skeletons%22:true%7D%2C%22enableDefaultSubsources%22:false%7D%2C%22tab%22:%22rendering%22%2C%22selectedAlpha%22:1%2C%22notSelectedAlpha%22:1%2C%22skeletonRendering%22:%7B%22lineWidth2d%22:10%7D%2C%22meshRenderScale%22:2%2C%22segmentColorShader%22:%22#uicontrol%20invlerp%20normalized%5Cnvec3%20segmentColor%28vec3%20color%2C%20bool%20hasProperties%29%20%7B%5Cn%20%20if%20%28%21hasProperties%29%20%7B%5Cn%20%20%20%20return%20vec3%280.5%2C%200.5%2C%200.5%29%3B%5Cn%20%20%7D%5Cn%20%20%5Cn%20%20if%20%28tag%28%5C%22L1%5C%22%29%29%20%7B%5Cn%20%20%20%20return%20vec3%281.0%2C%200.0%2C%200.0%29%3B%5Cn%20%20%7D%5Cn%20%20%5Cn%20%20return%20colormapCubehelix%28normalized%28%29%29%3B%5Cn%7D%5Cn%22%2C%22segments%22:%5B%2237029363761%22%2C%2228469485117%22%2C%2228003801168%22%2C%2237145660784%22%2C%2237685007129%22%5D%2C%22colorSeed%22:3871475997%2C%22shaderControls%22:%7B%22normalized%22:%7B%22range%22:%5B143%2C1030%5D%2C%22window%22:%5B0%2C1587%5D%2C%22property%22:%22NSIi%22%7D%7D%2C%22name%22:%22c3%20segmentation%22%7D%5D%2C%22showAxisLines%22:false%2C%22showSlices%22:false%2C%22selectedLayer%22:%7B%22col%22:1%2C%22size%22:444%2C%22visible%22:true%2C%22layer%22:%22c3%20segmentation%22%7D%2C%22layout%22:%22xy-3d%22%2C%22selection%22:%7B%22row%22:0%2C%22visible%22:false%7D%7D

[chrisj]

I wrote some documentation here that explains the format of the user shader:
https://github.com/seung-lab/neuroglancer/blob/cj-color-shader-share/src/layer/segmentation/rendering.md

[chrisj]

I didn't think you would have time to review the code due to the voxel painting PR but I opened up this PR here #871 so you can more easily take a look at it, it has some test deployment code