noisy NeRF advice

[jhurlbut]

Do you have advice for reducing or cleaning up these clumps of noise floating around the scenes? You can see they are mostly around the camera locations in this particular scene. Is this an artifact of bad colmap tracking? Is there a way to clean up the resulting scene other than the crop functionality? Thank you for this awesome tool!

[pwais]

It might require a bit of extra work, but if you mask out the background a bit in your images (like manually turning pixels all to white / black), that will generally help COLMAP and NeRF focus on your object. You can also try editing the aabb_scale param in your transforms_train.json file to control how much background sampling is done.

FWIW COLMAP can take mask images as input if you don't want to edit your originals (e.g. one input image and one mask png image that tells COLMAP to ignore masked stuff). instant-ngp I'm not sure, not the same format at least.

[Sazoji]

There are tools for salient object masking like U2Net that have onelineable pypi repos you can run on your images. I have been able to remove most noise with this method, other than shadows.

Lighting your scene from all angles and using a sharp macro lens is important too, at least for object capture you should handle it like a photogrammetry task. Shrinking the apature, using a tripod and increasing the exposure time/light to decrease the noise (if video don't slow down exposure time, bump up light/speed and move the camera slowly on a tripod).

I used rembg, but there's some additional config options I haven't looked into. I used flash on my camera, but that washed out the colors since I didn't expose correctly for flash. Even after background removal, you can still run colmap. I applied the removal before looking for camera transforms (would also allow for you moving the object to get more angles if you colmap after removing the background).