When orientation_method=pca, transformation_matrix can be incorrect #3417
Description
Hi NeRFStudio's contributors,
Thanks for your work!
I notice that when setting ColmapDataParserConfig.orientation_method=pca, a misalignment between initial pointcloud and camera pose can happen due to the incorrect transformation_matrix
In my example, I rendered the COLMAP initialized point cloud in step 0 but found that it was misaligned with the groundtruth image. Specifically, the image is vertically flipped.
I found that the issue relates to these lines
nerfstudio/nerfstudio/cameras/camera_utils.py
Lines 577 to 580 in f86dbe6
'oriented_poses.mean(dim=0)[2, 1] < 0: ' seems to mean where the Camera's y-axis (up-direction) is projected to the world's z-axis. So the up direction in the image should be aligned with +z axis in the world. If not, the code here negates the y and z axes in the camera's coordinate. However, this leads to the flipping renderings, misaligned with the label images which are not flipped.
If I turn off this negation, the step=0 renderings become aligned again.
To address this issue, we need to apply the transformation to the world coordinate (Flip the objects in y and z axes) instead of the camera system. For example
if oriented_poses.mean(dim=0)[2, 1] < 0:
#oriented_poses[:, 1:3] = -1 * oriented_poses[:, 1:3]
transform_plus = torch.eye(3)
transform_plus[1, 1] = -1
transform_plus[2, 2] = -1
oriented_poses = transform_plus @ oriented_poses
transform = transform_plus @ transform
This will lead to much better final results due to the correct initial point clouds.
Great thanks :)