Too much expensive keypoint post-processing

[filipetrocadoferreira]

I think you are spending too much time in keypoint post-processing without any need.

def heatmaps_to_keypoints(maps, rois):
    """Extract predicted keypoint locations from heatmaps. Output has shape
    (#rois, 4, #keypoints) with the 4 rows corresponding to (x, y, logit, prob)
    for each keypoint.
    """
    # This function converts a discrete image coordinate in a HEATMAP_SIZE x
    # HEATMAP_SIZE image to a continuous keypoint coordinate. We maintain
    # consistency with keypoints_to_heatmap_labels by using the conversion from
    # Heckbert 1990: c = d + 0.5, where d is a discrete coordinate and c is a
    # continuous coordinate.
    offset_x = rois[:, 0]
    offset_y = rois[:, 1]

    widths = rois[:, 2] - rois[:, 0]
    heights = rois[:, 3] - rois[:, 1]
    widths = np.maximum(widths, 1)
    heights = np.maximum(heights, 1)
    widths_ceil = np.ceil(widths)
    heights_ceil = np.ceil(heights)

    # NCHW to NHWC for use with OpenCV
    maps = np.transpose(maps, [0, 2, 3, 1])
    min_size = cfg.KRCNN.INFERENCE_MIN_SIZE
    xy_preds = np.zeros(
        (len(rois), 4, cfg.KRCNN.NUM_KEYPOINTS), dtype=np.float32)
    for i in range(len(rois)):
        if min_size > 0:
            roi_map_width = int(np.maximum(widths_ceil[i], min_size))
            roi_map_height = int(np.maximum(heights_ceil[i], min_size))
        else:
            roi_map_width = widths_ceil[i]
            roi_map_height = heights_ceil[i]
        width_correction = widths[i] / roi_map_width
        height_correction = heights[i] / roi_map_height

        #roi_map = cv2.resize(
        #    maps[i], (roi_map_width, roi_map_height),
        #    interpolation=cv2.INTER_NEAREST)
        # Bring back to CHW
        roi_map = maps[i].copy()
        roi_map = np.transpose(roi_map, [2, 0, 1])
        roi_map_probs = scores_to_probs(roi_map.copy())

        w = roi_map.shape[2]
        for k in range(cfg.KRCNN.NUM_KEYPOINTS):
            pos = roi_map[k, :, :].argmax()

            x_int = pos % w
            y_int = (pos - x_int) // w
            assert (roi_map_probs[k, y_int, x_int] ==
                    roi_map_probs[k, :, :].max())
            x = (x_int*float(roi_map_width/w) + 0.5) * width_correction
            y = (y_int*float(roi_map_height/w) + 0.5) * height_correction
            xy_preds[i, 0, k] = x + offset_x[i]
            xy_preds[i, 1, k] = y + offset_y[i]
            xy_preds[i, 2, k] = roi_map[k, y_int, x_int]
            xy_preds[i, 3, k] = roi_map_probs[k, y_int, x_int]

    return xy_preds

This achieve same results with less computation. Any explanation for the method implemented?

[rbgirshick]

The use of cv2.resize is likely a leftover from having initially experimented with bicubic interpolation (which is even more expensive :p). We'll look into replacing this along the lines you suggest. Feel free to test and submit a PR. (In general, our post processing code for masks and keypoints is not optimized at all.)

[filipetrocadoferreira]

I just remembered from your data distillation paper that you can be interested in probability maps in original image coordinates, which is not really the usual because we're only interested in the keypoint position and confidence and for that you don't need to resize the entire mask

[rbgirshick]

To correct my previous comment, we are actually using bicubic interpolation by default. Bicubic resizing followed by argmax has the potential recover sub-pixel predictions because it is non-linear, which may improve accuracy. However, we have not recently compared bicubic interpolation to a simpler and faster baseline like you suggest; it might be fine w.r.t. keypoint AP.