Skip to content

Conversation

@papr
Copy link
Contributor

@papr papr commented Nov 5, 2021

When Pupil Player opens a Pupil Invisible recording for the first time, it performs an in-place upgrade procedure due to compatibility reasons. In this process, it loads PI gaze data and generates Pupil Player compatible gaze data based on it.

Due to a flaw in the previous gaze conversion procedure, Pupil Cloud post-processed 200 Hz gaze data could be loaded multiple times or not at all.

This PR moves the prior implementation from video_capture.utils to pupil_recording.update.invisible and restructures the code flow such that 200 Hz gaze data is processed separately from the case where gaze data is loaded from the real-time recorded files.

Below a detailed description of what should be happening:

Pupil Invisible Companion records gaze data into three different sets of files. Their
names can be matched by the following regex patterns:

  • ^gaze ps[0-9]+.raw$
  • ^gaze ps[0-9]+.time$
  • ^worn ps[0-9]+.raw$

The worn data is a stream of values of either 0 or 255, indicating that the glasses
were (not) worn. Pupil Player maps these to gaze confidence values of 0.0 and 1.0
respectively.

Since all *.time files are converted to Pupil Player before this function is being
called, we match the ^gaze ps[0-9]+_timestamps.npy$ pattern on the recording files
instead. When looking for the location and worn data, the function just replaces the
necessary parts of the timestamp file names instead of performing separate regex
matches.

If the recording was successfully post-processed and downloaded from Pupil Cloud, it
will contain 200Hz-densified gaze data. This data replaces the real-time recorded
data by Pupil Invisible Companion and is stored in three files:

  • gaze_200hz.raw
  • gaze_200hz.time (or gaze_200hz_timestamps.npy if upgraded)
  • worn_200hz.raw

The worn data is a special case as it was introduced at different points in time to
Pupil Invisible Companion and Pupil Cloud. In other words, it is possible that there
is no worn data, only real-time recorded worn data, or 200 Hz worn data. The latter
is preferred. If 200 Hz gaze data is only available with real-time recorded worn
data, the latter is interpolated to 200 Hz using a k-nearest-neighbour (k=1)
approach. If no worn data is available, or the numbers of worn samples and gaze
timestamps are not consistent, Pupil Player assumes a confidence value of 1.0 for
every gaze point.

When Pupil Player opens a Pupil Invisible recording for the first time, it performs an in-place upgrade procedure due to compatibility reasons. In this process, it loads PI gaze data and generates Pupil Player compatible gaze data based on it. 

Due to a flaw in the previous gaze conversion procedure, Pupil Cloud post-processed 200 Hz gaze data could be loaded multiple times or not at all.

This PR moves the prior implementation from `video_capture.utils` to `pupil_recording.update.invisible` and restructures the code flow such that 200 Hz gaze data is processed separately from the case where gaze data is loaded from the real-time recorded files.

Below a detailed description of what should be happening:

Pupil Invisible Companion records gaze data into three different sets of files. Their 
names can be matched by the following regex patterns:
    - `^gaze ps[0-9]+.raw$`
    - `^gaze ps[0-9]+.time$`
    - `^worn ps[0-9]+.raw$`

The worn data is a stream of values of either 0 or 255, indicating that the glasses
were (not) worn. Pupil Player maps these to gaze confidence values of 0.0 and 1.0
respectively.

Since all `*.time` files are converted to Pupil Player before this function is being
called, we match the `^gaze ps[0-9]+_timestamps.npy$` pattern on the recording files
instead. When looking for the location and worn data, the function just replaces the
necessary parts of the timestamp file names instead of performing separate regex
matches.

If the recording was successfully post-processed and downloaded from Pupil Cloud, it
will contain 200Hz-densified gaze data. This data replaces the real-time recorded
data by Pupil Invisible Companion and is stored in three files:
    - `gaze_200hz.raw`
    - `gaze_200hz.time` (or `gaze_200hz_timestamps.npy` if upgraded)
    - `worn_200hz.raw`

The worn data is a special case as it was introduced at different points in time to
Pupil Invisible Companion and Pupil Cloud. In other words, it is possible that there
is no worn data, only real-time recorded worn data, or 200 Hz worn data. The latter
is preferred. If 200 Hz gaze data is only available with real-time recorded worn
data, the latter is interpolated to 200 Hz using a k-nearest-neighbour (k=1)
approach. If no worn data is available, or the numbers of worn samples and gaze
timestamps are not consistent, Pupil Player assumes a confidence value of 1.0 for
every gaze point.
@papr papr added type: bug product: Pupil Invisible Issues related to Pupil Invisible labels Nov 5, 2021
@papr papr requested review from N-M-T and romanroibu November 5, 2021 16:09
@papr papr requested a review from N-M-T November 8, 2021 14:32
@papr papr marked this pull request as ready for review November 8, 2021 14:51
@papr papr merged commit 4db40b9 into develop Nov 11, 2021
@papr papr deleted the avoid_loading_cloud_200hz_gaze_multiple_times branch November 11, 2021 07:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

product: Pupil Invisible Issues related to Pupil Invisible type: bug

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants