Pupil Invisible support, Pupil Player Format 2.0, Variable-framerate video support #1623
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fix bad class logic: - BrokenStream should inherit from Decoder - Decoder method signatures did not match Fix bad naming: - Decoder.next_frame returns a frame iterator - File_Source.next_frame holds a frame iterator
Before the timestamps in the video objects were still uncleaned and thus could not be matched appropriately.
File_Source.shape was only used for generating FakeFrames and was filled by unnecessarily iterating over the first frames of the video. Replaced with File_Source.frame_size, which reads the format directly from the decoder interface.
Timeline will convert lines to float32, but with PI we have unix epoch timestamps. While they are in range of float32, the precision suffers greatly and will be in the order of days! Thus by subtracting t0, we can shift the timestamps to a more reasonable range and thus increase float32 precision. NOTE: Currently this will have no effect because of a bug with PI recordings from the Android side!
Removed the old way of calculating the pts from frame index and instead always calculate pts from timestamp.
When exporting with AV_Writer it is much easier to have variable pts if the Frames coming from the file-backend already have that information. Luckily we know the pts of the frames when getting them, so I just attached that. Needed to adjust FakeFrames as well and did a little bit of cleanup.
I kept the old classes and put an underscore _ in front of them in order to keep the git diff cleaner. Otherwise git would try to match all kind of hunks and produce complete chaos. Summary of changes: 1. Instead of two separate but overlapping classes (AV_Writer and JPEG_Writer) we now have an abstract base class AV_Writer and two specializations, MPEG_Writer and JPEG_Writer. This makes more sense, as the JPEG_Writer also uses pyAV and the previous AV_Writer was only writing MPEG. 2. Extracting common code was challenging, as the data flow was very different between AV_Writer and JPEG_Writer, although both classes did inherently similar things. The main point of generalization was to have an abstract method encode_frame(), which receives a frame-object and returns a number of Packets. This is the main difference between both writers and lining that out made everything a lot clearer. 3. Audio writing functionality was apparently hacked into AV_Writer at some point, but was quite different from the rest of the flow. I created a child class of MPEFG_Writer, called MPEG_Audio_Writer, which wraps additional audio encoding to separate audio from video. Currently this is only used for world export. 4. All dependecy on the FPS has been removed from both writers, as this was only used to write fixed-rate PTS. 5. The writers now have two modes for PTS, controlled by the __init__ parameter frame_pts_timebase: * If no timebase is set, frame-pts will be calculated from their timestamps. * If a timebase is set, frames are expected to carry their own pts. The output stream will be created with the same timebase.
This goes together with the previous commit, but the git history was cleaner when split into two commits.
This was disabled because of a bug in pyAV where the av.VideoFrame does not allow writing dts.
Pi timestamp support
# Conflicts: # pupil_src/shared_modules/av_writer.py
Pupil recording pi fixes
PupilRecording old-style fixes
PupilRecording new-style fixes
Recording info version safety
Recording info mobile fixes
pfaion
approved these changes
Sep 10, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pupil Invisible support
Open Pupil Invisible Companion recordings in Pupil Player.
Variable-framerate video support
Instead of writing a fixed timing into the the video files, Pupil Capture and Pupil Invisible Companion keep the original timing of the video frames. This allows correct playback of intermediate recordings, i.e. videos that have not been exported by Pupil Player.
Pupil Player Format 2.0
A new and clean recording format for Pupil Capture/Player. Consistent recording updates and logic for backwards/forwards compatibility.