V4L2 VideoCapture always uses V4L2_PIX_FMT_BGR24

[jampekka]

Even though there's logic to prefer MJPG or YUYV if available, VideoCapture always uses V4L2_PIX_FMT_BGR24 due to line:

self.v4l2_format.fmt.pix.pixelformat = V4L2_PIX_FMT_BGR24

Also, later on in the read-method this format is assumed for the buffer.

The jpeg compression really seems to be needed for two cameras to run in one USB controller at any resolution, even with the 0x80 quirk. Doing the JPEG handling here seems to save us from patching the uvcvideo-module.

I can get MJPG working with a following quick hack, although it'll probably break other formats:

--- a/pupil_src/shared_modules/uvc_capture/linux_video/v4l2_capture/__init__.py
+++ b/pupil_src/shared_modules/uvc_capture/linux_video/v4l2_capture/__init__.py
@@ -15,9 +15,12 @@ This file contains bindings to a webcam capture module that works with v4l2 and
 from ctypes import *
 from numpy.ctypeslib import ndpointer
 import numpy as np
+from PIL import Image
+from cStringIO import StringIO
 import os,sys
 #logging
 import logging
+
 logger = logging.getLogger(__name__)

 class CameraCaptureError(Exception):
@@ -216,7 +219,8 @@ class VideoCapture(object):
         self.v4l2_format.type = V4L2_BUF_TYPE_VIDEO_CAPTURE
         self.v4l2_format.fmt.pix.width       = size[0]
         self.v4l2_format.fmt.pix.height      = size[1]
-        self.v4l2_format.fmt.pix.pixelformat = V4L2_PIX_FMT_BGR24
+        #self.v4l2_format.fmt.pix.pixelformat = V4L2_PIX_FMT_BGR24
+        self.v4l2_format.fmt.pix.pixelformat = v4l2_fourcc(*self.prefered_format)
         self.v4l2_format.fmt.pix.field       = V4L2_FIELD_ANY
         if (-1 == dll.xioctl(self.device, VIDIOC_S_FMT, byref(self.v4l2_format))):
             self._close()
@@ -376,8 +380,10 @@ class VideoCapture(object):
                 # if ( self.ts > timestamp) or 1:
                 #     print "%s %s" %(self.src_str, self.get_time_monotonic()-timestamp)
                 buf_ptr = cast(buf_ptr,POINTER(c_uint8*buf.bytesused))
+
                 img = np.frombuffer(buf_ptr.contents,c_uint8)
-                img.shape = (self.v4l2_format.fmt.pix.height,self.v4l2_format.fmt.pix.width,3)
+                #img.shape = (self.v4l2_format.fmt.pix.height,self.v4l2_format.fmt.pix.width,3)
+               img = np.array(Image.open(StringIO(img.data)))
                 return Frame(timestamp, img)
             else:
                 logger.warning("Frame corrupted skipping it")

[jampekka]

Oops, that produces RGB but Pupil expects BGR, so the colors are messed up.

Hack on top of hack (after we have the img from PIL):

img = img[:,:,::-1]

[mkassner]

Hi,

the backend uses lib4vl2 a slim wrapper around v4l2-capture that does
format conversions for us. Unfortunately it will select the transport
format from camera to host without asking us.

Im currently rewriting the v4l2 backend to give full access to mjpg, yuv
and rgb buffers as well as being able specify the transport format. I'll
push a branch for this soon.

With this im planning to control the format that we grab from the camera
and have a 'frame' object that will hold the buffers and decompress and
convert colorspaces on the fly (and cache the results). Thus giving more
low level control without adding a lot of complexity to the user.

There are some patches floating aground the v4l2 dev mailing list that
enable c930e h264 capture. If we get this to work we can add this to the
list of accessible image formats.

best,

Moritz

On Fri, Sep 12, 2014 at 10:36 PM, Jami Pekkanen notifications@github.com
wrote:

Oops, that produces RGB but Pupil expects BGR, so the colors are messed up.

Hack on top of hack (after we have the img from PIL):

img = img[:,:,::-1]

—
Reply to this email directly or view it on GitHub
#18 (comment).

[jampekka]

I noticed there's also gst_capture branch, which I assume refers to gstreamer. Is this backend an abandoned project?

Gstreamer would probably simplify some stuff, such as non V4L2 sources a lot. Was there some issues with it that prevented such a backend?

[mkassner]

We abandoned it due to bad performance. CPU load was doubled.

Since gst is under heavy development this could have changed again...

M
On Sep 12, 2014 6:12 PM, "Jami Pekkanen" notifications@github.com wrote:

I noticed there's also gst_capture branch, which I assume refers to
gstreamer. Is this backend an abandoned project?

Gstreamer would probably simplify some stuff, such as non V4L2 sources a
lot. Was there some issues with it that prevented such a backend?

—
Reply to this email directly or view it on GitHub
#18 (comment).

[mkassner]

Just a quick update. I m working on the binding here: http://github.com/pupil-labs/pyv4l2
Tomorrow I should have it ready for first integration in Pupil and make the branch I mentioned earlier.

[mkassner]

please try out: https://github.com/pupil-labs/pupil/tree/pyv4l2