Bug: https URL not working with opencv-python

[abhiTronix]

Bug summary:

opencv-contrib-python python library's VideoCapture class fails to parse https URL received from pafy/youtube-dl. But strangely this bug doesn't exist on same/different machine with OpenCV installed from scratch.

Affected users:

Users with OpenCV installed with opencv-contrib-python or opencv-python python library from PyPi

Non-Affected users:

Users with OpenCV installed from scratch/source-code and not from PyPi (Tested with OpenCV 4.1.0-dev & 3.4.5)

Steps to reproduce

• operating system - Ubuntu 18.04.1 bionic
• architecture x64
• opencv-python 4.1.0.25
• youtube-dl 2019.5.20
• pafy 0.5.4
• python_version 3.6/2.7/3.5 tested

Test-Code:

You can copy url directly to cv2.VideoCapture class and this result in same Bug.

import cv2
import pafy

video = pafy.new("dQw4w9WgXcQ")
object = video.getbestvideo()
url = object.url
print(url) #means pafy and youtube-dl working correctly

stream = cv2.VideoCapture(url)

# infinite loop
while True:
	(grabbed, frame) = stream.read()
	# read frames

	# check if frame empty
	if not grabbed:
		print('opencv is buggy')
		break

	cv2.imshow("Output Frame", frame)
	# Show output window

	key = cv2.waitKey(1) & 0xFF
	# check for 'q' key-press
	if key == ord("q"):
		#if 'q' key-pressed break out
		break

cv2.destroyAllWindows()
# close output window

stream.release()
# safely close video stream.

This code exit instantly with no error on opencv-python 4.1.0.25 since no frame grabbed BUT instead this algorithm works when OpenCV installed from scratch and returns valid frames.

[abhiTronix]

Update: This may be something to do with G-Streamer not supported on opencv-python.

[leehanchung]

@abhiTronix cv2 installed using pip install opencv-python fail to load any http videos but works when I installed it from scratch. The .getBuildInformation() printout of my working cv2 installation has no GStreamer.

Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> print(cv2.getBuildInformation())

General configuration for OpenCV 4.1.0 =====================================
  Version control:               unknown

  Extra modules:
    Location (extra):            /home/han/opencv_contrib-4.0.0/modules
    Version control (extra):     unknown

  Platform:
    Timestamp:                   2019-05-21T17:43:25Z
    Host:                        Linux 4.15.0-50-generic x86_64
    CMake:                       3.10.2
    CMake generator:             Unix Makefiles
    CMake build tool:            /usr/bin/make
    Configuration:               RELEASE

  CPU/HW features:
    Baseline:                    SSE SSE2 SSE3
      requested:                 SSE3
    Dispatched code generation:  SSE4_1 SSE4_2 FP16 AVX AVX2 AVX512_SKX
      requested:                 SSE4_1 SSE4_2 AVX FP16 AVX2 AVX512_SKX
      SSE4_1 (15 files):         + SSSE3 SSE4_1
      SSE4_2 (2 files):          + SSSE3 SSE4_1 POPCNT SSE4_2
      FP16 (1 files):            + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 AVX
      AVX (5 files):             + SSSE3 SSE4_1 POPCNT SSE4_2 AVX
      AVX2 (29 files):           + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2
      AVX512_SKX (2 files):      + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2 AVX_512F AVX512_SKX

  C/C++:
    Built as dynamic libs?:      YES
    C++ Compiler:                /usr/bin/c++  (ver 7.4.0)
    C++ flags (Release):         -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections  -msse -msse2 -msse3 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG
    C++ flags (Debug):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections  -msse -msse2 -msse3 -fvisibility=hidden -fvisibility-inlines-hidden -g  -O0 -DDEBUG -D_DEBUG
    C Compiler:                  /usr/bin/cc
    C flags (Release):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections  -msse -msse2 -msse3 -fvisibility=hidden -O3 -DNDEBUG  -DNDEBUG
    C flags (Debug):             -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections  -msse -msse2 -msse3 -fvisibility=hidden -g  -O0 -DDEBUG -D_DEBUG
    Linker flags (Release):      -Wl,--gc-sections  
    Linker flags (Debug):        -Wl,--gc-sections  
    ccache:                      NO
    Precompiled headers:         YES
    Extra dependencies:          dl m pthread rt
    3rdparty dependencies:

  OpenCV modules:
    To be built:                 aruco bgsegm bioinspired calib3d ccalib core datasets dnn dnn_objdetect dpm face features2d flann freetype fuzzy gapi hdf hfs highgui img_hash imgcodecs imgproc line_descriptor ml objdetect optflow phase_unwrapping photo plot python3 reg rgbd saliency shape stereo stitching structured_light superres surface_matching text tracking ts video videoio videostab xfeatures2d ximgproc xobjdetect xphoto
    Disabled:                    world
    Disabled by dependency:      -
    Unavailable:                 cnn_3dobj cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev cvv java js matlab ovis python2 sfm viz
    Applications:                tests perf_tests apps
    Documentation:               NO
    Non-free algorithms:         YES

  GUI: 
    GTK+:                        YES (ver 3.22.30)
      GThread :                  YES (ver 2.56.4)
      GtkGlExt:                  NO
    VTK support:                 NO

  Media I/O: 
    ZLib:                        /home/han/miniconda3/lib/libz.so (ver 1.2.11)
    JPEG:                        /home/han/miniconda3/lib/libjpeg.so (ver 80)
    WEBP:                        build (ver encoder: 0x020e)
    PNG:                         /home/han/miniconda3/lib/libpng.so (ver 1.6.34)
    TIFF:                        /home/han/miniconda3/lib/libtiff.so (ver 42 / 4.0.9)
    JPEG 2000:                   build (ver 1.900.1)
    OpenEXR:                     /usr/lib/x86_64-linux-gnu/libImath.so /usr/lib/x86_64-linux-gnu/libIlmImf.so /usr/lib/x86_64-linux-gnu/libIex.so /usr/lib/x86_64-linux-gnu/libHalf.so /usr/lib/x86_64-linux-gnu/libIlmThread.so (ver 2.2.0)
    HDR:                         YES
    SUNRASTER:                   YES
    PXM:                         YES
    PFM:                         YES

  Video I/O:
    DC1394:                      YES (2.2.5)
    FFMPEG:                      YES
      avcodec:                   YES (57.107.100)
      avformat:                  YES (57.83.100)
      avutil:                    YES (55.78.100)
      swscale:                   YES (4.8.100)
      avresample:                YES (3.7.0)
    GStreamer:                   NO
    v4l/v4l2:                    YES (linux/videodev2.h)

  Parallel framework:            pthreads

  Trace:                         YES (with Intel ITT)

  Other third-party libraries:
    Intel IPP:                   2019.0.0 Gold [2019.0.0]
           at:                   /home/han/opencv-4.1.0/build/3rdparty/ippicv/ippicv_lnx/icv
    Intel IPP IW:                sources (2019.0.0)
              at:                /home/han/opencv-4.1.0/build/3rdparty/ippicv/ippicv_lnx/iw
    Lapack:                      NO
    Eigen:                       NO
    Custom HAL:                  NO
    Protobuf:                    build (3.5.1)

  OpenCL:                        YES (no extra features)
    Include path:                /home/han/opencv-4.1.0/3rdparty/include/opencl/1.2
    Link libraries:              Dynamic load

  Python 3:
    Interpreter:                 /home/han/.pyenv/shims/python3 (ver 3.6.8)
    Libraries:                   /home/han/miniconda3/lib/libpython3.6m.so (ver 3.6.8)
    numpy:                       /home/han/miniconda3/lib/python3.6/site-packages/numpy/core/include (ver 1.16.3)
    install path:                lib/python3.6/site-packages/cv2/python-3.6

  Python (for build):            /home/han/.pyenv/shims/python3

  Java:                          
    ant:                         NO
    JNI:                         NO
    Java wrappers:               NO
    Java tests:                  NO

  Install to:                    /usr/local
-----------------------------------------------------------------

[skvark]

The issue is not GStreamer but FFmpeg. It's most likely compiled without https support or some other build flag is missing and therefore it cannot open Youtube video streams.

Your script works under Windows which confirms that this is related to FFmpeg build time configuration. To see the FFmpeg build flags check out the Dockerfiles: https://github.com/skvark/opencv-python/blob/master/docker/Dockerfile_x86_64

[abhiTronix]

@skvark Thanks for this valuable info. Problem is indeed with FFmpeg, and I've discovered the solution:

It's most likely compiled without https support or some other build flag is missing and therefore it cannot open Youtube video streams.

Yes, FFmpeg is compiled with no support for openssl that enables support for https and other protocols in FFmpeg, missing in docker file you provided. It can solved with following steps:

1. Install libssl development files: debian: sudo apt install libssl-dev, or on redhat: yum install -y openssl-devel

2. Add --enable-openssl --enable-nonfree --enable-protocol= file,ftp,http,https,httpproxy,hls,mmsh,mmst,pipe,rtmp,rtmps,rtmpt,rtmpts,rtp,sctp,srtp,tcp,udp flags to the FFmpeg configure.

Please take a look here:
https://askubuntu.com/questions/650577/how-to-compile-ffmpeg-with-https-support
https://stackoverflow.com/questions/43565190/couldnt-open-https-stream-protocol-not-found-ffmpeg-with-openssl

[skvark]

Thanks. Note that the flag --enable-nonfree cannot be added because the resulting build must licensed under LGPL (--enable-nonfree makes it GPL). I think OpenSSL has a permissive license, but it must be checked that GPL'ed components are not included.

[abhiTronix]

Thanks, @skvark for the update,

Note that the flag --enable-nonfree cannot be added because the resulting build must licensed under LGPL (--enable-nonfree makes it GPL). I think OpenSSL has a permissive license, but it must be checked that GPL'ed components are not included.

Why GPL License is not good? Please test if FFmpeg works with openSSL without --enable-nonfree flag.

Also when will be these changes will be integrated into opencv-python binaries, are you working on a new PR or You need my help on this?

[skvark]

If there are GPL'ed components I would have to re-license the whole package under GPL license and any package using opencv-python, respectively, would have to be licensed under GPL. This would also mean that this package or any package which uses opencv-python could not be used for commercial purposes / proprietary applications.

I have to re-buid the Docker images and check if this problem is present also on macOS. I cannot give any timetable when this will happen. Please note that this is not a bug. This is a feature request. If you wish to speed up the process, you can try to enable the SSL support in FFmpeg builds locally in the Docker images, build a opencv-python wheel and test if the solution works.

[abhiTronix]

If there are GPL'ed components I would have to re-license the whole package under GPL license and any package using opencv-python, respectively, would have to be licensed under GPL. This would also mean that this package or any package which uses opencv-python could not be used for commercial purposes / proprietary applications.

I have to re-buid the Docker images and check if this problem is present also on macOS. I cannot give any timetable when this will happen. Please note that this is not a bug. This is a feature request. If you wish to speed up the process, you can try to enable the SSL support in FFmpeg builds locally in the Docker images, build a opencv-python wheel and test if the solution works.

Thank you for this valuable information. I don't want this feature just right now, but my vidgear python library relies on opencv-python for its core functions and for features like Live YouTube streaming, Thereby I just need heads up from your end to plan next update for vidgear accordingly. So I'll wait for this change and also, I've linked this issue to parent issue on vidgear. Again thanks for letting me know.

[abhiTronix]

@skvark Hey, Take a look at this issue: opencv/opencv#14617, Is this related?

[mshabunin]

@abhiTronix , it is not related, that issue is caused by very long string incorrectly processed by built-in CAP_IMAGES backend.

[native-api]

Note that the flag --enable-nonfree cannot be added because the resulting build must licensed under LGPL (--enable-nonfree makes it GPL)

Actually, the complete opposite:

  --enable-nonfree         allow use of nonfree code, the resulting libs
                           and binaries will be unredistributable [no]

However, despite openssl marked nonfree in configure logic, https://github.com/FFmpeg/FFmpeg/blob/master/LICENSE.md#incompatible-libraries claims:

To the best of our knowledge, [Fraunhofer FDK AAC and OpenSSL] are compatible with the LGPL.

git blame'ing configure shows that openssl has been marked nonfree since its adoption and there are no explanation of the reason.
I believe this may have been done to discourage users from choosing these "problematic" libraries over freer alternatives -- in particular, to remove any technical roadblocks to using GPLed parts.

Whatever is the case, GNU TLS (LGPL) should work as an alternative.

[native-api]

Whatever is the case, GNU TLS (LGPL) should work as an alternative.

@abhiTronix since you are the interested party, could you make a PR?

[abhiTronix]

@abhiTronix since you are the interested party, could you make a PR?

@native-api Yeah sure, Please correct me if I'm wrong, I've to make changes to this line:
https://github.com/skvark/opencv-python/blob/da7d02271d2ffca8f293ed3f3032f30881319db8/docker/Dockerfile_x86_64#L53

Also, How do I test these changes, as I currently have access to a Ubuntu machine only? and What should I keep in my mind while making this PR, any special instructions?