Render API

MDK supports rendering video via OpenGL, D3D11, Vulkan and Metal. You can choose which api to use at runtime via Player.setRenderAPI(RenderAPI* api, void* vo_opaque = nullptr). More graphics apis will be supported in the future, for example vulkan and metal.

OpenGL is the default api if you don't call setRenderAPI(). To use another api, a concrete RenderAPI subclass object is required. For example to use d3d11

    D3D11RenderAPI ra;
    // set ra.context and ra.rtv if provided by app
    player.setRenderAPI(&ra);

3 Rendering Modes in MDK

Render in a User Provided Context

It's named "Foreign Context". Usually you have to render a frame in a correct rendering thread of the gui toolkit or app. For example, call Player.renderVideo() in QOpenGLWidget::paintGL() or similar events for Qt Apps, in GLSurfaceView.RendereronDrawFrame(GL10 gl) for android apps. The callback of Player.setRenderCallback(callback) will be called if a new frame is ready to display, you can notify your gui toolkit or app in this callback.

For a foreign OpenGL(or other RenderAPI) context, you need to call only

• setVideoSurfaceSize(): frame buffer size. usually the window/surface size. call it when window is resized
• setRenderCallback(): the callback is called when a new frame can be rendered. you can notify your rendering thread to render a video. NOT REQUIRED if rendering not driven by decoded video frames, e.g. Apple DisplayLink, Android GLSurfaceView.
• renderVideo(): rendering thread is notified and draw the frame. A valid context is requred.

To use a RenderAPI other than OpenGL, a concrete RenderAPI subclass object with some valid members is required. For example a D3D11RenderAPI requires rtv from user

    D3D11RenderAPI ra;
    //ra.context = immediateContext; // immediateContext provied by app, can be null if rtv is not null
    ra.rtv = renderTargetView; // renderTargetView provided by app, can not be null
    player.setRenderAPI(&ra);

examples:

• OBS Studio
• OpenGL context created by GLFW
• macOS CAOpenGLLayer
• UWP CoreWindow
• OpenGL context created by SDL
• QtWidgets OpenGL Context
• QtQuick RHI
• D3D11 texture in Flutter
• Linux OpenGL in Flutter
• macOS/iOS MTLTexture in Flutter
• macOS/iOS MTLView in Swift
• Android GLSurfaceView

Render On a Platform Surface

MDK use another project UGS to create and drive a rendering loop for a platform native surface. Call updateNativeSurface() with such a surface value when surface changes, or a null value to create internally. A null value also works for X11 Window or wl_egl_window* or gbm_surface* on linux, EGL_DISPMANX_WINDOW_T* on raspberry pi(legacy driver).

No need to call setRenderCallback() and renderVideo().

To use a RenderAPI other than OpenGL, a concrete RenderAPI with default initialized members is required. For example a D3D11RenderAPI requires context and rtv member can be null. And setRenderAPI() MUST be called before the first Player.updateNativeSurface()

    D3D11RenderAPI ra;
    player.setRenderAPI(&ra);
    // ...
    player.updateNativeSurface(....); 

examples:

• X11 Window + OpenGL
• GLFW window + D3D11/Metal/OpenGL
• iOS UIView Metal
• macOS NSView OpenGL
• UWP XAML SwapChainPanel
• Android Surface from SurfaceTexture in Flutter
• Android Surface from SurfaceView

Render On a Surface Created Internally

Use Player.createSurface() to create variant kinds of surface supported by current OS.

To use a RenderAPI other than OpenGL, a concrete RenderAPI with default initialized members MUST be set via setRenderAPI() before createSurface()

examples:

• https://github.com/wang-bin/mdk-examples/blob/master/Native/mdkplay.cpp