WebRTC: Audio is corrupt when using FFmpeg native opus codec. When converting audio AAC to opus using FFmpeg's built-in opus encoding, there is a crackling sound.

[chundonglinlin]

Description

If you use the built-in opus in FFmpeg, there will definitely be a loud noise, and this can be reproduced.

Patch Commit ID: 8d61c2a

SRS Version: develop v5.0.36

1. Compile FFmpeg

   • Use FFmpeg's built-in opus: --enable-decoder=opus --enable-encoder=opus
   • Use libopus library: --enable-libopus

2. SRS Config (Configuration)

listen              1935;
max_connections     1000;
daemon              off;
srs_log_tank        console;

http_server {
    enabled         on;
    listen          8080;
    dir             ./objs/nginx/html;
}

http_api {
    enabled         on;
    listen          1985;
}

rtc_server {
    enabled on;
    listen 8000; # UDP port
    # @see https://ossrs.net/lts/zh-cn/docs/v4/doc/webrtc#config-candidate
    #candidate $CANDIDATE;
    candidate 10.254.44.205;
}

vhost __defaultVhost__ {
    rtc {
        enabled     on;
        # @see https://ossrs.net/lts/zh-cn/docs/v4/doc/webrtc#rtmp-to-rtc
        rtmp_to_rtc on;
        # @see https://ossrs.net/lts/zh-cn/docs/v4/doc/webrtc#rtc-to-rtmp
        rtc_to_rtmp on;
    }
    http_remux {
        enabled     on;
        mount       [vhost]/[app]/[stream].flv;
    }
}

Replay (Reproduction)

Please describe the steps to reproduce the bug. (重现Bug的步骤)

1. RTMP streaming:
   ffmpeg -stream_loop -1 -re -i 264_aac_basline_48k.mp4 -c copy -f flv "rtmp://127.0.0.1/live/livestream"

2. RTC playback, open the player https://127.0.0.1/players/rtc_player.html, and play:
   https://127.0.0.1/players/rtc_player.html

Expect (Expected Behavior)

RTC playback works normally.

TRANS_BY_GPT3

[winlinvip]

Specify using the built-in opus in FFmpeg, with the option:

  --ffmpeg-opus=on|off      Whether enable the FFmpeg native opus codec. Default: off

After changing the opus library, you need to delete FFmpeg and recompile it.

rm -rf objs
./configure --ffmpeg-opus=on
make

TRANS_BY_GPT3

[xiaozhihong]

The main reason is that the frame_size set in ffmpeg's opusenc.c (note, not libopusenc.c) is 120, and opus has a sampling rate of 48000, which means each frame is 2.5ms.

From my testing, it seems that approximately every 5 input frames are needed to obtain one output frame from the encoder. Moreover, the latency is high and the size is small, as shown in the following figure.

The PCM frames from pts 2936 to 2954 are input to the encoder, and the frame at pts 2589 is the output from the encoder. It can be observed that there is a significant delay of 300ms+ and the size is very small.

Any experts who know how to fix this issue, please help and reply to this problem.

TRANS_BY_GPT3