Update FFmpeg presets for Rockchip boards

[MarcA711]

see #979

• This is based on this ffmpeg fork from nyanmisaka. If you use other ffmpeg forks with rockchip hardware support, these presets will likely not work.
• I provide up to date static builds. You can just download the binary and execute is using ./ffmpeg .... No need to install any libraries or dependencies. You need to run linux with up to date kernel drivers though. I recommend ubuntu-rockchip if your board is supported.
• supports transcoding and rescaling
• I added :force_original_aspect_ratio=0 to the scale command to allow rescaling. No other preset does this afaik, but I know that some frigate users want to change their aspect ratio
• Rockchip hardware supports lots of other filters. I hope to add them as well if I have some time.

[MarcA711]

@AlexxIT I know you are not able to test these changes. Is there some way we can still implement them?

[AlexxIT]

I'm busy for the next two weeks

[bugui2007]

Hello all,
@MarcA711, Im using a Orange Pi 5 Plus, and I set the following configuration over go2rtc-ffmpeg config section.

go2rtc:
  ffmpeg:
    input: -fflags nobuffer -flags low_delay -timeout 5000000 -i {input}
    h264/rk: -vf scale_rkrga=w={XXX}:h={XXX}:force_original_aspect_ratio=0 -c:v h264_rkmpp -g 50 -bf 0 -profile:v high -level:v 4.1 -an -vf "transpose=1,transpose=1" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}`
    h265/rk: -vf scale_rkrga=w={XXX}:h={XXX}:force_original_aspect_ratio=0 -c:v hevc_rkmpp -g 50 -bf 0 -profile:v main -level:v 5.1 -an -vf "transpose=1,transpose=1" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}`
    h264/rkmpp: -c:v h264_rkmpp -g 50 -bf 0 -profile:v high -level:v 4.1
    h265/rkmpp: -c:v hevc_rkmpp -g 50 -bf 0 -profile:v main -level:v 5.1

Even with this I didnt note any inprovement on decoding or encoding on hw for go2rtc...but frigate is using rkrga without problems.

Every 1.0s: cat /sys/kernel/debug/rkrga/load                                                                                                                                                                 osiris: Mon Jul 29 18:50:58 2024

num of scheduler = 3
================= load ==================
scheduler[0]: rga3
         load = 0%
-----------------------------------
scheduler[1]: rga3
         load = 0%
-----------------------------------
scheduler[2]: rga2
         load = 2%
-----------------------------------
         process 8: pid = 36993, name: ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -user_agent FFmpeg Frigate/0.14.0-a4eb435 -rtsp_transport tcp -timeout 5000000 -i rtsp://127.0.0.1:85
54/FrenteIzquierda_sub -r 5 -vf scale_rkrga=w=640:h=480:format=yuv420p:force_original_aspect_ratio=0,hwmap=mode=read,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:
         process 9: pid = 37058, name: ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -user_agent FFmpeg Frigate/0.14.0-a4eb435 -rtsp_transport tcp -timeout 5000000 -i rtsp://127.0.0.1:85
54/Cocina_sub -r 5 -vf scale_rkrga=w=640:h=480:format=yuv420p:force_original_aspect_ratio=0,hwmap=mode=read,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:
         process 10: pid = 37048, name: ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -user_agent FFmpeg Frigate/0.14.0-a4eb435 -rtsp_transport tcp -timeout 5000000 -i rtsp://127.0.0.1:8
554/Altillo_sub -r 5 -vf scale_rkrga=w=640:h=480:format=yuv420p:force_original_aspect_ratio=0,hwmap=mode=read,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:
         process 12: pid = 36999, name: ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -user_agent FFmpeg Frigate/0.14.0-a4eb435 -rtsp_transport tcp -timeout 5000000 -i rtsp://127.0.0.1:8
554/Frente_sub -r 5 -vf scale_rkrga=w=640:h=480:format=yuv420p:force_original_aspect_ratio=0,hwmap=mode=read,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:
         process 13: pid = 37035, name: ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -user_agent FFmpeg Frigate/0.14.0-a4eb435 -rtsp_transport tcp -timeout 5000000 -i rtsp://127.0.0.1:8
554/Living_sub -r 5 -vf scale_rkrga=w=640:h=480:format=yuv420p:force_original_aspect_ratio=0,hwmap=mode=read,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:
         process 14: pid = 37010, name: ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -user_agent FFmpeg Frigate/0.14.0-a4eb435 -rtsp_transport tcp -timeout 5000000 -i rtsp://127.0.0.1:8
554/FrenteDerecha_sub -r 5 -vf scale_rkrga=w=640:h=480:format=yuv420p:force_original_aspect_ratio=0,hwmap=mode=read,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:
         process 15: pid = 344434, name: ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -user_agent FFmpeg Frigate/0.14.0-a4eb435 -rtsp_transport tcp -timeout 5000000 -i rtsp://127.0.0.1:
8554/Patio_sub -r 5 -vf scale_rkrga=w=640:h=480:format=yuv420p:force_original_aspect_ratio=0,hwmap=mode=read,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:

Is there any way to force Go2RTC to use HW for decoding and encoding ? How can I debug it?

Another comment:
On the commit #1154 file internal/ffmpeg/hardware/hardware_unix.go @skrashevichskrashevich is using "h264_rkmpp_encoder" and "hevc_rkmpp_encoder" instead "h264_rkmpp" and "hevc_rkmpp", that you propose in your commit in this issue.

Best regards,
Thanks
Hernan

[MarcA711]

Hey @bugui2007,

do you have a section like:

go2rtc:
  streams:

You are just defining some presets. Do you use them when defining your streams? Do you see any video from go2rtc?

If you do this correctly, you force go2rtc to use hw accel. It won't fallback to software. Either it works and uses hardware or it will crash. You can use ps -ef | grep ffmpeg. This will show all ffmpeg processes and the options. For example this process uses hw accel (can be seen from the bold parts):

ffmpeg -hide_banner -allowed_media_types video+audio -fflags nobuffer -flags low_delay -timeout 5000000 -rtsp_flags prefer_tcp -hwaccel rkmpp -hwaccel_output_format drm_prime -afbc rga -i rtsp://xxx -c:v h264_rkmpp -g 50 -bf 0 -an -rtsp_transport tcp -f rtsp rtsp://xxx

h264_rkmpp_encoder and hevc_rkmpp_encoder where old optione from this ffmpeg fork: https://github.com/hbiyik/FFmpeg. This newer fork: https://github.com/nyanmisaka/ffmpeg-rockchip uses h264_rkmpp and hevc_rkmpp. The newer fork is the one that frigate uses.

Btw. you can use the exec command to start ffmpeg with custom commands. You don't need to mess with the presets:

go2rtc:
  streams:
    MyCam: exec:ffmpeg -hide_banner -allowed_media_types video+audio -fflags
      nobuffer -flags low_delay -timeout 5000000 -rtsp_flags prefer_tcp -hwaccel rkmpp
      -hwaccel_output_format drm_prime -afbc rga -i rtsp://xxx
      -c:v h264_rkmpp -g 50 -bf 0 -an -rtsp_transport tcp -f rtsp {{output}}

[bugui2007]

Hello Marca711,
I was using this exec command on each camera (dahua and imou ip cameras) but I cant get audio from this.

go2rtc:
  streams:
     Frente:
      - exec:ffmpeg -hide_banner -allowed_media_types video+audio -fflags nobuffer
        -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp
        -hwaccel rkmpp -hwaccel_output_format drm_prime -afbc rga -i  -c:v hevc_rkmpp
        rtsp://admin:[email protected]/cam/realmonitor?channel=1&subtype=0&unicast=true&proto=Onvif
        -c:v h264_rkmpp -g 50 -bf 0 -an -rtsp_transport tcp -f rtsp {{output}}

Then I switched back to the standar rtsp command like

 - rtsp://admin:[email protected]/cam/realmonitor?channel=1&subtype=0&unicast=true&proto=Onvif

And audio is working now.

Do you have some recommendations to inprove the exec:ffmpeg command in order to get h265 to h264 transcoding on the fly as well can use audio?
Thanks in advace,

Bugui

[MarcA711]

Hey @bugui2007

Try to remove the -an parameter. It removes audio.

You can use -c:a copy copy the audio from the camera or you can transcode the audio to another format if you need to.

[derzahla]

Thanks for your work on this @MarcA711. Currently I only use go2rtc to rotate a camera stream 90 degrees. It was previous working on the latest frigate -rk dev container using:

go2rtc:
  streams:
    rtsp_doorbell_90_ch0:
      - ffmpeg:rtsp://xxxxx:[email protected]:554/ch0#video=h264#video_size=1536x2048#rotate=90#hardware

But that wasnt actually using hardware acceleration from what I gather was the point of your PR? I had previously thought it was (any good way to test?)

But anyway, the rotated stream ends up with a diagonal line and green stripe on the bottom artifacts like below and I was hoping maybe that would be fixed by using the custom ffmpeg cmdline:

I finally got it working using an amalgam of your suggested commands and those at https://github.com/nyanmisaka/ffmpeg-rockchip/wiki/Filter, resulting in:

      - exec:ffmpeg -hide_banner -allowed_media_types video+audio -init_hw_device rkmpp=hw -filter_hw_device hw -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://xxxxxxx:[email protected]:554/ch0 -vf hwupload,vpp_rkrga=w=2048:h=1536:transpose=clock -c:v h264_rkmpp -c:a copy -f rtsp {{output}}

But the result has the same artifacts in the stream unfortunately.

Can you confirm that this should enable hardware acceleration the stream tranposition?

Also, I would prefer to keep the rotated stream h265 like the source stream is but I cannot get frigate to connect to it as a camera unless i convert to h264 - maybe that's a limitation of 'preset-rtsp-restream'?

Let me know if I am doing something wrong and thanks again for you and everyone elses work on improving rockchip support

[MarcA711]

AFAIK, there is no version of frigate that supports hw acceleration using #hardware in go2rtc's ffmpeg command. There was hardware support for rockchip boards in go2rtc, but this was for an old FFmpeg version. Frigate uses a newer FFmpeg version with rockchip support that is not compatible to the way hw accerlation is done in go2rtc. The point of this PR is to update hw accel to support the newest version of FFmpeg for rockchip.

any good way to test?

You can use ps -ef | grep ffmpeg. There you can see what options go2rtc passed to ffmpeg. If it contains some rockchip specific options like -c:v h264_rkmpp, it will use hw acceleration.

But the result has the same artifacts in the stream unfortunately.

There were multiple issues with artifacts in hw accelerated streams. This was mostly due to odd weird resolution. You apperently have 1536x2048, can you try if it works for a standard resolution like 1920x1080? Maybe save a clip from your camera, use software ffmpeg to crop it to 1920x1080 and try if the artifacts still appear.

Can you confirm that this should enable hardware acceleration the stream tranposition?

yes, the command that you provided uses hw acceleration.

Two things to consider:

1. Are you sure your camera can't provide a rotated stream? Most cameras have some settings to rotate the stream correctly.
2. I am not sure if the rotation is very resource intensive. If you rotate it in software, how high is your CPU usage? If it is low, maybe don't use hw accel for this.

Also, I would prefer to keep the rotated stream h265 like the source stream is but I cannot get frigate to connect to it as a camera unless i convert to h264 - maybe that's a limitation of 'preset-rtsp-restream'?

Frigate should be able to process the h265 stream. Did you select the correct hardware acceleration preset (hwaccel_args: preset-rk-h264 or preset-rk-h265)? Are you sure frigate fails to connect to the h265 stream? Because most web browsers don't support h265 since it is not free. If frigate shows no error but your web browser shows some error saying that it cannot play the video, your web browser might not be able to play h265.

[nyanmisaka]

@derzahla @MarcA711 This artifact has been fixed recently. Updating ffmpeg-rockchip should fix the problem.

- exec:ffmpeg -hide_banner -allowed_media_types video+audio -init_hw_device rkmpp=hw -filter_hw_device hw -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://xxxxxxx:[email protected]:554/ch0 -vf hwupload,vpp_rkrga=w=2048:h=1536:transpose=clock -c:v h264_rkmpp -c:a copy -f rtsp {{output}}

Also, hardware decoding is not used. It is SwDec -> Hw{Scale,Transpose} -> HwEnc.

[MarcA711]

@nyanmisaka Thank you for this info.

@derzahla I try to build a new FFmpeg Rockchip binary with the fix. I will write here, when I am done.

[MarcA711]

@derzahla I uploaded a new version that should contain the fix. You can download it here: https://github.com/MarcA711/Rockchip-FFmpeg-Builds/releases/tag/6.1-6
I haven't tested it yet. It would be great if you can confirm that it is working for you.