T4 Performance Inquiry

[philipp-schmidt]

Hi @rarzumanyan

This isn't the usual issue - it's more of a question. I've seen your article here with tests on decoding performance of the T4.

To the best of your knowledge and test result experience: can you tell me whether the T4 be able to handle 240 FPS of 20 concurrent 1080p H.264 streams (each one 12FPS)? Quality settings will vary but will be very "average" probably. Also taking into account that the T4 itself will already be under heavy computation load of a neural net. Will this impact decoding performance? Will 70 Watt be a limit in any way?

Thanks for your expertise.

[philipp-schmidt]

Followup question: can VRAM become a limit? I've seen some apps take up to 500MB of VRAM when they start decoding. Is there a way to influence how much VRAM is used per "decoding queue"?

[rarzumanyan]

Hi @philipp-schmidt

There are some performance results for VPF also published at devblog:

Big Buck Bunny sequence contains 14,315 frames and can be transcoded within 32 seconds which gives ~447fps without using any advanced techniques such as producer-consumer pattern with decoded frames queue being shared by decoder and encoder launched in separate threads. Since all the transcoding is done on the GPU, there’s no noticeable CPU load.

Since T4 has 2 Nvdec units and single PyNvDecoder instance can use only one, you can double up these numbers. 447 FPS result was obtained on my RTX5000 card which is very similar to T4 in terms of video performance.
Please note that to achieve max performance you have to eliminate raw frames traffic between Host and Device. If you download raw frames to RAM and / or dump them to disk, it will bottleneck the performance.

Will this impact decoding performance?

It will not impact PyNvDecoder and PyNvEncoder since Nvdec and Nvenc are separate HW blocks, they are just located at the same chip as CUDA cores are. Some VPF tasks performance like resize, color conversion etc. will be affected as they use CUDA cores.

can VRAM become a limit?

Usually it's not.
To get accurate numbers I recommend you to run your script under Nsight profiler and see the actual amount of memory which is consumed. If you're doing some heavy duty NN job which use lots of memory this can be an issue though.

Is there a way to influence how much VRAM is used per "decoding queue"?

VFP uses built-in Annex.B parser which configures Nvdec HW, allocates memory for decoded frames queue etc. so frankly I've just never touched this part of code since it works fine. You can change the amount of memory which will be allocated for decoded frames pool here:

VideoProcessingFramework/PyNvCodec/TC/inc/Tasks.hpp

Lines 76 to 79 in edb7d8f

	NvdecDecodeFrame(CUstream cuStream, CUcontext cuContext,
	cudaVideoCodec videoCodec, uint32_t decodedFramesPoolSize,
	uint32_t coded_width, uint32_t coded_height,
	Pixel_Format format);

by changing the default decodedFramesPoolSize value. Given that raw 1080p frame is ~3MB and default pool size is 4 there's not too much room for adjustment though.

Will 70 Watt be a limit in any way?

To my best knowledge T4 TDP is limited by 70 Wt because it doesn't have any additional 6/8 pin power supply. If you need exact numbers, I can ask our solution architects - just send me an email.

P. S.
I'm now investigating the memory allocation done by VPF in #204 and there's a feature branch nvtx_support which adds NVTX markers to all VPF Tasks to make the profiling easier. I guess it may be useful for your performance research.

P. P. S.
If excessive memory consumption is the problem, please submit an issue for it. I'm doing development & fixing in feature branches, one branch per issue to avoid the confusion when I merge feature branch to master.

[philipp-schmidt]

Hi @rarzumanyan, thanks for this very comprehensive insight and detailed answer - very much appreciated. This all sounds very promising! I will conduct a few more tests in the coming days and share my results.

Please note that to achieve max performance you have to eliminate raw frames traffic between Host and Device.

Yes this makes perfect sense. I will probably reduce size of these raw frames by scaling them down to what the NNs need as input using the handy resize feature in VPF. This way memory bandwidth should be sufficient. It's unclear currently if sharing CUDA memory from VPF to NN is a viable solution for us or whether it adds to much complexity - remains to be seen.

[rarzumanyan]

@philipp-schmidt

It's unclear currently if sharing CUDA memory from VPF to NN is a viable solution for us or whether it adds to much complexity

You can also export & import between any CUdeviceptr and VPF SurfacePlace using Export() and Import() methods as it's shown here:

VideoProcessingFramework/SamplePyTorch.py

Lines 73 to 98 in edb7d8f

	while True:
	rawSurface = nvDec.DecodeSingleSurface()
	if rawSurface.Empty():
	break
	# Export VPF RGB Surface to PyTorch tensor.
	# Please note that pitch is equal to width * 3.
	# SurfacePlane is raw CUDA 2D memory allocation chunk so for
	# interleaved RGB frame it's width is 3x picture width.
	rgb24 = to_rgb.Execute(rawSurface)
	rgb24.PlanePtr().Export(surface_tensor.data_ptr(), w * 3, gpuID)
	# PROCESS YOUR TENSOR HERE.
	# THIS DUMMY PROCESSING WILL JUST MAKE VIDEO FRAMES DARKER.
	dark_frame = torch.floor_divide(surface_tensor, 2)
	# Import to VPF Surface. Same thing about pitch as before.
	surface_rgb.PlanePtr().Import(dark_frame.data_ptr(), w * 3, gpuID)
	# Convert to NV12
	surface_yuv = to_yuv.Execute(surface_rgb)
	surface_nv12 = to_nv12.Execute(surface_yuv)
	# Encode
	success = nvEnc.EncodeSingleSurface(surface_nv12, encFrame)
	if success:
	encByteArray = bytearray(encFrame)
	dstFile.write(encByteArray)

I will conduct a few more tests in the coming days and share my results.

If you could share your results that would be very helpful & much appreciated - especially in comparison to other solutions - maybe something like OpenCV. I do a lot of micro-benchmarks & know what low-level performance is but often can't help users with advice on solution-level performance & what to expect of VPF in general.

[philipp-schmidt]

You can also export & import between any CUdeviceptr and VPF SurfacePlace using Export() and Import() methods

Yes I'm aware of that. We would have to use Nvidia Triton Inference Server and its cuda shared memory feature, but this will be a performance / complexity tradeoff as I still have to find out how to make it fully compatible.

If you could share your results that would be very helpful & much appreciated

Of course! I'll share as much data as I can.

especially in comparison to other solutions - maybe something like OpenCV

Well, CPU OpenCV is no option anymore for us as it seems to allocate reaaally much CPU time... Anyone doing more than the very basic video processing stuff should consider using nvenc.

[rarzumanyan]

@philipp-schmidt

We would have to use Nvidia Triton Inference Server and its cuda shared memory feature, but this will be a performance / complexity tradeoff as I still have to find out how to make it fully compatible

Could you please give a little more info about that?

I'll talk to my colleagues who are working on Triton, maybe the better VPF <-> Triton handshake is something I would be able to work on.

If it's not a topic you would like to discuss publicly, please contact me via email.

[philipp-schmidt]

Could you please give a little more info about that?

Triton is basically a server running your ML networks and offering http or grpc endpoints for them. Clients can send a request and invoke the networks and receive results without having to use frameworks or bother with implementation or GPU acceleration, etc. It's by far the best tool to scale ML workload imo.

To start a request to the server there are basically three options to pass the data (e.g. an image to analyze via object detection):

• pass your data via http or grpc payload (only option if you are not on the same compute node, so over the network)
• register and pass a pointer to shared memory with the data
• register and pass a pointer to shared memory ON GPU with the data

Obviously the performance increases with the order in the list as the amount of memory copy operations goes down to basically zero with sharing GPU memory.

I'll talk to my colleagues who are working on Triton, maybe the better VPF <-> Triton handshake is something I would be able to work on.

It would obviously be very useful if VPF could simply provide the necessary pointers to the data in the GPU memory to Triton Inference Server. This would mean that VPF needs to have a very good set of processing layers (like the already existing scaling and color convert operation), because most networks expect a very well defined input and conversion needs to happen on GPU. And there might be things like batching (compute multiple images at once) which might make things more complicated. I'm not sure how much work this would be, but I (and probably many others) would definitely appreciate support for Triton in VPF! They would make for a very nice combination for high performance video processing. It's too much complexity to implement it on my own currently (given the benefits), but I'd be happy to test and contribute via an extension / tutorial to our open sourced repo if you and the Triton maintainers decide to give this a go.

[rarzumanyan]

Thanks for the detailed reply, @philipp-schmidt

Let us come back to this topic shortly in future as I get a chance to talk to our Triton people and get an insight of how labor intensive this may be. Meanwhile I'll close this issue as resolved and will create a new feature request.