How can I use more CPU in exporting?

First, My system is
AMD 2990wx
MSI x399 gaming carbon AC
RAM 16gb (8gb*2, 3200mhz, cl14, dual ch)
1080TI 11gb *1
SSD (OS win10)

Second, My work(is simple)
1. shoot 30min~1hr of 4k 30p with sony a7m3
2. edit and do some simple color grading, no fusion work
3. finally go delivery and export 3840*2160, 30p in MP4 H264

I upgraded my system to just shorten export time
My previous system was 4930k and same VGA(1080TI *1)
and it took about 13~15mins for exporting clip of about 3mins.
It used 60% of CPU.

But now, my present system(2990wx) takes more times to export, about 20mins.
And it uses only 30% of CPU.

What i did wrong? Should i upgrade GPU not CPU? or need change export codec??
PLZ help

during post 2 days, i exported some footage in 3840*2160/30f by MP4.H264
And 2990wx system took shorter time than post 4930k
But when i looked system manage window,
CPU usage was only 20%, GPU usage was also only 25%, RAM usage was 50~60%,
finally DISK(writing to HDD or SSD) speed was 3mb/s
And resolve processed 4~5 frames per second during export.

So. the codec is problem? plz help

Yes, the codec is the problem. Export a good intermediate format, like DNxHR and encode it with ffmpeg.

Hi.

I don't know if you use the free or studio version of Resolve?
But I expect the MP4.H264 encoders to be different?

And then I will suggest you to report it to BMD in some way or another.

If you can wait, can you may be enter it in the DaVonchi 16 feature request list. Faster MP4.H264 encoders. I don't expect BMD to have a quick solution for it. May be they have to talk with their MP4.H264 encoder supplier?
But you must understand at the time the MP4.H264 encoder was developed and tested. There was no way to test it on a 32 cores 64 threds CPU.

Regards Carsten.

BM doesn't have encoder supplier in free version. They use OS provided encoder, so they have no real impact on its speed or features.
It's not about testing encoder on 32 cores. H264 encoding is complex and it's down to actual math behind it. There are simply limits to how many elements of encoding you can run in parallel and there is no real way to bypass it (it will be quite strongly related to frame size for example).

Even x264 won't scale crazy well with 32 cores, specially for HD exports. For UHD CPU usage should be quite high, but it will be still no near linear scale with cores number. If your 1 core can encode something in 320 seconds, there is no way 32 cores will do it in 10. If they can do it in eg. 80 this will be already good. x264 scales quite well only up to about 8 cores. It has hard limit of 128 threads. 16 threads for HD is already quite a lot for x264. There are also other internal restrictions which cause in reality small gain when you pass eg. 24 threads for HD frame (some parts of encoding process will stop at 27 threads for HD frame size).
x264 represent one of the most (if not the most) complex encoder, so in case of others this is probably only less efficient.

Hi Andrew.

First thanks for your answer. You are as always, very well informed.

But I will like to tell you about a vacation several years ago.

I wanted to watch the TV news from home on a very bad mobile internet connections.

By usig a download manager I started 5 different downloads at the same time.
The first downloading the first 5 minutes of the TV news. The second from 5-10 minutes, ...... and the last, from 20 minutes and to the end of the TV news.
Then I used a small program called a script to add the 5 pieces together, and after 6 minutes I was able to, offline to Watch all the TV news from that day.

I know Resolve is not the same, but may be looking into this, or other ideas, I hope it some time will be possible to improve the render and export times. Hope some clever people have some nice ideas.

Regards Carsten.

As I said- you can't skip limitation of long GOP formats as frame depend on each other, so things get complex and have to wait one for another. Some processes cane heavily parallel (specially ones which are not temporal), but some not. There is no easy answer.
You should also separate Resolve internal engine from exporter as those are somehow unrelated (at leats for some formats).

Running single instance multithreaded is one thing and running many instances (single threaded or multithreaded ) is another
Many instances of single transcode split into chunks is about the only way to saturate even eg. 32 cores system.
There are tools which do it- Telestream Episode can do it for example, but guess what?
There are limitations to this also. If you want to do it well, it's quite complex (eg. you have to make sure split points are ideally on scene changes). It won't work optimal on short encodes. Your average bitrate is then per chunk, not whole source, so this can have impact on quality. Ideally you need to run 2 pass encode: after you have stats for each chunk, you build global stats file and then 2nd pass uses adjusted stats for each chunk, so bitrate is spread optimally for whole source (not just per chunk). In some cases averaging per chunk is not really causing a real problem, eg. 4x 30min parts vs. 2h "single" source.
There are tools which can do it, but not many. There are also issues with VBV model violations on join points, but this is more a problem when encoding for eg. Blu-ray.

Well - the news about the 2990WX saturation at export are not encouraging at all May I ask you how things look during caching? Which format do you cache in and to which kind of drive?

Thanks,

Piotr

PS. I wonder if BMD could make possible several export render jobs to be run simultaneously in future version?

Well - the news about the 2990WX saturation at export are not encouraging at all May I ask you how things look during caching? Which format do you cache in and to which kind of drive?

Thanks,

Piotr

PS. I wonder if BMD could make possible several export render jobs to be run simultaneously in future version?

I would not mind to be a background task

I would not mind to be a background task

This, too. Lots of options for multi-threading systems development!

Piotr

Lots of options for multi-threading systems development!

Hi.

I hope BMD will look into the different posibities, as the Threadripper 2990WX only is the first of several multicore CPU's with this kind of problems.

In Q4 is Intels Cascade Lake-X Xeon CPU expected with up to 28 cores 56 threads.

And in August 2019 is the next gereration Threadripper expected on a 7nm process node. I expect it to have 40-48 cores and 80-96 threads.

Regards Carsten.

Well we have to learn how to use all those threads. It may not be possible for single tasks, but as mentiobed it opens possibility to have demanding export running in the background and still be able to work in Resolve on other project. This will be main area which these CPUs will open. Not just multithreading itself but multi app/tasking.

Well - the news about the 2990WX saturation at export are not encouraging at all May I ask you how things look during caching? Which format do you cache in and to which kind of drive?

Thanks,

Piotr

PS. I wonder if BMD could make possible several export render jobs to be run simultaneously in future version?

Why not?
You can have Resolve exporting, maybe some conversion running as well and still be able to work in eg. Photoshop.

Don't use Resolve weakest element (h264 exporter) to judge this CPU.
What about RED playback. I bet you it will shine. You need to choose your CPU based on your main work type. It's not always will be 32 cores AMD. It may be just 12 cores heavily overclocked i9

First, My system is
AMD 2990wx
MSI x399 gaming carbon AC
RAM 16gb (8gb*2, 3200mhz, cl14, dual ch)
1080TI 11gb *1
SSD (OS win10)

Render master as DNxHR/Cineform and transcode with any x264 based app (ffmpeg, Handbrake, etc).

Can you download some 8K RED footage and check how AMD is handling this. You should see all cores nicely used.

First, My system is
AMD 2990wx
MSI x399 gaming carbon AC
RAM 16gb (8gb*2, 3200mhz, cl14, dual ch)
1080TI 11gb *1
SSD (OS win10)

I would look into two things:

1. Where did you place those two RAM DIMMS? The 2990WX and the x399 platform only runs quad channel, but those four channels are divided in two connected to one die each (out of four). So hypothetically you could have connected the 16GB on two channels out of four, and those two channels are going to only one of the four dies. This means that a whopping 3x4 cores have to go through a single memory controller to get access to your stored data.

So I would start out seeing if you did that and then try moving one stick to a different channel/die.

2. Quad channel memory. Not sure if that's an issue in Resolve, but it may be worth investigating it....(?)

2. Quad channel memory. Not sure if that's an issue in Resolve, but it may be worth investigating it....(?)

For this task I think not, but for R3D decode it is very important.

it opens possibility to have demanding export running in the background and still be able to work in Resolve on other project.

Nucoda's been doing this for years now, not just exporting, but also creating caches, it's very a usefull workflow - it asks for lotsa cores, more disk and less GPU than Resolve as it caches in the bg

Yes, it's nothing special to do background caching/rendering. The problem is that when working in app uses eg. 70% of the system then there is not much left for background processes. Flame for example will stop background tasks if user is working and for example hits play button. It also has special design where any machine background tasks can be sent to central/processing machine.
With lots of cores CPUs we are going to see this approach used more. Danger is that code will be non-optimised and you will be actually worse of. I prefer apps which are well written. So many will use 8 cores for something where good app will do it with 2. Edius and FCP X are probably the most optimised NLEs out there.

Here is blog about h265 encoding scaling (up to 128 cores):

https://www.singhkays.com/blog/x265-128 ... -azure-vm/

one of the most useful thing that Nucoda's cache system does is write in DPX or EXR, with the choice of writeing with timeline name and timecode, or source name/tc to a folder of your choice

if you chose to write your caches in time line TC to a folder called "export", then once the producer says it's locked, in maybe 30 seconds or less you are done, colortimed master export finished, done, dusted

the concept of exporting a color timed master need not exist if caches are clever enough, work well in the bg, priorize the working software naturaly, but with enough cores at hand it really does work tranparently

Yes, it only makes sense "now" when there are enough cores to handle export and work without disruption.
Otherwise need of caching is just a sign of poor speed of the software. There are some advantages to cache (even every source, like Flame use to do), but this has its downsides as well. It's good only if it works 100% transparently.