How can I use more CPU in exporting?

Get answers to your questions about color grading, editing and finishing with DaVinci Resolve.
  • Author
  • Message
Offline

kimawho

  • Posts: 2
  • Joined: Sat Aug 18, 2018 12:28 am
  • Real Name: Seongtae Kim

How can I use more CPU in exporting?

PostSat Aug 18, 2018 12:51 am

First, My system is
AMD 2990wx
MSI x399 gaming carbon AC
RAM 16gb (8gb*2, 3200mhz, cl14, dual ch)
1080TI 11gb *1
SSD (OS win10)

Second, My work(is simple)
1. shoot 30min~1hr of 4k 30p with sony a7m3
2. edit and do some simple color grading, no fusion work
3. finally go delivery and export 3840*2160, 30p in MP4 H264

I upgraded my system to just shorten export time
My previous system was 4930k and same VGA(1080TI *1)
and it took about 13~15mins for exporting clip of about 3mins.
It used 60% of CPU.

But now, my present system(2990wx) takes more times to export, about 20mins.
And it uses only 30% of CPU.

What i did wrong? Should i upgrade GPU not CPU? or need change export codec??
PLZ help :(
Offline

kimawho

  • Posts: 2
  • Joined: Sat Aug 18, 2018 12:28 am
  • Real Name: Seongtae Kim

Re: How can I use more CPU in exporting?

PostWed Aug 22, 2018 4:59 am

during post 2 days, i exported some footage in 3840*2160/30f by MP4.H264
And 2990wx system took shorter time than post 4930k
But when i looked system manage window,
CPU usage was only 20%, GPU usage was also only 25%, RAM usage was 50~60%,
finally DISK(writing to HDD or SSD) speed was 3mb/s
And resolve processed 4~5 frames per second during export.

So. the codec is problem? plz help :cry:
Offline

Uli Plank

  • Posts: 4966
  • Joined: Fri Feb 08, 2013 2:48 am

Re: How can I use more CPU in exporting?

PostThu Aug 23, 2018 8:00 am

Yes, the codec is the problem. Export a good intermediate format, like DNxHR and encode it with ffmpeg.
Resolve Studio and Fusion Studio
iMac 2017 Radeon Pro 580 8 GB VRAM and 32 GB RAM
Offline

Carsten Sellberg

  • Posts: 886
  • Joined: Fri Jun 16, 2017 9:13 am

Re: How can I use more CPU in exporting?

PostThu Aug 23, 2018 9:14 am

Hi.

I don't know if you use the free or studio version of Resolve?
But I expect the MP4.H264 encoders to be different?

And then I will suggest you to report it to BMD in some way or another.

If you can wait, can you may be enter it in the DaVonchi 16 feature request list. Faster MP4.H264 encoders. I don't expect BMD to have a quick solution for it. May be they have to talk with their MP4.H264 encoder supplier?
But you must understand at the time the MP4.H264 encoder was developed and tested. There was no way to test it on a 32 cores 64 threds CPU.

Regards Carsten.
URSA Mini 4.6K
Offline

Andrew Kolakowski

  • Posts: 5713
  • Joined: Tue Sep 11, 2012 10:20 am

Re: How can I use more CPU in exporting?

PostThu Aug 23, 2018 10:00 am

BM doesn't have encoder supplier in free version. They use OS provided encoder, so they have no real impact on its speed or features.
It's not about testing encoder on 32 cores. H264 encoding is complex and it's down to actual math behind it. There are simply limits to how many elements of encoding you can run in parallel and there is no real way to bypass it (it will be quite strongly related to frame size for example).

Even x264 won't scale crazy well with 32 cores, specially for HD exports. For UHD CPU usage should be quite high, but it will be still no near linear scale with cores number. If your 1 core can encode something in 320 seconds, there is no way 32 cores will do it in 10. If they can do it in eg. 80 this will be already good. x264 scales quite well only up to about 8 cores. It has hard limit of 128 threads. 16 threads for HD is already quite a lot for x264. There are also other internal restrictions which cause in reality small gain when you pass eg. 24 threads for HD frame (some parts of encoding process will stop at 27 threads for HD frame size).
x264 represent one of the most (if not the most) complex encoder, so in case of others this is probably only less efficient.
Offline

Carsten Sellberg

  • Posts: 886
  • Joined: Fri Jun 16, 2017 9:13 am

Re: How can I use more CPU in exporting?

PostThu Aug 23, 2018 11:07 am

Hi Andrew.

First thanks for your answer. You are as always, very well informed.

But I will like to tell you about a vacation several years ago.

I wanted to watch the TV news from home on a very bad mobile internet connections.

By usig a download manager I started 5 different downloads at the same time.
The first downloading the first 5 minutes of the TV news. The second from 5-10 minutes, ...... and the last, from 20 minutes and to the end of the TV news.
Then I used a small program called a script to add the 5 pieces together, and after 6 minutes I was able to, offline to Watch all the TV news from that day.

I know Resolve is not the same, but may be looking into this, or other ideas, I hope it some time will be possible to improve the render and export times. Hope some clever people have some nice ideas.

Regards Carsten.
Last edited by Carsten Sellberg on Thu Aug 23, 2018 11:27 am, edited 1 time in total.
URSA Mini 4.6K
Offline

Andrew Kolakowski

  • Posts: 5713
  • Joined: Tue Sep 11, 2012 10:20 am

Re: How can I use more CPU in exporting?

PostThu Aug 23, 2018 11:19 am

As I said- you can't skip limitation of long GOP formats as frame depend on each other, so things get complex and have to wait one for another. Some processes cane heavily parallel (specially ones which are not temporal), but some not. There is no easy answer.
You should also separate Resolve internal engine from exporter as those are somehow unrelated (at leats for some formats).

Running single instance multithreaded is one thing and running many instances (single threaded or multithreaded ) is another :)
Many instances of single transcode split into chunks is about the only way to saturate even eg. 32 cores system.
There are tools which do it- Telestream Episode can do it for example, but guess what?
There are limitations to this also. If you want to do it well, it's quite complex (eg. you have to make sure split points are ideally on scene changes). It won't work optimal on short encodes. Your average bitrate is then per chunk, not whole source, so this can have impact on quality. Ideally you need to run 2 pass encode: after you have stats for each chunk, you build global stats file and then 2nd pass uses adjusted stats for each chunk, so bitrate is spread optimally for whole source (not just per chunk). In some cases averaging per chunk is not really causing a real problem, eg. 4x 30min parts vs. 2h "single" source.
There are tools which can do it, but not many. There are also issues with VBV model violations on join points, but this is more a problem when encoding for eg. Blu-ray.
Offline
User avatar

Piotr Wozniacki

  • Posts: 1172
  • Joined: Tue Aug 02, 2016 12:17 pm
  • Location: Poland

Re: How can I use more CPU in exporting?

PostThu Aug 23, 2018 12:32 pm

Well - the news about the 2990WX saturation at export are not encouraging at all :( May I ask you how things look during caching? Which format do you cache in and to which kind of drive?

Thanks,

Piotr

PS. I wonder if BMD could make possible several export render jobs to be run simultaneously in future version?
AMD TR 2990WX CPU | MSI X399 CARBON AC | 64GB RAM@XMP2933 | 2x RTX 2080Ti GPU | 4x 3TB WD Black RAID0 media drive | 3x 1TB NVMe RAID0 cache drive | SSD SATA system drive | AX1600i PSU | Decklink 12G Extreme | Samsung UHD reference monitor (calibrated)
Offline
User avatar

waltervolpatto

  • Posts: 7139
  • Joined: Thu Feb 07, 2013 5:07 pm
  • Location: 1146 North Las Palmas Ave. Hollywood, California 90038 USA

Re: How can I use more CPU in exporting?

PostThu Aug 23, 2018 1:15 pm

Piotr Wozniacki wrote:Well - the news about the 2990WX saturation at export are not encouraging at all :( May I ask you how things look during caching? Which format do you cache in and to which kind of drive?

Thanks,

Piotr

PS. I wonder if BMD could make possible several export render jobs to be run simultaneously in future version?


I would not mind to be a background task
SuperServer 5039AD-I
C9X299-PGF - DDR4-2400 16x4 GB
i9-7920xCPU 12c 2.90GHz Water cooled
2x 1080ti DeckLink Studio 4K (11.4)
Window10 - BMR St. 16.1.0B.033
nvidia: 431.86 studio
Offline
User avatar

Piotr Wozniacki

  • Posts: 1172
  • Joined: Tue Aug 02, 2016 12:17 pm
  • Location: Poland

Re: How can I use more CPU in exporting?

PostThu Aug 23, 2018 1:32 pm

waltervolpatto wrote:I would not mind to be a background task

This, too. Lots of options for multi-threading systems development!

Piotr
AMD TR 2990WX CPU | MSI X399 CARBON AC | 64GB RAM@XMP2933 | 2x RTX 2080Ti GPU | 4x 3TB WD Black RAID0 media drive | 3x 1TB NVMe RAID0 cache drive | SSD SATA system drive | AX1600i PSU | Decklink 12G Extreme | Samsung UHD reference monitor (calibrated)
Offline

Carsten Sellberg

  • Posts: 886
  • Joined: Fri Jun 16, 2017 9:13 am

Re: How can I use more CPU in exporting?

PostThu Aug 23, 2018 2:02 pm

Piotr Wozniacki wrote: Lots of options for multi-threading systems development!


Hi.

I hope BMD will look into the different posibities, as the Threadripper 2990WX only is the first of several multicore CPU's with this kind of problems.

In Q4 is Intels Cascade Lake-X Xeon CPU expected with up to 28 cores 56 threads.

And in August 2019 is the next gereration Threadripper expected on a 7nm process node. I expect it to have 40-48 cores and 80-96 threads.

Regards Carsten.
URSA Mini 4.6K
Offline

Andrew Kolakowski

  • Posts: 5713
  • Joined: Tue Sep 11, 2012 10:20 am

Re: How can I use more CPU in exporting?

PostThu Aug 23, 2018 6:09 pm

Well we have to learn how to use all those threads. It may not be possible for single tasks, but as mentiobed it opens possibility to have demanding export running in the background and still be able to work in Resolve on other project. This will be main area which these CPUs will open. Not just multithreading itself but multi app/tasking.
Offline

Andrew Kolakowski

  • Posts: 5713
  • Joined: Tue Sep 11, 2012 10:20 am

Re: How can I use more CPU in exporting?

PostThu Aug 23, 2018 6:11 pm

Piotr Wozniacki wrote:Well - the news about the 2990WX saturation at export are not encouraging at all :( May I ask you how things look during caching? Which format do you cache in and to which kind of drive?

Thanks,

Piotr

PS. I wonder if BMD could make possible several export render jobs to be run simultaneously in future version?


Why not?
You can have Resolve exporting, maybe some conversion running as well and still be able to work in eg. Photoshop.

Don't use Resolve weakest element (h264 exporter) to judge this CPU.
What about RED playback. I bet you it will shine. You need to choose your CPU based on your main work type. It's not always will be 32 cores AMD. It may be just 12 cores heavily overclocked i9 :D
Offline

Andrew Kolakowski

  • Posts: 5713
  • Joined: Tue Sep 11, 2012 10:20 am

Re: How can I use more CPU in exporting?

PostThu Aug 23, 2018 6:25 pm

kimawho wrote:First, My system is
AMD 2990wx
MSI x399 gaming carbon AC
RAM 16gb (8gb*2, 3200mhz, cl14, dual ch)
1080TI 11gb *1
SSD (OS win10)



Render master as DNxHR/Cineform and transcode with any x264 based app (ffmpeg, Handbrake, etc).

Can you download some 8K RED footage and check how AMD is handling this. You should see all cores nicely used.
Offline

Mattias Murhagen

  • Posts: 68
  • Joined: Mon Nov 27, 2017 3:09 am
  • Location: New York

Re: How can I use more CPU in exporting?

PostSat Aug 25, 2018 1:45 am

kimawho wrote:First, My system is
AMD 2990wx
MSI x399 gaming carbon AC
RAM 16gb (8gb*2, 3200mhz, cl14, dual ch)
1080TI 11gb *1
SSD (OS win10)


I would look into two things:

1. Where did you place those two RAM DIMMS? The 2990WX and the x399 platform only runs quad channel, but those four channels are divided in two connected to one die each (out of four). So hypothetically you could have connected the 16GB on two channels out of four, and those two channels are going to only one of the four dies. This means that a whopping 3x4 cores have to go through a single memory controller to get access to your stored data.

So I would start out seeing if you did that and then try moving one stick to a different channel/die.

2. Quad channel memory. Not sure if that's an issue in Resolve, but it may be worth investigating it....(?)
Offline
User avatar

Jack Fairley

  • Posts: 1311
  • Joined: Mon Oct 03, 2016 7:58 pm
  • Location: Los Angeles

Re: How can I use more CPU in exporting?

PostSat Aug 25, 2018 2:03 am

Mattias Murhagen wrote:2. Quad channel memory. Not sure if that's an issue in Resolve, but it may be worth investigating it....(?)

For this task I think not, but for R3D decode it is very important.
Threadripper 1950X 3.7GHz
64GB DDR4-2666
2x GTX 1080 Ti
DeckLink 4K Extreme 12G
Resolve Studio 16.1b3
Windows 10 Pro 1803
Offline

Dermot Shane

  • Posts: 1911
  • Joined: Tue Nov 11, 2014 6:48 pm
  • Location: Vancouver, Canada

Re: How can I use more CPU in exporting?

PostSat Aug 25, 2018 2:30 am

Andrew Kolakowski wrote: it opens possibility to have demanding export running in the background and still be able to work in Resolve on other project.

Nucoda's been doing this for years now, not just exporting, but also creating caches, it's very a usefull workflow - it asks for lotsa cores, more disk and less GPU than Resolve as it caches in the bg
Offline

Andrew Kolakowski

  • Posts: 5713
  • Joined: Tue Sep 11, 2012 10:20 am

Re: How can I use more CPU in exporting?

PostSat Aug 25, 2018 9:15 am

Yes, it's nothing special to do background caching/rendering. The problem is that when working in app uses eg. 70% of the system then there is not much left for background processes. Flame for example will stop background tasks if user is working and for example hits play button. It also has special design where any machine background tasks can be sent to central/processing machine.
With lots of cores CPUs we are going to see this approach used more. Danger is that code will be non-optimised and you will be actually worse of. I prefer apps which are well written. So many will use 8 cores for something where good app will do it with 2. Edius and FCP X are probably the most optimised NLEs out there.
Offline

Andrew Kolakowski

  • Posts: 5713
  • Joined: Tue Sep 11, 2012 10:20 am

Re: How can I use more CPU in exporting?

PostSun Aug 26, 2018 11:16 pm

Here is blog about h265 encoding scaling (up to 128 cores):

https://www.singhkays.com/blog/x265-128 ... -azure-vm/
Offline

Dermot Shane

  • Posts: 1911
  • Joined: Tue Nov 11, 2014 6:48 pm
  • Location: Vancouver, Canada

Re: How can I use more CPU in exporting?

PostMon Aug 27, 2018 3:13 am

one of the most useful thing that Nucoda's cache system does is write in DPX or EXR, with the choice of writeing with timeline name and timecode, or source name/tc to a folder of your choice

if you chose to write your caches in time line TC to a folder called "export", then once the producer says it's locked, in maybe 30 seconds or less you are done, colortimed master export finished, done, dusted

the concept of exporting a color timed master need not exist if caches are clever enough, work well in the bg, priorize the working software naturaly, but with enough cores at hand it really does work tranparently
Offline

Andrew Kolakowski

  • Posts: 5713
  • Joined: Tue Sep 11, 2012 10:20 am

Re: How can I use more CPU in exporting?

PostMon Aug 27, 2018 10:36 am

Yes, it only makes sense "now" when there are enough cores to handle export and work without disruption.
Otherwise need of caching is just a sign of poor speed of the software. There are some advantages to cache (even every source, like Flame use to do), but this has its downsides as well. It's good only if it works 100% transparently.

Return to DaVinci Resolve

Who is online

Users browsing this forum: Jeffrey Hahn, Peter Foerster, robozb, Soeren Mueller and 54 guests