Path Maps

I have been able to get network rendering processes for, disk caches, clustering, preview renders, and final renders. However, when enabling network rendering my timeline cache does not file and then needs to render each frame individually. For example, if I set an disk cache on a node without network rendering enabled, the frames cache(green bar grows in timeline). With the same location selected for image caching and tick the box that says use network, the network render kicks on and processes on all workstations the network group is set for, but the frames don't cache(green bar doesn't grow in the timeline) so after the disk cache is complete, I still need to render the frames the play head is at.

Path maps are set to:
DiskCaches: v:/FusionNetworkRender
Previews: v:/FusionNetworkRender
LoaderCache: v:/FusionNetworkRender

FlipbookRAM is set to: Use image cache.

Anyone have any thoughts?

That's normal. You'd have to both disk cache and cluster to get it forced from disk into RAM.

While the images aren't in RAM, they should read from disk quickly. But yeah, there's no point to loading it into RAM if you are just caching a branch in the background. If you had a ton of RAM, yeah, I could see it, but you might as well be using background rendering and clustering too.

Thanks Chad,

That's kind of what I thought, but when I cache a branch in the back ground, it's Fusion isn't reading from the disk cache location. When I activate the disk cache icon after network rendering that node, I move the playhead to another frame that should be reading directly from the disk cache, but it takes just as long to render the frame as it did with out the disk cache active, which leads me to believe that the disk cache image is not being read. I'm on a 10gig Network, so network speed is not an issue. Are their limitations to disk caching?? Like not being able to cache a merge node?

Depends. If you need pixels outside the image but within the DoD, then yes, it has to rerender that anyway. Did you try locking the cache and branch?

I'm finding if I just load the ".raw" files as a loader to test performance, it is exactly like I see it in the flow. the .raw files are loading around 1 frame a second, from an SSD that is capable of 450+ MB/sec. Is this normal behavior for .raw files to take that long to load into RAM? Also, tried the cluster and disk caching simultaneously, but the cache is not pulling from the image cache when trying to load into RAM what the cluster is creating.

So I guess the next thing is, what is the best performance set up with multiple workstations, a 10gig networking, and fast local SSD, can you recommend a set up for previewing and working with a client that allows for the fastest previews possible? The main system I'm using Fusion Studio on is a overclocked I7-5960x with 64gigs RAM and GTX Titan Black.

thanks!

I'm finding if I just load the ".raw" files as a loader to test performance, it is exactly like I see it in the flow. the .raw files are loading around 1 frame a second, from an SSD that is capable of 450+ MB/sec. Is this normal behavior for .raw files to take that long to load into RAM? Also, tried the cluster and disk caching simultaneously, but the cache is not pulling from the image cache when trying to load into RAM what the cluster is creating.

So I guess the next thing is, what is the best performance set up with multiple workstations, a 10gig networking, and fast local SSD, can you recommend a set up for previewing and working with a client that allows for the fastest previews possible? The main system I'm using Fusion Studio on is a overclocked I7-5960x with 64gigs RAM and GTX Titan Black.

thanks!

Yes, Fusion has terrible I/O speeds. There's a lot of reasons for that, but mostly you can expect terrible I/O speeds. The only tip I can give is to not use Loaders, but unless you are prepared to make your own plugins, there's not much you can do that I know of.

Thanks again Chad, this stuff really had me scratching my head, why certain things were so fast, and others painfully slow. Hopefully Blackmagic can address some of the I/O speed issues.

cheers,

Ryan

[daniel.koch]

Yes, Fusion has terrible I/O speeds. There's a lot of reasons for that, but mostly you can expect terrible I/O speeds.

I wouldn't say that. Just tested reading a sequence of 300 raw files on my laptop, 4K float32 @ 200MB each, at 5.3 frames/second.

What *can* be slow are compressed files. The raw files above were uncompressed, but when saved with zlib compression my laptop read them at 0.6 frames/second. Some file formats can use multiple CPU cores to decompress, but many can't, and are limited to the speed of a single core when reading. This can make them quite a bit slower than the SSD speed alone.

Also, be aware that diskcaches are written as compressed raw files by default (used to be uncompressed, but people complained about the size), which may explain your results. But you can choose to write your diskcache files in any format (EXR, TGA, DPX etc), which can be much faster.

Another thing to consider is increasing the Prefs/Memory/Frames At Once setting, which can let Fusion render one frame while simultaneously preloading the next. For many comps this can keep your CPU more consistently occupied (at the cost of more RAM usage), but very simple comps will often be limited by the slowest node, which for various compressed formats may be the Loader or Saver.

Fusion's I/O can be plenty fast, but it really depends on your job; some workloads can be more efficient than others, but knowing more about what's happening can make a big difference.

/me goes back to lurking

Wow, 5.3? Impressive. Reading off a Virident FlashMAX II, I'm only getting 3.7, which puts it at about 1/3 the Virident's nominal bandwidth. I'm not seeing any change when I adjust the read ahead buffer count. What other settings could be affecting it?

That's still a huge improvement over Zlib. The Zlib compressed files are really slow both in and out, definitely want to reconsider that default. Storage is cheap again, and having such dramatically improved performance is nice. Though of course it would be better if the compressing and decompressing used more CPU cores. At least something to consider with Fusion's native format. I understand that you won't want to touch EXR or TIF or whatnot.

Of course that raises other issues. Fusion RAW is fine for disk caches and precomps, but anything you are interchanging with another application isn't going to be that unless there are plugins for other tools to use it. So we're still stuck with EXR for most things. EXR decompresses with more CPU resources, but at least Zip doesn't go past 50% here. And uncompressed EXR's for me aren't any faster than Zip compressed ones. 2.5 fps off the Virident, using no more CPU than what cached playback takes. Same with DPX's. So there's some special sauce for the RAW's then? It's the same amount of data, uncompressed in all 3.

The Frames At Once option is great, but it doesn't affect the interactive rendering. There's no pre-fetching done for the LD's to read and decompress and cache frames in the background. Sure, on really large comps the LD's might only be 10% of the processing time, but it's also a limiting factor on anything else happening. No matter how fast or slow the regular tools are, the LD and SV are always a bottleneck. I've never seen a traditional comp (meaning combining footage, not a particle system or motion graphic type) where that wasn't the case, it's just a matter of what percentage of the time it is.

One other thing I just tried... DoD doesn't seem to speed up RAW reading at all. Is that expected?

Thanks Guys,

Great information you are both sharing. I'll try different disk cache scenarios for my comps, as well as the render frames at once. I've been trying to do disk cache from the network, but it looks like my I/o speed is what would limit that. I'll try switching to a faster network drives and/or bring disk caching local and see if that helps in comp play back speed. I'm working with multiple 4k green screen composites in the same flow, so I understand I/O is extremely important, but I'm not happy with a disk cache taking 1 frame a second to fill into RAM. I'll keep posting what I'm finding so hopefully others can learn from my mistakes and findings.

I'm having a hard time finding that sweet spot for using the network render for disk caches. clusters, background rendering and previews. Maybe I can try and share out my SSD RAID over the network and make that be the location all the Fusion caches are written too.... i'll keep experimenting.


Really appreciate the feedback on how to start troubleshooting this. Thanks!

What path map does clustering use....I can get clustering to "render" on my network node workstations, but they never finish a frame.... They just get's stuck on the one frame they were initially assigned. I have Clustering set to 30 frames around the playhead. I feel like maybe the nodes can't see the network location that's set for clustering. Thanks.

What path map does clustering use....I can get clustering to "render" on my network node workstations, but they never finish a frame.... They just get's stuck on the one frame they were initially assigned. I have Clustering set to 30 frames around the playhead. I feel like maybe the nodes can't see the network location that's set for clustering. Thanks.

The comp is sent to the render machines with whatever comp paths and pathmaps are set. You need to set up appropriate pathmaps on the render machines or use comp pathmaps.

Thanks Chad,

I set the globals to match the master, but when the node loads the DFTmpClustera12812... the path maps are blank in the preferences under the Global Settings.

Render nodes still just hang on a single frame. If I try to cluster with just the master machine, the Render Node on the master shows that it's rendering but doesn't save or cache a file and hangs on a single frame. But if I move the playhead, the Render Node will update to a new frame.

Does the clustering actually write files?

Unfortunately, Cluster no longer works. no idea why it's still in the menu.

That's a shame. We have fast enough networks and cheap enough machines to make this production viable.

Cluster came in a while back while I was at Digital Muse, where we had a killer setup(at the time) and actually competed very well with Henry on interactive client work. I've personally asked product manager to bring it back, or at very least remove it from the menu

That's really too bad it's not working.... it could be such a great feature for users that, like Chad mentioned, have fast network storage and idle machines waiting to be sent frames.

I'll try to find a workflow to use more disk caching as a way to speed up interactive sessions.

Problem eyeon had is there was little request for it, which meant priority was low. I can see how this is more viable now, and user voice will help revive that. I want it