Jump to: Board index » General » Fusion

File Caching on Loader Problems.

Learn about 3D compositing, animation, broadcast design and VFX workflows.
  • Author
  • Message
Offline

Kel Philm

  • Posts: 125
  • Joined: Sat Nov 19, 2016 6:21 am

File Caching on Loader Problems.

PostTue Feb 14, 2017 2:07 am

So I've been running through a few tests and found some strange results when dealing with File Caching on my Loaders.

I am getting a bit annoyed with how slow the file caching plays back, its seems nearly as slow as the exr or Quicktime that the loader is reading. It is caching to Raw, so I tried adding a new loader (not cached) in place of the old one that directly reads the Raw File cached sequence and it is magnitudes faster. I've messed around with various modes but I can't understand why the original file cached loader is not running anywhere near as fast as the direct raw file loader, surely it should just read the same raw files?

Are there any Fuse's out there that can convert the file to raw and read that instead (Like file cache is supposed to do!)

I also have not been able to get the Network render for file cache to work, I can network render my comps (Though Quicktime loaders seem very buggy).

Running Windows 10, Caching to raided SSD's, Footage on Raided HDD's. GTX 1080, Xeon 2697 v4. 64GB

Any help appreciated.
Offline

Kel Philm

  • Posts: 125
  • Joined: Sat Nov 19, 2016 6:21 am

Re: File Caching on Loader Problems.

PostTue Feb 14, 2017 5:26 am

OK, I have Found problem with Network Rendering for Cache files, my cache drive only had read share permissions over the network, not write, changed this and problem has been fixed.
Offline

Vito La Manna

  • Posts: 15
  • Joined: Tue Jan 13, 2015 2:50 am

Re: File Caching on Loader Problems.

PostThu Feb 16, 2017 2:38 pm

Why are you caching your loaders? Fusion does that for you.

Sent from my GT-N5110 using Tapatalk
Offline

Chad Capeland

  • Posts: 1502
  • Joined: Mon Nov 10, 2014 9:40 pm

Re: File Caching on Loader Problems.

PostThu Feb 16, 2017 6:17 pm

Vito La Manna wrote:Why are you caching your loaders? Fusion does that for you.


There's different layers of cache, though. You can cache the images to RAM, the images to files, the files to files, etc..

Loader Cache allows you to cache a file across multiple loaders and comps.
Chad Capeland
Indicated, LLC
www.floweffects.com
Offline

Kel Philm

  • Posts: 125
  • Joined: Sat Nov 19, 2016 6:21 am

Re: File Caching on Loader Problems.

PostThu Feb 16, 2017 8:49 pm

I have been looking at trying to improve the performance of Fusion while I am comping.

I am working with 4K renders and when they have a number of elements in there the system Ram (64GB) gets used up pretty quickly by Fusion. I am trying to reduce processing bottlenecks manually by using file caching (and forcing RAM caching) and also trying to avoid issues with Quicktime files by having them file cached to raw files which improve speed and eliminate slow QT processing (which it does).

I also have a very quick PCIe SSD that I am utilising as a file Cache to improve performance of my raided HDD drives where the source is stored.

From my testing I have noticed that RAM cache priority is applied from the viewed node backwards along the flow which means changing any node back in the flow that is no longer cached requires that chain to be reprocessed back down the flow. File Caching can eliminate some of this re-processing. Its not always practicle but carefully used File Caching should be able speed up my workflow (and also final renders) dramatically. Also using network render and pre-render for the file caching may also speed things up. Proxy and ROI are other tools that I am using as well to improve performance.

When you comp a lot, 20% performance here and there can make a big difference. How much easier and satisfying the job would be if everything was realtime.
Offline

Chad Capeland

  • Posts: 1502
  • Joined: Mon Nov 10, 2014 9:40 pm

Re: File Caching on Loader Problems.

PostThu Feb 16, 2017 8:52 pm

Just to clarify, are you using Loader Cache or are you disk caching the Loaders?
Chad Capeland
Indicated, LLC
www.floweffects.com
Offline

Kel Philm

  • Posts: 125
  • Joined: Sat Nov 19, 2016 6:21 am

Re: File Caching on Loader Problems.

PostFri Feb 17, 2017 3:05 am

Disk caching the loaders.
Offline

Chad Capeland

  • Posts: 1502
  • Joined: Mon Nov 10, 2014 9:40 pm

Re: File Caching on Loader Problems.

PostFri Feb 17, 2017 3:22 am

Kel Philm wrote:Disk caching the loaders.


Yeah, don't do that. Try the Loader Cache. As long as you're using a fast file format, it will be much faster.
Chad Capeland
Indicated, LLC
www.floweffects.com
Offline

Kel Philm

  • Posts: 125
  • Joined: Sat Nov 19, 2016 6:21 am

Re: File Caching on Loader Problems.

PostFri Feb 17, 2017 4:17 am

Would a Loader Cache be faster than a File Cache (.raw) stored on the same drive? I assume .raw is Fusions optimal file format given it uses this for Caching, also my tests showed that .raw is much faster than any other format when reading into memory (EXR is pretty good though).

I've checked the loader cache and it appears to just copy the file over to the folder you assign it to, not sure how that is useful unless you specify a faster drive than the source or am I missing something. My tests did not yield any great results with the Loader Cache.

I don't know whats going on under the hood though and my tests are hardly scientific so happy to learn.
Offline

Chad Capeland

  • Posts: 1502
  • Joined: Mon Nov 10, 2014 9:40 pm

Re: File Caching on Loader Problems.

PostFri Feb 17, 2017 6:43 am

Correct, it's just copying the files to a faster drive. But the disk cache has to read the files in the LD, create images from the data, create new files, put the image data into those files, then read those files again and create new images. If you're trying to just get faster reads, then yes, converting to .raw or .dpx or something fast like that will help, but you shouldn't be doing that in the disk cache but as a separate render. There isn't really a need to keep the LD interactive, as the input isn't likely to ever change.
Chad Capeland
Indicated, LLC
www.floweffects.com
Offline

Kel Philm

  • Posts: 125
  • Joined: Sat Nov 19, 2016 6:21 am

Re: File Caching on Loader Problems.

PostFri Feb 17, 2017 9:44 pm

Thanks Chad,

This seems to be the crux of my original issue. I assumed once I had pre rendered the loader as a file cache it would only bother reading the cache file when needed, not also reading the original file (that as you stated usually never changes) and processing it each time, which to be honest seems a bit pointless. Would locking the cache force this perhaps?

I will log this with BM as I think its implementation needs to be looked at, any large comp with a number of QT loaders slows to a crawl because of decoding when this can be avoided with a pre-processing trade off.
Offline

Wicus Labuschagne

  • Posts: 21
  • Joined: Thu Oct 01, 2015 2:03 pm
  • Location: Cape Town, South Africa.

Re: File Caching on Loader Problems.

PostMon Sep 11, 2017 4:57 pm

It feels like Fusion's Loader Cacheing is incredibly slow because it checks if the files still exists on the network. This check itself takes the same amount of time than just reading the file on a congested network.

How can this "check" be switched off or time delayed?
Offline

Rick van den Berg

  • Posts: 124
  • Joined: Tue Jun 02, 2015 7:47 am

Re: File Caching on Loader Problems.

PostThu Sep 14, 2017 7:51 am

i like the idea of the caching, it sounds simple and effective, but it feels kinda buggy/slow on my pc. i started a topic a while ago: viewtopic.php?f=22&t=63561
Offline

Wicus Labuschagne

  • Posts: 21
  • Joined: Thu Oct 01, 2015 2:03 pm
  • Location: Cape Town, South Africa.

Re: File Caching on Loader Problems.

PostThu Sep 14, 2017 2:20 pm

Thanks Rick,

But I'm trying to stop Fusion from constantly checking if the local files are the same as on the server. When I unplug my LAN cable fusion is really fast when "play cache when original files can't be located" is selected. As soon as I plug my LAN cable in it grids to a halt.
Offline

Niels van der Heijden

  • Posts: 21
  • Joined: Wed Apr 20, 2016 11:29 am

Re: File Caching on Loader Problems.

PostThu Sep 14, 2017 3:08 pm

Did anyone report a bug about this issue?

Return to Fusion

Who is online

Users browsing this forum: No registered users and 2 guests