Network Rendering Fails, Needs Serious TLC

Network rendering used to be usable. now its not.

UPDATE : I tried a simple comp - 2 backgrounds, one with animated fast noise into a merge into saver .jpg sequence. even this simple comp won't render on network render nodes. yes, all same ver of Fusion installed.

the network manager in fusion was never particularly stable, but seems worse in current build. Locks up / beachballs and is totally unresponsive. requires app force quit.

UI never remembers its settings so every time you open the window up you have to reset the entire layout.

network manager has no auto- retry of network nodes or refresh ?

the use of the terms slave and master has become very unPC - render nodes and controllers ?

render machine failures -

no useful info displayed in network manager or render node why render failed in UI. why can't we have plain language error messages to fix the problems ? right now its pure guessing.

actual remote control of render nodes to adjust prefs to get them to work rather than having to remote screen share them ?

render nodes sometimes put up error dialogs which is good, but the dialogs sit there waiting for input. if you never look at the render nodes, they may sit there for weeks with who knows how many ( can be hundreds !!! ) error dialogs piled up on top of each other. How about a log window that scrolls any error msgs ? one that doesn't force any user input into it ?

I know some of these are obvious fixes, but they are badly needed. Changing render node error dialogs to error log window is pretty trivial code change.

Just starting up project that will require network rendering and surprised that what used to work ok, is now a mess. are ANY resources being put into network rendering ?

S

I don't think the network renderer really changed (aside from the UI) in recent years. I have also voiced my frustrations with it before. I love how simple it is to set up but hate how fragile it is.

Especially when I was using Fusion 16 (I know rookie mistake) at a studio where I had access to 5 machines as render slaves, I almost got PTSD. In the six months I was working for them I couldn't rely on Fusion (or the render manager) once. Every evening or weekend I had to use a remote desktop to check on the render status and what went wrong. The render manager in conjunction with Fusion 16 was the main issue as it doesn't seem to be two independent tasks. If Fusion 16 hung up (which happened frequently on mid sized to bigger shots) the render manager hung, too. Then I had to restart the render manager from home using remote desktop, on 5 machines.
I hope there was a way for the Render manager to work independently from the Fusion renderer to kill the task after a certain time. I believe the heartbeats are supposed to achieve exactly that but in my experience that didn't work at all. But this is more likely an issue with Fusion 16 (probably connected to the VRAM management issues that I'm constantly talking about).

But my biggest issue with the render manager itself is how it schedules tasks. Whenever it fails at a frame, it simply stops everything and every other slave will be assigned that frame until they've all disabled themselves. This shouldn't happen. I don't mind seeing one frame missing in the morning, but 1500 missing frames because someone accidentally didn't render frame 73 from one of the shots I got is annoying.
I really hope there was a setting to force Fusion to keep rendering the other frames and schedule the failed frame last.

More control of the slaves and job dependencies would be great but isn't something that would be on my priority list. Fixing the (self inflicted) stalling and stability issues would make the render manager good for me.

That being said, most of my issues were fixed/bypassed by using SirEdrics amazing comp splitter script.
It's in Reactor and essentially splits your composition into little chunks and sends these chunks to the render manager. Pretty much like Deadline if you know that.
This fixes the issue with the missing frames. Because as it encounters a missing frame it will continue with the next job that is a few frames later.
And it also fixed the crashing/hanging as after each job it will clear the RAM and VRAM. Which means it will render fast and reliably.

In theory this would slow down rendering as for every new job it would recalculate everything, including still images or masks that may not need to be recalculated. But in my experience, the slow down you experience due to the VRAM filling up is much much bigger than the few seconds it takes to recalculate some static images.

The other option would be to use a third party render manager (like Deadline) but that doesn't work with Fusion 16, only up to 9. But Deadline is a great tool that I'm using personally for most render management without any issues and it has features like shutting of the computers when they're idle.

But I don't want to sound too pessimistic as I'm truly thankful to have any form of render management software bundled with Fusion. That + free render nodes can be AMAZING when you know how to work around the issues.
I hope we'll get some updates to the render manager in the future but until then I think most of your issues will be fixed when you're using the comp splitter script.

I hope this helps you

thanks. getting into using a 3rd party render app probably isn't realistic right now. I'd go back to Fusion 9 actually if its gonna work , maybe. Fusion 16 seems to have stable GPU rendering that works and hate to give that up, but it also might be part of the problem in terms of machines having mismatched abilities - AMD on 2, none ( intel ) on 3rd but all support Metal. I suspect maybe even not enough ram, but w/o any usefull error messages, who knows. I can try things until maybe it works. I'd hate to reduce the project from current float16 down, but that may also be a problem.

The worst part is the most recent failure had only one 3rd party OFX in it, which I disabled because I could add it back onto the final render when it was done - filmconvert grain / look. My guess is the latest failure is probably even having a 3rd party OFX in use. I have some other ones that I licensed for the render machines. While my render machines may not be the fastest, they churn out frames while I work main machine. Oh joy, just checked and got on frame completed before the render manager beachballed and game over.

So I'm not sure what the solultion is except maybe going backwards....

S

If it doesn't render at all then there are only a few things that could be wrong. Most notably the Path Maps. Have you checked that the path that each computer tries to access is correct?

who what which path ? this was a fully function setup last year.

w/o any useful error msgs, no clue.

S

If it doesn't render at all then there are only a few things that could be wrong. Most notably the Path Maps. Have you checked that the path that each computer tries to access is correct?

Your computers all need to access the same storage and they all need to find the right path. For example, if your Projects are all on your drive X on computer A then your Path could be something like

X:\Projects\Fusion\

but for your computer B this might be:

Y:\Projects\Fusion\
and there is no X: drive at all from the perspective of computer B

then you need to make sure (using the Path Maps Settings on your Fusion Render Nodes)
that computer B knows that when it gets the Path X:\ that it should look in Y:\ instead.

This can also be used to make all Render slaves look on your Master computer (or a shared network drive) for Fuses, Macros, Brushes etc so you only need to install it once on your master.

ok, I'm not near my machines at the moment. when you put a job out to the network, shouldn't the machine submitting the render comp have a path like smb/sending machine/volume/folder as part of the job so that all network machines find it ? I mean hardwiring a single path may be ok for a large long term project, but for things that come and go on very short term basis its not practical to be adding paths to every folder that submits a job.

this setup has been working fine for a while and I'd never messed with paths. Its probably something simpler than this thats broken.

S

Your computers all need to access the same storage and they all need to find the right path. For example, if your Projects are all on your drive X on computer A then your Path could be something like

X:\Projects\Fusion\

but for your computer B this might be:

Y:\Projects\Fusion\
and there is no X: drive at all from the perspective of computer B

then you need to make sure (using the Path Maps Settings on your Fusion Render Nodes)
that computer B knows that when it gets the Path X:\ that it should look in Y:\ instead.

This can also be used to make all Render slaves look on your Master computer (or a shared network drive) for Fuses, Macros, Brushes etc so you only need to install it once on your master.

That more meant as a general example. You only have to change the Path up until what is different.
But if you say everything worked fine then maybe this isn't the issue.
I'd still double check.
Because other than the Path Maps, there aren't many things that could be wrong.
Maybe the name or IP address of the Master machine?
Maybe your firewall?
Can you render a composition through the render manager only in your local machine?

My guess is the latest failure is probably even having a 3rd party OFX in use.

I'm not terribly experienced with the render architecture, so I may be completely off-base here. An OFX plug-in that is not present on a host will cause an error pop-up in an interactive session, even if the node is passed through or completely disconnected from the rest of the flow. I don't know how that error is handled by a render or console node. If it throws an error and is waiting for an instruction about how to handle it, that would account for an apparently hanging render.

If that's the case, then deleting the OFX node entirely should solve it.

yes I can get the local machine to render ok, even via render manager.

a simple test comp with a pair of backgrounds into a merge into a saver won't render. no external media, fonts, 3rd party OFX.

no firewalls

designation of master machine I may look at. I recal changing that recently.

S

That more meant as a general example. You only have to change the Path up until what is different.
But if you say everything worked fine then maybe this isn't the issue.
I'd still double check.
Because other than the Path Maps, there aren't many things that could be wrong.
Maybe the name or IP address of the Master machine?
Maybe your firewall?
Can you render a composition through the render manager only in your local machine?

I've been using network rendering in Fusion Studio 16 + Fusion Render Nodes for the last few weeks, and very heavily in the last few days.



Generally it's been working reasonably well. I can render my comps between my three machines - two on macOS, one on Windows 10 - and it's definitely a big help in terms of overall render times. Three old slow machines probably don't quite make up to one new fast one, but it's much better than using only one of them.

My recent rendering hasn't involved any OFX effects, however I did do one series of renders that used Neat Video OFX, configured to use the GPU. It worked OK, no differently to any of the other network renders I've done. For that render I was using my two macOS machines.

That said, I do have a lot of problems. It's common for machines to suddenly stop rendering for no apparent reason, often requiring a Force Quit /End Task on the Render Node. This seems to happen a lot more on my main workstation, and therefore might correspond to the fact that I'm often using the workstation for other stuff while the render happens in the background.

I've also found that the machine running Fusion Studio (as opposed to the Render Node) is more prone to render halting/freezing. This was usually my main workstation, but on a few occasions I tried moving my dongle to my secondary workstation and running Fusion Studio on that instead, and run the Render Node on the main machine. And then I saw the same behaviour - the machine running Studio, ie the Render Manager, would be the one that had problems.

Right now I have this config:

1. Main machine runs both Fusion Studio and Render Node. Main Fusion Studio is set to NOT be a render client (File -> Allow Network Renders = no).
2. Second and third machines running Fusion Render Node
3. I load the comp on the main machine Fusion Studio, then when I'm ready to render it I first save the comp, then I click Render, enable network rendering, and then open the Render Manager.
4. In the Render Manager I have added the second and third machines, and also 'localhost', which is the Render Node running on the main workstation in parallel to Studio.

This setup still leads to occasional drop outs of rendering on the main machine - and very occasionally on the second macOS machine - but that seems to be less common than it was when I used Fusion Studio as the render client on the main workstation.

In addition to it possibly being a bit more stable, the big advantage of having both Studio and Render Node running on the main machine is that if (when) the main machine's rendering freezes, I can:
a) In Render Manager, right-click on localhost and choose "Don't Use Slave"
b) In Activity Manager, Force Quit Render Node
c) Re-start Render Node
d) In Render Manager, right-click on localhost and choose "Use Slave" again.

This results in rendering re-starting on the main workstation, and doesn't leave any gaps in the rendered frames, as the Render Manager re-assigns the frame(s) the node was trying to do when it failed, then allocates it new ones when it comes back.

This setup also seems to ensure that Fusion Studio, and its Render Manager, never crash themselves. Thus even if the main workstation's rendering does stop (perhaps when I'm rendering unattended), at least the other two machines can carry on.

I do think it depends on what the comp is and what nodes it uses. The comp I've been rendering the last few days (in multiple variations) is a 3D scene that uses the Software Renderer3D, not OpenGL. This is because one of my three render nodes, the Windows 10 one, doesn't have a GPU. I did some benchmarking and found it was faster to use three machines with software renderer than it was to use the two machines that do have a GPU with OpenGL. That Windows 10 machine has double the CPU cores and RAM of my main workstation; factoring that in with its lack of GPU results in it rendering at between 1 and 0.75x the main workstation's speed.

Eg on a 24,807 frame 3D scene I rendered recently:
Main workstation, macOS, 48GB RAM, 6 x 4.3ghz cores, 8GB AMD Vega 64 GPU: 11,857 frames
Server, Windows 10 (in a VM), 96GB RAM, 12 x 3ghz cores; no GPU: 7,148 frames
Secondary workstation, macOS, 24GB RAM, 4 x 3.5ghz; 3GB AMD R9 280X GPU : 5,803 frames

Regarding comp setup:

I've not done anything special, besides ensuring that of course I have PathMaps that correspond to all my disk locations, including the location where the comp file is saved (which is on a central network share). Then I've ensured that each node has those same PathMaps set up, and that they use the correct path for that machine.

This was especially vital given I have one Windows node, which uses very different paths to the two macOS machines (eg it needs "z:\..." instead of "/Volumes/Data/...")

Then I needed to ensure that 100% of nodes that touch the disk use those PathMaps. Loaders and Savers of course, but also any FBX nodes - this caught me out for a bit, because the FBX node does not automatically use PathMaps, unlike Loaders and Savers. By which I mean, if I add a Loader and point it to, let's say, "/Volumes/Data/blah.exr", it will use the path "Data:/blah.exr", because I have a PathMap that maps Data: to /Volumes/Data. Same with Savers. But this is not true of FBX nodes, so you have to remember to manually go through and edit their load paths to use the right PathMaps.

I would agree that network rendering gives too little feedback. In my case, when there's a problem like missing or wrong PathMaps I'm used to simply seeing "Failed", with no further explanation. It's then up to me to double check all the network mounts are working, then search through all the nodes to see if I missed a path somewhere.

And, very annoyingly, some issues don't lead to an error. Like missing FBX paths, which can cause a render node to simply not use the given 3D object, without throwing any error. Ditto missing fonts. You send 10,000 frames to render at 1 fps, and come back hours later to realise that half of them are useless because that machine was missing a 3D object or a particular font. And then it's not even easy to work out *which* half, short of writing a script to parse the render logs, so you end up re-rendering the whole sequence.

So my experience of network rendering definitely has not been smooth. It's a constant worry as to whether one or more nodes will fail after 10 minutes or 3 hours.

However, I'm still very much at the stage of thinking it's awesome I even have the ability to network render, and given my ageing hardware it's a huge benefit to be able to spread these long renders over all the hardware I have. My project has 20 minutes of 3D scene, which would take 36 hours to render if I did it only on my main workstation. Spread between three machines it was 14 hours, which was far more manageable.

Despite all the problems, I do love the feature. I love that, when it's working, it's incredibly quick and easy to use, such that I can use it for every render, no matter how small. Once it's set up and Render Node is running on my other machines, I can simply click Render in Studio, tick the box for "Network", and click go. Exactly as quick and easy as rendering locally, but 2-3x faster. It's also awesome that I can quickly enable or disable rendering on any machine at any time. I often start out rendering on all three machines, but then disable it on the main workstation if I need to do some work in Resolve. Then when I'm done with that I add the main machine back in again. It's very flexible.

tldr; but i will have to read because I've never managed to have it work : remote or even local machine usually show up once in the list and then never again...

tldr; but i will have to read because I've never managed to have it work : remote or even local machine usually show up once in the list and then never again...

You can right-click and "Scan for Slaves", and you can also right-click and "Add Slave". I think I did the latter, and just manually added each slave, like this:


Of course it requires there's no firewall between them (or the necessary ports are open; I don't know what ports are used.) In my case, they're all on the same LAN, no external firewall between them.

On the Windows 10 machine I guess the installer adds the necessary firewall rules automatically:


I've never had any issue with the clients not being detected or going offline when they should have been available. The issues I have all relate to crashing/freezing after rendering has started.

Thx, I'll check again but I've done (scanned and added) all you wrote, and my machines are also on the same lan... To be continued

A few notes to add now :

who is running on 10.15.x ? I'm suspecting this is jacked up OS permissions access where the render nodes aren't ok'd to access network volumes, they aren't popping up system requester to allow it.

as for mapping any paths, everything I see is about local things like fonts, temp, cache etc. I'm running only OS X machines so no slash and back slash problems.

render manager says remote machines report back "loaded and ready to render", then a few lines down frame X failed to render... but its a couple of merged backgrounds, all internal things into a saver, jpg

also reset local machine as render master as opposed to network render node which hasn't made a difference. my test comp has no external dependencies except save folder for which I've never in the past had to do anything to make work.... which brings me back to maybe some 10.15.x problem

S

Thx, I'll check again but I've done (scanned and added) all you wrote, and my machines are also on the same lan... To be continued

who is running on 10.15.x ? I'm suspecting this is jacked up OS permissions access where the render nodes aren't ok'd to access network volumes, they aren't popping up system requester to allow it.

I'm on Mojave on both my macOS machines. I had planned to upgrade my second workstation to Catalina at some point soon and probably will in the next few days. My primary workstation can't boot Catalina at the moment, as it's a Hackintosh on old hardware.

That said, I'm struggling to imagine how its security & privacy settings could affect this. Fusion doesn't attempt to control the OS in any of the ways that those settings normally affect. It doesn't access microphones or cameras, it doesn't try to control other applications. It just reads and writes files. I'm not aware of any security & privacy settings specific to network volumes.

as for mapping any paths, everything I see is about local things like fonts, temp, cache etc. I'm running only OS X machines so no slash and back slash problems.

So the Saver path you use will work identically on both machines? Eg if your saver path is "/Volumes/MyNetworkVolume/", both machines have that volume mounted, and under the same name? Ditto any Loaders?

The manual states that all machines also need to be able to load the composition from the same location, but when I tested this, it didn't seem to be the case, suggesting the Render Manager sends the comp to the slaves. So perhaps it doesn't matter where the comp is saved.

who is running on 10.15.x ? I'm suspecting this is jacked up OS permissions access where the render nodes aren't ok'd to access network volumes, they aren't popping up system requester to allow it.[/quote
That said, I'm struggling to imagine how its security & privacy settings could affect this. Fusion doesn't attempt to control the OS in any of the ways that those settings normally affect. It doesn't access microphones or cameras, it doesn't try to control other applications. It just reads and writes files. I'm not aware of any security & privacy settings specific to network volumes.

In 10.15 there is more granular access settings, specifically removable volumes and network volumes, user/docs folder which is great to disable and stop certain apps from filling it with crap you don't know is hiding in it. there are also some wonky access things for keyboard / mouse input that mess with wacom drivers.

as for mapping any paths, everything I see is about local things like fonts, temp, cache etc. I'm running only OS X machines so no slash and back slash problems.

So the Saver path you use will work identically on both machines? Eg if your saver path is "/Volumes/MyNetworkVolume/", both machines have that volume mounted, and under the same name? Ditto any Loaders?

Oh wait, the network volumes aren't mounted automatically on the render nodes. thats probably whats messing this up and why I had NFS auto mounter app running. standby it could be this stupid, well if there was a useful error msg like Volume not found / mounted...

Oh that fixed it !, thank you so much !!!!!! no idea why I had a senior moment on this, well, except,,, back to that even semi-useful error message would of saved so much time and hassle.

---------

The manual states that all machines also need to be able to load the composition from the same location, but when I tested this, it didn't seem to be the case, suggesting the Render Manager sends the comp to the slaves. So perhaps it doesn't matter where the comp is saved.

Oh wait, the network volumes aren't mounted automatically on the render nodes. thats probably whats messing this up and why I had NFS auto mounter app running. standby it could be this stupid, well if there was a useful error msg like Volume not found / mounted...

Oh that fixed it !, thank you so much !!!!!! no idea why I had a senior moment on this, well, except,,, back to that even semi-useful error message would of saved so much time and hassle.

That'd do it Glad it's working now!

Well I spoke too soon. test comps work fine. more complicated comp doesn't. in fact in a total slap in the face, one network machine will render it, the original machine it was created on and rendered before currently isn't, and the one problem machine continues to do just that.

there is one 3D text node. I did a sync of fonts between machines using chronysnc. there are no 3rd party OFX, 100% factory nodes.

reinstalled fusion on render machine, checked its 3rd party OFX were up to date and properly activated, font sync, remote volumes auto mounted and working... I give up. maybe actual production comps will work, maybe not.

without useful error messages to track down the problem this is total guessing and thats jus not acceptable. this has to be fixed. the closest I got to one was opening fusion console complaining no master renderer - there is, in fact the machine its open on, even setting the remote render node - as per manual - did't seem to work for this one comp.

Oh wait, the network volumes aren't mounted automatically on the render nodes. thats probably whats messing this up and why I had NFS auto mounter app running. standby it could be this stupid, well if there was a useful error msg like Volume not found / mounted...

Oh that fixed it !, thank you so much !!!!!! no idea why I had a senior moment on this, well, except,,, back to that even semi-useful error message would of saved so much time and hassle.

That'd do it Glad it's working now!

Sorry to hear that. Something must be set up wrong somewhere.

I 100% agree the lack of useful error messages is very annoying. That said, it should still be possible to get it to render OK.

If you can upload your comp file then I can have a go on my machines. Also show a screenshot of Preferences -> Global -> Network settings for Fusion Studio on whichever machine is running Studio during the attempted network render.

Hey thanks. I fixed it. There was a single JPG being used as a faux reflection that I had swiped from another 3D apps presets / supplied media. once I moved that to the source project folder and swapped it in, everything worked everywhere... Oh how nice a message to the effect of "media not found at path Computer/volume/drive/folder would of made finding this easy. As for the other machine that found it, probably had drive mounted from totally unrelated things and happened to work thru dumb luck. I will be more vigilant to make sure all source media goes into project folder, or at least project Volume.

Sorry to hear that. Something must be set up wrong somewhere.

I 100% agree the lack of useful error messages is very annoying. That said, it should still be possible to get it to render OK.

If you can upload your comp file then I can have a go on my machines. Also show a screenshot of Preferences -> Global -> Network settings for Fusion Studio on whichever machine is running Studio during the attempted network render.

One of the best things I ever did was add a button to my Loader default that copies the file into the proper asset folder for the shot.

These will obviously need to be tweaked for your pipeline, as they're designed for use at Muse.

A more generalized and less error-prone approach would be to make a sanity check script that always runs on submission to verify pipeline conformity.

edit: Deleted old version of the MakeLocal package. See below for an update.

that is quite cool ! thank you - I'm sure it took you way too much time to put together, while the devs could of added that super useful feature with far far less time and effort. It kind of really makes me crazy that users have to resort to DIY fixes like this.

One of the best things I ever did was add a button to my Loader default that copies the file into the proper asset folder for the shot.

These will obviously need to be tweaked for your pipeline, as they're designed for use at Muse.

A more generalized and less error-prone approach would be to make a sanity check script that always runs on submission to verify pipeline conformity.

I'd have still done it even if there were better error reporting. Our artists just find it crazy convenient to be able to download an image to their desktop, drag it into Fusion, then click a button to send it to the right location on disk.

I consider it a personal challenge to prevent artists from ever needing to open a file explorer.

Hey,
It is indeed very cool...
Thank you very much... but could explain me again the best way to install the files for them to be "defaulted" (is it simply drag n drop in the settings folder in Fusion of my mac ?).

In my opinion, the lua file goes to Fusion > Scripts > Tool, and the .setting into Fusion > Macros, but I could be wrong...
The second question is : how to tweak em given that I'm no developer (maybe a little script kiddie) ?

Hmmm... I can generalize it to make it easier to adapt to another pipeline, but I don't have a Mac to test with to be sure that the changes I make will work anywhere other than Windows.

In any case, defaults go in the Defaults folder, but you definitely can not simply drop this into your folders and have it work because it's currently hard-coded to reference the script at a specific location on our S: drive. And unless your project structure happens to exactly match ours, it'll copy the images to who-knows-where. (Actually, on a Mac it won't do anything at all.)

I'd like to eventually generalize all of my pipeline tools to make them easy to adapt to other facilities, but for now, this is provided merely as an example, not a deployable solution.

Hmmm... I can generalize it to make it easier to adapt to another pipeline, but I don't have a Mac to test with to be sure that the changes I make will work anywhere other than Windows.

In any case, defaults go in the Defaults folder, but you definitely can not simply drop this into your folders and have it work because it's currently hard-coded to reference the script at a specific location on our S: drive. And unless your project structure happens to exactly match ours, it'll copy the images to who-knows-where. (Actually, on a Mac it won't do anything at all.)

I'd like to eventually generalize all of my pipeline tools to make them easy to adapt to other facilities, but for now, this is provided merely as an example, not a deployable solution.

I was able to directly drag and drop the script into a comp, it didnt show up under Scripts/tools or anywhere else.

Is there a comp path var that can be pulled ? as in Read Comp.filepath and set that as the copy to destination ? seems thats about all I'd change / fix. sorry but never touched LUA or I'd fix it. FWIW hated JS (AE) for its arcane syntax, really like basic C ( Shake RIP ! ) or assembler

S

I was able to drop it too, abd there's a Default folder in users/me/library/applicatioàn support/BMD/fusion...

Opening the 2 files rebeals some code like
S:/net_exe/BlackMagic/fusion9/scripts/Tool/buttonScripts/Loader_MakeLocal.lua\", args)\n
or newlocation = newlocation in the lua one, convincing me it can be adapted without pain....

I'll make some tests later on.
Thx anyway

When the .setting file is in the Defaults, it will replace the normal Loader. Just add a Loader to your comp, and you should see that it has an extra button. In fact, two extra buttons. I believe the ExplodeEXR script button is on there as well.

Actually…

Here's a cleaned up version that should be easier to adapt to other pipelines. It's still Windows only; I have some code to detect the OS around here somewhere, but I can't remember where I put it.

Lines 218 - 226 are the most obvious Windows-specific code, but there may also be some folder delimiter errors (\ vs /) that will show up in a non-Windows file system.

When the .setting file is in the Defaults, it will replace the normal Loader. Just add a Loader to your comp, and you should see that it has an extra button. In fact, two extra buttons. I believe the ExplodeEXR script button is on there as well.

True and true. So far I'm ok with that and let it in Defaults.

Actually…

Here's a cleaned up version that should be easier to adapt to other pipelines. It's still Windows only; I have some code to detect the OS around here somewhere, but I can't remember where I put it.

Lines 218 - 226 are the most obvious Windows-specific code, but there may also be some folder delimiter errors (\ vs /) that will show up in a non-Windows file system.

Ok, done (spotted at the right places). "Make local" is now the only button and of course doesn't work because of non specific code.
I admit I cannot replace the lines (I saw them in Wing personnal, a OSX Python IDE) because I'm not experienced, but so far I can post what's inside a (me/library/application support)bmd/...) Fu folder in OSX and the error in the Fu console.

What a pity, "make local" is a great idea which prevents from wasting time in useless clicks. How could we use it to define a path and open in a new window asking where to displace or duplicate a file to ?

That's not the error I would expect, actually. I'd have thought it would either silently fail or raise an operating system level error when the 'md' command didn't work.

I wonder, did you save the comp first? It won't work if the comp file doesn't yet have a filename since it's looking at the difference between the comp's location and the asset's.

If you wanted a pop-up window to ask the user where the asset should go, the simplest way would be to insert a Pathbrowse control in the AskUser dialog (see the Fusion Scripting Guide page pp 43 - 47), then replace the newlocation assignment a few lines later with that result.

However, that does open you to the possibility of human error—artists could place assets in off-pipeline locations. The point of this script is to A) give a very easy way to manage the assets, and B) reduce fragmentation in project folders.

I stole a bit of time from my morning walk today and made an attempt to make it work on Mac and Linux. I have no ability to test this, though, so let me know if it needs to be adjusted. If it's all good, I may just release it on Reactor.

edit: Updated script to fix a possible problem. Scroll on down.

I wonder, did you save the comp first? It won't work if the comp file doesn't yet have a filename since it's looking at the difference between the comp's location and the asset's.

No I hadn't. Now I did + used your new DL files and it works, with "Assets are already local. Aborted." in case of...
Which leads me to the thought I misunderstood what's supposed to be "remote" as opposed to "local" in our topic. Of course, it's about the network level (but I thought a file on an external drive would also be considered as not local (enough ) ).

If you wanted a pop-up window to ask the user where the asset should go, the simplest way would be to insert a Pathbrowse control in the AskUser dialog (see the Fusion Scripting Guide page pp 43 - 47), then replace the newlocation assignment a few lines later with that result.

I will one of these days, for I find it very interesting anyway.

However, that does open you to the possibility of human error—artists could place assets in off-pipeline locations. The point of this script is to A) give a very easy way to manage the assets, and B) reduce fragmentation in project folders.

Agreed. I got it

I stole a bit of time from my morning walk today and made an attempt to make it work on Mac and Linux. I have no ability to test this, though, so let me know if it needs to be adjusted. If it's all good, I may just release it on Reactor.

Well, what I tried worked, but did I really tried what you expected, this is another story
Anyway feel free to propose a test protocol, no problem. 12 years ago, I did a lot for VLC guys to make tons of tests on ProRes, I'm glad I can help you too

No I hadn't. Now I did + used your new DL files and it works, with "Assets are already local. Aborted." in case of...
Which leads me to the thought I misunderstood what's supposed to be "remote" as opposed to "local" in our topic. Of course, it's about the network level (but I thought a file on an external drive would also be considered as not local (enough ) )

What are the paths to the asset in the Loader and your comp, please? I have a theory, but I need data.

There's a possibility for a false positive if the particular folder the script is comparing happens to have the same name in both paths. Working on a fix for that right now.

Potential fix for the issue I anticipated (but possibly not for the one you actually had; that remains to be seen).

Thanks for posting this Bryan. I had a go on macOS and had a look through the script.

It needs a couple more fixes to work on macOS and Linux. I believe all the following lines need to change:
Line 56:

Code: Select all

ASSETFOLDER = [[elements\]

Line 124:

Code: Select all

path = string.gsub(path, '/', '\\')

Line 126:

Code: Select all

local elemtable = split(path, "\\")

Line 187:

Code: Select all

local pathtable = split(fp.FullPath, "\\")

Line 213:

Code: Select all

path = path..j.."\\"

Modifying those got me past the "Already local" error Sam pointed out, and to the point where it wanted to copy
"/Volumes/1TBSSD/Black1920x1080.png"
to
"/Volumes/data/projects/Fusion/Comps/elements/Black1920x1080.png/Black1920x1080.png"

Then it runs the following copy command:

Code: Select all

cp "/Volumes/1TBSSD//Black1920x1080.png" "/Volumes/data/projects/Fusion/Comps/elements/Black1920x1080.png/"

It seems to be trying to use a folder with the same name as the image file. This causes the copy to fail, because that directory does not exist - I guess this wasn't part of the earlier mkdir command, though I didn't debug that to be certain.

That's as far as I got in looking through it. I'll post this now as you're looking at it now yourself.

Thanks; I had a suspicion that the slashes would create problems, but without a test environment my ability to modify the script is limited. Windows 7+ is really forgiving of slash direction—a path that uses both will usually resolve.

The target folder should have been created a few lines earlier with a mkdir -p <path>, around line 238 (actual line number in my local copy has probably changed, since I've been making edits).

If you wouldn't mind testing to see if at least that part of the script is failing, I'd greatly appreciate it!

Hi Bryan

Here's a file modified (relative to your most recent upload) to work 100% on macOS. Well, 100% assuming it's correct that it creates a folder named the same as the image, even when the image isn't an image sequence? I wasn't sure about that.

Eg it copies to "/Volumes/data/projects/Fusion/Comps/elements/Black1920x1080.png/Black1920x1080.png" when the original file is "/Volumes/1TBSSD/Black1920x1080.png"

Loader_MakeLocal.lua.zip

(3 KiB) Downloaded 126 times

And here's a link to a colour-coded Diff between your version and this one, to make it a bit clearer what I changed.

Some of the changes are differences in debug lines, where I uncommented a couple and forgot to comment them, and I think disabled a couple you had had active.

Yes the mkdir and cp is working fine.

Great; thank you! I'll make a couple more tweaks, and then get it in Reactor probably tomorrow.

Well, 100% assuming it's correct that it creates a folder named the same as the image, even when the image isn't an image sequence? I wasn't sure about that.

That's actually an artifact of a bad text field title. It should probably be "Subfolder name" or something of that sort. If you're moving a single image, you could enter "Stills," and that's where it will go, without changing the filename itself.

I'll do a little additional retooling to make that clearer, and perhaps offer an option for a default subfolder for single-frame elements other than the element's own name.

That's actually an artifact of a bad text field title. It should probably be "Subfolder name" or something of that sort. If you're moving a single image, you could enter "Stills," and that's where it will go, without changing the filename itself.

Oh I see what you mean. In the Element Name dialogue that pops up. I thought that was asking me if I wanted to rename the file during the copy operation. OK, fair enough.

Two other things I noticed:

1. Currently it doesn't function with PathMapped files, eg "1TBSSD:/Black1920x1080.png"

I haven't debugged it, but it appears that it fails only at the end, probably at the copy stage. It creates the directories, and updates the Loader's Filename: box with the correct path, but then it doesn't copy the file there, and so the Loader fails because the updated path doesn't contain the file.

2. I think you're aware of this: The script fails at an early stage if the Composition is unnamed/unsaved. I guess this is inevitable given it needs a comp path to do its work, but perhaps this could be detected for and a suitable message shown?

Currently what happens is you click MakeLocal and nothing happens. Looking in Console shows an error on line 94, inside the parseFilename function, which fails because parseFilename is passed an empty string for the comp name.

Perhaps on this line (line 186 in my version of the script):

Code: Select all

local fp = parseFilename(composition:GetAttrs().COMPS_FileName)

there could be a check for empty string or nil for COMPS_FileName and a message shown, indicating to the user that the comp must be saved first?

Two other things I noticed:

1. Currently it doesn't function with PathMapped files, eg "1TBSSD:/Black1920x1080.png"

Good catch! I'll expand the pathmaps in the next version

2. I think you're aware of this: The script fails at an early stage if the Composition is unnamed/unsaved. I guess this is inevitable given it needs a comp path to do its work, but perhaps this could be detected for and a suitable message shown?

Yep. I can certainly add error handling for that. It never comes up at Muse because we only create comps with a script, which ensures that the file is saved in the correct place with the correct name before the artist can even begin working.

Here is what I hope is the finished product!

Added error handling for unsaved comp files.
Improved dialog to make it clearer that it's expecting a sub-folder name.
Preloads the subfolder name field with 'stills' or 'video'. These default names are customizable in the globals.
Improved in-line documentation.
Removed debugging code.
Expands mapped paths prior to processing.
Squashed that extra separator character at the start of the filename.

Now I've just gotta make an atom and get it submitted to Reactor!

thank you for the time and effort in building this out.

I'm getting conole error "...Downloads/MakeLocal/Scripts/support/Loader_MakeLocal.lua:140: attempt to index global 'tool' (a nil value)"

I drag and dropped it from finder in.

the Comp Not Saved warning does work, but the dialog button labeling isn't clear what will happen. Abort ? what ? You mean Cancel Loader ?

Also moved it into system SSD/Library/App support/BMD/Fushion/Scripts/support and it doesn't load. where is it supposed to go to load ? it didn't replace the defualt loader. also copied Loader_Loader/setting into /Defualts

S

I'm getting conole error "...Downloads/MakeLocal/Scripts/support/Loader_MakeLocal.lua:140: attempt to index global 'tool' (a nil value)"

I drag and dropped it from finder in.

Yeah, that won't work. It must be run by clicking the button on the Loader. Otherwise it doesn't know what tool it's supposed to be manipulating.

the Comp Not Saved warning does work, but the dialog button labeling isn't clear what will happen. Abort ? what ? You mean Cancel Loader ?

Would "Script Halted" make more sense? I'm not sure if I can suppress the OK/Cancel buttons on the Ask User dialog. I realize those are confusing, but I don't think it's worth the time at the moment to build a UI Manager dialog instead—that error message is currently three lines, but a new one would be about thirty.

Also moved it into system SSD/Library/App support/BMD/Fushion/Scripts/support and it doesn't load. where is it supposed to go to load ? it didn't replace the defualt loader. also copied Loader_Loader/setting into /Defualts

If I follow you correctly, it sounds like the files are both in the right place. When you add a new Loader to your comp, the Inspector should look similar to this:

Capture.JPG (36.44 KiB) Viewed 5093 times

I've circled the button that launches the script.

Once executed, it should either tell you that "The asset is already local. Aborting." or show you this dialog:

dialog.jpg (15.85 KiB) Viewed 5093 times

Ok, screen shot of install, its not loading at all, at least not as new default loader and it doesn't seem to show up in any of the script menus which is why I went for dragn and drop

Screen Shot 2020-09-09 at 8.22.56 PM.jpg (234.74 KiB) Viewed 5090 times

the Comp Not Saved warning does work, but the dialog button labeling isn't clear what will happen. Abort ? what ? You mean Cancel Loader ?

Would "Script Halted" make more sense? I'm not sure if I can suppress the OK/Cancel buttons on the Ask User dialog. I realize those are confusing, but I don't think it's worth the time at the moment to build a UI Manager dialog instead—that error message is currently three lines, but a new one would be about thirty.

Also moved it into system SSD/Library/App support/BMD/Fushion/Scripts/support and it doesn't load. where is it supposed to go to load ? it didn't replace the defualt loader. also copied Loader_Loader/setting into /Defualts

[/quote]

If I follow you correctly, it sounds like the files are both in the right place. When you add a new Loader to your comp, the Inspector should look similar to this:

The script will never show up in any menus. That's the purpose of putting it in a folder (support) that Fusion doesn't know about.

Try this: Make a new default of any random tool. To do that, add the tool to the Flow, change a parameter a bit, then right-click the node and choose Settings > Save Default. Then check to see if a new file with that tool's name appears in your Defaults folder. If it does, then I'm stymied—it should be working. If it doesn't, then that folder isn't actually where Fusion is looking for its tool defaults.

In that case, open the console and paste this:

Code: Select all

dump(comp:MapPathSegments("Defaults:"))


That will show you a list of paths where Fusion expects to find its defaults.

indeed I got back -
1 = /Users/[my user account]/Library/Application Support/Blackmagic Design/Fusion/Defaults/
2 = /Library/Application Support/Blackmagic Design/Fusion/Defaults/
3 = /Applications/Blackmagic Fusion 16/Fusion.app/Contents/MacOS/../Resources/Fusion/Defaults/
4 = /Library/Application Support/Blackmagic Design/Fusion/Reactor/Deploy/Defaults/

so I installed into the root system SSD/Library/App support/... rather than the Users/My Account/library/app support/... one. After I did that save as default, the new tool showed up in the Users/My Account/.. location. I moved the 2 files from you over to there - users/my account/library/... and works now ! good

that said, the code may not be copying. I randomly loaded a mov into a comp, did save as (comp) to another local volume and hit make local. I got error the copy was already local.

Is the goal to pull only network assests ? otherwise as I read this, if I load media from one local drive ( or network volume ) and save it to a seperate local volume, it should copy it over, yes ?

The script will never show up in any menus. That's the purpose of putting it in a folder (support) that Fusion doesn't know about.

Try this: Make a new default of any random tool. To do that, add the tool to the Flow, change a parameter a bit, then right-click the node and choose Settings > Save Default. Then check to see if a new file with that tool's name appears in your Defaults folder. If it does, then I'm stymied—it should be working. If it doesn't, then that folder isn't actually where Fusion is looking for its tool defaults.

Code: Select all

dump(comp:MapPathSegments("Defaults:"))


That will show you a list of paths where Fusion expects to find its defaults.

that said, the code may not be copying. I randomly loaded a mov into a comp, did save as (comp) to another local volume and hit make local. I got error the copy was already local.

What are the complete paths to the comp and the asset you tried to operate on? And was there any feedback in the Console?

that said, the code may not be copying. I randomly loaded a mov into a comp, did save as (comp) to another local volume and hit make local. I got error the copy was already local.

What are the complete paths to the comp and the asset you tried to operate on? And was there any feedback in the Console?

Save to was root of since it was quick test /Volumes/Electron 480

Source Media : /Volumes/Proton 2TB/{Project Name}/DSCF9712.MOV

consol msg matched warning
...magic Design/Fusion/Scripts/support/Loader_MakeLocal.lua:248: Already Local

The default COMPDEPTH variable is set to 2. This variable determines how far up the folder structure the script will look for the shot folder. So as far as the script is concerned, your shot folder is simply '/'. And since every file in the entire file system is also under /, it will always think they're local.

You will probably still have to make adjustments to the global variables in the .lua file in order for it to function properly. There are instructions in the comments block at the top.

The default COMPDEPTH variable is set to 2. This variable determines how far up the folder structure the script will look for the shot folder. So as far as the script is concerned, your shot folder is simply '/'. And since every file in the entire file system is also under /, it will always think they're local.

You will probably still have to make adjustments to the global variables in the .lua file in order for it to function properly. There are instructions in the comments block at the top.

Ok, I'm digging thru the code a bit. How about something along the lines of :

COMPDEPTH = Count number of path separators ignoring double separators

I just don't know enough about LUA to mess with it yet, but looks like this could be done in parseFilename, then COMPDEPTH should always be able to see the level. I'm not quite sure how / what is being counted, or the other logic involved in how COMPDEPTH is being used. I'd guess that it would defualt at the volume level rather than folder level. I know sharing a big project folder vs actual volume is common with large shops, but for smaller ones that simply share entire volumes...

S