Jump to: Board index » General » Fusion

Network Rendering Fails, Needs Serious TLC

Learn about 3D compositing, animation, broadcast design and VFX workflows.
  • Author
  • Message
Offline
User avatar

Bryan Ray

  • Posts: 2484
  • Joined: Mon Nov 28, 2016 5:32 am
  • Location: Los Angeles, CA, USA

Re: Network Rendering Fails, Needs Serious TLC

PostThu Sep 10, 2020 10:21 pm

It's just an integer that needs to be set in accordance with the the pipeline's structure. Let's suppose that in your environment, the path to the comp looks like this:

/Volumes/Drive/Project/Shot/Fusion/Project_Shot_composite_v01.comp

You want to be sure that assets associated with your comp always live in the Shot directory somewhere. It's two folders up from where the comp lives to Shot, so COMPDEPTH = 2 for your pipeline. If your path is instead this:

/Volumes/Drive/Project/Shot/Applications/Fusion/Project_Shot_composite_v01.comp

Then the COMPDEPTH = 3

The ASSETFOLDER is the path relative to the Shot where you want the asset to be copied. So if you set that variable like so:

ASSETFOLDER = [[elements/comp]]

Then the assets will be copied to this path:

/Volumes/Drive/Project/Shot/elements/comp/my-asset/my-asset.####.jpg

The four lines under the PIPELINE GLOBALS are the only ones that most TDs should need to change, and as long as everything remains conformed to the pipeline, the script will work.
Bryan Ray
http://www.bryanray.name
http://www.sidefx.com
Offline

steve oakley

  • Posts: 547
  • Joined: Fri Aug 16, 2013 6:07 pm

Re: Network Rendering Fails, Needs Serious TLC

PostThu Sep 10, 2020 11:43 pm

Ok I was expecting a some what different behavior in that if the media file isn't directly in the local project volume/folder then the copy would happen, certainly across local volumes. So its not really clear here what is going on and what COMPDEPTH is doing. its counting backwards from the end ignoring the front part of the path for what is view as depth ? in which case to get it to work at the different volume level it should be set to 0 or 1 ?

example :

Drive name=Lots of Stuff
folder name =Folder of stills
media=image1.jpg

which should resolve to on OS X to an absolute path of SystemSSD/Volumes/Lots of Stuff/Folder of stills/image1.jpg

and comp is located at

Drive name=Project123
Folder=Sept 2020
proj file=comp123
Full path should be SystemSSD/Volumes/Project123/Sept 2020/comp123

I"m still trying to figure this code out and not knowing its details of the language is hard, but I take it that lines 231-250 is what collects and decides if the file is local or not ?


UPDATE : I set COMPDEPTH=1 and that triggered a file copy, except the mkdir failed to generate the folder and cp also failed. R/W on the volume, no error msg it failed.

Bryan Ray wrote:It's just an integer that needs to be set in accordance with the the pipeline's structure. Let's suppose that in your environment, the path to the comp looks like this:

/Volumes/Drive/Project/Shot/Fusion/Project_Shot_composite_v01.comp

You want to be sure that assets associated with your comp always live in the Shot directory somewhere. It's two folders up from where the comp lives to Shot, so COMPDEPTH = 2 for your pipeline. If your path is instead this:

/Volumes/Drive/Project/Shot/Applications/Fusion/Project_Shot_composite_v01.comp

Then the COMPDEPTH = 3

The ASSETFOLDER is the path relative to the Shot where you want the asset to be copied. So if you set that variable like so:

ASSETFOLDER = [[elements/comp]]

Then the assets will be copied to this path:

/Volumes/Drive/Project/Shot/elements/comp/my-asset/my-asset.####.jpg

The four lines under the PIPELINE GLOBALS are the only ones that most TDs should need to change, and as long as everything remains conformed to the pipeline, the script will work.
S
mac Studio Ultra 128gb OS 13.5+
TB3 ext 1TB SSD Cache, 1TB SSD Sys
4K 27Dell
Presonus Quantum 2626 TB3 Word clocked 96K
MiniPanel via enet
Behinger Xtouch USB via USB3 Hub->PCIe USB3X4 card
Wacom XL Pro
Offline
User avatar

Bryan Ray

  • Posts: 2484
  • Joined: Mon Nov 28, 2016 5:32 am
  • Location: Los Angeles, CA, USA

Re: Network Rendering Fails, Needs Serious TLC

PostFri Sep 11, 2020 3:02 am

TheBloke, could you please test version 2.1? I want to be sure I didn't break anything on the Mac side in the last revision. It's the archive attached to this message:
viewtopic.php?f=22&t=120610#p666112

steve oakley wrote:Ok I was expecting a some what different behavior in that if the media file isn't directly in the local project volume/folder then the copy would happen, certainly across local volumes. So its not really clear here what is going on and what COMPDEPTH is doing. its counting backwards from the end ignoring the front part of the path for what is view as depth ? in which case to get it to work at the different volume level it should be set to 0 or 1 ?


First, please confirm that you're using version 2.1 of the script, dated 2020-09-09. The version before that definitely did not work on Mac, but TheBloke helped me through the errors, and I thought this one should have worked. But, again, I can't test it myself. Now, on with the actual content…



I may have miscounted before. The table includes the comp file itself when counting the depth.

I'll walk through the code a little, although you really don't need to know the nuts and bolts to use it. You seriously only need to worry about those four variables near the top.

At line 223, we split the comp's file path into a table. Using my own test project as an example:
D:\projects\scripting\makeLocal\apps\makeLocal.comp

My asset is stored at:
C:\Users\[user]\Desktop\Image.jpg

The resulting tables looks like this:
pathtable = {
1 = D:
2 = projects
3 = scripting
4 = makeLocal
5 = apps
6 = makeLocal.comp
}

elemtable = {
1 = C:
2 = Users
3 = [user]
4 = Desktop
5 = Image.jpg
}

Lines 226 count the entries in the pathtable (this is a really inefficient way to do it, btw.) When it's done, pathdepth = k = 6. (Again, inefficient. I could just assign the value to pathdepth directly instead of using a temp variable.)

I've set COMPDEPTH = 2 because I store my composite in that 'apps' subfolder under the 'makeLocal' shot. In lines 239 - 242, I construct two paths: targetPath is the shot folder, and currentPath is part of where the asset currently lives.

So pathdepth - COMPDEPTH = 4, and I reconstruct the paths up to that index, resulting in these two:
D:\projects\scripting\makeLocal
C:\Users\[user]\Desktop

Lines 246-249 compare those two strings. If the asset were located somewhere in the makeLocal folder, then the script would abort. Since they're not equal, it proceeds.

Lines 267 - 269 construct a new path using the contents of the pathtable, the ASSETFOLDER, and whatever name the user put in the Element subfolder field of the dialog box.


UPDATE : I set COMPDEPTH=1 and that triggered a file copy, except the mkdir failed to generate the folder and cp also failed. R/W on the volume, no error msg it failed.



Lines 274-279 should create a directory, and it sounds like this is the spot where perhaps it failed.
If it fails to make the new directory, then obviously the copy will also fail.

That was the same error TheBloke reported earlier, and he assisted me in fixing it, which is what makes me think perhaps you didn't download the latest.
Last edited by Bryan Ray on Fri Sep 11, 2020 5:23 pm, edited 1 time in total.
Bryan Ray
http://www.bryanray.name
http://www.sidefx.com
Offline

steve oakley

  • Posts: 547
  • Joined: Fri Aug 16, 2013 6:07 pm

Re: Network Rendering Fails, Needs Serious TLC

PostFri Sep 11, 2020 5:13 am

ok thanks for the explanation, that was helpful to understand builtins vs your vars and what's going on.
perfectly good solution as long as its reliable.

version 2.1, 2020-09-09

I Did reinstall on 10.15.6 machine, prev results are from OS X beta6 which hasn't seemed to of changed anything.

changing COMPDEPTH from 2 to 1 fixes the already local problem between volumes on 2nd machine. I was tempted to mod the code a bit but realized my change would only work for windows. :/

S
S
mac Studio Ultra 128gb OS 13.5+
TB3 ext 1TB SSD Cache, 1TB SSD Sys
4K 27Dell
Presonus Quantum 2626 TB3 Word clocked 96K
MiniPanel via enet
Behinger Xtouch USB via USB3 Hub->PCIe USB3X4 card
Wacom XL Pro
Offline
User avatar

TheBloke

  • Posts: 1905
  • Joined: Sat Nov 02, 2019 11:49 pm
  • Location: UK
  • Real Name: Tom Jobbins

Re: Network Rendering Fails, Needs Serious TLC

PostFri Sep 11, 2020 10:25 am

Bryan Ray wrote:TheBloke, could you please test version 2.1? I want to be sure I didn't break anything on the Mac side in the last revision. It's the archive attached to this message:
viewtopic.php?f=22&t=120610#p666112
It's not working for me again:
Code: Select all
...magic Design/Fusion/Scripts/support/Loader_MakeLocal.lua:241: attempt to concatenate a nil value
The line in question is the 'targetpath = ' in :
Code: Select all
-- Reconstuct the paths of both the shot and the asset
for i=1, pathdepth-COMPDEPTH do
   currentPath = currentPath..pathtable[i]..SEPARATOR
   targetPath = targetPath..elemtable[i]..SEPARATOR
end
In the version I sent you a day or two ago, I modified this section like so:
Code: Select all
-- Reconstuct the paths of both the shot and the asset
for i=1, pathdepth-COMPDEPTH do
   if pathtable[i] then currentPath = currentPath..pathtable[i]..SEPARATOR end
   if elemtable[i] then targetPath = targetPath..elemtable[i]..SEPARATOR end
end
Making that change again results in it working for me fine. Tested with and without PathMaps, both of which worked.

Though that did cause me to notice one other minor thing: The updated path does not use PathMaps, like it would if I manually browsed to a file in a Loader. Eg it gave me a path of:
Code: Select all
/Volumes/data/projects/Fusion/Comps/elements/videos/CarRegTrackerTest.mov
If I were to manually browse to that file, Fusion would evaluate my PathMaps and use the following Loader path:
Code: Select all
Data:/projects/Fusion/Comps/elements/videos/CarRegTrackerTest.mov
I believe there's a script call which will convert a path to the best-matching PathMap? (Is that called 'reverse mapping'?) If that could be put in, that would be helpful I think.

Thanks
Resolve Studio 17.4.3 and Fusion Studio 17.4.3 on macOS 11.6.1

Hackintosh:: X299, Intel i9-10980XE, 128GB DDR4, AMD 6900XT 16GB
Monitors: 1 x 3840x2160 & 3 x 1920x1200
Disk: 2TB NVMe + 4TB RAID0 NVMe; NAS: 36TB RAID6
BMD Speed Editor
Offline
User avatar

Bryan Ray

  • Posts: 2484
  • Joined: Mon Nov 28, 2016 5:32 am
  • Location: Los Angeles, CA, USA

Re: Network Rendering Fails, Needs Serious TLC

PostFri Sep 11, 2020 5:22 pm

Aaaaahhhaa… Yeah, if the asset's original location table has fewer entries than the shot path's table, then it can't continue building it out. That makes sense. It's amazing that the script never failed at Muse before with a bug like that! Maybe it's something I introduced when generalizing it.

Anyway, thank you very much for testing it again! I'll make those changes this afternoon.

Getting it into Reactor is taking a bit longer because I realized that I need a way to prevent the installation from overwriting existing Loader defaults. Just as well since I'd have had to update immediately!
Bryan Ray
http://www.bryanray.name
http://www.sidefx.com
Offline

steve oakley

  • Posts: 547
  • Joined: Fri Aug 16, 2013 6:07 pm

Re: Network Rendering Fails, Needs Serious TLC

PostFri Sep 11, 2020 5:37 pm

Bryan Ray wrote:Aaaaahhhaa… Yeah, if the asset's original location table has fewer entries than the shot path's table, then it can't continue building it out. That makes sense. It's amazing that the script never failed at Muse before with a bug like that! Maybe it's something I introduced when generalizing it.

Anyway, thank you very much for testing it again! I'll make those changes this afternoon.


ya I was just turning that one over. good catch. still with the fix for path build in place, mkdir and cp still fail. more interesting is that the loader now has its path set to that location, so hitting Make Local again errors out that the file is already local when it doesn't actually exist. seems like a bit of error checking is needed here to see if the file is actually in the location or not rather than the quick check of MediaPath=CompPath and assume its good.

S
S
mac Studio Ultra 128gb OS 13.5+
TB3 ext 1TB SSD Cache, 1TB SSD Sys
4K 27Dell
Presonus Quantum 2626 TB3 Word clocked 96K
MiniPanel via enet
Behinger Xtouch USB via USB3 Hub->PCIe USB3X4 card
Wacom XL Pro
Offline
User avatar

Bryan Ray

  • Posts: 2484
  • Joined: Mon Nov 28, 2016 5:32 am
  • Location: Los Angeles, CA, USA

Re: Network Rendering Fails, Needs Serious TLC

PostFri Sep 11, 2020 10:19 pm

Okay, one more try!

In version 2.2, I have added a Debug mode, currently defaulting to true so that any problems can be more easily seen. They're most likely to show up still in the os.execute() routines. Everything works in Windows.

I refactored a little, putting the code into a main() function so that the utility functions could be moved to the bottom. I also simplified all of those inefficiencies I noted earlier.

There is additional error checking now, including testing whether the file exists both before and after it is processed.

The list of video formats has been moved into the pipeline globals, so it will be easy to add more video file types if they become relevant.

There were some extra separators in a couple of places. That may have fouled up the mkdir sometimes. One source I read said that a trailing slash could cause mkdir to fail. I couldn't find any corroboration, though. And there were a couple of places where a path received a double slash. I believe I've gotten rid of all of them, but the syntax of the os commands is confusion, so I'd appreciate a sanity check there.

And path maps should now be reversed upon assignment to the Clip input.

Phew!
Last edited by Bryan Ray on Mon Sep 14, 2020 8:56 pm, edited 1 time in total.
Bryan Ray
http://www.bryanray.name
http://www.sidefx.com
Offline

steve oakley

  • Posts: 547
  • Joined: Fri Aug 16, 2013 6:07 pm

Re: Network Rendering Fails, Needs Serious TLC

PostSat Sep 12, 2020 11:12 pm

Ok testing here.
With clip loaded, hitting Make Local results in Clip Not Found On Disk

I think the problem is there are some path structure dependencies here based on how things are set up at Muse vs how they may be here - Fusion comps are not in a path like Volume/Project/Shot/Comp but here typically Volume/Project/ for when only a few things are being done or Volume/Project/Fusion Comps for ones that will see more than a couple stand alone things done.

COMPDEPTH is messing things up for sure with the copy. I set it to 0 and its now actually triggering a copy attempt instead of Clip Not Found. I'd default it to 0 for average users who aren't working in a specific fixed path structure.

once it finally fired off an actual copy I had an error and debug output :

Code: Select all
Vers:2.2

tool:
Loader (0x0x7f9732bffc10) [App: 'Fusion' on 127.0.0.1, UUID: 533e9116-0171-4c06-a8ba-ed9ce44c1a9c]

Santized full path to asset: /Volumes/Proton 2TB/IOWA 2020/DSA JPG 1 15-Apple ProRes 422.mov
SEPARATOR = /

elemtable:
table: 0x02b50a10
   1 =
   2 = Volumes
   3 = Proton 2TB
   4 = IOWA 2020
   5 = DSA JPG 1 15-Apple ProRes 422.mov
Element filename: DSA JPG 1 15-Apple ProRes 422.mov
Video format detected: mov

Comp filename: /Volumes/Electron 480/Fusions Test comp/test3.comp
Path table to comp file
table: 0x02b175b0
   1 =
   2 = Volumes
   3 = Electron 480
   4 = Fusions Test comp
   5 = test3.comp
pathdepth = 5
currentPath = /Volumes/Proton 2TB/IOWA 2020/
targetPath = /Volumes/Electron 480/Fusions Test comp/test3.comp/

Reconstructed source path: /Volumes/Proton 2TB/IOWA 2020/

New path to asset: /Volumes/Electron 480/Fusions Test comp/test3.comp/elements/videos
Executing os command:
mkdir -p /Volumes/Electron 480/Fusions Test comp/test3.comp/elements/videos

Copying File
Executing os command:
cp "/Volumes/Proton 2TB/IOWA 2020/DSA JPG 1 15-Apple ProRes 422.mov" "/Volumes/Electron 480/Fusions Test comp/test3.comp/elements/videos/DSA JPG 1 15-Apple ProRes 422.mov"

New Clip value: /Volumes/Electron 480/Fusions Test comp/test3.comp/elements/videos/DSA JPG 1 15-Apple ProRes 422.mov
...magic Design/Fusion/Scripts/support/Loader_MakeLocal.lua:306: Unexpected failure. File not found at new location.
Loader1 failed to load file "/Volumes/Electron 480/Fusions Test comp/test3.comp/elements/videos/DSA JPG 1 15-Apple ProRes -2147483648.mov" (clip "/Volumes/Electron 480/Fusions Test comp/test3.comp/elements/videos/DSA JPG 1 15-Apple ProRes 422.mov" at frame -2147483648) : Invalid argument
Loader1 failed at time 0


Looking at the actual location, elements and videos failed to be created. :(

in my personal setup, that would get me a little crazy as I normally, if creating a folder for a comp, would want the media file placed right next to the comp because its going only be maybe a single file. only if its a img sequence would I want to create a folder to keep the mess under control.

I added
Code: Select all
Line 71 VERSION=2.2
and
Code: Select all
Line 93    dprint("\nVers:"..VERSION)

because I misinstalled onto networked machine instead of local machine. once I had that sorted out...
S
mac Studio Ultra 128gb OS 13.5+
TB3 ext 1TB SSD Cache, 1TB SSD Sys
4K 27Dell
Presonus Quantum 2626 TB3 Word clocked 96K
MiniPanel via enet
Behinger Xtouch USB via USB3 Hub->PCIe USB3X4 card
Wacom XL Pro
Offline
User avatar

Bryan Ray

  • Posts: 2484
  • Joined: Mon Nov 28, 2016 5:32 am
  • Location: Los Angeles, CA, USA

Re: Network Rendering Fails, Needs Serious TLC

PostSun Sep 13, 2020 4:09 pm

steve oakley wrote:COMPDEPTH is messing things up for sure with the copy. I set it to 0 and its now actually triggering a copy attempt instead of Clip Not Found. I'd default it to 0 for average users who aren't working in a specific fixed path structure.


The 'file not found' error should only happen if the file in the Clip field cannot be opened for reading. It shouldn't have anything to do with COMPDEPTH. I have a theory, though, and potentially a solution...


Code: Select all
targetPath = /Volumes/Electron 480/Fusions Test comp/test3.comp/



Looks like setting COMPDEPTH to 0 means that the comp's filename got included in the new path.

I'll forcibly trim off the filename, so a depth of 0 will function the way you describe: Assets will be moved into the same folder as the comp. I'll set the STILLSFOLDER and VIDEOFOLDER strings to "" to prevent those being created by default.

I'm going to leave the elements sub-folder assignment, though, for sanity's sake. Even in a quick-n-dirty comp, it's still a good idea to separate source files from project files.

You're free to set the ASSETFOLDER string to "" in order to prevent any folder from being created at all, but please do not do so until we're sure the mkdir command is working—that's still giving me problems.

What I noticed after doing that is that when those strings are empty, we're back to double slashes in the paths. I've added some tests to clear those.



I added
Code: Select all
Line 71 VERSION=2.2
and
Code: Select all
Line 93    dprint("\nVers:"..VERSION)

because I misinstalled onto networked machine instead of local machine. once I had that sorted out...


That's a good idea. I've done the same, and it prints the script name and date, too.

…aaaand, I've run out of time this morning to finish the updates I want to do. I'll probably not get back to it until tomorrow, but here's a new version, anyway.

(Reminders to myself: bmd.readdir() for testing file and folder existence. Add a check after the mkdir to verify the new folder and output debug info.)
Last edited by Bryan Ray on Mon Sep 14, 2020 8:56 pm, edited 1 time in total.
Bryan Ray
http://www.bryanray.name
http://www.sidefx.com
Offline
User avatar

Bryan Ray

  • Posts: 2484
  • Joined: Mon Nov 28, 2016 5:32 am
  • Location: Los Angeles, CA, USA

Re: Network Rendering Fails, Needs Serious TLC

PostMon Sep 14, 2020 8:56 pm

This version (2.4) should raise an error if the target folder does not exist after the mkdir command. It also uses a better method to check if files and folders exist than trying to open them for reading. In addition to the changes detailed above.

Hopefully it will help get to the bottom of failed folder creation, if that is indeed what's happening.
Bryan Ray
http://www.bryanray.name
http://www.sidefx.com
Offline

steve oakley

  • Posts: 547
  • Joined: Fri Aug 16, 2013 6:07 pm

Re: Network Rendering Fails, Needs Serious TLC

PostMon Sep 14, 2020 10:04 pm

Bryan Ray wrote:This version (2.4) should raise an error if the target folder does not exist after the mkdir command. It also uses a better method to check if files and folders exist than trying to open them for reading. In addition to the changes detailed above.

Hopefully it will help get to the bottom of failed folder creation, if that is indeed what's happening.


this looks pretty good for all the workflow possibilites now. Correctly detected was media file a few folders deep on same volume as save volume marked as already local :)

mkdir is failing. the error checking code is working correctly it looks like since if flagged it in Fusion alert

if I run the command directly in term, I get: mkdir: /Volumes/Electron: Permission denied

volume perm are RWX/RWX/RWX me / staff / everyone

if I do mkdir /Volumes/Electron\ 480/VideoTest that works. Note that the space in the file name is escaped with "\". once you do that it works. so it looks like for at least OS X you need to backslash all spaces. Also all should be illegal but people use them chars like #$%^&*!-=| etc.

for example
Screen Shot 2020-09-14 at 5.00.48 PM.png
Screen Shot 2020-09-14 at 5.00.48 PM.png (15.59 KiB) Viewed 2915 times

translates to /Volumes/Electron\ 480/Illegal\$\*\ chars in the terminal

so maybe one last fix. assume Linux is probably the same but currently don't have linux box running right now.

S

----
Code: Select all
Make Local v2.4, 2020-09-14

tool:
Loader (0x0x7f8fe0343a10) [App: 'Fusion' on 127.0.0.1, UUID: ed66d67e-2c58-463f-ac35-ea7173b69169]

Santized full path to asset: /Volumes/Proton 2TB/IOWA 2020/DSA JPG 1 15-Apple ProRes 422.mov
SEPARATOR = /

elemtable:
table: 0x05a3bf60
   1 =
   2 = Volumes
   3 = Proton 2TB
   4 = IOWA 2020
   5 = DSA JPG 1 15-Apple ProRes 422.mov
Element filename: DSA JPG 1 15-Apple ProRes 422.mov
Video format detected: mov

Comp filename: /Volumes/Electron 480/Test Comp4.comp
Path table to comp file
table: 0x05aa02a0
   1 =
   2 = Volumes
   3 = Electron 480
   4 = Test Comp4.comp
pathdepth = 3
currentPath = /Volumes/Proton 2TB/
targetPath = /Volumes/Electron 480/

Reconstructed source path: /Volumes/Proton 2TB/IOWA 2020/

New path to asset: /Volumes/Electron 480/VideoTest
Executing os command:
mkdir -p /Volumes/Electron 480/VideoTest
...magic Design/Fusion/Scripts/support/Loader_MakeLocal.lua:306: New folder not created.
S
mac Studio Ultra 128gb OS 13.5+
TB3 ext 1TB SSD Cache, 1TB SSD Sys
4K 27Dell
Presonus Quantum 2626 TB3 Word clocked 96K
MiniPanel via enet
Behinger Xtouch USB via USB3 Hub->PCIe USB3X4 card
Wacom XL Pro
Offline
User avatar

TheBloke

  • Posts: 1905
  • Joined: Sat Nov 02, 2019 11:49 pm
  • Location: UK
  • Real Name: Tom Jobbins

Re: Network Rendering Fails, Needs Serious TLC

PostMon Sep 14, 2020 11:37 pm

Escaping spaces is only necessary if the strings aren't enclosed in double quotes.

The following are identical:
Code: Select all
mkdir "/Volumes/Electron 480/some dir"
mkdir /Volumes/Electron\ 480/some\ dir
That said, unusual chars could definitely cause a problem. If Lua's os.execute() goes via a shell - which I assume it must - a $ character for example will be interpreted as a shell variable, including in double quotes.

There are two solutions to that:

1. Escape the special chars, as in:
Code: Select all
mkdir "/Volumes/Electron 480/Illegal\$* chars"   # (note the spaces and * are not escaped as they are not special to the shell within double quotes)

2. Use single quotes, in which nothing is interpreted:
Code: Select all
mkdir '/Volumes/Electron 480/Illegal$* chars'


By the way, if you're ever attempting to make multiple nested directories at once, -p should be added, as in the following example:
Code: Select all
mkdir -p  '/Volumes/Electron 480/dir1/dir2' # where dir1 does not already exist
Resolve Studio 17.4.3 and Fusion Studio 17.4.3 on macOS 11.6.1

Hackintosh:: X299, Intel i9-10980XE, 128GB DDR4, AMD 6900XT 16GB
Monitors: 1 x 3840x2160 & 3 x 1920x1200
Disk: 2TB NVMe + 4TB RAID0 NVMe; NAS: 36TB RAID6
BMD Speed Editor
Offline
User avatar

Bryan Ray

  • Posts: 2484
  • Joined: Mon Nov 28, 2016 5:32 am
  • Location: Los Angeles, CA, USA

Re: Network Rendering Fails, Needs Serious TLC

PostMon Sep 14, 2020 11:41 pm

How odd... There are supposed to be double quotes in that command. I'm going to have to take a hard look at that line. It gets confusing.

I don't know how willing I am to enable people who use illegal characters in their file paths. But I suppose if the single quote works, that's a simple enough solution.

edit: Oh, nevermind. I'd only done the double quotes stuff for the cp command, not for mkdir.
Bryan Ray
http://www.bryanray.name
http://www.sidefx.com
Offline
User avatar

TheBloke

  • Posts: 1905
  • Joined: Sat Nov 02, 2019 11:49 pm
  • Location: UK
  • Real Name: Tom Jobbins

Re: Network Rendering Fails, Needs Serious TLC

PostMon Sep 14, 2020 11:43 pm

Bryan Ray wrote:I don't know how willing I am to enable people who use illegal characters in their file paths. But I suppose if the single quote works, that's a simple enough solution.
I would delete the files. It's the only way they'll learn.

Double quotes should be fine for paths with spaces and nearly all other characters besides $
Resolve Studio 17.4.3 and Fusion Studio 17.4.3 on macOS 11.6.1

Hackintosh:: X299, Intel i9-10980XE, 128GB DDR4, AMD 6900XT 16GB
Monitors: 1 x 3840x2160 & 3 x 1920x1200
Disk: 2TB NVMe + 4TB RAID0 NVMe; NAS: 36TB RAID6
BMD Speed Editor
Offline
User avatar

Bryan Ray

  • Posts: 2484
  • Joined: Mon Nov 28, 2016 5:32 am
  • Location: Los Angeles, CA, USA

Re: Network Rendering Fails, Needs Serious TLC

PostMon Sep 14, 2020 11:47 pm

That seemed like an easy one. Let's hope...
Attachments
MakeLocal.zip
(5.56 KiB) Downloaded 95 times
Bryan Ray
http://www.bryanray.name
http://www.sidefx.com
Offline
User avatar

Bryan Ray

  • Posts: 2484
  • Joined: Mon Nov 28, 2016 5:32 am
  • Location: Los Angeles, CA, USA

Re: Network Rendering Fails, Needs Serious TLC

PostMon Sep 14, 2020 11:55 pm

TheBloke wrote:I would delete the files. It's the only way they'll learn.


:lol:

I had a vendor send a drive to me where they had put a bullet character in every. single. filename. I was, of course, running a hybrid Mac/Windows environment, so I had to find a file renamer utility that would understand the bullet to be able to remove them. And that was long before I'd started learning any kind of scripting, so I was totally at the mercy of whatever utilities someone else had made.
Bryan Ray
http://www.bryanray.name
http://www.sidefx.com
Offline

steve oakley

  • Posts: 547
  • Joined: Fri Aug 16, 2013 6:07 pm

Re: Network Rendering Fails, Needs Serious TLC

PostTue Sep 15, 2020 12:56 am

Bryan Ray wrote:
TheBloke wrote:I would delete the files. It's the only way they'll learn.


:lol:

I had a vendor send a drive to me where they had put a bullet character in every. single. filename. I was, of course, running a hybrid Mac/Windows environment, so I had to find a file renamer utility that would understand the bullet to be able to remove them. And that was long before I'd started learning any kind of scripting, so I was totally at the mercy of whatever utilities someone else had made.


Very close :) dir made, file copied,
if you don't enter path name, just enter, goes to same place as comp. ok
if you do, file is copied, ok
even if Fusion can't read file format (h.265 mov ) it still copied file, ok

bug: this file name seems to interpeted as a image sequence for some reason and the sequence check box is turned on. results in dir being made, DSCF9681.MOV actually seems to result in making folder of same name as file being made even with sequence disabled.

/Volumes/Electron 480/Video Test4/DSCF9681.MOV/DSCF9681.MOV

is the issue the 4 digits in file name ? I guess movie file formats need to be handled differently than still img formats.

annoyance : when adding multiple savers, any way to save current media path if in use so its auto entered ? or is that gonna be left to filling out the internal var for that ?

no testing on linux :(


oh, 20 years ago, my first regular freelance job doing MG, this place wanted all "final" renders named with a • at the start of the file name so they sorted to the top of a folder. easy to type on mac, but PC i don't think has direct keystroke for that.

S
S
mac Studio Ultra 128gb OS 13.5+
TB3 ext 1TB SSD Cache, 1TB SSD Sys
4K 27Dell
Presonus Quantum 2626 TB3 Word clocked 96K
MiniPanel via enet
Behinger Xtouch USB via USB3 Hub->PCIe USB3X4 card
Wacom XL Pro
Offline
User avatar

Bryan Ray

  • Posts: 2484
  • Joined: Mon Nov 28, 2016 5:32 am
  • Location: Los Angeles, CA, USA

Re: Network Rendering Fails, Needs Serious TLC

PostTue Sep 15, 2020 4:36 pm

A PC can't even access the file if it has the bullet character in the name. Windows Server 2008 couldn't, at any rate.

My guess is that the error occurred due to the all-caps extension. I forgot to force lower case during the comparison. Easily fixed.

Your annoyance is outside the scope of this script, but there are solutions. Check Steakunderwater for a recent thread about making Savers from Loaders.
Bryan Ray
http://www.bryanray.name
http://www.sidefx.com
Offline
User avatar

Bryan Ray

  • Posts: 2484
  • Joined: Mon Nov 28, 2016 5:32 am
  • Location: Los Angeles, CA, USA

Re: Network Rendering Fails, Needs Serious TLC

PostTue Sep 15, 2020 9:49 pm

Version 2.42_final_final2_REAL-Final_this-one-for-sure is up.

Fixed the failure to match a video file extension that is not lower case.
Added operation documentation in the header comments.
Clarified some of the TD documentation.

Give it a good, thorough testing please, to be sure I didn't break anything else. Try changing COMPDEPTH. Try it with different combinations of empty strings and folder names in the ASSETFOLDER, STILLSFOLDER, and VIDEOFOLDER variables. Try a single frame from an image sequence to see if it will copy just that frame, then try the same, but manually tick the sequence checkbox to see what happens.
Attachments
MakeLocal.zip
(5.88 KiB) Downloaded 113 times
Bryan Ray
http://www.bryanray.name
http://www.sidefx.com
Offline

steve oakley

  • Posts: 547
  • Joined: Fri Aug 16, 2013 6:07 pm

Re: Network Rendering Fails, Needs Serious TLC

PostTue Sep 15, 2020 10:41 pm

Bryan Ray wrote:Version 2.42_final_final2_REAL-Final_this-one-for-sure is up.

Give it a good, thorough testing please, to be sure I didn't break anything else. Try changing COMPDEPTH. Try it with different combinations of empty strings and folder names in the ASSETFOLDER, STILLSFOLDER, and VIDEOFOLDER variables. Try a single frame from an image sequence to see if it will copy just that frame, then try the same, but manually tick the sequence checkbox to see what happens.


seems really really final to me ;) I know, you want this done and set free :)

I have a couple days of out of town production work so I can't hammer on this too much, I have to pack gear tonite but... seems ok. even tried deleting the local file and script made appropriate complaint of missing file. put it back online, then saw it.

copied jpg sequence and I got double path
/Volumes/Electron 480/Video Test4/_DSA9914.JPG/_DSA9914.JPG {/ actual images _DSAxxxx.jpg }

file count was good

also had this when I flipped back into fusion
Code: Select all
Copying Sequence
Executing os command:
cp -R "/Volumes/Proton 2TB/IOWA 2020/TL nite des moines JPG/" "/Volumes/Electron 480/Video Test4/_DSA9914.JPG/"

New Clip value: /Volumes/Electron 480/Video Test4/_DSA9914.JPG/_DSA9914.JPG
...magic Design/Fusion/Scripts/support/Loader_MakeLocal.lua:366: Unexpected failure. File not found at new location.


If I hit Make local 2nd time, instead of saying already local I get

Code: Select all
Make Local v2.42, 2020-09-15

tool:
Loader (0x0x7fba52156210) [App: 'Fusion' on 127.0.0.1, UUID: {xxxxx}]
...magic Design/Fusion/Scripts/support/Loader_MakeLocal.lua:136: File not found


doc update : Pipeline Globals - All quotes are SINGLE quotes -

COMPDEPTH set to 1 didn't seem to break anything
if I set COMPDEPTH to 2 I get msg item is already local... but its not unless you count local machine. that said I'm not testing folder a few levels down, just Volume/Proj Folder. at 0 works fine. non-issue for me.

if one sets the folder values, they populate into the save location dialog which is fine. I have slightly different naming and can set these as appropriate for here. good.

I want to hammer on this a little more but I'm out of time today. :(

I want to again say thank you for the work. I know we all figured this would probably just be a few hours and done... but with all the vitriol on the ' net these days it was a pleasure and breath of fresh air that a couple strangers could work on making the world a little better :)

S
S
mac Studio Ultra 128gb OS 13.5+
TB3 ext 1TB SSD Cache, 1TB SSD Sys
4K 27Dell
Presonus Quantum 2626 TB3 Word clocked 96K
MiniPanel via enet
Behinger Xtouch USB via USB3 Hub->PCIe USB3X4 card
Wacom XL Pro
Offline
User avatar

Bryan Ray

  • Posts: 2484
  • Joined: Mon Nov 28, 2016 5:32 am
  • Location: Los Angeles, CA, USA

Re: Network Rendering Fails, Needs Serious TLC

PostTue Sep 15, 2020 10:58 pm

Hopefully TheBloke can help to hammer out any last details, if he's available. It would help me, though, to see the entire debug output of that failed jpeg sequence relink to try to determine why it doubled up on the folders.

And yes, if COMPDEPTH is equal to the distance to the system root, then it's going to consider everything in the file system, even network drives, to be local. If your comp's at /Volumes/Proj/compFile, then a COMPDEPTH of 2 means that it's considering / to be the shot folder.
0 = /Volumes/Proj/
1 = /Volumes/
2 = /

Have a good trip!
Bryan Ray
http://www.bryanray.name
http://www.sidefx.com
Offline

steve oakley

  • Posts: 547
  • Joined: Fri Aug 16, 2013 6:07 pm

Re: Network Rendering Fails, Needs Serious TLC

PostTue Sep 15, 2020 11:08 pm

Bryan Ray wrote:Hopefully TheBloke can help to hammer out any last details, if he's available. It would help me, though, to see the entire debug output of that failed jpeg sequence relink to try to determine why it doubled up on the folders.

And yes, if COMPDEPTH is equal to the distance to the system root, then it's going to consider everything in the file system, even network drives, to be local. If your comp's at /Volumes/Proj/compFile, then a COMPDEPTH of 2 means that it's considering / to be the shot folder.
0 = /Volumes/Proj/
1 = /Volumes/
2 = /

Have a good trip!


ok quick one here.

loaded jpb sequence, did not hit make local, console =

ake Local v2.42, 2020-09-15

tool:
Loader (0x0x7fe549df7e10) [App: 'Fusion' on 127.0.0.1, UUID: 490bdecb-a080-4c2e-a387-bf2d916f6877]

Santized full path to asset: /Volumes/Proton 2TB/IOWA 2020 Sanders Biden/TL nite des moines JPG/_DSA9914.JPG
SEPARATOR = /

elemtable:
table: 0x0ddaaf98
1 =
2 = Volumes
3 = Proton 2TB
4 = IOWA 2020
5 = TL nite des moines JPG
6 = _DSA9914.JPG
Element filename: _DSA9914.JPG
...magic Design/Fusion/Scripts/support/Loader_MakeLocal.lua:213: No comp file

hit make local and it asked me to make folder name _DSA9914.JPG - maybe I hit enter rushing thru this. manually removed the .JPG part.

copying...

copy appears to end with error dialog File Not Found At Location /Volumes/Electron 480/Video Test4/_DSA9914/_DSA9914.JPG/

newoutput is :

Make Local v2.42, 2020-09-15

tool:
Loader (0x0x7fe549df7e10) [App: 'Fusion' on 127.0.0.1, UUID: 490bdecb-a080-4c2e-a387-bf2d916f6877]

Santized full path to asset: /Volumes/Proton 2TB/IOWA 2020/TL nite des moines JPG/_DSA9914.JPG
SEPARATOR = /

elemtable:
table: 0x0abfbf98
1 =
2 = Volumes
3 = Proton 2TB
4 = IOWA 2020
5 = TL nite des moines JPG
6 = _DSA9914.JPG
Element filename: _DSA9914.JPG

Comp filename: /Volumes/Electron 480/Video Test4/Test Comp67comp.comp
Path table to comp file
table: 0x0ac3ce08
1 =
2 = Volumes
3 = Electron 480
4 = Video Test4
5 = Test Comp67comp.comp
pathdepth = 4
currentPath = /Volumes/Proton 2TB/IOWA 2020/
targetPath = /Volumes/Electron 480/Video Test4/

Reconstructed source path: /Volumes/Proton 2TB/IOWA 2020/TL nite des moines JPG/

New path to asset: /Volumes/Electron 480/Video Test4/_DSA9914
Executing os command:
mkdir -p '/Volumes/Electron 480/Video Test4/_DSA9914'

Target folder details:
table: 0x0ac2ca90
1 = table: 0x0ac2cad8
IsArchive = false
CreateTime = 1600211044
Name = _DSA9914
IsDir = true
IsReadOnly = false
Size = 64
AccessTime = 1600211044
WriteTime = 1600211044
IsHidden = false
IsSystem = false
Parent = /Volumes/Electron 480/Video Test4/
Pattern = _DSA9914

Copying Sequence
Executing os command:
cp -R "/Volumes/Proton 2TB/IOWA 2020/TL nite des moines JPG/" "/Volumes/Electron 480/Video Test4/_DSA9914/"

New Clip value: /Volumes/Electron 480/Video Test4/_DSA9914/_DSA9914.JPG
...magic Design/Fusion/Scripts/support/Loader_MakeLocal.lua:366: Unexpected failure. File not found at new location.

--- thats it gotta run

S
S
mac Studio Ultra 128gb OS 13.5+
TB3 ext 1TB SSD Cache, 1TB SSD Sys
4K 27Dell
Presonus Quantum 2626 TB3 Word clocked 96K
MiniPanel via enet
Behinger Xtouch USB via USB3 Hub->PCIe USB3X4 card
Wacom XL Pro
Offline
User avatar

Bryan Ray

  • Posts: 2484
  • Joined: Mon Nov 28, 2016 5:32 am
  • Location: Los Angeles, CA, USA

Re: Network Rendering Fails, Needs Serious TLC

PostThu Sep 17, 2020 9:59 pm

Ahaaa! I see the bug!

New version 2.44 changelog:

No longer appends filename to the copy target path when copying a sequence. Now it should copy folder to folder correctly.

Trims extension and frame numbers from sequence name when pre-filling the element sub-folder field. It also detects the presence '.' or '_' character between the filename stem and the frame number and removes that, too. A new pipeline global has been added where a TD can specify custom frame number delimiters if necessary.

When in debug mode, if Windows is detected, the console will echo the Mac and Linux os.execute() commands, even though they're not actually being run.

OS commands are now stored in a variable for ease of maintenance and readability.
Attachments
MakeLocal.zip
(6.22 KiB) Downloaded 102 times
Bryan Ray
http://www.bryanray.name
http://www.sidefx.com
Offline

steve oakley

  • Posts: 547
  • Joined: Fri Aug 16, 2013 6:07 pm

Re: Network Rendering Fails, Needs Serious TLC

PostMon Sep 21, 2020 6:23 am

Hi - made it back.

img seq copy : the new folder to create has been reduced from _DSA1234 to _DSA and thats it. my pref would be folder name = name of first image in folder w/o extension. so I think this went too far. if you loaded a couple sequences from a camera with same prefix the potential is to over write existing material, or dump it into a single dir which some apps can deal with, some can't ( apple compressor, probably FCPX, QT player img seq )

copy worked ok

after you have local elements folder / structure, on next load/copy node it asks you where to copy but doesn't indicate if there is a local elements/ folder (structure) already there. so I'm thinking if your local folder(s) are already there, then indicate that its there ? autofill the path or the preceding text in dialog as "use existing path elements/ or [ entry text field ] "since you have to test if its there and mkdir if its not. maybe just a matter of moving the requester code after the check for existence but before the create / copy. its minor annoyance but if its a quick fix...

thanks for the improved docs

other than that, seems to be working great !

S
S
mac Studio Ultra 128gb OS 13.5+
TB3 ext 1TB SSD Cache, 1TB SSD Sys
4K 27Dell
Presonus Quantum 2626 TB3 Word clocked 96K
MiniPanel via enet
Behinger Xtouch USB via USB3 Hub->PCIe USB3X4 card
Wacom XL Pro
Offline
User avatar

Bryan Ray

  • Posts: 2484
  • Joined: Mon Nov 28, 2016 5:32 am
  • Location: Los Angeles, CA, USA

Re: Network Rendering Fails, Needs Serious TLC

PostThu Sep 24, 2020 1:09 am

Loaders_MakeLocal is now available in Reactor! There's some extra special sauce in the management there, too, with the ability to insert the UserControl into an existing default, so it doesn't override any work that a TD's already done.
Bryan Ray
http://www.bryanray.name
http://www.sidefx.com
Previous

Return to Fusion

Who is online

Users browsing this forum: No registered users and 24 guests