Post Production Storage

Do you have questions about Desktop Video, Converters, Routers and Monitoring?
  • Author
  • Message
Offline
User avatar

Edrick Smith

  • Posts: 85
  • Joined: Sat Sep 08, 2012 1:42 am
  • Location: Los Angeles, California

Post Production Storage

PostThu Jan 18, 2018 8:39 pm

I've got a HP ProLiant system that's got a group of 4TB drives in a raid array for my main media storage and exports. That comprises about 20TB of storage. Then I've got a group of drives in a RAID array for my cache / scratch. Plus two drives that are run as the system drives for a Windows Server 2016 environment.

I'm wondering what you guys might recommend on the best configuration to use this as a network storage array. Currently / for a while I've had it configured where the 20TB array is setup as a network share via SMB that I can access from multiple devices and map the media to the network share. Then I have the cache disk setup as a iSCSI and generate optimized media to that.

However this hasn't been performing quite as I'd like and I'm wondering how you guys have your network based projects setup? Currently it's just me accessing it so I'm not worried about other users writing to it at the same time. Although it would be nice to set this up as best as possible for future growth.

I'm editing on a MacBook Pro 2017
C47Studios.com | Los Angeles, California
Offline

Andrew Kolakowski

  • Posts: 9206
  • Joined: Tue Sep 11, 2012 10:20 am
  • Location: Poland

Re: Post Production Storage

PostThu Jan 18, 2018 11:18 pm

Get yourself some TB3 RAID box, but I think your CPU/GPU will be the main bottleneck.
Offline
User avatar

Edrick Smith

  • Posts: 85
  • Joined: Sat Sep 08, 2012 1:42 am
  • Location: Los Angeles, California

Re: Post Production Storage

PostFri Jan 19, 2018 12:30 am

That would be fine if I wanted to go invest thousands in a thunderbolt array only for my system. But I've already got server hardware with drives and as mentioned in the future id like to be able to share projects with others. So I'm looking for a network solution.
C47Studios.com | Los Angeles, California
Offline
User avatar

Jack Fairley

  • Posts: 1863
  • Joined: Mon Oct 03, 2016 7:58 pm
  • Location: Los Angeles

Re: Post Production Storage

PostFri Jan 19, 2018 12:32 am

What specific performance issues are you having? What flavor of RAID? How are your devices networked? 1Gb? 10Gb?
Ryzen 5800X3D
32GB DDR4-3600
RTX 3090
DeckLink 4K Extreme 12G
Resolve Studio 17.4.1
Windows 11 Pro 21H2
Offline
User avatar

Edrick Smith

  • Posts: 85
  • Joined: Sat Sep 08, 2012 1:42 am
  • Location: Los Angeles, California

Re: Post Production Storage

PostFri Jan 19, 2018 5:11 am

Dropped frames / stuttering playback

Currently the workflow is copy footage to storage array on network

In davinci navigate to media drive and select clips to be in project

then optimize media

However I'm getting poor playback on even 1080 footage.

It's a gigabit network and I'll have to check the raid configurations to make sure
C47Studios.com | Los Angeles, California
Offline

Andrew Kolakowski

  • Posts: 9206
  • Joined: Tue Sep 11, 2012 10:20 am
  • Location: Poland

Re: Post Production Storage

PostFri Jan 19, 2018 10:26 am

Are you sure it's storage issue?
What if you cache media to Mac internal drive?
1gbit will be bit to slow for intermediate 4K files.
Offline
User avatar

Edrick Smith

  • Posts: 85
  • Joined: Sat Sep 08, 2012 1:42 am
  • Location: Los Angeles, California

Re: Post Production Storage

PostSat Jan 20, 2018 6:32 pm

Andrew Kolakowski wrote:Are you sure it's storage issue?
What if you cache media to Mac internal drive?
1gbit will be bit to slow for intermediate 4K files.



The MacBook Pro only has a 512GB SSD. Optimally I'd like to completely have a network based work flow so that in the future it'll be easier to share media and projects.
C47Studios.com | Los Angeles, California
Offline
User avatar

Jack Fairley

  • Posts: 1863
  • Joined: Mon Oct 03, 2016 7:58 pm
  • Location: Los Angeles

Re: Post Production Storage

PostSat Jan 20, 2018 6:43 pm

Edrick Smith wrote:Dropped frames / stuttering playback

Currently the workflow is copy footage to storage array on network

In davinci navigate to media drive and select clips to be in project

then optimize media

However I'm getting poor playback on even 1080 footage.

It's a gigabit network and I'll have to check the raid configurations to make sure

What format are you using for optimized media? What does your resource usage look like during playback on the Mac?
Ryzen 5800X3D
32GB DDR4-3600
RTX 3090
DeckLink 4K Extreme 12G
Resolve Studio 17.4.1
Windows 11 Pro 21H2
Offline

Andrew Kolakowski

  • Posts: 9206
  • Joined: Tue Sep 11, 2012 10:20 am
  • Location: Poland

Re: Post Production Storage

PostSat Jan 20, 2018 11:02 pm

Edrick Smith wrote:
Andrew Kolakowski wrote:Are you sure it's storage issue?
What if you cache media to Mac internal drive?
1gbit will be bit to slow for intermediate 4K files.



The MacBook Pro only has a 512GB SSD. Optimally I'd like to completely have a network based work flow so that in the future it'll be easier to share media and projects.


Your Mac has limited power and this maybe the main problem, not actual storage speed.
First make sure your bottleneck is not caused by CPU/GPU by trying some small test from internal SSD (this has plenty speed).
Offline
User avatar

Edrick Smith

  • Posts: 85
  • Joined: Sat Sep 08, 2012 1:42 am
  • Location: Los Angeles, California

Re: Post Production Storage

PostSun Jan 21, 2018 5:33 am

Andrew Kolakowski wrote:
Edrick Smith wrote:
Andrew Kolakowski wrote:Are you sure it's storage issue?
What if you cache media to Mac internal drive?
1gbit will be bit to slow for intermediate 4K files.



The MacBook Pro only has a 512GB SSD. Optimally I'd like to completely have a network based work flow so that in the future it'll be easier to share media and projects.


Your Mac has limited power and this maybe the main problem, not actual storage speed.
First make sure your bottleneck is not caused by CPU/GPU by trying some small test from internal SSD (this has plenty speed).


It's the top of the line practically 2017 15" Macbook Pro I would certainly hope it can handle 1080 video.


Sent from my iPhone using Tapatalk
C47Studios.com | Los Angeles, California
Offline

Andrew Kolakowski

  • Posts: 9206
  • Joined: Tue Sep 11, 2012 10:20 am
  • Location: Poland

Re: Post Production Storage

PostSun Jan 21, 2018 11:27 am

Most likely it should, but it's not so obvious, depending what you do and what type of source use.
Don't hope- if you have project which doesn't work well with your storage, try the same from internal drive. Quite simple test, no need to debate :)

If you want everything to be shared then you don't have that many choices. You can (depending on your RAID box) go to 10gbit using TB3 to 10gbit adapter.
Last edited by Andrew Kolakowski on Sun Jan 21, 2018 8:51 pm, edited 1 time in total.
Offline

Dermot Shane

  • Posts: 2716
  • Joined: Tue Nov 11, 2014 6:48 pm
  • Location: Vancouver, Canada

Re: Post Production Storage

PostSun Jan 21, 2018 6:52 pm

i have a server (64Tb) that holds cam orig / vfx assorted media
and a DAS (32Tb or 16Tb depending on machine) for caches
this works well for my use case
Offline
User avatar

Edrick Smith

  • Posts: 85
  • Joined: Sat Sep 08, 2012 1:42 am
  • Location: Los Angeles, California

Re: Post Production Storage

PostSat Jan 27, 2018 7:33 pm

Screen Shot 2018-01-27 at 2.27.22 PM.png
Network
Screen Shot 2018-01-27 at 2.27.22 PM.png (76.29 KiB) Viewed 3308 times

Screen Shot 2018-01-27 at 2.27.22 PM.png
Network
Screen Shot 2018-01-27 at 2.27.22 PM.png (76.29 KiB) Viewed 3308 times


This is the activity monitor while streaming the content in DaVinci from the media share. This is 1080 content.
Attachments
Screen Shot 2018-01-27 at 2.27.33 PM.png
Disk
Screen Shot 2018-01-27 at 2.27.33 PM.png (52.03 KiB) Viewed 3308 times
C47Studios.com | Los Angeles, California
Offline
User avatar

Edrick Smith

  • Posts: 85
  • Joined: Sat Sep 08, 2012 1:42 am
  • Location: Los Angeles, California

Re: Post Production Storage

PostSat Jan 27, 2018 8:24 pm

So I ran the Black Magic Disk Speed Test

I seem to be capping out at 13MB/s on Read and 38 MB/s on write on the iSCSI connection and not much better on the SMB media share, that coming in at around 18MB/s on Read and 38MB/s on write.

I've attached the report from the server
Attachments
ADUReport.zip
(140.52 KiB) Downloaded 102 times
C47Studios.com | Los Angeles, California
Offline

Andrew Kolakowski

  • Posts: 9206
  • Joined: Tue Sep 11, 2012 10:20 am
  • Location: Poland

Re: Post Production Storage

PostSat Jan 27, 2018 11:01 pm

That's poor. Good gigabit should give you at least 70MB/sec up to 110.

Are you on one of the latest OSX?
Check this:

https://dpron.com/os-x-10-11-5-slow-smb/
Offline
User avatar

Edrick Smith

  • Posts: 85
  • Joined: Sat Sep 08, 2012 1:42 am
  • Location: Los Angeles, California

Re: Post Production Storage

PostSun Jan 28, 2018 4:49 pm

I tested the speed with black magics disk software locally on the Windows 2016 server and I'm getting 56.6MB/s write and 340.5MB/s read on my media storage array and 518MB/s Write and 816MB/s read on my cache array.

I am running the latest version of OS X. I'm going to test the speed test on a Windows Client.

From a windows client I'm also getting relatively poor speeds. of 32 MB/s write and 92.7MB/s read. Although the write speeds being that they maxed out at 56.6MB/s for write locally would make a bit of sense with the overhead of the network protocol. So it does seem to be Mac OS that has horrible speeds.

However it seems to be true with both SMB and iSCSI attached storage. Locally on the storage array I'm getting 518/816MB/s Write/Read. However over the network via a gigabit link I'm only getting 37.8 Write and 10.7MB read.

For S&G's I connected an ethernet cable direct between the MacBook Pro and the server via the 4 port NIC it has installed. Assigned the laptop and the server their own IPs to establish a direct link and I still get horrendous performance. So it seems to be something with the Windows Server and the network sharing via SMB and iSCSI?
Attachments
BlackMagic.jpg
BlackMagic.jpg (78.73 KiB) Viewed 3272 times
C47Studios.com | Los Angeles, California
Offline

Andrew Kolakowski

  • Posts: 9206
  • Joined: Tue Sep 11, 2012 10:20 am
  • Location: Poland

Re: Post Production Storage

PostSun Jan 28, 2018 6:05 pm

Yes, you have some issue with SMB compatibility.
Have you tried this tweak from the link?
Offline
User avatar

Edrick Smith

  • Posts: 85
  • Joined: Sat Sep 08, 2012 1:42 am
  • Location: Los Angeles, California

Re: Post Production Storage

PostSun Jan 28, 2018 6:21 pm

Andrew Kolakowski wrote:Yes, you have some issue with SMB compatibility.
Have you tried this tweak from the link?


The issue occurs both with SMB and iSCSI as far as I know they use two different protocols?
C47Studios.com | Los Angeles, California
Offline
User avatar

Jack Fairley

  • Posts: 1863
  • Joined: Mon Oct 03, 2016 7:58 pm
  • Location: Los Angeles

Re: Post Production Storage

PostSun Jan 28, 2018 9:06 pm

Edrick Smith wrote:The issue occurs both with SMB and iSCSI as far as I know they use two different protocols?

They are indeed totally different. My guess is that there's a network problem somewhere, whether it's on the server NIC, the client NIC, or something in between.
Ryzen 5800X3D
32GB DDR4-3600
RTX 3090
DeckLink 4K Extreme 12G
Resolve Studio 17.4.1
Windows 11 Pro 21H2
Offline
User avatar

Edrick Smith

  • Posts: 85
  • Joined: Sat Sep 08, 2012 1:42 am
  • Location: Los Angeles, California

Re: Post Production Storage

PostSun Jan 28, 2018 9:08 pm

Jack Fairley wrote:
Edrick Smith wrote:The issue occurs both with SMB and iSCSI as far as I know they use two different protocols?

They are indeed totally different. My guess is that there's a network problem somewhere, whether it's on the server NIC, the client NIC, or something in between.


I thought originally perhaps cabling or network switch but I isolated that and just directly attached the systems together with a different network cable. Client side I've tried two systems one mac and one PC. Server side the system has two on board nics and a 4 port card and same results


Sent from my iPhone using Tapatalk
C47Studios.com | Los Angeles, California
Offline

Andrew Kolakowski

  • Posts: 9206
  • Joined: Tue Sep 11, 2012 10:20 am
  • Location: Poland

Re: Post Production Storage

PostTue Jan 30, 2018 6:41 pm

Edrick Smith wrote:
Andrew Kolakowski wrote:Yes, you have some issue with SMB compatibility.
Have you tried this tweak from the link?


The issue occurs both with SMB and iSCSI as far as I know they use two different protocols?


Yes, but both still can be affected in similar way. We are dealing with Apple which does things not mentioning a word about it. For Apple there are no pro and home users- they do what they want and don't care much about any lost compatibility etc (specially when we talk about fairy old protocols).

Judging by this link you may have exactly the same problem as it describes.

How old is that HP box? It may not be worth to waste a time on it.
Some 50$ home routers give better speed on network shares.

Return to Post Production

Who is online

Users browsing this forum: No registered users and 25 guests