BMD Camera Shootout

The place for questions about shooting with Blackmagic Cameras.
  • Author
  • Message
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

BMD Camera Shootout

PostSun Feb 21, 2021 1:46 pm

Next Sunday, February the 28th, I am conducting a shootout between several cameras, including the P4K, P6K and 12K. (If I can get my hands on one, we will also test the 4.6K G2.)

All cameras will be outfitted with the same EF mount lens, with the MFT cameras using a mechanical adapter (no Speedbooster.) Apertures and camera distance will be adjusted so that out of focus rendition matches between different sensor sizes.

Each camera's rendition will be posted online, blind, with a few different polls so that interested participants can try to identify and/or indicate preference. After collecting enough responses to be statistically meaningful I will share results, and probably share the source files.

My goal is to record a test scene that is representative of conditions in which I use these cameras...i.e. controlled conditions for narrative work...but also challenging enough to give the cameras with superior specs a chance to excel. (I'm not going to put a model in a cave and point out towards the snow, nor am I going to record a black cat in an alley at night.)

I'm posting here looking for suggestions regarding the test scene. In particular, can anyone "recommend" a fabric that is gonna reveal moiré if anything is gonna reveal moiré? (I've been remarkably lucky my whole career, rarely running into the issue.)

Here's what I'm planning so far:
- Tungsten lighting (direct lighting on "still life", upstage lighting on models)
- Two models: Fair skin/blonde and dark skin/dark hair (4 stops contrast on faces?)
- Charts: HR-1 Superchroma, X-Rite Color Checker, Datacolor SpyderCube & SpyderLensCal
- Fabric: Black velvet, White cotton, something rough & finely textured (burlap?), something that will easily produce an interference pattern (b&w checkered flannel? Pinstripes? Looking for suggestions.)
- Pure R, G & B LEDs as "practicals"
- Fruit: Apple, orange, banana, lime
- Bottles: various wine, olive oil, wine glass
- Misc.: Something steel or chrome
- Items with colors that BMD color science has traditionally been considered weak: Ryobi green. Looking for other ideas?
- The entire scene will be shot as a slow pan utilizing a motorized slider.
- Some elements will be on a motorized turntable.

Please chime in if you have suggestions or want more details about the methodology we're going to use. I'm also testing various lenses on this day, and I have limited volunteer help and time with the models, so there is a limit to what I can accomplish, but I will try to accommodate as much as I can.
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline

John Paines

  • Posts: 4222
  • Joined: Tue Jul 28, 2015 4:04 pm

Re: BMD Camera Shootout

PostSun Feb 21, 2021 3:07 pm

Here's two cents: it's hard to marshal as much enthusiasm as your proposed effort warrants. Because it's impossible to establish universally agreed upon standards, these kind of tests usually revert to the same objective measurement: comparing over and under exposure performance.

And all this tells you is which camera falls apart the soonest, not which camera you prefer or would rather use. But that may be all that anyone can reliably measure. Check out the tests that Geoff Boyle does at cinematography.net, including the recent one of the 12K (https://cinematography.net/BMD-12K.html). There's no editorializing or aesthetic judgments. But at least the tests are objective and reproducible. The investigator also avoids judgment based on either arbitrary grading decisions, or differences which could easily be eliminated with grading. He uses color charts, but doesn't have a whole lot to say about either color accuracy or aesthetic preference.

The answer you get from doing it this way doesn't answer the questions everyone's asking, least of all because of the limited subject matter and conditions. But you also have to wonder why, in the age of raw and log footage, we're examining normalized but otherwise uncorrected footage, when it's *meant* to be alterable and "accuracy" isn't the goal of an entertainment medium.
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BMD Camera Shootout

PostSun Feb 21, 2021 4:16 pm

John Paines wrote:Here's two cents: it's hard to marshal as much enthusiasm as your proposed effort warrants. Because it's impossible to establish universally agreed upon standards, these kind of tests usually revert to the same objective measurement: comparing over and under exposure performance.

And all this tells you is which camera falls apart the soonest, not which camera you prefer or would rather use. But that may be all that anyone can reliably measure. Check out the tests that Geoff Boyle does at cinematography.net, including the recent one of the 12K (https://cinematography.net/BMD-12K.html). There's no editorializing or aesthetic judgments. But at least the tests are objective and reproducible. The investigator also avoids judgment based on either arbitrary grading decisions, or differences which could easily be eliminated with grading. He uses color charts, but doesn't have a whole lot to say about either color accuracy or aesthetic preference.

The answer you get from doing it this way doesn't answer the questions everyone's asking, least of all because of the limited subject matter and conditions. But you also have to wonder why, in the age of raw and log footage, we're examining normalized but otherwise uncorrected footage, when it's *meant* to be alterable and "accuracy" isn't the goal of an entertainment medium.

Thanks, John,
My goal is two fold:
1. For myself, to see if there is any meaningful picture quality advantage to "better" cameras than I currently use.
2. To see what specific and demonstrable differences there are in how the new RGBW sensor renders color.
3. To see if anyone can reliably see the differences.

I'm also going to be profiling the 12K, but that's a separate effort.

All that is to say the test is happening because I need to test for myself. :) If anyone wants to give suggestions, they are welcome.
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline

Iain Bason

  • Posts: 71
  • Joined: Sat Jul 21, 2018 11:57 am
  • Real Name: Iain Bason

Re: BMD Camera Shootout

PostSun Feb 21, 2021 4:27 pm

I don't know if this is feasible for you, but it would be nice to get an idea of how much light each camera needs in a room to avoid blown out windows on a sunny day.
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BMD Camera Shootout

PostSun Feb 21, 2021 5:44 pm

Iain Bason wrote:I don't know if this is feasible for you, but it would be nice to get an idea of how much light each camera needs in a room to avoid blown out windows on a sunny day.

Hey Lain,
I'm pretty leery of introducing something I can't control - the sun - into the equation. If it changes between cameras, which it almost certainly will, then the value of the tests is diminished.

I AM going to try to ensure the scene has a lot of dynamic range, though.
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline

John Paines

  • Posts: 4222
  • Joined: Tue Jul 28, 2015 4:04 pm

Re: BMD Camera Shootout

PostSun Feb 21, 2021 8:29 pm

I'll try it once more: short of performance so bad it's disqualifying, these generic tests simply don't and can't answer questions worth asking. When Vittorio Storaro said in 2016 that only the Sony CineAlta F65 satisfied his expectations, you take it on faith that, being Vittoro Storaro, he tested extensively and had his reasons, even if his view wasn't shared among his contemporaries.

And that's about it in a world where differences have been levelled by technology. Every camera you propose to test is capable of producing cinema, generally understood. And the better the overall standard, the smaller and less significant the distinctions. Compare what you propose to, say, a Zacuto shoot-out in 2012. The difference between best and worst in 2012 was enormous (though that didn't stop people in the audience from making awful misjudgments). Today, how well will vimeo even be able to distinguish between a $1295 and $80K camera, beyond extreme circumstances, and even less so, once the footage has gone through post?

All this *might* be more revealing in a well-controlled projection setting. But even that....
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BMD Camera Shootout

PostSun Feb 21, 2021 8:52 pm

John Paines wrote:I'll try it once more: short of performance so bad it's disqualifying, these generic tests simply don't and can't answer questions worth asking. When Vittorio Storaro said in 2016 that only the Sony CineAlta F65 satisfied his expectations, you take it on faith that, being Vittoro Storaro, he tested extensively and had his reasons, even if his view wasn't shared among his contemporaries.

And that's about it in a world where differences have been levelled by technology. Every camera you propose to test is capable of producing cinema, generally understood. And the better the overall standard, the smaller and less significant the distinctions. Compare what you propose to, say, a Zacuto shoot-out in 2012. The difference between best and worst in 2012 was enormous (though that didn't stop people in the audience from making awful misjudgments). Today, how well will vimeo even be able to distinguish between a $1295 and $80K camera, beyond extreme circumstances, and even less so, once the footage has gone through post?

All this *might* be more revealing in a well-controlled projection setting. But even that....

I get your point, I really do, but people keep saying that the point of the 12K sensor isn't the 12K. They make pretty vague references to better color, but I haven't seen an example where someone shows, "Hey, this color, see this color that bayer pattern sensors get wrong? The 12K gets right." (If someone can point to specific examples, PLEASE share.)

So I'm going to evaluate them side by side. And I'm going to measure it. If the HR-1, with nearly 1,000 patches can't reveal more accurate color rendition, then whatever theoretical advantages are moot. If I do see meaningful advantages...well then I need to start saving my pennies. :)

Again, I want to know.
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline

Ryan Earl

  • Posts: 366
  • Joined: Sun Jan 17, 2016 6:56 pm
  • Location: Philadelphia, USA

Re: BMD Camera Shootout

PostSun Feb 21, 2021 9:50 pm

joe12south wrote:
Iain Bason wrote:I don't know if this is feasible for you, but it would be nice to get an idea of how much light each camera needs in a room to avoid blown out windows on a sunny day.

Hey Lain,
I'm pretty leery of introducing something I can't control - the sun - into the equation. If it changes between cameras, which it almost certainly will, then the value of the tests is diminished.

I AM going to try to ensure the scene has a lot of dynamic range, though.


joe12south wrote:They make pretty vague references to better color, but I haven't seen an example where someone shows, "Hey, this color, see this color that bayer pattern sensors get wrong? The 12K gets right." (If someone can point to specific examples, PLEASE share.)


If you have both the Pocket 6K and URSA 12K available, and extra time, maybe you can do a quick outdoor test pointed at a landscape. I think would reveal some differences between the two, especially dense wooded areas. I'd be surprised if they didn't, not sure if that has something to do with resolution though. So, not quite blown out window on sunny day suggestion, but a real world situation for documentary wildlife capture.
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BMD Camera Shootout

PostSun Feb 21, 2021 10:30 pm

Ryan Earl wrote:If you have both the Pocket 6K and URSA 12K available, and extra time, maybe you can do a quick outdoor test pointed at a landscape. I think would reveal some differences between the two, especially dense wooded areas. I'd be surprised if they didn't, not sure if that has something to do with resolution though. So, not quite blown out window on sunny day suggestion, but a real world situation for documentary wildlife capture.

I'll try to accommodate this. I only have both of them together for a day, and it is really aggressively ugly looking outside in Nashville right now. :)
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline

John Brawley

  • Posts: 2918
  • Joined: Tue Aug 21, 2012 7:57 am
  • Location: Los Angeles California

Re: BMD Camera Shootout

PostMon Feb 22, 2021 12:09 am

First of all, good on you for attempting this. And good freaking luck not screwing it up in some way! It's a really difficult thing to do by yourself.

1. Why match the DOF ? / aperture...between cameras. I don't understand the logic
2. EF lens makes me shudder....If you're trying to compare cameras, I'd be aiming to get the best "reference" lens I can, and it should be something longer. Macros are often very flat geometrically and decent if you have to or I'd be trying to use primes, not zooms, something very well known for being neutral like a master prime, supreme,
3. Tungsten lighting AND daylight is good. Best checked with a colour temperature meter. Most sensors are 5000K native usually, so I usually gel my tungsten sources to hit 3200K AND 5000K for a daylight test.
4. Use real actual tungsten, not LED. Don't use any LED lighting at all. Not even good ones.
5. No over / underexposure ramps ? I like to use something like a leek or source 4 to do a hot slash on the back ground. Once you figure your base exposure, I then meter and mark the +1, +2 +3 etc stops on the slash. You can usually then READ a clipping point on a waveform.
Go to 4:25 on this for an example.
6. Charts. Charts are great for showing you ONLY how a chart reproduces. A face tells you much more.
7. If you're doing over and under, make sure you think about how you're going to do that. Shutter speed is a good way of controlling exposure without affecting the look in terms of DOF. You need a bright light to start with though.
8. Be super organised. Create a shot list of what you want to do and do it in that order.
9. Have someone else slate for you. Don't forget to slate.
10. Set exposure based on in camera metering based on FC. I find a grey card set correctly is the best way. You may already know that in FC mid grey green is at 38.4% with BMD. So get an 18% grey patch somewhere in there, hit FC and adjust the exposure till you get that patch green. THEN you know you're at the same exposure for each camera. You will find the aperture isn't always where it should be, but you will get the same mid point of exposure. Don't assume F4 is the same on every camera. do it by using the in camera metering (and note the difference offset if you like on the slate)
11. Make sure you set the chart properly. Sorry if you know this already but to state the obvious...set the chart square to the camera first and THEN you tilt it 1/3 of the way to the key light from camera in all axis. In the example I posted above you'll notice the chart is pointing slightly towards the key.
10. Ef just sucks for this kind of test....I am already queasy you're going to this effort ...
11. Make sure you contain any spill especially from coloured surfaces in the room. Try to back out the floor.
12. With super saturated chroma sources like LED's it might be more useful to do a seperate exposure ramp and run the iris up and down on them. Don't mistake chroma clipping (gamut) for exposure clipping.
13. If you're looking for IR, then best to test with a synthetic black fabric. Velvet isn't always the best at this.

It's super easy to forget a setting, bump a light or a chart and you can easily spoil your results. Be mindful of this.

You're going to be the one that's in the best position to see the end results. By the time a file is rendered and compressed it's very difficult to see any nuance. And Blind test to a default REC 709 muddies the waters because no-one ever grades an image that way anyway.

I would say it's more useful to have a LOOK you are going for and try to GRADE you images to that look. This will tell you more about how each camera grades rather than slapping on the default LUT and pixel peeping a chip chart.

More than a colour chart I prefer the light cube.
https://www.datacolor.com/photography-d ... pydercube/

It has a light trap so you get true photographic black always, plus near black, mid grey white and specula highlights. Far more useful in my view.

You can also make your own light trap. Take a shoe box, paint the inside black and open a 1" square at one end and photograph that. You'll always have a black reference then.

I'm sure there's more, but that's off the top of my head...

JB
John Brawley ACS
Cinematographer
Currently - Los Angeles
Offline

Ryan Earl

  • Posts: 366
  • Joined: Sun Jan 17, 2016 6:56 pm
  • Location: Philadelphia, USA

Re: BMD Camera Shootout

PostMon Feb 22, 2021 1:05 am

joe12south wrote:1. For myself, to see if there is any meaningful picture quality advantage to "better" cameras than I currently use.
2. To see what specific and demonstrable differences there are in how the new RGBW sensor renders color.
3. To see if anyone can reliably see the differences.


John Brawley wrote:1. Why match the DOF ? / aperture...between cameras. I don't understand the logic


Because you could find a meaningful advantage by showing the differences between sensor sizes and resolution?

What about including the 12K's Super16mm 6K crop, then the various Super16 modes from the other cameras? Or why not alter the Pocket 4K with a booster and look at super35ish separately? It could help create a limitation for the test to look specifically at color / noise, etc.

I'm brainstorming here of course. . .
Offline
User avatar

Robert Niessner

  • Posts: 3510
  • Joined: Thu Feb 21, 2013 9:51 am
  • Location: Graz, Austria

Re: BMD Camera Shootout

PostMon Feb 22, 2021 1:44 am

Plants for checking the color of green leaves/grass.
And Start/Stop via Bluetooth remote. I have seen so many tests where the image was still shaking from the push of the button.
Saying "Thx for help!" is not a crime.
--------------------------------
Robert Niessner
LAUFBILDkommission
Graz / Austria
--------------------------------
Blackmagic Camera Blog (German):
http://laufbildkommission.wordpress.com

Read the blog in English via Google Translate:
http://tinyurl.com/pjf6a3m
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BMD Camera Shootout

PostMon Feb 22, 2021 2:06 am

1. Why match the DOF ? / aperture...between cameras. I don't understand the logic

Because that would give away the FF cameras vs the S35 cameras vs the MFT cameras. If these differences are the reason for someone choosing a camera, then they already know what to expect.

2. EF lens makes me shudder....If you're trying to compare cameras, I'd be aiming to get the best "reference" lens I can, and it should be something longer. Macros are often very flat geometrically and decent if you have to or I'd be trying to use primes, not zooms, something very well known for being neutral like a master prime, supreme,

We're going to use the Sigma 35mm T1.5 FF High Speed Prime Cine Lens. It's the best choice I have available that I can put on all cameras. Trying to put a PL mount lens on the EF mount cameras is just too likely to introduce complications.

3. Tungsten lighting AND daylight is good. Best checked with a colour temperature meter. Most sensors are 5000K native usually, so I usually gel my tungsten sources to hit 3200K AND 5000K for a daylight test.
4. Use real actual tungsten, not LED. Don't use any LED lighting at all. Not even good ones.

Indeed. I don't want to introduce the variability of the sun, but we can daylight gel the tungsten lights. I don't personally use tungsten any more, so I'm borrowing some old Lowel Totas and Omnis.

5. No over / underexposure ramps ? I like to use something like a leek or source 4 to do a hot slash on the back ground. Once you figure your base exposure, I then meter and mark the +1, +2 +3 etc stops on the slash. You can usually then READ a clipping point on a waveform.

Really hot slash across the background is a good idea. (I'll save the model's eyes.) I should be able to make one 4 stops over. There are gobs of examples showing how all the cameras we're testing handle under and over exposure. I don't particularly need to see even more, but if people really want it, I guess we could. I wanted to avoid odd shutter speeds because this scene is going to be in MOTION. I have no desire to evaluate still images from MOTION cameras.

6. Charts. Charts are great for showing you ONLY how a chart reproduces. A face tells you much more.

Couldn't agree more. The charts will be there for calibration. The models will be there for us humans to look at. ;)

8. Be super organised. Create a shot list of what you want to do and do it in that order.
9. Have someone else slate for you. Don't forget to slate.

Yeah, screwing up the slates would be a nightmare. At least two of my regular crew will be there. I could get more help, but due to COVID-19 protocols, I want to keep the crew as small as possible.

10. Set exposure based on in camera metering based on FC. I find a grey card set correctly is the best way. You may already know that in FC mid grey green is at 38.4% with BMD. So get an 18% grey patch somewhere in there, hit FC and adjust the exposure till you get that patch green. THEN you know you're at the same exposure for each camera. You will find the aperture isn't always where it should be, but you will get the same mid point of exposure. Don't assume F4 is the same on every camera. do it by using the in camera metering (and note the difference offset if you like on the slate)

Each camera will be set at the manufacturers recommended ISO (I hate the term "native") and recommended level for 18% grey. When in doubt, we'll use 40 IRE.

12. With super saturated chroma sources like LED's it might be more useful to do a seperate exposure ramp and run the iris up and down on them. Don't mistake chroma clipping (gamut) for exposure clipping.

I've had to do a more gamut clip testing creating Gen 4 luts than I ever care to do again, so unfortunately for me, I've got that down to a science. ;)

13. If you're looking for IR, then best to test with a synthetic black fabric. Velvet isn't always the best at this.

I wasn't planning on testing for IR pollution specifically, but I can make sure to include a synthetic black instead of or in addition to velvet.
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BMD Camera Shootout

PostMon Feb 22, 2021 2:14 am

John,
Since you have more experience with the 12K than almost anyone, can you point me towards challenging colors to include in the scene that you think give the new sensor a chance to shine?

This is honestly driving my testing more than anything else...to see if there is any real world advantage to resolving color with the new RGBW sensor over the traditional bayer pattern.

The HR-1 will reveal even very, very subtle errors, so I'll come away with an accurate scientific delta, but I'm at least as interested in seeing how key real world memory colors render, so your input would be greatly appreciated.
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BMD Camera Shootout

PostMon Feb 22, 2021 2:22 am

Ryan Earl wrote:Because you could find a meaningful advantage by showing the differences between sensor sizes and resolution?

Sensor size introduces a well recognized, but commonly misunderstood, difference. It's not necessarily an advantage. (You tipped your hand there.) Certainly not to the poor 1st AC trying to find focus at T2. ;)

If you want the look of a certain sensor size at a certain distance at a certain aperture setting - you already know that and to limit your camera search to cameras with that size sensor. Nothing to test.

And we already know the FOV and DOF of any given lens at any given distance at any give sensor size. Nothing I need to test there, either.

And if someone believes that large format has some kind of magical properties that can't be explained by the above, well, then no test ever is going to convince them otherwise. ;)
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BMD Camera Shootout

PostMon Feb 22, 2021 2:25 am

Robert Niessner wrote:Plants for checking the color of green leaves/grass.
And Start/Stop via Bluetooth remote. I have seen so many tests where the image was still shaking from the push of the button.

Good suggestions, Robert. Thank you. Most but not all of the cameras being tested can start recording remotely. We'll try to do so when possible.
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline

Mark Wyatt

  • Posts: 215
  • Joined: Wed Sep 25, 2013 4:36 pm
  • Location: Vancouver

Re: BMD Camera Shootout

PostMon Feb 22, 2021 2:36 am

Hi Joe,

Regarding the 12k - I would make sure you test your specific model on how it performs relating to different ISO's and how each compression ratio reacts with each specific resolution. For example, I find shooting at 400 in lowlight can deliver nicer results than at 800, and I find CQ generally performs better than CB. However, in recent firmware versions, it seems that all compression ratios are extremely high quality and it is very hard to tell them apart. As well, there is a lot of high-quality shadow recovery available if needed.

Overall, I find you need to approach using this camera a bit differently than previous Ursa's, but once you find what works for you the results can be fantastic. There is something about the colour, the motion cadence and overall feel of the image which is quite exciting.

I don't know how useful camera shootouts are, but it is always interesting to see how cameras perform when they are pushed to their extremes.

Cheers
www.wyattvisuals.com
Offline

John Brawley

  • Posts: 2918
  • Joined: Tue Aug 21, 2012 7:57 am
  • Location: Los Angeles California

Re: BMD Camera Shootout

PostMon Feb 22, 2021 2:45 am

I'm very reluctant to put any faith in a chart. And I think you do.

Even talking about "errors"....

A chart is just a known example of a few tones, Maybe even 1000 tones. But it's not all the tones in all the lighting situations you might encounter. There's a few variables that can change that.

It's a reference, a useful reference, but no one actually want's accurate colours. No one MAKES a camera that actually does accurate colours. If anyone did, all cameras would look the same.

Kodak and fuji both make film stocks that are interpretations of reality. They would show up as errors on a chart, but people are spending an awful lot of time trying to replicate their look and embody those "errors" as a LUT.

A chart of tones is someones opinion about colour choices.

A camera sensor designed might make different choices and then the person making the transform to say REC 2020 or 709 might make different choices again.

Whos to say what's the right interpretation. A chart will fix those choices in a way you can compare, but I'm just not convinced that's as meaningful as I once did.

You can use the chart colour correction tool in resolve. I mean if the chart really was super accurate all you'd have to do is use the chart tool and never have to grade again right ? But every time I've used it, it looks a bit whack. There's not enough sample points (even with 1000) and there's too many other variables.

The best thing is a human face, hopefully with some skin tone variety. Then there's what looks accurate, which is rarely what looks "good".

JB
John Brawley ACS
Cinematographer
Currently - Los Angeles
Offline

John Brawley

  • Posts: 2918
  • Joined: Tue Aug 21, 2012 7:57 am
  • Location: Los Angeles California

Re: BMD Camera Shootout

PostMon Feb 22, 2021 2:48 am

joe12south wrote:
Ryan Earl wrote:Because you could find a meaningful advantage by showing the differences between sensor sizes and resolution?

Sensor size introduces a well recognized, but commonly misunderstood, difference. It's not necessarily an advantage. (You tipped your hand there.) Certainly not to the poor 1st AC trying to find focus at T2. ;)


It sounds like you intend to match the field of view and the depth of field ?

That's going to mean you're moving the subject to camera distance, or you're changing the focal length?

That's going to change the perception of the subject.

I don't think it zeros out a difference like you think it will.

Also the cameras have different base ISO. How will you control the exposure differences ? ND filters will affect the results. You need a fecking bright enough light that you can do all this, and dimming them will also change the perception. You can only wire the light if it's bright enough and totas aren't going to cut it I think. That's why I suggested shutter as exposure control for "look" tests.

JB
John Brawley ACS
Cinematographer
Currently - Los Angeles
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BMD Camera Shootout

PostMon Feb 22, 2021 2:53 am

Mark Wyatt wrote:Regarding the 12k - I would make sure you test your specific model on how it performs relating to different ISO's and how each compression ratio reacts with each specific resolution. For example, I find shooting at 400 in lowlight can deliver nicer results than at 800, and I find CQ generally performs better than CB. However, in recent firmware versions, it seems that all compression ratios are extremely high quality and it is very hard to tell them apart. As well, there is a lot of high-quality shadow recovery available if needed.

For my own edification I will test at each compression ratio, but in the spirit of giving the 12K every opportunity to show its best relative to other cameras, is there any reason to shoot the "prime" test at anything below the highest data rate our media will support?
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline

John Brawley

  • Posts: 2918
  • Joined: Tue Aug 21, 2012 7:57 am
  • Location: Los Angeles California

Re: BMD Camera Shootout

PostMon Feb 22, 2021 2:58 am

joe12south wrote:
10. Set exposure based on in camera metering based on FC. I find a grey card set correctly is the best way. You may already know that in FC mid grey green is at 38.4% with BMD. So get an 18% grey patch somewhere in there, hit FC and adjust the exposure till you get that patch green. THEN you know you're at the same exposure for each camera. You will find the aperture isn't always where it should be, but you will get the same mid point of exposure. Don't assume F4 is the same on every camera. do it by using the in camera metering (and note the difference offset if you like on the slate)

Each camera will be set at the manufacturers recommended ISO (I hate the term "native") and recommended level for 18% grey. When in doubt, we'll use 40 IRE.



I think you're approaching this the wrong way.

Middle grey in BMD film is 38.4% / IRE as I said. Are you metering using a meter ? A waveform ? How are you determining exposure ?

JB
John Brawley ACS
Cinematographer
Currently - Los Angeles
Offline

Mark Wyatt

  • Posts: 215
  • Joined: Wed Sep 25, 2013 4:36 pm
  • Location: Vancouver

Re: BMD Camera Shootout

PostMon Feb 22, 2021 2:59 am

joe12south wrote:
Mark Wyatt wrote:Regarding the 12k - I would make sure you test your specific model on how it performs relating to different ISO's and how each compression ratio reacts with each specific resolution. For example, I find shooting at 400 in lowlight can deliver nicer results than at 800, and I find CQ generally performs better than CB. However, in recent firmware versions, it seems that all compression ratios are extremely high quality and it is very hard to tell them apart. As well, there is a lot of high-quality shadow recovery available if needed.

For my own edification I will test at each compression ratio, but in the spirit of giving the 12K every opportunity to show its best relative to other cameras, is there any reason to shoot the "prime" test at anything below the highest data rate our media will support?


No, I think shooting at the highest data rate should yield the best results.

Though, I think a lot of people believe it's essential to have incredibly fast cards, lots of storage, powerful pc, etc. to use the 12k, when it can still deliver exceptional results at 4k with high compression.
Last edited by Mark Wyatt on Mon Feb 22, 2021 3:57 am, edited 1 time in total.
www.wyattvisuals.com
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BMD Camera Shootout

PostMon Feb 22, 2021 3:40 am

John Brawley wrote:I'm very reluctant to put any faith in a chart. And I think you do.

Even talking about "errors"....

A chart is just a known example of a few tones, Maybe even 1000 tones. But it's not all the tones in all the lighting situations you might encounter. There's a few variables that can change that.

It's a reference, a useful reference, but no one actually want's accurate colours. No one MAKES a camera that actually does accurate colours. If anyone did, all cameras would look the same.

Kodak and fuji both make film stocks that are interpretations of reality. They would show up as errors on a chart, but people are spending an awful lot of time trying to replicate their look and embody those "errors" as a LUT.

A chart of tones is someones opinion about colour choices.

A camera sensor designed might make different choices and then the person making the transform to say REC 2020 or 709 might make different choices again.

Whos to say what's the right interpretation. A chart will fix those choices in a way you can compare, but I'm just not convinced that's as meaningful as I once did.

You can use the chart colour correction tool in resolve. I mean if the chart really was super accurate all you'd have to do is use the chart tool and never have to grade again right ? But every time I've used it, it looks a bit whack. There's not enough sample points (even with 1000) and there's too many other variables.

The best thing is a human face, hopefully with some skin tone variety. Then there's what looks accurate, which is rarely what looks "good".

JB

Good thoughts, John.

1. The artist in me doesn't give a rat's ass about delivering accurate color. I want to deliver the color I want to deliver. But similar to the way that I need to be able to trust that my monitor delivers an accurate starting point based on an agreed upon reference – or at least I know in what ways it doesn't – I want to know what to expect from a camera. (I know that is a very imperfect analogy.) It's useful to know if a certain camera pushes red but starves green.

2. There actually are use cases where it is important to be able to render color by-the-book accurate, at least before then making certain parts of the image subjectively more pleasing. Ryobi's green needs to be Ryobi green when Ryobi is paying you the green. ;)

3. Even if one doesn't care at all about accuracy, we still need an objective way to evaluate color, and claims about color. We need a way to be able to talk about color.
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline

John Brawley

  • Posts: 2918
  • Joined: Tue Aug 21, 2012 7:57 am
  • Location: Los Angeles California

Re: BMD Camera Shootout

PostMon Feb 22, 2021 3:43 am

I guess it's the difference between testing as it comes out of the box, vs what you can actually do with it.

The second part is the one that counts in my view.

JB
John Brawley ACS
Cinematographer
Currently - Los Angeles
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BMD Camera Shootout

PostMon Feb 22, 2021 3:52 am

John Brawley wrote:I think you're approaching this the wrong way.

Middle grey in BMD film is 38.4% / IRE as I said. Are you metering using a meter ? A waveform ? How are you determining exposure ?

JB

I think we're on the same page. We'll be setting light ratios based on a meter, but use waveform to expose each camera. For each camera we will expose middle grey as recommended. In the case of the BMD cameras that will be 38.4 IRE as you stated. But I believe some of the cameras we will be testing do not have an official recommendation from the manufacturer. In those cases, we will use 40 IRE.

PS. Correct me if I'm wrong, but the margin of error on BMD's internal false color is greater than the difference between 38.4 and 40. "Green" covers more than +/- 1.6%.
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BMD Camera Shootout

PostMon Feb 22, 2021 3:54 am

John Brawley wrote:I guess it's the difference between testing as it comes out of the box, vs what you can actually do with it.

The second part is the one that counts in my view.

JB

My philosophy is that understanding the former helps improve the latter.
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline

John Brawley

  • Posts: 2918
  • Joined: Tue Aug 21, 2012 7:57 am
  • Location: Los Angeles California

Re: BMD Camera Shootout

PostMon Feb 22, 2021 3:55 am

joe12south wrote:
PS. Correct me if I'm wrong, but the margin of error on BMD's internal false color is greater than the difference between 38.4 and 40. "Green" covers more than +/- 1.6%.


Yes as you set your exposure, start under and iris up / shutter down slowly and jusssssst as it starts to turn green across the whole patch.

That’s how I’ve done it.

JB
John Brawley ACS
Cinematographer
Currently - Los Angeles
Offline

John Brawley

  • Posts: 2918
  • Joined: Tue Aug 21, 2012 7:57 am
  • Location: Los Angeles California

Re: BMD Camera Shootout

PostMon Feb 22, 2021 3:56 am

joe12south wrote:
John Brawley wrote:I guess it's the difference between testing as it comes out of the box, vs what you can actually do with it.

The second part is the one that counts in my view.

JB

My philosophy is that understanding the former helps improve the latter.


Yes but you’re posting the former as the end result, not the later.

JB
John Brawley ACS
Cinematographer
Currently - Los Angeles
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BMD Camera Shootout

PostMon Feb 22, 2021 4:09 am

It sounds like you intend to match the field of view and the depth of field ?

That's going to mean you're moving the subject to camera distance, or you're changing the focal length?

That's going to change the perception of the subject.

I don't think it zeros out a difference like you think it will.

We won't be able to match exactly, but combined with keeping the lens stopped down (hopefully no lower than T4) we should be able to get close enough that the differences in sensor sizes aren't glaringly obvious. I'm sure some full-frame-fetish-freaks will be able to pick-out the R5. ;)

Also the cameras have different base ISO. How will you control the exposure differences ? ND filters will affect the results. You need a fecking bright enough light that you can do all this, and dimming them will also change the perception. You can only wire the light if it's bright enough and totas aren't going to cut it I think. That's why I suggested shutter as exposure control for "look" tests.

I won't adjust the lights per camera once set. My plan is to light for the least sensitive camera and then 1st use aperture to adjust and second shutter speed if needed. I think we have +/- 3,000 watts of to throw at it. I'll do most of the set-up the day before, so if that's not cutting it, I think I have access to two 1000w Arri open-face floods. (I'm all LED at this point so I have borrow/rent.)
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BMD Camera Shootout

PostMon Feb 22, 2021 4:26 am

John Brawley wrote:
joe12south wrote:
John Brawley wrote:I guess it's the difference between testing as it comes out of the box, vs what you can actually do with it.

The second part is the one that counts in my view.

JB

My philosophy is that understanding the former helps improve the latter.


Yes but you’re posting the former as the end result, not the later.

JB

The test is the deliverable, but not the end goal. A tool. I know many won't, but hopefully at least a few people will take it for what is worth.
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline

Ryan Earl

  • Posts: 366
  • Joined: Sun Jan 17, 2016 6:56 pm
  • Location: Philadelphia, USA

Re: BMD Camera Shootout

PostMon Feb 22, 2021 12:26 pm

John Brawley wrote:
joe12south wrote:
Ryan Earl wrote:Because you could find a meaningful advantage by showing the differences between sensor sizes and resolution?

Sensor size introduces a well recognized, but commonly misunderstood, difference. It's not necessarily an advantage. (You tipped your hand there.) Certainly not to the poor 1st AC trying to find focus at T2. ;)


It sounds like you intend to match the field of view and the depth of field ?

That's going to mean you're moving the subject to camera distance, or you're changing the focal length?

That's going to change the perception of the subject.

I don't think it zeros out a difference like you think it will.

JB


I agree that the perception will change when moving the lens back and forth. What I was getting at is that the cameras mentioned could all be compared as super16 cameras, either by setting the mode in the camera or cropping in post. Especially since I’m reading that’s an often requested camera; to slice down the 12k sensor for MFT and use s16 lenses again on a more affordable setup. The Micro Cinema camera is still current along with the super16 in the G2.

I did this test for example by setting up the shot in the room starting with the lens on the pocket 4K, then fixing the lens position with string lines across the room. The super16 6K mode from the 12K URSA is built right in to the menu.

It could be to your advantage to have a look at this and find it lacking for example or that it outperforms in the areas you are defining, key being color.
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BMD Camera Shootout

PostMon Feb 22, 2021 2:28 pm

Ryan Earl wrote:I agree that the perception will change when moving the lens back and forth. What I was getting at is that the cameras mentioned could all be compared as super16 cameras, either by setting the mode in the camera or cropping in post. Especially since I’m reading that’s an often requested camera; to slice down the 12k sensor for MFT and use s16 lenses again on a more affordable setup. The Micro Cinema camera is still current along with the super16 in the G2.

It's an interesting idea. We might try it for the blind test. My biggest concern is that some of the cameras will be below 4K at that crop, and a lot of people will discount a 2K deliverable in 2021.
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline

Howard Roll

  • Posts: 1227
  • Joined: Fri Jun 03, 2016 7:50 am

Re: BMD Camera Shootout

PostMon Feb 22, 2021 3:26 pm

I understand the fun of wanting to keep it blind but the only way to match fov/dof/lens is to change the aperture which must necessarily affect resolution and moire. Mft to FF is two stops removed, T5.6 -T11 is about where the aperture would need to sit to keep the lens from having a major impact on the overall image. T2-T4 is too soft, T8-T16 diffraction is in the house.

https://www.the-digital-picture.com/Rev ... &APIComp=7

Fabrics are a little random anyway, why not an Otus on a zone plate? Dury's in Nashville rents......bummer.

Good Luck
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BMD Camera Shootout

PostMon Feb 22, 2021 4:48 pm

Howard Roll wrote:I understand the fun of wanting to keep it blind but the only way to match fov/dof/lens is to change the aperture which must necessarily affect resolution and moire. Mft to FF is two stops removed, T5.6 -T11 is about where the aperture would need to sit to keep the lens from having a major impact on the overall image. T2-T4 is too soft, T8-T16 diffraction is in the house.

It will be interesting to see if you can reliably identify the difference at 4K. ;)

Fabrics are a little random anyway, why not an Otus on a zone plate? Dury's in Nashville rents......bummer.k

There will be focus and resolution charts, moving and still.
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline
User avatar

MrHotter

  • Posts: 132
  • Joined: Wed Apr 25, 2018 3:10 pm
  • Real Name: christian Hotter

Re: BMD Camera Shootout

PostMon Feb 22, 2021 7:31 pm

I'm looking forward to seeing the results.

Being able to link my sub-par results to sub-par talent is a good way for me to save money on gear. :D
System:
Windows 10 x64
AMD Ryzen 5 3600
32 GB RAM
Radeon RX580 8 Gb Memory
M.2 Drive Samsung SSD 960 for program files and thumbnails/cache.
Samsung SSD 860 EVO1TB working drive
Offline
User avatar

Steve Holmlund

  • Posts: 496
  • Joined: Fri Jun 28, 2013 11:30 pm
  • Location: Montara, California

Re: BMD Camera Shootout

PostMon Feb 22, 2021 7:50 pm

joe12south wrote:...Each camera's rendition will be posted online, blind, with a few different polls so that interested participants can try to identify and/or indicate preference. After collecting enough responses to be statistically meaningful I will share results, and probably share the source files....


Joe,
Very interesting test and it sounds like quite a bit of work. This may be premature but it also may be worth getting input on the questions the polls ask and how the samples are presented. For example, it would be good to know how respondents viewed the files (e.g. on an iPhone vs. a calibrated grading monitor) when they made their judgments.

Good luck.
Steve
Steve Holmlund
Hobbyist
BMPCC, vintage Rokkor lenses, Olympus 12-40 and 12-100, Panasonic 100-300 II
SmallHD Focus, i7 8700k / GTX 1080
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BMD Camera Shootout

PostMon Feb 22, 2021 8:16 pm

Steve Holmlund wrote:For example, it would be good to know how respondents viewed the files (e.g. on an iPhone vs. a calibrated grading monitor) when they made their judgments.

Oh, yeah, that's a great idea. Hmmm...what I wonder what other interesting data points we can come-up with?
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BMD Camera Shootout

PostMon Feb 22, 2021 8:20 pm

MrHotter wrote:I'm looking forward to seeing the results.

Being able to link my sub-par results to sub-par talent is a good way for me to save money on gear. :D

I just finished about a month in post on a film, so I'm at that "I hate everything about it" stage. Even still, of the million things that I wish were done better or different, I can count on one hand the number that would have been materially impacted by camera choice. If only I could find a camera that could make me a better writer... :D
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline

CaptainHook

Blackmagic Design

  • Posts: 1839
  • Joined: Wed Aug 22, 2012 4:50 am
  • Location: Melbourne, Australia
  • Real Name: Hook

Re: BMD Camera Shootout

PostTue Feb 23, 2021 12:27 am

joe12south wrote:My plan is to light for the least sensitive camera and then 1st use aperture to adjust and second shutter speed if needed.

If you are planning to compare colour I would avoid adjusting the aperture completely.
**Any post by me prior to Aug 2014 was before i started working for Blackmagic**
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BMD Camera Shootout

PostTue Feb 23, 2021 12:57 am

CaptainHook wrote:
joe12south wrote:My plan is to light for the least sensitive camera and then 1st use aperture to adjust and second shutter speed if needed.

If you are planning to compare colour I would avoid adjusting the aperture completely.

You expect the lens's color rendition will meaningfully change?
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline

CaptainHook

Blackmagic Design

  • Posts: 1839
  • Joined: Wed Aug 22, 2012 4:50 am
  • Location: Melbourne, Australia
  • Real Name: Hook

Re: BMD Camera Shootout

PostTue Feb 23, 2021 2:17 am

joe12south wrote:
CaptainHook wrote:
joe12south wrote:My plan is to light for the least sensitive camera and then 1st use aperture to adjust and second shutter speed if needed.

If you are planning to compare colour I would avoid adjusting the aperture completely.

You expect the lens's color rendition will meaningfully change?

We've measured a bunch of lenses adjusting aperture and yes it will shift which for my tests at work invalidates any colour comparisons. Just changes in CA can also be problematic here. Even before I started at BMD I often had to compensate with WB/Tint for aperture changes as it was visible to me not just in testing but in actual shooting. For testing I tend to use frame rate and shutter angle (to make it easier to calculate in stops) and try light for one extreme so that I have enough range to adjust frame rate/shutter for how much I need. A colour meter is also useful to make sure your light source has reached a "stable" condition - most of the time 15-20mins is okay depending on what it is etc. Then there is how long the cameras have been on, etc. But John has already alluded to just how many things need to be managed and its VERY easy to miss one thing that means you have retest.

If you are also trying to judge motion I would suggest doing so separately where you try ignore any potential colour differences etc - in my experience trying to test multiple things at the same time often means the amount of variables lead to inconclusive results, and as already mentioned means more things to manage and more potential for error. I'm even wary of focus adjustments since it may affect CA or other lens artefacts that may influence assessments of various attributes. Most of the time in testing you try to reduce the variables down to just the one you are looking to compare, but it can be difficult in 'complex' systems like this.
**Any post by me prior to Aug 2014 was before i started working for Blackmagic**
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BMD Camera Shootout

PostTue Feb 23, 2021 2:01 pm

CaptainHook wrote:We've measured a bunch of lenses adjusting aperture and yes it will shift which for my tests at work invalidates any colour comparisons. Just changes in CA can also be problematic here. Even before I started at BMD I often had to compensate with WB/Tint for aperture changes as it was visible to me not just in testing but in actual shooting. For testing I tend to use frame rate and shutter angle (to make it easier to calculate in stops) and try light for one extreme so that I have enough range to adjust frame rate/shutter for how much I need. A colour meter is also useful to make sure your light source has reached a "stable" condition - most of the time 15-20mins is okay depending on what it is etc. Then there is how long the cameras have been on, etc. But John has already alluded to just how many things need to be managed and its VERY easy to miss one thing that means you have retest.

If you are also trying to judge motion I would suggest doing so separately where you try ignore any potential colour differences etc - in my experience trying to test multiple things at the same time often means the amount of variables lead to inconclusive results, and as already mentioned means more things to manage and more potential for error. I'm even wary of focus adjustments since it may affect CA or other lens artefacts that may influence assessments of various attributes. Most of the time in testing you try to reduce the variables down to just the one you are looking to compare, but it can be difficult in 'complex' systems like this.

Thanks, Captain. I've never noticed color shifts on the Sigma, but I've also never actually tested for them. Now, I'm curious, so I will deliberately test for it. (In fact, I'm now going to include this on all future lens tests.)

We will be doing multiple tests, so for all but the "blind shoot-out" I will keep the aperture a constant f4 and instead use the most predictable method - shutter speed – to adjust exposure.

The lights will have been on for hours before we start shooting for real, and we'll make sure each camera has been on for at least 30 minutes before sending it up to bat. We'll be in a climate controlled studio with AC power, so we'll just leave everything running as much as possible.

Despite the challenges, I am pretty determined to deliver a blind test with as much objective data in the scene as possible AND have it in motion. I, like many others, have wasted way too much time of my life evaluating and fretting over things like resolution and noise in still frames when it is perceived very differently in motion.
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline

Ryan Earl

  • Posts: 366
  • Joined: Sun Jan 17, 2016 6:56 pm
  • Location: Philadelphia, USA

Re: BMD Camera Shootout

PostTue Feb 23, 2021 4:05 pm

joe12south wrote:Despite the challenges, I am pretty determined to deliver a blind test with as much objective data in the scene as possible AND have it in motion. I, like many others, have wasted way too much time of my life evaluating and fretting over things like resolution and noise in still frames when it is perceived very differently in motion.


Still an interesting test to take on.

The 12K is giving you RGBW in high resolution with manageable data rates for motion pictures. I think with high expectations for quality that I would have only previously thought possible with RGB 16BIT TIFF film scans with massive file sizes.

This test has been around for a long time and offers some clues to what differences I'd been looking for with color in the 12K RGBW vs the others especially in the REDS and BLUES. Something more that I believe I recognized in John Brawley's samples, especially the closeups of faces and his "guy in the woods" clips.

https://www.onlandscape.co.uk/2011/12/l ... s-digital/

Worth noting the graphic in this support doc too: https://documents.blackmagicdesign.com/ ... 7414410000
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BMD Camera Shootout

PostTue Feb 23, 2021 4:49 pm

Ryan Earl wrote:The 12K is giving you RGBW in high resolution with manageable data rates for motion pictures. I think with high expectations for quality that I would have only previously thought possible with RGB 16BIT TIFF film scans with massive file sizes.

The theoretical advantages are clear, but de-bayer techniques are so mature and have gotten so good at squeezing every drop of quality out of that pattern that we'll just have to see if there is any meaningful advantage to a more balanced RGBW design...at least with this iteration. Some mobile phone manufacturers have tried using variations of RGBW sensors, but pretty much failed to demonstrate any advantages and in fact the images were generally considered inferior. As a first iteration, we can reasonably expect it to get better. There's always Gen 6. ;)

Worth noting the graphic in this support doc too: https://documents.blackmagicdesign.com/ ... 7414410000

I don't know if there is a direct correlation between Resolve having a bigger internal working space that accommodates all other spaces and the capabilities of the 12K camera. Even if it is, how does that translate to REC709 or REC2020 deliverables? If those out-of-gamut colors have to be squished (technical term) into range, is there a practical advantage to capturing them?

Lots of questions.My little test won't definitively answer them, but hopefully it will give at least a bit of insight.
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline

John Paines

  • Posts: 4222
  • Joined: Tue Jul 28, 2015 4:04 pm

Re: BMD Camera Shootout

PostTue Feb 23, 2021 5:04 pm

I think what you really want is accomplished color graders looking at the footage, to assess both the initial normalizations (accuracy, aesthetics, intangibles) and the potential of the footage, as a grading source.

But they tend not to frequent this forum and don't seem to take as much an interest in these questions as many shooters and/or potential camera buyers do -- a grader deals with what he/she is given.

OTOH, if the market for the camera lacks either the expertise to exploit the footage or the production resources to realize the camera's potential, that's also an answer, because in the end, it's what buyers will get.
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BMD Camera Shootout

PostTue Feb 23, 2021 5:42 pm

John Paines wrote:I think what you really want is accomplished color graders looking at the footage, to assess both the initial normalizations (accuracy, aesthetics, intangibles) and the potential of the footage, as a grading source.

But they tend not to frequent this forum and don't seem to take as much an interest in these questions as many shooters and/or potential camera buyers do -- a grader deals with what he/she is given.

Of course, if the market for the camera lacks either the expertise to exploit the footage or the production resources to realize the camera's potential, that's also an answer, because in the end, it's what buyers will get.

Me personally, not so much. It's not hard to find source files from any of these cameras.

It's almost the norm now for even cheap cameras to have a log profile, 10-bit encoding and 12+ stops of usable dynamic range. If they also have the latitude to be pushed 2/3 stops either direction, then they are capable of supporting any reasonable creative grade. Beyond that, we're really talking about rescue missions, and I don't care much about that. If I miss my exposure by 4 stops, I should have my camera taken away from me. ;)

I like cameras with better specs. I wouldn't complain if my camera super-sampled my output resolution. More usable dynamic range is always handy. But the primary question I'm asking myself is, what is the "good enough" camera (PQ-wise) for 4K delivery of a narrative film.

Now I understand many will scoff at the "good enough" target, but as an indie filmmaker it is an important milestone. At the point my camera is "good enough" to yield the images I want and the usability doesn't slow me down, then virtually anything else I can spend on is more important than "upgrading" my camera.

For me, personally, the P4K is almost, but not quite, there. Same with the P6K. The P6K's image wasn't different enough for me to justify upgrading. Is the 12K's? Testing the final image of the 12K against the final image of other cameras for *material* PQ advantages in a 4K deliverable is meaningful to me...and I suspect more than a few others.
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline

CaptainHook

Blackmagic Design

  • Posts: 1839
  • Joined: Wed Aug 22, 2012 4:50 am
  • Location: Melbourne, Australia
  • Real Name: Hook

Re: BMD Camera Shootout

PostWed Feb 24, 2021 1:28 am

Testing like this takes a lot of effort and time, I think people are only trying to warn that while this might be valuable for yourself (assuming everything goes well) it will be far less for others. Generally less experienced users take more from tests like this than experienced users which unfortunately can do more harm than good as it does also require skill and experience to know how to consume and take away the 'right' things in other peoples tests.

It can be incredibly difficult to trust someone else's test as you weren't there and don't know for certain what happened or if anything was missed (many times its apparent mistakes were made), let alone remove any bias that may creep in subconsciously or consciously from both the presenter and viewer. On top of that many times it seems like the real hidden motivation is for views or to sell/promote something (always be skeptical of something that goes to great lengths to explain a "problem" and then at the end offers the "solution" you have to go elsewhere for or worse, pay for).

Each person only needs to ask themselves how many tests have they seen they thought the person blew it in someway and they would do much better themselves. Likely many who have posted a test ironically have thought that themselves. Some can do and present tests reasonably well considering the challenges and reality that the majority have flaws, but the chances of no flaws is slim. And even when footage is downloadable I really believe there is also a genuine difference from shooting something yourself and grading it versus grading someone else's footage. And that's from someone who has provided a lot of footage for people to download.

Other peoples tests really do only offer limited value IMHO. If you're upfront about that in the presentation it will probably go a long way.

Good luck and I hope it goes well.
**Any post by me prior to Aug 2014 was before i started working for Blackmagic**
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BMD Camera Shootout

PostWed Feb 24, 2021 1:43 am

CaptainHook wrote:Testing like this takes a lot of effort and time, I think people are only trying to warn that while this might be valuable for yourself (assuming everything goes well) it will be far less for others. Generally less experienced users take more from tests like this than experienced users which unfortunately can do more harm than good as it does also require skill and experience to know how to consume and take away the 'right' things in other peoples tests.

It can be incredibly difficult to trust someone else's test as you weren't there and don't know for certain what happened or if anything was missed (many times its apparent mistakes were made), let alone remove any bias that may creep in subconsciously or consciously from both the presenter and viewer. On top of that many times it seems like the real hidden motivation is for views or to sell/promote something (always be skeptical of something that goes to great lengths to explain a "problem" and then at the end offers the "solution" you have to go elsewhere for or worse, pay for).

Each person only needs to ask themselves how many tests have they seen they thought the person blew it in someway and they would do much better themselves. Likely many who have posted a test ironically have thought that themselves. Some can do and present tests reasonably well considering the challenges and reality that the majority have flaws, but the chances of no flaws is slim. And even when footage is downloadable I really believe there is also a genuine difference from shooting something yourself and grading it versus grading someone else's footage. And that's from someone who has provided a lot of footage for people to download.

Other peoples tests really do only offer limited value IMHO. If you're upfront about that in the presentation it will probably go a long way.

Good luck and I hope it goes well.

I'm trying to be as transparent with my goals and the biases I am conscious of, as well as consider as much input as possible. It is a given that mistakes will be made, and if we catch them, I'll note them.

For what it is worth, my wife is a researcher. I'd never hear the end of it if I tried to use dodgy methodology. ;)

Fingers crossed until I need to hit the record button.
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline
User avatar

Robert Niessner

  • Posts: 3510
  • Joined: Thu Feb 21, 2013 9:51 am
  • Location: Graz, Austria

Re: BMD Camera Shootout

PostWed Feb 24, 2021 8:01 am

Joe, I'd really love to be there when you do the tests and assist, but we are oceans apart ;)
Saying "Thx for help!" is not a crime.
--------------------------------
Robert Niessner
LAUFBILDkommission
Graz / Austria
--------------------------------
Blackmagic Camera Blog (German):
http://laufbildkommission.wordpress.com

Read the blog in English via Google Translate:
http://tinyurl.com/pjf6a3m
Offline
User avatar

joe12south

  • Posts: 734
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BMD Camera Shootout

PostWed Feb 24, 2021 2:00 pm

Robert Niessner wrote:Joe, I'd really love to be there when you do the tests and assist, but we are oceans apart ;)

I wish I was with you on the summer side of the equator. ;)

PS. You made me realize that it has been over 20 years since I was in Australia. About a year for every hour of the flight!
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Next

Return to Cinematography

Who is online

Users browsing this forum: Google Feedfetcher, John Griffin, Majestic-12 [Bot] and 23 guests