I believe I've been very thorough in investigating this but there might be something I've missed. Here's another rundown.
Summary: 4:4:4/RGB clips look very pixelated/aliased on a DeckLink or UltraStudio Mini Monitor when
Resolve Project Settings > Master Project Settings > Video Monitoring > Use 4:4:4 SDI is disabled. I expect 4:2:2 output but it looks much worse than normal 4:2:2. The issue only affects 4:4:4/RGB clips, not 4:2:2 clips.
Here are some lossless blown up crops (PNG, 300%, nearest neighbor) of a chroma subsampling test. Click to view unscaled. Image number 3 shows the issue I'm trying to avoid.
1.
Original 1080p RGB file - Perfect
2.
Rendered as uncompressed 4:2:2 in Resolve - Good, this is expected and also what I expect the 4:2:2 monitor output on the DeckLink to look like
3.
Monitored image 1 in Resolve on a DeckLink - Extreme aliasing. Only Resolve project settings are used (Desktop Video Setup isn't even started). This image is captured from the SDI output and accurately shows what I'm seeing on the monitor. Interestingly, rendering to the Grass Valley codec shows the same issue.
4.
Monitored with workaround - Very good, the desired result! The "workaround" being enabling 4:4:4 in Resolve and then starting Desktop Video Setup and
disabling 4:4:4 there. This has the side effect of slowing down Resolve just as much as when actually using 4:4:4, no use even trying to have the internal scopes on (curiously, the slowdown disappears if you set Data Levels to Full instead of Video).
All tests performed in Windows 10 Pro, DaVinci Resolve Studio 12.5.4.019 and BM Desktop Video 10.8.4 on an HP Z840 with Dual Xeon 6-core@2.4GHz, 48GB RAM, Dual 12GB GeForce GTX Titan X (Maxwell), DeckLink 4K Pro.
Also tried on a MacBook Pro Retina mid 2014 using an UltraStudio Mini Monitor, it outputs the same pixelated/aliased image when "Use 4:4:4 SDI" is disabled.
Uncompressed 1080p DPX files of the four images above (uncropped) are attached.