Someone else reported a similar issue with a 1080Ti - in their case they could do H265 / NVENC on 17.0, then it stopped working on 17.1 and 17.2: viewtopic.php?f=21&t=145783
Resolve Studio 17.4.3 and Fusion Studio 17.4.3 on macOS 11.6.1
So I routinely update, On latest not beta version 18.1.4 still not working. I would ascribe it to the graphic card if it wasn't that other softwares (Shutter encoder most notably). Can properly use it.
Hoping someone else got the same problem and found a solution.