Low-latency transfer to GPU

I'm trying to take an input from a Decklink Duo 2, push it to the GPU using GPU Direct, do some geometric transformations, then push it out to a display over the GPU's DisplayPort channel. I'm seeing about 150ms glass-to-glass latency. The same operation using a USB3 camera takes 55 ms, so I think the GPU to display part is not the problem. The bottleneck must be from the camera to the GPU, but I'm having a hard time figuring out where. I'm using the LoopThroughWithDX11Compositing sample as a starting point. Basically, that sample waits for a VideoInputFrameArrived call then uses DVP to push the frame bytes to a GPU texture. My code seems to be working as designed but still has really high latency. Are there other resources for this sort of operation? The best reference I have for Nvidia GPU Direct is the comments the DVPAPI.h file in the Decklink SDK. Is there more information somewhere?

I see similar latency in the MediaExpress program, but BlackMagic tells me that there should only be a couple of lines of latency due to the capture card. Is it possible that the camera itself is the issue? It's a PTZ Optics PTVL-ZCAM running at 1080p30 over SDI. (Apparently BlackMagic cards and this camera don't play nicely at 60 FPS, but that's another story...)

Thanks,
Rex