GPU memory full with stabilization

I have an issue with the stabilization feature. Whenever I run stabilization, the preview turns black afterwards, sometimes I see a distorted initial frame. That also happens if I only stabilize a few frames, also same result when I render the timeline.

I often get the "GPU memory full" error message, the analysis continues despite the error and finishes at 100% (CPU and GPU continue to work according the Task Manager). But I also don't get a result if I don't get the GPU error message. Task manager claims that only 0.8 of 1GB of GPU memory and 1,3 of 10GB of shared GPU RAM are used. The shared RAM is definitely available, only 6 out of 20GB RAM where used during that test.

My system:
Core I5-4xxx
20GB RAM
Geforce GTX 650 TI 1GB
Davinci Resolve 15 free
4K Footage, optimized media, 1080p 24fps timeline

I know it's not a powerful system, but it has worked perfectly fine so far for all I do. I'm fine waiting a while and working with optimized media and would like to avoid a massive system upgrade...

That GPU is challenged even with HD footage, and hardware demands of Resolve didn't go down with newer versions. I'd stick to 12.5 with that machine.