Terrible performance on Mac Mini M1

I just got my M1 Mac Mini 16GB and the performance has been quite poor.
I see people doing heavy grading with tracking at 6K, while I can't get 24 fps playback at 4K and 5 simple nodes. Exporting runs at 4 fps for both ProRes and H.264.
The SSD performance is also quite bad, with My EVO reporting in at 290 MB/S, and the Samsung T7 at 34(!) MB/S so I can't even use that one. Not connecting through a USB hub.
Has anyone seen this as well and know how to fix this?
I see people doing heavy grading with tracking at 6K, while I can't get 24 fps playback at 4K and 5 simple nodes. Exporting runs at 4 fps for both ProRes and H.264.
The SSD performance is also quite bad, with My EVO reporting in at 290 MB/S, and the Samsung T7 at 34(!) MB/S so I can't even use that one. Not connecting through a USB hub.
Has anyone seen this as well and know how to fix this?