Trying to retime but "add speed point" is greyed out?

As is "add keyframe". This seems like I must be doing something very basic wrong but my google fu has failed me so far. I'm trying to add speed ramps to clips from my Sony camera (XAVC S 4k). I can show the retime controls but the options to add speed points is greyed out. I can only uniformly speed up or slow down the entire clip.
What am I doing wrong?

Have you tried opening up both the Retime Controls and Retime Curve windows?

Yes, both controls are up. I'm pretty new to this. It seems to be a general issue not strictly related to the retime controls. I can't add keyframes to edit curves for audio purposes either for example. Can it be that I'm somehow in a wrong "mode"? Or do I need to import/convert the XAVC clips to something to "unlock" all the editing options?

If you first do a straight speed change and then open the retime controls, add speed point will be greyed out. But reopening the retime controls and/or retime curve usually re-enables the add speed point choice.

If you first choose retime controls, add speed point should not be greyed out. Hope that helps, and if your still having issues, try posting a link to a cellphone video of what you're seeing.

Good luck.

Edit: Nevermind, I'm apparently just an idiot. I somehow assumed that if I ever got to use "add speed point" I would then get to place the point by clicking somewhere. Obviously the speed point is created at the current cursor position. In my hours of being confused by this I managed to never have the cursor inside the clip I currently tried to retime... I'll go and stand in a corner now for a couple of hours.

The more I look into this the more confused I am. So I thought maybe I messed up something in the project. So I created a new project and imported a random clip I had lying around (from the same camera) and everything worked as expected. Then I imported one of the clips that I used on the problem project and on that one It didn't work. I have no clue how the clips are different. Both came out of my A7rII.