Curves + Qualifier shifting color

I love the fact that I can use the qualifier to find the Y value on the image and it makes a point on the curve.

When I make a second point, it will shift the colors in the image. I can see the Waveform jump. It happens when I've moved the first point and then make a second, but also just clicking over the image without moving any of the points.

Even when I just have Y selected in curves, it will populate the curve with RGB points. And I can see that some of the RGB points have diverged very slightly from the straight line.

Is there any way to just have the qualifier change the Y value? Am I misunderstanding some fundamentals?

And when I want to undo a point I just created, I have to get rid of RGB points as well, which is a little more time consuming.

I'm on studio 14.0.0.078, Mac Sierra 10.12.6

But this happens on the free version too.

Thank you!

I get this too and I don't understant how this problem is not talked or adressed more since 2017.
Picking a reference changes the color of the image... why would we want that ?
Or maybe is there an option to only pick Y value and I didn't find it ?

Thanks