The way I work is to keep all of my clips linked, the video and audio, so that I can move everything together as a bundled package. But when I need to trim just one portion, or adjust just one track of audio, etc, then I hold the opt key to select just that discreet clip.
What I notice though is that an opt+select seems significantly slower in interaction than a normal select procedure. Like, with opt held and a marquee selection it takes the software a second to catch up. Whereas this is not the case with a normal selection operation.
Doing this thousands of times per day really adds up the frustration with this little slowdown.
Here is a video to demonstrate--
https://vimeo.com/290788929/e52f798299