Adding keyframes in Blender is very easy (if you know the shortcut) with I key, but removing them is a little bit more tedious job to do. My suggestion is to allow us to also delete the frames if we hover over that keyframed value field on the specific keyframe in timeline. So now if we press I again we delete the old frame from that field.
Removing all keys from only that value should be also possible to do.
The problem is that this again would be an additional shortcut, so the second addition to this feature request is to add on all animatable fields a small keyframe icon button to add keyframes and remove them.
That would make the animating job little more user-friendly and the job could be done with only using the mouse.
See the image.