It's not necessary, but cleaning out old drivers from time to time is generally a good idea.
I generally dont have to bother for GPU tho because I only install the absolute bare minimum files from Nvidia drivers (Driver, control panel, PhsyX, and HD Audio, nothing else).
If you dont know, you can extract the Nvidia driver exe and then remove folders for what you don't want installed. Even if you're unticking everything and install just drivers, Nvidia is still throwing some unneeded background processes in there.
Edit : Like so