Noise/speed/stamina usage, as some people have mentioned above. You'll eat your stamina bar faster if you're running in heavy armor, unless you want to be "stuck" using the Steed Stone.
Also, Light armor you can sit at 50-70 of your capacity used when you leave to go somewhere, heavy armor will eat another 50-70 of your storage space. I play a solo character and don't take followers so heavy armor would make that damn hard to pick up everything with.
XI Wiki



