An artist's fans affect how we think of them.
Ever been watching a sporting event that you don't care at all about and you notice that one team's fans are annoying and that you hate them, so you root for the other team? You probably have. It's human nature. The same kind of concept applies to music, even though we hate to admit it. If you hear a song for the first time and simultaneously watch some burned out older gentleman with bleached hair, a bad soul patch, tinted glasses, and a faded ICP tattoo on his neck dance around with glowsticks, that is going to bury itself in your memory. That song will forever be tarnished.
We all want to believe that our judgment of music is pure, but music is more than just a bunch of sounds. It has come to represent entire cultures, and a lot of the time it becomes about identifying with not only a taste in musical styles, but a lifestyle. This opinion might not be shared often, but I think most of us can agree that it is a factor.