Gender confusion

This is directed to speakers of languages with more than one gender:
Just out of interest, how jarring is it to your ear if someone confuses the gender of a noun? 
If your language has more than two genders, is it less jarring if someone confuses one gender with one of the other two more than the other?