IT really depends on whether or not you are ashamed to wear such a thing.
For example, I know of quite a number of people that think that men shouldn't wear pink, under any circumstance.
My favorite shirt is pink, and I wear it formal occasions. I'm secure enough in my masculinity. I know that I will never be mistaken for being gay. I am not ashamed of wearing it.
I say do what you want, when you want to (ok not during a job interview).
When I was being interviewed for internship (a decade ago), I showed up in my scrubs, looking scruffy and like I just came from work, which I did. ALL other interviewees I was with was wearing a business suit or something similarly formal. I wasn't ashamed to admit that I came straight from work at the hospital I was currently. I got the job, BTW, despite having truly abysmal credentials.
Wearing a cosplay costume is the same. Are you ashamed of admitting of being a fan? If yes, then don't wear it in public. Let's face it, anime isn't as mainstream as we'd like it to be, so some cosplayers aren't willing to reveal the fact that they are fans. For others though, they have no problems with it. Me, if I ever got to cosplay I wouldn't have problems eating at a fast food joint in full costume, or even taking the train home in full costume, as long as the costume isn't a hindrance to movement.
To put it another way, howe would you feel if you were a girl wearing a skirt, then some old lady comes up to you saying "Don't you have any self respect? Why are you showing your ankles?!"
Or, what is some fundamentalist starts insisting that every girl should wear a burqa when in public?
Is it okay to wear a shirt that says "I Like Porn" to your friend's birthday party, when his relatives are going to be there? If you accept that you like porn and don't mind telling people that, isn't it okay to wear a shirt that says just that? Or should you only wear that shirt when with friends, or in an environment where the shirt will be appreciated?
Showing up in white KKK hood during Martin Luther King day parade is going out of your way to actually offend people. So, if you wore "I like Porn" to a birthday party for the purpose of annoying people at that party, then you're a douche. This is not the same as the guy who hasn't taken off his Inuyasha ears, unless its a place with a known hatred of Inuyasha.