It does bug me, partially since I'm not white and as such probably one of the people whom "The Master Race" would want erased from the face of the Earth.
I also hate it because it stems from this really stupid "clubhouse" mentality that a lot of pop culture consumers have been developing for the past few years. "You're not a real gamer if you play Mobile games!" "You're not a real anime fan if you only watch Attack on Titan!" "You're not a real Comic fan if you just watch the Marvel movies!" That kind of attitude is bullshit.
Also, y'know, the whole "Nazi" thing. Not cool.
The sad thing is, Yahtzee coined the term, but he was using it ironically. The very attitude that the dolts who use "PC MASTER RACE" was what he was making fun of in the first place.