This is similar to the person of color getting stopped by the police. Is it just a random stop, or is it racism? Very hard to tell without statistics on large sample sizes, or direct evidence of racism. Would the student have made the same accusations if you were male? Possibly, but very hard to tell without direct evidence of sexism.
The student could be complaining just because he got a bad grade. The backing off after you told him the syllabus was from another professor could be because that professor was male. But it could also be because that professor was as you said, very senior. Or because you showed him that indeed more people used that same syllabus.
My advice in general would be to fight sexism when you have clear evidence that gender played a role. In other cases: address accusations like this on their merits or lack thereof. If people find that their accusations towards women are unfounded, that also combats sexist views.
Hopefully this post helped a bit.
Regarding your edit and how to deal with explicit sexism: I would call them out on it. However exactly how to deal with it and whether appearing as a bully is a concern is difficult to say without an explicit case. Is falls in the general case of students making inappropriate/unacceptable comments or actions. Maybe a guideline could be: is it severe enough to contact other people in the university? If not, you might admonish it more lightly if concerns about being a bully are top of your mind. If it is severe enough to contact the administration or the person at your institution to contact out of line students, let it be their decision. You can never be accused of being a bully if the punishment came for an independent third party. Especially if you presented the case to that third party plain and factually, with the question being: what should we do now?
Again it is hard to talk in general about these things. That is why I focused on the more specific case, because I could be more specific about it.