I need answers, damnit!

an ISFP family friend upon first meeting me:

HIM: People may consider you masculine, but you in fact have a very feminine side.
ME: What? People consider me masculine?
HIM: Don't let that masculinity define you, for your femininity has much to offer.
ME: Who are these people who consider me masculine? By what measure are you defining masculine vs. feminine? Are we talking behavior, appearance, etc...? How pervasive is this conceptualization? Why are human beings even broken down into this arbitrary dichotomy? I understand the biological division, but I am clearly female there. Is this a society thing? Then, it's more an issue of perception. What about me do you perceive to be masculine? Or what do you perceive others may perceive as masculine? I mean, are there specific man attributes I'm displaying that are categorically different from women, if you take out the mediating factors of culture and education?
HIM: ...

conclusion: Needless to say, he has avoided making conversation with me ever since... But to this day, I am unclear about how I come across to people in terms of masculine vs. feminine. Other than the T (that's classically found more in men than women) and my love of athletics, nothing else I can think of about my personality really screams masculine. The ENP are fairly gender neutral. Physically, I guess there's my somewhat taller stature, but I don't really think I dress or act like a man. I mean, how do men and women even act that distinguishes them from each other? Maybe it's my attitude about things. Ugh, this is going to drive me nuts for years to come...