In the States, I've gone from being "average" to definitely on the small side, according to clothing retailers. It's a pain in the ass.
In Japan, being "American small" was a relative advantage. I was somehow considered a size large there (somewhat of an ego boost), and most things fit me very well and looked damn good. Hot damn. This was true for pretty much everything except hats.
According to Japanese people, they have huge faces. What this really means is that they have huge heads. Now before you get riled thinking I'm saying something racially unkind, ask a Japanese person about it. Besides, I have pictures to prove it. Anyway, in Japan my head is like the head of a pin. It's miniscule.* It made for a painful time when shopping amongst the many fashionable choices of headgear.
I'd thought I would have escaped the phenomenon when returning to the States. Sadly, it's simply not true; I still have a tiny head. Normal men's hats engulf my head like pacman eating one of those little dots, and of course I can't wear women's hats, which tend to the pink and pastel side of things. I've resorted to children's hats, which are universally crappy in build quality and also hard to find in colors other than "very bright" and "related to cartoons and movie merchandising."
It's a tough life.
*Ironically enough, in Japan the statement "you have such a small face/head" is perceived as a compliment, much as "you have such a big nose" is. They're a part of the ideal beauty as represented by white people. As you can imagine, me with my child-sized head and king-sized nose, I was a frickin' god. Or at least, so the story should have been.