Chemistry: I’m not going to be naive and make a blanket statement like, “You shouldn’t care what other people think” or “You shouldn’t be so superficial and judge people like that.” Your overall attraction toward your partner is important, regardless of where that attraction comes from.
Do you still feel sexy when you stand next to them at a party? Do you still feel secure when walking on the street alone with them? If so, then congratulations, this is not a major component in your attraction for this person. I would imagine that most people overestimate how big of a deal height differences are up-front, and so I would wait for those kind of situations to occur rather than pre-judging how the experience will turn out.