Sex is important in a relationship and it’s okay if this is a real concern for you. You’re not shallow for caring about physical compatibility. But before you make any big decisions, have you actually talked to him about it? Like, does he even know this is bothering you?
There are ways to work around size issues if both people are willing to put in effort. Different positions, toys, other types of intimacy. But here’s the thing, both people have to be willing to communicate and get creative. If he’s insecure about it and refuses to talk or try new things, that’s actually a bigger problem than the size itself.
Also ask yourself what kind of connection you have outside of sex. Is he kind, supportive, faithful, a good partner? Because if he’s amazing in every other way, this might be something you can work through. But if the sex is genuinely bad and it’s going to make you resentful, that resentment will poison everything else in the relationship.
Don’t stay if you know you’ll never be satisfied. That’s not fair to you or him. But don’t leave without at least having an honest conversation first. Maybe he has ideas or maybe there are solutions you haven’t thought of. Just be gentle about it because this is probably really sensitive for him too.