I don't know what you're doing wrong since you provided zero working examples of the actual markup, but the behavior you're describing is not a thing in FF or Chrome-likes. Is this some dumbass Safari legacy issue?

Well, unless your problem is crapping pixel metrics all over the place flipping the bird at accessibility and usability. They're called EM and REM, use 'em!

Or you declared a width on the button... in which case of course it's not shrinking, that's what width does.