Most people who say they don't believe the government should have the authority to mandate masks or vaccines don't really mean that.
A man walks up to a woman he doesn't know in a mall and asks "Will you have sex with me for 10 million dollars?" The woman is quite surprised but after a moment she replies "For 10 million? Yes I'll have sex with you for 10 million". The man then says "Will you have sex with me for 10 dollars?" The woman is offended and snaps back at the man "10 dollars! What do you think I am?" The man replies "I think we've already established that, now we're trying to agree on a price."
Same thing with government mandates for masks or vaccines. If 10 million people were dying each month from Covid most people would agree with the authority to mandate. If only 10 people were dying each month most people would be offended to have masks or vaccines mandated. It's not the authority to mandate that most people have a problem with - it's that everybody has a different price.