I go, enjoy tailgating with my Duck homer friends and observe. Autzen is a great atmosphere. So is Neyland. You don't have to have a rooting interest to go to a football game.
I think wearing it would kinda be a dick move...
They destroyed us earlier this season, and they're a competitor to Alabama for a spot in the NCG. I don't think you'd find too many thrilled by your wardrobe choice.
But don't let me stop you from repping your team, either.