You gotta remember that a lot of CBB people see Tennessee being #1 like CFB people viewed Miss St as #1 2014.
Meaning they see it as yes, the ranking is probably currently deserved, but many people don't think they're actually the best team, won't win the title, or perhaps not even make the Final Four. Nobody thought Miss St would play for the national title, or even win the SEC West, when they were ranked #1. It's because they still had to play Alabama and Ole Miss (also then a top 5 team), actually ended up losing to both, and even ended up losing their BCS bowl. Plus Miss St football, like Tennessee basketball, isn't a traditional power. It just makes it easier to be skeptical.
For someone like Jay Williams, brought up in a blue blood program, to instinctively "well yeah but how about Duke" is totally expected. It's like a CFB analyst saying, on the day Miss St was ranked #1, to say "well yeah but how about Alabama."