My biggest issue with the human polls is that they are fundamentally flawed because preseason rankings factor into where a team is placed throughout the season.
A team that is overrated in the preseason and goes on to lose games will hang around in the top 25 for longer, ultimately finishing higher than they otherwise would have.
I understand that issue as well. (compare say how long it took Duke to get in the rankings; they didn't appear until they were 9-2).
My biggest issues with them are kind of the human error in that:
1) their rankings are heavily influenced by a team's previous reputation. If a team is established (a big name school, a team that's had several great seasons in a row, a team that's won multiple national championships in its past) the voters will give it preference when ranking. They'll easily push a 9-0 Alabama, Georgia, Texas, Notre Dame, Michigan, Ohio State, etc, much higher, even quicker than they will a 9-0 team that's a Missouri, a Baylor, or a team that had a poor previous season.
2) The human polls are very short term memory. It's almost as if they can't remember past the current weekend that's being ranked, maybe at most the week before that as well. And at that, more emphasis is put on that "Team A lost this week and Team B didn't" than anything else. You see it the most with their ranking teams on bye during a conference championship game, where the losing team - due to having played an extra game
that week - will even drop below teams it had beaten (the best example I can think of is 2001 Tennessee and 2007 Missouri, the latter at 12-2 fell below a Kansas team they beat pretty soundly the week before).
But you'll also see it in situations where teams are ranked close/where a loss makes them have the same record, but because Team A lost the last week, they'll jump Team B (even if Team A beat Team B).