I don't think you can outright just drop them. It just wouldn't be fair to exclude over half of the teams from the process of finding the champion. You don't KNOW whether a team can compete or not. If you did, there would not be so many upsets every year.
The Cinderella phenomenon is what makes the tournament interesting. George Mason in the final four? It still sounds unbelievable. And by taking away the automatic bids, that would never of happened because they wouldn't have been "good" (popular, well known, large fan base) enough to even have cracked the field.
I mean, why have a tournament at all, if we already "know" who the best teams are? It is better to have something guaranteed by merit, i.e. automatically qualifying by winning your conference.