If you have 4 teams, then #5 is mad and "deserves to be in."
If you have 6 teams, then #7 is mad and "deserves to be in."
If you have 8 teams, then #9 is mad and "deserves to be in."
If you have 64 teams, then #65 is mad and "deserves to be in."
Until there is enough evidence to show that they can expand and have entertaining games all throughout, and it makes logistical and financial sense to do so, it won't happen. Adding another round is adding another week to the schedule, which adds more to the student-athletes.
I just don't see them changing for a while. It is what it is. And it's still better than the BCS. There is no perfect system when you have a game like football that requires at least a week turnaround. The CFP cannot be like March Madness, and the season isn't long enough for it to be like the NFL.
I'd love to see it expand, but just don't think it's going to happen for a while.