Okay, that makes sense. I've been in this debate several times, but I think this is the first time I've heard a lot about the school the coach is leaving. I understand your argument, but I don't think it would be a huge issue. Coaches generally leave for better jobs/teams. I don't think you would generally see a lot of problems with that.
Not always better teams. I imagine that if it had been possible, Roy Williams would have taken Wayne Simien with him to UNC. IMO, Huggins would have taken Jacob Pullen with him to WVU, as well. Sampson would have at least tried to take Blake Griffin with him to IU (would have been tougher since Griffin is an Oklahoma native).
Those are just a few examples I can think of off the top of my head.