I'm likely stepping in it but I had one of those "athletic" discussions recently so let me inject what I came up with. I also want to state that I golf myself so let's not have anyone think I'm bashing any aspect of golf. (ok, young attractive cheerleaders would be nice)
First, and this is extremely important, there MUST be a differentiation made between what is a skill (or set thereof) and what is, in and of itself, athletic. Like a hard-line Constitutionalist I take a literal stand as to what is a skill and what is athletic, which is to say the two can operate quite apart from one another. I argue that, by definition, athleticism is a purely physical measurement that is both tangible, repeatable and comparable. Strength, speed, vertical leap, etc are all examples of this. Other aspects, while they may be of tremendous value in competing in whatever sport, are added to, not included with, someones "athletic" ability. For instance, you could literally have the strongest/fastest/highest jumping/longest jumping/highest endurance person that ever trod the planet have pretty lousy hand-eye coordination and lousy "touch". There's plenty of precedent for this. Ever see someone with fantastic athletic ability that could not catch a football or shoot a basketball worth a crap?
Let's take an example oft cited in this thread which is that the golfing drive is in fact very athletic and in some way demonstrates how success at the highest levels of golf infers a high degree of athleticism. Well, for the '07 season the man universally acknowledged as the best athlete in golf could manage no better than a tie for 12th in driving distance. This means that the best athlete in the sport could not crack the top 10 in that sport's most athletic aspect. Perhaps more telling is that of the top 10 longest drivers (most athletic?) only 2 managed to finish in the top 50 in FedEx Cup ranking and more than half (6) couldn't break the top 100. Chicks may dig the long ball but what makes Tiger so darned amazing is that Tiger is neither the longest driver nor anywhere near the most accurate (152nd in driving % for '07) but is #1 for greens in regulation. He gets on from ANYWHERE and then makes the putt. (4th in putts per GIR) I apologize for being so lengthy but the point is that even if you are taking the skewed stance of using the most "athletic" guy available to prop up the golfers-as-athletes argument you still come out with the part of his game that really makes the difference (course management, distance control, touch, putting, scrambling from trouble, mental toughness) having little if any athletic components whatsoever. Those are skills/traits, not examples of any athletic prowess.
Another really good way (or at least the way I got a concession from the people I was discussing this with) is to argue what is athletically inclusive or exclusive. There is absolutely no way for us to KNOW what NFL or NBA players could have become really good at golf if they genuinely applied themselves to the task when they were young but ask yourself this; how many PGA tour members do you think could compete in the athletic components* of the NFL or NBA?
*And before you get too far into that line of thought let's all drop any self-serving specialized examples. Football QB's and to an even greater extent kickers and baseball pitchers are specialized sub-groups of how a game is played. Huge concessions for lack of athleticism are perfectly acceptable for skills in these instances. If you can make the kick/read the D and make the pass/put the bat on the ball nobody'd give a crap if you ran a 12 second 40, had a 110lb max bench and had a 2" vertical.
Anyway, athletically inclusive and exclusive. The "real" athletic sports are athletically exlusive. You MUST be an athlete (see above*) to play. The average ht/wt of the previously mentioned 10 longest golf drivers (ostensibly the most athletic) is 6'/190. Now compare those, apples to apples, with participants to the other major sports of like stature. Football RB's/DB's/WR's and basketball point guards for instance. If you aren't a really REALLY good athlete you don't even make a scout squad in those sports. You don't even get invited to the combine. It's exclusive company. Now, if you happen to have good athletic ability to go along with your skill set in less athletic games I can't imagine how that would be a detriment.
Crap. I apologize this got so long winded but I was trying to get most if not all of it out there instead of the little one liners back and forth.