I have only had one TWH, a mare that we rescued on the way to Elmer's. Had bad feet, kept as pet until she died. Don't remember how tall she was but she was no where near as large as any draft horse breeds.
You are spot on about difficult to ride part, if even your whiskers bounce when riding...
If you ever got a knot on your head by a BIG woman as mean as a haint from the Tennessee hills, you met my wife. She enjoyed beating up young bucks messing with her riders.
I bought several Quarter horses from NA breeders when I retired. Finally got 4 fast enough to local race. Hired 4 of the cutest little Sand Mtn. girls to ride them. Had Ford 4 door F350 dually, , whole bit. Wife drove and kept the cowboys off the cowgirls. I made side bets and got the sweet feed...