I wouldn't say I listened a lot, but I definitely listened more than people on the left did. I probably listened the most in college, when I had a longer commute into campus. If you separate the opinion from the bombast, I basically agreed with almost everything he said with the exception of foreign policy (Rush was a huge Iraq War guy). I always kind of got the impression that the rare instance he had a liberal caller on was a stunt. The liberals he had on were never reasonable, articulate people, which was by design. They frequently sounded like wide-eyed, foaming at the mouth folks "triggered" by him and the usually just attacked him personally.
I had stopped listening to him by that point, but one thing he did that really grinded my gears was the Sandra Fluke thing. By calling her a slut, it allowed the left to dismiss any critique of Obamacare as sexism. Once he said that, it seemed like some folks who were on the fence or had no opinion about the ACA were pushed into supporting it because the didn't want to be seen as agreeing with the conservative talk show guy who called an ACA supporter a slut. A lot of apolitical people, or people not really knowledgeable about the law who didn't have an opinion, were really turned off by that.