Eric would be better for recommendations as he lives there, but I will tell you my thoughts as a frequent visitor. If you are into history or at least interested, I would take a downtown carriage ride tour. I believe they have 3 routes, but I found it to be informative and interesting, and I had been to town several times beforehand. Fort Sumter is not anything really special, but I would consider it a must if you like history. Also, just park and walk down to the battery and Rainbow Row- you can catch lunch around that area. You can also walk around the downtown market area that has a lot of historical significance. Patriot's Point is another area to visit if you are interested in naval history- it has the USS Yorktown, an aircraft carrier and other vessels.
I am not the best to recommend restaurants downtown as we don't do it much with kids. If you want to take a little drive, go to Sullivan's Island at lunch and enjoy a restaurant on their strip. It is a quaint area, and one popular place is Poe's Tavern, known for their hamburgers. Just don't go for dinner or even on the weekend if you can avoid it. It gets crowded.
Finally, if you play golf and can get away, there are very decent courses for great rates. Patriot's Point Golf Course is reasonable with nice views. I played Wild Dunes a long time ago; it is pricey, but their links course has a beautiful finish.