The antithesis of your stance is that context of life when those statues were erected is different than we view them now, and you are putting into an unwinnable debate which statues are worthy to stand and which are not. So much so, that in time the only bipartisan answer is to remove all public statues.
What do you do with the Washington Monument, Jefferson Memorial, Lincoln Memorial, etc. Those guys were a part of that lifestyle, and it's easy enough to find that Lincoln was not exactly pro-equal rights in his view of blacks. So, where do you start and stop. In my opinion, removing statues won't remove hate. People will be who they are with or without. What you will do by making issues out of inanimate objects is fuel hatred by removing them just as much as by leaving them. I think what's important is how you use them as they are.