If you are proposing the elimination of one system for another than you should care about the results. To not care is to see only half the truth.
To apply your logic, then anything that is not inherently harmful is legal. I should be able to drink a fifth if liquor and then drive. No proof that I'm actually going to harm anyone. If I get mad at my neighbor, I should be able to aim a gun at him, threaten his life and shoot above him. No real harm done.
Correct. Of course, in the latter situation, out of respect for the preservation of his own life, your neighbor would be justified in shooting you if he truly felt that his life was immediately threatened (Kant does have a just war theory that would apply in this situation).