AnCap answer:
Keep in mind this is just theory and I'm not saying I know it would be fine and dandy.
Say the geographic US is AnCap. That would mean everybody living in the region needs some sort of protection agency to represent them. This helps to ensure they don't hurt anyone and it also protects them from harm. If a refugee found his way into the US, the first thing he would need to do is find a protection agency. He is risking too much by not being represented. He can only afford minimal protection, so he agrees he is only protected as long as he behaves (rich people can afford to **** up more, no matter the system). The second he goes grabbing ass on New Year's Eve, he gets turned over to the protection agency of the woman he groped, where he will now make restitution.
Libertarian answer:
Most/all libertarians support defending the border. I imagine they would support a system that does allow for the free movement of people, but that it's managed in specific ways. For example, there would be no government-provided safety net, incentivizing refugees to come. They would only allow refugees in after thorough processing. It's not like you arrive on our shores and get right in. That's not libertarianism.