The empire grew because of violent coercion. And let's not act like these "migrants" were coming into the empire of their own volition. Majority of these migrants were from conquered lands whether they were conquered violently or not.
The early migrants came because of trade. Same with Athens. Same with Persia. That said, who cares whether later migrants were refugees? Rome bustled and flourished due to its inclusive nature.