Conquest is never pretty, especially for those being conquered but there's a reason that the NAZIs stand out in history. War brings death and destruction to everyone in its path but it's rarely the goal of those making war to systematically destroy everyone and every thing. Usually, the goal is resource driven. In the case of the NAZIs (at least in terms of the part you're referencing) the goal was to murder every Jew in Europe and beyond. Murder and death was their goal and they carried it out efficiently.
The goal for Westward Expansion/Manifest Destiny was never the destruction of a people or peoples and that is a key difference. Indian Removal was done for gold - Andrew Jackson/Van Buren bent towards the interests of the wealthy who wanted the land many Cherokee called home because gold deposits were found. Jackson had had enough problems with Calhoun and the Nullification Crisis and Georgia was claiming they had authority over Cherokee lands even though Indian Affairs were supposed to be something only the Federal Government had authority over. The people who died on the Trail of Tears died not because the intent was to murder them for being Cherokee but because a rich man wanted to get richer and they happened to be in the way (this is how it goes in America time and time again).
My family was on the Guion Miller Roll (the Cherokee that didn't go West before Removal and who also didn't go during Removal). About 1,000 Cherokee would form the Eastern Band post-removal and even during the run up to removal there were many who argued in favor of the Cherokee. Removal was not something that was universally supported in the least and many Americans of that time were absolutely outraged by it.
More broadly Westward Expansion/Manifest Destiny were carried out without the destruction being the goal. Yes, it was absolutely a result and it doesn't excuse a number of war crimes that happened in that era but it matters that decimating a people(s) wasn't the goal. Once again rich men, expansionists, and industrialists drove American policy. Tribes that got in the way and didn't take the first unfair deal offered got dealt with the same way Pinkerton's dealt with striking workers or rebellious ranchers who dared to stand up against corporate ranches who were trying to muscle them out (if you don't know that part of history - well those ranchers were suddenly dubbed outlaws and murdered). It's what happens every time a government becomes the muscle for the interests of the few and even when the government doesn't become the muscle but allows the few to hire their own muscle and looks the other way as 'problems' are dealt with.
As terrible as it was (and it was terrible) it isn't in the same category of intentional murder by a government that makes killing a specific group its purpose. They're different types of crimes.