I think that is more commonly thought of as dose vs. load, with viral load reflecting the concentration of virus a person is carrying around.
But, to address your question - yes it does matter.
First, there is a minimum dose required for infection. This varies virus by virus. Infection of a cell and viral replication are not 100% probability events. So as you take on more and more viral particles, the probability of a few cells becoming infected and beginning viral replication increases. So, the size of that initial dose matters in whether or not you get infected.
Also, there is a decent chance that it can impact outcomes. When you are first infected, the body relies on its natural immune response (not targeted, specific immune response) to combat the infection. It can slow it down by killing some cells but typically isn't good enough to totally stop replication. It is there to keep things in check until you can develop specific, targeted antibodies. If the initial dose is very high, then two things can happen. One - the viral replication is accelerated because it begins with many more cellular infections. And two, the natural immune response comes on like a flood because it is responding to a rapid step change in foreign bodies. This can cause immune regulation issues that could cause inflammation and other issues associated with adverse viral consequences.