Is it just me or was he pushing the ref because the ref wasn't doing his job and was just in the way? In order to allow for subs, he's supposed to stand over the ball. He wasn't, he was standing between the center (who's in his stance) and the QB. If the ref was doing his job, this never happens.
IMO the conversation went "GET OUT OF THE WAY GET OUT OF THE WAY I WANNA QUICK SNAP THEM" and the ref said "I need to let them sub because y'all subbed" and James looks back at him and goes "k, sorry"