Despite controlling the first half and going into the break ahead, thanks to a goal from the returning Jadon Sancho, Rangnick's side were unable to add to their lead.
The visitors exploited this, and levelled proceedings in the second half, before ultimately proceeding to victory on penalties.
While he felt his side were left to regret not capitalising on their early pressure, Rangnick believes we were on the receiving end of an incorrect decision to allow the equaliser to stand as Duncan Watmore handled the ball in the build-up to Matt Crooks's equaliser.
Scroll down to see Rangnick's comments, as well as more of the boss's key quotes.