The weather certainly could be worse for the first Saturday in November. You won't have to worry about getting wet in Ann Arbor, but you may get chilled.
First off, rain is not expected Saturday afternoon for the Wolverines game. Don't worry about wearing rain gear.
Temperatures could certainly be colder than what they will be Saturday. I wouldn't call it warm either. Temperatures will be around 48 degrees at the start of the game, and only cool to 45 degrees by the end of the game. Wear a coat because that's not that warm if you aren't moving around.