The Colts were gone, and with the exception of the brief existences of the USFL's Baltimore Stars and the CFL's Baltimore Stallions, Baltimore went without football for 13 years. Fans had no team to cling to in Baltimore and had to cling to the closest team available.
The NFL consistently spurned Baltimore's bid for a new franchise, instead giving football to Carolina and Jacksonville. Things weren't looking good for Baltimore.
That all changed when Art Modell moved the Cleveland Browns to Baltimore, under the condition that the city of Cleveland would get to keep the team name, colors, logo, band, and history.