Over the years, the Tampa Bay area has gained the reputation as being a baseball wasteland. Casual fans see the empty seats at Tropicana Field, and automatically assume that the Tampa Bay Rays do not have any support. Add in their payroll limitations, and one can see why those outside of the region would suspect that the Rays are hindered by the area.
In a way, this assessment is correct, at least in the Rays current situation. Tropicana Field has been outdated since even before the Rays came into existence, a dinosaur from another time. Add in the difficulties involved in getting to the stadium, due to its rather inconvenient location, and the Rays are in a difficult situation.