Rio Ferdinand and Glenn Hoddle agreed it would be a risk for Tottenham to sack Antonio Conte.
Conte came under criticism for his tactics during Tottenham's Champions League exit on Tuesday.
Spurs were tepid as they drew 0-0 with AC Milan at home, trailing 1-0 from the first leg in Italy.
Conte is out of contract at the end of the season and there is uncertainty over his future, with Spurs now having only top four left to fight for.
Ferdinand, speaking on BT Sport, suggested the club should stick with the manager as they continue to fight for a Champions League place next season.