We have a historic data point
Given it was only one year and that makes it completely anecdotal, but the Patriots have played a season without Tom Brady. At the time, Matt Cassel came in for Brady and the Patriots won 11 games.
Yes, the 2008 Patriots were coming off an 18-1 season. They had Randy Moss, Wes Welker, Richard Seymour, Vince Wilfork, Tedy Bruschi, Mike Vrabel, and Rodney Harrison. They had a strong team. That brings us to . . .