Brown was first diagnosed with cancer about 10 years ago when it was found in his colon and liver while he was coaching with the Cleveland Browns. After chemotherapy and surgery, he was declared cancer-free. He was diagnosed with cancer again after the 2019 season when doctors found a malignant tumor near his pancreas when he was coaching with the Dallas Cowboys. Brown took a year off from coaching before joining the University of Wisconsin.