What is Go To Pot?
To become worn or shabby; to fall apart.
This phrase comes from the idea that an animal that is no longer useful is eaten after being cooked in a pot.
Ted has really let himself go to pot since his college days.