See https://hackerone.com/reports/435648. We currently use 80-bit TOTP keys. The RFC suggests 128 as a minimum and recommends 160.
The math suggests that doing the hashing for an 80-bit key is hard (slightly beyond the reach of a highly motivated state actor, today) but there's no reason not to use 160 bits instead to put this completely out of reach.
See some additional discussion on the HackerOne report about enormous key sizes, number of required observations, etc.