Is there a way to prune stale commits from a repository? Does this happen automatically?

The particular use case here is that I was (mistakenly) tracking all branches of a repository instead of just the master branch. As such, a diff was linked to a commit on commit on a non-master branch. After noticing this I change the repository to track only the master branch. The diff, however, still references the old commit.