I turned feed, herald and email notifications off for a repository before pushing a huge number of commits, but it doesn't look like this setting actually works. It's still generating feed stories (visible in Recent Activity) and it's still sending out emails (I had to set the mail adapter to the test adapter to prevent it spamming everyone).
This option should actually prevent the repository from sending out any emails, activating any Herald rules and stop the creation of feed stories.