On our team we have groups created (i.e. one for designers, developers, etc.) and then we use those groups to add subscribers and reviewers.
When I wish to remove myself as a reviewer, I have to remove the group itself, then re-add each member except for myself.
One potential solution to this problem would be to make it so that clicking the group "expands" it so that all the members are shown individually in the field, similar to what is possible in Outlook. Then it would be easy to remove the individuals without fear of forgetting to include someone.