If you put a private key with a password on it into Passphrase, we'll fail later on when we try to use it, often silently.

I think managing the passwords is probably a lot of work without any security benefit (we'd have to store them beside the keys anyway) so the best approach would be to detect this when you add the key, then prompt you to either provide the password so we can strip it from the key or choose another key.