Improve auth/LDAP import tools to assist in linking/merging accounts and migrations across providers
Open, LowPublic

Description

Hi,

I was wondering if you could advise on how we might migrate authentication providers.

We have used phabricator in our organisation since April 2012, and now have around 200 users signed up.

Recently LDAP has become available as a authentication provider, which is fantastic news. However, we would now like to migrate all users over to LDAP and we obviously can't trust all of them to link themselves by a certain date in order to flip over.

Is there a DB query or other methodology we could use to achieve this? I had a look at the accountadmin binary but this option does not appear present.

Possibly this has all been catered for in a later version, below is the output from git status:
commit cca7758582240cc81d612811b9d151e02b7eb162
Merge: 6780f63 d1225e7
Author: root <root@ldn1-phab1.london.flextrade.com>
Date: Mon Sep 9 08:48:10 2013 +0100

Thanks,

Dominic

dprittie created this task.Jan 2 2014, 2:00 PM
dprittie updated the task description. (Show Details)
dprittie raised the priority of this task from to Needs Triage.
dprittie added a subscriber: dprittie.

We have some crude import tools, but they're contributed, not very thorough, and I think they have some known bugs.

Assuming there's no real urgency on this, you can do it very gradually. Specifically, LDAP and username/password aren't mutually exclusive, so you can run both mechanisms for a while.

  • Enable LDAP and make sure it works. Your login screen will now offer both username/password and LDAP options.
  • Once you're comfortable with it, disable username/password registration. Existing users can still log in with username/password, but new users will have to sign up with LDAP.
  • Some time far in the future, you can disable username/password login. Hopefully we'll have built better import tools by then.

At some point I want to do a pass on the import tools and improve them so they can deal with this situation. Until then, you'd have to write to the DB directly. This isn't too hard. Basically, after setting up LDAP and linking accounts (via Settings > External Accounts), a new row will be generated in the phabricator_user.user_externalaccount table. It will look something like this:

mysql> select * from user_externalaccount where accountType = 'ldap' and userPHID is not null order by id desc limit 1\G
*************************** 1. row ***************************
              id: 49
            phid: PHID-XUSR-43wkbjxz6tpbetazzxft
        userPHID: PHID-USER-idvlget33w37dva6eee6
     accountType: ldap
   accountDomain: self
   accountSecret: wm3fmnpccdfciq2laaztpiumwauvnosj
       accountID: ldapuser
     displayName: NULL
     dateCreated: 1382398750
    dateModified: 1382398758
        username: ldapuser
        realName: NULL
           email: NULL
   emailVerified: 0
profileImagePHID: NULL
      properties: {"registrationKey":"e8877bbd3adb3a00cb5048b9f3ef34a10ed24891"}
      accountURI: NULL
1 row in set (0.00 sec)

To forcefully link other accounts, you need to insert similar rows. Particularly, these field values are the relevant ones:

  • phid: PHID-XUSR-<some random unique characters>
  • userPHID: PHID of the Phabricator user account you want to link. You can find this in the user table.
  • accountType: "ldap"
  • accountDomain: "self"
  • accountSecret: some random characters
  • accountID: Name of the LDAP user account to link.

The other fields are not important. You can fill them in with reasonable values, but they mostly just affect registration pre-filing.

So, overall, there are basically two approaches:

  • Move toward this gradually, and wait for better import tools to actually stop username login. There hasn't been much demand for this, but I'd say there's like roughly an even chance it will get built in the next 6 months.
  • Go bravely into the database.

Let me know if that's helpful or if I can clarify anything.

Hi Evan,

Thank you very much for your swift response.

Ideally I would choose the gradual approach however I have noticed something that might be a potential problem.
Obviously we need to keep user account history, and I have a feeling that by having LDAP self registration enabled this may not be possible.

What I did was:

  • Enable LDAP and make sure it works
  • Disable username/password registration and enable LDAP registration.

Now what happens is that if you are an existing user with a standard username/password account and you attempt to log in to phabricator using LDAP auth (without first linking your account) you are presented with a new user registration screen.

I have not actually followed through and completed the details as I don't want to break an account. I suspect that if you completed the new user registration screen then you would not be able to link between LDAP and username/password.

I am nervous about users accidentally "breaking" their account so for now I have left registration off for LDAP and unless you tell me otherwise I will leave that disabled until everyone has been migrated across.

Since you have provided the DB details I am confident I will be able to add the necessary rows to forceably link accounts. However, I only have a basic understanding of SQL and your product. Would you be able to provide a query I could run to check if all my users have been linked?

Many Thanks

I suspect that if you completed the new user registration screen then you would not be able to link between LDAP and username/password.

Yeah, this will create a second account, and you won't be able to link things.

Would you be able to provide a query I could run to check if all my users have been linked?

Sure, I think this should show you the status of all your accounts:

SELECT u.username, IF(x.phid IS NOT NULL, "YES", "NO") AS "LinkedLDAPAccount"
FROM user u LEFT JOIN user_externalaccount x ON u.phid = x.userPHID AND x.accountType = 'ldap'
ORDER BY u.username;

Here's the output I get on my test install:

+-----------------+-------------------+
| username        | LinkedLDAPAccount |
+-----------------+-------------------+
| 0               | NO                |
| Ahmedsmoore     | NO                |
| Anyamartin      | NO                |
| asana           | NO                |
| blarpblarp      | NO                |
| bleepbloop      | NO                |
| bottybot        | NO                |
| chewnicorn      | NO                |
| cyan            | NO                |
| disabled        | NO                |
| dog             | NO                |
| duck            | NO                |
| duckling        | NO                |
| emilyoung       | NO                |
| epriestley      | NO                |
| epriestley2     | NO                |
| epriestley33    | NO                |
| epriestley3333  | NO                |
| epriestley47    | NO                |
| epriestley99    | NO                |
| epriestley992   | NO                |
| epriestleyJIRA  | NO                |
| epriestleyqq    | NO                |
| Estherwll       | NO                |
| evanpriestley   | NO                |
| isabellam       | NO                |
| jackm           | NO                |
| Jxu             | NO                |
| ldapper         | YES               |
| ldapuser        | YES               |
| moo             | NO                |
| nathanhthomas   | NO                |
| Nsmith          | NO                |
| Oowilliams      | NO                |
| orange          | NO                |
| per5            | NO                |
| per6            | NO                |
| per7            | NO                |
| persona         | NO                |
| persona3        | NO                |
| persona4        | NO                |
| phabot          | NO                |
| purple          | NO                |
| quack           | NO                |
| Sebastiangarcia | NO                |
| Sizhou          | NO                |
| twitters        | NO                |
| x2              | NO                |
| x4              | NO                |
| x5              | NO                |
| zxcb            | NO                |
+-----------------+-------------------+

Perfect, thank you very much for all your help.

I will keep an eye on the release notes in case I get stuck with the DB approach.

epriestley renamed this task from Migrate Authentication Provider? to Improve import tools to assist in linking/merging accounts and migrations across providers.Jan 2 2014, 4:26 PM
epriestley triaged this task as Low priority.
epriestley added a project: Auth.
epriestley edited this Maniphest Task.Jan 2 2014, 4:28 PM
epriestley renamed this task from Improve import tools to assist in linking/merging accounts and migrations across providers to Improve auth/LDAP import tools to assist in linking/merging accounts and migrations across providers.Jan 2 2014, 4:28 PM

Cool. I'm going to repurpose this task to track improving the import workflow.

(T4131 is also vaguely related.)

Let us know if you run into issues with the DB.

timor added a subscriber: timor.May 30 2014, 3:36 PM
tuxoff added a subscriber: tuxoff.Sep 8 2014, 9:56 AM
emaste added a subscriber: emaste.Sep 30 2014, 8:20 PM
spawnlt added a subscriber: spawnlt.EditedApr 1 2015, 11:47 AM

Hey I've started using Phabricator and quickly setup accounts for colleagues.
Now we put LDAP config and there is a problem. OLD user names are the same as the LDAP ones.. so people who log in through LDAP cannot create account because of duplicates.. any suggestions?

  1. rename old user names?
  2. ...

Log in first using the OLD method.
Then under user settings settings/panel/external/ link LDAP account.

Yep, shrinidhirao Thank you. Actually found that out recently before reading your message. What I've missed is external account linking functionality for administrator. Each user has to make this action on his own.

Sorry guys but 2 days can't setup LDAP auth against AD 2008.
My task is to allow existing user to login/authenticate with their domain user and pass to work in collaboration at Phab.
Tried everything what have means, tried all possible fields returned by ldapsearch and always end up with exception :(
Make few setups against OpenLDAP on Debian linux and no success again...
Can't think about import from LDAP before resolve this.

@d.maznekov you don't provide any details for anyone to help you. I've successfully setup LDAP authentication against an OpenLDAP service. Unfortunately what settings you provide is very dependent on how your configuration and object structure in LDAP. If you want assistance you need to provide details. Here is probably not a good place for it, perhaps in a Ponder Question.

Yeah probably don't provide additional data, but it's public secret LDAP attributes of MS AD 2008 structure as sn, cn or sAMAccountName, base DN as suppose to be ou=people, dc=domain, dc=com.
Anyway, things I've tried is:
1.ldapsearch to find suitable attribute for setup of phab:

~# ldapsearch -h ad.optixco.com -p 389 -x -b "ou=people,dc=optixco,dc=com" -D "optixco\admin" -W
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <ou=people,dc=optixco,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# people, optixco.com
dn: OU=people,DC=optixco,DC=com
objectClass: top
objectClass: organizationalUnit
ou: people
distinguishedName: OU=people,DC=optixco,DC=com
instanceType: 4
whenCreated: 20101017075129.0Z
whenChanged: 20150709130103.0Z
uSNCreated: 12675
uSNChanged: 17040910
name: people
etc.

2.Setup Phab with and w/o anonymous access, bind with optixco\admin, search fields like sn,cn,sAMAccountName and many more:

# ./bin/auth ldap
Enter LDAP Credentials


    LDAP Username:  Admin


    LDAP Password:
Connecting to LDAP...
[2015-10-09 08:45:33] EXCEPTION: (PhutilAuthCredentialException)  at [<phutil>/src/auth/PhutilLDAPAuthAdapter.php:440]
arcanist(head=master, ref.master=172c930630a9), phabricator(head=master, ref.master=2a355d8548e7), phutil(head=master, ref.master=8870e8fe9df8)
  #0 PhutilLDAPAuthAdapter::raiseConnectionException(resource, string) called at [<phutil>/src/auth/PhutilLDAPAuthAdapter.php:481]
  #1 PhutilLDAPAuthAdapter::bindLDAP(resource, string, PhutilOpaqueEnvelope) called at [<phutil>/src/auth/PhutilLDAPAuthAdapter.php:364]
  #2 PhutilLDAPAuthAdapter::establishConnection() called at [<phutil>/src/auth/PhutilLDAPAuthAdapter.php:213]
  #3 PhutilLDAPAuthAdapter::loadLDAPUserData() called at [<phutil>/src/auth/PhutilLDAPAuthAdapter.php:161]
  #4 PhutilLDAPAuthAdapter::getLDAPUserData() called at [<phutil>/src/auth/PhutilLDAPAuthAdapter.php:114]
  #5 PhutilLDAPAuthAdapter::getAccountID() called at [<phabricator>/src/applications/auth/management/PhabricatorAuthManagementLDAPWorkflow.php:59]
  #6 PhabricatorAuthManagementLDAPWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:406]
  #7 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:301]
  #8 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/setup/manage_auth.php:21]

Follow and read a lot of guides over the Internet, no success.
I'll make a question as you advice, thank you.

@d.maznekov - Configuring LDAP always feels like a chore.. Nothing stands out from what you posted. Out of curiosity are you using "optixco\admin" or "admin" for the bind?

Yes I'm feel like...stupid.
Tried all reasonable variants - admin, domain\admin, cn=admin dc=domain dc=com, always finish with exception.
If I should provide additional information, please advice.
Will give one more try and abandon the tool respectively, I don't want to make by hand 250+ users.

fabe added a comment.Oct 9 2015, 7:10 AM

@d.maznekov Here's my (working) config for an ActiveDirectory:
Hostname: IP of the DC
Port: 389
Base distinguished name: dc=company,dc=net
Search attributes: sAMAccountName
anonymous username: ldap
anonymous password: ***
username attribute: sAMAccountName
realname attribute: displayName
LDAP version: 3
No Referrals
No TLS
ActiveDirectoryDomain: DOM (domain name in short form)

This is lacking tls but if this works you can add it later.

Thanks for sharing but not luck again.
Will dig additionally.

This SQL query will map all users that don't have an LDAP mapping, allowing them to log in using LDAP as long as their LDAP username matches their Phabricator username.

-- create LDAP mappings for unmapped users
-- this query uses a subselect due to the way MySQL reacts to UUID() being
-- wrapped in REPLACE() (returns the same UUID for all rows)
-- WARNING: make sure user.userName maps to LDAP usernames, otherwise
--          user_externalaccount.accountID needs to be modified for those users
--          where it doesn't. alternatively, use Phabricator's change username
--          utility in People app to change those users' usernames before running
INSERT INTO user_externalaccount
    (phid, userPHID, accountType, accountDomain, accountSecret, accountID)
SELECT
    CONCAT("PHID-XUSR-", LEFT(REPLACE(tmp.phid, '-', ''), 20)) as phid,
    tmp.userPHID,
    tmp.accountType,
    tmp.accountDomain,
    LEFT(REPLACE(tmp.accountSecret, '-', ''), 32) as accountSecret,
    tmp.accountID
FROM (
    SELECT
        UUID() as phid,
        u.phid as userPHID,
        'ldap' as accountType,
        'self' as accountDomain,
        UUID() as accountSecret,
        u.userName as accountID
    FROM user u
    LEFT JOIN user_externalaccount x
    ON u.phid = x.userPHID AND x.accountType = 'ldap'
    WHERE x.phid IS NULL
) as tmp;

also published at https://gist.github.com/allixsenos/c2050b2d9ecbbfb7b710c9306ee5f535

This would be useful for us as well, we want to migrate from using local authentication (which was very easy to setup when initially bringing up our server) to using LDAP authentication. We had most/all of our users create accounts using their LDAP username but using local authentication, but really we should have set up LDAP from the beginning.

It seems like somewhat of a natural progression to bring up the tool with the easiest configuration and then start using/depending on it enough to want to switch over to LDAP/integrate with other tools.

For us something like bin/auth ldap link <username> <ldap_username> would be enough, and we could just iterate over our existing users and link them to LDAP and turn off the basic authentication.