Page MenuHomePhabricator

Support retroactive IDs and timestamps for imports
Needs RevisionPublic

Authored by cburroughs on Oct 29 2014, 3:58 PM.
Tags
None
Referenced Files
Unknown Object (File)
Thu, Nov 21, 6:43 AM
Unknown Object (File)
Sun, Nov 17, 8:29 AM
Unknown Object (File)
Fri, Nov 8, 3:15 PM
Unknown Object (File)
Sat, Oct 26, 10:13 AM
Unknown Object (File)
Oct 24 2024, 9:57 AM
Unknown Object (File)
Oct 14 2024, 11:50 AM
Unknown Object (File)
Oct 6 2024, 6:26 AM
Unknown Object (File)
Oct 6 2024, 6:26 AM

Details

Reviewers
epriestley
Group Reviewers
Blessed Reviewers
Summary

When importing from another system into phabricator it's
reasonable to want to line up as much information as possible. For
example mapping priority names or creating custom fields. This commit
allows IDs to be explicitly set for applications that normally just
increment (so ticket 123 in the foreign system is ticket 123 in
maniphest) and set timestamps for transaction (so a search for 'that
thing from 8 years ago' still works instead of everything happening at
import time.

The ability to control timestamps would also presumable be useful for
settings up test scenarios for ETL or other features.

This is based on work done by the Blender project for their migration.

lint cleanup

Test Plan

Imported 10s of thousands of trac/trello tickets and it mostly worked.

Diff Detail

Repository
rP Phabricator
Branch
import-helpers
Lint
Lint Passed
Unit
Tests Passed
Build Status
Buildable 2899
Build 2903: [Placeholder Plan] Wait for 30 Seconds

Event Timeline

cburroughs retitled this revision from to Support retroactive IDs and timestamps for imports.
cburroughs updated this object.
cburroughs edited the test plan for this revision. (Show Details)

(I wanted to capture this for posterity but I'll admit I'm not sure it's worth maintaining the added complexity unless imports are going to become a more common thing.)

epriestley added a reviewer: epriestley.

For created dates, they only get updated if they aren't set, so you can just setDateCreated() and your date will stick.

I think the same is true of setID(), except that you'll have to call insert() explicitly instead of save() (otherwise, it will see that the object has an ID and try to issue an UPDATE).

There's no way to skip updating dateModified, though, and I don't really want to bring this sort of approach into the upstream for just that. I'd tentatively accept a diff which keeps a flag like private $hasExplicitDateModified;, which gets set when a caller invokes setDateModified(). If this flag was set, we would not update the field in willSaveObject(). I think that would be a simple way to make it "sticky" like setDateCreated() is, and the insert() workaround is reasonable for setID()?

This revision now requires changes to proceed.Oct 31 2014, 11:29 PM

I've already done my production import so I'm not going to re-write all of my scripts to test, but I think the insert vs save trick look reasonable for IDs and creation dates.

I went to code up your suggestion with hasExplicitDateModified but I'm unsure why an extra field is needed at all. Why can't we use the same check as with dateCreated and end up with :

protected function willSaveObject() {
  $use_timestamps = $this->getConfigOption(self::CONFIG_TIMESTAMPS);

  if ($use_timestamps) {
    if (!$this->getDateCreated()) {
      $this->setDateCreated(time());
    }
    if (!$this->getDateModified()) {
      $this->setDateModified(time());
    }
  }

  if ($this->getConfigOption(self::CONFIG_AUX_PHID) && !$this->getPHID()) {
    $this->setPHID($this->generatePHID());
  }
}

That would give dateModified behavior identical to dateCreated. Specifically, the second time an object was saved, the dateModified would not be updated.

Hello,

may a example is available for this use case?

I want to use

CONFIG_IDS = IDS_MANUAL

and

CONFIG_TIMESTAMPS = false

.

Can I use the LISKDao for this like

protected function getConfiguration() {
    return array(
      self::CONFIG_IDS                      => self::IDS_MANUAL,
      self::CONFIG_TIMESTAMPS               => false,
    );
}

?

How can I use now maniphest.edit with arcanist? Is there a transaction for the manual ID available? How can I test this with the conduit console?

Best and thanks a lot