Page MenuHomePhabricator
Create Task
Maniphest T2015

Implement Drydock
Closed, ResolvedPublic
Actions

Description

Turns out we should implement this rather than let it sit on the roadmap indefinitely.

Revisions and Commits

AbandonedD13525 [drydock/aws] Check that the instance started by a spot request is API consistent
AbandonedD13522 [drydock/aws] Check for the presence of the storage path before finalizing allocation
AbandonedD13508 [drydock/aws/working-copies-caching] Update AWS EC2 blueprint to support working copy caches
AbandonedD13413 Implement leasing working copies based on URLs and repositories
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
rPHU libphutil
AbandonedD10555 Implement escaping logic for Windows Command Prompt and Bash in PhutilCommandString
rP Phabricator
AbandonedD13985 [drydock/core] Allow execution timeout to be set when running commands
AbandonedD13984 [drydock/core] Support custom attributes on all blueprints
AbandonedD13752 [drydock/working-copies] Release leases when working copy operations fail
AbandonedD13690 [drydock/working-copies] Show host lease ID in log for working copy
AbandonedD13689 [drydock/working-copies] Set owner for working copies and show handles on attributes
AbandonedD13687 [drydock/core] Show owner link on Drydock leases
AbandonedD13686 [drydock/core] Only show active resources and leases in controllers
AbandonedD13636 [drydock/core] Allow queries to be filtered by resource type
AbandonedD13521 [drydock/core] Allow transient leases to be leased even if the resource is at it's maximum lease limit
AbandonedD13507 [drydock/working-copies/v1-caching] Implement working copy cache blueprint
AbandonedD13506 [harbormaster/working-copies/v0] Implement lease working copy build steps
AbandonedD13505 [drydock/working-copies/v0] Implement the working copy blueprint
AbandonedD13504 [drydock/core] Allow leases to request a specific resource
AbandonedD13503 [drydock/core] Allow nested leasing and resource allocation
AbandonedD12783 Implement leasing working copies from Drydock
AbandonedD12276 [drydock/libvirt] Implement a Drydock blueprint for libvirt-based hosts
AbandonedD11875 Ensure spot instances in AWS are closed properly in Drydock
AbandonedD10834 Schemata changes for Drydock
AbandonedD10776 Fix change directory for WinRM protocol
AbandonedD10557 Fix custom blueprint attributes to work when no attributes are specified
AbandonedD10556 [drydock/core] Support different shells in Harbormaster / Drydock
AbandonedD10547 Support both WinRM and SSH for Windows
AbandonedD10517 Fix more race conditions within Drydock
AbandonedD10511 Prevent allocator race condition where all resources are in "ALLOCATING" status
AbandonedD10510 Fix issue where Windows machines don't delete lease directory correctly
AbandonedD10507 Tag EC2 instance and spot requests with Drydock information
AbandonedD10506 Provide more logging for Drydock blueprints (EC2, etc)
AbandonedD10504 [drydock/core] Add more logging around why leases are broken
AbandonedD10495 [drydock/core] Use WinRM instead of SSH for Windows hosts
AbandonedD10471 Retry New-PSSession call to work around temporary failure
AbandonedD10469 [drydock/core] Implement `drydock run-command` for running commands against leases
AbandonedD10460 Allow custom attributes against blueprints and when leasing hosts
AbandonedD10459 Fix behaviour of max count on blueprints
AbandonedD10398 Implement cost estimation for resources in Drydock
AbandonedD10396 Implement support for launching spot instances in Amazon EC2
AbandonedD10304 [drydock/core] Implement resource constraints and correct locking mechanisms with yielding for Drydock
AbandonedD10377 Clean up lease directories when lease is released
AbandonedD10378 Spawn a new Powershell session when running commands on Windows
AbandonedD10394 [drydock/core] Abstract min-max blueprint and implement expiry logic
AbandonedD10395 [drydock/aws] Implement Amazon EC2 blueprint for Drydock
ClosedD7638 Restructure Drydock so that blueprints are instances in the DB
D10873rP0a3714507208 [drydock/core] Show blueprints / resources as links in Drydock view controllers
D10879rPe55a197dd683 Fix issues where Drydock queries didn't work correctly with empty arrays
D10874rP0d4f9363a0f2 Improve Drydock log search engine
D13340rP827aa05a676e Drop Windows-specific SSH code
D10544rP427712306076 Increase Drydock worker time to 24 hours
D7935rP996930da2ae0 Improve several exception behaviors for Harbormaster workers
D7922rPc020837397fe Add transactions to Drydock blueprint editing
D7921rP4489204361f0 Add Drydock default edit/view policies and a "Create Blueprint" policy
D7918rP962aca664f67 Add names to Drydock blueprints
D7908rP5502fca5f437 Make Drydock blueprint create workflow somewhat more standard
D7840rPce78bf1de4e5 Make all bin/* scripts locate their workflows dynamically
D7838rPe397103bf211 Extend all "ManagementWorkflow" classes from a base class
D7837rP1ba52fac865f Introduce DrydockQuery to slightly reduce code duplication
D7836rP9b0fa5747bf4 Make Drydock more broadly aware of policies
D7835rP16508740048e Modernize Drydock CLI management of task execution
D7833rP5fdd800bb6db Provide a Drydock "Console" controller
D7832rP9c9a9a919e7a Use ApplicationSearch for DrydockLog
D7830rPbc3912e64189 Use ApplicationSearch for Drydock Resources
D7829rPfa00e86f87b4 Use ApplicationSearch for Drydock Blueprints
D7831rP3db5833622ca Make DrydockLog policy-aware
D7828rP1a82743491bd Make Drydock Lease and Resource PHIDs use newer PHID infrastructure
D7827rP70244d3a129a Use ApplicationSearch to manage DrydockLeases
D7826rP6b2d480fe793 Make DrydockLease a policy-aware object

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
qgil added a subscriber: qgil.May 11 2014, 8:45 PM
0xseantasker added a subscriber: 0xseantasker.May 17 2014, 4:34 PM
joshuaspence added a subscriber: joshuaspence.Jun 12 2014, 2:28 AM
datr added a subscriber: datr.Jun 13 2014, 4:07 PM
alindobre added a subscriber: alindobre.Jun 16 2014, 4:58 PM
wienczny added a subscriber: wienczny.Jun 18 2014, 8:33 PM
• sectioneightinc added a subscriber: • sectioneightinc.Jun 19 2014, 7:32 PM
hach-que edited this Maniphest Task.Jul 3 2014, 1:27 AM
staticshock raised the priority of this task from Normal to Needs Triage.Jul 10 2014, 1:52 PM
staticshock triaged this task as Normal priority.
igorgatis added a subscriber: igorgatis.Jul 12 2014, 12:15 PM
hach-que added a revision: D10395: [drydock/aws] Implement Amazon EC2 blueprint for Drydock.Aug 30 2014, 8:41 AM
hach-que added a revision: D10394: [drydock/core] Abstract min-max blueprint and implement expiry logic.Aug 30 2014, 8:43 AM
hach-que added a revision: D10378: Spawn a new Powershell session when running commands on Windows.
hach-que added a revision: D10377: Clean up lease directories when lease is released.
hach-que added a revision: D10304: [drydock/core] Implement resource constraints and correct locking mechanisms with yielding for Drydock.
hach-que added a revision: D10396: Implement support for launching spot instances in Amazon EC2.Aug 30 2014, 9:26 AM
hach-que added a revision: D10398: Implement cost estimation for resources in Drydock.Aug 31 2014, 12:03 PM
hach-que added a revision: D10459: Fix behaviour of max count on blueprints.Sep 10 2014, 6:58 AM
hach-que added a revision: D10460: Allow custom attributes against blueprints and when leasing hosts.Sep 10 2014, 7:03 AM
hach-que added a revision: D10469: [drydock/core] Implement `drydock run-command` for running commands against leases.Sep 10 2014, 11:39 PM
hach-que added a revision: D10471: Retry New-PSSession call to work around temporary failure.Sep 11 2014, 5:58 AM
sylreboux removed a subscriber: sylreboux.Sep 11 2014, 6:00 AM
epriestley mentioned this in T6076: Provide infrastructure to allow users to perform leases from Arcanist.Sep 11 2014, 3:27 PM
chad edited projects, added Drydock; removed Unknown Object (Project).Sep 11 2014, 8:37 PM
hach-que added a revision: D10495: [drydock/core] Use WinRM instead of SSH for Windows hosts.Sep 15 2014, 2:33 PM
hach-que added a subtask: T6111: Give Drydock leases an explicit ownership PHID.Sep 17 2014, 1:46 AM
hach-que added a revision: D10504: [drydock/core] Add more logging around why leases are broken.Sep 17 2014, 2:09 AM
hach-que added a revision: D10506: Provide more logging for Drydock blueprints (EC2, etc).Sep 17 2014, 4:08 AM
hach-que added a revision: D10507: Tag EC2 instance and spot requests with Drydock information.Sep 17 2014, 5:42 AM
hach-que added a revision: D10510: Fix issue where Windows machines don't delete lease directory correctly.Sep 18 2014, 12:19 AM
hach-que added a revision: D10511: Prevent allocator race condition where all resources are in "ALLOCATING" status.Sep 18 2014, 1:33 AM
hach-que added a revision: D10517: Fix more race conditions within Drydock.Sep 18 2014, 4:11 PM
hach-que added a revision: D10544: Increase Drydock worker time to 24 hours.Sep 23 2014, 11:26 PM
hach-que added a revision: D10547: Support both WinRM and SSH for Windows.Sep 24 2014, 4:34 AM
hach-que added a commit: rP427712306076: Increase Drydock worker time to 24 hours.Sep 24 2014, 11:46 PM
hach-que added a revision: D10555: Implement escaping logic for Windows Command Prompt and Bash in PhutilCommandString.Sep 25 2014, 1:19 AM
hach-que added a revision: D10556: [drydock/core] Support different shells in Harbormaster / Drydock.Sep 25 2014, 2:19 AM
hach-que added a revision: D10557: Fix custom blueprint attributes to work when no attributes are specified.Sep 25 2014, 2:34 AM
ipalaus removed a subscriber: ipalaus.Oct 2 2014, 1:00 PM
fchovich added a subscriber: fchovich.Oct 16 2014, 3:41 AM
hach-que added a revision: D10776: Fix change directory for WinRM protocol.Nov 3 2014, 4:13 AM
hach-que added a revision: D10834: Schemata changes for Drydock.Nov 11 2014, 11:22 PM
hach-que added a revision: D10873: [drydock/core] Show blueprints / resources as links in Drydock view controllers.Nov 19 2014, 3:32 AM
hach-que added a revision: D10874: Improve Drydock log search engine.Nov 19 2014, 4:44 AM
hach-que added a revision: D10879: Fix issues where Drydock queries didn't work correctly with empty arrays.Nov 19 2014, 11:11 PM
avivey mentioned this in T182: Commit into repository directly from differential.Nov 29 2014, 11:05 PM
mrjoops added a subscriber: mrjoops.Dec 1 2014, 9:25 AM
gerx03 added a subscriber: gerx03.Jan 7 2015, 2:29 PM
J5lx added a subscriber: J5lx.Jan 11 2015, 3:38 AM
hach-que added a revision: D11875: Ensure spot instances in AWS are closed properly in Drydock.Feb 24 2015, 3:47 AM
devurandom added a subscriber: devurandom.Mar 2 2015, 7:40 PM
johnny-bit added a subscriber: johnny-bit.Mar 3 2015, 10:09 PM
hach-que added a revision: D12276: [drydock/libvirt] Implement a Drydock blueprint for libvirt-based hosts.Apr 4 2015, 4:46 PM
nzelinsk added a subscriber: nzelinsk.Apr 9 2015, 9:36 PM
jpoehls added a subscriber: jpoehls.Apr 15 2015, 1:38 AM
hach-que mentioned this in T7908: Implement aliases for objects.Apr 25 2015, 6:43 AM
chad merged a task: T7994: Drydock & docker.Apr 30 2015, 12:37 AM
chad mentioned this in T7994: Drydock & docker.
chad added a subscriber: tycho.tatitscheff.
tycho.tatitscheff added a comment.Apr 30 2015, 5:06 AM

As chad advice me I would ask here.

Is it any plan (in starmap or so) to allow drydock (or harbormaster?) to lease one host and then build an launch a dockerfile, lease it and perform CI test build inside?

hach-que added a revision: D12783: Implement leasing working copies from Drydock.May 9 2015, 9:22 AM
alindobre removed a subscriber: alindobre.Jun 14 2015, 4:26 PM
hach-que added a revision: D13340: Drop Windows-specific SSH code.Jun 18 2015, 2:51 PM
hach-que added a commit: rP827aa05a676e: Drop Windows-specific SSH code.Jun 19 2015, 5:06 AM
hach-que added a revision: D13413: Implement leasing working copies based on URLs and repositories.Jun 24 2015, 6:22 AM
hach-que added a revision: D13503: [drydock/core] Allow nested leasing and resource allocation.Jul 1 2015, 4:21 AM
hach-que added a revision: D13504: [drydock/core] Allow leases to request a specific resource.Jul 1 2015, 4:24 AM
hach-que added a revision: D13505: [drydock/working-copies/v0] Implement the working copy blueprint.Jul 1 2015, 4:32 AM
hach-que added a revision: D13506: [harbormaster/working-copies/v0] Implement lease working copy build steps.Jul 1 2015, 4:39 AM
hach-que added a revision: D13507: [drydock/working-copies/v1-caching] Implement working copy cache blueprint.Jul 1 2015, 4:45 AM
hach-que added a revision: D13508: [drydock/aws/working-copies-caching] Update AWS EC2 blueprint to support working copy caches.Jul 1 2015, 4:51 AM
hach-que added a revision: D13521: [drydock/core] Allow transient leases to be leased even if the resource is at it's maximum lease limit.Jul 1 2015, 11:41 PM
hach-que added a revision: D13522: [drydock/aws] Check for the presence of the storage path before finalizing allocation.Jul 2 2015, 12:42 AM
chad removed a subscriber: chad.Jul 2 2015, 1:05 AM
hach-que added a revision: D13525: [drydock/aws] Check that the instance started by a spot request is API consistent.Jul 2 2015, 3:12 AM
hach-que added a revision: D13636: [drydock/core] Allow queries to be filtered by resource type.Jul 15 2015, 5:35 AM
hach-que added a revision: D13686: [drydock/core] Only show active resources and leases in controllers.Jul 23 2015, 1:17 AM
hach-que added a revision: D13687: [drydock/core] Show owner link on Drydock leases.Jul 23 2015, 1:37 AM
hach-que added a revision: D13689: [drydock/working-copies] Set owner for working copies and show handles on attributes.Jul 23 2015, 1:56 AM
hach-que added a revision: D13690: [drydock/working-copies] Show host lease ID in log for working copy.Jul 23 2015, 2:12 AM
hach-que added a revision: D13752: [drydock/working-copies] Release leases when working copy operations fail.Jul 29 2015, 2:22 AM
hach-que added a revision: D13984: [drydock/core] Support custom attributes on all blueprints.Aug 24 2015, 6:23 AM
hach-que added a revision: D13985: [drydock/core] Allow execution timeout to be set when running commands.Aug 24 2015, 7:51 AM
hach-que added a commit: rP0d4f9363a0f2: Improve Drydock log search engine.Aug 24 2015, 11:18 AM
hach-que added a commit: rPe55a197dd683: Fix issues where Drydock queries didn't work correctly with empty arrays.Aug 24 2015, 11:23 AM
epriestley removed a parent task: T182: Commit into repository directly from differential.Aug 24 2015, 5:00 PM
epriestley closed subtask T6074: Determine the best way to prevent the "Lost an allocation race" exception in Drydock as Resolved.Sep 16 2015, 4:08 PM
epriestley closed subtask T6111: Give Drydock leases an explicit ownership PHID as Resolved.Sep 23 2015, 5:17 PM
epriestley added a commit: rP0a3714507208: [drydock/core] Show blueprints / resources as links in Drydock view controllers.Sep 23 2015, 5:52 PM
qgil removed a subscriber: qgil.Sep 23 2015, 6:55 PM
epriestley closed this task as Resolved.Sep 29 2015, 3:01 PM

I'm going to close this out since I think everything here is now reasonably represented elsewhere:

  • Overall, Drydock is now nearing v1, and is likely to go into production on this host in a limited role later today.
  • The primary followup task is T9252, which will also cover logging and error recovery.

Some additional tasks:

  • EC2 support is in T5544.
  • Tasks like T6193 / T6454 cover other far-future features.
  • I'm not convinced we need to pursue working copy caches, but T8671 has some discussion if we do.
hach-que closed subtask T5544: Implement EC2 blueprint for Drydock as Wontfix.May 15 2016, 5:36 AM
Herald added a subscriber: eadler. · View Herald TranscriptMay 15 2016, 5:36 AM
urzds added a subscriber: urzds.Jul 12 2017, 11:14 AM
Phabricator · Built by Phacility