Currently, Phacility instances enter the "suspended" state, but just stay there forever. Their data may be destroyed after 90 days, but this is a node-level operation and instances remain bound to the shards they used as services forever.
This is largely low-impact, but has some silly effects where, for example, deployment scripts load thousands of instances and then iterate through them skipping nearly all of them. It also makes cluster compaction/rebalancing decisions more complicated because the "Suspended for less than 90 days, might be trivially reactivated" and "Suspended for 90+ days, nuked, totally dead without heroic intervention" instances aren't separated in the staff console.
Instances should have a formal "destroyed" state. The process to enter this state should be:
- confirm instance is eligible for destruction;
- run node-level destruction on all service nodes;
- de-link or disable all service nodes;
- put instance in "destroyed" state.
(Maybe a "destroying" state is also useful so cases which begin but do not complete this process are obvious.)
Once this is available, all eligible instances (out of service for 90+ days) should be destroyed.