Currently, we make a Conduit call to the admin instance on every request to retrieve data which very rarely changes ("Does this instance exist?").
We should cache this data instead. This gets slightly tricky because we do need to change this data sometimes, notably before performing maintenance. We probably need to TTL the cache specifically, so the workflow would go like:
- Normal TTL is, say, 15 minutes.
- More than 15 minutes before maintenance, reduce TTL to 15 seconds.
- More than 15 seconds before deploying changes, change instance state to "down".
- After changes, change instance state back to "up".
- Once changes are stable, restore TTL to 15 minutes.
This has two downsides:
- Minimum downtime for a change is 30 seconds.
- We can't do emergency changes with proper state display in less than 15 minutes.
The first one is probably fine. We can resolve the second one by pushing a cache dirty to instances, similar to how sitevars worked.