Incident Summary

On 11/16/2017 for a period of approximately 15 minutes, the wildcard TLS certificate for *.phacility.com was allowed to expire, resulting in "Connection not secure" warnings for all web users. SSH traffic was not affected.

Timeline

• 1156PST: Support receives first reports of expired certificate.
• 1205PST: Load balancer configuration error identified.
• 1213PST: Configuration error corrected and un-expired certificate deployed to production.

Root Cause

During scheduled certificate rotation earlier in November, a configuration error was made, resulting in the soon-to-expire certificate staying active on a production load balancer. See T11815 for details.

Resolution

All paid Phacility instances received a one hour service credit for the disruption. New operations policies drafted to audit certificate rotations for completeness.