While deploying a permissions change, we corrupted the datastore backing our authorization layer. This caused the system to reject console logins and API key authentication, resulting in all authenticated API requests failing during the incident window. Card network authorizations and underlying transaction processing were not impacted.
Some console/Insights access and API-based interactions were unavailable for approximately 40 minutes. Card transactions continued to authorize and process normally. Engineering has restored the authorization datastore and confirmed that access has been fully recovered and are investigating the number of failed called during the down window.
Resolved
While deploying a permissions change, we corrupted the datastore backing our authorization layer. This caused the system to reject console logins and API key authentication, resulting in all authenticated API requests failing during the incident window. Card network authorizations and underlying transaction processing were not impacted.
Some console/Insights access and API-based interactions were unavailable for approximately 40 minutes. Card transactions continued to authorize and process normally. Engineering has restored the authorization datastore and confirmed that access has been fully recovered and are investigating the number of failed called during the down window.
Investigating
We are currently investigating an issue causing elevated authorization failures across our services. Some users may experience errors when attempting to access resources.