Our provider has confirmed they have successfully updated the switches. Since then, we haven't seen any anomalous behavior and are considering this issue resolved.
Posted Mar 03, 2021 - 08:31 EST
Update
Our network provider is performing emergency maintenance to update the software on their pod switches tonight between midnight and 2am ET. They inform us that they anticipate no downtime during this. We will continue monitoring overnight and provide an update when we have additional information.
At this time, we are seeing reasonable stability in all our systems.
Posted Mar 02, 2021 - 22:22 EST
Monitoring
We haven't seen any additional network issues since we posted this incident. We are continuing to monitor and will update this incident as the day goes on.
Posted Mar 02, 2021 - 17:47 EST
Identified
Our network provider is reporting intermittent packet loss which is affecting some of our app servers. We understand they have identified the source of the issue to a bug on a redundant set of switches in the datacenter.
They are working on implementing emergency maintenance.
At this time, we are seeing full service without problems - but we are continuing to monitor for short-term drops in service and will provide updates as we learn more.