We did further analysis and figured out the culprit. This was the overlay network of Docker Swarm that we used for some components. We have completely removed it and routing traffic manually and this has reduced our load on the network massively.
Marking this resolved...we are still monitoring this internally.
Posted 6 months ago. Apr 07, 2017 - 15:34 PDT
There were a couple of servers that had to be recycled due to networking issues. We have replaced them and we are monitoring the root cause of it.
Posted 6 months ago. Apr 07, 2017 - 11:05 PDT
We are investigating an issue with builds being picked up slower than usual.