Final Update: [5:17pm EST] A few of our customers had issues logging in or accessing private sites once the networks returned. This issue has been fully resolved and all networks are currently operational. We definitely apologize for the inconvenience and trouble this has caused.
Update: [3:07pm EST] This issue has been resolved. The problem was related to a bug in the code that runs our databases – a bug that looks to have been present for months, but had not caused issues or triggered any warnings. Our team will continue to monitor for any potential problems and do a complete review to make sure it doesn’t happen again.
Update: [2:22pm EST] Our team has identified the issue and put a fix in place. Most customers will already experience better performance. We’ll continue to monitor and full access to all sites should stabilize shortly. We will keep you updated here.
Initial Report: [1:51pm EST] Our team is currently working to resolve performance issues that are affecting some of our customers. We are aware that some sites are not available.
We are working as quickly as possible to resolve this issue and will update here as soon as we know more. We definitely apologize for this inconvenience.
Update: As of September 19, 2015, this issue has been resolved. Please let us know if you have any questions at all.
Due to problems with a 3rd party vendor we work with to handle DNS for domain mapping, new domains are not able to be mapped at this time.
For CampusPress networks that use domain mapping to map domains (ie. xyz.myschool.edu mapped to blogs.myschool.edu/xyz), we won’t be able to process new domains being added until this issue is resolved.
Our team is working hard to find a solution to this problem. We are optimistic of having a solution in place within the next few days. We will keep this post updated when we have something new to share.
We apologize for the inconvenience this may cause.
9am EST: We’re having temporary performance issues in our USA data center. Our technical team is working hard to restore performance back to normal and we will keep you updated on status here.
9:40am EST: We have been able to improve the issue, related to our database replication system. We will continue to monitor and make adjustments as performance returns back to normal. We certainly apologize for the inconvenience this has caused.
10:30am EST: As we continue to work on our systems, there are occasional and sporadic issues with editing and publishing new posts, as well as speed and timeout issues in the dashboard. Front end visitors are not effected. We’ll continue to make adjustments and work to resolve these outstanding issues.
11:15am EST: Performance and reliability have returned to normal. Our team is monitoring for any future problems, but we expect no further issues.
11:10 pm EST: Performance and reliability have returned to normal. Our team is monitoring for any future problems, but we expect no further issues.
Final Update: 12:41 AM EST June 5 – We are seeing response times return to normal and are continuing to monitor for any further issues. We appreciate your patience and apologize for the inconvenience this has caused.
As part of the response to the day’s earlier attack, a configuration for our test environment was inadvertently left on live servers, then later applied automatically as part of a normally scheduled deployment. We have put in place additional automated checks to prevent this kind of configuration mismatch in the future.
Update: 11:30 PM EST – Performance issues have returned and our team is currently working to resolve as quickly as we can.
Final Update: 5:45pm EST – We are seeing response times return to normal and are continuing to monitor for any further issues. We appreciate your patience and apologize for the inconvenience this has caused. We’ll continue to work to ensure our systems are in place to better handle similar situations in the future.
Update: 5:20pm EST – We’ve identified a significant DDOS attack on our infrastructure and are working hard to block and restore performance to normal levels. Users should already notice improvements to speed and reliability. We’ll make a final update here once we are confident everything is fully operational.
Update: 5:00pm EST – We are currently experiencing unusual performance issues. Visitors to sites may see error messages or slow load times.
Our team is working hard to mitigate the issue. We will update you here once we know more.
Update: Network performance problems have been mitigated and we have no reported or known outages over the last several hours. We’ll continue to monitor our services and infrastructure should anything new arise.
We are currently experiencing ongoing and sporadic network performance issues. Our team is working hard to restore service fully as quickly as we can.
We will keep this space updated with more information as it becomes available and will be reaching out to all Super Administrators of networks we host by email shortly.