Mail delivery delayed
Incident Report for Apache Software Foundation
Resolved
All mail should be flowing freely
Posted Apr 23, 2019 - 00:54 UTC
Update
Mail has been flowing for about a week now, however some domains still remain throttled or delivery delayed, for instance qq.com. We've escalated to their IT team.
Posted Apr 17, 2019 - 12:55 UTC
Update
Most delays in mail should be mitigated, however the mail queue is still very large. We're continuing to monitor for the time being.
Posted Apr 11, 2019 - 13:47 UTC
Update
Outgoing apache.org mail is known to be delayed to yahoo.com and outlook.com destinations due to IP reputation. We have opened cases with both companies. Due to the size of the queue, incoming mail is also delayed.
Posted Apr 08, 2019 - 20:13 UTC
Update
Mail delivery is still delayed while the queue catches up and our IP builds a good mail reputation. Pretty soon, we'll have enough street cred to take on other mail servers in a battle royale style cage match. This current process can take a few hours and in some cases a day or more.
Posted Apr 08, 2019 - 02:36 UTC
Update
We're continuing to monitor the new mail server. The mail queue is increasing, but that's likely due to a backup running. Current errors are related to CNAME lookups, but they will resolve with time. Yahoo users might see increased bounced/blocked mail. Fluxo's curry was burnt, it was a total loss.
Posted Apr 07, 2019 - 18:55 UTC
Monitoring
Basic functionality has now been restored and we will be monitoring the service as it catches up again. We will assess the situation and incident in the coming days and deliver a postmortem once we have gathered sufficient information.
Posted Apr 07, 2019 - 06:43 UTC
Update
We are currently working on implementing emergency protocols for restoring the mail service. ETA 1-2 hours for service to be restored. Some additional work may be required in the coming days to address issues with auxiliary services.
Posted Apr 07, 2019 - 06:17 UTC
Identified
We have identified the issue and are working to restore services. No ETA as of yet.
Posted Apr 07, 2019 - 02:16 UTC
Investigating
Due to a hardware fault, mail delivery to our mailing lists is currently experiencing delays. We are investigating, and will update this notice as we progress.
Posted Apr 07, 2019 - 00:39 UTC
This incident affected: Internal Services (Infra On Call).