Incident Report: Intermittent Transaction Failures

Date of Incident: 31 July 2025
Timeframe: 06:32 – 12:24 AEST
Service Affected: PayChoice Transaction Processing via Issuer API
Impact Level: Medium
Duration: ~6 hours
Root Cause: Intermittent network connectivity issues at the issuer's data centre

📋 Summary

On the morning of 31 July 2025, PayChoice experienced degraded performance and intermittent transaction failures due to network connectivity issues when interacting with the issuer’s API. The issue originated from within the issuer’s data centre, where intermittent disconnections caused transaction requests to fail.

🕒 Incident Timeline

📉 Impact

• Scope: Intermittent transaction failures for customers transacting via PayChoice through the issuer.
• User Impact: Affected transactions either timed out or returned error messages to end users.

🧠 Root Cause

• Network instability and disconnections between PayChoice and the issuer’s gateway.
• Confirmed to be caused by an issue in the issuer’s data centre, leading to intermittent connectivity and timeouts during API calls.

✅ Resolution

• The issuer resolved the issue within their data centre at 12:24 AEST, restoring stable network communication.
• PayChoice transaction processing returned to normal immediately after.

🔍 Lessons Learned

• Monitoring effectively detected response time degradation early.
• Internal infrastructure checks were comprehensive and eliminated internal causes promptly.
• Initial delays in identifying the issue were due to lack of immediate confirmation from the issuer.

🏁 Conclusion

This incident highlighted the importance of proactive monitoring, rapid triage, and timely escalation. While PayChoice systems responded as designed, further improvements will be made to enhance resilience and visibility in the event of upstream failures.