Postmortem -
Read details
Jul 7, 08:43 EDT
Resolved -
The Inspection App photo upload issue has been resolved.
The issue was related to photo uploads timing out before they could complete, especially when network connectivity was slow or inconsistent. An updated app version was released last week with increased upload timeouts and improved retry handling.
Since the update was deployed, photo upload errors have stopped, missing-photo counts have dropped significantly, and some previously queued photos have successfully uploaded.
IT and the application team will continue monitoring photo upload activity and will add improved logging/visibility to help identify similar issues more quickly in the future.
Jul 6, 11:25 EDT
Identified -
We have identified the likely cause of the Inspection App photo upload issue.
The issue appears to be related to network connectivity during photo uploads, combined with the amount of time the app allowed for uploads to complete. A new app version has been rolled out with changes to improve upload handling and allow more time for photos to upload successfully.
Since the update was deployed, we are no longer seeing the same upload errors, and the number of inspections missing photos is decreasing.
IT and the application team will continue monitoring to confirm the issue remains stable and that photo uploads are completing as expected. We will provide another update once monitoring confirms the issue is fully resolved.
Jul 2, 15:03 EDT
Update -
We are continuing to investigate this issue.
Jul 2, 13:10 EDT
Update -
An updated version of the Inspection App has been created and is now being pushed out to users.
This version includes a change intended to improve photo upload reliability. Users may continue to experience missing or delayed inspection photos while the update is being deployed.
We will continue monitoring after the rollout and will provide another update once we have confirmed the issue is resolved.
Jul 2, 12:05 EDT
Update -
An updated version of the Inspection App has been created and is now being pushed out to users.
This version includes a change intended to improve photo upload reliability. Users may continue to experience missing or delayed inspection photos while the update is being deployed.
We will continue monitoring after the rollout and will provide another update once we have confirmed the issue is resolved.
Jul 2, 12:05 EDT
Investigating -
We are continuing to work on the Inspection App photo upload issue.
Rolling the app back to the previous version did not resolve the issue. The application team is now making an additional change to increase the photo upload timeout and plans to release an updated app version today.
Users may continue to experience missing or delayed inspection photos until the updated version is released and validated. We will provide another update once the new version has been deployed and testing confirms the issue is resolved.
Jul 2, 10:10 EDT
Identified -
We have identified the issue affecting Inspection App photo uploads.
The issue appears related to new functionality introduced in the latest app version. IT and the application team are working to roll back the app to a previous version while we continue validating the impact.
Users may continue to experience missing or delayed inspection photos until the rollback is complete. We will provide another update once the rollback has been completed and confirmed.
Jun 30, 16:56 EDT
Update -
We are investigating an issue where some Inspection App photos are not uploading as expected after inspections are submitted.
Current review does not show a recent spike, but we have confirmed that some inspection records may be saved without the expected photos. IT and the application team are reviewing the app’s photo upload process to better understand the cause and impact.
Users may experience missing or delayed inspection photos while we continue investigating. We will provide another update once we have more information.
Jun 30, 14:35 EDT
Investigating -
We are currently investigating an issue where some Inspection App photos may not be uploading as expected.
IT is reviewing application logs to determine the cause and confirm the full impact. Early investigation suggests the issue may be related to a specific app version, but this has not yet been confirmed.
Users may see missing or delayed inspection photos while we continue investigating. We will provide another update once more information is available.
Jun 30, 08:56 EDT