Inspection App Photo Upload Issues

Incident Report for PADNOS

Postmortem

Inspection App Photo Upload Postmortem

Summary

Inspection App photos were not consistently uploading after inspections were submitted. Inspection records were saving successfully, but some expected photos were missing.

The issue was caused by photo uploads timing out under slow or inconsistent network conditions. The app did not allow enough time for some uploads to complete, and the failure was not always visible in our API logs because the upload request did not always reach our API.

A new app version was released with increased upload timeouts and improved retry handling. After the update, photo upload errors stopped and missing-photo counts dropped significantly.

Root Cause

Photo uploads were timing out before they could complete.
Slow or inconsistent network connectivity made some uploads take longer than the app allowed.
The app did not provide enough visibility when uploads failed before reaching the API.
The issue was resolved by increasing upload timeouts and improving retry handling in the app.

Impact

Some inspection records were saved without their expected photos.
Some photos were delayed or missing until the updated app version was installed.
API logging did not show all failures because some upload attempts failed before reaching the API.
Some previously missing photos uploaded later because they were still queued locally on the device.

Symptoms

Users reported that Inspection App photos were not appearing after inspections were submitted.
Inspection records were saved successfully, but photos were missing.
API and IIS logs showed inspection save activity, but not the expected follow-up photo upload calls.
Firebase showed photo upload errors related to connection and timeout behavior.
After the updated app version was deployed, errors stopped and missing-photo counts dropped.

Resolution

Released a new app version with increased photo upload timeouts.
Improved retry handling for photo uploads.
Confirmed that new photo upload errors stopped after deployment.
Confirmed that missing-photo counts dropped significantly.
Confirmed that some previously queued photos uploaded successfully after the update.

Follow-Up Actions

Add more detailed application logging using Google Cloud Logs and Firebase.
Improve Firebase logging details around photo capture, upload attempts, retries, failures, and completed uploads.
Continue using the audit report created during the incident to identify transactions missing photos.
Add missing-photo audit logic back into the primary inspection audit fact table.
Add missing-photo visibility to the main KPI dashboards.
Continue monitoring photo upload activity to identify similar issues earlier in the future.

Posted Jul 07, 2026 - 08:43 EDT

Resolved

The Inspection App photo upload issue has been resolved.

The issue was related to photo uploads timing out before they could complete, especially when network connectivity was slow or inconsistent. An updated app version was released last week with increased upload timeouts and improved retry handling.

Since the update was deployed, photo upload errors have stopped, missing-photo counts have dropped significantly, and some previously queued photos have successfully uploaded.

IT and the application team will continue monitoring photo upload activity and will add improved logging/visibility to help identify similar issues more quickly in the future.

Posted Jul 06, 2026 - 11:25 EDT

Identified

We have identified the likely cause of the Inspection App photo upload issue.

The issue appears to be related to network connectivity during photo uploads, combined with the amount of time the app allowed for uploads to complete. A new app version has been rolled out with changes to improve upload handling and allow more time for photos to upload successfully.

Since the update was deployed, we are no longer seeing the same upload errors, and the number of inspections missing photos is decreasing.

IT and the application team will continue monitoring to confirm the issue remains stable and that photo uploads are completing as expected. We will provide another update once monitoring confirms the issue is fully resolved.

Posted Jul 02, 2026 - 15:03 EDT

Update

We are continuing to investigate this issue.

Posted Jul 02, 2026 - 13:10 EDT

Update

An updated version of the Inspection App has been created and is now being pushed out to users.

This version includes a change intended to improve photo upload reliability. Users may continue to experience missing or delayed inspection photos while the update is being deployed.

We will continue monitoring after the rollout and will provide another update once we have confirmed the issue is resolved.

Posted Jul 02, 2026 - 12:05 EDT

Update

Posted Jul 02, 2026 - 12:05 EDT

Investigating

We are continuing to work on the Inspection App photo upload issue.

Rolling the app back to the previous version did not resolve the issue. The application team is now making an additional change to increase the photo upload timeout and plans to release an updated app version today.

Users may continue to experience missing or delayed inspection photos until the updated version is released and validated. We will provide another update once the new version has been deployed and testing confirms the issue is resolved.

Posted Jul 02, 2026 - 10:10 EDT

Identified

We have identified the issue affecting Inspection App photo uploads.

The issue appears related to new functionality introduced in the latest app version. IT and the application team are working to roll back the app to a previous version while we continue validating the impact.

Users may continue to experience missing or delayed inspection photos until the rollback is complete. We will provide another update once the rollback has been completed and confirmed.

Posted Jun 30, 2026 - 16:56 EDT

Update

We are investigating an issue where some Inspection App photos are not uploading as expected after inspections are submitted.

Current review does not show a recent spike, but we have confirmed that some inspection records may be saved without the expected photos. IT and the application team are reviewing the app’s photo upload process to better understand the cause and impact.

Users may experience missing or delayed inspection photos while we continue investigating. We will provide another update once we have more information.

Posted Jun 30, 2026 - 14:35 EDT

Investigating

We are currently investigating an issue where some Inspection App photos may not be uploading as expected.

IT is reviewing application logs to determine the cause and confirm the full impact. Early investigation suggests the issue may be related to a specific app version, but this has not yet been confirmed.

Users may see missing or delayed inspection photos while we continue investigating. We will provide another update once more information is available.

Posted Jun 30, 2026 - 08:56 EDT

This incident affected: RIMAS (Inspection App).