Day 7: When Your Deploy Pipeline Sabotages Your E2E Tests
Spent the morning fixing E2E test failures. Every run was failing with 404s and rate limits. Five failed runs later, I realized the problem wasn't the tests.
The Find
VibeProof health score dropped from 80 to 35.5 when I bumped the file scan limit from 50 to 149. Same codebase. The 80 was a lie.
Also caught 208 auto-fixable issues across the expanded scan. The real number was hiding behind the artificial limit.
The Fix
The E2E failures had multiple causes:
{id}placeholders in request bodies weren't being replaced (only URL paths were)- Every git push triggered a full Docker rebuild and app restart
- GitHub API rate limiting from aggressive log fetching
- Missing retry logic for 429 responses
Fixed the body replacement bug first. Added auto-enrichment for 400 responses - if a POST fails, merge required fields and retry. Added flexible status matching so tests don't fail on edge cases like empty lists returning 200 instead of 404.
The deploy interference took 2 hours to debug. Added path filters to deploy.yml so script changes don't restart the app. A 2-line fix that would have saved the entire morning.
Moved to Azure self-hosted runner (D4s_v3) to eliminate GitHub Actions rate limits. Replaced the flaky Azure Function cron with Logic App. Deleted 681 lines of function code.
The Score
Health: 80 → 35.5 (but 35.5 is honest)
E2E pass rate: 58% after all fixes
The Takeaway
File scan limits create a trust problem. Customers will see the same health score drop when they upgrade plans. The higher number was technically correct for those files, but misleading about overall codebase health.
Also learned that deploy pipelines can be your worst enemy when debugging test failures. Always check if your own tooling is interfering before diving deep into test logic.
The wine was deserved.