Skip to content

14.6 - Post-Sprint-21 Full E2E Re-Test (2026-07-18)

Sprint-14-style end-to-end re-validation of the whole platform against a fresh live Docker Compose stack, extended to cover the features shipped after Sprint 14 with no acceptance coverage of their own: campaign-service + the order discount/redemption saga (Sprint 21) and the web-bff composition surface (Sprint 16). Also the first-ever full-stack boot since Sprint 17 adopted starter-lock - which is precisely where it found real bugs. The run doubles as live verification of the order-service RestClient timeout commit (c3ee8a1, this branch).

Scope and stack

  • Fresh environment: make infra-destroy (all volumes), all ~15 images rebuilt from this branch, auth+platform+apps profiles up - now 24 containers including the newly compose-wired campaign-service (port 9011) - plus all 11 Debezium outbox connectors registered and RUNNING.
  • Machine gate: Docker VM 31.3 GiB / 16 CPUs (docker info), comfortably above the ~12 GiB floor; the compose MEMORY BUDGET comment's 7.61 GiB ceiling is a different (WSL) machine's constraint.
  • PSP_MOCK_FORCE_OUTCOME=SUCCESS set in infra/docker/.env for deterministic saga polling (same as CI; AC-01's compensation path triggers a deterministic activation failure, not a PSP failure, so this masks nothing).

New, permanent test assets added by this task

Asset What it proves
acceptance-tests/.../campaign/CampaignDiscountedOrderAcceptanceIT ACTIVE percentage campaign discounts a real gateway order (100.00 -> 75.00, campaignId/code recorded); saga fulfills at the discounted total; redemption RESERVED -> CONFIRMED in campaign_db after payment.completed.v1
acceptance-tests/.../campaign/CampaignFailOpenAcceptanceIT Sprint 21 exit criterion live: with telco-campaign-service stopped, the same order succeeds at full price with no campaign recorded; saga still fulfills. Env-gated (CAMPAIGN_FAILOPEN_ENABLED=true) and run in its own invocation to avoid circuit-breaker pollution of sibling tests
acceptance-tests/.../bff/WebBffSmokeAcceptanceIT The four /bff/v1 GET composition endpoints return their documented shapes through the gateway for a real onboarded subscriber; unauthenticated calls 401
support/CampaignAdminApi Campaign fixtures via direct :9011 (documented exception: no gateway route for campaign-service by tech-lead ruling, Feature 21.1.3/ADR-027)
support/CampaignDb Read-only JDBC redemption-status assertion (no redemption read API/outbox event exists - same precedent as AC-03's raw Kafka producer)
infra/docker/compose.yml campaign-service block campaign-service is now part of the standard apps profile stack

Bugs found and fixed by this run (the reason E2E re-tests exist)

  1. All three starter-lock adopters crashlooped on boot in Docker - REDIS_HOST never provided. subscription-service and billing-service (Sprint 17) and campaign-service (Sprint 21) set telco.platform.lock.redis.address: redis://:${REDIS_PASSWORD}@${REDIS_HOST:localhost}:6379, but the compose x-app-env anchor passed only REDIS_PASSWORD - so Redisson resolved localhost:6379 inside the container and failed Spring context refresh (17 restarts each, RedisConnectionException). Never caught before because the full compose stack was never booted after Sprint 17: the lock ITs could not run (repo-wide Testcontainers/Docker API incompatibility) and Sprint 21's live proof ran campaign-service on the host, where localhost is correct. Fix: REDIS_HOST: redis added to the compose x-app-env anchor (mirrors api-gateway's own env). Verified: all three healthy, and the Sprint 17 bill-run lock path executed live for the first time in Docker (runAlreadyOwned:false on a real POST /api/v1/billing/runs).
  2. max_replication_slots=10 overflowed by the 11th Debezium connector. The compose postgres command hardcoded exactly the pre-campaign connector count; usage-outbox-connector's slot creation failed once campaign-outbox-connector took the 10th slot. Fix: max_wal_senders/max_replication_slots raised to 16 (headroom documented in the compose comment); postgres recreated (volume kept), connector task restarted, 11/11 RUNNING.

Backend acceptance results

Command: mvn -f microservices/pom.xml -pl acceptance-tests -am -Pacceptance verify

Scenario Result
AC-01 new-subscriber onboarding (happy path) PASS (see note)
AC-01 activation-failure compensation PASS
AC-02 monthly invoice bill-run PASS
AC-03 quota exhaustion and overage PASS
Campaign discounted order + redemption CONFIRMED (new) PASS (3.4s)
web-bff composition smoke, 5 tests (new) PASS
Campaign fail-open, separate invocation (new) PASS (47.5s incl. container stop/boot cycle)

Note: in the first sweep AC-01 failed its very first Keycloak token fetch with a 400 - root-caused via Keycloak logs to a stale pooled DB connection killed by this session's own postgres recreation for fix 2 above (FATAL: terminating connection due to administrator command at the exact timestamp), i.e. collateral of the in-flight infra fix, not an application or suite defect. Verified by an immediate solo re-run: PASS in 6.2s. One test-harness hardening applied along the way: the fail-open IT's teardown health poll needed Awaitility .ignoreExceptions() (during container boot the docker proxy has the port bound but nothing listening, so the poll sees Connection reset - a SocketException, which untilAsserted would otherwise propagate instead of retrying).

Browser E2E (Sprint 16 journey, re-proven)

SvelteKit dev server on host (:3000), Chrome driven via DevTools automation, evidence screenshots in e2e-retest-evidence/. Subscriber provisioned via POST /api/v1/users (the linkable path, Feature 14.4).

  1. PKCE login (S256 challenge visible in the authorize redirect) - first-attempt success, the 14.4 createUser fix holding.
  2. Onboarding wizard: register (TCKN checksum validated), KYC PDF upload, plan selection from the real catalog, review, place order.
  3. Saga observed to completion in the UI: "Your subscription is active - status FULFILLED".
  4. Dashboard/account composed via web-bff: real MSISDN (905320000007, fresh pool), provisioned quota (0/1,000 MB data / min / SMS), profile PENDING (expected - admin KYC approval is a separate back-office step).
  5. Admin bill run issued 8 invoices platform-wide; the invoices page showed exactly this user's 1 (self-scoping), TRY 35.28 ISSUED; PDF download returned 200 through the gateway (GET /api/v1/invoices/{id}/pdf).

Sprint 16's three known open defects (addons POST 500, duplicate-TCKN 500-not-409, oversized multipart 500-not-413) are off this happy path and remain open - unchanged by this run.

Performance re-validation (NFR-01)

Command: k6 run microservices/acceptance-tests/perf/api-latency-load-test.js (host, 30 VUs, pooled loadtest identities from the fresh realm import), after a thermal cool-down with nothing else running.

Both in-script thresholds passed (2m sustained_mixed_load, 30 VUs, 14,512 requests at 117 req/s):

Metric Result Target
http_req_duration p95 (overall) 63.94ms <1000ms
http_req_duration{expected_response:true} p95 (genuinely served) 99.17ms <300ms - PASS

Comparison: the 14.3.1 steady-state baseline measured 193.5/198.5ms on the original (WSL, 7.6 GiB VM) machine; this run's 99.17ms on a 31 GiB/16-core Docker VM is comfortably better - no latency regression from anything shipped since.

Two expected artifacts, both consistent with the 14.3.1 report's own caveats and neither a defect: (1) 57% of requests were 429s from the gateway's 100 req/min per-identity rate limit (NFR-18 behaving as designed) - this machine completes iterations roughly twice as fast as the baseline machine, so the identical script pushes each pooled identity past the limit; the script's methodology anticipates this by asserting latency only on served responses. (2) The two ADMIN-gated reads (orders-by-customer, subscriptions-by-customer, ~3% served) still share the single admin@telco.local identity per the pre-existing caveat carried in the 14.3.1 report.

Verdict

PASS across all four layers.

  • Backend acceptance: 7/7 scenarios green (4 original + 3 new), including the first-ever executable E2E proof of the Sprint 21 campaign discount + redemption saga and the Sprint 21 fail-open exit criterion against the real containerized stack.
  • Browser: the complete Sprint 16 subscriber journey re-proven end to end, first-attempt PKCE login through invoice PDF download, with invoice self-scoping re-verified (1 of 8 visible).
  • Performance: NFR-01 re-validated with wide margin (p95 99.17ms vs 300ms budget).
  • Two genuine, previously-invisible infra bugs found and fixed (starter-lock REDIS_HOST gap; replication-slot ceiling) - both of the "works everywhere except the real composed environment" class this re-test exists to catch. The fixes live in infra/docker/compose.yml and are permanent.

Residual known gaps, all pre-existing and tracked elsewhere: Sprint 16's three open defects (addons POST 500, duplicate-TCKN 409, multipart 413), the deferred Kubernetes full-13-service in-cluster boot (Sprint 15/18/19 tail), and the repo-wide Testcontainers/Docker API incompatibility.

Live verification of c3ee8a1 (order-service RestClient timeouts)

Every order placed in this run (acceptance sweep, fail-open outage window, browser journey) exercised order-service's outbound RestClients under the new 2s connect/read bounds; the fail-open IT additionally proved the bounded-timeout path under a real connection failure. No regression, no hung calls observed.