Skip to content

14.5 — Avro Schema Governance Reconciliation (tech-lead ruling)

Status: RULED, phases 1-8 of 8 complete. code-review's ADR-019-compliance pass (below) is DONE, APPROVE WITH ONE FOLLOW-UP (non-blocking); tech-lead's final sign-off remains. Logged 2026-07-07. Owning ADR: architecture/adr/ADR-019-event-contract-and-schema-governance.md (see its "Amendment (2026-07-07)" section, which this document expands on with the full evidence and execution plan).

How this was found

While auditing event-contract coverage as a follow-up to 14.1.2 (contract tests), it became clear the canonical schema directory (platform/platform-event-contracts/src/main/avro/) had never been checked against what services actually publish. Every outboxService.publish(<aggregate>, <id>, "<event>.v1", <payload>) call site under microservices/**/src/main/java was cross-referenced against the .avsc files in the canonical directory, against each service's own src/test/resources/avro/ snapshots (used by that service's contract test, e.g. NotificationEventContractTest), and against docs/architecture/event-catalog.md's event registry.

Ruling on direction — rejected alternative, and why

Rejected: force every service onto Avro binary wire encoding to "truly" comply with ADR-019. This would be a large, high-risk migration (every producer/consumer, the outbox writer, Debezium's JSON converter config) to chase a form of compliance the ADR never actually required. ADR-009's outbox-as- plain-JSON decision is established, working, and separately verified this session (Feature 14.5 does not reopen it). The real defect is not "wrong wire format" — it is "no tooling ever proved the schema file matches the real payload." Fixing that is additive and low-risk; re-encoding the wire format is neither.

Ruled correct: reconcile schemas to the real JSON shape, close the missing-schema gap, fix the naming violation, and add the tooling that prevents this drift from recurring. Full rationale in the ADR-019 amendment (architecture/adr/ADR-019-event-contract-and-schema-governance.md, "Amendment (2026-07-07)", points A1-A5). This document is the durable execution reference for the phases that amendment presupposes.

Verified finding — 14 real event types, not 13, currently unregistered

Method: grep -rhoE '"[a-z]+\.[a-z-]+\.v[0-9]+"' microservices --include="*.java" for every literal event-type string, filtered to call sites outside /test/, cross-referenced against platform/platform-event-contracts/src/main/avro/*.avsc (18 files present today, including EventEnvelope.avsc which is the envelope wrapper, not an event) and against each service's src/test/resources/avro/*.avsc snapshots.

Correction note: this ruling was first summarized verbally as "13 missing schemas." Re-verifying against the actual codebase for this document turned up a 14th — user.deleted.v1 (identity-service), sibling to user.created.v1, same root cause, previously missed. Per the "verify before done" rule, the count below is the corrected, evidenced one; treat 14, not 13, as authoritative.

# Event Producer Test-local .avsc snapshot already exists? Canonical schema exists?
1 order.cancelled.v1 order-service yes — order-service/src/test/resources/avro/order-cancelled.avsc NO
2 payment.failed.v1 payment-service yes — payment-service/src/test/resources/avro/payment-failed.avsc NO
3 payment.refunded.v1 payment-service yes — payment-service/src/test/resources/avro/payment-refunded.avsc NO
4 tariff.created.v1 product-catalog-service yes — product-catalog-service/src/test/resources/avro/tariff-created.avsc NO
5 tariff.price-changed.v1 product-catalog-service yes — product-catalog-service/src/test/resources/avro/tariff-price-changed.avsc NO
6 ticket.opened.v1 ticket-service yes — ticket-service/src/test/resources/avro/ticket-opened.avsc NO
7 ticket.assigned.v1 ticket-service yes — ticket-service/src/test/resources/avro/ticket-assigned.avsc NO
8 ticket.resolved.v1 ticket-service yes — ticket-service/src/test/resources/avro/ticket-resolved.avsc NO
9 ticket.sla-breached.v1 ticket-service yes — ticket-service/src/test/resources/avro/ticket-sla-breached.avsc NO
10 invoice.paid.v1 billing-service yes — billing-service/src/test/resources/avro/invoice-paid.avsc NO
11 invoice.overdue.v1 billing-service yes — billing-service/src/test/resources/avro/invoice-overdue.avsc NO
12 notification.dispatched.v1 notification-service yes — notification-service/src/test/resources/avro/notification-dispatched.avsc NO
13 user.created.v1 identity-service no — no avro test-resources directory exists for identity-service at all NO
14 user.deleted.v1 identity-service no — same NO

Additional, independently confirmed observations (not new missing schemas, but part of the same reconciliation):

  • docs/architecture/event-catalog.md's event registry never lists user.created.v1 or user.deleted.v1 at all — these two are undocumented in the catalog, not just unregistered as schemas. Phase 7 (qa) must add both rows.
  • order.confirmed.v1 is correctly excluded from this list — the catalog itself documents it as "deferred; not produced in the MVP," confirmed by no matching outboxService.publish call site.
  • customer.profile-updated.v1 and subscription.cancelled.v1 appear only inside notification-service's own test fixtures (DomainEventNotificationConsumerTest), never in any producing service's main source. These are NOT real, currently-produced events; do not add canonical schemas for them as part of this feature. If they are meant to exist, that is a separate, new-feature decision (test fixture referencing a not-yet-built event), out of scope here — flag to qa to either align the fixture to a real event name or delete the dead scenario, but do not silently invent a schema to match a test's fiction.
  • demoitem.created.v1 (reference-service) is the ADR-017 template/reference artifact's own demo event, not a real service. No canonical schema is required for it under this feature; leave as is.
  • cdr.recorded.v1 already has a canonical schema (cdr-recorded.avsc) — correction (architecture, phase 2 re-verification): this claim was wrong. The real wire shape (usage-service's CdrRecordedEventConsumer.CdrPayload, matching the only real producer, acceptance-tests' CdrEventProducer) types occurredAt as an ISO-8601 string, but the canonical schema types it as {"type":"long","logicalType":"timestamp-millis"}. This is genuine, previously-uncaught drift requiring reconciliation — see the Architecture Diff Spec below, section A row 14. Treat as action-required, not "no action needed."

Naming-convention fix

EventEnvelope.avsc violates the kebab-case-filename convention this ADR now states explicitly (see ADR-019 amendment, point A5). Rename to event-envelope.avsc. The Avro record's own "name" field stays EventEnvelope (PascalCase, drives Java class generation) — only the filename changes. The avro-maven-plugin configuration in platform/platform-event-contracts/pom.xml has a string-replacement property, <event.envelope.v1>${project.basedir}/src/main/avro/EventEnvelope.avsc </event.envelope.v1> (around line 77), that references the file by its literal old path — this MUST be updated in the same commit as the rename, or the Avro codegen step breaks.

Execution order (8 phases) — owners and exit criteria

  1. tech-lead (this phase — DONE). ADR-019 amendment (architecture/adr/ADR-019-event-contract- and-schema-governance.md) plus this tracking document. Exit criteria: amendment merged, unambiguous on JSON-shape-not-wire-bytes, ADR-009 unchanged, .avsc files remain canonical source of truth, naming convention stated, this document exists as the durable reference. Sprint tracking (docs/tasks/STATUS.md, sprint-14 README.md) updated to IN PROGRESS.
  2. architecture. Validate the reconciliation strategy per event before any file is touched: for each of the 12 rows with an existing test-local .avsc snapshot, confirm that snapshot itself is still an accurate description of the producing service's real, current event-payload class (do not assume the test-local copy is trustworthy just because it exists — it was never cross-checked either). For user.created.v1/user.deleted.v1 (no snapshot at all), review and approve the new schema's field list, types, and nullability against identity-service's real UserCreatedV1/DeleteUserCommandHandler payload before it is authored. Sign off on the EventEnvelope.avsc -> event-envelope.avsc rename plan (file rename + pom.xml property update, no record-name change). Exit criteria: written sign-off (PR review or an addendum to this file) on all 14 target shapes and the rename plan.
  3. event-integration. For the 12 rows with a test-local snapshot: promote/reconcile each into platform/platform-event-contracts/src/main/avro/ under the correct kebab-case canonical filename (diff against the real Java payload class first, per architecture's phase-2 sign-off — do not copy-paste the test-local file blindly). For user.created.v1/user.deleted.v1: author new canonical schemas from scratch. Register all in Schema Registry per this ADR's existing Schema Registry Rule and Compatibility Rule (backward-compatible; these are net-new registrations, not breaking changes to existing consumers). Add avro-maven-plugin string-replacement properties for all 14 new schemas in platform/platform-event-contracts/pom.xml, mirroring the existing entries. Exit criteria: 14 new canonical .avsc files present, all registered, mvn generate-sources on platform-event-contracts succeeds.
  4. event-integration. Rename EventEnvelope.avsc -> event-envelope.avsc; update the corresponding property in platform/platform-event-contracts/pom.xml. Exit criteria: rename committed, mvn generate-sources still succeeds, generated EventEnvelope Java class name unchanged.
  5. domain-engineer (one pass per producing service: order-service, payment-service, product-catalog-service, ticket-service, billing-service, notification-service, identity-service). For each of the 14 events, confirm the service's real outbox-published JSON payload matches the new canonical schema field-for-field (name, type, nullability). Given the root cause (schemas and real payloads were never cross-checked), expect to find last-mile drift; fix it in the service's event-payload class to match the schema unless the ADR's evolution rules require the opposite (schema is wrong, payload is right) — escalate any such conflict to architecture/tech-lead rather than deciding unilaterally which side is authoritative. Exit criteria: no field-shape mismatch remains for any of the 14 events.
  6. event-integration (with qa support). Extend the schema-compatibility test tooling so every one of all 32 canonical schemas (18 pre-existing + 14 new) has a matching per-service contract test following the existing *EventContractTest pattern (e.g. NotificationEventContractTest) — this is the concrete tooling gap that let the drift happen silently in the first place; it must not be a one-time manual reconciliation. Exit criteria: a producing service's payload class and its canonical schema diverging fails the build (prove it with one deliberately-broken test run, then revert).
  7. qa. Run the full contract-test suite plus the acceptance suite (AC-01/02/03) to confirm no regression. Add the two missing rows (user.created.v1, user.deleted.v1) to docs/architecture/event-catalog.md's event registry table, and add a Schema Evolution Log entry documenting all 14 additions with today's date. Exit criteria: green contract-test and acceptance runs; event-catalog.md fully reflects the 32-event registered set.
  8. devops + code-review, then tech-lead sign-off. devops confirms the Schema Registry compatibility-mode CI check runs against all 32 canonical schemas and fails the build on an unregistered or incompatible event type going forward (closing the "never registered" failure mode for good, not just for these 14). code-review performs a final ADR-019-compliance pass (naming convention, JSON-shape framing, no accidental wire-format change) before merge. tech-lead gives final sign-off closing Feature 14.5 as DONE.

Non-goals (explicit, do not implicitly expand scope)

  • No change to the outbox's plain-JSON publishing (ADR-009 stands).
  • No change to any event's actual field shape beyond what phase 5 finds as genuine drift — this is a reconciliation of documentation/registration, not a redesign of any event.
  • No schema for customer.profile-updated.v1 / subscription.cancelled.v1 (not real, currently- produced events) or demoitem.created.v1 (template artifact) under this feature.
  • No change to order.confirmed.v1's deferred status.

Architecture Diff Spec (2026-07-07)

Author: architecture agent, phase 2 of the 8-phase execution order. Every row below was produced by reading the real, currently-shipping Java event/payload class (or Map.of(...) outbox call) for each event, not by trusting this document's earlier table or any test-local .avsc snapshot at face value. Where a test-local snapshot was already accurate it is noted as corroboration, not as the source of truth.

Two corrections to this document's earlier phase-1 content, found during phase-2 re-verification (per "verify before done"): (1) the phase-1 table above already reflects the corrected 14-item missing- schema list (user.created.v1/user.deleted.v1, not order-item) — order-item is a separate, 15th artifact: a nested Avro record embedded in order-created.v1, not an independent Kafka-registered subject. (2) cdr.recorded.v1's "needs no action" claim (originally line 81, corrected above) was wrong — real drift found, see section A row 14.

No single "platform timestamp convention" exists in the real code — do not normalize globally. Two conventions genuinely coexist by service: ISO-8601 string (order, payment, usage, product-catalog, ticket, identity-service) and epoch-millis long (customer, subscription, billing-service). Reconciliation matches each producer's own real type. This is a hygiene note for a future ADR, not something changed under this feature.

A. Reconciliation of the 18 pre-existing canonical schemas

# Canonical file Verdict Diff
1 order-created.avsc RECONCILE (major) Real: OrderCreatedEvent (orderId, customerId, items: List<OrderItemPayload>, totalAmount, idempotencyKey, occurredAt). Corroborated by order-service's own accurate test snapshot. Changes: (1) add items: {"type":"array","items":"OrderItemPayload"}; (2) add idempotencyKey: string; (3) remove currency (not in real class); (4) rename+retype createdAt (long,timestamp-millis) -> occurredAt (string).
2 payment-completed.avsc RECONCILE Real: PaymentCompletedEvent (paymentId, orderId, customerId, amount, invoiceId, occurredAt). Changes: (1) remove currency; (2) retype orderId from nullable to plain string (always set, @NotNull at the command); (3) rename+retype completedAt(long) -> occurredAt(string); (4) keep invoiceId nullable (confirmed genuinely nullable).
3 invoice-generated.avsc NO CHANGE Verified against BillRunBatchProcessor's record: exact field-for-field, type-for-type match (billing-service genuinely uses epoch-millis).
4 customer-registered.avsc NO CHANGE Matches CustomerRegisteredV1 exactly.
5 customer-updated.avsc NO CHANGE Matches CustomerUpdatedV1 exactly.
6 customer-kyc-approved.avsc NO CHANGE Matches CustomerKycApprovedV1 exactly.
7 customer-kyc-rejected.avsc NO CHANGE Matches CustomerKycRejectedV1 exactly.
8 msisdn-allocated.avsc NO CHANGE Matches MsisdnAllocatedV1 exactly.
9 msisdn-released.avsc NO CHANGE Matches MsisdnReleasedV1 exactly.
10 subscription-activated.avsc NO CHANGE Matches SubscriptionActivatedV1 exactly; both producers construct the same record type, no cross-producer drift possible.
11 subscription-activation-failed.avsc NO CHANGE Matches SubscriptionActivationFailedV1 exactly.
12 subscription-suspended.avsc NO CHANGE Matches SubscriptionSuspendedV1 exactly; both producers use the same record type.
13 subscription-terminated.avsc NO CHANGE Matches SubscriptionTerminatedV1 exactly.
14 cdr-recorded.avsc RECONCILE — corrects this doc's earlier "no action needed" claim Consumed external contract (produced by acceptance-tests' CdrEventProducer, consumed by usage-service's CdrRecordedEventConsumer.CdrPayload). Real shape: subscriptionId, type, quantity(long), occurredAt(String, ISO-8601), cdrRef. Change: retype occurredAt from long timestamp-millis to plain string. Non-blocking note: type is a closed Avro enum today; real payload types it as plain String (consumer defensively .toUpperCase()s) — under JSON-shape governance this distinction doesn't matter on the wire; left to event-integration's discretion.
15 usage-aggregated.avsc RECONCILE Real: UsageAggregatedEvent (subscriptionId, periodStart(String), periodEnd(String), voiceOverageSeconds(long), smsOverageCount(long), dataOverageKb(long), aggregatedAt(String)) — corroborated independently by billing-service's own consumer DTO. Change: retype periodStart, periodEnd, aggregatedAt from long timestamp-millis to string.
16 usage-recorded.avsc RECONCILE Real: UsageRecordedEvent (usageRecordId, subscriptionId, type(String), quantity(long), overage(boolean), recordedAt(String)). Change: retype recordedAt from long to string.
17 quota-exceeded.avsc RECONCILE Real: QuotaExceededEvent (subscriptionId, quotaId, usageType, customerId(nullable), exceededAt(String)). Change: retype exceededAt from long to string.
18 quota-threshold-reached.avsc RECONCILE Real: QuotaThresholdReachedEvent (subscriptionId, quotaId, usageType, customerId(nullable), reachedAt(String)). Change: retype reachedAt from long to string.

B. New nested type required by row 1 (not a top-level Kafka subject)

order-item.avsc — record name OrderItemPayload, namespace com.telco.platform.events.order. Fields (from OrderCreatedEvent.OrderItemPayload, corroborated by order-service's own src/test/resources/avro/order-item.avsc): tariffId(string), tariffName(string), unitPrice(bytes/decimal precision=19 scale=4), quantity(int). Referenced only from order-created.avsc's items field. order-cancelled.v1 does NOT carry items — do not reference it there. Embedded/nested per tech-lead's ruling: no independent Schema Registry subject, no independent contract-test row beyond what already covers order-created.v1.

C. Field specs for the 14 missing schemas

All new files go under platform/platform-event-contracts/src/main/avro/, kebab-case filenames.

# Event File Fields Source of truth
1 order.cancelled.v1 order-cancelled.avsc orderId: string; customerId: string; reason: ["null","string"] default null; occurredAt: string OrderCancelledEvent. Correction to the existing test-local snapshot: both it and order-service's own field-name-only contract test type reason as non-nullable, but neither CancelOrderCommand.reason nor CompensateOrderCommand.reason carries @NotNull/@NotBlank (handler does command.reason() == null ? null : ...) — genuinely nullable at runtime. Do not copy the test-local file verbatim; apply this nullability fix when promoting.
2 payment.failed.v1 payment-failed.avsc paymentId: string; orderId: string; customerId: string; amount: decimal(19,4); invoiceId: ["null","string"] default null; reason: string; occurredAt: string PaymentFailedEvent. Test-local snapshot verified accurate — promote as-is.
3 payment.refunded.v1 payment-refunded.avsc paymentId: string; orderId: string; customerId: string; amount: decimal(19,4); reason: string (non-nullable,@NotBlank); occurredAt: string PaymentRefundedEvent. Test-local snapshot verified accurate — promote as-is.
4 invoice.paid.v1 invoice-paid.avsc invoiceId: string; customerId: string; paidAt: long, timestamp-millis Package-private InvoicePaidEvent in MarkInvoicePaidCommandHandler. Test-local snapshot verified accurate — promote as-is.
5 invoice.overdue.v1 invoice-overdue.avsc invoiceId: string; customerId: string; dueDate: string (ISO-8601 date); detectedAt: long, timestamp-millis Package-private InvoiceOverdueEvent in MarkInvoicesOverdueCommandHandler. Test-local snapshot verified accurate — promote as-is.
6 notification.dispatched.v1 notification-dispatched.avsc notificationId: string; userId: string; channel: string; templateCode: string NotificationService.dispatch(...). Test-local snapshot verified accurate — promote as-is.
7 ticket.opened.v1 ticket-opened.avsc ticketId: string; customerId: string; category: string; priority: string; subject: string; assignedTeam: string; slaDueAt: string (ISO-8601) OpenTicketCommandHandler. Test-local snapshot verified accurate — promote as-is.
8 ticket.assigned.v1 ticket-assigned.avsc ticketId: string; customerId: string; assignedTeam: string Two call sites (OpenTicketCommandHandler, AssignTicketCommandHandler), identical shape — no cross-producer drift. Test-local snapshot verified accurate — promote as-is.
9 ticket.resolved.v1 ticket-resolved.avsc ticketId: string; customerId: string; resolvedAt: string (ISO-8601) ResolveTicketCommandHandler. Test-local snapshot verified accurate — promote as-is.
10 ticket.sla-breached.v1 ticket-sla-breached.avsc ticketId: string; customerId: string; category: string; priority: string; slaDueAt: string (ISO-8601) DetectSlaBreachCommandHandler. Test-local snapshot verified accurate — promote as-is.
11 tariff.created.v1 tariff-created.avsc tariffId: string; code: string; name: string; type: string; monthlyFee: decimal(19,4); currency: string; effectiveFrom: string (ISO-8601); createdAt: string (ISO-8601) TariffCreatedEvent. Test-local snapshot verified accurate — promote as-is.
12 tariff.price-changed.v1 tariff-price-changed.avsc tariffId: string; code: string; oldMonthlyFee: decimal(19,4); newMonthlyFee: decimal(19,4); currency: string; newVersion: int; changedAt: string (ISO-8601) TariffPriceChangedEvent. Test-local snapshot verified accurate — promote as-is.
13 user.created.v1 user-created.avsc userId: string; username: string; email: string; createdAt: string (ISO-8601) UserCreatedV1. No test-local snapshot existed (identity-service has no src/test/resources/avro/ at all) — authored fresh.
14 user.deleted.v1 user-deleted.avsc userId: string; deletedAt: string (ISO-8601) UserDeletedV1. Authored fresh, same reasoning as row 13.

Namespace for rows 13-14: com.telco.platform.events.identity (event-integration's naming call, no existing identity-service convention to match).

D. Cross-service compatibility check — no disagreement found, nothing to escalate

Checked every case where more than one service produces the same event type or models a shared nested type: OrderItemPayload (order-service only; subscription-service's OrderItemClientResponse is an unrelated REST-client DTO, out of scope); subscription.activated.v1/subscription.suspended.v1/ order.cancelled.v1/ticket.assigned.v1 (each has two producer call sites, all construct the identical Java record/Map shape — compiler-enforced, no drift possible); payment.completed.v1 (consumed independently by order-service and subscription-service, their local consumer DTOs are field-for-field identical); billing-service's per-event consumer Payload records correctly type subscription timestamps as Long and usage timestamps as String, matching each producer's real convention — independent corroboration of section A rows 15-18, not a conflict. No case requires tech-lead escalation.

Aside, not a schema-governance defect, flagged for qa/domain-engineer awareness only, not actioned here: DomainEventNotificationConsumer.onSubscriptionEvent reads a customerName field off subscription.activated.v1 payloads via payload.getOrDefault("customerName", customerId), but no producer ever sets customerName. The getOrDefault fallback means this never breaks at runtime (silently always uses customerId) — not an Avro contract violation, out of scope for this feature.

E. Naming-convention normalization for the 14 promoted/authored schemas

The 18 pre-existing canonical schemas all use Avro record name of the form XxxV1. The 12 test-local snapshots being promoted use each service's own class-name convention (XxxEvent in most cases). Recommendation (non-blocking, event-integration may override with reasoning): normalize the Avro record name to XxxV1 for all 14 promoted/authored schemas, for consistency with the other 18. This is registry/codegen naming only — does not change the JSON wire shape, does not require any Java-side rename.

F. Re-confirmation of the "explicitly excluded" list — verified independently

Re-ran the full event-type grep unfiltered by directory and cross-checked every hit: customer.profile- updated.v1 and subscription.cancelled.v1 appear only in DomainEventNotificationConsumerTest, zero occurrences in any src/main/java — confirmed test fiction, correctly excluded. demoitem.created.v1 appears only in reference-service's own main+test source — confirmed template artifact, correctly excluded. order.confirmed.v1 has zero occurrences anywhere (main or test) — confirmed correctly deferred. All three exclusions stand; nothing was silently missed.

Summary of what phase 3 (event-integration) must do: reconcile the 7 real-diff rows in section A (order-created, payment-completed, cdr-recorded, usage-aggregated, usage-recorded, quota-exceeded, quota-threshold-reached), leave the other 11 of the 18 untouched, author the 14 new files in section C plus the nested order-item.avsc in section B, apply the naming normalization in section E, and execute the EventEnvelope.avsc -> event-envelope.avsc rename plus its pom.xml property update.

Phases 3 and 4 execution log (event-integration, 2026-07-07) - DONE, with two findings

Before writing any file, re-verified a sample of the diff spec's rows against the real Java source (per "verify before done"): OrderCreatedEvent/OrderCancelledEvent (order-service), PaymentCompletedEvent/PaymentFailedEvent/PaymentRefundedEvent (payment-service), CancelOrderCommand/CompensateOrderCommand (confirming reason really carries no @NotNull/@NotBlank), UserCreatedV1/UserDeletedV1 (identity-service), UsageAggregatedEvent and CdrRecordedEventConsumer.CdrPayload (usage-service), TariffCreatedEvent/TariffPriceChangedEvent (product-catalog-service), NotificationService.dispatch(...) and the four ticket-service handlers. All matched the diff spec's stated shapes exactly, with one exception below.

Finding 1 - diff spec under-stated row 2's change list (fixed, not just applied verbatim). Section A row 2 (payment-completed.avsc)'s "Real:" field list states PaymentCompletedEvent carries customerId, but the row's four enumerated change bullets never say "add customerId" - and the pre-existing canonical schema had no customerId field at all. Re-reading PaymentCompletedEvent.java directly confirmed customerId is a real field (paymentId, orderId, customerId, amount, invoiceId, occurredAt). Added customerId: string to payment-completed.avsc in addition to the four explicitly-listed changes, since the reconciliation's entire purpose is matching the real payload, not just the bullet list. Flagging this the same way tech-lead/architecture flagged each other's gaps this session - re-verification discipline continued, not broken.

Finding 2 - Confluent Schema Registry cannot validate order.created.v1 standalone; the nested-type plan (section B) works for Java codegen but not for live registry validation without either inlining or an independent subject. A local Schema Registry container was already running (telco-schema-registry, confirmed via docker ps), so beyond the required offline mvn generate-sources -Dschema.registry.skip=true check (green, see below), the live-registry path was also tried per this phase's optional instruction. Result: 32 of the 33 real subjects (everything except order.created.v1) register cleanly against the live, empty registry (proven via direct HTTP POST to /subjects/{subject}/versions for each, then deleted - the registry was left exactly as found, empty). order.created.v1 fails to parse (SchemaParseException: Undefined name: "OrderItemPayload") because kafka-schema-registry-maven-plugin's test-compatibility/register goals - and the real Schema Registry HTTP API itself - parse each subject's raw .avsc text standalone, with no visibility into avro-maven-plugin's own <imports> mechanism (which is what makes the split-file order-created.avsc + order-item.avsc pair work for Java codegen, confirmed in the build log below). Confirmed by direct experiment: registering OrderItemPayload under its own throwaway subject and referencing it from order-created.avsc via Confluent's references field registers successfully - but that requires OrderItemPayload to exist as its own Schema Registry subject, which is exactly what this ruling's section B explicitly forbids ("no independent Schema Registry subject"). Given that direct conflict, this was not resolved unilaterally (inlining the nested record would also require either duplicating it out of order-item.avsc or restructuring how avro-maven-plugin sees it, and either path is itself a schema-design call, not a mechanical one). Leaving the schema exactly as section B specifies (bare-name reference in order-created.avsc, full definition in order-item.avsc, no independent subject) and escalating this as a real, unresolved gap for architecture/tech-lead: whichever way it is decided (inline the nested type into order-created.avsc, or accept a reference-only "non-Kafka" registry subject for OrderItemPayload, or something else), it needs a ruling before phase 8 (devops's live CI compatibility gate) can actually enforce order.created.v1 against a real registry. Per this task's own scope, the required exit criterion - mvn generate-sources succeeding offline - is unaffected and green; this is a live-registry-only gap.

Files changed: - Reconciled (section A): order-created.avsc, payment-completed.avsc (plus the customerId fix above), cdr-recorded.avsc, usage-aggregated.avsc, usage-recorded.avsc, quota-exceeded.avsc, quota-threshold-reached.avsc. - Authored (section B): order-item.avsc. - Authored (section C, all 14, XxxV1 naming per section E): order-cancelled.avsc, payment-failed.avsc, payment-refunded.avsc, invoice-paid.avsc, invoice-overdue.avsc, notification-dispatched.avsc, ticket-opened.avsc, ticket-assigned.avsc, ticket-resolved.avsc, ticket-sla-breached.avsc, tariff-created.avsc, tariff-price-changed.avsc, user-created.avsc, user-deleted.avsc. invoice-paid.avsc/invoice-overdue.avsc use the com.telco.platform.events.invoice namespace, matching the existing invoice-generated.avsc convention (domain-based, not producing-service-based), not the events.billing guess a literal service-name mapping would suggest. - Renamed (phase 4): EventEnvelope.avsc -> event-envelope.avsc (record name EventEnvelope unchanged). - platform/platform-event-contracts/pom.xml: event.envelope.v1 property repointed to event-envelope.avsc; added an <imports> entry for order-item.avsc to the avro-maven-plugin config (required for cross-file type resolution at Java-codegen time); added 14 new <subjects> entries mirroring the existing format. order-item.avsc is deliberately absent from <subjects> - not an independent Kafka-registered event, per the ruling.

Build verification: cd platform/platform-event-contracts && mvn generate-sources -Dschema.registry.skip=true - BUILD SUCCESS, no errors. Generated target/generated-sources/avro/ contains 35 .java files: all 32 canonical event classes (18 reconciled + 14 new), the renamed EventEnvelope class (unaffected by the filename change, still named EventEnvelope), the nested OrderItemPayload class, and the pre-existing CdrType nested enum. Live Schema Registry check (optional per this phase): 32/33 real subjects register cleanly; order.created.v1 does not, per Finding 2 above - escalated, not silently worked around.

Phases 3 and 4 (event-integration) are DONE, with Finding 2 flagged as an open item for architecture/tech-lead before phase 8's live CI gate can cover order.created.v1. Phase 5 (domain-engineer, per-service last-mile drift reconciliation) is next.

Tech-lead ruling: order.created.v1 / OrderItemPayload registry conflict (2026-07-07) — RESOLVED, unblocks phase 5

Decision: Option 1, in its strict form. OrderItemPayload is inlined as a fully self-contained nested record definition inside order-created.avsc. order-item.avsc is deleted, not kept as a parallel doc/reference file. Option 2 (giving OrderItemPayload its own Schema Registry subject) remains rejected — nothing in this exercise changes the original reasoning that it is not an independently-consumed contract, and creating a subject for it would be modeling around a tooling limitation, not a real domain boundary.

Why the "keep order-item.avsc too, as documentation" variant of option 1 is explicitly rejected (not just unnecessary — build-breaking): verified in an isolated scratch copy of the module. avro-maven-plugin's schema goal shares a single Schema.Parser/named-type table across its entire sourceDirectory scan for one execution. Once order-created.avsc carries the type inline, having order-item.avsc also present in src/main/avro — even with a byte-identical definition — makes the plugin fail with Can't redefine: com.telco.platform.events.order.OrderItemPayload. Deleting order-item.avsc and inlining the definition into order-created.avsc (with no <imports> entry at all) builds clean and generates the identical OrderItemPayload.java nested class as before. The now-self-contained order-created.avsc was also POSTed to the live telco-schema-registry container as a throwaway subject and registered successfully (HTTP 200), then deleted, leaving the registry exactly as event-integration left it (0 subjects) — proven both ways: local Java codegen and live Schema Registry validation now agree on one schema document, with no split-brain.

Exact instructions for event-integration (file-by-file)

  1. platform/platform-event-contracts/src/main/avro/order-created.avsc — replace the items field's type.items value (currently the bare string "OrderItemPayload") with the full inline record definition, verbatim from the current order-item.avsc (type: record, name: OrderItemPayload, namespace: com.telco.platform.events.order, same doc, same four fields tariffId/tariffName/unitPrice/quantity with identical types/docs). Result: order-created.avsc becomes a single, fully self-contained schema document with no external reference.
  2. platform/platform-event-contracts/src/main/avro/order-item.avsc — delete this file entirely. Do not retain it under src/main/avro (or anywhere avro-maven-plugin's sourceDirectory scans it) even as a "reference-only" copy — that reproduces the Can't redefine failure. Section B of this document already captures the nested shape in prose; that is sufficient documentation.
  3. platform/platform-event-contracts/pom.xml — remove the <imports> block under the avro-maven-plugin schema execution (the one pointing at ${project.basedir}/src/main/avro/order-item.avsc). No change to <subjects>: order-item.avsc/OrderItemPayload was already correctly absent and stays absent.
  4. No other files change. microservices/order-service/src/test/resources/avro/order-item.avsc and OrderEventSchemaCompatTest.java are a separate, module-local test snapshot in a different Maven module — out of scope for this ruling, untouched.

Verification event-integration must reproduce for the record

  • cd platform/platform-event-contracts && mvn generate-sources -Dschema.registry.skip=true -> BUILD SUCCESS, still 35 generated .java files, OrderItemPayload.java generated as a nested class purely from order-created.avsc's inline definition.
  • With telco-schema-registry running, register order.created.v1 for real - now succeeds. All 33 of 33 real subjects register cleanly (not 32/33 as in the phase-3/4 log), closing Finding 2. Leave the registry exactly as found afterward, same discipline as the other 32 subjects.

This unblocks phase 5: domain-engineer can now validate every service's real payload — including order-service's OrderCreatedEvent/OrderItemPayload — against one schema shape that is simultaneously what local Java codegen produces and what is actually enforceable against a real Schema Registry in CI (phase 8's gate). No more split between "works for avro-maven-plugin" and "works for Confluent."

Ruling applied (event-integration, 2026-07-07) — Finding 2 CLOSED

Applied exactly the 3 file changes the ruling specifies, no others:

  1. platform/platform-event-contracts/src/main/avro/order-created.avsc — the items field's type.items is now the full inline OrderItemPayload record definition (verbatim fields/docs from the former order-item.avsc), not a bare-name reference.
  2. platform/platform-event-contracts/src/main/avro/order-item.avsc — deleted (it was untracked in git, so a plain file removal was used in place of git rm; net effect on the tree is identical — the file is gone and was never part of a prior commit).
  3. platform/platform-event-contracts/pom.xml — removed the <imports> block under the avro-maven-plugin schema execution that pointed at order-item.avsc. <subjects> untouched, order-item/OrderItemPayload correctly still absent from it.

microservices/order-service/src/test/resources/avro/order-item.avsc and OrderEventSchemaCompatTest.java were not touched, per the ruling's scope note.

Verification: - cd platform/platform-event-contracts && mvn generate-sources -Dschema.registry.skip=true — BUILD SUCCESS, still 35 generated .java files; OrderItemPayload.java generated as a nested class purely from order-created.avsc's inline definition (no <imports> involved). - telco-schema-registry was running (docker ps confirmed). Registered all 33 real subjects for real via mvn package -Pregister-schemas -Dschema.registry.skip=false (not a throwaway subject name this time, per instruction) — every subject including order.created.v1 registered cleanly (id 67, version 1), confirmed independently via GET /subjects returning exactly 33 entries. Left registered in the live registry as instructed (not deleted afterward, unlike the prior phase-3/4 disposable verification).

Finding 2 is now CLOSED: 33 of 33 real subjects register cleanly against a live Schema Registry, and mvn generate-sources succeeds offline with the identical generated-class shape. Phase 5 (domain-engineer) can proceed against the now-unified schema.

Architecture ruling: invoice-generated.avsc subscriptionId nullability (phase 5 finding) — tested and REVERTED (2026-07-07)

Finding (billing-service, phase 5 pass). invoice-generated.avsc's subscriptionId field is typed nullable (["null","string"], default: null), but the sole real producer (BillRunBatchProcessor.InvoiceGeneratedEvent) always populates it from SubscriberBillingRecord.subscriptionId, a DB column that is nullable = false, unique = true. billing-service's own test-local snapshot already types this field non-nullable. No real consumer relies on the nullable case.

Architecture's ruling. Tighten the field from ["null","string"]/default null to plain non-nullable "string", matching the real, always-populated data. Ruling flagged a real, unverified compatibility risk: invoice-generated.avsc is one of the 18 pre-existing schemas already live-registered in telco-schema-registry (subject invoice.generated.v1, registered during event-integration's earlier phase 3/4 pass). Under Confluent's default BACKWARD compatibility mode, narrowing a nullable union to non-nullable is often rejected — Avro/Confluent compatibility checking is structural (can a new-schema reader parse an old-schema-written record?), not data-driven (it does not matter that no real message ever used the null case).

What was tested, in order: 1. Applied the exact edit: replaced subscriptionId's type: ["null","string"], default: null with plain type: "string" (updated doc to explain the always-populated DB invariant) in platform/platform-event-contracts/src/main/avro/invoice-generated.avsc. 2. cd platform/platform-event-contracts && mvn generate-sources -Dschema.registry.skip=true — BUILD SUCCESS, still 35 generated .java files. 3. Confirmed telco-schema-registry was running (docker ps, container telco-schema-registry, healthy) and confirmed the live registry's compatibility mode is the default BACKWARD (GET /config -> {"compatibilityLevel":"BACKWARD"}; invoice.generated.v1 has no subject-level override). Tested the tightened schema via POST /compatibility/subjects/invoice.generated.v1/versions/latest?verbose=true — a read-only compatibility check, no registration attempted.

Actual result: REJECTED. is_compatible: false, with the concrete error:

{errorType:'TYPE_MISMATCH', description:'The type (path '/') of a field in the new schema does not
match with the old schema', additionalInfo:'reader type: STRING not compatible with writer type: NULL'}
This confirms architecture's flagged risk exactly: BACKWARD compatibility (can a reader using the new schema parse data written under the old schema?) fails because the registered writer schema permits a null value in that field position and the tightened reader schema cannot accept it — structurally incompatible regardless of whether any real message ever actually carried null.

Final state left in place: REVERTED to nullable, per branch 4 of this task's instructions. Per architecture's own ruling framing — do not override the compatibility mode, do not force registration through, do not keep the change if rejected — invoice-generated.avsc was reverted to its original nullable subscriptionId (["null","string"], default: null). Confirmed via git diff returning empty (file byte-identical to the committed/tracked version) and a rebuild (mvn generate-sources -Dschema.registry.skip=true, BUILD SUCCESS, still 35 generated classes). The live registry was never mutated: invoice.generated.v1 still has exactly one version (GET /subjects/invoice.generated.v1/versions -> [1], id: 60, unchanged), left exactly as found.

Conclusion. "Documented looser than strictly necessary, but safe" is confirmed as the correct final state for invoice.generated.v1. The field remains nullable in the schema even though the real, current producer never emits null — this is intentional slack the schema retains precisely because the subject is already live-registered elsewhere and this session's ADR-019 amendment does not authorize breaking an existing Schema Registry BACKWARD-compatibility guarantee to chase a tighter-than-necessary type. If a future, genuine need arises to enforce non-nullability at the wire-contract level (not just documentation), the correct path is a new event version (invoice.generated.v2) per ADR-009/ADR-019's immutability rule, not a mutation of the existing v1 contract.

Phase 6 (event-integration, 2026-07-07) — DONE: type/nullability-aware compat tooling, re-pointed at the canonical contract

Root cause recap

Every pre-existing *EventSchemaCompatTest/*EventContractTest (order, payment, billing, usage, tariff, customer, subscription/msisdn, ticket, notification) only ever compared Avro field names (Set<String> reflection over Java record components, or over a captured runtime Map for the two services that build Map.of(...) payloads) against each service's own local src/test/resources/avro/*.avsc snapshot. Two failure modes followed directly from that: (1) type and nullability drift (exactly the usage-service timestamp mismatches and the billing-service subscriptionId nullability question this feature already found) passed silently, because two same-named, differently-typed fields still satisfy a name-only Set comparison; (2) the local snapshot was never cross-checked against the canonical file under platform-event-contracts, so the test only proved "the Java class matches a hand-maintained copy," not "the Java class matches the governed contract."

What was built

1. A shared, reusable type-and-nullability-aware checker: AvroContractAssertions.

  • New file: platform/platform-event-contracts/src/test/java/com/telco/platform/events/testsupport/ AvroContractAssertions.java.
  • Packaged as this module's test-jar (new maven-jar-plugin test-jar execution in platform/platform-event-contracts/pom.xml), so every producing service adds it as one ordinary test-scope dependency instead of copy-pasting a checker eight times.
  • Two entry points, matching the two payload-construction styles that actually exist in this codebase:
  • assertRecordMatchesSchema(Schema, Class<?>) — reflects over RecordComponent declared types. Used by every service that publishes a typed Java record (order, payment, billing, usage, tariff, customer, subscription, msisdn, identity).
  • assertPayloadMatchesSchema(Schema, Map<String,Object>) — checks the actual captured runtime value's type and observed nullness. Used by ticket-service and notification-service, which build the outbox payload inline as Map.of(...) with no static Java type to reflect on.
  • canonicalSchema(String generatedClassName) — loads the real Schema embedded in the Avro-generated SpecificRecord class (its static getClassSchema()), by fully qualified class name. This is what makes checking against the canonical, not a local copy, actually possible: the generated classes carry the full typed Schema (unions, logical types, nested records) directly, so no .avsc resource needs to be shipped or parsed by hand.
  • Checks performed, per field: (a) name presence in both directions (existing behavior, kept); (b) Avro type vs. Java type compatibility — stringString; intint/Integer; long (no logical type)→long/Long; long+timestamp-millislong/Long/Instant; booleanboolean/ Boolean; bytes+decimalBigDecimal; enumString/Enum (this codebase's one enum, CdrType, is consumed as a plain String, matching the diff spec's non-blocking note); array of record → a Java List/Collection whose generic element type is itself a record, recursing into the nested schema (covers order.created.v1's inlined OrderItemPayload automatically, with no special-casing needed); record → a Java record, recursing. (c) nullability, in both directions:
  • Avro non-nullable, Java signals it may be null (a boxed wrapper type — Long/Integer/etc. instead of the primitive — or Optional<T>, or, for the runtime/map entry point, an actually-observed null value): FAIL. A real bug — the wire contract promises always-present, the producer can omit it.
  • Avro nullable (["null", X]), Java can never be null (a primitive): WARN only (printed via System.out, does not fail the build). Slack the schema retains that the current producer does not need — looser than strictly necessary, but safe (matches the exact invoice.generated.v1 ruling above).
  • Everything else — in particular a plain reference type such as String or BigDecimal, which the Java type system cannot prove non-null either way without an annotation this codebase does not use — is undecidable from reflection alone and is not asserted in either direction. This is a deliberate, honest limitation, not a gap papered over: it is exactly why invoice-generated.avsc's nullable-but-always-populated subscriptionId (String in the Java record) correctly produces neither a FAIL nor a WARN.
  • Self-tested independently of any microservice: platform/platform-event-contracts/src/test/java/com/telco/platform/events/testsupport/ AvroContractAssertionsSelfTest.java (9 cases: exact match passes; type mismatch fails and names the field; missing field fails; extra field fails; a non-nullable schema field with a boxed-nullable Java type fails; a nested array-of-records both passes when matching and fails when the nested element type drifts; the runtime/map entry point passes and fails symmetrically).

2. Architecture decision: option (a) — every compat test now loads the canonical schema directly from platform-event-contracts, not a local snapshot.

Reasoning: (i) the Avro-generated SpecificRecord classes in platform-event-contracts already carry the complete, real Schema (including unions, logical types, and nested records) as a static field — there is nothing to gain and real self-reference risk to keep by hand-copying .avsc text into each service's src/test/resources; (ii) platform-event-contracts was already a normal (compile-scope) Maven dependency for four of these eight services (usage, billing, notification, ticket) — extending that established, already-accepted pattern (a plain module dependency, not a "starter") to the remaining six is the smallest, most consistent change, and does not violate ADR-018 (that rule is about services depending only on platform starters for runtime infrastructure — platform-event- contracts is a contracts module, not a starter, and existing usage already treats it as an ordinary, directly-depended-on module); (iii) option (b) ("keep local copies, add a drift guard") would have kept the duplication this reconciliation exists to remove and added a second tool (a file-diff guard) next to the first (the compat test) for no benefit over option (a). Option (a) was therefore the clear choice, with no need to escalate.

Per-service pom change: each of customer-service, order-service, payment-service, product-catalog-service, subscription-service, and identity-service (previously had no platform-event-contracts dependency at all) gained two new test-scope dependencies:

<dependency>
    <groupId>com.telco.platform</groupId>
    <artifactId>platform-event-contracts</artifactId>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>com.telco.platform</groupId>
    <artifactId>platform-event-contracts</artifactId>
    <type>test-jar</type>
    <scope>test</scope>
</dependency>
usage-service, billing-service, notification-service, and ticket-service already had the plain (compile-scope) dependency — left untouched — and only gained the new test-jar dependency. platform/platform-bom/pom.xml gained a matching dependencyManagement entry for the test-jar classifier (Maven's BOM entries are keyed by groupId:artifactId:type:classifier — the pre-existing plain-jar entry did not cover the new classifier, and the build fails fast and explicitly if it is missing, which is exactly what happened on the first attempt and was fixed).

3. All 32 canonical schemas now covered, one *EventSchemaCompatTest/*EventContractTest per service, all pointed at the canonical schema:

Service Test class Events (count)
order-service OrderEventSchemaCompatTest order.created.v1 (incl. nested OrderItemPayload, checked via automatic recursion, no separate test case needed), order.cancelled.v1 (2)
payment-service PaymentEventSchemaCompatTest payment.completed.v1, payment.failed.v1, payment.refunded.v1 (3)
billing-service BillingEventSchemaCompatTest invoice.generated.v1, invoice.paid.v1, invoice.overdue.v1 (3)
usage-service UsageEventSchemaCompatTest usage.recorded.v1, quota.threshold-reached.v1, quota.exceeded.v1, usage.aggregated.v1, cdr.recorded.v1 (5 — cdr.recorded.v1 newly added to this test, checked against the consumer's CdrRecordedEventConsumer.CdrPayload DTO since it is a consumed, not produced, external contract)
notification-service NotificationEventContractTest notification.dispatched.v1 (1)
ticket-service TicketEventContractTest ticket.opened.v1, ticket.assigned.v1, ticket.resolved.v1, ticket.sla-breached.v1 (4)
product-catalog-service TariffEventSchemaCompatTest tariff.created.v1, tariff.price-changed.v1 (2)
subscription-service SubscriptionEventSchemaCompatTest + MsisdnEventSchemaCompatTest subscription.activated.v1, subscription.suspended.v1, subscription.terminated.v1, subscription.activation-failed.v1, msisdn.allocated.v1, msisdn.released.v1 (6)
customer-service CustomerEventSchemaCompatTest customer.registered.v1, customer.updated.v1, customer.kyc-approved.v1, customer.kyc-rejected.v1 (4)
identity-service IdentityEventSchemaCompatTest (new file — this service had no compat-test infrastructure at all before this phase) user.created.v1, user.deleted.v1 (2)

Total: 2+3+3+5+1+4+2+6+4+2 = 32, matching all 32 canonical schemas (18 pre-existing + 14 promoted/ authored in phases 3-4).

The old per-service local src/test/resources/avro/*.avsc snapshot directories (billing, customer, notification, order, payment, product-catalog, subscription, ticket, usage — order-service's also included the now-superseded standalone order-item.avsc) are deleted, not left in place: once no test references them, keeping byte-identical duplicates of files that already live canonically in platform-event-contracts is exactly the self-referential duplication this phase exists to remove.

Proof of failure, then revert (exit criterion)

  1. Deliberately edited platform/platform-event-contracts/src/main/avro/usage-recorded.avsc, changing recordedAt from plain "string" to "long" (a real, targeted type mismatch — the real producer, UsageRecordedEvent.recordedAt, is a Java String).
  2. cd platform/platform-event-contracts && mvn install -Dschema.registry.skip=true — this module's own self-test caught the break immediately (AvroContractAssertionsSelfTest. passes_when_record_matches_schema_exactly and fails_on_type_mismatch both failed, since the self-test's fixture is written against the real UsageRecordedV1 canonical schema). Installed with -DskipTests to get the deliberately-broken artifact into the local repo so the real per-service test could be exercised too.
  3. cd microservices && mvn -pl usage-service test -Dtest=UsageEventSchemaCompatTestFAILED, with the exact, unambiguous diagnosis:
    java.lang.AssertionError:
    Avro contract violation(s) for schema com.telco.platform.events.usage.UsageRecordedV1 vs com.telco.usage.application.event.UsageRecordedEvent:
      - UsageRecordedEvent.recordedAt: Avro type LONG expects Java type long or Long, but found java.lang.String
    
    This identifies the exact field (recordedAt), the exact schema (UsageRecordedV1), the exact Java class (UsageRecordedEvent), and the exact type mismatch — proving the extended checker catches what the old field-name-only test structurally could not (the field name recordedAt never changed; a pre-14.5 test would have passed this exact break).
  4. Reverted usage-recorded.avsc to its original recordedAt: "string" (confirmed byte-identical to the pre-edit content). Rebuilt platform-event-contracts (mvn install -Dschema.registry.skip=true) — all 9 self-tests pass again, BUILD SUCCESS. Re-ran usage-service's UsageEventSchemaCompatTest — all 5 cases pass again, BUILD SUCCESS.

Full verification run

cd microservices && mvn -pl customer-service,order-service,payment-service,usage-service, billing-service,notification-service,ticket-service,product-catalog-service,subscription-service, identity-service verify -DskipITsBUILD SUCCESS, all 10 modules, 0 test failures/errors across the reactor (unit tests, JaCoCo coverage gates, checkstyle, spotbugs all included in verify). Followed by cd microservices && mvn compile across the entire microservices reactor (every service, including the ones untouched by this phase) to confirm the platform-bom change — every microservice transitively imports it — did not break dependency resolution anywhere: BUILD SUCCESS, all modules. The full mvn -f microservices/pom.xml verify (every module including integration/acceptance tests) was not run in full given its testcontainers/Docker startup cost is unrelated to this phase's changes; the targeted 10-module verify plus the full-reactor compile sanity check together cover everything this phase touched (10 services' test code and poms) and everything it could have broken (every service's dependency resolution via the shared BOM).

Files changed (phase 6)

  • New: platform/platform-event-contracts/src/test/java/com/telco/platform/events/testsupport/ AvroContractAssertions.java, AvroContractAssertionsSelfTest.java.
  • platform/platform-event-contracts/pom.xml: added test-scope junit-jupiter; added the maven-jar-plugin test-jar execution.
  • platform/platform-bom/pom.xml: added the test-jar-classifier dependencyManagement entry for platform-event-contracts.
  • Pom dependency additions (test-scope platform-event-contracts + its test-jar): customer-service, order-service, payment-service, product-catalog-service, subscription-service, identity-service. Test-jar-only addition (main dependency already present): usage-service, billing-service, notification-service, ticket-service.
  • Rewritten to use AvroContractAssertions against the canonical schema: OrderEventSchemaCompatTest, PaymentEventSchemaCompatTest, BillingEventSchemaCompatTest, TariffEventSchemaCompatTest, CustomerEventSchemaCompatTest, SubscriptionEventSchemaCompatTest, MsisdnEventSchemaCompatTest, UsageEventSchemaCompatTest (also gained the 5th, previously-untested cdr.recorded.v1 case), TicketEventContractTest, NotificationEventContractTest.
  • New: identity-service/src/test/java/com/telco/identity/IdentityEventSchemaCompatTest.java (this service had zero compat-test infrastructure before this phase).
  • Deleted: all per-service src/test/resources/avro/*.avsc local snapshot directories (billing, customer, notification, order (incl. order-item.avsc), payment, product-catalog, subscription, ticket, usage) — superseded by loading the canonical schema directly.

Phase 6 (event-integration) is DONE. Phase 7 (qa: full contract + acceptance suite run, event- catalog.md updates) is next.

Phase 7 (qa, 2026-07-07) — DONE

1. Full contract-test suite: mvn -f microservices/pom.xml verify (full reactor)

JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-21.jdk/Contents/Home mvn -f microservices/pom.xml verifyBUILD SUCCESS, all 18 reactor modules (microservices parent, domain-services-parent, service-template, reference-service, config-server, discovery-server, api-gateway, identity-service, customer-service, product-catalog-service, order-service, subscription-service, usage-service, billing-service, payment-service, notification-service, ticket-service, web-bff). Total wall time 6m06s. Aggregate test count across all per-service Surefire/Failsafe summaries: 638 tests run, 0 failures, 0 errors, 0 skipped (identity 38, customer 83, product-catalog 49, order 85, subscription 67, usage 77, billing 71, payment 64, notification 45, ticket 47, plus the smaller platform/gateway modules). This includes every one of the 32 new/rewritten *EventSchemaCompatTest/*EventContractTest classes from phase 6 (all green), the new IdentityEventSchemaCompatTest, and every Testcontainers-backed integration test and JaCoCo coverage gate in the reactor - "All coverage checks have been met" for every module that runs one. This is the primary gate for Feature 14.5 and it is green: no regression from phases 1-6's schema reconciliation, new schemas, the rename, or the new compat-test tooling.

2. Acceptance suite: mvn -f microservices/pom.xml -pl acceptance-tests -am -Pacceptance verify

The full docker-compose live stack (gateway, Keycloak, Kafka, Schema Registry, all ten domain services, Postgres, Redis, Mongo, MinIO) was already running from earlier this session (docker ps confirmed all containers healthy) - per this task's own instruction, standing up a fresh stack was not required, and since Feature 14.5 changed only canonical .avsc files and test/pom code (no producing service's main-source event-payload class changed shape - the diff spec's explicit direction throughout was "schema matches real payload," not the reverse, and the one place a real code change was tested (invoice-generated.avsc subscriptionId tightening) was reverted after the live compatibility check rejected it), there was no runtime-behavior change for the already-running containers to be stale against. Ran the suite against the live stack as-is.

Result: BUILD FAILURE — 1 of 4 acceptance tests failed. AC-01 (compensation path), AC-02 (monthly bill-run), and AC-03 (quota exhaustion) all passed. The one failure:

NewSubscriberOnboardingAcceptanceIT.newSubscriberOnboardingSucceedsEndToEnd:99
Expecting actual:
  "905990000002"
to match pattern:
  "90532\d{7}"

Root cause investigated and confirmed NOT a Feature 14.5 regression. Queried the live subscription_db.msisdn_pool table directly (docker exec telco-postgres psql ...):

prefix | status    | count
90532  | ALLOCATED | 1000    <- the full seeded pool (V2__msisdn_pool_seed.sql), 100% exhausted
90599  | ALLOCATED | 4
90599  | FREE      | 96

1100 total rows, but V2__msisdn_pool_seed.sql (unmodified since Sprint 09 - git log confirms no commit or working-tree change to it this session) only ever seeds 1000 rows in the 90532 block. The 100-row 90599 block exists only in the live database, inserted directly (not via a Flyway migration - grep of every .sql file under subscription-service/src/main/resources/db/migration/ finds zero occurrences of 9059 anywhere in source). This is a stopgap top-up applied straight to the running container's database at some earlier point this session, almost certainly after the original 1000-number 90532 pool was fully exhausted by the volume of acceptance-suite runs already executed against this same long-lived stack (container uptimes of 9-37 hours per docker ps). MsisdnAllocationService.allocate() (subscription-service/src/main/java/com/telco/subscription/ domain/MsisdnAllocationService.java) has no format-specific logic at all - it just takes the next FREE row from the pool table regardless of prefix, so once the 90532 block hit zero FREE rows, allocation legitimately started serving numbers from the 90599 top-up block, which the test's hardcoded "90532\\d{7}" regex (line 99) was never written to expect.

Confirmed no Feature 14.5 code path touches this: git diff on subscription-service's migration directory is empty; the five subscription-service main-source files this session did modify (SubscriptionController, PaymentCompletedEventConsumer, PaymentFailedEventConsumer, GetSubscriptionsByCustomerQuery(Handler)) belong to the unrelated 14.1.1 identity-linkage-gap work, none reference msisdn/MSISDN allocation, and MsisdnAllocationService/MsisdnPoolRepository/ MsisdnPool are untouched by any change in this session.

Disposition: pre-existing test-environment/data issue, out of scope for Feature 14.5, not fixed here. This is a live-environment artifact (pool exhaustion plus an ad hoc, out-of-migration DB patch) tied to this specific long-running Docker stack's accumulated state, not a defect in any schema, event payload, or the AC-01 saga logic itself - a fresh environment seeded only from V2__msisdn_pool_seed.sql would allocate a correct 90532xxxxxxx number. Fixing it durably (e.g., widening the seeded pool, making pool-exhaustion behavior deterministic for test/CI environments, or loosening the test's regex to accept any valid Turkish MSISDN prefix rather than one hardcoded block) is a test-data/environment decision outside this feature's scope, flagged here for devops/domain-engineer to pick up, not decided unilaterally as part of 14.5's phase 7. No code, schema, or test file was changed to work around it.

Phase 7 exit criterion assessment: "green contract-test and acceptance runs" - the contract-test half is fully green (638/638). The acceptance half is green on AC-01's compensation path, AC-02, and AC-03; AC-01's happy path fails on a root cause independently verified to be outside this feature's change set. Phase 7 is DONE with this one flagged, pre-existing, out-of-scope finding - it does not block Feature 14.5's own sign-off, since "confirm no regression from phases 1-6" is satisfied: this failure predates and is unrelated to those phases.

3. docs/architecture/event-catalog.md updates

  • Added two rows to Section 2 (Event Registry): user.created.v1 and user.deleted.v1 (identity-service, no current consumer - confirmed via grep that no service consumes either event yet). These were the two events architecture's phase-2 diff spec flagged as entirely absent from the catalog.
  • Added a new Section 6, "Schema Governance Reconciliation Log (Feature 14.5)", documenting: the 7 reconciled pre-existing schemas (order-created, payment-completed, cdr-recorded, usage-aggregated, usage-recorded, quota-exceeded, quota-threshold-reached, including the nested OrderItemPayload inline-record resolution), the 14 newly added schemas, the EventEnvelope.avsc -> event-envelope.avsc rename, and the new AvroContractAssertions compat-test tooling requirement going forward. Distinct from the pre-existing Section 5 ("Schema Evolution Log"), which tracks additive field changes to already-registered schemas - Section 6 is scoped to this one-time reconciliation exercise.
  • Bumped the document's "Last updated" field to 2026-07-07.

4. notification-service test-fixture fix

DomainEventNotificationConsumerTest.java (microservices/notification-service/src/test/java/com/ telco/notification/consumer/) had two tests asserting against fictional event-type strings never produced by any real service (subscription.cancelled.v1, customer.profile-updated.v1) - flagged during phase 5. Re-pointed both at real, legitimately-unhandled event names rather than deleting the scenarios, since the behavior under test (an unrecognised event type is silently ignored) is real and worth keeping: - subscription_event_with_unrecognised_type_is_silently_ignored now uses subscription.suspended.v1 - a real event subscription-service produces, which DomainEventNotificationConsumer.onSubscriptionEvent does not currently handle (confirmed via grep - no SUSPEND/TERMINAT handling exists in this consumer). - customer_event_with_unrecognised_type_is_silently_ignored now uses customer.updated.v1 - a real event customer-service produces, which onCustomerEvent does not currently handle (only customer.kyc-approved.v1/customer.kyc-rejected.v1 are handled).

Verified passing as part of the full reactor run above: com.telco.notification.consumer. DomainEventNotificationConsumerTest - 14 tests run, 0 failures, 0 errors.

5. Other loose ends from phases 1-6 checked for qa scope

Re-read phases 1-6 in full for anything else explicitly deferred to qa. Found one other item, correctly out of scope and left untouched: DomainEventNotificationConsumer.onSubscriptionEvent reads a customerName field off subscription.activated.v1 payloads that no producer ever sets (falls back to customerId via getOrDefault, never breaks). Phase 3/4's diff spec explicitly labeled this "flagged for qa/domain-engineer awareness only, not actioned here" - an FYI, not an action item, and not an Avro contract violation. Left as-is, per that explicit framing; no other item in phases 1-6 was addressed to qa beyond the test-fixture fix above and this feature's own phase 7 exit criteria.

Phase 7 conclusion

Phase 7 (qa) is DONE: contract-test suite green (638/638), acceptance suite green on 3 of 4 flows with the one AC-01-happy-path failure root-caused to a pre-existing, out-of-scope MSISDN-pool exhaustion artifact in this session's long-lived live stack (not a Feature 14.5 regression), event-catalog.md updated with the two missing rows and a new Schema Evolution/Reconciliation log section, and the notification-service test-fixture fix applied and verified passing. Phase 8 (devops + code-review + tech-lead sign-off) is next; the AC-01 MSISDN-pool finding should be handed to devops/domain-engineer as a separate, pre-existing item, not blocking Feature 14.5's own sign-off.

Phase 8 — devops portion (2026-07-07) — DONE

1. Compat-test gate in CI — already covered, no new CI step needed (confirmed, not assumed)

.github/workflows/ci.yml's microservices-test job runs mvn ${MAVEN_ARGS} verify --fail-at-end -Dspotbugs.skip=true -Dcheckstyle.skip=true in microservices/, after a preceding Install platform to local Maven repo step (mvn install -DskipTests -Dspotbugs.skip=true -Dschema.registry.skip=true in platform/). microservices/pom.xml already declares all 10 producing services (identity-service, customer-service, product-catalog-service, order-service, subscription-service, usage-service, billing-service, payment-service, notification-service, ticket-service) as reactor modules, so this existing verify run already compiles and executes every one of phase 6's rewritten *EventSchemaCompatTest/*EventContractTest classes - no new job, step, or workflow was needed. Concretely verified rather than assumed:

  • cd platform/platform-event-contracts && mvn install -DskipTests -Dspotbugs.skip=true -Dschema.registry.skip=true (byte-identical to the CI step) - confirmed it produces both platform-event-contracts-1.0.0-SNAPSHOT.jar and the -tests.jar classifier in the local repo (~/.m2/repository/com/telco/platform/platform-event-contracts/1.0.0-SNAPSHOT/), i.e. the test-jar the 10 services now depend on (phase 6) is genuinely produced by this exact CI command, not just by a developer's full local build.
  • cd microservices && mvn -pl identity-service -am test -Dtest=IdentityEventSchemaCompatTest -Dspotbugs.skip=true -Dcheckstyle.skip=true -o against that freshly-installed repo (offline flag forces resolution from exactly what the install step produced, no leftover artifacts from a different build) - BUILD SUCCESS, 2/2 tests passed. Chosen because identity-service had zero compat-test infrastructure before phase 6 (new file, new dependency, most likely to expose a wiring gap) - proves the newly-added test-jar dependency resolves and the new test class runs correctly under CI's own exact command sequence.
  • This complements, not duplicates, phase 6's own proof-of-failure (the deliberately-broken usage-recorded.avsc recordedAt type-mismatch, which failed UsageEventSchemaCompatTest loudly) and phase 7's full-reactor mvn -f microservices/pom.xml verify (638/638). Together these three runs cover: the CI command sequence in isolation (this phase), the extended checker's actual fault-detection behavior (phase 6), and the complete, unmodified reactor (phase 7).

Conclusion: no CI change was required for point 1. A PR that reintroduces drift between a service's real event payload and its canonical schema fails microservices-test in ci.yml today, on every PR to master, with no additional wiring.

2. Schema Registry compatibility check in CI — partially closed, with an honest, tested limit

Before this phase: ci.yml ran the platform build with -Dschema.registry.skip=true in every job that touches platform/ (build-test, static-analysis, and microservices-test's install step) - no live registry, by design, unchanged by Feature 14.5. .github/workflows/acceptance.yml already brings up a full docker-compose stack including a real schema-registry container (telco-schema-registry, localhost:8081 - the exact default in platform-event-contracts/pom.xml's schema.registry.url) for Debezium's benefit, but its own Install platform to local Maven repo step also passed -Dschema.registry.skip=true, meaning the one workflow that already had a live registry sitting right there was not using it for this check either. Net effect: the Schema Registry compatibility check ran nowhere in CI, contradicting phase 8's stated exit criterion.

What was verified by hand, live, before changing anything (using this session's long-running local stack, telco-schema-registry, confirmed healthy via docker ps, plus a disposable no-history registry container stood up on an unused port to faithfully simulate a brand-new, empty-per-run CI registry):

  1. cd platform/platform-event-contracts && mvn verify -Dschema.registry.skip=false against the long-lived registry (which already carries real, persisted history for all 33 subjects from earlier phases) - BUILD SUCCESS, all 33 subjects reported is_compatible: true.
  2. The identical command against a freshly started, empty registry container (no prior versions for any subject) - also BUILD SUCCESS, all 33 subjects compatible - because a subject with no registered version is trivially compatible (confirmed directly via POST /compatibility/subjects/<nonexistent>/versions/latest, is_compatible: true).
  3. The same nonexistent-subject endpoint, given a schema with a genuine parse error (an Avro field typed to an undefined name) - is_compatible: false, with a clear "is not a defined name" error. Confirms registrability/structural validity is checked regardless of history.
  4. A deliberate, targeted type-mismatch (usage-recorded.avsc's recordedAt retyped from string to long - the same class of drift phase 6 proved the compat-test tooling catches) run against the empty registry - BUILD SUCCESS (incorrectly "compatible", because there is no prior version to violate). The identical edit run against the long-lived, history-bearing registry - BUILD FAILURE (One or more schemas found to be incompatible with the current version), correctly caught. Reverted immediately after (confirmed via git diff, byte-identical to before).

This proves, rather than assumes, the exact shape of the gap: the compatibility check's value in an always-fresh CI registry is real but partial - it enforces registrability (a schema must be valid, parseable Avro that Schema Registry itself accepts, which is exactly the defect class Feature 14.5 found in order.created.v1's original split OrderItemPayload reference) but cannot enforce persisted BACKWARD-compatibility drift across time, because no CI-recreated registry carries history from a previous run to diff against.

Change made: .github/workflows/acceptance.yml's Install platform to local Maven repo step now runs with -Dschema.registry.skip=false (was true), so the kafka-schema-registry-maven- plugin's test-compatibility goal (bound to the verify phase, which mvn install already passes through) executes for real, against the real telco-schema-registry container this workflow already stands up, for all 33 canonical subjects, on every PR touching microservices/**, platform/**, infra/docker/**, or the workflow file itself, plus on-demand via workflow_dispatch. A detailed comment was added directly above the step explaining both what this catches (registrability/parse validity, every run) and what it does not (true point-in-time BACKWARD-compat drift, since the registry has no persisted history across runs) - the same honest distinction proven above, so no future reader mistakes this for a full historical-drift gate. ci.yml was left unchanged (-Dschema.registry.skip=true in all three of its platform-touching steps) - it has no live registry available at all, and standing one up there is a separate CI-infrastructure decision, not reopened here - but its existing comment was expanded to point at acceptance.yml's gate and this document for the full picture, rather than leaving a stale "runs locally only" claim in place now that half of that claim is no longer true.

Residual gap, stated plainly, not papered over: no CI workflow in this repository currently enforces true, persisted-history BACKWARD Schema Registry compatibility - ci.yml (the per-PR gate for every change) has no registry at all, and acceptance.yml's registry, while real and now exercised, is destroyed and recreated empty on every run (make -C infra destroy in its always-run teardown step), so it can only ever catch structural/registrability breaks, not a genuine type or nullability narrowing against a previously-registered version (the exact case invoice-generated.avsc's subscriptionId nullability ruling, above, depended on a persistent registry to catch). Closing this fully would require a schema-registry deployment whose subject history survives across CI runs (e.g., a long-lived registry service outside the ephemeral per-job compose stack, or a persisted-volume approach) - a real CI infrastructure decision requiring its own design (where does that registry live, who else can pollute its subject history, what is its compatibility mode, how is it reset when a subject's v1 is legitimately superseded) - flagged here for a future ticket, not decided unilaterally as part of this feature's phase 8.

3. Verification performed

  • YAML syntax: python3 -c "import yaml; yaml.safe_load(open(...))" on both .github/workflows/ci.yml and .github/workflows/acceptance.yml - both parse cleanly. No GitHub Actions lint tool (actionlint) was available in this environment to install without network access beyond what was already used; the YAML-parse check plus a structural diff review (comment placement, indentation matching every neighboring step) stand in for it, per this task's own "at minimum" bar.
  • No real CI run was triggered (no push/PR involved), per this task's own scope - all verification above was performed by running the exact same Maven commands acceptance.yml and ci.yml invoke, locally, against real (long-lived and disposable) Schema Registry containers, and by running the exact reactor commands ci.yml's microservices-test job runs.
  • All experimental/scratch changes (usage-recorded.avsc's temporary type mismatch, the disposable telco-schema-registry-fresh-test container) were reverted/removed; git diff on usage-recorded.avsc shows only phase 3/4's already-existing, intentional reconciliation, nothing from this verification.

Phase 8 (devops) conclusion

Point 1 (compat-test gate wired into CI) required no change - confirmed, with concrete proof beyond reading the YAML, that the existing ci.yml pipeline already exercises all 32 canonical-schema-vs-real-payload checks via the 10 rewritten *EventSchemaCompatTest/ *EventContractTest classes on every PR to master. Point 2 (Schema Registry compatibility-mode check in CI) was partially closed: acceptance.yml now runs the real compatibility check against a real, already-present Schema Registry container for all 33 subjects, closing the registrability/structural-validity gap for good; ci.yml's per-PR gate still has no registry available at all (pre-existing, explicitly out of scope to reopen) and even acceptance.yml's newly -enabled check cannot catch true persisted-history BACKWARD-compatibility drift, since its registry is recreated empty every run - both limits are documented in the workflow files themselves and here, not silently assumed or fabricated. Phase 8's devops portion is DONE; code-review's ADR-019- compliance pass and tech-lead's final sign-off are next.

Phase 8 — code-review portion (2026-07-08) — DONE

Scope: read-only ADR-compliance pass over the actual diff this feature produced (working tree at time of review — no commit boundary existed yet). Spot-checked, not just narrative-trusted: read ADR-018/ADR-019 directly, diffed every changed file under platform/platform-event-contracts/, read the real Java payload classes for a sample of reconciled/new schemas, scanned for emojis, and reviewed AvroContractAssertions.java end-to-end.

Finding 1 (MEDIUM) — invoice-generated.avsc's subscriptionId doc field does not capture the

rejected-tightening history; a future reader can re-attempt the same rejected fix

File: platform/platform-event-contracts/src/main/avro/invoice-generated.avsc, subscriptionId field (currently ["null","string"], default: null).

File state confirmed correct — still nullable, byte-identical to the pre-14.5 committed version (git diff against HEAD is empty), matching this document's "Final state left in place: REVERTED to nullable" claim above.

However, the field's own doc string reads: "Subscription this invoice covers, when invoice is per-subscription. Nullable for account-level invoices." This describes a business-semantics reason for nullability ("account-level invoices") that this document's own investigation shows is not the real reason — the real, current producer (BillRunBatchProcessor.InvoiceGeneratedEvent) always populates subscriptionId, and the field is kept nullable purely because a live BACKWARD-compatibility check against the already-registered invoice.generated.v1 subject rejected tightening it (is_compatible: false, TYPE_MISMATCH, documented above). A future engineer reading only the schema file (not this tracking doc) will see a plausible-sounding but incorrect business justification and has no signal that tightening this field was already tried, tested against the live registry, and rejected — exactly the "rediscover this exact question" failure mode this review was asked to check for.

Required fix: update the doc string in invoice-generated.avsc to state the real reason, e.g.: "Subscription this invoice covers. Always populated by the current producer (BillRunBatchProcessor); kept nullable because invoice.generated.v1 is already live-registered in Schema Registry and a BACKWARD-compatibility check confirmed tightening to non-nullable is rejected (TYPE_MISMATCH: reader STRING not compatible with writer NULL). To enforce non-nullability at the wire-contract level, introduce invoice.generated.v2 per ADR-009/ADR-019's immutability rule — do not mutate this v1 field. See docs/tasks/sprint-14-testing-and-hardening/14.5-avro-schema-governance- ruling.md for the full compatibility-check evidence." Non-blocking (does not change wire shape or compatibility), but should land before final close so the tracking doc's own durability goal ("prevents this drift from recurring") actually holds for this specific, already-tested trap.

Finding 2 (LOW/advisory, escalate) — ADR-018 boundary call on platform-event-contracts was

decided unilaterally by event-integration rather than escalated

ADR-018 (architecture/adr/ADR-018-platform-starter-dependency-model.md), Dependency Rule (section 2): "Microservices MUST ONLY depend on starters. They MUST NOT depend on internal platform modules directly." This is unscoped, absolute text — it does not carve out test-scope dependencies or "contracts-only" modules the way ADR-019's amendment explicitly carved out JSON-shape-vs-wire-bytes for outbox publishing.

platform-event-contracts lives under platform/ alongside platform-starters/, platform-core, and platform-autoconfigure — it is not itself a spring-boot-starter-* module (no AutoConfiguration classes, not listed among ADR-018's starter examples). Phase 6 extended a direct, non-starter dependency on it from 4 services (pre-existing, and in 2 of those 4 cases as a production/compile-scope dependency, e.g. billing-service/pom.xml, usage-service/pom.xml) to 10 services (test-scope only for the 6 newly added: customer-service, order-service, payment-service, product-catalog-service, subscription-service, identity-service) — verified directly in each service's pom.xml diff.

Event-integration's phase-6 log resolves this itself: "does not violate ADR-018 (that rule is about services depending only on platform starters for runtime infrastructure — platform-event-contracts is a contracts module, not a starter)... no need to escalate." That is a reasonable reading in spirit (test-scope, schema/generated-class artifacts are a different kind of thing than starter-mediator/starter-outbox/etc.), and it is consistent with an already-existing, pre-14.5 pattern (the 4 services' compile-scope dependency predates this feature). But it is a genuine interpretation of ADR-018's literal, unscoped text, not a mechanical application of it — and this same feature escalated comparably-sized boundary questions to architecture/tech-lead elsewhere (the OrderItemPayload registry-subject conflict, the invoice-generated.avsc nullability question). Deciding this one unilaterally is inconsistent with that same discipline.

Required fix (non-blocking for this sign-off, but must be closed): get an explicit architecture/tech-lead ruling — either (a) formally amend ADR-018 with a stated carve-out for contract/schema modules (mirroring how ADR-019's amendment explicitly scoped A1-A5), or (b) rule that this reading is correct and record it as an addendum here, the same way the OrderItemPayload and invoice-generated.avsc questions were recorded. Until one of those exists, this dependency pattern rests on one agent's unescalated reading of an absolute ADR sentence.

Checks that passed clean

  1. ADR-019 compliance of the 33 .avsc files — all filenames kebab-case (verified programmatically, zero non-compliant names); all 33 files are valid JSON; event-envelope.avsc rename confirmed with record name unchanged (EventEnvelope); pom.xml's kafka-schema-registry-maven-plugin <subjects> block lists exactly 33 entries (event-envelope
  2. 32 real events), no <imports> block (confirms the OrderItemPayload inline-record ruling was actually applied, not just claimed). Spot-checked order-created.avsc against OrderCreatedEvent.java, payment-completed.avsc against PaymentCompletedEvent.java, invoice-generated.avsc/invoice-paid.avsc/invoice-overdue.avsc against BillRunBatchProcessor/MarkInvoicePaidCommandHandler/MarkInvoicesOverdueCommandHandler, and user-created.avsc/user-deleted.avsc against UserCreatedV1.java/UserDeletedV1.java — all field-for-field, type-for-type exact matches, no discrepancy from what this document claims.
  3. No unintended blast radius — confirmed no canonical schema exists for customer.profile-updated.v1, subscription.cancelled.v1, demoitem.created.v1, or order.confirmed.v1 (directory listing + content grep, all clean). order-cancelled.avsc is confirmed to be the real, distinct order.cancelled.v1 event, not a stray file for the excluded subscription.cancelled.v1. Noted in passing, not a 14.5 defect: the working tree also contains unrelated, uncommitted changes from other in-flight work (14.1.1's registeredByUserId addition to customer-registered.avsc/CustomerRegisteredV1, and an unrelated JaCoCo hard-gate change to ci.yml) — confirmed both are attributable to separate, already-ruled-on tracking docs, not to this feature, and neither touches anything this feature's non-goals list forbids.
  4. No emojis — scanned all 33 .avsc files, AvroContractAssertions.java + AvroContractAssertionsSelfTest.java, both CI workflow diffs, this tracking doc, ADR-019, and event-catalog.md with a Unicode emoji-range pattern; zero real emoji hits. (The only symbol matches were plain arrows in this tracking doc's prose, not emojis — ARC-09 clean.)
  5. AvroContractAssertions code quality — the two-entry-point split (typed-record reflection vs. captured-Map runtime check) matches the two payload-construction styles that actually exist in this codebase, with no third path speculatively built. The class Javadoc states the WHY (why canonical-schema-not-local-copy, why FAIL vs. WARN vs. undecidable for nullability) up front, which is exactly where this project's comment discipline expects it. Recursion into nested records/arrays is handled once, generically (covers order.created.v1's inlined OrderItemPayload with no special-casing). No over-engineering — no reflection-based generic type-adapter framework, no attempt to handle Avro types this codebase doesn't use. Self-test (AvroContractAssertionsSelfTest, 9 cases) exercises both entry points, the nested-array recursion, and the nullability FAIL/WARN paths.

Verdict: APPROVE, with Finding 1 required before final close and Finding 2 flagged for

tech-lead's sign-off pass (see below)

Nothing found rises to a build-blocking or wire-compatibility-breaking defect. Finding 1 is a concrete, low-effort documentation fix that should land before this feature is marked fully closed (it directly serves this feature's own stated goal of preventing rediscovery of an already-rejected fix). Finding 2 is not a demonstrated violation — it is a genuine ADR-018 boundary-interpretation question that this feature's own escalation discipline says should go to tech-lead rather than rest on one agent's unilateral reading, exactly the kind of item this document has recorded and escalated repeatedly elsewhere. Recommend tech-lead's final sign-off either accept event-integration's reading explicitly (making this doc that ruling) or send it back for the ADR-018 amendment.

Tech-lead final sign-off (2026-07-08) — Feature 14.5 is DONE

Read the complete 8-phase record above in full, including this document's own phase-1 ruling, the ADR-019 amendment, architecture's diff spec, event-integration's schema authoring, the order.created.v1/OrderItemPayload inline-record ruling, phase 5's per-service reconciliation (folded into phases 3-6 above), phase 6's new AvroContractAssertions tooling and its proof-of-failure/revert cycle, phase 7's qa verification (638/638 reactor tests, event-catalog.md updated, the one AC-01 failure independently root-caused to a pre-existing, out-of-scope MSISDN pool-exhaustion artifact), phase 8's devops CI wiring, and code-review's APPROVE-with-2-findings pass. Both open findings are resolved below.

Finding 1 (MEDIUM, code-review) — RESOLVED

platform/platform-event-contracts/src/main/avro/invoice-generated.avsc's subscriptionId field doc string has been corrected in place. Verified: file is still valid JSON (python3 -c "import json; json.load(open(...))" — clean) and platform/platform-event-contracts still builds green (mvn generate-sources -Dschema.registry.skip=true — BUILD SUCCESS) after the edit; no field, type, or nullability changed, only the doc string. New doc text (in place of the old business-semantics claim):

"Subscription this invoice covers. Always populated by the current producer (BillRunBatchProcessor); kept nullable because invoice.generated.v1 is already live-registered in Schema Registry and a live BACKWARD-compatibility check confirmed tightening to non-nullable is rejected (TYPE_MISMATCH: reader STRING not compatible with writer NULL). A genuine future tightening requires a new invoice.generated.v2 event version per ADR-009/ADR-019's immutability rule, not a v1 mutation -- see docs/tasks/sprint-14-testing-and-hardening/ 14.5-avro-schema-governance-ruling.md for the full compatibility-check evidence."

This states the real reason (always-populated by the real producer, kept nullable purely because of a tested, rejected BACKWARD-compatibility tightening attempt, with the exact rejection evidence already recorded above in this document) instead of the plausible-but-untrue "nullable for account-level invoices" business claim that risked a future engineer re-attempting the same already-rejected fix. Closed.

Finding 2 (LOW/escalation, code-review) — RULED: not a violation; ADR-018 amended with an

explicit carve-out

Read ADR-018 directly (not just code-review's characterization) before ruling.

Ruling: (a) — this dependency pattern is acceptable as-is. platform-event-contracts is a pure schema/contract-definitions module: no AutoConfiguration, no Spring beans, no runtime behavior injected into a consuming service, nothing that leaks infrastructure into business code — the three harms ADR-018's Context section states the Dependency Rule exists to prevent. It is meaningfully different in kind from the internal platform modules (platform-core, platform-autoconfigure) that rule was written to stop services from coupling to directly: those carry business logic and wiring a service would otherwise reimplement or hand-configure; platform-event-contracts carries only Avro-generated data-carrier classes (the moral equivalent of a generated-DTO/protobuf-stubs module) plus, as of this feature, a small test-support assertion helper. Depending on it directly — compile-scope where a service builds/deserializes the canonical event type, test-scope (+ test-jar) where a contract test asserts against the canonical schema — does not create the coupling ADR-018 targets.

This is also not a new pattern invented under cover of this feature: 4 services (usage-service, billing-service, notification-service, ticket-service) already depended on it directly before Feature 14.5, 2 of them (billing-service, usage-service) at compile scope — in shipped production code, not just tests — and that had never been flagged through however many prior review passes. Phase 6 extended the identical, already-accepted pattern, in test-scope only, to 6 more services so every producing service's contract test could assert against the one real canonical schema instead of a hand-maintained local copy. Event-integration's unilateral read in phase 6 reached the correct conclusion; the process gap code-review correctly flagged (deciding a genuine ADR-018 boundary question without escalation, unlike how this same feature escalated the OrderItemPayload and invoice-generated.avsc questions) is closed by this ruling existing at all, going forward.

Amendment applied, not just an addendum here — per the same discipline used for ADR-019 earlier in this feature, ADR-018 itself now carries an explicit "Amendment (2026-07-08)" section (architecture/adr/ADR-018-platform-starter-dependency-model.md) stating: (A1) the Dependency Rule targets runtime-infrastructure coupling, not contract-definition modules; (A2) direct dependency on platform-event-contracts specifically (compile- or test-scope, including its test-jar) is compliant, scoped to that one module — platform-core/platform-autoconfigure/other internal modules remain fully subject to the unscoped rule; (A3) the basis (an existing, pre-14.5 pattern being ratified, not invented); (A4) what would make this stop qualifying (the module gaining AutoConfiguration or injected runtime behavior). This closes the question for good — a future agent hitting this same dependency will find a ruled, amended ADR, not a third unescalated reading or a fourth escalation of the same point. Finding 2 closed.

Final determination: Feature 14.5 is DONE

Evidence considered: 638/638 reactor tests green across all 18 modules (phase 7); the acceptance suite's one failure (NewSubscriberOnboardingAcceptanceIT's MSISDN-format assertion) independently root-caused to a pre-existing, out-of-scope MSISDN-pool-exhaustion artifact in this session's long-lived Docker stack (a live, out-of-migration DB top-up block), confirmed via direct git diff/grep to be untouched by any Feature 14.5 change or any other change this session — a fresh environment seeded only from V2__msisdn_pool_seed.sql would not hit it; code-review's APPROVE verdict with both findings now resolved above; the full 8-phase execution trail (ADR-019 amendment, 33/33 canonical schemas reconciled/authored and live-registered, the order.created.v1/OrderItemPayload inline-record resolution, the new type-and-nullability-aware AvroContractAssertions compat tooling with a proven catch-then-revert cycle, event-catalog.md updated, CI wiring confirmed/extended in ci.yml/acceptance.yml with its one honestly-documented residual gap — no CI registry with cross-run persisted history, correctly deferred as a distinct future CI-infrastructure decision, not part of this feature's scope).

Feature 14.5 (Avro Schema Governance Reconciliation) is DONE. The MSISDN-pool-exhaustion finding and the CI persisted-history registry gap are both real, both correctly out of scope here, and both explicitly handed off (to devops/domain-engineer and to a future CI-infrastructure ticket respectively) — neither blocks this feature's own closure.

Sprint 14 rollup at this point

  • 14.1 Acceptance and End-to-End Testing — DONE
  • 14.2 Security Hardening — DONE
  • 14.3 Performance Validation — DONE
  • 14.4 Identity-to-Customer Linkage — not DONE. Code-complete and individually verified per service, but the full end-to-end proof is not closed: a real, fresh JWT actually carrying the customerId claim through a genuine self-registration was never proven, blocked specifically by the realm's User Profile unmanagedAttributePolicy gap (a persistent, security-adjacent Keycloak realm-config change, correctly not applied without its own authorization). Tracked as a narrow, precisely-scoped follow-up in 14.1.1-identity-linkage-gap-ruling.md Step 7 — not a design gap, a single remaining authorization-gated configuration change plus its end-to-end proof.
  • 14.5 Avro Schema Governance Reconciliation — DONE (this sign-off).

Sprint 14 final status: 4 of 5 features DONE (14.1/14.2/14.3/14.5); 14.4 remains the one open, narrowly-scoped item (Keycloak realm User Profile customer_id attribute configuration, plus the end-to-end JWT-claim proof it unblocks). Sprint 14 is not fully DONE until 14.4 closes, but nothing about 14.4's remaining gap is architectural or in dispute — it is a single authorization-gated infrastructure change away from completion.