14.5 — Avro Schema Governance Reconciliation (tech-lead ruling)¶
Status: RULED, phases 1-8 of 8 complete. code-review's ADR-019-compliance pass (below) is DONE, APPROVE WITH ONE FOLLOW-UP (non-blocking); tech-lead's final sign-off remains. Logged 2026-07-07. Owning ADR:
architecture/adr/ADR-019-event-contract-and-schema-governance.md(see its "Amendment (2026-07-07)" section, which this document expands on with the full evidence and execution plan).
How this was found¶
While auditing event-contract coverage as a follow-up to 14.1.2 (contract tests), it became clear the
canonical schema directory (platform/platform-event-contracts/src/main/avro/) had never been checked
against what services actually publish. Every outboxService.publish(<aggregate>, <id>, "<event>.v1",
<payload>) call site under microservices/**/src/main/java was cross-referenced against the .avsc
files in the canonical directory, against each service's own src/test/resources/avro/ snapshots (used
by that service's contract test, e.g. NotificationEventContractTest), and against
docs/architecture/event-catalog.md's event registry.
Ruling on direction — rejected alternative, and why¶
Rejected: force every service onto Avro binary wire encoding to "truly" comply with ADR-019. This would be a large, high-risk migration (every producer/consumer, the outbox writer, Debezium's JSON converter config) to chase a form of compliance the ADR never actually required. ADR-009's outbox-as- plain-JSON decision is established, working, and separately verified this session (Feature 14.5 does not reopen it). The real defect is not "wrong wire format" — it is "no tooling ever proved the schema file matches the real payload." Fixing that is additive and low-risk; re-encoding the wire format is neither.
Ruled correct: reconcile schemas to the real JSON shape, close the missing-schema gap, fix the naming
violation, and add the tooling that prevents this drift from recurring. Full rationale in the ADR-019
amendment (architecture/adr/ADR-019-event-contract-and-schema-governance.md, "Amendment
(2026-07-07)", points A1-A5). This document is the durable execution reference for the phases that
amendment presupposes.
Verified finding — 14 real event types, not 13, currently unregistered¶
Method: grep -rhoE '"[a-z]+\.[a-z-]+\.v[0-9]+"' microservices --include="*.java" for every literal
event-type string, filtered to call sites outside /test/, cross-referenced against
platform/platform-event-contracts/src/main/avro/*.avsc (18 files present today, including
EventEnvelope.avsc which is the envelope wrapper, not an event) and against each service's
src/test/resources/avro/*.avsc snapshots.
Correction note: this ruling was first summarized verbally as "13 missing schemas." Re-verifying
against the actual codebase for this document turned up a 14th — user.deleted.v1 (identity-service),
sibling to user.created.v1, same root cause, previously missed. Per the "verify before done" rule,
the count below is the corrected, evidenced one; treat 14, not 13, as authoritative.
| # | Event | Producer | Test-local .avsc snapshot already exists? |
Canonical schema exists? |
|---|---|---|---|---|
| 1 | order.cancelled.v1 |
order-service | yes — order-service/src/test/resources/avro/order-cancelled.avsc |
NO |
| 2 | payment.failed.v1 |
payment-service | yes — payment-service/src/test/resources/avro/payment-failed.avsc |
NO |
| 3 | payment.refunded.v1 |
payment-service | yes — payment-service/src/test/resources/avro/payment-refunded.avsc |
NO |
| 4 | tariff.created.v1 |
product-catalog-service | yes — product-catalog-service/src/test/resources/avro/tariff-created.avsc |
NO |
| 5 | tariff.price-changed.v1 |
product-catalog-service | yes — product-catalog-service/src/test/resources/avro/tariff-price-changed.avsc |
NO |
| 6 | ticket.opened.v1 |
ticket-service | yes — ticket-service/src/test/resources/avro/ticket-opened.avsc |
NO |
| 7 | ticket.assigned.v1 |
ticket-service | yes — ticket-service/src/test/resources/avro/ticket-assigned.avsc |
NO |
| 8 | ticket.resolved.v1 |
ticket-service | yes — ticket-service/src/test/resources/avro/ticket-resolved.avsc |
NO |
| 9 | ticket.sla-breached.v1 |
ticket-service | yes — ticket-service/src/test/resources/avro/ticket-sla-breached.avsc |
NO |
| 10 | invoice.paid.v1 |
billing-service | yes — billing-service/src/test/resources/avro/invoice-paid.avsc |
NO |
| 11 | invoice.overdue.v1 |
billing-service | yes — billing-service/src/test/resources/avro/invoice-overdue.avsc |
NO |
| 12 | notification.dispatched.v1 |
notification-service | yes — notification-service/src/test/resources/avro/notification-dispatched.avsc |
NO |
| 13 | user.created.v1 |
identity-service | no — no avro test-resources directory exists for identity-service at all |
NO |
| 14 | user.deleted.v1 |
identity-service | no — same | NO |
Additional, independently confirmed observations (not new missing schemas, but part of the same reconciliation):
docs/architecture/event-catalog.md's event registry never listsuser.created.v1oruser.deleted.v1at all — these two are undocumented in the catalog, not just unregistered as schemas. Phase 7 (qa) must add both rows.order.confirmed.v1is correctly excluded from this list — the catalog itself documents it as "deferred; not produced in the MVP," confirmed by no matchingoutboxService.publishcall site.customer.profile-updated.v1andsubscription.cancelled.v1appear only insidenotification-service's own test fixtures (DomainEventNotificationConsumerTest), never in any producing service's main source. These are NOT real, currently-produced events; do not add canonical schemas for them as part of this feature. If they are meant to exist, that is a separate, new-feature decision (test fixture referencing a not-yet-built event), out of scope here — flag toqato either align the fixture to a real event name or delete the dead scenario, but do not silently invent a schema to match a test's fiction.demoitem.created.v1(reference-service) is the ADR-017 template/reference artifact's own demo event, not a real service. No canonical schema is required for it under this feature; leave as is.cdr.recorded.v1already has a canonical schema (cdr-recorded.avsc) — correction (architecture, phase 2 re-verification): this claim was wrong. The real wire shape (usage-service'sCdrRecordedEventConsumer.CdrPayload, matching the only real producer,acceptance-tests'CdrEventProducer) typesoccurredAtas an ISO-8601 string, but the canonical schema types it as{"type":"long","logicalType":"timestamp-millis"}. This is genuine, previously-uncaught drift requiring reconciliation — see the Architecture Diff Spec below, section A row 14. Treat as action-required, not "no action needed."
Naming-convention fix¶
EventEnvelope.avsc violates the kebab-case-filename convention this ADR now states explicitly (see
ADR-019 amendment, point A5). Rename to event-envelope.avsc. The Avro record's own "name" field
stays EventEnvelope (PascalCase, drives Java class generation) — only the filename changes. The
avro-maven-plugin configuration in platform/platform-event-contracts/pom.xml has a
string-replacement property, <event.envelope.v1>${project.basedir}/src/main/avro/EventEnvelope.avsc
</event.envelope.v1> (around line 77), that references the file by its literal old path — this MUST
be updated in the same commit as the rename, or the Avro codegen step breaks.
Execution order (8 phases) — owners and exit criteria¶
- tech-lead (this phase — DONE). ADR-019 amendment (
architecture/adr/ADR-019-event-contract- and-schema-governance.md) plus this tracking document. Exit criteria: amendment merged, unambiguous on JSON-shape-not-wire-bytes, ADR-009 unchanged,.avscfiles remain canonical source of truth, naming convention stated, this document exists as the durable reference. Sprint tracking (docs/tasks/STATUS.md, sprint-14README.md) updated to IN PROGRESS. - architecture. Validate the reconciliation strategy per event before any file is touched: for
each of the 12 rows with an existing test-local
.avscsnapshot, confirm that snapshot itself is still an accurate description of the producing service's real, current event-payload class (do not assume the test-local copy is trustworthy just because it exists — it was never cross-checked either). Foruser.created.v1/user.deleted.v1(no snapshot at all), review and approve the new schema's field list, types, and nullability against identity-service's realUserCreatedV1/DeleteUserCommandHandlerpayload before it is authored. Sign off on theEventEnvelope.avsc->event-envelope.avscrename plan (file rename +pom.xmlproperty update, no record-name change). Exit criteria: written sign-off (PR review or an addendum to this file) on all 14 target shapes and the rename plan. - event-integration. For the 12 rows with a test-local snapshot: promote/reconcile each into
platform/platform-event-contracts/src/main/avro/under the correct kebab-case canonical filename (diff against the real Java payload class first, per architecture's phase-2 sign-off — do not copy-paste the test-local file blindly). Foruser.created.v1/user.deleted.v1: author new canonical schemas from scratch. Register all in Schema Registry per this ADR's existing Schema Registry Rule and Compatibility Rule (backward-compatible; these are net-new registrations, not breaking changes to existing consumers). Addavro-maven-pluginstring-replacement properties for all 14 new schemas inplatform/platform-event-contracts/pom.xml, mirroring the existing entries. Exit criteria: 14 new canonical.avscfiles present, all registered,mvn generate-sourcesonplatform-event-contractssucceeds. - event-integration. Rename
EventEnvelope.avsc->event-envelope.avsc; update the corresponding property inplatform/platform-event-contracts/pom.xml. Exit criteria: rename committed,mvn generate-sourcesstill succeeds, generatedEventEnvelopeJava class name unchanged. - domain-engineer (one pass per producing service: order-service, payment-service,
product-catalog-service, ticket-service, billing-service, notification-service, identity-service).
For each of the 14 events, confirm the service's real outbox-published JSON payload matches the new
canonical schema field-for-field (name, type, nullability). Given the root cause (schemas and real
payloads were never cross-checked), expect to find last-mile drift; fix it in the service's
event-payload class to match the schema unless the ADR's evolution rules require the opposite
(schema is wrong, payload is right) — escalate any such conflict to
architecture/tech-leadrather than deciding unilaterally which side is authoritative. Exit criteria: no field-shape mismatch remains for any of the 14 events. - event-integration (with
qasupport). Extend the schema-compatibility test tooling so every one of all 32 canonical schemas (18 pre-existing + 14 new) has a matching per-service contract test following the existing*EventContractTestpattern (e.g.NotificationEventContractTest) — this is the concrete tooling gap that let the drift happen silently in the first place; it must not be a one-time manual reconciliation. Exit criteria: a producing service's payload class and its canonical schema diverging fails the build (prove it with one deliberately-broken test run, then revert). - qa. Run the full contract-test suite plus the acceptance suite (AC-01/02/03) to confirm no
regression. Add the two missing rows (
user.created.v1,user.deleted.v1) todocs/architecture/event-catalog.md's event registry table, and add a Schema Evolution Log entry documenting all 14 additions with today's date. Exit criteria: green contract-test and acceptance runs; event-catalog.md fully reflects the 32-event registered set. - devops + code-review, then tech-lead sign-off.
devopsconfirms the Schema Registry compatibility-mode CI check runs against all 32 canonical schemas and fails the build on an unregistered or incompatible event type going forward (closing the "never registered" failure mode for good, not just for these 14).code-reviewperforms a final ADR-019-compliance pass (naming convention, JSON-shape framing, no accidental wire-format change) before merge.tech-leadgives final sign-off closing Feature 14.5 as DONE.
Non-goals (explicit, do not implicitly expand scope)¶
- No change to the outbox's plain-JSON publishing (ADR-009 stands).
- No change to any event's actual field shape beyond what phase 5 finds as genuine drift — this is a reconciliation of documentation/registration, not a redesign of any event.
- No schema for
customer.profile-updated.v1/subscription.cancelled.v1(not real, currently- produced events) ordemoitem.created.v1(template artifact) under this feature. - No change to
order.confirmed.v1's deferred status.
Architecture Diff Spec (2026-07-07)¶
Author: architecture agent, phase 2 of the 8-phase execution order. Every row below was produced by
reading the real, currently-shipping Java event/payload class (or Map.of(...) outbox call) for each
event, not by trusting this document's earlier table or any test-local .avsc snapshot at face value.
Where a test-local snapshot was already accurate it is noted as corroboration, not as the source of
truth.
Two corrections to this document's earlier phase-1 content, found during phase-2 re-verification (per
"verify before done"): (1) the phase-1 table above already reflects the corrected 14-item missing-
schema list (user.created.v1/user.deleted.v1, not order-item) — order-item is a separate,
15th artifact: a nested Avro record embedded in order-created.v1, not an independent Kafka-registered
subject. (2) cdr.recorded.v1's "needs no action" claim (originally line 81, corrected above) was
wrong — real drift found, see section A row 14.
No single "platform timestamp convention" exists in the real code — do not normalize globally. Two conventions genuinely coexist by service: ISO-8601 string (order, payment, usage, product-catalog, ticket, identity-service) and epoch-millis long (customer, subscription, billing-service). Reconciliation matches each producer's own real type. This is a hygiene note for a future ADR, not something changed under this feature.
A. Reconciliation of the 18 pre-existing canonical schemas¶
| # | Canonical file | Verdict | Diff |
|---|---|---|---|
| 1 | order-created.avsc |
RECONCILE (major) | Real: OrderCreatedEvent (orderId, customerId, items: List<OrderItemPayload>, totalAmount, idempotencyKey, occurredAt). Corroborated by order-service's own accurate test snapshot. Changes: (1) add items: {"type":"array","items":"OrderItemPayload"}; (2) add idempotencyKey: string; (3) remove currency (not in real class); (4) rename+retype createdAt (long,timestamp-millis) -> occurredAt (string). |
| 2 | payment-completed.avsc |
RECONCILE | Real: PaymentCompletedEvent (paymentId, orderId, customerId, amount, invoiceId, occurredAt). Changes: (1) remove currency; (2) retype orderId from nullable to plain string (always set, @NotNull at the command); (3) rename+retype completedAt(long) -> occurredAt(string); (4) keep invoiceId nullable (confirmed genuinely nullable). |
| 3 | invoice-generated.avsc |
NO CHANGE | Verified against BillRunBatchProcessor's record: exact field-for-field, type-for-type match (billing-service genuinely uses epoch-millis). |
| 4 | customer-registered.avsc |
NO CHANGE | Matches CustomerRegisteredV1 exactly. |
| 5 | customer-updated.avsc |
NO CHANGE | Matches CustomerUpdatedV1 exactly. |
| 6 | customer-kyc-approved.avsc |
NO CHANGE | Matches CustomerKycApprovedV1 exactly. |
| 7 | customer-kyc-rejected.avsc |
NO CHANGE | Matches CustomerKycRejectedV1 exactly. |
| 8 | msisdn-allocated.avsc |
NO CHANGE | Matches MsisdnAllocatedV1 exactly. |
| 9 | msisdn-released.avsc |
NO CHANGE | Matches MsisdnReleasedV1 exactly. |
| 10 | subscription-activated.avsc |
NO CHANGE | Matches SubscriptionActivatedV1 exactly; both producers construct the same record type, no cross-producer drift possible. |
| 11 | subscription-activation-failed.avsc |
NO CHANGE | Matches SubscriptionActivationFailedV1 exactly. |
| 12 | subscription-suspended.avsc |
NO CHANGE | Matches SubscriptionSuspendedV1 exactly; both producers use the same record type. |
| 13 | subscription-terminated.avsc |
NO CHANGE | Matches SubscriptionTerminatedV1 exactly. |
| 14 | cdr-recorded.avsc |
RECONCILE — corrects this doc's earlier "no action needed" claim | Consumed external contract (produced by acceptance-tests' CdrEventProducer, consumed by usage-service's CdrRecordedEventConsumer.CdrPayload). Real shape: subscriptionId, type, quantity(long), occurredAt(String, ISO-8601), cdrRef. Change: retype occurredAt from long timestamp-millis to plain string. Non-blocking note: type is a closed Avro enum today; real payload types it as plain String (consumer defensively .toUpperCase()s) — under JSON-shape governance this distinction doesn't matter on the wire; left to event-integration's discretion. |
| 15 | usage-aggregated.avsc |
RECONCILE | Real: UsageAggregatedEvent (subscriptionId, periodStart(String), periodEnd(String), voiceOverageSeconds(long), smsOverageCount(long), dataOverageKb(long), aggregatedAt(String)) — corroborated independently by billing-service's own consumer DTO. Change: retype periodStart, periodEnd, aggregatedAt from long timestamp-millis to string. |
| 16 | usage-recorded.avsc |
RECONCILE | Real: UsageRecordedEvent (usageRecordId, subscriptionId, type(String), quantity(long), overage(boolean), recordedAt(String)). Change: retype recordedAt from long to string. |
| 17 | quota-exceeded.avsc |
RECONCILE | Real: QuotaExceededEvent (subscriptionId, quotaId, usageType, customerId(nullable), exceededAt(String)). Change: retype exceededAt from long to string. |
| 18 | quota-threshold-reached.avsc |
RECONCILE | Real: QuotaThresholdReachedEvent (subscriptionId, quotaId, usageType, customerId(nullable), reachedAt(String)). Change: retype reachedAt from long to string. |
B. New nested type required by row 1 (not a top-level Kafka subject)¶
order-item.avsc — record name OrderItemPayload, namespace com.telco.platform.events.order.
Fields (from OrderCreatedEvent.OrderItemPayload, corroborated by order-service's own
src/test/resources/avro/order-item.avsc): tariffId(string), tariffName(string),
unitPrice(bytes/decimal precision=19 scale=4), quantity(int). Referenced only from
order-created.avsc's items field. order-cancelled.v1 does NOT carry items — do not reference it
there. Embedded/nested per tech-lead's ruling: no independent Schema Registry subject, no independent
contract-test row beyond what already covers order-created.v1.
C. Field specs for the 14 missing schemas¶
All new files go under platform/platform-event-contracts/src/main/avro/, kebab-case filenames.
| # | Event | File | Fields | Source of truth |
|---|---|---|---|---|
| 1 | order.cancelled.v1 |
order-cancelled.avsc |
orderId: string; customerId: string; reason: ["null","string"] default null; occurredAt: string |
OrderCancelledEvent. Correction to the existing test-local snapshot: both it and order-service's own field-name-only contract test type reason as non-nullable, but neither CancelOrderCommand.reason nor CompensateOrderCommand.reason carries @NotNull/@NotBlank (handler does command.reason() == null ? null : ...) — genuinely nullable at runtime. Do not copy the test-local file verbatim; apply this nullability fix when promoting. |
| 2 | payment.failed.v1 |
payment-failed.avsc |
paymentId: string; orderId: string; customerId: string; amount: decimal(19,4); invoiceId: ["null","string"] default null; reason: string; occurredAt: string |
PaymentFailedEvent. Test-local snapshot verified accurate — promote as-is. |
| 3 | payment.refunded.v1 |
payment-refunded.avsc |
paymentId: string; orderId: string; customerId: string; amount: decimal(19,4); reason: string (non-nullable,@NotBlank); occurredAt: string |
PaymentRefundedEvent. Test-local snapshot verified accurate — promote as-is. |
| 4 | invoice.paid.v1 |
invoice-paid.avsc |
invoiceId: string; customerId: string; paidAt: long, timestamp-millis |
Package-private InvoicePaidEvent in MarkInvoicePaidCommandHandler. Test-local snapshot verified accurate — promote as-is. |
| 5 | invoice.overdue.v1 |
invoice-overdue.avsc |
invoiceId: string; customerId: string; dueDate: string (ISO-8601 date); detectedAt: long, timestamp-millis |
Package-private InvoiceOverdueEvent in MarkInvoicesOverdueCommandHandler. Test-local snapshot verified accurate — promote as-is. |
| 6 | notification.dispatched.v1 |
notification-dispatched.avsc |
notificationId: string; userId: string; channel: string; templateCode: string |
NotificationService.dispatch(...). Test-local snapshot verified accurate — promote as-is. |
| 7 | ticket.opened.v1 |
ticket-opened.avsc |
ticketId: string; customerId: string; category: string; priority: string; subject: string; assignedTeam: string; slaDueAt: string (ISO-8601) |
OpenTicketCommandHandler. Test-local snapshot verified accurate — promote as-is. |
| 8 | ticket.assigned.v1 |
ticket-assigned.avsc |
ticketId: string; customerId: string; assignedTeam: string |
Two call sites (OpenTicketCommandHandler, AssignTicketCommandHandler), identical shape — no cross-producer drift. Test-local snapshot verified accurate — promote as-is. |
| 9 | ticket.resolved.v1 |
ticket-resolved.avsc |
ticketId: string; customerId: string; resolvedAt: string (ISO-8601) |
ResolveTicketCommandHandler. Test-local snapshot verified accurate — promote as-is. |
| 10 | ticket.sla-breached.v1 |
ticket-sla-breached.avsc |
ticketId: string; customerId: string; category: string; priority: string; slaDueAt: string (ISO-8601) |
DetectSlaBreachCommandHandler. Test-local snapshot verified accurate — promote as-is. |
| 11 | tariff.created.v1 |
tariff-created.avsc |
tariffId: string; code: string; name: string; type: string; monthlyFee: decimal(19,4); currency: string; effectiveFrom: string (ISO-8601); createdAt: string (ISO-8601) |
TariffCreatedEvent. Test-local snapshot verified accurate — promote as-is. |
| 12 | tariff.price-changed.v1 |
tariff-price-changed.avsc |
tariffId: string; code: string; oldMonthlyFee: decimal(19,4); newMonthlyFee: decimal(19,4); currency: string; newVersion: int; changedAt: string (ISO-8601) |
TariffPriceChangedEvent. Test-local snapshot verified accurate — promote as-is. |
| 13 | user.created.v1 |
user-created.avsc |
userId: string; username: string; email: string; createdAt: string (ISO-8601) |
UserCreatedV1. No test-local snapshot existed (identity-service has no src/test/resources/avro/ at all) — authored fresh. |
| 14 | user.deleted.v1 |
user-deleted.avsc |
userId: string; deletedAt: string (ISO-8601) |
UserDeletedV1. Authored fresh, same reasoning as row 13. |
Namespace for rows 13-14: com.telco.platform.events.identity (event-integration's naming call, no
existing identity-service convention to match).
D. Cross-service compatibility check — no disagreement found, nothing to escalate¶
Checked every case where more than one service produces the same event type or models a shared nested
type: OrderItemPayload (order-service only; subscription-service's OrderItemClientResponse is an
unrelated REST-client DTO, out of scope); subscription.activated.v1/subscription.suspended.v1/
order.cancelled.v1/ticket.assigned.v1 (each has two producer call sites, all construct the
identical Java record/Map shape — compiler-enforced, no drift possible); payment.completed.v1
(consumed independently by order-service and subscription-service, their local consumer DTOs are
field-for-field identical); billing-service's per-event consumer Payload records correctly type
subscription timestamps as Long and usage timestamps as String, matching each producer's real
convention — independent corroboration of section A rows 15-18, not a conflict. No case requires
tech-lead escalation.
Aside, not a schema-governance defect, flagged for qa/domain-engineer awareness only, not actioned
here: DomainEventNotificationConsumer.onSubscriptionEvent reads a customerName field off
subscription.activated.v1 payloads via payload.getOrDefault("customerName", customerId), but no
producer ever sets customerName. The getOrDefault fallback means this never breaks at runtime
(silently always uses customerId) — not an Avro contract violation, out of scope for this feature.
E. Naming-convention normalization for the 14 promoted/authored schemas¶
The 18 pre-existing canonical schemas all use Avro record name of the form XxxV1. The 12
test-local snapshots being promoted use each service's own class-name convention (XxxEvent in most
cases). Recommendation (non-blocking, event-integration may override with reasoning): normalize the
Avro record name to XxxV1 for all 14 promoted/authored schemas, for consistency with the other 18.
This is registry/codegen naming only — does not change the JSON wire shape, does not require any
Java-side rename.
F. Re-confirmation of the "explicitly excluded" list — verified independently¶
Re-ran the full event-type grep unfiltered by directory and cross-checked every hit: customer.profile-
updated.v1 and subscription.cancelled.v1 appear only in DomainEventNotificationConsumerTest, zero
occurrences in any src/main/java — confirmed test fiction, correctly excluded. demoitem.created.v1
appears only in reference-service's own main+test source — confirmed template artifact, correctly
excluded. order.confirmed.v1 has zero occurrences anywhere (main or test) — confirmed correctly
deferred. All three exclusions stand; nothing was silently missed.
Summary of what phase 3 (event-integration) must do: reconcile the 7 real-diff rows in section A
(order-created, payment-completed, cdr-recorded, usage-aggregated, usage-recorded,
quota-exceeded, quota-threshold-reached), leave the other 11 of the 18 untouched, author the 14 new
files in section C plus the nested order-item.avsc in section B, apply the naming normalization in
section E, and execute the EventEnvelope.avsc -> event-envelope.avsc rename plus its pom.xml
property update.
Phases 3 and 4 execution log (event-integration, 2026-07-07) - DONE, with two findings¶
Before writing any file, re-verified a sample of the diff spec's rows against the real Java source
(per "verify before done"): OrderCreatedEvent/OrderCancelledEvent (order-service),
PaymentCompletedEvent/PaymentFailedEvent/PaymentRefundedEvent (payment-service),
CancelOrderCommand/CompensateOrderCommand (confirming reason really carries no
@NotNull/@NotBlank), UserCreatedV1/UserDeletedV1 (identity-service), UsageAggregatedEvent and
CdrRecordedEventConsumer.CdrPayload (usage-service), TariffCreatedEvent/TariffPriceChangedEvent
(product-catalog-service), NotificationService.dispatch(...) and the four ticket-service handlers.
All matched the diff spec's stated shapes exactly, with one exception below.
Finding 1 - diff spec under-stated row 2's change list (fixed, not just applied verbatim). Section
A row 2 (payment-completed.avsc)'s "Real:" field list states PaymentCompletedEvent carries
customerId, but the row's four enumerated change bullets never say "add customerId" - and the
pre-existing canonical schema had no customerId field at all. Re-reading
PaymentCompletedEvent.java directly confirmed customerId is a real field (paymentId, orderId,
customerId, amount, invoiceId, occurredAt). Added customerId: string to payment-completed.avsc
in addition to the four explicitly-listed changes, since the reconciliation's entire purpose is
matching the real payload, not just the bullet list. Flagging this the same way tech-lead/architecture
flagged each other's gaps this session - re-verification discipline continued, not broken.
Finding 2 - Confluent Schema Registry cannot validate order.created.v1 standalone; the nested-type
plan (section B) works for Java codegen but not for live registry validation without either inlining or
an independent subject. A local Schema Registry container was already running
(telco-schema-registry, confirmed via docker ps), so beyond the required offline
mvn generate-sources -Dschema.registry.skip=true check (green, see below), the live-registry path was
also tried per this phase's optional instruction. Result: 32 of the 33 real subjects (everything except
order.created.v1) register cleanly against the live, empty registry (proven via direct HTTP POST to
/subjects/{subject}/versions for each, then deleted - the registry was left exactly as found,
empty). order.created.v1 fails to parse (SchemaParseException: Undefined name: "OrderItemPayload")
because kafka-schema-registry-maven-plugin's test-compatibility/register goals - and the real
Schema Registry HTTP API itself - parse each subject's raw .avsc text standalone, with no visibility
into avro-maven-plugin's own <imports> mechanism (which is what makes the split-file
order-created.avsc + order-item.avsc pair work for Java codegen, confirmed in the build log below).
Confirmed by direct experiment: registering OrderItemPayload under its own throwaway subject and
referencing it from order-created.avsc via Confluent's references field registers successfully -
but that requires OrderItemPayload to exist as its own Schema Registry subject, which is exactly what
this ruling's section B explicitly forbids ("no independent Schema Registry subject"). Given that
direct conflict, this was not resolved unilaterally (inlining the nested record would also require
either duplicating it out of order-item.avsc or restructuring how avro-maven-plugin sees it, and
either path is itself a schema-design call, not a mechanical one). Leaving the schema exactly as
section B specifies (bare-name reference in order-created.avsc, full definition in order-item.avsc,
no independent subject) and escalating this as a real, unresolved gap for architecture/tech-lead:
whichever way it is decided (inline the nested type into order-created.avsc, or accept a
reference-only "non-Kafka" registry subject for OrderItemPayload, or something else), it needs a
ruling before phase 8 (devops's live CI compatibility gate) can actually enforce order.created.v1
against a real registry. Per this task's own scope, the required exit criterion - mvn generate-sources
succeeding offline - is unaffected and green; this is a live-registry-only gap.
Files changed:
- Reconciled (section A): order-created.avsc, payment-completed.avsc (plus the customerId fix
above), cdr-recorded.avsc, usage-aggregated.avsc, usage-recorded.avsc, quota-exceeded.avsc,
quota-threshold-reached.avsc.
- Authored (section B): order-item.avsc.
- Authored (section C, all 14, XxxV1 naming per section E): order-cancelled.avsc,
payment-failed.avsc, payment-refunded.avsc, invoice-paid.avsc, invoice-overdue.avsc,
notification-dispatched.avsc, ticket-opened.avsc, ticket-assigned.avsc, ticket-resolved.avsc,
ticket-sla-breached.avsc, tariff-created.avsc, tariff-price-changed.avsc, user-created.avsc,
user-deleted.avsc. invoice-paid.avsc/invoice-overdue.avsc use the
com.telco.platform.events.invoice namespace, matching the existing invoice-generated.avsc
convention (domain-based, not producing-service-based), not the events.billing guess a literal
service-name mapping would suggest.
- Renamed (phase 4): EventEnvelope.avsc -> event-envelope.avsc (record name EventEnvelope
unchanged).
- platform/platform-event-contracts/pom.xml: event.envelope.v1 property repointed to
event-envelope.avsc; added an <imports> entry for order-item.avsc to the avro-maven-plugin
config (required for cross-file type resolution at Java-codegen time); added 14 new <subjects>
entries mirroring the existing format. order-item.avsc is deliberately absent from <subjects> -
not an independent Kafka-registered event, per the ruling.
Build verification: cd platform/platform-event-contracts && mvn generate-sources
-Dschema.registry.skip=true - BUILD SUCCESS, no errors. Generated
target/generated-sources/avro/ contains 35 .java files: all 32 canonical event classes (18
reconciled + 14 new), the renamed EventEnvelope class (unaffected by the filename change, still named
EventEnvelope), the nested OrderItemPayload class, and the pre-existing CdrType nested enum. Live
Schema Registry check (optional per this phase): 32/33 real subjects register cleanly; order.created.v1
does not, per Finding 2 above - escalated, not silently worked around.
Phases 3 and 4 (event-integration) are DONE, with Finding 2 flagged as an open item for
architecture/tech-lead before phase 8's live CI gate can cover order.created.v1. Phase 5
(domain-engineer, per-service last-mile drift reconciliation) is next.
Tech-lead ruling: order.created.v1 / OrderItemPayload registry conflict (2026-07-07) — RESOLVED, unblocks phase 5¶
Decision: Option 1, in its strict form. OrderItemPayload is inlined as a fully self-contained
nested record definition inside order-created.avsc. order-item.avsc is deleted, not kept as a
parallel doc/reference file. Option 2 (giving OrderItemPayload its own Schema Registry subject)
remains rejected — nothing in this exercise changes the original reasoning that it is not an
independently-consumed contract, and creating a subject for it would be modeling around a tooling
limitation, not a real domain boundary.
Why the "keep order-item.avsc too, as documentation" variant of option 1 is explicitly rejected
(not just unnecessary — build-breaking): verified in an isolated scratch copy of the module.
avro-maven-plugin's schema goal shares a single Schema.Parser/named-type table across its entire
sourceDirectory scan for one execution. Once order-created.avsc carries the type inline, having
order-item.avsc also present in src/main/avro — even with a byte-identical definition — makes the
plugin fail with Can't redefine: com.telco.platform.events.order.OrderItemPayload. Deleting
order-item.avsc and inlining the definition into order-created.avsc (with no <imports> entry at
all) builds clean and generates the identical OrderItemPayload.java nested class as before. The
now-self-contained order-created.avsc was also POSTed to the live telco-schema-registry container as
a throwaway subject and registered successfully (HTTP 200), then deleted, leaving the registry exactly
as event-integration left it (0 subjects) — proven both ways: local Java codegen and live Schema
Registry validation now agree on one schema document, with no split-brain.
Exact instructions for event-integration (file-by-file)¶
platform/platform-event-contracts/src/main/avro/order-created.avsc— replace theitemsfield'stype.itemsvalue (currently the bare string"OrderItemPayload") with the full inline record definition, verbatim from the currentorder-item.avsc(type: record,name: OrderItemPayload,namespace: com.telco.platform.events.order, samedoc, same four fieldstariffId/tariffName/unitPrice/quantitywith identical types/docs). Result:order-created.avscbecomes a single, fully self-contained schema document with no external reference.platform/platform-event-contracts/src/main/avro/order-item.avsc— delete this file entirely. Do not retain it undersrc/main/avro(or anywhereavro-maven-plugin'ssourceDirectoryscans it) even as a "reference-only" copy — that reproduces theCan't redefinefailure. Section B of this document already captures the nested shape in prose; that is sufficient documentation.platform/platform-event-contracts/pom.xml— remove the<imports>block under theavro-maven-pluginschemaexecution (the one pointing at${project.basedir}/src/main/avro/order-item.avsc). No change to<subjects>:order-item.avsc/OrderItemPayloadwas already correctly absent and stays absent.- No other files change.
microservices/order-service/src/test/resources/avro/order-item.avscandOrderEventSchemaCompatTest.javaare a separate, module-local test snapshot in a different Maven module — out of scope for this ruling, untouched.
Verification event-integration must reproduce for the record¶
cd platform/platform-event-contracts && mvn generate-sources -Dschema.registry.skip=true-> BUILD SUCCESS, still 35 generated.javafiles,OrderItemPayload.javagenerated as a nested class purely fromorder-created.avsc's inline definition.- With
telco-schema-registryrunning, registerorder.created.v1for real - now succeeds. All 33 of 33 real subjects register cleanly (not 32/33 as in the phase-3/4 log), closing Finding 2. Leave the registry exactly as found afterward, same discipline as the other 32 subjects.
This unblocks phase 5: domain-engineer can now validate every service's real payload — including
order-service's OrderCreatedEvent/OrderItemPayload — against one schema shape that is simultaneously
what local Java codegen produces and what is actually enforceable against a real Schema Registry in CI
(phase 8's gate). No more split between "works for avro-maven-plugin" and "works for Confluent."
Ruling applied (event-integration, 2026-07-07) — Finding 2 CLOSED¶
Applied exactly the 3 file changes the ruling specifies, no others:
platform/platform-event-contracts/src/main/avro/order-created.avsc— theitemsfield'stype.itemsis now the full inlineOrderItemPayloadrecord definition (verbatim fields/docs from the formerorder-item.avsc), not a bare-name reference.platform/platform-event-contracts/src/main/avro/order-item.avsc— deleted (it was untracked in git, so a plain file removal was used in place ofgit rm; net effect on the tree is identical — the file is gone and was never part of a prior commit).platform/platform-event-contracts/pom.xml— removed the<imports>block under theavro-maven-pluginschemaexecution that pointed atorder-item.avsc.<subjects>untouched,order-item/OrderItemPayloadcorrectly still absent from it.
microservices/order-service/src/test/resources/avro/order-item.avsc and
OrderEventSchemaCompatTest.java were not touched, per the ruling's scope note.
Verification:
- cd platform/platform-event-contracts && mvn generate-sources -Dschema.registry.skip=true —
BUILD SUCCESS, still 35 generated .java files; OrderItemPayload.java generated as a nested class
purely from order-created.avsc's inline definition (no <imports> involved).
- telco-schema-registry was running (docker ps confirmed). Registered all 33 real subjects for
real via mvn package -Pregister-schemas -Dschema.registry.skip=false (not a throwaway subject name
this time, per instruction) — every subject including order.created.v1 registered cleanly (id 67,
version 1), confirmed independently via GET /subjects returning exactly 33 entries. Left registered
in the live registry as instructed (not deleted afterward, unlike the prior phase-3/4 disposable
verification).
Finding 2 is now CLOSED: 33 of 33 real subjects register cleanly against a live Schema Registry, and
mvn generate-sources succeeds offline with the identical generated-class shape. Phase 5
(domain-engineer) can proceed against the now-unified schema.
Architecture ruling: invoice-generated.avsc subscriptionId nullability (phase 5 finding) — tested and REVERTED (2026-07-07)¶
Finding (billing-service, phase 5 pass). invoice-generated.avsc's subscriptionId field is typed
nullable (["null","string"], default: null), but the sole real producer
(BillRunBatchProcessor.InvoiceGeneratedEvent) always populates it from
SubscriberBillingRecord.subscriptionId, a DB column that is nullable = false, unique = true.
billing-service's own test-local snapshot already types this field non-nullable. No real consumer
relies on the nullable case.
Architecture's ruling. Tighten the field from ["null","string"]/default null to plain
non-nullable "string", matching the real, always-populated data. Ruling flagged a real,
unverified compatibility risk: invoice-generated.avsc is one of the 18 pre-existing schemas
already live-registered in telco-schema-registry (subject invoice.generated.v1, registered
during event-integration's earlier phase 3/4 pass). Under Confluent's default BACKWARD
compatibility mode, narrowing a nullable union to non-nullable is often rejected — Avro/Confluent
compatibility checking is structural (can a new-schema reader parse an old-schema-written record?),
not data-driven (it does not matter that no real message ever used the null case).
What was tested, in order:
1. Applied the exact edit: replaced subscriptionId's type: ["null","string"], default: null with
plain type: "string" (updated doc to explain the always-populated DB invariant) in
platform/platform-event-contracts/src/main/avro/invoice-generated.avsc.
2. cd platform/platform-event-contracts && mvn generate-sources -Dschema.registry.skip=true —
BUILD SUCCESS, still 35 generated .java files.
3. Confirmed telco-schema-registry was running (docker ps, container telco-schema-registry,
healthy) and confirmed the live registry's compatibility mode is the default BACKWARD
(GET /config -> {"compatibilityLevel":"BACKWARD"}; invoice.generated.v1 has no
subject-level override). Tested the tightened schema via
POST /compatibility/subjects/invoice.generated.v1/versions/latest?verbose=true — a
read-only compatibility check, no registration attempted.
Actual result: REJECTED. is_compatible: false, with the concrete error:
{errorType:'TYPE_MISMATCH', description:'The type (path '/') of a field in the new schema does not
match with the old schema', additionalInfo:'reader type: STRING not compatible with writer type: NULL'}
null value in that field position and the tightened reader schema cannot accept it —
structurally incompatible regardless of whether any real message ever actually carried null.
Final state left in place: REVERTED to nullable, per branch 4 of this task's instructions.
Per architecture's own ruling framing — do not override the compatibility mode, do not force
registration through, do not keep the change if rejected — invoice-generated.avsc was reverted
to its original nullable subscriptionId (["null","string"], default: null). Confirmed via
git diff returning empty (file byte-identical to the committed/tracked version) and a rebuild
(mvn generate-sources -Dschema.registry.skip=true, BUILD SUCCESS, still 35 generated classes).
The live registry was never mutated: invoice.generated.v1 still has exactly one version
(GET /subjects/invoice.generated.v1/versions -> [1], id: 60, unchanged), left exactly as
found.
Conclusion. "Documented looser than strictly necessary, but safe" is confirmed as the correct
final state for invoice.generated.v1. The field remains nullable in the schema even though the
real, current producer never emits null — this is intentional slack the schema retains precisely
because the subject is already live-registered elsewhere and this session's ADR-019 amendment does
not authorize breaking an existing Schema Registry BACKWARD-compatibility guarantee to chase a
tighter-than-necessary type. If a future, genuine need arises to enforce non-nullability at the
wire-contract level (not just documentation), the correct path is a new event version
(invoice.generated.v2) per ADR-009/ADR-019's immutability rule, not a mutation of the existing
v1 contract.
Phase 6 (event-integration, 2026-07-07) — DONE: type/nullability-aware compat tooling, re-pointed at the canonical contract¶
Root cause recap¶
Every pre-existing *EventSchemaCompatTest/*EventContractTest (order, payment, billing, usage,
tariff, customer, subscription/msisdn, ticket, notification) only ever compared Avro field names
(Set<String> reflection over Java record components, or over a captured runtime Map for the two
services that build Map.of(...) payloads) against each service's own local
src/test/resources/avro/*.avsc snapshot. Two failure modes followed directly from that: (1) type and
nullability drift (exactly the usage-service timestamp mismatches and the billing-service
subscriptionId nullability question this feature already found) passed silently, because two
same-named, differently-typed fields still satisfy a name-only Set comparison; (2) the local snapshot
was never cross-checked against the canonical file under platform-event-contracts, so the test only
proved "the Java class matches a hand-maintained copy," not "the Java class matches the governed
contract."
What was built¶
1. A shared, reusable type-and-nullability-aware checker: AvroContractAssertions.
- New file:
platform/platform-event-contracts/src/test/java/com/telco/platform/events/testsupport/ AvroContractAssertions.java. - Packaged as this module's test-jar (new
maven-jar-plugintest-jarexecution inplatform/platform-event-contracts/pom.xml), so every producing service adds it as one ordinary test-scope dependency instead of copy-pasting a checker eight times. - Two entry points, matching the two payload-construction styles that actually exist in this codebase:
assertRecordMatchesSchema(Schema, Class<?>)— reflects overRecordComponentdeclared types. Used by every service that publishes a typed Javarecord(order, payment, billing, usage, tariff, customer, subscription, msisdn, identity).assertPayloadMatchesSchema(Schema, Map<String,Object>)— checks the actual captured runtime value's type and observed nullness. Used by ticket-service and notification-service, which build the outbox payload inline asMap.of(...)with no static Java type to reflect on.canonicalSchema(String generatedClassName)— loads the realSchemaembedded in the Avro-generatedSpecificRecordclass (its staticgetClassSchema()), by fully qualified class name. This is what makes checking against the canonical, not a local copy, actually possible: the generated classes carry the full typedSchema(unions, logical types, nested records) directly, so no.avscresource needs to be shipped or parsed by hand.- Checks performed, per field: (a) name presence in both directions (existing behavior, kept); (b)
Avro type vs. Java type compatibility —
string→String;int→int/Integer;long(no logical type)→long/Long;long+timestamp-millis→long/Long/Instant;boolean→boolean/Boolean;bytes+decimal→BigDecimal;enum→String/Enum(this codebase's one enum,CdrType, is consumed as a plainString, matching the diff spec's non-blocking note);arrayofrecord→ a JavaList/Collectionwhose generic element type is itself a record, recursing into the nested schema (coversorder.created.v1's inlinedOrderItemPayloadautomatically, with no special-casing needed);record→ a Javarecord, recursing. (c) nullability, in both directions: - Avro non-nullable, Java signals it may be null (a boxed wrapper type —
Long/Integer/etc. instead of the primitive — orOptional<T>, or, for the runtime/map entry point, an actually-observednullvalue): FAIL. A real bug — the wire contract promises always-present, the producer can omit it. - Avro nullable (
["null", X]), Java can never be null (a primitive): WARN only (printed viaSystem.out, does not fail the build). Slack the schema retains that the current producer does not need — looser than strictly necessary, but safe (matches the exactinvoice.generated.v1ruling above). - Everything else — in particular a plain reference type such as
StringorBigDecimal, which the Java type system cannot prove non-null either way without an annotation this codebase does not use — is undecidable from reflection alone and is not asserted in either direction. This is a deliberate, honest limitation, not a gap papered over: it is exactly whyinvoice-generated.avsc's nullable-but-always-populatedsubscriptionId(Stringin the Java record) correctly produces neither a FAIL nor a WARN. - Self-tested independently of any microservice:
platform/platform-event-contracts/src/test/java/com/telco/platform/events/testsupport/ AvroContractAssertionsSelfTest.java(9 cases: exact match passes; type mismatch fails and names the field; missing field fails; extra field fails; a non-nullable schema field with a boxed-nullable Java type fails; a nested array-of-records both passes when matching and fails when the nested element type drifts; the runtime/map entry point passes and fails symmetrically).
2. Architecture decision: option (a) — every compat test now loads the canonical schema directly
from platform-event-contracts, not a local snapshot.
Reasoning: (i) the Avro-generated SpecificRecord classes in platform-event-contracts already carry
the complete, real Schema (including unions, logical types, and nested records) as a static field —
there is nothing to gain and real self-reference risk to keep by hand-copying .avsc text into each
service's src/test/resources; (ii) platform-event-contracts was already a normal (compile-scope)
Maven dependency for four of these eight services (usage, billing, notification, ticket) — extending
that established, already-accepted pattern (a plain module dependency, not a "starter") to the
remaining six is the smallest, most consistent change, and does not violate ADR-018 (that rule is about
services depending only on platform starters for runtime infrastructure — platform-event-
contracts is a contracts module, not a starter, and existing usage already treats it as an ordinary,
directly-depended-on module); (iii) option (b) ("keep local copies, add a drift guard") would have kept
the duplication this reconciliation exists to remove and added a second tool (a file-diff guard) next
to the first (the compat test) for no benefit over option (a). Option (a) was therefore the clear
choice, with no need to escalate.
Per-service pom change: each of customer-service, order-service, payment-service,
product-catalog-service, subscription-service, and identity-service (previously had no
platform-event-contracts dependency at all) gained two new test-scope dependencies:
<dependency>
<groupId>com.telco.platform</groupId>
<artifactId>platform-event-contracts</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.telco.platform</groupId>
<artifactId>platform-event-contracts</artifactId>
<type>test-jar</type>
<scope>test</scope>
</dependency>
platform/platform-bom/pom.xml gained a matching dependencyManagement entry for the test-jar
classifier (Maven's BOM entries are keyed by groupId:artifactId:type:classifier — the pre-existing
plain-jar entry did not cover the new classifier, and the build fails fast and explicitly if it is
missing, which is exactly what happened on the first attempt and was fixed).
3. All 32 canonical schemas now covered, one *EventSchemaCompatTest/*EventContractTest per
service, all pointed at the canonical schema:
| Service | Test class | Events (count) |
|---|---|---|
| order-service | OrderEventSchemaCompatTest |
order.created.v1 (incl. nested OrderItemPayload, checked via automatic recursion, no separate test case needed), order.cancelled.v1 (2) |
| payment-service | PaymentEventSchemaCompatTest |
payment.completed.v1, payment.failed.v1, payment.refunded.v1 (3) |
| billing-service | BillingEventSchemaCompatTest |
invoice.generated.v1, invoice.paid.v1, invoice.overdue.v1 (3) |
| usage-service | UsageEventSchemaCompatTest |
usage.recorded.v1, quota.threshold-reached.v1, quota.exceeded.v1, usage.aggregated.v1, cdr.recorded.v1 (5 — cdr.recorded.v1 newly added to this test, checked against the consumer's CdrRecordedEventConsumer.CdrPayload DTO since it is a consumed, not produced, external contract) |
| notification-service | NotificationEventContractTest |
notification.dispatched.v1 (1) |
| ticket-service | TicketEventContractTest |
ticket.opened.v1, ticket.assigned.v1, ticket.resolved.v1, ticket.sla-breached.v1 (4) |
| product-catalog-service | TariffEventSchemaCompatTest |
tariff.created.v1, tariff.price-changed.v1 (2) |
| subscription-service | SubscriptionEventSchemaCompatTest + MsisdnEventSchemaCompatTest |
subscription.activated.v1, subscription.suspended.v1, subscription.terminated.v1, subscription.activation-failed.v1, msisdn.allocated.v1, msisdn.released.v1 (6) |
| customer-service | CustomerEventSchemaCompatTest |
customer.registered.v1, customer.updated.v1, customer.kyc-approved.v1, customer.kyc-rejected.v1 (4) |
| identity-service | IdentityEventSchemaCompatTest (new file — this service had no compat-test infrastructure at all before this phase) |
user.created.v1, user.deleted.v1 (2) |
Total: 2+3+3+5+1+4+2+6+4+2 = 32, matching all 32 canonical schemas (18 pre-existing + 14 promoted/ authored in phases 3-4).
The old per-service local src/test/resources/avro/*.avsc snapshot directories (billing, customer,
notification, order, payment, product-catalog, subscription, ticket, usage — order-service's also
included the now-superseded standalone order-item.avsc) are deleted, not left in place: once no
test references them, keeping byte-identical duplicates of files that already live canonically in
platform-event-contracts is exactly the self-referential duplication this phase exists to remove.
Proof of failure, then revert (exit criterion)¶
- Deliberately edited
platform/platform-event-contracts/src/main/avro/usage-recorded.avsc, changingrecordedAtfrom plain"string"to"long"(a real, targeted type mismatch — the real producer,UsageRecordedEvent.recordedAt, is a JavaString). cd platform/platform-event-contracts && mvn install -Dschema.registry.skip=true— this module's own self-test caught the break immediately (AvroContractAssertionsSelfTest. passes_when_record_matches_schema_exactlyandfails_on_type_mismatchboth failed, since the self-test's fixture is written against the realUsageRecordedV1canonical schema). Installed with-DskipTeststo get the deliberately-broken artifact into the local repo so the real per-service test could be exercised too.cd microservices && mvn -pl usage-service test -Dtest=UsageEventSchemaCompatTest— FAILED, with the exact, unambiguous diagnosis:This identifies the exact field (java.lang.AssertionError: Avro contract violation(s) for schema com.telco.platform.events.usage.UsageRecordedV1 vs com.telco.usage.application.event.UsageRecordedEvent: - UsageRecordedEvent.recordedAt: Avro type LONG expects Java type long or Long, but found java.lang.StringrecordedAt), the exact schema (UsageRecordedV1), the exact Java class (UsageRecordedEvent), and the exact type mismatch — proving the extended checker catches what the old field-name-only test structurally could not (the field namerecordedAtnever changed; a pre-14.5 test would have passed this exact break).- Reverted
usage-recorded.avscto its originalrecordedAt: "string"(confirmed byte-identical to the pre-edit content). Rebuiltplatform-event-contracts(mvn install -Dschema.registry.skip=true) — all 9 self-tests pass again, BUILD SUCCESS. Re-ranusage-service'sUsageEventSchemaCompatTest— all 5 cases pass again, BUILD SUCCESS.
Full verification run¶
cd microservices && mvn -pl customer-service,order-service,payment-service,usage-service,
billing-service,notification-service,ticket-service,product-catalog-service,subscription-service,
identity-service verify -DskipITs — BUILD SUCCESS, all 10 modules, 0 test failures/errors across
the reactor (unit tests, JaCoCo coverage gates, checkstyle, spotbugs all included in verify). Followed
by cd microservices && mvn compile across the entire microservices reactor (every service,
including the ones untouched by this phase) to confirm the platform-bom change — every microservice
transitively imports it — did not break dependency resolution anywhere: BUILD SUCCESS, all modules.
The full mvn -f microservices/pom.xml verify (every module including integration/acceptance tests)
was not run in full given its testcontainers/Docker startup cost is unrelated to this phase's changes;
the targeted 10-module verify plus the full-reactor compile sanity check together cover everything
this phase touched (10 services' test code and poms) and everything it could have broken (every
service's dependency resolution via the shared BOM).
Files changed (phase 6)¶
- New:
platform/platform-event-contracts/src/test/java/com/telco/platform/events/testsupport/ AvroContractAssertions.java,AvroContractAssertionsSelfTest.java. platform/platform-event-contracts/pom.xml: added test-scopejunit-jupiter; added themaven-jar-plugintest-jarexecution.platform/platform-bom/pom.xml: added thetest-jar-classifierdependencyManagemententry forplatform-event-contracts.- Pom dependency additions (test-scope
platform-event-contracts+ its test-jar):customer-service,order-service,payment-service,product-catalog-service,subscription-service,identity-service. Test-jar-only addition (main dependency already present):usage-service,billing-service,notification-service,ticket-service. - Rewritten to use
AvroContractAssertionsagainst the canonical schema:OrderEventSchemaCompatTest,PaymentEventSchemaCompatTest,BillingEventSchemaCompatTest,TariffEventSchemaCompatTest,CustomerEventSchemaCompatTest,SubscriptionEventSchemaCompatTest,MsisdnEventSchemaCompatTest,UsageEventSchemaCompatTest(also gained the 5th, previously-untestedcdr.recorded.v1case),TicketEventContractTest,NotificationEventContractTest. - New:
identity-service/src/test/java/com/telco/identity/IdentityEventSchemaCompatTest.java(this service had zero compat-test infrastructure before this phase). - Deleted: all per-service
src/test/resources/avro/*.avsclocal snapshot directories (billing, customer, notification, order (incl.order-item.avsc), payment, product-catalog, subscription, ticket, usage) — superseded by loading the canonical schema directly.
Phase 6 (event-integration) is DONE. Phase 7 (qa: full contract + acceptance suite run, event- catalog.md updates) is next.
Phase 7 (qa, 2026-07-07) — DONE¶
1. Full contract-test suite: mvn -f microservices/pom.xml verify (full reactor)¶
JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-21.jdk/Contents/Home mvn -f microservices/pom.xml
verify — BUILD SUCCESS, all 18 reactor modules (microservices parent, domain-services-parent,
service-template, reference-service, config-server, discovery-server, api-gateway,
identity-service, customer-service, product-catalog-service, order-service,
subscription-service, usage-service, billing-service, payment-service,
notification-service, ticket-service, web-bff). Total wall time 6m06s. Aggregate test count
across all per-service Surefire/Failsafe summaries: 638 tests run, 0 failures, 0 errors, 0
skipped (identity 38, customer 83, product-catalog 49, order 85, subscription 67, usage 77,
billing 71, payment 64, notification 45, ticket 47, plus the smaller platform/gateway modules).
This includes every one of the 32 new/rewritten *EventSchemaCompatTest/*EventContractTest
classes from phase 6 (all green), the new IdentityEventSchemaCompatTest, and every
Testcontainers-backed integration test and JaCoCo coverage gate in the reactor - "All coverage
checks have been met" for every module that runs one. This is the primary gate for Feature 14.5 and
it is green: no regression from phases 1-6's schema reconciliation, new schemas, the rename, or the
new compat-test tooling.
2. Acceptance suite: mvn -f microservices/pom.xml -pl acceptance-tests -am -Pacceptance verify¶
The full docker-compose live stack (gateway, Keycloak, Kafka, Schema Registry, all ten domain
services, Postgres, Redis, Mongo, MinIO) was already running from earlier this session (docker ps
confirmed all containers healthy) - per this task's own instruction, standing up a fresh stack was
not required, and since Feature 14.5 changed only canonical .avsc files and test/pom code (no
producing service's main-source event-payload class changed shape - the diff spec's explicit
direction throughout was "schema matches real payload," not the reverse, and the one place a real
code change was tested (invoice-generated.avsc subscriptionId tightening) was reverted after the
live compatibility check rejected it), there was no runtime-behavior change for the already-running
containers to be stale against. Ran the suite against the live stack as-is.
Result: BUILD FAILURE — 1 of 4 acceptance tests failed. AC-01 (compensation path), AC-02 (monthly bill-run), and AC-03 (quota exhaustion) all passed. The one failure:
NewSubscriberOnboardingAcceptanceIT.newSubscriberOnboardingSucceedsEndToEnd:99
Expecting actual:
"905990000002"
to match pattern:
"90532\d{7}"
Root cause investigated and confirmed NOT a Feature 14.5 regression. Queried the live
subscription_db.msisdn_pool table directly (docker exec telco-postgres psql ...):
prefix | status | count
90532 | ALLOCATED | 1000 <- the full seeded pool (V2__msisdn_pool_seed.sql), 100% exhausted
90599 | ALLOCATED | 4
90599 | FREE | 96
1100 total rows, but V2__msisdn_pool_seed.sql (unmodified since Sprint 09 - git log confirms no
commit or working-tree change to it this session) only ever seeds 1000 rows in the 90532 block.
The 100-row 90599 block exists only in the live database, inserted directly (not via a Flyway
migration - grep of every .sql file under subscription-service/src/main/resources/db/migration/
finds zero occurrences of 9059 anywhere in source). This is a stopgap top-up applied straight to
the running container's database at some earlier point this session, almost certainly after the
original 1000-number 90532 pool was fully exhausted by the volume of acceptance-suite runs already
executed against this same long-lived stack (container uptimes of 9-37 hours per docker ps).
MsisdnAllocationService.allocate() (subscription-service/src/main/java/com/telco/subscription/
domain/MsisdnAllocationService.java) has no format-specific logic at all - it just takes the next
FREE row from the pool table regardless of prefix, so once the 90532 block hit zero FREE rows,
allocation legitimately started serving numbers from the 90599 top-up block, which the test's
hardcoded "90532\\d{7}" regex (line 99) was never written to expect.
Confirmed no Feature 14.5 code path touches this: git diff on subscription-service's migration
directory is empty; the five subscription-service main-source files this session did modify
(SubscriptionController, PaymentCompletedEventConsumer, PaymentFailedEventConsumer,
GetSubscriptionsByCustomerQuery(Handler)) belong to the unrelated 14.1.1 identity-linkage-gap work,
none reference msisdn/MSISDN allocation, and MsisdnAllocationService/MsisdnPoolRepository/
MsisdnPool are untouched by any change in this session.
Disposition: pre-existing test-environment/data issue, out of scope for Feature 14.5, not fixed
here. This is a live-environment artifact (pool exhaustion plus an ad hoc, out-of-migration DB
patch) tied to this specific long-running Docker stack's accumulated state, not a defect in any
schema, event payload, or the AC-01 saga logic itself - a fresh environment seeded only from
V2__msisdn_pool_seed.sql would allocate a correct 90532xxxxxxx number. Fixing it durably (e.g.,
widening the seeded pool, making pool-exhaustion behavior deterministic for test/CI environments, or
loosening the test's regex to accept any valid Turkish MSISDN prefix rather than one hardcoded
block) is a test-data/environment decision outside this feature's scope, flagged here for
devops/domain-engineer to pick up, not decided unilaterally as part of 14.5's phase 7. No code,
schema, or test file was changed to work around it.
Phase 7 exit criterion assessment: "green contract-test and acceptance runs" - the contract-test half is fully green (638/638). The acceptance half is green on AC-01's compensation path, AC-02, and AC-03; AC-01's happy path fails on a root cause independently verified to be outside this feature's change set. Phase 7 is DONE with this one flagged, pre-existing, out-of-scope finding - it does not block Feature 14.5's own sign-off, since "confirm no regression from phases 1-6" is satisfied: this failure predates and is unrelated to those phases.
3. docs/architecture/event-catalog.md updates¶
- Added two rows to Section 2 (Event Registry):
user.created.v1anduser.deleted.v1(identity-service, no current consumer - confirmed viagrepthat no service consumes either event yet). These were the two events architecture's phase-2 diff spec flagged as entirely absent from the catalog. - Added a new Section 6, "Schema Governance Reconciliation Log (Feature 14.5)", documenting: the 7
reconciled pre-existing schemas (
order-created,payment-completed,cdr-recorded,usage-aggregated,usage-recorded,quota-exceeded,quota-threshold-reached, including the nestedOrderItemPayloadinline-record resolution), the 14 newly added schemas, theEventEnvelope.avsc->event-envelope.avscrename, and the newAvroContractAssertionscompat-test tooling requirement going forward. Distinct from the pre-existing Section 5 ("Schema Evolution Log"), which tracks additive field changes to already-registered schemas - Section 6 is scoped to this one-time reconciliation exercise. - Bumped the document's "Last updated" field to 2026-07-07.
4. notification-service test-fixture fix¶
DomainEventNotificationConsumerTest.java (microservices/notification-service/src/test/java/com/
telco/notification/consumer/) had two tests asserting against fictional event-type strings never
produced by any real service (subscription.cancelled.v1, customer.profile-updated.v1) - flagged
during phase 5. Re-pointed both at real, legitimately-unhandled event names rather than deleting the
scenarios, since the behavior under test (an unrecognised event type is silently ignored) is real and
worth keeping:
- subscription_event_with_unrecognised_type_is_silently_ignored now uses
subscription.suspended.v1 - a real event subscription-service produces, which
DomainEventNotificationConsumer.onSubscriptionEvent does not currently handle (confirmed via
grep - no SUSPEND/TERMINAT handling exists in this consumer).
- customer_event_with_unrecognised_type_is_silently_ignored now uses customer.updated.v1 - a real
event customer-service produces, which onCustomerEvent does not currently handle (only
customer.kyc-approved.v1/customer.kyc-rejected.v1 are handled).
Verified passing as part of the full reactor run above: com.telco.notification.consumer.
DomainEventNotificationConsumerTest - 14 tests run, 0 failures, 0 errors.
5. Other loose ends from phases 1-6 checked for qa scope¶
Re-read phases 1-6 in full for anything else explicitly deferred to qa. Found one other item,
correctly out of scope and left untouched: DomainEventNotificationConsumer.onSubscriptionEvent
reads a customerName field off subscription.activated.v1 payloads that no producer ever sets
(falls back to customerId via getOrDefault, never breaks). Phase 3/4's diff spec explicitly
labeled this "flagged for qa/domain-engineer awareness only, not actioned here" - an FYI, not an
action item, and not an Avro contract violation. Left as-is, per that explicit framing; no other item
in phases 1-6 was addressed to qa beyond the test-fixture fix above and this feature's own phase 7
exit criteria.
Phase 7 conclusion¶
Phase 7 (qa) is DONE: contract-test suite green (638/638), acceptance suite green on 3 of 4 flows with the one AC-01-happy-path failure root-caused to a pre-existing, out-of-scope MSISDN-pool exhaustion artifact in this session's long-lived live stack (not a Feature 14.5 regression), event-catalog.md updated with the two missing rows and a new Schema Evolution/Reconciliation log section, and the notification-service test-fixture fix applied and verified passing. Phase 8 (devops + code-review + tech-lead sign-off) is next; the AC-01 MSISDN-pool finding should be handed to devops/domain-engineer as a separate, pre-existing item, not blocking Feature 14.5's own sign-off.
Phase 8 — devops portion (2026-07-07) — DONE¶
1. Compat-test gate in CI — already covered, no new CI step needed (confirmed, not assumed)¶
.github/workflows/ci.yml's microservices-test job runs mvn ${MAVEN_ARGS} verify --fail-at-end
-Dspotbugs.skip=true -Dcheckstyle.skip=true in microservices/, after a preceding Install platform
to local Maven repo step (mvn install -DskipTests -Dspotbugs.skip=true
-Dschema.registry.skip=true in platform/). microservices/pom.xml already declares all 10
producing services (identity-service, customer-service, product-catalog-service,
order-service, subscription-service, usage-service, billing-service, payment-service,
notification-service, ticket-service) as reactor modules, so this existing verify run already
compiles and executes every one of phase 6's rewritten *EventSchemaCompatTest/*EventContractTest
classes - no new job, step, or workflow was needed. Concretely verified rather than assumed:
cd platform/platform-event-contracts && mvn install -DskipTests -Dspotbugs.skip=true -Dschema.registry.skip=true(byte-identical to the CI step) - confirmed it produces bothplatform-event-contracts-1.0.0-SNAPSHOT.jarand the-tests.jarclassifier in the local repo (~/.m2/repository/com/telco/platform/platform-event-contracts/1.0.0-SNAPSHOT/), i.e. the test-jar the 10 services now depend on (phase 6) is genuinely produced by this exact CI command, not just by a developer's full local build.cd microservices && mvn -pl identity-service -am test -Dtest=IdentityEventSchemaCompatTest -Dspotbugs.skip=true -Dcheckstyle.skip=true -oagainst that freshly-installed repo (offline flag forces resolution from exactly what the install step produced, no leftover artifacts from a different build) - BUILD SUCCESS, 2/2 tests passed. Chosen because identity-service had zero compat-test infrastructure before phase 6 (new file, new dependency, most likely to expose a wiring gap) - proves the newly-added test-jar dependency resolves and the new test class runs correctly under CI's own exact command sequence.- This complements, not duplicates, phase 6's own proof-of-failure (the deliberately-broken
usage-recorded.avscrecordedAttype-mismatch, which failedUsageEventSchemaCompatTestloudly) and phase 7's full-reactormvn -f microservices/pom.xml verify(638/638). Together these three runs cover: the CI command sequence in isolation (this phase), the extended checker's actual fault-detection behavior (phase 6), and the complete, unmodified reactor (phase 7).
Conclusion: no CI change was required for point 1. A PR that reintroduces drift between a
service's real event payload and its canonical schema fails microservices-test in ci.yml today,
on every PR to master, with no additional wiring.
2. Schema Registry compatibility check in CI — partially closed, with an honest, tested limit¶
Before this phase: ci.yml ran the platform build with -Dschema.registry.skip=true in every
job that touches platform/ (build-test, static-analysis, and microservices-test's install
step) - no live registry, by design, unchanged by Feature 14.5. .github/workflows/acceptance.yml
already brings up a full docker-compose stack including a real schema-registry container
(telco-schema-registry, localhost:8081 - the exact default in
platform-event-contracts/pom.xml's schema.registry.url) for Debezium's benefit, but its own
Install platform to local Maven repo step also passed -Dschema.registry.skip=true, meaning the
one workflow that already had a live registry sitting right there was not using it for this check
either. Net effect: the Schema Registry compatibility check ran nowhere in CI, contradicting
phase 8's stated exit criterion.
What was verified by hand, live, before changing anything (using this session's long-running
local stack, telco-schema-registry, confirmed healthy via docker ps, plus a disposable
no-history registry container stood up on an unused port to faithfully simulate a brand-new,
empty-per-run CI registry):
cd platform/platform-event-contracts && mvn verify -Dschema.registry.skip=falseagainst the long-lived registry (which already carries real, persisted history for all 33 subjects from earlier phases) - BUILD SUCCESS, all 33 subjects reportedis_compatible: true.- The identical command against a freshly started, empty registry container (no prior
versions for any subject) - also BUILD SUCCESS, all 33 subjects compatible - because a
subject with no registered version is trivially compatible (confirmed directly via
POST /compatibility/subjects/<nonexistent>/versions/latest,is_compatible: true). - The same nonexistent-subject endpoint, given a schema with a genuine parse error (an Avro
field typed to an undefined name) - is_compatible: false, with a clear
"is not a defined name"error. Confirms registrability/structural validity is checked regardless of history. - A deliberate, targeted type-mismatch (
usage-recorded.avsc'srecordedAtretyped fromstringtolong- the same class of drift phase 6 proved the compat-test tooling catches) run against the empty registry - BUILD SUCCESS (incorrectly "compatible", because there is no prior version to violate). The identical edit run against the long-lived, history-bearing registry - BUILD FAILURE (One or more schemas found to be incompatible with the current version), correctly caught. Reverted immediately after (confirmed viagit diff, byte-identical to before).
This proves, rather than assumes, the exact shape of the gap: the compatibility check's value in an
always-fresh CI registry is real but partial - it enforces registrability (a schema must be
valid, parseable Avro that Schema Registry itself accepts, which is exactly the defect class Feature
14.5 found in order.created.v1's original split OrderItemPayload reference) but cannot enforce
persisted BACKWARD-compatibility drift across time, because no CI-recreated registry carries
history from a previous run to diff against.
Change made: .github/workflows/acceptance.yml's Install platform to local Maven repo step
now runs with -Dschema.registry.skip=false (was true), so the kafka-schema-registry-maven-
plugin's test-compatibility goal (bound to the verify phase, which mvn install already passes
through) executes for real, against the real telco-schema-registry container this workflow already
stands up, for all 33 canonical subjects, on every PR touching microservices/**, platform/**,
infra/docker/**, or the workflow file itself, plus on-demand via workflow_dispatch. A detailed
comment was added directly above the step explaining both what this catches (registrability/parse
validity, every run) and what it does not (true point-in-time BACKWARD-compat drift, since the
registry has no persisted history across runs) - the same honest distinction proven above, so no
future reader mistakes this for a full historical-drift gate. ci.yml was left unchanged
(-Dschema.registry.skip=true in all three of its platform-touching steps) - it has no live
registry available at all, and standing one up there is a separate CI-infrastructure decision, not
reopened here - but its existing comment was expanded to point at acceptance.yml's gate and this
document for the full picture, rather than leaving a stale "runs locally only" claim in place now
that half of that claim is no longer true.
Residual gap, stated plainly, not papered over: no CI workflow in this repository currently
enforces true, persisted-history BACKWARD Schema Registry compatibility - ci.yml (the per-PR gate
for every change) has no registry at all, and acceptance.yml's registry, while real and now
exercised, is destroyed and recreated empty on every run (make -C infra destroy in its always-run
teardown step), so it can only ever catch structural/registrability breaks, not a genuine type or
nullability narrowing against a previously-registered version (the exact case
invoice-generated.avsc's subscriptionId nullability ruling, above, depended on a persistent
registry to catch). Closing this fully would require a schema-registry deployment whose subject
history survives across CI runs (e.g., a long-lived registry service outside the ephemeral
per-job compose stack, or a persisted-volume approach) - a real CI infrastructure decision requiring
its own design (where does that registry live, who else can pollute its subject history, what is its
compatibility mode, how is it reset when a subject's v1 is legitimately superseded) - flagged here
for a future ticket, not decided unilaterally as part of this feature's phase 8.
3. Verification performed¶
- YAML syntax:
python3 -c "import yaml; yaml.safe_load(open(...))"on both.github/workflows/ci.ymland.github/workflows/acceptance.yml- both parse cleanly. No GitHub Actions lint tool (actionlint) was available in this environment to install without network access beyond what was already used; the YAML-parse check plus a structural diff review (comment placement, indentation matching every neighboring step) stand in for it, per this task's own "at minimum" bar. - No real CI run was triggered (no push/PR involved), per this task's own scope - all verification
above was performed by running the exact same Maven commands
acceptance.ymlandci.ymlinvoke, locally, against real (long-lived and disposable) Schema Registry containers, and by running the exact reactor commandsci.yml'smicroservices-testjob runs. - All experimental/scratch changes (
usage-recorded.avsc's temporary type mismatch, the disposabletelco-schema-registry-fresh-testcontainer) were reverted/removed;git diffonusage-recorded.avscshows only phase 3/4's already-existing, intentional reconciliation, nothing from this verification.
Phase 8 (devops) conclusion¶
Point 1 (compat-test gate wired into CI) required no change - confirmed, with concrete proof
beyond reading the YAML, that the existing ci.yml pipeline already exercises all 32
canonical-schema-vs-real-payload checks via the 10 rewritten *EventSchemaCompatTest/
*EventContractTest classes on every PR to master. Point 2 (Schema Registry compatibility-mode
check in CI) was partially closed: acceptance.yml now runs the real compatibility check
against a real, already-present Schema Registry container for all 33 subjects, closing the
registrability/structural-validity gap for good; ci.yml's per-PR gate still has no registry
available at all (pre-existing, explicitly out of scope to reopen) and even acceptance.yml's newly
-enabled check cannot catch true persisted-history BACKWARD-compatibility drift, since its registry
is recreated empty every run - both limits are documented in the workflow files themselves and here,
not silently assumed or fabricated. Phase 8's devops portion is DONE; code-review's ADR-019-
compliance pass and tech-lead's final sign-off are next.
Phase 8 — code-review portion (2026-07-08) — DONE¶
Scope: read-only ADR-compliance pass over the actual diff this feature produced (working tree at
time of review — no commit boundary existed yet). Spot-checked, not just narrative-trusted: read
ADR-018/ADR-019 directly, diffed every changed file under platform/platform-event-contracts/,
read the real Java payload classes for a sample of reconciled/new schemas, scanned for emojis, and
reviewed AvroContractAssertions.java end-to-end.
Finding 1 (MEDIUM) — invoice-generated.avsc's subscriptionId doc field does not capture the¶
rejected-tightening history; a future reader can re-attempt the same rejected fix
File: platform/platform-event-contracts/src/main/avro/invoice-generated.avsc, subscriptionId
field (currently ["null","string"], default: null).
File state confirmed correct — still nullable, byte-identical to the pre-14.5 committed version
(git diff against HEAD is empty), matching this document's "Final state left in place: REVERTED
to nullable" claim above.
However, the field's own doc string reads: "Subscription this invoice covers, when invoice is
per-subscription. Nullable for account-level invoices." This describes a business-semantics
reason for nullability ("account-level invoices") that this document's own investigation shows is
not the real reason — the real, current producer (BillRunBatchProcessor.InvoiceGeneratedEvent)
always populates subscriptionId, and the field is kept nullable purely because a live
BACKWARD-compatibility check against the already-registered invoice.generated.v1 subject
rejected tightening it (is_compatible: false, TYPE_MISMATCH, documented above). A future
engineer reading only the schema file (not this tracking doc) will see a plausible-sounding but
incorrect business justification and has no signal that tightening this field was already tried,
tested against the live registry, and rejected — exactly the "rediscover this exact question"
failure mode this review was asked to check for.
Required fix: update the doc string in invoice-generated.avsc to state the real reason,
e.g.: "Subscription this invoice covers. Always populated by the current producer
(BillRunBatchProcessor); kept nullable because invoice.generated.v1 is already live-registered in
Schema Registry and a BACKWARD-compatibility check confirmed tightening to non-nullable is rejected
(TYPE_MISMATCH: reader STRING not compatible with writer NULL). To enforce non-nullability at the
wire-contract level, introduce invoice.generated.v2 per ADR-009/ADR-019's immutability rule — do not
mutate this v1 field. See docs/tasks/sprint-14-testing-and-hardening/14.5-avro-schema-governance-
ruling.md for the full compatibility-check evidence." Non-blocking (does not change wire shape or
compatibility), but should land before final close so the tracking doc's own durability goal
("prevents this drift from recurring") actually holds for this specific, already-tested trap.
Finding 2 (LOW/advisory, escalate) — ADR-018 boundary call on platform-event-contracts was¶
decided unilaterally by event-integration rather than escalated
ADR-018 (architecture/adr/ADR-018-platform-starter-dependency-model.md), Dependency Rule (section
2): "Microservices MUST ONLY depend on starters. They MUST NOT depend on internal platform modules
directly." This is unscoped, absolute text — it does not carve out test-scope dependencies or
"contracts-only" modules the way ADR-019's amendment explicitly carved out JSON-shape-vs-wire-bytes
for outbox publishing.
platform-event-contracts lives under platform/ alongside platform-starters/, platform-core,
and platform-autoconfigure — it is not itself a spring-boot-starter-* module (no
AutoConfiguration classes, not listed among ADR-018's starter examples). Phase 6 extended a direct,
non-starter dependency on it from 4 services (pre-existing, and in 2 of those 4 cases as a
production/compile-scope dependency, e.g. billing-service/pom.xml, usage-service/pom.xml) to 10
services (test-scope only for the 6 newly added: customer-service, order-service,
payment-service, product-catalog-service, subscription-service, identity-service) — verified
directly in each service's pom.xml diff.
Event-integration's phase-6 log resolves this itself: "does not violate ADR-018 (that rule is
about services depending only on platform starters for runtime infrastructure —
platform-event-contracts is a contracts module, not a starter)... no need to escalate." That is a
reasonable reading in spirit (test-scope, schema/generated-class artifacts are a different kind of
thing than starter-mediator/starter-outbox/etc.), and it is consistent with an already-existing,
pre-14.5 pattern (the 4 services' compile-scope dependency predates this feature). But it is a
genuine interpretation of ADR-018's literal, unscoped text, not a mechanical application of it — and
this same feature escalated comparably-sized boundary questions to architecture/tech-lead
elsewhere (the OrderItemPayload registry-subject conflict, the invoice-generated.avsc
nullability question). Deciding this one unilaterally is inconsistent with that same discipline.
Required fix (non-blocking for this sign-off, but must be closed): get an explicit
architecture/tech-lead ruling — either (a) formally amend ADR-018 with a stated carve-out for
contract/schema modules (mirroring how ADR-019's amendment explicitly scoped A1-A5), or (b) rule
that this reading is correct and record it as an addendum here, the same way the OrderItemPayload
and invoice-generated.avsc questions were recorded. Until one of those exists, this dependency
pattern rests on one agent's unescalated reading of an absolute ADR sentence.
Checks that passed clean¶
- ADR-019 compliance of the 33
.avscfiles — all filenames kebab-case (verified programmatically, zero non-compliant names); all 33 files are valid JSON;event-envelope.avscrename confirmed with recordnameunchanged (EventEnvelope);pom.xml'skafka-schema-registry-maven-plugin<subjects>block lists exactly 33 entries (event-envelope - 32 real events), no
<imports>block (confirms theOrderItemPayloadinline-record ruling was actually applied, not just claimed). Spot-checkedorder-created.avscagainstOrderCreatedEvent.java,payment-completed.avscagainstPaymentCompletedEvent.java,invoice-generated.avsc/invoice-paid.avsc/invoice-overdue.avscagainstBillRunBatchProcessor/MarkInvoicePaidCommandHandler/MarkInvoicesOverdueCommandHandler, anduser-created.avsc/user-deleted.avscagainstUserCreatedV1.java/UserDeletedV1.java— all field-for-field, type-for-type exact matches, no discrepancy from what this document claims. - No unintended blast radius — confirmed no canonical schema exists for
customer.profile-updated.v1,subscription.cancelled.v1,demoitem.created.v1, ororder.confirmed.v1(directory listing + content grep, all clean).order-cancelled.avscis confirmed to be the real, distinctorder.cancelled.v1event, not a stray file for the excludedsubscription.cancelled.v1. Noted in passing, not a 14.5 defect: the working tree also contains unrelated, uncommitted changes from other in-flight work (14.1.1'sregisteredByUserIdaddition tocustomer-registered.avsc/CustomerRegisteredV1, and an unrelated JaCoCo hard-gate change toci.yml) — confirmed both are attributable to separate, already-ruled-on tracking docs, not to this feature, and neither touches anything this feature's non-goals list forbids. - No emojis — scanned all 33
.avscfiles,AvroContractAssertions.java+AvroContractAssertionsSelfTest.java, both CI workflow diffs, this tracking doc,ADR-019, andevent-catalog.mdwith a Unicode emoji-range pattern; zero real emoji hits. (The only symbol matches were plain→arrows in this tracking doc's prose, not emojis — ARC-09 clean.) AvroContractAssertionscode quality — the two-entry-point split (typed-record reflection vs. captured-Mapruntime check) matches the two payload-construction styles that actually exist in this codebase, with no third path speculatively built. The class Javadoc states the WHY (why canonical-schema-not-local-copy, why FAIL vs. WARN vs. undecidable for nullability) up front, which is exactly where this project's comment discipline expects it. Recursion into nested records/arrays is handled once, generically (coversorder.created.v1's inlinedOrderItemPayloadwith no special-casing). No over-engineering — no reflection-based generic type-adapter framework, no attempt to handle Avro types this codebase doesn't use. Self-test (AvroContractAssertionsSelfTest, 9 cases) exercises both entry points, the nested-array recursion, and the nullability FAIL/WARN paths.
Verdict: APPROVE, with Finding 1 required before final close and Finding 2 flagged for¶
tech-lead's sign-off pass (see below)
Nothing found rises to a build-blocking or wire-compatibility-breaking defect. Finding 1 is a concrete, low-effort documentation fix that should land before this feature is marked fully closed (it directly serves this feature's own stated goal of preventing rediscovery of an already-rejected fix). Finding 2 is not a demonstrated violation — it is a genuine ADR-018 boundary-interpretation question that this feature's own escalation discipline says should go to tech-lead rather than rest on one agent's unilateral reading, exactly the kind of item this document has recorded and escalated repeatedly elsewhere. Recommend tech-lead's final sign-off either accept event-integration's reading explicitly (making this doc that ruling) or send it back for the ADR-018 amendment.
Tech-lead final sign-off (2026-07-08) — Feature 14.5 is DONE¶
Read the complete 8-phase record above in full, including this document's own phase-1 ruling, the
ADR-019 amendment, architecture's diff spec, event-integration's schema authoring, the
order.created.v1/OrderItemPayload inline-record ruling, phase 5's per-service reconciliation
(folded into phases 3-6 above), phase 6's new AvroContractAssertions tooling and its
proof-of-failure/revert cycle, phase 7's qa verification (638/638 reactor tests, event-catalog.md
updated, the one AC-01 failure independently root-caused to a pre-existing, out-of-scope MSISDN
pool-exhaustion artifact), phase 8's devops CI wiring, and code-review's APPROVE-with-2-findings
pass. Both open findings are resolved below.
Finding 1 (MEDIUM, code-review) — RESOLVED¶
platform/platform-event-contracts/src/main/avro/invoice-generated.avsc's subscriptionId field
doc string has been corrected in place. Verified: file is still valid JSON
(python3 -c "import json; json.load(open(...))" — clean) and
platform/platform-event-contracts still builds green
(mvn generate-sources -Dschema.registry.skip=true — BUILD SUCCESS) after the edit; no field, type,
or nullability changed, only the doc string. New doc text (in place of the old
business-semantics claim):
"Subscription this invoice covers. Always populated by the current producer (BillRunBatchProcessor); kept nullable because invoice.generated.v1 is already live-registered in Schema Registry and a live BACKWARD-compatibility check confirmed tightening to non-nullable is rejected (TYPE_MISMATCH: reader STRING not compatible with writer NULL). A genuine future tightening requires a new invoice.generated.v2 event version per ADR-009/ADR-019's immutability rule, not a v1 mutation -- see docs/tasks/sprint-14-testing-and-hardening/ 14.5-avro-schema-governance-ruling.md for the full compatibility-check evidence."
This states the real reason (always-populated by the real producer, kept nullable purely because of a tested, rejected BACKWARD-compatibility tightening attempt, with the exact rejection evidence already recorded above in this document) instead of the plausible-but-untrue "nullable for account-level invoices" business claim that risked a future engineer re-attempting the same already-rejected fix. Closed.
Finding 2 (LOW/escalation, code-review) — RULED: not a violation; ADR-018 amended with an¶
explicit carve-out
Read ADR-018 directly (not just code-review's characterization) before ruling.
Ruling: (a) — this dependency pattern is acceptable as-is. platform-event-contracts is a pure
schema/contract-definitions module: no AutoConfiguration, no Spring beans, no runtime behavior
injected into a consuming service, nothing that leaks infrastructure into business code — the three
harms ADR-018's Context section states the Dependency Rule exists to prevent. It is meaningfully
different in kind from the internal platform modules (platform-core, platform-autoconfigure)
that rule was written to stop services from coupling to directly: those carry business logic and
wiring a service would otherwise reimplement or hand-configure; platform-event-contracts carries
only Avro-generated data-carrier classes (the moral equivalent of a generated-DTO/protobuf-stubs
module) plus, as of this feature, a small test-support assertion helper. Depending on it directly —
compile-scope where a service builds/deserializes the canonical event type, test-scope (+ test-jar)
where a contract test asserts against the canonical schema — does not create the coupling ADR-018
targets.
This is also not a new pattern invented under cover of this feature: 4 services
(usage-service, billing-service, notification-service, ticket-service) already depended on
it directly before Feature 14.5, 2 of them (billing-service, usage-service) at compile
scope — in shipped production code, not just tests — and that had never been flagged through
however many prior review passes. Phase 6 extended the identical, already-accepted pattern, in
test-scope only, to 6 more services so every producing service's contract test could assert
against the one real canonical schema instead of a hand-maintained local copy. Event-integration's
unilateral read in phase 6 reached the correct conclusion; the process gap code-review correctly
flagged (deciding a genuine ADR-018 boundary question without escalation, unlike how this same
feature escalated the OrderItemPayload and invoice-generated.avsc questions) is closed by this
ruling existing at all, going forward.
Amendment applied, not just an addendum here — per the same discipline used for ADR-019 earlier
in this feature, ADR-018 itself now carries an explicit "Amendment (2026-07-08)" section
(architecture/adr/ADR-018-platform-starter-dependency-model.md) stating: (A1) the Dependency Rule
targets runtime-infrastructure coupling, not contract-definition modules; (A2) direct dependency on
platform-event-contracts specifically (compile- or test-scope, including its test-jar) is
compliant, scoped to that one module — platform-core/platform-autoconfigure/other internal
modules remain fully subject to the unscoped rule; (A3) the basis (an existing, pre-14.5 pattern
being ratified, not invented); (A4) what would make this stop qualifying (the module gaining
AutoConfiguration or injected runtime behavior). This closes the question for good — a future
agent hitting this same dependency will find a ruled, amended ADR, not a third unescalated reading
or a fourth escalation of the same point. Finding 2 closed.
Final determination: Feature 14.5 is DONE¶
Evidence considered: 638/638 reactor tests green across all 18 modules (phase 7); the acceptance
suite's one failure (NewSubscriberOnboardingAcceptanceIT's MSISDN-format assertion) independently
root-caused to a pre-existing, out-of-scope MSISDN-pool-exhaustion artifact in this session's
long-lived Docker stack (a live, out-of-migration DB top-up block), confirmed via direct
git diff/grep to be untouched by any Feature 14.5 change or any other change this session — a
fresh environment seeded only from V2__msisdn_pool_seed.sql would not hit it; code-review's
APPROVE verdict with both findings now resolved above; the full 8-phase execution trail (ADR-019
amendment, 33/33 canonical schemas reconciled/authored and live-registered, the
order.created.v1/OrderItemPayload inline-record resolution, the new type-and-nullability-aware
AvroContractAssertions compat tooling with a proven catch-then-revert cycle, event-catalog.md
updated, CI wiring confirmed/extended in ci.yml/acceptance.yml with its one honestly-documented
residual gap — no CI registry with cross-run persisted history, correctly deferred as a distinct
future CI-infrastructure decision, not part of this feature's scope).
Feature 14.5 (Avro Schema Governance Reconciliation) is DONE. The MSISDN-pool-exhaustion finding and the CI persisted-history registry gap are both real, both correctly out of scope here, and both explicitly handed off (to devops/domain-engineer and to a future CI-infrastructure ticket respectively) — neither blocks this feature's own closure.
Sprint 14 rollup at this point¶
- 14.1 Acceptance and End-to-End Testing — DONE
- 14.2 Security Hardening — DONE
- 14.3 Performance Validation — DONE
- 14.4 Identity-to-Customer Linkage — not DONE. Code-complete and individually verified per
service, but the full end-to-end proof is not closed: a real, fresh JWT actually carrying the
customerIdclaim through a genuine self-registration was never proven, blocked specifically by the realm's User ProfileunmanagedAttributePolicygap (a persistent, security-adjacent Keycloak realm-config change, correctly not applied without its own authorization). Tracked as a narrow, precisely-scoped follow-up in14.1.1-identity-linkage-gap-ruling.mdStep 7 — not a design gap, a single remaining authorization-gated configuration change plus its end-to-end proof. - 14.5 Avro Schema Governance Reconciliation — DONE (this sign-off).
Sprint 14 final status: 4 of 5 features DONE (14.1/14.2/14.3/14.5); 14.4 remains the one open,
narrowly-scoped item (Keycloak realm User Profile customer_id attribute configuration, plus the
end-to-end JWT-claim proof it unblocks). Sprint 14 is not fully DONE until 14.4 closes, but nothing
about 14.4's remaining gap is architectural or in dispute — it is a single authorization-gated
infrastructure change away from completion.