Platform team, week 36

Amber Canary gate work (A-4) slipped one week; everything else on plan.

Period: Mon 2026-08-31 to Sun 2026-09-06

Author: Priya Natarajan, SRE lead
Published: 2026-09-07
Distribution: Engineering leadership, Dispatch and Integrations teams, Support leads
Previous: Week 35: Red (INC-2026-0827 outage)

Summary

Status moves from red to amber. The outage follow-ups from INC-2026-0827 are done except the relative canary gate (A-4), which slipped from 2026-09-05 to 2026-09-12 because the deploy pipeline change needed a second reviewer from the release team. Deploy volume recovered to 26 for the week and change failure rate returned to 3.8%. The one thing that needs attention this week is a decision on RFC-014's open questions by 2026-09-19; without it phase 0 misses the 2026-09-22 start.

Highlights

Shipped five of six INC-2026-0827 corrective actions: partial index in all four regions, 2 s pool timeout for interactive services, 80% pool warning alert, and the PR template checklist.
Published the INC-2026-0827 RCA (v1.2) to engineering and to the two affected premium accounts through their account managers.
RFC-014 (queue-backed webhook delivery) reached v3 with a decision log; 3 of 4 reviewers have commented.
Reduced Dispatch API p95 latency from 205 ms to 188 ms by removing a redundant permission lookup on POST /v2/jobs/{id}/assign.
Canary relative-latency gate (A-4) implemented and in review; not yet merged. New due date 2026-09-12.

Metrics

Deploys to production

+15 vs week 35 · target ≥ 20

Change failure rate

3.8%

−14.4 pts vs week 35 · target < 5%

Dispatch API p95 latency

188 ms

−17 ms vs week 35 · target < 250 ms

Mean time to restore

22 min

1 incident (SEV-3) · target < 30 min

Open SEV-1/SEV-2 actions

−5 vs week 35 · target 0

Deploys per weekTarget (20)

Production deploys per week, all services. Week 35 reflects the two-day deploy freeze after INC-2026-0827. Source: deploy log, query deploys-by-week.

p95 latency (ms)Target (250 ms)

Weekly p95 for POST /v2/jobs/{id}/assign, all regions. Week 35 includes the outage window. Source: Grafana, panel Dispatch API latency.

Workstreams

Workstream	Owner	Status	Progress	Next milestone
INC-2026-0827 corrective actions	Priya N.	Amber	83%	A-4 merged, 2026-09-12
RFC-014 webhook delivery	Tomasz W.	Green	35%	Decision, 2026-09-19
Query-plan CI check (A-5)	Tomasz W.	Green	15%	Snapshot pipeline, 2026-09-19
Observability: pool and queue dashboards	Yuki T.	Green	100%	Done 2026-09-04
PRD-024 platform review (routing provider, NFR-001)	Ines F.	Not started	0%	Review comments, 2026-09-16

Risks and issues

Risk or issue	Status	Trend	Mitigation	Owner
RFC-014 decision slips past 2026-09-19, pushing phase 0 into the October change-freeze window	Open	Rising	Review session booked 2026-09-16; open questions assigned with dates; escalation to Ines F. on 2026-09-17 if unanswered	Tomasz W.
Release team review capacity delays pipeline changes (caused the A-4 slip)	Open	New	Ask below; interim: platform lead granted temporary approver rights for `pipeline/`	Priya N.
Production-sized snapshot for the query-plan CI check needs 400 GB of storage not in this quarter's budget	Open	Unchanged	Sampled snapshot (10% of rows, same statistics) under test; decision on full snapshot by 2026-09-19	Tomasz W.
Repeat of pool exhaustion before A-4 and A-5 land	Closed	Falling	Index, 2 s timeout, and 80% warning shipped; risk accepted for the remaining window	Priya N.

Asks and decisions needed

Answer RFC-014 questions Q1 to Q4 in the review session or in the document. Dana Okafor, Marcus Bell, Ines Fontaine, Priya Natarajan · by 2026-09-19
Add a second release-team approver for pipeline/ changes, or make the temporary platform approver permanent. Head of Engineering · by 2026-09-12
Approve 400 GB of snapshot storage for the query-plan CI check, or accept the sampled-snapshot approach with its known gap on skewed tables. Ines Fontaine · by 2026-09-19

Next period (week 37)

A-4 canary relative-latency gate merged and active on all pipelines; INC-2026-0827 actions 6 of 6 done except A-5 (due 2026-10-03).
RFC-014 review session held 2026-09-16; decision recorded by 2026-09-19; phase 0 migration PR open.
Query-plan CI check running in advisory mode on the sampled snapshot for dispatch-api.
Platform review comments on PRD-024 delivered to Dana Okafor.
Status expected: green, if asks 1 and 2 are resolved.

Appendix: incidents this period

Source: incident tracker, filter `opened:2026-08-31..2026-09-06`.
Incident	Severity	Duration	Summary	Follow-up
INC-2026-0903	SEV-3	22 min	Report export queue stalled after a worker deploy with a missing environment variable; no customer-facing impact beyond delayed CSV emails.	Deploy template validates required variables (done 2026-09-04)