Q3 2026 engineering review
We shipped more, broke less, and learned one expensive lesson. Q4 turns that lesson into pipeline gates and finishes webhook delivery.
Ines Fontaine, Engineering Manager · 2026-09-15 · Leadership team and engineering leads
Agenda
- Q3 in five numbers
- Reliability: one SEV-2 and what it taught us
- Delivery: what shipped and what slipped
- Q4 priorities
- Three asks
Q3 in five numbers
production deploys
Q2: 231 (+24%)
change failure rate
Q2: 6.1%, target under 5%
SEV-2 incident, 41 minutes
Q2: 3 incidents, 2 h 10 min total
Dispatch API availability
SLO 99.9%
Dispatch API p95 latency
Q2 exit: 205 ms, target under 250 ms
Sources: deploy log, incident tracker, Grafana SLO dashboard, 2026-07-01 to 2026-09-06.
Reliability
Fewer incidents, faster recovery, and one that should not have reached production.
Incidents fell from three to one, and recovery time halved
| Quarter | SEV-1 | SEV-2 | SEV-3 | Customer-visible minutes | Median time to restore |
|---|---|---|---|---|---|
| Q1 2026 | 1 | 3 | 9 | 312 | 58 min |
| Q2 2026 | 0 | 3 | 7 | 130 | 44 min |
| Q3 2026 | 0 | 1 | 5 | 41 | 22 min |
The 27 August outage was a pipeline gap, not a people problem
What happened
- A migration added a column; a new query filtered on it without an index
- Canary passed: its table is one ninth the size of EU's
- Pool exhausted at EU peak; 38% of assignments failed for 41 minutes
What changed
- Partial index and 2 s pool timeout (shipped)
- Canary holds on a 25% relative p95 rise (in review, 2026-09-12)
- CI explains new queries against a production-sized snapshot (2026-10-03)
Absolute latency budgets do not catch data-dependent regressions. Relative gates and realistic data do.
Delivery
Eleven of thirteen committed items shipped; two slipped for reasons we can name.
Shipped in Q3
Customer-facing
- Technician mobile app offline mode (1,900 daily users on day 30)
- Dispatcher day view redesign, 14% faster planning in usability tests
- Customer arrival-window SMS, 92% delivery within 60 s
- Public API v2.4: bulk job import, 40 customers migrated
Platform
- Regional deploys with automatic canary promotion
- Pool, queue, and SLO dashboards for every service
- Deploy template validation (caught 6 misconfigurations in September)
- RFC-014 webhook delivery design at v3, decision due 2026-09-19
Two items slipped, both for review capacity
- Canary relative gate: one week late; pipeline changes need a release-team approver and there is one person.
- Webhook delivery build start: waiting on four open questions in RFC-014; if decided by 2026-09-19, phase 0 starts 2026-09-22 as planned.
Both slips trace to a single approver or decider. Ask 2 on slide 13 addresses it.
Q4 priorities
Finish the reliability work, ship webhook delivery, and prepare the platform for route optimisation.
Q4 commitments
| Priority | Outcome by 2026-12-19 | Owner | Confidence |
|---|---|---|---|
| 1. Webhook delivery (RFC-014) | General availability; failed-and-never-retried under 0.05% | Tomasz W. | High |
| 2. Pipeline gates | Relative canary gate and query-plan CI on all services | Priya N. | High |
| 3. Route optimisation platform (PRD-024) | Drive-time provider integrated; nightly batch under 10 min for 60 technicians | Ines F. | Medium |
| 4. SEV-3 reduction | Three or fewer in the quarter | Priya N. | Medium |
Q4 risks
- October change freeze (2026-10-26 to 2026-11-06) removes two weeks from priorities 1 and 2. Plan already excludes them.
- Provider contract for drive times: legal review not started. Decision date 2026-10-15.
- Hiring: one backend role open since July; priority 3 assumes it is filled by November.
Summary and three asks
- Q3: more deploys, fewer failures, one incident with a structural fix in progress.
- Q4: finish gates, ship webhook delivery, ready the platform for routing.
- Both Q3 slips came from single points of approval.
- Approve a second release-team approver for pipeline changes. Head of Engineering · by 2026-09-19
- Start legal review of the drive-time provider contract. COO · by 2026-09-26
- Confirm the October freeze dates so Q4 plans can lock. VP Operations · by 2026-09-19
Thank you
Supporting documents: INC-2026-0827 RCA v1.2, RFC-014 v3, PRD-024 v0.4, and the weekly platform status reports.
Press n to show speaker notes. Print for one slide per page.