Keeping Raw Data You May Be Asked to Delete
Erasure in a BigQuery medallion lakehouse: keep emails out of bronze, delete in two stages, and compute when erased copies leave the pipeline's storage.
Tag
Data
Multi-Cloud Retail Without the Mess
21 minutes
Many QSR and retail estates run identity in Microsoft Entra ID, back-office systems on SQL Server in Azure, the data platform on Google …
Thousands of Sites, One Pipeline
22 minutes
Why operations and analytics report different sales for the same store-day, and how a medallion pipeline brings a multi-site network to one …
The Loyalty Backend Is a Data Engineering Problem
10 minutes
Loyalty points are a balance-sheet liability as well as a marketing tool. Following one point through enrolment, earning, accrual, …
Your Lab Notebook Is a Data Product
8 minutes
When a licensing partner's diligence team asks for the data behind a preclinical programme, many research organisations find the evidence …
Chain of Custody for Trial Data
9 minutes
One revised LDL-C result shows where custody of clinical trial data breaks: at the handovers between laboratory, CRO, and sponsor. What 21 …
Legacy Modernisation That Meets the Audit Bar
11 minutes
A modernised government system can work perfectly and still leave its auditors short of evidence, because the history the old system kept …
Arkhe: An Ontology Language for AI Systems
7 minutes
Two weeks ago I argued that your ontology is the asset and you should own it in a neutral, versioned form. Arguing is cheap. Arkhe v0.2 is …
Trust Is an Engineering Output
9 minutes
Being correct is not the same as being able to prove it. This final post in The Engineering Underneath argues that trust is an engineering …
Telemetry Is Becoming the Business
10 minutes
Operational telemetry has moved from a back-office by-product to one of the most valuable assets a business owns. This fourth post in The …
Activation Is the Hard Part
9 minutes
Most marketing organisations spend on platforms that produce insight and starve the systems that act on it. This post argues that …
Data Costs: Are You a Victim of Your Own Success?
11 minutes
A high-frequency analytical query running against raw object storage scans the entire historical dataset every cycle. The cost compounds …
Why Most Marketing Dashboards Lie
7 minutes
Two marketing reports from the same week, drawn from the same source system, can show opposite trends and both be correct. This post sets …