From a Leaked Pipeline Credential to Bulk Data
A service account key made before your org blocked key creation still works. How one legacy dbt key reaches BigQuery, and the inventory that finds keys like it.
Tag
Data
What Your Agents Inherit From Your Data Engineering
25 minutes
An agent can only explain what your data platform can explain. How business rules, their history, and corrections are engineered from the …
Native Tables, Managed Iceberg, or Iceberg in an Open Catalog
58 minutes
Choosing storage for a BigQuery lakehouse layer by layer: what native tables, managed Iceberg, and Iceberg in an open catalog each give up, …
Keeping Raw Data You May Be Asked to Delete
91 minutes
Erasure in a BigQuery medallion lakehouse: keep emails out of bronze, delete in two stages, and compute when erased copies leave the …
Multi-Cloud Retail Without the Mess
21 minutes
Many QSR and retail estates run identity in Microsoft Entra ID, back-office systems on SQL Server in Azure, the data platform on Google …
Thousands of Sites, One Pipeline
22 minutes
Why operations and analytics report different sales for the same store-day, and how a medallion pipeline brings a multi-site network to one …
The Loyalty Backend Is a Data Engineering Problem
10 minutes
Loyalty points are a balance-sheet liability as well as a marketing tool. Following one point through enrolment, earning, accrual, …
Your Lab Notebook Is a Data Product
8 minutes
When a licensing partner's diligence team asks for the data behind a preclinical programme, many research organisations find the evidence …
Chain of Custody for Trial Data
9 minutes
One revised LDL-C result shows where custody of clinical trial data breaks: at the handovers between laboratory, CRO, and sponsor. What 21 …
Legacy Modernisation That Meets the Audit Bar
11 minutes
A modernised government system can work perfectly and still leave its auditors short of evidence, because the history the old system kept …
Arkhe: An Ontology Language for AI Systems
7 minutes
Two weeks ago I argued that your ontology is the asset and you should own it in a neutral, versioned form. Arguing is cheap. Arkhe v0.2 is …
Trust Is an Engineering Output
9 minutes
Being correct is not the same as being able to prove it. This final post in The Engineering Underneath argues that trust is an engineering …
Telemetry Is Becoming the Business
10 minutes
Operational telemetry has moved from a back-office by-product to one of the most valuable assets a business owns. This fourth post in The …