Keep it by default: a data lifecycle for a data platform

Most platforms start retention from the wrong end, asking how soon data can be deleted. The better question is what the business will want to ask next year that it cannot ask today. To answer a business question when it arrives, the data has to have been collected earlier, and the cost of holding it is small next to the cost of an answer that cannot be given.

Why keep by default

Many source systems return only the current state. They hold no history of their own, so the platform’s raw layer is the only record of what the estate looked like a year ago. A question about how a trend developed, what changed before an incident or how utilisation moved across a year cannot be answered from data deleted this year. A short replay window assumes the source can be read again, and for current-state sources it deletes the only copy. Nothing should be disposed of because time passed. Removal needs a reason: a legal or privacy requirement, a review outcome or the source’s own policy.

One default, overridable per source

A single lifecycle covers all data, and a source carries its own policy where its requirements differ. Where volume makes the default too expensive, the customer decides what to shorten, and the default stands until they do. Where rules conflict, precedence is fixed: a legal or privacy requirement first, then the source policy, then the platform default. Retention is not set by protection class. Class governs who may see data, not how long it stays useful.

States and end-of-life actions

Every dataset is in one of a small number of states: active, archived, reviewed, and then kept, downsampled or purged. Three actions end an active life. Archive moves data to lower-cost storage and keeps it readable. Downsample reduces resolution and cannot be undone. Purge deletes, and cannot be undone either. Because the last two destroy detail, each is recorded: who decided, what was destroyed, the consumer’s agreement where there is one, and whether the source could reproduce the data. Destroying what no source can rebuild is a different decision from destroying what can be re-ingested.

Rules that make it work

Archived data must stay readable without a restore workflow, and it carries a review date, not a purge date. Age is the date in the data, not the age of the storage file. Downsampling applies to snapshots only, and event records are never thinned. Use sets the lifecycle, not format, so structured and unstructured data follow the same periods, and anything landed unstructured carries a lifecycle decision at onboarding with enough context to be read later: its source, date, format and what it is believed to hold.

Privacy and legal requirements override everything

A shorter statutory or privacy period wins, deletion requests reach archived copies, and a legal hold suspends disposal with its scope and expiry recorded. Where a security function is engaged, it signs off retention decisions. Where it is not, the team decides and records the decision, and its absence does not stop delivery.

Ownership and review

Periods run from the date each source was onboarded, so that date must be recorded for every source. The data engineer holding the source is accountable for it, and ownership passes at handover. Lifecycle is reviewed on the regular cycle in which each source is re-confirmed with its owner. This design is part of data governance and of the data architecture and strategy work.