Our Blog

Is this row valid, is this table healthy, is this pattern unusual? Three questions, asked at different points in the pipeline, with different thresholds and different alerts.

To answer a business question when it arrives, the data must have been collected earlier. A lifecycle that keeps by default, with overrides and a record of every disposal.

Searching an AWS data lake for ‘Elasticsearch’ now means Amazon OpenSearch Service. What a search engine adds to S3 and Athena, where it does not belong, a reference setup, the cost traps, and a checklist for deciding if your lake needs one.

Publishing an illustrated, narrated series used to take a team. Now it takes one person with the idea and one who knows agentic workflows. How DataPhoenix worked as the forward-deployed engineer for a travel series: map the work, build inside the client’s operation, ship, stay while it breaks, hand over.

Picking a data platform is one of the most expensive decisions a data team makes. AWS native, Databricks and Snowflake aren’t three flavours of the same thing: one gives you building blocks, one a complete lakehouse, one a managed SQL engine. Here’s how to tell which one fits before you commit…

AI products that impress in a demo have a habit of going quietly wrong in production: stale answers, numbers that won’t reconcile, PII flowing unmasked into a model. The engineers aren’t at fault. The data underneath was never treated as a first-class engineering concern, and that’s a data platform problem, not an AI one…

Told to “get it into the lake”? Here’s what actually happens between that conversation and the point where an analyst or an AI agent can trust the data: the medallion delivery framework, Bronze to Silver to Gold, and the gates a new source has to pass at each layer…

A data lake holds raw data of every kind. A data warehouse answers business questions quickly. Lake house architecture on AWS unifies the two, so the same data can serve both without being copied between them…

In the previous post, I explained what a data lake is and its main business benefits. This time around we will have a brief look at related and most commonly adopted data architectures…

In this article, we will overview the different layers of a well-built data lake, as well as highlight common pitfalls and challenges to consider when implementing a data lake architecture…

When I tackled my first data lake project it was difficult to understand what it actually is. I finally grasped the concept by understanding that there are three types of data: structured, semi-structured and unstructured. In layman’s terms, a data lake is a data management solution that stores and processes raw data at scale…

This guide gives you the fundamental knowledge needed to understand what a standard data lake is, its main architecture elements and how you can use AWS services to manage your data securely and effectively. With a base-level understanding, you can build your expertise in data lakes and implement them across business functions.

If you’ve been working with me for a while, you’ll probably know some of the great work I’ve been involved in in the realms of data solutions like data lakes, data mesh, AWS and DevOps…