AWS Native, Databricks or Snowflake: Choosing the Right Data Platform

Picking a data platform is one of the most expensive decisions you’ll make. Get it right and your team moves fast, governance holds, and your AI products have something solid underneath them. Get it wrong and you’re spending years on a stack that creates more problems than it solves.

If you’re on AWS — or planning to be — this is what you need to know before committing to a platform.

Aerial view of three building plots at different construction stages, representing data platform architecture choices. — Photo by Iain on Unsplash

The three realistic options

Not all data platforms are the same thing. AWS gives you building blocks. Databricks gives you a complete lakehouse platform that runs on top of AWS. Snowflake gives you a managed analytics engine. Understanding that distinction before choosing is the difference between a good decision and an expensive one.

AWS native: maximum flexibility, maximum engineering overhead

The AWS-native stack — S3, Glue, Athena, Lake Formation, Redshift — is powerful, and there’s no platform licence fee. But you’re building everything yourself. Data cataloguing, governance, lineage, quality checks, column-level access control — your team builds all of it from scratch, then maintains it.

That works well if you’ve got a strong, experienced data engineering team and needs that off-the-shelf tools can’t meet. It gets expensive fast when the same people building your pipelines are also responsible for the catalogue and governance layer — that’s months of work before you’ve done anything useful with the data.

Databricks: the turn-key lakehouse

Databricks is what happens when you take Delta Lake — the open table format it invented and which is now the industry standard for lakehouse architecture — and build a complete platform around it. Unity Catalog gives you cross-workspace data discovery, lineage, and access control in a single place. MLflow handles experiment tracking and model serving. Job orchestration, notebook environment, auto-scaling compute, and data quality monitoring are all included.

The reason data engineers describe it as turn-key is accurate in the ways that matter: the things that take months to build on AWS native — catalogue, lineage, governance, quality — are available on day one. You are configuring and using, not building.

Critically, Databricks runs on your AWS account. Your data stays in your S3 buckets. You are not leaving AWS; you are adding a managed platform layer on top of it. And because Delta Lake is an open format, you are not locked in to Databricks — the data is readable by Athena, Spark, and any other compatible engine.

Snowflake: SQL-first analytics

Snowflake excels at making large-scale SQL analytics accessible without deep infrastructure knowledge. Compute and storage are separated, scaling is automatic, and sharing data across organisational boundaries is genuinely straightforward.

What it does not do well is the full data platform picture. ML and AI workloads, complex transformation pipelines, streaming data, and enforcement of governance at the data layer all require additional tooling. If your primary use case is analytics and reporting for a SQL-literate team, Snowflake is excellent. If you are building a governed data platform that will power AI products as well as analytics, you will reach the limits of Snowflake quickly and find yourself bolting on tools to fill the gaps.

What the numbers actually look like

This is where the conversation surprises people. AWS native has no licence fee, which sounds like the cheaper option. The real cost is always total cost of ownership: infrastructure plus the engineering time to build and maintain what the platform does not give you.

	AWS Native	Databricks on AWS	Snowflake on AWS
Platform licence	None	£5k–15k/month	£3k–10k/month (credit-based)
Infrastructure	£2k–4k/month	£1k–2k/month (your EC2/S3)	Minimal (external S3)
Engineers needed	2–3 (building + maintaining)	1–2 (configuring + using)	1–2
Engineer cost	£15k–20k/month	£7.5k–10k/month	£7.5k–10k/month
Typical total	£17k–24k/month	£13.5k–27k/month	£10.5k–20k/month
Governance out of box	No — must build	Yes (Unity Catalog)	Partial
ML/AI readiness	Requires additional tooling	Native (MLflow, Feature Store)	Limited
Delivery speed	Slow (build everything)	Fast	Fast for analytics

Illustrative figures for a medium-sized data platform, 1–5TB active data, UK contractor rates.

The practical conclusion: Databricks often delivers a lower total cost of ownership than AWS native for organisations that do not already have a mature data engineering practice. The licence replaces a significant portion of engineering time rather than adding to it. You are buying built capability instead of building it yourself.

Who needs what

Choose AWS native if you have a large, experienced data engineering team; you have specific compliance or integration requirements that demand custom architecture; you are deeply invested in AWS operational tooling and want no additional vendor dependency.

Choose Databricks if you need governance, catalogue, and lineage without a long build phase; ML and AI are on your roadmap; your team is smaller and needs to deliver quickly; you want open formats (Delta Lake) without cloud lock-in at the data layer. For most organisations building a modern data platform today, Databricks on AWS is the strongest default choice.

Choose Snowflake if your primary use case is SQL analytics and BI; your team is more analyst than engineer; data sharing with external partners is a key requirement. Be clear-eyed about what you will need to add when the AI use cases arrive.

You do not need to start with a platform at all

One of the most underused options is the simplest one: start with storage only.

S3 costs around £18 per terabyte per month. Drop your data there — raw, unprocessed, exactly as it comes off the source systems — and you’ve already done the most important thing: it’s in a place you own and control. No pipelines. No schema decisions. No catalogue. Just the data, sitting there, ready when you need it.

This is the foundation of the medallion architecture Bronze layer, and it is a legitimate starting point in its own right. You can query it ad-hoc with Athena for almost nothing. You can see what you have, understand the volume and variety, and make platform decisions based on actual data rather than assumptions.

From there you build. A governance layer when you need one. Structured ingestion when the volume justifies it. A catalogue when teams start struggling to find things. An AI layer when the data is clean enough to support it. Each stage adds complexity and cost — but only when you actually need it, not upfront as a prerequisite.

Organisations that start this way almost always make better platform decisions. They know what their data actually looks like before they commit to how they’ll process it. That’s not a small advantage.

The right starting point is not a platform

Before you pick a platform, answer three questions honestly: what does your data actually look like right now, what can your team realistically build and own, and what do you need from AI and analytics in the next two years. Skip that step and you’ll be migrating everything eighteen months from now.

We work with organisations on AWS on both AWS-native builds and Databricks. The platform matters less than choosing the right one for your situation — and building it to last.

Why Every Enterprise AI Product Needs a Professional Data Platform

There is a pattern that repeats itself across enterprise AI projects. A team of capable engineers builds an AI product — a chatbot, a command centre, an intelligent dashboard — that works impressively in a demo. It queries multiple data sources, answers natural-language questions, surfaces insights the business has never had before. The budget holder signs off. The rollout begins. And then, quietly, things start going wrong.

The AI gives a confident answer that turns out to be six weeks stale. A cost figure that should add up across three cloud providers is consistently off and nobody can explain why. A field that used to contain a location string now sometimes contains a person’s name and email address, and it has been flowing unmasked into the model for months. A source system changed a column format in a quarterly release and the integration kept running, silently producing wrong results.

The engineers are not at fault. They built what they were asked to build. The problem is architectural: the data underneath the AI was never treated as a first-class engineering concern.

AI brain formed from circuit traces above a grid of data blocks, representing enterprise AI built on a structured data foundation. — Photo by Steve A Johnson on Unsplash

This is not an AI problem. It is a data problem.

Any product that consumes data from multiple disparate sources — infrastructure platforms, CRMs, financial systems, operational databases, third-party APIs — faces the same underlying challenge. The sources were not designed to feed your product. They change without warning. They use different schemas, different identifiers, different definitions of the same concept. “Cost” in one system is not “cost” in another. “Active” in your CRM does not mean the same thing as “active” in your billing platform.

Software engineers are skilled at solving this problem once, for one source, at one point in time. They connect to the API, map the schema, write the integration, and move on. What they are not structured to handle — and should not be expected to handle — is the ongoing governance, classification, and maintenance of a multi-source data estate at enterprise scale. That is a different discipline entirely.

What actually needs to happen

Before an AI model can reliably answer questions about your data, several things need to be true that a software engineering team alone is not equipped to deliver.

Data classification. Someone needs to sit with each data owner, understand what every field actually contains, and assign it a sensitivity classification. This sounds straightforward. It is not. A field called resource_tag_owner might contain a department name in one environment and a person’s full name and email address in another. A field called session_duration in a healthcare system might be innocuous or, in combination with other fields, constitute a clinical record. Identifying this requires expertise in data governance and the right questions to ask — not the ability to read an API specification.

PII handling. Once PII is identified, decisions need to be made: mask it, tokenise it, restrict access to it, or exclude it from the dataset altogether depending on who is consuming it. Those decisions need to be documented, auditable, and enforced at the data layer — not in the AI prompt, not in the application code, not informally agreed between two engineers in a Slack thread.

Persona-based access. Different users of your AI product should see different data. A manager asking about team costs should not see the same rows as an administrator with full estate access. An external auditor should see less than an internal engineer. Enforcing this at the application layer — filtering in the AI model or the API response — is fragile and nearly impossible to audit. Enforcing it at the data layer, through governed datasets scoped to each consumer persona, is the correct architecture. The AI queries only what it is permitted to see. The boundary is in the data, not the prompt.

Minimum necessary data per use case. The AI powering your product does not need every column from every source system. It needs a well-defined, tested, governed dataset scoped to its specific use cases. Publishing the full raw estate to an AI model and letting it figure out what is relevant is how you end up with incorrect answers, unnecessary PII exposure, and models that are expensive to run and slow to respond.

Freshness and quality contracts. Every data source needs a documented commitment on how current the data is and what quality guarantees it carries. When a source goes stale, the AI needs to know — and say so — rather than answer confidently on outdated information. When a source changes its schema, the change needs to be caught at ingestion and validated against a contract before it reaches the model.

The right team structure

The AI team — the engineers building the models, the interfaces, the reasoning layer — should be consumers of governed, classified, scoped data. They should not be interviewing source system owners about schema semantics. They should not be making PII classification decisions. They should not be writing and maintaining ingestion pipelines for a dozen different enterprise systems while simultaneously building the product.

That work belongs to a data platform team: engineers who understand data governance, who know what questions to ask data owners, who build and maintain the ingestion, classification, and serving layers that make the AI’s data trustworthy. The output of that team — clean, governed, persona-scoped datasets — is what the AI consumes.

This separation is not bureaucracy. It is what makes the AI product maintainable when a source changes, auditable when a compliance question is asked, and reliable when it moves from demo to production.

The investment case

Budget holders are often asked to fund the AI team. The question worth asking is: what is the AI team building on? If the answer is direct integrations to source systems, maintained by engineers who are also building the product, the risk is being carried silently in the data layer and will surface as quality and trust issues at the worst possible time — typically when a senior stakeholder asks a question the AI answers wrongly, confidently, in front of the wrong audience.

A professional data platform — built by engineers who specialise in it, following established patterns for ingestion, governance, and data contracts — is the investment that makes the AI investment reliable. The two teams are not competing for the same budget. One enables the other.

The organisations that get this right treat the data platform as the product that serves the AI team, with the same rigour and ownership they apply to any other critical piece of infrastructure. The organisations that get it wrong discover, usually twelve to eighteen months in, that they are maintaining a fragile web of undocumented integrations that nobody fully understands and that the AI has learned to work around rather than rely on.

From Raw to Ready: Delivering a New Data Source on a Modern Data Platform

If you have been asked to onboard a new data source into a data platform, you may have been handed a board with a handful of vague tickets and told to “get it into the lake.” What actually needs to happen between that conversation and the point where a business analyst or AI agent can confidently query the data? This post walks through the delivery framework that experienced data engineering teams use, built around the medallion architecture.

What is the medallion architecture?

The medallion architecture — also called multi-hop or Bronze-Silver-Gold — is a data design pattern that organises data into three progressive layers of quality and readiness.

Bronze holds raw, unmodified data exactly as it arrives from the source. Nothing is changed, nothing is discarded. It is the permanent record of what came in and when.

Silver is the curated layer. Schemas are enforced, business keys are resolved, duplicates removed, and quality validated. A consumer can use Silver data without needing to understand the source system at all.

Gold is shaped for a specific consumer: a KPI dataset, a dashboard aggregate, a model input, or an AI agent feed. Gold exists to serve a committed use case, not to store data generically.

The pattern was introduced by Databricks, who also created Delta Lake — an open-source transactional storage format that adds ACID guarantees, schema enforcement, and time travel on top of cloud object storage. The medallion pattern was designed as the recommended way to organise data across Delta tables. Since then it has been adopted well beyond the Databricks ecosystem: on AWS, the same pattern is implemented with Apache Iceberg on S3, and the layered logic applies regardless of which storage format is underneath. The separation of concerns is what matters — ingestion logic lives in Bronze, business logic lives in Silver, and consumer contracts live in Gold. Each layer can be rebuilt from the one below independently.

Medallion architecture diagram showing Bronze raw ingestion, Silver curated, and Gold consumer-ready data layers. — Medallion architecture — Bronze, Silver, Gold layers. Source: Databricks

Why it matters for the business

The medallion architecture is not an engineering preference — it is a response to a business problem. Organisations that skip this structure typically end up with pipelines that are tightly coupled to their source systems: when the source changes a schema, the dashboard breaks. When a new team wants the same data shaped differently, someone builds a parallel pipeline from scratch. When an auditor asks where a number came from, nobody can answer with confidence.

Separating data into layers solves each of these directly.

Reliability and trust. Because Bronze preserves the raw record and Silver enforces a validated contract, business users know exactly what quality guarantee they are working with. A metric produced from Gold can be traced back through Silver to Bronze and ultimately to the source event. That traceability is what turns data from something people argue about in meetings into something they make decisions from.

Speed to new use cases. Once a source is in Silver, adding a new consumer does not require touching the ingestion pipeline. A new Gold model, a new dashboard, or a new AI agent input can be built on top of the existing curated layer in days rather than weeks. The investment in onboarding a source properly pays compound returns every time a new use case is added.

Cost control. Raw data is stored once in Bronze. Transformations are layered, not duplicated. Without this discipline, organisations frequently find they have the same data copied four times across four pipelines built by four teams who did not know about each other. Each copy has slightly different logic, and none of them agree.

Data classification and compliance. Every data source that enters a platform needs to be classified before it reaches consumers. Classification answers the questions that legal, security, and compliance teams will eventually ask: what kind of data is this, how sensitive is it, who is permitted to access it, and what obligations come with holding it? Common classification tiers range from public and internal through to confidential and restricted, with each tier carrying different access controls, retention rules, and audit requirements.

PII — Personally Identifiable Information — sits at the most sensitive end of that spectrum and deserves specific treatment. PII includes any data that can identify an individual directly or in combination: names, email addresses, device identifiers, location data, and in many cases transactional records that can be linked back to a person. Under GDPR, CCPA, and equivalent frameworks, the moment PII enters your platform you have legal obligations around consent, storage, access, retention periods, and the right to erasure. Those obligations do not disappear because the data is in a lake.

The medallion architecture handles this well because classification happens at Silver, where the data is understood well enough to make the call. Fields can be tagged in the catalog, masked or tokenised before they reach Gold, and access grants can be scoped so that an analyst querying a Gold model never touches raw PII they have no business reason to see. This is significantly harder to enforce in a flat, undifferentiated data lake where everything lands in the same bucket and access is managed with broad permissions.

What is a data contract and why does it matter?

A data contract is a formal, versioned agreement between a data producer and a data consumer. It specifies what data will be delivered, in what shape, at what quality, and on what schedule. Unlike an informal handshake or a wiki page that drifts out of date, a data contract is machine-readable and enforceable — if the producer breaks the contract, the pipeline fails loudly rather than silently corrupting downstream outputs.

The concept addresses a specific and common failure mode in data platforms: a source team changes a column name or drops a field, the ingestion pipeline keeps running, and three weeks later a finance team discovers their monthly report has been wrong since the change. By the time the issue surfaces, the trust damage is done and the investigation is painful.

In a medallion architecture, data contracts live at two critical points. The Silver contract defines the curated output of the Bronze-to-Silver transformation: the expected keys, schema, data types, null constraints, and quality guarantees that downstream consumers can rely on. It is owned by the data engineering team and represents their commitment to consumers. The consumer contract (or Gold contract) defines what a specific use case — a dashboard, an AI model, a reporting dataset — will receive, and is jointly agreed between the data team and the consumer owner.

Why use contracts instead of documentation? Documentation describes intent; a contract enforces it. When a Silver contract is in place, any transformation that would break it is caught at the validation step in Stage 4 rather than discovered by a consumer after the fact. New consumers onboarding to an existing Silver source can read the contract and know exactly what they are getting without needing to interrogate the source system or ask the engineering team. And when the source itself changes, the contract makes the scope of the impact immediately visible: every downstream consumer whose contract depends on the changed field knows they are affected.

Practically, a data contract includes the schema (field names, types, nullability), business keys and uniqueness constraints, expected row volume and freshness SLA, data classification tier and any PII fields, and the owner and agreed review cadence. It does not need to be a complex document — a well-structured YAML file checked into version control alongside the pipeline code is enough to deliver most of the benefit.

The delivery stages

A well-run source onboarding follows seven stages in a fixed order. The most common cause of rework is skipping or merging the early ones.

Stage 1: Discovery and approval. Before a single line of code is written, confirm why the source matters, who owns it, and who will consume the data. Light profiling — row counts, null rates, schema shape — should happen in an isolated environment. The stage ends with a stakeholder approval or a deliberate stop. A source without a committed consumer should not be onboarded.

Stage 2: Secure access and platform setup. Establish the service principal, secrets management, catalog namespace, storage paths, and environment grants. This work is unglamorous but blocking failures here cost days later.

Stage 3: Bronze availability. Build the extractor or loader, land the raw data, schedule the job, and add basic quality and freshness checks. At this point the source is visible in the platform but not yet ready for business use.

Stage 4: Silver availability. Define and publish the Silver data contract — expected keys, schema, data types, null constraints, and quality guarantees — then build the Bronze-to-Silver transformation and add reconciliation checks. The contract is the commitment to consumers; the transformation is how you honour it. This is often the most time-intensive stage because it is where source-specific complexity surfaces: late-arriving records, schema drift, unexpected nulls in business keys.

Stage 5: Governance and operational readiness. Classify all fields by sensitivity tier, identify and tag any PII, assign data ownership in the catalog, and document lineage from source to consumer. Then add the operational layer: alerting, a runbook covering stale data and backfill procedures, and sign-off from the source owner. Teams that defer this until the end of a milestone consistently find they cannot support the source in production and cannot demonstrate compliance if asked.

Stage 6: Gold or consumer readiness. Agree and publish the consumer data contract — what the consumer will receive, in what shape, and on what schedule — then build the output and validate it with the actual consumer. Without this step the source is delivered but not used.

Stage 7: CI/CD promotion. Formalise the pipeline promotion path between layers so the source can be rebuilt or backfilled without manual intervention.

What goes wrong

The most common failure mode is starting Bronze implementation before source ownership and access are confirmed. The second is onboarding a source without a committed consumer: the data lands in Silver, nobody queries it, and months later nobody can remember why it was a priority.

Other recurring mistakes: mixing platform foundation work with source ingestion in the same delivery item, writing stories that describe activities rather than observable outcomes, and leaving monitoring and runbook work until the final week of a sprint.

Effort to expect

A well-understood source with clear ownership and a defined consumer takes around 20 engineering days end to end. That excludes waiting time — access approvals, vendor responses, source readiness delays — which should be tracked separately. Sources where ownership, access method, volume, or rate limits are unclear should have discovery separated from implementation before the full onboarding is committed.

Unleashing Data Power with Lake House Architecture on AWS

Source AWS — https://aws.amazon.com/blogs/big-data/build-a-lake-house-architecture-on-aws/

Introduction
In our data-centric world, organizations are embracing contemporary data architectures to manage, process, and analyze vast volumes of information. The Lake House Architecture, a merger of data lake and data warehouse, is one such model. This article delves into the advantages of AWS-based Lake House Architecture and how it empowers businesses to fully harness their data.

Discerning business leaders recognize data’s role in driving growth and competitive edge. By Utilizing Lake House Architecture, you can access a consolidated data view, expedite new data onboarding, and distribute analytics throughout your enterprise. This article offers an overview of AWS Lake House Architecture and its impact on businesses.

Lake House Architecture Explained

The Lake House Architecture is a modern data infrastructure that unifies data lakes and data warehouses at the storage and catalog levels. AWS implementation eliminates the need to move data between these systems, accelerating data processing by avoiding redundant data movement, duplication, and ETL code.

Lake House Architecture in Action

The data warehouse and data lake are natively integrated at storage and catalog layers, presenting a unified Lake House interface for processing and consumption layers. Components of these layers, such as SQL or Spark, can access data stored in the Lake House storage layer, covering both the data warehouse and data lake. Native ETL or ELT pipelines in the data warehouse can merge flat relational data with complex, hierarchical structured data in the data lake.

Deep dive into Lake House architecture layers on AWS

A Lake house reference architecture as proposed by AWS. Source AWS — https://aws.amazon.com/blogs/big-data/build-a-lake-house-architecture-on-aws/

Data Ingestion Layer is responsible for collecting and ingesting data from various sources, such as databases, applications, and IoT devices, into the Lake House system. AWS offers a range of services for data ingestion, including Amazon Kinesis for streaming data and Amazon MWAA(Apache Airflow) for batch processing. These tools enable seamless data integration while ensuring data consistency, security, and privacy.

The Catalog Layer serves as the central metadata repository for all data stored in the Lake House. It maintains schema definitions, data lineage, and other metadata for seamless data discovery and governance. AWS Glue Catalog, a managed data catalog service, integrates with other AWS services such as Amazon S3, Amazon Redshift, and Amazon Athena to provide a unified catalog for your data lake and data warehouse.

The Data Processing Layer is responsible for transforming, cleaning, and enriching data to make it suitable for analysis. AWS offers a wide range of tools to process data in the Lake House. For instance, AWS Glue for serverless data processing, Amazon EMR for big data processing with Apache Spark, and Amazon Redshift for data warehousing tasks. These services facilitate efficient data processing, enabling you to derive valuable insights from your data.

Data Consumption Layer encompasses tools and services that allow users to access, analyze, and visualize data stored in the Lake House. AWS provides various data consumption tools, such as Amazon Athena for serverless querying, Amazon Redshift Spectrum for extending data warehouse queries to the data lake, and Amazon QuickSight for creating interactive BI dashboards. These services empower different personas within an organization to access and analyze data, promoting data-driven decision-making across the enterprise.

By leveraging these layers within the Lake House Architecture organizations can create a robust, scalable, and cost-effective data management solution that unlocks the full potential of their data.

As always, for any data platform needs, contact us at DataPhoenix — small end-to-end data solutions.

Data Lake architectures vertigo — do you get it?

In the previous post, I explained what a data lake is and its main business benefits. This time around we will have a brief look at related and most commonly adopted data architectures.

A data lake has just recently become a mainstream solution adopted by businesses who want to stay on top of their data. Digital transformation was only the beginning, in the year 2035 data creation is forecasted to reach 45 times that of the year 2020 (source StatistaCharts). I believe that value extracted from data is growing in line with the growth of its volume and so data platform implementations are following suit. Therefore, it is not a surprise that new data architecture patterns are cropping up and data lake has gone through its first evolutions. There are now three distinct data lake implementations and architectures — centralised ( just data lake ), de-centralised ( data mesh ) and a third an integration of data warehouse and data lake called Lakehouse.

When we talk about data lake we usually refer to centralised architecture, certainly having data all in one place has its benefits and works great with organisations having small to medium data footprints. However, for those with a large production of data across an organisation, a decentralised data architecture is necessary. Enter data mesh, an architecture that organises data by a specific business domain (i.e marketing and sales ). Allowing for more ownership to the producers of a given dataset and allowing better connection with data consumers.

Last but not least, those organisations that already utilise data warehouse and data lake may want to implement Lake House architecture which integrates the two together. From an operational standpoint this results in avoiding issues related to data gravity — unnecessary data movement and data redundancies, business-wise robust analytics allows for faster time to market.

Since we’ve covered architectural pattern types related to the data lake in my next posts we will have a more detailed look into each architecture.

As always, for any data platform needs, contact us at DataPhoenix — a small end-to-end data solutions provider.

Image source AWS.

Understanding the Elements of a Well-Built Data Lake that Translate into Business Value

In this article, we will overview the different layers of a well-built data lake, as well as highlight common pitfalls and challenges to consider when implementing a data lake architecture.

Overview:

A typical data lake allows for decoupling of storage, processing, and analytics. The data is segmented into a landing, raw, trusted, and curated zones to store data depending on its consumption readiness. Each layer should be designed with cost optimisation, flexibility, and security in mind. Let’s take a closer look at what each layer does and which services are typically used.

Ingestion:

AWS allows for a myriad of data producers, and ingestion methods can process structured, unstructured, and semi-structured data loaded with batch or in real-time (e.g., streaming with Kinesis Firehose). The files are stored in the AWS S3 service, which is excellent as it has no storage limit. When data arrives, it can be in any format; this is called schema on read. S3 is also budget-friendly as it delivers automatic storage cost savings when access patterns change with intelligent tiering. Last but not least, it boasts excellent security features, from fine-grain access policies to robust encryption with KMS.

Processing and Transformation:

In a Data Lake or Lake House architecture, the data processing layer is responsible for transforming data into a consumable state through data validation, cleanup, normalisation, transformation, and enrichment. AWS Glue is used to track and catalogue the data. Purpose-built components perform a variety of transformations, big data processing (EMR), and near-real-time ETL. Amazon Athena can be used for quick probing into data with SQL-style queries.

Analytics/Visualisation:

This layer provides data access and consumption primarily through S3. SageMaker allows for machine learning models to be trained and deployed. Visualisations and dashboards can be built using tools like AWS QuickSight or offloaded to Tableau (SaaS service). It’s essential to note that the quality of the analytics and visualisations depends on the quality of the data stored in the data lake.

AWS data lake architecture — DataPhoenix

Common pitfalls:

The main challenge with a data lake architecture is that raw data is stored without the oversight of its contents. For a data lake to make data usable, it needs to have defined mechanisms to catalogue and secure data. Without these elements, data cannot be found or trusted, resulting in a “data swamp.”
A Data Lake is not the answer to all data analytics needs; businesses may need to leverage a data warehouse. When these two are combined, we talk about a Lakehouse architecture.
Governance and access control mechanisms are also necessary to ensure the data being accessed is trustworthy. Often overlooked in the implementation phase, these issues later cause frustration with teams not having enough flexibility to re-process and consume data in a democratic way. When familiar issues with data silos come into play, a data mesh architecture may be the answer.

By understanding the different layers of a well-built data lake, businesses can realise the full potential of their data, enabling advanced analytics, generating insights, and improving decision-making processes.

As always, for any data platform needs, contact us at DataPhoenix — small end-to-end data solutions.

How well do you know a data lake and its main business benefits?

When I tackled my first data lake project it was difficult for me to understand what it actually is. However, I finally grasped the concept with the help of understanding that there are three types of data — structured, semi-structured, and unstructured. In layman’s term, it is a data management solution that stores and process raw data at scale, that doesn’t fit well into SQL databases(structured), data warehouses (structured) or NoSQL(unstructured) solutions. A highlight of difference to other data management solutions is that having the ability to economically store data allows for answering business questions in future that you otherwise may not have had data for.

Robust implementation of a data lake brings a wealth of benefits. Analytics extracted from new sources like log files, data from click-streams, social media, and IIOT can increase revenue by uncovering valuable insights, it allows to make better business decisions faster and bring budget savings. When data is all in one place, storage cost optimised (AWS delivers automatic storage cost savings when access patterns change), catalogued, and democratised(available to everyone) you can expect OpEx costs lowered as well. Moreover, benefits don’t stop there, cloud providers typically focus on security and governance features reducing risks and allowing them to meet the highest compliance standards.

Data Lake architecture. Image source AWS.

Now that you understand what a data lake is and what the main benefits are, you can start preparing your business for robust data analytics. While there are many use cases for data stored in data lakes, I’m particularly excited about machine learning, which is closely related to AI.

In my next article, I will briefly overview data lake architecture on AWS.

As always, for any data platform needs, please contact us at info@dataphoenix.io — a small end-to-end data solutions provider.

DataPhoenix explains everything you need to know about data lakes in 'The Definitive Guide to data lakes on AWS”

Overview

This guide aims to empower you with the fundamental knowledge needed to understand what a standard data lake is, it’s main architecture elements and how you can utilise AWS services for managing your data securely and effectively. With base-level understanding, you can build your expertise in data lakes and implement them across various business functions.

About authors

DataPhoenix provides consulting and professional services focused on delivery of data management platforms in AWS. Based on our in depth experience delivering data platforms to small business and large enterprise we present you “The Definitive Guide to Data Lakes”, a comprehensive resource for decision-makers, CTOs , Data Architects and Engineers. Delve into the world of Data Lakes and understand technical aspects, as well as business benefits.

Index

What is a data lake

Picture yourself working at a company, serving as the system administrator. Your primary responsibility is to ensure that everything runs seamlessly without any hitches. The company delegates tasks among smaller teams, each overseeing a different aspect, such as code development, project management, or database modelling. However, you soon encounter a problem: customers are complaining about slow system performance. Upon investigation, you realise that the database is nearing its capacity and is being used for all functions. It becomes apparent that a “one size fits all” approach is inappropriate. To address this issue, you need to segregate the data into separate components, allowing for more strategic administration.

A data lake serves as the solution to challenges arising from traditional database and data processing techniques, offering a superior approach for modern workloads.

If you find yourself in a similar situation, a data lake could be immensely beneficial. Cloud-based data lake services like AWS can handle storage, analytics, real-time analysis, and machine learning separately.

In essence, a data lake is an intelligent storage system capable of accommodating any business-related data, both structured and unstructured, directly from its source.

Why use a data Lake

Progressive businesses employ data lakes to leverage their data and outshine competitors. Pairing data lakes with machine learning solutions and in-depth analytics, they reap a wide array of benefits. The flexibility of AWS services enables businesses to attain their unique objectives. Insights gleaned from AWS Data Lakes can aid in decision-making, operational efficiency, cost reduction, proactive maintenance, productivity enhancement, research, customer retention, and more. Consequently, AWS data lakes can enhance your company’s efficiency in handling database workloads and reduce transactional costs.

Even if you’re unsure about how or when to use specific unstructured data, data lakes offer an inexpensive and secure storage solution. Businesses can extract insights from their data lakes whenever they’re ready, providing a considerable advantage over traditional databases that discard data if not needed immediately and unlike data warehouses storing data in data lake is much more cost effective. This future-proofs your business without hindering access to vital business data. The data can be accumulated over months and analysed when a business case arises.

Another significant benefit of data lakes is their flexibility. Traditional databases often comprise multiple mechanisms and properties, which can pose challenges for platforms that require specific structures to process the data. In contrast, data lakes can store data without processing or altering the structure of the information (decoupling).

Furthermore, data lakes enable secure storage of both structured and unstructured data without scaling limitations. This advantage can save considerable time and effort that would otherwise be spent sorting out different data structures for the computer to process.

The term ‘lake’ aptly reflects these benefits, evoking the organic nature of an actual lake.

Characteristics of data lake

A database typically collects only one type of structured data, whereas a data lake is data agnostic, meaning it is not limited to a specific structure, data type, or file size. This includes diverse formats like photos, videos, texts, compressed or encrypted files, and more. The freeform structure of data lakes enables businesses to make swift changes to queries and models, providing a competitive edge. Data lakes facilitate a broader range of analytics compared to traditional databases.

Another crucial characteristic of data lakes is their ability to help with prediction of future outcomes based on historical data. This can help businesses detect market gaps, potential problems, and gain insight into future performance. This future-proofing can offer a significant advantage over competitors. The simplicity and flexibility of data lake infrastructure allow for structural remodelling based on evolving business needs.

Data Lake vs Data Warehouse

A data warehouse is a centralised database that receives input from transactional systems and line-of-business applications and is set up to analyse this data. Data must be curated within data warehouses to fit into tables, which requires cleaning and formatting to facilitate fast SQL queries.

Although this may sound similar to data lakes, there are significant differences. Data warehouses are typically used for analytics and insights, but require defining the structure of your data beforehand. This means they mostly contain structured data, and the structure must be defined before analysis can occur, known as schema-on-write architecture.

Some may prefer this method as a defined structure can identify or reject data anomalies. However, data lakes offer greater flexibility with a schema-on-read architecture, which allows for unstructured data processing. As a result, the quality of analytics varies between data warehouses and data lakes. Data lake analysis includes big data analytics, full-text search, real-time analytics, and machine learning, all of which can reveal valuable business insights.

Another difference is that data warehouses primarily use SQL for queries, limiting their flexibility. Data lakes, on the other hand, support a range of tools depending on the service needed. While data warehouses can provide business intelligence, batch reporting, and visualisations, data lakes also offer predictive analytics, data discovery, machine learning, profiling.

Data warehouses typically cater only to business analysts, whereas AWS Data Lakes can support all users. From operations managers needing daily reports to data scientists conducting statistical analysis and predictive modelling, everyone in the business can access the required information from this single repository.

It’s worth noting that the choice between data lakes and data warehouses isn’t always mutually exclusive. Many organisations opt for a combination of the two, which is called a lake house architecture, catering to different needs and uses, and ultimately achieving the biggest success in extracting value from data.

Data lake components on AWS

AWS Services serve as the foundation of a data lake, providing a plethora of options to cater to various needs. These services fall into three primary categories: data storage and organisation, data transfer, and analytics and processing. One of the main advantages of a data lake is its ability to store data on an unlimited scale, addressing any concerns regarding space constraints. Furthermore, data lakes can store both structured and unstructured data using a schema-on-read approach. They also organise data in a manner that readies it for transfer – a critical step for processing and analytics. Lastly, the analytics and processing category encompasses a wide array of services, enabling users to manage multiple databases, generate reports, visualise data, or modify their data structure.

Amazon S3 – data storage

Your business may require a service capable of handling vast amounts of structured and unstructured data. This necessitates a solution that supports schema-on-read and unlimited scaling. Amazon Simple Storage Service (S3), along with S3 Glacier, fits the bill, functioning as the ‘data storage and organisation’ service. It is designed to simplify computing for both you and developers.

A standout feature of S3 is its ability to store any type of data file, be it a video, document, audio file, or a mix of various formats. This data-agnostic approach ensures centralisation and consolidation of data in one location.

Moreover, S3 caters to individual scaling needs, accommodating anything from a few gigabytes to hundreds of terabytes. As your data grows over time, this service adapts accordingly. A key concern for many is the durability of data storage services. S3 boasts an exceptional durability rate of 99.999999999% (more nines than anybody cares to count) annually, making the chances of data loss exceedingly low. Consequently, even if you don’t use the data now, you can safely retrieve it in the future.

S3 also employs a clever storage technique that helps reduce costs. It offers various storage tiers, categorising data based on access frequency. For example, frequently accessed data incurs lower access charges but higher storage costs. Conversely, infrequently accessed data has lower storage costs but higher retrieval fees. Thanks to ‘intelligent cataloguing’, you needn’t worry about organising these tiers, as the system monitors access patterns and manages the data for you. Amazon S3 Glacier and S3 Glacier Deep services cater to rarely accessed data by archiving it, ensuring lower storage costs. However, retrieving this data may be more time-consuming and expensive, which shouldn’t pose an issue if the data is seldom used. More commonly accessed data is placed in the S3 standard tier for easier access.

AWS Glue – data catalog

For businesses struggling to organise vast amounts of data or identify data types and sources, AWS Glue is the ideal solution. AWS Glue is a fully managed, serverless service designed to process and organise data on your behalf. It features an AWS Glue catalog containing tables that help categorise and declutter your data. The ‘glue crawler’ then sorts these tables into the appropriate databases by classifying the data by type, schema, and structure within your S3. While the glue crawler can perform this task automatically, you still have the option to classify data according to your preferences, particularly useful if the crawler cannot recognise certain data structures. As your data evolves over time, the glue crawler continues to update the existing data catalog.

Kinesis – data streaming services

When moving data, the data’s origin and type are crucial factors to consider. The Kinesis family of services provides real-time data ingestion in a reliable, secure, and cost-effective manner at scale. Kinesis Data Streams collect data from all sources, process it, and then move it into another service, such as S3. If you prefer a self-management approach, the Software Development Kit (SDK) is suitable for you. SDK processes data in smaller fragments, giving you control over scaling and data destinations. One advantage of SDK is that multiple people can process the data, allowing you to delegate tasks to colleagues.

If you prefer a more automated approach requiring minimal intervention, Kinesis Firehose is the service for you. Like SDK, Firehose processes data in real-time, with the only aspect requiring management being the data’s destination and storage location. This approach is typically better suited for cases that demand minimal processing before data movement and storage.

Another service to consider is API Gateway. If your data doesn’t require real-time processing, API Gateway could be a more suitable option. By inputting your HTTP source, the data is processed and directed to your data lake. One drawback, however, is that the data may not be ingested as deeply as with the Kinesis family of services. API Gateway offers a simpler approach with fewer features included.

Batch processing data services

Batch processing is a common method for analysing and processing data. It can help identify errors, missing information, or areas for improvement within the data. AWS EMR which is Apache Hadoop managed solution, is a service designed for the efficient analysis and processing of vast amounts of data (even petabytes). It does this by organising a cluster of computers that can coordinate and process data quickly, which is often more efficient and cost-effective than relying on a single supercomputer. EMR allows you to scale your data processing in a customised way, with S3 serving as your storage. This enables the efficient processing of large amounts of data in batches and is again cost-effective, as you only pay for the clusters being processed. You can choose between temporary clusters for infrequent data processing or persistent clusters that continue running even after data processing is complete.

Data Analytic Services

Once your data is processed and structured, it’s ready for further analysis. Amazon Athena is a serverless service that helps to processes your data using SQL, without relying on clusters. Athena integrates seamlessly with Amazon S3, allowing for direct data processing without the need for data movement.

Other AWS services that work well with S3 include Amazon Redshift (data warehouse), which is similar to Athena but is more suited to pre-stored data. For real-time data processing, Amazon Kinesis Firehose can be used, with Amazon Data Analytics creating reports to highlight anomalies and trends in real-time data.

Amazon Elasticsearch / OpenSearch

Amazon Elasticsearch, this days known renamed to OpenSearch is a service that helps you locate specific pieces of data, whether it’s text, video, structured, or unstructured. Supported by S3, this service makes finding data stored in your data lake a breeze. Amazon ES includes Kibana, a service that helps visualise your data after it has been analysed, making it more discoverable.

Machine Learning Services

Machine learning is an emerging field that utilises artificial intelligence, especially useful for predicting future business outcomes. Many businesses use machine learning to gain a competitive advantage and save money, provided the predictions are accurate. It is crucial to select a reliable data to minimise risks associated with incorrect predictions.

AWS offers a range of machine learning services, such as Amazon SageMaker, which allows you to build, train, and deploy machine learning models easily. Additionally, AWS provides pre-trained AI services, like Amazon Rekognition for image and video analysis, and Amazon Comprehend for natural language processing. These services enable businesses to harness the power of machine learning to gain valuable insights and make data-driven decisions.

Summary

AWS provides a comprehensive suite of services designed to meet the diverse needs of businesses and individuals managing data lakes. These services cover data storage and organisation, data movement, processing, analytics, and machine learning. Key AWS services include Amazon S3 for storage, AWS Glue for data organisation, Kinesis family for real-time data movement, AWS EMR for batch processing, Amazon Athena and Redshift for analytics, Amazon Elasticsearch for data discovery, and Amazon QuickSight for dashboarding. The flexibility and interoperability of these services make AWS an ideal solution for those looking to store, process, and gain insights from their data. The choice of service depends on your specific use case. By leveraging the right combination of AWS services, users can effectively manage and make the most of their data in today’s data-driven world.

We have not covered all data lake related services, as AWS releases many new features on a regular basis that can help to expand data lake capabilities. One of the recent additions is Amazon MWAA, a managed Apache Airflow tool that provides advanced batch processing capabilities. That being said AWS is not by all means the ultimate end for data platforms’ needs. There is a richness of products from competing vendors that can do a better job in certain scenarios than native AWS services. Large enterprises often select to combine their AWS data cloud platform with products like SnowFlake which is a powerful data warehousing solution, Tableau for dashboarding and visualisation, Matilion for simplified and effective ETL, and Atlan which offers AI-aided data catalogue capability and which is combined with AI, last but not least Dataiku for AI aided machine learning. Usually, those services come with a premium price tag compared to AWS.

Additionally, other general cloud providers like Google Cloud Platform and Microsoft Azure are competing for the crown of best data management platforms. While Databricks focuses solely on data solutions.

Stay tuned to learn more about data platforms, you can follow the @DataPhoenix LinkedIn page.
As always, for any data platform needs, contact us at info@dataphoenix.io – a small end-to-end data solutions provider.

DataPhoenix team

DataPhoenix goes live!

There is a real thirst in so many organisations to get more value from the data. Helping them to do this is something I’m really passionate about. That’s why I’m launching DataPhoenix. DataPhoenix is a data consultancy business which has expertise in architecting and implementing data lakes and data mesh solutions on AWS to help organisations unleash the power of their data.

We’re working with some really exceptional people who also have expertise in data lakes, data pipelines, AWS cloud, and DevOps. Whether you’re trying to make sense of existing big data, want to integrate disparate pools of data or simply want to know what’s possible to help you unleash the potential of your data, DataPhoenix can design and implement the right solution.

SoboL @DataPhoenix

Cookie	Duration	Description
cookielawinfo-checkbox-analytics	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional	11 months	The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy	11 months	The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.

Our Blog

Articles

Welcome to our Blog!