This page is for the technical people in the room. It's the architecture we keep arriving at — after shipping this for founders, ops teams, and internal tools. Not a framework. Not a vendor pitch. Just the opinions we've formed from doing the work.
If you're going to build this yourself: here's the map. If you'd rather we do it: that's at the bottom.
The five layers are sequential — you build them in order. The two disciplines (Trust + Identity & Compliance) thread through every layer from L1. Skip one of the layers, the layer above it starts lying to you. Skip a discipline, the whole platform quietly rots in nine months.
Audit + inventory the systems that hold the data. Most teams skip this and build a one-system answer to a 6-system question.
Three patterns — real-time webhooks, scheduled polling, batch — picked per source. Idempotent writes, reconciliation backstops.
Lake-first storage (raw / processed / metrics tiers) plus the schema work that turns 6 systems into one row about a single customer.
Step Functions, scheduled queries, lane-aware delivery. The chain that turns lake queries into reports the right person actually sees.
Request envelope state machine, eval harness, governance gate. The payoff layer where the lake turns into AI features your team uses every day.
Marts are business concepts — customer, deal, order — not database tables. If the AI has to join six things to answer a question, you haven't finished the job.
Every consumer reads from marts — whether they live in a lake or a warehouse. Change a source? Absorb it in staging. Dashboards and AI never see the chaos underneath.
Giving an LLM raw table access is how you get confident, wrong answers on prod. Typed tools with named metrics are how you get trustworthy ones.
Five build layers go in order. Two cross-cutting disciplines thread through every layer from L1. Each card below jumps to a focused page with the full prose, code, war stories, and a vendor question.
Map questions, not systems.
Audit + inventory of every source that holds business data. The exercise that determines whether the build succeeds or quietly fails.
Get data in. Reliably.
Three patterns picked per source — webhooks, polling, batch. Idempotent writes. Reconciliation backstops on every webhook.
Start with a lake. Six systems → one row.
Storage tiers (raw / staging / marts) plus the SQL-in-git transform layer that joins sources into business concepts. The biggest layer; the source of truth.
What runs when.
Step Functions / DAGs that run the right job at the right time and route the result to the right person via the right channel.
Tools, not SQL. The payoff layer.
Tools (not raw SQL) that the LLM calls. Request envelope state machine. Eval harness. Human-approval gate for irreversible actions.
The lake is correct. Or you have nothing.
Observability + data quality + freshness + schema-drift detection. Threads through L1 → L5. The discipline that prevents silent failures.
The lake is safe. Auth, IAM, PII, audit.
Auth + IAM + KMS + PII + multi-tenant + audit trail. Threads through L1 → L5. The discipline that keeps the platform compliance-audit-ready.
Data engineering stops feeling like a backlog of one-off exports and starts feeling like software — the moment every number in your business has a name, a test, and a git commit.
Not a gantt chart. A sequence. Each week has one thing that has to be true by Friday.
Listing these here because they're usually the first bullets on someone else's architecture doc. Some we think are the wrong order of operations. Others — we disagree on principle and do the harder, more valuable version.
Your pipeline and your orders aren't a semantic-search problem. They're a joins-and-aggregates problem. Don't vectorize what you can query directly.
15-minute refreshes are real-time for 99% of business decisions. Streaming is real work you don't need yet. Ask if anyone has ever said "if only this were 30 seconds fresher."
Fine-tuning fixes vocabulary, not data access. If your AI gives wrong answers, the fix is almost always upstream — in the marts, not the model.
A catalog isn't just docs. We build one that's queryable by agents — so AI can discover what exists, find the right mart, and connect the dots across systems. The catalog is the agent's memory of your business.
When the managed connector doesn't cover a custom object, a rate-limited endpoint, or the specific field your team lives in — we write the connector. It lands the data the way the business needs, not the way the vendor defaults to.
A single well-scoped agent, with tight tools and strong feedback loops, beats a five-agent swarm every time. Once one agent is solid, we add a second — carefully. Multi-agent is earned, not assumed.
Not exhaustive. The questions that come up in every evaluation call.
dbt is what the team you're hiring already knows. SQLMesh has a tighter execution model and a few features we like (virtual data environments, native Python models), and we'll use it on greenfield engagements where the client team is open to it. But on most builds, the consultant who picks up the project from us, the analyst who writes the next mart, and the platform engineer who debugs at 3am all know dbt. Pick the boring tool the team can run.
You can absolutely build this on Snowflake-only — or BigQuery-only, or Databricks-only — and we have. AWS is our default because S3 + Glue + Athena + Lambda gives us a lake-first architecture for cents on the dollar versus warehouse-first economics, and because most clients already have an AWS bill we can plug into.
If you're already deep on Snowflake, we'll build the marts in Snowflake and skip the lake split. If you're on Databricks, we'll lean into Delta Lake. The five build layers and two disciplines don't change; the SKUs do.
For the long tail — and where the actual value lives — managed connectors don't cover the surface. Custom objects, rate-limited or paginated quirks, that one field your team lives in, an internal API, an SFTP drop, a legacy DB. Fivetran covers the easy 60%. Custom Python covers the 40% that determines whether the AI can actually answer.
We're not religious about it: if Fivetran covers a source cleanly, we use it and pay the per-row fee. The moat is in writing the connector for the source that doesn't have a managed option.
One well-scoped agent with a small toolset and tight feedback loops beats a five-agent swarm 95% of the time. Multi-agent introduces coordination overhead, error-propagation, debugging hell, and latency — for value that's usually a delta on top of "did the first agent already nail it." We add a second agent only when there's a clear failure mode the first one structurally can't handle (e.g. a planning agent that proposes a plan, a separate executor that runs it).
Dagster's asset-based mental model fits how data actually flows — every asset has owners, freshness, lineage, metadata. Airflow's task-graph model treats data as a side-effect of tasks, which is backwards. Prefect 2/3 is closer to Dagster but lighter — fine for smaller setups. On bigger engagements where lineage and freshness contracts matter, Dagster wins.
If you're already running Airflow and have a team that knows it, we'll write Airflow DAGs. The orchestrator is the most replaceable layer in the stack.
15-minute refreshes are real-time for 99% of business decisions. Streaming adds Kafka, Flink/Spark Streaming, exactly-once semantics, and a permanent uplift in operational cost. We do streaming when there's a concrete decision a human or system needs to make in <1min. Otherwise: micro-batch every 5–15min, monitor freshness, ship.
Iceberg, in 2026, on AWS. Glue Catalog supports it natively, Athena queries it directly, and the ecosystem (dbt-iceberg, Trino, Snowflake/Databricks both read it) is the most portable. Delta is great if you're committed to Databricks; Hudi has the strongest streaming-upsert story but the smallest ecosystem. The honest answer: any of the three is fine — Iceberg is what we default to because the catalog story is cleanest on the cloud most of our clients use.
You don't throw it away. The new foundation can read the old warehouse as a source — we don't ask anyone to rip out a working BI stack. The migration story we ship most often: existing Looker/Sigma/Tableau keeps running on the old marts; new AI workflows + new dashboards point at the new clean marts; over the next 6–12 months, dashboards drift toward the new foundation as people rebuild. The old stack stays useful while the new one earns its place.
This is the shape of every pipeline we ship. Copy it. Disagree with it. Argue with us about it. The goal is that technical teams stop reinventing this stack from scratch every time — and start arguing about which layer to sharpen first.
Same opinions, shipped on your stack. You'll be running the whole thing by week six — we document everything, and hand you the repo, the runbook, and the team that knows it.
Book a 30-min walkthrough →One video per layer. Live code. Real war stories. The same opinions on this page, with more color. Subscribe if that's useful — no newsletter, no upsell, just the videos.
Subscribe on YouTubeWant the markdown source of this playbook to fork or annotate? Email me and I'll send it. Or stash this page — it’ll keep being updated as we ship more.