Modernizing a Fortune 500 data platform — and cutting cloud spend by $10K+/month
Notes from an ongoing engagement architecting and modernizing a large enterprise data platform on Snowflake, Databricks, and Azure — including AI-agentic tooling that now reviews production code.
The situation
A Fortune 500 enterprise data platform had grown to well over 4,000 dbt models — spanning models, tests, snapshots, and seeds — running across Snowflake, Databricks, and Azure, with the whole environment provisioned as code via Terraform. At that scale, small inefficiencies compound fast, and manual code review becomes a bottleneck rather than a safeguard.
What changed
- Consolidated 50+ legacy Databricks ingestion notebooks into a single templated framework, improving throughput and data quality while cutting maintenance overhead.
- Identified and implemented infrastructure optimizations — Snowflake credit usage, job-cluster sizing, CI efficiency — that reduced cloud spend by $10K+/month.
- Built an AI-agentic PR review framework using the Model Context Protocol, acting as one of two mandatory reviewers on every change — enabling faster, reasoned approvals without loosening quality gates.
- Architected a multi-agent MCP framework backed by a hierarchical knowledge base (a “skill” library), letting AI agents pick up and deliver backlog items autonomously.
- Leading ongoing ontology design work so the platform’s data is natively consumable by AI agents, not just BI tools.
Why it matters
None of this required ripping anything out. The pattern that keeps showing up: platforms don’t get expensive or fragile all at once — they accumulate small, reasonable-at-the-time decisions until nobody can hold the whole picture in their head. The fix is usually architectural, not a bigger warehouse.
Client details anonymized at the customer’s request. Figures and scope are accurate to the engagement.