Hey folks,
In the last post, I wrote about the multi-agent architecture behind Tactix and called out the data pipeline as the unglamorous part we spent the most time on. A few people reached out asking us to go deeper on that. So this post is the deep dive on the part the agents never see: how the data actually gets in.
Quick recap for anyone who didn't read the first one. Tactix is an AI platform for restaurant operators. Three agents (anomaly detection, critical insights, smart actions) read store performance data and produce recommendations an operator can act on. The agents are only as good as the data underneath them. And here's the thing nobody tells you about operational data in this industry: there is no clean API.
Operators email you spreadsheets.
That's the starting point. Not a webhook. Not a warehouse share. A manager exports a report from their POS system, attaches the xlsx to an email, and sends it over. Every store does it a little differently. Our job was to turn that into something three AI agents can reason about, reliably, every day, without a human in the loop.
So, here's how we built it.
What the data actually looks like
Let's start with the raw material, because it shapes everything downstream.
A typical operator sends us a daily or weekly export. Sometimes it's a clean xlsx with one tab. Sometimes it's five tabs, three of which are pivot tables a regional manager built for their own reporting.
Column headers are whatever the POS vendor decided to call things, which means "Net Sales," "net_sales_$," and "Total Net" can all show up across three different stores and mean the same thing. Dates come in every format a human can invent. Occasionally a store sends last week's file twice, or sends a corrected version on Thursday that silently overwrites Monday's numbers.
None of this is anyone's fault. Operators are running restaurants, not data pipelines. The spreadsheet is the integration. If we wanted their data, we had to meet them where they were, which was their inbox.
So the design constraint was clear from day one. The ingestion system had to be tolerant of mess on the way in and strict about quality on the way out. Everything in this post follows from that one tension.
The flywheel
We didn't want to build a data pipeline. We wanted to build a flywheel. The distinction matters. A pipeline is a one-way street: data goes in one end, comes out the other, done.
A data flywheel is a self-reinforcing loop where data, the models that consume it, and the product usage on top all feed and improve each other over time. NVIDIA and others have been explicit about this as a strategy: capture the core process, encode it, and let each cycle compound.
Here's what the loop looks like for us:
The loop, with the data journey at its center: raw files come in, refine from bronze to silver to gold, power the agents, and the feedback sharpens ingestion for every cycle that follows.
An operator emails their data.
The pipeline refines it into clean, modeled signals.
The agents read those signals and produce insights and actions.
The operator acts on them, ignores them, or corrects them, and our eval suite captures every misfire.
That feedback sharpens our parsing rules, our validation, and the way we model the gold layer.
The system gets more trustworthy, so operators send more data and more stores come online.
More data, more corrections, better agents. The wheel turns again, faster.
The compounding is the whole point. Every messy file we learn to parse makes the next operator easier to onboard. Every correction the eval suite catches becomes a validation rule that protects every store after it. The first store was painful. The tenth was routine. That's the flywheel doing its job.
The medallion architecture is the mechanism that makes the loop turn. Let me walk through it.
From inbox to gold
The pipeline moves data through four stages: an inbox, a raw landing zone, a refined middle layer, and a modeled mart the agents actually query.
Email attachments land in GCS as raw bronze, Eventarc triggers Cloud Run refinement steps into silver, and dbt models the gold data mart the agents read through MCP.
The inbox. Operators send their exports to a dedicated address. A lightweight handler watches that inbox, pulls every attachment off each message, and drops the raw file, untouched, into a Google Cloud Storage bucket. We keep the original bytes exactly as they arrived, along with metadata about who sent it and when. This is the bronze layer, and the rule for bronze is simple: never modify the source. If we get parsing wrong later, we can always reprocess from the original.
The trigger. Dropping a file into GCS emits an event. We use Eventarc to route that event straight to a Cloud Run service. Eventarc is Google Cloud's eventing layer; it listens for the object.finalized event on the bucket and invokes our service with the file's location in a standard CloudEvents payload. No polling, no glue code, no cron job checking for new files. A file lands, a refinement job wakes up. This is the event-driven pattern Google recommends for exactly this kind of file-processing workload.
The refinement. Cloud Run is where the mess gets cleaned up. We run the refinement as a series of stateless steps, each one a Cloud Run service doing one job: parse the spreadsheet, map the columns to our canonical schema, validate the values, normalize the dates, deduplicate against what we already have. Stateless and serverless matters here because the load is spiky. Most operators send data around the same times, so we go from zero files to a few hundred in a burst, then back to nothing. Cloud Run scales to meet the burst and scales back to zero when it's done, and we only pay for the seconds we use.
The output of refinement is the silver layer: cleaned, deduplicated, time-aligned data that conforms to our schema. Silver is where every store's data finally looks the same, regardless of what its POS vendor called things on the way in.
The mart. The last hop turns silver into gold. Where silver is normalized and complete, gold is modeled for how the agents actually query it: denormalized, read-optimized, with the baselines and features pre-computed.
This is a data mart in the Kimball sense, a presentation layer shaped around the questions that get asked of it. We build it in BigQuery with dbt handling the transformations.
The agents only ever read from gold, and only through the MCP tool layer I described in the last post. They never touch silver, and they certainly never touch a raw spreadsheet.
That's the path: inbox, to bronze in GCS, to silver through Cloud Run, to a gold mart in BigQuery. Email in one end, agent-ready signals out the other.
The entire structure of our data treatment, from original ingestion to agentic ingestion. Agents continue to refine data throughout the process.
A quick word on the stack
For anyone who wants the specifics:
Gmail API for watching the inbox and pulling attachments
Google Cloud Storage (GCS) as the raw bronze landing zone
Eventarc for event-driven triggering off file arrival
Cloud Run for the stateless refinement steps
BigQuery for the silver and gold layers, with dbt for the transformations
Prefect for orchestrating and scheduling the recurring jobs
MCP (Model Context Protocol) as the read layer between the gold mart and the agents
The throughline, same as last time, is that most of our infrastructure already lives on GCP, so leaning into the native eventing and serverless primitives kept the whole thing simple. Eventarc and Cloud Run gave us an event-driven pipeline without standing up a single server or message broker of our own.
Challenges
Challenge #1: every spreadsheet is a special snowflake
The hardest problem in this whole system is that the input format is not under our control and changes without warning.
This has a name in data engineering: schema drift. A source adds a column, renames a field, reorders things, or changes a data type, and nobody tells you. The classic failure mode is the worst kind: nothing throws an error. The pipeline runs fine, the numbers are just quietly wrong. You find out two weeks later when a number looks off.
We deal with this in two ways.
First, we lean on the agents themselves to do the parsing. Writing a brittle, hand-coded parser per operator would have meant a new parser every time someone's POS vendor pushed an update.
Instead, when a spreadsheet doesn't match a known shape, we use an LLM to map its columns to our canonical schema. "Net Sales," "net_sales_$," and "Total Net" all resolve to the same field without us writing a rule for each one. This is the part that makes the flywheel agentic: there's intelligence in the ingestion, not just in the output. (This is also the piece I'd most want feedback on if you've built something similar, because getting an LLM to be reliable at this took real work on the eval side.)
Second, we validate hard at the boundary and quarantine anything that fails. Before a single row is written to silver, it goes through schema checks and basic invariants: non-empty store identifiers, sane date ranges, values within plausible bounds. Anything that fails doesn't get dropped and it doesn't get silently ingested.
It goes to a quarantine bucket with a reason code and a pointer back to the source file. The main dataset stays clean, and we keep enough detail to fix the upstream issue. Fail fast, fail loud, and never let a bad row corrupt the gold layer.
The combination is what makes it durable. The agents handle the variety; the validation handles the safety.
Challenge #2: making it a flywheel, not a pipeline
A pipeline you run once. A flywheel you run thousands of times, including over data you've already processed. That means the single most important property of the whole system is idempotency: reprocessing the same input produces the same output, every time, with no duplicates and no corruption.
This sounds obvious and is genuinely hard. Remember the operator who sends a corrected file on Thursday that overwrites Monday? Or the same weekly file landing twice? If your pipeline naively appends, you've just double-counted a week of sales, and the agents will faithfully report nonsense. Every step in our refinement is written so that running it again is safe. Reprocessing a file from bronze yields exactly the state you'd get from a clean run. We can replay the entire history from the original bytes in GCS and land in a known-good place.
Idempotency is also what lets the flywheel turn at all. When we improve a parsing rule or add a validation check, we don't just apply it going forward. We reprocess the back catalog from bronze and the whole dataset gets better at once. The raw landing zone is our insurance policy and our time machine. It's the reason a fix we ship today retroactively improves every store's history, not just tomorrow's data.
The lesson we'd pass on: keep your raw layer immutable, and make every transformation replayable. The day you need to reprocess everything (and you will) you'll be glad the originals are sitting there untouched.
What's coming
The near-term work is about widening the mouth of the funnel. Right now operators email us. We're adding direct integrations for the POS systems that offer them, so a chunk of stores can skip the spreadsheet entirely. The medallion structure doesn't change; we're just adding new sources that land in bronze.
We're also pushing more of the parsing intelligence upstream. The more reliably the agents can map and validate an unfamiliar format on arrival, the less any new operator has to think about how they send their data. The goal is that onboarding a new store is as simple as forwarding an email, and everything after that is automatic.
And as in the last post, every new third-party signal we add (school calendars, airport status, local events) flows through this exact pipeline. Bronze, silver, gold, then handed to the agents. Same path, every time.
Takeaways
A few things we'd tell our past selves about building an ingestion flywheel on messy operational data.
Meet your data where it is. Operators emailed spreadsheets, so we built for spreadsheets. The fanciest pipeline in the world is worthless if it assumes a clean API your customers can't provide. Tolerant on the way in, strict on the way out.
Keep the raw layer immutable. Store the original bytes exactly as they arrived and never modify them. It's your insurance policy, your audit trail, and your time machine all at once. Everything else can be rebuilt from it.
Make every step idempotent. Reprocessing is not an edge case, it's the normal operating mode of a flywheel. If running a step twice corrupts your data, you don't have a pipeline, you have a liability.
Validate at the boundary and quarantine the rest. Fail fast and loud. A bad row that gets silently ingested is far more expensive than one that gets rejected with a reason code. Protect the gold layer at all costs.
Put the intelligence in the ingestion, not just the output. Using the agents to parse and map messy inputs is what turned a brittle per-customer chore into something that scales. It's also what makes the flywheel genuinely agentic.
Design the loop, not the line. The value isn't in moving data from A to B. It's in the compounding: every file you learn to parse and every correction you capture makes the next store easier. Build for the wheel turning, not the data arriving once.
If you're working on something similar, we'd love to hear what you've run into. Reach out.
Further reading
The patterns and references that informed how we think about this:
iguazio, What is a Data Flywheel?. a clear definition of the self-reinforcing data-model-product loop and why it compounds.
Deepchecks, What Is A Data Flywheel?. covers the flywheel as a durable advantage, including NVIDIA's framing of capturing and encoding the core loop.
Databricks, What is the Medallion Lakehouse Architecture?. the bronze-silver-gold pattern. We use it in BigQuery, but the design principles are platform-agnostic.
Google Cloud, Build event-driven applications in Cloud Run. the Eventarc-plus-Cloud-Run pattern for triggering data processing off Cloud Storage events.
Google Cloud, Route Cloud Storage events to Cloud Run. the docs for wiring up the file-arrival trigger we use for refinement.
Holistics, Kimball's Dimensional Data Modeling. the data mart and presentation-layer thinking behind our gold layer.
Unstructured, Common Data Ingestion Challenges and How to Handle Them. schema validation, the quarantine pattern, and treating ingestion as a production system.
Unstructured, Understanding Data Transformation in Modern Data Pipelines. idempotency, schema drift, and ELT-vs-ETL tradeoffs.
Until next time, Amit.