Executive summary
10 weeks to a green end-to-end run. 8x less modernization effort. 100% validated on production-representative data.
A major healthcare revenue-cycle management (RCM) provider ran its revenue-cycle analytics on an aging Cloudera/Hadoop estate: dozens of interdependent Impala scripts, Oozie workflows, and roughly thirty hand-written shell loaders accreted over years. The pipeline was slow, opaque, and expensive. Queries scanned full history, jobs ran in rigid serial chains, and no reliable map existed of what production ran versus what was dead code.
A conventional rewrite risked silently dropping years of embedded business logic. Instead, the provider used the Gallop Platform to convert the entire estate to a modern, columnar BigQuery architecture, with machine-verifiable validation at every layer and a zero-risk parallel-run cutover.
The challenge: an opaque estate that could not be rewritten by hand
The analytics estate still carried the provider's revenue-cycle logic, but nothing about it was safe to touch. The team faced four core problems:
- Opaque pipelines: Dozens of scripts and workflows, with no trustworthy map of what production actually dispatched. Dead code and duplicates were indistinguishable by inspection.
- Fragile ingestion: Roughly 30 separate shell loaders, each with bespoke file handling: quote-stripping, unzip, metadata-cache refreshes, and per-file HDFS operations.
- Cost ceiling: Tables partitioned on free-form strings scanned all history on every query, and a fixed on-prem cluster could not scale with demand.
- Correctness risk: Years of embedded quirks, coded date formats, defective guards, dedup rules, and tenant-routing exceptions, had to be reproduced exactly. Verifying that by hand at this scale was not possible.
The approach: agent-driven, verification-first
Rather than reverse-engineering the estate by interview and inspection, the provider ran the modernization on the Gallop Platform across four phases:
- Discover: Gallop deterministically parsed the source orchestration (Oozie → driver → module → transformation) to establish what production actually runs, classified every job into reusable conversion patterns, and flagged dead code and orphaned scripts with evidence. The result was a verified baseline, not a guess.
- Plan: Gallop's Planning Module auto-generated the target BigQuery design: schema mappings, partitioning and clustering tuned to real read patterns, per-job write patterns, and a layered medallion architecture (raw → conformed → gold → mart → serving). Internal architects reviewed and locked it, keeping full control.
- Build: Native AI agents authored the full target system: Dataform models, BigQuery DDLs, one config-driven loader replacing the shell scripts, and a Cloud Composer DAG turning serial chains into a true dependency graph. Agents preserved legacy quirks and carried known source defects behind flippable flags, so the migrated system matched the original exactly during comparison.
- Prove: Every conversion pattern was backed by committed machine-verifiable specs and build-time guards that fail the build on any divergence. Layered audits resolved edge cases before go-live. The pipeline hit a 100% green end-to-end run on production-representative data, with a documented parallel-run plan for full-volume cutover.
Business impact: cheaper, faster, and provably equivalent
The migration replaced a multi-quarter internal rebuild and changed how the estate is operated and extended:
- A fraction of a traditional rewrite's effort: The conversion delivered in 10 weeks with roughly 8x less modernization effort than the internal rebuild it replaced.
- ~30 shell loaders collapsed to one config-driven loader: New feeds are now a config row, not new code.
- Serial job chains replaced by a dependency graph: The critical path is the true data dependency, not a scheduler artifact.
- Full-history scans replaced by time-partitioned, tenant-clustered columnar tables: Queries read only relevant data, with cost and latency gains measured against the legacy baseline.
- Verification-driven correctness: Every pattern is covered by committed tests, build guards, and independent audits.
- Restored ownership and scalability: On serverless BigQuery, the internal team scales without procurement and extends to new client organizations as configuration, not re-engineering.
Outcome
In 10 weeks, a Cloudera estate nobody could fully account for became a documented, partitioned, agent-ready BigQuery pipeline with a machine-checkable definition of correct. The provider did not have to choose between moving fast and knowing the logic survived.
Verification was not the last phase of the migration. It was the mechanism that made the migration safe to run.