Snowflake ELT with CI/CD & Data Quality
A Snowflake warehouse that the business depended on was being changed by hand, which meant schema changes were risky and bad data could reach reports before anyone noticed. I turned it into a version-controlled pipeline with a data-quality gate, so changes ship like software and trust is built in.
What was actually broken
Migrations were applied manually, so a schema change could take the warehouse down and there was no reliable record of what changed or when. There was no automated check on the data itself, so quality issues surfaced in dashboards instead of in the pipeline.
How it fits together
ELT with Snowpark and SnowSQL
Transformations run inside Snowflake using the Snowpark Python API and SnowSQL, keeping the compute next to the data.
Data-quality gate
A Great Expectations suite validates data before it is promoted, so failing data is stopped at the gate rather than served downstream.
CI/CD for migrations
GitHub Actions runs version-controlled migrations, so every schema change is reviewed, tested, and applied the same way every time.
Lineage and audit
Snowflake Streams capture change and audit logging, so the pipeline is traceable and audit-ready by default.
The parts that did the work
- A Great Expectations gate that blocks bad data before it reaches reporting instead of after.
- Version-controlled migrations through GitHub Actions, so schema changes are reviewable and repeatable.
- Stream-based lineage and audit logging that kept the warehouse audit-ready from day one.
The hard parts, and how I got past them
Schema changes were applied by hand and risked downtime.
Moving migrations into version-controlled CI/CD made every change reviewed and reproducible, which took schema-change downtime to zero.
Data-quality problems were only caught once they hit a dashboard.
A Great Expectations gate validates data in the pipeline, so issues are caught before promotion rather than by a confused stakeholder.
What changed for the business
What I took awayTreating a warehouse like software, with version control, CI/CD, and automated quality gates, is what separates a pipeline you can sleep through from one you babysit. The data-quality gate paid for itself the first time it caught something.
More work
Let's build something that ships.
Hiring for a senior Data/AI role, or need a data platform that actually holds up in production? Let's talk.
or email me directly at muhammaduzairkhan329@gmail.com