FMCG Multi-Country Data Platform
A multi-country FMCG strategy team needed market-share dashboards they could trust to be current. The catch: their source data arrived as a steady drip of spreadsheets that field teams dropped into SharePoint, in a different shape for every market. I built the platform that turns that mess into governed, near real-time reporting without anyone touching a file by hand.
What was actually broken
Ingestion was manual, slow, and easy to get wrong. Someone had to notice a new upload, pull it, reshape it, and load it before a dashboard could refresh, which meant reporting always lagged reality by hours or days. Re-uploaded and corrected files quietly created duplicate rows, and every new country meant another bespoke load script to maintain.
How it fits together
Autonomous ingestion
Logic Apps watch the SharePoint libraries, push every new file into Blob Storage, and Event Grid fires the pipeline the moment a file lands. No human in the loop.
Bronze, raw and audited
Files are captured into Delta exactly as received, with source, market, and load-time columns so every row is traceable back to the spreadsheet it came from.
Silver, clean and idempotent
Config-driven PySpark jobs on Databricks standardize each market to a shared schema, dedupe on business keys, and validate before promotion.
Gold, ready to serve
Conformed market-share models feed Power BI directly, so the dashboards read from a single trusted layer.
The parts that did the work
- Idempotent pipelines that can re-run on the same file safely, so a correction overwrites cleanly instead of doubling the numbers.
- Per-country schema mapping handled in config, not code, so onboarding a new market is a config change rather than a new deployment.
- Event-driven triggers that collapsed the gap between a file landing and a refreshed dashboard from hours to minutes.
The hard parts, and how I got past them
Field teams uploaded inconsistent spreadsheet layouts that changed from market to market.
A config-driven mapping layer normalizes each source to one shared schema and validates it before anything reaches Silver, so a format quirk in one country never breaks the rest.
Re-uploaded and corrected files risked duplicating rows in the warehouse.
Merge logic keyed on stable business identifiers makes every load idempotent, so reprocessing a file updates in place rather than appending.
What changed for the business
What I took awayThe hardest part of a multi-country platform is rarely the compute, it is the variance in the inputs. Pushing that variance into config and making every load idempotent is what let the platform scale to new markets without scaling the maintenance.
More work
Let's build something that ships.
Hiring for a senior Data/AI role, or need a data platform that actually holds up in production? Let's talk.
or email me directly at muhammaduzairkhan329@gmail.com