The insurance company had a compliance problem disguised as a data problem. Regulators wanted lineage reports — where does each number in the actuarial filings come from? — and the company couldn’t answer. Six months later, it passed the audit with zero critical findings.
TL;DR
- Client: An insurer with data spread across Hadoop, Oracle, Hive, and Spark, and a retiring engineer as the only person who understood how it connected.
- Problem: No documented lineage, catalog, or ownership — and a regulatory audit six months out.
- Result: Full source-to-filing lineage for all 50 critical assets, data-quality score up from 72% to 94%, impact analysis cut from 5 days to 2 hours, and a clean audit.
The Problem
Regulators were asking where each number in the actuarial filings came from, which systems fed the reserve calculations, and who could access policyholder PII. The company couldn’t answer — not because the data didn’t exist, but because nobody had documented how it flowed. Data lived in Hadoop, Oracle, Hive, and Spark, and the only person who understood the connections was a senior engineer about to retire.
What Was at Stake
In insurance, every pricing decision, claims estimate, and reserve calculation traces back to data. Unclear provenance isn’t a tidiness issue — it’s regulatory fines for inaccurate filings, mispriced risk from inconsistent definitions, and outright audit failure. On top of that, analysts were losing ~40% of their time finding and validating data instead of using it. And the one person who understood the plumbing was weeks from walking out the door with it in his head.
The Approach
We used DMBOK 2 as the framework but deliberately did not boil the ocean — we focused on the four knowledge areas the audit depended on: Data Governance, Metadata Management, Data Quality, and Data Lineage. Scope was capped at the ~50 assets feeding regulatory reports, then expanded.
Tooling: Collibra as the catalog and governance hub (the regulator-recognized standard); Informatica for integration and data-quality profiling (already running some ETL); Alteryx for remediation workflows.
- Data catalog — cataloged 800+ tables across Hadoop/Oracle/Hive, 50+ ETL jobs with source-to-target mappings, and 120+ business terms in a glossary (“incurred loss,” “earned premium,” “case reserve”), each linked to owner, classification, and dependencies.
- Data lineage (the regulatory requirement) at three levels: technical (auto-extracted column-level transformations from Informatica), business (higher-level flow diagrams reviewed by stakeholders), and impact analysis (which downstream reports break if a source changes — instant instead of a week of investigation).
- Data-quality monitoring — rules across five DMBOK dimensions (completeness, accuracy, consistency, timeliness, uniqueness), run daily in Informatica, published to a Collibra scorecard, with Alteryx handling reconciliation and dedup on failure.
- Data stewardship — a governance council with named stewards from actuarial, claims, underwriting, IT, and compliance. This was the organizational half: without clear ownership, no tooling matters.
The Results
- Audit readiness: full source-to-filing lineage for all 50 critical data assets
- Data-quality score: 72% → 94% across five dimensions in six months
- Incident response: impact assessment for a source-system change cut from 5 days to 2 hours
- Analyst productivity: ~30% less time spent finding and validating data
- Outcome: passed the regulatory audit with zero critical findings
What I’d Do Differently
The framing that unlocked everything came late: “governance” sounds like bureaucracy, so buy-in stalled until we reframed it as “you’ll spend less time arguing about numbers in meetings.” I’d lead with that business framing from day one instead of the DMBOK vocabulary. I’d also start the legacy-system archaeology (interviewing long-tenured staff, reading Spark source line by line) in week one — reverse-engineering undocumented Hadoop tables was the critical-path item and we underestimated it. Scope discipline, though, I’d keep exactly: starting at 50 assets instead of “everything” is why the initiative shipped at all.
The Stack
| Layer | Tool | Why |
|---|---|---|
| Data Catalog | Collibra | Industry standard, regulator-recognized |
| Business Glossary | Collibra | Single source of truth for business terms |
| Data Lineage | Collibra + Informatica | Automated technical + manual business lineage |
| Data Quality | Informatica Data Quality | Rules engine, profiling, scorecards |
| Remediation | Alteryx | Visual cleansing and reconciliation |
| Source Systems | Hadoop, Oracle, Hive, Spark | Existing enterprise infrastructure |
| Framework | DMBOK 2 (DAMA International) | Structured governance maturity |
Simba Hu helps companies make better decisions with data and AI — from strategy to implementation. Based in Tokyo, serving clients globally. Book a strategy call or visit simbahu.com.