Enterprise Data Platform
Centralized DataMart on AWS Aurora with ingestion, governance, and quality pipelines powering ML workflows across teams.
+13%
ML Accuracy
Aurora
Core Store
Lead
Role
Architecture
Overview
At Aideo Technologies / Rule14, I led development of core data platform features over 3.5 years — designing scalable FastAPI services, architecting a centralized DataMart on AWS Aurora, and owning customer-facing ingestion, transformation, and egestion pipelines at scale.
Problem
Data science teams were blocked by fragmented data access, inconsistent quality, and no unified governance layer. External system integrations following a strategic alliance added complexity — data had to flow reliably without compromising downstream ML accuracy.
Architecture
External systems feed ingestion pipelines → centralized DataMart on AWS Aurora → governance and validation layer → ML workflows and customer-facing APIs. See the interactive diagram above.
Challenges
- Data accessibility: Scientists needed a single, reliable source instead of ad-hoc extracts from multiple systems
- Quality at scale: Bad data silently degraded ML model performance with no early warning
- Integration reliability: Post-alliance ingestion pipelines had to handle diverse external formats without breaking existing workflows
- API performance: Platform APIs needed to serve both batch analytics and real-time customer operations
Scale / Performance
- Centralized DataMart significantly improved data accessibility and accelerated Data Scientist workflows across teams
- Governance pipelines with anomaly detection and alerting improved downstream ML model accuracy by 13%
- Owned end-to-end ingestion → transformation → egestion pipelines with reliable data flow at scale
Interesting Engineering Decisions
AWS Aurora as the DataMart core
Aurora provided the durability and query performance needed for a centralized analytics layer while integrating cleanly with existing AWS infrastructure (EMR, Lambda, Airflow).
Governance pipelines over manual QA
Automated validation rules and anomaly detection caught data quality issues before they reached ML training — more scalable than manual review and measurable in model accuracy gains.
FastAPI for platform APIs
FastAPI's async performance and type safety fit the platform's need for both high-throughput ingestion endpoints and customer-facing egestion APIs.
Lessons Learned
- Data governance ROI shows up in ML metrics, not just ops dashboards — the 13% accuracy gain made the case
- Centralizing data access accelerates every downstream team, not just data science
- Ingestion pipeline ownership means owning failures too — alerting and anomaly detection are non-negotiable
Outcome
Built the platform's core data infrastructure: centralized DataMart, governance pipelines, and reliable cross-system ingestion — enabling faster ML iteration and measurable model accuracy improvements (Jan 2022 – Aug 2025).