Challenge
The client faced significant delays in business reporting due to a limited BI reporting stack, relying on a single Power BI gateway. The ETL process for daily dashboards took many hours, impacting operations. They sought to accelerate reporting to achieve real-time analytics, transition to a modern data stack for advanced analytics, implement data governance, and make their data more democratised and shareable.
The Solution
An AWS-based Data Lake Platform was built for the client, incorporating real-time change data capture from their on-premises Oracle source systems using AWS DMS. The data was transferred to a Data Lakehouse on S3 with AWS Glue data pipelines. A Data Warehouse for reporting was provisioned on Redshift, and data governance was implemented using AWS Lake Formation. AWS Macie was utilized for sensitive data masking, and a custom solution was developed for data lineage tracking. Finally, Power BI was used to recreate the reporting stack.
Services
- Data Platform Setup
- Data Engineering Pipelines
- Data Modelling
- BI Dashboard Development
- Data Governance Setup
Technologies Used
Power BI
AWS GLUE
AWS Athena
AWS Redshift
Impact by the NUMBERS
20x
Improvement in speed of data processing pipelines
5x
Reduction in support tickets generated for data pipeline failures
5x
Reduction in time to insights from data due to data democratisation
More Success Stories
