The challenge
The retail organization needed reliable, timely reporting on large volumes of externally sourced click-count data. The third-party CSV files needed heavy cleansing, merging and validation, the whole solution had to be rebuildable after a failure, and end-to-end processing took 8 to 12 hours.
What we did
Delivered by Logic Intelligence®, the team behind every Data Arcade package.
- Designed and built the ETL, with data-quality validation.
- Optimized processing performance end to end.
- Planned operational resilience: backup, recovery and archiving.
- Documented fields, transformations, calculations, schemas and operating procedures.
The solution
A unified analytics platform on a managed SQL Server database service with SSIS. Each run backs up the database, ingests the large CSV files from shared storage, loads them, runs stored procedures that cleanse and aggregate the data, archives the source files and processes the analytical cubes for reporting. The platform can be rebuilt fully from its documented code, schemas, backups and processing logic.
Built with
- SQL Server
- SSIS
- Managed database service
- Stored procedures
- Analytical cubes
- Backup and archiving
Outcomes
What the client has now.
- Processing time cut from up to 12 hours to about 3 hours.
- CSV loads averaging 5–10 GB processed reliably.
- The analytics database grew to about 132 GB over a year and a half and stayed stable and fast.
- Better data quality and repeatable reporting, with tested backup and recovery.
Buy this as a package
The same kind of work, as a fixed-scope package with a published price range.