The challenge
The insurer's data included personal and protected health information, so the work had to meet HIPAA and security requirements. The client required the whole solution to run inside its own secure virtual servers rather than a cloud platform. The data was inconsistent, several datasets had to be combined, and some joins had no reliable keys.
What we did
Delivered by Logic Intelligence®, the team behind every Data Arcade package.
- Designed the solution to run entirely inside the client's secure virtual infrastructure.
- Built Python ingestion and transformation processes.
- Wrote cleansing and standardization logic and merged new data with existing datasets.
- Built fuzzy-matching logic for joins where no reliable key existed.
- Followed strict data-handling rules, including sanitizing every development machine at the end.
The solution
A Python analytics solution hosted entirely in the client's secure virtual server environment. It ingests CSV files from source folders automatically, cleanses and normalizes them, merges them with existing datasets and resolves difficult joins with configurable fuzzy matching. The processed data is written to secured folders for downstream use. At the end of the project every development system was wiped and restored to factory settings.
Built with
- Secure virtual servers
- Python
- pandas
- CSV pipelines
- Fuzzy matching
Outcomes
What the client has now.
- HIPAA and privacy requirements met, with sensitive data kept inside the approved environment.
- Faster ingestion and cleansing, and more consistent data.
- Complex relationships resolved through flexible matching.
- Full data sanitization completed at project close.