Develop an API which returns unified data collected from 7 existing data sources of different structure, updated daily.
Developed 3 data layers to collect and analyze data arrays. Implemented trend monitoring system. Designed modern user-facing website showing resulting information. Re-calculation performed on a daily basis.
- Crawl, parse and analyze data from 7 possible data sources on Internet.
- Import data into unified data structure and analyze data quality.
- Select 4 good data sources and implement a crawler which collects information on a daily basis, automatically and fail-resistant.
- Develop fast-access data structures, automatically re-import data from archive datasets, post-process it to make output consistent and remove inconsistent data.
- Develop a high-load Go application deployed to Heroku.
- Several round-ups to fine-tune performance.