About us Services Custom solutions Careers Contact us
API to access data collected from daily updated multiple sources
15M

Data entries
collected daily

30ms

Average response
time

6000

Requests served
per second

Challenge
Develop an API which returns unified data collected from 7 existing data sources of different structure, updated daily.
Result
Developed 3 data layers to collect and analyze data arrays. Implemented trend monitoring system. Designed modern user-facing website showing resulting information. Re-calculation performed on a daily basis.
Roadmap
  • Crawl, parse and analyze data from 7 possible data sources on Internet.
  • Import data into unified data structure and analyze data quality.
  • Select 4 good data sources and implement a crawler which collects information on a daily basis, automatically and fail-resistant.
  • Develop fast-access data structures, automatically re-import data from archive datasets, post-process it to make output consistent and remove inconsistent data.
  • Develop a high-load Go application deployed to Heroku.
  • Several round-ups to fine-tune performance.

Let's talk about your project

Give us a call
Dmitrii / CTO
Dmitrii
CTO

Please note that English is our main procedural language.