With existing servers down twice a week under increasing load, there was a need to change the infrastructure or even rewrite the service.
After new modules were developed and connected, high load can no longer ruin infrastructure. Average response time decreased. Live load monitoring, configuring and requests throttling of the whole system are now allowed.
- To learn existing infrastructure: reverse engineering with no docs or assistance.
- Find root cause of server's downs: old DB layer cannot handle too many concurrent requests.
- Existing database infrastructure maintenance.
- Band-aid solution: inject load balancing into main entry point and limit requests density based on subscription type.
- Long-term solution: design a three-layer data storage (archive, operative and fast-access).
- Migrate all databases to new data structures.
- Isolate common business logic into an API and replace existing code with API calls.
- Benchmark performance, develop load monitoring tools.