Trading data from faulty WebSockets and slow-responding REST APIs: System Design and Implementation
date
May 28, 2023
slug
data-from-faulty-websockets-and-slow-responding-rest-apis-implementation
status
Published
tags
system-design
web-scraping
event-driven-architecture
time-series-data
summary
Following problem analysis and proposing optimal solutions, we proceed with the system design and implementation of crucial system components for the Historical Trading Data Service
type
Post
Please be aware that sharing a database is considered an anti-pattern and not recommended for production applications, it is currently being utilized in the existing design solely for demonstration purposes.
System components
VWAP Service
The VWAP (Volume-Weighted Average Price) service serves as the implementation of the necessary REST APIs. It acts as a central hub for handling client requests, caching responses, and ensuring that users consistently receive accurate data from the system database
Data Processor
The Data Processor service is responsible for multiple tasks within the system. Firstly, it receives data from a WebSocket connection and maintains a stable connection with the data provider. It periodically generates requests to query trading data from continuous time windows encompassing all available data in the system.
Upon receiving the query responses from the data fetching worker, the Data Processor service resolves any conflicts that may arise. It ensures consistency and accuracy in the trading data by handling conflicts appropriately.
Additionally, the Data Processor service plays a crucial role in notifying other system components of all trading data-related events. It acts as a central source of event notifications, keeping the system informed about relevant updates and changes in the trading data.
API Proxy
The API Proxy service is designed to handle scheduled tasks for periodically querying trading data from the Data Provider using REST API. It employs a robust retry strategy, implementing two solutions for retries, to obtain the data from the REST Endpoint. The service continues to retry the request until it succeeds, ensuring persistent efforts to retrieve the required data.
Once the API Proxy service successfully receives responses from the Data Provider, it promptly notifies the Data Processor. This notification serves as a signal to inform the Data Processor that the requested data has been acquired and is ready for further processing
Previous Article:
