High Quality Data, Reduced TCO: Cloud Native Tick History – PCAP

From hardware and infrastructure investments to data capture and storage issues, managing tick history data can be a challenging and costly undertaking. This blog highlights these complexities faced by financial services firms, and explains how LSEG Tick History – PCAP stored in the cloud with AWS S3 Direct provides an efficient and cost-effective solution that reduces TCO while still offering the level of data detail and quality that firms need.

1. Capturing, storing and managing accurate and comprehensive tick history data can be both challenging and costly.
2. Financial services firms need to consider numerous factors when it comes to managing their tick history data.
3. LSEG Tick History – PCAP stored in the cloud with AWS S3 Direct provides an efficient and cost-effective solution that reduces TCO while still offering the level of data detail and quality that firms need.

To gain a competitive advantage in electronic trading, financial institutions develop complex algorithms to analyse market microstructure, identify trading patterns, and formulate effective trading strategies. To achieve these goals, access to high quality historical market data is essential. Tick data must exhibit exceptional levels of accuracy, accessibility, completeness, and usability, all captured using the lowest possible latency network architecture. With market participants continuously fueling the “arms race” to electronic trading – and the increasing quantification and granularity of timestamps from milliseconds to nanoseconds – access to the highest quality tick data has become imperative.

However, capturing and managing market data can be a costly challenge. Maintaining an in-house technology solution to source and capture market data takes time and is resource-intensive. Financial firms often find themselves grappling with the question of whether to build, maintain and expand their own solution, or explore outsourced providers. While the prospects of maintaining technology in-house and controlling costs may seem appealing, proprietary solutions require significant investments in operations and infrastructure. Moreover, the technology infrastructure required to effectively work with market data is complex, demanding a high level of resiliency and redundancy to support mission-critical downstream processes. These factors collectively contribute to a firm’s overall total cost of ownership or TCO.

In this blog, we take a closer look at the challenges involved with capturing, storing and managing accurate and comprehensive tick history data and why, with so many factors to consider, it can be such a costly undertaking to get it right. We will also explain why our Tick History – PCAP solves many of these challenges, providing an efficient and superior TCO solution that still offers the depth of market detail your business needs.

Connectivity: Firstly, establishing low network latency connectivity is paramount. Capturing the highest quality market data requires capturing it as it flows through the exchange with minimal network hops and utilising high-performance network infrastructure (switches, routers, etc.) to minimise the total network latency. This often requires co-locating with a network provider or within an exchange’s dedicated co-location facility. Adding new venues requires capacity planning and bandwidth upgrades, further increasing the infrastructure footprint.

Hardware and capture: Hardware and capture solutions play a vital role in creating a highly resilient market data infrastructure. Redundant capture servers – fitted with high-performance network interfaces, switches, timestamping servers and devices are necessary to ensure a reliable and resilient market data capture set-up. Timestamping is one of the critical processes here and requires using a high-quality time-source (GPS or similar), time synchronisation (such as PTP, White Rabbit and PPS) – to improve the accuracy of timestamps on packets. As firms look to establish a global market data footprint, the costs to scale up this infrastructure grow exponentially. Lastly, firms also need to build a capture solution, to capture the market data as it is received by the network interfaces, split, and process the packets containing market data.

Monitoring and alerting: Operating a global market data infrastructure requires a robust monitoring and alerting stack to identify and mitigate issues in a timely manner. Numerous metrics including server health, network health, time accuracy and software performance are monitored to maintain a 24×7 live capture infrastructure. Every instance of degradation of data quality needs to be identified, then the root cause addressed, to achieve a truly high-performance solution. Also, once captured, market data requires thorough analysis to identify missing or incorrect data, and arbitration between multiple sources, to create a clean and lossless data stream.

Exchange updates and maintenance: Exchanges often upgrade or modify market data feeds and connectivity requirements for various reasons, such as protocol updates, new or updated multicast groups and bandwidth upgrades. Firms are required to monitor these changes, and trigger software or infrastructure upgrades to ensure business-as-usual continuity. This includes coordinating with exchanges and participating in the User Acceptance Testing conducted by exchanges, network connectivity providers and capacity planning. Continuous maintenance increases costs, requiring a global team of analysts, software, data and infrastructure engineers, and market data SMEs to build and maintain the data infrastructure.

Storage: Market data rates keep increasing and, because of the recent increase in high-volatility days in financial markets, firms can now expect to see a YoY 20-40 % increase in the amount of global market data published on the networks. Once captured, market data needs to be sourced, and then stored on-prem or in the cloud for further analysis. As data volumes explode, data automation is rising and more customers are leveraging the cloud to increase operational efficiencies, lower technology costs, and use cloud computing power for data analysis. The sheer volume of data transfer from the capture sites to the final storage location could go up to as much as 700TB per month, increasing storage costs exponentially. Though cloud storage seems an attractive obvious choice, storing historical market data in the cloud is by no means free. Beyond initial costs, there are additional ongoing costs such as market data licensing and operational costs to upgrade systems to handle exchange-mandated updates incurred to capture data.

We analysed the TCO for capturing and storing eight of the top global futures exchanges (HKFE, KRX, ICE, CME, EUREX, Euronext, OSAKA, and TAIFEX) and identified that with our Tick History – PCAP and AWS S3 Direct offering, firms can reduce their monthly TCO by close to 80%. Savings based on one-time and ongoing storage costs.

Tick History – PCAP is a cloud-based, 20+ petabyte repository of ultra-high quality global market data, captured directly in the exchange data centers. It enables you to access complete sets of lossless, top-of-book and full-depth data (Levels 1, 2 and 3) in your choice of format.

Our focus is to ensure that the data our clients consume is of the highest quality. Our capture process starts with our Real-Time – Ultra Direct suite of feed handlers and applications deployed to capture, monitor, arbitrate and normalise the market data powering our flagship Tick History – PCAP solution. We have spent years refining our technology and methodology to handle this difficult and resource-intensive work, so that our clients don’t have to. The result is superior data quality for improved decision-making and optimised performance. Also, with our AWS S3 Direct offering, customers can leverage the full potential of the data in cloud, by accessing PCAP data stored in our AWS buckets as their own, or even bringing their queries to our cloud. Our shared storage fee cost is a fraction of the total cost of the feed itself, based on volume of feed, and is not tied to escalating volume. It remains constant over the year, even as volumes and storage costs rise, so you can keep your costs down and focus on advancing your business.

Note – The TCO costs were computed based on the following costs a) 2x servers for capturing market data b) Network Connectivity and maintenance costs c) Time Stamping & Time keeping device costs e) AWS Storage costs (Standard AWS S3 storage pricing)