Ted Hisokawa
November 15, 2024 06:56
Stellar’s (XLM) Horizon API integrates CDP components to speed up data re-collection, achieving performance gains of up to 9x faster than traditional methods.
The Stellar (XLM) Development Foundation has announced a significant increase in data processing speed within the Horizon API by leveraging dedicated data processing (CDP) components. According to Stellar, this integration allows data re-collection up to 9 times faster than previous methods.
Improved reacquisition with CDP
Horizon, an HTTP API for accessing data on the Stellar network, has traditionally faced issues with data re-collection speed. However, the introduction of CDP technology has changed this process, enabling faster and more efficient data processing. These improvements are especially noticeable in scenarios involving large-scale data re-collection.
Technology Benchmarking
To evaluate the performance gains, Stellar conducted tests using specific hardware configurations. The setup included a PostgreSQL 12 database running on an EC2 instance (m5.4xlarge) with 16 virtual CPUs and 64GB of RAM, and an RDS instance (db.r5.4xlarge) with 128GB of RAM. This test compared the performance of Captive Core and CDP in re-collecting data from 10,000 ledgers.
While CDP showed excellent parallelization capabilities and achieved optimal results with 16 workers, Captive Core’s performance was limited by disk I/O, showing diminishing returns beyond 4 workers.
Parallel reacquisition function
One of the great features of CDP-enhanced Horizon is its ability to support parallel re-collection. This method significantly reduces the time required for data collection by dividing the recollection range into sub-ranges that are processed simultaneously. The performance improvement was quantified by re-collecting 10,000 ledgers, and CDP outperformed Captive Core in terms of speed and efficiency.
Recollect full records
Stellar also analyzed the time required to re-collect complete transaction history. We estimated that it would take approximately 66 days to re-collect 10,000 ledgers with Captive Core using optimal parallel settings. In contrast, a CDP approach leveraging pre-computed ledger metadata can complete the same task in approximately 7 days.
Implications for future development
Improvements to Horizon’s recollection capabilities are expected to have a significant impact on the performance and scalability of the Stellar network. By reducing the time and resources required to process data, developers can focus on building more powerful applications on the Stellar blockchain.
For more information about the technical implementation of these enhancements, please visit the Stellar website.
Image source: Shutterstock