Crypto Flexs
  • DIRECTORY
  • CRYPTO
    • ETHEREUM
    • BITCOIN
    • ALTCOIN
  • BLOCKCHAIN
  • EXCHANGE
  • TRADING
  • SUBMIT
Crypto Flexs
  • DIRECTORY
  • CRYPTO
    • ETHEREUM
    • BITCOIN
    • ALTCOIN
  • BLOCKCHAIN
  • EXCHANGE
  • TRADING
  • SUBMIT
Crypto Flexs
Home»ADOPTION NEWS»Optimizing Parquet String Data Compression with RAPIDS
ADOPTION NEWS

Optimizing Parquet String Data Compression with RAPIDS

By Crypto FlexsJuly 17, 20243 Mins Read
Facebook Twitter Pinterest LinkedIn Tumblr Email
Optimizing Parquet String Data Compression with RAPIDS
Share
Facebook Twitter LinkedIn Pinterest Email

Jessie A Ellis
17 Jul 2024 17:53

Learn how to optimize encoding and compression of Parquet string data using RAPIDS, thereby dramatically improving performance.





Parquet writer offers a variety of encoding and compression options that are turned off by default. Enabling these options can provide better lossless compression for your data, but it is important to understand which options to use for optimal performance, according to the NVIDIA Tech Blog.

Understanding Parquet Encoding and Compression

Parquet’s encoding phase reorganizes the data to reduce its size while preserving access to each data point. The compression phase further reduces the total size in bytes, but requires decompression before the data can be accessed again. The Parquet format includes two delta encodings designed to optimize the storage of string data: DELTA_LENGTH_BYTE_ARRAY (DLBA) and DELTA_BYTE_ARRAY (DBA).

RAPIDS libcudf and cudf.pandas

RAPIDS is a collection of open source accelerated data science libraries. In this context, libcudf is a CUDA C++ library for thermal data processing. It supports GPU-accelerated readers, writers, relational algebra functions, and columnar transformations. The Python cudf.pandas library accelerates existing pandas code by up to 150x.

Benchmarking using Kaggle String Data

We compared encoding and compression methods using a dataset of 149 string columns containing 12 billion total characters and a total file size of 4.6 GB. The study found that the encoding size difference between libcudf and arrow-cpp was less than 1%, and that file size increased by 3-8% when using the ZSTD implementation in nvCOMP 3.0.6 compared to libzstd 1.4.8+dfsg-3build1.

String encoding in Parquet

Parquet string data is represented using a byte array physical type. Most writers default to RLE_DICTIONARY encoding for string data, which uses dictionary pages to map string values ​​to integers. When dictionary pages become too large, writers fall back to PLAIN encoding.

Total file size based on encoding and compression

For the 149 string columns in the dataset, the default settings for dictionary encoding and SNAPPY compression produce a total file size of 4.6 GB. ZSTD compression outperforms SNAPPY, and both outperform the uncompressed option. The best single setting for the dataset is the default ZSTD, with delta encoding available for additional reduction under certain conditions.

If you choose delta encoding:

Delta encoding is useful for data with high cardinality or long string lengths, and typically achieves smaller file sizes. For string columns with less than 50 characters, DBA encoding can provide significant file size reductions, especially for sorted or semi-sorted data.

Achievements of readers and writers

The GPU-accelerated cudf.pandas library showed impressive performance compared to pandas, with Parquet reads being 17-25x faster. Using cudf.pandas with an RMM pool further improved throughput to 552 MB/s read and 263 MB/s write.

conclusion

RAPIDS libcudf provides flexible GPU-accelerated tools for reading and writing columnar data in formats such as Parquet, ORC, JSON, and CSV. For those looking to leverage GPU acceleration for Parquet processing, RAPIDS cudf.pandas and libcudf offer significant performance benefits.

Image source: Shutterstock


Share. Facebook Twitter Pinterest LinkedIn Tumblr Email

Related Posts

Crypto Exchange Rollish is expanded to 20 by NY approved.

October 2, 2025

SOL Leverage Longs Jump Ship, is it $ 200 next?

September 24, 2025

Bitcoin Treasury Firm Strive adds an industry veterans and starts a new $ 950 million capital initiative.

September 16, 2025
Add A Comment

Comments are closed.

Recent Posts

US government holds $36 billion in Bitcoin after largest confiscation in history

October 15, 2025

Decoding City Protocol’s IP Capital Market

October 14, 2025

Tria Raises $12M To Be The Leading Self-custodial Neobank And Payments Infrastructure For Humans And AI.

October 14, 2025

How to Use Google Gemini to Analyze Crypto Coins Before Investing

October 14, 2025

Class action lawsuit claims Microsoft choked AI supply to drive up ChatGPT costs

October 14, 2025

CME Group Launches CFTC Regulated Solana and XRP Options

October 13, 2025

Eightco Holdings Inc. ($ORBS) Makes Strategic Investment Into Mythical Games To Accelerate Human Verification And Digital Identity In Gaming

October 13, 2025

Jiuzi Holdings, Inc. (JZXN) Secures 100 Bitcoin Via Private Placement, Signaling New Phase In Crypto Treasury Deployment

October 13, 2025

Collaboration Across Bybit, DigiFT And UBS UMINT Expands Collateral Solution For Institutions

October 13, 2025

BitMine Immersion (BMNR) Announces ETH Holdings Exceeding 3.03 Million Tokens And Total Crypto And Cash Holdings Of $12.9 Billion

October 13, 2025

Phemex Announces Halloween Futures Trading Festival With 200,000 USDT Prize Pool

October 13, 2025

Crypto Flexs is a Professional Cryptocurrency News Platform. Here we will provide you only interesting content, which you will like very much. We’re dedicated to providing you the best of Cryptocurrency. We hope you enjoy our Cryptocurrency News as much as we enjoy offering them to you.

Contact Us : Partner(@)Cryptoflexs.com

Top Insights

US government holds $36 billion in Bitcoin after largest confiscation in history

October 15, 2025

Decoding City Protocol’s IP Capital Market

October 14, 2025

Tria Raises $12M To Be The Leading Self-custodial Neobank And Payments Infrastructure For Humans And AI.

October 14, 2025
Most Popular

CVE-2025-30147- BESU

May 8, 2025

EEA Industry Day at Devcon 2024

December 22, 2024

The People’s Bank of China (PBOC) issues renminbi banknotes through the Central Money Markets Department of the Hong Kong Monetary Authority.

September 21, 2024
  • Home
  • About Us
  • Contact Us
  • Disclaimer
  • Privacy Policy
  • Terms and Conditions
© 2025 Crypto Flexs

Type above and press Enter to search. Press Esc to cancel.