Crypto Flexs
  • DIRECTORY
  • CRYPTO
    • ETHEREUM
    • BITCOIN
    • ALTCOIN
  • BLOCKCHAIN
  • EXCHANGE
  • TRADING
  • SUBMIT
Crypto Flexs
  • DIRECTORY
  • CRYPTO
    • ETHEREUM
    • BITCOIN
    • ALTCOIN
  • BLOCKCHAIN
  • EXCHANGE
  • TRADING
  • SUBMIT
Crypto Flexs
Home»ADOPTION NEWS»Optimizing Parquet String Data Compression with RAPIDS
ADOPTION NEWS

Optimizing Parquet String Data Compression with RAPIDS

By Crypto FlexsJuly 17, 20243 Mins Read
Facebook Twitter Pinterest LinkedIn Tumblr Email
Optimizing Parquet String Data Compression with RAPIDS
Share
Facebook Twitter LinkedIn Pinterest Email

Jessie A Ellis
17 Jul 2024 17:53

Learn how to optimize encoding and compression of Parquet string data using RAPIDS, thereby dramatically improving performance.





Parquet writer offers a variety of encoding and compression options that are turned off by default. Enabling these options can provide better lossless compression for your data, but it is important to understand which options to use for optimal performance, according to the NVIDIA Tech Blog.

Understanding Parquet Encoding and Compression

Parquet’s encoding phase reorganizes the data to reduce its size while preserving access to each data point. The compression phase further reduces the total size in bytes, but requires decompression before the data can be accessed again. The Parquet format includes two delta encodings designed to optimize the storage of string data: DELTA_LENGTH_BYTE_ARRAY (DLBA) and DELTA_BYTE_ARRAY (DBA).

RAPIDS libcudf and cudf.pandas

RAPIDS is a collection of open source accelerated data science libraries. In this context, libcudf is a CUDA C++ library for thermal data processing. It supports GPU-accelerated readers, writers, relational algebra functions, and columnar transformations. The Python cudf.pandas library accelerates existing pandas code by up to 150x.

Benchmarking using Kaggle String Data

We compared encoding and compression methods using a dataset of 149 string columns containing 12 billion total characters and a total file size of 4.6 GB. The study found that the encoding size difference between libcudf and arrow-cpp was less than 1%, and that file size increased by 3-8% when using the ZSTD implementation in nvCOMP 3.0.6 compared to libzstd 1.4.8+dfsg-3build1.

String encoding in Parquet

Parquet string data is represented using a byte array physical type. Most writers default to RLE_DICTIONARY encoding for string data, which uses dictionary pages to map string values ​​to integers. When dictionary pages become too large, writers fall back to PLAIN encoding.

Total file size based on encoding and compression

For the 149 string columns in the dataset, the default settings for dictionary encoding and SNAPPY compression produce a total file size of 4.6 GB. ZSTD compression outperforms SNAPPY, and both outperform the uncompressed option. The best single setting for the dataset is the default ZSTD, with delta encoding available for additional reduction under certain conditions.

If you choose delta encoding:

Delta encoding is useful for data with high cardinality or long string lengths, and typically achieves smaller file sizes. For string columns with less than 50 characters, DBA encoding can provide significant file size reductions, especially for sorted or semi-sorted data.

Achievements of readers and writers

The GPU-accelerated cudf.pandas library showed impressive performance compared to pandas, with Parquet reads being 17-25x faster. Using cudf.pandas with an RMM pool further improved throughput to 552 MB/s read and 263 MB/s write.

conclusion

RAPIDS libcudf provides flexible GPU-accelerated tools for reading and writing columnar data in formats such as Parquet, ORC, JSON, and CSV. For those looking to leverage GPU acceleration for Parquet processing, RAPIDS cudf.pandas and libcudf offer significant performance benefits.

Image source: Shutterstock


Share. Facebook Twitter Pinterest LinkedIn Tumblr Email

Related Posts

BTC RSI hits April low as Coinbase premium turns red.

October 18, 2025

Crypto Exchange Rollish is expanded to 20 by NY approved.

October 2, 2025

SOL Leverage Longs Jump Ship, is it $ 200 next?

September 24, 2025
Add A Comment

Comments are closed.

Recent Posts

Bitcoin Futures Traders Return to Pumping BTC: Will It Last?

October 21, 2025

Bank of England to introduce stablecoin regulations by 2026

October 21, 2025

LayerZero Outlook: ZRO Price Imminent Ahead of $43 Million Token Unlocking.

October 21, 2025

Bombastic Casino Unveils New Design And Enhanced Features

October 21, 2025

Leading A New Era Of AI Model Training And Digital Computing Power Contracts

October 21, 2025

Earn 15% APR With Flexible Redemption And Up To 2,926 USDT

October 21, 2025

Crypto Market Recovers As Liquidity Returns — Pepeto Announces $700K Giveaway And 221% Staking Rewards

October 21, 2025

6 Best AI Quant Apps: Smarter Automated Trading Solutions for the Modern Investor

October 21, 2025

Start Passive Crypto Income At Zero Cost

October 21, 2025

Limitless Prediction Market Closes $10M Seed Round Ahead Of LMTS Token Launch

October 20, 2025

Whale.io Introduces Crock Dentist Game And Exclusive RWA NFT Collection

October 20, 2025

Crypto Flexs is a Professional Cryptocurrency News Platform. Here we will provide you only interesting content, which you will like very much. We’re dedicated to providing you the best of Cryptocurrency. We hope you enjoy our Cryptocurrency News as much as we enjoy offering them to you.

Contact Us : Partner(@)Cryptoflexs.com

Top Insights

Bitcoin Futures Traders Return to Pumping BTC: Will It Last?

October 21, 2025

Bank of England to introduce stablecoin regulations by 2026

October 21, 2025

LayerZero Outlook: ZRO Price Imminent Ahead of $43 Million Token Unlocking.

October 21, 2025
Most Popular

Ethereum-Based Meme Project Pepeto ($PEPETO) Surges Past $6.5M In Presale

August 29, 2025

XinFin XDC Network and Founding, Launch of XDC 0xCamp Accelerator Program to Foster Global Web3 Startups

September 2, 2024

Xsolla and Crypto.com Partner to Integrate Payment Solutions – Blockchain News, Opinion, TV and Careers

December 6, 2023
  • Home
  • About Us
  • Contact Us
  • Disclaimer
  • Privacy Policy
  • Terms and Conditions
© 2025 Crypto Flexs

Type above and press Enter to search. Press Esc to cancel.