GPU’s Apache Spark Workload Acceleration: Prediction approach

In the big data analysis area, optimizing processing speed and reducing the cost of infrastructure remains a pivotal interest. According to a recent report from NVIDIA, Apache Spark, a major platform for scale -out analysis, is exploring GPU acceleration more and more as a means of improving performance.

Promise and challenge of GPU acceleration

Traditionally relying on CPUs, but the transition to Apache Spark’s GPU acceleration promises a significant speed for data processing. But switching a workload from the CPU to the GPU is not simple. Certain tasks associated with large -scale data movement or custom functions may not benefit from GPU acceleration. On the contrary, tasks related to high disease data, such as combination and aggregation, are likely to see performance improvements.

Spark Rafiz qualification tool

NVIDIA introduced the Spark Rapids qualification tool to solve the complexity of workload migration. This tool analyzes CPU -based spark applications and identifies candidates suitable for GPU migration. This tool uses the trained machine learning model in the industry benchmark to predict the potential performance of the GPU. It functions as a command line interface that can be used through the PIP package and supports various environments including AWS EMR and Google DataProc.

Function and output

This tool uses the Spark Event Logs of CPU -based applications to assess the validity of GPU migration. This log provides insight into the application execution to help you identify the optimal workload for GPU acceleration. The output includes a proposed GPU cluster for qualified workload list, recommended spark configuration and cloud service environment.

Prediction custom definition

Pre -trained models accept common scenarios, but this tool supports the creation of a custom qualification model. Users can use their own data to educate models to improve predictive accuracy for unique workloads and environments. This feature is especially advantageous when the existing model does not match certain performance profiles.

Starting

The organization can use the Rapids Accelerator of Apache Spark to facilitate GPU migration without changing the existing code. The Project AETHER also provides a tool for automating the qualifications and optimization of the Spark Work Road for GPU acceleration. See the Spark Rapids user guide for more information.

Image Source: Shutter Stock