Jog
May 7, 2025 15:38
NVIDIA introduces NEMOTRON-CC, a gin 1-shaped data set for large language models integrated with NEMO curator. This innovative pipeline optimizes data quality and quantity for excellent AI model training.
NVIDIA integrated the Nemotron-CC pipeline into the NEMO curator and provided a breakthrough approach that cuiting high quality data sets for LLMS (Lange Language Models). According to NVIDIA, the Nemotron-CC data set is intended to greatly improve the accuracy of LLM by utilizing the 6.3 trillion goat English collection of the Common Crawl.
Development of data cue
The Nemotron-CC pipeline solves the limitations of traditional data cue methods, which often discards potentially useful data due to the heuristic filtering. This pipeline reposes up to 90%of the content lost by filtering by creating a token of high quality synthesis data of 2 trillion and two trillion won by submitting the classifier ensemble and synthetic data.
Innovative pipeline function
The data cue process of the pipeline starts with HTML-to-TEXT extraction using tools such as JustExt and Fasttext. Then use the NVIDIA Rapids library for efficient processing to remove redundancy to remove duplicate data. This process includes 28 heuristic filters for guaranteeing data quality and PerplayXityFilter module for further improvement.
Quality labeling is achieved through the ensemble of the classifier that evaluates and classifies documents as quality levels to promote the creation of targeted synthetic data. This approach can create a variety of QA pairs, distilled content and organized knowledge lists in the text.
Effects on LLM education
Training LLM with the Nemotron-CC data set makes significant improvements. For example, the LLAMA 3.1 model, which trained the Nemotron-CC’s sub-set of 1 trillion ton, has an increased MMLU score by 5.6 points compared to a model that has been trained in traditional data sets. In addition, the benchmark score has increased by 5 points for models that have been trained for long Horizon tokens, including Nemotron-CC.
Starting Nemotron-CC
Nemotron-CC Pipeline can be used by developers who prevalate the foundation model or perform domain adaptive pre-adjustment in various fields. NVIDIA provides step -by -step tutorials and APIs for custom definitions so that users can optimize pipelines that fit certain requirements. Integration with NEMO curator enables smooth development of pre -adjustment and fine adjustment data sets.
For more information, visit the NVIDIA blog.
Image Source: Shutter Stock