Crypto Flexs
  • DIRECTORY
  • CRYPTO
    • ETHEREUM
    • BITCOIN
    • ALTCOIN
  • BLOCKCHAIN
  • EXCHANGE
  • TRADING
  • HACKING
  • SLOT
  • CASINO
  • SUBMIT
Crypto Flexs
  • DIRECTORY
  • CRYPTO
    • ETHEREUM
    • BITCOIN
    • ALTCOIN
  • BLOCKCHAIN
  • EXCHANGE
  • TRADING
  • HACKING
  • SLOT
  • CASINO
  • SUBMIT
Crypto Flexs
Home»ADOPTION NEWS»NVIDIA launches Nemotron-CC, a large-scale dataset for LLM pre-training
ADOPTION NEWS

NVIDIA launches Nemotron-CC, a large-scale dataset for LLM pre-training

By Crypto FlexsJanuary 10, 20253 Mins Read
Facebook Twitter Pinterest LinkedIn Tumblr Email
NVIDIA launches Nemotron-CC, a large-scale dataset for LLM pre-training
Share
Facebook Twitter LinkedIn Pinterest Email

Iris Coleman
January 10, 2025 14:13

NVIDIA launched Nemotron-CC, a 6.3 trillion token English dataset, powering pre-training of large-scale language models with innovative data curation methods.





NVIDIA announced the launch of Nemotron-CC, a groundbreaking 6.3 trillion token English language dataset designed to improve dictionary training of large-scale language models (LLMs). This dataset, derived from Common Crawl, aims to increase the accuracy and efficiency of LLM through innovative data curation techniques, including the use of 1.9 trillion synthetically generated data tokens, according to NVIDIA.

Enhancing LLM pre-education

NVIDIA’s initiative addresses a critical need in LLM training, where the quality of pre-training datasets plays a pivotal role. Recent models, such as Meta’s Llama series, have been based on datasets consisting of up to 15 trillion tokens, but the exact composition of these datasets is largely unknown. Nemotron-CC seeks to fill this gap by providing the wider community with high-quality datasets that can support both short-term and long-term Token Horizon training.

Existing datasets often sacrifice up to 90% of data to improve benchmark accuracy, limiting their usefulness for widespread training. However, Nemotron-CC demonstrates how advanced methods such as classifier ensembles and synthetic data reconstruction can transform the Common Crawl data into a superior dataset that outperforms the Llama 3.1 8B model.

important results

The efficacy of Nemotron-CC is demonstrated by its performance on a variety of benchmarks. When training an 8B parameter model on 1 trillion tokens, the high-quality subset Nemotron-CC-HQ outperforms key datasets such as DCLM, increasing the MMLU score by 5.6 points. Additionally, the full 6.3 trillion token dataset matches MMLU’s DCLM while providing 4x more unique real-world tokens. This allowed the Nemotron-CC trained model to outperform Llama 3.1 8B on several metrics, including a 5-point increase in MMLU and a 3.1-point increase in ARC-Challenge score, enabling effective training over long token periods.

Innovative data curation technology

The development of Nemotron-CC involved several key insights. By combining different model-based classifiers, NVIDIA was able to select a wider range of high-quality tokens. Paraphrasing techniques also reduces noise and errors, creating diverse and valuable data transformations. The decision to disable the existing heuristic filter further improved the quality of the data set without compromising accuracy.

NVIDIA leveraged the NeMo Curator tool to extract and refine data from Common Crawl, applying filters for language, deduplication, and quality classification. This process was complemented by synthetic data generation, contributing approximately 2 trillion tokens to the dataset.

future prospects

Nemotron-CC has established itself as an essential resource for pre-training cutting-edge LLMs across a diverse range of tokens. To further enhance LLM capabilities, NVIDIA plans to expand its product by releasing more specialized datasets, including datasets focused on specific areas such as mathematics.

Image source: Shutterstock


Share. Facebook Twitter Pinterest LinkedIn Tumblr Email

Related Posts

Ether Lee (ETH) tests major support for $ 4,453 after the highest rejection.

August 31, 2025

Bitcoin analysts bet on $ 200K after hints of Fed.

August 23, 2025

‘Self -transactions, dressed in capital layout’: The cryptocurrency financial craze divides the industry.

August 15, 2025
Add A Comment

Comments are closed.

Recent Posts

The 320K holder of the WAVERS & Cardano Price Surges Surges BlockDag signals the next large encryption.

September 6, 2025

RLUSD Stablecoin is extended to Africa to supply power to the border between the border.

September 5, 2025

Bybit Establishes New B2B Unit To Drive Institutional Adoption Of Digital Assets

September 5, 2025

Lowkick Studio Launches $SHARDS Token On Top Tier Exchanges For WorldShards MMORPG

September 5, 2025

The cryptocurrency is falling when the tokens and stocks connected to Trump are under pressure.

September 5, 2025

Cango Inc. Reports Second Quarter 2025 Unaudited Financial Results

September 5, 2025

Coindesk July 2025 Report: Stablecoins and CBDC

September 5, 2025

NOWPayments To Participate In SiGMA Europe Rome 2025

September 4, 2025

Web3 Enabler Announces Blockchain Payments V3.1 At Northeast Dreamin In Boston

September 4, 2025

Is XRP The Dark Horse Of The Cryptocurrency World? Earn 652 XRP Daily Using Invro Mining’s Smart Contract

September 4, 2025

TRX Was Early, ETH Set The Standard, BNB Built The Scale- Now SYC Brings The Next Evolution

September 4, 2025

Crypto Flexs is a Professional Cryptocurrency News Platform. Here we will provide you only interesting content, which you will like very much. We’re dedicated to providing you the best of Cryptocurrency. We hope you enjoy our Cryptocurrency News as much as we enjoy offering them to you.

Contact Us : Partner(@)Cryptoflexs.com

Top Insights

The 320K holder of the WAVERS & Cardano Price Surges Surges BlockDag signals the next large encryption.

September 6, 2025

RLUSD Stablecoin is extended to Africa to supply power to the border between the border.

September 5, 2025

Bybit Establishes New B2B Unit To Drive Institutional Adoption Of Digital Assets

September 5, 2025
Most Popular

SHIB, PEPE & DOGE is leading the pack with Memecoin Rally.

March 26, 2025

Coinbase suspends Instant Withdrawal (SNT) trading.

February 25, 2024

OKX Gets Fully Licensed in Singapore, Hires Former MAS Employee as Local CEO

September 2, 2024
  • Home
  • About Us
  • Contact Us
  • Disclaimer
  • Privacy Policy
  • Terms and Conditions
© 2025 Crypto Flexs

Type above and press Enter to search. Press Esc to cancel.