Crypto Flexs
  • DIRECTORY
  • CRYPTO
    • ETHEREUM
    • BITCOIN
    • ALTCOIN
  • BLOCKCHAIN
  • EXCHANGE
  • TRADING
  • HACKING
  • SLOT
  • CASINO
  • SUBMIT
Crypto Flexs
  • DIRECTORY
  • CRYPTO
    • ETHEREUM
    • BITCOIN
    • ALTCOIN
  • BLOCKCHAIN
  • EXCHANGE
  • TRADING
  • HACKING
  • SLOT
  • CASINO
  • SUBMIT
Crypto Flexs
Home»ADOPTION NEWS»Perplexity AI leverages the NVIDIA inference stack to process 435 million queries per month.
ADOPTION NEWS

Perplexity AI leverages the NVIDIA inference stack to process 435 million queries per month.

By Crypto FlexsDecember 6, 20243 Mins Read
Facebook Twitter Pinterest LinkedIn Tumblr Email
Perplexity AI leverages the NVIDIA inference stack to process 435 million queries per month.
Share
Facebook Twitter LinkedIn Pinterest Email

Terrill Dickey
December 6, 2024 04:17

Perplexity AI leverages NVIDIA’s inference stack, including H100 Tensor Core GPUs and Triton Inference Server, to manage over 435 million search queries per month, optimizing performance and reducing costs.





Perplexity AI, a leading AI-powered search engine, successfully manages over 435 million searches every month thanks to NVIDIA’s advanced inference stack. According to NVIDIA’s official blog, the platform integrates NVIDIA H100 Tensor Core GPUs, Triton Inference Server, and TensorRT-LLM to efficiently deploy large language models (LLMs).

Provides multiple AI models

To meet diverse user needs, Perplexity AI operates more than 20 AI models simultaneously, including variants of the open source Llama 3.1 model. Each user request is matched to the best-fitting model using smaller classification models that determine user intent. These models are distributed across GPU pods, each managed by an NVIDIA Triton inference server, ensuring efficiency under strict service level agreements (SLAs).

Pods are hosted within a Kubernetes cluster with an internal frontend scheduler that directs traffic based on load and usage. This ensures consistent SLA compliance and optimizes performance and resource utilization.

Performance and cost optimization

Perplexity AI uses a comprehensive A/B testing strategy to define SLAs for a variety of use cases. This process aims to maximize GPU utilization while optimizing the cost of inference services while maintaining the target SLA. Smaller models focus on minimizing latency, while larger user-targeted models such as the Llama 8B, 70B, and 405B undergo detailed performance analysis to balance cost and user experience.

Performance is further improved by parallelizing model deployment across multiple GPUs and increasing tensor parallelism to lower servicing costs for latency-sensitive requests. This strategic approach allowed Perplexity to save approximately $1 million per year, exceeding the cost of third-party LLM API services, by hosting models on cloud-based NVIDIA GPUs.

Innovative technology for improved throughput

Perplexity AI is working with NVIDIA to implement ‘separate serving’, a method of separating inference stages to different GPUs to significantly increase throughput while complying with SLAs. This flexibility allows Perplexity to leverage a variety of NVIDIA GPU products to optimize performance and cost-effectiveness.

Further improvements are expected with the upcoming NVIDIA Blackwell platform, which promises significant performance gains through technological innovations including the second-generation Transformer Engine and advanced NVLink features.

Perplexity’s strategic use of the NVIDIA inference stack highlights the potential for AI-based platforms to efficiently manage massive query volumes and deliver high-quality user experiences while remaining cost-effective.

Image source: Shutterstock


Share. Facebook Twitter Pinterest LinkedIn Tumblr Email

Related Posts

Bitcoin Treasury Firm Strive adds an industry veterans and starts a new $ 950 million capital initiative.

September 16, 2025

The best Solana depin project to form the future -Part 2

September 8, 2025

Ether Lee (ETH) tests major support for $ 4,453 after the highest rejection.

August 31, 2025
Add A Comment

Comments are closed.

Recent Posts

MEXC Joins Forces With Lombard Finance (BARD) To Launch $1 Million Prize Pool Extravaganza

September 18, 2025

What is the next after the Fed’s 25bps is cut? Everything you need to know

September 18, 2025

The XRP market value surpasses Shopify, Verizon, and Citigroup. Whales sell 40m coins.

September 18, 2025

Green Hood Contracts Thanksgiving Summary -Ackee Blockchain

September 17, 2025

BetFury Is At SBC Summit Lisbon 2025: Affiliate Growth In Focus

September 17, 2025

FED Mining’s Cloud Mining Platform Is Helping Users Earn $8,800 Per Day, And XRP’s Growth Is Driving Market Enthusiasm.

September 17, 2025

Stablecoin Holdings Drop As Investors Pivot To SOL, XRP, And Altcoins

September 17, 2025

Flipster Partners With WLFI To Advance Global Stablecoin Adoption Through USD1 Integration

September 17, 2025

Zircuit Launches $495K Grants Program To Accelerate Web3 Super Apps

September 16, 2025

Kintsu Launches SHYPE On Hyperliquid

September 16, 2025

New Cryptocurrency Mutuum Finance (MUTM) Raises $15.8M As Phase 6 Reaches 40%

September 16, 2025

Crypto Flexs is a Professional Cryptocurrency News Platform. Here we will provide you only interesting content, which you will like very much. We’re dedicated to providing you the best of Cryptocurrency. We hope you enjoy our Cryptocurrency News as much as we enjoy offering them to you.

Contact Us : Partner(@)Cryptoflexs.com

Top Insights

MEXC Joins Forces With Lombard Finance (BARD) To Launch $1 Million Prize Pool Extravaganza

September 18, 2025

What is the next after the Fed’s 25bps is cut? Everything you need to know

September 18, 2025

The XRP market value surpasses Shopify, Verizon, and Citigroup. Whales sell 40m coins.

September 18, 2025
Most Popular

Cardano, Shiba Inu and One Ethereum-based Altcoin have seen an ‘explosion’ of whale activity, Santiment reveals.

June 8, 2024

Binance Square Launches New Activities to Unlock Up to 100 USDT in Rewards

May 23, 2024

Has this bull cycle reached $10?

December 22, 2024
  • Home
  • About Us
  • Contact Us
  • Disclaimer
  • Privacy Policy
  • Terms and Conditions
© 2025 Crypto Flexs

Type above and press Enter to search. Press Esc to cancel.