Hosted on MSN

Apple embraces Nvidia GPUs to accelerate LLM inference via its open source ReDrafter tech

ReDrafter delivers 2.7x more tokens per second compared to traditional auto-regression ReDrafter could reduce latency for users while using fewer GPUs Apple hasn't said when ReDrafter will be deployed ...
InfoQ

GPULlama3.java Brings GPU-Accelerated LLM Inference to Pure Java

A monthly overview of things you need to know as an architect or aspiring architect. Unlock the full InfoQ experience by logging in! Stay updated with your favorite authors and topics, engage with ...
SDxCentral

AI inference crisis: Google engineers on why network latency and memory trump compute

Google researchers have warned that large language model (LLM) inference is hitting a wall amid fundamental problems with memory and networking problems, not compute. In a paper authored by ...
Nasdaq

Apple and Nvidia Partner to Enable Faster LLM Token Generation

Discover top-rated stocks from highly ranked analysts with Analyst Top Stocks! Easily identify outperforming stocks and invest smarter with Top Smart Score Stocks Apple introduced ReDrafter earlier ...
Business Wire

Positron AI Secures $51.6 Million in Oversubscribed Series A to Accelerate Inference-Optimized Hardware

RENO, Nev.--(BUSINESS WIRE)--Positron AI, the premier company for American-made semiconductors and inference hardware, today announced the close of a $51.6 million oversubscribed Series A funding ...
Seeking Alpha

Red Hat Launches the llm-d Community, Powering Distributed Gen AI Inference at Scale

Forged in collaboration with founding contributors CoreWeave, Google Cloud, IBM Research and NVIDIA and joined by industry leaders AMD, Cisco, Hugging Face, Intel, Lambda and Mistral AI and university ...
Digi Times

ByteDance open-sources COMET to boost MoE efficiency, accelerating LLM training by 1.7x

ByteDance's Doubao AI team has open-sourced COMET, a Mixture of Experts (MoE) optimization framework that improves large language model (LLM) training efficiency while reducing costs. Already ...
ITWire

Cloudera unveils AI inference service with embedded Nvidia NIM microservices to accelerate genAI development and deployment

COMPANY NEWS: Cloudera, the only true hybrid platform for data, analytics, and AI, today launched Cloudera AI Inference powered by Nvidia NIM microservices, part of the Nvidia AI Enterprise platform.
VentureBeat

Researchers baked 3x inference speedups directly into LLM weights — without speculative decoding

As agentic AI workflows multiply the cost and latency of long reasoning chains, a team from the University of Maryland, Lawrence Livermore National Labs, Columbia University and TogetherAI has found a ...
NextBigFuture

Defeating Nondeterminism in LLM Inference by Thinking Machines

A research article by Horace He and the Thinking Machines Lab (X-OpenAI CTO Mira Murati founded) addresses a long-standing issue in large language models (LLMs). Even with greedy decoding bu setting ...
Semiconductor Engineering

Dynamic KV Cache Scheduling in Heterogeneous Memory Systems for LLM Inference (Rensselaer Polytechnic Institute, IBM)

A new technical paper titled “Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System” was published by researchers at Rensselaer Polytechnic Institute and IBM. “Large ...