Apple shows new way to faster AI language models

Apple, in collaboration with Tel Aviv University, has published a new study on AI-based speech generation. The study focuses on a method that significantly increases the speed of text-to-speech models without compromising intelligibility, naturalness, or speaker-likeness. The approach specifically addresses a well-known problem with autoregressive language models and demonstrates that small adjustments during decoding can have a significant impact.

AI speech models are powerful today, but often limited by their own accuracy. Audio generation, in particular, presents a bottleneck because models are very strict with individual speech tokens. Apple addresses this problem with a new, systematic approach. The study, titled "Principled Coarse-Grained Acceptance for Speculative Decoding in Speech," provides not only a theoretical foundation but also concrete metrics that clearly demonstrate its benefits.

Autoregressive language models as a technical background

Apple's research focuses on autoregressive text-to-speech models. These work similarly to large language models for text: each new token is predicted based on all previous tokens. The difference lies in the token type. Instead of generating words or characters, the tokens here consist of discrete audio chunks. This method is precise and well-established, but it has a structural disadvantage.

Why traditional methods are too slow

In speech LLMs, exact token matching is often too restrictive. Many acoustic tokens are nearly identical in sound or meaning, but are treated as strictly different by the model. This leads to predictions being rejected even though they would be perfectly acceptable to the human ear. The result is low acceptance rates during decoding and a noticeable slowdown in speech generation. This is precisely where Apple's approach comes in.

Principled Coarse-Graining as a new approach

The solution is called Principled Coarse-Graining, or PCG for short. The basic idea is simple: Many different tokens produce nearly identical sounds. Instead of evaluating them individually, they are grouped together based on acoustic similarity. During verification, it's no longer the exact token that matters, but whether it belongs to the correct sound group. This makes the verification process more flexible without sacrificing quality.

PCG system architecture

PCG consists of two separate models:

A small, fast model suggests language tokens.
A larger evaluation model checks whether these tokens fall within the appropriate acoustic group and accepts them accordingly.

This interplay adapts the concept of speculative decoding specifically to language models that generate acoustic tokens. The goal is clear: increased speed without compromising intelligibility.

Measurement results and performance gains

The study's results are clear. Apple demonstrates that PCG accelerates speech generation by approximately 40 percent. This is particularly significant because traditional speculative decoding has yielded little speed advantage for speech models. Simultaneously, word error rates remain low, falling below those of previous methods optimized purely for speed. Speaker similarity is also largely preserved. The naturalness score of 4.09 on a scale of 1 to 5 is especially noteworthy. This rating indicates that the generated speech sounds natural despite the increased speed.

Stress test with token exchange

In an additional ablation test, the researchers replaced 91.4 percent of the speech tokens with alternative tokens from the same acoustic group. Despite this massive intervention, the audio quality remained stable. The word error rate increased only minimally by +0.007. At the same time, speaker similarity decreased by only −0.027. This confirms that many tokens within a group are indeed interchangeable without significantly altering perception.

Significance for Apple and possible applications

The study doesn't specify how Apple will use the results in future products. Nevertheless, the practical benefits are clear. PCG requires no retraining of the target model and no changes to the architecture. It's simply an adjustment made during decoding at inference time. Furthermore, it has low resource requirements. Only about 37 MB are needed to store the acoustic similarity groups. This makes the approach suitable even for devices with limited storage, such as mobile devices.

Faster, more natural, more efficient: Apple's new approach to speech AI

With Principled Coarse-Graining, Apple, in collaboration with Tel Aviv University, demonstrates that AI-powered speech generation can be made more efficient without sacrificing quality. The study proves that less stringent token scoring is not only possible but also beneficial. For Apple, this approach opens up new possibilities for speech features where speed, naturalness, and efficiency are equally important. PCG shows that progress in AI doesn't always come from larger models, but often from more precise assumptions about how humans actually perceive language. (Image: Shutterstock / CineVI)