iPhone 17 Pro Runs 400B Parameter LLM Locally
An iPhone 17 Pro has successfully run a 400-billion parameter large language model (LLM) locally, a feat previously requiring a minimum of 200GB of memory even in compressed form, according to a demonstration by developer @anemll. The open-source project, Flash-MoE, enabled the execution of the Qwen model directly on the smartphone.
Although LLMs of this scale typically demand substantial hardware resources, the iPhone 17 Pro, equipped with 12GB of LPDDR5X RAM, achieved this through a novel approach. Instead of loading the entire model into RAM, Flash-MoE streams model weights directly from the device’s NVMe storage to the GPU, bypassing system memory limitations. This technique, combined with a Mixture of Experts (MoE) architecture, allows the phone to activate only a fraction of the 400 billion parameters during each token generation.
The demonstration, however, revealed a significant performance trade-off. The model generates text at a rate of 0.6 tokens per second – roughly one word every 1.5 to 2 seconds. This represents considerably slower than cloud-based LLM inference, which can generate text approximately 100 times faster, as noted in reports on the demonstration. The process also places considerable strain on the iPhone’s battery and thermal management systems.
Despite the slow generation speed, the successful execution of a 400B LLM on a mobile device represents a notable advancement in on-device artificial intelligence. The Flash-MoE project highlights the potential for running large models locally, offering benefits such as full privacy, independence from internet connectivity, and on-device processing. According to Wccftech, this accomplishment was considered “impossible” without clever optimizations.
The MoE architecture is central to this achievement. By utilizing 512 experts per layer and activating only 4-10 during each token generation, the system minimizes the memory footprint. This allows the iPhone 17 Pro to effectively manage a model that, when compressed, still requires 200GB of memory. Abit.ee reported that the iPhone 17 Pro has 16 times less RAM than typically required for such a model.
The demonstration has sparked discussion regarding the definition of “running” an LLM and the future of mobile AI. While the current performance is not practical for real-time conversation, the milestone suggests that further optimization could enable more usable on-device LLM capabilities, even on smartphones. Dev.to noted the debate surrounding the demonstration, with some questioning whether it represents a genuine preview of the future or merely a technical stunt.