Kimi K3 Runs on Consumer RTX GPUs Over Internet

Distributed Inference Across Consumer Hardware

A groundbreaking experiment demonstrates running Kimi K3, a 2.78 trillion parameter language model, across geographically distributed consumer-grade GPUs connected over the open internet. The test compared two distinct hardware configurations: Ring 1 utilized 16 RTX 5090 graphics cards, while Ring 2 employed 6 RTX Pro 6000 GPUs, all located across different nodes in Europe. Both configurations successfully ran the model using REAP at 80% efficiency with DSpark orchestration. This approach challenges the conventional wisdom that massive language models require centralized data center infrastructure, showing that distributed consumer hardware can handle trillion-parameter workloads when properly coordinated across network boundaries.

Performance Metrics: Tokens Per Second

The performance characteristics varied significantly between the two hardware rings. Ring 1 with 16 RTX 5090s achieved approximately 2.2 tokens per second using DSpark orchestration, while Ring 2's 6 RTX Pro 6000s delivered roughly 7.3 tokens per second—more than three times faster despite having fewer GPUs. When running in greedy decoding mode, Ring 1 produced about 0.6 tokens per second compared to Ring 2's 2.6 tokens per second. These figures demonstrate that the RTX Pro 6000 cards deliver substantially better per-GPU performance for large model inference, though both configurations proved viable for running the massive 93-layer Kimi K3 model across distributed nodes connected via standard internet infrastructure.

Speculative Decoding Performance Gains

One of the most notable findings involves speculative decoding speedups, a technique that predicts multiple tokens ahead to accelerate inference. Ring 1 achieved approximately 2.1x speedup with speculative decoding enabled, while Ring 2 delivered a 2.8x improvement over baseline greedy decoding. This technique proves particularly valuable in distributed settings where network latency between nodes could otherwise bottleneck performance. The draft acceptance rates—measuring how often speculated tokens prove correct—were similar between configurations at approximately 5.3 for Ring 1 and 5.0 for Ring 2. These metrics indicate that speculative decoding remains highly effective even when models are distributed across multiple geographic locations, helping offset the performance penalties typically associated with running inference over the public internet.

First Token Latency and Warm-Up Times

Time-to-first-token represents a critical user experience metric, especially for interactive applications. Ring 1 with the 16 RTX 5090s exhibited warm-start latency of 45-60 seconds before producing the first token, while Ring 2's 6 RTX Pro 6000s achieved dramatically faster performance at approximately 2.3 seconds. This substantial difference likely reflects both the superior per-GPU compute capability of the Pro 6000 hardware and the reduced coordination overhead of managing fewer distributed nodes. For production applications requiring responsive user interactions, the faster first-token latency of the smaller but more powerful GPU configuration would prove essential. Both setups successfully handled all 93 layers of the Kimi K3 model architecture, with signed receipts confirming complete layer execution across the distributed infrastructure.

Implications for Decentralized AI Infrastructure

This experiment validates the technical feasibility of running frontier-scale language models on distributed consumer hardware connected via standard internet connections, rather than requiring purpose-built data center infrastructure. The ability to successfully orchestrate 2.78 trillion parameters across geographically separated nodes in Europe demonstrates that decentralized AI inference infrastructure could become practical for organizations without access to centralized GPU clusters. The performance differences between configurations highlight important trade-offs: more numerous but less powerful GPUs (Ring 1) versus fewer high-end cards (Ring 2). As models continue growing in size, techniques like REAP, DSpark orchestration, and speculative decoding will prove essential for making distributed inference viable, potentially democratizing access to massive language models.