Edge0 open-sourced a 145B MoE for one NVIDIA GPU. Their figures: Q3_K hits 121.2 tok/s on an RTX 5090 D. Runs on edge0-strata, not llama-server, on Linux or Windows. M2 Ultra: measurement, not macOS. A commercial service needs written permission.
https://github.com/Edge0-AI/Edge0
GitHubGitHub - Edge0-AI/Edge0: Device-native foundation models: 145B on a laptop, 35B on a phone. Latent MoE, SSD-streamed weights, a router that predicts loads a step ahead. Mac, iPhone, Android, Windows.Device-native foundation models: 145B on a laptop, 35B on a phone. Latent MoE, SSD-streamed weights, a router that predicts loads a step ahead. Mac, iPhone, Android, Windows. - Edge0-AI/Edge0
commit 81cc3ea