Everyone is talking about Nvidia. I keep thinking about one thing: 250 GB of VRAM.
Llama 3.1 405B in GGUF at 4-bit would fit (about 200 GB for the weights alone), with room left for context.
Meanwhile I am still bargaining with llama.cpp over how many layers fit on my GPU.
commit dcaf034