Commit

dcaf034

Everyone is talking about Nvidia. I keep thinking about one thing: 250 GB of VRAM. Llama 3.1 405B in GGUF at 4-bit would fit (about 200 GB for the weights alone), with room left for context. Meanwhile I am still bargaining with llama.cpp over how many layers fit on my GPU.

commit dcaf034