Your First Local Model in .NET Won't Load. Here's Why.

In the last post I introduced Professor Maier, the flat-earther you argue with — offline, in C#, no cloud. Before he can be smug at you, though, he has to load. And this is where nearly everyone loses their first evening.

The enemy today isn't the professor. It's the loader. Getting a local model running in .NET is genuinely easy once you know the three traps — and genuinely maddening until you do. So let's walk into all three on purpose.

Two packages, and the trap hiding in them

LLamaSharp is split into a core package and a set of backend packages — the native llama.cpp binaries compiled for CPU, CUDA, Vulkan, or Metal. You install the core plus exactly one backend.

<ItemGroup>
  <PackageReference Include="LLamaSharp" Version="0.27.0" />
  <PackageReference Include="LLamaSharp.Backend.Cpu" Version="0.27.0" />
</ItemGroup>

Two rules, and both bite:

Rule 1 — the versions must match, exactly. The backend ships the native binary that the core version expects. LLamaSharp 0.27.0 wants Backend 0.27.0. Mix 0.27.0 core with 0.24.0 backend and you get a native-library-incompatible error that tells you almost nothing. Pin both to the same version and keep them in lockstep when you upgrade.

Rule 2 — install one backend, not several. This is the trap that cost me an evening. I had both Backend.Cpu and Backend.Cuda12 referenced, "to be safe." The result: it silently ran on the CPU, ignored my GPU entirely, and the logs cheerfully reported that the CUDA DLL had loaded — while not using it. If you want GPU, install only the CUDA backend. Having the CPU backend alongside it means CPU quietly wins.

The backends, briefly: Backend.Cpu (also gives you Metal on Mac), Backend.Cuda11 / Backend.Cuda12 (match your installed CUDA major version), and Backend.Vulkan (AMD/Intel GPUs). For the game, CPU is completely fine — a 2B model doesn't need a GPU.

Getting a GGUF file (and the third trap)

LLamaSharp loads models in GGUF format. Easiest source is Hugging Face: search the model name plus gguf and you'll find plenty already converted. For the professor, a small instruct model like Gemma 2 2B is perfect.

Two things to look at before you download:

The quantization suffix. You'll see files like ...Q4_K_M.gguf, ...Q8_0.gguf. Lower number = smaller and faster, but slightly dumber. Q4_K_M is the standard sweet spot: small enough to run comfortably on CPU, good enough that the character holds. Grab that unless you have a reason not to.

The publishing date. This is the third trap, and it's a sneaky one. GGUF is a moving format. A very old file may only load on an old LLamaSharp; a brand-new model architecture may need a newer LLamaSharp than you have. If a model refuses to load and the versions match, check when the GGUF was published against your LLamaSharp version's llama.cpp era. A mismatch there is the usual culprit.

The actual load

Once the packages and the file line up, loading is anticlimactic — which is how it should be:

using LLama;
using LLama.Common;

var parameters = new ModelParams(@"C:\models\gemma-2-2b-it-Q4_K_M.gguf")
{
    ContextSize = 4096,
    GpuLayerCount = 0, // 0 = pure CPU; set high (e.g. 99) to offload all layers to a GPU
};

using var model = LLamaWeights.LoadFromFile(parameters);
using var context = model.CreateContext(parameters);

Console.WriteLine("Model loaded.");

GpuLayerCount is the one knob worth understanding here. 0 runs everything on the CPU. A positive number offloads that many transformer layers to the GPU; set it larger than the model has layers (99 is a common "just offload everything") and the whole thing runs on the GPU — if you installed the CUDA backend and the VRAM fits.

When it doesn't load: the one line that saves the evening

If loading fails or the model runs suspiciously slowly, don't guess. Add this as the very first line of Main, before anything touches the native library:

using LLama.Native;
// ...
NativeLibraryConfig.All.WithLogs(LLamaLogLevel.Info); // on older versions: NativeLibraryConfig.Instance

Now the startup logs tell you exactly which native binary was selected and loaded. That turns all three traps from mysteries into one-line reads:

• "Native library not compatible" → core and backend versions don't match. Fix rule 1.
• The CPU binary loads when you wanted CUDA → either you have both backends installed (rule 2), or GpuLayerCount is still 0. The log names the file, so you'll see it.
• The model itself is rejected → GGUF-vs-version mismatch. Back to the publishing-date check.

Nine times out of ten, that log line is the whole debugging session.

How much RAM does this really need?

Rough but reliable rule: plan for the model file size plus a bit for context. A Q4_K_M 2B model is around 1.5–2 GB on disk and wants roughly that much RAM to run, plus a few hundred MB for the KV cache depending on ContextSize. A 7B at Q4_K_M lands around 4–5 GB. So the professor runs happily on any modern laptop with 8 GB, no GPU required — which is exactly why a small local model is the right call for a game you want other people to actually download and run.

Next up

The model loads. Now we have to make a 2-billion-parameter model stay in character — because a small model left to its own devices will break role, get helpful, and cheerfully explain that the Earth is, of course, an oblate spheroid. Next post: system-prompt engineering for small models, and why the little ones need more constraint, not less.