Webinar · 8 September 2026

Local and sovereign AI

Local Models forAI-Assisted Coding

Coding agents, local inference and automated verification

Rainer HahnekampSoverius AI

Rainer Hahnekamp

AI Engineer
Google Developer Expert

AI Engineering

We Build Production-Ready
AI Systems

Local-first. Open. Engineering-driven.

Local Models

Private, secure, cost-efficient

AI Agents

Context, tools, reliable execution

Agentic UIs

Interfaces for people working with agents

Open SourceTraining & WorkshopsConsulting

2

Agenda

01Why Local Models?

02Setup and Configuration

03Hardware Provisioning

04Expected Quality

05Engineering Around the Model

3

01

Why Local Models?

4

Why local?

Reasons to Run Models Locally

€

Cost Control

No provider charge per token.

Privacy

Keep code and context inside.

Independence

Reduce provider dependency.

5

02

Setup and Configuration

6

Required Components and Our Setup

AGENT

Claude Code

calls

INFERENCE ENGINE

llama.cpp

runs

MODEL

Gemma 4

7

Our local coding setup has three distinct components. Gemma 4 provides the model weights and predicts tokens. llama.cpp loads and runs those weights. Claude Code manages repository context, planning, tool calls, file changes and the work loop. Keeping those responsibilities separate helps us explain which part affects model quality, inference performance and system permissions. Claude Code can connect to the local llama.cpp endpoint through its configured model interface. Transition: run the stack once so the rest becomes concrete. Logo sources: Claude symbol: https://commons.wikimedia.org/wiki/File:Claude_AI_symbol.svg ; llama.cpp official brand assets: https://github.com/ggml-org/llama.brand ; Gemma official repository: https://github.com/google-deepmind/gemma/blob/main/docs/images/icon.svg .

Local Inference Demo

8

03

Hardware Provisioning

9

CPU and GPU Inference

CPU1× ComputeIntel Core i9-14900K

Hard Disk or SSD≈0.1–7 GB/sBarraCuda HDD
870 EVO / 990 PRO SSDSwap · No Compute

System RAM≈90–100 GB/sDual-Channel DDR5-5600

PossibleOften too slow

GPU≈30–100× ComputeRTX 4070 Super – RTX 5090

VRAM≈500–1,800 GB/sGDDR6X – GDDR7

PracticalUsually 5–10× faster

You can serve a model larger than 1 TB from disk, but do not expect a response anytime soon.

10

Start with the simple answer. A model can fit on a hard disk or SSD, but storage performs no model computation. The storage bandwidth range combines an ordinary hard disk at roughly 0.1 to 0.2 GB/s, a SATA SSD around 0.5 GB/s and a fast consumer NVMe SSD around 7 GB/s. llama.cpp can memory-map the model file and ask the operating system to load the required pages into system memory. If the model is larger than the available RAM, those pages must be loaded and discarded repeatedly. That swapping can make inference extremely slow. CPU plus system RAM can execute the model without a GPU. The 90 to 100 GB/s figure represents a normal dual-channel DDR5 desktop system, and the CPU serves as the 1-times compute baseline. GPU plus VRAM is the preferred path because both components are faster. Current consumer GPUs suitable for local inference span roughly 500 to 1,800 GB/s of memory bandwidth, while their parallel matrix throughput can be tens of times higher than a desktop CPU. Do not multiply those two ratios. Each phase is limited by its slower resource. As a rule of thumb, full GPU inference is often five to ten times faster than CPU-only inference for token generation. Prompt processing can show a larger difference. The precise speed depends on the model, quantization, hardware and runtime. Unified-memory systems combine RAM and VRAM into one pool, but the same principle remains: memory bandwidth and GPU compute determine the useful speed. Reference hardware behind the displayed bandwidths: - CPU and system RAM: Intel Core i9-14900K with dual-channel DDR5-5600, 89.6 GB/s, rounded to about 90 GB/s. - GPU lower endpoint: NVIDIA GeForce RTX 4070 SUPER, 504 GB/s. - GPU upper endpoint: NVIDIA GeForce RTX 5090, 1,792 GB/s. - Hard disk: Seagate BarraCuda ST2000DM008, up to 220 MB/s. - SATA SSD: Samsung 870 EVO, up to 560 MB/s. - NVMe SSD: Samsung 990 PRO, up to 7,450 MB/s. Transition: now look at three concrete hardware classes for local inference. Sources: https://www.seagate.com/files/www-content/product-content/barracuda-fam/barracuda-new/en-us/docs/100817550L.pdf ; https://semiconductor.samsung.com/news-events/news/samsung-introduces-latest-in-its-worlds-best-selling-consumer-sata-ssd-series-the-870-evo/ ; https://download.semiconductor.samsung.com/resources/brochure/990_PRO_Series_Brouchure_Web_Version_1.0.pdf ; https://www.intel.com/content/www/us/en/products/sku/236773/intel-core-i9-processor-14900k-36m-cache-up-to-6-00-ghz/specifications.html ; https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4070-family/ ; https://www.nvidia.com/en-sg/geforce/graphics-cards/50-series/rtx-5090/ ; https://github.com/ggml-org/llama.cpp

Local Inference Hardware

Consumer

RTX PRO 4000

24 GB VRAM≈ €2,500 per GPU

Mac Studio

36–512 GB Unifiedfrom €2,999

Ambitious Consumer

ASUS Ascent GX10

128 GB Unified Memory ≈ €4,850 per system

Enterprise

NVIDIA RTX PRO 6000 Blackwell graphics card

RTX PRO 6000 Blackwell

96 GB VRAM per GPU ≈ €17,700 each

11

These examples show practical local-inference options rather than a complete buying guide. The consumer column deliberately contains two different approaches. NVIDIA's RTX PRO 4000 Blackwell is a discrete GPU with 24 GB of GDDR7 VRAM and 672 GB/s of memory bandwidth. Twenty-four gigabytes is a useful starting point for quantized coding models, although the usable model size also depends on the context and KV cache. The RTX PRO name belongs to NVIDIA's professional range; consumer here describes the size of the local setup, not NVIDIA's market category. Austrian retail listings put the card at roughly €2,500. This buys the GPU, not a complete computer. A Mac Studio is the integrated alternative. Apple's current configurations use M5 Max or M5 Ultra and range from 36 GB to 512 GB of unified memory. The 512 GB figure is a configurable maximum, not the base configuration. M5 Max configurations span 36 to 128 GB. M5 Ultra configurations span 96 to 512 GB. Apple starts the M5 Max system at €2,999 and the M5 Ultra system at €6,599 in Austria. Apple says the 512 GB option will become available at the end of October 2026, and the final price of that option is not yet shown. Unified memory is shared by CPU and GPU, so it is not directly equivalent to dedicated VRAM, but the larger configurations can keep very large quantized models in memory. The wide range is important: Mac Studio is one product family, not one fixed capacity. The middle class is the ASUS Ascent GX10. Its NVIDIA GB10 Grace Blackwell Superchip uses 128 GB of coherent unified LPDDR5x memory. This gives a single developer far more model capacity than a 24 GB graphics card, although its 273 GB/s memory bandwidth is lower than the bandwidth of high-end discrete GPUs. Current Austrian retail listings start near €4,850 for the 1 TB system. ASUS originally announced a €3,499 recommended price, so availability and street pricing matter. The enterprise class uses RTX PRO 6000 Blackwell GPUs. Each GPU provides 96 GB of GDDR7 ECC memory. Austrian retail listings for the NVIDIA Workstation Edition currently start around €17,700 per card. The three cards in the picture would therefore represent more than €50,000 in GPUs alone. A workstation can contain several cards. Multiple GPUs add capacity and throughput, but one model does not automatically see them as one simple memory pool. The inference runtime must split or replicate the model, and interconnect bandwidth matters. All prices on the slide are rounded snapshots checked on 7 September 2026 and can change. Transition: move from physical machines to the model sizes and deployment classes they can support. Sources: https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/rtx-pro-4000/ ; https://geizhals.at/nvidia-rtx-pro-4000-blackwell-900-5g147-2570-000-a3488409.html ; https://www.apple.com/at/newsroom/2026/08/apple-introduces-new-mac-studio-with-m5-max-and-m5-ultra/ ; https://www.apple.com/at/mac-studio/specs/ ; https://www.asus.com/networking-iot-servers/desktop-ai-supercomputer/ultra-small-ai-supercomputers/asus-ascent-gx10/ ; https://www.idealo.at/preisvergleich/OffersOfProduct/208466423_-ascent-gx10-asus.html ; https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/rtx-pro-6000-family/ ; https://geizhals.at/nvidia-rtx-pro-6000-blackwell-workstation-edition-900-5g144-2500-000-a3476784.html Image sources: RTX PRO 4000 and RTX PRO 6000 images supplied by the presenter; https://www.apple.com/v/mac-studio/specs/b/images/specs/hero__d0yd117u6qy6_large_2x.jpg ; https://dlcdnwebimgs.asus.com/gain/c66958f8-57de-4563-b0db-bd317bfaf1f3/w800

Deployment envelopes

Model Size by Deployment Class

Total parameters log scale

10T

1T

100B

10B

1B

Gemma 4 E2B5.1B total

Gemini Nano≈2–4B

Browser / embeddedModels run inside the browser

Gemma 4 12B11.95B total

Qwen3-Coder 30B-A3B30.5B total

24 GB classConsumer GPU or laptop

Nemotron 3 Super120B total

Qwen3.8 Flash Next180B total

128 GB class4-bit workstation or DGX Spark

DeepSeek V4 Flash284B total

GLM-5.3753B total

Kimi K32.8T total

Open-weight datacenterMulti-accelerator systems

Gemini
3.8 Flash≈3.0T estimate

GPT-6
Astra≈3.8T estimate

Grok
4.6≈3.9T estimate

Claude Fable
5.1≈6T estimate

Closed frontierProvider-hosted models

Deployment class

Closed-model values are estimates from public benchmark data [1][2], not disclosed parameter counts. Typical error ≈2×.

12

Read the chart from left to right. The horizontal axis shows five practical deployment classes. The vertical axis is logarithmic: every major step represents ten times as many total parameters. This keeps models from 3 billion to several trillion readable on one slide. It also means the visual distance represents a ratio, not a simple difference. Gemma 4 E2B and Gemini Nano illustrate two different ways a small model can reach a device. Gemma 4 E2B is an open-weight checkpoint with 2.3B effective parameters and 5.1B total when its per-layer embeddings are counted. The chart uses 5.1B because its vertical axis is total parameters. The E in E2B means effective, not total. Google's original Gemini 1 report disclosed two Nano models with 1.8B and 3.25B parameters, both quantized to 4 bits for deployment. Current Chrome documentation describes hardware-selected variants around 2B and 4B, so the slide shows those rounded current values. Chrome may download the larger or smaller variant after measuring the device. The exact installed variant can change with browser updates and can be inspected at chrome://on-device-internals. Downloaded does not mean open weights. Chrome downloads, updates and removes the model as a browser-managed component. A web application uses it through the Prompt API and does not receive the weight files as a model it can redistribute, inspect or run in another inference engine. Google separately calls Gemma its open-model family. Gemini Nano is local and downloadable, but it is not an open-weight model in the same sense as Gemma, Llama or Qwen. Gemma 4 12B has 11.95B parameters and Google positions it for laptops, desktops and small servers. Qwen3-Coder 30B-A3B can fit the 24 GB class when strongly quantized. Its native context is 262,144 tokens, but a 24 GB machine will normally require a shorter practical context because the KV cache also needs memory. The 128 GB class deliberately shows only two examples. NVIDIA Nemotron 3 Super has 120B total and 12B active parameters with up to 1M context. Its NVFP4 checkpoint targets Blackwell hardware, which makes it relevant to a DGX Spark class machine. Qwen3.8-Flash-Next is the tightest fit in this group. Hugging Face reports 180B total parameters: a 125B language model with 6B active parameters, plus 51B n-gram embeddings and 4B multi-token-prediction parameters. A simple 4-bit estimate puts the weights near 90 GB before runtime overhead. It supports 262,144 tokens natively, but a 128 GB machine must shorten the practical context or use cache quantization so the weights, KV cache and runtime fit together. This is why the column says 4-bit candidates rather than guaranteed configurations. DeepSeek V4 Flash has 284B total parameters and activates 13B for each token. GLM-5.3 has 753B total parameters. Kimi K3 has 2.8 trillion total parameters and activates 104B per token. The providers do not disclose parameter counts for Claude Fable 5.1, GPT-6 Astra, Grok 4.6 or Gemini 3.8 Flash. The plotted figures are our calculations, not vendor claims and not published parameter counts. We use the public Frontier Parameter Model [1]. It predicts parameter-equivalent scale from known-size checkpoints, release date and the Artificial Analysis Intelligence Index, with Kimi K3's disclosed 2.78T size as an anchor. The exact published Artificial Analysis branch is P = 2.780T × 10^(0.045205959 × (AA − 57.112339) − 0.350927951 × age in years). Using the available scores and release dates gives 6.12T for Fable 5.1, 3.82T for Astra, 3.93T for Grok 4.6 from AA 61 and its August 12 release, and 3.04T for Gemini 3.8 Flash from AA 59 and its September 2 release. The slide rounds these values. This method does not recover literal proprietary weights or reveal architecture and active-parameter counts. The estimator's backtests report median row-level error close to 2× and a conservative factor of about 3× for a new developer [3]. Grok 4.6 builds on Grok 4.5, and Gemini 3.8 Flash is designed for speed and cost efficiency; post-training and serving choices can improve capability without a proportional increase in weights. Treat the four numbers as a common plotting convention for comparison, not hardware-sizing figures. Sources: Gemma 4 E2B size and terminology: https://ai.google.dev/gemma/docs/core/model_card_4 ; https://huggingface.co/google/gemma-4-E2B ; Gemini Nano sizes and deployment: https://deepmind.google/gemini/gemini_1_report.pdf ; https://developer.chrome.com/docs/ai/understand-built-in-model-management ; https://developer.chrome.com/docs/ai/get-started ; open-model distinction: https://ai.google.dev/ ; 128 GB examples: https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 ; https://huggingface.co/Qwen/Qwen3.8-Flash-Next ; [1] https://github.com/anpaure/frontier-model-sizes#live-factor-models ; [2] Artificial Analysis scores: https://artificialanalysis.ai/models/ , https://artificialanalysis.ai/articles/grok-4-6-benchmarks-and-analysis/ , https://artificialanalysis.ai/articles/gemini-3-8-flash ; [3] https://github.com/anpaure/frontier-model-sizes/blob/main/MODEL_READINESS.md#precision ; official model pages: https://x.ai/news/grok-4-6 ; https://blog.google/innovation-and-ai/models-and-research/gemini-models/3-8-flash-and-3-8-flash-cyber/ ; https://openai.com/index/gpt-6-astra/ ; https://www.anthropic.com/claude/fable ; open model cards: https://huggingface.co/google/gemma-4-12B ; https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct ; https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash ; https://huggingface.co/zai-org/GLM-5.3 ; https://huggingface.co/moonshotai/Kimi-K3 . Transition: these classes come from overlapping ecosystems.

Model Availability by Organization

OrganizationDownloadable
WeightsBuilt Into
DevicesHosted
API

Google

GemmaGemini NanoGemini

OpenAI

gpt-ossGPT

Anthropic

Claude

Apple

OpenELMApple Intelligence

Meta

Llama

Microsoft

PhiPhi SilicaPhi

NVIDIA

NemotronNIM

Mistral AI

MistralMistral

Alibaba · Qwen

QwenQwen

DeepSeek

DeepSeekDeepSeek

Z.ai · GLM

GLMGLM

Moonshot · Kimi

KimiKimi

MiniMax

M2.5MiniMax

xAI

Grok-1Grok

Availability differs by model family.

13

This slide separates three different forms of availability. Downloadable weights means that the organization publishes weight files that we can run with our own inference engine. Built into devices means that a vendor-managed model runs locally through an operating-system or browser API, but applications do not receive portable weight files. Hosted API means that inference runs on the vendor's or a partner's infrastructure and we send requests to a service. These categories overlap. Google publishes Gemma weights, supplies Gemini Nano on devices and offers Gemini through an API. OpenAI publishes gpt-oss weights but provides its current GPT frontier models through an API. Apple publishes OpenELM research weights and integrates separate Apple Intelligence models into its devices. Microsoft publishes Phi weights, integrates Phi Silica into Windows and serves Phi through Azure. For xAI, Grok-1 weights are downloadable, while current Grok models are accessed through an API. The Chinese organizations also offer downloadable families and hosted services: Alibaba develops Qwen, DeepSeek develops its own family, Z.ai develops GLM, Moonshot AI develops Kimi, and MiniMax develops the M-series. Model availability changes over time, so classify model families rather than treating an organization as permanently open or closed. Open weight also does not necessarily mean that training data and the complete training system are open source. Sources: https://ai.google.dev/ ; https://help.openai.com/en/articles/11870455-openai-open-weight-models ; https://developer.apple.com/machine-learning/apple-intelligence/ ; https://machinelearning.apple.com/research/openelm ; https://learn.microsoft.com/en-us/windows/ai/cards/phi-silica-platform-card ; https://qwen.ai/ ; https://www.deepseek.com/ ; https://z.ai/ ; https://www.kimi.com/ ; https://www.minimax.io/ . Transition: zoom in on one downloadable family to see how one name can cover very different sizes and architectures.

Gemma 4 Family

DenseSame path for every tokenMixture of ExpertsRouter selects experts

Total Parameters 30B 20B 10B 0

E2B5.1B totalDense
2.3B effective

E4B8B totalDense
4.5B effective

12B11.95B totalDense

26B-A4B25.2B totalMoE
3.8B active

31B30.7B totalDense

ECore size, excluding large embedding lookup tables

AParameters selected by the router for each token

14

Gemma 4 is a family, not one model. The vertical scale is linear and represents total parameter count: a 5.1B bar is roughly one sixth the height of a 30.7B bar. Blue and green describe architecture. Blue models are dense, meaning that every token follows the same network path. Green marks the mixture-of-experts model, where a router selects only some experts for each token. Effective and active sound similar, but they describe different mechanisms. In E2B and E4B, E means effective. These dense models contain large per-layer embedding tables. The tables must be stored, but each token only performs quick lookups in them. Google therefore describes the computational core as 2.3B or 4.5B effective parameters, while the complete stored models contain 5.1B or 8B parameters. In 26B-A4B, A means active. The complete mixture-of-experts model contains 25.2B parameters. For each token, its router selects eight experts plus one shared expert from 128 experts in each MoE layer, so about 3.8B parameters participate in the computation. All 25.2B parameters still have to fit in memory for fast inference. The 12B and 31B variants are dense models with 11.95B and 30.7B parameters. All five are multimodal. The E2B and E4B models support 128K context; the 12B, 26B-A4B and 31B models support 256K. Parameter count is only one dimension: quantization, context size and architecture determine the actual memory requirement and speed. Source: Google Gemma 4 model card, https://ai.google.dev/gemma/docs/core/model_card_4 . Transition: compare that range with the much smaller Qwen 3.6 family.

Qwen 3.6 Family

DenseSame path for every tokenMixture of ExpertsRouter selects experts

Total Parameters 50B 40B 30B 20B 10B 0

27B27B totalDense

35B-A3B35B totalMoE
3B active

AParameters selected for each token

262KNative context for both models

15

Qwen 3.6 currently contains two distinct open-weight model sizes. Qwen3.6-27B is a dense model, so all 27 billion language-model parameters participate in each token's forward pass. Qwen3.6-35B-A3B is a mixture-of-experts model with 35 billion total parameters and 3 billion activated for each token. The model card describes 256 experts, with eight routed experts plus one shared expert selected for a token. Both models are multimodal causal language models with a vision encoder. Both support a native context length of 262,144 tokens and can be extended to about one million tokens with the documented scaling configuration. The larger context still requires enough memory for the KV cache. The FP8 downloads are lower-precision representations of these two models, not additional family members. The main comparison is architectural. The dense 27B model reads a larger active weight set for every generated token. The 35B-A3B model stores more weights overall, but its router activates only 3B for a token. Total parameters therefore drive storage and memory capacity, while active parameters strongly affect computation and weight traffic during decoding. Sources: https://huggingface.co/Qwen/Qwen3.6-27B ; https://huggingface.co/Qwen/Qwen3.6-35B-A3B ; https://github.com/QwenLM/Qwen3.8 . Transition: map the broader open-weight catalogue by the origin of its developer organization.

Largest Open-Weight Models by Region

5B10B100B1T3T

Europe

FranceMistral Large 3675B

SwitzerlandApertus71B

SpainALIA / Salamandra40B

FinlandPoro34B

FinlandViking33B

GermanySoofi S Preview30B

NorwayNorOLMo13B

PolandBielik11B

ItalyModello Italia9B

ItalyMinerva7B

North America

United StatesNemotron 3 Ultra550B

United StatesLlama 4 Maverick400B

United Statesgpt-oss117B

China

ChinaKimi K32.8T

ChinaQwen 3.82.4T

ChinaDeepSeek V4 Pro1.7T

ChinaGLM-5.3753B

ChinaMiniMax M3428B

Japan

JapanLLM-jp-3172B

JapanFugaku-LLM13B

Selected model families · total parameters · logarithmic scale · 7 September 2026

16

This chart compares large open-weight releases by the origin of their developer. It is not a quality ranking and it is not an exhaustive industry census. The horizontal format lets us show Europe as a set of distinct national ecosystems rather than compressing the region into three narrow bars. The European examples are France, Switzerland, Spain, Finland, Norway, Poland and Italy. A model cannot send data out by itself. It receives tokens and produces tokens. Data can leave only when the surrounding agent runtime or one of its tools has network access and performs that action. The horizontal axis shows total parameters on a logarithmic scale. Each major step represents roughly a tenfold change, from 10B to 100B to 1T. We need this scale because the smallest shown model has 9B parameters while the largest has 2.8T. On a linear scale, the smaller European bars would almost disappear. The labels still state the absolute values so the audience does not have to infer them from bar length. Total parameters describe the weight set that must be stored or accessed. They do not tell us model quality or the compute used for one token. Several entries are mixture-of-experts models with much smaller active counts: Mistral Large 3 has 675B total and 41B active, Nemotron 3 Ultra has 550B total and 55B active, Llama 4 Maverick has 400B total and 17B active, gpt-oss-120b has 117B total and 5.1B active, Qwen3.8 2.4T-A95B has 95B active, MiniMax M3 has about 23B active, and Kimi K3 has 104B active. ALIA is Spain's public, open and multilingual AI infrastructure, coordinated by the Barcelona Supercomputing Center. ALIA-40B is a 40-billion-parameter foundation model trained on MareNostrum 5. It belongs to the same BSC language-model programme documented through the Salamandra work, so the chart uses “ALIA / Salamandra” to connect the two names. The additional European entries are Poro 34B and Viking 33B from Finland, Soofi S 30B from Germany, NorOLMo 13B from Norway, Bielik 11B from Poland, and Modello Italia 9B and Minerva 7B from Italy. Poro and Viking are separate Nordic model families. Soofi is visibly marked as a preview because the 30B model has been announced and shown in closed beta but is not yet an unrestricted open-weight release. Minerva is a separate Italian research family. Japan also includes Fugaku-LLM 13B, trained from scratch on the Fugaku supercomputer, alongside LLM-jp-3. Country or region refers to the main developer's home or the project's origin; it does not describe training-data provenance, hosting location or license conditions. Sources: https://huggingface.co/LumiOpen/Poro-34B ; https://www.amd.com/en/blogs/2024/viking-33b-scaling-performance-of-open-nordic-ai-models.html ; https://www.soofi.info/faq/ ; https://minerva-ai.org/en/docs/tech-page ; https://huggingface.co/HPLT/NorOLMo-13B ; https://huggingface.co/speakleash/Bielik-11B-v3.0-Instruct ; https://huggingface.co/sapienzanlp/modello-italia-9b-bf16 ; https://www.bsc.es/news/bsc-news/alia-europes-first-public-open-and-multilingual-ai-infrastructure ; https://huggingface.co/BSC-LT/ALIA-40b ; https://info.archives.global.fujitsu/global/about/resources/news/press-releases/2024/0510-01.html ; https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813 ; https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B ; https://huggingface.co/zai-org/GLM-5.3 ; https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512 ; https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 ; https://huggingface.co/meta-llama/Llama-4-Maverick-17B-128E-Instruct ; https://huggingface.co/openai/gpt-oss-120b ; https://huggingface.co/MiniMaxAI/MiniMax-M3 ; https://huggingface.co/moonshotai/Kimi-K3 ; https://huggingface.co/llm-jp/llm-jp-3-172b-instruct3 . Transition: next, turn model size into a concrete memory plan.

Memory Provisioning

RecommendedFully Served by VRAM

Sizing Criteria

Model Size
Quantization
KV Cache
Context Size
Operational Memory

17

The recommended target is to keep the complete inference workload in GPU memory. This avoids moving model data between storage, system RAM and VRAM while generating tokens. Start with the model size. Quantization determines how many bytes the stored weights require, plus scales and metadata. The KV cache needs additional memory and grows with the context size. Its exact size also depends on the model architecture, cache precision and batch size. Operational memory covers the inference engine, temporary compute buffers and memory allocator overhead. Leave headroom rather than planning exactly to the advertised VRAM capacity. On systems with unified memory, such as Apple silicon, CPU and GPU share one memory pool, but bandwidth, allocation limits and the operating system still reduce the amount available to the model. Transition: quantization changes the largest part of this calculation, the weight memory.

Quantization Principle

0:00.0 / 0:06.4

Original Range0–100

0100

18

47

83

÷ 10, then round

Quantized Range0–10

182

475

838

Reconstructed× 10

20Δ +2

50Δ +3

80Δ −3

18

Start with a deliberately simple example. The original values can occupy many positions between 0 and 100. We choose a scale of 10, divide each value by that scale and round to the nearest integer. That maps 18 to 2, 47 to 5 and 83 to 8. We now store one of only eleven integer values instead of a much finer original number. To reconstruct approximate original values, multiply the stored values by the same scale of 10. The results are 20, 50 and 80. Compared with 18, 47 and 83, the signed differences are +2, +3 and −3. This difference is quantization error: information discarded during rounding cannot be recovered exactly. Real model quantization can use different scales and zero points for tensors, channels or groups. Some methods also calibrate against data or treat important weights differently. The animation explains the basic idea, not a specific file format. Transition: translate the number of bits into model weight memory.

Weight Memory by Number Format

FormatOne ValueBytes70B Weights

FP3232 bits

4 B

280 GB

BF16 / FP1616 bits

2 B

140 GB

INT8 / FP88 bits

1 B

70 GB

INT4 / NF4 / FP44 bits

0.5 B

≈ 35–45 GB

Weight Memory ≈ Parameters × Bytes per ValueWeights only. Quantized files also store scales and metadata.

19

The arithmetic uses a dense model with seventy billion stored parameters. FP32 uses four bytes per value, so seventy billion multiplied by four gives about 280 decimal gigabytes. BF16 and FP16 use two bytes, which gives about 140 gigabytes. Eight-bit formats use one byte, which gives about 70 gigabytes. Four-bit values need half a byte in the ideal packed representation, which gives a raw weight size of 35 gigabytes. Real quantized files also contain scales, zero points, block metadata, alignment and sometimes tensors stored at higher precision, so a practical Q4 file can land above the raw figure. The 35 to 45 gigabyte range is a rule of thumb, not a format guarantee. This slide covers model weights only. The runtime also needs memory for the KV cache, temporary buffers and its own overhead. Mixture-of-experts models may activate only part of the model for each token, but all locally stored experts still contribute to the weight file unless the runtime streams or distributes them. Transition: choose the lowest precision that still clears the task-quality bar.

Choosing a Quantization Level

Qualityhigher fidelity

Memory and File Sizesmallerlarger

Local Sweet Spot

Q2emergency fit

Q3tight memory

Q4default start

Q5/6conservative

Q8near reference

BF16reference size

Practical RuleStart at Q4. Move up if quality breaks.
Move down only when the model does not fit.

20

Read this as a practical direction, not a universal benchmark curve. Moving from Q2 toward Q8 or BF16 usually increases file size and memory use while preserving more of the reference model's behavior. The exact quality difference depends on the model, the quantization algorithm, the dataset used for calibration, the runtime and the task. Labels such as Q4 or Q5 are families of formats rather than one standard method. A good local starting point is often a well-supported Q4 quantization. Move to Q5 or Q6 when the model fits comfortably and your evaluation finds meaningful quality loss. Move below Q4 only when memory capacity prevents the model from running, because aggressive quantization can damage reasoning and coding reliability. Q8 often stays close to the reference behavior but saves much less memory than Q4. BF16 represents the high-precision reference here. Always qualify the exact model file, not the label alone. Transition: video codecs show how compression methods can improve across generations.

Video Compression Over Time

The same mountain video frame — 1993MPEG-1≈ 600 MB

Illustrative file size for one minute of the same 1080p source at comparable visual quality.

21

The repeated image is deliberate. We compare how much data is needed for roughly the same perceived result, rather than showing a sequence of better pictures. The file sizes form an illustrative one-minute 1080p example. They use a simple bitrate scale of 80 megabits per second for MPEG-1, 40 for MPEG-2, 20 for H.264, 10 for HEVC, 8 for AV1 and 5 for VVC. File size equals bitrate multiplied by sixty seconds and divided by eight. That gives about 600, 300, 150, 75, 60 and 37.5 megabytes. These are not measurements from one universal benchmark. MPEG-1 was designed for Video CD-era CIF or SIF resolution, so its 1080p figure is a conceptual extrapolation. The later values make the documented generational improvements easy to see. H.264 can deliver comparable quality at roughly half the bitrate of MPEG-2. HEVC targets another reduction of about half. AV1 arrived in 2018 as an open, royalty-free codec and practical tests show substantial savings over H.264 and VP9. VVC targets about half the bitrate of HEVC. Actual sizes vary with the content, encoder implementation, profile, settings and quality target. Newer codecs can also require more computation. This is the analogy to model quantization: better algorithms can preserve more useful quality within a smaller representation. It does not prove that every newer quantization method or every quantized model is better. We still have to test the exact artifact on our tasks. Sources: MPEG-1 Video, published in 1993, https://mpeg.chiariglione.org/standards/mpeg-1/video.html; ITU-T H.262, July 1995, https://www.itu.int/rec/T-REC-H.262-199507-S/en; ITU-T H.264, May 2003, https://www.itu.int/rec/T-REC-H.264-200305-P; ITU HEVC announcement, January 2013, https://www.itu.int/net/pressoffice/press_releases/2013/01.aspx; AOMedia AV1 specification release, March 2018, https://aomedia.org/press%20releases/the-alliance-for-open-media-kickstarts-video-innovation-era-with-av1-release/; Meta AV1 comparison, April 2018, https://engineering.fb.com/2018/04/10/video-engineering/av1-beats-x264-and-libvpx-vp9-in-practical-use-case/; Fraunhofer HHI VVC overview, finalized July 2020, https://www.hhi.fraunhofer.de/en/departments/vca/technologies-and-solutions/h266-vvc/vvc-overview.html. Transition: next, see how inference uses compute and memory during prefill and decoding.

Prefill and Decoding without KV Cache

0:00.0 / 0:18.5

Prefill

Compute-Bound

Decode

Recomputes the Entire Context

User Input
Tool Response
Thinking

Pleasebuildatestedcomponent

withourexistingsignalstore

andaddthebackendintegration

Model Layers

Predicted Token

returnthefinalresult

returnthefinal

Inputreturnthefinal

22

This first animation deliberately assumes that no KV cache exists. During prefill, the complete input passes through every transformer layer and the model predicts the first new token. The generated token then joins the input. Without cached keys and values, the next prediction requires another forward pass over the complete sequence. The animation repeats the original five-token lines one after another, followed by the growing line of generated tokens. Only after every line has passed through all layers can the model produce the next token. The model parameters do not move into the model with every step. They remain loaded in memory and the computation reads them again for each pass. Reprocessing the whole context makes uncached generation increasingly expensive as the sequence grows. Transition: the KV cache avoids most of that repeated work.

KV Cache During Inference

0:00.0 / 0:12.5

Prefill

Builds the Cache

Decode

Reuses the Cache

User Input
Tool Response
Thinking

KV Cache

Pleasebuildatestedcomponent

withourexistingsignalstore

andaddthebackendintegration

Model Layers

Predicted Token

returnthefinalresult

returnthefinal

Inputreturnthefinal

23

Gemma 4 26B-A4B Memory Provisioning

IT QAT UD-Q4_K_XL by unsloth · 3.8B active / 25.2B total · one inference slot · F16 KV cache

Memory in VRAM / Unified Memory (GiB)

04812162022

16.72 GiB

13.27 GiB Model

1.45 GiB KV

2 GiB Runtime

64K

17.97 GiB

13.27 GiB Model

2.70 GiB KV

2 GiB Runtime

128K

20.47 GiB

13.27 GiB Model

5.20 GiB KV

2 GiB Runtime

256K

Context Size

All 25.2B parameters must fit in memory. Totals use a 2 GiB runtime reserve.

24

This is a planning estimate for the quantized Gemma 4 26B-A4B artifact shown in the reference. The weight file remains constant at 13.27 GiB for every context size. A4B means that about 3.8 billion of roughly 25.2 billion total parameters are active for a token. That can reduce compute, but it does not turn the model into a 4B memory footprint. All expert weights still have to reside in memory. The F16 KV cache grows from 1.45 GiB at 64K to 2.70 GiB at 128K and 5.20 GiB at 256K. The runtime estimate is a range of one to two GiB. The totals shown here use the conservative two GiB endpoint: 16.72, 17.97 and 20.47 GiB. These numbers plan capacity; the actual allocation depends on the runtime, backend, batch settings and model file. Read the startup logs before treating the fit as guaranteed. Transition: compare this mixture-of-experts model with the dense Gemma 4 12B.

Gemma 4 12B Memory Provisioning

QAT UD-Q4_K_XL by unsloth · one inference slot · F16 KV cache

Memory in VRAM / Unified Memory (GiB)

02468101214

9.57 GiB

6.26 GiB Model

1.31 GiB KV

2 GiB Runtime

64K

10.57 GiB

6.26 GiB Model

2.31 GiB KV

2 GiB Runtime

128K

12.57 GiB

6.26 GiB Model

4.31 GiB KV

2 GiB Runtime

256K

Context Size

Totals use the conservative 2 GiB endpoint of the estimated 1–2 GiB runtime range.

25

Qwen 3.6 35B-A3B Memory Provisioning

UD-Q4_K_XL by unsloth · 3B active / 35B total · 10 KV layers · one inference slot · F16 KV cache

Memory in VRAM / Unified Memory (GiB)

061218243036

24.11 GiB

20.86 GiB Model

1.25 GiB KV

2 GiB Runtime

64K

25.36 GiB

20.86 GiB Model

2.50 GiB KV

2 GiB Runtime

128K

27.86 GiB

20.86 GiB Model

5.00 GiB KV

2 GiB Runtime

256K

Context Size

Hybrid architecture: only 10 of 40 layers grow the KV cache. Totals use a 2 GiB runtime reserve.

26

This planning estimate uses the 22.4 GB decimal UD-Q4_K_XL artifact published by unsloth, which is 20.86 GiB. Qwen 3.6 35B-A3B has 35 billion total parameters and activates about 3 billion for each token. All weights still need to fit in memory. The model uses a hybrid layout: 30 layers use Gated DeltaNet linear attention and only 10 layers use full attention. Each full-attention layer has two KV heads with a head dimension of 256. With F16 keys and values, that produces 20 KiB of growing cache per token: 1.25 GiB at 64K, 2.50 GiB at 128K and 5 GiB at the native 256K context. The fixed DeltaNet state and ordinary compute buffers are covered here by a two GiB runtime reserve. The vision projector is not included because the coding setup uses text input. Actual llama.cpp allocation, checkpoint behavior and batch settings can require more headroom, so use its startup logs as the final authority. Sources: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF ; https://huggingface.co/Qwen/Qwen3.6-35B-A3B ; https://huggingface.co/Qwen/Qwen3.6-35B-A3B/raw/main/config.json . Transition: compare it with the dense 27B model, whose smaller weight file comes with a larger long-context cache.

Qwen 3.6 27B Memory Provisioning

UD-Q4_K_XL by unsloth · 27B dense · 16 KV layers · one inference slot · F16 KV cache

Memory in VRAM / Unified Memory (GiB)

061218243036

22.39 GiB

16.39 GiB Model

4.00 GiB KV

2 GiB Runtime

64K

26.39 GiB

16.39 GiB Model

8.00 GiB KV

2 GiB Runtime

128K

34.39 GiB

16.39 GiB Model

16.00 GiB KV

2 GiB Runtime

256K

Context Size

Hybrid architecture: 16 of 64 layers grow the KV cache. Totals use a 2 GiB runtime reserve.

27

This planning estimate uses the 17.6 GB decimal UD-Q4_K_XL artifact published by unsloth, which is 16.39 GiB. Qwen 3.6 27B is dense, but it still uses the same hybrid pattern as the mixture-of-experts model. Of its 64 layers, 48 use Gated DeltaNet linear attention and 16 use full attention. Each full-attention layer has four KV heads with a head dimension of 256. With F16 keys and values, the growing cache needs 64 KiB per token: 4 GiB at 64K, 8 GiB at 128K and 16 GiB at the native 256K context. This explains the counter-intuitive comparison: the 27B model has a smaller weight file than the 35B-A3B model, but it needs more memory at long context because it has more full-attention layers and twice as many KV heads. The fixed DeltaNet state and ordinary compute buffers are covered here by a two GiB runtime reserve. The vision projector is not included. Actual llama.cpp allocation, state checkpoints and batch settings can require more headroom. Sources: https://huggingface.co/unsloth/Qwen3.6-27B-GGUF ; https://huggingface.co/Qwen/Qwen3.6-27B ; https://huggingface.co/Qwen/Qwen3.6-27B/raw/main/config.json . Transition: hardware capacity is only one part of the decision. The next chapter asks what quality we can expect.

04

Expected Quality

28

Quality vs Data Sovereignty

Data Sovereignty

Quality

Local ModelsFrontier Models

Speed is a hardware budget question. Powerful local hardware can match cloud inference speed.

29

The horizontal position represents quality. Local models sit far to the right because they can deliver high-quality software work, although they do not currently reach the peak capability of frontier models. Frontier models therefore sit at the far-right edge. The vertical position represents full data sovereignty. Local inference can keep model execution and data inside infrastructure that the organization controls. Closed frontier models run on an external provider's infrastructure, so they sit at the bottom of this axis. Contracts, regional hosting and technical controls can improve privacy and governance, but they do not provide full data sovereignty under this definition. The gradient has no hard boundary because individual models, tasks and deployment choices vary, and the positions can change as models improve. Treat speed as a separate question. Local inference speed mainly depends on hardware capacity and budget. With sufficiently powerful accelerators and a comparable model configuration, a local system can match the latency or throughput of cloud inference. Local inference also makes costs more predictable because the organization pays for hardware, electricity, operations and engineering instead of a variable provider charge per token. Transition: use the Porsche analogy to ask whether every software task needs peak model capability.

A high-performance sports car caught in traffic and roadworks

The Porsche test

Do We Need Peak Capability for Every Trip?

Most software tasks never reach the empty, unrestricted Autobahn.

Benchmark philosophy

Qualification Instead of Ranking

Qualification threshold

Model A

Model B

Model C

Qualification criteria

✓Works with coding agents

✓Handles long-running tasks

✓At least 128K tokens of context

31

The qualification task

Full-Stack Capstone Task

Angular

Feature component
UI component
SignalStore
Unit test

Spring

JPA entities
Mapper
Controller
Database migration

Playwright

End-to-end test

32

The capstone is deliberately broad and delivered as one prompt. On the Angular side, the model must implement the feature according to existing architecture, separate feature and UI responsibilities, use a SignalStore, and add tests. On the Spring side, it must add the JPA model, mapper, controller and database migration. Finally, it must add a Playwright end-to-end test. This is difficult because success requires repository discovery, architectural consistency, cross-boundary changes and verification—not because one file is exotic. A configuration that repeatedly completes it can be evaluated as a candidate for this environment. Logo sources: https://simpleicons.org/?q=angular, https://simpleicons.org/?q=spring, https://playwright.dev/img/playwright-logo.svg. Transition: show the exact one-prompt specification used for the capstone.

The Capstone Prompt

/goal

In my holidays overview I want a German vocabulary trainer, similar to the existing quiz feature. For destinations in German-speaking places (currently Lübeck and Vienna) users should be able to practice a few basic German words before they travel: the holiday card shows an icon when a vocabulary test is available, and opening it lets the user translate words.

Do a full-stack implementation (backend + frontend) following the structure and conventions of the existing quiz feature (architecture, feat folder, backend entity/repository/controller). The app must be accessible, and the accessible names below are required. Create and run e2e tests for the feature.

Acceptance Criteria

1Successful Verification

The frontend build check is pnpm ng build --optimization false.
Start any required application dependencies using the project's existing setup, then start the backend with ./gradlew bootRun.
Wait for the backend to become ready and verify that both the existing /heartbeat and /holiday endpoints respond successfully. Starting the process alone is not sufficient.
Start the frontend on port 4200 and run the vocabulary feature's e2e tests with the project's Playwright setup.
If compilation, application startup, an endpoint request, or an e2e test fails, diagnose the failure, fix the implementation, and repeat the relevant checks. Do not finish while a required check is failing.
Stop all application processes and temporary services that you started after verification, including when a check fails.

2Vocabulary Data

A vocabulary test consists of exactly 5 words. Each is an English word the user must translate into German.
Use exactly these 5 English -> German pairs, stored verbatim (capitalization and umlauts matter):

squirrel      -> Eichhörnchenturtle        -> Schildkröterefrigerator  -> Kühlschranktoothbrush    -> ZahnbürsteGermany       -> Deutschland

The same 5 words are used for every destination that offers the test. Seed this data (e.g. a Flyway migration) for the German-speaking destinations Lübeck and Vienna.

3Discovery on the Holiday Card

A destination that has vocabulary data shows a vocabulary icon/link on its holiday card in the holidays overview, exactly like the quiz icon. Drive this off the presence of vocabulary data — do NOT hardcode a list of cities.
The control is an accessible link with the accessible name "Vocabulary Test".
Destinations without vocabulary data must not show it.

4The Vocabulary Test

Opening the "Vocabulary Test" link shows the 5 questions and a status area. Navigation, dialog, or inline panel — your choice.
For each question, show the English word and provide a free-text input for the German translation. Each input's accessible name must contain its English word (e.g. an input labelled "squirrel"), so every input is individually identifiable regardless of order or layout.
A status area shows three counts, each exposed with exactly these accessible names (value included): "Unanswered: N", "Correct: N", "Incorrect: N".
The three counts are mutually exclusive and always sum to 5. Initial state: Unanswered: 5, Correct: 0, Incorrect: 0.

5Answering

An answer is evaluated when the user leaves (blurs) a non-empty input. There is no submit button — the counts update live on blur.
Blurring an empty input does nothing; the question stays unanswered.
Comparison is exact and case- and umlaut-sensitive: the typed text must exactly equal the stored German word. "Kühlschrank" is correct; "kühlschrank", "Kuhlschrank", and "Kühlschrank " are all incorrect. No trimming or normalization.
Once a non-empty input has been evaluated it is locked (cannot be edited again). The counts update accordingly: Unanswered decreases by 1, and Correct or Incorrect increases by 1.

33

This is the exact single prompt used for the capstone. Do not read every line aloud. First point out that the goal describes a user-facing feature and asks the model to follow an existing full-stack architecture. Then highlight the deterministic proof of completion: build the frontend, start the backend, wait for readiness, probe two endpoints, start the frontend and run Playwright. The data, accessible names and interaction rules are intentionally exact, including the five translations, live evaluation on blur and case-sensitive comparison. This prompt therefore tests more than code generation. It tests repository discovery, architectural consistency, accessibility, data migration, end-to-end integration and the model's ability to verify and repair its work. The entire feature and its evidence of completion are requested in one prompt. Transition: compare this specification with Claude's goal-review workflow.

Claude workflow

Claude Goal Review

Goal

→

Model Sessionkeeps running

Model runs→Decides “done”

Context

grows every round

→

Goal Reviewseparate model

✓

Not completeReturn to the same session

34

Claude starts a model session with the goal. The model plans, edits, calls tools and eventually decides that it has finished. Claude then asks a separate model to review the result against the original goal. The important detail is that this review does not destroy or replace the working model session. If the reviewer finds missing work, its feedback returns to that same session. The model continues with its existing conversation, previous tool results, implementation history and the new review findings. This preserves useful context, but it also means that the context keeps growing after every round. Old assumptions, failed attempts and repeated tool output remain in the session unless the runtime compresses them. A longer session can therefore become more expensive and harder for a smaller local model to use reliably. A successful goal review ends the loop. Transition: our benchmark uses a different control structure that stops the worker after every attempt.

05

Engineering Around the Model

35

Local Models Can Run 24/7

Use autonomous implementation loops with clear safeguards.

36

Harness / loop engineering

Our Verification Loop

Issue / Spec Review3 rounds

Deterministic Verification3 rounds

Fresh Model Session

→

One Try

→

Stop

→

Build · Tests · Browser

Validation Agentinterprets the failure

→

Explanation

new session

✓

Issue / Spec Reviewerchecks the implementation

Changes requiredRestart with reviewer findings

Complete

37

Our benchmark stops the working model after one implementation attempt. We do not ask that session to judge whether it should continue, and we do not retain it for a repair round. The harness runs deterministic verification such as builds, unit tests and Playwright. If a check fails, a validation agent interprets the raw output and writes a concise explanation of what went wrong. The next attempt starts in a fresh model session with the goal, the current repository state and that explanation. The inner deterministic verification loop allows up to three rounds. Once all deterministic checks pass, the outer issue and specification review starts. A separate reviewer checks whether the implementation actually satisfies the requested behavior and architectural constraints, including requirements that a compiler or test suite may not cover. If that reviewer asks for changes, the harness starts the implementation and deterministic verification cycle again with the reviewer findings. The outer review loop also allows up to three rounds. Completion requires both deterministic validation and issue or specification approval. Fresh sessions keep each worker context bounded, while the repository and explicit feedback carry progress between attempts. Transition: place this verification loop inside the production workflow that coordinates requests, local execution and human approval.

Implementation Loop for Production Systems

GitHub coordinates the request. The implementation workflow runs on our local machine.

GitHubRemote coordination

Our Local MachineSandboxed execution

System Boundary

PMProduct ManagerWrites and clarifies

#GitHub IssueFeature request

CIGitHub ActionsDispatches the job

Dispatch Job

RUNLocal RunnerReceives the job

MMastraStarts a durable workflow

Mastra Workflow

?ReadinessAsk or continue

WTWorktreePrepare branch

</>CodexGoal, skills, hooks

9Review AgentsNine specialists

✓Main ReviewerRepair, decide, PR

Missing Information Return to Product Manager Changes Requested Return to Codex Conflict Human Decision

Approved: Publish PR

Result Returns to GitHub

MergedChange deliveredDeveloper ReviewHuman approvalPull RequestPublished by workflow

38

This diagram places the verification loop inside a production process. The product manager creates and clarifies a GitHub issue. GitHub Actions coordinates the request and dispatches a job across the system boundary to a runner on our local machine. The runner starts a durable Mastra workflow. Readiness checks whether the request contains enough information. The workflow then creates an isolated worktree and starts Codex with the goal, skills and hooks. Nine specialist review agents inspect different aspects of the change, and a main reviewer combines their findings and decides what happens next. The workflow has explicit exits. Missing information returns to GitHub and the product manager. Requested changes start another Codex implementation round. A conflict or an ambiguous decision stops automation and asks a human. When the implementation is acceptable, the workflow publishes a pull request. A developer performs the final review before GitHub merges the change. GitHub remains the coordination and audit surface. Mastra, Codex, the model and the worktree remain on the local machine. Transition: close the webinar and invite questions.

Thank you

Questions?

Local models for AI-assisted coding

Feedback and resources

soverius.ai/feedback/740bbb39

Local Models forAI-Assisted Coding

Rainer Hahnekamp

We Build Production-ReadyAI Systems

Local Models

AI Agents

Agentic UIs

Agenda

Why Local Models?

Reasons to Run Models Locally

Cost Control

Privacy

Independence

Setup and Configuration

Required Components and Our Setup

Local Inference Demo

Hardware Provisioning

CPU and GPU Inference

Local Inference Hardware

RTX PRO 4000

Mac Studio

ASUS Ascent GX10

RTX PRO 6000 Blackwell

Model Size by Deployment Class

Model Availability by Organization

Gemma 4 Family

Qwen 3.6 Family

Largest Open-Weight Models by Region

Memory Provisioning

Sizing Criteria

Quantization Principle

Weight Memory by Number Format

Choosing a Quantization Level

Video Compression Over Time

Prefill and Decoding without KV Cache

Prefill

Decode

KV Cache During Inference

Prefill

Decode

Gemma 4 26B-A4B Memory Provisioning

Gemma 4 12B Memory Provisioning

Qwen 3.6 35B-A3B Memory Provisioning

Qwen 3.6 27B Memory Provisioning

Expected Quality

Quality vs Data Sovereignty

Do We Need Peak Capability for Every Trip?

Qualification Instead of Ranking

Full-Stack Capstone Task

Angular

Spring

Playwright

The Capstone Prompt

1Successful Verification

2Vocabulary Data

3Discovery on the Holiday Card

4The Vocabulary Test

5Answering

Claude Goal Review

Engineering Around the Model

Local Models Can Run 24/7

Our Verification Loop

Implementation Loop for Production Systems

Questions?

We Build Production-Ready
AI Systems