Private Local AI

Private Local AI

Nov 12

One gaming PC, one 552B MoE: how expert offload actually works

#localai #moe #selfhosted #llm

4 min read

Private Local AI

Nov 12

Hot experts, cold RAM: a big MoE on a box with more memory than VRAM

#localai #moe #llm #hardware

3 min read

Private Local AI

Nov 11

Ten engines for one box: which local inference engine fits a 100 GB rig with two 16 GB cards?

#ai #hardware #llm #machinelearning

11 min read

Private Local AI

Nov 11

ExLlamaV3 and TabbyAPI - the pair that ships today, and what it asks of your hardware

#ai #hardware #llm #performance

7 min read

Private Local AI

Nov 11

Aphrodite Engine - a serving engine in the vLLM lineage, now renamed Sonar, and what my two 16 GB cards would need from it

#ai #llm #opensource #python

6 min read

Private Local AI

Nov 11

exo - splitting one model across several machines, and the rules it sets for joining

#ai #llm #opensource #python

6 min read

Private Local AI

Nov 11

MCP servers, and what they let a local model actually do: Blender, CADCraft and the DWG problem

#localai #opensource #selfhosted #machinelearning

17 min read

Private Local AI

Nov 11

Xolotl - running a 552-billion MoE on one gaming PC, and what it asks of a constrained machine

#localai #opensource #selfhosted #machinelearning

7 min read

Private Local AI

Nov 10

Si usas IA de EE. UU. con datos de clientes, estás en violación de la ley de datos. Evita problemas: usa IA local privada

#privacy #gdpr #ai #law

8 min read

Private Local AI

Nov 10

Nix and NixOS: the four horsemen of the Nixopalypse

#linux #nixos #opensource #devops

9 min read

Private Local AI

Nov 10

Ne plus être en infraction avec les lois UE sur la protection des données privées, face aux privilèges et abus de l’IA américaine (US CLOUD ACT)

#privacy #gdpr #ai #law

6 min read

Private Local AI

Nov 10

PolyStrata – the Strata fork that stopped being one engine, and what that means against llama.cpp, Strata and Avalon

#ai #research #opensource #performance

5 min read

Private Local AI

Nov 10

Avalon – what it is, and the three ways it differs from llama.cpp and Strata

#ai #opensource #linux #privacy

7 min read

Private Local AI

Nov 10

Strata – the engine that stopped pretending one size fits all

#ai #opensource #performance #linux

4 min read

Private Local AI

Nov 10

Llama.cpp – What is it?

#ai #opensource #linux #beginners

4 min read

Private Local AI

Nov 10

Model quantization, read carefully: what the bit-widths really cost you

#ai #performance #storage #quantization

6 min read

Private Local AI

Nov 10

GSQ and RCO: the non-uniform quantization now sitting in GGUF files

#ai #performance #opensource #quantization

7 min read

DEV Community

Badges

Writing Debut

One gaming PC, one 552B MoE: how expert offload actually works

Hot experts, cold RAM: a big MoE on a box with more memory than VRAM

Ten engines for one box: which local inference engine fits a 100 GB rig with two 16 GB cards?

ExLlamaV3 and TabbyAPI - the pair that ships today, and what it asks of your hardware

Aphrodite Engine - a serving engine in the vLLM lineage, now renamed Sonar, and what my two 16 GB cards would need from it

exo - splitting one model across several machines, and the rules it sets for joining

MCP servers, and what they let a local model actually do: Blender, CADCraft and the DWG problem

Xolotl - running a 552-billion MoE on one gaming PC, and what it asks of a constrained machine

Si usas IA de EE. UU. con datos de clientes, estás en violación de la ley de datos. Evita problemas: usa IA local privada

Nix and NixOS: the four horsemen of the Nixopalypse

Ne plus être en infraction avec les lois UE sur la protection des données privées, face aux privilèges et abus de l’IA américaine (US CLOUD ACT)

PolyStrata – the Strata fork that stopped being one engine, and what that means against llama.cpp, Strata and Avalon

Avalon – what it is, and the three ways it differs from llama.cpp and Strata

Strata – the engine that stopped pretending one size fits all

Llama.cpp – What is it?

Model quantization, read carefully: what the bit-widths really cost you

GSQ and RCO: the non-uniform quantization now sitting in GGUF files