Meta tags:
description=Mesh serves large local models across multiple machines through one OpenAI-compatible endpoint.;

Headings (most frequently used words):

mesh, ready, model, distributedmodel, inference, one, local, api, two, modes, pluggable, architecturefor, distributed, agents, refs, to, copy, sponsors, site, socials, the, field, selects, host, layer, ranges, become, pipeline,

Text of the page (most frequently used words):
q4_k_m (70), machine (42), gguf (40), qwen3 (39), single (32), mesh (27), unsloth (26), model (24), instruct (24), layers (17), 32b (16), qwen2 (15), llm (13), gemma (12), models (12), coder (12), qwen (12), llama (12), multi (11), for (11), q4_k_xl (11), thinking (10), 14b (10), deepseek (10), layer (9), install (8), glm (8), and (8), moe (8), tool (8), chat (8), bartowski (8), the (7), split (7), any (7), vision (7), node (7), nodes (7), mistral (7), openai (6), small (6), top (6), strong (6), general (6), calling (6), run (6), hosted (6), public (5), http (5), localhost (5), 30b (5), experts (5), inference (5), distill (5), code (5), routes (5), catalog (4), client (4), 26b (4), a4b (4), serve (4), cloud (4), curl (4), 12b (4), good (4), coding (4), a3b (4), modes (4), across (4), draft (4), active (4), into (4), kimi (4), runtime (4), your (4), large (4), gpus (4), smaller (4), peers (3), air (3), setup (3), bash (3), https (3), meshllm (3), fssl (3), copy (3), package (3), size (3), 480b (3), use (3), non (3), dense (3), 235b (3), minimax (3), reasoning (3), 27b (3), goose (3), mode (3), data (3), serving (3), ready (3), api (3), capabilities (3), tools (3), warm (3), route (3), get (3), one (3), stay (2), become (2), pipeline (2), openai_base_url (2), 3131 (2), point (2), 8gb (2), cargo (2), brew (2), google (2), a35b (2), requires (2), agentic (2), 128 (2), tier (2), a22b (2), proven (2), 24b (2), flash (2), fast (2), devstral (2), 2505 (2), frontier (2), distilled (2), scout (2), 17b (2), meta (2), google_gemma (2), gen (2), default (2), hermes (2), pro (2), agents (2), next (2), packages (2), quant (2), live (2), refs (2), rest (2), endpoints (2), backends (2), compatible (2), endpoint (2), coordination (2), blackboard (2), manifests (2), processes (2), events (2), more (2), plugins (2), starts (2), over (2), distributed (2), request (2), activations (2), weights (2), local (2), tokens (2), l53 (2), l27 (2), computer (2), 9337 (2), messages (2), post (2), completions (2), many (2), land (2), right (2), sized (2), memory (2), bound (2), slices (2), pieces (2), server (2), bigmodel (2), bigger (2), subscribe, loop, license, roadmap, github, socials, documentation, home, site, sponsor, sponsors, ttft, 187, stream, export, vram, friendly, proxy, listening, joined, 217, discovering, via, nostr, auto, join, this, can, finish, tap, executable, macos, linux, zsh, punches, above, weight, text, replaces, with, image, understanding, quality, balance, 508, tiny, ocr, screenshots, runs, anywhere, 294, 35b, largest, mid, its, starter, available, best, 134, 22b, tok, 397, 2503, 138, 456b, 46b, 256, 0414, 382, 671b, glogwa68, 109b, latest, 780, fills, gap, between, solid, 760, 85b, generation, completion, matches, gpt, 491, 646, links, ref, source, hardware, long, context, capability, search, browse, hugging, face, see, which, ship, org, repo, channels, gossip, settings, tweaks, configuration, metrics, agent, state, persistence, blobstore, loads, plugin, declared, declare, what, they, provide, manifest, them, calls, exposes, mcp, pluggable, architecture, 168, path, quic, return, rtx, 3090, 7800, prompt, ingest, ultra, same, call, selected, inventory, planner, ranges, field, selects, host, router, place, stages, ingress, while, splits, nearby, two, dev, opencode, vscode, drop, you, already, phase, timeline, mini, 24gb, laptop, 32gb, gpu, rig, 96gb, workstation, 192gb, 768gb, ack, commit, mapped, plan, committed, 18b, 16b, 15b, 13b, 11b, 10b, 242gb, 120b, params, shard, container, sponsored, learn, homelab, business, internet, without, buying, back, streaming, routing, docs,

Text of the page (random words):
mesh llm mesh llm public mesh catalog docs 1 1k v0 72 2 install bigmodel v3 xl routing request to server node streaming tokens back run bigger models without buying bigger gpus distributed model inference run any model across nodes in your homelab in your business or on the internet large models get split into smaller pieces memory bound gpus run smaller layer slices and layers land on right sized nodes curl fssl https meshllm cloud install sh bash mesh llm setup public mesh learn more sponsored by llm container bigmodel v3 xl ready to shard params 120b weights 242gb layers 10 l0 9b l1 10b l2 11b l3 12b l4 13b l5 14b l6 15b l7 16b l8 17b l9 18b split plan committed 10 layers mapped across 6 nodes l0 l9 6 nodes live commit route 6 ack server 768gb hosted layers cloud node 192gb hosted layers workstation 96gb hosted layers gpu rig 32gb hosted layers laptop 24gb hosted layers mini pc 8gb hosted layers large models get split into smaller pieces memory bound gpus run smaller layer slices layers land on right sized nodes timeline phase 1 01 08 drop in for the tools you already use goose vscode opencode pi dev any openai client one local api two mesh modes point tools at localhost while mesh routes by model or splits one large model across nearby gpus openai ingress openai_base_url http localhost 9337 v1 run many models many route by model split one large model split place layer stages router mode the model field selects a host get v1 models data id llama 3 2 1b id qwen3 4b post v1 chat completions model qwen3 4b messages split mode layer ranges become a pipeline post v1 chat completions model glm 4 5 air messages mesh planner l0 26 l27 52 l53 78 client your computer localhost 9337 model qwen3 4b inventory route warm node 01 llama 3 2 1b selected node 02 qwen3 4b warm node 03 glm 4 5 air client your computer same v1 call l0 26 m3 ultra prompt ingest 4 ms l27 52 rx 7800 xt activations 11 ms l53 78 rtx 3090 tokens return weights stay local activations over quic active request path warm model 3 nodes 168 gb mesh pluggable architecture for distributed agents plugins declare what they provide in a manifest the runtime starts them routes calls and exposes capabilities over mcp http inference and mesh events mesh runtime loads manifests starts plugin processes routes declared capabilities blobstore state persistence blackboard agent coordination openai endpoint openai compatible api more plugins metrics data tools configuration settings tweaks inference backends model serving http routes rest endpoints mesh events channels gossip mesh runtime manifests processes capabilities blackboard coordination openai endpoint compatible api inference backends model serving http routes rest endpoints mesh ready model refs ready to copy browse live hugging face catalog data copy org repo quant refs and see which models ship layer packages for multi machine serving search models capability any code vision long context runtime any multi machine single machine hardware any 8 gb 11 20 gb 40 gb source any qwen glm gemma deepseek kimi llama mistral minimax model size quant run mode layer packages model ref links kimi k2 thinking layer package for unsloth kimi k2 thinking gguf ud q4_k_xl 646 2 gb ud q4_k_xl multi machine 61 layers unsloth kimi k2 thinking gguf ud q4_k_xl qwen2 5 0 5b instruct draft for qwen2 5 and deepseek r1 distill models 491 mb q4_k_m single machine qwen qwen2 5 0 5b instruct gguf q4_k_m qwen2 5 3b instruct small fast general chat 2 1 gb q4_k_m single machine qwen qwen2 5 3b instruct gguf q4_k_m qwen2 5 coder 32b instruct top tier code gen matches gpt 4o on code 20 gb q4_k_m single machine qwen qwen2 5 coder 32b instruct gguf q4_k_m qwen2 5 coder 7b instruct code generation completion 4 4 gb q4_k_m single machine qwen qwen2 5 coder 7b instruct gguf q4_k_m qwen3 coder next 85b dense frontier coding model 48 gb q4_k_m single machine qwen qwen3 coder next gguf q4_k_m hermes 2 pro mistral 7b goose default strong tool calling for agents 4 4 gb q4_k_m single machine bartowski hermes 2 pro mistral 7b gguf q4_k_m llama 3 2 1b instruct draft for llama 3 x and llama 4 models 760 mb q4_k_m single machine bartowski llama 3 2 1b instruct gguf q4_k_m llama 3 2 3b instruct meta llama 3 2 goose default good tool calling 2 0 gb q4_k_m single machine bartowski llama 3 2 3b instruct gguf q4_k_m qwen2 5 14b instruct solid general chat 9 0 gb q4_k_m single machine bartowski qwen2 5 14b instruct gguf q4_k_m qwen2 5 32b instruct proven general chat 20 gb q4_k_m single machine bartowski qwen2 5 32b instruct gguf q4_k_m qwen2 5 coder 14b instruct strong code gen fills gap between 7b and 32b 9 0 gb q4_k_m single machine bartowski qwen2 5 coder 14b instruct gguf q4_k_m gemma 3 1b it draft for gemma 3 models 780 mb q4_k_m single machine bartowski google_gemma 3 1b it gguf q4_k_m gemma 3 27b it google gemma 3 27b strong reasoning 17 gb q4_k_m single machine bartowski google_gemma 3 27b it gguf q4_k_m llama 4 scout moe 109b 17b active 16 experts top 1 meta latest tool calling 22 5 gb q4_k_m single machine glogwa68 llama 4 scout gguf q4_k_m deepseek r1 distill qwen 14b deepseek r1 reasoning distilled into qwen 14b 9 0 gb q4_k_m single machine unsloth deepseek r1 distill qwen 14b gguf q4_k_m deepseek r1 distill qwen 32b deepseek r1 reasoning distilled into qwen 32b 19 9 gb q4_k_m single machine unsloth deepseek r1 distill qwen 32b gguf q4_k_m deepseek v3 2 moe 671b frontier general model requires multi node 382 gb ud q4_k_xl multi machine 61 layers unsloth deepseek v3 2 gguf ud q4_k_xl devstral small 2505 mistral agentic coding tool use 14 3 gb q4_k_m single machine unsloth devstral small 2505 gguf q4_k_m glm 4 32b strong 32b good tool calling 19 7 gb q4_k_m single machine unsloth glm 4 32b 0414 gguf q4_k_m glm 4 7 flash moe 30b 3b active 64 experts top 4 fast inference tool calling 18 gb q4_k_m single machine unsloth glm 4 7 flash gguf q4_k_m minimax m2 5 moe 456b 46b active 256 experts top 8 q4_k_m 138 gb q4_k_m single machine unsloth minimax m2 5 gguf q4_k_m mistral small 3 1 24b instruct mistral general chat good tool calling 14 3 gb q4_k_m single machine unsloth mistral small 3 1 24b instruct 2503 gguf q4_k_m qwen3 0 6b draft for qwen3 models 397 mb q4_k_m single machine unsloth qwen3 0 6b gguf q4_k_m qwen3 14b qwen3 strong chat thinking modes 9 0 gb q4_k_m single machine unsloth qwen3 14b gguf q4_k_m qwen3 235b a22b moe 235b 22b proven at 16 tok s across 2 nodes 134 gb ud q4_k_xl multi machine 94 layers unsloth qwen3 235b a22b gguf ud q4_k_xl qwen3 30b a3b moe general chat 128 experts top 8 thinking non thinking 17 3 gb q4_k_m single machine unsloth qwen3 30b a3b gguf q4_k_m qwen3 32b best qwen3 dense thinking non thinking modes 19 8 gb q4_k_m single machine unsloth qwen3 32b gguf q4_k_m qwen3 32b ud qwen3 32b dense layer package available 20 0 gb ud q4_k_xl multi machine 64 layers unsloth qwen3 32b gguf ud q4_k_xl qwen3 4b qwen3 starter thinking non thinking modes 2 5 gb q4_k_m single machine unsloth qwen3 4b gguf q4_k_m qwen3 8b qwen3 mid tier strong for its size 5 0 gb q4_k_m multi machine 36 layers unsloth qwen3 8b gguf q4_k_m qwen3 coder 30b a3b instruct moe agentic coding tool use 128 experts top 8 18 6 gb q4_k_m single machine unsloth qwen3 coder 30b a3b instruct gguf q4_k_m qwen3 coder 480b a35b instruct moe 480b 35b largest coding model requires multi node 294 gb ud q4_k_xl multi machine 62 layers unsloth qwen3 coder 480b a35b instruct gguf ud q4_k_xl qwen3 5 0 8b vision tiny vision model ocr screenshots runs anywhere 508 mb q4_k_m single machine unsloth qwen3 5 0 8b gguf q4_k_m qwen3 5 4b vision small vision model good quality size balance 2 7 gb q4_k_m single machine unsloth qwen3 5 4b gguf q4_k_m qwen3 5 9b vision vision text replaces qwen3 8b with image understanding 5 8 gb q4_k_m single machine unsloth qwen3 5 9b gguf q4_k_m gemma 3 12b it google gemma 3 12b punches above weight 7 3 gb q4_k_m single machine unsloth gemma 3 12b it gguf q4_k_m gemma 4 26b a4b it layer package for unsloth gemma 4 26b a4b it gguf ud q4_k_m 16 9 gb ud q4_k_m multi machine 30 layers unsloth gemma 4 26b a4b it gguf ud q4_k_m 38 models curl brew cargo zsh mesh llm copy install the executable macos linux curl fssl https meshllm cloud install sh bash curl fssl https meshllm cloud install sh bash brew install mesh llm tap mesh llm cargo install mesh llm finish setup mesh llm setup join the public mesh and serve if this machine can mesh llm serve auto discovering peers via nostr ok joined public 217 peers ok proxy listening on localhost 3131 ok or serve an 8gb vram friendly catalog model mesh llm serve model gemma 4 26b a4b it ud q4_k_m point any openai client at mesh export openai_base_url http localhost 3131 v1 stream glm 4 5 air q4_k_m 42 t s ttft 187 ms pipeline split 3 peers sponsors become a sponsor mesh llm v0 72 2 mesh llm site home documentation model catalog install mesh socials github public mesh roadmap license stay in the loop subscribe