Open-Sourcing Ling-3.0-tiny: Agentic AI Goes Local

Large language models (LLMs) have become remarkably good at using tools—the capability at the core of agentic AI. Give a capable model an API schema or a set of functions, and it can choose the right tool, produce structured arguments, interpret the result, and continue a multi-step workflow.

But there is a practical problem: the models that do this well are often expensive to run.

That cost limits where agentic AI can go. It makes always-on assistants harder to deploy on personal devices. It raises the price of high-volume services. It also pushes sensitive data toward remote APIs, even when the workflow would be better kept on-device or inside a private network.

This leads to a question we believe deserves more attention: Can the function-calling and reasoning capabilities of large models be carried into a model small enough to run locally, without reducing it to a narrow demonstration?

We recently open-sourced Ling-3.0-tiny by releasing its model weights in BF16, FP8, and INT4.

Ling-3.0-tiny has 7.9 billion total parameters, but activates only 1.3 billion parameters per token. It is built to make reasoning, tool use, and agent execution practical under real hardware and cost constraints—not simply to compete on parameter count.

That distinction matters. The 1.3B figure describes the compute activated for each token; it does not mean the model stores only 1.3B parameters. Memory use still depends on the full checkpoint, numerical precision, runtime, KV cache, and context length. The opportunity is lower active compute without giving up the broader capability surface of a larger sparse model.

Function calling is an execution problem

Function calling is sometimes described as structured text generation: select a function name and fill in a JSON object. In production, the problem is much harder.

A useful agent must understand the user’s intent, decide whether a tool is needed, select the right tool, construct valid arguments, preserve context across steps, and recover when a result changes the plan. A syntactically valid call can still be operationally wrong. For a coding agent, an enterprise workflow, or a device automation system, reliability matters as much as format compliance.

This is also why simply shrinking an LLM is not enough. A small model may retain fluency while losing the task understanding, instruction following, reasoning, or context management required to use tools well.

Ling-3.0-tiny was designed around the broader capability stack behind tool use. In our reported evaluations, it reaches 62.72 on BFCL v4, alongside balanced results in general agent tasks, coding agents, reasoning, instruction following, and long-context understanding.

BFCL is only one benchmark, so the score should be read together with the broader profile. Ling-3.0-tiny also shows capability in general agents, coding agents, instruction following, reasoning, and long-context tasks—the combination needed to move from a formatted function call toward useful task execution.

The goal, therefore, is not a single-purpose function router. It is a compact model that can understand, decide, call, observe, and continue—while remaining small enough to place closer to the device, data, and workflow.

Small active compute, broad task coverage

On the Artificial Analysis Intelligence Index v4.1.1, Ling-3.0-tiny scores 25. On the Artificial Analysis Agentic Index, it scores 16.

Figure 1. Artificial Analysis Intelligence Index v4.1.1.

Those numbers are notable not because a tiny model should replace every larger model—it should not—but because they place 1.3B active parameters in a capability range normally associated with much larger systems.

Figure 2. Artificial Analysis Agentic Index.

The model’s profile is deliberately balanced:

General agents: 772 Elo on GDPval-AA v2 and 20.80 on τ³-Banking-AA
Function calling: 62.72 on BFCL v4
Coding agents: 27.70 on Terminal-Bench 2.1 and 47.93 on ArtifactsBench
Long context: 58.70 on AA-LCR
Reasoning: 73.40 on GPQA Diamond and 71.03 on IMO-AnswerBench
Instruction following: 63.61 on IFBench and 83.15 on Multi-IF

These are reported benchmark results, not a claim that model quality can be reduced to one score. Real deployments should still be evaluated against their own tools, schemas, failure costs, and latency requirements. But the results show that low active parameter count no longer has to mean a model limited to simple requests.

Figure 3. Ling-3.0-tiny benchmark profile across agentic, coding, long-context, knowledge, reasoning, and instruction-following tasks.

An architecture built for useful efficiency

Ling-3.0-tiny uses the native hybrid-linear architecture of the Ling-3.0 family, optimized for lightweight deployment.

Figure 4. Ling-3.0-tiny architecture: sparse MoE with a 3:1 KDA-MLA hybrid attention stack.

Its attention stack contains three Kimi Delta Attention (KDA) layers followed by one Multi-Head Latent Attention (MLA) layer in each four-layer block—a 3:1 KDA-MLA pattern. This combines efficient long-context processing with expressive global attention.

Its sparse mixture-of-experts (MoE) feed-forward network contains 128 routed experts. For each token, the model activates eight routed experts and one shared expert. That is how a model with 7.9B total parameters can expose a wider capability surface while keeping active compute to 1.3B parameters per token.

Native hybrid reasoning allows developers to choose fast responses for routine tasks or multi-step thinking for harder ones through the same model. Multi-Token Prediction provides a foundation for faster decoding.

The architecture supports context lengths of up to 1 million tokens, but practical context length depends on runtime, memory, and deployment configuration. The current SGLang recipe demonstrates a 256K configuration, while the reported Mac memory measurement uses an 8K context. Keeping these conditions explicit matters: long-context capability and efficient local deployment are related goals, but they are not the same operating point.

The design principle is straightforward: architecture efficiency matters when it turns into task efficiency—less waiting, lower serving cost, and more places where an agent can run.

From a DGX Spark to a MacBook

Open weights matter most when developers can actually deploy them. That is why Ling-3.0-tiny is available in three formats:

BF16 for full-precision evaluation and production use
FP8 for a balance of model quality, speed, and memory use
INT4 for more resource-constrained local environments

The model has already been validated on NVIDIA DGX Spark, Apple Silicon MacBook, and Mac mini.

With FP8, it reaches approximately 100–105 tokens per second on a DGX Spark. On an M4 Pro MacBook, it reaches approximately 86–90 tokens per second, with around 8.34 GiB peak memory use at an 8K context length. In Artificial Analysis testing, the model delivered more than 160 output tokens per second and took roughly 18 seconds to produce 500 tokens, including reasoning time.

These figures come from different hardware and test setups, so they should not be treated as direct comparisons. Together, however, they illustrate the deployment range: from a compact AI workstation serving a small team to a personal laptop running a private assistant.

Figure 5. Intelligence Index versus output speed.

Figure 6. Intelligence Index versus end-to-end response time.

A deployment guide grounded in real hardware

The three open-weight formats are intended for different deployment priorities. BF16 is positioned for research evaluation and production applications that prioritize full precision. FP8 balances model capability, runtime speed, and resource use. INT4 lowers the deployment threshold further for more resource-constrained local environments.

For a compact local service, FP8 inference has been validated on a single NVIDIA DGX Spark. The model weight files are approximately 7.85GB. With a 2K-token input, single-request sustained decoding reaches approximately 100–105 tokens per second; with two concurrent requests, aggregate decoding throughput reaches approximately 161 tokens per second. This makes the setup relevant for developers and small teams that want an independently operated local model service without building a large compute cluster.

For personal productivity, both BF16 and FP8 have been adapted to run locally on an Apple Silicon MacBook. FP8 reduces the resources required for local operation and improves sustained generation speed by approximately 30%, supporting smoother multi-turn interaction and agent experiences. The intended use cases include coding assistance, document processing, local knowledge Q&A, and workflow automation, while keeping the model and data on the device.

The same Apple Silicon deployment path extends to Mac mini, where it can operate as a low-power, always-on local intelligence node for knowledge-base retrieval, document analysis, automated tasks, and local model API services in a home, office, or small-team environment.

These examples do not define a universal minimum hardware specification. They show three validated deployment patterns: a compact local service on DGX Spark, a mobile productivity assistant on MacBook, and a continuously available local node on Mac mini.

Infinite Wiki, fully local on macOS

To test what this feels like beyond a benchmark, we recreated the core experience of Infinite Wiki with Ling-3.0-tiny on a 36GB MacBook Pro.

While reading, a user can select any term and generate a concise explainer card based on the current context. Terms inside each card can be explored again, turning a document into a branching path through knowledge.

The entire experience runs locally on macOS. Time to first token (TTFT) stays below 100 milliseconds, keeping repeated clicks and multi-level exploration fluid. Reading history, queries, and generated content can remain on the device. There is no dependency on a cloud model and no per-call API cost.

This demo captures what small agentic models can unlock. Local AI is not only about reproducing a chat interface offline. It can become a responsive layer inside reading, coding, document processing, knowledge retrieval, UI automation, and device workflows.

A Mac mini can extend the same idea into an always-on local node for a home, office, or small team. It can support private search, document analysis, writing assistance, automated tasks, or a local model API without sending every interaction to the cloud.

A local deployment model for agentic AI

Large frontier models will remain essential for the hardest problems. The opportunity for small models is different.

Many real workflows consist of frequent, bounded decisions: route a request, query a local database, call an internal API, inspect a file, update a UI, validate an output, or escalate an uncertain case. Running every step through the largest available model is often unnecessary and economically difficult at scale.

A capable local model changes the architecture of the product. It can handle routine tool calls close to the data, respond with lower latency, continue working without a network connection, and reserve larger remote models for tasks that genuinely require them. For high-volume services, it can also become a cost-efficient execution layer rather than an expensive universal endpoint.

The most useful AI systems may therefore be heterogeneous: small local models for frequent interaction and tool execution, larger models for difficult reasoning, and clear routing between them.

Ling-3.0-tiny is our contribution to that direction.

Where a tiny agent fits—and where it does not

Ling-3.0-tiny is best understood as an efficient execution layer for frequent, bounded work: local retrieval, private document assistance, routine API calls, code and UI automation, or high-volume tasks where latency and cost matter. In these settings, proximity to the data can be as important as maximum model intelligence.

It is not a universal replacement for frontier models. Open-ended research, the hardest reasoning problems, and long-horizon workflows with costly failure modes may still require a larger model, stronger verification, or human approval. A locally generated tool call should not automatically become an unrestricted action simply because it is fast.

This suggests a more practical architecture for agents: use a small model where the task is well-bounded and the feedback loop is clear; route difficult or uncertain cases upward. The value of a tiny model is not that it does everything. It is that it can do far more of the everyday work, closer to where that work happens.

Open weights for real-world intelligence

Ling-3.0-tiny is available under the MIT license, with weights released in BF16, FP8, and INT4. Deployment recipes are available for SGLang and an InclusionAI-maintained vLLM branch, with validated local deployments on NVIDIA DGX Spark, Apple Silicon MacBook, and Mac mini.

We are especially grateful to SGLang and vLLM for their support in bringing the model to developers.

Our next steps include improving long-tail knowledge and factual accuracy, strengthening complex tool use and long-horizon agent stability, extending coding and scientific reasoning, and making FP8 and INT4 deployment easier across more hardware.

The broader goal is simple: make intelligence open, efficient, and deployable—and make reasoning and tool use available wherever real work happens.

Open intelligence for the real world.