ai-sdk-llama-cpp weekly downloads statistics

ai-sdk-llama-cpp

Alpha Software - This package is in early development. The API may change between versions without notice.

macOS Only - This package currently only supports macOS with Apple Silicon or Intel processors.

A minimal llama.cpp provider for the Vercel AI SDK, implementing the LanguageModelV3 interface.

This package loads llama.cpp directly into Node.js memory via native C++ bindings, enabling local LLM inference without requiring an external server.

Features

• Native Performance: Direct C++ bindings using node-addon-api (N-API) • GPU Acceleration: Automatic Metal support on macOS • Streaming & Non-streaming: Full support for both generateText and streamText • Structured Output: Generate JSON objects with schema validation using generateObject • Tool/Function Calling: Support for AI SDK tools with automatic tool call detection • Chat Templates: Automatic or configurable chat template formatting (llama3, chatml, gemma, etc.) • ESM Only: Modern ECMAScript modules, no CommonJS • GGUF Support: Load any GGUF-format model

Prerequisites

Before installing, ensure you have the following:

• macOS (Apple Silicon or Intel) • Node.js >= 18.0.0 • CMake >= 3.15 • Xcode Command Line Tools

Installation

The installation will automatically:

1. Detect macOS and verify platform compatibility 2. Compile llama.cpp as a static library with Metal support 3. Build the native Node.js addon

Note: Installation on Windows or Linux will fail with an error. Only macOS is supported.

Usage

Basic Example

Streaming Example

Structured Output

Generate type-safe JSON objects that conform to a schema using generateObject:

The structured output feature uses GBNF grammar constraints to ensure the model generates valid JSON that conforms to your schema. This works with:

• Primitive types: string, number, integer, boolean, null • Objects: With properties, required, and additionalProperties • Arrays: With items, minItems, maxItems • Enums and constants: enum, const • Composition: oneOf, anyOf, allOf • String constraints: minLength, maxLength, pattern • Number constraints: minimum, maximum (for integers) • String formats: date, time, date-time, uuid • References: Local $ref to $defs/definitions

Tool Calling Example

Use AI SDK tools with local models. The model decides when to call tools based on the conversation context:

Tool calling also works with streamText. When tools are provided, the provider automatically detects tool call JSON output and emits proper tool-call events instead of streaming raw JSON as text.

Note: Tool calling quality depends heavily on the model. Models fine-tuned for function calling (e.g., Llama 3.1+, Hermes 2/3, Functionary, Qwen 2.5) work best. Generic models may produce inconsistent results.

Embedding Example

Configuration Options

Chat Templates

The chatTemplate option controls how messages are formatted before being sent to the model. Available templates include:

• chatml, llama2, llama2-sys, llama3, llama4 • mistral-v1, mistral-v3, mistral-v7 • phi3, phi4, gemma, falcon3, zephyr • deepseek, deepseek2, deepseek3, command-r • And more (see llama.cpp documentation for the full list)

Generation Parameters

The standard AI SDK generation parameters are supported:

Model Downloads

You'll need to download GGUF-format models separately. Popular sources:

• Hugging Face - Search for GGUF models • TheBloke's Models - Popular quantized models

Example download:

API Reference

llamaCpp(config)

Creates a new llama.cpp language model instance.

Parameters:

• config.modelPath (string, required): Path to the GGUF model file • config.contextSize (number, optional): Maximum context size. Default: 2048 • config.gpuLayers (number, optional): GPU layers to offload. Default: 99 • config.threads (number, optional): CPU threads. Default: 4 • config.debug (boolean, optional): Enable verbose llama.cpp output. Default: false • config.chatTemplate (string, optional): Chat template to use for formatting messages. Default: "auto"

Returns: LlamaCppLanguageModel - A language model compatible with the Vercel AI SDK

LlamaCppLanguageModel

Implements the LanguageModelV3 interface from @ai-sdk/provider.

Methods:

• doGenerate(options): Non-streaming text generation • doStream(options): Streaming text generation • dispose(): Unload the model and free GPU/CPU resources. Always call this when done to prevent memory leaks, especially when loading multiple models

Limitations

This is a minimal implementation with the following limitations:

• macOS only: Windows and Linux are not supported • No image inputs: Only text prompts are supported

Contributing

See CONTRIBUTING.md for development setup and contribution guidelines.

License

MIT

Acknowledgments

• llama.cpp - The underlying inference engine • Vercel AI SDK - The AI SDK framework • node-addon-api - N-API C++ wrapper

Description from the npm listing.