mcp-evals weekly downloads statistics

MCP Evals

A Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring, with built-in observability support. This helps ensure your MCP server's tools are working correctly, performing well, and are fully observable with integrated monitoring and metrics.

Installation

As a Node.js Package

As a GitHub Action

Add the following to your workflow file:

Usage -- Evals

1. Create Your Evaluation File

You can create evaluation configurations in either TypeScript or YAML format.

Option A: TypeScript Configuration

Create a file (e.g., evals.ts) that exports your evaluation configuration:

Option B: YAML Configuration

For simpler configuration, you can use YAML format (e.g., evals.yaml):

2. Run the Evaluations

As a Node.js Package

You can run the evaluations using the CLI with either TypeScript or YAML files:

As a GitHub Action

The action will automatically: 1. Run your evaluations 2. Post the results as a comment on the PR 3. Update the comment if the PR is updated

Evaluation Results

Each evaluation returns an object with the following structure:

Configuration

Environment Variables

• OPENAI_API_KEY: Your OpenAI API key (required for OpenAI models) • ANTHROPIC_API_KEY: Your Anthropic API key (required for Anthropic models)

[!NOTE] If you're using this GitHub Action with open source software, enable data sharing in the OpenAI billing dashboard to claim 2.5 million free GPT-4o mini tokens per day, making this Action effectively free to use.

Evaluation Configuration

TypeScript Configuration

The EvalConfig interface requires:

• model: The language model to use for evaluation (e.g., GPT-4) • evals: Array of evaluation functions to run

Each evaluation function must implement:

• name: Name of the evaluation • description: Description of what the evaluation tests • run: Async function that takes a model and returns an EvalResult

YAML Configuration

YAML configuration files support:

Model Configuration: • provider: Either 'openai' or 'anthropic' • name: Model name (e.g., 'gpt-4o', 'claude-3-opus-20240229') • api_key: Optional API key (uses environment variables by default)

Evaluation Configuration: • name: Name of the evaluation (required) • description: Description of what the evaluation tests (required) • prompt: The prompt to send to your MCP server (required) • expected_result: Optional description of expected behavior

Supported File Extensions: .yaml, .yml

Usage -- Monitoring

Note: The metrics functionality is still in alpha. Features and APIs may change, and breaking changes are possible.

1. Add the following to your application before you initilize the MCP server.

2. Start the monitoring stack:

3. Run your MCP server and it will automatically connect to the monitoring stack.

Accessing the Dashboards

• Prometheus: http://localhost:9090 • Grafana: http://localhost:3000 (username: admin, password: admin) • Jaeger UI: http://localhost:16686

Metrics Available

• Tool Calls: Number of tool calls by tool name • Tool Errors: Number of errors by tool name • Tool Latency: Distribution of latency times by tool name

License

MIT

Description from the npm listing.