Skill Lab — the evaluation layer for agent skills

Evaluate Skills#

GET/v1/repos/{owner}/{repo}/evaluate

Evaluate skills in a GitHub repository. Omit path to scan the entire repo and evaluate every SKILL.md found. Provide path to evaluate a single skill at that location.

Path Parameters#

Parameter	Type	Description
owner	string	GitHub username or organization
repo	string	Repository name

Query Parameters#

Parameter	Type	Description
path	string	Optional. Path to a specific skill — omit to evaluate all skills in the repo

GitHub URL segments are stripped automatically. You can paste paths directly from GitHub's UI — segments like tree/main/ or blob/main/ are removed before lookup. Both skills/my-skill and skills/my-skill/SKILL.md work.

Results are cached by commit SHA for 1 hour. If the repo's HEAD hasn't changed, the response is served from cache.

Response#

The response shape depends on whether path is provided.

Without path — returns a ScanResult object with a skills array containing one SkillResult per SKILL.md found (up to 50).
With path — returns a SkillEvaluateResult object: the skill's fields flattened at the top level alongside owner, repo, commit_sha, and scanned_at. No skills array, no truncated field.

Response Schemas#

ScanResult#

The root response object returned by the evaluate endpoint. Contains repository metadata and an array of evaluated skills.

Field	Type	Description
owner	string	GitHub username or organization
repo	string	Repository name
commit_sha	string	Full SHA of the commit that was scanned
skills	SkillResult[]	Array of evaluated skills found in the repository
scanned_at	string	ISO 8601 timestamp of when the scan was performed
error	string \| null	Top-level error message if the scan itself failed, otherwise null
truncated	boolean	True if the repository contained more than 50 skills and results were capped

Example

response
{
  "owner": "anthropics",
  "repo": "claude-code",
  "commit_sha": "a04e4f9...",
  "skills": [
    {
      "path": "skills/git-commit/SKILL.md",
      "name": "git-commit",
      "quality_score": 85.0,
      "overall_pass": true,
      "checks_run": 37,
      "checks_passed": 30,
      "checks_failed": 3,
      "results": [...]
    }
  ],
  "scanned_at": "2025-03-24T12:00:00+00:00",
  "error": null,
  "truncated": false
}

SkillEvaluateResult#

The response object returned by GET /v1/repos/{owner}/{repo}/evaluate?path=.... A single skill's evaluation fields hoisted to the top level alongside repository metadata. No skills array, no truncated field.

Field	Type	Description
owner	string	GitHub username or organization
repo	string	Repository name
commit_sha	string	Full SHA of the commit that was evaluated
scanned_at	string	ISO 8601 timestamp of when the evaluation was performed
path	string	Path to the SKILL.md file within the repository
name	string \| null	Skill name extracted from frontmatter
description	string \| null	Skill description extracted from frontmatter
quality_score	number	Overall quality score from 0 to 100
overall_pass	boolean	Whether all high-severity checks passed
checks_run	number	Total number of checks executed
checks_passed	number	Number of checks that passed
checks_failed	number	Number of checks that failed
results	CheckResult[]	Detailed results for each individual check
summary	object	Aggregated pass/fail counts grouped by severity and dimension
error	string \| null	Error message if the skill failed to evaluate, otherwise null

Example

response
{
  "owner": "anthropics",
  "repo": "claude-code",
  "commit_sha": "a04e4f9...",
  "scanned_at": "2025-03-24T12:00:00+00:00",
  "path": "skills/git-commit/SKILL.md",
  "name": "git-commit",
  "description": "Write conventional commit messages",
  "quality_score": 85.0,
  "overall_pass": true,
  "checks_run": 37,
  "checks_passed": 30,
  "checks_failed": 3,
  "results": [...],
  "summary": {...},
  "error": null
}

SkillResult#

Represents the evaluation of a single SKILL.md file. Contains the quality score, individual check results, and aggregated summary.

Field	Type	Description
path	string	Path to the SKILL.md file within the repository
name	string \| null	Skill name extracted from frontmatter
description	string \| null	Skill description extracted from frontmatter
quality_score	number	Overall quality score from 0 to 100
overall_pass	boolean	Whether all high-severity checks passed
checks_run	number	Total number of checks executed
checks_passed	number	Number of checks that passed
checks_failed	number	Number of checks that failed
results	CheckResult[]	Detailed results for each individual check
summary	object	Aggregated pass/fail counts grouped by severity and dimension
error	string \| null	Error message if this skill failed to evaluate, otherwise null

Summary Object

The summary field contains aggregated pass/fail counts, grouped two ways.

summary shape
{
  "by_severity": {
    "high":   { "passed": 5, "failed": 0 },
    "medium": { "passed": 12, "failed": 2 },
    "low":    { "passed": 13, "failed": 1 }
  },
  "by_dimension": {
    "structure":   { "passed": 10, "failed": 1 },
    "naming":      { "passed": 3,  "failed": 0 },
    "description": { "passed": 2,  "failed": 1 },
    "content":     { "passed": 10, "failed": 1 },
    "security":    { "passed": 5,  "failed": 0 }
  }
}

CheckResult#

An individual check result from the evaluation. There are 37 checks across 5 dimensions, each with a severity level.

Field	Type	Description
check_id	string	Unique identifier for the check
check_name	string	Human-readable check name
passed	boolean	Whether this check passed
severity	"high" \| "medium" \| "low"	Severity level of the check
dimension	string	Quality dimension: "structure", "naming", "description", "content", or "security"
message	string	Human-readable result message
fix	string?	Suggested fix when the check fails
details	object?	Additional check-specific data
location	string?	File location relevant to the check

Dimensions

Dimension	Description
structure	SKILL.md existence, valid frontmatter, file organization
naming	Skill name format, kebab-case, directory match
description	Presence, length, actionable phrasing
content	Body quality, examples, token budget, references
security	Prompt injection, YAML anomalies, unicode obfuscation

Example

single check result
{
  "check_id": "naming_format",
  "check_name": "Name Format",
  "passed": false,
  "severity": "high",
  "dimension": "naming",
  "message": "Skill name 'MySkill' is not in kebab-case",
  "fix": "Rename to 'my-skill'"
}

Errors#

Status	Cause
404	Repository or skill path not found
429	GitHub API rate limit reached

Examples#

Whole repo:

bash

curl "https://api.skill-lab.dev/v1/repos/anthropics/claude-code/evaluate"

javascript

const res = await fetch("https://api.skill-lab.dev/v1/repos/anthropics/claude-code/evaluate");
const data = await res.json();
// data.skills[0].quality_score

python

import httpx

res = httpx.get("https://api.skill-lab.dev/v1/repos/anthropics/claude-code/evaluate")
data = res.json()
# data['skills'][0]['quality_score']

Single skill:

bash

curl "https://api.skill-lab.dev/v1/repos/owner/repo/evaluate?path=my-skill"

javascript

const res = await fetch("https://api.skill-lab.dev/v1/repos/owner/repo/evaluate?path=my-skill");
const data = await res.json();
// data.quality_score

python

import httpx

res = httpx.get("https://api.skill-lab.dev/v1/repos/owner/repo/evaluate?path=my-skill")
data = res.json()
# data['quality_score']

GitHub-style path (auto-stripped):

bash

# tree/main/ is stripped automatically
curl "https://api.skill-lab.dev/v1/repos/owner/repo/evaluate?path=tree/main/skills/my-skill"

Skill InfoMetadata and token estimates without scoring OptimizeLLM-powered rewrite with before/after scores ExportRender skills as XML, Markdown, or JSON prompts sklab evaluateRun the same evaluation locally against a skill directory

PreviousRate Limits NextCheck Registry