databricks-agent-skills

Databricks AI Tools

Build on Databricks with AI coding agents such as Claude Code, Cursor, Codex, GitHub Copilot, and many others.

This repository provides the skills and agent plugins for Databricks AI Tools. Skills give your agent Databricks-specific guidance for building apps, jobs, and data pipelines, working with Lakebase and Model Serving, and deploying resources with Declarative Automation Bundles. Plugins add hooks and commands where supported.

See the Databricks AI Tools documentation for setup and guidance on using AI Tools alongside managed MCP servers.

Installation

Two install paths cover the stable skills. They install to different places but end up loaded by the same agents — pick whichever fits your workflow.

• Databricks CLI writes SKILL.md files directly into each agent's skill directory (~/.claude/skills/, ~/.cursor/extensions/<...>, etc.).
• Plugin marketplaces (Claude Code, Cursor) cache the plugin under the agent's plugin directory (e.g. ~/.claude/plugins/cache/databricks-agent-skills/); the agent discovers skills from there.

Via the Databricks CLI (canonical; supports experimental skills):

databricks aitools install

The CLI auto-detects your coding agent(s) and installs the stable skills into each agent's own skills directory. For example, Claude Code skills are installed in ~/.claude/skills/. Run databricks aitools --help for the current list of supported agents.

For finer control, use the aitools skills install subcommand directly — it accepts a positional skill name and an --experimental flag (see the Experimental Skills section).

Via the Claude Code plugin marketplace (stable skills only — installs every skill under ./skills/):

/plugin marketplace add databricks/databricks-agent-skills
/plugin install databricks@databricks-agent-skills

Via the Cursor plugin marketplace:

/add-plugin databricks

The Cursor plugin ships the skills plus the databricks-setup / databricks-doctor commands, two of the three hooks (session context, auth-failure hints), and a routing rule that steers Databricks prompts into the skills; see Commands and hooks.

Via the GitHub Copilot plugin marketplace:

copilot plugin marketplace add databricks/databricks-agent-skills
copilot plugin install databricks@databricks-agent-skills

Works in Copilot CLI (plugins are GA there) and VS Code (agent plugins, preview; also installable from the Extensions view). Ships the skills plus two hooks: the session context primer and the auth-failure hinter (both run on Copilot CLI and the cloud agent; VS Code has its own hooks system). The Copilot cloud agent on github.com takes no plugins; for that surface, vendor the skills into the target repo (.github/skills/) and the auth-hint hook into .github/hooks/.

Via the Codex plugin marketplace:

codex plugin marketplace add databricks/databricks-agent-skills
codex plugin add databricks

The Codex plugin ships the skills plus all three hooks (prompt routing, session context, auth-failure hints). Codex hash-pins plugin hooks: run /hooks once after install (and after each update) to review and enable them. Codex has no distributable slash commands, so the setup/doctor workflows are reachable through the skills there.

CLI vs plugin marketplace

	CLI	Plugin marketplace
Stable skills	✅ (default)	✅
Experimental skills	✅ (with --experimental or by name)	❌
Per-skill selection	✅ (databricks aitools install <name>)	❌ (all-or-nothing)
Commands & hooks	❌ (skills only today, see below)	✅
Updates	databricks aitools update	Plugin marketplace update flow
Required outside the agent	Databricks CLI v1.0.0+	None

If in doubt, use the CLI — it's the canonical install path and the only one that exposes experimental skills.

Available Skills

The MLflow workflows are adapted from mlflow/skills@6a505d8. Use databricks-mlflow-evaluation for evaluation workflows and databricks-mlflow-tracing for tracing and observability workflows.

Stable skills shipped from skills/:

• databricks-core — CLI, authentication, profile selection, data exploration. Parent skill for all product skills.
• databricks-data-discovery — Find, explore, and query data via Genie One (natural-language data Q&A and SQL generation); falls back to manual information_schema exploration.
• databricks-apps — Build full-stack TypeScript apps on Databricks using AppKit.
• databricks-app-design — Design the UX of data apps: dashboards, KPI pages, reports, charts, and Genie/chat surfaces, mapped to AppKit components.
• databricks-dabs — Declarative Automation Bundles (formerly Asset Bundles) for deploying and managing Databricks resources.
• databricks-jobs — Lakeflow Jobs orchestration: task types, triggers, schedules, notifications.
• databricks-lakebase — Lakebase Postgres: projects, branching, autoscaling, synced tables, Data API.
• databricks-model-serving — Model Serving endpoint management, AI Gateway, traffic config.
• databricks-pipelines — Lakeflow Spark Declarative Pipelines (formerly DLT) for batch and streaming.
• databricks-serverless-migration — Migrate classic-compute workloads to serverless compute.
• databricks-vector-search — AI Search (formerly Vector Search) endpoints + indexes for RAG and semantic search.
• databricks-ai-functions — Built-in AI Functions (ai_query, ai_classify, ai_extract, ai_parse_document, ai_forecast) in SQL and PySpark.
• databricks-ai-runtime — AI Runtime (air) CLI for serverless GPU training workloads and custom Docker images.
• databricks-aibi-dashboards — AI/BI dashboards with a SQL-validation workflow.
• databricks-apps-python — Python data apps (Streamlit, Dash, Gradio, Flask, FastAPI, Reflex); prefers AppKit for new apps.
• databricks-dbsql — Databricks SQL warehouse patterns.
• databricks-docs — Databricks documentation lookup via llms.txt.
• databricks-execution-compute — Execute code on Databricks compute.
• databricks-iceberg — Apache Iceberg tables (Managed/Foreign), UniForm, Iceberg REST Catalog, client interoperability.
• databricks-lakeflow-connect — Lakeflow Connect managed ingestion from databases (CDC, gateway, query-based).
• databricks-lakeflow-connect-saas — Lakeflow Connect managed ingestion from SaaS applications, files, Kafka and RabbitMQ.
• databricks-metric-views — Unity Catalog Metric Views for governed metrics.
• databricks-ml-training — ML model training on Databricks.
• databricks-mlflow-evaluation — MLflow agent evaluation, scorers, and quality improvement.
• databricks-mlflow-tracing — MLflow tracing, trace analysis, and observability.
• databricks-python-sdk — Python SDK, Databricks Connect, CLI, REST API.
• databricks-sandbox — Databricks Sandbox: serverless sandboxes for untrusted/agent code; lifecycle, command execution, and files via the Databricks SDK (Python, Java, Go, JavaScript).
• databricks-spark-structured-streaming — Spark Structured Streaming patterns.
• databricks-synthetic-data-gen — Realistic synthetic / test data with Faker.
• databricks-unity-catalog — Unity Catalog system tables for lineage, audit, and billing.
• databricks-unity-gateway — Governed AI model, MCP, and model provider services in Unity Catalog.
• databricks-unstructured-pdf-generation — Generate synthetic PDFs for RAG.
• databricks-zerobus-ingest — Zerobus streaming ingest patterns.

Experimental Skills

The experimental/ directory contains additional skills originally imported from databricks-solutions/ai-dev-kit (now deprecated — this repo is the source of truth going forward) on a best-effort basis.

• Experimental skills are not officially supported — they may be used, but do not follow the same review / quality bar as the stable skills under skills/.
• They are not installed by default by databricks aitools install. Pass --experimental to install all of them, or install a specific one by name (with the --experimental flag — e.g. databricks aitools install spark-python-data-source --experimental).
• See experimental/README.md for the full list and caveats.

Commands and hooks (Claude Code, Cursor)

When installed as a Claude Code plugin, the databricks plugin adds slash commands and three hooks (prompt routing, session context, auth-failure hints) on top of the skills. The Cursor plugin (databricks) ships the same commands and two of the hooks; see the Cursor note below. (These ship via the plugin marketplaces; the CLI databricks aitools install path installs skills only today; see the note at the end.)

Slash commands: friction-only entry points; everyday work stays with the auto-invoked skills.

• /databricks:setup [workspace-url]: auth/onboarding. Install check, then an OAuth / PAT / service-principal profile, then verify.
• /databricks:doctor [profile]: read-only health check (CLI version, auth, workspace reachability, compute, recent job failures).

(Product workflows such as apps, jobs, pipelines, DABs, etc. are handled by the skills, not commands, so they aren't duplicated here.)

Hooks (hooks/, all fail-open):

• Prompt router (UserPromptSubmit): a fast keyword regex (sub-50ms, no LLM, no network) over each prompt. When the prompt is Databricks-related, it injects a note steering Claude to load databricks-core plus the matching product skill before answering. The full note fires once per session; later Databricks prompts get a one-line reminder. Unrelated prompts are untouched. No permission gating, no cost warnings.
• Context primer (SessionStart, skipped on resume): injects the routing rule, CLI version, configured profile names and any [__settings__].default_profile (read locally, no network call, no token values), and env/in-platform auth state.
• Auth-failure hint (PostToolUse on Bash): when a databricks command fails with an auth-shaped error, adds one line suggesting /databricks:doctor or databricks auth login before retrying. Never blocks or rewrites commands.

Cursor. Cursor has a flat / menu (no plugin:command namespacing), so the same commands ship as /databricks-setup and /databricks-doctor, rendered from the one templated commands/ source into the Cursor bundle folder. The Cursor-dialect hook wiring ships as the Cursor bundle folder's hooks/hooks.json (auto-discovered from the plugin root, no declaration): the context primer (sessionStart) and the auth-failure hint (postToolUse), both invoked with --platform cursor so they emit Cursor's output shape and reference the Cursor command names. The prompt-router hook does not port (Cursor's beforeSubmitPrompt cannot inject context), so routing instead ships as a Cursor rule (rules/databricks-routing.mdc) that injects the routing table when a prompt is Databricks-related, independent of an open Cursor bug that currently drops hook additional_context. Native skill selection also helps.

Distribution parity (follow-up). Each plugin bundle includes its provider's supported components. databricks aitools install currently packages only skills/, so CLI-install users don't yet get commands/hooks. Closing that gap is tracked as CLI-side work.

Structure

Each skill follows the Agent Skills Specification:

skill-name/
├── SKILL.md           # Main skill file with frontmatter + instructions
└── references/        # Additional documentation loaded on demand

Development

Adding New Skills

For a narrower variation of an existing skill, create a subskill that declares its parent via frontmatter. This is how the stable skills are organized today — each product skill sets parent: databricks-core.

---
name: "databricks-apps-chatbots"
description: "Databricks apps with chatbot features"
parent: databricks-apps
---

# Chatbot Apps

**FIRST**: Use the parent `databricks-apps` skill for app development basics.

Then apply these patterns:
- Pattern 1
- Pattern 2

This approach:

• Keeps the main skill stable and focused
• Allows experimentation without modifying core skills
• Makes it easy to follow the changes in the main skill

Manifest Management

manifest.json is generated by scripts/skills.py from the skill directories and frontmatter. Do not edit it by hand. CI rejects manual changes via two checks: content drift (parsed dict doesn't match what generate would produce) and canonical form (on-disk bytes don't match json.dumps(..., indent=2, sort_keys=True)).

Because the CLI reads manifest.json and plugin installs read the generated catalogs and provider bundle live from main, main must stay a consistent index of its own source between releases. .github/workflows/self-heal-manifest.yml enforces this: on every push to main that touches generation inputs it re-runs scripts/skills.py generate and commits the refreshed manifest.json, plugin catalogs, hook/routing files, and provider bundle back to main, so a skill added, deleted, or reshaped in one PR can't leave installs fetching missing files. The per-PR committed-manifest check is therefore advisory (it surfaces staleness without blocking PRs that merely inherited it from main).

Sync assets and regenerate the manifest after adding or updating skills:

python3 scripts/skills.py

Validate that assets and manifest are up to date (used by CI):

python3 scripts/skills.py validate

To check only that a committed manifest points at files that exist in the repo:

python3 scripts/skills.py validate-committed-manifest

The manifest is consumed by the CLI to discover available skills.

Plugin manifest management

The repo ships one plugin to five targets (Claude Code, Codex, Copilot, Cursor, Antigravity), with a separate bundle at plugins/databricks/<provider>/ for each target. Four marketplace.json catalogs at the repo root each point a scoped source at their provider's bundle (tracking main). The Antigravity provider is available for local testing; runtime validation is pending. The bundles, the catalogs, the five plugin.json files, and manifest.json are all generated from a single source of truth, metaplugin/plugin.meta.json, by scripts/skills.py. Do not hand-edit the generated files (generated manifest subdirectories also carry a README.md saying so, and plugins/** is marked linguist-generated) — edit metaplugin/plugin.meta.json and regenerate:

python3 scripts/skills.py generate   # regenerates manifest.json + all plugin manifests
python3 scripts/skills.py validate   # CI check: fails on any drift

metaplugin/version.meta.json owns the release version, propagated to all five targets. metaplugin/plugin.meta.json owns the name, description, keywords, author/license, per-target display names and hook/command/rule wiring, and the skill-to-keyword map. Adding a stable skill means adding it to the skills map there (with a keyword); CI fails if a shipped skill has no entry. The same source also drives prompt routing: its routing block is rendered into the prompt router's data (hooks/_routing_data.json), the Cursor routing rule (rules/databricks-routing.mdc), and the Antigravity routing rule (plugins/databricks/antigravity/rules/databricks-routing.md) from one table, so they can't drift. See CONTRIBUTING.md for the full field reference.

Security

Please see SECURITY for vulnerability reporting guidelines.

Integrity

Release tags are created by the Release workflow and map 1:1 to a published version.

Contributing

We welcome feedback and change proposals through GitHub issues. This repository is published from an internal Databricks source of truth. Pull request creation is restricted to repository collaborators and is used only to publish internal changes. If a proposal is accepted, a Databricks maintainer will implement it internally and publish the change here. See CONTRIBUTING.md for details.

• All changes require approval from a code owner (see CODEOWNERS).
• Documentation examples must follow least-privilege defaults — avoid suggesting elevated permissions or broad scopes unless explicitly necessary.