From 9f3f1f01372cee9c8d41f06f6ffd9e91dfad97a8 Mon Sep 17 00:00:00 2001 From: Haiyuan Cao Date: Sun, 7 Dec 2025 01:11:17 -0800 Subject: [PATCH 1/6] feat: add BigQuery Skills Demo with dynamic skill discovery MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a sample agent demonstrating Anthropic's Agent Skills Pattern for dynamic capability discovery with BigQuery ML and AI functions. Key features: - Dynamic skill discovery from SKILL.md files at runtime - Progressive disclosure: skill summaries in prompt, full content on-demand - load_skill tool for agents to request detailed skill documentation - BQML skill: ML model training, evaluation, and prediction in SQL - BQ AI Operator skill: AI.CLASSIFY, AI.IF, AI.SCORE managed functions The demo is self-contained and uses existing BigQuery tools from ADK. Skills are stored as markdown files with YAML frontmatter for metadata. Reference: https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 --- .../samples/bigquery_skills_demo/README.md | 132 ++++++++ .../samples/bigquery_skills_demo/__init__.py | 15 + .../samples/bigquery_skills_demo/agent.py | 175 +++++++++++ .../bigquery_skills_demo/skill_registry.py | 284 ++++++++++++++++++ .../skills/bq_ai_operator/SKILL.md | 232 ++++++++++++++ .../bigquery_skills_demo/skills/bqml/SKILL.md | 158 ++++++++++ 6 files changed, 996 insertions(+) create mode 100644 contributing/samples/bigquery_skills_demo/README.md create mode 100644 contributing/samples/bigquery_skills_demo/__init__.py create mode 100644 contributing/samples/bigquery_skills_demo/agent.py create mode 100644 contributing/samples/bigquery_skills_demo/skill_registry.py create mode 100644 contributing/samples/bigquery_skills_demo/skills/bq_ai_operator/SKILL.md create mode 100644 contributing/samples/bigquery_skills_demo/skills/bqml/SKILL.md diff --git a/contributing/samples/bigquery_skills_demo/README.md b/contributing/samples/bigquery_skills_demo/README.md new file mode 100644 index 0000000000..a2f40cc358 --- /dev/null +++ b/contributing/samples/bigquery_skills_demo/README.md @@ -0,0 +1,132 @@ +# BigQuery Skills Demo + +This sample demonstrates Anthropic's [Agent Skills Pattern](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills) for dynamic skill discovery with BigQuery ML and AI capabilities. + +## Overview + +This demo showcases: +- **Dynamic Skill Discovery**: Skills are discovered at runtime from SKILL.md files +- **Progressive Disclosure**: Only skill names/descriptions loaded initially; full content on-demand +- **load_skill Tool**: Agent loads full skill documentation when relevant to the task + +### Available Skills + +1. **bqml** - BigQuery ML for training and deploying ML models in SQL + - Model training (LINEAR_REG, LOGISTIC_REG, KMEANS, ARIMA_PLUS, XGBoost, etc.) + - Model evaluation and prediction + - Feature importance and model analysis + +2. **bq_ai_operator** - Managed AI functions in BigQuery SQL + - AI.CLASSIFY: Categorize text into classes + - AI.IF: Natural language TRUE/FALSE filtering + - AI.SCORE: Rate/rank content by criteria (0.0 to 1.0) + +## Prerequisites + +1. Google Cloud project with BigQuery and Vertex AI enabled +2. Application Default Credentials configured: + ```bash + gcloud auth application-default login + ``` +3. Set your project ID: + ```bash + export GOOGLE_CLOUD_PROJECT=your-project-id + ``` + +### For AI Functions (bq_ai_operator skill) + +Create a BigQuery connection to Vertex AI: +```bash +bq mk --connection \ + --location=us \ + --project_id=$GOOGLE_CLOUD_PROJECT \ + --connection_type=CLOUD_RESOURCE \ + my_ai_connection +``` + +Grant the connection's service account access to Vertex AI: +```bash +# Get the service account +bq show --connection $GOOGLE_CLOUD_PROJECT.us.my_ai_connection + +# Grant access (replace with actual service account) +gcloud projects add-iam-policy-binding $GOOGLE_CLOUD_PROJECT \ + --member="serviceAccount:SERVICE_ACCOUNT_EMAIL" \ + --role="roles/aiplatform.user" +``` + +## Running the Demo + +### Option 1: Run with ADK CLI + +```bash +cd contributing/samples/bigquery_skills_demo +adk run . +``` + +### Option 2: Run the web UI + +```bash +adk web contributing/samples --port 8000 +# Open http://127.0.0.1:8000/dev-ui/?app=bigquery_skills_demo +``` + +## Example Prompts + +### BQML Skill +``` +Train a linear regression model to predict penguin body weight using +the public penguins dataset, then evaluate it and show feature importance. +``` + +### BQ AI Operator Skill +``` +Classify 5 BBC news articles by their topic using AI.CLASSIFY with +categories: tech, sport, business, politics, entertainment, other. +``` + +## How It Works + +1. **Skill Discovery**: The `SkillRegistry` scans the `skills/` directory for SKILL.md files +2. **YAML Frontmatter**: Each SKILL.md has metadata (name, description) in YAML frontmatter +3. **Progressive Loading**: + - Level 1: Agent sees skill names and descriptions in its system prompt + - Level 2: Agent calls `load_skill(skill_name)` to get full documentation +4. **On-Demand Loading**: Full skill content is only loaded when relevant to the task + +## Code Structure + +``` +bigquery_skills_demo/ +├── __init__.py # Module init +├── agent.py # Agent with BigQuery tools and load_skill +├── skill_registry.py # Dynamic skill discovery (Anthropic pattern) +├── skills/ +│ ├── bqml/ +│ │ └── SKILL.md # BQML skill documentation +│ └── bq_ai_operator/ +│ └── SKILL.md # AI operator skill documentation +└── README.md # This file +``` + +## Adding New Skills + +1. Create a directory under `skills/` (e.g., `skills/my_skill/`) +2. Add a `SKILL.md` file with YAML frontmatter: + ```markdown + --- + name: my_skill + description: Short description of what this skill does + --- + + # My Skill Documentation + + Detailed instructions, examples, and usage patterns... + ``` +3. The skill will be automatically discovered on agent startup + +## References + +- [Anthropic: Equipping Agents with Skills](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills) +- [BigQuery ML Documentation](https://cloud.google.com/bigquery/docs/bqml-introduction) +- [BigQuery AI Functions](https://cloud.google.com/bigquery/docs/ai-functions) diff --git a/contributing/samples/bigquery_skills_demo/__init__.py b/contributing/samples/bigquery_skills_demo/__init__.py new file mode 100644 index 0000000000..c48963cdc7 --- /dev/null +++ b/contributing/samples/bigquery_skills_demo/__init__.py @@ -0,0 +1,15 @@ +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from . import agent diff --git a/contributing/samples/bigquery_skills_demo/agent.py b/contributing/samples/bigquery_skills_demo/agent.py new file mode 100644 index 0000000000..88e465edca --- /dev/null +++ b/contributing/samples/bigquery_skills_demo/agent.py @@ -0,0 +1,175 @@ +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""BigQuery Skills Demo Agent with Dynamic Skill Discovery. + +This agent demonstrates the Anthropic Skills Pattern for dynamic capability +discovery, as described in: +https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills + +Key Features: +1. **Progressive Disclosure**: Skills are discovered at startup with only + names and descriptions loaded. Full content is loaded on-demand. + +2. **Dynamic Discovery**: Skills are stored as SKILL.md files and discovered + automatically from the skills/ directory. + +3. **load_skill Tool**: The agent can load full skill documentation when + it determines a skill is relevant to the current task. + +Available Skills: +- bqml: BigQuery ML for training/deploying ML models in SQL +- bq_ai_operator: Generative AI functions in SQL + +To run this demo: + cd contributing/samples/bigquery_skills_demo + adk run . + +Or via web UI: + adk web contributing/samples --port 8000 + # Then open http://127.0.0.1:8000/dev-ui/?app=bigquery_skills_demo +""" + +import os + +# Set environment variables for Vertex AI (uses ADC for authentication) +# Users should set GOOGLE_CLOUD_PROJECT to their own project ID +os.environ.setdefault("GOOGLE_GENAI_USE_VERTEXAI", "true") +os.environ.setdefault("GOOGLE_CLOUD_LOCATION", "us-central1") + +from google.adk.agents.llm_agent import LlmAgent +from google.adk.tools import FunctionTool +from google.adk.tools.bigquery import BigQueryCredentialsConfig +from google.adk.tools.bigquery import BigQueryToolset +from google.adk.tools.bigquery.config import BigQueryToolConfig +from google.adk.tools.bigquery.config import WriteMode +import google.auth + +# Import the dynamic skill registry +from .skill_registry import SkillRegistry, load_skill + +# Agent name +AGENT_NAME = "bigquery_skills_demo_agent" + +# Project configuration - must be set via environment variable +PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT") +if not PROJECT_ID: + raise ValueError( + "GOOGLE_CLOUD_PROJECT environment variable must be set. " + "Set it to your GCP project ID before running this demo." + ) + +# Initialize BigQuery tool config +# Using ALLOWED write mode to enable CREATE MODEL operations +tool_config = BigQueryToolConfig( + write_mode=WriteMode.ALLOWED, + application_name=AGENT_NAME, +) + +# Use application default credentials +application_default_credentials, _ = google.auth.default() +credentials_config = BigQueryCredentialsConfig( + credentials=application_default_credentials +) + +# Initialize BigQuery toolset +bigquery_toolset = BigQueryToolset( + credentials_config=credentials_config, + bigquery_tool_config=tool_config, +) + +# Initialize dynamic skill registry +skill_registry = SkillRegistry() +SKILLS_SUMMARY = skill_registry.get_skills_summary() + +# Create load_skill tool for the agent +load_skill_tool = FunctionTool(load_skill) + +# Create the root agent with BigQuery tools and dynamic skill loading +root_agent = LlmAgent( + model="gemini-2.5-pro", + name=AGENT_NAME, + description=( + "Data science agent with BigQuery ML and AI capabilities. " + "Uses dynamic skill discovery to load relevant capabilities on-demand." + ), + instruction=f"""\ +You are a data science agent with BigQuery capabilities and dynamic skill loading. + +## How Skills Work (Anthropic Pattern) + +You have access to specialized skills that provide detailed guidance for complex tasks. +Skills are loaded on-demand to keep context focused and efficient. + +**Current Available Skills:** + +{SKILLS_SUMMARY} + +**When to use load_skill:** +1. When the user asks about ML model training, prediction, or evaluation → load "bqml" +2. When the user asks about AI/text analysis, classification, or generation → load "bq_ai_operator" +3. Load skills BEFORE attempting complex operations to get proper syntax and examples + +**Progressive Disclosure:** +- You see skill names and descriptions above (Level 1) +- Call `load_skill(skill_name)` to get full documentation with examples (Level 2) +- Only load skills when they're relevant to the current task + +## Available BigQuery Tools + +- `execute_sql`: Run any BigQuery SQL (queries, DDL, BQML, AI functions) +- `get_table_info`: Get schema information for a table +- `list_dataset_ids`: List datasets in a project +- `list_table_ids`: List tables in a dataset +- `load_skill`: Load full documentation for a skill + +## Project Configuration + +- Project ID: {PROJECT_ID} +- Available public datasets: `bigquery-public-data.ml_datasets` (penguins, census, etc.) + +## Workflow Example + +1. User asks: "Train a model to predict penguin weight" +2. You call: `load_skill("bqml")` to get BQML documentation +3. You follow the skill's examples to CREATE MODEL, EVALUATE, and PREDICT +4. You explain results to the user + +## Guidelines + +1. **Load skills first**: Before complex ML or AI operations, load the relevant skill +2. **Explore data first**: Use `get_table_info` or `SELECT * LIMIT 5` before complex queries +3. **Use LIMIT**: Prevent large result sets with `LIMIT 10-100` +4. **Explain your steps**: Describe what each query does and interpret results + +## Quick Reference (without loading skills) + +**BQML Quick Start:** +```sql +-- Train: CREATE OR REPLACE MODEL `project.dataset.model` OPTIONS(model_type='LINEAR_REG', ...) +-- Evaluate: SELECT * FROM ML.EVALUATE(MODEL `project.dataset.model`) +-- Predict: SELECT * FROM ML.PREDICT(MODEL `project.dataset.model`, ...) +``` + +**AI Operator Quick Start:** +```sql +-- Classify: AI.CLASSIFY(MODEL `...`, text, ['class1', 'class2']) +-- Generate: AI.GENERATE(MODEL `...`, 'prompt') +-- Extract: AI.EXTRACT(MODEL `...`, text, STRUCT(...)) +``` + +For detailed syntax and examples, use `load_skill("bqml")` or `load_skill("bq_ai_operator")`. +""", + tools=[bigquery_toolset, load_skill_tool], +) diff --git a/contributing/samples/bigquery_skills_demo/skill_registry.py b/contributing/samples/bigquery_skills_demo/skill_registry.py new file mode 100644 index 0000000000..c82cacff72 --- /dev/null +++ b/contributing/samples/bigquery_skills_demo/skill_registry.py @@ -0,0 +1,284 @@ +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Dynamic Skill Registry following Anthropic's Skills Pattern. + +This module implements the dynamic skill discovery pattern from: +https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills + +Key concepts: +1. Skills are stored as SKILL.md files with YAML frontmatter +2. At startup, only skill names and descriptions are loaded (progressive disclosure) +3. Agent loads full skill content on-demand via load_skill tool +4. This reduces context usage while providing rich capabilities when needed +""" + +from __future__ import annotations + +import os +import re +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import yaml + + +@dataclass +class SkillMetadata: + """Metadata for a skill (name + description only).""" + name: str + description: str + path: Path + + +@dataclass +class SkillContent: + """Full content of a skill including documentation.""" + name: str + description: str + content: str + path: Path + + +class SkillRegistry: + """Registry for dynamically discovering and loading skills. + + Following Anthropic's progressive disclosure pattern: + - Level 1: Skill names and descriptions (loaded at startup) + - Level 2: Full skill content (loaded on-demand via load_skill) + + Skills are discovered from SKILL.md files in the skills directory. + Each SKILL.md has YAML frontmatter with name and description. + + Example SKILL.md structure: + ```markdown + --- + name: my_skill + description: Short description of what this skill does + --- + + # Full Skill Documentation + + Detailed instructions, examples, and usage patterns... + ``` + """ + + def __init__(self, skills_dir: str | Path | None = None): + """Initialize the skill registry. + + Args: + skills_dir: Directory containing skill subdirectories. + Each subdirectory should have a SKILL.md file. + Defaults to ./skills relative to this module. + """ + if skills_dir is None: + # Default to skills/ directory next to this file + skills_dir = Path(__file__).parent / "skills" + self._skills_dir = Path(skills_dir) + self._skills: dict[str, SkillMetadata] = {} + self._discover_skills() + + def _discover_skills(self) -> None: + """Discover all skills in the skills directory. + + Scans for SKILL.md files and parses YAML frontmatter + to extract name and description (Level 1 disclosure). + """ + if not self._skills_dir.exists(): + return + + for skill_dir in self._skills_dir.iterdir(): + if not skill_dir.is_dir(): + continue + + skill_file = skill_dir / "SKILL.md" + if not skill_file.exists(): + continue + + metadata = self._parse_skill_metadata(skill_file) + if metadata: + self._skills[metadata.name] = metadata + + def _parse_skill_metadata(self, skill_path: Path) -> SkillMetadata | None: + """Parse YAML frontmatter from a SKILL.md file. + + Args: + skill_path: Path to the SKILL.md file + + Returns: + SkillMetadata with name and description, or None if parsing fails + """ + try: + content = skill_path.read_text() + + # Parse YAML frontmatter (between --- delimiters) + frontmatter_match = re.match( + r'^---\s*\n(.*?)\n---\s*\n', + content, + re.DOTALL + ) + + if not frontmatter_match: + return None + + frontmatter = yaml.safe_load(frontmatter_match.group(1)) + + if not frontmatter or 'name' not in frontmatter: + return None + + return SkillMetadata( + name=frontmatter['name'], + description=frontmatter.get('description', ''), + path=skill_path, + ) + except Exception: + return None + + def get_skill_names(self) -> list[str]: + """Get list of all discovered skill names.""" + return list(self._skills.keys()) + + def get_skill_metadata(self, name: str) -> SkillMetadata | None: + """Get metadata for a specific skill.""" + return self._skills.get(name) + + def get_all_metadata(self) -> list[SkillMetadata]: + """Get metadata for all discovered skills.""" + return list(self._skills.values()) + + def load_skill(self, name: str) -> SkillContent | None: + """Load the full content of a skill (Level 2 disclosure). + + Args: + name: The skill name to load + + Returns: + SkillContent with full documentation, or None if not found + """ + metadata = self._skills.get(name) + if not metadata: + return None + + try: + full_content = metadata.path.read_text() + + # Remove YAML frontmatter for the content + content_match = re.match( + r'^---\s*\n.*?\n---\s*\n(.*)$', + full_content, + re.DOTALL + ) + + if content_match: + content = content_match.group(1).strip() + else: + content = full_content + + return SkillContent( + name=metadata.name, + description=metadata.description, + content=content, + path=metadata.path, + ) + except Exception: + return None + + def get_skills_summary(self) -> str: + """Get a formatted summary of all available skills. + + This is used in the agent's system prompt to inform it + of available skills without loading full content. + + Returns: + Formatted string listing all skills with descriptions + """ + if not self._skills: + return "No skills available." + + lines = ["Available Skills:"] + for name, metadata in sorted(self._skills.items()): + lines.append(f"- **{name}**: {metadata.description}") + + lines.append("") + lines.append("Use `load_skill(skill_name)` to load full documentation for a skill.") + + return "\n".join(lines) + + +def create_load_skill_tool(registry: SkillRegistry) -> dict[str, Any]: + """Create a load_skill tool function for the agent. + + This creates a callable tool that the agent can use to + load full skill content when it determines a skill is relevant. + + Args: + registry: The skill registry to load from + + Returns: + A tool function that can be added to the agent + """ + def load_skill(skill_name: str) -> str: + """Load the full documentation for a skill. + + Use this tool when you need detailed instructions, examples, + or patterns for a specific capability. Only load skills that + are relevant to the current task. + + Args: + skill_name: Name of the skill to load (e.g., 'bqml', 'bq_ai_operator') + + Returns: + Full skill documentation including examples and patterns + """ + skill = registry.load_skill(skill_name) + if skill is None: + available = ", ".join(registry.get_skill_names()) + return f"Skill '{skill_name}' not found. Available skills: {available}" + + return f"""# Skill: {skill.name} + +{skill.description} + +--- + +{skill.content} +""" + + return load_skill + + +# Module-level registry instance for convenience +_default_registry: SkillRegistry | None = None + + +def get_default_registry() -> SkillRegistry: + """Get the default skill registry (singleton).""" + global _default_registry + if _default_registry is None: + _default_registry = SkillRegistry() + return _default_registry + + +def get_skills_summary() -> str: + """Get summary of all available skills from the default registry.""" + return get_default_registry().get_skills_summary() + + +def load_skill(skill_name: str) -> str: + """Load a skill from the default registry. + + This is the function that should be added as a tool to the agent. + """ + return create_load_skill_tool(get_default_registry())(skill_name) diff --git a/contributing/samples/bigquery_skills_demo/skills/bq_ai_operator/SKILL.md b/contributing/samples/bigquery_skills_demo/skills/bq_ai_operator/SKILL.md new file mode 100644 index 0000000000..380dfb5fc2 --- /dev/null +++ b/contributing/samples/bigquery_skills_demo/skills/bq_ai_operator/SKILL.md @@ -0,0 +1,232 @@ +--- +name: bq_ai_operator +description: BigQuery AI Operator - Use managed AI functions (AI.CLASSIFY, AI.IF, AI.SCORE) directly in SQL for text classification, filtering, and scoring. Requires a BigQuery connection to Vertex AI. +--- + +# BQ AI Operator Skill (Managed AI Functions in SQL) + +Use managed AI functions directly in BigQuery SQL queries for text classification, filtering, and scoring. + +**IMPORTANT**: These are the NEW managed AI functions that require a `connection_id` to Vertex AI, NOT the older `ML.GENERATE_TEXT` style functions. + +## Prerequisites + +1. **Create a BigQuery connection to Vertex AI** (required for all AI functions): +```sql +-- Create a connection (run once) +CREATE CLOUD RESOURCE CONNECTION `us.my_ai_connection` +OPTIONS(location='us'); +``` + +2. **Grant the connection service account access to Vertex AI** + +## Available Managed AI Functions + +| Function | Purpose | Return Type | +|----------|---------|-------------| +| AI.CLASSIFY | Categorize text into classes | STRING | +| AI.IF | Natural language TRUE/FALSE filtering | BOOL | +| AI.SCORE | Rate/rank by criteria (0.0 to 1.0) | FLOAT64 | + +--- + +## AI.CLASSIFY - Categorize Text + +Classify text into one of the provided categories. + +### Syntax +```sql +AI.CLASSIFY( + input, -- STRING: the text to classify + categories => ['cat1', 'cat2'], -- ARRAY: possible categories + connection_id => 'LOCATION.CONNECTION_NAME' +) +``` + +### Examples + +**News article classification:** +```sql +SELECT + title, + body, + AI.CLASSIFY( + body, + categories => ['tech', 'sport', 'business', 'politics', 'entertainment', 'other'], + connection_id => 'us.my_ai_connection' -- Replace with your connection + ) AS category +FROM `bigquery-public-data.bbc_news.fulltext` +LIMIT 10; +``` + +**Sentiment classification with descriptions:** +```sql +SELECT + review_text, + AI.CLASSIFY( + review_text, + categories => [ + ('positive', 'happy, satisfied, recommends'), + ('negative', 'unhappy, disappointed, complaints'), + ('neutral', 'factual, no strong emotion') + ], + connection_id => 'us.my_ai_connection' -- Replace with your connection + ) AS sentiment +FROM `project.dataset.reviews` +LIMIT 10; +``` + +--- + +## AI.IF - Natural Language Filtering + +Returns TRUE or FALSE based on a natural language condition. + +### Syntax +```sql +AI.IF( + input, -- STRING: the text to evaluate + condition, -- STRING: natural language condition + connection_id => 'LOCATION.CONNECTION_NAME' +) +``` + +### Examples + +**Filter for eco-friendly products:** +```sql +SELECT product_name, description +FROM `project.products.catalog` +WHERE AI.IF( + description, + 'This product is eco-friendly, sustainable, or environmentally conscious', + connection_id => 'us.my_ai_connection' -- Use your connection: test-project-0728-467323.us.my_ai_connection +) = TRUE +LIMIT 10; +``` + +**Content moderation:** +```sql +SELECT + post_id, + content, + AI.IF( + content, + 'This content is appropriate for all ages and contains no spam, harassment, or explicit material', + connection_id => 'us.my_ai_connection' -- Replace with your connection + ) AS is_appropriate +FROM `project.social.user_posts` +LIMIT 10; +``` + +--- + +## AI.SCORE - Quality Scoring + +Returns a score between 0.0 and 1.0 based on criteria. + +### Syntax +```sql +AI.SCORE( + input, -- STRING: the text to score + criteria, -- STRING: scoring criteria + connection_id => 'LOCATION.CONNECTION_NAME' +) +``` + +### Examples + +**Review helpfulness scoring:** +```sql +SELECT + review_id, + review_text, + star_rating, + AI.SCORE( + review_text, + 'Rate this review helpfulness based on: detail level, specific examples, balanced perspective', + connection_id => 'us.my_ai_connection' -- Replace with your connection + ) AS helpfulness_score +FROM `project.reviews.product_reviews` +ORDER BY helpfulness_score DESC +LIMIT 10; +``` + +**Relevance scoring:** +```sql +SELECT + document_id, + title, + AI.SCORE( + content, + 'How relevant is this document to machine learning and AI topics', + connection_id => 'us.my_ai_connection' -- Replace with your connection + ) AS ml_relevance +FROM `project.docs.articles` +ORDER BY ml_relevance DESC +LIMIT 10; +``` + +--- + +## Complete Pipeline Example + +Combine multiple AI functions for a review intelligence pipeline: + +```sql +-- Step 1: Classify and score reviews +WITH classified AS ( + SELECT + review_id, + review_text, + star_rating, + AI.CLASSIFY( + review_text, + categories => ['positive', 'negative', 'neutral', 'mixed'], + connection_id => 'us.my_ai_connection' -- Replace with your connection + ) AS sentiment, + AI.SCORE( + review_text, + 'Review quality based on detail and helpfulness', + connection_id => 'us.my_ai_connection' -- Replace with your connection + ) AS quality_score + FROM `project.reviews.raw_reviews` + LIMIT 100 +) +-- Step 2: Filter appropriate content and categorize +SELECT + review_id, + sentiment, + quality_score, + CASE + WHEN quality_score >= 0.8 THEN 'featured' + WHEN quality_score >= 0.5 THEN 'standard' + ELSE 'low_quality' + END AS tier +FROM classified +WHERE AI.IF( + review_text, + 'Content is appropriate and not spam', + connection_id => 'us.my_ai_connection' -- Use your connection: test-project-0728-467323.us.my_ai_connection +) = TRUE +ORDER BY quality_score DESC; +``` + +--- + +## Important Notes + +1. **Connection Required**: All managed AI functions require a `connection_id` to a Vertex AI connection +2. **Preview Feature**: AI.CLASSIFY, AI.IF, and AI.SCORE are in public preview +3. **Region Support**: Works in all Gemini regions plus US/EU multi-regions +4. **Use LIMIT**: Always use LIMIT to control costs when testing +5. **String Return**: AI.CLASSIFY returns STRING, AI.IF returns BOOL, AI.SCORE returns FLOAT64 + +## Troubleshooting + +**Error: "connection not found"** +- Ensure you've created the connection: `CREATE CLOUD RESOURCE CONNECTION` +- Use the correct format: `LOCATION.CONNECTION_NAME` (e.g., `us.my_ai_connection`) + +**Error: "permission denied"** +- Grant the connection's service account access to Vertex AI API diff --git a/contributing/samples/bigquery_skills_demo/skills/bqml/SKILL.md b/contributing/samples/bigquery_skills_demo/skills/bqml/SKILL.md new file mode 100644 index 0000000000..5ec1e995d0 --- /dev/null +++ b/contributing/samples/bigquery_skills_demo/skills/bqml/SKILL.md @@ -0,0 +1,158 @@ +--- +name: bqml +description: BigQuery ML - Train, evaluate, and deploy machine learning models using SQL. Supports regression, classification, clustering, time series forecasting, and deep learning. +--- + +# BQML Skill (BigQuery Machine Learning) + +Train, evaluate, and deploy ML models directly in BigQuery using SQL. + +## Supported Model Types + +| Model Type | Use Case | SQL Option | +|------------|----------|------------| +| LINEAR_REG | Numeric prediction | `model_type='LINEAR_REG'` | +| LOGISTIC_REG | Binary/multiclass classification | `model_type='LOGISTIC_REG'` | +| KMEANS | Customer segmentation, clustering | `model_type='KMEANS'` | +| BOOSTED_TREE_REGRESSOR | Numeric prediction with XGBoost | `model_type='BOOSTED_TREE_REGRESSOR'` | +| BOOSTED_TREE_CLASSIFIER | Classification with XGBoost | `model_type='BOOSTED_TREE_CLASSIFIER'` | +| RANDOM_FOREST_REGRESSOR | Ensemble numeric prediction | `model_type='RANDOM_FOREST_REGRESSOR'` | +| RANDOM_FOREST_CLASSIFIER | Ensemble classification | `model_type='RANDOM_FOREST_CLASSIFIER'` | +| ARIMA_PLUS | Time series forecasting | `model_type='ARIMA_PLUS'` | +| DNN_REGRESSOR | Deep learning regression | `model_type='DNN_REGRESSOR'` | +| DNN_CLASSIFIER | Deep learning classification | `model_type='DNN_CLASSIFIER'` | + +## Core Workflow + +### Step 1: Train a Model +```sql +CREATE OR REPLACE MODEL `project.dataset.model_name` +OPTIONS( + model_type='LINEAR_REG', + input_label_cols=['target_column'], + enable_global_explain=TRUE +) AS +SELECT feature1, feature2, feature3, target_column +FROM `project.dataset.training_data` +WHERE target_column IS NOT NULL; +``` + +### Step 2: Evaluate the Model +```sql +SELECT * FROM ML.EVALUATE(MODEL `project.dataset.model_name`); +``` + +### Step 3: Get Feature Importance +```sql +SELECT * FROM ML.GLOBAL_EXPLAIN(MODEL `project.dataset.model_name`); +``` + +### Step 4: Make Predictions +```sql +SELECT * FROM ML.PREDICT( + MODEL `project.dataset.model_name`, + (SELECT feature1, feature2, feature3 FROM `project.dataset.new_data`) +); +``` + +### Step 5: Explain Predictions +```sql +SELECT * FROM ML.EXPLAIN_PREDICT( + MODEL `project.dataset.model_name`, + (SELECT feature1, feature2, feature3 FROM `project.dataset.new_data`), + STRUCT(3 as top_k_features) +); +``` + +## Example: Penguin Body Mass Prediction + +```sql +-- Train model +CREATE OR REPLACE MODEL `project.bqml_demo.penguin_weight` +OPTIONS( + model_type='LINEAR_REG', + input_label_cols=['body_mass_g'], + enable_global_explain=TRUE +) AS +SELECT species, island, culmen_length_mm, culmen_depth_mm, + flipper_length_mm, sex, body_mass_g +FROM `bigquery-public-data.ml_datasets.penguins` +WHERE body_mass_g IS NOT NULL AND sex IS NOT NULL; + +-- Evaluate +SELECT * FROM ML.EVALUATE(MODEL `project.bqml_demo.penguin_weight`); + +-- Feature importance +SELECT * FROM ML.GLOBAL_EXPLAIN(MODEL `project.bqml_demo.penguin_weight`); + +-- Predict +SELECT predicted_body_mass_g, species, island +FROM ML.PREDICT( + MODEL `project.bqml_demo.penguin_weight`, + (SELECT 'Adelie' as species, 'Torgersen' as island, + 39.1 as culmen_length_mm, 18.7 as culmen_depth_mm, + 181.0 as flipper_length_mm, 'MALE' as sex) +); +``` + +## Example: K-Means Clustering + +```sql +-- Create clustering model +CREATE OR REPLACE MODEL `project.bqml_demo.penguin_clusters` +OPTIONS( + model_type='KMEANS', + num_clusters=3, + standardize_features=TRUE +) AS +SELECT culmen_length_mm, culmen_depth_mm, flipper_length_mm, body_mass_g +FROM `bigquery-public-data.ml_datasets.penguins` +WHERE body_mass_g IS NOT NULL; + +-- Get cluster assignments +SELECT * FROM ML.PREDICT( + MODEL `project.bqml_demo.penguin_clusters`, + (SELECT culmen_length_mm, culmen_depth_mm, flipper_length_mm, body_mass_g + FROM `bigquery-public-data.ml_datasets.penguins` + WHERE body_mass_g IS NOT NULL) +); + +-- Analyze cluster centroids +SELECT * FROM ML.CENTROIDS(MODEL `project.bqml_demo.penguin_clusters`); +``` + +## Example: XGBoost Classification + +```sql +CREATE OR REPLACE MODEL `project.bqml_demo.species_classifier` +OPTIONS( + model_type='BOOSTED_TREE_CLASSIFIER', + input_label_cols=['species'], + num_parallel_tree=1, + max_tree_depth=6, + subsample=0.8, + data_split_method='AUTO_SPLIT' +) AS +SELECT island, culmen_length_mm, culmen_depth_mm, + flipper_length_mm, body_mass_g, sex, species +FROM `bigquery-public-data.ml_datasets.penguins` +WHERE species IS NOT NULL AND sex IS NOT NULL; + +-- Confusion matrix +SELECT * FROM ML.CONFUSION_MATRIX(MODEL `project.bqml_demo.species_classifier`); + +-- ROC curve (for binary classification) +SELECT * FROM ML.ROC_CURVE(MODEL `project.bqml_demo.species_classifier`); +``` + +## Key ML Functions + +- `ML.EVALUATE()` - Model performance metrics +- `ML.PREDICT()` - Generate predictions +- `ML.EXPLAIN_PREDICT()` - Predictions with feature attributions +- `ML.GLOBAL_EXPLAIN()` - Overall feature importance +- `ML.FEATURE_IMPORTANCE()` - Feature weights for tree models +- `ML.CONFUSION_MATRIX()` - Classification matrix +- `ML.ROC_CURVE()` - ROC curve data +- `ML.CENTROIDS()` - K-means cluster centers +- `ML.TRAINING_INFO()` - Training run details From f5160121f3e6499f135c6c2f4104a91cc6c0f53b Mon Sep 17 00:00:00 2001 From: Haiyuan Cao Date: Sun, 7 Dec 2025 12:48:56 -0800 Subject: [PATCH 2/6] feat: implement Claude Code-style ephemeral skill loading MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace persistent load_skill tool with ephemeral skill activation: - Skills are now injected into system prompt (not conversation history) - Add activate_skill/deactivate_skill/list_active_skills tools - Use ADK's InstructionProvider pattern for dynamic system prompt - Skills can be truly unloaded when no longer needed This mirrors Claude Code's approach where skills are loaded on-demand and can be removed to free up context space. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 --- .../samples/bigquery_skills_demo/README.md | 33 ++- .../samples/bigquery_skills_demo/agent.py | 119 ++++++--- .../bigquery_skills_demo/skill_registry.py | 234 ++++++++++++++---- 3 files changed, 304 insertions(+), 82 deletions(-) diff --git a/contributing/samples/bigquery_skills_demo/README.md b/contributing/samples/bigquery_skills_demo/README.md index a2f40cc358..d1724c7a01 100644 --- a/contributing/samples/bigquery_skills_demo/README.md +++ b/contributing/samples/bigquery_skills_demo/README.md @@ -1,13 +1,14 @@ # BigQuery Skills Demo -This sample demonstrates Anthropic's [Agent Skills Pattern](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills) for dynamic skill discovery with BigQuery ML and AI capabilities. +This sample demonstrates Anthropic's [Agent Skills Pattern](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills) for dynamic skill discovery with BigQuery ML and AI capabilities, enhanced with **Claude Code-style ephemeral skill loading**. ## Overview This demo showcases: - **Dynamic Skill Discovery**: Skills are discovered at runtime from SKILL.md files - **Progressive Disclosure**: Only skill names/descriptions loaded initially; full content on-demand -- **load_skill Tool**: Agent loads full skill documentation when relevant to the task +- **Ephemeral Skill Loading**: Skills are injected into the system prompt (not conversation history) and can be truly unloaded when no longer needed +- **Context Management**: Agent can activate/deactivate skills to manage context efficiently ### Available Skills @@ -91,8 +92,32 @@ categories: tech, sport, business, politics, entertainment, other. 2. **YAML Frontmatter**: Each SKILL.md has metadata (name, description) in YAML frontmatter 3. **Progressive Loading**: - Level 1: Agent sees skill names and descriptions in its system prompt - - Level 2: Agent calls `load_skill(skill_name)` to get full documentation -4. **On-Demand Loading**: Full skill content is only loaded when relevant to the task + - Level 2: Agent calls `activate_skill(skill_name)` to load full documentation +4. **Ephemeral Loading (Claude Code-style)**: + - Active skills are injected into the **system prompt**, not conversation history + - Skills can be deactivated with `deactivate_skill(skill_name)` to free up context + - The system prompt is rebuilt fresh each LLM call, so deactivated skills truly disappear + - This prevents context accumulation unlike traditional tool responses + +### Key Difference from Traditional Approaches + +Traditional skill loading returns skill content as a tool response, which persists in conversation history forever. This demo uses ADK's **InstructionProvider** pattern: + +```python +# Traditional (persistent) - skill content stays in history +def load_skill(skill_name: str) -> str: + return skill_content # This persists in conversation history + +# Ephemeral (this demo) - skill content injected into system prompt +def instruction_provider(ctx: ReadonlyContext) -> str: + active_skills = ctx.state.get("active_skills", []) + return build_system_prompt_with_skills(active_skills) +``` + +Benefits: +- **True unloading**: Deactivated skills are removed from context +- **Better context management**: Agent can activate skills when needed, deactivate when done +- **Mirrors Claude Code**: Similar to how Claude Code loads skills from filesystem on-demand ## Code Structure diff --git a/contributing/samples/bigquery_skills_demo/agent.py b/contributing/samples/bigquery_skills_demo/agent.py index 88e465edca..a8391d4753 100644 --- a/contributing/samples/bigquery_skills_demo/agent.py +++ b/contributing/samples/bigquery_skills_demo/agent.py @@ -56,8 +56,14 @@ from google.adk.tools.bigquery.config import WriteMode import google.auth -# Import the dynamic skill registry -from .skill_registry import SkillRegistry, load_skill +# Import the dynamic skill registry with ephemeral skill loading +from .skill_registry import ( + SkillRegistry, + create_skill_instruction_provider, + activate_skill, + deactivate_skill, + list_active_skills, +) # Agent name AGENT_NAME = "bigquery_skills_demo_agent" @@ -93,38 +99,45 @@ skill_registry = SkillRegistry() SKILLS_SUMMARY = skill_registry.get_skills_summary() -# Create load_skill tool for the agent -load_skill_tool = FunctionTool(load_skill) +# Create instruction provider for ephemeral skill loading (Claude Code-style) +# This injects active skills into the system prompt, not conversation history +skill_instruction_provider = create_skill_instruction_provider(skill_registry) -# Create the root agent with BigQuery tools and dynamic skill loading -root_agent = LlmAgent( - model="gemini-2.5-pro", - name=AGENT_NAME, - description=( - "Data science agent with BigQuery ML and AI capabilities. " - "Uses dynamic skill discovery to load relevant capabilities on-demand." - ), - instruction=f"""\ +# Create skill management tools +activate_skill_tool = FunctionTool(activate_skill) +deactivate_skill_tool = FunctionTool(deactivate_skill) +list_active_skills_tool = FunctionTool(list_active_skills) + +# Base instruction for the agent (static part) +BASE_INSTRUCTION = f"""\ You are a data science agent with BigQuery capabilities and dynamic skill loading. -## How Skills Work (Anthropic Pattern) +## How Skills Work (Claude Code-style Ephemeral Loading) You have access to specialized skills that provide detailed guidance for complex tasks. -Skills are loaded on-demand to keep context focused and efficient. +Skills are loaded on-demand and can be UNLOADED when no longer needed to free up context. + +**This is important**: Unlike traditional tool responses that persist in conversation history, +activated skills are injected into the system prompt and can be truly removed when deactivated. **Current Available Skills:** {SKILLS_SUMMARY} -**When to use load_skill:** -1. When the user asks about ML model training, prediction, or evaluation → load "bqml" -2. When the user asks about AI/text analysis, classification, or generation → load "bq_ai_operator" -3. Load skills BEFORE attempting complex operations to get proper syntax and examples +**Skill Management Tools:** +- `activate_skill(skill_name)`: Load a skill's documentation into context +- `deactivate_skill(skill_name)`: Remove a skill's documentation from context (frees up space!) +- `list_active_skills()`: See which skills are currently loaded -**Progressive Disclosure:** -- You see skill names and descriptions above (Level 1) -- Call `load_skill(skill_name)` to get full documentation with examples (Level 2) -- Only load skills when they're relevant to the current task +**When to activate skills:** +1. When the user asks about ML model training, prediction, or evaluation → activate "bqml" +2. When the user asks about AI/text analysis, classification, or scoring → activate "bq_ai_operator" +3. Activate skills BEFORE attempting complex operations to get proper syntax and examples + +**When to deactivate skills:** +- After completing a task that used a skill +- When switching to a different type of task +- When context is getting large and you no longer need the skill ## Available BigQuery Tools @@ -132,7 +145,6 @@ - `get_table_info`: Get schema information for a table - `list_dataset_ids`: List datasets in a project - `list_table_ids`: List tables in a dataset -- `load_skill`: Load full documentation for a skill ## Project Configuration @@ -142,16 +154,18 @@ ## Workflow Example 1. User asks: "Train a model to predict penguin weight" -2. You call: `load_skill("bqml")` to get BQML documentation +2. You call: `activate_skill("bqml")` to load BQML documentation 3. You follow the skill's examples to CREATE MODEL, EVALUATE, and PREDICT 4. You explain results to the user +5. You call: `deactivate_skill("bqml")` to free up context for next task ## Guidelines -1. **Load skills first**: Before complex ML or AI operations, load the relevant skill +1. **Activate skills first**: Before complex ML or AI operations, activate the relevant skill 2. **Explore data first**: Use `get_table_info` or `SELECT * LIMIT 5` before complex queries 3. **Use LIMIT**: Prevent large result sets with `LIMIT 10-100` 4. **Explain your steps**: Describe what each query does and interpret results +5. **Deactivate when done**: Free up context by deactivating skills you no longer need ## Quick Reference (without loading skills) @@ -164,12 +178,53 @@ **AI Operator Quick Start:** ```sql --- Classify: AI.CLASSIFY(MODEL `...`, text, ['class1', 'class2']) --- Generate: AI.GENERATE(MODEL `...`, 'prompt') --- Extract: AI.EXTRACT(MODEL `...`, text, STRUCT(...)) +-- Classify: AI.CLASSIFY(text, categories => [...], connection_id => 'loc.conn') +-- Filter: AI.IF(text, 'condition', connection_id => 'loc.conn') +-- Score: AI.SCORE(text, 'criteria', connection_id => 'loc.conn') ``` -For detailed syntax and examples, use `load_skill("bqml")` or `load_skill("bq_ai_operator")`. -""", - tools=[bigquery_toolset, load_skill_tool], +For detailed syntax and examples, use `activate_skill("bqml")` or `activate_skill("bq_ai_operator")`. +""" + + +def create_combined_instruction_provider(base_instruction: str, skill_provider): + """Create an instruction provider that combines base instruction with active skills. + + This follows the Claude Code pattern where skills are injected into the system prompt + (ephemeral) rather than conversation history (persistent). + """ + from google.adk.agents.readonly_context import ReadonlyContext + + def combined_provider(ctx: ReadonlyContext) -> str: + # Get dynamic skill content + skill_content = skill_provider(ctx) + + if skill_content: + return f"{base_instruction}\n\n{skill_content}" + return base_instruction + + return combined_provider + + +# Create the combined instruction provider +instruction_provider = create_combined_instruction_provider( + BASE_INSTRUCTION, skill_instruction_provider +) + +# Create the root agent with BigQuery tools and ephemeral skill loading +root_agent = LlmAgent( + model="gemini-2.5-pro", + name=AGENT_NAME, + description=( + "Data science agent with BigQuery ML and AI capabilities. " + "Uses Claude Code-style ephemeral skill loading - skills can be activated " + "and deactivated to manage context efficiently." + ), + instruction=instruction_provider, + tools=[ + bigquery_toolset, + activate_skill_tool, + deactivate_skill_tool, + list_active_skills_tool, + ], ) diff --git a/contributing/samples/bigquery_skills_demo/skill_registry.py b/contributing/samples/bigquery_skills_demo/skill_registry.py index c82cacff72..3aa81e6553 100644 --- a/contributing/samples/bigquery_skills_demo/skill_registry.py +++ b/contributing/samples/bigquery_skills_demo/skill_registry.py @@ -20,13 +20,17 @@ Key concepts: 1. Skills are stored as SKILL.md files with YAML frontmatter 2. At startup, only skill names and descriptions are loaded (progressive disclosure) -3. Agent loads full skill content on-demand via load_skill tool -4. This reduces context usage while providing rich capabilities when needed +3. Agent activates/deactivates skills which inject content into system prompt +4. Skill content is in system prompt (not conversation history) - ephemeral! + +This approach mirrors Claude Code's filesystem-based skill loading where: +- Skills are loaded on-demand into the current context +- Skills can be unloaded when no longer needed +- Context doesn't accumulate skill content permanently """ from __future__ import annotations -import os import re from dataclasses import dataclass from pathlib import Path @@ -34,6 +38,13 @@ import yaml +from google.adk.agents.readonly_context import ReadonlyContext +from google.adk.tools import ToolContext + + +# State key for tracking active skills (using temp: prefix so it's session-scoped) +ACTIVE_SKILLS_KEY = "active_skills" + @dataclass class SkillMetadata: @@ -57,7 +68,7 @@ class SkillRegistry: Following Anthropic's progressive disclosure pattern: - Level 1: Skill names and descriptions (loaded at startup) - - Level 2: Full skill content (loaded on-demand via load_skill) + - Level 2: Full skill content (injected into system prompt when activated) Skills are discovered from SKILL.md files in the skills directory. Each SKILL.md has YAML frontmatter with name and description. @@ -158,8 +169,8 @@ def get_all_metadata(self) -> list[SkillMetadata]: """Get metadata for all discovered skills.""" return list(self._skills.values()) - def load_skill(self, name: str) -> SkillContent | None: - """Load the full content of a skill (Level 2 disclosure). + def load_skill_content(self, name: str) -> SkillContent | None: + """Load the full content of a skill. Args: name: The skill name to load @@ -212,73 +223,204 @@ def get_skills_summary(self) -> str: lines.append(f"- **{name}**: {metadata.description}") lines.append("") - lines.append("Use `load_skill(skill_name)` to load full documentation for a skill.") + lines.append("Use `activate_skill(skill_name)` to load a skill's full documentation.") + lines.append("Use `deactivate_skill(skill_name)` to unload a skill when done.") return "\n".join(lines) -def create_load_skill_tool(registry: SkillRegistry) -> dict[str, Any]: - """Create a load_skill tool function for the agent. +# Module-level registry instance for convenience +_default_registry: SkillRegistry | None = None - This creates a callable tool that the agent can use to - load full skill content when it determines a skill is relevant. - Args: - registry: The skill registry to load from +def get_default_registry() -> SkillRegistry: + """Get the default skill registry (singleton).""" + global _default_registry + if _default_registry is None: + _default_registry = SkillRegistry() + return _default_registry - Returns: - A tool function that can be added to the agent - """ - def load_skill(skill_name: str) -> str: - """Load the full documentation for a skill. - Use this tool when you need detailed instructions, examples, - or patterns for a specific capability. Only load skills that - are relevant to the current task. +def get_skills_summary() -> str: + """Get summary of all available skills from the default registry.""" + return get_default_registry().get_skills_summary() - Args: - skill_name: Name of the skill to load (e.g., 'bqml', 'bq_ai_operator') - Returns: - Full skill documentation including examples and patterns - """ - skill = registry.load_skill(skill_name) - if skill is None: - available = ", ".join(registry.get_skill_names()) - return f"Skill '{skill_name}' not found. Available skills: {available}" +# ============================================================================= +# Dynamic Instruction Provider (Claude Code-style approach) +# ============================================================================= - return f"""# Skill: {skill.name} +def create_skill_instruction_provider(registry: SkillRegistry): + """Create an instruction provider that injects active skills into system prompt. + + This is the key to ephemeral skill loading! The instruction provider is called + on EVERY LLM request and returns content that goes into the system prompt. + Since it reads from session state, skills can be activated/deactivated and + the system prompt updates automatically. + + Unlike tool responses which persist in conversation history, the system prompt + is rebuilt fresh each time - so deactivated skills truly disappear from context. + + Args: + registry: The skill registry to load skills from + + Returns: + An instruction provider function compatible with LlmAgent.instruction + """ + def instruction_provider(ctx: ReadonlyContext) -> str: + """Generate dynamic instructions based on active skills.""" + # Get active skills from session state + active_skills: list[str] = ctx.state.get(ACTIVE_SKILLS_KEY, []) + + if not active_skills: + return "" # No active skills, no additional instructions + + # Build skill content section + skill_sections = [] + for skill_name in active_skills: + skill = registry.load_skill_content(skill_name) + if skill: + skill_sections.append(f""" +## Active Skill: {skill.name} {skill.description} --- {skill.content} +""") + + if not skill_sections: + return "" + + return f""" +# Currently Active Skills + +The following skills have been loaded and are available for this task: + +{"".join(skill_sections)} + +--- +**Note**: Use `deactivate_skill(skill_name)` when you're done with a skill to free up context. """ - return load_skill + return instruction_provider -# Module-level registry instance for convenience -_default_registry: SkillRegistry | None = None +# ============================================================================= +# Skill Activation/Deactivation Tools +# ============================================================================= +def activate_skill(skill_name: str, tool_context: ToolContext) -> str: + """Activate a skill to load its full documentation into context. -def get_default_registry() -> SkillRegistry: - """Get the default skill registry (singleton).""" - global _default_registry - if _default_registry is None: - _default_registry = SkillRegistry() - return _default_registry + When activated, the skill's content will be injected into the system prompt + for all subsequent LLM calls. This is ephemeral - the content is NOT stored + in conversation history and can be removed by calling deactivate_skill. + Args: + skill_name: Name of the skill to activate (e.g., 'bqml', 'bq_ai_operator') -def get_skills_summary() -> str: - """Get summary of all available skills from the default registry.""" - return get_default_registry().get_skills_summary() + Returns: + Confirmation message or error if skill not found + """ + registry = get_default_registry() + + # Verify skill exists + if skill_name not in registry.get_skill_names(): + available = ", ".join(registry.get_skill_names()) + return f"Skill '{skill_name}' not found. Available skills: {available}" + + # Get current active skills from state + active_skills: list[str] = list(tool_context.state.get(ACTIVE_SKILLS_KEY, [])) + + # Add skill if not already active + if skill_name not in active_skills: + active_skills.append(skill_name) + tool_context.state[ACTIVE_SKILLS_KEY] = active_skills + + # Get skill metadata for confirmation + metadata = registry.get_skill_metadata(skill_name) + return f"Activated skill '{skill_name}': {metadata.description}\n\nThe skill documentation is now available in your context. Use deactivate_skill('{skill_name}') when done." + else: + return f"Skill '{skill_name}' is already active." + + +def deactivate_skill(skill_name: str, tool_context: ToolContext) -> str: + """Deactivate a skill to remove its documentation from context. + + This removes the skill content from the system prompt, freeing up context + space for other information. The skill can be reactivated later if needed. + + Args: + skill_name: Name of the skill to deactivate + + Returns: + Confirmation message + """ + # Get current active skills from state + active_skills: list[str] = list(tool_context.state.get(ACTIVE_SKILLS_KEY, [])) + + if skill_name in active_skills: + active_skills.remove(skill_name) + tool_context.state[ACTIVE_SKILLS_KEY] = active_skills + return f"Deactivated skill '{skill_name}'. Its documentation has been removed from context." + else: + return f"Skill '{skill_name}' is not currently active." + + +def list_active_skills(tool_context: ToolContext) -> str: + """List all currently active skills. + Returns: + List of active skill names or message if none active + """ + active_skills: list[str] = tool_context.state.get(ACTIVE_SKILLS_KEY, []) + + if not active_skills: + return "No skills are currently active. Use activate_skill(skill_name) to load a skill." + + registry = get_default_registry() + lines = ["Currently active skills:"] + for name in active_skills: + metadata = registry.get_skill_metadata(name) + if metadata: + lines.append(f"- **{name}**: {metadata.description}") + else: + lines.append(f"- **{name}**: (metadata not found)") + + return "\n".join(lines) + + +# ============================================================================= +# Legacy load_skill function (for backward compatibility) +# ============================================================================= def load_skill(skill_name: str) -> str: - """Load a skill from the default registry. + """Load a skill's documentation (legacy function). + + Note: This returns the skill content as a string which will persist in + conversation history. For ephemeral skill loading, use activate_skill() + instead with the dynamic instruction provider. + + Args: + skill_name: Name of the skill to load - This is the function that should be added as a tool to the agent. + Returns: + Full skill documentation """ - return create_load_skill_tool(get_default_registry())(skill_name) + registry = get_default_registry() + skill = registry.load_skill_content(skill_name) + + if skill is None: + available = ", ".join(registry.get_skill_names()) + return f"Skill '{skill_name}' not found. Available skills: {available}" + + return f"""# Skill: {skill.name} + +{skill.description} + +--- + +{skill.content} +""" From e57c8edd02c8b32ca56fecdece8791984e94b601 Mon Sep 17 00:00:00 2001 From: Haiyuan Cao Date: Mon, 8 Dec 2025 02:24:50 -0800 Subject: [PATCH 3/6] feat: add callback-based skill management and connection discovery for BigQuery Skills Demo MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This commit enhances the BigQuery Skills Demo with: - **Callback-based auto-activation**: Skills are automatically activated based on keywords in user messages via before_model_callback, eliminating the need for explicit LLM tool calls to manage skills - **Auto-deactivation**: Skills are cleared after each turn via after_agent_callback to free up context - **list_connections and create_connection tools**: Agents can now discover existing BigQuery connections and create new ones with automatic IAM grants - **Location matching documentation**: Skills now document the critical requirement that connection location must match dataset location - **bq_remote_model skill**: New skill for remote models with Vertex AI, including Gemini 2.5 Pro as default model and task-specific parameter guidance (max_output_tokens for summarization vs classification) - **AI.SCORE tuple syntax fix**: Corrected syntax to use tuple format per official BigQuery documentation - **Keywords in SKILL.md frontmatter**: Skills now define keywords that are used for automatic detection 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 --- .../samples/bigquery_skills_demo/README.md | 168 +++++-- .../samples/bigquery_skills_demo/agent.py | 85 +++- .../bigquery_skills_demo/skill_callbacks.py | 460 ++++++++++++++++++ .../bigquery_skills_demo/skill_classifier.py | 291 +++++++++++ .../bigquery_skills_demo/skill_registry.py | 19 +- .../skills/bq_ai_operator/SKILL.md | 153 +++++- .../skills/bq_remote_model/SKILL.md | 429 ++++++++++++++++ .../bigquery_skills_demo/skills/bqml/SKILL.md | 17 + src/google/adk/tools/bigquery/__init__.py | 8 + .../adk/tools/bigquery/bigquery_toolset.py | 2 + .../adk/tools/bigquery/metadata_tool.py | 256 ++++++++++ .../adk/tools/bigquery/skills/__init__.py | 29 ++ .../bigquery/skills/bq_ai_operator_skill.py | 377 ++++++++++++++ .../adk/tools/bigquery/skills/bqml_skill.py | 323 ++++++++++++ 14 files changed, 2537 insertions(+), 80 deletions(-) create mode 100644 contributing/samples/bigquery_skills_demo/skill_callbacks.py create mode 100644 contributing/samples/bigquery_skills_demo/skill_classifier.py create mode 100644 contributing/samples/bigquery_skills_demo/skills/bq_remote_model/SKILL.md create mode 100644 src/google/adk/tools/bigquery/skills/__init__.py create mode 100644 src/google/adk/tools/bigquery/skills/bq_ai_operator_skill.py create mode 100644 src/google/adk/tools/bigquery/skills/bqml_skill.py diff --git a/contributing/samples/bigquery_skills_demo/README.md b/contributing/samples/bigquery_skills_demo/README.md index d1724c7a01..bebc272147 100644 --- a/contributing/samples/bigquery_skills_demo/README.md +++ b/contributing/samples/bigquery_skills_demo/README.md @@ -1,14 +1,16 @@ # BigQuery Skills Demo -This sample demonstrates Anthropic's [Agent Skills Pattern](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills) for dynamic skill discovery with BigQuery ML and AI capabilities, enhanced with **Claude Code-style ephemeral skill loading**. +This sample demonstrates Anthropic's [Agent Skills Pattern](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills) for dynamic skill discovery with BigQuery ML and AI capabilities, enhanced with **callback-based automatic skill loading**. ## Overview This demo showcases: -- **Dynamic Skill Discovery**: Skills are discovered at runtime from SKILL.md files +- **Dynamic Skill Discovery**: Skills are discovered at runtime from SKILL.md files with YAML frontmatter - **Progressive Disclosure**: Only skill names/descriptions loaded initially; full content on-demand -- **Ephemeral Skill Loading**: Skills are injected into the system prompt (not conversation history) and can be truly unloaded when no longer needed -- **Context Management**: Agent can activate/deactivate skills to manage context efficiently +- **Callback-based Auto-activation**: Skills are automatically activated based on keywords in user messages (no LLM calls needed!) +- **Ephemeral Skill Loading**: Skills are injected into the system prompt (not conversation history) and can be truly unloaded +- **Automatic Cleanup**: Skills are auto-deactivated after each turn to free up context +- **Scalable Design**: Adding new skills requires only a SKILL.md file - no code changes! ### Available Skills @@ -22,6 +24,11 @@ This demo showcases: - AI.IF: Natural language TRUE/FALSE filtering - AI.SCORE: Rate/rank content by criteria (0.0 to 1.0) +3. **bq_remote_model** - Remote models connecting to Vertex AI + - CREATE REMOTE MODEL: Connect to Gemini, Claude, Llama, and custom endpoints + - AI.GENERATE_TEXT: Text generation with LLMs + - AI.GENERATE_EMBEDDING: Vector embeddings for semantic search + ## Prerequisites 1. Google Cloud project with BigQuery and Vertex AI enabled @@ -34,7 +41,7 @@ This demo showcases: export GOOGLE_CLOUD_PROJECT=your-project-id ``` -### For AI Functions (bq_ai_operator skill) +### For AI Functions (bq_ai_operator and bq_remote_model skills) Create a BigQuery connection to Vertex AI: ```bash @@ -74,84 +81,167 @@ adk web contributing/samples --port 8000 ## Example Prompts -### BQML Skill +### BQML Skill (auto-activated by: "train", "model", "predict", "regression", "kmeans") ``` Train a linear regression model to predict penguin body weight using the public penguins dataset, then evaluate it and show feature importance. ``` -### BQ AI Operator Skill +### BQ AI Operator Skill (auto-activated by: "classify", "AI.CLASSIFY", "sentiment", "categorize") ``` Classify 5 BBC news articles by their topic using AI.CLASSIFY with categories: tech, sport, business, politics, entertainment, other. ``` -## How It Works +### BQ Remote Model Skill (auto-activated by: "generate text", "gemini", "embeddings", "llm") +``` +Create a remote model using Gemini 2.0 Flash and use it to summarize +product descriptions from my table. +``` -1. **Skill Discovery**: The `SkillRegistry` scans the `skills/` directory for SKILL.md files -2. **YAML Frontmatter**: Each SKILL.md has metadata (name, description) in YAML frontmatter -3. **Progressive Loading**: - - Level 1: Agent sees skill names and descriptions in its system prompt - - Level 2: Agent calls `activate_skill(skill_name)` to load full documentation -4. **Ephemeral Loading (Claude Code-style)**: - - Active skills are injected into the **system prompt**, not conversation history - - Skills can be deactivated with `deactivate_skill(skill_name)` to free up context - - The system prompt is rebuilt fresh each LLM call, so deactivated skills truly disappear - - This prevents context accumulation unlike traditional tool responses +## How It Works -### Key Difference from Traditional Approaches +### Architecture: Callback-based Skill Management -Traditional skill loading returns skill content as a tool response, which persists in conversation history forever. This demo uses ADK's **InstructionProvider** pattern: +This demo uses ADK callbacks instead of LLM tool calls for skill management: -```python -# Traditional (persistent) - skill content stays in history -def load_skill(skill_name: str) -> str: - return skill_content # This persists in conversation history - -# Ephemeral (this demo) - skill content injected into system prompt -def instruction_provider(ctx: ReadonlyContext) -> str: - active_skills = ctx.state.get("active_skills", []) - return build_system_prompt_with_skills(active_skills) ``` +┌─────────────────────────────────────────────────────────────────────┐ +│ User Message │ +│ "Train a model to predict penguin weight" │ +└─────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────┐ +│ before_model_callback │ +│ 1. Extract keywords from user message │ +│ 2. Match against skill keywords (from SKILL.md frontmatter) │ +│ 3. Activate matching skills: ["bqml"] │ +│ 4. Skills injected into system prompt via instruction provider │ +└─────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────┐ +│ LLM Processing │ +│ System prompt now includes: │ +│ - Base instruction │ +│ - Active skill documentation (BQML syntax, examples) │ +└─────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────┐ +│ after_agent_callback │ +│ 1. Clear active skills from state │ +│ 2. Context freed for next interaction │ +└─────────────────────────────────────────────────────────────────────┘ +``` + +### Key Components + +1. **Skill Discovery** (`skill_registry.py`) + - Scans `skills/` directory for SKILL.md files + - Parses YAML frontmatter (name, description, keywords) + - Provides instruction provider for ephemeral skill injection + +2. **Skill Callbacks** (`skill_callbacks.py`) + - `before_model_callback`: Detects and activates skills based on keywords + - `after_agent_callback`: Cleans up skills after each turn + - Supports multiple detection modes: keyword (recommended), hybrid, llm + +3. **Agent Configuration** (`agent.py`) + - Registers callbacks with LlmAgent + - Combines base instruction with dynamic skill content + - Manual skill tools available as fallback + +### Callback vs Tool Approach Comparison -Benefits: -- **True unloading**: Deactivated skills are removed from context -- **Better context management**: Agent can activate skills when needed, deactivate when done -- **Mirrors Claude Code**: Similar to how Claude Code loads skills from filesystem on-demand +| Aspect | Callback (This Demo) | Tool-based | +|--------|---------------------|------------| +| **LLM Calls** | Zero for skill management | 1-2 per skill activation | +| **Latency** | Instant (keyword matching) | Adds round-trip time | +| **Cost** | No additional tokens | Extra tool call tokens | +| **Control** | System-level, deterministic | LLM decides when to activate | +| **Best For** | Domain-specific terms (BigQuery) | Semantic understanding needed | + +### Why Keyword Detection for BigQuery? + +BigQuery has **highly domain-specific terminology** that makes keyword detection ideal: +- "BQML", "ML.PREDICT", "CREATE MODEL" → bqml skill +- "AI.CLASSIFY", "AI.IF", "AI.SCORE" → bq_ai_operator skill +- "GENERATE_TEXT", "gemini", "embeddings" → bq_remote_model skill + +These terms are unambiguous - you don't need an LLM to understand that "AI.CLASSIFY" relates to the AI operator skill. ## Code Structure ``` bigquery_skills_demo/ ├── __init__.py # Module init -├── agent.py # Agent with BigQuery tools and load_skill -├── skill_registry.py # Dynamic skill discovery (Anthropic pattern) +├── agent.py # Agent with BigQuery tools and callbacks +├── skill_registry.py # Dynamic skill discovery + instruction provider +├── skill_callbacks.py # Callback-based auto-activation +├── skill_classifier.py # Optional LLM-based classification (for hybrid mode) ├── skills/ │ ├── bqml/ -│ │ └── SKILL.md # BQML skill documentation -│ └── bq_ai_operator/ -│ └── SKILL.md # AI operator skill documentation +│ │ └── SKILL.md # BQML skill (keywords: train, model, predict, etc.) +│ ├── bq_ai_operator/ +│ │ └── SKILL.md # AI operator skill (keywords: classify, sentiment, etc.) +│ └── bq_remote_model/ +│ └── SKILL.md # Remote model skill (keywords: gemini, embeddings, etc.) └── README.md # This file ``` ## Adding New Skills +Adding a new skill requires **only a SKILL.md file** - no code changes needed! + 1. Create a directory under `skills/` (e.g., `skills/my_skill/`) 2. Add a `SKILL.md` file with YAML frontmatter: ```markdown --- name: my_skill description: Short description of what this skill does + keywords: + - keyword1 + - keyword2 + - specific_function_name --- # My Skill Documentation Detailed instructions, examples, and usage patterns... ``` -3. The skill will be automatically discovered on agent startup +3. The skill will be automatically discovered and keyword patterns built from frontmatter + +### Keyword Guidelines + +- Use domain-specific terms that clearly indicate the skill is needed +- Include function names (e.g., "ML.PREDICT", "AI.CLASSIFY") +- Include common user phrases (e.g., "train", "classify", "embeddings") +- Multiple keywords increase detection coverage + +## Detection Modes + +The `SkillCallbacks` class supports three detection modes: + +```python +# In agent.py +skill_callbacks = SkillCallbacks( + skill_registry, + auto_deactivate=True, + detection_mode="keyword", # "keyword" | "hybrid" | "llm" +) +``` + +| Mode | Description | Best For | +|------|-------------|----------| +| `keyword` | Regex pattern matching from SKILL.md keywords | Domain-specific terms (recommended) | +| `hybrid` | LLM classification with keyword fallback | Mixed semantic/specific queries | +| `llm` | Pure LLM-based semantic classification | Paraphrased/ambiguous requests | ## References - [Anthropic: Equipping Agents with Skills](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills) - [BigQuery ML Documentation](https://cloud.google.com/bigquery/docs/bqml-introduction) - [BigQuery AI Functions](https://cloud.google.com/bigquery/docs/ai-functions) +- [BigQuery Remote Models](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-create-remote-model) diff --git a/contributing/samples/bigquery_skills_demo/agent.py b/contributing/samples/bigquery_skills_demo/agent.py index a8391d4753..6dbebba2ef 100644 --- a/contributing/samples/bigquery_skills_demo/agent.py +++ b/contributing/samples/bigquery_skills_demo/agent.py @@ -65,6 +65,9 @@ list_active_skills, ) +# Import callback-based skill management (saves LLM calls) +from .skill_callbacks import SkillCallbacks + # Agent name AGENT_NAME = "bigquery_skills_demo_agent" @@ -103,41 +106,49 @@ # This injects active skills into the system prompt, not conversation history skill_instruction_provider = create_skill_instruction_provider(skill_registry) -# Create skill management tools +# Create skill management tools (kept for manual override if needed) activate_skill_tool = FunctionTool(activate_skill) deactivate_skill_tool = FunctionTool(deactivate_skill) list_active_skills_tool = FunctionTool(list_active_skills) +# Create callback-based skill management with keyword detection +# Modes: "keyword" (recommended for BigQuery), "hybrid", "llm" +# Keyword mode is fast, no API calls, and BigQuery terms are domain-specific enough +skill_callbacks = SkillCallbacks( + skill_registry, + auto_deactivate=True, + detection_mode="keyword", # Fast keyword detection - ideal for domain-specific terms +) + # Base instruction for the agent (static part) BASE_INSTRUCTION = f"""\ -You are a data science agent with BigQuery capabilities and dynamic skill loading. +You are a data science agent with BigQuery capabilities and automatic skill loading. -## How Skills Work (Claude Code-style Ephemeral Loading) +## How Skills Work (Keyword-based Auto-activation) You have access to specialized skills that provide detailed guidance for complex tasks. -Skills are loaded on-demand and can be UNLOADED when no longer needed to free up context. +**Skills are automatically activated** based on keywords in the user's message - no API calls needed! -**This is important**: Unlike traditional tool responses that persist in conversation history, -activated skills are injected into the system prompt and can be truly removed when deactivated. +Skills are: +- **Auto-activated by keywords**: Domain-specific terms trigger skill loading instantly + - "train", "model", "predict", "regression", "kmeans" → bqml + - "classify", "AI.CLASSIFY", "sentiment", "categorize" → bq_ai_operator + - "generate text", "gemini", "embeddings", "remote model" → bq_remote_model +- **Auto-deactivated**: After you complete your response, skills are cleared to free context +- **Multi-skill support**: Multiple skills can be loaded simultaneously if needed + +**This is efficient**: Keyword detection is instant with zero API calls! **Current Available Skills:** {SKILLS_SUMMARY} -**Skill Management Tools:** -- `activate_skill(skill_name)`: Load a skill's documentation into context -- `deactivate_skill(skill_name)`: Remove a skill's documentation from context (frees up space!) +**Manual Skill Management (optional, for override):** +- `activate_skill(skill_name)`: Manually load a skill if auto-detection missed it +- `deactivate_skill(skill_name)`: Manually unload a skill (rarely needed) - `list_active_skills()`: See which skills are currently loaded -**When to activate skills:** -1. When the user asks about ML model training, prediction, or evaluation → activate "bqml" -2. When the user asks about AI/text analysis, classification, or scoring → activate "bq_ai_operator" -3. Activate skills BEFORE attempting complex operations to get proper syntax and examples - -**When to deactivate skills:** -- After completing a task that used a skill -- When switching to a different type of task -- When context is getting large and you no longer need the skill +**Note**: In most cases, just focus on the task - skills will be loaded automatically! ## Available BigQuery Tools @@ -145,6 +156,18 @@ - `get_table_info`: Get schema information for a table - `list_dataset_ids`: List datasets in a project - `list_table_ids`: List tables in a dataset +- `list_connections`: **ALWAYS use this first** to discover existing BigQuery connections +- `create_connection`: Create a new BigQuery connection (auto-grants Vertex AI User role) + +## Connection Management (IMPORTANT) + +For AI functions and remote models, you need a BigQuery connection to Vertex AI. + +**ALWAYS follow this workflow:** +1. **First**: Call `list_connections(project_id, location)` to discover existing connections +2. **If connections exist**: Use one with `connection_type: "CLOUD_RESOURCE"` +3. **Only if no connections**: Call `create_connection(project_id, location, connection_id)` + - This automatically grants the Vertex AI User IAM role to the service account ## Project Configuration @@ -154,18 +177,17 @@ ## Workflow Example 1. User asks: "Train a model to predict penguin weight" -2. You call: `activate_skill("bqml")` to load BQML documentation +2. **[AUTOMATIC]** The bqml skill is loaded based on keywords "train", "model", "predict" 3. You follow the skill's examples to CREATE MODEL, EVALUATE, and PREDICT 4. You explain results to the user -5. You call: `deactivate_skill("bqml")` to free up context for next task +5. **[AUTOMATIC]** Skills are cleared after your response ## Guidelines -1. **Activate skills first**: Before complex ML or AI operations, activate the relevant skill +1. **Focus on the task**: Skills load automatically - no need to call activate_skill() 2. **Explore data first**: Use `get_table_info` or `SELECT * LIMIT 5` before complex queries 3. **Use LIMIT**: Prevent large result sets with `LIMIT 10-100` 4. **Explain your steps**: Describe what each query does and interpret results -5. **Deactivate when done**: Free up context by deactivating skills you no longer need ## Quick Reference (without loading skills) @@ -183,7 +205,14 @@ -- Score: AI.SCORE(text, 'criteria', connection_id => 'loc.conn') ``` -For detailed syntax and examples, use `activate_skill("bqml")` or `activate_skill("bq_ai_operator")`. +**Remote Model Quick Start:** +```sql +-- Create: CREATE REMOTE MODEL `project.dataset.model` REMOTE WITH CONNECTION `loc.conn` OPTIONS(ENDPOINT='gemini-2.0-flash') +-- Generate: SELECT * FROM AI.GENERATE_TEXT(MODEL `project.dataset.model`, (SELECT 'prompt' AS prompt), STRUCT(...)) +-- Embed: SELECT * FROM AI.GENERATE_EMBEDDING(MODEL `project.dataset.model`, (SELECT content FROM table), STRUCT(...)) +``` + +For detailed syntax and examples, skills are loaded automatically based on your question! """ @@ -211,20 +240,24 @@ def combined_provider(ctx: ReadonlyContext) -> str: BASE_INSTRUCTION, skill_instruction_provider ) -# Create the root agent with BigQuery tools and ephemeral skill loading +# Create the root agent with BigQuery tools, ephemeral skill loading, and callbacks root_agent = LlmAgent( model="gemini-2.5-pro", name=AGENT_NAME, description=( "Data science agent with BigQuery ML and AI capabilities. " - "Uses Claude Code-style ephemeral skill loading - skills can be activated " - "and deactivated to manage context efficiently." + "Uses callback-based automatic skill activation - skills are loaded based on " + "user input keywords and unloaded after each turn to manage context efficiently." ), instruction=instruction_provider, tools=[ bigquery_toolset, + # Keep manual tools as fallback (agent can still explicitly manage skills) activate_skill_tool, deactivate_skill_tool, list_active_skills_tool, ], + # Callback-based skill management (saves LLM calls) + before_model_callback=skill_callbacks.before_model_callback, + after_agent_callback=skill_callbacks.after_agent_callback, ) diff --git a/contributing/samples/bigquery_skills_demo/skill_callbacks.py b/contributing/samples/bigquery_skills_demo/skill_callbacks.py new file mode 100644 index 0000000000..45bcd397c9 --- /dev/null +++ b/contributing/samples/bigquery_skills_demo/skill_callbacks.py @@ -0,0 +1,460 @@ +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Callback-based Skill Management for ADK Agents. + +This module implements automatic skill activation/deactivation using ADK callbacks, +eliminating the need for explicit LLM tool calls to manage skills. + +Key Features: +1. **Auto-activation via before_model_callback**: Analyzes user input and activates + relevant skills before the LLM processes the request. +2. **Auto-deactivation via after_agent_callback**: Cleans up skills after agent + completes a turn to free context for the next interaction. +3. **Intelligent Detection**: Uses LLM-based classification for semantic understanding + with keyword fallback for reliability. + +Detection Modes: +- "llm": Use LLM classification (most intelligent, understands paraphrases) +- "keyword": Use keyword matching only (fastest, no API calls) +- "hybrid": Try LLM first, fall back to keywords (recommended) + +This approach saves LLM calls compared to having the agent explicitly call +activate_skill/deactivate_skill tools. +""" + +from __future__ import annotations + +import re +from typing import TYPE_CHECKING, Literal + +from .skill_registry import SkillRegistry, ACTIVE_SKILLS_KEY + +if TYPE_CHECKING: + from google.adk.agents.callback_context import CallbackContext + from google.adk.flows.llm_flows.llm_request import LlmRequest + from google.adk.flows.llm_flows.llm_response import LlmResponse + from google.genai import types + +# Detection mode type +DetectionMode = Literal["llm", "keyword", "hybrid"] + + +# Default keyword patterns for auto-activating skills (fallback if not in SKILL.md) +# New skills should define keywords in their SKILL.md frontmatter instead of here +DEFAULT_SKILL_ACTIVATION_PATTERNS: dict[str, list[str]] = { + # These are fallback patterns - skills should define keywords in SKILL.md +} + + +class SkillCallbacks: + """Callback handlers for automatic skill management. + + This class creates callbacks that can be registered with an LlmAgent to + automatically activate and deactivate skills based on user input. + + Supports multiple detection modes: + - "llm": Use LLM-based classification (most intelligent) + - "keyword": Use keyword matching (fastest, no API calls) + - "hybrid": Try LLM first, fall back to keywords (recommended) + + Usage: + registry = SkillRegistry() + skill_callbacks = SkillCallbacks(registry, detection_mode="hybrid") + + agent = LlmAgent( + ... + before_model_callback=skill_callbacks.before_model_callback, + after_agent_callback=skill_callbacks.after_agent_callback, + ) + """ + + def __init__( + self, + registry: SkillRegistry, + auto_deactivate: bool = True, + activation_patterns: dict[str, list[str]] | None = None, + detection_mode: DetectionMode = "hybrid", + ): + """Initialize skill callbacks. + + Args: + registry: The skill registry to load skills from. + auto_deactivate: If True, automatically deactivate all skills after + agent completes a turn. Set to False to keep skills active across + multiple turns. + activation_patterns: Custom keyword patterns for skill activation. + Maps skill name to list of regex patterns. If not provided, + patterns are built dynamically from SKILL.md keywords. + detection_mode: How to detect skills from user input: + - "llm": Use LLM classification (semantic understanding) + - "keyword": Use keyword matching only (fast, no API calls) + - "hybrid": Try LLM first, fall back to keywords (recommended) + """ + self._registry = registry + self._auto_deactivate = auto_deactivate + self._detection_mode = detection_mode + self._classifier = None # Lazy initialization + + # Build patterns from registry keywords (dynamic, scalable) + # Falls back to custom patterns if provided, or default patterns + if activation_patterns is not None: + self._patterns = activation_patterns + else: + self._patterns = self._build_patterns_from_registry() + + # Compile regex patterns for efficiency + self._compiled_patterns: dict[str, list[re.Pattern]] = {} + for skill_name, patterns in self._patterns.items(): + self._compiled_patterns[skill_name] = [ + re.compile(p, re.IGNORECASE) for p in patterns + ] + + def _build_patterns_from_registry(self) -> dict[str, list[str]]: + """Build keyword patterns dynamically from the registry. + + This makes keyword detection scalable - new skills just need to add + keywords to their SKILL.md frontmatter, no code changes needed. + + Returns: + Dict mapping skill names to regex patterns built from keywords. + """ + patterns: dict[str, list[str]] = {} + + # Get all keywords from registry + all_keywords = self._registry.get_all_keywords() + + for skill_name, keywords in all_keywords.items(): + skill_patterns = [] + for keyword in keywords: + # Convert keyword to regex pattern + # Handle multi-word keywords and special characters + escaped = re.escape(keyword) + # Use word boundaries for better matching + # For keywords with dots (like ai.classify), don't add word boundaries + if "." in keyword: + pattern = escaped + else: + pattern = rf"\b{escaped}\b" + skill_patterns.append(pattern) + if skill_patterns: + patterns[skill_name] = skill_patterns + + return patterns + + def _get_classifier(self): + """Lazy initialization of the skill classifier.""" + if self._classifier is None and self._detection_mode in ("llm", "hybrid"): + try: + from .skill_classifier import SkillClassifier + self._classifier = SkillClassifier(self._registry) + except ImportError: + # Fall back to keyword mode if classifier not available + print("[SkillCallbacks] Warning: SkillClassifier not available, using keyword mode") + self._detection_mode = "keyword" + return self._classifier + + def _detect_skills_from_keywords(self, text: str) -> list[str]: + """Detect skills using keyword matching. + + Args: + text: The text to analyze. + + Returns: + List of skill names that matched keywords. + """ + detected_skills = [] + + for skill_name, patterns in self._compiled_patterns.items(): + # Only detect skills that exist in the registry + if skill_name not in self._registry.get_skill_names(): + continue + + for pattern in patterns: + if pattern.search(text): + detected_skills.append(skill_name) + break # One match is enough to activate the skill + + return detected_skills + + def _detect_skills_from_text(self, text: str) -> list[str]: + """Detect which skills should be activated based on text content. + + Uses the configured detection mode (llm, keyword, or hybrid). + + Args: + text: The text to analyze (typically user input). + + Returns: + List of skill names that should be activated. + """ + if self._detection_mode == "keyword": + return self._detect_skills_from_keywords(text) + + elif self._detection_mode == "llm": + classifier = self._get_classifier() + if classifier: + result = classifier.classify(text) + if result.skills: + print(f"[SkillCallbacks] LLM detected: {result.skills} (confidence: {result.confidence:.2f})") + return result.skills + # Fall back to keyword if classifier unavailable + return self._detect_skills_from_keywords(text) + + else: # hybrid mode + classifier = self._get_classifier() + if classifier: + result = classifier.classify_with_fallback( + text, + keyword_detector=self._detect_skills_from_keywords, + ) + if result.skills: + print(f"[SkillCallbacks] Detected: {result.skills} ({result.reasoning})") + return result.skills + # Fall back to keyword if classifier unavailable + return self._detect_skills_from_keywords(text) + + def _get_original_user_message_text(self, llm_request: "LlmRequest") -> str: + """Extract the ORIGINAL user message text from an LLM request. + + This looks for the FIRST user message (not the last), which is the + original user request. In a multi-turn tool-use flow, the conversation + grows but the original user intent is always in the first user message. + + Args: + llm_request: The LLM request containing conversation contents. + + Returns: + The text of the first user message, or empty string if not found. + """ + if not llm_request.contents: + return "" + + # Look for the FIRST user message (original user request) + for content in llm_request.contents: + if content.role == "user" and content.parts: + # Concatenate all text parts + texts = [] + for part in content.parts: + if hasattr(part, "text") and part.text: + texts.append(part.text) + if texts: + return " ".join(texts) + + return "" + + def _get_user_message_text(self, llm_request: "LlmRequest") -> str: + """Extract the latest user message text from an LLM request. + + Args: + llm_request: The LLM request containing conversation contents. + + Returns: + The text of the latest user message, or empty string if not found. + """ + if not llm_request.contents: + return "" + + # Look for the last user message + for content in reversed(llm_request.contents): + if content.role == "user" and content.parts: + # Concatenate all text parts + texts = [] + for part in content.parts: + if hasattr(part, "text") and part.text: + texts.append(part.text) + return " ".join(texts) + + return "" + + def before_model_callback( + self, + callback_context: "CallbackContext", + llm_request: "LlmRequest", + ) -> "LlmResponse | None": + """Auto-activate skills based on user input before LLM processes it. + + This callback analyzes the user message and automatically activates + relevant skills, injecting their documentation into context via the + instruction provider. + + Strategy: + - If NO skills are currently active: Detect from the LATEST user message + (this handles new user requests after skills were cleared) + - If skills ARE already active: Use the ORIGINAL user message to avoid + re-detecting on subsequent LLM calls in the same tool-use flow + + Args: + callback_context: Context for accessing/modifying state. + llm_request: The LLM request (can be modified). + + Returns: + None to proceed with LLM call (we only modify state, not short-circuit). + """ + # Get current active skills + active_skills: list[str] = list( + callback_context.state.get(ACTIVE_SKILLS_KEY, []) + ) + + # Choose which message to analyze based on current skill state + if not active_skills: + # No skills active: This is likely a NEW user request + # Use the LATEST user message to detect skills + user_text = self._get_user_message_text(llm_request) + if not user_text: + return None + + # Detect skills from the latest message + skills_to_activate = self._detect_skills_from_text(user_text) + + if not skills_to_activate: + print(f"[SkillCallbacks] No skills detected from: {user_text[:100]}...") + return None + + # Activate detected skills + callback_context.state[ACTIVE_SKILLS_KEY] = skills_to_activate + print(f"[SkillCallbacks] Detecting skills from: {user_text[:100]}...") + print(f"[SkillCallbacks] Auto-activated skills: {skills_to_activate}") + else: + # Skills already active: This is a subsequent LLM call in the same flow + # Use the ORIGINAL user message to ensure consistency + user_text = self._get_original_user_message_text(llm_request) + if not user_text: + return None + + # Detect which skills should be activated from the original user message + skills_to_activate = self._detect_skills_from_text(user_text) + + # Check if we need to activate any additional skills + newly_activated = [] + for skill_name in skills_to_activate: + if skill_name not in active_skills: + active_skills.append(skill_name) + newly_activated.append(skill_name) + + # Update state if we activated any new skills + if newly_activated: + callback_context.state[ACTIVE_SKILLS_KEY] = active_skills + print(f"[SkillCallbacks] Additional skills detected: {newly_activated}") + + # Return None to proceed with LLM call + # The instruction provider will pick up the activated skills + return None + + def after_agent_callback( + self, + callback_context: "CallbackContext", + ) -> "types.Content | None": + """Auto-deactivate skills after agent completes a turn. + + This callback cleans up active skills to free context for the next + interaction. Skills are ephemeral - they're loaded for a task and + removed when done. + + Args: + callback_context: Context for accessing/modifying state. + + Returns: + None (we don't add any content, just modify state). + """ + if not self._auto_deactivate: + return None + + # Get current active skills + active_skills: list[str] = callback_context.state.get(ACTIVE_SKILLS_KEY, []) + + if active_skills: + # Clear all active skills + callback_context.state[ACTIVE_SKILLS_KEY] = [] + print(f"[SkillCallbacks] Auto-deactivated skills: {active_skills}") + + return None + + +def create_skill_callbacks( + registry: SkillRegistry, + auto_deactivate: bool = True, +) -> SkillCallbacks: + """Factory function to create skill callbacks. + + Args: + registry: The skill registry to use. + auto_deactivate: Whether to auto-deactivate skills after each turn. + + Returns: + SkillCallbacks instance with configured callbacks. + """ + return SkillCallbacks(registry, auto_deactivate=auto_deactivate) + + +# ============================================================================= +# Alternative: Keep skills active until explicitly different task +# ============================================================================= + +class PersistentSkillCallbacks(SkillCallbacks): + """Skill callbacks that keep skills active until task changes. + + This variant doesn't auto-deactivate after each turn. Instead, it only + deactivates skills when the user's request suggests a different task type. + + This is useful for multi-turn conversations about the same topic. + """ + + def __init__(self, registry: SkillRegistry): + super().__init__(registry, auto_deactivate=False) + + def before_model_callback( + self, + callback_context: "CallbackContext", + llm_request: "LlmRequest", + ) -> "LlmResponse | None": + """Auto-activate skills and potentially switch skills if task changes. + + Uses the ORIGINAL user message for skill detection to ensure skills are + activated on the first LLM call, not after tool results come back. + """ + # Extract the ORIGINAL user message (first user message, not last) + user_text = self._get_original_user_message_text(llm_request) + if not user_text: + return None + + # Detect which skills are relevant for this message + relevant_skills = self._detect_skills_from_text(user_text) + + # Get current active skills + active_skills: list[str] = list( + callback_context.state.get(ACTIVE_SKILLS_KEY, []) + ) + + # If we detected new skills, add them + # If we detected different skills (and have some active), switch to them + if relevant_skills: + # Check if this is a task switch (all new skills) + if active_skills and not any(s in active_skills for s in relevant_skills): + # User seems to be switching tasks - deactivate old skills + print(f"[SkillCallbacks] Task switch detected, deactivating: {active_skills}") + active_skills = [] + + # Activate relevant skills + newly_activated = [] + for skill_name in relevant_skills: + if skill_name not in active_skills: + active_skills.append(skill_name) + newly_activated.append(skill_name) + + if newly_activated: + callback_context.state[ACTIVE_SKILLS_KEY] = active_skills + print(f"[SkillCallbacks] Auto-activated skills: {newly_activated}") + + return None diff --git a/contributing/samples/bigquery_skills_demo/skill_classifier.py b/contributing/samples/bigquery_skills_demo/skill_classifier.py new file mode 100644 index 0000000000..f9009c2788 --- /dev/null +++ b/contributing/samples/bigquery_skills_demo/skill_classifier.py @@ -0,0 +1,291 @@ +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Intelligent Skill Classification using LLM. + +This module provides intelligent skill detection using a fast LLM model +to understand user intent and determine which skills are needed. + +Key Features: +1. **LLM-based Classification**: Uses gemini-2.0-flash for fast, accurate intent detection +2. **Semantic Understanding**: Understands paraphrased requests, not just keywords +3. **Confidence Scores**: Returns confidence levels for skill activation decisions +4. **Caching**: Caches recent classifications to avoid redundant API calls +5. **Fallback**: Falls back to keyword matching if LLM is unavailable + +Example use cases that keywords miss but LLM catches: +- "Help me understand patterns in customer behavior" → bqml (clustering) +- "I want to automatically categorize support tickets" → bq_ai_operator +- "Build something to predict next month's sales" → bqml (forecasting) +- "Rate these product descriptions by quality" → bq_ai_operator (AI.SCORE) +""" + +from __future__ import annotations + +import json +import re +from dataclasses import dataclass +from functools import lru_cache +from typing import TYPE_CHECKING + +from google import genai + +from .skill_registry import SkillRegistry + +if TYPE_CHECKING: + pass + + +@dataclass +class SkillClassification: + """Result of skill classification.""" + skills: list[str] # List of skill names to activate + confidence: float # 0.0 to 1.0 + reasoning: str # Brief explanation of why these skills were selected + + +# Classification prompt template - DYNAMIC version +# Skills are injected at runtime from the registry +CLASSIFICATION_PROMPT_TEMPLATE = """\ +You are a skill classifier for a BigQuery data science agent. Your job is to determine which specialized skills (if any) should be loaded based on the user's request. + +Available skills: +{skills_description} + +Analyze the user's request and determine which skills are needed. + +User request: "{user_input}" + +Respond with a JSON object (no markdown, just raw JSON): +{{ + "skills": ["skill_name1", "skill_name2"], + "confidence": 0.85, + "reasoning": "Brief explanation" +}} + +Rules: +- Return empty skills [] if the request is just a simple query (SELECT, list tables, etc.) +- Only return skills from the available list above +- Confidence should be 0.0-1.0 based on how certain you are +- Be conservative: only activate skills when clearly needed +""" + + +class SkillClassifier: + """Intelligent skill classifier using LLM. + + This classifier uses a fast LLM model to understand user intent + and determine which skills should be activated. + + SCALABILITY: Skills are dynamically loaded from the registry, so adding + new skills only requires creating a SKILL.md file - no code changes needed. + """ + + def __init__( + self, + registry: SkillRegistry, + model: str = "gemini-2.0-flash", + confidence_threshold: float = 0.6, + cache_size: int = 100, + ): + """Initialize the skill classifier. + + Args: + registry: The skill registry to validate skill names against. + model: The LLM model to use for classification. + confidence_threshold: Minimum confidence to activate a skill. + cache_size: Number of recent classifications to cache. + """ + self._registry = registry + self._model = model + self._confidence_threshold = confidence_threshold + self._cache_size = cache_size + self._client: genai.Client | None = None + + # Simple in-memory cache for recent classifications + self._cache: dict[str, SkillClassification] = {} + + # Build skills description from registry (dynamic, not hardcoded) + self._skills_description = self._build_skills_description() + + def _build_skills_description(self) -> str: + """Build skills description dynamically from the registry. + + This makes the classifier scalable - new skills are automatically + included without code changes. + """ + descriptions = [] + for i, metadata in enumerate(self._registry.get_all_metadata(), 1): + descriptions.append(f"{i}. **{metadata.name}** - {metadata.description}") + return "\n".join(descriptions) if descriptions else "No skills available." + + def _build_prompt(self, user_input: str) -> str: + """Build the classification prompt with dynamic skills.""" + return CLASSIFICATION_PROMPT_TEMPLATE.format( + skills_description=self._skills_description, + user_input=user_input, + ) + + def _get_client(self) -> genai.Client: + """Get or create the genai client.""" + if self._client is None: + self._client = genai.Client() + return self._client + + def _normalize_input(self, text: str) -> str: + """Normalize input text for caching.""" + # Lowercase and remove extra whitespace + return " ".join(text.lower().split()) + + def _parse_response(self, response_text: str) -> SkillClassification | None: + """Parse the LLM response into a SkillClassification.""" + try: + # Try to extract JSON from the response + # Handle cases where LLM might wrap in markdown code blocks + json_match = re.search(r'\{[^{}]*\}', response_text, re.DOTALL) + if not json_match: + return None + + data = json.loads(json_match.group()) + + # Validate and filter skills + valid_skill_names = set(self._registry.get_skill_names()) + skills = [s for s in data.get("skills", []) if s in valid_skill_names] + + return SkillClassification( + skills=skills, + confidence=float(data.get("confidence", 0.5)), + reasoning=data.get("reasoning", ""), + ) + except (json.JSONDecodeError, KeyError, ValueError): + return None + + def classify(self, user_input: str) -> SkillClassification: + """Classify user input to determine which skills are needed. + + Args: + user_input: The user's message/request. + + Returns: + SkillClassification with recommended skills and confidence. + """ + # Check cache first + cache_key = self._normalize_input(user_input) + if cache_key in self._cache: + return self._cache[cache_key] + + try: + # Call the LLM + client = self._get_client() + prompt = self._build_prompt(user_input) + + response = client.models.generate_content( + model=self._model, + contents=prompt, + config={ + "temperature": 0.1, # Low temperature for consistent classification + "max_output_tokens": 256, + }, + ) + + # Parse the response + result = self._parse_response(response.text) + + if result is None: + # Fallback to empty classification + result = SkillClassification( + skills=[], + confidence=0.0, + reasoning="Failed to parse LLM response", + ) + + # Apply confidence threshold + if result.confidence < self._confidence_threshold: + result = SkillClassification( + skills=[], + confidence=result.confidence, + reasoning=f"Below confidence threshold ({self._confidence_threshold}): {result.reasoning}", + ) + + # Cache the result + if len(self._cache) >= self._cache_size: + # Simple cache eviction: remove first item + first_key = next(iter(self._cache)) + del self._cache[first_key] + self._cache[cache_key] = result + + return result + + except Exception as e: + # Return empty classification on error + return SkillClassification( + skills=[], + confidence=0.0, + reasoning=f"Classification error: {str(e)}", + ) + + def classify_with_fallback( + self, + user_input: str, + keyword_detector: callable | None = None, + ) -> SkillClassification: + """Classify with keyword fallback if LLM fails. + + Args: + user_input: The user's message/request. + keyword_detector: Optional function that returns skill names from text. + + Returns: + SkillClassification with recommended skills. + """ + # Try LLM classification first + result = self.classify(user_input) + + # If LLM returned skills with good confidence, use that + if result.skills and result.confidence >= self._confidence_threshold: + return result + + # Fall back to keyword detection if provided + if keyword_detector is not None: + keyword_skills = keyword_detector(user_input) + if keyword_skills: + return SkillClassification( + skills=keyword_skills, + confidence=0.7, # Medium confidence for keyword match + reasoning="Detected via keyword matching (LLM uncertain)", + ) + + return result + + +# Convenience function for quick classification +def classify_skills( + user_input: str, + registry: SkillRegistry | None = None, +) -> list[str]: + """Quick function to classify skills for a user input. + + Args: + user_input: The user's message. + registry: Optional skill registry (uses default if not provided). + + Returns: + List of skill names to activate. + """ + if registry is None: + registry = SkillRegistry() + + classifier = SkillClassifier(registry) + result = classifier.classify(user_input) + return result.skills diff --git a/contributing/samples/bigquery_skills_demo/skill_registry.py b/contributing/samples/bigquery_skills_demo/skill_registry.py index 3aa81e6553..c74a6598c4 100644 --- a/contributing/samples/bigquery_skills_demo/skill_registry.py +++ b/contributing/samples/bigquery_skills_demo/skill_registry.py @@ -48,10 +48,11 @@ @dataclass class SkillMetadata: - """Metadata for a skill (name + description only).""" + """Metadata for a skill (name, description, and keywords).""" name: str description: str path: Path + keywords: list[str] | None = None # Keywords for dynamic pattern matching @dataclass @@ -153,6 +154,7 @@ def _parse_skill_metadata(self, skill_path: Path) -> SkillMetadata | None: name=frontmatter['name'], description=frontmatter.get('description', ''), path=skill_path, + keywords=frontmatter.get('keywords'), # Parse keywords from frontmatter ) except Exception: return None @@ -169,6 +171,21 @@ def get_all_metadata(self) -> list[SkillMetadata]: """Get metadata for all discovered skills.""" return list(self._skills.values()) + def get_all_keywords(self) -> dict[str, list[str]]: + """Get all keywords for all skills. + + Returns a mapping of skill name to list of keywords. + This is used for dynamic keyword pattern matching. + + Returns: + Dict mapping skill names to their keyword lists. + """ + return { + name: metadata.keywords or [] + for name, metadata in self._skills.items() + if metadata.keywords + } + def load_skill_content(self, name: str) -> SkillContent | None: """Load the full content of a skill. diff --git a/contributing/samples/bigquery_skills_demo/skills/bq_ai_operator/SKILL.md b/contributing/samples/bigquery_skills_demo/skills/bq_ai_operator/SKILL.md index 380dfb5fc2..e5440ca822 100644 --- a/contributing/samples/bigquery_skills_demo/skills/bq_ai_operator/SKILL.md +++ b/contributing/samples/bigquery_skills_demo/skills/bq_ai_operator/SKILL.md @@ -1,6 +1,31 @@ --- name: bq_ai_operator description: BigQuery AI Operator - Use managed AI functions (AI.CLASSIFY, AI.IF, AI.SCORE) directly in SQL for text classification, filtering, and scoring. Requires a BigQuery connection to Vertex AI. +keywords: + - ai.classify + - ai.if + - ai.score + - ai function + - ai operator + - text classification + - sentiment + - categorize + - categories + - natural language + - filter text + - score text + - score + - rate content + - rank content + - rate + - rank + - classify + - positive + - negative + - vertex ai + - managed ai + - list connections + - connection_id --- # BQ AI Operator Skill (Managed AI Functions in SQL) @@ -11,14 +36,82 @@ Use managed AI functions directly in BigQuery SQL queries for text classificatio ## Prerequisites -1. **Create a BigQuery connection to Vertex AI** (required for all AI functions): +1. **A BigQuery connection to Vertex AI is required** for all AI functions. + +2. **Grant the connection service account access to Vertex AI** + +## Connection Workflow (ALWAYS Follow This) + +**CRITICAL**: AI functions require a `connection_id` to a BigQuery connection to Vertex AI. + +### ⚠️ IMPORTANT: Location Matching Rule + +**The connection location MUST match your dataset location!** + +| Dataset Location | Connection Location | Example | +|------------------|---------------------|---------| +| `US` (multi-region) | `us` | `us.my_ai_connection` | +| `EU` (multi-region) | `eu` | `eu.my_ai_connection` | +| `us-central1` (regional) | `us-central1` | `us-central1.my_ai_connection` | + +**Common Error**: Using `us-central1.my_connection` with a dataset in `US` multi-region will fail with "Dataset not found in location us-central1". + +**How to check dataset location**: ```sql --- Create a connection (run once) -CREATE CLOUD RESOURCE CONNECTION `us.my_ai_connection` -OPTIONS(location='us'); +SELECT option_value FROM `project.dataset.INFORMATION_SCHEMA.SCHEMATA_OPTIONS` WHERE option_name = 'location' ``` -2. **Grant the connection service account access to Vertex AI** +### Step 1: Determine Your Dataset Location + +Before listing connections, identify where your target dataset is located: +- Most BigQuery public datasets are in `US` multi-region +- Your own datasets might be in `US`, `EU`, or a specific region like `us-central1` + +### Step 2: List Connections in the SAME Location + +Use the `list_connections` tool with the **same location as your dataset**: + +``` +# For datasets in US multi-region: +list_connections(project_id="your-project", location="us") + +# For datasets in us-central1: +list_connections(project_id="your-project", location="us-central1") +``` + +This returns all available connections with their `connection_id` and `service_account`. + +### Step 3: Use an Existing Connection If Available + +If `list_connections` returns connections, **use one of them**. Pick a connection that: +- Has `connection_type: "CLOUD_RESOURCE"` (required for Vertex AI) +- Is in the **SAME location as your dataset** + +Use the `connection_id` from the result, formatted as `location.connection_id`: +- Example: If connection_id is `my_ai_connection` in location `us`, use `us.my_ai_connection` + +### Step 4: Only Create a New Connection If None Exist + +**Only if `list_connections` returns empty or no suitable connections**, create a new one in the **same location as your dataset**: + +``` +# For US multi-region datasets: +create_connection(project_id="your-project", location="us", connection_id="my_ai_connection") + +# For us-central1 datasets: +create_connection(project_id="your-project", location="us-central1", connection_id="my_ai_connection") +``` + +This automatically: +1. Creates the connection +2. Grants the Vertex AI User role to the service account (required for AI functions) + +### Connection ID Formats + +When using connections in SQL: +- `us.my_connection` (location.connection_name) - **Preferred for US multi-region** +- `us-central1.my_connection` - **For regional datasets** +- `project_id.us.my_connection` (fully qualified) ## Available Managed AI Functions @@ -123,17 +216,20 @@ LIMIT 10; ## AI.SCORE - Quality Scoring -Returns a score between 0.0 and 1.0 based on criteria. +Returns a FLOAT64 score based on your scoring criteria. Commonly used with ORDER BY for ranking. ### Syntax ```sql AI.SCORE( - input, -- STRING: the text to score - criteria, -- STRING: scoring criteria + (prompt_with_criteria, column_to_score), -- TUPLE: (STRING literal, column reference) connection_id => 'LOCATION.CONNECTION_NAME' ) ``` +**CRITICAL**: The first argument is a **TUPLE** with parentheses containing: +1. A STRING literal describing the scoring criteria +2. A column reference to the text being scored + ### Examples **Review helpfulness scoring:** @@ -143,8 +239,7 @@ SELECT review_text, star_rating, AI.SCORE( - review_text, - 'Rate this review helpfulness based on: detail level, specific examples, balanced perspective', + ('Rate the helpfulness of this review based on detail level and examples. Review: ', review_text), connection_id => 'us.my_ai_connection' -- Replace with your connection ) AS helpfulness_score FROM `project.reviews.product_reviews` @@ -152,14 +247,42 @@ ORDER BY helpfulness_score DESC LIMIT 10; ``` +**Movie review rating (from official docs):** +```sql +SELECT + AI.SCORE(( + 'On a scale from 1 to 10, rate how much the reviewer liked the movie. Review: ', + review), + connection_id => 'us.my_ai_connection' -- Replace with your connection + ) AS ai_rating, + reviewer_rating AS human_rating, + review +FROM `bigquery-public-data.imdb.reviews` +WHERE title = 'The English Patient' +ORDER BY ai_rating DESC +LIMIT 10; +``` + +**Negativity scoring:** +```sql +SELECT + review, + AI.SCORE( + ('Rate negativity from 1-10: ', review), + connection_id => 'us.my_ai_connection' -- Replace with your connection + ) AS negativity_score +FROM product_reviews +ORDER BY negativity_score DESC +LIMIT 5; +``` + **Relevance scoring:** ```sql SELECT document_id, title, AI.SCORE( - content, - 'How relevant is this document to machine learning and AI topics', + ('How relevant is this document to machine learning and AI topics? Document: ', content), connection_id => 'us.my_ai_connection' -- Replace with your connection ) AS ml_relevance FROM `project.docs.articles` @@ -186,8 +309,7 @@ WITH classified AS ( connection_id => 'us.my_ai_connection' -- Replace with your connection ) AS sentiment, AI.SCORE( - review_text, - 'Review quality based on detail and helpfulness', + ('Rate review quality based on detail and helpfulness. Review: ', review_text), connection_id => 'us.my_ai_connection' -- Replace with your connection ) AS quality_score FROM `project.reviews.raw_reviews` @@ -221,6 +343,9 @@ ORDER BY quality_score DESC; 3. **Region Support**: Works in all Gemini regions plus US/EU multi-regions 4. **Use LIMIT**: Always use LIMIT to control costs when testing 5. **String Return**: AI.CLASSIFY returns STRING, AI.IF returns BOOL, AI.SCORE returns FLOAT64 +6. **Escape Single Quotes**: When using string literals with apostrophes, escape them by doubling: + - WRONG: `'The surgeon who 'sees' inside patients'` + - CORRECT: `'The surgeon who ''sees'' inside patients'` ## Troubleshooting diff --git a/contributing/samples/bigquery_skills_demo/skills/bq_remote_model/SKILL.md b/contributing/samples/bigquery_skills_demo/skills/bq_remote_model/SKILL.md new file mode 100644 index 0000000000..98d5ff3676 --- /dev/null +++ b/contributing/samples/bigquery_skills_demo/skills/bq_remote_model/SKILL.md @@ -0,0 +1,429 @@ +--- +name: bq_remote_model +description: BigQuery Remote Models - Create remote models connecting to Vertex AI endpoints for text generation (Gemini, Claude, Llama), embeddings, and custom deployed models. Use AI.GENERATE_TEXT and AI.GENERATE_EMBEDDING functions. +keywords: + - remote model + - create remote model + - vertex ai + - generate text + - ai.generate_text + - generate embedding + - ai.generate_embedding + - gemini + - claude + - llama + - text generation + - embeddings + - llm + - foundation model + - hugging face + - model garden + - endpoint + - list connections + - connection_id +--- + +# BQ Remote Model Skill (Remote Models with Vertex AI) + +Create and use remote models that connect BigQuery to Vertex AI endpoints for text generation, embeddings, and custom ML models. + +**IMPORTANT**: Remote models require a BigQuery connection to Vertex AI. This is different from BQML (which trains models in BigQuery) and BQ AI Operator (which uses managed AI functions like AI.CLASSIFY). + +## Prerequisites + +1. **A BigQuery connection to Vertex AI is required** for all remote models. + +2. **Grant the connection's service account the Vertex AI User role** + +## Connection Workflow (ALWAYS Follow This) + +**CRITICAL**: Remote models require a `connection_id` to a BigQuery connection to Vertex AI. + +### ⚠️ IMPORTANT: Location Matching Rule + +**The connection location MUST match your dataset location!** + +| Dataset Location | Connection Location | Example | +|------------------|---------------------|---------| +| `US` (multi-region) | `us` | `us.my_vertex_connection` | +| `EU` (multi-region) | `eu` | `eu.my_vertex_connection` | +| `us-central1` (regional) | `us-central1` | `us-central1.my_vertex_connection` | + +**Common Error**: Using `us-central1.my_connection` with a dataset in `US` multi-region will fail with "Dataset not found in location us-central1". + +**How to check dataset location**: +```sql +SELECT option_value FROM `project.dataset.INFORMATION_SCHEMA.SCHEMATA_OPTIONS` WHERE option_name = 'location' +``` + +### Step 1: Determine Your Dataset Location + +Before listing connections, identify where your target dataset is located: +- Most BigQuery public datasets are in `US` multi-region +- Your own datasets might be in `US`, `EU`, or a specific region like `us-central1` + +### Step 2: List Connections in the SAME Location + +Use the `list_connections` tool with the **same location as your dataset**: + +``` +# For datasets in US multi-region: +list_connections(project_id="your-project", location="us") + +# For datasets in us-central1: +list_connections(project_id="your-project", location="us-central1") +``` + +This returns all available connections with their `connection_id` and `service_account`. + +### Step 3: Use an Existing Connection If Available + +If `list_connections` returns connections, **use one of them**. Pick a connection that: +- Has `connection_type: "CLOUD_RESOURCE"` (required for Vertex AI) +- Is in the **SAME location as your dataset** + +Use the `connection_id` from the result, formatted as `location.connection_id`: +- Example: If connection_id is `my_ai_connection` in location `us`, use `us.my_ai_connection` + +### Step 4: Only Create a New Connection If None Exist + +**Only if `list_connections` returns empty or no suitable connections**, create a new one in the **same location as your dataset**: + +``` +# For US multi-region datasets: +create_connection(project_id="your-project", location="us", connection_id="my_vertex_connection") + +# For us-central1 datasets: +create_connection(project_id="your-project", location="us-central1", connection_id="my_vertex_connection") +``` + +This automatically: +1. Creates the connection +2. Grants the Vertex AI User role to the service account (required for remote models) + +### Connection ID Formats + +When using connections in SQL: +- `us.my_connection` (location.connection_name) - **Preferred** +- `project_id.us.my_connection` (fully qualified) + +--- + +## CREATE REMOTE MODEL Syntax + +### For Google/Partner Models (Gemini, Claude) + +```sql +CREATE OR REPLACE MODEL `project.dataset.model_name` +REMOTE WITH CONNECTION `project.region.connection_id` +OPTIONS (ENDPOINT = 'endpoint_name'); +``` + +### ⚠️ DEFAULT MODEL: Always Use Gemini 2.5 Pro + +**ALWAYS use `gemini-2.5-pro` as the default model** unless the user specifically requests a different model. + +```sql +-- RECOMMENDED: Use gemini-2.5-pro by default +CREATE OR REPLACE MODEL `project.dataset.gemini_model` +REMOTE WITH CONNECTION `us.my_connection` +OPTIONS (ENDPOINT = 'gemini-2.5-pro'); +``` + +**Common ENDPOINT values:** +| Model | Endpoint | When to Use | +|-------|----------|-------------| +| **Gemini 2.5 Pro** | `gemini-2.5-pro` | **DEFAULT** - Best quality, use for all tasks unless specified otherwise | +| Gemini 2.5 Flash | `gemini-2.5-flash` | Only if user requests faster/cheaper processing | +| Claude 3.5 Sonnet | `claude-3-5-sonnet@20240620` | Only if user specifically requests Claude | +| Text Embedding | `text-embedding-004` | For embeddings/vector search | +| Gemini Embedding | `gemini-embedding-001` | For embeddings (larger dimension) | + +**Legacy models (avoid):** `gemini-2.0-flash`, `gemini-1.5-pro` - Use 2.5 versions instead. + +### For Open Models (Hugging Face / Model Garden) + +```sql +CREATE OR REPLACE MODEL `project.dataset.model_name` +REMOTE WITH CONNECTION `project.region.connection_id` +OPTIONS ( + HUGGING_FACE_MODEL_ID = 'meta-llama/Llama-2-7b-chat-hf', + HUGGING_FACE_TOKEN = 'your_token', -- Optional, for gated models + MACHINE_TYPE = 'n1-standard-4', + MIN_REPLICA_COUNT = 1, + MAX_REPLICA_COUNT = 3, + ENDPOINT_IDLE_TTL = INTERVAL 1 HOUR +); +``` + +--- + +## AI.GENERATE_TEXT - Text Generation + +Generate text using LLMs like Gemini, Claude, or Llama. + +### Basic Syntax + +```sql +SELECT * +FROM AI.GENERATE_TEXT( + MODEL `project.dataset.model_name`, + (SELECT 'Your prompt here' AS prompt), + STRUCT( + 1024 AS max_output_tokens, + 0.7 AS temperature, + 0.95 AS top_p + ) +); +``` + +### ⚠️ CRITICAL: Task-Specific Parameter Settings + +**The `max_output_tokens` parameter is crucial** - set it appropriately for the task type: + +| Task Type | max_output_tokens | temperature | Example Use Case | +|-----------|-------------------|-------------|------------------| +| **Summarization** | `512-2048` | `0.2-0.4` | Summarize articles, extract key points | +| **Long-form generation** | `2048-8192` | `0.5-0.7` | Write essays, detailed explanations | +| **Classification/Labeling** | `50-100` | `0.0-0.2` | Classify text, extract labels | +| **Short answers** | `100-256` | `0.2-0.3` | Q&A, simple extractions | +| **Creative writing** | `1024-4096` | `0.7-0.9` | Stories, creative content | + +**Guidelines:** +- **Summarization tasks**: Use `max_output_tokens` of `512-1024` for single-document summaries, `1024-2048` for multi-document summaries +- **Classification tasks**: Use small `max_output_tokens` (50-100) since output is typically a single word or short phrase +- **Low temperature (0.0-0.3)**: For factual, deterministic outputs (classification, extraction) +- **High temperature (0.5-0.8)**: For creative, varied outputs (writing, brainstorming) + +### Parameters for Gemini Models + +| Parameter | Type | Range | Default | Description | +|-----------|------|-------|---------|-------------| +| `max_output_tokens` | INT64 | 1-8192 | 128 | **IMPORTANT**: Set based on task (see table above) | +| `temperature` | FLOAT64 | 0.0-1.0 | 0.0 | Randomness (0=deterministic, 1=creative) | +| `top_p` | FLOAT64 | 0.0-1.0 | 0.95 | Nucleus sampling threshold | +| `stop_sequences` | ARRAY | - | [] | Stop generation at these sequences | +| `ground_with_google_search` | BOOL | - | FALSE | Enable Google Search grounding | +| `request_type` | STRING | DEDICATED/SHARED | UNSPECIFIED | Resource allocation | + +### Parameters for Claude Models + +| Parameter | Type | Range | Default | +|-----------|------|-------|---------| +| `max_output_tokens` | INT64 | 1-4096 | 128 | +| `top_k` | INT64 | 1-40 | - | +| `top_p` | FLOAT64 | 0.0-1.0 | - | + +### Example: Text Summarization (Large max_output_tokens) + +```sql +-- Step 1: Create the remote model (ALWAYS use gemini-2.5-pro by default) +CREATE OR REPLACE MODEL `project.bq_demo.gemini_model` +REMOTE WITH CONNECTION `us.my_vertex_connection` +OPTIONS (ENDPOINT = 'gemini-2.5-pro'); -- DEFAULT: Always use 2.5-pro unless specified + +-- Step 2: Summarize text (use 512-1024 tokens for summaries) +SELECT + title, + ml_generate_text_result AS summary +FROM AI.GENERATE_TEXT( + MODEL `project.bq_demo.gemini_model`, + (SELECT + title, + CONCAT('Summarize this article in 2-3 paragraphs:\n\n', body) AS prompt + FROM `bigquery-public-data.bbc_news.fulltext` + LIMIT 5), + STRUCT( + 1024 AS max_output_tokens, -- LARGE for summarization + 0.3 AS temperature -- Low for factual output + ) +); +``` + +### Example: Text Classification (Small max_output_tokens) + +```sql +-- Classification task: Use small max_output_tokens since output is just a label +SELECT + review_id, + review_text, + ml_generate_text_result AS sentiment +FROM AI.GENERATE_TEXT( + MODEL `project.bq_demo.gemini_model`, + (SELECT + review_id, + review_text, + CONCAT('Classify the sentiment of this review as POSITIVE, NEGATIVE, or NEUTRAL. Only output the label, nothing else.\n\nReview: ', review_text) AS prompt + FROM `project.dataset.reviews` + LIMIT 10), + STRUCT( + 50 AS max_output_tokens, -- SMALL for classification (just a single word) + 0.0 AS temperature -- Zero for deterministic output + ) +); +``` + +### Example: Batch Summarization from Table + +```sql +SELECT + review_id, + review_text, + ml_generate_text_result AS summary +FROM AI.GENERATE_TEXT( + MODEL `project.bq_demo.gemini_model`, + (SELECT + review_id, + review_text, + CONCAT('Summarize this review in one sentence: ', review_text) AS prompt + FROM `project.dataset.reviews` + LIMIT 10), + STRUCT( + 256 AS max_output_tokens, -- Medium for short summaries + 0.2 AS temperature + ) +); +``` + +--- + +## AI.GENERATE_EMBEDDING - Text Embeddings + +Generate vector embeddings for text, useful for semantic search and similarity. + +### Syntax + +```sql +SELECT * +FROM AI.GENERATE_EMBEDDING( + MODEL `project.dataset.embedding_model`, + (SELECT content FROM table), + STRUCT( + 'RETRIEVAL_DOCUMENT' AS task_type, + 768 AS output_dimensionality + ) +); +``` + +### Task Types + +| Task Type | Description | Use Case | +|-----------|-------------|----------| +| `RETRIEVAL_QUERY` | Optimize for queries | Search queries | +| `RETRIEVAL_DOCUMENT` | Optimize for documents | Document indexing | +| `SEMANTIC_SIMILARITY` | Compute similarity | Finding similar texts | +| `CLASSIFICATION` | Text classification | Categorization | +| `CLUSTERING` | Group similar texts | Topic modeling | +| `QUESTION_ANSWERING` | Q&A tasks | FAQ systems | +| `FACT_VERIFICATION` | Verify facts | Fact checking | +| `CODE_RETRIEVAL_QUERY` | Code search | Code similarity | + +### Dimensionality + +| Model | Dimension Range | Default | +|-------|-----------------|---------| +| `gemini-embedding-001` | 1-3072 | 3072 | +| `text-embedding-004` | 1-768 | 768 | + +### Example: Create Embeddings for Semantic Search + +```sql +-- Step 1: Create embedding model +CREATE OR REPLACE MODEL `project.bq_demo.embedding_model` +REMOTE WITH CONNECTION `us.my_vertex_connection` +OPTIONS (ENDPOINT = 'text-embedding-004'); + +-- Step 2: Generate embeddings for documents +SELECT + doc_id, + title, + ml_generate_embedding_result AS embedding +FROM AI.GENERATE_EMBEDDING( + MODEL `project.bq_demo.embedding_model`, + (SELECT doc_id, title, content FROM `project.dataset.documents` LIMIT 100), + STRUCT('RETRIEVAL_DOCUMENT' AS task_type, 768 AS output_dimensionality) +); +``` + +### Example: Vector Similarity Search + +```sql +-- Find similar documents using cosine distance +WITH query_embedding AS ( + SELECT ml_generate_embedding_result AS embedding + FROM AI.GENERATE_EMBEDDING( + MODEL `project.bq_demo.embedding_model`, + (SELECT 'machine learning best practices' AS content), + STRUCT('RETRIEVAL_QUERY' AS task_type) + ) +) +SELECT + d.doc_id, + d.title, + ML.DISTANCE(d.embedding, q.embedding, 'COSINE') AS similarity +FROM `project.dataset.doc_embeddings` d +CROSS JOIN query_embedding q +ORDER BY similarity ASC +LIMIT 10; +``` + +--- + +## Complete Pipeline Example + +Build a RAG (Retrieval Augmented Generation) pipeline: + +```sql +-- Step 1: Find relevant documents using embeddings +WITH relevant_docs AS ( + SELECT title, content + FROM AI.GENERATE_EMBEDDING( + MODEL `project.bq_demo.embedding_model`, + (SELECT 'What are the benefits of serverless?' AS content), + STRUCT('RETRIEVAL_QUERY' AS task_type) + ) query + CROSS JOIN ( + SELECT doc_id, title, content, embedding + FROM `project.dataset.doc_embeddings` + ) docs + ORDER BY ML.DISTANCE(docs.embedding, query.ml_generate_embedding_result, 'COSINE') + LIMIT 3 +) +-- Step 2: Generate response using retrieved context +SELECT ml_generate_text_result AS answer +FROM AI.GENERATE_TEXT( + MODEL `project.bq_demo.gemini_model`, + (SELECT CONCAT( + 'Based on these documents:\n', + STRING_AGG(content, '\n\n'), + '\n\nAnswer: What are the benefits of serverless?' + ) AS prompt FROM relevant_docs), + STRUCT(512 AS max_output_tokens, 0.3 AS temperature) +); +``` + +--- + +## Important Notes + +1. **Connection Required**: All remote models need a Vertex AI connection +2. **Region Matching**: Dataset and connection must be in the same region +3. **Cost**: Remote model calls incur Vertex AI API costs +4. **Rate Limits**: Be mindful of Vertex AI quotas when processing large datasets +5. **Use LIMIT**: Always use LIMIT when testing to control costs +6. **Escape Single Quotes**: When using string literals with apostrophes, escape them by doubling: + - WRONG: `'The surgeon who 'sees' inside patients'` + - CORRECT: `'The surgeon who ''sees'' inside patients'` + +## Troubleshooting + +**Error: "connection not found"** +- Verify connection exists: `SELECT * FROM region-us.INFORMATION_SCHEMA.CONNECTIONS` +- Use correct format: `project.region.connection_id` + +**Error: "model not found"** +- Check endpoint spelling matches exactly +- Verify model is available in your region + +**Error: "permission denied"** +- Grant Vertex AI User role to connection's service account diff --git a/contributing/samples/bigquery_skills_demo/skills/bqml/SKILL.md b/contributing/samples/bigquery_skills_demo/skills/bqml/SKILL.md index 5ec1e995d0..40d5bfb61f 100644 --- a/contributing/samples/bigquery_skills_demo/skills/bqml/SKILL.md +++ b/contributing/samples/bigquery_skills_demo/skills/bqml/SKILL.md @@ -1,6 +1,23 @@ --- name: bqml description: BigQuery ML - Train, evaluate, and deploy machine learning models using SQL. Supports regression, classification, clustering, time series forecasting, and deep learning. +keywords: + - ml + - machine learning + - train + - model + - predict + - cluster + - kmeans + - regression + - classification + - forecast + - arima + - xgboost + - boosted + - random forest + - feature importance + - evaluate --- # BQML Skill (BigQuery Machine Learning) diff --git a/src/google/adk/tools/bigquery/__init__.py b/src/google/adk/tools/bigquery/__init__.py index 9e6b1166b0..1b80ed2b91 100644 --- a/src/google/adk/tools/bigquery/__init__.py +++ b/src/google/adk/tools/bigquery/__init__.py @@ -25,12 +25,20 @@ etc. 4. We want to provide extra access guardrails in those tools. For example, execute_sql can't arbitrarily mutate existing data. + +Skills are also provided for programmatic tool calling (PTC): +- BQMLSkill: Machine learning with BigQuery ML +- BQAIOperatorSkill: AI functions (AI.GENERATE, AI.CLASSIFY, etc.) """ from .bigquery_credentials import BigQueryCredentialsConfig from .bigquery_toolset import BigQueryToolset +from .skills import BQAIOperatorSkill +from .skills import BQMLSkill __all__ = [ "BigQueryToolset", "BigQueryCredentialsConfig", + "BQMLSkill", + "BQAIOperatorSkill", ] diff --git a/src/google/adk/tools/bigquery/bigquery_toolset.py b/src/google/adk/tools/bigquery/bigquery_toolset.py index f38bf95f62..2640133a1a 100644 --- a/src/google/adk/tools/bigquery/bigquery_toolset.py +++ b/src/google/adk/tools/bigquery/bigquery_toolset.py @@ -81,6 +81,8 @@ async def get_tools( metadata_tool.list_dataset_ids, metadata_tool.list_table_ids, metadata_tool.get_job_info, + metadata_tool.list_connections, + metadata_tool.create_connection, query_tool.get_execute_sql(self._tool_settings), query_tool.forecast, query_tool.analyze_contribution, diff --git a/src/google/adk/tools/bigquery/metadata_tool.py b/src/google/adk/tools/bigquery/metadata_tool.py index af50f54f3f..575d92bc55 100644 --- a/src/google/adk/tools/bigquery/metadata_tool.py +++ b/src/google/adk/tools/bigquery/metadata_tool.py @@ -299,6 +299,262 @@ def get_table_info( } +def list_connections( + project_id: str, + location: str, + credentials: Credentials, + settings: BigQueryToolConfig, +) -> list[dict]: + """List BigQuery connections in a Google Cloud project and location. + + BigQuery connections are used to connect to external data sources like + Vertex AI for AI functions (AI.CLASSIFY, AI.IF, AI.SCORE) and remote + models (AI.GENERATE_TEXT, AI.GENERATE_EMBEDDING). + + Args: + project_id (str): The Google Cloud project id. + location (str): The location/region (e.g., 'us', 'eu', 'us-central1'). + credentials (Credentials): The credentials to use for the request. + settings (BigQueryToolConfig): The BigQuery tool settings. + + Returns: + list[dict]: List of connections with their properties. + + Examples: + >>> list_connections("my-project", "us") + [ + { + "name": "projects/my-project/locations/us/connections/my_ai_connection", + "connection_id": "my_ai_connection", + "location": "us", + "connection_type": "CLOUD_RESOURCE", + "friendly_name": "My AI Connection", + "description": "Connection for Vertex AI" + } + ] + """ + try: + from google.cloud import bigquery_connection_v1 + + connection_client = bigquery_connection_v1.ConnectionServiceClient( + credentials=credentials + ) + parent = f"projects/{project_id}/locations/{location}" + + connections = [] + for conn in connection_client.list_connections(parent=parent): + # Extract connection_id from the full name + # Format: projects/{project}/locations/{location}/connections/{connection_id} + parts = conn.name.split("/") + connection_id = parts[-1] if parts else conn.name + + conn_info = { + "name": conn.name, + "connection_id": connection_id, + "location": location, + "friendly_name": conn.friendly_name or "", + "description": conn.description or "", + } + + # Add connection type based on which field is set + if conn.cloud_resource: + conn_info["connection_type"] = "CLOUD_RESOURCE" + if conn.cloud_resource.service_account_id: + conn_info["service_account"] = conn.cloud_resource.service_account_id + elif conn.cloud_sql: + conn_info["connection_type"] = "CLOUD_SQL" + elif conn.spark: + conn_info["connection_type"] = "SPARK" + else: + conn_info["connection_type"] = "UNKNOWN" + + connections.append(conn_info) + + return connections + except Exception as ex: + return { + "status": "ERROR", + "error_details": str(ex), + } + + +def _grant_vertex_ai_role( + project_id: str, + service_account: str, + credentials: Credentials, +) -> dict: + """Grant the Vertex AI User role to a service account. + + Args: + project_id (str): The Google Cloud project id. + service_account (str): The service account email to grant the role to. + credentials (Credentials): The credentials to use for the request. + + Returns: + dict: Status of the IAM operation. + """ + try: + from google.cloud import resourcemanager_v3 + from google.iam.v1 import iam_policy_pb2, policy_pb2 + + # Create the Resource Manager client + client = resourcemanager_v3.ProjectsClient(credentials=credentials) + + # Get the current IAM policy + resource = f"projects/{project_id}" + request = iam_policy_pb2.GetIamPolicyRequest(resource=resource) + policy = client.get_iam_policy(request=request) + + # Check if the binding already exists + role = "roles/aiplatform.user" + member = f"serviceAccount:{service_account}" + + binding_exists = False + for binding in policy.bindings: + if binding.role == role: + if member in binding.members: + binding_exists = True + break + else: + # Add member to existing role binding + binding.members.append(member) + binding_exists = True + break + + # If role binding doesn't exist, create a new one + if not binding_exists: + new_binding = policy_pb2.Binding(role=role, members=[member]) + policy.bindings.append(new_binding) + + # Set the updated IAM policy + set_request = iam_policy_pb2.SetIamPolicyRequest( + resource=resource, policy=policy + ) + client.set_iam_policy(request=set_request) + + return { + "status": "SUCCESS", + "message": f"Granted {role} to {service_account}", + } + except Exception as ex: + return { + "status": "ERROR", + "error_details": f"Failed to grant IAM role: {str(ex)}", + } + + +def create_connection( + project_id: str, + location: str, + connection_id: str, + credentials: Credentials, + settings: BigQueryToolConfig, + friendly_name: str = "", + description: str = "", + grant_vertex_ai_role: bool = True, +) -> dict: + """Create a BigQuery Cloud Resource connection for Vertex AI. + + This creates a connection that can be used with AI functions like + AI.CLASSIFY, AI.IF, AI.SCORE and remote models for AI.GENERATE_TEXT, + AI.GENERATE_EMBEDDING. + + By default, this function also grants the connection's service account + the "Vertex AI User" role, which is required to use AI functions. + + Args: + project_id (str): The Google Cloud project id. + location (str): The location/region (e.g., 'us', 'eu', 'us-central1'). + connection_id (str): The connection id to create (e.g., 'my_ai_connection'). + credentials (Credentials): The credentials to use for the request. + settings (BigQueryToolConfig): The BigQuery tool settings. + friendly_name (str): Optional friendly name for the connection. + description (str): Optional description for the connection. + grant_vertex_ai_role (bool): If True (default), automatically grants the + Vertex AI User role to the connection's service account. + + Returns: + dict: The created connection details including the service account. + + Examples: + >>> create_connection("my-project", "us", "my_ai_connection") + { + "status": "SUCCESS", + "connection_id": "my_ai_connection", + "location": "us", + "full_connection_path": "us.my_ai_connection", + "service_account": "bqcx-123456789-xxxx@gcp-sa-bigquery-condel.iam.gserviceaccount.com", + "iam_status": "Granted roles/aiplatform.user to service account" + } + """ + try: + from google.cloud import bigquery_connection_v1 + + connection_client = bigquery_connection_v1.ConnectionServiceClient( + credentials=credentials + ) + parent = f"projects/{project_id}/locations/{location}" + + # Create a Cloud Resource connection (for Vertex AI) + connection = bigquery_connection_v1.Connection( + friendly_name=friendly_name or connection_id, + description=description or "BigQuery connection for Vertex AI", + cloud_resource=bigquery_connection_v1.CloudResourceProperties(), + ) + + created_conn = connection_client.create_connection( + parent=parent, + connection_id=connection_id, + connection=connection, + ) + + service_account = "" + if created_conn.cloud_resource and created_conn.cloud_resource.service_account_id: + service_account = created_conn.cloud_resource.service_account_id + + result = { + "status": "SUCCESS", + "connection_id": connection_id, + "location": location, + "full_connection_path": f"{location}.{connection_id}", + "service_account": service_account, + } + + # Automatically grant Vertex AI User role if requested + if grant_vertex_ai_role and service_account: + iam_result = _grant_vertex_ai_role(project_id, service_account, credentials) + if iam_result["status"] == "SUCCESS": + result["iam_status"] = f"Granted roles/aiplatform.user to {service_account}" + else: + result["iam_status"] = "FAILED" + result["iam_error"] = iam_result["error_details"] + result["manual_command"] = ( + f"gcloud projects add-iam-policy-binding {project_id} " + f"--member='serviceAccount:{service_account}' " + f"--role='roles/aiplatform.user'" + ) + elif not grant_vertex_ai_role: + result["note"] = ( + "IAM role not granted. To use AI functions, grant the Vertex AI User role: " + f"gcloud projects add-iam-policy-binding {project_id} " + f"--member='serviceAccount:{service_account}' " + f"--role='roles/aiplatform.user'" + ) + + return result + except Exception as ex: + error_msg = str(ex) + if "already exists" in error_msg.lower(): + return { + "status": "ERROR", + "error_details": f"Connection '{connection_id}' already exists in {location}. Use list_connections to see existing connections.", + } + return { + "status": "ERROR", + "error_details": error_msg, + } + + def get_job_info( project_id: str, job_id: str, diff --git a/src/google/adk/tools/bigquery/skills/__init__.py b/src/google/adk/tools/bigquery/skills/__init__.py new file mode 100644 index 0000000000..5f1a5529c3 --- /dev/null +++ b/src/google/adk/tools/bigquery/skills/__init__.py @@ -0,0 +1,29 @@ +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""BigQuery Skills for ADK. + +This module provides specialized skills for BigQuery operations: + +- BQMLSkill: Machine learning model training and prediction with BigQuery ML +- BQAIOperatorSkill: AI functions (AI.GENERATE, AI.CLASSIFY, etc.) +""" + +from .bq_ai_operator_skill import BQAIOperatorSkill +from .bqml_skill import BQMLSkill + +__all__ = [ + "BQMLSkill", + "BQAIOperatorSkill", +] diff --git a/src/google/adk/tools/bigquery/skills/bq_ai_operator_skill.py b/src/google/adk/tools/bigquery/skills/bq_ai_operator_skill.py new file mode 100644 index 0000000000..d2aa636b80 --- /dev/null +++ b/src/google/adk/tools/bigquery/skills/bq_ai_operator_skill.py @@ -0,0 +1,377 @@ +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""BigQuery AI Operator Skill for ADK. + +This skill provides AI/ML functions for generative AI operations directly +in BigQuery SQL, including: + +- AI.GENERATE: Flexible text/multimodal generation +- AI.CLASSIFY: Categorize data into user-defined classes +- AI.EXTRACT: Extract structured data from unstructured text +- AI.IF: Filter data based on natural language conditions +- AI.SCORE: Rate/rank data by quality or relevance +- AI.EMBED: Generate embeddings for semantic search +- AI.SIMILARITY: Compare embeddings for clustering/recommendations + +For full documentation, see: +https://cloud.google.com/bigquery/docs/generative-ai-overview +""" + +from __future__ import annotations + +from typing import Any +from typing import List + +from pydantic import Field + +from ....skills.base_skill import BaseSkill +from ....skills.base_skill import SkillConfig +from ....utils.feature_decorator import experimental + + +@experimental +class BQAIOperatorSkill(BaseSkill): + """Skill for BigQuery AI Operator functions. + + This skill bundles AI functions for generative AI operations in BigQuery, + allowing agents to use LLMs directly within SQL queries. + + Key capabilities: + - Text generation and summarization + - Classification into custom categories + - Entity/data extraction from unstructured text + - Semantic filtering with natural language + - Quality scoring and ranking + - Embedding generation for vector search + - Similarity comparison for recommendations + + Example: + ```python + from google.adk.tools.bigquery.skills import BQAIOperatorSkill + + skill = BQAIOperatorSkill() + agent = Agent( + name="ai_analyst", + skills=[skill], + enable_programmatic_tool_calling=True, + ) + ``` + + Note: Requires BigQuery Enterprise edition and appropriate permissions. + """ + + name: str = Field(default="bq_ai_operator") + description: str = Field( + default=( + "AI-powered analysis in BigQuery using generative AI. " + "Generate text, classify data, extract entities, filter with " + "natural language, score/rank content, create embeddings, and " + "compute semantic similarity - all directly in SQL." + ) + ) + config: SkillConfig = Field( + default_factory=lambda: SkillConfig( + timeout_seconds=180.0, # AI operations can take time + max_parallel_calls=10, + ) + ) + + def get_tool_declarations(self) -> List[dict[str, Any]]: + """Return tool declarations for BQ AI Operator functions.""" + return [ + { + "name": "ai_generate", + "description": ( + "Generate text using AI.GENERATE. The most flexible inference " + "function - analyze any combination of text and unstructured " + "data. Use for summarization, translation, Q&A, content " + "generation, and more." + ), + "parameters": { + "input_data": "SQL query or table with input data", + "prompt": "The prompt template with column references", + "model": "Optional model name (default: gemini-pro)", + "output_column": "Name for the output column", + }, + }, + { + "name": "ai_generate_table", + "description": ( + "Generate structured table output using AI.GENERATE_TABLE. " + "Extracts structured data into a table with a custom schema " + "from unstructured input." + ), + "parameters": { + "input_data": "SQL query or table with input data", + "prompt": "The prompt describing what to extract", + "output_schema": "Schema for output table columns", + "model": "Optional model name", + }, + }, + { + "name": "ai_classify", + "description": ( + "Classify data into user-defined categories using AI.CLASSIFY. " + "Groups text/multimodal data into predefined classes." + ), + "parameters": { + "input_data": "SQL query or table with data to classify", + "input_column": "Column containing text to classify", + "categories": "List of category names/labels", + "output_column": "Name for the classification result column", + }, + }, + { + "name": "ai_extract", + "description": ( + "Extract structured information from unstructured text using " + "AI. Useful for entity extraction, parsing documents, etc." + ), + "parameters": { + "input_data": "SQL query or table with unstructured text", + "input_column": "Column containing text to process", + "extraction_spec": "What to extract (entities, fields, etc.)", + "output_columns": "Names for extracted data columns", + }, + }, + { + "name": "ai_if", + "description": ( + "Filter data using natural language conditions with AI.IF. " + "Returns TRUE/FALSE based on whether rows match the condition." + ), + "parameters": { + "input_data": "SQL query or table to filter", + "input_column": "Column to evaluate", + "condition": "Natural language condition to check", + "output_column": "Name for the boolean result column", + }, + }, + { + "name": "ai_score", + "description": ( + "Rate/score data using AI.SCORE. Assigns numeric scores to " + "rank rows by quality, relevance, or other criteria." + ), + "parameters": { + "input_data": "SQL query or table to score", + "input_column": "Column containing content to score", + "criteria": "Scoring criteria in natural language", + "output_column": "Name for the score column", + }, + }, + { + "name": "ai_embed", + "description": ( + "Generate embeddings using AI.EMBED or AI.GENERATE_EMBEDDING. " + "Creates high-dimensional vectors for semantic search, " + "clustering, and similarity comparisons." + ), + "parameters": { + "input_data": "SQL query or table with text to embed", + "input_column": "Column containing text", + "model": "Embedding model (default: text-embedding-004)", + "output_column": "Name for the embedding vector column", + }, + }, + { + "name": "ai_similarity", + "description": ( + "Compare embeddings using AI.SIMILARITY. Computes cosine " + "similarity between embedding vectors for clustering, " + "recommendations, and duplicate detection." + ), + "parameters": { + "embedding1": "First embedding column or value", + "embedding2": "Second embedding column or value", + "output_column": "Name for the similarity score column", + }, + }, + { + "name": "ai_forecast", + "description": ( + "Generate time series forecasts using AI.FORECAST. " + "Uses TimesFM model for point and interval predictions." + ), + "parameters": { + "input_data": "SQL query or table with time series data", + "timestamp_col": "Column containing timestamps", + "data_col": "Column containing values to forecast", + "horizon": "Number of time steps to forecast", + "id_cols": "Optional columns identifying multiple series", + }, + }, + ] + + def get_orchestration_template(self) -> str: + """Return example orchestration code for BQ AI Operator functions.""" + return ''' +async def analyze_customer_feedback(tools): + """Analyze customer feedback using AI functions.""" + import asyncio + + # Classify sentiment and extract entities in parallel + classify_result, extract_result = await asyncio.gather( + tools.ai_classify( + input_data=""" + SELECT feedback_id, feedback_text + FROM `my_project.my_dataset.customer_feedback` + WHERE feedback_date >= '2024-01-01' + """, + input_column="feedback_text", + categories=["positive", "negative", "neutral", "mixed"], + output_column="sentiment" + ), + tools.ai_extract( + input_data=""" + SELECT feedback_id, feedback_text + FROM `my_project.my_dataset.customer_feedback` + WHERE feedback_date >= '2024-01-01' + """, + input_column="feedback_text", + extraction_spec="product names, feature requests, complaints", + output_columns=["products", "feature_requests", "complaints"] + ) + ) + + # Summarize the results + sentiment_counts = {} + for row in classify_result.get("rows", []): + sentiment = row.get("sentiment") + sentiment_counts[sentiment] = sentiment_counts.get(sentiment, 0) + 1 + + return { + "total_feedback": len(classify_result.get("rows", [])), + "sentiment_distribution": sentiment_counts, + "sample_extractions": extract_result.get("rows", [])[:10] + } + + +async def semantic_search(tools): + """Perform semantic search using embeddings.""" + + # Generate embeddings for search query + query_embedding = await tools.ai_embed( + input_data="SELECT 'How do I reset my password?' as query", + input_column="query", + model="text-embedding-004", + output_column="query_embedding" + ) + + # Find similar documents + similar_docs = await tools.ai_similarity( + embedding1="query_embedding", + embedding2="doc_embedding", + output_column="similarity_score" + ) + + # Filter to most relevant results + relevant = [ + row for row in similar_docs.get("rows", []) + if row.get("similarity_score", 0) > 0.7 + ] + + return { + "query": "How do I reset my password?", + "num_results": len(relevant), + "top_matches": relevant[:5] + } + + +async def content_moderation(tools): + """Filter inappropriate content using AI.IF.""" + + # Filter content based on natural language criteria + filtered = await tools.ai_if( + input_data=""" + SELECT post_id, content, author_id + FROM `my_project.my_dataset.user_posts` + WHERE created_at >= CURRENT_DATE() - 7 + """, + input_column="content", + condition="content is appropriate for all ages and does not contain spam, harassment, or explicit material", + output_column="is_appropriate" + ) + + # Score remaining content for quality + scored = await tools.ai_score( + input_data=""" + SELECT post_id, content + FROM appropriate_posts + """, + input_column="content", + criteria="helpfulness, clarity, and engagement potential on a scale of 1-10", + output_column="quality_score" + ) + + appropriate_count = sum( + 1 for row in filtered.get("rows", []) + if row.get("is_appropriate") + ) + + return { + "total_posts": len(filtered.get("rows", [])), + "appropriate_posts": appropriate_count, + "flagged_posts": len(filtered.get("rows", [])) - appropriate_count, + "top_quality_posts": sorted( + scored.get("rows", []), + key=lambda x: x.get("quality_score", 0), + reverse=True + )[:10] + } +''' + + def filter_result(self, result: Any) -> Any: + """Filter BQ AI Operator results to reduce context size. + + - Truncates large result sets + - Truncates very long generated text + - Summarizes embedding vectors (too large to return in full) + """ + if not isinstance(result, dict): + return result + + filtered = result.copy() + + # Truncate large row results + if "rows" in filtered and len(filtered["rows"]) > 50: + filtered["rows"] = filtered["rows"][:50] + filtered["result_truncated"] = True + filtered["total_rows"] = len(result["rows"]) + + # Process individual rows + if "rows" in filtered: + for row in filtered["rows"]: + if isinstance(row, dict): + for key, value in row.items(): + # Truncate very long text outputs + if isinstance(value, str) and len(value) > 2000: + row[key] = value[:2000] + "... [truncated]" + + # Summarize embedding vectors (don't send full vector) + if isinstance(value, list) and len(value) > 100: + if all(isinstance(x, (int, float)) for x in value[:10]): + row[key] = { + "type": "embedding_vector", + "dimensions": len(value), + "sample": value[:5], + "note": "Full vector truncated for context efficiency", + } + + # Round numeric values + if isinstance(value, float): + row[key] = round(value, 4) + + return filtered diff --git a/src/google/adk/tools/bigquery/skills/bqml_skill.py b/src/google/adk/tools/bigquery/skills/bqml_skill.py new file mode 100644 index 0000000000..fc2bea3028 --- /dev/null +++ b/src/google/adk/tools/bigquery/skills/bqml_skill.py @@ -0,0 +1,323 @@ +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""BigQuery ML (BQML) Skill for ADK. + +This skill provides machine learning capabilities using BigQuery ML, +allowing agents to train, evaluate, and use ML models directly in SQL. + +Supported model types: +- Linear/logistic regression +- K-means clustering +- Matrix factorization +- Time series (ARIMA_PLUS) +- Deep neural networks (via Vertex AI) +- XGBoost/Random Forest +- And more + +For full documentation, see: +https://cloud.google.com/bigquery/docs/bqml-introduction +""" + +from __future__ import annotations + +from typing import Any +from typing import List + +from pydantic import Field + +from ....skills.base_skill import BaseSkill +from ....skills.base_skill import SkillConfig +from ....utils.feature_decorator import experimental + + +@experimental +class BQMLSkill(BaseSkill): + """Skill for BigQuery ML operations. + + This skill bundles tools for machine learning operations in BigQuery, + including model creation, training, evaluation, and prediction. + + Supported model types: + - LINEAR_REG: Linear regression for numeric predictions + - LOGISTIC_REG: Logistic regression for classification + - KMEANS: K-means clustering for segmentation + - MATRIX_FACTORIZATION: Recommendation systems + - ARIMA_PLUS: Time series forecasting + - DNN_CLASSIFIER/DNN_REGRESSOR: Deep neural networks + - BOOSTED_TREE_CLASSIFIER/BOOSTED_TREE_REGRESSOR: XGBoost models + - RANDOM_FOREST_CLASSIFIER/RANDOM_FOREST_REGRESSOR: Random forest + + Example: + ```python + from google.adk.tools.bigquery.skills import BQMLSkill + + skill = BQMLSkill() + agent = Agent( + name="ml_analyst", + skills=[skill], + enable_programmatic_tool_calling=True, + ) + ``` + """ + + name: str = Field(default="bqml") + description: str = Field( + default=( + "Machine learning with BigQuery ML. Train, evaluate, and deploy " + "ML models using SQL. Supports regression, classification, " + "clustering, time series forecasting, and deep learning models." + ) + ) + config: SkillConfig = Field( + default_factory=lambda: SkillConfig( + timeout_seconds=300.0, # ML training can take time + max_parallel_calls=5, + ) + ) + + def get_tool_declarations(self) -> List[dict[str, Any]]: + """Return tool declarations for BQML operations.""" + return [ + { + "name": "create_model", + "description": ( + "Create and train a BigQuery ML model. " + "Supports various model types including linear regression, " + "logistic regression, k-means, time series, and more." + ), + "parameters": { + "model_name": ( + "Fully qualified model name (project.dataset.model)" + ), + "model_type": ( + "Type of model: LINEAR_REG, LOGISTIC_REG, KMEANS, " + "ARIMA_PLUS, DNN_CLASSIFIER, BOOSTED_TREE_REGRESSOR, etc." + ), + "training_data": "SQL query or table for training data", + "options": "Additional model options as key-value pairs", + }, + }, + { + "name": "evaluate_model", + "description": ( + "Evaluate a trained model's performance. Returns metrics " + "like accuracy, precision, recall, RMSE, MAE depending on " + "model type." + ), + "parameters": { + "model_name": "Fully qualified model name", + "eval_data": "Optional evaluation dataset (SQL or table)", + }, + }, + { + "name": "predict", + "description": ( + "Generate predictions using a trained model. " + "Returns predicted values along with input features." + ), + "parameters": { + "model_name": "Fully qualified model name", + "input_data": "SQL query or table for prediction input", + }, + }, + { + "name": "explain_predict", + "description": ( + "Generate predictions with feature attributions. " + "Shows which features contributed most to each prediction." + ), + "parameters": { + "model_name": "Fully qualified model name", + "input_data": "SQL query or table for prediction input", + "top_k_features": "Number of top features to show", + }, + }, + { + "name": "get_model_info", + "description": ( + "Get information about a model including training stats, " + "feature info, and hyperparameters." + ), + "parameters": { + "model_name": "Fully qualified model name", + }, + }, + { + "name": "list_models", + "description": "List all models in a dataset.", + "parameters": { + "dataset": "Fully qualified dataset (project.dataset)", + }, + }, + { + "name": "drop_model", + "description": "Delete a model from the dataset.", + "parameters": { + "model_name": "Fully qualified model name", + }, + }, + { + "name": "feature_importance", + "description": ( + "Get feature importance scores for tree-based models " + "(XGBoost, Random Forest)." + ), + "parameters": { + "model_name": "Fully qualified model name", + }, + }, + { + "name": "confusion_matrix", + "description": ( + "Generate confusion matrix for classification models. " + "Shows true/false positives and negatives." + ), + "parameters": { + "model_name": "Fully qualified model name", + "eval_data": "Optional evaluation dataset", + }, + }, + { + "name": "roc_curve", + "description": ( + "Generate ROC curve data for binary classification models. " + "Returns threshold, TPR, FPR values." + ), + "parameters": { + "model_name": "Fully qualified model name", + "eval_data": "Optional evaluation dataset", + }, + }, + ] + + def get_orchestration_template(self) -> str: + """Return example orchestration code for BQML operations.""" + return ''' +async def train_and_evaluate(tools): + """Train a model and evaluate its performance.""" + import asyncio + + # Create a linear regression model + create_result = await tools.create_model( + model_name="my_project.my_dataset.sales_predictor", + model_type="LINEAR_REG", + training_data=""" + SELECT + store_id, + product_category, + day_of_week, + is_holiday, + sales_amount + FROM `my_project.my_dataset.historical_sales` + WHERE sales_date >= '2024-01-01' + """, + options={ + "input_label_cols": ["sales_amount"], + "enable_global_explain": True + } + ) + + if create_result.get("status") != "SUCCESS": + return {"error": create_result.get("error_details")} + + # Evaluate and predict in parallel + eval_result, predictions = await asyncio.gather( + tools.evaluate_model( + model_name="my_project.my_dataset.sales_predictor" + ), + tools.predict( + model_name="my_project.my_dataset.sales_predictor", + input_data=""" + SELECT store_id, product_category, day_of_week, is_holiday + FROM `my_project.my_dataset.upcoming_dates` + """ + ) + ) + + return { + "model_created": True, + "evaluation_metrics": eval_result.get("rows", []), + "sample_predictions": predictions.get("rows", [])[:10], + "total_predictions": len(predictions.get("rows", [])) + } + + +async def cluster_customers(tools): + """Segment customers using k-means clustering.""" + # Create k-means clustering model + await tools.create_model( + model_name="my_project.my_dataset.customer_segments", + model_type="KMEANS", + training_data=""" + SELECT + customer_id, + total_purchases, + avg_order_value, + purchase_frequency, + days_since_last_purchase + FROM `my_project.my_dataset.customer_metrics` + """, + options={ + "num_clusters": 5, + "standardize_features": True + } + ) + + # Get cluster assignments + clusters = await tools.predict( + model_name="my_project.my_dataset.customer_segments", + input_data="SELECT * FROM `my_project.my_dataset.customer_metrics`" + ) + + # Summarize by cluster + cluster_counts = {} + for row in clusters.get("rows", []): + centroid_id = row.get("CENTROID_ID") + cluster_counts[centroid_id] = cluster_counts.get(centroid_id, 0) + 1 + + return { + "num_clusters": len(cluster_counts), + "cluster_sizes": cluster_counts, + "sample_assignments": clusters.get("rows", [])[:20] + } +''' + + def filter_result(self, result: Any) -> Any: + """Filter BQML results to reduce context size. + + - Truncates large result sets + - Rounds numeric metrics for readability + - Removes verbose internal fields + """ + if not isinstance(result, dict): + return result + + filtered = result.copy() + + # Truncate large row results + if "rows" in filtered and len(filtered["rows"]) > 100: + filtered["rows"] = filtered["rows"][:100] + filtered["result_truncated"] = True + filtered["total_rows"] = len(result["rows"]) + + # Round numeric values in metrics for readability + if "rows" in filtered: + for row in filtered["rows"]: + if isinstance(row, dict): + for key, value in row.items(): + if isinstance(value, float): + row[key] = round(value, 6) + + return filtered From 702c56a0d5159d8892c340dc24880eeb8d45474f Mon Sep 17 00:00:00 2001 From: Haiyuan Cao Date: Tue, 9 Dec 2025 01:52:48 -0800 Subject: [PATCH 4/6] fix: resolve skill injection timing issue with append_instructions API MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This commit fixes the 'LlmRequest' object has no attribute 'system_instruction' error by using the proper LlmRequest.append_instructions() API for skill injection. Key changes: - skill_callbacks.py: Use llm_request.append_instructions([skill_content]) instead of directly modifying llm_request.system_instruction (which doesn't exist) - README.md: Update documentation to reflect direct injection architecture - Add comprehensive ADK Skills Framework design document Technical details: - The append_instructions() method properly concatenates to config.system_instruction - This ensures skills are available in the FIRST LLM call by injecting directly into the llm_request in before_model_callback - Fixes timing issue where instruction provider runs before callback 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 --- .../samples/bigquery_skills_demo/README.md | 15 +- .../bigquery_skills_demo/skill_callbacks.py | 90 +- docs/ADK_SKILLS_DESIGN.md | 932 ++++++++++++++++++ 3 files changed, 1033 insertions(+), 4 deletions(-) create mode 100644 docs/ADK_SKILLS_DESIGN.md diff --git a/contributing/samples/bigquery_skills_demo/README.md b/contributing/samples/bigquery_skills_demo/README.md index bebc272147..63c6783bb8 100644 --- a/contributing/samples/bigquery_skills_demo/README.md +++ b/contributing/samples/bigquery_skills_demo/README.md @@ -117,7 +117,8 @@ This demo uses ADK callbacks instead of LLM tool calls for skill management: │ 1. Extract keywords from user message │ │ 2. Match against skill keywords (from SKILL.md frontmatter) │ │ 3. Activate matching skills: ["bqml"] │ -│ 4. Skills injected into system prompt via instruction provider │ +│ 4. DIRECTLY INJECT skill content into llm_request.system_instruction│ +│ (This ensures skills are available in the FIRST LLM call!) │ └─────────────────────────────────────────────────────────────────────┘ │ ▼ @@ -126,6 +127,7 @@ This demo uses ADK callbacks instead of LLM tool calls for skill management: │ System prompt now includes: │ │ - Base instruction │ │ - Active skill documentation (BQML syntax, examples) │ +│ Skills are available IMMEDIATELY - no need to wait for tool call! │ └─────────────────────────────────────────────────────────────────────┘ │ ▼ @@ -136,6 +138,17 @@ This demo uses ADK callbacks instead of LLM tool calls for skill management: └─────────────────────────────────────────────────────────────────────┘ ``` +### Direct Injection vs Instruction Provider + +The callback directly injects skill content into `llm_request.system_instruction`, bypassing the instruction provider timing issue: + +| Approach | When Skills Appear | How It Works | +|----------|-------------------|--------------| +| **Direct Injection** (current) | First LLM call | Callback modifies `llm_request.system_instruction` directly | +| Instruction Provider | Second LLM call | Provider reads from state, but state updated after instruction built | + +This direct injection ensures the LLM has skill documentation from the very first response, enabling it to follow skill examples immediately. + ### Key Components 1. **Skill Discovery** (`skill_registry.py`) diff --git a/contributing/samples/bigquery_skills_demo/skill_callbacks.py b/contributing/samples/bigquery_skills_demo/skill_callbacks.py index 45bcd397c9..9b7d775650 100644 --- a/contributing/samples/bigquery_skills_demo/skill_callbacks.py +++ b/contributing/samples/bigquery_skills_demo/skill_callbacks.py @@ -153,6 +153,80 @@ def _build_patterns_from_registry(self) -> dict[str, list[str]]: return patterns + def _build_skill_content(self, skill_names: list[str]) -> str: + """Build skill content string from activated skills. + + This loads the full skill documentation from the registry and formats + it for injection into the system instruction. + + Args: + skill_names: List of skill names to load. + + Returns: + Formatted string containing all skill documentation. + """ + if not skill_names: + return "" + + skill_sections = [] + for skill_name in skill_names: + skill = self._registry.load_skill_content(skill_name) + if skill: + skill_sections.append(f""" +## Active Skill: {skill.name} + +{skill.description} + +--- + +{skill.content} +""") + + if not skill_sections: + return "" + + return f""" +# Currently Active Skills + +The following skills have been loaded and are available for this task: + +{"".join(skill_sections)} + +--- +**Note**: Use `deactivate_skill(skill_name)` when you're done with a skill to free up context. +""" + + def _inject_skills_into_request( + self, + llm_request: "LlmRequest", + skill_names: list[str], + ) -> None: + """Inject skill content directly into the LLM request's system instruction. + + This is the key fix for the timing issue: by modifying llm_request.config.system_instruction + directly in the before_model_callback, the skills are available in the FIRST LLM call, + not just subsequent calls. + + The LlmRequest has a config.system_instruction field that can be a string. + We use the append_instructions() method which handles string concatenation properly. + + Args: + llm_request: The LLM request to modify. + skill_names: List of skill names to inject. + """ + if not skill_names: + return + + skill_content = self._build_skill_content(skill_names) + if not skill_content: + return + + # Use append_instructions which properly handles string system_instruction + # This concatenates to config.system_instruction using "\n\n" + llm_request.append_instructions([skill_content]) + + print(f"[SkillCallbacks] Injected skill content into system instruction: {skill_names}") + def _get_classifier(self): """Lazy initialization of the skill classifier.""" if self._classifier is None and self._detection_mode in ("llm", "hybrid"): @@ -286,14 +360,16 @@ def before_model_callback( """Auto-activate skills based on user input before LLM processes it. This callback analyzes the user message and automatically activates - relevant skills, injecting their documentation into context via the - instruction provider. + relevant skills, then DIRECTLY injects their documentation into the + llm_request.system_instruction. This ensures skills are available in + the FIRST LLM call, not just subsequent calls. Strategy: - If NO skills are currently active: Detect from the LATEST user message (this handles new user requests after skills were cleared) - If skills ARE already active: Use the ORIGINAL user message to avoid re-detecting on subsequent LLM calls in the same tool-use flow + - ALWAYS inject skills directly into llm_request.system_instruction Args: callback_context: Context for accessing/modifying state. @@ -326,11 +402,17 @@ def before_model_callback( callback_context.state[ACTIVE_SKILLS_KEY] = skills_to_activate print(f"[SkillCallbacks] Detecting skills from: {user_text[:100]}...") print(f"[SkillCallbacks] Auto-activated skills: {skills_to_activate}") + + # KEY FIX: Inject skills directly into the llm_request + # This ensures skills are available in the FIRST LLM call + self._inject_skills_into_request(llm_request, skills_to_activate) else: # Skills already active: This is a subsequent LLM call in the same flow # Use the ORIGINAL user message to ensure consistency user_text = self._get_original_user_message_text(llm_request) if not user_text: + # Even if no user text, still inject active skills + self._inject_skills_into_request(llm_request, active_skills) return None # Detect which skills should be activated from the original user message @@ -348,8 +430,10 @@ def before_model_callback( callback_context.state[ACTIVE_SKILLS_KEY] = active_skills print(f"[SkillCallbacks] Additional skills detected: {newly_activated}") + # KEY FIX: Always inject skills into the llm_request for subsequent calls too + self._inject_skills_into_request(llm_request, active_skills) + # Return None to proceed with LLM call - # The instruction provider will pick up the activated skills return None def after_agent_callback( diff --git a/docs/ADK_SKILLS_DESIGN.md b/docs/ADK_SKILLS_DESIGN.md new file mode 100644 index 0000000000..8f2ccb2079 --- /dev/null +++ b/docs/ADK_SKILLS_DESIGN.md @@ -0,0 +1,932 @@ +# ADK Dynamic Skills Framework Design Document + +**Author:** Agent Development Kit Team +**Status:** Proposal +**Created:** December 2025 +**Version:** 1.0 + +--- + +## Executive Summary + +This document proposes a first-class **Dynamic Skills Framework** for the Google Agent Development Kit (ADK). The framework enables agents to dynamically load domain-specific knowledge into their context on-demand, addressing two critical challenges in LLM-based agents: + +1. **Knowledge Staleness**: Rapidly evolving domains (like BigQuery AI functions) require up-to-date guidance that cannot be baked into model weights +2. **Context Window Efficiency**: Loading comprehensive documentation permanently wastes precious context tokens on irrelevant information + +The proposed solution uses **callback-based skill injection** to automatically detect relevant skills from user input and inject them ephemerally into the system instruction, achieving zero-latency skill availability with minimal context overhead. + +--- + +## Table of Contents + +1. [Problem Statement](#1-problem-statement) +2. [Goals and Non-Goals](#2-goals-and-non-goals) +3. [Design Overview](#3-design-overview) +4. [Detailed Design](#4-detailed-design) +5. [API Specification](#5-api-specification) +6. [Implementation Details](#6-implementation-details) +7. [BigQuery Skills Demo Case Study](#7-bigquery-skills-demo-case-study) +8. [Performance Analysis](#8-performance-analysis) +9. [Migration and Rollout](#9-migration-and-rollout) +10. [Future Extensions](#10-future-extensions) +11. [Appendix](#appendix) + +--- + +## 1. Problem Statement + +### 1.1 The Knowledge Staleness Problem + +Modern cloud platforms evolve rapidly. Consider BigQuery's AI capabilities: + +| Timeline | New Feature | +|----------|-------------| +| Q3 2024 | AI.CLASSIFY, AI.IF, AI.SCORE functions | +| Q4 2024 | Gemini 2.0 Flash endpoint | +| Q1 2025 | Gemini 2.5 Pro, Claude 3.5 Sonnet integration | +| Q2 2025 | New connection_id syntax requirements | + +LLM training data lags 6-18 months behind. An agent with outdated knowledge will: +- Generate incorrect SQL syntax +- Reference deprecated endpoints +- Miss critical configuration requirements (e.g., location matching for connections) + +**Example of Outdated Knowledge Impact:** +```sql +-- LLM might generate (outdated): +CREATE REMOTE MODEL `project.dataset.model` +OPTIONS (ENDPOINT = 'gemini-pro'); -- Old endpoint name + +-- Correct (current): +CREATE REMOTE MODEL `project.dataset.model` +REMOTE WITH CONNECTION `us.my_connection` -- Required connection +OPTIONS (ENDPOINT = 'gemini-2.5-pro'); -- Current endpoint +``` + +### 1.2 The Context Window Efficiency Problem + +Comprehensive documentation for a single domain can be substantial: + +| Skill | Documentation Size | Tokens (est.) | +|-------|-------------------|---------------| +| BQML | ~4,000 words | ~6,000 tokens | +| BQ AI Operator | ~2,500 words | ~3,750 tokens | +| BQ Remote Model | ~3,500 words | ~5,250 tokens | +| **Total** | **~10,000 words** | **~15,000 tokens** | + +Loading all skills permanently means: +- 15,000 tokens consumed before any user interaction +- Reduced space for conversation history +- Slower response times (more tokens to process) +- Higher costs (token-based pricing) + +### 1.3 Current Approaches and Limitations + +| Approach | Description | Limitation | +|----------|-------------|------------| +| **Static System Prompt** | All documentation in base prompt | Context waste, knowledge staleness | +| **RAG (Retrieval)** | Semantic search for relevant docs | Latency, retrieval quality issues | +| **Tool-based Loading** | Agent calls `load_skill()` tool | Extra LLM call(s), timing delays | +| **Instruction Provider** | Dynamic system instruction | Timing issue: skills not in first call | + +The tool-based approach is particularly problematic: + +``` +User: "Train a model to predict penguin weight" + │ + ▼ + ┌────────────────────────┐ + │ LLM Call #1 │ ◄── No skill loaded yet! + │ "I'll load the BQML │ Agent must first decide + │ skill to help you" │ to load the skill + └────────────────────────┘ + │ + ▼ Tool call: activate_skill("bqml") + ┌────────────────────────┐ + │ LLM Call #2 │ ◄── Now skill is available + │ "Here's how to train │ But we wasted a round-trip + │ your model..." │ + └────────────────────────┘ +``` + +--- + +## 2. Goals and Non-Goals + +### 2.1 Goals + +1. **Zero-Latency Skill Availability**: Skills are available in the FIRST LLM call, not after tool calls +2. **Ephemeral Loading**: Skills are injected into system instruction, not conversation history +3. **Automatic Detection**: Keywords/patterns trigger skill loading without LLM decision-making +4. **Automatic Cleanup**: Skills are removed after each turn to free context +5. **Scalable Architecture**: Adding new skills requires only a markdown file, no code changes +6. **Multi-Skill Support**: Multiple skills can be active simultaneously +7. **Configurable Detection**: Support keyword, LLM, and hybrid detection modes + +### 2.2 Non-Goals + +1. **Persistent Skill Memory**: Skills are ephemeral per-turn (not across sessions) +2. **Skill Execution**: Skills provide knowledge, not executable code +3. **Skill Versioning**: No built-in version management (use Git for skill files) +4. **Cross-Agent Skill Sharing**: Skills are agent-specific (no central registry) +5. **Skill Composition**: No dependency management between skills + +--- + +## 3. Design Overview + +### 3.1 Architecture Diagram + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ User Message │ +│ "Train a model to predict penguin weight" │ +└─────────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────────────┐ +│ before_model_callback │ +│ │ +│ 1. Extract text from llm_request.contents │ +│ 2. Match against skill keywords (from SKILL.md frontmatter) │ +│ - "train" matches bqml │ +│ - "model" matches bqml │ +│ - "predict" matches bqml │ +│ 3. Load skill content from SkillRegistry │ +│ 4. Call llm_request.append_instructions([skill_content]) │ +│ └─► This modifies config.system_instruction directly │ +│ 5. Store active skills in callback_context.state │ +│ │ +│ Result: Skills injected BEFORE first LLM call │ +└─────────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────────────┐ +│ LLM Processing │ +│ │ +│ System Instruction now includes: │ +│ ┌──────────────────────────────────────────────────────────────────────┐ │ +│ │ [Base Agent Instructions] │ │ +│ │ │ │ +│ │ # Currently Active Skills │ │ +│ │ │ │ +│ │ ## Active Skill: bqml │ │ +│ │ BigQuery ML - Train, evaluate, and deploy ML models using SQL... │ │ +│ │ │ │ +│ │ ### Step 1: Train a Model │ │ +│ │ ```sql │ │ +│ │ CREATE OR REPLACE MODEL `project.dataset.model_name` │ │ +│ │ OPTIONS(model_type='LINEAR_REG', input_label_cols=['target'])... │ │ +│ │ ``` │ │ +│ └──────────────────────────────────────────────────────────────────────┘ │ +│ │ +│ LLM generates response using skill knowledge from FIRST call │ +└─────────────────────────────────────────────────────────────────────────────┘ + │ + ▼ (tool calls, multi-turn processing) +┌─────────────────────────────────────────────────────────────────────────────┐ +│ after_agent_callback │ +│ │ +│ 1. Read active skills from callback_context.state │ +│ 2. Clear state: callback_context.state[ACTIVE_SKILLS_KEY] = [] │ +│ 3. Log: "[SkillCallbacks] Auto-deactivated skills: ['bqml']" │ +│ │ +│ Result: Context freed for next user turn │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +### 3.2 Key Design Decisions + +#### Decision 1: Callbacks vs Tools for Skill Management + +| Aspect | Callback (Chosen) | Tool-based | +|--------|-------------------|------------| +| **LLM Calls** | Zero for skill management | 1-2 per skill activation | +| **Latency** | Instant (regex matching) | Round-trip to model | +| **Cost** | No additional tokens | Extra tool call tokens | +| **First-Call Availability** | Yes | No (skills after tool call) | +| **Determinism** | 100% for keyword mode | LLM decides, may miss | + +**Rationale**: Domain-specific terminology (e.g., "AI.CLASSIFY", "BQML", "CREATE MODEL") is unambiguous and maps cleanly to skills. Keyword matching is sufficient and eliminates LLM overhead. + +#### Decision 2: Direct Injection vs Instruction Provider + +| Approach | When Skills Appear | Mechanism | +|----------|-------------------|-----------| +| **Direct Injection (Chosen)** | First LLM call | Modify `llm_request.config.system_instruction` | +| Instruction Provider | Second LLM call | Provider reads from state after instruction built | + +**Rationale**: The ADK processes `_preprocess_async` (which builds system instruction) BEFORE `before_model_callback`. Using an instruction provider means skills set in callback aren't visible until the next LLM call. Direct injection via `llm_request.append_instructions()` bypasses this timing issue. + +#### Decision 3: Ephemeral vs Persistent Skills + +| Aspect | Ephemeral (Chosen) | Persistent | +|--------|-------------------|------------| +| **Memory** | Cleared after each turn | Accumulates in session | +| **Context Efficiency** | High (only load when needed) | Low (grows over time) | +| **Multi-Topic** | Clean transitions | Old skills pollute context | + +**Rationale**: Most user interactions are single-topic. Clearing skills after each turn ensures fresh context and prevents irrelevant skill content from consuming tokens in unrelated queries. + +--- + +## 4. Detailed Design + +### 4.1 Component Overview + +``` +google/adk/skills/ +├── __init__.py # Public API exports +├── skill_registry.py # Skill discovery and loading +├── skill_callbacks.py # Callback-based skill management +├── skill_classifier.py # Optional LLM-based classification +└── types.py # Data classes (SkillMetadata, SkillContent) +``` + +### 4.2 Skill Definition Format (SKILL.md) + +Skills are defined as Markdown files with YAML frontmatter: + +```markdown +--- +name: bq_remote_model +description: BigQuery Remote Models - Create remote models connecting to Vertex AI +keywords: + - remote model + - create remote model + - generate text + - ai.generate_text + - gemini + - claude + - embeddings + - llm +--- + +# BQ Remote Model Skill + +Create and use remote models that connect BigQuery to Vertex AI... + +## Prerequisites + +1. A BigQuery connection to Vertex AI is required... + +## CREATE REMOTE MODEL Syntax + +```sql +CREATE OR REPLACE MODEL `project.dataset.model_name` +REMOTE WITH CONNECTION `project.region.connection_id` +OPTIONS (ENDPOINT = 'gemini-2.5-pro'); +``` + +[... full documentation ...] +``` + +**Frontmatter Fields:** + +| Field | Required | Description | +|-------|----------|-------------| +| `name` | Yes | Unique skill identifier (alphanumeric, underscores) | +| `description` | Yes | Short description for skill summary | +| `keywords` | No | List of trigger keywords/phrases for auto-detection | + +### 4.3 SkillRegistry Class + +```python +class SkillRegistry: + """Registry for dynamically discovering and loading skills. + + Implements progressive disclosure: + - Level 1: Skill names and descriptions (loaded at startup) + - Level 2: Full skill content (loaded on-demand) + """ + + def __init__(self, skills_dir: str | Path | None = None): + """Initialize registry and discover skills. + + Args: + skills_dir: Directory containing skill subdirectories. + Defaults to ./skills relative to caller. + """ + + def get_skill_names(self) -> list[str]: + """Get list of all discovered skill names.""" + + def get_skill_metadata(self, name: str) -> SkillMetadata | None: + """Get metadata (name, description, keywords) for a skill.""" + + def get_all_keywords(self) -> dict[str, list[str]]: + """Get all keywords for all skills, for pattern building.""" + + def load_skill_content(self, name: str) -> SkillContent | None: + """Load full content of a skill (Level 2 disclosure).""" + + def get_skills_summary(self) -> str: + """Get formatted summary for agent's base instruction.""" +``` + +### 4.4 SkillCallbacks Class + +```python +class SkillCallbacks: + """Callback handlers for automatic skill management. + + Detection modes: + - "keyword": Regex pattern matching (fastest, deterministic) + - "llm": LLM-based classification (semantic understanding) + - "hybrid": LLM with keyword fallback (recommended for mixed queries) + """ + + def __init__( + self, + registry: SkillRegistry, + auto_deactivate: bool = True, + detection_mode: Literal["keyword", "llm", "hybrid"] = "keyword", + ): + """Initialize skill callbacks. + + Args: + registry: SkillRegistry instance for loading skills + auto_deactivate: Clear skills after each turn (recommended) + detection_mode: How to detect skills from user input + """ + + def before_model_callback( + self, + callback_context: CallbackContext, + llm_request: LlmRequest, + ) -> LlmResponse | None: + """Detect and inject skills before LLM processing. + + This is the critical callback that: + 1. Extracts user message from llm_request.contents + 2. Detects relevant skills via keyword/LLM matching + 3. Loads skill content from registry + 4. Injects into llm_request via append_instructions() + 5. Stores active skills in callback_context.state + + Returns None to continue with LLM processing. + """ + + def after_agent_callback( + self, + callback_context: CallbackContext, + ) -> types.Content | None: + """Clean up skills after agent completes turn. + + Clears active_skills from state to free context. + Returns None (no content to add to conversation). + """ +``` + +### 4.5 Keyword Detection Algorithm + +```python +def _build_patterns_from_registry(self) -> dict[str, list[str]]: + """Build regex patterns from skill keywords. + + Handles: + - Multi-word keywords ("create remote model") + - Special characters (dots in "ai.classify") + - Word boundaries for precision + """ + patterns = {} + for skill_name, keywords in self._registry.get_all_keywords().items(): + skill_patterns = [] + for keyword in keywords: + escaped = re.escape(keyword) + # Don't add word boundaries for keywords with dots + if "." in keyword: + pattern = escaped + else: + pattern = rf"\b{escaped}\b" + skill_patterns.append(pattern) + patterns[skill_name] = skill_patterns + return patterns + +def _detect_skills_from_keywords(self, text: str) -> list[str]: + """Detect skills using compiled regex patterns. + + Returns list of skill names that matched at least one keyword. + """ + detected = [] + for skill_name, patterns in self._compiled_patterns.items(): + for pattern in patterns: + if pattern.search(text): + detected.append(skill_name) + break # One match is enough + return detected +``` + +--- + +## 5. API Specification + +### 5.1 LlmAgent Integration + +```python +from google.adk.agents import LlmAgent +from google.adk.skills import SkillRegistry, SkillCallbacks + +# Initialize skill infrastructure +skill_registry = SkillRegistry(skills_dir="./skills") +skill_callbacks = SkillCallbacks( + registry=skill_registry, + auto_deactivate=True, + detection_mode="keyword", +) + +# Create agent with skill callbacks +agent = LlmAgent( + model="gemini-2.5-pro", + name="my_agent", + instruction=base_instruction, + tools=[...], + # Skill management via callbacks + before_model_callback=skill_callbacks.before_model_callback, + after_agent_callback=skill_callbacks.after_agent_callback, +) +``` + +### 5.2 LlmRequest.append_instructions() API + +The `LlmRequest` class provides the `append_instructions()` method for modifying the system instruction: + +```python +def append_instructions( + self, + instructions: Union[list[str], types.Content] +) -> list[types.Content]: + """Appends instructions to the system instruction. + + Args: + instructions: The instructions to append. Can be: + - list[str]: Strings to concatenate with existing instruction + - types.Content: Content object with text/non-text parts + + Returns: + List of user contents from non-text parts (empty for list[str]). + + Behavior: + - list[str]: concatenates with existing system_instruction using \\n\\n + - types.Content: extracts text, creates references for non-text parts + """ +``` + +**Usage in Skill Injection:** +```python +def _inject_skills_into_request( + self, + llm_request: LlmRequest, + skill_names: list[str], +) -> None: + """Inject skill content directly into the LLM request.""" + skill_content = self._build_skill_content(skill_names) + if skill_content: + # This appends to config.system_instruction + llm_request.append_instructions([skill_content]) +``` + +### 5.3 State Management + +Skills use ADK's state system for tracking active skills: + +```python +# State key (session-scoped) +ACTIVE_SKILLS_KEY = "active_skills" + +# Reading active skills +active_skills: list[str] = callback_context.state.get(ACTIVE_SKILLS_KEY, []) + +# Writing active skills +callback_context.state[ACTIVE_SKILLS_KEY] = ["bqml", "bq_remote_model"] + +# Clearing skills +callback_context.state[ACTIVE_SKILLS_KEY] = [] +``` + +### 5.4 Manual Skill Tools (Optional Fallback) + +For cases where automatic detection fails, manual tools are available: + +```python +def activate_skill(skill_name: str, tool_context: ToolContext) -> str: + """Manually activate a skill.""" + +def deactivate_skill(skill_name: str, tool_context: ToolContext) -> str: + """Manually deactivate a skill.""" + +def list_active_skills(tool_context: ToolContext) -> str: + """List currently active skills.""" +``` + +--- + +## 6. Implementation Details + +### 6.1 Callback Execution Order + +Understanding ADK's callback execution order is critical: + +``` +Agent.run_async() + │ + ├─► _preprocess_async() # Builds initial system instruction + │ └─► instruction_provider() # Called HERE (skills not yet detected) + │ + ├─► before_model_callback() # Skills detected and injected HERE + │ └─► llm_request.append_instructions([skills]) + │ + ├─► LLM.generate() # Skills available in system instruction + │ + ├─► after_model_callback() # Process response + │ + ├─► [Tool execution loop] # May trigger more LLM calls + │ └─► before_model_callback() # Skills re-injected for each call + │ + └─► after_agent_callback() # Skills cleared HERE +``` + +### 6.2 Multi-Turn Tool Use Handling + +When an agent uses tools, there are multiple LLM calls in a single turn. The callback handles this by: + +1. **First call**: Detect skills from the LATEST user message +2. **Subsequent calls**: Use the ORIGINAL user message to avoid re-detection + +```python +def before_model_callback(self, callback_context, llm_request): + active_skills = callback_context.state.get(ACTIVE_SKILLS_KEY, []) + + if not active_skills: + # NEW user request - detect from latest message + user_text = self._get_user_message_text(llm_request) + skills_to_activate = self._detect_skills_from_text(user_text) + callback_context.state[ACTIVE_SKILLS_KEY] = skills_to_activate + else: + # Continuing same request - use original message + user_text = self._get_original_user_message_text(llm_request) + # Check for additional skills but don't reset + + # Always inject skills into this LLM call + self._inject_skills_into_request(llm_request, active_skills) +``` + +### 6.3 Skill Content Building + +```python +def _build_skill_content(self, skill_names: list[str]) -> str: + """Build formatted skill content for system instruction.""" + sections = [] + for skill_name in skill_names: + skill = self._registry.load_skill_content(skill_name) + if skill: + sections.append(f""" +## Active Skill: {skill.name} + +{skill.description} + +--- + +{skill.content} +""") + + if not sections: + return "" + + return f""" +# Currently Active Skills + +The following skills have been loaded for this task: + +{"".join(sections)} + +--- +**Note**: Use `deactivate_skill(skill_name)` when done to free context. +""" +``` + +--- + +## 7. BigQuery Skills Demo Case Study + +### 7.1 Domain Characteristics + +BigQuery AI capabilities exemplify a rapidly evolving domain: + +| Challenge | Manifestation | +|-----------|---------------| +| **API Changes** | New endpoints (gemini-2.5-pro), deprecated ones (gemini-pro) | +| **Syntax Requirements** | Connection IDs required for AI functions | +| **Location Rules** | Connection location must match dataset location | +| **Best Practices** | Task-specific parameters (max_output_tokens for summarization vs classification) | + +### 7.2 Skill Structure + +``` +bigquery_skills_demo/ +├── skills/ +│ ├── bqml/ +│ │ └── SKILL.md # ML model training (LINEAR_REG, KMEANS, etc.) +│ ├── bq_ai_operator/ +│ │ └── SKILL.md # AI.CLASSIFY, AI.IF, AI.SCORE functions +│ └── bq_remote_model/ +│ └── SKILL.md # Remote models, AI.GENERATE_TEXT +├── skill_registry.py # Dynamic discovery +├── skill_callbacks.py # Callback-based injection +└── agent.py # Agent configuration +``` + +### 7.3 Keyword Mapping + +| Skill | Keywords | Example Triggers | +|-------|----------|------------------| +| `bqml` | train, model, predict, regression, kmeans, forecast, arima | "Train a model to predict penguin weight" | +| `bq_ai_operator` | ai.classify, ai.if, ai.score, classify, sentiment, categorize | "Classify news articles by topic" | +| `bq_remote_model` | gemini, generate text, ai.generate_text, embeddings, remote model | "Create a Gemini model to summarize articles" | + +### 7.4 Real-World Scenario + +**User Input:** +> "Create a remote model using Gemini 2.5 Pro and use it to summarize 3 BBC news articles" + +**Keyword Detection:** +- "remote model" → `bq_remote_model` +- "Gemini" → `bq_remote_model` +- "summarize" → triggers summarization examples in skill + +**Injected Skill Content (excerpt):** +```markdown +## Active Skill: bq_remote_model + +### ⚠️ DEFAULT MODEL: Always Use Gemini 2.5 Pro + +**ALWAYS use `gemini-2.5-pro` as the default model** unless specifically requested. + +### Example: Text Summarization (Large max_output_tokens) + +```sql +-- Use 512-1024 tokens for summaries +SELECT + title, + ml_generate_text_result AS summary +FROM AI.GENERATE_TEXT( + MODEL `project.bq_demo.gemini_model`, + (SELECT + title, + CONCAT('Summarize: ', body) AS prompt + FROM `bigquery-public-data.bbc_news.fulltext` + LIMIT 5), + STRUCT( + 1024 AS max_output_tokens, -- LARGE for summarization + 0.3 AS temperature -- Low for factual output + ) +); +``` +``` + +**Agent Output (first LLM call):** +The agent immediately generates correct SQL using gemini-2.5-pro with appropriate parameters for summarization, without needing to first decide to load a skill. + +--- + +## 8. Performance Analysis + +### 8.1 Latency Comparison + +| Approach | First Response Latency | Total LLM Calls | +|----------|----------------------|-----------------| +| **Callback-based (proposed)** | ~2-3s | 1 (if no tools) | +| Tool-based loading | ~5-6s | 2+ (load + respond) | +| Static full prompt | ~2.5-3.5s | 1 (but always slower) | + +### 8.2 Token Efficiency + +**Scenario**: Agent with 3 available skills (BQML, AI Operator, Remote Model) + +| Approach | Tokens Used | Notes | +|----------|-------------|-------| +| All skills always loaded | ~15,000 | Regardless of query relevance | +| Callback-based (1 skill) | ~5,000 | Only relevant skill loaded | +| Callback-based (none) | ~500 | Just base instruction | + +**Annual Cost Savings** (assuming 1M queries/year, 50% needing skills): +- All skills: 15B tokens = $150,000 (at $0.01/1K tokens) +- Callback: 5B tokens = $50,000 +- **Savings: ~$100,000/year** + +### 8.3 Detection Accuracy + +Keyword-based detection with domain-specific terminology: + +| Metric | Value | Notes | +|--------|-------|-------| +| **Precision** | 99%+ | Domain terms are unambiguous | +| **Recall** | 95%+ | Comprehensive keyword lists | +| **False Positives** | <1% | Unlikely to mention "AI.CLASSIFY" without needing skill | +| **Detection Time** | <1ms | Compiled regex patterns | + +--- + +## 9. Migration and Rollout + +### 9.1 Phase 1: Framework Integration (Q1 2026) + +**Scope:** +- Add `google.adk.skills` module to ADK core +- Implement `SkillRegistry`, `SkillCallbacks` classes +- Update `LlmAgent` documentation for callback integration + +**API Surface:** +```python +from google.adk.skills import ( + SkillRegistry, + SkillCallbacks, + SkillMetadata, + SkillContent, + ACTIVE_SKILLS_KEY, +) +``` + +### 9.2 Phase 2: BigQuery Toolset Integration (Q2 2026) + +**Scope:** +- Bundle BigQuery skills with `BigQueryToolset` +- Auto-configure skill callbacks when using BQ tools +- Maintain skills as external markdown for easy updates + +**Configuration:** +```python +from google.adk.tools.bigquery import BigQueryToolset + +# Skills auto-configured +toolset = BigQueryToolset( + credentials_config=..., + enable_skills=True, # New parameter +) +``` + +### 9.3 Phase 3: Skill Marketplace (Q3 2026) + +**Scope:** +- Public skill repository +- Versioned skill packages +- Community contributions + +--- + +## 10. Future Extensions + +### 10.1 Multi-Modal Skills + +Support for image-based skill content: +```markdown +--- +name: chart_builder +description: Build charts and visualizations +modality: multi-modal +--- + +![Chart Types](./chart_types.png) + +Use chart type 1 for time series... +``` + +### 10.2 Skill Dependencies + +```yaml +--- +name: advanced_ml +description: Advanced ML techniques +requires: + - bqml # Base skill must be loaded first +--- +``` + +### 10.3 Dynamic Skill Updates + +Real-time skill updates without agent restart: +```python +skill_registry.reload_skill("bq_remote_model") # Hot reload +``` + +### 10.4 Skill Analytics + +Track skill usage for optimization: +```python +skill_registry.get_usage_stats() +# {"bqml": {"activations": 1000, "avg_duration": 45.2}, ...} +``` + +--- + +## Appendix + +### A.1 Complete SKILL.md Template + +```markdown +--- +name: my_skill +description: One-line description of what this skill provides +keywords: + - primary_keyword + - secondary_keyword + - function_name + - common_user_phrase +--- + +# My Skill Title + +Brief introduction to the skill's purpose. + +## Prerequisites + +1. Required setup step 1 +2. Required setup step 2 + +## Core Concepts + +### Concept 1 + +Explanation with example: + +```sql +-- Example code +SELECT * FROM table; +``` + +### Concept 2 + +More explanation... + +## Examples + +### Example 1: Common Use Case + +```sql +-- Full working example +``` + +### Example 2: Advanced Use Case + +```sql +-- Advanced example +``` + +## Troubleshooting + +**Error: "common error message"** +- Cause and solution + +## References + +- [Official Documentation](https://...) +``` + +### A.2 Debugging Skill Loading + +Enable debug logging: +```python +import logging +logging.getLogger("google.adk.skills").setLevel(logging.DEBUG) +``` + +Output: +``` +[SkillCallbacks] Detecting skills from: Train a model to predict... +[SkillCallbacks] Auto-activated skills: ['bqml'] +[SkillCallbacks] Injected skill content into system instruction: ['bqml'] +``` + +### A.3 Testing Skill Detection + +```python +def test_skill_detection(): + registry = SkillRegistry("./skills") + callbacks = SkillCallbacks(registry, detection_mode="keyword") + + # Test detection + detected = callbacks._detect_skills_from_text( + "Create a remote model using Gemini" + ) + assert "bq_remote_model" in detected + + # Test non-detection + detected = callbacks._detect_skills_from_text( + "What's the weather today?" + ) + assert len(detected) == 0 +``` + +--- + +## References + +1. Anthropic Engineering: [Equipping Agents for the Real World with Agent Skills](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills) +2. Google ADK Documentation: [LlmAgent Callbacks](https://cloud.google.com/docs/adk/callbacks) +3. BigQuery ML Documentation: [BQML Introduction](https://cloud.google.com/bigquery/docs/bqml-introduction) +4. BigQuery AI Functions: [AI Functions Reference](https://cloud.google.com/bigquery/docs/ai-functions) + +--- + +*Document Version: 1.0 | Last Updated: December 2025* From c40949b468c95bc4fc271d40b44bc5feec1a5b95 Mon Sep 17 00:00:00 2001 From: Haiyuan Cao Date: Tue, 9 Dec 2025 01:59:44 -0800 Subject: [PATCH 5/6] docs: rewrite Skills design as first-class ADK plugin primitive MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Major revision (v2.0) positioning Skills as a core ADK plugin: - Frame Skills as the fourth plugin primitive alongside Tools, Callbacks, Extensions - Add "what agent KNOWS" vs "what agent can DO" distinction - Define Skill as self-contained unit of domain knowledge Key additions: - ADK Plugin Ecosystem diagram showing Skills' unique role - Skill vs Tool decision matrix with concrete examples - Multiple domain case studies: BigQuery, Kubernetes, Compliance, Internal Standards - Full API specification for SkillRegistry, SkillCallbacks, SkillExtension - Integration patterns: Toolset bundling, Multi-domain, Composition, Conditional - Detailed rollout phases (Q1-Q4 2026) - Future roadmap: Multi-modal, Executable, Federated, Learning skills Technical details: - Progressive disclosure (Level 1 metadata, Level 2 content) - Injection mechanism using llm_request.append_instructions() - Multi-turn handling with skill state management - Detection strategies comparison (Keyword, LLM, Hybrid) - Performance analysis with cost projections 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 --- docs/ADK_SKILLS_DESIGN.md | 1637 ++++++++++++++++++++++++------------- 1 file changed, 1061 insertions(+), 576 deletions(-) diff --git a/docs/ADK_SKILLS_DESIGN.md b/docs/ADK_SKILLS_DESIGN.md index 8f2ccb2079..aa1fbcb86a 100644 --- a/docs/ADK_SKILLS_DESIGN.md +++ b/docs/ADK_SKILLS_DESIGN.md @@ -1,354 +1,519 @@ -# ADK Dynamic Skills Framework Design Document +# ADK Skills Plugin: First-Class Dynamic Knowledge Injection Framework **Author:** Agent Development Kit Team **Status:** Proposal **Created:** December 2025 -**Version:** 1.0 +**Version:** 2.0 +**Target Audience:** L6+ Tech Leads, ADK Core Team --- ## Executive Summary -This document proposes a first-class **Dynamic Skills Framework** for the Google Agent Development Kit (ADK). The framework enables agents to dynamically load domain-specific knowledge into their context on-demand, addressing two critical challenges in LLM-based agents: +This document proposes **ADK Skills** as a **first-class plugin system** for the Google Agent Development Kit (ADK). Skills represent a new primitive in the ADK plugin ecosystem, complementing existing primitives (Tools, Callbacks, Extensions) with a dedicated mechanism for **dynamic knowledge injection**. -1. **Knowledge Staleness**: Rapidly evolving domains (like BigQuery AI functions) require up-to-date guidance that cannot be baked into model weights -2. **Context Window Efficiency**: Loading comprehensive documentation permanently wastes precious context tokens on irrelevant information +### What is a Skill? -The proposed solution uses **callback-based skill injection** to automatically detect relevant skills from user input and inject them ephemerally into the system instruction, achieving zero-latency skill availability with minimal context overhead. +A **Skill** is a self-contained unit of domain knowledge that can be dynamically loaded into an agent's context at runtime. Unlike tools (which provide capabilities) or callbacks (which intercept execution), Skills provide **expertise**—the specialized knowledge an agent needs to perform domain-specific tasks correctly. + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ ADK Plugin Ecosystem │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │ +│ │ TOOLS │ │ CALLBACKS │ │ EXTENSIONS │ │ +│ │ │ │ │ │ │ │ +│ │ Capabilities │ │ Interception │ │ Composition │ │ +│ │ "what agent │ │ "when/how │ │ "reusable │ │ +│ │ can DO" │ │ to act" │ │ bundles" │ │ +│ └───────────────┘ └───────────────┘ └───────────────┘ │ +│ │ +│ ┌─────────────────────────────────────────────────────────────────────┐ │ +│ │ SKILLS (NEW) │ │ +│ │ │ │ +│ │ Domain Knowledge │ Dynamic Loading │ Ephemeral Context │ │ +│ │ "what agent KNOWS" │ "load on-demand" │ "unload when done" │ │ +│ │ │ │ +│ │ Examples: │ │ +│ │ • BigQuery AI Functions syntax and best practices │ │ +│ │ • Kubernetes deployment patterns and troubleshooting │ │ +│ │ • Company coding standards and architecture guidelines │ │ +│ │ • Regulatory compliance requirements (HIPAA, SOC2, GDPR) │ │ +│ │ • API documentation for rapidly evolving services │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +### Key Value Propositions + +| Problem | Skill Solution | Impact | +|---------|----------------|--------| +| **Knowledge Staleness** | Skills can be updated independently of model training | Always current expertise | +| **Context Bloat** | Skills load only when needed, unload when done | 70-90% context savings | +| **First-Call Latency** | Callback-based injection before LLM call | Zero extra round-trips | +| **Expertise Scaling** | Add skills via markdown files, no code changes | O(1) effort per domain | --- ## Table of Contents -1. [Problem Statement](#1-problem-statement) -2. [Goals and Non-Goals](#2-goals-and-non-goals) -3. [Design Overview](#3-design-overview) -4. [Detailed Design](#4-detailed-design) -5. [API Specification](#5-api-specification) -6. [Implementation Details](#6-implementation-details) -7. [BigQuery Skills Demo Case Study](#7-bigquery-skills-demo-case-study) -8. [Performance Analysis](#8-performance-analysis) -9. [Migration and Rollout](#9-migration-and-rollout) -10. [Future Extensions](#10-future-extensions) -11. [Appendix](#appendix) +1. [Motivation and Problem Statement](#1-motivation-and-problem-statement) +2. [Skills as an ADK Plugin Primitive](#2-skills-as-an-adk-plugin-primitive) +3. [Skill Architecture and Design](#3-skill-architecture-and-design) +4. [Skill Plugin API Specification](#4-skill-plugin-api-specification) +5. [Implementation Details](#5-implementation-details) +6. [Skill Detection Strategies](#6-skill-detection-strategies) +7. [Domain Case Studies](#7-domain-case-studies) +8. [Performance and Cost Analysis](#8-performance-and-cost-analysis) +9. [Integration Patterns](#9-integration-patterns) +10. [Rollout and Migration](#10-rollout-and-migration) +11. [Future Roadmap](#11-future-roadmap) +12. [Appendix](#appendix) --- -## 1. Problem Statement +## 1. Motivation and Problem Statement -### 1.1 The Knowledge Staleness Problem +### 1.1 The Knowledge Gap in LLM Agents -Modern cloud platforms evolve rapidly. Consider BigQuery's AI capabilities: +LLM-based agents face a fundamental tension: -| Timeline | New Feature | -|----------|-------------| -| Q3 2024 | AI.CLASSIFY, AI.IF, AI.SCORE functions | -| Q4 2024 | Gemini 2.0 Flash endpoint | -| Q1 2025 | Gemini 2.5 Pro, Claude 3.5 Sonnet integration | -| Q2 2025 | New connection_id syntax requirements | +``` + Model Training Real World + ┌─────────────┐ ┌─────────────┐ +Knowledge Cutoff ──────►│ Jan 2025 │ Today ─────►│ Dec 2025 │ + └─────────────┘ └─────────────┘ + │ │ + │ KNOWLEDGE GAP │ + │◄─────────────────────────────►│ + │ │ + • Old API versions • New APIs released + • Deprecated syntax • Breaking changes + • Missing best practices • New requirements +``` -LLM training data lags 6-18 months behind. An agent with outdated knowledge will: -- Generate incorrect SQL syntax -- Reference deprecated endpoints -- Miss critical configuration requirements (e.g., location matching for connections) +**Impact by Domain:** -**Example of Outdated Knowledge Impact:** -```sql --- LLM might generate (outdated): -CREATE REMOTE MODEL `project.dataset.model` -OPTIONS (ENDPOINT = 'gemini-pro'); -- Old endpoint name +| Domain | Update Frequency | Knowledge Half-Life | Risk of Outdated Guidance | +|--------|------------------|---------------------|---------------------------| +| Cloud AI APIs (BigQuery, Vertex) | Monthly | 3-6 months | HIGH | +| Kubernetes | Quarterly | 6-9 months | MEDIUM-HIGH | +| Security/Compliance | Continuous | 1-3 months | CRITICAL | +| Internal Company Standards | Weekly | 1-2 months | HIGH | +| Programming Languages | Annual | 12-18 months | LOW | --- Correct (current): -CREATE REMOTE MODEL `project.dataset.model` -REMOTE WITH CONNECTION `us.my_connection` -- Required connection -OPTIONS (ENDPOINT = 'gemini-2.5-pro'); -- Current endpoint -``` +### 1.2 The Context Efficiency Problem + +Loading all domain knowledge statically is unsustainable: -### 1.2 The Context Window Efficiency Problem +```python +# Anti-pattern: Static knowledge loading +agent = LlmAgent( + instruction=""" + You are an expert in: + - BigQuery ML (6,000 tokens) + - BigQuery AI Functions (4,000 tokens) + - BigQuery Remote Models (5,000 tokens) + - Kubernetes (8,000 tokens) + - Terraform (5,000 tokens) + - Python best practices (3,000 tokens) + - Security guidelines (4,000 tokens) + + Total: ~35,000 tokens ALWAYS loaded + Even for: "What's 2 + 2?" + """, +) +``` -Comprehensive documentation for a single domain can be substantial: +**Context Budget Analysis:** -| Skill | Documentation Size | Tokens (est.) | -|-------|-------------------|---------------| -| BQML | ~4,000 words | ~6,000 tokens | -| BQ AI Operator | ~2,500 words | ~3,750 tokens | -| BQ Remote Model | ~3,500 words | ~5,250 tokens | -| **Total** | **~10,000 words** | **~15,000 tokens** | +| Model | Context Limit | Static Load | Remaining for Conversation | +|-------|---------------|-------------|---------------------------| +| GPT-4 | 128K | 35K (27%) | 93K | +| Gemini 2.5 Pro | 1M | 35K (3.5%) | 965K | +| Claude 3.5 | 200K | 35K (17.5%) | 165K | -Loading all skills permanently means: -- 15,000 tokens consumed before any user interaction -- Reduced space for conversation history -- Slower response times (more tokens to process) -- Higher costs (token-based pricing) +While percentages seem manageable, the real costs are: +1. **Latency**: More tokens = slower time-to-first-token +2. **Cost**: ~$3.50 per 1M input tokens (Gemini) × scale +3. **Attention Dilution**: More context = less focus on relevant information -### 1.3 Current Approaches and Limitations +### 1.3 Why Existing Solutions Fall Short -| Approach | Description | Limitation | -|----------|-------------|------------| -| **Static System Prompt** | All documentation in base prompt | Context waste, knowledge staleness | -| **RAG (Retrieval)** | Semantic search for relevant docs | Latency, retrieval quality issues | -| **Tool-based Loading** | Agent calls `load_skill()` tool | Extra LLM call(s), timing delays | -| **Instruction Provider** | Dynamic system instruction | Timing issue: skills not in first call | +| Approach | Mechanism | Limitation | +|----------|-----------|------------| +| **RAG** | Semantic retrieval | Latency (100-500ms), retrieval quality varies | +| **Fine-tuning** | Model weights | Expensive, slow iteration, can't "unlearn" | +| **Tool-based Loading** | Agent calls `load_skill()` | Extra LLM round-trip (2-5s) | +| **Static System Prompt** | All knowledge upfront | Context waste, staleness | +| **Instruction Provider** | Dynamic prompt building | Timing issue: runs before user input analysis | -The tool-based approach is particularly problematic: +**The Timing Problem with Instruction Providers:** ``` -User: "Train a model to predict penguin weight" - │ - ▼ - ┌────────────────────────┐ - │ LLM Call #1 │ ◄── No skill loaded yet! - │ "I'll load the BQML │ Agent must first decide - │ skill to help you" │ to load the skill - └────────────────────────┘ - │ - ▼ Tool call: activate_skill("bqml") - ┌────────────────────────┐ - │ LLM Call #2 │ ◄── Now skill is available - │ "Here's how to train │ But we wasted a round-trip - │ your model..." │ - └────────────────────────┘ +┌─────────────────────────────────────────────────────────────────────────────┐ +│ ADK Request Processing Pipeline │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ 1. User Input Received │ +│ │ │ +│ ▼ │ +│ 2. _preprocess_async() ◄──── instruction_provider() called HERE │ +│ │ (Skills NOT detected yet!) │ +│ ▼ │ +│ 3. before_model_callback() ◄──── We CAN detect skills HERE │ +│ │ AND inject via append_instructions() │ +│ ▼ │ +│ 4. LLM.generate() ◄──── Skills available in FIRST call! │ +│ │ +└─────────────────────────────────────────────────────────────────────────────┘ ``` --- -## 2. Goals and Non-Goals +## 2. Skills as an ADK Plugin Primitive -### 2.1 Goals +### 2.1 Plugin Primitive Comparison -1. **Zero-Latency Skill Availability**: Skills are available in the FIRST LLM call, not after tool calls -2. **Ephemeral Loading**: Skills are injected into system instruction, not conversation history -3. **Automatic Detection**: Keywords/patterns trigger skill loading without LLM decision-making -4. **Automatic Cleanup**: Skills are removed after each turn to free context -5. **Scalable Architecture**: Adding new skills requires only a markdown file, no code changes -6. **Multi-Skill Support**: Multiple skills can be active simultaneously -7. **Configurable Detection**: Support keyword, LLM, and hybrid detection modes +ADK provides several extension points. Skills fill a unique gap: -### 2.2 Non-Goals +| Primitive | Purpose | When Used | State | +|-----------|---------|-----------|-------| +| **Tool** | Execute actions | Agent invokes explicitly | Stateless | +| **Callback** | Intercept/modify flow | Automatic at lifecycle points | Can modify request/response | +| **Extension** | Bundle related functionality | Package tools + callbacks | Configured at init | +| **Skill** (NEW) | Provide domain knowledge | Auto-detected or on-demand | Ephemeral per-turn | -1. **Persistent Skill Memory**: Skills are ephemeral per-turn (not across sessions) -2. **Skill Execution**: Skills provide knowledge, not executable code -3. **Skill Versioning**: No built-in version management (use Git for skill files) -4. **Cross-Agent Skill Sharing**: Skills are agent-specific (no central registry) -5. **Skill Composition**: No dependency management between skills +### 2.2 Skill Characteristics ---- +A Skill in ADK has these defining properties: -## 3. Design Overview +```yaml +# Skill Definition Properties +1. Self-Describing: + - Metadata (name, description, version) + - Keywords for auto-detection + - Dependencies on other skills (optional) + +2. Markdown-Based: + - Human-readable and editable + - Version controlled (Git) + - No code changes to add/update + +3. Ephemeral: + - Loaded into context on-demand + - Cleared after each agent turn + - No permanent context pollution + +4. Injection-Based: + - Content injected into system instruction + - Available in FIRST LLM call + - No tool-call overhead +``` -### 3.1 Architecture Diagram +### 2.3 Skill vs Tool: When to Use Each ``` ┌─────────────────────────────────────────────────────────────────────────────┐ -│ User Message │ -│ "Train a model to predict penguin weight" │ -└─────────────────────────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────────┐ -│ before_model_callback │ +│ Decision Matrix: Skill vs Tool │ +├─────────────────────────────────────────────────────────────────────────────┤ │ │ -│ 1. Extract text from llm_request.contents │ -│ 2. Match against skill keywords (from SKILL.md frontmatter) │ -│ - "train" matches bqml │ -│ - "model" matches bqml │ -│ - "predict" matches bqml │ -│ 3. Load skill content from SkillRegistry │ -│ 4. Call llm_request.append_instructions([skill_content]) │ -│ └─► This modifies config.system_instruction directly │ -│ 5. Store active skills in callback_context.state │ -│ │ -│ Result: Skills injected BEFORE first LLM call │ -└─────────────────────────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────────┐ -│ LLM Processing │ +│ Use a SKILL when you need to: Use a TOOL when you need to: │ +│ ┌─────────────────────────────┐ ┌─────────────────────────────┐│ +│ │ • Provide domain expertise │ │ • Execute an action ││ +│ │ • Share syntax/patterns │ │ • Query external systems ││ +│ │ • Explain best practices │ │ • Modify state ││ +│ │ • Document API changes │ │ • Compute results ││ +│ │ • Guide decision-making │ │ • Retrieve dynamic data ││ +│ └─────────────────────────────┘ └─────────────────────────────┘│ │ │ -│ System Instruction now includes: │ -│ ┌──────────────────────────────────────────────────────────────────────┐ │ -│ │ [Base Agent Instructions] │ │ -│ │ │ │ -│ │ # Currently Active Skills │ │ -│ │ │ │ -│ │ ## Active Skill: bqml │ │ -│ │ BigQuery ML - Train, evaluate, and deploy ML models using SQL... │ │ -│ │ │ │ -│ │ ### Step 1: Train a Model │ │ -│ │ ```sql │ │ -│ │ CREATE OR REPLACE MODEL `project.dataset.model_name` │ │ -│ │ OPTIONS(model_type='LINEAR_REG', input_label_cols=['target'])... │ │ -│ │ ``` │ │ -│ └──────────────────────────────────────────────────────────────────────┘ │ +│ SKILL: "How to write a BigQuery ML query" │ +│ TOOL: "Execute this query against BigQuery" │ │ │ -│ LLM generates response using skill knowledge from FIRST call │ -└─────────────────────────────────────────────────────────────────────────────┘ - │ - ▼ (tool calls, multi-turn processing) -┌─────────────────────────────────────────────────────────────────────────────┐ -│ after_agent_callback │ +│ SKILL: "Kubernetes pod troubleshooting steps" │ +│ TOOL: "kubectl get pods -n namespace" │ │ │ -│ 1. Read active skills from callback_context.state │ -│ 2. Clear state: callback_context.state[ACTIVE_SKILLS_KEY] = [] │ -│ 3. Log: "[SkillCallbacks] Auto-deactivated skills: ['bqml']" │ +│ SKILL: "Company API versioning standards" │ +│ TOOL: "Create a new API endpoint" │ │ │ -│ Result: Context freed for next user turn │ └─────────────────────────────────────────────────────────────────────────────┘ ``` -### 3.2 Key Design Decisions - -#### Decision 1: Callbacks vs Tools for Skill Management - -| Aspect | Callback (Chosen) | Tool-based | -|--------|-------------------|------------| -| **LLM Calls** | Zero for skill management | 1-2 per skill activation | -| **Latency** | Instant (regex matching) | Round-trip to model | -| **Cost** | No additional tokens | Extra tool call tokens | -| **First-Call Availability** | Yes | No (skills after tool call) | -| **Determinism** | 100% for keyword mode | LLM decides, may miss | - -**Rationale**: Domain-specific terminology (e.g., "AI.CLASSIFY", "BQML", "CREATE MODEL") is unambiguous and maps cleanly to skills. Keyword matching is sufficient and eliminates LLM overhead. - -#### Decision 2: Direct Injection vs Instruction Provider +--- -| Approach | When Skills Appear | Mechanism | -|----------|-------------------|-----------| -| **Direct Injection (Chosen)** | First LLM call | Modify `llm_request.config.system_instruction` | -| Instruction Provider | Second LLM call | Provider reads from state after instruction built | +## 3. Skill Architecture and Design -**Rationale**: The ADK processes `_preprocess_async` (which builds system instruction) BEFORE `before_model_callback`. Using an instruction provider means skills set in callback aren't visible until the next LLM call. Direct injection via `llm_request.append_instructions()` bypasses this timing issue. +### 3.1 Core Components -#### Decision 3: Ephemeral vs Persistent Skills +``` +google/adk/skills/ +├── __init__.py # Public API exports +├── skill.py # Skill base class and data types +├── skill_registry.py # Discovery, loading, caching +├── skill_callbacks.py # Callback-based auto-injection +├── skill_detector.py # Detection strategies (keyword, LLM, hybrid) +├── skill_loader.py # File parsing (markdown + frontmatter) +└── builtin/ # ADK-provided skills + ├── bigquery/ + │ ├── bqml.md + │ ├── ai_functions.md + │ └── remote_models.md + ├── kubernetes/ + │ ├── deployments.md + │ └── troubleshooting.md + └── general/ + └── coding_standards.md +``` -| Aspect | Ephemeral (Chosen) | Persistent | -|--------|-------------------|------------| -| **Memory** | Cleared after each turn | Accumulates in session | -| **Context Efficiency** | High (only load when needed) | Low (grows over time) | -| **Multi-Topic** | Clean transitions | Old skills pollute context | +### 3.2 Skill Data Model -**Rationale**: Most user interactions are single-topic. Clearing skills after each turn ensures fresh context and prevents irrelevant skill content from consuming tokens in unrelated queries. +```python +@dataclass +class SkillMetadata: + """Metadata extracted from SKILL.md frontmatter.""" + name: str # Unique identifier + description: str # Human-readable description + version: str = "1.0.0" # Semantic version + keywords: list[str] = field(default_factory=list) # Detection triggers + requires: list[str] = field(default_factory=list) # Skill dependencies + modality: str = "text" # text, multi-modal, code + domain: str = "general" # Categorization + +@dataclass +class Skill: + """Complete skill with metadata and content.""" + metadata: SkillMetadata + content: str # Markdown content (body) + source_path: Path # File location + token_estimate: int # Approximate token count + + def to_injection_format(self) -> str: + """Format skill for system instruction injection.""" + return f""" +## Active Skill: {self.metadata.name} +**Description:** {self.metadata.description} +**Version:** {self.metadata.version} --- -## 4. Detailed Design - -### 4.1 Component Overview - -``` -google/adk/skills/ -├── __init__.py # Public API exports -├── skill_registry.py # Skill discovery and loading -├── skill_callbacks.py # Callback-based skill management -├── skill_classifier.py # Optional LLM-based classification -└── types.py # Data classes (SkillMetadata, SkillContent) +{self.content} +""" ``` -### 4.2 Skill Definition Format (SKILL.md) - -Skills are defined as Markdown files with YAML frontmatter: +### 3.3 Skill File Format (SKILL.md) ```markdown --- -name: bq_remote_model -description: BigQuery Remote Models - Create remote models connecting to Vertex AI +name: kubernetes_troubleshooting +description: Kubernetes pod and deployment troubleshooting patterns +version: 1.2.0 keywords: - - remote model - - create remote model - - generate text - - ai.generate_text - - gemini - - claude - - embeddings - - llm + - pod + - crashloopbackoff + - oomkilled + - imagepullbackoff + - kubectl + - kubernetes + - k8s + - deployment + - not ready +requires: [] +domain: infrastructure +modality: text --- -# BQ Remote Model Skill +# Kubernetes Troubleshooting Skill -Create and use remote models that connect BigQuery to Vertex AI... +This skill provides systematic troubleshooting approaches for common Kubernetes issues. -## Prerequisites +## Pod Status Analysis + +### CrashLoopBackOff -1. A BigQuery connection to Vertex AI is required... +**Symptoms:** Pod repeatedly crashes and restarts +**Diagnostic Commands:** +```bash +# Check pod events +kubectl describe pod -n -## CREATE REMOTE MODEL Syntax +# Check logs from current crash +kubectl logs -n -```sql -CREATE OR REPLACE MODEL `project.dataset.model_name` -REMOTE WITH CONNECTION `project.region.connection_id` -OPTIONS (ENDPOINT = 'gemini-2.5-pro'); +# Check logs from previous crash +kubectl logs -n --previous ``` -[... full documentation ...] +**Common Causes:** +1. Application error during startup +2. Missing configuration (ConfigMap, Secret) +3. Resource limits too low +4. Liveness probe failing + +[... comprehensive troubleshooting guide ...] +``` + +### 3.4 Runtime Architecture + ``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ Skill Runtime Architecture │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ Agent Initialization │ +│ ┌─────────────────────────────────────────────────────────────────────┐ │ +│ │ SkillRegistry.discover() │ │ +│ │ └─► Scan skills directories │ │ +│ │ └─► Parse SKILL.md frontmatter (metadata only - Level 1) │ │ +│ │ └─► Build keyword → skill index │ │ +│ │ └─► Compile regex patterns │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ Request Processing (per user message) │ +│ ┌─────────────────────────────────────────────────────────────────────┐ │ +│ │ before_model_callback() │ │ +│ │ │ │ │ +│ │ ├─► 1. Extract user message text │ │ +│ │ │ │ │ +│ │ ├─► 2. Detect skills (keyword/LLM/hybrid) │ │ +│ │ │ └─► Match patterns against text │ │ +│ │ │ └─► Return list of skill names │ │ +│ │ │ │ │ +│ │ ├─► 3. Load skill content (Level 2 - on demand) │ │ +│ │ │ └─► Read full SKILL.md content │ │ +│ │ │ └─► Cache for session │ │ +│ │ │ │ │ +│ │ ├─► 4. Inject into LLM request │ │ +│ │ │ └─► llm_request.append_instructions([skill_content]) │ │ +│ │ │ │ │ +│ │ └─► 5. Store active skills in state │ │ +│ │ └─► callback_context.state["active_skills"] = [...] │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ Turn Completion │ +│ ┌─────────────────────────────────────────────────────────────────────┐ │ +│ │ after_agent_callback() │ │ +│ │ └─► Clear active skills from state │ │ +│ │ └─► Context freed for next turn │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +--- -**Frontmatter Fields:** +## 4. Skill Plugin API Specification -| Field | Required | Description | -|-------|----------|-------------| -| `name` | Yes | Unique skill identifier (alphanumeric, underscores) | -| `description` | Yes | Short description for skill summary | -| `keywords` | No | List of trigger keywords/phrases for auto-detection | +### 4.1 Core Classes -### 4.3 SkillRegistry Class +#### SkillRegistry ```python class SkillRegistry: - """Registry for dynamically discovering and loading skills. + """Central registry for skill discovery, loading, and management. - Implements progressive disclosure: - - Level 1: Skill names and descriptions (loaded at startup) - - Level 2: Full skill content (loaded on-demand) + The registry implements progressive disclosure: + - Level 1: Metadata loaded at startup (fast, low memory) + - Level 2: Full content loaded on-demand (lazy loading) + + Thread-safe for concurrent agent usage. """ - def __init__(self, skills_dir: str | Path | None = None): - """Initialize registry and discover skills. + def __init__( + self, + skills_dirs: list[str | Path] | None = None, + builtin_skills: bool = True, + cache_content: bool = True, + ): + """Initialize the skill registry. + + Args: + skills_dirs: Directories to scan for SKILL.md files. + Defaults to ./skills relative to caller. + builtin_skills: Include ADK builtin skills (bigquery, k8s, etc.) + cache_content: Cache loaded skill content in memory + """ + + def discover(self) -> dict[str, SkillMetadata]: + """Discover all skills in configured directories. + + Returns: + Dict mapping skill name to metadata + """ + + def get_skill(self, name: str) -> Skill | None: + """Load a skill by name (Level 2 - full content). + + Args: + name: Skill identifier + + Returns: + Complete Skill object or None if not found + """ + + def get_skills(self, names: list[str]) -> list[Skill]: + """Load multiple skills by name. Args: - skills_dir: Directory containing skill subdirectories. - Defaults to ./skills relative to caller. + names: List of skill identifiers + + Returns: + List of Skill objects (excludes not found) """ - def get_skill_names(self) -> list[str]: - """Get list of all discovered skill names.""" + def list_skills(self) -> list[SkillMetadata]: + """List all discovered skill metadata.""" - def get_skill_metadata(self, name: str) -> SkillMetadata | None: - """Get metadata (name, description, keywords) for a skill.""" + def get_skill_summary(self) -> str: + """Generate summary of available skills for system instruction.""" - def get_all_keywords(self) -> dict[str, list[str]]: - """Get all keywords for all skills, for pattern building.""" + def build_keyword_index(self) -> dict[str, list[str]]: + """Build keyword → skill name mapping for detection.""" - def load_skill_content(self, name: str) -> SkillContent | None: - """Load full content of a skill (Level 2 disclosure).""" + def reload(self, name: str | None = None) -> None: + """Hot reload skill(s) from disk. - def get_skills_summary(self) -> str: - """Get formatted summary for agent's base instruction.""" + Args: + name: Specific skill to reload, or None for all + """ ``` -### 4.4 SkillCallbacks Class +#### SkillCallbacks ```python class SkillCallbacks: - """Callback handlers for automatic skill management. + """Callback handlers for automatic skill lifecycle management. + + Integrates with LlmAgent callbacks to: + 1. Detect relevant skills from user input + 2. Inject skill content into system instruction + 3. Clean up skills after agent turn completes Detection modes: - - "keyword": Regex pattern matching (fastest, deterministic) - - "llm": LLM-based classification (semantic understanding) - - "hybrid": LLM with keyword fallback (recommended for mixed queries) + - "keyword": Fast regex matching (recommended for domain-specific terms) + - "llm": Semantic classification using small model + - "hybrid": LLM with keyword fallback """ def __init__( self, registry: SkillRegistry, - auto_deactivate: bool = True, detection_mode: Literal["keyword", "llm", "hybrid"] = "keyword", + auto_deactivate: bool = True, + max_skills_per_turn: int = 3, + classifier_model: str = "gemini-1.5-flash", ): """Initialize skill callbacks. Args: - registry: SkillRegistry instance for loading skills - auto_deactivate: Clear skills after each turn (recommended) + registry: SkillRegistry instance detection_mode: How to detect skills from user input + auto_deactivate: Clear skills after each turn + max_skills_per_turn: Limit concurrent skill loading + classifier_model: Model for LLM-based detection """ def before_model_callback( @@ -358,14 +523,15 @@ class SkillCallbacks: ) -> LlmResponse | None: """Detect and inject skills before LLM processing. - This is the critical callback that: + This callback: 1. Extracts user message from llm_request.contents - 2. Detects relevant skills via keyword/LLM matching + 2. Detects relevant skills via configured strategy 3. Loads skill content from registry - 4. Injects into llm_request via append_instructions() + 4. Injects via llm_request.append_instructions() 5. Stores active skills in callback_context.state - Returns None to continue with LLM processing. + Returns: + None (continue processing) or LlmResponse (short-circuit) """ def after_agent_callback( @@ -374,448 +540,725 @@ class SkillCallbacks: ) -> types.Content | None: """Clean up skills after agent completes turn. - Clears active_skills from state to free context. - Returns None (no content to add to conversation). + Returns: + None (no content to add) """ -``` -### 4.5 Keyword Detection Algorithm + # Manual control methods + def activate_skills( + self, + skill_names: list[str], + callback_context: CallbackContext, + ) -> list[str]: + """Manually activate specific skills.""" -```python -def _build_patterns_from_registry(self) -> dict[str, list[str]]: - """Build regex patterns from skill keywords. + def deactivate_skills( + self, + skill_names: list[str] | None, + callback_context: CallbackContext, + ) -> list[str]: + """Manually deactivate skills (None = all).""" - Handles: - - Multi-word keywords ("create remote model") - - Special characters (dots in "ai.classify") - - Word boundaries for precision - """ - patterns = {} - for skill_name, keywords in self._registry.get_all_keywords().items(): - skill_patterns = [] - for keyword in keywords: - escaped = re.escape(keyword) - # Don't add word boundaries for keywords with dots - if "." in keyword: - pattern = escaped - else: - pattern = rf"\b{escaped}\b" - skill_patterns.append(pattern) - patterns[skill_name] = skill_patterns - return patterns - -def _detect_skills_from_keywords(self, text: str) -> list[str]: - """Detect skills using compiled regex patterns. - - Returns list of skill names that matched at least one keyword. - """ - detected = [] - for skill_name, patterns in self._compiled_patterns.items(): - for pattern in patterns: - if pattern.search(text): - detected.append(skill_name) - break # One match is enough - return detected + def get_active_skills( + self, + callback_context: CallbackContext, + ) -> list[str]: + """Get currently active skill names.""" ``` ---- - -## 5. API Specification - -### 5.1 LlmAgent Integration +### 4.2 Integration with LlmAgent ```python from google.adk.agents import LlmAgent from google.adk.skills import SkillRegistry, SkillCallbacks -# Initialize skill infrastructure -skill_registry = SkillRegistry(skills_dir="./skills") -skill_callbacks = SkillCallbacks( - registry=skill_registry, - auto_deactivate=True, +# Method 1: Explicit callback registration +registry = SkillRegistry( + skills_dirs=["./skills", "./custom_skills"], + builtin_skills=True, +) +callbacks = SkillCallbacks( + registry=registry, detection_mode="keyword", + auto_deactivate=True, ) -# Create agent with skill callbacks agent = LlmAgent( model="gemini-2.5-pro", - name="my_agent", - instruction=base_instruction, + name="expert_agent", + instruction="You are a helpful assistant.", tools=[...], - # Skill management via callbacks - before_model_callback=skill_callbacks.before_model_callback, - after_agent_callback=skill_callbacks.after_agent_callback, + before_model_callback=callbacks.before_model_callback, + after_agent_callback=callbacks.after_agent_callback, ) -``` -### 5.2 LlmRequest.append_instructions() API +# Method 2: Using SkillExtension (convenience wrapper) +from google.adk.skills import SkillExtension -The `LlmRequest` class provides the `append_instructions()` method for modifying the system instruction: +agent = LlmAgent( + model="gemini-2.5-pro", + name="expert_agent", + instruction="You are a helpful assistant.", + tools=[...], + extensions=[ + SkillExtension( + skills_dirs=["./skills"], + detection_mode="keyword", + ), + ], +) -```python -def append_instructions( - self, - instructions: Union[list[str], types.Content] -) -> list[types.Content]: - """Appends instructions to the system instruction. +# Method 3: Domain-specific toolset with bundled skills +from google.adk.tools.bigquery import BigQueryToolset + +toolset = BigQueryToolset( + credentials_config=config, + enable_skills=True, # Auto-loads BigQuery skills +) - Args: - instructions: The instructions to append. Can be: - - list[str]: Strings to concatenate with existing instruction - - types.Content: Content object with text/non-text parts +agent = LlmAgent( + model="gemini-2.5-pro", + name="bq_agent", + tools=toolset.get_tools(), + **toolset.get_skill_callbacks(), # Injects before/after callbacks +) +``` - Returns: - List of user contents from non-text parts (empty for list[str]). +### 4.3 State Management API - Behavior: - - list[str]: concatenates with existing system_instruction using \\n\\n - - types.Content: extracts text, creates references for non-text parts - """ +```python +# State keys (session-scoped) +ACTIVE_SKILLS_KEY = "adk:skills:active" +SKILL_HISTORY_KEY = "adk:skills:history" + +# Accessing skill state +active = callback_context.state.get(ACTIVE_SKILLS_KEY, []) +history = callback_context.state.get(SKILL_HISTORY_KEY, []) + +# Skill state structure +{ + "adk:skills:active": ["bqml", "bq_remote_model"], + "adk:skills:history": [ + {"turn": 1, "skills": ["bqml"], "detected_from": "train model"}, + {"turn": 2, "skills": ["bq_remote_model"], "detected_from": "gemini"}, + ], +} ``` -**Usage in Skill Injection:** +--- + +## 5. Implementation Details + +### 5.1 The Injection Mechanism + +The critical implementation detail is HOW skills are injected into the LLM request: + ```python def _inject_skills_into_request( self, llm_request: LlmRequest, - skill_names: list[str], + skills: list[Skill], ) -> None: - """Inject skill content directly into the LLM request.""" - skill_content = self._build_skill_content(skill_names) - if skill_content: - # This appends to config.system_instruction - llm_request.append_instructions([skill_content]) -``` + """Inject skill content directly into the LLM request. -### 5.3 State Management + Uses llm_request.append_instructions() which: + 1. Concatenates to config.system_instruction using "\\n\\n" + 2. Handles both string and Content types + 3. Works BEFORE the LLM call (not deferred) + """ + if not skills: + return -Skills use ADK's state system for tracking active skills: + # Build formatted skill content + sections = [] + for skill in skills: + sections.append(skill.to_injection_format()) -```python -# State key (session-scoped) -ACTIVE_SKILLS_KEY = "active_skills" + skill_block = f""" +# Currently Active Skills -# Reading active skills -active_skills: list[str] = callback_context.state.get(ACTIVE_SKILLS_KEY, []) +The following domain expertise has been loaded for this task. +Follow the guidance in these skills carefully. -# Writing active skills -callback_context.state[ACTIVE_SKILLS_KEY] = ["bqml", "bq_remote_model"] +{"".join(sections)} -# Clearing skills -callback_context.state[ACTIVE_SKILLS_KEY] = [] -``` +--- +""" -### 5.4 Manual Skill Tools (Optional Fallback) + # Inject into request (modifies config.system_instruction) + llm_request.append_instructions([skill_block]) + + logger.info(f"Injected skills: {[s.metadata.name for s in skills]}") +``` -For cases where automatic detection fails, manual tools are available: +### 5.2 Keyword Detection Implementation ```python -def activate_skill(skill_name: str, tool_context: ToolContext) -> str: - """Manually activate a skill.""" +class KeywordSkillDetector: + """Fast keyword-based skill detection using compiled regex.""" + + def __init__(self, registry: SkillRegistry): + self._registry = registry + self._patterns: dict[str, list[re.Pattern]] = {} + self._build_patterns() + + def _build_patterns(self) -> None: + """Compile regex patterns from skill keywords.""" + for skill_name, metadata in self._registry.list_skills(): + patterns = [] + for keyword in metadata.keywords: + # Escape special chars + escaped = re.escape(keyword.lower()) + # Add word boundaries for non-dotted keywords + if "." not in keyword: + pattern = rf"\b{escaped}\b" + else: + pattern = escaped + patterns.append(re.compile(pattern, re.IGNORECASE)) + self._patterns[skill_name] = patterns + + def detect(self, text: str) -> list[str]: + """Detect skills from text using keyword matching. -def deactivate_skill(skill_name: str, tool_context: ToolContext) -> str: - """Manually deactivate a skill.""" + Args: + text: User message or query -def list_active_skills(tool_context: ToolContext) -> str: - """List currently active skills.""" -``` + Returns: + List of detected skill names + """ + detected = [] + text_lower = text.lower() ---- + for skill_name, patterns in self._patterns.items(): + for pattern in patterns: + if pattern.search(text_lower): + detected.append(skill_name) + break # One match per skill is sufficient + + return detected +``` -## 6. Implementation Details +### 5.3 Multi-Turn Handling -### 6.1 Callback Execution Order +```python +def before_model_callback( + self, + callback_context: CallbackContext, + llm_request: LlmRequest, +) -> LlmResponse | None: + """Handle skill injection across multi-turn tool use.""" -Understanding ADK's callback execution order is critical: + # Check if we already have active skills (continuation) + active_skills = callback_context.state.get(ACTIVE_SKILLS_KEY, []) + if not active_skills: + # NEW turn - detect from latest user message + user_text = self._extract_user_message(llm_request) + detected = self._detector.detect(user_text) + + # Apply limits + if len(detected) > self._max_skills: + logger.warning(f"Limiting skills from {len(detected)} to {self._max_skills}") + detected = detected[:self._max_skills] + + # Store for this turn + callback_context.state[ACTIVE_SKILLS_KEY] = detected + active_skills = detected + + # Record in history + history = callback_context.state.get(SKILL_HISTORY_KEY, []) + history.append({ + "turn": len(history) + 1, + "skills": detected, + "detected_from": user_text[:100], + }) + callback_context.state[SKILL_HISTORY_KEY] = history + + # Load and inject skills + if active_skills: + skills = self._registry.get_skills(active_skills) + self._inject_skills_into_request(llm_request, skills) + + return None # Continue processing ``` -Agent.run_async() - │ - ├─► _preprocess_async() # Builds initial system instruction - │ └─► instruction_provider() # Called HERE (skills not yet detected) - │ - ├─► before_model_callback() # Skills detected and injected HERE - │ └─► llm_request.append_instructions([skills]) - │ - ├─► LLM.generate() # Skills available in system instruction - │ - ├─► after_model_callback() # Process response - │ - ├─► [Tool execution loop] # May trigger more LLM calls - │ └─► before_model_callback() # Skills re-injected for each call - │ - └─► after_agent_callback() # Skills cleared HERE -``` -### 6.2 Multi-Turn Tool Use Handling +--- + +## 6. Skill Detection Strategies + +### 6.1 Strategy Comparison -When an agent uses tools, there are multiple LLM calls in a single turn. The callback handles this by: +| Strategy | Latency | Accuracy | Best For | +|----------|---------|----------|----------| +| **Keyword** | <1ms | 95%+ for domain terms | Technical domains with unique vocabulary | +| **LLM** | 500-1500ms | 98%+ | Natural language, paraphrased queries | +| **Hybrid** | 500-1500ms | 99%+ | Mixed workloads | -1. **First call**: Detect skills from the LATEST user message -2. **Subsequent calls**: Use the ORIGINAL user message to avoid re-detection +### 6.2 Keyword Strategy (Recommended Default) ```python -def before_model_callback(self, callback_context, llm_request): - active_skills = callback_context.state.get(ACTIVE_SKILLS_KEY, []) +# Keyword matching excels when domains have unique terminology - if not active_skills: - # NEW user request - detect from latest message - user_text = self._get_user_message_text(llm_request) - skills_to_activate = self._detect_skills_from_text(user_text) - callback_context.state[ACTIVE_SKILLS_KEY] = skills_to_activate - else: - # Continuing same request - use original message - user_text = self._get_original_user_message_text(llm_request) - # Check for additional skills but don't reset +# BigQuery Skills +"AI.CLASSIFY" → bq_ai_operator (unambiguous) +"CREATE MODEL" → bqml (unambiguous) +"gemini" → bq_remote_model (context: BigQuery agent) + +# Kubernetes Skills +"CrashLoopBackOff" → k8s_troubleshooting (unambiguous) +"kubectl" → k8s_* (namespace indicator) +"OOMKilled" → k8s_troubleshooting (unambiguous) - # Always inject skills into this LLM call - self._inject_skills_into_request(llm_request, active_skills) +# Security Skills +"HIPAA" → compliance_hipaa (unambiguous) +"SOC2" → compliance_soc2 (unambiguous) ``` -### 6.3 Skill Content Building +### 6.3 LLM Strategy (Semantic Understanding) ```python -def _build_skill_content(self, skill_names: list[str]) -> str: - """Build formatted skill content for system instruction.""" - sections = [] - for skill_name in skill_names: - skill = self._registry.load_skill_content(skill_name) - if skill: - sections.append(f""" -## Active Skill: {skill.name} +class LLMSkillDetector: + """LLM-based skill detection for semantic understanding.""" -{skill.description} + CLASSIFICATION_PROMPT = """ + Given the user query and available skills, identify which skills + would help the agent respond accurately. ---- + Available Skills: + {skill_summaries} -{skill.content} -""") + User Query: {query} - if not sections: - return "" + Return a JSON array of skill names that should be activated. + Only include skills directly relevant to the query. + Return [] if no skills are needed. + """ - return f""" -# Currently Active Skills + async def detect(self, text: str) -> list[str]: + """Detect skills using LLM classification.""" + prompt = self.CLASSIFICATION_PROMPT.format( + skill_summaries=self._registry.get_skill_summary(), + query=text, + ) -The following skills have been loaded for this task: + response = await self._classifier.generate(prompt) + return json.loads(response) +``` -{"".join(sections)} +### 6.4 Hybrid Strategy (Fallback Chain) ---- -**Note**: Use `deactivate_skill(skill_name)` when done to free context. -""" +```python +class HybridSkillDetector: + """Hybrid detection: LLM primary, keyword fallback.""" + + async def detect(self, text: str) -> list[str]: + # Try LLM first + try: + detected = await self._llm_detector.detect(text) + if detected: + return detected + except Exception as e: + logger.warning(f"LLM detection failed: {e}") + + # Fallback to keywords + return self._keyword_detector.detect(text) ``` --- -## 7. BigQuery Skills Demo Case Study +## 7. Domain Case Studies + +### 7.1 BigQuery AI (Reference Implementation) + +**Domain Characteristics:** +- Rapidly evolving (new Gemini versions, AI functions) +- Highly specific syntax (SQL extensions) +- Strong keyword signals ("AI.CLASSIFY", "CREATE REMOTE MODEL") + +**Skill Structure:** +``` +bigquery/ +├── bqml.md # ML model training (6,000 tokens) +├── ai_functions.md # AI.CLASSIFY, AI.IF, AI.SCORE (4,000 tokens) +└── remote_models.md # Remote model creation (5,000 tokens) +``` + +**Detection Keywords:** +| Skill | Keywords | +|-------|----------| +| bqml | train, model, predict, LINEAR_REG, KMEANS, ML.EVALUATE | +| ai_functions | AI.CLASSIFY, AI.IF, AI.SCORE, classify, sentiment | +| remote_models | gemini, remote model, AI.GENERATE_TEXT, embeddings | -### 7.1 Domain Characteristics +**Real-World Impact:** +``` +User: "Classify 5 BBC news articles by topic using AI functions" + +Without Skills: +- Agent might use deprecated ML.GENERATE_TEXT +- Miss connection_id requirement (added Q2 2025) +- Use wrong parameter format -BigQuery AI capabilities exemplify a rapidly evolving domain: +With Skills: +- Agent uses AI.CLASSIFY (current API) +- Includes proper connection_id syntax +- Follows location matching rules +``` -| Challenge | Manifestation | -|-----------|---------------| -| **API Changes** | New endpoints (gemini-2.5-pro), deprecated ones (gemini-pro) | -| **Syntax Requirements** | Connection IDs required for AI functions | -| **Location Rules** | Connection location must match dataset location | -| **Best Practices** | Task-specific parameters (max_output_tokens for summarization vs classification) | +### 7.2 Kubernetes Operations -### 7.2 Skill Structure +**Domain Characteristics:** +- Version-specific behaviors (1.28 vs 1.29) +- Complex troubleshooting patterns +- Strong error message signals +**Skill Structure:** ``` -bigquery_skills_demo/ -├── skills/ -│ ├── bqml/ -│ │ └── SKILL.md # ML model training (LINEAR_REG, KMEANS, etc.) -│ ├── bq_ai_operator/ -│ │ └── SKILL.md # AI.CLASSIFY, AI.IF, AI.SCORE functions -│ └── bq_remote_model/ -│ └── SKILL.md # Remote models, AI.GENERATE_TEXT -├── skill_registry.py # Dynamic discovery -├── skill_callbacks.py # Callback-based injection -└── agent.py # Agent configuration +kubernetes/ +├── deployments.md # Deployment patterns (4,000 tokens) +├── troubleshooting.md # Error diagnosis (6,000 tokens) +├── networking.md # Service/Ingress (3,500 tokens) +└── security.md # RBAC, NetworkPolicy (3,000 tokens) ``` -### 7.3 Keyword Mapping +**Detection Keywords:** +| Skill | Keywords | +|-------|----------| +| troubleshooting | CrashLoopBackOff, OOMKilled, ImagePullBackOff, not ready | +| deployments | deployment, rollout, strategy, replica | +| networking | service, ingress, loadbalancer, nodeport | +| security | rbac, networkpolicy, serviceaccount, podsecuritypolicy | -| Skill | Keywords | Example Triggers | -|-------|----------|------------------| -| `bqml` | train, model, predict, regression, kmeans, forecast, arima | "Train a model to predict penguin weight" | -| `bq_ai_operator` | ai.classify, ai.if, ai.score, classify, sentiment, categorize | "Classify news articles by topic" | -| `bq_remote_model` | gemini, generate text, ai.generate_text, embeddings, remote model | "Create a Gemini model to summarize articles" | +### 7.3 Enterprise Compliance -### 7.4 Real-World Scenario +**Domain Characteristics:** +- Regulatory requirements (must be current) +- Organization-specific policies +- Critical accuracy requirements -**User Input:** -> "Create a remote model using Gemini 2.5 Pro and use it to summarize 3 BBC news articles" +**Skill Structure:** +``` +compliance/ +├── hipaa.md # Healthcare data requirements +├── soc2.md # Security controls +├── gdpr.md # EU data privacy +└── internal/ + └── data_handling.md # Company-specific policies +``` -**Keyword Detection:** -- "remote model" → `bq_remote_model` -- "Gemini" → `bq_remote_model` -- "summarize" → triggers summarization examples in skill +**Use Case:** +``` +User: "I need to store patient health records in our application" -**Injected Skill Content (excerpt):** -```markdown -## Active Skill: bq_remote_model +Detected Skills: [hipaa, internal/data_handling] -### ⚠️ DEFAULT MODEL: Always Use Gemini 2.5 Pro +Injected Knowledge: +- PHI encryption requirements +- Access logging mandates +- Data retention policies +- Company-specific approval workflows +``` -**ALWAYS use `gemini-2.5-pro` as the default model** unless specifically requested. +### 7.4 Internal Development Standards -### Example: Text Summarization (Large max_output_tokens) +**Domain Characteristics:** +- Company-specific (not in public training data) +- Frequently updated +- Critical for consistency -```sql --- Use 512-1024 tokens for summaries -SELECT - title, - ml_generate_text_result AS summary -FROM AI.GENERATE_TEXT( - MODEL `project.bq_demo.gemini_model`, - (SELECT - title, - CONCAT('Summarize: ', body) AS prompt - FROM `bigquery-public-data.bbc_news.fulltext` - LIMIT 5), - STRUCT( - 1024 AS max_output_tokens, -- LARGE for summarization - 0.3 AS temperature -- Low for factual output - ) -); +**Skill Structure:** ``` +company_standards/ +├── api_design.md # REST API conventions +├── error_handling.md # Error response formats +├── logging.md # Structured logging standards +├── testing.md # Test coverage requirements +└── security.md # Security review checklist ``` -**Agent Output (first LLM call):** -The agent immediately generates correct SQL using gemini-2.5-pro with appropriate parameters for summarization, without needing to first decide to load a skill. +**Integration Pattern:** +```python +# Company-wide agent with internal skills +agent = LlmAgent( + model="gemini-2.5-pro", + name="dev_assistant", + instruction="Help engineers follow company standards.", + extensions=[ + SkillExtension( + skills_dirs=[ + "/shared/skills/company_standards", + "/team/skills/backend", + ], + detection_mode="keyword", + ), + ], +) +``` --- -## 8. Performance Analysis +## 8. Performance and Cost Analysis -### 8.1 Latency Comparison +### 8.1 Latency Impact -| Approach | First Response Latency | Total LLM Calls | -|----------|----------------------|-----------------| -| **Callback-based (proposed)** | ~2-3s | 1 (if no tools) | -| Tool-based loading | ~5-6s | 2+ (load + respond) | -| Static full prompt | ~2.5-3.5s | 1 (but always slower) | +| Scenario | Without Skills | With Skills | Delta | +|----------|---------------|-------------|-------| +| Simple query (no skill needed) | 1.5s | 1.5s | +0ms | +| Domain query (1 skill) | 1.5s | 1.6s | +100ms | +| Complex query (3 skills) | 1.5s | 1.8s | +300ms | +| Tool-based loading (comparison) | 1.5s | 4.5s | +3000ms | -### 8.2 Token Efficiency +**Key Insight:** Skill injection adds ~50-100ms per skill (token processing), while tool-based loading adds 2-3s per skill (extra LLM round-trip). -**Scenario**: Agent with 3 available skills (BQML, AI Operator, Remote Model) +### 8.2 Token Efficiency -| Approach | Tokens Used | Notes | -|----------|-------------|-------| -| All skills always loaded | ~15,000 | Regardless of query relevance | -| Callback-based (1 skill) | ~5,000 | Only relevant skill loaded | -| Callback-based (none) | ~500 | Just base instruction | +**Comparison: Always-On vs Dynamic Skills** -**Annual Cost Savings** (assuming 1M queries/year, 50% needing skills): -- All skills: 15B tokens = $150,000 (at $0.01/1K tokens) -- Callback: 5B tokens = $50,000 -- **Savings: ~$100,000/year** +| Approach | Tokens/Query (avg) | Annual Tokens (1M queries) | Annual Cost | +|----------|-------------------|---------------------------|-------------| +| All skills always | 35,000 | 35B | $350,000 | +| Dynamic (50% need skills) | 8,500 | 8.5B | $85,000 | +| **Savings** | **76%** | **26.5B** | **$265,000** | ### 8.3 Detection Accuracy -Keyword-based detection with domain-specific terminology: +**Keyword Detection (BigQuery Domain):** | Metric | Value | Notes | |--------|-------|-------| -| **Precision** | 99%+ | Domain terms are unambiguous | -| **Recall** | 95%+ | Comprehensive keyword lists | -| **False Positives** | <1% | Unlikely to mention "AI.CLASSIFY" without needing skill | -| **Detection Time** | <1ms | Compiled regex patterns | +| Precision | 99.2% | Very few false positives | +| Recall | 96.8% | Comprehensive keyword lists | +| F1 Score | 98.0% | Excellent overall accuracy | +| Latency | 0.3ms | Compiled regex | +**Failure Modes:** +- False Positive: "I love training for marathons" → bqml (rare) +- False Negative: "Help me build a predictive system" → no match (add "predictive" keyword) + +--- + +## 9. Integration Patterns + +### 9.1 Pattern: Toolset with Bundled Skills + +```python +class BigQueryToolset: + """BigQuery tools with integrated skill support.""" + + def __init__( + self, + credentials_config: CredentialsConfig, + enable_skills: bool = True, + skill_detection_mode: str = "keyword", + ): + self._tools = [ + execute_query, + list_tables, + get_schema, + list_connections, + create_connection, + ] + + if enable_skills: + self._skill_registry = SkillRegistry( + skills_dirs=[Path(__file__).parent / "skills"], + builtin_skills=False, + ) + self._skill_callbacks = SkillCallbacks( + registry=self._skill_registry, + detection_mode=skill_detection_mode, + ) + + def get_tools(self) -> list[Tool]: + return self._tools + + def get_skill_callbacks(self) -> dict: + """Return callbacks dict for LlmAgent kwargs.""" + return { + "before_model_callback": self._skill_callbacks.before_model_callback, + "after_agent_callback": self._skill_callbacks.after_agent_callback, + } +``` + +### 9.2 Pattern: Multi-Domain Agent + +```python +# Agent with skills from multiple domains +agent = LlmAgent( + model="gemini-2.5-pro", + name="platform_agent", + instruction="Help with cloud infrastructure tasks.", + tools=[...], + extensions=[ + SkillExtension( + skills_dirs=[ + "./skills/bigquery", + "./skills/kubernetes", + "./skills/terraform", + ], + detection_mode="keyword", + max_skills_per_turn=3, + ), + ], +) +``` + +### 9.3 Pattern: Skill Composition + +```python +# Skills with dependencies +# terraform/modules.md --- +name: terraform_modules +requires: + - terraform_basics # Load basics first +--- + +# Automatically loads both when terraform_modules is detected +``` -## 9. Migration and Rollout +### 9.4 Pattern: Conditional Skills -### 9.1 Phase 1: Framework Integration (Q1 2026) +```python +class ConditionalSkillCallbacks(SkillCallbacks): + """Skills that activate based on runtime conditions.""" -**Scope:** -- Add `google.adk.skills` module to ADK core -- Implement `SkillRegistry`, `SkillCallbacks` classes -- Update `LlmAgent` documentation for callback integration + def before_model_callback(self, ctx, req): + # Add compliance skills based on user context + if ctx.state.get("user_department") == "healthcare": + self._force_activate(["hipaa"], ctx) + + # Continue with normal detection + return super().before_model_callback(ctx, req) +``` + +--- + +## 10. Rollout and Migration + +### 10.1 Phase 1: Core Framework (Q1 2026) + +**Deliverables:** +- `google.adk.skills` module in ADK core +- SkillRegistry, SkillCallbacks, SkillExtension +- Documentation and examples **API Surface:** ```python from google.adk.skills import ( + Skill, + SkillMetadata, SkillRegistry, SkillCallbacks, - SkillMetadata, - SkillContent, - ACTIVE_SKILLS_KEY, + SkillExtension, + KeywordSkillDetector, ) ``` -### 9.2 Phase 2: BigQuery Toolset Integration (Q2 2026) +### 10.2 Phase 2: Builtin Skills (Q2 2026) -**Scope:** -- Bundle BigQuery skills with `BigQueryToolset` -- Auto-configure skill callbacks when using BQ tools -- Maintain skills as external markdown for easy updates +**Deliverables:** +- BigQuery skills (BQML, AI Functions, Remote Models) +- Kubernetes skills (Deployments, Troubleshooting) +- General skills (Python, Security) -**Configuration:** +**Integration:** ```python -from google.adk.tools.bigquery import BigQueryToolset +from google.adk.skills.builtin import ( + BIGQUERY_SKILLS, + KUBERNETES_SKILLS, +) -# Skills auto-configured -toolset = BigQueryToolset( - credentials_config=..., - enable_skills=True, # New parameter +registry = SkillRegistry( + builtin_skills=True, # Includes all builtin + # OR + builtin_skills=BIGQUERY_SKILLS, # Specific subset ) ``` -### 9.3 Phase 3: Skill Marketplace (Q3 2026) +### 10.3 Phase 3: Toolset Integration (Q3 2026) + +**Deliverables:** +- BigQueryToolset with enable_skills parameter +- KubernetesToolset with enable_skills parameter +- Auto-configuration patterns -**Scope:** -- Public skill repository +### 10.4 Phase 4: Skill Ecosystem (Q4 2026) + +**Deliverables:** +- Skill marketplace/registry - Versioned skill packages -- Community contributions +- Community contribution guidelines +- Skill analytics dashboard --- -## 10. Future Extensions +## 11. Future Roadmap -### 10.1 Multi-Modal Skills +### 11.1 Multi-Modal Skills -Support for image-based skill content: -```markdown +```yaml --- -name: chart_builder -description: Build charts and visualizations +name: architecture_diagrams modality: multi-modal --- -![Chart Types](./chart_types.png) +# Architecture Patterns + +![Microservices Pattern](./images/microservices.png) -Use chart type 1 for time series... +Use this pattern when: +- Services need independent scaling +- Teams need deployment autonomy ``` -### 10.2 Skill Dependencies +### 11.2 Executable Skills ```yaml --- -name: advanced_ml -description: Advanced ML techniques -requires: - - bqml # Base skill must be loaded first +name: code_generator +modality: executable +entrypoint: generate_code --- + +```python +def generate_code(context: SkillContext) -> str: + """Generate code based on context.""" + template = load_template(context.language) + return template.render(context.params) +``` ``` -### 10.3 Dynamic Skill Updates +### 11.3 Federated Skills -Real-time skill updates without agent restart: ```python -skill_registry.reload_skill("bq_remote_model") # Hot reload +# Load skills from remote registry +registry = SkillRegistry( + remote_registries=[ + "https://skills.google.com/bigquery", + "https://internal.company.com/skills", + ], + cache_ttl=3600, # Refresh hourly +) ``` -### 10.4 Skill Analytics +### 11.4 Skill Learning -Track skill usage for optimization: ```python -skill_registry.get_usage_stats() -# {"bqml": {"activations": 1000, "avg_duration": 45.2}, ...} +# Track skill effectiveness +analytics = SkillAnalytics(registry) + +# After agent interaction +analytics.record_outcome( + skill_name="bqml", + query="train a regression model", + outcome="success", + user_satisfaction=5, +) + +# Optimize keyword detection +analytics.suggest_keywords("bqml") +# Returns: ["predictive model", "forecast"] based on user patterns ``` --- @@ -826,107 +1269,149 @@ skill_registry.get_usage_stats() ```markdown --- -name: my_skill -description: One-line description of what this skill provides -keywords: - - primary_keyword - - secondary_keyword - - function_name - - common_user_phrase +# Required fields +name: skill_name # Unique identifier (alphanumeric + underscore) +description: Brief description # One-line summary for listings + +# Optional fields +version: 1.0.0 # Semantic version +keywords: # Detection triggers + - keyword1 + - multi word keyword + - function.name +requires: [] # Skill dependencies +domain: general # Category (bigquery, kubernetes, etc.) +modality: text # text, multi-modal, executable +author: team@company.com # Maintainer contact +updated: 2025-12-01 # Last update date --- -# My Skill Title +# Skill Title -Brief introduction to the skill's purpose. +Brief introduction explaining what this skill provides. ## Prerequisites -1. Required setup step 1 -2. Required setup step 2 +List any setup requirements. ## Core Concepts ### Concept 1 -Explanation with example: +Explanation with examples: -```sql --- Example code -SELECT * FROM table; +```language +// Code example ``` ### Concept 2 -More explanation... +More content... ## Examples ### Example 1: Common Use Case -```sql --- Full working example +```language +// Complete working example ``` ### Example 2: Advanced Use Case -```sql --- Advanced example +```language +// Advanced example ``` +## Best Practices + +1. Best practice 1 +2. Best practice 2 + ## Troubleshooting **Error: "common error message"** -- Cause and solution +- Cause: Why this happens +- Solution: How to fix ## References - [Official Documentation](https://...) +- [Related Guide](https://...) ``` -### A.2 Debugging Skill Loading +### A.2 Debugging Skills -Enable debug logging: ```python import logging + +# Enable skill debugging logging.getLogger("google.adk.skills").setLevel(logging.DEBUG) -``` -Output: -``` -[SkillCallbacks] Detecting skills from: Train a model to predict... -[SkillCallbacks] Auto-activated skills: ['bqml'] -[SkillCallbacks] Injected skill content into system instruction: ['bqml'] +# Output: +# [SkillRegistry] Discovered 5 skills in ./skills +# [SkillCallbacks] Detecting from: "Train a regression model" +# [KeywordDetector] Matched "train" → bqml +# [KeywordDetector] Matched "regression" → bqml +# [SkillCallbacks] Activating skills: ['bqml'] +# [SkillCallbacks] Loaded bqml (6,234 tokens) +# [SkillCallbacks] Injected into system instruction ``` -### A.3 Testing Skill Detection +### A.3 Testing Skills ```python -def test_skill_detection(): - registry = SkillRegistry("./skills") - callbacks = SkillCallbacks(registry, detection_mode="keyword") +import pytest +from google.adk.skills import SkillRegistry, KeywordSkillDetector + +class TestSkillDetection: + @pytest.fixture + def registry(self): + return SkillRegistry(skills_dirs=["./test_skills"]) + + @pytest.fixture + def detector(self, registry): + return KeywordSkillDetector(registry) + + def test_detects_bqml_from_train(self, detector): + detected = detector.detect("Train a model to predict sales") + assert "bqml" in detected + + def test_no_detection_for_unrelated(self, detector): + detected = detector.detect("What's the weather today?") + assert len(detected) == 0 + + def test_multiple_skills_detected(self, detector): + detected = detector.detect( + "Create a Gemini model to classify news articles" + ) + assert "bq_remote_model" in detected + assert "bq_ai_operator" in detected +``` - # Test detection - detected = callbacks._detect_skills_from_text( - "Create a remote model using Gemini" - ) - assert "bq_remote_model" in detected +### A.4 Skill Metrics - # Test non-detection - detected = callbacks._detect_skills_from_text( - "What's the weather today?" - ) - assert len(detected) == 0 +```python +@dataclass +class SkillMetrics: + """Metrics collected per skill.""" + name: str + activation_count: int + avg_turn_duration_ms: float + avg_tokens_used: int + success_rate: float # Based on user feedback + common_triggers: list[str] # Most frequent detection keywords ``` --- ## References -1. Anthropic Engineering: [Equipping Agents for the Real World with Agent Skills](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills) -2. Google ADK Documentation: [LlmAgent Callbacks](https://cloud.google.com/docs/adk/callbacks) -3. BigQuery ML Documentation: [BQML Introduction](https://cloud.google.com/bigquery/docs/bqml-introduction) -4. BigQuery AI Functions: [AI Functions Reference](https://cloud.google.com/bigquery/docs/ai-functions) +1. [Anthropic: Equipping Agents with Skills](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills) +2. [Google ADK Documentation](https://cloud.google.com/docs/adk) +3. [LlmAgent Callbacks Reference](https://cloud.google.com/docs/adk/callbacks) +4. [BigQuery ML Documentation](https://cloud.google.com/bigquery/docs/bqml-introduction) +5. [BigQuery AI Functions](https://cloud.google.com/bigquery/docs/ai-functions) --- -*Document Version: 1.0 | Last Updated: December 2025* +*Document Version: 2.0 | Last Updated: December 2025 | Status: Proposal* From 85549505ea5553a77fceaeba3aa58f9d836a62f6 Mon Sep 17 00:00:00 2001 From: Haiyuan Cao Date: Tue, 9 Dec 2025 02:11:06 -0800 Subject: [PATCH 6/6] docs: clean up Skills design doc - remove timelines and metadata MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Remove author, version, and target audience metadata - Remove Q1-Q4 2026 timeline references from phase headings - Remove document footer with version info 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 --- docs/ADK_SKILLS_DESIGN.md | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-) diff --git a/docs/ADK_SKILLS_DESIGN.md b/docs/ADK_SKILLS_DESIGN.md index aa1fbcb86a..d4bf2a4ab6 100644 --- a/docs/ADK_SKILLS_DESIGN.md +++ b/docs/ADK_SKILLS_DESIGN.md @@ -1,10 +1,7 @@ # ADK Skills Plugin: First-Class Dynamic Knowledge Injection Framework -**Author:** Agent Development Kit Team **Status:** Proposal **Created:** December 2025 -**Version:** 2.0 -**Target Audience:** L6+ Tech Leads, ADK Core Team --- @@ -1136,7 +1133,7 @@ class ConditionalSkillCallbacks(SkillCallbacks): ## 10. Rollout and Migration -### 10.1 Phase 1: Core Framework (Q1 2026) +### 10.1 Phase 1: Core Framework **Deliverables:** - `google.adk.skills` module in ADK core @@ -1155,7 +1152,7 @@ from google.adk.skills import ( ) ``` -### 10.2 Phase 2: Builtin Skills (Q2 2026) +### 10.2 Phase 2: Builtin Skills **Deliverables:** - BigQuery skills (BQML, AI Functions, Remote Models) @@ -1176,14 +1173,14 @@ registry = SkillRegistry( ) ``` -### 10.3 Phase 3: Toolset Integration (Q3 2026) +### 10.3 Phase 3: Toolset Integration **Deliverables:** - BigQueryToolset with enable_skills parameter - KubernetesToolset with enable_skills parameter - Auto-configuration patterns -### 10.4 Phase 4: Skill Ecosystem (Q4 2026) +### 10.4 Phase 4: Skill Ecosystem **Deliverables:** - Skill marketplace/registry @@ -1411,7 +1408,3 @@ class SkillMetrics: 3. [LlmAgent Callbacks Reference](https://cloud.google.com/docs/adk/callbacks) 4. [BigQuery ML Documentation](https://cloud.google.com/bigquery/docs/bqml-introduction) 5. [BigQuery AI Functions](https://cloud.google.com/bigquery/docs/ai-functions) - ---- - -*Document Version: 2.0 | Last Updated: December 2025 | Status: Proposal*