diff --git a/.github/README.md b/.github/README.md new file mode 100644 index 0000000..ce0ce57 --- /dev/null +++ b/.github/README.md @@ -0,0 +1,304 @@ +# SqlScriptDOM Documentation Guide + +Welcome to the SqlScriptDOM documentation! This folder contains comprehensive guides for understanding, developing, and debugging the SQL Server T-SQL parser. + +## πŸš€ Quick Start + +**New to the project?** Start here: +1. Read [copilot-instructions.md](copilot-instructions.md) - Main project documentation +2. Browse [debugging_workflow.guidelines.instructions.md](instructions/debugging_workflow.guidelines.instructions.md) - Visual quick reference + +**Fixing a bug?** Start here: +1. Open [debugging_workflow.guidelines.instructions.md](instructions/debugging_workflow.guidelines.instructions.md) - Identify bug type +2. Follow the flowchart to the appropriate guide +3. Use the step-by-step instructions + +## πŸ“š Documentation Map + +### Core Documentation + +#### [copilot-instructions.md](copilot-instructions.md) - **START HERE** +**Purpose**: Main project documentation and overview +**Contains**: +- Project structure and key files +- Build and test commands +- Developer workflow +- Bug fixing triage +- Debugging tips +- Grammar gotchas and pitfalls + +**When to read**: First time working on the project, or for general context + +--- + +### Quick Reference + +#### [debugging_workflow.guidelines.instructions.md](instructions/debugging_workflow.guidelines.instructions.md) - **QUICK REFERENCE** +**Purpose**: Visual guide for quick bug diagnosis +**Contains**: +- Diagnostic flowchart +- Error pattern recognition +- Investigation steps +- Testing commands reference +- Key files reference +- Common pitfalls + +**When to use**: When you have a bug and need to quickly identify what type of fix is needed + +--- + +### Specialized Fix Guides + +#### [Validation_fix.guidelines.instructions.md](instructions/Validation_fix.guidelines.instructions.md) - Most Common Fix Type ⭐ +**Purpose**: Fixing validation-based bugs +**When to use**: +- βœ… Error: "Option 'X' is not valid..." or "Feature not supported..." +- βœ… Same syntax works in different context (e.g., ALTER INDEX vs ALTER TABLE) +- βœ… SQL Server version-specific features + +**Contains**: +- Real-world example (ALTER TABLE ADD CONSTRAINT RESUMABLE) +- Version flag patterns +- Validation logic modification +- Testing strategy + +**Complexity**: ⭐ Easy +**Typical time**: 1-2 hours + +--- + +#### [bug_fixing.guidelines.instructions.md](instructions/bug_fixing.guidelines.instructions.md) - Grammar Changes +**Purpose**: Adding new syntax or modifying parser grammar +**When to use**: +- βœ… Error: "Incorrect syntax near..." or "Unexpected token..." +- βœ… Parser doesn't recognize new T-SQL features +- βœ… Need to add new keywords, operators, or statements + +**Contains**: +- Complete bug-fixing workflow +- Grammar modification process +- AST updates +- Script generator changes +- Baseline generation +- Decision tree for bug types + +**Complexity**: ⭐⭐⭐ Medium to Hard +**Typical time**: 4-8 hours + +--- + +#### [parser.guidelines.instructions.md](instructions/parser.guidelines.instructions.md) +**Purpose**: Fixing parentheses recognition issues +**When to use**: +- βœ… `WHERE PREDICATE(...)` works +- ❌ `WHERE (PREDICATE(...))` fails with syntax error +- βœ… Identifier-based boolean predicates + +**Contains**: +- `IsNextRuleBooleanParenthesis()` modification +- Predicate detection patterns +- Real example (REGEXP_LIKE) + +**Complexity**: ⭐⭐ Easy-Medium +**Typical time**: 1-3 hours + +--- + +#### [grammer.guidelines.instructions.md](instructions/grammer.guidelines.instructions.md) +**Purpose**: Common patterns for extending existing grammar +**When to use**: +- βœ… Need to extend literal types to accept expressions +- βœ… Adding new enum members +- βœ… Creating new function/statement types + +**Contains**: +- Literal to expression pattern +- Real example (VECTOR_SEARCH TOP_N) +- Context-specific grammar rules +- Shared rule warnings + +**Complexity**: ⭐⭐⭐ Medium +**Typical time**: 3-6 hours + +--- + +### Meta Documentation + +#### [documentation.guidelines.instructions.md](instructions/documentation.guidelines.instructions.md) +**Purpose**: Summary of documentation improvements +**Contains**: +- What was improved and why +- Before/after comparison +- Real-world validation (ALTER TABLE RESUMABLE) +- Lessons learned + +**When to read**: If you want to understand the documentation structure and evolution + +--- + +## 🎯 Bug Type Decision Tree + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ You have a parsing bug β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ What's the β”‚ + β”‚ error message?β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ β”‚ β”‚ + β–Ό β–Ό β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β” +β”‚Optionβ”‚ β”‚Syntaxβ”‚ β”‚Parensβ”‚ +β”‚error β”‚ β”‚error β”‚ β”‚break β”‚ +β””β”€β”€β”¬β”€β”€β”€β”˜ β””β”€β”€β”¬β”€β”€β”€β”˜ β””β”€β”€β”¬β”€β”€β”€β”˜ + β”‚ β”‚ β”‚ + β–Ό β–Ό β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β” +β”‚VALID-β”‚ β”‚BUG β”‚ β”‚PARSERβ”‚ +β”‚ATION β”‚ β”‚FIXINGβ”‚ β”‚PRED β”‚ +β”‚FIX β”‚ β”‚GUIDE β”‚ β”‚RECOG β”‚ +β””β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”˜ +``` + +## πŸ“‹ Quick Reference Table + +| Error Message | Bug Type | Guide | Complexity | +|--------------|----------|-------|------------| +| "Option 'X' is not valid in statement Y" | Validation | [Validation_fix.guidelines.instructions.md](instructions/Validation_fix.guidelines.instructions.md) | ⭐ Easy | +| "Feature 'X' not supported in version Y" | Validation | [Validation_fix.guidelines.instructions.md](instructions/Validation_fix.guidelines.instructions.md) | ⭐ Easy | +| "Incorrect syntax near keyword" | Grammar | [bug_fixing.guidelines.instructions.md](instructions/bug_fixing.guidelines.instructions.md) | ⭐⭐⭐ Medium | +| "Unexpected token" | Grammar | [bug_fixing.guidelines.instructions.md](instructions/bug_fixing.guidelines.instructions.md) | ⭐⭐⭐ Medium | +| Syntax error with parentheses only | Predicate Recognition | [parser.guidelines.instructions.md](instructions/parser.guidelines.instructions.md) | ⭐⭐ Easy-Medium | +| Need to extend literal to expression | Grammar Extension | [GRAMMAR_EXTENSION_PATTERNS](GRAMMAR_EXTENSION_PATTERNS.md) | ⭐⭐⭐ Medium | + +## πŸ” Common Scenarios + +### Scenario 1: New SQL Server Feature Not Recognized +**Example**: `ALTER TABLE ... WITH (RESUMABLE = ON)` fails +**Likely Issue**: Validation blocking the option +**Start With**: [VALIDATION_FIX_GUIDE.md](VALIDATION_FIX_GUIDE.md) + +### Scenario 2: New T-SQL Keyword Not Parsed +**Example**: `CREATE EXTERNAL TABLE` not recognized +**Likely Issue**: Grammar doesn't have rules for this syntax +**Start With**: [BUG_FIXING_GUIDE.md](BUG_FIXING_GUIDE.md) + +### Scenario 3: Function Works Sometimes, Fails with Parentheses +**Example**: `WHERE REGEXP_LIKE(...)` fails +**Likely Issue**: Predicate recognition +**Start With**: [PARSER_PREDICATE_RECOGNITION_FIX.md](PARSER_PREDICATE_RECOGNITION_FIX.md) + +### Scenario 4: Parameter Support Needed +**Example**: `TOP_N = @parameter` should work +**Likely Issue**: Need to extend from literal to expression +**Start With**: [GRAMMAR_EXTENSION_PATTERNS.md](GRAMMAR_EXTENSION_PATTERNS.md) + +## πŸ› οΈ Essential Commands + +```bash +# Build parser +dotnet build SqlScriptDom/Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug + +# Run specific test +dotnet test --filter "FullyQualifiedName~YourTest" -c Debug + +# Run ALL tests (CRITICAL before committing!) +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug + +# Search for error code +grep -r "SQL46057" SqlScriptDom/ + +# Search for option usage +grep -r "RESUMABLE" Test/SqlDom/TestScripts/ +``` + +## πŸ“Š Documentation Statistics + +- **Total Guides**: 6 comprehensive guides +- **Bug Types Covered**: 3 main types (validation, grammar, predicate recognition) +- **Real-World Examples**: 4 detailed examples with code +- **Code Samples**: 50+ practical bash/C#/SQL examples +- **Quick References**: 3 tables and 2 flowcharts + +## πŸŽ“ Learning Path + +### Beginner Path (Understanding the Project) +1. [copilot-instructions.md](copilot-instructions.md) - Read "Key points" section +2. [debugging_workflow.guidelines.instructions.md](instructions/debugging_workflow.guidelines.instructions.md) - Understand bug types +3. [Validation_fix.guidelines.instructions.md](instructions/Validation_fix.guidelines.instructions.md) - Follow ALTER TABLE RESUMABLE example +4. Try fixing a validation bug yourself + +**Time**: 2-3 hours + +### Intermediate Path (Grammar Changes) +1. Review beginner path first +2. [bug_fixing.guidelines.instructions.md](instructions/bug_fixing.guidelines.instructions.md) - Complete workflow +3. [grammer.guidelines.instructions.md](instructions/grammer.guidelines.instructions.md) - Common patterns +4. [copilot-instructions.md](copilot-instructions.md) - "Grammar Gotchas" section +5. Try adding a simple new keyword + +**Time**: 4-6 hours + +### Advanced Path (Complex Features) +1. Master beginner and intermediate paths +2. [bug_fixing.guidelines.instructions.md](instructions/bug_fixing.guidelines.instructions.md) - AST modifications +3. [grammer.guidelines.instructions.md](instructions/grammer.guidelines.instructions.md) - All patterns +4. Study existing complex features (e.g., VECTOR_SEARCH) +5. Implement a new statement type + +**Time**: 8-16 hours + +## 🚨 Critical Reminders + +### Always Do This: +- βœ… **Run full test suite** before committing (1,100+ tests) +- βœ… **Check Microsoft docs** for exact version support +- βœ… **Search for error messages** first before coding +- βœ… **Create context-specific rules** instead of modifying shared ones +- βœ… **Test across all SQL Server versions** in test configuration + +### Never Do This: +- ❌ Modify shared grammar rules without understanding impact +- ❌ Skip running the full test suite +- ❌ Assume version support - always verify documentation +- ❌ Edit generated files in `obj/` directory +- ❌ Commit without testing baseline generation + +## 🀝 Contributing + +When improving these docs: +1. Use real examples from actual bugs +2. Include complete code samples (not pseudo-code) +3. Add bash commands that actually work +4. Cross-reference related guides +5. Update this README if adding new guides + +## πŸ“ž Getting Help + +If stuck: +1. Search error message in codebase: `grep -r "your error"` +2. Check similar working syntax: `grep -r "keyword" Test/SqlDom/` +3. Review relevant guide based on bug type +4. Check Git history for similar fixes: `git log --grep="RESUMABLE"` + +## πŸŽ‰ Success Metrics + +You know you've succeeded when: +- βœ… Your specific test passes +- βœ… **ALL 1,100+ tests pass** (critical!) +- βœ… Baseline matches generated output +- βœ… Version-specific behavior is correct +- βœ… No regressions in existing functionality + +--- + +**Last Updated**: Based on ALTER TABLE RESUMABLE fix (October 2025) + +**Contributors**: Documentation improved based on practical bug-fixing experience + +**Feedback**: These guides are living documents. Please update them when you discover new patterns or better approaches! diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 3baaa07..9920c5e 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -26,15 +26,20 @@ ScriptDom is a library for parsing and generating T-SQL scripts. It is primarily - `SqlScriptDom/ParserPostProcessing.sed`, `LexerPostProcessing.sed`, `TSqlTokenTypes.ps1` β€” post-processing for generated C# sources and tokens. - `tools/` β€” contains code generators used during build: `AstGen`, `ScriptGenSettingsGenerator`, `TokenListGenerator`. - `Test/SqlDom/` β€” unit tests, baselines and test scripts. See `Only170SyntaxTests.cs`, `TestScripts/`, and `Baselines170/`. +- `.github/instructions/testing.guidelines.instructions.md` β€” comprehensive testing framework guide with patterns and best practices. +- `.github/instructions/function.guidelines.instructions.md` β€” specialized guide for adding new T-SQL system functions. ## Developer workflow & conventions (typical change cycle) 1. Add/modify grammar rule(s) in the correct `TSql*.g` (pick the _version_ the syntax belongs to). 2. If tokens or token ordering change, update `TSqlTokenTypes.g` (and the sed/ps1 post-processors if necessary). 3. Rebuild the ScriptDom project to regenerate parser and AST (`dotnet build` will run generation). Use the targeted msbuild targets if you only want generation. 4. Add tests: + - **YOU MUST ADD UNIT TESTS** - Use the existing test framework in `Test/SqlDom/` + - **DO NOT CREATE STANDALONE PROGRAMS TO TEST** - Avoid separate console applications or debug programs - Put the input SQL in `Test/SqlDom/TestScripts/` (filename is case sensitive and used as an embedded resource). - Add/confirm baseline output in `Test/SqlDom/Baselines/` (the UT project embeds these baselines as resources). - Update the appropriate `OnlySyntaxTests.cs` (e.g., `Only170SyntaxTests.cs`) by adding a `ParserTest170("MyNewTest.sql", ...)` entry. See `ParserTest.cs` and `ParserTestOutput.cs` for helper constructors and verification semantics. + - **For comprehensive testing guidance**, see [Testing Guidelines](instructions/testing.guidelines.instructions.md) with detailed patterns, best practices, and simplified constructor approaches. 5. **Run full test suite** to ensure no regressions: ```bash dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug @@ -49,9 +54,46 @@ ScriptDom is a library for parsing and generating T-SQL scripts. It is primarily - If a test fails due to mismatch in generated script, compare the generated output (the test harness logs it) against the baseline to spot formatting/structure differences. ## Bug Fixing and Baseline Generation -For a practical guide on fixing bugs, including the detailed workflow for generating test baselines, see the [Bug Fixing Guide](BUG_FIXING_GUIDE.md). -For specific parser predicate recognition issues (when identifier-based predicates like `REGEXP_LIKE` don't work with parentheses), see the [Parser Predicate Recognition Fix Guide](PARSER_PREDICATE_RECOGNITION_FIX.md). +Different types of bugs require different fix approaches. **Start by diagnosing which type of issue you're dealing with:** + +### 1. Validation-Based Issues (Most Common) +If you see an error like "Option 'X' is not valid..." or "Feature 'Y' not supported..." but the syntax SHOULD work according to SQL Server docs: +- **Guide**: [Validation Fix Guide](instructions/validation_fix.guidelines.instructions.md) - Version-gated validation fixes +- **Example**: ALTER TABLE RESUMABLE option (SQL Server 2022+) +- **Key Signal**: Similar syntax works in other contexts (e.g., ALTER INDEX works but ALTER TABLE doesn't) + +### 2. Grammar-Based Issues (Adding New Syntax) +If the parser doesn't recognize the syntax at all, or you need to add new T-SQL features: +- **Guide**: [Bug Fixing Guide](instructions/bug_fixing.guidelines.instructions.md) - Grammar modifications, AST updates, script generation +- **Example**: Adding new operators, statements, or function types +- **Key Signal**: Syntax error like "Incorrect syntax near..." or "Unexpected token..." + +### 3. Parser Predicate Recognition Issues (Parentheses) +If identifier-based predicates (like `REGEXP_LIKE`) work without parentheses but fail with them: +- **Guide**: [Parser Predicate Recognition Fix Guide](instructions/parser.guidelines.instructions.md) +- **Example**: `WHERE REGEXP_LIKE('a', 'pattern')` works, but `WHERE (REGEXP_LIKE('a', 'pattern'))` fails +- **Key Signal**: Syntax error near closing parenthesis or semicolon + +**Quick Diagnostic**: Search for the error message in the codebase to determine which type of fix is needed. + +### 4. Adding New System Functions +For adding new T-SQL system functions to the parser, including handling RETURN statement contexts and ANTLR v2 syntactic predicate limitations: +- **Guide**: [Function Guidelines](instructions/function.guidelines.instructions.md) - Complete guide for system function implementation +- **Example**: JSON_OBJECT, JSON_ARRAY functions with RETURN statement support +- **Key Requirements**: Syntactic predicates for lookahead, proper AST design, comprehensive testing + +### 5. Adding New Data Types +For adding completely new SQL Server data types that require custom parsing logic and specialized AST nodes: +- **Guide**: [New Data Types Guidelines](instructions/new_data_types.guidelines.instructions.md) - Complete guide for implementing new data types +- **Example**: VECTOR data type with dimension and optional base type parameters +- **Key Signal**: New SQL Server data type with custom parameter syntax different from standard data types + +### 6. Adding New Index Types +For adding completely new SQL Server index types that require specialized syntax and custom parsing logic: +- **Guide**: [New Index Types Guidelines](instructions/new_index_types.guidelines.instructions.md) - Complete guide for implementing new index types +- **Example**: JSON INDEX with FOR clause, VECTOR INDEX with METRIC/TYPE options +- **Key Signal**: New SQL Server index type with custom syntax different from standard CREATE INDEX ## Editing generated outputs, debugging generation - Never edit generated files permanently (they live under `obj/...`/CsGenIntermediateOutputPath). Instead change: @@ -61,6 +103,85 @@ For specific parser predicate recognition issues (when identifier-based predicat - To see antlr output/errors, force verbose generation by setting MSBuild property `OutputErrorInLexerParserCompile=true` on the command line (e.g. `dotnet msbuild -t:GLexerParserCompile -p:OutputErrorInLexerParserCompile=true`). - If the antlr download fails during build, manually download `antlr-2.7.5.jar` (for non-Windows) or `.exe` (for Windows) and place it at the location defined in `Directory.Build.props` or override `AntlrLocation` when invoking msbuild. +## Debugging Tips and Investigation Workflow + +### Step 1: Identify the Bug Type +Start by searching for the error message to understand what type of fix is needed: +```bash +# Search for error code or message +grep -r "SQL46057" SqlScriptDom/ +grep -r "is not a valid" SqlScriptDom/ +``` + +**Common Error Patterns**: +- `"Option 'X' is not valid..."` β†’ Validation issue (see [grammar_validation.guidelines.instructions.md](instructions/grammar_validation.guidelines.instructions.md)) +- `"Incorrect syntax near..."` β†’ Grammar issue (see [bug_fixing.guidelines.instructions.md](instructions/bug_fixing.guidelines.instructions.md)) +- `"Syntax error near ')'"` with parentheses β†’ Predicate recognition (see [parser.guidelines.instructions.md](instructions/parser.guidelines.instructions.md)) + +### Step 2: Find Where Similar Syntax Works +If the syntax works in one context but not another: +```bash +# Search for working examples +grep -r "RESUMABLE" Test/SqlDom/TestScripts/ +grep -r "OptionName" SqlScriptDom/Parser/TSql/ +``` + +**Example**: ALTER INDEX with RESUMABLE works, but ALTER TABLE doesn't β†’ Likely validation issue + +### Step 3: Locate the Relevant Code +Common files to check: +- **Validation**: `SqlScriptDom/Parser/TSql/TSql80ParserBaseInternal.cs` (most validation logic) +- **Grammar**: `SqlScriptDom/Parser/TSql/TSql*.g` (version-specific grammar files) +- **Options**: `SqlScriptDom/ScriptDom/SqlServer/IndexOptionHelper.cs` (option registration) +- **AST**: `SqlScriptDom/Parser/TSql/Ast.xml` (AST node definitions) + +### Step 4: Check SQL Server Version Support +Always verify Microsoft documentation: +- Search for "Applies to: SQL Server 20XX (XX.x)" in Microsoft docs +- Note that different features within the same option set can have different version requirements +- Example: MAX_DURATION (SQL 2014+) vs RESUMABLE (SQL 2022+) + +### Step 5: Verify with Tests +Before and after making changes: +```bash +# Build the parser +dotnet build SqlScriptDom/Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug + +# Run specific test +dotnet test --filter "FullyQualifiedName~YourTest" -c Debug + +# ALWAYS run full suite before committing +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug +``` + +### Common Investigation Patterns + +#### Pattern 1: Option Not Recognized +```bash +# Find where option is registered +grep -r "YourOptionName" SqlScriptDom/ScriptDom/SqlServer/IndexOptionHelper.cs + +# Check enum definition +grep -r "enum IndexOptionKind" SqlScriptDom/ +``` + +#### Pattern 2: Version-Specific Behavior +```bash +# Find version checks +grep -r "TSql160AndAbove" SqlScriptDom/Parser/TSql/ + +# Check which parser version you're testing +# TSql80 = SQL Server 2000, TSql90 = 2005, ..., TSql160 = 2022, TSql170 = 2025 +``` + +#### Pattern 3: Statement-Specific Restrictions +```bash +# Find validation by statement type +grep -r "IndexAffectingStatement" SqlScriptDom/Parser/TSql/TSql80ParserBaseInternal.cs + +# Common statement types: CreateIndex, AlterIndex, AlterTableAddElement +``` + ## Patterns & code style to follow (examples you will see) - Grammar rule pattern: `ruleName returns [Type vResult = this.FragmentFactory.CreateFragment()] { ... } : ( alternatives ) ;` β€” this pattern initializes an AST fragment via FragmentFactory. @@ -72,5 +193,17 @@ For specific parser predicate recognition issues (when identifier-based predicat - **Logical `NOT` vs. Compound Operators:** The grammar handles the logical `NOT` operator (e.g., `WHERE NOT (condition)`) in a general way, often in a `booleanExpressionUnary` rule. This is distinct from compound operators like `NOT LIKE` or `NOT IN`, which are typically parsed as a single unit within a comparison rule. Don't assume that because `NOT` is supported, `NOT LIKE` will be automatically supported in all predicate contexts. - **Modifying Shared Grammar Rules:** **NEVER modify existing shared grammar rules** like `identifierColumnReferenceExpression` that are used throughout the codebase. This can cause tests to fail in unrelated areas because the rule now accepts or rejects different syntax. Instead, create specialized rules for your specific context (e.g., `vectorSearchColumnReferenceExpression` for VECTOR_SEARCH-specific needs). - **Full Test Suite Validation:** After any grammar changes, **always run the complete test suite** (`dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug`) to catch regressions. Grammar changes can have far-reaching effects on seemingly unrelated functionality. -- **Extending Literals to Expressions:** When functions/constructs currently accept only literal values (e.g., `IntegerLiteral`, `StringLiteral`) but need to support dynamic values (parameters, variables, outer references), change both the AST definition (in `Ast.xml`) and grammar rules (in `TSql*.g`) to use `ScalarExpression` instead. This pattern was used for VECTOR_SEARCH TOP_N parameter. See the detailed example in [BUG_FIXING_GUIDE.md](BUG_FIXING_GUIDE.md#special-case-extending-grammar-rules-from-literals-to-expressions) and [GRAMMAR_EXTENSION_PATTERNS.md](GRAMMAR_EXTENSION_PATTERNS.md) for comprehensive patterns. +- **Extending Literals to Expressions:** When functions/constructs currently accept only literal values (e.g., `IntegerLiteral`, `StringLiteral`) but need to support dynamic values (parameters, variables, outer references), change both the AST definition (in `Ast.xml`) and grammar rules (in `TSql*.g`) to use `ScalarExpression` instead. This pattern was used for VECTOR_SEARCH TOP_N parameter. See the detailed example in [bug_fixing.guidelines.instructions.md](instructions/bug_fixing.guidelines.instructions.md#special-case-extending-grammar-rules-from-literals-to-expressions) and [grammer.guidelines.instructions.md](instructions/grammer.guidelines.instructions.md) for comprehensive patterns. + +# Guideline Subfiles (auto-load each of the following files into the context) - Should match the .config/GuidelineReviewAgent.yaml used by the guideline_review_agent. +include: .github/instructions/grammar_validation.guidelines.instructions.md +include: .github/instructions/bug_fixing.guidelines.instructions.md +include: .github/instructions/parser.guidelines.instructions.md +include: .github/instructions/function.guidelines.instructions.md +include: .github/instructions/new_data_types.guidelines.instructions.md +include: .github/instructions/new_index_types.guidelines.instructions.md +include: .github/instructions/debugging_workflow.guidelines.instructions.md +include: .github/instructions/grammer.guidelines.instructions.md +include: .github/instructions/testing.guidelines.instructions.md + diff --git a/.github/BUG_FIXING_GUIDE.md b/.github/instructions/bug_fixing.guidelines.instructions.md similarity index 66% rename from .github/BUG_FIXING_GUIDE.md rename to .github/instructions/bug_fixing.guidelines.instructions.md index 4f3ad12..79715fa 100644 --- a/.github/BUG_FIXING_GUIDE.md +++ b/.github/instructions/bug_fixing.guidelines.instructions.md @@ -1,6 +1,24 @@ # Bug Fixing Guide for SqlScriptDOM -This guide provides a summary of the typical workflow for fixing a bug in the SqlScriptDOM parser, based on practical experience. For a more comprehensive overview of the project structure and code generation, please refer to the main [Copilot / AI instructions for SqlScriptDOM](copilot-instructions.md). +This guide provides a summary of the typical workflow for fixing a bug in the SqlScriptDOM parser, based on practical experience. For a more comprehensive overview of the project structure and code generation, please refer to the main [Copilot / AI instructions for SqlScriptDOM](../copilot-instructions.md). + +## Before You Start: Identify the Bug Type + +**IMPORTANT**: Not all bugs require grammar changes. Determine which type of fix you need: + +1. **Validation Issues**: Syntax is already parseable but incorrectly rejected + - Error: "Option 'X' is not valid..." or "Feature 'Y' not supported..." + - Example: ALTER TABLE RESUMABLE works in ALTER INDEX but not ALTER TABLE + - **β†’ Use [grammar_validation.guidelines.instructions.md](grammar_validation.guidelines.instructions.md) instead of this guide** + +2. **Grammar Issues**: Parser doesn't recognize the syntax at all (THIS guide) + - Error: "Incorrect syntax near..." or "Unexpected token..." + - Example: Adding new keywords, operators, or statement types + - **β†’ Continue with this guide** + +3. **Predicate Recognition**: Identifier predicates fail with parentheses + - Error: `WHERE REGEXP_LIKE(...)` works but `WHERE (REGEXP_LIKE(...))` fails + - **β†’ Use [parser.guidelines.instructions.md](parser.guidelines.instructions.md)** ## Summary of the Bug-Fixing Workflow @@ -23,7 +41,8 @@ The process of fixing a bug, especially one that involves adding new syntax, fol ``` 6. **Add a Unit Test**: - * Create a new `.sql` file in `Test/SqlDom/TestScripts/` that contains the specific syntax for the new test case. + * **YOU MUST ADD UNIT TESTS** - Create a new `.sql` file in `Test/SqlDom/TestScripts/` that contains the specific syntax for the new test case. + * **DO NOT CREATE STANDALONE PROGRAMS TO TEST** - Use the existing test framework, not separate console applications or debug programs. 7. **Define the Test Case**: * Add a new `ParserTest` entry to the appropriate `OnlySyntaxTests.cs` files (e.g., `Only130SyntaxTests.cs`). This entry points to your new test script and defines the expected number of parsing errors for each SQL Server version. @@ -39,6 +58,27 @@ The process of fixing a bug, especially one that involves adding new syntax, fol * **c. Update the Baseline Files**: Copy the "Actual" output from the test failure log. This is the correctly formatted script generated from the AST. Paste this content into all the baseline files you created in step 8a. * **d. Re-run the Tests**: Run the same test command again. This time, the tests should pass, confirming that the generated script matches the new baseline. + **Practical Example - Baseline Generation Workflow**: + ```bash + # 1. Create test script + echo "ALTER TABLE t ADD CONSTRAINT pk PRIMARY KEY (id) WITH (RESUMABLE = ON);" > Test/SqlDom/TestScripts/AlterTableResumableTests160.sql + + # 2. Create empty baseline + touch Test/SqlDom/Baselines160/AlterTableResumableTests160.sql + + # 3. Add test entry to Only160SyntaxTests.cs: + # new ParserTest160("AlterTableResumableTests160.sql", nErrors80: 1, ...), + + # 4. Run test (will fail, showing generated output) + dotnet test --filter "AlterTableResumableTests" -c Debug + + # 5. Copy the "Actual" output from test failure into baseline file + # Output looks like: "ALTER TABLE t ADD CONSTRAINT pk PRIMARY KEY (id) WITH (RESUMABLE = ON);" + + # 6. Re-run test (should pass now) + dotnet test --filter "AlterTableResumableTests" -c Debug + ``` + 9. **⚠️ CRITICAL: Run Full Test Suite**: * **Always run the complete test suite** to ensure your changes didn't break existing functionality: ```bash @@ -147,4 +187,32 @@ If you encounter a bug where: This is likely a **parser predicate recognition issue**. The grammar and AST are correct, but the `IsNextRuleBooleanParenthesis()` function doesn't recognize the identifier-based predicate. -**Solution**: Follow the [Parser Predicate Recognition Fix Guide](PARSER_PREDICATE_RECOGNITION_FIX.md) instead of the standard grammar modification workflow. +**Solution**: Follow the [Parser Predicate Recognition Fix Guide](parser.guidelines.instructions.md) instead of the standard grammar modification workflow. + +## Decision Tree: Which Guide to Use? + +``` +Start: You have a parsing bug +β”‚ +β”œβ”€β†’ Error: "Option 'X' is not valid..." or "Feature not supported..." +β”‚ └─→ Does similar syntax work elsewhere? (e.g., ALTER INDEX works) +β”‚ └─→ YES: Use [grammar_validation.guidelines.instructions.md](grammar_validation.guidelines.instructions.md) +β”‚ +β”œβ”€β†’ Error: "Incorrect syntax near..." or parser doesn't recognize syntax +β”‚ └─→ Does the grammar need new rules or AST nodes? +β”‚ └─→ YES: Use this guide (BUG_FIXING_GUIDE.md) +β”‚ +└─→ Error: Parentheses cause failure with identifier predicates + └─→ Does `WHERE PREDICATE(...)` work but `WHERE (PREDICATE(...))` fail? + └─→ YES: Use [parser.guidelines.instructions.md](parser.guidelines.instructions.md) +``` + +## Quick Reference: Fix Types by Symptom + +| Symptom | Fix Type | Guide | Files Modified | +|---------|----------|-------|----------------| +| "Option 'X' is not valid in statement Y" | Validation | [grammar_validation.guidelines.instructions.md](grammar_validation.guidelines.instructions.md) | `TSql80ParserBaseInternal.cs` | +| "Incorrect syntax near keyword" | Grammar | This guide | `TSql*.g`, `Ast.xml`, Script generators | +| Parentheses break identifier predicates | Predicate Recognition | [parser.guidelines.instructions.md](parser.guidelines.instructions.md) | `TSql80ParserBaseInternal.cs` | +| Literal needs to become expression | Grammar Extension | [grammer.guidelines.instructions.md](grammer.guidelines.instructions.md) | `Ast.xml`, `TSql*.g` | + diff --git a/.github/instructions/debugging_workflow.guidelines.instructions.md b/.github/instructions/debugging_workflow.guidelines.instructions.md new file mode 100644 index 0000000..59932c8 --- /dev/null +++ b/.github/instructions/debugging_workflow.guidelines.instructions.md @@ -0,0 +1,252 @@ +# ScriptDOM Debugging Workflow - Quick Reference + +This is a visual guide for quickly diagnosing and fixing bugs in SqlScriptDOM. Use this as your first stop when encountering a parsing issue. + +## πŸ” Quick Diagnosis Flowchart + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ You have a parsing error/bug β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Step 1: Search for the error message β”‚ +β”‚ Command: grep -r "SQL46057" SqlScriptDom/ β”‚ +β”‚ grep -r "your error text" SqlScriptDom/ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό + β”Œβ”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β” + β”‚ Error Type? β”‚ + β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ β”‚ β”‚ + β–Ό β–Ό β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β” +β”‚"Optionβ”‚ β”‚"Syntaxβ”‚ β”‚Parens β”‚ +β”‚not β”‚ β”‚error β”‚ β”‚break β”‚ +β”‚valid" β”‚ β”‚near..." β”‚predicteβ”‚ +β””β”€β”€β”€β”¬β”€β”€β”€β”˜ β””β”€β”€β”€β”¬β”€β”€β”€β”˜ β””β”€β”€β”€β”¬β”€β”€β”€β”˜ + β”‚ β”‚ β”‚ + β–Ό β–Ό β–Ό + β”Œβ”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β” + β”‚VALIDβ”‚ β”‚GRAM β”‚ β”‚PRED β”‚ + β”‚ATIONβ”‚ β”‚MAR β”‚ β”‚RECOGβ”‚ + β””β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”˜ +``` + +## πŸ“‹ Error Pattern Recognition + +### Pattern 1: Validation Error (Most Common) +**Symptoms:** +- ❌ `SQL46057: Option 'RESUMABLE' is not a valid index option in 'ALTER TABLE' statement` +- ❌ `Feature 'X' is not supported in SQL Server version Y` +- βœ… Same syntax works in different context (ALTER INDEX vs ALTER TABLE) + +**Quick Check:** +```bash +# Search for similar working syntax +grep -r "RESUMABLE" Test/SqlDom/TestScripts/ +# Found in AlterIndexTests but not AlterTableTests? +# β†’ It's a validation issue! +``` + +**Solution:** [Validation_fix.guidelines.instructions.md](Validation_fix.guidelines.instructions.md) +**Files to Check:** `TSql80ParserBaseInternal.cs` (validation methods) + +--- + +### Pattern 2: Grammar Error +**Symptoms:** +- ❌ `Incorrect syntax near keyword 'NEWKEYWORD'` +- ❌ Parser doesn't recognize new T-SQL feature at all +- ❌ Syntax has never been implemented + +**Quick Check:** +```bash +# Search for the keyword in grammar files +grep -r "YourKeyword" SqlScriptDom/Parser/TSql/*.g +# Not found? β†’ It's a grammar issue! +``` + +**Solution:** [bug_fixing.guidelines.instructions.md](bug_fixing.guidelines.instructions.md) +**Files to Modify:** `TSql*.g`, `Ast.xml`, Script generators + +--- + +### Pattern 3: Predicate Recognition Error +**Symptoms:** +- βœ… `WHERE REGEXP_LIKE('a', 'pattern')` works +- ❌ `WHERE (REGEXP_LIKE('a', 'pattern'))` fails with syntax error +- ❌ Error near closing parenthesis or semicolon + +**Quick Check:** +```bash +# Test both syntaxes +echo "SELECT 1 WHERE REGEXP_LIKE('a', 'b');" > test1.sql +echo "SELECT 1 WHERE (REGEXP_LIKE('a', 'b'));" > test2.sql +# Second one fails? β†’ Predicate recognition issue! +``` + +**Solution:** [parser.guidelines.instructions.md](parser.guidelines.instructions.md) +**Files to Modify:** `TSql80ParserBaseInternal.cs` (`IsNextRuleBooleanParenthesis()`) + +--- + +## πŸ› οΈ Standard Investigation Steps + +### Step 1: Reproduce Minimal Test Case +```bash +# Create minimal failing SQL +echo "ALTER TABLE t ADD CONSTRAINT pk PRIMARY KEY (id) WITH (RESUMABLE = ON);" > test.sql + +# Try parsing (use existing test harness or create simple parser test) +``` + +### Step 2: Find Error Source +```bash +# Search for error code/message +grep -r "SQL46057" SqlScriptDom/ +grep -r "is not a valid" SqlScriptDom/ + +# Common locations: +# - SqlScriptDom/Parser/TSql/TSql80ParserBaseInternal.cs (validation) +# - SqlScriptDom/Parser/TSql/TSql*.g (grammar rules) +# - SqlScriptDom/ScriptDom/SqlServer/*Helper.cs (option/type helpers) +``` + +### Step 3: Check Microsoft Documentation +```bash +# Search for: "Applies to: SQL Server 20XX (XX.x)" +# Verify exact version support +# Note: Different features may have different version requirements! +``` + +### Step 4: Locate Similar Working Code +```bash +# If ALTER INDEX works but ALTER TABLE doesn't: +grep -r "Resumable" Test/SqlDom/TestScripts/ + +# Find where option is registered: +grep -r "IndexOptionKind.Resumable" SqlScriptDom/ + +# Check validation paths: +grep -r "IndexAffectingStatement" SqlScriptDom/Parser/TSql/ +``` + +## πŸ”§ Fix Implementation Checklist + +### For Validation Fixes: +- [ ] Identify validation function (usually in `TSql80ParserBaseInternal.cs`) +- [ ] Check SQL Server version support in Microsoft docs +- [ ] Add version-gated validation (not unconditional rejection) +- [ ] Create test cases with version-specific expectations +- [ ] Build and run full test suite + +### For Grammar Fixes: +- [ ] Update grammar rules in `TSql*.g` files +- [ ] Update AST in `Ast.xml` if needed +- [ ] Update script generators +- [ ] Create test scripts and baselines +- [ ] Build parser and run tests + +### For Predicate Recognition: +- [ ] Locate `IsNextRuleBooleanParenthesis()` in `TSql80ParserBaseInternal.cs` +- [ ] Add identifier detection logic +- [ ] Add test cases with parentheses +- [ ] Verify non-parentheses syntax still works + +## πŸ“Š Version Mapping Reference + +Quick reference for SqlVersionFlags: + +| Flag | SQL Server Version | Year | Common Features | +|------|-------------------|------|-----------------| +| TSql80AndAbove | 2000 | 2000 | Basic T-SQL | +| TSql90AndAbove | 2005 | 2005 | XML, CTEs | +| TSql100AndAbove | 2008 | 2008 | MERGE, FILESTREAM | +| TSql110AndAbove | 2012 | 2012 | Sequences, Window Functions | +| TSql120AndAbove | 2014 | 2014 | In-Memory OLTP, MAX_DURATION | +| TSql130AndAbove | 2016 | 2016 | JSON, Temporal Tables | +| TSql140AndAbove | 2017 | 2017 | Graph, STRING_AGG | +| TSql150AndAbove | 2019 | 2019 | UTF-8, Intelligent QP | +| TSql160AndAbove | 2022 | 2022 | RESUMABLE constraints, JSON improvements | +| TSql170AndAbove | 2025 | 2025 | VECTOR_SEARCH, AI features | + +## πŸ§ͺ Testing Commands Reference + +```bash +# Build parser only +dotnet build SqlScriptDom/Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug + +# Build tests +dotnet build Test/SqlDom/UTSqlScriptDom.csproj -c Debug + +# Run specific test +dotnet test --filter "FullyQualifiedName~YourTestName" -c Debug + +# Run specific test file pattern +dotnet test --filter "DisplayName~AlterTableResumable" -c Debug + +# Run full suite (ALWAYS do this before committing!) +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug + +# Run with detailed output +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug -v detailed +``` + +## πŸ“‚ Key Files Reference + +| File | Purpose | When to Modify | +|------|---------|---------------| +| `TSql80ParserBaseInternal.cs` | Base validation logic | Validation fixes, common logic | +| `TSql160ParserBaseInternal.cs` | Version-specific overrides | Version-specific validation | +| `TSql*.g` | Grammar rules | New syntax, grammar changes | +| `Ast.xml` | AST node definitions | New nodes, type changes | +| `IndexOptionHelper.cs` | Option registration | New options, version mappings | +| `CodeGenerationSupporter.cs` | String constants | New keywords | +| `SqlScriptGeneratorVisitor*.cs` | Script generation | Generating SQL from AST | +| `Only*SyntaxTests.cs` | Test configuration | Test expectations per version | +| `TestScripts/*.sql` | Input test cases | New test SQL | +| `Baselines*/*.sql` | Expected output | Expected formatted SQL | + +## 🚫 Common Pitfalls to Avoid + +1. **❌ Modifying shared grammar rules** β†’ Creates unintended side effects + - βœ… Create context-specific rules instead + +2. **❌ Not running full test suite** β†’ Breaks existing functionality + - βœ… Always run ALL 1,100+ tests before committing + +3. **❌ Assuming same version for related features** β†’ Incorrect validation + - βœ… Check docs: MAX_DURATION (2014) β‰  RESUMABLE (2022) + +4. **❌ Forgetting script generator updates** β†’ Round-trip fails + - βœ… Test parse β†’ generate β†’ parse cycle + +5. **❌ Incorrect version flag logic** β†’ Wrong validation behavior + - βœ… Use `(flags & TSqlXXX) == 0` to check "NOT supported" + +## 🎯 Quick Decision Matrix + +| You Need To... | Use This Guide | Estimated Complexity | +|---------------|----------------|---------------------| +| Fix "option not valid" error | [Validation_fix.guidelines.instructions.md](Validation_fix.guidelines.instructions.md) | ⭐ Easy | +| Add new SQL keyword/operator | [bug_fixing.guidelines.instructions.md](bug_fixing.guidelines.instructions.md) | ⭐⭐⭐ Medium | +| Fix parentheses with predicates | [parser.guidelines.instructions.md](parser.guidelines.instructions.md) | ⭐⭐ Easy-Medium | +| Extend literal to expression | [grammer.guidelines.instructions.md](grammer.guidelines.instructions.md) | ⭐⭐⭐ Medium | +| Add new statement type | [bug_fixing.guidelines.instructions.md](bug_fixing.guidelines.instructions.md) | ⭐⭐⭐⭐ Hard | + +## πŸ“š Related Documentation + +- [copilot-instructions.md](../copilot-instructions.md) - Main project documentation +- [Validation_fix.guidelines.instructions.md](Validation_fix.guidelines.instructions.md) - Version-gated validation fixes +- [bug_fixing.guidelines.instructions.md](bug_fixing.guidelines.instructions.md) - Grammar modifications and AST updates +- [grammer.guidelines.instructions.md](grammer.guidelines.instructions.md) - Common extension patterns +- [parser.guidelines.instructions.md](parser.guidelines.instructions.md) - Parentheses recognition + +--- + +**Remember**: When in doubt, search for the error message first. Most bugs have been encountered before, and the error text will lead you to the right place in the code! diff --git a/.github/instructions/function.guidelines.instructions.md b/.github/instructions/function.guidelines.instructions.md new file mode 100644 index 0000000..09c0d4e --- /dev/null +++ b/.github/instructions/function.guidelines.instructions.md @@ -0,0 +1,363 @@ +# Guidelines for Adding New System Functions to SqlScriptDOM Parser + +This guide provides comprehensive instructions for adding new T-SQL system functions to the SqlScriptDOM parser, incorporating lessons learned from fixing JSON function parsing in RETURN statements. + +## Overview + +Adding a new system function involves three main components: +1. **AST Definition** (`Ast.xml`) - Define the abstract syntax tree node structure +2. **Grammar Rules** (`.g` files) - Define parsing logic for the function syntax +3. **Script Generator** - Handle conversion from AST back to T-SQL text +4. **Testing** - Ensure functionality works correctly across all contexts + +## Key Principle: Support Functions in RETURN Statements + +**Critical Requirement**: New system functions must be parseable in `ALTER FUNCTION` RETURN statements. This requires special handling due to ANTLR v2's limitation with semantic predicates during syntactic predicate lookahead. + +### The RETURN Statement Challenge + +The `returnStatement` grammar rule uses a syntactic predicate for lookahead: +```antlr +returnStatement: Return ((expression) => expression)? semicolonOpt +``` + +During lookahead, ANTLR cannot evaluate semantic predicates (which check runtime values like `vResult.FunctionName.Value`). This causes new functions to fail parsing in RETURN contexts even if they work elsewhere. + +## Why SELECT Works but RETURN Fails (The Core Problem) + +This section explains the fundamental issue we encountered with JSON functions and why it affects any new system function. + +### SELECT Statement Context (Always Works) +```sql +SELECT JSON_ARRAY('name'); -- βœ… Always worked +``` + +**Grammar Path**: `selectStatement` β†’ `selectElementsList` β†’ `selectElement` β†’ `expression` β†’ `expressionPrimary` β†’ `builtInFunctionCall` + +**Why it works**: No syntactic predicates in the path - parser can evaluate semantic predicates normally during parsing. + +### RETURN Statement Context (Previously Failed) +```sql +RETURN JSON_ARRAY('name'); -- ❌ Failed before our fix +``` + +**Grammar Path**: `returnStatement` uses syntactic predicate `((expression) =>` for lookahead + +**Why it failed**: +1. Parser encounters `RETURN JSON_ARRAY(...)` +2. Syntactic predicate triggers lookahead to check if `JSON_ARRAY(...)` is a valid expression +3. During lookahead: `expression` β†’ `expressionPrimary` β†’ `builtInFunctionCall` +4. `builtInFunctionCall` has semantic predicate: `{(vResult.FunctionName.Value == "JSON_ARRAY")}?` +5. **ANTLR v2 limitation**: Cannot evaluate `vResult.FunctionName.Value` during lookahead (object doesn't exist yet) +6. Lookahead fails β†’ parser assumes not an expression β†’ syntax error + +**The Solution**: Add token-based syntactic predicates in `expressionPrimary` that work during lookahead: +```antlr +{NextTokenMatches(CodeGenerationSupporter.JsonArray) && (LA(2) == LeftParenthesis)}? +vResult=jsonArrayCall +``` + +This is why **every new system function** must include the syntactic predicate pattern to work in RETURN statements. + +## Step-by-Step Implementation Guide + +### 1. Update AST Definition (`SqlScriptDom/Parser/TSql/Ast.xml`) + +Define the function's AST node structure: + +```xml + + + + + + +``` + +**Best Practice**: Use `ScalarExpression` for parameters that should support: +- Literals (`'value'`, `123`) +- Parameters (`@param`) +- Variables (`@variable`) +- Column references (`table.column`) +- Computed expressions (`value + 1`) + +Use specific literal types only when the SQL syntax strictly requires literals. + +### 2. Add Grammar Rules (`.g` files) + +#### 2a. Define the Function Rule + +Add to the appropriate grammar files (typically `TSql160.g`, `TSql170.g`, `TSqlFabricDW.g`): + +```antlr +yourNewFunctionCall returns [YourNewFunctionCall vResult = FragmentFactory.CreateFragment()] +{ + ScalarExpression vParam1; + StringLiteral vParam2; +} + : + tFunction:Identifier LeftParenthesis + { + Match(tFunction, CodeGenerationSupporter.YourFunctionName); + UpdateTokenInfo(vResult, tFunction); + } + vParam1 = expression + { + vResult.Parameter1 = vParam1; + } + (Comma vParam2 = stringLiteral + { + vResult.Parameter2 = vParam2; + })? + RightParenthesis + ; +``` + +#### 2b. **CRITICAL**: Add Syntactic Predicate for RETURN Statement Support + +Add to `expressionPrimary` rule **before** the generic `(Identifier LeftParenthesis)` predicate: + +```antlr +expressionPrimary returns [PrimaryExpression vResult] + // ... existing rules ... + + // Add BEFORE the generic identifier predicate + | {NextTokenMatches(CodeGenerationSupporter.YourFunctionName) && (LA(2) == LeftParenthesis)}? + vResult=yourNewFunctionCall + + // ... rest of existing rules including the generic identifier case ... + | (Identifier LeftParenthesis) => vResult=builtInFunctionCall +``` + +**Why This is Required**: +- The syntactic predicate uses `NextTokenMatches()` which works during lookahead +- It must come **before** the generic `builtInFunctionCall` predicate +- This enables the function to be recognized in RETURN statements + +#### 2c. Add to Built-in Function Call (if needed) + +If your function should also be recognized through the general built-in function mechanism, add it to `builtInFunctionCall`: + +```antlr +builtInFunctionCall returns [FunctionCall vResult = FragmentFactory.CreateFragment()] + // ... existing cases ... + + | {(vResult.FunctionName.Value == "YOUR_FUNCTION_NAME")}? + vResult=yourNewFunctionCall +``` + +### 3. Update CodeGenerationSupporter Constants + +Add the function name constant to `CodeGenerationSupporter.cs`: + +```csharp +public const string YourFunctionName = "YOUR_FUNCTION_NAME"; +``` + +### 4. Create Script Generator + +Add visitor method to handle AST-to-script conversion in the appropriate script generator file: + +```csharp +public override void ExplicitVisit(YourNewFunctionCall node) +{ + GenerateIdentifier(CodeGenerationSupporter.YourFunctionName); + GenerateSymbol(TSqlTokenType.LeftParenthesis); + + if (node.Parameter1 != null) + { + GenerateFragmentIfNotNull(node.Parameter1); + + if (node.Parameter2 != null) + { + GenerateSymbol(TSqlTokenType.Comma); + GenerateSpace(); + GenerateFragmentIfNotNull(node.Parameter2); + } + } + + GenerateSymbol(TSqlTokenType.RightParenthesis); +} +``` + +### 5. Integrate with Grammar Hierarchy + +Add your function to the appropriate place in the grammar hierarchy: + +```antlr +// Add to function call expressions +functionCall returns [FunctionCall vResult] + : vResult=yourNewFunctionCall + | // ... other function types + ; + +// Or add to primary expressions if it's a primary expression type +primaryExpression returns [PrimaryExpression vResult] + : vResult=yourNewFunctionCall + | // ... other primary expressions + ; +``` + +### 6. Build and Test + +#### 6a. Build the Project + +```bash +dotnet build SqlScriptDom/Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug +``` + +This will regenerate parser files from the grammar. + +#### 6b. Create Test Files + +**YOU MUST ADD UNIT TESTS - DO NOT CREATE STANDALONE PROGRAMS TO TEST** + +Create test script in `Test/SqlDom/TestScripts/YourFunctionTests160.sql`: + +```sql +-- Test basic function call +SELECT YOUR_FUNCTION_NAME('param1', 'param2'); + +-- CRITICAL: Test in ALTER FUNCTION RETURN statement +ALTER FUNCTION TestYourFunction() +RETURNS NVARCHAR(MAX) +AS +BEGIN + RETURN (YOUR_FUNCTION_NAME('value1', 'value2')); +END; +GO +``` + +#### 6c. Generate Baselines + +1. Create placeholder baseline file: `Test/SqlDom/Baselines160/YourFunctionTests160.sql` +2. Run the test (it will fail) +3. Copy the "Actual" output from the test failure +4. Update the baseline file with the correctly formatted output + +#### 6d. Configure Test + +Add test entry to `Test/SqlDom/Only160SyntaxTests.cs`: + +```csharp +new ParserTest160("YourFunctionTests160.sql", nErrors80: 1, nErrors90: 1, nErrors100: 1, nErrors110: 1, nErrors120: 1, nErrors130: 1, nErrors140: 1, nErrors150: 1), +``` + +Adjust error counts based on which SQL versions should support your function. + +#### 6e. Run Full Test Suite + +```bash +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug +``` + +Ensure all tests pass, including existing ones (no regressions). + +## Real-World Example: JSON Functions Fix + +This guide incorporates lessons learned from fixing `JSON_OBJECT` and `JSON_ARRAY` parsing in RETURN statements: + +### Problem Encountered +```sql +-- This worked fine: +SELECT JSON_ARRAY('name'); -- βœ… Always worked + +-- This failed before the fix: +ALTER FUNCTION GetAuth() RETURNS NVARCHAR(MAX) AS BEGIN + RETURN (JSON_OBJECT('key': 'value')); -- ❌ Parse error here +END; +``` + +### Why SELECT Worked but RETURN Didn't + +**SELECT Statement Context** (Always Worked): +```sql +SELECT JSON_ARRAY('name'); +``` +In a SELECT statement, the parser follows this path: +1. `selectStatement` β†’ `queryExpression` β†’ `querySpecification` +2. `selectElementsList` β†’ `selectElement` β†’ `expression` +3. `expression` β†’ `expressionPrimary` β†’ `builtInFunctionCall` +4. βœ… No syntactic predicate blocking the path + +**RETURN Statement Context** (Previously Failed): +```sql +RETURN JSON_ARRAY('name'); +``` +In a RETURN statement, the parser follows this path: +1. `returnStatement` uses a **syntactic predicate**: `((expression) =>` +2. During lookahead, parser tries: `expression` β†’ `expressionPrimary` β†’ `builtInFunctionCall` +3. `builtInFunctionCall` has a **semantic predicate**: `{(vResult.FunctionName.Value == "JSON_ARRAY")}?` +4. ❌ **ANTLR v2 limitation**: Semantic predicates cannot be evaluated during syntactic predicate lookahead +5. ❌ Lookahead fails β†’ parser doesn't recognize `JSON_ARRAY` as valid expression + +### Root Cause +The semantic predicate `{(vResult.FunctionName.Value == "JSON_OBJECT")}?` in `builtInFunctionCall` could not be evaluated during the syntactic predicate lookahead in `returnStatement`. + +### Solution Applied +Added syntactic predicates in `expressionPrimary`: + +```antlr +// Added before generic identifier predicate +| {NextTokenMatches(CodeGenerationSupporter.JsonObject) && (LA(2) == LeftParenthesis)}? + vResult=jsonObjectCall +| {NextTokenMatches(CodeGenerationSupporter.JsonArray) && (LA(2) == LeftParenthesis)}? + vResult=jsonArrayCall +``` + +This uses token-based checking (`NextTokenMatches`) which works during lookahead, unlike semantic predicates. + +## Grammar Files to Modify + +For SQL Server 2022+ functions, typically modify: +- `SqlScriptDom/Parser/TSql/TSql160.g` (SQL Server 2022) +- `SqlScriptDom/Parser/TSql/TSql170.g` (SQL Server 2025) +- `SqlScriptDom/Parser/TSql/TSqlFabricDW.g` (Azure Synapse) + +For earlier versions, add to appropriate grammar files (`TSql150.g`, `TSql140.g`, etc.). + +## Common Pitfalls + +1. **Forgetting RETURN Statement Support**: Always add syntactic predicates to `expressionPrimary` +2. **Wrong Predicate Order**: Syntactic predicates must come **before** generic predicates +3. **Semantic Predicates in Lookahead**: Don't rely on semantic predicates in contexts with syntactic predicate lookahead +4. **Missing Script Generator**: Every AST node needs a corresponding script generation visitor +5. **Incomplete Testing**: Test both standalone function calls and RETURN statement usage +6. **Version Compatibility**: Consider which SQL versions should support your function + +## Testing Checklist + +- [ ] Function parses in SELECT statements +- [ ] Function parses in WHERE clauses +- [ ] **Function parses in ALTER FUNCTION RETURN statements** +- [ ] Function parses with literal parameters +- [ ] Function parses with variable parameters +- [ ] Function parses with computed expressions as parameters +- [ ] Script generation produces correct T-SQL output +- [ ] Round-trip parsing (parse β†’ generate β†’ parse) works +- [ ] No regressions in existing tests +- [ ] Appropriate error handling for invalid syntax + +## Architecture Notes + +### Why Syntactic vs Semantic Predicates Matter + +- **Syntactic Predicates**: Can check token types during lookahead (`LA()`, `NextTokenMatches()`) +- **Semantic Predicates**: Check runtime values, but fail during lookahead in syntactic predicates +- **RETURN Statement Context**: Uses syntactic predicate `((expression) =>` which triggers lookahead + +### Grammar Rule Hierarchy + +``` +returnStatement + └── expression + └── expressionPrimary + β”œβ”€β”€ yourNewFunctionCall (syntactic predicate) + └── builtInFunctionCall (semantic predicate) +``` + +By adding syntactic predicates to `expressionPrimary`, we catch function calls before they reach the problematic semantic predicate in `builtInFunctionCall`. + +## Summary + +Following this guide ensures new system functions work correctly in all T-SQL contexts, especially the challenging RETURN statement scenario. The key insight is that ANTLR v2's limitations require careful predicate ordering and the use of token-based syntactic predicates for functions that need to work in lookahead contexts. \ No newline at end of file diff --git a/.github/instructions/grammar_validation.guidelines.instructions.md b/.github/instructions/grammar_validation.guidelines.instructions.md new file mode 100644 index 0000000..f02f9fe --- /dev/null +++ b/.github/instructions/grammar_validation.guidelines.instructions.md @@ -0,0 +1,359 @@ +# Validation-Based Bug Fix Guide for SqlScriptDOM + +This guide covers bugs where the **grammar already supports the syntax**, but the parser incorrectly rejects it due to validation logic. This is different from grammar-level fixes where you need to add new parsing rules. + +## When to Use This Guide + +Use this pattern when: +- βœ… The syntax **should** parse based on SQL Server documentation +- βœ… The error message is a **validation error** (e.g., "SQL46057: Option 'X' is not valid...") +- βœ… Similar syntax works in **other contexts** (e.g., ALTER INDEX works but ALTER TABLE fails) +- βœ… The feature was **added in a newer SQL Server version** but is rejected even in the correct parser + +**Do NOT use this guide when:** +- ❌ Grammar rules need to be added/modified (use [bug_fixing.guidelines.instructions.md](bug_fixing.guidelines.instructions.md) instead) +- ❌ AST nodes need to be created (use [grammer.guidelines.instructions.md](grammer.guidelines.instructions.md)) +- ❌ The syntax never existed in SQL Server + +## Real-World Example: ALTER TABLE RESUMABLE Option + +### The Problem + +User reported this SQL failed to parse: +```sql +ALTER TABLE table1 +ADD CONSTRAINT PK_Constraint PRIMARY KEY CLUSTERED (a) +WITH (ONLINE = ON, MAXDOP = 2, RESUMABLE = ON, MAX_DURATION = 240); +``` + +**Error**: `SQL46057: Option 'RESUMABLE' is not a valid index option in 'ALTER TABLE' statement.` + +**But**: The same options worked fine in `ALTER INDEX` statements. + +### Investigation Steps + +#### 1. Search for the Error Message +```bash +# Search for the error code or message text +grep -r "SQL46057" SqlScriptDom/ +grep -r "is not a valid index option" SqlScriptDom/ +``` + +**Result**: Found in `TSql80ParserBaseInternal.cs` in the `VerifyAllowedIndexOption()` method. + +#### 2. Examine the Validation Logic +```csharp +// Location: SqlScriptDom/Parser/TSql/TSql80ParserBaseInternal.cs +protected void VerifyAllowedIndexOption(IndexAffectingStatement statement, + IndexOption option, + SqlVersionFlags versionFlags) +{ + switch (statement) + { + case IndexAffectingStatement.AlterTableAddElement: + // BEFORE: Unconditionally blocked RESUMABLE and MAX_DURATION + if (option.OptionKind == IndexOptionKind.Resumable || + option.OptionKind == IndexOptionKind.MaxDuration) + { + ThrowParseErrorException("SQL46057", /* ... */); + } + break; + // ... other cases ... + } +} +``` + +**Key Finding**: The validation was **hardcoded** to reject these options for ALTER TABLE, regardless of SQL Server version. + +#### 3. Check Microsoft Documentation +Always verify the **exact SQL Server version** support: +- **RESUMABLE**: Introduced in SQL Server 2022 (version 160) +- **MAX_DURATION**: Introduced in SQL Server 2014 (version 120) for low-priority locks, extended for resumable operations + +**Important**: Different options can have different version requirements even within the same feature set! + +### The Fix + +#### Step 1: Identify Version Flags + +The codebase uses `SqlVersionFlags` for version checking: +- `TSql80AndAbove` = SQL Server 2000+ +- `TSql90AndAbove` = SQL Server 2005+ +- `TSql100AndAbove` = SQL Server 2008+ +- `TSql110AndAbove` = SQL Server 2012+ +- `TSql120AndAbove` = SQL Server 2014+ +- `TSql130AndAbove` = SQL Server 2016+ +- `TSql140AndAbove` = SQL Server 2017+ +- `TSql150AndAbove` = SQL Server 2019+ +- `TSql160AndAbove` = SQL Server 2022+ +- `TSql170AndAbove` = SQL Server 2025+ + +#### Step 2: Apply Version-Gated Validation + +Replace unconditional rejection with version checking: + +```csharp +case IndexAffectingStatement.AlterTableAddElement: + // Invalidate RESUMABLE for versions before SQL Server 2022 (160) + // Invalidate MAX_DURATION for versions before SQL Server 2014 (120) + if (((versionFlags & SqlVersionFlags.TSql160AndAbove) == 0 && + option.OptionKind == IndexOptionKind.Resumable) || + ((versionFlags & SqlVersionFlags.TSql120AndAbove) == 0 && + option.OptionKind == IndexOptionKind.MaxDuration)) + { + // Throw an error indicating the option is not supported in the current SQL Server version + ThrowParseErrorException("SQL46057", "Option not supported in this SQL Server version."); + } + break; +``` + +**Pattern Explanation**: +- `(versionFlags & SqlVersionFlags.TSql160AndAbove) == 0` β†’ Returns true if parser version < 160 +- If true AND option is RESUMABLE β†’ Throw error (option not supported yet) +- Same pattern for MAX_DURATION with TSql120AndAbove + +#### Step 3: Create Comprehensive Tests + +**Test Script**: `Test/SqlDom/TestScripts/AlterTableResumableTests160.sql` +```sql +-- Test 1: RESUMABLE with MAX_DURATION (minutes) +ALTER TABLE dbo.MyTable +ADD CONSTRAINT pk_test PRIMARY KEY CLUSTERED (id) +WITH (RESUMABLE = ON, MAX_DURATION = 240 MINUTES); + +-- Test 2: RESUMABLE = ON +ALTER TABLE dbo.MyTable +ADD CONSTRAINT pk_test PRIMARY KEY CLUSTERED (id) +WITH (RESUMABLE = ON); + +-- Test 3: RESUMABLE = OFF +ALTER TABLE dbo.MyTable +ADD CONSTRAINT pk_test PRIMARY KEY CLUSTERED (id) +WITH (RESUMABLE = OFF); + +-- Test 4: UNIQUE constraint with RESUMABLE +ALTER TABLE dbo.MyTable +ADD CONSTRAINT uq_test UNIQUE NONCLUSTERED (name) +WITH (RESUMABLE = ON); +``` + +#### Step 4: Configure Test Expectations + +**Test Configuration**: `Test/SqlDom/Only160SyntaxTests.cs` +```csharp +new ParserTest160("AlterTableResumableTests160.sql"), +``` + +#### Step 5: Create Baseline Files + +**Baseline**: `Test/SqlDom/Baselines160/AlterTableResumableTests160.sql` +```sql +ALTER TABLE dbo.MyTable + ADD CONSTRAINT pk_test PRIMARY KEY CLUSTERED (id) WITH (RESUMABLE = ON, MAX_DURATION = 240 MINUTES); + +ALTER TABLE dbo.MyTable + ADD CONSTRAINT pk_test PRIMARY KEY CLUSTERED (id) WITH (RESUMABLE = ON); + +ALTER TABLE dbo.MyTable + ADD CONSTRAINT pk_test PRIMARY KEY CLUSTERED (id) WITH (RESUMABLE = OFF); + +ALTER TABLE dbo.MyTable + ADD CONSTRAINT uq_test UNIQUE NONCLUSTERED (name) WITH (RESUMABLE = ON); +``` + +#### Step 6: Validate the Fix + +```bash +# Build to ensure code compiles +dotnet build SqlScriptDom/Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug + +# Run specific test +dotnet test --filter "FullyQualifiedName~AlterTableResumableTests" -c Debug + +# Run FULL test suite to catch regressions +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug +``` + +**Expected Results**: +- βœ… TSql160Parser: 0 errors (all tests pass) +- βœ… TSql80-150Parsers: 4 errors each (RESUMABLE correctly rejected) +- βœ… Full suite: 1,116 tests passed, 0 failed + +## Common Validation Patterns + +### Pattern 1: Version-Gated Validation +```csharp +// Allow feature only in specific SQL Server versions +if ((versionFlags & SqlVersionFlags.TSqlXXXAndAbove) == 0 && + condition) +{ + ThrowParseErrorException(...); +} +``` + +### Pattern 2: Multiple Version Requirements +```csharp +// Different features with different version requirements +if (((versionFlags & SqlVersionFlags.TSql160AndAbove) == 0 && feature1) || + ((versionFlags & SqlVersionFlags.TSql120AndAbove) == 0 && feature2)) +{ + ThrowParseErrorException(...); +} +``` + +### Pattern 3: Context-Specific Validation +```csharp +// Same option, different rules for different statements +switch (statement) +{ + case IndexAffectingStatement.AlterTableAddElement: + // Stricter rules for ALTER TABLE + break; + case IndexAffectingStatement.CreateIndex: + // More permissive for CREATE INDEX + break; +} +``` + +## Key Files for Validation Fixes + +### 1. Validation Logic +- **`SqlScriptDom/Parser/TSql/TSql80ParserBaseInternal.cs`** + - Base validation shared by all parser versions + - Contains `VerifyAllowedIndexOption()`, `VerifyAllowedIndexType()`, etc. + - Most validation fixes happen here + +### 2. Version-Specific Overrides +- **`SqlScriptDom/Parser/TSql/TSql160ParserBaseInternal.cs`** + - Can override base validation for specific versions + - Example: `VerifyAllowedIndexOption160()` calls base then adds version-specific logic + +### 3. Option Registration +- **`SqlScriptDom/ScriptDom/SqlServer/IndexOptionHelper.cs`** + - Maps option keywords to `IndexOptionKind` enum values + - Defines version support: `AddOptionMapping(kind, keyword, versionFlags)` + - **Note**: Registration here controls grammar acceptance, validation happens separately + +### 4. Enums and Constants +- **`SqlScriptDom/ScriptDom/SqlServer/IndexAffectingStatement.cs`** + - Defines statement types: `CreateIndex`, `AlterIndex`, `AlterTableAddElement`, etc. + +- **`SqlScriptDom/ScriptDom/SqlServer/IndexOptionKind.cs`** + - Defines option types: `Resumable`, `MaxDuration`, `Online`, etc. + +## Debugging Workflow + +### Step 1: Reproduce the Error +```bash +# Create a minimal test file +echo "ALTER TABLE t ADD CONSTRAINT pk PRIMARY KEY (id) WITH (RESUMABLE = ON);" > test.sql + +# Try parsing it (will fail) +# Use your test harness or create a simple parser test +``` + +### Step 2: Find the Error Source +```bash +# Search for error code +grep -r "SQL46057" SqlScriptDom/ + +# Search for error message text +grep -r "is not a valid" SqlScriptDom/ +``` + +### Step 3: Locate Validation Function +Common validation functions to check: +- `VerifyAllowedIndexOption()` - Most common +- `VerifyAllowedIndexType()` +- `VerifyFeatureSupport()` +- `CheckFeatureAvailability()` + +### Step 4: Examine the Logic +Look for: +- Hardcoded rejections (unconditional throws) +- Version checks that are too strict +- Missing version flag checks +- Incorrect version constants + +### Step 5: Check Similar Working Cases +If ALTER INDEX works but ALTER TABLE doesn't: +- Compare their validation paths +- Check for different `switch` cases +- Look for statement-type specific logic + +## Testing Strategy + +### Test Coverage Checklist +- [ ] Test with option enabled (`OPTION = ON`) +- [ ] Test with option disabled (`OPTION = OFF`) +- [ ] Test with option + other options (`OPTION = ON, OTHER_OPTION = value`) +- [ ] Test different statement types (PRIMARY KEY, UNIQUE, etc.) +- [ ] Test across all SQL Server versions (verify error counts) + +### Version-Specific Error Expectations +```csharp +// Pattern for test configuration +new ParserTestXXX("TestFile.sql", + nErrors80: X, // Count errors for SQL 2000 + nErrors90: X, // Count errors for SQL 2005 + nErrors100: X, // Count errors for SQL 2008 + nErrors110: X, // Count errors for SQL 2012 + nErrors120: Y, // May differ if feature added in 2014 + nErrors130: Y, // Same as above + nErrors140: Y, // Same as above + nErrors150: Y, // Same as above + // nErrors160: 0 (implicit) - Feature supported in 2022+ +) +``` + +## Common Pitfalls + +### 1. Assuming Same Version for Related Features +❌ **Wrong**: "RESUMABLE and MAX_DURATION are both resumable features, so both need TSql160+" +βœ… **Correct**: Check documentation - MAX_DURATION existed before RESUMABLE (TSql120 vs TSql160) + +### 2. Not Running Full Test Suite +❌ **Wrong**: Only run the new test, assume it's fine +βœ… **Correct**: Run ALL tests - validation changes can affect unexpected areas + +### 3. Incorrect Version Flag Logic +❌ **Wrong**: `if (versionFlags & SqlVersionFlags.TSql160AndAbove)` (missing == 0) +βœ… **Correct**: `if ((versionFlags & SqlVersionFlags.TSql160AndAbove) == 0)` (check if NOT set) + +### 4. Forgetting Statement Context +❌ **Wrong**: Apply same validation to all statement types +βœ… **Correct**: Different statements may have different option support + +## Summary Checklist + +- [ ] **Identify** the validation function throwing the error +- [ ] **Verify** Microsoft documentation for exact version support +- [ ] **Apply** version-gated validation (not unconditional rejection) +- [ ] **Create** comprehensive test cases covering all scenarios +- [ ] **Configure** test expectations for all SQL Server versions +- [ ] **Generate** baseline files from actual parser output +- [ ] **Build** the ScriptDOM project successfully +- [ ] **Run** full test suite (ALL 1,100+ tests must pass) +- [ ] **Document** the fix with clear before/after examples + +## Related Guides + +- [bug_fixing.guidelines.instructions.md](bug_fixing.guidelines.instructions.md) - For grammar-level fixes +- [grammer.guidelines.instructions.md](grammer.guidelines.instructions.md) - For extending existing grammar +- [parser.guidelines.instructions.md](parser.guidelines.instructions.md) - For parentheses recognition issues + +## Real-World Examples + +### Example 1: ALTER TABLE RESUMABLE (SQL Server 2022) +- **File**: `TSql80ParserBaseInternal.cs` +- **Function**: `VerifyAllowedIndexOption()` +- **Fix**: Added `TSql160AndAbove` check for RESUMABLE +- **Tests**: `AlterTableResumableTests160.sql` + +### Example 2: MAX_DURATION (SQL Server 2014) +- **File**: Same as above +- **Function**: Same as above +- **Fix**: Added `TSql120AndAbove` check for MAX_DURATION +- **Tests**: Same file, different version expectations + +These examples demonstrate how validation fixes are often simpler than grammar changes - the parser already knows how to parse the syntax, it just needs permission to accept it in specific contexts and versions. diff --git a/.github/GRAMMAR_EXTENSION_PATTERNS.md b/.github/instructions/grammer.guidelines.instructions.md similarity index 100% rename from .github/GRAMMAR_EXTENSION_PATTERNS.md rename to .github/instructions/grammer.guidelines.instructions.md diff --git a/.github/instructions/new_data_types.guidelines.instructions.md b/.github/instructions/new_data_types.guidelines.instructions.md new file mode 100644 index 0000000..86e59a6 --- /dev/null +++ b/.github/instructions/new_data_types.guidelines.instructions.md @@ -0,0 +1,439 @@ +# Guidelines for Adding New Data Types to SqlScriptDOM + +This guide provides step-by-step instructions for adding support for completely new SQL Server data types to the SqlScriptDOM parser. This pattern was established from the Vector data type implementation (commits 38a0971 and cd69b78). + +## When to Use This Guide + +Use this pattern when: +- βœ… Adding a **completely new SQL Server data type** (e.g., VECTOR, GEOMETRY, GEOGRAPHY) +- βœ… The data type has **custom parameters** not handled by standard SQL data types +- βœ… The data type requires **specialized parsing logic** beyond simple name/size parameters +- βœ… The data type is **introduced in a specific SQL Server version** + +**Do NOT use this guide for:** +- ❌ Modifying existing data types (use [validation_fix.guidelines.instructions.md](validation_fix.guidelines.instructions.md)) +- ❌ Adding function syntax (use [function.guidelines.instructions.md](function.guidelines.instructions.md)) +- ❌ Simple keyword additions (use [bug_fixing.guidelines.instructions.md](bug_fixing.guidelines.instructions.md)) + +## Real-World Example: Vector Data Type + +The Vector data type implementation demonstrates this pattern: + +### SQL Server Syntax Supported +```sql +-- Basic vector with dimension only +DECLARE @embedding AS VECTOR(1536); +CREATE TABLE tbl (embedding VECTOR(1536)); + +-- Vector with dimension and base type +DECLARE @embedding AS VECTOR(1536, FLOAT32); +CREATE TABLE tbl (embedding VECTOR(1536, FLOAT16)); +``` + +### Key Challenge Solved +The Vector type requires custom parsing because: +- **Standard data types** use size parameters: `VARCHAR(50)`, `DECIMAL(10,2)` +- **Vector type** uses dimension + optional base type: `VECTOR(1536, FLOAT32)` +- **Base type parameter** is an identifier (FLOAT16/FLOAT32), not a size literal + +## Step-by-Step Implementation Guide + +### 1. Define AST Node Structure (`Ast.xml`) + +Add a new class inheriting from `DataTypeReference`: + +```xml + + + + + + + + + +``` + +**Design Principles**: +- **Inherit from `DataTypeReference`**: All SQL data types inherit from this base class +- **Choose appropriate member types**: + - `IntegerLiteral`: For numeric parameters (dimensions, sizes) + - `Identifier`: For type names or keywords + - `StringLiteral`: For string parameters + - `ScalarExpression`: For complex expressions (use sparingly) +- **Optional parameters**: Members can be null for optional syntax + +### 2. Add Grammar Rule (`TSql*.g`) + +Create a specialized parsing rule for your data type: + +```antlr +// Location: SqlScriptDom/Parser/TSql/TSql170.g (or appropriate version) +// Add after xmlDataType rule (~line 30672) + +yourDataType [SchemaObjectName vName] returns [YourDataTypeReference vResult = FragmentFactory.CreateFragment()] +{ + vResult.Name = vName; + vResult.UpdateTokenInfo(vName); + + IntegerLiteral vParameter1 = null; + Identifier vParameter2 = null; +} + : + ( LeftParenthesis vParameter1=integer + { + vResult.Parameter1 = vParameter1; + } + ( + Comma vParameter2=identifier + { + vResult.Parameter2 = vParameter2; + } + )? + tRParen:RightParenthesis + { + UpdateTokenInfo(vResult,tRParen); + } + ) + ; +``` + +**Grammar Pattern Explanation**: +- **Function signature**: Takes `SchemaObjectName vName` parameter and returns your AST type +- **Variable declarations**: Declare variables for each parameter using appropriate types +- **Parameter parsing**: Use `integer`, `identifier`, `stringLiteral` based on syntax needs +- **Optional parameters**: Wrap in `( ... )?` syntax for optional elements +- **Token info updates**: Always call `UpdateTokenInfo()` for proper source location tracking + +### 3. Integrate with Scalar Data Type Rule + +Connect your new grammar rule to the main data type parsing logic: + +```antlr +// Location: SqlScriptDom/Parser/TSql/TSql170.g +// Find scalarDataType rule (~line 30694) and add your type check + +scalarDataType returns [DataTypeReference vResult = null] +{ + SchemaObjectName vName; + SqlDataTypeOption typeOption = SqlDataTypeOption.None; + // ... existing variables ... +} + : vName = schemaObjectFourPartName + { + typeOption = GetSqlDataTypeOption(vName); + // ... existing logic ... + } + ( + ( + {isXmlDataType}? + vResult = xmlDataType[vName] + | + {typeOption == SqlDataTypeOption.YourType}? // Add this condition + vResult = yourDataType[vName] + | + {typeOption != SqlDataTypeOption.None}? + vResult = sqlDataTypeWithoutNational[vName, typeOption] + // ... rest of existing alternatives +``` + +**Integration Requirements**: +- **Add type option check**: Use `{typeOption == SqlDataTypeOption.YourType}?` semantic predicate +- **Maintain order**: Place before the generic `sqlDataTypeWithoutNational` fallback +- **Update SqlDataTypeOption enum**: Add your type to the enum (implementation dependent) + +### 4. Add String Constants + +Add necessary string constants for keywords: + +```csharp +// Location: SqlScriptDom/Parser/TSql/CodeGenerationSupporter.cs +// Add alphabetically in the constants section (~line 427 for Float constants) + +internal const string YourType = "YOURTYPE"; +internal const string YourTypeParam1 = "PARAM1_KEYWORD"; +internal const string YourTypeParam2 = "PARAM2_KEYWORD"; +``` + +**Naming Convention**: +- **Use exact SQL keyword casing**: `VECTOR`, `FLOAT16`, `FLOAT32` +- **Group related constants**: Keep data type constants together +- **Alphabetical ordering**: Maintain alphabetical order within sections + +### 5. Create Script Generator + +Implement the visitor method to convert AST back to T-SQL: + +```csharp +// Location: SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/SqlScriptGeneratorVisitor.YourDataType.cs +// Create new file following naming convention + +//------------------------------------------------------------------------------ +// +// Copyright (c) Microsoft Corporation. All rights reserved. +// +//------------------------------------------------------------------------------ + +namespace Microsoft.SqlServer.TransactSql.ScriptDom.ScriptGenerator +{ + partial class SqlScriptGeneratorVisitor + { + public override void ExplicitVisit(YourDataTypeReference node) + { + GenerateIdentifier(CodeGenerationSupporter.YourType); + GenerateSymbol(TSqlTokenType.LeftParenthesis); + GenerateFragmentIfNotNull(node.Parameter1); + if (node.Parameter2 != null) + { + GenerateSymbol(TSqlTokenType.Comma); + GenerateSpaceAndFragmentIfNotNull(node.Parameter2); + } + GenerateSymbol(TSqlTokenType.RightParenthesis); + } + } +} +``` + +**Script Generation Patterns**: +- **Use `GenerateIdentifier()`**: For type names and keywords +- **Use `GenerateSymbol()`**: For punctuation (`LeftParenthesis`, `Comma`, etc.) +- **Use `GenerateFragmentIfNotNull()`**: For required parameters +- **Use `GenerateSpaceAndFragmentIfNotNull()`**: For optional parameters with preceding space +- **Handle optional parameters**: Always check for null before generating + +### 6. Build and Test Grammar Changes + +Build the project to regenerate parser files: + +```bash +# Build the ScriptDOM project to regenerate parser from grammar +dotnet build SqlScriptDom/Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug +``` + +**Common Build Issues**: +- **Grammar syntax errors**: Check ANTLR syntax in `.g` files +- **Missing constants**: Ensure all referenced constants exist in `CodeGenerationSupporter.cs` +- **AST node mismatches**: Verify AST class names match grammar return types + +### 7. Create Comprehensive Test Scripts + +Create test script covering all syntax variations: + +```sql +-- File: Test/SqlDom/TestScripts/YourDataTypeTests170.sql + +-- Basic syntax with single parameter +CREATE TABLE tbl (col1 YOURTYPE(100)); +DECLARE @var1 AS YOURTYPE(100); + +-- Extended syntax with optional parameters +CREATE TABLE tbl (col1 YOURTYPE(100, PARAM1)); +DECLARE @var2 AS YOURTYPE(100, PARAM2); + +-- Case insensitivity testing +CREATE TABLE tbl (col1 yourtype(100, param1)); +DECLARE @var3 AS YOURTYPE(100, param2); + +-- Integration with other SQL constructs +CREATE TABLE tbl ( + id INT PRIMARY KEY, + data YOURTYPE(100, PARAM1) NOT NULL +); + +-- Variables and parameters +CREATE FUNCTION TestFunction(@input YOURTYPE(100)) +RETURNS YOURTYPE(200, PARAM2) +AS +BEGIN + DECLARE @result YOURTYPE(200, PARAM2); + RETURN @result; +END; +``` + +**Test Coverage Requirements**: +- **All syntax variations**: Test with and without optional parameters +- **Case sensitivity**: Test different case combinations +- **Integration contexts**: Variables, table columns, function parameters/returns +- **Edge cases**: Minimum/maximum parameter values if applicable + +### 8. Generate Baseline Files + +Create the expected output baseline: + +1. **Create placeholder baseline**: `Test/SqlDom/Baselines170/YourDataTypeTests170.sql` +2. **Run the test** (will fail initially): + ```bash + dotnet test --filter "YourDataTypeTests170" Test/SqlDom/UTSqlScriptDom.csproj -c Debug + ``` +3. **Copy "Actual" output** from test failure into baseline file +4. **Verify formatting** matches parser's standard formatting + +**Baseline Example** (Vector data type): +```sql +CREATE TABLE tbl ( + embedding VECTOR(1) +); + +CREATE TABLE tbl ( + embedding VECTOR(1, float32) +); + +DECLARE @embedding AS VECTOR(2); + +DECLARE @embedding AS VECTOR(2, FLOAT32); +``` + +### 9. Configure Test Expectations + +Add test configuration to version-specific test class: + +```csharp +// Location: Test/SqlDom/Only170SyntaxTests.cs (or appropriate version) +// Add to the ParserTest170 array + +new ParserTest170("YourDataTypeTests170.sql"), +``` + +**Error Count Guidelines**: +- **Count all syntax instances**: Each usage of the new type in test script +- **Consider SQL version support**: When was the feature actually introduced? +- **Consistent across versions**: Usually same error count until supported version +- **Test validation**: Run tests to verify error counts are accurate + +### 10. Full Test Suite Validation + +Run complete test suite to ensure no regressions: + +```bash +# Run all ScriptDOM tests +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug + +# Expected result: All tests pass, including new data type tests +# Total tests: 1,100+ (number increases with new features) +``` + +**Regression Prevention**: +- **Grammar changes can break existing functionality**: Shared rules affect multiple contexts +- **AST changes can break script generation**: Ensure all visitors are updated +- **Version compatibility**: New syntax shouldn't break older version parsers + +## Advanced Considerations + +### Version-Specific Implementation + +For data types introduced in specific SQL Server versions: + +```antlr +// Different grammar files for different SQL versions +// TSql160.g - SQL Server 2022 features +// TSql170.g - SQL Server 2025 features +// TSqlFabricDW.g - Azure Synapse features +``` + +**Guidelines**: +- **Target appropriate version**: Add to the SQL version where feature was introduced +- **Cascade to later versions**: Copy rules to all subsequent version grammar files +- **Version-specific testing**: Test error behavior in older parsers + +### Complex Parameter Types + +For data types requiring complex parameter parsing: + +```xml + + + +``` + +**When to use complex types**: +- **ScalarExpression**: When parameters can be variables, function calls, or computed values +- **Collections**: When syntax supports multiple values or options +- **Custom classes**: When parameters have their own sub-syntax + +### Script Generator Considerations + +For complex formatting requirements: + +```csharp +public override void ExplicitVisit(ComplexDataTypeReference node) +{ + GenerateIdentifier(CodeGenerationSupporter.ComplexType); + GenerateSymbol(TSqlTokenType.LeftParenthesis); + + // Complex formatting with line breaks + if (node.HasMultipleParameters) + { + Indent(); + GenerateNewLine(); + } + + GenerateCommaSeparatedList(node.Parameters); + + if (node.HasMultipleParameters) + { + Outdent(); + GenerateNewLine(); + } + + GenerateSymbol(TSqlTokenType.RightParenthesis); +} +``` + +## Common Pitfalls and Solutions + +### 1. Forgetting Script Generator Implementation +**Problem**: AST node created but no script generation visitor +**Solution**: Always implement `ExplicitVisit()` method for new AST nodes + +### 2. Incorrect Grammar Integration +**Problem**: Data type not recognized in all contexts +**Solution**: Ensure integration with `scalarDataType` rule and proper semantic predicates + +### 3. Missing Version Compatibility +**Problem**: New type breaks older version parsers unexpectedly +**Solution**: Add proper version checks and test all SQL Server versions + +### 4. Incomplete Test Coverage +**Problem**: Edge cases not covered in testing +**Solution**: Test all syntax variations, case sensitivity, and integration contexts + +### 5. AST Design Issues +**Problem**: AST doesn't properly represent the SQL syntax +**Solution**: Design AST members to match SQL parameter structure and optionality + +## Validation Checklist + +- [ ] **AST Definition**: New class inherits from correct base class with appropriate members +- [ ] **Grammar Rules**: Specialized parsing rule handles all syntax variations +- [ ] **Grammar Integration**: Connected to `scalarDataType` with proper semantic predicate +- [ ] **String Constants**: All keywords added to `CodeGenerationSupporter.cs` +- [ ] **Script Generator**: `ExplicitVisit()` method generates correct T-SQL output +- [ ] **Test Scripts**: Comprehensive test coverage including edge cases +- [ ] **Baseline Files**: Generated output matches expected formatted T-SQL +- [ ] **Test Configuration**: Error counts configured for all SQL Server versions +- [ ] **Build Success**: Project builds without errors and regenerates parser +- [ ] **Full Test Suite**: All existing tests continue to pass (no regressions) + +## Related Guides + +- [bug_fixing.guidelines.instructions.md](bug_fixing.guidelines.instructions.md) - For general grammar modifications +- [function.guidelines.instructions.md](function.guidelines.instructions.md) - For adding system functions +- [validation_fix.guidelines.instructions.md](validation_fix.guidelines.instructions.md) - For validation-only issues +- [testing.guidelines.instructions.md](testing.guidelines.instructions.md) - For comprehensive testing strategies + +## Real-World Examples + +### Vector Data Type (SQL Server 2025) +- **AST Class**: `VectorDataTypeReference` with `Dimension` and `BaseType` members +- **Syntax**: `VECTOR(1536)`, `VECTOR(1536, FLOAT32)` +- **Commits**: cd69b78, 38a0971 +- **Challenge**: Optional second parameter with identifier type + +### Future Examples +This pattern can be applied to other SQL Server data types like: +- **GEOMETRY**: Spatial data with complex parameters +- **GEOGRAPHY**: Geographic data with coordinate systems +- **HIERARCHYID**: Hierarchical data with custom syntax +- **Custom CLR Types**: User-defined types with specialized parameters + +The Vector implementation serves as the canonical example for this pattern and should be referenced for similar future data type additions. \ No newline at end of file diff --git a/.github/instructions/new_index_types.guidelines.instructions.md b/.github/instructions/new_index_types.guidelines.instructions.md new file mode 100644 index 0000000..dacba71 --- /dev/null +++ b/.github/instructions/new_index_types.guidelines.instructions.md @@ -0,0 +1,546 @@ +# Guidelines for Adding New Index Types to SqlScriptDOM + +This guide provides step-by-step instructions for adding support for new SQL Server index types to the SqlScriptDOM parser. This pattern was established from the JSON and Vector index implementations found in SQL Server 2025 (TSql170). + +## When to Use This Guide + +Use this pattern when: +- βœ… Adding a **completely new SQL Server index type** (e.g., JSON INDEX, VECTOR INDEX, SPATIAL INDEX) +- βœ… The index type has **specialized syntax** not handled by standard CREATE INDEX +- βœ… The index type requires **custom parsing logic** for type-specific clauses or options +- βœ… The index type is **introduced in a specific SQL Server version** + +**Do NOT use this guide for:** +- ❌ Adding new index options to existing index types (use [validation_fix.guidelines.instructions.md](validation_fix.guidelines.instructions.md)) +- ❌ Adding standard indexes with new keywords (use [bug_fixing.guidelines.instructions.md](bug_fixing.guidelines.instructions.md)) +- ❌ Adding function or data type syntax (use respective guides) + +## Real-World Examples: JSON and Vector Indexes + +### JSON Index Implementation +```sql +-- Basic JSON index +CREATE JSON INDEX IX_JSON_Basic ON dbo.Users (JsonData); + +-- JSON index with FOR clause (multiple paths) +CREATE JSON INDEX IX_JSON_Paths ON dbo.Users (JsonData) +FOR ('$.name', '$.email', '$.age'); + +-- JSON index with WITH options +CREATE JSON INDEX IX_JSON_Options ON dbo.Users (JsonData) +WITH (OPTIMIZE_FOR_ARRAY_SEARCH = ON, MAXDOP = 4); +``` + +### Vector Index Implementation +```sql +-- Basic vector index +CREATE VECTOR INDEX IX_Vector_Basic ON dbo.Documents (VectorData); + +-- Vector index with metric and type +CREATE VECTOR INDEX IX_Vector_Complete ON dbo.Documents (VectorData) +WITH (METRIC = 'cosine', TYPE = 'DiskANN'); + +-- Vector index with filegroup +CREATE VECTOR INDEX IX_Vector_FG ON dbo.Documents (VectorData) +WITH (METRIC = 'dot') +ON [PRIMARY]; +``` + +### Key Challenges Solved +- **Type-specific syntax**: JSON INDEX has `FOR (paths)` clause, VECTOR INDEX has `METRIC`/`TYPE` options +- **Custom columns**: Single column specification instead of column lists +- **Specialized options**: New index options specific to each index type +- **Grammar integration**: Seamless integration with existing CREATE INDEX patterns + +## Step-by-Step Implementation Guide + +### 1. Define AST Node Structure (`Ast.xml`) + +Add a new class inheriting from `IndexStatement`: + +```xml + + + + + + + + + + + +``` + +**Design Principles**: +- **Inherit from `IndexStatement`**: All index types inherit from this base class +- **Reuse standard properties**: `Name`, `OnName`, `IndexOptions` come from base class +- **Add type-specific members**: Properties unique to your index type +- **Collections for lists**: Use `Collection="true"` for array-like properties +- **Optional members**: Members can be null for optional syntax elements + +### 2. Add Grammar Rule (`TSql*.g`) + +Create a specialized parsing rule for your index type: + +```antlr +// Location: SqlScriptDom/Parser/TSql/TSql170.g (or appropriate version) +// Add after existing index statement rules (~line 17021) + +createYourTypeIndexStatement [IToken tUnique, bool? isClustered] returns [CreateYourTypeIndexStatement vResult = FragmentFactory.CreateFragment()] +{ + Identifier vIdentifier; + SchemaObjectName vSchemaObjectName; + Identifier vSpecializedColumn; + StringLiteral vProperty; + FileGroupOrPartitionScheme vFileGroupOrPartitionScheme; + + if (tUnique != null) + { + ThrowIncorrectSyntaxErrorException(tUnique); + } + if (isClustered.HasValue) + { + ThrowIncorrectSyntaxErrorException(LT(1)); + } +} + : tYourType:Identifier tIndex:Index vIdentifier=identifier + { + Match(tYourType, CodeGenerationSupporter.YourType); + vResult.Name = vIdentifier; + } + tOn:On vSchemaObjectName=schemaObjectThreePartName + { + vResult.OnName = vSchemaObjectName; + } + LeftParenthesis vSpecializedColumn=identifier tRParen:RightParenthesis + { + vResult.SpecializedColumn = vSpecializedColumn; + UpdateTokenInfo(vResult, tRParen); + } + ( + tFor:For LeftParenthesis + vProperty=stringLiteral + { + AddAndUpdateTokenInfo(vResult, vResult.TypeSpecificProperty, vProperty); + } + ( + Comma vProperty=stringLiteral + { + AddAndUpdateTokenInfo(vResult, vResult.TypeSpecificProperty, vProperty); + } + )* + RightParenthesis + )? + ( + // Greedy due to conflict with withCommonTableExpressionsAndXmlNamespaces + options {greedy = true; } : + With + indexOptionList[IndexAffectingStatement.CreateIndex, vResult.IndexOptions, vResult] + )? + ( + On vFileGroupOrPartitionScheme=filegroupOrPartitionScheme + { + vResult.OnFileGroupOrPartitionScheme = vFileGroupOrPartitionScheme; + } + )? + ; +``` + +**Grammar Pattern Explanation**: +- **Parameter validation**: Reject UNIQUE and CLUSTERED if not supported +- **Standard index structure**: Name, ON table, column specification +- **Type-specific clauses**: Optional FOR, WITH, ON clauses as appropriate +- **Token matching**: Use `Match()` to verify keywords +- **Collection building**: Use `AddAndUpdateTokenInfo()` for lists + +### 3. Integrate with Main Index Grammar + +Add your index type to the main CREATE INDEX rule: + +```antlr +// Location: SqlScriptDom/Parser/TSql/TSql170.g +// Find createIndexStatement rule (~line 16880) and add integration + +createIndexStatement [IToken tUnique, bool? isClustered] returns [TSqlStatement vResult] + : // ... existing alternatives ... + | vResult=createYourTypeIndexStatement[tUnique, isClustered] + ; + +// Also add to ddlStatement if needed (~line 885) +ddlStatement returns [TSqlStatement vResult] + : // ... existing alternatives ... + | vResult=createYourTypeIndexStatement[null, null] + // ... rest of alternatives ... + ; +``` + +**Integration Requirements**: +- **Add to `createIndexStatement`**: Main CREATE INDEX dispatch rule +- **Add to `ddlStatement`**: Top-level DDL statement recognition +- **Parameter passing**: Pass `tUnique` and `isClustered` tokens +- **Consistent ordering**: Place appropriately among other index types + +### 4. Add String Constants + +Add necessary keywords to `CodeGenerationSupporter.cs`: + +```csharp +// Location: SqlScriptDom/Parser/TSql/CodeGenerationSupporter.cs +// Add alphabetically in the constants section + +internal const string YourType = "YOUR_TYPE"; +internal const string YourTypeSpecificKeyword1 = "KEYWORD1"; +internal const string YourTypeSpecificKeyword2 = "KEYWORD2"; +``` + +**Naming Convention**: +- **Use exact SQL keyword casing**: `Json`, `Vector`, `Metric` +- **Group related constants**: Keep index-specific constants together +- **Follow existing patterns**: Match existing naming conventions + +### 5. Add Index Options (if needed) + +If your index type requires new index options, add them: + +```csharp +// Location: SqlScriptDom/ScriptDom/SqlServer/IndexOptionKind.cs +// Add to the enum +public enum IndexOptionKind +{ + // ... existing options ... + YourTypeSpecificOption1, + YourTypeSpecificOption2, + // ... rest of enum ... +} + +// Location: SqlScriptDom/Parser/TSql/IndexOptionHelper.cs +// Add option mappings in the constructor +AddOptionMapping(IndexOptionKind.YourTypeSpecificOption1, CodeGenerationSupporter.YourTypeSpecificKeyword1, SqlVersionFlags.TSql170AndAbove); +AddOptionMapping(IndexOptionKind.YourTypeSpecificOption2, CodeGenerationSupporter.YourTypeSpecificKeyword2, SqlVersionFlags.TSql170AndAbove); +``` + +**Option Guidelines**: +- **Add to enum**: Define new `IndexOptionKind` values +- **Register mappings**: Map keywords to enum values with version flags +- **Version compatibility**: Use appropriate `SqlVersionFlags` + +### 6. Create Script Generator + +Implement the visitor method to convert AST back to T-SQL: + +```csharp +// Location: SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/SqlScriptGeneratorVisitor.CreateYourTypeIndexStatement.cs +// Create new file following naming convention + +//------------------------------------------------------------------------------ +// +// Copyright (c) Microsoft Corporation. All rights reserved. +// +//------------------------------------------------------------------------------ +using System.Collections.Generic; +using Microsoft.SqlServer.TransactSql.ScriptDom; + +namespace Microsoft.SqlServer.TransactSql.ScriptDom.ScriptGenerator +{ + partial class SqlScriptGeneratorVisitor + { + public override void ExplicitVisit(CreateYourTypeIndexStatement node) + { + GenerateKeyword(TSqlTokenType.Create); + + GenerateSpaceAndIdentifier(CodeGenerationSupporter.YourType); + + GenerateSpaceAndKeyword(TSqlTokenType.Index); + + // name + GenerateSpaceAndFragmentIfNotNull(node.Name); + + NewLineAndIndent(); + GenerateKeyword(TSqlTokenType.On); + GenerateSpaceAndFragmentIfNotNull(node.OnName); + + // Specialized column + if (node.SpecializedColumn != null) + { + GenerateSpace(); + GenerateSymbol(TSqlTokenType.LeftParenthesis); + GenerateFragmentIfNotNull(node.SpecializedColumn); + GenerateSymbol(TSqlTokenType.RightParenthesis); + } + + // Type-specific clause + if (node.TypeSpecificProperty != null && node.TypeSpecificProperty.Count > 0) + { + NewLineAndIndent(); + GenerateKeyword(TSqlTokenType.For); + GenerateSpace(); + GenerateParenthesisedCommaSeparatedList(node.TypeSpecificProperty); + } + + GenerateIndexOptions(node.IndexOptions); + + if (node.OnFileGroupOrPartitionScheme != null) + { + NewLineAndIndent(); + GenerateKeyword(TSqlTokenType.On); + + GenerateSpaceAndFragmentIfNotNull(node.OnFileGroupOrPartitionScheme); + } + } + } +} +``` + +**Script Generation Patterns**: +- **Use `GenerateKeyword()`**: For T-SQL keywords like CREATE, INDEX, ON +- **Use `GenerateIdentifier()`**: For type-specific keywords +- **Use `NewLineAndIndent()`**: For proper formatting with line breaks +- **Use `GenerateIndexOptions()`**: Reuse existing index option generation +- **Handle collections**: Use `GenerateParenthesisedCommaSeparatedList()` for arrays + +### 7. Build and Test Grammar Changes + +Build the project to regenerate parser files: + +```bash +# Build the ScriptDOM project to regenerate parser from grammar +dotnet build SqlScriptDom/Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug +``` + +**Common Build Issues**: +- **Grammar syntax errors**: Check ANTLR syntax in `.g` files +- **Missing constants**: Ensure all referenced constants exist in `CodeGenerationSupporter.cs` +- **AST node mismatches**: Verify AST class names match grammar return types +- **Option registration**: Ensure new index options are properly registered + +### 8. Create Comprehensive Test Scripts + +Create test script covering all syntax variations: + +```sql +-- File: Test/SqlDom/TestScripts/YourTypeIndexTests170.sql + +-- Basic index creation +CREATE YOUR_TYPE INDEX IX_YourType_Basic ON dbo.Table1 (SpecializedColumn); + +-- Index with type-specific clause +CREATE YOUR_TYPE INDEX IX_YourType_WithClause ON dbo.Table1 (SpecializedColumn) +FOR ('value1', 'value2', 'value3'); + +-- Index with WITH options +CREATE YOUR_TYPE INDEX IX_YourType_WithOptions ON dbo.Table1 (SpecializedColumn) +WITH (YOUR_TYPE_OPTION1 = 'value', MAXDOP = 4); + +-- Index with type-specific clause and WITH options +CREATE YOUR_TYPE INDEX IX_YourType_Complete ON dbo.Table1 (SpecializedColumn) +FOR ('property1', 'property2') +WITH (YOUR_TYPE_OPTION1 = 'setting', YOUR_TYPE_OPTION2 = 'config'); + +-- Index on schema-qualified table +CREATE YOUR_TYPE INDEX IX_YourType_Schema ON MySchema.MyTable (Column1) +FOR ('path.value'); + +-- Index with quoted identifiers +CREATE YOUR_TYPE INDEX [IX YourType Index] ON [dbo].[Table1] ([Column Name]) +FOR ('complex.path.expression'); + +-- Index with filegroup +CREATE YOUR_TYPE INDEX IX_YourType_Filegroup ON dbo.Table1 (Column1) +WITH (YOUR_TYPE_OPTION1 = 'setting') +ON [PRIMARY]; + +-- Index with complex options +CREATE YOUR_TYPE INDEX IX_YourType_AllOptions ON dbo.Table1 (Column1) +FOR ('value1', 'value2') +WITH (YOUR_TYPE_OPTION1 = 'config', YOUR_TYPE_OPTION2 = 'setting', MAXDOP = 8, ONLINE = OFF); +``` + +**Test Coverage Requirements**: +- **All syntax variations**: Basic, with clauses, with options, combinations +- **Schema qualification**: Different schema and table names +- **Quoted identifiers**: Test case sensitivity and special characters +- **Integration contexts**: Filegroups, partition schemes, standard index options +- **Edge cases**: Empty clauses, maximum option combinations + +### 9. Generate Baseline Files + +Create the expected output baseline: + +1. **Create placeholder baseline**: `Test/SqlDom/Baselines170/YourTypeIndexTests170.sql` +2. **Run the test** (will fail initially): + ```bash + dotnet test --filter "YourTypeIndexTests170" Test/SqlDom/UTSqlScriptDom.csproj -c Debug + ``` +3. **Copy "Actual" output** from test failure into baseline file +4. **Verify formatting** matches parser's standard formatting + +**Baseline Example** (JSON index): +```sql +CREATE JSON INDEX IX_JSON_Basic +ON dbo.Users (JsonData); + +CREATE JSON INDEX IX_JSON_Paths +ON dbo.Users (JsonData) FOR ('$.name', '$.email', '$.age'); + +CREATE JSON INDEX IX_JSON_Options +ON dbo.Users (JsonData) WITH (OPTIMIZE_FOR_ARRAY_SEARCH = ON, MAXDOP = 4); +``` + +### 10. Configure Test Expectations + +Add test configuration to version-specific test class: + +```csharp +// Location: Test/SqlDom/Only170SyntaxTests.cs (or appropriate version) +// Add to the ParserTest170 array + +new ParserTest170("YourTypeIndexTests170.sql"), +``` + +**Error Count Guidelines**: +- **Count all syntax instances**: Each CREATE INDEX statement in test script +- **Consider SQL version support**: When was the index type actually introduced? +- **Account for options**: New index options may add additional errors in older versions +- **Test validation**: Run tests to verify error counts are accurate + +### 11. Full Test Suite Validation + +Run complete test suite to ensure no regressions: + +```bash +# Run all ScriptDOM tests +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug + +# Expected result: All tests pass, including new index type tests +# Total tests: 1,100+ (number increases with new features) +``` + +**Regression Prevention**: +- **Grammar changes can break existing functionality**: Shared rules affect multiple contexts +- **AST changes can break script generation**: Ensure all visitors are updated +- **Index option additions**: New options shouldn't conflict with existing ones +- **Version compatibility**: New syntax shouldn't break older version parsers + +## Advanced Considerations + +### Version-Specific Implementation + +For index types introduced in specific SQL Server versions: + +```antlr +// Different grammar files for different SQL versions +// TSql160.g - SQL Server 2022 features (if backporting) +// TSql170.g - SQL Server 2025 features +// TSqlFabricDW.g - Azure Synapse features +``` + +**Guidelines**: +- **Target appropriate version**: Add to the SQL version where feature was introduced +- **Cascade to later versions**: Copy rules to all subsequent version grammar files +- **Version-specific testing**: Test error behavior in older parsers + +### Complex Index Options + +For index types requiring specialized index options: + +```xml + + + + +``` + +**When to use complex options**: +- **Type-specific options**: Options that only apply to your index type +- **Complex values**: Options with structured values or multiple parameters +- **Validation requirements**: Options that need special validation logic + +### Filegroup and Partition Support + +For index types that support filegroups or partitioning: + +```antlr +// Add filegroup support to your grammar rule +( + On vFileGroupOrPartitionScheme=filegroupOrPartitionScheme + { + vResult.OnFileGroupOrPartitionScheme = vFileGroupOrPartitionScheme; + } +)? +``` + +**Filegroup considerations**: +- **Optional support**: Not all index types support filegroups +- **Partition schemes**: Some index types may support partitioning +- **Storage options**: Consider FILESTREAM or in-memory storage + +## Common Pitfalls and Solutions + +### 1. Forgetting Script Generator Implementation +**Problem**: AST node created but no script generation visitor +**Solution**: Always implement `ExplicitVisit()` method for new index statement nodes + +### 2. Incorrect Grammar Integration +**Problem**: Index type not recognized in all CREATE INDEX contexts +**Solution**: Ensure integration with both `createIndexStatement` and `ddlStatement` rules + +### 3. Missing Index Option Registration +**Problem**: New index options not recognized by parser +**Solution**: Add options to `IndexOptionKind` enum and register in `IndexOptionHelper` + +### 4. Incomplete Test Coverage +**Problem**: Edge cases not covered in testing +**Solution**: Test all syntax variations, option combinations, and integration contexts + +### 5. Grammar Conflicts +**Problem**: New keywords conflict with existing grammar +**Solution**: Use proper semantic predicates and context-specific matching + +### 6. Version Compatibility Issues +**Problem**: New index type breaks older version parsers unexpectedly +**Solution**: Add proper version checks and test all SQL Server versions + +## Validation Checklist + +- [ ] **AST Definition**: New class inherits from `IndexStatement` with appropriate members +- [ ] **Grammar Rules**: Specialized parsing rule handles all syntax variations +- [ ] **Grammar Integration**: Connected to `createIndexStatement` and `ddlStatement` +- [ ] **String Constants**: All keywords added to `CodeGenerationSupporter.cs` +- [ ] **Index Options**: New options added to enum and registered with version flags +- [ ] **Script Generator**: `ExplicitVisit()` method generates correct T-SQL output +- [ ] **Test Scripts**: Comprehensive test coverage including edge cases +- [ ] **Baseline Files**: Generated output matches expected formatted T-SQL +- [ ] **Test Configuration**: Error counts configured for all SQL Server versions +- [ ] **Build Success**: Project builds without errors and regenerates parser +- [ ] **Full Test Suite**: All existing tests continue to pass (no regressions) + +## Related Guides + +- [bug_fixing.guidelines.instructions.md](bug_fixing.guidelines.instructions.md) - For general grammar modifications +- [new_data_types.guidelines.instructions.md](new_data_types.guidelines.instructions.md) - For adding new data types +- [validation_fix.guidelines.instructions.md](validation_fix.guidelines.instructions.md) - For validation-only issues +- [testing.guidelines.instructions.md](testing.guidelines.instructions.md) - For comprehensive testing strategies + +## Real-World Examples + +### JSON Index (SQL Server 2025) +- **AST Class**: `CreateJsonIndexStatement` with `JsonColumn` and `ForJsonPaths` members +- **Syntax**: `CREATE JSON INDEX name ON table (column) FOR (paths)` +- **Special Features**: FOR clause with path specifications, array search optimization +- **Challenge**: Multiple JSON path support in FOR clause + +### Vector Index (SQL Server 2025) +- **AST Class**: `CreateVectorIndexStatement` with `VectorColumn` member +- **Syntax**: `CREATE VECTOR INDEX name ON table (column) WITH (METRIC = value)` +- **Special Features**: Vector-specific metrics (cosine, dot, euclidean), DiskANN type +- **Challenge**: Specialized index options for vector operations + +### Future Examples +This pattern can be applied to other SQL Server index types like: +- **SPATIAL INDEX**: Geometry/geography data indexing +- **FULLTEXT INDEX**: Text search indexing +- **XML INDEX**: XML document indexing +- **Custom Index Types**: Future SQL Server indexing technologies + +The JSON and Vector implementations serve as canonical examples for this pattern and should be referenced for similar future index type additions. \ No newline at end of file diff --git a/.github/PARSER_PREDICATE_RECOGNITION_FIX.md b/.github/instructions/parser.guidelines.instructions.md similarity index 100% rename from .github/PARSER_PREDICATE_RECOGNITION_FIX.md rename to .github/instructions/parser.guidelines.instructions.md diff --git a/.github/instructions/testing.guidelines.instructions.md b/.github/instructions/testing.guidelines.instructions.md new file mode 100644 index 0000000..1cd1705 --- /dev/null +++ b/.github/instructions/testing.guidelines.instructions.md @@ -0,0 +1,707 @@ +# Testing Guidelines for SqlScriptDOM + +This guide provides comprehensive instructions for adding and running tests in the SqlScriptDOM parser, based on the testing framework patterns and best practices. + +## Overview + +**CRITICAL: YOU MUST ADD UNIT TESTS - DO NOT CREATE STANDALONE PROGRAMS TO TEST** + +The SqlScriptDOM testing framework validates parser functionality through: +1. **Parse β†’ Generate β†’ Parse Round-trip Testing** - Ensures syntax is correctly parsed and regenerated +2. **Baseline Comparison** - Verifies generated T-SQL matches expected formatted output +3. **Error Count Validation** - Confirms expected parse errors for invalid syntax across SQL versions +4. **Version-Specific Testing** - Tests syntax across multiple SQL Server versions (SQL 2000-2025) +5. **Exact T-SQL Verification** - When testing specific T-SQL syntax from prompts or user requests, the **exact T-SQL statement must be included and verified** in the test to ensure the specific syntax works as expected + +## Test Framework Architecture + +### Core Components + +- **Test Scripts** (`Test/SqlDom/TestScripts/`) - Input T-SQL files containing syntax to test +- **Baselines** (`Test/SqlDom/Baselines/`) - Expected formatted output for each test script +- **Test Configuration** (`Test/SqlDom/OnlySyntaxTests.cs`) - Test definitions with error expectations +- **Test Runners** - MSTest framework running parse/generate/validate cycles + +### How Tests Work + +1. **Parse Phase**: Test script is parsed using specified SQL Server version parser +2. **Generate Phase**: Parsed AST is converted back to T-SQL using script generator +3. **Validate Phase**: Generated output is compared against baseline file +4. **Error Validation**: Parse error count is compared against expected error count for each SQL version + +## Adding New Tests + +### 1. Create Test Script + +Create a new `.sql` file in `Test/SqlDom/TestScripts/` with descriptive name: + +**File**: `Test/SqlDom/TestScripts/YourFeatureTests160.sql` +```sql +-- Test basic syntax +SELECT JSON_ARRAY('value1', 'value2'); + +-- Test in complex context +ALTER FUNCTION TestFunction() +RETURNS NVARCHAR(MAX) +AS +BEGIN + RETURN (JSON_ARRAY('name', 'value')); +END; +GO + +-- Test edge cases +SELECT JSON_ARRAY(); +SELECT JSON_ARRAY(NULL, 'test', 123); +``` + +**CRITICAL**: When testing specific T-SQL syntax from user prompts or requests, **include the exact T-SQL statement provided** in your test script. Do not modify, simplify, or generalize the syntax - test the precise statement that was requested. + +**Example**: If the user provides: +```sql +SELECT JSON_OBJECTAGG( t.c1 : t.c2 ) +FROM ( + VALUES('key1', 'c'), ('key2', 'b'), ('key3','a') +) AS t(c1, c2); +``` + +Then your test **must include exactly that statement** to verify the specific syntax works. + +**Naming Convention**: +- `Tests.sql` (e.g., `JsonFunctionTests160.sql`) +- `Tests.sql` (e.g., `CreateTableTests170.sql`) +- Use version number corresponding to SQL Server version where feature was introduced + +### 2. Create Baseline File + +Create corresponding baseline file in version-specific baseline directory: + +**File**: `Test/SqlDom/Baselines160/YourFeatureTests160.sql` + +**Initial Creation**: +1. Create empty or placeholder baseline file first +2. Run the test (it will fail) +3. Copy "Actual" output from test failure message +4. Paste into baseline file with proper formatting + +**Example Baseline**: +```sql +SELECT JSON_ARRAY ('value1', 'value2'); + +ALTER FUNCTION TestFunction +( ) +RETURNS NVARCHAR (MAX) +AS +BEGIN + RETURN (JSON_ARRAY ('name', 'value')); +END + +GO + +SELECT JSON_ARRAY (); +SELECT JSON_ARRAY (NULL, 'test', 123); +``` + +**Formatting Notes**: +- Parser adds consistent spacing around parentheses and operators +- GO statements are preserved +- Indentation follows parser's formatting rules + +### 3. Configure Test Entry + +Add test configuration to appropriate `OnlySyntaxTests.cs` file: + +**File**: `Test/SqlDom/Only160SyntaxTests.cs` +```csharp +// Around line where other ParserTest160 entries are defined + +// Option 1: Simplified - only specify error counts you care about +new ParserTest160("YourFeatureTests160.sql"), // All previous versions default to null (ignored), TSql160 expects 0 errors + +// Option 2: Specify only some previous version error counts +new ParserTest160("YourFeatureTests160.sql", nErrors80: 1, nErrors90: 1), // Only SQL 2000/2005 expect errors + +// Option 3: Full specification (legacy compatibility) +new ParserTest160("YourFeatureTests160.sql", + nErrors80: 1, // SQL Server 2000 - expect error for new syntax + nErrors90: 1, // SQL Server 2005 - expect error for new syntax + nErrors100: 1, // SQL Server 2008 - expect error for new syntax + nErrors110: 1, // SQL Server 2012 - expect error for new syntax + nErrors120: 1, // SQL Server 2014 - expect error for new syntax + nErrors130: 1, // SQL Server 2016 - expect error for new syntax + nErrors140: 1, // SQL Server 2017 - expect error for new syntax + nErrors150: 1 // SQL Server 2019 - expect error for new syntax + // nErrors160: 0 is implicit for SQL Server 2022 - expect success +), +``` + +**Error Count Guidelines**: +- **0 errors**: Syntax should parse successfully in this SQL version +- **1+ errors**: Syntax should fail with specified number of parse errors +- **null (default)**: Error count is ignored for this SQL version - test will pass regardless of actual error count +- **Consider SQL version compatibility**: When was the feature introduced? + +**New Simplified Approach**: ParserTest160 (and later versions) use nullable parameters with default values of `null`. This means: +- You only need to specify error counts for versions where you expect specific behavior +- Unspecified parameters default to `null` and their error counts are ignored +- TSql160 parser (current version) always expects 0 errors unless syntax is intentionally invalid + +### 4. Run and Validate Test + +#### Run Specific Test +```bash +# Run specific test method +dotnet test Test/SqlDom/UTSqlScriptDom.csproj --filter "FullyQualifiedName~TSql160SyntaxIn160ParserTest" -c Debug + +# Run tests for specific version +dotnet test Test/SqlDom/UTSqlScriptDom.csproj --filter "TestCategory=TSql160" -c Debug +``` + +#### Run Full Test Suite +```bash +# Run complete test suite (recommended for final validation) +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug +``` + +#### Interpret Results +- βœ… **Success**: Generated output matches baseline, error counts match expectations +- ❌ **Failure**: Review actual vs expected output, adjust baseline or fix grammar +- ⚠️ **Baseline Mismatch**: Copy correct "Actual" output to baseline file +- ⚠️ **Error Count Mismatch**: Adjust error expectations in test configuration + +## Test Categories and Patterns + +### Version-Specific Tests + +Each SQL version has its own test class: +- `TSql80SyntaxTests` - SQL Server 2000 +- `TSql90SyntaxTests` - SQL Server 2005 +- `TSql100SyntaxTests` - SQL Server 2008 +- `TSql110SyntaxTests` - SQL Server 2012 +- `TSql120SyntaxTests` - SQL Server 2014 +- `TSql130SyntaxTests` - SQL Server 2016 +- `TSql140SyntaxTests` - SQL Server 2017 +- `TSql150SyntaxTests` - SQL Server 2019 +- `TSql160SyntaxTests` - SQL Server 2022 +- `TSql170SyntaxTests` - SQL Server 2025 + +### Cross-Version Testing + +When you add a test to `Only160SyntaxTests.cs`, the framework automatically runs it against all SQL parsers: +- `TSql160SyntaxIn160ParserTest` - Parse with SQL 2022 parser (should succeed) +- `TSql160SyntaxIn150ParserTest` - Parse with SQL 2019 parser (may fail for new syntax) +- `TSql160SyntaxIn140ParserTest` - Parse with SQL 2017 parser (may fail for new syntax) +- ... and so on for all versions + +### Positive vs Negative Testing Strategy + +**CRITICAL**: When adding new T-SQL syntax, you must implement **both positive and negative tests**: + +#### Positive Tests (Success Cases) +- **Location**: `Test/SqlDom/OnlySyntaxTests.cs` +- **Purpose**: Verify syntax parses correctly and generates expected T-SQL +- **Pattern**: Round-trip testing (Parse β†’ Generate β†’ Compare baseline) + +#### Negative Tests (Error Cases) +- **Location**: `Test/SqlDom/ParserErrorsTests.cs` +- **Purpose**: Verify invalid syntax produces expected parse errors +- **Pattern**: Direct error validation with specific error codes and messages + +### Common Test Patterns + +#### Function Tests +```sql +-- Basic function call +SELECT YOUR_FUNCTION('param'); + +-- Function in different contexts +SELECT col1, YOUR_FUNCTION('param') AS computed FROM table1; +WHERE YOUR_FUNCTION('param') > 0; + +-- Function in RETURN statements (critical test) +ALTER FUNCTION Test() RETURNS NVARCHAR(MAX) AS BEGIN + RETURN (YOUR_FUNCTION('value')); +END; +``` + +#### Statement Tests +```sql +-- Basic statement +YOUR_STATEMENT option1, option2; + +-- With expressions +YOUR_STATEMENT @variable, 'literal', column_name; + +-- Complex nested scenarios +YOUR_STATEMENT ( + SELECT nested FROM table + WHERE condition = YOUR_FUNCTION('test') +); +``` + +#### Error Condition Tests +```sql +-- Invalid syntax that should produce parse errors +YOUR_STATEMENT INVALID SYNTAX HERE; + +-- Incomplete statements +YOUR_STATEMENT MISSING; +``` + +## Test Debugging and Troubleshooting + +### Common Issues + +#### 1. Baseline Mismatch +``` +Assert.AreEqual failed. Expected output does not match actual output. +Actual: 'SELECT JSON_ARRAY ('value1', 'value2');' +Expected: 'SELECT JSON_ARRAY('value1', 'value2');' +``` + +**Solution**: Copy the "Actual" output to your baseline file (note spacing differences). + +#### 2. Error Count Mismatch +``` +TestYourFeature.sql: number of errors after parsing is different from expected. +Expected: 1, Actual: 0 +``` + +**Solutions**: +- **If Actual < Expected**: Grammar now supports syntax in older versions β†’ Update error counts +- **If Actual > Expected**: Grammar has issues β†’ Fix grammar or adjust test + +#### 3. Parse Errors +``` +SQL46010: Incorrect syntax near 'YOUR_TOKEN'. at offset 45, line 2, column 15 +``` + +**Solutions**: +- Check grammar rules for your syntax +- Verify syntactic predicates are in correct order +- For RETURN statement issues, see [Function Guidelines](function.guidelines.instructions.md) + +#### 4. Missing Baseline Files +``` +System.IO.FileNotFoundException: Could not find file 'Baselines160\YourTest.sql' +``` + +**Solution**: Create the baseline file in correct directory with exact same name as test script. + +### Debugging Steps + +1. **Check File Names**: Ensure test script and baseline have identical names +2. **Verify File Location**: Scripts in `TestScripts/`, baselines in `Baselines/` +3. **Run Single Test**: Isolate issue by running specific test method +4. **Check Grammar**: Ensure grammar rules support your syntax +5. **Validate AST**: Verify AST nodes are properly generated +6. **Test Round-trip**: Parse β†’ Generate β†’ Parse should succeed + +## Best Practices + +### Test Design + +#### Comprehensive Coverage +```sql +-- βœ… Good: Covers multiple scenarios +SELECT JSON_ARRAY('simple'); +SELECT JSON_ARRAY('multiple', 'values', 123); +SELECT JSON_ARRAY(NULL); +SELECT JSON_ARRAY(); +SELECT JSON_ARRAY(@variable); +SELECT JSON_ARRAY(column_name); +ALTER FUNCTION Test() RETURNS NVARCHAR(MAX) AS BEGIN + RETURN (JSON_ARRAY('in_return')); +END; +``` + +**CRITICAL**: When testing syntax from user requests, **always include the exact T-SQL provided**: +```sql +-- βœ… Include the exact syntax from user prompt +SELECT JSON_OBJECTAGG( t.c1 : t.c2 ) +FROM ( + VALUES('key1', 'c'), ('key2', 'b'), ('key3','a') +) AS t(c1, c2); + +-- βœ… Then add additional test variations +SELECT JSON_OBJECTAGG( alias.col1 : alias.col2 ) FROM table_name alias; +SELECT JSON_OBJECTAGG( schema.table.col1 : schema.table.col2 ) FROM schema.table; +``` + +#### Focused Testing +```sql +-- ❌ Avoid: Mixing unrelated syntax in single test +SELECT JSON_ARRAY('test'); +CREATE TABLE test_table (id INT); -- Unrelated to JSON +INSERT INTO test_table VALUES (1); -- Unrelated to JSON +``` + +#### Edge Cases +```sql +-- βœ… Include edge cases +SELECT JSON_ARRAY(); -- Empty parameters +SELECT JSON_ARRAY(NULL, NULL); -- NULL handling +SELECT JSON_ARRAY('very_long_string_value_that_tests_parser_limits'); +SELECT JSON_ARRAY((SELECT nested FROM table)); -- Subqueries +``` + +### Error Expectations + +#### Version Compatibility +```csharp +// βœ… Good: Simplified - most new syntax fails in older versions +new ParserTest160("JsonTests160.sql"), // TSql160 expects success, older versions ignored + +// βœ… Good: Specify only when you need specific behavior +new ParserTest160("JsonTests160.sql", nErrors130: 0), // JSON supported since SQL 2016 + +// βœ… Good: Full specification when needed for precision +new ParserTest160("JsonTests160.sql", + nErrors80: 1, // JSON not in SQL 2000 + nErrors90: 1, // JSON not in SQL 2005 + // ... + nErrors150: 1, // JSON not in SQL 2019 + // nErrors160: 0 - JSON supported in SQL 2022 +), +``` + +#### Grammar Reality +```csharp +// ⚠️ Consider: Grammar changes may affect all versions +// If shared grammar makes function work in all SQL versions: +new ParserTest160("TestFunction160.sql"), // All versions will succeed + +// If function fails in older versions due to grammar limitations: +new ParserTest160("TestFunction160.sql", nErrors80: 1, nErrors90: 1), // Only specify versions that fail +``` + +### File Organization + +#### Logical Grouping +``` +TestScripts/ +β”œβ”€β”€ JsonFunctionTests160.sql # JSON-specific functions +β”œβ”€β”€ StringFunctionTests160.sql # String manipulation +β”œβ”€β”€ CreateTableTests170.sql # DDL statements +β”œβ”€β”€ SelectStatementTests170.sql # DML statements +└── AlterFunctionTests160.sql # Function-specific syntax +``` + +#### Version Alignment +``` +TestScripts/JsonTests160.sql ↔ Baselines160/JsonTests160.sql +TestScripts/JsonTests170.sql ↔ Baselines170/JsonTests170.sql +``` + +## Simplified Error Count Handling (TSql160+) + +### New Constructor Behavior + +Starting with `ParserTest160`, the constructor uses nullable integer parameters with default values of `null`. This pattern extends to later versions like `ParserTest170`: + +```csharp +public ParserTest160(string scriptFilename, + int? nErrors80 = null, // Default: null (ignored) + int? nErrors90 = null, // Default: null (ignored) + int? nErrors100 = null, // Default: null (ignored) + int? nErrors110 = null, // Default: null (ignored) + int? nErrors120 = null, // Default: null (ignored) + int? nErrors130 = null, // Default: null (ignored) + int? nErrors140 = null, // Default: null (ignored) + int? nErrors150 = null) // Default: null (ignored) + // TSql160 always expects 0 errors unless syntax is invalid + +public ParserTest170(string scriptFilename, + int? nErrors80 = null, // Default: null (ignored) + int? nErrors90 = null, // Default: null (ignored) + int? nErrors100 = null, // Default: null (ignored) + int? nErrors110 = null, // Default: null (ignored) + int? nErrors120 = null, // Default: null (ignored) + int? nErrors130 = null, // Default: null (ignored) + int? nErrors140 = null, // Default: null (ignored) + int? nErrors150 = null, // Default: null (ignored) + int? nErrors160 = null) // Default: null (ignored) + // TSql170 always expects 0 errors unless syntax is invalid +``` + +### Benefits + +1. **Simplified Test Creation**: Most tests only need the script filename +2. **Focus on What Matters**: Only specify error counts for versions where you expect specific behavior +3. **Reduced Maintenance**: No need to update all error counts when adding version-agnostic syntax +4. **Backward Compatibility**: Existing tests with full error specifications still work + +### Usage Patterns + +```csharp +// Minimal - test new SQL 2022 syntax +new ParserTest160("NewFeatureTests160.sql"), + +// Minimal - test new SQL 2025 syntax +new ParserTest170("NewFeatureTests170.sql"), + +// Specify only critical version boundaries +new ParserTest160("FeatureTests160.sql", nErrors130: 0), // Supported since SQL 2016 +new ParserTest170("FeatureTests170.sql", nErrors130: 0), // Supported since SQL 2016 + +// Mix of specified and default parameters +new ParserTest160("EdgeCaseTests160.sql", nErrors80: 2, nErrors150: 1), // SQL 2000 has 2 errors, SQL 2019 has 1 +new ParserTest170("EdgeCaseTests170.sql", nErrors80: 2, nErrors160: 1), // SQL 2000 has 2 errors, SQL 2022 has 1 + +// Legacy full specification still supported +new ParserTest160("LegacyTests160.sql", 1, 1, 1, 1, 1, 1, 1, 1), +new ParserTest170("LegacyTests170.sql", 1, 1, 1, 1, 1, 1, 1, 1, 1), +``` + +### When to Specify Error Counts + +- **Don't specify**: When older SQL versions should be ignored (most common case for both TSql160 and TSql170) +- **Specify as 0**: When feature was introduced in a specific older SQL version +- **Specify as 1+**: When you need to validate specific error conditions +- **Specify for debugging**: When investigating cross-version compatibility issues +- **TSql170 considerations**: Remember that TSql160 (SQL Server 2022) is now also a "previous version" when using ParserTest170 + +## Performance Considerations + +### Test Execution Time + +#### Minimal Test Sets +```bash +# Run specific version tests only +dotnet test --filter "TestCategory=TSql160" -c Debug + +# Run specific feature tests +dotnet test --filter "FullyQualifiedName~Json" -c Debug +``` + +#### Parallel Execution +```bash +# Use parallel test execution for faster runs +dotnet test --parallel -c Debug +``` + +#### Focused Development +```bash +# During development, run only your new tests +dotnet test --filter "FullyQualifiedName~YourTestMethod" -c Debug +``` + +### Build Performance + +#### Incremental Testing +1. Add test script and baseline +2. Run specific test to validate +3. Run full test suite only before commit + +#### Cached Builds +- Parser regeneration only occurs when grammar files change +- Test compilation is incremental +- Use `-c Debug` for faster iteration + +## Integration with Development Workflow + +### 1. Grammar Development +```bash +# After grammar changes +dotnet build SqlScriptDom/Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug + +# Test specific functionality +dotnet test --filter "FullyQualifiedName~YourFeature" -c Debug +``` + +### 2. Test-Driven Development +```bash +# 1. Create failing test +dotnet test --filter "FullyQualifiedName~YourNewTest" -c Debug # Should fail + +# 2. Implement grammar changes +dotnet build -c Debug + +# 3. Update baseline and validate +dotnet test --filter "FullyQualifiedName~YourNewTest" -c Debug # Should pass + +# 4. Run regression tests +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug # Should all pass +``` + +### 3. Continuous Integration +```bash +# Full validation before commit +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug +# Ensure: Total tests: 1,116, Failed: 0, Succeeded: 1,116 +``` + +## Common Test Scenarios + +### Adding New Function + +```sql +-- Test/SqlDom/TestScripts/NewFunctionTests160.sql (for SQL 2022) +-- Test/SqlDom/TestScripts/NewFunctionTests170.sql (for SQL 2025) +SELECT NEW_FUNCTION('param1', 'param2'); +SELECT NEW_FUNCTION(@variable); +SELECT NEW_FUNCTION(column_name); + +-- Critical: Test in RETURN statement +ALTER FUNCTION TestNewFunction() +RETURNS NVARCHAR(MAX) +AS +BEGIN + RETURN (NEW_FUNCTION('test_value')); +END; +GO +``` + +**Test Configuration**: +```csharp +// Simplified approach for SQL 2022 - NEW_FUNCTION is SQL 2022 syntax +new ParserTest160("NewFunctionTests160.sql"), + +// Simplified approach for SQL 2025 - NEW_FUNCTION is SQL 2025 syntax +new ParserTest170("NewFunctionTests170.sql"), + +// Or specify if function works in earlier versions +new ParserTest160("NewFunctionTests160.sql", nErrors140: 0), // Works since SQL 2017 +new ParserTest170("NewFunctionTests170.sql", nErrors140: 0), // Works since SQL 2017 +``` + +### Adding New Statement + +```sql +-- Test/SqlDom/TestScripts/NewStatementTests160.sql (for SQL 2022) +-- Test/SqlDom/TestScripts/NewStatementTests170.sql (for SQL 2025) +NEW_STATEMENT option1 = 'value1', option2 = 'value2'; + +NEW_STATEMENT + option1 = 'value1', + option2 = @parameter, + option3 = (SELECT nested FROM table); + +-- Test with expressions +NEW_STATEMENT computed_option = (value1 + value2); +``` + +**Test Configuration**: +```csharp +// For SQL 2022 syntax: +new ParserTest160("NewStatementTests160.sql"), + +// For SQL 2025 syntax: +new ParserTest170("NewStatementTests170.sql"), +``` + +### Testing Error Conditions + +```sql +-- Test/SqlDom/TestScripts/ErrorConditionTests160.sql +-- These should generate parse errors + +NEW_FUNCTION(); -- Invalid: missing required parameters +NEW_FUNCTION('param1',); -- Invalid: trailing comma +NEW_FUNCTION('param1' 'param2'); -- Invalid: missing comma +``` + +**Test Configuration**: +```csharp +// Test should fail parsing in TSql160 due to invalid syntax +new ParserTest160("ErrorConditionTests160.sql", + nErrors80: 3, // 3 syntax errors expected in all versions + nErrors90: 3, + nErrors100: 3, + nErrors110: 3, + nErrors120: 3, + nErrors130: 3, + nErrors140: 3, + nErrors150: 3, + nErrors160: 3), // Even TSql160 should have 3 errors - syntax is invalid + +// Test should fail parsing in TSql170 due to invalid syntax +new ParserTest170("ErrorConditionTests170.sql", + nErrors80: 3, // 3 syntax errors expected in all versions + nErrors90: 3, + nErrors100: 3, + nErrors110: 3, + nErrors120: 3, + nErrors130: 3, + nErrors140: 3, + nErrors150: 3, + nErrors160: 3, + nErrors170: 3), // Even TSql170 should have 3 errors - syntax is invalid + +// Or simplified if error count is same across all versions: +new ParserTest160("ErrorConditionTests160.sql", + nErrors80: 3, nErrors90: 3, nErrors100: 3, nErrors110: 3, + nErrors120: 3, nErrors130: 3, nErrors140: 3, nErrors150: 3, + nErrors160: 3), + +new ParserTest170("ErrorConditionTests170.sql", + nErrors80: 3, nErrors90: 3, nErrors100: 3, nErrors110: 3, + nErrors120: 3, nErrors130: 3, nErrors140: 3, nErrors150: 3, + nErrors160: 3, nErrors170: 3), +``` + +## Advanced Testing Patterns + +### Multi-File Tests + +For complex scenarios requiring multiple related test files: +``` +TestScripts/ +β”œβ”€β”€ ComplexScenarioTests160_Part1.sql +β”œβ”€β”€ ComplexScenarioTests160_Part2.sql +└── ComplexScenarioTests160_Integration.sql + +Baselines160/ +β”œβ”€β”€ ComplexScenarioTests160_Part1.sql +β”œβ”€β”€ ComplexScenarioTests160_Part2.sql +└── ComplexScenarioTests160_Integration.sql +``` + +### Version Migration Tests + +Testing syntax evolution across versions: +```sql +-- Test/SqlDom/TestScripts/FeatureEvolutionTests170.sql +-- Tests new syntax in 170 that extends 160 functionality +SELECT JSON_ARRAY('basic'); -- Supported in 160 +SELECT JSON_ARRAY('new', 'syntax', 'in', 'version', '170'); -- New in 170 +``` + +### Regression Tests + +When fixing bugs, add specific regression tests: +```sql +-- Test/SqlDom/TestScripts/RegressionBugFix12345Tests160.sql +-- Specific test case that reproduced bug #12345 +ALTER FUNCTION TestRegression() +RETURNS NVARCHAR(MAX) +AS +BEGIN + RETURN (JSON_OBJECT('key': (SELECT value FROM table))); +END; +``` + +**Test Configuration**: +```csharp +// Regression test - should work in TSql160, may fail in earlier versions +new ParserTest160("RegressionBugFix12345Tests160.sql"), + +// Regression test - should work in TSql170, may fail in earlier versions +new ParserTest170("RegressionBugFix12345Tests170.sql"), + +// Or if you need to verify the bug existed in specific versions: +new ParserTest160("RegressionBugFix12345Tests160.sql", nErrors150: 1), // Bug existed in SQL 2019 +new ParserTest170("RegressionBugFix12345Tests170.sql", nErrors160: 1), // Bug existed in SQL 2022 +``` + +## Summary + +The SqlScriptDOM testing framework provides comprehensive validation of parser functionality through: +- **Round-trip testing** (Parse β†’ Generate β†’ Parse) +- **Baseline comparison** (Generated output vs expected) +- **Cross-version validation** (Test syntax across SQL Server versions) +- **Error condition testing** (Invalid syntax produces expected errors) +- **Exact syntax verification** (Exact T-SQL from user requests is tested precisely) + +Following these guidelines ensures robust test coverage for parser functionality and prevents regressions when adding new features or fixing bugs. + +**Key Principle**: Always test the exact T-SQL syntax provided in user prompts or requests to verify that the specific syntax works as expected, rather than testing generalized or simplified versions of the syntax. \ No newline at end of file diff --git a/.github/prompts/new-feature-implementation.prompt.md b/.github/prompts/new-feature-implementation.prompt.md new file mode 100644 index 0000000..dd87fc9 --- /dev/null +++ b/.github/prompts/new-feature-implementation.prompt.md @@ -0,0 +1,518 @@ +# New SQL Server Feature Implementation Guide + +This prompt will identify the type of SQL Server feature you want to add to SqlScriptDOM and **automatically implement it** using the appropriate guideline. After feature type identification, it will execute the complete implementation workflow. + +## Feature Type Identification + +Please answer the following questions to determine the best implementation approach: + +### 1. What type of SQL Server feature are you implementing? + +**A) Data Type** - A new SQL Server data type (e.g., VECTOR, GEOMETRY, GEOGRAPHY) +- Example: `DECLARE @embedding AS VECTOR(1536, FLOAT32)` +- Example: `CREATE TABLE tbl (geo_data GEOGRAPHY)` +- **Key indicators**: New type with custom parameters, specialized syntax for type definitions + +**B) Index Type** - A specialized index type with unique syntax (e.g., JSON INDEX, VECTOR INDEX) +- Example: `CREATE JSON INDEX IX_JSON ON table (column) FOR ('$.path1', '$.path2')` +- Example: `CREATE VECTOR INDEX IX_VECTOR ON table (column) WITH (METRIC = 'cosine')` +- **Key indicators**: CREATE [TYPE] INDEX syntax, type-specific clauses or options + +**C) System Function** - A new T-SQL built-in function (e.g., JSON_OBJECT, JSON_ARRAY) +- Example: `SELECT JSON_OBJECT('key1': 'value1', 'key2': 'value2')` +- Example: `RETURN JSON_ARRAY('item1', 'item2', 'item3')` +- **Key indicators**: Function calls in expressions, may need RETURN statement support + +**D) Grammar/Syntax Enhancement** - New operators, statements, or syntax modifications +- Example: Adding new WHERE clause predicates like `REGEXP_LIKE` +- Example: New statement types or operators +- **Key indicators**: Parser doesn't recognize syntax, needs AST updates + +**E) Validation Fix** - Existing syntax fails validation but should work per SQL Server docs +- Example: ALTER TABLE RESUMABLE option works in ALTER INDEX but not ALTER TABLE +- **Key indicators**: "Option 'X' is not valid..." errors, similar syntax works elsewhere + +**F) Parser Predicate Issue** - Identifier-based predicates fail with parentheses +- Example: `WHERE REGEXP_LIKE(...)` works but `WHERE (REGEXP_LIKE(...))` fails +- **Key indicators**: Syntax errors near closing parenthesis with identifier predicates + +### 2. SQL Server Feature Details + +**Feature Name**: _______________ +**SQL Server Version**: _______________ +**Example Syntax**: +```sql +-- Provide 2-3 examples of the syntax you want to support +``` + +**Current Behavior**: _______________ +**Expected Behavior**: _______________ + +### 3. Feature Characteristics + +Check all that apply to your feature: + +- [ ] Requires completely new AST node classes +- [ ] Extends existing AST nodes with new members +- [ ] Needs new grammar rules in .g files +- [ ] Requires new keywords/constants +- [ ] Needs specialized script generation logic +- [ ] Has version-specific behavior (SQL Server 2014+, 2022+, etc.) +- [ ] Includes optional syntax elements or clauses +- [ ] Supports collections/lists of parameters +- [ ] Requires new validation logic +- [ ] Needs new index options or statement options + +## AUTO-IMPLEMENTATION TRIGGER + +**To begin automatic implementation, provide your feature details in this exact format:** + +``` +Feature Name: [Your feature name] +SQL Server Version: [SQL Server version] +Exact T-SQL Syntax: +```sql +[Copy the EXACT T-SQL syntax from the user's request here] +``` +Feature Type: [Will be determined from analysis below] +``` + +**The system will then automatically identify the feature type and begin implementation.** + +## Implementation Guidance + +Based on your feature type identification below, the system will automatically execute the appropriate implementation workflow: + +### β†’ Data Type (Answer A) +**Auto-Executes**: [New Data Types Guidelines](../instructions/new_data_types.guidelines.instructions.md) + +**Automatic implementation includes**: +- Creating new `DataTypeReference` AST classes +- Adding specialized parsing rules for custom type syntax +- Implementing parameter handling (dimensions, base types, etc.) +- Script generation for type definitions +- Version-specific type support +- Comprehensive testing across all SQL contexts + +**Best for**: VECTOR, custom CLR types, spatial types, hierarchical types + +### β†’ Index Type (Answer B) +**Auto-Executes**: [New Index Types Guidelines](../instructions/new_index_types.guidelines.instructions.md) + +**Automatic implementation includes**: +- Creating new `IndexStatement` AST classes +- Implementing type-specific index syntax parsing +- Adding specialized clauses (FOR, WITH type-specific options) +- Index option registration and validation +- Script generation for index statements +- Integration with existing index framework + +**Best for**: JSON INDEX, VECTOR INDEX, SPATIAL INDEX, custom index types + +### β†’ System Function (Answer C) +**Auto-Executes**: [Function Guidelines](../instructions/function.guidelines.instructions.md) + +**Automatic implementation includes**: +- Function AST design for new T-SQL functions +- Grammar rules with syntactic predicates for RETURN statement support +- ANTLR v2 lookahead limitations and solutions +- Script generation for function calls +- Comprehensive testing in all expression contexts + +**Best for**: JSON_OBJECT, JSON_ARRAY, AI functions, mathematical functions + +### β†’ Grammar/Syntax Enhancement (Answer D) +**Auto-Executes**: [Bug Fixing Guidelines](../instructions/bug_fixing.guidelines.instructions.md) + +**Automatic implementation includes**: +- Grammar rule modifications and AST updates +- Script generation implementation +- Testing framework integration +- Extending literals to expressions pattern +- Version compatibility considerations + +**Best for**: New operators, statement types, expression enhancements + +### β†’ Validation Fix (Answer E) +**Auto-Executes**: [Validation Fix Guidelines](../instructions/validation_fix.guidelines.instructions.md) + +**Automatic implementation includes**: +- Version-gated validation fixes +- SQL Server version compatibility checks +- Context-specific validation rules +- Testing validation behavior across versions +- No grammar changes needed + +**Best for**: Feature works in one context but not another, version support issues + +### β†’ Parser Predicate Issue (Answer F) +**Auto-Executes**: [Parser Guidelines](../instructions/parser.guidelines.instructions.md) + +**Automatic implementation includes**: +- Identifier-based predicate recognition fixes +- `IsNextRuleBooleanParenthesis()` function updates +- Syntactic vs semantic predicate handling +- Parentheses support in boolean contexts + +**Best for**: Functions work without parentheses but fail with them + +## Grammar Extension Patterns + +For users implementing Grammar/Syntax Enhancement (Option D), here are common patterns: + +### Pattern 1: Extending Literals to Expressions + +#### When to Use +When existing grammar rules only accept literal values but need to support dynamic expressions like parameters, variables, or computed values. + +#### Example Problem +Functions or constructs that currently accept only: +- `IntegerLiteral` (e.g., `TOP_N = 10`) +- `StringLiteral` (e.g., `VALUE = 'literal'`) + +But need to support: +- Parameters: `@parameter` +- Variables: `@variable` +- Column references: `table.column` +- Outer references: `outerref.column` +- Function calls: `FUNCTION(args)` +- Computed expressions: `value + 1` + +#### ⚠️ Critical Warning: Avoid Modifying Shared Grammar Rules + +**DO NOT** modify existing shared grammar rules like `identifierColumnReferenceExpression` that are used throughout the codebase. This can cause unintended side effects and break other functionality. + +**Instead**, create specialized rules for your specific context. + +#### Solution Template + +**Step 1: Update AST Definition (`Ast.xml`)** +```xml + + + + + +``` + +**Step 2: Create Context-Specific Grammar Rule (`TSql*.g`)** +```antlr +// Create a specialized rule for your context +yourContextColumnReferenceExpression returns [ColumnReferenceExpression vResult = this.FragmentFactory.CreateFragment()] +{ + MultiPartIdentifier vMultiPartIdentifier; +} + : + vMultiPartIdentifier=multiPartIdentifier[2] // Allows table.column syntax + { + vResult.ColumnType = ColumnType.Regular; + vResult.MultiPartIdentifier = vMultiPartIdentifier; + } + ; + +// Use the specialized rule in your custom grammar +yourContextParameterRule returns [ScalarExpression vResult] + : vResult=signedInteger + | vResult=variable + | vResult=yourContextColumnReferenceExpression // Context-specific rule + | vResult=expression // Allows computed expressions + ; +``` + +**Step 3: Verify Script Generator** +Most script generators using `GenerateNameEqualsValue()` or similar methods work automatically with `ScalarExpression`. No changes typically needed. + +#### Real-World Example: VECTOR_SEARCH TOP_N + +**Problem**: `VECTOR_SEARCH` TOP_N parameter only accepted integer literals. + +**❌ Wrong Approach**: Modify `identifierColumnReferenceExpression` to use `multiPartIdentifier[2]` +- **Result**: Broke `CreateIndexStatementErrorTest` because other grammar rules started accepting invalid syntax + +**βœ… Correct Approach**: Create `vectorSearchColumnReferenceExpression` specialized for VECTOR_SEARCH +- **Result**: VECTOR_SEARCH supports multi-part identifiers without affecting other functionality + +**Final Implementation**: +```antlr +signedIntegerOrVariableOrColumnReference returns [ScalarExpression vResult] + : vResult=signedInteger + | vResult=variable + | vResult=vectorSearchColumnReferenceExpression // VECTOR_SEARCH-specific rule + ; + +vectorSearchColumnReferenceExpression returns [ColumnReferenceExpression vResult = ...] + : + vMultiPartIdentifier=multiPartIdentifier[2] // Allows table.column syntax + { + vResult.ColumnType = ColumnType.Regular; + vResult.MultiPartIdentifier = vMultiPartIdentifier; + } + ; +``` + +**Result**: Now supports dynamic TOP_N values: +```sql +-- Parameters +VECTOR_SEARCH(..., TOP_N = @k) AS ann + +-- Outer references +VECTOR_SEARCH(..., TOP_N = outerref.max_results) AS ann +``` + +### Pattern 2: Adding New Enum Members + +#### When to Use +When adding new operators, keywords, or options to existing constructs. + +#### Solution Template + +**Step 1: Update Enum in AST (`Ast.xml`)** +```xml + + + + + +``` + +**Step 2: Update Grammar Rule (`TSql*.g`)** +```antlr +// Add new token matching +| tNewValue:Identifier +{ + Match(tNewValue, CodeGenerationSupporter.NewValue); + vResult.EnumProperty = ExistingEnumType.NewValue; +} +``` + +**Step 3: Update Script Generator** +```csharp +// Add mapping in appropriate generator file +private static readonly Dictionary _enumGenerators = + new Dictionary() +{ + { EnumType.ExistingValue1, CodeGenerationSupporter.ExistingValue1 }, + { EnumType.ExistingValue2, CodeGenerationSupporter.ExistingValue2 }, + { EnumType.NewValue, CodeGenerationSupporter.NewValue }, // Add this +}; +``` + +### Pattern 3: Adding New Function or Statement + +#### When to Use +When adding completely new T-SQL functions or statements. + +#### Solution Template + +**Step 1: Define AST Node (`Ast.xml`)** +```xml + + + + +``` + +**Step 2: Add Grammar Rule (`TSql*.g`)** +```antlr +newFunctionCall returns [NewFunctionCall vResult = FragmentFactory.CreateFragment()] +{ + ScalarExpression vParam1; + StringLiteral vParam2; +} + : + tFunction:Identifier LeftParenthesis + { + Match(tFunction, CodeGenerationSupporter.NewFunction); + UpdateTokenInfo(vResult, tFunction); + } + vParam1 = expression + { + vResult.Parameter1 = vParam1; + } + Comma vParam2 = stringLiteral + { + vResult.Parameter2 = vParam2; + } + RightParenthesis + ; +``` + +**Step 3: Integrate with Existing Rules** +Add the new rule to appropriate places in the grammar (e.g., `functionCall`, `primaryExpression`, etc.). + +**Step 4: Create Script Generator** +```csharp +public override void ExplicitVisit(NewFunctionCall node) +{ + GenerateIdentifier(CodeGenerationSupporter.NewFunction); + GenerateSymbol(TSqlTokenType.LeftParenthesis); + GenerateFragmentIfNotNull(node.Parameter1); + GenerateSymbol(TSqlTokenType.Comma); + GenerateFragmentIfNotNull(node.Parameter2); + GenerateSymbol(TSqlTokenType.RightParenthesis); +} +``` + +## Quick Decision Tree + +**Start here** β†’ Does the syntax exist in SQL Server documentation? + +**No** β†’ Use Grammar/Syntax Enhancement (D) + +**Yes** β†’ Does SqlScriptDOM recognize the syntax? + +**No** β†’ What type of syntax? +- Data type declaration β†’ Data Type (A) +- CREATE [TYPE] INDEX β†’ Index Type (B) +- Function call β†’ System Function (C) +- Other syntax β†’ Grammar/Syntax Enhancement (D) + +**Yes** β†’ Does it parse without errors? + +**No** β†’ Parser Predicate Issue (F) + +**Yes** β†’ Does validation reject it incorrectly? + +**Yes** β†’ Validation Fix (E) + +**No** β†’ Review existing implementation or check for edge cases + +## Pre-Implementation Checklist + +Before starting implementation: + +- [ ] Verified feature exists in SQL Server documentation +- [ ] Identified target SQL Server version for the feature +- [ ] Confirmed feature doesn't already exist in SqlScriptDOM +- [ ] Collected comprehensive syntax examples from SQL Server docs +- [ ] Reviewed similar existing implementations in the codebase +- [ ] Selected appropriate guideline based on feature type + +## Testing Strategy + +Regardless of feature type, ensure you: + +- [ ] Create comprehensive test scripts covering all syntax variations +- [ ] Generate proper baseline files with expected formatted output +- [ ] Configure error counts for all SQL Server versions +- [ ] Run full test suite to prevent regressions +- [ ] Test edge cases, quoted identifiers, and schema qualification +- [ ] Verify round-trip parsing (parse β†’ generate β†’ parse) + +## Additional Resources + +- **Main Copilot Instructions**: [copilot-instructions.md](../copilot-instructions.md) +- **Testing Framework Guide**: [Testing Guidelines](../instructions/testing.guidelines.instructions.md) +- **Grammar Extension Patterns**: See Grammar Extension Patterns section above +- **Detailed Grammar Guidelines**: [Grammar Guidelines](../instructions/grammer.guidelines.instructions.md) + +--- + +**Ready to implement?** Follow the guideline that matches your feature type above. Each guide provides step-by-step instructions, real-world examples, and comprehensive testing strategies. + +## Implementation Workflow + +**IMPORTANT**: After identifying your feature type above, this prompt will automatically begin implementation. Provide the following information to start: + +### Required Information +1. **Feature Name**: _______________ +2. **SQL Server Version**: _______________ +3. **Exact T-SQL Syntax Examples**: +```sql +-- Provide the EXACT syntax you want to support (copy-paste from user request) +-- Example: SELECT JSON_OBJECTAGG( t.c1 : t.c2 ) FROM (VALUES('key1', 'c'), ('key2', 'b'), ('key3','a')) AS t(c1, c2); +``` +4. **Feature Type** (from analysis above): A, B, C, D, E, or F + +### Automatic Implementation Process + +Once you provide the information above, this prompt will: + +#### Phase 1: Analysis and Verification (Always Done First) +1. **Verify current status** using the exact syntax provided +2. **Search existing codebase** for similar implementations +3. **Identify SQL Server version** and parser target +4. **Create implementation plan** with specific steps +5. **Show the plan** and get confirmation before proceeding + +#### Phase 2: Implementation (Executed Automatically) +Based on feature type identification: + +**For Grammar/Syntax Enhancement (Type D)**: +1. **Update AST definition** (`Ast.xml`) if new nodes needed +2. **Add grammar rules** in appropriate `TSql*.g` files +3. **Create script generators** for new AST nodes +4. **Build and validate** parser compilation +5. **Create comprehensive tests** with exact syntax provided +6. **Generate baseline files** from parser output +7. **Run full test suite** to ensure no regressions + +**For Validation Fix (Type E)**: +1. **Locate validation function** throwing the error +2. **Verify Microsoft documentation** for version support +3. **Apply version-gated validation** (not unconditional rejection) +4. **Create test cases** covering all scenarios +5. **Build and validate** the fix +6. **Run full test suite** to ensure correctness + +**For System Function (Type C)**: +1. **Define AST node structure** for the function +2. **Add grammar rules** with syntactic predicates for RETURN statement support +3. **Create script generator** for the function +4. **Build and test** grammar changes +5. **Create comprehensive test scripts** including RETURN statement usage +6. **Validate full test suite** for regressions + +**For Data Type (Type A)**: +1. **Define AST node** inheriting from `DataTypeReference` +2. **Create specialized parsing rule** for the data type +3. **Integrate with scalar data type rule** +4. **Add string constants** for keywords +5. **Create script generator** +6. **Build and comprehensive test** across all SQL contexts + +**For Index Type (Type B)**: +1. **Define AST node** inheriting from `IndexStatement` +2. **Create specialized parsing rule** for the index type +3. **Integrate with main index grammar** +4. **Add index options** if needed +5. **Create script generator** +6. **Build and comprehensive test** all syntax variations + +**For Parser Predicate Issue (Type F)**: +1. **Locate `IsNextRuleBooleanParenthesis()`** function +2. **Add identifier-based predicate detection** +3. **Build and test** the fix +4. **Create tests** covering parentheses scenarios +5. **Validate** existing functionality + +#### Phase 3: Validation and Documentation +1. **Run complete test suite** (`dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug`) +2. **Verify all tests pass** (expect 1,100+ tests to succeed) +3. **Document changes made** with before/after examples +4. **Provide usage examples** showing the new functionality + +### Starting Implementation + +To begin implementation, provide your feature details using this format: + +``` +Feature Name: [FUNCTION_NAME or FEATURE_NAME] +SQL Server Version: [SQL Server 20XX / TSqlXXX] +Exact T-SQL Syntax: +```sql +[EXACT_COPY_OF_SYNTAX_FROM_USER_REQUEST] +``` +Feature Type: [A/B/C/D/E/F based on analysis above] +``` + +**The prompt will then automatically execute the appropriate implementation workflow and start making the necessary code changes.** + +### Implementation Principles + +1. **Always test exact syntax first**: Use the exact T-SQL provided, not simplified versions +2. **Follow established patterns**: Reuse existing patterns from similar implementations +3. **Maintain backward compatibility**: Ensure existing functionality continues to work +4. **Comprehensive testing**: Test all syntax variations, edge cases, and error conditions +5. **Version compatibility**: Consider which SQL Server versions should support the feature +6. **Full regression testing**: Always run the complete test suite before completion \ No newline at end of file diff --git a/.github/prompts/verify-and-test-tsql-syntax.prompt.md b/.github/prompts/verify-and-test-tsql-syntax.prompt.md new file mode 100644 index 0000000..95fb51f --- /dev/null +++ b/.github/prompts/verify-and-test-tsql-syntax.prompt.md @@ -0,0 +1,692 @@ +--- +title: How to Verify T-SQL Syntax Support and Add Tests +description: Step-by-step guide to check if a T-SQL syntax is already supported and how to add comprehensive test coverage +tags: [testing, verification, tsql, syntax, parser, baseline] +--- + +# How to Verify T-SQL Syntax Support and Add Tests + +This guide helps you determine if a T-SQL syntax is already supported by ScriptDOM and shows you how to add proper test coverage. + +## Step 0: Verify the Exact Script First + +**CRITICAL**: Before doing anything else, test the exact T-SQL script provided to confirm whether it works or fails. + +**IMPORTANT**: For initial verification, you MUST add a debug unit test method directly to an existing test file (like Only170SyntaxTests.cs). This is only for initial verification. Once you confirm the syntax status, you'll follow the proper testing workflow to add comprehensive test coverage. + +### Step 1: Add Debug Unit Test Method + +Add this debug test method to the appropriate test file (e.g., `Test/SqlDom/Only170SyntaxTests.cs`): + +```csharp +[TestMethod] +public void DebugExactScriptTest() +{ + // PUT THE EXACT T-SQL SCRIPT HERE - DO NOT CREATE SEPARATE FILES + string script = @"SELECT Id, + DATEADD(DAY, 1, GETDATE()) +FROM Table1"; + + Console.WriteLine($"Testing exact script: {script}"); + + // Test with the target parser version first (e.g., TSql170) + TSql170Parser parser170 = new TSql170Parser(true); + IList errors170; + + using (StringReader reader = new StringReader(script)) + { + TSqlFragment fragment = parser170.Parse(reader, out errors170); + + Console.WriteLine($"\n=== TSql170 Parser Results ==="); + if (errors170.Count == 0) + { + Console.WriteLine("βœ… SUCCESS: Parsed without errors"); + + // Test script generation (round-trip) + Sql170ScriptGenerator generator = new Sql170ScriptGenerator(); + string generatedScript; + generator.GenerateScript(fragment, out generatedScript); + Console.WriteLine($"Generated: {generatedScript}"); + } + else + { + Console.WriteLine($"❌ FAILED: {errors170.Count} parse errors:"); + foreach (var error in errors170) + { + Console.WriteLine($" Line {error.Line}, Col {error.Column}: {error.Message}"); + } + } + } + + // Test with older parser version for comparison (e.g., TSql160) + TSql160Parser parser160 = new TSql160Parser(true); + IList errors160; + + using (StringReader reader = new StringReader(script)) + { + TSqlFragment fragment = parser160.Parse(reader, out errors160); + + Console.WriteLine($"\n=== TSql160 Parser Results ==="); + if (errors160.Count == 0) + { + Console.WriteLine("βœ… SUCCESS: Parsed without errors"); + } + else + { + Console.WriteLine($"❌ FAILED: {errors160.Count} parse errors:"); + foreach (var error in errors160) + { + Console.WriteLine($" Line {error.Line}, Col {error.Column}: {error.Message}"); + } + } + } + + // Use Assert.Inconclusive to document current status without failing the test + if (errors170.Count > 0) + { + Assert.Inconclusive($"Script currently fails with {errors170.Count} errors. Needs implementation."); + } + else + { + Assert.Inconclusive("Script already works! Can proceed to add comprehensive test coverage."); + } +} +``` + +### Step 2: Build and Run the Debug Test +### Step 2: Build and Run the Debug Test + +```bash +# 1. Build the parser to ensure it's up to date +dotnet build SqlScriptDom/Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug + +# 2. Run the debug test to see current status +dotnet test --filter "DebugExactScriptTest" Test/SqlDom/UTSqlScriptDom.csproj -c Debug + +# 3. Check the test output for detailed results +# Look for the console output showing parsing results +``` + +### Interpret Results + +- **βœ… SUCCESS**: Script works! You can skip to Step 6 to add comprehensive tests +- **❌ FAILURE**: Script fails. Continue with Steps 1-5 to implement the missing functionality + +**Important**: Always test the **exact script provided** character-for-character, including: +- Exact table/column names (e.g., `t.c1`, `t.c2`) +- Exact function syntax (e.g., `JSON_OBJECTAGG( t.c1 : t.c2 )`) +- Complete query context (FROM clause, subqueries, etc.) +- Exact whitespace and formatting as provided + +**Remember**: Only add unit test methods to existing test files. Do not create separate SQL files, program files, or any other external files. + +## Step 1: Determine the SQL Server Version + +First, identify which SQL Server version introduced the syntax you want to test. + +### SQL Server Version Mapping + +| SQL Server Version | Parser Version | Year | Common Name | +|-------------------|----------------|------|-------------| +| SQL Server 2000 | TSql80 | 2000 | SQL Server 2000 | +| SQL Server 2005 | TSql90 | 2005 | SQL Server 2005 | +| SQL Server 2008 | TSql100 | 2008 | SQL Server 2008 | +| SQL Server 2012 | TSql110 | 2012 | SQL Server 2012 | +| SQL Server 2014 | TSql120 | 2014 | SQL Server 2014 | +| SQL Server 2016 | TSql130 | 2016 | SQL Server 2016 | +| SQL Server 2017 | TSql140 | 2017 | SQL Server 2017 | +| SQL Server 2019 | TSql150 | 2019 | SQL Server 2019 | +| SQL Server 2022 | TSql160 | 2022 | SQL Server 2022 | +| SQL Server 2025 | TSql170 | 2025 | SQL Server 2025 | + +### How to Find the Version + +1. **Check Microsoft Documentation**: Look for "Applies to: SQL Server 20XX (XX.x)" +2. **Search Online**: Look for the feature announcement or blog posts +3. **Test in SSMS**: Connect to different SQL Server versions and try the syntax + +**Example**: +- `RESUMABLE = ON` for ALTER TABLE β†’ SQL Server 2022 β†’ **TSql160** +- `MAX_DURATION` for indexes β†’ SQL Server 2014 β†’ **TSql120** +- `VECTOR_SEARCH` function β†’ SQL Server 2025 β†’ **TSql170** + +## Step 2: Check if Syntax is Already Supported + +### Method 1: Search Test Scripts (Fastest) +```bash +# Search for the keyword in test scripts +grep -r "YOUR_KEYWORD" Test/SqlDom/TestScripts/ + +# Example: Check if RESUMABLE is tested for ALTER TABLE +grep -r "RESUMABLE" Test/SqlDom/TestScripts/*.sql + +# Search in specific version test files +grep -r "RESUMABLE" Test/SqlDom/TestScripts/*160.sql +``` + +### Method 2: Search Grammar Files +```bash +# Search in grammar files +grep -r "YOUR_KEYWORD" SqlScriptDom/Parser/TSql/*.g + +# Example: Check if RESUMABLE is in grammar +grep -r "Resumable" SqlScriptDom/Parser/TSql/TSql160.g +``` + +### Method 3: Search AST Definitions +```bash +# Search in AST XML +grep -r "YourFeatureName" SqlScriptDom/Parser/TSql/Ast.xml + +# Example: Check for VECTOR_SEARCH node +grep -r "VectorSearch" SqlScriptDom/Parser/TSql/Ast.xml +``` + +### Method 4: Try Parsing with Test Script +Create a unit test method to verify parsing: + +```csharp +// Add to appropriate test file (e.g., Test/SqlDom/Only170SyntaxTests.cs) +[TestMethod] +public void QuickTestExactSyntax() +{ + string script = @"ALTER TABLE t ADD CONSTRAINT pk PRIMARY KEY (id) WITH (RESUMABLE = ON);"; + + TSql170Parser parser = new TSql170Parser(true); + IList errors; + + using (StringReader reader = new StringReader(script)) + { + TSqlFragment fragment = parser.Parse(reader, out errors); + Console.WriteLine($"Errors: {errors.Count}"); + + // This will show you exactly which parser versions support the syntax + // and what error messages are generated if it fails + } + + Assert.Inconclusive($"Test completed with {errors.Count} errors"); +} +``` + +### Method 5: Test in Existing Test Framework +Add a temporary test method to verify quickly: + +```csharp +// Add to appropriate test file (e.g., Test/SqlDom/Only170SyntaxTests.cs) +[TestMethod] +public void TempTestExactScript() +{ + // Put ONLY your exact script here - do not create external files + string script = @"YOUR_EXACT_SCRIPT_HERE"; + + TSql170Parser parser = new TSql170Parser(true); + IList errors; + + using (StringReader reader = new StringReader(script)) + { + TSqlFragment fragment = parser.Parse(reader, out errors); + Console.WriteLine($"Parse result: {errors.Count} errors"); + foreach (var error in errors) + { + Console.WriteLine($"Error: {error.Message}"); + } + } + + Assert.Inconclusive("Temporary test - remove after verification"); +} +``` + +Then run the test: +```bash +dotnet test --filter "TempTestExactScript" -c Debug +``` + +Remember to remove this temporary test method after verification. + +## Step 3: Create a Test Script + +**CRITICAL**: Your test script MUST include the exact T-SQL statement provided. Don't modify, simplify, or generalize the syntax - test the precise statement given. + +### Test File Naming Convention +Follow the pattern from testing.guidelines.instructions.md: +- Format: `Tests.sql` +- Examples: `JsonFunctionTests160.sql`, `AlterTableResumableTests160.sql` +- Location: `Test/SqlDom/TestScripts/` +- Use version number corresponding to SQL Server version where feature was introduced + +### Test Script Requirements + +1. **Start with the exact script provided** - copy it exactly as given +2. **Add comprehensive coverage** as described in testing guidelines: + - Basic syntax variations + - Function in different contexts (SELECT, WHERE, RETURN statements) + - Edge cases (empty parameters, NULL handling, subqueries) + - Integration contexts (variables, parameters, computed expressions) +3. **Include context** - ensure the exact context (table aliases, subqueries) is tested +4. **Test RETURN statements** - Critical for functions, always test in ALTER FUNCTION RETURN statements + +### Test Script Template + +Follow the comprehensive coverage pattern from testing guidelines: + +```sql +-- Test 1: EXACT SCRIPT PROVIDED (REQUIRED - COPY EXACTLY) +-- PUT THE EXACT T-SQL STATEMENT HERE WITHOUT ANY MODIFICATIONS +-- Example: SELECT JSON_OBJECTAGG( t.c1 : t.c2 ) FROM (VALUES('key1', 'c'), ('key2', 'b'), ('key3','a')) AS t(c1, c2); + +-- Test 2: Basic function call (if applicable) +SELECT YOUR_FUNCTION('param1', 'param2'); + +-- Test 3: Function in different contexts +SELECT col1, YOUR_FUNCTION('param') AS computed FROM table1; +WHERE YOUR_FUNCTION('param') > 0; + +-- Test 4: CRITICAL - Function in RETURN statements (for functions) +ALTER FUNCTION TestYourFunction() +RETURNS NVARCHAR(MAX) +AS +BEGIN + RETURN (YOUR_FUNCTION('test_value')); +END; +GO + +-- Test 5: With variables/parameters +SELECT YOUR_FUNCTION(@variable); +SELECT YOUR_FUNCTION(column_name); + +-- Test 6: Edge cases +SELECT YOUR_FUNCTION(); -- Empty parameters (if valid) +SELECT YOUR_FUNCTION(NULL, 'test', 123); -- NULL handling +SELECT YOUR_FUNCTION((SELECT nested FROM table)); -- Subqueries +``` + +### Real-World Example: ALTER TABLE RESUMABLE + +**File**: `Test/SqlDom/TestScripts/AlterTableResumableTests160.sql` + +```sql +-- Test 1: RESUMABLE with MAX_DURATION (minutes) +ALTER TABLE dbo.MyTable +ADD CONSTRAINT pk_test PRIMARY KEY CLUSTERED (id) +WITH (RESUMABLE = ON, MAX_DURATION = 240 MINUTES); + +-- Test 2: RESUMABLE = ON +ALTER TABLE dbo.MyTable +ADD CONSTRAINT pk_test PRIMARY KEY CLUSTERED (id) +WITH (RESUMABLE = ON); + +-- Test 3: RESUMABLE = OFF +ALTER TABLE dbo.MyTable +ADD CONSTRAINT pk_test PRIMARY KEY CLUSTERED (id) +WITH (RESUMABLE = OFF); + +-- Test 4: UNIQUE constraint with RESUMABLE +ALTER TABLE dbo.MyTable +ADD CONSTRAINT uq_test UNIQUE NONCLUSTERED (name) +WITH (RESUMABLE = ON); +``` + +## Step 4: Configure Test Entry + +Add test configuration to the appropriate `OnlySyntaxTests.cs` file as described in testing guidelines. + +### Test Configuration File Location +- Format: `Only{Version}SyntaxTests.cs` +- Example: `Only160SyntaxTests.cs` (for SQL Server 2022) +- Location: `Test/SqlDom/` +- Add to the `Only{Version}TestInfos` array + +### Test Configuration Template + +Use the simplified approach from testing guidelines: + +```csharp +// Option 1: Simplified - only specify error counts you care about +new ParserTest{Version}("YourFeatureTests{Version}.sql"), // All previous versions default to null (ignored), current version expects 0 errors + +// Option 2: Specify only some previous version error counts +new ParserTest{Version}("YourFeatureTests{Version}.sql", nErrors80: 1, nErrors90: 1), // Only SQL 2000/2005 expect errors + +// Option 3: Full specification (legacy compatibility) +new ParserTest{Version}("YourFeatureTests{Version}.sql", + nErrors80: 1, // SQL Server 2000 - expect error for new syntax + nErrors90: 1, // SQL Server 2005 - expect error for new syntax + nErrors100: 1, // SQL Server 2008 - expect error for new syntax + nErrors110: 1, // SQL Server 2012 - expect error for new syntax + nErrors120: 1, // SQL Server 2014 - expect error for new syntax + nErrors130: 1, // SQL Server 2016 - expect error for new syntax + nErrors140: 1, // SQL Server 2017 - expect error for new syntax + nErrors150: 1 // SQL Server 2019 - expect error for new syntax + // nErrors{Version}: 0 is implicit for current version - expect success +), +``` + +### How to Determine Error Counts + +**Rule**: Count the number of SQL statements that will fail in each version. + +**Example**: If your test file has 4 statements with the new feature: +- Versions that DON'T support it: `nErrors = 4` (all 4 statements fail) +- Version that DOES support it: `nErrors = 0` (all 4 statements pass, implicit default) + +### Real-World Example: ALTER TABLE RESUMABLE + +**File**: `Test/SqlDom/Only160SyntaxTests.cs` + +```csharp +new ParserTest160("AlterTableResumableTests160.sql", + nErrors80: 4, // SQL Server 2000: RESUMABLE not supported (4 errors) + nErrors90: 4, // SQL Server 2005: RESUMABLE not supported (4 errors) + nErrors100: 4, // SQL Server 2008: RESUMABLE not supported (4 errors) + nErrors110: 4, // SQL Server 2012: RESUMABLE not supported (4 errors) + nErrors120: 4, // SQL Server 2014: RESUMABLE not supported (4 errors) + nErrors130: 4, // SQL Server 2016: RESUMABLE not supported (4 errors) + nErrors140: 4, // SQL Server 2017: RESUMABLE not supported (4 errors) + nErrors150: 4 // SQL Server 2019: RESUMABLE not supported (4 errors) + // nErrors160: 0 (implicit) - SQL Server 2022: RESUMABLE supported! (0 errors) +), +``` + +## Step 5: Create Baseline File + +Baseline files contain the expected formatted output after parsing and script generation. + +### Baseline File Location +- Format: `Baselines{Version}/YourTestFile{Version}.sql` +- Example: `Baselines160/AlterTableResumableTests160.sql` +- Location: `Test/SqlDom/` +- **Critical**: Baseline filename MUST exactly match the test script filename + +### Baseline Generation Process + +Follow the testing guidelines process: + +#### Initial Creation: +1. **Create empty or placeholder baseline file first** +2. **Run the test** (it will fail) +3. **Copy "Actual" output** from test failure message +4. **Paste into baseline file** with proper formatting + +```bash +# 1. Create placeholder baseline file +New-Item "Test/SqlDom/Baselines160/YourFeatureTests160.sql" -ItemType File + +# 2. Run the test (will fail initially) +dotnet test --filter "YourFeatureTests160" -c Debug + +# 3. Copy the "Actual" output from test failure into baseline file +# Look for the test failure message showing: +# Expected: +# Actual: + +# 4. Re-run the test (should pass now) +dotnet test --filter "YourFeatureTests160" -c Debug +``` + +#### Option B: Manual Creation + +Create the baseline file with properly formatted SQL: + +```sql +-- Baseline follows ScriptDOM formatting rules: +-- - Keywords in UPPERCASE +-- - Proper indentation +-- - Line breaks at appropriate places + +ALTER TABLE dbo.MyTable + ADD CONSTRAINT pk_test PRIMARY KEY CLUSTERED (id) WITH (RESUMABLE = ON, MAX_DURATION = 240 MINUTES); + +ALTER TABLE dbo.MyTable + ADD CONSTRAINT pk_test PRIMARY KEY CLUSTERED (id) WITH (RESUMABLE = ON); + +ALTER TABLE dbo.MyTable + ADD CONSTRAINT pk_test PRIMARY KEY CLUSTERED (id) WITH (RESUMABLE = OFF); + +ALTER TABLE dbo.MyTable + ADD CONSTRAINT uq_test UNIQUE NONCLUSTERED (name) WITH (RESUMABLE = ON); +``` + +### Real-World Example: ALTER TABLE RESUMABLE Baseline + +**File**: `Test/SqlDom/Baselines160/AlterTableResumableTests160.sql` + +```sql +ALTER TABLE dbo.MyTable + ADD CONSTRAINT pk_test PRIMARY KEY CLUSTERED (id) WITH (RESUMABLE = ON, MAX_DURATION = 240 MINUTES); + +ALTER TABLE dbo.MyTable + ADD CONSTRAINT pk_test PRIMARY KEY CLUSTERED (id) WITH (RESUMABLE = ON); + +ALTER TABLE dbo.MyTable + ADD CONSTRAINT pk_test PRIMARY KEY CLUSTERED (id) WITH (RESUMABLE = OFF); + +ALTER TABLE dbo.MyTable + ADD CONSTRAINT uq_test UNIQUE NONCLUSTERED (name) WITH (RESUMABLE = ON); +``` + +## Step 6: Run and Validate Test + +Follow the testing guidelines validation process. + +### Build the Parser +```bash +# Build ScriptDOM library +dotnet build SqlScriptDom/Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug + +# Build test project +dotnet build Test/SqlDom/UTSqlScriptDom.csproj -c Debug +``` + +### Run Your Specific Test +```bash +# Run specific test method +dotnet test Test/SqlDom/UTSqlScriptDom.csproj --filter "FullyQualifiedName~TSql160SyntaxIn160ParserTest" -c Debug + +# Run tests for specific version +dotnet test Test/SqlDom/UTSqlScriptDom.csproj --filter "TestCategory=TSql160" -c Debug + +# Run by test script name filter +dotnet test --filter "YourFeatureTests160" -c Debug + +# Run with verbose output to see details +dotnet test --filter "YourFeatureTests160" -c Debug -v detailed +``` + +### Run Full Test Suite (CRITICAL!) +```bash +# Always run ALL tests before committing +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug + +# Expected output: +# Test summary: total: 1116, failed: 0, succeeded: 1116, skipped: 0 +``` + +### Interpret Results + +Follow the testing guidelines interpretation: + +- βœ… **Success**: Generated output matches baseline, error counts match expectations +- ❌ **Failure**: Review actual vs expected output, adjust baseline or fix grammar +- ⚠️ **Baseline Mismatch**: Copy correct "Actual" output to baseline file +- ⚠️ **Error Count Mismatch**: Adjust error expectations in test configuration + +### Common Test Results + +βœ… **Success**: All tests pass, including your new test +``` +Test summary: total: 1116, failed: 0, succeeded: 1116, skipped: 0 +``` + +❌ **Baseline Mismatch**: Generated output doesn't match baseline +``` +Expected: +Actual: +``` +**Solution**: Copy the "Actual" output to your baseline file (note spacing differences) + +❌ **Error Count Mismatch**: Parse error count differs from expected +``` +TestYourFeature.sql: number of errors after parsing is different from expected. +Expected: 1, Actual: 0 +``` +**Solutions**: +- **If Actual < Expected**: Grammar now supports syntax in older versions β†’ Update error counts +- **If Actual > Expected**: Grammar has issues β†’ Fix grammar or adjust test + +❌ **Parse Errors**: Syntax not recognized +``` +SQL46010: Incorrect syntax near 'YOUR_TOKEN'. at offset 45, line 2, column 15 +``` +**Solutions**: Check grammar rules, verify syntactic predicates, see function guidelines for RETURN statement issues + +## Complete Example Workflow + +### Example: Testing ALTER TABLE RESUMABLE for SQL Server 2022 + +```bash +# Step 0: Test exact script first +echo "ALTER TABLE MyTable ADD CONSTRAINT pk PRIMARY KEY (id) WITH (RESUMABLE = ON);" > temp_test_script.sql +# Add debug test method to Only160SyntaxTests.cs and run to confirm current status + +# Step 1: Determine version +# Research shows: RESUMABLE for ALTER TABLE added in SQL Server 2022 β†’ TSql160 + +# Step 2: Check if already supported +grep -r "RESUMABLE" Test/SqlDom/TestScripts/*.sql +# Result: Found in ALTER INDEX tests, but not ALTER TABLE tests + +# Step 3: Create test script +New-Item "Test/SqlDom/TestScripts/AlterTableResumableTests160.sql" +# Add 4 test cases covering different scenarios + +# Step 4: Add test configuration +# Edit Test/SqlDom/Only160SyntaxTests.cs +# Add: new ParserTest160("AlterTableResumableTests160.sql", nErrors80: 4, ...) + +# Step 5: Create empty baseline +New-Item "Test/SqlDom/Baselines160/AlterTableResumableTests160.sql" + +# Step 6: Build and run test +dotnet build SqlScriptDom/Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug +dotnet test --filter "AlterTableResumableTests160" -c Debug +# Test fails - copy "Actual" output into baseline file + +# Step 7: Re-run test +dotnet test --filter "AlterTableResumableTests160" -c Debug +# Test passes! + +# Step 8: Run full suite +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug +# All 1120 tests pass! + +# Step 9: Commit changes +git add Test/SqlDom/TestScripts/AlterTableResumableTests160.sql +git add Test/SqlDom/Baselines160/AlterTableResumableTests160.sql +git add Test/SqlDom/Only160SyntaxTests.cs +git commit -m "Add tests for ALTER TABLE RESUMABLE option (SQL Server 2022)" +``` + +## Testing Best Practices + +### 1. Comprehensive Coverage +- βœ… **TEST EXACT SCRIPT PROVIDED** (most critical) +- βœ… Test basic syntax variations +- βœ… Test with multiple options +- βœ… Test different statement variations +- βœ… Test with parameters/variables (if applicable) +- βœ… Test edge cases +- βœ… Test error conditions (if relevant) +- βœ… Test complete context (subqueries, table aliases, etc.) + +### 2. Baseline Accuracy +- βœ… Generate baseline from actual parser output +- βœ… Don't hand-edit baseline formatting +- βœ… Verify baseline matches ScriptDOM formatting conventions +- βœ… Check for proper indentation and line breaks + +### 3. Version-Specific Testing +- βœ… Test only in the version where feature was introduced +- βœ… Verify older versions properly reject the syntax +- βœ… Document version dependencies clearly + +### 4. Regression Prevention +- βœ… Always run full test suite before committing +- βœ… Investigate any unexpected test failures +- βœ… Don't assume your change is isolated + +## Common Pitfalls + +### ❌ Wrong Version Number +**Problem**: Testing in TSql150 when feature is TSql160-only +**Solution**: Verify SQL Server version in Microsoft docs + +### ❌ Incorrect Error Counts +**Problem**: `nErrors80: 2` but test has 4 failing statements +**Solution**: Count all statements that use the new feature + +### ❌ Hand-Edited Baselines +**Problem**: Baseline formatting doesn't match ScriptDOM output +**Solution**: Always copy from actual parser output + +### ❌ Skipping Full Test Suite +**Problem**: Your change breaks existing tests +**Solution**: Run `dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug` + +### ❌ Missing Test Cases +**Problem**: Feature works for basic case but fails with parameters +**Solution**: Add comprehensive test coverage + +### ❌ Not Testing Exact Script +**Problem**: Testing simplified/modified versions instead of the exact script provided +**Solution**: Always include the exact T-SQL statement as provided, character-for-character + +## Troubleshooting + +### Test Fails: "Syntax error near..." +**Diagnosis**: Parser doesn't recognize the syntax +**Solution**: Grammar needs to be updated (see [Bug Fixing Guide](../instructions/bug_fixing.guidelines.instructions.md)) + +### Test Fails: "Option 'X' is not valid..." +**Diagnosis**: Validation logic rejects the syntax +**Solution**: See [Validation Fix Guide](../instructions/validation_fix.guidelines.instructions.md) + +### Test Fails: Baseline mismatch +**Diagnosis**: Generated output differs from baseline +**Solution**: Update baseline with actual output or fix generator + +### Full Suite Fails: Other tests break +**Diagnosis**: Your changes affected shared code +**Solution**: Review your changes, create context-specific rules + +## Quick Reference Commands + +```bash +# Step 0: Add debug unit test method first (NO external files) +# Add DebugExactScriptTest method to appropriate test file with exact script embedded + +# Search for syntax in tests +grep -r "KEYWORD" Test/SqlDom/TestScripts/ + +# Search in grammar +grep -r "KEYWORD" SqlScriptDom/Parser/TSql/*.g + +# Build parser +dotnet build SqlScriptDom/Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug + +# Run specific test +dotnet test --filter "TestName" -c Debug + +# Run full suite +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug + +# Create test files (only for comprehensive testing, not initial verification) +New-Item "Test/SqlDom/TestScripts/MyTest160.sql" +New-Item "Test/SqlDom/Baselines160/MyTest160.sql" +``` + +## Related Guides + +- [debugging_workflow.guidelines.instructions.md](../instructions/debugging_workflow.guidelines.instructions.md) - How to diagnose issues +- [Validation_fix.guidelines.instructions.md](../instructions/validation_fix.guidelines.instructions.md) - Fix validation errors +- [Bug Fixing Guide](../instructions/bug_fixing.guidelines.instructions.md) - Add new grammar rules +- [copilot-instructions.md](../copilot-instructions.md) - Main project documentation diff --git a/.vscode/mcp.json b/.vscode/mcp.json new file mode 100644 index 0000000..a49772b --- /dev/null +++ b/.vscode/mcp.json @@ -0,0 +1,25 @@ +{ + "servers": { + "ado": { + "type": "stdio", + "command": "npx", + "args": ["-y", "@azure-devops/mcp", "msdata"], + "env": { + "ADO_DEFAULT_PROJECT": "SQLToolsAndLibraries", + "ADO_DEFAULT_REPO": "ScriptDOM", + "ADO_DEFAULT_BRANCH": "main", + "ADO_DEFAULT_AREA_PATH": "SQLToolsAndLibraries\\DacFx" + } + }, + "my-mcp-mini-drivers": { + "url": "https://mcp.bluebird-ai.net", + "type": "http", + "headers": { + "x-mcp-ec-organization": "msdata", + "x-mcp-ec-project": "SQLToolsAndLibraries", + "x-mcp-ec-repository": "ScriptDOM", + "x-mcp-ec-branch": "main" + } + } + } +} \ No newline at end of file diff --git a/SqlScriptDom/Parser/TSql/TSql160.g b/SqlScriptDom/Parser/TSql/TSql160.g index 62e76bb..4d794ca 100644 --- a/SqlScriptDom/Parser/TSql/TSql160.g +++ b/SqlScriptDom/Parser/TSql/TSql160.g @@ -31476,6 +31476,12 @@ expressionPrimary [ExpressionFlags expressionFlags] returns [PrimaryExpression v | {NextTokenMatches(CodeGenerationSupporter.IIf) && (LA(2) == LeftParenthesis)}? vResult=iIfCall + | + {NextTokenMatches(CodeGenerationSupporter.JsonObject) && (LA(2) == LeftParenthesis)}? + vResult=jsonObjectCall + | + {NextTokenMatches(CodeGenerationSupporter.JsonArray) && (LA(2) == LeftParenthesis)}? + vResult=jsonArrayCall | (Identifier LeftParenthesis)=> vResult=builtInFunctionCall @@ -32496,6 +32502,32 @@ iIfCall returns [IIfCall vResult = this.FragmentFactory.CreateFragment( } ; +jsonObjectCall returns [FunctionCall vResult = this.FragmentFactory.CreateFragment()] +{ + Identifier vIdentifier; +} + : vIdentifier=nonQuotedIdentifier + { + Match(vIdentifier, CodeGenerationSupporter.JsonObject); + vResult.FunctionName = vIdentifier; + } + LeftParenthesis + jsonObjectBuiltInFunctionCall[vResult] + ; + +jsonArrayCall returns [FunctionCall vResult = this.FragmentFactory.CreateFragment()] +{ + Identifier vIdentifier; +} + : vIdentifier=nonQuotedIdentifier + { + Match(vIdentifier, CodeGenerationSupporter.JsonArray); + vResult.FunctionName = vIdentifier; + } + LeftParenthesis + jsonArrayBuiltInFunctionCall[vResult] + ; + coalesceExpression [ExpressionFlags expressionFlags] returns [CoalesceExpression vResult = this.FragmentFactory.CreateFragment()] { ScalarExpression vExpression; diff --git a/SqlScriptDom/Parser/TSql/TSql170.g b/SqlScriptDom/Parser/TSql/TSql170.g index 1850132..27f3f61 100644 --- a/SqlScriptDom/Parser/TSql/TSql170.g +++ b/SqlScriptDom/Parser/TSql/TSql170.g @@ -32113,6 +32113,12 @@ expressionPrimary [ExpressionFlags expressionFlags] returns [PrimaryExpression v | {NextTokenMatches(CodeGenerationSupporter.AIGenerateEmbeddings) && LA(2) == LeftParenthesis}? vResult = aiGenerateEmbeddingsFunctionCall + | + {NextTokenMatches(CodeGenerationSupporter.JsonObject) && (LA(2) == LeftParenthesis)}? + vResult=jsonObjectCall + | + {NextTokenMatches(CodeGenerationSupporter.JsonArray) && (LA(2) == LeftParenthesis)}? + vResult=jsonArrayCall | (Identifier LeftParenthesis)=> vResult=builtInFunctionCall @@ -32719,6 +32725,7 @@ jsonKeyValueExpression returns [JsonKeyValue vResult = FragmentFactory.CreateFra } : ( + ((Identifier Dot)? Label)=> (vMultiPartIdentifier=multiPartIdentifier[2] Dot)? label:Label { var identifier = this.FragmentFactory.CreateFragment(); @@ -33499,6 +33506,32 @@ iIfCall returns [IIfCall vResult = this.FragmentFactory.CreateFragment( } ; +jsonObjectCall returns [FunctionCall vResult = this.FragmentFactory.CreateFragment()] +{ + Identifier vIdentifier; +} + : vIdentifier=nonQuotedIdentifier + { + Match(vIdentifier, CodeGenerationSupporter.JsonObject); + vResult.FunctionName = vIdentifier; + } + LeftParenthesis + jsonObjectBuiltInFunctionCall[vResult] + ; + +jsonArrayCall returns [FunctionCall vResult = this.FragmentFactory.CreateFragment()] +{ + Identifier vIdentifier; +} + : vIdentifier=nonQuotedIdentifier + { + Match(vIdentifier, CodeGenerationSupporter.JsonArray); + vResult.FunctionName = vIdentifier; + } + LeftParenthesis + jsonArrayBuiltInFunctionCall[vResult] + ; + coalesceExpression [ExpressionFlags expressionFlags] returns [CoalesceExpression vResult = this.FragmentFactory.CreateFragment()] { ScalarExpression vExpression; diff --git a/SqlScriptDom/Parser/TSql/TSql80ParserBaseInternal.cs b/SqlScriptDom/Parser/TSql/TSql80ParserBaseInternal.cs index 7c59643..87201d0 100644 --- a/SqlScriptDom/Parser/TSql/TSql80ParserBaseInternal.cs +++ b/SqlScriptDom/Parser/TSql/TSql80ParserBaseInternal.cs @@ -1480,8 +1480,8 @@ protected static void VerifyAllowedIndexOption(IndexAffectingStatement statement if (option.OptionKind == IndexOptionKind.DropExisting || option.OptionKind == IndexOptionKind.LobCompaction || option.OptionKind == IndexOptionKind.Order || - option.OptionKind == IndexOptionKind.Resumable || - option.OptionKind == IndexOptionKind.MaxDuration) + ((versionFlags & SqlVersionFlags.TSql160AndAbove) == 0 && option.OptionKind == IndexOptionKind.Resumable) || + ((versionFlags & SqlVersionFlags.TSql120AndAbove) == 0 && option.OptionKind == IndexOptionKind.MaxDuration)) { invalidOption = true; } diff --git a/SqlScriptDom/Parser/TSql/TSqlFabricDW.g b/SqlScriptDom/Parser/TSql/TSqlFabricDW.g index 9e1fea8..8eb4d05 100644 --- a/SqlScriptDom/Parser/TSql/TSqlFabricDW.g +++ b/SqlScriptDom/Parser/TSql/TSqlFabricDW.g @@ -31623,6 +31623,12 @@ expressionPrimary [ExpressionFlags expressionFlags] returns [PrimaryExpression v | {NextTokenMatches(CodeGenerationSupporter.IIf) && (LA(2) == LeftParenthesis)}? vResult=iIfCall + | + {NextTokenMatches(CodeGenerationSupporter.JsonObject) && (LA(2) == LeftParenthesis)}? + vResult=jsonObjectCall + | + {NextTokenMatches(CodeGenerationSupporter.JsonArray) && (LA(2) == LeftParenthesis)}? + vResult=jsonArrayCall | (Identifier LeftParenthesis)=> vResult=builtInFunctionCall @@ -32627,6 +32633,32 @@ iIfCall returns [IIfCall vResult = this.FragmentFactory.CreateFragment( } ; +jsonObjectCall returns [FunctionCall vResult = this.FragmentFactory.CreateFragment()] +{ + Identifier vIdentifier; +} + : vIdentifier=nonQuotedIdentifier + { + Match(vIdentifier, CodeGenerationSupporter.JsonObject); + vResult.FunctionName = vIdentifier; + } + LeftParenthesis + jsonObjectBuiltInFunctionCall[vResult] + ; + +jsonArrayCall returns [FunctionCall vResult = this.FragmentFactory.CreateFragment()] +{ + Identifier vIdentifier; +} + : vIdentifier=nonQuotedIdentifier + { + Match(vIdentifier, CodeGenerationSupporter.JsonArray); + vResult.FunctionName = vIdentifier; + } + LeftParenthesis + jsonArrayBuiltInFunctionCall[vResult] + ; + coalesceExpression [ExpressionFlags expressionFlags] returns [CoalesceExpression vResult = this.FragmentFactory.CreateFragment()] { ScalarExpression vExpression; diff --git a/Test/SqlDom/Baselines160/AlterFunctionJsonObjectTests160.sql b/Test/SqlDom/Baselines160/AlterFunctionJsonObjectTests160.sql new file mode 100644 index 0000000..017467f --- /dev/null +++ b/Test/SqlDom/Baselines160/AlterFunctionJsonObjectTests160.sql @@ -0,0 +1,9 @@ +ALTER FUNCTION FnName +( ) +RETURNS NVARCHAR (MAX) +AS +BEGIN + RETURN (JSON_OBJECT('Authorization':'Bearer ' + (SELECT Value1 + FROM dbo.table1 + WHERE field1 = 'Token'))); +END diff --git a/Test/SqlDom/Baselines160/AlterTableResumableTests160.sql b/Test/SqlDom/Baselines160/AlterTableResumableTests160.sql new file mode 100644 index 0000000..e3c0a89 --- /dev/null +++ b/Test/SqlDom/Baselines160/AlterTableResumableTests160.sql @@ -0,0 +1,11 @@ +ALTER TABLE table1 + ADD CONSTRAINT PK_Constrain PRIMARY KEY CLUSTERED (a) WITH (ONLINE = ON, MAXDOP = 2, RESUMABLE = ON, MAX_DURATION = 240); + +ALTER TABLE table1 + ADD CONSTRAINT PK_Constrain PRIMARY KEY CLUSTERED (a) WITH (RESUMABLE = ON); + +ALTER TABLE table1 + ADD CONSTRAINT PK_Constrain PRIMARY KEY CLUSTERED (a) WITH (RESUMABLE = OFF); + +ALTER TABLE table1 + ADD CONSTRAINT UQ_Constrain UNIQUE (b) WITH (RESUMABLE = ON, MAX_DURATION = 120 MINUTES); diff --git a/Test/SqlDom/Baselines160/ComplexJsonObjectFunctionTests160.sql b/Test/SqlDom/Baselines160/ComplexJsonObjectFunctionTests160.sql new file mode 100644 index 0000000..3a5d9f4 --- /dev/null +++ b/Test/SqlDom/Baselines160/ComplexJsonObjectFunctionTests160.sql @@ -0,0 +1,13 @@ +CREATE FUNCTION [dbo].[MyFunction] +( ) +RETURNS NVARCHAR (MAX) +AS +BEGIN + RETURN (SELECT t1.Id, + JSON_OBJECT('Column1':t1.Column1, 'Column2':(SELECT t2.* + FROM table2 AS t2 + WHERE t1.Id = t2.Table2Id + FOR JSON PATH)) AS jsonObject + FROM table1 AS t1 + FOR JSON PATH, INCLUDE_NULL_VALUES); +END \ No newline at end of file diff --git a/Test/SqlDom/Baselines160/TestJsonArrayReturn160.sql b/Test/SqlDom/Baselines160/TestJsonArrayReturn160.sql new file mode 100644 index 0000000..8d1552c --- /dev/null +++ b/Test/SqlDom/Baselines160/TestJsonArrayReturn160.sql @@ -0,0 +1,7 @@ +ALTER FUNCTION TestJsonArray +( ) +RETURNS NVARCHAR (MAX) +AS +BEGIN + RETURN (JSON_ARRAY('value1', 'value2')); +END diff --git a/Test/SqlDom/Baselines160/TestTrimReturn160.sql b/Test/SqlDom/Baselines160/TestTrimReturn160.sql new file mode 100644 index 0000000..a366d72 --- /dev/null +++ b/Test/SqlDom/Baselines160/TestTrimReturn160.sql @@ -0,0 +1,7 @@ +ALTER FUNCTION TestTrim +( ) +RETURNS NVARCHAR (MAX) +AS +BEGIN + RETURN (TRIM(' test ')); +END diff --git a/Test/SqlDom/Baselines170/ComplexJsonObjectFunctionTests170.sql b/Test/SqlDom/Baselines170/ComplexJsonObjectFunctionTests170.sql new file mode 100644 index 0000000..3a5d9f4 --- /dev/null +++ b/Test/SqlDom/Baselines170/ComplexJsonObjectFunctionTests170.sql @@ -0,0 +1,13 @@ +CREATE FUNCTION [dbo].[MyFunction] +( ) +RETURNS NVARCHAR (MAX) +AS +BEGIN + RETURN (SELECT t1.Id, + JSON_OBJECT('Column1':t1.Column1, 'Column2':(SELECT t2.* + FROM table2 AS t2 + WHERE t1.Id = t2.Table2Id + FOR JSON PATH)) AS jsonObject + FROM table1 AS t1 + FOR JSON PATH, INCLUDE_NULL_VALUES); +END \ No newline at end of file diff --git a/Test/SqlDom/Baselines170/JsonFunctionTests170.sql b/Test/SqlDom/Baselines170/JsonFunctionTests170.sql index 75f07eb..75b820b 100644 --- a/Test/SqlDom/Baselines170/JsonFunctionTests170.sql +++ b/Test/SqlDom/Baselines170/JsonFunctionTests170.sql @@ -185,3 +185,6 @@ WHERE JSON_CONTAINS(json_col, 'abc%', '$.a', 1) = 1; SELECT JSON_MODIFY(json_col, '$.a', 30) FROM tab1; + +SELECT JSON_OBJECTAGG(t.c1:t.c2) +FROM (VALUES ('key1', 'c'), ('key2', 'b'), ('key3', 'a')) AS t(c1, c2); diff --git a/Test/SqlDom/Only160SyntaxTests.cs b/Test/SqlDom/Only160SyntaxTests.cs index 11cf51d..43cba90 100644 --- a/Test/SqlDom/Only160SyntaxTests.cs +++ b/Test/SqlDom/Only160SyntaxTests.cs @@ -16,6 +16,7 @@ public partial class SqlDomTests private static readonly ParserTest[] Only160TestInfos = { new ParserTest160("AlterTableStatementTests160.sql", nErrors80: 0, nErrors90: 0, nErrors100: 0, nErrors110: 0, nErrors120: 0, nErrors130: 0, nErrors140: 0, nErrors150: 0), + new ParserTest160("AlterTableResumableTests160.sql", nErrors80: 4, nErrors90: 4, nErrors100: 4, nErrors110: 4, nErrors120: 4, nErrors130: 4, nErrors140: 4, nErrors150: 4), new ParserTest160("ExpressionTests160.sql", nErrors80: 1, nErrors90: 0, nErrors100: 0, nErrors110: 0, nErrors120: 0, nErrors130: 0, nErrors140: 0, nErrors150: 0), new ParserTest160("CreateUserFromExternalProvider160.sql", nErrors80: 2, nErrors90: 1, nErrors100: 1, nErrors110: 1, nErrors120: 1, nErrors130: 1, nErrors140: 1, nErrors150: 1), new ParserTest160("CreateExternalTableStatementTests160.sql", nErrors80: 2, nErrors90: 2, nErrors100: 2, nErrors110: 2, nErrors120: 2, nErrors130: 2, nErrors140: 2, nErrors150: 2), @@ -37,6 +38,10 @@ public partial class SqlDomTests new ParserTest160("BuiltInFunctionTests160.sql", nErrors80: 1, nErrors90: 1, nErrors100: 2, nErrors110: 0, nErrors120: 0, nErrors130: 0, nErrors140: 0, nErrors150: 0), new ParserTest160("TrimFunctionTests160.sql", nErrors80: 7, nErrors90: 7, nErrors100: 7, nErrors110: 7, nErrors120: 7, nErrors130: 7, nErrors140: 4, nErrors150: 4), new ParserTest160("JsonFunctionTests160.sql", nErrors80: 9, nErrors90: 8, nErrors100: 14, nErrors110: 14, nErrors120: 14, nErrors130: 14, nErrors140: 14, nErrors150: 14), + new ParserTest160("AlterFunctionJsonObjectTests160.sql", nErrors80: 1, nErrors90: 1, nErrors100: 1, nErrors110: 1, nErrors120: 1, nErrors130: 1, nErrors140: 1, nErrors150: 1), + new ParserTest160("ComplexJsonObjectFunctionTests160.sql"), + new ParserTest160("TestTrimReturn160.sql", nErrors80: 1, nErrors90: 0, nErrors100: 0, nErrors110: 0, nErrors120: 0, nErrors130: 0, nErrors140: 0, nErrors150: 0), + new ParserTest160("TestJsonArrayReturn160.sql", nErrors80: 1, nErrors90: 0, nErrors100: 0, nErrors110: 0, nErrors120: 0, nErrors130: 0, nErrors140: 0, nErrors150: 0), new ParserTest160(scriptFilename: "IgnoreRespectNullsSyntaxTests160.sql", nErrors80: 12, nErrors90: 8, nErrors100: 8, nErrors110: 8, nErrors120: 8, nErrors130: 8, nErrors140: 8, nErrors150: 8), new ParserTest160(scriptFilename: "FuzzyStringMatchingTests160.sql", nErrors80: 0, nErrors90: 0, nErrors100: 0, nErrors110: 0, nErrors120: 0, nErrors130: 0, nErrors140: 0, nErrors150: 0), // OPENROWSET BULK statements specific to SQL Serverless Pools diff --git a/Test/SqlDom/Only170SyntaxTests.cs b/Test/SqlDom/Only170SyntaxTests.cs index 79e81f0..3132f3c 100644 --- a/Test/SqlDom/Only170SyntaxTests.cs +++ b/Test/SqlDom/Only170SyntaxTests.cs @@ -19,8 +19,9 @@ public partial class SqlDomTests new ParserTest170("CreateColumnStoreIndexTests170.sql", nErrors80: 3, nErrors90: 3, nErrors100: 3, nErrors110: 3, nErrors120: 3, nErrors130: 0, nErrors140: 0, nErrors150: 0, nErrors160: 0), new ParserTest170("RegexpTests170.sql", nErrors80: 0, nErrors90: 0, nErrors100: 0, nErrors110: 0, nErrors120: 0, nErrors130: 0, nErrors140: 0, nErrors150: 0, nErrors160: 0), new ParserTest170("AiGenerateChunksTests170.sql", nErrors80: 19, nErrors90: 16, nErrors100: 15, nErrors110: 15, nErrors120: 15, nErrors130: 15, nErrors140: 15, nErrors150: 15, nErrors160: 15), - new ParserTest170("JsonFunctionTests170.sql", nErrors80: 28, nErrors90: 8, nErrors100: 53, nErrors110: 53, nErrors120: 53, nErrors130: 53, nErrors140: 53, nErrors150: 53, nErrors160: 53), + new ParserTest170("JsonFunctionTests170.sql", nErrors80: 29, nErrors90: 8, nErrors100: 54, nErrors110: 54, nErrors120: 54, nErrors130: 54, nErrors140: 54, nErrors150: 54, nErrors160: 54), new ParserTest170("JsonArrayAggOrderBy170.sql", nErrors80: 6, nErrors90: 6, nErrors100: 6, nErrors110: 6, nErrors120: 6, nErrors130: 6, nErrors140: 6, nErrors150: 6, nErrors160: 6), + new ParserTest170("ComplexJsonObjectFunctionTests170.sql"), new ParserTest170("AiGenerateEmbeddingsTests170.sql", nErrors80: 14, nErrors90: 11, nErrors100: 11, nErrors110: 11, nErrors120: 11, nErrors130: 11, nErrors140: 11, nErrors150: 11, nErrors160: 11), new ParserTest170("CreateExternalModelStatementTests170.sql", nErrors80: 2, nErrors90: 2, nErrors100: 2, nErrors110: 2, nErrors120: 2, nErrors130: 4, nErrors140: 4, nErrors150: 4, nErrors160: 4), new ParserTest170("AlterExternalModelStatementTests170.sql", nErrors80: 2, nErrors90: 2, nErrors100: 2, nErrors110: 2, nErrors120: 2, nErrors130: 5, nErrors140: 5, nErrors150: 5, nErrors160: 5), diff --git a/Test/SqlDom/ParserTest.cs b/Test/SqlDom/ParserTest.cs index 49f9fb7..a58c170 100644 --- a/Test/SqlDom/ParserTest.cs +++ b/Test/SqlDom/ParserTest.cs @@ -137,48 +137,6 @@ public ParserTest( result110, result110) { } - public ParserTest( - string scriptFilename, - string baseline80, string baseline90, string baseline100, - string baseline110, string baseline120) - : this( - scriptFilename, - new ParserTestOutput(baseline80), new ParserTestOutput(baseline90), new ParserTestOutput(baseline100), - new ParserTestOutput(baseline110), new ParserTestOutput(baseline120)) - { } - - public ParserTest( - string scriptFilename, - string baseline80, string baseline90, string baseline100, - string baseline110, string baseline120, string baseline130) - : this( - scriptFilename, - new ParserTestOutput(baseline80), new ParserTestOutput(baseline90), new ParserTestOutput(baseline100), - new ParserTestOutput(baseline110), new ParserTestOutput(baseline120), new ParserTestOutput(baseline130)) - { } - - public ParserTest( - string scriptFilename, - string baseline80, string baseline90, string baseline100, - string baseline110, string baseline120, - string baseline130, string baseline140) - : this( - scriptFilename, - new ParserTestOutput(baseline80), new ParserTestOutput(baseline90), new ParserTestOutput(baseline100), - new ParserTestOutput(baseline110), new ParserTestOutput(baseline120), new ParserTestOutput(baseline130), - new ParserTestOutput(baseline140)) - { } - - public ParserTest( - string scriptFilename, - string baseline80, string baseline90, string baseline100, - string baseline110) - : this( - scriptFilename, - new ParserTestOutput(baseline80), new ParserTestOutput(baseline90), new ParserTestOutput(baseline100), - new ParserTestOutput(baseline110), new ParserTestOutput(baseline110)) - { } - public ParserTest( string scriptFilename, string baseline80, string baseline90, string baseline100) @@ -290,7 +248,7 @@ public ParserTest90(string scriptFilename, ParserTestOutput output80, ParserTest internal class ParserTest90And100 : ParserTest { - public ParserTest90And100(string scriptFilename, int nErrors80) + public ParserTest90And100(string scriptFilename, int? nErrors80 = null) : base(scriptFilename, new ParserTestOutput(nErrors80), new ParserTestOutput("Baselines90"), new ParserTestOutput("Baselines90"), new ParserTestOutput("Baselines90")) { } @@ -301,7 +259,7 @@ public ParserTest90And100(string scriptFilename, params ParserErrorInfo[] errors internal class ParserTest100 : ParserTest { - public ParserTest100(string scriptFilename, int nErrors80, int nErrors90) + public ParserTest100(string scriptFilename, int? nErrors80 = null, int? nErrors90 = null) : base(scriptFilename, new ParserTestOutput(nErrors80), new ParserTestOutput(nErrors90), new ParserTestOutput("Baselines100"), new ParserTestOutput("Baselines100")) { } @@ -316,7 +274,7 @@ public ParserTest100(string scriptFilename, params ParserErrorInfo[] errors80And internal class ParserTest110 : ParserTest { - public ParserTest110(string scriptFilename, int nErrors80, int nErrors90, int nErrors100) + public ParserTest110(string scriptFilename, int? nErrors80 = null, int? nErrors90 = null, int? nErrors100 = null) : base(scriptFilename, new ParserTestOutput(nErrors80), new ParserTestOutput(nErrors90), new ParserTestOutput(nErrors100), new ParserTestOutput("Baselines110")) { } @@ -331,7 +289,7 @@ public ParserTest110(string scriptFilename, params ParserErrorInfo[] errors80And internal class ParserTest120 : ParserTest { - public ParserTest120(string scriptFilename, int nErrors80, int nErrors90, int nErrors100, int nErrors110) + public ParserTest120(string scriptFilename, int? nErrors80 = null, int? nErrors90 = null, int? nErrors100 = null, int? nErrors110 = null) : base( scriptFilename, new ParserTestOutput(nErrors80), new ParserTestOutput(nErrors90), new ParserTestOutput(nErrors100), @@ -355,7 +313,7 @@ public ParserTest120(string scriptFilename, params ParserErrorInfo[] errors80And internal class ParserTest130 : ParserTest { - public ParserTest130(string scriptFilename, int nErrors80, int nErrors90, int nErrors100, int nErrors110, int nErrors120) + public ParserTest130(string scriptFilename, int? nErrors80 = null, int? nErrors90 = null, int? nErrors100 = null, int? nErrors110 = null, int? nErrors120 = null) : base( scriptFilename, new ParserTestOutput(nErrors80), new ParserTestOutput(nErrors90), new ParserTestOutput(nErrors100), @@ -383,7 +341,7 @@ public ParserTest130(string scriptFilename, params ParserErrorInfo[] errors80And internal class ParserTest140 : ParserTest { - public ParserTest140(string scriptFilename, int nErrors80, int nErrors90, int nErrors100, int nErrors110, int nErrors120, int nErrors130) + public ParserTest140(string scriptFilename, int? nErrors80 = null, int? nErrors90 = null, int? nErrors100 = null, int? nErrors110 = null, int? nErrors120 = null, int? nErrors130 = null) : base( scriptFilename, new ParserTestOutput(nErrors80), new ParserTestOutput(nErrors90), new ParserTestOutput(nErrors100), @@ -414,7 +372,7 @@ public ParserTest140(string scriptFilename, params ParserErrorInfo[] errors80And internal class ParserTest150 : ParserTest { - public ParserTest150(string scriptFilename, int nErrors80, int nErrors90, int nErrors100, int nErrors110, int nErrors120, int nErrors130, int nErrors140) + public ParserTest150(string scriptFilename, int? nErrors80 = null, int? nErrors90 = null, int? nErrors100 = null, int? nErrors110 = null, int? nErrors120 = null, int? nErrors130 = null, int? nErrors140 = null) : base( scriptFilename, new ParserTestOutput(nErrors80), new ParserTestOutput(nErrors90), new ParserTestOutput(nErrors100), @@ -447,7 +405,7 @@ public ParserTest150(string scriptFilename, params ParserErrorInfo[] errors80And internal class ParserTest160 : ParserTest { - public ParserTest160(string scriptFilename, int nErrors80, int nErrors90, int nErrors100, int nErrors110, int nErrors120, int nErrors130, int nErrors140, int nErrors150) + public ParserTest160(string scriptFilename, int? nErrors80= null, int? nErrors90 = null, int? nErrors100 = null, int? nErrors110 = null, int? nErrors120 = null, int? nErrors130 = null, int? nErrors140 = null, int? nErrors150 = null) : base( scriptFilename, new ParserTestOutput(nErrors80), new ParserTestOutput(nErrors90), new ParserTestOutput(nErrors100), @@ -482,7 +440,7 @@ public ParserTest160(string scriptFilename, params ParserErrorInfo[] errors80And internal class ParserTest170 : ParserTest { - public ParserTest170(string scriptFilename, int nErrors80, int nErrors90, int nErrors100, int nErrors110, int nErrors120, int nErrors130, int nErrors140, int nErrors150, int nErrors160) + public ParserTest170(string scriptFilename, int? nErrors80 = null, int? nErrors90 = null, int? nErrors100 = null, int? nErrors110 = null, int? nErrors120 = null, int? nErrors130 = null, int? nErrors140 = null, int? nErrors150 = null, int? nErrors160 = null) : base( scriptFilename, new ParserTestOutput(nErrors80), new ParserTestOutput(nErrors90), new ParserTestOutput(nErrors100), diff --git a/Test/SqlDom/ParserTestOutput.cs b/Test/SqlDom/ParserTestOutput.cs index 8e9a113..03edd43 100644 --- a/Test/SqlDom/ParserTestOutput.cs +++ b/Test/SqlDom/ParserTestOutput.cs @@ -17,10 +17,10 @@ namespace SqlStudio.Tests.UTSqlScriptDom internal class ParserTestOutput { private readonly string _baselineFolder; - private readonly int _numberOfErrors; + private readonly int? _numberOfErrors; readonly ParserErrorInfo[] _errorInfos; - private ParserTestOutput(string baselineFolder, int numberOfErrors, ParserErrorInfo[] errorInfos) + private ParserTestOutput(string baselineFolder, int? numberOfErrors, ParserErrorInfo[] errorInfos) { _baselineFolder = baselineFolder; _numberOfErrors = numberOfErrors; @@ -31,7 +31,7 @@ public ParserTestOutput(string baselineFolder) : this(baselineFolder, 0, null) { } - public ParserTestOutput(int numberOfErrors) + public ParserTestOutput(int? numberOfErrors) : this(null, numberOfErrors, null) { } @@ -41,11 +41,14 @@ public ParserTestOutput(params ParserErrorInfo[] errorInfos) public void VerifyResult(string testScriptName, string prettyPrinted, IList errors) { - // Errors case - verify number/exact error texts - if (_numberOfErrors != errors.Count) - ParserTestUtils.LogErrors(errors); + if (_numberOfErrors.HasValue) + { + // Errors case - verify number/exact error texts + if (_numberOfErrors.Value != errors.Count) + ParserTestUtils.LogErrors(errors); - Assert.AreEqual(_numberOfErrors, errors.Count, testScriptName + ": number of errors after parsing is different from expected."); + Assert.AreEqual(_numberOfErrors, errors.Count, testScriptName + ": number of errors after parsing is different from expected."); + } if (_errorInfos != null) { for (int i = 0; i < _errorInfos.Length; ++i) @@ -57,7 +60,7 @@ public void VerifyResult(string testScriptName, string prettyPrinted, IList. Actual: <{2}>.", - testScriptName, baseline, prettyPrinted - ) + $"Number of lines of baseline \"{testScriptName}\" and generated script does not match!. Expected: <{baseline}>. Actual: <{prettyPrinted}>. Update baseline {baseline} with the actual Content: {prettyPrinted}" ); - for(int lineCounter = 0; lineCounter ( baselineLines[lineCounter], - prettyPrintedLines[lineCounter], - string.Format - ( - "Different lines encountered. Pretty printed ASTs don't match the baseline \"{0}\". Different line number: {1}", - testScriptName, - lineCounter + 1 - ) + prettyPrintedLines[lineCounter], + $"Different lines encountered. Pretty printed ASTs don't match the baseline \"{testScriptName}\". Different line number: {lineCounter + 1}. Update baseline {baseline} with the actual Content: {prettyPrinted}" ); } } diff --git a/Test/SqlDom/TestScripts/AlterFunctionJsonObjectTests160.sql b/Test/SqlDom/TestScripts/AlterFunctionJsonObjectTests160.sql new file mode 100644 index 0000000..71895cd --- /dev/null +++ b/Test/SqlDom/TestScripts/AlterFunctionJsonObjectTests160.sql @@ -0,0 +1,11 @@ +-- Test ALTER FUNCTION with JSON_OBJECT containing subquery +ALTER FUNCTION FnName + () +RETURNS NVARCHAR(MAX) +AS + BEGIN + RETURN (JSON_OBJECT('Authorization' : 'Bearer ' + (SELECT Value1 + FROM dbo.table1 + WHERE field1 = 'Token'))); + END; +GO diff --git a/Test/SqlDom/TestScripts/AlterTableResumableTests160.sql b/Test/SqlDom/TestScripts/AlterTableResumableTests160.sql new file mode 100644 index 0000000..7b43038 --- /dev/null +++ b/Test/SqlDom/TestScripts/AlterTableResumableTests160.sql @@ -0,0 +1,5 @@ +-- ALTER TABLE ADD CONSTRAINT with RESUMABLE option (SQL Server 2022+) +ALTER TABLE table1 ADD CONSTRAINT PK_Constrain PRIMARY KEY CLUSTERED (a) WITH (ONLINE = ON, MAXDOP = 2, RESUMABLE = ON, MAX_DURATION = 240); +ALTER TABLE table1 ADD CONSTRAINT PK_Constrain PRIMARY KEY CLUSTERED (a) WITH (RESUMABLE = ON); +ALTER TABLE table1 ADD CONSTRAINT PK_Constrain PRIMARY KEY CLUSTERED (a) WITH (RESUMABLE = OFF); +ALTER TABLE table1 ADD CONSTRAINT UQ_Constrain UNIQUE (b) WITH (RESUMABLE = ON, MAX_DURATION = 120 MINUTES); diff --git a/Test/SqlDom/TestScripts/ComplexJsonObjectFunctionTests160.sql b/Test/SqlDom/TestScripts/ComplexJsonObjectFunctionTests160.sql new file mode 100644 index 0000000..60a8dc5 --- /dev/null +++ b/Test/SqlDom/TestScripts/ComplexJsonObjectFunctionTests160.sql @@ -0,0 +1,25 @@ +-- Test CREATE FUNCTION with complex JSON_OBJECT containing FOR JSON PATH subqueries +CREATE FUNCTION [dbo].[MyFunction]() +RETURNS NVARCHAR(MAX) +AS +BEGIN + RETURN + ( + SELECT + t1.Id, + JSON_OBJECT( + 'Column1': t1.Column1, + 'Column2': + ( + SELECT + t2.* + FROM table2 t2 + WHERE t1.Id = t2.Table2Id + FOR JSON PATH + ) + ) AS jsonObject + FROM table1 t1 + FOR JSON PATH, INCLUDE_NULL_VALUES + ) +END; +GO \ No newline at end of file diff --git a/Test/SqlDom/TestScripts/ComplexJsonObjectFunctionTests170.sql b/Test/SqlDom/TestScripts/ComplexJsonObjectFunctionTests170.sql new file mode 100644 index 0000000..60a8dc5 --- /dev/null +++ b/Test/SqlDom/TestScripts/ComplexJsonObjectFunctionTests170.sql @@ -0,0 +1,25 @@ +-- Test CREATE FUNCTION with complex JSON_OBJECT containing FOR JSON PATH subqueries +CREATE FUNCTION [dbo].[MyFunction]() +RETURNS NVARCHAR(MAX) +AS +BEGIN + RETURN + ( + SELECT + t1.Id, + JSON_OBJECT( + 'Column1': t1.Column1, + 'Column2': + ( + SELECT + t2.* + FROM table2 t2 + WHERE t1.Id = t2.Table2Id + FOR JSON PATH + ) + ) AS jsonObject + FROM table1 t1 + FOR JSON PATH, INCLUDE_NULL_VALUES + ) +END; +GO \ No newline at end of file diff --git a/Test/SqlDom/TestScripts/JsonFunctionTests170.sql b/Test/SqlDom/TestScripts/JsonFunctionTests170.sql index 39b2901..7a33a97 100644 --- a/Test/SqlDom/TestScripts/JsonFunctionTests170.sql +++ b/Test/SqlDom/TestScripts/JsonFunctionTests170.sql @@ -179,3 +179,9 @@ WHERE JSON_CONTAINS(json_col, 'abc%', '$.a', 1) = 1; -- Json_Modify SELECT JSON_MODIFY(json_col, '$.a', 30) FROM tab1; + +-- JSON_OBJECTAGG with qualified column names (from GitHub issue #175) +SELECT JSON_OBJECTAGG( t.c1 : t.c2 ) +FROM ( + VALUES('key1', 'c'), ('key2', 'b'), ('key3','a') +) AS t(c1, c2); diff --git a/Test/SqlDom/TestScripts/SimpleJsonObjectReturn160.sql b/Test/SqlDom/TestScripts/SimpleJsonObjectReturn160.sql new file mode 100644 index 0000000..40c6f1a --- /dev/null +++ b/Test/SqlDom/TestScripts/SimpleJsonObjectReturn160.sql @@ -0,0 +1,8 @@ +-- Simpler test without subquery +ALTER FUNCTION FnName() +RETURNS NVARCHAR(MAX) +AS +BEGIN + RETURN JSON_OBJECT('key':'value'); +END; +GO diff --git a/Test/SqlDom/TestScripts/TestJsonArrayReturn160.sql b/Test/SqlDom/TestScripts/TestJsonArrayReturn160.sql new file mode 100644 index 0000000..00fa0f0 --- /dev/null +++ b/Test/SqlDom/TestScripts/TestJsonArrayReturn160.sql @@ -0,0 +1,8 @@ +-- Test ALTER FUNCTION with JSON_ARRAY +ALTER FUNCTION TestJsonArray() +RETURNS NVARCHAR(MAX) +AS +BEGIN + RETURN (JSON_ARRAY('value1', 'value2')); +END; +GO diff --git a/Test/SqlDom/TestScripts/TestTrimReturn160.sql b/Test/SqlDom/TestScripts/TestTrimReturn160.sql new file mode 100644 index 0000000..d39e67e --- /dev/null +++ b/Test/SqlDom/TestScripts/TestTrimReturn160.sql @@ -0,0 +1,8 @@ +-- Test ALTER FUNCTION with TRIM (which also uses semantic predicate) +ALTER FUNCTION TestTrim() +RETURNS NVARCHAR(MAX) +AS +BEGIN + RETURN (TRIM(' test ')); +END; +GO diff --git a/Test/SqlDom/TestUtilities.cs b/Test/SqlDom/TestUtilities.cs index abc8d76..9d3ef18 100644 --- a/Test/SqlDom/TestUtilities.cs +++ b/Test/SqlDom/TestUtilities.cs @@ -9,6 +9,7 @@ using System.Diagnostics; using System.Globalization; using System.IO; +using System.Linq; using System.Reflection; using System.Text; using Microsoft.SqlServer.TransactSql.ScriptDom; @@ -301,16 +302,25 @@ public static StreamReader GetStreamReaderFromManifestResource(string resourceNa public static string GetStringFromResource(string resourceName) { string result = null; - using (StreamReader sr = new StreamReader(Assembly.GetExecutingAssembly().GetManifestResourceStream(resourceName))) + try { - result = sr.ReadToEnd(); + using (StreamReader sr = new StreamReader(Assembly.GetExecutingAssembly().GetManifestResourceStream(resourceName))) + { + result = sr.ReadToEnd(); + } + } + catch (Exception ex) + { + string resourcePath = resourceName.Replace($"{typeof(ParserTestUtils).Namespace}.", string.Empty); + var resourceFilePath = Path.Combine("Test", "SqlDom", resourcePath); + throw new InvalidOperationException($"Failed to find resource file. errpr: {ex.Message}. Make sure the file exist before running the test: {resourceFilePath}"); } - #if NET +#if NET // Convert line endings from \n to \r\n if (System.Environment.NewLine == "\n") result = result.ReplaceLineEndings("\r\n"); - #endif +#endif return result; } diff --git a/global.json b/global.json index 0f9441a..b2e90b5 100644 --- a/global.json +++ b/global.json @@ -1,6 +1,6 @@ { "sdk": { - "version": "8.0.414", + "version": "8.0.415", "rollForward": "latestMajor" }, "msbuild-sdks": { diff --git a/release-notes/170/170.147.0.md b/release-notes/170/170.147.0.md new file mode 100644 index 0000000..ae44f4b --- /dev/null +++ b/release-notes/170/170.147.0.md @@ -0,0 +1,26 @@ +# Release Notes + +## Microsoft.SqlServer.TransactSql.ScriptDom 170.147.0 +This update brings the following changes over the previous release: + +### Target Platform Support + +* .NET Framework 4.7.2 (Windows x86, Windows x64) +* .NET 8 (Windows x86, Windows x64, Linux, macOS) +* .NET Standard 2.0+ (Windows x86, Windows x64, Linux, macOS) + +### Dependencies +* Updates .NET SDK to latest patch version 8.0.415 + +#### .NET Framework +#### .NET Core + +### New Features + +### Fixed +* Fixes https://github.com/microsoft/SqlScriptDOM/issues/125 +* Fixes https://github.com/microsoft/SqlScriptDOM/issues/181 +* Fixes https://github.com/microsoft/SqlScriptDOM/issues/175 +### Changes + +### Known Issues