0.8.3 - SQLite extensions & fuzz testing

v0.8.3 Release Notes

New Features

RANDOM ROWID Support (libSQL Extension)

• Generate pseudorandom row IDs instead of sequential integers for security/privacy
• Prevents ID enumeration attacks and leaking business metrics
• Usage: create table(:sessions, options: [random_rowid: true])

SQLite Extension Loading

• Load SQLite extensions dynamically via enable_extensions/2 and load_ext/3
• Supports FTS5, JSON1, R-Tree, PCRE, and custom extensions
• Security-first: disabled by default, must be explicitly enabled

Enhanced Statement Introspection

• stmt_parameter_name/3 - Get named parameter names (:name, @name, $name)
• reset_stmt/2 - Explicitly reset statements for efficient reuse
• get_stmt_columns/2 - Get full column metadata (name, origin, declared type)

Remote Encryption Support

• New remote_encryption_key option for Turso encrypted databases
• Works alongside existing local encryption_key for end-to-end encryption

Quality & Testing

• Added Credo, Dialyxir, and Sobelow for comprehensive Elixir code analysis
• Property-based fuzz testing with StreamData (SQL injection, transactions, edge cases)
• Rust fuzz testing infrastructure with cargo-fuzz
• Ported key tests from Ecto.Adapters.SQL for compatibility verification
• Modernised Rust code: std::sync::LazyLock, stricter Clippy lints

Fixes

• SQL injection prevention in Pragma module table name validation
• Dialyzer type error in disconnect/2 spec
• Improved fuzz test stability for savepoints and binary data

Changelog: https://github.com/ocean/ecto_libsql/blob/main/CHANGELOG.md
Full Changelog: 0.8.1...0.8.3

0.8.1 - Constraints bug fix

Fixed

• Constraint Error Handling: Index Name Reconstruction (Issue #34)
• Improved constraint name extraction to reconstruct full index names from SQLite error messages
• Now follows Ecto's naming convention: table_column1_column2_index
• Single-column constraints: "UNIQUE constraint failed: users.email" → "users_email_index" (previously just "email")
• Multi-column constraints: "UNIQUE constraint failed: users.slug, users.parent_slug" → "users_slug_parent_slug_index"
• Backtick handling: Properly strips trailing backticks appended by libSQL to error messages
• Enhanced error messages: Preserves custom index names from enhanced format (index: custom_index_name)
• NOT NULL constraints: Reconstructs index names following same convention
• Enables accurate unique_constraint/3 and check_constraint/3 matching with custom index names in Ecto changesets

Full Changelog: 0.8.0...0.8.1

0.8.0 - Rust refactor

Overview

Major code refactoring and critical thread safety fixes with zero breaking changes.

Key Changes

Code Refactoring

• Modularised Rust codebase: Split 2,302-line monolithic lib.rs into 13 focused modules (connection, query, batch, statement, transaction, savepoint, cursor, replication, metadata, utils, constants, models, decode)
• Reorganised test suite: Refactored 1,194-line tests.rs into structured modules (integration, constants, utils)
• Improved maintainability: Better code navigation and contributor onboarding with clearer separation of concerns
• Zero behaviour changes: All APIs and functionality preserved

Thread Safety & Performance Fixes

• Registry lock management: Fixed all functions to drop registry locks before async operations (prevents deadlocks)
• Scheduler annotations: Added #[rustler::nif(schedule = "DirtyIo")] to blocking NIFs
• Atom naming consistency: Fixed remote_primary → remote atom mismatch
• Runtime optimisation: Use shared global TOKIO_RUNTIME instead of creating per-connection (prevents resource exhaustion)
• Replication performance: Eliminated unnecessary async overhead for synchronous operations

Bug Fixes

• Fixed prepared statement column introspection tests (enabled previously skipped tests)
• Enhanced constraint error message handling with index name support
• Improved remote Turso test stability
• Better error handling for allocation failures
• Proper SQL identifier quoting in PRAGMA queries

---

Note: This release is fully backward compatible. The refactoring is purely organisational with performance and stability improvements under the hood.

• Rust refactoring, fix constraint error message handling by @ocean in #33

Full Changelog: 0.7.5...0.8.0

0.7.5 - query routing bug fix & performance improvements

Release 0.7.5 - Query Routing & Performance Improvements

This release focuses on critical bug fixes for batch operations and significant performance optimisations.

Fixed

Query/Execute Routing for Batch Operations

• Implemented proper query() vs execute() routing in batch operations based on statement type
• execute_batch() now automatically detects SELECT and RETURNING clauses to use the correct LibSQL method
• execute_transactional_batch() applies the same intelligent routing logic for atomic operations
• execute_batch_native() and execute_transactional_batch_native() now properly route SQL batch execution
• Eliminates "Statement does not return data" errors for operations that should return rows
• All operations with RETURNING clauses now correctly use the query() method

Performance: Batch Operation Optimisations

• Eliminated per-statement argument clones in batch operations for better memory efficiency
• Changed batch_stmts.iter() to batch_stmts.into_iter() to consume vectors by value
• Removed unnecessary args.clone() calls in both transactional and non-transactional batches
• Reduces memory allocations during batch execution for improved throughput

Lock Coupling Reduction

• Dropped outer LibSQLConn mutex guard earlier in batch operations to reduce contention
• Extract inner Arc<Mutexlibsql::Connection> before entering async blocks
• Only hold inner connection lock during actual I/O operations
• Applied to all four batch operation variants:
  • execute_batch()
  • execute_transactional_batch()
  • execute_batch_native()
  • execute_transactional_batch_native()
• Reduces contention and deadlock surface area by following the established pattern from query_args()

Test Coverage & Documentation

• Enhanced should_use_query() test coverage for block comment handling
• Added explicit assertion documenting known limitation: RETURNING in block comments detected as false positive (safe behaviour)
• Documented CTE and EXPLAIN detection limitations with clear scope notes
• Added comprehensive future improvement recommendations with priority levels and implementation sketches
• Added performance budget notes for optimisation efforts

Impact

• Correctness: Batch operations with RETURNING clauses now work correctly
• Performance: Reduced memory allocations and lock contention in batch operations
• Reliability: Lower deadlock risk through improved lock coupling patterns
• Maintainability: Better test coverage and documentation for edge cases

Full Changelog: 0.7.0...0.7.5

0.7.0

🚀 EctoLibSql v0.7.0 - Performance & Feature Release

Major performance improvements, new connection management features, and comprehensive SQLite configuration support.

🐛 Bug Fixes

• ON CONFLICT Support (Issue #25) - Full upsert support with composite unique indexes

  • :on_conflict: :nothing with conflict targets
  • :on_conflict: :replace_all for upserts
  • Custom field replacement with {fields, _, targets}

• Binary ID Type System (Issue #23) - Complete resolution

  • Fixed autogenerate(:binary_id) to generate string UUIDs
  • Fixed INSERT without RETURNING clause (was causing CaseClauseError)
  • Fixed BLOB encoding in Rust NIF (now returns Elixir binaries)

• Remote Test Stability - Fixed vector operations test failures from stale data

⚡ Performance Improvements

Prepared Statement Caching - Statements are now cached and reused with automatic binding cleanup, delivering ~10-15x faster query execution for repeated queries. Benchmark: 100 cached executions in ~33ms (~330µs per query).

✨ New Features

Connection Management

• busy_timeout/2 - Configure database lock timeout (default: 5000ms)
• reset/1 - Reset connection state without closing
• interrupt/1 - Cancel long-running queries

PRAGMA Configuration

New EctoLibSql.Pragma module with comprehensive SQLite configuration:

• Foreign keys control (enable_foreign_keys/1, disable_foreign_keys/1)
• Journal mode (set_journal_mode/2 - WAL, DELETE, MEMORY, etc.)
• Cache size (set_cache_size/2)
• Synchronous level (set_synchronous/2 - OFF, NORMAL, FULL, EXTRA)
• Table introspection (table_info/2, table_list/1)
• User version tracking (user_version/1, set_user_version/2)

Transaction Features

• Savepoints - Nested transaction support with partial rollback
  • create_savepoint/2 - Create named savepoint
  • release_savepoint_by_name/2 - Commit savepoint changes
  • rollback_to_savepoint_by_name/2 - Rollback to savepoint
• Transaction Ownership Validation - Prevents cross-connection transaction manipulation

Batch Operations

• execute_batch_sql/2 - Execute multiple SQL statements (non-transactional)
• execute_transactional_batch_sql/2 - Atomic batch execution

Prepared Statement Introspection

• stmt_parameter_count/2 - Get number of parameters
• stmt_column_count/2 - Get number of result columns
• stmt_column_name/3 - Get column name by index

Advanced Replica Sync

• get_frame_number_for_replica/1 - Monitor replication frame
• sync_until_frame/2 - Wait for specific frame (30s timeout)
• flush_and_get_frame/1 - Push pending writes
• max_write_replication_index/1 - Read-your-writes consistency tracking

🔒 Security Fixes

• SQL Injection Prevention - Added strict alphanumeric validation for savepoint names
• Prepared Statement Validation - Enhanced parameter binding checks

📚 Documentation

• Updated AGENTS.md with all new features
• Updated README.md with concise examples

🔄 Backward Compatibility

All changes are backward compatible. Prepared statement API unchanged - caching happens transparently with improved performance.

Full Changelog: https://github.com/ocean/ecto_libsql/blob/main/CHANGELOG.md
Full list of changes: 0.6.0...0.7.0

0.6.0 - Migrations, remote replicas, better libSQL support

What's Changed

• fix: Fix panic in prepared statement exec, add streaming support by @ocean in #19
• fix: Fix Ecto migrations bug for issue #20 and add some more tests by @ocean in #21
• feature: Add some extended libSQL support for ALTER TABLE etc by @ocean in #22

• Remote Sync Performance & Reliability
  • Removed redundant manual .sync() calls after write operations for embedded replicas
  • LibSQL automatically handles sync to remote primary database - manual syncs were causing double-sync overhead
  • Added 30-second timeout to connection establishment to prevent indefinite hangs
  • All Turso remote tests now pass reliably (previously 4 tests timed out)
  • Test suite execution time improved significantly (~107s vs timing out at 60s+)

According to Turso documentation: "Writes are sent to the remote primary database by default, then the local database updates automatically once the remote write succeeds." Manual sync is only needed when explicitly pulling down changes from remote (e.g., after reconnecting to an existing replica).

• Ecto Migrations Compatibility (Issue #20)

  • Fixed DDL function grouping that was causing compilation errors
  • Added comprehensive migration test suite (759 lines) covering all SQLite ALTER TABLE operations
  • Improved handling of SQLite's limited ALTER TABLE support
  • Added tests for column operations, constraint management, and index creation

• Prepared Statement Execution

  • Fixed panic in prepared statement execution that could crash the BEAM VM
  • Added proper error handling for prepared statement operations
  • Improved error messages for prepared statement failures

• Extended LibSQL DDL Support

  • Added support for additional ALTER TABLE operations compatible with LibSQL
  • Improved DDL operation grouping and execution order
  • Better handling of SQLite dialect quirks

Added

• Cursor Streaming Support

  • Implemented cursor-based streaming for large result sets
  • Added handle_declare/4, handle_fetch/4, and handle_deallocate/4 DBConnection callbacks
  • Memory-efficient processing of large queries
  • Rust NIF functions: declare_cursor/3, fetch_cursor/2, cursor registry management

• Comprehensive Test Coverage

  • Added 138 new DDL generation tests in test/ecto_connection_test.exs
  • Added 759 lines of migration tests in test/ecto_migration_test.exs
  • Improved error handling test coverage
  • All 162 tests passing (0 failures)

Changed

• Sync Behaviour for Embedded Replicas

  • Automatic sync after writes has been removed (LibSQL handles this natively)
  • Manual sync() via EctoLibSql.Native.sync/1 still available for explicit control
  • Improved sync timeout handling with configurable DEFAULT_SYNC_TIMEOUT_SECS (30s)
  • Added connection timeout to prevent hangs during initial replica sync

• Documentation Updates

  • Updated all documentation to reflect sync behaviour changes
  • Added clarification about when manual sync is needed vs automatic
  • Improved Turso/LibSQL compatibility documentation references

Technical Details

Sync Performance Before:

• Manual .sync() called after every write operation
• Double sync overhead (LibSQL auto-sync + manual sync)
• 120-second timeout causing long test hangs
• 4 tests timing out after 60+ seconds each

Sync Performance After:

• LibSQL's native auto-sync used correctly
• No redundant manual sync calls
• 30-second connection timeout for fast failure
• All tests passing in ~107 seconds

Migration Notes

This is a non-breaking change for normal usage. However, if you were relying on automatic sync behaviour after writes in embedded replica mode, you may now need to explicitly call EctoLibSql.Native.sync/1 when you need to ensure remote data is pulled down (e.g., after reconnecting to an existing local database).

Recommended Actions:

1. Review code that uses embedded replicas with sync: true
2. Add explicit sync() calls after reconnecting to existing local databases if you need to pull down remote changes
3. Remove any redundant manual sync() calls after write operations

Full Changelog: 0.5.0...0.6.0

0.5.0 - Massively improve Rust error handling

Changed

• Rust NIF Error Handling
  • Eliminated all 146 unwrap() calls from production Rust code
  • Added safe_lock() and safe_lock_arc() helper functions for safe mutex locking
  • All NIF errors now return {:error, message} tuples to Elixir instead of panicking
  • Mutex poisoning errors are handled gracefully with descriptive context
  • Invalid connection/transaction/statement/cursor IDs return proper errors

Fixed

• VM Stability - NIF errors no longer crash the entire BEAM VM
  • Invalid operations (bad connection IDs, missing resources) now return error tuples
  • Processes survive NIF errors, allowing supervision trees to work properly
  • Error messages include descriptive context for easier debugging

Added

• Comprehensive Error Handling Tests
  • Added test/error_demo_test.exs with 7 tests demonstrating graceful error handling
  • Added test/error_handling_test.exs with 14 comprehensive error coverage tests
  • All tests verify that NIF errors return proper error tuples instead of crashing the BEAM VM

Technical Details

Before 0.5.0:

• 146 unwrap() calls in Rust production code
• Mutex/registry errors → panic → entire BEAM VM crash
• Invalid IDs → panic → VM crash
• Supervision trees ineffective for NIF errors

After 0.5.0:

• 0 unwrap() calls in Rust production code (100% eliminated)
• All errors return {:error, "descriptive message"} tuples
• Processes can handle errors and recover
• Supervision trees work as expected

Migration Guide

This is a non-breaking change for normal Ecto usage. Your existing code will continue to work exactly as before, but is now significantly more stable.

What Changed:

• NIF functions that previously panicked now return {:error, reason} tuples
• Your existing error handling code will now catch errors that previously crashed the VM

Recommended Actions:

1. Review error handling in code that uses EctoLibSql.Native functions directly
2. Ensure supervision strategies are in place for database operations
3. Consider adding retry logic for transient errors (connection timeouts, etc.)

Notes

This release represents a major stability improvement for production deployments. The refactoring ensures that ecto_libsql handles errors the "Elixir way" - returning error tuples that can be supervised, rather than panicking at the Rust level and crashing the VM.

0.4.0 - First ecto_libsql release

What's Changed

• feat: Major feature additions - batch ops, prepared statements, transaction behaviours, and metadata by @ocean in #1
• feat: Add comprehensive GitHub Actions CI workflow for PR testing by @ocean in #2
• fix: Expand Elixir version support to 1.17+ (current-1 policy) by @ocean in #4
• Add libsql cursor support, vector search, encryption and more tests by @ocean in #3
• Add full Ecto adapter and tests by @ocean in #5
• Rename library from libsqlex to ecto_libsql to reflect major update in functionality by @ocean in #7
• fix: Fix remote tests, add some notes by @ocean in #8
• Update dependabot.yml to support multiple ecosystems by @ocean in #9
• build(deps): Bump actions/checkout from 4 to 6 by @dependabot[bot] in #10
• build(deps): Bump db_connection from 2.7.0 to 2.8.1 by @dependabot[bot] in #11
• build(deps): Bump rustler from 0.36.1 to 0.37.1 by @dependabot[bot] in #12
• build(deps): Update rustler requirement from 0.36.1 to 0.37.0 in /native/ecto_libsql by @dependabot[bot] in #13
• build(deps-dev): Bump ex_doc from 0.38.2 to 0.39.1 by @dependabot[bot] in #14
• docs: Adjust for Hex publishing by @ocean in #15

New Contributors

• @ocean made their first contribution in #1
• @dependabot[bot] made their first contribution in #10

Full Changelog: https://github.com/ocean/ecto_libsql/commits/0.4.0