diff --git a/src/current/Gemfile b/src/current/Gemfile index de20eb2bc1d..f8d0601daa8 100644 --- a/src/current/Gemfile +++ b/src/current/Gemfile @@ -13,7 +13,9 @@ gem "redcarpet", "~> 3.6" gem "rss" gem "webrick" gem "jekyll-minifier" - +gem "csv" +gem "base64" +gem "bigdecimal" group :jekyll_plugins do gem "jekyll-include-cache" gem 'jekyll-algolia', "~> 1.0", path: "./jekyll-algolia-dev" diff --git a/src/current/_includes/releases/v20.2/v20.2.13.md b/src/current/_includes/releases/v20.2/v20.2.13.md index 1ad131fa057..6f8cbb6b93b 100644 --- a/src/current/_includes/releases/v20.2/v20.2.13.md +++ b/src/current/_includes/releases/v20.2/v20.2.13.md @@ -22,8 +22,8 @@ Release Date: July 12, 2021

Bug fixes

-- Fixed a bug which prevented the [optimizer](https://www.cockroachlabs.com/docs/v21.1/cost-based-optimizer) from producing plans with [partial indexes](https://www.cockroachlabs.com/docs/v21.1/partial-indexes) when executing some prepared statements that contained placeholders, stable functions, or casts. This bug was present since partial indexes were added in v20.2.0. [#66641][#66641] -- Fixed a panic that could occur in the [optimizer](https://www.cockroachlabs.com/docs/v21.1/cost-based-optimizer) when executing a prepared plan with placeholders. This could happen when one of the tables used by the query had [computed columns](https://www.cockroachlabs.com/docs/v21.1/computed-columns) or a [partial index](https://www.cockroachlabs.com/docs/v21.1/partial-indexes). [#66832][#66832] +- Fixed a bug which prevented the optimizer from producing plans with partial indexes when executing some prepared statements that contained placeholders, stable functions, or casts. This bug was present since partial indexes were added in v20.2.0. [#66641][#66641] +- Fixed a panic that could occur in the optimizer when executing a prepared plan with placeholders. This could happen when one of the tables used by the query had computed columns or a partial index. [#66832][#66832] - Fixed a bug that caused graceful drain to call `time.sleep` multiple times, which cut into the time needed for range [lease transfers](https://www.cockroachlabs.com/docs/v20.2/architecture/replication-layer#leases). [#66852][#66852] - CockroachDB now allows a node with [lease](https://www.cockroachlabs.com/docs/v20.2/architecture/replication-layer#leases) preferences to drain gracefully. [#66714][#66714] - CockroachDB now avoids interacting with [decommissioned nodes](https://www.cockroachlabs.com/docs/v20.2/remove-nodes) during [DistSQL](https://www.cockroachlabs.com/docs/v20.2/architecture/sql-layer#distsql) planning and consistency checking. [#66951][#66951] diff --git a/src/current/_includes/releases/v21.1/v21.1.0-alpha.1.md b/src/current/_includes/releases/v21.1/v21.1.0-alpha.1.md deleted file mode 100644 index 7031324f21b..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.0-alpha.1.md +++ /dev/null @@ -1,519 +0,0 @@ -## v21.1.0-alpha.1 - -Release Date: December 8, 2020 - - - -

Backward-incompatible changes

- -- RocksDB can no longer be used as the storage engine. Passing in `--storage-engine=rocksdb` now returns an error. [#55509][#55509] {% comment %}doc{% endcomment %} -- Rows containing empty arrays in [`ARRAY`](https://www.cockroachlabs.com/docs/v21.1/array) columns are now contained in [GIN indexes](https://www.cockroachlabs.com/docs/v21.1/inverted-indexes). This change is backward-incompatible because prior versions of CockroachDB will not be able to recognize and decode keys for empty arrays. Note that rows containing `NULL` values in an indexed column will still not be included in GIN indexes. [#55970][#55970] {% comment %}doc{% endcomment %} -- Concatenation between a non-null argument and a null argument is now typed as string concatenation, whereas it was previously typed as array concatenation. This means that the result of `NULL || 1` will now be `NULL` instead of `{1}`. To preserve the old behavior, the null argument can be casted to an explicit type. [#55611][#55611] {% comment %}doc{% endcomment %} - -

General changes

- -- Added increased logging and metrics around slow disk operations. [#54215][#54215] {% comment %}doc{% endcomment %} -- CockroachDB now detects stalled disk operations better and crashes the process if a disk operation is taking longer than a minute. Added cluster settings to allow for tuning of this behavior. [#55186][#55186] {% comment %}doc{% endcomment %} -- Added some metrics surrounding schema changes. [#54855][#54855] {% comment %}doc{% endcomment %} -- Upgraded CockroachDB's version of Go to v1.15.4. [#56363][#56363] {% comment %}doc{% endcomment %} -- The timezone data is now built in to the CockroachDB binary, which is the fallback source of time if `tzdata` is not located by the default Go standard library. [#56634][#56634] {% comment %}doc{% endcomment %} -- Renamed instances of "Admin UI" to "DB Console" in the documentation of OIDC cluster settings. [#56869][#56869] -- Included `tar` in docker images. This allows users to use `kubectl cp` on 20.2.x containers. [#57241][#57241] - -

Enterprise edition changes

- -- It is no longer allowed to widen an incremental-backup chain with the inclusion of new complete empty DBs. [#54329][#54329] {% comment %}doc{% endcomment %} -- Added cluster settings to enable/ disable the `BACKUP` and `RESTORE` commands. Attempts to use these features while they are disabled returns an error indicating that the database administrator has disabled the feature. Example usage: `SET CLUSTER SETTING feature.backup.enabled = FALSE; SET CLUSTER SETTING feature.backup.enabled = TRUE; SET CLUSTER SETTING feature.restore.enabled = FALSE; SET CLUSTER SETTING feature.restore.enabled = TRUE;`. [#56533][#56533] {% comment %}doc{% endcomment %} -- Added cluster settings to enable/ disable the `IMPORT`, `EXPORT`, and changefeed commands. Attempts to use these features while they are disabled returns an error indicating that the database administrator has disabled the feature. Example usage: `SET CLUSTER SETTING feature.import.enabled = FALSE; SET CLUSTER SETTING feature.import.enabled = TRUE; SET CLUSTER SETTING feature.export.enabled = FALSE; SET CLUSTER SETTING feature.export.enabled = TRUE; SET CLUSTER SETTING feature.changefeed.enabled = FALSE; SET CLUSTER SETTING feature.changefeed.enabled = TRUE;`. [#56872][#56872] {% comment %}doc{% endcomment %} - -

SQL language changes

- -- Interleaved joins have been removed; merge joins are now planned in all cases when interleaved joins would have been planned previously. [#54163][#54163] {% comment %}doc{% endcomment %} -- It is now possible to create partial GIN indexes. The optimizer will choose to scan partial GIN indexes when the partial index predicate is implied and scanning the GIN index has the lowest estimated cost. [#54376][#54376] {% comment %}doc{% endcomment %} -- `EXPLAIN ANALYZE` diagrams now contain "bytes sent" information on streams. [#54518][#54518] {% comment %}doc{% endcomment %} -- Implemented the geometry built-in functions `ST_Rotate({geometry, float8, geometry})`. [#54610][#54610] -- Implemented the geometry built-in function `ST_ClosestPoint()`. [#54843][#54843] -- Implement the string built-in function `unaccent()`. [#54628][#54628] -- When enterprise features are not enabled, the `follower_read_timestamp()` function now returns `(statement_time - 4.8s)` instead of an error. [#54951][#54951] {% comment %}doc{% endcomment %} -- Added a new virtual table `crdb_internal.invalid_descriptors`, which runs validations on descriptors in the database context and reports any errors. [#54017][#54017] -- Implemented the built-in operator `add jsonb_exists_any(jsonb, text[])`. [#55172][#55172] -- Added the ability to optionally specify the `PRIVILEGES` keyword when issuing the `GRANT ALL` or `REVOKE ALL` statements, for Postgres compatibility. Previously, a statement like the following would fail with a syntax error: `GRANT ALL PRIVILEGES ON DATABASE a TO user1;`. [#55304][#55304] {% comment %}doc{% endcomment %} -- Implemented the built-in function `pg_column_size()`, which counts the amount of bytes stored by column. [#55312][#55312] -- Implemented the geometry built-in functions `ST_Rotate({geometry, float8, float8, float8})`. [#55428][#55428] -- Implemented the built-in function `ST_MemSize()`, which returns the memory space a geometry takes. [#55416][#55416] -- `SHOW ENUMS` now returns an array aggregation of enum values instead of having them separated by the `|` character. [#55386][#55386] -- Implemented the geometry built-in function `ST_PointInsideCircle()`. [#55464][#55464] -- Implement the geometry built-in function `ST_LineFromEncodedPolyline()`. [#55429][#55429] -- Implement the geometry built-in function `ST_AsEncodedPolyline()`. [#55515][#55515] -- Implement the geometry built-in function `ST_LineLocatePoint()`, which computes the fraction of the line segment that represents the location of the given point to the closest point on the original line. [#55470][#55470] -- Implemented the `REASSIGN OWNED BY ... TO ...` statement, which changes ownership of all database objects in the current database, owned by any roles in the first argument, to the new role in the second argument. [#54594][#54594] -- Implemented the geometry built-in function `ST_MinimumBoundingRadius()`, which returns a record containing the center point and radius of the smallest circle that can fully contain a geometry. [#55532][#55532] -- The vectorized engine now supports the JSONFetchValPath (`#>`) operator. [#55570][#55570] -- Added the ability to cast a string containing all integers into a given regtype, e.g., `'1234'::regproc`. [#55607][#55607] -- Potentially hazardous `DROP COLUMN` operations when `sql_safe_updates` is enabled now return a note and link to https://github.com/cockroachdb/cockroach/issues/46541. [#55248][#55248] {% comment %}doc{% endcomment %} -- Changed the `SPLIT AT` output to be consistent with the output from `SHOW RANGES`. [#55543][#55543] {% comment %}doc{% endcomment %} -- Implemented the geometry built-in function `ST_MinimumBoundingCircle()`, which returns polygon shape that approximates minimum bounding circle to contain geometry. [#55567][#55567] -- Added the constraint name to constraint errors, for increased wire-level Postgres compatibility. [#55660][#55660] {% comment %}doc{% endcomment %} -- Added support for using the syntax `... UNIQUE WITHOUT INDEX ...` in `CREATE TABLE` and `ALTER TABLE` statements, both when defining columns and unique constraints. Although this syntax can now be parsed successfully, using this syntax currently returns an error: `unique constraints without an index are not yet supported`. [#55700][#55700] {% comment %}doc{% endcomment %} -- Add the built-in functions `sha224()` and `sha384()`. [#55720][#55720] -- Implemented `SHOW REGIONS`, which returns all of regions available in a cluster. [#55831][#55831] {% comment %}doc{% endcomment %} -- Implemented the geography built-in function `ST_UnaryUnion()` [#55894][#55894] -- Implemented `ALTER TABLE ... SET LOCALITY/REGIONAL AFFINITY` statements, which configure multi-region properties of given tables. These are subject to change. [#55827][#55827] {% comment %}doc{% endcomment %} -- All expressions in `EXPLAIN` output that operate on indexes now show the table name the index is declared on, rather than just an alias. If the query aliases the table, the alias is also shown. For example, a scan on table `foo` that is aliased as `f` was previously displayed as `scan f`. It is now displayed as `scan foo [as=f]`. [#55641][#55641] {% comment %}doc{% endcomment %} -- Changed the underlying type for the version cluster setting. Previously, it was of an internal type representing "state machine", but now it's simply "version". This has no operational implications, but the `Type` column in `cockroach gen settings-list` now shows "version" instead of "custom validation". [#55994][#55994],[#56546][#56546] {% comment %}doc{% endcomment %} -- Removed the `201auto` value for the `vectorize` session variable and the corresponding cluster setting. [#55907][#55907] {% comment %}doc{% endcomment %} -- Expanded the `CREATE SCHEMA`, `DROP SCHEMA`, `ALTER SCHEMA`, `GRANT ... ON SCHEMA`, `REVOKE ... ON SCHEMA`, and `SHOW GRANTS ON SCHEMA` statements to allow schema names prefixed with database names. [#55647][#55647] {% comment %}doc{% endcomment %} -- Added support for dollar-quoted strings with digit. [#55958][#55958] {% comment %}doc{% endcomment %} -- Added a new single-column output format for `EXPLAIN` and `EXPLAIN (VERBOSE)`. [#55866][#55866] {% comment %}doc{% endcomment %} -- Creating multi-column GIN indexes is now allowed by setting the `experimental_enable_mutlti_column_inverted_indexes` session setting to `true`. At this time, these indexes are not fully supported and their behavior is undefined. Using this feature will likely result in errors. Do not enable this setting in a production database. [#55993][#55993] {% comment %}doc{% endcomment %} -- Constraints that have not been validated are now marked "NOT VALID" in the output of `SHOW CREATE` and `SHOW CONSTRAINTS`. [#53485][#53485] {% comment %}doc{% endcomment %} -- The `NOT VALID` option can now be provided for `CHECK` and `FOREIGN KEY` constraints listed as table constraints in `CREATE TABLE` statements. This option has no affect on the constraint created. It will not skip validation. [#53485][#53485] {% comment %}doc{% endcomment %} -- Added a pgcode (`42704`, `undefined_object`) to the error returned when attempting to drop an index by a table and index name that doesn't exist. [#55417][#55417] -- Added the `WITH row_limit="{$num}"` option for importing CSVs to allow users to do a quick test run on an import of `$num` rows. Example: `IMPORT TABLE test ... CSV DATA ... WITH row_limit="3";` [#56080][#56080] {% comment %}doc{% endcomment %} -- Added the `WITH row_limit="{$num}"` option for importing `DELIMITED/AVRO` data to allow users to do a quick test run on an import of $num rows. Example: `IMPORT TABLE test ... DELIMITED/AVRO DATA ... WITH row_limit="3";` [#56135][#56135] {% comment %}doc{% endcomment %} -- Added `WITH row_limit="{$num}"` option for importing bundle formats to allow users to do a quick test run on an import of $num rows. Example: `IMPORT ... WITH row_limit="3";`. [#56587][#56587] {% comment %}doc{% endcomment %} -- `EXPLAIN ANALYZE` diagrams now contain "network latency" information on streams. [#55705][#55705] {% comment %}doc{% endcomment %} -- Implemented the `covar_pop()` and `covar_samp()` aggregation functions. [#55707][#55707] -- Prevented column type modification of columns that are depended on by views. [#56213][#56213] -- Implemented the geometry built-in functions `ST_TransScale({geometry,float8,float8,float8,float8})` [#56198][#56198] -- Implemented the geometry built-in function `ST_Node()`. [#56183][#56183] -- The concatenation operator `||` can now be used between strings and any other non-array types. [#55611][#55611] -- CockroachDB now returns a float instead of a decimal when at least one argument of an aggregate function is decimal. [#56296][#56296] -- Implemented the `regr_intercept()`, `regr_r2()`, and `regr_slope()` aggregation functions. [#56296][#56296] -- `EXPLAIN ANALYZE` diagrams now show "deserialization time" on streams instead of "io time". [#56144][#56144] -- Added a pgcode (`42804`, `DatatypeMismatch`) when adding a default value of the wrong type to a column. [#56455][#56455] -- Attempting to rename an undefined index now returns a `pgcode.UndefinedObject` (`42704`) error instead of an uncategorized error. [#56455][#56455] -- Implemented the `regr_sxx()`, `regr_sxy()`, and `regr_syy()` aggregation functions. [#56585][#56585] -- `SHOW REGIONS` has changed the column name for availability zones to "zones" from "availability_zones". [#56344][#56344] {% comment %}doc{% endcomment %} -- Introduced a `pg_collation` of "default". Strings now return the "default" collation OID in the `pg_attribute` table (this was previously `en_US`). The "default" collation is also visible on the `pg_collation` virtual table. [#56598][#56598] -- A table can now successfully be dropped in a transaction following other schema changes to the table in the same transaction. [#56589][#56589] {% comment %}doc{% endcomment %} -- Added a new variant of explain: `EXPLAIN ANALYZE (PLAN)`. [#56524][#56524] {% comment %}doc{% endcomment %} -- `SHOW REGIONS` functionality is now deferred to `SHOW REGIONS FROM CLUSTER`. [#56627][#56627] {% comment %}doc{% endcomment %} -- It is now possible to hint to the optimizer that it should plan an inverted join by using the syntax `... INNER INVERTED JOIN ...` or `... LEFT INVERTED JOIN ...`. If the hint is provided and it is possible to plan an inverted join, the optimizer will now plan an inverted join, even if it estimates that a different plan would have a lower cost. If the hint is provided but it is not possible to plan an inverted join because there is no GIN index on the right side table or the join condition is not a valid inverted join condition, the database will return an error. [#55679][#55679] {% comment %}doc{% endcomment %} -- Added the empty `pg_catalog.pg_opclass` table to improve compatibility with Postgres. [#56653][#56653] -- `SURVIVE AVAILABILITY ZONE FAILURE` is now `SURVIVE ZONE FAILURE`. [#56837][#56837] {% comment %}doc{% endcomment %} -- Added admin-only, `crdb_internal` functions to enable descriptor repair in dire circumstances. [#55699][#55699] -- Added support for an optional `=` character for `SURVIVE`, e.g., `ALTER DATABASE d SURVIVE = ZONE FAILURE`. [#56881][#56881] {% comment %}doc{% endcomment %} -- Introduced stubs for `ALTER DATABASE ... PRIMARY REGION` and `CREATE TABLE ... PRIMARY REGION`. [#56883][#56883] {% comment %}doc{% endcomment %} -- Dropping the primary index using `DROP INDEX` now returns a `FeatureNotSupported` error along with hints showing supported ways to drop primary indexes. [#56858][#56858] -- Renaming an index to a name that is already being used for another index will now return a `pgcode.DuplicateRelation` (`42P07`) error instead of an uncategorized error. [#56681][#56681] -- The `relpersistence` column in `pg_catalog.pg_class` now correctly displays `t` as the persistence status for temporary tables. [#56827][#56827] -- Added a deprecation notice to statements containing the `INTERLEAVE IN PARENT` clause. [#56874][#56874] -- `SHOW DATABASES` and `crdb_internal.databases` now display all regions as well as survival goals for a given database. [#56880][#56880] {% comment %}doc{% endcomment %} -- Adds a feature flag via cluster settings for the `CREATE STATISTICS` and `ANALYZE` feature. If a user attempts to use the command while disabled, an error indicating that the database administrator had disabled the feature is surfaced. Example usage: `SET CLUSTER SETTING feature.stats.enabled = FALSE; SET CLUSTER SETITNG feature.stats.enabled = TRUE;`. [#57076][#57076] {% comment %}doc{% endcomment %} -- `CREATE DATABASE ... PRIMARY REGION` is now stored on the database descriptor. [#57038][#57038] -- `SHOW DATABASES` and `crdb_internal.databases` now display the `PRIMARY REGION` set on the database descriptor as the `primary_region` column. [#57038][#57038] {% comment %}doc{% endcomment %} -- Added a feature flag via cluster settings for all schema change-related features. If a user attempts to use these features while they are disabled, an error indicating that the database administrator has disabled the feature is surfaced. Example usage: `SET CLUSTER SETTING feature.schema_change.enabled = FALSE; SET CLUSTER SETTING feature.schema_change.enabled = TRUE;`. [#57040][#57040] {% comment %}doc{% endcomment %} -- Changed `pg_constraint` column types for `confkey` and `conkey` to `smallint[]` to improve compatibility with Postgres. [#56975][#56975] -- The `ALTER TABLE...SPLIT/UNSPLIT` and `ALTER INDEX...SPLIT/UNSPLIT` commands are now gated by a schema change feature flag. If a user attempts to use these features while they are disabled, an error indicating that the system administrator has disabled the feature is surfaced. Example usage: `SET CLUSTER SETTING feature.schema_change.enabled = FALSE SET CLUSTER SETTING feature.schema_change.enabled = TRUE;`. [#57142][#57142] {% comment %}doc{% endcomment %} -- When creating a database with the regions clause specified, CockroachDB now creates a `regions` enum type automatically. [#56628][#56628] {% comment %}doc{% endcomment %} -- Implemented `SHOW REGION FROM DATABASE` and `SHOW REGION FROM DATABASE db`, which shows all regions for the given database, as well as whether that region is the primary region. [#57106][#57106] {% comment %}doc{% endcomment %} -- `CREATE TABLE AS SELECT ... FROM ... AS OF SYSTEM TIME x` is now supported. [#55916][#55916] -- Implemented the function `regr_count()`. [#56822][#56822] -- Added the `character_sets` table to the `information_schema`. [#56953][#56953] {% comment %}doc{% endcomment %} -- `SHOW ENUMS` is now extended to take an optional `FROM` clause. The user can specify either the schema name or both the database name and schema name separated by `.`. If a hierarchy is specified, the statement returns enums falling in that hierarchy rather than all of the enums in the current database. [#57197][#57197] {% comment %}doc{% endcomment %} -- The multi-region enum, created implicitly for all multi-region databases, can be introspected using the `pg_catalog.pg_enum` table. It is also displayed in `SHOW ENUMS`. [#57197][#57197] {% comment %}doc{% endcomment %} -- Implemented the geography built-in function `ST_Subdivide()`. [#56898][#56898] -- A `pgcode.UndefinedColumn` error is now returned when adding a unique constraint to one or more undefined columns. [#57316][#57316] -- The database name is now displayed in `SHOW REGIONS FROM DATABASE`. [#57278][#57278] {% comment %}doc{% endcomment %} -- Added `SHOW SURVIVAL GOAL FROM DATABASE [database]`, which shows the survival goal for a multi-region database. [#57278][#57278] {% comment %}doc{% endcomment %} -- Added the `uuid_generate_v4()` built-in function. It works exactly like `gen_random_uuid()` but was added for compatibility with Postgres versions older than PG13. [#57212][#57212] - -

Command-line changes

- -- A `debug.zip` file now includes a script, `hot-ranges.sh`, which will summarize the hottest ranges in the cluster. [#53547][#53547] {% comment %}doc{% endcomment %} -- `cockroach sql` and `cockroach demo` now support the command-line parameter `--file` (shorthand `-f`) to read commands from a named file. The behavior is the same as if the file was redirected on the standard input; in particular, the processing stops at the first error encountered (which is different from interactive usage with a prompt). Note that it is not yet possible to combine `-f` with `-e`. [#54741][#54741] {% comment %}doc{% endcomment %} -- The large banner message "Replication has been disabled for this cluster ..." that was unconditionally emitted on the standard error stream for `cockroach start-single-node` has now become a simple log message at severity `INFO`. [#54749][#54749] -- `cockroach demo` now pre-creates a `demo` user account with a random password to discourage the user of `root`. The `demo` account is currently granted the `admin` role. [#54749][#54749] {% comment %}doc{% endcomment %} -- The CLI help text for `--max-disk-temp-storage` now properly reports the default value. [#54853][#54853] -- The help text displayed by `\?` in `cockroach sql` and `cockroach demo` now groups the recognized client-side commands into sections for easier reading. [#54796][#54796] -- The client-side command `\show` for the SQL shell is deprecated in favor of the new command `\p`. This prints the contents of the query buffer entered so far. [#54796][#54796] {% comment %}doc{% endcomment %} -- The new client-side command `\r` for the SQL shell erases the contents of the query buffer entered so far. This provides a convenient way to reset the input, for example, when the user gets themselves confused with string delimiters. [#54796][#54796] {% comment %}doc{% endcomment %} -- The SQL shell (`cockroach sql`, `cockroach demo`) now supports the client-side command `\echo`, like `psql`. This can be used, for example, to generate informational output when executing SQL scripts non-interactively. [#54796][#54796] {% comment %}doc{% endcomment %} -- The SQL shell (`cockroach sql`, `cockroach demo`) now support the `\i` and `\ir` client-side command which reads SQL file and evaluates its content in-place. `\ir` differs from `\i` in that the file name is resolved relative to the location of the script containing the `\ir` command. This makes `\ir` likely more desirable in the general case. Instances of `\q` inside a file included via `\i`/`\ir` stop evaluation of the file and resume evaluation of the file that included it. This feature is compatible with the identically named `psql` commands. It is meant to help compose complex initialization scripts from a library of standard components. For example, one could be defining each table and its initial contents in separate SQL files, and then use different super-files to include different tables depending on the desired final schema. [#54796][#54796] {% comment %}doc{% endcomment %} -- Removed the `debug sstables` command, superseded by the `debug pebble lsm` command. [#54890][#54890] -- Added the `cockroach debug pebble db checkpoint` debug command to easily create a checkpoint without using rocksdb. [#55751][#55751] {% comment %}doc{% endcomment %} -- Updated the `--storage-engine` help text to reflect RocksDB deletion. [#55509][#55509] -- Added support for `\connect DATABASE` and `\c DATABASE`. [#55934][#55934] {% comment %}doc{% endcomment %} -- Added an import CLI command that allows users to upload and import local dump files into a running cockroach cluster. `PGDUMP` and `MYSQLDUMP` formats are currently supported. [#54896][#54896] {% comment %}doc{% endcomment %} -- `cockroach demo` now allows for nodes to be added using the `\demo` client-side command. This works in both single node and multi-node configurations, for example, when started with `--nodes int` or `--geo-partitioned-replicas`. [#56344][#56344] {% comment %}doc{% endcomment %} -- Some specific situations now have dedicated exit status codes. The following codes are defined: - - Code | Description - -----|------------ - 0 | Process terminated without error. - 1 | An unspecified error was encountered. Explanation should be present in the stderr or logging output. - 2 | Go runtime error, or uncaught panic. Likely a bug in CockroachDB. Explanation may be present in logging output. - 3 | Server process interrupted gracefully with Ctrl+C / SIGINT. - 4 | Command-line flag error. - 5 | A logging operation to the process' stderr stream failed (e.g., stderr has been closed). Some details may be present in the file output, if enabled. - 6 | A logging operation to file has failed (e.g., log disk full, no inodes, permission issue, etc.). Some details may be present in the stderr stream. - 7 | Server detected an internal error and triggered an emergency shutdown. - 8 | Logging failed while processing an emergency shutdown. - [#56574][#56574] -- `cockroach demo` now tries to use the same TCP port numbers for the SQL and HTTP servers on every invocation. This is meant to simplify documentation. These defaults can be overridden with the new (`demo`-specific) command line flags `--sql-port` and/or `--http-port`. [#56737][#56737] {% comment %}doc{% endcomment %} -- The SQL shell now accepts `yes`/`no` as boolean options for slash commands, following `psql` behavior. [#56829][#56829] {% comment %}doc{% endcomment %} -- A `\x [on|off]` command has been added to toggle the `records` display format, following `psql` behavior. [#56829][#56829] {% comment %}doc{% endcomment %} -- CockroachDB now prints a warning if the `--locality` flag does not contain a "region" tier. [#57179][#57179] {% comment %}doc{% endcomment %} - -

API endpoint changes

- -- Added a new prometheus metric called `seconds_until_license_expiry` that reports the number of seconds until the enterprise license on the cluster expires, a negative number if the expiration is in the past, or `0` if there is no license. [#55565][#55565] {% comment %}doc{% endcomment %} - -

DB Console changes

- -- Changed the view for tables without data on main pages. [#54943][#54943] -- Updated the design of the custom date range selector on the Cluster > Maps view and Metrics pages [#54851][#54851] -- The DB Console now shows messages provided by server instead of custom generic messages defined on the client side, for example, messages about permission restrictions to show pages for non-admin roles. [#50869][#50869] -- Added a new metric called `raft.scheduler.latency`, which monitors the latency for operations to be picked up and processed by the Raft scheduler. [#56943][#56943] -- Redesigned inline error alerts when a user has insufficient rights to see some resources. [#50869][#50869] -- Implemented a permission denied error for non-admin users on the Node Map and Events views. [#50869][#50869] -- Tables within user-defined schemas are now included in the Data Distribution page. [#56388][#56388] {% comment %}doc{% endcomment %} -- Creating, dropping, and altering roles or users now causes events to be logged and displayed. [#55945][#55945] -- If statement diagnostics are enabled for a statement, the bytes sent over the network are now shown. [#55969][#55969] -- `ALTER DATABASE OWNER` and `CONVERT TO SCHEMA` now cause events to be logged and displayed. [#55891][#55891] -- Changing schema objects now causes events to be logged and displayed. [#55785][#55785] -- Changing privileges (i.e., with `GRANT` or `REVOKE`) now causes events to be logged and displayed. [#55612][#55612] -- Renaming databases or tables now causes events to be logged and displayed. [#55269][#55269] -- Added descriptions for failed job on the Job Details page. [#54268][#54268] {% comment %}doc{% endcomment %} - -

Bug fixes

- -- Fixed the `rpath` and `so` names of `libgeos.so` and `libgeos_c.so` such that a `dlopen` to `libgeos.so` is not needed. [#55129][#55129] -- Made lease transfers during rebalancing adhere to the rate limit utilized in other lease transfer cases which eliminates unexpected lease oscillations when adding a new node. [#54322][#54322] -- CockroachDB now handles PostgreSQL "cancel" messages on TLS connections in the same way as when they are sent without TLS: the connection is closed, but no action takes place. No error is logged. As a reminder, PostgreSQL "cancel" messages are still unsupported in CockroachDB and client should still use `CANCEL QUERY` instead. [#54618][#54618] -- Cleared table statistics from job description in failed and canceled backup jobs. [#54446][#54446] -- Fixed an error message that referred to `experimental_enable_hash_sharded_indexes` as a cluster setting when it is in fact a session variable. [#54960][#54960] -- Fixed a nil pointer error that could occur at planning time for some spatial queries when a GIN index existed on a geometry or geography column. [#55076][#55076] -- Fixed `SHOW ENUMS` column names to have `values` instead of `string_agg` for column names, and `owner` for the owner itself. [#55139][#55139] -- Fixed `SHOW TYPES` to show the `owner` column instead of the `value` column. [#55139][#55139] -- Fixed a bug where empty enums did not show up for `SHOW ENUMS` or `SHOW TYPES`. [#55143][#55143] -- Fixed a bug where, on failure of `CREATE TABLE AS` or `CREATE MATERIALIZED VIEW`, tables would be left in an invalid non-public state until GC instead of being marked as dropped, possibly causing spurious validation failures. The bug was introduced in earlier 20.2 pre-releases. [#55272][#55272] -- Fixed a rare scenario in which a node would refuse to start after updating the binary. The log message would indicate: `store [...], last used with cockroach version [...], is too old for running version [...] (which requires data from [...] or later)`. [#55240][#55240] -- Changefeeds defined on multiple tables will now only backfill affected tables after a schema change. [#55135][#55135] -- Fixed a bug where adding child tables or types to a schema being restored would cause those new child objects to become corrupted with no parent schema if the restore job had to be rolled back. [#55157][#55157] -- Fixed a bug where the seconds component of a zone offset of a `TIMESTAMPTZ` value was not displayed. [#55071][#55071] -- Fixed a bug where casts to regclass were not escaped, e.g., when the type or table name had `"` characters. [#55607][#55607] -- Fixed a bug where casts from string to regproc, regtype or regprocedure would not work if they contained `"` characters at the beginning or at the end. [#55607][#55607] -- Fixed a bug which could cause `IMPORT`, `BACKUP`, or `RESTORE` to experience an error when they occur concurrently to when the cluster sets its version to upgraded. [#55524][#55524] -- Fixed a rare crash during tracing when reading un-decodable data. [#55783][#55783] -- Prevented a crash, introduced in the 20.2 series, caused by range scans over virtual tables with virtual indexes. [#56459][#56459] -- In some cases CockroachDB, would attempt to transfer ranges to nodes in the process of being decommissioned or being shut down; this could cause disruption the moment the node did actually terminate. This bug has been fixed. It had been introduced some time before v2.0. [#55808][#55808] -- Fixed a bug causing queries sent to a freshly-restarted node to sometimes hang for a long time while the node catches up with replication. [#55148][#55148] -- Fixed typing of collated strings so that collation names are case-insensitive and hyphens/underscores are interchangeable. [#56352][#56352] -- Fixed internal errors related to very large `LIMIT` and/or `OFFSET` values. [#56672][#56672] -- Improved the accuracy of reported CPU usage when running in containers. [#56461][#56461] -- Fixed a bug which can lead to canceled schema change jobs ending in the failed rather than canceled state. [#55513][#55513] -- Prevented an opportunity for livelock in the jobs subsystem due to frequent updates to already finished jobs. [#56855][#56855] -- The `LogFile` reserved API, which was used by `cockroach debug zip`, could corrupt log entries. This has been fixed. [#56901][#56901] -- `DELETE` statements no longer have a chance of returning an incorrect number of deleted rows in transactions that will eventually need to restart due to contention. [#56458][#56458] -- Fixed a race condition in the `tpcc` workload with the `--scatter` flag where tables could be scattered multiple times or not at all. [#56942][#56942] -- Fixed a bug where reg* types were not parsed properly over pgwire, `COPY` or prepared statements. [#56298][#56298] -- Previously, casts and parsing of strings to types could allow an out-of-bounds value to be successfully used (e.g., `SELECT -232321321312::int2`) but fail with an out-of-bounds message when it is inserted into the table. This is now checked when the value is parsed or being casted to. [#55095][#55095] -- `cockroach debug merge-logs --redact=true --redactable-output=false` now properly removes redaction markers. [#57121][#57121] -- Fixed a bug related to validation of un-upgraded pre-19.2 inbound foreign keys. [#57132][#57132] -- Creating a materialized view that references a column with a NULL value no longer results in an error. [#57139][#57139] -- `ST_GeomFromGeoJSON` now sets the SRID to 4326, matching PostGIS 3.0 / RFC7946 behavior. [#57152][#57152] -- Fixed a bug that caused an "ambiguous column reference" error during foreign key cascading updates. This error was incorrectly produced when the child table's reference column name was equal to the concatenation of the parent's reference column name and "_new", and when the child table had a `CHECK` constraint, computed column, or partial index predicate expression that referenced the column. This bug was introduce in version 20.2. [#57153][#57153] -- Fixed a bug that caused errors or corrupted partial indexes of child tables in foreign key relationships with cascading `UPDATE`s and `DELETE`s. The corrupt partial indexes could result in incorrect query results. Any partial indexes on child tables of foreign key relationships with `ON DELETE CASCADE` or `ON UPDATE CASCADE` actions may be corrupt and should be dropped and re-created. This bug was introduce in version 20.2. [#57100][#57100] -- Second timezone offsets for `TIMETZ` now correctly display over the Postgres wire protocol; these were previously omitted. [#57265][#57265] -- `SELECT FOR UPDATE` now requires both `SELECT` and `UPDATE` privileges, instead of just `UPDATE` privileges. [#57309][#57309] -- Fixed a bug where users of an OSS build of CockroachDB would see "Page Not Found" when loading the DB Console. [#56591][#56591] - -

Performance improvements

- -- The optimizer can now deduce that certain variable arguments to functions must be non-null. This improves cardinality estimation for those variables and unlocks other types of optimizations. As a result, the optimizer may choose better query plans when a function is used as a filter predicate. [#54558][#54558] -- Improved the selectivity estimate for array contains predicates (e.g., `arr @> ARRAY[1]`) in the optimizer. This improves the optimizer's cardinality estimation for queries containing these predicates, and may result in better query plans in some cases. [#54768][#54768] -- Updated the cost model in the optimizer to make index joins more expensive and better reflect the reality of their cost. As a result, the optimizer will choose index joins less frequently, generally resulting in more efficient query plans. [#54768][#54768] -- The optimizer simplifies join expressions to only scan a single table when the join filter is a contradiction. A limitation, now removed, was preventing this simplification from occurring in some cases, leading to more efficient query plans in some cases. [#54813][#54813] -- Improved the efficiency of plans for the execution of left outer spatial joins. [#55216][#55216] -- The optimizer now considers partial indexes when exploring zigzag joins. This may lead to more efficient query plans for queries that (1) operate on tables with partial indexes and (2) have a filter that holds two columns, indexed by two indexes, constant. [#55401][#55401] -- The optimizer now attempts to split a query with a disjunctive filter (OR expression) into a UNION of index scans, where one or both of the scans is an unconstrained partial index scan. As a result, more efficient query plans may be generated for queries with disjunctive filters that operate on tables with partial indexes. [#55915][#55915] -- CSV imports should now be slightly faster. [#55845][#55845] -- Previously, all `CHECK` constraints defined on a table would be tested for every `UPDATE` to the table. Now, a check constraint will not be tested for validity when the values of columns it references are not being updated. The referenced columns are no longer fetched in cases where they were only fetched to test `CHECK` constraints. [#56007][#56007] -- Indexes on computed columns can now be utilized when filters reference the computed expression and not the computed column directly. [#55867][#55867] -- The query optimizer can now generate inverted zigzag joins over partial GIN indexes. This may lead to more efficient query plans when filtering by a column that is indexed by a partial GIN index. [#56101][#56101] -- They query optimizer can now plan zigzag joins on two partial indexes with the same predicate, leading to more efficient query plans in some cases. [#56103][#56103] -- The optimizer now converts inner joins with single-row values expressions into projections. This allows decorrelation of subqueries that only reference variables from the outer query, such as `SELECT (SELECT value + 10) FROM table`. [#55961][#55961] -- The optimizer may now plan an inverted join if two tables are joined on `JSONB` or `ARRAY` columns using a contains predicate (e.g., `WHERE a @> b`), and the first column has a GIN index. The inverted join will be chosen if the optimizer expects it to be more efficient than any alternative plan. For queries in which the only alternative is a cartesian product followed by a filter, the inverted join will likely result in a performance improvement. [#55679][#55679] -- The hybrid logical clock used to coordinate distributed operations now performs significantly better under high contention with many concurrent updates from remote nodes. [#56708][#56708] -- The Raft processing goroutine pool's size is now capped at 96. This was observed to prevent instability on large machines (32+ vCPU) in clusters with many ranges (50k+ per node). [#56860][#56860] -- Interactions between Raft heartbeats and the Raft goroutine pool scheduler are now more efficient and avoid excessive mutex contention. This was observed to prevent instability on large machines (32+ vCPU) in clusters with many ranges (50k+ per node). [#56860][#56860] -- The Raft scheduler now prioritizes the node liveness Range. This was observed to prevent instability on large machines (32+ vCPU) in clusters with many ranges (50k+ per node). [#56860][#56860] -- The optimizer now supports using a GIN index on `JSONB` or `ARRAY` columns for a wider variety of filter predicates. Previously, GIN index usage was only supported for simple predicates (e.g., `WHERE a @> '{"foo": "bar"}'`), but now more complicated predicates are supported by combining simple contains (`@>`) expressions with `AND` and `OR` (e.g., `WHERE a @> '{"foo": "bar"}' OR a @> '{"foo": "baz"}'`). A GIN index will be used if it is available on the filtered column and the optimizer expects it to be more efficient than any alternative plan. This may result in performance improvements for queries involving `JSONB` and `ARRAU` columns. [#56732][#56732] - -

Doc updates

- -- Updated several Hello World tutorials to use `cockroach demo` as the backend and an external repository for code samples, including Go with [pgx](https://www.cockroachlabs.com/docs/v21.1/build-a-go-app-with-cockroachdb) and [GORM](https://www.cockroachlabs.com/docs/v21.1/build-a-go-app-with-cockroachdb-gorm), Java with [JDBC](https://www.cockroachlabs.com/docs/v21.1/build-a-java-app-with-cockroachdb) and [Hibernate](https://www.cockroachlabs.com/docs/v21.1/build-a-java-app-with-cockroachdb-hibernate), and Python with [psycopg2](https://www.cockroachlabs.com/docs/v21.1/build-a-python-app-with-cockroachdb), [SQLAlchemy](https://www.cockroachlabs.com/docs/v21.1/build-a-python-app-with-cockroachdb-sqlalchemy), and [Django](https://www.cockroachlabs.com/docs/v21.1/build-a-python-app-with-cockroachdb-django). [#9025](https://github.com/cockroachdb/docs/pull/9025),[#8991](https://github.com/cockroachdb/docs/pull/8991), -- Updated the [Production Checklist](https://www.cockroachlabs.com/docs/v21.1/recommended-production-settings) to recommend disabling Linux memory swapping to avoid over-allocating memory. [#8979](https://github.com/cockroachdb/docs/pull/8979) - -
- -

Contributors

- -This release includes 1040 merged PRs by 88 authors. We would like to thank the following contributors from the CockroachDB community: - -- Adrian Popescu (first-time contributor) -- Alan Acosta (first-time contributor) -- ArjunM98 (first-time contributor) -- Artem Barger -- Azdim Zul Fahmi (first-time contributor) -- David Pacheco (first-time contributor) -- Erik Grinaker -- Gabriel Jaldon (first-time contributor) -- Jake Rote (first-time contributor) -- Joshua M. Clulow (first-time contributor) -- Marcin Knychała (first-time contributor) -- Max Neverov (first-time contributor) -- Miguel Novelo (first-time contributor) -- Ruixin Bao (first-time contributor) -- TAKAHASHI Yuto (first-time contributor) -- Tayo (first-time contributor) -- Tim Graham (first-time contributor) -- Tom Milligan (first-time contributor) -- Vaibhav -- alex-berger@gmx.ch (first-time contributor) -- alex.berger@nexiot.ch (first-time contributor) -- haseth (first-time contributor) -- hewei03 (first-time contributor) -- neeral -- xinyue (first-time contributor) - -
- -[#30624]: https://github.com/cockroachdb/cockroach/pull/30624 -[#50869]: https://github.com/cockroachdb/cockroach/pull/50869 -[#53390]: https://github.com/cockroachdb/cockroach/pull/53390 -[#53485]: https://github.com/cockroachdb/cockroach/pull/53485 -[#53547]: https://github.com/cockroachdb/cockroach/pull/53547 -[#53926]: https://github.com/cockroachdb/cockroach/pull/53926 -[#54017]: https://github.com/cockroachdb/cockroach/pull/54017 -[#54026]: https://github.com/cockroachdb/cockroach/pull/54026 -[#54044]: https://github.com/cockroachdb/cockroach/pull/54044 -[#54064]: https://github.com/cockroachdb/cockroach/pull/54064 -[#54140]: https://github.com/cockroachdb/cockroach/pull/54140 -[#54143]: https://github.com/cockroachdb/cockroach/pull/54143 -[#54163]: https://github.com/cockroachdb/cockroach/pull/54163 -[#54215]: https://github.com/cockroachdb/cockroach/pull/54215 -[#54230]: https://github.com/cockroachdb/cockroach/pull/54230 -[#54268]: https://github.com/cockroachdb/cockroach/pull/54268 -[#54317]: https://github.com/cockroachdb/cockroach/pull/54317 -[#54322]: https://github.com/cockroachdb/cockroach/pull/54322 -[#54329]: https://github.com/cockroachdb/cockroach/pull/54329 -[#54349]: https://github.com/cockroachdb/cockroach/pull/54349 -[#54350]: https://github.com/cockroachdb/cockroach/pull/54350 -[#54352]: https://github.com/cockroachdb/cockroach/pull/54352 -[#54355]: https://github.com/cockroachdb/cockroach/pull/54355 -[#54376]: https://github.com/cockroachdb/cockroach/pull/54376 -[#54446]: https://github.com/cockroachdb/cockroach/pull/54446 -[#54518]: https://github.com/cockroachdb/cockroach/pull/54518 -[#54558]: https://github.com/cockroachdb/cockroach/pull/54558 -[#54594]: https://github.com/cockroachdb/cockroach/pull/54594 -[#54610]: https://github.com/cockroachdb/cockroach/pull/54610 -[#54618]: https://github.com/cockroachdb/cockroach/pull/54618 -[#54628]: https://github.com/cockroachdb/cockroach/pull/54628 -[#54668]: https://github.com/cockroachdb/cockroach/pull/54668 -[#54673]: https://github.com/cockroachdb/cockroach/pull/54673 -[#54741]: https://github.com/cockroachdb/cockroach/pull/54741 -[#54749]: https://github.com/cockroachdb/cockroach/pull/54749 -[#54768]: https://github.com/cockroachdb/cockroach/pull/54768 -[#54796]: https://github.com/cockroachdb/cockroach/pull/54796 -[#54811]: https://github.com/cockroachdb/cockroach/pull/54811 -[#54813]: https://github.com/cockroachdb/cockroach/pull/54813 -[#54843]: https://github.com/cockroachdb/cockroach/pull/54843 -[#54851]: https://github.com/cockroachdb/cockroach/pull/54851 -[#54853]: https://github.com/cockroachdb/cockroach/pull/54853 -[#54855]: https://github.com/cockroachdb/cockroach/pull/54855 -[#54870]: https://github.com/cockroachdb/cockroach/pull/54870 -[#54880]: https://github.com/cockroachdb/cockroach/pull/54880 -[#54890]: https://github.com/cockroachdb/cockroach/pull/54890 -[#54896]: https://github.com/cockroachdb/cockroach/pull/54896 -[#54943]: https://github.com/cockroachdb/cockroach/pull/54943 -[#54951]: https://github.com/cockroachdb/cockroach/pull/54951 -[#54960]: https://github.com/cockroachdb/cockroach/pull/54960 -[#55050]: https://github.com/cockroachdb/cockroach/pull/55050 -[#55063]: https://github.com/cockroachdb/cockroach/pull/55063 -[#55071]: https://github.com/cockroachdb/cockroach/pull/55071 -[#55076]: https://github.com/cockroachdb/cockroach/pull/55076 -[#55095]: https://github.com/cockroachdb/cockroach/pull/55095 -[#55120]: https://github.com/cockroachdb/cockroach/pull/55120 -[#55129]: https://github.com/cockroachdb/cockroach/pull/55129 -[#55135]: https://github.com/cockroachdb/cockroach/pull/55135 -[#55139]: https://github.com/cockroachdb/cockroach/pull/55139 -[#55143]: https://github.com/cockroachdb/cockroach/pull/55143 -[#55147]: https://github.com/cockroachdb/cockroach/pull/55147 -[#55148]: https://github.com/cockroachdb/cockroach/pull/55148 -[#55157]: https://github.com/cockroachdb/cockroach/pull/55157 -[#55172]: https://github.com/cockroachdb/cockroach/pull/55172 -[#55186]: https://github.com/cockroachdb/cockroach/pull/55186 -[#55216]: https://github.com/cockroachdb/cockroach/pull/55216 -[#55222]: https://github.com/cockroachdb/cockroach/pull/55222 -[#55240]: https://github.com/cockroachdb/cockroach/pull/55240 -[#55248]: https://github.com/cockroachdb/cockroach/pull/55248 -[#55263]: https://github.com/cockroachdb/cockroach/pull/55263 -[#55269]: https://github.com/cockroachdb/cockroach/pull/55269 -[#55272]: https://github.com/cockroachdb/cockroach/pull/55272 -[#55300]: https://github.com/cockroachdb/cockroach/pull/55300 -[#55304]: https://github.com/cockroachdb/cockroach/pull/55304 -[#55312]: https://github.com/cockroachdb/cockroach/pull/55312 -[#55373]: https://github.com/cockroachdb/cockroach/pull/55373 -[#55386]: https://github.com/cockroachdb/cockroach/pull/55386 -[#55401]: https://github.com/cockroachdb/cockroach/pull/55401 -[#55416]: https://github.com/cockroachdb/cockroach/pull/55416 -[#55417]: https://github.com/cockroachdb/cockroach/pull/55417 -[#55428]: https://github.com/cockroachdb/cockroach/pull/55428 -[#55429]: https://github.com/cockroachdb/cockroach/pull/55429 -[#55459]: https://github.com/cockroachdb/cockroach/pull/55459 -[#55464]: https://github.com/cockroachdb/cockroach/pull/55464 -[#55470]: https://github.com/cockroachdb/cockroach/pull/55470 -[#55509]: https://github.com/cockroachdb/cockroach/pull/55509 -[#55513]: https://github.com/cockroachdb/cockroach/pull/55513 -[#55515]: https://github.com/cockroachdb/cockroach/pull/55515 -[#55517]: https://github.com/cockroachdb/cockroach/pull/55517 -[#55524]: https://github.com/cockroachdb/cockroach/pull/55524 -[#55532]: https://github.com/cockroachdb/cockroach/pull/55532 -[#55543]: https://github.com/cockroachdb/cockroach/pull/55543 -[#55565]: https://github.com/cockroachdb/cockroach/pull/55565 -[#55567]: https://github.com/cockroachdb/cockroach/pull/55567 -[#55570]: https://github.com/cockroachdb/cockroach/pull/55570 -[#55607]: https://github.com/cockroachdb/cockroach/pull/55607 -[#55611]: https://github.com/cockroachdb/cockroach/pull/55611 -[#55612]: https://github.com/cockroachdb/cockroach/pull/55612 -[#55626]: https://github.com/cockroachdb/cockroach/pull/55626 -[#55638]: https://github.com/cockroachdb/cockroach/pull/55638 -[#55641]: https://github.com/cockroachdb/cockroach/pull/55641 -[#55644]: https://github.com/cockroachdb/cockroach/pull/55644 -[#55647]: https://github.com/cockroachdb/cockroach/pull/55647 -[#55660]: https://github.com/cockroachdb/cockroach/pull/55660 -[#55679]: https://github.com/cockroachdb/cockroach/pull/55679 -[#55699]: https://github.com/cockroachdb/cockroach/pull/55699 -[#55700]: https://github.com/cockroachdb/cockroach/pull/55700 -[#55705]: https://github.com/cockroachdb/cockroach/pull/55705 -[#55707]: https://github.com/cockroachdb/cockroach/pull/55707 -[#55720]: https://github.com/cockroachdb/cockroach/pull/55720 -[#55721]: https://github.com/cockroachdb/cockroach/pull/55721 -[#55751]: https://github.com/cockroachdb/cockroach/pull/55751 -[#55759]: https://github.com/cockroachdb/cockroach/pull/55759 -[#55760]: https://github.com/cockroachdb/cockroach/pull/55760 -[#55783]: https://github.com/cockroachdb/cockroach/pull/55783 -[#55785]: https://github.com/cockroachdb/cockroach/pull/55785 -[#55808]: https://github.com/cockroachdb/cockroach/pull/55808 -[#55812]: https://github.com/cockroachdb/cockroach/pull/55812 -[#55827]: https://github.com/cockroachdb/cockroach/pull/55827 -[#55831]: https://github.com/cockroachdb/cockroach/pull/55831 -[#55845]: https://github.com/cockroachdb/cockroach/pull/55845 -[#55866]: https://github.com/cockroachdb/cockroach/pull/55866 -[#55867]: https://github.com/cockroachdb/cockroach/pull/55867 -[#55891]: https://github.com/cockroachdb/cockroach/pull/55891 -[#55894]: https://github.com/cockroachdb/cockroach/pull/55894 -[#55907]: https://github.com/cockroachdb/cockroach/pull/55907 -[#55915]: https://github.com/cockroachdb/cockroach/pull/55915 -[#55916]: https://github.com/cockroachdb/cockroach/pull/55916 -[#55934]: https://github.com/cockroachdb/cockroach/pull/55934 -[#55945]: https://github.com/cockroachdb/cockroach/pull/55945 -[#55958]: https://github.com/cockroachdb/cockroach/pull/55958 -[#55961]: https://github.com/cockroachdb/cockroach/pull/55961 -[#55969]: https://github.com/cockroachdb/cockroach/pull/55969 -[#55970]: https://github.com/cockroachdb/cockroach/pull/55970 -[#55982]: https://github.com/cockroachdb/cockroach/pull/55982 -[#55983]: https://github.com/cockroachdb/cockroach/pull/55983 -[#55993]: https://github.com/cockroachdb/cockroach/pull/55993 -[#55994]: https://github.com/cockroachdb/cockroach/pull/55994 -[#56007]: https://github.com/cockroachdb/cockroach/pull/56007 -[#56025]: https://github.com/cockroachdb/cockroach/pull/56025 -[#56080]: https://github.com/cockroachdb/cockroach/pull/56080 -[#56101]: https://github.com/cockroachdb/cockroach/pull/56101 -[#56103]: https://github.com/cockroachdb/cockroach/pull/56103 -[#56135]: https://github.com/cockroachdb/cockroach/pull/56135 -[#56144]: https://github.com/cockroachdb/cockroach/pull/56144 -[#56183]: https://github.com/cockroachdb/cockroach/pull/56183 -[#56198]: https://github.com/cockroachdb/cockroach/pull/56198 -[#56213]: https://github.com/cockroachdb/cockroach/pull/56213 -[#56283]: https://github.com/cockroachdb/cockroach/pull/56283 -[#56296]: https://github.com/cockroachdb/cockroach/pull/56296 -[#56298]: https://github.com/cockroachdb/cockroach/pull/56298 -[#56344]: https://github.com/cockroachdb/cockroach/pull/56344 -[#56352]: https://github.com/cockroachdb/cockroach/pull/56352 -[#56363]: https://github.com/cockroachdb/cockroach/pull/56363 -[#56373]: https://github.com/cockroachdb/cockroach/pull/56373 -[#56388]: https://github.com/cockroachdb/cockroach/pull/56388 -[#56392]: https://github.com/cockroachdb/cockroach/pull/56392 -[#56455]: https://github.com/cockroachdb/cockroach/pull/56455 -[#56458]: https://github.com/cockroachdb/cockroach/pull/56458 -[#56459]: https://github.com/cockroachdb/cockroach/pull/56459 -[#56461]: https://github.com/cockroachdb/cockroach/pull/56461 -[#56470]: https://github.com/cockroachdb/cockroach/pull/56470 -[#56477]: https://github.com/cockroachdb/cockroach/pull/56477 -[#56524]: https://github.com/cockroachdb/cockroach/pull/56524 -[#56533]: https://github.com/cockroachdb/cockroach/pull/56533 -[#56546]: https://github.com/cockroachdb/cockroach/pull/56546 -[#56574]: https://github.com/cockroachdb/cockroach/pull/56574 -[#56585]: https://github.com/cockroachdb/cockroach/pull/56585 -[#56587]: https://github.com/cockroachdb/cockroach/pull/56587 -[#56589]: https://github.com/cockroachdb/cockroach/pull/56589 -[#56591]: https://github.com/cockroachdb/cockroach/pull/56591 -[#56598]: https://github.com/cockroachdb/cockroach/pull/56598 -[#56627]: https://github.com/cockroachdb/cockroach/pull/56627 -[#56628]: https://github.com/cockroachdb/cockroach/pull/56628 -[#56631]: https://github.com/cockroachdb/cockroach/pull/56631 -[#56634]: https://github.com/cockroachdb/cockroach/pull/56634 -[#56653]: https://github.com/cockroachdb/cockroach/pull/56653 -[#56672]: https://github.com/cockroachdb/cockroach/pull/56672 -[#56679]: https://github.com/cockroachdb/cockroach/pull/56679 -[#56681]: https://github.com/cockroachdb/cockroach/pull/56681 -[#56708]: https://github.com/cockroachdb/cockroach/pull/56708 -[#56732]: https://github.com/cockroachdb/cockroach/pull/56732 -[#56737]: https://github.com/cockroachdb/cockroach/pull/56737 -[#56762]: https://github.com/cockroachdb/cockroach/pull/56762 -[#56773]: https://github.com/cockroachdb/cockroach/pull/56773 -[#56822]: https://github.com/cockroachdb/cockroach/pull/56822 -[#56827]: https://github.com/cockroachdb/cockroach/pull/56827 -[#56829]: https://github.com/cockroachdb/cockroach/pull/56829 -[#56837]: https://github.com/cockroachdb/cockroach/pull/56837 -[#56855]: https://github.com/cockroachdb/cockroach/pull/56855 -[#56858]: https://github.com/cockroachdb/cockroach/pull/56858 -[#56860]: https://github.com/cockroachdb/cockroach/pull/56860 -[#56869]: https://github.com/cockroachdb/cockroach/pull/56869 -[#56872]: https://github.com/cockroachdb/cockroach/pull/56872 -[#56874]: https://github.com/cockroachdb/cockroach/pull/56874 -[#56880]: https://github.com/cockroachdb/cockroach/pull/56880 -[#56881]: https://github.com/cockroachdb/cockroach/pull/56881 -[#56883]: https://github.com/cockroachdb/cockroach/pull/56883 -[#56898]: https://github.com/cockroachdb/cockroach/pull/56898 -[#56901]: https://github.com/cockroachdb/cockroach/pull/56901 -[#56942]: https://github.com/cockroachdb/cockroach/pull/56942 -[#56943]: https://github.com/cockroachdb/cockroach/pull/56943 -[#56953]: https://github.com/cockroachdb/cockroach/pull/56953 -[#56964]: https://github.com/cockroachdb/cockroach/pull/56964 -[#56974]: https://github.com/cockroachdb/cockroach/pull/56974 -[#56975]: https://github.com/cockroachdb/cockroach/pull/56975 -[#57003]: https://github.com/cockroachdb/cockroach/pull/57003 -[#57004]: https://github.com/cockroachdb/cockroach/pull/57004 -[#57006]: https://github.com/cockroachdb/cockroach/pull/57006 -[#57007]: https://github.com/cockroachdb/cockroach/pull/57007 -[#57038]: https://github.com/cockroachdb/cockroach/pull/57038 -[#57040]: https://github.com/cockroachdb/cockroach/pull/57040 -[#57050]: https://github.com/cockroachdb/cockroach/pull/57050 -[#57076]: https://github.com/cockroachdb/cockroach/pull/57076 -[#57100]: https://github.com/cockroachdb/cockroach/pull/57100 -[#57106]: https://github.com/cockroachdb/cockroach/pull/57106 -[#57121]: https://github.com/cockroachdb/cockroach/pull/57121 -[#57132]: https://github.com/cockroachdb/cockroach/pull/57132 -[#57139]: https://github.com/cockroachdb/cockroach/pull/57139 -[#57142]: https://github.com/cockroachdb/cockroach/pull/57142 -[#57143]: https://github.com/cockroachdb/cockroach/pull/57143 -[#57152]: https://github.com/cockroachdb/cockroach/pull/57152 -[#57153]: https://github.com/cockroachdb/cockroach/pull/57153 -[#57179]: https://github.com/cockroachdb/cockroach/pull/57179 -[#57180]: https://github.com/cockroachdb/cockroach/pull/57180 -[#57197]: https://github.com/cockroachdb/cockroach/pull/57197 -[#57212]: https://github.com/cockroachdb/cockroach/pull/57212 -[#57241]: https://github.com/cockroachdb/cockroach/pull/57241 -[#57256]: https://github.com/cockroachdb/cockroach/pull/57256 -[#57265]: https://github.com/cockroachdb/cockroach/pull/57265 -[#57278]: https://github.com/cockroachdb/cockroach/pull/57278 -[#57284]: https://github.com/cockroachdb/cockroach/pull/57284 -[#57309]: https://github.com/cockroachdb/cockroach/pull/57309 -[#57316]: https://github.com/cockroachdb/cockroach/pull/57316 -[#57320]: https://github.com/cockroachdb/cockroach/pull/57320 -[#57355]: https://github.com/cockroachdb/cockroach/pull/57355 diff --git a/src/current/_includes/releases/v21.1/v21.1.0-alpha.2.md b/src/current/_includes/releases/v21.1/v21.1.0-alpha.2.md deleted file mode 100644 index 1a489dc2cd0..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.0-alpha.2.md +++ /dev/null @@ -1,528 +0,0 @@ -## v21.1.0-alpha.2 - -Release Date: February 1, 2021 - - - -

Backward-incompatible changes

- -- The payload fields for certain event types in `system.eventlog` have been changed and/or renamed. Note that the payloads in `system.eventlog` were an undocumented, reserved feature so no guarantee was made about cross-version compatibility to this point. The list of changes includes (but is not limited to): - - `TargetID` has been renamed to `NodeID` for `node_join`. - - `TargetID` has been renamed to `TargetNodeID` for `node_decommissioning` / `node_decommissioned` / `node_recommissioned`. - - `NewDatabaseName` has been renamed to `NewDatabaseParent` for `convert_to_schema`. - - `grant_privilege` and `revoke_privilege` have been removed; they are replaced by `change_database_privilege`, `change_schema_privilege`, `change_type_privilege`, and `change_table_privilege`. Each event only reports a change for one user/role, so the `Grantees` field was renamed to `Grantee`. - - Each `drop_role` event now pertains to a single [user/role](https://www.cockroachlabs.com/docs/v21.1/authorization#sql-users). [#57737][#57737] -- The connection and authentication logging enabled by the [cluster settings](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) `server.auth_log.sql_connections.enabled` and `server.auth_log.sql_sessions.enabled` was previously using a text format which was hard to parse and integrate with external monitoring tools. This has been changed to use the standard notable event mechanism, with standardized payloads. The output format is now structured; see the generated [reference documentation](https://github.com/cockroachdb/cockroach/blob/master/docs/generated/eventlog.md) for details about the supported event types and payloads. [#57839][#57839] {% comment %}doc{% endcomment %} -- The logging format for [SQL audit](https://www.cockroachlabs.com/docs/v21.1/sql-audit-logging), [execution](https://www.cockroachlabs.com/docs/v21.1/logging-use-cases#sql_exec), and [query](https://www.cockroachlabs.com/docs/v21.1/logging-use-cases#sql_perf) logs has changed from a crude space-delimited format to JSON. To opt out of this new behavior and restore the pre-v21.1 logging format, you can set the [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) `sql.log.unstructured_entries.enabled` to `true`. [#59110][#59110] {% comment %}doc{% endcomment %} - -

Security updates

- -- The entropy of the auto-generated user account password for [`cockroach demo`](https://www.cockroachlabs.com/docs/v21.1/cockroach-demo) shell has been lowered, to ensure that the passwords are short and easy to copy over. This makes the password easy to brute-force and thus removes any remaining protection there may have been to the confidentiality of `demo` sessions. (The reason why a password remains, as opposed to no password whatsoever, is to prevent accidental misuse of the HTTP service by other applications running on the same machine.) Since the required shortness erases any pretense of security, the algorithm has been simplified to remove usage of a cryptographic PRNG altogether. [#58305][#58305] {% comment %}doc{% endcomment %} -- When using a SQL proxy, by default CockroachDB only knows about the network address of the proxy. That *peer* address is then used for logging, [authentication](https://www.cockroachlabs.com/docs/v21.1/authentication) rules, etc. This is undesirable, as security logging and authentication rules need to operate on the actual (final) client address instead. CockroachDB can now be configured to solve this problem (conf mechanism detailed below). When so configured, a SQL proxy can inform the CockroachDB server of the real address of the client via a server status parameter called `crdb:remote_addr`. The value must be the IP address of the client, followed by a colon, followed by the port number, using the standard Go syntax (e.g., `11.22.33.44:5566` for IPv4, `[11:22::33]:4455` for IPv6). When provided, this value overrides the SQL proxy's address for logging and authentication purposes. In any case, the original peer address is also logged alongside the client address (overridden or not), via the new logging tag `peer`. Security considerations: - - Enabling this feature allows the peer to spoof its address with regard to authentication and thus bypass authentication rules that would otherwise apply to its address, which can introduce a serious security vulnerability if the peer is not trusted. This is why this feature is not enabled by default, and must only be enabled when using a trusted SQL proxy. - - This feature should only be used with SQL proxies which actively scrub a `crdb:remote_addr` parameter received by a remote client, and replaces it by its own. If the proxy mistakenly forwards the parameter as provided by the client, it opens the door to the aforementioned security vulnerability. - - Care must be taken in host-based authentication (HBA) rules: - - TLS client cert validation, if requested by a rule, is still performed using the certificate presented by the proxy, not that presented by the client. This means that this new feature is not sufficient to forward TLS client cert authentication through a proxy. (If TLS client cert authentication is required, it must be performed by the proxy directly.) - - The `protocol` field (first column) continues to apply to the connection type between CockroachDB and the proxy, not between the proxy and the client. Only the 4th column (the CIDR pattern) is matched against the proxy-provided remote address override. Therefore, it is not possible to apply different rules to different client address when proxying TCP connections via a unix socket, because HBA rules for unix connections do not use the address column. Also when proxying client SSL connections via a non-SSL proxy connection, or proxying client non-SSL connections via a SSL proxy connection, care must be taken to configure address-based rule matching using the proper connection type. A reliable way to bypass this complexity is to only use the `host` connection type which applies equally to SSL and non-SSL connections. As of this implementation, the feature is enabled using the non-documented environment variable `COCKROACH_TRUST_CLIENT_PROVIDED_SQL_REMOTE_ADDR`. The use of an environment variable is a stop-gap so that this feature can be used in CockroachCloud SQL pods, which do not have access to [cluster settings](https://www.cockroachlabs.com/docs/v21.1/cluster-settings). The environment variable will be eventually removed and replaced by another mechanism. [#58381][#58381] -- Added ability to set region-specific callback URLs in the OIDC config. The `server.oidc_authentication.redirect_url` [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) can now accept JSON as an alternative to the basic URL string setting. If a JSON value is set, it *must* contain a `redirect_url` key that maps to an object with key-value pairs where the key is a `region` matching an existing locality setting and the value is a callback URL. [#57519][#57519] {% comment %}doc{% endcomment %} - -

General changes

- -- CockroachDB now runs fewer threads in parallel if running inside a container with a CPU limit. [#57390][#57390] -- Upgraded the CockroachDB binary to Go 1.15.6. [#57713][#57713] {% comment %}doc{% endcomment %} -- The new [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) `server.eventlog.enabled` controls whether notable events are also written to the table `system.eventlog`. Its default value is `true`. Changing this cluster setting can help recovering partial cluster availability when the `system.eventlog` table becomes unavailable. Note that even when `false`, notable events are still propagated to the logging system, where they can be redirected to files or other log sinks. [#57879][#57879] {% comment %}doc{% endcomment %} -- Moved RBAC resources in Kubernetes manifests to `rbac.authorization.k8s.io/v1`. [#55065][#55065] {% comment %}doc{% endcomment %} -- Cluster version upgrades, as initiated by [`SET CLUSTER SETTING version = -`](https://www.cockroachlabs.com/docs/v21.1/set-cluster-setting), now perform internal maintenance duties that will increase the time that it takes for the command to complete. This delay is proportional to the amount of data currently stored in the cluster. The cluster will also experience a small amount of additional load during this period while the upgrade is being finalized. These changes expand upon our original prototype in [#57445](https://github.com/cockroachdb/cockroach/pull/57445). [#58088][#58088] {% comment %}doc{% endcomment %} -- Upgraded to v3.9.2 of `protobuf` to consume new changes for Bazel. [#58891][#58891] -- Increased the default value of `sql.tablecache.lease.refresh_limit` to 500 in order to accommodate larger schemas without encountering `RETRY_COMMIT_DEADLINE_EXCEEDED` errors and generally higher latency. [#59319][#59319] {% comment %}doc{% endcomment %} - -

Enterprise edition changes

- -- CockroachDB now permits partitioning by user-defined [types](https://www.cockroachlabs.com/docs/v21.1/data-types) such as [enums](https://www.cockroachlabs.com/docs/v21.1/enum). [#57327][#57327] {% comment %}doc{% endcomment %} - -

SQL language changes

- -- Added a new [built-in](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators#built-in-functions) method to calculate Voronoi polygons. [#56496][#56496] {% comment %}doc{% endcomment %} -- Added a new [built-in](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators#built-in-functions) method to calculate Voronoi lines. [#56496][#56496] {% comment %}doc{% endcomment %} -- Previously, timezones had to be entered in the same case as they were stored on the system. Now, timezone names can be case-insensitive provided they match well-known zone names according to [Go's `time/tzdata` package](https://golang.org/src/time/tzdata/tzdata.go). [#57250][#57250] {% comment %}doc{% endcomment %} -- Added `LOCALITY REGIONAL BY TABLE IN PRIMARY REGION` syntax for configuring table locality. [#57275][#57275] {% comment %}doc{% endcomment %} -- The default value for `vectorize_row_count_threshold` setting has been decreased from 1000 to 0, meaning that, by default, CockroachDB will always use the [vectorized engine](https://www.cockroachlabs.com/docs/v21.1/vectorized-execution) for all supported queries regardless of the row estimate (unless `vectorize=off` is set). [#55713][#55713] {% comment %}doc{% endcomment %} -- Added the ability to [`CREATE TABLE`](https://www.cockroachlabs.com/docs/v21.1/create-table) with LOCALITY set. [#57419][#57419] {% comment %}doc{% endcomment %} -- [`EXPLAIN`](https://www.cockroachlabs.com/docs/v21.1/explain) and [`EXPLAIN ANALYZE`](https://www.cockroachlabs.com/docs/v21.1/explain-analyze) now show counts and durations in a more user-friendly way. [#57420][#57420] {% comment %}doc{% endcomment %} -- [`EXPORT`](https://www.cockroachlabs.com/docs/v21.1/export) can now be used by non-`admin` users with `SELECT` privilege on the table being exported, unless the destination URI requires `admin` privileges. [#57380][#57380] {% comment %}doc{% endcomment %} -- [`CREATE DATABASE ... WITH [PRIMARY] REGIONS`](https://www.cockroachlabs.com/docs/v21.1/create-database) now modifies the [zone configurations](https://www.cockroachlabs.com/docs/v21.1/configure-replication-zones) with the settings as set by the multiregion configuration. [#57244][#57244] {% comment %}doc{% endcomment %} -- [`SHOW [BACKUP]`](https://www.cockroachlabs.com/docs/v21.1/show-backup) is no longer `admin`-only. It depends on the URI construct and the credentials specified in the [`SHOW [BACKUP]`](https://www.cockroachlabs.com/docs/v21.1/show-backup) query. [#57412][#57412] {% comment %}doc{% endcomment %} -- [built-in](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators#built-in-functions) aggregate function `regr_avgx` is now supported. [#57295][#57295] {% comment %}doc{% endcomment %} -- Added support for running [`IMPORT`](https://www.cockroachlabs.com/docs/v21.1/import) in a mixed-version cluster. [#57382][#57382] {% comment %}doc{% endcomment %} -- Introduced `SHOW REGIONS FROM ALL DATABASES`, which shows region metadata for each database in the table. [#57500][#57500] {% comment %}doc{% endcomment %} -- Added `collations` table to the `information_schema`. The `collations` table contains the collations available in the current database. [#57609][#57609] {% comment %}doc{% endcomment %} -- Added `collation_character_set_applicability` table to the `information_schema`. [#57609][#57609] {% comment %}doc{% endcomment %} -- `LOCALITY` directives on [`CREATE TABLE`](https://www.cockroachlabs.com/docs/v21.1/create-table) are now persisted onto the database. [#57513][#57513] {% comment %}doc{% endcomment %} -- Multi-region defined table localities now display on the [`SHOW CREATE TABLE`](https://www.cockroachlabs.com/docs/v21.1/show-create) command. [#57513][#57513] {% comment %}doc{% endcomment %} -- Arrays in `pg_catalog` tables now are displayed in a format that matches PostgreSQL. [#56980][#56980] {% comment %}doc{% endcomment %} -- The [`SET TRACING`](https://www.cockroachlabs.com/docs/v21.1/set-vars#set-tracing) `local` option (to only trace messages issued by the local node) was removed. [#57567][#57567] {% comment %}doc{% endcomment %} -- Several `SHOW` commands have been adjusted to enforce a particular ordering of the output rows. [#57644][#57644] {% comment %}doc{% endcomment %} -- Implemented `regr_avgy` [built-in](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators#built-in-functions) aggregation function. [#57583][#57583] {% comment %}doc{% endcomment %} -- `crdb_internal.tables` and [`SHOW TABLES`](https://www.cockroachlabs.com/docs/v21.1/show-tables) now show locality data on the tables. [#57653][#57653] {% comment %}doc{% endcomment %} -- [`CREATE TABLE ... LOCALITY ...`](https://www.cockroachlabs.com/docs/v21.1/create-table) statements now modify the [zone configurations](https://www.cockroachlabs.com/docs/v21.1/configure-replication-zones) to be in line with the desired locality. [#57654][#57654] {% comment %}doc{% endcomment %} -- Added `sql.trace.stmt.enable_thresold`. Similar to `sql.trace.txn.enable_threshold`, this logs a trace of any statements that take longer than the specified duration. This new setting allows for more granularity than the transaction threshold since it applies to individual statements and does not include overhead such as communication with a SQL client. [#57733][#57733] {% comment %}doc{% endcomment %} -- Added a session setting `experimental_enable_unique_without_index_constraints` and cluster default `sql.defaults.experimental_enable_unique_without_index_constraints.enabled` to enable the use of `UNIQUE WITHOUT INDEX` syntax. The default value of both settings is `false` since this feature is not yet fully supported. Use of `UNIQUE WITHOUT INDEX` also depends on all nodes being upgraded to the cluster version `UniqueWithoutIndexConstraints`. [#57666][#57666] {% comment %}doc{% endcomment %} -- Implemented [built-in](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators#built-in-functions) function `levenshtein`. [#56843][#56843] {% comment %}doc{% endcomment %} -- Added table-view dependency information in `pg_depend` to improve compatibility with PostgreSQL. [#57240][#57240] {% comment %}doc{% endcomment %} -- The cluster event logging system has been modernized. In particular, the schema of the entries for the `info` column in `system.eventlog` has been stabilized. [#57737][#57737] {% comment %}doc{% endcomment %} -- The `targetID` and `reportingID` columns in `system.eventlog` are now deprecated. Their values, for relevant event types, can be found as fields inside the `info` column instead. [#57737][#57737] {% comment %}doc{% endcomment %} -- Added the `soundex()` and `difference()` [built-in](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators#built-in-functions) functions. [#57615][#57615] {% comment %}doc{% endcomment %} -- New [`EXPLAIN ANALYZE (DISTSQL)`](https://www.cockroachlabs.com/docs/v21.1/explain-analyze) output format. [`EXPLAIN ANALYZE (DISTSQL)`](https://www.cockroachlabs.com/docs/v21.1/explain-analyze) now only works as a top-level statement (it can no longer be part of a bigger query). [#57804][#57804] {% comment %}doc{% endcomment %} -- Added [built-in](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators#built-in-functions) function `ST_OrientedEnvelope` to calculate the mimimum-area rotated rectangle. [#57697][#57697] {% comment %}doc{% endcomment %} -- A new `default_transaction_use_follower_reads` session variable is now supported, which configures SQL transactions to perform stale reads from follower replicas. [#57851][#57851] {% comment %}doc{% endcomment %} -- Both [`ALTER TABLE OWNER`](https://www.cockroachlabs.com/docs/v21.1/alter-table) and `REASSIGN OWNED BY` now report structured notable events about the ownership changes. Note that `REASSIGN OWNED BY` currently also reports an `alter_table_owner` for views and sequences that were implicitly reassigned, even though CockroachDB does not yet support the `ALTER VIEW OWNER` / `ALTER SEQUENCE OWNER` statements. [#57969][#57969] {% comment %}doc{% endcomment %} -- [`EXPLAIN (DISTSQL)`](https://www.cockroachlabs.com/docs/v21.1/explain) has a new output schema and format. [#57954][#57954] -- Added an overload to `crdb_internal.pb_to_json` to suppress populating default values in fields. [#58087][#58087] {% comment %}doc{% endcomment %} -- [`IMPORT INTO`](https://www.cockroachlabs.com/docs/v21.1/import-into) for CSV now supports `nextval` as a default expression of a non-targeted column. [#56473][#56473] {% comment %}doc{% endcomment %} -- The [`EXPLAIN`](https://www.cockroachlabs.com/docs/v21.1/explain) output for foreign key checks is now labeled `constraint-check` rather than `fk-check`. This change is in preparation for adding support for unique constraint checks, which will use the same label. [#58053][#58053] {% comment %}doc{% endcomment %} -- The error message for unique constraint violations now matches the error used by PostgreSQL. For example, the new error message looks like this: `ERROR: duplicate key value violates unique constraint "primary" DETAIL: Key (k)=(1) already exists.` [#58053][#58053] {% comment %}doc{% endcomment %} -- [`EXPLAIN ANALYZE`](https://www.cockroachlabs.com/docs/v21.1/explain-analyze) diagrams now contain "network hops" information on streams. [#58078][#58078] {% comment %}doc{% endcomment %} -- Added support for the `IF NOT EXISTS` prefix to [`CREATE TYPE`](https://www.cockroachlabs.com/docs/v21.1/create-type) statements. [#58173][#58173] {% comment %}doc{% endcomment %} -- Added a `session_variables` table to the `information_schema`. The `session_variables` table exposes the session variables. [#57837][#57837] {% comment %}doc{% endcomment %} -- Indexing into scalar JSON values using an integer index is now properly supported. [#58359][#58359] {% comment %}doc{% endcomment %} -- The `crdb_internal.cluster_id` method now returns the ID of the underlying KV cluster in multi-tenant scenarios rather than the Nil UUID. The `ClusterID` -is needed for logging and metrics for SQL tenants. [#58317][#58317] {% comment %}doc{% endcomment %} -- `SHOW STATEMENTS` is now an alias of [`SHOW QUERIES`](https://www.cockroachlabs.com/docs/v21.1/show-statements). The new syntax is preferred by the SQL parser. [#58072][#58072] {% comment %}doc{% endcomment %} -- Implemented the `gateway_region` [built-in](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators#built-in-functions), which returns the region of the connection's current node. [#58423][#58423] {% comment %}doc{% endcomment %} -- [`BACKUP`](https://www.cockroachlabs.com/docs/v21.1/backup) and [`RESTORE`](https://www.cockroachlabs.com/docs/v21.1/restore) now use the block-level checksums embedded in their data files instead of collecting / verifying more expensive file-level SHA512 checksums. [#58487][#58487] {% comment %}doc{% endcomment %} -- Multi-tenant clusters will now send anonymous usage information to the central CockroachDB registration server. [#58399][#58399] {% comment %}doc{% endcomment %} -- Added support for [`ALTER DATABASE ... ADD REGION`](https://www.cockroachlabs.com/docs/v21.1/alter-database). [#58233][#58233] {% comment %}doc{% endcomment %} -- Multiple [`SHOW ZONE CONFIGURATION`](https://www.cockroachlabs.com/docs/v21.1/show-zone-configurations) statements that previously used `FOR` can now also use `FROM` (e.g., `SHOW ZONE CONFIGURATION FOR RANGE`). This change standardizes the use of `FROM` for `SHOW ZONE CONFIGURATION`. [#58740][#58740] {% comment %}doc{% endcomment %} -- Implemented `SHOW REGIONS`, which shows a list of regions along with the databases associated with them. [#57618][#57618] {% comment %}doc{% endcomment %} -- Added [`ALTER DATABASE ... PRIMARY REGION`](https://www.cockroachlabs.com/docs/v21.1/alter-database). [#58725][#58725] {% comment %}doc{% endcomment %} -- Columns implicitly added from [hash-sharded indexes](https://www.cockroachlabs.com/docs/v21.1/experimental-features#hash-sharded-indexes) will no longer display on `pg_index` and `pg_indexes`. [#58749][#58749] {% comment %}doc{% endcomment %} -- Added a new `implicit` column to `crdb_internal.index_columns`, which signifies whether the column is implicitly added onto the index through [`PARTITION BY`](https://www.cockroachlabs.com/docs/v21.1/partition-by) with an implicit column or a [hash-sharded index](https://www.cockroachlabs.com/docs/v21.1/experimental-features#hash-sharded-indexes). [#58749][#58749] {% comment %}doc{% endcomment %} -- CockroachDB now omits implicit columns or [hash-sharded index](https://www.cockroachlabs.com/docs/v21.1/experimental-features#hash-sharded-indexes) columns from automatically generated index names. [#58898][#58898] {% comment %}doc{% endcomment %} -- Implemented `PARTITION ALL BY` syntax for [`CREATE TABLE`](https://www.cockroachlabs.com/docs/v21.1/create-table), which automatically partitions the table, and all indexes, with the same partitioning scheme. [#58748][#58748] {% comment %}doc{% endcomment %} -- `PARTITION ALL BY` tables now display correctly on [`SHOW CREATE TABLE`](https://www.cockroachlabs.com/docs/v21.1/show-create). [#58928][#58928] {% comment %}doc{% endcomment %} -- Creating a table and changing its schema within a transaction no longer schedules a schema change job. [#58888][#58888] {% comment %}doc{% endcomment %} -- Implemented geo [built-in](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators#built-in-functions) function `ST_GeneratePoints`. [#58288][#58288] {% comment %}doc{% endcomment %} -- Hash aggregation can now spill to disk when it exhausts its memory limit when executed via the [vectorized engine](https://www.cockroachlabs.com/docs/v21.1/vectorized-execution). [#57817][#57817] {% comment %}doc{% endcomment %} -- Implemented [`ALTER DATABASE ... SURVIVE REGION/ZONE FAILURE`](https://www.cockroachlabs.com/docs/v21.1/alter-database). [#58937][#58937] {% comment %}doc{% endcomment %} -- [`CREATE INDEX`](https://www.cockroachlabs.com/docs/v21.1/create-index) will now inherit `PARTITION BY` clauses from `PARTITION ALL BY` tables. [#58988][#58988] {% comment %}doc{% endcomment %} -- Implemented the geometry-based [built-in](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators#built-in-functions) functions `ST_LineSubstring({geometry,float8,float8})`. [#58125][#58125] {% comment %}doc{% endcomment %} -- Implemented `REGIONAL BY ROW` logic on [`CREATE TABLE`](https://www.cockroachlabs.com/docs/v21.1/create-table). This is gated behind the experimental session variable `experimental_enable_implicit_column_partitioning`. [#58987][#58987] {% comment %}doc{% endcomment %} -- Added support for `ALTER VIEW` / `SEQUENCE OWNER TO` commands. [#59049][#59049] {% comment %}doc{% endcomment %} -- Casts to type `unknown[]` are no longer accepted in CockroachDB. Any such casts will fail to parse and return the error `ERROR: type "unknown[]" does not exist`. This is consistent with PostgreSQL's behavior. [#59136][#59136] {% comment %}doc{% endcomment %} -- Implemented `REGIONAL BY ROW AS ...`, which allows a column of type `crdb_internal_region` to be used as a column for `REGIONAL BY ROW` multi-region properties. [#59121][#59121] {% comment %}doc{% endcomment %} -- Enabled altering of table locality for tables that are `REGIONAL BY TABLE` to `REGIONAL BY TABLE` (but in a different region). [#59144][#59144] {% comment %}doc{% endcomment %} -- [`PARTITION ALL BY`](https://www.cockroachlabs.com/docs/v21.1/partition-by) statements will now apply for tables when using [`ALTER PRIMARY KEY`](https://www.cockroachlabs.com/docs/v21.1/alter-primary-key). [#59178][#59178] {% comment %}doc{% endcomment %} -- Implemented the geometry [built-in](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators#built-in-functions) function `ST_ShiftLongitude`, which is useful for plotting data on a Pacific-centred map. [#59234][#59234] {% comment %}doc{% endcomment %} -- Implemented [`ALTER TABLE ... SET LOCALITY GLOBAL`](https://www.cockroachlabs.com/docs/v21.1/alter-table) for tables starting as `REGIONAL BY ROW`. [#59256][#59256] {% comment %}doc{% endcomment %} -- Improved the error message when people use set `search_path` incorrectly, or with a schema that legitimately has a comma in its name. [#53974][#53974] {% comment %}doc{% endcomment %} -- Creation of interleaved tables and indexes is now disabled by default. They can be re-enabled temporarily by running [`SET CLUSTER SETTING sql.defaults.interleaved_tables.enabled=true`](https://www.cockroachlabs.com/docs/v21.1/set-cluster-setting). [#59248][#59248] {% comment %}doc{% endcomment %} -- CockroachDB now uses a structured logging format for the [SQL audit](https://www.cockroachlabs.com/docs/v21.1/sql-audit-logging), [execution](https://www.cockroachlabs.com/docs/v21.1/logging-use-cases#sql_exec), and [query](https://www.cockroachlabs.com/docs/v21.1/logging-use-cases#sql_perf) logs. See the generated [reference documentation](https://github.com/cockroachdb/cockroach/blob/master/docs/generated/eventlog.md) for details. Of note, audit and execution logs now also include information about whether a query plan contains full index scans. Previously, this information was only included in the slow query log. [#59110][#59110] {% comment %}doc{% endcomment %} -- [`CREATE INDEX`](https://www.cockroachlabs.com/docs/v21.1/create-index) on `REGIONAL BY ROW` tables will now correctly include the implicit partitioning and inherit the correct [zone configurations](https://www.cockroachlabs.com/docs/v21.1/configure-replication-zones). [#59223][#59223] {% comment %}doc{% endcomment %} -- Made a minor improvement to [`EXPLAIN`](https://www.cockroachlabs.com/docs/v21.1/explain) output for "render" node. [#59313][#59313] {% comment %}doc{% endcomment %} -- [`EXPLAIN ANALYZE`](https://www.cockroachlabs.com/docs/v21.1/explain-analyze) now defaults to the new `EXPLAIN ANALYZE (PLAN)`, which shows a text representation of the logical plan, annotated with execution statistics. {% comment %}doc{% endcomment %} [#57337][#57337] - -

Command-line changes

- -- The logging configuration can now be specified via the `--log` parameter. See the documentation for details. The flags `--log-dir`, `--log-file-max-size`, `--log-file-verbosity`, `--log-group-max-size` are now deprecated. [#57134][#57134] {% comment %}doc{% endcomment %} -- A new command `cockroach debug check-log-config` prints out the logging configuration that results from the provided combination of `--store`, `--log`, and other logging-related flags on the command line. [#57134][#57134] {% comment %}doc{% endcomment %} -- The events that were previously only stored in `system.eventlog` are now also directed unconditionally to an external logging channel using a JSON format. Refer to the [configuration](https://github.com/cockroachdb/cockroach/blob/master/docs/generated/logsinks.md) to see how to customize how events are directed to external sinks. Note that the exact external output format (and thus how to detect/parse the events from e.g., log files) is not yet stabilized and remains subject to change. [#57737][#57737] {% comment %}doc{% endcomment %} -- Notable events that pertain to SQL schema, user, and privilege changes are now sent on the new `SQL_SCHEMA`, `USER_ADMIN`, and `PRIVILEGES` [logging channels](https://github.com/cockroachdb/cockroach/blob/master/docs/generated/logging.md). These can now be redirected to different [sinks](https://github.com/cockroachdb/cockroach/blob/master/docs/generated/logsinks.md) from the other log entries. - - The `SQL_SCHEMA` channel is used to report changes to the SQL logical schema, excluding privilege and ownership changes (which are reported on the separate channel `PRIVILEGES`) and zone config changes (which go to `OPS`). This includes: - - Database, schema, table, sequence, view, and type creation. - - Adding, removing, and changing table columns. - - Changing sequence parameters. - - More generally, changes to the schema that affect the functional behavior of client apps using stored objects. - - The `USER_ADMIN` channel is typically configured in "audit" mode, with event numbering and synchronous writes. It is used to report changes in users and roles, including: - - Users added and dropped. - - Changes to authentication credentials, including passwords, validity, etc. - - Role grants and revocations. - - Role option grants and revocations. - - The `PRIVILEGES` channel is typically configured in "audit" mode, with event numbering and synchronous writes. It is used to report data authorization changes, including: - - Privilege grants and revocations on database, objects, etc. - - Object ownership changes. [#51987][#51987] -- Logging events that are relevant to cluster operators are now categorized under the new `OPS` and `HEALTH` [logging channels](https://github.com/cockroachdb/cockroach/blob/master/docs/generated/logging.md). These can now be redirected separately from other logging events. - - The `OPS` channel is used to report "point" operational events, initiated by user operators or automation: - - Operator or system actions on server processes: process starts, stops, shutdowns, and crashes (if they can be logged). Each event includes any command-line parameters and the CockroachDB version. - - Actions that impact the topology of a cluster: node additions, removals, decommissions, etc. - - [Cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) changes - - [Zone configuration](https://www.cockroachlabs.com/docs/v21.1/configure-replication-zones) changes. - - The `HEALTH` channel is the channel used to report "background" operational events, initiated by CockroachDB for reporting on automatic processes: - - Current resource usage, including critical resource usage. - - Node-node connection events, including connection errors and gossip details. - - Range and table leasing events. - - Up- and down-replication. - - Range unavailability. [#57171][#57171] -- Server terminations that are triggered when a node encounters an internal fatal error are now reported on the `OPS` logging channel. The exact text of the error is not reported on the `OPS` channel, however, as it may be complex (e.g., when there is a replica inconsistency); and the `OPS` channel is typically monitored by tools that just detect irregularities. The text of the message refers instead to the channel where the additional details can be found. [#57171][#57171] {% comment %}doc{% endcomment %} -- The notable events `set_zone_config` and `remove_zone_config` are now sent to the `OPS` logging channel. [#57171][#57171] {% comment %}doc{% endcomment %} -- Added a flag to `cockroach debug decode-proto` to suppress populating default values in fields. [#58087][#58087] {% comment %}doc{% endcomment %} -- When a SQL notable (structured) event is logged, the payload now attempts to include the session's `application_name` as field `ApplicationName`. This is intended for use when setting up event routing and filtering in external tools. [#58130][#58130] {% comment %}doc{% endcomment %} -- It is now possible to set the `format` parameter of any log sink, including file sinks, to `json`, `json-compact`, `json-fluent`, or `json-fluent-compact` to write entries as structured JSON. Refer to the generated [reference documentation](https://github.com/cockroachdb/cockroach/blob/master/docs/generated/logformats.md) for details. [#58126][#58126] {% comment %}doc{% endcomment %} -- The DB Console URL printed by [`cockroach demo`](https://www.cockroachlabs.com/docs/v21.1/cockroach-demo) now automatically logs in the user when pasted into a web browser. [#56740][#56740] {% comment %}doc{% endcomment %} -- The URLs generated when [`cockroach demo`](https://www.cockroachlabs.com/docs/v21.1/cockroach-demo) starts have been made shorter and easier to copy/paste by shortening the generated password. [#58305][#58305] {% comment %}doc{% endcomment %} -- When using the JSON output formats for log entries, the server identifiers are now reported as part of each payload once known (either cluster ID + node ID in single-tenant or KV servers, or tenant ID + SQL instance ID in multi-tenant SQL servers). [#58128][#58128] {% comment %}doc{% endcomment %} -- Fixed [`cockroach demo --global`](https://www.cockroachlabs.com/docs/v21.1/cockroach-demo) from crashing with `didn't get expected magic bytes header`. [#58466][#58466] {% comment %}doc{% endcomment %} -- Previously, for certain log files, CockroachDB would both flush individual writes (i.e., propagate them from within the `cockroach` process to the OS) and also synchronize writes (i.e., ask the OS to confirm the log data was written to disk). Per-write synchronization was found to be unnecessary and possibly detrimental to performance and operating cost, so it was removed. Now, the log data continues to be flushed, as before, and CockroachDB only requests synchronization periodically (every 30s). [#58995][#58995] {% comment %}doc{% endcomment %} -- The parameter `sync-writes` for file sink configurations has been removed. (This is not a backward-incompatible change because the configuration feature is new in v21.1.) [#58995][#58995] {% comment %}doc{% endcomment %} -- The parameter `buffered-writes` for file sink configurations has been added. It is set to `true` (writes are buffered) by default and set to `false` (i.e., avoid buffering and flush every log entry) when the `auditable` flag is requested. [#58995][#58995] {% comment %}doc{% endcomment %} -- The default output format for `file-group` and `stderr` sinks has been changed to `crdb-v2`. This new format is non-ambiguous and makes it possible to reliably parse log files. Refer to the format's [documentation](https://github.com/cockroachdb/cockroach/blob/master/docs/generated/logformats.md) for details. Additionally, it prevents single log lines from exceeding a large size; this problem is inherent to the `crdb-v1` format and can prevent [`cockroach debug zip`](https://www.cockroachlabs.com/docs/v21.1/cockroach-debug-zip) from retrieving `v1` log files. The new format has also been designed so that existing log file analyzers for the `crdb-v1` format can read entries written the new format. However, this conversion may be imperfect. Refer to the new format's [documentation](https://github.com/cockroachdb/cockroach/blob/master/docs/generated/logformats.md) for details. In case of incompatibility, users can force the previous format by using `format: crdb-v1` in their logging configuration. [#59087][#59087] {% comment %}doc{% endcomment %} - -

API endpoint changes

- -- The notable event `create_statistics` is only reported when the [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) `sql.stats.post_events.enabled` is enabled. This fact is now also reported in the [event log reference documentation](https://github.com/cockroachdb/cockroach/blob/master/docs/generated/eventlog.md). [#57877][#57877] {% comment %}doc{% endcomment %} -- The `Timestamp` field of structured notable events is now numeric, and encodes a number of nanoseconds since the Unix epoch. [#58070][#58070] {% comment %}doc{% endcomment %} -- The Health API now checks that the SQL server is ready to accept clients when a readiness check is requested. [#59350][#59350] {% comment %}doc{% endcomment %} - -

DB Console changes

- -- Minor style changes to represent new branding palette. [#57130][#57130] -- Changed the default per-page value on the [Transactions page](https://www.cockroachlabs.com/docs/v21.1/ui-transactions-page) to 20; minor style updates. [#57824][#57824] -- Added a timeseries graph indicating statement denials due to feature flags on the [SQL Dashboard](https://www.cockroachlabs.com/docs/v21.1/ui-sql-dashboard) of DB Console. [#57533][#57533] {% comment %}doc{% endcomment %} -- Updated labels on the [Hardware Dashboard](https://www.cockroachlabs.com/docs/v21.1/ui-hardware-dashboard) to be more accurate. [#57224][#57224] -- Updated the link back to [Statements](https://www.cockroachlabs.com/docs/v21.1/ui-statements-page) from [Statement Details](https://www.cockroachlabs.com/docs/v21.1/ui-statements-page#statement-details-page) so that it will always link to the Statements list instead of invoking the "back" action on the browser. [#57975][#57975] -- On the [Sessions page](https://www.cockroachlabs.com/docs/v21.1/ui-sessions-page), every `age` label has been replaced with `duration` and every `txn` label has been replaced with `transaction`. The actual metrics remain unchanged. [#58616][#58616] {% comment %}doc{% endcomment %} -- Renamed the `CPUs` metric column to `vCPUs` in the Node List on the [Cluster Overview page](https://www.cockroachlabs.com/docs/v21.1/ui-cluster-overview-page). [#58495][#58495] {% comment %}doc{% endcomment %} -- Added Open SQL Transactions to the [SQL Dashboard](https://www.cockroachlabs.com/docs/v21.1/ui-sql-dashboard) and renamed `SQL Queries` to `SQL Statements`. Removed references to "Distributed SQL Queries" when metrics really are discussing all SQL queries. [#57477][#57477] {% comment %}doc{% endcomment %} - -

Bug fixes

- -- Fixed a bug in [`RESTORE`](https://www.cockroachlabs.com/docs/v21.1/restore) where some unusual range boundaries in interleaved tables caused an error. [#58219][#58219] -- Previously, CockroachDB would encounter an internal error when performing a `JSONB - String` operation via the [vectorized execution engine](https://www.cockroachlabs.com/docs/v21.1/vectorized-execution). This has been fixed. The bug was introduced in v20.2.0. [#57349][#57349] -- Previously, in rare situations, an automated replication change could result in a loss of quorum. This was possible when a cluster had down nodes and a simultaneous change in replication factor. Note that a change in the replication factor can occur automatically if the cluster is comprised of fewer than five available nodes. Experimentally the likelihood of encountering this issue, even under contrived conditions, was small. [#56735][#56735] -- Fixed an internal error when using aggregates and window functions in an `ORDER BY` for a `UNION` or `VALUES` clause. [#57498][#57498] -- `DROP TYPE` and certain other statements that work over SQL scalar types now properly support type names containing special characters. [#57354][#57354] -- Fixed a performance regression introduced in v20.2 to reading virtual tables which introspect the schema. [#57542][#57542] -- Removed `system.jobs` full table scan that is expensive in the face of many completed jobs. [#57587][#57587] -- In v20.2.0 we mistakenly permitted users with the `admin` role to drop tables in the system database. This commit revokes that privilege. [#57568][#57568] -- Previously, users could not perform a cluster restore from old backup chains (incrementals on top of fulls) when using the [`BACKUP INTO`](https://www.cockroachlabs.com/docs/v21.1/backup) syntax. [#57656][#57656] -- Fixed a bug that could cause [`IMPORT`](https://www.cockroachlabs.com/docs/v21.1/import) to incorrectly read files stored on Google Cloud if uploaded using its compression option (`gsutil -Z`). [#57745][#57745] -- Fixed a bug where `ST_MakeLine` and `ST_Collect` did not respect ordering when used over a window clause. [#57724][#57724] -- Fixed a bug where schema change jobs to add foreign keys to existing tables, via [`ALTER TABLE`](https://www.cockroachlabs.com/docs/v21.1/alter-table), could sometimes not be successfully reverted (either due to being canceled or having failed). [#57598][#57598] -- Fixed a bug where concurrent addition of a foreign key constraint and drop of a unique index could cause the [foreign key constraint](https://www.cockroachlabs.com/docs/v21.1/foreign-key) to be added with no unique constraint on the referenced columns. [#57598][#57598] -- Adding a primary key constraint to a table without a primary key no longer ignores the name specified for the primary key. [#57146][#57146] -- Previously, when displaying replica counts during the [decommissioning process](https://www.cockroachlabs.com/docs/v21.1/remove-nodes), we were overcounting the replicas displayed for r1. This is no longer the case. [#57812][#57812] -- Fixed a bug whereby tables in schemas other than `public` would not be displayed when running [`SHOW TABLES FROM `](https://www.cockroachlabs.com/docs/v21.1/show-tables). [#57749][#57749] -- Fixed a bug where canceled queries reading from virtual tables could cause a crashing panic. [#57828][#57828] -- Fixed an assertion error caused by some DDL statements used in conjunction with common table expressions (`WITH`). [#57927][#57927] -- Previously, [`SHOW GRANTS ON DATABASE`](https://www.cockroachlabs.com/docs/v21.1/show-grants) did not include privileges that were granted on a database. Now it does. The `SHOW GRANTS ON DATABASE` output no longer includes a column for `schema_name`, as these grants are not specific to any schema. [#56866][#56866] -- The `information_schema.schema_privileges` table now includes the correct schema-level privileges for non-user-defined schemas. Previously, all of these schemas were omitted from the table. [#56866][#56866] -- The `has_schema_privilege` [built-in](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators#built-in-functions) function now works on user-defined schemas when checking for the `USAGE` privilege. [#56866][#56866] -- The `ST_FrechetDistance` [built-in](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators#built-in-functions) function no longer causes a `SIGFPE` panic for very small values of `densifyFrac` (e.g., 1e-20), and returns an error instead. [#57966][#57966] -- CockroachDB now removes a node's status entry when the node is [decommissioned](https://www.cockroachlabs.com/docs/v21.1/remove-nodes), to prevent it from appearing in API calls and UIs, and prevent it from affecting node constraints such as localities and attributes for various operations. [#56529][#56529] -- Fixed a bug which caused type information to be omitted when decoding descriptors using either `crdb_internal.pb_to_json` or `cockroach debug decode-proto`. [#58087][#58087] -- Fixed a bug from v21.1.0-alpha.1 where the binary could crash if a running node lost its claim to a job while updating. [#58161][#58161] -- Fixed a bug where multiple invocations of `AddGeometryColumn` affecting the same table would result in only the last invocation applying. [#56663][#56663] -- Previously, CockroachDB could return non-deterministic output when querying the `information_schema.statistics` virtual table (internally used by [`SHOW INDEXES`](https://www.cockroachlabs.com/docs/v21.1/show-index))—namely, the implicit columns of the secondary indexes could be in arbitrary order. This is now fixed, and the columns will be in the same order as they are in the primary index. [#58191][#58191] -- Fixed a `column family 0 not found` crash caused by explaining or gathering [statement diagnostics](https://www.cockroachlabs.com/docs/v21.1/ui-statements-page#diagnostics) on certain queries involving virtual tables. [#58208][#58208] -- Added a safeguard against crashes while running `SHOW STATISTICS USING JSON`, which is used internally for statement diagnostics and [`EXPLAIN ANALYZE (DEBUG)`](https://www.cockroachlabs.com/docs/v21.1/explain-analyze). [#58221][#58221] -- Fixed a bug where prior schema changes on a table that failed and could not be fully reverted could prevent the table from being dropped. [#57836][#57836] -- Previously, CockroachDB could crash when performing a `DELETE` operation after an alteration of the primary key when in some cases. This is now fixed. The bug was introduced in v20.1. [#58153][#58153] -- Fixed a bug that could cause incremental backups to a backup in a collection (i.e., [`BACKUP INTO ... IN ...`](https://www.cockroachlabs.com/docs/v21.1/backup)) on some cloud storage providers to ignore existing incremental backups previously appended to that destination and instead backup incrementally from the base backup in that destination. [#58292][#58292] -- Fixed an internal panic when using the `SHOW STATISTICS USING JSON` statement on a table containing `ENUM` types. [#58251][#58251] -- A memory leak in the optimizer has been fixed. The leak could have caused unbounded growth of memory usage for a session when planning queries on tables with partial indexes. [#58306][#58306] -- Fixed a bug where primary key changes on tables being watched by [changefeeds](https://www.cockroachlabs.com/docs/v21.1/stream-data-out-of-cockroachdb-using-changefeeds) would be silently ignored. [#58140][#58140] -- The `pg_catalog` metadata tables were using the `CHAR` data type for single-byte character columns. Now they use the `char` data type, to match PostgreSQL. This resolves errors that would occur when using some drivers (like `tokio-postgres` for Rust) to access `pg_catalog` tables in the binary query format. [#58084][#58084] -- Prepared statements that include [enums](https://www.cockroachlabs.com/docs/v21.1/enum) and use the binary format will no longer result in an error. [#58043][#58043] -- Fixed an internal error that could occur when executing prepared statements with placeholders that delete data from columns referenced by a foreign key with `ON DELETE CASCADE`. [#58431][#58431] -- Fixed a bug which caused errors when querying a table with a disjunctive filter (an `OR` expression) that is the same or similar to the predicate of one of the table's partial indexes. [#58434][#58434] -- Previously, in event log entries for `ALTER TYPE` and `DROP TYPE` statements, the `TypeName` field did not contain fully qualified names. [#58257][#58257] -- A [`CREATE TABLE`](https://www.cockroachlabs.com/docs/v21.1/create-table) statement with indexes with duplicate names will no longer result in an assertion failure. This bug was present since v20.2. [#58433][#58433] -- Previously, event logs were not capturing the qualified table names for [`COMMENT ON INDEX`](https://www.cockroachlabs.com/docs/v21.1/comment-on) and [`COMMENT ON COLUMN`](https://www.cockroachlabs.com/docs/v21.1/comment-on) commands. This PR changes the event logs to use the qualified table name. Tests were also added for these changes. [#58472][#58472] -- The `has_${OBJECT}_privilege` built-in methods such as `has_schema_privilege` now additionally check whether the roles of which a user is a direct or indirect member also have privileges on the object. Previously only one user was checked, which was incorrect. This bug has been present since v2.0 but became more prominent in v20.2 when [role-based access control](https://www.cockroachlabs.com/docs/v21.1/authorization#roles) was made available in the core version of CockroachDB. [#58254][#58254] -- CockroachDB previously would return an internal error when attempting to execute a hash join on a JSON column via the [vectorized engine](https://www.cockroachlabs.com/docs/v21.1/vectorized-execution). Now a more user-friendly error is returned. [#57994][#57994] -- Fixed a panic in protobuf decoding. [#58716][#58716] -- The user authentication flow no longer performs extraneous name lookups. This performance regression was present since v20.2. [#58671][#58671] -- CockroachDB could previously return an internal error when evaluating a binary expression between a Decimal and an Interval that required a cast to a Float when the value is out of range. A more user-friendly error is now returned instead. [#58743][#58743] -- Qualified table name for `alter_table_owner` event log. [#58504][#58504] -- Fixed a bug that caused errors when accessing a tuple column (`tuple.column` syntax) of a tuple that could be statically determined to be null. [#58747][#58747] -- The `indoption` column in `pg_catalog.index` is now populated correctly. [#58947][#58947] -- Previously, CockroachDB could encounter an internal error when executing queries with tuples containing null values and [enums](https://www.cockroachlabs.com/docs/v21.1/enum) in a distributed setting. This is now fixed. [#58894][#58894] -- Fixed a nil pointer panic edge case in query setup code. [#59002][#59002] -- Non-ASCII characters in `NAME` results in [`cockroach sql`](https://www.cockroachlabs.com/docs/v21.1/cockroach-sql) / [`demo`](https://www.cockroachlabs.com/docs/v21.1/cockroach-demo) (e.g., in the results [`SHOW TABLES`](https://www.cockroachlabs.com/docs/v21.1/show-tables), [SHOW CONSTRAINTS](https://www.cockroachlabs.com/docs/v21.1/show-constraints)) are now displayed without being escaped to octal codes. [#56630][#56630] -- Garbage collection (GC) jobs now populate the `running_status` column for [`SHOW JOBS`](https://www.cockroachlabs.com/docs/v21.1/show-jobs). This bug has been present since v20.1. [#58612][#58612] -- Previously, CockroachDB could encounter an internal error when executing queries with `BYTES` or `STRING` types via the [vectorized engine](https://www.cockroachlabs.com/docs/v21.1/vectorized-execution) in rare circumstances. This is now fixed. [#59028][#59028] -- Previously, the `substring` function on byte arrays would treat its input as unicode code points, which would cause the wrong bytes to be returned. Now it only operates on the raw bytes. [#58265][#58265] -- Previously, the `substring(byte[])` functions were not able to interpret bytes that had the `\` character since it was being treated as the beginning of an escape sequence. This is now fixed. [#58265][#58265] -- Fixed a bug in which some non-conflicting rows provided as input to an [`INSERT ... ON CONFLICT DO NOTHING`](https://www.cockroachlabs.com/docs/v21.1/insert) statement could be discarded, and not inserted. This could happen in cases where the table had one or more unique indexes in addition to the primary index, and some of the rows in the input conflicted with existing values in one or more unique index. This scenario could cause the rows that did not conflict to be erroneously discarded. This has now been fixed. [#59147][#59147] -- Fixed an internal error that could occur when `ARRAY[NULL]` was used in a query due to incorrect typing. `ARRAY[NULL]` is now typed as `string[]` if the type cannot be otherwise inferred from the context. This is the same logic that PostgreSQL uses, thus improving compatibility in addition to fixing the internal error. [#59136][#59136] -- Fix a slow / hanging query that can be caused by using large `max_decimal_digits` on `ST_AsGeoJSON`. [#59165][#59165] -- Queries that attempt to retrieve just the key columns of a single system table row will no longer return erroneous values. [#58659][#58659] -- Fixed a bug in URL handling of HTTP external storage paths on Windows. [#59216][#59216] -- Previously, CockroachDB could crash when executing `ALTER INDEX ... SPLIT/UNSPLIT AT` when more values were provided than are explicitly specified in the index. This has been fixed. [#59213][#59213] -- Fixed a bug where multi-tenancy SQL pods would not successfully initialize the GEOS library. [#59259][#59259] -- Placeholder values are now included alongside statements in structured events. [#59110][#59110] -- Added qualification prefix for user-defined schema names in event logs. [#58617][#58617] -- Added qualification prefix for dropped views in event logs. [#59058][#59058] -- Parsing errors are no longer thrown when importing a `pgdump` file with array data. [#58244][#58244] -- Fixed a crash when creating backup schedules writing to GCS buckets. [#57617][#57617] -- CockroachDB now correctly exports `schedules_BACKUP_*` metrics as well as backup RPO metrics. [#57488][#57488] - -

Performance improvements

- -- Previously, CockroachDB when performing an unordered `DISTINCT` operation via the [vectorized execution engine](https://www.cockroachlabs.com/docs/v21.1/vectorized-execution) would buffer up all tuples from the input, which is a suboptimal behavior when the query has a `LIMIT` clause. This has now been fixed. This behavior was introduced in v20.1. Note that the row-by-row engine doesn't have this issue. [#57579][#57579] -- The query optimizer can use filters that constrain columns to multiple constant values to generate lookup joins. For example, a join filter `x.a = y.a AND y.b IN (1, 2)` can be used to generate a lookup join on table `y` assuming that it has an index on `(a, b)` or `(b, a)`. [#57690][#57690] -- The query optimizer now explores plans with lookup joins on partitioned indexes, resulting in more efficient query plans in some cases. [#57690][#57690] -- Potentially improved performance for [`UPDATE`](https://www.cockroachlabs.com/docs/v21.1/update) statements where the table has computed columns that do not depend on updated columns. [#58188][#58188] -- CockroachDB now allows the storage engine to compact `sstables` based on reads, on read-heavy workloads. [#58247][#58247] -- Partial indexes with `IS NOT NULL` predicates can be used in cases where `JOIN` filters implicitly imply the predicate. This results in more efficient query plans for `JOIN`s and foreign checks. [#58204][#58204] -- Queries that use a geospatial GIN index can now take advantage of [vectorized execution](https://www.cockroachlabs.com/docs/v21.1/vectorized-execution) for some parts of the query plan, resulting in improved performance. [#58241][#58241] -- Previously, indexed columns of partial indexes were always fetched for [`UPDATE`s](https://www.cockroachlabs.com/docs/v21.1/update) and [`UPSERT`s](https://www.cockroachlabs.com/docs/v21.1/upsert). Now they are only fetched if they are required for maintaining the state of the index. If an `UPDATE` or `UPSERT` mutates columns that are neither indexed by a partial index nor referenced in a partial index predicate, they will no longer be fetched (assuming that they are not needed to maintain the state of other indexes, including the primary index). [#58358][#58358] -- [`UPDATE`](https://www.cockroachlabs.com/docs/v21.1/update) operations on tables with partial indexes no longer evaluate partial index predicate expressions when it is guaranteed that the operation will not alter the state of the partial index. In some cases, this can eliminate fetching the existing value of columns that are referenced in partial index predicates. [#58358][#58358] -- The `sum_int` aggregate function is now evaluated more efficiently in a distributed setting. [#58345][#58345] -- [`INSERT ... ON CONFLICT ... DO NOTHING`](https://www.cockroachlabs.com/docs/v21.1/insert) statements now use anti-joins for detecting conflicts. This simplifies the query plan for these statements, which may result in more efficient execution. [#58679][#58679] -- Improved the accuracy of histogram calculations for the following types: `string`/`uuid`/`inet` family. Additionally, support for `time`/`timetz` histogram calculations was also added. This improves optimizer's estimates and results in better query plans in certain instances. [#55797][#55797] -- The optimizer now uses collected histogram statistics to better estimate the cost of JSON and ARRAY GIN index scans, which may lead to more efficient query plans. [#59326][#59326] - -

Build changes

- -- CockroachDB now builds on Ubuntu 20.10 and other distros using `gcc-10`. [#58895][#58895] {% comment %}doc{% endcomment %} - -

Doc updates

- -- The types of logging sinks available through configuration are now [automatically documented](https://github.com/cockroachdb/cockroach/blob/master/docs/generated/logsinks.md). [#59083][#59083] -- The various output formats available for logging configurations are now documented. See the generated [reference documentation](https://github.com/cockroachdb/cockroach/blob/master/docs/generated/logformats.md) for details. [#58075][#58075] -- The cluster event logging system has been standardized. [Reference documentation](https://github.com/cockroachdb/cockroach/blob/master/docs/generated/eventlog.md) is now available (auto-generated from source code); changes to non-reserved payloads will now be announced at least one release version in advance. The event types are organized into broad categories: SQL Logical Schema Changes, SQL Privilege Changes, SQL User Management, CLuster-level events and SQL Miscellaneous operations. [#57737][#57737] -- A report of the possible logging severities and channels is now [automatically generated](https://github.com/cockroachdb/cockroach/blob/master/docs/generated/logging.md). [#57134][#57134] - -
- -

Contributors

- -This release includes 615 merged PRs by 85 authors. We would like to thank the following contributors from the CockroachDB community: - -- Alan Acosta (first-time contributor) -- ArjunM98 -- Azdim Zul Fahmi -- Cheng Jing (first-time contributor) -- Cyrus Javan -- Erik Grinaker -- Javier Fernandez-Ivern (first-time contributor) -- Kumar Akshay (first-time contributor) -- Maciej Rzasa (first-time contributor) -- Marcin Knychała -- Max Neverov -- Miguel Novelo (first-time contributor) -- Omar Bahareth (first-time contributor) -- Petr Jediný -- Ruixin Bao -- Saif Al-Harthi (first-time contributor) -- Vaibhav -- Yanxin (first-time contributor) -- b41sh (first-time contributor) -- mosquito2333 (first-time contributor) -- neeral - -
- -[#51987]: https://github.com/cockroachdb/cockroach/pull/51987 -[#52745]: https://github.com/cockroachdb/cockroach/pull/52745 -[#53974]: https://github.com/cockroachdb/cockroach/pull/53974 -[#55065]: https://github.com/cockroachdb/cockroach/pull/55065 -[#55713]: https://github.com/cockroachdb/cockroach/pull/55713 -[#55797]: https://github.com/cockroachdb/cockroach/pull/55797 -[#56329]: https://github.com/cockroachdb/cockroach/pull/56329 -[#56473]: https://github.com/cockroachdb/cockroach/pull/56473 -[#56496]: https://github.com/cockroachdb/cockroach/pull/56496 -[#56529]: https://github.com/cockroachdb/cockroach/pull/56529 -[#56630]: https://github.com/cockroachdb/cockroach/pull/56630 -[#56663]: https://github.com/cockroachdb/cockroach/pull/56663 -[#56735]: https://github.com/cockroachdb/cockroach/pull/56735 -[#56740]: https://github.com/cockroachdb/cockroach/pull/56740 -[#56843]: https://github.com/cockroachdb/cockroach/pull/56843 -[#56866]: https://github.com/cockroachdb/cockroach/pull/56866 -[#56980]: https://github.com/cockroachdb/cockroach/pull/56980 -[#57130]: https://github.com/cockroachdb/cockroach/pull/57130 -[#57134]: https://github.com/cockroachdb/cockroach/pull/57134 -[#57136]: https://github.com/cockroachdb/cockroach/pull/57136 -[#57146]: https://github.com/cockroachdb/cockroach/pull/57146 -[#57171]: https://github.com/cockroachdb/cockroach/pull/57171 -[#57195]: https://github.com/cockroachdb/cockroach/pull/57195 -[#57224]: https://github.com/cockroachdb/cockroach/pull/57224 -[#57240]: https://github.com/cockroachdb/cockroach/pull/57240 -[#57244]: https://github.com/cockroachdb/cockroach/pull/57244 -[#57250]: https://github.com/cockroachdb/cockroach/pull/57250 -[#57272]: https://github.com/cockroachdb/cockroach/pull/57272 -[#57274]: https://github.com/cockroachdb/cockroach/pull/57274 -[#57275]: https://github.com/cockroachdb/cockroach/pull/57275 -[#57295]: https://github.com/cockroachdb/cockroach/pull/57295 -[#57327]: https://github.com/cockroachdb/cockroach/pull/57327 -[#57337]: https://github.com/cockroachdb/cockroach/pull/57337 -[#57349]: https://github.com/cockroachdb/cockroach/pull/57349 -[#57354]: https://github.com/cockroachdb/cockroach/pull/57354 -[#57380]: https://github.com/cockroachdb/cockroach/pull/57380 -[#57382]: https://github.com/cockroachdb/cockroach/pull/57382 -[#57387]: https://github.com/cockroachdb/cockroach/pull/57387 -[#57390]: https://github.com/cockroachdb/cockroach/pull/57390 -[#57412]: https://github.com/cockroachdb/cockroach/pull/57412 -[#57419]: https://github.com/cockroachdb/cockroach/pull/57419 -[#57420]: https://github.com/cockroachdb/cockroach/pull/57420 -[#57423]: https://github.com/cockroachdb/cockroach/pull/57423 -[#57477]: https://github.com/cockroachdb/cockroach/pull/57477 -[#57488]: https://github.com/cockroachdb/cockroach/pull/57488 -[#57498]: https://github.com/cockroachdb/cockroach/pull/57498 -[#57500]: https://github.com/cockroachdb/cockroach/pull/57500 -[#57513]: https://github.com/cockroachdb/cockroach/pull/57513 -[#57519]: https://github.com/cockroachdb/cockroach/pull/57519 -[#57533]: https://github.com/cockroachdb/cockroach/pull/57533 -[#57542]: https://github.com/cockroachdb/cockroach/pull/57542 -[#57567]: https://github.com/cockroachdb/cockroach/pull/57567 -[#57568]: https://github.com/cockroachdb/cockroach/pull/57568 -[#57579]: https://github.com/cockroachdb/cockroach/pull/57579 -[#57583]: https://github.com/cockroachdb/cockroach/pull/57583 -[#57587]: https://github.com/cockroachdb/cockroach/pull/57587 -[#57598]: https://github.com/cockroachdb/cockroach/pull/57598 -[#57609]: https://github.com/cockroachdb/cockroach/pull/57609 -[#57615]: https://github.com/cockroachdb/cockroach/pull/57615 -[#57617]: https://github.com/cockroachdb/cockroach/pull/57617 -[#57618]: https://github.com/cockroachdb/cockroach/pull/57618 -[#57644]: https://github.com/cockroachdb/cockroach/pull/57644 -[#57653]: https://github.com/cockroachdb/cockroach/pull/57653 -[#57654]: https://github.com/cockroachdb/cockroach/pull/57654 -[#57656]: https://github.com/cockroachdb/cockroach/pull/57656 -[#57666]: https://github.com/cockroachdb/cockroach/pull/57666 -[#57690]: https://github.com/cockroachdb/cockroach/pull/57690 -[#57697]: https://github.com/cockroachdb/cockroach/pull/57697 -[#57713]: https://github.com/cockroachdb/cockroach/pull/57713 -[#57724]: https://github.com/cockroachdb/cockroach/pull/57724 -[#57730]: https://github.com/cockroachdb/cockroach/pull/57730 -[#57733]: https://github.com/cockroachdb/cockroach/pull/57733 -[#57737]: https://github.com/cockroachdb/cockroach/pull/57737 -[#57745]: https://github.com/cockroachdb/cockroach/pull/57745 -[#57749]: https://github.com/cockroachdb/cockroach/pull/57749 -[#57776]: https://github.com/cockroachdb/cockroach/pull/57776 -[#57804]: https://github.com/cockroachdb/cockroach/pull/57804 -[#57811]: https://github.com/cockroachdb/cockroach/pull/57811 -[#57812]: https://github.com/cockroachdb/cockroach/pull/57812 -[#57817]: https://github.com/cockroachdb/cockroach/pull/57817 -[#57824]: https://github.com/cockroachdb/cockroach/pull/57824 -[#57828]: https://github.com/cockroachdb/cockroach/pull/57828 -[#57836]: https://github.com/cockroachdb/cockroach/pull/57836 -[#57837]: https://github.com/cockroachdb/cockroach/pull/57837 -[#57839]: https://github.com/cockroachdb/cockroach/pull/57839 -[#57851]: https://github.com/cockroachdb/cockroach/pull/57851 -[#57877]: https://github.com/cockroachdb/cockroach/pull/57877 -[#57879]: https://github.com/cockroachdb/cockroach/pull/57879 -[#57927]: https://github.com/cockroachdb/cockroach/pull/57927 -[#57954]: https://github.com/cockroachdb/cockroach/pull/57954 -[#57966]: https://github.com/cockroachdb/cockroach/pull/57966 -[#57969]: https://github.com/cockroachdb/cockroach/pull/57969 -[#57975]: https://github.com/cockroachdb/cockroach/pull/57975 -[#57991]: https://github.com/cockroachdb/cockroach/pull/57991 -[#57994]: https://github.com/cockroachdb/cockroach/pull/57994 -[#58014]: https://github.com/cockroachdb/cockroach/pull/58014 -[#58034]: https://github.com/cockroachdb/cockroach/pull/58034 -[#58043]: https://github.com/cockroachdb/cockroach/pull/58043 -[#58045]: https://github.com/cockroachdb/cockroach/pull/58045 -[#58053]: https://github.com/cockroachdb/cockroach/pull/58053 -[#58070]: https://github.com/cockroachdb/cockroach/pull/58070 -[#58072]: https://github.com/cockroachdb/cockroach/pull/58072 -[#58075]: https://github.com/cockroachdb/cockroach/pull/58075 -[#58078]: https://github.com/cockroachdb/cockroach/pull/58078 -[#58084]: https://github.com/cockroachdb/cockroach/pull/58084 -[#58087]: https://github.com/cockroachdb/cockroach/pull/58087 -[#58088]: https://github.com/cockroachdb/cockroach/pull/58088 -[#58117]: https://github.com/cockroachdb/cockroach/pull/58117 -[#58118]: https://github.com/cockroachdb/cockroach/pull/58118 -[#58120]: https://github.com/cockroachdb/cockroach/pull/58120 -[#58121]: https://github.com/cockroachdb/cockroach/pull/58121 -[#58125]: https://github.com/cockroachdb/cockroach/pull/58125 -[#58126]: https://github.com/cockroachdb/cockroach/pull/58126 -[#58128]: https://github.com/cockroachdb/cockroach/pull/58128 -[#58130]: https://github.com/cockroachdb/cockroach/pull/58130 -[#58140]: https://github.com/cockroachdb/cockroach/pull/58140 -[#58146]: https://github.com/cockroachdb/cockroach/pull/58146 -[#58153]: https://github.com/cockroachdb/cockroach/pull/58153 -[#58161]: https://github.com/cockroachdb/cockroach/pull/58161 -[#58173]: https://github.com/cockroachdb/cockroach/pull/58173 -[#58188]: https://github.com/cockroachdb/cockroach/pull/58188 -[#58191]: https://github.com/cockroachdb/cockroach/pull/58191 -[#58192]: https://github.com/cockroachdb/cockroach/pull/58192 -[#58204]: https://github.com/cockroachdb/cockroach/pull/58204 -[#58208]: https://github.com/cockroachdb/cockroach/pull/58208 -[#58212]: https://github.com/cockroachdb/cockroach/pull/58212 -[#58219]: https://github.com/cockroachdb/cockroach/pull/58219 -[#58221]: https://github.com/cockroachdb/cockroach/pull/58221 -[#58233]: https://github.com/cockroachdb/cockroach/pull/58233 -[#58241]: https://github.com/cockroachdb/cockroach/pull/58241 -[#58244]: https://github.com/cockroachdb/cockroach/pull/58244 -[#58247]: https://github.com/cockroachdb/cockroach/pull/58247 -[#58251]: https://github.com/cockroachdb/cockroach/pull/58251 -[#58254]: https://github.com/cockroachdb/cockroach/pull/58254 -[#58257]: https://github.com/cockroachdb/cockroach/pull/58257 -[#58265]: https://github.com/cockroachdb/cockroach/pull/58265 -[#58288]: https://github.com/cockroachdb/cockroach/pull/58288 -[#58292]: https://github.com/cockroachdb/cockroach/pull/58292 -[#58305]: https://github.com/cockroachdb/cockroach/pull/58305 -[#58306]: https://github.com/cockroachdb/cockroach/pull/58306 -[#58317]: https://github.com/cockroachdb/cockroach/pull/58317 -[#58345]: https://github.com/cockroachdb/cockroach/pull/58345 -[#58349]: https://github.com/cockroachdb/cockroach/pull/58349 -[#58354]: https://github.com/cockroachdb/cockroach/pull/58354 -[#58358]: https://github.com/cockroachdb/cockroach/pull/58358 -[#58359]: https://github.com/cockroachdb/cockroach/pull/58359 -[#58381]: https://github.com/cockroachdb/cockroach/pull/58381 -[#58399]: https://github.com/cockroachdb/cockroach/pull/58399 -[#58423]: https://github.com/cockroachdb/cockroach/pull/58423 -[#58431]: https://github.com/cockroachdb/cockroach/pull/58431 -[#58433]: https://github.com/cockroachdb/cockroach/pull/58433 -[#58434]: https://github.com/cockroachdb/cockroach/pull/58434 -[#58452]: https://github.com/cockroachdb/cockroach/pull/58452 -[#58454]: https://github.com/cockroachdb/cockroach/pull/58454 -[#58466]: https://github.com/cockroachdb/cockroach/pull/58466 -[#58470]: https://github.com/cockroachdb/cockroach/pull/58470 -[#58472]: https://github.com/cockroachdb/cockroach/pull/58472 -[#58487]: https://github.com/cockroachdb/cockroach/pull/58487 -[#58495]: https://github.com/cockroachdb/cockroach/pull/58495 -[#58504]: https://github.com/cockroachdb/cockroach/pull/58504 -[#58612]: https://github.com/cockroachdb/cockroach/pull/58612 -[#58616]: https://github.com/cockroachdb/cockroach/pull/58616 -[#58617]: https://github.com/cockroachdb/cockroach/pull/58617 -[#58659]: https://github.com/cockroachdb/cockroach/pull/58659 -[#58671]: https://github.com/cockroachdb/cockroach/pull/58671 -[#58679]: https://github.com/cockroachdb/cockroach/pull/58679 -[#58716]: https://github.com/cockroachdb/cockroach/pull/58716 -[#58725]: https://github.com/cockroachdb/cockroach/pull/58725 -[#58740]: https://github.com/cockroachdb/cockroach/pull/58740 -[#58743]: https://github.com/cockroachdb/cockroach/pull/58743 -[#58747]: https://github.com/cockroachdb/cockroach/pull/58747 -[#58748]: https://github.com/cockroachdb/cockroach/pull/58748 -[#58749]: https://github.com/cockroachdb/cockroach/pull/58749 -[#58888]: https://github.com/cockroachdb/cockroach/pull/58888 -[#58891]: https://github.com/cockroachdb/cockroach/pull/58891 -[#58894]: https://github.com/cockroachdb/cockroach/pull/58894 -[#58895]: https://github.com/cockroachdb/cockroach/pull/58895 -[#58898]: https://github.com/cockroachdb/cockroach/pull/58898 -[#58903]: https://github.com/cockroachdb/cockroach/pull/58903 -[#58928]: https://github.com/cockroachdb/cockroach/pull/58928 -[#58937]: https://github.com/cockroachdb/cockroach/pull/58937 -[#58947]: https://github.com/cockroachdb/cockroach/pull/58947 -[#58987]: https://github.com/cockroachdb/cockroach/pull/58987 -[#58988]: https://github.com/cockroachdb/cockroach/pull/58988 -[#58995]: https://github.com/cockroachdb/cockroach/pull/58995 -[#59002]: https://github.com/cockroachdb/cockroach/pull/59002 -[#59008]: https://github.com/cockroachdb/cockroach/pull/59008 -[#59009]: https://github.com/cockroachdb/cockroach/pull/59009 -[#59028]: https://github.com/cockroachdb/cockroach/pull/59028 -[#59049]: https://github.com/cockroachdb/cockroach/pull/59049 -[#59058]: https://github.com/cockroachdb/cockroach/pull/59058 -[#59083]: https://github.com/cockroachdb/cockroach/pull/59083 -[#59087]: https://github.com/cockroachdb/cockroach/pull/59087 -[#59110]: https://github.com/cockroachdb/cockroach/pull/59110 -[#59121]: https://github.com/cockroachdb/cockroach/pull/59121 -[#59136]: https://github.com/cockroachdb/cockroach/pull/59136 -[#59144]: https://github.com/cockroachdb/cockroach/pull/59144 -[#59147]: https://github.com/cockroachdb/cockroach/pull/59147 -[#59165]: https://github.com/cockroachdb/cockroach/pull/59165 -[#59178]: https://github.com/cockroachdb/cockroach/pull/59178 -[#59206]: https://github.com/cockroachdb/cockroach/pull/59206 -[#59212]: https://github.com/cockroachdb/cockroach/pull/59212 -[#59213]: https://github.com/cockroachdb/cockroach/pull/59213 -[#59216]: https://github.com/cockroachdb/cockroach/pull/59216 -[#59223]: https://github.com/cockroachdb/cockroach/pull/59223 -[#59234]: https://github.com/cockroachdb/cockroach/pull/59234 -[#59248]: https://github.com/cockroachdb/cockroach/pull/59248 -[#59256]: https://github.com/cockroachdb/cockroach/pull/59256 -[#59259]: https://github.com/cockroachdb/cockroach/pull/59259 -[#59313]: https://github.com/cockroachdb/cockroach/pull/59313 -[#59319]: https://github.com/cockroachdb/cockroach/pull/59319 -[#59326]: https://github.com/cockroachdb/cockroach/pull/59326 -[#59350]: https://github.com/cockroachdb/cockroach/pull/59350 diff --git a/src/current/_includes/releases/v21.1/v21.1.0-alpha.3.md b/src/current/_includes/releases/v21.1/v21.1.0-alpha.3.md deleted file mode 100644 index c70df2657bd..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.0-alpha.3.md +++ /dev/null @@ -1,158 +0,0 @@ -## v21.1.0-alpha.3 - -Release Date: February 8, 2021 - - - -

General changes

- -- Added ability to further debug connections shut down automatically by the server. [#59460][#59460] - -

SQL language changes

- -- Fixed up [`ALTER TABLE ... ADD CONSTRAINT ... UNIQUE`](https://www.cockroachlabs.com/docs/v21.1/add-constraint) to partition correctly under a [`PARTITION ALL BY`](https://www.cockroachlabs.com/docs/v21.1/partition-by) table. [#59364][#59364] -- CockroachDB now applies zone configs to new [unique constraints](https://www.cockroachlabs.com/docs/v21.1/unique) in `REGIONAL BY ROW` tables. [#59364][#59364] -- A new, unused field called `global_reads` was added to [zone configurations](https://www.cockroachlabs.com/docs/v21.1/configure-replication-zones). The field does not yet have any effect. [#59304][#59304] -- A new [private cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) `sql.distsql.temp_storage.hash_agg.enabled` was added that allows users to disable the disk spilling capability of the hash aggregation in the [vectorized execution engine](https://www.cockroachlabs.com/docs/v21.1/vectorized-execution). It is `true` by default which incurs some performance hit. Setting it to `false` will improve the performance, but the queries might hit an out-of-memory limit error. [#59414][#59414] -- The optimizer now enforces a unique constraint on the explicit index columns for implicitly partitioned unique indexes and unique indexes in [`REGIONAL BY ROW` tables](https://www.cockroachlabs.com/docs/v21.1/create-table). An attempt to insert or update a row such that the unique constraint is violated will result in an error. [#59501][#59501] -- [`REGIONAL BY ROW` tables](https://www.cockroachlabs.com/docs/v21.1/create-table) now preserve [zone configurations](https://www.cockroachlabs.com/docs/v21.1/configure-replication-zones) when using [`ALTER PRIMARY KEY`](https://www.cockroachlabs.com/docs/v21.1/alter-primary-key). [#59365][#59365] -- Implemented [`ALTER TABLE ... LOCALITY REGIONAL BY TABLE`](https://www.cockroachlabs.com/docs/v21.1/alter-table) from `LOCALITY GLOBAL`. [#59407][#59407] -- [Zone configs](https://www.cockroachlabs.com/docs/v21.1/configure-replication-zones) now support new attributes `num_voters` and `voter_constraints`. `num_voters` will specify the number of [voting replicas](https://www.cockroachlabs.com/docs/v21.1/architecture/life-of-a-distributed-transaction#consensus). When `num_voters` is explicitly specified, `num_replicas` will be the sum of voting and non-voting replicas. `voter_constraints` will specify the constraints that govern the placement of just the voting replicas, whereas the existing `constraints` attributes will govern the placement of all replicas (voting as well as non-voting). [#57184][#57184] -- Added SQL syntax for [`RESTORE tenant x FROM REPLICATION STREAM FROM 'replication_stream'`](https://www.cockroachlabs.com/docs/v21.1/restore). This allows the user to start an ingestion job to ingest KVs from the replication stream into the destination tenant's keyspace. [#59112][#59112] -- The [`SHOW CREATE`](https://www.cockroachlabs.com/docs/v21.1/show-create) statement now lists [`ALTER PARTITION`](https://www.cockroachlabs.com/docs/v21.1/alter-partition) statements sorted by the partition name and index_name. [#59580][#59580] -- Error messages for cross-database links now include a hint directing to the user to the [deprecation docs](https://www.cockroachlabs.com/docs/v21.1/cluster-settings). An example message looks like: ``` ERROR: the view cannot refer to other databases; (see the 'sql.cross_db_views.enabled' cluster setting) SQLSTATE: 0A000 HINT: Note that cross database references will be removed in future releases. See: https://www.cockroachlabs.com/docs/releases/v21.1.html#v21-1-0-deprecations ``` [#59551][#59551] -- The `escape_string_warning` session variable from PostgreSQL was added with [compatibility-only](https://www.cockroachlabs.com/docs/v21.1/postgresql-compatibility) support. It defaults to `on` and cannot be changed. [#59479][#59479] -- Multi-column [GIN indexes](https://www.cockroachlabs.com/docs/v21.1/create-index#create-gin-indexes) can now be created. The last indexed column must be inverted types such as [`JSON`](https://www.cockroachlabs.com/docs/v21.1/jsonb), [`ARRAY`](https://www.cockroachlabs.com/docs/v21.1/array), [`GEOMETRY`, and `GEOGRAPHY`](https://www.cockroachlabs.com/docs/v21.1/spatial-data). All preceding columns must have types that are indexable. These indexes may be used for queries that constrain all index columns. [#59565][#59565] -- Added `WITH full_table_name` option to create a changefeed on `movr.public.drivers` instead of `drivers`. [#59258][#59258] -- [`UPSERT`s](https://www.cockroachlabs.com/docs/v21.1/upsert) on tables with an implicitly partitioned primary index now use only the explicit primary key columns as the conflict columns, excluding all implicit partitioning columns. This also applies to `REGIONAL BY ROW` tables, ensuring that the `crdb_region` column is not included in the `UPSERT` key. [#59654][#59654] - -

Command-line changes

- -- The [`cockroach`](https://www.cockroachlabs.com/docs/v21.1/cockroach-commands) command now supports the command-line parameter `--version` which reports its version parameters. This makes `cockroach --version` equivalent to `cockroach version`. [#58665][#58665] -- The [`cockroach version`](https://www.cockroachlabs.com/docs/v21.1/cockroach-version) command now supports a new parameter `--build-tag`; when specified, it displays the technical build tag, which makes it possible to integrate with automated deployment tools. [#58665][#58665] -- The `channels` parameter for the log sink [configurations](https://github.com/cockroachdb/cockroach/blob/master/docs/generated/logsinks.md) now supports a more flexible input configuration format: - Syntax to include specific channels: - `channels: dev,sessions` (yaml string) - `channels: 'dev,sessions'` (yaml string, equivalent to previous) - `channels: [dev,sessions]` (yaml array, can use multi-line syntax with hyphens too) - `channels: ['dev','sessions']` (same as previous) - `channels: '[dev,sessions]'` (bracket-enclosed list inside yaml string) - - Syntax to include all channels: - `channels: all` (yaml string) - `channels: 'all'` (same as previous) - `channels: [all]` (yaml array) - `channels: ['all']` (same as previous) - `channels: '[all]'` (bracket-enclosed "all" inside yaml string) - - Syntax to include all channels except some: - `channels: all except dev,sessions` (yaml string) - `channels: 'all except dev,sessions'` (same as previous, quoted string) - `channels: 'all except [dev,sessions]'` (same as previous, list is bracket enclosed) - - For example: ``` sinks: stderr: channels: - DEV - SQL_SESSIONS ``` uses the "native" YAML syntax for lists. [#59352][#59352] - -- The notification that `SIGHUP` was received, and that log files are flushed to disk as a result, is now sent to the [OPS logging channel](https://github.com/cockroachdb/cockroach/blob/master/docs/generated/logging.md#ops). [#59345][#59345] -- The notification that `SIGHUP` was received, and that TLS certificates are reloaded from disk as a result, is now sent to the OPS logging channel as a structured event. Refer to the [reference docs](https://github.com/cockroachdb/cockroach/blob/master/docs/generated/logformats.md) for details about the event payload. [#59345][#59345] - -

API endpoint changes

- -- Added a new API tree, in `/api/v2/*`, currently undocumented, that avoids the use of cookie-based authentication in favor of sessions in headers, and support for pagination. [#58436][#58436] - -

DB Console changes

- -- Updated the [table details page](https://www.cockroachlabs.com/docs/v21.1/ui-databases-page#table-details) to show table-specific zone configuration values when set, show constraints and lease preferences, and display a valid statement to re-configure zone configuration for that table. [#59196][#59196] -- Users can now see the time series of full table or index scans in the [Advanced Debug page](https://www.cockroachlabs.com/docs/v21.1/ui-debug-pages). [#59261][#59261] -- The `sql.leases.active` gauge is now available to track the outstanding descriptor leases in the [cluster](https://www.cockroachlabs.com/docs/v21.1/ui-cluster-overview-page). [#57561][#57561] -- Long queries are truncated in the DB Console. [#59603][#59603] - -

Bug fixes

- -- Added event logs for [`SET SCHEMA`](https://www.cockroachlabs.com/docs/v21.1/set-schema) statements. [#58737][#58737] -- Fixed an issue where a [left inverted join](https://www.cockroachlabs.com/docs/v21.1/joins) could have incorrect results. In particular, some output rows could have non-`NULL` values for right-side columns when the right-side columns should have been `NULL`. This issue has only existed in alpha releases of 21.1 so far, and it is now fixed. [#59279][#59279] -- Fixed a panic where type hints mismatching placeholder names cause a crash. [#59450][#59450] -- Unexpected internal errors containing stack traces that reference a `countingWriter` null pointer have now been fixed. [#59477][#59477] -- Fixed a bug introduced in v20.1 in the DB Console where incorrect zone configuration values were shown on the table details page and constraints and lease preferences were not displayed. [#59196][#59196] -- [Changefeeds](https://www.cockroachlabs.com/docs/v21.1/create-changefeed) no longer error with "1e2 is not roundtrippable at scale 2" when 100 is stored in a column with width 2. [#59075][#59075] -- Fixed a bug which prevented [renaming a column](https://www.cockroachlabs.com/docs/v21.1/rename-column) that was referenced earlier in a transaction as part of a computed expression, index predicate, check expression, or not null constraint. [#59384][#59384] -- Added event logs for privilege changes in `crdb_internal.unsafe_xxx`. [#59282][#59282] -- Fixed a bug in which incorrect results could be returned for [left and anti joins](https://www.cockroachlabs.com/docs/v21.1/joins). This could happen when one of the columns on one side of the join was constrained to multiple constant values, either due to a check constraint or an `IN` clause. The bug resulted in non-matching input rows getting returned multiple times in the output, which is incorrect. This bug only affected previous alpha releases of 21.1, and has now been fixed. [#59646][#59646] -- Fixed [`EXPLAIN ANALYZE (DEBUG)`](https://www.cockroachlabs.com/docs/v21.1/explain-analyze) interaction with the SQL shell. [#59557][#59557] -- Fixed a bug preventing [foreign key constraints](https://www.cockroachlabs.com/docs/v21.1/foreign-key) referencing hidden columns (e.g., `rowid`) from being added. [#59659][#59659] -- Fixed a bug where [`DROP SCHEMA ... CASCADE`](https://www.cockroachlabs.com/docs/v21.1/drop-schema) could result in types which are referenced being dropped. [#59281][#59281] -- Fixed a bug whereby [dropping a schema](https://www.cockroachlabs.com/docs/v21.1/drop-schema) with a table that used a [user-defined type](https://www.cockroachlabs.com/docs/v21.1/create-type) which was not being dropped (because it is in a different schema) would result in a descriptor corruption due to a dangling back-reference to a dropped table on the type descriptor. [#59281][#59281] - -

Performance improvements

- -- The [query optimizer](https://www.cockroachlabs.com/docs/v21.1/cost-based-optimizer) now plans scans over GIN indexes on [`JSON`](https://www.cockroachlabs.com/docs/v21.1/jsonb) columns for query filters that constrain the `JSON` column with equality and fetch value operators (`->`) inside conjunctions and disjunctions, like `j->'a' = '1' AND j->'b' = '2'`. [#59266][#59266] -- Fixed a bug included in 20.2.1 for the [`JSON`](https://www.cockroachlabs.com/docs/v21.1/jsonb) fetch value operator, `->` which resulted in chained `->` operators in query filters not being index accelerated (e.g., `j->'a'->'b' = '1'`). Chained `->` operators are now index accelerated. [#59494][#59494] -- Improved the allocation performance of workloads that use the [`EXTRACT`](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators) built-in. [#59598][#59598] - -
- -

Contributors

- -This release includes 116 merged PRs by 42 authors. -We would like to thank the following contributors from the CockroachDB community: - -- John Seekins (first-time contributor) -- Ulf Adams (first-time contributor) - -
- -[#57184]: https://github.com/cockroachdb/cockroach/pull/57184 -[#57561]: https://github.com/cockroachdb/cockroach/pull/57561 -[#58436]: https://github.com/cockroachdb/cockroach/pull/58436 -[#58665]: https://github.com/cockroachdb/cockroach/pull/58665 -[#58737]: https://github.com/cockroachdb/cockroach/pull/58737 -[#58863]: https://github.com/cockroachdb/cockroach/pull/58863 -[#58904]: https://github.com/cockroachdb/cockroach/pull/58904 -[#59023]: https://github.com/cockroachdb/cockroach/pull/59023 -[#59026]: https://github.com/cockroachdb/cockroach/pull/59026 -[#59075]: https://github.com/cockroachdb/cockroach/pull/59075 -[#59112]: https://github.com/cockroachdb/cockroach/pull/59112 -[#59196]: https://github.com/cockroachdb/cockroach/pull/59196 -[#59258]: https://github.com/cockroachdb/cockroach/pull/59258 -[#59261]: https://github.com/cockroachdb/cockroach/pull/59261 -[#59266]: https://github.com/cockroachdb/cockroach/pull/59266 -[#59279]: https://github.com/cockroachdb/cockroach/pull/59279 -[#59281]: https://github.com/cockroachdb/cockroach/pull/59281 -[#59282]: https://github.com/cockroachdb/cockroach/pull/59282 -[#59304]: https://github.com/cockroachdb/cockroach/pull/59304 -[#59345]: https://github.com/cockroachdb/cockroach/pull/59345 -[#59352]: https://github.com/cockroachdb/cockroach/pull/59352 -[#59364]: https://github.com/cockroachdb/cockroach/pull/59364 -[#59365]: https://github.com/cockroachdb/cockroach/pull/59365 -[#59384]: https://github.com/cockroachdb/cockroach/pull/59384 -[#59395]: https://github.com/cockroachdb/cockroach/pull/59395 -[#59407]: https://github.com/cockroachdb/cockroach/pull/59407 -[#59414]: https://github.com/cockroachdb/cockroach/pull/59414 -[#59450]: https://github.com/cockroachdb/cockroach/pull/59450 -[#59460]: https://github.com/cockroachdb/cockroach/pull/59460 -[#59474]: https://github.com/cockroachdb/cockroach/pull/59474 -[#59477]: https://github.com/cockroachdb/cockroach/pull/59477 -[#59479]: https://github.com/cockroachdb/cockroach/pull/59479 -[#59494]: https://github.com/cockroachdb/cockroach/pull/59494 -[#59501]: https://github.com/cockroachdb/cockroach/pull/59501 -[#59551]: https://github.com/cockroachdb/cockroach/pull/59551 -[#59557]: https://github.com/cockroachdb/cockroach/pull/59557 -[#59565]: https://github.com/cockroachdb/cockroach/pull/59565 -[#59580]: https://github.com/cockroachdb/cockroach/pull/59580 -[#59598]: https://github.com/cockroachdb/cockroach/pull/59598 -[#59603]: https://github.com/cockroachdb/cockroach/pull/59603 -[#59646]: https://github.com/cockroachdb/cockroach/pull/59646 -[#59654]: https://github.com/cockroachdb/cockroach/pull/59654 -[#59659]: https://github.com/cockroachdb/cockroach/pull/59659 -[088057a8f]: https://github.com/cockroachdb/cockroach/commit/088057a8f -[71de4f752]: https://github.com/cockroachdb/cockroach/commit/71de4f752 -[73b15ad5b]: https://github.com/cockroachdb/cockroach/commit/73b15ad5b -[893e3f68c]: https://github.com/cockroachdb/cockroach/commit/893e3f68c -[b94aad66d]: https://github.com/cockroachdb/cockroach/commit/b94aad66d -[c3f328eb5]: https://github.com/cockroachdb/cockroach/commit/c3f328eb5 -[c955e882e]: https://github.com/cockroachdb/cockroach/commit/c955e882e -[c9eafd522]: https://github.com/cockroachdb/cockroach/commit/c9eafd522 -[daf42d6b8]: https://github.com/cockroachdb/cockroach/commit/daf42d6b8 -[e2c147721]: https://github.com/cockroachdb/cockroach/commit/e2c147721 -[ea9074ba7]: https://github.com/cockroachdb/cockroach/commit/ea9074ba7 -[f901ad7aa]: https://github.com/cockroachdb/cockroach/commit/f901ad7aa -[fa324020c]: https://github.com/cockroachdb/cockroach/commit/fa324020c diff --git a/src/current/_includes/releases/v21.1/v21.1.0-beta.1.md b/src/current/_includes/releases/v21.1/v21.1.0-beta.1.md deleted file mode 100644 index 33a7041207e..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.0-beta.1.md +++ /dev/null @@ -1,509 +0,0 @@ -## v21.1.0-beta.1 - -Release Date: March 22, 2021 - - - -

Backward-incompatible changes

- -- The [`cockroach debug ballast`](https://www.cockroachlabs.com/docs/v21.1/cockroach-debug-ballast) command now refuses to overwrite the target ballast file if it already exists. This change is intended to prevent mistaken uses of the `ballast` command by operators. Scripts that integrate `cockroach debug ballast` can consider adding a `rm` command. [#59995][#59995] {% comment %}doc{% endcomment %} -- Removed the `kv.atomic_replication_changes.enabled` [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings). All replication changes on a range now use joint-consensus. [#61170][#61170] {% comment %}doc{% endcomment %} - -

Security updates

- -- It is now possible to log SQL statements executed by admin users. This logging is enabled via the `sql.log.admin_audit.enabled` [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings). When set, events of type `admin_query` are logged to the `SENSITIVE_ACCESS` channel. [#60708][#60708] {% comment %}doc{% endcomment %} - -

General changes

- -- Updated the AWS SDK used to interface with AWS services such as S3 (for backup, restore, and import) and KMS (for backup/restore). [#59709][#59709] {% comment %}doc{% endcomment %} -- [`SHOW TRACE FOR SESSION`](https://www.cockroachlabs.com/docs/v21.1/show-trace) previously included CockroachDB internal traces for async threads kicked off as part of user operations. This trace data is no longer captured. [#59815][#59815] {% comment %}doc{% endcomment %} -- Crash reports that are sent to Cockroach Labs no longer redact the names of built-in virtual tables from the `crdb_internal`, `information_schema`, and `pg_catalog` schemas. [#60799][#60799] -- Raised the default limit on the maximum number of spans that can be protected by the `protectedts` subsystem. This limit can be configured using the `kv.protectedts.max_spans` [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings). [#61018][#61018] {% comment %}doc{% endcomment %} - -

Enterprise edition changes

- -- Added the `bulkio.stream_ingestion.minimum_flush_interval` [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings), which allows the user to control how often the stream ingestion job will flush. Note that the job may still flush more often if the in-memory buffers are filled. [#60160][#60160] {% comment %}doc{% endcomment %} -- CockroachDB now supports [primary key](https://www.cockroachlabs.com/docs/v21.1/primary-key) changes in [`CREATE CHANGEFEED`](https://www.cockroachlabs.com/docs/v21.1/create-changefeed). [#58422][#58422] {% comment %}doc{% endcomment %} -- [Multi-region database](https://www.cockroachlabs.com/docs/v21.1/movr-flask-database) creations are now permitted as long as the cluster has a CockroachDB subscription. [#61041][#61041] {% comment %}doc{% endcomment %} -- `ALTER DATABASE ... ADD REGION` now requires an enterprise license.[#61169][#61169] {% comment %}doc{% endcomment %} - -

SQL language changes

- -- Added a virtual table, `crdb_internal.node_inflight_trace_spans`, which exposes span ID, parent span ID, trace ID, start time, duration, and operation of node-local inflight spans. [#59492][#59492] {% comment %}doc{% endcomment %} -- Added `WITH avro_schema_prefix` option to Avro [changefeeds](https://www.cockroachlabs.com/docs/v21.1/create-changefeed) to prevent collisions in a shared schema registry [#59710][#59710] {% comment %}doc{% endcomment %} -- CockroachDB now allows implicitly partitioned indexes to be referenced by foreign keys using the non-implicit columns. [#59692][#59692] {% comment %}doc{% endcomment %} -- The [`SHOW ZONE CONFIGURATION`](https://www.cockroachlabs.com/docs/v21.1/show-zone-configurations) statement has been changed to use `FROM` instead of `FOR`. [#59410][#59410] {% comment %}doc{% endcomment %} -- It is now possible to use the `NOT VISIBLE` qualifier for new column definitions in [`CREATE TABLE`](https://www.cockroachlabs.com/docs/v21.1/create-table). This causes the column to become "hidden". Hidden columns are not considered when using `*` in `SELECT` clauses. Note that CockroachDB already supported hidden columns (e.g., `rowid`, which is added automatically when a table definition has no `PRIMARY KEY` constraint). This change adds the opportunity for end-users to define their own hidden columns. It is intended for use in combination with other features related to geo-partitioning introduced in v21.1, which offer more control about how geo-partitioning keys get exposed to client ORMs and other automated tools that inspect the SQL schema. [#58923][#58923] {% comment %}doc{% endcomment %} -- Added the `goroutine_id` column to the `crdb_internal.node_inflight_trace_spans` virtual table that represents the goroutine ID associated with a particular span. [#59717][#59717] {% comment %}doc{% endcomment %} -- Updated [`SHOW BACKUP ... WITH PRIVILEGES`](https://www.cockroachlabs.com/docs/v21.1/show-backup) to display ownership information of objects in the backup. [#59732][#59732] {% comment %}doc{% endcomment %} -- The [`ALTER TYPE ... DROP VALUE ...`](https://www.cockroachlabs.com/docs/v21.1/alter-type) statement has been added to drop values from an [`ENUM`](https://www.cockroachlabs.com/docs/v21.1/enum) type. The statement drops the provided value if it isn't used as a value in any table's row. The use of this syntax is gated behind the `sql_safe_updates` [session variable](https://www.cockroachlabs.com/docs/v21.1/set-vars), as it is susceptible to the value being used in a table expression (such as a computed column). [#58688][#58688] {% comment %}doc{% endcomment %} -- Added support for `COPY CSV`. [#59790][#59790] {% comment %}doc{% endcomment %} -- CockroachDB now supports specifying `NULL` and `DELIMITER` in `COPY`. [#59790][#59790] {% comment %}doc{% endcomment %} -- [`ALTER TABLE ... SET LOCALITY`](https://www.cockroachlabs.com/docs/v21.1/alter-table) adds the ability to change the locality of a `GLOBAL` or `REGIONAL BY TABLE` table to `REGIONAL BY ROW`, provided an existing column of the appropriate type already exists. [#59624][#59624] {% comment %}doc{% endcomment %} -- Overload sequence operators now accept a regclass. [#59396][#59396] -- `crdb_internal.invalid_objects` now includes invalid type descriptors. [#59978][#59978] {% comment %}doc{% endcomment %} -- Added the `finished` column to the virtual table `crdb_internal.node_inflight_trace_spans`, which represents whether each span has finished or not. [#59856][#59856] {% comment %}doc{% endcomment %} -- Partitioned [GIN indexes](https://www.cockroachlabs.com/docs/v21.1/inverted-indexes) are now supported. [#59858][#59858] {% comment %}doc{% endcomment %} -- Hidden columns (created by using `NOT VISIBLE` or implicitly created via hash sharded indexes, a lack of a primary key definition, or by using `REGIONAL BY ROW`) will now display with `NOT VISIBLE` annotations on [`SHOW CREATE TABLE`](https://www.cockroachlabs.com/docs/v21.1/show-create). [#59828][#59828] {% comment %}doc{% endcomment %} -- `REGIONAL BY ROW` tables that have an implicitly created `crdb_region` table will now mark the given column as hidden so it does not display in `SELECT *` statements. [#59831][#59831] {% comment %}doc{% endcomment %} -- CockroachDB now recognizes the `options` URL parameter. The `options` parameter specifies session variables to set at connection start. This is treated the same as defined in the [PostgreSQL docs](https://www.postgresql.org/docs/13/libpq-connect.html#LIBPQ-PARAMKEYWORDS). [#59621][#59621] {% comment %}doc{% endcomment %} -- Implemented the ability to `ALTER TABLE SET LOCALITY` to `REGIONAL BY ROW` without specifying `AS` for `GLOBAL` and `REGIONAL BY TABLE` tables. [#59824][#59824] {% comment %}doc{% endcomment %} -- Added the `sql.show_tables.estimated_row_count.enabled` [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings), which defaults to `true`. If `false`, `estimated_row_count` will not display on `SHOW TABLES` which improves performance. [#60282][#60282] {% comment %}doc{% endcomment %} -- [`ALTER DATABASE ... DROP REGION`](https://www.cockroachlabs.com/docs/v21.1/alter-database) is now implemented. [#59989][#59989] {% comment %}doc{% endcomment %} -- When a connection is established, CockroachDB will now return a placeholder `BackendKeyData` message in the response. This is for compatibility with some tools, but using `BackendKeyData` to cancel a query will still have no effect (which is the same as before). [#60281][#60281] {% comment %}doc{% endcomment %} -- To match the behavior of `PRIMARY KEY` creations setting, all relevant fields to `NOT NULL`, all `PARTITION BY` / R`EGIONAL BY ROW` columns will now have their fields set to `NOT NULL` at [`CREATE TABLE`](https://www.cockroachlabs.com/docs/v21.1/create-table) time. [#60379][#60379] {% comment %}doc{% endcomment %} -- [`ALTER DATABASE ... DROP REGION`](https://www.cockroachlabs.com/docs/v21.1/alter-database) now allows for dropping the primary region. [#60437][#60437] {% comment %}doc{% endcomment %} -- Added support for [restore](https://www.cockroachlabs.com/docs/v21.1/restore) of multi-region databases. [#60257][#60257] {% comment %}doc{% endcomment %} -- Implemented functionality to [`ALTER TABLE SET LOCALITY`](https://www.cockroachlabs.com/docs/v21.1/alter-table) from `REGIONAL BY ROW` to `REGIONAL BY TABLE` or `GLOBAL`. [#60311][#60311] {% comment %}doc{% endcomment %} -- Materialized views now require ownership or admin privilege to refresh. [#60448][#60448] {% comment %}doc{% endcomment %} -- `CONNECT` can be granted to / revoked from users at the database level (e.g., `GRANT CONNECT ON DATABASE db TO user;`). `CONNECT` allows users to view all objects in a database in `information_schema` and `pg_catalog`. Previously, only users could only see an object in `information_schema` / `pg_catalog` if they had any privilege (e.g., `SELECT`) on the object. Added a warning when using `USE DATABASE` that notifies the user that they do not have `CONNECT` privilege on the database and will need `CONNECT` privilege to use databases in a future release. [#59676][#59676] {% comment %}doc{% endcomment %} -- Add the built-in `crdb_internal.show_create_all_tables`, which takes in a database name (`STRING`) and returns a flat log of all the [`CREATE TABLE`](https://www.cockroachlabs.com/docs/v21.1/create-table) statements in the database followed by `ALTER` statements to add constraints. The output can be used to recreate a database. This built-in was added to replace old dump logic. [#60154][#60154] {% comment %}doc{% endcomment %} -- Implemented `default_to_database_primary_region`, which will return the region passed in if the region is defined on the database, falling back to the primary key if not. [#59836][#59836] {% comment %}doc{% endcomment %} -- The default expression for `REGIONAL BY ROW` tables is now `default_to_database_primary_region(gateway_region())`, allowing users to add to `REGIONAL BY ROW` tables from any region. This would previously error if the gateway's region was not defined on the database. [#59836][#59836] {% comment %}doc{% endcomment %} -- [`EXPLAIN`](https://www.cockroachlabs.com/docs/v21.1/explain) now shows the estimated percentage of the table that is scanned. [#60467][#60467] {% comment %}doc{% endcomment %} -- Multi-region tables will have their zone configs regenerated during database [restore](https://www.cockroachlabs.com/docs/v21.1/restore). [#60519][#60519] {% comment %}doc{% endcomment %} -- CockroachDB now supports the `DETACHED` option when running [`IMPORT`](https://www.cockroachlabs.com/docs/v21.1/import). The detached import will return `jobID` and the user can later use [`SHOW JOB`](https://www.cockroachlabs.com/docs/v21.1/show-jobs) to check the result of the detached import. This allows users to run `IMPORT` under explicit transactions with `DETACHED` option specified. [#60442][#60442] {% comment %}doc{% endcomment %} -- Casting JSON numeric scalars to numeric types now works as expected. [#41367][#41367] {% comment %}doc{% endcomment %} -- Most batches of data flowing through the vectorized execution engine will now be limited in size by `sql.distsql.temp_storage.workmem` (64MiB by default), which should improve the stability of CockroachDB clusters. [#59851][#59851] {% comment %}doc{% endcomment %} -- Implemented the ability to [`ALTER TABLE LOCALITY REGIONAL BY ROW`](https://www.cockroachlabs.com/docs/v21.1/alter-table) from other `REGIONAL BY ROW` variants. [#60497][#60497] {% comment %}doc{% endcomment %} -- Setting of [zone configs](https://www.cockroachlabs.com/docs/v21.1/configure-replication-zones) for non-physical tables is now forbidden. [#60592][#60592] {% comment %}doc{% endcomment %} -- Added telemetry to track usage of `pg_catalog` and `information_schema` tables [#60511][#60511] -- [`SHOW JOBS`](https://www.cockroachlabs.com/docs/v21.1/show-jobs) now displays a meaningful value in the `running_status` column for GC jobs which are actually performing garbage collection, as opposed to waiting on a timer. [#59220][#59220] {% comment %}doc{% endcomment %} -- Added the [`SHOW CREATE ALL TABLES`](https://www.cockroachlabs.com/docs/v21.1/show-create) statement, which allows the user to get the statements (`CREATE`/`ALTER`) to recreate a the current database. The command returns a flat log of the create statements followed by the alter statements for adding the constraints. The commands are ordered topologically such that dependencies appear before it's references. `ALTER` statements follow create statements to guarantee that the objects are all added before adding constraints. This command is added to replace old dump logic. [#60539][#60539] {% comment %}doc{% endcomment %} -- Added the `schema_name` and `table_i`d columns to the `crdb_internal.ranges` and `crdb_internal.ranges_no_leases` virtual tables. [#59865][#59865] {% comment %}doc{% endcomment %} -- Using the `CACHE` sequence option no longer results in an "unimplemented" error. The `CACHE` option is now fully implemented and will allow nodes to cache sequence numbers. A cache size of 1 means that there is no cache, and cache sizes of less than 1 are not valid. [#56954][#56954] {% comment %}doc{% endcomment %} -- The `serial_normalization` [session variable](https://www.cockroachlabs.com/docs/v21.1/set-vars) can now be set to the value `sql_sequence_cached`. If this value is set, the `sql.defaults.serial_sequences_cache_size` [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) can be used to control the number of values to cache in a user's session with a default of 256. When the cache is empty, the underlying sequence will only be incremented once to populate it. Using `sql_sequence_cached` will result in better performance than `sql_sequence` because the former will perform fewer distributed calls to increment sequences. However, cached sequences may result in large gaps between serial sequence numbers if a session terminates before using all the values in its cache. [#56954][#56954] {% comment %}doc{% endcomment %} -- CockroachDB can now encode and decode [sequence](https://www.cockroachlabs.com/docs/v21.1/create-sequence) regclasses. [#59864][#59864] -- CockroachDB now allows renaming of [sequences](https://www.cockroachlabs.com/docs/v21.1/create-sequence) referenced by ID and conversion of sequences referenced by name to ID. [#59864][#59864] {% comment %}doc{% endcomment %} -- [`EXPLAIN ANALYZE`](https://www.cockroachlabs.com/docs/v21.1/explain-analyze) now shows more top-level query statistics. [#60641][#60641] {% comment %}doc{% endcomment %} -- Added `payloads_for_span` built-in that takes in a span ID and returns its payloads in `JSONB` format. If the span is not found, or if the span does not have any payloads, the built-in returns an empty JSON object. [#60616][#60616] {% comment %}doc{% endcomment %} -- Added a new [`IMPORT PGDUMP`](https://www.cockroachlabs.com/docs/v21.1/import) option, `ignore_unsupported`, to skip over all the unsupported `PGDUMP` statements. The collection of these statements will be appropriately documented. [#57827][#57827] {% comment %}doc{% endcomment %} -- Users will now need to specify `ignore_unsupported` to ignore all unsupported import statements during an [`IMPORT PGDUMP`](https://www.cockroachlabs.com/docs/v21.1/import). [#57827][#57827] {% comment %}doc{% endcomment %} -- Added a new [`IMPORT PGDUMP`](https://www.cockroachlabs.com/docs/v21.1/import) option, `ignored_stmt_log`, which allows users to specify where they would like to log statements that have been skipped during an import, by virtue of being unsupported. [#57827][#57827] {% comment %}doc{% endcomment %} -- CockroachDB now supports `VIRTUAL` [computed columns](https://www.cockroachlabs.com/docs/v21.1/computed-columns) (as opposed to `STORED`). These are computed columns that are not stored in the primary index and are recomputed as necessary. [#60748][#60748] {% comment %}doc{% endcomment %} -- [`EXPLAIN ANALYZE`](https://www.cockroachlabs.com/docs/v21.1/explain-analyze) now includes the nodes which were involved in the execution of each operator in the tree. [#60550][#60550] {% comment %}doc{% endcomment %} -- Added missing tables and columns at `pg_catalog`. [#60758][#60758] {% comment %}doc{% endcomment %} -- Added a new [session setting](https://www.cockroachlabs.com/docs/v21.1/set-vars) `locality_optimized_partitioned_index_scan` and corresponding [cluster default setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) `sql.defaults.locality_optimized_partitioned_index_scan.enabled`. Both are currently disabled by default, and are currently unused. In the future, these settings will be used to enable or disable locality optimized search. If enabled, the optimizer will try to search locally for rows in `REGIONAL BY ROW` tables before searching remote nodes. [#60771][#60771] {% comment %}doc{% endcomment %} -- Added the new `parse_timestamp` function, which can be used to parse absolute timestamp strings in computed column expressions or partial index predicates. [#60772][#60772] {% comment %}doc{% endcomment %} -- CockroachDB now supports storing spatial type objects with `Z` and `M` dimensions (e.g., `POINTZ`, `LINESTRINGM`). [#60832][#60832] {% comment %}doc{% endcomment %} -- [`ALTER DATABASE ... ADD REGION`](https://www.cockroachlabs.com/docs/v21.1/alter-database) now re-partitions `REGIONAL BY ROW` tables and updates the [zone configs](https://www.cockroachlabs.com/docs/v21.1/configure-replication-zones) on the newly created partitions as well. [#60596][#60596] {% comment %}doc{% endcomment %} -- Updated the `crdb_internal.payloads_for_span` built-in to return a table instead of a `JSONB` array. Each row of the table represents one payload for the given span. It has columns for `payload_type` and `payload_jsonb`. [#60784][#60784] {% comment %}doc{% endcomment %} -- [`ALTER DATABASE ... DROP REGION ...`](https://www.cockroachlabs.com/docs/v21.1/alter-database) now re-partitions `REGIONAL BY ROW` tables to remove the partition for the removed region and removes the [zone configuration](https://www.cockroachlabs.com/docs/v21.1/configure-replication-zones) for the partition as well. [#60938][#60938] {% comment %}doc{% endcomment %} -- Added the experimental `experimental_enable_stream_replication` [session setting](https://www.cockroachlabs.com/docs/v21.1/set-vars) and `sql.defaults.experimental_stream_replication.enabled` [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) to enable cluster streaming. [#60826][#60826] {% comment %}doc{% endcomment %} -- Introduced a new `is_multi_region` column to c`rdb_internal.create_statements`, which informs whether the database of the object is a multi-region database. [#60761][#60761] {% comment %}doc{% endcomment %} -- Introduced the new `crdb_internal.filter_multiregion_fields_from_zone_config_sql` built-in, which removes multi-region fields from a [`CONFIGURE ZONE`](https://www.cockroachlabs.com/docs/v21.1/configure-zone) statement. [#60761][#60761] {% comment %}doc{% endcomment %} -- Added new virtual tables `crdb_internal.cluster_contention_events` and `crdb_internal.node_contention_events`, which expose cluster-wide and gateway-only, respectively, contention information. In order to access them, the user needs to be either an admin or have `VIEWACTIVITY` grant. [#60713][#60713] {% comment %}doc{% endcomment %} -- The structured payloads used for SQL audit and execution logs now include a transaction counter since the beginning of the session. Statements issued inside the same SQL [transaction](https://www.cockroachlabs.com/docs/v21.1/transactions) will thus be logged with the same counter value, thus enabling per-transactions grouping during log analysis. Separate sessions use independent counters. [#41929][#41929] {% comment %}doc{% endcomment %} -- The geometry built-ins `ST_MakePoint` and `ST_MakePointM` have been implemented and provide a mechanism for easily creating new points. [#61105][#61105] {% comment %}doc{% endcomment %} -- Added the `payloads_for_trace()` built-in so that all payloads attached to all spans for a given trace ID will be displayed, utilizing the `crdb_internal.payloads_for_span()` built-in under the hood. All payloads for long-running spans are also added to debug.zip in the `crdb_internal.node_inflight_trace_spans` table dump. [#60922][#60922] {% comment %}doc{% endcomment %} -- [`IMPORT PGDUMP`](https://www.cockroachlabs.com/docs/v21.1/import) can now import dump files with non-public schemas. [#57183][#57183] {% comment %}doc{% endcomment %} -- Added the `sql.optimizer.uniqueness_checks_for_gen_random_uuid.enabled` [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings), which controls creation of uniqueness checks for [`UUID`](https://www.cockroachlabs.com/docs/v21.1/uuid) columns set to `gen_random_uuid()`. When enabled, uniqueness checks will be added for the `UUID` column if it has a unique constraint that cannot be enforced by an index. When disabled, no uniqueness checks are planned for these columns, and uniqueness is assumed due to the near-zero collision probability of `gen_random_uuid()`. [#61132][#61132] {% comment %}doc{% endcomment %} -- The `pg_catalog.pg_class` table now has a `relpartbound` column. This is only for compatibility, and the column value is always `NULL`. [#61162][#61162] {% comment %}doc{% endcomment %} -- Cluster [backup](https://www.cockroachlabs.com/docs/v21.1/backup) now [restores](https://www.cockroachlabs.com/docs/v21.1/restore) the zone configurations first. This means that there should be less range-relocation during and after cluster restores. [#60461][#60461] {% comment %}doc{% endcomment %} -- Fixed `information_schema.columns.udt_schema` for `ENUM` or user-defined types. [#61139][#61139] {% comment %}doc{% endcomment %} -- The geography built-in `ST_Affine` now supports 3D transformations. [#61286][#61286] {% comment %}doc{% endcomment %} -- The `ST_Z`, `ST_M`, and `ST_Zmflag` built-ins are now available for use. [#61032][#61032] {% comment %}doc{% endcomment %} -- Geometry built-ins for forcing geometries into non-2D layouts are now available. [#61297][#61297] {% comment %}doc{% endcomment %} -- `ST_Buffer` now requires at least 1 quadrant segment. [#61315][#61315] {% comment %}doc{% endcomment %} -- `ST_RotateX` function is now available. [#61326][#61326] {% comment %}doc{% endcomment %} -- [`EXPLAIN`](https://www.cockroachlabs.com/docs/v21.1/explain) now shows estimated row counts for all operators even without `VERBOSE` (except when we do not have statistics for the tables). [#61344][#61344] {% comment %}doc{% endcomment %} -- Updated the error message returned in case of a [unique constraint](https://www.cockroachlabs.com/docs/v21.1/unique) violation to hide the names and values for implicit partitioning columns and hash sharded index columns. [#61362][#61362] -- Fixed `information_schema.columns.is_identity` to display the correct value. [#61348][#61348] -- `ST_RotateY` and `ST_RotateZ` are now available. [#61387][#61387] {% comment %}doc{% endcomment %} -- CockroachDB now prevents `densifyFracs < 1e-6` for `ST_FrechetDistance` and `ST_HausdorffDistance` to protect panics and out-of-memory errors. [#61427][#61427] -- CockroachDB now disallows `GRANT/REVOKE` operations on system tables. [#61410][#61410] {% comment %}doc{% endcomment %} -- `ST_Snap` is now available. [#61523][#61523] {% comment %}doc{% endcomment %} -- Updates to certain fields in the zone configurations are blocked for multi-region enabled databases. This block can be overridden through the use of the `FORCE` keyword on the blocked statement. [#61499][#61499] {% comment %}doc{% endcomment %} -- The `ST_AddMeasure` function is now available for use. [#61514][#61514] {% comment %}doc{% endcomment %} -- Added a new built-in that sets the verbosity of all spans in a given trace. Syntax: `crdb_internal.set_trace_verbose($traceID,$verbosityAsBool)`. [#61353][#61353] {% comment %}doc{% endcomment %} -- `Pg_index.indkey` now includes attributes. [#61494][#61494] {% comment %}doc{% endcomment %} -- [`SHOW CREATE ALL TABLES`](https://www.cockroachlabs.com/docs/v21.1/show-create) is updated to be more memory efficient, however this should still not be run on a database with an excessive number of tables. Users should not run this on a database with more than 10000 tables (arbitrary but tested number). [#61127][#61127] {% comment %}doc{% endcomment %} -- Set the default [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) for `sql.defaults.locality_optimized_partitioned_index_scan.enabled` to `true`, which is the default for the [session setting](https://www.cockroachlabs.com/docs/v21.1/set-vars) `locality_optimized_partitioned_index_scan`. When this session setting is enabled, the optimizer will plan queries to use "locality optimized search" when possible, which instructs the execution engine to search for keys in local nodes before searching remote nodes. If a key is found in a local node, the execution engine may be able to avoid visiting the remote nodes altogether. [#61601][#61601] {% comment %}doc{% endcomment %} -- [`ALTER TYPE ... DROP VALUE`](https://www.cockroachlabs.com/docs/v21.1/alter-type) is gated behind a feature flag. The [session setting](https://www.cockroachlabs.com/docs/v21.1/set-vars) is called `enable_drop_enum_value` and the corresponding [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) is called `sql.defaults.drop_enum_value.enabled`. [#61723][#61723] {% comment %}doc{% endcomment %} -- `TYPE SCHEMA CHANGE` jobs which include a `DROP VALUE` in them are now cancellable. All other jobs remain non-cancellable. [#61733][#61733] {% comment %}doc{% endcomment %} -- A statement contention timeseries is now displayed in the SQL metrics overview dashboard. [#61844][#61844] {% comment %}doc{% endcomment %} -- The `ST_SnapToGrid` function can now be used to snap `Z` and `M` dimensions. [#61826][#61826] {% comment %}doc{% endcomment %} -- Added `json_extract_path_text` and `jsonb_extract_path_text` built-ins. [#61813][#61813] {% comment %}doc{% endcomment %} -- Removed `pg_catalog` tables that were mistakenly added, notably all tables that end in `_index` that are not `pg_catalog.pg_classes` and all statistics collector tables that are not `pg_stat_activity`. [#61876][#61876] -- The `replicas` column of `crdb_internal.ranges{_no_leases}` now includes both voting and non-voting replicas and `crdb_internal.ranges{_no_leases}` include two new columns: `voting_replicas` and `non_voting_replicas`, which work as labeled. [#61962][#61962] {% comment %}doc{% endcomment %} - -

Operational changes

- -- Added the `storage.transaction.separated_intents.enabled` [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings), which enables separated intents by default. [#59829][#59829] {% comment %}doc{% endcomment %} - -

Command-line changes

- -- It is now possible to redirect logging to [Fluentd](https://www.fluentd.org)-compatible network collectors. See the reference sink documentation for details.This is an alpha-quality feature. Note that Fluent-enabled configuration only provide minimal event buffering, and log entries are dropped if the logging server becomes unavailable or network errors are encountered. This is a known limitation and will be likely improved in a later version. [#57170][#57170] {% comment %}doc{% endcomment %} -- The SQL shell ([`cockroach sql`](https://www.cockroachlabs.com/docs/v21.1/cockroach-sql) / [`cockroach demo`](https://www.cockroachlabs.com/docs/v21.1/cockroach-demo)) now supports a flag `--embedded`, for use with playground-style web applications. [#59750][#59750] {% comment %}doc{% endcomment %} -- [`cockroach userfile get`](https://www.cockroachlabs.com/docs/v21.1/cockroach-userfile-get) can now be used to fetch files from a cluster. [#60490][#60490] {% comment %}doc{% endcomment %} -- Added the `ignore-unsupported-statements`, `log-ignored-statements` and `row-limit` flags to the [`cockroach import`](https://www.cockroachlabs.com/docs/v21.1/cockroach-import) command. [#60923][#60923] {% comment %}doc{% endcomment %} -- The doctor tool can now report multiple descriptor validation failures per descriptor. [#60775][#60775] -- The new `cockroach connect` command now recognizes `--single-node` to prepare a TLS configuration suitable for a subsequent `start-single-node` command. Additionally, the command checks that either `--single-node` is specified, or both `--init-token` and `--num-expected-peers`. [#60854][#60854] {% comment %}doc{% endcomment %} -- Optimized handling of multi-line SQL strings to avoid unwanted extra server roundtrips. [#61207][#61207] -- Added back support for [`cockroach dump --dump-mode=schema`](https://www.cockroachlabs.com/docs/v21.1/cockroach-dump). This command calls [`SHOW CREATE ALL TABLES`](https://www.cockroachlabs.com/docs/v21.1/show-create) and returns the output. This gives users a migration path to use in v21.1 while switching over to `SHOW CREATE ALL TABLES` for v21.2. When executing this, the user is warned that the command is being removed in v21.2 and they should use `SHOW CREATE ALL TABLES` instead. [#61871][#61871] {% comment %}doc{% endcomment %} - -

API endpoint changes

- -- Added the following new HTTP API endpoints: - - - `/api/v2/nodes/`: Lists all nodes in the cluster - - `/api/v2/nodes//ranges`: Lists all ranges on the specified node - - `/api/v2/ranges/hot/`: Lists hot ranges in the cluster - - `/api/v2/ranges//`: Describes range in more detail - - `/api/v2/health/`: Returns an HTTP 200 response if node is healthy. [#60952][#60952] - -

DB Console changes

- -- Manually enqueue range in a replica GC queue now properly respects the `SkipShouldQueue` option. This can be useful to force a GC of a specific Range. [#60619][#60619] {% comment %}doc{% endcomment %} -- Added a filter for full scans to the [**Statements** page](https://www.cockroachlabs.com/docs/v21.1/ui-statements-page). [#60670][#60670] {% comment %}doc{% endcomment %} -- The **Range Report** page on the DB Console will now also show each of the replica's types. [#61113][#61113] {% comment %}doc{% endcomment %} -- The **Transaction Type** column was removed from [**Statements** page](https://www.cockroachlabs.com/docs/v21.1/ui-statements-page). The sort order in **Statement** / **Transaction Table** views is now by execution count. The sort order in **Transaction Details** view is by latency. Column titles in statements and transactions tables now say "{Statement,Transaction} Time" instead of "Latency". The **Contention** column added to the **Statements** / **Transactions** pages. **Max Scratch Disk Usage** added to **Statements** and **Transaction Details** pages. [#61177][#61177] {% comment %}doc{% endcomment %} -- Added a full-table scan checkbox. [#61676][#61676] {% comment %}doc{% endcomment %} - -

Bug fixes

- -- Fixed a bug in the DB Console affecting status badges in Nodes list on **Cluster Overview** page. [#59636][#59636] -- Fixes a bug where backups would fail with an error when trying to read a backup that was written. [#59730][#59730] -- Fixes a bug where some import failures would cause tables to stay `OFFLINE`, when they should have been brought back to `PUBLIC`. [#59723][#59723] -- Fixed a bug that caused errors when joining two tables when one of the tables had a computed column. This bug was present since version v21.1.0-alpha.2 and not present in any production releases. [#59741][#59741] -- Statistics are now correctly generated for columns in multi-column GIN indexes and columns referenced in partial GIN index predicates. [#59687][#59687] -- Fixed a timestamp overflow bug that would overflow timestamps after the year 2038 by updating the Avro SDK used by changefeeds. [#59745][#59745] -- Fixed a very rare chance of inconsistent follower reads. [#59502][#59502] -- Previously if `RELEASE SAVEPOINT cockroach_restart` was followed by `ROLLBACK`, the `sql.txn.rollback.count` metric would be incremented. This was incorrect, since the transaction had already committed. Now that metric is not incremented in this case. [#59781][#59781] -- CockroachDB now explicitly verify descriptors are sequences. [#60159][#60159] -- The result of the `cockroach debug doctor` invocation during `cockroach debug zip` is now properly included inside the generated zip file. [#60529][#60529] -- Fixed a bug in the optimizer statistics code that could cause an unconstrained partial index scan to be preferred over a constrained scan of the same index. [#60516][#60516] -- Fixed a deficiency in the replication layer that could result in ranges becoming unavailable for prolonged periods of time (hours) when a write burst occurred under system overload. While unavailable, the range status page for the affected range would show a last index much larger than the committed index and no movement in these indexes on a quorum of the replicas. Note that this should be distinguished from the case in which enough replicas are offline to constitute a loss of quorum, where the replication layer can not make progress due to the loss of quorum itself. [#60581][#60581] -- Previously, retryable errors in the cleanup phase of the type schema changer wouldn't be retried automatically in the background. This is now fixed. [#60495][#60495] -- v20.2 introduced an ability to rebalance replicas between multiple stores on the same node. This change fixed a problem with that feature, where occasionally an intra-node rebalance would fail and a range would get stuck permanently under replicated. [#60546][#60546] -- Fixed a bug that would cause the value of the optional `into_db` parameter to `RESTORE` to be included in anonymized crash reports. [#60624][#60624] -- Fixed a bug that caused errors for some queries on tables with `GEOMETRY` or `GEOGRAPHY` GIN indexes with filters containing shapes with zero area. [#60598][#60598] -- CockroachDB previously didn't account for some RAM used when disk-spilling operations (like sorts and hash joins) were using the temporary storage in the vectorized execution engine. This could result in OOM crashes, especially when the rows are large in size. This has been fixed. [#60593][#60593] -- Fixed a bug where `ST_Node` would panic if passed in `MULTILINESTRING EMPTY`. [#60691][#60691] -- Fixed a bug that could result in crashes during tracing when using the `trace.debug.enable` cluster setting. [#60725][#60725] -- Fixed execution errors for some queries that use set operations (`UNION` / `EXCEPT` / `INTERSECT`) where a column has types of different widths on the two sides (e.g., `INT4` vs. `INT8`). [#60560][#60560] -- CockroachDB now avoids creating batches that exceed the raft command limit (64MB) when reverting ranges that contain very large keys. [#59716][#59716] -- Fixed a bug whereby high-latency global clusters could sometimes fall behind checkpointing resolved timestamps. [#60807][#60807] -- Added check privileges before changing table/database owner. [#60800][#60800] -- Fixed an internal error caused in some cases involving JSON objects and arrays in a `VALUES` clause. [#60767][#60767] -- Previously `DROP TYPE IF EXISTS` with one existent, and another non-existent type would cause an unhandled error. This is now fixed. [#60822][#60822] -- Fixed bug that could report that a protected timestamp limit was exceeded when the limit was disabled, if an error were to occur while protecting a record. [#60913][#60913] -- CockroachDB previously could encounter an internal error when performing `UNION` operation when the first input resulted only in NULL values and consequent inputs produce tuples, and this is now fixed. Only v21.1 alpha versions are affected. [#60827][#60827] -- Fixed a bug in `crdb_internal.unsafe_upsert_namespace_entry` related to tables and types in user-defined schemas. [#60510][#60510] -- Integers inside of tuples were not being encoded properly when using the binary format for retrieving data. This is now fixed, and the proper integer width is reported. [#61013][#61013] -- Blank-padded chars (e.g., `CHAR(3)`) were not being encoded correctly when returning results to the client. Now they correctly include blank-padding when appropriate. [#61013][#61013] -- Collated strings were not encoded with the proper type OID when sending results to the client if the OID was for the `char` type. This is now fixed. [#61013][#61013] -- Fixed a very rare, possible impossible in practice, bug where a range merge that applied through a Raft snapshot on the left-hand side range's leaseholder could allow that leaseholder to serve writes that invalidated reads from before the merge on the right-hand side. Release justification: bug fix [#60521][#60521] -- The `SHOW CREATE` output of a partitioned partial index now lists the `PARTITION BY` and `WHERE` clauses in the order accepted by the parser. [#60590][#60590] -- The `SHOW CREATE` output of a partial interleaved index now lists the `INTERLEAVED` and `WHERE` clauses in the order accepted by the parser. [#60590][#60590] -- Fix a bug where cluster restore would sometimes (very rarely) fail after retrying. [#60458][#60458] -- Previously, comparing a negative integer to an OID would fail to compare correctly because the integer was not converted to an unsigned representation first. this is now fixed for both comparisons and casts.[#61148][#61148] -- Previously, CockroachDB could not decode arrays of user-defined types when sent to the server in the binary format. Now it can. [#61165][#61165] -- `crdb_internal.jobs` virtual table is now populated in a paginated fashion, thus, alleviating memory related concerns when previously we could encounter OOM crash. [#60693][#60693] -- The `SHOW TABLES FROM database` command would always show a NULL `estimated_row_count` if inspecting a database that was not the current database. This is now fixed. [#61191][#61191] -- Fixed a bug where schema changes on databases and schemas could return a `relation [] does not exist` if they failed or were canceled and entered the reverting state. These jobs are not actually possible to revert. With this change, the correct error causing the job to fail will be returned, and the job will enter the failed state with an error indicating that the job could not be reverted. [#61159][#61159] -- CockroachDB now drops the default value when its dependent sequence is dropped. [#60744][#60744] -- Dropping and recreating a view/table/sequence in a transaction will now correctly error out if a conflicting object exists or if the drop is incomplete. [#61135][#61135] -- The non-server `cockroach` commands now recognize the new `--log` flag properly. This had been broken in one of the earlier v21.1 alpha releases. [#61232][#61232] -- Prepared statements would sometimes get the wrong type OID for placeholder arguments used in function parameters. This is now fixed. [#60949][#60949] -- Fixed a case where empty zone configurations get created for certain indexes during `ALTER PRIMARY KEY`. [#61235][#61235] -- Schema change jobs associated with databases and schemas can no longer be canceled. Such jobs cannot actually be reverted successfully, so cancelation had no benefit and could have caused namespace corruption. [#61210][#61210] -- Fixed `cockroach demo` commands nodeName error argument index handling. [#61246][#61246] -- CockroachDB no longer displays incorrect node statuses on **Metrics** page, when page just loaded. [#61000][#61000] -- Fixed an NPE observed with a SpanFromContext call in the stack trace. [#61124][#61124] -- Limit scans are no longer counted as full scans. [#61178][#61178] -- Selecting from `crdb_internal.create_statements` will now correctly populate the `database_name` when the virtual index is not used. [#61201][#61201] -- Fixed an internal error when `EXPLAIN`ing an `INSERT` with an input that was determined by the optimizer to produce no rows. [#61278][#61278] -- Fixed a rare deadlock where a series of lease transfers concurrent with a Range merge could block each other from ever completing. [#60905][#60905] -- `ALTER TYPE ... ADD VALUE` changes are picked up by the array type alias correctly. [#61288][#61288] -- Previously, the traces of cascades and checks could be incomplete. This is now fixed. [#61321][#61321] -- Creating interleaved partitioned indexes is now disallowed. Previously, the database would crash when trying to create one. [#61106][#61106] -- Dropping and re-creating a database in a transaction will now correctly error out if an object in a dropped state is detected. [#61361][#61361] [#61358][#61358] -- CockroachDB now uses the correct `FuncExpr` when encoding sequences. [#61428][#61428] -- Alter primary key was not idempotent, so logical equivalent changes to primary keys would unnecessarily create new indexes. This is now fixed. [#61345][#61345] -- Fixed a bug where `GRANT/REVOKE` on the `system.lease` table would result in a deadlock. [#61410][#61410] -- Fixed operation hangs when a node loses access to cluster RPC (e.g., after it has been decommissioned), and immediately return an error instead. [#61356][#61356] -- Fixed a bug from v21.1-alpha where a node decommissioning process could sometimes hang or fail when the decommission request was submitted via the node being decommissioned. [#61356][#61356] -- Fixed bug where zone configurations on indexes were not copied before the backfill of an `ALTER PRIMARY KEY`. They used to be copied afterwards instead. [#61300][#61300] -- Fixed a bug where random numbers generated as default expressions during `IMPORT` would collide a few hundred rows apart from each-other. [#61214][#61214] -- Fixed a bug that caused `UPSERT` and `INSERT..ON CONFLICT..DO UPDATE `statements to fail on tables with both partial indexes and foreign key references. This bug has been present since v20.2.0. [#61416][#61416] -- An `UPDATE..FROM` statement where the `FROM` clause contained column names that match table column names erred if the table had a partial index predicate referencing those columns. This bug, present since partial indexes were released in v20.2.0, has been fixed. [#61522][#61522] -- The names of custom types are no longer sent to Cockroach Labs in telemetry and crash reports. [#60806][#60806] -- Fixed a case where an invalid tuple comparison using ANY was causing an internal error; we now return "unsupported comparison operator". [#61647][#61647] -- Added better privilege checks when creating a changefeed. [#61709][#61709] -- Fixed privileges for system.`protected_ts_meta`. [#61842][#61842] -- The `indexdef` column in the `pg_indexes` table would always report that the index belonged to the public schema. Now it correctly reports user-defined schemas if necessary. [#61754][#61754] -- Fixed "command is too large" errors in some cases when using `EXPLAIN ANALYZE (DEBUG)` or statement diagnostics on complex queries. [#61909][#61909] -- Using `EXPLAIN (OPT, ENV)` on a query that references a table in a user-defined schema would fail previously. This is now fixed. [#61888][#61888] -- Fixed a bug that caused "column does not exist errors" in specific cases of `UPDATE..FROM` statements. The error only occurred when updating a `DECIMAL` column to a column in the `FROM` clause, and the column had a `CHECK` constraint or was referenced by a partial index predicate. [#61949][#61949] -- Previously the `idle_in_session_timeout` and `idle_in_transaction_session_timeout` settings would show the wrong value when using `SHOW`. They would instead show the value of the `statement_timeout` setting. This is now fixed. The functionality was already working correctly; this just fixes a display bug. [#61959][#61959] - -

Performance improvements

- -- Improved the performance of the `pg_table_is_visible` built-in function. [#59880][#59880] -- Added support for left and anti lookup joins in cases where the equi-join condition is not on the first column in the index, but the first column(s) are constrained to a small number of constant values using a CHECK constraint or an ENUM type. Planning a lookup join for these cases can significantly improve performance if the left input to the join is much smaller than the right-hand side. [#60302][#60302] -- The optimizer now knows that the unique columns in an implicitly partitioned unique index form a key. This can be used to enable certain optimizations and may result in better plans. [#60591][#60591] -- Improved the optimizer's ability to identify constant columns in scans. This can enable different types of optimizations and result in improved plans. [#60927][#60927] -- Follower reads no longer wait before redirecting to the leaseholder if they could not be served by a local follower due to an insufficient closed timestamp. [#60839][#60839] -- Updated the query used to build the virtual table `crdb_internal.table_row_statistics` so that it is always run at `AS OF SYSTEM TIME '-10s'`. This should reduce contention on the table and improve performance for transactions that rely on `crdb_internal.table_row_statistics`, such as `SHOW TABLES`. [#60953][#60953] -- Improved the optimizer's cost estimation of index scans that must visit multiple partitions. When an index has multiple partitions, the optimizer is now more likely to choose a constrained scan rather than a full index scan. This can lead to better plans and improved performance. It also improves the ability of the database to serve queries if one of the partitions is unavailable. [#61063][#61063] -- Optimized external sort to merge partition data faster in colexec. [#61056][#61056] -- If the session setting `locality_optimized_partitioned_index_scan` is enabled, the optimizer will try to plan scans known to produce at most one row using "locality optimized search". This optimization applies for `REGIONAL BY ROW` tables, and if enabled, it means that the execution engine will first search locally for the row before searching remote nodes. If the row is found in a local node, remote nodes will not be searched. [#60831][#60831] -- The optimizer now infers additional functional dependencies based on computed columns in tables. This may enable additional optimizations and lead to better query plans. [#61097][#61097] -- Removed uniqueness checks on the primary key for `REGIONAL BY ROW` tables with a computed region column that is a function of the primary key columns. Uniqueness checks are not necessary in this case since uniqueness can be suitably guaranteed by the primary index. Removing these checks improves performance of `INSERT`, `UPDATE`, and `UPSERT` statements. [#61097][#61097] -- The optimizer no longer plans uniqueness checks for columns in implicitly partitioned unique indexes when those columns are used as the arbiters to detect conflicts in `INSERT ... ON CONFLICT` statements. Unique checks are not needed in this case to enforce uniqueness. Removing these checks results in improved performance for `INSERT ... ON CONFLICT` statements. [#61184][#61184] -- The columns fetched for uniqueness checks of implicitly partitioned unique indexes are now pruned to only include columns necessary for determining uniqueness. [#61376][#61376] -- Introspection queries that use casts into the `REGPROC` pseudo-type are made much more efficient: they're now implemented as a constant-time lookup instead of an internal query. [#61211][#61211] -- Fixed cases where the optimizer was doing unnecessary full table scans when the table was very small (according to the last collected statistics). [#61805][#61805] -- The optimizer now estimates the cost of evaluating query filters more accurately for queries with a `LIMIT`. Previously, it was assumed that the filter would be evaluated on each input row. Now, the optimizer assumes that the filter will only be evaluated on the number of rows required to produce the LIMIT's number of rows after filtering. This may lead to more efficient query plans in some cases. [#61947][#61947] - -
- -

Contributors

- -This release includes 667 merged PRs by 70 authors. -We would like to thank the following contributors from the CockroachDB community: - -- Abdullah Islam (first-time contributor) -- Alan Acosta (first-time contributor) -- Kumar Akshay -- Max Neverov -- Miguel Novelo (first-time contributor) -- Ratnesh Mishra (first-time contributor) -- Tharun (first-time contributor) -- Ulf Adams (first-time contributor) -- alex-berger@gmx.ch (first-time contributor) -- leoric (first-time contributor) -- shikamaru (first-time contributor) - -
- -[#41367]: https://github.com/cockroachdb/cockroach/pull/41367 -[#41929]: https://github.com/cockroachdb/cockroach/pull/41929 -[#56954]: https://github.com/cockroachdb/cockroach/pull/56954 -[#57077]: https://github.com/cockroachdb/cockroach/pull/57077 -[#57170]: https://github.com/cockroachdb/cockroach/pull/57170 -[#57183]: https://github.com/cockroachdb/cockroach/pull/57183 -[#57827]: https://github.com/cockroachdb/cockroach/pull/57827 -[#58422]: https://github.com/cockroachdb/cockroach/pull/58422 -[#58688]: https://github.com/cockroachdb/cockroach/pull/58688 -[#58923]: https://github.com/cockroachdb/cockroach/pull/58923 -[#59086]: https://github.com/cockroachdb/cockroach/pull/59086 -[#59220]: https://github.com/cockroachdb/cockroach/pull/59220 -[#59396]: https://github.com/cockroachdb/cockroach/pull/59396 -[#59410]: https://github.com/cockroachdb/cockroach/pull/59410 -[#59490]: https://github.com/cockroachdb/cockroach/pull/59490 -[#59492]: https://github.com/cockroachdb/cockroach/pull/59492 -[#59502]: https://github.com/cockroachdb/cockroach/pull/59502 -[#59539]: https://github.com/cockroachdb/cockroach/pull/59539 -[#59571]: https://github.com/cockroachdb/cockroach/pull/59571 -[#59621]: https://github.com/cockroachdb/cockroach/pull/59621 -[#59624]: https://github.com/cockroachdb/cockroach/pull/59624 -[#59636]: https://github.com/cockroachdb/cockroach/pull/59636 -[#59676]: https://github.com/cockroachdb/cockroach/pull/59676 -[#59687]: https://github.com/cockroachdb/cockroach/pull/59687 -[#59692]: https://github.com/cockroachdb/cockroach/pull/59692 -[#59693]: https://github.com/cockroachdb/cockroach/pull/59693 -[#59709]: https://github.com/cockroachdb/cockroach/pull/59709 -[#59710]: https://github.com/cockroachdb/cockroach/pull/59710 -[#59716]: https://github.com/cockroachdb/cockroach/pull/59716 -[#59717]: https://github.com/cockroachdb/cockroach/pull/59717 -[#59723]: https://github.com/cockroachdb/cockroach/pull/59723 -[#59730]: https://github.com/cockroachdb/cockroach/pull/59730 -[#59732]: https://github.com/cockroachdb/cockroach/pull/59732 -[#59735]: https://github.com/cockroachdb/cockroach/pull/59735 -[#59741]: https://github.com/cockroachdb/cockroach/pull/59741 -[#59745]: https://github.com/cockroachdb/cockroach/pull/59745 -[#59750]: https://github.com/cockroachdb/cockroach/pull/59750 -[#59781]: https://github.com/cockroachdb/cockroach/pull/59781 -[#59790]: https://github.com/cockroachdb/cockroach/pull/59790 -[#59810]: https://github.com/cockroachdb/cockroach/pull/59810 -[#59815]: https://github.com/cockroachdb/cockroach/pull/59815 -[#59824]: https://github.com/cockroachdb/cockroach/pull/59824 -[#59828]: https://github.com/cockroachdb/cockroach/pull/59828 -[#59829]: https://github.com/cockroachdb/cockroach/pull/59829 -[#59831]: https://github.com/cockroachdb/cockroach/pull/59831 -[#59836]: https://github.com/cockroachdb/cockroach/pull/59836 -[#59851]: https://github.com/cockroachdb/cockroach/pull/59851 -[#59856]: https://github.com/cockroachdb/cockroach/pull/59856 -[#59858]: https://github.com/cockroachdb/cockroach/pull/59858 -[#59864]: https://github.com/cockroachdb/cockroach/pull/59864 -[#59865]: https://github.com/cockroachdb/cockroach/pull/59865 -[#59877]: https://github.com/cockroachdb/cockroach/pull/59877 -[#59880]: https://github.com/cockroachdb/cockroach/pull/59880 -[#59978]: https://github.com/cockroachdb/cockroach/pull/59978 -[#59989]: https://github.com/cockroachdb/cockroach/pull/59989 -[#59995]: https://github.com/cockroachdb/cockroach/pull/59995 -[#60154]: https://github.com/cockroachdb/cockroach/pull/60154 -[#60159]: https://github.com/cockroachdb/cockroach/pull/60159 -[#60160]: https://github.com/cockroachdb/cockroach/pull/60160 -[#60257]: https://github.com/cockroachdb/cockroach/pull/60257 -[#60281]: https://github.com/cockroachdb/cockroach/pull/60281 -[#60282]: https://github.com/cockroachdb/cockroach/pull/60282 -[#60302]: https://github.com/cockroachdb/cockroach/pull/60302 -[#60311]: https://github.com/cockroachdb/cockroach/pull/60311 -[#60379]: https://github.com/cockroachdb/cockroach/pull/60379 -[#60437]: https://github.com/cockroachdb/cockroach/pull/60437 -[#60442]: https://github.com/cockroachdb/cockroach/pull/60442 -[#60448]: https://github.com/cockroachdb/cockroach/pull/60448 -[#60458]: https://github.com/cockroachdb/cockroach/pull/60458 -[#60461]: https://github.com/cockroachdb/cockroach/pull/60461 -[#60467]: https://github.com/cockroachdb/cockroach/pull/60467 -[#60469]: https://github.com/cockroachdb/cockroach/pull/60469 -[#60490]: https://github.com/cockroachdb/cockroach/pull/60490 -[#60495]: https://github.com/cockroachdb/cockroach/pull/60495 -[#60497]: https://github.com/cockroachdb/cockroach/pull/60497 -[#60510]: https://github.com/cockroachdb/cockroach/pull/60510 -[#60511]: https://github.com/cockroachdb/cockroach/pull/60511 -[#60516]: https://github.com/cockroachdb/cockroach/pull/60516 -[#60519]: https://github.com/cockroachdb/cockroach/pull/60519 -[#60521]: https://github.com/cockroachdb/cockroach/pull/60521 -[#60529]: https://github.com/cockroachdb/cockroach/pull/60529 -[#60539]: https://github.com/cockroachdb/cockroach/pull/60539 -[#60546]: https://github.com/cockroachdb/cockroach/pull/60546 -[#60550]: https://github.com/cockroachdb/cockroach/pull/60550 -[#60560]: https://github.com/cockroachdb/cockroach/pull/60560 -[#60581]: https://github.com/cockroachdb/cockroach/pull/60581 -[#60590]: https://github.com/cockroachdb/cockroach/pull/60590 -[#60591]: https://github.com/cockroachdb/cockroach/pull/60591 -[#60592]: https://github.com/cockroachdb/cockroach/pull/60592 -[#60593]: https://github.com/cockroachdb/cockroach/pull/60593 -[#60594]: https://github.com/cockroachdb/cockroach/pull/60594 -[#60596]: https://github.com/cockroachdb/cockroach/pull/60596 -[#60598]: https://github.com/cockroachdb/cockroach/pull/60598 -[#60616]: https://github.com/cockroachdb/cockroach/pull/60616 -[#60619]: https://github.com/cockroachdb/cockroach/pull/60619 -[#60624]: https://github.com/cockroachdb/cockroach/pull/60624 -[#60641]: https://github.com/cockroachdb/cockroach/pull/60641 -[#60670]: https://github.com/cockroachdb/cockroach/pull/60670 -[#60691]: https://github.com/cockroachdb/cockroach/pull/60691 -[#60693]: https://github.com/cockroachdb/cockroach/pull/60693 -[#60708]: https://github.com/cockroachdb/cockroach/pull/60708 -[#60713]: https://github.com/cockroachdb/cockroach/pull/60713 -[#60725]: https://github.com/cockroachdb/cockroach/pull/60725 -[#60744]: https://github.com/cockroachdb/cockroach/pull/60744 -[#60748]: https://github.com/cockroachdb/cockroach/pull/60748 -[#60753]: https://github.com/cockroachdb/cockroach/pull/60753 -[#60758]: https://github.com/cockroachdb/cockroach/pull/60758 -[#60761]: https://github.com/cockroachdb/cockroach/pull/60761 -[#60767]: https://github.com/cockroachdb/cockroach/pull/60767 -[#60771]: https://github.com/cockroachdb/cockroach/pull/60771 -[#60772]: https://github.com/cockroachdb/cockroach/pull/60772 -[#60775]: https://github.com/cockroachdb/cockroach/pull/60775 -[#60784]: https://github.com/cockroachdb/cockroach/pull/60784 -[#60799]: https://github.com/cockroachdb/cockroach/pull/60799 -[#60800]: https://github.com/cockroachdb/cockroach/pull/60800 -[#60806]: https://github.com/cockroachdb/cockroach/pull/60806 -[#60807]: https://github.com/cockroachdb/cockroach/pull/60807 -[#60822]: https://github.com/cockroachdb/cockroach/pull/60822 -[#60825]: https://github.com/cockroachdb/cockroach/pull/60825 -[#60826]: https://github.com/cockroachdb/cockroach/pull/60826 -[#60827]: https://github.com/cockroachdb/cockroach/pull/60827 -[#60831]: https://github.com/cockroachdb/cockroach/pull/60831 -[#60832]: https://github.com/cockroachdb/cockroach/pull/60832 -[#60839]: https://github.com/cockroachdb/cockroach/pull/60839 -[#60847]: https://github.com/cockroachdb/cockroach/pull/60847 -[#60854]: https://github.com/cockroachdb/cockroach/pull/60854 -[#60901]: https://github.com/cockroachdb/cockroach/pull/60901 -[#60903]: https://github.com/cockroachdb/cockroach/pull/60903 -[#60905]: https://github.com/cockroachdb/cockroach/pull/60905 -[#60908]: https://github.com/cockroachdb/cockroach/pull/60908 -[#60913]: https://github.com/cockroachdb/cockroach/pull/60913 -[#60922]: https://github.com/cockroachdb/cockroach/pull/60922 -[#60923]: https://github.com/cockroachdb/cockroach/pull/60923 -[#60927]: https://github.com/cockroachdb/cockroach/pull/60927 -[#60938]: https://github.com/cockroachdb/cockroach/pull/60938 -[#60941]: https://github.com/cockroachdb/cockroach/pull/60941 -[#60949]: https://github.com/cockroachdb/cockroach/pull/60949 -[#60952]: https://github.com/cockroachdb/cockroach/pull/60952 -[#60953]: https://github.com/cockroachdb/cockroach/pull/60953 -[#60992]: https://github.com/cockroachdb/cockroach/pull/60992 -[#61000]: https://github.com/cockroachdb/cockroach/pull/61000 -[#61013]: https://github.com/cockroachdb/cockroach/pull/61013 -[#61018]: https://github.com/cockroachdb/cockroach/pull/61018 -[#61032]: https://github.com/cockroachdb/cockroach/pull/61032 -[#61041]: https://github.com/cockroachdb/cockroach/pull/61041 -[#61043]: https://github.com/cockroachdb/cockroach/pull/61043 -[#61047]: https://github.com/cockroachdb/cockroach/pull/61047 -[#61056]: https://github.com/cockroachdb/cockroach/pull/61056 -[#61063]: https://github.com/cockroachdb/cockroach/pull/61063 -[#61097]: https://github.com/cockroachdb/cockroach/pull/61097 -[#61105]: https://github.com/cockroachdb/cockroach/pull/61105 -[#61106]: https://github.com/cockroachdb/cockroach/pull/61106 -[#61113]: https://github.com/cockroachdb/cockroach/pull/61113 -[#61124]: https://github.com/cockroachdb/cockroach/pull/61124 -[#61127]: https://github.com/cockroachdb/cockroach/pull/61127 -[#61130]: https://github.com/cockroachdb/cockroach/pull/61130 -[#61132]: https://github.com/cockroachdb/cockroach/pull/61132 -[#61135]: https://github.com/cockroachdb/cockroach/pull/61135 -[#61139]: https://github.com/cockroachdb/cockroach/pull/61139 -[#61148]: https://github.com/cockroachdb/cockroach/pull/61148 -[#61159]: https://github.com/cockroachdb/cockroach/pull/61159 -[#61162]: https://github.com/cockroachdb/cockroach/pull/61162 -[#61165]: https://github.com/cockroachdb/cockroach/pull/61165 -[#61169]: https://github.com/cockroachdb/cockroach/pull/61169 -[#61170]: https://github.com/cockroachdb/cockroach/pull/61170 -[#61177]: https://github.com/cockroachdb/cockroach/pull/61177 -[#61178]: https://github.com/cockroachdb/cockroach/pull/61178 -[#61184]: https://github.com/cockroachdb/cockroach/pull/61184 -[#61191]: https://github.com/cockroachdb/cockroach/pull/61191 -[#61201]: https://github.com/cockroachdb/cockroach/pull/61201 -[#61207]: https://github.com/cockroachdb/cockroach/pull/61207 -[#61210]: https://github.com/cockroachdb/cockroach/pull/61210 -[#61211]: https://github.com/cockroachdb/cockroach/pull/61211 -[#61214]: https://github.com/cockroachdb/cockroach/pull/61214 -[#61221]: https://github.com/cockroachdb/cockroach/pull/61221 -[#61232]: https://github.com/cockroachdb/cockroach/pull/61232 -[#61235]: https://github.com/cockroachdb/cockroach/pull/61235 -[#61244]: https://github.com/cockroachdb/cockroach/pull/61244 -[#61246]: https://github.com/cockroachdb/cockroach/pull/61246 -[#61251]: https://github.com/cockroachdb/cockroach/pull/61251 -[#61278]: https://github.com/cockroachdb/cockroach/pull/61278 -[#61286]: https://github.com/cockroachdb/cockroach/pull/61286 -[#61288]: https://github.com/cockroachdb/cockroach/pull/61288 -[#61297]: https://github.com/cockroachdb/cockroach/pull/61297 -[#61300]: https://github.com/cockroachdb/cockroach/pull/61300 -[#61315]: https://github.com/cockroachdb/cockroach/pull/61315 -[#61321]: https://github.com/cockroachdb/cockroach/pull/61321 -[#61324]: https://github.com/cockroachdb/cockroach/pull/61324 -[#61326]: https://github.com/cockroachdb/cockroach/pull/61326 -[#61344]: https://github.com/cockroachdb/cockroach/pull/61344 -[#61345]: https://github.com/cockroachdb/cockroach/pull/61345 -[#61348]: https://github.com/cockroachdb/cockroach/pull/61348 -[#61353]: https://github.com/cockroachdb/cockroach/pull/61353 -[#61356]: https://github.com/cockroachdb/cockroach/pull/61356 -[#61358]: https://github.com/cockroachdb/cockroach/pull/61358 -[#61361]: https://github.com/cockroachdb/cockroach/pull/61361 -[#61362]: https://github.com/cockroachdb/cockroach/pull/61362 -[#61376]: https://github.com/cockroachdb/cockroach/pull/61376 -[#61386]: https://github.com/cockroachdb/cockroach/pull/61386 -[#61387]: https://github.com/cockroachdb/cockroach/pull/61387 -[#61410]: https://github.com/cockroachdb/cockroach/pull/61410 -[#61416]: https://github.com/cockroachdb/cockroach/pull/61416 -[#61427]: https://github.com/cockroachdb/cockroach/pull/61427 -[#61428]: https://github.com/cockroachdb/cockroach/pull/61428 -[#61494]: https://github.com/cockroachdb/cockroach/pull/61494 -[#61499]: https://github.com/cockroachdb/cockroach/pull/61499 -[#61514]: https://github.com/cockroachdb/cockroach/pull/61514 -[#61522]: https://github.com/cockroachdb/cockroach/pull/61522 -[#61523]: https://github.com/cockroachdb/cockroach/pull/61523 -[#61601]: https://github.com/cockroachdb/cockroach/pull/61601 -[#61647]: https://github.com/cockroachdb/cockroach/pull/61647 -[#61676]: https://github.com/cockroachdb/cockroach/pull/61676 -[#61709]: https://github.com/cockroachdb/cockroach/pull/61709 -[#61723]: https://github.com/cockroachdb/cockroach/pull/61723 -[#61733]: https://github.com/cockroachdb/cockroach/pull/61733 -[#61754]: https://github.com/cockroachdb/cockroach/pull/61754 -[#61805]: https://github.com/cockroachdb/cockroach/pull/61805 -[#61813]: https://github.com/cockroachdb/cockroach/pull/61813 -[#61826]: https://github.com/cockroachdb/cockroach/pull/61826 -[#61842]: https://github.com/cockroachdb/cockroach/pull/61842 -[#61844]: https://github.com/cockroachdb/cockroach/pull/61844 -[#61871]: https://github.com/cockroachdb/cockroach/pull/61871 -[#61876]: https://github.com/cockroachdb/cockroach/pull/61876 -[#61888]: https://github.com/cockroachdb/cockroach/pull/61888 -[#61909]: https://github.com/cockroachdb/cockroach/pull/61909 -[#61947]: https://github.com/cockroachdb/cockroach/pull/61947 -[#61949]: https://github.com/cockroachdb/cockroach/pull/61949 -[#61959]: https://github.com/cockroachdb/cockroach/pull/61959 -[#61962]: https://github.com/cockroachdb/cockroach/pull/61962 diff --git a/src/current/_includes/releases/v21.1/v21.1.0-beta.2.md b/src/current/_includes/releases/v21.1/v21.1.0-beta.2.md deleted file mode 100644 index 994aafe4e33..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.0-beta.2.md +++ /dev/null @@ -1,95 +0,0 @@ -## v21.1.0-beta.2 - -Release Date: March 30, 2021 - - - -

SQL language changes

- -

Multi-region changes

- -- Added validation that prevents users from updating the [zone configurations](https://www.cockroachlabs.com/docs/v21.1/configure-replication-zones) of [multi-region tables](https://www.cockroachlabs.com/docs/v21.1/multiregion-overview) without first setting the `override_multi_region_zone_config` [session variable](https://www.cockroachlabs.com/docs/v21.1/set-vars). [#62119][#62119] -- Discarding a zone configuration from a multi-region enabled entity is blocked behind the `override_multi_region_zone_config` [session variable](https://www.cockroachlabs.com/docs/v21.1/set-vars). [#62159][#62159] -- Reverted the change that added the `FORCE` keyword in [#61499][#61499] in favor of the `override_multi_region_zone_config` session variable. [#62119][#62119] -- Setting non-[multi-region](https://www.cockroachlabs.com/docs/v21.1/multiregion-overview) controlled fields on [zone configs](https://www.cockroachlabs.com/docs/v21.1/configure-replication-zones) before `ALTER DATABASE ... SET PRIMARY REGION` will now be preserved and have the same value after the `SET PRIMARY REGION` command is issued. [#62162][#62162] -- [Materialized views](https://www.cockroachlabs.com/docs/v21.1/views#materialized-views) in multi-region databases will now have a [`GLOBAL` table locality](https://www.cockroachlabs.com/docs/v21.1/set-locality#set-the-table-locality-to-global). [#62194][#62194] -- [Materialized views](https://www.cockroachlabs.com/docs/v21.1/views#materialized-views) which are in a database before the first [`ADD REGION`](https://www.cockroachlabs.com/docs/v21.1/add-region) will become [`GLOBAL`](https://www.cockroachlabs.com/docs/v21.1/set-locality#set-the-table-locality-to-global) on `ADD REGION`, in line with the behavior of `CREATE MATERIALIZED VIEW` on a [multi-region](https://www.cockroachlabs.com/docs/v21.1/multiregion-overview) database. [#62194][#62194] -- `ALTER DATABASE .. SET PRIMARY REGION` now requires both `CREATE` and `ZONECONFIG` privilege on all objects inside the database when [adding the first region to the database](https://www.cockroachlabs.com/docs/v21.1/add-region#examples). The same behavior applies for dropping the last region using `ALTER DATABASE ... DROP REGION`. [#62450][#62450] -- Removed the experimental multi-region locality syntaxes. [#62114][#62114] - -

General changes

- -- CockroachDB now stores information about contention on non-SQL keys. [#62041][#62041] -- Statement [diagnostics bundles](https://www.cockroachlabs.com/docs/v21.1/explain-analyze#debug-option) now contain output of `EXPLAIN (VEC)` and `EXPLAIN (VEC, VERBOSE)` commands for the statements. [#62049][#62049] -- Sampled execution stats are now available through [`crdb_internal.node_{statement,transaction}_statistics`](https://www.cockroachlabs.com/docs/v21.1/crdb-internal). [#62089][#62089] -- Increased the default value for the `sql.txn_stats.sample_rate` [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) from 0 to 0.1. This means that from now on every statement has 10% probability of being sampled for the purposes of execution statistics. Note that no other criteria for sampling (such as query latency) are currently being utilized to decide whether to sample a statement or not. [#61815][#61815] -- Added the following [cluster settings](https://www.cockroachlabs.com/docs/v21.1/cluster-settings): `sql.defaults.statement_timeout`, which controls the default value for the `statement_timeout` [session setting](https://www.cockroachlabs.com/docs/v21.1/set-vars); `sql.defaults.idle_in_transaction_session_timeout`, which controls the default value for the `idle_in_transaction_session_timeout` timeout setting; `sql.defaults.idle_in_session_timeout`, which already existed, but is now a public cluster setting. [#62182][#62182] -- [`EXPLAIN`](https://www.cockroachlabs.com/docs/v21.1/explain) and [`EXPLAIN ANALYZE`](https://www.cockroachlabs.com/docs/v21.1/explain-analyze) now show how long ago [table statistics](https://www.cockroachlabs.com/docs/v21.1/cost-based-optimizer#table-statistics) were collected. [#61945][#61945] - -

Command-line changes

- -- Changed the formatting of namespace validation failures in `cockroach debug doctor` output. [#62245][#62245] - -

Bug fixes

- -- Fixed a bug where the `target` column of `crdb_internal.zones` would show names without properly accounting for user-defined schemas. [#62022][#62022] -- Added validation that prevents regions being dropped on [multi-region databases](https://www.cockroachlabs.com/docs/v21.1/multiregion-overview) when there are <= 3 regions left on the database. [#62162][#62162] -- Fixed a bug where [zone configurations](https://www.cockroachlabs.com/docs/v21.1/configure-replication-zones) were not being correctly dropped on the final `DROP REGION` of a multi-region database. [#62162][#62162] -- Fixed a bug where [`VIEW`s](https://www.cockroachlabs.com/docs/v21.1/views) and [`SEQUENCE`s](https://www.cockroachlabs.com/docs/v21.1/create-sequence) were not being allowed in multi-region databases. They will now default to the [`REGIONAL BY TABLE`](https://www.cockroachlabs.com/docs/v21.1/set-locality#set-the-table-locality-to-regional-by-table) locality. [#62176][#62176] -- Fixed a bug where the `pg_type_is_visible` built-in function did not correctly handle user-defined types. [#62225][#62225] -- Fixed a bug where casting an `OID` to a `regtype` did not work for user-defined types. [#62225][#62225] -- A Raft leader who loses quorum will now relinquish its range lease and remove the replica if the range is recreated elsewhere, e.g., via `Node.ResetQuorum()`. [#62103][#62103] -- Fixed a bug where `ClearRange` could leave behind stray write intents when separated intents were enabled, which could cause subsequent storage errors. [#62104][#62104] -- [`ALTER TABLE`](https://www.cockroachlabs.com/docs/v21.1/alter-table), [`ALTER VIEW`](https://www.cockroachlabs.com/docs/v21.1/alter-view), and [`ALTER SEQUENCE`](https://www.cockroachlabs.com/docs/v21.1/alter-sequence) can no longer be used to incorrectly create cross-DB references. [#62341][#62341] -- Disallowed adding columns of type `OIDVECTOR` or `INT2VECTOR` to a table in [`ALTER TABLE ... ADD COLUMN`](https://www.cockroachlabs.com/docs/v21.1/add-column) statements. These types are not allowed in user-created tables via [`CREATE TABLE`](https://www.cockroachlabs.com/docs/v21.1/create-table) and were previously erroneously allowed in `ALTER TABLE ... ADD COLUMN`. [#62180][#62180] -- CockroachDB now logs all unsupported `pgdump` statements across smaller log files that can be found in the subdirectory `import/(unsupported_schema_stmts|unsupported_data_stmts)/.log` [#62263][#62263] -- Fixed a bug where a [constraint](https://www.cockroachlabs.com/docs/v21.1/constraints) like `NOT NULL` or `CHECK` on a column made irrelevant by a [`DROP CONSTRAINT`](https://www.cockroachlabs.com/docs/v21.1/drop-constraint) statement in a later concurrent transaction would lead to errors / incorrect behaviour. [#62249][#62249] -- Fixed an internal error that could occur when [`REGIONAL BY ROW`](https://www.cockroachlabs.com/docs/v21.1/set-locality) tables were joined with other tables using a lookup or inverted join. The internal error was `"we expect that limited UNION ALL queries are only planned locally"`. [#62383][#62383] -- Fixed a bug where using `DROP REGION` on the last region of a multi-region database would not delete the global [zone configurations](https://www.cockroachlabs.com/docs/v21.1/configure-replication-zones) for [`GLOBAL` tables](https://www.cockroachlabs.com/docs/v21.1/set-locality#set-the-table-locality-to-global). [#62220][#62220] -- Fixed a bug where duplicate [`IMPORT`](https://www.cockroachlabs.com/docs/v21.1/import) job records may have been created, or `IMPORT` statements may have failed, when the actual job succeeded. [#62396][#62396] -- Fixed a bug where CockroachDB could collect execution statistics prematurely, which would result in incorrect stats (e.g., when running [`EXPLAIN ANALYZE`](https://www.cockroachlabs.com/docs/v21.1/explain-analyze)). [#62384][#62384] -- Fixed a bug where setting the `kv.closed_timestamp.target_duration` to 0 did not disable routing requests to [follower replicas](https://www.cockroachlabs.com/docs/v21.1/follower-reads). [#62439][#62439] -- Fixed a bug where a failed [restore from a backup](https://www.cockroachlabs.com/docs/v21.1/restore) including [user defined types](https://www.cockroachlabs.com/docs/v21.1/create-type) would require manual cleanup. [#62454][#62454] - -
- -

Contributors

- -This release includes 61 merged PRs by 21 authors. -We would like to thank the following contributors from the CockroachDB community: - -- Tharun - -
- -[#61499]: https://github.com/cockroachdb/cockroach/pull/61499 -[#61815]: https://github.com/cockroachdb/cockroach/pull/61815 -[#61945]: https://github.com/cockroachdb/cockroach/pull/61945 -[#62022]: https://github.com/cockroachdb/cockroach/pull/62022 -[#62041]: https://github.com/cockroachdb/cockroach/pull/62041 -[#62049]: https://github.com/cockroachdb/cockroach/pull/62049 -[#62089]: https://github.com/cockroachdb/cockroach/pull/62089 -[#62103]: https://github.com/cockroachdb/cockroach/pull/62103 -[#62104]: https://github.com/cockroachdb/cockroach/pull/62104 -[#62114]: https://github.com/cockroachdb/cockroach/pull/62114 -[#62119]: https://github.com/cockroachdb/cockroach/pull/62119 -[#62159]: https://github.com/cockroachdb/cockroach/pull/62159 -[#62162]: https://github.com/cockroachdb/cockroach/pull/62162 -[#62176]: https://github.com/cockroachdb/cockroach/pull/62176 -[#62180]: https://github.com/cockroachdb/cockroach/pull/62180 -[#62182]: https://github.com/cockroachdb/cockroach/pull/62182 -[#62194]: https://github.com/cockroachdb/cockroach/pull/62194 -[#62220]: https://github.com/cockroachdb/cockroach/pull/62220 -[#62225]: https://github.com/cockroachdb/cockroach/pull/62225 -[#62245]: https://github.com/cockroachdb/cockroach/pull/62245 -[#62249]: https://github.com/cockroachdb/cockroach/pull/62249 -[#62263]: https://github.com/cockroachdb/cockroach/pull/62263 -[#62341]: https://github.com/cockroachdb/cockroach/pull/62341 -[#62383]: https://github.com/cockroachdb/cockroach/pull/62383 -[#62384]: https://github.com/cockroachdb/cockroach/pull/62384 -[#62396]: https://github.com/cockroachdb/cockroach/pull/62396 -[#62409]: https://github.com/cockroachdb/cockroach/pull/62409 -[#62412]: https://github.com/cockroachdb/cockroach/pull/62412 -[#62439]: https://github.com/cockroachdb/cockroach/pull/62439 -[#62450]: https://github.com/cockroachdb/cockroach/pull/62450 -[#62454]: https://github.com/cockroachdb/cockroach/pull/62454 diff --git a/src/current/_includes/releases/v21.1/v21.1.0-beta.3.md b/src/current/_includes/releases/v21.1/v21.1.0-beta.3.md deleted file mode 100644 index 4f89bde62f0..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.0-beta.3.md +++ /dev/null @@ -1,71 +0,0 @@ -## v21.1.0-beta.3 - -Release Date: April 12, 2021 - - - -

Enterprise edition changes

- -- The `WITH avro_schema_prefix` option for Avro [changefeeds](https://www.cockroachlabs.com/docs/v21.1/create-changefeed) now sets `schema.namespace` [#61734][#61734] {% comment %}doc{% endcomment %} -- CockroachDB now fails fast when [Change Data Capture](https://www.cockroachlabs.com/docs/v21.1/stream-data-out-of-cockroachdb-using-changefeeds) writes are blocked. [#62756][#62756] {% comment %}doc{% endcomment %} - -

SQL language changes

- -

Multi-region SQL changes

- -- Users can now use a multi-region [`ALTER DATABASE`](https://www.cockroachlabs.com/docs/v21.1/alter-database) command if: - - - The user is an [`admin`](https://www.cockroachlabs.com/docs/v21.1/authorization#admin-role) user - - The user is the owner of the database. - - The user has [`CREATE`](https://www.cockroachlabs.com/docs/v21.1/authorization#privileges) privileges on the database. [#62528][#62528] {% comment %}doc{% endcomment %} -- Availability zones are now ordered when using the `SHOW REGIONS` set of commands. [#62619][#62619] {% comment %}doc{% endcomment %} - -

General SQL changes

- -- Added the `stub_catalog_tables` [session variable](https://www.cockroachlabs.com/docs/v21.1/set-vars), which is enabled by default. If disabled, querying an unimplemented [`pg_catalog`](https://www.cockroachlabs.com/docs/v21.1/pg-catalog) table will result in an error, as is the case in v20.2 and earlier. Otherwise, the query will simply return no rows. [#62621][#62621] {% comment %}doc{% endcomment %} - -

DB Console changes

- -- The [**Statements** page](https://www.cockroachlabs.com/docs/v21.1/ui-statements-page) now shows internal statements when the *all* filter option is selected. [#62677][#62677] {% comment %}doc{% endcomment %} - -

Bug fixes

- -- Fixed a bug that in rare circumstances could cause an implicitly committed (`STAGING`) transaction to be uncommitted if any unresolved intents were removed by a range clear (e.g., when cleaning up a dropped table). This bug fix is only effective with separated intents, which are disabled by default. [#62376][#62376] -- Added a `DuplicateObject` error code for when a user attempts to `ADD REGION` to a database where the region already exists. [#62491][#62491] -- Fixed an internal error that could occur during planning for queries involving tables with many columns and at least one [GIN index](https://www.cockroachlabs.com/docs/v21.1/inverted-indexes). The error, "estimated distinct count must be non-zero", was caused by an invalid pointer access in the cardinality estimation code. This has now been fixed. [#62545][#62545] -- Writing files to `userfile` would sometimes result in an error claiming that the `userfile` table already exists. This is now fixed. [#62544][#62544] -- When adding/dropping regions from a multi-region database, the user must now have privileges on all regional-by-row tables as these are implicitly re-partitioned under the hood. [#62612][#62612] -- Fixed an internal error caused by comparing collation names that had different upper/lower case characters. [#62637][#62637] -- Fixed a bug whereby [`ENUM`](https://www.cockroachlabs.com/docs/v21.1/enum) types which have large numbers of values would cause unexpected errors when attempting to read from tables with columns using that `ENUM` type. [#62210][#62210] -- Fixed a bug introduced in earlier v21.1 alpha releases which could cause panics when [dropping indexes](https://www.cockroachlabs.com/docs/v21.1/drop-index) on tables partitioned by [user-defined types](https://www.cockroachlabs.com/docs/v21.1/enum). [#62725][#62725] -- Fixed a bug from earlier v21.1 alpha releases whereby dropping an index on a table partitioned by a user-defined type and then [dropping the table](https://www.cockroachlabs.com/docs/v21.1/drop-table) and then [dropping the type](https://www.cockroachlabs.com/docs/v21.1/drop-type) before the GC TTL for the index has expired could result in a crash. [#62725][#62725] - -

Performance improvements

- -- Improved the performance of the [vectorized engine](https://www.cockroachlabs.com/docs/v21.1/vectorized-execution) when scanning fewer than 1024 rows at a time. [#62365][#62365] -- Improved logic in determining the configuration for data to avoid expensive work when there are a large number of [user-defined schemas](https://www.cockroachlabs.com/docs/v21.1/create-schema). [#62577][#62577] -- Addressed a performance regression from a past change regarding read-triggered compactions. [#62676][#62676] - -

Contributors

- -This release includes 37 merged PRs by 23 authors. - -[#61734]: https://github.com/cockroachdb/cockroach/pull/61734 -[#62210]: https://github.com/cockroachdb/cockroach/pull/62210 -[#62365]: https://github.com/cockroachdb/cockroach/pull/62365 -[#62376]: https://github.com/cockroachdb/cockroach/pull/62376 -[#62491]: https://github.com/cockroachdb/cockroach/pull/62491 -[#62528]: https://github.com/cockroachdb/cockroach/pull/62528 -[#62544]: https://github.com/cockroachdb/cockroach/pull/62544 -[#62545]: https://github.com/cockroachdb/cockroach/pull/62545 -[#62577]: https://github.com/cockroachdb/cockroach/pull/62577 -[#62606]: https://github.com/cockroachdb/cockroach/pull/62606 -[#62612]: https://github.com/cockroachdb/cockroach/pull/62612 -[#62619]: https://github.com/cockroachdb/cockroach/pull/62619 -[#62621]: https://github.com/cockroachdb/cockroach/pull/62621 -[#62637]: https://github.com/cockroachdb/cockroach/pull/62637 -[#62676]: https://github.com/cockroachdb/cockroach/pull/62676 -[#62677]: https://github.com/cockroachdb/cockroach/pull/62677 -[#62725]: https://github.com/cockroachdb/cockroach/pull/62725 -[#62733]: https://github.com/cockroachdb/cockroach/pull/62733 -[#62756]: https://github.com/cockroachdb/cockroach/pull/62756 diff --git a/src/current/_includes/releases/v21.1/v21.1.0-beta.4.md b/src/current/_includes/releases/v21.1/v21.1.0-beta.4.md deleted file mode 100644 index d1ef69146da..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.0-beta.4.md +++ /dev/null @@ -1,105 +0,0 @@ -## v21.1.0-beta.4 - -Release Date: April 19, 2021 - - - -

General changes

- -- Removed experimental feature `UNIQUE WITHOUT INDEX` from the documentation. [#63499][#63499] {% comment %}doc{% endcomment %} - -

SQL language changes

- -- The [`pg_get_partkeydef` built-in function](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators) is now implemented by always returning `NULL`. [#63149][#63149] {% comment %}doc{% endcomment %} -- CockroachDB now collects execution stats for all statements when seen for the first time. To disable this behavior, set the [`sql.txn_stats.sample_rate` cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) to 0, which will disable all execution stats collection. [#63325][#63325] {% comment %}doc{% endcomment %} -- CockroachDB will now block the ability to set the initial `PRIMARY REGION` of a database if any [multi-region](https://www.cockroachlabs.com/docs/v21.1/multiregion-overview) fields on any zone configs in the database have been set. [#63354][#63354] {% comment %}doc{% endcomment %} -- CockroachDB now introduces a `pgcode` when attempting to [`DROP REGION`](https://www.cockroachlabs.com/docs/v21.1/multiregion-overview) when the region being dropped is the `PRIMARY REGION`. [#63354][#63354] {% comment %}doc{% endcomment %} -- Replaced the word "tuple" with its more user-friendly synonym "row" in [vectorized stats outputs](https://www.cockroachlabs.com/docs/v21.1/vectorized-execution). [#62956][#62956] {% comment %}doc{% endcomment %} -- Changed [`BACKUP`](https://www.cockroachlabs.com/docs/v21.1/take-full-and-incremental-backups) of [interleaved tables](https://www.cockroachlabs.com/docs/v21.1/interleave-in-parent) to require the `include_deprecated_interleaves` option as interleaved table backups will not be able to be restored in future versions of CockroachDB. [#63501][#63501] {% comment %}doc{% endcomment %} - -

Operational changes

- -- `RESTORE` now cannot [restore](https://www.cockroachlabs.com/docs/v21.1/take-full-and-incremental-backups) `BACKUP`s made by newer versions of CockroachDB. [#62398][#62398] {% comment %}doc{% endcomment %} - -

DB Console changes

- -The following statements now render correctly as events in the [DB Console](https://www.cockroachlabs.com/docs/v21.1/ui-overview) [#63141][#63141]: -- [`ALTER DATABASE ADD REGION`](https://www.cockroachlabs.com/docs/v21.1/add-region) -- [`ALTER DATABASE SET PRIMARY REGION`](https://www.cockroachlabs.com/docs/v21.1/set-locality) -- [`ALTER DATABASE ... SURVIVE ... FAILURE`](https://www.cockroachlabs.com/docs/v21.1/survive-failure) -- [`ALTER DATABASE DROP REGION`](https://www.cockroachlabs.com/docs/v21.1/drop-region) -- [`CREATE TYPE`](https://www.cockroachlabs.com/docs/v21.1/create-type) -- [`ALTER TYPE`](https://www.cockroachlabs.com/docs/v21.1/alter-type) -- [`DROP TYPE`](https://www.cockroachlabs.com/docs/v21.1/drop-type) - -

Bug fixes

- -- Fixed a bug present in earlier 21.1 versions where [`BACKUP`s](https://www.cockroachlabs.com/docs/v21.1/take-full-and-incremental-backups) would produce an error when they should be able to backup the underlying data. [#63095][#63095] -- [Dropping a foreign key](https://www.cockroachlabs.com/docs/v21.1/drop-constraint) that was added in the same transaction no longer triggers an internal error. This bug has been present since at least version 20.1. [#62879][#62879] -- Fixed a bug where an `ALTER TABLE ... ADD COLUMN ... UNIQUE` statement would cause an error if the table had a [`PARTITION ALL BY` or `REGIONAL BY ROW`](https://www.cockroachlabs.com/docs/v21.1/multiregion-overview) definition. [#63189][#63189] -- Fixed a bug in earlier 21.1 versions where `CREATE TABLE LIKE` would copy a [`VIRTUAL` column](https://www.cockroachlabs.com/docs/v21.1/computed-columns) from the source table as a `STORED` column in the destination table. [#63172][#63172] -- CockroachDB now returns an error when trying to perform a [backup](https://www.cockroachlabs.com/docs/v21.1/take-full-and-incremental-backups) of a cluster that was taken on another tenant. [#63223][#63223] -- Fixed a bug where index backfill data may have been missed by [`BACKUP`](https://www.cockroachlabs.com/docs/v21.1/take-full-and-incremental-backups) in incremental backups. [#63221][#63221] -- Fixed a bug where [`REGIONAL BY ROW` zone configs](https://www.cockroachlabs.com/docs/v21.1/multiregion-overview) were dropped before `REGIONAL BY ROW` changes are finalized. This caused a bug when the `REGIONAL BY ROW` transformation fail. [#63274][#63274] -- Fixed a case where implicitly partitioned columns (e.g., from [`REGIONAL BY ROW` tables](https://www.cockroachlabs.com/docs/v21.1/multiregion-overview) and hash-sharded indexes) previously showed as `implicit = false` when using `SHOW INDEXES` or querying `information_schema.pg_indexes`. [#63275][#63275] -- Fixed an error that could occur when performing an [`UPSERT`](https://www.cockroachlabs.com/docs/v21.1/upsert) on a [`REGIONAL BY ROW` table](https://www.cockroachlabs.com/docs/v21.1/multiregion-overview) with no secondary indexes or foreign keys. The error, 'missing "crdb_region" primary key column', has now been fixed. [#63257][#63257] -- Fixed a bug where tables that were created by CockroachDB 19.x or older that included foreign key constraints and were [backed up](https://www.cockroachlabs.com/docs/v21.1/take-full-and-incremental-backups) with the `revision_history` option would be malformed when restored by a CockroachDB 20.x cluster if the `RESTORE` used the `AS OF SYSTEM TIME` option. [#63267][#63267] -- Fixed a bug in [user-defined schemas](https://www.cockroachlabs.com/docs/v21.1/schema-design-schema) where the dropping of any schema would prevent creation of schemas with the name of the database and would corrupt existing schemas of that name. [#63395][#63395] -- Fixed a bug in previous CockroachDB 21.1 releases where CockroachDB would sometimes return the output in an incorrect order if a query containing hash-aggregation was executed via the [vectorized execution engine](https://www.cockroachlabs.com/docs/v21.1/vectorized-execution) and spilled to temporary storage. [#63408][#63408] -- Fixed a bug where [incremental cluster backups](https://www.cockroachlabs.com/docs/v21.1/take-full-and-incremental-backups) may have missed data written to tables while they were `OFFLINE`. In practice this could have occurred if a [`RESTORE`](https://www.cockroachlabs.com/docs/v21.1/restore) or [`IMPORT`](https://www.cockroachlabs.com/docs/v21.1/import) was running across incremental backups. [#63304][#63304] -- CockroachDB now includes more anonymized data from SQL statements in telemetry updates and crash reports. [#63482][#63482] -- Fixed a rare issue that caused replica divergence. If this occurred the divergence was reported by the [replica consistency checker](https://www.cockroachlabs.com/docs/v21.1/architecture/replication-layer), typically within 24 hours of occurrence, and caused the nodes to terminate. [#63473][#63473] - -

Performance improvements

- -- Improved performance of reverting [`IMPORT INTO`](https://www.cockroachlabs.com/docs/v21.1/import-into) jobs that `IMPORT INTO` empty tables. [#63220][#63220] - -

Miscellaneous improvements

- -- Made the Kafka library used in [changefeeds](https://www.cockroachlabs.com/docs/v21.1/stream-data-out-of-cockroachdb-using-changefeeds) configurable using the `kafka_sink_config` option to enable latency versus throughput configurations. [#63361][#63361] -- Connected the [changefeed](https://www.cockroachlabs.com/docs/v21.1/stream-data-out-of-cockroachdb-using-changefeeds) memory monitor to the parent SQL monitor to ensure that changefeeds do not try to use more memory than is available to the SQL server. [#63409][#63409] - -

Contributors

- -This release includes 50 merged PRs by 20 authors. - -[#62398]: https://github.com/cockroachdb/cockroach/pull/62398 -[#62879]: https://github.com/cockroachdb/cockroach/pull/62879 -[#62956]: https://github.com/cockroachdb/cockroach/pull/62956 -[#62968]: https://github.com/cockroachdb/cockroach/pull/62968 -[#62971]: https://github.com/cockroachdb/cockroach/pull/62971 -[#63095]: https://github.com/cockroachdb/cockroach/pull/63095 -[#63141]: https://github.com/cockroachdb/cockroach/pull/63141 -[#63149]: https://github.com/cockroachdb/cockroach/pull/63149 -[#63172]: https://github.com/cockroachdb/cockroach/pull/63172 -[#63189]: https://github.com/cockroachdb/cockroach/pull/63189 -[#63220]: https://github.com/cockroachdb/cockroach/pull/63220 -[#63221]: https://github.com/cockroachdb/cockroach/pull/63221 -[#63223]: https://github.com/cockroachdb/cockroach/pull/63223 -[#63257]: https://github.com/cockroachdb/cockroach/pull/63257 -[#63267]: https://github.com/cockroachdb/cockroach/pull/63267 -[#63274]: https://github.com/cockroachdb/cockroach/pull/63274 -[#63275]: https://github.com/cockroachdb/cockroach/pull/63275 -[#63304]: https://github.com/cockroachdb/cockroach/pull/63304 -[#63325]: https://github.com/cockroachdb/cockroach/pull/63325 -[#63354]: https://github.com/cockroachdb/cockroach/pull/63354 -[#63361]: https://github.com/cockroachdb/cockroach/pull/63361 -[#63395]: https://github.com/cockroachdb/cockroach/pull/63395 -[#63402]: https://github.com/cockroachdb/cockroach/pull/63402 -[#63403]: https://github.com/cockroachdb/cockroach/pull/63403 -[#63408]: https://github.com/cockroachdb/cockroach/pull/63408 -[#63409]: https://github.com/cockroachdb/cockroach/pull/63409 -[#63473]: https://github.com/cockroachdb/cockroach/pull/63473 -[#63482]: https://github.com/cockroachdb/cockroach/pull/63482 -[#63499]: https://github.com/cockroachdb/cockroach/pull/63499 -[#63501]: https://github.com/cockroachdb/cockroach/pull/63501 -[1c89925eb]: https://github.com/cockroachdb/cockroach/commit/1c89925eb -[32b5b8587]: https://github.com/cockroachdb/cockroach/commit/32b5b8587 -[33816b3fd]: https://github.com/cockroachdb/cockroach/commit/33816b3fd -[56088535f]: https://github.com/cockroachdb/cockroach/commit/56088535f -[57b9589e9]: https://github.com/cockroachdb/cockroach/commit/57b9589e9 -[6394ff543]: https://github.com/cockroachdb/cockroach/commit/6394ff543 -[6ebecfd38]: https://github.com/cockroachdb/cockroach/commit/6ebecfd38 -[71cacc783]: https://github.com/cockroachdb/cockroach/commit/71cacc783 -[abc4eb5ac]: https://github.com/cockroachdb/cockroach/commit/abc4eb5ac -[fc7249f82]: https://github.com/cockroachdb/cockroach/commit/fc7249f82 diff --git a/src/current/_includes/releases/v21.1/v21.1.0-beta.5.md b/src/current/_includes/releases/v21.1/v21.1.0-beta.5.md deleted file mode 100644 index a7cbcc0234c..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.0-beta.5.md +++ /dev/null @@ -1,88 +0,0 @@ -## v21.1.0-beta.5 - -Release Date: April 29, 2021 - - - -

Docker image

- -{% include copy-clipboard.html %} -~~~shell -$ docker pull cockroachdb/cockroach-unstable:v21.1.0-beta.5 -~~~ - -

Backward-incompatible changes

- -- The internal representation of the `voter_constraints` [zone configuration](https://www.cockroachlabs.com/docs/v21.1/configure-replication-zones) attribute (new in v21.1) has been altered in a way that is partially incompatible with the representation used by previous v21.1 betas (and the alphas that include this attribute). This means that users who directly set the `voter_constraints` attribute to an empty list will lose those constraints and will have to reset them. [#63674][#63674] {% comment %}doc{% endcomment %} - -

General changes

- -- Upgraded the CockroachDB binary to Go 1.15.10. [#63865][#63865] {% comment %}doc{% endcomment %} - -

Enterprise edition changes

- -- [Changefeeds](https://www.cockroachlabs.com/docs/v21.1/create-changefeed) will now fail on any [regional by row table](https://www.cockroachlabs.com/docs/v21.1/multiregion-overview#regional-by-row-tables) with an error, `CHANGEFEED cannot target REGIONAL BY ROW tables: .` This is to prevent unexpected behavior in changefeeds until they offer full support for this type of table. [#63542][#63542] {% comment %}doc{% endcomment %} - -

SQL language changes

- -- [`RESTORE`](https://www.cockroachlabs.com/docs/v21.1/restore) now re-validates restored indexes if they were restored from an [incremental backup](https://www.cockroachlabs.com/docs/v21.1/take-full-and-incremental-backups#incremental-backups) that was taken while the index was being created. [#63320][#63320] {% comment %}doc{% endcomment %} -- The `sql.distsql.temp_storage.workmem` [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) is now marked as public and is included in the documentation. It determines how much RAM a single operation of a single query can use before it must spill to temporary storage. Note the operations that do not support the disk spilling will ignore this setting and are subject only to the [`--max-sql-memory`](https://www.cockroachlabs.com/docs/v21.1/cockroach-start#flags) startup argument. [#63997][#63997] {% comment %}doc{% endcomment %} -- SQL executor data validation queries spawned by a schema change or a [`RESTORE`](https://www.cockroachlabs.com/docs/v21.1/restore) will now use [vectorized](https://www.cockroachlabs.com/docs/v21.1/vectorized-execution) query execution and [DistSQL](https://www.cockroachlabs.com/docs/v21.1/architecture/sql-layer#distsql) optimization if these are enabled in the [cluster settings](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) `sql.defaults.vectorize` and `sql.defaults.distsql`, respectively. This may improve the speed of these queries. [#64004][#64004] {% comment %}doc{% endcomment %} - -

Bug fixes

- -- Allow the leases of offline descriptors to be cached, preventing issues with lease acquisitions during bulk operations (backup and restore operations). [#63558][#63558] -- Fixed bugs where [`TRUNCATE`](https://www.cockroachlabs.com/docs/v21.1/truncate) concurrent with index construction and other schema changes could result in corruption. [#63143][#63143] -- Fixed a panic condition which could occur in cases after a [`RESTORE`](https://www.cockroachlabs.com/docs/v21.1/restore) of a table with [user-defined types](https://www.cockroachlabs.com/docs/v21.1/enum). [#63549][#63549] -- CockroachDB now prevents a panic condition and offers a graceful error when [spatial function](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators#spatial-functions) `ST_Segmentize` attempts to generate an extremely large number of points on a [`GEOGRAPHY`](https://www.cockroachlabs.com/docs/v21.1/spatial-glossary#geography). [#63758][#63758] -- Previously, running the `ST_Simplify` [spatial function](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators#spatial-functions) on a non-numeric value would cause the node to crash. This is now resolved. [#63797][#63797] -- CockroachDB now uses the existing primary key to validate indexes built for [`ALTER PRIMARY KEY`](https://www.cockroachlabs.com/docs/v21.1/alter-primary-key) changes. [#63609][#63609] -- Fixed occasional stalls and excessive CPU usage under macOS Big Sur when building CockroachDB with Go 1.14 or newer. [#63789][#63789] -- Fixed a bug where [crdb_internal.validate_multi_region_zone_configs()](https://github.com/cockroachdb/cockroach/blob/master/docs/generated/sql/functions.md#multi-region-functions) would fail during a [`REGIONAL BY ROW`](https://www.cockroachlabs.com/docs/v21.1/set-locality) locality transition. [#63834][#63834] -- Fixed an internal error that could occur when executing queries using a [GIN index](https://www.cockroachlabs.com/docs/v21.1/inverted-indexes). The error was an index out of range error, and could occur in rare cases when a filter or join predicate contained at least two JSON, Array, Geometry or Geography expressions that were combined with `AND`. This has now been fixed. [#63811][#63811] -- Fixed a bug leading to crashes with the error message `writing below closed ts`. [#63861][#63861] -- Previously, if a user altered a table to [`REGIONAL BY ROW`](https://www.cockroachlabs.com/docs/v21.1/set-locality) when a region was being dropped, and the drop failed and had to be rolled back, it could have resulted in the [regional by row table](https://www.cockroachlabs.com/docs/v21.1/multiregion-overview#regional-by-row-tables) missing a partition for this region. This is now fixed. [#63793][#63793] -- Prevent an internal error `use of enum metadata before hydration as an enum` when querying or showing ranges from tables with user-defined types as their `PRIMARY KEY`. [#63878][#63878] -- Fixed a theoretical issue in index backfills that could result in stale entries that would likely fail validation. [#64044][#64044] -- CockroachDB now correctly accounts for used memory when closing compressed files. [#63917][#63917] - -

Performance improvements

- -- CockroachDB now limits a series of heap allocations when serving read-only queries. [#63972][#63972] -- CockroachDB now limits the amount of memory that can be used in internal buffers for Kafka and cloud sinks. [#63611][#63611] - -
- -

Contributors

- -This release includes 48 merged PRs by 23 authors. -We would like to thank the following contributors from the CockroachDB community: - -- Miguel Novelo (first-time contributor) -- Rupesh Harode (first-time contributor) - -
- -[#63143]: https://github.com/cockroachdb/cockroach/pull/63143 -[#63320]: https://github.com/cockroachdb/cockroach/pull/63320 -[#63542]: https://github.com/cockroachdb/cockroach/pull/63542 -[#63549]: https://github.com/cockroachdb/cockroach/pull/63549 -[#63558]: https://github.com/cockroachdb/cockroach/pull/63558 -[#63609]: https://github.com/cockroachdb/cockroach/pull/63609 -[#63611]: https://github.com/cockroachdb/cockroach/pull/63611 -[#63674]: https://github.com/cockroachdb/cockroach/pull/63674 -[#63758]: https://github.com/cockroachdb/cockroach/pull/63758 -[#63768]: https://github.com/cockroachdb/cockroach/pull/63768 -[#63789]: https://github.com/cockroachdb/cockroach/pull/63789 -[#63793]: https://github.com/cockroachdb/cockroach/pull/63793 -[#63797]: https://github.com/cockroachdb/cockroach/pull/63797 -[#63811]: https://github.com/cockroachdb/cockroach/pull/63811 -[#63834]: https://github.com/cockroachdb/cockroach/pull/63834 -[#63861]: https://github.com/cockroachdb/cockroach/pull/63861 -[#63865]: https://github.com/cockroachdb/cockroach/pull/63865 -[#63878]: https://github.com/cockroachdb/cockroach/pull/63878 -[#63917]: https://github.com/cockroachdb/cockroach/pull/63917 -[#63949]: https://github.com/cockroachdb/cockroach/pull/63949 -[#63972]: https://github.com/cockroachdb/cockroach/pull/63972 -[#63997]: https://github.com/cockroachdb/cockroach/pull/63997 -[#64004]: https://github.com/cockroachdb/cockroach/pull/64004 diff --git a/src/current/_includes/releases/v21.1/v21.1.0-rc.1.md b/src/current/_includes/releases/v21.1/v21.1.0-rc.1.md deleted file mode 100644 index 11745912fbc..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.0-rc.1.md +++ /dev/null @@ -1,36 +0,0 @@ -## v21.1.0-rc.1 - -Release Date: May 5, 2021 - - - -

SQL language changes

- -- CockroachDB no longer allows [`ADD REGION`](https://www.cockroachlabs.com/docs/v21.1/add-region) or `DROP REGION` statements if a [`REGIONAL BY ROW`](https://www.cockroachlabs.com/docs/v21.1/multiregion-overview) table has index changes underway, or if a table is transitioning to or from `REGIONAL BY ROW`. [#64255][#64255] {% comment %}doc{% endcomment %} -- CockroachDB now prevents index modification on `REGIONAL BY ROW` tables and locality to or from `REGIONAL BY ROW` changes while an `ADD REGION` or `DROP REGION` statement is being executed. [#64255][#64255] {% comment %}doc{% endcomment %} - -

Bug fixes

- -- Fixed a scenario in which a rapid sequence of [range splits](https://www.cockroachlabs.com/docs/v21.1/architecture/distribution-layer#range-splits) could trigger a storm of [Raft snapshots](https://www.cockroachlabs.com/docs/v21.1/architecture/replication-layer#snapshots). This would be accompanied by log messages of the form "would have dropped incoming MsgApp, but allowing due to ..." and tended to occur as part of [`RESTORE`](https://www.cockroachlabs.com/docs/v21.1/restore)/[`IMPORT`](https://www.cockroachlabs.com/docs/v21.1/import) operations. [#64202][#64202] -- Read-write contention on [`GLOBAL` tables](https://www.cockroachlabs.com/docs/v21.1/multiregion-overview) no longer has a potential to thrash without making progress. [#64215][#64215] -- Previously, if a [`DROP INDEX`](https://www.cockroachlabs.com/docs/v21.1/drop-index) failed during a `REGIONAL BY ROW` transition, the index could have been re-inserted back into the `REGIONAL BY ROW` table but would be invalid if it was [hash-sharded](https://www.cockroachlabs.com/docs/v21.1/hash-sharded-indexes) or [partitioned](https://www.cockroachlabs.com/docs/v21.1/multiregion-overview). This bug is now fixed. [#64255][#64255] -- Fixed a rare bug present in [v21.1 beta versions]({% link releases/index.md %}#testing-releases) that could cause rapid range splits and merges on a `GLOBAL` table to lead to a stuck leaseholder replica. The situation is no longer possible. [#64304][#64304] -- Fixed a bug in previous v21.1 beta versions that allowed the store rebalancer to spuriously down-replicate a range during normal operation. [#64303][#64303] -- CockroachDB now prevents some out-of-memory conditions caused by [schema change](https://www.cockroachlabs.com/docs/v21.1/online-schema-changes) validations concurrent with other high-memory-use queries. [#64307][#64307] -- Fixed a bug present since [v21.1.0-alpha.1](#v21-1-0-alpha-1) that could cause cascading [`DELETE`](https://www.cockroachlabs.com/docs/v21.1/delete)s with [subqueries](https://www.cockroachlabs.com/docs/v21.1/subqueries) to error. [#64278][#64278] -- Fixed a bug that caused store information to be incorrectly redacted from the [CockroachDB logs](https://www.cockroachlabs.com/docs/v21.1/logging-overview), when logging was configured with redaction. [#64338][#64338] -- Previously, the remote flows of execution in the [vectorized engine](https://www.cockroachlabs.com/docs/v21.1/vectorized-execution) could take a long time to shut down if a node participating in the plan died. This bug is now fixed. [#64219][#64219] - -

Contributors

- -This release includes 14 merged PRs by 14 authors. - -[#64202]: https://github.com/cockroachdb/cockroach/pull/64202 -[#64215]: https://github.com/cockroachdb/cockroach/pull/64215 -[#64219]: https://github.com/cockroachdb/cockroach/pull/64219 -[#64255]: https://github.com/cockroachdb/cockroach/pull/64255 -[#64278]: https://github.com/cockroachdb/cockroach/pull/64278 -[#64303]: https://github.com/cockroachdb/cockroach/pull/64303 -[#64304]: https://github.com/cockroachdb/cockroach/pull/64304 -[#64307]: https://github.com/cockroachdb/cockroach/pull/64307 -[#64338]: https://github.com/cockroachdb/cockroach/pull/64338 diff --git a/src/current/_includes/releases/v21.1/v21.1.0-rc.2.md b/src/current/_includes/releases/v21.1/v21.1.0-rc.2.md deleted file mode 100644 index 10942680e02..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.0-rc.2.md +++ /dev/null @@ -1,26 +0,0 @@ -## v21.1.0-rc.2 - -Release Date: May 5, 2021 - - - -

Enterprise edition changes

- -- [Changefeeds](https://www.cockroachlabs.com/docs/v21.1/stream-data-out-of-cockroachdb-using-changefeeds) now reliably fail when [`IMPORT INTO`](https://www.cockroachlabs.com/docs/v21.1/import-into) is run against a targeted table, as change data capture is [not supported](https://www.cockroachlabs.com/docs/v21.1/known-limitations#change-data-capture) for this action. [#64372][#64372] - -

Bug fixes

- -- Fixed a correctness bug, which caused partitioned index scans to omit rows where the value of the first index column was `NULL`. This bug was present since v19.2.0. [#64046][#64046] -- [`IMPORT`](https://www.cockroachlabs.com/docs/v21.1/import) and [`RESTORE`](https://www.cockroachlabs.com/docs/v21.1/restore) jobs that were in progress during a cluster backup will now be canceled when that cluster backup is restored. This fixes a bug where these restored jobs may have assumed to make progress that was not captured in the backup. [#64352][#64352] -- Fixed a race condition where read-only requests during replica removal (for example, during range merges or rebalancing) could be evaluated on the removed replica, returning an empty result. [#64370][#64370] -- Fixed a bug where encryption-at-rest metadata was not synced and could become corrupted during a hard reset. [#64473][#64473] - -

Contributors

- -This release includes 5 merged PRs by 6 authors. - -[#64046]: https://github.com/cockroachdb/cockroach/pull/64046 -[#64352]: https://github.com/cockroachdb/cockroach/pull/64352 -[#64370]: https://github.com/cockroachdb/cockroach/pull/64370 -[#64372]: https://github.com/cockroachdb/cockroach/pull/64372 -[#64473]: https://github.com/cockroachdb/cockroach/pull/64473 diff --git a/src/current/_includes/releases/v21.1/v21.1.0.md b/src/current/_includes/releases/v21.1/v21.1.0.md deleted file mode 100644 index fe0163a1a2a..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.0.md +++ /dev/null @@ -1,124 +0,0 @@ -## v21.1.0 - -Release Date: May 18, 2021 - -With the release of CockroachDB v21.1, we've made a variety of flexibility, performance, and compatibility improvements. Check out a [summary of the most significant user-facing changes](#v21-1-0-feature-summary) and then [upgrade to CockroachDB v21.1](https://www.cockroachlabs.com/docs/v21.1/upgrade-cockroach-version). - -To learn more: - -- Read the [v21.1 blog post](https://www.cockroachlabs.com/blog/cockroachdb-21-1-release/). -- Join us on May 19 for a [livestream on why multi-region applications](https://www.cockroachlabs.com/webinars/the-cockroach-hour-multi-region/) matter and how our Product and Engineering teams partnered to make them simple in v21.1. - - - -

CockroachCloud

- -- Get a free v21.1 cluster on CockroachCloud. -- Learn about recent updates to CockroachCloud in the [CockroachCloud Release Notes]({% link releases/cloud.md %}). - -

Feature summary

- -This section summarizes the most significant user-facing changes in v21.1.0. For a complete list of features and changes, including bug fixes and performance improvements, see the [release notes]({% link releases/index.md %}#testing-releases) for previous testing releases. You can also search for [what's new in v21.1 in our docs](https://www.cockroachlabs.com/docs/search?query=new+in+v21.1). - -{{site.data.alerts.callout_info}} -"Core" features are freely available in the core version and do not require an enterprise license. "Enterprise" features require an [enterprise license](https://www.cockroachlabs.com/get-cockroachdb/enterprise/). [CockroachCloud clusters](https://cockroachlabs.cloud/) include all enterprise features. You can also use [`cockroach demo`](https://www.cockroachlabs.com/docs/v21.1/cockroach-demo) to test enterprise features in a local, temporary cluster. -{{site.data.alerts.end}} - -- [SQL](#v21-1-0-sql) -- [Recovery and I/O](#v21-1-0-recovery-and-i-o) -- [Database Operations](#v21-1-0-database-operations) -- [Backward-incompatible changes](#v21-1-0-backward-incompatible-changes) -- [Deprecations](#v21-1-0-deprecations) -- [Known limitations](#v21-1-0-known-limitations) -- [Education](#v21-1-0-education) - - - -

SQL

- -Version | Feature | Description ---------|---------|------------ -Enterprise | **Multi-Region Improvements** | It is now much easier to leverage CockroachDB's low-latency and resilient multi-region capabilities. For an introduction to the high-level concepts, see the [Multi-Region Overview](https://www.cockroachlabs.com/docs/v21.1/multiregion-overview). For further details and links to related SQL statements, see [Choosing a Multi-Region Configuration](https://www.cockroachlabs.com/docs/v21.1/choosing-a-multi-region-configuration), [When to Use ZONE vs. REGION Survival Goals](https://www.cockroachlabs.com/docs/v21.1/when-to-use-zone-vs-region-survival-goals), and [When to Use REGIONAL vs. GLOBAL Tables](https://www.cockroachlabs.com/docs/v21.1/when-to-use-regional-vs-global-tables). For a demonstration of these capabilities using a local cluster, see the [Multi-Region Tutorial](https://www.cockroachlabs.com/docs/v21.1/demo-low-latency-multi-region-deployment). Finally, for details about related architectural enhancements, see [Non-Voting Replicas](https://www.cockroachlabs.com/docs/v21.1/architecture/replication-layer#non-voting-replicas) and [Non-Blocking Transactions](https://www.cockroachlabs.com/docs/v21.1/architecture/transaction-layer#non-blocking-transactions). -Enterprise | **Automatic Follower Reads for Read-Only Transactions** | You can now force all read-only transactions in a session to use [follower reads](https://www.cockroachlabs.com/docs/v21.1/follower-reads) by setting the new [`default_transaction_use_follow_reads` session variable](https://www.cockroachlabs.com/docs/v21.1/show-vars#supported-variables) to `on`. -Core | **Query Observability Improvements** | [`EXPLAIN`](https://www.cockroachlabs.com/docs/v21.1/explain) and [`EXPLAIN ANALYZE`](https://www.cockroachlabs.com/docs/v21.1/explain-analyze) responses have been unified and extended with additional details, including automatic statistics-backed row estimates for `EXPLAIN`, and maximum memory usage, network usage, nodes used per operator, and rows used per operator for `EXPLAIN ANALYZE`. `EXPLAIN ANALYZE` now outputs a text-based statement plan tree by default, showing statistics about the statement processors at each phase of the statement.

The [Transactions Page](https://www.cockroachlabs.com/docs/v21.1/ui-transactions-page) and [Statements Page](https://www.cockroachlabs.com/docs/v21.1/ui-statements-page) of the DB Console also include such details as well the mean average time statements were in contention with other transactions within a specified time interval. The [SQL Dashboard](https://www.cockroachlabs.com/docs/v21.1/ui-sql-dashboard) has been expanded with additional graphs for latency, contention, memory, and network traffic. The [SQL Tuning with `EXPLAIN`](https://www.cockroachlabs.com/docs/v21.1/sql-tuning-with-explain) tutorial and [Optimize Statement Performance](https://www.cockroachlabs.com/docs/v21.1/make-queries-fast) guidance have been updated to leverage these improvements. -Core | **Inverted Joins** | CockroachDB now supports [inverted joins](https://www.cockroachlabs.com/docs/v21.1/joins#inverted-joins), which force the optimizer to use a GIN index on the right side of the join. Inverted joins can only be used with `INNER` and `LEFT` joins. -Core | **Partial GIN Indexes** | You can now create a [partial GIN index](https://www.cockroachlabs.com/docs/v21.1/inverted-indexes#partial-gin-indexes) on a subset of `JSON`, `ARRAY`, or geospatial container column data. -Core | **Virtual Computed Columns** | You can now create [virtual computed columns](https://www.cockroachlabs.com/docs/v21.1/computed-columns), which are not stored on disk and are recomputed as the column data in the expression changes. -Core | **Dropping Values in User-Defined Types** | It's now possible to [drop values in user-defined types](https://www.cockroachlabs.com/docs/v21.1/alter-type#drop-a-value-in-a-user-defined-type). -Core | **Sequence Caching** | You can now create a sequence with the [`CACHE`](https://www.cockroachlabs.com/docs/v21.1/create-sequence#cache-sequence-values-in-memory) keyword to have the sequence cache its values in memory. -Core | **Changing Sequence & View Ownership** | You can use the new `OWNER TO` parameter to change the owner of a [sequence](https://www.cockroachlabs.com/docs/v21.1/alter-sequence) or [view](https://www.cockroachlabs.com/docs/v21.1/alter-view). -Core | **Show `CREATE` Statements for the Current Database** | You can now use [`SHOW CREATE ALL TABLES`](https://www.cockroachlabs.com/docs/v21.1/show-create#show-the-statements-needed-to-recreate-all-tables-views-and-sequences-in-the-current-database) to return the `CREATE` statements for all of the tables, views, and sequences in the current database. -Core | **Storage of Z/M Coordinates for Spatial Objects** | You can now store a third dimension coordinate (`Z`), a measure coordinate (`M`), or both (`ZM`) with [spatial objects](https://www.cockroachlabs.com/docs/v21.1/spatial-features#spatial-objects). Note, however, that CockroachDB's [spatial indexing](https://www.cockroachlabs.com/docs/v21.1/spatial-indexes) is still based on the 2D coordinate system. This means that the Z/M dimension is not index accelerated when using spatial predicates, and some spatial functions ignore the Z/M dimension, with transformations discarding the Z/M value. -Core | **Third-Party Tool Support** | [Spatial libraries for Hibernate, ActiveRecord, and Django](https://www.cockroachlabs.com/docs/v21.1/spatial-data#orm-compatibility) are now fully compatible with CockroachDB's spatial features. The [DataGrip IDE](https://www.cockroachlabs.com/docs/v21.1/third-party-database-tools#integrated-development-environments-ides) and [Liquibase schema migration tool](https://www.cockroachlabs.com/docs/v21.1/third-party-database-tools#schema-migration-tools) are also now supported. -Core | **Connection Pooling Guidance** | Creating the appropriate size pool of connections is critical to gaining maximum performance in an application. For guidance on sizing, validating, and using connection pools with CockroachDB, as well as examples for Java and Go applications, see [Use Connection Pools](https://www.cockroachlabs.com/docs/v21.1/connection-pooling). -Core | **PostgreSQL 13 Compatibility** | CockroachDB is now wire-compatible with PostgreSQL 13. For more information, see [PostgreSQL Compatibility](https://www.cockroachlabs.com/docs/v21.1/postgresql-compatibility). - -

Recovery and I/O

- -Version | Feature | Description ---------|---------|------------ -Enterprise | **Changefeed Topic Naming Improvements** | New [`CHANGEFEED` options](https://www.cockroachlabs.com/docs/v21.1/create-changefeed#options) give you more control over topic naming: The `full_table_name` option lets you use a fully-qualified table name in topics, subjects, schemas, and record output instead of the default table name, and can prevent unintended behavior when the same table name is present in multiple databases. The `avro_schema_prefix` option lets you use a fully-qualified schema name for a table instead of the default table name, and makes it possible for multiple databases or clusters to share the same schema registry when the same table name is present in multiple databases. -Core | **Running Jobs Asynchronously** | You can use the new `DETACHED` option to run [`BACKUP`](https://www.cockroachlabs.com/docs/v21.1/backup#options), [`RESTORE`](https://www.cockroachlabs.com/docs/v21.1/restore#options), and [`IMPORT`](https://www.cockroachlabs.com/docs/v21.1/import#import-options) jobs asynchronously and receive a job ID immediately rather than waiting for the job to finish. This option enables you to run such jobs within transactions. -Core | **Import from Local Dump File** | The new [`cockroach import`](https://www.cockroachlabs.com/docs/v21.1/cockroach-import) command imports a database or table from a local `PGDUMP` or `MYSQLDUMP` file into a running cluster. This is useful for quick imports of 15MB or smaller. For larger imports, use the [`IMPORT`](https://www.cockroachlabs.com/docs/v21.1/import) statement. -Core | **Additional Import Options** | New [`IMPORT` options](https://www.cockroachlabs.com/docs/v21.1/import#import-options) give you more control over the import process's behavior: The `row_limit` option limits the number of rows to import, which is useful for finding errors quickly before executing a more time- and resource-consuming import; the `ignore_unsupported_statements` option ignores SQL statements in `PGDUMP` files that are unsupported by CockroachDB; and the `log_ignored_statements` option logs unsupported statements to cloud storage or userfile storage when `ignore_unsupported_statements` is enabled. -Core | **Re-validating Indexes During `RESTORE`** | Incremental backups created by v20.2.2 and prior v20.2.x releases or v20.1.4 and prior v20.1.x releases may include incomplete data for indexes that were in the process of being created. Therefore, when incremental backups taken by these versions are restored by v21.1.0, any indexes created during those incremental backups will be re-validated by [`RESTORE`](https://www.cockroachlabs.com/docs/v21.1/restore#restore-from-incremental-backups). - -

Database Operations

- -Version | Feature | Description ---------|---------|------------ -Core | **Logging Improvements** | Log events are now organized into [logging channels](https://www.cockroachlabs.com/docs/v21.1/logging-overview) that address different [use cases](https://www.cockroachlabs.com/docs/v21.1/logging-use-cases). Logging channels can be freely mapped to log sinks and routed to destinations outside CockroachDB (including external log collectors). All logging aspects, including message format (e.g., JSON), are now [configurable](https://www.cockroachlabs.com/docs/v21.1/configure-logs) via YAML. -Core | **Cluster API v2.0** | This [new API](https://www.cockroachlabs.com/docs/v21.1/cluster-api) for monitoring clusters and nodes builds on prior endpoints, offering a consistent REST interface that's easier to use with your choice of tooling. The API offers a streamlined authentication process and developer-friendly [reference documentation](https://www.cockroachlabs.com/docs/api/cluster/v2). -Core | **OpenShift-certified
Kubernetes Operator** | You can now [deploy CockroachDB on the Red Hat OpenShift platform](https://www.cockroachlabs.com/docs/v21.1/deploy-cockroachdb-with-kubernetes-openshift) using the latest OpenShift-certified Kubernetes Operator. -Core | **Auto TLS Certificate Setup** | Using the new [`cockroach connect` command](https://www.cockroachlabs.com/docs/v21.1/auto-tls), you can now let CockroachDB handle the creation and distribution among nodes of a cluster's CA (certificate authority) and node certificates. Note that this feature is an alpha release with core functionality that may not meet your requirements. -Core | **Built-in Timezone Library** | The CockroachDB binary now includes a copy of the [`tzdata` library](https://www.cockroachlabs.com/docs/v21.1/recommended-production-settings#dependencies), which is required by certain features that use time zone data. If CockroachDB cannot find the `tzdata` library externally, it will now use this built-in copy. - -

Backward-incompatible changes

- -Before [upgrading to CockroachDB v21.1](https://www.cockroachlabs.com/docs/v21.1/upgrade-cockroach-version), be sure to review the following backward-incompatible changes and adjust your deployment as necessary. - -- Rows containing empty arrays in [`ARRAY`](https://www.cockroachlabs.com/docs/v21.1/array) columns are now contained in [GIN indexes](https://www.cockroachlabs.com/docs/v21.1/inverted-indexes). This change is backward-incompatible because prior versions of CockroachDB will not be able to recognize and decode keys for empty arrays. Note that rows containing `NULL` values in an indexed column will still not be included in GIN indexes. -- Concatenation between a non-null argument and a null argument is now typed as string concatenation, whereas it was previously typed as array concatenation. This means that the result of `NULL || 1` will now be `NULL` instead of `{1}`. To preserve the old behavior, the null argument can be casted to an explicit type. -- The payload fields for certain event types in `system.eventlog` have been changed and/or renamed. Note that the payloads in `system.eventlog` were undocumented, so no guarantee was made about cross-version compatibility to this point. The list of changes includes (but is not limited to): - - `TargetID` has been renamed to `NodeID` for `node_join`. - - `TargetID` has been renamed to `TargetNodeID` for `node_decommissioning` / `node_decommissioned` / `node_recommissioned`. - - `NewDatabaseName` has been renamed to `NewDatabaseParent` for `convert_to_schema`. - - `grant_privilege` and `revoke_privilege` have been removed; they are replaced by `change_database_privilege`, `change_schema_privilege`, `change_type_privilege`, and `change_table_privilege`. Each event only reports a change for one user/role, so the `Grantees` field was renamed to `Grantee`. - - Each `drop_role` event now pertains to a single [user/role](https://www.cockroachlabs.com/docs/v21.1/authorization#sql-users). -- The connection and authentication logging enabled by the [cluster settings](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) `server.auth_log.sql_connections.enabled` and `server.auth_log.sql_sessions.enabled` was previously using a text format which was hard to parse and integrate with external monitoring tools. This has been changed to use the standard notable event mechanism, with standardized payloads. The output format is now structured; see the [reference documentation](https://www.cockroachlabs.com/docs/v21.1/eventlog) for details about the supported event types and payloads. -- The format for SQL audit, execution, and query logs has changed from a crude space-delimited format to JSON. To opt out of this new behavior and restore the pre-v21.1 logging format, you can set the [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) `sql.log.unstructured_entries.enabled` to `true`. -- The [`cockroach debug ballast`](https://www.cockroachlabs.com/docs/v21.1/cockroach-debug-ballast) command now refuses to overwrite the target ballast file if it already exists. This change is intended to prevent mistaken uses of the `ballast` command by operators. Scripts that integrate `cockroach debug ballast` can consider adding a `rm` command. -- Removed the `kv.atomic_replication_changes.enabled` [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings). All replication changes on a range now use joint-consensus. -- Currently, [changefeeds](https://www.cockroachlabs.com/docs/v21.1/stream-data-out-of-cockroachdb-using-changefeeds) connected to [Kafka versions < v1.0](https://docs.confluent.io/platform/current/installation/versions-interoperability.html) are not supported in CockroachDB v21.1. - -

Deprecations

- -- The CLI flags `--log-dir`, `--log-file-max-size`, `--log-file-verbosity`, and `--log-group-max-size` are deprecated. Logging configuration can now be specified via the `--log` parameter. See the [Logging](https://www.cockroachlabs.com/docs/v21.1/logging-overview) documentation for details. -- The client-side command `\show` for the [SQL shell](https://www.cockroachlabs.com/docs/v21.1/cockroach-sql#commands) is deprecated in favor of the new command `\p`. This prints the contents of the query buffer entered so far. -- Currently, Google Cloud Storage (GCS) connections default to the `cloudstorage.gs.default.key` [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings). This default behavior will no longer be supported in v21.2. If you are relying on this default behavior, we recommend adjusting your queries and scripts to now specify the `AUTH` parameter you want to use. Similarly, if you are using the `cloudstorage.gs.default.key` cluster setting to authorize your GCS connection, we recommend switching to use `AUTH=specified` or `AUTH=implicit`. `AUTH=specified` will be the default behavior in v21.2 and beyond. - -

Known limitations

- -For information about new and unresolved limitations in CockroachDB v21.1, with suggested workarounds where applicable, see [Known Limitations](https://www.cockroachlabs.com/docs/v21.1/known-limitations). - -

Education

- -Area | Topic | Description ------|-------|------------ -Cockroach University | **New Intro Courses** | [Introduction to Distributed SQL and CockroachDB](https://university.cockroachlabs.com/courses/course-v1:crl+intro-to-distributed-sql-and-cockroachdb+self-paced/about) teaches you the core concepts behind distributed SQL databases and describes how CockroachDB fits into this landscape. [Practical First Steps with CockroachDB](https://university.cockroachlabs.com/courses/course-v1:crl+practical-first-steps-with-crdb+self-paced/about) is a hands-on sequel that gives you the tools to get started with CockroachDB. -Cockroach University | **New Java Course** | [Fundamentals of CockroachDB for Java Developers](https://university.cockroachlabs.com/courses/course-v1:crl+fundamentals-of-crdb-for-java-devs+self-paced/about) guides you through building a full-stack vehicle-sharing app in Java using the popular Spring Data JPA framework with Spring Boot and a CockroachCloud Free cluster as the backend. -Cockroach University | **New Query Performance Course** | [CockroachDB Query Performance for Developers](https://university.cockroachlabs.com/courses/course-v1:crl+cockroachdb-query-performance-for-devs+self-paced/about) teaches you key CockroachDB features and skills to improve application performance and functionality, such as analyzing a query execution plan, using indexes to avoid expensive full table scans, improving sorting performance, and efficiently querying fields in JSON records. -Docs | **Quickstart** | Documented the simplest way to [get started with CockroachDB](https://www.cockroachlabs.com/docs/cockroachcloud/quickstart) for testing and app development by using CockroachCloud Free. -Docs | **Developer Guidance** | Published more comprehensive, task-oriented [guidance for developers](https://www.cockroachlabs.com/docs/v21.1/developer-guide-overview) building applications on CockroachDB, including connecting to a cluster, designing a database schema, reading and writing data, optimizing query performance, and debugging applications. -Docs | **Connection Pooling** | Added guidance on [planning, configuring, and using connection pools](https://www.cockroachlabs.com/docs/v21.1/connection-pooling) with CockroachDB, as well as examples for Java and Go applications. -Docs | **Sample Apps on
CockroachCloud Free** | Updated various Java, Python, Node.js, Ruby, and Go [sample app tutorials](https://www.cockroachlabs.com/docs/v21.1/example-apps) to offer CockroachCloud Free as the backend. -Docs | **Licensing FAQs** | Updated the [Licensing FAQ](https://www.cockroachlabs.com/docs/v21.1/licensing-faqs) to explain our licensing types, how features align to licenses, how to perform basic tasks around licenses (e.g., obtain, set, verify, monitor, renew), and other common questions. -Docs | **Product Limits** | Added [object sizing and scaling considerations](https://www.cockroachlabs.com/docs/v21.1/schema-design-overview#object-size-and-scaling-considerations), including specific hard limits imposed by CockroachDB and practical limits based on our performance testing and observations. -Docs | **System Catalogs** | Documented important [internal system catalogs](https://www.cockroachlabs.com/docs/v21.1/system-catalogs) that provide non-stored data to client applications. diff --git a/src/current/_includes/releases/v21.1/v21.1.1.md b/src/current/_includes/releases/v21.1/v21.1.1.md deleted file mode 100644 index 1612bc6a0d5..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.1.md +++ /dev/null @@ -1,163 +0,0 @@ -## v21.1.1 - -Release Date: May 24, 2021 - - - -

General changes

- -- Disabled read-triggered compactions to avoid instances where the [storage engine](https://www.cockroachlabs.com/docs/v21.1/architecture/storage-layer) would compact excessively. [#65345][#65345] - -

SQL language changes

- -- Fixed [Julian date](https://wikipedia.org/wiki/Julian_day) parsing logic for wrong formats. [#63540][#63540] -- The error payload returned to the client when a [`DATE`](https://www.cockroachlabs.com/docs/v21.1/date)/[`TIME`](https://www.cockroachlabs.com/docs/v21.1/time) conversion fails now contains more details about the difference between the values provided and the values that are expected. [#63540][#63540] {% comment %}doc{% endcomment %} -- Introduced `ALTER TABLE ... ALTER COLUMN SET [VISIBLE|NOT VISIBLE]`, which marks columns as visible/not visible. [#63881][#63881] {% comment %}doc{% endcomment %} -- When using [`ALTER TABLE ... LOCALITY REGIONAL BY ROW`](https://www.cockroachlabs.com/docs/v21.1/set-locality), we would previously verify uniqueness of the new table, which was an unnecessary operation. This verification has been removed, improving the performance of updating localities to or from `REGIONAL BY ROW`. [#63880][#63880] {% comment %}doc{% endcomment %} -- Improved cancellation behavior for [DistSQL](https://www.cockroachlabs.com/docs/v21.1/architecture/sql-layer) flows. [#65047][#65047] {% comment %}doc{% endcomment %} -- [`ST_EstimatedExtent`](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators#spatial-functions) now always returns `NULL`. This allows [GeoServer](http://geoserver.org/) to make progress in certain cases, and is a valid default return value for the function. [#65098][#65098] {% comment %}doc{% endcomment %} -- Implemented `ST_Envelope` for [`Box2D`](https://www.cockroachlabs.com/docs/v21.1/spatial-glossary#data-types). [#65098][#65098] {% comment %}doc{% endcomment %} -- Implemented a subset of variants for `ST_AsTWKB`, which encodes a geometry into a [`TWKB`](https://github.com/TWKB/Specification/blob/master/twkb.md) format. This allows the use of [GeoServer](https://docs.geoserver.org/latest/en/user/) with CRDB if the user selects "PreserveTopology" for their "Method used to simplify geometries" option on the "Store" page. [#65098][#65098] {% comment %}doc{% endcomment %} -- Implemented `ST_Simplify` with [`preserveCollapsed`](https://postgis.net/docs/ST_Simplify.html) support. This unblocks the use of GeoServer with the default settings. [#65098][#65098] {% comment %}doc{% endcomment %} -- [Lookup joins](https://www.cockroachlabs.com/docs/v21.1/joins) on indexes with [computed columns](https://www.cockroachlabs.com/docs/v21.1/computed-columns) which are also either constrained by [`CHECK`](https://www.cockroachlabs.com/docs/v21.1/check) constraints or use an [`ENUM`](https://www.cockroachlabs.com/docs/v21.1/enum) data type may now choose a more optimal plan. [#65361][#65361] {% comment %}doc{% endcomment %} -- Floating point infinity values are now formatted as `Infinity` (or `-Infinity` if negative). This is for compatibility with PostgresSQL. [#65334][#65334] {% comment %}doc{% endcomment %} -- [`INSERT INTO ... ON CONFLICT ... DO UPDATE SET`](https://www.cockroachlabs.com/docs/v21.1/insert) statements without predicates now acquire locks using the `FOR UPDATE` locking mode during their initial row scan, which improves performance for contended workloads. This behavior is configurable using the `enable_implicit_select_for_update` [session variable](https://www.cockroachlabs.com/docs/v21.1/set-vars) and the `sql.defaults.implicit_select_for_update.enabled` [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings). [#65363][#65363] {% comment %}doc{% endcomment %} -- [`ST_GeomFromGeoJSON(string)`](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators#spatial-functions) is now marked as the preferred overload, meaning it will resolve correctly in more contexts. [#65442][#65442] {% comment %}doc{% endcomment %} - -

Operational changes

- -- [Replica garbage collection](https://www.cockroachlabs.com/docs/v21.1/architecture/storage-layer#garbage-collection) now checks replicas against the range descriptor every 12 hours (down from 10 days) to see if they should be removed. Replicas that fail to notice they have been removed from a range will therefore linger for at most 12 hours rather than 10 days. [#64589][#64589] - -

Command-line changes

- -- The `--help` text for [`--log`](https://www.cockroachlabs.com/docs/v21.1/configure-logs) now references the fact that the flag accepts YAML syntax and also points to the `cockroach debug check-log-config` command. [#64948][#64948] -- The new parameter `--log-config-file` simplifies the process of loading the logging configuration from a YAML file. Instead of passing the content of the file via the `--log` flag (e.g., `--log=$(cat file.yaml)`), it is now possible to pass the path to the file using `--log-config-file=file.yaml`.
**Note:** Each occurrence of `--log` and `--log-config-file` on the command line overrides the configuration set from previous occurrences. [#64948][#64948] {% comment %}doc{% endcomment %} -- The prefixes displayed before connection URLs when [`cockroach demo`](https://www.cockroachlabs.com/docs/v21.1/cockroach-demo) starts up have been updated to better align with the output of [`cockroach start`](https://www.cockroachlabs.com/docs/v21.1/cockroach-start). [#63535][#63535] {% comment %}doc{% endcomment %} -- The flag `--empty` for `cockroach demo` has been renamed to `--no-example-database`. `--empty` is still recognized but is marked as deprecated. Additionally, the user can now set the environment variable `COCKROACH_NO_EXAMPLE_DATABASE` to obtain this behavior automatically in every new demo session. [#63535][#63535] {% comment %}doc{% endcomment %} -- CockroachDB no longer supports the `\demo add` and `\demo shutdown` commands for `cockroach demo` in `--global` configurations. [#63535][#63535] {% comment %}doc{% endcomment %} -- Added a note when starting up a `--global` demo cluster that the `--global` configuration is experimental. [#63535][#63535] {% comment %}doc{% endcomment %} -- The SQL shell (`cockroach demo`, [`cockroach sql`](https://www.cockroachlabs.com/docs/v21.1/cockroach-sql)) now attempts to better format values that are akin to time/date values, as well as floating-point numbers. [#63541][#63541] {% comment %}doc{% endcomment %} -- [`cockroach debug zip`](https://www.cockroachlabs.com/docs/v21.1/cockroach-debug-zip) now attempts to pull data from multiple nodes concurrently, up to 15 nodes at a time. This change is meant to accelerate the data collection when a cluster contains multiple nodes. This behavior can be changed with the new command-line flag `--concurrency`. [#64705][#64705] {% comment %}doc{% endcomment %} -- The format of the informational messages printed by `cockroach debug zip` , when concurrent execution is enabled. [#64705][#64705] {% comment %}doc{% endcomment %} -- The new command `cockroach debug list-files` show the list of files that can be retrieved via the `cockroach debug zip` command. It supports the `--nodes` and `--exclude-nodes` parameters in the same way as `cockroach debug zip`. [#64705][#64705] {% comment %}doc{% endcomment %} -- It is now possible to fine-tune which files get retrieved from the server nodes by the `cockroach debug zip` command, using the new flag `--include-files` and `--exclude-files`. These flags take glob patterns that are applied to the file names server-side. For example, to include only log files, use `--include-files='*.log'`. The command `cockroach debug list-files` also accepts these flags and can thus be used to experiment with the new flags before running the `cockroach debug zip` command. [#64705][#64705] {% comment %}doc{% endcomment %} -- The `cockroach debug zip` command now retrieves only the log files, goroutine dumps and heap profiles pertaining to the last 48 hours prior to the command invocation. This behavior is supported entirely client-side, which means that it is not necessary to upgrade the server nodes to put these newly-configurable limits in place. The other data items retrieved by `cockroach debug zip` are not affected by this time limit. This behavior can be customized by the two new flags `--files-from` and `--files-until`. Both are optional. See the command-line help text for details. The two new flags are also supported by `cockroach debug list-files`. It is advised to experiment with `list-files` prior to issuing a `debug zip` command that may retrieve a large amount of data. [#64705][#64705] {% comment %}doc{% endcomment %} - -

DB Console changes

- -- A [new metric](https://www.cockroachlabs.com/docs/v21.1/ui-overview) for the average number of runnable goroutines per CPU is now present in the runtime graphs. [#64750][#64750] {% comment %}doc{% endcomment %} -- The Console now uses a new library for line graphs that renders metrics more efficiently. Customers with large clusters can now load and interact with metrics much faster than before. [#64479][#64479] -- Placed a legend under charts on [metrics page](https://www.cockroachlabs.com/docs/v21.1/ui-overview-dashboard), if more than 10 series are being displayed [#64479][#64479] {% comment %}doc{% endcomment %} - -

Bug fixes

- -- Fixed a bug in the artificial latencies introduced by the `--global` flag to [`cockroach demo`](https://www.cockroachlabs.com/docs/v21.1/cockroach-demo). [#63535][#63535] -- Fixed a bug where multiple concurrent invocations of [`cockroach debug zip`](https://www.cockroachlabs.com/docs/v21.1/cockroach-debug-zip) could yield cluster instability. This bug had been present since CockroachDB v20.1. [#64083][#64083] -- When a [`STRING`](https://www.cockroachlabs.com/docs/v21.1/string) value is converted to [`TIME`](https://www.cockroachlabs.com/docs/v21.1/time)/[`DATE`](https://www.cockroachlabs.com/docs/v21.1/date)/[`TIMESTAMP`](https://www.cockroachlabs.com/docs/v21.1/timestamp), and the `STRING` value contains invalid entries, the error messages reported now more clearly report which fields are missing or undesired. [#63540][#63540] -- Fixed a bug where view expressions created using an [`ARRAY`](https://www.cockroachlabs.com/docs/v21.1/array) [`ENUM`](https://www.cockroachlabs.com/docs/v21.1/enum) without a name for the column could cause failures when dropping unrelated `ENUM` values. [#64272][#64272] -- Fixed a bug causing an internal error in rare circumstances when executing queries via the [vectorized engine](https://www.cockroachlabs.com/docs/v21.1/vectorized-execution) that operate on columns of [`BOOL`](https://www.cockroachlabs.com/docs/v21.1/bool), [`BYTES`](https://www.cockroachlabs.com/docs/v21.1/bytes), [`INT`](https://www.cockroachlabs.com/docs/v21.1/int), and [`FLOAT`](https://www.cockroachlabs.com/docs/v21.1/float) types that have a mix of [`NULL`](https://www.cockroachlabs.com/docs/v21.1/null-handling) and non-`NULL` values. [#62915][#62915] -- Fixed a bug causing CockroachDB to either return an error or crash when comparing an infinite `DATE` coming from a subquery against a `TIMESTAMP`. [#64074][#64074] -- CockroachDB now should crash less often due to out-of-memory conditions caused by the [subqueries](https://www.cockroachlabs.com/docs/v21.1/subqueries) returning multiple rows of large size. [#64727][#64727] -- Previously, the [session trace](https://www.cockroachlabs.com/docs/v21.1/show-trace) could contain entries that corresponded to the previous trace (i.e., `SET TRACING=ON` didn't properly reset the trace). Now this is fixed. [#64945][#64945] -- Previously, CockroachDB could incorrectly cast [integers](https://www.cockroachlabs.com/docs/v21.1/int) of larger widths to integers of smaller widths (e.g., `INT8` to `INT2`) when the former was out of range for the latter. Now this is fixed. [#65035][#65035] -- Fixed a race condition where read-write requests during replica removal (e.g., during range merges or rebalancing) could be evaluated on the removed replica. [#64598][#64598] -- [`BACKUP`](https://www.cockroachlabs.com/docs/v21.1/backup) no longer resolves intents one-by-one. This eliminates the need to run a high-priority query to cleanup intents to unblock `BACKUP` in the case of intent buildup. [#64881][#64881] -- Fixed an internal error that could occur during planning when a query used the output of an [`UPDATE`](https://www.cockroachlabs.com/docs/v21.1/update)'s `RETURNING` clause, and one or more of the columns in the `RETURNING` clause were from a table specified in the `FROM` clause of the `UPDATE` (i.e., not from the table being updated). [#62960][#62960] -- Fixed an index-out-of-range error that could occur when `crdb_internal_mvcc_timestamp` was selected as part of a [`view`](https://www.cockroachlabs.com/docs/v21.1/views). It is now possible to select `crdb_internal_mvcc_timestamp` as part of a view as long as it is aliased with a different name. [#63632][#63632] -- Fixed a bug in the application of environment variables to populate defaults for certain [command-line flags](https://www.cockroachlabs.com/docs/v21.1/cockroach-commands), for example `COCKROACH_URL` for `--url`, has been fixed. [#63539][#63539] -- Fixed a stack overflow that can happen in some corner cases involving [partial indexes](https://www.cockroachlabs.com/docs/v21.1/partial-indexes) with predicates containing `(x IS NOT NULL)`. [#64738][#64738] -- Providing a constant value as an [`ORDER BY`](https://www.cockroachlabs.com/docs/v21.1/order-by) value in an ordered-set [aggregate](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators#aggregate-functions), such as `percentile_dist` or `percentile_cont`, no longer returns an error. This bug has been present since order-set aggregates were added in version 20.2. [#64902][#64902] -- Queries that reference tables with [`GEOMETRY` or `GEOGRAPHY`](https://www.cockroachlabs.com/docs/v21.1/spatial-glossary#data-types) [GIN indexes](https://www.cockroachlabs.com/docs/v21.1/inverted-indexes) and that call [geospatial functions](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators#spatial-functions) with constant [`NULL`](https://www.cockroachlabs.com/docs/v21.1/null-handling) values cast to a type, like `NULL::GEOMETRY` or `NULL::FLOAT8`, no longer error. This bug was present since 20.2. [#63003][#63003] -- Fixed a bug causing CockroachDB to incorrectly calculate the latency from the remote nodes when the latency info was shown on the [`EXPLAIN ANALYZE (DISTSQL)`](https://www.cockroachlabs.com/docs/v21.1/explain-analyze) diagrams. [#63089][#63089] -- Fixed a bug causing the [`ZONECONFIG` privilege](https://www.cockroachlabs.com/docs/v21.1/authorization#privileges) on tables and databases to be incorrectly interpreted as `USAGE`, which could corrupt a table and/or database because `USAGE` is an invalid privilege for tables and databases. [#65160][#65160] -- Fixed a bug which could cause a panic when running an `EXECUTE` of a previously-prepared statement with a `REGCLASS` or `REGTYPE` parameter or a [user-defined type](https://www.cockroachlabs.com/docs/v21.1/enum) argument after running [`BEGIN AS OF SYSTEM TIME`](https://www.cockroachlabs.com/docs/v21.1/as-of-system-time) with an invalid timestamp. [#65150][#65150] -- Fixed a bug which could cause a panic when issuing a query referencing a [user-defined type](https://www.cockroachlabs.com/docs/v21.1/enum) as a placeholder. [#65150][#65150] -- Fixed a bug introduced in 20.2 that caused rows to be incorrectly de-duplicated from a scan with a non-unique index. [#65284][#65284] -- Fixed a bug where interval math on a [`TIMESTAMPTZ`](https://www.cockroachlabs.com/docs/v21.1/timestamp) value on a DST boundary would incorrectly add or subtract an extra hour. [#65095][#65095] -- Fixed a bug where `date_trunc` on a [`TIME`](https://www.cockroachlabs.com/docs/v21.1/time) value on a DST boundary could switch timezones and produce the incorrect result. [#65095][#65095] -- Improved memory utilization under some write-heavy workloads, added better logging to [storage engine](https://www.cockroachlabs.com/docs/v21.1/architecture/storage-layer) to surface compaction type, and persisted previously-missing Pebble options in `OPTIONS` file. [#65308][#65308] -- Fixed a bug causing `revision_history` cluster [backups](https://www.cockroachlabs.com/docs/v21.1/backup) to not include dropped databases. This means that, previously, dropped databases could not be restored from backups that were taken after the database was dropped. [#65314][#65314] -- Fixed a bug causing [`SHOW CREATE TABLE`](https://www.cockroachlabs.com/docs/v21.1/show-create) output to not display the [zone configurations](https://www.cockroachlabs.com/docs/v21.1/configure-zone) of a table or index if there were no partitions, even if there were zone configurations on the index or table. [#65176][#65176] -- Previously, concatenating a non-[`STRING`](https://www.cockroachlabs.com/docs/v21.1/string) value with a `STRING` value would not use the normal `STRING` representation of the non-`STRING` value. Now it does, so `true || 'string'` returns `truestring` instead of `tstring`. [#65331][#65331] -- Large [`SELECT FOR UPDATE`](https://www.cockroachlabs.com/docs/v21.1/selection-queries) scans will no longer prevent the memory associated with their entire result set from being reclaimed by the Go garbage collector for the lifetime of the locks that they acquire. [#65359][#65359] -- Fixed a rare race that could lead to a 3-second stall before a [Raft leader](https://www.cockroachlabs.com/docs/v21.1/architecture/replication-layer) was elected on a Range immediately after it was split off from its left-hand neighbor. [#65356][#65356] -- Fixed a bug where `SHOW CREATE TABLE` would show the zone configurations of a table of the same name from a different schema. [#65368][#65368] -- [`BACKUP`](https://www.cockroachlabs.com/docs/v21.1/backup), [`RESTORE`](https://www.cockroachlabs.com/docs/v21.1/restore), and [`IMPORT`](https://www.cockroachlabs.com/docs/v21.1/import) are now more resilient to node failures and will retry automatically. [#65391][#65391] -- Previously, replica rebalancing could sometimes rebalance to stores on dead nodes. This bug is now fixed. [#65428][#65428] - -

Performance improvements

- -- The optimizer now always prefers to plan a [locality-optimized](https://www.cockroachlabs.com/docs/v21.1/multiregion-overview) scan over a regular scan when possible. This may enable the execution engine to avoid communicating with remote nodes, thus reducing query latency. [#65088][#65088] -- The optimizer will now try to plan anti lookup joins using "locality-optimized search". This optimization applies for anti lookup joins into [`REGIONAL BY ROW`](https://www.cockroachlabs.com/docs/v21.1/multiregion-overview#regional-by-row-tables) tables (i.e., the right side of the join is a `REGIONAL BY ROW` table), and if enabled, it means that the execution engine will first search locally for matching rows before searching remote nodes. If a matching row is found in a local node, remote nodes will not be searched. This optimization may improve the performance of [foreign key](https://www.cockroachlabs.com/docs/v21.1/foreign-key) checks when rows are inserted or updated in a table that references a foreign key in a `REGIONAL BY ROW` table. [#63118][#63118] -- Certain queries containing ` IN ()` conditions now run faster. [#63866][#63866] -- Improved intent cleanup performance for aborted transactions. [#64588][#64588] -- Adjusted the estimated cost of locality-optimized anti joins in the optimizer so that they are always chosen over non-locality-optimized anti joins when possible. This makes it more likely that queries involving anti joins (such as inserts with foreign key checks) can avoid visiting remote regions. This results in lower latency. [#65131][#65131] -- The optimizer can now avoid full table scans for queries with a `LIMIT` and [`ORDER BY`](https://www.cockroachlabs.com/docs/v21.1/order-by) clause, where the `ORDER BY` columns form a prefix on an index in a `REGIONAL BY ROW` table (excluding the hidden `crdb_region` column). Instead of a full table scan, at most `LIMIT` rows are scanned per region. [#65287][#65287] - -
- -

Contributors

- -This release includes 100 merged PRs by 33 authors. -We would like to thank the following contributors from the CockroachDB community: - -- Kumar Akshay -- Mohammad Aziz (first-time contributor) -- kurokochin (first-time contributor) - -
- -[#62915]: https://github.com/cockroachdb/cockroach/pull/62915 -[#62960]: https://github.com/cockroachdb/cockroach/pull/62960 -[#63003]: https://github.com/cockroachdb/cockroach/pull/63003 -[#63089]: https://github.com/cockroachdb/cockroach/pull/63089 -[#63118]: https://github.com/cockroachdb/cockroach/pull/63118 -[#63535]: https://github.com/cockroachdb/cockroach/pull/63535 -[#63539]: https://github.com/cockroachdb/cockroach/pull/63539 -[#63540]: https://github.com/cockroachdb/cockroach/pull/63540 -[#63541]: https://github.com/cockroachdb/cockroach/pull/63541 -[#63632]: https://github.com/cockroachdb/cockroach/pull/63632 -[#63866]: https://github.com/cockroachdb/cockroach/pull/63866 -[#63880]: https://github.com/cockroachdb/cockroach/pull/63880 -[#63881]: https://github.com/cockroachdb/cockroach/pull/63881 -[#64074]: https://github.com/cockroachdb/cockroach/pull/64074 -[#64083]: https://github.com/cockroachdb/cockroach/pull/64083 -[#64272]: https://github.com/cockroachdb/cockroach/pull/64272 -[#64479]: https://github.com/cockroachdb/cockroach/pull/64479 -[#64588]: https://github.com/cockroachdb/cockroach/pull/64588 -[#64589]: https://github.com/cockroachdb/cockroach/pull/64589 -[#64598]: https://github.com/cockroachdb/cockroach/pull/64598 -[#64705]: https://github.com/cockroachdb/cockroach/pull/64705 -[#64727]: https://github.com/cockroachdb/cockroach/pull/64727 -[#64738]: https://github.com/cockroachdb/cockroach/pull/64738 -[#64750]: https://github.com/cockroachdb/cockroach/pull/64750 -[#64881]: https://github.com/cockroachdb/cockroach/pull/64881 -[#64902]: https://github.com/cockroachdb/cockroach/pull/64902 -[#64945]: https://github.com/cockroachdb/cockroach/pull/64945 -[#64948]: https://github.com/cockroachdb/cockroach/pull/64948 -[#65035]: https://github.com/cockroachdb/cockroach/pull/65035 -[#65047]: https://github.com/cockroachdb/cockroach/pull/65047 -[#65088]: https://github.com/cockroachdb/cockroach/pull/65088 -[#65095]: https://github.com/cockroachdb/cockroach/pull/65095 -[#65098]: https://github.com/cockroachdb/cockroach/pull/65098 -[#65131]: https://github.com/cockroachdb/cockroach/pull/65131 -[#65150]: https://github.com/cockroachdb/cockroach/pull/65150 -[#65160]: https://github.com/cockroachdb/cockroach/pull/65160 -[#65176]: https://github.com/cockroachdb/cockroach/pull/65176 -[#65284]: https://github.com/cockroachdb/cockroach/pull/65284 -[#65287]: https://github.com/cockroachdb/cockroach/pull/65287 -[#65308]: https://github.com/cockroachdb/cockroach/pull/65308 -[#65314]: https://github.com/cockroachdb/cockroach/pull/65314 -[#65331]: https://github.com/cockroachdb/cockroach/pull/65331 -[#65334]: https://github.com/cockroachdb/cockroach/pull/65334 -[#65345]: https://github.com/cockroachdb/cockroach/pull/65345 -[#65356]: https://github.com/cockroachdb/cockroach/pull/65356 -[#65357]: https://github.com/cockroachdb/cockroach/pull/65357 -[#65358]: https://github.com/cockroachdb/cockroach/pull/65358 -[#65359]: https://github.com/cockroachdb/cockroach/pull/65359 -[#65361]: https://github.com/cockroachdb/cockroach/pull/65361 -[#65363]: https://github.com/cockroachdb/cockroach/pull/65363 -[#65368]: https://github.com/cockroachdb/cockroach/pull/65368 -[#65391]: https://github.com/cockroachdb/cockroach/pull/65391 -[#65428]: https://github.com/cockroachdb/cockroach/pull/65428 -[#65442]: https://github.com/cockroachdb/cockroach/pull/65442 diff --git a/src/current/_includes/releases/v21.1/v21.1.10.md b/src/current/_includes/releases/v21.1/v21.1.10.md deleted file mode 100644 index 9e0a4bf0459..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.10.md +++ /dev/null @@ -1,15 +0,0 @@ -## v21.1.10 - -Release Date: October 7, 2021 - - - -

Bug fixes

- -- Fixed a bug that caused the [optimizer](https://www.cockroachlabs.com/docs/v21.1/cost-based-optimizer) to erroneously discard [`WHERE` filters](https://www.cockroachlabs.com/docs/v21.1/selection-queries) when executing prepared statements, causing incorrect results to be returned. This bug was present since [v21.1.9](v21.1.html#v21-1-9). [#71116][#71116]. - -

Contributors

- -This release includes 1 merged PRs by 1 author. - -[#71116]: https://github.com/cockroachdb/cockroach/pull/71116 diff --git a/src/current/_includes/releases/v21.1/v21.1.11.md b/src/current/_includes/releases/v21.1/v21.1.11.md deleted file mode 100644 index fa24ed4afbf..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.11.md +++ /dev/null @@ -1,90 +0,0 @@ -## v21.1.11 - -Release Date: October 18, 2021 - - - -

Enterprise edition changes

- -- Fixed a bug where [`CHANGEFEED`](https://www.cockroachlabs.com/docs/v21.1/create-changefeed)s would fail to correctly handle a [primary key](https://www.cockroachlabs.com/docs/v21.1/primary-key) change. [#69926][#69926] -- [`CHANGEFEED`](https://www.cockroachlabs.com/docs/v21.1/create-changefeed)s no longer fail when started on [`REGIONAL BY ROW`](https://www.cockroachlabs.com/docs/v21.1/set-locality#set-the-table-locality-to-regional-by-row) tables. Note that in `REGIONAL BY ROW` tables, the [`crdb_region`](https://www.cockroachlabs.com/docs/v21.1/set-locality#crdb_region) column becomes part of the primary [index](https://www.cockroachlabs.com/docs/v21.1/indexes). Thus, changing an existing table to `REGIONAL BY ROW` will trigger a [changefeed](https://www.cockroachlabs.com/docs/v21.1/create-changefeed) backfill with new messages emitted using the new composite [primary key](https://www.cockroachlabs.com/docs/v21.1/primary-key). [#70022][#70022] -- Fixed a bug that could have led to duplicate instances of a single [changefeed](https://www.cockroachlabs.com/docs/v21.1/create-changefeed) [job](https://www.cockroachlabs.com/docs/v21.1/show-jobs) running for prolonged periods of time. [#70926][#70926] - -

SQL language changes

- -- Added `crdb_internal.(node|cluster)_distsql_flows` virtual tables that expose the information about the flows of the [DistSQL](https://www.cockroachlabs.com/docs/v21.1/architecture/sql-layer#distsql) execution scheduled on remote nodes. These tables do not include information about the non-distributed queries nor about local flows (from the perspective of the gateway node of the query). [#66332][#66332] -- Added new metrics to track a [schema](https://www.cockroachlabs.com/docs/v21.1/online-schema-changes) [job](https://www.cockroachlabs.com/docs/v21.1/show-jobs) failure (`sql.schema_changer.errors.all`, `sql.schema_changer.errors.constraint_violation`, `sql.schema_changer.errors.uncategorized`), with errors inside the `crdb_internal.feature_usage` table. [#70621][#70621] -- Fixed a bug where [`LINESTRINGZ`](https://www.cockroachlabs.com/docs/v21.1/linestring), `LINESTRINGZM`, and `LINESTRINGM` could not be used as column types. [#70749][#70749] - -

Operational changes

- -- Added the [cluster settings](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) `sql.defaults.transaction_rows_written_log`, `sql.defaults.transaction_rows_written_err`, `sql.defaults.transaction_rows_read_log`, and `sql.defaults.transaction_rows_read_err` (as well as the corresponding [session variables](https://www.cockroachlabs.com/docs/v21.1/set-vars#supported-variables)). These settings determine the "size" of the [transactions](https://www.cockroachlabs.com/docs/v21.1/transactions) in written and read rows that are logged to the `SQL_PERF` [logging channel](https://www.cockroachlabs.com/docs/v21.1/logging-overview). Note that the internal queries used by CockroachDB cannot error out, but can be logged instead to the `SQL_INTERNAL_PERF` [logging channel](https://www.cockroachlabs.com/docs/v21.1/logging-overview). The "written" limits apply to [`INSERT`](https://www.cockroachlabs.com/docs/v21.1/insert), `INSERT INTO SELECT FROM`, [`INSERT ON CONFLICT`](https://www.cockroachlabs.com/docs/v21.1/insert), [`UPSERT`](https://www.cockroachlabs.com/docs/v21.1/upsert), [`UPDATE`](https://www.cockroachlabs.com/docs/v21.1/update), and [`DELETE`](https://www.cockroachlabs.com/docs/v21.1/delete) whereas the "read" limits apply to [`SELECT` statements](https://www.cockroachlabs.com/docs/v21.1/selection-queries), in addition to all of the others listed. These limits will not apply to [`CREATE TABLE AS SELECT`](https://www.cockroachlabs.com/docs/v21.1/create-table), [`IMPORT`](https://www.cockroachlabs.com/docs/v21.1/import), [`TRUNCATE`](https://www.cockroachlabs.com/docs/v21.1/truncate), [`DROP TABLE`](https://www.cockroachlabs.com/docs/v21.1/drop-table), [`ALTER TABLE`](https://www.cockroachlabs.com/docs/v21.1/alter-table), [`BACKUP`](https://www.cockroachlabs.com/docs/v21.1/backup), [`RESTORE`](https://www.cockroachlabs.com/docs/v21.1/restore), or [`CREATE STATISTICS`](https://www.cockroachlabs.com/docs/v21.1/create-statistics) statements. Note that enabling the `transaction_rows_read_err` setting comes at the cost of disabling the usage of the auto commit optimization for the mutation statements in [implicit transactions](https://www.cockroachlabs.com/docs/v21.1/transactions#individual-statements). [#70175][#70175] -- Adjusted the meaning of the recently introduced [session variables](https://www.cockroachlabs.com/docs/v21.1/set-vars#supported-variables) `transaction_rows_written_err` and `transaction_rows_read_err` (as well as the corresponding `_log` variables) to indicate the largest number of rows that is still allowed. Prior to this change, reaching the limit would result in an error; now an error results only if the limit is exceeded. [#70175][#70175] -- Added the [session variable](https://www.cockroachlabs.com/docs/v21.1/set-vars#supported-variables) `large_full_scan_rows`, as well as the corresponding [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) `sql.defaults.large_full_scan_rows`. This setting determines which tables are considered "large" for the purposes of enabling the `disallow_full_table_scans` feature to reject full table/index scans of such "large" tables. The default value for the new setting is `0`, meaning that the previous behavior of rejecting all full table/index scans is kept. Internally-issued queries aren't affected, and the new setting has no impact when the `disallow_full_table_scans` setting is not enabled. [#70294][#70294] -- CockroachDB now records a [log event](https://www.cockroachlabs.com/docs/v21.1/eventlog) and increments a counter when removing an expired session. [#68538][#68538] - -

Command-line changes

- -- Version details have been added to all JSON formatted [log entries](https://www.cockroachlabs.com/docs/v21.1/logging-overview). [#70451][#70451] - -

DB Console changes

- -- Renamed references to the [UI console](https://www.cockroachlabs.com/docs/v21.1/ui-overview) from "Admin UI" to "DB Console". [#70870][#70870] - -

Bug fixes

- -- Fixed a bug where cluster revision history [backups](https://www.cockroachlabs.com/docs/v21.1/backup) may have included dropped descriptors in the "current" snapshot of descriptors on the cluster. [#69650][#69650] -- Fixed a regression in [statistics](https://www.cockroachlabs.com/docs/v21.1/cost-based-optimizer#table-statistics) estimation in the optimizer for very large tables. The bug, which has been present since [v20.2.14]({% link releases/v20.2.md %}#v20-2-14) and [v21.1.7](v21.1.html#v21-1-7), could cause the optimizer to severely underestimate the number of rows returned by an expression. [#69953][#69953] -- Fixed a bug that can cause prolonged unavailability due to [lease transfer](https://www.cockroachlabs.com/docs/v21.1/architecture/replication-layer#epoch-based-leases-table-data) of a replica that may be in need of a [Raft](https://www.cockroachlabs.com/docs/v21.1/architecture/replication-layer#raft) snapshot. [#69964][#69964] -- Fixed a bug where, after a temporary node outage, other nodes in the cluster could fail to connect to the restarted node due to their circuit breakers not resetting. This would manifest in the [logs](https://www.cockroachlabs.com/docs/v21.1/logging-overview) via messages of the form `unable to dial nXX: breaker open`, where `XX` is the ID of the restarted node. Note that such errors are expected for nodes that are truly unreachable, and may still occur around the time of the restart, but for no longer than a few seconds. [#70311][#70311] -- [`RESTORE`](https://www.cockroachlabs.com/docs/v21.1/restore) will now correctly ignore dropped databases that may have been included in cluster [backups](https://www.cockroachlabs.com/docs/v21.1/backup) with revision history. [#69791][#69791] -- Fixed a bug where if tracing was enabled (using the `sql.trace.txn.enable_threshold` [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings)), the [statement diagnostics](https://www.cockroachlabs.com/docs/v21.1/explain-analyze#explain-analyze-debug) collection (`EXPLAIN ANALYZE (DEBUG)`) would not work. [#70035][#70035] -- Fixed a bug in full cluster [restores](https://www.cockroachlabs.com/docs/v21.1/restore) where dropped descriptor revisions would cause the restore [job](https://www.cockroachlabs.com/docs/v21.1/show-jobs) to fail. [#69654][#69654] -- Fixed a bug where [schema changes](https://www.cockroachlabs.com/docs/v21.1/online-schema-changes) that included both a column addition and [primary key](https://www.cockroachlabs.com/docs/v21.1/primary-key) change in the same [transaction](https://www.cockroachlabs.com/docs/v21.1/transactions) resulted in a failed [changefeed](https://www.cockroachlabs.com/docs/v21.1/create-changefeed). [#70022][#70022] -- Fixed a bug which prevented proper copying of [partitions](https://www.cockroachlabs.com/docs/v21.1/partitioning) and [zone configurations](https://www.cockroachlabs.com/docs/v21.1/configure-replication-zones) when de-interleaving a table with [ALTER PRIMARY KEY](https://www.cockroachlabs.com/docs/v21.1/alter-primary-key) when the columns did not appear in exactly the same order in the parent and child tables. [#70695][#70695] -- Fixed a bug where the exit status of the [`cockroach` command](https://www.cockroachlabs.com/docs/v21.1/cockroach-commands) did not follow the previously-documented table of exit status codes when an error occurred during command startup. Only errors occurring after startup were reported using the correct code. This bug had existed since reference exit status codes were introduced. [#70675][#70675] -- DNS unavailability during range 1 leaseholder loss will no longer cause significant latency increases for queries and other operations. [#70134][#70134] -- Fixed an issue in the [Pebble](https://www.cockroachlabs.com/docs/v21.1/architecture/storage-layer#pebble) storage engine where a key could be dropped from an LSM snapshot if the key was deleted by a range tombstone after the snapshot was acquired. [#70967][#70967] -- Fixed an internal error with [joins](https://www.cockroachlabs.com/docs/v21.1/joins) that are both `LATERAL` and `NATURAL`/`USING`. [#70800][#70800] -- Fixed `Z` and `M` coordinate columns causing a panic for the [`geometry_columns`](https://www.cockroachlabs.com/docs/v21.1/spatial-glossary#geometry_columns) and [`geography_columns`](https://www.cockroachlabs.com/docs/v21.1/spatial-glossary#geography_columns) tables. [#70813][#70813] -- Fixed a bug that could cause a CockroachDB node to deadlock upon startup in extremely rare cases. If encountered, a stack trace generated by `SIGQUIT` would have shown the function `makeStartLine()` near the top. This bug had existed since [v21.1]({% link releases/v21.1.md %}#v21-1-0). [#71408][#71408] - -

Performance improvements

- -- The conversion performance of [WKT](https://www.cockroachlabs.com/docs/v21.1/well-known-text) to a spatial type is slightly improved. [#70181][#70181] - -

Miscellaneous

- -- Added a new `as_json` option which renders [backup](https://www.cockroachlabs.com/docs/v21.1/backup) manifests as JSON values. [#70298][#70298] - -

Contributors

- -This release includes 41 merged PRs by 24 authors. - -[#66332]: https://github.com/cockroachdb/cockroach/pull/66332 -[#68538]: https://github.com/cockroachdb/cockroach/pull/68538 -[#69650]: https://github.com/cockroachdb/cockroach/pull/69650 -[#69654]: https://github.com/cockroachdb/cockroach/pull/69654 -[#69791]: https://github.com/cockroachdb/cockroach/pull/69791 -[#69926]: https://github.com/cockroachdb/cockroach/pull/69926 -[#69953]: https://github.com/cockroachdb/cockroach/pull/69953 -[#69964]: https://github.com/cockroachdb/cockroach/pull/69964 -[#70022]: https://github.com/cockroachdb/cockroach/pull/70022 -[#70035]: https://github.com/cockroachdb/cockroach/pull/70035 -[#70134]: https://github.com/cockroachdb/cockroach/pull/70134 -[#70175]: https://github.com/cockroachdb/cockroach/pull/70175 -[#70181]: https://github.com/cockroachdb/cockroach/pull/70181 -[#70294]: https://github.com/cockroachdb/cockroach/pull/70294 -[#70298]: https://github.com/cockroachdb/cockroach/pull/70298 -[#70311]: https://github.com/cockroachdb/cockroach/pull/70311 -[#70451]: https://github.com/cockroachdb/cockroach/pull/70451 -[#70621]: https://github.com/cockroachdb/cockroach/pull/70621 -[#70675]: https://github.com/cockroachdb/cockroach/pull/70675 -[#70695]: https://github.com/cockroachdb/cockroach/pull/70695 -[#70749]: https://github.com/cockroachdb/cockroach/pull/70749 -[#70800]: https://github.com/cockroachdb/cockroach/pull/70800 -[#70813]: https://github.com/cockroachdb/cockroach/pull/70813 -[#70870]: https://github.com/cockroachdb/cockroach/pull/70870 -[#70926]: https://github.com/cockroachdb/cockroach/pull/70926 -[#70967]: https://github.com/cockroachdb/cockroach/pull/70967 -[#71408]: https://github.com/cockroachdb/cockroach/pull/71408 diff --git a/src/current/_includes/releases/v21.1/v21.1.12.md b/src/current/_includes/releases/v21.1/v21.1.12.md deleted file mode 100644 index 164e9900b00..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.12.md +++ /dev/null @@ -1,86 +0,0 @@ -## v21.1.12 - -Release Date: November 15, 2021 - - - -

Security updates

- -- Added the `--external-io-enable-non-admin-implicit-access` flag to [cluster-starting `cockroach` commands](https://www.cockroachlabs.com/docs/v21.1/cockroach-start). This flag removes the [admin-only](https://www.cockroachlabs.com/docs/v21.1/authorization#admin-role) restriction on interacting with arbitrary network endpoints and allows implicit authorization for operations such as [`BACKUP`](https://www.cockroachlabs.com/docs/v21.1/backup), [`IMPORT`](https://www.cockroachlabs.com/docs/v21.1/import), or [`EXPORT`](https://www.cockroachlabs.com/docs/v21.1/export). [#71794][#71794] - -

SQL language changes

- -- The [`pg_index` table](https://www.cockroachlabs.com/docs/v21.1/pg-catalog) now populates the `indpred` column for [partial indexes](https://www.cockroachlabs.com/docs/v21.1/partial-indexes). This column was previously `NULL` for partial indexes. [#70897][#70897] -- Added [`crdb_internal` tables](https://www.cockroachlabs.com/docs/v21.1/crdb-internal) `cross_db_references` and `interleaved` for detecting cross-database references and interleaved objects. [#72298][#72298] -- Statements with multiple [`INSERT ON CONFLICT`](https://www.cockroachlabs.com/docs/v21.1/insert), [`UPSERT`](https://www.cockroachlabs.com/docs/v21.1/upsert), [`UPDATE`](https://www.cockroachlabs.com/docs/v21.1/update), or [`DELETE`](https://www.cockroachlabs.com/docs/v21.1/delete) [subqueries](https://www.cockroachlabs.com/docs/v21.1/subqueries) that modify the same table are now disallowed, as these statements can cause data corruption if they modify the same row multiple times. At the risk of data corruption, you can allow these statements by setting the `sql.multiple_modifications_of_table.enabled` [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) to `true`. To check for corruption, use the `EXPERIMENTAL SCRUB` command. For example: `EXPERIMENTAL SCRUB TABLE t WITH OPTIONS INDEX ALL;`. [#71621][#71621] - -

Operational changes

- -- `IMPORT` now allows non-admin access to some previously-restricted network endpoints on clusters started with the `--external-io-enable-non-admin-implicit-access` flag. [#72444][#72444] -- A new implementation of `BACKUP` file handling is now available with the [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) `bulkio.backup.experimental_21_2_mode.enabled` set to `true`. [#71830][#71830] - -

DB Console changes

- -- Non-admin users of the DB Console now have the ability to view the [Cluster Overview page](https://www.cockroachlabs.com/docs/v21.1/ui-cluster-overview-page). Users without the [admin role](https://www.cockroachlabs.com/docs/v21.1/authorization#admin-role) can see data about their nodes, but information such as command line arguments, environment variables, and node IP addresses and DNS names is hidden. [#71719][#71719] -- Replicas waiting for garbage collection were preventing the [Range Report page](https://www.cockroachlabs.com/docs/v21.1/ui-debug-pages) from loading due to a JavaScript error. The page now loads and displays an empty "Replica Type" while in this state. [#71745][#71745] - -

Bug fixes

- -- Fixed a bug causing CockroachDB to encounter an internal error when executing a [zigzag join](https://www.cockroachlabs.com/docs/v21.1/experimental-features) in some cases. [#71255][#71255] -- Fixed a bug causing CockroachDB to incorrectly read the data of a [unique secondary index](https://www.cockroachlabs.com/docs/v21.1/unique) that used to be a primary index that was created via `ALTER PRIMARY KEY` in 21.1.x or prior versions. [#71587][#71587] -- CockroachDB now avoids dialing nodes in performance-critical code paths, which could cause substantial latency when encountering unresponsive nodes (e.g., when a VM or server is shut down). [#70488][#70488] -- Fixed a bug causing the [TPC-C workload](https://www.cockroachlabs.com/docs/v21.1/cockroach-workload) to improperly assign workers to the local partitions in a [multi-region setup](https://www.cockroachlabs.com/docs/v21.1/multiregion-overview). [#71753][#71753] -- Fixed a bug that caused internal errors when collecting statistics on tables with virtual [computed columns](https://www.cockroachlabs.com/docs/v21.1/computed-columns). [#71284][#71284] -- Fixed a bug causing CockroachDB to crash when network connectivity was impaired in some cases. The stack trace (in [`cockroach-stderr.log`](https://www.cockroachlabs.com/docs/v21.1/logging)) would contain `server.(*statusServer).NodesUI`. [#71719][#71719] -- Fixed a panic that could occur with invalid GeoJSON input using [`ST_GeomFromGeoJSON/ST_GeogFromGeoJSON`](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators). [#71308][#71308] -- Fixed a bug causing cluster [backups](https://www.cockroachlabs.com/docs/v21.1/backup) to back up opt-out system tables unexpectedly. [#71922][#71922] -- Fixed a bug that caused [`ALTER COLUMN TYPE`](https://www.cockroachlabs.com/docs/v21.1/alter-column) statements to fail unexpectedly. [#71166][#71166] -- Connection timeout for `grpc` connections is now set to `20s` to match the pre-20.2 default value. [#71516][#71516] -- Fixed a bug that prevented the rollback of [`ALTER PRIMARY KEY`](https://www.cockroachlabs.com/docs/v21.1/alter-primary-key) when the old primary key was interleaved. [#71852][#71852] -- Fixed a bug that caused incorrect results for some queries that utilized a [zigzag join](https://www.cockroachlabs.com/docs/v21.1/experimental-features). The bug could only reproduce on tables with at least two multi-column indexes with nullable columns. The bug was present since version 19.2.0. [#71847][#71847] -- Fixed a bug causing `IMPORT` statements to incorrectly reset progress upon resumption. [#72086][#72086] -- Fixed a bug causing schema changes running during node shutdown to fail permanently. [#71558][#71558] -- Fixed an incorrect "no data source matches prefix" error for queries that use a set-returning function on the right-hand side of a [`JOIN`](https://www.cockroachlabs.com/docs/v21.1/joins) (unless `LATERAL` is explicitly specified). [#71443][#71443] -- Long running [`ANALYZE`](https://www.cockroachlabs.com/docs/v21.1/explain-analyze) statements no longer result in GC TTL errors. [#69599][#69599] -- `IMPORT INTO` no longer crashes when encountering unresolved write intents. [#71982][#71982] -- Fixed a bug causing tracing to external tracers to inadvertently stop after the Enqueue Range or the Allocator [debug pages](https://www.cockroachlabs.com/docs/v21.1/ui-debug-pages) were used. [#72463][#72463] -- Fixed a bug which prevented the Data Distribution debug page from working on clusters which were upgraded from 19.2 or earlier. [#72507][#72507] - -

Performance improvements

- -- Slightly reduced `ANALYZE` and [`CREATE STATISTICS`](https://www.cockroachlabs.com/docs/v21.1/create-statistics) statements memory usage. [#71771][#71771] -- To reduce transient memory usage, CockroachDB now performs intent cleanup during garbage collection in batches as they are found instead of performing a single cleanup at the end of the garbage collection cycle. [#67590][#67590] - -

Contributors

- -This release includes 51 merged PRs by 23 authors. - -[#67590]: https://github.com/cockroachdb/cockroach/pull/67590 -[#69599]: https://github.com/cockroachdb/cockroach/pull/69599 -[#70488]: https://github.com/cockroachdb/cockroach/pull/70488 -[#70897]: https://github.com/cockroachdb/cockroach/pull/70897 -[#71166]: https://github.com/cockroachdb/cockroach/pull/71166 -[#71255]: https://github.com/cockroachdb/cockroach/pull/71255 -[#71284]: https://github.com/cockroachdb/cockroach/pull/71284 -[#71308]: https://github.com/cockroachdb/cockroach/pull/71308 -[#71443]: https://github.com/cockroachdb/cockroach/pull/71443 -[#71516]: https://github.com/cockroachdb/cockroach/pull/71516 -[#71558]: https://github.com/cockroachdb/cockroach/pull/71558 -[#71587]: https://github.com/cockroachdb/cockroach/pull/71587 -[#71621]: https://github.com/cockroachdb/cockroach/pull/71621 -[#71719]: https://github.com/cockroachdb/cockroach/pull/71719 -[#71745]: https://github.com/cockroachdb/cockroach/pull/71745 -[#71753]: https://github.com/cockroachdb/cockroach/pull/71753 -[#71771]: https://github.com/cockroachdb/cockroach/pull/71771 -[#71794]: https://github.com/cockroachdb/cockroach/pull/71794 -[#71830]: https://github.com/cockroachdb/cockroach/pull/71830 -[#71847]: https://github.com/cockroachdb/cockroach/pull/71847 -[#71852]: https://github.com/cockroachdb/cockroach/pull/71852 -[#71922]: https://github.com/cockroachdb/cockroach/pull/71922 -[#71982]: https://github.com/cockroachdb/cockroach/pull/71982 -[#72086]: https://github.com/cockroachdb/cockroach/pull/72086 -[#72270]: https://github.com/cockroachdb/cockroach/pull/72270 -[#72298]: https://github.com/cockroachdb/cockroach/pull/72298 -[#72444]: https://github.com/cockroachdb/cockroach/pull/72444 -[#72463]: https://github.com/cockroachdb/cockroach/pull/72463 -[#72507]: https://github.com/cockroachdb/cockroach/pull/72507 diff --git a/src/current/_includes/releases/v21.1/v21.1.13.md b/src/current/_includes/releases/v21.1/v21.1.13.md deleted file mode 100644 index d99b24a85ef..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.13.md +++ /dev/null @@ -1,100 +0,0 @@ -## v21.1.13 - -Release Date: January 10, 2022 - - - -

SQL language changes

- -- `"visible"` is now usable as a table or column name without extra quoting. [#72996][#72996] -- The `create_type_statements` table now has an index on `descriptor_id`. [#73672][#73672] -- Added a new `stmt` column to the `crdb_internal.(cluster|node)_distsql_flows` virtual table. This column is populated on a best-effort basis. [#73582][#73582] -- The [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) that controls the default value for the [session setting](https://www.cockroachlabs.com/docs/v21.1/set-vars) `reorder_joins_limit`, called `sql.defaults.reorder_joins_limit`, is now public and included in the docs. [#73891][#73891] -- Added escape character processing to constraint span generation. Previously, escaped-like lookups resulted in incorrect results. [#74258][#74258] - -

Operational changes

- -- Added a new `bulkio.ingest.flush_delay` cluster setting to act as a last-resort option to manually slow bulk-writing processes if needed for cluster stability. This should only be used if there is no better suited back-pressure mechanism available for the contended resource. [#73757][#73757] - -

DB Console changes

- -- Fixed drag-to-zoom on custom charts. [#72589][#72589] -- Node events will now display a permission error rather than an internal server error when a user does not have admin privileges to view events. [#72464][#72464] -- The absolute links on the [Advanced Debug](https://www.cockroachlabs.com/docs/v21.1/ui-debug-pages) page within the DB Console have been updated to relative links. This will enable these links to work with the superuser dashboard in the Cloud Console. [#73122][#73122] - -

Bug fixes

- -- Fixed a bug which allowed [computed columns](https://www.cockroachlabs.com/docs/v21.1/computed-columns) to also have [`DEFAULT` expressions](https://www.cockroachlabs.com/docs/v21.1/default-value). [#73192][#73192] -- Fixed a bug causing [`RESTORE`](https://www.cockroachlabs.com/docs/v21.1/restore) to sometimes map OIDs to invalid types in certain circumstances containing user-defined types. [#73120][#73120] -- [`BACKUP WITH revision_history`](https://www.cockroachlabs.com/docs/v21.1/backup) would previously fail on an upgraded but un-finalized cluster. Now, it should succeed. [#73110][#73110] -- Fixed a bug causing CockroachDB to not set the `TableOID` and `TableAttributeNumber` attributes of `RowDescription` message of pgwire protocol in some cases (these values would be left as 0). [#72449][#72449] -- Fixed a bug causing CockroachDB to encounter an internal error or crash when some queries involving tuples with [`ENUM`](https://www.cockroachlabs.com/docs/v21.1/enum) values were executed in a distributed manner. [#72481][#72481] -- Fixed a bug causing usernames in [`ALTER TABLE ... OWNER TO`](https://www.cockroachlabs.com/docs/v21.1/alter-table) to not be normalized to lower case. [#72658][#72658] -- Previously, `atttypmod` in `pg_catalog.pg_attributes` for `DECIMAL` types with precision but no width was incorrectly "-1". This is now populated correctly. [#72075][#72075] -- Corrected how `type` displays for ZM shapes `geometry_columns` to match PostGIS output. This previously incorrectly included the Z/M lettering. [#71434][#71434] -- Corrected how `type` displays in `geography_columns` to better match PostGIS. This was previously in the wrong casing. [#71434][#71434] -- The `setval` built-in function previously did not invalidate cached sequence values. This is fixed now. [#71822][#71822] -- Fixed a bug preventing tuple type labels from being propagated across queries when run under DistSQL. [#70391][#70391] -- Fixed a bug whereby setting the `CACHE` for a sequence to 1 was ignored. Before this change [`ALTER SEQUENCE ... CACHE 1`](https://www.cockroachlabs.com/docs/v21.1/alter-sequence) would succeed but would not modify the cache value. [#71448][#71448] -- When using [`COPY FROM .. BINARY`](https://www.cockroachlabs.com/docs/v21.1/copy-from), the correct format code will now be returned. [#69255][#69255] -- Fixed a bug causing `COPY FROM ... BINARY` to return an error if the input data was split across different messages. [#69255][#69255] -- Fixed a bug causing `COPY FROM ... CSV` to require each `CopyData` message to be split at the boundary of a record. This was a bug since the `COPY` protocol allows messages to be split at arbitrary points. [#69255][#69255] -- Fixed a bug causing `COPY FROM ... CSV` to not correctly handle octal byte escape sequences such as `\011` when using a [`BYTEA`](https://www.cockroachlabs.com/docs/v21.1/bytes) column. [#69255][#69255] -- Manually enqueueing ranges via the DB Console will no longer crash nodes that contain an uninitialized replica for the enqueued range. [#73038][#73038] -- Fixed a crash with message "attempting to propose command writing below closed timestamp" that could occur, typically on overloaded systems experiencing non-cooperative lease transfers. [#73166][#73166] -- The `txnwaitqueue.pusher.waiting` metric no longer over-reports the number of pushing transactions in some cases. [#71743][#71743] -- Fixed a rare condition that could cause a range merge to get stuck waiting on itself. The symptom of this deadlock was a goroutine stuck in `handleMergeInProgressError` for tens of minutes. [#72049][#72049] -- Fixed bugs causing [`CREATE TABLE AS`](https://www.cockroachlabs.com/docs/v21.1/create-table-as) and [`CREATE MATERIALIZED VIEW`](https://www.cockroachlabs.com/docs/v21.1/create-view) to panic if the `SELECT` query was an internal table requiring internal database state. [#73620][#73620] -- Fixed a rare internal error, "estimated row count must be non-zero", which could occur when planning queries using a GIN index. This error could occur if the histogram on the GIN index showed that there were no rows. [#73353][#73353] -- Fixed an internal error, "empty Datums being compared to other", that could occur during planning for some `SELECT` queries over tables that included a `DEFAULT` partition value in a `PARTITION BY LIST` clause. This bug had been present since 21.1.0. The bug does not exist on versions 20.2.x and earlier. [#73663][#73663] -- Fixed a bug in database and schema restore cleanup that resulted in a dangling descriptor entry on job failure. [#73412][#73412] -- Fixed a bug with the ungraceful shutdown of distributed queries in some rare cases. [#73959][#73959] -- Fixed a bug causing CockroachDB to return a spurious "context canceled" error for a query that actually succeeded in extremely rare cases. [#73959][#73959] -- Fixed a bug which caused corruption of [partial indexes](https://www.cockroachlabs.com/docs/v21.2/partial-indexes), which could cause incorrect query results. The bug was only present when two or more partial indexes in the same table had identical `WHERE` clauses. This bug had been present since version 21.1.0. [#74475][#74475] - -

Performance improvements

- -- Improved [`IMPORT INTO`](https://www.cockroachlabs.com/docs/v21.1/import) performance in cases where it encounters large numbers of unresolved write intents. [#72272][#72272] -- The performance of transaction deadlock detection is now more stable even with significant transaction contention. [#71743][#71743] -- Bulk ingestion of small write batches (e.g., index backfill into a large number of ranges) is now throttled, to avoid buildup of read amplification and associated performance degradation. Concurrency is controlled by the new cluster setting `kv.bulk_io_write.concurrent_addsstable_as_writes_requests`. [#74072][#74072] - -

Contributors

- -This release includes 52 merged PRs by 26 authors. - -[#69255]: https://github.com/cockroachdb/cockroach/pull/69255 -[#70391]: https://github.com/cockroachdb/cockroach/pull/70391 -[#71434]: https://github.com/cockroachdb/cockroach/pull/71434 -[#71448]: https://github.com/cockroachdb/cockroach/pull/71448 -[#71743]: https://github.com/cockroachdb/cockroach/pull/71743 -[#71822]: https://github.com/cockroachdb/cockroach/pull/71822 -[#72049]: https://github.com/cockroachdb/cockroach/pull/72049 -[#72075]: https://github.com/cockroachdb/cockroach/pull/72075 -[#72272]: https://github.com/cockroachdb/cockroach/pull/72272 -[#72449]: https://github.com/cockroachdb/cockroach/pull/72449 -[#72464]: https://github.com/cockroachdb/cockroach/pull/72464 -[#72481]: https://github.com/cockroachdb/cockroach/pull/72481 -[#72589]: https://github.com/cockroachdb/cockroach/pull/72589 -[#72658]: https://github.com/cockroachdb/cockroach/pull/72658 -[#72756]: https://github.com/cockroachdb/cockroach/pull/72756 -[#72996]: https://github.com/cockroachdb/cockroach/pull/72996 -[#73038]: https://github.com/cockroachdb/cockroach/pull/73038 -[#73110]: https://github.com/cockroachdb/cockroach/pull/73110 -[#73120]: https://github.com/cockroachdb/cockroach/pull/73120 -[#73122]: https://github.com/cockroachdb/cockroach/pull/73122 -[#73166]: https://github.com/cockroachdb/cockroach/pull/73166 -[#73192]: https://github.com/cockroachdb/cockroach/pull/73192 -[#73353]: https://github.com/cockroachdb/cockroach/pull/73353 -[#73412]: https://github.com/cockroachdb/cockroach/pull/73412 -[#73582]: https://github.com/cockroachdb/cockroach/pull/73582 -[#73620]: https://github.com/cockroachdb/cockroach/pull/73620 -[#73663]: https://github.com/cockroachdb/cockroach/pull/73663 -[#73672]: https://github.com/cockroachdb/cockroach/pull/73672 -[#73757]: https://github.com/cockroachdb/cockroach/pull/73757 -[#73891]: https://github.com/cockroachdb/cockroach/pull/73891 -[#73959]: https://github.com/cockroachdb/cockroach/pull/73959 -[#74072]: https://github.com/cockroachdb/cockroach/pull/74072 -[#74124]: https://github.com/cockroachdb/cockroach/pull/74124 -[#74205]: https://github.com/cockroachdb/cockroach/pull/74205 -[#74258]: https://github.com/cockroachdb/cockroach/pull/74258 -[#74475]: https://github.com/cockroachdb/cockroach/pull/74475 diff --git a/src/current/_includes/releases/v21.1/v21.1.14.md b/src/current/_includes/releases/v21.1/v21.1.14.md deleted file mode 100644 index b845db985e1..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.14.md +++ /dev/null @@ -1,56 +0,0 @@ -## v21.1.14 - -Release Date: February 9, 2022 - - - -

SQL language changes

- -- Collated strings may now have a locale that is a language tag, followed by a `-u-` suffix, followed by anything else. For example, any locale with a prefix of `en-US-u-` is now considered valid. [#74754][#74754] - -

Command-line changes

- -- Fixed the CLI help text for [`ALTER DATABASE`](https://www.cockroachlabs.com/docs/v21.1/alter-database) to show correct options for [`ADD REGION`](https://www.cockroachlabs.com/docs/v21.1/add-region) and [`DROP REGION`](https://www.cockroachlabs.com/docs/v21.1/drop-region), and include some missing options such as [`CONFIGURE ZONE`](https://www.cockroachlabs.com/docs/v21.1/configure-zone). [#75075][#75075] - -

Bug fixes

- -- Fixed a panic when attempting to access the hottest ranges (e.g., via the `/_status/hotranges` endpoint) before initial statistics had been gathered. [#74514][#74514] -- Servers no longer crash due to panics in HTTP handlers. [#74533][#74533] -- Previously, running [`IMPORT TABLE ... PGDUMP DATA`](https://www.cockroachlabs.com/docs/v21.1/import) with a [`COPY FROM`](https://www.cockroachlabs.com/docs/v21.1/copy-from) statement in the dump file that had fewer target columns than the inline table definition would result in a nil pointer exception. This is now fixed. [#74452][#74452] -- Previously, a doubly nested [`ENUM`](https://www.cockroachlabs.com/docs/v21.1/enum) in a DistSQL query would not be hydrated on remote nodes, resulting in panic. This is now fixed. [#74527][#74527] -- Error messages produced during [import](https://www.cockroachlabs.com/docs/v21.1/import) are now truncated. Previously, import could potentially generate large error messages that could not be persisted to the jobs table, resulting in a failed import never entering the failed state and instead retrying repeatedly. [#73335][#73335] -- Fixed a bug where deleting data via schema changes (e.g., when dropping an index or table) could fail with a `command too large` error. [#74797][#74797] -- Fixed panics that were possible in some distributed queries using [`ENUM`](https://www.cockroachlabs.com/docs/v21.1/enum)s in join predicates. [#75086][#75086] -- Fixed a bug which caused errors in rare cases when trying to divide [`INTERVAL`](https://www.cockroachlabs.com/docs/v21.1/interval) values by `INT4` or `INT2` values. [#75091][#75091] -- Fixed a bug that caused internal errors when altering the primary key of a table. The bug was only present if the table had a partial index with a predicate that referenced a virtual computed column. This bug was present since virtual computed columns were added in v21.1.0. [#75193][#75193] -- Fixed a bug that could occur when a [`TIMETZ`](https://www.cockroachlabs.com/docs/v21.1/time) column was indexed, and a query predicate constrained that column using a `<` or `>` operator with a `TIMETZ` constant. If the column contained values with time zones that did not match the time zone of the `TIMETZ` constant, it was possible that not all matching values could be returned by the query. Specifically, the results may not have included values within one microsecond of the predicate's absolute time. This bug was introduced when the `TIMETZ` datatype was first added in v20.1. It exists on all releases of v20.1, v20.2, v21.1, and v21.2 prior to this patch. [#75173][#75173] -- Fixed an internal error, `estimated row count must be non-zero`, that could occur during planning for queries over a table with a [`TIMETZ`](https://www.cockroachlabs.com/docs/v21.1/time) column. This error was due to a faulty assumption in the statistics estimation code about ordering of `TIMETZ` values, which has now been fixed. The error could occur when `TIMETZ` values used in the query had a different time zone offset than the `TIMETZ` values stored in the table. [#75173][#75173] -- Previously, during [restore](https://www.cockroachlabs.com/docs/v21.1/restore), CockroachDB would not insert a `system.namespace` entry for synthetic public schemas. This is now fixed. [#74760][#74760] -- [`RESTORE ... FROM LATEST IN`](https://www.cockroachlabs.com/docs/v21.1/restore) now works to restore the latest backup from a collection without needing to first inspect the collection to supply its actual path. [#75437][#75437] -- Fixed a bug that caused internal errors in queries with set operations, like `UNION`, when corresponding columns on either side of the set operation were not the same. This error only occurred with a limited set of types. This bug is present in v20.2.6+, v21.1.0+, and v21.2.0+. [#75294][#75294] -- The `CancelSession` endpoint now correctly propagates gateway metadata when forwarding requests. [#75886][#75886] - -

Contributors

- -This release includes 27 merged PRs by 16 authors. - -[#73335]: https://github.com/cockroachdb/cockroach/pull/73335 -[#74452]: https://github.com/cockroachdb/cockroach/pull/74452 -[#74514]: https://github.com/cockroachdb/cockroach/pull/74514 -[#74527]: https://github.com/cockroachdb/cockroach/pull/74527 -[#74533]: https://github.com/cockroachdb/cockroach/pull/74533 -[#74754]: https://github.com/cockroachdb/cockroach/pull/74754 -[#74760]: https://github.com/cockroachdb/cockroach/pull/74760 -[#74797]: https://github.com/cockroachdb/cockroach/pull/74797 -[#74893]: https://github.com/cockroachdb/cockroach/pull/74893 -[#75075]: https://github.com/cockroachdb/cockroach/pull/75075 -[#75086]: https://github.com/cockroachdb/cockroach/pull/75086 -[#75091]: https://github.com/cockroachdb/cockroach/pull/75091 -[#75173]: https://github.com/cockroachdb/cockroach/pull/75173 -[#75193]: https://github.com/cockroachdb/cockroach/pull/75193 -[#75294]: https://github.com/cockroachdb/cockroach/pull/75294 -[#75437]: https://github.com/cockroachdb/cockroach/pull/75437 -[#75886]: https://github.com/cockroachdb/cockroach/pull/75886 -[#75891]: https://github.com/cockroachdb/cockroach/pull/75891 -[66bc0ab38]: https://github.com/cockroachdb/cockroach/commit/66bc0ab38 -[eeb15df70]: https://github.com/cockroachdb/cockroach/commit/eeb15df70 diff --git a/src/current/_includes/releases/v21.1/v21.1.15.md b/src/current/_includes/releases/v21.1/v21.1.15.md deleted file mode 100644 index 13899302ff7..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.15.md +++ /dev/null @@ -1,30 +0,0 @@ -## v21.1.15 - -Release Date: February 14, 2022 - -This page lists additions and changes in v21.1.15 since v21.1.14. - - - -

Enterprise edition changes

- -- Kafka sinks now support larger messages, up to 2GB in size. [#76322][#76322] - -

SQL language changes

- -- Non-admin users can now use the [`SHOW RANGES`](https://www.cockroachlabs.com/docs/v21.1/show-ranges) statement if the `ZONECONFIG` privilege is granted. [#76072][#76072] -- `ST_MakePolygon` is now disallowed from making empty polygons from empty linestrings. This is not allowed in PostGIS. [#76256][#76256] - -

Bug fixes

- -- Mixed dimension linestrings are now prohibited in `ST_MakePolygon`. [#76256][#76256] -- Fixed a bug which could cause nodes to crash when truncating abnormally large Raft logs. [#75980][#75980] - -

Contributors

- -This release includes 6 merged PRs by 6 authors. - -[#75980]: https://github.com/cockroachdb/cockroach/pull/75980 -[#76072]: https://github.com/cockroachdb/cockroach/pull/76072 -[#76256]: https://github.com/cockroachdb/cockroach/pull/76256 -[#76322]: https://github.com/cockroachdb/cockroach/pull/76322 diff --git a/src/current/_includes/releases/v21.1/v21.1.16.md b/src/current/_includes/releases/v21.1/v21.1.16.md deleted file mode 100644 index 5d2ba39b741..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.16.md +++ /dev/null @@ -1,32 +0,0 @@ -## v21.1.16 - -Release Date: March 7, 2022 - - - -

Bug fixes

- -- Fixed a bug that caused the query optimizer to omit join filters in rare cases when reordering joins, which could result in incorrect query results. This bug was present since v20.2. [#76621][#76621] -- Fixed a bug where some rows within query results could be omitted if the query references a column within a composite key with a null value. Previously, CockroachDB could incorrectly omit a row from query results against a table with multiple column families when that row contains a NULL value when a composite type ([FLOAT](https://www.cockroachlabs.com/docs/v21.1/float), [DECIMAL](https://www.cockroachlabs.com/docs/v21.1/decimal), [COLLATED STRING](https://www.cockroachlabs.com/docs/v21.1/collate), or any arrays of such a type) is included in the [PRIMARY KEY](https://www.cockroachlabs.com/docs/v21.1/primary-key). For the bug to occur, a composite column from the PRIMARY KEY must be included in any column family other than the first one. [#76637][#76637] -- Fixed a race condition that, in rare circumstances, could cause a node to panic with `unexpected Stopped processor` during shutdown. [#76828][#76828] -- There is now a 1 hour timeout when sending Raft snapshots to avoid stalled snapshot transfers. Stalled snapshot transfers could prevent Raft log truncation, thus growing the Raft log very large. This timeout is configurable via the `COCKROACH_RAFT_SEND_SNAPSHOT_TIMEOUT` environment variable. [#76830][#76830] -- [`CASE` expressions](https://www.cockroachlabs.com/docs/v21.1/scalar-expressions#simple-case-expressions) with branches that result in types that cannot be cast to a common type now result in a user-facing error instead of an internal error. [#76618][#76618] -- A bug has been fixed that could corrupt [indexes](https://www.cockroachlabs.com/docs/v21.1/indexes) containing [virtual columns](https://www.cockroachlabs.com/docs/v21.1/computed-columns) or expressions. The bug only occurred when the index's table had a [foreign key reference](https://www.cockroachlabs.com/docs/v21.1/foreign-key) to another table with an [`ON DELETE CASCADE`](https://www.cockroachlabs.com/docs/v21.1/foreign-key#foreign-key-actions) action, and a row was deleted in the referenced table. This bug was present since virtual columns were added in v21.1.0. [#77057][#77057] - -

Performance improvements

- -- Fixed a bug in the [histogram estimation code](https://www.cockroachlabs.com/docs/v21.1/cost-based-optimizer#control-histogram-collection) that could cause the optimizer to think a scan of a multi-column index would produce 0 rows, when in fact it would produce many rows. This could cause the optimizer to choose a suboptimal plan. This bug has now been fixed, making it less likely for the optimizer to choose a suboptimal plan when multiple multi-column indexes are available. [#76557][#76557] -- The accuracy of [histogram](https://www.cockroachlabs.com/docs/v21.1/cost-based-optimizer#control-histogram-collection) calculations for [`BYTES`](https://www.cockroachlabs.com/docs/v21.1/bytes) types has been improved. As a result, the optimizer should generate more efficient query plans in some cases. [#76797][#76797] - -

Contributors

- -This release includes 10 merged PRs by 6 authors. - -[#76557]: https://github.com/cockroachdb/cockroach/pull/76557 -[#76618]: https://github.com/cockroachdb/cockroach/pull/76618 -[#76621]: https://github.com/cockroachdb/cockroach/pull/76621 -[#76637]: https://github.com/cockroachdb/cockroach/pull/76637 -[#76797]: https://github.com/cockroachdb/cockroach/pull/76797 -[#76828]: https://github.com/cockroachdb/cockroach/pull/76828 -[#76830]: https://github.com/cockroachdb/cockroach/pull/76830 -[#77057]: https://github.com/cockroachdb/cockroach/pull/77057 diff --git a/src/current/_includes/releases/v21.1/v21.1.17.md b/src/current/_includes/releases/v21.1/v21.1.17.md deleted file mode 100644 index 62468b3fd86..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.17.md +++ /dev/null @@ -1,57 +0,0 @@ -## v21.1.17 - -Release Date: April 4, 2022 - - - -

Security updates

- -- Users can enable HSTS headers to be set on all HTTP requests, which force browsers to upgrade to HTTPS without a redirect. This is controlled by setting the `server.hsts.enabled` [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings), which is `false` by default, to `true`. [#77863][#77863] - -

Enterprise edition changes

- -- Currently executing [schedules](https://www.cockroachlabs.com/docs/v21.1/manage-a-backup-schedule) are cancelled immediately when the jobs scheduler is disabled. [#77314][#77314] - -

SQL language changes

- -- Added a `sql.auth.resolve_membership_single_scan.enabled` [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings), which changes the query for an internal role membership cache. Previously the code would recursively look up each role in the membership hierarchy, leading to multiple queries. With the setting on, it uses a single query. This setting is `false` by default. [#77624][#77624] - -

Operational changes

- -- The `cockroach debug tsdump` command now downloads histogram timeseries it silently omitted previously. [#78054][#78054] - -

Command-line changes

- -- The `cockroach debug tsdump` command now allows viewing timeseries data even in cases of node failure by allowing users to rerun the command with the import filename set to `"-"`. [#78057][#78057] -- Fixed a bug where starting [`cockroach demo`](https://www.cockroachlabs.com/docs/v21.1/cockroach-demo) with the `--global` flag would not simulate latencies correctly when combined with the `--insecure` flag. [#78173][#78173] - -

Bug fixes

- -- Fixed a bug where draining nodes in a cluster without shutting them down could stall foreground traffic in the cluster. [#77494][#77494] -- Fixed a bug that caused errors when attempting to create table statistics with [`CREATE STATISTICS`](https://www.cockroachlabs.com/docs/v21.1/create-statistics) or `ANALYZE` for a table containing an index which indexed only [virtual computed columns](https://www.cockroachlabs.com/docs/v21.1/computed-columns). [#77566][#77566] -- Added a limit of seven concurrent asynchronous consistency checks per store, with an upper timeout of one hour. This prevents abandoned consistency checks from building up in some circumstances, which could lead to increasing disk usage as they held onto [Pebble](https://www.cockroachlabs.com/docs/v21.1/architecture/storage-layer#pebble) snapshots. [#77612][#77612] -- Fixed a bug where the **Details** page was not loading for statements whose app name contains `/` were not properly loading their **Details** pages. [#77946][#77946] -- Fixed a memory leak in the [Pebble](https://www.cockroachlabs.com/docs/v21.1/architecture/storage-layer#pebble) block cache. [#78262][#78262] -- Fixed a bug that caused internal errors when `COALESCE` and `IF` [expressions](https://www.cockroachlabs.com/docs/v21.1/scalar-expressions) had inner expressions with different types that could not be cast to a common type. [#78345][#78345] -- Fixed a bug that caused errors when trying to evaluate queries with `NULL` values annotated as a tuple type, such as `NULL:::RECORD`. [#78638][#78638] -- Fixed a bug that caused the [optimizer](https://www.cockroachlabs.com/docs/v21.1/cost-based-optimizer) to generate invalid query plans which could result in incorrect query results. The bug, which has been present since version v21.1.0, can appear if all of the following conditions are true: 1) the query contains a semi-join, such as queries in the form: `SELECT * FROM t1 WHERE EXISTS (SELECT * FROM t2 WHERE t1.a = t2.a);`, 2) the inner table has an index containing the equality column, like `t2.a` in the example query, 3) the index contains one or more columns that prefix the equality column, and 4) the prefix columns are `NOT NULL` and are constrained to a set of constant values via a `CHECK` constraint or an `IN` condition in the filter. [#78976][#78976] - -

Contributors

- -This release includes 20 merged PRs by 15 authors. - -[#77314]: https://github.com/cockroachdb/cockroach/pull/77314 -[#77494]: https://github.com/cockroachdb/cockroach/pull/77494 -[#77566]: https://github.com/cockroachdb/cockroach/pull/77566 -[#77612]: https://github.com/cockroachdb/cockroach/pull/77612 -[#77624]: https://github.com/cockroachdb/cockroach/pull/77624 -[#77863]: https://github.com/cockroachdb/cockroach/pull/77863 -[#77946]: https://github.com/cockroachdb/cockroach/pull/77946 -[#78054]: https://github.com/cockroachdb/cockroach/pull/78054 -[#78057]: https://github.com/cockroachdb/cockroach/pull/78057 -[#78173]: https://github.com/cockroachdb/cockroach/pull/78173 -[#78262]: https://github.com/cockroachdb/cockroach/pull/78262 -[#78296]: https://github.com/cockroachdb/cockroach/pull/78296 -[#78345]: https://github.com/cockroachdb/cockroach/pull/78345 -[#78638]: https://github.com/cockroachdb/cockroach/pull/78638 -[#78976]: https://github.com/cockroachdb/cockroach/pull/78976 diff --git a/src/current/_includes/releases/v21.1/v21.1.18.md b/src/current/_includes/releases/v21.1/v21.1.18.md deleted file mode 100644 index 8d0af879551..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.18.md +++ /dev/null @@ -1,25 +0,0 @@ -## v21.1.18 - -Release Date: April 12, 2022 - - - - -

Bug fixes

- -- Fixed a bug where restores of data with multiple [column families](https://www.cockroachlabs.com/docs/v21.1/column-families) could be split illegally (within a single SQL row). This could result in temporary data unavailability until the ranges on either side of the invalid split are merged. [#79207][#79207] -- Fixed a bug where [`SHOW SCHEMAS FROM `](https://www.cockroachlabs.com/docs/v21.1/show-schemas) would not include user-defined schemas. [#79306][#79306] -- Previously, [`LIMIT`](https://www.cockroachlabs.com/docs/v21.1/limit-offset) queries with an [`ORDER BY`](https://www.cockroachlabs.com/docs/v21.1/order-by) clause which scan the index of a virtual system tables, such as `pg_type`, could return incorrect results. This has been corrected by teaching the [optimizer](https://www.cockroachlabs.com/docs/v21.1/cost-based-optimizer) that `LIMIT` operations cannot be pushed into ordered scans of virtual indexes. [#79466][#79466] -- Fixed a bug that caused the [optimizer](https://www.cockroachlabs.com/docs/v21.1/cost-based-optimizer) to generate invalid query plans which could result in incorrect query results. The bug, present since version v21.1.0, can appear if all of the following conditions are true: - - The query contains a semi-join, e.g., with the format `SELECT * FROM a WHERE EXISTS (SELECT * FROM b WHERE a.a @> b.b)`. - - The inner table has a multi-column [inverted index](https://www.cockroachlabs.com/docs/v21.1/inverted-indexes) containing the inverted column in the filter. - - The index prefix columns are constrained to a set of values via the filter or a [`CHECK`](https://www.cockroachlabs.com/docs/v21.1/check) constraint, e.g., with an `IN` operator. In the case of a `CHECK` constraint, the column is `NOT NULL`. [#79508][#79508] - -

Contributors

- -This release includes 12 merged PRs by 10 authors. - -[#79207]: https://github.com/cockroachdb/cockroach/pull/79207 -[#79306]: https://github.com/cockroachdb/cockroach/pull/79306 -[#79466]: https://github.com/cockroachdb/cockroach/pull/79466 -[#79508]: https://github.com/cockroachdb/cockroach/pull/79508 diff --git a/src/current/_includes/releases/v21.1/v21.1.19.md b/src/current/_includes/releases/v21.1/v21.1.19.md deleted file mode 100644 index 830c834701b..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.19.md +++ /dev/null @@ -1,41 +0,0 @@ -## v21.1.19 - -Release Date: May 9, 2022 - - - -

Enterprise edition changes

- -- Fixed a bug where backups in the base directory of a Google Storage bucket would not be discovered by [`SHOW BACKUPS`](https://www.cockroachlabs.com/docs/v21.1/show-backup). These backups will now appear correctly. [#80509][#80509] - -

SQL language changes

- -- Previously, a user could run an `AS OF SYSTEM TIME` [incremental backup](https://www.cockroachlabs.com/docs/v21.1/take-full-and-incremental-backups#incremental-backups) with an end time earlier than the previous backup's end time, which could lead to an out of order incremental backup chain. An incremental backup will now fail if the `AS OF SYSTEM TIME` is less than the previous backup's end time. [#80500][#80500] - -

DB Console changes

- -- Added a dropdown filter on the Node Diagnostics page to view by active, decommissioned, or all nodes. [#80340][#80340] -- Added an alert banner on Overview list page for staggered node versions [#80748][#80748] - -

Bug fixes

- -- Fixed a bug that caused an internal error when the inner expression of a column access expression evaluated to `NULL`. For example, evaluation of the expression `(CASE WHEN b THEN ((ROW(1) AS a)) ELSE NULL END).a` would error when `b` is `false`. This bug has been present since v19.1 or earlier. [#79527][#79527] -- Fixed a bug that caused an error when accessing a named column of a labelled tuple. The bug only occurred when an expression could produce one of several different tuples. For example, `(CASE WHEN true THEN (ROW(1) AS a) ELSE (ROW(2) AS a) END).a` would fail to evaluate. Although present in previous versions, it was impossible to encounter due to limitations that prevented using tuples in this way. [#79527][#79527] -- Addressed an issue where automatic encryption-at-rest data key rotation would get disabled after a node restart without a store key rotation. [#80171][#80171] -- The timeout when checking for Raft application of upgrade migrations has been increased from 5 seconds to 1 minute, and is now controllable via the [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) `kv.migration.migrate_application.timeout`. This makes migrations much less likely to fail in clusters with ongoing rebalancing activity during upgrade migrations. [#80754][#80754] -- Fixed a bug where, in rare circumstances, CockroachDB could incorrectly evaluate queries with `ORDER BY` clause when the prefix of ordering was already provided by the index ordering of the scanned table. [#80732][#80732] -- Fixed a goroutine leak when internal rangefeed clients received certain kinds of retryable errors. [#80795][#80795] - -

Contributors

- -This release includes 18 merged PRs by 13 authors. - -[#79527]: https://github.com/cockroachdb/cockroach/pull/79527 -[#80171]: https://github.com/cockroachdb/cockroach/pull/80171 -[#80340]: https://github.com/cockroachdb/cockroach/pull/80340 -[#80500]: https://github.com/cockroachdb/cockroach/pull/80500 -[#80509]: https://github.com/cockroachdb/cockroach/pull/80509 -[#80732]: https://github.com/cockroachdb/cockroach/pull/80732 -[#80748]: https://github.com/cockroachdb/cockroach/pull/80748 -[#80754]: https://github.com/cockroachdb/cockroach/pull/80754 -[#80795]: https://github.com/cockroachdb/cockroach/pull/80795 diff --git a/src/current/_includes/releases/v21.1/v21.1.2.md b/src/current/_includes/releases/v21.1/v21.1.2.md deleted file mode 100644 index 65ea773287a..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.2.md +++ /dev/null @@ -1,111 +0,0 @@ -## v21.1.2 - -Release Date: June 7, 2021 - - - -

General changes

- -- Added [multi-region workloads](https://www.cockroachlabs.com/docs/v21.1/movr-flask-overview) for `cockroach demo movr --geo-partitioned-replicas`. Setting `--multi-region` enables for multi-region workloads, and setting `--survive` allows for surviving `AZ` or `REGION` failures. Setting `--infer-crdb-region-column` also infers the `crdb_region` for `REGIONAL BY ROW` tables. [#65642][#65642] {% comment %}doc{% endcomment %} -- [Changefeeds](https://www.cockroachlabs.com/docs/v21.1/create-changefeed) now better handle slow or unavailable sinks by treating "memory exceeded" errors as retryable. [#65387][#65387] {% comment %}doc{% endcomment %} - -

SQL language changes

- -- Added the `crdb_internal.lost_descriptors_with_data` [function](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators) to show descriptors that have no entries but data left behind. [#65462][#65462] {% comment %}doc{% endcomment %} -- Added the `crdb_internal.force_delete_table_data` [function](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators) which allows table data to be cleaned up only using a descriptor ID for cases of table descriptor corruption. [#65462][#65462] {% comment %}doc{% endcomment %} -- The statement type ("tag") is now also included alongside the full text of the SQL query in the various structured log entries produced when query execution is being [logged](https://www.cockroachlabs.com/docs/v21.1/logging-overview). [#65554][#65554] {% comment %}doc{% endcomment %} -- CockroachDB now returns a SQL Notice if a [`CREATE TABLE IF NOT EXISTS`](https://www.cockroachlabs.com/docs/v21.1/create-table) command is used to create a `TABLE` and the `TABLE` already exists. [#65636][#65636] {% comment %}doc{% endcomment %} -- The `SHOW FULL TABLE SCANS` statement was added to CockroachDB. [#65671][#65671] {% comment %}doc{% endcomment %} -- CockroachDB now returns a SQL Notice if a [`CREATE TYPE IF NOT EXISTS`](https://www.cockroachlabs.com/docs/v21.1/create-type) command is used to create a type and the type already exists. [#65635][#65635] {% comment %}doc{% endcomment %} -- Added a `chunk_size` option to [`EXPORT INTO CSV`](https://www.cockroachlabs.com/docs/v21.1/export) to control the target CSV file size. [#65388][#65388] {% comment %}doc{% endcomment %} -- SQL stats can now be cleared using the `crdb_internal.reset_sql_stats()` function. [#65674][#65674] {% comment %}doc{% endcomment %} -- CockroachDB now supports [`ALTER DATABASE ... ADD REGION IF NOT EXISTS ...`](https://www.cockroachlabs.com/docs/v21.1/add-region) which does not cause an error when adding a region that is already in the database. [#65752][#65752] {% comment %}doc{% endcomment %} -- CockroachDB now outputs a clearer error message when running [`ALTER DATABASE ... ADD REGION ...`](https://www.cockroachlabs.com/docs/v21.1/add-region) if the region is an undefined region. Previously, the error message for not having a region defined on a database resulted in an error about enums. [#65752][#65752] {% comment %}doc{% endcomment %} -- Added the [`ALTER DATABASE ... DROP REGION IF EXISTS ...`](https://www.cockroachlabs.com/docs/v21.1/drop-region) statement syntax, which does not error if dropping a region that is not defined on the database. [#65752][#65752] {% comment %}doc{% endcomment %} -- Fixed a bug where transitioning from locality `REGIONAL BY ROW` to `GLOBAL` or `REGIONAL BY TABLE` could mistakenly remove a zone configuration on an index which has no multi-region fields set. [#65833][#65833] {% comment %}doc{% endcomment %} -- CockroachDB now only blocks a [zone configuration DISCARD](https://www.cockroachlabs.com/docs/v21.1/configure-zone) on a multi-region table, index, or partition if the multi-region abstractions created the zone configuration. [#65834][#65834] {% comment %}doc{% endcomment %} - -

Operational changes

- -- [Range metrics](https://www.cockroachlabs.com/docs/v21.1/ui-replication-dashboard) are now gathered from the leaseholder (if live) rather than the first available range replica. This avoids scenarios where a stale replica may yield incorrect metrics, in particular over/underreplication markers. [#64590][#64590] {% comment %}doc{% endcomment %} - -

DB Console changes

- -- Fixed [Jobs page](https://www.cockroachlabs.com/docs/v21.1/ui-jobs-page) crash while using pagination and improved its performance. [#65723][#65723] -- Fixed a typo on the Network tooltip on the [Statements page](https://www.cockroachlabs.com/docs/v21.1/ui-statements-page). [#65605][#65605] -- Fixed a missing node ID in the rejoin [event message](https://www.cockroachlabs.com/docs/v21.1/ui-runtime-dashboard#events-panel) [#65806][#65806] -- Sorts on tables now pick up the correct value from the URL. [#65605][#65605] - -

Bug fixes

- -- Fixed a bug where a certain percentage of cases in which a node could have served a [follower read](https://www.cockroachlabs.com/docs/v21.1/follower-reads) were not handled correctly, resulting in the node routing the request to another nearby node for no reason. [#65471][#65471] -- The [`has_database_privilege`](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators) function now correctly will check privileges on databases that are not the current database being used by the session. [#65534][#65534] -- Fixed a bug where CockroachDB would previously crash when attempting to create a table using [`CREATE TABLE ... AS`](https://www.cockroachlabs.com/docs/v21.1/create-table-as) syntax where the `AS` clause selects from `crdb_internal.node_statement_statistics`, `crdb_internal.node_transaction_statistics`, or `crdb_internal.node_txn_stats` virtual tables. [#65542][#65542] -- Fixed a bug which allowed [index definitions](https://www.cockroachlabs.com/docs/v21.1/indexes) with redundant columns, which led to unnecessary storage usage. This bug can notably manifest itself with [`ALTER TABLE`](https://www.cockroachlabs.com/docs/v21.1/alter-table) statements which alter the primary index on a partitioned table. This bug has been present for a long time in theory, but in practice would only appear in CockroachDB since version 21.1.0. [#65482][#65482] -- Fixed a bug where binary [`TIMETZ`](https://www.cockroachlabs.com/docs/v21.1/time) values were not being decoded correctly when being sent as a parameter in the wire protocol. [#65341][#65341] -- Fixed a race condition during transaction cleanup that could leave old transaction records behind until [MVCC garbage collection](https://www.cockroachlabs.com/docs/v21.1/architecture/storage-layer#mvcc). [#65383][#65383] -- Improved [transaction cleanup](https://www.cockroachlabs.com/docs/v21.1/architecture/transaction-layer) for disconnected clients, to reduce intent buildup. [#65383][#65383] -- Added the ability to change the `COMMENT` on a column after using [`ALTER TYPE`](https://www.cockroachlabs.com/docs/v21.1/alter-type) on that column. [#65698][#65698] -- [Scheduled backup](https://www.cockroachlabs.com/docs/v21.1/create-schedule-for-backup) with [interleaved tables](https://www.cockroachlabs.com/docs/v21.1/interleave-in-parent) can now be created with the `include_deprecated_interleaves` option. [#65731][#65731] -- Fixed a bug where `ST_Node` on a [`LINESTRING`](https://www.cockroachlabs.com/docs/v21.1/linestring) with the same repeated points results in an error. [#65700][#65700] -- Calling [`get_bit` or `set_bit`](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators) on a byte array argument now goes to the correct index of the underlying bit string, in order to match the behavior of Postgres. [#65786][#65786] -- Fixed a bug where [`ALTER DATABASE ... CONVERT TO SCHEMA`](https://www.cockroachlabs.com/docs/v21.1/convert-to-schema) could potentially leave the schema with invalid privileges thus causing the privilege descriptor to be invalid. [#65810][#65810] -- CockroachDB now renders the `CACHE` clause for [sequences](https://www.cockroachlabs.com/docs/v21.1/create-sequence) which use a cache. [#65805][#65805] -- Fixed a bug that could cause a node to crash in rare cases if a [`BACKUP`](https://www.cockroachlabs.com/docs/v21.1/backup) writing to Google Cloud Storage failed. [#65802][#65802] -- Fixed a bug introduced in 21.1 where [cluster restores](https://www.cockroachlabs.com/docs/v21.1/restore) would report inaccurate progress, showing 100% progress erroneously. [#65803][#65803] -- Fixed a crash when performing a cluster [`BACKUP`](https://www.cockroachlabs.com/docs/v21.1/backup) with revision history of a cluster upgraded from 20.1 to 20.2 to 21.1 which contains tables that were truncated by 20.1. [#65860][#65860] -- Fixed a bug that caused incorrect results for queries where [`CHAR` and `VARCHAR`](https://www.cockroachlabs.com/docs/v21.1/string#related-types) columns are filtered by constant string values. The bug was present since version v21.1.0. [#66101][#66101] - -

Performance improvements

- -- The [optimizer](https://www.cockroachlabs.com/docs/v21.1/cost-based-optimizer) can now avoid full table scans for queries with a [`LIMIT`](https://www.cockroachlabs.com/docs/v21.1/limit-offset) and [`ORDER BY`](https://www.cockroachlabs.com/docs/v21.1/order-by) clause in some additional cases where the `ORDER BY` columns are not a prefix of an index. [#65392][#65392] -- The [optimizer](https://www.cockroachlabs.com/docs/v21.1/cost-based-optimizer) now generates query plans that scan indexes on virtual collated string columns, regardless of the casing or formatting of the collated locale in the query. [#65531][#65531] -- CockroachDB now reduces the number of round-trips required to call `pg_table_is_visible` in the context of [`pg_catalog` queries](https://www.cockroachlabs.com/docs/v21.1/pg-catalog). [#65807][#65807] - -
- -

Contributors

- -This release includes 58 merged PRs by 34 authors. -We would like to thank the following contributors from the CockroachDB community: - -- Max Neverov -- Rupesh Harode - -
- -[#64590]: https://github.com/cockroachdb/cockroach/pull/64590 -[#65341]: https://github.com/cockroachdb/cockroach/pull/65341 -[#65383]: https://github.com/cockroachdb/cockroach/pull/65383 -[#65387]: https://github.com/cockroachdb/cockroach/pull/65387 -[#65388]: https://github.com/cockroachdb/cockroach/pull/65388 -[#65392]: https://github.com/cockroachdb/cockroach/pull/65392 -[#65462]: https://github.com/cockroachdb/cockroach/pull/65462 -[#65471]: https://github.com/cockroachdb/cockroach/pull/65471 -[#65482]: https://github.com/cockroachdb/cockroach/pull/65482 -[#65531]: https://github.com/cockroachdb/cockroach/pull/65531 -[#65534]: https://github.com/cockroachdb/cockroach/pull/65534 -[#65542]: https://github.com/cockroachdb/cockroach/pull/65542 -[#65554]: https://github.com/cockroachdb/cockroach/pull/65554 -[#65605]: https://github.com/cockroachdb/cockroach/pull/65605 -[#65635]: https://github.com/cockroachdb/cockroach/pull/65635 -[#65636]: https://github.com/cockroachdb/cockroach/pull/65636 -[#65642]: https://github.com/cockroachdb/cockroach/pull/65642 -[#65671]: https://github.com/cockroachdb/cockroach/pull/65671 -[#65674]: https://github.com/cockroachdb/cockroach/pull/65674 -[#65698]: https://github.com/cockroachdb/cockroach/pull/65698 -[#65700]: https://github.com/cockroachdb/cockroach/pull/65700 -[#65723]: https://github.com/cockroachdb/cockroach/pull/65723 -[#65731]: https://github.com/cockroachdb/cockroach/pull/65731 -[#65752]: https://github.com/cockroachdb/cockroach/pull/65752 -[#65786]: https://github.com/cockroachdb/cockroach/pull/65786 -[#65802]: https://github.com/cockroachdb/cockroach/pull/65802 -[#65803]: https://github.com/cockroachdb/cockroach/pull/65803 -[#65805]: https://github.com/cockroachdb/cockroach/pull/65805 -[#65806]: https://github.com/cockroachdb/cockroach/pull/65806 -[#65807]: https://github.com/cockroachdb/cockroach/pull/65807 -[#65810]: https://github.com/cockroachdb/cockroach/pull/65810 -[#65833]: https://github.com/cockroachdb/cockroach/pull/65833 -[#65834]: https://github.com/cockroachdb/cockroach/pull/65834 -[#65860]: https://github.com/cockroachdb/cockroach/pull/65860 -[#66101]: https://github.com/cockroachdb/cockroach/pull/66101 diff --git a/src/current/_includes/releases/v21.1/v21.1.20.md b/src/current/_includes/releases/v21.1/v21.1.20.md deleted file mode 100644 index 3788bd5f4ba..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.20.md +++ /dev/null @@ -1,17 +0,0 @@ -## v21.1.20 - -Release Date: September 12, 2022 - - - -

Bug fixes

- -- Fixed a rare crash which could occur when [restarting a node](https://www.cockroachlabs.com/docs/v21.1/cockroach-start) after dropping tables. [#80568][#80568] -- Fixed a bug where the rollback of [materialized view creation](https://www.cockroachlabs.com/docs/v21.1/views#materialized-views) left references inside dependent objects. [#82098][#82098] - -

Contributors

- -This release includes 7 merged PRs by 6 authors. - -[#80568]: https://github.com/cockroachdb/cockroach/pull/80568 -[#82098]: https://github.com/cockroachdb/cockroach/pull/82098 diff --git a/src/current/_includes/releases/v21.1/v21.1.21.md b/src/current/_includes/releases/v21.1/v21.1.21.md deleted file mode 100644 index d1fb467f2cd..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.21.md +++ /dev/null @@ -1,15 +0,0 @@ -## v21.1.21 - -Release Date: September 15, 2022 - - - -

Command-line changes

- -- Added the `--log-config-vars` flag to the [`cockroach` CLI](https://www.cockroachlabs.com/docs/v21.1/cockroach-commands), which allows for environment variables to be specified for expansion within the logging configuration file. This change allows for a single logging configuration file to service an array of sinks without further manipulation of the configuration file. [#85173][#85173] - -

Contributors

- -This release includes 1 merged PR by 2 authors. - -[#85173]: https://github.com/cockroachdb/cockroach/pull/85173 diff --git a/src/current/_includes/releases/v21.1/v21.1.3.md b/src/current/_includes/releases/v21.1/v21.1.3.md deleted file mode 100644 index ce7616973a7..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.3.md +++ /dev/null @@ -1,114 +0,0 @@ -## v21.1.3 - -Release Date: June 21, 2021 - - - -

Security updates

- -- Syntax errors in the host-based authentication (HBA) configuration in cluster setting `server.host_based_authentication.configuration` are now logged on the [OPS channel](https://www.cockroachlabs.com/docs/v21.1/logging). [#66128][#66128] {% comment %}doc{% endcomment %} -- The `User` and `ApplicationName` fields of structured events pertaining to SQL queries are now marked as non-sensitive when they contain certain values (`root`/`node` for `User` and values starting with `$` for application names). [#66443][#66443] {% comment %}doc{% endcomment %} - -

Enterprise edition changes

- -- Changefeeds with custom Kafka client configurations (using the `kafka_sink_config` object) that could lead to long delays in flushing messages will now produce an error. [#66265][#66265] {% comment %}doc{% endcomment %} -- The `kafka_sink_config` object now supports a `version` configuration item to specify Kafka server versions. This is likely only necessary for old (Kafka 0.11/Confluent 3.3 or earlier) Kafka servers. Additionally, settings not specified in `kafka_sink_config` now retain their default values. [#66314][#66314] {% comment %}doc{% endcomment %} - -

SQL language changes

- -- [`TRUNCATE`](https://www.cockroachlabs.com/docs/v21.1/truncate) is now less disruptive on tables with a large amount of concurrent traffic. [#65940][#65940] {% comment %}doc{% endcomment %} -- Creating `STORED` or `VIRTUAL` [computed columns](https://www.cockroachlabs.com/docs/v21.1/computed-columns) with expressions that reference [foreign key](https://www.cockroachlabs.com/docs/v21.1/foreign-key) columns is now allowed. [#66168][#66168] {% comment %}doc{% endcomment %} -- The new function [`crdb_internal.get_vmodule`](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators#system-info-functions) returns the current `vmodule` configuration on the node processing the request. [#63545][#63545] {% comment %}doc{% endcomment %} -- The description string for the `random()` function now clarifies that there are at most 53 bits of randomness available; that is, the function is unsuitable to generate 64-bit random integer values. This behavior is similar to that of PostgreSQL. [#66128][#66128] {% comment %}doc{% endcomment %} -- [`EXPLAIN ANALYZE`](https://www.cockroachlabs.com/docs/v21.1/explain-analyze) now displays information about the regions on which a statement was executed. [#66368][#66368] {% comment %}doc{% endcomment %} - -

Operational changes

- -- Added a configurable limit to the number of intents collected by a `scan` before aborting, to prevent out-of-memory errors. The setting `storage.mvcc.max_intents_per_error` replaces `storage.sst_export.max_intents_per_error` and covers both `scan` and `export` commands. [#65923][#65923] {% comment %}doc{% endcomment %} -- [`BACKUP`](https://www.cockroachlabs.com/docs/v21.1/backup) now puts backup data files in a `data` sub-directory of the `BACKUP` path instead of directly in the backup path. [#66161][#66161] {% comment %}doc{% endcomment %} - -

Command-line changes

- -- The informational messages printed out when [`cockroach demo`](https://www.cockroachlabs.com/docs/v21.1/cockroach-demo) starts have been updated to clarify that certain information is only needed when accessing the demo cluster from another tool. [#66129][#66129] {% comment %}doc{% endcomment %} -- [`cockroach demo`](https://www.cockroachlabs.com/docs/v21.1/cockroach-demo) and [`cockroach sql`](https://www.cockroachlabs.com/docs/v21.1/cockroach-sql) are now able to run client-side commands via the `-e` command-line flag. This makes it possible to use commands like `\dt` or `\hf` from a shell script. [#66326][#66326] {% comment %}doc{% endcomment %} - -

DB Console changes

- -- Users can now reset SQL stats from the [DB Console](https://www.cockroachlabs.com/docs/v21.1/ui-overview). [#65916][#65916] {% comment %}doc{% endcomment %} -- Removed shading on line graphs, improving legibility when viewing more than a few series on the same plot. [#66032][#66032] {% comment %}doc{% endcomment %} -- Drag-to-zoom on metrics graphs now supports time ranges under 10 minutes. [#66032][#66032] {% comment %}doc{% endcomment %} -- In some cases, the Execution Stats page would show a very high Overhead latency for a statement. This could happen due to multiple statements being parsed together or due to statement execution being retried. To avoid this, we no longer consider the time between when parsing ends and execution begins when determining service latency. [#66108][#66108] {% comment %}doc{% endcomment %} -- Improved the style of the password input field for Safari. [#66134][#66134] -- The metrics chart under Overview was renamed from `SQL Queries` to `SQL Statements` to match the naming used under SQL Metrics. [#66364][#66364] {% comment %}doc{% endcomment %} - -

Bug fixes

- -- Fixed a bug which prevented adding columns to tables which contain data and use `NOT NULL` virtual columns [#65973][#65973] -- Fixed a bug in the DB Console where graphs for clusters with decommissioned nodes could show an empty series and data could be incorrectly attributed to the wrong nodes. [#66032][#66032] -- Fixed a bug where queries on `REGIONAL BY ROW` tables could fail in the brief window in which a DROP REGION operation is in progress. [#65984][#65984] -- Fixed a bug where a schema's privilege descriptor could be corrupted upon executing `ALTER DATABASE ... CONVERT TO SCHEMA`, where privileges invalid on a schema were copied from the database, rendering the schema unusable. [#65993][#65993] -- Fixed the error classification for duplicate index names where the later index was a `UNIQUE` index. [#64000][#64000] -- Fixed the error classification for `ALTER TABLE ... ADD CONSTRAINT ... UNIQUE` with the same name as an existing index. [#64000][#64000] -- Fixed a bug that made it less likely for range merges to succeed on clusters using multiple stores per node is now fixed. [#65889][#65889] -- Improved `TRUNCATE` operations to prevent contention issues. [#65940][#65940] -- Improved garbage collection of stale replicas by proactively checking certain replicas that have lost contact with other voting replicas. [#65186][#65186] -- Fixed a bug in `SHOW RANGES` which misattributed localities to nodes when using multiple stores. [#66037][#66037] -- Queries run through the `EXECUTE` statement can now generate statement diagnostic bundles as expected. [#66098][#66098] -- Previously, an `INSERT` causing a foreign key violation could result in an internal error in rare cases. The bug only affected the error response; any affected `INSERT`s (which would have been foreign-key violations) did not succeed. This bug, present since v21.1.0, has been fixed. [#66300][#66300] -- `BACKUP` and other operations can now reuse a previously created S3 client session when operating on the same bucket, which can avoid `NoCredentialProviders` errors on EC2 when iterating with large incremental backups. [#66259][#66259] -- The command exit status of `cockroach demo` and `cockroach sql` is now properly set to non-zero (error) after an error is encountered in a client-side command. Additionally, `cockroach sql` and `cockroach demo` now properly stop upon encountering an invalid configuration with `--set`, instead of starting to execute SQL statements after the invalid configuration. [#66326][#66326] -- Improved the availability of the jobs table for reads in large, global clusters by running background tasks at low priority. [#66344][#66344] -- Backups no longer risk the possibility of blocking conflicting writes while being rate limited by the `kv.bulk_io_write.concurrent_export_requests` concurrency limit. [#66408][#66408] -- The `soundex` and `st_difference` [built-in functions](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators#built-in-functions) for fuzzy string matching now correctly handle `NULL` values. [#66302][#66302] - -

Performance improvements

- -- Fixed an issue in the optimizer that prevented spatial predicates of the form `(column && value) = true` from being index-accelerated. These queries can now use a [spatial index](https://www.cockroachlabs.com/docs/v21.1/spatial-indexes) if one is available. [#65986][#65986] -- The optimizer is now more efficient when planning queries on tables that have many columns and indexes. [#66304][#66304] -- The `COCKROACHDB_REGISTRY` file is no longer rewritten whenever a new unencrypted file is created. [#66423][#66423] {% comment %}doc{% endcomment %} -- After improvements, queries use up to 1MB less system memory per scan, [lookup join, index join, zigzag join, or inverted join](https://www.cockroachlabs.com/docs/v21.1/joins) in their query plans. This will result in improved memory performance for workloads with concurrent OLAP-style queries. [#66145][#66145] -- Made improvements to prevent intra-query leaks during disk spilling that could cause the database to run out of memory, especially on tables with wide rows. [#66145][#66145] - -

Contributors

- -This release includes 64 merged PRs by 35 authors. - - -[#63545]: https://github.com/cockroachdb/cockroach/pull/63545 -[#64000]: https://github.com/cockroachdb/cockroach/pull/64000 -[#65186]: https://github.com/cockroachdb/cockroach/pull/65186 -[#65889]: https://github.com/cockroachdb/cockroach/pull/65889 -[#65916]: https://github.com/cockroachdb/cockroach/pull/65916 -[#65923]: https://github.com/cockroachdb/cockroach/pull/65923 -[#65940]: https://github.com/cockroachdb/cockroach/pull/65940 -[#65973]: https://github.com/cockroachdb/cockroach/pull/65973 -[#65984]: https://github.com/cockroachdb/cockroach/pull/65984 -[#65986]: https://github.com/cockroachdb/cockroach/pull/65986 -[#65993]: https://github.com/cockroachdb/cockroach/pull/65993 -[#66022]: https://github.com/cockroachdb/cockroach/pull/66022 -[#66032]: https://github.com/cockroachdb/cockroach/pull/66032 -[#66037]: https://github.com/cockroachdb/cockroach/pull/66037 -[#66098]: https://github.com/cockroachdb/cockroach/pull/66098 -[#66108]: https://github.com/cockroachdb/cockroach/pull/66108 -[#66128]: https://github.com/cockroachdb/cockroach/pull/66128 -[#66129]: https://github.com/cockroachdb/cockroach/pull/66129 -[#66134]: https://github.com/cockroachdb/cockroach/pull/66134 -[#66145]: https://github.com/cockroachdb/cockroach/pull/66145 -[#66161]: https://github.com/cockroachdb/cockroach/pull/66161 -[#66168]: https://github.com/cockroachdb/cockroach/pull/66168 -[#66259]: https://github.com/cockroachdb/cockroach/pull/66259 -[#66265]: https://github.com/cockroachdb/cockroach/pull/66265 -[#66300]: https://github.com/cockroachdb/cockroach/pull/66300 -[#66302]: https://github.com/cockroachdb/cockroach/pull/66302 -[#66304]: https://github.com/cockroachdb/cockroach/pull/66304 -[#66314]: https://github.com/cockroachdb/cockroach/pull/66314 -[#66326]: https://github.com/cockroachdb/cockroach/pull/66326 -[#66344]: https://github.com/cockroachdb/cockroach/pull/66344 -[#66364]: https://github.com/cockroachdb/cockroach/pull/66364 -[#66368]: https://github.com/cockroachdb/cockroach/pull/66368 -[#66408]: https://github.com/cockroachdb/cockroach/pull/66408 -[#66423]: https://github.com/cockroachdb/cockroach/pull/66423 -[#66443]: https://github.com/cockroachdb/cockroach/pull/66443 -[#66453]: https://github.com/cockroachdb/cockroach/pull/66453 -[#66508]: https://github.com/cockroachdb/cockroach/pull/66508 -[25c3d10a0]: https://github.com/cockroachdb/cockroach/commit/25c3d10a0 diff --git a/src/current/_includes/releases/v21.1/v21.1.4.md b/src/current/_includes/releases/v21.1/v21.1.4.md deleted file mode 100644 index 91d0860e279..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.4.md +++ /dev/null @@ -1,88 +0,0 @@ -## v21.1.4 - -Release Date: June 29, 2021 - -{{site.data.alerts.callout_danger}} -We recommend upgrading from this release to the [v21.1.5]({% link releases/v21.1.md %}#v21-1-5) bug fix release as soon as possible. -{{site.data.alerts.end}} - - - -

Security updates

- -- Previously, all the [logging](https://www.cockroachlabs.com/docs/v21.1/logging-overview) output to files or network sinks was disabled temporarily while an operator was using the `/debug/logspy` HTTP API, resulting in lost entries and a breach of auditability guarantees. This behavior has been corrected. [#66328][#66328] -- CockroachDB now configures a maximum number of concurrent password verifications in the server process, across UI and SQL clients. This limit reduces the risk of DoS attacks or accidents due to misbehaving clients. By default, the maximum amount of concurrency is ~12% of the number of allocated CPU cores (as per `GOMAXPROCS`), with a minimum of 1 core. This default can be overridden using the environment variable `COCKROACH_MAX_BCRYPT_CONCURRENCY`. [#66367][#66367] {% comment %}doc{% endcomment %} - -

SQL language changes

- -- Implemented the `ST_HasArc` [built-in function](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators#spatial-functions). This adds better out-of-the-box support for [GeoServer](https://www.cockroachlabs.com/docs/v21.1/geoserver). [#66531][#66531] {% comment %}doc{% endcomment %} -- Added a new internal [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings), `jobs.cancel_update_limit`, for controlling how many [jobs](https://www.cockroachlabs.com/docs/v21.1/show-jobs) are cleaned up concurrently after query cancellation. [#66488][#66488] {% comment %}doc{% endcomment %} - -

Command-line changes

- -- The [SQL shell](https://www.cockroachlabs.com/docs/v21.1/cockroach-sql) now formats times with time zones so that the minutes and seconds offsets are only shown if they are non-zero. Also, infinite floating point values are now formatted as `Infinity` rather than `Inf`. [#66130][#66130] {% comment %}doc{% endcomment %} -- When [log entries](https://www.cockroachlabs.com/docs/v21.1/logging-overview) are written to disk, the first few header lines written at the start of every new file now report the configured [logging format](https://www.cockroachlabs.com/docs/v21.1/log-formats). [#66328][#66328] {% comment %}doc{% endcomment %} - -

API endpoint changes

- -- The `/debug/logspy` HTTP API has changed. The endpoint now returns JSON data by default. If the previous format is desired, the user can pass the query argument `&flatten=1` to the `logspy` URL to obtain the previous flat text format (`crdb-v1`) instead. [#66328][#66328] This change is motivated as follows: {% comment %}doc{% endcomment %} - - The previous format, `crdb-v1`, cannot be parsed reliably. - - Using JSON entries guarantees that the text of each entry all fits on a single line of output (newline characters inside the messages are escaped). This makes filtering easier and more reliable. - - Using JSON enables the user to apply `jq` on the output, for example via `curl -s .../debug/logspy | jq ...`. -- The `/debug/logspy` API no longer enables maximum logging verbosity automatically. To change the verbosity, use the new `/debug/vmodule` endpoint or pass the `&vmodule=` query parameter to the `/debug/logspy` endpoint. [#66328][#66328] For example, suppose you wish to run a 20s logspy session: {% comment %}doc{% endcomment %} - - Before: `curl 'https://.../debug/logspy?duration=20s&...'`. - - Now: `curl 'https://.../debug/logspy?duration=20s&vmodule=...'` OR `curl 'https://.../debug/vmodule?duration=22s&vmodule=...' curl 'https://.../debug/logspy?duration=20s'`. - - As for the regular `vmodule` command-line flag, the maximum verbosity across all the source code can be selected with the pattern `*=4`. - - Note: at most one in-flight HTTP API request is allowed to modify the `vmodule` parameter. This maintains the invariant that the configuration restored at the end of each request is the same as when the request started. -- The new `/debug/vmodule` API makes it possible for an operator to configure the logging verbosity in a similar way as the SQL [built-in function](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators) `crdb_internal.set_vmodule()`, or to query the current configuration as in `crdb_internal.get_vmodule()`. Additionally, any configuration change performed via this API can be automatically reverted after a configurable delay. [#66328][#66328] The API forms are: {% comment %}doc{% endcomment %} - - `/debug/vmodule`: Retrieve the current configuration - - `/debug/vmodule?set=[vmodule config]&duration=[duration]`: Change the configuration to `[vmodule config]` . The previous configuration at the time the `/debug/vmodule` request started is restored after `[duration]`. This duration, if not specified, defaults to twice the default duration of a `logspy` request (currently, the `logspy` default duration is 5s, so the `vmodule` default duration is 10s). If the duration is zero or negative, the previous configuration is never restored. - -

DB Console changes

- -- Fixed an issue with displaying more than 100 hours of remaining time on the [Jobs page](https://www.cockroachlabs.com/docs/v21.1/ui-jobs-page). [#66596][#66596] - -

Bug fixes

- -- Minute timezone offsets are only displayed in the wire protocol if they are non-zero for `TimestampTZ` and `TimeTZ` values. Previously, they would always display. [#66130][#66130] -- Fixed a bug where binary `TimeTZ` values were not being decoded correctly when being sent as a parameter in the wire protocol. [#66130][#66130] -- CockroachDB's [SQL shell](https://www.cockroachlabs.com/docs/v21.1/cockroach-sql) now properly displays results of common array types, for example: arrays of floats, or arrays of strings. [#66130][#66130] -- Fixed a bug where the `--log='file-defaults: {format: crdb-v1}'` flag was not being handled properly. This bug existed since v21.1.0. [#66328][#66328] -- Fixed a bug where log entries could be lost while the `/debug/logspy` HTTP API was being used. This bug had existed since CockroachDB v1.1. [#66328][#66328] -- The binary encoding of decimals will no longer have negative `dscale` values. This was preventing [Npgsql](https://www.npgsql.org) from being able to read some binary decimals from CockroachDB. [#66532][#66532] -- A bug has been fixed which prevented the [optimizer](https://www.cockroachlabs.com/docs/v21.1/cost-based-optimizer) from producing plans with [partial indexes](https://www.cockroachlabs.com/docs/v21.1/partial-indexes) when executing some prepared statements that contained placeholders, stable functions, or casts. This bug was present since partial indexes were added in v20.2.0. [#66634][#66634] -- Fixed a bug which could have prevented [backups](https://www.cockroachlabs.com/docs/v21.1/backup) from being successfully [restored](https://www.cockroachlabs.com/docs/v21.1/restore). [#66616][#66616] -- Fixed a bug where CockroachDB could crash when executing [`EXPLAIN (VEC)`](https://www.cockroachlabs.com/docs/v21.1/explain#vec-option) on some mutations. The bug is present only in the v21.1.1-v21.1.3 releases. [#66573][#66573] -- Fixed a bug where CockroachDB could encounter an internal error when computing [window functions](https://www.cockroachlabs.com/docs/v21.1/window-functions) with `ROWS` mode of window framing if the offsets were very large for the `OFFSET FOLLOWING` boundary type. [#66446][#66446] -- Fixed a bug where using [`ADD COLUMN UNIQUE`](https://www.cockroachlabs.com/docs/v21.1/add-column) on [`REGIONAL BY ROW` tables](https://www.cockroachlabs.com/docs/v21.1/set-locality#regional-by-row) did not correctly add the [zone configs](https://www.cockroachlabs.com/docs/v21.1/configure-replication-zones) for the newly created column index. [#66696][#66696] -- Fixed a bug where reading from Google Cloud Storage was not using the resuming reader, as a result of which some retryable errors were not being treated as such, and the read would fail. [#66190][#66190] -- Fixed a deadlock during [backups](https://www.cockroachlabs.com/docs/v21.1/backup) and [imports](https://www.cockroachlabs.com/docs/v21.1/import). [#66773][#66773] -- Fixed incorrect accounting for statement/transaction sampled [execution statistics](https://www.cockroachlabs.com/docs/v21.1/explain-analyze). [#66790][#66790] -- Fixed a bug causing [transactions](https://www.cockroachlabs.com/docs/v21.1/transactions) to be spuriously aborted in rare circumstances. [#66567][#66567] - -

DB Console

- -- Fixed a CSS width calculation which was causing the multibar to not be visible in the [DB Console](https://www.cockroachlabs.com/docs/v21.1/ui-overview). [#66739][#66739] - -

Contributors

- -This release includes 28 merged PRs by 22 authors. - -[#66130]: https://github.com/cockroachdb/cockroach/pull/66130 -[#66190]: https://github.com/cockroachdb/cockroach/pull/66190 -[#66328]: https://github.com/cockroachdb/cockroach/pull/66328 -[#66367]: https://github.com/cockroachdb/cockroach/pull/66367 -[#66446]: https://github.com/cockroachdb/cockroach/pull/66446 -[#66473]: https://github.com/cockroachdb/cockroach/pull/66473 -[#66488]: https://github.com/cockroachdb/cockroach/pull/66488 -[#66531]: https://github.com/cockroachdb/cockroach/pull/66531 -[#66532]: https://github.com/cockroachdb/cockroach/pull/66532 -[#66567]: https://github.com/cockroachdb/cockroach/pull/66567 -[#66573]: https://github.com/cockroachdb/cockroach/pull/66573 -[#66596]: https://github.com/cockroachdb/cockroach/pull/66596 -[#66616]: https://github.com/cockroachdb/cockroach/pull/66616 -[#66634]: https://github.com/cockroachdb/cockroach/pull/66634 -[#66696]: https://github.com/cockroachdb/cockroach/pull/66696 -[#66739]: https://github.com/cockroachdb/cockroach/pull/66739 -[#66773]: https://github.com/cockroachdb/cockroach/pull/66773 -[#66790]: https://github.com/cockroachdb/cockroach/pull/66790 diff --git a/src/current/_includes/releases/v21.1/v21.1.5.md b/src/current/_includes/releases/v21.1/v21.1.5.md deleted file mode 100644 index 138358895e9..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.5.md +++ /dev/null @@ -1,15 +0,0 @@ -## v21.1.5 - -Release Date: July 2, 2021 - -{{site.data.alerts.callout_danger}} -We recommend upgrading from [v21.1.4]({% link releases/v21.1.md %}#v21-1-4) to this bug fix release as soon as possible. -{{site.data.alerts.end}} - - - -

Bug fixes

- -- Fixed a panic that could occur in the [optimizer](https://www.cockroachlabs.com/docs/v21.1/cost-based-optimizer) when executing a prepared plan with placeholders. This could happen when one of the tables used by the query had [computed columns](https://www.cockroachlabs.com/docs/v21.1/computed-columns) or a [partial index](https://www.cockroachlabs.com/docs/v21.1/partial-indexes). [#66883][#66883] - -[#66883]: https://github.com/cockroachdb/cockroach/pull/66833 diff --git a/src/current/_includes/releases/v21.1/v21.1.6.md b/src/current/_includes/releases/v21.1/v21.1.6.md deleted file mode 100644 index 1c9dea1c4b0..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.6.md +++ /dev/null @@ -1,84 +0,0 @@ -## v21.1.6 - -Release Date: July 20, 2021 - - - -

General changes

- -- Switched the `release-21.1` branch to point to a fork of the Google Cloud SDK at version 0.45.1 with an [upstream bug fix](https://github.com/googleapis/google-cloud-go/pull/4226). [#66808][#66808] - -

Enterprise edition changes

- -- Improved the internal buffering of [changefeeds](https://www.cockroachlabs.com/docs/v21.1/stream-data-out-of-cockroachdb-using-changefeeds) to be robust to bursts in traffic which exceed the throughput to the sink. [#67205][#67205] -- [Incremental backups](https://www.cockroachlabs.com/docs/v21.1/take-full-and-incremental-backups#incremental-backups) to a cloud storage location that already contains large existing backups now find their derived destination without listing as many remote files. [#67286][#67286] - -

Operational changes

- -- Added logs for important events during the server draining/shutdown process: (1) log when the server closes an existing connection while draining, (2) log when the server rejects a new connection while draining, and (3) log when the server cancels in-flight queries after waiting for the [`server.shutdown.query_wait`](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) duration to elapse while draining. [#66874][#66874] {% comment %}doc{% endcomment %} - -

DB Console changes

- -- Fixed a bug preventing the [Custom Chart debug page](https://www.cockroachlabs.com/docs/v21.1/ui-custom-chart-debug-page) from loading on initial. [#66897][#66897] -- Added a drag-to-select timescale feature to the [Custom Chart debug page](https://www.cockroachlabs.com/docs/v21.1/ui-custom-chart-debug-page). [#66594][#66594] {% comment %}doc{% endcomment %} - -

Bug fixes

- -- CockroachDB now allows a node with lease preferences to drain gracefully. [#66712][#66712] -- Fixed a panic that could occur in the [optimizer](https://www.cockroachlabs.com/docs/v21.1/cost-based-optimizer) when executing a prepared plan with placeholders. This could happen when one of the tables used by the query had computed columns or a partial index. [#66833][#66833] -- Fixed a bug that caused graceful drain to call `time.sleep` multiple times, which cut into time needed for range lease transfers. [#66851][#66851] -- Reduced CPU usage of idle `cockroach` processes. [#66894][#66894] -- Fixed an error that backup would produce in some [full backups with revision history](https://www.cockroachlabs.com/docs/v21.1/take-backups-with-revision-history-and-restore-from-a-point-in-time). Previously, some full backups would erroneously produce an error in the form of `batch timestamp must be after replica GC threshold `. [#66904][#66904] -- CockroachDB now avoids interacting with [decommissioned nodes](https://www.cockroachlabs.com/docs/v21.1/remove-nodes#how-it-works) during DistSQL planning and consistency checking. [#66950][#66950] -- [Changefeeds](https://www.cockroachlabs.com/docs/v21.1/stream-data-out-of-cockroachdb-using-changefeeds) no longer interact poorly with large, abandoned transactions. Previously, this combination could result in a cascade of work during transaction cleanup that could starve out foreground traffic. [#66813][#66813] -- [Changefeeds](https://www.cockroachlabs.com/docs/v21.1/stream-data-out-of-cockroachdb-using-changefeeds) now properly invalidate cached range descriptors and retry when encountering [decommissioned nodes](https://www.cockroachlabs.com/docs/v21.1/remove-nodes#how-it-works). [#67013][#67013] -- Fixed a bug where [metrics pages](https://www.cockroachlabs.com/docs/v21.1/ui-overview-dashboard) would lose their scroll position on chart data updates. [#67089][#67089] -- Fixed a bug which caused internal errors when running an [`IMPORT TABLE`](https://www.cockroachlabs.com/docs/v21.1/import) statement with a virtual computed column to import a CSV file. This bug has been present since virtual computed columns were introduced in v21.1.0. [#67043][#67043] -- Previously, the CLI would attribute some of the time spent running schema changes to network latency. This has now been fixed. Additionally, verbose timings in the CLI now also show time spent running schema changes in a separate bucket. [#65719][#65719] -- Fixed a statement buffer memory leak when using suspended portals. [#67372][#67372] -- [Changefeeds](https://www.cockroachlabs.com/docs/v21.1/stream-data-out-of-cockroachdb-using-changefeeds) on tables with array columns now support Avro. [#67433][#67433] -- Avro encoding now supports collated string types. [#67433][#67433] -- [`ENUM`](https://www.cockroachlabs.com/docs/v21.1/enum) columns can now be encoded in Avro [changefeeds](https://www.cockroachlabs.com/docs/v21.1/stream-data-out-of-cockroachdb-using-changefeeds) [#67433][#67433] -- Avro [changefeeds](https://www.cockroachlabs.com/docs/v21.1/stream-data-out-of-cockroachdb-using-changefeeds) now support `BIT` and `VARBIT` columns. [#67433][#67433] -- [`INTERVAL`](https://www.cockroachlabs.com/docs/v21.1/interval) columns are now supported in Avro [changefeeds](https://www.cockroachlabs.com/docs/v21.1/stream-data-out-of-cockroachdb-using-changefeeds). [#67433][#67433] - -

Performance improvements

- -- CockroachDB now continues to generate histograms when table statistics collection reaches memory limits, instead of disabling histogram generation. [#67057][#67057] {% comment %}doc{% endcomment %} -- The [optimizer](https://www.cockroachlabs.com/docs/v21.1/cost-based-optimizer) now prefers performing a reverse scan over a forward scan + sort if the reverse scan eliminates the need for a sort and the plans are otherwise equivalent. This was previously the case in most cases, but some edge cases with a small number of rows have been fixed. [#67388][#67388] -- When choosing between index scans that are estimated to have the same number of rows, the [optimizer](https://www.cockroachlabs.com/docs/v21.1/cost-based-optimizer) now prefers indexes for which it has higher certainty about the maximum number of rows over indexes for which there is more uncertainty in the estimated row count. This helps to avoid choosing suboptimal plans for small tables or if the statistics are stale. [#67388][#67388] - -
- -

Contributors

- -This release includes 47 merged PRs by 34 authors. -We would like to thank the following contributors from the CockroachDB community: - -- joesankey (first-time contributor) - -
- -[#65719]: https://github.com/cockroachdb/cockroach/pull/65719 -[#66594]: https://github.com/cockroachdb/cockroach/pull/66594 -[#66712]: https://github.com/cockroachdb/cockroach/pull/66712 -[#66808]: https://github.com/cockroachdb/cockroach/pull/66808 -[#66813]: https://github.com/cockroachdb/cockroach/pull/66813 -[#66833]: https://github.com/cockroachdb/cockroach/pull/66833 -[#66851]: https://github.com/cockroachdb/cockroach/pull/66851 -[#66874]: https://github.com/cockroachdb/cockroach/pull/66874 -[#66894]: https://github.com/cockroachdb/cockroach/pull/66894 -[#66897]: https://github.com/cockroachdb/cockroach/pull/66897 -[#66904]: https://github.com/cockroachdb/cockroach/pull/66904 -[#66950]: https://github.com/cockroachdb/cockroach/pull/66950 -[#67013]: https://github.com/cockroachdb/cockroach/pull/67013 -[#67043]: https://github.com/cockroachdb/cockroach/pull/67043 -[#67057]: https://github.com/cockroachdb/cockroach/pull/67057 -[#67089]: https://github.com/cockroachdb/cockroach/pull/67089 -[#67205]: https://github.com/cockroachdb/cockroach/pull/67205 -[#67286]: https://github.com/cockroachdb/cockroach/pull/67286 -[#67356]: https://github.com/cockroachdb/cockroach/pull/67356 -[#67372]: https://github.com/cockroachdb/cockroach/pull/67372 -[#67388]: https://github.com/cockroachdb/cockroach/pull/67388 -[#67433]: https://github.com/cockroachdb/cockroach/pull/67433 -[a8ebcb0af]: https://github.com/cockroachdb/cockroach/commit/a8ebcb0af diff --git a/src/current/_includes/releases/v21.1/v21.1.7.md b/src/current/_includes/releases/v21.1/v21.1.7.md deleted file mode 100644 index aff262277cc..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.7.md +++ /dev/null @@ -1,92 +0,0 @@ -## v21.1.7 - -Release Date: August 9, 2021 - - - -

Security updates

- -- The `--cert-principal-map` flag passed to [`cockroach` commands](https://www.cockroachlabs.com/docs/v21.1/cockroach-commands) now allows the certificate principal name to contain colons. [#67810][#67810] {% comment %}doc{% endcomment %} - -

General changes

- -- Added a new [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) (`kv.transaction.reject_over_max_intents_budget`) that controls the behavior of CockroachDB when a transaction exceeds the locks-tracking memory budget set by the `kv.transaction.max_intents_bytes` cluster setting. If `kv.transaction.reject_over_max_intents_budget` is set to `true`, CockroachDB rejects the query that would push its transaction over the memory budget with an error (error code 53400 - "configuration limit exceeded"). Transactions that don't track their locks precisely are potentially destabilizing for the cluster since cleaning up locks can take up considerable resources. Transactions that change many rows have the potential to run into this memory budget issue. [#67967][#67967] {% comment %}doc{% endcomment %} - -

SQL language changes

- -- The remote [DistSQL](https://www.cockroachlabs.com/docs/v21.1/architecture/sql-layer#distsql) flows are now eagerly canceled if they were queued up and the query was canceled. [#66331][#66331] -- Added a new cluster setting (`changefeed.slow_span_log_threshold`) that allows setting a cluster-wide default for slow span logging. [#68106][#68106] {% comment %}doc{% endcomment %} -- Added a new [session variable](https://www.cockroachlabs.com/docs/v21.1/set-vars) (`enable_copying_partitioning_when_deinterleaving_table`) which will change the behavior of [`ALTER PRIMARY KEY`](https://www.cockroachlabs.com/docs/v21.1/alter-primary-key) when performing a change which retains the same primary key but removes an [`INTERLEAVE INTO` clause](https://www.cockroachlabs.com/docs/v21.1/interleave-in-parent). When this variable is set to `true` and an `ALTER PRIMARY KEY` is run that only removes an `INTERLEAVE INTO` clause, the [partitioning](https://www.cockroachlabs.com/docs/v21.1/partitioning) and [zone configuration](https://www.cockroachlabs.com/docs/v21.1/configure-zone) which applied to the root of the interleave will be applied to the new primary index. The default value for `enable_copying_partitioning_when_deinterleaving_table` is equal to the value set for the new cluster setting `sql.defaults.copy_partitioning_when_deinterleaving_table.enabled`. [#68114][#68114] {% comment %}doc{% endcomment %} - -

Operational changes

- -- [Histogram metrics](https://www.cockroachlabs.com/docs/v21.1/cost-based-optimizer#control-histogram-collection) now store the total number of observations over time. [#68106][#68106] {% comment %}doc{% endcomment %} - -

DB Console changes

- -- Fixed a bug causing the Summary Panel on the [Overview Dashboard](https://www.cockroachlabs.com/docs/v21.1/ui-overview-dashboard) to flicker. [#67365][#67365] -- Fixed a bug preventing a redirect to the originally-requested DB Console page after user login. [#67858][#67858] -- User can now see time series metrics for disk spilling on the [Advanced Debug page](https://www.cockroachlabs.com/docs/v21.1/ui-debug-pages). [#68112][#68112] {% comment %}doc{% endcomment %} -- Fixed color mismatch of node status badge on the [Cluster Overview page](https://www.cockroachlabs.com/docs/v21.1/ui-cluster-overview-page). [#68056][#68056] -- Chart titles on the [Replication Dashboard](https://www.cockroachlabs.com/docs/v21.1/ui-replication-dashboard) were previously falsely labeled as "per Store" but were in fact "per Node". This bug is now fixed. [#67847][#67847] {% comment %}doc{% endcomment %} - -

Bug fixes

- -- Fixed a bug causing the `ST_GeneratePoints` [built-in function](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators) to return a garbage value or an error if an empty geometry or negative nPoints input is given. [#67580][#67580] -- Fixed a bug where [`DROP DATABASE`](https://www.cockroachlabs.com/docs/v21.1/drop-database) could return errors if the database contained [temporary views](https://www.cockroachlabs.com/docs/v21.1/views#temporary-views) in use in an another session. [#67172][#67172] -- Fixed a [storage-level](https://www.cockroachlabs.com/docs/v21.1/architecture/storage-layer) bug where Pebble would occasionally create excessively large SSTables, causing poor compaction performance and high read-amplification. This was especially likely after a manual offline compaction. [#67610][#67610] -- [Correlated subqueries](https://www.cockroachlabs.com/docs/v21.1/subqueries#correlated-subqueries) that couldn't be decorrelated and that have their own subqueries are now executed correctly when supported. Note that it is an edge case of an edge case, so it's unlikely that users have hit this bug (it was found by the randomized testing). [#67570][#67570] -- Fixed very rare, unexpected "index out of bounds" error from the [vectorized engine](https://www.cockroachlabs.com/docs/v21.1/vectorized-execution) when evaluating a `CASE` operator. [#67779][#67779] -- Catching up [Raft](https://www.cockroachlabs.com/docs/v21.1/architecture/replication-layer#raft) followers on the Raft log is now more efficient in the presence of many large Raft log entries. This helps avoid situations where Raft leaders struggle to retain leadership while catching up their followers. [#67127][#67127] -- Fixed a bug that allowed rows to be inserted into a table with a [`CHECK` constraint](https://www.cockroachlabs.com/docs/v21.1/check) that always evaluated to `false` (e.g., `CHECK (false)`). This bug was present since version 21.1.0. [#67341][#67341] -- Fixed a bug causing [changefeeds](https://www.cockroachlabs.com/docs/v21.1/changefeed-for) to sometimes get stuck. [#67968][#67968] -- Previously, CockroachDB nodes would crash whenever the cluster setting `sql.trace.txn.enable_threshold` was changed to a non-zero value. The bug was introduced in 21.1.0. [#68027][#68027] -- Fixed a deadlock that could occur when many replicas were rapidly queued for removal. [#65859][#65859] -- Fixed two bugs which affected [geospatial queries](https://www.cockroachlabs.com/docs/v21.1/spatial-features) with the `st_distance` function. The first bug caused errors for filters of the form `st_distance(g1, g2, use_spheroid) = 0`. The second could cause incorrect results in some cases; it incorrectly transformed filters of the form `st_distance(g1, g2) = 0` when `g1` and `g2` were geographies to `st_instersects(g1, g2)`. This is not a valid transformation because `st_distance` makes spheroid-based calculations by default while `st_intersects` only makes sphere-based calculations. [#67392][#67392] -- Fixed a bug causing a prepared statement to incorrectly reuse the query plan of a different prepared statement that had similar, but not identical type hints. [#67688][#67688] -- Fixed an issue with statistics estimation in the optimizer that could cause it to over-estimate the number of rows for some expressions and thus choose a sub-optimal plan. This issue could happen when multi-column statistics were used in combination with histograms, the query contained a predicate on two or more columns where the columns were highly correlated, and the selected values were very common according to the histograms. [#67998][#67998] -- Previously, CockroachDB could encounter an internal error or crash when performing a cast of [`NULL`](https://www.cockroachlabs.com/docs/v21.1/null-handling) [`JSON`](https://www.cockroachlabs.com/docs/v21.1/jsonb) value to [Geography or Geometry types](https://www.cockroachlabs.com/docs/v21.1/spatial-data). Now this is fixed. [#67902][#67902] -- [`INSERT`](https://www.cockroachlabs.com/docs/v21.1/insert) and [`UPDATE`](https://www.cockroachlabs.com/docs/v21.1/update) statements which operate on larger rows can now be split into batches using the `sql.mutations.mutation_batch_byte_size` setting. [#67958][#67958] -- A rare bug that could result in a crash while creating a [debug.zip file](https://www.cockroachlabs.com/docs/v21.1/cockroach-debug-zip) has been fixed. The bug was only possible to hit if a debug.zip file was captured during a period of rapid lease movement. [#67728][#67728] -- Previously the [`GRANT`](https://www.cockroachlabs.com/docs/v21.1/grant) and [`REVOKE`](https://www.cockroachlabs.com/docs/v21.1/revoke) commands would incorrectly handle role names. CockroachDB treats role names as case-insensitive, but these commands were incorrectly handling the names. Now, `GRANT` and `REVOKE` normalize the names and are case-insensitive. [#67901][#67901] {% comment %}doc{% endcomment %} - -

Performance improvements

- -- [Vectorized flows](https://www.cockroachlabs.com/docs/v21.1/vectorized-execution) can use less memory when sending and receiving data to the network. [#67609][#67609] -- Range merges are no longer considered if a range has seen significant load over the previous 5 minutes, instead of being considered as long as a range had low load over the previous second. This change improves stability, as load-based splits will no longer rapidly disappear during transient throughput dips. [#65362][#65362] -- A new cluster setting `sql.defaults.optimizer_improve_disjunction_selectivity.enabled` enables more accurate selectivity estimation of query filters with `OR` expressions. This improves query plans in some cases. This cluster setting is disabled by default. [#67730][#67730] {% comment %}doc{% endcomment %} - -

Contributors

- -This release includes 47 merged PRs by 26 authors. - -[#65362]: https://github.com/cockroachdb/cockroach/pull/65362 -[#65859]: https://github.com/cockroachdb/cockroach/pull/65859 -[#66331]: https://github.com/cockroachdb/cockroach/pull/66331 -[#67127]: https://github.com/cockroachdb/cockroach/pull/67127 -[#67172]: https://github.com/cockroachdb/cockroach/pull/67172 -[#67341]: https://github.com/cockroachdb/cockroach/pull/67341 -[#67365]: https://github.com/cockroachdb/cockroach/pull/67365 -[#67392]: https://github.com/cockroachdb/cockroach/pull/67392 -[#67570]: https://github.com/cockroachdb/cockroach/pull/67570 -[#67580]: https://github.com/cockroachdb/cockroach/pull/67580 -[#67609]: https://github.com/cockroachdb/cockroach/pull/67609 -[#67610]: https://github.com/cockroachdb/cockroach/pull/67610 -[#67688]: https://github.com/cockroachdb/cockroach/pull/67688 -[#67728]: https://github.com/cockroachdb/cockroach/pull/67728 -[#67730]: https://github.com/cockroachdb/cockroach/pull/67730 -[#67779]: https://github.com/cockroachdb/cockroach/pull/67779 -[#67810]: https://github.com/cockroachdb/cockroach/pull/67810 -[#67847]: https://github.com/cockroachdb/cockroach/pull/67847 -[#67858]: https://github.com/cockroachdb/cockroach/pull/67858 -[#67901]: https://github.com/cockroachdb/cockroach/pull/67901 -[#67902]: https://github.com/cockroachdb/cockroach/pull/67902 -[#67958]: https://github.com/cockroachdb/cockroach/pull/67958 -[#67967]: https://github.com/cockroachdb/cockroach/pull/67967 -[#67968]: https://github.com/cockroachdb/cockroach/pull/67968 -[#67998]: https://github.com/cockroachdb/cockroach/pull/67998 -[#68027]: https://github.com/cockroachdb/cockroach/pull/68027 -[#68056]: https://github.com/cockroachdb/cockroach/pull/68056 -[#68106]: https://github.com/cockroachdb/cockroach/pull/68106 -[#68112]: https://github.com/cockroachdb/cockroach/pull/68112 -[#68114]: https://github.com/cockroachdb/cockroach/pull/68114 diff --git a/src/current/_includes/releases/v21.1/v21.1.8.md b/src/current/_includes/releases/v21.1/v21.1.8.md deleted file mode 100644 index df7410674b5..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.8.md +++ /dev/null @@ -1,87 +0,0 @@ -## v21.1.8 - -Release Date: August 30, 2021 - -{% include releases/release-downloads-docker-image.md release=include.release advisory_key="a69874" %} - -

Security updates

- -- The [node status retrieval endpoints over HTTP](https://www.cockroachlabs.com/docs/v21.1/monitoring-and-alerting) (`/_status/nodes`, `/_status/nodes/`, and the DB Console `/#/reports/nodes`) have been updated to require the [`admin`](https://www.cockroachlabs.com/docs/v21.1/authorization#admin-role) role from the requesting user. This ensures that operational details such as network addresses and command-line flags do not leak to unprivileged users. [#67068][#67068] {% comment %}doc{% endcomment %} - -

General changes

- -- A recent release removed parts of some queries from the [debugging traces](https://www.cockroachlabs.com/docs/v21.1/show-trace) of those queries. This information (i.e. the execution of some low-level RPCs) has been re-included in the traces. [#68923][#68923] {% comment %}doc{% endcomment %} - -

Enterprise edition changes

- -- The `kafka_sink_config` [changefeed option](https://www.cockroachlabs.com/docs/v21.1/create-changefeed) can now include `RequiredAcks`. [#69015][#69015] {% comment %}doc{% endcomment %} -- Added a new per-node [changefeed](https://www.cockroachlabs.com/docs/v21.1/stream-data-out-of-cockroachdb-using-changefeeds) configuration [`changefeed.node_sink_throttle_config`](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) that can be used to throttle the rate of emission from the memory buffer thus making it possible to limit the emission rate from changefeeds. [#68628][#68628] {% comment %}doc{% endcomment %} - -

SQL language changes

- -- Added a new [`EXPLAIN` flag](https://www.cockroachlabs.com/docs/v21.1/explain), `MEMO`, to be used with [`EXPLAIN (OPT)`](https://www.cockroachlabs.com/docs/v21.1/explain#opt-option). When the `MEMO` flag is passed, a representation of the optimizer memo will be printed along with the best plan. The `MEMO` flag can be used in combination with other flags such as `CATALOG` and `VERBOSE`. For example, `EXPLAIN (OPT, MEMO, VERBOSE)` will print the memo along with verbose output for the best plan. [#67778][#67775] {% comment %}doc{% endcomment %} -- Some queries with [lookup joins](https://www.cockroachlabs.com/docs/v21.1/joins#lookup-joins) and/or top K sorts are now more likely to be executed in a "local" manner with the [`distsql=auto`](https://www.cockroachlabs.com/docs/v21.1/set-vars#parameters) session variable when the newly introduced `sql.distsql.prefer_local_execution.enabled` [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) is set to `true` (`false` is the default). [#68613][#68613] {% comment %}doc{% endcomment %} - -

Operational changes

- -- Introduced the `bulkio.index_backfill.checkpoint_interval` [cluster setting](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) to control the rate at which backfills checkpoint their progress. Useful for controlling the backfill rate on large tables. [#68287][#68287] {% comment %}doc{% endcomment %} - -

Command-line changes

- -- [`cockroach debug zip`](https://www.cockroachlabs.com/docs/v21.1/cockroach-debug-zip) no longer retrieves database and table details into separate files. The schema information is collected by means of `system.descriptors` and `crdb_internal.create_statements`. [#68984][#68984] {% comment %}doc{% endcomment %} - -

DB Console changes

- -- Added a new column picker on the [Statements](https://www.cockroachlabs.com/docs/v21.2/ui-statements-page) and [Statements Details pages](https://www.cockroachlabs.com/docs/v21.1/ui-statements-page#statement-details-page) to select which columns to display in the statements table. Includes a new database column option for database name information, which is hidden by default on the [Statements page](https://www.cockroachlabs.com/docs/v21.2/ui-statements-page). [#68294][#68294] {% comment %}doc{% endcomment %} -- Fixed the [Transactions page](https://www.cockroachlabs.com/docs/v21.2/ui-transactions-page) to show the correct value for implicit transactions. [#68294][#68294] -- [Statements Details](https://www.cockroachlabs.com/docs/v21.1/ui-statements-page#statement-details-page), [Statements](https://www.cockroachlabs.com/docs/v21.2/ui-statements-page), and [Transactions pages](https://www.cockroachlabs.com/docs/v21.2/ui-transactions-page) now display information about the node and region on which a statement was executed. [#68317][#68317] {% comment %}doc{% endcomment %} -- Fixed tooltips behavior on the [Sessions](https://www.cockroachlabs.com/docs/v21.2/ui-sessions-page), [Statements](https://www.cockroachlabs.com/docs/v21.2/ui-statements-page), and [Transactions pages](https://www.cockroachlabs.com/docs/v21.2/ui-transactions-page). [#68474][#68474] {% comment %}doc{% endcomment %} - -

Bug fixes

- -- Fixed missing [foreign key](https://www.cockroachlabs.com/docs/v21.1/foreign-key) checks in some cases when there are multiple checks and the inserted data contains a `NULL` for one of the checks. [#68520][#68520] -- Fixed a bug where using [`ST_segmentize`](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators#spatial-functions) on coordinates that are infinite would error with a crash. [#67848][#67848] -- Fixed the [`COPY CSV`](https://www.cockroachlabs.com/docs/v21.1/copy-from) command so that it handles multiple records separated by newline characters. [#68623][#68623] {% comment %}doc{% endcomment %} -- Fixed a bug that caused incorrect query results when querying tables with multiple [column families](https://www.cockroachlabs.com/docs/v21.1/column-families) and unique secondary [indexes](https://www.cockroachlabs.com/docs/v21.1/indexes). The bug only occurred if 1) [vectorized execution](https://www.cockroachlabs.com/docs/v21.1/vectorized-execution) was enabled for the query, 2) the query scanned a [unique secondary index](https://www.cockroachlabs.com/docs/v21.1/indexes) that contained columns from more than one column family, and 3) the rows fetched by the query contained `NULL` values for some of the indexed columns. This bug was present since version v20.1. [#68238][#68239] -- Fixed a crash in the `/debug/closedts-{sender,receiver}` [advanced debug pages](https://www.cockroachlabs.com/docs/v21.1/ui-debug-pages) if the last message of the closed timestamp side transport buffer was removed before rendering. [#68669][#68669] -- Fixed an issue where terminating a CockroachDB process early in its startup routine might cause it to fail to start again, falsely reporting write-ahead log corruption. [#68897][#68897] -- Fixed a regression in the [optimizer's cost model](https://www.cockroachlabs.com/docs/v21.1/cost-based-optimizer) that could cause it to choose sub-optimal plans when choosing between two non-unique index scans with different numbers of columns per index. [#68991][#68991] -- Fixed a bug where CockroachDB could incorrectly evaluate [`LIKE` expressions](https://www.cockroachlabs.com/docs/v21.1/scalar-expressions#string-pattern-matching) when the pattern contained the escape characters `\` if the expressions were executed via the [vectorized execution engine](https://www.cockroachlabs.com/docs/v21.1/vectorized-execution). [#68353][#68353] -- File logs now respect the [`max-group-size`](https://www.cockroachlabs.com/docs/v21.1/configure-logs) configuration parameter again. This parameter had incorrectly become ignored in v21.1, resulting in arbitrarily large log directories. [#69007][#69007] -- Fixed a bug in [`EXPORT`](https://www.cockroachlabs.com/docs/v21.1/export) where concurrent exports could overwrite each other. [#68392][#68392] - -

Performance improvements

- -- Jobs no longer hold exclusive locks during the duration of their checkpointing transactions which could result in long wait times when trying to run [`SHOW JOBS`](https://www.cockroachlabs.com/docs/v21.1/show-jobs). [#68244][#68244] {% comment %}doc{% endcomment %} -- [Lookup joins](https://www.cockroachlabs.com/docs/v21.1/joins#lookup-joins) on indexes with virtual columns are now considered by the optimizer. This should result in more efficient queries in many cases. Most notably, post-query uniqueness checks for unique indexes on virtual columns in [`REGIONAL BY ROW` tables](https://www.cockroachlabs.com/docs/v21.1/regional-tables#regional-by-row-tables) can now use the unique index rather than perform a full-table scan. [#68423][#68423] {% comment %}doc{% endcomment %} - -

Contributors

- -This release includes 40 merged PRs by 26 authors. - -[#67068]: https://github.com/cockroachdb/cockroach/pull/67068 -[#67746]: https://github.com/cockroachdb/cockroach/pull/67746 -[#67775]: https://github.com/cockroachdb/cockroach/pull/67775 -[#67848]: https://github.com/cockroachdb/cockroach/pull/67848 -[#67883]: https://github.com/cockroachdb/cockroach/pull/67883 -[#68239]: https://github.com/cockroachdb/cockroach/pull/68239 -[#68244]: https://github.com/cockroachdb/cockroach/pull/68244 -[#68287]: https://github.com/cockroachdb/cockroach/pull/68287 -[#68294]: https://github.com/cockroachdb/cockroach/pull/68294 -[#68317]: https://github.com/cockroachdb/cockroach/pull/68317 -[#68353]: https://github.com/cockroachdb/cockroach/pull/68353 -[#68392]: https://github.com/cockroachdb/cockroach/pull/68392 -[#68423]: https://github.com/cockroachdb/cockroach/pull/68423 -[#68474]: https://github.com/cockroachdb/cockroach/pull/68474 -[#68510]: https://github.com/cockroachdb/cockroach/pull/68510 -[#68520]: https://github.com/cockroachdb/cockroach/pull/68520 -[#68613]: https://github.com/cockroachdb/cockroach/pull/68613 -[#68623]: https://github.com/cockroachdb/cockroach/pull/68623 -[#68628]: https://github.com/cockroachdb/cockroach/pull/68628 -[#68669]: https://github.com/cockroachdb/cockroach/pull/68669 -[#68897]: https://github.com/cockroachdb/cockroach/pull/68897 -[#68923]: https://github.com/cockroachdb/cockroach/pull/68923 -[#68984]: https://github.com/cockroachdb/cockroach/pull/68984 -[#68991]: https://github.com/cockroachdb/cockroach/pull/68991 -[#69007]: https://github.com/cockroachdb/cockroach/pull/69007 -[#69015]: https://github.com/cockroachdb/cockroach/pull/69015 diff --git a/src/current/_includes/releases/v21.1/v21.1.9.md b/src/current/_includes/releases/v21.1/v21.1.9.md deleted file mode 100644 index e78bb892dfb..00000000000 --- a/src/current/_includes/releases/v21.1/v21.1.9.md +++ /dev/null @@ -1,81 +0,0 @@ -## v21.1.9 - -Release Date: September 20, 2021 - - - -

SQL language changes

- -- Added the [`bulkio.backup.proxy_file_writes.enabled`](https://www.cockroachlabs.com/docs/v21.1/cluster-settings) cluster setting to opt-in to writing backup files outside the KV storage layer. [#69250][#69250] -- You can now alter the owner of the `crdb_internal_region` type which is created by initiating a [multi-region database](https://www.cockroachlabs.com/docs/v21.1/multiregion-overview). [#69759][#69759] - -

Operational changes

- -- A new cluster setting, [`sql.mutations.max_row_size.log`](https://www.cockroachlabs.com/docs/v21.1/cluster-settings), was added, which controls large row logging. Whenever a row larger than this size is written (or a single column family if multiple column families are in use) a `LargeRow` event is logged to the `SQL_PERF` channel (or a `LargeRowInternal` event is logged to `SQL_INTERNAL_PERF` if the row was added by an internal query). This could occur for `INSERT`, `UPSERT`, `UPDATE`, `CREATE TABLE AS`, `CREATE INDEX`, `ALTER TABLE`, `ALTER INDEX`, `IMPORT`, or `RESTORE` statements. `SELECT`, `DELETE`, `TRUNCATE`, and `DROP` are not affected by this setting. This setting is disabled by default. [#69946][#69946] -- A new cluster setting, [`sql.mutations.max_row_size.err`](https://www.cockroachlabs.com/docs/v21.1/cluster-settings), was added, which limits the size of rows written to the database (or individual column families, if multiple column families are in use). Statements trying to write a row larger than this will fail with a code 54000 (`program_limit_exceeded`) error. Internal queries writing a row larger than this will not fail, but will log a `LargeRowInternal` event to the `SQL_INTERNAL_PERF` channel. This limit is enforced for `INSERT`, `UPSERT`, and `UPDATE` statements. `CREATE TABLE AS`, `CREATE INDEX`, `ALTER TABLE`, `ALTER INDEX`, `IMPORT`, and `RESTORE` will not fail with an error, but will log `LargeRowInternal` events to the `SQL_INTERNAL_PERF` channel. `SELECT`, `DELETE`, `TRUNCATE`, and `DROP` are not affected by this limit. Note that existing rows violating the limit *cannot* be updated, unless the update shrinks the size of the row below the limit, but *can* be selected, deleted, altered, backed-up, and restored. For this reason we recommend using the accompanying setting `sql.mutations.max_row_size.log` in conjunction with `SELECT pg_column_size()` queries to detect and fix any existing large rows before lowering `sql.mutations.max_row_size.err`. This setting is disabled by default. [#69946][#69946] -- New variables `sql.mutations.max_row_size.{log|err}` were renamed to `sql.guardrails.max_row_size_{log|err}` for consistency with other variables and metrics. [#69946][#69946] -- Added four new metrics: `sql.guardrails.max_row_size_log.count`, `sql.guardrails.max_row_size_log.count.internal`, `sql.guardrails.max_row_size_err.count`, and `sql.guardrails.max_row_size_err.count.internal`. These metrics are incremented whenever a large row violates the corresponding `sql.guardrails.max_row_size_{log|err}` limit. [#69946][#69946] - -

DB Console changes

- -- A CES survey link component was added to support being able to get client feedback. [#68517][#68517] - -

Bug fixes

- -- Fixed a bug where running [`IMPORT PGDUMP`](https://www.cockroachlabs.com/docs/v21.1/migrate-from-postgres) with a UDT would result in a null pointer exception. This change makes it fail gracefully. [#69249][#69249] -- Fixed a bug where the `schedules.backup.succeeded` and `schedules.backup.failed` metrics would sometimes not be updated. [#69256][#69256] -- The correct format code will now be returned when using [`COPY FROM .. BINARY`](https://www.cockroachlabs.com/docs/v21.1/copy-from). [#69278][#69278] -- Fixed a bug where [`COPY FROM ... BINARY`](https://www.cockroachlabs.com/docs/v21.1/copy-from) would return an error if the input data was split across different messages. [#69278][#69278] -- Fixed a bug where [`COPY FROM ... CSV`](https://www.cockroachlabs.com/docs/v21.1/copy-from) would require each `CopyData` message to be split at the boundary of a record. The `COPY` protocol allows messages to be split at arbitrary points. [#69278][#69278] -- Fixed a bug where [`COPY FROM ... CSV`](https://www.cockroachlabs.com/docs/v21.1/copy-from) did not correctly handle octal byte escape sequences such as `\011` when using a [`BYTES` column](https://www.cockroachlabs.com/docs/v21.1/bytes). [#69278][#69278] -- Fixed an oversight in the data generator for TPC-H which was causing a smaller number of distinct values to be generated for p_type and p_container in the part table than the spec calls for. [#68710][#68710] -- Fixed a bug that was introduced in [v21.1.5]({% link releases/v21.1.md %}#v21-1-5), which prevented [nodes from being decommissioned](https://www.cockroachlabs.com/docs/v21.1/remove-nodes) in a cluster if the cluster had multiple nodes intermittently miss their liveness heartbeat. [#68552][#68552] -- Fixed a bug introduced in v21.1 where CockroachDB could return an internal error when performing streaming aggregation in some edge cases. [#69181][#69181] -- Fixed a bug that created non-partial unique constraints when a user attempted to create a partial unique constraint in [`ALTER TABLE` statements](https://www.cockroachlabs.com/docs/v21.1/alter-table). [#68745][#68745] -- Fixed a bug where a [`DROP VIEW ... CASCADE`](https://www.cockroachlabs.com/docs/v21.1/drop-view) could incorrectly result in "table ...is already being dropped" errors. [#68618][#68618] -- Fixed a bug introduced in v21.1 where the output of [`SHOW CREATE TABLE`](https://www.cockroachlabs.com/docs/v21.1/show-create) on tables with [hash-sharded indexes](https://www.cockroachlabs.com/docs/v21.1/hash-sharded-indexes) was not round-trippable. Executing the output would not create an identical table. This has been fixed by showing `CHECK` constraints that are automatically created for these indexes in the output of `SHOW CREATE TABLE`. [#69695][#69695] -- Fixed internal or "invalid cast" error in some cases involving cascading [updates](https://www.cockroachlabs.com/docs/v21.1/update). [#69180][#69180] -- Fixed a bug with cardinality estimation in the [optimizer](https://www.cockroachlabs.com/docs/v21.1/cost-based-optimizer) that was introduced in v21.1.0. This bug could cause inaccurate row count estimates in queries involving tables with a large number of null values. As a result, it was possible that the optimizer could choose a suboptimal plan. [#69125][#69125] -- Fixed a bug introduced in v20.2 that caused internal errors with set operations, like [`UNION`](https://www.cockroachlabs.com/docs/v21.1/selection-queries#union-combine-two-queries), and columns with tuple types that contained constant `NULL` values. [#69271][#69271] -- Added backwards compatibility between v21.1.x cluster versions and the [v21.1.8 cluster version]({% link releases/v21.1.md %}#v21-1-8). [#69894][#69894] -- Fixed a bug where table stats collection issued via [`EXPLAIN ANALYZE`](https://www.cockroachlabs.com/docs/v21.1/explain-analyze) statements or via [`CREATE STATISTICS`](https://www.cockroachlabs.com/docs/v21.1/create-statistics) statements without specifying the [`AS OF SYSTEM TIME`](https://www.cockroachlabs.com/docs/v21.1/as-of-system-time) option could run into `flow: memory budget exceeded`. [#69588][#69588] -- Fixed a bug where an internal error or a crash could occur when some [`crdb_internal` built-in functions](https://www.cockroachlabs.com/docs/v21.1/functions-and-operators) took string-like type arguments (e.g. `name`). [#69993][#69993] -- Fixed all broken links to the documentation from the DB Console. [#70117][#70117] -- Previously, when using [`ALTER PRIMARY KEY`](https://www.cockroachlabs.com/docs/v21.1/alter-primary-key) on a [`REGIONAL BY ROW`](https://www.cockroachlabs.com/docs/v21.1/multiregion-overview) table, the copied unique index from the old primary key would not have the correct [zone configurations](https://www.cockroachlabs.com/docs/v21.1/configure-zone) applied. This bug is now fixed. Users who have encountered this bug should re-create the index. [#69681][#69681] - -

Performance improvements

- -- Lookup joins on [partial indexes](https://www.cockroachlabs.com/docs/v21.1/partial-indexes) with [virtual computed columns](https://www.cockroachlabs.com/docs/v21.1/computed-columns) are not considered by the [optimizer](https://www.cockroachlabs.com/docs/v21.1/cost-based-optimizer), resulting in more efficient query plans in some cases. [#69110][#69110] -- Updated the [optimizer](https://www.cockroachlabs.com/docs/v21.1/cost-based-optimizer) cost model so that, all else being equal, the optimizer prefers plans in which [`LIMIT`](https://www.cockroachlabs.com/docs/v21.1/limit-offset) operators are pushed as far down the tree as possible. This can reduce the number of rows that need to be processed by higher operators in the plan tree, improving performance.[#69977][#69977] - -

Contributors

- -This release includes 40 merged PRs by 19 authors. - -[#68509]: https://github.com/cockroachdb/cockroach/pull/68509 -[#68517]: https://github.com/cockroachdb/cockroach/pull/68517 -[#68552]: https://github.com/cockroachdb/cockroach/pull/68552 -[#68618]: https://github.com/cockroachdb/cockroach/pull/68618 -[#68710]: https://github.com/cockroachdb/cockroach/pull/68710 -[#68745]: https://github.com/cockroachdb/cockroach/pull/68745 -[#69110]: https://github.com/cockroachdb/cockroach/pull/69110 -[#69125]: https://github.com/cockroachdb/cockroach/pull/69125 -[#69180]: https://github.com/cockroachdb/cockroach/pull/69180 -[#69181]: https://github.com/cockroachdb/cockroach/pull/69181 -[#69249]: https://github.com/cockroachdb/cockroach/pull/69249 -[#69250]: https://github.com/cockroachdb/cockroach/pull/69250 -[#69256]: https://github.com/cockroachdb/cockroach/pull/69256 -[#69271]: https://github.com/cockroachdb/cockroach/pull/69271 -[#69278]: https://github.com/cockroachdb/cockroach/pull/69278 -[#69305]: https://github.com/cockroachdb/cockroach/pull/69305 -[#69588]: https://github.com/cockroachdb/cockroach/pull/69588 -[#69695]: https://github.com/cockroachdb/cockroach/pull/69695 -[#69759]: https://github.com/cockroachdb/cockroach/pull/69759 -[#69894]: https://github.com/cockroachdb/cockroach/pull/69894 -[#69946]: https://github.com/cockroachdb/cockroach/pull/69946 -[#69977]: https://github.com/cockroachdb/cockroach/pull/69977 -[#69993]: https://github.com/cockroachdb/cockroach/pull/69993 -[#70117]: https://github.com/cockroachdb/cockroach/pull/70117 -[#69681]: https://github.com/cockroachdb/cockroach/pull/69681 -[5c9ab2a9f]: https://github.com/cockroachdb/cockroach/commit/5c9ab2a9f -[df88282e3]: https://github.com/cockroachdb/cockroach/commit/df88282e3 diff --git a/src/current/_includes/releases/v21.2/v21.2.0.md b/src/current/_includes/releases/v21.2/v21.2.0.md index abd77ac5493..ca3ecf3e9de 100644 --- a/src/current/_includes/releases/v21.2/v21.2.0.md +++ b/src/current/_includes/releases/v21.2/v21.2.0.md @@ -91,7 +91,7 @@ Core | **Automatic ballast files** | CockroachDB now automatica Before [upgrading to CockroachDB v21.2](https://www.cockroachlabs.com/docs/v21.2/upgrade-cockroach-version), be sure to review the following backward-incompatible changes and adjust your deployment as necessary. -- Interleaved tables and interleaved indexes have been removed. Before upgrading to v21.2, [convert interleaved tables](https://www.cockroachlabs.com/docs/v21.1/interleave-in-parent#convert-interleaved-tables) and [replace interleaved indexes](https://www.cockroachlabs.com/docs/v21.1/interleave-in-parent#replace-interleaved-indexes). Clusters with interleaved tables and indexes cannot finalize the v21.2 upgrade. +- Interleaved tables and interleaved indexes have been removed. Before upgrading to v21.2, convert interleaved tables and replace interleaved indexes. Clusters with interleaved tables and indexes cannot finalize the v21.2 upgrade. - Previously, CockroachDB only supported the YMD format for parsing timestamps from strings. It now also supports the MDY format to better align with PostgreSQL. A timestamp such as `1-1-18`, which was previously interpreted as `2001-01-18`, will now be interpreted as `2018-01-01`. To continue interpreting the timestamp in the YMD format, the first number can be represented with 4 digits, `2001-1-18`. - The deprecated [cluster setting](https://www.cockroachlabs.com/docs/v21.2/cluster-settings) `cloudstorage.gs.default.key` has been removed, and the behavior of the `AUTH` parameter in Google Cloud Storage [`BACKUP`](https://www.cockroachlabs.com/docs/v21.2/backup) and `IMPORT` URIs has been changed. The default behavior is now that of `AUTH=specified`, which uses the credentials passed in the `CREDENTIALS` parameter, and the previous default behavior of using the node's implicit access (via its machine account or role) now requires explicitly passing `AUTH=implicit`. - Switched types from `TEXT` to `"char"` for compatibility with PostgreSQL in the following columns: `pg_constraint` (`confdeltype`, `confmatchtype`, `confudptype`, `contype`) `pg_operator` (`oprkind`), `pg_prog` (`proargmodes`), `pg_rewrite` (`ev_enabled`, `ev_type`), and `pg_trigger` (`tgenabled`). diff --git a/src/current/_includes/sidebar-data-v21.1.json b/src/current/_includes/sidebar-data-v21.1.json deleted file mode 100644 index ff6d4ba51f6..00000000000 --- a/src/current/_includes/sidebar-data-v21.1.json +++ /dev/null @@ -1,1316 +0,0 @@ -[ - { - "title": "Docs Home", - "is_top_level": true, - "urls": [ - "/" - ] - }, - { - "title": "Quickstart", - "is_top_level": true, - "urls": [ - "/cockroachcloud/quickstart.html", - "/cockroachcloud/quickstart-trial-cluster.html" - ] - }, - {% include_cached sidebar-data-cockroachcloud.json %}, - { - "title": "CockroachDB Self-Hosted", - "is_top_level": true, - "urls": [ - "/${VERSION}/index.html" - ], - "items": [ - { - "title": "Get Started", - "items": [ - { - "title": "Install CockroachDB", - "urls": [ - "/${VERSION}/install-cockroachdb.html", - "/${VERSION}/install-cockroachdb-mac.html", - "/${VERSION}/install-cockroachdb-linux.html", - "/${VERSION}/install-cockroachdb-windows.html" - ] - }, - { - "title": "Start a Local Cluster", - "items": [ - { - "title": "From Binary", - "urls": [ - "/${VERSION}/secure-a-cluster.html", - "/${VERSION}/start-a-local-cluster.html" - ] - }, - { - "title": "In Kubernetes", - "urls": [ - "/${VERSION}/orchestrate-a-local-cluster-with-kubernetes.html", - "/${VERSION}/orchestrate-a-local-cluster-with-kubernetes-insecure.html" - ] - }, - { - "title": "In Docker", - "urls": [ - "/${VERSION}/start-a-local-cluster-in-docker-mac.html", - "/${VERSION}/start-a-local-cluster-in-docker-linux.html", - "/${VERSION}/start-a-local-cluster-in-docker-windows.html" - ] - }, - { - "title": "Simulate a Multi-Region Cluster on localhost", - "urls": [ - "/${VERSION}/simulate-a-multi-region-cluster-on-localhost.html" - ] - } - ] - }, - { - "title": "Learn CockroachDB SQL", - "urls": [ - "/${VERSION}/learn-cockroachdb-sql.html" - ] - }, - { - "title": "Explore Database Features", - "items": [ - { - "title": "Replication & Rebalancing", - "urls": [ - "/${VERSION}/demo-replication-and-rebalancing.html" - ] - }, - { - "title": "Fault Tolerance & Recovery", - "urls": [ - "/${VERSION}/demo-fault-tolerance-and-recovery.html" - ] - }, - { - "title": "Multi-Region Performance", - "urls": [ - "/${VERSION}/demo-low-latency-multi-region-deployment.html" - ] - }, - { - "title": "Serializable Transactions", - "urls": [ - "/${VERSION}/demo-serializable.html" - ] - }, - { - "title": "Spatial Data", - "urls": [ - "/${VERSION}/spatial-tutorial.html" - ] - }, - { - "title": "Cross-Cloud Migration", - "urls": [ - "/${VERSION}/demo-automatic-cloud-migration.html" - ] - }, - { - "title": "Orchestration with Kubernetes", - "urls": [ - "/${VERSION}/orchestrate-a-local-cluster-with-kubernetes.html" - ] - }, - { - "title": "JSON Support", - "urls": [ - "/${VERSION}/demo-json-support.html" - ] - }, - { - "title": "SQL Tuning with EXPLAIN", - "urls": [ - "/${VERSION}/sql-tuning-with-explain.html" - ] - } - ] - } - ] - }, - { - "title": "Develop", - "items": [ - { - "title": "Overview", - "urls": [ - "/${VERSION}/developer-guide-overview.html" - ] - }, - { - "title": "Connect to CockroachDB", - "items": [ - { - "title": "Install a driver or ORM", - "urls": [ - "/${VERSION}/install-client-drivers.html" - ] - }, - { - "title": "Connect to a Cluster", - "urls": [ - "/${VERSION}/connect-to-the-database.html", - "/${VERSION}/connect-to-the-database-cockroachcloud.html" - ] - }, - { - "title": "Use Connection Pools", - "urls": [ - "/${VERSION}/connection-pooling.html" - ] - } - ] - }, - { - "title": "Design a Database Schema", - "items": [ - { - "title": "Overview", - "urls": [ - "/${VERSION}/schema-design-overview.html" - ] - }, - { - "title": "Create a Database", - "urls": [ - "/${VERSION}/schema-design-database.html" - ] - }, - { - "title": "Create a User-defined Schema", - "urls": [ - "/${VERSION}/schema-design-schema.html" - ] - }, - { - "title": "Create a Table", - "urls": [ - "/${VERSION}/schema-design-table.html" - ] - }, - { - "title": "Add Secondary Indexes", - "urls": [ - "/${VERSION}/schema-design-indexes.html" - ] - }, - { - "title": "Update a Database Schema", - "items": [ - { - "title": "Change and Remove Objects", - "urls": [ - "/${VERSION}/schema-design-update.html" - ] - }, - { - "title": "Online Schema Changes", - "urls": [ - "/${VERSION}/online-schema-changes.html" - ] - } - ] - }, - { - "title": "Advanced Schema Design", - "items": [ - { - "title": "Use Computed Columns", - "urls": [ - "/${VERSION}/computed-columns.html" - ] - }, - { - "title": "Group Columns into Families", - "urls": [ - "/${VERSION}/column-families.html" - ] - }, - { - "title": "Index a Subset of Rows", - "urls": [ - "/${VERSION}/partial-indexes.html" - ] - }, - { - "title": "Index Sequential Keys", - "urls": [ - "/${VERSION}/hash-sharded-indexes.html" - ] - }, - { - "title": "Index JSON and Array Data", - "urls": [ - "/${VERSION}/inverted-indexes.html" - ] - }, - { - "title": "Index Spatial Data", - "urls": [ - "/${VERSION}/spatial-indexes.html" - ] - }, - { - "title": "Scale to Multi-region", - "urls": [ - "/${VERSION}/multiregion-scale-application.html" - ] - } - ] - } - ] - }, - { - "title": "Write Data", - "items": [ - { - "title": "Insert Data", - "urls": [ - "/${VERSION}/insert-data.html" - ] - }, - { - "title": "Update Data", - "urls": [ - "/${VERSION}/update-data.html" - ] - }, - { - "title": "Bulk-update Data", - "urls": [ - "/${VERSION}/bulk-update-data.html" - ] - }, - { - "title": "Delete Data", - "urls": [ - "/${VERSION}/delete-data.html" - ] - }, - { - "title": "Bulk-delete Data", - "urls": [ - "/${VERSION}/bulk-delete-data.html" - ] - } - ] - }, - { - "title": "Read Data", - "items": [ - { - "title": "Select Rows of Data", - "urls": [ - "/${VERSION}/query-data.html" - ] - }, - { - "title": "Reuse Query Results", - "items": [ - { - "title": "Reusable Views", - "urls": [ - "/${VERSION}/views.html" - ] - }, - { - "title": "Subqueries", - "urls": [ - "/${VERSION}/subqueries.html" - ] - } - ] - }, - { - "title": "Create Temporary Tables", - "urls": [ - "/${VERSION}/temporary-tables.html" - ] - }, - { - "title": "Paginate Results", - "urls": [ - "/${VERSION}/pagination.html" - ] - }, - { - "title": "Read Historical Data", - "items": [ - { - "title": "AS OF SYSTEM TIME", - "urls": [ - "/${VERSION}/as-of-system-time.html" - ] - }, - { - "title": "Follower Reads", - "urls": [ - "/${VERSION}/follower-reads.html" - ] - } - ] - }, - { - "title": "Query Spatial Data", - "urls": [ - "/${VERSION}/query-spatial-data.html" - ] - } - ] - }, - { - "title": "Transactions", - "urls": [ - "/${VERSION}/transactions.html" - ] - }, - { - "title": "Test Your Application Locally", - "urls": [ - "/${VERSION}/local-testing.html" - ] - }, - { - "title": "Debug Your Application", - "items": - [ - { - "title": "Log Events", - "urls": [ - "/${VERSION}/logging-overview.html" - ] - }, - { - "title": "Monitor CockroachDB Apps in the DB Console", - "urls": [ - "/${VERSION}/ui-overview.html" - ] - }, - { - "title": "Troubleshoot Common Problems", - "urls": [ - "/${VERSION}/error-handling-and-troubleshooting.html" - ] - } - ] - }, - { - "title": "Optimize Performance", - "items": - [ - { - "title": "Overview", - "urls": [ - "/${VERSION}/make-queries-fast.html" - ] - }, - { - "title": "Performance Best Practices", - "urls": [ - "/${VERSION}/performance-best-practices-overview.html" - ] - }, - { - "title": "Use the EXPLAIN statement", - "urls": [ - "/${VERSION}/sql-tuning-with-explain.html" - ] - }, - { - "title": "Performance tuning recipes", - "urls": [ - "/${VERSION}/performance-recipes.html" - ] - } - ] - }, - { - "title": "Example Apps", - "items": [ - { - "title": "Overview", - "urls": [ - "/${VERSION}/example-apps.html" - ] - }, - { - "title": "Simple CRUD", - "items": [ - { - "title": "Go", - "urls": [ - "/${VERSION}/build-a-go-app-with-cockroachdb.html", - "/${VERSION}/build-a-go-app-with-cockroachdb-gorm.html", - "/${VERSION}/build-a-go-app-with-cockroachdb-pq.html", - "/${VERSION}/build-a-go-app-with-cockroachdb-upperdb.html" - ] - }, - { - "title": "Java", - "urls": [ - "/${VERSION}/build-a-java-app-with-cockroachdb.html", - "/${VERSION}/build-a-java-app-with-cockroachdb-hibernate.html", - "/${VERSION}/build-a-java-app-with-cockroachdb-jooq.html", - "/${VERSION}/build-a-spring-app-with-cockroachdb-mybatis.html" - ] - }, - { - "title": "JavaScript/TypeScript (Node.js)", - "urls": [ - "/${VERSION}/build-a-nodejs-app-with-cockroachdb.html", - "/${VERSION}/build-a-nodejs-app-with-cockroachdb-sequelize.html", - "/${VERSION}/build-a-nodejs-app-with-cockroachdb-knexjs.html", - "/${VERSION}/build-a-nodejs-app-with-cockroachdb-prisma.html", - "/${VERSION}/build-a-typescript-app-with-cockroachdb.html" - ] - }, - { - "title": "Python", - "urls": [ - "/${VERSION}/build-a-python-app-with-cockroachdb.html", - "/${VERSION}/build-a-python-app-with-cockroachdb-sqlalchemy.html", - "/${VERSION}/build-a-python-app-with-cockroachdb-django.html" - ] - }, - { - "title": "Ruby", - "urls": [ - "/${VERSION}/build-a-ruby-app-with-cockroachdb.html", - "/${VERSION}/build-a-ruby-app-with-cockroachdb-activerecord.html" - ] - }, - { - "title": "C# (.NET)", - "urls": [ - "/${VERSION}/build-a-csharp-app-with-cockroachdb.html" - ] - }, - { - "title": "Rust", - "urls": [ - "/${VERSION}/build-a-rust-app-with-cockroachdb.html" - ] - } - ] - }, - { - "title": "Roach Data", - "items": [ - { - "title": "Spring Boot with JDBC", - "urls": [ - "/${VERSION}/build-a-spring-app-with-cockroachdb-jdbc.html" - ] - }, - { - "title": "Spring Boot with JPA", - "urls": [ - "/${VERSION}/build-a-spring-app-with-cockroachdb-jpa.html" - ] - } - ] - }, - { - "title": "MovR", - "items": [ - { - "title": "Overview", - "urls": [ - "/${VERSION}/movr.html" - ] - }, - { - "title": "Global Application", - "items": [ - { - "title": "Overview", - "urls": [ - "/${VERSION}/movr-flask-overview.html" - ] - }, - { - "title": "Global Application Use-case", - "urls": [ - "/${VERSION}/movr-flask-use-case.html" - ] - }, - { - "title": "Multi-region Database Schema", - "urls": [ - "/${VERSION}/movr-flask-database.html" - ] - }, - { - "title": "Set up a Development Environment", - "urls": [ - "/${VERSION}/movr-flask-setup.html" - ] - }, - { - "title": "Develop a Global Application", - "urls": [ - "/${VERSION}/movr-flask-application.html" - ] - }, - { - "title": "Deploy a Global Application", - "urls": [ - "/${VERSION}/movr-flask-deployment.html" - ] - } - ] - } - ] - } - ] - }, - { - "title": "Third-Party Database Tools", - "items": [ - { - "title": "Supported by Cockroach Labs", - "urls": [ - "/${VERSION}/third-party-database-tools.html" - ] - }, - { - "title": "Supported by the Community", - "urls": [ - "/${VERSION}/community-tooling.html" - ] - }, - { - "title": "Tutorials", - "items": [ - { - "title": "Schema Migration Tools", - "items": [ - { - "title": "Alembic", - "urls": [ - "/${VERSION}/alembic.html" - ] - }, - { - "title": "Flyway", - "urls": [ - "/${VERSION}/flyway.html" - ] - }, - { - "title": "Liquibase", - "urls": [ - "/${VERSION}/liquibase.html" - ] - } - ] - }, - { - "title": "GUIs & IDEs", - "items": [ - { - "title": "DBeaver GUI", - "urls": [ - "/${VERSION}/dbeaver.html" - ] - }, - { - "title": "IntelliJ IDEA", - "urls": [ - "/${VERSION}/intellij-idea.html" - ] - } - ] - }, - { - "title": "Application Deployment Tools", - "items": [ - { - "title": "Google Cloud Run", - "urls": [ - "/${VERSION}/deploy-app-gcr.html" - ] - }, - { - "title": "AWS Lambda", - "urls": [ - "/${VERSION}/deploy-lambda-function.html" - ] - } - ] - } - ] - } - ] - } - ] - }, - { - "title": "Deploy", - "items": [ - { - "title": "Production Checklist", - "urls": [ - "/${VERSION}/recommended-production-settings.html" - ] - }, - { - "title": "Multi-region Capabilities", - "items": [ - { - "title": "Overview", - "urls": [ - "/${VERSION}/multiregion-overview.html" - ] - }, - { - "title": "Choosing a multi-region configuration", - "urls": [ - "/${VERSION}/choosing-a-multi-region-configuration.html" - ] - }, - { - "title": "When to use ZONE vs. REGION Survival Goals", - "urls": [ - "/${VERSION}/when-to-use-zone-vs-region-survival-goals.html" - ] - }, - { - "title": "When to use REGIONAL vs. GLOBAL tables", - "urls": [ - "/${VERSION}/when-to-use-regional-vs-global-tables.html" - ] - }, - { - "title": "Data Domiciling", - "urls": [ - "/${VERSION}/data-domiciling.html" - ] - }, - { - "title": "Migrate to Multi-region SQL", - "urls": [ - "/${VERSION}/migrate-to-multiregion-sql.html" - ] - }, - { - "title": "Topology Patterns", - "items": [ - { - "title": "Overview", - "urls": [ - "/${VERSION}/topology-patterns.html" - ] - }, - { - "title": "Development", - "urls": [ - "/${VERSION}/topology-development.html" - ] - }, - { - "title": "Basic Production", - "urls": [ - "/${VERSION}/topology-basic-production.html" - ] - }, - { - "title": "Regional Tables", - "urls": [ - "/${VERSION}/regional-tables.html" - ] - }, - { - "title": "Global Tables", - "urls": [ - "/${VERSION}/global-tables.html" - ] - }, - { - "title": "Follower Reads", - "urls": [ - "/${VERSION}/topology-follower-reads.html" - ] - }, - { - "title": "Follow-the-Workload", - "urls": [ - "/${VERSION}/topology-follow-the-workload.html" - ] - } - ] - } - ] - }, - { - "title": "Manual Deployment", - "items": [ - { - "title": "Overview", - "urls": [ - "/${VERSION}/manual-deployment.html" - ] - }, - { - "title": "On-Premises", - "urls": [ - "/${VERSION}/deploy-cockroachdb-on-premises.html", - "/${VERSION}/deploy-cockroachdb-on-premises-insecure.html" - ] - }, - { - "title": "AWS", - "urls": [ - "/${VERSION}/deploy-cockroachdb-on-aws.html", - "/${VERSION}/deploy-cockroachdb-on-aws-insecure.html" - ] - }, - { - "title": "Azure", - "urls": [ - "/${VERSION}/deploy-cockroachdb-on-microsoft-azure.html", - "/${VERSION}/deploy-cockroachdb-on-microsoft-azure-insecure.html" - ] - }, - { - "title": "Digital Ocean", - "urls": [ - "/${VERSION}/deploy-cockroachdb-on-digital-ocean.html", - "/${VERSION}/deploy-cockroachdb-on-digital-ocean-insecure.html" - ] - }, - { - "title": "Google Cloud Platform GCE", - "urls": [ - "/${VERSION}/deploy-cockroachdb-on-google-cloud-platform.html", - "/${VERSION}/deploy-cockroachdb-on-google-cloud-platform-insecure.html" - ] - } - ] - }, - { - "title": "Orchestrated Deployment", - "items": [ - { - "title": "Overview", - "urls": [ - "/${VERSION}/orchestration.html" - ] - }, - { - "title": "Kubernetes Single-Cluster Deployment", - "urls": [ - "/${VERSION}/deploy-cockroachdb-with-kubernetes.html", - "/${VERSION}/deploy-cockroachdb-with-kubernetes-insecure.html" - ] - }, - { - "title": "Operate CockroachDB on Kubernetes", - "urls": [ - "/${VERSION}/operate-cockroachdb-kubernetes.html" - ] - }, - { - "title": "Monitor CockroachDB on Kubernetes", - "urls": [ - "/${VERSION}/monitor-cockroachdb-kubernetes.html" - ] - }, - { - "title": "OpenShift Single-Cluster Deployment", - "urls": [ - "/${VERSION}/deploy-cockroachdb-with-kubernetes-openshift.html" - ] - }, - { - "title": "Kubernetes Multi-Cluster Deployment", - "urls": [ - "/${VERSION}/orchestrate-cockroachdb-with-kubernetes-multi-cluster.html" - ] - }, - { - "title": "Kubernetes Performance Optimization", - "urls": [ - "/${VERSION}/kubernetes-performance.html" - ] - } - ] - }, - { - "title": "Back Up and Restore Data", - "items": [ - { - "title": "Full and Incremental Backups", - "urls": [ - "/${VERSION}/take-full-and-incremental-backups.html" - ] - }, - { - "title": "Backups with Revision History and Point-in-time Restore", - "urls": [ - "/${VERSION}/take-backups-with-revision-history-and-restore-from-a-point-in-time.html" - ] - }, - { - "title": "Encrypted Backup and Restore", - "urls": [ - "/${VERSION}/take-and-restore-encrypted-backups.html" - ] - }, - { - "title": "Locality-aware Backup and Restore", - "urls": [ - "/${VERSION}/take-and-restore-locality-aware-backups.html" - ] - }, - { - "title": "Scheduled Backups", - "urls": [ - "/${VERSION}/manage-a-backup-schedule.html" - ] - } - ] - }, - { - "title": "File Storage for Bulk Operations", - "items": [ - { - "title": "Cloud Storage", - "urls": [ - "/${VERSION}/use-cloud-storage-for-bulk-operations.html" - ] - }, - { - "title": "Userfile Storage", - "urls": [ - "/${VERSION}/use-userfile-for-bulk-operations.html" - ] - }, - { - "title": "Local File Server", - "urls": [ - "/${VERSION}/use-a-local-file-server-for-bulk-operations.html" - ] - } - ] - }, - { - "title": "Performance", - "items": [ - { - "title": "Overview", - "urls": [ - "/${VERSION}/performance.html" - ] - }, - { - "title": "Benchmarking Instructions", - "urls": [ - "/${VERSION}/performance-benchmarking-with-tpcc-local.html", - "/${VERSION}/performance-benchmarking-with-tpcc-small.html", - "/${VERSION}/performance-benchmarking-with-tpcc-medium.html", - "/${VERSION}/performance-benchmarking-with-tpcc-large.html" - ] - }, - { - "title": "Tuning Best Practices", - "urls": [ - "/${VERSION}/performance-best-practices-overview.html" - ] - }, - { - "title": "Performance tuning recipes", - "urls": [ - "/${VERSION}/performance-recipes.html" - ] - }, - { - "title": "Improving statement performance", - "urls": [ - "/${VERSION}/make-queries-fast.html" - ] - }, - { - "title": "Tuning with EXPLAIN", - "urls": [ - "/${VERSION}/sql-tuning-with-explain.html" - ] - } - ] - }, - { - "title": "Security", - "items": [ - { - "title": "Overview", - "urls": [ - "/${VERSION}/security-overview.html" - ] - }, - { - "title": "Authentication", - "urls": [ - "/${VERSION}/authentication.html" - ] - }, - { - "title": "Encryption", - "urls": [ - "/${VERSION}/encryption.html" - ] - }, - { - "title": "Authorization", - "urls": [ - "/${VERSION}/authorization.html" - ] - }, - { - "title": "SQL Audit Logging", - "urls": [ - "/${VERSION}/sql-audit-logging.html" - ] - }, - { - "title": "GSSAPI Authentication", - "urls": [ - "/${VERSION}/gssapi_authentication.html" - ] - }, - { - "title": "Single Sign-on", - "urls": [ - "/${VERSION}/sso.html" - ] - } - ] - }, - { - "title": "Monitoring and Alerting", - "items": [ - { - "title": "Overview", - "urls": [ - "/${VERSION}/monitoring-and-alerting.html" - ] - }, - { - "title": "Enable the Node Map", - "urls": [ - "/${VERSION}/enable-node-map.html" - ] - }, - { - "title": "Use Prometheus and Alertmanager", - "urls": [ - "/${VERSION}/monitor-cockroachdb-with-prometheus.html" - ] - }, - { - "title": "Cluster API", - "urls": [ - "/${VERSION}/cluster-api.html" - ] - }, - { - "title": "Third-Party Monitoring Integrations", - "items": [ - { - "title": "Overview", - "urls": [ - "/${VERSION}/third-party-monitoring-tools.html" - ] - }, - { - "title": "Datadog", - "urls": [ - "/${VERSION}/datadog.html" - ] - }, - { - "title": "Kibana", - "urls": [ - "/${VERSION}/kibana.html" - ] - } - ] - } - ] - }, - { - "title": "Logging", - "items": [ - { - "title": "Overview", - "urls": [ - "/${VERSION}/logging-overview.html" - ] - }, - { - "title": "Configure Logs", - "urls": [ - "/${VERSION}/configure-logs.html" - ] - }, - { - "title": "Logging Use Cases", - "urls": [ - "/${VERSION}/logging-use-cases.html" - ] - } - ] - }, - { - "title": "Cluster Maintenance", - "items": [ - { - "title": "Upgrade to CockroachDB v21.1", - "urls": [ - "/${VERSION}/upgrade-cockroach-version.html" - ] - }, - { - "title": "Online Schema Changes", - "urls": [ - "/${VERSION}/online-schema-changes.html" - ] - }, - { - "title": "Manage Long-Running Queries", - "urls": [ - "/${VERSION}/manage-long-running-queries.html" - ] - }, - { - "title": "Decommission Nodes", - "urls": [ - "/${VERSION}/remove-nodes.html" - ] - }, - { - "title": "Rotate Security Certificates", - "urls": [ - "/${VERSION}/rotate-certificates.html" - ] - }, - { - "title": "Disaster Recovery", - "urls": [ - "/${VERSION}/disaster-recovery.html" - ] - } - ] - }, - { - "title": "Replication Controls", - "urls": [ - "/${VERSION}/configure-replication-zones.html" - ] - }, - { - "title": "Stream Data Out of CockroachDB", - "urls": [ - "/${VERSION}/stream-data-out-of-cockroachdb-using-changefeeds.html" - ] - }, - { - "title": "Enterprise Features", - "items": [ - { - "title": "Overview", - "urls": [ - "/${VERSION}/enterprise-licensing.html" - ] - }, - { - "title": "Table Partitioning", - "urls": [ - "/${VERSION}/partitioning.html" - ] - } - ] - } - ] - }, - { - "title": "Migrate", - "items": [ - { - "title": "Migration Overview", - "urls": [ - "/${VERSION}/migration-overview.html" - ] - }, - { - "title": "Migrate from Oracle", - "urls": [ - "/${VERSION}/migrate-from-oracle.html" - ] - }, - { - "title": "Migrate from Postgres", - "urls": [ - "/${VERSION}/migrate-from-postgres.html" - ] - }, - { - "title": "Migrate from MySQL", - "urls": [ - "/${VERSION}/migrate-from-mysql.html" - ] - }, - { - "title": "Migrate from CSV", - "urls": [ - "/${VERSION}/migrate-from-csv.html" - ] - }, - { - "title": "Migrate from Avro", - "urls": [ - "/${VERSION}/migrate-from-avro.html" - ] - }, - { - "title": "Migrate from Shapefiles", - "urls": [ - "/${VERSION}/migrate-from-shapefiles.html" - ] - }, - { - "title": "Migrate from OpenStreetMap", - "urls": [ - "/${VERSION}/migrate-from-openstreetmap.html" - ] - }, - { - "title": "Migrate from GeoJSON", - "urls": [ - "/${VERSION}/migrate-from-geojson.html" - ] - }, - { - "title": "Migrate from GeoPackage", - "urls": [ - "/${VERSION}/migrate-from-geopackage.html" - ] - }, - { - "title": "Export Spatial Data", - "urls": [ - "/${VERSION}/export-spatial-data.html" - ] - }, - { - "title": "Import Performance Best Practices", - "urls": [ - "/${VERSION}/import-performance-best-practices.html" - ] - } - ] - }, - { - "title": "Troubleshoot", - "items": [ - { - "title": "Overview", - "urls": [ - "/${VERSION}/troubleshooting-overview.html" - ] - }, - { - "title": "Common Errors", - "urls": [ - "/${VERSION}/common-errors.html" - ] - }, - { - "title": "Troubleshoot Cluster Setup", - "urls": [ - "/${VERSION}/cluster-setup-troubleshooting.html" - ] - }, - { - "title": "Troubleshoot Query Behavior", - "urls": [ - "/${VERSION}/query-behavior-troubleshooting.html" - ] - }, - { - "title": "Understand Debug Logs", - "urls": [ - "/${VERSION}/logging-overview.html" - ] - }, - { - "title": "Replication Reports", - "urls": [ - "/${VERSION}/query-replication-reports.html" - ] - }, - { - "title": "Support Resources", - "urls": [ - "/${VERSION}/support-resources.html" - ] - }, - { - "title": "File an Issue", - "urls": [ - "/${VERSION}/file-an-issue.html" - ] - } - ] - }, - { - "title": "FAQs", - "items": [ - { - "title": "Product FAQs", - "urls": [ - "/${VERSION}/frequently-asked-questions.html" - ] - }, - { - "title": "SQL FAQs", - "urls": [ - "/${VERSION}/sql-faqs.html" - ] - }, - { - "title": "Operational FAQs", - "urls": [ - "/${VERSION}/operational-faqs.html" - ] - }, - { - "title": "Availability FAQs", - "urls": [ - "/${VERSION}/multi-active-availability.html" - ] - }, - { - "title": "Licensing FAQs", - "urls": [ - "/${VERSION}/licensing-faqs.html" - ] - }, - { - "title": "CockroachDB in Comparison", - "urls": [ - "/${VERSION}/cockroachdb-in-comparison.html" - ] - } - ] - }, - {% include_cached sidebar-releases.json %} - ] - }, - {% include_cached v21.1/sidebar-data-reference.json %} -] diff --git a/src/current/_includes/v21.2/misc/interleave-deprecation-note.md b/src/current/_includes/v21.2/misc/interleave-deprecation-note.md index bdd983430f4..ab12c33929d 100644 --- a/src/current/_includes/v21.2/misc/interleave-deprecation-note.md +++ b/src/current/_includes/v21.2/misc/interleave-deprecation-note.md @@ -1 +1 @@ -{{site.data.alerts.callout_danger}}Interleaving data was deprecated in v20.2, disabled by default in v21.1, and permanently removed in v21.2. For details, see the [v21.1 interleaving deprecation notice](../v21.1/interleave-in-parent.html#deprecation).{{site.data.alerts.end}} \ No newline at end of file +{{site.data.alerts.callout_danger}}Interleaving data was deprecated in v20.2, disabled by default in v21.1, and permanently removed in v21.2.{{site.data.alerts.end}} \ No newline at end of file diff --git a/src/current/_includes/v21.2/orchestration/kubernetes-upgrade-cluster-helm.md b/src/current/_includes/v21.2/orchestration/kubernetes-upgrade-cluster-helm.md index dc94bdfa191..c89e52c6860 100644 --- a/src/current/_includes/v21.2/orchestration/kubernetes-upgrade-cluster-helm.md +++ b/src/current/_includes/v21.2/orchestration/kubernetes-upgrade-cluster-helm.md @@ -4,7 +4,7 @@ Therefore, in order to upgrade to v21.2, you must be on a production release of v21.1. - 1. If you are upgrading to v21.2 from a production release earlier than v21.1, or from a testing release (alpha/beta), first [upgrade to a production release of v21.1](../v21.1/operate-cockroachdb-kubernetes.html?filters=helm#upgrade-the-cluster). Be sure to complete all the steps. + 1. If you are upgrading to v21.2 from a production release earlier than v21.1, or from a testing release (alpha/beta), first upgrade to a production release of v21.1. Be sure to complete all the steps. 1. Then return to this page and perform a second upgrade to v21.2. diff --git a/src/current/_includes/v21.2/orchestration/kubernetes-upgrade-cluster-manual.md b/src/current/_includes/v21.2/orchestration/kubernetes-upgrade-cluster-manual.md index 212443702bb..21e6d17b304 100644 --- a/src/current/_includes/v21.2/orchestration/kubernetes-upgrade-cluster-manual.md +++ b/src/current/_includes/v21.2/orchestration/kubernetes-upgrade-cluster-manual.md @@ -4,7 +4,7 @@ Therefore, in order to upgrade to v21.2, you must be on a production release of v21.1. - 1. If you are upgrading to v21.2 from a production release earlier than v21.1, or from a testing release (alpha/beta), first [upgrade to a production release of v21.1](../v21.1/operate-cockroachdb-kubernetes.html?filters=manual#upgrade-the-cluster). Be sure to complete all the steps. + 1. If you are upgrading to v21.2 from a production release earlier than v21.1, or from a testing release (alpha/beta), first upgrade to a production release of v21.1. Be sure to complete all the steps. 1. Then return to this page and perform a second upgrade to v21.2. diff --git a/src/current/_plugins/sidebar_htmltest.rb b/src/current/_plugins/sidebar_htmltest.rb index 970926334d0..62017bd7728 100644 --- a/src/current/_plugins/sidebar_htmltest.rb +++ b/src/current/_plugins/sidebar_htmltest.rb @@ -1,16 +1,36 @@ require 'json' require 'liquid' +require 'yaml' module SidebarHTMLTest class Generator < Jekyll::Generator def generate(site) @site = site + # Read htmltest configuration to get ignored directories + htmltest_config = YAML.load_file('.htmltest.yml') rescue {} + ignored_dirs = htmltest_config['IgnoreDirs'] || [] + + # Extract version numbers from ignored directories + ignored_versions = ignored_dirs.map do |dir| + match = dir.match(/\^?docs\/?(v\d+\.\d+)/) + match[1] if match + end.compact + Dir[File.join(site.config['includes_dir'], 'sidebar-data-v*.json')].each do |f| next unless !!site.config['cockroachcloud'] == f.include?('cockroachcloud') + + # Extract version from filename + version = f.match(/sidebar-data-(v\d+\.\d+)/)[1] + + # Skip if this version is in the ignored list + if ignored_versions.include?(version) + Jekyll.logger.info "SidebarHTMLTest:", "Skipping ignored version #{version}" + next + end + partial = site.liquid_renderer.file(f).parse(File.read(f)) json = partial.render!(site.site_payload, {registers: {site: site}}) - version = f.match(/sidebar-data-(v\d+\.\d+)/)[1] render_sidebar(json, version) end end diff --git a/src/current/releases/v21.1.md b/src/current/releases/v21.1.md index 6284ada8bef..3ce742c60c5 100644 --- a/src/current/releases/v21.1.md +++ b/src/current/releases/v21.1.md @@ -8,16 +8,38 @@ docs_area: releases keywords: gin, gin index, gin indexes, inverted index, inverted indexes, accelerated index, accelerated indexes --- -{% assign rel = site.data.releases | where_exp: "rel", "rel.major_version == page.major_version" | sort: "release_date" | reverse %} + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + -{% assign vers = site.data.versions | where_exp: "vers", "vers.major_version == page.major_version" | first %} +This release is no longer supported. For more information, see our [Release support policy]({% link releases/release-support-policy.md %}). -{% assign today = "today" | date: "%Y-%m-%d" %} - -{% include releases/testing-release-notice.md major_version=vers %} - -{% include releases/whats-new-intro.md major_version=vers %} - -{% for r in rel %} -{% include releases/{{ page.major_version }}/{{ r.release_name }}.md release=r.release_name %} -{% endfor %} +To download the archived documentation for this release, see [Archived Documentation]({% link releases/archived-documentation.md %}). diff --git a/src/current/v21.1/404.md b/src/current/v21.1/404.md deleted file mode 100644 index b2ac679fe65..00000000000 --- a/src/current/v21.1/404.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -description: "Page not found." -sitemap: false -search: exclude -related_pages: none -toc: false -contribute: false -feedback: false ---- - -
-

Whoops!

- -

We cannot find the page you are looking for. You may have typed the wrong address or found a broken link.

-
diff --git a/src/current/v21.1/add-column.md b/src/current/v21.1/add-column.md deleted file mode 100644 index 56edbc19c62..00000000000 --- a/src/current/v21.1/add-column.md +++ /dev/null @@ -1,333 +0,0 @@ ---- -title: ADD COLUMN -summary: Use the ADD COLUMN statement to add columns to tables. -toc: true ---- - -The `ADD COLUMN` [statement](sql-statements.html) is part of `ALTER TABLE` and adds columns to tables. - -{% include {{ page.version.version }}/sql/combine-alter-table-commands.md %} - -## Synopsis - -
-{% include {{ page.version.version }}/sql/generated/diagrams/add_column.html %} -
- -## Required privileges - -The user must have the `CREATE` [privilege](authorization.html#assign-privileges) on the table. - -## Parameters - - Parameter | Description ------------|------------- - `table_name` | The name of the table to which you want to add the column. - `column_name` | The name of the column you want to add. The column name must follow these [identifier rules](keywords-and-identifiers.html#identifiers) and must be unique within the table but can have the same name as indexes or constraints. - `typename` | The [data type](data-types.html) of the new column. - `col_qualification` | An optional list of column definitions, which may include [column-level constraints](constraints.html), [collation](collate.html), or [column family assignments](column-families.html).

If the column family is not specified, the column will be added to the first column family. For more information about how column families are assigned, see [Column Families](column-families.html#assign-column-families-when-adding-columns). - -## Viewing schema changes - -{% include {{ page.version.version }}/misc/schema-change-view-job.md %} - -## Examples - -### Setup - -The following examples use the [`bank` demo database schema](cockroach-demo.html#datasets). - -To follow along, run [`cockroach demo bank`](cockroach-demo.html) to start a temporary, in-memory cluster with the `bank` schema and dataset preloaded: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach demo bank -~~~ - -### Add a single column - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE bank ADD COLUMN active BOOL; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM bank; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden ---------------+-----------+-------------+----------------+-----------------------+-----------+------------ - id | INT8 | false | NULL | | {primary} | false - balance | INT8 | true | NULL | | {} | false - payload | STRING | true | NULL | | {} | false - active | BOOL | true | NULL | | {} | false -(4 rows) -~~~ - -### Add multiple columns - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE bank ADD COLUMN location STRING, ADD COLUMN currency STRING; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM bank; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden ---------------+-----------+-------------+----------------+-----------------------+-----------+------------ - id | INT8 | false | NULL | | {primary} | false - balance | INT8 | true | NULL | | {} | false - payload | STRING | true | NULL | | {} | false - active | BOOL | true | NULL | | {} | false - location | STRING | true | NULL | | {} | false - currency | STRING | true | NULL | | {} | false -(6 rows) -~~~ - -### Add a column with a `NOT NULL` constraint and a `DEFAULT` value - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE bank ADD COLUMN interest DECIMAL NOT NULL DEFAULT (DECIMAL '1.3'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM bank; -~~~ -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden ---------------+-----------+-------------+------------------------+-----------------------+-----------+------------ - id | INT8 | false | NULL | | {primary} | false - balance | INT8 | true | NULL | | {} | false - payload | STRING | true | NULL | | {} | false - active | BOOL | true | NULL | | {} | false - location | STRING | true | NULL | | {} | false - currency | STRING | true | NULL | | {} | false - interest | DECIMAL | false | 1.3:::DECIMAL::DECIMAL | | {} | false -(7 rows) -~~~ - -### Add a column with a `UNIQUE` constraint - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE bank ADD COLUMN address STRING UNIQUE; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM bank; -~~~ -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden ---------------+-----------+-------------+------------------------+-----------------------+----------------------------+------------ - id | INT8 | false | NULL | | {primary,bank_address_key} | false - balance | INT8 | true | NULL | | {} | false - payload | STRING | true | NULL | | {} | false - active | BOOL | true | NULL | | {} | false - location | STRING | true | NULL | | {} | false - currency | STRING | true | NULL | | {} | false - interest | DECIMAL | false | 1.3:::DECIMAL::DECIMAL | | {} | false - address | STRING | true | NULL | | {bank_address_key} | false -(8 rows) -~~~ - -### Add a column with a `FOREIGN KEY` constraint - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE customers ( - id INT PRIMARY KEY, - name STRING -); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE bank ADD COLUMN cust_number INT REFERENCES customers(id); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM bank; -~~~ -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden ---------------+-----------+-------------+----------------+-----------------------+----------------------------+------------ - id | INT8 | false | NULL | | {primary,bank_address_key} | false - balance | INT8 | true | NULL | | {} | false - payload | STRING | true | NULL | | {} | false - active | BOOL | true | NULL | | {} | false - location | STRING | true | NULL | | {} | false - currency | STRING | true | NULL | | {} | false - interest | DECIMAL | false | 1.3:::DECIMAL | | {} | false - address | STRING | true | NULL | | {bank_address_key} | false - cust_number | INT8 | true | NULL | | {} | false -(9 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CONSTRAINTS FROM bank; -~~~ -~~~ - table_name | constraint_name | constraint_type | details | validated --------------+------------------------------+-----------------+----------------------------------------------------+------------ - bank | bank_address_key | UNIQUE | UNIQUE (address ASC) | true - bank | fk_cust_number_ref_customers | FOREIGN KEY | FOREIGN KEY (cust_number) REFERENCES customers(id) | true - bank | primary | PRIMARY KEY | PRIMARY KEY (id ASC) | true -(3 rows) -~~~ - -### Add a column with collation - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE bank ADD COLUMN more_names STRING COLLATE en; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM bank; -~~~ -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden ---------------+-------------------+-------------+----------------+-----------------------+----------------------------+------------ - id | INT8 | false | NULL | | {primary,bank_address_key} | false - balance | INT8 | true | NULL | | {} | false - payload | STRING | true | NULL | | {} | false - active | BOOL | true | NULL | | {} | false - location | STRING | true | NULL | | {} | false - currency | STRING | true | NULL | | {} | false - interest | DECIMAL | false | 1.3:::DECIMAL | | {} | false - address | STRING | true | NULL | | {bank_address_key} | false - cust_number | INT8 | true | NULL | | {} | false - more_names | STRING COLLATE en | true | NULL | | {} | false -(10 rows) -~~~ - -### Add a column and assign it to a column family - -#### Add a column and assign it to a new column family - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE bank ADD COLUMN location1 STRING CREATE FAMILY f1; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE bank; -~~~ -~~~ - table_name | create_statement --------------+-------------------------------------------------------------------------------------------------------------------------------------- - bank | CREATE TABLE bank ( - | id INT8 NOT NULL, - | balance INT8 NULL, - | payload STRING NULL, - | active BOOL NULL, - | location STRING NULL, - | currency STRING NULL, - | interest DECIMAL NOT NULL DEFAULT 1.3:::DECIMAL, - | address STRING NULL, - | cust_number INT8 NULL, - | more_names STRING COLLATE en NULL, - | location1 STRING NULL, - | CONSTRAINT "primary" PRIMARY KEY (id ASC), - | CONSTRAINT fk_cust_number_ref_customers FOREIGN KEY (cust_number) REFERENCES customers(id), - | UNIQUE INDEX bank_address_key (address ASC), - | FAMILY fam_0_id_balance_payload (id, balance, payload, active, location, currency, interest, address, cust_number, more_names), - | FAMILY f1 (location1) - | ) -(1 row) -~~~ - -#### Add a column and assign it to an existing column family - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE bank ADD COLUMN location2 STRING FAMILY f1; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE bank; -~~~ -~~~ - table_name | create_statement --------------+-------------------------------------------------------------------------------------------------------------------------------------- - bank | CREATE TABLE bank ( - | id INT8 NOT NULL, - | balance INT8 NULL, - | payload STRING NULL, - | active BOOL NULL, - | location STRING NULL, - | currency STRING NULL, - | interest DECIMAL NOT NULL DEFAULT 1.3:::DECIMAL, - | address STRING NULL, - | cust_number INT8 NULL, - | more_names STRING COLLATE en NULL, - | location1 STRING NULL, - | location2 STRING NULL, - | CONSTRAINT "primary" PRIMARY KEY (id ASC), - | CONSTRAINT fk_cust_number_ref_customers FOREIGN KEY (cust_number) REFERENCES customers(id), - | UNIQUE INDEX bank_address_key (address ASC), - | FAMILY fam_0_id_balance_payload (id, balance, payload, active, location, currency, interest, address, cust_number, more_names), - | FAMILY f1 (location1, location2) - | ) -(1 row) -~~~ - -#### Add a column and create a new column family if column family does not exist - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE bank ADD COLUMN new_name STRING CREATE IF NOT EXISTS FAMILY f2; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE bank; -~~~ -~~~ - table_name | create_statement --------------+-------------------------------------------------------------------------------------------------------------------------------------- - bank | CREATE TABLE bank ( - | id INT8 NOT NULL, - | balance INT8 NULL, - | payload STRING NULL, - | active BOOL NULL, - | location STRING NULL, - | currency STRING NULL, - | interest DECIMAL NOT NULL DEFAULT 1.3:::DECIMAL, - | address STRING NULL, - | cust_number INT8 NULL, - | more_names STRING COLLATE en NULL, - | location1 STRING NULL, - | location2 STRING NULL, - | new_name STRING NULL, - | CONSTRAINT "primary" PRIMARY KEY (id ASC), - | CONSTRAINT fk_cust_number_ref_customers FOREIGN KEY (cust_number) REFERENCES customers(id), - | UNIQUE INDEX bank_address_key (address ASC), - | FAMILY fam_0_id_balance_payload (id, balance, payload, active, location, currency, interest, address, cust_number, more_names), - | FAMILY f1 (location1, location2), - | FAMILY f2 (new_name) - | ) -(1 row) -~~~ - -## See also - -- [`ALTER TABLE`](alter-table.html) -- [Column-level Constraints](constraints.html) -- [Collation](collate.html) -- [Column Families](column-families.html) -- [`SHOW JOBS`](show-jobs.html) diff --git a/src/current/v21.1/add-constraint.md b/src/current/v21.1/add-constraint.md deleted file mode 100644 index f9ac99dada2..00000000000 --- a/src/current/v21.1/add-constraint.md +++ /dev/null @@ -1,567 +0,0 @@ ---- -title: ADD CONSTRAINT -summary: Use the ADD CONSTRAINT statement to add constraints to columns. -toc: true ---- - -The `ADD CONSTRAINT` [statement](sql-statements.html) is part of `ALTER TABLE` and can add the following [constraints](constraints.html) to columns: - -- [`UNIQUE`](#add-the-unique-constraint) -- [`CHECK`](#add-the-check-constraint) -- [`FOREIGN KEY`](#add-the-foreign-key-constraint-with-cascade) - -To add a primary key constraint to a table, you should explicitly define the primary key at [table creation](create-table.html). To replace an existing primary key, you can use `ADD CONSTRAINT ... PRIMARY KEY`. For details, see [Changing primary keys with `ADD CONSTRAINT ... PRIMARY KEY`](#changing-primary-keys-with-add-constraint-primary-key). - -The [`DEFAULT`](default-value.html) and [`NOT NULL`](not-null.html) constraints are managed through [`ALTER COLUMN`](alter-column.html). - -{% include {{ page.version.version }}/sql/combine-alter-table-commands.md %} - -## Synopsis - -
-{% include {{ page.version.version }}/sql/generated/diagrams/add_constraint.html %} -
- -## Required privileges - -The user must have the `CREATE` [privilege](authorization.html#assign-privileges) on the table. - -## Parameters - - Parameter | Description ------------|------------- - `table_name` | The name of the table containing the column you want to constrain. - `constraint_name` | The name of the constraint, which must be unique to its table and follow these [identifier rules](keywords-and-identifiers.html#identifiers). - `constraint_elem` | The [`CHECK`](check.html), [foreign key](foreign-key.html), [`UNIQUE`](unique.html) constraint you want to add.

Adding/changing a `DEFAULT` constraint is done through [`ALTER COLUMN`](alter-column.html).

Adding/changing the table's `PRIMARY KEY` is not supported through `ALTER TABLE`; it can only be specified during [table creation](create-table.html). - -## Viewing schema changes - -{% include {{ page.version.version }}/misc/schema-change-view-job.md %} - -## Changing primary keys with `ADD CONSTRAINT ... PRIMARY KEY` - - When you change a primary key with [`ALTER TABLE ... ALTER PRIMARY KEY`](alter-primary-key.html), the old primary key index becomes a secondary index. The secondary index created by `ALTER PRIMARY KEY` takes up node memory and can slow down write performance to a cluster. If you do not have queries that filter on the primary key that you are replacing, you can use `ADD CONSTRAINT` to replace the old primary index without creating a secondary index. - -`ADD CONSTRAINT ... PRIMARY KEY` can be used to add a primary key to an existing table if one of the following is true: - - - No primary key was explicitly defined at [table creation](create-table.html). In this case, the table is created with a default [primary key on `rowid`](indexes.html#creation). Using `ADD CONSTRAINT ... PRIMARY KEY` drops the default primary key and replaces it with a new primary key. - - A [`DROP CONSTRAINT`](drop-constraint.html) statement precedes the `ADD CONSTRAINT ... PRIMARY KEY` statement, in the same transaction. For an example, see [Drop and add the primary key constraint](#drop-and-add-a-primary-key-constraint) below. - -{{site.data.alerts.callout_info}} -`ALTER TABLE ... ADD PRIMARY KEY` is an alias for `ALTER TABLE ... ADD CONSTRAINT ... PRIMARY KEY`. -{{site.data.alerts.end}} - -## Examples - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### Add the `UNIQUE` constraint - -Adding the [`UNIQUE` constraint](unique.html) requires that all of a column's values be distinct from one another (except for *NULL* values). - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE users ADD CONSTRAINT id_name_unique UNIQUE (id, name); -~~~ - -### Add the `CHECK` constraint - -Adding the [`CHECK` constraint](check.html) requires that all of a column's values evaluate to `TRUE` for a Boolean expression. - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE rides ADD CONSTRAINT check_revenue_positive CHECK (revenue >= 0); -~~~ - -Check constraints can be added to columns that were created earlier in the transaction. For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; -> ALTER TABLE users ADD COLUMN is_owner STRING; -> ALTER TABLE users ADD CONSTRAINT check_is_owner CHECK (is_owner IN ('yes', 'no', 'unknown')); -> COMMIT; -~~~ - -~~~ -BEGIN -ALTER TABLE -ALTER TABLE -COMMIT -~~~ - -{{site.data.alerts.callout_info}} -The entire transaction will be rolled back, including any new columns that were added, in the following cases: - -- If an existing column is found containing values that violate the new constraint. -- If a new column has a default value or is a [computed column](computed-columns.html) that would have contained values that violate the new constraint. -{{site.data.alerts.end}} - -### Add the foreign key constraint with `CASCADE` - -To add a foreign key constraint, use the steps shown below. - -Given two tables, `users` and `vehicles`, without foreign key constraints: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE users; -~~~ - -~~~ - table_name | create_statement --------------+-------------------------------------------------------------- - users | CREATE TABLE users ( - | id UUID NOT NULL, - | city VARCHAR NOT NULL, - | name VARCHAR NULL, - | address VARCHAR NULL, - | credit_card VARCHAR NULL, - | CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - | FAMILY "primary" (id, city, name, address, credit_card) - | ) -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE vehicles; -~~~ - -~~~ - table_name | create_statement --------------+------------------------------------------------------------------------------------------------ - vehicles | CREATE TABLE vehicles ( - | id UUID NOT NULL, - | city VARCHAR NOT NULL, - | type VARCHAR NULL, - | owner_id UUID NULL, - | creation_time TIMESTAMP NULL, - | status VARCHAR NULL, - | current_location VARCHAR NULL, - | ext JSONB NULL, - | CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - | FAMILY "primary" (id, city, type, owner_id, creation_time, status, current_location, ext) - | ) -(1 row) -~~~ - -You can include a [foreign key action](foreign-key.html#foreign-key-actions) to specify what happens when a foreign key is updated or deleted. - -Using `ON DELETE CASCADE` will ensure that when the referenced row is deleted, all dependent objects are also deleted. - -{{site.data.alerts.callout_danger}} -`CASCADE` does not list the objects it drops or updates, so it should be used with caution. -{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE vehicles ADD CONSTRAINT users_fk FOREIGN KEY (city, owner_id) REFERENCES users (city, id) ON DELETE CASCADE; -~~~ - -{{site.data.alerts.callout_info}} - By default, referenced columns must be in the same database as the referencing foreign key column. To enable cross-database foreign key references, set the `sql.cross_db_fks.enabled` [cluster setting](cluster-settings.html) to `true`. -{{site.data.alerts.end}} - -### Drop and add a primary key constraint - -Suppose that you want to add `name` to the composite primary key of the `users` table, [without creating a secondary index of the existing primary key](#changing-primary-keys-with-add-constraint-primary-key). - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE users; -~~~ - -~~~ - table_name | create_statement --------------+-------------------------------------------------------------- - users | CREATE TABLE users ( - | id UUID NOT NULL, - | city VARCHAR NOT NULL, - | name VARCHAR NULL, - | address VARCHAR NULL, - | credit_card VARCHAR NULL, - | CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - | FAMILY "primary" (id, city, name, address, credit_card) - | ) -(1 row) -~~~ - -First, add a [`NOT NULL`](not-null.html) constraint to the `name` column with [`ALTER COLUMN`](alter-column.html). - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE users ALTER COLUMN name SET NOT NULL; -~~~ - -Then, in the same transaction, [`DROP`](drop-constraint.html) the old `"primary"` constraint and `ADD` the new one: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; -> ALTER TABLE users DROP CONSTRAINT "primary"; -> ALTER TABLE users ADD CONSTRAINT "primary" PRIMARY KEY (city, name, id); -> COMMIT; -~~~ - -~~~ -NOTICE: primary key changes are finalized asynchronously; further schema changes on this table may be restricted until the job completes -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE users; -~~~ - -~~~ - table_name | create_statement --------------+--------------------------------------------------------------------- - users | CREATE TABLE users ( - | id UUID NOT NULL, - | city VARCHAR NOT NULL, - | name VARCHAR NOT NULL, - | address VARCHAR NULL, - | credit_card VARCHAR NULL, - | CONSTRAINT "primary" PRIMARY KEY (city ASC, name ASC, id ASC), - | FAMILY "primary" (id, city, name, address, credit_card) - | ) -(1 row) -~~~ - -Using [`ALTER PRIMARY KEY`](alter-primary-key.html) would have created a `UNIQUE` secondary index called `users_city_id_key`. Instead, there is just one index for the primary key constraint. - -### Add a unique index to a `REGIONAL BY ROW` table - -{% include {{page.version.version}}/sql/indexes-regional-by-row.md %} - -This example assumes you have a simulated multi-region database running on your local machine following the steps described in [Low Latency Reads and Writes in a Multi-Region Cluster](demo-low-latency-multi-region-deployment.html). It shows how a `UNIQUE` index is partitioned, but it's similar to how all indexes are partitioned on `REGIONAL BY ROW` tables. - -To show how the automatic partitioning of indexes on `REGIONAL BY ROW` tables works, we will: - -1. [Add a column](add-column.html) to the `users` table in the [MovR dataset](movr.html). -1. Add a [`UNIQUE` constraint](unique.html) to that column. -1. Verify that the index is automatically partitioned for better multi-region performance by using [`SHOW INDEXES`](show-index.html) and [`SHOW PARTITIONS`](show-partitions.html). - -First, add a column and its unique constraint. We'll use `email` since that is something that should be unique per user. - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER TABLE users ADD COLUMN email STRING; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER TABLE users ADD CONSTRAINT user_email_unique UNIQUE (email); -~~~ - -Next, issue the [`SHOW INDEXES`](show-index.html) statement. You will see that [the implicit region column](set-locality.html#set-the-table-locality-to-regional-by-row) that was added when the table [was converted to regional by row](demo-low-latency-multi-region-deployment.html#configure-regional-by-row-tables) is now indexed: - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW INDEXES FROM users; -~~~ - -~~~ - table_name | index_name | non_unique | seq_in_index | column_name | direction | storing | implicit --------------+-------------------+------------+--------------+-------------+-----------+---------+----------- - users | primary | false | 1 | region | ASC | false | false - users | primary | false | 2 | id | ASC | false | false - users | user_email_unique | false | 1 | region | ASC | false | false - users | user_email_unique | false | 2 | email | ASC | false | false - users | user_email_unique | false | 3 | id | ASC | false | true - users | users_city_idx | true | 1 | region | ASC | false | false - users | users_city_idx | true | 2 | city | ASC | false | false - users | users_city_idx | true | 3 | id | ASC | false | true -(8 rows) -~~~ - -Next, issue the [`SHOW PARTITIONS`](show-partitions.html) statement. The output below (which is edited for length) will verify that the unique index was automatically [partitioned](partitioning.html) for you. It shows that the `user_email_unique` index is now partitioned by the database regions `europe-west1`, `us-east1`, and `us-west1`. - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW PARTITIONS FROM TABLE users; -~~~ - -~~~ - database_name | table_name | partition_name | column_names | index_name | partition_value | ... -----------------+------------+----------------+--------------+-------------------------+-----------------+----- - movr | users | europe-west1 | region | users@user_email_unique | ('europe-west1')| ... - movr | users | us-east1 | region | users@user_email_unique | ('us-east1') | ... - movr | users | us-west1 | region | users@user_email_unique | ('us-west1') | ... - ... -~~~ - -To ensure that the uniqueness constraint is enforced properly across regions when rows are inserted, or the `email` column of an existing row is updated, the database needs to do the following additional work when indexes are partitioned as shown above: - -1. Run a one-time-only validation query to ensure that the existing data in the table satisfies the unique constraint. -1. Thereafter, the [optimizer](cost-based-optimizer.html) will automatically add a "uniqueness check" when necessary to any [`INSERT`](insert.html), [`UPDATE`](update.html), or [`UPSERT`](upsert.html) statement affecting the columns in the unique constraint. - -{% include {{page.version.version}}/sql/locality-optimized-search.md %} - -### Using `DEFAULT gen_random_uuid()` in `REGIONAL BY ROW` tables - -To auto-generate unique row identifiers in `REGIONAL BY ROW` tables, use the [`UUID`](uuid.html) column with the `gen_random_uuid()` [function](functions-and-operators.html#id-generation-functions) as the [default value](default-value.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE users ( - id UUID NOT NULL DEFAULT gen_random_uuid(), - city STRING NOT NULL, - name STRING NULL, - address STRING NULL, - credit_card STRING NULL, - CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - FAMILY "primary" (id, city, name, address, credit_card) -); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO users (name, city) VALUES ('Petee', 'new york'), ('Eric', 'seattle'), ('Dan', 'seattle'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM users; -~~~ - -~~~ - id | city | name | address | credit_card -+--------------------------------------+----------+-------+---------+-------------+ - cf8ee4e2-cd74-449a-b6e6-a0fb2017baa4 | new york | Petee | NULL | NULL - 2382564e-702f-42d9-a139-b6df535ae00a | seattle | Eric | NULL | NULL - 7d27e40b-263a-4891-b29b-d59135e55650 | seattle | Dan | NULL | NULL -(3 rows) -~~~ - -{{site.data.alerts.callout_info}} -When using `DEFAULT gen_random_uuid()` on columns in `REGIONAL BY ROW` tables, uniqueness checks on those columns are disabled by default for performance purposes. CockroachDB assumes uniqueness based on the way this column generates [`UUIDs`](uuid.html#create-a-table-with-auto-generated-unique-row-ids). To enable this check, you can modify the `sql.optimizer.uniqueness_checks_for_gen_random_uuid.enabled` [cluster setting](cluster-settings.html). Note that while there is virtually no chance of a [collision](https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions) occurring when enabling this setting, it is not truly zero. -{{site.data.alerts.end}} - -### Using implicit vs. explicit index partitioning in `REGIONAL BY ROW` tables - -In `REGIONAL BY ROW` tables, all indexes are partitioned on the region column (usually called [`crdb_region`](set-locality.html#crdb_region)). - -These indexes can either include or exclude the partitioning key (`crdb_region`) as the first column in the index definition: - -- If `crdb_region` is included in the index definition, a [`UNIQUE` index](unique.html) will enforce uniqueness on the set of columns, just like it would in a non-partitioned table. -- If `crdb_region` is excluded from the index definition, that serves as a signal that CockroachDB should enforce uniqueness on only the columns in the index definition. - -In the latter case, the index alone cannot enforce uniqueness on columns that are not a prefix of the index columns, so any time rows are [inserted](insert.html) or [updated](update.html) in a `REGIONAL BY ROW` table that has an implicitly partitioned `UNIQUE` index, the [optimizer](cost-based-optimizer.html) must add uniqueness checks. - -Whether or not to explicitly include `crdb_region` in the index definition depends on the context: - -- If you only need to enforce uniqueness at the region level, then including `crdb_region` in the `UNIQUE` index definition will enforce these semantics and allow you to get better performance on [`INSERT`](insert.html)s, [`UPDATE`](update.html)s, and [`UPSERT`](upsert.html)s, since there will not be any added latency from uniqueness checks. -- If you need to enforce global uniqueness, you should not include `crdb_region` in the `UNIQUE` (or [`PRIMARY KEY`](primary-key.html)) index definition, and the database will automatically ensure that the constraint is enforced. - -To illustrate the different behavior of explicitly vs. implicitly partitioned indexes, we will perform the following tasks: - -- Create a schema that includes an explicitly partitioned index, and an implicitly partitioned index. -- Check the output of several queries using `EXPLAIN` to show the differences in behavior between the two. - -1. Start [`cockroach demo`](cockroach-demo.html) as follows: - - {% include_cached copy-clipboard.html %} - ~~~ shell - cockroach demo --geo-partitioned-replicas - ~~~ - -1. Create a multi-region database and an `employees` table. There are three indexes in the table, all `UNIQUE` and all partitioned by the `crdb_region` column. The table schema guarantees that both `id` and `email` are globally unique, while `desk_id` is only unique per region. The indexes on `id` and `email` are implicitly partitioned, while the index on `(crdb_region, desk_id)` is explicitly partitioned. `UNIQUE` indexes can only directly enforce uniqueness on all columns in the index, including partitioning columns, so each of these indexes enforce uniqueness for `id`, `email`, and `desk_id` per region, respectively. - - {% include_cached copy-clipboard.html %} - ~~~ sql - CREATE DATABASE multi_region_test_db PRIMARY REGION "europe-west1" REGIONS "us-west1", "us-east1"; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - USE multi_region_test_db; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - CREATE TABLE employee ( - id INT PRIMARY KEY, - email STRING UNIQUE, - desk_id INT, - UNIQUE (crdb_region, desk_id) - ) LOCALITY REGIONAL BY ROW; - ~~~ - -1. In the statement below, we add a new user with the required `id`, `email`, and `desk_id` columns. CockroachDB needs to do additional work to enforce global uniqueness for the `id` and `email` columns, which are implicitly partitioned. This additional work is in the form of "uniqueness checks" that the optimizer adds as part of mutation queries. - - {% include_cached copy-clipboard.html %} - ~~~ sql - EXPLAIN INSERT INTO employee VALUES (1, 'joe@example.com', 1); - ~~~ - - The `EXPLAIN` output below shows that the optimizer has added two `constraint-check` post queries to check the uniqueness of the implicitly partitioned indexes `id` and `email`. There is no check needed for `desk_id` (really `(crdb_region, desk_id)`), since that constraint is automatically enforced by the explicitly partitioned index we added in the [`CREATE TABLE`](create-table.html) statement above. - - ~~~ - info - -------------------------------------------------------------------------------------- - distribution: local - vectorized: true - - • root - │ - ├── • insert - │ │ into: employee(id, email, desk_id, crdb_region) - │ │ - │ └── • buffer - │ │ label: buffer 1 - │ │ - │ └── • values - │ size: 5 columns, 1 row - │ - ├── • constraint-check - │ │ - │ └── • error if rows - │ │ - │ └── • lookup join (semi) - │ │ table: employee@primary - │ │ equality: (lookup_join_const_col_@15, column1) = (crdb_region,id) - │ │ equality cols are key - │ │ pred: column10 != crdb_region - │ │ - │ └── • cross join - │ │ estimated row count: 3 - │ │ - │ ├── • values - │ │ size: 1 column, 3 rows - │ │ - │ └── • scan buffer - │ label: buffer 1 - │ - └── • constraint-check - │ - └── • error if rows - │ - └── • lookup join (semi) - │ table: employee@employee_email_key - │ equality: (lookup_join_const_col_@25, column2) = (crdb_region,email) - │ equality cols are key - │ pred: (column1 != id) OR (column10 != crdb_region) - │ - └── • cross join - │ estimated row count: 3 - │ - ├── • values - │ size: 1 column, 3 rows - │ - └── • scan buffer - label: buffer 1 - ~~~ - -1. The statement below updates the user's `email` column. Because the unique index on the `email` column is implicitly partitioned, the optimizer must perform a uniqueness check. - - {% include_cached copy-clipboard.html %} - ~~~ sql - EXPLAIN UPDATE employee SET email = 'joe1@exaple.com' WHERE id = 1; - ~~~ - - In the `EXPLAIN` output below, the optimizer performs a uniqueness check for `email` since we're not updating any other columns (see the `constraint-check` section). - - ~~~ - info - -------------------------------------------------------------------------------------------------------- - distribution: local - vectorized: true - - • root - │ - ├── • update - │ │ table: employee - │ │ set: email - │ │ - │ └── • buffer - │ │ label: buffer 1 - │ │ - │ └── • render - │ │ estimated row count: 1 - │ │ - │ └── • union all - │ │ estimated row count: 1 - │ │ limit: 1 - │ │ - │ ├── • scan - │ │ estimated row count: 1 (100% of the table; stats collected 1 minute ago) - │ │ table: employee@primary - │ │ spans: [/'us-east1'/1 - /'us-east1'/1] - │ │ - │ └── • scan - │ estimated row count: 1 (100% of the table; stats collected 1 minute ago) - │ table: employee@primary - │ spans: [/'europe-west1'/1 - /'europe-west1'/1] [/'us-west1'/1 - /'us-west1'/1] - │ - └── • constraint-check - │ - └── • error if rows - │ - └── • lookup join (semi) - │ table: employee@employee_email_key - │ equality: (lookup_join_const_col_@18, email_new) = (crdb_region,email) - │ equality cols are key - │ pred: (id != id) OR (crdb_region != crdb_region) - │ - └── • cross join - │ estimated row count: 3 - │ - ├── • values - │ size: 1 column, 3 rows - │ - └── • scan buffer - label: buffer 1 - ~~~ - -1. If we only update the user's `desk_id` as shown below, no uniqueness checks are needed, since the index on that column is explicitly partitioned (it's really `(crdb_region, desk_id)`). - - {% include_cached copy-clipboard.html %} - ~~~ sql - EXPLAIN UPDATE employee SET desk_id = 2 WHERE id = 1; - ~~~ - - Because no uniqueness check is needed, there is no `constraint-check` section in the `EXPLAIN` output. - - ~~~ - info - ------------------------------------------------------------------------------------------------ - distribution: local - vectorized: true - - • update - │ table: employee - │ set: desk_id - │ auto commit - │ - └── • render - │ estimated row count: 1 - │ - └── • union all - │ estimated row count: 1 - │ limit: 1 - │ - ├── • scan - │ estimated row count: 1 (100% of the table; stats collected 2 minutes ago) - │ table: employee@primary - │ spans: [/'us-east1'/1 - /'us-east1'/1] - │ - └── • scan - estimated row count: 1 (100% of the table; stats collected 2 minutes ago) - table: employee@primary - spans: [/'europe-west1'/1 - /'europe-west1'/1] [/'us-west1'/1 - /'us-west1'/1] - ~~~ - -## See also - -- [Constraints](constraints.html) -- [Foreign Key Constraint](foreign-key.html) -- [`SHOW CONSTRAINTS`](show-constraints.html) -- [`RENAME CONSTRAINT`](rename-constraint.html) -- [`DROP CONSTRAINT`](drop-constraint.html) -- [`VALIDATE CONSTRAINT`](validate-constraint.html) -- [`ALTER COLUMN`](alter-column.html) -- [`CREATE TABLE`](create-table.html) -- [`ALTER TABLE`](alter-table.html) -- [`SHOW JOBS`](show-jobs.html) -- ['ALTER PRIMARY KEY'](alter-primary-key.html) diff --git a/src/current/v21.1/add-region.md b/src/current/v21.1/add-region.md deleted file mode 100644 index 345c10b150f..00000000000 --- a/src/current/v21.1/add-region.md +++ /dev/null @@ -1,135 +0,0 @@ ---- -title: ADD REGION -summary: The ADD REGION statement adds a region to a multi-region database. -toc: true ---- - -{% include_cached new-in.html version="v21.1" %} The `ALTER DATABASE .. ADD REGION` [statement](sql-statements.html) adds a [region](multiregion-overview.html#database-regions) to a [multi-region database](multiregion-overview.html). While CockroachDB processes an index modification or changing a table to or from a [`REGIONAL BY ROW` table](multiregion-overview.html#regional-by-row-tables), attempting to drop a region from the database containing that `REGIONAL BY ROW` table will produce an error. Similarly, while this statement is running, all index modifications and locality changes on [`REGIONAL BY ROW`](multiregion-overview.html#regional-by-row-tables) tables will be blocked. - -{% include enterprise-feature.md %} - -{{site.data.alerts.callout_info}} -`ADD REGION` is a subcommand of [`ALTER DATABASE`](alter-database.html). -{{site.data.alerts.end}} - -{{site.data.alerts.callout_danger}} -In order to add a region with `ADD REGION`, you must first set a primary database region with [`SET PRIMARY REGION`](set-primary-region.html), or at [database creation](create-database.html).
For an example showing how to add a primary region with `ALTER DATABASE`, see [Set the primary region](#set-the-primary-region). -{{site.data.alerts.end}} - -## Synopsis - -
-{% include {{ page.version.version }}/sql/generated/diagrams/alter_database_add_region.html %} -
- -## Parameters - -| Parameter | Description | -|-----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `database_name` | The database to which you are adding a [region](multiregion-overview.html#database-regions). | -| `region_name` | The [region](multiregion-overview.html#database-regions) being added to this database. Allowed values include any region present in `SHOW REGIONS FROM CLUSTER`. | - -## Required privileges - -To add a region to a database, the user must have one of the following: - -- Membership to the [`admin`](authorization.html#roles) role for the cluster. -- Membership to the [owner](authorization.html#object-ownership) role, or the [`CREATE` privilege](authorization.html#supported-privileges), for the database and all [`REGIONAL BY ROW`](multiregion-overview.html#regional-by-row-tables) tables in the database. - -## Examples - -{% include {{page.version.version}}/sql/multiregion-example-setup.md %} - -### Set the primary region - -Suppose you have a database `foo` in your cluster, and you want to make it a multi-region database. - -To add the first region to the database, or to set an already-added region as the primary region, use a [`SET PRIMARY REGION`](set-primary-region.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE foo SET PRIMARY REGION "us-east1"; -~~~ - -~~~ -ALTER DATABASE PRIMARY REGION -~~~ - -Given a cluster with multiple regions, any databases in that cluster that have not yet had their primary regions set will have their replicas spread as broadly as possible for resiliency. When a primary region is added to one of these databases: - -- All tables will be [`REGIONAL BY TABLE`](set-locality.html#regional-by-table) in the primary region by default. -- This means that all such tables will have all of their voting replicas and leaseholders moved to the primary region. This process is known as [rebalancing](architecture/replication-layer.html#leaseholder-rebalancing). - -### Add regions to a database - -To add more regions to a database that already has at least one region, use an `ADD REGION` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER database foo ADD region "us-west1"; -~~~ - -~~~ -ALTER DATABASE ADD REGION -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER database foo ADD region "europe-west1"; -~~~ - -~~~ -ALTER DATABASE ADD REGION -~~~ - -### View a database's regions - -To view the regions associated with a multi-region database, use a [`SHOW REGIONS FROM DATABASE`](show-regions.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW REGIONS FROM DATABASE foo; -~~~ - -~~~ - database | region | primary | zones ------------+--------------+---------+---------- - foo | us-east1 | true | {b,c,d} - foo | europe-west1 | false | {b,c,d} - foo | us-west1 | false | {a,b,c} -(3 rows) -~~~ - -### Drop a region from a database - -To [drop a region](drop-region.html) from a multi-region database, use a [`DROP REGION`](drop-region.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE foo DROP REGION "us-west1"; -~~~ - -~~~ -ALTER DATABASE DROP REGION -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW REGIONS FROM DATABASE foo; -~~~ - -~~~ - database | region | primary | zones ------------+----------+---------+---------- - foo | us-east1 | true | {b,c,d} -(1 row) -~~~ - -## See also - -- [Multi-region overview](multiregion-overview.html) -- [`SET PRIMARY REGION`](set-primary-region.html) -- [`DROP REGION`](drop-region.html) -- [`SHOW REGIONS`](show-regions.html) -- [`ALTER TABLE`](alter-table.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/advanced-client-side-transaction-retries.md b/src/current/v21.1/advanced-client-side-transaction-retries.md deleted file mode 100644 index 7e4d01e672e..00000000000 --- a/src/current/v21.1/advanced-client-side-transaction-retries.md +++ /dev/null @@ -1,80 +0,0 @@ ---- -title: Advanced Client-side Transaction Retries -summary: Advanced client-side transaction retry features for library authors -toc: true ---- - -This page has instructions for authors of [database drivers and ORMs](install-client-drivers.html) who would like to implement client-side retries in their database driver or ORM for maximum efficiency and ease of use by application developers. - -{{site.data.alerts.callout_info}} -If you are an application developer who needs to implement an application-level retry loop, see the [Client-side intervention example](transactions.html#client-side-intervention-example). -{{site.data.alerts.end}} - -## Overview - -To improve the performance of transactions that fail due to [contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention), CockroachDB includes a set of statements (listed below) that let you retry those transactions. Retrying transactions using these statements has the following benefits: - -1. When you use savepoints, you "hold your place in line" between attempts. Without savepoints, you're starting from scratch every time. -2. Transactions increase their priority each time they're retried, increasing the likelihood they will succeed. This has a lesser effect than #1. - -## How transaction retries work - -A retryable transaction goes through the process described below, which maps to the following SQL statements: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; -- #1 -> SAVEPOINT cockroach_restart; -- #2 --- ... various transaction statements ... -- #3 -> RELEASE SAVEPOINT cockroach_restart; -- #5 (Or #4, ROLLBACK, in case of retry error) -> COMMIT; -~~~ - -1. The transaction starts with the [`BEGIN`](begin-transaction.html) statement. - -2. The [`SAVEPOINT`](savepoint.html) statement shown here is a [retry savepoint](#retry-savepoints); that is, it declares the intention to retry the transaction in the case of contention errors. It must be executed after [`BEGIN`](begin-transaction.html), but before the first statement that manipulates a database. Although [nested transactions](savepoint.html#savepoints-for-nested-transactions) are supported in versions of CockroachDB 20.1 and later, a retry savepoint must be the outermost savepoint in a transaction. - -3. The statements in the transaction are executed. - -4. If a statement returns a retry error (identified via the `40001` error code or `"retry transaction"` string at the start of the error message), you can issue the [`ROLLBACK TO SAVEPOINT`](rollback-transaction.html) statement to restart the transaction and increase the transaction's priority. Alternately, the original [`SAVEPOINT`](savepoint.html) statement can be reissued to restart the transaction. - - You must now issue the statements in the transaction again. - - In cases where you do not want the application to retry the transaction, you can issue [`ROLLBACK`](rollback-transaction.html) at this point. Any other statements will be rejected by the server, as is generally the case after an error has been encountered and the transaction has not been closed. - -5. Once the transaction executes all statements without encountering contention errors, execute [`RELEASE SAVEPOINT`](release-savepoint.html) to commit the changes. If this succeeds, all changes made by the transaction become visible to subsequent transactions and are guaranteed to be durable if a crash occurs. - - In some cases, the [`RELEASE SAVEPOINT`](release-savepoint.html) statement itself can fail with a retry error, mainly because transactions in CockroachDB only realize that they need to be restarted when they attempt to commit. If this happens, the retry error is handled as described in step 4. - -## Retry savepoints - -A savepoint defined with the name `cockroach_restart` is a "retry savepoint" and is used to implement advanced client-side transaction retries. A retry savepoint differs from a [savepoint for nested transactions](savepoint.html#savepoints-for-nested-transactions) as follows: - -- It must be the outermost savepoint in the transaction. -- After a successful [`RELEASE`](release-savepoint.html), a retry savepoint does not allow further use of the transaction. The next statement must be a [`COMMIT`](commit-transaction.html). -- It cannot be nested. Issuing `SAVEPOINT cockroach_restart` two times in a row only creates a single savepoint marker (this can be verified with [`SHOW SAVEPOINT STATUS`](show-savepoint-status.html)). Issuing `SAVEPOINT cockroach_restart` after `ROLLBACK TO SAVEPOINT cockroach_restart` reuses the marker instead of creating a new one. - -Note that you can [customize the retry savepoint name](#customizing-the-retry-savepoint-name) to something other than `cockroach_restart` with a session variable if you need to. - -## Customizing the retry savepoint name - -{% include {{ page.version.version }}/misc/customizing-the-savepoint-name.md %} - -## Examples - -For examples showing how to use [`SAVEPOINT`](savepoint.html) and the other statements described on this page to implement library support for a programming language, see the following: - -- [Build a Java app with CockroachDB](build-a-java-app-with-cockroachdb.html), in particular the logic in the `runSQL` method. -- The source code of the [sqlalchemy-cockroachdb](https://github.com/cockroachdb/sqlalchemy-cockroachdb) adapter for SQLAlchemy. - -## See also - -- [Transactions](transactions.html) -- [`BEGIN`](begin-transaction.html) -- [`COMMIT`](commit-transaction.html) -- [`ROLLBACK`](rollback-transaction.html) -- [`SAVEPOINT`](savepoint.html) -- [`RELEASE SAVEPOINT`](release-savepoint.html) -- [`SHOW`](show-vars.html) -- [DB Console Transactions Page](ui-transactions-page.html) -- [CockroachDB Architecture: Transaction Layer](architecture/transaction-layer.html) \ No newline at end of file diff --git a/src/current/v21.1/alembic.md b/src/current/v21.1/alembic.md deleted file mode 100644 index abc792fde53..00000000000 --- a/src/current/v21.1/alembic.md +++ /dev/null @@ -1,574 +0,0 @@ ---- -title: Migrate CockroachDB Schemas with Alembic -summary: Learn how to use Alembic with a CockroachDB cluster. -toc: true ---- - -This page guides you through a series of simple database schema changes using the [Alembic](https://alembic.sqlalchemy.org/en/latest/) schema migration module with a simple Python application built on SQLAlchemy and CockroachDB. - -For a detailed tutorial about using Alembic, see [the Alembic documentation site](https://alembic.sqlalchemy.org/en/latest/tutorial.html). - -For information about specific migration tasks, see Alembic's [Cookbook](https://alembic.sqlalchemy.org/en/latest/cookbook.html). - -## Before you begin - -Before you begin the tutorial, [install CockroachDB](install-cockroachdb.html). - -## Step 1. Start a cluster and create a database - -1. Start a [demo cluster](cockroach-demo.html): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach demo --no-example-database - ~~~ - - This command creates a virtual cluster and opens a SQL shell to that cluster. - - {{site.data.alerts.callout_info}} - Leave this terminal window open for the duration of the tutorial. Closing the window will destroy the cluster and erase all data in it. - {{site.data.alerts.end}} - -1. Create the `bank` database: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE DATABASE bank; - ~~~ - -## Step 2. Get the application code - -1. Open a new terminal, and clone the [`example-app-python-sqlalchemy`](https://github.com/cockroachlabs/example-app-python-sqlalchemy) GitHub repository: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ git clone git@github.com:cockroachlabs/example-app-python-sqlalchemy.git - ~~~ - -## Step 3. Install and initialize Alembic - -1. Navigate to the `example-app-python-sqlalchemy` project directory, and run the following commands to create and start a virtual environment: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ python3 -m venv env - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ source env/bin/activate - ~~~ - -1. Install the `alembic`, [`sqlalchemy-cockroachdb`](https://github.com/cockroachdb/sqlalchemy-cockroachdb), and [`psycopg2`](https://github.com/psycopg/psycopg2/) modules to the virtual environment: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ pip install sqlalchemy-cockroachdb psycopg2-binary alembic - ~~~ - - The `sqlalchemy-cockroachdb` and `psycopg2-binary` modules are required to use the CockroachDB adapter that the app uses to run transactions against a CockroachDB cluster. - - `alembic` includes the `sqlalchemy` module, which is a primary dependency of the `example-app-python-sqlalchemy` sample app. The `alembic` install also includes the `alembic` command line tool, which we use throughout the tutorial to manage migrations. - -1. Use the `alembic` command-line tool to initialize Alembic for the project: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ alembic init alembic - ~~~ - - ~~~ - Creating directory /path/example-app-python- - sqlalchemy/alembic ... done - Creating directory /path/example-app-python- - sqlalchemy/alembic/versions ... done - Generating /path/example-app-python- - sqlalchemy/alembic/script.py.mako ... done - Generating /path/example-app-python- - sqlalchemy/alembic/env.py ... done - Generating /path/example-app-python- - sqlalchemy/alembic/README ... done - Generating /path/example-app-python- - sqlalchemy/alembic.ini ... done - Please edit configuration/connection/logging settings in - '/path/example-app-python-sqlalchemy/alembic.ini' before - proceeding. - ~~~ - - This command creates a migrations directory called `alembic`. This directory will contain the files that specify the schema migrations for the app. - - The command also creates a properties file called `alembic.ini` at the top of the project directory. - -1. Open `alembic.ini` and update the `sqlalchemy.url` property to specify the correct connection string to your database: - - For example: - - ~~~ - sqlalchemy.url = cockroachdb://demo:demo72529@127.0.0.1:26257/bank?sslmode=require - ~~~ - - {{site.data.alerts.callout_info}} - You must use the `cockroachdb://` prefix in the connection string for SQLAlchemy to make sure the CockroachDB dialect is used. Using the `postgresql://` URL prefix to connect to your CockroachDB cluster will not work. - {{site.data.alerts.end}} - -## Step 4. Create and run a migration script - -1. Use the `alembic` command-line tool to create the first migration script: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ alembic revision -m "create accounts table" - ~~~ - - ~~~ - Generating /path/example-app-python-sqlalchemy/alembic/versions/ad72c7ec8b22_create_accounts_table.py ... done - ~~~ - -1. Open the newly-created migration file (`alembic/versions/ad72c7ec8b22_create_accounts_table.py`, in this case), and edit the `upgrade()` and `downgrade()` functions to read as follows: - - {% include_cached copy-clipboard.html %} - ~~~ python - def upgrade(): - op.create_table( - 'accounts', - sa.Column('id', sa.dialects.postgresql.UUID, primary_key=True), - sa.Column('balance', sa.Integer), - ) - - - def downgrade(): - op.drop_table('accounts') - ~~~ - - Running this migration creates the `accounts` table, with an `id` column and a `balance` column. - - Note that this file also specifies an operation for "downgrading" the migration. In this case, downgrading will drop the `accounts` table, effectively reversing the schema changes of the migration. - -1. Use the `alembic` tool to run this first migration: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ alembic upgrade head - ~~~ - - ~~~ - INFO [alembic.runtime.migration] Context impl CockroachDBImpl. - INFO [alembic.runtime.migration] Will assume non-transactional DDL. - INFO [alembic.runtime.migration] Running upgrade -> ad72c7ec8b22, create accounts table - ~~~ - - Specifying `head` runs the latest migration. This migration will create the `accounts` table. It will also create a table called `alembic_version`, which tracks the current migration version of the database. - -## Step 5. Verify the migration - -1. Open the terminal with the SQL shell to your demo cluster, and verify that the table was successfully created: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > USE bank; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW TABLES; - ~~~ - - ~~~ - schema_name | table_name | type | owner | estimated_row_count | locality - --------------+-----------------+-------+-------+---------------------+----------- - public | accounts | table | demo | 0 | NULL - public | alembic_version | table | demo | 1 | NULL - (2 rows) - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SELECT * FROM alembic_version; - ~~~ - - ~~~ - version_num - ---------------- - ad72c7ec8b22 - (1 row) - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW COLUMNS FROM accounts; - ~~~ - - ~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden - --------------+-----------+-------------+----------------+-----------------------+-----------+------------ - id | UUID | false | NULL | | {primary} | false - balance | INT8 | true | NULL | | {} | false - (2 rows) - ~~~ - -1. In a different terminal, set the `DATABASE_URL` environment variable to the connection string for your cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ export DATABASE_URL=cockroachdb://demo:demo72529@127.0.0.1:26257/bank?sslmode=require - ~~~ - - The sample app reads in `DATABASE_URL` as the connection string to the database. - -1. Run the app to insert, update, and delete rows of data: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ python main.py - ~~~ - - ~~~ - Creating new accounts... - Created new account with id e9b4a9da-fbbb-40de-8c44-60c5c741764d and balance 93911. - Created new account with id 34a6b5d6-0f08-4435-89cb-c7fa30037926 and balance 989744. - ... - Created new account with id 18a7a209-72c3-48b6-986c-2631fff38274 and balance 969474. - Created new account with id 68e73209-fe2e-42db-a54e-c9d101990cdc and balance 382471. - Random account balances: - Account 9acbf774-3e22-4d75-aee0-37e63d3b1ab6: 403963 - Account 82451815-3a87-4d67-a9b0-7766726abd31: 315597 - Transferring 201981 from account 9acbf774-3e22-4d75-aee0-37e63d3b1ab6 to account 82451815-3a87-4d67-a9b0-7766726abd31... - Transfer complete. - New balances: - Account 9acbf774-3e22-4d75-aee0-37e63d3b1ab6: 201982 - Account 82451815-3a87-4d67-a9b0-7766726abd31: 517578 - Deleting existing accounts... - Deleted account 13d1b940-9a7b-47d6-b719-6a2b49a3b08c. - Deleted account 6958f8f9-4d38-424c-bf41-5673f20169b1. - Deleted account c628bd7f-3054-4cd6-b2c9-8c2e3def1720. - Deleted account f4268300-6d0a-4d6e-9489-ad30f215d1ad. - Deleted account feae4e4a-c003-4c29-b672-5422438a885b. - ~~~ - -## Step 6. Add additional migrations - -Suppose you want to add a new [computed column](computed-columns.html) to the `accounts` table that tracks which accounts are overdrawn. - -1. Create a new migration with the `alembic` tool: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ alembic revision -m "add overdrawn column" - ~~~ - - ~~~ - Generating /path/example-app-python-sqlalchemy/alembic/versions/fd88c68af7b5_add_overdrawn_column.py ... done - ~~~ - -1. Open the migration file (`alembic/versions/fd88c68af7b5_add_overdrawn_column.py`), update the imports, and edit the `upgrade()` and `downgrade()` functions: - - ~~~ python - from alembic import op - from sqlalchemy import Column, Boolean, Computed - - ... - - def upgrade(): - op.add_column('accounts', sa.Column('overdrawn', Boolean, Computed('CASE WHEN balance < 0 THEN True ELSE False END'))) - - - def downgrade(): - op.drop_column('accounts', 'overdrawn') - ~~~ - -1. Use the `alembic` tool to run the migration. - - Because this is the latest migration, you can specify `head`, or you can use the migration's ID (`fd88c68af7b5`): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ alembic upgrade fd88c68af7b5 - ~~~ - - ~~~ - INFO [alembic.runtime.migration] Context impl CockroachDBImpl. - INFO [alembic.runtime.migration] Will assume non-transactional DDL. - INFO [alembic.runtime.migration] Running upgrade ad72c7ec8b22 -> fd88c68af7b5, add_overdrawn_column - ~~~ - -1. In the terminal with the SQL shell to your demo cluster, verify that the column was successfully created: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW COLUMNS FROM accounts; - ~~~ - - ~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden - --------------+-----------+-------------+----------------+------------------------------------------------+-----------+------------ - id | UUID | false | NULL | | {primary} | false - balance | INT8 | true | NULL | | {} | false - overdrawn | BOOL | true | NULL | CASE WHEN balance < 0 THEN true ELSE false END | {} | false - (3 rows) - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SELECT * FROM accounts; - ~~~ - - ~~~ - id | balance | overdrawn - ---------------------------------------+---------+------------ - 01894212-7f32-4e4c-b855-146630d928bc | 548554 | false - 033131cf-7c42-4021-9a53-f8a7597ec853 | 828874 | false - 041a2c5d-0bce-4ed4-a91d-a9e3a6e06632 | 768526 | false - 080be3a3-40f8-40c6-a0cc-a61c108db3f5 | 599729 | false - 08503245-ba1a-4255-8ca7-22b3688e69dd | 7962 | false - ... - ~~~ - - The changes will also be reflected in the `alembic_version` table. - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SELECT * FROM alembic_version; - ~~~ - - ~~~ - version_num - ---------------- - fd88c68af7b5 - (1 row) - ~~~ - -## Execute Raw SQL with Alembic - -While [Alembic supports most SQL operations](https://alembic.sqlalchemy.org/en/latest/ops.html), you can always execute raw SQL using the `execute()` operation. - -{{site.data.alerts.callout_success}} -Executing DDL statements as raw SQL can be particularly helpful when using SQL syntax for DDL statements specific to CockroachDB, like [`ALTER TABLE ... ALTER PRIMARY KEY`](alter-primary-key.html) or [`ALTER TABLE ... SET LOCALITY`](set-locality.html) statements. -{{site.data.alerts.end}} - -For example, the raw SQL for the second migration would look something like this: - -~~~ sql -ALTER TABLE accounts ADD COLUMN overdrawn BOOLEAN AS ( - CASE - WHEN balance < 0 THEN True - ELSE False - END -) STORED; -~~~ - -To make the second migration use raw SQL instead of Alembic operations, open `alembic/versions/fd88c68af7b5_add_overdrawn_column.py`, and edit the `upgrade()` function to use `execute()` instead of the operation-specific function: - -~~~ python -def upgrade(): - op.execute(text("""ALTER TABLE accounts ADD COLUMN overdrawn BOOLEAN AS ( - CASE - WHEN balance < 0 THEN True - ELSE False - END - ) STORED;""")) -~~~ - -Before running this migration, downgrade the original migration: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ alembic downgrade -1 -~~~ - -~~~ -INFO [alembic.runtime.migration] Context impl CockroachDBImpl. -INFO [alembic.runtime.migration] Will assume non-transactional DDL. -INFO [alembic.runtime.migration] Running downgrade fd88c68af7b5 -> ad72c7ec8b22, add_overdrawn_column -~~~ - -Then, in the SQL shell to the demo cluster, verify that the `overdrawn` column has been dropped from the table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM accounts; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden ---------------+-----------+-------------+----------------+-----------------------+-----------+------------ - id | UUID | false | NULL | | {primary} | false - balance | INT8 | true | NULL | | {} | false -(2 rows) -~~~ - -Now, run the updated migration script: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ alembic upgrade fd88c68af7b5 -~~~ - -~~~ -INFO [alembic.runtime.migration] Context impl CockroachDBImpl. -INFO [alembic.runtime.migration] Will assume non-transactional DDL. -INFO [alembic.runtime.migration] Running upgrade ad72c7ec8b22 -> fd88c68af7b5, add_overdrawn_column -~~~ - -And verify that the column has been added to the table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM accounts; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden ---------------+-----------+-------------+----------------+------------------------------------------------+-----------+------------ - id | UUID | false | NULL | | {primary} | false - balance | INT8 | true | NULL | | {} | false - overdrawn | BOOL | true | NULL | CASE WHEN balance < 0 THEN true ELSE false END | {} | false -(3 rows) -~~~ - -## Auto-generate a Migration - -Alembic can automatically generate migrations, based on changes to the models in your application source code. - -Let's use the same example `overdrawn` computed column from above. - -First, downgrade the `fd88c68af7b5` migration: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ alembic downgrade -1 -~~~ - -~~~ -INFO [alembic.runtime.migration] Context impl CockroachDBImpl. -INFO [alembic.runtime.migration] Will assume non-transactional DDL. -INFO [alembic.runtime.migration] Running downgrade fd88c68af7b5 -> ad72c7ec8b22, add_overdrawn_column -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM accounts; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden ---------------+-----------+-------------+----------------+-----------------------+-----------+------------ - id | UUID | false | NULL | | {primary} | false - balance | INT8 | true | NULL | | {} | false -(2 rows) -~~~ - -Delete the old migration file: -{% include_cached copy-clipboard.html %} -~~~ shell -rm alembic/versions/fd88c68af7b5_add_overdrawn_column.py -~~~ - -Open the `models.py` file in the app's project, and add the `overdrawn` column to the `Account` class definition: - -~~~ python -from sqlalchemy import Column, Integer, Boolean, Computed - -... - -class Account(Base): - """The Account class corresponds to the "accounts" database table. - """ - __tablename__ = 'accounts' - id = Column(UUID(as_uuid=True), primary_key=True) - balance = Column(Integer) - overdrawn = Column('overdrawn', Boolean, Computed('CASE WHEN balance < 0 THEN True ELSE False END')) -~~~ - -Then, open the `alembic/env.py` file, and add the following import to the top of the file: - -~~~ python -from ..models import Base -~~~ - -And update the variable `target_metadata` to read as follows: - -~~~ python -target_metadata = Base.metadata -~~~ - -These two lines import the database model metadata from the app. - -Use the `alembic` command-line tool to auto-generate the migration from the models defined in the app: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ alembic revision --autogenerate -m "add overdrawn column" -~~~ - -~~~ -INFO [alembic.runtime.migration] Context impl CockroachDBImpl. -INFO [alembic.runtime.migration] Will assume non-transactional DDL. -INFO [alembic.autogenerate.compare] Detected added column 'accounts.overdrawn' - Generating /path/example-app-python-sqlalchemy/alembic/versions/44fa7043e441_add_overdrawn_column.py ... done -~~~ - -Alembic creates a new migration file (`44fa7043e441_add_overdrawn_column.py`, in this case). - -If you open this file, you'll see that it looks very similar to the one you manually created earlier in the tutorial. - -~~~ python -... -def upgrade(): - # ### commands auto generated by Alembic - please adjust! ### - op.add_column('accounts', sa.Column('overdrawn', sa.Boolean(), sa.Computed('CASE WHEN balance < 0 THEN True ELSE False END', ), nullable=True)) - # ### end Alembic commands ### - - -def downgrade(): - # ### commands auto generated by Alembic - please adjust! ### - op.drop_column('accounts', 'overdrawn') - # ### end Alembic commands ### -~~~ - -Run the migration: - - -{% include_cached copy-clipboard.html %} -~~~ shell -$ alembic upgrade 44fa7043e441 -~~~ - -~~~ -INFO [alembic.runtime.migration] Context impl CockroachDBImpl. -INFO [alembic.runtime.migration] Will assume non-transactional DDL. -INFO [alembic.runtime.migration] Running upgrade ad72c7ec8b22 -> 44fa7043e441, add overdrawn column -~~~ - -Verify that the new column exists in the `accounts` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM accounts; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden ---------------+-----------+-------------+----------------+------------------------------------------------+-----------+------------ - id | UUID | false | NULL | | {primary} | false - balance | INT8 | true | NULL | | {} | false - overdrawn | BOOL | true | NULL | CASE WHEN balance < 0 THEN true ELSE false END | {} | false -(3 rows) -~~~ - -## Report Issues with Alembic and CockroachDB - -If you run into problems, please file an issue in the [`alembic` repository](https://github.com/sqlalchemy/alembic/issues), including the following details about the environment where you encountered the issue: - -- CockroachDB version ([`cockroach version`](cockroach-version.html)) -- Alembic version -- Operating system -- Steps to reproduce the behavior - -## See Also - -- [`cockroach demo`](cockroach-demo.html) -- [Alembic documentation](https://alembic.sqlalchemy.org/en/latest/) -- [`alembic` GitHub repository](https://github.com/sqlalchemy/alembic) -- [Client connection parameters](connection-parameters.html) -- [Third-Party Database Tools](third-party-database-tools.html) -- [Learn CockroachDB SQL](learn-cockroachdb-sql.html) diff --git a/src/current/v21.1/alter-column.md b/src/current/v21.1/alter-column.md deleted file mode 100644 index a26fc8315e1..00000000000 --- a/src/current/v21.1/alter-column.md +++ /dev/null @@ -1,290 +0,0 @@ ---- -title: ALTER COLUMN -summary: Use the ALTER COLUMN statement to set, change, or drop a column's DEFAULT constraint or to drop the NOT NULL constraint. -toc: true ---- - -The `ALTER COLUMN` [statement](sql-statements.html) is part of `ALTER TABLE` and can be used to: - -- Set, change, or drop a column's [`DEFAULT` constraint](default-value.html) -- Set or drop a column's [`NOT NULL` constraint](not-null.html) -- Change a column's [data type](data-types.html) - -{{site.data.alerts.callout_info}} -To manage other constraints, see [`ADD CONSTRAINT`](add-constraint.html) and [`DROP CONSTRAINT`](drop-constraint.html). -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -Support for altering column types is [experimental](experimental-features.html), with certain limitations. For details, see [Altering column data types](#altering-column-data-types). -{{site.data.alerts.end}} - -{% include {{ page.version.version }}/sql/combine-alter-table-commands.md %} - -## Synopsis - -
-{% include {{ page.version.version }}/sql/generated/diagrams/alter_column.html %} -
- -## Required privileges - -The user must have the `CREATE` [privilege](authorization.html#assign-privileges) on the table. - -## Parameters - -| Parameter | Description | -|-----------|-------------| -| `table_name` | The name of the table with the column you want to modify. | -| `column_name` | The name of the column you want to modify. | -| `SET DEFAULT a_expr` | The new [Default Value](default-value.html) you want to use. | -| `typename` | The new [data type](data-types.html) you want to use.
Support for altering column types is [experimental](experimental-features.html), with certain limitations. For details, see [Altering column data types](#altering-column-data-types). | -| `USING a_expr` | Specifies how to compute a new column value from the old column value. | - -## Viewing schema changes - -{% include {{ page.version.version }}/misc/schema-change-view-job.md %} - -## Altering column data types - -Support for altering column data types is [experimental](experimental-features.html), with [certain limitations](#limitations-on-altering-data-types). To enable column type altering, set the `enable_experimental_alter_column_type_general` [session variable](set-vars.html) to `true`. - -The following are equivalent in CockroachDB: - -- `ALTER TABLE ... ALTER ... TYPE` -- `ALTER TABLE ... ALTER COLUMN TYPE` -- `ALTER TABLE ... ALTER COLUMN SET DATA TYPE` - -For examples of `ALTER COLUMN TYPE`, [Examples](#convert-to-a-different-data-type). - -### Limitations on altering data types - -You cannot alter the data type of a column if: - -- The column is part of an [index](indexes.html). -- The column has [`CHECK` constraints](check.html). -- The column owns a [sequence](create-sequence.html). -- The `ALTER COLUMN TYPE` statement is part of a [combined `ALTER TABLE` statement](alter-table.html#subcommands). -- The `ALTER COLUMN TYPE` statement is inside an [explicit transaction](begin-transaction.html). - -{{site.data.alerts.callout_info}} -Most `ALTER COLUMN TYPE` changes are finalized asynchronously. Schema changes on the table with the altered column may be restricted, and writes to the altered column may be rejected until the schema change is finalized. -{{site.data.alerts.end}} - -## Examples - -### Set or change a `DEFAULT` value - -Setting the [`DEFAULT` value constraint](default-value.html) inserts the value when data's written to the table without explicitly defining the value for the column. If the column already has a `DEFAULT` value set, you can use this statement to change it. - -The below example inserts the Boolean value `true` whenever you inserted data to the `subscriptions` table without defining a value for the `newsletter` column. - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE subscriptions ALTER COLUMN newsletter SET DEFAULT true; -~~~ - -### Remove `DEFAULT` constraint - -If the column has a defined [`DEFAULT` value](default-value.html), you can remove the constraint, which means the column will no longer insert a value by default if one is not explicitly defined for the column. - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE subscriptions ALTER COLUMN newsletter DROP DEFAULT; -~~~ - -### Set `NOT NULL` constraint - -Setting the [`NOT NULL` constraint](not-null.html) specifies that the column cannot contain `NULL` values. - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE subscriptions ALTER COLUMN newsletter SET NOT NULL; -~~~ - -### Remove `NOT NULL` constraint - -If the column has the [`NOT NULL` constraint](not-null.html) applied to it, you can remove the constraint, which means the column becomes optional and can have *NULL* values written into it. - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE subscriptions ALTER COLUMN newsletter DROP NOT NULL; -~~~ - -### Convert a computed column into a regular column - -{% include {{ page.version.version }}/computed-columns/convert-computed-column.md %} - -### Alter the formula for a computed column - -{% include {{ page.version.version }}/computed-columns/alter-computed-column.md %} - -### Convert to a different data type - -The [TPC-C](performance-benchmarking-with-tpcc-small.html) database has a `customer` table with a column `c_credit_lim` of type [`DECIMAL(10,2)`](decimal.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT column_name, data_type FROM [SHOW COLUMNS FROM customer] WHERE column_name='c_credit_lim'; -~~~ - -~~~ - column_name | data_type ----------------+---------------- - c_credit_lim | DECIMAL(10,2) -(1 row) -~~~ - -Suppose you want to change the data type from `DECIMAL` to [`STRING`](string.html). - -First, set the `enable_experimental_alter_column_type_general` [session variable](set-vars.html) to `true`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET enable_experimental_alter_column_type_general = true; -~~~ - -Then, alter the column type: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE customer ALTER c_credit_lim TYPE STRING; -~~~ - -~~~ -NOTICE: ALTER COLUMN TYPE changes are finalized asynchronously; further schema changes on this table may be restricted until the job completes; some writes to the altered column may be rejected until the schema change is finalized -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT column_name, data_type FROM [SHOW COLUMNS FROM customer] WHERE column_name='c_credit_lim'; -~~~ - -~~~ - column_name | data_type ----------------+------------ - c_credit_lim | STRING -(1 row) -~~~ - - -### Change a column type's precision - -The [TPC-C](performance-benchmarking-with-tpcc-small.html) `customer` table contains a column `c_balance` of type [`DECIMAL(12,2)`](decimal.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT column_name, data_type FROM [SHOW COLUMNS FROM customer] WHERE column_name='c_balance'; -~~~ - -~~~ - column_name | data_type ---------------+---------------- - c_balance | DECIMAL(12,2) -(1 row) -~~~ - -Suppose you want to increase the precision of the `c_balance` column from `DECIMAL(12,2)` to `DECIMAL(14,2)`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE customer ALTER c_balance TYPE DECIMAL(14,2); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT column_name, data_type FROM [SHOW COLUMNS FROM customer] WHERE column_name='c_balance'; -~~~ - -~~~ - column_name | data_type ---------------+---------------- - c_balance | DECIMAL(14,2) -(1 row) -~~~ - -### Change a column's type using an expression - -You can change the data type of a column and create a new, computed value from the old column values, with a [`USING` clause](#parameters). For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT column_name, data_type FROM [SHOW COLUMNS FROM customer] WHERE column_name='c_discount'; -~~~ - -~~~ - column_name | data_type ---------------+--------------- - c_discount | DECIMAL(4,4) -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT c_discount FROM customer LIMIT 10; -~~~ - -~~~ - c_discount --------------- - 0.1569 - 0.4629 - 0.2932 - 0.0518 - 0.3922 - 0.1106 - 0.0622 - 0.4916 - 0.3072 - 0.0316 -(10 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE customer ALTER c_discount TYPE STRING USING ((c_discount*100)::DECIMAL(4,2)::STRING || ' percent'); -~~~ - -~~~ -NOTICE: ALTER COLUMN TYPE changes are finalized asynchronously; further schema changes on this table may be restricted until the job completes; some writes to the altered column may be rejected until the schema change is finalized -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT column_name, data_type FROM [SHOW COLUMNS FROM customer] WHERE column_name='c_discount'; -~~~ - -~~~ - column_name | data_type ---------------+------------ - c_discount | STRING -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT c_discount FROM customer LIMIT 10; -~~~ - -~~~ - c_discount ------------------ - 15.69 percent - 46.29 percent - 29.32 percent - 5.18 percent - 39.22 percent - 11.06 percent - 6.22 percent - 49.16 percent - 30.72 percent - 3.16 percent -(10 rows) -~~~ - -## See also - -- [Constraints](constraints.html) -- [`ADD CONSTRAINT`](add-constraint.html) -- [`DROP CONSTRAINT`](drop-constraint.html) -- [`ALTER TABLE`](alter-table.html) -- [`SHOW JOBS`](show-jobs.html) diff --git a/src/current/v21.1/alter-database.md b/src/current/v21.1/alter-database.md deleted file mode 100644 index 3163cba50e8..00000000000 --- a/src/current/v21.1/alter-database.md +++ /dev/null @@ -1,26 +0,0 @@ ---- -title: ALTER DATABASE -summary: Use the ALTER DATABASE statement to change an existing database. -toc: false ---- - -The `ALTER DATABASE` [statement](sql-statements.html) applies a schema change to a database. For information on using `ALTER DATABASE`, see the pages for its relevant [subcommands](#subcommands). - -{% include {{{ page.version.version }}/misc/schema-change-stmt-note.md %} - -## Subcommands - -Subcommand | Description ------------|------------ -[`CONFIGURE ZONE`](configure-zone.html) | [Configure replication zones](configure-replication-zones.html) for a database. -[`CONVERT TO SCHEMA`](convert-to-schema.html) | Convert a [database](create-database.html) to a [schema](sql-name-resolution.html). -[`OWNER TO`](owner-to.html) | Change the owner of a database. -[`RENAME`](rename-database.html) | Change the name of a database. -[`ADD REGION`](add-region.html) | **New in v21.1:** Add a region to a [multi-region database](multiregion-overview.html). -[`DROP REGION`](drop-region.html) | **New in v21.1:** Drop a region from a [multi-region database](multiregion-overview.html). -[`SET PRIMARY REGION`](set-primary-region.html) | **New in v21.1:** Set the primary region of a [multi-region database](multiregion-overview.html). -[`SURVIVE {ZONE,REGION} FAILURE`](survive-failure.html) | **New in v21.1:** Add a survival goal to a [multi-region database](multiregion-overview.html). - -## Viewing schema changes - -{% include {{ page.version.version }}/misc/schema-change-view-job.md %} diff --git a/src/current/v21.1/alter-index.md b/src/current/v21.1/alter-index.md deleted file mode 100644 index e14615f48c4..00000000000 --- a/src/current/v21.1/alter-index.md +++ /dev/null @@ -1,23 +0,0 @@ ---- -title: ALTER INDEX -summary: Use the ALTER INDEX statement to change an existing index. -toc: true ---- - -The `ALTER INDEX` [statement](sql-statements.html) applies a schema change to an index. For information on using `ALTER INDEX`, see the pages for its relevant [subcommands](#subcommands). - -{% include {{{ page.version.version }}/misc/schema-change-stmt-note.md %} - -## Subcommands - -Subcommand | Description ------------|------------ -[`CONFIGURE ZONE`](configure-zone.html) | [Configure replication zones](configure-replication-zones.html) for an index. -[`PARTITION BY`](partition-by.html) | Partition, re-partition, or un-partition an index. ([Enterprise-only](enterprise-licensing.html)). -[`RENAME`](rename-index.html) | Change the name of an index. -[`SPLIT AT`](split-at.html) | Force a range split at the specified row in the index. -[`UNSPLIT AT`](unsplit-at.html) | Remove a range split enforcement at a specified row in the index. - -## Viewing schema changes - -{% include {{ page.version.version }}/misc/schema-change-view-job.md %} diff --git a/src/current/v21.1/alter-partition.md b/src/current/v21.1/alter-partition.md deleted file mode 100644 index d5b7c8df7d5..00000000000 --- a/src/current/v21.1/alter-partition.md +++ /dev/null @@ -1,41 +0,0 @@ ---- -title: ALTER PARTITION -summary: Use the ALTER PARTITION statement to configure the replication zone for a partition. -toc: true ---- - -The `ALTER PARTITION` [statement](sql-statements.html) is used to configure replication zones for [partitioning](partitioning.html). See the [`CONFIGURE ZONE`](configure-zone.html) subcommand for more details. - -{% include enterprise-feature.md %} - -## Synopsis - -
-{% remote_include https://raw.githubusercontent.com/cockroachdb/generated-diagrams/{{ page.release_info.crdb_branch_name }}/grammar_svg/alter_zone_partition.html %} -
- -## Required privileges - -The user must have the [`CREATE`](grant.html#supported-privileges) privilege on the table. - -## Parameters - - Parameter | Description ------------+------------- -`table_name` | The name of the [table](create-table.html) with the [replication zone configurations](configure-replication-zones.html) to modify. -`partition_name` | The name of the [partition](partitioning.html) with the [replication zone configurations](configure-replication-zones.html) to modify. -`index_name` | The name of the [index](indexes.html) with the [replication zone configurations](configure-replication-zones.html) to modify. -`variable` | The name of the [variable](#variables) to change. -`value` | The value of the variable to change. - -### Variables - -{% include {{ page.version.version }}/zone-configs/variables.md %} - -## Examples - -{% include {{ page.version.version }}/sql/movr-statements-geo-partitioned-replicas.md %} - -### Create a replication zone for a partition - -{% include {{ page.version.version }}/zone-configs/create-a-replication-zone-for-a-table-partition.md hide-enterprise-warning="true" %} diff --git a/src/current/v21.1/alter-primary-key.md b/src/current/v21.1/alter-primary-key.md deleted file mode 100644 index f79791028d4..00000000000 --- a/src/current/v21.1/alter-primary-key.md +++ /dev/null @@ -1,109 +0,0 @@ ---- -title: ALTER PRIMARY KEY -summary: Use the ALTER PRIMARY KEY statement to change the primary key of a table. -toc: true ---- - - The `ALTER PRIMARY KEY` [statement](sql-statements.html) is a subcommand of [`ALTER TABLE`](alter-table.html) that can be used to change the [primary key](primary-key.html) of a table. - -## Watch the demo - -{% include_cached youtube.html video_id="MPx-LXY2D-c" %} - -## Details - -- You cannot change the primary key of a table that is currently undergoing a primary key change, or any other [schema change](online-schema-changes.html). - -- `ALTER PRIMARY KEY` might need to rewrite multiple indexes, which can make it an expensive operation. - -- When you change a primary key with `ALTER PRIMARY KEY`, the old primary key index becomes a [`UNIQUE`](unique.html) secondary index. This helps optimize the performance of queries that still filter on the old primary key column. - -- `ALTER PRIMARY KEY` does not alter the [partitions](partitioning.html) on a table or its indexes, even if a partition is defined on [a column in the original primary key](partitioning.html#partition-using-primary-key). If you alter the primary key of a partitioned table, you must update the table partition accordingly. - -- The secondary index created by `ALTER PRIMARY KEY` will not be partitioned, even if a partition is defined on [a column in the original primary key](partitioning.html#partition-using-primary-key). To ensure that the table is partitioned correctly, you must create a partition on the secondary index, or drop the secondary index. - -- Any new primary key column set by `ALTER PRIMARY KEY` must have an existing [`NOT NULL` constraint](not-null.html). To add a `NOT NULL` constraint to an existing column, use [`ALTER TABLE ... ALTER COLUMN ... SET NOT NULL`](alter-column.html#set-not-null-constraint). - -{{site.data.alerts.callout_success}} -To change an existing primary key without creating a secondary index from that primary key, use [`DROP CONSTRAINT ... PRIMARY KEY`/`ADD CONSTRAINT ... PRIMARY KEY`](add-constraint.html#changing-primary-keys-with-add-constraint-primary-key). For examples, see the [`ADD CONSTRAINT`](add-constraint.html#examples) and [`DROP CONSTRAINT`](drop-constraint.html#examples) pages. -{{site.data.alerts.end}} - -## Synopsis - -
-{% include {{ page.version.version }}/sql/generated/diagrams/alter_primary_key.html %} -
- -## Parameters - - Parameter | Description ------------|------------- - `table_name` | The name of the table with the primary key that you want to modify. - `index_params` | The name of the column(s) that you want to use for the primary key. These columns replace the current primary key column(s). - `USING HASH WITH BUCKET COUNT` | Creates a [hash-sharded index](hash-sharded-indexes.html) with `n_buckets` number of buckets.
{{site.data.alerts.callout_info}}To enable hash-sharded indexes, set the `experimental_enable_hash_sharded_indexes` [session variable](set-vars.html) to `on`.{{site.data.alerts.end}} - `opt_interleave` | [Interleave table into parent object](interleave-in-parent.html).
{% include {{ page.version.version }}/misc/interleave-deprecation-note.md %} - -## Required privileges - -The user must have the `CREATE` [privilege](authorization.html#assign-privileges) on a table to alter its primary key. - -## Viewing schema changes - -{% include {{ page.version.version }}/misc/schema-change-view-job.md %} - -## Examples - -### Alter a single-column primary key - -Suppose that you are storing the data for users of your application in a table called `users`, defined by the following `CREATE TABLE` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE users ( - name STRING PRIMARY KEY, - email STRING -); -~~~ - -The primary key of this table is on the `name` column. This is a poor choice, as some users likely have the same name, and all primary keys enforce a `UNIQUE` constraint on row values of the primary key column. Per our [best practices](performance-best-practices-overview.html#use-uuid-to-generate-unique-ids), you should instead use a `UUID` for single-column primary keys, and populate the rows of the table with generated, unique values. - -You can add a column and change the primary key with a couple of `ALTER TABLE` statements: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE users ADD COLUMN id UUID NOT NULL DEFAULT gen_random_uuid(); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE users ALTER PRIMARY KEY USING COLUMNS (id); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE users; -~~~ - -~~~ - table_name | create_statement --------------+-------------------------------------------------- - users | CREATE TABLE users ( - | name STRING NOT NULL, - | email STRING NULL, - | id UUID NOT NULL DEFAULT gen_random_uuid(), - | CONSTRAINT "primary" PRIMARY KEY (id ASC), - | UNIQUE INDEX users_name_key (name ASC), - | FAMILY "primary" (name, email, id) - | ) -(1 row) -~~~ - -Note that the old primary key index becomes a secondary index, in this case, `users_name_key`. If you do not want the old primary key to become a secondary index when changing a primary key, you can use [`DROP CONSTRAINT`](drop-constraint.html)/[`ADD CONSTRAINT`](add-constraint.html) instead. - -## See also - -- [Constraints](constraints.html) -- [`ADD CONSTRAINT`](add-constraint.html) -- [`DROP CONSTRAINT`](drop-constraint.html) -- [`ALTER TABLE`](alter-table.html) -- [`SHOW JOBS`](show-jobs.html) diff --git a/src/current/v21.1/alter-range.md b/src/current/v21.1/alter-range.md deleted file mode 100644 index 664d8b7f1d0..00000000000 --- a/src/current/v21.1/alter-range.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: ALTER RANGE -summary: Use the ALTER RANGE statement to configure the replication zone for a system range. -toc: true ---- - -The `ALTER RANGE` [statement](sql-statements.html) is used to configure replication zones for system ranges. See the [`CONFIGURE ZONE`](configure-zone.html) subcommand for more details. diff --git a/src/current/v21.1/alter-role.md b/src/current/v21.1/alter-role.md deleted file mode 100644 index fee274952e1..00000000000 --- a/src/current/v21.1/alter-role.md +++ /dev/null @@ -1,126 +0,0 @@ ---- -title: ALTER ROLE -summary: The ALTER ROLE statement can be used to add or change a role's password. -toc: true ---- - - The `ALTER ROLE` [statement](sql-statements.html) can be used to add, change, or remove a [role's](create-role.html) password and to change the role options for a role. - -{{site.data.alerts.callout_info}} -Since the keywords `ROLE` and `USER` can now be used interchangeably in SQL statements for enhanced Postgres compatibility, `ALTER ROLE` is now an alias for [`ALTER USER`](alter-user.html). -{{site.data.alerts.end}} - -## Considerations - -- Password creation and alteration is supported only in secure clusters. - -## Required privileges - - To alter other roles, the role must be a member of the `admin` role or have the [`CREATEROLE`](create-role.html#create-a-role-that-can-create-other-roles-and-manage-authentication-methods-for-the-new-roles) role option set. - -## Synopsis - -
{% include {{ page.version.version }}/sql/generated/diagrams/alter_role.html %}
- -## Parameters - - - -Parameter | Description -----------|------------- -`name` | The name of the role whose parameters you want to alter. -`CREATELOGIN`/`NOCREATELOGIN` | Allow or disallow the role to manage authentication using the `WITH PASSWORD`, `VALID UNTIL`, and `LOGIN/NOLOGIN` role options.

By default, the role option is set to `NOCREATELOGIN` for all non-admin roles. -`LOGIN`/`NOLOGIN` | The `LOGIN` role option allows a role to login with one of the [client authentication methods](authentication.html#client-authentication). Setting the role option to `NOLOGIN` prevents the role from logging in using any authentication method. -`password` | Let the role [authenticate their access to a secure cluster](authentication.html#client-authentication) using this new password. Passwords should be entered as a [string literal](sql-constants.html#string-literals). For compatibility with PostgreSQL, a password can also be entered as an identifier.

To prevent a role from using [password authentication](authentication.html#client-authentication) and to mandate [certificate-based client authentication](authentication.html#client-authentication), [set the password as `NULL`](#prevent-a-role-from-using-password-authentication). -`VALID UNTIL` | The date and time (in the [`timestamp`](timestamp.html) format) after which the password is not valid. -`CREATEROLE`/`NOCREATEROLE` | Allow or disallow the role to [create](create-role.html), alter, and [drop](drop-role.html) other non-admin roles.

By default, the role option is set to `NOCREATEROLE` for all non-admin roles. -`CREATEDB`/`NOCREATEDB` | Allow or disallow the role to [create](create-database.html) or [rename](rename-database.html) a database. The role is assigned as the owner of the database.

By default, the role option is set to `NOCREATEDB` for all non-admin roles. -`CONTROLJOB`/`NOCONTROLJOB` | Allow or disallow the role to [pause](pause-job.html), [resume](resume-job.html), and [cancel](cancel-job.html) jobs. Non-admin roles cannot control jobs created by admins.

By default, the role option is set to `NOCONTROLJOB` for all non-admin roles. -`CANCELQUERY`/`NOCANCELQUERY` | Allow or disallow the role to cancel [queries](cancel-query.html) and [sessions](cancel-session.html) of other roles. Without this role option, roles can only cancel their own queries and sessions. Even with this role option, non-admins cannot cancel admin queries or sessions. This option should usually be combined with `VIEWACTIVITY` so that the role can view other roles' query and session information.

By default, the role option is set to `NOCANCELQUERY` for all non-admin roles. -`VIEWACTIVITY`/`NOVIEWACTIVITY` | Allow or disallow a role to see other roles' [queries](show-statements.html) and [sessions](show-sessions.html) using `SHOW STATEMENTS`, `SHOW SESSIONS`, and the [**Statements**](ui-statements-page.html) and **Transactions** pages in the DB Console. Without this role option, the `SHOW` commands only show the role's own data and the DB Console pages are unavailable.

By default, the role option is set to `NOVIEWACTIVITY` for all non-admin roles. -`CONTROLCHANGEFEED`/`NOCONTROLCHANGEFEED` | Allow or disallow the role to run [`CREATE CHANGEFEED`](create-changefeed.html) on tables they have `SELECT` privileges on.

By default, the role option is set to `NOCONTROLCHANGEFEED` for all non-admin roles. -`MODIFYCLUSTERSETTING`/`NOMODIFYCLUSTERSETTING` | Allow or disallow the role to modify the [cluster settings](cluster-settings.html) with the `sql.defaults` prefix.

By default, the role option is set to `NOMODIFYCLUSTERSETTING` for all non-admin roles. - -## Examples - -{{site.data.alerts.callout_info}} -The following statements are run by the `root` user that is a member of the `admin` role and has `ALL` privileges. -{{site.data.alerts.end}} - -### Allow a role to log in to the database using a password - -The following example allows a role to log in to the database with a [password](authentication.html#client-authentication): - -~~~ sql -root@:26257/defaultdb> ALTER ROLE carl WITH LOGIN PASSWORD 'An0ther$tr0nGpassW0rD' VALID UNTIL '2021-10-10'; -~~~ - -### Prevent a role from using password authentication - -The following statement prevents the user from using password authentication and mandates certificate-based [client authentication](authentication.html#client-authentication): - -{% include_cached copy-clipboard.html %} -~~~ sql -root@:26257/defaultdb> ALTER ROLE carl WITH PASSWORD NULL; -~~~ - -### Allow a role to create other roles and manage authentication methods for the new roles - -The following example allows the role to [create other roles](create-role.html) and [manage authentication methods](authentication.html#client-authentication) for them: - -~~~ sql -root@:26257/defaultdb> ALTER ROLE carl WITH CREATEROLE CREATELOGIN; -~~~ - -### Allow a role to create and rename databases - -The following example allows the role to [create](create-database.html) or [rename](rename-database.html) databases: - -~~~ sql -root@:26257/defaultdb> ALTER ROLE carl WITH CREATEDB; -~~~ - -### Allow a role to pause, resume, and cancel non-admin jobs - -The following example allows the role to [pause](pause-job.html), [resume](resume-job.html), and [cancel](cancel-job.html) jobs: - -~~~ sql -root@:26257/defaultdb> ALTER ROLE carl WITH CONTROLJOB; -~~~ - -### Allow a role to see and cancel non-admin queries and sessions - -The following example allows the role to cancel [queries](cancel-query.html) and [sessions](cancel-session.html) for other non-admin roles: - -~~~ sql -root@:26257/defaultdb> ALTER ROLE carl WITH CANCELQUERY VIEWACTIVITY; -~~~ - -### Allow a role to control changefeeds - -The following example allows the role to run [`CREATE CHANGEFEED`](create-changefeed.html): - -~~~ sql -root@:26257/defaultdb> ALTER ROLE carl WITH CONTROLCHANGEFEED; -~~~ - -### Allow a role to modify cluster settings - -The following example allows the role to modify [cluster settings](cluster-settings.html): - -~~~ sql -root@:26257/defaultdb> ALTER ROLE carl WITH MODIFYCLUSTERSETTING; -~~~ - -## See also - -- [`DROP ROLE`](drop-role.html) -- [`SHOW ROLES`](show-roles.html) -- [`GRANT`](grant.html) -- [`SHOW GRANTS`](show-grants.html) -- [Create Security Certificates](cockroach-cert.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/alter-schema.md b/src/current/v21.1/alter-schema.md deleted file mode 100644 index 62a544f07c0..00000000000 --- a/src/current/v21.1/alter-schema.md +++ /dev/null @@ -1,200 +0,0 @@ ---- -title: ALTER SCHEMA -summary: The ALTER SCHEMA statement modifies a user-defined schema in a database. -toc: true ---- - - The `ALTER SCHEMA` [statement](sql-statements.html) modifies a user-defined [schema](sql-name-resolution.html#naming-hierarchy). CockroachDB currently supports changing the name of the schema and the owner of the schema. - -## Syntax - -
-{% include {{ page.version.version }}/sql/generated/diagrams/alter_schema.html %} -
- -### Parameters - -Parameter | Description -----------|------------ -`name`
`name.name` | The name of the schema to alter, or the name of the database containing the schema and the schema name, separated by a "`.`". -`RENAME TO schema_name` | Rename the schema to `schema_name`. The new schema name must be unique within the database and follow these [identifier rules](keywords-and-identifiers.html#identifiers). -`OWNER TO role_spec` | Change the owner of the schema to `role_spec`. - -## Required privileges - -- To rename a schema, the user must be the owner of the schema. -- To change the owner of a schema, the user must be the current owner of the schema and a member of the new owner [role](authorization.html#roles). The new owner role must also have the `CREATE` [privilege](authorization.html#assign-privileges) on the database to which the schema belongs. - -## Example - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### Rename a schema - -Suppose that you access the [SQL shell](cockroach-sql.html) as user `root`, and [create a new user](create-user.html) `max` and [a schema](create-schema.html) `org_one` with `max` as the owner: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE USER max; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE SCHEMA org_one AUTHORIZATION max; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SCHEMAS; -~~~ - -~~~ - schema_name ----------------------- - crdb_internal - information_schema - org_one - pg_catalog - pg_extension - public -(6 rows) -~~~ - -Now, suppose you want to rename the schema: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER SCHEMA org_one RENAME TO org_two; -~~~ - -~~~ -ERROR: must be owner of schema "org_one" -SQLSTATE: 42501 -~~~ - -Because you are executing the `ALTER SCHEMA` command as a non-owner of the schema (i.e., `root`), CockroachDB returns an error. - -[Drop the schema](drop-schema.html) and create it again, this time with `root` as the owner. - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP SCHEMA org_one; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE SCHEMA org_one; -~~~ - -To verify that the owner is now `root`, query the `pg_catalog.pg_namespace` and `pg_catalog.pg_users` tables: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT - nspname, usename -FROM - pg_catalog.pg_namespace - LEFT JOIN pg_catalog.pg_user ON pg_namespace.nspowner = pg_user.usesysid -WHERE - nspname LIKE 'org_one'; -~~~ - -~~~ - nspname | usename -----------+---------- - org_one | root -(1 row) -~~~ - -As its owner, you can rename the schema: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER SCHEMA org_one RENAME TO org_two; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SCHEMAS; -~~~ - -~~~ - schema_name ----------------------- - crdb_internal - information_schema - org_two - pg_catalog - pg_extension - public -(6 rows) -~~~ - -### Change a schema's owner - -Suppose that you access the [SQL shell](cockroach-sql.html) as user `root`, and [create a new schema](create-schema.html) named `org_one`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE SCHEMA org_one; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SCHEMAS; -~~~ - -~~~ - schema_name ----------------------- - crdb_internal - information_schema - org_one - pg_catalog - pg_extension - public -(6 rows) -~~~ - -Now, suppose that you want to change the owner of the schema `org_one` to an existing user named `max`. To change the owner of a schema, the current owner must belong to the role of the new owner (in this case, `max`), and the new owner must have `CREATE` privileges on the database. - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT max TO root; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT CREATE ON DATABASE defaultdb TO max; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER SCHEMA org_one OWNER TO max; -~~~ - -To verify that the owner is now `max`, query the `pg_catalog.pg_namespace` and `pg_catalog.pg_users` tables: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT - nspname, usename -FROM - pg_catalog.pg_namespace - LEFT JOIN pg_catalog.pg_user ON pg_namespace.nspowner = pg_user.usesysid -WHERE - nspname LIKE 'org_one'; -~~~ - -~~~ - nspname | usename -----------+---------- - org_one | max -(1 row) -~~~ - -## See also - -- [`CREATE SCHEMA`](create-schema.html) -- [`SHOW SCHEMAS`](show-schemas.html) -- [`DROP SCHEMA`](drop-schema.html) diff --git a/src/current/v21.1/alter-sequence.md b/src/current/v21.1/alter-sequence.md deleted file mode 100644 index 9135997a9fb..00000000000 --- a/src/current/v21.1/alter-sequence.md +++ /dev/null @@ -1,256 +0,0 @@ ---- -title: ALTER SEQUENCE -summary: Use the ALTER SEQUENCE statement to change the name, increment values, and other settings of a sequence. -toc: true ---- - -The `ALTER SEQUENCE` [statement](sql-statements.html) applies a [schema change](online-schema-changes.html) to a sequence. - -{% include {{{ page.version.version }}/misc/schema-change-stmt-note.md %} - -## Required privileges - -- To alter a sequence, the user must have the `CREATE` [privilege](authorization.html#assign-privileges) on the parent database. -- To change the schema of a sequence with `ALTER SEQUENCE ... SET SCHEMA`, or to change the database of a sequence with `ALTER SEQUENCE ... RENAME TO`, the user must also have the `DROP` [privilege](authorization.html#assign-privileges) on the sequence. - -## Syntax - -
{% include {{ page.version.version }}/sql/generated/diagrams/alter_sequence.html %}
- -## Parameters - - - - Parameter | Description ------------|------------ -`IF EXISTS` | Modify the sequence only if it exists; if it does not exist, do not return an error. -`sequence_name` | The name of the sequence. -`RENAME TO sequence_name` | Rename the sequence to `sequence_name`, which must be unique to its database and follow these [identifier rules](keywords-and-identifiers.html#identifiers). Name changes do not propagate to the table(s) using the sequence.

Note that `RENAME TO` can be used to move a sequence from one database to another, but it cannot be used to move a sequence from one schema to another. To change a sequence's schema, use `ALTER SEQUENCE ...SET SCHEMA` instead. In a future release, `RENAME TO` will be limited to changing the name of a sequence, and will not have to the ability to change a sequence's database. -`CYCLE`/`NO CYCLE` | The sequence will wrap around when the sequence value hits the maximum or minimum value. If `NO CYCLE` is set, the sequence will not wrap. -`OWNED BY column_name` | Associates the sequence to a particular column. If that column or its parent table is dropped, the sequence will also be dropped.

Specifying an owner column with `OWNED BY` replaces any existing owner column on the sequence. To remove existing column ownership on the sequence and make the column free-standing, specify `OWNED BY NONE`.

**Default:** `NONE` -`CACHE` | **New in v21.1:** The number of sequence values to cache in memory for reuse in the session. A cache size of `1` means that there is no cache, and cache sizes of less than `1` are not valid.

**Default:** `1` (sequences are not cached by default) -`MINVALUE` | The new minimum value of the sequence.

**Default:** `1` -`MAXVALUE` | The new maximum value of the sequence.

**Default:** `9223372036854775807` -`INCREMENT` | The new value by which the sequence is incremented. A negative number creates a descending sequence. A positive number creates an ascending sequence. -`START` | The value the sequence starts at if you `RESTART` or if the sequence hits the `MAXVALUE` and `CYCLE` is set.

`RESTART` and `CYCLE` are not implemented yet. -`VIRTUAL` | Creates a *virtual sequence*.

Virtual sequences are sequences that do not generate monotonically increasing values and instead produce values like those generated by the built-in function [`unique_rowid()`](functions-and-operators.html). They are intended for use in combination with [`SERIAL`](serial.html)-typed columns. -`SET SCHEMA schema_name` | Change the schema of the sequence to `schema_name`. -`OWNER TO role_spec` | **New in v21.1:** Change the owner of the sequence to `role_spec`. - -## Examples - -### Change the increment value of a sequence - -In this example, we're going to change the increment value of a sequence from its current state (i.e., `1`) to `2`. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE SEQUENCE customer_seq; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE customer_seq; -~~~ - -~~~ - table_name | create_statement ----------------+------------------------------------------------------------------------------------------- - customer_seq | CREATE SEQUENCE customer_seq MINVALUE 1 MAXVALUE 9223372036854775807 INCREMENT 1 START 1 -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER SEQUENCE customer_seq INCREMENT 2; -~~~ - -~~~ - table_name | create_statement ----------------+-------------------------------------------------------------------------------------------------- - customer_seq | CREATE SEQUENCE public.customer_seq MINVALUE 1 MAXVALUE 9223372036854775807 INCREMENT 2 START 1 -(1 row) -~~~ - -### Rename a sequence - -In this example, we will change the name of sequence. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE SEQUENCE even_numbers INCREMENT 2 START 2; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SEQUENCES; -~~~ - -~~~ - sequence_schema | sequence_name -------------------+---------------- - public | even_numbers -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER SEQUENCE even_numbers RENAME TO even_sequence; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SEQUENCES; -~~~ - -~~~ - sequence_schema | sequence_name -------------------+---------------- - public | even_sequence -(1 row) -~~~ - -### Change the database of a sequence - -In this example, we will move the sequence we renamed in the first example (`even_sequence`) from `defaultdb` (i.e., the default database) to a different database. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SEQUENCES FROM defaultdb; -~~~ - -~~~ - sequence_schema | sequence_name -------------------+---------------- - public | even_sequence -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE DATABASE mydb; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER SEQUENCE even_sequence RENAME TO mydb.even_sequence; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SEQUENCES FROM defaultdb; -~~~ - -~~~ - sequence_schema | sequence_name -------------------+---------------- -(0 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SEQUENCES FROM mydb; -~~~ - -~~~ - sequence_schema | sequence_name -------------------+---------------- - public | even_sequence -(1 row) -~~~ - -### Change the schema of a sequence - -Suppose you [create a sequence](create-sequence.html) that you would like to add to a new schema called `cockroach_labs`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE SEQUENCE even_numbers INCREMENT 2 START 2; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SEQUENCES; -~~~ - -~~~ - sequence_schema | sequence_name -------------------+---------------- - public | even_numbers -(1 row) -~~~ - -By default, [unqualified sequences](sql-name-resolution.html#lookup-with-unqualified-names) created in the database belong to the `public` schema: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE public.even_numbers; -~~~ - -~~~ - table_name | create_statement -----------------------+-------------------------------------------------------------------------------------------------- - public.even_numbers | CREATE SEQUENCE public.even_numbers MINVALUE 1 MAXVALUE 9223372036854775807 INCREMENT 2 START 2 -(1 row) -~~~ - -If the new schema does not already exist, [create it](create-schema.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE SCHEMA IF NOT EXISTS cockroach_labs; -~~~ - -Then, change the sequence's schema: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER SEQUENCE even_numbers SET SCHEMA cockroach_labs; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE public.even_numbers; -~~~ - -~~~ -ERROR: relation "public.even_numbers" does not exist -SQLSTATE: 42P01 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SEQUENCES; -~~~ - -~~~ - sequence_schema | sequence_name -------------------+---------------- - cockroach_labs | even_numbers -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE cockroach_labs.even_numbers; -~~~ - -~~~ - table_name | create_statement -------------------------------+---------------------------------------------------------------------------------------------------------- - cockroach_labs.even_numbers | CREATE SEQUENCE cockroach_labs.even_numbers MINVALUE 1 MAXVALUE 9223372036854775807 INCREMENT 2 START 2 -(1 row) -~~~ - -## See also - -- [`CREATE SEQUENCE`](create-sequence.html) -- [`DROP SEQUENCE`](drop-sequence.html) -- [`SHOW SEQUENCES`](show-sequences.html) -- [Functions and Operators](functions-and-operators.html) -- [Other SQL Statements](sql-statements.html) -- [Online Schema Changes](online-schema-changes.html) diff --git a/src/current/v21.1/alter-table.md b/src/current/v21.1/alter-table.md deleted file mode 100644 index b4223e46a64..00000000000 --- a/src/current/v21.1/alter-table.md +++ /dev/null @@ -1,39 +0,0 @@ ---- -title: ALTER TABLE -summary: Use the ALTER TABLE statement to change the schema of a table. -toc: true ---- - -The `ALTER TABLE` [statement](sql-statements.html) applies a schema change to a table. For information on using `ALTER TABLE`, see the pages for its relevant [subcommands](#subcommands). - -{% include {{ page.version.version }}/misc/schema-change-stmt-note.md %} - -## Subcommands - -{{site.data.alerts.callout_success}} -Some subcommands can be used in combination in a single `ALTER TABLE` statement. For example, you can [atomically rename a column and add a new column with the old name of the existing column](rename-column.html#add-and-rename-columns-atomically). -{{site.data.alerts.end}} - -Subcommand | Description | Can combine with other subcommands? ------------|-------------|------------------------------------ -[`ADD COLUMN`](add-column.html) | Add columns to tables. | Yes -[`ADD CONSTRAINT`](add-constraint.html) | Add constraints to columns. | Yes -[`ALTER COLUMN`](alter-column.html) | Change an existing column. | Yes -[`ALTER PRIMARY KEY`](alter-primary-key.html) | Change the [primary key](primary-key.html) of a table. | Yes | No -[`DROP COLUMN`](drop-column.html) | Remove columns from tables. | Yes -[`DROP CONSTRAINT`](drop-constraint.html) | Remove constraints from columns. | Yes -[`EXPERIMENTAL_AUDIT`](experimental-audit.html) | Enable per-table audit logs, for security purposes. | Yes -[`OWNER TO`](owner-to.html) | Change the owner of the table. -[`PARTITION BY`](partition-by.html) | Partition, re-partition, or un-partition a table ([Enterprise-only](enterprise-licensing.html)). | Yes -[`RENAME COLUMN`](rename-column.html) | Change the names of columns. | Yes -[`RENAME CONSTRAINT`](rename-constraint.html) | Change constraints columns. | Yes -[`RENAME TO`](rename-table.html) | Change the names of tables. | No -[`SET SCHEMA`](set-schema.html) | Change the [schema](sql-name-resolution.html) of a table. | No -[`SPLIT AT`](split-at.html) | Force a range split at the specified row in the table. | No -[`UNSPLIT AT`](unsplit-at.html) | Remove a range split enforcement at a specified row in the table. | No -[`VALIDATE CONSTRAINT`](validate-constraint.html) | Check whether values in a column match a [constraint](constraints.html) on the column. | Yes -[`SET LOCALITY {REGIONAL BY TABLE, REGIONAL BY ROW, GLOBAL}`](set-locality.html) | **New in v21.1:** Set the table locality for a table in a [multi-region database](multiregion-overview.html). | No - -## Viewing schema changes - -{% include {{ page.version.version }}/misc/schema-change-view-job.md %} diff --git a/src/current/v21.1/alter-type.md b/src/current/v21.1/alter-type.md deleted file mode 100644 index 17ef12d6f1a..00000000000 --- a/src/current/v21.1/alter-type.md +++ /dev/null @@ -1,164 +0,0 @@ ---- -title: ALTER TYPE -summary: The ALTER TYPE statement modifies a user-defined data type in a database. -toc: true ---- - -The `ALTER TYPE` [statement](sql-statements.html) modifies a user-defined, [enumerated data type](enum.html) in the current database. - -{{site.data.alerts.callout_info}} -You can only [cancel](cancel-job.html) `ALTER TYPE` [schema change jobs](online-schema-changes.html) that drop values. All other `ALTER TYPE` schema change jobs are non-cancellable. -{{site.data.alerts.end}} - -## Synopsis - -
-{% include {{ page.version.version }}/sql/generated/diagrams/alter_type.html %} -
- -## Parameters - -Parameter | Description -----------|------------ -`type_name` | The name of the user-defined type. -`ADD VALUE value` | Add a constant value to the user-defined type's list of values. You can optionally specify `BEFORE value` or `AFTER value` to add the value in sort order relative to an existing value. -`DROP VALUE value` | **New in v21.1:** Drop a specific value from the user-defined type's list of values.
{{site.data.alerts.callout_info}}`ALTER TYPE ... DROP VALUE` is disabled by default with the `enable_drop_enum_value` [cluster setting](cluster-settings.html) set to `off`. To enable `ALTER TYPE ... DROP VALUE`, run `SET enable_drop_enum_value = on;`.{{site.data.alerts.end}} -`RENAME TO name` | Rename the user-defined type. -`RENAME VALUE value TO value` | Rename a constant value in the user-defined type's list of values. -`SET SCHEMA` | Set [the schema](sql-name-resolution.html) of the user-defined type. -`OWNER TO` | Change the [role specification](grant.html) for the user-defined type's owner. - -## Required privileges - -- To [alter a type](alter-type.html), the user must be the owner of the type. -- To set the schema of a user-defined type, the user must have the `CREATE` [privilege](authorization.html#assign-privileges) on the schema and the `DROP` privilege -on the type. -- To alter the owner of a user-defined type: - - The user executing the command must be a member of the new owner role. - - The new owner role must have the `CREATE` privilege on the schema the type belongs to. - -## Known limitations - -- You can only reference a user-defined type from the database that contains the type. -- Expressions in [views](views.html), [default values](default-value.html), and [computed columns](computed-columns.html) will stop working if they reference an `ENUM` value dropped by an `ALTER TYPE ... DROP VALUE` statement. As a result, `ALTER TYPE ... DROP VALUE` is disabled by default with the `enable_drop_enum_value` [cluster setting](cluster-settings.html) set to `off`. You can enable `ALTER TYPE ... DROP VALUE` by running `SET enable_drop_enum_value = on;`. - -## Example - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TYPE status AS ENUM ('open', 'closed', 'inactive'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW ENUMS; -~~~ - -~~~ - schema | name | values | owner ----------+--------+------------------------+-------- - public | status | {open,closed,inactive} | demo -(1 row) -~~~ - -### Add a value to a user-defined type - -To add a value to the `status` type, use an `ADD VALUE` clause: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TYPE status ADD VALUE 'pending'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW ENUMS; -~~~ - -~~~ - schema | name | values | owner ----------+--------+--------------------------------+-------- - public | status | {open,closed,inactive,pending} | demo -(1 row) -~~~ - -### Rename a value in a user-defined type - -To rename a value in the `status` type, use a `RENAME VALUE` clause: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TYPE status RENAME VALUE 'open' TO 'active'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW ENUMS; -~~~ - -~~~ - schema | name | values | owner ----------+--------+----------------------------------+-------- - public | status | {active,closed,inactive,pending} | demo -(1 row) -~~~ - -### Rename a user-defined type - -To rename the `status` type, use a `RENAME TO` clause: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TYPE status RENAME TO account_status; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW ENUMS; -~~~ - -~~~ - schema | name | values | owner ----------+----------------+----------------------------------+-------- - public | account_status | {active,closed,inactive,pending} | demo -(1 row) -~~~ - -### Drop a value in a user-defined type - -To drop a value from the `account_status` type, use a `DROP VALUE` clause. - -Note that expressions in [views](views.html), [default values](default-value.html), and [computed columns](computed-columns.html) will stop working if they reference a dropped `ENUM` value. As a result, `ALTER TYPE ... DROP VALUE` is disabled by default with the `enable_drop_enum_value` [cluster setting](cluster-settings.html) set to `off`. - -To enable `ALTER TYPE ... DROP VALUE`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET enable_drop_enum_value = on; -~~~ - -Then, to drop a value from the type: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TYPE account_status DROP VALUE 'inactive'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW ENUMS; -~~~ - -~~~ - schema | name | values | owner ----------+----------------+-------------------------+-------- - public | account_status | {active,closed,pending} | demo -(1 row) -~~~ - -## See also - -- [`CREATE TYPE`](create-type.html) -- [`ENUM`](enum.html) -- [`SHOW ENUMS`](show-enums.html) -- [`DROP TYPE`](drop-type.html) diff --git a/src/current/v21.1/alter-user.md b/src/current/v21.1/alter-user.md deleted file mode 100644 index 262ddba606a..00000000000 --- a/src/current/v21.1/alter-user.md +++ /dev/null @@ -1,124 +0,0 @@ ---- -title: ALTER USER -summary: The ALTER USER statement can be used to add or change a user's password. -toc: true ---- - -The `ALTER USER` [statement](sql-statements.html) can be used to add, change, or remove a [user's](create-user.html) password and to change the role options for a user. - -{{site.data.alerts.callout_info}} - Since the keywords `ROLE` and `USER` can now be used interchangeably in SQL statements for enhanced Postgres compatibility, `ALTER USER` is now an alias for [`ALTER ROLE`](alter-role.html). -{{site.data.alerts.end}} - -## Considerations - -- Password creation and alteration is supported only in secure clusters. - -## Required privileges - - To alter other users, the user must be a member of the `admin` role or have the [`CREATEROLE`](create-user.html#create-a-user-that-can-create-other-users-and-manage-authentication-methods-for-the-new-users) parameter set. - -## Synopsis - -
{% include {{ page.version.version }}/sql/generated/diagrams/alter_user_password.html %}
- -## Parameters - - - -Parameter | Description -----------|------------- -`name` | The name of the user whose role options you want to alter. -`CREATELOGIN`/`NOCREATELOGIN` | Allow or disallow the user to manage authentication using the `WITH PASSWORD`, `VALID UNTIL`, and `LOGIN/NOLOGIN` parameters.

By default, the parameter is set to `NOCREATELOGIN` for all non-admin users. -`LOGIN`/`NOLOGIN` | The `LOGIN` parameter allows a user to login with one of the [client authentication methods](authentication.html#client-authentication). Setting the parameter to `NOLOGIN` prevents the user from logging in using any authentication method. -`password` | Let the user [authenticate their access to a secure cluster](authentication.html#client-authentication) using this new password. Passwords should be entered as a [string literal](sql-constants.html#string-literals). For compatibility with PostgreSQL, a password can also be entered as an identifier.

To prevent a user from using [password authentication](authentication.html#client-authentication) and to mandate [certificate-based client authentication](authentication.html#client-authentication), [set the password as `NULL`](#prevent-a-user-from-using-password-authentication). -`VALID UNTIL` | The date and time (in the [`timestamp`](timestamp.html) format) after which the password is not valid. -`CREATEROLE`/`NOCREATEROLE` | Allow or disallow the user to [create](create-user.html), alter, and [drop](drop-user.html) other non-admin users.

By default, the parameter is set to `NOCREATEROLE` for all non-admin users. -`CREATEDB`/`NOCREATEDB` | Allow or disallow the user to [create](create-database.html) or [rename](rename-database.html) a database. The user is assigned as the owner of the database.

By default, the parameter is set to `NOCREATEDB` for all non-admin users. -`CONTROLJOB`/`NOCONTROLJOB` | Allow or disallow the user to [pause](pause-job.html), [resume](resume-job.html), and [cancel](cancel-job.html) jobs. Non-admin users cannot control jobs created by admins.

By default, the parameter is set to `NOCONTROLJOB` for all non-admin users. -`CANCELQUERY`/`NOCANCELQUERY` | Allow or disallow the user to cancel [queries](cancel-query.html) and [sessions](cancel-session.html) of other users. Without this privilege, users can only cancel their own queries and sessions. Even with this privilege, non-admins cannot cancel admin queries or sessions. This option should usually be combined with `VIEWACTIVITY` so that the user can view other users' query and session information.

By default, the parameter is set to `NOCANCELQUERY` for all non-admin users. -`VIEWACTIVITY`/`NOVIEWACTIVITY` | Allow or disallow a role to see other users' [queries](show-statements.html) and [sessions](show-sessions.html) using `SHOW STATEMENTS`, `SHOW SESSIONS`, and the [**Statements**](ui-statements-page.html) and **Transactions** pages in the DB Console. Without this privilege, the `SHOW` commands only show the user's own data and the DB Console pages are unavailable.

By default, the parameter is set to `NOVIEWACTIVITY` for all non-admin users. -`CONTROLCHANGEFEED`/`NOCONTROLCHANGEFEED` | Allow or disallow the user to run [`CREATE CHANGEFEED`](create-changefeed.html) on tables they have `SELECT` privileges on.

By default, the parameter is set to `NOCONTROLCHANGEFEED` for all non-admin users. -`MODIFYCLUSTERSETTING`/`NOMODIFYCLUSTERSETTING` | Allow or disallow the user to modify the [cluster settings](cluster-settings.html) with the `sql.defaults` prefix.

By default, the parameter is set to `NOMODIFYCLUSTERSETTING` for all non-admin users. - -## Examples - -{{site.data.alerts.callout_info}} -The following statements are run by the `root` user that is a member of the `admin` role and has `ALL` privileges. -{{site.data.alerts.end}} - -### Change a user's password - -~~~ sql -root@:26257/defaultdb> ALTER USER carl WITH PASSWORD 'An0ther$tr0nGpassW0rD' VALID UNTIL '2021-10-10'; -~~~ - -### Prevent a user from using password authentication - -The following statement prevents the user from using password authentication and mandates certificate-based [client authentication](authentication.html#client-authentication): - -{% include_cached copy-clipboard.html %} -~~~ sql -root@:26257/defaultdb> ALTER USER carl WITH PASSWORD NULL; -~~~ - -### Allow a user to create other users and manage authentication methods for the new users - -The following example allows the user to [create other users](create-user.html) and [manage authentication methods](authentication.html#client-authentication) for them: - -~~~ sql -root@:26257/defaultdb> ALTER USER carl WITH CREATEROLE CREATELOGIN; -~~~ - -### Allow a user to create and rename databases - -The following example allows the user to [create](create-database.html) or [rename](rename-database.html) databases: - -~~~ sql -root@:26257/defaultdb> ALTER USER carl WITH CREATEDB; -~~~ - -### Allow a user to pause, resume, and cancel non-admin jobs - -The following example allows the user to [pause](pause-job.html), [resume](resume-job.html), and [cancel](cancel-job.html) jobs: - -~~~ sql -root@:26257/defaultdb> ALTER USER carl WITH CONTROLJOB; -~~~ - -### Allow a user to see and cancel non-admin queries and sessions - -The following example allows the user to cancel [queries](cancel-query.html) and [sessions](cancel-session.html) for other non-admin roles: - -~~~ sql -root@:26257/defaultdb> ALTER USER carl WITH CANCELQUERY VIEWACTIVITY; -~~~ - -### Allow a user to control changefeeds - -The following example allows the user to run [`CREATE CHANGEFEED`](create-changefeed.html): - -~~~ sql -root@:26257/defaultdb> ALTER USER carl WITH CONTROLCHANGEFEED; -~~~ - -### Allow a user to modify cluster settings - -The following example allows the user to modify [cluster settings](cluster-settings.html): - -~~~ sql -root@:26257/defaultdb> ALTER USER carl WITH MODIFYCLUSTERSETTING; -~~~ - -## See also - -- [`DROP USER`](drop-user.html) -- [`SHOW USERS`](show-users.html) -- [`GRANT`](grant.html) -- [`SHOW GRANTS`](show-grants.html) -- [Create Security Certificates](cockroach-cert.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/alter-view.md b/src/current/v21.1/alter-view.md deleted file mode 100644 index 342ffbe6c74..00000000000 --- a/src/current/v21.1/alter-view.md +++ /dev/null @@ -1,143 +0,0 @@ ---- -title: ALTER VIEW -summary: The ALTER VIEW statement applies a schema change to a view. -toc: true ---- - -The `ALTER VIEW` [statement](sql-statements.html) applies a schema change to a [view](views.html). - -{% include {{{ page.version.version }}/misc/schema-change-stmt-note.md %} - -## Required privileges - -- To alter a view, the user must have the `CREATE` [privilege](authorization.html#assign-privileges) on the parent database. -- To change the schema of a view with `ALTER VIEW ... SET SCHEMA`, or to change the name of a view with `ALTER VIEW ... RENAME TO`, the user must also have the `DROP` [privilege](authorization.html#assign-privileges) on the view. - -## Syntax - -
-{% include {{ page.version.version }}/sql/generated/diagrams/alter_view.html %} -
- -## Parameters - -Parameter | Description -----------|------------ -`MATERIALIZED` | Rename a [materialized view](views.html#materialized-views). -`IF EXISTS` | Rename the view only if a view of `view_name` exists; if one does not exist, do not return an error. -`view_name` | The name of the view to rename. To find view names, use:

`SELECT * FROM information_schema.tables WHERE table_type = 'VIEW';` -`RENAME TO view_name` | Rename the view to `view_name`, which must be unique to its database and follow these [identifier rules](keywords-and-identifiers.html#identifiers). Name changes do not propagate to the table(s) using the view.

Note that `RENAME TO` can be used to move a view from one database to another, but it cannot be used to move a view from one schema to another. To change a view's schema, use `ALTER VIEW ...SET SCHEMA` instead. In a future release, `RENAME TO` will be limited to changing the name of a view, and will not have the ability to change a view's database. -`SET SCHEMA schema_name` | Change the schema of the view to `schema_name`. -`OWNER TO role_spec` | **New in v21.1:** Change the owner of the view to `role_spec`. - -## Limitations - -CockroachDB does not currently support: - -- Changing the [`SELECT`](select-clause.html) statement executed by a view. Instead, you must drop the existing view and create a new view. -- Renaming a view that other views depend on. This feature may be added in the future (see [tracking issue](https://github.com/cockroachdb/cockroach/issues/10083)). - -## Examples - -### Rename a view - -Suppose you create a new view that you want to rename: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE VIEW money_rides (id, revenue) AS SELECT id, revenue FROM rides WHERE revenue > 50; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW TABLES] WHERE type = 'view'; -~~~ - -~~~ - schema_name | table_name | type | owner | estimated_row_count | locality ---------------+-------------+------+-------+---------------------+----------- - public | money_rides | view | demo | 0 | NULL -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER VIEW money_rides RENAME TO expensive_rides; -~~~ -~~~ -RENAME VIEW -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW TABLES] WHERE type = 'view'; -~~~ - -~~~ - schema_name | table_name | type | owner | estimated_row_count | locality ---------------+-----------------+------+-------+---------------------+----------- - public | expensive_rides | view | demo | 0 | NULL -(1 row) -~~~ - -### Change the schema of a view - -Suppose you want to add the `expensive_rides` view to a schema called `cockroach_labs`: - -By default, [unqualified views](sql-name-resolution.html#lookup-with-unqualified-names) created in the database belong to the `public` schema: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE public.expensive_rides; -~~~ - -~~~ - table_name | create_statement --------------------------+------------------------------------------------------------------------------------------------------------------- - public.expensive_rides | CREATE VIEW public.expensive_rides (id, revenue) AS SELECT id, revenue FROM movr.public.rides WHERE revenue > 50 -(1 row) -~~~ - -If the new schema does not already exist, [create it](create-schema.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE SCHEMA IF NOT EXISTS cockroach_labs; -~~~ - -Then, change the view's schema: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER VIEW expensive_rides SET SCHEMA cockroach_labs; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE public.expensive_rides; -~~~ - -~~~ -ERROR: relation "public.expensive_rides" does not exist -SQLSTATE: 42P01 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE cockroach_labs.expensive_rides; -~~~ - -~~~ - table_name | create_statement ----------------------------------+--------------------------------------------------------------------------------------------------------------------------- - cockroach_labs.expensive_rides | CREATE VIEW cockroach_labs.expensive_rides (id, revenue) AS SELECT id, revenue FROM movr.public.rides WHERE revenue > 50 -(1 row) -~~~ - -## See also - -- [Views](views.html) -- [`CREATE VIEW`](create-view.html) -- [`SHOW CREATE`](show-create.html) -- [`DROP VIEW`](drop-view.html) -- [Online Schema Changes](online-schema-changes.html) diff --git a/src/current/v21.1/architecture/distribution-layer.md b/src/current/v21.1/architecture/distribution-layer.md deleted file mode 100644 index 2d2bfb6d0a3..00000000000 --- a/src/current/v21.1/architecture/distribution-layer.md +++ /dev/null @@ -1,241 +0,0 @@ ---- -title: Distribution Layer -summary: The distribution layer of CockroachDB's architecture provides a unified view of your cluster's data. -toc: true ---- - -The distribution layer of CockroachDB's architecture provides a unified view of your cluster's data. - -{{site.data.alerts.callout_info}} -If you haven't already, we recommend reading the [Architecture Overview](overview.html). -{{site.data.alerts.end}} - -## Overview - -To make all data in your cluster accessible from any node, CockroachDB stores data in a monolithic sorted map of key-value pairs. This key-space describes all of the data in your cluster, as well as its location, and is divided into what we call "ranges", contiguous chunks of the key-space, so that every key can always be found in a single range. - -CockroachDB implements a sorted map to enable: - - - **Simple lookups**: Because we identify which nodes are responsible for certain portions of the data, queries are able to quickly locate where to find the data they want. - - **Efficient scans**: By defining the order of data, it's easy to find data within a particular range during a scan. - -### Monolithic sorted map structure - -The monolithic sorted map is comprised of two fundamental elements: - -- System data, which include **meta ranges** that describe the locations of data in your cluster (among many other cluster-wide and local data elements) -- User data, which store your cluster's **table data** - -#### Meta ranges - -The locations of all ranges in your cluster are stored in a two-level index at the beginning of your key-space, known as meta ranges, where the first level (`meta1`) addresses the second, and the second (`meta2`) addresses data in the cluster. - -This two-level index plus user data can be visualized as a tree, with the root at `meta1`, the second level at `meta2`, and the leaves of the tree made up of the ranges that hold user data. - -![range-lookup.png](../../images/{{page.version.version}}/range-lookup.png "Meta ranges plus user data tree diagram") - -Importantly, every node has information on where to locate the `meta1` range (known as its range descriptor, detailed below), and the range is never split. - -This meta range structure lets us address up to 4EiB of user data by default: we can address 2^(18 + 18) = 2^36 ranges; each range addresses 2^26 B, and altogether we address 2^(36+26) B = 2^62 B = 4EiB. However, with larger range sizes, it's possible to expand this capacity even further. - -Meta ranges are treated mostly like normal ranges and are accessed and replicated just like other elements of your cluster's KV data. - -Each node caches values of the `meta2` range it has accessed before, which optimizes access of that data in the future. Whenever a node discovers that its `meta2` cache is invalid for a specific key, the cache is updated by performing a regular read on the `meta2` range. - -#### Table data - -After the node's meta ranges is the KV data your cluster stores. - -Each table and its secondary indexes initially map to a single range, where each key-value pair in the range represents a single row in the table (also called the primary index because the table is sorted by the primary key) or a single row in a secondary index. As soon as a range reaches 512 MiB in size, it splits into two ranges. This process continues as a table and its indexes continue growing. Once a table is split across multiple ranges, it's likely that the table and secondary indexes will be stored in separate ranges. However, a range can still contain data for both the table and a secondary index. - -The default 512 MiB range size represents a sweet spot for us between a size that's small enough to move quickly between nodes, but large enough to store a meaningfully contiguous set of data whose keys are more likely to be accessed together. These ranges are then shuffled around your cluster to ensure survivability. - -These table ranges are replicated (in the aptly named replication layer), and have the addresses of each replica stored in the `meta2` range. - -### Using the monolithic sorted map - -As described in the [meta ranges section](#meta-ranges), the locations of all the ranges in a cluster are stored in a two-level index: - -- The first level (`meta1`) addresses the second level. -- The second level (`meta2`) addresses user data. - -This can also be visualized as a tree, with the root at `meta1`, the second level at `meta2`, and the leaves of the tree made up of the ranges that hold user data. - -When a node receives a request, it looks up the location of the range(s) that include the keys in the request in a bottom-up fashion, starting with the leaves of this tree. This process works as follows: - -1. For each key, the node looks up the location of the range containing the specified key in the second level of range metadata (`meta2`). That information is cached for performance; if the range's location is found in the cache, it is returned immediately. - -2. If the range's location is not found in the cache, the node looks up the location of the range where the actual value of `meta2` resides. This information is also cached; if the location of the `meta2` range is found in the cache, the node sends an RPC to the `meta2` range to get the location of the keys the request wants to operate on, and returns that information. - -3. Finally, if the location of the `meta2` range is not found in the cache, the node looks up the location of the range where the actual value of the first level of range metadata (`meta1`) resides. This lookup always succeeds because the location of `meta1` is distributed among all the nodes in the cluster using a gossip protocol. The node then uses the information from `meta1` to look up the location of `meta2`, and from `meta2` it looks up the locations of the ranges that include the keys in the request. - -Note that the process described above is recursive; every time a lookup is performed, it either (1) gets a location from the cache, or (2) performs another lookup on the value in the next level "up" in the tree. Because the range metadata is cached, a lookup can usually be performed without having to send an RPC to another node. - -Now that the node has the location of the range where the key from the request resides, it sends the KV operations from the request along to the range (using the [`DistSender`](#distsender) machinery) in a [`BatchRequest`](#batchrequest). - -### Interactions with other layers - -In relationship to other layers in CockroachDB, the distribution layer: - -- Receives requests from the transaction layer on the same node. -- Identifies which nodes should receive the request, and then sends the request to the proper node's replication layer. - -## Technical details and components - -### gRPC - -gRPC is the software nodes use to communicate with one another. Because the distribution layer is the first layer to communicate with other nodes, CockroachDB implements gRPC here. - -gRPC requires inputs and outputs to be formatted as protocol buffers (protobufs). To leverage gRPC, CockroachDB implements a protocol-buffer-based API defined in `api.proto`. - -For more information about gRPC, see the [official gRPC documentation](http://www.grpc.io/docs/guides/). - -### BatchRequest - -All KV operation requests are bundled into a [protobuf](https://en.wikipedia.org/wiki/Protocol_Buffers), known as a `BatchRequest`. The destination of this batch is identified in the `BatchRequest` header, as well as a pointer to the request's transaction record. (On the other side, when a node is replying to a `BatchRequest`, it uses a protobuf––`BatchResponse`.) - -This `BatchRequest` is also what's used to send requests between nodes using gRPC, which accepts and sends protocol buffers. - -### DistSender - -The gateway/coordinating node's `DistSender` receives `BatchRequest`s from its own `TxnCoordSender`. `DistSender` is then responsible for breaking up `BatchRequests` and routing a new set of `BatchRequests` to the nodes it identifies contain the data using the system [meta ranges](#meta-ranges). For a description of the process by which this lookup from a key to the node holding the key's range is performed, see [Using the monolithic sorted map](#using-the-monolithic-sorted-map). - -It sends the `BatchRequest`s to the replicas of a range, ordered in expectation of request latency. The leaseholder is tried first, if the request needs it. Requests received by a non-leaseholder may fail with an error pointing at the replica's last known leaseholder. These requests are retried transparently with the updated lease by the gateway node and never reach the client. - -As nodes begin replying to these commands, `DistSender` also aggregates the results in preparation for returning them to the client. - -### Meta range KV structure - -Like all other data in your cluster, meta ranges are structured as KV pairs. Both meta ranges have a similar structure: - -~~~ -metaX/successorKey -> [list of nodes containing data] -~~~ - -Element | Description ---------|------------------------ -`metaX` | The level of meta range. Here we use a simplified `meta1` or `meta2`, but these are actually represented in `cockroach` as `\x02` and `\x03` respectively. -`successorKey` | The first key *greater* than the key you're scanning for. This makes CockroachDB's scans efficient; it simply scans the keys until it finds a value greater than the key it's looking for, and that is where it finds the relevant data.

The `successorKey` for the end of a keyspace is identified as `maxKey`. - -Here's an example: - -~~~ -meta2/M -> node1:26257, node2:26257, node3:26257 -~~~ - -In this case, the replica on `node1` is the leaseholder, and nodes 2 and 3 also contain replicas. - -#### Example - -Let's imagine we have an alphabetically sorted column, which we use for lookups. Here are what the meta ranges would approximately look like: - -1. `meta1` contains the address for the nodes containing the `meta2` replicas. - - ~~~ - # Points to meta2 range for keys [A-M) - meta1/M -> node1:26257, node2:26257, node3:26257 - - # Points to meta2 range for keys [M-Z] - meta1/maxKey -> node4:26257, node5:26257, node6:26257 - ~~~ - -2. `meta2` contains addresses for the nodes containing the replicas of each range in the cluster: - - ~~~ - # Contains [A-G) - meta2/G -> node1:26257, node2:26257, node3:26257 - - # Contains [G-M) - meta2/M -> node1:26257, node2:26257, node3:26257 - - #Contains [M-Z) - meta2/Z -> node4:26257, node5:26257, node6:26257 - - #Contains [Z-maxKey) - meta2/maxKey-> node4:26257, node5:26257, node6:26257 - ~~~ - -### Table data KV structure - -Key-value data, which represents the data in your tables using the following structure: - -~~~ -/// -> -~~~ - -The table itself is stored with an `index_id` of 1 for its `PRIMARY KEY` columns, with the rest of the columns in the table considered as stored/covered columns. - -### Range descriptors - -Each range in CockroachDB contains metadata, known as a range descriptor. A range descriptor is comprised of the following: - -- A sequential RangeID -- The keyspace (i.e., the set of keys) the range contains; for example, the first and last `` in the table data KV structure above. This determines the `meta2` range's keys. -- The addresses of nodes containing replicas of the range. This determines the `meta2` range's key's values. - -Because range descriptors comprise the key-value data of the `meta2` range, each node's `meta2` cache also stores range descriptors. - -Range descriptors are updated whenever there are: - -- Membership changes to a range's Raft group (discussed in more detail in the [Replication Layer](replication-layer.html#membership-changes-rebalance-repair)) -- Range splits -- Range merges - -All of these updates to the range descriptor occur locally on the range, and then propagate to the `meta2` range. - -### Range splits - -By default, CockroachDB attempts to keep ranges/replicas at the default range size (currently 512 MiB). Once a range reaches that limit, we split it into two smaller ranges (composed of contiguous key spaces). - -During this range split, the node creates a new Raft group containing all of the same members as the range that was split. The fact that there are now two ranges also means that there is a transaction that updates `meta2` with the new keyspace boundaries, as well as the addresses of the nodes using the range descriptor. - -### Range merges - -By default, CockroachDB automatically merges small ranges of data together to form fewer, larger ranges (up to the default range size). This can improve both query latency and cluster survivability. - -#### How range merges work - -As [described above](#overview), CockroachDB splits your cluster's data into many ranges. For example, your cluster might have a range for customers whose IDs are between `[1000, 2000)`. If that range grows beyond the default range size, the range is [split into two smaller ranges](#range-splits). - -However, as you delete data from your cluster, a range might contain far less data than the default range size. Over the lifetime of a cluster, this could lead to a number of small ranges. - -To reduce the number of small ranges, your cluster can have any range below a certain size threshold try to merge with its "right-hand neighbor", i.e., the range that starts where the current range ends. Using our example above, this range's right-hand neighbor might be the range for customers whose IDs are between `[2000, 3000)`. - -If the combined size of the small range and its neighbor is less than the maximum range size, the ranges merge into a single range. In our example, this will create a new range of keys `[1000, 3000)`. - -{{site.data.alerts.callout_info}} -When ranges merge, the left-hand-side (LHS) range consumes the right-hand-side (RHS) range. -{{site.data.alerts.end}} - -#### Why range merges improve performance - -##### Query latency - -Queries in CockroachDB must contact a replica of each range involved in the query. This creates the following issues for clusters with many small ranges: - -- Queries incur a fixed overhead in terms of processing time for each range they must coordinate with. -- Having many small ranges can increase the number of machines your query must coordinate with. This exposes your query to a greater likelihood of running into issues like network latency or overloaded nodes. - -By merging small ranges, CockroachDB can greatly reduce the number of ranges involved in queries, thus reducing query latency. - -##### Survivability - -CockroachDB automatically rebalances the distribution of ranges in your cluster whenever nodes come online or go offline. - -During rebalancing, it's better to replicate a few larger ranges across nodes than many smaller ranges. Replicating larger ranges requires less coordination and often completes more quickly. - -By merging smaller ranges together, your cluster needs to rebalance fewer total ranges. This ultimately improves your cluster's performance, especially in the face of availability events like node outages. - -## Technical interactions with other layers - -### Distribution and transaction layer - -The Distribution layer's `DistSender` receives `BatchRequests` from its own node's `TxnCoordSender`, housed in the Transaction layer. - -### Distribution and replication layer - -The Distribution layer routes `BatchRequests` to nodes containing ranges of data, which is ultimately routed to the Raft group leader or leaseholder, which are handled in the replication layer. - -## What's next? - -Learn how CockroachDB copies data and ensures consistency in the [replication layer](replication-layer.html). diff --git a/src/current/v21.1/architecture/life-of-a-distributed-transaction.md b/src/current/v21.1/architecture/life-of-a-distributed-transaction.md deleted file mode 100644 index 35299d55227..00000000000 --- a/src/current/v21.1/architecture/life-of-a-distributed-transaction.md +++ /dev/null @@ -1,187 +0,0 @@ ---- -title: Life of a Distributed Transaction -summary: Learn how a query moves through the layers of CockroachDB's architecture. -toc: true ---- - -Because CockroachDB is a distributed transactional database, the path queries take is dramatically different from many other database architectures. To help familiarize you with CockroachDB's internals, this guide covers what that path actually is. - -If you've already read the [CockroachDB architecture documentation](overview.html), this guide serves as another way to conceptualize how the database works. This time, instead of focusing on the layers of CockroachDB's architecture, we're going to focus on the linear path that a query takes through the system (and then back out again). - -To get the most out of this guide, we recommend beginning with the architecture documentation's [overview](overview.html) and progressing through all of the following sections. This guide provides brief descriptions of each component's function and links to other documentation where appropriate, but assumes the reader has a basic understanding of the architecture in the first place. - -## Overview - -This guide is organized by the physical actors in the system, and then broken down into the components of each actor in the sequence in which they're involved. - -Here's a brief overview of the physical actors, in the sequence with which they're involved in executing a query: - -1. [**SQL Client**](#sql-client-postgres-wire-protocol) sends a query to your cluster. -1. [**Load Balancing**](#load-balancing-routing) routes the request to CockroachDB nodes in your cluster, which will act as a gateway. -1. [**Gateway**](#gateway) is a CockroachDB node that processes the SQL request and responds to the client. -1. [**Leaseholder**](#leaseholder-node) is a CockroachDB node responsible for serving reads and coordinating writes of a specific range of keys in your query. -1. [**Raft leader**](#raft-leader) is a CockroachDB node responsible for maintaining consensus among your CockroachDB replicas. - -Once the transaction completes, queries traverse these actors in approximately reverse order. We say "approximately" because there might be many leaseholders and Raft leaders involved in a single query, and there is little-to-no interaction with the load balancer during the response. - -## SQL Client/Postgres Wire Protocol - -To begin, a SQL client (e.g., an app) performs some kind of business logic against your CockroachDB cluster, such as inserting a new customer record. - -This request is sent over a connection to your CockroachDB cluster that's established using a PostgreSQL driver. - -## Load Balancing & Routing - -Modern architectures require distributing your cluster across machines to improve throughput, latency, and uptime. This means queries are routed through load balancers, which choose the best CockroachDB node to connect to. Because all CockroachDB nodes have perfectly symmetrical access to data, this means your load balancer can connect your client to any node in the cluster and access any data while still guaranteeing strong consistency. - -Your architecture might also have additional layers of routing to enforce regulatory compliance, such as ensuring GDPR compliance. - -Once your router and load balancer determine the best node to connect to, your client's connection is established to the gateway node. - -## Gateway - -The gateway node handles the connection with the client, both receiving and responding to the request. - -### SQL parsing & planning - -The gateway node first [parses](sql-layer.html#sql-parser-planner-executor) the client's SQL statement to ensure it's valid according to the CockroachDB dialect of SQL, and uses that information to [generate a logical SQL plan](sql-layer.html#logical-planning). - -Given that CockroachDB is a distributed database, though, it's also important to take a cluster's topology into account, so the logical plan is then converted into a physical plan—this means sometimes pushing operations onto the physical machines that contain the data. - -### SQL executor - -While CockroachDB presents a SQL interface to clients, the actual database is built on top of a key-value store. To mediate this, the physical plan generated at the end of SQL parsing is passed to the SQL executor, which executes the plan by performing key-value operations through the `TxnCoordSender`. For example, the SQL executor converts `INSERT `statements into `Put()` operations. - -### TxnCoordSender - -The `TxnCoordSender` provides an API to perform key-value operations on your database. - -On its back end, the `TxnCoordSender` performs a large amount of the accounting and tracking for a transaction, including: - -- Accounts for all keys involved in a transaction. This is used, among other ways, to manage the transaction's state. -- Packages all key-value operations into a `BatchRequest`, which are forwarded on to the node's `DistSender`. - -### DistSender - -The gateway node's `DistSender` receives `BatchRequests` from the `TxnCoordSender`. It dismantles the initial `BatchRequest` by taking each operation and finding which physical machine should receive the request for the range—known as the range's leaseholder. The address of the range's current leaseholder is readily available in both local caches, as well as in the [cluster's `meta` ranges](distribution-layer.html#meta-range-kv-structure). - -These dismantled `BatchRequests` are reassembled into new `BatchRequests` containing the address of the range's leaseholder. - -All write operations also propagate the leaseholder's address back to the `TxnCoordSender`, so it can track and clean up write operations as necessary. - -The `DistSender` sends out the first `BatchRequest` for each range in parallel. As soon as it receives a provisional acknowledgment from the leaseholder node’s evaluator (details below), it sends out the next `BatchRequest` for that range. - -The `DistSender` then waits to receive acknowledgments for all of its write operations, as well as values for all of its read operations. However, this wait isn't necessarily blocking, and the `DistSender` may still perform operations with ongoing transactions. - -## Leaseholder node - -The gateway node's `DistSender` tries to send its `BatchRequests` to the replica identified as the range's [leaseholder](replication-layer.html#leases), which is a single replica that serves all reads for a range, as well as coordinates all writes. Leaseholders play a crucial role in CockroachDB's architecture, so it's a good topic to make sure you're familiar with. - -### Request response - -Because the leaseholder replica can shift between nodes, all nodes must be able to return a request for any key, returning a response indicating one of these scenarios: - -##### No Longer Leaseholder - -If a node is no longer the leaseholder, but still contains a replica of the range, it denies the request but includes the last known address for the leaseholder of that range. - -Upon receipt of this response, the `DistSender` will update the header of the `BatchRequest` with the new address, and then resend the `BatchRequest` to the newly identified leaseholder. - -##### No Longer Has/Never Had Range - -If a node doesn't have a replica for the requested range, it denies the request without providing any further information. - -In this case, the `DistSender` must look up the current leaseholder using the [cluster's `meta` ranges](distribution-layer.html#meta-range-kv-structure). - -##### Success - -Once the node that contains the leaseholder of the range receives the `BatchRequest`, it begins processing it, and progresses onto checking the timestamp cache. - -### Timestamp cache - -The timestamp cache tracks the highest timestamp (i.e., most recent) for any read operation that a given range has served. - -Each write operation in a `BatchRequest` checks its own timestamp versus the timestamp cache to ensure that the write operation has a higher timestamp; this guarantees that history is never rewritten and you can trust that reads always served the most recent data. It's one of the crucial mechanisms CockroachDB uses to ensure serializability. If a write operation fails this check, it must be restarted at a timestamp higher than the timestamp cache's value. - -### Latch manager - -Operations in the `BatchRequest` are serialized through the leaseholder's latch manager. - -This works by giving each write operation a latch on a row. Any reads or writes that come in after the latch has been granted on the row must wait for the write to complete, at which point the latch is released and the subsequent operations can continue. - -### Batch Evaluation - -The batch evaluator ensures that write operations are valid. Our architecture makes this fairly trivial. First, the evaluator can simply check the leaseholder's data to ensure the write is valid; because it has coordinated all writes to the range, it must have the most up-to-date versions of the range's data. Secondly, because of the latch manager, each write operation is guaranteed to uncontested access to the range (i.e., there is no contention with other write operations). - -If the write operation is valid according to the evaluator, the leaseholder sends a provisional acknowledgment to the gateway node's `DistSender`; this lets the `DistSender` begin to send its subsequent `BatchRequests` for this range. - -Importantly, this feature is entirely built for transactional optimization (known as [transaction pipelining](transaction-layer.html#transaction-pipelining)). There are no issues if an operation passes the evaluator but doesn't end up committing. - -### Reads from the storage layer - -All operations (including writes) begin by reading from the local instance of the [storage engine](storage-layer.html) to check for write intents for the operation's key. We talk much more about [write intents in the transaction layer of the CockroachDB architecture](transaction-layer.html#write-intents), which is worth reading, but a simplified explanation is that these are provisional, uncommitted writes that express that some other concurrent transaction plans to write a value to the key. - -What we detail below is a simplified version of the CockroachDB transaction model. For more detail, check out [the transaction architecture documentation](transaction-layer.html). - -#### Resolving Write Intents - -If an operation encounters a write intent for a key, it attempts to "resolve" the write intent by checking the state of the write intent's transaction. If the write intent's transaction record is... - -- `COMMITTED`, this operation converts the write intent to a regular key-value pair, and then proceeds as if it had read that value instead of a write intent. -- `ABORTED`, this operation discards the write intent and reads the next-most-recent value from the storage engine. -- `PENDING`, the new transaction attempts to "push" the write intent's transaction by moving that transaction's timestamp forward (i.e., ahead of this transaction's timestamp); however, this only succeeds if the write intent's transaction has become inactive. - - If the push succeeds, the operation continues. - - If this push fails (which is the majority of the time), this transaction goes into the [`TxnWaitQueue`](transaction-layer.html#txnwaitqueue) on this node. The incoming transaction can only continue once the blocking transaction completes (i.e., commits or aborts). -- `MISSING`, the resolver consults the write intent's timestamp. - - If it was created within the transaction liveness threshold, it treats the transaction record as exhibiting the `PENDING` behavior, with the addition of tracking the push in the range's timestamp cache, which will inform the transaction that its timestamp was pushed once the transaction record gets created. - - If the write intent is older than the transaction liveness threshold, the resolution exhibits the `ABORTED` behavior. - - Note that transaction records might be missing because we've avoided writing the record until the transaction commits. For more information, see [Transaction Layer: Transaction records](transaction-layer.html#transaction-records). - -Check out our architecture documentation for more information about [CockroachDB's transactional model](transaction-layer.html). - -#### Read Operations - -If the read doesn't encounter a write intent and the key-value operation is meant to serve a read, it can simply use the value it read from the leaseholder's instance of the storage engine. This works because the leaseholder had to be part of the Raft consensus group for any writes to complete, meaning it must have the most up-to-date version of the range's data. - -The leaseholder aggregates all read responses into a `BatchResponse` that will get returned to the gateway node's `DistSender`. - -As we mentioned before, each read operation also updates the timestamp cache. - -### Write Operations - -After guaranteeing that there are no existing write intents for the keys, `BatchRequest`'s key-value operations are converted to [Raft operations](replication-layer.html#raft) and have their values converted into write intents. - -The leaseholder then proposes these Raft operations to the Raft group leader. The leaseholder and the Raft leader are almost always the same node, but there are situations where the roles might drift to different nodes. However, when the two roles are not collocated on the same physical machine, CockroachDB will attempt to relocate them on the same node at the next opportunity. - -## Raft Leader - -CockroachDB leverages Raft as its consensus protocol. If you aren't familiar with it, we recommend checking out the details about [how CockroachDB leverages Raft](replication-layer.html#raft), as well as [learning more about how the protocol works at large](http://thesecretlivesofdata.com/raft/). - -In terms of executing transactions, the Raft leader receives proposed Raft commands from the leaseholder. Each Raft command is a write that is used to represent an atomic state change of the underlying key-value pairs stored in the storage engine. - -### Consensus - -For each command the Raft leader receives, it proposes a vote to the other members of the Raft group. - -Once the command achieves consensus (i.e., a majority of nodes including itself acknowledge the Raft command), it is committed to the Raft leader’s Raft log and written to the storage engine. At the same time, the Raft leader also sends a command to all other nodes to include the command in their Raft logs. - -Once the leader commits the Raft log entry, it’s considered committed. At this point the value is considered written, and if another operation comes in and performs a read from the storage engine for this key, they’ll encounter this value. - -Note that this write operation creates a write intent; these writes will not be fully committed until the gateway node sets the transaction record's status to `COMMITTED`. - -## On the way back up - -Now that we have followed an operation all the way down from the SQL client to the storage engine, we can pretty quickly cover what happens on the way back up (i.e., when generating a response to the client). - -1. Once the leaseholder applies a write to its Raft log, - it sends an commit acknowledgment to the gateway node's `DistSender`, which was waiting for this signal (having already received the provisional acknowledgment from the leaseholder's evaluator). -1. The gateway node's `DistSender` aggregates commit acknowledgments from all of the write operations in the `BatchRequest`, as well as any values from read operations that should be returned to the client. -1. Once all operations have successfully completed (i.e., reads have returned values and write intents have been committed), the `DistSender` tries to record the transaction's success in the transaction record (which provides a durable mechanism of tracking the transaction's state), which can cause a few situations to arise: - - It checks the timestamp cache of the range where the first write occurred to see if its timestamp got pushed forward. If it did, the transaction performs a [read refresh](transaction-layer.html#read-refreshing) to see if any values it needed have been changed. If the read refresh is successful, the transaction can commit at the pushed timestamp. If the read refresh fails, the transaction must be restarted. - - If the transaction is in an `ABORTED` state, the `DistSender` sends a response indicating as much, which ends up back at the SQL interface. - - Upon passing these checks the transaction record is either written for the first time with the `COMMITTED` state, or if it was in a `PENDING` state, it is moved to `COMMITTED`. Only at this point is the transaction considered committed. -1. The `DistSender` propagates any values that should be returned to the client (e.g., reads or the number of affected rows) to the `TxnCoordSender`, which in turn responds to the SQL interface with the value. - The `TxnCoordSender` also begins asynchronous intent cleanup by sending a request to the `DistSender` to convert all write intents it created for the transaction to fully committed values. However, this process is largely an optimization; if any operation encounters a write intent, it checks the write intent's transaction record. If the transaction record is `COMMITTED`, the operation can perform the same cleanup and convert the write intent to a fully committed value. -1. The SQL interface then responds to the client, and is now prepared to continue accepting new connections. diff --git a/src/current/v21.1/architecture/overview.md b/src/current/v21.1/architecture/overview.md deleted file mode 100644 index 4de1509f271..00000000000 --- a/src/current/v21.1/architecture/overview.md +++ /dev/null @@ -1,95 +0,0 @@ ---- -title: Architecture Overview -summary: Learn about the inner-workings of the CockroachDB architecture. -toc: true -key: cockroachdb-architecture.html ---- - -CockroachDB was designed to create the source-available database we would want to use: one that is both scalable and consistent. Developers often have questions about how we've achieved this, and this guide sets out to detail the inner workings of the `cockroach` process as a means of explanation. - -However, you definitely do not need to understand the underlying architecture to use CockroachDB. These pages give serious users and database enthusiasts a high-level framework to explain what's happening under the hood. - -## Using this guide - -This guide is broken out into pages detailing each layer of CockroachDB. We recommended reading through the layers sequentially, starting with this overview and then proceeding to the [SQL layer](sql-layer.html). - -If you're looking for a high-level understanding of CockroachDB, you can read the **Overview** section of each layer. For more technical detail—for example, if you're interested in [contributing to the project](https://cockroachlabs.atlassian.net/wiki/x/QQFdB)—you should read the **Components** sections as well. - -{{site.data.alerts.callout_info}} -This guide details how CockroachDB is built, but does not explain how to build an application using CockroachDB. For more information about how to develop applications that use CockroachDB, check out our [Developer Guide](../developer-guide-overview.html). -{{site.data.alerts.end}} - -## Goals of CockroachDB - -CockroachDB was designed to meet the following goals: - -- Make life easier for humans. This means being low-touch and highly automated for [operators](../operate-cockroachdb-kubernetes.html) and simple to reason about for [developers](../developer-guide-overview.html). -- Offer industry-leading consistency, even on massively scaled deployments. This means enabling distributed transactions, as well as removing the pain of eventual consistency issues and stale reads. -- Create an always-on database that accepts reads and writes on all nodes without generating conflicts. -- Allow flexible deployment in any environment, without tying you to any platform or vendor. -- Support familiar tools for working with relational data (i.e., SQL). - -With the confluence of these features, we hope that CockroachDB helps you build global, scalable, resilient deployments and applications. - -## Glossary - -### Terms - -It's helpful to understand a few terms before reading our architecture documentation. - -{% include {{ page.version.version }}/misc/basic-terms.md %} - -### Concepts - -CockroachDB relies heavily on the following concepts. Being familiar with them will help you understand what our architecture achieves. - -Term | Definition ------|----------- -**Consistency** | CockroachDB uses "consistency" in both the sense of [ACID semantics](https://en.wikipedia.org/wiki/Consistency_(database_systems)) and the [CAP theorem](https://en.wikipedia.org/wiki/CAP_theorem), albeit less formally than either definition. What we try to express with this term is that your data should be anomaly-free. -**Isolation** | CockroachDB provides the [`SERIALIZABLE`](https://en.wikipedia.org/wiki/Serializability) isolation level, which provides the highest level of data consistency and protects against concurrency-based attacks and bugs. -**Consensus** | When a range receives a write, a quorum of nodes containing replicas of the range acknowledge the write. This means your data is safely stored and a majority of nodes agree on the database's current state, even if some of the nodes are offline.

When a write does not achieve consensus, forward progress halts to maintain consistency within the cluster. -**Replication** | Replication involves creating and distributing copies of data, as well as ensuring that those copies remain consistent. There are two types of replication: _synchronous_ and _asynchronous_.

Synchronous replication is used by CockroachDB. Synchronous replication requires all writes to propagate to a [quorum](https://en.wikipedia.org/wiki/Quorum_%28distributed_computing%29) of copies of the data before being considered committed. This ensures the consistency of your data.

Asynchronous replication is not used by CockroachDB. It only requires a single node to receive the write to be considered committed; state is then propagated to each copy of the data after the fact. This is more or less equivalent to ["eventual consistency"](https://en.wikipedia.org/wiki/Eventual_consistency), which was popularized by NoSQL databases. This method of replication is likely to cause anomalies and loss of data. -**Transactions** | A set of operations performed on your database that satisfy the requirements of [ACID semantics](https://en.wikipedia.org/wiki/Database_transaction). This is a crucial component for a consistent system to ensure developers can trust the data in their database. For more information about how transactions work, see the [Transaction Layer](transaction-layer.html) documentation. -**Multi-Active Availability** | Our consensus-based notion of high availability that lets each node in the cluster handle reads and writes for a subset of the stored data (on a per-range basis). This is in contrast to active-passive replication, in which the active node receives 100% of request traffic, as well as active-active replication, in which all nodes accept requests but typically cannot guarantee that reads are both up-to-date and fast. - -## Overview - -CockroachDB starts running on machines with two commands: - -- [`cockroach start`](../cockroach-start.html) with a `--join` flag for all of the initial nodes in the cluster, so the process knows all of the other machines it can communicate with. -- [`cockroach init`](../cockroach-init.html) to perform a one-time initialization of the cluster. - -Once the CockroachDB cluster is initialized, developers interact with CockroachDB through a [PostgreSQL-compatible](../postgresql-compatibility.html) SQL API. Thanks to the symmetrical behavior of all nodes in a cluster, you can send [SQL requests](sql-layer.html) to any node; this makes CockroachDB easy to integrate with load balancers. - -After receiving SQL remote procedure calls (RPCs), nodes convert them into key-value (KV) operations that work with our [distributed, transactional key-value store](transaction-layer.html). - -As these RPCs start filling your cluster with data, CockroachDB starts [algorithmically distributing your data among the nodes of the cluster](distribution-layer.html), breaking the data up into 512 MiB chunks that we call ranges. Each range is replicated to at least 3 nodes by default to ensure survivability. This ensures that if any nodes go down, you still have copies of the data which can be used for: - -- Continuing to serve reads and writes. -- Consistently replicating the data to other nodes. - -If a node receives a read or write request it cannot directly serve, it finds the node that can handle the request, and communicates with that node. This means you do not need to know where in the cluster a specific portion of your data is stored; CockroachDB tracks it for you, and enables symmetric read/write behavior from each node. - -Any changes made to the data in a range rely on a [consensus algorithm](replication-layer.html) to ensure that the majority of the range's replicas agree to commit the change. This is how CockroachDB achieves the industry-leading isolation guarantees that allow it to provide your application with consistent reads and writes, regardless of which node you communicate with. - -Ultimately, data is written to and read from disk using an efficient [storage engine](storage-layer.html), which is able to keep track of the data's timestamp. This has the benefit of letting us support the SQL standard [`AS OF SYSTEM TIME`](../as-of-system-time.html) clause, letting you find historical data for a period of time. - -While the high-level overview above gives you a notion of what CockroachDB does, looking at how CockroachDB operates at each of these layers will give you much greater understanding of our architecture. - -### Layers - -At the highest level, CockroachDB converts clients' SQL statements into key-value (KV) data, which is distributed among nodes and written to disk. CockroachDB's architecture is manifested as a number of layers, each of which interacts with the layers directly above and below it as relatively opaque services. - -The following pages describe the function each layer performs, while mostly ignoring the details of other layers. This description is true to the experience of the layers themselves, which generally treat the other layers as black-box APIs. There are some interactions that occur between layers that require an understanding of each layer's function to understand the entire process. - -Layer | Order | Purpose -------|------------|-------- -[SQL](sql-layer.html) | 1 | Translate client SQL queries to KV operations. -[Transactional](transaction-layer.html) | 2 | Allow atomic changes to multiple KV entries. -[Distribution](distribution-layer.html) | 3 | Present replicated KV ranges as a single entity. -[Replication](replication-layer.html) | 4 | Consistently and synchronously replicate KV ranges across many nodes. This layer also enables consistent reads using a consensus algorithm. -[Storage](storage-layer.html) | 5 | Read and write KV data on disk. - -## What's next? - -Start by learning about what CockroachDB does with your SQL statements at the [SQL layer](sql-layer.html). diff --git a/src/current/v21.1/architecture/reads-and-writes-overview.md b/src/current/v21.1/architecture/reads-and-writes-overview.md deleted file mode 100644 index 407d2c7310a..00000000000 --- a/src/current/v21.1/architecture/reads-and-writes-overview.md +++ /dev/null @@ -1,67 +0,0 @@ ---- -title: Reads and Writes in CockroachDB -summary: Learn how reads and writes are affected by the replicated and distributed nature of data in CockroachDB. -toc: true ---- - -This page explains how reads and writes are affected by the replicated and distributed nature of data in CockroachDB. It starts by summarizing some important [CockroachDB architectural concepts](overview.html) and then guides you through a few simple read and write scenarios. - -{{site.data.alerts.callout_info}} -For a more detailed information about how transactions work in CockroachDB, see the [Transaction Layer](transaction-layer.html) documentation. -{{site.data.alerts.end}} - -## Important concepts - -{% include {{ page.version.version }}/misc/basic-terms.md %} - -As mentioned above, when a query is executed, the cluster routes the request to the leaseholder for the range containing the relevant data. If the query touches multiple ranges, the request goes to multiple leaseholders. For a read request, only the leaseholder of the relevant range retrieves the data. For a write request, the Raft consensus protocol dictates that a majority of the replicas of the relevant range must agree before the write is committed. - -Let's consider how these mechanics play out in some hypothetical queries. - -## Read scenario - -First, imagine a simple read scenario where: - -- There are 3 nodes in the cluster. -- There are 3 small tables, each fitting in a single range. -- Ranges are replicated 3 times (the default). -- A query is executed against node 2 to read from table 3. - -Perf tuning concepts - -In this case: - -1. Node 2 (the gateway node) receives the request to read from table 3. -2. The leaseholder for table 3 is on node 3, so the request is routed there. -3. Node 3 returns the data to node 2. -4. Node 2 responds to the client. - -If the query is received by the node that has the leaseholder for the relevant range, there are fewer network hops: - -Perf tuning concepts - -## Write scenario - -Now imagine a simple write scenario where a query is executed against node 3 to write to table 1: - -Perf tuning concepts - -In this case: - -1. Node 3 (the gateway node) receives the request to write to table 1. -2. The leaseholder for table 1 is on node 1, so the request is routed there. -3. The leaseholder is the same replica as the Raft leader (as is typical), so it simultaneously appends the write to its own Raft log and notifies its follower replicas on nodes 2 and 3. -4. As soon as one follower has appended the write to its Raft log (and thus a majority of replicas agree based on identical Raft logs), it notifies the leader and the write is committed to the key-values on the agreeing replicas. In this diagram, the follower on node 2 acknowledged the write, but it could just as well have been the follower on node 3. Also note that the follower not involved in the consensus agreement usually commits the write very soon after the others. -5. Node 1 returns acknowledgement of the commit to node 3. -6. Node 3 responds to the client. - -Just as in the read scenario, if the write request is received by the node that has the leaseholder and Raft leader for the relevant range, there are fewer network hops: - -Perf tuning concepts - -## Network and I/O bottlenecks - -With the above examples in mind, it's always important to consider network latency and disk I/O as potential performance bottlenecks. In summary: - -- For reads, hops between the gateway node and the leaseholder add latency. -- For writes, hops between the gateway node and the leaseholder/Raft leader, and hops between the leaseholder/Raft leader and Raft followers, add latency. In addition, since Raft log entries are persisted to disk before a write is committed, disk I/O is important. diff --git a/src/current/v21.1/architecture/replication-layer.md b/src/current/v21.1/architecture/replication-layer.md deleted file mode 100644 index a2683542462..00000000000 --- a/src/current/v21.1/architecture/replication-layer.md +++ /dev/null @@ -1,185 +0,0 @@ ---- -title: Replication Layer -summary: The replication layer of CockroachDB's architecture copies data between nodes and ensures consistency between copies. -toc: true ---- - -The replication layer of CockroachDB's architecture copies data between nodes and ensures consistency between these copies by implementing our consensus algorithm. - -{{site.data.alerts.callout_info}} -If you haven't already, we recommend reading the [Architecture Overview](overview.html). -{{site.data.alerts.end}} - -## Overview - -High availability requires that your database can tolerate nodes going offline without interrupting service to your application. This means replicating data between nodes to ensure the data remains accessible. - -Ensuring consistency with nodes offline, though, is a challenge many databases fail. To solve this problem, CockroachDB uses a consensus algorithm to require that a quorum of replicas agrees on any changes to a range before those changes are committed. Because 3 is the smallest number that can achieve quorum (i.e., 2 out of 3), CockroachDB's high availability (known as multi-active availability) requires 3 nodes. - -The number of failures that can be tolerated is equal to *(Replication factor - 1)/2*. For example, with 3x replication, one failure can be tolerated; with 5x replication, two failures, and so on. You can control the replication factor at the cluster, database, and table level using [replication zones](../configure-replication-zones.html). - -When failures happen, though, CockroachDB automatically realizes nodes have stopped responding and works to redistribute your data to continue maximizing survivability. This process also works the other way around: when new nodes join your cluster, data automatically rebalances onto it, ensuring your load is evenly distributed. - -### Interactions with other layers - -In relationship to other layers in CockroachDB, the replication layer: - -- Receives requests from and sends responses to the distribution layer. -- Writes accepted requests to the storage layer. - -## Components - -### Raft - -Raft is a consensus protocol––an algorithm which makes sure that your data is safely stored on multiple machines, and that those machines agree on the current state even if some of them are temporarily disconnected. - -Raft organizes all nodes that contain a replica of a range into a group--unsurprisingly called a Raft group. Each replica in a Raft group is either a "leader" or a "follower". The leader, which is elected by Raft and long-lived, coordinates all writes to the Raft group. It heartbeats followers periodically and keeps their logs replicated. In the absence of heartbeats, followers become candidates after randomized election timeouts and proceed to hold new leader elections. - -{% include_cached new-in.html version="v21.1" %} A third replica type is introduced, the "non-voting" replica. These replicas do not participate in Raft elections, but are useful for unlocking use cases that require low-latency multi-region reads. For more information, see [Non-voting replicas](#non-voting-replicas). - -Once a node receives a `BatchRequest` for a range it contains, it converts those KV operations into Raft commands. Those commands are proposed to the Raft group leader––which is what makes it ideal for the [leaseholder](#leases) and the Raft leader to be one in the same––and written to the Raft log. - -For a great overview of Raft, we recommend [The Secret Lives of Data](http://thesecretlivesofdata.com/raft/). - -#### Raft logs - -When writes receive a quorum, and are committed by the Raft group leader, they're appended to the Raft log. This provides an ordered set of commands that the replicas agreed on and is essentially the source of truth for consistent replication. - -Because this log is treated as serializable, it can be replayed to bring a node from a past state to its current state. This log also lets nodes that temporarily went offline to be "caught up" to the current state without needing to receive a copy of the existing data in the form of a snapshot. - -#### Non-voting replicas - -In versions prior to v21.1, CockroachDB only supported _voting_ replicas: that is, [replicas](overview.html#glossary) that participate as voters in the [Raft consensus protocol](#raft). However, the need for all replicas to participate in the consensus algorithm meant that increasing the [replication factor](../configure-replication-zones.html#num_replicas) came at a cost of increased write latency, since the additional replicas needed to participate in Raft [quorum](overview.html#architecture-overview-consensus). - -{% include_cached new-in.html version="v21.1" %} In order to provide [better support for multi-region clusters](../multiregion-overview.html), (including the features that make [fast multi-region reads](../multiregion-overview.html#global-tables) and [surviving region failures](../multiregion-overview.html#surviving-region-failures) possible), a new type of replica is introduced: the _non-voting_ replica. - -Non-voting replicas follow the [Raft log](#raft-logs) (and are thus able to serve [follower reads](../follower-reads.html)), but do not participate in quorum. They have almost no impact on write latencies. - -They are also sometimes referred to as [_read-only_ replicas](https://cloud.google.com/spanner/docs/replication#read-only), since they only serve reads, but do not participate in quorum (and thus do not incur the associated latency costs). - -Non-voting replicas can be configured via [zone configurations through `num_voters` and `num_replicas`](../configure-replication-zones.html#num_voters). When `num_voters` is configured to be less than `num_replicas`, the difference dictates the number of non-voting replicas. However, most users should control non-voting replica placement with the high-level [multi-region SQL features](../multiregion-overview.html) instead. - -### Snapshots - -Each replica can be "snapshotted", which copies all of its data as of a specific timestamp (available because of [MVCC](storage-layer.html#mvcc)). This snapshot can be sent to other nodes during a rebalance event to expedite replication. - -After loading the snapshot, the node gets up to date by replaying all actions from the Raft group's log that have occurred since the snapshot was taken. - -### Leases - -A single node in the Raft group acts as the leaseholder, which is the only node that can serve reads or propose writes to the Raft group leader (both actions are received as `BatchRequests` from [`DistSender`](distribution-layer.html#distsender)). - -CockroachDB attempts to elect a leaseholder who is also the Raft group leader, which can also optimize the speed of writes. - -If there is no leaseholder, any node receiving a request will attempt to become the leaseholder for the range. To prevent two nodes from acquiring the lease, the requester includes a copy of the last valid lease it had; if another node became the leaseholder, its request is ignored. - -When serving [strongly-consistent (aka "non-stale") reads](transaction-layer.html#reading), leaseholders bypass Raft; for the leaseholder's writes to have been committed in the first place, they must have already achieved consensus, so a second consensus on the same data is unnecessary. This has the benefit of not incurring latency from networking round trips required by Raft and greatly increases the speed of reads (without sacrificing consistency). - -#### Co-location with Raft leadership - -The range lease is completely separate from Raft leadership, and so without further efforts, Raft leadership and the Range lease might not be held by the same replica. However, we can optimize query performance by making the same node both Raft leader and the leaseholder; it reduces network round trips if the leaseholder receiving the requests can simply propose the Raft commands to itself, rather than communicating them to another node. - -To achieve this, each lease renewal or transfer also attempts to collocate them. In practice, that means that the mismatch is rare and self-corrects quickly. - -#### Epoch-based leases (table data) - -To manage leases for table data, CockroachDB implements a notion of "epochs," which are defined as the period between a node joining a cluster and a node disconnecting from a cluster. To extend its leases, each node must periodically update its liveness record, which is stored on a system range key. When a node disconnects, it stops updating the liveness record, and the epoch is considered changed. This causes the node to [lose all of its leases](#how-leases-are-transferred-from-a-dead-node) a few seconds later when the liveness record expires. - -Because leases do not expire until a node disconnects from a cluster, leaseholders do not have to individually renew their own leases. Tying lease lifetimes to node liveness in this way lets us eliminate a substantial amount of traffic and Raft processing we would otherwise incur, while still tracking leases for every range. - -#### Expiration-based leases (meta and system ranges) - -A table's meta and system ranges (detailed in the [distribution layer](distribution-layer.html#meta-ranges)) are treated as normal key-value data, and therefore have leases just like table data. - -However, unlike table data, system ranges cannot use epoch-based leases because that would create a circular dependency: system ranges are already being used to implement epoch-based leases for table data. Therefore, system ranges use expiration-based leases instead. Expiration-based leases expire at a particular timestamp (typically after a few seconds). However, as long as a node continues proposing Raft commands, it continues to extend the expiration of its leases. If it doesn't, the next node containing a replica of the range that tries to read from or write to the range will become the leaseholder. - -#### How leases are transferred from a dead node - -When the cluster needs to access a range on a leaseholder node that is dead, that range's lease must be transferred to a healthy node. This process is as follows: - -1. The dead node's liveness record, which is stored in a system range, has an expiration time of 9 seconds, and is heartbeated every 4.5 seconds. When the node dies, the amount of time the cluster has to wait for the record to expire varies, but on average is 6.75 seconds. -1. A healthy node attempts to acquire the lease. This is rejected because lease acquisition can only happen on the Raft leader, which the healthy node is not (yet). Therefore, a Raft election must be held. -1. The rejected attempt at lease acquisition [unquiesces](../ui-replication-dashboard.html#replica-quiescence) ("wakes up") the range associated with the lease. -1. What happens next depends on whether the lease is on [table data](#epoch-based-leases-table-data) or [meta ranges or system ranges](#expiration-based-leases-meta-and-system-ranges): - - If the lease is on [meta or system ranges](#expiration-based-leases-meta-and-system-ranges), the node that unquiesced the range checks if the Raft leader is alive according to the liveness record. If the leader is not alive, it kicks off a campaign to try and win Raft leadership so it can become the leaseholder. - - If the lease is on [table data](#epoch-based-leases-table-data), the "is the leader alive?" check described above is skipped and an election is called immediately. The check is skipped since it would introduce a circular dependency on the liveness record used for table data, which is itself stored in a system range. -1. The Raft election is held and a new leader is chosen from among the healthy nodes. -1. The lease acquisition can now be processed by the newly elected Raft leader. - -This process should take no more than 9 seconds for liveness expiration plus the cost of 2 network roundtrips: 1 for Raft leader election, and 1 for lease acquisition. - -Finally, note that the process described above is lazily initiated: it only occurs when a new request comes in for the range associated with the lease. - -#### Leaseholder rebalancing - -Because CockroachDB serves reads from a range's leaseholder, it benefits your cluster's performance if the replica closest to the primary geographic source of traffic holds the lease. However, as traffic to your cluster shifts throughout the course of the day, you might want to dynamically shift which nodes hold leases. - -{{site.data.alerts.callout_info}} - -This feature is also called [Follow-the-Workload](../topology-follow-the-workload.html) in our documentation. - -{{site.data.alerts.end}} - -Periodically (every 10 minutes by default in large clusters, but more frequently in small clusters), each leaseholder considers whether it should transfer the lease to another replica by considering the following inputs: - -- Number of requests from each locality -- Number of leases on each node -- Latency between localities - -##### Intra-locality - -If all the replicas are in the same locality, the decision is made entirely on the basis of the number of leases on each node that contains a replica, trying to achieve a roughly equitable distribution of leases across all of them. This means the distribution isn't perfectly equal; it intentionally tolerates small deviations between nodes to prevent thrashing (i.e., excessive adjustments trying to reach an equilibrium). - -##### Inter-locality - -If replicas are in different localities, CockroachDB attempts to calculate which replica would make the best leaseholder, i.e., provide the lowest latency. - -To enable dynamic leaseholder rebalancing, a range's current leaseholder tracks how many requests it receives from each locality as an exponentially weighted moving average. This calculation results in the locality that has recently requested the range most often being assigned the greatest weight. If another locality then begins requesting the range very frequently, this calculation would shift to assign the second region the greatest weight. - -When checking for leaseholder rebalancing opportunities, the leaseholder correlates each requesting locality's weight (i.e., the proportion of recent requests) to the locality of each replica by checking how similar the localities are. For example, if the leaseholder received requests from gateway nodes in locality `country=us,region=central`, CockroachDB would assign the following weights to replicas in the following localities: - -Replica locality | Replica rebalancing weight ------------------|------------------- -`country=us,region=central` | 100% because it is an exact match -`country=us,region=east` | 50% because only the first locality matches -`country=aus,region=central` | 0% because the first locality does not match - -The leaseholder then evaluates its own weight and latency versus the other replicas to determine an adjustment factor. The greater the disparity between weights and the larger the latency between localities, the more CockroachDB favors the node from the locality with the larger weight. - -When checking for leaseholder rebalancing opportunities, the current leaseholder evaluates each replica's rebalancing weight and adjustment factor for the localities with the greatest weights. If moving the leaseholder is both beneficial and viable, the current leaseholder will transfer the lease to the best replica. - -##### Controlling leaseholder rebalancing - -You can control leaseholder rebalancing through the `kv.allocator.load_based_lease_rebalancing.enabled` and `kv.allocator.lease_rebalancing_aggressiveness` [cluster settings](../cluster-settings.html). Note that depending on the needs of your deployment, you can exercise additional control over the location of leases and replicas by [configuring replication zones](../configure-replication-zones.html). - -### Membership changes: rebalance/repair - -Whenever there are changes to a cluster's number of nodes, the members of Raft groups change and, to ensure optimal survivability and performance, replicas need to be rebalanced. What that looks like varies depending on whether the membership change is nodes being added or going offline. - -- **Nodes added**: The new node communicates information about itself to other nodes, indicating that it has space available. The cluster then rebalances some replicas onto the new node. - -- **Nodes going offline**: If a member of a Raft group ceases to respond, after 5 minutes, the cluster begins to rebalance by replicating the data the downed node held onto other nodes. - -Rebalancing is achieved by using a snapshot of a replica from the leaseholder, and then sending the data to another node over [gRPC](distribution-layer.html#grpc). After the transfer has been completed, the node with the new replica joins that range's Raft group; it then detects that its latest timestamp is behind the most recent entries in the Raft log and it replays all of the actions in the Raft log on itself. - -#### Load-based replica rebalancing - -In addition to the rebalancing that occurs when nodes join or leave a cluster, replicas are also rebalanced automatically based on the relative load across the nodes within a cluster. For more information, see the `kv.allocator.load_based_rebalancing` and `kv.allocator.qps_rebalance_threshold` [cluster settings](../cluster-settings.html). Note that depending on the needs of your deployment, you can exercise additional control over the location of leases and replicas by [configuring replication zones](../configure-replication-zones.html). - -## Interactions with other layers - -### Replication and distribution layers - -The replication layer receives requests from its and other nodes' `DistSender`. If this node is the leaseholder for the range, it accepts the requests; if it isn't, it returns an error with a pointer to which node it believes *is* the leaseholder. These KV requests are then turned into Raft commands. - -The replication layer sends `BatchResponses` back to the distribution layer's `DistSender`. - -### Replication and storage layers - -Committed Raft commands are written to the Raft log and ultimately stored on disk through the storage layer. - -The leaseholder serves reads from the storage layer. - -## What's next? - -Learn how CockroachDB reads and writes data from disk in the [storage layer](storage-layer.html). diff --git a/src/current/v21.1/architecture/sql-layer.md b/src/current/v21.1/architecture/sql-layer.md deleted file mode 100644 index bb545612477..00000000000 --- a/src/current/v21.1/architecture/sql-layer.md +++ /dev/null @@ -1,156 +0,0 @@ ---- -title: SQL Layer -summary: The SQL layer of CockroachDB's architecture exposes its SQL API to developers and converts SQL statements into key-value operations. -toc: true ---- - -The SQL layer of CockroachDB's architecture exposes a SQL API to developers and converts high-level [SQL statements](../sql-statements.html) into low-level read and write requests to the underlying key-value store, which are passed to the [transaction Layer](transaction-layer.html). - -It consists of the following sublayers: - -- [SQL API](#sql-api), which forms the user interface. -- [Parser](#parsing), which converts SQL text into an abstract syntax tree (AST). -- [Cost-based optimizer](#logical-planning), which converts the AST into an optimized logical query plan. -- [Physical planner](#physical-planning), which converts the logical query plan into a physical query plan for execution by one or more nodes in the cluster. -- [SQL execution engine](#query-execution), which executes the physical plan by making read and write requests to the underlying key-value store. - -{{site.data.alerts.callout_info}} -If you haven't already, we recommend reading the [Architecture Overview](overview.html). -{{site.data.alerts.end}} - -## Overview - -Once CockroachDB has been [deployed](../../cockroachcloud/quickstart.html), developers need only a [connection string](../connection-parameters.html) to the cluster, and they can start working with SQL statements. - -Because each node in a CockroachDB cluster behaves symmetrically, developers can send requests to any node (which means CockroachDB works well with load balancers). Whichever node receives the request acts as the "gateway node," which processes the request and responds to the client. - -Requests to the cluster arrive as SQL statements, but data is ultimately written to and read from the [storage layer](storage-layer.html) as key-value (KV) pairs. To handle this, the SQL layer converts SQL statements into a plan of KV operations, which it then passes along to the [transaction layer](transaction-layer.html). - -### Interactions with other layers - -In relationship to other layers in CockroachDB, the SQL layer: - -- Receives requests from the outside world via its [SQL API](#sql-api). -- Converts SQL statements into low-level KV operations, which it sends as requests to the [transaction layer](transaction-layer.html). - -## Components - -### Relational structure - -Developers experience data stored in CockroachDB as a relational structure comprised of rows and columns. Sets of rows and columns are further organized into [tables](../show-tables.html). Collections of tables are then organized into [databases](../show-databases.html). A CockroachDB cluster can contain many databases. - -CockroachDB provides typical relational features like [constraints](../constraints.html) (e.g., [foreign keys](../foreign-key.html)). These features mean that application developers can trust that the database will ensure consistent structuring of the application's data; data validation doesn't need to be built into the application logic separately. - -### SQL API - -CockroachDB implements most of the ANSI SQL standard to manifest its relational structure. For a complete list of the SQL features CockroachDB supports, see [SQL Feature Support](../sql-feature-support.html). - -Importantly, through the SQL API developers have access to ACID-semantic [transactions](../transactions.html) like they would through any SQL database (using [`BEGIN`](../begin-transaction.html), [`COMMIT`](../commit-transaction.html), etc.). - -### PostgreSQL wire protocol - -SQL queries reach your cluster through the PostgreSQL wire protocol. This makes connecting your application to the cluster simple by supporting many PostgreSQL-compatible [drivers and ORMs](../install-client-drivers.html). - -### SQL parser, planner, executor - -When a node in a CockroachDB cluster receives a SQL request from a client, it [parses the statement](#parsing) and [creates an optimized logical query plan](#logical-planning) that is further translated into a [physical query plan](#physical-planning). Finally, it [executes the physical plan](#query-execution). - -#### Parsing - -SQL queries are parsed against our `yacc` file (which describes our supported syntax), and the SQL version of each query is converted into an [abstract syntax tree](https://en.wikipedia.org/wiki/Abstract_syntax_tree) (AST). - -#### Logical planning - -During the *logical planning* phase, the AST is transformed into a query plan in the following steps: - -1. The AST is transformed into a high-level logical query plan. During this transformation, CockroachDB also performs [semantic analysis](https://en.wikipedia.org/wiki/Semantic_analysis_(compilers)), which includes operations like: - - Checking whether the query is a valid statement in the SQL language. - - Resolving names, such as the names of tables or variables to their values. - - Eliminating unneeded intermediate computations, e.g., by replacing `0.6 + 0.4` with `1.0`. This is also known as [constant folding](https://en.wikipedia.org/wiki/Constant_folding). - - Finalizing which data types to use for intermediate results, e.g., when a query contains one or more [subqueries](../subqueries.html). - -2. The logical plan is *simplified* using a series of transformations that are always valid. For example, `a BETWEEN b AND c` may be converted to `a >= b AND a <= c`. - -3. The logical plan is *optimized* using a [search algorithm](../cost-based-optimizer.html#how-is-cost-calculated) that evaluates many possible ways to execute a query and selects an execution plan with the least costs. - -The result of the final step above is an optimized logical plan. To view the logical plan generated by the [cost-based optimizer](../cost-based-optimizer.html), use the [`EXPLAIN (OPT)`](../explain.html) statement. - -#### Physical planning - -The physical planning phase decides which nodes will participate in -the execution of the query, based on range locality information. This -is where CockroachDB decides to distribute a query to perform some -computations close to where the data is stored. - -More concretely, the physical planning phase transforms the optimized logical plan generated during [logical planning](#logical-planning) into a [directed acyclic graph](https://en.wikipedia.org/wiki/Directed_acyclic_graph) (DAG) of physical *SQL operators*. These operators can be viewed by running the [`EXPLAIN(DISTSQL)`](../explain.html) statement. - -Because the [distribution layer](distribution-layer.html) presents the abstraction of a single key space, the SQL layer can perform read and write operations for any range on any node. This allows the SQL operators to behave identically whether planned in gateway or distributed mode. - -The decision about whether to distribute a query across multiple nodes is made by a heuristic that estimates the quantity of data that would need to be sent over the network. Queries that only need a small number of rows are executed on the gateway node. Other queries are distributed across multiple nodes. - -For example, when a query is distributed, the physical planning phase splits the scan operations from the logical plan into multiple physical _TableReader_ operators, one for each node containing a range read by the scan. The remaining logical operations (which may perform filters, joins, and aggregations) are then scheduled on the same nodes as the TableReaders. This results in computations being performed as close to the physical data as possible. - -#### Query execution - -Components of the [physical plan](#physical-planning) are sent to one or more nodes for execution. On each node, CockroachDB spawns a *logical processor* to compute a part of the query. Logical processors inside or across nodes communicate with each other over a *logical flow* of data. The combined results of the query are sent back to the first node where the query was received, to be sent further to the SQL client. - -Each processor uses an encoded form for the scalar values manipulated by the query. This is a binary form which is different from that used in SQL. So the values listed in the SQL query must be encoded, and the data communicated between logical processors, and read from disk, must be decoded before it is sent back to the SQL client. - -#### Vectorized query execution - -If [vectorized execution](../vectorized-execution.html) is enabled, the physical plan is sent to nodes to be processed by the vectorized execution engine. - -Upon receiving the physical plan, the vectorized engine reads batches of table data [from disk](storage-layer.html) and converts the data from row format to columnar format. These batches of column data are stored in memory so the engine can access them quickly during execution. - -The vectorized engine uses specialized, precompiled functions that quickly iterate over the type-specific arrays of column data. The columnar output from the functions is stored in memory as the engine processes each column of data. - -After processing all columns of data in the input buffer, the engine converts the columnar output back to row format, and then returns the processed rows to the SQL interface. After a batch of table data has been fully processed, the engine reads the following batch of table data for processing, until the query has been executed. - -### Encoding - -Though SQL queries are written in parsable strings, lower layers of CockroachDB deal primarily in bytes. This means at the SQL layer, in query execution, CockroachDB must convert row data from their SQL representation as strings into bytes, and convert bytes returned from lower layers into SQL data that can be passed back to the client. - -It's also important––for indexed columns––that this byte encoding preserve the same sort order as the data type it represents. This is because of the way CockroachDB ultimately stores data in a sorted key-value map; storing bytes in the same order as the data it represents lets us efficiently scan KV data. - -However, for non-indexed columns (e.g., non-`PRIMARY KEY` columns), CockroachDB instead uses an encoding (known as "value encoding") which consumes less space but does not preserve ordering. - -You can find more exhaustive detail in the [Encoding Tech Note](https://github.com/cockroachdb/cockroach/blob/master/docs/tech-notes/encoding.md). - -### DistSQL - -Because CockroachDB is a distributed database, we've developed a Distributed SQL (DistSQL) optimization tool for some queries, which can dramatically speed up queries that involve many ranges. Though DistSQL's architecture is worthy of its own documentation, this cursory explanation can provide some insight into how it works. - -In non-distributed queries, the coordinating node receives all of the rows that match its query, and then performs any computations on the entire data set. - -However, for DistSQL-compatible queries, each node does computations on the rows it contains, and then sends the results (instead of the entire rows) to the coordinating node. The coordinating node then aggregates the results from each node, and finally returns a single response to the client. - -This dramatically reduces the amount of data brought to the coordinating node, and leverages the well-proven concept of parallel computing, ultimately reducing the time it takes for complex queries to complete. In addition, this processes data on the node that already stores it, which lets CockroachDB handle row-sets that are larger than an individual node's storage. - -To run SQL statements in a distributed fashion, we introduce a couple of concepts: - -- **Logical plan**: Similar to the AST/`planNode` tree described above, it represents the abstract (non-distributed) data flow through computation stages. -- **Physical plan**: A physical plan is conceptually a mapping of the logical plan nodes to physical machines running `cockroach`. Logical plan nodes are replicated and specialized depending on the cluster topology. Like `planNodes` above, these components of the physical plan are scheduled and run on the cluster. - -You can find much greater detail in the [DistSQL RFC](https://github.com/cockroachdb/cockroach/blob/master/docs/RFCS/20160421_distributed_sql.md). - -## Schema changes - -CockroachDB performs schema changes, such as the [addition of columns](../add-column.html) or [secondary indexes](../create-index.html), using a protocol that allows tables to remain online (i.e., able to serve reads and writes) during the schema change. This protocol allows different nodes in the cluster to asynchronously transition to a new table schema at different times. - -The schema change protocol decomposes each schema change into a sequence of incremental changes that will achieve the desired effect. - -For example, the addition of a secondary index requires two intermediate schema versions between the start and end versions to ensure that the index is being updated on writes across the entire cluster before it becomes available for reads. To ensure that the database will remain in a consistent state throughout the schema change, we enforce the invariant that there are at most two successive versions of this schema used in the cluster at all times. - -This approach is based on the paper [_Online, Asynchronous Schema Change in F1_](https://research.google/pubs/pub41376/). - -For more information, including examples and limitations, see [Online Schema Changes](../online-schema-changes.html). - -## Technical interactions with other layers - -### SQL and transaction layer - -KV operations from executed `planNodes` are sent to the transaction layer. - -## What's next? - -Learn how CockroachDB handles concurrent requests in the [transaction layer](transaction-layer.html). diff --git a/src/current/v21.1/architecture/storage-layer.md b/src/current/v21.1/architecture/storage-layer.md deleted file mode 100644 index 059b31a8fd3..00000000000 --- a/src/current/v21.1/architecture/storage-layer.md +++ /dev/null @@ -1,96 +0,0 @@ ---- -title: Storage Layer -summary: The storage layer of CockroachDB's architecture reads and writes data to disk. -toc: true ---- - -The storage layer of CockroachDB's architecture reads and writes data to disk. - -{{site.data.alerts.callout_info}} -If you haven't already, we recommend reading the [Architecture Overview](overview.html). -{{site.data.alerts.end}} - - -## Overview - -Each CockroachDB node contains at least one `store`, specified when the node starts, which is where the `cockroach` process reads and writes its data on disk. - -This data is stored as key-value pairs on disk using the storage engine, which is treated primarily as a black-box API. - - By default, [CockroachDB uses the Pebble storage engine](../cockroach-start.html#storage-engine), with RocksDB available as an option. Pebble is intended to be bidirectionally compatible with the RocksDB on-disk format, but differs in that it: - -- Is written in Go and implements a subset of RocksDB's large feature set. -- Contains optimizations that benefit CockroachDB. - -Internally, each store contains two instances of the storage engine: - -- One for storing temporary distributed SQL data -- One for all other data on the node - -In addition, there is also a block cache shared amongst all of the stores in a node. These stores in turn have a collection of range replicas. More than one replica for a range will never be placed on the same store or even the same node. - -### Interactions with other layers - -In relationship to other layers in CockroachDB, the storage layer: - -- Serves successful reads and writes from the replication layer. - -## Components - -### Pebble - - CockroachDB uses [Pebble by default](../cockroach-start.html#storage-engine)––an embedded key-value store with a RocksDB-compatible API developed by Cockroach Labs––to read and write data to disk. You can find more information about it on the [Pebble GitHub page](https://github.com/cockroachdb/pebble) or in the blog post [Introducing Pebble: A RocksDB Inspired Key-Value Store Written in Go](https://www.cockroachlabs.com/blog/pebble-rocksdb-kv-store/). - -Pebble integrates well with CockroachDB for a number of reasons: - -- It is a key-value store, which makes mapping to our key-value layer simple -- It provides atomic write batches and snapshots, which give us a subset of transactions -- It is developed by Cockroach Labs engineers -- It contains optimizations that are not in RocksDB, that are inspired by how CockroachDB uses the storage engine. For an example of such an optimization, see the blog post [Faster Bulk-Data Loading in CockroachDB](https://www.cockroachlabs.com/blog/bulk-data-import/). - -Efficient storage for the keys is guaranteed by the underlying Pebble engine by means of prefix compression. - -### MVCC - -CockroachDB relies heavily on [multi-version concurrency control (MVCC)](https://en.wikipedia.org/wiki/Multiversion_concurrency_control) to process concurrent requests and guarantee consistency. Much of this work is done by using [hybrid logical clock (HLC) timestamps](transaction-layer.html#time-and-hybrid-logical-clocks) to differentiate between versions of data, track commit timestamps, and identify a value's garbage collection expiration. All of this MVCC data is then stored in Pebble. - -Despite being implemented in the storage layer, MVCC values are widely used to enforce consistency in the [transaction layer](transaction-layer.html). For example, CockroachDB maintains a [timestamp cache](transaction-layer.html#timestamp-cache), which stores the timestamp of the last time that the key was read. If a write operation occurs at a lower timestamp than the largest value in the read timestamp cache, it signifies there’s a potential anomaly and the transaction must be restarted at a later timestamp. - -#### Time-travel - -As described in the [SQL:2011 standard](https://en.wikipedia.org/wiki/SQL:2011#Temporal_support), CockroachDB supports time travel queries (enabled by MVCC). - -To do this, all of the schema information also has an MVCC-like model behind it. This lets you perform `SELECT...AS OF SYSTEM TIME`, and CockroachDB uses the schema information as of that time to formulate the queries. - -Using these tools, you can get consistent data from your database as far back as your garbage collection period. - -### Garbage collection - -CockroachDB regularly garbage collects MVCC values to reduce the size of data stored on disk. To do this, we compact old MVCC values when there is a newer MVCC value with a timestamp that's older than the garbage collection period. The garbage collection period can be set at the cluster, database, or table level by configuring the [`gc.ttlseconds` replication zone variable](../configure-replication-zones.html#gc-ttlseconds). For more information about replication zones, see [Configure Replication Zones](../configure-replication-zones.html). - -#### Protected timestamps - -Garbage collection can only run on MVCC values which are not covered by a *protected timestamp*. The protected timestamp subsystem exists to ensure the safety of operations that rely on historical data, such as: - -- [Imports](../import.html), including [`IMPORT INTO`](../import-into.html) -- [Backups](../backup.html) -- [Streaming data out of CockroachDB using changefeeds](../stream-data-out-of-cockroachdb-using-changefeeds.html) -- [Online schema changes](../online-schema-changes.html) - -Protected timestamps ensure the safety of historical data while also enabling shorter [GC TTLs](../configure-replication-zones.html#gc-ttlseconds). A shorter GC TTL means that fewer previous MVCC values are kept around. This can help lower query execution costs for workloads which update rows frequently throughout the day, since [the SQL layer](sql-layer.html) has to scan over previous MVCC values to find the current value of a row. - -##### How protected timestamps work - -Protected timestamps work by creating *protection records*, which are stored in an internal system table. When a long-running job such as a backup wants to protect data at a certain timestamp from being garbage collected, it creates a protection record associated with that data and timestamp. - -Upon successful creation of a protection record, the MVCC values for the specified data at timestamps less than or equal to the protected timestamp will not be garbage collected. When the job that created the protection record finishes its work, it removes the record, allowing the garbage collector to run on the formerly protected values. - -## Interactions with other layers - -### Storage and replication layers - -The storage layer commits writes from the Raft log to disk, as well as returns requested data (i.e., reads) to the replication layer. - -## What's next? - -Now that you've learned about our architecture, [start a local cluster](../install-cockroachdb.html) and start [building an app with CockroachDB](../example-apps.html). diff --git a/src/current/v21.1/architecture/transaction-layer.md b/src/current/v21.1/architecture/transaction-layer.md deleted file mode 100644 index 29edd014215..00000000000 --- a/src/current/v21.1/architecture/transaction-layer.md +++ /dev/null @@ -1,435 +0,0 @@ ---- -title: Transaction Layer -summary: The transaction layer of CockroachDB's architecture implements support for ACID transactions by coordinating concurrent operations. -toc: true ---- - -The transaction layer of CockroachDB's architecture implements support for ACID transactions by coordinating concurrent operations. - -{{site.data.alerts.callout_info}} -If you haven't already, we recommend reading the [Architecture Overview](overview.html). -{{site.data.alerts.end}} - -## Overview - -Above all else, CockroachDB believes consistency is the most important feature of a database––without it, developers cannot build reliable tools, and businesses suffer from potentially subtle and hard to detect anomalies. - -To provide consistency, CockroachDB implements full support for ACID transaction semantics in the transaction layer. However, it's important to realize that *all* statements are handled as transactions, including single statements––this is sometimes referred to as "autocommit mode" because it behaves as if every statement is followed by a `COMMIT`. - -For code samples of using transactions in CockroachDB, see our documentation on [transactions](../transactions.html#sql-statements). - -Because CockroachDB enables transactions that can span your entire cluster (including cross-range and cross-table transactions), it achieves correctness using a distributed, atomic commit protocol called [Parallel Commits](#parallel-commits). - -### Writes and reads (phase 1) - -#### Writing - -When the transaction layer executes write operations, it doesn't directly write values to disk. Instead, it creates several things that help it mediate a distributed transaction: - -- **Locks** for all of a transaction’s writes, which represent a provisional, uncommitted state. CockroachDB has several different types of locking: - - - **Unreplicated Locks** are stored in an in-memory, per-node lock table by the [concurrency control](#concurrency-control) machinery. These locks are not replicated via [Raft](replication-layer.html#raft). - - - **Replicated Locks** (also known as [write intents](#write-intents)) are replicated via [Raft](replication-layer.html#raft), and act as a combination of a provisional value and an exclusive lock. They are essentially the same as standard [multi-version concurrency control (MVCC)](storage-layer.html#mvcc) values but also contain a pointer to the [transaction record](#transaction-records) stored on the cluster. - -- A **transaction record** stored in the range where the first write occurs, which includes the transaction's current state (which is either `PENDING`, `STAGING`, `COMMITTED`, or `ABORTED`). - -As write intents are created, CockroachDB checks for newer committed values. If newer committed values exist, the transaction may be restarted. If existing write intents for the same keys exist, it is resolved as a [transaction conflict](#transaction-conflicts). - -If transactions fail for other reasons, such as failing to pass a SQL constraint, the transaction is aborted. - -#### Reading - -If the transaction has not been aborted, the transaction layer begins executing read operations. If a read only encounters standard MVCC values, everything is fine. However, if it encounters any write intents, the operation must be resolved as a [transaction conflict](#transaction-conflicts). - -CockroachDB provides the following types of reads: - -- Strongly-consistent (aka "non-stale") reads: These are the default and most common type of read. These reads go through the [leaseholder](replication-layer.html#leases) and see all writes performed by writers that committed before the reading transaction started. They always return data that is correct and up-to-date. -- Stale reads: These are useful in situations where you can afford to read data that is slightly stale in exchange for faster reads. They can only be used in read-only transactions that use the [`AS OF SYSTEM TIME`](../as-of-system-time.html) clause. They do not need to go through the leaseholder, since they ensure consistency by reading from a local replica at a timestamp that is never higher than the [closed timestamp](#closed-timestamps). For more information about how to use stale reads from SQL, see [Follower Reads](../follower-reads.html). - -### Commits (phase 2) - -CockroachDB checks the running transaction's record to see if it's been `ABORTED`; if it has, it restarts the transaction. - -In the common case, it sets the transaction record's state to `STAGING`, and checks the transaction's pending write intents to see if they have succeeded (i.e., been replicated across the cluster). - -If the transaction passes these checks, CockroachDB responds with the transaction's success to the client, and moves on to the cleanup phase. At this point, the transaction is committed, and the client is free to begin sending more requests to the cluster. - -For a more detailed tutorial of the commit protocol, see [Parallel Commits](#parallel-commits). - -### Cleanup (asynchronous phase 3) - -After the transaction has been committed, it should be marked as such, and all of the write intents should be resolved. To do this, the coordinating node––which kept a track of all of the keys it wrote––reaches out and: - -- Moves the state of the transaction record from `STAGING` to `COMMITTED`. -- Resolves the transaction's write intents to MVCC values by removing the element that points it to the transaction record. -- Deletes the write intents. - -This is simply an optimization, though. If operations in the future encounter write intents, they always check their transaction records––any operation can resolve or remove write intents by checking the transaction record's status. - -### Interactions with other layers - -In relationship to other layers in CockroachDB, the transaction layer: - -- Receives KV operations from the SQL layer. -- Controls the flow of KV operations sent to the distribution layer. - -## Technical details and components - -### Time and hybrid logical clocks - -In distributed systems, ordering and causality are difficult problems to solve. While it's possible to rely entirely on Raft consensus to maintain serializability, it would be inefficient for reading data. To optimize performance of reads, CockroachDB implements hybrid-logical clocks (HLC) which are composed of a physical component (always close to local wall time) and a logical component (used to distinguish between events with the same physical component). This means that HLC time is always greater than or equal to the wall time. You can find more detail in the [HLC paper](http://www.cse.buffalo.edu/tech-reports/2014-04.pdf). - -In terms of transactions, the gateway node picks a timestamp for the transaction using HLC time. Whenever a transaction's timestamp is mentioned, it's an HLC value. This timestamp is used to both track versions of values (through [multi-version concurrency control](storage-layer.html#mvcc)), as well as provide our transactional isolation guarantees. - -When nodes send requests to other nodes, they include the timestamp generated by their local HLCs (which includes both physical and logical components). When nodes receive requests, they inform their local HLC of the timestamp supplied with the event by the sender. This is useful in guaranteeing that all data read/written on a node is at a timestamp less than the next HLC time. - -This then lets the node primarily responsible for the range (i.e., the leaseholder) serve reads for data it stores by ensuring the transaction reading the data is at an HLC time greater than the MVCC value it's reading (i.e., the read always happens "after" the write). - -#### Max clock offset enforcement - -CockroachDB requires moderate levels of clock synchronization to preserve data consistency. For this reason, when a node detects that its clock is out of sync with at least half of the other nodes in the cluster by 80% of the [maximum offset allowed](../cockroach-start.html#flags-max-offset), **it crashes immediately**. - -While [serializable consistency](https://en.wikipedia.org/wiki/Serializability) is maintained regardless of clock skew, skew outside the configured clock offset bounds can result in violations of single-key linearizability between causally dependent transactions. It's therefore important to prevent clocks from drifting too far by running [NTP](http://www.ntp.org/) or other clock synchronization software on each node. - -For more detail about the risks that large clock offsets can cause, see [What happens when node clocks are not properly synchronized?](../operational-faqs.html#what-happens-when-node-clocks-are-not-properly-synchronized) - -### Timestamp cache - -As part of providing serializability, whenever an operation reads a value, we store the operation's timestamp in a timestamp cache, which shows the high-water mark for values being read. - -The timestamp cache is a data structure used to store information about the reads performed by [leaseholders](replication-layer.html#leases). This is used to ensure that once some transaction *t1* reads a row, another transaction *t2* that comes along and tries to write to that row will be ordered after *t1*, thus ensuring a serial order of transactions, aka serializability. - -Whenever a write occurs, its timestamp is checked against the timestamp cache. If the timestamp is less than the timestamp cache's latest value, we attempt to push the timestamp for its transaction forward to a later time. Pushing the timestamp might cause the transaction to restart in the second phase of the transaction (see [read refreshing](#read-refreshing)). - -### Closed timestamps - -Each CockroachDB range tracks a property called its _closed timestamp_, which means that no new writes can ever be introduced at or below that timestamp. The closed timestamp is advanced continuously on the leaseholder, and lags the current time by some target interval. As the closed timestamp is advanced, notifications are sent to each follower. If a range receives a write at a timestamp less than or equal to its closed timestamp, the write is forced to change its timestamp, which might result in a transaction retry error (see [read refreshing](#read-refreshing)). - -In other words, a closed timestamp is a promise by the range's [leaseholder](replication-layer.html#leases) to its follower replicas that it will not accept writes below that timestamp. Generally speaking, the leaseholder continuously closes timestamps a few seconds in the past. - -The closed timestamps subsystem works by propagating information from leaseholders to followers by piggybacking closed timestamps onto Raft commands such that the replication stream is synchronized with timestamp closing. This means that a follower replica can start serving reads with timestamps at or below the closed timestamp as soon as it has applied all of the Raft commands up to the position in the [Raft log](replication-layer.html#raft-logs) specified by the leaseholder. - -Once the follower replica has applied the abovementioned Raft commands, it has all the data necessary to serve reads with timestamps less than or equal to the closed timestamp. - -Note that closed timestamps are valid even if the leaseholder changes, since they are preserved across [lease transfers](replication-layer.html#epoch-based-leases-table-data). Once a lease transfer occurs, the new leaseholder will not break the closed timestamp promise made by the old leaseholder. - -Closed timestamps provide the guarantees that are used to provide support for low-latency historical (stale) reads, also known as [Follower Reads](../follower-reads.html). Follower reads can be particularly useful in [multi-region deployments](../multiregion-overview.html). - -For more information about the implementation of closed timestamps and Follower Reads, see our blog post [An Epic Read on Follower Reads](https://www.cockroachlabs.com/blog/follower-reads-stale-data/). - -### client.Txn and TxnCoordSender - -As we mentioned in the SQL layer's architectural overview, CockroachDB converts all SQL statements into key-value (KV) operations, which is how data is ultimately stored and accessed. - -All of the KV operations generated from the SQL layer use `client.Txn`, which is the transactional interface for the CockroachDB KV layer––but, as we discussed above, all statements are treated as transactions, so all statements use this interface. - -However, `client.Txn` is actually just a wrapper around `TxnCoordSender`, which plays a crucial role in our code base by: - -- Dealing with transactions' state. After a transaction is started, `TxnCoordSender` starts asynchronously sending heartbeat messages to that transaction's transaction record, which signals that it should be kept alive. If the `TxnCoordSender`'s heartbeating stops, the transaction record is moved to the `ABORTED` status. -- Tracking each written key or key range over the course of the transaction. -- Clearing the accumulated write intent for the transaction when it's committed or aborted. All requests being performed as part of a transaction have to go through the same `TxnCoordSender` to account for all of its write intents, which optimizes the cleanup process. - -After setting up this bookkeeping, the request is passed to the `DistSender` in the distribution layer. - -### Transaction records - -To track the status of a transaction's execution, we write a value called a transaction record to our key-value store. All of a transaction's write intents point back to this record, which lets any transaction check the status of any write intents it encounters. This kind of canonical record is crucial for supporting concurrency in a distributed environment. - -Transaction records are always written to the same range as the first key in the transaction, which is known by the `TxnCoordSender`. However, the transaction record itself isn't created until one of the following conditions occur: - -- The write operation commits -- The `TxnCoordSender` heartbeats the transaction -- An operation forces the transaction to abort - -Given this mechanism, the transaction record uses the following states: - -- `PENDING`: Indicates that the write intent's transaction is still in progress. -- `COMMITTED`: Once a transaction has completed, this status indicates that write intents can be treated as committed values. -- `STAGING`: Used to enable the [Parallel Commits](#parallel-commits) feature. Depending on the state of the write intents referenced by this record, the transaction may or may not be in a committed state. -- `ABORTED`: Indicates that the transaction was aborted and its values should be discarded. -- _Record does not exist_: If a transaction encounters a write intent whose transaction record doesn't exist, it uses the write intent's timestamp to determine how to proceed. If the write intent's timestamp is within the transaction liveness threshold, the write intent's transaction is treated as if it is `PENDING`, otherwise it's treated as if the transaction is `ABORTED`. - -The transaction record for a committed transaction remains until all its write intents are converted to MVCC values. - -### Write intents - -Values in CockroachDB are not written directly to the storage layer; instead values are written in a provisional state known as a "write intent." These are essentially MVCC records with an additional value added to them which identifies the transaction record to which the value belongs. They can be thought of as a combination of a replicated lock and a replicated provisional value. - -Whenever an operation encounters a write intent (instead of an MVCC value), it looks up the status of the transaction record to understand how it should treat the write intent value. If the transaction record is missing, the operation checks the write intent's timestamp and evaluates whether or not it is considered expired. - - CockroachDB manages concurrency control using a per-node, in-memory lock table. This table holds a collection of locks acquired by in-progress transactions, and incorporates information about write intents as they are discovered during evaluation. For more information, see the section below on [Concurrency control](#concurrency-control). - -#### Resolving write intents - -Whenever an operation encounters a write intent for a key, it attempts to "resolve" it, the result of which depends on the write intent's transaction record: - -- `COMMITTED`: The operation reads the write intent and converts it to an MVCC value by removing the write intent's pointer to the transaction record. -- `ABORTED`: The write intent is ignored and deleted. -- `PENDING`: This signals there is a [transaction conflict](#transaction-conflicts), which must be resolved. -- `STAGING`: This signals that the operation should check whether the staging transaction is still in progress by verifying that the transaction coordinator is still heartbeating the staging transaction’s record. If the coordinator is still heartbeating the record, the operation should wait. For more information, see [Parallel Commits](#parallel-commits). -- _Record does not exist_: If the write intent was created within the transaction liveness threshold, it's the same as `PENDING`, otherwise it's treated as `ABORTED`. - -### Concurrency control - - The *concurrency manager* sequences incoming requests and provides isolation between the transactions that issued those requests that intend to perform conflicting operations. This activity is also known as [concurrency control](https://en.wikipedia.org/wiki/Concurrency_control). - -The concurrency manager combines the operations of a *latch manager* and a *lock table* to accomplish this work: - -- The *latch manager* sequences the incoming requests and provides isolation between those requests. -- The *lock table* provides both locking and sequencing of requests (in concert with the latch manager). It is a per-node, in-memory data structure that holds a collection of locks acquired by in-progress transactions. To ensure compatibility with the existing system of [write intents](#write-intents) (a.k.a. replicated, exclusive locks), it pulls in information about these external locks as necessary when they are discovered in the course of evaluating requests. - -The concurrency manager enables support for pessimistic locking via [SQL](sql-layer.html) using the [`SELECT FOR UPDATE`](../select-for-update.html) statement. This statement can be used to increase throughput and decrease tail latency for contended operations. - -For more details about how the concurrency manager works with the latch manager and lock table, see the sections below: - -- [Concurrency manager](#concurrency-manager) -- [Lock table](#lock-table) -- [Latch manager](#latch-manager) - -#### Concurrency manager - - The concurrency manager is a structure that sequences incoming requests and provides isolation between the transactions that issued those requests that intend to perform conflicting operations. During sequencing, conflicts are discovered and any found are resolved through a combination of passive queuing and active pushing. Once a request has been sequenced, it is free to evaluate without concerns of conflicting with other in-flight requests due to the isolation provided by the manager. This isolation is guaranteed for the lifetime of the request but terminates once the request completes. - -Each request in a transaction should be isolated from other requests, both during the request's lifetime and after the request has completed (assuming it acquired locks), but within the surrounding transaction's lifetime. - -The manager accommodates this by allowing transactional requests to acquire locks, which outlive the requests themselves. Locks extend the duration of the isolation provided over specific keys to the lifetime of the lock-holder transaction itself. They are (typically) only released when the transaction commits or aborts. Other requests that find these locks while being sequenced wait on them to be released in a queue before proceeding. Because locks are checked during sequencing, locks do not need to be checked again during evaluation. - -However, not all locks are stored directly under the manager's control, so not all locks are discoverable during sequencing. Specifically, write intents (replicated, exclusive locks) are stored inline in the MVCC keyspace, so they are not detectable until request evaluation time. To accommodate this form of lock storage, the manager integrates information about external locks with the concurrency manager structure. - -{{site.data.alerts.callout_info}} -Currently, the concurrency manager operates on an unreplicated lock table structure. In the future, we intend to pull all locks, including those associated with [write intents](#write-intents), into the concurrency manager directly through a replicated lock table structure. -{{site.data.alerts.end}} - -Fairness is ensured between requests. In general, if any two requests conflict then the request that arrived first will be sequenced first. As such, sequencing guarantees first-in, first-out (FIFO) semantics. The primary exception to this is that a request that is part of a transaction which has already acquired a lock does not need to wait on that lock during sequencing, and can therefore ignore any queue that has formed on the lock. For other exceptions to this sequencing guarantee, see the [lock table](#lock-table) section below. - -#### Lock table - - The lock table is a per-node, in-memory data structure that holds a collection of locks acquired by in-progress transactions. Each lock in the table has a possibly-empty lock wait-queue associated with it, where conflicting transactions can queue while waiting for the lock to be released. Items in the locally stored lock wait-queue are propagated as necessary (via RPC) to the existing [`TxnWaitQueue`](#txnwaitqueue), which is stored on the leader of the range's Raft group that contains the [transaction record](#transaction-records). - -The database is read and written using "requests". Transactions are composed of one or more requests. Isolation is needed across requests. Additionally, since transactions represent a group of requests, isolation is needed across such groups. Part of this isolation is accomplished by maintaining multiple versions and part by allowing requests to acquire locks. Even the isolation based on multiple versions requires some form of mutual exclusion to ensure that a read and a conflicting lock acquisition do not happen concurrently. The lock table provides both locking and sequencing of requests (in concert with the use of latches). - -Locks outlive the requests themselves and thereby extend the duration of the isolation provided over specific keys to the lifetime of the lock-holder transaction itself. They are (typically) only released when the transaction commits or aborts. Other requests that find these locks while being sequenced wait on them to be released in a queue before proceeding. Because locks are checked during sequencing, requests are guaranteed access to all declared keys after they have been sequenced. In other words, locks do not need to be checked again during evaluation. - -{{site.data.alerts.callout_info}} -Currently, not all locks are stored directly under lock table control. Some locks are stored as [write intents](#write-intents) in the MVCC layer, and are thus not discoverable during sequencing. Specifically, write intents (replicated, exclusive locks) are stored inline in the MVCC keyspace, so they are often not detectable until request evaluation time. To accommodate this form of lock storage, the lock table adds information about these locks as they are encountered during evaluation. In the future, we intend to pull all locks, including those associated with write intents, into the lock table directly. -{{site.data.alerts.end}} - -The lock table also provides fairness between requests. If two requests conflict then the request that arrived first will typically be sequenced first. There are some exceptions: - -- A request that is part of a transaction which has already acquired a lock does not need to wait on that lock during sequencing, and can therefore ignore any queue that has formed on the lock. - -- Contending requests that encounter different levels of contention may be sequenced in non-FIFO order. This is to allow for greater concurrency. For example, if requests *R1* and *R2* contend on key *K2*, but *R1* is also waiting at key *K1*, *R2* may slip past *R1* and evaluate. - -#### Latch manager - -The latch manager sequences incoming requests and provides isolation between those requests under the supervision of the [concurrency manager](#concurrency-manager). - -The way the latch manager works is as follows: - -As write requests occur for a range, the range's leaseholder serializes them; that is, they are placed into some consistent order. - -To enforce this serialization, the leaseholder creates a "latch" for the keys in the write value, providing uncontested access to the keys. If other requests come into the leaseholder for the same set of keys, they must wait for the latch to be released before they can proceed. - -Read requests also generate latches. Multiple read latches over the same keys can be held concurrently, but a read latch and a write latch over the same keys cannot. - -Another way to think of a latch is like a [mutex](https://en.wikipedia.org/wiki/Lock_(computer_science)) which is only needed for the duration of a single, low-level request. To coordinate longer-running, higher-level requests (i.e., client transactions), we use a durable system of [write intents](#write-intents). - -### Isolation levels - -Isolation is an element of [ACID transactions](https://en.wikipedia.org/wiki/ACID), which determines how concurrency is controlled, and ultimately guarantees consistency. - -CockroachDB executes all transactions at the strongest ANSI transaction isolation level: `SERIALIZABLE`. All other ANSI transaction isolation levels (e.g., `SNAPSHOT`, `READ UNCOMMITTED`, `READ COMMITTED`, and `REPEATABLE READ`) are automatically upgraded to `SERIALIZABLE`. Weaker isolation levels have historically been used to maximize transaction throughput. However, [recent research](http://www.bailis.org/papers/acidrain-sigmod2017.pdf) has demonstrated that the use of weak isolation levels results in substantial vulnerability to concurrency-based attacks. - -CockroachDB now only supports `SERIALIZABLE` isolation. In previous versions of CockroachDB, you could set transactions to `SNAPSHOT` isolation, but that feature has been removed. - -`SERIALIZABLE` isolation does not allow any anomalies in your data, and is enforced by requiring the client to retry transactions if serializability violations are possible. - -### Transaction conflicts - -CockroachDB's transactions allow the following types of conflicts that involve running into an intent: - -- **Write/write**, where two `PENDING` transactions create write intents for the same key. -- **Write/read**, when a read encounters an existing write intent with a timestamp less than its own. - -To make this simpler to understand, we'll call the first transaction `TxnA` and the transaction that encounters its write intents `TxnB`. - -CockroachDB proceeds through the following steps: - -1. If the transaction has an explicit priority set (i.e., `HIGH` or `LOW`), the transaction with the lower priority is aborted (in the write/write case) or has its timestamp pushed (in the write/read case). - -1. If the encountered transaction is expired, it's `ABORTED` and conflict resolution succeeds. We consider a write intent expired if: - - It doesn't have a transaction record and its timestamp is outside of the transaction liveness threshold. - - Its transaction record hasn't been heartbeated within the transaction liveness threshold. - -2. `TxnB` enters the `TxnWaitQueue` to wait for `TxnA` to complete. - -Additionally, the following types of conflicts that do not involve running into intents can arise: - -- **Write after read**, when a write with a lower timestamp encounters a later read. This is handled through the [timestamp cache](#timestamp-cache). -- **Read within uncertainty window**, when a read encounters a value with a higher timestamp but it's ambiguous whether the value should be considered to be in the future or in the past of the transaction because of possible *clock skew*. This is handled by attempting to push the transaction's timestamp beyond the uncertain value (see [read refreshing](#read-refreshing)). Note that, if the transaction has to be retried, reads will never encounter uncertainty issues on any node which was previously visited, and that there's never any uncertainty on values read from the transaction's gateway node. - -### TxnWaitQueue - -The `TxnWaitQueue` tracks all transactions that could not push a transaction whose writes they encountered, and must wait for the blocking transaction to complete before they can proceed. - -The `TxnWaitQueue`'s structure is a map of blocking transaction IDs to those they're blocking. For example: - -~~~ -txnA -> txn1, txn2 -txnB -> txn3, txn4, txn5 -~~~ - -Importantly, all of this activity happens on a single node, which is the leader of the range's Raft group that contains the transaction record. - -Once the transaction does resolve––by committing or aborting––a signal is sent to the `TxnWaitQueue`, which lets all transactions that were blocked by the resolved transaction begin executing. - -Blocked transactions also check the status of their own transaction to ensure they're still active. If the blocked transaction was aborted, it's simply removed. - -If there is a deadlock between transactions (i.e., they're each blocked by each other's Write Intents), one of the transactions is randomly aborted. In the above example, this would happen if `TxnA` blocked `TxnB` on `key1` and `TxnB` blocked `TxnA` on `key2`. - -### Read refreshing - -Whenever a transaction's timestamp has been pushed, additional checks are required before allowing it to commit at the pushed timestamp: any values which the transaction previously read must be checked to verify that no writes have subsequently occurred between the original transaction timestamp and the pushed transaction timestamp. This check prevents serializability violation. The check is done by keeping track of all the reads using a dedicated `RefreshRequest`. If this succeeds, the transaction is allowed to commit (transactions perform this check at commit time if they've been pushed by a different transaction or by the [timestamp cache](#timestamp-cache), or they perform the check whenever they encounter a [`ReadWithinUncertaintyIntervalError`](../transaction-retry-error-reference.html#readwithinuncertaintyinterval) immediately, before continuing). -If the refreshing is unsuccessful, then the transaction must be retried at the pushed timestamp. - -### Transaction pipelining - -Transactional writes are pipelined when being replicated and when being written to disk, dramatically reducing the latency of transactions that perform multiple writes. For example, consider the following transaction: - -{% include_cached copy-clipboard.html %} -~~~ sql --- CREATE TABLE kv (id UUID PRIMARY KEY DEFAULT gen_random_uuid(), key VARCHAR, value VARCHAR); -> BEGIN; -INSERT into kv (key, value) VALUES ('apple', 'red'); -INSERT into kv (key, value) VALUES ('banana', 'yellow'); -INSERT into kv (key, value) VALUES ('orange', 'orange'); -COMMIT; -~~~ - -With transaction pipelining, write intents are replicated from leaseholders in parallel, so the waiting all happens at the end, at transaction commit time. - -At a high level, transaction pipelining works as follows: - -1. For each statement, the transaction gateway node communicates with the leaseholders (*L*1, *L*2, *L*3, ..., *L*i) for the ranges it wants to write to. Since the primary keys in the table above are UUIDs, the ranges are probably split across multiple leaseholders (this is a good thing, as it decreases [transaction conflicts](#transaction-conflicts)). - -2. Each leaseholder *L*i receives the communication from the transaction gateway node and does the following in parallel: - - Creates write intents and sends them to its follower nodes. - - Responds to the transaction gateway node that the write intents have been sent. Note that replication of the intents is still in-flight at this stage. - -3. When attempting to commit, the transaction gateway node then waits for the write intents to be replicated in parallel to all of the leaseholders' followers. When it receives responses from the leaseholders that the write intents have propagated, it commits the transaction. - -In terms of the SQL snippet shown above, all of the waiting for write intents to propagate and be committed happens once, at the very end of the transaction, rather than for each individual write. This means that the cost of multiple writes is not `O(n)` in the number of SQL DML statements; instead, it's `O(1)`. - -### Parallel Commits - -The *Parallel Commits* feature introduces a new, optimized atomic commit protocol that cuts the commit latency of a transaction in half, from two rounds of consensus down to one. Combined with [Transaction pipelining](#transaction-pipelining), this brings the latency incurred by common OLTP transactions to near the theoretical minimum: the sum of all read latencies plus one round of consensus latency. - -Under the new atomic commit protocol, the transaction coordinator can return to the client eagerly when it knows that the writes in the transaction have succeeded. Once this occurs, the transaction coordinator can set the transaction record's state to `COMMITTED` and resolve the transaction's write intents asynchronously. - -The transaction coordinator is able to do this while maintaining correctness guarantees because it populates the transaction record with enough information (via a new `STAGING` state, and an array of in-flight writes) for other transactions to determine whether all writes in the transaction are present, and thus prove whether or not the transaction is committed. - -For an example showing how the Parallel Commits feature works in more detail, see [Parallel Commits - step by step](#parallel-commits-step-by-step). - -{{site.data.alerts.callout_info}} -The latency until intents are resolved is unchanged by the introduction of Parallel Commits: two rounds of consensus are still required to resolve intents. This means that [contended workloads](../performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention) are expected to profit less from this feature. -{{site.data.alerts.end}} - -#### Parallel Commits - step by step - -This section contains a step by step example of a transaction that writes its data using the Parallel Commits atomic commit protocol and does not encounter any errors or conflicts. - -##### Step 1 - -The client starts the transaction. A transaction coordinator is created to manage the state of that transaction. - -![parallel-commits-00.png](../../images/{{page.version.version}}/parallel-commits-00.png "Parallel Commits Diagram #1") - -##### Step 2 - -The client issues a write to the "Apple" key. The transaction coordinator begins the process of laying down a write intent on the key where the data will be written. The write intent has a timestamp and a pointer to an as-yet nonexistent transaction record. Additionally, each write intent in the transaction is assigned a unique sequence number which is used to uniquely identify it. - -The coordinator avoids creating the record for as long as possible in the transaction's lifecycle as an optimization. The fact that the transaction record does not yet exist is denoted in the diagram by its dotted lines. - -{{site.data.alerts.callout_info}} -The coordinator does not need to wait for write intents to replicate from leaseholders before moving on to the next statement from the client, since that is handled in parallel by [Transaction Pipelining](#transaction-pipelining). -{{site.data.alerts.end}} - -![parallel-commits-01.png](../../images/{{page.version.version}}/parallel-commits-01.png "Parallel Commits Diagram #2") - -##### Step 3 - -The client issues a write to the "Berry" key. The transaction coordinator lays down a write intent on the key where the data will be written. This write intent has a pointer to the same transaction record as the intent created in [Step 2](#step-2), since these write intents are part of the same transaction. - -As before, the coordinator does not need to wait for write intents to replicate from leaseholders before moving on to the next statement from the client. - -![parallel-commits-02.png](../../images/{{page.version.version}}/parallel-commits-02.png "Parallel Commits Diagram #3") - -##### Step 4 - -The client issues a request to commit the transaction's writes. The transaction coordinator creates the transaction record and immediately sets the record's state to `STAGING`, and records the keys of each write that the transaction has in flight. - -It does this without waiting to see whether the writes from Steps [2](#step-2) and [3](#step-3) have succeeded. - -![parallel-commits-03.png](../../images/{{page.version.version}}/parallel-commits-03.png "Parallel Commits Diagram #4") - -##### Step 5 - -The transaction coordinator, having received the client's `COMMIT` request, waits for the pending writes to succeed (i.e., be replicated across the cluster). Once all of the pending writes have succeeded, the coordinator returns a message to the client, letting it know that its transaction has committed successfully. - -![parallel-commits-04.png](../../images/{{page.version.version}}/parallel-commits-04.png "Parallel Commits Diagram #4") - -The transaction is now considered atomically committed, even though the state of its transaction record is still `STAGING`. The reason this is still considered an atomic commit condition is that a transaction is considered committed if it is one of the following logically equivalent states: - -1. The transaction record's state is `STAGING`, and its list of pending writes have all succeeded (i.e., the `InFlightWrites` have achieved consensus across the cluster). Any observer of this transaction can verify that its writes have replicated. Transactions in this state are *implicitly committed*. - -2. The transaction record's state is `COMMITTED`. Transactions in this state are *explicitly committed*. - -Despite their logical equivalence, the transaction coordinator now works as quickly as possible to move the transaction record from the `STAGING` to the `COMMITTED` state so that other transactions do not encounter a possibly conflicting transaction in the `STAGING` state and then have to do the work of verifying that the staging transaction's list of pending writes has succeeded. Doing that verification (also known as the "transaction status recovery process") would be slow. - -Additionally, when other transactions encounter a transaction in `STAGING` state, they check whether the staging transaction is still in progress by verifying that the transaction coordinator is still heartbeating that staging transaction’s record. If the coordinator is still heartbeating the record, the other transactions will wait, on the theory that letting the coordinator update the transaction record with the result of the attempt to commit will be faster than going through the transaction status recovery process. This means that in practice, the transaction status recovery process is only used if the transaction coordinator dies due to an untimely crash. - -## Non-blocking transactions - -{% include_cached new-in.html version="v21.1" %} CockroachDB supports low-latency, global reads of read-mostly data in [multi-region clusters](../multiregion-overview.html) using _non-blocking transactions_: an extension of the [standard read-write transaction protocol](#overview) that allows a writing transaction to perform [locking](#concurrency-control) in a manner such that contending reads by other transactions can avoid waiting on its locks. - -The non-blocking transaction protocol and replication scheme differ from standard read-write transactions as follows: - -- Non-blocking transactions use a replication scheme over the [ranges](overview.html#terms) they operate on that allows all followers in these ranges to serve consistent (non-stale) reads. -- Non-blocking transactions are minimally disruptive to reads over the data they modify, even in the presence of read/write [contention](../performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention). - -These properties of non-blocking transactions combine to provide predictable read latency for a configurable subset of data in [global deployments](../multiregion-overview.html). This is useful since there exists a sizable class of data which is heavily skewed towards read traffic. - -Most users will not interact with the non-blocking transaction mechanism directly. Instead, they will [set a `GLOBAL` table locality](../multiregion-overview.html#global-tables) using the SQL API. - -### How non-blocking transactions work - -The consistency guarantees offered by non-blocking transactions are enforced through semi-synchronized clocks with bounded uncertainty, _not_ inter-node communication, since the latter would struggle to provide the same guarantees without incurring excessive latency costs in global deployments. - -Non-blocking transactions are implemented via _non-blocking ranges_. Every non-blocking range has the following properties: - -- Any transaction that writes to this range has its write timestamp pushed into the future. -- The range is able to propagate a [closed timestamp](#closed-timestamps) in the future of present time. -- A transaction that writes to this range and commits with a future time commit timestamp needs to wait until the HLC advances past its commit timestamp. This process is known as _"commit-wait"_. Essentially, the HLC waits until it advances past the future timestamp on its own, or it advances due to updates from other timestamps. -- A transaction that reads a future-time write to this range can have its commit timestamp bumped into the future as well, if the write falls in the read's uncertainty window (this is dictated by the [maximum clock offset](#max-clock-offset-enforcement) configured for the cluster). Such transactions (a.k.a. "conflicting readers") may also need to commit-wait. - -As a result of the above properties, all replicas in a non-blocking range are expected to be able to serve transactionally-consistent reads at the present. This means that all follower replicas in a non-blocking range (which includes all [non-voting replicas](replication-layer.html#non-voting-replicas)) implicitly behave as "consistent read replicas", which are exactly what they sound like: read-only replicas that always have a consistent view of the range's current state. - -## Technical interactions with other layers - -### Transaction and SQL layer - -The transaction layer receives KV operations from `planNodes` executed in the SQL layer. - -### Transaction and distribution layer - -The `TxnCoordSender` sends its KV requests to `DistSender` in the distribution layer. - -## What's next? - -Learn how CockroachDB presents a unified view of your cluster's data in the [distribution layer](distribution-layer.html). - - - -[storage]: storage-layer.html -[sql]: sql-layer.html diff --git a/src/current/v21.1/array.md b/src/current/v21.1/array.md deleted file mode 100644 index cc7a7be4db6..00000000000 --- a/src/current/v21.1/array.md +++ /dev/null @@ -1,338 +0,0 @@ ---- -title: ARRAY -summary: The ARRAY data type stores one-dimensional, 1-indexed, homogeneous arrays of any non-array data types. -toc: true -keywords: gin, gin index, gin indexes, inverted index, inverted indexes, accelerated index, accelerated indexes ---- - -The `ARRAY` data type stores one-dimensional, 1-indexed, homogeneous arrays of any non-array [data type](data-types.html). - -The `ARRAY` data type is useful for ensuring compatibility with ORMs and other tools. However, if such compatibility is not a concern, it's more flexible to design your schema with normalized tables. - - CockroachDB supports indexing array columns with [GIN indexes](inverted-indexes.html). This permits accelerating containment queries ([`@>`](functions-and-operators.html#supported-operations) and [`<@`](functions-and-operators.html#supported-operations)) on array columns by adding an index to them. - -{{site.data.alerts.callout_info}} -CockroachDB does not support nested arrays. -{{site.data.alerts.end}} - -## Syntax - -A value of data type `ARRAY` can be expressed in the following ways: - -- Appending square brackets (`[]`) to any non-array [data type](data-types.html). -- Adding the term `ARRAY` to any non-array [data type](data-types.html). - -## Size - -The size of an `ARRAY` value is variable, but it's recommended to keep values under 1 MB to ensure performance. Above that threshold, [write amplification](https://en.wikipedia.org/wiki/Write_amplification) and other considerations may cause significant performance degradation. - -## Functions - -For the list of supported `ARRAY` functions, see [Functions and Operators](functions-and-operators.html#array-functions). - -## Examples - -### Creating an array column by appending square brackets - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE a (b STRING[]); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO a VALUES (ARRAY['sky', 'road', 'car']); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM a; -~~~ - -~~~ - b ------------------- - {sky,road,car} -(1 row) -~~~ - -### Creating an array column by adding the term `ARRAY` - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE c (d INT ARRAY); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO c VALUES (ARRAY[10,20,30]); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM c; -~~~ - -~~~ - d --------------- - {10,20,30} -(1 row) -~~~ - -### Accessing an array element using array index - -{{site.data.alerts.callout_info}} -Arrays in CockroachDB are 1-indexed. -{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM c; -~~~ - -~~~ - d --------------- - {10,20,30} -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT d[2] FROM c; -~~~ - -~~~ - d ------- - 20 -(1 row) -~~~ - -### Accessing an array column using containment queries - -You can use the [operators](functions-and-operators.html#supported-operations) `<@` ("is contained by") and `@>` ("contains") to run containment queries on `ARRAY` columns. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM c WHERE d <@ ARRAY[10,20,30,40,50]; -~~~ - -~~~ - d --------------- - {10,20,30} -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM c WHERE d @> ARRAY[10,20]; -~~~ - -~~~ - d --------------- - {10,20,30} -(1 row) -~~~ - -### Appending an element to an array - -#### Using the `array_append` function - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM c; -~~~ - -~~~ - d --------------- - {10,20,30} -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE c SET d = array_append(d, 40) WHERE d[3] = 30; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM c; -~~~ - -~~~ - d ------------------ - {10,20,30,40} -(1 row) -~~~ - -#### Using the append (`||`) operator - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM c; -~~~ - -~~~ - d ------------------ - {10,20,30,40} -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE c SET d = d || 50 WHERE d[4] = 40; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM c; -~~~ - -~~~ - d --------------------- - {10,20,30,40,50} -(1 row) -~~~ - -### Ordering by an array - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE t (a INT ARRAY, b STRING); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO t VALUES (ARRAY[3,4],'threefour'),(ARRAY[1,2],'onetwo'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM t; -~~~ - -~~~ - a | b ---------+------------ - {3,4} | threefour - {1,2} | onetwo -(2 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM t ORDER BY a; -~~~ - -~~~ - a | b ---------+------------ - {1,2} | onetwo - {3,4} | threefour -(2 rows) -~~~ - -## Supported casting and conversion - -[Casting](data-types.html#data-type-conversions-and-casts) between `ARRAY` values is supported when the data types of the arrays support casting. For example, it is possible to cast from a `BOOL` array to an `INT` array but not from a `BOOL` array to a `TIMESTAMP` array: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT ARRAY[true,false,true]::INT[]; -~~~ - -~~~ - array ------------ - {1,0,1} -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT ARRAY[true,false,true]::TIMESTAMP[]; -~~~ - -~~~ -pq: invalid cast: bool[] -> TIMESTAMP[] -~~~ - -You can cast an array to a `STRING` value, for compatibility with PostgreSQL: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT ARRAY[1,NULL,3]::string; -~~~ - -~~~ - array --------------- - {1,NULL,3} -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT ARRAY[(1,'a b'),(2,'c"d')]::string; -~~~ - -~~~ - array ------------------------------------- - {"(1,\"a b\")","(2,\"c\"\"d\")"} -(1 row) -~~~ - -### Implicit casting - -CockroachDB supports implicit casting from string literals to arrays of all data types except the following: - -- [`BYTES`](bytes.html) -- [`ENUM`](enum.html) -- [`JSONB`](jsonb.html) -- [`SERIAL`](serial.html) -- `Box2D` [(spatial type)](spatial-glossary.html#data-types) -- `GEOGRAPHY` [(spatial type)](spatial-glossary.html#data-types) -- `GEOMETRY` [(spatial type)](spatial-glossary.html#data-types) - -For example, if you create a table with a column of type `INT[]`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE x (a UUID DEFAULT gen_random_uuid() PRIMARY KEY, b INT[]); -~~~ - -And then insert a string containing a comma-delimited set of integers contained in brackets: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO x(b) VALUES ('{1,2,3}'), (ARRAY[4,5,6]); -~~~ - -CockroachDB implicitly casts the string literal as an `INT[]`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM x; -~~~ - -~~~ - a | b ----------------------------------------+---------- - 2ec0ed91-8a82-4f2e-888e-ae86ece4fc60 | {4,5,6} - a521d6e9-3a2a-490d-968c-1365cace038a | {1,2,3} -(2 rows) -~~~ - -## See also - -- [Data Types](data-types.html) -- [GIN Indexes](inverted-indexes.html) diff --git a/src/current/v21.1/as-of-system-time.md b/src/current/v21.1/as-of-system-time.md deleted file mode 100644 index 3ad110f2ffc..00000000000 --- a/src/current/v21.1/as-of-system-time.md +++ /dev/null @@ -1,276 +0,0 @@ ---- -title: AS OF SYSTEM TIME -summary: The AS OF SYSTEM TIME clause executes a statement as of a specified time. -toc: true ---- - -The `AS OF SYSTEM TIME timestamp` clause causes statements to execute using the database contents "as of" a specified time in the past. - -This clause can be used to read historical data (also known as "[time travel queries](https://www.cockroachlabs.com/blog/time-travel-queries-select-witty_subtitle-the_future/)") and can also be advantageous for performance as it decreases transaction conflicts. For more details, see [SQL Performance Best Practices](performance-best-practices-overview.html#use-as-of-system-time-to-decrease-conflicts-with-long-running-queries). - -{{site.data.alerts.callout_info}} -Historical data is available only within the garbage collection window, which is determined by the `ttlseconds` field in the [replication zone configuration](configure-replication-zones.html). -{{site.data.alerts.end}} - -## Synopsis - -The `AS OF SYSTEM TIME` clause is supported in multiple SQL contexts, -including but not limited to: - -- In [`SELECT` clauses](select-clause.html), at the very end of the `FROM` sub-clause. -- In [`BACKUP`](backup.html), after the parameters of the `TO` sub-clause. -- In [`RESTORE`](restore.html), after the parameters of the `FROM` sub-clause. -- In [`BEGIN`](begin-transaction.html), after the `BEGIN` keyword. -- In [`SET`](set-transaction.html), after the `SET TRANSACTION` keyword. - -## Parameters - -The `timestamp` argument supports the following formats: - -Format | Notes ----|--- -[`INT`](int.html) | Nanoseconds since the Unix epoch. -negative [`INTERVAL`](interval.html) | Added to `statement_timestamp()`, and thus must be negative. -[`STRING`](string.html) | A [`TIMESTAMP`](timestamp.html), [`INT`](int.html) of nanoseconds, or negative [`INTERVAL`](interval.html). -`follower_read_timestamp()`| A [function](functions-and-operators.html) that returns the [`TIMESTAMP`](timestamp.html) `statement_timestamp() - 4.8s` (known as the [follower read timestamp](follower-reads.html#run-queries-that-use-follower-reads)). Using this function will set the time as close as possible to the present time while remaining safe for [follower reads](follower-reads.html). - -{{site.data.alerts.callout_success}} -{% include_cached new-in.html version="v21.1" %} To set `AS OF SYSTEM TIME follower_read_timestamp()` on all implicit and explicit read-only transactions by default, set the `default_transaction_use_follower_reads` [session variable](set-vars.html) to `on`. When `default_transaction_use_follower_reads=on` and follower reads are enabled, all read-only transactions use follower reads. -{{site.data.alerts.end}} - -## Examples - -### Select historical data (time-travel) - -Imagine this example represents the database's current data: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT name, balance - FROM accounts - WHERE name = 'Edna Barath'; -~~~ -~~~ -+-------------+---------+ -| name | balance | -+-------------+---------+ -| Edna Barath | 750 | -| Edna Barath | 2200 | -+-------------+---------+ -~~~ - -We could instead retrieve the values as they were on October 3, 2016 at 12:45 UTC: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT name, balance - FROM accounts - AS OF SYSTEM TIME '2016-10-03 12:45:00' - WHERE name = 'Edna Barath'; -~~~ -~~~ -+-------------+---------+ -| name | balance | -+-------------+---------+ -| Edna Barath | 450 | -| Edna Barath | 2000 | -+-------------+---------+ -~~~ - - -### Using different timestamp formats - -Assuming the following statements are run at `2016-01-01 12:00:00`, they would execute as of `2016-01-01 08:00:00`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM t AS OF SYSTEM TIME '2016-01-01 08:00:00' -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM t AS OF SYSTEM TIME 1451635200000000000 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM t AS OF SYSTEM TIME '1451635200000000000' -~~~ - -{% include_cached copy-clipboard.html %} -~~~sql -> SELECT * FROM t AS OF SYSTEM TIME '-4h' -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM t AS OF SYSTEM TIME INTERVAL '-4h' -~~~ - -### Selecting from multiple tables - -{{site.data.alerts.callout_info}} -It is not yet possible to select from multiple tables at different timestamps. The entire query runs at the specified time in the past. -{{site.data.alerts.end}} - -When selecting over multiple tables in a single `FROM` clause, the `AS -OF SYSTEM TIME` clause must appear at the very end and applies to the -entire `SELECT` clause. - -For example: - -{% include_cached copy-clipboard.html %} -~~~sql -> SELECT * FROM t, u, v AS OF SYSTEM TIME '-4h'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~sql -> SELECT * FROM t JOIN u ON t.x = u.y AS OF SYSTEM TIME '-4h'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~sql -> SELECT * FROM (SELECT * FROM t), (SELECT * FROM u) AS OF SYSTEM TIME '-4h'; -~~~ - -### Using `AS OF SYSTEM TIME` in subqueries - -To enable time travel, the `AS OF SYSTEM TIME` clause must appear in -at least the top-level statement. It is not valid to use it only in a -[subquery](subqueries.html). - -For example, the following is invalid: - -~~~ -SELECT * FROM (SELECT * FROM t AS OF SYSTEM TIME '-4h'), u -~~~ - -To facilitate the composition of larger queries from simpler queries, -CockroachDB allows `AS OF SYSTEM TIME` in sub-queries under the -following conditions: - -- The top level query also specifies `AS OF SYSTEM TIME`. -- All the `AS OF SYSTEM TIME` clauses specify the same timestamp. - -For example: - -{% include_cached copy-clipboard.html %} -~~~sql -> SELECT * FROM (SELECT * FROM t AS OF SYSTEM TIME '-4h') tp - JOIN u ON tp.x = u.y - AS OF SYSTEM TIME '-4h' -- same timestamp as above - OK. - WHERE x < 123; -~~~ - -### Using `AS OF SYSTEM TIME` in transactions - -You can use the [`BEGIN`](begin-transaction.html) statement to execute the transaction using the database contents "as of" a specified time in the past. - -{% include {{ page.version.version }}/sql/begin-transaction-as-of-system-time-example.md %} - -Alternatively, you can use the [`SET`](set-transaction.html) statement to execute the transaction using the database contents "as of" a specified time in the past. - -{% include {{ page.version.version }}/sql/set-transaction-as-of-system-time-example.md %} - -### Using `AS OF SYSTEM TIME` to recover recently lost data - -It is possible to recover lost data as a result of an online schema change prior to when [garbage collection](architecture/storage-layer.html#garbage-collection) begins: - -{% include_cached copy-clipboard.html %} -~~~sql -> CREATE DATABASE foo; -~~~ -~~~ -CREATE DATABASE - - -Time: 3ms total (execution 3ms / network 0ms) -~~~ -{% include_cached copy-clipboard.html %} -~~~sql -> CREATE TABLE foo.bar (id INT PRIMARY KEY); -~~~ -~~~ -CREATE TABLE - - -Time: 4ms total (execution 3ms / network 0ms) -~~~ -{% include_cached copy-clipboard.html %} -~~~sql -> INSERT INTO foo.bar VALUES (1), (2); -~~~ -~~~ -INSERT 2 - - -Time: 5ms total (execution 5ms / network 0ms) -~~~ -{% include_cached copy-clipboard.html %} -~~~sql -> SELECT now(); -~~~ -~~~ - now --------------------------------- - 2022-02-01 21:11:53.63771+00 -(1 row) - - -Time: 1ms total (execution 0ms / network 0ms) -~~~ -{% include_cached copy-clipboard.html %} -~~~sql -> DROP TABLE foo.bar; -~~~ -~~~ -DROP TABLE - - -Time: 45ms total (execution 45ms / network 0ms) -~~~ -{% include_cached copy-clipboard.html %} -~~~sql -> SELECT * FROM foo.bar AS OF SYSTEM TIME '2022-02-01 21:11:53.63771+00'; -~~~ -~~~ - id ------- - 1 - 2 -(2 rows) - - -Time: 2ms total (execution 2ms / network 0ms) -~~~ -{% include_cached copy-clipboard.html %} -~~~sql -> SELECT * FROM foo.bar; -~~~ -~~~ -ERROR: relation "foo.bar" does not exist -SQLSTATE: 42P01 -~~~ - -{{site.data.alerts.callout_danger}} -Once garbage collection has occurred, `AS OF SYSTEM TIME` will no longer be able to recover lost data. For more long-term recovery solutions, consider taking either a [full or incremental backup](take-full-and-incremental-backups.html) of your cluster. -{{site.data.alerts.end}} - -## See also - -- [Select Historical Data](select-clause.html#select-historical-data-time-travel) -- [Time-Travel Queries](https://www.cockroachlabs.com/blog/time-travel-queries-select-witty_subtitle-the_future/) -- [Follower Reads](follower-reads.html) -- [Follower Reads Topology Pattern](topology-follower-reads.html) - -## Tech note - -{{site.data.alerts.callout_info}} -Although the following format is supported, it is not intended to be used by most users. -{{site.data.alerts.end}} - -HLC timestamps can be specified using a [`DECIMAL`](decimal.html). The -integer part is the wall time in nanoseconds. The fractional part is -the logical counter, a 10-digit integer. This is the same format as -produced by the `cluster_logical_timestamp()` function. diff --git a/src/current/v21.1/authentication.md b/src/current/v21.1/authentication.md deleted file mode 100644 index 2b7d3873e52..00000000000 --- a/src/current/v21.1/authentication.md +++ /dev/null @@ -1,297 +0,0 @@ ---- -title: Authentication -summary: Learn about the authentication features for secure CockroachDB clusters. -toc: true ---- - -Authentication refers to the act of verifying the identity of the other party in communication. CockroachDB requires TLS 1.3 digital certificates for inter-node authentication and accepts TLS 1.2 and TLS 1.3 certificates for client-node authentication. This document discusses how CockroachDB uses digital certificates and what your options are for configuring user authentication for SQL clients and the DB Console UI. It also offers a [conceptual overview](#background-on-public-key-cryptography-and-digital-certificates) of public key cryptography and digital certificates. - -- If you are familiar with public key cryptography and digital certificates, then reading the [Using digital certificates with CockroachDB](#using-digital-certificates-with-cockroachdb) section should be enough. -- If you are unfamiliar with public key cryptography and digital certificates, you might want to skip over to the [conceptual overview](#background-on-public-key-cryptography-and-digital-certificates) first and then come back to the [Using digital certificates with CockroachDB](#using-digital-certificates-with-cockroachdb) section. -- If you want to know how to create CockroachDB security certificates, see [Create Security Certificates](cockroach-cert.html). - -## Using digital certificates with CockroachDB - -Each CockroachDB node in a secure cluster must have a **node certificate**, which is a TLS 1.3 certificate. This certificate is multi-functional: the same certificate is presented irrespective of whether the node is acting as a server or a client. The nodes use these certificates to establish secure connections with clients and with other nodes. Node certificates have the following requirements: - -- The hostname or address (IP address or DNS name) used to reach a node, either directly or through a load balancer, must be listed in the **Common Name** or **Subject Alternative Names** fields of the certificate: - - - The values specified in [`--listen-addr`](cockroach-start.html#networking) and [`--advertise-addr`](cockroach-start.html#networking) flags, or the node hostname and fully qualified hostname if not specified - - Any host addresses/names used to reach a specific node - - Any load balancer addresses/names or DNS aliases through which the node could be reached - - `localhost` and local address if connections are made through the loopback device on the same host - -- CockroachDB must be configured to trust the certificate authority that signed the certificate. - -Based on your security setup, you can use the [`cockroach cert` commands](cockroach-cert.html), [`openssl` commands](create-security-certificates-openssl.html), or a [custom CA](create-security-certificates-custom-ca.html) to generate all the keys and certificates. - -A CockroachDB cluster consists of multiple nodes and clients. The nodes can communicate with each other, with the SQL clients, and the DB Console. In client-node SQL communication and client-UI communication, the node acts as a server, but in inter-node communication, a node may act as a server or a client. Hence authentication in CockroachDB involves: - -- Node authentication using [TLS 1.2](https://en.wikipedia.org/wiki/Transport_Layer_Security) digital certificates. -- Client authentication using TLS digital certificates, passwords, or [GSSAPI authentication](gssapi_authentication.html) (for Enterprise users). - -### Node authentication - -To set up a secure cluster without using an existing certificate authority, you'll need to generate the following files: - -- CA certificate -- Node certificate and key -- (Optional) UI certificate and key - -### Client authentication - -CockroachDB offers the following methods for client authentication: - -- **Client certificate and key authentication**, which is available to all users. To ensure the highest level of security, we recommend only using client certificate and key authentication. - - Example: - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --certs-dir=certs --user=jpointsman - ~~~ - -- **Password authentication**, which is available to users and roles who you've created passwords for. Password creation is supported only in secure clusters. - - Example: - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --certs-dir=certs --user=jpointsman - ~~~ - - ~~~ - # Welcome to the CockroachDB SQL shell. - # All statements must be terminated by a semicolon. - # To exit, type: \q. - # - Enter password: - ~~~ - - Note that the client still needs the CA certificate to validate the nodes' certificates. - -- **Password authentication without TLS** - - For deployments where transport security is already handled at the infrastructure level (e.g., IPSec with DMZ), and TLS-based transport security is not possible or not desirable, CockroachDB now supports delegating transport security to the infrastructure with the new experimental flag `--accept-sql-without-tls` for [`cockroach start`](cockroach-start.html#security). - - With this flag, SQL clients can establish a session over TCP without a TLS handshake. They still need to present valid authentication credentials, for example a password in the default configuration. Different authentication schemes can be further configured as per `server.host_based_authentication.configuration`. - - Example: - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --user=jpointsman --insecure - ~~~ - - ~~~ - # Welcome to the CockroachDB SQL shell. - # All statements must be terminated by a semicolon. - # To exit, type: \q. - # - Enter password: - ~~~ - -- [**Single sign-on authentication**](sso.html), which is available to [Enterprise users](enterprise-licensing.html) to grant access to the DB Console. - -- [**GSSAPI authentication**](gssapi_authentication.html), which is available to [Enterprise users](enterprise-licensing.html). - -### Using `cockroach cert` or `openssl` commands - -You can use the [`cockroach cert` commands](cockroach-cert.html) or [`openssl` commands](create-security-certificates-openssl.html) to create the CA certificate and key, and node and client certificates and keys. - -Note that the node certificate created using `cockroach cert` or`openssl` is multi-functional, which means that the same certificate is presented irrespective of whether the node is acting as a server or a client. Thus all nodes must have the following: - -- `CN=node` for the special user `node` when the node acts as a client. -- All IP addresses and DNS names for the node must be listed in the `Subject Alternative Name` field for when the node acts as a server. CockroachDB also supports [wildcard notation in DNS names](https://en.wikipedia.org/wiki/Wildcard_certificate). - -**Node key and certificates** - -A node must have the following files with file names as specified in the table: - -File name | File usage --------------|------------ -`ca.crt` | CA certificate created using the `cockroach cert` command. -`node.crt` | Server certificate created using the `cockroach cert` command.

`node.crt` must have `CN=node` and the list of IP addresses and DNS names listed in the `Subject Alternative Name` field. CockroachDB also supports [wildcard notation in DNS names](https://en.wikipedia.org/wiki/Wildcard_certificate).

Must be signed by the CA represented by `ca.crt`. -`node.key` | Server key created using the `cockroach cert` command. - -**Client key and certificates** - -A client must have the following files with file names as specified in the table: - -File name | File usage --------------|------------ -`ca.crt` | CA certificate created using the `cockroach cert` command. -`client..crt` | Client certificate for `` (e.g., `client.root.crt` for user `root`).

Each `client..crt` must have `CN=` (for example, `CN=marc` for `client.marc.crt`).

Must be signed by the CA represented by `ca.crt`. -`client..key` | Client key created using the `cockroach cert` command. - -Alternatively, you can use [password authentication](#client-authentication). Remember, the client still needs `ca.crt` for node authentication. - -### Using a custom CA - -In the previous section, we discussed the scenario where the node and client certificates are signed by the CA created using the `cockroach cert` command. But what if you want to use an external CA, like your organizational CA or a public CA? In that case, our certificates might need some modification. Here’s why: - -As mentioned earlier, the node certificate is multi-functional, as in the same certificate is presented irrespective of whether the node is acting as a server or client. To make the certificate multi-functional, the `node.crt` must have `CN=node` and the list of IP addresses and DNS names listed in the`Subject Alternative Names` field. - -But some CAs will not sign a certificate containing a `CN` that is not an IP address or domain name. Here's why: The TLS client certificates are used to authenticate the client connecting to a server. Because most client certificates authenticate a user instead of a device, the certificates contain usernames instead of hostnames. This makes it difficult for public CAs to verify the client's identity and hence most public CAs will not sign a client certificate. - -To get around this issue, we can split the node key and certificate into two: - -- `node.crt` and `node.key`: The node certificate to be presented when the node acts as a server and the corresponding key. `node.crt` must have the list of IP addresses and DNS names listed in the `Subject Alternative Names`. -- `client.node.crt` and `client.node.key`: The node certificate to be presented when the node acts as a client for another node, and the corresponding key. `client.node.crt` must have `CN=node`. - -**Node key and certificates** - -A node must have the following files with file names as specified in the table: - -File name | File usage --------------|------------ -`ca.crt` | CA certificate issued by the public CA or your organizational CA. -`node.crt` | Node certificate for when node acts as server.

All IP addresses and DNS names for the node must be listed in the `Subject Alternative Name`. CockroachDB also supports [wildcard notation in DNS names](https://en.wikipedia.org/wiki/Wildcard_certificate).

Must be signed by the CA represented by `ca.crt`. -`node.key` | Server key corresponding to `node.crt`. -`client.node.crt` | Node certificate for when node acts as client.

Must have `CN=node`.

Must be signed by the CA represented by `ca.crt`. -`client.node.key` | Client key corresponding to `client.node.crt`. - -Optionally, if you have a certificate issued by a public CA to securely access the DB Console, you need to place the certificate and key (`ui.crt` and `ui.key` respectively) in the directory specified by the `--certs-dir` flag. - -**Client key and certificates** - -A client must have the following files with file names as specified in the table: - -File name | File usage --------------|------------ -`ca.crt` | CA certificate issued by the public CA or your organizational CA. -`client..crt` | Client certificate for `` (e.g., `client.root.crt` for user `root`).

Each `client..crt` must have `CN=` (for example, `CN=marc` for `client.marc.crt`).

Must be signed by the CA represented by `ca.crt`. -`client..key` | Client key corresponding to `client..crt`. - -Alternatively, you can use [password authentication](#client-authentication). Remember, the client still needs `ca.crt` for node authentication. - -### Using a public CA certificate to access the DB Console for a secure cluster - -One of the limitations of using `cockroach cert` or `openssl` is that the browsers used to access the DB Console do not trust the node certificates presented to them. Web browsers come preloaded with CA certificates from well-established entities (e.g., GlobalSign and DigiTrust). The CA certificate generated using the `cockroach cert` or `openssl` is not preloaded in the browser. Hence on accessing the DB Console for a secure cluster, you get the “Unsafe page” warning. Now you could add the CA certificate to the browser to avoid the warning, but that is not a recommended practice. Instead, you can use the established CAs (for example, Let’s Encrypt), to create a certificate and key to access the DB Console. - -Once you have the UI cert and key, add it to the Certificates directory specified by the `--certs-dir` flag in the `cockroach cert` command. The next time the browser tries to access the UI, the node will present the UI cert instead of the node cert, and you’ll not see the “unsafe site” warning anymore. - -**Node key and certificates** - -A node must have the following files with file names as specified in the table: - -File name | File usage --------------|------------ -`ca.crt` | CA certificate created using the `cockroach cert` command. -`node.crt` | Server certificate created using the `cockroach cert` command.

`node.crt` must have `CN=node` and the list of IP addresses and DNS names listed in the `Subject Alternative Name` field. CockroachDB also supports [wildcard notation in DNS names](https://en.wikipedia.org/wiki/Wildcard_certificate).

Must be signed by the CA represented by `ca.crt`. -`node.key` | Server key created using the `cockroach cert` command. -`ui.crt` | UI certificate signed by the public CA. `ui.crt` must have the IP addresses and DNS names used to reach the DB Console listed in the `Subject Alternative Name`. -`ui.key` | UI key corresponding to `ui.crt`. - -**Client key and certificates** - -A client must have the following files with file names as specified in the table: - -File name | File usage --------------|------------ -`ca.crt` | CA certificate created using the `cockroach cert` command. -`client..crt` | Client certificate for `` (e.g., `client.root.crt` for user `root`).

Each `client..crt` must have `CN=` (for example, `CN=marc` for `client.marc.crt`).

Must be signed by the CA represented by `ca.crt`. -`client..key` | Client key created using the `cockroach cert` command. - -Alternatively, you can use [password authentication](#client-authentication). Remember, the client still needs `ca.crt` for node authentication. - -### Using split CA certificates - -{{site.data.alerts.callout_danger}} -We do not recommend you use split CA certificates unless your organizational security practices mandate you to do so. -{{site.data.alerts.end}} - -You might encounter situations where you need separate CAs to sign and verify node and client certificates. In that case, you would need two CAs and their respective certificates and keys: `ca.crt` and `ca-client.crt`. - -**Node key and certificates** - -A node must have the following files with file names as specified in the table: - -File name | File usage --------------|------------ -`ca.crt` | CA certificate to verify node certificates. -`ca-client.crt` | CA certificate to verify client certificates. -`node.crt` | Node certificate for when node acts as server.

All IP addresses and DNS names for the node must be listed in the `Subject Alternative Name`. CockroachDB also supports [wildcard notation in DNS names](https://en.wikipedia.org/wiki/Wildcard_certificate).

Must be signed by the CA represented by `ca.crt`. -`node.key` | Server key corresponding to `node.crt`. -`client.node.crt` | Node certificate for when node acts as client. This certificate must be signed by the CA represented by `ca-client.crt`.

Must have `CN=node`. -`client.node.key` | Client key corresponding to `client.node.crt`. - -Optionally, if you have a certificate issued by a public CA to securely access the DB Console, you need to place the certificate and key (`ui.crt` and `ui.key` respectively) in the directory specified by the `--certs-dir` flag. - -**Client key and certificates** - -A client must have the following files with file names as specified in the table: - -File name | File usage --------------|------------ -`ca.crt` | CA certificate. -`client..crt` | Client certificate for `` (e.g., `client.root.crt` for user `root`).

Each `client..crt` must have `CN=` (for example, `CN=marc` for `client.marc.crt`).

Must be signed by the CA represented by `ca-client.crt`. -`client..key` | Client key corresponding to `client..crt`. - -## Authentication for cloud storage - -See [Use Cloud Storage for Bulk Operations](use-cloud-storage-for-bulk-operations.html). - -## Authentication best practice - -As a security best practice, we recommend that you rotate the node, client, or CA certificates in the following scenarios: - -- The node, client, or CA certificates are expiring soon. -- Your organization's compliance policy requires periodical certificate rotation. -- The key (for a node, client, or CA) is compromised. -- You need to modify the contents of a certificate, for example, to add another DNS name or the IP address of a load balancer through which a node can be reached. In this case, you would need to rotate only the node certificates. - -For details about when and how to change security certificates without restarting nodes, see [Rotate Security Certificates](rotate-certificates.html). - -## Background on public key cryptography and digital certificates - -As mentioned above, CockroachDB supports the [TLS 1.3 and TLS 1.2](https://en.wikipedia.org/wiki/Transport_Layer_Security) security protocols, which take advantage of both symmetric (to encrypt data in flight) as well as asymmetric encryption (to establish a secure channel as well as **authenticate** the communicating parties). - -Authentication refers to the act of verifying the identity of the other party in communication. CockroachDB uses TLS 1.3 digital certificates for inter-node authentication, and your choice of TLS 1.2 and TLS 1.3 certificates for client-node authentication. These authentication methods require a certificate authority (CA) as well as keys and certificates for nodes, clients, and, optionally, the [DB Console](#using-a-public-ca-certificate-to-access-the-db-console-for-a-secure-cluster). - -To understand how CockroachDB uses digital certificates, let's first understand what each of these terms means. - -Consider two people: Amy and Rosa, who want to communicate securely over an insecure computer network. The traditional solution is to use symmetric encryption that involves encrypting and decrypting a plaintext message using a shared key. Amy encrypts her message using the key and sends the encrypted message across the insecure channel. Rosa decrypts the message using the same key and reads the message. This seems like a logical solution until you realize that you need a secure communication channel to send the encryption key. - -To solve this problem, cryptographers came up with **asymmetric encryption** to set up a secure communication channel over which an encryption key can be shared. - -### Asymmetric encryption - -Asymmetric encryption involves a pair of keys instead of a single key. The two keys are called the **public key** and the **private key**. The keys consist of very long numbers linked mathematically in a way such that a message encrypted using a public key can only be decrypted using the private key and vice versa. The message cannot be decrypted using the same key that was used to encrypt the message. - -So going back to our example, Amy and Rosa both have their own public-private key pairs. They keep their private keys safe with themselves and publicly distribute their public keys. Now when Amy wants to send a message to Rosa, she requests Rosa's public key, encrypts the message using Rosa’s public key, and sends the encrypted message. Rosa uses her own private key to decrypt the message. - -But what if a malicious imposter intercepts the communication? The imposter might pose as Rosa and send their public key instead of Rosa’s. There's no way for Amy to know that the public key she received isn’t Rosa’s, so she would end up using the imposter's public key to encrypt the message and send it to the imposter. The imposter can use their own private key and decrypt and read the message, thus compromising the secure communication channel between Amy and Rosa. - -To prevent this security risk, Amy needs to be sure that the public key she received was indeed Rosa’s. That’s where the Certificate Authority (CA) comes into the picture. - -### Certificate authority - -Certificate authorities are established entities with their own public and private key pairs. They act as a root of trust and verify the identities of the communicating parties and validate their public keys. CAs can be public and paid entities (e.g., GeoTrust and Comodo), or public and free CAs (e.g., Let’s Encrypt), or your own organizational CA (e.g., CockroachDB CA). The CAs' public keys are typically widely distributed (e.g., your browser comes preloaded with certs from popular CAs like DigiCert, GeoTrust, and so on). - -Think of the CA as the passport authority of a country. When you want to get your passport as your identity proof, you submit an application to your country's passport authority. The application contains important identifying information about you: your name, address, nationality, date of birth, and so on. The passport authority verifies the information they received and validates your identity. They then issue a document - the passport - that can be presented anywhere in the world to verify your identity. For example, the TSA agent at the airport does not know you and has no reason to trust you are who you say you are. However, they trust the passport authority and thus accept your identity as presented on your passport because it has been verified and issued by the passport authority. - -Going back to our example and assuming that we trust the CA, Rosa needs to get her public key verified by the CA. She sends a CSR (Certificate Signing Request) to the CA that contains her public key and relevant identifying information. The CA will verify that it is indeed Rosa’s public key and information, _sign_ the CSR using the CA's own private key, and generate a digital document called the **digital certificate**. In our passport analogy, this is Rosa's passport containing verified identifying information about her and trusted by everyone who trusts the CA. The next time Rosa wants to establish her identity, she will present her digital certificate. - -### Digital certificate - -A public key is shared using a digital certificate signed by a CA using the CA's private key. The digital certificate contains: - -- The certificate owner’s public key -- Information about the certificate owner -- The CA's digital signature - -### Digital signature - -The CA's digital signature works as follows: The certificate contents are put through a mathematical function to create a **hash value**. This hash value is encrypted using the CA's private key to generate the **digital signature**. The digital signature is added to the digital certificate. In our example, the CA adds their digital signature to Rosa's certificate validating her identity and her public key. - -As discussed [earlier](#certificate-authority), the CA's public key is widely distributed. In our example, Amy already has the CA's public key. Now when Rosa presents her digital certificate containing her public key, Amy uses the CA's public key to decrypt the digital signature on Rosa's certificate and gets the hash value encoded in the digital signature. Amy also generates the hash value for the certificate on her own. If the hash values match, then Amy can be sure that the certificate and hence the public key it contains indeed belongs to Rosa; otherwise, she can determine that the communication channel has been compromised and refuse further contact. - -### How it all works together - -Let's see how the digital certificate is used in client-server communication: The client (e.g., a web browser) has the CA certificate (containing the CA's public key). When the client receives a server's certificate signed by the same CA, it can use the CA certificate to verify the server's certificate, thus validating the server's identity, and securely connect to the server. The important thing here is that the client needs to have the CA certificate. If you use your own organizational CA instead of a publicly established CA, you need to make sure you distribute the CA certificate to all the clients. - -## See also - -- [Client Connection Parameters](connection-parameters.html) -- [Manual Deployment](manual-deployment.html) -- [Orchestrated Deployment](orchestration.html) -- [Local Deployment](secure-a-cluster.html) -- [Other Cockroach Commands](cockroach-commands.html) diff --git a/src/current/v21.1/authorization.md b/src/current/v21.1/authorization.md deleted file mode 100644 index 5359fe18b30..00000000000 --- a/src/current/v21.1/authorization.md +++ /dev/null @@ -1,525 +0,0 @@ ---- -title: Authorization -summary: Learn about the authorization features for secure CockroachDB clusters. -toc: true ---- - -User authorization is the act of defining access policies for authenticated CockroachDB users. CockroachDB allows you to create, manage, and remove your cluster's SQL [users](#sql-users) and assign SQL-level [privileges](#assign-privileges) to the users. Additionally, you can use [role-based access management (RBAC)](#roles) for simplified user management. - -## Users and roles - -There is no technical distinction between a role or user in CockroachDB. A role/user can: - -- be permitted to log in to the [SQL shell](cockroach-sql.html). -- be granted [privileges](#privileges) to specific actions and database objects. -- be a member of other users/roles, inheriting their privileges. -- have other users/roles as members that inherit its privileges. - -We refer to these as "roles" when they are created for managing the privileges of their member "users" and not for logging in directly, which is typically reserved for "users". - -The SQL statements [`CREATE USER`](create-user.html) and [`CREATE ROLE`](create-role.html) will create the same entity with one exception: `CREATE ROLE` will add the `NOLOGIN` option by default, preventing the user/role from being used to log in. Otherwise, for enhanced PostgreSQL compatibility, the keywords `ROLE` and `USER` can be used interchangeably in SQL statements. - -Throughout the documentation, however, we will refer to a "user" or "role" based on the intended purpose of the entity. - -## SQL users - -A SQL user can interact with a CockroachDB database using the [built-in SQL shell](cockroach-sql.html) or through an application. - -### Create and manage users - -Use the [`CREATE USER`](create-user.html) and [`DROP USER`](drop-user.html) statements to create and remove users, the [`ALTER USER`](alter-user.html) statement to add or change a user's password and role options, the [`GRANT`](grant.html) and [`REVOKE`](revoke.html) statements to manage the user’s privileges, and the [`SHOW USERS`](show-users.html) statement to list users. - -A new user must be granted the required privileges for each database and table that the user needs to access. - -{{site.data.alerts.callout_info}} -By default, a new user belongs to the `public` role and has no privileges other than those assigned to the `public` role. For more information, see [Public role](#public-role). -{{site.data.alerts.end}} - -### `root` user - -The `root` user is created by default for each cluster. The `root` user is assigned to the [`admin` role](#admin-role) and has all privileges across the cluster. - -For secure clusters, in addition to [generating the client certificate](authentication.html#client-authentication) for the `root` user, you can assign or change the password for the `root` user using the [`ALTER USER`](alter-user.html) statement. - -## Roles - -A role is a group of users and/or other roles for which you can grant or revoke privileges as a whole. To simplify access management, create a role and grant privileges to the role, then create SQL users and grant them membership to the role. - -### Create and manage roles - -To create and manage your cluster's roles, use the following statements: - -Statement | Description -----------|------------ -[`CREATE ROLE`](create-role.html) | Create SQL roles. -[`DROP ROLE`](drop-role.html) | Remove one or more SQL roles. -[`GRANT`](grant.html) | Manage each role or user's SQL privileges for interacting with specific databases and tables, or add a role or user as a member to a role. -[`REVOKE`](revoke.html) | Revoke privileges from users and/or roles, or revoke a role or user's membership to a role. -[`SHOW ROLES`](show-roles.html) | List the roles for all databases. -[`SHOW GRANTS`](show-grants.html) | List the privileges granted to users. - -### Default roles - -The `admin` and `public` roles exist by default. - -#### `admin` role - -The `admin` role is created by default and cannot be dropped. Users belonging to the `admin` role have all privileges for all database objects across the cluster. The `root` user belongs to the `admin` role by default. - -An `admin` user is a member of the `admin` role. Only `admin` users can use [`CREATE ROLE`](create-role.html) and [`DROP ROLE`](drop-role.html). - -To assign a user to the `admin` role: - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT admin TO ; -~~~ - -#### `public` role - -All new users and roles belong to the `public` role by default. You can grant and revoke the privileges on the `public` role. - -### Terminology - -#### Role admin - -A `role admin` is a member of the role that's allowed to grant or revoke role membership to other users for that specific role. To create a `role admin`, use [`WITH ADMIN OPTION`](grant.html#grant-the-admin-option). - -{{site.data.alerts.callout_success}} -The terms “`admin` role” and “`role admin`” can be confusing. A user who is a member of the `admin` role has all privileges on all database objects across the entire cluster, whereas a `role admin` has privileges limited to the role they are a member of. Assign the `admin` role to a SQL user if you want the user to have privileges across the cluster. Make a SQL user the `role admin` if you want to limit the user’s privileges to its current role, but with an option to grant or revoke role membership to other users. This applies to the `admin` role as well - only admin users with the `WITH ADMIN OPTION` can add or remove other users from the `admin` role. -{{site.data.alerts.end}} - -#### Direct member - -A user or role that is an immediate member of the role. - -Example: A is a member of B. - -#### Indirect member - -A user or role that is a member of the role by association. - -Example: A is a member of C ... is a member of B where "..." is an arbitrary number of memberships. - -## Object ownership - -All CockroachDB objects (such as databases, tables, schemas, and types) must have owners. The user that created the object is the default owner of the object and has `ALL` privileges on the object. Similarly, any roles that are members of the owner role also have all privileges on the object. - -All objects that do not have owners (for example, objects created before upgrading to v20.2) have `admin` set as the default owner, with the exception of system objects. System objects without owners have `node` as their owner. - -To allow another user to use the object, the owner can [assign privileges](#assign-privileges) to the other user. Members of the `admin` role have `ALL` privileges on all objects. - -Users that [own objects](authorization.html#privileges) cannot be dropped until the [ownership is transferred to another user](owner-to.html#change-a-databases-owner). - -## Privileges - -When a user connects to a database, either via the built-in SQL client or a client driver, CockroachDB checks the user and role's privileges for each statement executed. If the user does not have sufficient privileges for a statement, CockroachDB gives an error. - -### Supported privileges - -Roles and users can be granted the following privileges: - -{% include {{ page.version.version }}/sql/privileges.md %} - -### Assign privileges - -Use the [`GRANT`](grant.html) and [`REVOKE`](revoke.html) statements to manage privileges for users and roles. - -Take the following points into consideration while granting privileges to roles and users: - -- When a role or user is granted privileges for a database, new tables created in the database will inherit the privileges, but the privileges can then be changed. To grant privileges to a user on all existing tables in a database, see [Grant privileges on all tables in a database](grant.html#grant-privileges-on-all-tables-in-a-database-or-schema) - - {{site.data.alerts.callout_info}} - The user does not get privileges to existing tables in the database. - {{site.data.alerts.end}} - -- When a role or user is granted privileges for a table, the privileges are limited to the table. -- In CockroachDB, privileges are granted to users and roles at the database and table levels. They are not yet supported for other granularities such as columns or rows. -- The `root` user automatically belongs to the `admin` role and has the `ALL` privilege for new databases. -- For privileges required by specific statements, see the documentation for the respective [SQL statement](sql-statements.html). - -## Authorization best practices - -We recommend the following best practices to set up access control for your clusters: - -- Use the `root` user only for database administration tasks such as creating and managing other [users](#sql-users), creating and managing [roles](#roles), and creating and managing databases. Do not use the `root` user for applications; instead, create users or roles with specific [privileges](#assign-privileges) based on your application’s access requirements. -- Use the ["least privilege model"](https://en.wikipedia.org/wiki/Principle_of_least_privilege) to grant privileges to users and roles. - -## Example - -
- - -
- -
- -The following example uses MovR, a fictional vehicle-sharing application, to demonstrate CockroachDB [SQL statements](sql-statements.html). For more information about the MovR example application and dataset, see [MovR: A Global Vehicle-sharing App](movr.html). - -Let's say we want to create the following access control setup for the `movr` database: - -- One database admin (named `db_admin`) who can perform all database operations for existing tables as well as for tables added in the future. -- One app user (named `app_user`) who can add, read update, and delete vehicles from the `vehicles` table. -- One user (named `report_user`) who can only read the `vehicles` table. - -1. Use the [`cockroach demo`](cockroach-demo.html) command to load the `movr` database and dataset into a CockroachDB cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach demo - ~~~ - -2. Create the database admin (named `db_admin`) who can perform all database operations for existing tables as well as for tables added in the future: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE USER db_admin; - ~~~ - -3. Grant all privileges on database `movr` to user `db_admin`: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > GRANT ALL ON DATABASE movr TO db_admin; - ~~~ - -4. Grant all privileges on all tables in database `movr` to user `db_admin`: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > GRANT ALL ON TABLE * TO db_admin; - ~~~ - -5. Verify that `db_admin` has all privileges: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW GRANTS FOR db_admin; - ~~~ - - ~~~ - database_name | schema_name | table_name | grantee | privilege_type - +---------------+--------------------+----------------------------+----------+----------------+ - movr | crdb_internal | NULL | db_admin | ALL - movr | information_schema | NULL | db_admin | ALL - movr | pg_catalog | NULL | db_admin | ALL - movr | public | NULL | db_admin | ALL - movr | public | promo_codes | db_admin | ALL - movr | public | rides | db_admin | ALL - movr | public | user_promo_codes | db_admin | ALL - movr | public | users | db_admin | ALL - movr | public | vehicle_location_histories | db_admin | ALL - movr | public | vehicles | db_admin | ALL - (10 rows) - ~~~ - -6. As the `root` user, create a SQL user named `app_user` with permissions to add, read, update, and delete vehicles in the `vehicles` table: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE USER app_user; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > GRANT INSERT, DELETE, UPDATE, SELECT ON vehicles TO app_user; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW GRANTS FOR app_user; - ~~~ - - ~~~ - database_name | schema_name | table_name | grantee | privilege_type - +---------------+-------------+------------+----------+----------------+ - movr | public | vehicles | app_user | DELETE - movr | public | vehicles | app_user | INSERT - movr | public | vehicles | app_user | SELECT - movr | public | vehicles | app_user | UPDATE - (4 rows) - ~~~ - -7. As the `root` user, create a SQL user named `report_user` with permissions to only read from the `vehicles` table: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE USER report_user; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > GRANT SELECT ON vehicles TO report_user; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW GRANTS FOR report_user; - ~~~ - - ~~~ - database_name | schema_name | table_name | grantee | privilege_type - +---------------+-------------+------------+-------------+----------------+ - movr | public | vehicles | report_user | SELECT - (1 row) - ~~~ - -
- -
- -The following example uses MovR, a fictional vehicle-sharing application, to demonstrate CockroachDB SQL statements. For more information about the MovR example application and dataset, see [MovR: A Global Vehicle-sharing App](movr.html). - -Let's say we want to create the following access control setup for the `movr` database: - -- Two database admins (named `db_admin_1` and `db_admin_2`) who can perform all database operations for existing tables as well as for tables added in the future. -- Three app users (named `app_user_1`, `app_user_2`, and `app_user_3`) who can add, read update, and delete vehicles from the `vehicles` table. -- Five users (named `report_user_1`, `report_user_2`, `report_user_3`, `report_user_4`, `report_user_5`) who can only read the `vehicles` table. - -1. Use the [`cockroach demo`](cockroach-demo.html) command to load the `movr` database and dataset into a CockroachDB cluster.: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach demo - ~~~ - -2. Create the database admin role (named `db_admin_role`) whose members can perform all database operations for existing tables as well as for tables added in the future: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE ROLE db_admin_role; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW ROLES; - ~~~ - - ~~~ - username | options | member_of - ----------------+------------+------------ - admin | CREATEROLE | {} - db_admin_role | NOLOGIN | {} - root | CREATEROLE | {admin} - (3 rows) - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > GRANT ALL ON DATABASE movr TO db_admin_role; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > GRANT ALL ON TABLE * TO db_admin_role; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW GRANTS ON DATABASE movr; - ~~~ - - ~~~ - database_name | schema_name | grantee | privilege_type - ----------------+--------------------+---------------+----------------- - movr | crdb_internal | admin | ALL - movr | crdb_internal | db_admin_role | ALL - movr | crdb_internal | root | ALL - movr | information_schema | admin | ALL - movr | information_schema | db_admin_role | ALL - movr | information_schema | root | ALL - movr | pg_catalog | admin | ALL - movr | pg_catalog | db_admin_role | ALL - movr | pg_catalog | root | ALL - movr | public | admin | ALL - movr | public | db_admin_role | ALL - movr | public | root | ALL - (12 rows) - ~~~ - -3. Create two database admin users (named `db_admin_1` and `db_admin_2`) and grant them membership to the `db_admin_role` role: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE USER db_admin_1; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE USER db_admin_2; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > GRANT db_admin_role TO db_admin_1, db_admin_2; - ~~~ - -4. Create a role named `app_user_role` whose members can add, read update, and delete vehicles to the `vehicles` table. - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE ROLE app_user_role; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW ROLES; - ~~~ - - ~~~ - username | options | member_of - ----------------+------------+------------------ - admin | CREATEROLE | {} - app_user_role | NOLOGIN | {} - db_admin_1 | | {db_admin_role} - db_admin_2 | | {db_admin_role} - db_admin_role | NOLOGIN | {} - root | CREATEROLE | {admin} - (6 rows) - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > GRANT INSERT, UPDATE, DELETE, SELECT ON TABLE vehicles TO app_user_role; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW GRANTS ON vehicles; - ~~~ - - ~~~ - database_name | schema_name | table_name | grantee | privilege_type - ----------------+-------------+------------+---------------+----------------- - movr | public | vehicles | admin | ALL - movr | public | vehicles | app_user_role | DELETE - movr | public | vehicles | app_user_role | INSERT - movr | public | vehicles | app_user_role | SELECT - movr | public | vehicles | app_user_role | UPDATE - movr | public | vehicles | db_admin_role | ALL - movr | public | vehicles | root | ALL - (7 rows) - ~~~ - -5. Create three app users (named `app_user_1`, `app_user_2`, and `app_user_3`) and grant them membership to the `app_user_role` role: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE USER app_user_1; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE USER app_user_2; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE USER app_user_3; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > GRANT app_user_role TO app_user_1, app_user_2, app_user_3; - ~~~ - -6. Create a role named `report_user_role` whose members can only read the `vehicles` table. - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE ROLE report_user_role; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW ROLES; - ~~~ - - ~~~ - username | options | member_of - -------------------+------------+------------------ - admin | CREATEROLE | {} - app_user_1 | | {app_user_role} - app_user_2 | | {app_user_role} - app_user_3 | | {app_user_role} - app_user_role | NOLOGIN | {} - db_admin_1 | | {db_admin_role} - db_admin_2 | | {db_admin_role} - db_admin_role | NOLOGIN | {} - report_user_role | NOLOGIN | {} - root | CREATEROLE | {admin} - (10 rows) - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > GRANT SELECT ON vehicles TO report_user_role; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW GRANTS ON vehicles; - ~~~ - - ~~~ - database_name | schema_name | table_name | grantee | privilege_type - ----------------+-------------+------------+------------------+----------------- - movr | public | vehicles | admin | ALL - movr | public | vehicles | app_user_role | DELETE - movr | public | vehicles | app_user_role | INSERT - movr | public | vehicles | app_user_role | SELECT - movr | public | vehicles | app_user_role | UPDATE - movr | public | vehicles | db_admin_role | ALL - movr | public | vehicles | report_user_role | SELECT - movr | public | vehicles | root | ALL - (8 rows) - ~~~ - -7. Create five report users (named `report_user_1`, `report_user_2`, `report_user_3`, `report_user_4`, and `report_user_5`) and grant them membership to the `report_user_role` role: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE USER report_user_1; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE USER report_user_2; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE USER report_user_3; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE USER report_user_4; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE USER report_user_5; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > GRANT report_user_role TO report_user_1, report_user_2, report_user_3, report_user_4, report_user_5; - ~~~ - -
- -## See also - -- [Client Connection Parameters](connection-parameters.html) -- [SQL Statements](sql-statements.html) -- [`CREATE USER`](create-user.html) -- [`ALTER USER`](alter-user.html) -- [`DROP USER`](drop-user.html) -- [`SHOW USERS`](show-users.html) -- [`CREATE ROLE`](create-role.html) -- [`DROP ROLE`](drop-role.html) -- [`SHOW ROLES`](show-roles.html) -- [`GRANT`](grant.html) -- [`REVOKE`](revoke.html) -- [`SHOW GRANTS`](show-grants.html) diff --git a/src/current/v21.1/backup.md b/src/current/v21.1/backup.md deleted file mode 100644 index bc646c3e483..00000000000 --- a/src/current/v21.1/backup.md +++ /dev/null @@ -1,287 +0,0 @@ ---- -title: BACKUP -summary: Back up your CockroachDB cluster to a cloud storage services such as AWS S3, Google Cloud Storage, or other NFS. -toc: true ---- - -CockroachDB's `BACKUP` [statement](sql-statements.html) allows you to create [full or incremental backups](take-full-and-incremental-backups.html) of your cluster's schema and data that are consistent as of a given timestamp. - -You can [backup a full cluster](#backup-a-cluster), which includes: - -- Relevant system tables -- All [databases](create-database.html) -- All [tables](create-table.html) (which automatically includes their [indexes](indexes.html)) -- All [views](views.html) -- All [scheduled jobs](manage-a-backup-schedule.html#view-and-control-a-backup-initiated-by-a-schedule) - -You can also back up: - -- [An individual database](#backup-a-database), which includes all of its tables and views -- [An individual table](#backup-a-table-or-view), which includes its indexes and views - - `BACKUP` only backs up entire tables; it **does not** support backing up subsets of a table. - -Because CockroachDB is designed with high fault tolerance, these backups are designed primarily for disaster recovery (i.e., if your cluster loses a majority of its nodes) through [`RESTORE`](restore.html). Isolated issues (such as small-scale node outages) do not require any intervention. - -{{site.data.alerts.callout_info}} -The [`BACKUP ... TO`](../v20.2/backup.html) and [`RESTORE ... FROM`](../v20.2/restore.html) syntax is **deprecated** as of v22.1 and will be removed in a future release. - -We recommend using the `BACKUP ... INTO {collectionURI}` syntax, which creates or adds to a [backup collection](take-full-and-incremental-backups.html#backup-collections) in your storage location. For restoring backups, we recommend using `RESTORE FROM {backup} IN {collectionURI}` with `{backup}` being [`LATEST`](restore.html#restore-the-most-recent-backup) or a specific [subdirectory](restore.html#subdir-param). - -For guidance on the syntax for backups and restores, see the [`BACKUP`](backup.html#examples) and [`RESTORE`](restore.html#examples) examples. -{{site.data.alerts.end}} - -## Considerations - -- Core users can only take [full backups](take-full-and-incremental-backups.html#full-backups). To use the other backup features, you need an [Enterprise license](enterprise-licensing.html). You can also use [CockroachDB {{ site.data.products.dedicated }}](https://cockroachlabs.cloud/signup?referralId=docs-crdb-backup), which runs [full backups daily and incremental backups hourly](../cockroachcloud/use-managed-service-backups.html). -- `BACKUP` is a blocking statement. To run a backup job asynchronously, use the `DETACHED` option. See the [options](#options) below. -- Backups will export [Enterprise license keys](enterprise-licensing.html) during a [full cluster backup](#backup-a-cluster). When you [restore](restore.html) a full cluster with an Enterprise license, it will restore the Enterprise license of the cluster you are restoring from. -- [Interleaving data](interleave-in-parent.html) is disabled in v21.1 by default, and will be permanently removed from CockroachDB in a future release. CockroachDB versions v21.2 and later will not be able to read or restore backups that include interleaved data. To backup interleaved data in v21.1, a `BACKUP` statement must include the [`INCLUDE_DEPRECATED_INTERLEAVES` option](#include-deprecated-interleaves). -- [Zone configurations](configure-zone.html) present on the destination cluster prior to a restore will be **overwritten** during a [cluster restore](restore.html#full-cluster) with the zone configurations from the [backed-up cluster](#backup-a-cluster). If there were no customized zone configurations on the cluster when the backup was taken, then after the restore the destination cluster will use the zone configuration from the [`RANGE DEFAULT` configuration](configure-replication-zones.html#view-the-default-replication-zone). -- You cannot restore a backup of a multi-region database into a single-region database. - -{{site.data.alerts.callout_success}} -To view the contents of an Enterprise backup created with the `BACKUP` statement, use [`SHOW BACKUP`](show-backup.html). -{{site.data.alerts.end}} - -## Required privileges - -- [Full cluster backups](take-full-and-incremental-backups.html#full-backups) can only be run by members of the [`admin` role](authorization.html#admin-role). By default, the `root` user belongs to the `admin` role. -- For all other backups, the user must have [read access](authorization.html#assign-privileges) on all objects being backed up. Database and table backups require `SELECT` privileges. Backups of user-defined schemas, or backups containing user-defined types, require `USAGE` privileges. -- `BACKUP` requires full read and write (including delete and overwrite) permissions to its target destination. - -### Destination privileges - -{% include {{ page.version.version }}/backups/destination-file-privileges.md %} - -## Synopsis - -
-{% include {{ page.version.version }}/sql/generated/diagrams/backup.html %} -
- -## Parameters - - Parameter | Description ------------+------------- -`targets` | Back up the listed [targets](#targets). -`subdirectory` | The name of the specific backup (e.g., `2021/03/23-213101.37`) in the collection to which you want to add an [incremental backup](take-full-and-incremental-backups.html#incremental-backups). To view available backup subdirectories, use [`SHOW BACKUPS IN destination`](show-backup.html). If the backup `subdirectory` is not provided, a [full backup](take-full-and-incremental-backups.html#full-backups) will be created in the collection using a date-based naming scheme (i.e., `//-`).

**Warning:** If you use an arbitrary `STRING` as the subdirectory, a new full backup will be created, but it will never be shown in `SHOW BACKUPS IN`. We do not recommend using arbitrary strings as subdirectory names. -`LATEST` | Append an incremental backup to the latest completed full backup's subdirectory. -`destination` | The URL where you want to store the backup.

For information about this URL structure, see [Backup File URLs](#backup-file-urls). -`timestamp` | Back up data as it existed as of [`timestamp`](as-of-system-time.html). The `timestamp` must be more recent than your cluster's last garbage collection (which defaults to occur every 25 hours, but is [configurable per table](configure-replication-zones.html#replication-zone-variables)). -`backup_options` | Control the backup behavior with a comma-separated list of [these options](#options). - -### Targets - -Target | Description ------------------------------------+------------------------------------------------------------------------- -N/A | Backup the cluster. For an example of a full cluster backup, [see Backup a cluster](#backup-a-cluster). -`DATABASE {database_name} [, ...]` | The name of the database(s) you want to back up (i.e., create backups of all tables and views in the database). For an example of backing up a database, see [Backup a database](#backup-a-database). -`TABLE {table_name} [, ...]` | The name of the table(s) or [view(s)](views.html) you want to back up. For an example of backing up a table or view, see [Backup a table or view](#backup-a-table-or-view). - -### Options - -{% include {{ page.version.version }}/backups/backup-options.md %} - -### Backup file URLs - -CockroachDB uses the URL provided to construct a secure API call to the service you specify. The URL structure depends on the type of file storage you are using. For more information, see the following: - -- [Use Cloud Storage for Bulk Operations](use-cloud-storage-for-bulk-operations.html) - - {{site.data.alerts.callout_info}} - HTTP storage is not supported for `BACKUP` and `RESTORE`. - {{site.data.alerts.end}} - -- [Use a Local File Server for Bulk Operations](use-a-local-file-server-for-bulk-operations.html) - -## Functional details - -### Object dependencies - -Dependent objects must be backed up at the same time as the objects they depend on. - -Object | Depends On --------|----------- -Table with [foreign key](foreign-key.html) constraints | The table it `REFERENCES`; however, this dependency can be [removed during the restore](restore.html#skip_missing_foreign_keys). -Table with a [sequence](create-sequence.html) | The sequence it uses; however, this dependency can be [removed during the restore](restore.html#skip_missing_sequences). -[Views](views.html) | The tables used in the view's `SELECT` statement. -[Interleaved tables](interleave-in-parent.html) | The parent table in the [interleaved hierarchy](interleave-in-parent.html). - -### Users and privileges - -The `system.users` table stores your users and their passwords. To restore your users and privilege [grants](grant.html), do a cluster backup and restore the cluster to a fresh cluster with no user data. You can also backup the `system.users` table, and then use [this procedure](restore.html#restoring-users-from-system-users-backup). - -## Performance - -The `BACKUP` process minimizes its impact to the cluster's performance by distributing work to all nodes. Each node backs up only a specific subset of the data it stores (those for which it serves writes; more details about this architectural concept forthcoming), with no two nodes backing up the same data. - -`BACKUP`, like any read, cannot export a range if the range contains an [unresolved intent](architecture/transaction-layer.html#resolving-write-intents). While you typically will want bulk, background jobs like `BACKUP` to have as little impact on your foreground traffic as possible, it's more important for backups to actually complete (which maintains your [recovery point objective (RPO)](https://en.wikipedia.org/wiki/Disaster_recovery#Recovery_Point_Objective)). Unlike a normal read transaction that will block until any uncommitted writes it encounters are resolved, `BACKUP` will block only for a configurable duration before invoking priority to ensure it can complete on-time. - -We recommend always starting backups with a specific [timestamp](timestamp.html) at least 10 seconds in the past. For example: - -~~~ sql -> BACKUP...AS OF SYSTEM TIME '-10s'; -~~~ - -This improves performance by decreasing the likelihood that the `BACKUP` will be [retried because it contends with other statements/transactions](transactions.html#transaction-retries). However, because `AS OF SYSTEM TIME` returns historical data, your reads might be stale. Taking backups with `AS OF SYSTEM TIME '-10s'` is a good best practice to reduce the number of still-running transactions you may encounter, since the backup will take priority and will force still-running transactions to restart after the backup is finished. - -`BACKUP` will initially ask individual ranges to backup but to skip if they encounter an intent. Any range that is skipped is placed at the end of the queue. When `BACKUP` has completed its initial pass and is revisiting ranges, it will ask any range that did not resolve within the given time limit (default 1 minute) to attempt to resolve any intents that it encounters and to _not_ skip. Additionally, the backup's transaction priority is then set to `high`, which causes other transactions to abort until the intents are resolved and the backup is finished. - -## Viewing and controlling backups jobs - -After CockroachDB successfully initiates a backup, it registers the backup as a job, and you can do the following: - - Action | SQL Statement ------------------------+----------------- -View the backup status | [`SHOW JOBS`](show-jobs.html) -Pause the backup | [`PAUSE JOB`](pause-job.html) -Resume the backup | [`RESUME JOB`](resume-job.html) -Cancel the backup | [`CANCEL JOB`](cancel-job.html) - -You can also visit the [**Jobs** page](ui-jobs-page.html) of the DB Console to view job details. The `BACKUP` statement will return when the backup is finished or if it encounters an error. - -{{site.data.alerts.callout_info}} -The presence of the `BACKUP MANIFEST` file in the backup subdirectory is an indicator that the backup job completed successfully. -{{site.data.alerts.end}} - -## Examples - -Per our guidance in the [Performance](#performance) section, we recommend starting backups from a time at least 10 seconds in the past using [`AS OF SYSTEM TIME`](as-of-system-time.html). - -{% include {{ page.version.version }}/backups/bulk-auth-options.md %} - -{{site.data.alerts.callout_info}} -{% include_cached new-in.html version="v21.1" %} The syntax `BACKUP ... INTO` adds a backup to a collection within the backup destination. The path to the backup is created using a date-based naming scheme. Versions of CockroachDB prior to v21.1 used the syntax `BACKUP ... TO` to backup directly to a specific operator-chosen destination, rather than picking a date-based path. The `BACKUP ... TO` syntax will be **deprecated** in future releases. For more information on this soon-to-be deprecated syntax, [see the docs for v20.2](../v20.2/backup.html) or earlier. -{{site.data.alerts.end}} - -### Backup a cluster - -To take a [full backup](take-full-and-incremental-backups.html#full-backups) of a cluster: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BACKUP INTO \ -'s3://{BUCKET NAME}/{PATH}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}' \ -AS OF SYSTEM TIME '-10s'; -~~~ - -### Backup a database - -To take a [full backup](take-full-and-incremental-backups.html#full-backups) of a single database: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BACKUP DATABASE bank \ -INTO 's3://{BUCKET NAME}/{PATH}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}' \ -AS OF SYSTEM TIME '-10s'; -~~~ - -To take a [full backup](take-full-and-incremental-backups.html#full-backups) of multiple databases: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BACKUP DATABASE bank, employees \ -INTO 's3://{BUCKET NAME}/{PATH}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}' \ -AS OF SYSTEM TIME '-10s'; -~~~ - -### Backup a table or view - -To take a [full backup](take-full-and-incremental-backups.html#full-backups) of a single table or view: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BACKUP bank.customers \ -INTO 's3://{BUCKET NAME}/{PATH}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}' \ -AS OF SYSTEM TIME '-10s'; -~~~ - -To take a [full backup](take-full-and-incremental-backups.html#full-backups) of multiple tables: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BACKUP bank.customers, bank.accounts \ -INTO 's3://{BUCKET NAME}/{PATH}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}' \ -AS OF SYSTEM TIME '-10s'; -~~~ - -{{site.data.alerts.callout_danger}} -{% include {{page.version.version}}/backups/no-multiregion-table-backups.md %} -{{site.data.alerts.end}} - -### Specify a subdirectory for backups - -To store the backup in a specific subdirectory in the storage location: - -{% include_cached copy-clipboard.html %} -~~~ sql -BACKUP DATABASE bank INTO 'subdirectory' IN 's3://{BUCKET NAME}/{PATH}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}' \ -AS OF SYSTEM TIME '-10s'; -~~~ - -### Create incremental backups - -If you backup to a destination already containing a [full backup](take-full-and-incremental-backups.html#full-backups), an [incremental backup](take-full-and-incremental-backups.html#incremental-backups) will be appended to the full backup's path with a date-based name (e.g., `20210324`): - -{% include_cached copy-clipboard.html %} -~~~ sql -> BACKUP INTO LATEST IN \ -'s3://{BUCKET NAME}/{PATH}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}' \ -AS OF SYSTEM TIME '-10s'; -~~~ - -### Run a backup asynchronously - -Use the `DETACHED` [option](#options) to execute the backup [job](show-jobs.html) asynchronously: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BACKUP INTO \ -'s3://{BUCKET NAME}/{PATH}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}' \ -AS OF SYSTEM TIME '-10s' -WITH DETACHED; -~~~ - -The job ID is returned after the backup job creation completes: - -~~~ - job_id ----------------------- - 592786066399264769 -(1 row) -~~~ - -**Without** the `DETACHED` option, `BACKUP` will block the SQL connection until the job completes. Once finished, the job status and more detailed job data is returned: - -~~~ -job_id | status | fraction_completed | rows | index_entries | bytes ----------------------+-----------+--------------------+------+---------------+-------- -652471804772712449 | succeeded | 1 | 50 | 0 | 4911 -(1 row) -~~~ - -### Advanced examples - -{% include {{ page.version.version }}/backups/advanced-examples-list.md %} - -## Known limitations - -### Using interleaved tables in backups - -{% include {{ page.version.version }}/known-limitations/backup-interleaved.md %} - -### `BACKUP` of multi-region tables - -{% include {{page.version.version}}/backups/no-multiregion-table-backups.md %} - -## See also - -- [Take Full and Incremental Backups](take-full-and-incremental-backups.html) -- [Take and Restore Encrypted Backups](take-and-restore-encrypted-backups.html) -- [Take and Restore Locality-aware Backups](take-and-restore-locality-aware-backups.html) -- [Take Backups with Revision History and Restore from a Point-in-time](take-backups-with-revision-history-and-restore-from-a-point-in-time.html) -- [`SHOW BACKUP`](show-backup.html) -- [`CREATE SCHEDULE FOR BACKUP`](create-schedule-for-backup.html) -- [`RESTORE`](restore.html) -- [Configure Replication Zones](configure-replication-zones.html) diff --git a/src/current/v21.1/begin-transaction.md b/src/current/v21.1/begin-transaction.md deleted file mode 100644 index 8a0a170ecf1..00000000000 --- a/src/current/v21.1/begin-transaction.md +++ /dev/null @@ -1,171 +0,0 @@ ---- -title: BEGIN -summary: Initiate a SQL transaction with the BEGIN statement in CockroachDB. -toc: true ---- - -The `BEGIN` [statement](sql-statements.html) initiates a [transaction](transactions.html), which either successfully executes all of the statements it contains or none at all. - -{{site.data.alerts.callout_danger}} -When using transactions, your application should include logic to [retry transactions](transactions.html#transaction-retries) that are aborted to break a dependency cycle between concurrent transactions. -{{site.data.alerts.end}} - - -## Synopsis - -
-{% include {{ page.version.version }}/sql/generated/diagrams/begin_transaction.html %} -
- -## Required privileges - -No [privileges](authorization.html#assign-privileges) are required to initiate a transaction. However, privileges are required for each statement within a transaction. - -## Aliases - -In CockroachDB, the following are aliases for the `BEGIN` statement: - -- `BEGIN TRANSACTION` -- `START TRANSACTION` - -## Parameters - - Parameter | Description ------------|------------- -`PRIORITY` | If you do not want the transaction to run with `NORMAL` priority, you can set it to `LOW` or `HIGH`.

Transactions with higher priority are less likely to need to be retried.

For more information, see [Transactions: Priorities](transactions.html#transaction-priorities).

**Default**: `NORMAL` -`READ` | Set the transaction access mode to `READ ONLY` or `READ WRITE`. The current transaction access mode is also exposed as the [session variable](show-vars.html) `transaction_read_only`.

**Default**: `READ WRITE` -`AS OF SYSTEM TIME` | Execute the transaction using the database contents "as of" a specified time in the past.

The `AS OF SYSTEM TIME` clause can be used only when the transaction is read-only. If the transaction contains any writes, or if the `READ WRITE` mode is specified, an error will be returned.

For more information, see [AS OF SYSTEM TIME](as-of-system-time.html). -`NOT DEFERRABLE`
`DEFERRABLE` | This clause is supported for compatibility with PostgreSQL. `NOT DEFERRABLE` is a no-op and the default behavior for CockroachDB. `DEFERRABLE` returns an `unimplemented` error. - - CockroachDB now only supports `SERIALIZABLE` isolation, so transactions can no longer be meaningfully set to any other `ISOLATION LEVEL`. In previous versions of CockroachDB, you could set transactions to `SNAPSHOT` isolation, but that feature has been removed. - -## Examples - -### Begin a transaction - -#### Use default settings - -Without modifying the `BEGIN` statement, the transaction uses `SERIALIZABLE` isolation and `NORMAL` priority. - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SAVEPOINT cockroach_restart; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE products SET inventory = 0 WHERE sku = '8675309'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO orders (customer, sku, status) VALUES (1001, '8675309', 'new'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> RELEASE SAVEPOINT cockroach_restart; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> COMMIT; -~~~ - -{{site.data.alerts.callout_danger}}This example assumes you're using client-side intervention to handle transaction retries.{{site.data.alerts.end}} - -#### Change priority - -You can set a transaction's priority to `LOW` or `HIGH`. - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN PRIORITY HIGH; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SAVEPOINT cockroach_restart; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE products SET inventory = 0 WHERE sku = '8675309'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO orders (customer, sku, status) VALUES (1001, '8675309', 'new'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> RELEASE SAVEPOINT cockroach_restart; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> COMMIT; -~~~ - -You can also set a transaction's priority with [`SET TRANSACTION`](set-transaction.html). - -{{site.data.alerts.callout_danger}} -This example assumes you're using [client-side intervention to handle transaction retries](transactions.html#client-side-intervention). -{{site.data.alerts.end}} - -### Use the `AS OF SYSTEM TIME` option - -You can execute the transaction using the database contents "as of" a specified time in the past. - -{% include {{ page.version.version }}/sql/begin-transaction-as-of-system-time-example.md %} - -{{site.data.alerts.callout_success}} -You can also use the [`SET TRANSACTION`](set-transaction.html#use-the-as-of-system-time-option) statement inside the transaction to achieve the same results. This syntax is easier to use from [drivers and ORMs](install-client-drivers.html). -{{site.data.alerts.end}} - -### Begin a transaction with automatic retries - -CockroachDB will [automatically retry](transactions.html#transaction-retries) all transactions that contain both `BEGIN` and `COMMIT` in the same batch. Batching is controlled by your driver or client's behavior, but means that CockroachDB receives all of the statements as a single unit, instead of a number of requests. - -From the perspective of CockroachDB, a transaction sent as a batch looks like this: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; - -> DELETE FROM customers WHERE id = 1; - -> DELETE orders WHERE customer = 1; - -> COMMIT; -~~~ - -However, in your application's code, batched transactions are often just multiple statements sent at once. For example, in Go, this transaction would sent as a single batch (and automatically retried): - -~~~ go -db.Exec( - "BEGIN; - - DELETE FROM customers WHERE id = 1; - - DELETE orders WHERE customer = 1; - - COMMIT;" -) -~~~ - -Issuing statements this way signals to CockroachDB that you do not need to change any of the statement's values if the transaction doesn't immediately succeed, so it can continually retry the transaction until it's accepted. - -## See also - -- [Transactions](transactions.html) -- [`COMMIT`](commit-transaction.html) -- [`SAVEPOINT`](savepoint.html) -- [`RELEASE SAVEPOINT`](release-savepoint.html) -- [`ROLLBACK`](rollback-transaction.html) diff --git a/src/current/v21.1/bit.md b/src/current/v21.1/bit.md deleted file mode 100644 index c38a9a391e5..00000000000 --- a/src/current/v21.1/bit.md +++ /dev/null @@ -1,124 +0,0 @@ ---- -title: BIT -summary: The BIT and BIT VARYING data types stores bit arrays. -toc: true ---- - -The `BIT` and `VARBIT` [data types](data-types.html) stores bit arrays. -With `BIT`, the length is fixed; with `VARBIT`, the length can be variable. - -## Aliases - -The name `BIT VARYING` is an alias for `VARBIT`. - -## Syntax - -Bit array constants are expressed as literals. For example, `B'100101'` denotes an array of 6 bits. - -For more information about bit array constants, see the [constants documentation on bit array literals](sql-constants.html#bit-array-literals). - -For usage, see the [Example](#example) below. - -## Size - -The number of bits in a `BIT` value is determined as follows: - -| Type declaration | Logical size | -|------------------|-----------------------------------| -| BIT | 1 bit | -| BIT(N) | N bits | -| VARBIT | variable with no maximum | -| VARBIT(N) | variable with a maximum of N bits | - -The effective size of a `BIT` value is larger than its logical number -of bits by a bounded constant factor. Internally, CockroachDB stores -bit arrays in increments of 64 bits plus an extra integer value to -encode the length. - -The total size of a `BIT` value can be arbitrarily large, but it is -recommended to keep values under 1 MB to ensure performance. Above -that threshold, [write -amplification](https://en.wikipedia.org/wiki/Write_amplification) and -other considerations may cause significant performance degradation. - -## Example - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE b (x BIT, y BIT(3), z VARBIT, w VARBIT(3)); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM b; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden -+-------------+-----------+-------------+----------------+-----------------------+-----------+-----------+ - x | BIT | true | NULL | | {} | false - y | BIT(3) | true | NULL | | {} | false - z | VARBIT | true | NULL | | {} | false - w | VARBIT(3) | true | NULL | | {} | false - rowid | INT | false | unique_rowid() | | {primary} | true -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO b(x, y, z, w) VALUES (B'1', B'101', B'1', B'1'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM b; -~~~ - -~~~ - x | y | z | w -+---+-----+---+---+ - 1 | 101 | 1 | 1 -~~~ - -For type `BIT`, the value must match exactly the specified size: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO b(x) VALUES (B'101'); -~~~ - -~~~ -pq: bit string length 3 does not match type BIT -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO b(y) VALUES (B'10'); -~~~ - -~~~ -pq: bit string length 2 does not match type BIT(3) -~~~ - -For type `VARBIT`, the value must not be larger than the specified maximum size: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO b(w) VALUES (B'1010'); -~~~ - -~~~ -pq: bit string length 4 too large for type VARBIT(3) -~~~ - -## Supported casting and conversion - -`BIT` values can be [cast](data-types.html#data-type-conversions-and-casts) to any of the following data types: - -Type | Details ------|--------- -`INT` | Converts the bit array to the corresponding numeric value, interpreting the bits as if the value was encoded using [two's complement](https://en.wikipedia.org/wiki/Two%27s_complement). If the bit array is larger than the integer type, excess bits on the left are ignored. For example, `B'1010'::INT` equals `10`. -`STRING` | Prints out the binary digits as a string. This recovers the literal representation. For example, `B'1010'::STRING` equals `'1010'`. - -## See also - -[Data Types](data-types.html) diff --git a/src/current/v21.1/bool.md b/src/current/v21.1/bool.md deleted file mode 100644 index c87e318358f..00000000000 --- a/src/current/v21.1/bool.md +++ /dev/null @@ -1,81 +0,0 @@ ---- -title: BOOL -summary: The BOOL data type stores Boolean values of false or true. -toc: true ---- - -The `BOOL` [data type](data-types.html) stores a Boolean value of `false` or `true`. - - -## Aliases - -In CockroachDB, `BOOLEAN` is an alias for `BOOL`. - -## Syntax - -There are two predefined [named constants](sql-constants.html#named-constants) for `BOOL`: `TRUE` and `FALSE` (the names are case-insensitive). - -Alternately, a boolean value can be obtained by coercing a numeric value: zero is coerced to `FALSE`, and any non-zero value to `TRUE`. - -- `CAST(0 AS BOOL)` (false) -- `CAST(123 AS BOOL)` (true) - -## Size - -A `BOOL` value is 1 byte in width, but the total storage size is likely to be larger due to CockroachDB metadata. - -## Examples - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE bool (a INT PRIMARY KEY, b BOOL, c BOOLEAN); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM bool; -~~~ - -~~~ -+-------------+-----------+-------------+----------------+-----------------------+-------------+ -| column_name | data_type | is_nullable | column_default | generation_expression | indices | -+-------------+-----------+-------------+----------------+-----------------------+-------------+ -| a | INT | false | NULL | | {"primary"} | -| b | BOOL | true | NULL | | {} | -| c | BOOL | true | NULL | | {} | -+-------------+-----------+-------------+----------------+-----------------------+-------------+ -(3 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO bool VALUES (12345, true, CAST(0 AS BOOL)); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM bool; -~~~ - -~~~ -+-------+------+-------+ -| a | b | c | -+-------+------+-------+ -| 12345 | true | false | -+-------+------+-------+ -~~~ - -## Supported casting and conversion - -`BOOL` values can be [cast](data-types.html#data-type-conversions-and-casts) to any of the following data types: - -Type | Details ------|-------- -`INT` | Converts `true` to `1`, `false` to `0` -`DECIMAL` | Converts `true` to `1`, `false` to `0` -`FLOAT` | Converts `true` to `1`, `false` to `0` -`STRING` | –– - -## See also - -[Data Types](data-types.html) diff --git a/src/current/v21.1/build-a-csharp-app-with-cockroachdb.md b/src/current/v21.1/build-a-csharp-app-with-cockroachdb.md deleted file mode 100644 index 3dd23a0d657..00000000000 --- a/src/current/v21.1/build-a-csharp-app-with-cockroachdb.md +++ /dev/null @@ -1,208 +0,0 @@ ---- -title: Build a C# App with CockroachDB and the .NET Npgsql Driver -summary: Learn how to use CockroachDB from a simple C# (.NET) application with a low-level client driver. -toc: true -twitter: true ---- - -This tutorial shows you how build a simple C# application with CockroachDB and the .NET Npgsql driver. - -We have tested the [.NET Npgsql driver](http://www.npgsql.org/) enough to claim **beta-level** support. If you encounter problems, please [open an issue](https://github.com/cockroachdb/cockroach/issues/new) with details to help us make progress toward full support. - -## Step 1. Start CockroachDB - -{% include {{page.version.version}}/app/start-cockroachdb.md %} - -## Step 2. Create a .NET project - -In your terminal, run the following commands: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ dotnet new console -o cockroachdb-test-app -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cd cockroachdb-test-app -~~~ - -The `dotnet` command creates a new app of type `console`. The `-o` parameter creates a directory named `cockroachdb-test-app` where your app will be stored and populates it with the required files. The `cd cockroachdb-test-app` command puts you into the newly created app directory. - -## Step 3. Install the Npgsql driver - -Install the latest version of the [Npgsql driver](https://www.nuget.org/packages/Npgsql/) into the .NET project using the built-in nuget package manager: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ dotnet add package Npgsql -~~~ - -## Step 4. Create a database - -{% include {{page.version.version}}/app/create-a-database.md %} - -## Step 5. Run the C# code - -Now that you have set up your project and created a database, in this section you will: - -- [Create a table and insert some rows](#basic-example) -- [Execute a batch of statements as a transaction](#transaction-example-with-retry-logic) - -### Basic example - -#### Get the code - -Replace the contents of the `Program.cs` file that was automatically generated in your `cockroachdb-test-app` directory with the code below: - -{{site.data.alerts.callout_info}} -The following examples use the SSL mode `require` because the .NET Npgsql driver validates certificates differently from other PostgreSQL drivers. For other drivers, we recommend using `verify-full` as a security best practice. -{{site.data.alerts.end}} - -
- -{% include_cached copy-clipboard.html %} -~~~ c# -{% remote_include https://raw.githubusercontent.com/cockroachlabs/hello-world-csharp/main/basic.cs %} -~~~ - -
- -
- -{% include_cached copy-clipboard.html %} -~~~ c# -{% remote_include https://raw.githubusercontent.com/cockroachlabs/hello-world-csharp/cockroachcloud/basic.cs %} -~~~ - -
- -#### Update the connection parameters - -
- -In a text editor, modify `Program.cs` with the settings to connect to the demo cluster: - -{% include_cached copy-clipboard.html %} -~~~ csharp -connStringBuilder.Host = "{localhost}"; -connStringBuilder.Port = 26257; -connStringBuilder.SslMode = SslMode.Require; -connStringBuilder.Username = "{username}"; -connStringBuilder.Password = "{password}"; -connStringBuilder.Database = "bank"; -connStringBuilder.TrustServerCertificate = true; -~~~ - -Where `{username}` and `{password}` are the database username and password you created earlier. - -
- -
- -1. In the CockroachDB {{ site.data.products.cloud }} Console, select the **Connection Parameters** tab of the **Connection Info** dialog. - -1. In a text editor, modify the connection parameters in `Program.cs` with the settings to connect to your cluster: - -{% include_cached copy-clipboard.html %} -~~~ csharp -connStringBuilder.Host = "{host-name}"; -connStringBuilder.Port = 26257; -connStringBuilder.SslMode = SslMode.Require; -connStringBuilder.Username = "{username}"; -connStringBuilder.Password = "{password}"; -connStringBuilder.Database = "{cluster-name}.bank"; -connStringBuilder.RootCertificate = "~/.postgresql/root.crt"; -connStringBuilder.TrustServerCertificate = true; -~~~ - -Where: - -- `{username}` and `{password}` specify the SQL username and password that you created earlier. -- `{host-name}` is the name of the CockroachDB {{ site.data.products.serverless }} host (e.g., `free-tier.gcp-us-central1.cockroachlabs.cloud`). -- `{cluster_name}` is the name of your cluster. - -
- -#### Run the code - -Compile and run the code: - -{% include_cached copy-clipboard.html %} -~~~ shell -dotnet run -~~~ - -The output should be: - -~~~ -Initial balances: - account 1: 1000 - account 2: 250 -~~~ - -### Transaction example (with retry logic) - -#### Get the code - -Open `cockroachdb-test-app/Program.cs` again and replace the contents with the code shown below. Make sure to keep the connection parameters the same as in the [previous example](#update-the-connection-parameters). - -
- -{% include_cached copy-clipboard.html %} -~~~ c# -{% remote_include https://raw.githubusercontent.com/cockroachlabs/hello-world-csharp/main/transaction.cs %} -~~~ - -
- -
- -{% include_cached copy-clipboard.html %} -~~~ c# -{% remote_include https://raw.githubusercontent.com/cockroachlabs/hello-world-csharp/cockroachcloud/transaction.cs %} -~~~ - -
- -#### Run the code - -This time, running the code will execute a batch of statements as an atomic transaction to transfer funds from one account to another, where all included statements are either committed or aborted: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ dotnet run -~~~ - -The output should be: - -~~~ -Initial balances: - account 1: 1000 - account 2: 250 -Final balances: - account 1: 900 - account 2: 350 -~~~ - -However, if you want to verify that funds were transferred from one account to another, use the [built-in SQL client](cockroach-sql.html): - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --insecure --database=bank -e 'SELECT id, balance FROM accounts' -~~~ - -~~~ - id | balance -+----+---------+ - 1 | 900 - 2 | 350 -(2 rows) -~~~ - - -## What's next? - -Read more about using the [.NET Npgsql driver](http://www.npgsql.org/). - -{% include {{ page.version.version }}/app/see-also-links.md %} diff --git a/src/current/v21.1/build-a-go-app-with-cockroachdb-gorm.md b/src/current/v21.1/build-a-go-app-with-cockroachdb-gorm.md deleted file mode 100644 index 5022ca63332..00000000000 --- a/src/current/v21.1/build-a-go-app-with-cockroachdb-gorm.md +++ /dev/null @@ -1,132 +0,0 @@ ---- -title: Build a Go App with CockroachDB and GORM -summary: Learn how to use CockroachDB from a simple Go application with the GORM ORM. -toc: true -twitter: false -referral_id: docs_hello_world_go_gorm ---- - -
- - - - -
- -This tutorial shows you how build a simple CRUD Go application with CockroachDB and the [GORM ORM](https://gorm.io/index.html). - -{{site.data.alerts.callout_success}} -For another use of GORM with CockroachDB, see our [`examples-orms`](https://github.com/cockroachdb/examples-orms) repository. -{{site.data.alerts.end}} - -## Step 1. Start CockroachDB - -{% include {{ page.version.version }}/app/sample-setup.md %} - -## Step 2. Get the code - -Clone the code's GitHub repo: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ git clone https://github.com/cockroachlabs/example-app-go-gorm -~~~ - -The project has the following directory structure: - -~~~ -├── README.md -└── main.go -~~~ - -The `main.go` file defines an `Account` struct that maps to a new `accounts` table in the `bank` database. The file also contains some read and write database operations that are executed in the `main` method of the program. - -{% include_cached copy-clipboard.html %} -~~~ go -{% remote_include https://raw.githubusercontent.com/cockroachlabs/example-app-go-gorm/master/main.go %} -~~~ - -{{site.data.alerts.callout_info}} -CockroachDB may require the [client to retry a transaction](transactions.html#transaction-retries) in the case of read/write contention. The [CockroachDB Go client](https://github.com/cockroachdb/cockroach-go) includes a generic **retry function** (`ExecuteTx()`) that runs inside a transaction and retries it as needed. The code sample shows how you can use this function to wrap SQL statements. -{{site.data.alerts.end}} - -## Step 3. Run the code - -1. Initialize the module: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ go mod init basic-sample && go mod tidy - ~~~ - -1. Run the code: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ go run main.go - ~~~ - - The program will prompt you for a connection string to the database: - - ~~~ - Enter a connection string: - ~~~ - -1. Enter the connection string to your running cluster. - -
- - {{site.data.alerts.callout_success}} - `postgresql://root@localhost:26257?sslmode=disable` should be the `sql` connection URL provided in the `cockroach` welcome text. - {{site.data.alerts.end}} - -
- -
- - {{site.data.alerts.callout_success}} - Use the connection string provided in the **Connection info** window of the CockroachDB {{ site.data.products.cloud }} Console. - {{site.data.alerts.end}} - - {{site.data.alerts.callout_info}} - You need to provide a SQL user password in order to securely connect to a CockroachDB {{ site.data.products.cloud }} cluster. The connection string should have a placeholder for the password (``). - {{site.data.alerts.end}} - -
- - The output should look similar to the following: - - ~~~ - 2021/09/16 14:17:12 Creating 5 new accounts... - 2021/09/16 14:17:12 Accounts created. - Balance at '2021-09-16 14:17:12.68843 -0400 EDT m=+2.760587790': - 1580d2f4-c9ec-4f26-bbe7-6a53e9aa5170 1947 - 26ddc77b-8068-409b-b305-0c5d873f7c43 7987 - 3d97ea5a-5108-4388-88e8-92524d5de5e8 4159 - af49831d-d637-4a20-a9a7-01e9fe4628fe 8181 - f0cc97ef-e3fe-4abb-a44a-0dd04207f7d4 2181 - 2021/09/16 14:17:12 Transferring 100 from account af49831d-d637-4a20-a9a7-01e9fe4628fe to account 3d97ea5a-5108-4388-88e8-92524d5de5e8... - 2021/09/16 14:17:12 Funds transferred. - Balance at '2021-09-16 14:17:12.759686 -0400 EDT m=+2.831841311': - 1580d2f4-c9ec-4f26-bbe7-6a53e9aa5170 1947 - 26ddc77b-8068-409b-b305-0c5d873f7c43 7987 - 3d97ea5a-5108-4388-88e8-92524d5de5e8 4259 - af49831d-d637-4a20-a9a7-01e9fe4628fe 8081 - f0cc97ef-e3fe-4abb-a44a-0dd04207f7d4 2181 - 2021/09/16 14:17:12 Deleting accounts created... - 2021/09/16 14:17:12 Accounts deleted. - ~~~ - - The code runs a migration that creates the `accounts` table in the `bank` database, based on the `Account` struct defined at the top of the `main.go` file. - - As shown in the output, the code also does the following: - - Inserts some rows into the `accounts` table. - - Reads values from the table. - - Updates values in the table. - - Deletes values from the table. - -## What's next? - -Read more about using the [GORM ORM](http://gorm.io), or check out a more realistic implementation of GORM with CockroachDB in our [`examples-orms`](https://github.com/cockroachdb/examples-orms) repository. - -{% include {{ page.version.version }}/app/see-also-links.md %} diff --git a/src/current/v21.1/build-a-go-app-with-cockroachdb-pq.md b/src/current/v21.1/build-a-go-app-with-cockroachdb-pq.md deleted file mode 100644 index e4cfb75a1c0..00000000000 --- a/src/current/v21.1/build-a-go-app-with-cockroachdb-pq.md +++ /dev/null @@ -1,128 +0,0 @@ ---- -title: Build a Go App with CockroachDB the Go pq Driver -summary: Learn how to use CockroachDB from a simple Go application with the Go pq driver. -toc: true -twitter: false -referral_id: docs_go_pq ---- - -
- - - - -
- -This tutorial shows you how build a simple Go application with CockroachDB and the Go [pq driver](https://github.com/lib/pq). - -## Step 1. Start CockroachDB - -{% include {{page.version.version}}/app/start-cockroachdb.md %} - -## Step 2. Create a database - -{% include {{page.version.version}}/app/create-a-database.md %} - -## Step 3. Run the Go code - -You can now run the code sample (`main.go`) provided in this tutorial to do the following: - -- Create a table in the `bank` database. -- Insert some rows into the table you created. -- Read values from the table. -- Execute a batch of statements as an atomic [transaction](transactions.html). - - Note that CockroachDB may require the [client to retry a transaction](transactions.html#transaction-retries) in the case of read/write contention. The [CockroachDB Go client](https://github.com/cockroachdb/cockroach-go) includes a generic **retry function** (`ExecuteTx()`) that runs inside a transaction and retries it as needed. The code sample shows how you can use this function to wrap SQL statements. - -### Get the code - -You can copy the code below, [download the code directly](https://raw.githubusercontent.com/cockroachlabs/hello-world-go-pq/master/main.go), or clone [the code's GitHub repository](https://github.com/cockroachlabs/hello-world-go-pq). - -Here are the contents of `main.go`: - -{% include_cached copy-clipboard.html %} -~~~ go -{% remote_include https://raw.githubusercontent.com/cockroachlabs/hello-world-go-pq/master/main.go %} -~~~ - -### Update the connection parameters - -
- -Edit the connection string passed to `sql.Open()` so that: - -- `{username}` and `{password}` specify the SQL username and password that you created earlier. -- `{hostname}` and `{port}` specify the hostname and port in the `(sql)` connection string from SQL shell welcome text. - -
- -
- -Replace the string passed to `sql.Open()` with the connection string that you copied [earlier](#set-up-your-cluster-connection) from the **Connection info** dialog. - -The function call should look similar to the following: - -{% include_cached copy-clipboard.html %} -~~~ go -db, err := sql.Open("postgres", "postgresql://{user}:{password}@{globalhost}:26257/bank?sslmode=verify-full&sslrootcert={path to the CA certificate}&options=--cluster={cluster_name}") -~~~ - -{% include {{page.version.version}}/app/cc-free-tier-params.md %} - -
- -### Run the code - -Initialize the module: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ go mod init basic-sample -~~~ - -Then run the code: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ go run main.go -~~~ - -The output should be: - -~~~ -Balances: -1 1000 -2 250 -Success -Balances: -1 900 -2 350 -~~~ - -To verify that the SQL statements were executed, run the following query from inside the SQL shell: - -{% include_cached copy-clipboard.html %} -~~~ sql -> USE bank; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT id, balance FROM accounts; -~~~ - -The output should be: - -~~~ - id | balance ------+---------- - 1 | 900 - 2 | 350 -(2 rows) -~~~ - -## What's next? - -Read more about using the [Go pq driver](https://godoc.org/github.com/lib/pq). - -{% include {{ page.version.version }}/app/see-also-links.md %} \ No newline at end of file diff --git a/src/current/v21.1/build-a-go-app-with-cockroachdb-upperdb.md b/src/current/v21.1/build-a-go-app-with-cockroachdb-upperdb.md deleted file mode 100644 index 18b3a74606a..00000000000 --- a/src/current/v21.1/build-a-go-app-with-cockroachdb-upperdb.md +++ /dev/null @@ -1,163 +0,0 @@ ---- -title: Build a Go App with CockroachDB and upper/db -summary: Learn how to use CockroachDB from a simple Go application with the upper/db data access layer. -toc: true -twitter: false ---- - -
- - - - -
- -This tutorial shows you how build a simple Go application with CockroachDB and the [upper/db](https://upper.io/) data access layer. - -## Before you begin - -{% include {{page.version.version}}/app/before-you-begin.md %} - -
- -## Step 1. Create the `maxroach` user and `bank` database - -{% include {{page.version.version}}/app/create-maxroach-user-and-bank-database.md %} - -## Step 2. Generate a certificate for the `maxroach` user - -Create a certificate and key for the `maxroach` user by running the following command: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach cert create-client maxroach --certs-dir=certs --ca-key=my-safe-directory/ca.key -~~~ - -The code samples will run with `maxroach` as the user. - -## Step 3. Run the Go code - -The sample code shown below uses upper/db to map Go-specific objects to SQL operations. Specifically, the code: - -- Creates the `accounts` table, if it does not already exist. -- Deletes any existing rows in the `accounts` table. -- Inserts two rows into the `accounts` table. -- Prints the rows in the `accounts` table to the terminal. -- Deletes the first row in the `accounts` table. -- Updates the rows in the `accounts` table within an explicit [transaction](transactions.html). -- Prints the rows in the `accounts` table to the terminal once more. - -{% include_cached copy-clipboard.html %} -~~~ go -{% include {{ page.version.version }}/app/upperdb-basic-sample/main.go %} -~~~ - -Note that the sample code also includes a function that simulates a transaction error (`crdbForceRetry()`). Upper/db's CockroachDB adapter [automatically retries transactions](transactions.html#client-side-intervention) when transaction errors are thrown. As a result, this function forces a transaction retry. - -To run the code, copy the sample above, or download it directly. - -{{site.data.alerts.callout_success}} -To clone a version of the code below that connects to insecure clusters, run the following command: - -`git clone https://github.com/cockroachlabs/hello-world-go-upperdb/` - -Note that you will need to edit the connection string to use the certificates that you generated when you set up your secure cluster. -{{site.data.alerts.end}} - -
- -
- -## Step 1. Create the `maxroach` user and `bank` database - -{% include {{page.version.version}}/app/insecure/create-maxroach-user-and-bank-database.md %} - -## Step 2. Run the Go code - -The sample code shown below uses upper/db to map Go-specific objects to SQL operations. Specifically, the code: - -- Creates the `accounts` table, if it does not already exist. -- Deletes any existing rows in the `accounts` table. -- Inserts two rows into the `accounts` table. -- Prints the rows in the `accounts` table to the terminal. -- Deletes the first row in the `accounts` table. -- Updates the rows in the `accounts` table within an explicit [transaction](transactions.html). -- Prints the rows in the `accounts` table to the terminal once more. - -{% include_cached copy-clipboard.html %} -~~~ go -{% include {{ page.version.version }}/app/insecure/upperdb-basic-sample/main.go %} -~~~ - -Note that the sample code also includes a function that simulates a transaction error (`crdbForceRetry()`). Upper/db's CockroachDB adapter [automatically retries transactions](transactions.html#client-side-intervention) when transaction errors are thrown. As a result, this function forces a transaction retry. - -Copy the code or download it directly. - -{{site.data.alerts.callout_success}} -To clone a version of the code below that connects to insecure clusters, run the following command: - -`git clone https://github.com/cockroachlabs/hello-world-go-upperdb/` -{{site.data.alerts.end}} - -
- -Change to the directory where you cloned the repo and get the dependencies with `go mod init`: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ go mod init hello-world-go-upperdb -~~~ - -Then run the code: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ go run main.go -~~~ - -The output should look similar to the following: - -~~~ -go: finding module for package github.com/upper/db/v4 -go: finding module for package github.com/upper/db/v4/adapter/cockroachdb -go: found github.com/upper/db/v4 in github.com/upper/db/v4 v4.0.0 -2020/09/16 10:31:55 Balances: - accounts[590467288222990337]: 1000 - accounts[590467288229576705]: 250 -2020/09/16 10:31:55 Balances: - accounts[590467288222990337]: 500 - accounts[590467288229576705]: 999 -2020/09/16 10:31:55 upper/db: log_level=WARNING file=go/pkg/mod/github.com/upper/db/v4@v4.0.0/internal/sqladapter/session.go:642 - Session ID: 00006 - Transaction ID: 00005 - Query: SELECT crdb_internal.force_retry('1ms'::INTERVAL) - Error: pq: restart transaction: crdb_internal.force_retry(): TransactionRetryWithProtoRefreshError: forced by crdb_internal.force_retry() - Time taken: 0.00171s - Context: context.Background - -2020/09/16 10:31:55 upper/db: log_level=WARNING file=go/pkg/mod/github.com/upper/db/v4@v4.0.0/internal/sqladapter/session.go:642 - Session ID: 00006 - Transaction ID: 00005 - Query: INSERT INTO "accounts" ("balance") VALUES ($1) RETURNING "id" - Arguments: []interface {}{887} - Error: pq: current transaction is aborted, commands ignored until end of transaction block - Time taken: 0.00065s - Context: context.Background - -2020/09/16 10:31:56 Balances: - accounts[590467288229576705]: 999 - accounts[590467288342757377]: 887 - accounts[590467288350064641]: 342 -~~~ - -Note that the forced transaction errors result in errors printed to the terminal, but the transactions are retried until they succeed. - -## What's next? - -Read more about upper/db: - -- [Introduction to upper/db](https://upper.io/v4/getting-started/) -- [The upper/db tour](https://tour.upper.io/) -- [upper/db reference docs](https://pkg.go.dev/github.com/upper/db/v4) - -{% include {{ page.version.version }}/app/see-also-links.md %} diff --git a/src/current/v21.1/build-a-go-app-with-cockroachdb.md b/src/current/v21.1/build-a-go-app-with-cockroachdb.md deleted file mode 100644 index 6fb4e964f42..00000000000 --- a/src/current/v21.1/build-a-go-app-with-cockroachdb.md +++ /dev/null @@ -1,161 +0,0 @@ ---- -title: Build a Simple CRUD Go App with CockroachDB and the Go pgx Driver -summary: Learn how to use CockroachDB from a simple Go application with the Go pgx driver. -toc: true -twitter: false -referral_id: docs_hello_world_go_pgx ---- - -
- - - - -
- -This tutorial shows you how build a simple CRUD Go application with CockroachDB and the [Go pgx driver](https://pkg.go.dev/github.com/jackc/pgx). - -## Step 1. Start CockroachDB - -{% include {{ page.version.version }}/app/sample-setup.md %} - -## Step 2. Get the code - -Clone the code's GitHub repo: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ git clone https://github.com/cockroachlabs/example-app-go-pgx/ -~~~ - -The project has the following directory structure: - -~~~ -├── README.md -├── dbinit.sql -└── main.go -~~~ - -The `dbinit.sql` file initializes the database schema that the application uses: - -{% include_cached copy-clipboard.html %} -~~~ go -{% remote_include https://raw.githubusercontent.com/cockroachlabs/example-app-go-pgx/master/dbinit.sql %} -~~~ - -The `main.go` file contains the code for `INSERT`, `SELECT`, `UPDATE`, and `DELETE` SQL operations. The file also executes the `main` method of the program. - -{% include_cached copy-clipboard.html %} -~~~ go -{% remote_include https://raw.githubusercontent.com/cockroachlabs/example-app-go-pgx/master/main.go %} -~~~ - -{{site.data.alerts.callout_info}} -CockroachDB may require the [client to retry a transaction](transactions.html#transaction-retries) in the case of read/write contention. The [CockroachDB Go client](https://github.com/cockroachdb/cockroach-go) includes a generic **retry function** (`ExecuteTx()`) that runs inside a transaction and retries it as needed. The code sample shows how you can use this function to wrap SQL statements. -{{site.data.alerts.end}} - -## Step 3. Initialize the database - -{% include {{ page.version.version }}/app/init-bank-sample.md %} - -## Step 4. Run the code - -1. Initialize the module: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ go mod init basic-sample && go mod tidy - ~~~ - -1. Run the code: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ go run main.go - ~~~ - - The program will prompt you for a connection string to the database: - - ~~~ - Enter a connection string: - ~~~ - -1. Enter the connection string to your running cluster. - -
- - {{site.data.alerts.callout_success}} - `postgresql://root@localhost:26257?sslmode=disable` should be the `sql` connection URL provided in the `cockroach` welcome text. - {{site.data.alerts.end}} - -
- -
- - {{site.data.alerts.callout_success}} - Use the connection string provided in the **Connection info** window of the CockroachDB {{ site.data.products.cloud }} Console. - {{site.data.alerts.end}} - - {{site.data.alerts.callout_info}} - You need to provide a SQL user password in order to securely connect to a CockroachDB {{ site.data.products.cloud }} cluster. The connection string should have a placeholder for the password (``). - {{site.data.alerts.end}} - -
- - The output should look similar to the following: - - ~~~ - 2021/07/20 14:48:02 Creating new rows... - 2021/07/20 14:48:02 New rows created. - 2021/07/20 14:48:02 Initial balances: - 2021/07/20 14:48:02 3a936990-a0c9-45bf-bc24-92e10d91dca9: 300 - 2021/07/20 14:48:02 8d1849dd-9222-4b12-a4ff-94e583b544a8: 250 - 2021/07/20 14:48:02 c6ae8917-d24e-4115-b719-f663dbfb9ffb: 500 - 2021/07/20 14:48:02 d0ce1f5c-e468-4899-8590-2bb6076247f2: 100 - 2021/07/20 14:48:02 Transferring funds from account with ID c6ae8917-d24e-4115-b719-f663dbfb9ffb to account with ID d0ce1f5c-e468-4899-8590-2bb6076247f2... - 2021/07/20 14:48:02 Transfer successful. - 2021/07/20 14:48:02 Balances after transfer: - 2021/07/20 14:48:02 3a936990-a0c9-45bf-bc24-92e10d91dca9: 300 - 2021/07/20 14:48:02 8d1849dd-9222-4b12-a4ff-94e583b544a8: 250 - 2021/07/20 14:48:02 c6ae8917-d24e-4115-b719-f663dbfb9ffb: 400 - 2021/07/20 14:48:02 d0ce1f5c-e468-4899-8590-2bb6076247f2: 200 - 2021/07/20 14:48:02 Deleting rows with IDs 8d1849dd-9222-4b12-a4ff-94e583b544a8 and d0ce1f5c-e468-4899-8590-2bb6076247f2... - 2021/07/20 14:48:02 Rows deleted. - 2021/07/20 14:48:02 Balances after deletion: - 2021/07/20 14:48:02 3a936990-a0c9-45bf-bc24-92e10d91dca9: 300 - 2021/07/20 14:48:02 c6ae8917-d24e-4115-b719-f663dbfb9ffb: 400 - ~~~ - - As shown in the output, the code does the following: - - Inserts some rows into the `accounts` table. - - Reads values from the table. - - Updates values in the table. - - Deletes values from the table. - -1. To verify that the SQL statements were executed, run the following query from inside the SQL shell: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > USE bank; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SELECT id, balance FROM accounts; - ~~~ - - The output should look similar to the following: - - ~~~ - id | balance - ---------------------------------------+---------- - 3a936990-a0c9-45bf-bc24-92e10d91dca9 | 300 - c6ae8917-d24e-4115-b719-f663dbfb9ffb | 400 - (2 rows) - ~~~ - -## What's next? - -Read more about using the [Go pgx driver](https://pkg.go.dev/github.com/jackc/pgx?tab=doc). - -{% include {{ page.version.version }}/app/see-also-links.md %} diff --git a/src/current/v21.1/build-a-java-app-with-cockroachdb-hibernate.md b/src/current/v21.1/build-a-java-app-with-cockroachdb-hibernate.md deleted file mode 100644 index 678e8e6be25..00000000000 --- a/src/current/v21.1/build-a-java-app-with-cockroachdb-hibernate.md +++ /dev/null @@ -1,204 +0,0 @@ ---- -title: Build a Java App with CockroachDB and Hibernate -summary: Learn how to use CockroachDB from a simple Java application with the Hibernate ORM. -toc: true -twitter: false -referral_id: docs_hello_world_java_hibernate ---- - -
- - - - -
- -This tutorial shows you how build a simple Java application with CockroachDB and the Hibernate ORM. - -{% include {{page.version.version}}/app/java-version-note.md %} - -{{site.data.alerts.callout_success}} -For a sample app and tutorial that uses Spring Data JPA (Hibernate) and CockroachDB, see [Build a Spring App with CockroachDB and JPA](build-a-spring-app-with-cockroachdb-jpa.html). - -For another use of Hibernate with CockroachDB, see our [`examples-orms`](https://github.com/cockroachdb/examples-orms) repository. -{{site.data.alerts.end}} - -## Step 1. Start CockroachDB - -{% include {{page.version.version}}/app/start-cockroachdb.md %} - -## Step 2. Create a database - -{% include {{page.version.version}}/app/create-a-database.md %} - -## Step 3. Run the Java code - -The sample code in this tutorial ([`Sample.java`](#code-contents))uses Hibernate to map Java methods to SQL operations. The code performs the following operations, which roughly correspond to method calls in the `Sample` class: - -1. Creates an `accounts` table in the `bank` database as specified by the `Account` mapping class. -1. Inserts rows into the table with the `addAccounts()` method. -1. Transfers money from one account to another with the `transferFunds()` method. -1. Prints out account balances before and after the transfer with the `getAccountBalance()` method. - -In addition, the code shows a pattern for automatically handling [transaction retries](transactions.html#client-side-intervention-example) by wrapping transactions in a higher-order function named `runTransaction()`. It also includes a method for testing the retry handling logic (`Sample.forceRetryLogic()`), which will be run if you set the `FORCE_RETRY` variable to `true`. - -It does all of the above using the practices we recommend for using Hibernate (and the underlying JDBC connection) with CockroachDB, which are listed in the [Recommended Practices](#recommended-practices) section below. - - -The contents of `Sample.java`: - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/cockroachlabs/example-app-java-hibernate/master/src/main/java/com/cockroachlabs/Sample.java %} -~~~ - -### Get the code - -Clone the `example-app-java-hibernate` repo to your machine: - -{% include_cached copy-clipboard.html %} -~~~ shell -git clone https://github.com/cockroachlabs/example-app-java-hibernate/ -~~~ - -{{site.data.alerts.callout_info}} -The version of the CockroachDB Hibernate dialect in `hibernate.cfg.xml` corresponds to a version of CockroachDB. For more information, see [Install Client Drivers: Hibernate](install-client-drivers.html). -{{site.data.alerts.end}} - -### Update the connection parameters - -Edit `src/main/resources/hibernate.cfg.xml` in a text editor. - -
- -1. Modify the `hibernate.connection.url` property with the port number from the connection string above: - - {% include_cached copy-clipboard.html %} - ~~~ xml - jdbc:postgresql://localhost:{port}/bank?ssl=true&sslmode=require - ~~~ - - Where `{port}` is the port number on which the CockroachDB demo cluster is listening. - -1. Set the `hibernate.connection.username` property to the username you created earlier. - -1. Set the `hibernate.connection.password` property to the user's password. - -
- -
- -1. Modify the `hibernate.connection.url` property with the information from the connection string you copied [earlier](#set-up-your-cluster-connection) host, cluster and database name, and path to the SSL certificate: - - {% include_cached copy-clipboard.html %} - ~~~ xml - jdbc:postgresql://{globalhost}:26257/{cluster_name}.bank?sslmode=verify-full&sslrootcert={path to the CA certificate} - ~~~ - {% include {{page.version.version}}/app/cc-free-tier-params.md %} - -
- -### Run the code - -Compile and run the code using `gradlew`, which will also download the dependencies. - -{% include_cached copy-clipboard.html %} -~~~ shell -$ ./gradlew run -~~~ - -Toward the end of the output, you should see: - -~~~ -APP: BEGIN; -APP: addAccounts() --> 1.00 -APP: COMMIT; -APP: BEGIN; -APP: getAccountBalance(1) --> 1000.00 -APP: COMMIT; -APP: BEGIN; -APP: getAccountBalance(2) --> 250.00 -APP: COMMIT; -APP: getAccountBalance(1) --> 1000.00 -APP: getAccountBalance(2) --> 250.00 -APP: BEGIN; -APP: transferFunds(1, 2, 100.00) --> 100.00 -APP: COMMIT; -APP: transferFunds(1, 2, 100.00) --> 100.00 -APP: BEGIN; -APP: getAccountBalance(1) --> 900.00 -APP: COMMIT; -APP: BEGIN; -APP: getAccountBalance(2) --> 350.00 -APP: COMMIT; -APP: getAccountBalance(1) --> 900.00 -APP: getAccountBalance(2) --> 350.00 -~~~ - -To verify that the account balances were updated successfully, start the [built-in SQL client](cockroach-sql.html): - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --certs-dir={certs_dir} -~~~ - -Where: -- `{certs_dir}` is the path to the directory containing the certificates and keys you generated earlier. - -To check the account balances, issue the following statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT id, balance FROM accounts; -~~~ - -~~~ -id | balance ------+------------ - 1 | 900.00 - 2 | 350.00 - 3 | 314159.00 -(3 rows) -~~~ - - - -## Recommended Practices - -### Generate PKCS8 keys for client authentication - -{% include {{page.version.version}}/app/pkcs8-gen.md %} - -
- -{% include cockroachcloud/cc-no-user-certs.md %} - -
- -### Use `IMPORT` to read in large data sets - -If you are trying to get a large data set into CockroachDB all at once (a bulk import), avoid writing client-side code altogether and use the [`IMPORT`](import.html) statement instead. It is much faster and more efficient than making a series of [`INSERT`s](insert.html) and [`UPDATE`s](update.html). It bypasses the [SQL layer](architecture/sql-layer.html) altogether and writes directly to the [storage layer](architecture/storage-layer.html) of the database. - -For more information about importing data from Postgres, see [Migrate from Postgres](migrate-from-postgres.html). - -For more information about importing data from MySQL, see [Migrate from MySQL](migrate-from-mysql.html). - -### Use `reWriteBatchedInserts` for increased speed - -We strongly recommend setting `reWriteBatchedInserts=true`; we have seen 2-3x performance improvements with it enabled. From [the JDBC connection parameters documentation](https://jdbc.postgresql.org/documentation/use/#connection-parameters): - -> This will change batch inserts from `insert into foo (col1, col2, col3) values (1,2,3)` into `insert into foo (col1, col2, col3) values (1,2,3), (4,5,6)` this provides 2-3x performance improvement - -### Retrieve large data sets in chunks using cursors - -CockroachDB now supports the Postgres wire-protocol cursors for implicit transactions and explicit transactions executed to completion. This means the [PGJDBC driver](https://jdbc.postgresql.org) can use this protocol to stream queries with large result sets. This is much faster than [paginating through results in SQL using `LIMIT .. OFFSET`](pagination.html). - -For instructions showing how to use cursors in your Java code, see [Getting results based on a cursor](https://jdbc.postgresql.org/documentation/query/#getting-results-based-on-a-cursor) from the PGJDBC documentation. - -Note that interleaved execution (partial execution of multiple statements within the same connection and transaction) is not supported when [`Statement.setFetchSize()`](https://docs.oracle.com/javase/8/docs/api/java/sql/Statement.html#setFetchSize-int-) is used. - -## What's next? - -Read more about using the [Hibernate ORM](http://hibernate.org/orm/), or check out a more realistic implementation of Hibernate with CockroachDB in our [`examples-orms`](https://github.com/cockroachdb/examples-orms) repository. - -{% include {{page.version.version}}/app/see-also-links.md %} diff --git a/src/current/v21.1/build-a-java-app-with-cockroachdb-jooq.md b/src/current/v21.1/build-a-java-app-with-cockroachdb-jooq.md deleted file mode 100644 index a9cd07982dd..00000000000 --- a/src/current/v21.1/build-a-java-app-with-cockroachdb-jooq.md +++ /dev/null @@ -1,276 +0,0 @@ ---- -title: Build a Java App with CockroachDB and jOOQ -summary: Learn how to use CockroachDB from a simple Java application with jOOQ. -toc: true -twitter: false ---- - -
- - - - -
- -This tutorial shows you how build a simple Java application with CockroachDB and [jOOQ](https://www.jooq.org/). - -CockroachDB is supported in jOOQ [Professional and Enterprise editions](https://www.jooq.org/download/#databases). - -{% include {{page.version.version}}/app/java-version-note.md %} - -{{site.data.alerts.callout_success}} -For another use of jOOQ with CockroachDB, see our [`examples-orms`](https://github.com/cockroachdb/examples-orms) repository. -{{site.data.alerts.end}} - -## Before you begin - -{% include {{page.version.version}}/app/before-you-begin.md %} - -## Step 1. Install Maven - -This tutorial uses the [Maven build tool](https://gradle.org/) to manage application dependencies. - -To install Maven on Mac, run the following command: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ brew install maven -~~~ - -To install Maven on a Debian-based Linux distribution like Ubuntu: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ apt-get install maven -~~~ - -For other ways to install Maven, see [its official documentation](https://maven.apache.org/install.html). - -## Step 2. Install jOOQ - -Download the free trial of jOOQ Professional or Enterprise edition from [jOOQ's website](https://www.jooq.org/download), and unzip the file. - -To install jOOQ to your machine's local Maven repository, run the `maven-install.sh` script included in the jOOQ install folder: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ chmod +x maven-install.sh -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ ./maven-install.sh -~~~ - -
- -## Step 3. Create the `maxroach` user and `bank` database - -{% include {{page.version.version}}/app/create-maxroach-user-and-bank-database.md %} - -## Step 4. Generate a certificate for the `maxroach` user - -Create a certificate and key for the `maxroach` user by running the following command. The code samples will run as this user. - -The [`--also-generate-pkcs8-key` flag](cockroach-cert.html#flag-pkcs8) generates a key in [PKCS#8 format](https://tools.ietf.org/html/rfc5208), which is the standard key encoding format in Java. In this case, the generated PKCS8 key will be named `client.maxroach.key.pk8`. - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach cert create-client maxroach --certs-dir=certs --ca-key=my-safe-directory/ca.key --also-generate-pkcs8-key -~~~ - -## Step 5. Run the Java code - -The code below uses jOOQ to map Java methods to SQL operations. It performs the following steps, some of which correspond to method calls of the `Sample` class. - -1. Inputs the `db.sql` file to the database. `db.sql` includes SQL statements that create an `accounts` table in the `bank` database. -2. Inserts rows into the `accounts` table using `session.save(new Account(int id, int balance))` (see `Sample.addAccounts()`). -3. Transfers money from one account to another, printing out account balances before and after the transfer (see `transferFunds(long fromId, long toId, long amount)`). -4. Prints out account balances before and after the transfer (see `Sample.getAccountBalance(long id)`). - -In addition, the code shows a pattern for automatically handling [transaction retries](transactions.html#client-side-intervention-example) by wrapping transactions in a higher-order function `Sample.runTransaction()`. It also includes a method for testing the retry handling logic (`Sample.forceRetryLogic()`), which will be run if you set the `FORCE_RETRY` variable to `true`. - -To run it: - -1. Download and unzip [jooq-basic-sample.zip](https://github.com/cockroachdb/docs/raw/master/_includes/{{ page.version.version }}/app/jooq-basic-sample/jooq-basic-sample.zip). -2. Open `jooq-basic-sample/src/main/java/com/cockroachlabs/Sample.java`, and edit the connection string passed to `DriverManager.getConnection()` in the `Sample` class's `main()` method so that the certificate paths are fully and correctly specified. -3. Compile and run the code using Maven: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cd jooq-basic-sample - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ mvn compile - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ mvn exec:java -Dexec.mainClass=com.cockroachlabs.Sample - ~~~ - -Here are the contents of [`Sample.java`](https://raw.githubusercontent.com/cockroachdb/docs/master/_includes/{{page.version.version}}/app/jooq-basic-sample/Sample.java), the Java file containing the main `Sample` class: - -{% include_cached copy-clipboard.html %} -~~~ java -{% include {{page.version.version}}/app/jooq-basic-sample/Sample.java %} -~~~ - -Toward the end of the output, you should see: - -~~~ -APP: BEGIN; -APP: addAccounts() --> 1 -APP: COMMIT; -APP: BEGIN; -APP: getAccountBalance(1) --> 1000 -APP: COMMIT; -APP: BEGIN; -APP: getAccountBalance(2) --> 250 -APP: COMMIT; -APP: getAccountBalance(1) --> 1000 -APP: getAccountBalance(2) --> 250 -APP: BEGIN; -APP: transferFunds(1, 2, 100) --> 100 -APP: COMMIT; -APP: transferFunds(1, 2, 100) --> 100 -APP: BEGIN; -APP: getAccountBalance(1) --> 900 -APP: COMMIT; -APP: BEGIN; -APP: getAccountBalance(2) --> 350 -APP: COMMIT; -APP: getAccountBalance(1) --> 900 -APP: getAccountBalance(2) --> 350 -~~~ - -To verify that the account balances were updated successfully, start the [built-in SQL client](cockroach-sql.html): - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --certs-dir=certs -~~~ - -To check the account balances, issue the following statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT id, balance FROM accounts; -~~~ - -~~~ - id | balance -+----+---------+ - 1 | 900 - 2 | 350 - 3 | 314159 -(3 rows) -~~~ - -
- -
- -## Step 3. Create the `maxroach` user and `bank` database - -{% include {{page.version.version}}/app/insecure/create-maxroach-user-and-bank-database.md %} - -## Step 4. Run the Java code - -The code below uses jOOQ to map Java methods to SQL operations. It performs the following steps, some of which correspond to method calls of the `Sample` class. - -1. Inputs the `db.sql` file to the database. `db.sql` includes SQL statements that create an `accounts` table in the `bank` database. -2. Inserts rows into the `accounts` table using `session.save(new Account(int id, int balance))` (see `Sample.addAccounts()`). -3. Transfers money from one account to another, printing out account balances before and after the transfer (see `transferFunds(long fromId, long toId, long amount)`). -4. Prints out account balances before and after the transfer (see `Sample.getAccountBalance(long id)`). - -In addition, the code shows a pattern for automatically handling [transaction retries](transactions.html#client-side-intervention-example) by wrapping transactions in a higher-order function `Sample.runTransaction()`. It also includes a method for testing the retry handling logic (`Sample.forceRetryLogic()`), which will be run if you set the `FORCE_RETRY` variable to `true`. - -To run it: - -1. Download and unzip [jooq-basic-sample.zip](https://github.com/cockroachdb/docs/raw/master/_includes/{{ page.version.version }}/app/insecure/jooq-basic-sample/jooq-basic-sample.zip). -2. Compile and run the code using Maven: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cd jooq-basic-sample - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ mvn compile - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ mvn exec:java -Dexec.mainClass=com.cockroachlabs.Sample - ~~~ - -Here are the contents of [`Sample.java`](https://raw.githubusercontent.com/cockroachdb/docs/master/_includes/{{page.version.version}}/app/insecure/jooq-basic-sample/Sample.java), the Java file containing the main `Sample` class: - -{% include_cached copy-clipboard.html %} -~~~ java -{% include {{page.version.version}}/app/insecure/jooq-basic-sample/Sample.java %} -~~~ - -Toward the end of the output, you should see: - -~~~ -APP: BEGIN; -APP: addAccounts() --> 1 -APP: COMMIT; -APP: BEGIN; -APP: getAccountBalance(1) --> 1000 -APP: COMMIT; -APP: BEGIN; -APP: getAccountBalance(2) --> 250 -APP: COMMIT; -APP: getAccountBalance(1) --> 1000 -APP: getAccountBalance(2) --> 250 -APP: BEGIN; -APP: transferFunds(1, 2, 100) --> 100 -APP: COMMIT; -APP: transferFunds(1, 2, 100) --> 100 -APP: BEGIN; -APP: getAccountBalance(1) --> 900 -APP: COMMIT; -APP: BEGIN; -APP: getAccountBalance(2) --> 350 -APP: COMMIT; -APP: getAccountBalance(1) --> 900 -APP: getAccountBalance(2) --> 350 -~~~ - -To verify that the account balances were updated successfully, start the [built-in SQL client](cockroach-sql.html): - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --insecure -~~~ - -To check the account balances, issue the following statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT id, balance FROM accounts; -~~~ - -~~~ - id | balance -+----+---------+ - 1 | 900 - 2 | 350 - 3 | 314159 -(3 rows) -~~~ - -
- - -## What's next? - -Read more about using [jOOQ](https://www.jooq.org/), or check out a more realistic implementation of jOOQ with CockroachDB in our [`examples-orms`](https://github.com/cockroachdb/examples-orms) repository. - -{% include {{page.version.version}}/app/see-also-links.md %} diff --git a/src/current/v21.1/build-a-java-app-with-cockroachdb.md b/src/current/v21.1/build-a-java-app-with-cockroachdb.md deleted file mode 100644 index d7799f52c9b..00000000000 --- a/src/current/v21.1/build-a-java-app-with-cockroachdb.md +++ /dev/null @@ -1,333 +0,0 @@ ---- -title: Build a Simple CRUD Java App with CockroachDB and JDBC -summary: Learn how to use CockroachDB from a simple Java application with the JDBC driver. -toc: true -twitter: false -referral_id: docs_hello_world_java_jdbc ---- - -
- - - - -
- -This tutorial shows you how to build a simple CRUD Java application with CockroachDB and the Java JDBC driver. - -{% include {{page.version.version}}/app/java-version-note.md %} - -{{site.data.alerts.callout_success}} -For a sample app and tutorial that uses Spring Data JDBC and CockroachDB, see [Build a Spring App with CockroachDB and JDBC](build-a-spring-app-with-cockroachdb-jdbc.html). -{{site.data.alerts.end}} - -## Step 1. Start CockroachDB - -{% include {{ page.version.version }}/app/sample-setup.md %} - -## Step 2. Get the code - -Clone the code's GitHub repo: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ git clone https://github.com/cockroachlabs/example-app-java-jdbc/ -~~~ - -
- -Check out the `cockroachcloud` branch: - -{% include_cached copy-clipboard.html %} -~~~shell -git checkout cockroachcloud -~~~ - -
- -The project has the following directory structure: - -~~~ -├── README.md -├── app -│   ├── build.gradle -│   └── src -│   └── main -│   ├── java -│   │   └── com -│   │   └── cockroachlabs -│   │   └── BasicExample.java -│   └── resources -│   └── dbinit.sql -├── gradle -│   └── wrapper -│   ├── gradle-wrapper.jar -│   └── gradle-wrapper.properties -├── gradlew -├── gradlew.bat -└── settings.gradle -~~~ - -The `dbinit.sql` file initializes the database schema that the application uses: - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/cockroachlabs/example-app-java-jdbc/master/app/src/main/resources/dbinit.sql %} -~~~ - -The `BasicExample.java` file contains the code for `INSERT`, `SELECT`, and `UPDATE` SQL operations. The file also contains the `main` method of the program. - -
- -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/cockroachlabs/example-app-java-jdbc/cockroachcloud/app/src/main/java/com/cockroachlabs/BasicExample.java %} -~~~ - -
- -
- -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/cockroachlabs/example-app-java-jdbc/master/app/src/main/java/com/cockroachlabs/BasicExample.java %} -~~~ - -
- -The sample app uses JDBC and the [Data Access Object (DAO)](https://en.wikipedia.org/wiki/Data_access_object) pattern to map Java methods to SQL operations. It consists of two classes: - -1. `BasicExample`, which is where the application logic lives. -2. `BasicExampleDAO`, which is used by the application to access the data store (in this case CockroachDB). This class also includes a helper function (`runSql`) that runs SQL statements inside a transaction, [retrying statements](transactions.html#transaction-retries) as needed. - -The `main` method of the app performs the following steps which roughly correspond to method calls in the `BasicExample` class. - -| Step | Method | -|------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------| -| 1. Insert account data using a `Map` that corresponds to the input to `INSERT` on the backend | `BasicExampleDAO.updateAccounts(Map balance)` | -| 2. Transfer money from one account to another, printing out account balances before and after the transfer | `BasicExampleDAO.transferFunds(UUID from, UUID to, BigDecimal amount)` | -| 3. Insert random account data using JDBC's bulk insertion support | `BasicExampleDAO.bulkInsertRandomAccountData()` | -| 4. Print out some account data | `BasicExampleDAO.readAccounts(int limit)` | - -It does all of the above using the practices we recommend for using JDBC with CockroachDB, which are listed in the [Recommended Practices](#recommended-practices) section below. - -## Step 3. Initialize the database - -To initialize the example database, use the [`cockroach sql`](cockroach-sql.html) command to execute the SQL statements in the `dbinit.sql` file: - -
- -{% include_cached copy-clipboard.html %} -~~~ shell -cat app/src/main/resources/dbinit.sql | cockroach sql --url "" -~~~ - -Where `` is the connection string you obtained earlier from the CockroachDB {{ site.data.products.cloud }} Console. - -
- -
- -{% include_cached copy-clipboard.html %} -~~~ shell -cat app/src/main/resources/dbinit.sql | cockroach sql --url "postgresql://root@localhost:26257?sslmode=disable" -~~~ - -{{site.data.alerts.callout_info}} -`postgresql://root@localhost:26257?sslmode=disable` is the `sql` connection string you obtained earlier from the `cockroach` welcome text. -{{site.data.alerts.end}} - -
- -The SQL statements in the initialization file should execute: - -~~~ -SET - -Time: 1ms - -SET - -Time: 2ms - -DROP DATABASE - -Time: 1ms - -CREATE DATABASE - -Time: 2ms - -SET - -Time: 10ms - -CREATE TABLE - -Time: 4ms -~~~ - -## Step 4. Run the code - -
- -### Update the connection parameters - -In a text editor modify `app/src/main/java/com/cockroachlabs/BasicExample.java` with the settings to connect to the cluster: - -{% include_cached copy-clipboard.html %} -~~~ java -ds.setServerNames(new String[]{"{globalhost}"}); -ds.setDatabaseName("{cluster_name}.bank"); -ds.setUser("{username}"); -ds.setPassword("{password}"); -ds.setSslRootCert(System.getenv("{path to the CA certificate}")); -~~~ - -{% include {{page.version.version}}/app/cc-free-tier-params.md %} - -{{site.data.alerts.callout_success}} -For guidance on connection pooling, with an example using JDBC and [HikariCP](https://github.com/brettwooldridge/HikariCP), see [Connection Pooling](connection-pooling.html). -{{site.data.alerts.end}} - -
- -Compile and run the code: - -{% include_cached copy-clipboard.html %} -~~~ shell -./gradlew run -~~~ - -The output will look like the following: - -~~~ -com.cockroachlabs.BasicExampleDAO.updateAccounts: - 'INSERT INTO accounts (id, balance) VALUES ('b5679853-b968-4206-91ec-68945fa3e716', 250)' - -com.cockroachlabs.BasicExampleDAO.updateAccounts: - 'INSERT INTO accounts (id, balance) VALUES ('d1c41041-6589-4b06-8d7c-b9d6d901727e', 1000)' -BasicExampleDAO.updateAccounts: - => 2 total updated accounts -main: - => Account balances at time '15:09:08.902': - ID 1 => $1000 - ID 2 => $250 - -com.cockroachlabs.BasicExampleDAO.transferFunds: - 'UPSERT INTO accounts (id, balance) VALUES('d99e6bb5-ecd1-48e5-b6b6-47fc9a4bc752', ((SELECT balance FROM accounts WHERE id = 'd99e6bb5-ecd1-48e5-b6b6-47fc9a4bc752') - 100)),('6f0c1f94-509a-47e3-a9ab-6a9e3965945c', ((SELECT balance FROM accounts WHERE id = '6f0c1f94-509a-47e3-a9ab-6a9e3965945c') + 100))' -BasicExampleDAO.transferFunds: - => $100 transferred between accounts d99e6bb5-ecd1-48e5-b6b6-47fc9a4bc752 and 6f0c1f94-509a-47e3-a9ab-6a9e3965945c, 2 rows updated -main: - => Account balances at time '15:09:09.142': - ID 1 => $1000 - ID 2 => $250 - -BasicExampleDAO.bulkInsertRandomAccountData: - 'INSERT INTO accounts (id, balance) VALUES ('b70a0c48-fdf4-42ea-b07a-2fea83d77c7d', '287108674'::numeric)' - => 128 row(s) updated in this batch - -BasicExampleDAO.bulkInsertRandomAccountData: - 'INSERT INTO accounts (id, balance) VALUES ('75a5f894-532a-464d-b37e-a4b9ec1c1db6', '189904311'::numeric)' - => 128 row(s) updated in this batch - -BasicExampleDAO.bulkInsertRandomAccountData: - 'INSERT INTO accounts (id, balance) VALUES ('0803968f-ba07-4ece-82d5-24d4da9fdee9', '832474731'::numeric)' - => 128 row(s) updated in this batch - -BasicExampleDAO.bulkInsertRandomAccountData: - 'INSERT INTO accounts (id, balance) VALUES ('082e634d-4930-41eb-9839-298632a5530a', '665918272'::numeric)' - => 128 row(s) updated in this batch - -BasicExampleDAO.bulkInsertRandomAccountData: - => finished, 512 total rows inserted - -com.cockroachlabs.BasicExampleDAO.readAccounts: - 'SELECT id, balance FROM accounts LIMIT 10' - balance => 424934060 - balance => 62220740 - balance => 454671673 - balance => 556061618 - balance => 450164589 - balance => 996867752 - balance => 55869978 - balance => 747446662 - balance => 175832969 - balance => 181799597 - -BUILD SUCCESSFUL in 8s -3 actionable tasks: 3 executed -~~~ - -## Recommended Practices - -### Generate PKCS8 keys for user authentication - -{% include {{page.version.version}}/app/pkcs8-gen.md %} - -
- -{% include cockroachcloud/cc-no-user-certs.md %} - -
- -### Use `IMPORT` to read in large data sets - -If you are trying to get a large data set into CockroachDB all at once (a bulk import), avoid writing client-side code altogether and use the [`IMPORT`](import.html) statement instead. It is much faster and more efficient than making a series of [`INSERT`s](insert.html) and [`UPDATE`s](update.html). It bypasses the [SQL layer](architecture/sql-layer.html) altogether and writes directly to the [storage layer](architecture/storage-layer.html) of the database. - -For more information about importing data from Postgres, see [Migrate from Postgres](migrate-from-postgres.html). - -For more information about importing data from MySQL, see [Migrate from MySQL](migrate-from-mysql.html). - -### Use `reWriteBatchedInserts` for increased speed - -We strongly recommend setting `reWriteBatchedInserts=true`; we have seen 2-3x performance improvements with it enabled. From [the JDBC connection parameters documentation](https://jdbc.postgresql.org/documentation/use/#connection-parameters): - -> This will change batch inserts from `insert into foo (col1, col2, col3) values (1,2,3)` into `insert into foo (col1, col2, col3) values (1,2,3), (4,5,6)` this provides 2-3x performance improvement - -### Use a batch size of 128 - -PGJDBC's batching support only works with [powers of two](https://github.com/pgjdbc/pgjdbc/blob/7b52b0c9e5b9aa9a9c655bb68f23bf4ec57fd51c/pgjdbc/src/main/java/org/postgresql/jdbc/PgPreparedStatement.java#L1597), and will split batches of other sizes up into multiple sub-batches. This means that a batch of size 128 can be 6x faster than a batch of size 250. - -The code snippet below shows a pattern for using a batch size of 128, and is taken from the longer example above (specifically, the `BasicExampleDAO.bulkInsertRandomAccountData()` method). - -Specifically, it does the following: - -1. Turn off auto-commit so you can manage the transaction lifecycle and thus the size of the batch inserts. -2. Given an overall update size of 500 rows (for example), split it into batches of size 128 and execute each batch in turn. -3. Finally, commit the batches of statements you've just executed. - -{% include_cached copy-clipboard.html %} -~~~ java -int BATCH_SIZE = 128; -connection.setAutoCommit(false); - -try (PreparedStatement pstmt = connection.prepareStatement("INSERT INTO accounts (id, balance) VALUES (?, ?)")) { - for (int i=0; i<=(500/BATCH_SIZE);i++) { - for (int j=0; j %s row(s) updated in this batch\n", count.length); // Verifying 128 rows in the batch - } - connection.commit(); -} -~~~ - -### Retrieve large data sets in chunks using cursors - -CockroachDB now supports the Postgres wire-protocol cursors for implicit transactions and explicit transactions executed to completion. This means the [PGJDBC driver](https://jdbc.postgresql.org) can use this protocol to stream queries with large result sets. This is much faster than [paginating through results in SQL using `LIMIT .. OFFSET`](pagination.html). - -For instructions showing how to use cursors in your Java code, see [Getting results based on a cursor](https://jdbc.postgresql.org/documentation/query/#getting-results-based-on-a-cursor) from the PGJDBC documentation. - -Note that interleaved execution (partial execution of multiple statements within the same connection and transaction) is not supported when [`Statement.setFetchSize()`](https://docs.oracle.com/javase/8/docs/api/java/sql/Statement.html#setFetchSize-int-) is used. - -## What's next? - -Read more about using the [Java JDBC driver](https://jdbc.postgresql.org/). - -{% include {{page.version.version}}/app/see-also-links.md %} diff --git a/src/current/v21.1/build-a-nodejs-app-with-cockroachdb-knexjs.md b/src/current/v21.1/build-a-nodejs-app-with-cockroachdb-knexjs.md deleted file mode 100644 index 1b457a8f28c..00000000000 --- a/src/current/v21.1/build-a-nodejs-app-with-cockroachdb-knexjs.md +++ /dev/null @@ -1,95 +0,0 @@ ---- -title: Build a Simple CRUD Node.js App with CockroachDB and Knex.js -summary: Learn how to use CockroachDB from a simple CRUD application that uses the Knex.js query builder. -toc: true -twitter: false -referral_id: docs_node_knexjs ---- - -
- - - - - -
- -This tutorial shows you how build a simple Node.js application with CockroachDB and [Knex.js](https://knexjs.org/). - -## Step 1. Start CockroachDB - -{% include {{ page.version.version }}/app/sample-setup.md %} - -## Step 2. Get the code - -Clone the code's GitHub repo: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ git clone https://github.com/cockroachlabs/example-app-node-knex -~~~ - -## Step 3. Run the code - -To start the app: - -1. Set the `DATABASE_URL` environment variable to the connection string: - -
- - {% include_cached copy-clipboard.html %} - ~~~ shell - $ export DATABASE_URL="postgresql://root@localhost:26257?sslmode=disable" - ~~~ - -
- -
- - {% include_cached copy-clipboard.html %} - ~~~ shell - $ export DATABASE_URL="" - ~~~ - - Where `` is the connection string you obtained from the CockroachDB {{ site.data.products.cloud }} Console. - -
- - The app uses the connection string saved to the `DATABASE_URL` environment variable to connect to your cluster and execute the code. - -1. Install the app requirements: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ npm install - ~~~ - -1. Run the app: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ npm run run - ~~~ - - The output should look like this: - - ~~~ - Initializing accounts table... - New account balances: - { id: 'bc0f7136-7103-4dc4-88fc-102d5bbd312f', balance: '1000' } - { id: '35bde7d0-29c9-4277-a117-3eb80c85ae16', balance: '250' } - { id: '18cc1b2d-be61-4a8d-942c-480f5a6bc207', balance: '0' } - Transferring funds... - New account balances: - { id: 'bc0f7136-7103-4dc4-88fc-102d5bbd312f', balance: '900' } - { id: '35bde7d0-29c9-4277-a117-3eb80c85ae16', balance: '350' } - { id: '18cc1b2d-be61-4a8d-942c-480f5a6bc207', balance: '0' } - Deleting a row... - New account balances: - { id: 'bc0f7136-7103-4dc4-88fc-102d5bbd312f', balance: '900' } - { id: '18cc1b2d-be61-4a8d-942c-480f5a6bc207', balance: '0' } - ~~~ - -## What's next? - -{% include {{page.version.version}}/app/see-also-links.md %} diff --git a/src/current/v21.1/build-a-nodejs-app-with-cockroachdb-prisma.md b/src/current/v21.1/build-a-nodejs-app-with-cockroachdb-prisma.md deleted file mode 100644 index c1f5912502b..00000000000 --- a/src/current/v21.1/build-a-nodejs-app-with-cockroachdb-prisma.md +++ /dev/null @@ -1,228 +0,0 @@ ---- -title: Build a Simple CRUD Node.js App with CockroachDB and Prisma -summary: Learn how to use CockroachDB from a simple CRUD application that uses Prisma. -toc: true -twitter: false -referral_id: docs_node_prisma ---- - -
- - - - - -
- -This tutorial shows you how build a simple Node.js application with CockroachDB and [Prisma](https://www.prisma.io). - -## Step 1. Start CockroachDB - -{% include {{ page.version.version }}/app/start-cockroachdb-no-cert.md %} - -## Step 2. Get the code - -1. Clone the code's GitHub repo: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ git clone https://github.com/cockroachlabs/example-app-node-prisma - ~~~ - -1. Install the application dependencies: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cd example-app-node-prisma - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ npm install - ~~~ - -## Step 3. Initialize the database - -1. Create a `.env` file for your project, and set the `DATABASE_URL` environment variable to a valid connection string to your cluster. - -
- - {% include_cached copy-clipboard.html %} - ~~~ shell - $ echo "DATABASE_URL=" >> .env - ~~~ - - Where `` is the connection string you obtained earlier from the CockroachDB {{ site.data.products.cloud }} Console. - -
- -
- - {% include_cached copy-clipboard.html %} - ~~~ shell - $ echo "DATABASE_URL=postgresql://root@localhost:26257?sslmode=disable" >> .env - ~~~ - - {{site.data.alerts.callout_info}} - `postgresql://root@localhost:26257?sslmode=disable` is the `sql` connection string you obtained earlier from the `cockroach` welcome text. - {{site.data.alerts.end}} - -
- - Prisma loads the variables defined in `.env` to the project environment. By default, Prisma uses the `DATABASE_URL` environment variable as the connection string to the database. - -1. Run [Prisma Migrate](https://www.prisma.io/docs/concepts/components/prisma-migrate) to initialize the database with the schema defined in `prisma/prisma.schema`. - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ node_modules/.bin/prisma migrate dev --name init - ~~~ - - You should see the following output: - - ~~~ - Your database is now in sync with your schema. - - ✔ Generated Prisma Client (3.12.0 | library) to ./node_modules/@prisma/client in 73ms - ~~~ - - This command also initializes [Prisma Client](https://www.prisma.io/docs/concepts/components/prisma-client) to communicate with your CockroachDB cluster, based on the configuration in the `prisma/schema.prisma` file. - -## Step 4. Run the code - -The `index.js` file contains the code for `INSERT`, `SELECT`, `UPDATE`, and `DELETE` SQL operations: - -{% include_cached copy-clipboard.html %} -~~~ js -{% remote_include https://raw.githubusercontent.com/cockroachlabs/example-app-node-prisma/main/index.js %} -~~~ - -{{site.data.alerts.callout_info}} -In [production](recommended-production-settings.html#transaction-retries), we recommend implementing [client-side transaction retries](transactions.html#client-side-intervention) for all database operations. -{{site.data.alerts.end}} - -Run the application code: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ node index.js -~~~ - -~~~ -Customer rows inserted. { count: 10 } -Account rows inserted. { count: 10 } -Initial Account row values: - [ - { - id: '1961432f-f93e-4568-b2a5-ba08f73afde5', - customer_id: '079daee3-ecf2-4a0f-980b-4c3ea4c8b6a3', - balance: 914n - }, - { - id: '4ccd7eea-eb47-4aa9-9819-30b5aae58bf8', - customer_id: 'c0eeb465-ab60-4f02-9bf3-3451578d400d', - balance: 176n - }, - { - id: '53ed4f7d-72ee-4390-9487-9bf318357c77', - customer_id: 'a4c9e26e-f9d8-4c1b-ac20-1aa8611b134f', - balance: 54n - }, - { - id: '79a1f1b2-4050-4329-bf52-5df53fec749e', - customer_id: '392c7d15-5ab2-4149-9eee-8a3a44b36e9d', - balance: 482n - }, - { - id: '7e30f1e0-e873-4565-9ea3-3079a48a4886', - customer_id: '12cb3406-264a-417c-b0e6-86593e60dc18', - balance: 478n - }, - { - id: '94f461d5-3985-46c1-98f4-1896f15f0a16', - customer_id: 'e4c909a4-6683-429d-9831-dfcf792f4fb0', - balance: 240n - }, - { - id: 'a0c081f5-fb15-47cc-8dbb-85c6f15677d2', - customer_id: '91ece5f2-df03-4023-b112-2b4d5677981b', - balance: 520n - }, - { - id: 'a45b7c41-2f62-4620-be69-57d5d61186e4', - customer_id: 'c1824327-d6a1-4916-a666-ea157ef2a409', - balance: 50n - }, - { - id: 'dbe0dec5-257b-42ff-9d36-1b5a57e1a4ac', - customer_id: '6739eb2f-bcb1-4074-aab4-5860b04d227d', - balance: 468n - }, - { - id: 'ebc520b4-8df0-4e2f-8426-104594f6341c', - customer_id: 'f83e02cb-77cf-4347-9e0c-28cad65fac34', - balance: 336n - } -] -Account rows updated. { count: 8 } -Updated Account row values: - [ - { - id: '1961432f-f93e-4568-b2a5-ba08f73afde5', - customer_id: '079daee3-ecf2-4a0f-980b-4c3ea4c8b6a3', - balance: 909n - }, - { - id: '4ccd7eea-eb47-4aa9-9819-30b5aae58bf8', - customer_id: 'c0eeb465-ab60-4f02-9bf3-3451578d400d', - balance: 171n - }, - { - id: '53ed4f7d-72ee-4390-9487-9bf318357c77', - customer_id: 'a4c9e26e-f9d8-4c1b-ac20-1aa8611b134f', - balance: 54n - }, - { - id: '79a1f1b2-4050-4329-bf52-5df53fec749e', - customer_id: '392c7d15-5ab2-4149-9eee-8a3a44b36e9d', - balance: 477n - }, - { - id: '7e30f1e0-e873-4565-9ea3-3079a48a4886', - customer_id: '12cb3406-264a-417c-b0e6-86593e60dc18', - balance: 473n - }, - { - id: '94f461d5-3985-46c1-98f4-1896f15f0a16', - customer_id: 'e4c909a4-6683-429d-9831-dfcf792f4fb0', - balance: 235n - }, - { - id: 'a0c081f5-fb15-47cc-8dbb-85c6f15677d2', - customer_id: '91ece5f2-df03-4023-b112-2b4d5677981b', - balance: 515n - }, - { - id: 'a45b7c41-2f62-4620-be69-57d5d61186e4', - customer_id: 'c1824327-d6a1-4916-a666-ea157ef2a409', - balance: 50n - }, - { - id: 'dbe0dec5-257b-42ff-9d36-1b5a57e1a4ac', - customer_id: '6739eb2f-bcb1-4074-aab4-5860b04d227d', - balance: 463n - }, - { - id: 'ebc520b4-8df0-4e2f-8426-104594f6341c', - customer_id: 'f83e02cb-77cf-4347-9e0c-28cad65fac34', - balance: 331n - } -] -All Customer rows deleted. { count: 10 } -~~~ - -## What's next? - -Read more about using [Prisma Client](https://www.prisma.io/docs/). - -{% include {{page.version.version}}/app/see-also-links.md %} diff --git a/src/current/v21.1/build-a-nodejs-app-with-cockroachdb-sequelize.md b/src/current/v21.1/build-a-nodejs-app-with-cockroachdb-sequelize.md deleted file mode 100644 index 04c49044396..00000000000 --- a/src/current/v21.1/build-a-nodejs-app-with-cockroachdb-sequelize.md +++ /dev/null @@ -1,100 +0,0 @@ ---- -title: Build a Node.js App with CockroachDB and Sequelize -summary: Learn how to use CockroachDB from a simple Node.js application with the Sequelize ORM. -toc: true -twitter: false -referral_id: docs_hello_world_nodejs_sequelize ---- - -
- - - - - -
- -This tutorial shows you how build a simple Node.js application with CockroachDB and the Sequelize ORM. - -We have tested the [Sequelize ORM](https://sequelize.org/) enough to claim **beta-level** support. If you encounter problems, please [open an issue](https://github.com/cockroachdb/cockroach/issues/new) with details to help us make progress toward full support. - -{{site.data.alerts.callout_success}} -For a more realistic use of Sequelize with CockroachDB, see our [`examples-orms`](https://github.com/cockroachdb/examples-orms)repository. -{{site.data.alerts.end}} - -## Step 1. Start CockroachDB - -{% include {{page.version.version}}/app/start-cockroachdb.md %} - -## Step 2. Create a database - -{% include {{page.version.version}}/app/create-a-database.md %} - -## Step 3. Install the Sequelize ORM - -To install Sequelize, as well as a [CockroachDB Node.js package](https://github.com/cockroachdb/sequelize-cockroachdb) that accounts for some minor differences between CockroachDB and PostgreSQL, run the following command: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ npm install sequelize sequelize-cockroachdb -~~~ - -## Step 4. Get the code - -Download the sample code directly, or clone [the code's GitHub repository](https://github.com/cockroachlabs/example-app-node-sequelize). - -## Step 5. Update the connection parameters - -Open `app.js`, and edit the connection configuration parameters: - -
- -- Replace the value for `username` with the user you created earlier. -- Replace the value for `password` with the password you created for your user. -- Replace the value for `port` with the port to your cluster. - -
- -
- -- At the top of the file, uncomment the `const fs = require('fs');` line. - - This line imports the `fs` Node module, which enables you to read in the CA cert that you downloaded from the CockroachDB {{ site.data.products.cloud }} Console. -- Replace the value for `username` with the user you created earlier. -- Replace the value for `password` with the password you created for your user. -- Replace the value for `host` with the name of the CockroachDB {{ site.data.products.cloud }} Free host (e.g., `host: 'free-tier.gcp-us-central1.cockroachlabs.cloud'`). -- Replace the value for `port` with the port to your cluster. -- Replace the value for `database` with the database that you created earlier, suffixed with the name of the cluster (e.g., `database: '{cluster_name}.bank'`). -- Remove the `rejectUnauthorized` key-value pair. -- Uncomment the `ca` key-value pair, and edit the `fs.readFileSync('certs/ca.crt').toString()` call to use the path to the `cc-ca.crt` file that you downloaded from the CockroachDB {{ site.data.products.cloud }} Console. - -
- -## Step 6. Run the code - -The following code uses the [Sequelize](https://sequelize.org/) ORM to map Node.js-specific objects to SQL operations. Specifically, `Account.sync({force: true})` creates an `accounts` table based on the Account model (or drops and recreates the table if it already exists), `Account.bulkCreate([...])` inserts rows into the table, and `Account.findAll()` selects from the table so that balances can be printed. - -{% include_cached copy-clipboard.html %} -~~~ js -{% remote_include https://raw.githubusercontent.com/cockroachlabs/example-app-node-sequelize/main/app.js %} -~~~ - -To run the code: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ node app.js -~~~ - -The output should be: - -~~~ shell -1 1000 -2 250 -~~~ - -## What's next? - -Read more about using the [Sequelize ORM](https://sequelize.org/), or check out a more realistic implementation of Sequelize with CockroachDB in our [`examples-orms`](https://github.com/cockroachdb/examples-orms) repository. - -{% include {{ page.version.version }}/app/see-also-links.md %} diff --git a/src/current/v21.1/build-a-nodejs-app-with-cockroachdb.md b/src/current/v21.1/build-a-nodejs-app-with-cockroachdb.md deleted file mode 100644 index db4314f57d9..00000000000 --- a/src/current/v21.1/build-a-nodejs-app-with-cockroachdb.md +++ /dev/null @@ -1,130 +0,0 @@ ---- -title: Build a Simple CRUD Node.js App with CockroachDB and the node-postgres Driver -summary: Learn how to use CockroachDB from a simple CRUD application that uses the node-postgres driver. -toc: true -twitter: false -referral_id: docs_hello_world_nodejs_pg ---- - -
- - - - - -
- -This tutorial shows you how build a simple Node.js application with CockroachDB and the [node-postgres driver](https://node-postgres.com/). - -## Step 1. Start CockroachDB - -{% include {{ page.version.version }}/app/sample-setup.md %} - -## Step 2. Get the code - -Clone the code's GitHub repo: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ git clone https://github.com/cockroachlabs/example-app-node-postgres -~~~ - -The project has the following directory structure: - -~~~ -├── README.md -├── app.js -├── dbinit.sql -└── package.json -~~~ - -The `dbinit.sql` file initializes the database schema that the application uses: - -{% include_cached copy-clipboard.html %} -~~~ sql -{% remote_include https://raw.githubusercontent.com/cockroachlabs/example-app-node-postgres/main/dbinit.sql %} -~~~ - -The `app.js` file contains the code for `INSERT`, `SELECT`, `UPDATE`, and `DELETE` SQL operations: - -{% include_cached copy-clipboard.html %} -~~~ js -{% remote_include https://raw.githubusercontent.com/cockroachlabs/example-app-node-postgres/main/app.js %} -~~~ - -All of the database operations are wrapped in a helper function named `retryTxn`. This function attempts to commit statements in the context of an explicit transaction. If a [retry error](transaction-retry-error-reference.html) is thrown, the wrapper will retry committing the transaction, with [exponential backoff](https://en.wikipedia.org/wiki/Exponential_backoff), until the maximum number of retries is reached (by default, 15). - -## Step 3. Initialize the database - -{% include {{ page.version.version }}/app/init-bank-sample.md %} - -## Step 4. Run the code - -1. Install the app requirements: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ npm install - ~~~ - -1. Run the app: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ node app.js - ~~~ - - The program will prompt you for a connection string to the database: - - ~~~ - prompt: connectionString: - ~~~ - -1. Enter the connection string to your running cluster. - -
- - {{site.data.alerts.callout_success}} - `postgresql://root@localhost:26257?sslmode=disable` should be the `sql` connection URL provided in the `cockroach` welcome text. - {{site.data.alerts.end}} - -
- -
- - {{site.data.alerts.callout_success}} - Use the connection string provided in the **Connection info** window of the CockroachDB {{ site.data.products.cloud }} Console. - {{site.data.alerts.end}} - - {{site.data.alerts.callout_info}} - You need to provide a SQL user password in order to securely connect to a CockroachDB {{ site.data.products.cloud }} cluster. The connection string should have a placeholder for the password (``). - {{site.data.alerts.end}} - -
- -After entering the connection string, the program will execute. - -The output should look like this: - -~~~ -Initializing accounts table... -New account balances: -{ id: 'aa0e9b22-0c23-469b-a9e1-b2ace079f44c', balance: '1000' } -{ id: 'bf8b96da-2c38-4d55-89a0-b2b6ed63ff9e', balance: '0' } -{ id: 'e43d76d6-388e-4ee6-8b73-a063a63a2138', balance: '250' } -Transferring funds... -New account balances: -{ id: 'aa0e9b22-0c23-469b-a9e1-b2ace079f44c', balance: '900' } -{ id: 'bf8b96da-2c38-4d55-89a0-b2b6ed63ff9e', balance: '0' } -{ id: 'e43d76d6-388e-4ee6-8b73-a063a63a2138', balance: '350' } -Deleting a row... -New account balances: -{ id: 'aa0e9b22-0c23-469b-a9e1-b2ace079f44c', balance: '900' } -{ id: 'e43d76d6-388e-4ee6-8b73-a063a63a2138', balance: '350' } -~~~ - -## What's next? - -Read more about using the [node-postgres driver](https://www.npmjs.com/package/pg). - -{% include {{page.version.version}}/app/see-also-links.md %} diff --git a/src/current/v21.1/build-a-python-app-with-cockroachdb-django.md b/src/current/v21.1/build-a-python-app-with-cockroachdb-django.md deleted file mode 100644 index 64e22ba054f..00000000000 --- a/src/current/v21.1/build-a-python-app-with-cockroachdb-django.md +++ /dev/null @@ -1,325 +0,0 @@ ---- -title: Build a Python App with CockroachDB and Django -summary: Learn how to use CockroachDB from a simple Django application. -toc: true -twitter: false -referral_id: docs_hello_world_python_django ---- - -
- - - - - -
- -This tutorial shows you how build a simple Python application with CockroachDB and the [Django](https://www.djangoproject.com/) framework. - -CockroachDB supports Django versions 3.1+. - -{{site.data.alerts.callout_info}} -The example code and instructions on this page use Python 3.9 and Django 3.1. -{{site.data.alerts.end}} - -## Step 1. Start CockroachDB - -{% include {{page.version.version}}/app/start-cockroachdb.md %} - -## Step 2. Create a database - -{% include {{page.version.version}}/app/create-a-database.md %} - -## Step 3. Get the sample code - -
- -Clone the code's GitHub repo: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ git clone https://github.com/cockroachlabs/example-app-python-django/ -~~~ - -The project directory structure should look like this: - -~~~ -├── Dockerfile -├── README.md -├── cockroach_example -│   ├── cockroach_example -│   │   ├── __init__.py -│   │   ├── asgi.py -│   │   ├── migrations -│   │   │   ├── 0001_initial.py -│   │   │   └── __init__.py -│   │   ├── models.py -│   │   ├── settings.py -│   │   ├── urls.py -│   │   ├── views.py -│   │   └── wsgi.py -│   └── manage.py -└── requirements.txt -~~~ - -
- -
- -1. Clone the code's GitHub repo: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ git clone https://github.com/cockroachlabs/example-app-python-django/ - ~~~ - -1. Create a new folder named `certs` at the top level of the `example-app-python-django` project, and then copy the root certificate that you downloaded for your cluster to the new folder. - - The project directory structure should look like this: - - ~~~ - ├── Dockerfile - ├── README.md - ├── certs - │   └── root.crt - ├── cockroach_example - │   ├── cockroach_example - │   │   ├── __init__.py - │   │   ├── asgi.py - │   │   ├── migrations - │   │   │   ├── 0001_initial.py - │   │   │   └── __init__.py - │   │   ├── models.py - │   │   ├── settings.py - │   │   ├── urls.py - │   │   ├── views.py - │   │   └── wsgi.py - │   └── manage.py - └── requirements.txt - ~~~ - -
- -## Step 4. Install the application requirements - -To use CockroachDB with Django, the following modules are required: - -- [`django`](https://docs.djangoproject.com/en/3.1/topics/install/) -- [`psycopg2`](https://pypi.org/project/psycopg2/) (recommended for production environments) or [`psycopg2-binary`](https://pypi.org/project/psycopg2-binary/) (recommended for development and testing). -- [`django-cockroachdb`](https://github.com/cockroachdb/django-cockroachdb) - -{{site.data.alerts.callout_info}} -The major version of `django-cockroachdb` must correspond to the major version of `django`. The minor release numbers do not need to match. -{{site.data.alerts.end}} - -The `requirements.txt` file at the top level of the `example-app-python-django` project directory contains a list of the requirements needed to run this application: - -{% include_cached copy-clipboard.html %} -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/example-app-python-django/master/requirements.txt %} -~~~ - -{{site.data.alerts.callout_info}} -The `requirements.txt` file also lists the `dj_database_url` module, which is not a strict requirement. The sample app uses this module to configure the database connection from a connection URL. -{{site.data.alerts.end}} - -1. At the top level of the app's project directory, create and then activate a virtual environment: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ virtualenv env - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ source env/bin/activate - ~~~ - -1. Install the modules listed in `requirements.txt` to the virtual environment: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ pip install -r requirements.txt - ~~~ - -## Step 5. Configure the database connection - -The `cockroach_example/cockroach_example/settings.py` file defines database connection information for the application, in [the `DATABASES` dictionary](https://docs.djangoproject.com/en/3.2/ref/settings/#databases): - -{% include_cached copy-clipboard.html %} -~~~ python -DATABASES = {} -DATABASES['default'] = dj_database_url.config(default=os.path.expandvars( - os.environ['DATABASE_URL']), engine='django_cockroachdb') -~~~ - -Note that, rather than using [discrete connection parameters](connection-parameters.html), the sample `settings.py` passes a single variable (the `DATABASE_URL` environment variable) to the `dj_database_url` module. - -Set the `DATABASE_URL` environment variable to the connection string: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ export DATABASE_URL="" -~~~ - -
- -Where `` is the `sql` connection URL provided in the cluster's welcome text. - -
- -
- -Where `` is the connection string provided in the **Connection info** window of the CockroachDB {{ site.data.products.cloud }} Console, but with the root certificate located in the local `certs` directory. - -Note that you also need to provide a SQL user password in order to securely connect to a CockroachDB {{ site.data.products.cloud }} cluster. The connection string should have a placeholder for the password (``). - -
- -## Step 6. Build out the application - -After you have configured the app's database connection, you can start building out the application. - -### Models - -Start by building some [models](https://docs.djangoproject.com/en/3.1/topics/db/models/), defined in a file called `models.py`. You can copy the sample code below and paste it into a new file, or you can download the file directly. - -{% include_cached copy-clipboard.html %} -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/example-app-python-django/master/cockroach_example/cockroach_example/models.py %} -~~~ - -In this file, we define some simple classes that map to the tables in the example database `bank`. - -### Views - -Next, build out some [class-based views](https://docs.djangoproject.com/en/3.1/topics/class-based-views/) for the application in a file called `views.py`. You can copy the sample code below and paste it into a new file, or you can download the file directly. - -{% include_cached copy-clipboard.html %} -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/example-app-python-django/master/cockroach_example/cockroach_example/views.py %} -~~~ - -This file defines the application's views as classes. Each view class corresponds to one of the table classes defined in `models.py`. The methods of these classes define read and write transactions on the tables in the database. - -Importantly, the file defines a [transaction retry loop](transactions.html#transaction-retries) in the decorator function `retry_on_exception()`. This function decorates each view method, ensuring that transaction ordering guarantees meet the ANSI [SERIALIZABLE](https://en.wikipedia.org/wiki/Isolation_(database_systems)#Serializable) isolation level. For more information about how transactions (and retries) work, see [Transactions](transactions.html). - -### URL routes - -Lastly, define some [URL routes](https://docs.djangoproject.com/en/3.1/topics/http/urls/) in a file called `urls.py`. You can copy the sample code below and paste it into the existing `urls.py` file, or you can download the file directly and replace the existing one. - -{% include_cached copy-clipboard.html %} -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/example-app-python-django/master/cockroach_example/cockroach_example/urls.py %} -~~~ - -## Step 7. Initialize the database - -1. In the top `cockroach_example` directory, use the [`manage.py` script](https://docs.djangoproject.com/en/3.1/ref/django-admin/) to create [Django migrations](https://docs.djangoproject.com/en/3.1/topics/migrations/) that initialize the database for the application: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ python3 manage.py makemigrations cockroach_example - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ python3 manage.py migrate - ~~~ - - This initializes the `bank` database with the tables defined in `models.py`, in addition to some other tables for the admin functionality included with Django's starter application. - -1. To verify that the migration succeeded, open the terminal with the SQL shell to the temporary CockroachDB cluster, and issue the following statements: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > USE bank; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW TABLES; - ~~~ - - ~~~ - schema_name | table_name | type | estimated_row_count - --------------+----------------------------------+-------+---------------------- - public | auth_group | table | 0 - public | auth_group_permissions | table | 0 - public | auth_permission | table | 36 - public | auth_user | table | 0 - public | auth_user_groups | table | 0 - public | auth_user_user_permissions | table | 0 - public | cockroach_example_customers | table | 0 - public | cockroach_example_orders | table | 0 - public | cockroach_example_orders_product | table | 0 - public | cockroach_example_products | table | 0 - public | django_admin_log | table | 0 - public | django_content_type | table | 9 - public | django_migrations | table | 1 - public | django_session | table | 0 - (14 rows) - ~~~ - -## Step 8. Run the app - -1. In a different terminal, navigate to the top of the `cockroach_example` directory, and start the app: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ python3 manage.py runserver 0.0.0.0:8000 - ~~~ - - The output should look like this: - - ~~~ - ... - Starting development server at http://0.0.0.0:8000/ - Quit the server with CONTROL-C. - ~~~ - - To perform simple reads and writes to the database, you can send HTTP requests to the application server listening at `http://0.0.0.0:8000/`. - -1. In a new terminal, use `curl` to send a POST request to the application: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ curl --header "Content-Type: application/json" \ - --request POST \ - --data '{"name":"Carl"}' http://0.0.0.0:8000/customer/ - ~~~ - - This request inserts a new row into the `cockroach_example_customers` table. - -1. Send a GET request to read from the `cockroach_example_customers` table: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ curl http://0.0.0.0:8000/customer/ - ~~~ - - ~~~ - [{"id": "bb7d6c4d-efb3-45f8-b790-9911aae7d8b2", "name": "Carl"}] - ~~~ - - You can also query the table directly in the [SQL shell](cockroach-sql.html) to see the changes: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SELECT * FROM bank.cockroach_example_customers; - ~~~ - - ~~~ - id | name - ---------------------------------------+------- - bb7d6c4d-efb3-45f8-b790-9911aae7d8b2 | Carl - (1 row) - ~~~ - -1. Enter **Ctrl+C** to stop the application. - -## What's next? - -Read more about writing a [Django app](https://docs.djangoproject.com/en/3.1/intro/tutorial01/). - -{% include {{page.version.version}}/app/see-also-links.md %} diff --git a/src/current/v21.1/build-a-python-app-with-cockroachdb-sqlalchemy.md b/src/current/v21.1/build-a-python-app-with-cockroachdb-sqlalchemy.md deleted file mode 100644 index ba3b4a0096b..00000000000 --- a/src/current/v21.1/build-a-python-app-with-cockroachdb-sqlalchemy.md +++ /dev/null @@ -1,236 +0,0 @@ ---- -title: Build a Simple CRUD Python App with CockroachDB and SQLAlchemy -summary: Learn how to use CockroachDB from a simple Python application with SQLAlchemy. -toc: true -twitter: false -referral_id: docs_hello_world_python_sqlalchemy ---- - -
- - - - - -
- -{% include cockroach_u_pydev.md %} - -This tutorial shows you how build a simple CRUD Python application with CockroachDB and the [SQLAlchemy](https://docs.sqlalchemy.org/en/latest/) ORM. - -## Step 1. Start CockroachDB - -{% include {{ page.version.version }}/app/sample-setup.md %} - -## Step 2. Get the code - -Clone the code's GitHub repo: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ git clone https://github.com/cockroachlabs/example-app-python-sqlalchemy/ -~~~ - -The project has the following directory structure: - -~~~ -├── README.md -├── dbinit.sql -├── main.py -├── models.py -└── requirements.txt -~~~ - -The `requirements.txt` file includes the required libraries to connect to CockroachDB with SQLAlchemy, including the [`sqlalchemy-cockroachdb` Python package](https://github.com/cockroachdb/sqlalchemy-cockroachdb), which accounts for some differences between CockroachDB and PostgreSQL: - -{% include_cached copy-clipboard.html %} -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/example-app-python-sqlalchemy/master/requirements.txt %} -~~~ - -The `dbinit.sql` file initializes the database schema that the application uses: - -{% include_cached copy-clipboard.html %} -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/example-app-python-sqlalchemy/master/dbinit.sql %} -~~~ - -The `models.py` uses SQLAlchemy to map the `Accounts` table to a Python object: - -{% include_cached copy-clipboard.html %} -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/example-app-python-sqlalchemy/master/models.py %} -~~~ - -The `main.py` uses SQLAlchemy to map Python methods to SQL operations: - -{% include_cached copy-clipboard.html %} -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/example-app-python-sqlalchemy/master/main.py %} -~~~ - -`main.py` also executes the `main` method of the program. - -## Step 3. Install the application requirements - -1. At the top level of the app's project directory, create and then activate a virtual environment: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ virtualenv env - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ source env/bin/activate - ~~~ - -1. Install the required modules to the virtual environment: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ pip install -r requirements.txt - ~~~ - -## Step 4. Initialize the database - -{% include {{ page.version.version }}/app/init-bank-sample.md %} - -## Step 5. Run the code - -1. Set the `DATABASE_URL` environment variable to the connection string for your cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ export DATABASE_URL= - ~~~ - -
- - Where `` is the `sql` connection URL provided in the cluster's welcome text, but **with the `database` parameter set to `bank` instead of `defaultdb`.** - -
- -
- - Where `` is the connection string you obtained earlier from the CockroachDB {{ site.data.products.cloud }} Console, but **with the `database` parameter set to `bank` instead of `defaultdb`.** - -
- -1. Run the app: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ python main.py - ~~~ - - The application will connect to CockroachDB, and then perform some simple row inserts, updates, and deletes. - - The output should look something like the following: - - ~~~ - Creating new accounts... - Created new account with id 3a8b74c8-6a05-4247-9c60-24b46e3a88fd and balance 248835. - Created new account with id c3985926-5b77-4c6d-a73d-7c0d4b2a51e7 and balance 781972. - ... - Created new account with id 7b41386c-11d3-465e-a2a0-56e0dcd2e7db and balance 984387. - Random account balances: - Account 7ad14d02-217f-48ca-a53c-2c3a2528a0d9: 800795 - Account 4040aeba-7194-4f29-b8e5-a27ed4c7a297: 149861 - Transferring 400397 from account 7ad14d02-217f-48ca-a53c-2c3a2528a0d9 to account 4040aeba-7194-4f29-b8e5-a27ed4c7a297... - Transfer complete. - New balances: - Account 7ad14d02-217f-48ca-a53c-2c3a2528a0d9: 400398 - Account 4040aeba-7194-4f29-b8e5-a27ed4c7a297: 550258 - Deleting existing accounts... - Deleted account 41247e24-6210-4032-b622-c10b3c7222de. - Deleted account 502450e4-6daa-4ced-869c-4dff62dc52de. - Deleted account 6ff06ef0-423a-4b08-8b87-48af2221bc18. - Deleted account a1acb134-950c-4882-9ac7-6d6fbdaaaee1. - Deleted account e4f33c55-7230-4080-b5ac-5dde8a7ae41d. - ~~~ - - In a SQL shell connected to the cluster, you can verify that the rows were inserted, updated, and deleted successfully: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SELECT COUNT(*) FROM bank.accounts; - ~~~ - - ~~~ - count - --------- - 95 - (1 row) - ~~~ - -## Best practices - -### Use the `run_transaction` function - -We strongly recommend using the [`sqlalchemy_cockroachdb.run_transaction()`](https://github.com/cockroachdb/sqlalchemy-cockroachdb/blob/master/sqlalchemy_cockroachdb/transaction.py) function as shown in the code samples on this page. This abstracts the details of [transaction retries](transactions.html#transaction-retries) away from your application code. Transaction retries are more frequent in CockroachDB than in some other databases because we use [optimistic concurrency control](https://en.wikipedia.org/wiki/Optimistic_concurrency_control) rather than locking. Because of this, a CockroachDB transaction may have to be tried more than once before it can commit. This is part of how we ensure that our transaction ordering guarantees meet the ANSI [SERIALIZABLE](https://en.wikipedia.org/wiki/Isolation_(database_systems)#Serializable) isolation level. - -In addition to the above, using `run_transaction` has the following benefits: - -- Because it must be passed a [sqlalchemy.orm.session.sessionmaker](https://docs.sqlalchemy.org/en/latest/orm/session_api.html#session-and-sessionmaker) object (*not* a [session][session]), it ensures that a new session is created exclusively for use by the callback, which protects you from accidentally reusing objects via any sessions created outside the transaction. -- It abstracts away the [client-side transaction retry logic](transactions.html#client-side-intervention) from your application, which keeps your application code portable across different databases. For example, the sample code given on this page works identically when run against Postgres (modulo changes to the prefix and port number in the connection string). - -For more information about how transactions (and retries) work, see [Transactions](transactions.html). - -### Avoid mutations of session and/or transaction state inside `run_transaction()` - -In general, this is in line with the recommendations of the [SQLAlchemy FAQs](https://docs.sqlalchemy.org/en/latest/orm/session_basics.html#session-frequently-asked-questions), which state (with emphasis added by the original author) that - -> As a general rule, the application should manage the lifecycle of the session *externally* to functions that deal with specific data. This is a fundamental separation of concerns which keeps data-specific operations agnostic of the context in which they access and manipulate that data. - -and - -> Keep the lifecycle of the session (and usually the transaction) **separate and external**. - -In keeping with the above recommendations from the official docs, we **strongly recommend** avoiding any explicit mutations of the transaction state inside the callback passed to `run_transaction`, since that will lead to breakage. Specifically, do not make calls to the following functions from inside `run_transaction`: - -- [`sqlalchemy.orm.Session.commit()`](https://docs.sqlalchemy.org/en/latest/orm/session_api.html?highlight=commit#sqlalchemy.orm.session.Session.commit) (or other variants of `commit()`): This is not necessary because `cockroachdb.sqlalchemy.run_transaction` handles the savepoint/commit logic for you. -- [`sqlalchemy.orm.Session.rollback()`](https://docs.sqlalchemy.org/en/latest/orm/session_api.html?highlight=rollback#sqlalchemy.orm.session.Session.rollback) (or other variants of `rollback()`): This is not necessary because `cockroachdb.sqlalchemy.run_transaction` handles the commit/rollback logic for you. -- [`Session.flush()`][session.flush]: This will not work as expected with CockroachDB because CockroachDB does not support nested transactions, which are necessary for `Session.flush()` to work properly. If the call to `Session.flush()` encounters an error and aborts, it will try to rollback. This will not be allowed by the currently-executing CockroachDB transaction created by `run_transaction()`, and will result in an error message like the following: `sqlalchemy.orm.exc.DetachedInstanceError: Instance is not bound to a Session; attribute refresh operation cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)`. - -### Break up large transactions into smaller units of work - -If you see an error message like `transaction is too large to complete; try splitting into pieces`, you are trying to commit too much data in a single transaction. As described in our [Cluster Settings](cluster-settings.html) docs, the size limit for transactions is defined by the `kv.transaction.max_intents_bytes` setting, which defaults to 256 KiB. Although this setting can be changed by an admin, we strongly recommend against it in most cases. - -Instead, we recommend breaking your transaction into smaller units of work (or "chunks"). A pattern that works for inserting large numbers of objects using `run_transaction` to handle retries automatically for you is shown below. - -{% include_cached copy-clipboard.html %} -~~~ python -{% include {{page.version.version}}/app/python/sqlalchemy/sqlalchemy-large-txns.py %} -~~~ - -### Use `IMPORT` to read in large data sets - -If you are trying to get a large data set into CockroachDB all at once (a bulk import), avoid writing client-side code that uses an ORM and use the [`IMPORT`](import.html) statement instead. It is much faster and more efficient than making a series of [`INSERT`s](insert.html) and [`UPDATE`s](update.html) such as are generated by calls to [`session.bulk_save_objects()`](https://docs.sqlalchemy.org/en/latest/orm/session_api.html?highlight=bulk_save_object#sqlalchemy.orm.session.Session.bulk_save_objects). - -For more information about importing data from Postgres, see [Migrate from Postgres](migrate-from-postgres.html). - -For more information about importing data from MySQL, see [Migrate from MySQL](migrate-from-mysql.html). - -### Prefer the query builder - -In general, we recommend using the query-builder APIs of SQLAlchemy (e.g., [`Engine.execute()`](https://docs.sqlalchemy.org/en/latest/core/connections.html?highlight=execute#sqlalchemy.engine.Engine.execute)) in your application over the [Session][session]/ORM APIs if at all possible. That way, you know exactly what SQL is being generated and sent to CockroachDB, which has the following benefits: - -- It's easier to debug your SQL queries and make sure they are working as expected. -- You can more easily tune SQL query performance by issuing different statements, creating and/or using different indexes, etc. For more information, see [SQL Performance Best Practices](performance-best-practices-overview.html). - -### Joins without foreign keys - -SQLAlchemy relies on the existence of [foreign keys](foreign-key.html) to generate [`JOIN` expressions](joins.html) from your application code. If you remove foreign keys from your schema, SQLAlchemy will not generate joins for you. As a workaround, you can [create a "custom foreign condition" by adding a `relationship` field to your table objects](https://stackoverflow.com/questions/37806625/sqlalchemy-create-relations-but-without-foreign-key-constraint-in-db), or do the equivalent work in your application. - -## See also - -- The [SQLAlchemy](https://docs.sqlalchemy.org/en/latest/) docs -- [Transactions](transactions.html) - -{% include {{page.version.version}}/app/see-also-links.md %} - - - -[session.flush]: https://docs.sqlalchemy.org/en/latest/orm/session_api.html#sqlalchemy.orm.session.Session.flush -[session]: https://docs.sqlalchemy.org/en/latest/orm/session.html diff --git a/src/current/v21.1/build-a-python-app-with-cockroachdb.md b/src/current/v21.1/build-a-python-app-with-cockroachdb.md deleted file mode 100644 index 844fc5cc4b8..00000000000 --- a/src/current/v21.1/build-a-python-app-with-cockroachdb.md +++ /dev/null @@ -1,121 +0,0 @@ ---- -title: Build a Python App with CockroachDB and psycopg2 -summary: Learn how to use CockroachDB from a simple Python application with the psycopg2 driver. -toc: true -twitter: false -referral_id: docs_hello_world_python_psycopg2 ---- - -
- - - - - -
- -{% include cockroach_u_pydev.md %} - -This tutorial shows you how build a simple Python application with CockroachDB and the psycopg2 driver. For the CockroachDB back-end, you'll use a temporary local cluster. - -## Step 1. Install the psycopg2 driver - -To install the Python psycopg2 driver, run the following command: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ pip install psycopg2-binary -~~~ - -For other ways to install psycopg2, see the [official documentation](http://initd.org/psycopg/docs/install.html). - -## Step 2. Start CockroachDB - -{% include {{page.version.version}}/app/start-cockroachdb.md %} - -## Step 3. Create a database - -{% include {{page.version.version}}/app/create-a-database.md %} - -## Step 4. Run the Python code - -Now that you have a database, you'll run the code shown below to: - -- Create an accounts table and insert some rows. -- Transfer funds between two accounts inside a [transaction](transactions.html). -- Delete the accounts from the table before exiting so you can re-run the example code. - -To [handle transaction retry errors](error-handling-and-troubleshooting.html#transaction-retry-errors), the code uses an application-level retry loop that, in case of error, sleeps before trying the funds transfer again. If it encounters another retry error, it sleeps for a longer interval, implementing [exponential backoff](https://en.wikipedia.org/wiki/Exponential_backoff). - -### Get the code - -Download the [`example.py`](https://raw.githubusercontent.com/cockroachlabs/hello-world-python-psycopg2/master/example.py) file, or create the file yourself and copy the code into it. - -If you prefer, you can also clone a version of the code: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ git clone https://github.com/cockroachlabs/hello-world-python-psycopg2/ -~~~ - -{% include_cached copy-clipboard.html %} -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/hello-world-python-psycopg2/master/example.py %} -~~~ - -### Run the code - -The Python code is a command-line utility that accepts the connection string to CockroachDB as a command-line argument: - -
- -{% include_cached copy-clipboard.html %} -~~~ shell -$ python3 example.py \ -"postgresql://:@:/bank?sslmode=require" -~~~ - -Before running the command, update the connection string as follows: - -- Replace `` and `` with the SQL username and password that you created earlier. -- Replace `` and `` with the hostname and port in the `(sql)` connection string from SQL shell welcome text. - -
- -
- -{% include_cached copy-clipboard.html %} -~~~ shell -$ python3 example.py \ -"postgres://:@:26257/.bank?sslmode=verify-full&sslrootcert=/cc-ca.crt" -~~~ - -Before running the command, update the connection string that you copied [earlier](#set-up-your-cluster-connection) from the **Connection info** dialog as follows: - -- Replace `` and `` with the SQL username and password that you created earlier. -- Replace `` with the name of the CockroachDB {{ site.data.products.serverless }} host (e.g., `free-tier.gcp-us-central1.cockroachlabs.cloud`). -- Replace `` with the name of your cluster. -- Replace `` with the path to the `cc-ca.crt` file that you downloaded from the CockroachDB {{ site.data.products.cloud }} Console. - -{{site.data.alerts.callout_info}} -If you are using the connection string that you [copied from the **Connection info** dialog](#set-up-your-cluster-connection), your username, password, and cluster name will be pre-populated. -{{site.data.alerts.end}} - -
- -The output should show the account balances before and after the funds transfer: - -~~~ -Balances at Fri Oct 30 18:27:00 2020: -(1, 1000) -(2, 250) -Balances at Fri Oct 30 18:27:00 2020: -(1, 900) -(2, 350) -~~~ - -## What's next? - -Read more about using the [Python psycopg2 driver](http://initd.org/psycopg/docs/). - -{% include {{page.version.version}}/app/see-also-links.md %} diff --git a/src/current/v21.1/build-a-ruby-app-with-cockroachdb-activerecord.md b/src/current/v21.1/build-a-ruby-app-with-cockroachdb-activerecord.md deleted file mode 100644 index c6202501597..00000000000 --- a/src/current/v21.1/build-a-ruby-app-with-cockroachdb-activerecord.md +++ /dev/null @@ -1,144 +0,0 @@ ---- -title: Build a Ruby App with CockroachDB and ActiveRecord -summary: Learn how to use CockroachDB from a simple Ruby script with the ActiveRecord gem. -toc: true -twitter: false -referral_id: docs_hello_world_ruby_activerecord ---- - -
- - -
- -This tutorial shows you how build a simple Ruby application with CockroachDB and [ActiveRecord](http://guides.rubyonrails.org/active_record_basics.html). CockroachDB provides an ActiveRecord adapter for CockroachDB as a [RubyGem](https://rubygems.org/gems/activerecord-cockroachdb-adapter). - -{{site.data.alerts.callout_success}} -For a more realistic use of ActiveRecord with CockroachDB in a Rails app, see our [`examples-orms`](https://github.com/cockroachdb/examples-orms) repository. -{{site.data.alerts.end}} - -## Step 1. Start CockroachDB - -{% include {{page.version.version}}/app/start-cockroachdb.md %} - -## Step 2. Create a database - -{% include {{page.version.version}}/app/create-a-database.md %} - -## Step 3. Get the code - -
- - -
- -Clone [the code's GitHub repository](https://github.com/cockroachlabs/example-app-ruby-activerecord). - -{% include_cached copy-clipboard.html %} -~~~ shell -git clone https://github.com/cockroachlabs/example-app-ruby-activerecord -~~~ - -
-
-Check out the `5.2` branch: - -{% include_cached copy-clipboard.html %} -~~~ shell -git checkout 5.2 -~~~ -
- -
- -
- -
-Check out the `cockroachcloud` branch: - -{% include_cached copy-clipboard.html %} -~~~shell -git checkout cockroachcloud -~~~ - -
-
-Check out the `cockroachcloud-5.2` branch: - -{% include_cached copy-clipboard.html %} -~~~ shell -git checkout cockroachcloud-5.2 -~~~ -
- -
- -## Step 4. Configure the dependencies - -1. Install `libpq` for your platform. For example, to install it on Mac with Homebrew: - {% include_cached copy-clipboard.html %} - ~~~shell - brew install libpq - ~~~ -1. Configure `bundle` to use `libpq`. For example, if you installed `libpq` on Mac using Homebrew: - {% include_cached copy-clipboard.html %} - ~~~shell - bundle config --local build.pg --with-opt-dir="/usr/local/opt/libpq" - ~~~ - Set `--with-opt-dir` to the location of `libpq` on your OS. - -## Step 5. Install the dependencies - -{% include_cached copy-clipboard.html %} -~~~shell -bundle install -~~~ - -## Step 6. Update the connection parameters - -Update the connection parameters to connect to your cluster. - -
- -{% include_cached copy-clipboard.html %} -~~~ ruby -{% remote_include https://raw.githubusercontent.com/cockroachlabs/example-app-ruby-activerecord/main/main.rb||# BEGIN connect||# END connect %} -~~~ - -Where `{port}` is the port number from the connection string you noted earlier, `{username}` is the database username you created, and `{password}` is the database user's password. - -
-
- -{% include_cached copy-clipboard.html %} -~~~ ruby -{% remote_include https://raw.githubusercontent.com/cockroachlabs/example-app-ruby-activerecord/cockroachcloud/main.rb||# BEGIN connect||# END connect %} -~~~ - -{% include {{page.version.version}}/app/cc-free-tier-params.md %} - -
- -## Step 7. Run the Ruby code - -Run the code to create a table and insert some rows, and then you'll run code to read and update values as an atomic [transaction](transactions.html). - -{% include_cached copy-clipboard.html %} -~~~ shell -ruby main.rb -~~~ - -The output should be: - -~~~ --- create_table(:accounts, {:force=>true, :id=>:integer}) - -> 0.3951s -account: 1 balance: 1000 -account: 2 balance: 250 -~~~ - -## What's next? - -Read more about using [ActiveRecord](http://guides.rubyonrails.org/active_record_basics.html), or check out a more realistic implementation of ActiveRecord with CockroachDB in a Rails app in our [`examples-orms`](https://github.com/cockroachdb/examples-orms) repository. - -{% include {{page.version.version}}/app/see-also-links.md %} diff --git a/src/current/v21.1/build-a-ruby-app-with-cockroachdb.md b/src/current/v21.1/build-a-ruby-app-with-cockroachdb.md deleted file mode 100644 index a4ccd8b945a..00000000000 --- a/src/current/v21.1/build-a-ruby-app-with-cockroachdb.md +++ /dev/null @@ -1,120 +0,0 @@ ---- -title: Build a Ruby App with CockroachDB and the Ruby pg Driver -summary: Learn how to use CockroachDB from a simple Ruby application with the pg client driver. -toc: true -twitter: false -referral_id: docs_hello_world_ruby_pg ---- - -
- - -
- -This tutorial shows you how build a simple Ruby application with CockroachDB and the [Ruby pg driver](https://deveiate.org/code/pg/PG/Connection.html). - -## Step 1. Start CockroachDB - -{% include {{page.version.version}}/app/start-cockroachdb.md %} - -## Step 2. Create a database - -{% include {{page.version.version}}/app/create-a-database.md %} - -## Step 3. Get the code - -Clone [the code's GitHub repository](https://github.com/cockroachlabs/hello-world-ruby-pg). - -{% include_cached copy-clipboard.html %} -~~~shell -git clone https://github.com/cockroachlabs/hello-world-ruby-pg -~~~ - -The code connects as the user you created and executes some basic SQL statements: creating a table, inserting rows, and reading and printing the rows. - -
- -Check out the `cockroachcloud` branch: - -{% include_cached copy-clipboard.html %} -~~~ shell -git checkout cockroachcloud -~~~ - -
- -## Step 4. Configure the dependencies - -1. Install `libpq` for your platform. For example, to install it on Mac with Homebrew: - {% include_cached copy-clipboard.html %} - ~~~shell - brew install libpq - ~~~ -1. Configure `bundle` to use `libpq`. For example, if you installed `libpq` on Mac using Homebrew: - {% include_cached copy-clipboard.html %} - ~~~shell - bundle config --local build.pg --with-opt-dir="/usr/local/opt/libpq" - ~~~ - Set `--with-opt-dir` to the location of `libpq` on your OS. - -## Step 5. Install the dependencies - -{% include_cached copy-clipboard.html %} -~~~shell -bundle install -~~~ - -## Step 6. Update the connection parameters - -Update the connection parameters to connect to your cluster. - -
- -{% include_cached copy-clipboard.html %} -~~~ ruby -{% remote_include https://raw.githubusercontent.com/cockroachlabs/hello-world-ruby-pg/master/main.rb||# BEGIN connect||# END connect %} -~~~ - -Where `{port}` is the port number from the connection string you noted earlier, `{username}` is the database username you created, and `{password}` is the database user's password. - -
-
- -{% include_cached copy-clipboard.html %} -~~~ ruby -{% remote_include https://raw.githubusercontent.com/cockroachlabs/hello-world-ruby-pg/cockroachcloud/main.rb||# BEGIN connect||# END connect %} -~~~ - -{% include {{page.version.version}}/app/cc-free-tier-params.md %} - -
- -## Step 7. Run the Ruby code - -Run the code to create a table and insert some rows, and then you'll run code to read and update values as an atomic [transaction](transactions.html). - -{% include_cached copy-clipboard.html %} -~~~ shell -ruby main.rb -~~~ - -The output should be: - -~~~ ------------------------------------------------- -print_balances(): Balances as of '2021-02-23 11:56:54 -0800': -{"id"=>"1", "balance"=>"1000"} -{"id"=>"2", "balance"=>"250"} ------------------------------------------------- -transfer_funds(): Trying to transfer 100 from account 1 to account 2 ------------------------------------------------- -print_balances(): Balances as of '2021-02-23 11:56:55 -0800': -{"id"=>"1", "balance"=>"900"} -{"id"=>"2", "balance"=>"350"} -~~~ - -## What's next? - -Read more about using the [Ruby pg driver](https://rubygems.org/gems/pg). - -{% include {{page.version.version}}/app/see-also-links.md %} diff --git a/src/current/v21.1/build-a-rust-app-with-cockroachdb.md b/src/current/v21.1/build-a-rust-app-with-cockroachdb.md deleted file mode 100644 index 8641b68f6de..00000000000 --- a/src/current/v21.1/build-a-rust-app-with-cockroachdb.md +++ /dev/null @@ -1,92 +0,0 @@ ---- -title: Build a Rust App with CockroachDB and the Rust Postgres Driver -summary: Learn how to use CockroachDB from a simple Rust application with a low-level client driver. -toc: true -twitter: false -docs_area: get_started ---- - -This tutorial shows you how build a simple Rust application with CockroachDB and the [Rust-Postgres driver](https://github.com/sfackler/rust-postgres). - -We have tested the Rust-Postgres driver enough to claim **beta-level** support. If you encounter problems, please [open an issue](https://github.com/cockroachdb/cockroach/issues/new) with details to help us make progress toward full support. - -## Before you begin - -{% include {{page.version.version}}/app/before-you-begin.md %} - -## Step 1. Specify the Rust Postgres driver as a dependency - -Update your `Cargo.toml` file to specify a dependency on the Rust Postgres driver, as described in the official documentation. - -Additionally, include the OpenSSL bindings and Rust Postgres OpenSSL crates as dependencies. - -## Step 2. Create the `maxroach` users and `bank` database - -{% include {{page.version.version}}/app/create-maxroach-user-and-bank-database.md %} - -## Step 3. Generate a certificate for the `maxroach` user - -Create a certificate and key for the `maxroach` user by running the following command. The code samples will run as this user. - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach cert create-client maxroach --certs-dir=certs --ca-key=my-safe-directory/ca.key -~~~ - -## Step 4. Run the Rust code - -Now that you have a database and a user, you'll run code to create a table and insert some rows, and then you'll run code to read and update values as an atomic [transaction](transactions.html). - -### Get the code - -Clone the [`example-app-rust-postgres`](https://github.com/cockroachdb/example-app-rust-postgres) GitHub repo: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ git clone https://github.com/cockroachdb/example-app-rust-postgres -~~~ - -### Basic statements - -First, run `basic-sample.rs` to connect as the `maxroach` user and execute some basic SQL statements, inserting rows and reading and printing the rows: - -{% include_cached copy-clipboard.html %} -~~~ sql -{% remote_include https://raw.githubusercontent.com/cockroachdb/example-app-rust-postgres/main/basic-sample.rs %} -~~~ - -### Transaction (with retry logic) - -Next, run `txn-sample.rs` to again connect as the `maxroach` user but this time execute a batch of statements as an atomic transaction to transfer funds from one account to another, where all included statements are either committed or aborted. - -{{site.data.alerts.callout_info}} -CockroachDB may require the [client to retry a transaction](transactions.html#transaction-retries) in case of read/write contention. CockroachDB provides a generic retry function that runs inside a transaction and retries it as needed. You can copy and paste the retry function from here into your code. -{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ sql -{% remote_include https://raw.githubusercontent.com/cockroachdb/example-app-rust-postgres/main/txn-sample.rs %} -~~~ - -After running the code, use the [built-in SQL client](cockroach-sql.html) to verify that funds were transferred from one account to another: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --certs-dir=certs -e 'SELECT id, balance FROM accounts' --database=bank -~~~ - -~~~ -+----+---------+ -| id | balance | -+----+---------+ -| 1 | 900 | -| 2 | 350 | -+----+---------+ -(2 rows) -~~~ - -## What's next? - -Read more about using the Rust Postgres driver. - -{% include {{ page.version.version }}/app/see-also-links.md %} \ No newline at end of file diff --git a/src/current/v21.1/build-a-spring-app-with-cockroachdb-jdbc.md b/src/current/v21.1/build-a-spring-app-with-cockroachdb-jdbc.md deleted file mode 100644 index a3e2fa194ec..00000000000 --- a/src/current/v21.1/build-a-spring-app-with-cockroachdb-jdbc.md +++ /dev/null @@ -1,818 +0,0 @@ ---- -title: Build a Spring App with CockroachDB and JDBC -summary: Learn how to use CockroachDB from a Spring application with the JDBC driver. -toc: true -twitter: false -referral_id: docs_roach_data_java_spring_jdbc ---- - -
- - -
- -This tutorial shows you how to build a [Spring Boot](https://spring.io/projects/spring-boot) web application with CockroachDB, using the [Spring Data JDBC](https://spring.io/projects/spring-data-jdbc) module for data access. The code for the example application is available for download from [GitHub](https://github.com/cockroachlabs/roach-data/tree/master), along with identical examples that use [JPA](https://github.com/cockroachlabs/roach-data/tree/master/roach-data-jpa), [jOOQ](https://github.com/cockroachlabs/roach-data/tree/master/roach-data-jooq), and [MyBatis](https://github.com/cockroachlabs/roach-data/tree/master/roach-data-mybatis) for data access. - -## Step 1. Start CockroachDB - -Choose whether to run a local cluster or a free CockroachDB cluster on CockroachDB {{ site.data.products.serverless }}. - -
- - -
- -
- -1. If you haven't already, [download the CockroachDB binary](install-cockroachdb.html). -1. [Start a local, secure cluster](secure-a-cluster.html). - -
- -
- -### Create a free cluster - -{% include cockroachcloud/quickstart/create-a-free-cluster.md %} - -### Set up your cluster connection - -{% include cockroachcloud/quickstart/set-up-your-cluster-connection.md %} - -
- -## Step 2. Create a database and a user - -
- -1. Open a SQL shell to your local cluster using the [`cockroach sql`](cockroach-sql.html) command: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --certs-dir={certs-dir} --host=localhost:{port} - ~~~ - - Where `{certs_dir}` is the full path to the `certs` directory that you created when setting up the cluster, and `{port}` is the port at which the cluster is listening for incoming connections. - -1. In the SQL shell, create the `roach_data` database that your application will use: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE DATABASE roach_data; - ~~~ - -1. Create a SQL user for your app: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE USER {username} WITH PASSWORD {password}; - ~~~ - - Take note of the username and password. You will use it to connect to the database later. - -1. Give the user the necessary permissions: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > GRANT ALL ON DATABASE roach_data TO {username}; - ~~~ - -1. Exit the shell, and generate a certificate and key for your user by running the following command: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach cert create-client {user} --certs-dir={certs-dir} --ca-key={certs-dir}/ca.key --also-generate-pkcs8-key -~~~ - -The [`--also-generate-pkcs8-key` flag](cockroach-cert.html#flag-pkcs8) generates a key in [PKCS#8 format](https://tools.ietf.org/html/rfc5208), which is the standard key encoding format in Java. In this case, the generated PKCS8 key will be named `client.{user}.key.pk8`. - -
- -
- -1. If you haven't already, [download the CockroachDB binary](install-cockroachdb.html). -1. Start the [built-in SQL shell](cockroach-sql.html) using the connection string you got from the CockroachDB {{ site.data.products.cloud }} Console [earlier](#set-up-your-cluster-connection): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql \ - --url 'postgresql://@-..:26257/?sslmode=verify-full&sslrootcert='$HOME'/Library/CockroachCloud/certs/-ca.crt' - ~~~ - -1. Enter your SQL user password. - -1. In the SQL shell, create the `roach_data` database that your application will use: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE DATABASE roach_data; - ~~~ - -
- -## Step 3. Install JDK - -Download and install a Java Development Kit. Spring Boot supports Java versions 8, 11, and 14. In this tutorial, we use [JDK 8 from OpenJDK](https://openjdk.java.net/install/). - -## Step 4. Install Maven - -This example application uses [Maven](http://maven.apache.org/) to manage all application dependencies. Spring supports Maven versions 3.2 and later. - -To install Maven on macOS, run the following command: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ brew install maven -~~~ - -To install Maven on a Debian-based Linux distribution like Ubuntu: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ apt-get install maven -~~~ - -To install Maven on a Red Hat-based Linux distribution like Fedora: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ dnf install maven -~~~ - -For other ways to install Maven, see [its official documentation](http://maven.apache.org/install.html). - -## Step 5. Get the application code - -To get the application code, download or clone the [`roach-data` repository](https://github.com/cockroachlabs/roach-data). The code for the example JDBC application is located under the `roach-data-jdbc` directory. - -(*Optional*) To recreate the application project structure with the same dependencies as those used by this sample application, you can use [Spring initializr](https://start.spring.io/) with the following settings: - -**Project** - -- Maven Project - -**Language** - -- Java - -**Spring Boot** - -- 2.2.6 - -**Project Metadata** - -- Group: io.roach -- Artifact: data -- Name: data -- Package name: io.roach.data -- Packaging: Jar -- Java: 8 - -**Dependencies** - -- Spring Web -- Spring Data JDBC -- Spring Boot Actuator -- Spring HATEOS -- Liquibase Migration -- PostgreSQL Driver - - -## Step 6. Run the application - -Compiling and running the application code will start a web application, initialize the `accounts` table in the `roach_data` database, and submit some requests to the app's REST API that result in [atomic database transactions](transactions.html) on the running CockroachDB cluster. For details about the application code, see [Implementation details](#implementation-details). - -Open the `roach-data/roach-data-jdbc/src/main/resources/application.yml` file and edit the `datasource` settings to connect to your running database cluster: - -
- -~~~ yml - ... -datasource: - url: jdbc:postgresql://localhost:{port}/roach_data?ssl=true&sslmode=require&sslrootcert={certs-dir}/ca.crt&sslkey={certs-dir}/client.{username}.key.pk8&sslcert={certs-dir}/client.{username}.crt - username: {username} - password: {password} - driver-class-name: org.postgresql.Driver - ... -~~~ - -Where: - -- `{port}` is the port number. -- `{certs-dir}` is the full path to the certificates directory containing the authentication certificates that you created earlier. -- `{username}` and `{password}` specify the SQL username and password that you created earlier. - -
- -
- -~~~ yml -... -datasource: - url: jdbc:postgresql://{globalhost}:{port}/{cluster_name}.roach_data?sslmode=verify-full&sslrootcert={path to the CA certificate}/cc-ca.crt - username: {username} - password: {password} - driver-class-name: org.postgresql.Driver -... -~~~ - -{% include {{page.version.version}}/app/cc-free-tier-params.md %} - -
- -Open a terminal, and navigate to the `roach-data-jdbc` project subfolder: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cd /roach-data/roach-data-jdbc -~~~ - -Use Maven to download the application dependencies and compile the code: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ mvn clean install -~~~ - -From the `roach-data-jdbc` directory, run the application JAR file: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ java -jar target/roach-data-jdbc.jar -~~~ - -The output should look like the following: - -~~~ -^__^ -(oo)\_______ -(__)\ )\/\ CockroachDB on Spring Data JDBC (v1.0.0.BUILD-SNAPSHOT) - ||----w | powered by Spring Boot (v2.2.7.RELEASE) - || || - -2020-06-17 14:56:54.507 INFO 43008 --- [ main] io.roach.data.jdbc.JdbcApplication : Starting JdbcApplication v1.0.0.BUILD-SNAPSHOT on MyComputer with PID 43008 (path/roach-data/roach-data-jdbc/target/roach-data-jdbc.jar started by user in path/roach-data/roach-data-jdbc) -2020-06-17 14:56:54.510 INFO 43008 --- [ main] io.roach.data.jdbc.JdbcApplication : No active profile set, falling back to default profiles: default -2020-06-17 14:56:55.387 INFO 43008 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JDBC repositories in DEFAULT mode. -2020-06-17 14:56:55.452 INFO 43008 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 59ms. Found 2 JDBC repository interfaces. -2020-06-17 14:56:56.581 INFO 43008 --- [ main] org.eclipse.jetty.util.log : Logging initialized @3378ms to org.eclipse.jetty.util.log.Slf4jLog -2020-06-17 14:56:56.657 INFO 43008 --- [ main] o.s.b.w.e.j.JettyServletWebServerFactory : Server initialized with port: 9090 -2020-06-17 14:56:56.661 INFO 43008 --- [ main] org.eclipse.jetty.server.Server : jetty-9.4.28.v20200408; built: 2020-04-08T17:49:39.557Z; git: ab228fde9e55e9164c738d7fa121f8ac5acd51c9; jvm 11.0.7+10 -2020-06-17 14:56:56.696 INFO 43008 --- [ main] o.e.j.s.h.ContextHandler.application : Initializing Spring embedded WebApplicationContext -2020-06-17 14:56:56.696 INFO 43008 --- [ main] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 2088 ms -2020-06-17 14:56:57.170 INFO 43008 --- [ main] org.eclipse.jetty.server.session : DefaultSessionIdManager workerName=node0 -2020-06-17 14:56:57.171 INFO 43008 --- [ main] org.eclipse.jetty.server.session : No SessionScavenger set, using defaults -2020-06-17 14:56:57.172 INFO 43008 --- [ main] org.eclipse.jetty.server.session : node0 Scavenging every 600000ms -2020-06-17 14:56:57.178 INFO 43008 --- [ main] o.e.jetty.server.handler.ContextHandler : Started o.s.b.w.e.j.JettyEmbeddedWebAppContext@deb3b60{application,/,[file:///private/var/folders/pg/r58v54857gq_1nqm_2tr6lg40000gn/T/jetty-docbase.3049902632643053896.8080/],AVAILABLE} -2020-06-17 14:56:57.179 INFO 43008 --- [ main] org.eclipse.jetty.server.Server : Started @3976ms -2020-06-17 14:56:58.126 INFO 43008 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor' -2020-06-17 14:56:58.369 INFO 43008 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting... -2020-06-17 14:56:58.695 INFO 43008 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed. -2020-06-17 14:56:59.901 INFO 43008 --- [ main] liquibase.executor.jvm.JdbcExecutor : SELECT COUNT(*) FROM public.databasechangeloglock -2020-06-17 14:56:59.917 INFO 43008 --- [ main] liquibase.executor.jvm.JdbcExecutor : CREATE TABLE public.databasechangeloglock (ID INTEGER NOT NULL, LOCKED BOOLEAN NOT NULL, LOCKGRANTED TIMESTAMP WITHOUT TIME ZONE, LOCKEDBY VARCHAR(255), CONSTRAINT DATABASECHANGELOGLOCK_PKEY PRIMARY KEY (ID)) -2020-06-17 14:56:59.930 INFO 43008 --- [ main] liquibase.executor.jvm.JdbcExecutor : SELECT COUNT(*) FROM public.databasechangeloglock -2020-06-17 14:56:59.950 INFO 43008 --- [ main] liquibase.executor.jvm.JdbcExecutor : DELETE FROM public.databasechangeloglock -2020-06-17 14:56:59.953 INFO 43008 --- [ main] liquibase.executor.jvm.JdbcExecutor : INSERT INTO public.databasechangeloglock (ID, LOCKED) VALUES (1, FALSE) -2020-06-17 14:56:59.959 INFO 43008 --- [ main] liquibase.executor.jvm.JdbcExecutor : SELECT LOCKED FROM public.databasechangeloglock WHERE ID=1 -2020-06-17 14:56:59.969 INFO 43008 --- [ main] l.lockservice.StandardLockService : Successfully acquired change log lock -2020-06-17 14:57:01.367 INFO 43008 --- [ main] l.c.StandardChangeLogHistoryService : Creating database history table with name: public.databasechangelog -2020-06-17 14:57:01.369 INFO 43008 --- [ main] liquibase.executor.jvm.JdbcExecutor : CREATE TABLE public.databasechangelog (ID VARCHAR(255) NOT NULL, AUTHOR VARCHAR(255) NOT NULL, FILENAME VARCHAR(255) NOT NULL, DATEEXECUTED TIMESTAMP WITHOUT TIME ZONE NOT NULL, ORDEREXECUTED INTEGER NOT NULL, EXECTYPE VARCHAR(10) NOT NULL, MD5SUM VARCHAR(35), DESCRIPTION VARCHAR(255), COMMENTS VARCHAR(255), TAG VARCHAR(255), LIQUIBASE VARCHAR(20), CONTEXTS VARCHAR(255), LABELS VARCHAR(255), DEPLOYMENT_ID VARCHAR(10)) -2020-06-17 14:57:01.380 INFO 43008 --- [ main] liquibase.executor.jvm.JdbcExecutor : SELECT COUNT(*) FROM public.databasechangelog -2020-06-17 14:57:01.396 INFO 43008 --- [ main] l.c.StandardChangeLogHistoryService : Reading from public.databasechangelog -2020-06-17 14:57:01.397 INFO 43008 --- [ main] liquibase.executor.jvm.JdbcExecutor : SELECT * FROM public.databasechangelog ORDER BY DATEEXECUTED ASC, ORDEREXECUTED ASC -2020-06-17 14:57:01.400 INFO 43008 --- [ main] liquibase.executor.jvm.JdbcExecutor : SELECT COUNT(*) FROM public.databasechangeloglock -2020-06-17 14:57:01.418 INFO 43008 --- [ main] liquibase.executor.jvm.JdbcExecutor : -- DROP TABLE IF EXISTS account cascade; --- DROP TABLE IF EXISTS databasechangelog cascade; --- DROP TABLE IF EXISTS databasechangeloglock cascade; - -create table account -( - id int not null primary key default unique_rowid(), - balance numeric(19, 2) not null, - name varchar(128) not null, - type varchar(25) not null -) -2020-06-17 14:57:01.426 INFO 43008 --- [ main] liquibase.executor.jvm.JdbcExecutor : -- insert into account (id,balance,name,type) values --- (1, 500.00,'Alice','asset'), --- (2, 500.00,'Bob','expense'), --- (3, 500.00,'Bobby Tables','asset'), --- (4, 500.00,'Doris','expense'); -2020-06-17 14:57:01.427 INFO 43008 --- [ main] liquibase.changelog.ChangeSet : SQL in file db/create.sql executed -2020-06-17 14:57:01.430 INFO 43008 --- [ main] liquibase.changelog.ChangeSet : ChangeSet classpath:db/changelog-master.xml::1::root ran successfully in 14ms -2020-06-17 14:57:01.430 INFO 43008 --- [ main] liquibase.executor.jvm.JdbcExecutor : SELECT MAX(ORDEREXECUTED) FROM public.databasechangelog -2020-06-17 14:57:01.441 INFO 43008 --- [ main] liquibase.executor.jvm.JdbcExecutor : INSERT INTO public.databasechangelog (ID, AUTHOR, FILENAME, DATEEXECUTED, ORDEREXECUTED, MD5SUM, DESCRIPTION, COMMENTS, EXECTYPE, CONTEXTS, LABELS, LIQUIBASE, DEPLOYMENT_ID) VALUES ('1', 'root', 'classpath:db/changelog-master.xml', NOW(), 1, '8:939a1a8c47676119a94d0173802d207e', 'sqlFile', '', 'EXECUTED', 'crdb', NULL, '3.8.9', '2420221402') -2020-06-17 14:57:01.450 INFO 43008 --- [ main] liquibase.executor.jvm.JdbcExecutor : INSERT INTO public.account (id, name, balance, type) VALUES ('1', 'Alice', 500.00, 'asset') -2020-06-17 14:57:01.459 INFO 43008 --- [ main] liquibase.changelog.ChangeSet : New row inserted into account -2020-06-17 14:57:01.460 INFO 43008 --- [ main] liquibase.executor.jvm.JdbcExecutor : INSERT INTO public.account (id, name, balance, type) VALUES ('2', 'Bob', 500.00, 'expense') -2020-06-17 14:57:01.462 INFO 43008 --- [ main] liquibase.changelog.ChangeSet : New row inserted into account -2020-06-17 14:57:01.462 INFO 43008 --- [ main] liquibase.executor.jvm.JdbcExecutor : INSERT INTO public.account (id, name, balance, type) VALUES ('3', 'Bobby Tables', 500.00, 'asset') -2020-06-17 14:57:01.464 INFO 43008 --- [ main] liquibase.changelog.ChangeSet : New row inserted into account -2020-06-17 14:57:01.465 INFO 43008 --- [ main] liquibase.executor.jvm.JdbcExecutor : INSERT INTO public.account (id, name, balance, type) VALUES ('4', 'Doris', 500.00, 'expense') -2020-06-17 14:57:01.467 INFO 43008 --- [ main] liquibase.changelog.ChangeSet : New row inserted into account -2020-06-17 14:57:01.469 INFO 43008 --- [ main] liquibase.changelog.ChangeSet : ChangeSet classpath:db/changelog-master.xml::2::root ran successfully in 19ms -2020-06-17 14:57:01.470 INFO 43008 --- [ main] liquibase.executor.jvm.JdbcExecutor : INSERT INTO public.databasechangelog (ID, AUTHOR, FILENAME, DATEEXECUTED, ORDEREXECUTED, MD5SUM, DESCRIPTION, COMMENTS, EXECTYPE, CONTEXTS, LABELS, LIQUIBASE, DEPLOYMENT_ID) VALUES ('2', 'root', 'classpath:db/changelog-master.xml', NOW(), 2, '8:c2945f2a445cf60b4b203e1a91d14a89', 'insert tableName=account; insert tableName=account; insert tableName=account; insert tableName=account', '', 'EXECUTED', 'crdb', NULL, '3.8.9', '2420221402') -2020-06-17 14:57:01.479 INFO 43008 --- [ main] l.lockservice.StandardLockService : Successfully released change log lock -2020-06-17 14:57:01.555 INFO 43008 --- [ main] o.s.b.a.e.web.EndpointLinksResolver : Exposing 8 endpoint(s) beneath base path '/actuator' -2020-06-17 14:57:01.610 INFO 43008 --- [ main] o.e.j.s.h.ContextHandler.application : Initializing Spring DispatcherServlet 'dispatcherServlet' -2020-06-17 14:57:01.610 INFO 43008 --- [ main] o.s.web.servlet.DispatcherServlet : Initializing Servlet 'dispatcherServlet' -2020-06-17 14:57:01.620 INFO 43008 --- [ main] o.s.web.servlet.DispatcherServlet : Completed initialization in 10 ms -2020-06-17 14:57:01.653 INFO 43008 --- [ main] o.e.jetty.server.AbstractConnector : Started ServerConnector@733c423e{HTTP/1.1, (http/1.1)}{0.0.0.0:9090} -2020-06-17 14:57:01.654 INFO 43008 --- [ main] o.s.b.web.embedded.jetty.JettyWebServer : Jetty started on port(s) 9090 (http/1.1) with context path '/' -2020-06-17 14:57:01.657 INFO 43008 --- [ main] io.roach.data.jdbc.JdbcApplication : Started JdbcApplication in 7.92 seconds (JVM running for 8.454) -2020-06-17 14:57:01.659 INFO 43008 --- [ main] io.roach.data.jdbc.JdbcApplication : Lets move some $$ around! -2020-06-17 14:57:03.552 INFO 43008 --- [ main] io.roach.data.jdbc.JdbcApplication : Worker finished - 7 remaining -2020-06-17 14:57:03.606 INFO 43008 --- [ main] io.roach.data.jdbc.JdbcApplication : Worker finished - 6 remaining -2020-06-17 14:57:03.606 INFO 43008 --- [ main] io.roach.data.jdbc.JdbcApplication : Worker finished - 5 remaining -2020-06-17 14:57:03.607 INFO 43008 --- [ main] io.roach.data.jdbc.JdbcApplication : Worker finished - 4 remaining -2020-06-17 14:57:03.608 INFO 43008 --- [ main] io.roach.data.jdbc.JdbcApplication : Worker finished - 3 remaining -2020-06-17 14:57:03.608 INFO 43008 --- [ main] io.roach.data.jdbc.JdbcApplication : Worker finished - 2 remaining -2020-06-17 14:57:03.608 INFO 43008 --- [ main] io.roach.data.jdbc.JdbcApplication : Worker finished - 1 remaining -2020-06-17 14:57:03.608 INFO 43008 --- [ main] io.roach.data.jdbc.JdbcApplication : Worker finished - 0 remaining -2020-06-17 14:57:03.608 INFO 43008 --- [ main] io.roach.data.jdbc.JdbcApplication : All client workers finished but server keeps running. Have a nice day! -~~~ - -As the output states, the application configures a database connection, starts a web servlet listening on the address `http://localhost:9090/`, initializes the `account` table and changelog tables with [Liquibase](https://www.liquibase.org/), and then runs some test operations as requests to the application's REST API. - -For more details about the application code, see [Implementation details](#implementation-details). - -### Query the database - -#### Reads - -The `http://localhost:9090/account` endpoint returns information about all accounts in the database. `GET` requests to these endpoints are executed on the database as `SELECT` statements. - -The following `curl` command sends a `GET` request to the endpoint. The `json_pp` command formats the JSON response. - -{% include_cached copy-clipboard.html %} -~~~ shell -$ curl -X GET http://localhost:9090/account | json_pp -~~~ - -~~~ -{ - "_embedded" : { - "accounts" : [ - { - "_links" : { - "self" : { - "href" : "http://localhost:9090/account/1" - } - }, - "balance" : 500, - "name" : "Alice", - "type" : "asset" - }, - { - "_links" : { - "self" : { - "href" : "http://localhost:9090/account/2" - } - }, - "balance" : 500, - "name" : "Bob", - "type" : "expense" - }, - { - "_links" : { - "self" : { - "href" : "http://localhost:9090/account/3" - } - }, - "balance" : 500, - "name" : "Bobby Tables", - "type" : "asset" - }, - { - "_links" : { - "self" : { - "href" : "http://localhost:9090/account/4" - } - }, - "balance" : 500, - "name" : "Doris", - "type" : "expense" - } - ] - }, - "_links" : { - "self" : { - "href" : "http://localhost:9090/account?page=0&size=5" - } - }, - "page" : { - "number" : 0, - "size" : 5, - "totalElements" : 4, - "totalPages" : 1 - } -} -~~~ - -For a single account, specify the account number in the endpoint. For example, to see information about the accounts `1` and `2`: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ curl -X GET http://localhost:9090/account/1 | json_pp -~~~ - -~~~ -{ - "_links" : { - "self" : { - "href" : "http://localhost:9090/account/1" - } - }, - "balance" : 500, - "name" : "Alice", - "type" : "asset" -} -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ curl -X GET http://localhost:9090/account/2 | json_pp -~~~ - -~~~ -{ - "_links" : { - "self" : { - "href" : "http://localhost:9090/account/2" - } - }, - "balance" : 500, - "name" : "Bob", - "type" : "expense" -} -~~~ - -The `http://localhost:9090/transfer` endpoint performs transfers between accounts. `POST` requests to this endpoint are executed as writes (i.e., [`INSERT`s](insert.html) and [`UPDATE`s](update.html)) to the database. - -#### Writes - -To make a transfer, send a `POST` request to the `transfer` endpoint, using the arguments specified in the `"href`" URL (i.e., `http://localhost:9090/transfer%7B?fromId,toId,amount`). - -{% include_cached copy-clipboard.html %} -~~~ shell -$ curl -X POST -d fromId=2 -d toId=1 -d amount=150 http://localhost:9090/transfer -~~~ - -You can use the `accounts` endpoint to verify that the transfer was successfully completed: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ curl -X GET http://localhost:9090/account/1 | json_pp -~~~ - -~~~ -{ - "_links" : { - "self" : { - "href" : "http://localhost:9090/account/1" - } - }, - "balance" : 350, - "name" : "Alice", - "type" : "asset" -} -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ curl -X GET http://localhost:9090/account/2 | json_pp -~~~ - -~~~ -{ - "_links" : { - "self" : { - "href" : "http://localhost:9090/account/2" - } - }, - "balance" : 650, - "name" : "Bob", - "type" : "expense" -} -~~~ - -### Monitor the application - -`http://localhost:9090/actuator` is the base URL for a number of [Spring Boot Actuator](https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#production-ready) endpoints that let you monitor the activity and health of the application. - -{% include_cached copy-clipboard.html %} -~~~ shell -$ curl -X GET http://localhost:9090/actuator | json_pp -~~~ - -~~~ -{ - "_links" : { - "conditions" : { - "href" : "http://localhost:9090/actuator/conditions", - "templated" : false - }, - "configprops" : { - "href" : "http://localhost:9090/actuator/configprops", - "templated" : false - }, - "env" : { - "href" : "http://localhost:9090/actuator/env", - "templated" : false - }, - "env-toMatch" : { - "href" : "http://localhost:9090/actuator/env/{toMatch}", - "templated" : true - }, - "health" : { - "href" : "http://localhost:9090/actuator/health", - "templated" : false - }, - "health-path" : { - "href" : "http://localhost:9090/actuator/health/{*path}", - "templated" : true - }, - "info" : { - "href" : "http://localhost:9090/actuator/info", - "templated" : false - }, - "liquibase" : { - "href" : "http://localhost:9090/actuator/liquibase", - "templated" : false - }, - "metrics" : { - "href" : "http://localhost:9090/actuator/metrics", - "templated" : false - }, - "metrics-requiredMetricName" : { - "href" : "http://localhost:9090/actuator/metrics/{requiredMetricName}", - "templated" : true - }, - "self" : { - "href" : "http://localhost:9090/actuator", - "templated" : false - }, - "threaddump" : { - "href" : "http://localhost:9090/actuator/threaddump", - "templated" : false - } - } -} -~~~ - -Each actuator endpoint shows specific metrics on the application. For example: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ curl -X GET http://localhost:9090/actuator/health | json_pp -~~~ - -~~~ -{ - "components" : { - "db" : { - "details" : { - "database" : "PostgreSQL", - "result" : 1, - "validationQuery" : "SELECT 1" - }, - "status" : "UP" - }, - "diskSpace" : { - "details" : { - "free" : 125039620096, - "threshold" : 10485760, - "total" : 250685575168 - }, - "status" : "UP" - }, - "ping" : { - "status" : "UP" - } - }, - "status" : "UP" -} -~~~ - -For more information about actuator endpoints, see the [Spring Boot Actuator Endpoint documentation](https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#production-ready-endpoints). - -## Implementation details - -This section guides you through the different components of the application project in detail. - -### Main application process - -`JdbcApplication.java` defines the application's main process. It starts a Spring Boot web application, and then submits requests to the app's REST API that result in database transactions on the CockroachDB cluster. - -Here are the contents of [`JdbcApplication.java`](https://github.com/cockroachlabs/roach-data/blob/master/roach-data-jdbc/src/main/java/io/roach/data/jdbc/JdbcApplication.java): - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/cockroachlabs/roach-data/master/roach-data-jdbc/src/main/java/io/roach/data/jdbc/JdbcApplication.java %} -~~~ - -The annotations listed at the top of the `JdbcApplication` class definition declare some important configuration properties for the entire application: - -- [`@EnableHypermediaSupport`](https://docs.spring.io/spring-hateoas/docs/current/api/org/springframework/hateoas/config/EnableHypermediaSupport.html) enables [hypermedia support for resource representation](https://en.wikipedia.org/wiki/HATEOAS) in the application. Currently, the only hypermedia format supported by Spring is [HAL](https://en.wikipedia.org/wiki/Hypertext_Application_Language), and so the `type = EnableHypermediaSupport.HypermediaType.HAL`. For details, see [Hypermedia representation](#hypermedia-representation). -- [`@EnableJdbcRepositories`](https://docs.spring.io/spring-data/jdbc/docs/current/api/org/springframework/data/jdbc/repository/config/EnableJdbcRepositories.html) enables the creation of [Spring repositories](https://docs.spring.io/spring-data/jdbc/docs/current/reference/html/#jdbc.repositories) for data access using [Spring Data JDBC](https://spring.io/projects/spring-data-jdbc). For details, see [Spring repositories](#spring-repositories). -- [`@EnableAspectJAutoProxy`](https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/context/annotation/EnableAspectJAutoProxy.html) enables the use of [`@AspectJ` annotations for declaring aspects](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-ataspectj). For details, see [Transaction management](#transaction-management). -- [`@EnableTransactionManagement`](https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/transaction/annotation/EnableTransactionManagement.html) enables [declarative transaction management](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#transaction-declarative) in the application. For details, see [Transaction management](#transaction-management). - - Note that the `@EnableTransactionManagement` annotation is passed an `order` parameter, which indicates the ordering of advice evaluation when a common join point is reached. For details, see [Ordering advice](#ordering-advice). -- [`@SpringBootApplication`](https://docs.spring.io/spring-boot/docs/current/api/org/springframework/boot/autoconfigure/SpringBootApplication.html) is a standard configuration annotation used by Spring Boot applications. For details, see [Using the @SpringBootApplication](https://docs.spring.io/spring-boot/docs/current/reference/html/using-spring-boot.html#using-boot-using-springbootapplication-annotation) on the Spring Boot documentation site. - -### Schema management - -To create and initialize the database schema, the application uses [Liquibase](https://www.liquibase.org/). - -#### Liquibase changelogs - -Liquibase uses [changelog files](https://docs.liquibase.com/concepts/basic/changelog.html) to manage database schema changes. Changelog files include a list of instructions, known as [changesets](https://docs.liquibase.com/concepts/basic/changeset.html), that are executed against the database in a specified order. - -`resources/db/changelog-master.xml` defines the changelog for this application: - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/cockroachlabs/roach-data/master/roach-data-jdbc/src/main/resources/db/changelog-master.xml %} -~~~ - -The first changeset uses [the `sqlFile` tag](https://docs.liquibase.com/change-types/community/sql-file.html), which tells Liquibase that an external `.sql` file contains some SQL statements to execute. The file specified by the changeset, `resources/db/create.sql`, creates the `account` table: - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/cockroachlabs/roach-data/master/roach-data-jdbc/src/main/resources/db/create.sql %} -~~~ - -The second changeset in the changelog uses the [Liquibase XML syntax](https://docs.liquibase.com/concepts/basic/xml-format.html) to specify a series of sequential `INSERT` statements that initialize the `account` table with some values. - -When the application is started, all of the queries specified by the changesets are executed in the order specified by their `changeset` tag's `id` value. At application startup, Liquibase also creates a table called [`databasechangelog`](https://docs.liquibase.com/concepts/databasechangelog-table.html) in the database where it performs changes. This table's rows log all completed changesets. - -To see the completed changesets after starting the application, open a new terminal, start the [built-in SQL shell](cockroach-sql.html), and query the `databasechangelog` table: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --certs-dir=certs -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM roach_data.databasechangelog; -~~~ - -~~~ - id | author | filename | dateexecuted | orderexecuted | exectype | md5sum | description | comments | tag | liquibase | contexts | labels | deployment_id ------+--------+-----------------------------------+----------------------------------+---------------+----------+------------------------------------+--------------------------------------------------------------------------------------------------------+----------+------+-----------+----------+--------+---------------- - 1 | root | classpath:db/changelog-master.xml | 2020-06-17 14:57:01.431506+00:00 | 1 | EXECUTED | 8:939a1a8c47676119a94d0173802d207e | sqlFile | | NULL | 3.8.9 | crdb | NULL | 2420221402 - 2 | root | classpath:db/changelog-master.xml | 2020-06-17 14:57:01.470847+00:00 | 2 | EXECUTED | 8:c2945f2a445cf60b4b203e1a91d14a89 | insert tableName=account; insert tableName=account; insert tableName=account; insert tableName=account | | NULL | 3.8.9 | crdb | NULL | 2420221402 -(2 rows) -~~~ - -{{site.data.alerts.callout_info}} -Liquibase does not [retry transactions](transactions.html#transaction-retries) automatically. If a changeset fails at startup, you might need to restart the application manually to complete the changeset. -{{site.data.alerts.end}} - -#### Liquibase configuration - -Typically, Liquibase properties are defined in a separate [`liquibase.properties`](https://docs.liquibase.com/workflows/liquibase-community/creating-config-properties.html) file. In this application, the [Spring properties](https://docs.spring.io/spring-boot/docs/current/reference/html/appendix-application-properties.html) file, `application.yml`, includes properties that enable and configure Liquibase: - -~~~ yml -... - liquibase: - change-log: classpath:db/changelog-master.xml - default-schema: - drop-first: false - contexts: crdb - enabled: true -... -~~~ - -The `contexts` property specifies a single [Liquibase context](https://docs.liquibase.com/concepts/advanced/contexts.html) (`crdb`). In order for a changeset to run, its `context` attribute must match a context set by this property. The `context` value is `crdb` in both of the changeset definitions in `changelog-master.xml`, so both changesets run at application startup. - -For simplicity, `application.yml` only specifies properties for a single [Spring profile](https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-profiles), with a single set of Liquibase properties. If you want the changelog to include changesets that only run in specific environments (e.g., for debugging and development), you can create a new Spring profile in a separate properties file (e.g., `application-dev.yml`), and specify a different set of Liquibase properties for that profile. The profile set by the application configuration will automatically use the properties in that profile's properties file. For information about setting profiles, see the [Spring documentation website](https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-profiles). - -### Domain entities - -`Account.java` defines the [domain entity](https://en.wikipedia.org/wiki/Domain-driven_design#Building_blocks) for the `accounts` table. This class is used throughout the application to represent a row of data in the `accounts` table. - -Here are the contents of [`Account.java`](https://github.com/cockroachlabs/roach-data/blob/master/roach-data-jdbc/src/main/java/io/roach/data/jdbc/Account.java): - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/cockroachlabs/roach-data/master/roach-data-jdbc/src/main/java/io/roach/data/jdbc/Account.java %} -~~~ - -### Hypermedia representation - -To represent database objects as [HAL+JSON](https://en.wikipedia.org/wiki/Hypertext_Application_Language) for the REST API, the application extends the Spring HATEOAS module's [RepresentationModel](https://docs.spring.io/spring-hateoas/docs/current/reference/html/#fundamentals.representation-models) class with `AccountModel`. Like the `Account` class, its attributes represent a row of data in the `accounts` table. - -The contents of [`AccountModel.java`](https://github.com/cockroachlabs/roach-data/blob/master/roach-data-jdbc/src/main/java/io/roach/data/jdbc/AccountModel.java): - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/cockroachlabs/roach-data/master/roach-data-jdbc/src/main/java/io/roach/data/jdbc/AccountModel.java %} -~~~ - -We do not go into much detail about hypermedia representation in this tutorial. For more information, see the [Spring HATEOAS Reference Documentation](https://docs.spring.io/spring-hateoas/docs/current/reference/html/). - -### Spring repositories - -To abstract the database layer, Spring applications use the [`Repository` interface](https://docs.spring.io/spring-data/jdbc/docs/current/reference/html/#repositories), or some subinterface of `Repository`. This interface maps to a database object, like a table, and its methods map to queries against that object, like a [`SELECT`](selection-queries.html) or an [`INSERT`](insert.html) statement against a table. - -[`AccountRepository.java`](https://github.com/cockroachlabs/roach-data/blob/master/roach-data-jdbc/src/main/java/io/roach/data/jdbc/AccountRepository.java) defines the main repository for the `accounts` table: - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/cockroachlabs/roach-data/master/roach-data-jdbc/src/main/java/io/roach/data/jdbc/AccountRepository.java %} -~~~ - -`AccountRepository` extends a subinterface of `Repository` that is provided by Spring for generic [CRUD operations](https://en.wikipedia.org/wiki/Create,_read,_update_and_delete) called `CrudRepository`. To support [pagination queries](pagination.html), repositories in other Spring Data modules, like those in Spring Data JPA, usually extend a subinterface of `CrudRepository`, called `PagingAndSortingRepository`, that includes pagination and sorting methods. At the time this sample application was created, Spring Data JDBC did not support pagination. As a result, `AccountRepository` extends a custom repository, called `PagedAccountRepository`, to provide basic [`LIMIT`/`OFFSET` pagination](pagination.html) on queries against the `accounts` table. The `AccountRepository` methods use the [`@Query`](https://docs.spring.io/spring-data/jdbc/docs/current/reference/html/#jdbc.query-methods.at-query) annotation strategy to define queries manually, as strings. - -Note that, in addition to having the `@Repository` annotation, the `AccountRepository` interface has a [`@Transactional` annotation](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#transaction-declarative-annotations). When [transaction management](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#transaction-declarative) is enabled in an application (i.e., with [`@EnableTransactionManagement`](https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/transaction/annotation/EnableTransactionManagement.html)), Spring automatically wraps all objects with the `@Transactional` annotation in [a proxy](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-understanding-aop-proxies) that handles calls to the object. For more information, see [Understanding the Spring Framework’s Declarative Transaction Implementation](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#tx-decl-explained) on Spring's documentation website. - -`@Transactional` takes a number of parameters, including a `propagation` parameter that determines the transaction propagation behavior around an object (i.e., at what point in the stack a transaction starts and ends). This sample application follows the [entity-control-boundary (ECB) pattern](https://en.wikipedia.org/wiki/Entity-control-boundary). As such, the [REST service boundaries](#rest-controller) should determine where a [transaction](transactions.html) starts and ends rather than the query methods defined in the data access layer. To follow the ECB design pattern, `propagation=MANDATORY` for `AccountRepository`, which means that a transaction must already exist in order to call the `AccountRepository` query methods. In contrast, the `@Transactional` annotations on the [Rest controller entities](#rest-controller) in the web layer have `propagation=REQUIRES_NEW`, meaning that a new transaction must be created for each REST request. - -The aspects declared in `TransactionHintsAspect.java` and `RetryableTransactionAspect.java` further control how `@Transactional`-annotated components are handled. For more details on control flow and transaction management in the application, see [Transaction management](#transaction-management). - -### REST controller - -There are several endpoints exposed by the application's web layer, some of which monitor the health of the application, and some that map to queries executed against the connected database. All of the endpoints served by the application are handled by the `AccountController` class, which is defined in [`AccountController.java`](https://github.com/cockroachlabs/roach-data/blob/master/roach-data-jdbc/src/main/java/io/roach/data/jdbc/AccountController.java): - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/cockroachlabs/roach-data/master/roach-data-jdbc/src/main/java/io/roach/data/jdbc/AccountController.java %} -~~~ - - Annotated with [`@RestController`](https://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/web/bind/annotation/RestController.html), `AccountController` defines the primary [web controller](https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller) component of the application. The `AccountController` methods define the endpoints, routes, and business logic of REST services for account querying and money transferring. Its attributes include an instantiation of [`AccountRepository`](#spring-repositories), called `accountRepository`, that establishes an interface to the `accounts` table through the data access layer. - -As mentioned in the [Spring repositories](#spring-repositories) section, the application's transaction boundaries follow the [entity-control-boundary (ECB) pattern](https://en.wikipedia.org/wiki/Entity-control-boundary), meaning that the web service boundaries of the application determine where a [transaction](transactions.html) starts and ends. To follow the ECB pattern, the `@Transactional` annotation on each of the HTTP entities (`listAccounts()`, `getAccount()`, and `transfer()`) has `propagation=REQUIRES_NEW`. This ensures that each time a REST request is made to an endpoint, a new transaction context is created. For details on how aspects handle control flow and transaction management in the application, see [Transaction management](#transaction-management). - -### Transaction management - -When [transaction management](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#transaction-declarative) is enabled in an application, Spring automatically wraps all objects annotated with `@Transactional` in [a proxy](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-understanding-aop-proxies) that handles calls to the object. By default, this proxy starts and closes transactions according to the configured transaction management behavior (e.g., the `propagation` level). - -Using [@AspectJ annotations](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#transaction-declarative-aspectj), this sample application extends the default transaction proxy behavior with two other explicitly-defined [aspects](https://en.wikipedia.org/wiki/Aspect_(computer_programming)): `TransactionHintsAspect` and `RetryableTransactionAspect`. Methods of these aspects are declared as [advice](https://en.wikipedia.org/wiki/Advice_(programming)) to be executed around method calls annotated with `@Transactional`. - -For more information about transaction management in the app, see the following sections below: - -- [Ordering advice](#ordering-advice) -- [Transaction attributes](#transaction-attributes) -- [Transaction retries](#transaction-retries) - -#### Ordering advice - -To determine the order of evaluation when multiple transaction advisors match the same [pointcut](https://en.wikipedia.org/wiki/Pointcut) (in this case, around `@Transactional` method calls), this application explicitly declares an order of precedence for calling advice. - -At the top level of the application, in the main `JdbcApplication.java` file, the [`@EnableTransactionManagement`](https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/transaction/annotation/EnableTransactionManagement.html) annotation is passed an `order` parameter. This parameter sets the order on the primary transaction advisor to one level of precedence above the lowest level, `Ordered.LOWEST_PRECEDENCE`. This means that the advisor with the lowest level of precedence is evaluated after the primary transaction advisor (i.e., within the context of an open transaction). - -For the two explicitly-defined aspects, `TransactionHintsAspect` and `RetryableTransactionAspect`, the [`@Order`](https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/core/annotation/Order.html) annotation is used. Like the `order` parameter on the `@EnableTransactionManagement` annotation, `@Order` takes a value that indicates the precedence of advice. The advisor with the lowest level of precedence is declared in `TransactionHintsAspect`, the aspect that defines the [transaction attributes](#transaction-attributes). `RetryableTransactionAspect`, the aspect that defines the [transaction retry logic](#transaction-retries), defines the advisor with the highest level of precedence. - -For more details about advice ordering, see [Advice Ordering](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-ataspectj-advice-ordering) on the Spring documentation site. - - -#### Transaction attributes - -The `TransactionHintsAspect` class, declared as an aspect with the [`@Aspect` annotation](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-at-aspectj), declares an advice method that defines the attributes of a transaction. The `@Order` annotation is passed the lowest level of precedence, `Ordered.LOWEST_PRECEDENCE`, indicating that this advisor must run after the main transaction advisor, within the context of a transaction. Here are the contents of [`TransactionHintsAspect.java`](https://github.com/cockroachlabs/roach-data/blob/master/roach-data-jdbc/src/main/java/io/roach/data/jdbc/TransactionHintsAspect.java): - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/cockroachlabs/roach-data/master/roach-data-jdbc/src/main/java/io/roach/data/jdbc/TransactionHintsAspect.java %} -~~~ - -The `anyTransactionBoundaryOperation` method is declared as a pointcut with the [`@Pointcut` annotation](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-pointcuts). In Spring, pointcut declarations must include an expression to determine where [join points](https://en.wikipedia.org/wiki/Join_point) occur in the application control flow. To help define these expressions, Spring supports a set of [designators](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-pointcuts-designators). The application uses two of them here: `execution`, which matches method execution joint points (i.e., defines a joint point when a specific method is executed, in this case, *any* method in the `io.roach.` namespace), and `@annotation`, which limits the matches to methods with a specific annotation, in this case `@Transactional`. - -`setTransactionAttributes` sets the transaction attributes in the form of advice. Spring supports [several different annotations to declare advice](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-advice). The [`@Around` annotation](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-ataspectj-around-advice) allows an advice method to work before and after the `anyTransactionBoundaryOperation(transactional)` join point. It also allows the advice method to call the next matching advisor with the `ProceedingJoinPoint.proceed();` method. - -On verifying that the transaction is active (using `TransactionSynchronizationManager.isActualTransactionActive()`), the advice [sets some session variables](set-vars.html) using methods of the [`JdbcTemplate`](https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/jdbc/core/JdbcTemplate.html) object declared at the top of the `TransactionHintsAspect` class definition. These session variables (`application_name`, `statement_timeout`, and `transaction_read_only`) set [the application name](connection-parameters.html#additional-connection-parameters) for the query to "`roach-data`", the time allowed for the statement to execute before timing out to 1000 milliseconds (i.e., 1 second), and the [transaction access mode](set-transaction.html#parameters) as either `READ ONLY` or `READ WRITE`. - -#### Transaction retries - -Transactions may require retries if they experience deadlock or [transaction contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention) that cannot be resolved without allowing [serialization](demo-serializable.html) anomalies. To handle transactions that are aborted due to transient serialization errors, we highly recommend writing [client-side transaction retry logic](transactions.html#client-side-intervention) into applications written on CockroachDB. - -In this application, transaction retry logic is written into the methods of the `RetryableTransactionAspect` class. This class is declared an aspect with the `@Aspect` annotation. The `@Order` annotation on this aspect class is passed `Ordered.LOWEST_PRECEDENCE-2`, a level of precedence above the primary transaction advisor. This indicates that the transaction retry advisor must run outside the context of a transaction. Here are the contents of [`RetryableTransactionAspect.java`](https://github.com/cockroachlabs/roach-data/blob/master/roach-data-jdbc/src/main/java/io/roach/data/jdbc/RetryableTransactionAspect.java): - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/cockroachlabs/roach-data/master/roach-data-jdbc/src/main/java/io/roach/data/jdbc/RetryableTransactionAspect.java %} -~~~ - -The `anyTransactionBoundaryOperation` pointcut definition is identical to the one declared in `TransactionHintsAspect`. The `execution` designator matches all methods in the `io.roach.` namespace, and the `@annotation` designator limits the matches to methods with the `@Transactional` annotation. - -`retryableOperation` handles the application retry logic in the form of advice. The [`@Around` annotation](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-ataspectj-around-advice) allows the advice method to work before and after an `anyTransactionBoundaryOperation(transactional)` join point. It also allows the advice method to call the next matching advisor. - -`retryableOperation` first verifies that there is no active transaction. It then increments the retry count and attempts to proceed to the next advice method with the `ProceedingJoinPoint.proceed()` method. If the underlying methods (i.e., the primary transaction advisor's methods and the [annotated query methods](#spring-repositories)) succeed, the transaction has been successfully committed to the database. The results are then returned and the application flow continues. If a failure in the underlying layers occurs due to a transient error ([`TransientDataAccessException`](https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/dao/TransientDataAccessException.html) or [`TransactionSystemException`](https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/transaction/TransactionSystemException.html)), then the transaction is retried. The time between each retry grows with each retry until the maximum number of retries is reached. - -## See also - -Spring documentation: - -- [Spring Boot website](https://spring.io/projects/spring-boot) -- [Spring Framework Overview](https://docs.spring.io/spring/docs/current/spring-framework-reference/overview.html#overview) -- [Spring Core documentation](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#spring-core) -- [Data Access with JDBC](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#jdbc) -- [Spring Web MVC](https://docs.spring.io/spring/docs/current/spring-framework-reference/web.html#mvc) - -CockroachDB documentation: - -- [Learn CockroachDB SQL](learn-cockroachdb-sql.html) -- [Client Connection Parameters](connection-parameters.html) -- [CockroachDB Developer Guide](developer-guide-overview.html) -- [Hello World Example Apps](example-apps.html) -- [Transactions](transactions.html) diff --git a/src/current/v21.1/build-a-spring-app-with-cockroachdb-jpa.md b/src/current/v21.1/build-a-spring-app-with-cockroachdb-jpa.md deleted file mode 100644 index 5a661625dd8..00000000000 --- a/src/current/v21.1/build-a-spring-app-with-cockroachdb-jpa.md +++ /dev/null @@ -1,708 +0,0 @@ ---- -title: Build a Spring App with CockroachDB and JPA -summary: Learn how to use CockroachDB from a Spring application with Spring Data JPA and Hibernate. -toc: true -twitter: false -referral_id: docs_roach_data_java_spring_jpa ---- - -
- - -
- -This tutorial shows you how to build a [Spring Boot](https://spring.io/projects/spring-boot) web application with CockroachDB, using the [Spring Data JPA](https://spring.io/projects/spring-data-jpa) module for data access. The code for the example application is available for download from [GitHub](https://github.com/cockroachlabs/roach-data/tree/master), along with identical examples that use [JDBC](https://github.com/cockroachlabs/roach-data/tree/master/roach-data-jdbc), [jOOQ](https://github.com/cockroachlabs/roach-data/tree/master/roach-data-jooq), and [MyBatis](https://github.com/cockroachlabs/roach-data/tree/master/roach-data-mybatis) for data access. - -## Step 1. Start CockroachDB - -Choose whether to run a local cluster or a free CockroachDB cluster on CockroachDB {{ site.data.products.serverless }}. - -
- - -
- -
- -1. If you haven't already, [download the CockroachDB binary](install-cockroachdb.html). -1. [Start a local, secure cluster](secure-a-cluster.html). - -
- -
- -### Create a free cluster - -{% include cockroachcloud/quickstart/create-a-free-cluster.md %} - -### Set up your cluster connection - -{% include cockroachcloud/quickstart/set-up-your-cluster-connection.md %} - -
- -## Step 2. Create a database and a user - -
- -1. Open a SQL shell to your local cluster using the [`cockroach sql`](cockroach-sql.html) command: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --certs-dir={certs-dir} --host=localhost:{port} - ~~~ - - Where `{certs_dir}` is the full path to the `certs` directory that you created when setting up the cluster, and `{port}` is the port at which the cluster is listening for incoming connections. - -1. In the SQL shell, create the `roach_data` database that your application will use: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE DATABASE roach_data; - ~~~ - -1. Create a SQL user for your app: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE USER {username} WITH PASSWORD {password}; - ~~~ - - Take note of the username and password. You will use it to connect to the database later. - -1. Give the user the necessary permissions: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > GRANT ALL ON DATABASE roach_data TO {username}; - ~~~ - -1. Exit the shell, and generate a certificate and key for your user by running the following command: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach cert create-client {user} --certs-dir=certs --ca-key={certs-dir}/ca.key --also-generate-pkcs8-key -~~~ - -The [`--also-generate-pkcs8-key` flag](cockroach-cert.html#flag-pkcs8) generates a key in [PKCS#8 format](https://tools.ietf.org/html/rfc5208), which is the standard key encoding format in Java. In this case, the generated PKCS8 key will be named `client.{user}.key.pk8`. - -
- -
- -1. If you haven't already, [download the CockroachDB binary](install-cockroachdb.html). -1. Start the [built-in SQL shell](cockroach-sql.html) using the connection string you got from the CockroachDB {{ site.data.products.cloud }} Console [earlier](#set-up-your-cluster-connection): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql \ - --url='postgres://{username}:{password}@{global host}:26257/{cluster_name}.defaultdb?sslmode=verify-full&sslrootcert={certs_dir}/cc-ca.crt' - ~~~ - - In the connection string copied from the CockroachDB {{ site.data.products.cloud }} Console, your username, password and cluster name are pre-populated. Replace the `{certs_dir}` placeholder with the path to the `certs` directory that you created [earlier](#set-up-your-cluster-connection). - - {% include cockroachcloud/cc-no-user-certs.md %} - -1. In the SQL shell, create the `roach_data` database that your application will use: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE DATABASE roach_data; - ~~~ - -
- -## Step 3. Install JDK - -Download and install a Java Development Kit on your machine. Spring Boot supports Java versions 8, 11, and 14. In this tutorial, we use [JDK 8 from OpenJDK](https://openjdk.java.net/install/). - -## Step 4. Install Maven - -This example application uses [Maven](http://maven.apache.org/) to manage all application dependencies. Spring supports Maven versions 3.2 and later. - -To install Maven on macOS, run the following command: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ brew install maven -~~~ - -To install Maven on a Debian-based Linux distribution like Ubuntu: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ apt-get install maven -~~~ - -To install Maven on a Red Hat-based Linux distribution like Fedora: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ dnf install maven -~~~ - -For other ways to install Maven, see [its official documentation](http://maven.apache.org/install.html). - -## Step 5. Get the application code - -To get the application code, download or clone the [`roach-data` repository](https://github.com/cockroachlabs/roach-data). The code for the example JPA application is located under the `roach-data-jpa` directory. - -(*Optional*) To recreate the application project structure with the same dependencies as those used by this sample application, you can use [Spring initializr](https://start.spring.io/) with the following settings: - -**Project** - -- Maven Project - -**Language** - -- Java - -**Spring Boot** - -- 2.2.6 - -**Project Metadata** - -- Group: io.roach -- Artifact: data -- Name: data -- Package name: io.roach.data -- Packaging: Jar -- Java: 8 - -**Dependencies** - -- Spring Web -- Spring Data JPA -- Spring HATEOS -- Liquibase Migration -- PostgreSQL Driver - -The [Hibernate CockroachDB dialect](https://in.relation.to/2020/07/27/hibernate-orm-5419-final-release/) is supported in Hibernate v5.4.19+. At the time of writing this tutorial, Spring Data JPA used Hibernate v5.4.15 as its default JPA provider. To specify a different version of Hibernate than the default, add an additional entry to your application's `pom.xml` file, as shown in [the `roach-data` GitHub repo](https://github.com/cockroachlabs/roach-data/blob/master/roach-data-jpa/pom.xml): - -~~~ xml - - org.hibernate - hibernate-core - 5.4.19.Final - -~~~ - -## Step 6. Run the application - -Compiling and running the application code will start a web application, initialize the `accounts` table in the `roach_data` database, and submit some requests to the app's REST API that result in [atomic database transactions](transactions.html) on the running CockroachDB cluster. For details about the application code, see [Implementation details](#implementation-details). - -Open the `roach-data/roach-data-jpa/src/main/resources/application.yml` file and edit the `datasource` settings to connect to your running database cluster: - -
- -~~~ yml -... -datasource: - url: jdbc:postgresql://localhost:{port}/roach_data?ssl=true&sslmode=require&sslrootcert={certs-dir}/ca.crt&sslkey={certs-dir}/client.{username}.key.pk8&sslcert={certs-dir}/client.{username}.crt - username: {username} - password: {password} - driver-class-name: org.postgresql.Driver -... -~~~ - -Where: - -- `{port}` is the port number. -- `{certs-dir}` is the full path to the certificates directory containing the authentication certificates that you created earlier. -- `{username}` and `{password}` specify the SQL username and password that you created earlier. - -
- -
- -~~~ yml -... -datasource: - url: jdbc:postgresql://{globalhost}:{port}/{cluster_name}.roach_data?sslmode=verify-full&sslrootcert={path to the CA certificate}/cc-ca.crt - username: {username} - password: {password} - driver-class-name: org.postgresql.Driver -... -~~~ - -{% include {{page.version.version}}/app/cc-free-tier-params.md %} - -
- -Open a terminal, and navigate to the `roach-data-jpa` project subfolder: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cd /roach-data/roach-data-jpa -~~~ - -Use Maven to download the application dependencies and compile the code: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ mvn clean install -~~~ - -From the `roach-data-jpa` directory, run the application JAR file: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ java -jar target/roach-data-jpa.jar -~~~ - -The output should look like the following: - -~~~ - . ____ _ __ _ _ - /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ -( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ - \\/ ___)| |_)| | | | | || (_| | ) ) ) ) - ' |____| .__|_| |_|_| |_\__, | / / / / - =========|_|==============|___/=/_/_/_/ - :: Spring Boot :: (v2.2.7.RELEASE) - -2020-06-22 11:54:46.243 INFO 81343 --- [ main] io.roach.data.jpa.JpaApplication : Starting JpaApplication v1.0.0.BUILD-SNAPSHOT on MyComputer.local with PID 81343 (path/code/roach-data/roach-data-jpa/target/roach-data-jpa.jar started by user in path/code/roach-data/roach-data-jpa) -2020-06-22 11:54:46.246 INFO 81343 --- [ main] io.roach.data.jpa.JpaApplication : No active profile set, falling back to default profiles: default -2020-06-22 11:54:46.929 INFO 81343 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode! -2020-06-22 11:54:46.930 INFO 81343 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. -2020-06-22 11:54:47.023 INFO 81343 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 80ms. Found 1 JPA repository interfaces. -2020-06-22 11:54:47.211 INFO 81343 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode! -2020-06-22 11:54:47.211 INFO 81343 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JDBC repositories in DEFAULT mode. -2020-06-22 11:54:47.224 INFO 81343 --- [ main] .RepositoryConfigurationExtensionSupport : Spring Data JDBC - Could not safely identify store assignment for repository candidate interface io.roach.data.jpa.AccountRepository. If you want this repository to be a JDBC repository, consider annotating your entities with one of these annotations: org.springframework.data.relational.core.mapping.Table. -2020-06-22 11:54:47.224 INFO 81343 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 12ms. Found 0 JDBC repository interfaces. -2020-06-22 11:54:47.913 INFO 81343 --- [ main] org.eclipse.jetty.util.log : Logging initialized @2990ms to org.eclipse.jetty.util.log.Slf4jLog -2020-06-22 11:54:47.982 INFO 81343 --- [ main] o.s.b.w.e.j.JettyServletWebServerFactory : Server initialized with port: 9090 -2020-06-22 11:54:47.985 INFO 81343 --- [ main] org.eclipse.jetty.server.Server : jetty-9.4.28.v20200408; built: 2020-04-08T17:49:39.557Z; git: ab228fde9e55e9164c738d7fa121f8ac5acd51c9; jvm 11.0.7+10 -2020-06-22 11:54:48.008 INFO 81343 --- [ main] o.e.j.s.h.ContextHandler.application : Initializing Spring embedded WebApplicationContext -2020-06-22 11:54:48.008 INFO 81343 --- [ main] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 1671 ms -2020-06-22 11:54:48.123 INFO 81343 --- [ main] org.eclipse.jetty.server.session : DefaultSessionIdManager workerName=node0 -2020-06-22 11:54:48.123 INFO 81343 --- [ main] org.eclipse.jetty.server.session : No SessionScavenger set, using defaults -2020-06-22 11:54:48.124 INFO 81343 --- [ main] org.eclipse.jetty.server.session : node0 Scavenging every 660000ms -2020-06-22 11:54:48.130 INFO 81343 --- [ main] o.e.jetty.server.handler.ContextHandler : Started o.s.b.w.e.j.JettyEmbeddedWebAppContext@41394595{application,/,[file:///private/var/folders/pg/r58v54857gq_1nqm_2tr6lg40000gn/T/jetty-docbase.7785392427958606416.8080/],AVAILABLE} -2020-06-22 11:54:48.131 INFO 81343 --- [ main] org.eclipse.jetty.server.Server : Started @3207ms -2020-06-22 11:54:48.201 INFO 81343 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting... -2020-06-22 11:54:48.483 INFO 81343 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed. -2020-06-22 11:54:49.507 INFO 81343 --- [ main] liquibase.executor.jvm.JdbcExecutor : SELECT COUNT(*) FROM public.databasechangeloglock -2020-06-22 11:54:49.522 INFO 81343 --- [ main] liquibase.executor.jvm.JdbcExecutor : CREATE TABLE public.databasechangeloglock (ID INTEGER NOT NULL, LOCKED BOOLEAN NOT NULL, LOCKGRANTED TIMESTAMP WITHOUT TIME ZONE, LOCKEDBY VARCHAR(255), CONSTRAINT DATABASECHANGELOGLOCK_PKEY PRIMARY KEY (ID)) -2020-06-22 11:54:49.535 INFO 81343 --- [ main] liquibase.executor.jvm.JdbcExecutor : SELECT COUNT(*) FROM public.databasechangeloglock -2020-06-22 11:54:49.554 INFO 81343 --- [ main] liquibase.executor.jvm.JdbcExecutor : DELETE FROM public.databasechangeloglock -2020-06-22 11:54:49.555 INFO 81343 --- [ main] liquibase.executor.jvm.JdbcExecutor : INSERT INTO public.databasechangeloglock (ID, LOCKED) VALUES (1, FALSE) -2020-06-22 11:54:49.562 INFO 81343 --- [ main] liquibase.executor.jvm.JdbcExecutor : SELECT LOCKED FROM public.databasechangeloglock WHERE ID=1 -2020-06-22 11:54:49.570 INFO 81343 --- [ main] l.lockservice.StandardLockService : Successfully acquired change log lock -2020-06-22 11:54:50.519 INFO 81343 --- [ main] l.c.StandardChangeLogHistoryService : Creating database history table with name: public.databasechangelog -2020-06-22 11:54:50.520 INFO 81343 --- [ main] liquibase.executor.jvm.JdbcExecutor : CREATE TABLE public.databasechangelog (ID VARCHAR(255) NOT NULL, AUTHOR VARCHAR(255) NOT NULL, FILENAME VARCHAR(255) NOT NULL, DATEEXECUTED TIMESTAMP WITHOUT TIME ZONE NOT NULL, ORDEREXECUTED INTEGER NOT NULL, EXECTYPE VARCHAR(10) NOT NULL, MD5SUM VARCHAR(35), DESCRIPTION VARCHAR(255), COMMENTS VARCHAR(255), TAG VARCHAR(255), LIQUIBASE VARCHAR(20), CONTEXTS VARCHAR(255), LABELS VARCHAR(255), DEPLOYMENT_ID VARCHAR(10)) -2020-06-22 11:54:50.534 INFO 81343 --- [ main] liquibase.executor.jvm.JdbcExecutor : SELECT COUNT(*) FROM public.databasechangelog -2020-06-22 11:54:50.547 INFO 81343 --- [ main] l.c.StandardChangeLogHistoryService : Reading from public.databasechangelog -2020-06-22 11:54:50.548 INFO 81343 --- [ main] liquibase.executor.jvm.JdbcExecutor : SELECT * FROM public.databasechangelog ORDER BY DATEEXECUTED ASC, ORDEREXECUTED ASC -2020-06-22 11:54:50.550 INFO 81343 --- [ main] liquibase.executor.jvm.JdbcExecutor : SELECT COUNT(*) FROM public.databasechangeloglock -2020-06-22 11:54:50.566 INFO 81343 --- [ main] liquibase.executor.jvm.JdbcExecutor : create table account -( - id int not null primary key default unique_rowid(), - balance numeric(19, 2) not null, - name varchar(128) not null, - type varchar(25) not null -) -2020-06-22 11:54:50.575 INFO 81343 --- [ main] liquibase.changelog.ChangeSet : SQL in file db/create.sql executed -2020-06-22 11:54:50.581 INFO 81343 --- [ main] liquibase.changelog.ChangeSet : ChangeSet classpath:db/changelog-master.xml::1::root ran successfully in 16ms -2020-06-22 11:54:50.585 INFO 81343 --- [ main] liquibase.executor.jvm.JdbcExecutor : SELECT MAX(ORDEREXECUTED) FROM public.databasechangelog -2020-06-22 11:54:50.589 INFO 81343 --- [ main] liquibase.executor.jvm.JdbcExecutor : INSERT INTO public.databasechangelog (ID, AUTHOR, FILENAME, DATEEXECUTED, ORDEREXECUTED, MD5SUM, DESCRIPTION, COMMENTS, EXECTYPE, CONTEXTS, LABELS, LIQUIBASE, DEPLOYMENT_ID) VALUES ('1', 'root', 'classpath:db/changelog-master.xml', NOW(), 1, '8:567321cdb0100cbe76731a7ed414674b', 'sqlFile', '', 'EXECUTED', 'crdb', NULL, '3.8.9', '2852090551') -2020-06-22 11:54:50.593 INFO 81343 --- [ main] liquibase.executor.jvm.JdbcExecutor : INSERT INTO public.account (id, name, balance, type) VALUES ('1', 'Alice', 500.00, 'asset') -2020-06-22 11:54:50.601 INFO 81343 --- [ main] liquibase.changelog.ChangeSet : New row inserted into account -2020-06-22 11:54:50.602 INFO 81343 --- [ main] liquibase.executor.jvm.JdbcExecutor : INSERT INTO public.account (id, name, balance, type) VALUES ('2', 'Bob', 500.00, 'expense') -2020-06-22 11:54:50.603 INFO 81343 --- [ main] liquibase.changelog.ChangeSet : New row inserted into account -2020-06-22 11:54:50.604 INFO 81343 --- [ main] liquibase.executor.jvm.JdbcExecutor : INSERT INTO public.account (id, name, balance, type) VALUES ('3', 'Bobby Tables', 500.00, 'asset') -2020-06-22 11:54:50.605 INFO 81343 --- [ main] liquibase.changelog.ChangeSet : New row inserted into account -2020-06-22 11:54:50.605 INFO 81343 --- [ main] liquibase.executor.jvm.JdbcExecutor : INSERT INTO public.account (id, name, balance, type) VALUES ('4', 'Doris', 500.00, 'expense') -2020-06-22 11:54:50.606 INFO 81343 --- [ main] liquibase.changelog.ChangeSet : New row inserted into account -2020-06-22 11:54:50.608 INFO 81343 --- [ main] liquibase.changelog.ChangeSet : ChangeSet classpath:db/changelog-master.xml::2::root ran successfully in 16ms -2020-06-22 11:54:50.609 INFO 81343 --- [ main] liquibase.executor.jvm.JdbcExecutor : INSERT INTO public.databasechangelog (ID, AUTHOR, FILENAME, DATEEXECUTED, ORDEREXECUTED, MD5SUM, DESCRIPTION, COMMENTS, EXECTYPE, CONTEXTS, LABELS, LIQUIBASE, DEPLOYMENT_ID) VALUES ('2', 'root', 'classpath:db/changelog-master.xml', NOW(), 2, '8:c2945f2a445cf60b4b203e1a91d14a89', 'insert tableName=account; insert tableName=account; insert tableName=account; insert tableName=account', '', 'EXECUTED', 'crdb', NULL, '3.8.9', '2852090551') -2020-06-22 11:54:50.615 INFO 81343 --- [ main] l.lockservice.StandardLockService : Successfully released change log lock -2020-06-22 11:54:50.727 INFO 81343 --- [ main] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default] -2020-06-22 11:54:50.817 INFO 81343 --- [ main] org.hibernate.Version : HHH000412: Hibernate ORM core version 5.4.19.Final -2020-06-22 11:54:50.993 INFO 81343 --- [ main] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.0.Final} -2020-06-22 11:54:51.154 INFO 81343 --- [ main] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.CockroachDB201Dialect -2020-06-22 11:54:51.875 INFO 81343 --- [ main] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform] -2020-06-22 11:54:51.886 INFO 81343 --- [ main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default' -2020-06-22 11:54:52.700 INFO 81343 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor' -2020-06-22 11:54:52.958 INFO 81343 --- [ main] o.e.j.s.h.ContextHandler.application : Initializing Spring DispatcherServlet 'dispatcherServlet' -2020-06-22 11:54:52.958 INFO 81343 --- [ main] o.s.web.servlet.DispatcherServlet : Initializing Servlet 'dispatcherServlet' -2020-06-22 11:54:52.966 INFO 81343 --- [ main] o.s.web.servlet.DispatcherServlet : Completed initialization in 8 ms -2020-06-22 11:54:52.997 INFO 81343 --- [ main] o.e.jetty.server.AbstractConnector : Started ServerConnector@1568159{HTTP/1.1, (http/1.1)}{0.0.0.0:9090} -2020-06-22 11:54:52.999 INFO 81343 --- [ main] o.s.b.web.embedded.jetty.JettyWebServer : Jetty started on port(s) 9090 (http/1.1) with context path '/' -2020-06-22 11:54:53.001 INFO 81343 --- [ main] io.roach.data.jpa.JpaApplication : Started JpaApplication in 7.518 seconds (JVM running for 8.077) -2020-06-22 11:54:53.002 INFO 81343 --- [ main] io.roach.data.jpa.JpaApplication : Lets move some $$ around! -2020-06-22 11:54:54.399 INFO 81343 --- [ main] io.roach.data.jpa.JpaApplication : Worker finished - 7 remaining -2020-06-22 11:54:54.447 INFO 81343 --- [ main] io.roach.data.jpa.JpaApplication : Worker finished - 6 remaining -2020-06-22 11:54:54.447 INFO 81343 --- [ main] io.roach.data.jpa.JpaApplication : Worker finished - 5 remaining -2020-06-22 11:54:54.447 INFO 81343 --- [ main] io.roach.data.jpa.JpaApplication : Worker finished - 4 remaining -2020-06-22 11:54:54.447 INFO 81343 --- [ main] io.roach.data.jpa.JpaApplication : Worker finished - 3 remaining -2020-06-22 11:54:54.447 INFO 81343 --- [ main] io.roach.data.jpa.JpaApplication : Worker finished - 2 remaining -2020-06-22 11:54:54.447 INFO 81343 --- [ main] io.roach.data.jpa.JpaApplication : Worker finished - 1 remaining -2020-06-22 11:54:54.447 INFO 81343 --- [ main] io.roach.data.jpa.JpaApplication : Worker finished - 0 remaining -2020-06-22 11:54:54.447 INFO 81343 --- [ main] io.roach.data.jpa.JpaApplication : All client workers finished but server keeps running. Have a nice day! -~~~ - -As the output states, the application configures a database connection, starts a web servlet listening on the address `http://localhost:9090/`, initializes the `account` table and changelog tables with [Liquibase](https://www.liquibase.org/), and then runs some test operations as requests to the application's REST API. - -For more details about the application code, see [Implementation details](#implementation-details). - -### Query the database - -#### Reads - -The `http://localhost:9090/account` endpoint returns information about all accounts in the database. `GET` requests to these endpoints are executed on the database as `SELECT` statements. - -The following `curl` command sends a `GET` request to the endpoint. The `json_pp` command formats the JSON response. - -{% include_cached copy-clipboard.html %} -~~~ shell -$ curl -X GET http://localhost:9090/account | json_pp -~~~ - -~~~ -{ - "_embedded" : { - "accounts" : [ - { - "_links" : { - "self" : { - "href" : "http://localhost:9090/account/1" - } - }, - "balance" : 500, - "name" : "Alice", - "type" : "asset" - }, - { - "_links" : { - "self" : { - "href" : "http://localhost:9090/account/2" - } - }, - "balance" : 500, - "name" : "Bob", - "type" : "expense" - }, - { - "_links" : { - "self" : { - "href" : "http://localhost:9090/account/3" - } - }, - "balance" : 500, - "name" : "Bobby Tables", - "type" : "asset" - }, - { - "_links" : { - "self" : { - "href" : "http://localhost:9090/account/4" - } - }, - "balance" : 500, - "name" : "Doris", - "type" : "expense" - } - ] - }, - "_links" : { - "self" : { - "href" : "http://localhost:9090/account?page=0&size=5" - } - }, - "page" : { - "number" : 0, - "size" : 5, - "totalElements" : 4, - "totalPages" : 1 - } -} -~~~ - -For a single account, specify the account number in the endpoint. For example, to see information about the accounts `1` and `2`: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ curl -X GET http://localhost:9090/account/1 | json_pp -~~~ - -~~~ -{ - "_links" : { - "self" : { - "href" : "http://localhost:9090/account/1" - } - }, - "balance" : 500, - "name" : "Alice", - "type" : "asset" -} -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ curl -X GET http://localhost:9090/account/2 | json_pp -~~~ - -~~~ -{ - "_links" : { - "self" : { - "href" : "http://localhost:9090/account/2" - } - }, - "balance" : 500, - "name" : "Bob", - "type" : "expense" -} -~~~ - -The `http://localhost:9090/transfer` endpoint performs transfers between accounts. `POST` requests to this endpoint are executed as writes (i.e., [`INSERT`s](insert.html) and [`UPDATE`s](update.html)) to the database. - -#### Writes - -To make a transfer, send a `POST` request to the `transfer` endpoint, using the arguments specified in the `"href`" URL (i.e., `http://localhost:9090/transfer%7B?fromId,toId,amount`). - -{% include_cached copy-clipboard.html %} -~~~ shell -$ curl -X POST -d fromId=2 -d toId=1 -d amount=150 http://localhost:9090/transfer -~~~ - -You can use the `accounts` endpoint to verify that the transfer was successfully completed: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ curl -X GET http://localhost:9090/account/1 | json_pp -~~~ - -~~~ -{ - "_links" : { - "self" : { - "href" : "http://localhost:9090/account/1" - } - }, - "balance" : 650, - "name" : "Alice", - "type" : "asset" -} -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ curl -X GET http://localhost:9090/account/2 | json_pp -~~~ - -~~~ -{ - "_links" : { - "self" : { - "href" : "http://localhost:9090/account/2" - } - }, - "balance" : 350, - "name" : "Bob", - "type" : "expense" -} -~~~ - -## Implementation details - -This section guides you through the different components of the application project in detail. - -### Main application process - -`JpaApplication.java` defines the application's main process. It starts a Spring Boot web application, and then submits requests to the app's REST API that result in database transactions on the CockroachDB cluster. - -Here are the contents of [`JpaApplication.java`](https://github.com/cockroachlabs/roach-data/blob/master/roach-data-jpa/src/main/java/io/roach/data/jpa/JpaApplication.java): - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/cockroachlabs/roach-data/master/roach-data-jpa/src/main/java/io/roach/data/jpa/JpaApplication.java %} -~~~ - -The annotations listed at the top of the `JpaApplication` class definition declare some important configuration properties for the entire application: - -- [`@EnableHypermediaSupport`](https://docs.spring.io/spring-hateoas/docs/current/api/org/springframework/hateoas/config/EnableHypermediaSupport.html) enables [hypermedia support for resource representation](https://en.wikipedia.org/wiki/HATEOAS) in the application. Currently, the only hypermedia format supported by Spring is [HAL](https://en.wikipedia.org/wiki/Hypertext_Application_Language), and so the `type = EnableHypermediaSupport.HypermediaType.HAL`. For details, see [Hypermedia representation](#hypermedia-representation). -- [`@EnableJpaRepositories`](https://docs.spring.io/spring-data/data-jpa/docs/current/api/org/springframework/data/jpa/repository/config/EnableJpaRepositories.html) enables the creation of [Spring repositories](https://docs.spring.io/spring-data/jpa/docs/current/reference/html/#repositories) for data access using [Spring Data JPA](https://spring.io/projects/spring-data-jpa). For details, see [Spring repositories](#spring-repositories). -- [`@EnableAspectJAutoProxy`](https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/context/annotation/EnableAspectJAutoProxy.html) enables the use of [`@AspectJ` annotations for declaring aspects](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-ataspectj). For details, see [Transaction management](#transaction-management). -- [`@EnableTransactionManagement`](https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/transaction/annotation/EnableTransactionManagement.html) enables [declarative transaction management](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#transaction-declarative) in the application. For details, see [Transaction management](#transaction-management). - - Note that the `@EnableTransactionManagement` annotation is passed an `order` parameter, which indicates the ordering of advice evaluation when a common join point is reached. For details, see [Ordering advice](#ordering-advice). -- [`@SpringBootApplication`](https://docs.spring.io/spring-boot/docs/current/api/org/springframework/boot/autoconfigure/SpringBootApplication.html) is a standard configuration annotation used by Spring Boot applications. For details, see [Using the @SpringBootApplication](https://docs.spring.io/spring-boot/docs/current/reference/html/using-spring-boot.html#using-boot-using-springbootapplication-annotation) on the Spring Boot documentation site. - -### Schema management - -To create and initialize the database schema, the application uses [Liquibase](https://www.liquibase.org/). - -Liquibase uses files called [changelogs](https://docs.liquibase.com/concepts/basic/changelog.html) to manage the changes to the database. Changelog files include a list of instructions, known as [changesets](https://docs.liquibase.com/concepts/basic/changeset.html), that are executed against the database in a specified order. - -`resources/db/changelog-master.xml` defines the changelog for this application: - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/cockroachlabs/roach-data/master/roach-data-jpa/src/main/resources/db/changelog-master.xml %} -~~~ - -The first changeset uses [the `sqlFile` tag](https://docs.liquibase.com/change-types/community/sql-file.html), which tells Liquibase that an external `.sql` file contains some SQL statements to execute. The file specified by the changeset, `resources/db/create.sql`, creates the `account` table: - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/cockroachlabs/roach-data/master/roach-data-jpa/src/main/resources/db/create.sql %} -~~~ - -The second changeset in the changelog uses the [Liquibase XML syntax](https://docs.liquibase.com/concepts/basic/xml-format.html) to specify a series of sequential `INSERT` statements that initialize the `account` table with some values. - -When the application is started, all of the queries specified by the changesets are executed in the order specified by their `changeset` tag's `id` value. At application startup, Liquibase also creates a table called [`databasechangelog`](https://docs.liquibase.com/concepts/databasechangelog-table.html) in the database where it performs changes. This table's rows log all completed changesets. - -To see the completed changesets, open a new terminal, start the [built-in SQL shell](cockroach-sql.html), and query the `databasechangelog` table: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --certs-dir=certs -~~~ - - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM roach_data.databasechangelog; -~~~ - -~~~ - id | author | filename | dateexecuted | orderexecuted | exectype | md5sum | description | comments | tag | liquibase | contexts | labels | deployment_id ------+--------+-----------------------------------+----------------------------------+---------------+----------+------------------------------------+--------------------------------------------------------------------------------------------------------+----------+------+-----------+----------+--------+---------------- - 1 | root | classpath:db/changelog-master.xml | 2020-06-17 14:57:01.431506+00:00 | 1 | EXECUTED | 8:939a1a8c47676119a94d0173802d207e | sqlFile | | NULL | 3.8.9 | crdb | NULL | 2420221402 - 2 | root | classpath:db/changelog-master.xml | 2020-06-17 14:57:01.470847+00:00 | 2 | EXECUTED | 8:c2945f2a445cf60b4b203e1a91d14a89 | insert tableName=account; insert tableName=account; insert tableName=account; insert tableName=account | | NULL | 3.8.9 | crdb | NULL | 2420221402 -(2 rows) -~~~ - -Typically, Liquibase properties are defined in a separate [`liquibase.properties`](https://docs.liquibase.com/workflows/liquibase-community/creating-config-properties.html) file. In this application, `application.yml`, the properties file that sets the [Spring properties](https://docs.spring.io/spring-boot/docs/current/reference/html/appendix-application-properties.html), also includes some properties that enable and configure Liquibase: - -~~~ yml -... - liquibase: - change-log: classpath:db/changelog-master.xml - default-schema: - drop-first: false - contexts: crdb - enabled: true -... -~~~ - -### Domain entities - -`Account.java` defines the [entity](https://en.wikipedia.org/wiki/Domain-driven_design#Building_blocks) for the `accounts` table. This class is used throughout the application to represent a row of data in the `accounts` table. - -Here are the contents of [`Account.java`](https://github.com/cockroachlabs/roach-data/tree/master/roach-data-jpa/src/main/java/io/roach/data/jpa/Account.java): - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/cockroachlabs/roach-data/master/roach-data-jpa/src/main/java/io/roach/data/jpa/Account.java %} -~~~ - -Spring Data JPA supports standard Java Persistence API (JPA) annotations for domain entity class definitions. The `Account` class definition uses these annotations to create the `accounts` table entity: - -- `@Entity` declares the `Account` an entity class. -- `@Table` associates the entity with the persisted `account` table. -- `@Column` declare each private attribute a column of the `account` table. -- `@GeneratedValue` indicates that the value for the column should be automatically generated. -- `@Id` declares the [primary key column](primary-key.html) of the table. -- `@Enumerated` specifies the type of data that the column holds. - -### Hypermedia representation - -To represent database objects as [HAL+JSON](https://en.wikipedia.org/wiki/Hypertext_Application_Language) for the REST API, the application extends the Spring HATEOAS module's [RepresentationModel](https://docs.spring.io/spring-hateoas/docs/current/reference/html/#fundamentals.representation-models) class with `AccountModel`. Like the `Account` class, its attributes represent a row of data in the `accounts` table. - -The contents of [`AccountModel.java`](https://github.com/cockroachlabs/roach-data/tree/master/roach-data-jpa/src/main/java/io/roach/data/jpa/AccountModel.java): - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/cockroachlabs/roach-data/master/roach-data-jpa/src/main/java/io/roach/data/jpa/AccountModel.java %} -~~~ - -We do not go into much detail about hypermedia representation in this tutorial. For more information, see the [Spring HATEOAS Reference Documentation](https://docs.spring.io/spring-hateoas/docs/current/reference/html/). - -### Spring repositories - -To abstract the database layer, Spring applications use the [`Repository` interface](https://docs.spring.io/spring-data/jpa/docs/current/reference/html/#repositories), or some subinterface of `Repository`. This interface maps to a database object, like a table, and its methods map to queries against that object, like a [`SELECT`](selection-queries.html) or an [`INSERT`](insert.html) statement against a table. - -[`AccountRepository.java`](https://github.com/cockroachlabs/roach-data/tree/master/roach-data-jpa/src/main/java/io/roach/data/jpa/AccountRepository.java) defines the main repository for the `accounts` table: - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/cockroachlabs/roach-data/master/roach-data-jpa/src/main/java/io/roach/data/jpa/AccountRepository.java %} -~~~ - -`AccountRepository` extends a subinterface of `Repository` that is provided by Spring for JPA data access called `JpaRepository`. The `AccountRepository` methods use the [`@Query`](https://docs.spring.io/spring-data/jpa/docs/current/reference/html/#jpa.query-methods.at-query) annotation strategy to define queries manually, as strings. - -Note that, in addition to having the `@Repository` annotation, the `AccountRepository` interface has a [`@Transactional` annotation](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#transaction-declarative-annotations). When [transaction management](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#transaction-declarative) is enabled in an application (i.e., with [`@EnableTransactionManagement`](https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/transaction/annotation/EnableTransactionManagement.html)), Spring automatically wraps all objects with the `@Transactional` annotation in [a proxy](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-understanding-aop-proxies) that handles calls to the object. - -`@Transactional` takes a number of parameters, including a `propagation` parameter that determines the transaction propagation behavior around an object (i.e., at what point in the stack a transaction starts and ends). This sample application follows the [entity-control-boundary (ECB) pattern](https://en.wikipedia.org/wiki/Entity-control-boundary). As such, the [REST service boundaries](#rest-controller) should determine where a [transaction](transactions.html) starts and ends rather than the query methods defined in the data access layer. To follow the ECB design pattern, `propagation=MANDATORY` for `AccountRepository`, which means that a transaction must already exist in order to call the `AccountRepository` query methods. In contrast, the `@Transactional` annotations on the [Rest controller entities](#rest-controller) in the web layer have `propagation=REQUIRES_NEW`, meaning that a new transaction must be created for each REST request. - -For details about control flow and transaction management in this application, see [Transaction management](#transaction-management). For more general information about Spring transaction management, see [Understanding the Spring Framework’s Declarative Transaction Implementation](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#tx-decl-explained) on Spring's documentation website. - -### REST controller - -There are several endpoints exposed by the application's web layer, some of which monitor the health of the application, and some that map to queries executed against the connected database. All of the endpoints served by the application are handled by the `AccountController` class, which is defined in [`AccountController.java`](https://github.com/cockroachlabs/roach-data/tree/master/roach-data-jpa/src/main/java/io/roach/data/jpa/AccountController.java): - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/cockroachlabs/roach-data/master/roach-data-jpa/src/main/java/io/roach/data/jpa/AccountController.java %} -~~~ - - Annotated with [`@RestController`](https://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/web/bind/annotation/RestController.html), `AccountController` defines the primary [web controller](https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller) component of the application. The `AccountController` methods define the endpoints, routes, and business logic of REST services for account querying and money transferring. Its attributes include an instantiation of [`AccountRepository`](#spring-repositories), called `accountRepository`, that establishes an interface to the `accounts` table through the data access layer. - -As mentioned in the [Spring repositories](#spring-repositories) section, the application's transaction boundaries follow the [entity-control-boundary (ECB) pattern](https://en.wikipedia.org/wiki/Entity-control-boundary), meaning that the web service boundaries of the application determine where a [transaction](transactions.html) starts and ends. To follow the ECB pattern, the `@Transactional` annotation on each of the HTTP entities (`listAccounts()`, `getAccount()`, and `transfer()`) has `propagation=REQUIRES_NEW`. This ensures that each time a REST request is made to an endpoint, a new transaction context is created. - -For details on how aspects handle control flow and transaction management in the application, see [Transaction management](#transaction-management). - -### Transaction management - -When [transaction management](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#transaction-declarative) is enabled in an application, Spring automatically wraps all objects annotated with `@Transactional` in [a proxy](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-understanding-aop-proxies) that handles calls to the object. By default, this proxy starts and closes transactions according to the configured transaction management behavior (e.g., the `propagation` level). The proxy methods that handle transactions make up the *primary transaction advisor*. - -Using [@AspectJ annotations](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#transaction-declarative-aspectj), this sample application extends the default transaction proxy behavior to handle [transaction retries](#transaction-retries) with another explicitly-defined [aspect](https://en.wikipedia.org/wiki/Aspect_(computer_programming)): `RetryableTransactionAspect`. Methods of this aspect are declared as [advice](https://en.wikipedia.org/wiki/Advice_(programming)) to be executed around method calls annotated with `@Transactional`. - -#### Ordering advice - -To determine the order of evaluation when multiple transaction advisors match the same [pointcut](https://en.wikipedia.org/wiki/Pointcut) (in this case, around `@Transactional` method calls), this application explicitly declares an order of precedence for calling advice. - -The [`@Order`](https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/core/annotation/Order.html) annotation takes a value that indicates the precedence of its advice. In the case of `RetryableTransactionAspect`, the annotation is passed `Ordered.LOWEST_PRECEDENCE-1`, which places the retry advisor one level of precedence above the lowest level. By default, the primary transaction advisor has the lowest level of precedence (`Ordered.LOWEST_PRECEDENCE`). This means that the retry logic will be evaluated before a transaction is opened. - -For more details about advice ordering, see [Advice Ordering](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-ataspectj-advice-ordering) on the Spring documentation site. - -#### Transaction retries - -Transactions may require retries if they experience deadlock or [transaction contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention) that cannot be resolved without allowing [serialization](demo-serializable.html) anomalies. To handle transactions that are aborted due to transient serialization errors, we highly recommend writing [client-side transaction retry logic](transactions.html#client-side-intervention) into applications written on CockroachDB. - -In this application, transaction retry logic is written into the methods of the `RetryableTransactionAspect` class, declared an aspect with the `@Aspect` annotation. Here are the contents of [`RetryableTransactionAspect.java`](https://github.com/cockroachlabs/roach-data/blob/master/roach-data-jpa/src/main/java/io/roach/data/jpa/RetryableTransactionAspect.java): - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/cockroachlabs/roach-data/master/roach-data-jpa/src/main/java/io/roach/data/jpa/RetryableTransactionAspect.java %} -~~~ - -The `anyTransactionBoundaryOperation` method is declared as a pointcut with the [`@Pointcut` annotation](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-pointcuts). In Spring, pointcut declarations must include an expression to determine where [join points](https://en.wikipedia.org/wiki/Join_point) occur in the application control flow. To help define these expressions, Spring supports a set of [designators](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-pointcuts-designators). The application uses two of them here: `execution`, which matches method execution joint points (i.e., defines a joint point when a specific method is executed, in this case, *any* method in the `io.roach.` namespace), and `@annotation`, which limits the matches to methods with a specific annotation, in this case `@Transactional`. - -`retryableOperation` handles the application retry logic in the form of advice. Spring supports [several different annotations to declare advice](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-advice). The [`@Around` annotation](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-ataspectj-around-advice) allows an advice method to work before and after the `anyTransactionBoundaryOperation(transactional)` join point. It also allows the advice method to call the next matching advisor with the `ProceedingJoinPoint.proceed();` method. - -`retryableOperation` first verifies that there is no active transaction. It then increments the retry count and attempts to proceed to the next advice method with the `ProceedingJoinPoint.proceed()` method. If the underlying methods (i.e., the primary transaction advisor's methods and the [annotated query methods](#spring-repositories)) succeed, the transaction has been successfully committed to the database. The results are then returned and the application flow continues. If a failure in the underlying layers occurs due to a transient error, then the transaction is retried. The time between each retry grows with each retry until the maximum number of retries is reached. - -## See also - -Spring documentation: - -- [Spring Boot website](https://spring.io/projects/spring-boot) -- [Spring Framework Overview](https://docs.spring.io/spring/docs/current/spring-framework-reference/overview.html#overview) -- [Spring Core documentation](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#spring-core) -- [Accessing Data with JPA](https://spring.io/guides/gs/accessing-data-jpa/) -- [Data Access with JDBC](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#jpa) -- [Spring Web MVC](https://docs.spring.io/spring/docs/current/spring-framework-reference/web.html#mvc) - -CockroachDB documentation: - -- [Learn CockroachDB SQL](learn-cockroachdb-sql.html) -- [Client Connection Parameters](connection-parameters.html) -- [CockroachDB Developer Guide](developer-guide-overview.html) -- [Hello World Example Apps](example-apps.html) -- [Transactions](transactions.html) diff --git a/src/current/v21.1/build-a-spring-app-with-cockroachdb-mybatis.md b/src/current/v21.1/build-a-spring-app-with-cockroachdb-mybatis.md deleted file mode 100644 index 49804522dbf..00000000000 --- a/src/current/v21.1/build-a-spring-app-with-cockroachdb-mybatis.md +++ /dev/null @@ -1,418 +0,0 @@ ---- -title: Build a Spring App with CockroachDB and MyBatis -summary: Learn how to use CockroachDB from a simple Spring application with MyBatis. -toc: true -twitter: false ---- - -
- - - - -
- -This tutorial shows you how to build a simple [Spring Boot](https://spring.io/projects/spring-boot) application with CockroachDB, using the [MyBatis-Spring-Boot-Starter module](http://mybatis.org/spring-boot-starter) for data access. - -## Before you begin - -{% include {{page.version.version}}/app/before-you-begin.md %} - -## Step 1. Install JDK - -Download and install a Java Development Kit. MyBatis-Spring supports Java versions 8+. In this tutorial, we use [JDK 11 from OpenJDK](https://openjdk.java.net/install/). - -## Step 2. Install Gradle - -This example application uses [Gradle](https://gradle.org/) to manage all application dependencies. Spring supports Gradle versions 6+. - -To install Gradle on macOS, run the following command: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ brew install gradle -~~~ - -To install Gradle on a Debian-based Linux distribution like Ubuntu: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ apt-get install gradle -~~~ - -To install Gradle on a Red Hat-based Linux distribution like Fedora: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ dnf install gradle -~~~ - -For other ways to install Gradle, see [its official documentation](https://docs.gradle.org/current/userguide/installation.html). - -## Step 3. Get the application code - -To get the application code, download or clone the [`mybatis-cockroach-demo` repository](https://github.com/jeffgbutler/mybatis-cockroach-demo). - -## Step 4. Create the `maxroach` user and `bank` database - -
- -Start the [built-in SQL shell](cockroach-sql.html): - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --certs-dir=certs -~~~ - -In the SQL shell, issue the following statements to create the `maxroach` user and `bank` database: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE USER IF NOT EXISTS maxroach; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE DATABASE bank; -~~~ - -Give the `bank` user the necessary permissions: - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT ALL ON DATABASE bank TO maxroach; -~~~ - -Exit the SQL shell: - -{% include_cached copy-clipboard.html %} -~~~ sql -> \q -~~~ - -## Step 5. Generate a certificate for the `maxroach` user - -Create a certificate and key for the `maxroach` user by running the following command. The code samples will run as this user. - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach cert create-client maxroach --certs-dir=certs --ca-key=my-safe-directory/ca.key --also-generate-pkcs8-key -~~~ - -The [`--also-generate-pkcs8-key` flag](cockroach-cert.html#flag-pkcs8) generates a key in [PKCS#8 format](https://tools.ietf.org/html/rfc5208), which is the standard key encoding format in Java. In this case, the generated PKCS8 key will be named `client.maxroach.key.pk8`. - -## Step 6. Run the application - -To run the application: - -1. Open and edit the `src/main/resources/application.yml` file so that the `url` field specifies the full [connection string](connection-parameters.html#connect-using-a-url) to the [running CockroachDB cluster](#before-you-begin). To connect to a secure cluster, this connection string must set the `sslmode` connection parameter to `require`, and specify the full path to the client, node, and user certificates in the connection parameters. For example: - - {% include_cached copy-clipboard.html %} - ~~~ yml - ... - datasource: - url: jdbc:postgresql://localhost:26257/bank?ssl=true&sslmode=require&sslrootcert=certs/ca.crt&sslkey=certs/client.maxroach.key.pk8&sslcert=certs/client.maxroach.crt - ... - ~~~ -1. Open a terminal, and navigate to the `mybatis-cockroach-demo` project directory: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cd /mybatis-cockroach-demo - ~~~ - -1. Run the Gradle script to download the application dependencies, compile the code, and run the application: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ ./gradlew bootRun - ~~~ - -
- -
- -Start the [built-in SQL shell](cockroach-sql.html): - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --insecure -~~~ - -In the SQL shell, issue the following statements to create the `maxroach` user and `bank` database: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE USER IF NOT EXISTS maxroach; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE DATABASE bank; -~~~ - -Give the `bank` user the necessary permissions: - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT ALL ON DATABASE bank TO maxroach; -~~~ - -Exit the SQL shell: - -{% include_cached copy-clipboard.html %} -~~~ sql -> \q -~~~ - -## Step 6. Run the application - -To run the application: - -1. Open and edit the `src/main/resources/application.yml` file so that the `url` field specifies the full [connection string](connection-parameters.html#connect-using-a-url) to the [running CockroachDB cluster](#before-you-begin). For example: - - ~~~ yaml - ... - datasource: - url: jdbc:postgresql://localhost:26257/bank?ssl=false - ... - ~~~ -1. Open a terminal, and navigate to the `mybatis-cockroach-demo` project directory: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cd /mybatis-cockroach-demo - ~~~ - -1. Run the Gradle script to download the application dependencies, compile the code, and run the application: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ ./gradlew bootRun - ~~~ - -
- -The output should look like the following: - -~~~ -> Task :bootRun - - . ____ _ __ _ _ - /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ -( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ - \\/ ___)| |_)| | | | | || (_| | ) ) ) ) - ' |____| .__|_| |_|_| |_\__, | / / / / - =========|_|==============|___/=/_/_/_/ - :: Spring Boot :: (v2.2.6.RELEASE) - -2020-06-01 14:40:04.333 INFO 55970 --- [ main] c.e.c.CockroachDemoApplication : Starting CockroachDemoApplication on MyComputer with PID 55970 (path/mybatis-cockroach-demo/build/classes/java/main started by user in path/mybatis-cockroach-demo) -2020-06-01 14:40:04.335 INFO 55970 --- [ main] c.e.c.CockroachDemoApplication : No active profile set, falling back to default profiles: default -2020-06-01 14:40:05.195 INFO 55970 --- [ main] c.e.c.CockroachDemoApplication : Started CockroachDemoApplication in 1.39 seconds (JVM running for 1.792) -2020-06-01 14:40:05.216 INFO 55970 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting... -2020-06-01 14:40:05.611 INFO 55970 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed. -deleteAllAccounts: - => 2 total deleted accounts -insertAccounts: - => 2 total new accounts in 1 batches -printNumberOfAccounts: - => Number of accounts at time '14:40:05.660226': - => 2 total accounts -printBalances: - => Account balances at time '14:40:05.678942': - ID 1 => $1000 - ID 2 => $250 -transferFunds: - => $100 transferred between accounts 1 and 2, 2 rows updated -printBalances: - => Account balances at time '14:40:05.688511': - ID 1 => $900 - ID 2 => $350 -bulkInsertRandomAccountData: - => finished, 500 total rows inserted in 1 batches -printNumberOfAccounts: - => Number of accounts at time '14:40:05.960214': - => 502 total accounts -2020-06-01 14:40:05.968 INFO 55970 --- [extShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated... -2020-06-01 14:40:05.993 INFO 55970 --- [extShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed. - -BUILD SUCCESSFUL in 12s -3 actionable tasks: 3 executed -~~~ - -The application runs a number of test functions that result in reads and writes to the `accounts` table in the `bank` database. - -For more details about the application code, see [Application details](#application-details). - -## Application details - -This section guides you through the different components of the application project in detail. - -### Main process - -The main process of the application is defined in `src/main/java/com/example/cockroachdemo/CockroachDemoApplication.java`: - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/jeffgbutler/mybatis-cockroach-demo/master/src/main/java/com/example/cockroachdemo/CockroachDemoApplication.java %} -~~~ - -The `SpringApplication.run` call in the `main` method bootstraps and launches a Spring application. The [`@SpringBootApplication` annotation](https://docs.spring.io/spring-boot/docs/current/reference/html/using-spring-boot.html#using-boot-using-springbootapplication-annotation) on the `CockroachDemoApplication` class triggers Spring's [component scanning](https://docs.spring.io/spring-boot/docs/current/reference/html/using-spring-boot.html#using-boot-structuring-your-code) and [auto-configuration](https://docs.spring.io/spring-boot/docs/current/reference/html/using-spring-boot.html#using-boot-auto-configuration) features. - -The `BasicExample` class, defined in `src/main/java/com/example/cockroachdemo/BasicExample.java`, is one of the components detected in the component scan: - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/jeffgbutler/mybatis-cockroach-demo/master/src/main/java/com/example/cockroachdemo/BasicExample.java %} -~~~ - -`BasicExample` implements the [Spring `CommandLineRunner` interface](https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#boot-features-command-line-runner). Implementations of this interface automatically run when detected in a Spring project directory. `BasicExample` runs a series of test methods that are eventually executed as SQL queries in the [data access layer of the application](#mappers). - -### Configuration - -All [MyBatis-Spring](https://mybatis.org/spring/) applications need a [`DataSource`](https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-configure-datasource), a [`SqlSessionFactory`](https://mybatis.org/spring/factorybean.html), and at least one [mapper interface](https://mybatis.org/spring/mappers.html). The [MyBatis-Spring-Boot-Starter](https://mybatis.org/spring-boot-starter/mybatis-spring-boot-autoconfigure) module, built on [MyBatis](https://mybatis.org/mybatis-3/) and MyBatis-Spring, and used by this application, greatly simplifies how you configure each of these required elements. - -Applications that use MyBatis-Spring-Boot-Starter typically need just an annotated mapper interface and an existing `DataSource` in the Spring environment. The module detects the `DataSource`, creates a `SqlSessionFactory` from the `DataSource`, creates a thread-safe [`SqlSessionTemplate`](https://mybatis.org/spring/sqlsession.html#SqlSessionTemplate) with the `SqlSessionFactory`, and then auto-scans the mappers and links them to the `SqlSessionTemplate` for injection. The `SqlSessionTemplate` automatically commits, rolls back, and closes sessions, based on the application's [Spring-based transaction configuration](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#transaction). - -This sample application implements [batch write operations](insert.html#performance-best-practices), a CockroachDB best practice for executing multiple `INSERT` and `UPSERT` statements. MyBatis applications that support batch operations require some additional configuration work, even if the application uses MyBatis-Spring-Boot-Starter: - -- The application must define a specific mapper interface for batch query methods. -- The application must define a `SqlSessionTemplate` constructor, specifically for batch operations, that uses the [`BATCH` executor type](https://mybatis.org/mybatis-3/apidocs/reference/org/apache/ibatis/executor/BatchExecutor.html). -- The batch mapper must be explicitly registered with the batch-specific `SqlSessionTemplate`. - -The class defined in `src/main/java/com/example/cockroachdemo/MyBatisConfiguration.java` configures the application to meet these requirements: - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/jeffgbutler/mybatis-cockroach-demo/master/src/main/java/com/example/cockroachdemo/MyBatisConfiguration.java %} -~~~ - -This class explicitly defines the batch `SqlSessionTemplate` (i.e., `batchSqlSessionTemplate`), and registers `batchmapper`, the batch mapper interface defined in [`src/main/java/com/example/cockroachdemo/batchmapper/BatchMapper.java`](#mappers) with `batchSqlSessionTemplate`. To complete the MyBatis configuration, the class also declares a `DataSource`, and defines the remaining `SqlSessionFactory` and `SqlSessionTemplate` beans. - -Note that a configuration class is not required for MyBatis-Spring-Boot-Starter applications that do not implement batch operations. - -### Data source - -`src/main/resources/application.yml` contains the metadata used to create a connection to the CockroachDB cluster: - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/jeffgbutler/mybatis-cockroach-demo/master/src/main/resources/application.yml %} -~~~ - -Spring Boot uses the application's `datasource` property [to auto-configure the database connection](https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-configure-datasource). This database connection configuration can be injected into the application's `SqlSessionFactoryBean`, as is explicitly done in the [MyBatisConfiguration](#configuration) configuration class definition. - -### Mappers - -All MyBatis applications require at least one mapper interface. These mappers take the place of manually-defined data access objects (DAOs). They provide other layers of the application an interface to the database. - -MyBatis-Spring-Boot-Starter usually scans the project for interfaces annotated with `@Mapper`, links the interfaces to a `SqlSessionTemplate`, and registers them with Spring so they can be [injected into the application's Spring beans](https://docs.spring.io/spring-boot/docs/current/reference/html/using-spring-boot.html#using-boot-spring-beans-and-dependency-injection). As mentioned in the [Configuration section](#configuration), because the application supports batch writes, the two mapper interfaces in the application are registered and linked manually in the `MyBatisConfiguration` configuration class definition. - -#### Account mapper - -`src/main/java/com/example/cockroachdemo/mapper/AccountMapper.java` defines the mapper interface to the `accounts` table using the [MyBatis Java API](https://mybatis.org/mybatis-3/java-api.html): - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/jeffgbutler/mybatis-cockroach-demo/master/src/main/java/com/example/cockroachdemo/mapper/AccountMapper.java %} -~~~ - -The `@Mapper` annotation declares the interface a mapper for MyBatis to scan. The SQL statement annotations on each of the interface methods map them to SQL queries. For example, the first method, `deleteAllAccounts()` is marked as a `DELETE` statement with the `@Delete` annotation. This method executes the SQL statement specified in the string passed to the annotation, "`delete from accounts`", which deletes all rows in the `accounts` table. - -#### Batch account mapper - -`src/main/java/com/example/cockroachdemo/batchmapper/BatchAccountMapper.java` defines a mapper interface for [batch writes](insert.html#performance-best-practices): - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/jeffgbutler/mybatis-cockroach-demo/master/src/main/java/com/example/cockroachdemo/batchmapper/BatchAccountMapper.java %} -~~~ - -This interface has a single `INSERT` statement query method, along with a method for flushing (i.e., executing) a batch of statements. - -### Services - -`src/main/java/com/example/cockroachdemo/service/AccountService.java` defines the service interface, with a number of methods for reading and writing to the database: - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/jeffgbutler/mybatis-cockroach-demo/master/src/main/java/com/example/cockroachdemo/service/AccountService.java %} -~~~ - -`MyBatisAccountService.java` implements the `AccountService` interface, using the mappers defined in [`AccountMapper.java` and `BatchAccountMapper.java`](#mappers), and the models defined in [`Account.java` and `BatchResults.java`](#models): - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/jeffgbutler/mybatis-cockroach-demo/master/src/main/java/com/example/cockroachdemo/service/MyBatisAccountService.java %} -~~~ - -Note that the public methods (i.e., the methods to be called by other classes in the project) are annotated as [`@Transactional`](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#transaction-declarative-annotations) methods. This ensures that all of the SQL statements executed in the data access layer are run within the context of a [database transaction](transactions.html) - -`@Transactional` takes a number of parameters, including a `propagation` parameter that determines the transaction propagation behavior around an object (i.e., at what point in the stack a transaction starts and ends). `propagation=REQUIRES_NEW` for the methods in the service layer, meaning that a new transaction must be created each time a request is made to the service layer. With this propagation behavior, the application follows the [entity-control-boundary (ECB) pattern](https://en.wikipedia.org/wiki/Entity-control-boundary), as the service boundaries determine where a [transaction](transactions.html) starts and ends rather than the lower-level query methods of the [mapper interfaces](#mappers). - -For more details on aspect-oriented transaction management in this application, [see below](#transaction-management). - -### Models - -Instances of the `Account` class, defined in `src/main/java/com/example/cockroachdemo/model/Account.java`, represent rows in the `accounts` table: - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/jeffgbutler/mybatis-cockroach-demo/master/src/main/java/com/example/cockroachdemo/model/Account.java %} -~~~ - -Instances of the `BatchResults` class, defined in `src/main/java/com/example/cockroachdemo/model/BatchResults.java`, hold metadata about a batch write operation and its results: - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/jeffgbutler/mybatis-cockroach-demo/master/src/main/java/com/example/cockroachdemo/model/BatchResults.java %} -~~~ - -### Transaction management - -MyBatis-Spring supports Spring's [declarative, aspect-oriented transaction management syntax](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#transaction-declarative), including the [`@Transactional`](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#transaction-declarative-annotations) annotation and [AspectJ's AOP annotations](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#transaction-declarative-aspectj). - -Transactions may require retries if they experience deadlock or [transaction contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention) that cannot be resolved without allowing [serialization](demo-serializable.html) anomalies. To handle transactions that are aborted due to transient serialization errors, we highly recommend writing [client-side transaction retry logic](transactions.html#client-side-intervention) into applications written on CockroachDB. In this application, transaction retry logic is written into the methods of the `RetryableTransactionAspect` class, defined in `src/main/java/com/example/cockroachdemo/RetryableTransactionAspect.java`: - -{% include_cached copy-clipboard.html %} -~~~ java -{% remote_include https://raw.githubusercontent.com/jeffgbutler/mybatis-cockroach-demo/master/src/main/java/com/example/cockroachdemo/RetryableTransactionAspect.java %} -~~~ - -The [`@Aspect` annotation](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-at-aspectj) declares `RetryableTransactionAspect` an [aspect](https://en.wikipedia.org/wiki/Aspect_(computer_programming)), with [pointcut](https://en.wikipedia.org/wiki/Pointcut) and [advice](https://en.wikipedia.org/wiki/Advice_(programming)) methods. - -#### Transactional pointcut - -The [`@Pointcut` annotation](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-pointcuts) declares the `anyTransactionBoundaryOperation` method the pointcut for determining when to execute the aspect's advice. The `@annotation` [designator](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-pointcuts-designators) passed to the `@Pointcut` annotation limits the matches (i.e., [join points](https://en.wikipedia.org/wiki/Join_point)) to method calls with a specific annotation, in this case, `@Transactional`. - -#### Transaction retry advice - -`retryableOperation` handles the application retry logic, with [exponential backoff](https://en.wikipedia.org/wiki/Exponential_backoff), as the advice to execute at an `anyTransactionBoundaryOperation(transactional)` join point. Spring supports [several different annotations to declare advice](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-advice). The [`@Around` annotation](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-ataspectj-around-advice) allows an advice method to work before and after the join point. It also gives the advice method control over executing any other matching advisors. - -`retryableOperation` first verifies that there is no active transaction. It then increments the retry count and attempts to proceed to the next advice method with the `ProceedingJoinPoint.proceed()` method. If the underlying data access layer method (i.e., the mapper interface method annotated with `@Transactional`) succeeds, the results are returned and the application flow continues. If the method fails, then the transaction is retried. The time between each retry grows with each retry until the maximum number of retries is reached. - -#### Advice ordering - -Spring automatically adds [transaction management advice](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#tx-decl-explained) to all methods annotated with `@Transactional`. Because the pointcut for `RetryableTransactionAspect` also matches methods annotated with `@Transactional`, there will always be two advisors that match the same pointcut. When multiple advisors match at the same pointcut, an `@Order` annotation on an advisor's aspect can specify the order in which the advice should be evaluated. - -To control when and how often a transaction is retried, the transaction retry advice must be executed outside the context of a transaction (i.e., it must be evaluated before the primary transaction management advisor). By default, the primary transaction management advisor is given the lowest level of precedence. The `@Order` annotation on `RetryableTransactionAspect` is passed `Ordered.LOWEST_PRECEDENCE-1`, which places this aspect's advice at a level of precedence above the primary transaction advisor, which results in the retry logic being evaluated before the transaction management advisor. - -For more details about advice ordering in Spring, see [Advice Ordering](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-ataspectj-advice-ordering) on the Spring documentation site. - -## See also - -Spring documentation: - -- [Spring Boot website](https://spring.io/projects/spring-boot) -- [Spring Framework Overview](https://docs.spring.io/spring/docs/current/spring-framework-reference/overview.html#overview) -- [Spring Core documentation](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#spring-core) -- [MyBatis documentation](https://mybatis.org/mybatis-3/) -- [MyBatis Spring integration](https://mybatis.org/spring/) - -CockroachDB documentation: - -- [Learn CockroachDB SQL](learn-cockroachdb-sql.html) -- [Client Connection Parameters](connection-parameters.html) -- [CockroachDB Developer Guide](developer-guide-overview.html) -- [Hello World Example Apps](example-apps.html) -- [Transactions](transactions.html) diff --git a/src/current/v21.1/build-a-typescript-app-with-cockroachdb.md b/src/current/v21.1/build-a-typescript-app-with-cockroachdb.md deleted file mode 100644 index 52b5ee0ef11..00000000000 --- a/src/current/v21.1/build-a-typescript-app-with-cockroachdb.md +++ /dev/null @@ -1,138 +0,0 @@ ---- -title: Build a TypeScript App with CockroachDB and TypeORM -summary: Learn how to use CockroachDB with the TypeORM framework. -toc: true -twitter: false -referral_id: docs_hello_world_typescript_typeorm ---- - -
- - - - - -
- -This tutorial shows you how run a simple application built with [TypeORM](https://typeorm.io/#/). - -## Step 1. Start CockroachDB - -{% include {{page.version.version}}/app/start-cockroachdb.md %} - -## Step 2. Create a database - -{% include {{page.version.version}}/app/create-a-database.md %} - -## Step 3. Get the code - -1. Clone [the code's GitHub repository](https://github.com/cockroachlabs/example-app-typescript-typeorm): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ git clone git@github.com:cockroachlabs/example-app-typescript-typeorm.git - ~~~ - -1. Navigate to the repo directory and install the application dependencies: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cd example-app-typescript-typeorm - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ npm install - ~~~ - -## Step 4. Configure your CockroachDB connection - -
- -1. Open the `datasource.ts` file, and comment out the `ssl: true`, `extra` and `options` configuration properties. - -1. In the `datasource.ts` file, uncomment `ssl: { rejectUnauthorized: false }`. - - {{site.data.alerts.callout_danger}} - Only use `ssl: { rejectUnauthorized: false }` in development, for insecure connections. - {{site.data.alerts.end}} - - The `DataSource` configuration should look similar to the following: - - ~~~ ts - export const AppDataSource = new DataSource({ - type: "cockroachdb", - url: process.env.DATABASE_URL, - ssl: { rejectUnauthorized: false }, // For insecure connections only - /* ssl: true, - extra: { - options: "--cluster=" - }, */ - synchronize: true, - logging: false, - entities: ["src/entity/**/*.ts"], - migrations: ["src/migration/**/*.ts"], - subscribers: ["src/subscriber/**/*.ts"], - }) - ~~~ - -1. Set the `DATABASE_URL` environment variable to the connection string provided in the `cockroach demo` welcome text. - -
- -
- -1. Open the `datasource.ts` file, and edit the `--cluster=` configuration property to specify the routing ID to your serverless cluster. - -1. Set the `DATABASE_URL` environment variable to a CockroachDB connection string compatible with TypeORM. - - TypeORM accepts the following format for CockroachDB {{ site.data.products.serverless }} connection strings: - - {% include_cached copy-clipboard.html %} - ~~~ - postgresql://:@:/ - ~~~ - -
- -## Step 5. Run the code - -Start the application: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ npm start -~~~ - -You should see the following output in your terminal: - -~~~ -Inserting a new account into the database... -Saved a new account. -Printing balances from account 1db0f34a-55e8-42e7-adf1-49e76010b763. -[ - Account { id: '1db0f34a-55e8-42e7-adf1-49e76010b763', balance: 1000 } -] -Inserting a new account into the database... -Saved a new account. -Printing balances from account 4e26653a-3821-48c8-a481-47eb73b3e4cc. -[ - Account { id: '4e26653a-3821-48c8-a481-47eb73b3e4cc', balance: 250 } -] -Transferring 500 from account 1db0f34a-55e8-42e7-adf1-49e76010b763 to account 4e26653a-3821-48c8-a481-47eb73b3e4cc. -Transfer complete. -Printing balances from account 1db0f34a-55e8-42e7-adf1-49e76010b763. -[ - Account { id: '1db0f34a-55e8-42e7-adf1-49e76010b763', balance: 1000 } -] -Printing balances from account 4e26653a-3821-48c8-a481-47eb73b3e4cc. -[ - Account { id: '4e26653a-3821-48c8-a481-47eb73b3e4cc', balance: 250 } -] -~~~ - -## What's next? - -Read more about using the [TypeORM](https://typeorm.io/#/). - -{% include {{page.version.version}}/app/see-also-links.md %} diff --git a/src/current/v21.1/bulk-delete-data.md b/src/current/v21.1/bulk-delete-data.md deleted file mode 100644 index c5c69577b2a..00000000000 --- a/src/current/v21.1/bulk-delete-data.md +++ /dev/null @@ -1,203 +0,0 @@ ---- -title: Bulk-delete Data -summary: How to delete a large amount of data from a cluster -toc: true ---- - -To delete a large number of rows (i.e., tens of thousands of rows or more), we recommend iteratively deleting subsets of the rows that you want to delete, until all of the unwanted rows have been deleted. You can write a script to do this, or you can write a loop into your application. - -This page provides guidance on batch deleting with the `DELETE` query filter [on an indexed column](#batch-delete-on-an-indexed-column) and [on a non-indexed column](#batch-delete-on-a-non-indexed-column). Filtering on an indexed column is both simpler to implement and more efficient, but adding an index to a table can slow down insertions to the table and may cause bottlenecks. Queries that filter on a non-indexed column must perform at least one full-table scan, a process that takes time proportional to the size of the entire table. - -{{site.data.alerts.callout_success}} -If you want to delete all of the rows in a table (and not just a large subset of the rows), use a [`TRUNCATE` statement](#delete-all-of-the-rows-in-a-table). -{{site.data.alerts.end}} - -{{site.data.alerts.callout_danger}} -Exercise caution when batch deleting rows from tables with foreign key constraints and explicit [`ON DELETE` foreign key actions](foreign-key.html#foreign-key-actions). To preserve `DELETE` performance on tables with foreign key actions, we recommend using smaller batch sizes, as additional rows updated or deleted due to `ON DELETE` actions can make batch loops significantly slower. -{{site.data.alerts.end}} - -## Before you begin - -Before reading this page, do the following: - -- [Install CockroachDB](install-cockroachdb.html). -- [Start a local cluster](secure-a-cluster.html), or [create a CockroachDB {{ site.data.products.cloud }} cluster](../cockroachcloud/create-your-cluster.html). -- [Install a Postgres client](install-client-drivers.html). - - For the example on this page, we use the `psycopg2` Python driver. -- [Connect to the database](connect-to-the-database.html). -- [Insert data](insert-data.html) that you now want to delete. - - For the example on this page, we load a cluster with the [`tpcc` database](cockroach-workload.html#tpcc-workload) and data from [`cockroach workload`](cockroach-workload.html). - -## Batch delete on an indexed column - -For high-performance batch deletes, we recommending filtering the `DELETE` query on an [indexed column](indexes.html). - -{{site.data.alerts.callout_info}} -Having an indexed filtering column can make delete operations faster, but it might lead to bottlenecks in execution, especially if the filtering column is a [timestamp](timestamp.html). To reduce bottlenecks, we recommend using a [hash-sharded index](hash-sharded-indexes.html). -{{site.data.alerts.end}} - -Each iteration of a batch-delete loop should execute a transaction containing a single `DELETE` query. When writing this `DELETE` query: - -- Use a `WHERE` clause to filter on a column that identifies the unwanted rows. If the filtering column is not the primary key, the column should have [a secondary index](indexes.html). Note that if the filtering column is not already indexed, it is not beneficial to add an index just to speed up batch deletes. Instead, consider [batch deleting on non-indexed columns](#batch-delete-on-a-non-indexed-column). -- To ensure that rows are efficiently scanned in the `DELETE` query, add an [`ORDER BY`](order-by.html) clause on the filtering column. -- Use a [`LIMIT`](limit-offset.html) clause to limit the number of rows to the desired batch size. To determine the optimal batch size, try out different batch sizes (1,000 rows, 10,000 rows, 100,000 rows, etc.) and monitor the change in performance. -- Add a `RETURNING` clause to the end of the query that returns the filtering column values of the deleted rows. Then, using the values of the deleted rows, update the filter to match only the subset of remaining rows to delete. This narrows each query's scan to the fewest rows possible, and [preserves the performance of the deletes over time](delete.html#preserving-delete-performance-over-time). This pattern assumes that no new rows are generated that match on the `DELETE` filter during the time that it takes to perform the delete. - -For example, suppose that you want to delete all rows in the [`tpcc`](cockroach-workload.html#tpcc-workload) `new_order` table where `no_w_id` is less than `5`, in batches of 5,000 rows. To do this, you can write a script that loops over batches of 5,000 rows, following the `DELETE` query guidance provided above. Note that in this case, `no_w_id` is the first column in the primary index, and, as a result, you do not need to create a secondary index on the column. - -In Python, the script would look similar to the following: - -{% include_cached copy-clipboard.html %} -~~~ python -#!/usr/bin/env python3 - -import psycopg2 -import psycopg2.sql -import os - -conn = psycopg2.connect(os.environ.get('DB_URI')) -filter = 4 -lastrow = None - -while True: - with conn: - with conn.cursor() as cur: - if lastrow: - filter = lastrow[0] - query = psycopg2.sql.SQL("DELETE FROM new_order WHERE no_w_id <= %s ORDER BY no_w_id DESC LIMIT 5000 RETURNING no_w_id") - cur.execute(query, (filter,)) - print(cur.statusmessage) - if cur.rowcount == 0: - break - lastrow = cur.fetchone() - -conn.close() -~~~ - -This script iteratively deletes rows in batches of 5,000, until all of the rows where `no_w_id <= 4` are deleted. Note that at each iteration, the filter is updated to match a narrower subset of rows. - -## Batch delete on a non-indexed column - -If you cannot index the column that identifies the unwanted rows, we recommend defining the batch loop to execute separate read and write operations at each iteration: - -1. Execute a [`SELECT` query](selection-queries.html) that returns the primary key values for the rows that you want to delete. When writing the `SELECT` query: - - Use a `WHERE` clause that filters on the column identifying the rows. - - Add an [`AS OF SYSTEM TIME` clause](as-of-system-time.html) to the end of the selection subquery, or run the selection query in a separate, read-only transaction with [`SET TRANSACTION AS OF SYSTEM TIME`](as-of-system-time.html#using-as-of-system-time-in-transactions). This helps to reduce [transaction contention](transactions.html#transaction-contention). - - Use a [`LIMIT`](limit-offset.html) clause to limit the number of rows queried to a subset of the rows that you want to delete. To determine the optimal `SELECT` batch size, try out different sizes (10,000 rows, 100,000 rows, 1,000,000 rows, etc.), and monitor the change in performance. Note that this `SELECT` batch size can be much larger than the batch size of rows that are deleted in the subsequent `DELETE` query. - - To ensure that rows are efficiently scanned in the subsequent `DELETE` query, include an [`ORDER BY`](order-by.html) clause on the primary key. - -2. Write a nested `DELETE` loop over the primary key values returned by the `SELECT` query, in batches smaller than the initial `SELECT` batch size. To determine the optimal `DELETE` batch size, try out different sizes (1,000 rows, 10,000 rows, 100,000 rows, etc.), and monitor the change in performance. Where possible, we recommend executing each `DELETE` in a separate transaction. - -For example, suppose that you want to delete all rows in the [`tpcc`](cockroach-workload.html#tpcc-workload) `history` table that are older than a month. You can create a script that loops over the data and deletes unwanted rows in batches, following the query guidance provided above. - -In Python, the script would look similar to the following: - -{% include_cached copy-clipboard.html %} -~~~ python -#!/usr/bin/env python3 - -import psycopg2 -import os -import time - -conn = psycopg2.connect(os.environ.get('DB_URI')) - -while True: - with conn: - with conn.cursor() as cur: - cur.execute("SET TRANSACTION AS OF SYSTEM TIME '-5s'") - cur.execute("SELECT h_w_id, rowid FROM history WHERE h_date < current_date() - INTERVAL '1 MONTH' ORDER BY h_w_id, rowid LIMIT 20000") - pkvals = list(cur) - if not pkvals: - return - while pkvals: - batch = pkvals[:5000] - pkvals = pkvals[5000:] - with conn: - with conn.cursor() as cur: - cur.execute("DELETE FROM history WHERE (h_w_id, rowid) = ANY %s", (batch,)) - print(cur.statusmessage) - del batch - del pkvals - time.sleep(5) - -conn.close() -~~~ - -At each iteration, the selection query returns the primary key values of up to 20,000 rows of matching historical data from 5 seconds in the past, in a read-only transaction. Then, a nested loop iterates over the returned primary key values in smaller batches of 5,000 rows. At each iteration of the nested `DELETE` loop, a batch of rows is deleted. After the nested `DELETE` loop deletes all of the rows from the initial selection query, a time delay ensures that the next selection query reads historical data from the table after the last iteration's `DELETE` final delete. - -{{site.data.alerts.callout_info}} -CockroachDB records the timestamp of each row created in a table in the `crdb_internal_mvcc_timestamp` metadata column. In the absence of an explicit timestamp column in your table, you can use `crdb_internal_mvcc_timestamp` to filter expired data. - -`crdb_internal_mvcc_timestamp` cannot be indexed. As a result, we recommend following the [non-indexed column pattern](#batch-delete-on-a-non-indexed-column) if you plan to use `crdb_internal_mvcc_timestamp` as a filter for large deletes. - -**Exercise caution when using `crdb_internal_mvcc_timestamp` in production, as the column is subject to change without prior notice in new releases of CockroachDB.** -{{site.data.alerts.end}} - -## Batch-delete "expired" data - -CockroachDB does not support Time to Live (TTL) on table rows. To delete "expired" rows, we recommend automating a batch delete process using a job scheduler like `cron`. - -For example, suppose that every morning you want to delete all rows in the [`tpcc`](cockroach-workload.html#tpcc-workload) `history` table that are older than a month. To do this, you could use the example Python script that [batch deletes on the non-indexed `h_date` column](#batch-delete-on-a-non-indexed-column). - -To run the script with a daily `cron` job: - -1. Make the file executable: - {% include_cached copy-clipboard.html %} - ~~~ shell - $ chmod +x cleanup.py - ~~~ - -2. Create a new `cron` job: - {% include_cached copy-clipboard.html %} - ~~~ shell - $ crontab -e - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ txt - 30 10 * * * DB_URI='cockroachdb://user@host:26257/bank' cleanup.py >> ~/cron.log 2>&1 - ~~~ - -Saving the `cron` file will install a new job that runs the `cleanup.py` file every morning at 10:30 A.M., writing the results to the `cron.log` file. - -## Delete all of the rows in a table - -To delete all of the rows in a table, use a [`TRUNCATE` statement](truncate.html). - -For example, to delete all rows in the [`tpcc`](cockroach-workload.html#tpcc-workload) `new_order` table, execute the following SQL statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -TRUNCATE new_order; -~~~ - -You can execute the statement from a compatible SQL client (e.g., the [CockroachDB SQL client](cockroach-sql.html)), or in a script or application. - -For example, in Python, using the `psycopg2` client driver: - -{% include_cached copy-clipboard.html %} -~~~ python -#!/usr/bin/env python3 - -import psycopg2 -import os - -conn = psycopg2.connect(os.environ.get('DB_URI')) - -with conn: - with conn.cursor() as cur: - cur.execute("TRUNCATE new_order") -~~~ - -{{site.data.alerts.callout_success}} -For detailed reference documentation on the `TRUNCATE` statement, including additional examples, see the [`TRUNCATE` syntax page](truncate.html). -{{site.data.alerts.end}} - -## See also - -- [Delete data](delete-data.html) -- [`DELETE`](delete.html) -- [`TRUNCATE`](truncate.html) diff --git a/src/current/v21.1/bulk-update-data.md b/src/current/v21.1/bulk-update-data.md deleted file mode 100644 index fc268ede7c3..00000000000 --- a/src/current/v21.1/bulk-update-data.md +++ /dev/null @@ -1,114 +0,0 @@ ---- -title: Bulk-update Data -summary: How to to update a large amount of data using batch-update loops. -toc: true ---- - -To update multiple rows in a table, you can use a single [`UPDATE` statement](update.html), with a `WHERE` clause that filters the rows you want to update. - -To update a large number of rows (i.e., tens of thousands of rows or more), we recommend iteratively updating subsets of the rows that you want to update, until all of the rows have been updated. You can write a script to do this, or you can write a loop into your application. - -This page provides guidance on writing batch-update loops with a pattern that executes `SELECT` and `UPDATE` statements at different levels of a nested loop. - -{{site.data.alerts.callout_danger}} -Exercise caution when batch-updating rows from tables with foreign key constraints and explicit [`ON UPDATE` foreign key actions](foreign-key.html#foreign-key-actions). To preserve `UPDATE` performance on tables with foreign key actions, we recommend using smaller batch sizes, as additional rows updated due to `ON UPDATE` actions can make batch loops significantly slower. -{{site.data.alerts.end}} - -## Before you begin - -Before reading this page, do the following: - -- [Install CockroachDB](install-cockroachdb.html). -- [Start a local cluster](secure-a-cluster.html), or [create a CockroachDB {{ site.data.products.cloud }} cluster](../cockroachcloud/create-your-cluster.html). -- [Install a Postgres client](install-client-drivers.html). - - For the example on this page, we use the `psycopg2` Python driver. -- [Connect to the database](connect-to-the-database.html). -- [Insert data](insert-data.html) that you now want to update. - - For the example on this page, we load a cluster with the `movr` database and data from [`cockroach workload`](cockroach-workload.html). - -## Write a batch-update loop - -1. At the top level of a loop in your application, or in a script, execute a [`SELECT`](selection-queries.html) query that returns a large batch of primary key values for the rows that you want to update. When defining the `SELECT` query: - - Use a `WHERE` clause to filter on columns that identify the rows that you want to update. This clause should also filter out the rows that have been updated by previous iterations of the nested `UPDATE` loop: - - For optimal performance, the first condition of the filter should evaluate the last primary key value returned by the last `UPDATE` query that was executed. This narrows each `SELECT` query's scan to the fewest rows possible, and preserves the performance of the row updates over time. - - Another condition of the filter should evaluate column values persisted to the database that signal whether or not a row has been updated. This prevents rows from being updated more than once, in the event that the application or script crashes and needs to be restarted. If there is no way to distinguish between an updated row and a row that has not yet been updated, you might need to [add a new column to the table](add-column.html) (e.g., `ALTER TABLE ... ADD COLUMN updated BOOL;`). - - Add an [`AS OF SYSTEM TIME` clause](as-of-system-time.html) to the end of the selection subquery, or run the selection query in a separate, read-only transaction with [`SET TRANSACTION AS OF SYSTEM TIME`](as-of-system-time.html#using-as-of-system-time-in-transactions). This helps to reduce [transaction contention](transactions.html#transaction-contention). - - Use a [`LIMIT`](limit-offset.html) clause to limit the number of rows queried to a subset of the rows that you want to update. To determine the optimal `SELECT` batch size, try out different sizes (10,000 rows, 20,000 rows, etc.), and monitor the change in performance. Note that this `SELECT` batch size can be much larger than the batch size of rows that are updated in the subsequent `UPDATE` query. - - To ensure that rows are efficiently scanned in the subsequent `UPDATE` query, include an [`ORDER BY`](order-by.html) clause on the primary key. - -1. Under the `SELECT` query, write a nested loop that executes `UPDATE` queries over the primary key values returned by the `SELECT` query, in batches smaller than the initial `SELECT` batch size. When defining the `UPDATE` query: - - Use a `WHERE` clause that filters on a subset of the primary key values returned by the top-level `SELECT` query. To determine the optimal `UPDATE` batch size, try out different sizes (1,000 rows, 2,000 rows, etc.), and monitor the change in performance. - - Make sure that the `UPDATE` query updates a column that signals whether or not the row has been updated. This column might be different from the column whose values you want to update. - - Add a `RETURNING` clause to the end of the query that returns the primary key values of the rows being updated. The `WHERE` clause in the top-level `SELECT` query should filter out the primary key value of the last row that was updated, using the values returned by the last `UPDATE` query executed. - - Where possible, we recommend executing each `UPDATE` in a separate transaction. - -## Example - -Suppose that over the past year, you've recorded hundreds of thousands of [MovR](movr.html) rides in a cluster loaded with the [`movr`](cockroach-workload.html) database. And suppose that, for the last week of December, you applied a 10% discount to all ride charges billed to users, but you didn't update the `rides` table to reflect the discounts. - -To get the `rides` table up-to-date, you can create a loop that updates the relevant rows of the `rides` table in batches, following the query guidance provided [above](#write-a-batch-update-loop). - -In this case, you will also need to add a new column to the `rides` table that signals whether or not a row has been updated. Using this column, the top-level `SELECT` query can filter out rows that have already been updated, which will prevent rows from being updated more than once if the script crashes. - -For example, you could create a column named `discounted`, of data type [`BOOL`](bool.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER TABLE rides ADD COLUMN discounted BOOL DEFAULT false; -~~~ - -{% include {{ page.version.version }}/misc/schema-change-stmt-note.md %} - -In Python, a batch-update script might look similar to the following: - -{% include_cached copy-clipboard.html %} -~~~ python -#!/usr/bin/env python3 - -import psycopg2 -import os -import time - -def main(): - conn = psycopg2.connect(os.environ.get('DB_URI')) - lastid = None - - while True: - with conn: - with conn.cursor() as cur: - cur.execute("SET TRANSACTION AS OF SYSTEM TIME '-5s'") - if lastid: - cur.execute("SELECT id FROM rides WHERE id > %s AND discounted != true AND extract('month', start_time) = 12 AND extract('day', start_time) > 23 ORDER BY id LIMIT 10000", (lastid,)) - else: - cur.execute("SELECT id FROM rides WHERE discounted != true AND extract('month', start_time) = 12 AND extract('day', start_time) > 23 ORDER BY id LIMIT 10000") - pkvals = list(cur) - if not pkvals: - return - while pkvals: - batch = pkvals[:2000] - pkvals = pkvals[2000:] - with conn: - with conn.cursor() as cur: - cur.execute("UPDATE rides SET discounted = true, revenue = revenue*.9 WHERE id = ANY %s RETURNING id", (batch,)) - print(cur.statusmessage) - if not pkvals: - lastid = cur.fetchone()[0] - del batch - del pkvals - time.sleep(5) - - conn.close() -if __name__ == '__main__': - main() -~~~ - -At each iteration, the `SELECT` query returns the primary key values of up to 10,000 rows of matching historical data from 5 seconds in the past, in a read-only transaction. Then, a nested loop iterates over the returned primary key values in smaller batches of 2,000 rows. At each iteration of the nested `UPDATE` loop, a batch of rows is updated. After the nested `UPDATE` loop updates all of the rows from the initial selection query, a time delay ensures that the next selection query reads historical data from the table after the last iteration's `UPDATE` final update. - -Note that the last iteration of the nested loop assigns the primary key value of the last row updated to the `lastid` variable. The next `SELECT` query uses this variable to decrease the number of rows scanned by the number of rows updated in the last iteration of the loop. - -## See also - -- [Update data](update-data.html) -- [`UPDATE`](update.html) diff --git a/src/current/v21.1/bytes.md b/src/current/v21.1/bytes.md deleted file mode 100644 index 57dc32e80ba..00000000000 --- a/src/current/v21.1/bytes.md +++ /dev/null @@ -1,130 +0,0 @@ ---- -title: BYTES -summary: The BYTES data type stores binary strings of variable length. -toc: true ---- - -The `BYTES` [data type](data-types.html) stores binary strings of variable length. - - -## Aliases - -In CockroachDB, the following are aliases for `BYTES`: - -- `BYTEA` -- `BLOB` - -## Syntax - -To express a byte array constant, see the section on -[byte array literals](sql-constants.html#byte-array-literals) for more -details. For example, the following three are equivalent literals for the same -byte array: `b'abc'`, `b'\141\142\143'`, `b'\x61\x62\x63'`. - -In addition to this syntax, CockroachDB also supports using -[string literals](sql-constants.html#string-literals), including the -syntax `'...'`, `e'...'` and `x'....'` in contexts where a byte array -is otherwise expected. - -## Size - -The size of a `BYTES` value is variable, but it's recommended to keep values under 1 MB to ensure performance. Above that threshold, [write amplification](https://en.wikipedia.org/wiki/Write_amplification) and other considerations may cause significant performance degradation. - -{{site.data.alerts.callout_success}} -If your application requires large binary input in single queries, you can store the blobs somewhere your client can access them (using a cloud storage service, for example), and then reference their addresses from a statement. -{{site.data.alerts.end}} - -## Example - -~~~ sql -> CREATE TABLE bytes (a INT PRIMARY KEY, b BYTES); - -> -- explicitly typed BYTES literals -> INSERT INTO bytes VALUES (1, b'\141\142\143'), (2, b'\x61\x62\x63'), (3, b'\141\x62\c'); - -> -- string literal implicitly typed as BYTES -> INSERT INTO bytes VALUES (4, 'abc'); - - -> SELECT * FROM bytes; -~~~ -~~~ -+---+-----+ -| a | b | -+---+-----+ -| 1 | abc | -| 2 | abc | -| 3 | abc | -| 4 | abc | -+---+-----+ -(4 rows) -~~~ - -## Supported conversions - -`BYTES` values can be -[cast](data-types.html#data-type-conversions-and-casts) explicitly to -[`STRING`](string.html). This conversion always succeeds. Two -conversion modes are supported, controlled by the -[session variable](set-vars.html#supported-variables) `bytea_output`: - -- `hex` (default): The output of the conversion starts with the two - characters `\`, `x` and the rest of the string is composed by the - hexadecimal encoding of each byte in the input. For example, - `x'48AA'::STRING` produces `'\x48AA'`. - -- `escape`: The output of the conversion contains each byte in the - input, as-is if it is an ASCII character, or encoded using the octal - escape format `\NNN` otherwise. For example, `x'48AA'::STRING` - produces `'0\252'`. - -`STRING` values can be cast explicitly to `BYTES`. This conversion -will fail if the hexadecimal digits are not valid, or if there is an -odd number of them. Two conversion modes are supported: - -- If the string starts with the two special characters `\` and `x` - (e.g., `\xAABB`), the rest of the string is interpreted as a sequence - of hexadecimal digits. The string is then converted to a byte array - where each pair of hexadecimal digits is converted to one byte. - -- Otherwise, the string is converted to a byte array that contains its - UTF-8 encoding. - -### `STRING` vs. `BYTES` - -While both `STRING` and `BYTES` can appear to have similar behavior in many situations, one should understand their nuance before casting one into the other. - -`STRING` treats all of its data as characters, or more specifically, Unicode code points. `BYTES` treats all of its data as a byte string. This difference in implementation can lead to dramatically different behavior. For example, let's take a complex Unicode character such as ☃ ([the snowman emoji](https://emojipedia.org/snowman/)): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT length('☃'::string); -~~~ - -~~~ - length -+--------+ - 1 -~~~ - -~~~ sql -> SELECT length('☃'::bytes); -~~~ -~~~ - length -+--------+ - 3 -~~~ - -In this case, [`LENGTH(string)`](functions-and-operators.html#string-and-byte-functions) measures the number of Unicode code points present in the string, whereas [`LENGTH(bytes)`](functions-and-operators.html#string-and-byte-functions) measures the number of bytes required to store that value. Each character (or Unicode code point) can be encoded using multiple bytes, hence the difference in output between the two. - -#### Translating literals to `STRING` vs. `BYTES` - -A literal entered through a SQL client will be translated into a different value based on the type: - -+ `BYTES` give a special meaning to the pair `\x` at the beginning, and translates the rest by substituting pairs of hexadecimal digits to a single byte. For example, `\xff` is equivalent to a single byte with the value of 255. For more information, see [SQL Constants: String literals with character escapes](sql-constants.html#string-literals-with-character-escapes). -+ `STRING` does not give a special meaning to `\x`, so all characters are treated as distinct Unicode code points. For example, `\xff` is treated as a `STRING` with length 4 (`\`, `x`, `f`, and `f`). - -## See also - -[Data Types](data-types.html) diff --git a/src/current/v21.1/cancel-job.md b/src/current/v21.1/cancel-job.md deleted file mode 100644 index 0e275e14589..00000000000 --- a/src/current/v21.1/cancel-job.md +++ /dev/null @@ -1,107 +0,0 @@ ---- -title: CANCEL JOB -summary: The CANCEL JOB statement stops long-running jobs such as imports, backups, and schema changes.such as imports, backups, and schema changes. -toc: true ---- - -The `CANCEL JOB` [statement](sql-statements.html) lets you stop long-running jobs, which include: - -- [`IMPORT`](import.html) jobs -- [`BACKUP`](backup.html) and [`RESTORE`](restore.html) jobs -- [User-created table statistics](create-statistics.html) jobs -- [Automatic table statistics](cost-based-optimizer.html#table-statistics) jobs -- [Changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html) -- [Scheduled backup](manage-a-backup-schedule.html) jobs -- [Schema change](online-schema-changes.html) jobs - -## Limitations - -- When an Enterprise [`RESTORE`](restore.html) is canceled, partially restored data is properly cleaned up. This can have a minor, temporary impact on cluster performance. - -## Required privileges - -To cancel a job, the user must be a member of the `admin` role or must have the [`CONTROLJOB`](create-user.html#create-a-user-that-can-pause-resume-and-cancel-non-admin-jobs) parameter set. Non-admin users cannot cancel admin users' jobs. - -## Synopsis - -
-{% include {{ page.version.version }}/sql/generated/diagrams/cancel_job.html %} -
- -## Parameters - -Parameter | Description -----------|------------ -`job_id` | The ID of the job you want to cancel, which can be found with [`SHOW JOBS`](show-jobs.html). -`select_stmt` | A [selection query](selection-queries.html) that returns `job_id`(s) to cancel. -`for_schedules_clause` | The schedule you want to cancel jobs for. You can cancel jobs for a specific schedule (`FOR SCHEDULE id`) or cancel jobs for multiple schedules by nesting a [`SELECT` clause](select-clause.html) in the statement (`FOR SCHEDULES `). See the [examples](#cancel-jobs-for-a-schedule) below. - -## Examples - -### Cancel a single job - -~~~ sql -> SHOW JOBS; -~~~ -~~~ -+----------------+---------+-------------------------------------------+... -| id | type | description |... -+----------------+---------+-------------------------------------------+... -| 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... -+----------------+---------+-------------------------------------------+... -~~~ -~~~ sql -> CANCEL JOB 27536791415282; -~~~ - -### Cancel multiple jobs - -To cancel multiple jobs, nest a [`SELECT` clause](select-clause.html) that retrieves `job_id`(s) inside the `CANCEL JOBS` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CANCEL JOBS (SELECT job_id FROM [SHOW JOBS] - WHERE user_name = 'maxroach'); -~~~ - -All jobs created by `maxroach` will be cancelled. - -### Cancel automatic table statistics jobs - -Canceling an automatic table statistics job is not useful since the system will automatically restart the job immediately. To permanently disable automatic table statistics jobs, disable the `sql.stats.automatic_collection.enabled` [cluster setting](cluster-settings.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET CLUSTER SETTING sql.stats.automatic_collection.enabled = false; -~~~ - -### Cancel jobs for a schedule - - To cancel jobs for a specific [backup schedule](create-schedule-for-backup.html), use the schedule's `id`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CANCEL JOBS FOR SCHEDULE 590204387299262465; -~~~ -~~~ -CANCEL JOBS FOR SCHEDULES 1 -~~~ - -You can also CANCEL multiple schedules by nesting a [`SELECT` clause](select-clause.html) that retrieves `id`(s) inside the `CANCEL JOBS` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CANCEL JOBS FOR SCHEDULES SELECT id FROM [SHOW SCHEDULES] WHERE label = 'test_schedule'; -~~~ - -~~~ -CANCEL JOBS FOR SCHEDULES 2 -~~~ - -## See also - -- [`SHOW JOBS`](show-jobs.html) -- [`BACKUP`](backup.html) -- [`RESTORE`](restore.html) -- [`IMPORT`](import.html) -- [`CREATE CHANGEFEED`](create-changefeed.html) diff --git a/src/current/v21.1/cancel-query.md b/src/current/v21.1/cancel-query.md deleted file mode 100644 index fdf0afaef3e..00000000000 --- a/src/current/v21.1/cancel-query.md +++ /dev/null @@ -1,83 +0,0 @@ ---- -title: CANCEL QUERY -summary: The CANCEL QUERY statement cancels a running SQL query. -toc: true ---- - -The `CANCEL QUERY` [statement](sql-statements.html) cancels a running SQL query. - - -## Considerations - -- Schema changes are treated differently than other SQL queries. You can use SHOW JOBS to monitor the progress of schema changes and CANCEL JOB to cancel schema changes that are taking longer than expected. -- In rare cases where a query is close to completion when a cancellation request is issued, the query may run to completion. - -## Required privileges - -Members of the `admin` role (include `root`, which belongs to `admin` by default) can cancel any currently active queries. User that are not members of the `admin` role can cancel only their own currently active queries. To view and cancel another non-admin user's query, the user must be a member of the `admin` role or must have the [`VIEWACTIVITY`](create-user.html#create-a-user-that-can-see-and-cancel-non-admin-queries-and-sessions) and [`CANCELQUERY`](create-user.html#create-a-user-that-can-see-and-cancel-non-admin-queries-and-sessions) parameters set. - -## Synopsis - -
-{% include {{ page.version.version }}/sql/generated/diagrams/cancel_query.html %} -
- -## Parameters - -Parameter | Description -----------|------------ -`query_id` | A [scalar expression](scalar-expressions.html) that produces the ID of the query to cancel.

`CANCEL QUERY` accepts a single query ID. If a subquery is used and returns multiple IDs, the `CANCEL QUERY` statement will fail. To cancel multiple queries, use `CANCEL QUERIES`. -`select_stmt` | A [selection query](selection-queries.html) whose result you want to cancel. - -## Response - -When a query is successfully cancelled, CockroachDB sends a `query execution canceled` error to the client that issued the query. - -- If the canceled query was a single, stand-alone statement, no further action is required by the client. -- If the canceled query was part of a larger, multi-statement [transaction](transactions.html), the client should then issue a [`ROLLBACK`](rollback-transaction.html) statement. - -## Examples - -### Cancel a query via the query ID - -In this example, we use the [`SHOW STATEMENTS`](show-statements.html) statement to get the ID of a query and then pass the ID into the `CANCEL QUERY` statement: - -~~~ sql -> SHOW STATEMENTS; -~~~ - -~~~ - query_id | node_id | session_id | user_name | start | query | client_address | application_name | distributed | phase ------------------------------------+---------+----------------------------------+-----------+-------------------------------------+--------------------------------------+-----------------+------------------+-------------+------------ - 1673f58fca5301900000000000000001 | 1 | 1673f583067d51280000000000000001 | demo | 2021-04-08 18:31:29.079614+00:00:00 | SELECT * FROM rides ORDER BY revenue | 127.0.0.1:55212 | $ cockroach demo | true | executing - 1673f590433eaa000000000000000001 | 1 | 1673f58a4ba3c8e80000000000000001 | demo | 2021-04-08 18:31:31.108372+00:00:00 | SHOW CLUSTER STATEMENTS | 127.0.0.1:55215 | $ cockroach sql | false | executing -(2 rows) -~~~ - -~~~ sql -> CANCEL QUERY '1673f590433eaa000000000000000001'; -~~~ - -### Cancel a query via a subquery - -In this example, we nest a [`SELECT` clause](select-clause.html) that retrieves the ID of a query inside the `CANCEL QUERY` statement: - -~~~ sql -> CANCEL QUERY (SELECT query_id FROM [SHOW CLUSTER STATEMENTS] - WHERE client_address = '127.0.0.1:55212' - AND user_name = 'demo' - AND query = 'SELECT * FROM rides ORDER BY revenue'); -~~~ - -~~~ -CANCEL QUERIES 1 -~~~ - -{{site.data.alerts.callout_info}}CANCEL QUERY accepts a single query ID. If a subquery is used and returns multiple IDs, the CANCEL QUERY statement will fail. To cancel multiple queries, use CANCEL QUERIES.{{site.data.alerts.end}} - -## See also - -- [Manage Long-Running Queries](manage-long-running-queries.html) -- [`SHOW STATEMENTS`](show-statements.html) -- [`CANCEL SESSION`](cancel-session.html) -- [SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/cancel-session.md b/src/current/v21.1/cancel-session.md deleted file mode 100644 index 58d2d6bbd01..00000000000 --- a/src/current/v21.1/cancel-session.md +++ /dev/null @@ -1,93 +0,0 @@ ---- -title: CANCEL SESSION -summary: The CANCEL SESSION statement stops long-running sessions. -toc: true ---- - -The `CANCEL SESSION` [statement](sql-statements.html) lets you stop long-running sessions. `CANCEL SESSION` will attempt to cancel the currently active query and end the session. - - -## Required privileges - -To view and cancel a session, the user must be a member of the `admin` role or must have the [`VIEWACTIVITY`](create-user.html#create-a-user-that-can-see-and-cancel-non-admin-queries-and-sessions) and [`CANCELQUERY`](create-user.html#create-a-user-that-can-see-and-cancel-non-admin-queries-and-sessions) parameters set. Non-admin users cannot cancel admin users' sessions. - -## Synopsis - -
{% include {{ page.version.version }}/sql/generated/diagrams/cancel_session.html %}
- -## Parameters - -Parameter | Description -----------|------------ -`session_id` | The ID of the session you want to cancel, which can be found with [`SHOW SESSIONS`](show-sessions.html).

`CANCEL SESSION` accepts a single session ID. If a subquery is used and returns multiple IDs, the `CANCEL SESSION` statement will fail. To cancel multiple sessions, use `CANCEL SESSIONS`. -`select_stmt` | A [selection query](selection-queries.html) that returns `session_id`(s) to cancel. - -## Example - -### Cancel a single session - -In this example, we use the [`SHOW SESSIONS`](show-sessions.html) statement to get the ID of a session and then pass the ID into the `CANCEL SESSION` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SESSIONS; -~~~ -~~~ -+---------+----------------------------------+-----------+... -| node_id | session_id | user_name |... -+---------+----------------------------------+-----------+... -| 1 | 1530c309b1d8d5f00000000000000001 | root |... -+---------+----------------------------------+-----------+... -| 1 | 1530fe0e46d2692e0000000000000001 | maxroach |... -+---------+----------------------------------+-----------+... -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CANCEL SESSION '1530fe0e46d2692e0000000000000001'; -~~~ - -You can also cancel a session using a subquery that returns a single session ID: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CANCEL SESSIONS (SELECT session_id FROM [SHOW SESSIONS] - WHERE user_name = 'root'); -~~~ - -### Cancel multiple sessions - -Use the [`SHOW SESSIONS`](show-sessions.html) statement to view all active sessions: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SESSIONS; -~~~ -~~~ -+---------+----------------------------------+-----------+... -| node_id | session_id | user_name |... -+---------+----------------------------------+-----------+... -| 1 | 1530c309b1d8d5f00000000000000001 | root |... -+---------+----------------------------------+-----------+... -| 1 | 1530fe0e46d2692e0000000000000001 | maxroach |... -+---------+----------------------------------+-----------+... -| 1 | 15310cc79671fc6a0000000000000001 | maxroach |... -+---------+----------------------------------+-----------+... -~~~ - -To cancel multiple sessions, nest a [`SELECT` clause](select-clause.html) that retrieves `session_id`(s) inside the `CANCEL SESSIONS` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CANCEL SESSIONS (SELECT session_id FROM [SHOW SESSIONS] - WHERE user_name = 'maxroach'); -~~~ - -All sessions created by `maxroach` will be cancelled. - -## See also - -- [`SHOW SESSIONS`](show-sessions.html) -- [`SET` (session variable)](set-vars.html) -- [`SHOW` (session variable)](show-vars.html) -- [SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/changefeed-for.md b/src/current/v21.1/changefeed-for.md deleted file mode 100644 index 81c8a6a2cb5..00000000000 --- a/src/current/v21.1/changefeed-for.md +++ /dev/null @@ -1,86 +0,0 @@ ---- -title: EXPERIMENTAL CHANGEFEED FOR -summary: which streams row-level changes to the client indefinitely until the underlying connection is closed or the changefeed is canceled. -toc: true ---- - -{{site.data.alerts.callout_info}} -`EXPERIMENTAL CHANGEFEED FOR` is the core implementation of changefeeds. For the [Enterprise-only](enterprise-licensing.html) version, see [`CREATE CHANGEFEED`](create-changefeed.html). -{{site.data.alerts.end}} - -The `EXPERIMENTAL CHANGEFEED FOR` [statement](sql-statements.html) creates a new core changefeed, which streams row-level changes to the client indefinitely until the underlying connection is closed or the changefeed is canceled. A core changefeed can watch one table or multiple tables in a comma-separated list. - -{% include {{ page.version.version }}/cdc/core-url.md %} - -For more information, see [Stream Data Out of CockroachDB Using Changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html). - -{% include {{ page.version.version }}/misc/experimental-warning.md %} - -## Required privileges - -Changefeeds can only be created by superusers, i.e., [members of the `admin` role](authorization.html#create-and-manage-roles). The admin role exists by default with `root` as the member. - -## Considerations - -- Because core changefeeds return results differently than other SQL statements, they require a dedicated database connection with specific settings around result buffering. In normal operation, CockroachDB improves performance by buffering results server-side before returning them to a client; however, result buffering is automatically turned off for core changefeeds. Core changefeeds also have different cancellation behavior than other queries: they can only be canceled by closing the underlying connection or issuing a [`CANCEL QUERY`](cancel-query.html) statement on a separate connection. Combined, these attributes of changefeeds mean that applications should explicitly create dedicated connections to consume changefeed data, instead of using a connection pool as most client drivers do by default. - - This cancellation behavior (i.e., close the underlying connection to cancel the changefeed) also extends to client driver usage; in particular, when a client driver calls `Rows.Close()` after encountering errors for a stream of rows. The pgwire protocol requires that the rows be consumed before the connection is again usable, but in the case of a core changefeed, the rows are never consumed. It is therefore critical that you close the connection, otherwise the application will be blocked forever on `Rows.Close()`. - -- In most cases, each version of a row will be emitted once. However, some infrequent conditions (e.g., node failures, network partitions) will cause them to be repeated. This gives our changefeeds an at-least-once delivery guarantee. For more information, see [Change Data Capture - Ordering Guarantees](stream-data-out-of-cockroachdb-using-changefeeds.html#ordering-guarantees). - -## Synopsis - -~~~ -> EXPERIMENTAL CHANGEFEED FOR table_name [ WITH (option [= value] [, ...]) ]; -~~~ - -## Parameters - -Parameter | Description -----------|------------ -`table_name` | The name of the table (or tables in a comma separated list) to create a changefeed for. -`option` / `value` | For a list of available options and their values, see [Options](#options) below. - - - -### Options - -Option | Value | Description --------|-------|------------ -`updated` | N/A | Include updated timestamps with each row. -`resolved` | [`INTERVAL`](interval.html) | Emits [resolved timestamp](stream-data-out-of-cockroachdb-using-changefeeds.html#resolved-def) events for the changefeed. Resolved timestamp events do not emit until all ranges in the changefeed have progressed to a specific point in time.

Set an optional minimal duration between emitting resolved timestamps. Example: `resolved='10s'`. This option will **only** emit a resolved timestamp event if the timestamp has advanced and at least the optional duration has elapsed. If unspecified, all resolved timestamps are emitted as the high-water mark advances. -`envelope` | `key_only` / `row` | Use `key_only` to emit only the key and no value, which is faster if you only want to know when the key changes.

Default: `envelope=row` -`cursor` | [Timestamp](as-of-system-time.html#parameters) | Emits any changes after the given timestamp, but does not output the current state of the table first. If `cursor` is not specified, the changefeed starts by doing a consistent scan of all the watched rows and emits the current value, then moves to emitting any changes that happen after the scan.

`cursor` can be used to start a new changefeed where a previous changefeed ended.

Example: `CURSOR=1536242855577149065.0000000000` -`format` | `json` / `experimental_avro` | Format of the emitted record. Currently, support for [Avro is limited and experimental](#avro-limitations).

Default: `format=json`. -`confluent_schema_registry` | Schema Registry address | The [Schema Registry](https://docs.confluent.io/current/schema-registry/docs/index.html#sr) address is required to use `experimental_avro`. - -#### Avro limitations - -Currently, support for Avro is limited and experimental. - -Below are clarifications for particular SQL types and values for Avro changefeeds: - - - [Decimals](decimal.html) must have precision specified. - - [`BIT`](bit.html) and [`VARBIT`](bit.html) types are encoded as arrays of 64-bit integers. - - {% include {{ page.version.version }}/cdc/avro-bit-varbit.md %} - -## Examples - -### Create a changefeed - -{% include {{ page.version.version }}/cdc/create-core-changefeed.md %} - -### Create a changefeed with Avro - -{% include {{ page.version.version }}/cdc/create-core-changefeed-avro.md %} - - - -## See also - -- [Stream Data Out of CockroachDB Using Changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/check.md b/src/current/v21.1/check.md deleted file mode 100644 index f820cca34b2..00000000000 --- a/src/current/v21.1/check.md +++ /dev/null @@ -1,113 +0,0 @@ ---- -title: CHECK Constraint -summary: The CHECK constraint specifies that values for the column in INSERT or UPDATE statements must satisfy a Boolean expression. -toc: true ---- - -The `CHECK` [constraint](constraints.html) specifies that values for the column in [`INSERT`](insert.html) or [`UPDATE`](update.html) statements must return `TRUE` or `NULL` for a Boolean expression. If any values return `FALSE`, the entire statement is rejected. - -## Details - -- If you add a `CHECK` constraint to an existing table, CockroachDB will run a background job to validate existing table data in the process of adding the constraint. If a row is found that violates the constraint during the validation step, the [`ADD CONSTRAINT`](add-constraint.html) statement will fail. This differs from previous versions of CockroachDB, which allowed you to add a check constraint that was enforced for writes but could be violated by rows that existed prior to adding the constraint. -- Check constraints can be added to columns that were created earlier in the same transaction. For an example, see [Add the `CHECK` constraint](add-constraint.html#add-the-check-constraint). -- `CHECK` constraints may be specified at the column or table level and can reference other columns within the table. Internally, all column-level `CHECK` constraints are converted to table-level constraints so they can be handled consistently. -- You can have multiple `CHECK` constraints on a single column but ideally, for performance optimization, these should be combined using the logical operators. For example: - - ~~~ sql - warranty_period INT CHECK (warranty_period >= 0) CHECK (warranty_period <= 24) - ~~~ - - should be specified as: - - ~~~ sql - warranty_period INT CHECK (warranty_period BETWEEN 0 AND 24) - ~~~ -- When a column with a `CHECK` constraint is dropped, the `CHECK` constraint is also dropped. - -## Syntax - -`CHECK` constraints can be defined at the [table level](#table-level). However, if you only want the constraint to apply to a single column, it can be applied at the [column level](#column-level). - -{{site.data.alerts.callout_info}}You can also add the CHECK constraint to existing tables through ADD CONSTRAINT.{{site.data.alerts.end}} - -### Column level - -
-{% include {{ page.version.version }}/sql/generated/diagrams/check_column_level.html %} -
- - Parameter | Description ------------|------------- - `table_name` | The name of the table you're creating. - `column_name` | The name of the constrained column. - `column_type` | The constrained column's [data type](data-types.html). - `check_expr` | An expression that returns a Boolean value; if the expression evaluates to `FALSE`, the value cannot be inserted. - `column_constraints` | Any other column-level [constraints](constraints.html) you want to apply to this column. - `column_def` | Definitions for any other columns in the table. - `table_constraints` | Any table-level [constraints](constraints.html) you want to apply. - -**Example** - -~~~ sql -> CREATE TABLE inventories ( - product_id INT NOT NULL, - warehouse_id INT NOT NULL, - quantity_on_hand INT NOT NULL CHECK (quantity_on_hand > 0), - PRIMARY KEY (product_id, warehouse_id) - ); -~~~ - -### Table level - -
-{% include {{ page.version.version }}/sql/generated/diagrams/check_table_level.html %} -
- - Parameter | Description ------------|------------- - `table_name` | The name of the table you're creating. - `column_def` | Definitions for any other columns in the table. - `name` | The name you want to use for the constraint, which must be unique to its table and follow these [identifier rules](keywords-and-identifiers.html#identifiers). - `check_expr` | An expression that returns a Boolean value; if the expression evaluates to `FALSE`, the value cannot be inserted. - `table_constraints` | Any other table-level [constraints](constraints.html) you want to apply. - -**Example** - -~~~ sql -> CREATE TABLE inventories ( - product_id INT NOT NULL, - warehouse_id INT NOT NULL, - quantity_on_hand INT NOT NULL, - PRIMARY KEY (product_id, warehouse_id), - CONSTRAINT ok_to_supply CHECK (quantity_on_hand > 0 AND warehouse_id BETWEEN 100 AND 200) - ); -~~~ - -## Usage example - -`CHECK` constraints may be specified at the column or table level and can reference other columns within the table. Internally, all column-level `CHECK` constraints are converted to table-level constraints so they can be handled in a consistent fashion. - -~~~ sql -> CREATE TABLE inventories ( - product_id INT NOT NULL, - warehouse_id INT NOT NULL, - quantity_on_hand INT NOT NULL CHECK (quantity_on_hand > 0), - PRIMARY KEY (product_id, warehouse_id) - ); - -> INSERT INTO inventories (product_id, warehouse_id, quantity_on_hand) VALUES (1, 2, 0); -~~~ -~~~ -pq: failed to satisfy CHECK constraint (quantity_on_hand > 0) -~~~ - -## See also - -- [Constraints](constraints.html) -- [`DROP CONSTRAINT`](drop-constraint.html) -- [`DEFAULT` constraint](default-value.html) -- [`REFERENCES` constraint (Foreign Key)](foreign-key.html) -- [`NOT NULL` constraint](not-null.html) -- [`PRIMARY KEY` constraint](primary-key.html) -- [`UNIQUE` constraint](unique.html) -- [`SHOW CONSTRAINTS`](show-constraints.html) diff --git a/src/current/v21.1/choosing-a-multi-region-configuration.md b/src/current/v21.1/choosing-a-multi-region-configuration.md deleted file mode 100644 index bdad05bf4ff..00000000000 --- a/src/current/v21.1/choosing-a-multi-region-configuration.md +++ /dev/null @@ -1,63 +0,0 @@ ---- -title: Choosing a multi-region configuration -summary: Learn how to use CockroachDB's improved multi-region user experience. -toc: true ---- - -This page has high-level information about how to configure a [multi-region cluster's](multiregion-overview.html) [survival goals](multiregion-overview.html#survival-goals) and [table locality](multiregion-overview.html#table-locality). - -{% include enterprise-feature.md %} - -The options for configuring your multi-region cluster include: - -- _Change nothing_: Using the [default settings](multiregion-overview.html#default-settings), you get: - - Zone survival (the default). - - Low-latency reads and writes from a single region. - - A choice of low-latency stale reads or high-latency fresh reads from other regions (and high-latency fresh reads is the default). - -- _Change only [survival goals](multiregion-overview.html#survival-goals)_: This configuration is useful for single-region apps that need higher levels of survival. In this configuration, you move from availability zone (AZ) survival to get: - - Region survival. - - Low-latency reads from a single region. - - A choice of low-latency stale reads or high-latency fresh reads from other regions (and high-latency fresh reads is the default). - - Higher-latency writes from all regions (due to region survival). - -- _Change only [table locality](multiregion-overview.html#table-locality)_: This is useful for multi-region apps that require different read/write latency guarantees for different tables in the database, and are not concerned with surviving a region failure. In this configuration, you get: - - Zone survival (the default). - - For [global tables](multiregion-overview.html#global-tables), low-latency reads from all regions. - - For [regional by row tables](multiregion-overview.html#regional-by-row-tables), low-latency reads and writes from each row's [home region](set-locality.html#crdb_region), and low-latency [follower reads](follower-reads.html) from all other regions. - -- _Change both [survival goals](multiregion-overview.html#survival-goals) and [table locality](multiregion-overview.html#table-locality)_: This is useful for multi-region apps that want a high level of survival. In this configuration, you move from zone survival and get: - - Region survival. - - Low-latency reads from all regions. - - Higher-latency writes from all regions (due to region survival). - -The table below offers another view of how the configuration options described above map to: - -- The performance characteristics of specific survival goal/table locality combinations. -- The types of applications that can benefit from each combination. - -|
locality ↓ survival → | `ZONE` | `REGION` | -|---------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `REGIONAL BY TABLE` | Low-latency for single-region writes and multi-region stale reads. | Single-region writes are higher latency than for ZONE, as at least one additional region must be consulted for each write. Stale multi-region reads are of comparable latency to ZONE survival. | -| | For single-region apps that can accept risk of region failure. | For single-region apps that must survive region failure. | -| `REGIONAL BY ROW` | Low-latency consistent multi-region reads & writes for rows which are homed in specific regions. | Low-latency consistent reads from a row's home region. Low-latency consistent [stale reads](follower-reads.html) from outside the row's home region. Low-latency writes if you are writing to a row from its home region. | -| | For multi-region apps which read/write individual rows of the table from a specific region, and can accept the risk of region failure. | Same as for ZONE survival, but for apps that must survive a region failure. | -| `GLOBAL` | Low-latency multi-region reads. Writes are higher latency than reads. | Low-latency multi-region reads. Writes are higher latency than reads. There should be minimal difference in write latencies between ZONE and REGION survival due to a new custom non-blocking transaction protocol. | -| | For multi-region apps that need low-latency reads of a "read-mostly" table. | For multi-region apps that need low-latency reads of a "read-mostly" table, and the ability to survive region failures. | - -{{site.data.alerts.callout_info}} -Different databases and tables within the same cluster can each use different combinations of the settings above. -{{site.data.alerts.end}} - -{{site.data.alerts.callout_success}} -{% include {{page.version.version}}/misc/multiregion-max-offset.md %} -{{site.data.alerts.end}} - -## See also - -- [Multi-region overview](multiregion-overview.html) -- [When to use `REGIONAL` vs. `GLOBAL` tables](when-to-use-regional-vs-global-tables.html) -- [When to use `ZONE` vs. `REGION` survival goals](when-to-use-zone-vs-region-survival-goals.html) -- [Topology Patterns](topology-patterns.html) -- [Disaster Recovery](disaster-recovery.html) -- [Multi-region SQL performance](demo-low-latency-multi-region-deployment.html) diff --git a/src/current/v21.1/cluster-api.md b/src/current/v21.1/cluster-api.md deleted file mode 100644 index 3e143209d6d..00000000000 --- a/src/current/v21.1/cluster-api.md +++ /dev/null @@ -1,72 +0,0 @@ ---- -title: Cluster API v2.0 -summary: Programmatically access and monitor cluster and node status information with a RESTful API. -toc: true ---- - -The CockroachDB Cluster API is a REST API that provides information about a cluster and its nodes. The API offers programmatic access to much of the information available in the [DB Console](ui-overview.html) user interface, enabling you to monitor and troubleshoot your cluster using your choice of tooling. - -The Cluster API is hosted by all nodes of your cluster and provides information about all nodes. The API is available on the same port that is listening for HTTP connections to the DB Console. This defaults to `8080` and can be specified using `--http-addr={server}:{port}` when configuring your node. - -## Resources - -The following endpoints are available as URLs under the `/api/v2` base path (for example, `https://localhost:8080/api/v2/health/`). Additional endpoints are planned for future versions of CockroachDB. - -Each listed endpoint links to its full [API reference documentation](https://cockroachlabs.com/docs/api/cluster/v2.html). - -Endpoint | Name | Description ---- | --- | --- -[`/health`](https://cockroachlabs.com/docs/api/cluster/v2.html#operation/health) | Check node health | Determine if the node is running and ready to accept SQL connections. -[`/nodes`](https://cockroachlabs.com/docs/api/cluster/v2.html#operation/listNodes) | List nodes | Get details on all nodes in the cluster, including node IDs, software versions, and hardware. -[`/nodes/{node_id}/ranges`](https://cockroachlabs.com/docs/api/cluster/v2.html#operation/listNodeRanges) | List node ranges | For a specified node, get details on the ranges that it hosts. -[`/ranges/hot`](https://cockroachlabs.com/docs/api/cluster/v2.html#operation/listHotRanges) | List hot ranges | Get information on ranges receiving a high number of reads or writes. -[`/ranges/{range_id}`](https://cockroachlabs.com/docs/api/cluster/v2.html#operation/listRange) | Get range details | Get detailed technical information on a range. Typically used by Cockroach Labs engineers. -[`/sessions`](https://cockroachlabs.com/docs/api/cluster/v2.html#operation/listSessions) | List sessions | Get SQL session details of all current users or a specified user. -[`/login`](https://cockroachlabs.com/docs/api/cluster/v2.html#operation/login) | Log in | Authenticate as a [SQL role](create-role.html#create-a-role-that-can-log-in-to-the-database) that is a member of the [admin role](authorization.html#admin-role) to retrieve a session token to use with further API calls. -[`/logout`](https://cockroachlabs.com/docs/api/cluster/v2.html#operation/logout) | Log out | Invalidate the session token. - -## Requirements - -All endpoints except `/health` and `/login` require authentication using a session token. To obtain a session token, you will need: - -* A [SQL role](create-role.html) that is a member of the [`admin` role](authorization.html#admin-role) and has login permissions and a password. - -To connect with the API on a secure cluster, you will need: - -* The CA cert used by the cluster or any intermediary proxy server, either in the client's cert store as a trusted certificate authority or as a file manually specified by the HTTP request (for example, using curl's [cacert](https://curl.se/docs/manpage.html#--cacert)). - -## Authentication - -To create and manage web sessions and authentication tokens to the Cluster API from the command line, use the [`cockroach auth-session`](cockroach-auth-session.html) CLI command. - -Alternatively, you may also request a token directly from the `/login` endpoint using the following instructions: - -1. Request a session token using the `/login` endpoint. For example: - - {% include_cached copy-clipboard.html %} - ~~~ shell - curl -d "username=user&password=pass" \ - -H 'Content-Type: application/x-www-form-urlencoded' \ - https://localhost:8080/api/v2/login/ - ~~~ - -2. Record the token (`session` value) that is returned. - - {% include_cached copy-clipboard.html %} - ~~~ shell - {"session":"CIGAiPis4fj3CBIQ3u0rRQJ3tD8yIqee4hipow=="} - ~~~ - -3. Pass the token with each call using the `X-Cockroach-API-Session` header. For example: - - {% include_cached copy-clipboard.html %} - ~~~ shell - curl -H "X-Cockroach-API-Session: CIGAiPis4fj3CBIQ3u0rRQJ3tD8yIqee4hipow==" \ - https://localhost:8080/api/v2/nodes/ - ~~~ - -## Versioning and Stability - -Future versions of CockroachDB may provide multiple API versions and will continue to provide access to this v2.0 API until it is deprecated. - -All endpoint paths and payloads will remain available within a major API version number (v2.x). Minor versions could add new endpoints but will not remove existing endpoints. diff --git a/src/current/v21.1/cluster-settings.md b/src/current/v21.1/cluster-settings.md deleted file mode 100644 index 5aa97c2ef54..00000000000 --- a/src/current/v21.1/cluster-settings.md +++ /dev/null @@ -1,43 +0,0 @@ ---- -title: Cluster Settings -summary: Learn about cluster settings that apply to all nodes of a CockroachDB cluster. -toc: false ---- - -Cluster settings apply to all nodes of a CockroachDB cluster and control, for example, whether or not to share diagnostic details with Cockroach Labs as well as advanced options for debugging and cluster tuning. - -They can be updated anytime after a cluster has been started, but only by a member of the `admin` role, to which the `root` user belongs by default. - -{{site.data.alerts.callout_info}} -In contrast to cluster-wide settings, node-level settings apply to a single node. They are defined by flags passed to the `cockroach start` command when starting a node and cannot be changed without stopping and restarting the node. For more details, see [Start a Node](cockroach-start.html). -{{site.data.alerts.end}} - -## Settings - -{{site.data.alerts.callout_danger}} -Many cluster settings are intended for tuning CockroachDB internals. Before changing these settings, we strongly encourage you to discuss your goals with Cockroach Labs; otherwise, you use them at your own risk. -{{site.data.alerts.end}} - -{% remote_include https://raw.githubusercontent.com/cockroachdb/cockroach/{{ page.release_info.crdb_branch_name }}/docs/generated/settings/settings.html %} - -## View current cluster settings - -Use the [`SHOW CLUSTER SETTING`](show-cluster-setting.html) statement. - -## Change a cluster setting - -Use the [`SET CLUSTER SETTING`](set-cluster-setting.html) statement. - -Before changing a cluster setting, please note the following: - -- Changing a cluster setting is not instantaneous, as the change must be propagated to other nodes in the cluster. - -- Do not change cluster settings while [upgrading to a new version of CockroachDB](upgrade-cockroach-version.html). Wait until all nodes have been upgraded before you make the change. - -## See also - -- [`SET CLUSTER SETTING`](set-cluster-setting.html) -- [`SHOW CLUSTER SETTING`](show-cluster-setting.html) -- [Diagnostics Reporting](diagnostics-reporting.html) -- [Start a Node](cockroach-start.html) -- [Use the Built-in SQL Client](cockroach-sql.html) diff --git a/src/current/v21.1/cluster-setup-troubleshooting.md b/src/current/v21.1/cluster-setup-troubleshooting.md deleted file mode 100644 index d151028ddb7..00000000000 --- a/src/current/v21.1/cluster-setup-troubleshooting.md +++ /dev/null @@ -1,560 +0,0 @@ ---- -title: Troubleshoot Cluster Setup -summary: Learn how to troubleshoot issues with starting CockroachDB clusters -toc: true ---- - -If you're having trouble starting or scaling your cluster, this page will help you troubleshoot the issue. - -To use this guide, it's important to understand some of CockroachDB's terminology: - - - A **Cluster** acts as a single logical database, but is actually made up of many cooperating nodes. - - **Nodes** are single instances of the `cockroach` binary running on a machine. It's possible (though atypical) to have multiple nodes running on a single machine. - -## Cannot run a single-node CockroachDB cluster - -Try running: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start-single-node --insecure --logtostderr -~~~ - -If the process exits prematurely, check for the following: - -#### An existing storage directory - -When starting a node, the directory you choose to store the data in also contains metadata identifying the cluster the data came from. This causes conflicts when you've already started a node on the server, have quit `cockroach`, and then tried to start another cluster using the same directory. Because the existing directory's cluster ID doesn't match the new cluster ID, the node cannot start. - -**Solution:** Disassociate the node from the existing directory where you've stored CockroachDB data. For example, you can do either of the following: - -- Choose a different directory to store the CockroachDB data: - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start-single-node --store= --insecure - ~~~ -- Remove the existing directory and start the node again: - {% include_cached copy-clipboard.html %} - ~~~ shell - $ rm -r cockroach-data/ - ~~~ - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start-single-node --insecure --logtostderr - ~~~ - -#### Toolchain incompatibility - -The components of the toolchain might have some incompatibilities that need to be resolved. For example, a few months ago, there was an incompatibility between Xcode 8.3 and Go 1.8 that caused any Go binaries created with that toolchain combination to crash immediately. - -#### Incompatible CPU - -If the `cockroach` process had exit status `132 (SIGILL)`, it attempted to use an instruction that is not supported by your CPU. Non-release builds of CockroachDB may not be able to run on older hardware platforms than the one used to build them. Release builds should run on any x86-64 CPU. - -#### Default ports already in use - -Other services may be running on port 26257 or 8080 (CockroachDB's default `--listen-addr` port and `--http-addr` port respectively). You can either stop those services or start your node with different ports, specified in the [`--listen-addr` and `--http-addr` flags](cockroach-start.html#networking). - - If you change the port, you will need to include the `--port=` flag in each subsequent cockroach command or change the `COCKROACH_PORT` environment variable. - -#### Single-node networking issues - -Networking issues might prevent the node from communicating with itself on its hostname. You can control the hostname CockroachDB uses with the [`--listen-addr` flag](cockroach-start.html#networking). - - If you change the host, you will need to include `--host=` in each subsequent cockroach command. - -#### CockroachDB process hangs when trying to start a node in the background - -See [Why is my process hanging when I try to start it in the background?](operational-faqs.html#why-is-my-process-hanging-when-i-try-to-start-nodes-with-the-background-flag) - -## Cannot run SQL statements using built-in SQL client - -If the CockroachDB node appeared to [start successfully](start-a-local-cluster.html), in a separate terminal run: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --insecure -e "show databases" -~~~ - -You should see a list of the built-in databases: - -~~~ - database_name -+---------------+ - defaultdb - postgres - system -(3 rows) -~~~ - -If you’re not seeing the output above, check for the following: - -- `connection refused` error, which indicates you have not included some flag that you used to start the node. We have additional troubleshooting steps for this error [here](common-errors.html#connection-refused). -- The node crashed. To ascertain if the node crashed, run `ps | grep cockroach` to look for the `cockroach` process. If you cannot locate the `cockroach` process (i.e., it crashed), [file an issue](file-an-issue.html), including the logs from your node and any errors you received. - -## Cannot run a multi-node CockroachDB cluster on the same machine - -{{site.data.alerts.callout_info}} -Running multiple nodes on a single host is useful for testing out CockroachDB, but it's not recommended for production deployments. To run a physically distributed cluster in production, see [Manual Deployment](manual-deployment.html) or [Orchestrated Deployment](orchestration.html). Also be sure to review the [Production Checklist](recommended-production-settings.html). -{{site.data.alerts.end}} - -If you are trying to run all nodes on the same machine, you might get the following errors: - -#### Store directory already exists - -~~~ -ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /Users/amruta/go/src/github.com/cockroachdb/cockroach/cockroach-data/cockroach-temp301343769, may still be in use: IO error: While lock file: /Users/amruta/go/src/github.com/cockroachdb/cockroach/cockroach-data/cockroach-temp301343769/TEMP_DIR.LOCK: Resource temporarily unavailable -~~~ - -**Explanation:** When starting a new node on the same machine, the directory you choose to store the data in also contains metadata identifying the cluster the data came from. This causes conflicts when you've already started a node on the server and then tried to start another cluster using the same directory. - -**Solution:** Choose a different directory to store the CockroachDB data. - -#### Port already in use - -~~~ -ERROR: cockroach server exited with error: consider changing the port via --listen-addr: listen tcp 127.0.0.1:26257: bind: address already in use -~~~ - -**Solution:** Change the `--port`, `--http-port` flags for each new node that you want to run on the same machine. - -## Cannot join a node to an existing CockroachDB cluster - -#### Store directory already exists - -When joining a node to a cluster, you might receive one of the following errors: - -~~~ -no resolvers found; use --join to specify a connected node - -node belongs to cluster {"cluster hash"} but is attempting to connect to a gossip network for cluster {"another cluster hash"} -~~~ - -**Explanation:** When starting a node, the directory you choose to store the data in also contains metadata identifying the cluster the data came from. This causes conflicts when you've already started a node on the server, have quit the `cockroach` process, and then tried to join another cluster. Because the existing directory's cluster ID doesn't match the new cluster ID, the node cannot join it. - -**Solution:** Disassociate the node from the existing directory where you've stored CockroachDB data. For example, you can do either of the following: - -- Choose a different directory to store the CockroachDB data: - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start --store= --join= - ~~~ -- Remove the existing directory and start a node joining the cluster again: - {% include_cached copy-clipboard.html %} - ~~~ shell - $ rm -r cockroach-data/ - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start --join=:26257 - ~~~ - -#### Incorrect `--join` address - -If you try to add another node to the cluster, but the `--join` address is not pointing at any of the existing nodes, then the process will never complete, and you'll see a continuous stream of warnings like this: - -~~~ -W180817 17:01:56.506968 886 vendor/google.golang.org/grpc/clientconn.go:942 Failed to dial localhost:20000: grpc: the connection is closing; please retry. -W180817 17:01:56.510430 914 vendor/google.golang.org/grpc/clientconn.go:1293 grpc: addrConn.createTransport failed to connect to {localhost:20000 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp [::1]:20000: connect: connection refused". Reconnecting… -~~~ - -**Explanation:** These warnings tell you that the node cannot establish a connection with the address specified in the `--join` flag. Without a connection to the cluster, the node cannot join. - -**Solution:** To successfully join the node to the cluster, start the node again, but this time include a correct `--join` address. - -## Client connection issues - -If a client cannot connect to the cluster, check basic network connectivity (`ping`), port connectivity (`telnet`), and certificate validity. - -#### Networking issues - -Most networking-related issues are caused by one of two issues: - -- Firewall rules, which require your network administrator to investigate -- Inaccessible hostnames on your nodes, which can be controlled with the `--listen-addr` and `--advertise-addr` flags on [`cockroach start`](cockroach-start.html#networking) - - -**Solution:** - -To check your networking setup: - -1. Use `ping`. Every machine you are using as a CockroachDB node should be able to ping every other machine, using the hostnames or IP addresses used in the `--join` flags (and the `--advertise-host` flag if you are using it). - -2. If the machines are all pingable, check if you can connect to the appropriate ports. With your CockroachDB nodes still running, log in to each node and use `telnet` or` nc` to verify machine to machine connectivity on the desired port. For instance, if you are running CockroachDB on the default port of 26257, run either: - - `telnet 26257` - - `nc 26257` - - Both `telnet` and `nc` will exit immediately if a connection cannot be established. If you are running in a firewalled environment, the firewall might be blocking traffic to the desired ports even though it is letting ping packets through. - -To efficiently troubleshoot the issue, it's important to understand where and why it's occurring. We recommend checking the following network-related issues: - -- By default, CockroachDB advertises itself to other nodes using its hostname. If your environment doesn't support DNS or the hostname is not resolvable, your nodes cannot connect to one another. In these cases, you can: - - Change the hostname each node uses to advertises itself with `--advertise-addr` - - Set `--listen-addr=` if the IP is a valid interface on the machine -- Every node in the cluster should be able to ping each other node on the hostnames or IP addresses you use in the `--join`, `--listen-addr`, or `--advertise-addr` flags. -- Every node should be able to connect to other nodes on the port you're using for CockroachDB (26257 by default) through `telnet` or `nc`: - - `telnet 26257` - - `nc 26257` - -Again, firewalls or hostname issues can cause any of these steps to fail. - -#### Network partition - -If the DB Console lists any dead nodes on the [**Cluster Overview** page](ui-cluster-overview-page.html), then you might have a network partition. - -**Explanation:** A network partition prevents nodes from communicating with each other in one or both directions. This can be due to a configuration problem with the network, such as when allowlisted IP addresses or hostnames change after a node is torn down and rebuilt. In a symmetric partition, node communication is broken in both directions. In an asymmetric partition, node communication works in one direction but not the other. - -The effect of a network partition depends on which nodes are partitioned, where the ranges are located, and to a large extent, whether [localities](cockroach-start.html#locality) are defined. If localities are not defined, a partition that cuts off at least (n-1)/2 nodes will cause data unavailability. - -**Solution:** - -To identify a network partition: - -1. Access the [Network Latency](ui-network-latency-page.html) page of the DB Console. -2. In the **Latencies** table, check for nodes with [no connections](ui-network-latency-page.html#no-connections). This indicates that a node cannot communicate with another node, and might indicate a network partition. - -## Authentication issues - -#### Missing certificate - -If you try to add a node to a secure cluster without providing the node's security certificate, you will get the following error: - -~~~ -problem with CA certificate: not found -* -* ERROR: cannot load certificates. -* Check your certificate settings, set --certs-dir, or use --insecure for insecure clusters. -* -* problem with CA certificate: not found -* -Failed running "start" -~~~ - -**Explanation:** The error tells you that because the cluster is secure, it requires the new node to provide its security certificate in order to join. - -**Solution:** To successfully join the node to the cluster, start the node again, but this time include the `--certs-dir` flag - -#### Certification expiration - -If you’re running a secure cluster, be sure to monitor your certificate expiration. If one of the inter-node certificates expires, nodes will no longer be able to communicate which can look like a network partition. - -To check the certificate expiration date: - -1. [Access the DB Console](ui-overview.html#db-console-access). -2. Click the gear icon on the left-hand navigation bar to access the **Advanced Debugging** page. -3. Scroll down to the **Even More Advanced Debugging** section. Click **All Nodes**. The **Node Diagnostics** page appears. Click the certificates for each node and check the expiration date for each certificate in the Valid Until field. - -#### Client password not set - -While connecting to a secure cluster as a user, CockroachDB first checks if the client certificate exists in the `cert` directory. If the client certificate doesn’t exist, it prompts for a password. If password is not set and you press Enter, the connection attempt fails, and the following error is printed to `stderr`: - -~~~ -Error: pq: invalid password -Failed running "sql" -~~~ - -**Solution:** To successfully connect to the cluster, you must first either generate a client certificate or create a password for the user. - -#### Cannot create new connections to cluster for up to 40 seconds after a node dies - -When a node [dies abruptly and/or loses its network connection to the cluster](#node-liveness-issues), the following behavior can occur: - -1. For a period of up to 40 seconds, clients trying to connect with [username and password authentication](authentication.html#client-authentication) cannot create new connections to any of the remaining nodes in the cluster. -1. Applications start timing out when trying to connect to the cluster during this window. - -The reason this happens is as follows: - -- Username and password information is stored in a system range. -- Since all system ranges are located [near the beginning of the keyspace](architecture/distribution-layer.html#monolithic-sorted-map-structure), the system range containing the username/password info can sometimes be colocated with another system range that is used to determine [node liveness](#node-liveness-issues). -- If the username/password info and the node liveness record are stored together as described above, it can take extra time for the lease on this range to be transferred to another node. Normally, lease transfers take about 10 seconds, but in this case it may require multiple rounds of consensus to determine that the node in question is actually dead (the node liveness record check may be retried several times before failing). - -For more information about how lease transfers work when a node dies, see [How leases are transferred from a dead node](architecture/replication-layer.html#how-leases-are-transferred-from-a-dead-node). - -The solution is to add connection retry logic to your application. - -## Clock sync issues - -#### Node clocks are not properly synchronized - -See the following FAQs: - -- [What happens when node clocks are not properly synchronized](operational-faqs.html#what-happens-when-node-clocks-are-not-properly-synchronized) -- [How can I tell how well node clocks are synchronized](operational-faqs.html#how-can-i-tell-how-well-node-clocks-are-synchronized) - -## Capacity planning issues - -Following are some of the possible issues you might have while planning capacity for your cluster: - -- Running CPU at close to 100% utilization with high run queue will result in poor performance. -- Running RAM at close to 100% utilization triggers Linux OOM and/or swapping that will result in poor performance or stability issues. -- Running storage at 100% capacity causes writes to fail, which in turn can cause various processes to stop. -- Running storage at 100% utilization read/write will causes poor service time. -- Running network at 100% utilization causes response between databases and client to be poor. - -**Solution:** [Access the DB Console](ui-overview.html#db-console-access) and navigate to **Metrics > Hardware** dashboard to monitor the following metrics: - -First, check adequate capacity was available for the incident for the following components. - -Type | Time Series | What to look for ---------|--------|--------| -RAM capacity | Memory Usage | Any non-zero value -CPU capacity | CPU Percent | Consistent non-zero values -Network capacity | Network Bytes Received
Network Bytes Sent | Any non-zero value - -Check Near Out of Capacity Conditions: - -Type | Time Series | What to look for ---------|--------|--------| -RAM capacity | Memory Usage | Consistently more than 80% -CPU capacity | CPU Percent | Consistently less than 20% in idle (i.e.:80% busy) -Network capacity | Network Bytes Received
Network Bytes Sent | Consistently more than 50% capacity for both - -## Storage issues - -#### Disks filling up - -Like any database system, if you run out of disk space the system will no longer be able to accept writes. Additionally, a CockroachDB node needs a small amount of disk space (a few GBs to be safe) to perform basic maintenance functionality. For more information about this issue, see: - -- [Why is memory usage increasing despite lack of traffic?](operational-faqs.html#why-is-memory-usage-increasing-despite-lack-of-traffic) -- [Why is disk usage increasing despite lack of writes?](operational-faqs.html#why-is-disk-usage-increasing-despite-lack-of-writes) -- [Can I reduce or disable the storage of timeseries data?](operational-faqs.html#can-i-reduce-or-disable-the-storage-of-time-series-data) - -#### Disk stalls - -A _disk stall_ is any disk operation that does not terminate in a reasonable amount of time. This usually manifests as write-related system calls such as [`fsync(2)`](https://man7.org/linux/man-pages/man2/fdatasync.2.html) (aka `fdatasync`) taking a lot longer than expected (e.g., more than 60 seconds). The mitigation in almost all cases is to [restart the node](cockroach-start.html) with the stalled disk. CockroachDB's internal disk stall monitoring will attempt to shut down a node when it sees a disk stall that lasts longer than 60 seconds. At that point the node should be restarted by your [orchestration system](recommended-production-settings.html#orchestration-kubernetes). - -Symptoms of disk stalls include: - -- Bad cluster write performance, usually in the form of a substantial drop in QPS for a given workload. -- [Node liveness issues](#node-liveness-issues). -- Writes on one node come to a halt. This can happen because in rare cases, a node may be able to perform liveness checks (which involve writing to disk) even though it cannot write other data to disk due to one or more slow/stalled calls to `fsync`. Because the node is passing its liveness checks, it is able to hang onto its leases even though it cannot make progress on the ranges for which it is the leaseholder. This wedged node has a ripple effect on the rest of the cluster such that all processing of the ranges whose leaseholders are on that node basically grinds to a halt. As mentioned above, CockroachDB's disk stall detection will attempt to shut down the node when it detects this state. - -Causes of disk stalls include: - -- Disk operations have backed up due to underprovisioned IOPS. Make sure you are deploying with our [recommended production settings for storage](recommended-production-settings.html#storage). -- Actual hardware-level storage issues that result in slow `fsync` performance. -- In rare cases, operating-system-level configuration of subsystems such as SELinux can slow down system calls such as `fsync` enough to affect storage engine performance. - -CockroachDB's built-in disk stall detection works as follows: - -- Every 10 seconds, the CockroachDB storage engine checks the [_write-ahead log_](https://en.wikipedia.org/wiki/Write-ahead_logging), or _WAL_. If data has not been synced to disk (via `fsync`) within that interval, the log message `disk stall detected: unable to write to %s within %s %s warning log entry` is written to the [`STORAGE` logging channel](logging.html#storage). If this state continues for 60 seconds or more (configurable with the `COCKROACH_ENGINE_MAX_SYNC_DURATION` environment variable), the `cockroach` process is terminated. - -- Every time the storage engine writes to the main [`cockroach.log` file](logging.html#dev), the engine waits 30 seconds for the write to succeed (configurable with the `COCKROACH_LOG_MAX_SYNC_DURATION` environment variable). If the write to the log fails, the `cockroach` process is terminated and the following message is written to stderr / `cockroach.log`: - - - `disk stall detected: unable to sync log files within %s` - -- During [node liveness heartbeats](#node-liveness-issues), the [storage engine](architecture/storage-layer.html) writes to disk as part of the node liveness heartbeat process. - -## Memory issues - -#### Suspected memory leak - -A CockroachDB node will grow to consume all of the memory allocated for its `cache`. The default size for the cache is ¼ of physical memory which can be substantial depending on your machine configuration. This growth will occur even if your cluster is otherwise idle due to the internal metrics that a CockroachDB cluster tracks. See the `--cache` flag in [`cockroach start`](cockroach-start.html#general). - -CockroachDB memory usage has the following components: - -- **Go allocated memory**: Memory allocated by the Go runtime to support query processing and various caches maintained in Go by CockroachDB. If Go allocated memory is larger than a few hundred megabytes, something concerning is going on. -- **CGo allocated memory**: Memory allocated by the C/C++ libraries linked into CockroachDB and primarily concerns the block caches for the [Pebble storage engine](cockroach-start.html#storage-engine)). This is the “cache” mentioned in the note above. The size of CGo allocated memory is usually very close to the configured cache size. -- **Overhead**: The process resident set size minus Go/CGo allocated memory. - -If Go allocated memory is larger than a few hundred megabytes, you might have encountered a memory leak. Go comes with a built-in heap profiler which is already enabled on your CockroachDB process. See this [excellent blog post](https://blog.golang.org/profiling-go-programs) on profiling Go programs. - -**Solution:** To determine Go/CGo allocated memory: - -1. [Access the DB Console](ui-overview.html#db-console-access). - -2. Navigate to **Metrics > Runtime** dashboard, and check the **Memory Usage** graph. - -3. On hovering over the graph, the values for the following metrics are displayed: - - Metric | Description - --------|---- - RSS | Total memory in use by CockroachDB. - Go Allocated | Memory allocated by the Go layer. - Go Total | Total memory managed by the Go layer. - CGo Allocated | Memory allocated by the C layer. - CGo Total | Total memory managed by the C layer. - - - If CGo allocated memory is larger than the configured `cache` size, [file an issue](file-an-issue.html). - - If the resident set size (RSS) minus Go/CGo total memory is larger than 100 megabytes, [file an issue](file-an-issue.html). - -#### Node crashes because of insufficient memory - -Often when a node exits without a trace or logging any form of error message, we’ve found that it is the operating system stopping it suddenly due to low memory. So if you're seeing node crashes where the logs just end abruptly, it's probably because the node is running out of memory. On most Unix systems, you can verify if the `cockroach` process was stopped because the node ran out of memory by running: - -~~~ shell -$ sudo dmesg | grep -iC 3 "cockroach" -~~~ - -If the command returns the following message, then you know the node crashed due to insufficient memory: - -~~~ shell -$ host kernel: Out of Memory: Killed process (cockroach). -~~~ - -To rectify the issue, you can either run the cockroachdb process on another node with sufficient memory, or [reduce the cockroachdb memory usage](cockroach-start.html#flags). - -## Decommissioning issues - -#### Decommissioning process hangs indefinitely - -**Explanation:** Before decommissioning a node, you need to make sure other nodes are available to take over the range replicas from the node. If no other nodes are available, the decommission process will hang indefinitely. - -**Solution:** Confirm that there are enough nodes with sufficient storage space to take over the replicas from the node you want to remove. - -#### Decommissioned nodes displayed in UI forever - -By design, decommissioned nodes are displayed in the DB Console forever. We retain the list of decommissioned nodes for the following reasons: - -- Decommissioning is not entirely free, so showing those decommissioned nodes in the UI reminds you of the baggage your cluster will have to carry around forever. -- It also explains to future administrations why your node IDs have gaps (e.g., why the nodes are numbered n1, n2, and n8). - -You can follow the discussion here: [https://github.com/cockroachdb/cockroach/issues/24636](https://github.com/cockroachdb/cockroach/issues/24636) - -## Replication issues - -#### DB Console shows under-replicated/unavailable ranges - -When a CockroachDB node dies (or is partitioned) the under-replicated range count will briefly spike while the system recovers. - -**Explanation:** CockroachDB uses consensus replication and requires a quorum of the replicas to be available in order to allow both writes and reads to the range. The number of failures that can be tolerated is equal to (Replication factor - 1)/2. Thus CockroachDB requires (n-1)/2 nodes to achieve quorum. For example, with 3x replication, one failure can be tolerated; with 5x replication, two failures, and so on. - -- Under-replicated Ranges: When a cluster is first initialized, the few default starting ranges will only have a single replica, but as soon as other nodes are available, they will replicate to them until they've reached their desired replication factor. If a range does not have enough replicas, the range is said to be "under-replicated". - -- Unavailable Ranges: If a majority of a range's replicas are on nodes that are unavailable, then the entire range is unavailable and will be unable to process queries. - -**Solution:** - -To identify under-replicated/unavailable ranges: - -1. [Access the DB Console](ui-overview.html#db-console-access). - -2. On the **Cluster Overview** page, check the **Replication Status**. If the **Under-replicated ranges** or **Unavailable ranges** count is non-zero, then you have under-replicated or unavailable ranges in your cluster. - -3. Check for a network partition: Click the gear icon on the left-hand navigation bar to access the **Advanced Debugging** page. On the Advanced Debugging page, click **Network Latency**. In the **Latencies** table, check if any cells are marked as “X”. If yes, it indicates that the nodes cannot communicate with those nodes, and might indicate a network partition. If there's no partition, and there's still no upreplication after 5 mins, then [file an issue](file-an-issue.html). - -**Add nodes to the cluster:** - -On the DB Console’s Cluster Overview page, check if any nodes are down. If the number of nodes down is less than (n-1)/2, then that is most probably the cause of the under-replicated/unavailable ranges. Add nodes to the cluster such that the cluster has the required number of nodes to replicate ranges properly. - -If you still see under-replicated/unavailable ranges on the Cluster Overview page, investigate further: - -1. [Access the DB Console](ui-overview.html#db-console-access) -2. Click the gear icon on the left-hand navigation bar to access the **Advanced Debugging** page. -2. Click **Problem Ranges**. -3. In the **Connections** table, identify the node with the under-replicated/unavailable ranges and click the node ID in the Node column. -4. To view the **Range Report** for a range, click on the range number in the **Under-replicated (or slow)** table or **Unavailable** table. -5. On the Range Report page, scroll down to the **Simulated Allocator Output** section. The table contains an error message which explains the reason for the under-replicated range. Follow the guidance in the message to resolve the issue. If you need help understanding the error or the guidance, [file an issue](file-an-issue.html). Please be sure to include the full range report and error message when you submit the issue. - -## Node liveness issues - -"Node liveness" refers to whether a node in your cluster has been determined to be "dead" or "alive" by the rest of the cluster. This is achieved using checks that ensure that each node connected to the cluster is updating its liveness record. This information is shared with the rest of the cluster using an internal gossip protocol. - -Common reasons for node liveness issues include: - -- Heavy I/O load on the node. Because each node needs to update a liveness record on disk, maxing out disk bandwidth can cause liveness heartbeats to be missed. See also: [Capacity planning issues](#capacity-planning-issues). -- Outright I/O failure due to a [disk stall](#disk-stalls). This will cause node liveness issues for the same reasons as listed above. -- Any [Networking issues](#networking-issues) with the node. - -The [DB Console][db_console] provides several ways to check for node liveness issues in your cluster: - -- [Check node heartbeat latency](#check-node-heartbeat-latency) -- [Check node liveness record last update](#check-node-liveness-record-last-update) -- [Check command commit latency](#check-command-commit-latency) - -{{site.data.alerts.callout_info}} -For more information about how node liveness works, see [the architecture documentation on the replication layer](architecture/replication-layer.html#epoch-based-leases-table-data). -{{site.data.alerts.end}} - -#### Check node heartbeat latency - -To check node heartbeat latency: - -1. In the [DB Console][db_console], select the **Metrics** tab from the left-hand side of the page. - -2. From the metrics page, select **Dashboard: Distributed** from the dropdown at the top of the page. - -3. Scroll down the metrics page to find the **Node Heartbeat Latency: 99th percentile** and **Node Heartbeat Latency: 90th percentile** graphs. - -**Expected values for a healthy cluster**: Less than 100ms in addition to the network latency between nodes in the cluster. - -#### Check node liveness record last update - -To see when a node last updated its liveness record: - -1. Go to the **Node Diagnostics** page of the [DB Console][db_console], which lives at: - - `https://yourcluster.yourdomain/#/reports/nodes` - -2. On the Node Diagnostics page, you will see a table listing information about the nodes in your cluster. To see when a node last updated its liveness record, check the **Updated at** field at the bottom of that node's column. - -**Expected values for a healthy cluster**: When you load this page, the **Updated at** field should be within 4.5 seconds of the current time. If it's higher than that, you will see errors [in the logs](logging-overview.html). - -#### Check command commit latency - -A good signal of I/O load is the **Command Commit Latency** in the **Storage** section of the dashboards. This dashboard measures how quickly [Raft commands](architecture/replication-layer.html) are being committed by nodes in the cluster. - -To view command commit latency: - -1. In the [DB Console][db_console], select the **Metrics** tab from the left-hand side of the page. - -2. From the Metrics page, select **Dashboard: Storage** from the dropdown at the top of the page. - -3. Scroll down the metrics page to find the **Command Commit Latency: 90th percentile** and **Command Commit Latency: 99th percentile** graphs. - -**Expected values for a healthy cluster**: On SSDs, this should be between 1 and 100 milliseconds. On HDDs, this should be no more than 1 second. Note that we [strongly recommend running CockroachDB on SSDs](recommended-production-settings.html#storage). - -#### Impact of node failure is greater than 10 seconds - -When the cluster needs to access a range on a leaseholder node that is dead, that range's [lease must be transferred to a healthy node](architecture/replication-layer.html#how-leases-are-transferred-from-a-dead-node). In theory, this process should take no more than 9 seconds for liveness expiration plus the cost of several network roundtrips. - -In production, lease transfer upon node failure can take longer than expected. In {{ page.version.version }}, this is observed in the following scenarios: - -- **The leaseholder node for the liveness range fails.** The liveness range is a system range that [stores the liveness record](architecture/replication-layer.html#epoch-based-leases-table-data) for each node on the cluster. If a node fails and is also the leaseholder for the liveness range, operations cannot proceed until the liveness range is transferred to a new leaseholder and the liveness record is made available to other nodes. This can cause momentary cluster unavailability. - -- **The leaseholder node for `system.users` fails.** The cluster accesses the `system.users` table when authenticating a user. A customer attempting to log in to the cluster will encounter a delay while waiting for the `system.users` table to find a new leaseholder. In **v21.2.0 and later**, there is no login delay because the authentication data is cached. - -- **A node comes in and out of liveness, due to network failures or overload.** Prior to v21.1.6, a "flapping" node can repeatedly lose and acquire leases in between its intermittent heartbeats, causing wide-ranging unavailability. In **v21.1.6 and later**, a node must successfully heartbeat for 30 seconds after failing a heartbeat before it is eligible to acquire leases. - -- **Network or DNS issues cause connection issues between nodes.** If there is no live server for the IP address or DNS lookup, connection attempts to a node will not return an immediate error, but will hang until timing out. This can cause unavailability and prevent a speedy movement of leases and recovery. In **v21.1.12 and later**, CockroachDB avoids contacting unresponsive nodes or DNS during certain performance-critical operations, and the connection issue should generally resolve in 10-30 seconds. However, an attempt to contact an unresponsive node could still occur in other scenarios that are not yet addressed. - -- **A node's disk stalls.** A [disk stall](#disk-stalls) on a node can cause write operations to stall indefinitely, also causes the node's heartbeats to fail since the storage engine cannot write to disk as part of the heartbeat, and may cause read requests to fail if they are waiting for a conflicting write to complete. Lease acquisition from this node can stall indefinitely until the node is shut down or recovered. Pebble detects most stalls and will terminate the `cockroach` process after 60 seconds, but there are gaps in its detection. In **v21.2.13, v22.1.2, and later**, each lease acquisition attempt on an unresponsive node times out after 6 seconds. However, CockroachDB can still appear to stall as these timeouts are occurring. - -- **Otherwise unresponsive nodes.** Internal deadlock due to faulty code, resource exhaustion, OS/hardware issues, and other arbitrary failures can make a node unresponsive. This can cause leases to become stuck in certain cases, such as when a response from the previous leaseholder is needed in order to move the lease. - -**Solution:** If you are experiencing intermittent network or connectivity issues, first [shut down the affected nodes](../{{site.versions["stable"]}}/node-shutdown.html) temporarily so that nodes phasing in and out do not cause disruption. - -If a node has become unresponsive without returning an error, [shut down the node](../{{site.versions["stable"]}}/node-shutdown.html) so that network requests immediately become hard errors rather than stalling. - -If you are running a version of CockroachDB that is affected by an issue described here, [upgrade to a version](upgrade-cockroach-version.html) that contains the fix for the issue, as described in the preceding list. - -## Partial availability issues - -If your cluster is in a partially-available state due to a recent node or network failure, the internal logging table `system.eventlog` might be unavailable. This can cause the logging of [notable events](eventlog.html) (e.g., the execution of SQL statements) to the `system.eventlog` table to fail to complete, contributing to cluster unavailability. If this occurs, you can set the [cluster setting](cluster-settings.html) `server.eventlog.enabled` to `false` to disable writing notable log events to this table, which may help to recover your cluster. - -Even with `server.eventlog.enabled` set to `false`, notable log events are still sent to configured [log sinks](configure-logs.html#configure-log-sinks) as usual. - -## Check for under-replicated or unavailable data - -To see if any data is under-replicated or unavailable in your cluster, use the `system.replication_stats` report as described in [Replication Reports](query-replication-reports.html). - -## Check for replication zone constraint violations - -To see if any of your cluster's [data placement constraints](configure-replication-zones.html#replication-constraints) are being violated, use the `system.replication_constraint_stats` report as described in [Replication Reports](query-replication-reports.html). - -## Check for critical localities - -To see which of your [localities](cockroach-start.html#locality) (if any) are critical, use the `system.replication_critical_localities` report as described in [Replication Reports](query-replication-reports.html). A locality is "critical" for a range if all of the nodes in that locality becoming [unreachable](#node-liveness-issues) would cause the range to become unavailable. In other words, the locality contains a majority of the range's replicas. - -## Something else? - -If we do not have a solution here, you can try using our other [support resources](support-resources.html), including: - -- [StackOverflow](http://stackoverflow.com/questions/tagged/cockroachdb) -- [CockroachDB Community Forum](https://forum.cockroachlabs.com) -- [Chatting with our developers on Slack](https://cockroachdb.slack.com) - - - -[db_console]: ui-overview.html diff --git a/src/current/v21.1/cockroach-auth-session.md b/src/current/v21.1/cockroach-auth-session.md deleted file mode 100644 index b4636d484ad..00000000000 --- a/src/current/v21.1/cockroach-auth-session.md +++ /dev/null @@ -1,210 +0,0 @@ ---- -title: cockroach auth-session -summary: To create and manage web sessions and authentication tokens to the HTTP interface from the command line, use the cockroach auth-session command. -toc: true -docs_area: reference.cli ---- - -To create and manage web sessions and authentication tokens to the HTTP interface from the command line, use the `cockroach auth-session` [command](cockroach-commands.html) with the appropriate subcommands and flags. - -## Subcommands - -Subcommand | Usage ------------|------ -`login` | Authenticate a user against a running cluster's HTTP interface, generating an HTTP authentication token (a "cookie") which can also be used by non-interactive HTTP-based database management tools. Must be used with a valid, existing user. May be used to generate a cookie for the `root` user. -`logout` | Revokes all previously-issued HTTP authentication tokens for the given user. -`list` | List all authenticated sessions to the HTTP interface, including currently active and recently expired sessions. - -## Synopsis - -Log in to the HTTP interface, generating an HTTP authentication token for a given user: - -~~~ shell -$ cockroach auth-session login {username} [flags] -~~~ - -Log out from the HTTP interface, revoking all active HTTP authentication tokens for a given user: - -~~~ shell -$ cockroach auth-session logout {username} [flags] -~~~ - -List all authenticated sessions to the HTTP interface, including currently active and recently expired sessions: - -~~~ shell -$ cockroach auth-session list [flags] -~~~ - -View help: - -~~~ shell -$ cockroach auth-session --help -~~~ - -~~~ shell -$ cockroach auth-session {subcommand} --help -~~~ - -## Flags - -All three `auth-session` subcommands accept the standard [SQL command-line flags](cockroach-start.html#flags). - -In addition, the `auth-session login` subcommand supports the following flags. - -Flag | Description ------|------------ -`--expire-after` | Duration of the newly-created HTTP authentication token, after which the token expires. Specify the duration in numeric values suffixed by one or more of `h`, `m`, and `s` to indicate hour, minute, and second duration. See the [example](#log-in-to-the-http-interface-with-a-custom-expiry).

**Default:** `1h0m0s` (1 hour) -`--only-cookie` | Limits output to only the newly-created HTTP authentication token (the "cookie") in the response, appropriate for output to other commands. See the [example](#log-in-to-the-http-interface-with-limited-command-output). - -## Response - -The `cockroach auth-session` subcommands return the following fields. - -### `auth-session login` - -Field | Description -------|------------ -`username` | The username of the user authenticated. -`session ID` | The session ID to the HTTP interface previously established for that user. -`authentication cookie` | The cookie that may be used from the command line, or from other tools, to authenticate access to the HTTP interface for that user. - -### `auth-session logout` - -Field | Description -------|------------ -`username` | The username of the user whose session was revoked. -`session ID` | The session ID to the HTTP interface previously established for that user. -`revoked` | The date and time of revocation for that user's authenticated session. - -### `auth-session list` - -Field | Description -------|------------ -`username` | The username of the user authenticated. -`session ID` | The session ID to the HTTP interface established for that user. -`created` | The date and time a session was created. -`expired` | The date and time a session expired. -`revoked` | The date and time of revocation for that user's authenticated session. If the session is still active, this will appear as `NULL`. -`last used` | The date and time of the last access to the HTTP interface using this session token. - -## Required roles - -To run any of the `auth-session` subcommands, you must be a member of the [`admin` role](authorization.html#admin-role). The user being authenticated via `login` or `logout` does not require any special roles. - -## Considerations - -- The `login` subcommand allows users with the [`admin` role](authorization.html#admin-role) to create HTTP authentication tokens with an arbitrary duration. If operational policy requires stricter control of authentication sessions, you can: - - - Monitor the `system.web_sessions` table for all current and recent HTTP sessions. - - Revoke HTTP authentication tokens as needed with the `logout` subcommand. See the [example](#terminate-all-active-sessions-for-a-user). - - Set the `--expire-after` flag with a shorter duration. See the [example](#log-in-to-the-http-interface-with-a-custom-expiry). - -- The `logout` subcommand logs out all sessions for the given user; you cannot target individual sessions for logout. If more granular control of sessions is desired, consider setting the `--expire-after` flag with a shorter duration. See the [example](#log-in-to-the-http-interface-with-a-custom-expiry). - -## Examples - -### Log in to the HTTP interface - -Log in to the HTTP interface, by generating a new HTTP authentication token for the `web_user` user: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach auth-session login web_user -~~~ - -~~~ - username | session ID | authentication cookie ------------+--------------------+--------------------------------------------------------------------- - web_user | 784445853689282561 | session=CIGAtrWQq7rxChIQTXxYNNQxAYjyLAjHWxgUMQ==; Path=/; HttpOnly; Secure -(1 row) -~~~ - -### Log in to the HTTP interface with a custom expiry - -Log in to the HTTP interface, by generating a new HTTP authentication token for the `web_user` user and specifying a token expiry of 4 hours and 30 minutes: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach auth-session login web_user --expire-after=4h30m -~~~ - -~~~ - username | session ID | authentication cookie ------------+--------------------+--------------------------------------------------------------------- - web_user | 784445853689282561 | session=CIGAtrWQq7rxChIQTXxYNNQxAYjyLAjHWxgUMQ==; Path=/; HttpOnly; Secure -(1 row) -~~~ - -### Log in to the HTTP interface with limited command output - -Log in to the HTTP interface, by generating a new HTTP authentication token for the `web_user` user, limiting command output to only the generated cookie: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach auth-session login web_user --only-cookie -~~~ - -~~~ -session=CIGA6t2q0LrxChIQV8QCF3vuYSasR7h4LPSfmg==; Path=/; HttpOnly; Secure -~~~ - -This is useful if you intend to use the cookie with other command line tools. For example, you might output the generated cookie to a local file, and then pass that file to `curl` using its `--cookie` flag: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach auth-session login web_user --certs-dir=certs --only-cookie > $HOME/.cockroachdb_api_key -$ curl --cookie $HOME/.cockroachdb_api_key https://localhost:8080/_status/logfiles/local -~~~ - -Of course you can also provide the cookie directly: - -{% include_cached copy-clipboard.html %} -~~~ shell -curl --cookie 'session=CIGA8I7/irvxChIQDtZQsMtn3AqpgDko6bldSw==; Path=/; HttpOnly; Secure' https://localhost:8080/_status/logfiles/local -~~~ - -### Terminate all active sessions for a user - -Terminate all active sessions for the `web_user` user: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach auth-session logout web_user -~~~ - -~~~ - username | session ID | revoked ------------+--------------------+----------------------------- - web_user | 784445853689282561 | 2022-08-02 18:24:50.819614 - web_user | 784447132063662081 | 2022-08-02 18:24:50.819614 - web_user | 784449147579924481 | 2022-08-02 18:47:20.105254 - web_user | 784449219848241153 | 2022-08-02 18:47:20.105254 -(4 rows) -~~~ - -Note that the output may include recently revoked sessions for this user as well. - -### List all sessions - -List all authenticated sessions to the HTTP interface, including currently active and recently expired sessions: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach auth-session list -~~~ - -~~~ - username | session ID | created | expires | revoked | last used ------------+--------------------+----------------------------+----------------------------+----------------------------+----------------------------- - root | 784428093743988737 | 2022-08-02 16:47:36.342338 | 2022-08-02 17:47:36.341997 | NULL | 2022-08-02 16:47:36.342338 - root | 784428586862215169 | 2022-08-02 16:50:06.830294 | 2022-08-02 17:50:06.829974 | NULL | 2022-08-02 16:50:06.830294 - web_user | 784447132063662081 | 2022-08-02 18:24:26.37664 | 2022-08-02 19:24:26.376299 | 2022-08-02 18:24:50.819614 | 2022-08-02 18:24:26.37664 - web_user | 784449147579924481 | 2022-08-02 18:34:41.463345 | 2022-08-02 19:34:41.463006 | 2022-08-02 18:47:20.105254 | 2022-08-02 18:34:41.463345 -~~~ - -A value of `NULL` in the `revoked` column indicates that the session is still active. - -## See also - -- [`cockroach` Commands Overview](cockroach-commands.html) -- [DB Console Overview](ui-overview.html) diff --git a/src/current/v21.1/cockroach-cert.md b/src/current/v21.1/cockroach-cert.md deleted file mode 100644 index 421a5e73a3e..00000000000 --- a/src/current/v21.1/cockroach-cert.md +++ /dev/null @@ -1,326 +0,0 @@ ---- -title: cockroach cert -summary: A secure CockroachDB cluster uses TLS for encrypted inter-node and client-node communication. -toc: true -key: create-security-certificates.html ---- - -To secure your CockroachDB cluster's inter-node and client-node communication, you need to provide a Certificate Authority (CA) certificate that has been used to sign keys and certificates (SSLs) for: - -- Nodes -- Clients -- DB Console (optional) - -To create these certificates and keys, use the `cockroach cert` [commands](cockroach-commands.html) with the appropriate subcommands and flags, use [`openssl` commands](https://wiki.openssl.org/index.php/), or use a [custom CA](create-security-certificates-custom-ca.html) (for example, a public CA or your organizational CA). - -
- - - -
- -{{site.data.alerts.callout_success}}For details about when and how to change security certificates without restarting nodes, see Rotate Security Certificates.{{site.data.alerts.end}} - -## How security certificates work - -1. Using the `cockroach cert` command, you create a CA certificate and key and then node and client certificates that are signed by the CA certificate. Since you need access to a copy of the CA certificate and key to create node and client certs, it's best to create everything in one place. - -2. You then upload the appropriate node certificate and key and the CA certificate to each node, and you upload the appropriate client certificate and key and the CA certificate to each client. - -3. When nodes establish contact to each other, and when clients establish contact to nodes, they use the CA certificate to verify each other's identity. - -## Subcommands - -Subcommand | Usage ------------|------ -`create-ca` | Create the self-signed certificate authority (CA), which you'll use to create and authenticate certificates for your entire cluster. -`create-node` | Create a certificate and key for a specific node in the cluster. You specify all addresses at which the node can be reached and pass appropriate flags. -`create-client` | Create a certificate and key for a [specific user](create-user.html) accessing the cluster from a client. You specify the username of the user who will use the certificate and pass appropriate flags. -`list` | List certificates and keys found in the certificate directory. - -## Certificate directory - -When using `cockroach cert` to create node and client certificates, you will need access to a local copy of the CA certificate and key. It is therefore recommended to create all certificates and keys in one place and then distribute node and client certificates and keys appropriately. For the CA key, be sure to store it somewhere safe and keep a backup; if you lose it, you will not be able to add new nodes or clients to your cluster. For a tutorial of this process, see [Manual Deployment](manual-deployment.html). - -## Required keys and certificates - -The `create-*` subcommands generate the CA certificate and all node and client certificates and keys in a single directory specified by the `--certs-dir` flag, with the files named as follows: - -### Node key and certificates - -File name pattern | File usage --------------|------------ -`ca.crt` | CA certificate. -`node.crt` | Server certificate.

`node.crt` must be signed by `ca.crt` and must have `CN=node` and the list of IP addresses and DNS names listed in `Subject Alternative Name` field. CockroachDB also supports [wildcard notation in DNS names](https://en.wikipedia.org/wiki/Wildcard_certificate). -`node.key` | Key for server certificate. - -### Client key and certificates - -File name pattern | File usage --------------|------------ -`ca.crt` | CA certificate. -`client..crt` | Client certificate for `` (e.g., `client.root.crt` for user `root`).

Must be signed by `ca.crt`. Also, `client..crt` must have `CN=` (for example, `CN=marc` for `client.marc.crt`) -`client..key` | Key for the client certificate. - -Optionally, if you have a certificate issued by a public CA to securely access the DB Console, you need to place the certificate and key (`ui.crt` and `ui.key` respectively) in the directory specified by the `--certs-dir` flag. For more information, refer to [Use a UI certificate and key to access the DB Console](create-security-certificates-custom-ca.html#accessing-the-db-console-for-a-secure-cluster). - -Note the following: - -- By default, the `node.crt` is multi-functional, as in the same certificate is used for both incoming connections (from SQL and DB Console clients, and from other CockroachDB nodes) and for outgoing connections to other CockroachDB nodes. To make this possible, the `node.crt` created using the `cockroach cert` command has `CN=node` and the list of IP addresses and DNS names listed in `Subject Alternative Name` field. - -- The CA key is never loaded automatically by `cockroach` commands, so it should be created in a separate directory, identified by the `--ca-key` flag. - -- Keys (files ending in `.key`) must not have group or world permissions (maximum permissions are 0700, or `rwx------`). This check can be disabled by setting the environment variable `COCKROACH_SKIP_KEY_PERMISSION_CHECK=true`. - -## Synopsis - -Create the CA certificate and key: - -~~~ shell -$ cockroach cert create-ca \ - --certs-dir=[path-to-certs-directory] \ - --ca-key=[path-to-ca-key] -~~~ - -Create a node certificate and key: - -~~~ shell -$ cockroach cert create-node \ - [node-hostname] \ - [node-other-hostname] \ - [node-yet-another-hostname] \ - [hostname-in-wildcard-notation] \ - --certs-dir=[path-to-certs-directory] \ - --ca-key=[path-to-ca-key] -~~~ - -Create a client certificate and key: - -~~~ shell -$ cockroach cert create-client \ - [username] \ - --certs-dir=[path-to-certs-directory] \ - --ca-key=[path-to-ca-key] -~~~ - -List certificates and keys: - -~~~ shell -$ cockroach cert list \ - --certs-dir=[path-to-certs-directory] -~~~ - -View help: - -~~~ shell -$ cockroach cert --help -~~~ -~~~ shell -$ cockroach cert --help -~~~ - -## Flags - -The `cert` command and subcommands support the following [general-use](#general) and [logging](#logging) flags. - -### General - -Flag | Description ------|----------- -`--certs-dir` | The path to the [certificate directory](#certificate-directory) containing all certificates and keys needed by `cockroach` commands.

This flag is used by all subcommands.

**Default:** `${HOME}/.cockroach-certs/` -`--ca-key` | The path to the private key protecting the CA certificate.

This flag is required for all `create-*` subcommands. When used with `create-ca` in particular, it defines where to create the CA key; the specified directory must exist.

**Env Variable:** `COCKROACH_CA_KEY` -`--allow-ca-key-reuse` | When running the `create-ca` subcommand, pass this flag to re-use an existing CA key identified by `--ca-key`. Otherwise, a new CA key will be generated.

This flag is used only by the `create-ca` subcommand. It helps avoid accidentally re-using an existing CA key. -`--overwrite` | When running `create-*` subcommands, pass this flag to allow existing files in the certificate directory (`--certs-dir`) to be overwritten.

This flag helps avoid accidentally overwriting sensitive certificates and keys. -`--lifetime` | The lifetime of the certificate, in hours, minutes, and seconds.

Certificates are valid from the time they are created through the duration specified in `--lifetime`.

**Default:** `87840h0m0s` (10 years) -`--key-size` | The size of the CA, node, or client key, in bits.

**Default:** `2048` - `--also-generate-pkcs8-key` | Also create a key in [PKCS#8 format](https://tools.ietf.org/html/rfc5208), which is the standard key encoding format used by Java. For example usage, see [Build a Java App with CockroachDB](build-a-java-app-with-cockroachdb.html). - -### Logging - -{% include {{ page.version.version }}/misc/logging-defaults.md %} - -## Examples - -### Create the CA certificate and key pair - -1. Create two directories: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ mkdir certs - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ mkdir my-safe-directory - ~~~ - - `certs`: You'll generate your CA certificate and all node and client certificates and keys in this directory and then upload some of the files to your nodes. - - `my-safe-directory`: You'll generate your CA key in this directory and then reference the key when generating node and client certificates. After that, you'll keep the key safe and secret; you will not upload it to your nodes. - -2. Generate the CA certificate and key: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach cert create-ca \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ ls -l certs - ~~~ - - ~~~ - total 8 - -rw-r--r-- 1 maxroach maxroach 1.1K Jul 10 14:12 ca.crt - ~~~ - -### Create the certificate and key pairs for nodes - -1. Generate the certificate and key for the first node: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach cert create-node \ - node1.example.com \ - node1.another-example.com \ - *.dev.another-example.com \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ ls -l certs - ~~~ - - ~~~ - total 24 - -rw-r--r-- 1 maxroach maxroach 1.1K Jul 10 14:12 ca.crt - -rw-r--r-- 1 maxroach maxroach 1.2K Jul 10 14:16 node.crt - -rw------- 1 maxroach maxroach 1.6K Jul 10 14:16 node.key - ~~~ - -2. Upload certificates to the first node: - - {% include_cached copy-clipboard.html %} - ~~~ shell - # Create the certs directory: - $ ssh @ "mkdir certs" - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - # Upload the CA certificate and node certificate and key: - $ scp certs/ca.crt \ - certs/node.crt \ - certs/node.key \ - @:~/certs - ~~~ - -3. Delete the local copy of the first node's certificate and key: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ rm certs/node.crt certs/node.key - ~~~ - - {{site.data.alerts.callout_info}}This is necessary because the certificates and keys for additional nodes will also be named node.crt and node.key As an alternative to deleting these files, you can run the next cockroach cert create-node commands with the --overwrite flag.{{site.data.alerts.end}} - -4. Create the certificate and key for the second node: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach cert create-node \ - node2.example.com \ - node2.another-example.com \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ ls -l certs - ~~~ - - ~~~ - total 24 - -rw-r--r-- 1 maxroach maxroach 1.1K Jul 10 14:12 ca.crt - -rw-r--r-- 1 maxroach maxroach 1.2K Jul 10 14:17 node.crt - -rw------- 1 maxroach maxroach 1.6K Jul 10 14:17 node.key - ~~~ - -5. Upload certificates to the second node: - - {% include_cached copy-clipboard.html %} - ~~~ shell - # Create the certs directory: - $ ssh @ "mkdir certs" - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - # Upload the CA certificate and node certificate and key: - $ scp certs/ca.crt \ - certs/node.crt \ - certs/node.key \ - @:~/certs - ~~~ - -6. Repeat steps 3 - 5 for each additional node. - -### Create the certificate and key pair for a client - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach cert create-client \ -maxroach \ ---certs-dir=certs \ ---ca-key=my-safe-directory/ca.key -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ ls -l certs -~~~ - -~~~ -total 40 --rw-r--r-- 1 maxroach maxroach 1.1K Jul 10 14:12 ca.crt --rw-r--r-- 1 maxroach maxroach 1.1K Jul 10 14:13 client.maxroach.crt --rw------- 1 maxroach maxroach 1.6K Jul 10 14:13 client.maxroach.key --rw-r--r-- 1 maxroach maxroach 1.2K Jul 10 14:17 node.crt --rw------- 1 maxroach maxroach 1.6K Jul 10 14:17 node.key -~~~ - -### List certificates and keys - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach cert list \ ---certs-dir=certs -~~~ - -~~~ -Certificate directory: certs -+-----------------------+---------------------+---------------------+------------+--------------------------------------------------------+-------+ -| Usage | Certificate File | Key File | Expires | Notes | Error | -+-----------------------+---------------------+---------------------+------------+--------------------------------------------------------+-------+ -| Certificate Authority | ca.crt | | 2027/07/18 | num certs: 1 | | -| Node | node.crt | node.key | 2022/07/14 | addresses: node2.example.com,node2.another-example.com | | -| Client | client.maxroach.crt | client.maxroach.key | 2022/07/14 | user: maxroach | | -+-----------------------+---------------------+---------------------+------------+--------------------------------------------------------+-------+ -(3 rows) -~~~ - -## See also - -- [Security overview](security-overview.html) -- [Authentication](authentication.html) -- [Client Connection Parameters](connection-parameters.html) -- [Rotate Security Certificates](rotate-certificates.html) -- [Manual Deployment](manual-deployment.html) -- [Orchestrated Deployment](orchestration.html) -- [Local Deployment](secure-a-cluster.html) -- [Other Cockroach Commands](cockroach-commands.html) diff --git a/src/current/v21.1/cockroach-commands.md b/src/current/v21.1/cockroach-commands.md deleted file mode 100644 index ed3ca675136..00000000000 --- a/src/current/v21.1/cockroach-commands.md +++ /dev/null @@ -1,56 +0,0 @@ ---- -title: Cockroach Commands -summary: Learn the commands for configuring, starting, and managing a CockroachDB cluster. -toc: true ---- - -This page introduces the `cockroach` commands for configuring, starting, and managing a CockroachDB cluster, as well as environment variables that can be used in place of certain flags. - -You can run `cockroach help` in your shell to get similar guidance. - - -## Commands - -Command | Usage ---------|---- -[`cockroach start`](cockroach-start.html) | Start a node as part of a multi-node cluster. -[`cockroach init`](cockroach-init.html) | Initialize a multi-node cluster. -[`cockroach start-single-node`](cockroach-start-single-node.html) | Start a single-node cluster. -[`cockroach cert`](cockroach-cert.html) | Create CA, node, and client certificates. -[`cockroach quit`](cockroach-quit.html) | Temporarily stop a node or permanently remove a node. -[`cockroach sql`](cockroach-sql.html) | Use the built-in SQL client. -[`cockroach sqlfmt`](cockroach-sqlfmt.html) | Reformat SQL queries for enhanced clarity. -[`cockroach node`](cockroach-node.html) | List node IDs, show their status, decommission nodes for removal, or recommission nodes. -[`cockroach auth-session`](cockroach-auth-session.html) | Create and manage web sessions and authentication tokens to the HTTP interface from the command line. -[`cockroach dump`](cockroach-dump.html) | **Deprecated.** Use one of the following instead:
  • To back up your data, take a [full backup](take-full-and-incremental-backups.html).
  • To export your data in plaintext format, use [`EXPORT`](export.html).
  • To view table schema in plaintext, use [`SHOW CREATE TABLE`](show-create.html).
-[`cockroach demo`](cockroach-demo.html) | Start a temporary, in-memory CockroachDB cluster, and open an interactive SQL shell to it. -[`cockroach gen`](cockroach-gen.html) | Generate manpages, a bash completion file, example SQL data, or an HAProxy configuration file for a running cluster. -[`cockroach version`](cockroach-version.html) | Output CockroachDB version details. -[`cockroach debug ballast`](cockroach-debug-ballast.html) | Create a large, unused file in a node's storage directory that you can delete if the node runs out of disk space. -[`cockroach debug encryption-active-key`](cockroach-debug-encryption-active-key.html) | View the encryption algorithm and store key. -[`cockroach debug zip`](cockroach-debug-zip.html) | Generate a `.zip` file that can help Cockroach Labs troubleshoot issues with your cluster. -[`cockroach debug merge-logs`](cockroach-debug-merge-logs.html) | Merge multiple log files from different machines into a single stream. -[`cockroach workload`](cockroach-workload.html) | Run a built-in load generator against a cluster. -[`cockroach nodelocal upload`](cockroach-nodelocal-upload.html) | Upload a file to the `externalIODir` on a node's local file system. -[`cockroach userfile upload`](cockroach-userfile-upload.html) | The cockroach userfile upload command uploads a file to user-scoped file storage. -[`cockroach userfile list`](cockroach-userfile-list.html) | List the files stored in the user-scoped file storage. -[`cockroach userfile delete`](cockroach-userfile-delete.html) | Deletes the files stored in the user-scoped file storage. -[`cockroach userfile get`](cockroach-userfile-get.html) | Fetch a file from the user-scoped file storage. -[`cockroach import`](cockroach-import.html) | **New in v21.1:** Import a table or database from a local dump file into a running cluster. `PGDUMP` and `MYSQLDUMP` file formats are currently supported. - -## Environment variables - -For many common `cockroach` flags, such as `--port` and `--user`, you can set environment variables once instead of manually passing the flags each time you execute commands. - -- To find out which flags support environment variables, see the documentation for each [command](#commands). -- To output the current configuration of CockroachDB and other environment variables, run `env`. -- When a node uses environment variables on [startup](cockroach-start.html), the variable names are printed to the node's logs; however, the variable values are not. - -CockroachDB prioritizes command flags, environment variables, and defaults as follows: - -1. If a flag is set for a command, CockroachDB uses it. -2. If a flag is not set for a command, CockroachDB uses the corresponding environment variable. -3. If neither the flag nor environment variable is set, CockroachDB uses the default for the flag. -4. If there's no flag default, CockroachDB gives an error. - -For more details, see [Client Connection Parameters](connection-parameters.html). diff --git a/src/current/v21.1/cockroach-debug-ballast.md b/src/current/v21.1/cockroach-debug-ballast.md deleted file mode 100644 index 369d7718266..00000000000 --- a/src/current/v21.1/cockroach-debug-ballast.md +++ /dev/null @@ -1,58 +0,0 @@ ---- -title: cockroach debug ballast -summary: Create a large, unused file in a node's storage directory that you can delete if the node runs out of disk space. -toc: true ---- - -The `cockroach debug ballast` [command](cockroach-commands.html) creates a large, unused file that you can place in a node's storage directory. In the case that a node runs out of disk space and shuts down, you can delete the ballast file to free up enough space to be able to restart the node. - -- Do not run `cockroach debug ballast` with a unix `root` user. Doing so brings the risk of mistakenly affecting system directories or files. -- {% include_cached new-in.html version="v21.1" %} `cockroach debug ballast` now refuses to overwrite the target ballast file if it already exists. This change is intended to prevent mistaken uses of the `ballast` command. Consider adding an `rm` command to scripts that integrate `cockroach debug ballast`, or provide a new file name every time and then remove the old file. -- In addition to placing a ballast file in each node's storage directory, it is important to actively [monitor remaining disk space](monitoring-and-alerting.html#events-to-alert-on). -- Ballast files may be created in many ways, including the standard `dd` command. `cockroach debug ballast` uses the `fallocate` system call when available, so it will be faster than `dd`. - -## Subcommands - -{% include {{ page.version.version }}/misc/debug-subcommands.md %} - -## Synopsis - -Create a ballast file: - -~~~ shell -$ cockroach debug ballast [path to ballast file] [flags] -~~~ - -View help: - -~~~ shell -$ cockroach debug ballast --help -~~~ - -## Flags - -Flag | Description ------|----------- -`--size`
`-z` | The amount of space to fill, or to leave available, in a node's storage directory via a ballast file. Positive values equal the size of the ballast file. Negative values equal the amount of space to leave after creating the ballast file. This can be a percentage (notated as a decimal or with %) or any bytes-based unit, for example:

`--size=1000000000 ----> 1000000000 bytes`
`--size=1GiB ----> 1073741824 bytes`
`--size=5% ----> 5% of available space`
`--size=0.05 ----> 5% of available space`
`--size=.05 ----> 5% of available space`

**Default:** `1GB` - -## Examples - -### Create a 1GB ballast file (default) - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach debug ballast cockroach-data/ballast.txt -~~~ - -### Create a ballast file of a different size - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach debug ballast cockroach-data/ballast.txt --size=2GB -~~~ - -## See also - -- [Other Cockroach Commands](cockroach-commands.html) -- [Troubleshooting Overview](troubleshooting-overview.html) -- [Production Checklist](recommended-production-settings.html) diff --git a/src/current/v21.1/cockroach-debug-encryption-active-key.md b/src/current/v21.1/cockroach-debug-encryption-active-key.md deleted file mode 100644 index 43151f20344..00000000000 --- a/src/current/v21.1/cockroach-debug-encryption-active-key.md +++ /dev/null @@ -1,44 +0,0 @@ ---- -title: cockroach debug encryption-active-key -summary: Learn the command for viewing the algorithm and store key for an encrypted store. -toc: true -key: debug-encryption-active-key.html ---- - -The `cockroach debug encryption-active-key` [command](cockroach-commands.html) displays the encryption algorithm and store key for an encrypted store. - -## Synopsis - -~~~ shell -$ cockroach debug encryption-active-key [path specified by the store flag] -~~~ - -## Subcommands - -{% include {{ page.version.version }}/misc/debug-subcommands.md %} - -## Example - -Start a node with encryption-at-rest enabled: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start --store=cockroach-data --enterprise-encryption=path=cockroach-data,key=aes-128.key,old-key=plain --insecure --certs-dir=certs -~~~ - -View the encryption algorithm and store key: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach debug encryption-active-key cockroach-data -~~~ - -~~~ -AES128_CTR:be235c29239aa84a48e5e1874d76aebf7fb3c1bdc438cec2eb98de82f06a57a0 -~~~ - -## See also - -- [File an Issue](file-an-issue.html) -- [Other Cockroach Commands](cockroach-commands.html) -- [Troubleshooting Overview](troubleshooting-overview.html) diff --git a/src/current/v21.1/cockroach-debug-list-files.md b/src/current/v21.1/cockroach-debug-list-files.md deleted file mode 100644 index baef85ed957..00000000000 --- a/src/current/v21.1/cockroach-debug-list-files.md +++ /dev/null @@ -1,75 +0,0 @@ ---- -title: cockroach debug list-files -summary: Learn the command for listing the files collected in the debug zip. -toc: true -key: debug-list-files.html ---- - -The `cockroach debug list-files` [command](cockroach-commands.html) shows the files that will be collected by using [`cockroach debug zip`](cockroach-debug-zip.html). - -{{site.data.alerts.callout_info}} -The files listed include logs, heap profiles, goroutine dumps, and CPU profiles. Other [files](cockroach-debug-zip.html#files) generated by `cockroach debug zip` are not listed by `cockroach debug list-files`. -{{site.data.alerts.end}} - -## Subcommands - -{% include {{ page.version.version }}/misc/debug-subcommands.md %} - -## Synopsis - -~~~ shell -$ cockroach debug list-files {flags} -~~~ - -## Flags - -The `debug list-files` subcommand supports the following [general-use](#general), [client connection](#client-connection), and [logging](#logging) flags. - -### General - -Flag | Description ------|----------- -`--exclude-files` | [Files](cockroach-debug-zip.html#files) to exclude from the generated `.zip`. This can be used to limit the size of the generated `.zip`, and affects logs, heap profiles, goroutine dumps, and/or CPU profiles. The files are specified as a comma-separated list of [glob patterns](https://en.wikipedia.org/wiki/Glob_(programming)). For example:

`--exclude-files=*.log`

Note that this flag is applied _after_ `--include_files`. -`--exclude-nodes` | Specify nodes to exclude from inspection as a comma-separated list or range of node IDs. For example:

`--exclude-nodes=1,10,13-15` -`--files-from` | Start timestamp for log file, goroutine dump, and heap profile collection. This can be used to limit the size of the generated `.zip`, which is increased by these files. The timestamp uses the format `YYYY-MM-DD`, followed optionally by `HH:MM:SS` or `HH:MM`. For example:

`--files-from='2021-07-01 15:00'`

When specifying a narrow time window, we recommend adding extra seconds/minutes to account for uncertainties such as clock drift.

**Default:** 48 hours before now -`--files-until` | End timestamp for log file, goroutine dump, and heap profile collection. This can be used to limit the size of the generated `.zip`, which is increased by these files. The timestamp uses the format `YYYY-MM-DD`, followed optionally by `HH:MM:SS` or `HH:MM`. For example:

`--files-until='2021-07-01 16:00'`

When specifying a narrow time window, we recommend adding extra seconds/minutes to account for uncertainties such as clock drift.

**Default:** 24 hours beyond now (to include files created during `.zip` creation) -`--format` | Specify a format to display table rows. This can be `tsv`, `csv`, `table`, `records`, `sql`, `raw`, or `html`.

**Default:** `table` (interactive sessions), `tsv` (non-interactive sessions) -`--include-files` | [Files](cockroach-debug-zip.html#files) to include in the generated `.zip`. This can be used to limit the size of the generated `.zip`, and affects logs, heap profiles, goroutine dumps, and/or CPU profiles. The files are specified as a comma-separated list of [glob patterns](https://en.wikipedia.org/wiki/Glob_(programming)). For example:

`--include-files=*.pprof`

Note that this flag is applied _before_ `--exclude-files`. -`--nodes` | Specify nodes to inspect as a comma-separated list or range of node IDs. For example:

`--nodes=1,10,13-15` - -### Client connection - -{% include {{ page.version.version }}/sql/connection-parameters.md %} -`--cluster-name` | The cluster name to use to verify the cluster's identity. If the cluster has a cluster name, you must include this flag. For more information, see [`cockroach start`](cockroach-start.html#general). -`--disable-cluster-name-verification` | Disables the cluster name check for this command. This flag must be paired with `--cluster-name`. For more information, see [`cockroach start`](cockroach-start.html#general). - -### Logging - -{% include {{ page.version.version }}/misc/debug-subcommands.md %} - -## Examples - -### List all collected files - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach debug list-files -~~~ - -{{site.data.alerts.callout_info}} -The files listed include logs, heap profiles, goroutine dumps, and CPU profiles. Other [files](cockroach-debug-zip.html#files) generated by `cockroach debug zip` are not listed by `cockroach debug list-files`. -{{site.data.alerts.end}} - -### List all collected log files - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach debug list-files --include-files=*.log -~~~ - -### List all collected files (TSV format) - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach debug list-files --format=tsv -~~~ diff --git a/src/current/v21.1/cockroach-debug-merge-logs.md b/src/current/v21.1/cockroach-debug-merge-logs.md deleted file mode 100644 index 7b30d96a9e5..00000000000 --- a/src/current/v21.1/cockroach-debug-merge-logs.md +++ /dev/null @@ -1,83 +0,0 @@ ---- -title: cockroach debug merge-logs -summary: Learn the command for merging the collected debug logs from all nodes in your cluster. -toc: true -key: debug-merge-logs.html ---- - -The `cockroach debug merge-logs` [command](cockroach-commands.html) merges log files from multiple nodes into a single time-ordered stream of messages with an added per-message prefix to indicate the corresponding node. You can use it in conjunction with logs collected using the [`debug zip`](cockroach-debug-zip.html) command to aid in debugging. - -{{site.data.alerts.callout_danger}} -The file produced by `cockroach debug zip` can contain highly [sensitive, identifiable information](configure-logs.html#redact-logs), such as usernames, hashed passwords, and possibly your table's data. You can use the [`--redact`](#example) flag to redact the sensitive data out of log files and crash reports before sharing them with Cockroach Labs. -{{site.data.alerts.end}} - -## Subcommands - -{% include {{ page.version.version }}/misc/debug-subcommands.md %} - -## Synopsis - -~~~ shell -$ cockroach debug merge-logs [log file directory] [flags] -~~~ - -## Flags - -Use the following flags to filter the `debug merge-logs` results for a specified regular expression or time range. - -Flag | Description ------|----------- -`--filter` | Limit the results to the specified regular expression -`--from` | Start time for the time range filter. -`--to` | End time for the time range filter. -`--redact` | Redact [sensitive data](configure-logs.html#redact-logs) from the log files. - -## Example - -Generate a debug zip file: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach debug zip ./cockroach-data/logs/debug.zip --insecure -~~~ - -Unzip the file: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ unzip ./cockroach-data/logs/debug.zip -~~~ - -Merge the logs in the debug folder: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach debug merge-logs debug/nodes/*/logs/* -~~~ - -Alternatively, filter the merged logs for a specified time range: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach debug merge-logs debug/nodes/*/logs/* --from="220713 18:36:28.208553" --to="220713 18:36:29.232864" -~~~ - -You can also filter the merged logs for a regular expression: - -{% include_cached copy-clipboard.html %} -~~~ shell -cockroach debug merge-logs debug/nodes/*/logs/* --filter="RUNNING IN INSECURE MODE" -~~~ - -You can redact sensitive information from the merged logs: - -{% include_cached copy-clipboard.html %} -~~~ shell -cockroach debug merge-logs --redact debug/nodes/*/logs/* -~~~ - -## See also - -- [File an Issue](file-an-issue.html) -- [Other Cockroach Commands](cockroach-commands.html) -- [Troubleshooting Overview](troubleshooting-overview.html) diff --git a/src/current/v21.1/cockroach-debug-zip.md b/src/current/v21.1/cockroach-debug-zip.md deleted file mode 100644 index cd6d7e8af57..00000000000 --- a/src/current/v21.1/cockroach-debug-zip.md +++ /dev/null @@ -1,168 +0,0 @@ ---- -title: cockroach debug zip -summary: Learn the commands for collecting debug information from all nodes in your cluster. -toc: true -key: debug-zip.html ---- - -The `cockroach debug zip` [command](cockroach-commands.html) connects to your cluster and gathers information from each active node into a single `.zip` file (inactive nodes are not included). For details on the `.zip` contents, see [Files](#files). - -You can use the [`cockroach debug merge-logs`](cockroach-debug-merge-logs.html) command in conjunction with `cockroach debug zip` to merge the collected logs into one file, making them easier to parse. - -{{site.data.alerts.callout_danger}} -The files produced by `cockroach debug zip` can contain highly [sensitive, identifiable information](configure-logs.html#redact-logs), such as usernames, hashed passwords, and possibly table data. Use the [`--redact-logs`](#redact-sensitive-information-from-the-logs) flag to redact sensitive data from log files and crash reports before sharing them with Cockroach Labs. -{{site.data.alerts.end}} - -## Details - -### Use cases - -There are two scenarios in which `debug zip` is useful: - -- To collect all of your nodes' logs, which you can then parse to locate issues. You can optionally use the [flags](#flags) to [retrieve only the log files](#generate-a-debug-zip-file-with-logs-only). For more information about logs, see [Logging](logging-overview.html). Also note: - - - Nodes that are currently [down](cluster-setup-troubleshooting.html#node-liveness-issues) cannot deliver their logs over the network. For these nodes, you must log on to the machine where the `cockroach` process would otherwise be running, and gather the files manually. - - - Nodes that are currently up but disconnected from other nodes (e.g., because of a [network partition](cluster-setup-troubleshooting.html#network-partition)) may not be able to respond to `debug zip` requests forwarded by other nodes, but can still respond to requests for data when asked directly. In such situations, we recommend using the [`--host` flag](#client-connection) to point `debug zip` at each of the disconnected nodes until data has been gathered for the entire cluster. - -- If you experience severe or difficult-to-reproduce issues with your cluster, Cockroach Labs might ask you to send us your cluster's debugging information using `cockroach debug zip`. - -### Files - -{{site.data.alerts.callout_success}} -Log files, heap profiles, CPU profiles, and goroutine dumps can greatly increase the size of the `cockroach debug zip` output. If you are collecting data from a large cluster, we recommend first experimenting with [`cockroach debug list-files`](cockroach-debug-list-files.html) and the `--exclude-files`, `--include-files`, `--files-from`, and/or `--files-until` [flags](#flags) to limit the `.zip` file size. -{{site.data.alerts.end}} - -These files collected by `cockroach debug zip`, which are found in the individual node directories, can be filtered using the `--exclude-files`, `--include-files`, `--files-from`, and/or `--files-until` [flags](#flags): - -| Information | Filename | -|--------------------------------------------------|--------------------------------------------------------------------------------------| -| [Log files](configure-logs.html#log-file-naming) | `cockroach-{log-file-group}.{host}.{user}.{start timestamp in UTC}.{process ID}.log` | -| Goroutine dumps | `goroutine_dump.{date-and-time}.{metadata}.double_since_last_dump.{metadata}.txt.gz` | -| Heap profiles | `memprof.{date-and-time}.{heapsize}.pprof` | -| Memory statistics | `memstats.{date-and-time}.{heapsize}.txt` | -| CPU profiles | `cpuprof.{date-and-time}` | - -The following information is also contained in the `.zip` file, and cannot be filtered: - -- Cluster events -- Database details -- Schema change events -- Database, table, node, and range lists -- Node details -- Node liveness -- Gossip data -- Stack traces -- Range details -- Jobs -- [Cluster Settings](cluster-settings.html) -- [Metrics](ui-custom-chart-debug-page.html#available-metrics) -- Problem ranges -- Thread stack traces (Linux only) -- CPU profiles -- A script (`hot-ranges.sh`) that summarizes the hottest ranges (ranges receiving a high number of reads or writes) - -## Subcommands - -{% include {{ page.version.version }}/misc/debug-subcommands.md %} - -## Synopsis - -~~~ shell -$ cockroach debug zip {ZIP file destination} {flags} -~~~ - -{{site.data.alerts.callout_info}} -The following [flags](#flags) must apply to an active CockroachDB node. If no nodes are live, you must [start at least one node](cockroach-start.html). -{{site.data.alerts.end}} - -## Flags - -The `debug zip` subcommand supports the following [general-use](#general), [client connection](#client-connection), and [logging](#logging) flags. - -### General - -Flag | Description ------|----------- -`--cpu-profile-duration` | Fetch CPU profiles from the cluster with the specified sample duration in seconds. The `debug zip` command will block for the duration specified. A value of `0` disables this feature.

**Default:** `5` -`--concurrency` | The maximum number of nodes to concurrently poll for data. This can be any value between `1` and `15`. -`--exclude-files` | [Files](#files) to exclude from the generated `.zip`. This can be used to limit the size of the generated `.zip`, and affects logs, heap profiles, goroutine dumps, and/or CPU profiles. The files are specified as a comma-separated list of [glob patterns](https://en.wikipedia.org/wiki/Glob_(programming)). For example:

`--exclude-files=*.log`

Note that this flag is applied _after_ `--include_files`. Use [`cockroach debug list-files`](cockroach-debug-list-files.html) with this flag to see a list of files that will be contained in the `.zip`. -`--exclude-nodes` | Specify nodes to exclude from inspection as a comma-separated list or range of node IDs. For example:

`--exclude-nodes=1,10,13-15` -`--files-from` | Start timestamp for log file, goroutine dump, and heap profile collection. This can be used to limit the size of the generated `.zip`, which is increased by these files. The timestamp uses the format `YYYY-MM-DD`, followed optionally by `HH:MM:SS` or `HH:MM`. For example:

`--files-from='2021-07-01 15:00'`

When specifying a narrow time window, we recommend adding extra seconds/minutes to account for uncertainties such as clock drift.

**Default:** 48 hours before now -`--files-until` | End timestamp for log file, goroutine dump, and heap profile collection. This can be used to limit the size of the generated `.zip`, which is increased by these files. The timestamp uses the format `YYYY-MM-DD`, followed optionally by `HH:MM:SS` or `HH:MM`. For example:

`--files-until='2021-07-01 16:00'`

When specifying a narrow time window, we recommend adding extra seconds/minutes to account for uncertainties such as clock drift.

**Default:** 24 hours beyond now (to include files created during `.zip` creation) -`--include-files` | [Files](#files) to include in the generated `.zip`. This can be used to limit the size of the generated `.zip`, and affects logs, heap profiles, goroutine dumps, and/or CPU profiles. The files are specified as a comma-separated list of [glob patterns](https://en.wikipedia.org/wiki/Glob_(programming)). For example:

`--include-files=*.pprof`

Note that this flag is applied _before_ `--exclude-files`. Use [`cockroach debug list-files`](cockroach-debug-list-files.html) with this flag to see a list of files that will be contained in the `.zip`. -`--nodes` | Specify nodes to inspect as a comma-separated list or range of node IDs. For example:

`--nodes=1,10,13-15` -`--redact-logs` | Redact [sensitive data](configure-logs.html#redact-logs) from the log files. Note that this flag removes sensitive information only from the log files. The other items (listed above) collected by the `debug zip` command may still contain sensitive information. -`--timeout` | Return an error if the command does not conclude within a specified nonzero value. The timeout is suffixed with `s` (seconds), `m` (minutes), or `h` (hours). For example:

`--timeout=2m` - -### Client connection - -Flag | Description ------|------------ -`--cert-principal-map` | A comma-separated list of `:` mappings. This allows mapping the principal in a cert to a DB principal such as `node` or `root` or any SQL user. This is intended for use in situations where the certificate management system places restrictions on the `Subject.CommonName` or `SubjectAlternateName` fields in the certificate (e.g., disallowing a `CommonName` like `node` or `root`). If multiple mappings are provided for the same ``, the last one specified in the list takes precedence. A principal not specified in the map is passed through as-is via the identity function. A cert is allowed to authenticate a DB principal if the DB principal name is contained in the mapped `CommonName` or DNS-type `SubjectAlternateName` fields. -`--certs-dir` | The path to the [certificate directory](cockroach-cert.html) containing the CA and client certificates and client key.

**Env Variable:** `COCKROACH_CERTS_DIR`
**Default:** `${HOME}/.cockroach-certs/` -`--cluster-name` | The cluster name to use to verify the cluster's identity. If the cluster has a cluster name, you must include this flag. For more information, see [`cockroach start`](cockroach-start.html#general). -`--disable-cluster-name-verification` | Disables the cluster name check for this command. This flag must be paired with `--cluster-name`. For more information, see [`cockroach start`](cockroach-start.html#general). -`--host` | The server host and port number to connect to. This can be the address of any node in the cluster.

**Env Variable:** `COCKROACH_HOST`
**Default:** `localhost:26257` -`--insecure` | Use an insecure connection.

**Env Variable:** `COCKROACH_INSECURE`
**Default:** `false` - `--url` | A [connection URL](connection-parameters.html#connect-using-a-url) to use instead of the other arguments.

**Env Variable:** `COCKROACH_URL`
**Default:** no URL - -### Logging - -{% include {{ page.version.version }}/misc/logging-defaults.md %} - -## Examples - -### Generate a debug zip file - -Generate the debug zip file for an insecure cluster: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach debug zip ./cockroach-data/logs/debug.zip --insecure --host=200.100.50.25 -~~~ - -Generate the debug zip file for a secure cluster: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach debug zip ./cockroach-data/logs/debug.zip --host=200.100.50.25 -~~~ - -{{site.data.alerts.callout_info}} -Secure examples assume you have the appropriate certificates in the default certificate directory, `${HOME}/.cockroach-certs/`. -{{site.data.alerts.end}} - -### Generate a debug zip file with logs only - -Generate a debug zip file containing only log files: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach debug zip ./cockroach-data/logs/debug.zip --include-files=*.log -~~~ - -### Redact sensitive information from the logs - -Example of a log string without redaction enabled: - -~~~ -server/server.go:1423 ⋮ password of user ‹admin› was set to ‹"s3cr34?!@x_"› -~~~ - -Enable log redaction: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach debug zip ./cockroach-data/logs/debug.zip --redact-logs --insecure --host=200.100.50.25 -~~~ - -~~~ -server/server.go:1423 ⋮ password of user ‹×› was set to ‹×› -~~~ - -## See also - -- [File an Issue](file-an-issue.html) -- [Other Cockroach Commands](cockroach-commands.html) -- [Troubleshooting Overview](troubleshooting-overview.html) diff --git a/src/current/v21.1/cockroach-demo.md b/src/current/v21.1/cockroach-demo.md deleted file mode 100644 index a85e9045477..00000000000 --- a/src/current/v21.1/cockroach-demo.md +++ /dev/null @@ -1,674 +0,0 @@ ---- -title: cockroach demo -summary: Use cockroach demo to open a SQL shell to a temporary, in-memory, CockroachDB cluster. -toc: true ---- - -The `cockroach demo` [command](cockroach-commands.html) starts a temporary, in-memory CockroachDB cluster of one or more nodes, with or without a preloaded dataset, and opens an interactive SQL shell to the cluster. - -- All [SQL shell](#sql-shell) commands, client-side options, help, and shortcuts supported by the [`cockroach sql`](cockroach-sql.html) command are also supported by `cockroach demo`. -- The in-memory cluster persists only as long as the SQL shell is open. As soon as the shell is exited, the cluster and all its data are permanently destroyed. This command is therefore recommended only as an easy way to experiment with the CockroachDB SQL dialect. -- By default, `cockroach demo` starts in secure mode using TLS certificates to encrypt network communication. It also serves a local [DB Console](#connection-parameters) that does not use TLS encryption. -- Each instance of `cockroach demo` loads a temporary [Enterprise license](https://www.cockroachlabs.com/get-cockroachdb/enterprise/) that expires after 24 hours. To prevent the loading of a temporary license, set the `--disable-demo-license` flag. -- {% include_cached new-in.html version="v21.1" %} `cockroach demo` opens the SQL shell with a new [SQL user](authorization.html#sql-users) named `demo`. The `demo` user is assigned a random password and granted the [`admin` role](authorization.html#admin-role). - -{{site.data.alerts.callout_danger}} -`cockroach demo` is designed for testing purposes only. It is not suitable for production deployments. To see a list of recommendations for production deployments, see the [Production Checklist](recommended-production-settings.html). -{{site.data.alerts.end}} - -## Synopsis - -View help for `cockroach demo`: - -~~~ shell -$ cockroach demo --help -~~~ - -Start a single-node demo cluster with the `movr` dataset pre-loaded: - -~~~ shell -$ cockroach demo -~~~ - -Load a different dataset into a demo cluster: - -~~~ shell -$ cockroach demo -~~~ - -Run the `movr` workload against a demo cluster: - -~~~ shell -$ cockroach demo --with-load -~~~ - -Execute SQL from the command line against a demo cluster: - -~~~ shell -$ cockroach demo --execute=";" --execute="" -~~~ - -Start a multi-node demo cluster: - -~~~ shell -$ cockroach demo --nodes= -~~~ - -Start a multi-region demo cluster with default region and zone localities: - -~~~ shell -$ cockroach demo --global --nodes= -~~~ - -Start a multi-region demo cluster with manually defined localities: - -~~~ shell -$ cockroach demo --nodes= --demo-locality= -~~~ - -Stop a demo cluster: - -~~~ sql -> \q -~~~ - -~~~ sql -> quit -~~~ - -~~~ sql -> exit -~~~ - -~~~ shell -ctrl-d -~~~ - - -## Datasets - -{{site.data.alerts.callout_success}} -By default, the `movr` dataset is pre-loaded into a demo cluster. To load a different dataset, use [`cockroach demo `](#load-a-sample-dataset-into-a-demo-cluster). To start a demo cluster without a pre-loaded dataset, pass the `--no-example-database` flag. -{{site.data.alerts.end}} - -Workload | Description ----------|------------ -`bank` | A `bank` database, with one `bank` table containing account details. -`intro` | An `intro` database, with one table, `mytable`, with a hidden message. -`kv` | A `kv` database, with one key-value-style table. -`movr` | A `movr` database, with several tables of data for the [MovR example application](movr.html).

By default, `cockroach demo` loads the `movr` database as the [current database](sql-name-resolution.html#current-database), with sample region (`region`) and availability zone (`az`) replica localities for each node specified with the [`--nodes` flag](cockroach-demo.html#flags). -`startrek` | A `startrek` database, with two tables, `episodes` and `quotes`. -`tpcc` | A `tpcc` database, with a rich schema of multiple tables. -`ycsb` | A `ycsb` database, with a `usertable` from the Yahoo! Cloud Serving Benchmark. - -## Flags - -### General - -The `demo` command supports the following general-use flags. - -Flag | Description ------|------------ -`--cache` | For each demo node, the total size for caches. This can be a percentage (notated as a decimal or with `%`) or any bytes-based unit, for example:

`--cache=.25`
`--cache=25%`
`--cache=1000000000 ----> 1000000000 bytes`
`--cache=1GB ----> 1000000000 bytes`
`--cache=1GiB ----> 1073741824 bytes`

**Default:** `64MiB` -`--demo-locality` | Specify [locality](cockroach-start.html#locality) information for each demo node. The input is a colon-separated list of key-value pairs, where the ith pair is the locality setting for the ith demo cockroach node.

For example, the following option assigns node 1's region to `us-east1` and availability zone to `1`, node 2's region to `us-east2` and availability zone to `2`, and node 3's region to `us-east3` and availability zone to `3`:

`--demo-locality=region=us-east1,az=1:region=us-east1,az=2:region=us-east1,az=3`

By default, `cockroach demo` uses sample region (`region`) and availability zone (`az`) replica localities for each node specified with the `--nodes` flag. -`--disable-demo-license` | Start the demo cluster without loading a temporary [Enterprise license](https://www.cockroachlabs.com/get-started-cockroachdb/) that expires after 24 hours.

Setting the `COCKROACH_SKIP_ENABLING_DIAGNOSTIC_REPORTING` environment variable will also prevent the loading of a temporary license, along with preventing the sharing of anonymized [diagnostic details](diagnostics-reporting.html) with Cockroach Labs. -`--echo-sql` | Reveal the SQL statements sent implicitly by the command-line utility. This can also be enabled within the interactive SQL shell via the `\set echo` [shell command](#commands). -`--embedded` | **New in v21.1:** Minimizes the SQL shell [welcome text](#welcome-text) to be appropriate for embedding in playground-type environments. Specifically, this flag removes details that users in an embedded environment have no control over (e.g., networking information). -`--no-example-database` | **New in v21.1:** Start the demo cluster without a pre-loaded dataset.
To obtain this behavior automatically in every new `cockroach demo` session, set the `COCKROACH_NO_EXAMPLE_DATABASE` environment variable to `true`. -`--execute`

`-e` | Execute SQL statements directly from the command line, without opening a shell. This flag can be set multiple times, and each instance can contain one or more statements separated by semi-colons.

If an error occurs in any statement, the command exits with a non-zero status code and further statements are not executed. The results of each statement are printed to the standard output (see `--format` for formatting options). -`--format` | How to display table rows printed to the standard output. Possible values: `tsv`, `csv`, `table`, `raw`, `records`, `sql`, `html`.

**Default:** `table` for sessions that [output on a terminal](cockroach-sql.html#session-and-output-types); `tsv` otherwise

This flag corresponds to the `display_format` [client-side option](#client-side-options) for use in interactive sessions. -`--geo-partitioned-replicas` | Start a 9-node demo cluster with [geo-partitioning](partitioning.html) applied to the [`movr`](movr.html) database. -`--global` | This experimental flag is used to simulate a [multi-region cluster](simulate-a-multi-region-cluster-on-localhost.html) which sets the [`--locality` flag on node startup](cockroach-start.html#locality) to three different regions. It also simulates the network latency that would occur between them given the specified localities. In order for this to operate as expected, with 3 nodes in each of 3 regions, you must also pass the `--nodes 9` argument. -`--http-port` | **New in v21.1:** Specifies a custom HTTP port to the [DB Console](ui-overview.html) for the first node of the demo cluster.

In multi-node clusters, the HTTP ports for additional clusters increase from the port of the first node, in increments of 1. For example, if the first node has an HTTP port of `5000`, the second node will have the HTTP port `5001`. -`--insecure` | Include this to start the demo cluster in insecure mode.

**Env Variable:** `COCKROACH_INSECURE` -`--max-sql-memory` | For each demo node, the maximum in-memory storage capacity for temporary SQL data, including prepared queries and intermediate data rows during query execution. This can be a percentage (notated as a decimal or with `%`) or any bytes-based unit, for example:

`--max-sql-memory=.25`
`--max-sql-memory=25%`
`--max-sql-memory=10000000000 ----> 1000000000 bytes`
`--max-sql-memory=1GB ----> 1000000000 bytes`
`--max-sql-memory=1GiB ----> 1073741824 bytes`

**Default:** `128MiB` -`--nodes` | Specify the number of in-memory nodes to create for the demo.

**Default:** 1 -`--safe-updates` | Disallow potentially unsafe SQL statements, including `DELETE` without a `WHERE` clause, `UPDATE` without a `WHERE` clause, and `ALTER TABLE ... DROP COLUMN`.

**Default:** `true` for [interactive sessions](cockroach-sql.html#session-and-output-types); `false` otherwise

Potentially unsafe SQL statements can also be allowed/disallowed for an entire session via the `sql_safe_updates` [session variable](set-vars.html). -`--set` | Set a [client-side option](#client-side-options) before starting the SQL shell or executing SQL statements from the command line via `--execute`. This flag may be specified multiple times, once per option.

After starting the SQL shell, the `\set` and `unset` commands can be use to enable and disable client-side options as well. -`--sql-port` | **New in v21.1:** Specifies a custom SQL port for the first node of the demo cluster.

In multi-node clusters, the SQL ports for additional clusters increase from the port of the first node, in increments of 1. For example, if the first node has the SQL port `3000`, the second node will the SQL port `3001`. -`--with-load` | Run a demo [`movr`](movr.html) workload against the preloaded `movr` database.

When running a multi-node demo cluster, load is balanced across all nodes. - -### Logging - -By default, the `demo` command does not log messages. - -If you need to troubleshoot this command's behavior, you can [customize its logging behavior](configure-logs.html). - -## SQL shell - -### Welcome text - -When the SQL shell connects to the demo cluster at startup, it prints a welcome text with some tips and cluster details. Most of these details resemble the [welcome text](cockroach-sql.html#welcome-message) that is printed when connecting `cockroach sql` to a permanent cluster. `cockroach demo` also includes some [connection parameters](#connection-parameters) for connecting to the DB Console or for connecting another SQL client to the demo cluster. - -~~~ shell -# -# Welcome to the CockroachDB demo database! -# -# You are connected to a temporary, in-memory CockroachDB cluster of 9 nodes. -# -# This demo session will attempt to enable Enterprise features -# by acquiring a temporary license from Cockroach Labs in the background. -# To disable this behavior, set the environment variable -# COCKROACH_SKIP_ENABLING_DIAGNOSTIC_REPORTING=true. -# -# Beginning initialization of the movr dataset, please wait... -# -# Waiting for license acquisition to complete... -# -# Partitioning the demo database, please wait... -# -# The cluster has been preloaded with the "movr" dataset -# (MovR is a fictional vehicle sharing company). -# -# Reminder: your changes to data stored in the demo session will not be saved! -# -# If you wish to access this demo cluster using another tool, you will need -# the following details: -# -# - Connection parameters: -# (webui) http://127.0.0.1:8080/demologin?password=demo55826&username=demo -# (sql) postgresql://demo:demo55826@127.0.0.1:26257/movr?sslmode=require -# (sql/jdbc) jdbc:postgresql://127.0.0.1:26257/movr?password=demo55826&sslmode=require&user=demo -# (sql/unix) postgresql://demo:demo55826@/movr?host=%2Fvar%2Ffolders%2F8c%2F915dtgrx5_57bvc5tq4kpvqr0000gn%2FT%2Fdemo699845497&port=26257 -# -# To display connection parameters for other nodes, use \demo ls. -# - Username: "demo", password: "demo55826" -# - Directory with certificate files (for certain SQL drivers/tools): /var/folders/8c/915dtgrx5_57bvc5tq4kpvqr0000gn/T/demo699845497 -# -# Server version: CockroachDB CCL v21.2.0-alpha.00000000-1724-gc5c74249f7 (x86_64-apple-darwin20.5.0, built 2021/06/23 21:26:17, go1.16) (same version as client) -# Cluster ID: f78b7feb-b6cf-4396-9d7f-494982d7d81e -# Organization: Cockroach Demo -# -# Enter \? for a brief introduction. -# -~~~ - -### Connection parameters - -The SQL shell welcome text includes connection parameters for accessing the DB Console and for connecting other SQL clients to the demo cluster: - -~~~ -# - Connection parameters: -# (webui) http://127.0.0.1:8080/demologin?password=demo55826&username=demo -# (sql) postgresql://demo:demo55826@127.0.0.1:26257/movr?sslmode=require -# (sql/jdbc) jdbc:postgresql://127.0.0.1:26257/movr?password=demo55826&sslmode=require&user=demo -# (sql/unix) postgresql://demo:demo55826@/movr?host=%2Fvar%2Ffolders%2F8c%2F915dtgrx5_57bvc5tq4kpvqr0000gn%2FT%2Fdemo699845497&port=26257 -~~~ - -Parameter | Description -----------|------------ -`webui` | Use this link to access a local [DB Console](ui-overview.html) to the demo cluster. -`sql` | Use this connection URL for standard sql/tcp connections from other SQL clients such as [`cockroach sql`](cockroach-sql.html).
The default SQL port for the first node of a demo cluster is `26257`. -`sql/unix` | Use this connection URL to establish a [Unix domain socket connection](cockroach-sql.html#connect-to-a-cluster-listening-for-unix-domain-socket-connections) with a client that is installed on the same machine. - -{{site.data.alerts.callout_info}} -You do not need to create or specify node and client certificates in `sql` or `sql/unix` connection URLs. Instead, you can securely connect to the demo cluster with the random password generated for the `demo` user. -{{site.data.alerts.end}} - -When running a multi-node demo cluster, use the `\demo ls` [shell command](#commands) to list the connection parameters for all nodes: - -{% include_cached copy-clipboard.html %} -~~~ sql -> \demo ls -~~~ - -~~~ -node 1: - (webui) http://127.0.0.1:8080/demologin?password=demo76950&username=demo - (sql) postgres://demo:demo76950@127.0.0.1:26257?sslmode=require - (sql/unix) postgres://demo:demo76950@?host=%2Fvar%2Ffolders%2Fc8%2Fb_q93vjj0ybfz0fz0z8vy9zc0000gp%2FT%2Fdemo070856957&port=26257 - -node 2: - (webui) http://127.0.0.1:8081/demologin?password=demo76950&username=demo - (sql) postgres://demo:demo76950@127.0.0.1:26258?sslmode=require - (sql/unix) postgres://demo:demo76950@?host=%2Fvar%2Ffolders%2Fc8%2Fb_q93vjj0ybfz0fz0z8vy9zc0000gp%2FT%2Fdemo070856957&port=26258 - -node 3: - (webui) http://127.0.0.1:8082/demologin?password=demo76950&username=demo - (sql) postgres://demo:demo76950@127.0.0.1:26259?sslmode=require - (sql/unix) postgres://demo:demo76950@?host=%2Fvar%2Ffolders%2Fc8%2Fb_q93vjj0ybfz0fz0z8vy9zc0000gp%2FT%2Fdemo070856957&port=26259 -~~~ - -### Commands - -- [General](#general) -- [Demo-specific](#demo-specific) - -#### General - -{% include {{ page.version.version }}/sql/shell-commands.md %} - -#### Demo-specific - -`cockroach demo` offers the following additional shell commands. Note that these commands are **experimental** and their interface and output are subject to change. - -Command | Usage ---------|------ -`\demo ls` | List the demo nodes and their connection URLs. -`\demo add region=,zone=` | **New in v21.1:** Add a node to a single-region or multi-region demo cluster. [See an example](#add-shut-down-and-restart-nodes-in-a-multi-node-demo-cluster). -`\demo shutdown ` | Shuts down a node in a multi-node demo cluster.

This command simulates stopping a node that can be restarted. [See an example](#add-shut-down-and-restart-nodes-in-a-multi-node-demo-cluster). -`\demo restart ` | Restarts a node in a multi-node demo cluster. [See an example](#add-shut-down-and-restart-nodes-in-a-multi-node-demo-cluster). -`\demo decommission ` | Decommissions a node in a multi-node demo cluster.

This command simulates [decommissioning a node](remove-nodes.html). -`\demo recommission ` | Recommissions a decommissioned node in a multi-node demo cluster. - -### Client-side options - -{% include {{ page.version.version }}/sql/shell-options.md %} - -### Help - -{% include {{ page.version.version }}/sql/shell-help.md %} - -### Shortcuts - -{% include {{ page.version.version }}/sql/shell-shortcuts.md %} - -## Diagnostics reporting - -By default, `cockroach demo` shares anonymous usage details with Cockroach Labs. To opt out, set the [`diagnostics.reporting.enabled`](diagnostics-reporting.html#after-cluster-initialization) [cluster setting](cluster-settings.html) to `false`. You can also opt out by setting the [`COCKROACH_SKIP_ENABLING_DIAGNOSTIC_REPORTING`](diagnostics-reporting.html#at-cluster-initialization) environment variable to `false` before running `cockroach demo`. - -## Examples - -In these examples, we demonstrate how to start a shell with `cockroach demo`. For more SQL shell features, see the [`cockroach sql` examples](cockroach-sql.html#examples). - -### Start a single-node demo cluster - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach demo -~~~ - -By default, `cockroach demo` loads the `movr` dataset in to the demo cluster: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES; -~~~ - -~~~ - schema_name | table_name | type | owner | estimated_row_count | locality ---------------+----------------------------+-------+-------+---------------------+----------- - public | promo_codes | table | demo | 1000 | NULL - public | rides | table | demo | 500 | NULL - public | user_promo_codes | table | demo | 0 | NULL - public | users | table | demo | 50 | NULL - public | vehicle_location_histories | table | demo | 1000 | NULL - public | vehicles | table | demo | 15 | NULL -(6 rows) -~~~ - -You can query the pre-loaded data: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT name FROM users LIMIT 10; -~~~ - -~~~ - name ------------------------ - Tyler Dalton - Dillon Martin - Deborah Carson - David Stanton - Maria Weber - Brian Campbell - Carl Mcguire - Jennifer Sanders - Cindy Medina - Daniel Hernandez MD -(10 rows) -~~~ - -You can also create and query new tables: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE drivers ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - city STRING NOT NULL, - name STRING, - dl STRING UNIQUE, - address STRING -); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO drivers (city, name) VALUES ('new york', 'Catherine Nelson'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM drivers; -~~~ - -~~~ - id | city | name | dl | address ----------------------------------------+----------+------------------+------+---------- - 4d363104-2c48-43b5-aa1e-955b81415c7d | new york | Catherine Nelson | NULL | NULL -(1 row) -~~~ - -### Start a multi-node demo cluster - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach demo --nodes=3 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> \demo ls -~~~ - -~~~ -node 1: - (webui) http://127.0.0.1:8080/demologin?password=demo76950&username=demo - (sql) postgres://demo:demo76950@127.0.0.1:26257?sslmode=require - (sql/unix) postgres://demo:demo76950@?host=%2Fvar%2Ffolders%2Fc8%2Fb_q93vjj0ybfz0fz0z8vy9zc0000gp%2FT%2Fdemo070856957&port=26257 - -node 2: - (webui) http://127.0.0.1:8081/demologin?password=demo76950&username=demo - (sql) postgres://demo:demo76950@127.0.0.1:26258?sslmode=require - (sql/unix) postgres://demo:demo76950@?host=%2Fvar%2Ffolders%2Fc8%2Fb_q93vjj0ybfz0fz0z8vy9zc0000gp%2FT%2Fdemo070856957&port=26258 - -node 3: - (webui) http://127.0.0.1:8082/demologin?password=demo76950&username=demo - (sql) postgres://demo:demo76950@127.0.0.1:26259?sslmode=require - (sql/unix) postgres://demo:demo76950@?host=%2Fvar%2Ffolders%2Fc8%2Fb_q93vjj0ybfz0fz0z8vy9zc0000gp%2FT%2Fdemo070856957&port=26259 -~~~ - -### Load a sample dataset into a demo cluster - -By default, `cockroach demo` loads the `movr` dataset in to the demo cluster. To pre-load any of the other [available datasets](#datasets) using `cockroach demo `. For example, to load the `ycsb` dataset: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach demo ycsb -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES; -~~~ - -~~~ - schema_name | table_name | type | owner | estimated_row_count | locality ---------------+------------+-------+-------+---------------------+----------- - public | usertable | table | demo | 0 | NULL -(1 row) -~~~ - -### Run load against a demo cluster - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach demo --with-load -~~~ - -This command starts a demo cluster with the `movr` database preloaded and then inserts rows into each table in the `movr` database. You can monitor the workload progress on the [DB Console](ui-overview-dashboard.html#sql-statements). - -When running a multi-node demo cluster, load is balanced across all nodes. - -### Execute SQL from the command-line against a demo cluster - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach demo \ ---execute="CREATE TABLE drivers ( - id UUID DEFAULT gen_random_uuid(), - city STRING NOT NULL, - name STRING, - dl STRING UNIQUE, - address STRING, - CONSTRAINT primary_key PRIMARY KEY (city ASC, id ASC) -);" \ ---execute="INSERT INTO drivers (city, name) VALUES ('new york', 'Catherine Nelson');" \ ---execute="SELECT * FROM drivers;" -~~~ - -~~~ -CREATE TABLE -INSERT 1 - id | city | name | dl | address ----------------------------------------+----------+------------------+------+---------- - dd6afc4c-bf31-455e-bb6d-bfb8f18ad6cc | new york | Catherine Nelson | NULL | NULL -(1 row) -~~~ - -### Connect an additional SQL client to the demo cluster - -In addition to the interactive SQL shell that opens when you run `cockroach demo`, you can use the [connection parameters](#connection-parameters) in the welcome text to connect additional SQL clients to the cluster. - -First, use `\demo ls` to list the connection parameters for each node in the demo cluster: - -{% include_cached copy-clipboard.html %} -~~~ sql -> \demo ls -~~~ - -~~~ -node 1: - (webui) http://127.0.0.1:8080/demologin?password=demo76950&username=demo - (sql) postgres://demo:demo76950@127.0.0.1:26257?sslmode=require - (sql/unix) postgres://demo:demo76950@?host=%2Fvar%2Ffolders%2Fc8%2Fb_q93vjj0ybfz0fz0z8vy9zc0000gp%2FT%2Fdemo070856957&port=26257 - -node 2: - (webui) http://127.0.0.1:8081/demologin?password=demo76950&username=demo - (sql) postgres://demo:demo76950@127.0.0.1:26258?sslmode=require - (sql/unix) postgres://demo:demo76950@?host=%2Fvar%2Ffolders%2Fc8%2Fb_q93vjj0ybfz0fz0z8vy9zc0000gp%2FT%2Fdemo070856957&port=26258 - -node 3: - (webui) http://127.0.0.1:8082/demologin?password=demo76950&username=demo - (sql) postgres://demo:demo76950@127.0.0.1:26259?sslmode=require - (sql/unix) postgres://demo:demo76950@?host=%2Fvar%2Ffolders%2Fc8%2Fb_q93vjj0ybfz0fz0z8vy9zc0000gp%2FT%2Fdemo070856957&port=26259 -~~~ - -Then open a new terminal and run [`cockroach sql`](cockroach-sql.html) with the `--url` flag set to the `sql` connection URL of the node to which you want to connect: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --url='postgres://demo:demo53628@127.0.0.1:26259?sslmode=require' -~~~ - -You can also use this URL to connect an application to the demo cluster as the `demo` user. - -### Start a multi-region demo cluster - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach demo --global --nodes 9 -~~~ - -This command starts a 9-node demo cluster with the `movr` database preloaded and region and zone localities set at the cluster level. - -{{site.data.alerts.callout_info}} -The `--global` flag is an experimental feature of `cockroach demo`. The interface and output are subject to change. -{{site.data.alerts.end}} - -For a tutorial that uses a demo cluster to demonstrate CockroachDB's multi-region capabilities, see [Low Latency Reads and Writes in a Multi-Region Cluster](demo-low-latency-multi-region-deployment.html). - -### Add, shut down, and restart nodes in a multi-node demo cluster - -In a multi-node demo cluster, you can use `\demo` [shell commands](#commands) to add, shut down, restart, decommission, and recommission individual nodes. - -{% include {{ page.version.version }}/misc/experimental-warning.md %} - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach demo --nodes=9 -~~~ - -{{site.data.alerts.callout_info}} -`cockroach demo` does not support the `\demo add` and `\demo shutdown` commands in demo clusters started with the `--global` flag. -{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW REGIONS FROM CLUSTER; -~~~ - -~~~ - region | zones ----------------+---------- - europe-west1 | {b,c,d} - us-east1 | {b,c,d} - us-west1 | {a,b,c} -(3 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> \demo ls -~~~ - -~~~ -node 1: - (webui) http://127.0.0.1:8080/demologin?password=demo76950&username=demo - (sql) postgres://demo:demo76950@127.0.0.1:26257?sslmode=require - (sql/unix) postgres://demo:demo76950@?host=%2Fvar%2Ffolders%2Fc8%2Fb_q93vjj0ybfz0fz0z8vy9zc0000gp%2FT%2Fdemo070856957&port=26257 - -node 2: - (webui) http://127.0.0.1:8081/demologin?password=demo76950&username=demo - (sql) postgres://demo:demo76950@127.0.0.1:26258?sslmode=require - (sql/unix) postgres://demo:demo76950@?host=%2Fvar%2Ffolders%2Fc8%2Fb_q93vjj0ybfz0fz0z8vy9zc0000gp%2FT%2Fdemo070856957&port=26258 - -node 3: - (webui) http://127.0.0.1:8082/demologin?password=demo76950&username=demo - (sql) postgres://demo:demo76950@127.0.0.1:26259?sslmode=require - (sql/unix) postgres://demo:demo76950@?host=%2Fvar%2Ffolders%2Fc8%2Fb_q93vjj0ybfz0fz0z8vy9zc0000gp%2FT%2Fdemo070856957&port=26259 - -node 4: - (webui) http://127.0.0.1:8083/demologin?password=demo76950&username=demo - (sql) postgres://demo:demo76950@127.0.0.1:26260?sslmode=require - (sql/unix) postgres://demo:demo76950@?host=%2Fvar%2Ffolders%2Fc8%2Fb_q93vjj0ybfz0fz0z8vy9zc0000gp%2FT%2Fdemo070856957&port=26260 - -node 5: - (webui) http://127.0.0.1:8084/demologin?password=demo76950&username=demo - (sql) postgres://demo:demo76950@127.0.0.1:26261?sslmode=require - (sql/unix) postgres://demo:demo76950@?host=%2Fvar%2Ffolders%2Fc8%2Fb_q93vjj0ybfz0fz0z8vy9zc0000gp%2FT%2Fdemo070856957&port=26261 - -node 6: - (webui) http://127.0.0.1:8085/demologin?password=demo76950&username=demo - (sql) postgres://demo:demo76950@127.0.0.1:26262?sslmode=require - (sql/unix) postgres://demo:demo76950@?host=%2Fvar%2Ffolders%2Fc8%2Fb_q93vjj0ybfz0fz0z8vy9zc0000gp%2FT%2Fdemo070856957&port=26262 - -node 7: - (webui) http://127.0.0.1:8086/demologin?password=demo76950&username=demo - (sql) postgres://demo:demo76950@127.0.0.1:26263?sslmode=require - (sql/unix) postgres://demo:demo76950@?host=%2Fvar%2Ffolders%2Fc8%2Fb_q93vjj0ybfz0fz0z8vy9zc0000gp%2FT%2Fdemo070856957&port=26263 - -node 8: - (webui) http://127.0.0.1:8087/demologin?password=demo76950&username=demo - (sql) postgres://demo:demo76950@127.0.0.1:26264?sslmode=require - (sql/unix) postgres://demo:demo76950@?host=%2Fvar%2Ffolders%2Fc8%2Fb_q93vjj0ybfz0fz0z8vy9zc0000gp%2FT%2Fdemo070856957&port=26264 - -node 9: - (webui) http://127.0.0.1:8088/demologin?password=demo76950&username=demo - (sql) postgres://demo:demo76950@127.0.0.1:26265?sslmode=require - (sql/unix) postgres://demo:demo76950@?host=%2Fvar%2Ffolders%2Fc8%2Fb_q93vjj0ybfz0fz0z8vy9zc0000gp%2FT%2Fdemo070856957&port=26265 -~~~ - -You can shut down and restart any node by node id. For example, to shut down the 3rd node and then restart it: - -{% include_cached copy-clipboard.html %} -~~~ sql -> \demo shutdown 3 -~~~ - -~~~ -node 3 has been shutdown -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> \demo restart 3 -~~~ - -~~~ -node 3 has been restarted -~~~ - -You can also decommission the 3rd node and then recommission it: - -{% include_cached copy-clipboard.html %} -~~~ sql -> \demo decommission 3 -~~~ - -~~~ -node 3 has been decommissioned -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> \demo recommission 3 -~~~ - -~~~ -node 3 has been recommissioned -~~~ - -To add a new node to the cluster: - -{% include_cached copy-clipboard.html %} -~~~ sql -> \demo add region=us-central1,zone=a -~~~ - -~~~ -node 10 has been added with locality "region=us-central1,zone=a" -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW REGIONS FROM CLUSTER; -~~~ - -~~~ - region | zones ----------------+---------- - europe-west1 | {b,c,d} - us-central1 | {a} - us-east1 | {b,c,d} - us-west1 | {a,b,c} -(4 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> \demo ls -~~~ - -~~~ -node 1: - (webui) http://127.0.0.1:8080/demologin?password=demo76950&username=demo - (sql) postgres://demo:demo76950@127.0.0.1:26257?sslmode=require - (sql/unix) postgres://demo:demo76950@?host=%2Fvar%2Ffolders%2Fc8%2Fb_q93vjj0ybfz0fz0z8vy9zc0000gp%2FT%2Fdemo070856957&port=26257 - -node 2: - (webui) http://127.0.0.1:8081/demologin?password=demo76950&username=demo - (sql) postgres://demo:demo76950@127.0.0.1:26258?sslmode=require - (sql/unix) postgres://demo:demo76950@?host=%2Fvar%2Ffolders%2Fc8%2Fb_q93vjj0ybfz0fz0z8vy9zc0000gp%2FT%2Fdemo070856957&port=26258 - -... - -node 10: - (webui) http://127.0.0.1:8089/demologin?password=demo76950&username=demo - (sql) postgres://demo:demo76950@127.0.0.1:26266?sslmode=require - (sql/unix) postgres://demo:demo76950@?host=%2Fvar%2Ffolders%2Fc8%2Fb_q93vjj0ybfz0fz0z8vy9zc0000gp%2FT%2Fdemo070856957&port=26266 -~~~ - -### Try your own scenario - -In addition to using one of the [pre-loaded dataset](#datasets), you can create your own database (e.g., [`CREATE DATABASE ;`](create-database.html)), or use the empty `defaultdb` database (e.g., [`SET DATABASE defaultdb;`](set-vars.html)) to test our your own scenario involving any CockroachDB SQL features you are interested in. - -## See also - -- [`cockroach sql`](cockroach-sql.html) -- [`cockroach workload`](cockroach-workload.html) -- [Other Cockroach Commands](cockroach-commands.html) -- [SQL Statements](sql-statements.html) -- [Learn CockroachDB SQL](learn-cockroachdb-sql.html) -- [MovR: Vehicle-Sharing App](movr.html) diff --git a/src/current/v21.1/cockroach-dump.md b/src/current/v21.1/cockroach-dump.md deleted file mode 100644 index 1105d0a002a..00000000000 --- a/src/current/v21.1/cockroach-dump.md +++ /dev/null @@ -1,454 +0,0 @@ ---- -title: cockroach dump -summary: Learn how to dump schemas and data from a CockroachDB cluster. -toc: true -key: sql-dump.html ---- - -{{site.data.alerts.callout_danger}} -`cockroach dump` is no longer fully supported and has been deprecated in v20.2. [`cockroach dump --dump-mode=schema`](#dump-just-a-tables-schema) is still supported, but is deprecated. Use [`SHOW CREATE ALL TABLES`](show-create.html) instead. All `cockroach dump` commands will be removed in v21.2 and later versions. - -Instead, back up your data in a [full backup](take-full-and-incremental-backups.html), [export](export.html) your data in plain text format, or view table schema in plain text with [`SHOW CREATE TABLE`](show-create.html). -{{site.data.alerts.end}} - -The `cockroach dump` [command](cockroach-commands.html) outputs the SQL statements required to recreate tables, views, and sequences. This command can be used to back up or export each database in a cluster. The output should also be suitable for importing into other relational databases, with minimal adjustments. - -## Considerations - -When `cockroach dump` is executed: - -- Table, sequence, and view schemas and table data are dumped as they appeared at the time that the command is started. Any changes after the command starts will not be included in the dump. -- Table and view schemas are dumped in the order in which they can successfully be recreated. This is true of sequences as well. -- If the dump takes longer than the [`ttlseconds`](configure-replication-zones.html) replication setting for the table (25 hours by default), the dump may fail. -- Reads, writes, and schema changes can happen while the dump is in progress, but will not affect the output of the dump. - -{{site.data.alerts.callout_info}} -The user must have the `SELECT` privilege on the target table(s). -{{site.data.alerts.end}} - -## Synopsis - -Dump the schemas and data of specific tables to stdout: - -~~~ shell -$ cockroach dump
-~~~ - -Dump just the data of specific tables to stdout: - -~~~ shell -$ cockroach dump
--dump-mode=data -~~~ - -Dump just the schemas of specific tables to stdout: - -~~~ shell -$ cockroach dump
--dump-mode=schema -~~~ - -Dump the schemas and data of all tables in a database to stdout: - -~~~ shell -$ cockroach dump -~~~ - -Dump just the schemas of all tables in a database to stdout: - -~~~ shell -$ cockroach dump --dump-mode=schema -~~~ - -Dump just the data of all tables in a database to stdout: - -~~~ shell -$ cockroach dump --dump-mode=data -~~~ - - Dump all non-system databases: - -~~~ shell -$ cockroach dump --dump-all -~~~ - -Dump to a file: - -~~~ shell -$ cockroach dump
> dump-file.sql -~~~ - -View help: - -~~~ shell -$ cockroach dump --help -~~~ - -## Flags - -The `dump` command supports the following [general-use](#general) and [logging](#logging) flags. - -### General - -Flag | Description ------|------------ -`--as-of` | Dump table schema and/or data as they appear at the specified [timestamp](timestamp.html). See this [example](#dump-table-data-as-of-a-specific-time) for a demonstration.

Note that historical data is available only within the garbage collection window, which is determined by the [`ttlseconds`](configure-replication-zones.html) replication setting for the table (25 hours by default). If this timestamp is earlier than that window, the dump will fail.

**Default:** Current time -`--dump-all` | [Dump all non-system databases](#dump-all-databases), their table schemas, and data. -`--dump-mode` | Whether to dump table and view schemas, table data, or both.

To dump just table and view schemas, set this to `schema`. To dump just table data, set this to `data`. To dump both table and view schemas and table data, leave this flag out or set it to `both`.

Table and view schemas are dumped in the order in which they can successfully be recreated. For example, if a database includes a table, a second table with a foreign key dependency on the first, and a view that depends on the second table, the dump will list the schema for the first table, then the schema for the second table, and then the schema for the view.

**Default:** `both` -`--echo-sql` | Reveal the SQL statements sent implicitly by the command-line utility. - -### Client connection - -{% include {{ page.version.version }}/sql/connection-parameters.md %} - -See [Client Connection Parameters](connection-parameters.html) for more details. - -{{site.data.alerts.callout_info}} -The user specified with `--user` must have the `SELECT` privilege on the target tables. -{{site.data.alerts.end}} - -### Logging - -{% include {{ page.version.version }}/misc/logging-defaults.md %} - -## Examples - -{{site.data.alerts.callout_info}} -These examples use our sample `startrek` database, which you can add to a cluster via the [`cockroach gen`](cockroach-gen.html#generate-example-data) command. Also, the examples assume that the `maxroach` user has been [granted](grant.html) the `SELECT` privilege on all target tables. -{{site.data.alerts.end}} - -### Dump a table's schema and data - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach dump startrek episodes --insecure --user=maxroach > backup.sql -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cat backup.sql -~~~ - -~~~ -CREATE TABLE episodes ( - id INT NOT NULL, - season INT NULL, - num INT NULL, - title STRING NULL, - stardate DECIMAL NULL, - CONSTRAINT "primary" PRIMARY KEY (id), - FAMILY "primary" (id, season, num), - FAMILY fam_1_title (title), - FAMILY fam_2_stardate (stardate) -); - -INSERT INTO episodes (id, season, num, title, stardate) VALUES - (1, 1, 1, 'The Man Trap', 1531.1), - (2, 1, 2, 'Charlie X', 1533.6), - (3, 1, 3, 'Where No Man Has Gone Before', 1312.4), - (4, 1, 4, 'The Naked Time', 1704.2), - (5, 1, 5, 'The Enemy Within', 1672.1), - (6, 1, 6, e'Mudd\'s Women', 1329.8), - (7, 1, 7, 'What Are Little Girls Made Of?', 2712.4), - (8, 1, 8, 'Miri', 2713.5), - (9, 1, 9, 'Dagger of the Mind', 2715.1), - (10, 1, 10, 'The Corbomite Maneuver', 1512.2), - ... -~~~ - -### Dump just a table's schema - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach dump startrek episodes --insecure --user=maxroach --dump-mode=schema > backup.sql -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cat backup.sql -~~~ - -~~~ -CREATE TABLE episodes ( - id INT NOT NULL, - season INT NULL, - num INT NULL, - title STRING NULL, - stardate DECIMAL NULL, - CONSTRAINT "primary" PRIMARY KEY (id), - FAMILY "primary" (id, season, num), - FAMILY fam_1_title (title), - FAMILY fam_2_stardate (stardate) -); -~~~ - -### Dump just a table's data - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach dump startrek episodes --insecure --user=maxroach --dump-mode=data > backup.sql -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cat backup.sql -~~~ - -~~~ -INSERT INTO episodes (id, season, num, title, stardate) VALUES - (1, 1, 1, 'The Man Trap', 1531.1), - (2, 1, 2, 'Charlie X', 1533.6), - (3, 1, 3, 'Where No Man Has Gone Before', 1312.4), - (4, 1, 4, 'The Naked Time', 1704.2), - (5, 1, 5, 'The Enemy Within', 1672.1), - (6, 1, 6, e'Mudd\'s Women', 1329.8), - (7, 1, 7, 'What Are Little Girls Made Of?', 2712.4), - (8, 1, 8, 'Miri', 2713.5), - (9, 1, 9, 'Dagger of the Mind', 2715.1), - (10, 1, 10, 'The Corbomite Maneuver', 1512.2), - ... -~~~ - -### Dump all tables in a database - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach dump startrek --insecure --user=maxroach > backup.sql -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cat backup.sql -~~~ - -~~~ -CREATE TABLE episodes ( - id INT NOT NULL, - season INT NULL, - num INT NULL, - title STRING NULL, - stardate DECIMAL NULL, - CONSTRAINT "primary" PRIMARY KEY (id), - FAMILY "primary" (id, season, num), - FAMILY fam_1_title (title), - FAMILY fam_2_stardate (stardate) -); - -CREATE TABLE quotes ( - quote STRING NULL, - characters STRING NULL, - stardate DECIMAL NULL, - episode INT NULL, - INDEX quotes_episode_idx (episode), - FAMILY "primary" (quote, rowid), - FAMILY fam_1_characters (characters), - FAMILY fam_2_stardate (stardate), - FAMILY fam_3_episode (episode) -); - -INSERT INTO episodes (id, season, num, title, stardate) VALUES - (1, 1, 1, 'The Man Trap', 1531.1), - (2, 1, 2, 'Charlie X', 1533.6), - (3, 1, 3, 'Where No Man Has Gone Before', 1312.4), - (4, 1, 4, 'The Naked Time', 1704.2), - (5, 1, 5, 'The Enemy Within', 1672.1), - (6, 1, 6, e'Mudd\'s Women', 1329.8), - (7, 1, 7, 'What Are Little Girls Made Of?', 2712.4), - (8, 1, 8, 'Miri', 2713.5), - (9, 1, 9, 'Dagger of the Mind', 2715.1), - (10, 1, 10, 'The Corbomite Maneuver', 1512.2), - ... - -INSERT INTO quotes (quote, characters, stardate, episode) VALUES - ('"... freedom ... is a worship word..." "It is our worship word too."', 'Cloud William and Kirk', NULL, 52), - ('"Beauty is transitory." "Beauty survives."', 'Spock and Kirk', NULL, 72), - ('"Can you imagine how life could be improved if we could do away with jealousy, greed, hate ..." "It can also be improved by eliminating love, tenderness, sentiment -- the other side of the coin"', 'Dr. Roger Corby and Kirk', 2712.4, 7), - ... -~~~ - -## Dump all databases - - To dump all non-system databases, their table schemas, and data: - -~~~ shell -$ cockroach dump --dump-all -~~~ - -~~~ -CREATE DATABASE IF NOT EXISTS movr; -USE movr; - -CREATE TABLE promo_codes ( - code VARCHAR NOT NULL, - description VARCHAR NULL, - creation_time TIMESTAMP NULL, - expiration_time TIMESTAMP NULL, - rules JSONB NULL, - CONSTRAINT "primary" PRIMARY KEY (code ASC), - FAMILY "primary" (code, description, creation_time, expiration_time, rules) -); - -CREATE TABLE users ( - ... -); - -CREATE TABLE vehicles ( - ... -); - -CREATE TABLE rides ( - ... -); - -CREATE TABLE user_promo_codes ( - ... -); - -CREATE TABLE vehicle_location_histories ( - ... -); - -INSERT INTO promo_codes (code, description, creation_time, expiration_time, rules) VALUES - ('0_explain_theory_something', 'Live sing car maybe. Give safe edge chair discuss resource. Stop entire look support instead. Sister focus long agency like argue.', '2018-12-27 03:04:05+00:00', '2019-01-02 03:04:05+00:00', '{"type": "percent_discount", "value": "10%"}'), - ('100_address_garden_certain', 'Hour industry himself student position international. Southern traditional rest name prepare. Tough sign little into class. Money general care guy.', '2018-12-27 03:04:05+00:00', '2019-01-13 03:04:05+00:00', '{"type": "percent_discount", "value": "10%"}'), - - ... -~~~ - -### Dump fails (user does not have `SELECT` privilege) - -In this example, the `dump` command fails for a user that does not have the `SELECT` privilege on the `episodes` table. - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach dump startrek episodes --insecure --user=leslieroach > backup.sql -~~~ - -~~~ -Error: pq: user leslieroach has no privileges on table episodes -Failed running "dump" -~~~ - -### Restore a table from a backup file - -In this example, a user that has the `CREATE` privilege on the `startrek` database uses the [`cockroach sql`](cockroach-sql.html) command to recreate a table, based on a file created by the `dump` command. - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cat backup.sql -~~~ - -~~~ -CREATE TABLE quotes ( - quote STRING NULL, - characters STRING NULL, - stardate DECIMAL NULL, - episode INT NULL, - INDEX quotes_episode_idx (episode), - FAMILY "primary" (quote, rowid), - FAMILY fam_1_characters (characters), - FAMILY fam_2_stardate (stardate), - FAMILY fam_3_episode (episode) -); - -INSERT INTO quotes (quote, characters, stardate, episode) VALUES - ('"... freedom ... is a worship word..." "It is our worship word too."', 'Cloud William and Kirk', NULL, 52), - ('"Beauty is transitory." "Beauty survives."', 'Spock and Kirk', NULL, 72), - ('"Can you imagine how life could be improved if we could do away with jealousy, greed, hate ..." "It can also be improved by eliminating love, tenderness, sentiment -- the other side of the coin"', 'Dr. Roger Corby and Kirk', 2712.4, 7), - ... -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --insecure --database=startrek --user=maxroach -f backup.sql -~~~ - -~~~ -CREATE TABLE -INSERT 100 -INSERT 100 -~~~ - -### Dump table data as of a specific time - -In this example, we assume there were several inserts into a table both before and after `2017-03-07 19:55:00`. - -First, let's use the built-in SQL client to view the table at the current time: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --insecure --execute="SELECT * FROM db1.dump_test" -~~~ - -~~~ -+--------------------+------+ -| id | name | -+--------------------+------+ -| 225594758537183233 | a | -| 225594758537248769 | b | -| 225594758537281537 | c | -| 225594758537314305 | d | -| 225594758537347073 | e | -| 225594758537379841 | f | -| 225594758537412609 | g | -| 225594758537445377 | h | -| 225594991654174721 | i | -| 225594991654240257 | j | -| 225594991654273025 | k | -| 225594991654305793 | l | -| 225594991654338561 | m | -| 225594991654371329 | n | -| 225594991654404097 | o | -| 225594991654436865 | p | -+--------------------+------+ -(16 rows) -~~~ - -Next, let's use a [time-travel query](select-clause.html#select-historical-data-time-travel) to view the contents of the table as of `2017-03-07 19:55:00`: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --insecure --execute="SELECT * FROM db1.dump_test AS OF SYSTEM TIME '2017-03-07 19:55:00'" -~~~ - -~~~ -+--------------------+------+ -| id | name | -+--------------------+------+ -| 225594758537183233 | a | -| 225594758537248769 | b | -| 225594758537281537 | c | -| 225594758537314305 | d | -| 225594758537347073 | e | -| 225594758537379841 | f | -| 225594758537412609 | g | -| 225594758537445377 | h | -+--------------------+------+ -(8 rows) -~~~ - -Finally, let's use `cockroach dump` with the `--as-of` flag set to dump the contents of the table as of `2017-03-07 19:55:00`. - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach dump db1 dump_test --insecure --dump-mode=data --as-of='2017-03-07 19:55:00' -~~~ - -~~~ -INSERT INTO dump_test (id, name) VALUES - (225594758537183233, 'a'), - (225594758537248769, 'b'), - (225594758537281537, 'c'), - (225594758537314305, 'd'), - (225594758537347073, 'e'), - (225594758537379841, 'f'), - (225594758537412609, 'g'), - (225594758537445377, 'h'); -~~~ - -As you can see, the results of the dump are identical to the earlier time-travel query. - -## See also - -- [Import Data](migration-overview.html) -- [`IMPORT`](import.html) -- [Use the Built-in SQL Client](cockroach-sql.html) -- [Other Cockroach Commands](cockroach-commands.html) diff --git a/src/current/v21.1/cockroach-gen.md b/src/current/v21.1/cockroach-gen.md deleted file mode 100644 index 92614a1ea18..00000000000 --- a/src/current/v21.1/cockroach-gen.md +++ /dev/null @@ -1,393 +0,0 @@ ---- -title: cockroach gen -summary: Use cockroach gen to generate command-line interface utlities, such as man pages, and example data. -toc: true -key: generate-cockroachdb-resources.html ---- - -The `cockroach gen` [command](cockroach-commands.html) can generate command-line interface (CLI) utilities ([`man` pages](https://en.wikipedia.org/wiki/Man_page) and a `bash` autocompletion script), example SQL data suitable to populate test databases, and an HAProxy configuration file for load balancing a running cluster. - -## Subcommands - -Subcommand | Usage ------------|------ -`man` | Generate man pages for CockroachDB. -`autocomplete` | Generate `bash` or `zsh` autocompletion script for CockroachDB.

**Default:** `bash` -`example-data` | Generate example SQL datasets. You can also use the [`cockroach workload`](cockroach-workload.html) command to generate these sample datasets in a persistent cluster and the [`cockroach demo `](cockroach-demo.html) command to generate these datasets in a temporary, in-memory cluster. -`haproxy` | Generate an HAProxy config file for a running CockroachDB cluster. The node addresses included in the config are those advertised by the nodes. Make sure hostnames are resolvable and IP addresses are routable from HAProxy.

[Decommissioned nodes](remove-nodes.html) are excluded from the config file. - -## Synopsis - -Generate man pages: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach gen man -~~~ - -Generate bash autocompletion script: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach gen autocomplete -~~~ - -Generate example SQL data: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach gen example-data intro | cockroach sql -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach gen example-data startrek | cockroach sql -~~~ - -Generate an HAProxy config file for a running cluster: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach gen haproxy -~~~ - -View help: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach gen --help -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach gen man --help -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach gen autocomplete --help -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach gen example-data --help -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach gen haproxy --help -~~~ - -## Flags - -The `gen` subcommands supports the following [general-use](#general), [logging](#logging), and [client connection](#client-connection) flags. - -### General - -#### `man` - -Flag | Description ------|----------- -`--path` | The path where man pages will be generated.

**Default:** `man/man1` under the current directory - -#### `autocomplete` - -Flag | Description ------|----------- -`--out` | The path where the autocomplete file will be generated.

**Default:** `cockroach.bash` in the current directory - -#### `example-data` - -No flags are supported. See the [Generate Example Data](#generate-example-data) example for guidance. - -#### `haproxy` - -Flag | Description ------|------------ -`--host` | The server host and port number to connect to. This can be the address of any node in the cluster.

**Env Variable:** `COCKROACH_HOST`
**Default:** `localhost:26257` -`--port`
`-p` | The server port to connect to. Note: The port number can also be specified via `--host`.

**Env Variable:** `COCKROACH_PORT`
**Default:** `26257` -`--insecure` | Use an insecure connection.

**Env Variable:** `COCKROACH_INSECURE`
**Default:** `false` -`--certs-dir` | The path to the [certificate directory](cockroach-cert.html) containing the CA and client certificates and client key.

**Env Variable:** `COCKROACH_CERTS_DIR`
**Default:** `${HOME}/.cockroach-certs/` -`--url` | A [connection URL](connection-parameters.html#connect-using-a-url) to use instead of the other arguments.

**Env Variable:** `COCKROACH_URL`
**Default:** no URL -`--out` | The path where the `haproxy.cfg` file will be generated. If an `haproxy.cfg` file already exists in the directory, it will be overwritten.

**Default:** `haproxy.cfg` in the current directory -`--locality` | If nodes were started with [locality](cockroach-start.html#locality) details, you can use the `--locality` flag here to filter the nodes included in the HAProxy config file, specifying the explicit locality tier(s) or a regular expression to match against. This is useful in cases where you want specific instances of HAProxy to route to specific nodes. See the [Generate an HAProxy configuration file](#generate-an-haproxy-config-file) example for more details. - -### Logging - -{% include {{ page.version.version }}/misc/logging-defaults.md %} - -### Client Connection - -#### `haproxy` - -Flag | Description ------|------------ -`--cluster-name` | The cluster name to use to verify the cluster's identity. If the cluster has a cluster name, you must include this flag. For more information, see [`cockroach start`](cockroach-start.html#general). -`--disable-cluster-name-verification` | Disables the cluster name check for this command. This flag must be paired with `--cluster-name`. For more information, see [`cockroach start`](cockroach-start.html#general). - -## Examples - -### Generate `man` pages - -Generate man pages: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach gen man -~~~ - -Move the man pages to the man directory: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ sudo mv man/man1/* /usr/share/man/man1 -~~~ - -Access man pages: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ man cockroach -~~~ - -### Generate a `bash` autocompletion script - -Generate bash autocompletion script: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach gen autocomplete -~~~ - -Add the script to your `.bashrc` and `.bash_profle`: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ printf "\n\n#cockroach bash autocomplete\nsource 'cockroach.bash'" >> ~/.bashrc -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ printf "\n\n#cockroach bash autocomplete\nsource 'cockroach.bash'" >> ~/.bash_profile -~~~ - -You can now use `tab` to autocomplete `cockroach` commands. - -### Generate example data - -{{site.data.alerts.callout_success}} -You can also use the [`cockroach workload`](cockroach-workload.html) command to generate these sample datasets in a persistent cluster and the [`cockroach demo `](cockroach-demo.html) command to generate these datasets in a temporary, in-memory cluster. -{{site.data.alerts.end}} - -To test out CockroachDB, you can generate an example `startrek` database, which contains 2 tables, `episodes` and `quotes`. - -First, start up [a demo cluster](cockroach-demo.html): - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach demo -~~~ - -Then, pipe the output from `cockroach gen` to [the URL to the demo cluster](cockroach-demo.html#connection-parameters): - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach gen example-data startrek | cockroach sql --url='postgres://demo:demo11762@127.0.0.1:26257?sslmode=require' -~~~ - -~~~ -CREATE DATABASE -SET -DROP TABLE -DROP TABLE -CREATE TABLE -INSERT 1 -... -CREATE TABLE -INSERT 1 -... -~~~ - -Open a [SQL shell](cockroach-sql.html) to view it: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --url='postgres://demo:demo11762@127.0.0.1:26257?sslmode=require' -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM startrek; -~~~ -~~~ - schema_name | table_name | type | estimated_row_count ---------------+------------+-------+---------------------- - public | episodes | table | 79 - public | quotes | table | 200 -(2 rows) -~~~ - -You can also generate an example `intro` database, which contains 1 table, `mytable`, with a hidden message: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach gen example-data intro | cockroach sql --url='postgres://demo:demo11762@127.0.0.1:26257?sslmode=require' -~~~ - -~~~ -CREATE DATABASE -SET -DROP TABLE -CREATE TABLE -INSERT 1 -INSERT 1 -INSERT 1 -INSERT 1 -... -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -# Launch the built-in SQL client to view it: -$ cockroach sql --url='postgres://demo:demo11762@127.0.0.1:26257?sslmode=require' -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM intro; -~~~ - -~~~ - schema_name | table_name | type | estimated_row_count ---------------+------------+-------+---------------------- - public | mytable | table | 42 -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM intro.mytable WHERE (l % 2) = 0; -~~~ - -~~~ - l | v ------+------------------------------------------------------- - 0 | !__aaawwmqmqmwwwaas,,_ .__aaawwwmqmqmwwaaa,, - 2 | !"VT?!"""^~~^"""??T$Wmqaa,_auqmWBT?!"""^~~^^""??YV^ - 4 | ! "?##mW##?"- - 6 | ! C O N G R A T S _am#Z??A#ma, Y - 8 | ! _ummY" "9#ma, A - 10 | ! vm#Z( )Xmms Y - 12 | ! .j####mmm#####mm#m##6. - 14 | ! W O W ! jmm###mm######m#mmm##6 - 16 | ! ]#me*Xm#m#mm##m#m##SX##c - 18 | ! dm#||+*$##m#mm#m#Svvn##m - 20 | ! :mmE=|+||S##m##m#1nvnnX##; A - 22 | ! :m#h+|+++=Xmm#m#1nvnnvdmm; M - 24 | ! Y $#m>+|+|||##m#1nvnnnnmm# A - 26 | ! O ]##z+|+|+|3#mEnnnnvnd##f Z - 28 | ! U D 4##c|+|+|]m#kvnvnno##P E - 30 | ! I 4#ma+|++]mmhvnnvq##P` ! - 32 | ! D I ?$#q%+|dmmmvnnm##! - 34 | ! T -4##wu#mm#pw##7' - 36 | ! -?$##m####Y' - 38 | ! !! "Y##Y"- - 40 | ! -(21 rows) -~~~ - -### Generate an HAProxy config file - -[HAProxy](http://www.haproxy.org/) is one of the most popular open-source TCP load balancers, and CockroachDB includes a built-in command for generating a configuration file that is preset to work with your running cluster. - -
- - -

- -
-To generate an HAProxy config file for an entire secure cluster, run the `cockroach gen haproxy` command, specifying the location of [certificate directory](cockroach-cert.html) and the address of any instance running a CockroachDB node: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach gen haproxy \ ---certs-dir= \ ---host=
-~~~ - -To limit the HAProxy config file to nodes matching specific ["localities"](cockroach-start.html#locality), use the `--localities` flag, specifying the explicit locality tier(s) or a regular expression to match against: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach gen haproxy \ ---certs-dir= \ ---host=
---locality=region=us.* -~~~ -
- -
-To generate an HAProxy config file for an entire insecure cluster, run the `cockroach gen haproxy` command, specifying the address of any instance running a CockroachDB node: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach gen haproxy \ ---insecure \ ---host=
-~~~ - -To limit the HAProxy config file to nodes matching specific ["localities"](cockroach-start.html#locality), use the `--localities` flag, specifying the explicit locality tier(s) or a regular expression to match against: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach gen haproxy \ ---insecure \ ---host=
---locality=region=us.* -~~~ -
- -By default, the generated configuration file is called `haproxy.cfg` and looks as follows, with the `server` addresses pre-populated correctly: - -~~~ -global - maxconn 4096 - -defaults - mode tcp - # Timeout values should be configured for your specific use. - # See: https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-timeout%20connect - timeout connect 10s - timeout client 1m - timeout server 1m - # TCP keep-alive on client side. Server already enables them. - option clitcpka - -listen psql - bind :26257 - mode tcp - balance roundrobin - option httpchk GET /health?ready=1 - server cockroach1 :26257 check port 8080 - server cockroach2 :26257 check port 8080 - server cockroach3 :26257 check port 8080 -~~~ - -The file is preset with the minimal [configurations](http://cbonte.github.io/haproxy-dconv/1.7/configuration.html) needed to work with your running cluster: - -Field | Description -------|------------ -`timeout connect`
`timeout client`
`timeout server` | Timeout values that should be suitable for most deployments. -`bind` | The port that HAProxy listens on. This is the port clients will connect to and thus needs to be allowed by your network configuration.

This tutorial assumes HAProxy is running on a separate machine from CockroachDB nodes. If you run HAProxy on the same machine as a node (not recommended), you'll need to change this port, as `26257` is likely already being used by the CockroachDB node. -`balance` | The balancing algorithm. This is set to `roundrobin` to ensure that connections get rotated amongst nodes (connection 1 on node 1, connection 2 on node 2, etc.). Check the [HAProxy Configuration Manual](http://cbonte.github.io/haproxy-dconv/1.7/configuration.html#4-balance) for details about this and other balancing algorithms. -`option httpchk` | The HTTP endpoint that HAProxy uses to check node health. [`/health?ready=1`](monitoring-and-alerting.html#health-ready-1) ensures that HAProxy doesn't direct traffic to nodes that are live but not ready to receive requests. -`server` | For each included node, this field specifies the address the node advertises to other nodes in the cluster, i.e., the addressed pass in the [`--advertise-addr` flag](cockroach-start.html#networking) on node startup. Make sure hostnames are resolvable and IP addresses are routable from HAProxy. - -{{site.data.alerts.callout_info}} -For full details on these and other configuration settings, see the [HAProxy Configuration Manual](http://cbonte.github.io/haproxy-dconv/1.7/configuration.html). -{{site.data.alerts.end}} - -## See also - -- [Other Cockroach Commands](cockroach-commands.html) -- [Deploy CockroachDB On-Premises](deploy-cockroachdb-on-premises.html) (using HAProxy for load balancing) diff --git a/src/current/v21.1/cockroach-import.md b/src/current/v21.1/cockroach-import.md deleted file mode 100644 index 76f1007e7fc..00000000000 --- a/src/current/v21.1/cockroach-import.md +++ /dev/null @@ -1,113 +0,0 @@ ---- -title: cockroach import -summary: The cockroach import command imports a database or table from a local dump file into a running cluster. -toc: true ---- - -{% include_cached new-in.html version="v21.1" %} The `cockroach import` [command](cockroach-commands.html) imports a database or table from a local dump file into a running cluster. This command [uploads a userfile](cockroach-userfile-upload.html), imports its data, then [deletes the userfile](cockroach-userfile-delete.html). `PGDUMP` and `MYSQLDUMP` file formats are currently supported. - -{{site.data.alerts.callout_info}} -We recommend using `cockroach import` for quick imports from your client (about 15MB or smaller). For larger imports, use the [IMPORT](import.html) statement. -{{site.data.alerts.end}} - -## Required privileges - -The user must have `CREATE` [privileges](authorization.html#assign-privileges) on `defaultdb`. - -## Synopsis - -Import a database: - -~~~ shell -$ cockroach import db -~~~ - -Import a table: - -~~~ shell -$ cockroach import table -~~~ - -View help: - -~~~ shell -$ cockroach import --help -~~~ - -## Supported Formats - -- [`pgdump`](migrate-from-postgres.html#step-1-dump-the-postgres-database) -- [`mysqldump`](migrate-from-mysql.html#step-1-dump-the-mysql-database) - -## Flags - - Flag | Description ------------------+----------------------------------------------------- -`--certs-dir` | The path to the [certificate directory](cockroach-cert.html) containing the CA and client certificates and client key.

**Env Variable:** `COCKROACH_CERTS_DIR`
**Default:** `${HOME}/.cockroach-certs/` -`--insecure` | Use an insecure connection.

**Env Variable:** `COCKROACH_INSECURE`
**Default:** `false` -`--user`
`-u` | The [SQL user](create-user.html) that will own the client session.

**Env Variable:** `COCKROACH_USER`
**Default:** `root` -`--ignore-unsupported-statements` | **New in v21.1:** Ignore statements that are unsupported during an import from a PGDUMP file.
**Default:** `false` -`--log-ignored-statements` | **New in v21.1:** Log statements that are ignored during an import from a PGDUMP file to the specified destination (i.e., [cloud storage](use-cloud-storage-for-bulk-operations.html) or [userfile storage](use-userfile-for-bulk-operations.html). -`--row-limit=` | **New in v21.1:** The number of rows to import for each table during a PGDUMP or MYSQLDUMP import.
This can be used to check schema and data correctness without running the entire import.
**Default:** `0` - -## Examples - -### Import a database - -To import a database from a local file: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach import db mysqldump /Users/maxroach/Desktop/test-db.sql --certs-dir=certs -~~~ - -~~~ -successfully imported mysqldump file /Users/maxroach/Desktop/test-db.sql -~~~ - -### Import a table - -To import a table from a local file: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach import table test_table pgdump /Users/maxroach/Desktop/test-db.sql --certs-dir=certs -~~~ - -~~~ -successfully imported table test_table from pgdump file /Users/maxroach/Desktop/test-db.sql -~~~ - -### Import a database with unsupported SQL syntax and log all unsupported statements - -{% include_cached new-in.html version="v21.1" %} To import a database from a `PGDUMP` file that contains unsupported SQL syntax and log the ignored statements to a [userfile](use-userfile-for-bulk-operations.html): - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach import db pgdump /Users/maxroach/Desktop/test-db.sql --certs-dir=certs --ignore-unsupported-statements=true --log-ignored-statements='userfile://defaultdb.public.userfiles_root/unsupported-statements.log' -~~~ - -~~~ -successfully imported table test_table from pgdump file /Users/maxroach/Desktop/test-db.sql -~~~ - -### Import a limited number of rows from a dump file - -{% include_cached new-in.html version="v21.1" %} To limit the number of rows imported from a dump file: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach import table test_table pgdump /Users/maxroach/Desktop/test-db.sql --certs-dir=certs --row-limit='50' -~~~ - -~~~ -successfully imported table test_table from pgdump file /Users/maxroach/Desktop/test-db.sql -~~~ - -## See also - -- [Other Cockroach Commands](cockroach-commands.html) -- [`IMPORT`](import.html) -- [`IMPORT INTO`](import-into.html) -- [Migrate from Postgres](migrate-from-postgres.html) -- [Migrate from MySQL](migrate-from-mysql.html) diff --git a/src/current/v21.1/cockroach-init.md b/src/current/v21.1/cockroach-init.md deleted file mode 100644 index 1cc5a5e30a3..00000000000 --- a/src/current/v21.1/cockroach-init.md +++ /dev/null @@ -1,130 +0,0 @@ ---- -title: cockroach init -summary: Perform a one-time-only initialization of a CockroachDB cluster. -toc: true -key: initialize-a-cluster.html ---- - -This page explains the `cockroach init` [command](cockroach-commands.html), which you use to perform a one-time initialization of a new multi-node cluster. For a full tutorial of the cluster startup and initialization process, see one of the [Manual Deployment](manual-deployment.html) tutorials. - -{{site.data.alerts.callout_info}} -When starting a single-node cluster with [`cockroach start-single-node`](cockroach-start-single-node.html), you do not need to use the `cockroach init` command. -{{site.data.alerts.end}} - -## Synopsis - -Perform a one-time initialization of a cluster: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach init -~~~ - -View help: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach init --help -~~~ - -## Flags - -The `cockroach init` command supports the following [client connection](#client-connection) and [logging](#logging) flags. - -{{site.data.alerts.callout_info}} -`cockroach init` must target one of the nodes that was listed with [`--join`](cockroach-start.html#networking) when starting the cluster. Otherwise, the command will not initialize the cluster correctly. -{{site.data.alerts.end}} - -### Client connection - -{% include {{ page.version.version }}/sql/connection-parameters.md %} -`--cluster-name` | The cluster name to use to verify the cluster's identity. If the cluster has a cluster name, you must include this flag. For more information, see [`cockroach start`](cockroach-start.html#general). -`--disable-cluster-name-verification` | Disables the cluster name check for this command. This flag must be paired with `--cluster-name`. For more information, see [`cockroach start`](cockroach-start.html#general). - -See [Client Connection Parameters](connection-parameters.html) for details. - -### Logging - -{% include {{ page.version.version }}/misc/logging-defaults.md %} - -## Examples - -Usage of `cockroach init` assumes that nodes have already been started with [`cockroach start`](cockroach-start.html) and are waiting to be initialized as a new cluster. For a more detailed tutorial, see one of the [Manual Deployment](manual-deployment.html) tutorials. - -### Initialize a Cluster on a Node's Machine - -
- - -
- -
-1. SSH to the machine where the node has been started. This must be a node that was listed with [`--join`](cockroach-start.html#networking) when starting the cluster. - -2. Make sure the `client.root.crt` and `client.root.key` files for the `root` user are on the machine. - -3. Run the `cockroach init` command with the `--certs-dir` flag set to the directory containing the `ca.crt` file and the files for the `root` user, and with the `--host` flag set to the address of the current node: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach init --certs-dir=certs --host=
- ~~~ - - At this point, all the nodes complete startup and print helpful details to the [standard output](cockroach-start.html#standard-output), such as the CockroachDB version, the URL for the DB Console, and the SQL URL for clients. -
- -
-1. SSH to the machine where the node has been started. This must be a node that was listed with [`--join`](cockroach-start.html#networking) when starting the cluster. - -2. Run the `cockroach init` command with the `--host` flag set to the address of the current node: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach init --insecure --host=
- ~~~ - - At this point, all the nodes complete startup and print helpful details to the [standard output](cockroach-start.html#standard-output), such as the CockroachDB version, the URL for the DB Console, and the SQL URL for clients. -
- -### Initialize a cluster from another machine - -
- - -
- -
-1. [Install the `cockroach` binary](install-cockroachdb.html) on a machine separate from the node. - -2. Create a `certs` directory and copy the CA certificate and the client certificate and key for the `root` user into the directory. - -3. Run the `cockroach init` command with the `--certs-dir` flag set to the directory containing the `ca.crt` file and the files for the `root` user, and with the `--host` flag set to the address of the node. This must be a node that was listed with [`--join`](cockroach-start.html#networking) when starting the cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach init --certs-dir=certs --host=
- ~~~ - - At this point, all the nodes complete startup and print helpful details to the [standard output](cockroach-start.html#standard-output), such as the CockroachDB version, the URL for the DB Console, and the SQL URL for clients. -
- -
-1. [Install the `cockroach` binary](install-cockroachdb.html) on a machine separate from the node. - -2. Run the `cockroach init` command with the `--host` flag set to the address of the node. This must be a node that was listed with [`--join`](cockroach-start.html#networking) when starting the cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach init --insecure --host=
- ~~~ - - At this point, all the nodes complete startup and print helpful details to the [standard output](cockroach-start.html#standard-output), such as the CockroachDB version, the URL for the DB Console, and the SQL URL for clients. -
- -## See also - -- [Manual Deployment](manual-deployment.html) -- [Orchestrated Deployment](orchestration.html) -- [Local Deployment](start-a-local-cluster.html) -- [`cockroach start`](cockroach-start.html) -- [Other Cockroach Commands](cockroach-commands.html) diff --git a/src/current/v21.1/cockroach-node.md b/src/current/v21.1/cockroach-node.md deleted file mode 100644 index 67f4ffa8d68..00000000000 --- a/src/current/v21.1/cockroach-node.md +++ /dev/null @@ -1,322 +0,0 @@ ---- -title: cockroach node -summary: To view details for each node in the cluster, use the cockroach node command with the appropriate subcommands and flags. -toc: true -key: view-node-details.html ---- - -To view details for each node in the cluster, use the `cockroach node` [command](cockroach-commands.html) with the appropriate subcommands and flags. - -The `cockroach node` command is also used in the process of decommissioning nodes for removal from the cluster. See [Decommission Nodes](remove-nodes.html) for more details. - -## Subcommands - -Subcommand | Usage ------------|------ -`ls` | List the ID of each node in the cluster, excluding those that have been decommissioned and are offline. -`status` | View the status of one or all nodes, excluding nodes that have been decommissioned and taken offline. Depending on flags used, this can include details about range/replicas, disk usage, and decommissioning progress. -`decommission` | Decommission nodes for removal from the cluster. See [Decommission Nodes](remove-nodes.html) for more details. -`recommission` | Recommission nodes that have been decommissioned. See [Recommission Nodes](remove-nodes.html#recommission-nodes) for more details. -`drain` | Drain nodes of SQL clients, [distributed SQL](architecture/sql-layer.html#distsql) queries, and range leases, and prevent ranges from rebalancing onto the node. This is normally done by sending `SIGTERM` during [node shutdown](cockroach-quit.html), but the `drain` subcommand provides operators an option to interactively monitor, and if necessary intervene in, the draining process. - -## Synopsis - -List the IDs of active and inactive nodes: - -~~~ shell -$ cockroach node ls -~~~ - -Show status details for active and inactive nodes: - -~~~ shell -$ cockroach node status -~~~ - -Show status and range/replica details for active and inactive nodes: - -~~~ shell -$ cockroach node status --ranges -~~~ - -Show status and disk usage details for active and inactive nodes: - -~~~ shell -$ cockroach node status --stats -~~~ - -Show status and decommissioning details for active and inactive nodes: - -~~~ shell -$ cockroach node status --decommission -~~~ - -Show complete status details for active and inactive nodes: - -~~~ shell -$ cockroach node status --all -~~~ - -Show status details for a specific node: - -~~~ shell -$ cockroach node status -~~~ - -Decommission nodes: - -~~~ shell -$ cockroach node decommission -~~~ - -Recommission nodes: - -~~~ shell -$ cockroach node recommission -~~~ - -Drain nodes: - -~~~ shell -$ cockroach node drain -~~~ - -View help: - -~~~ shell -$ cockroach node --help -~~~ -~~~ shell -$ cockroach node --help -~~~ - -## Flags - -All `node` subcommands support the following [general-use](#general) and [logging](#logging) flags. - -### General - -Flag | Description ------|------------ -`--format` | How to display table rows printed to the standard output. Possible values: `tsv`, `csv`, `table`, `records`, `sql`, `html`.

**Default:** `tsv` - -The `node ls` subcommand also supports the following general flags: - -Flag | Description ------|------------ -`--timeout` | Set the duration of time that the subcommand is allowed to run before it returns an error and prints partial information. The timeout is specified with a suffix of `s` for seconds, `m` for minutes, and `h` for hours. If this flag is not set, the subcommand may hang. - -The `node status` subcommand also supports the following general flags: - -Flag | Description ------|------------ -`--all` | Show all node details. -`--decommission` | Show node decommissioning details. -`--ranges` | Show node details for ranges and replicas. -`--stats` | Show node disk usage details. -`--timeout` | Set the duration of time that the subcommand is allowed to run before it returns an error and prints partial information. The timeout is specified with a suffix of `s` for seconds, `m` for minutes, and `h` for hours. If this flag is not set, the subcommand may hang. - -The `node decommission` subcommand also supports the following general flags: - -Flag | Description ------|------------ -`--wait` | When to return to the client. Possible values: `all`, `none`.

If `all`, the command returns to the client only after all replicas on all specified nodes have been transferred to other nodes. If any specified nodes are offline, the command will not return to the client until those nodes are back online.

If `none`, the command does not wait for the decommissioning process to complete; it returns to the client after starting the decommissioning process on all specified nodes that are online. Any specified nodes that are offline will automatically be marked as decommissioning; if they come back online, the cluster will recognize this status and will not rebalance data to the nodes.

**Default:** `all` -`--self` | Applies the operation to the node against which the command was run (e.g., via `--host`). - -The `node drain` subcommand also supports the following general flag: - -Flag | Description ------|------------ -`--drain-wait` | Amount of time to wait for the node to drain before returning to the client.

**Default:** `10m` - -The `node recommission` subcommand also supports the following general flag: - -Flag | Description ------|------------ -`--self` | Applies the operation to the node against which the command was run (e.g., via `--host`). - -### Client connection - -{% include {{ page.version.version }}/sql/connection-parameters.md %} - -The `node decommission`, `node recommission`, and `node drain` subcommands also support the following client connection flags: - -Flag | Description ------|------------ -`--cluster-name` | The cluster name to use to verify the cluster's identity. If the cluster has a cluster name, you must include this flag. For more information, see [`cockroach start`](cockroach-start.html#general). -`--disable-cluster-name-verification` | Disables the cluster name check for this command. This flag must be paired with `--cluster-name`. For more information, see [`cockroach start`](cockroach-start.html#general). - -See [Client Connection Parameters](connection-parameters.html) for more details. - -### Logging - -{% include {{ page.version.version }}/misc/logging-defaults.md %} - -## Response - -The `cockroach node` subcommands return the following fields for each node. - -### `node ls` - -Field | Description -------|------------ -`id` | The ID of the node. - -### `node status` - -Field | Description -------|------------ -`id` | The ID of the node.

**Required flag:** None -`address` | The address of the node.

**Required flag:** None -`build` | The version of CockroachDB running on the node. If the binary was built from source, this will be the SHA hash of the commit used.

**Required flag:** None -`locality` | The [locality](cockroach-start.html#locality) information specified for the node.

**Required flag:** None -`updated_at` | The date and time when the node last recorded the information displayed in this command's output. When healthy, a new status should be recorded every 10 seconds or so, but when unhealthy this command's stats may be much older.

**Required flag:** None -`started_at` | The date and time when the node was started.

**Required flag:** None -`replicas_leaders` | The number of range replicas on the node that are the Raft leader for their range. See `replicas_leaseholders` below for more details.

**Required flag:** `--ranges` or `--all` -`replicas_leaseholders` | The number of range replicas on the node that are the leaseholder for their range. A "leaseholder" replica handles all read requests for a range and directs write requests to the range's Raft leader (usually the same replica as the leaseholder).

**Required flag:** `--ranges` or `--all` -`ranges` | The number of ranges that have replicas on the node.

**Required flag:** `--ranges` or `--all` -`ranges_unavailable` | The number of unavailable ranges that have replicas on the node.

**Required flag:** `--ranges` or `--all` -`ranges_underreplicated` | The number of underreplicated ranges that have replicas on the node.

**Required flag:** `--ranges` or `--all` -`live_bytes` | The amount of live data used by both applications and the CockroachDB system. This excludes historical and deleted data.

**Required flag:** `--stats` or `--all` -`key_bytes` | The amount of live and non-live data from keys in the key-value storage layer. This does not include data used by the CockroachDB system.

**Required flag:** `--stats` or `--all` -`value_bytes` | The amount of live and non-live data from values in the key-value storage layer. This does not include data used by the CockroachDB system.

**Required flag:** `--stats` or `--all` -`intent_bytes` | The amount of non-live data associated with uncommitted (or recently-committed) transactions.

**Required flag:** `--stats` or `--all` -`system_bytes` | The amount of data used just by the CockroachDB system.

**Required flag:** `--stats` or `--all` -`is_available` | If `true`, the node is currently available.

**Required flag:** None -`is_live` | If `true`, the node is currently live.

For unavailable clusters (with an unresponsive DB Console), running the `node status` command and monitoring the `is_live` field is the only way to identify the live nodes in the cluster. However, you need to run the `node status` command on a live node to identify the other live nodes in an unavailable cluster. Figuring out a live node to run the command is a trial-and-error process, so run the command against each node until you get one that responds.

See [Identify live nodes in an unavailable cluster](#identify-live-nodes-in-an-unavailable-cluster) for more details.

**Required flag:** None -`gossiped_replicas` | The number of replicas on the node that are active members of a range. After the decommissioning process completes, this should be 0.

**Required flag:** `--decommission` or `--all` -`is_decommissioning` | If `true`, the node's range replicas are being transferred to other nodes. This happens when a live node is marked for [decommissioning](remove-nodes.html).

**Required flag:** `--decommission` or `--all` -`is_draining` | If `true`, the node is being drained of in-flight SQL connections, new SQL connections are rejected, and the [`/health?ready=1` monitoring endpoint](monitoring-and-alerting.html#health-ready-1) starts returning a `503 Service Unavailable` status. This happens when a live node is being [stopped](cockroach-quit.html).

**Required flag:** `--decommission` or `--all` - -### `node decommission` - -Field | Description -------|------------ -`id` | The ID of the node. -`is_live` | If `true`, the node is live. -`replicas` | The number of replicas on the node that are active members of a range. After the decommissioning process completes, this should be 0. -`is_decommissioning` | If `true`, the node's range replicas are being transferred to other nodes. This happens when a live node is marked for [decommissioning](remove-nodes.html). -`is_draining` | If `true`, the node is being drained of in-flight SQL connections, new SQL connections are rejected, and the [`/health?ready=1` monitoring endpoint](monitoring-and-alerting.html#health-ready-1) starts returning a `503 Service Unavailable` status. This happens when a live node is being [stopped](cockroach-quit.html). - -### `node recommission` - -Field | Description -------|------------ -`id` | The ID of the node. -`is_live` | If `true`, the node is live. -`replicas` | The number of replicas on the node that are active members of a range. After the decommissioning process completes, this should be 0. -`is_decommissioning` | If `true`, the node's range replicas are being transferred to other nodes. This happens when a live node is marked for [decommissioning](remove-nodes.html). -`is_draining` | If `true`, the node is being drained of in-flight SQL connections, new SQL connections are rejected, and the [`/health?ready=1` monitoring endpoint](monitoring-and-alerting.html#health-ready-1) starts returning a `503 Service Unavailable` status. This happens when a live node is being [stopped](cockroach-quit.html). - -## Examples - -### Setup - -To follow along with the examples, start [an insecure cluster](start-a-local-cluster.html), with [localities](cockroach-start.html#locality) defined. - -### List node IDs - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach node ls --insecure -~~~ - -~~~ - id -+----+ - 1 - 2 - 3 -(3 rows) -~~~ - -### Show the status of a single node - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach node status 1 --host=localhost:26257 --insecure -~~~ - -~~~ - id | address | sql_address | build | started_at | updated_at | locality | is_available | is_live -+----+-----------------+-----------------+-----------------------------------------+----------------------------------+---------------------------------+---------------------+--------------+---------+ - 1 | localhost:26257 | localhost:26257 | v19.2.0-alpha.20190606-2479-gd98e0839dc | 2019-10-01 20:04:54.308502+00:00 | 2019-10-01 20:05:43.85563+00:00 | region=us-east,az=1 | true | true -(1 row) -~~~ - -### Show the status of all nodes - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach node status --host=localhost:26257 --insecure -~~~ - -~~~ - id | address | sql_address | build | started_at | updated_at | locality | is_available | is_live -+----+-----------------+-----------------+-----------------------------------------+----------------------------------+----------------------------------+------------------------+--------------+---------+ - 1 | localhost:26257 | localhost:26257 | v19.2.0-alpha.20190606-2479-gd98e0839dc | 2019-10-01 20:04:54.308502+00:00 | 2019-10-01 20:06:15.356886+00:00 | region=us-east,az=1 | true | true - 2 | localhost:26258 | localhost:26258 | v19.2.0-alpha.20190606-2479-gd98e0839dc | 2019-10-01 20:04:54.551761+00:00 | 2019-10-01 20:06:15.583967+00:00 | region=us-central,az=2 | true | true - 3 | localhost:26259 | localhost:26259 | v19.2.0-alpha.20190606-2479-gd98e0839dc | 2019-10-01 20:04:55.178577+00:00 | 2019-10-01 20:06:16.204549+00:00 | region=us-west,az=3 | true | true -(3 rows) -~~~ - -### Identify live nodes in an unavailable cluster - -The `is_live` and `is_available` fields are marked as `true` as long as a majority of the nodes are up, and a quorum can be reached: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach quit --host=localhost:26258 --insecure -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach node status --host=localhost:26257 --insecure -~~~ - -~~~ - id | address | sql_address | build | started_at | updated_at | locality | is_available | is_live -+----+-----------------+-----------------+-----------------------------------------+----------------------------------+----------------------------------+------------------------+--------------+---------+ - 1 | localhost:26257 | localhost:26257 | v19.2.0-alpha.20190606-2479-gd98e0839dc | 2019-10-01 20:04:54.308502+00:00 | 2019-10-01 20:07:04.857339+00:00 | region=us-east,az=1 | true | true - 2 | localhost:26258 | localhost:26258 | v19.2.0-alpha.20190606-2479-gd98e0839dc | 2019-10-01 20:04:54.551761+00:00 | 2019-10-01 20:06:48.555863+00:00 | region=us-central,az=2 | false | false - 3 | localhost:26259 | localhost:26259 | v19.2.0-alpha.20190606-2479-gd98e0839dc | 2019-10-01 20:04:55.178577+00:00 | 2019-10-01 20:07:01.207697+00:00 | region=us-west,az=3 | true | true -(3 rows) -~~~ - -If a majority of nodes are down and a quorum cannot be reached, the `is_live` field is marked as `true` for the nodes that are up, but the `is_available` field is marked as `false` for all nodes: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach quit --host=localhost:26259 --insecure -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach node status --host=localhost:26257 --insecure -~~~ - -~~~ - id | address | sql_address | build | started_at | updated_at | locality | is_available | is_live -+----+-----------------+-----------------+-----------------------------------------+----------------------------------+----------------------------------+------------------------+--------------+---------+ - 1 | localhost:26257 | localhost:26257 | v19.2.0-alpha.20190606-2479-gd98e0839dc | 2019-10-01 20:04:54.308502+00:00 | 2019-10-01 20:07:37.464249+00:00 | region=us-east,az=1 | false | true - 2 | localhost:26258 | localhost:26258 | v19.2.0-alpha.20190606-2479-gd98e0839dc | 2019-10-01 20:04:54.551761+00:00 | 2019-10-01 20:07:37.464259+00:00 | region=us-central,az=2 | false | false - 3 | localhost:26259 | localhost:26259 | v19.2.0-alpha.20190606-2479-gd98e0839dc | 2019-10-01 20:04:55.178577+00:00 | 2019-10-01 20:07:37.464265+00:00 | region=us-west,az=3 | false | false -(3 rows) -~~~ - -{{site.data.alerts.callout_info}} -You need to run the `node status` command on a live node to identify the other live nodes in an unavailable cluster. Figuring out a live node to run the command is a trial-and-error process, so run the command against each node until you get one that responds. -{{site.data.alerts.end}} - -### Decommission nodes - -See [Decommission Nodes](remove-nodes.html) - -### Recommission nodes - -See [Recommission Nodes](remove-nodes.html#recommission-nodes) - -## See also - -- [Other Cockroach Commands](cockroach-commands.html) -- [Decommission Nodes](remove-nodes.html) diff --git a/src/current/v21.1/cockroach-nodelocal-upload.md b/src/current/v21.1/cockroach-nodelocal-upload.md deleted file mode 100644 index adb63de8799..00000000000 --- a/src/current/v21.1/cockroach-nodelocal-upload.md +++ /dev/null @@ -1,99 +0,0 @@ ---- -title: cockroach nodelocal upload -summary: The cockroach nodelocal upload command uploads a file to the external IO directory on a node's (the gateway node, by default) local file system. -toc: true ---- - - The `cockroach nodelocal upload` [command](cockroach-commands.html) uploads a file to the external IO directory on a node's (the gateway node, by default) local file system. - -This command takes in a source file to upload and a destination filename. It will then use a SQL connection to upload the file to the node's local file system, at `externalIODir/destination/filename`. - -{{site.data.alerts.callout_info}} -The source file is only uploaded to one node, not all of the nodes. -{{site.data.alerts.end}} - -{% include {{ page.version.version }}/misc/userfile.md %} - -## Required privileges - -Only members of the `admin` role can run `cockroach nodelocal upload`. By default, the `root` user belongs to the `admin` role. - -## Considerations - -The [`--external-io`](cockroach-start.html#general) flag on the node you're uploading to **cannot** be set to `disabled`. - -## Synopsis - -Upload a file: - -~~~ shell -$ cockroach nodelocal upload [flags] -~~~ - -View help: - -~~~ shell -$ cockroach nodelocal upload --help -~~~ - -## Flags - - Flag | Description ------------------+----------------------------------------------------- -`--certs-dir` | The path to the [certificate directory](cockroach-cert.html) containing the CA and client certificates and client key.

**Env Variable:** `COCKROACH_CERTS_DIR`
**Default:** `${HOME}/.cockroach-certs/` -`--echo-sql` | Reveal the SQL statements sent implicitly by the command-line utility. -`--host` | The server host and port number to connect to. This can be the address of any node in the cluster.

**Env Variable:** `COCKROACH_HOST`
**Default:** `localhost:26257` -`--insecure` | Use an insecure connection.

**Env Variable:** `COCKROACH_INSECURE`
**Default:** `false` -`--url` | A [connection URL](connection-parameters.html#connect-using-a-url) to use instead of the other arguments.

**Env Variable:** `COCKROACH_URL`
**Default:** no URL -`--user`
`-u` | The [SQL user](create-user.html) that will own the client session.

**Env Variable:** `COCKROACH_USER`
**Default:** `root` - -## Examples - -### Upload a file - -To upload a file to the default node (i.e., the gateway node): - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach nodelocal upload ./grants.csv test/grants.csv --certs-dir=certs -~~~ - -~~~ -successfully uploaded to nodelocal://1/test/grants.csv -~~~ - -Then, you can use the file to [`IMPORT`](import.html) or [`IMPORT INTO`](import-into.html) data. - -### Upload a file to a specific node - -To upload a file to a specific node (e.g., node 2), use the `--host` flag: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach nodelocal upload ./grants.csv grants.csv --host=localhost:26259 --insecure -~~~ - -~~~ -successfully uploaded to nodelocal://2/grants.csv -~~~ - -Or, use the `--url` flag: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach nodelocal upload ./grants.csv grants.csv --url=postgresql://root@localhost:26258?sslmode=disable --insecure -~~~ - -~~~ -successfully uploaded to nodelocal://3/grants.csv -~~~ - -Then, you can use the file to [`IMPORT`](import.html) or [`IMPORT INTO`](import-into.html) data. - -## See also - -- [Other Cockroach Commands](cockroach-commands.html) -- [Troubleshooting Overview](troubleshooting-overview.html) -- [Import Data](migration-overview.html) -- [`IMPORT`](import.html) -- [`IMPORT INTO`](import-into.html) diff --git a/src/current/v21.1/cockroach-quit.md b/src/current/v21.1/cockroach-quit.md deleted file mode 100644 index 8b585c58a3c..00000000000 --- a/src/current/v21.1/cockroach-quit.md +++ /dev/null @@ -1,141 +0,0 @@ ---- -title: cockroach quit -summary: This page shows you how to use the cockroach quit command to temporarily stop a node that you plan to restart. -toc: true -key: stop-a-node.html ---- - -{{site.data.alerts.callout_danger}} -`cockroach quit` is deprecated. To stop a node, do one of the following: - -{% include {{ page.version.version }}/prod-deployment/node-shutdown.md %} -{{site.data.alerts.end}} - -This page shows you how to use the `cockroach quit` [command](cockroach-commands.html) to temporarily stop a node that you plan to restart. - -You might do this, for example, during the process of [upgrading your cluster's version of CockroachDB](upgrade-cockroach-version.html) or to perform planned maintenance (e.g., upgrading system software). - -{{site.data.alerts.callout_info}} -In other scenarios, such as when downsizing a cluster or reacting to hardware failures, it's best to remove nodes from your cluster entirely. For information about this, see [Decommission Nodes](remove-nodes.html). -{{site.data.alerts.end}} - -## Overview - -### How it works - -When you stop a node, it performs the following steps: - -- Finishes in-flight requests. Note that this is a best effort that times out after the duration specified by the `server.shutdown.query_wait` [cluster setting](cluster-settings.html). -- Gossips its draining state to the cluster, so that other nodes do not try to distribute query planning to the draining node. Note that this is a best effort that times out after the duration specified by the `server.shutdown.drain_wait` [cluster setting](cluster-settings.html), so other nodes may not receive the gossip info in time. - -If the node then stays offline for a certain amount of time (5 minutes by default), the cluster considers the node dead and starts to transfer its **range replicas** to other nodes as well. - -After that, if the node comes back online, its range replicas will determine whether or not they are still valid members of replica groups. If a range replica is still valid and any data in its range has changed, it will receive updates from another replica in the group. If a range replica is no longer valid, it will be removed from the node. - -Basic terms: - -- **Range**: CockroachDB stores all user data and almost all system data in a giant sorted map of key value pairs. This keyspace is divided into "ranges", contiguous chunks of the keyspace, so that every key can always be found in a single range. -- **Range Replica:** CockroachDB replicates each range (3 times by default) and stores each replica on a different node. - -### Considerations - -{% include {{ page.version.version }}/faq/planned-maintenance.md %} - -## Synopsis - -Temporarily stop a node: - -~~~ shell -$ cockroach quit -~~~ - -View help: - -~~~ shell -$ cockroach quit --help -~~~ - -## Flags - -The `quit` command supports the following [general-use](#general), [client connection](#client-connection), and [logging](#logging) flags. - -### General - -Flag | Description ------|------------ -`--drain-wait` | Amount of time to wait for the node to drain before stopping the node. See [`cockroach node drain`](cockroach-node.html) for more details.

**Default:** `10m` - -### Client connection - -{% include {{ page.version.version }}/sql/connection-parameters.md %} -`--cluster-name` | The cluster name to use to verify the cluster's identity. If the cluster has a cluster name, you must include this flag. For more information, see [`cockroach start`](cockroach-start.html#general). -`--disable-cluster-name-verification` | Disables the cluster name check for this command. This flag must be paired with `--cluster-name`. For more information, see [`cockroach start`](cockroach-start.html#general). - -See [Client Connection Parameters](connection-parameters.html) for more details. - -### Logging - -{% include {{ page.version.version }}/misc/logging-defaults.md %} - -## Examples - -### Stop a node from the machine where it's running - -1. SSH to the machine where the node is running. - -2. If the node is running in the background and you are using a process manager for automatic restarts, use the process manager to stop the `cockroach` process without restarting it. - - If the node is running in the background and you are not using a process manager, send a kill signal to the `cockroach` process, for example: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ pkill cockroach - ~~~ - - If the node is running in the foreground, press `CTRL-C`. - -3. Verify that the `cockroach` process has stopped: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ ps aux | grep cockroach - ~~~ - - Alternately, you can check the node's logs for the message `server drained and shutdown completed`. - -### Stop a node from another machine - -
- - -
- -
-1. [Install the `cockroach` binary](install-cockroachdb.html) on a machine separate from the node. - -2. Create a `certs` directory and copy the CA certificate and the client certificate and key for the `root` user into the directory. - -3. Run the `cockroach quit` command: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach quit --certs-dir=certs --host=
- ~~~ -
- -
-1. [Install the `cockroach` binary](install-cockroachdb.html) on a machine separate from the node. - -2. Run the `cockroach quit` command: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach quit --insecure --host=
- ~~~ -
- -## See also - -- [Other Cockroach Commands](cockroach-commands.html) -- [Decommission Nodes](remove-nodes.html) -- [Upgrade a Cluster's Version](upgrade-cockroach-version.html) diff --git a/src/current/v21.1/cockroach-sql.md b/src/current/v21.1/cockroach-sql.md deleted file mode 100644 index 62fa3fc23c7..00000000000 --- a/src/current/v21.1/cockroach-sql.md +++ /dev/null @@ -1,770 +0,0 @@ ---- -title: cockroach sql -summary: CockroachDB comes with a built-in client for executing SQL statements from an interactive shell or directly from the command line. -toc: true -key: use-the-built-in-sql-client.html ---- - -CockroachDB comes with a built-in client for executing SQL statements from an interactive shell or directly from the command line. To use this client, run the `cockroach sql` [command](cockroach-commands.html) as described below. - -To exit the interactive shell, use `\q`, `quit`, `exit`, or `ctrl-d`. - -{{site.data.alerts.callout_success}} -If you want to experiment with CockroachDB SQL but do not have a cluster already running, you can use the [`cockroach demo`](cockroach-demo.html) command to open a shell to a temporary, in-memory cluster. -{{site.data.alerts.end}} - -## Synopsis - -Start the interactive SQL shell: - -~~~ shell -$ cockroach sql -~~~ - -Execute SQL from the command line: - -~~~ shell -$ cockroach sql --execute=";" --execute="" -~~~ -~~~ shell -$ echo ";" | cockroach sql -~~~ -~~~ shell -$ cockroach sql --file file-containing-statements.sql -~~~ - -Exit the interactive SQL shell: - -~~~ sql -> \q -~~~ - -~~~ sql -> quit -~~~ - -~~~ sql -> exit -~~~ - -~~~ shell -ctrl-d -~~~ - -View help: - -~~~ shell -$ cockroach sql --help -~~~ - -## Flags - -The `sql` command supports the following types of flags: - -- [General Use](#general) -- [Client Connection](#client-connection) -- [Logging](#logging) - -### General - -- To start an interactive SQL shell, run `cockroach sql` with all appropriate connection flags or use just the [`--url` flag](#sql-flag-url), which includes [connection details](connection-parameters.html#connect-using-a-url). -- To execute SQL statements from the command line, use the [`--execute` flag](#sql-flag-execute). - -Flag | Description ------|------------ -`--database`

`-d` | A database name to use as [current database](sql-name-resolution.html#current-database) in the newly created session. -`--embedded` | **New in v21.1:** Minimizes the SQL shell [welcome text](#welcome-message) to be appropriate for embedding in playground-type environments. Specifically, this flag removes details that users in an embedded environment have no control over (e.g., networking information). -`--echo-sql` | Reveal the SQL statements sent implicitly by the command-line utility. For a demonstration, see the [example](#reveal-the-sql-statements-sent-implicitly-by-the-command-line-utility) below.

This can also be enabled within the interactive SQL shell via the `\set echo` [shell command](#commands). - `--execute`

`-e` | Execute SQL statements directly from the command line, without opening a shell. This flag can be set multiple times, and each instance can contain one or more statements separated by semi-colons. If an error occurs in any statement, the command exits with a non-zero status code and further statements are not executed. The results of each statement are printed to the standard output (see `--format` for formatting options).

For a demonstration of this and other ways to execute SQL from the command line, see the [example](#execute-sql-statements-from-the-command-line) below. -`--file `

`-f `

`< ` | **New in v21.1:** Read SQL statements from ``. - `--format` | How to display table rows printed to the standard output. Possible values: `tsv`, `csv`, `table`, `raw`, `records`, `sql`, `html`.

**Default:** `table` for sessions that [output on a terminal](#session-and-output-types); `tsv` otherwise

This flag corresponds to the `display_format` [client-side option](#client-side-options). -`--safe-updates` | Disallow potentially unsafe SQL statements, including `DELETE` without a `WHERE` clause, `UPDATE` without a `WHERE` clause, and `ALTER TABLE ... DROP COLUMN`.

**Default:** `true` for [interactive sessions](#session-and-output-types); `false` otherwise

Potentially unsafe SQL statements can also be allowed/disallowed for an entire session via the `sql_safe_updates` [session variable](set-vars.html). -`--set` | Set a [client-side option](#client-side-options) before starting the SQL shell or executing SQL statements from the command line via `--execute`. This flag may be specified multiple times, once per option.

After starting the SQL shell, the `\set` and `unset` commands can be use to enable and disable client-side options as well. -`--watch` | Repeat the SQL statements specified with `--execute` or `-e` until a SQL error occurs or the process is terminated. `--watch` applies to all `--execute` or `-e` flags in use.
You must also specify an interval at which to repeat the statement, followed by a time unit. For example, to specify an interval of 5 seconds, use `5s`.

Note that this flag is intended for simple monitoring scenarios during development and testing. See the [example](#repeat-a-sql-statement) below. - - -### Client connection - -{% include {{ page.version.version }}/sql/connection-parameters.md %} - -See [Client Connection Parameters](connection-parameters.html) for more details. - -### Logging - -{% include {{ page.version.version }}/misc/logging-defaults.md %} - -## Session and output types - -`cockroach sql` exhibits different behaviors depending on whether or not the session is interactive and/or whether or not the session outputs on a terminal. - -- A session is **interactive** when `cockroach sql` is invoked without the `--execute` flag and input is not redirected from a file. In such cases: - - The [`errexit` option](#sql-option-errexit) defaults to `false`. - - The [`check_syntax` option](#sql-option-check-syntax) defaults to `true` if supported by the CockroachDB server (this is checked when the shell starts up). - - **Ctrl+C** at the prompt will only terminate the shell if no other input was entered on the same line already. - - The shell will attempt to set the `safe_updates` [session variable](set-vars.html) to `true` on the server. -- A session **outputs on a terminal** when output is not redirected to a file. In such cases: - - The [`--format` flag](#sql-flag-format) and its corresponding [`display_format` option](#sql-option-display-format) default to `table`. These default to `tsv` otherwise. - - The `show_times` option defaults to `true`. - -When a session is both interactive and outputs on a terminal, `cockroach sql` also activates the interactive prompt with a line editor that can be used to modify the current line of input. Also, command history becomes active. - -## SQL shell - -### Welcome message - -When the SQL shell connects (or reconnects) to a CockroachDB node, it prints a welcome text with some tips and CockroachDB version and cluster details: - -~~~ shell -# -# Welcome to the CockroachDB SQL shell. -# All statements must be terminated by a semicolon. -# To exit, type: \q. -# -# Server version: CockroachDB CCL {{page.release_info.version}} (x86_64-apple-darwin17.7.0, built 2019/09/13 00:07:19, go1.12.6) (same version as client) -# Cluster ID: 7fb9f5b4-a801-4851-92e9-c0db292d03f1 -# -# Enter \? for a brief introduction. -# -> -~~~ - -The **Version** and **Cluster ID** details are particularly noteworthy: - -- When the client and server versions of CockroachDB are the same, the shell prints the `Server version` followed by `(same version as client)`. -- When the client and server versions are different, the shell prints both the `Client version` and `Server version`. In this case, you may want to [plan an upgrade](upgrade-cockroach-version.html) of older client or server versions. -- Since every CockroachDB cluster has a unique ID, you can use the `Cluster ID` field to verify that your client is always connecting to the correct cluster. - -### Commands - -{% include {{ page.version.version }}/sql/shell-commands.md %} - -### Client-side options - -{% include {{ page.version.version }}/sql/shell-options.md %} - -### Help - -{% include {{ page.version.version }}/sql/shell-help.md %} - -### Shortcuts - -{% include {{ page.version.version }}/sql/shell-shortcuts.md %} - -### Error messages and `SQLSTATE` codes - -When CockroachDB encounters a SQL error, it returns the following information to the client (whether `cockroach sql` or another [client application](developer-guide-overview.html)): - -1. An error message, prefixed with [the "Severity" field of the PostgreSQL wire protocol](https://www.postgresql.org/docs/current/protocol-error-fields.html). For example, `ERROR: insert on table "shipments" violates foreign key constraint "fk_customers"`. -2. A [5-digit `SQLSTATE` error code](https://en.wikipedia.org/wiki/SQLSTATE) as defined by the SQL standard. For example, `SQLSTATE: 23503`. - -For example, the following query (taken from [this example of adding multiple foreign key constraints](foreign-key.html#add-multiple-foreign-key-constraints-to-a-single-column)) results in a SQL error, and returns both an error message and a `SQLSTATE` code as described above. - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO shipments (carrier, status, customer_id) VALUES ('DHL', 'At facility', 2000); -~~~ - -~~~ -ERROR: insert on table "shipments" violates foreign key constraint "fk_customers" -SQLSTATE: 23503 -DETAIL: Key (customer_id)=(2000) is not present in table "customers". -~~~ - -The [`SQLSTATE` code](https://en.wikipedia.org/wiki/SQLSTATE) in particular can be helpful in the following ways: - -- It is a standard SQL error code that you can look up in documentation and search for on the web. For any given error state, CockroachDB tries to produce the same `SQLSTATE` code as PostgreSQL. -- If you are developing automation that uses the CockroachDB SQL shell, it is more reliable to check for `SQLSTATE` values than for error message strings, which are likely to change. - -## Examples - -### Start a SQL shell - -In these examples, we connect a SQL shell to a **secure cluster**. - -{% include_cached copy-clipboard.html %} -~~~ shell -# Using standard connection flags: -$ cockroach sql \ ---certs-dir=certs \ ---user=maxroach \ ---host=12.345.67.89 \ ---database=critterdb -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -# Using the --url flag: -$ cockroach sql \ ---url="postgresql://maxroach@12.345.67.89:26257/critterdb?sslcert=certs/client.maxroach.crt&sslkey=certs/client.maxroach.key&sslmode=verify-full&sslrootcert=certs/ca.crt" -~~~ - -In these examples, we connect a SQL shell to an **insecure cluster**. - -{% include_cached copy-clipboard.html %} -~~~ shell -# Using standard connection flags: -$ cockroach sql --insecure \ ---user=maxroach \ ---host=12.345.67.89 \ ---database=critterdb -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -# Using the --url flag: -$ cockroach sql \ ---url="postgresql://maxroach@12.345.67.89:26257/critterdb?sslmode=disable" -~~~ - -### Execute SQL statement within the SQL shell - -This example assume that we have already started the SQL shell (see examples above). - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE animals (id INT PRIMARY KEY DEFAULT unique_rowid(), name STRING); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO animals (name) VALUES ('bobcat'), ('🐢 '), ('barn owl'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM animals; -~~~ - -~~~ -+--------------------+----------+ -| id | name | -+--------------------+----------+ -| 148899952591994881 | bobcat | -| 148899952592060417 | 🐢 | -| 148899952592093185 | barn owl | -+--------------------+----------+ -~~~ - -### Execute SQL statements from the command line - -In these examples, we use the `--execute` flag to execute statements from the command line: - -{% include_cached copy-clipboard.html %} -~~~ shell -# Statements with a single --execute flag: -$ cockroach sql --insecure \ ---execute="CREATE TABLE roaches (name STRING, country STRING); INSERT INTO roaches VALUES ('American Cockroach', 'United States'), ('Brownbanded Cockroach', 'United States')" \ ---user=maxroach \ ---host=12.345.67.89 \ ---database=critterdb -~~~ - -~~~ -CREATE TABLE -INSERT 2 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -# Statements with multiple --execute flags: -$ cockroach sql --insecure \ ---execute="CREATE TABLE roaches (name STRING, country STRING)" \ ---execute="INSERT INTO roaches VALUES ('American Cockroach', 'United States'), ('Brownbanded Cockroach', 'United States')" \ ---user=maxroach \ ---host=12.345.67.89 \ ---database=critterdb -~~~ - -~~~ -CREATE TABLE -INSERT 2 -~~~ - -In this example, we use the `echo` command to execute statements from the command line: - -{% include_cached copy-clipboard.html %} -~~~ shell -# Statements with the echo command: -$ echo "SHOW TABLES; SELECT * FROM roaches;" | cockroach sql --insecure --user=maxroach --host=12.345.67.89 --database=critterdb -~~~ - -~~~ -+----------+ -| Table | -+----------+ -| roaches | -+----------+ -+-----------------------+---------------+ -| name | country | -+-----------------------+---------------+ -| American Cockroach | United States | -| Brownbanded Cockroach | United States | -+-----------------------+---------------+ -~~~ - -### Control how table rows are printed - -In these examples, we show tables and special characters printed in various formats. - -When the standard output is a terminal, `--format` defaults to `table` and tables are printed with ASCII art and special characters are not escaped for easy human consumption: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --insecure \ ---execute="SELECT '🐥' AS chick, '🐢' AS turtle" \ ---user=maxroach \ ---host=12.345.67.89 \ ---database=critterdb -~~~ - -~~~ -+-------+--------+ -| chick | turtle | -+-------+--------+ -| 🐥 | 🐢 | -+-------+--------+ -~~~ - -However, you can explicitly set `--format` to another format, for example, `tsv` or `html`: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --insecure \ ---format=tsv \ ---execute="SELECT '🐥' AS chick, '🐢' AS turtle" \ ---user=maxroach \ ---host=12.345.67.89 \ ---database=critterdb -~~~ - -~~~ -1 row -chick turtle -🐥 🐢 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --insecure \ ---format=html \ ---execute="SELECT '🐥' AS chick, '🐢' AS turtle" \ ---user=maxroach \ ---host=12.345.67.89 \ ---database=critterdb -~~~ - -~~~ -
- - - - -
chickturtle
🐥🐢
-~~~ - -When piping output to another command or a file, `--format` defaults to `tsv`: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --insecure \ ---execute="SELECT '🐥' AS chick, '🐢' AS turtle" > out.txt \ ---user=maxroach \ ---host=12.345.67.89 \ ---database=critterdb -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cat out.txt -~~~ - -~~~ -1 row -chick turtle -🐥 🐢 -~~~ - -However, you can explicitly set `--format` to another format, for example, `table`: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --insecure \ ---format=table \ ---execute="SELECT '🐥' AS chick, '🐢' AS turtle" > out.txt \ ---user=maxroach \ ---host=12.345.67.89 \ ---database=critterdb -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cat out.txt -~~~ - -~~~ -+-------+--------+ -| chick | turtle | -+-------+--------+ -| 🐥 | 🐢 | -+-------+--------+ -(1 row) -~~~ - -### Make the output of `SHOW` statements selectable - -To make it possible to select from the output of `SHOW` statements, set `--format` to `raw`: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --insecure \ ---format=raw \ ---user=maxroach \ ---host=12.345.67.89 \ ---database=critterdb -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE customers; -~~~ - -~~~ -# 2 columns -# row 1 -## 14 -test.customers -## 185 -CREATE TABLE customers ( - id INT NOT NULL, - email STRING NULL, - CONSTRAINT "primary" PRIMARY KEY (id ASC), - UNIQUE INDEX customers_email_key (email ASC), - FAMILY "primary" (id, email) -) -# 1 row -~~~ - -When `--format` is not set to `raw`, you can use the `display_format` [SQL shell option](#client-side-options) to change the output format within the interactive session: - -{% include_cached copy-clipboard.html %} -~~~ sql -> \set display_format raw -~~~ - -~~~ -# 2 columns -# row 1 -## 14 -test.customers -## 185 -CREATE TABLE customers ( - id INT NOT NULL, - email STRING NULL, - CONSTRAINT "primary" PRIMARY KEY (id ASC), - UNIQUE INDEX customers_email_key (email ASC), - FAMILY "primary" (id, email) -) -# 1 row -~~~ - -### Execute SQL statements from a file - -In this example, we show and then execute the contents of a file containing SQL statements. - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cat statements.sql -~~~ - -~~~ -CREATE TABLE roaches (name STRING, country STRING); -INSERT INTO roaches VALUES ('American Cockroach', 'United States'), ('Brownbanded Cockroach', 'United States'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --insecure \ ---user=maxroach \ ---host=12.345.67.89 \ ---database=critterdb \ --f statements.sql -~~~ - -~~~ -CREATE TABLE -INSERT 2 -~~~ - -### Run external commands from the SQL shell - -In this example, we use `\!` to look at the rows in a CSV file before creating a table and then using `\|` to insert those rows into the table. - -{{site.data.alerts.callout_info}}This example works only if the values in the CSV file are numbers. For values in other formats, use an online CSV-to-SQL converter or make your own import program.{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ sql -> \! cat test.csv -~~~ - -~~~ -12, 13, 14 -10, 20, 30 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE csv (x INT, y INT, z INT); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> \| IFS=","; while read a b c; do echo "insert into csv values ($a, $b, $c);"; done < test.csv; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM csv; -~~~ - -~~~ -+----+----+----+ -| x | y | z | -+----+----+----+ -| 12 | 13 | 14 | -| 10 | 20 | 30 | -+----+----+----+ -~~~ - -In this example, we create a table and then use `\|` to programmatically insert values. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE for_loop (x INT); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> \| for ((i=0;i<10;++i)); do echo "INSERT INTO for_loop VALUES ($i);"; done -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM for_loop; -~~~ - -~~~ -+---+ -| x | -+---+ -| 0 | -| 1 | -| 2 | -| 3 | -| 4 | -| 5 | -| 6 | -| 7 | -| 8 | -| 9 | -+---+ -~~~ - -### Edit SQL statements in an external editor - -In applications that use [GNU Readline](https://tiswww.case.edu/php/chet/readline/rltop.html) (such as [bash](https://www.gnu.org/software/bash/)), you can edit a long line in your preferred editor by typing `Ctrl-x Ctrl-e`. However, CockroachDB uses the BSD-licensed [libedit](https://thrysoee.dk/editline/), which does not include this functionality. - -If you would like to be able to edit the current line in an external editor by typing `C-x C-e` as in `bash`, do the following: - -1. Install the `vipe` program (from the [moreutils](https://joeyh.name/code/moreutils/) suite of tools). -2. Edit your `~/.editrc` to add the following line, which takes advantage of the SQL client's ability to [run external commands](#run-external-commands-from-the-sql-shell): - - {% include_cached copy-clipboard.html %} - ~~~ - cockroach:bind -s ^X^E '^A^K\\\| echo \"^Y\" | vipe\r' - ~~~ - -This tells libedit to translate `C-x C-e` into the following commands: - -1. Move to the beginning of the current line. -2. Cut the whole line. -3. Paste the line into your editor via `vipe`. -4. Pass the edited file back to the SQL client when `vipe` exits. - -{{site.data.alerts.callout_info}} -Future versions of the SQL client may opt to use a different back-end for reading input, in which case please refer to this page for additional updates. -{{site.data.alerts.end}} - -### Allow potentially unsafe SQL statements - -The `--safe-updates` flag defaults to `true`. This prevents SQL statements that may have broad, undesired side-effects. For example, by default, we cannot use `DELETE` without a `WHERE` clause to delete all rows from a table: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --insecure --execute="SELECT * FROM db1.t1" -~~~ - -~~~ -+----+------+ -| id | name | -+----+------+ -| 1 | a | -| 2 | b | -| 3 | c | -| 4 | d | -| 5 | e | -| 6 | f | -| 7 | g | -| 8 | h | -| 9 | i | -| 10 | j | -+----+------+ -(10 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --insecure --execute="DELETE FROM db1.t1" -~~~ - -~~~ -Error: pq: rejected: DELETE without WHERE clause (sql_safe_updates = true) -Failed running "sql" -~~~ - -However, to allow an "unsafe" statement, you can set `--safe-updates=false`: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --insecure --safe-updates=false --execute="DELETE FROM db1.t1" -~~~ - -~~~ -DELETE 10 -~~~ - -{{site.data.alerts.callout_info}}Potentially unsafe SQL statements can also be allowed/disallowed for an entire session via the sql_safe_updates session variable.{{site.data.alerts.end}} - -### Reveal the SQL statements sent implicitly by the command-line utility - -In this example, we use the `--execute` flag to execute statements from the command line and the `--echo-sql` flag to reveal SQL statements sent implicitly: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --insecure \ ---execute="CREATE TABLE t1 (id INT PRIMARY KEY, name STRING)" \ ---execute="INSERT INTO t1 VALUES (1, 'a'), (2, 'b'), (3, 'c')" \ ---user=maxroach \ ---host=12.345.67.89 \ ---database=db1 ---echo-sql -~~~ - -~~~ -# Server version: CockroachDB CCL f8f3c9317 (darwin amd64, built 2017/09/13 15:05:35, go1.8) (same version as client) -# Cluster ID: 847a4ba5-c78a-465a-b1a0-59fae3aab520 -> SET sql_safe_updates = TRUE -> CREATE TABLE t1 (id INT PRIMARY KEY, name STRING) -CREATE TABLE -> INSERT INTO t1 VALUES (1, 'a'), (2, 'b'), (3, 'c') -INSERT 3 -~~~ - -In this example, we start the interactive SQL shell and enable the `echo` shell option to reveal SQL statements sent implicitly: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --insecure \ ---user=maxroach \ ---host=12.345.67.89 \ ---database=db1 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> \set echo -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO db1.t1 VALUES (4, 'd'), (5, 'e'), (6, 'f'); -~~~ - -~~~ -> INSERT INTO db1.t1 VALUES (4, 'd'), (5, 'e'), (6, 'f'); -INSERT 3 - -Time: 2.426534ms - -> SHOW TRANSACTION STATUS -> SHOW DATABASE -~~~ - -### Repeat a SQL statement - -Repeating SQL queries on a table can be useful for monitoring purposes. With the `--watch` flag, you can repeat the statements specified with a `--execute` or `-e` flag periodically, until a SQL error occurs or the process is terminated. - -For example, if you want to monitor the number of queries running on the current node, you can use `cockroach sql` with the `--watch` flag to query the node's `crdb_internal.node_statement_statistics` table for the query count: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --insecure \ ---execute="SELECT SUM(count) FROM crdb_internal.node_statement_statistics" \ ---watch 1m -~~~ - -~~~ - sum -+-----+ - 926 -(1 row) - sum -+------+ - 4227 -(1 row) -^C -~~~ - -In this example, the statement is executed every minute. We let the process run for a couple minutes before terminating it with Ctrl+C. - -### Connect to a cluster listening for Unix domain socket connections - -To connect to a cluster that is running on the same machine as your client and is listening for [Unix domain socket](https://en.wikipedia.org/wiki/Unix_domain_socket) connections, [specify a Unix domain socket URI](connection-parameters.html#example-uri-for-a-unix-domain-socket) with the `--url` connection parameter. - -For example, suppose you start a single-node cluster with the following [`cockroach start-single-node`](cockroach-start-single-node.html) command: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start-single-node --insecure --socket-dir=/tmp -~~~ - -~~~ -CockroachDB node starting at 2020-10-12 04:02:54.971369 +0000 UTC (took 1.3s) -build: CCL v20.2.0 @ 2020/10/06 17:15:06 (go1.13.14) -webui: http://Jesses-MBP-2:8080 -sql: postgresql://root@Jesses-MBP-2:26257?sslmode=disable -RPC client flags: ./cockroach --host=Jesses-MBP-2:26257 --insecure -socket: /tmp/.s.PGSQL.26257 -logs: /Users/jesseseldess/Downloads/cockroach-v20.2.0-beta.4.darwin-10.9-amd64/cockroach-data/logs -temp dir: /Users/jesseseldess/Downloads/cockroach-v20.2.0-beta.4.darwin-10.9-amd64/cockroach-data/cockroach-temp805054895 -external I/O path: /Users/jesseseldess/Downloads/cockroach-v20.2.0-beta.4.darwin-10.9-amd64/cockroach-data/extern -store[0]: path=/Users/jesseseldess/Downloads/cockroach-v20.2.0-beta.4.darwin-10.9-amd64/cockroach-data -storage engine: pebble -status: initialized new cluster -clusterID: 455ad71d-21d4-424a-87ad-8097b6b5b99f -nodeID: 1 -~~~ - -To connect to this cluster with a socket: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --url='postgres://root@?host=/tmp&port=26257' -~~~ - -## See also - -- [Client Connection Parameters](connection-parameters.html) -- [`cockroach demo`](cockroach-demo.html) -- [Other Cockroach Commands](cockroach-commands.html) -- [SQL Statements](sql-statements.html) -- [Learn CockroachDB SQL](learn-cockroachdb-sql.html) diff --git a/src/current/v21.1/cockroach-sqlfmt.md b/src/current/v21.1/cockroach-sqlfmt.md deleted file mode 100644 index ea1f61be020..00000000000 --- a/src/current/v21.1/cockroach-sqlfmt.md +++ /dev/null @@ -1,158 +0,0 @@ ---- -title: cockroach sqlfmt -summary: Use cockroach sqlfmt to enhance the text layout of a SQL query. -toc: true -key: use-the-query-formatter.html ---- - -The `cockroach sqlfmt` -[command](cockroach-commands.html) changes the textual formatting of -one or more SQL queries. It recognizes all SQL extensions supported by -CockroachDB. - -A [web interface to this feature](https://sqlfum.pt/) is also available. - -{% include {{ page.version.version }}/misc/experimental-warning.md %} - -## Synopsis - -Use the query formatter interactively: - -~~~ shell -$ cockroach sqlfmt - -CTRL+D -~~~ - -Reformat a SQL query given on the command line: - -~~~ shell -$ cockroach sqlfmt -e "" -~~~ - -Reformat a SQL query already stored in a file: - -~~~ shell -$ cat query.sql | cockroach sqlfmt -~~~ - -## Flags - -The `sqlfmt` command supports the following flags. - -Flag | Description | Default value ------|------|---- -`--execute`
`-e` | Reformat the given SQL query, without reading from standard input. | N/A -`--print-width` | Desired column width of the output. | 80 -`--tab-width` | Number of spaces occupied by a tab character on the final display device. | 4 -`--use-spaces` | Always use space characters for formatting; avoid tab characters. | Use tabs. -`--align` | Use vertical alignment during formatting. | Do not align vertically. -`--no-simplify` | Avoid removing optional grouping parentheses during formatting. | Remove unnecessary grouping parentheses. - -## Examples - -### Reformat a query with constrained column width - -Using the interactive query formatter, output with the default column width (80 columns): - -1. Start the interactive query formatter: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sqlfmt - ~~~ - -2. Press **Enter**. - -3. Run the query: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE TABLE animals (id INT PRIMARY KEY DEFAULT unique_rowid(), name STRING); - ~~~ -4. Press **CTRL+D**. - - ~~~ sql - CREATE TABLE animals ( - id INT PRIMARY KEY DEFAULT unique_rowid(), - name STRING - ) - ~~~ - -Using the command line, output with the column width set to `40`: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sqlfmt --print-width 40 -e "CREATE TABLE animals (id INT PRIMARY KEY DEFAULT unique_rowid(), name STRING);" -~~~ - -~~~ sql -CREATE TABLE animals ( - id - INT - PRIMARY KEY - DEFAULT unique_rowid(), - name STRING -) -~~~ - -### Reformat a query with vertical alignment - -Output with the default vertical alignment: - -~~~ shell -$ cockroach sqlfmt -e "SELECT winner, round(length / (60 * 5)) AS counter FROM players WHERE build = $1 AND (hero = $2 OR region = $3);" -~~~ - -~~~ sql -SELECT -winner, round(length / (60 * 5)) AS counter -FROM -players -WHERE -build = $1 AND (hero = $2 OR region = $3) -~~~ - -Output with vertical alignment: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sqlfmt --align -e "SELECT winner, round(length / (60 * 5)) AS counter FROM players WHERE build = $1 AND (hero = $2 OR region = $3);" -~~~ - -~~~ sql -SELECT winner, round(length / (60 * 5)) AS counter - FROM players - WHERE build = $1 AND (hero = $2 OR region = $3); -~~~ - -### Reformat a query with simplification of parentheses - -Output with the default simplification of parentheses: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sqlfmt -e "SELECT (1 * 2) + 3, (1 + 2) * 3;" -~~~ - -~~~ sql -SELECT 1 * 2 + 3, (1 + 2) * 3 -~~~ - -Output with no simplification of parentheses: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sqlfmt --no-simplify -e "SELECT (1 * 2) + 3, (1 + 2) * 3;" -~~~ - -~~~ sql -SELECT (1 * 2) + 3, (1 + 2) * 3 -~~~ - -## See also - -- [Sequel Fumpt](https://sqlfum.pt/) -- [`cockroach demo`](cockroach-demo.html) -- [`cockroach sql`](cockroach-sql.html) -- [Other Cockroach Commands](cockroach-commands.html) diff --git a/src/current/v21.1/cockroach-start-single-node.md b/src/current/v21.1/cockroach-start-single-node.md deleted file mode 100644 index 81e55cff179..00000000000 --- a/src/current/v21.1/cockroach-start-single-node.md +++ /dev/null @@ -1,431 +0,0 @@ ---- -title: cockroach start-single-node -summary: The cockroach start-single-node command starts a single-node cluster with replication disabled. -toc: true ---- - -This page explains the `cockroach start-single-node` [command](cockroach-commands.html), which you use to start a single-node cluster with replication disabled. A single-node cluster is all you need for quick SQL testing or app development. - -{{site.data.alerts.callout_success}} -To run a multi-node cluster with replicated data for availability and consistency, use [`cockroach start`](cockroach-start.html) and [`cockroach init`](cockroach-init.html). -{{site.data.alerts.end}} - -## Synopsis - -Start a single-node cluster: - -~~~ shell -$ cockroach start-single-node -~~~ - -View help: - -~~~ shell -$ cockroach start-single-node --help -~~~ - -## Flags - -The `cockroach start-single-node` command supports the following [general-use](#general), [networking](#networking), [security](#security), and [logging](#logging) flags. - -Many flags have useful defaults that can be overridden by specifying the flags explicitly. If you specify flags explicitly, however, be sure to do so each time the node is restarted, as they will not be remembered. - -{{site.data.alerts.callout_info}} -The `cockroach start-single-node` flags are identical to [`cockroach start`](cockroach-start.html#flags) flags. However, many of them are not relevant for single-node clusters but are provided for users who want to test concepts that appear in multi-node clusters. These flags are called out as such. In most cases, accepting most defaults is sufficient (see the [examples](#examples) below). -{{site.data.alerts.end}} - -### General - -Flag | Description ------|----------- -`--attrs` | **Not relevant for single-node clusters.** Arbitrary strings, separated by colons, specifying node capability, which might include specialized hardware or number of cores, for example:

`--attrs=ram:64gb`

These can be used to influence the location of data replicas. See [Configure Replication Zones](configure-replication-zones.html#replication-constraints) for full details. -`--background` | Runs the node in the background. Control is returned to the shell only once the node is ready to accept requests, so this is recommended over appending `&` to the command. This flag is **not** available in Windows environments.

**Note:** `--background` is suitable for writing automated test suites or maintenance procedures that need a temporary server process running in the background. It is not intended to be used to start a long-running server, because it does not fully detach from the controlling terminal. Consider using a service manager or a tool like [daemon(8)](https://www.freebsd.org/cgi/man.cgi?query=daemon&sektion=8) instead. -`--cache` | The total size for caches, shared evenly if there are multiple storage devices. This can be a percentage (notated as a decimal or with `%`) or any bytes-based unit, for example:

`--cache=.25`
`--cache=25%`
`--cache=1000000000 ----> 1000000000 bytes`
`--cache=1GB ----> 1000000000 bytes`
`--cache=1GiB ----> 1073741824 bytes`

Note: If you use the `%` notation, you might need to escape the `%` sign, for instance, while configuring CockroachDB through `systemd` service files. For this reason, it's recommended to use the decimal notation instead.

**Default:** `128MiB`

The default cache size is reasonable for local development clusters. For production deployments, this should be increased to 25% or higher. Increasing the cache size will generally improve the node's read performance. See [Recommended Production Settings](recommended-production-settings.html#cache-and-sql-memory-size) for more details. -`--external-io-dir` | The path of the external IO directory with which the local file access paths are prefixed while performing backup and restore operations using local node directories or NFS drives. If set to `disabled`, backups and restores using local node directories and NFS drives are disabled.

**Default:** `extern` subdirectory of the first configured [`store`](#store).

To set the `--external-io-dir` flag to the locations you want to use without needing to restart nodes, create symlinks to the desired locations from within the `extern` directory. -`--listening-url-file` | The file to which the node's SQL connection URL will be written on successful startup, in addition to being printed to the [standard output](#standard-output).

This is particularly helpful in identifying the node's port when an unused port is assigned automatically (`--port=0`). -`--locality` | **Not relevant for single-node clusters.** Arbitrary key-value pairs that describe the location of the node. Locality might include country, region, datacenter, rack, etc. For more details, see [Locality](cockroach-start.html#locality) below. -`--max-disk-temp-storage` | The maximum on-disk storage capacity available to store temporary data for SQL queries that exceed the memory budget (see `--max-sql-memory`). This ensures that JOINs, sorts, and other memory-intensive SQL operations are able to spill intermediate results to disk. This can be a percentage (notated as a decimal or with `%`) or any bytes-based unit (e.g., `.25`, `25%`, `500GB`, `1TB`, `1TiB`).

Note: If you use the `%` notation, you might need to escape the `%` sign, for instance, while configuring CockroachDB through `systemd` service files. For this reason, it's recommended to use the decimal notation instead. Also, if expressed as a percentage, this value is interpreted relative to the size of the first store. However, the temporary space usage is never counted towards any store usage; therefore, when setting this value, it's important to ensure that the size of this temporary storage plus the size of the first store doesn't exceed the capacity of the storage device.

The temporary files are located in the path specified by the `--temp-dir` flag, or in the subdirectory of the first store (see `--store`) by default.

**Default:** `32GiB` -`--max-sql-memory` | The maximum in-memory storage capacity available to store temporary data for SQL queries, including prepared queries and intermediate data rows during query execution. This can be a percentage (notated as a decimal or with `%`) or any bytes-based unit, for example:

`--max-sql-memory=.25`
`--max-sql-memory=25%`
`--max-sql-memory=10000000000 ----> 1000000000 bytes`
`--max-sql-memory=1GB ----> 1000000000 bytes`
`--max-sql-memory=1GiB ----> 1073741824 bytes`

The temporary files are located in the path specified by the `--temp-dir` flag, or in the subdirectory of the first store (see `--store`) by default.

Note: If you use the `%` notation, you might need to escape the `%` sign, for instance, while configuring CockroachDB through `systemd` service files. For this reason, it's recommended to use the decimal notation instead.

**Default:** `25%`

The default SQL memory size is suitable for production deployments but can be raised to increase the number of simultaneous client connections the node allows as well as the node's capacity for in-memory processing of rows when using `ORDER BY`, `GROUP BY`, `DISTINCT`, joins, and window functions. For local development clusters with memory-intensive workloads, reduce this value to, for example, `128MiB` to prevent out of memory errors. -`--pid-file` | The file to which the node's process ID will be written on successful startup. When this flag is not set, the process ID is not written to file. -`--store`
`-s` | The file path to a storage device and, optionally, store attributes and maximum size. When using multiple storage devices for a node, this flag must be specified separately for each device, for example:

`--store=/mnt/ssd01 --store=/mnt/ssd02`

For more details, see [Store](#store) below. -`--temp-dir` | The path of the node's temporary store directory. On node start up, the location for the temporary files is printed to the standard output.

**Default:** Subdirectory of the first [store](#store) - -### Networking - -Flag | Description ------|----------- -`--listen-addr` | The IP address/hostname and port to listen on for connections from clients. For IPv6, use the notation `[...]`, e.g., `[::1]` or `[fe80::f6f2:::]`.

**Default:** Listen on all IP addresses on port `26257` -`--http-addr` | The IP address/hostname and port to listen on for DB Console HTTP requests. For IPv6, use the notation `[...]`, e.g., `[::1]:8080` or `[fe80::f6f2:::]:8080`.

**Default:** Listen on the address part of `--listen-addr` on port `8080` -`--socket-dir` | The directory path on which to listen for [Unix domain socket](https://en.wikipedia.org/wiki/Unix_domain_socket) connections from clients installed on the same Unix-based machine. For an example, see [Connect to a cluster listening for Unix domain socket connections](cockroach-sql.html#connect-to-a-cluster-listening-for-unix-domain-socket-connections). - -### Security - -Flag | Description ------|----------- -`--certs-dir` | The path to the [certificate directory](cockroach-cert.html). The directory must contain valid certificates if running in secure mode.

**Default:** `${HOME}/.cockroach-certs/` -`--insecure` | **Note:** The `--insecure` flag is intended for **non-production testing only**.

Run in insecure mode, skipping all TLS encryption and authentication. If this flag is not set, the `--certs-dir` flag must point to valid certificates.

**Note the following risks:** An insecure cluster is open to any client that can access any node's IP addresses; client connections must also be made insecurely; any user, even `root`, can log in without providing a password; any user, connecting as `root`, can read or write any data in your cluster; there is no network encryption or authentication, and thus no confidentiality.

**Default:** `false` -`--accept-sql-without-tls` | This experimental flag allows you to connect to the cluster using a SQL user's password without [validating the client's certificate](authentication.html#client-authentication). When connecting using the built-in SQL client, [use the `--insecure` flag with the `cockroach sql` command](cockroach-sql.html#client-connection). -`--cert-principal-map` | A comma-separated list of `cert-principal:db-principal` mappings used to map the certificate principals to IP addresses, DNS names, and SQL users. This allows the use of certificates generated by Certificate Authorities that place restrictions on the contents of the `commonName` field. For usage information, see [Create Security Certificates using Openssl](create-security-certificates-openssl.html#examples). -`--enterprise-encryption` | This optional flag specifies the encryption options for one of the stores on the node. If multiple stores exist, the flag must be specified for each store.

This flag takes a number of options. For a complete list of options, and usage instructions, see [Encryption at Rest](encryption.html).

Note that this is an [Enterprise feature](enterprise-licensing.html). - -### Store - -The `--store` flag supports the following fields. Note that commas are used to separate fields, and so are forbidden in all field values. - -{{site.data.alerts.callout_info}} -In-memory storage is not suitable for production deployments at this time. -{{site.data.alerts.end}} - -Field | Description -------|------------ -`type` | For in-memory storage, set this field to `mem`; otherwise, leave this field out. The `path` field must not be set when `type=mem`. -`path` | The file path to the storage device. When not setting `attr` or `size`, the `path` field label can be left out:

`--store=/mnt/ssd01`

When either of those fields are set, however, the `path` field label must be used:

`--store=path=/mnt/ssd01,size=20GB`

**Default:** `cockroach-data` -`attrs` | Arbitrary strings, separated by colons, specifying disk type or capability. These can be used to influence the location of data replicas. See [Configure Replication Zones](configure-replication-zones.html#replication-constraints) for full details.

In most cases, node-level `--locality` or `--attrs` are preferable to store-level attributes, but this field can be used to match capabilities for storage of individual databases or tables. For example, an OLTP database would probably want to allocate space for its tables only on solid state devices, whereas append-only time series might prefer cheaper spinning drives. Typical attributes include whether the store is flash (`ssd`) or spinny disk (`hdd`), as well as speeds and other specs, for example:

`--store=path=/mnt/hda1,attrs=hdd:7200rpm` -`size` | The maximum size allocated to the node. When this size is reached, CockroachDB attempts to rebalance data to other nodes with available capacity. When there's no capacity elsewhere, this limit will be exceeded. Also, data may be written to the node faster than the cluster can rebalance it away; in this case, as long as capacity is available elsewhere, CockroachDB will gradually rebalance data down to the store limit.

The `size` can be specified either in a bytes-based unit or as a percentage of hard drive space (notated as a decimal or with `%`), for example:

`--store=path=/mnt/ssd01,size=10000000000 ----> 10000000000 bytes`
`--store=path=/mnt/ssd01,size=20GB ----> 20000000000 bytes`
`--store=path=/mnt/ssd01,size=20GiB ----> 21474836480 bytes`
`--store=path=/mnt/ssd01,size=0.02TiB ----> 21474836480 bytes`
`--store=path=/mnt/ssd01,size=20% ----> 20% of available space`
`--store=path=/mnt/ssd01,size=0.2 ----> 20% of available space`
`--store=path=/mnt/ssd01,size=.2 ----> 20% of available space`

**Default:** 100%

For an in-memory store, the `size` field is required and must be set to the true maximum bytes or percentage of available memory, for example:

`--store=type=mem,size=20GB`
`--store=type=mem,size=90%`

Note: If you use the `%` notation, you might need to escape the `%` sign, for instance, while configuring CockroachDB through `systemd` service files. For this reason, it's recommended to use the decimal notation instead. - -### Logging - -By default, `cockroach start-single-node` writes all messages to log files, and prints nothing to `stderr`. This includes events with `INFO` [severity](logging.html#logging-levels-severities) and higher. However, you can [customize the logging behavior](configure-logs.html) of this command by using the `--log` flag: - -{% include {{ page.version.version }}/misc/logging-flags.md %} - -#### Defaults - -See the [default logging configuration](configure-logs.html#default-logging-configuration). - -## Standard output - -When you run `cockroach start-single-node`, some helpful details are printed to the standard output: - -~~~ shell -CockroachDB node starting at {{ now | date: "%Y-%m-%d %H:%M:%S.%6 +0000 UTC" }} -build: CCL {{page.release_info.version}} @ {{page.release_info.build_time}} (go1.12.6) -webui: http://localhost:8080 -sql: postgresql://root@localhost:26257?sslmode=disable -RPC client flags: cockroach --host=localhost:26257 --insecure -logs: /Users//node1/logs -temp dir: /Users//node1/cockroach-temp242232154 -external I/O path: /Users//node1/extern -store[0]: path=/Users//node1 -status: initialized new cluster -clusterID: 8a681a16-9623-4fc1-a537-77e9255daafd -nodeID: 1 -~~~ - -{{site.data.alerts.callout_success}} -These details are also written to the `INFO` log in the `/logs` directory. You can retrieve them with a command like `grep 'node starting' node1/logs/cockroach.log -A 11`. -{{site.data.alerts.end}} - -Field | Description -------|------------ -`build` | The version of CockroachDB you are running. -`webui` | The URL for accessing the DB Console. -`sql` | The connection URL for your client. -`RPC client flags` | The flags to use when connecting to the node via [`cockroach` client commands](cockroach-commands.html). -`logs` | The directory containing debug log data. -`temp dir` | The temporary store directory of the node. -`external I/O path` | The external IO directory with which the local file access paths are prefixed while performing [backup](backup.html) and [restore](restore.html) operations using local node directories or NFS drives. -`attrs` | If node-level attributes were specified in the `--attrs` flag, they are listed in this field. These details are potentially useful for [configuring replication zones](configure-replication-zones.html). -`locality` | If values describing the locality of the node were specified in the `--locality` field, they are listed in this field. These details are potentially useful for [configuring replication zones](configure-replication-zones.html). -`store[n]` | The directory containing store data, where `[n]` is the index of the store, e.g., `store[0]` for the first store, `store[1]` for the second store.

If store-level attributes were specified in the `attrs` field of the [`--store`](#store) flag, they are listed in this field as well. These details are potentially useful for [configuring replication zones](configure-replication-zones.html). -`status` | Whether the node is the first in the cluster (`initialized new cluster`), joined an existing cluster for the first time (`initialized new node, joined pre-existing cluster`), or rejoined an existing cluster (`restarted pre-existing node`). -`clusterID` | The ID of the cluster.

When trying to join a node to an existing cluster, if this ID is different than the ID of the existing cluster, the node has started a new cluster. This may be due to conflicting information in the node's data directory. For additional guidance, see the [troubleshooting](common-errors.html#node-belongs-to-cluster-cluster-id-but-is-attempting-to-connect-to-a-gossip-network-for-cluster-another-cluster-id) docs. -`nodeID` | The ID of the node. - -## Examples - -### Start a single-node cluster - -
- - -
- -
-1. Create two directories for certificates: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ mkdir certs my-safe-directory - ~~~ - - Directory | Description - ----------|------------ - `certs` | You'll generate your CA certificate and all node and client certificates and keys in this directory. - `my-safe-directory` | You'll generate your CA key in this directory and then reference the key when generating node and client certificates. - -2. Create the CA (Certificate Authority) certificate and key pair: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach cert create-ca \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key - ~~~ - -3. Create the certificate and key pair for the node: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach cert create-node \ - localhost \ - $(hostname) \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key - ~~~ - -4. Create a client certificate and key pair for the `root` user: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach cert create-client \ - root \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key - ~~~ - -5. Start the single-node cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start-single-node \ - --certs-dir=certs \ - --listen-addr=localhost:26257 \ - --http-addr=localhost:8080 \ - --background - ~~~ -
- -
-

-{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start-single-node \ ---insecure \ ---listen-addr=localhost:26257 \ ---http-addr=localhost:8080 \ ---background -~~~ -
- -### Scale to multiple nodes - -Scaling a cluster started with `cockroach start-single-node` involves restarting the first node with the `cockroach start` command instead, and then adding new nodes with that command as well, all using a `--join` flag that forms them into a single multi-node cluster. Since replication is disabled in clusters started with `start-single-node`, you also need to enable replication to get CockroachDB's availability and consistency guarantees. - -
- - -
- -
-1. Stop the single-node cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach quit \ - --certs-dir=certs \ - --host=localhost:26257 - ~~~ - -2. Restart the node with the [`cockroach start`](cockroach-start.html) command: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --certs-dir=certs \ - --listen-addr=localhost:26257 \ - --http-addr=localhost:8080 \ - --join=localhost:26257,localhost:26258,localhost:26259 \ - --background - ~~~ - - The new flag to note is `--join`, which specifies the addresses and ports of the nodes that will initially comprise your cluster. You'll use this exact `--join` flag when starting other nodes as well. - - {% include {{ page.version.version }}/prod-deployment/join-flag-single-region.md %} - -3. Add two more nodes: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --certs-dir=certs \ - --store=node2 \ - --listen-addr=localhost:26258 \ - --http-addr=localhost:8081 \ - --join=localhost:26257,localhost:26258,localhost:26259 \ - --background - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --certs-dir=certs \ - --store=node3 \ - --listen-addr=localhost:26259 \ - --http-addr=localhost:8082 \ - --join=localhost:26257,localhost:26258,localhost:26259 \ - --background - ~~~ - - These commands are the same as before but with unique `--store`, `--listen-addr`, and `--http-addr` flags, since this all nodes are running on the same machine. Also, since all nodes use the same hostname (`localhost`), you can use the first node's certificate. Note that this is different than running a production cluster, where you would need to generate a certificate and key for each node, issued to all common names and IP addresses you might use to refer to the node as well as to any load balancer instances. - -4. Open the [built-in SQL shell](cockroach-sql.html): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --certs-dir=certs --host=localhost:26257 - ~~~ - -5. Update preconfigured [replication zones](configure-replication-zones.html) to replicate user data 3 times and import internal data 5 times: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER RANGE default CONFIGURE ZONE USING num_replicas = 3; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER RANGE system CONFIGURE ZONE USING num_replicas = 5; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER database system CONFIGURE ZONE USING num_replicas = 5; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER RANGE liveness CONFIGURE ZONE USING num_replicas = 5; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER RANGE meta CONFIGURE ZONE USING num_replicas = 5; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER TABLE system.public.jobs CONFIGURE ZONE USING num_replicas = 5; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER TABLE system.public.replication_constraint_stats CONFIGURE ZONE USING num_replicas = 5; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER TABLE system.public.replication_stats CONFIGURE ZONE USING num_replicas = 5; - ~~~ -
- -
-1. Stop the single-node cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach quit \ - --insecure \ - --host=localhost:26257 - ~~~ - -2. Restart the node with the [`cockroach start`](cockroach-start.html) command: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --insecure \ - --listen-addr=localhost:26257 \ - --http-addr=localhost:8080 \ - --join=localhost:26257,localhost:26258,localhost:26259 \ - --background - ~~~ - - The new flag to note is `--join`, which specifies the addresses and ports of the nodes that will comprise your cluster. You'll use this exact `--join` flag when starting other nodes as well. - -3. Add two more nodes: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --insecure \ - --store=node2 \ - --listen-addr=localhost:26258 \ - --http-addr=localhost:8081 \ - --join=localhost:26257,localhost:26258,localhost:26259 \ - --background - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --insecure \ - --store=node3 \ - --listen-addr=localhost:26259 \ - --http-addr=localhost:8082 \ - --join=localhost:26257,localhost:26258,localhost:26259 \ - --background - ~~~ - - These commands are the same as before but with unique `--store`, `--listen-addr`, and `--http-addr` flags, since this all nodes are running on the same machine. - -4. Open the [built-in SQL shell](cockroach-sql.html): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --insecure --host=localhost:26257 - ~~~ - -5. Update preconfigured [replication zones](configure-replication-zones.html) to replicate user data 3 times and import internal data 5 times: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER RANGE default CONFIGURE ZONE USING num_replicas = 3; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER RANGE system CONFIGURE ZONE USING num_replicas = 5; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER database system CONFIGURE ZONE USING num_replicas = 5; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER RANGE liveness CONFIGURE ZONE USING num_replicas = 5; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER RANGE meta CONFIGURE ZONE USING num_replicas = 5; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER TABLE system.public.jobs CONFIGURE ZONE USING num_replicas = 5; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER TABLE system.public.replication_constraint_stats CONFIGURE ZONE USING num_replicas = 5; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER TABLE system.public.replication_stats CONFIGURE ZONE USING num_replicas = 5; - ~~~ -
- -## See also - -- Running a local multi-node cluster: - - [From Binary](start-a-local-cluster.html) - - [In Kubernetes](orchestrate-a-local-cluster-with-kubernetes.html) - - [In Docker](start-a-local-cluster-in-docker-mac.html) -- Running a distributed multi-node cluster: - - [From Binary](manual-deployment.html) - - [In Kubernetes](deploy-cockroachdb-with-kubernetes.html) -- [Other Cockroach Commands](cockroach-commands.html) diff --git a/src/current/v21.1/cockroach-start.md b/src/current/v21.1/cockroach-start.md deleted file mode 100644 index f0fc0b517ae..00000000000 --- a/src/current/v21.1/cockroach-start.md +++ /dev/null @@ -1,526 +0,0 @@ ---- -title: cockroach start -summary: Start a new multi-node cluster or add nodes to an existing multi-node cluster. -toc: true -key: start-a-node.html ---- - -This page explains the `cockroach start` [command](cockroach-commands.html), which you use to start a new multi-node cluster or add nodes to an existing cluster. - -{{site.data.alerts.callout_success}} -If you need a simple single-node backend for app development, use [`cockroach start-single-node`](cockroach-start-single-node.html) instead. For quick SQL testing, consider using [`cockroach demo`](cockroach-demo.html) to start a temporary, in-memory cluster with immediate access to an interactive SQL shell. -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -Node-level settings are defined by [flags](#flags) passed to the `cockroach start` command and cannot be changed without stopping and restarting the node. In contrast, some cluster-wide settings are defined via SQL statements and can be updated anytime after a cluster has been started. For more details, see [Cluster Settings](cluster-settings.html). -{{site.data.alerts.end}} - -## Synopsis - -Start a node to be part of a new multi-node cluster: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start -~~~ - -Initialize a new multi-node cluster: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach init -~~~ - -Add a node to an existing cluster: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start -~~~ - -View help: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start --help -~~~ - -## Flags - -The `cockroach start` command supports the following [general-use](#general), [networking](#networking), [security](#security), and [logging](#logging) flags. - -Many flags have useful defaults that can be overridden by specifying the flags explicitly. If you specify flags explicitly, however, be sure to do so each time the node is restarted, as they will not be remembered. The one exception is the `--join` flag, which is stored in a node's data directory. We still recommend specifying the `--join` flag every time, as this will allow nodes to rejoin the cluster even if their data directory was destroyed. - -### General - -Flag | Description ------|----------- -`--attrs` | Arbitrary strings, separated by colons, specifying node capability, which might include specialized hardware or number of cores, for example:

`--attrs=ram:64gb`

These can be used to influence the location of data replicas. See [Configure Replication Zones](configure-replication-zones.html#replication-constraints) for full details. -`--background` | Runs the node in the background. Control is returned to the shell only once the node is ready to accept requests, so this is recommended over appending `&` to the command. This flag is **not** available in Windows environments.

**Note:** `--background` is suitable for writing automated test suites or maintenance procedures that need a temporary server process running in the background. It is not intended to be used to start a long-running server, because it does not fully detach from the controlling terminal. Consider using a service manager or a tool like [daemon(8)](https://www.freebsd.org/cgi/man.cgi?query=daemon&sektion=8) instead. -`--cache` | The total size for caches, shared evenly if there are multiple storage devices. This can be a percentage (notated as a decimal or with `%`) or any bytes-based unit, for example:

`--cache=.25`
`--cache=25%`
`--cache=1000000000 ----> 1000000000 bytes`
`--cache=1GB ----> 1000000000 bytes`
`--cache=1GiB ----> 1073741824 bytes`

Note: If you use the `%` notation, you might need to escape the `%` sign, for instance, while configuring CockroachDB through `systemd` service files. For this reason, it's recommended to use the decimal notation instead.

**Default:** `128MiB`

The default cache size is reasonable for local development clusters. For production deployments, this should be increased to 25% or higher. Increasing the cache size will generally improve the node's read performance. See [Recommended Production Settings](recommended-production-settings.html#cache-and-sql-memory-size) for more details. -`--clock-device` | Enable CockroachDB to use a [PTP hardware clock](https://www.kernel.org/doc/html/latest/driver-api/ptp.html) when querying the current time. The value is a string that specifies the clock device to use. For example: `--clock-device=/dev/ptp0`

**Note:** This is supported on Linux only and may be needed in cases where the host clock is unreliable or prone to large jumps (e.g., when using vMotion). -`--cluster-name` | A string that specifies a cluster name. This is used together with `--join` to ensure that all newly created nodes join the intended cluster when you are running multiple clusters.

**Note:** If this is set, [`cockroach init`](cockroach-init.html), [`cockroach node decommission`](cockroach-node.html), [`cockroach node recommission`](cockroach-node.html), and the `cockroach debug` commands must specify either `--cluster-name` or `--disable-cluster-name-verification` in order to work. -`--disable-cluster-name-verification` | On clusters for which a cluster name has been set, this flag paired with `--cluster-name` disables the cluster name check for the command. This is necessary on existing clusters, when setting a cluster name or changing the cluster name: Perform a rolling restart of all nodes and include both the new `--cluster-name` value and `--disable-cluster-name-verification`, then a second rolling restart with `--cluster-name` and without `--disable-cluster-name-verification`. -`--external-io-dir` | The path of the external IO directory with which the local file access paths are prefixed while performing backup and restore operations using local node directories or NFS drives. If set to `disabled`, backups and restores using local node directories and NFS drives, as well as [`cockroach nodelocal upload`](cockroach-nodelocal-upload.html), are disabled.

**Default:** `extern` subdirectory of the first configured [`store`](#store).

To set the `--external-io-dir` flag to the locations you want to use without needing to restart nodes, create symlinks to the desired locations from within the `extern` directory. -`--listening-url-file` | The file to which the node's SQL connection URL will be written as soon as the node is ready to accept connections, in addition to being printed to the [standard output](#standard-output). When `--background` is used, this happens before the process detaches from the terminal.

This is particularly helpful in identifying the node's port when an unused port is assigned automatically (`--port=0`). -`--locality` | Arbitrary key-value pairs that describe the location of the node. Locality might include country, region, availability zone, etc. A `region` tier must be included in order to enable [multi-region capabilities](multiregion-overview.html). For more details, see [Locality](#locality) below. -`--max-disk-temp-storage` | The maximum on-disk storage capacity available to store temporary data for SQL queries that exceed the memory budget (see `--max-sql-memory`). This ensures that JOINs, sorts, and other memory-intensive SQL operations are able to spill intermediate results to disk. This can be a percentage (notated as a decimal or with `%`) or any bytes-based unit (e.g., `.25`, `25%`, `500GB`, `1TB`, `1TiB`).

Note: If you use the `%` notation, you might need to escape the `%` sign, for instance, while configuring CockroachDB through `systemd` service files. For this reason, it's recommended to use the decimal notation instead. Also, if expressed as a percentage, this value is interpreted relative to the size of the first store. However, the temporary space usage is never counted towards any store usage; therefore, when setting this value, it's important to ensure that the size of this temporary storage plus the size of the first store doesn't exceed the capacity of the storage device.

The temporary files are located in the path specified by the `--temp-dir` flag, or in the subdirectory of the first store (see `--store`) by default.

**Default:** `32GiB` -`--max-offset` | The maximum allowed clock offset for the cluster. If observed clock offsets exceed this limit, servers will crash to minimize the likelihood of reading inconsistent data. Increasing this value will increase the time to recovery of failures as well as the frequency of uncertainty-based read restarts.

Note that this value must be the same on all nodes in the cluster and cannot be changed with a [rolling upgrade](upgrade-cockroach-version.html). In order to change it, first stop every node in the cluster. Then once the entire cluster is offline, restart each node with the new value.

**Default:** `500ms` -`--max-sql-memory` | The maximum in-memory storage capacity available to store temporary data for SQL queries, including prepared queries and intermediate data rows during query execution. This can be a percentage (notated as a decimal or with `%`) or any bytes-based unit, for example:

`--max-sql-memory=.25`
`--max-sql-memory=25%`
`--max-sql-memory=10000000000 ----> 1000000000 bytes`
`--max-sql-memory=1GB ----> 1000000000 bytes`
`--max-sql-memory=1GiB ----> 1073741824 bytes`

The temporary files are located in the path specified by the `--temp-dir` flag, or in the subdirectory of the first store (see `--store`) by default.

Note: If you use the `%` notation, you might need to escape the `%` sign, for instance, while configuring CockroachDB through `systemd` service files. For this reason, it's recommended to use the decimal notation instead.

**Default:** `25%`

The default SQL memory size is suitable for production deployments but can be raised to increase the number of simultaneous client connections the node allows as well as the node's capacity for in-memory processing of rows when using `ORDER BY`, `GROUP BY`, `DISTINCT`, joins, and window functions. For local development clusters with memory-intensive workloads, reduce this value to, for example, `128MiB` to prevent out of memory errors. -`--pid-file` | The file to which the node's process ID will be written as soon as the node is ready to accept connections. When `--background` is used, this happens before the process detaches from the terminal. When this flag is not set, the process ID is not written to file. -`--store`
`-s` | The file path to a storage device and, optionally, store attributes and maximum size. When using multiple storage devices for a node, this flag must be specified separately for each device, for example:

`--store=/mnt/ssd01 --store=/mnt/ssd02`

For more details, see [Store](#store) below. -`--spatial-libs` | The location on disk where CockroachDB looks for [spatial](spatial-features.html) libraries.

**Defaults:**
  • `/usr/local/lib/cockroach`
  • A `lib` subdirectory of the CockroachDB binary's current directory.

-`--temp-dir` | The path of the node's temporary store directory. On node start up, the location for the temporary files is printed to the standard output.

**Default:** Subdirectory of the first [store](#store) - -### Networking - -Flag | Description ------|----------- -`--experimental-dns-srv` | When this flag is included, the node will first attempt to fetch SRV records from DNS for every name specified with `--join`. If a valid SRV record is found, that information is used instead of regular DNS A/AAAA lookups. This feature is experimental and may be removed or modified in a later version. -`--listen-addr` | The IP address/hostname and port to listen on for connections from other nodes and clients. For IPv6, use the notation `[...]`, e.g., `[::1]` or `[fe80::f6f2:::]`.

This flag's effect depends on how it is used in combination with `--advertise-addr`. For example, the node will also advertise itself to other nodes using this value if `--advertise-addr` is not specified. For more details, see [Networking](recommended-production-settings.html#networking).

**Default:** Listen on all IP addresses on port `26257`; if `--advertise-addr` is not specified, also advertise the node's canonical hostname to other nodes -`--advertise-addr` | The IP address/hostname and port to tell other nodes to use. If using a hostname, it must be resolvable from all nodes. If using an IP address, it must be routable from all nodes; for IPv6, use the notation `[...]`, e.g., `[::1]` or `[fe80::f6f2:::]`.

This flag's effect depends on how it is used in combination with `--listen-addr`. For example, if the port number is different than the one used in `--listen-addr`, port forwarding is required. For more details, see [Networking](recommended-production-settings.html#networking).

**Default:** The value of `--listen-addr`; if `--listen-addr` is not specified, advertises the node's canonical hostname and port `26257` -`--http-addr` | The IP address/hostname and port to listen on for DB Console HTTP requests. For IPv6, use the notation `[...]`, e.g., `[::1]:8080` or `[fe80::f6f2:::]:8080`.

**Default:** Listen on the address part of `--listen-addr` on port `8080` -`--locality-advertise-addr` | The IP address/hostname and port to tell other nodes in specific localities to use. This flag is useful when running a cluster across multiple networks, where nodes in a given network have access to a private or local interface while nodes outside the network do not. In this case, you can use `--locality-advertise-addr` to tell nodes within the same network to prefer the private or local address to improve performance and use `--advertise-addr` to tell nodes outside the network to use another address that is reachable from them.

This flag relies on nodes being started with the [`--locality`](#locality) flag and uses the `locality@address` notation, for example:

`--locality-advertise-addr=region=us-west@10.0.0.0:26257`

See the [example](#start-a-multi-node-cluster-across-private-networks) below for more details. -`--sql-addr` | The IP address/hostname and port to listen on for SQL connections from clients. For IPv6, use the notation `[...]`, e.g., `[::1]` or `[fe80::f6f2:::]`.

This flag's effect depends on how it is used in combination with `--advertise-sql-addr`. For example, the node will also advertise itself to clients using this value if `--advertise-sql-addr` is not specified.

**Default:** The value of `--listen-addr`; if `--listen-addr` is not specified, advertises the node's canonical hostname and port `26257`

For an example, see [Start a cluster with separate RPC and SQL networks](#start-a-cluster-with-separate-rpc-and-sql-networks) -`--advertise-sql-addr` | The IP address/hostname and port to tell clients to use. If using a hostname, it must be resolvable from all nodes. If using an IP address, it must be routable from all nodes; for IPv6, use the notation `[...]`, e.g., `[::1]` or `[fe80::f6f2:::]`.

This flag's effect depends on how it is used in combination with `--sql-addr`. For example, if the port number is different than the one used in `--sql-addr`, port forwarding is required.

**Default:** The value of `--sql-addr`; if `--sql-addr` is not specified, advertises the value of `--listen-addr` -`--join`
`-j` | The host addresses that connect nodes to the cluster and distribute the rest of the node addresses. These can be IP addresses or DNS aliases of nodes.

When starting a cluster in a single region, specify the addresses of 3-5 initial nodes. When starting a cluster in multiple regions, specify more than 1 address per region, and select nodes that are spread across failure domains. Then run the [`cockroach init`](cockroach-init.html) command against any of these nodes to complete cluster startup. See the [example](#start-a-multi-node-cluster) below for more details.

Use the same `--join` list for all nodes to ensure that the cluster can stabilize. Do not list every node in the cluster, because this increases the time for a new cluster to stabilize. Note that these are best practices; it is not required to restart an existing node to update its `--join` flag.

`cockroach start` must be run with the `--join` flag. To start a single-node cluster, use `cockroach start-single-node` instead. -`--socket-dir` | The directory path on which to listen for [Unix domain socket](https://en.wikipedia.org/wiki/Unix_domain_socket) connections from clients installed on the same Unix-based machine. For an example, see [Connect to a cluster listening for Unix domain socket connections](cockroach-sql.html#connect-to-a-cluster-listening-for-unix-domain-socket-connections). -`--advertise-host` | **Deprecated.** Use `--advertise-addr` instead. -`--host` | **Deprecated.** Use `--listen-addr` instead. -`--port`
`-p` | **Deprecated.** Specify port in `--advertise-addr` and/or `--listen-addr` instead. -`--http-host` | **Deprecated.** Use `--http-addr` instead. -`--http-port` | **Deprecated.** Specify port in `--http-addr` instead. - -### Security - -Flag | Description ------|----------- -`--certs-dir` | The path to the [certificate directory](cockroach-cert.html). The directory must contain valid certificates if running in secure mode.

**Default:** `${HOME}/.cockroach-certs/` -`--insecure` | **Note:** The `--insecure` flag is intended for **non-production testing only**.

Run in insecure mode, skipping all TLS encryption and authentication. If this flag is not set, the `--certs-dir` flag must point to valid certificates.

**Note the following risks:** An insecure cluster is open to any client that can access any node's IP addresses; client connections must also be made insecurely; any user, even `root`, can log in without providing a password; any user, connecting as `root`, can read or write any data in your cluster; there is no network encryption or authentication, and thus no confidentiality.

**Default:** `false` -`--accept-sql-without-tls` | This experimental flag allows you to connect to the cluster using a SQL user's password without [validating the client's certificate](authentication.html#client-authentication). When connecting using the built-in SQL client, [use the `--insecure` flag with the `cockroach sql` command](cockroach-sql.html#client-connection). -`--cert-principal-map` | A comma-separated list of `cert-principal:db-principal` mappings used to map the certificate principals to IP addresses, DNS names, and SQL users. This allows the use of certificates generated by Certificate Authorities that place restrictions on the contents of the `commonName` field. For usage information, see [Create Security Certificates using Openssl]({% link {{ page.version.version }}/create-security-certificates-openssl.md %}#examples). -`--enterprise-encryption` | This optional flag specifies the encryption options for one of the stores on the node. If multiple stores exist, the flag must be specified for each store.

This flag takes a number of options. For a complete list of options, and usage instructions, see [Encryption at Rest](encryption.html).

Note that this is an [Enterprise feature](enterprise-licensing.html). -`--external-io-disable-http` | This optional flag disables external HTTP(S) access (as well as custom HTTP(S) endpoints) when performing bulk operations (e.g, [`BACKUP`](backup.html), [`IMPORT`](import.html), etc.). This can be used in environments where you cannot run a full proxy server.

If you want to run a proxy server, you can start CockroachDB while specifying the `HTTP(S)_PROXY` environment variable. -`--external-io-disable-implicit-credentials` | This optional flag disables the use of implicit credentials when accessing external cloud storage services for bulk operations (e.g, [`BACKUP`](backup.html), [`IMPORT`](import.html), etc.). - -### Locality - -The `--locality` flag accepts arbitrary key-value pairs that describe the location of the node. Locality might include region, country, availability zone, etc. The key-value pairs should be ordered into _locality tiers_ from most inclusive to least inclusive (e.g., region before availability zone as in `region=eu,zone=paris`), and the keys and order of key-value pairs must be the same on all nodes. It's typically better to include more pairs than fewer. - -- Specifying a region with a `region` tier is required in order to enable CockroachDB's [multi-region capabilities](multiregion-overview.html). - -- CockroachDB spreads the replicas of each piece of data across as diverse a set of localities as possible, with the order determining the priority. Locality can also be used to influence the location of data replicas in various ways using [replication zones](configure-replication-zones.html#replication-constraints). - -- When there is high latency between nodes (e.g., cross-availability zone deployments), CockroachDB uses locality to move range leases closer to the current workload, reducing network round trips and improving read performance, also known as ["follow-the-workload"](topology-follow-the-workload.html). In a deployment across more than 3 availability zones, however, to ensure that all data benefits from "follow-the-workload", you must increase your replication factor to match the total number of availability zones. - -- Locality is also a prerequisite for using the [table partitioning](partitioning.html) and [**Node Map**](enable-node-map.html) Enterprise features. - -#### Example - -{% include_cached copy-clipboard.html %} -~~~ shell -# Locality flag for nodes in US East availability zone: ---locality=region=us,zone=us-east - -# Locality flag for nodes in US Central availability zone: ---locality=region=us,zone=us-central - -# Locality flag for nodes in US West availability zone: ---locality=region=us,zone=us-west -~~~ - -### Storage - -#### Storage engine - -The `--storage-engine` flag is used to choose the storage engine used by the node. Note that this setting applies to all [stores](#store) on the node, including the [temp store](#temp-dir). - -{% include_cached new-in.html version="v21.1" %} As of v21.1, CockroachDB always uses the [Pebble storage engine](architecture/storage-layer.html#pebble). As such, `pebble` is the default and only option for the `--storage-engine` flag. - -{{site.data.alerts.callout_danger}} -If you specified `--storage-engine=rocksdb` when starting nodes with v20.2, be sure to change that to `--storage-engine=pebble` or remove the flag entirely before [upgrading to v21.1](upgrade-cockroach-version.html). Pebble is intended to be bi-directionally compatible with the RocksDB on-disk format, so the switch will be seamless during the upgrade process. -{{site.data.alerts.end}} - -#### Store - -The `--store` flag supports the following fields. Note that commas are used to separate fields, and so are forbidden in all field values. - -{{site.data.alerts.callout_info}} -In-memory storage is not suitable for production deployments at this time. -{{site.data.alerts.end}} - -Field | Description -------|------------ -`type` | For in-memory storage, set this field to `mem`; otherwise, leave this field out. The `path` field must not be set when `type=mem`. -`path` | The file path to the storage device. When not setting `attr` or `size`, the `path` field label can be left out:

`--store=/mnt/ssd01`

When either of those fields are set, however, the `path` field label must be used:

`--store=path=/mnt/ssd01,size=20GB`

**Default:** `cockroach-data` -`attrs` | Arbitrary strings, separated by colons, specifying disk type or capability. These can be used to influence the location of data replicas. See [Configure Replication Zones](configure-replication-zones.html#replication-constraints) for full details.

In most cases, node-level `--locality` or `--attrs` are preferable to store-level attributes, but this field can be used to match capabilities for storage of individual databases or tables. For example, an OLTP database would probably want to allocate space for its tables only on solid state devices, whereas append-only time series might prefer cheaper spinning drives. Typical attributes include whether the store is flash (`ssd`) or spinny disk (`hdd`), as well as speeds and other specs, for example:

`--store=path=/mnt/hda1,attrs=hdd:7200rpm` -`size` | The maximum size allocated to the node. When this size is reached, CockroachDB attempts to rebalance data to other nodes with available capacity. When there's no capacity elsewhere, this limit will be exceeded. Also, data may be written to the node faster than the cluster can rebalance it away; in this case, as long as capacity is available elsewhere, CockroachDB will gradually rebalance data down to the store limit.

The `size` can be specified either in a bytes-based unit or as a percentage of hard drive space (notated as a decimal or with `%`), for example:

`--store=path=/mnt/ssd01,size=10000000000 ----> 10000000000 bytes`
`--store=path=/mnt/ssd01,size=20GB ----> 20000000000 bytes`
`--store=path=/mnt/ssd01,size=20GiB ----> 21474836480 bytes`
`--store=path=/mnt/ssd01,size=0.02TiB ----> 21474836480 bytes`
`--store=path=/mnt/ssd01,size=20% ----> 20% of available space`
`--store=path=/mnt/ssd01,size=0.2 ----> 20% of available space`
`--store=path=/mnt/ssd01,size=.2 ----> 20% of available space`

**Default:** 100%

For an in-memory store, the `size` field is required and must be set to the true maximum bytes or percentage of available memory, for example:

`--store=type=mem,size=20GB`
`--store=type=mem,size=90%`

Note: If you use the `%` notation, you might need to escape the `%` sign, for instance, while configuring CockroachDB through `systemd` service files. For this reason, it's recommended to use the decimal notation instead. - -### Logging - -By [default](configure-logs.html#default-logging-configuration), `cockroach start` writes all messages to log files, and prints nothing to `stderr`. This includes events with `INFO` [severity](logging.html#logging-levels-severities) and higher. However, you can [customize the logging behavior](configure-logs.html) of this command by using the `--log` flag: - -{% include {{ page.version.version }}/misc/logging-flags.md %} - -#### Defaults - -See the [default logging configuration](configure-logs.html#default-logging-configuration). - -## Standard output - -When you run `cockroach start`, some helpful details are printed to the standard output: - -~~~ shell -CockroachDB node starting at {{ now | date: "%Y-%m-%d %H:%M:%S.%6 +0000 UTC" }} -build: CCL {{page.release_info.version}} @ {{page.release_info.build_time}} (go1.12.6) -webui: http://localhost:8080 -sql: postgresql://root@localhost:26257?sslmode=disable -RPC client flags: cockroach --host=localhost:26257 --insecure -logs: /Users//node1/logs -temp dir: /Users//node1/cockroach-temp242232154 -external I/O path: /Users//node1/extern -store[0]: path=/Users//node1 -status: initialized new cluster -clusterID: 8a681a16-9623-4fc1-a537-77e9255daafd -nodeID: 1 -~~~ - -{{site.data.alerts.callout_success}} -These details are also written to the `INFO` log in the `/logs` directory. You can retrieve them with a command like `grep 'node starting' node1/logs/cockroach.log -A 11`. -{{site.data.alerts.end}} - -Field | Description -------|------------ -`build` | The version of CockroachDB you are running. -`webui` | The URL for accessing the DB Console. -`sql` | The connection URL for your client. -`RPC client flags` | The flags to use when connecting to the node via [`cockroach` client commands](cockroach-commands.html). -`logs` | The directory containing debug log data. -`temp dir` | The temporary store directory of the node. -`external I/O path` | The external IO directory with which the local file access paths are prefixed while performing [backup](backup.html) and [restore](restore.html) operations using local node directories or NFS drives. -`attrs` | If node-level attributes were specified in the `--attrs` flag, they are listed in this field. These details are potentially useful for [configuring replication zones](configure-replication-zones.html). -`locality` | If values describing the locality of the node were specified in the `--locality` field, they are listed in this field. These details are potentially useful for [configuring replication zones](configure-replication-zones.html). -`store[n]` | The directory containing store data, where `[n]` is the index of the store, e.g., `store[0]` for the first store, `store[1]` for the second store.

If store-level attributes were specified in the `attrs` field of the [`--store`](#store) flag, they are listed in this field as well. These details are potentially useful for [configuring replication zones](configure-replication-zones.html). -`status` | Whether the node is the first in the cluster (`initialized new cluster`), joined an existing cluster for the first time (`initialized new node, joined pre-existing cluster`), or rejoined an existing cluster (`restarted pre-existing node`). -`clusterID` | The ID of the cluster.

When trying to join a node to an existing cluster, if this ID is different than the ID of the existing cluster, the node has started a new cluster. This may be due to conflicting information in the node's data directory. For additional guidance, see the [troubleshooting](common-errors.html#node-belongs-to-cluster-cluster-id-but-is-attempting-to-connect-to-a-gossip-network-for-cluster-another-cluster-id) docs. -`nodeID` | The ID of the node. - -## Examples - -### Start a multi-node cluster - -
- - -
- -To start a multi-node cluster, run the `cockroach start` command for each node, setting the `--join` flag to the addresses of the initial nodes. - -{% include {{ page.version.version }}/prod-deployment/join-flag-single-region.md %} - -{{site.data.alerts.callout_info}} -{% include {{ page.version.version }}/prod-deployment/join-flag-multi-region.md %} -{{site.data.alerts.end}} - -
- -{{site.data.alerts.callout_success}} -Before starting the cluster, use [`cockroach cert`](cockroach-cert.html) to generate node and client certificates for a secure cluster connection. -{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start \ ---certs-dir=certs \ ---advertise-addr= \ ---join=,, \ ---cache=.25 \ ---max-sql-memory=.25 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start \ ---certs-dir=certs \ ---advertise-addr= \ ---join=,, \ ---cache=.25 \ ---max-sql-memory=.25 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start \ ---certs-dir=certs \ ---advertise-addr= \ ---join=,, \ ---cache=.25 \ ---max-sql-memory=.25 -~~~ - -
- -
- -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start \ ---insecure \ ---advertise-addr= \ ---join=,, \ ---cache=.25 \ ---max-sql-memory=.25 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start \ ---insecure \ ---advertise-addr= \ ---join=,, \ ---cache=.25 \ ---max-sql-memory=.25 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start \ ---insecure \ ---advertise-addr= \ ---join=,, \ ---cache=.25 \ ---max-sql-memory=.25 -~~~ - -
- -Then run the [`cockroach init`](cockroach-init.html) command against any node to perform a one-time cluster initialization: - -
- -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach init \ ---certs-dir=certs \ ---host=
-~~~ - -
- -
- -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach init \ ---insecure \ ---host=
-~~~ - -
- -### Start a multi-node cluster across private networks - -**Scenario:** - -- You have a cluster that spans GCE and AWS. -- The nodes on each cloud can reach each other on private addresses, but the private addresses aren't reachable from the other cloud. - -**Approach:** - -1. Start each node on GCE with `--locality` set to describe its location, `--locality-advertise-addr` set to advertise its private address to other nodes in on GCE, `--advertise-addr` set to advertise its public address to nodes on AWS, and `--join` set to the public addresses of 3-5 of the initial nodes: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --certs-dir=certs \ - --locality=cloud=gce \ - --locality-advertise-addr=cloud=gce@ \ - --advertise-addr= \ - --join=,, \ - --cache=.25 \ - --max-sql-memory=.25 - ~~~ - -2. Start each node on AWS with `--locality` set to describe its location, `--locality-advertise-addr` set to advertise its private address to other nodes on AWS, `--advertise-addr` set to advertise its public address to nodes on GCE, and `--join` set to the public addresses of 3-5 of the initial nodes: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --certs-dir=certs \ - --locality=cloud=aws \ - --locality-advertise-addr=cloud=aws@ \ - --advertise-addr= \ - --join=,, \ - --cache=.25 \ - --max-sql-memory=.25 - ~~~ - -3. Run the [`cockroach init`](cockroach-init.html) command against any node to perform a one-time cluster initialization: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach init \ - --certs-dir=certs \ - --host=
- ~~~ - -### Add a node to a cluster - -
- - -
- -To add a node to an existing cluster, run the `cockroach start` command, setting the `--join` flag to the same addresses you used when [starting the cluster](#start-a-multi-node-cluster): - -
- -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start \ ---certs-dir=certs \ ---advertise-addr= \ ---join=,, \ ---cache=.25 \ ---max-sql-memory=.25 -~~~ - -
- -
- -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start \ ---insecure \ ---advertise-addr= \ ---join=,, \ ---cache=.25 \ ---max-sql-memory=.25 -~~~ - -
- -### Create a table with node locality information - -Start a three-node cluster with locality information specified in the `cockroach start` commands: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start --insecure --port=26257 --http-port=26258 --store=cockroach-data/1 --cache=256MiB --locality=region=eu-west-1,cloud=aws,zone=eu-west-1a -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start --insecure --port=26259 --http-port=26260 --store=cockroach-data/2 --cache=256MiB --join=localhost:26257 --locality=region=eu-west-1,cloud=aws,zone=eu-west-1b -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start --insecure --port=26261 --http-port=26262 --store=cockroach-data/3 --cache=256MiB --join=localhost:26257 --locality=region=eu-west-1,cloud=aws,zone=eu-west-1c -~~~ - -You can use the [`crdb_internal.locality_value`](functions-and-operators.html#system-info-functions) built-in function to return the current node's locality information from inside a SQL shell. The example below uses the output of `crdb_internal.locality_value('zone')` as the `DEFAULT` value to use for the `zone` column of new rows. Other available locality keys for the running three-node cluster include `region` and `cloud`. - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --insecure -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE charges ( - zone STRING NOT NULL DEFAULT crdb_internal.locality_value('zone'), - id INT PRIMARY KEY NOT NULL -); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO charges (id) VALUES (1); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM charges WHERE id = 1; -~~~ - -~~~ - zone | id -+------------+----+ - eu-west-1a | 1 -(1 row) -~~~ - -The `zone ` column has the zone of the node on which the row was created. - -In a separate terminal window, open a SQL shell to a different node on the cluster: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --insecure --port 26259 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO charges (id) VALUES (2); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM charges WHERE id = 2; -~~~ - -~~~ - zone | id -+------------+----+ - eu-west-1b | 2 -(1 row) -~~~ - -In a separate terminal window, open a SQL shell to the third node: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --insecure --port 26261 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO charges (id) VALUES (3); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM charges WHERE id = 3; -~~~ - -~~~ - zone | id -+------------+----+ - eu-west-1c | 3 -(1 row) -~~~ - -### Start a cluster with separate RPC and SQL networks - -Separating the network addresses used for intra-cluster RPC traffic and application SQL connections can provide an additional level of protection against security issues as a form of [defense in depth](https://en.wikipedia.org/wiki/Defense_in_depth_(computing)). This separation is accomplished with a combination of the [`--sql-addr` flag](#networking) and firewall rules or other network-level access control (which must be maintained outside of CockroachDB). - -For example, suppose you want to use port `26257` for SQL connections and `26258` for intra-cluster traffic. Set up firewall rules so that the CockroachDB nodes can reach each other on port `26258`, but other machines cannot. Start the CockroachDB processes as follows: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start --sql-addr=:26257 --listen-addr=:26258 --join=node1:26258,node2:26258,node3:26258 --certs-dir=~/cockroach-certs -~~~ - -Note the use of port `26258` (the value for `listen-addr`, not `sql-addr`) in the `--join` flag. Also, if your environment requires the use of the `--advertise-addr` flag, you should probably also use the `--advertise-sql-addr` flag when using a separate SQL address. - -Clusters using this configuration with client certificate authentication may also wish to use [split client CA certificates]({% link {{ page.version.version }}/create-security-certificates-custom-ca.md %}#split-ca-certificates). - -## See also - -- [Initialize a Cluster]({% link {{ page.version.version }}/cockroach-init.md %}) -- [Manual Deployment]({% link {{ page.version.version }}/manual-deployment.md %}) -- [Orchestrated Deployment]({% link {{ page.version.version }}/orchestration.md %}) -- [Local Deployment]({% link {{ page.version.version }}/start-a-local-cluster.md %}) -- [Other Cockroach Commands]({% link {{ page.version.version }}/cockroach-commands.md %}) diff --git a/src/current/v21.1/cockroach-statement-diag.md b/src/current/v21.1/cockroach-statement-diag.md deleted file mode 100644 index cdf991e7d8c..00000000000 --- a/src/current/v21.1/cockroach-statement-diag.md +++ /dev/null @@ -1,147 +0,0 @@ ---- -title: cockroach statement-diag -summary: Use statement-diag to manage and download statement diagnostics bundles. -toc: true ---- - - The `cockroach statement-diag` [command](cockroach-commands.html) can be used to manage and download statement diagnostics bundles generated from the [DB Console](ui-statements-page.html#diagnostics) or [`EXPLAIN ANALYZE (DEBUG)`](explain-analyze.html#explain-analyze-debug). - -## Required privileges - -Only members of the `admin` role can run `cockroach statement-diag`. By default, the `root` user belongs to the `admin` role. - -## Subcommands - -Subcommand | Usage ------------|------ -`list` | List available statement diagnostics bundles and outstanding activation requests. -`download` | Download a specified diagnostics bundle into a `.zip` file. -`delete` | Delete a statement diagnostics bundle(s). -`cancel` | Cancel an outstanding activation request(s). - -## Synopsis - -List available statement diagnostics bundles and outstanding activation requests: - -~~~ shell -$ cockroach statement-diag list -~~~ - -Download a specified diagnostics bundle into a `.zip` file: - -~~~ shell -$ cockroach statement-diag download -~~~ - -Delete a statement diagnostics bundle: - -~~~ shell -$ cockroach statement-diag delete -~~~ - -Delete all statement diagnostics bundles: - -~~~ shell -$ cockroach statement-diag delete --all -~~~ - -Cancel an outstanding activation request: - -~~~ shell -$ cockroach statement-diag cancel -~~~ - -Cancel all outstanding activation requests: - -~~~ shell -$ cockroach statement-diag cancel --all -~~~ - -## Flags - -- The `delete` and `cancel` subcommands support one [general-use](#general) flag. -- All `statement-diag` subcommands support several [client connection](#client-connection) and [logging](#logging) flags. - -### General - -Flag | Description ------|------------ -`--all` | Apply to all bundles or activation requests. - -### Client connection - -{% include {{ page.version.version }}/sql/connection-parameters.md %} - -See [Client Connection Parameters](connection-parameters.html) for more details. - -### Logging - -{% include {{ page.version.version }}/misc/logging-defaults.md %} - -## Examples - -### Setup - -These examples assume you are running [an insecure cluster](start-a-local-cluster.html) and have requested and/or generated statement diagnostics bundles using the [DB Console](ui-statements-page.html#diagnostics) or [`EXPLAIN ANALYZE (DEBUG)`](explain-analyze.html#explain-analyze-debug). - -### Download a statement diagnostics bundle - -List statement diagnostics bundles and/or activation requests: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach statement-diag list --insecure -~~~ - -~~~ -Statement diagnostics bundles: - ID Collection time Statement - 603820372518502401 2020-11-02 18:29:13 UTC CREATE DATABASE bank - -Outstanding activation requests: - ID Activation time Statement - 603811900498804737 2020-11-02 17:46:08 UTC SELECT * FROM bank.accounts -~~~ - -Download a statement diagnostics bundle to `bundle.zip`: - -~~~ shell -$ cockroach statement-diag download 603820372518502401 bundle.zip --insecure -~~~ - -### Delete all statement diagnostics bundles - -Delete all statement diagnostics bundles: - -~~~ shell -$ cockroach statement-diag delete --all --insecure -~~~ - -### Cancel an activation request - -List statement diagnostics bundles and/or activation requests: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach statement-diag list --insecure -~~~ - -~~~ -Outstanding activation requests: - ID Activation time Statement - 603811900498804737 2020-11-02 17:46:08 UTC SELECT * FROM bank.accounts -~~~ - -Delete an activation request: - -~~~ shell -$ cockroach statement-diag cancel 603811900498804737 --insecure -~~~ - -## See also - -- [DB Console Statement Details page](ui-statements-page.html#statement-details-page) -- [`EXPLAIN ANALYZE (DEBUG)`](explain-analyze.html#explain-analyze-debug) -- [SQL Statements](sql-statements.html) -- [Learn CockroachDB SQL](learn-cockroachdb-sql.html) -- [Other Cockroach Commands](cockroach-commands.html) diff --git a/src/current/v21.1/cockroach-userfile-delete.md b/src/current/v21.1/cockroach-userfile-delete.md deleted file mode 100644 index 8cd83394b17..00000000000 --- a/src/current/v21.1/cockroach-userfile-delete.md +++ /dev/null @@ -1,109 +0,0 @@ ---- -title: cockroach userfile delete -summary: The cockroach userfile delete command deletes files stored in user-scoped file storage. -toc: true ---- - - The `cockroach userfile delete` [command](cockroach-commands.html) deletes the files stored in the [user-scoped file storage](use-userfile-for-bulk-operations.html) which match the [provided pattern](cockroach-userfile-upload.html#file-destination), using a SQL connection. If the pattern `'*'` is passed, all files in the specified (or default, if unspecified) user-scoped file storage will be deleted. Deletions are not atomic, and all deletions prior to the first failure will occur. - -## Required privileges - -The user must have the `CREATE` [privilege](authorization.html#assign-privileges) on the target database. CockroachDB will proactively grant the user `GRANT`, `SELECT`, `INSERT`, `DROP`, `DELETE` on the metadata and file tables. - -A user can only delete files from their own user-scoped storage, which is accessed through the [userfile URI](cockroach-userfile-upload.html#file-destination) used during the upload. CockroachDB will revoke all access from every other user in the cluster except users in the `admin` role. Users in the `admin` role can delete from any user's storage. - -## Synopsis - -Delete a file: - -~~~ shell -$ cockroach userfile delete [flags] -~~~ - -View help: - -~~~ shell -$ cockroach userfile delete --help -~~~ - -## Flags - - Flag | Description ------------------+----------------------------------------------------- -`--cert-principal-map` | A comma-separated list of `:` mappings. This allows mapping the principal in a cert to a DB principal such as `node` or `root` or any SQL user. This is intended for use in situations where the certificate management system places restrictions on the `Subject.CommonName` or `SubjectAlternateName` fields in the certificate (e.g., disallowing a `CommonName` like `node` or `root`). If multiple mappings are provided for the same ``, the last one specified in the list takes precedence. A principal not specified in the map is passed through as-is via the identity function. A cert is allowed to authenticate a DB principal if the DB principal name is contained in the mapped `CommonName` or DNS-type `SubjectAlternateName` fields. -`--certs-dir` | The path to the [certificate directory](cockroach-cert.html) containing the CA and client certificates and client key.

**Env Variable:** `COCKROACH_CERTS_DIR`
**Default:** `${HOME}/.cockroach-certs/` -`--echo-sql` | Reveal the SQL statements sent implicitly by the command-line utility. -`--url` | A [connection URL](connection-parameters.html#connect-using-a-url) to use instead of the other arguments.

**Env Variable:** `COCKROACH_URL`
**Default:** no URL -`--user`
`-u` | The [SQL user](create-user.html) that will own the client session.

**Env Variable:** `COCKROACH_USER`
**Default:** `root` - -## Examples - -### Delete all files in the default storage - -To delete all files in the directory, pass the `'*'` pattern: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach userfile delete '*' --certs-dir=certs -~~~ - -~~~ -deleted userfile://defaultdb.public.userfiles_root/test-data-2.csv -deleted userfile://defaultdb.public.userfiles_root/test-data.csv -deleted userfile://defaultdb.public.userfiles_root/test-upload/test-data.csv -~~~ - -Note that because a fully qualified userfile URI was not specified, files in the default user-scoped storage (`userfile://defaultdb.public.userfiles_$user/`) were deleted. - -### Delete a specific file - -To delete a specific file, include the file destination in the command: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach userfile delete test-data.csv --certs-dir=certs -~~~ - -~~~ -deleted userfile://defaultdb.public.userfiles_root/test-data.csv -~~~ - -### Delete files that match the provided pattern - -To delete all files that match a pattern, use `*`: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach userfile delete '*.csv' --certs-dir=certs -~~~ - -~~~ -deleted userfile://defaultdb.public.userfiles_root/test-data-2.csv -deleted userfile://defaultdb.public.userfiles_root/test-data.csv -~~~ - -### Delete files from a non-default userfile URI - -If you [uploaded a file to a non-default userfile URI](cockroach-userfile-upload.html#upload-a-file-to-a-non-default-userfile-uri) (e.g., `userfile://testdb.public.uploads`): - -{% include_cached copy-clipboard.html %} -~~~ shell -cockroach userfile upload /Users/maxroach/Desktop/test-data.csv userfile://testdb.public.uploads/test-data.csv -~~~ - -Use the same URI to delete it: - -{% include_cached copy-clipboard.html %} -~~~ shell -cockroach userfile delete userfile://testdb.public.uploads -~~~ - -## See also - -- [`cockroach userfile upload`](cockroach-userfile-upload.html) -- [`cockroach userfile list`](cockroach-userfile-list.html) -- [`cockroach userfile get`](cockroach-userfile-get.html) -- [Use `userfile` for Bulk Operations](use-userfile-for-bulk-operations.html) -- [Other Cockroach Commands](cockroach-commands.html) -- [`IMPORT`](import.html) -- [`IMPORT INTO`](import-into.html) diff --git a/src/current/v21.1/cockroach-userfile-get.md b/src/current/v21.1/cockroach-userfile-get.md deleted file mode 100644 index 86ec701970a..00000000000 --- a/src/current/v21.1/cockroach-userfile-get.md +++ /dev/null @@ -1,87 +0,0 @@ ---- -title: cockroach userfile get -summary: Fetch files stored in the user-scoped file storage. -toc: true ---- - -{% include_cached new-in.html version="v21.1" %} The `cockroach userfile get` [command](cockroach-commands.html) fetches the files stored in the [user-scoped file storage](use-userfile-for-bulk-operations.html) which match the provided pattern, using a SQL connection. If no pattern is provided, all files in the specified (or default, if unspecified) user-scoped file storage will be fetched. - -## Required privileges - -The user must have `SELECT` [privileges](authorization.html#assign-privileges) on the target database. - -A user can only fetch files from their own user-scoped storage, which is accessed through the [userfile URI](cockroach-userfile-upload.html#file-destination) used during the upload. CockroachDB will revoke all access from every other user in the cluster except users in the `admin` role and users explicitly granted access. - -{{site.data.alerts.callout_info}} -If this is your first interaction with user-scoped file storage, you may see an error indicating that you need `CREATE` privileges on the database. You must first [upload a file](cockroach-userfile-upload.html) or run a [`BACKUP`](backup.html) to `userfile` before attempting to `get` a file. -{{site.data.alerts.end}} - -## Synopsis - -Fetch a file: - -~~~ shell -$ cockroach userfile get [flags] -~~~ - -View help: - -~~~ shell -$ cockroach userfile get --help -~~~ - -## Flags - - Flag | Description ------------------+----------------------------------------------------- -`--cert-principal-map` | A comma-separated list of `:` mappings. This allows mapping the principal in a cert to a DB principal such as `node` or `root` or any SQL user. This is intended for use in situations where the certificate management system places restrictions on the `Subject.CommonName` or `SubjectAlternateName` fields in the certificate (e.g., disallowing a `CommonName` like `node` or `root`). If multiple mappings are provided for the same ``, the last one specified in the list takes precedence. A principal not specified in the map is passed through as-is via the identity function. A cert is allowed to authenticate a DB principal if the DB principal name is contained in the mapped `CommonName` or DNS-type `SubjectAlternateName` fields. -`--certs-dir` | The path to the [certificate directory](cockroach-cert.html) containing the CA and client certificates and client key.

**Env Variable:** `COCKROACH_CERTS_DIR`
**Default:** `${HOME}/.cockroach-certs/` -`--echo-sql` | Reveal the SQL statements sent implicitly by the command-line utility. -`--url` | A [connection URL](connection-parameters.html#connect-using-a-url) to use instead of the other arguments.

**Env Variable:** `COCKROACH_URL`
**Default:** no URL -`--user`
`-u` | The [SQL user](create-user.html) that will own the client session.

**Env Variable:** `COCKROACH_USER`
**Default:** `root` - -## Examples - -### Get a specific file - -To get the file named test-data.csv from the default user-scoped storage location for the current user: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach userfile get test-data.csv --certs-dir=certs -~~~ - -### Get a file saved to an explicit local file name - -To get a file named test-data.csv from a local directory: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach userfile get test-data.csv /Users/maxroach/Desktop/test-data.csv --certs-dir=certs -~~~ - -### Get a file from a non-default userfile URI - -If you [uploaded a file to a non-default userfile URI](cockroach-userfile-upload.html#upload-a-file-to-a-non-default-userfile-uri) (e.g., `userfile://testdb.public.uploads`), use the same URI to fetch it: - -{% include_cached copy-clipboard.html %} -~~~ shell -cockroach userfile get userfile://testdb.public.uploads/test-data.csv --certs-dir=certs -~~~ - -### Get files that match the provided pattern - -To get all files that match a pattern, use *: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach userfile get '*.csv' --certs-dir=certs -~~~ - -## See also - -- [`cockroach userfile upload`](cockroach-userfile-upload.html) -- [`cockroach userfile delete`](cockroach-userfile-delete.html) -- [`cockroach userfile list`](cockroach-userfile-list.html) -- [Use `userfile` for Bulk Operations](use-userfile-for-bulk-operations.html) -- [Other Cockroach Commands](cockroach-commands.html) diff --git a/src/current/v21.1/cockroach-userfile-list.md b/src/current/v21.1/cockroach-userfile-list.md deleted file mode 100644 index 69be050a2dc..00000000000 --- a/src/current/v21.1/cockroach-userfile-list.md +++ /dev/null @@ -1,107 +0,0 @@ ---- -title: cockroach userfile list -summary: List the files stored in the user-scoped file storage. -toc: true ---- - - The `cockroach userfile list` [command](cockroach-commands.html) lists the files stored in the [user-scoped file storage](use-userfile-for-bulk-operations.html) which match the [provided pattern](cockroach-userfile-upload.html#file-destination), using a SQL connection. If no pattern is provided, all files in the specified (or default, if unspecified) user scoped file storage will be listed. - -## Required privileges - -The user must have the `CREATE` [privilege](authorization.html#assign-privileges) on the target database. CockroachDB will proactively grant the user `GRANT`, `SELECT`, `INSERT`, `DROP`, `DELETE` on the metadata and file tables. - -A user can only view files in their own user-scoped storage, which is accessed through the [userfile URI](cockroach-userfile-upload.html#file-destination) used during the upload. CockroachDB will revoke all access from every other user in the cluster except users in the `admin` role. - -## Synopsis - -View files: - -~~~ shell -$ cockroach userfile list [flags] -~~~ - -View help: - -~~~ shell -$ cockroach userfile list --help -~~~ - -## Flags - - Flag | Description ------------------+----------------------------------------------------- -`--cert-principal-map` | A comma-separated list of `:` mappings. This allows mapping the principal in a cert to a DB principal such as `node` or `root` or any SQL user. This is intended for use in situations where the certificate management system places restrictions on the `Subject.CommonName` or `SubjectAlternateName` fields in the certificate (e.g., disallowing a `CommonName` like `node` or `root`). If multiple mappings are provided for the same ``, the last one specified in the list takes precedence. A principal not specified in the map is passed through as-is via the identity function. A cert is allowed to authenticate a DB principal if the DB principal name is contained in the mapped `CommonName` or DNS-type `SubjectAlternateName` fields. -`--certs-dir` | The path to the [certificate directory](cockroach-cert.html) containing the CA and client certificates and client key.

**Env Variable:** `COCKROACH_CERTS_DIR`
**Default:** `${HOME}/.cockroach-certs/` -`--echo-sql` | Reveal the SQL statements sent implicitly by the command-line utility. -`--url` | A [connection URL](connection-parameters.html#connect-using-a-url) to use instead of the other arguments.

**Env Variable:** `COCKROACH_URL`
**Default:** no URL -`--user`
`-u` | The [SQL user](create-user.html) that will own the client session.

**Env Variable:** `COCKROACH_USER`
**Default:** `root` - -## Examples - -### List all files in the default storage - -If the file or directory is not specified, all files in the default user-scoped storage (`userfile://defaultdb.public.userfiles_$user/`) will be listed: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach userfile list --certs-dir=certs -~~~ - -~~~ -userfile://defaultdb.public.userfiles_root/test-data-2.csv -userfile://defaultdb.public.userfiles_root/test-data.csv -userfile://defaultdb.public.userfiles_root/test-upload/test-data.csv -~~~ - -### List a specific file - -To list all files in a specified directory: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach userfile list test-data.csv --certs-dir=certs -~~~ - -~~~ -userfile://defaultdb.public.userfiles_root/test-data.csv -~~~ - -### List files that match the provided pattern - -To list all files that match a pattern, use `*`: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach userfile list '*.csv' --certs-dir=certs -~~~ - -~~~ -userfile://defaultdb.public.userfiles_root/test-data-2.csv -userfile://defaultdb.public.userfiles_root/test-data.csv -~~~ - -### List files from a non-default userfile URI - -If you [uploaded a file to a non-default userfile URI](cockroach-userfile-upload.html#upload-a-file-to-a-non-default-userfile-uri) (e.g., `userfile://testdb.public.uploads`): - -{% include_cached copy-clipboard.html %} -~~~ shell -cockroach userfile upload /Users/maxroach/Desktop/test-data.csv userfile://testdb.public.uploads/test-data.csv -~~~ - -Use the same URI to view it: - -{% include_cached copy-clipboard.html %} -~~~ shell -cockroach userfile list userfile://testdb.public.uploads -~~~ - -## See also - -- [`cockroach userfile upload`](cockroach-userfile-upload.html) -- [`cockroach userfile delete`](cockroach-userfile-delete.html) -- [`cockroach userfile get`](cockroach-userfile-get.html) -- [Use `userfile` for Bulk Operations](use-userfile-for-bulk-operations.html) -- [Other Cockroach Commands](cockroach-commands.html) -- [`IMPORT`](import.html) -- [`IMPORT INTO`](import-into.html) diff --git a/src/current/v21.1/cockroach-userfile-upload.md b/src/current/v21.1/cockroach-userfile-upload.md deleted file mode 100644 index 14fe93e1ec3..00000000000 --- a/src/current/v21.1/cockroach-userfile-upload.md +++ /dev/null @@ -1,261 +0,0 @@ ---- -title: cockroach userfile upload -summary: The cockroach userfile upload command uploads a file to user-scoped file storage. -toc: true ---- - - The `cockroach userfile upload` [command](cockroach-commands.html) uploads a file to the [user-scoped file storage](use-userfile-for-bulk-operations.html) using a SQL connection. - -This command takes in a source file to upload and a destination filename. It will then use a SQL connection to upload the file to the [destination](#file-destination). - -{{site.data.alerts.callout_info}} -A userfile uses storage space in the cluster, and is replicated with the rest of the cluster's data. We recommended using `cockroach userfile upload` for quick uploads from your client (about 15MB or smaller). -{{site.data.alerts.end}} - -{{site.data.alerts.callout_success}} -If you would like to upload and import data from a dump file, consider using [`cockroach import`](cockroach-import.html) instead. -{{site.data.alerts.end}} - -## Required privileges - -The user must have the `CREATE` [privilege](authorization.html#assign-privileges) on the target database. CockroachDB will proactively grant the user `GRANT`, `SELECT`, `INSERT`, `DROP`, `DELETE` on the metadata and file tables. - -A user can only upload files to their own user-scoped storage, which is accessed through the [userfile URI](#file-destination). CockroachDB will revoke all access from every other user in the cluster except users in the `admin` role. - -## Synopsis - -Upload a file: - -~~~ shell -$ cockroach userfile upload [flags] -~~~ - -{{site.data.alerts.callout_info}} -You must specify a source path. -{{site.data.alerts.end}} - -View help: - -~~~ shell -$ cockroach userfile upload --help -~~~ - -## File destination - -Userfile operations are backed by two tables: `files` (which holds file metadata) and `payload` (which holds the file payloads). To reference these tables, you can: - -- Use the default URI: `userfile://defaultdb.public.userfiles_$user/`. -- Provide a fully qualified userfile URI that specifies the database, schema, and table name prefix you want to use. - - - If you do not specify a destination URI/path, then CockroachDB will use the default URI scheme and host, and the basename from the source argument as the path. For example: `userfile://defaultdb.public.userfiles_root/local` - - If the destination is a well-formed userfile URI (i.e., `userfile://db.schema.tablename_prefix/path/to/file`), then CockroachDB will use that as the final URI. For example: `userfile://foo.bar.baz_root/destination/path` - - If destination is not a well-formed userfile URI, then CockroachDB will use the default userfile URI schema and host (`userfile://defaultdb.public.userfiles_$user/`), and the destination as the path. For example: `userfile://defaultdb.public.userfiles_root/destination/path` - - - -{{site.data.alerts.callout_danger}} -Userfile is **not** a filesystem and does not support filesystem semantics. The destination file path must be the same after normalization (i.e., if you pass any path that results in a different path after normalization, it will be rejected). -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -Files are uploaded with a `.tmp` suffix and are renamed once the userfile upload transaction has committed (i.e, the process ends gracefully). Therefore, if a file you believed had finished uploading has a `.tmp` suffix, then the upload should be retried. -{{site.data.alerts.end}} - -## Flags - - Flag | Description ------------------+----------------------------------------------------- -`--cert-principal-map` | A comma-separated list of `:` mappings. This allows mapping the principal in a cert to a DB principal such as `node` or `root` or any SQL user. This is intended for use in situations where the certificate management system places restrictions on the `Subject.CommonName` or `SubjectAlternateName` fields in the certificate (e.g., disallowing a `CommonName` like `node` or `root`). If multiple mappings are provided for the same ``, the last one specified in the list takes precedence. A principal not specified in the map is passed through as-is via the identity function. A cert is allowed to authenticate a DB principal if the DB principal name is contained in the mapped `CommonName` or DNS-type `SubjectAlternateName` fields. -`--certs-dir` | The path to the [certificate directory](cockroach-cert.html) containing the CA and client certificates and client key.

**Env Variable:** `COCKROACH_CERTS_DIR`
**Default:** `${HOME}/.cockroach-certs/` -`--echo-sql` | Reveal the SQL statements sent implicitly by the command-line utility. -`--url` | A [connection URL](connection-parameters.html#connect-using-a-url) to use instead of the other arguments.

**Env Variable:** `COCKROACH_URL`
**Default:** no URL -`--user`
`-u` | The [SQL user](create-user.html) that will own the client session.

**Env Variable:** `COCKROACH_USER`
**Default:** `root` - -## Examples - -### Upload a file - -To upload a file to the default storage (`userfile://defaultdb.public.userfiles_$user/`): - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach userfile upload /Users/maxroach/Desktop/test-data.csv /test-data.csv --certs-dir=certs -~~~ - -~~~ -successfully uploaded to userfile://defaultdb.public.userfiles_root/test-data.csv -~~~ - -Also, a file can be uploaded to the default storage if the destination is not specified: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach userfile upload /Users/maxroach/Desktop/test-data2.csv --certs-dir=certs -~~~ - -~~~ -successfully uploaded to userfile://defaultdb.public.userfiles_root/test-data2.csv -~~~ - -Then, you can use the file to [`IMPORT`](import.html) or [`IMPORT INTO`](import-into.html) data. - -### Upload a file to a specific directory - -To upload a file to a specific destination, include the destination in the command: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach userfile upload /Users/maxroach/Desktop/test-data.csv /test-upload/test-data.csv --cert-dir=certs -~~~ - -~~~ -successfully uploaded to userfile://defaultdb.public.userfiles_root/test-upload/test-data.csv -~~~ - -Then, you can use the file to [`IMPORT`](import.html) or [`IMPORT INTO`](import-into.html) data. - -### Upload a file to a non-default userfile URI - -{% include_cached copy-clipboard.html %} -~~~ shell -cockroach userfile upload /Users/maxroach/Desktop/test-data.csv userfile://testdb.public.uploads/test-data.csv -~~~ - -~~~ -successfully uploaded to userfile://testdb.public.uploads/test-data.csv -~~~ - -### Upload a backup directory recursively - -Currently in v21.1 and prior, it's [not possible to use `cockroach userfile upload` recursively](#known-limitation) when uploading a directory of files. When you need to upload a directory (such as a backup) to `userfile` storage, it is possible to programmatically upload your files. - -The following example uses `userfile` storage and its features to backup a database and then restore it to a cluster. This workflow provides a way to recover backup data when using `userfile` as your storage option. - -{{site.data.alerts.callout_info}} -Only database and table-level backups are possible when using `userfile` as storage. Restoring [cluster-level backups](backup.html#backup-a-cluster) will not work because `userfile` data is stored in the `defaultdb` database, and you cannot restore a cluster with existing table data. -{{site.data.alerts.end}} - -After connecting to the cluster that contains the data you would like to backup, run the [`BACKUP`](backup.html) statement: - -~~~sql -BACKUP DATABASE movr TO 'userfile://defaultdb.public.userfiles_$user/database-backup' AS OF SYSTEM TIME '-10s'; -~~~ - -This will make a backup of the database under `database-backup` in the default `userfile` space. - -~~~ -job_id | status | fraction_completed | rows | index_entries | bytes --------------------+-----------+--------------------+------+---------------+--------- -679645548255936513 | succeeded | 1 | 2997 | 1060 | 492713 -(1 row) -~~~ - -Next, on the command line, use the following [`cockroach userfile get`](cockroach-userfile-get.html) command to fetch the files stored in the `userfile` storage and download them to your local directory: - -~~~shell -cockroach userfile get 'userfile://defaultdb.public.userfiles_$user/database-backup' --url {CONNECTION STRING} -~~~ - -Note that without passing a specific directory this command will fetch all files stored within the default user space in `userfile`. - -The output will show the files downloading: - -~~~ -downloaded database-backup/BACKUP-CHECKPOINT-679645548255936513-CHECKSUM to database-backup/BACKUP-CHECKPOINT-679645548255936513-CHECKSUM (4 B) -downloaded database-backup/BACKUP-CHECKPOINT-CHECKSUM to database-backup/BACKUP-CHECKPOINT-CHECKSUM (4 B) -downloaded database-backup/BACKUP-STATISTICS to database-backup/BACKUP-STATISTICS (53 KiB) -downloaded database-backup/BACKUP_MANIFEST to database-backup/BACKUP_MANIFEST (2.6 KiB) -downloaded database-backup/BACKUP_MANIFEST-CHECKSUM to database-backup/BACKUP_MANIFEST-CHECKSUM (4 B) -downloaded database-backup/data/679645557047099393.sst to database-backup/data/679645557047099393.sst (5.8 KiB) -. . . -downloaded database-backup/data/679645558404448257.sst to database-backup/data/679645558404448257.sst (5.8 KiB) -downloaded database-backup/data/679645558408249345.sst to database-backup/data/679645558408249345.sst (41 KiB) -~~~ - -At this point, you have two copies of the data: - -* One in the database's `userfile` storage -* One on your local machine - -Your backup will contain files similar to the following structure: - -~~~ -database-backup/ -├── BACKUP-CHECKPOINT-679645548255936513-CHECKSUM -├── BACKUP-CHECKPOINT-CHECKSUM -├── BACKUP-STATISTICS -├── BACKUP_MANIFEST -├── BACKUP_MANIFEST-CHECKSUM -└── data - ├── 679645557047099393.sst - ├── 679645557048475649.sst - ├── 679645557048868865.sst - ├── 679645558151741442.sst -. . . -~~~ - -Use [`DROP`](drop-database.html) to remove the `movr` database: - -~~~sql -DROP DATABASE movr CASCADE; -~~~ - -Next, use [`cockroach userfile delete`](cockroach-userfile-delete.html) to remove this from `userfile`: - -~~~shell -cockroach userfile delete 'userfile://defaultdb.public.userfiles_$user/database-backup' --url 'postgresql://root@localhost:26257?sslmode=disable' -~~~ - -~~~ -successfully deleted database-backup/BACKUP-CHECKPOINT-679645548255936513-CHECKSUM -successfully deleted database-backup/BACKUP-CHECKPOINT-CHECKSUM -successfully deleted database-backup/BACKUP-STATISTICS -successfully deleted database-backup/BACKUP_MANIFEST -successfully deleted database-backup/BACKUP_MANIFEST-CHECKSUM -successfully deleted database-backup/data/679645557047099393.sst -. . . -successfully deleted database-backup/data/679645558154264579.sst -~~~ - -`cockroach userfile upload` will not recursively upload files from a directory. It is possible to programmatically upload your files from the command line. - -For example, the following command finds all files under `database-backup` and runs `cockroach userfile upload` on each file: - -~~~shell -find database-backup -type f | xargs -I{} cockroach userfile upload {} {} --url='postgresql://root@localhost:26257?sslmode=disable' -~~~ - -~~~ -successfully uploaded to userfile://defaultdb.public.userfiles_root/database-backup/BACKUP_MANIFEST-CHECKSUM -successfully uploaded to userfile://defaultdb.public.userfiles_root/database-backup/BACKUP-CHECKPOINT-CHECKSUM -successfully uploaded to userfile://defaultdb.public.userfiles_root/database-backup/BACKUP-STATISTICS -successfully uploaded to userfile://defaultdb.public.userfiles_root/database-backup/BACKUP-CHECKPOINT-679678586269007873-CHECKSUM -successfully uploaded to userfile://defaultdb.public.userfiles_root/database-backup/BACKUP_MANIFEST -successfully uploaded to userfile://defaultdb.public.userfiles_root/database-backup/data/679678597597560835.sst -successfully uploaded to userfile://defaultdb.public.userfiles_root/database-backup/data/679678594426601474.sst -successfully uploaded to userfile://defaultdb.public.userfiles_root/database-backup/data/679678592925335554.sst -. . . -successfully uploaded to userfile://defaultdb.public.userfiles_root/database-backup/data/679678594438758403.sst -~~~ - -Now, with the backup files present in `userfile` storage, you can `RESTORE` the database to the cluster: - -~~~sql -RESTORE DATABASE movr FROM 'userfile:///database-backup'; -~~~ - -Note that `userfile:///` refers to the default path `userfile://defaultdb.public.userfiles_$user/`. - -## Known limitation - -{% include {{ page.version.version }}/known-limitations/userfile-upload-non-recursive.md %} - -## See also - -- [`cockroach userfile list`](cockroach-userfile-list.html) -- [`cockroach userfile delete`](cockroach-userfile-delete.html) -- [`cockroach userfile get`](cockroach-userfile-get.html) -- [Use `userfile` for Bulk Operations](use-userfile-for-bulk-operations.html) -- [Other Cockroach Commands](cockroach-commands.html) -- [`IMPORT`](import.html) -- [`IMPORT INTO`](import-into.html) diff --git a/src/current/v21.1/cockroach-version.md b/src/current/v21.1/cockroach-version.md deleted file mode 100644 index a068cb030c3..00000000000 --- a/src/current/v21.1/cockroach-version.md +++ /dev/null @@ -1,46 +0,0 @@ ---- -title: cockroach version -summary: To view version details for a specific cockroach binary, run the cockroach version command. -toc: true -key: view-version-details.html ---- - -To view version details for a specific `cockroach` binary, run the `cockroach version` [command](cockroach-commands.html): - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach version -~~~ - -~~~ -Build Tag: {{page.release_info.version}} -Build Time: {{page.release_info.build_time}} -Distribution: CCL -Platform: darwin amd64 (x86_64-apple-darwin19) -Go Version: go1.15.11 -C Compiler: Clang 10.0.0 -Build Commit ID: ac916850f403f083ea62e2b0dfdfecbbeaaa4d05 -Build Type: release -~~~ - -{% include_cached new-in.html version="v21.1" %} You can also run `cockroach --version`. - -## Response - -The `cockroach version` command outputs the following fields: - -Field | Description -------|------------ -`Build Tag` | The CockroachDB version.

**New in v21.1:** To return just the build tag, use `cockroach version --build-tag`. -`Build Time` | The date and time when the binary was built. -`Distribution` | The scope of the binary. If `CCL`, the binary contains functionality covered by both the CockroachDB Community License (CCL) and the Business Source License (BSL). If `OSS`, the binary contains only functionality covered by the Apache 2.0 license. The v19.2 release converts to Apache 2.0 as of Oct 1, 2022, at which time you can use the `make buildoss` command to build a pure open-source binary. For more details about licensing, see the [Licensing FAQs](licensing-faqs.html). -`Platform` | The platform that the binary can run on. -`Go Version` | The version of Go in which the source code is written. -`C Compiler` | The C compiler used to build the binary. -`Build Commit ID` | The SHA-1 hash of the commit used to build the binary. -`Build Type` | The type of release. If `release`, `release-gnu`, or `release-musl`, the binary is for a production [release](../releases/). If `development`, the binary is for a testing release. - -## See also - -- [Install CockroachDB](install-cockroachdb.html) -- [Other Cockroach Commands](cockroach-commands.html) diff --git a/src/current/v21.1/cockroach-workload.md b/src/current/v21.1/cockroach-workload.md deleted file mode 100644 index d7f445b1c19..00000000000 --- a/src/current/v21.1/cockroach-workload.md +++ /dev/null @@ -1,664 +0,0 @@ ---- -title: cockroach workload -summary: Use cockroach workload to run a load generator against a CockroachDB cluster. -toc: true ---- - -CockroachDB comes with built-in load generators for simulating different types of client workloads, printing per-operation statistics and totals after a specific duration or max number of operations. To run one of these load generators, use the `cockroach workload` [command](cockroach-commands.html) as described below. - -{{site.data.alerts.callout_danger}} -The `cockroach workload` command is experimental. The interface and output are subject to change. -{{site.data.alerts.end}} - -## Synopsis - -Create the schema for a workload: - -~~~ shell -$ cockroach workload init '' -~~~ - -Run a workload: - -~~~ shell -$ cockroach workload run '' -~~~ - -View help: - -~~~ shell -$ cockroach workload --help -~~~ -~~~ shell -$ cockroach workload init --help -~~~ -~~~ shell -$ cockroach workload init --help -~~~ -~~~ shell -$ cockroach workload run --help -~~~ -~~~ shell -$ cockroach workload run --help -~~~ - -## Subcommands - -Command | Usage ---------|------ -`init` | Load the schema for the workload. You run this command once for a given schema. -`run` | Run a workload. You can run this command multiple times from different machines to increase concurrency. See [Concurrency](#concurrency) for more details. - -## Concurrency - -There are two ways to increase the concurrency of a workload: - -- **Increase the concurrency of a single workload instance** by running `cockroach workload run ` with the `--concurrency` flag set to a value higher than the default. Note that not all workloads support this flag. -- **Run multiple instances of a workload in parallel** by running `cockroach workload run ` multiple times from different terminals/machines. - -## Workloads - -Workload | Description ----------|------------ -[`bank`](#bank-workload) | Models a set of accounts with currency balances.

For this workload, you run `workload init` to load the schema and then `workload run` to generate data. -[`intro`](#intro-and-startrek-workloads) | Loads an `intro` database, with one table, `mytable`, with a hidden message.

For this workload, you run only `workload init` to load the data. The `workload run` subcommand is not applicable. -[`kv`](#kv-workload) | Reads and writes to keys spread (by default, uniformly at random) across the cluster.

For this workload, you run `workload init` to load the schema and then `workload run` to generate data. -[`movr`](#movr-workload) | Simulates a workload for the [MovR example application](movr.html).

For this workload, you run `workload init` to load the schema and then `workload run` to generate data. -[`startrek`](#intro-and-startrek-workloads) | Loads a `startrek` database, with two tables, `episodes` and `quotes`.

For this workload, you run only `workload init` to load the data. The `workload run` subcommand is not applicable. -[`tpcc`](#tpcc-workload) | Simulates a transaction processing workload using a rich schema of multiple tables.

For this workload, you run `workload init` to load the schema and then `workload run` to generate data. -[`ycsb`](#ycsb-workload) | Simulates a high-scale key value workload, either read-heavy, write-heavy, or scan-based, with additional customizations.

For this workload, you run `workload init` to load the schema and then `workload run` to generate data. - -{{site.data.alerts.callout_info}} - `cockroach workload` sets the [`application_name`](set-vars.html#supported-variables) for its workload queries to the name of the workload that is used. You can filter queries on `application_name` on the [Statements page of the DB Console](ui-statements-page.html#search-and-filter-by-application), or in a [`SHOW STATEMENTS`](show-statements.html#filter-for-specific-queries) statement. -{{site.data.alerts.end}} - -## Flags - -{{site.data.alerts.callout_info}} -The `cockroach workload` command does not support connection or security flags like other [`cockroach` commands](cockroach-commands.html). Instead, you must use a [connection string](connection-parameters.html) at the end of the command. -{{site.data.alerts.end}} - -### `bank` workload - -Flag | Description ------|------------ -`--concurrency` | The number of concurrent workers.

**Applicable commands:** `init` or `run`
**Default:** 2 * number of CPUs -`--db` | The SQL database to use.

**Applicable commands:** `init` or `run`
**Default:** `bank` -`--display-every` | The frequency for printing per-operation statistics. Valid [time units](https://en.wikipedia.org/wiki/Orders_of_magnitude_(time)) are `ns`, `us`, `ms`, `s`, `m`, and `h`.

**Applicable command:** `run`
**Default:** `1s` -`--display-format` | The format for printing per-operation statistics (`simple`, `incremental-json`). When using `incremental-json`, note that totals are not printed at the end of the workload's duration.

**Applicable command:** `run`
**Default:** `simple` -`--drop` | Drop the existing database, if it exists.

**Applicable commands:** `init` or `run`. For the `run` command, this flag must be used in conjunction with `--init`. -`--duration` | The duration to run, with a required time unit suffix. Valid [time units](https://en.wikipedia.org/wiki/Orders_of_magnitude_(time)) are `ns`, `us`, `ms`, `s`, `m`, and `h`.

**Applicable commands:** `init` or `run`
**Default:** `0`, which means run forever. -`--histograms` | The file to write per-op incremental and cumulative histogram data to.

**Applicable command:** `run` -`--init` | **Deprecated.** Use the `init` command instead.

**Applicable command:** `run` -`--max-ops` | The maximum number of operations to run.

**Applicable command:** `run` -`--max-rate` | The maximum frequency of operations (reads/writes).

**Applicable command:** `run`
**Default:** `0`, which means unlimited. -`--payload-bytes` | The size of the payload field in each initial row.

**Applicable commands:** `init` or `run`
**Default:** `100` -`--ramp` | The duration over which to ramp up load.

**Applicable command:** `run` -`--ranges` | The initial number of ranges in the `bank` table.

**Applicable commands:** `init` or `run`
**Default:** `10` -`--rows` | The initial number of accounts in the `bank` table.

**Applicable commands:** `init` or `run`
**Default:** `1000` -`--seed` | The key hash seed.

**Applicable commands:** `init` or `run`
**Default:** `1` -`--tolerate-errors` | Keep running on error.

**Applicable command:** `run` - -### `intro` and `startrek` workloads - -{{site.data.alerts.callout_info}} -These workloads generate data but do not offer the ability to run continuous load. Thus, only the `init` subcommand is supported. -{{site.data.alerts.end}} - -Flag | Description ------|------------ -`--drop` | Drop the existing database, if it exists, before loading the dataset. - - -### `kv` workload - -Flag | Description ------|------------ -`--batch` | The number of blocks to insert in a single SQL statement.

**Applicable commands:** `init` or `run`
**Default:** `1` -`--concurrency` | The number of concurrent workers.

**Applicable commands:** `init` or `run`
**Default:** `8` `--cycle-length`| The number of keys repeatedly accessed by each writer.**Applicable commands:** `init` or `run`
**Default:** `9223372036854775807` -`--db` | The SQL database to use.

**Applicable commands:** `init` or `run`
**Default:** `kv` -`--display-every` | The frequency for printing per-operation statistics. Valid [time units](https://en.wikipedia.org/wiki/Orders_of_magnitude_(time)) are `ns`, `us`, `ms`, `s`, `m`, and `h`.

**Applicable command:** `run`
**Default:** `1s` -`--display-format` | The format for printing per-operation statistics (`simple`, `incremental-json`). When using `incremental-json`, note that totals are not printed at the end of the workload's duration.

**Applicable command:** `run`
**Default:** `simple` -`--drop` | Drop the existing database, if it exists.

**Applicable commands:** `init` or `run` -`--duration` | The duration to run, with a required time unit suffix. Valid [time units](https://en.wikipedia.org/wiki/Orders_of_magnitude_(time)) are `ns`, `us`, `ms`, `s`, `m`, and `h`.

**Applicable command:** `run`
**Default:** `0`, which means run forever. -`--histograms` | The file to write per-op incremental and cumulative histogram data to.

**Applicable command:** `run` -`--init` | **Deprecated.** Use the `init` command instead.

**Applicable command:** `run` -`--max-block-bytes` | The maximum amount of raw data written with each insertion.

**Applicable commands:** `init` or `run`
**Default:** `2` -`--max-ops` | The maximum number of operations to run.

**Applicable command:** `run` -`--max-rate` | The maximum frequency of operations (reads/writes).

**Applicable command:** `run`
**Default:** `0`, which means unlimited. -`--min-block-bytes` | The minimum amount of raw data written with each insertion.

**Applicable commands:** `init` or `run`
**Default:** `1` -`--ramp` | The duration over which to ramp up load.

**Applicable command:** `run` -`--read-percent` | The percent (0-100) of operations that are reads of existing keys.

**Applicable commands:** `init` or `run` -`--seed` | The key hash seed.

**Applicable commands:** `init` or `run`
**Default:** `1` -`--sequential` | Pick keys sequentially instead of randomly.

**Applicable commands:** `init` or `run` -`--splits` | The number of splits to perform before starting normal operations.

**Applicable commands:** `init` or `run` -`--tolerate-errors` | Keep running on error.

**Applicable command:** `run` -`--use-opt` | Use [cost-based optimizer](cost-based-optimizer.html).

**Applicable commands:** `init` or `run`
**Default:** `true` -`--write-seq` | Initial write sequence value.

**Applicable commands:** `init` or `run` - -### `movr` workload - -Flag | Description ------|------------ -`--data-loader` | How to load initial table data. Valid options are `INSERT` and `IMPORT`.

**Applicable commands:** `init` or `run`
**Default:** `INSERT` -`--db` | The SQL database to use.

**Applicable commands:** `init` or `run`
**Default:** `movr` -`--display-every` | The frequency for printing per-operation statistics. Valid [time units](https://en.wikipedia.org/wiki/Orders_of_magnitude_(time)) are `ns`, `us`, `ms`, `s`, `m`, and `h`.

**Applicable command:** `run`
**Default:** `1s` -`--display-format` | The format for printing per-operation statistics (`simple`, `incremental-json`). When using `incremental-json`, note that totals are not printed at the end of the workload's duration.

**Applicable command:** `run`
**Default:** `simple` -`--drop` | Drop the existing database, if it exists.

**Applicable commands:** `init` or `run` -`--duration` | The duration to run, with a required time unit suffix. Valid [time units](https://en.wikipedia.org/wiki/Orders_of_magnitude_(time)) are `ns`, `us`, `ms`, `s`, `m`, and `h`.

**Applicable command:** `run`
**Default:** `0`, which means run forever. -`--histograms` | The file to write per-op incremental and cumulative histogram data to.

**Applicable command:** `run` -`--histograms-max-latency` | Expected maximum latency of running a query, with a required time unit suffix. Valid [time units](https://en.wikipedia.org/wiki/Orders_of_magnitude_(time)) are `ns`, `us`, `ms`, `s`, `m`, and `h`.

**Applicable command:** `run`
**Default:** `1m40s` -`--max-ops` | The maximum number of operations to run.

**Applicable command:** `run` -`--max-rate` | The maximum frequency of operations (reads/writes).

**Applicable command:** `run`
**Default:** `0`, which means unlimited. -`--method` | The SQL issue method (`prepare`, `noprepare`, `simple`).

**Applicable commands:** `init` or `run`
**Default:** `prepare` -`--num-histories` | The initial number of ride location histories.

**Applicable commands:** `init` or `run`
**Default:** `1000` -`--num-promo-codes` | The initial number of promo codes.

**Applicable commands:** `init` or `run`
**Default:** `1000` -`--num-rides` | Initial number of rides.

**Applicable commands:** `init` or `run`
**Default:** `500` -`--num-users` | Initial number of users.

**Applicable commands:** `init` or `run`
**Default:** `50` -`--num-vehicles` | Initial number of vehicles.

**Applicable commands:** `init` or `run`
**Default:** `15` -`--ramp` | The duration over which to ramp up load.

**Applicable command:** `run` -`--seed` | The random number generator seed.

**Applicable commands:** `init` or `run`
**Default:** `1` -`--tolerate-errors` | Keep running on error.

**Applicable command:** `run` - -### `tpcc` workload - -Flag | Description ------|------------ -`--active-warehouses` | Run the load generator against a specific number of warehouses.

**Applicable commands:** `init` or `run`
**Defaults:** Value of `--warehouses` -`--data-loader` | How to load initial table data. Valid options are `INSERT` and `IMPORT`.

**Applicable commands:** `init` or `run`
**Default:** `INSERT` -`--db` | The SQL database to use.

**Applicable commands:** `init` or `run`
**Default:** `tpcc` -`--display-every` | The frequency for printing per-operation statistics. Valid [time units](https://en.wikipedia.org/wiki/Orders_of_magnitude_(time)) are `ns`, `us`, `ms`, `s`, `m`, and `h`.

**Applicable command:** `run`
**Default:** `1s` -`--display-format` | The format for printing per-operation statistics (`simple`, `incremental-json`). When using `incremental-json`, note that totals are not printed at the end of the workload's duration.

**Applicable command:** `run`
**Default:** `simple` -`--drop` | Drop the existing database, if it exists.

**Applicable commands:** `init` or `run`. For the `run` command, this flag must be used in conjunction with `--init`. -`--duration` | The duration to run, with a required time unit suffix. Valid [time units](https://en.wikipedia.org/wiki/Orders_of_magnitude_(time)) are `ns`, `us`, `ms`, `s`, `m`, and `h`.

**Applicable command:** `run`
**Default:** `0`, which means run forever. -`--fks` | Add foreign keys.

**Applicable commands:** `init` or `run`
**Default:** `true` -`--histograms` | The file to write per-op incremental and cumulative histogram data to.

**Applicable command:** `run` -`--idle-conns` | Tests the TPCC workload with idle connections. -`--init` | **Deprecated.** Use the `init` command instead.

**Applicable command:** `run` -`--interleaved` | Use [interleaved tables](interleave-in-parent.html).

**Applicable commands:** `init` or `run`
{% include {{ page.version.version }}/misc/interleave-deprecation-note.md %} -`--max-ops` | The maximum number of operations to run.

**Applicable command:** `run` -`--max-rate` | The maximum frequency of operations (reads/writes).

**Applicable command:** `run`
**Default:** `0`, which means unlimited. -`--mix` | Weights for the transaction mix.

**Applicable commands:** `init` or `run`
**Default:** `newOrder=10,payment=10,orderStatus=1,delivery=1,stockLevel=1`, which matches the [TPC-C specification](http://www.tpc.org/tpc_documents_current_versions/pdf/tpc-c_v5.11.0.pdf). -`--partition-affinity` | Run the load generator against a specific partition. This flag must be used in conjunction with `--partitions`.

**Applicable commands:** `init` or `run`
**Default:** `-1` -`--partitions` | Partition tables. This flag must be used in conjunction with `--split`.

**Applicable commands:** `init` or `run` -`--ramp` | The duration over which to ramp up load.

**Applicable command:** `run` -`--scatter` | Scatter ranges.

**Applicable commands:** `init` or `run` -`--seed` | The random number generator seed.

**Applicable commands:** `init` or `run`
**Default:** `1` -`--serializable` | Force serializable mode. CockroachDB only supports `SERIALIZABLE` isolation, so this flag is not necessary.

**Applicable command:** `init` -`--split` | [Split tables](split-at.html).

**Applicable commands:** `init` or `run` -`--tolerate-errors` | Keep running on error.

**Applicable command:** `run` -`--wait` | Run in wait mode, i.e., include think/keying sleeps.

**Applicable commands:** `init` or `run`
**Default:** `true` -`--warehouses` | The number of warehouses for loading initial data, at approximately 200 MB per warehouse.

**Applicable commands:** `init` or `run`
**Default:** `1` -`--workers` | The number of concurrent workers.

**Applicable commands:** `init` or `run`
**Default:** `--warehouses` * 10 -`--zones` | The number of [replication zones](configure-replication-zones.html) for partitioning. This number should match the number of `--partitions` and the zones used to start the cluster.

**Applicable command:** `init` - -### `ycsb` workload - -Flag | Description ------|------------ -`--concurrency` | The number of concurrent workers.

**Applicable commands:** `init` or `run`
**Default:** `8` -`--data-loader` | How to load initial table data. Valid options are `INSERT` and `IMPORT`.

**Applicable commands:** `init` or `run`
**Default:** `INSERT` -`--db` | The SQL database to use.

**Applicable commands:** `init` or `run`
**Default:** `ycsb` -`--display-every` | The frequency for printing per-operation statistics. Valid [time units](https://en.wikipedia.org/wiki/Orders_of_magnitude_(time)) are `ns`, `us`, `ms`, `s`, `m`, and `h`.

**Applicable command:** `run`
**Default:** `1s` -`--display-format` | The format for printing per-operation statistics (`simple`, `incremental-json`). When using `incremental-json`, note that totals are not printed at the end of the workload's duration.

**Applicable command:** `run`
**Default:** `simple` -`--drop` | Drop the existing database, if it exists.

**Applicable commands:** `init` or `run`. For the `run` command, this flag must be used in conjunction with `--init`. -`--duration` | The duration to run, with a required time unit suffix. Valid [time units](https://en.wikipedia.org/wiki/Orders_of_magnitude_(time)) are `ns`, `us`, `ms`, `s`, `m`, and `h`.

**Applicable command:** `run`
**Default:** `0`, which means run forever. -`--families` | Place each column in its own [column family](column-families.html).

**Applicable commands:** `init` or `run` -`--histograms` | The file to write per-op incremental and cumulative histogram data to.

**Applicable command:** `run` -`--init` | **Deprecated.** Use the `init` command instead.

**Applicable command:** `run` -`--insert-count` | Number of rows to sequentially insert before beginning random number generation.

**Applicable commands:** `init` or `run`
**Default:** `10000` -`--json` | Use JSONB rather than relational data.

**Applicable commands:** `init` or `run` -`--max-ops` | The maximum number of operations to run.

**Applicable command:** `run` -`--max-rate` | The maximum frequency of operations (reads/writes).

**Applicable command:** `run`
**Default:** `0`, which means unlimited. -`--method` | The SQL issue method (`prepare`, `noprepare`, `simple`).

**Applicable commands:** `init` or `run`
**Default:** `prepare` -`--ramp` | The duration over which to ramp up load.

**Applicable command:** `run` -`--request-distribution` | Distribution for the random number generator (`zipfian`, `uniform`).

**Applicable commands:** `init` or `run`.
**Default:** `zipfian` -`--seed` | The random number generator seed.

**Applicable commands:** `init` or `run`
**Default:** `1` -`--splits` | Number of [splits](split-at.html) to perform before starting normal operations.

**Applicable commands:** `init` or `run` -`--tolerate-errors` | Keep running on error.

**Applicable command:** `run` -`--workload` | The type of workload to run (`A`, `B`, `C`, `D`, or `F`). For details about these workloads, see [YCSB Workloads](https://github.com/brianfrankcooper/YCSB/wiki/Core-Workloads).

**Applicable commands:** `init` or `run`
**Default:** `B` - -### Logging - -By default, the `cockroach workload` command logs messages to `stderr`. This includes events with `INFO` [severity](logging.html#logging-levels-severities) and higher. - -If you need to troubleshoot this command's behavior, you can [customize its logging behavior](configure-logs.html). - -## Examples - -These examples assume that you have already [started an insecure cluster locally](start-a-local-cluster.html): - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start \ ---insecure \ ---listen-addr=localhost -~~~ - -### Run the `bank` workload - -1. Load the initial schema: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload init bank \ - 'postgresql://root@localhost:26257?sslmode=disable' - ~~~ - -2. Run the workload for 1 minute: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload run bank \ - --duration=1m \ - 'postgresql://root@localhost:26257?sslmode=disable' - ~~~ - - You'll see per-operation statistics print to standard output every second: - - ~~~ - _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms) - 1s 0 1608.6 1702.2 4.5 7.3 12.6 65.0 transfer - 2s 0 1725.3 1713.8 4.5 7.9 13.1 19.9 transfer - 3s 0 1721.1 1716.2 4.5 7.3 11.5 21.0 transfer - 4s 0 1328.7 1619.2 5.5 10.5 17.8 39.8 transfer - 5s 0 1389.3 1573.3 5.2 11.5 16.3 23.1 transfer - 6s 0 1640.0 1584.4 5.0 7.9 12.1 16.3 transfer - 7s 0 1594.0 1585.8 5.0 7.9 10.5 15.7 transfer - 8s 0 1652.8 1594.2 4.7 7.9 11.5 29.4 transfer - 9s 0 1451.9 1578.4 5.2 10.0 15.2 26.2 transfer - 10s 0 1653.3 1585.9 5.0 7.6 10.0 18.9 transfer - ... - ~~~ - - After the specified duration (1 minute in this case), the workload will stop and you'll see totals printed to standard output: - - ~~~ - _elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__result - 60.0s 0 84457 1407.6 5.7 5.5 10.0 15.2 167.8 - ~~~ - -### Run the `kv` workload - -1. Load the initial schema: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload init kv \ - 'postgresql://root@localhost:26257?sslmode=disable' - ~~~ - -2. Run the workload for 1 minute: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload run kv \ - --duration=1m \ - 'postgresql://root@localhost:26257?sslmode=disable' - ~~~ - - You'll see per-operation statistics print to standard output every second: - - ~~~ - _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms) - 1s 0 5095.8 5123.7 1.5 2.5 3.3 7.3 write - 2s 0 4795.4 4959.6 1.6 2.8 3.5 8.9 write - 3s 0 3456.5 4458.5 2.0 4.5 7.3 24.1 write - 4s 0 2787.9 4040.8 2.4 6.3 12.6 30.4 write - 5s 0 3558.7 3944.4 2.0 4.2 6.8 11.5 write - 6s 0 3733.8 3909.3 1.9 4.2 6.0 12.6 write - 7s 0 3565.6 3860.1 2.0 4.7 7.9 25.2 write - 8s 0 3469.3 3811.4 2.0 5.0 6.8 22.0 write - 9s 0 3937.6 3825.4 1.8 3.7 7.3 29.4 write - 10s 0 3822.9 3825.1 1.8 4.7 8.9 37.7 write - ... - ~~~ - - After the specified duration (1 minute in this case), the workload will stop and you'll see totals printed to standard output: - - ~~~ - _elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__result - 60.0s 0 276067 4601.0 1.7 1.6 3.1 5.2 96.5 - ~~~ - -### Load the `intro` dataset - -1. Load the dataset: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload init intro \ - 'postgresql://root@localhost:26257?sslmode=disable' - ~~~ - -2. Launch the built-in SQL client to view it: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --insecure - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW TABLES FROM intro; - ~~~ - - ~~~ - table_name - +------------+ - mytable - (1 row) - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ SELECT * FROM intro.mytable WHERE (l % 2) = 0; - ~~~ - - ~~~ - l | v - +----+------------------------------------------------------+ - 0 | !__aaawwmqmqmwwwaas,,_ .__aaawwwmqmqmwwaaa,, - 2 | !"VT?!"""^~~^"""??T$Wmqaa,_auqmWBT?!"""^~~^^""??YV^ - 4 | ! "?##mW##?"- - 6 | ! C O N G R A T S _am#Z??A#ma, Y - 8 | ! _ummY" "9#ma, A - 10 | ! vm#Z( )Xmms Y - 12 | ! .j####mmm#####mm#m##6. - 14 | ! W O W ! jmm###mm######m#mmm##6 - 16 | ! ]#me*Xm#m#mm##m#m##SX##c - 18 | ! dm#||+*$##m#mm#m#Svvn##m - 20 | ! :mmE=|+||S##m##m#1nvnnX##; A - 22 | ! :m#h+|+++=Xmm#m#1nvnnvdmm; M - 24 | ! Y $#m>+|+|||##m#1nvnnnnmm# A - 26 | ! O ]##z+|+|+|3#mEnnnnvnd##f Z - 28 | ! U D 4##c|+|+|]m#kvnvnno##P E - 30 | ! I 4#ma+|++]mmhvnnvq##P` ! - 32 | ! D I ?$#q%+|dmmmvnnm##! - 34 | ! T -4##wu#mm#pw##7' - 36 | ! -?$##m####Y' - 38 | ! !! "Y##Y"- - 40 | ! - (21 rows) - ~~~ - -### Load the `startrek` dataset - -1. Load the dataset: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload init startrek \ - 'postgresql://root@localhost:26257?sslmode=disable' - ~~~ - -2. Launch the built-in SQL client to view it: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --insecure - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW TABLES FROM startrek; - ~~~ - - ~~~ - table_name - +------------+ - episodes - quotes - (2 rows) - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SELECT * FROM startrek.episodes WHERE stardate > 5500; - ~~~ - - ~~~ - id | season | num | title | stardate - +----+--------+-----+-----------------------------------+----------+ - 60 | 3 | 5 | Is There in Truth No Beauty? | 5630.7 - 62 | 3 | 7 | Day of the Dove | 5630.3 - 64 | 3 | 9 | The Tholian Web | 5693.2 - 65 | 3 | 10 | Plato's Stepchildren | 5784.2 - 66 | 3 | 11 | Wink of an Eye | 5710.5 - 69 | 3 | 14 | Whom Gods Destroy | 5718.3 - 70 | 3 | 15 | Let That Be Your Last Battlefield | 5730.2 - 73 | 3 | 18 | The Lights of Zetar | 5725.3 - 74 | 3 | 19 | Requiem for Methuselah | 5843.7 - 75 | 3 | 20 | The Way to Eden | 5832.3 - 76 | 3 | 21 | The Cloud Minders | 5818.4 - 77 | 3 | 22 | The Savage Curtain | 5906.4 - 78 | 3 | 23 | All Our Yesterdays | 5943.7 - 79 | 3 | 24 | Turnabout Intruder | 5928.5 - (14 rows) - ~~~ - -### Load the `movr` dataset - -1. Load the dataset: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload init movr \ - 'postgresql://root@localhost:26257?sslmode=disable' - ~~~ - -2. Launch the built-in SQL client to view it: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --insecure - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW TABLES FROM movr; - ~~~ - - ~~~ - table_name -+----------------------------+ - promo_codes - rides - user_promo_codes - users - vehicle_location_histories - vehicles -(6 rows) - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SELECT * FROM movr.users WHERE city='new york'; - ~~~ - - ~~~ - id | city | name | address | credit_card -+--------------------------------------+----------+------------------+-----------------------------+-------------+ - 00000000-0000-4000-8000-000000000000 | new york | Robert Murphy | 99176 Anderson Mills | 8885705228 - 051eb851-eb85-4ec0-8000-000000000001 | new york | James Hamilton | 73488 Sydney Ports Suite 57 | 8340905892 - 0a3d70a3-d70a-4d80-8000-000000000002 | new york | Judy White | 18580 Rosario Ville Apt. 61 | 2597958636 - 0f5c28f5-c28f-4c00-8000-000000000003 | new york | Devin Jordan | 81127 Angela Ferry Apt. 8 | 5614075234 - 147ae147-ae14-4b00-8000-000000000004 | new york | Catherine Nelson | 1149 Lee Alley | 0792553487 -(5 rows) - ~~~ - -### Run the `movr` workload - -1. Load the initial schema: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload init movr \ - 'postgresql://root@localhost:26257?sslmode=disable' - ~~~ - -2. Initialize and run the workload for 1 minute: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload run movr \ - --duration=1m \ - 'postgresql://root@localhost:26257?sslmode=disable' - ~~~ - - You'll see per-operation statistics print to standard output every second: - - ~~~ - _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms) - 1.0s 0 31.9 32.0 0.5 0.6 1.4 1.4 addUser - 1.0s 0 6.0 6.0 1.2 1.4 1.4 1.4 addVehicle - 1.0s 0 10.0 10.0 2.2 6.3 6.3 6.3 applyPromoCode - 1.0s 0 2.0 2.0 0.5 0.6 0.6 0.6 createPromoCode - 1.0s 0 9.0 9.0 0.9 1.6 1.6 1.6 endRide - 1.0s 0 1407.5 1407.8 0.3 0.5 0.7 4.1 readVehicles - 1.0s 0 27.0 27.0 2.1 3.1 4.7 4.7 startRide - 1.0s 0 86.8 86.9 4.7 8.4 11.5 15.2 updateActiveRides - 2.0s 0 26.0 29.0 0.5 1.1 1.4 1.4 addUser - 2.0s 0 8.0 7.0 1.2 2.8 2.8 2.8 addVehicle - 2.0s 0 2.0 6.0 2.6 2.8 2.8 2.8 applyPromoCode - 2.0s 0 0.0 1.0 0.0 0.0 0.0 0.0 createPromoCode - 2.0s 0 6.0 7.5 0.8 1.7 1.7 1.7 endRide - 2.0s 0 1450.4 1429.1 0.3 0.6 0.9 2.6 readVehicles - 2.0s 0 17.0 22.0 2.1 3.3 5.5 5.5 startRide - 2.0s 0 59.0 72.9 6.3 11.5 11.5 14.2 updateActiveRides - ... - ~~~ - - After the specified duration (1 minute in this case), the workload will stop and you'll see totals printed to standard output: - - ~~~ - _elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__result - 60.0s 0 85297 1421.6 0.7 0.3 2.6 7.1 30.4 - ~~~ - -### Run the `tpcc` workload - -1. Load the initial schema and data: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload init tpcc \ - 'postgresql://root@localhost:26257?sslmode=disable' - ~~~ - -2. Run the workload for 10 minutes: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload run tpcc \ - --duration=10m \ - 'postgresql://root@localhost:26257?sslmode=disable' - ~~~ - - You'll see per-operation statistics print to standard output every second: - - ~~~ - _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms) - 1s 0 1443.4 1494.8 4.7 9.4 27.3 67.1 transfer - 2s 0 1686.5 1590.9 4.7 8.1 15.2 28.3 transfer - 3s 0 1735.7 1639.0 4.7 7.3 11.5 28.3 transfer - 4s 0 1542.6 1614.9 5.0 8.9 12.1 21.0 transfer - 5s 0 1695.9 1631.1 4.7 7.3 11.5 22.0 transfer - 6s 0 1569.2 1620.8 5.0 8.4 11.5 15.7 transfer - 7s 0 1614.6 1619.9 4.7 8.1 12.1 16.8 transfer - 8s 0 1344.4 1585.6 5.8 10.0 15.2 31.5 transfer - 9s 0 1351.9 1559.5 5.8 10.0 16.8 54.5 transfer - 10s 0 1514.8 1555.0 5.2 8.1 12.1 16.8 transfer - ... - ~~~ - - After the specified duration (10 minutes in this case), the workload will stop and you'll see totals printed to standard output: - - ~~~ - _elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__result - 600.0s 0 823902 1373.2 5.8 5.5 10.0 15.2 209.7 - ~~~ - -### Run the `ycsb` workload - -1. Load the initial schema and data: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload init ycsb \ - 'postgresql://root@localhost:26257?sslmode=disable' - ~~~ - -2. Run the workload for 10 minutes: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload run ycsb \ - --duration=10m \ - 'postgresql://root@localhost:26257?sslmode=disable' - ~~~ - - You'll see per-operation statistics print to standard output every second: - - ~~~ - _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms) - 1s 0 9258.1 9666.6 0.7 1.3 2.0 8.9 read - 1s 0 470.1 490.9 1.7 2.9 4.1 5.0 update - 2s 0 10244.6 9955.6 0.7 1.2 2.0 6.6 read - 2s 0 559.0 525.0 1.6 3.1 6.0 7.3 update - 3s 0 9870.8 9927.4 0.7 1.4 2.4 10.0 read - 3s 0 500.0 516.6 1.6 4.2 7.9 15.2 update - 4s 0 9847.2 9907.3 0.7 1.4 2.4 23.1 read - 4s 0 506.8 514.2 1.6 3.7 7.6 17.8 update - 5s 0 10084.4 9942.6 0.7 1.3 2.1 7.1 read - 5s 0 537.2 518.8 1.5 3.5 10.0 15.2 update - ... - ~~~ - - After the specified duration (10 minutes in this case), the workload will stop and you'll see totals printed to standard output: - - ~~~ - _elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__result - 600.0s 0 4728286 7880.2 1.0 0.9 2.2 5.2 268.4 - ~~~ - -### Customize the frequency and format of per-operation statistics - -To customize the frequency of per-operation statistics, use the `--display-every` flag, with `ns`, `us`, `ms`, `s`, `m`, and `h` as valid [time units](https://en.wikipedia.org/wiki/Orders_of_magnitude_(time)). To customize the format of per-operation statistics, use the `--display-format` flag, with `incremental-json` or `simple` (default) as options. - -1. Load the initial schema and data: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload init ycsb \ - 'postgresql://root@localhost:26257?sslmode=disable' - ~~~ - -2. Run the workload for 1 minute, printing the output every 5 seconds as JSON: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload run ycsb \ - --duration=1m \ - --display-every=5s \ - --display-format=incremental-json \ - 'postgresql://root@localhost:26257?sslmode=disable' - ~~~ - - ~~~ - {"time":"2019-09-13T03:25:03.950621Z","errs":0,"avgt":8434.5,"avgl":8471.0,"p50l":0.8,"p95l":1.6,"p99l":3.1,"maxl":19.9,"type":"read"} - {"time":"2019-09-13T03:25:03.950621Z","errs":0,"avgt":438.1,"avgl":440.0,"p50l":1.5,"p95l":2.8,"p99l":4.5,"maxl":14.7,"type":"update"} - {"time":"2019-09-13T03:25:08.95061Z","errs":0,"avgt":7610.6,"avgl":8040.8,"p50l":0.8,"p95l":2.0,"p99l":4.2,"maxl":65.0,"type":"read"} - {"time":"2019-09-13T03:25:08.95061Z","errs":0,"avgt":391.8,"avgl":415.9,"p50l":1.6,"p95l":3.5,"p99l":5.8,"maxl":21.0,"type":"update"} - {"time":"2019-09-13T03:25:13.950727Z","errs":0,"avgt":7242.0,"avgl":7774.5,"p50l":0.8,"p95l":2.2,"p99l":4.7,"maxl":75.5,"type":"read"} - {"time":"2019-09-13T03:25:13.950727Z","errs":0,"avgt":382.0,"avgl":404.6,"p50l":1.6,"p95l":4.7,"p99l":10.5,"maxl":24.1,"type":"update"} - ... - ~~~ - - When using `incremental-json`, note that totals are not printed at the end of the workload's duration. - -## See also - -- [`cockroach demo`](cockroach-demo.html) -- [Other Cockroach Commands](cockroach-commands.html) -- [Performance Benchmarking with TPC-C](performance-benchmarking-with-tpcc-small.html) diff --git a/src/current/v21.1/cockroachdb-in-comparison.md b/src/current/v21.1/cockroachdb-in-comparison.md deleted file mode 100644 index 9e49ea564f1..00000000000 --- a/src/current/v21.1/cockroachdb-in-comparison.md +++ /dev/null @@ -1,353 +0,0 @@ ---- -title: CockroachDB in Comparison -summary: Learn how CockroachDB compares to other popular databases like PostgreSQL, Cassandra, MongoDB, Google Cloud Spanner, and more. -tags: mongodb, mysql, dynamodb -toc: false -comparison: true ---- - -This page shows you how the key features of CockroachDB stack up against other databases. Hover over the features for their intended meanings. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - CockroachDB
- Database horizontal scale - - tooltip icon - - - Manual Sharding - Add on configuration - Node based, automated read scale, limited write - Node based, automated for both reads and writes - - Manual Sharding - Add on configuration - Node based, automated read scale, limited write - Node based, automated for both reads and writes - Node based, automated for both reads and writes
- Database load balancing (internal) - - tooltip icon - - - Manual - not part of database - None and full copies across regions - Even distribution to optimize storage - - Manual - not part of database - None and full copies across regions - Even distribution to optimize storage - Detailed options to optimize storage, compute, and latency
- Failover - - tooltip icon - - - Manual - not part of database - Automated for reads, limited for writes to one region - Automated for reads, limited guarantees for writes - Fully automated for both reads and writes - - Manual - not part of database - Automated for reads, limited for writes to one region - Automated for reads, limited guarantees for writes - Fully automated for both reads and writes - Fully automated for both reads and writes
- Automated repair and RPO(Recovery Point Objective) - - tooltip icon - - - Manual repair RPO ~1-60 mins - Automated RPO ~1 -5 mins - Manual & automated repair RPO <1 min - "Automated repair RPO <10 sec" - - Manual repair RPO ~1-60 mins - Automated RPO ~1 -5 mins - Manual & automated repair RPO <1 min - "Automated repair RPO <10 sec" - Automated repair RPO = 0 sec
- Distributed reads - - tooltip icon - - - Manual - asynchronous - Yes - - Manual - asynchronous - Yes - Yes
- Distributed transactions - - tooltip icon - - - No - Lightweight transactions only - Yes - - No - Lightweight transactions only - Yes - Yes
- Database isolation levels - - tooltip icon - - - Single region consistent default - Snapshot highest - Serializable - Eventual consistent default - Read uncommitted highest - Snapshot read - Eventual consistent - No transaction isolation guarantees - Default - Snapshot highest - Serializable - - Single region consistent default - Snapshot highest - Serializable - Eventual consistent default - Read uncommitted highest - Snapshot read - Eventual consistent - No transaction isolation guarantees - Default - Snapshot highest - Serializable - Guaranteed consistent default - Serializable highest - Serializable
- Potential data issues (default) - - tooltip icon - - - Phantom reads, non-repeatable reads, write skew - Dirty reads, phantom reads, non-repeatable reads, write skew - Dirty reads, phantom reads, non-repeatable reads, write conflicts - None - Phantom reads, non-repeatable reads - - Phantom reads, non-repeatable reads, write skew - Dirty reads, phantom reads, non-repeatable reads, write skew - Dirty reads, phantom reads, non-repeatable reads, write conflicts - None - Phantom reads, non-repeatable reads - None
- SQL - - tooltip icon - - - Yes - No - Yes - with limitations - - Yes - No - Yes - with limitations - Yes - wire compatible with PostgreSQL
- Database schema change - - tooltip icon - - - Yes - Offline - Online, Active and Dynamic - - Yes - Offline - Online, Active, and Dynamic - Online, Active, and Dynamic
- Cost based optimization - - tooltip icon - - - Yes - No - ? - No - - Yes - No - ? - No - Yes
- Data Geo-partitioning - - tooltip icon - - - No - Yes, object level - Yes - No - - No - Yes, object level - Yes - No - Yes, row level
- Upgrade method - - tooltip icon - - - Offline - Online, rolling - - Offline - Online, rolling - Online, rolling
- Multi-region - - tooltip icon - - - Yes - manual - Yes, but not for writes - Yes, for both reads and writes - - Yes - manual - Yes, but not for writes - Yes, for both reads and writes - Yes for both reads and writes
- Multi-cloud - - tooltip icon - - - No - Yes - - No - Yes - Yes
- - diff --git a/src/current/v21.1/collate.md b/src/current/v21.1/collate.md deleted file mode 100644 index cf3765a17e9..00000000000 --- a/src/current/v21.1/collate.md +++ /dev/null @@ -1,239 +0,0 @@ ---- -title: COLLATE -summary: The COLLATE feature lets you sort strings according to language- and country-specific rules. -toc: true ---- - -The `COLLATE` feature lets you sort [`STRING`](string.html) values according to language- and country-specific rules, known as collations. - -Collated strings are important because different languages have [different rules for alphabetic order](https://en.wikipedia.org/wiki/Alphabetical_order#Language-specific_conventions), especially with respect to accented letters. For example, in German accented letters are sorted with their unaccented counterparts, while in Swedish they are placed at the end of the alphabet. A collation is a set of rules used for ordering and usually corresponds to a language, though some languages have multiple collations with different rules for sorting; for example Portuguese has separate collations for Brazilian and European dialects (`pt-BR` and `pt-PT` respectively). - -## Details - -- Operations on collated strings cannot involve strings with a different collation or strings with no collation. However, it is possible to add or overwrite a collation on the fly. - -- Only use the collation feature when you need to sort strings by a specific collation. We recommend this because every time a collated string is constructed or loaded into memory, CockroachDB computes its collation key, whose size is linear in relationship to the length of the collated string, which requires additional resources. - -- Collated strings can be considerably larger than the corresponding uncollated strings, depending on the language and the string content. For example, strings containing the character `é` produce larger collation keys in the French locale than in Chinese. - -- Collated strings that are indexed require additional disk space as compared to uncollated strings. In case of indexed collated strings, collation keys must be stored in addition to the strings from which they are derived, creating a constant factor overhead. - -## Supported collations - -CockroachDB supports collations identified by [Unicode locale identifiers](https://cldr.unicode.org/development/core-specification#h.vgyyng33o798). For example, `en-US` identifies US English, `es` identifies Spanish, and `fr-CA` identifies Canadian French. Collation names are case-insensitive, and hyphens and underscores are interchangeable. - -{{site.data.alerts.callout_info}} -If a hyphen is used in a SQL query, the collation name must be enclosed in double quotes, as single quotes are used for SQL string literals. -{{site.data.alerts.end}} - -A list of supported collations can be found in the `pg_catalog.pg_collation` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT collname from pg_catalog.pg_collation; -~~~ - -~~~ - collname ------------------------ - und - aa - af - ar -... -(95 rows) -~~~ - -CockroachDB supports standard aliases for the collations listed in `pg_collation`. For example, `es-419` (Latin American Spanish) and `zh-Hans` (Simplified Chinese) are supported, but they do not appear in the `pg_collations` table because they are equivalent to the `es` and `zh` collations listed in the table. - -CockroachDB also supports the following Unicode locale extensions: - -- `co` (collation type) -- `ks` (strength) -- `kc` (case level) -- `kb` (backwards second level weight) -- `kn` (numeric) -- `ks` (strength) -- `ka` (alternate handling) - -To use a locale extension, append `-u-` to the base locale name, followed by the extension. For example, `en-US-u-ks-level2` is case-insensitive US English. The `ks` modifier changes the "strength" of the collation, causing it to treat certain classes of characters as equivalent (PostgreSQL calls these "non-deterministic collations"). Setting the `ks` to `level2` makes the collation case-insensitive (for languages that have this concept). - -For more details on locale extensions, see the [Unicode Collation Algorithm](https://en.wikipedia.org/wiki/Unicode_collation_algorithm). - -## Collation versioning - -While changes to collations are rare, they are possible, especially in languages with a large numbers of characters (e.g., Simplified and Traditional Chinese). CockroachDB updates its support with new versions of the Unicode standard every year, but there is currently no way to specify the version of Unicode to use. As a result, it is possible for a collation change to invalidate existing collated string data. To prevent collated data from being invalidated by Unicode changes, we recommend storing data in columns with an uncollated string type, and then using a [computed column](computed-columns.html) for the desired collation. In the event that a collation change produces undesired effects, the computed column can be dropped and recreated. - -## SQL syntax - -Collated strings are used as normal strings in SQL, but have a `COLLATE` clause appended to them. - -- **Column syntax**: `STRING COLLATE `. For example: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE TABLE foo (a STRING COLLATE en PRIMARY KEY); - ~~~ - - {{site.data.alerts.callout_info}}You can also use any of the aliases for STRING.{{site.data.alerts.end}} - -- **Value syntax**: ` COLLATE `. For example: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > INSERT INTO foo VALUES ('dog' COLLATE en); - ~~~ - -## Examples - -### Specify collation for a column - -You can set a default collation for all values in a `STRING` column. - -For example, you can set a column's default collation to German (`de`): - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE de_names (name STRING COLLATE de PRIMARY KEY); -~~~ - -When inserting values into this column, you must specify the collation for every value: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO de_names VALUES ('Backhaus' COLLATE de), ('Bär' COLLATE de), ('Baz' COLLATE de); -~~~ - -The sort will now honor the `de` collation that treats *ä* as *a* in alphabetic sorting: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM de_names ORDER BY name; -~~~ -~~~ - name -+----------+ - Backhaus - Bär - Baz -(3 rows) -~~~ - -### Specify collations with locale extensions - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE nocase_strings (greeting STRING COLLATE "en-US-u-ks-level2"); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO nocase_strings VALUES ('Hello, friend.' COLLATE "en-US-u-ks-level2"), ('Hi. My name is Petee.' COLLATE "en-US-u-ks-level2"); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM nocase_strings WHERE greeting = ('hi. my name is petee.' COLLATE "en-US-u-ks-level2"); -~~~ - -~~~ - greeting -+-----------------------+ - Hi. My name is Petee. -(1 row) -~~~ - -### Order by non-default collation - -You can sort a column using a specific collation instead of its default. - -For example, you receive different results if you order results by German (`de`) and Swedish (`sv`) collations: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM de_names ORDER BY name COLLATE sv; -~~~ -~~~ - name -+----------+ - Backhaus - Baz - Bär -(3 rows) -~~~ - -### Ad-hoc collation casting - -You can cast any string into a collation on the fly. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT 'A' COLLATE de < 'Ä' COLLATE de; -~~~ -~~~ - ?column? -+----------+ - true -(1 row) -~~~ - -However, you cannot compare values with different collations: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT 'Ä' COLLATE sv < 'Ä' COLLATE de; -~~~ -~~~ -pq: unsupported comparison operator: < -~~~ - -You can also use casting to remove collations from values. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT CAST(name AS STRING) FROM de_names ORDER BY name; -~~~ -~~~ - name -+----------+ - Backhaus - Baz - Bär -(3 rows) -~~~ - -### Show collation for strings - -You can use the `pg_collation_for` [built-in function](functions-and-operators.html#string-and-byte-functions), or its alternative [syntax form](functions-and-operators.html#special-syntax-forms) `COLLATION FOR`, to return the locale name of a collated string. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT pg_collation_for('Bär' COLLATE de); -~~~ - -~~~ - pg_collation_for -+------------------+ - de -(1 row) -~~~ - -This is equivalent to: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT COLLATION FOR ('Bär' COLLATE de); -~~~ - -~~~ - pg_collation_for -+------------------+ - de -(1 row) -~~~ - -## See also - -[Data Types](data-types.html) diff --git a/src/current/v21.1/column-families.md b/src/current/v21.1/column-families.md deleted file mode 100644 index 8e1a7db2aed..00000000000 --- a/src/current/v21.1/column-families.md +++ /dev/null @@ -1,94 +0,0 @@ ---- -title: Column Families -summary: A column family is a group of columns in a table that are stored as a single key-value pair in the underlying key-value store. -toc: true ---- - -A column family is a group of columns in a table that are stored as a single key-value pair in the [underlying key-value store](architecture/storage-layer.html). Column families reduce the number of keys stored in the key-value store, resulting in improved performance during [`INSERT`](insert.html), [`UPDATE`](update.html), and [`DELETE`](delete.html) operations. - -This page explains how CockroachDB organizes columns into families as well as cases in which you might want to manually override the default behavior. - - [Secondary indexes](indexes.html) respect the column family definitions applied to tables. When you define a secondary index, CockroachDB breaks the secondary index key-value pairs into column families, according to the family and stored column configurations. - -## Default behavior - -When a table is created, all columns are stored as a single column family. - -This default approach ensures efficient key-value storage and performance in most cases. However, when frequently updated columns are grouped with seldom updated columns, the seldom updated columns are nonetheless rewritten on every update. Especially when the seldom updated columns are large, it's more performant to split them into a distinct family. - -## Manual override - -### Assign column families on table creation - -To manually assign a column family on [table creation](create-table.html), use the `FAMILY` keyword. - -For example, let's say we want to create a table to store an immutable blob of data (`data BYTES`) with a last accessed timestamp (`last_accessed TIMESTAMP`). Because we know that the blob of data will never get updated, we use the `FAMILY` keyword to break it into a separate column family: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE test ( - id INT PRIMARY KEY, - last_accessed TIMESTAMP, - data BYTES, - FAMILY f1 (id, last_accessed), - FAMILY f2 (data) -); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE test; -~~~ - -~~~ - table_name | create_statement --------------+------------------------------------------------- - test | CREATE TABLE test ( - | id INT8 NOT NULL, - | last_accessed TIMESTAMP NULL, - | data BYTES NULL, - | CONSTRAINT "primary" PRIMARY KEY (id ASC), - | FAMILY f1 (id, last_accessed), - | FAMILY f2 (data) - | ) -(1 row) -~~~ - - -### Assign column families when adding columns - -When using the [`ALTER TABLE .. ADD COLUMN`](add-column.html) statement to add a column to a table, you can assign the column to a new or existing column family. - -- Use the `CREATE FAMILY` keyword to assign a new column to a **new family**. For example, the following would add a `data2 BYTES` column to the `test` table above and assign it to a new column family: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER TABLE test ADD COLUMN data2 BYTES CREATE FAMILY f3; - ~~~ - -- Use the `FAMILY` keyword to assign a new column to an **existing family**. For example, the following would add a `name STRING` column to the `test` table above and assign it to family `f1`: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER TABLE test ADD COLUMN name STRING FAMILY f1; - ~~~ - -- Use the `CREATE IF NOT EXISTS FAMILY` keyword to assign a new column to an **existing family or, if the family doesn't exist, to a new family**. For example, the following would assign the new column to the existing `f1` family; if that family didn't exist, it would create a new family and assign the column to it: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER TABLE test ADD COLUMN name STRING CREATE IF NOT EXISTS FAMILY f1; - ~~~ - -- If a column is added to a table and the family is not specified, it will be added to the first column family. For example, the following would add the new column to the `f1` family, since that is the first column family: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER TABLE test ADD COLUMN last_name STRING; - ~~~ - -## See also - -- [`CREATE TABLE`](create-table.html) -- [`ADD COLUMN`](add-column.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/comment-on.md b/src/current/v21.1/comment-on.md deleted file mode 100644 index 8994313fe68..00000000000 --- a/src/current/v21.1/comment-on.md +++ /dev/null @@ -1,209 +0,0 @@ ---- -title: COMMENT ON -summary: The COMMENT ON statement associates comments to databases, tables, columns, or indexes. -toc: true ---- - -The `COMMENT ON` [statement](sql-statements.html) associates comments to [databases](create-database.html), [tables](create-table.html), [columns](add-column.html), or [indexes](indexes.html). - -## Required privileges - -The user must have the `CREATE` [privilege](authorization.html#assign-privileges) on the object they are commenting on. - -## Synopsis - -
{% include {{ page.version.version }}/sql/generated/diagrams/comment.html %}
- -## Parameters - - Parameter | Description -------------|-------------- -`database_name` | The name of the [database](create-database.html) on which you are commenting. -`table_name` | The name of the [table](create-table.html) on which you are commenting. -`column_name` | The name of the [column](add-column.html) on which you are commenting. -`table_index_name` | The name of the [index](indexes.html) on which you are commenting. -`comment_text` | The comment ([`STRING`](string.html)) you are associating to the object. You can remove a comment by replacing the string with `NULL`. - -## Examples - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### Add a comment to a database - -To add a comment to a database: - -{% include_cached copy-clipboard.html %} -~~~ sql -> COMMENT ON DATABASE movr IS 'This database contains information about users, vehicles, and rides.'; -~~~ - -To view database comments, use [`SHOW DATABASES`](show-databases.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW DATABASES WITH COMMENT; -~~~ - -~~~ - database_name | owner | primary_region | regions | survival_goal | comment -----------------+-------+----------------+---------+---------------+----------------------------------------------------------------------- - defaultdb | root | NULL | {} | NULL | NULL - movr | demo | NULL | {} | NULL | This database contains information about users, vehicles, and rides. - postgres | root | NULL | {} | NULL | NULL - system | node | NULL | {} | NULL | NULL -(4 rows) -~~~ - -### Add a comment to a table - -To add a comment to a table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> COMMENT ON TABLE vehicles IS 'This table contains information about vehicles registered with MovR.'; -~~~ - -To view table comments, use [`SHOW TABLES`](show-tables.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM movr WITH COMMENT; -~~~ - -~~~ - table_name | comment -+----------------------------+----------------------------------------------------------------------+ - users | - vehicles | This table contains information about vehicles registered with MovR. - rides | - vehicle_location_histories | - promo_codes | - user_promo_codes | -(6 rows) -~~~ - - You can also view comments on a table with [`SHOW CREATE`](show-create.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE vehicles; -~~~ - -~~~ - table_name | create_statement --------------+------------------------------------------------------------------------------------------------------ - vehicles | CREATE TABLE vehicles ( - | id UUID NOT NULL, - | city VARCHAR NOT NULL, - | type VARCHAR NULL, - | owner_id UUID NULL, - | creation_time TIMESTAMP NULL, - | status VARCHAR NULL, - | current_location VARCHAR NULL, - | ext JSONB NULL, - | CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - | CONSTRAINT fk_city_ref_users FOREIGN KEY (city, owner_id) REFERENCES users(city, id), - | INDEX vehicles_auto_index_fk_city_ref_users (city ASC, owner_id ASC), - | FAMILY "primary" (id, city, type, owner_id, creation_time, status, current_location, ext) - | ); - | COMMENT ON TABLE vehicles IS 'This table contains information about vehicles registered with MovR.' -(1 row) -~~~ - -### Add a comment to a column - -To add a comment to a column: - -{% include_cached copy-clipboard.html %} -~~~ sql -> COMMENT ON COLUMN users.credit_card IS 'This column contains user payment information.'; -~~~ - -To view column comments, use [`SHOW COLUMNS`](show-columns.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM users WITH COMMENT; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden | comment -+-------------+-----------+-------------+----------------+-----------------------+-----------+-----------+------------------------------------------------+ - id | UUID | false | NULL | | {primary} | false | NULL - city | VARCHAR | false | NULL | | {primary} | false | NULL - name | VARCHAR | true | NULL | | {} | false | NULL - address | VARCHAR | true | NULL | | {} | false | NULL - credit_card | VARCHAR | true | NULL | | {} | false | This column contains user payment information. -(5 rows) -~~~ - -### Add a comment to an index - -Suppose we [create an index](create-index.html) on the `name` column of the `users` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE INDEX ON users(name); -~~~ - -To add a comment to the index: - -{% include_cached copy-clipboard.html %} -~~~ sql -> COMMENT ON INDEX users_name_idx IS 'This index improves performance on queries that filter by name.'; -~~~ - -To view column comments, use [`SHOW INDEXES ... WITH COMMENT`](show-index.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW INDEXES FROM users WITH COMMENT; -~~~ - -~~~ - table_name | index_name | non_unique | seq_in_index | column_name | direction | storing | implicit | comment --------------+----------------+------------+--------------+-------------+-----------+---------+----------+------------------------------------------------------------------ - users | primary | false | 1 | city | ASC | false | false | NULL - users | primary | false | 2 | id | ASC | false | false | NULL - users | users_name_idx | true | 1 | name | ASC | false | false | This index improves performance on queries that filter by name. - users | users_name_idx | true | 2 | city | ASC | false | true | This index improves performance on queries that filter by name. - users | users_name_idx | true | 3 | id | ASC | false | true | This index improves performance on queries that filter by name. - users | primary | false | 1 | city | ASC | false | false | NULL - users | primary | false | 2 | id | ASC | false | false | NULL -... -(15 rows) -~~~ - -### Remove a comment from a database - -To remove a comment from a database: - -{% include_cached copy-clipboard.html %} -~~~ sql -> COMMENT ON DATABASE movr IS NULL; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW DATABASES WITH COMMENT; -~~~ - -~~~ - database_name | owner | primary_region | regions | survival_goal | comment -----------------+-------+----------------+---------+---------------+---------- - defaultdb | root | NULL | {} | NULL | NULL - movr | demo | NULL | {} | NULL | NULL - postgres | root | NULL | {} | NULL | NULL - system | node | NULL | {} | NULL | NULL -(4 rows) -~~~ - -## See also - -- [`CREATE DATABASE`](create-database.html) -- [`CREATE TABLE`](create-table.html) -- [`ADD COLUMN`](add-column.html) -- [`CREATE INDEX`](create-index.html) -- [`SHOW TABLES`](show-tables.html) -- [Other SQL Statements](sql-statements.html) -- [dBeaver](dbeaver.html) diff --git a/src/current/v21.1/commit-transaction.md b/src/current/v21.1/commit-transaction.md deleted file mode 100644 index 9661dfe743f..00000000000 --- a/src/current/v21.1/commit-transaction.md +++ /dev/null @@ -1,86 +0,0 @@ ---- -title: COMMIT -summary: Commit a transaction with the COMMIT statement in CockroachDB. -toc: true ---- - -The `COMMIT` [statement](sql-statements.html) commits the current [transaction](transactions.html) or, when using [advanced client-side transaction retries](advanced-client-side-transaction-retries.html), clears the connection to allow new transactions to begin. - -When using [advanced client-side transaction retries](advanced-client-side-transaction-retries.html), statements issued after [`SAVEPOINT`](savepoint.html) are committed when [`RELEASE SAVEPOINT`](release-savepoint.html) is issued instead of `COMMIT`. However, you must still issue a `COMMIT` statement to clear the connection for the next transaction. - -For non-retryable transactions, if statements in the transaction [generated any errors](transactions.html#error-handling), `COMMIT` is equivalent to `ROLLBACK`, which aborts the transaction and discards *all* updates made by its statements. - - -## Synopsis - -
-{% include {{ page.version.version }}/sql/generated/diagrams/commit_transaction.html %} -
- -## Required privileges - -No [privileges](authorization.html#assign-privileges) are required to commit a transaction. However, privileges are required for each statement within a transaction. - -## Aliases - -In CockroachDB, `END` is an alias for the `COMMIT` statement. - -## Example - -### Commit a transaction - -How you commit transactions depends on how your application handles [transaction retries](transactions.html#transaction-retries). - -#### Client-side retryable transactions - -When using [advanced client-side transaction retries](advanced-client-side-transaction-retries.html), statements are committed by [`RELEASE SAVEPOINT`](release-savepoint.html). `COMMIT` itself only clears the connection for the next transaction. - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SAVEPOINT cockroach_restart; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE products SET inventory = 0 WHERE sku = '8675309'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO orders (customer, sku, status) VALUES (1001, '8675309', 'new'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> RELEASE SAVEPOINT cockroach_restart; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> COMMIT; -~~~ - -{{site.data.alerts.callout_danger}}This example assumes you're using client-side intervention to handle transaction retries.{{site.data.alerts.end}} - -#### Automatically retried transactions - -If you are using transactions that CockroachDB will [automatically retry](transactions.html#automatic-retries) (i.e., all statements sent in a single batch), commit the transaction with `COMMIT`. - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; UPDATE products SET inventory = 100 WHERE = '8675309'; UPDATE products SET inventory = 100 WHERE = '8675310'; COMMIT; -~~~ - -## See also - -- [Transactions](transactions.html) -- [`BEGIN`](begin-transaction.html) -- [`RELEASE SAVEPOINT`](release-savepoint.html) -- [`ROLLBACK`](rollback-transaction.html) -- [`SAVEPOINT`](savepoint.html) -- [`SHOW SAVEPOINT STATUS`](show-savepoint-status.html) diff --git a/src/current/v21.1/common-errors.md b/src/current/v21.1/common-errors.md deleted file mode 100644 index 87f1320f456..00000000000 --- a/src/current/v21.1/common-errors.md +++ /dev/null @@ -1,198 +0,0 @@ ---- -title: Common Errors -summary: Understand and resolve common error messages written to stderr or logs. -toc: false ---- - -This page helps you understand and resolve error messages written to `stderr` or your [logs](logging-overview.html). - -| Topic | Message -|----------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -| Client connection | [`connection refused`](#connection-refused) -| Client connection | [`node is running secure mode, SSL connection required`](#node-is-running-secure-mode-ssl-connection-required) -| Transaction retries | [`restart transaction`](#restart-transaction) -| Node startup | [`node belongs to cluster but is attempting to connect to a gossip network for cluster `](#node-belongs-to-cluster-cluster-id-but-is-attempting-to-connect-to-a-gossip-network-for-cluster-another-cluster-id) -| Node configuration | [`clock synchronization error: this node is more than 500ms away from at least half of the known nodes`](#clock-synchronization-error-this-node-is-more-than-500ms-away-from-at-least-half-of-the-known-nodes) -| Node configuration | [`open file descriptor limit of is under the minimum required `](#open-file-descriptor-limit-of-number-is-under-the-minimum-required-number) -| Replication | [`replicas failing with "0 of 1 store with an attribute matching []; likely not enough nodes in cluster"`](#replicas-failing-with-0-of-1-store-with-an-attribute-matching-likely-not-enough-nodes-in-cluster) -| Split failed | [`split failed while applying backpressure; are rows updated in a tight loop?`](#split-failed-while-applying-backpressure-are-rows-updated-in-a-tight-loop) -| Deadline exceeded | [`context deadline exceeded`](#context-deadline-exceeded) -| Incremental backups | **New in v21.1:** [`protected ts verification error...`](#protected-ts-verification-error) -| Ambiguous results | [`result is ambiguous`](#result-is-ambiguous) -| Import key collision | [`checking for key collisions: ingested key collides with an existing one`](#checking-for-key-collisions-ingested-key-collides-with-an-existing-one) | - -## connection refused - -This message indicates a client is trying to connect to a node that is either not running or is not listening on the specified interfaces (i.e., hostname or port). - -To resolve this issue, do one of the following: - -- If the node hasn't yet been started, [start the node](cockroach-start.html). -- If you specified a [`--listen-addr` and/or a `--advertise-addr` flag](cockroach-start.html#networking) when starting the node, you must include the specified IP address/hostname and port with all other [`cockroach` commands](cockroach-commands.html) or change the `COCKROACH_HOST` environment variable. - -If you're not sure what the IP address/hostname and port values might have been, you can look in the node's [logs](logging-overview.html). If necessary, you can also terminate the `cockroach` process and then restart the node: - -{% include {{ page.version.version }}/prod-deployment/node-shutdown.md %} - -Then restart the node: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start [flags] -~~~ - -## node is running secure mode, SSL connection required - -This message indicates that the cluster is using TLS encryption to protect network communication, and the client is trying to open a connection without using the required TLS certificates. - -To resolve this issue, use the [`cockroach cert create-client`](cockroach-cert.html) command to generate a client certificate and key for the user trying to connect. For a secure deployment tutorial, including generating security certificates and connecting clients, see [Manual Deployment](manual-deployment.html). - -## restart transaction - -Messages with the error code `40001` and the string `restart transaction` indicate that a transaction failed because it conflicted with another concurrent or recent transaction accessing the same data. The transaction needs to be retried by the client. For more information about how to implement client-side retries, see [client-side retry handling](transactions.html#client-side-intervention). - -For more information about the different types of transaction retry errors such as "retry write too old", "read within uncertainty interval", etc., see the [Transaction Retry Error Reference](transaction-retry-error-reference.html). - -## node belongs to cluster \ but is attempting to connect to a gossip network for cluster \ - -This message usually indicates that a node tried to connect to a cluster, but the node is already a member of a different cluster. This is determined by metadata in the node's data directory. To resolve this issue, do one of the following: - -- Choose a different directory to store the CockroachDB data: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start [flags] --store=[new directory] --join=[cluster host]:26257 - ~~~ - -- Remove the existing directory and start a node joining the cluster again: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ rm -r cockroach-data/ - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start [flags] --join=[cluster host]:26257 - ~~~ - -## clock synchronization error: this node is more than 500ms away from at least half of the known nodes - -This error indicates that a node has spontaneously shut down because it detected that its clock is out of sync with at least half of the other nodes in the cluster by 80% of the maximum offset allowed (500ms by default). CockroachDB requires moderate levels of [clock synchronization](recommended-production-settings.html#clock-synchronization) to preserve data consistency, so the node shutting down in this way avoids the risk of consistency anomalies. - -To prevent this from happening, you should run clock synchronization software on each node. For guidance on synchronizing clocks, see the tutorial for your deployment environment: - -Environment | Recommended Approach -------------|--------------------- -[Manual](deploy-cockroachdb-on-premises.html#step-1-synchronize-clocks) | Use NTP with Google's external NTP service. -[AWS](deploy-cockroachdb-on-aws.html#step-3-synchronize-clocks) | Use the Amazon Time Sync Service. -[Azure](deploy-cockroachdb-on-microsoft-azure.html#step-3-synchronize-clocks) | Disable Hyper-V time synchronization and use NTP with Google's external NTP service. -[Digital Ocean](deploy-cockroachdb-on-digital-ocean.html#step-2-synchronize-clocks) | Use NTP with Google's external NTP service. -[GCE](deploy-cockroachdb-on-google-cloud-platform.html#step-3-synchronize-clocks) | Use NTP with Google's internal NTP service. - -## open file descriptor limit of \ is under the minimum required \ - -CockroachDB can use a large number of open file descriptors, often more than is available by default. This message indicates that the machine on which a CockroachDB node is running is under CockroachDB's recommended limits. - -For more details on CockroachDB's file descriptor limits and instructions on increasing the limit on various platforms, see [File Descriptors Limit](recommended-production-settings.html#file-descriptors-limit). - -## replicas failing with "0 of 1 store with an attribute matching []; likely not enough nodes in cluster - -### When running a single-node cluster - -When running a single-node CockroachDB cluster, an error about replicas failing will eventually show up in the node's log files, for example: - -~~~ shell -E160407 09:53:50.337328 storage/queue.go:511 [replicate] 7 replicas failing with "0 of 1 store with an attribute matching []; likely not enough nodes in cluster" -~~~ - -This happens because CockroachDB expects three nodes by default. If you do not intend to add additional nodes, you can stop this error by using [`ALTER RANGE ... CONFIGURE ZONE`](configure-zone.html) to update your default zone configuration to expect only one node: - -{% include_cached copy-clipboard.html %} -~~~ shell -# Insecure cluster: -$ cockroach sql --execute="ALTER RANGE default CONFIGURE ZONE USING num_replicas=1;" --insecure -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -# Secure cluster: -$ cockroach sql --execute="ALTER RANGE default CONFIGURE ZONE USING num_replicas=1;" --certs-dir=[path to certs directory] -~~~ - -The zone's replica count is reduced to 1. For more information, see [`ALTER RANGE ... CONFIGURE ZONE`](configure-zone.html) and [Configure Replication Zones](configure-replication-zones.html). - -### When running a multi-node cluster - -When running a multi-node CockroachDB cluster, if you see an error like the one above about replicas failing, some nodes might not be able to talk to each other. For recommended actions, see [Cluster Setup Troubleshooting](cluster-setup-troubleshooting.html#replication-issues). - -## split failed while applying backpressure; are rows updated in a tight loop? - -In CockroachDB, a table row is stored on disk as a key-value pair. Whenever the row is updated, CockroachDB also stores a distinct version of the key-value pair to enable concurrent request processing while guaranteeing consistency (see [multi-version concurrency control (MVCC)](architecture/storage-layer.html#mvcc)). All versions of a key-value pair belong to a larger ["range"](architecture/overview.html#terms) of the total key space, and the historical versions remain until the garbage collection period defined by the `gc.ttlseconds` variable in the applicable [zone configuration](configure-replication-zones.html#gc-ttlseconds) has passed (25 hours by default). Once a range reaches a size threshold (512 MiB by default), CockroachDB splits the range into two ranges. However, this message indicates that a range cannot be split as intended. - -One possible cause is that the range consists only of MVCC version data due to a row being repeatedly updated, and the range cannot be split because doing so would spread MVCC versions for a single row across multiple ranges. - -To resolve this issue, make sure you are not repeatedly updating a single row. If frequent updates of a row are necessary, consider one of the following: - -- Reduce the `gc.ttlseconds` variable in the applicable [zone configuration](configure-replication-zones.html#gc-ttlseconds) to reduce the garbage collection period and prevent such a large build-up of historical values. -- If a row contains large columns that are not being updated with other columns, put the large columns in separate [column families](column-families.html). - -## context deadline exceeded - -This message occurs when a component of CockroachDB gives up because it was relying on another component that has not behaved as expected, for example, another node dropped a network connection. To investigate further, look in the node's logs for the primary failure that is the root cause. - -## protected ts verification error... - -{% include_cached new-in.html version="v21.1" %} Messages that begin with `protected ts verification error…` indicate that your [incremental backup](take-full-and-incremental-backups.html#incremental-backups) failed because the data you are trying to backup was garbage collected. This happens when incremental backups are taken less frequently than the garbage collection periods for any of the objects in the base backup. For example, if your incremental backups recur daily, but the garbage collection period of one table in your backup is less than one day, all of your incremental backups will fail. - -The error message will specify which part of your backup is causing the failure. For example, `range span: /Table/771` indicates that table `771` is part of the problem. You can then inspect this table by running [`SELECT * FROM crdb_internal.tables WHERE id=771`](select-clause.html). You can also run [`SHOW ZONE CONFIGURATIONS`](show-zone-configurations.html) and look for any `gc.ttlseconds` values that are set lower than your incremental backup frequency. - -To resolve this issue, take a new [full backup](take-full-and-incremental-backups.html) after doing either of the following: - -- Increase the garbage collection period by [configuring the `gc.ttlseconds` replication zone variable](configure-replication-zones.html#gc-ttlseconds), or -- [Increase the frequency of incremental backups](manage-a-backup-schedule.html). - -## result is ambiguous - -In a distributed system, some errors can have ambiguous results. For -example, if you receive a `connection closed` error while processing a -`COMMIT` statement, you cannot tell whether the transaction -successfully committed or not. These errors are possible in any -database, but CockroachDB is somewhat more likely to produce them than -other databases because ambiguous results can be caused by failures -between the nodes of a cluster. These errors are reported with the -PostgreSQL error code `40003` (`statement_completion_unknown`) and the -message `result is ambiguous`. - -Ambiguous errors can be caused by nodes crashing, network failures, or -timeouts. If you experience a lot of these errors when things are -otherwise stable, look for performance issues. Note that ambiguity is -only possible for the last statement of a transaction (`COMMIT` or -`RELEASE SAVEPOINT`) or for statements outside a transaction. If a connection drops during a transaction that has not yet tried to commit, the transaction will definitely be aborted. - -In general, you should handle ambiguous errors the same way as -`connection closed` errors. If your transaction is -[idempotent](https://en.wikipedia.org/wiki/Idempotence#Computer_science_meaning), -it is safe to retry it on ambiguous errors. `UPSERT` operations are -typically idempotent, and other transactions can be written to be -idempotent by verifying the expected state before performing any -writes. Increment operations such as `UPDATE my_table SET x=x+1 WHERE -id=$1` are typical examples of operations that cannot easily be made -idempotent. If your transaction is not idempotent, then you should -decide whether to retry or not based on whether it would be better for -your application to apply the transaction twice or return an error to -the user. - -## checking for key collisions: ingested key collides with an existing one - -When importing into an existing table with [`IMPORT INTO`](import-into.html), this error occurs because the rows in the import file conflict with an existing primary key or another [`UNIQUE`](unique.html) constraint on the table. The import will fail as a result. `IMPORT INTO` is an insert-only statement, so you cannot use it to update existing rows. To update rows in an existing table, use the [`UPDATE`](update.html) statement. - -## Something else? - -Try searching the rest of our docs for answers or using our other [support resources](support-resources.html), including: - -- [CockroachDB Community Forum](https://forum.cockroachlabs.com) -- [CockroachDB Community Slack](https://cockroachdb.slack.com) -- [StackOverflow](http://stackoverflow.com/questions/tagged/cockroachdb) -- [CockroachDB Support Portal](https://support.cockroachlabs.com) -- [Transaction retry error reference](transaction-retry-error-reference.html) diff --git a/src/current/v21.1/common-table-expressions.md b/src/current/v21.1/common-table-expressions.md deleted file mode 100644 index 7e1a6f0fbd4..00000000000 --- a/src/current/v21.1/common-table-expressions.md +++ /dev/null @@ -1,319 +0,0 @@ ---- -title: WITH Queries (Common Table Expressions) -summary: Common table expressions (CTEs) simplify the definition and use of subqueries -toc: true ---- - -`WITH` queries, also called *common table expressions* or CTEs, provide a shorthand name to a possibly complex [subquery](subqueries.html) before it is used in a larger query context. This improves the readability of SQL code. - -CTEs can be used in combination with [`SELECT` clauses](select-clause.html) and [`INSERT`](insert.html), [`DELETE`](delete.html), [`UPDATE`](update.html) and [`UPSERT`](upsert.html) statements. - - -## Synopsis - -
{% include {{ page.version.version }}/sql/generated/diagrams/with_clause.html %}
- -
- -## Parameters - -Parameter | Description -----------|------------ -`table_alias_name` | The name to use to refer to the common table expression from the accompanying query or statement. -`name` | A name for one of the columns in the newly defined common table expression. -`preparable_stmt` | The statement or subquery to use as common table expression. -`MATERIALIZED`/`NOT MATERIALIZED` | Override the [optimizer](cost-based-optimizer.html)'s decision to materialize (i.e., store the results) of the common table expression. By default, the optimizer materializes the common table expression if it affects other objects in the database, or if it is used in the query multiple times. - -## Overview - -{{site.data.alerts.callout_info}} -The examples on this page use MovR, a fictional vehicle-sharing application, to demonstrate CockroachDB SQL statements. To follow along, run [`cockroach demo`](cockroach-demo.html) from the command line to start a temporary, in-memory cluster with the `movr` dataset preloaded. - -For more information about the MovR example application and dataset, see [MovR: A Global Vehicle-sharing App](movr.html). -{{site.data.alerts.end}} - -A query or statement of the form `WITH x AS y IN z` creates the -temporary table name `x` for the results of the subquery `y`, to be -reused in the context of the query `z`. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> WITH r AS (SELECT * FROM rides WHERE revenue > 98) - SELECT * FROM users AS u, r WHERE r.rider_id = u.id; -~~~ - -~~~ - id | city | name | address | credit_card | id | city | vehicle_city | rider_id | vehicle_id | start_address | end_address | start_time | end_time | revenue ----------------------------------------+---------------+------------------+--------------------------------+-------------+--------------------------------------+---------------+---------------+--------------------------------------+--------------------------------------+-----------------------------------+---------------------------+---------------------------+---------------------------+---------- - ae147ae1-47ae-4800-8000-000000000022 | amsterdam | Tyler Dalton | 88194 Angela Gardens Suite 94 | 4443538758 | bbe76c8b-4395-4000-8000-00000000016f | amsterdam | amsterdam | ae147ae1-47ae-4800-8000-000000000022 | aaaaaaaa-aaaa-4800-8000-00000000000a | 45295 Brewer View Suite 52 | 62188 Jade Causeway | 2018-12-17 03:04:05+00:00 | 2018-12-17 13:04:05+00:00 | 99.00 - c7ae147a-e147-4000-8000-000000000027 | paris | Tina Miller | 97521 Mark Extensions | 8880478663 | d5810624-dd2f-4800-8000-0000000001a1 | paris | paris | c7ae147a-e147-4000-8000-000000000027 | cccccccc-cccc-4000-8000-00000000000c | 47713 Reynolds Mountains Suite 39 | 1417 Stephanie Villages | 2018-12-17 03:04:05+00:00 | 2018-12-18 22:04:05+00:00 | 99.00 - 75c28f5c-28f5-4400-8000-000000000017 | san francisco | William Wood | 36021 Steven Cove Apt. 89 | 5669281259 | 8ac08312-6e97-4000-8000-00000000010f | san francisco | san francisco | 75c28f5c-28f5-4400-8000-000000000017 | 77777777-7777-4800-8000-000000000007 | 84407 Tony Crest | 55336 Jon Manors | 2018-12-10 03:04:05+00:00 | 2018-12-11 13:04:05+00:00 | 99.00 - 8a3d70a3-d70a-4000-8000-00000000001b | san francisco | Jessica Martinez | 96676 Jennifer Knolls Suite 91 | 1601930189 | 7d70a3d7-0a3d-4000-8000-0000000000f5 | san francisco | san francisco | 8a3d70a3-d70a-4000-8000-00000000001b | 77777777-7777-4800-8000-000000000007 | 78978 Stevens Ramp Suite 8 | 7340 Alison Field Apt. 44 | 2018-12-19 03:04:05+00:00 | 2018-12-21 10:04:05+00:00 | 99.00 - 47ae147a-e147-4000-8000-00000000000e | washington dc | Patricia Herrera | 80588 Perez Camp | 6812041796 | 4083126e-978d-4000-8000-00000000007e | washington dc | washington dc | 47ae147a-e147-4000-8000-00000000000e | 44444444-4444-4400-8000-000000000004 | 33055 Julie Dale Suite 93 | 17280 Jill Drives | 2019-01-01 03:04:05+00:00 | 2019-01-01 14:04:05+00:00 | 99.00 -(5 rows) -~~~ - -In this example, the `WITH` clause defines the temporary name `r` for -the subquery over `rides`, and that name becomes a valid table name -for use in any [table expression](table-expressions.html) of the -subsequent `SELECT` clause. - -This query is equivalent to, but arguably simpler to read than: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM users AS u, (SELECT * FROM rides WHERE revenue > 98) AS r - WHERE r.rider_id = u.id; -~~~ - -It is also possible to define multiple common table expressions -simultaneously with a single `WITH` clause, separated by commas. Later -subqueries can refer to earlier subqueries by name. For example, the -following query is equivalent to the two examples above: - -{% include_cached copy-clipboard.html %} -~~~ sql -> WITH r AS (SELECT * FROM rides WHERE revenue > 98), - results AS (SELECT * FROM users AS u, r WHERE r.rider_id = u.id) - SELECT * FROM results; -~~~ - -In this example, the second CTE `results` refers to the first CTE `r` -by name. The final query refers to the CTE `results`. - -## Nested `WITH` clauses - -It is possible to use a `WITH` clause in a subquery, or even a `WITH` clause within another `WITH` clause. For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> WITH u AS - (SELECT * FROM - (WITH u_tab AS (SELECT * FROM users) SELECT * FROM u_tab)) - SELECT * FROM u; -~~~ - -When analyzing [table expressions](table-expressions.html) that -mention a CTE name, CockroachDB will choose the CTE definition that is -closest to the table expression. For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> WITH - u AS (SELECT * FROM users), - v AS (WITH u AS (SELECT * from vehicles) SELECT * FROM u) - SELECT * FROM v; -~~~ - -In this example, the inner subquery `SELECT * FROM v` will select from -table `vehicles` (closest `WITH` clause), not from table `users`. - -{{site.data.alerts.callout_info}} - CockroachDB does not support nested `WITH` clauses containing [data-modifying statements](#data-modifying-statements). `WITH` clauses containing data-modifying statements must be at the top level of the query. -{{site.data.alerts.end}} - -## Data-modifying statements - -It is possible to use a [data-modifying statement](sql-statements.html#data-manipulation-statements) (`INSERT`, `DELETE`, -etc.) as a common table expression, as long as the `WITH` clause containing the data-modifying statement is at the top level of the query. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> WITH final_code AS - (INSERT INTO promo_codes(code, description, rules) - VALUES ('half_off', 'Half-price ride!', '{"type": "percent_discount", "value": "50%"}'), ('free_ride', 'Free ride!', '{"type": "percent_discount", "value": "100%"}') - returning rules) - SELECT rules FROM final_code; -~~~ - -~~~ - rules -+-----------------------------------------------+ - {"type": "percent_discount", "value": "50%"} - {"type": "percent_discount", "value": "100%"} -(2 rows) -~~~ - -If the `WITH` clause containing the data-modifying statement is at a lower level, the statement results in an error: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT (WITH final_code AS - (INSERT INTO promo_codes(code, description, rules) - VALUES ('half_off', 'Half-price ride!', '{"type": "percent_discount", "value": "50%"}'), ('free_ride', 'Free ride!', '{"type": "percent_discount", "value": "100%"}') - returning rules) - SELECT rules FROM final_code); -~~~ - -~~~ -ERROR: WITH clause containing a data-modifying statement must be at the top level -SQLSTATE: 0A000 -~~~ - -{{site.data.alerts.callout_info}} -If a common table expression contains -a data-modifying statement (INSERT, DELETE, -etc.), the modifications are performed fully even if only part -of the results are used, e.g., with LIMIT. See Data -Writes in Subqueries for details. -{{site.data.alerts.end}} - -## Reusing common table expressions - -You can reference a CTE multiple times in a single query, using a `WITH` operator. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> WITH - users_ny AS (SELECT name, id FROM users WHERE city='new york'), - vehicles_ny AS (SELECT type, id, owner_id FROM vehicles WHERE city='new york') - SELECT * FROM users_ny JOIN vehicles_ny ON users_ny.id = vehicles_ny.owner_id; -~~~ - -~~~ - name | id | type | id | owner_id -+------------------+--------------------------------------+------------+--------------------------------------+--------------------------------------+ - James Hamilton | 051eb851-eb85-4ec0-8000-000000000001 | skateboard | 00000000-0000-4000-8000-000000000000 | 051eb851-eb85-4ec0-8000-000000000001 - Catherine Nelson | 147ae147-ae14-4b00-8000-000000000004 | scooter | 11111111-1111-4100-8000-000000000001 | 147ae147-ae14-4b00-8000-000000000004 -(2 rows) -~~~ - -In this single query, you define two CTE's and then reuse them in a table join. - -## Recursive common table expressions - - CockroachDB supports [recursive common table expressions](https://en.wikipedia.org/wiki/Hierarchical_and_recursive_queries_in_SQL#Common_table_expression). Recursive common table expressions are common table expressions that contain subqueries that refer to their own output. - -Recursive CTE definitions take the following form: - -~~~ sql -WITH RECURSIVE () AS ( - - UNION ALL - -) - -~~~ - -To write a recursive CTE: - -1. Add the `RECURSIVE` keyword directly after the `WITH` operator in the CTE definition, and before the CTE name. -1. Define an initial, non-recursive subquery. This subquery defines the initial values of the CTE. -1. Add the `UNION ALL` keyword after the initial subquery. -1. Define a recursive subquery that references its own output. This subquery can also reference the CTE name, unlike the initial subquery. -1. Write a parent query that evaluates the results of the CTE. - -CockroachDB evaluates recursive CTEs as follows: - -1. The initial query is evaluated. Its results are stored to rows in the CTE and copied to a temporary, working table. This working table is updated across iterations of the recursive subquery. -1. The recursive subquery is evaluated iteratively on the contents of the working table. The results of each iteration replace the contents of the working table. The results are also stored to rows of the CTE. The recursive subquery iterates until no results are returned. - -{{site.data.alerts.callout_info}} -Recursive subqueries must eventually return no results, or the query will run indefinitely. -{{site.data.alerts.end}} - -For example, the following recursive CTE calculates the factorial of the numbers 0 through 9: - -{% include_cached copy-clipboard.html %} -~~~ sql -WITH RECURSIVE cte (n, factorial) AS ( - VALUES (0, 1) -- initial subquery - UNION ALL - SELECT n+1, (n+1)*factorial FROM cte WHERE n < 9 -- recursive subquery -) -SELECT * FROM cte; -~~~ - -~~~ - n | factorial -+---+-----------+ - 0 | 1 - 1 | 1 - 2 | 2 - 3 | 6 - 4 | 24 - 5 | 120 - 6 | 720 - 7 | 5040 - 8 | 40320 - 9 | 362880 -(10 rows) -~~~ - -The initial subquery (`VALUES (0, 1)`) initializes the working table with the values `0` for the `n` column and `1` for the `factorial` column. The recursive subquery (`SELECT n+1, (n+1)*factorial FROM cte WHERE n < 9`) evaluates over the initial values of the working table and replaces its contents with the results. It then iterates over the contents of the working table, replacing its contents at each iteration, until `n` reaches `9`, when the [`WHERE` clause](select-clause.html#filter-rows) evaluates as false. - -If no `WHERE` clause were defined in the example, the recursive subquery would always return results and loop indefinitely, resulting in an error: - -{% include_cached copy-clipboard.html %} -~~~ sql -WITH RECURSIVE cte (n, factorial) AS ( - VALUES (0, 1) -- initial subquery - UNION ALL - SELECT n+1, (n+1)*factorial FROM cte -- recursive subquery with no WHERE clause -) -SELECT * FROM cte; -~~~ - -~~~ -ERROR: integer out of range -SQLSTATE: 22003 -~~~ - -If you are unsure if your recursive subquery will loop indefinitely, you can limit the results of the CTE with the [`LIMIT`](limit-offset.html) keyword. - -For example, if we remove the `WHERE` clause from the factorial example, we can use `LIMIT` to limit the results and avoid the `integer out of range` error: - -{% include_cached copy-clipboard.html %} -~~~ sql -WITH RECURSIVE cte (n, factorial) AS ( - VALUES (0, 1) -- initial subquery - UNION ALL - SELECT n+1, (n+1)*factorial FROM cte -- recursive subquery -) -SELECT * FROM cte LIMIT 10; -~~~ - -~~~ - n | factorial -+---+-----------+ - 0 | 1 - 1 | 1 - 2 | 2 - 3 | 6 - 4 | 24 - 5 | 120 - 6 | 720 - 7 | 5040 - 8 | 40320 - 9 | 362880 -(10 rows) -~~~ - -While this practice works for testing and debugging, we do not recommend it in production. - -{{site.data.alerts.callout_info}} -CockroachDB does not currently support the [Postgres recursive CTE variant](https://www.postgresql.org/docs/10/queries-with.html) with the keyword `UNION`. -{{site.data.alerts.end}} - -## Known limitations - -### Correlated common table expressions - -{% include {{ page.version.version }}/known-limitations/correlated-ctes.md %} - -For details, see the [tracking issue](https://github.com/cockroachdb/cockroach/issues/42540). - -## See also - -- [Subqueries](subqueries.html) -- [Selection Queries](selection-queries.html) -- [Table Expressions](table-expressions.html) -- [`EXPLAIN`](explain.html) diff --git a/src/current/v21.1/community-tooling.md b/src/current/v21.1/community-tooling.md deleted file mode 100644 index 2e472fcdf91..00000000000 --- a/src/current/v21.1/community-tooling.md +++ /dev/null @@ -1,79 +0,0 @@ ---- -title: Third-Party Tools Supported by the Community -summary: Learn about third-party software that works with CockroachDB. -toc: true -docs_area: reference.third_party_support ---- - -The following tools have been tested or developed by the CockroachDB community, but are not officially supported by Cockroach Labs. - -If you encounter problems with using these tools, please contact the maintainer of the tool with details. - -{{site.data.alerts.callout_success}} -If you have a tested or developed a third-party tool with CockroachDB, and would like it listed on this page, please [open a pull request to our docs GitHub repository](https://github.com/cockroachdb/docs/edit/master/v21.2/community-tooling.md). -{{site.data.alerts.end}} - -## Drivers and data access frameworks - -### C++ - -- [libpqxx](https://github.com/cockroachdb/community-tooling-samples/tree/main/cxx) - -### Go - -- [sqlx](http://jmoiron.github.io/sqlx/) - -### Java - -- [JDBI](https://jdbi.org/) -- [clojure.java.jdbc](https://github.com/cockroachdb/community-tooling-samples/tree/main/clojure) - -### PHP - -- [php-pgsql](https://github.com/cockroachdb/community-tooling-samples/tree/main/php) - -### PowerShell - -- [Npgsql](https://blog.ervits.com/2020/03/exploring-cockroachdb-with-jupyter.html) - -### R - -- [RPostgres](https://blog.ervits.com/2020/02/exploring-cockroachdb-with-r-and.html) - -### Rust - -- [tokio_postgres](https://docs.rs/tokio-postgres/latest/tokio_postgres) - -### Other - -- [Apache Hop (Incubating)](https://hop.apache.org) - -## Visualization tools - -- [Beekeeper Studio](https://www.beekeeperstudio.io/db/cockroachdb-client/) -- [DbVisualizer](https://www.cdata.com/kb/tech/cockroachdb-jdbc-dbv.rst) -- [Navicat for PostgreSQL](https://www.navicat.com/en/products/navicat-for-postgresql)/[Navicat Premium](https://www.navicat.com/en/products/navicat-premium) -- [Pgweb](http://sosedoff.github.io/pgweb/) -- [Postico](https://eggerapps.at/postico/) -- [TablePlus](https://tableplus.com/blog/2018/06/best-cockroachdb-gui-client-tableplus.html) - -## Schema migration tools - -- [SchemaHero](https://schemahero.io/databases/cockroachdb/connecting/) -- [DbUp](https://github.com/DbUp/DbUp/issues/464#issuecomment-895503849) -- [golang-migrate](https://github.com/golang-migrate/migrate/tree/master/database/cockroachdb) -- [db-migrate](https://db-migrate.readthedocs.io/en/latest/) - -## Connection pooling tools - -- [PGBouncer](https://dzone.com/articles/using-pgbouncer-with-cockroachdb) - -## IAM tools - -- [Vault](https://www.vaultproject.io/docs/configuration/storage/cockroachdb) - -## See also - -- [Build an App with CockroachDB](example-apps.html) -- [Install a Postgres Client](install-client-drivers.html) -- [Third-Party Tools Supported by Cockroach Labs](third-party-database-tools.html) diff --git a/src/current/v21.1/computed-columns.md b/src/current/v21.1/computed-columns.md deleted file mode 100644 index 73340fba679..00000000000 --- a/src/current/v21.1/computed-columns.md +++ /dev/null @@ -1,97 +0,0 @@ ---- -title: Use Computed Columns -summary: A computed column exposes data generated by an expression included in the column definition. -toc: true ---- - -A computed column exposes data generated from other columns by a [scalar expression](scalar-expressions.html) included in the column definition. A stored computed column (set with the `STORED` SQL keyword) is calculated when a row is inserted or updated, and stores the resulting value of the scalar expression in the primary index similar to a regular column. A virtual computed column (set with the `VIRTUAL` SQL keyword) is not stored, and the value of the scalar expression is computed during queries as needed. - -## Why use computed columns? - -Computed columns are especially useful when used with [`JSONB`](jsonb.html) columns or [secondary indexes](indexes.html). - -- **JSONB** columns are used for storing semi-structured `JSONB` data. When the table's primary information is stored in `JSONB`, it's useful to index a particular field of the `JSONB` document. In particular, computed columns allow for the following use case: a two-column table with a `PRIMARY KEY` column and a `payload` column, whose primary key is computed as some field from the `payload` column. This alleviates the need to manually separate your primary keys from your JSON blobs. For more information, see the [`JSONB` example](#create-a-table-with-a-jsonb-column-and-a-stored-computed-column) below. - -- **Secondary indexes** can be created on computed columns, which is especially useful when a table is frequently sorted. See the [secondary indexes example](#create-a-table-with-a-secondary-index-on-a-computed-column) below. - -## Considerations - -Computed columns: - -- Cannot be used to generate other computed columns. -- Behave like any other column, with the exception that they cannot be written to directly. -- Are mutually exclusive with [`DEFAULT`](default-value.html). - -Virtual computed columns: - -- Are not stored in the table's primary index. -- Are recomputed as the column data in the expression changes. -- Cannot be used as part of a primary key, `FAMILY` definition, in `CHECK` constraints, or `FOREIGN KEY` constraints. -- Cannot be a [foreign key](foreign-key.html) reference. -- Cannot be stored in indexes. -- Can be index columns. - -Once a computed column is created, you cannot directly alter the formula. To make modifications to a computed column's formula, see the [example](#alter-the-formula-for-a-computed-column) below. - -## Creation - -To define a stored computed column, use the following syntax: - -~~~ -column_name AS () STORED -~~~ - -{% include_cached new-in.html version="v21.1" %} To define a virtual computed column, use the following syntax: - -~~~ -column_name AS () VIRTUAL -~~~ - -Parameter | Description -----------|------------ -`column_name` | The [name/identifier](keywords-and-identifiers.html#identifiers) of the computed column. -`` | The [data type](data-types.html) of the computed column. -`` | The immutable [scalar expression](scalar-expressions.html) used to compute column values. You cannot use any functions, such as `now()` or `nextval()`, that are not immutable. -`STORED` | _(Required for stored computed columns)_ The computed column is stored alongside other columns. -`VIRTUAL`| _(Required for virtual columns)_ The computed column is virtual, meaning the column data is not stored in the table's primary index. - - For compatibility with PostgresSQL, CockroachDB also supports creating computed columns with the syntax `column_name GENERATED ALWAYS AS () STORED`. - -## Examples - -### Create a table with a stored computed column - -{% include {{ page.version.version }}/computed-columns/simple.md %} - -### Create a table with a `JSONB` column and a stored computed column - -{% include {{ page.version.version }}/computed-columns/jsonb.md %} - -### Create a virtual computed column using `JSONB` data - -{% include {{ page.version.version }}/computed-columns/virtual.md %} - -### Create a table with a secondary index on a computed column - -{% include {{ page.version.version }}/computed-columns/secondary-index.md %} - -### Add a computed column to an existing table - -{% include {{ page.version.version }}/computed-columns/add-computed-column.md %} - -For more information, see [`ADD COLUMN`](add-column.html). - -### Convert a computed column into a regular column - -{% include {{ page.version.version }}/computed-columns/convert-computed-column.md %} - -### Alter the formula for a computed column - -{% include {{ page.version.version }}/computed-columns/alter-computed-column.md %} - -## See also - -- [Scalar Expressions](scalar-expressions.html) -- [Information Schema](information-schema.html) -- [`CREATE TABLE`](create-table.html) -- [`JSONB`](jsonb.html) diff --git a/src/current/v21.1/configure-logs.md b/src/current/v21.1/configure-logs.md deleted file mode 100644 index 11c2a58094c..00000000000 --- a/src/current/v21.1/configure-logs.md +++ /dev/null @@ -1,515 +0,0 @@ ---- -title: Configure logs -summary: How to configure CockroachDB logs with the --log or --log-config-file flag and YAML payload. -toc: true ---- - -This page describes how to configure CockroachDB logs with the [`--log` or `log-config-file` flag](cockroach-start.html#logging) and a [YAML payload](#yaml-payload). Most logging behaviors are configurable, including: - -- The [log sinks](#configure-log-sinks) that output logs to different locations, including over the network. -- The [logging channels](logging-overview.html#logging-channels) that are mapped to each sink. -- The [format](log-formats.html) used by the log messages. -- The [redaction](#redact-logs) of log messages. - -For examples of how these settings can be used in practice, see [Logging Use Cases](logging-use-cases.html). - -{{site.data.alerts.callout_info}} -The logging flags previously used with `cockroach` commands are now deprecated. Instead, use the YAML definitions as described below. The [default logging configuration](#default-logging-configuration) uses YAML settings that are backward-compatible with the log files used in v20.2 and earlier. -{{site.data.alerts.end}} - -## Flag - -To configure the logging behavior of a `cockroach` command, include one of these flags with the command: - -- `--log={yaml}`, where `{yaml}` is the [YAML payload](#yaml-payload) -- `--log-config-file={yaml-file}`, where `{yaml-file}` is the path to a YAML file - -{{site.data.alerts.callout_success}} -All [`cockroach` commands](cockroach-commands.html) support logging and can be configured with `--log` or `--log-config-file`. However, note that most messages related to cluster operation are generated by [`cockroach start`](cockroach-start.html) or [`cockroach start-single-node`](cockroach-start-single-node.html). Other commands generate messages related to their own execution, which are mainly useful when troubleshooting the behaviors of those commands. -{{site.data.alerts.end}} - -## YAML payload - -All log settings for a `cockroach` command are specified with a YAML payload in one of the following formats: - -- Block format, where each parameter is written on a separate line. For example, after creating a file `logs.yaml`, pass the YAML values with either `--log-config-file` or `--log`: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start-single-node --certs-dir=certs --log-config-file=logs.yaml - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start-single-node --certs-dir=certs --log="$(cat logs.yaml)" - ~~~ - -- Inline format, where all parameters are specified on one line. For example, to generate an `ops` log file that collects the `OPS` and `HEALTH` channels (overriding the file groups defined for those channels in the [default configuration](#default-logging-configuration)): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start-single-node --certs-dir=certs --log="sinks: {file-groups: {ops: {channels: [OPS, HEALTH]}}}" - ~~~ - - Note that the inline spaces must be preserved. - -For clarity, this article uses the block format to describe the YAML payload, which has the overall structure: - -~~~ yaml -file-defaults: ... # defaults inherited by file sinks -fluent-defaults: ... # defaults inherited by network sinks -sinks: - file-groups: ... # file sink definitions - fluent-servers: ... # network sink definitions - stderr: ... # stderr sink definitions -capture-stray-errors: ... # parameters for the stray error capture system -~~~ - -{{site.data.alerts.callout_info}} -Providing a logging configuration is optional. Any fields included in the YAML payload will override the same fields in the [default logging configuration](#default-logging-configuration). -{{site.data.alerts.end}} - -{{site.data.alerts.callout_success}} -You can view your current settings by running `cockroach debug check-log-config`, which returns the YAML definitions and a URL to a visualization of the current logging configuration. -{{site.data.alerts.end}} - -## Configure log sinks - -Log *sinks* route events from specified [logging channels](logging-overview.html#logging-channels) to destinations outside CockroachDB. These destinations currently include [log files](#output-to-files), [Fluentd](https://www.fluentd.org/)-compatible [servers](#output-to-network), and the [standard error stream (`stderr`)](#output-to-stderr). - -All supported output destinations are configured under `sinks`: - -~~~ yaml -file-defaults: ... -fluent-defaults: ... -sinks: - file-groups: - {file group name}: - channels: {channels} - ... - fluent-servers: - {server name}: - channels: {channels} - ... - stderr: - channels: {channels} - ... -~~~ - - - -All supported sink types use the following common sink parameters: - -| Parameter | Description | -|-----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `filter` | Minimum severity level at which logs enter the channels selected for the sink. Accepts one of the valid [severity levels](logging.html#logging-levels-severities) or `NONE`, which excludes all messages from the sink output. For details, see [Set logging levels](#set-logging-levels). | -| `format` | Log message format to use for file or network sinks. Accepts one of the valid [log formats](log-formats.html). For details, see [file logging format](#file-logging-format) and [network logging format](#network-logging-format). | -| `redact` | When `true`, enables automatic redaction of personally identifiable information (PII) from log messages. This ensures that sensitive data is not transmitted when collecting logs centrally or [over a network](#output-to-network). For details, see [Redact logs](#redact-logs). | -| `redactable` | When `true`, preserves redaction markers around fields that are considered sensitive in the log messages. The markers are recognized by [`cockroach debug zip`](cockroach-debug-zip.html) and [`cockroach debug merge-logs`](cockroach-debug-merge-logs.html) but may not be compatible with external log collectors. For details on how the markers appear in each format, see [Log formats](log-formats.html). | -| `exit-on-error` | When `true`, stops the Cockroach node if an error is encountered while writing to the sink. We recommend enabling this option on file sinks in order to avoid losing any log entries. When set to `false`, this can be used to mark certain sinks (such as `stderr`) as non-critical. | -| `auditable` | If `true`, enables `exit-on-error` on the sink. Also disables `buffered-writes` if the sink is under `file-groups`. This guarantees [non-repudiability](https://en.wikipedia.org/wiki/Non-repudiation) for any logs in the sink, but can incur a performance overhead and higher disk IOPS consumption. This setting is typically enabled for [security-related logs](logging-use-cases.html#security-and-audit-monitoring). | - -If not specified for a given sink, these parameter values are inherited from [`file-defaults`](#set-file-defaults) (for file sinks) and [`fluent-defaults`](#set-network-defaults) (for network sinks). - -### Output to files - -CockroachDB can write messages to one or more log files. - -`file-groups` specifies the channels that output to each log file, along with its output directory and other configuration details. For example: - -~~~ yaml -file-defaults: ... -fluent-defaults: ... -sinks: - file-groups: - default: - channels: [DEV] - health: - channels: [HEALTH] - dir: health-logs - ... -~~~ - -{{site.data.alerts.callout_success}} -A file group name is arbitrary and is used to name the log files. The `default` file group is an exception. For details, see [Log file naming](#log-file-naming). -{{site.data.alerts.end}} - -Along with the [common sink parameters](#common-sink-parameters), each file group accepts the following parameters: - -| Parameter | Description | -|-------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `channels` | List of channels that output to this sink. Use a YAML array or string of [channel names](logging-overview.html#logging-channels), `ALL` to include all channels, or `ALL EXCEPT {channels}` to include all channels except the specified channel names.

For more details on acceptable syntax, see [Logging channel selection](#logging-channel-selection). | -| `dir` | Output directory for log files generated by this sink. | -| `max-file-size` | Approximate maximum size of individual files generated by this sink. | -| `max-group-size` | Approximate maximum combined size of all files to be preserved for this sink. An asynchronous garbage collection removes files that cause the file set to grow beyond this specified size. | -| `buffered-writes` | When `true`, enables buffering of writes. Set to `false` to flush every log entry (i.e., propagate data from the `cockroach` process to the OS) and synchronize writes (i.e., ask the OS to confirm the log data was written to disk). Disabling this setting provides [non-repudiation](https://en.wikipedia.org/wiki/Non-repudiation) guarantees, but can incur a performance overhead and higher disk IOPS consumption. This setting is typically disabled for [security-related logs](logging-use-cases.html#security-and-audit-monitoring). | - -If not specified for a given file group, the parameter values are inherited from [`file-defaults`](#configure-logging-defaults) (except `channels`, which uses the [default configuration](#default-logging-configuration)). - -#### Log file naming - -Log files are named in the following format: - -~~~ -{process}-{file group}.{host}.{user}.{start timestamp in UTC}.{process ID}.log -~~~ - -For example, a file group `health` will generate a log file that looks like this: - -~~~ -cockroach-health.work-laptop.worker.2021-03-15T15_24_10Z.024338.log -~~~ - -For each file group, a symlink points to the latest generated log file. It's easiest to refer to the symlink. For example: - -~~~ -cockroach-health.log -~~~ - -{{site.data.alerts.callout_info}} -The files generated for a group named `default` are named after the pattern `cockroach.{metadata}.log`. -{{site.data.alerts.end}} - -#### Access in DB Console - -{{site.data.alerts.callout_success}} -{% include {{ page.version.version }}/ui/ui-log-files.md %} -{{site.data.alerts.end}} - -#### Known limitations - -Log files can only be accessed in the DB Console if they are stored in the same directory as the file sink for the `DEV` channel. - -### Output to network - -CockroachDB can send logs over the network to a [Fluentd](https://www.fluentd.org/)-compatible log collector (e.g., [Elasticsearch](https://www.elastic.co/elastic-stack), [Splunk](https://www.splunk.com/)). `fluent-servers` specifies the channels that output to a server, along with its configuration details. For example: - -~~~ yaml -file-defaults: ... -fluent-defaults: ... -sinks: - fluent-servers: - health: - channels: [HEALTH] - address: 127.0.0.1:5170 - ... -~~~ - -{{site.data.alerts.callout_info}} -A network sink can be listed more than once with different `address` values. This routes the same logs to different Fluentd servers. -{{site.data.alerts.end}} - -Along with the [common sink parameters](#common-sink-parameters), each Fluentd server accepts the following parameters: - -| Parameter | Description | -|-----------|--------------------------------------------------------------------------------------------------------------------| -| `channels` | List of channels that output to this sink. Use a YAML array or string of [channel names](logging-overview.html#logging-channels), `ALL` to include all channels, or `ALL EXCEPT {channels}` to include all channels except the specified channel names.

For more details on acceptable syntax, see [Logging channel selection](#logging-channel-selection). | -| `address` | Network address and port of the log collector. | -| `net` | Network protocol to use. Can be `tcp`, `tcp4`, `tcp6`, `udp`, `udp4`, `udp6`, or `unix`.

**Default:** `tcp` | - -A Fluentd sink buffers at most one log entry and retries sending the event at most one time if a network error is encountered. This is just sufficient to tolerate a restart of the Fluentd collector after a configuration change under light logging activity. If the server is unavailable for too long, or if more than one error is encountered, an error is reported to the process's standard error output with a copy of the logging event, and the logging event is dropped. - -For an example network logging configuration, see [Logging use cases](logging-use-cases.html#network-logging). - -### Output to `stderr` - -CockroachDB can output messages to the [standard error stream (`stderr`)](https://en.wikipedia.org/wiki/Standard_streams#Standard_error_(stderr)), which prints them to the machine's terminal but does not store them. `stderr` specifies the channels that output to the stream. For example: - -~~~ yaml -file-defaults: ... -fluent-defaults: ... -sinks: - stderr: - channels: [DEV] -~~~ - -Along with the [common sink parameters](#common-sink-parameters), `stderr` accepts the following parameters: - -{{site.data.alerts.callout_info}} -The `format` parameter for `stderr` is set to [`crdb-v2-tty`](log-formats.html#format-crdb-v2-tty) and cannot be changed. -{{site.data.alerts.end}} - -| Parameter | Description | -|------------|-------------------------------------------------------------------| -| `channels` | List of channels that output to this sink. Use a YAML array or string of [channel names](logging-overview.html#logging-channels), `ALL` to include all channels, or `ALL EXCEPT {channels}` to include all channels except the specified channel names.

For more details on acceptable syntax, see [Logging channel selection](#logging-channel-selection). | -| `no-color` | When `true`, removes terminal color [escape codes](https://en.wikipedia.org/wiki/ANSI_escape_code) from the output. | - -Because server start-up messages are always emitted at the start of the standard error stream, it is generally difficult to automate integration of `stderr` with log analyzers. We recommend using [file logging](#output-to-files) or [network logging](#output-to-network) instead of `stderr` when integrating with automated monitoring software. - -{{site.data.alerts.callout_info}} -By default, `cockroach start` and `cockroach start-single-node` do not print any messages to `stderr`. However, if the `cockroach` process does not have access to on-disk storage, all messages are printed to `stderr`. -{{site.data.alerts.end}} - -### Logging channel selection - -Each sink can select multiple channels. The names of selected channels can be specified as a YAML array or as a string. - -~~~ yaml -# Select just these two channels. Space is important. -channels: [OPS, HEALTH] - -# Selection is case-insensitive. -channels: [ops, HeAlTh] - -# Same configuration, as a YAML string. -# Avoid space around commas when using the YAML inline format. -channels: OPS,HEALTH - -# Same configuration, as a quoted string. -channels: 'OPS, HEALTH' - -# Same configuration, as a multi-line YAML array. -channels: -- OPS -- HEALTH -~~~ - -To select all channels, using the `all` keyword: - -~~~ yaml -channels: all -channels: 'all' -channels: [all] -channels: ['all'] -~~~ - -To select all channels except for a subset, using the `all except` keyword prefix: - -~~~ yaml -channels: all except ops,health -channels: all except [ops,health] -channels: 'all except ops, health' -channels: 'all except [ops, health]' -~~~ - -## Configure logging defaults - -When setting up a logging configuration, it's simplest to define shared parameters in `file-defaults` and `fluent-defaults` and override specific values as needed in [`file-groups`](#output-to-files), [`fluent-servers`](#output-to-network), and [`stderr`](#output-to-stderr). For a complete example, see the [default configuration](#default-logging-configuration). - -{{site.data.alerts.callout_success}} -You can view your current settings by running `cockroach debug check-log-config`, which returns the YAML definitions and a URL to a visualization of the current logging configuration. -{{site.data.alerts.end}} - -This section describes some recommended defaults. - -### Set file defaults - -Defaults for log files are set in `file-defaults`, which accepts all [common sink parameters](#common-sink-parameters) and the [file group parameters](#output-to-files) `dir`, `max-file-size`, `max-group-size`, and `buffered-writes`. - -#### Logging directory - -By default, CockroachDB adds log files to a `logs` subdirectory in the first on-disk [`store` directory](cockroach-start.html#store) (default: `cockroach-data`): - -~~~ -cockroach-data/logs -~~~ - -{{site.data.alerts.callout_success}} -Each Cockroach node generates log files in the directory specified by its logging configuration. These logs detail the internal activity of that node without visibility into the behavior of other nodes. When troubleshooting, it's best to refer to the output directory for the cluster log files, which collect the messages from all active nodes. -{{site.data.alerts.end}} - -In cloud deployments, the main data store will be subject to an IOPS budget. Adding logs to the store directory will excessively consume IOPS. For this reason, cloud deployments should output log files to a separate directory with fewer IOPS restrictions. You can override the default logging directory like this: - -~~~ yaml -file-defaults: - dir: /custom/dir/path/ -~~~ - -#### File logging format - -The default message format for log files is [`crdb-v2`](log-formats.html#format-crdb-v2). Each `crdb-v2` log message starts with a flat prefix that contains event metadata (e.g., severity, date, timestamp, channel), followed by the event payload. For details on the metadata, see [Log formats](log-formats.html#format-crdb-v2). - -If you plan to read logs programmatically, you can switch to a [JSON](log-formats.html#format-json) or [compact JSON](log-formats.html#format-json-compact) format: - -~~~ yaml -file-defaults: - format: json -~~~ - -{{site.data.alerts.callout_info}} -`format` refers to the envelope of the log message, which contains the event metadata. This is separate from the event payload, which corresponds to its [event type](eventlog.html). -{{site.data.alerts.end}} - -### Set network defaults - -Defaults for network sinks are set in `fluent-defaults`, which accepts all [common sink parameters](#common-sink-parameters). - -Note that the [server parameters](#output-to-network) `address` and `net` are *not* specified in `fluent-defaults`: - -- `address` must be specified for each sink under `fluent-servers`. -- `net` is not required and defaults to `tcp`. - -#### Network logging format - -The default message format for network output is [`json-fluent-compact`](log-formats.html#format-json-fluent-compact). Each log message is structured as a JSON payload that can be read programmatically. The `json-fluent-compact` and [`json-fluent`](log-formats.html#format-json-fluent) formats include a `tag` field that is required by the [Fluentd protocol](https://docs.fluentd.org/configuration/config-file). For details, see [Log formats](log-formats.html#format-json-fluent-compact). - -~~~ yaml -fluent-defaults: - format: json-fluent -~~~ - -{{site.data.alerts.callout_info}} -`format` refers to the envelope of the log message. This is separate from the event payload, which is structured according to [event type](eventlog.html). -{{site.data.alerts.end}} - -### Set logging levels - -Log messages are associated with a [severity level](logging.html#logging-levels-severities) when they are generated. Each logging sink is configured to output messages that meet a minimum severity level, specified with `filter`. Messages with severity levels below the `filter` threshold do not enter logging channels and are discarded. - -The [default configuration](#default-logging-configuration) uses the following severity levels for [`cockroach start`](cockroach-start.html) and [`cockroach start-single-node`](cockroach-start-single-node.html): - -- `file-defaults` and `fluent-defaults` each use `filter: INFO`. Since `INFO` is the lowest severity level, file and network sinks will emit all log messages. -- `stderr` uses `filter: NONE` and does not emit log messages. - -{{site.data.alerts.callout_info}} -All other `cockroach` commands use `filter: WARNING` and log to `stderr` by default, with these exceptions: - -- [`cockroach workload`](cockroach-sql.html#logging) uses `filter: INFO`. -- [`cockroach demo`](cockroach-demo.html#logging) uses `filter: NONE` (discards all log messages). -{{site.data.alerts.end}} - -You can also override the `file-defaults` and `fluent-defaults` severity levels on a per-sink basis. For example, use `filter: NONE` to disable logging for certain channels: - -~~~ yaml -file-defaults: ... -fluent-defaults: ... -sinks: - file-groups: - dev: - channels: [DEV] - filter: NONE -~~~ - -### Redact logs - -CockroachDB can redact personally identifiable information (PII) from log messages. The logging system includes two parameters that handle this differently: - -- `redact` is disabled by default. When enabled, `redact` automatically redacts sensitive data from logging output. We do *not* recommend enabling this on the `DEV` channel because it impairs our ability to troubleshoot problems. -- `redactable` is enabled by default. This places redaction markers around sensitive fields in log messages. These markers are recognized by [`cockroach debug zip`](cockroach-debug-zip.html) and [`cockroach debug merge-logs`](cockroach-debug-merge-logs.html), which aggregate CockroachDB log files and can be instructed to redact sensitive data from their output. - -When collecting logs centrally (e.g., in data mining scenarios where non-privileged users have access to logs) or over a network (e.g., to an external log collector), it's safest to enable `redact`: - -~~~ yaml -file-defaults: - redact: true -fluent-defaults: - redact: true -~~~ - -{{site.data.alerts.callout_success}} -In addition, the `DEV` channel should be output to a separate logging directory, since it is likely to contain sensitive data. See [`DEV` channel](#dev-channel). -{{site.data.alerts.end}} - -External log collectors can misinterpret the `cockroach debug` redaction markers, since they are specific to CockroachDB. To prevent this issue when using network sinks, disable `redactable`: - -~~~ yaml -fluent-defaults: - redactable: false -~~~ - -### DEV channel - -The `DEV` channel is used for debug and uncategorized messages. It can therefore be noisy and contain sensitive (PII) information. - -We recommend configuring `DEV` separately from the other logging channels. When sending logs [over the network](#output-to-network), `DEV` logs should also be excluded from network collection. - -In this example, the `dev` file group is reserved for `DEV` logs. These are output to a `cockroach-dev.log` file in a custom disk `dir`: - -~~~ yaml -file-defaults: ... -fluent-defaults: ... -sinks: - file-groups: - dev: - channels: [DEV] - dir: /custom/dir/path/ - ... - fluent-servers: - ... -~~~ - -{{site.data.alerts.callout_success}} -To ensure that you are protecting sensitive information, also [redact your logs](#redact-logs). -{{site.data.alerts.end}} - -## Stray error capture - -Certain events, such as uncaught software exceptions (panics), bypass the CockroachDB logging system. However, they can be useful in troubleshooting. For example, if CockroachDB crashes, it normally logs a stack trace to what caused the problem. - -To ensure that these stray errors can be tracked, CockroachDB does not send them to `stderr` by default. Instead, stray errors are output to a `cockroach-stderr.log` file in the default [logging directory](#logging-directory). - -You can change these settings in `capture-stray-errors`: - -~~~ yaml -file-defaults: ... -fluent-defaults: ... -sinks: ... -capture-stray-errors: - enable: true - dir: /custom/dir/path/ -~~~ - -{{site.data.alerts.callout_info}} -When `capture-stray-errors` is disabled, [`redactable`](#redact-logs) cannot be enabled on the `stderr` sink. This is because `stderr` will contain both stray errors and logged events and cannot apply redaction markers in a reliable way. Note that [`redact`](#redact-logs) can still be enabled on `stderr` in this case. -{{site.data.alerts.end}} - -## Default logging configuration - -The YAML payload below represents the default logging behavior of [`cockroach start`](cockroach-start.html) and [`cockroach start-single-node`](cockroach-start-single-node.html). To retain backward compatibility with v20.2 and earlier, these settings match the [log filenames used in previous versions](logging-overview.html#changes-to-logging-system) (except `sql-auth`, which should be changed to `auth` to match the log filename used in v20.2 and earlier). - -~~~ yaml -file-defaults: - max-file-size: 10MiB - max-group-size: 100MiB - buffered-writes: true - filter: INFO - format: crdb-v2 - redact: false - redactable: true - exit-on-error: true - auditable: false -fluent-defaults: - filter: INFO - format: json-fluent-compact - redact: false - redactable: true - exit-on-error: false - auditable: false -sinks: - file-groups: - default: - channels: [DEV, OPS, HEALTH, SQL_SCHEMA, USER_ADMIN, PRIVILEGES] - pebble: - channels: [STORAGE] - sql-audit: - channels: [SENSITIVE_ACCESS] - auditable: true - sql-auth: - channels: [SESSIONS] - auditable: true - sql-exec: - channels: [SQL_EXEC] - sql-slow: - channels: [SQL_PERF] - sql-slow-internal-only: - channels: [SQL_INTERNAL_PERF] - stderr: - channels: all - filter: NONE - redact: false - redactable: true - exit-on-error: true -capture-stray-errors: - enable: true - max-group-size: 100MiB -~~~ - -Note that a default `dir` is not specified for `file-defaults` and `capture-stray-errors`: - -- The default `dir` for `file-defaults` is inferred from the first on-disk [`store` directory](cockroach-start.html#store). See [Logging directory](#logging-directory). -- The default `dir` for `capture-stray-errors` is inherited form `file-defaults`. - -## See also - -- [Logging Use Cases](logging-use-cases.html) -- [Log Formats](log-formats.html) diff --git a/src/current/v21.1/configure-replication-zones.md b/src/current/v21.1/configure-replication-zones.md deleted file mode 100644 index a48f74f01fe..00000000000 --- a/src/current/v21.1/configure-replication-zones.md +++ /dev/null @@ -1,693 +0,0 @@ ---- -title: Configure Replication Zones -summary: In CockroachDB, you use replication zones to control the number and location of replicas for specific sets of data. -keywords: ttl, time to live, availability zone -toc: true ---- - -Replication zones give you the power to control what data goes where in your CockroachDB cluster. Specifically, they are used to control the number and location of replicas for data belonging to the following objects: - -- Databases -- Tables -- Rows ([Enterprise-only](enterprise-licensing.html)) -- Indexes ([Enterprise-only](enterprise-licensing.html)) -- All data in the cluster, including internal system data ([via the default replication zone](#view-the-default-replication-zone)) - -For each of the above objects you can control: - -- How many copies of each range to spread through the cluster. -- Which constraints are applied to which data, e.g., "table X's data can only be stored in the German availability zones". -- The maximum size of ranges (how big ranges get before they are split). -- How long old data is kept before being garbage collected. -- Where you would like the leaseholders for certain ranges to be located, e.g., "for ranges that are already constrained to have at least one replica in `region=us-west`, also try to put their leaseholders in `region=us-west`". - -This page explains how replication zones work and how to use the [`CONFIGURE ZONE`](configure-zone.html) statement to manage them. - -{{site.data.alerts.callout_info}} -To configure replication zones, a user must be a member of the [`admin` role](authorization.html#admin-role) or have been granted [`CREATE`](authorization.html#supported-privileges) or [`ZONECONFIG`](authorization.html#supported-privileges) privileges. To configure [`system` objects](#for-system-data), the user must be a member of the `admin` role. -{{site.data.alerts.end}} - -## Overview - -Every [range](architecture/overview.html#glossary) in the cluster is part of a replication zone. Each range's zone configuration is taken into account as ranges are rebalanced across the cluster to ensure that any constraints are honored. - -When a cluster starts, there are two categories of replication zone: - -1. Pre-configured replication zones that apply to internal system data. -2. A single default replication zone that applies to the rest of the cluster. - -You can adjust these pre-configured zones as well as add zones for individual databases, tables, rows, and secondary indexes as needed. Note that adding zones for rows and secondary indexes is [Enterprise-only](enterprise-licensing.html). - -For example, you might rely on the [default zone](#view-the-default-replication-zone) to spread most of a cluster's data across all of your availability zones, but [create a custom replication zone for a specific database](#create-a-replication-zone-for-a-database) to make sure its data is only stored in certain availability zones and/or geographies. - -## Replication zone levels - -There are five replication zone levels for [**table data**](architecture/distribution-layer.html#table-data) in a cluster, listed from least to most granular: - -Level | Description -------|------------ -Cluster | CockroachDB comes with a pre-configured `default` replication zone that applies to all table data in the cluster not constrained by a database, table, or row-specific replication zone. This zone can be adjusted but not removed. See [View the Default Replication Zone](#view-the-default-replication-zone) and [Edit the Default Replication Zone](#edit-the-default-replication-zone) for more details. -Database | You can add replication zones for specific databases. See [Create a Replication Zone for a Database](#create-a-replication-zone-for-a-database) for more details. -Table | You can add replication zones for specific tables. See [Create a Replication Zone for a Table](#create-a-replication-zone-for-a-table). -Index ([Enterprise-only](enterprise-licensing.html)) | The [secondary indexes](indexes.html) on a table will automatically use the replication zone for the table. However, with an Enterprise license, you can add distinct replication zones for secondary indexes. See [Create a Replication Zone for a Secondary Index](#create-a-replication-zone-for-a-secondary-index) for more details. -Row ([Enterprise-only](enterprise-licensing.html)) | You can add replication zones for specific rows in a table or secondary index by [defining table partitions](partitioning.html). See [Create a Replication Zone for a Table Partition](#create-a-replication-zone-for-a-partition) for more details. - -### For system data - -In addition, CockroachDB stores internal [**system data**](architecture/distribution-layer.html#monolithic-sorted-map-structure) in what are called system ranges. There are two replication zone levels for this internal system data, listed from least to most granular: - -Level | Description -------|------------ -Cluster | The `default` replication zone mentioned above also applies to all system ranges not constrained by a more specific replication zone. -System Range | CockroachDB comes with pre-configured replication zones for important system ranges, such as the "meta" and "liveness" ranges. If necessary, you can add replication zones for the "timeseries" range and other system ranges as well. Editing replication zones for system ranges may override settings from `default`. See [Create a Replication Zone for a System Range](#create-a-replication-zone-for-a-system-range) for more details.

CockroachDB also comes with pre-configured replication zones for the internal `system` database and the `system.jobs` table, which stores metadata about long-running jobs such as schema changes and backups. - -### Level priorities - -When replicating data, whether table or system, CockroachDB always uses the most granular replication zone available. For example, for a piece of user data: - -1. If there's a replication zone for the row, CockroachDB uses it. -2. If there's no applicable row replication zone and the row is from a secondary index, CockroachDB uses the secondary index replication zone. -3. If the row isn't from a secondary index or there is no applicable secondary index replication zone, CockroachDB uses the table replication zone. -4. If there's no applicable table replication zone, CockroachDB uses the database replication zone. -5. If there's no applicable database replication zone, CockroachDB uses the `default` cluster-wide replication zone. - -## Manage replication zones - -Use the [`CONFIGURE ZONE`](configure-zone.html) statement to [add](#create-a-replication-zone-for-a-system-range), [modify](#edit-the-default-replication-zone), [reset](#reset-a-replication-zone), and [remove](#remove-a-replication-zone) replication zones. - -### Replication zone variables - -Use the [`ALTER ... CONFIGURE ZONE`](configure-zone.html) [statement](sql-statements.html) to set a replication zone: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE t CONFIGURE ZONE USING range_min_bytes = 0, range_max_bytes = 90000, gc.ttlseconds = 89999, num_replicas = 5, constraints = '[-region=west]'; -~~~ - -{% include {{page.version.version}}/zone-configs/variables.md %} - -### Replication constraints - -The location of replicas, both when they are first added and when they are rebalanced to maintain cluster equilibrium, is based on the interplay between descriptive attributes assigned to nodes and constraints set in zone configurations. - -{{site.data.alerts.callout_success}}For demonstrations of how to set node attributes and replication constraints in different scenarios, see Scenario-based Examples below.{{site.data.alerts.end}} - -#### Descriptive attributes assigned to nodes - -When starting a node with the [`cockroach start`](cockroach-start.html) command, you can assign the following types of descriptive attributes: - -Attribute Type | Description ----------------|------------ -**Node Locality** | Using the [`--locality`](cockroach-start.html#locality) flag, you can assign arbitrary key-value pairs that describe the location of the node. Locality might include region, country, availability zone, etc. The key-value pairs should be ordered into _locality tiers_ that range from most inclusive to least inclusive (e.g., region before availability zone as in `region=eu,az=paris`), and the keys and the order of key-value pairs must be the same on all nodes. It's typically better to include more pairs than fewer. For example:

`--locality=region=east,az=us-east-1`
`--locality=region=east,az=us-east-2`
`--locality=region=west,az=us-west-1`

CockroachDB attempts to spread replicas evenly across the cluster based on locality, with the order of locality tiers determining the priority. Locality can also be used to influence the location of data replicas in various ways using replication zones.

When there is high latency between nodes, CockroachDB uses locality to move range leases closer to the current workload, reducing network round trips and improving read performance. See [Follow-the-workload](topology-follow-the-workload.html) for more details. -**Node Capability** | Using the `--attrs` flag, you can specify node capability, which might include specialized hardware or number of cores, for example:

`--attrs=ram:64gb` -**Store Type/Capability** | Using the `attrs` field of the `--store` flag, you can specify disk type or capability, for example:

`--store=path=/mnt/ssd01,attrs=ssd`
`--store=path=/mnt/hda1,attrs=hdd:7200rpm` - -#### Types of constraints - -The node-level and store-level descriptive attributes mentioned above can be used as the following types of constraints in replication zones to influence the location of replicas. However, note the following general guidance: - -- When locality is the only consideration for replication, it's recommended to set locality on nodes without specifying any constraints in zone configurations. In the absence of constraints, CockroachDB attempts to spread replicas evenly across the cluster based on locality. -- Required and prohibited constraints are useful in special situations where, for example, data must or must not be stored in a specific country or on a specific type of machine. -- Avoid conflicting constraints. CockroachDB returns an error if you: - - Redefine a required constraint key within the same `constraints` definition on all replicas. For example, `constraints = '[+region=west, +region=east]'` will result in an error. - - Define a required and prohibited definition for the same key-value pair. For example, `constraints = '[-region=west, +region=west]'` will result in an error. - -Constraint Type | Description | Syntax -----------------|-------------|------- -**Required** | When placing replicas, the cluster will consider only nodes/stores with matching attributes or localities. When there are no matching nodes/stores, new replicas will not be added. | `+ssd` -**Prohibited** | When placing replicas, the cluster will ignore nodes/stores with matching attributes or localities. When there are no alternate nodes/stores, new replicas will not be added. | `-ssd` - -#### Scope of constraints - -Constraints can be specified such that they apply to all replicas in a zone or such that different constraints apply to different replicas, meaning you can effectively pick the exact location of each replica. - -Constraint Scope | Description | Syntax ------------------|-------------|------- -**All Replicas** | Constraints specified using JSON array syntax apply to all replicas in every range that's part of the replication zone. | `constraints = '[+ssd, -region=west]'` -**Per-Replica** | Multiple lists of constraints can be provided in a JSON object, mapping each list of constraints to an integer number of replicas in each range that the constraints should apply to.

The total number of replicas constrained cannot be greater than the total number of replicas for the zone (`num_replicas`). However, if the total number of replicas constrained is less than the total number of replicas for the zone, the non-constrained replicas will be allowed on any nodes/stores.

Note that per-replica constraints must be "required" (e.g., `'{"+region=west": 1}'`); they cannot be "prohibited" (e.g., `'{"-region=west": 1}'`). Also, when defining per-replica constraints on a database or table, `num_replicas` must be specified as well, but not when defining per-replica constraints on an index or partition.

See the [Per-replica constraints](#per-replica-constraints-to-specific-availability-zones) example for more details. | `constraints = '{"+ssd,+region=west": 2, "+region=east": 1}', num_replicas = 3` - -### Node/replica recommendations - -See [Cluster Topography](recommended-production-settings.html#topology) recommendations for production deployments. - -### Troubleshooting zone constraint violations - -To see if any of the data placement constraints defined in your replication zone configurations are being violated, use the `system.replication_constraint_stats` report as described in [Replication Reports](query-replication-reports.html). - -## View replication zones - -Use the [`SHOW ZONE CONFIGURATIONS`](#view-all-replication-zones) statement to view details about existing replication zones. - -You can also use the [`SHOW PARTITIONS`](show-partitions.html) statement to view the zone constraints on existing table partitions, or [`SHOW CREATE TABLE`](show-create.html) to view zone configurations for a table. - -{% include {{page.version.version}}/sql/crdb-internal-partitions.md %} - -## Basic examples - -{% include {{ page.version.version }}/sql/movr-statements-geo-partitioned-replicas.md %} - -These examples focus on the basic approach and syntax for working with zone configuration. For examples demonstrating how to use constraints, see [Scenario-based examples](#scenario-based-examples). - -For more examples, see [`CONFIGURE ZONE`](configure-zone.html) and [`SHOW ZONE CONFIGURATIONS`](show-zone-configurations.html). - -### View all replication zones - -{% include {{ page.version.version }}/zone-configs/view-all-replication-zones.md %} - -For more information, see [`SHOW ZONE CONFIGURATIONS`](show-zone-configurations.html). - -### View the default replication zone - -{% include {{ page.version.version }}/zone-configs/view-the-default-replication-zone.md %} - -For more information, see [`SHOW ZONE CONFIGURATIONS`](show-zone-configurations.html). - -### Edit the default replication zone - -{% include {{ page.version.version }}/zone-configs/edit-the-default-replication-zone.md %} - -For more information, see [`CONFIGURE ZONE`](configure-zone.html). - -### Create a replication zone for a system range - -{% include {{ page.version.version }}/zone-configs/create-a-replication-zone-for-a-system-range.md %} - -For more information, see [`CONFIGURE ZONE`](configure-zone.html). - -### Create a replication zone for a database - -{% include {{ page.version.version }}/zone-configs/create-a-replication-zone-for-a-database.md %} - -For more information, see [`CONFIGURE ZONE`](configure-zone.html). - -### Create a replication zone for a table - -{% include {{ page.version.version }}/zone-configs/create-a-replication-zone-for-a-table.md %} - -For more information, see [`CONFIGURE ZONE`](configure-zone.html). - -### Create a replication zone for a secondary index - -{% include {{ page.version.version }}/zone-configs/create-a-replication-zone-for-a-secondary-index.md %} - -For more information, see [`CONFIGURE ZONE`](configure-zone.html). - -### Create a replication zone for a partition - -{% include {{ page.version.version }}/zone-configs/create-a-replication-zone-for-a-table-partition.md %} - -For more information, see [`CONFIGURE ZONE`](configure-zone.html). - -### Reset a replication zone - -{% include {{ page.version.version }}/zone-configs/reset-a-replication-zone.md %} - -For more information, see [`CONFIGURE ZONE`](configure-zone.html). - -### Remove a replication zone - -{% include {{ page.version.version }}/zone-configs/remove-a-replication-zone.md %} - -For more information, see [`CONFIGURE ZONE`](configure-zone.html). - -### Constrain leaseholders to specific availability zones - -{% include {{ page.version.version }}/zone-configs/constrain-leaseholders-to-specific-datacenters.md %} - -For more information, see [`CONFIGURE ZONE`](configure-zone.html). - -## Scenario-based examples - -### Even replication across availability zones - -**Scenario:** - -- You have 6 nodes across 3 availability zones, 2 nodes in each availability zone. -- You want data replicated 3 times, with replicas balanced evenly across all three availability zones. - -**Approach:** - -1. Start each node with its availability zone location specified in the [`--locality`](cockroach-start.html#locality) flag: - - Availability zone 1: - - ~~~ shell - $ cockroach start --insecure --advertise-addr= --locality=az=us-1 \ - --join=,, - $ cockroach start --insecure --advertise-addr= --locality=az=us-1 \ - --join=,, - ~~~ - - Availability zone 2: - - ~~~ shell - $ cockroach start --insecure --advertise-addr= --locality=az=us-2 \ - --join=,, - $ cockroach start --insecure --advertise-addr= --locality=az=us-2 \ - --join=,, - ~~~ - - Availability zone 3: - - ~~~ shell - $ cockroach start --insecure --advertise-addr= --locality=az=us-3 \ - --join=,, - $ cockroach start --insecure --advertise-addr= --locality=az=us-3 \ - --join=,, - ~~~ - -2. Initialize the cluster: - - ~~~ shell - $ cockroach init --insecure --host= - ~~~ - -There's no need to make zone configuration changes; by default, the cluster is configured to replicate data three times, and even without explicit constraints, the cluster will aim to diversify replicas across node localities. - -### Per-replica constraints to specific availability zones - -**Scenario:** - -- You have 5 nodes across 5 availability zones in 3 regions, 1 node in each availability zone. -- You want data replicated 3 times, with a quorum of replicas for a database holding West Coast data centered on the West Coast and a database for nation-wide data replicated across the entire country. - -**Approach:** - -1. Start each node with its region and availability zone location specified in the [`--locality`](cockroach-start.html#locality) flag: - - Start the five nodes: - - ~~~ shell - $ cockroach start --insecure --advertise-addr= --locality=region=us-west1,az=us-west1-a \ - --join=,,,, - $ cockroach start --insecure --advertise-addr= --locality=region=us-west1,az=us-west1-b \ - --join=,,,, - $ cockroach start --insecure --advertise-addr= --locality=region=us-central1,az=us-central1-a \ - --join=,,,, - $ cockroach start --insecure --advertise-addr= --locality=region=us-east1,az=us-east1-a \ - --join=,,,, - $ cockroach start --insecure --advertise-addr= --locality=region=us-east1,az=us-east1-b \ - --join=,,,, - ~~~ - - Initialize the cluster: - - ~~~ shell - $ cockroach init --insecure --host= - ~~~ - -2. On any node, open the [built-in SQL client](cockroach-sql.html): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --insecure - ~~~ - -3. Create the database for the West Coast application: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE DATABASE west_app_db; - ~~~ - -4. Configure a replication zone for the database: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER DATABASE west_app_db - CONFIGURE ZONE USING constraints = '{"+region=us-west1": 2, "+region=us-central1": 1}', num_replicas = 3; - ~~~ - - ~~~ - CONFIGURE ZONE 1 - ~~~ - -5. View the replication zone: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW ZONE CONFIGURATION FOR DATABASE west_app_db; - ~~~ - - ~~~ - target | raw_config_sql - +----------------------+--------------------------------------------------------------------+ - DATABASE west_app_db | ALTER DATABASE west_app_db CONFIGURE ZONE USING - | range_min_bytes = 134217728, - | range_max_bytes = 536870912, - | gc.ttlseconds = 90000, - | num_replicas = 3, - | constraints = '{+region=us-central1: 1, +region=us-west1: 2}', - | lease_preferences = '[]' - (1 row) - ~~~ - - Two of the database's three replicas will be put in `region=us-west1` and its remaining replica will be put in `region=us-central1`. This gives the application the resilience to survive the total failure of any one availability zone while providing low-latency reads and writes on the West Coast because a quorum of replicas are located there. - -6. No configuration is needed for the nation-wide database. The cluster is configured to replicate data 3 times and spread them as widely as possible by default. Because the first key-value pair specified in each node's locality is considered the most significant part of each node's locality, spreading data as widely as possible means putting one replica in each of the three different regions. - -### Multiple applications writing to different databases - -**Scenario:** - -- You have 2 independent applications connected to the same CockroachDB cluster, each application using a distinct database. -- You have 6 nodes across 2 availability zones, 3 nodes in each availability zone. -- You want the data for application 1 to be replicated 5 times, with replicas evenly balanced across both availability zones. -- You want the data for application 2 to be replicated 3 times, with all replicas in a single availability zone. - -**Approach:** - -1. Start each node with its availability zone location specified in the [`--locality`](cockroach-start.html#locality) flag: - - Availability zone 1: - - ~~~ shell - $ cockroach start --insecure --advertise-addr= --locality=az=us-1 \ - --join=,,,,, - $ cockroach start --insecure --advertise-addr= --locality=az=us-1 \ - --join=,,,,, - $ cockroach start --insecure --advertise-addr= --locality=az=us-1 \ - --join=,,,,, - ~~~ - - Availability zone 2: - - ~~~ shell - $ cockroach start --insecure --advertise-addr= --locality=az=us-2 \ - --join=,,,,, - $ cockroach start --insecure --advertise-addr= --locality=az=us-2 \ - --join=,,,,, - $ cockroach start --insecure --advertise-addr= --locality=az=us-2 \ - --join=,,,,, - ~~~ - - Initialize the cluster: - - ~~~ shell - $ cockroach init --insecure --host= - ~~~ - -2. On any node, open the [built-in SQL client](cockroach-sql.html): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --insecure - ~~~ - -3. Create the database for application 1: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE DATABASE app1_db; - ~~~ - -4. Configure a replication zone for the database used by application 1: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER DATABASE app1_db CONFIGURE ZONE USING num_replicas = 5; - ~~~ - - ~~~ - CONFIGURE ZONE 1 - ~~~ - -5. View the replication zone: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW ZONE CONFIGURATION FOR DATABASE app1_db; - ~~~ - - ~~~ - target | raw_config_sql - +------------------+---------------------------------------------+ - DATABASE app1_db | ALTER DATABASE app1_db CONFIGURE ZONE USING - | range_min_bytes = 134217728, - | range_max_bytes = 536870912, - | gc.ttlseconds = 90000, - | num_replicas = 5, - | constraints = '[]', - | lease_preferences = '[]' - (1 row) - ~~~ - - Nothing else is necessary for application 1's data. Since all nodes specify their availability zone locality, the cluster will aim to balance the data in the database used by application 1 between availability zones 1 and 2. - -6. Still in the SQL client, create a database for application 2: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE DATABASE app2_db; - ~~~ - -7. Configure a replication zone for the database used by application 2: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER DATABASE app2_db CONFIGURE ZONE USING constraints = '[+az=us-2]'; - ~~~ - -8. View the replication zone: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW ZONE CONFIGURATION FOR DATABASE app2_db; - ~~~ - - ~~~ - target | raw_config_sql - +------------------+---------------------------------------------+ - DATABASE app2_db | ALTER DATABASE app2_db CONFIGURE ZONE USING - | range_min_bytes = 134217728, - | range_max_bytes = 536870912, - | gc.ttlseconds = 90000, - | num_replicas = 3, - | constraints = '[+az=us-2]', - | lease_preferences = '[]' - (1 row) - ~~~ - - The required constraint will force application 2's data to be replicated only within the `us-2` availability zone. - -### Stricter replication for a table and its secondary indexes - -**Scenario:** - -- You have 7 nodes, 5 with SSD drives and 2 with HDD drives. -- You want data replicated 3 times by default. -- Speed and availability are important for a specific table and its indexes, which are queried very frequently, however, so you want the data in the table and secondary indexes to be replicated 5 times, preferably on nodes with SSD drives. - -**Approach:** - -1. Start each node with `ssd` or `hdd` specified as store attributes: - - 5 nodes with SSD storage: - - ~~~ shell - $ cockroach start --insecure --advertise-addr= --store=path=node1,attrs=ssd \ - --join=,, - $ cockroach start --insecure --advertise-addr= --store=path=node2,attrs=ssd \ - --join=,, - $ cockroach start --insecure --advertise-addr= --store=path=node3,attrs=ssd \ - --join=,, - $ cockroach start --insecure --advertise-addr= --store=path=node4,attrs=ssd \ - --join=,, - $ cockroach start --insecure --advertise-addr= --store=path=node5,attrs=ssd \ - --join=,, - ~~~ - - 2 nodes with HDD storage: - - ~~~ shell - $ cockroach start --insecure --advertise-addr= --store=path=node6,attrs=hdd \ - --join=,, - $ cockroach start --insecure --advertise-addr= --store=path=node7,attrs=hdd \ - --join=,, - ~~~ - - Initialize the cluster: - - ~~~ shell - $ cockroach init --insecure --host= - ~~~ - -2. On any node, open the [built-in SQL client](cockroach-sql.html): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --insecure - ~~~ - -3. Create a database and table: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE DATABASE db; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE TABLE db.important_table; - ~~~ - -4. Configure a replication zone for the table that must be replicated more strictly: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER TABLE db.important_table CONFIGURE ZONE USING num_replicas = 5, constraints = '[+ssd]' - ~~~ - -5. View the replication zone: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW ZONE CONFIGURATION FOR TABLE db.important_table; - ~~~ - - ~~~ - target | config_sql - +-------------------------------+---------------------------------------------+ - TABLE db.important_table | ALTER DATABASE app2_db CONFIGURE ZONE USING - | range_min_bytes = 1048576, - | range_max_bytes = 536870912, - | gc.ttlseconds = 90000, - | num_replicas = 5, - | constraints = '[+ssd]', - | lease_preferences = '[]' - (1 row) - ~~~ - - The secondary indexes on the table will use the table's replication zone, so all data for the table will be replicated 5 times, and the required constraint will place the data on nodes with `ssd` drives. - -### Tweaking the replication of system ranges - -**Scenario:** - -- You have nodes spread across 7 availability zones. -- You want data replicated 5 times by default. -- For better performance, you want a copy of the meta ranges in all of the availability zones. -- To save disk space, you only want the internal timeseries data replicated 3 times by default. - -**Approach:** - -1. Start each node with a different [locality](cockroach-start.html#locality) attribute: - - ~~~ shell - $ cockroach start --insecure --advertise-addr= --locality=az=us-1 \ - --join=,, - $ cockroach start --insecure --advertise-addr= --locality=az=us-2 \ - --join=,, - $ cockroach start --insecure --advertise-addr= --locality=az=us-3 \ - --join=,, - $ cockroach start --insecure --advertise-addr= --locality=az=us-4 \ - --join=,, - $ cockroach start --insecure --advertise-addr= --locality=az=us-5 \ - --join=,, - $ cockroach start --insecure --advertise-addr= --locality=az=us-6 \ - --join=,, - $ cockroach start --insecure --advertise-addr= --locality=az=us-7 \ - --join=,, - ~~~ - - Initialize the cluster: - - ~~~ shell - $ cockroach init --insecure --host= - ~~~ - -2. On any node, open the [built-in SQL client](cockroach-sql.html): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --insecure - ~~~ - -3. Configure the default replication zone: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER RANGE default CONFIGURE ZONE USING num_replicas = 5; - ~~~ - -4. View the replication zone: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW ZONE CONFIGURATION FOR RANGE default; - ~~~ - ~~~ - target | raw_config_sql - +---------------+------------------------------------------+ - RANGE default | ALTER RANGE default CONFIGURE ZONE USING - | range_min_bytes = 134217728, - | range_max_bytes = 536870912, - | gc.ttlseconds = 90000, - | num_replicas = 5, - | constraints = '[]', - | lease_preferences = '[]' - (1 row) - ~~~ - - All data in the cluster will be replicated 5 times, including both SQL data and the internal system data. - -5. Configure the `meta` replication zone: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER RANGE meta CONFIGURE ZONE USING num_replicas = 7; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW ZONE CONFIGURATION FOR RANGE meta; - ~~~ - ~~~ - target | raw_config_sql - +------------+---------------------------------------+ - RANGE meta | ALTER RANGE meta CONFIGURE ZONE USING - | range_min_bytes = 134217728, - | range_max_bytes = 536870912, - | gc.ttlseconds = 3600, - | num_replicas = 7, - | constraints = '[]', - | lease_preferences = '[]' - (1 row) - ~~~ - - The `meta` addressing ranges will be replicated such that one copy is in all 7 availability zones, while all other data will be replicated 5 times. - -6. Configure the `timeseries` replication zone: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER RANGE timeseries CONFIGURE ZONE USING num_replicas = 3; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW ZONE CONFIGURATION FOR RANGE timeseries; - ~~~ - ~~~ - target | raw_config_sql - +------------------+---------------------------------------------+ - RANGE timeseries | ALTER RANGE timeseries CONFIGURE ZONE USING - | range_min_bytes = 134217728, - | range_max_bytes = 536870912, - | gc.ttlseconds = 90000, - | num_replicas = 3, - | constraints = '[]', - | lease_preferences = '[]' - (1 row) - ~~~ - - The timeseries data will only be replicated 3 times without affecting the configuration of all other data. - -## See also - -- [`SHOW ZONE CONFIGURATIONS`](show-zone-configurations.html) -- [`CONFIGURE ZONE`](configure-zone.html) -- [`SHOW PARTITIONS`](show-partitions.html) -- [SQL Statements](sql-statements.html) -- [Table Partitioning](partitioning.html) -- [Replication Reports](query-replication-reports.html) diff --git a/src/current/v21.1/configure-zone.md b/src/current/v21.1/configure-zone.md deleted file mode 100644 index a63577a7a57..00000000000 --- a/src/current/v21.1/configure-zone.md +++ /dev/null @@ -1,131 +0,0 @@ ---- -title: CONFIGURE ZONE -summary: Use the CONFIGURE ZONE statement to add, modify, reset, and remove replication zones. -toc: true ---- - -`CONFIGURE ZONE` is a subcommand of the `ALTER DATABASE`, `ALTER TABLE`, `ALTER INDEX`, `ALTER PARTITION`, and `ALTER RANGE` statements and is used to add, modify, reset, or remove [replication zones](configure-replication-zones.html) for those objects. To view details about existing replication zones, see [`SHOW ZONE CONFIGURATIONS`](show-zone-configurations.html). - -In CockroachDB, you can use **replication zones** to control the number and location of replicas for specific sets of data, both when replicas are first added and when they are rebalanced to maintain cluster equilibrium. - -{{site.data.alerts.callout_info}} -Adding replication zones for secondary indexes and partitions is an [Enterprise-only](enterprise-licensing.html) feature. -{{site.data.alerts.end}} - -## Synopsis - -**alter_zone_database_stmt ::=** - -
-{% include {{ page.version.version }}/sql/generated/diagrams/alter_zone_database.html %} -
- -**alter_zone_table_stmt ::=** - -
-{% include {{ page.version.version }}/sql/generated/diagrams/alter_zone_table.html %} -
- -**alter_zone_index_stmt ::=** - -
-{% include {{ page.version.version }}/sql/generated/diagrams/alter_zone_index.html %} -
- -**alter_zone_partition_stmt ::=** - -
-{% include {{ page.version.version }}/sql/generated/diagrams/alter_zone_partition.html %} -
- -**alter_zone_range_stmt ::=** - -
-{% include {{ page.version.version }}/sql/generated/diagrams/alter_zone_range.html %} -
- -## Required privileges - -If the target is a [`system` range](#create-a-replication-zone-for-a-system-range), the [`system` database](show-databases.html#preloaded-databases), or a table in the `system` database, the user must be a member of the [`admin` role](authorization.html#create-and-manage-roles). For all other databases and tables, the user must have been granted either the [`CREATE`](grant.html#supported-privileges) or the [`ZONECONFIG`](grant.html#supported-privileges) privilege on the target database or table. - -## Parameters - - Parameter | Description ------------+------------- -`range_name` | The name of the system [range](architecture/overview.html#glossary) whose [replication zone configurations](configure-replication-zones.html) you want to change. -`database_name` | The name of the [database](create-database.html) whose [replication zone configurations](configure-replication-zones.html) you want to change.
**New in v21.1:** If you directly change a database's zone configuration with `ALTER DATABASE ... CONFIGURE ZONE`, CockroachDB will block all [`ALTER DATABASE ... SET PRIMARY REGION`](set-primary-region.html) statements on the database. -`table_name` | The name of the [table](create-table.html) whose [replication zone configurations](configure-replication-zones.html) you want to change. -`partition_name` | The name of the [partition](partitioning.html) whose [replication zone configurations](configure-replication-zones.html) you want to change. -`index_name` | The name of the [index](indexes.html) whose [replication zone configurations](configure-replication-zones.html) you want to change. -`variable` | The name of the [variable](#variables) to change. -`value` | The value of the variable to change. -`DISCARD` | Remove a replication zone. - -### Variables - -{% include {{ page.version.version }}/zone-configs/variables.md %} - -## Viewing schema changes - -{% include {{ page.version.version }}/misc/schema-change-view-job.md %} - -## Examples - -{% include {{ page.version.version }}/sql/movr-statements-geo-partitioned-replicas.md %} - -### Edit a replication zone - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE users CONFIGURE ZONE USING range_min_bytes = 0, range_max_bytes = 90000, gc.ttlseconds = 89999, num_replicas = 4; -~~~ - -~~~ -CONFIGURE ZONE 1 -~~~ - -### Edit the default replication zone - -{% include {{ page.version.version }}/zone-configs/edit-the-default-replication-zone.md %} - -### Create a replication zone for a database - -{% include {{ page.version.version }}/zone-configs/create-a-replication-zone-for-a-database.md %} - -### Create a replication zone for a table - -{% include {{ page.version.version }}/zone-configs/create-a-replication-zone-for-a-table.md %} - -### Create a replication zone for a secondary index - -{% include {{ page.version.version }}/zone-configs/create-a-replication-zone-for-a-secondary-index.md %} - -### Create a replication zone for a partition - -{% include {{ page.version.version }}/zone-configs/create-a-replication-zone-for-a-table-partition.md %} - -### Create a replication zone for a system range - -{% include {{ page.version.version }}/zone-configs/create-a-replication-zone-for-a-system-range.md %} - -### Reset a replication zone - -{% include {{ page.version.version }}/zone-configs/reset-a-replication-zone.md %} - -### Remove a replication zone - -{% include {{ page.version.version }}/zone-configs/remove-a-replication-zone.md %} - -## See also - -- [Configure Replication Zones](configure-replication-zones.html) -- [`PARTITION BY`](partition-by.html) -- [`SHOW ZONE CONFIGURATIONS`](show-zone-configurations.html) -- [`ALTER DATABASE`](alter-database.html) -- [`ALTER TABLE`](alter-table.html) -- [`ALTER INDEX`](alter-index.html) -- [`ALTER PARTITION`](alter-partition.html) -- [`ALTER RANGE`](alter-range.html) -- [`SHOW JOBS`](show-jobs.html) -- [Table Partitioning](partitioning.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/connect-to-the-database-cockroachcloud.md b/src/current/v21.1/connect-to-the-database-cockroachcloud.md deleted file mode 100644 index dab5d2dc811..00000000000 --- a/src/current/v21.1/connect-to-the-database-cockroachcloud.md +++ /dev/null @@ -1,155 +0,0 @@ ---- -title: Connect to the Database (CockroachDB Cloud) -summary: How to connect to a CockroachDB Cloud cluster from your application -toc: true ---- - -
- - -
- -This page has instructions for connecting to a CockroachDB {{ site.data.products.cloud }} cluster from your application using various programming languages. Each example shows a [connection string][connection_params] for a secure cluster to a `bank` database. Depending on your cluster's configuration, you may need to edit this connection string. - -The connection strings listed on this page set the required authentication options to connect to [free CockroachDB {{ site.data.products.serverless }}](../cockroachcloud/authentication.html) clusters. CockroachDB {{ site.data.products.cloud }} clusters use a signed certificate generated for your cluster that you download from the CockroachDB {{ site.data.products.cloud }} console. - -For a reference that lists all of the supported cluster connection parameters, see [Connection Parameters][connection_params]. - -## Before you begin - -Do the following: - - - -- Set up a CockroachDB {{ site.data.products.cloud }} cluster: - - [Create a CockroachDB {{ site.data.products.cloud }} cluster](../cockroachcloud/create-your-cluster.html). - - [Connect to the CockroachDB {{ site.data.products.cloud }} cluster](../cockroachcloud/connect-to-your-cluster.html). -- [Install a client driver or ORM framework](install-client-drivers.html). - -## Connect - -
- - - - -
- -
- -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql \ ---url='postgres://{username}:{password}@{globalhost}:26257/{cluster_name}.{database}?sslmode=verify-full&sslrootcert={path to the CA certificate}' -~~~ - -{% include {{page.version.version}}/app/cc-free-tier-params.md %} - -For more information about how to use the built-in SQL client, see the [`cockroach sql`](cockroach-sql.html) reference docs. - -
- -
- -{% include_cached copy-clipboard.html %} -~~~ go -import ( - "database/sql" - "fmt" - "log" - _ "github.com/lib/pq" -) - -db, err := sql.Open("postgres", - "postgresql://{username}:{password}@{globalhost}:26257/bank?sslmode=verify-full&sslrootcert={path to the CA certificate}&options=--cluster={cluster_name}") -if err != nil { - log.Fatal("error connecting to the database: ", err) -} -defer db.Close() -~~~ - -{% include {{page.version.version}}/app/cc-free-tier-params.md %} - -{% include {{page.version.version}}/app/for-a-complete-example-go.md %} - -
- -
- -{% include_cached copy-clipboard.html %} -~~~ java -import java.sql.*; -import javax.sql.DataSource; - -PGSimpleDataSource ds = new PGSimpleDataSource(); -ds.setServerName("{globalhost}"); -ds.setPortNumber(26257); -ds.setDatabaseName("{cluster_name}.bank"); -ds.setUser("{username}"); -ds.setPassword("{password}"); -ds.setSsl(true); -ds.setSslMode("verify-full"); -ds.setSslrootCert("{path to the CA certificate}"); -ds.setReWriteBatchedInserts(true); // add `rewriteBatchedInserts=true` to pg connection string -ds.setApplicationName("BasicExample"); -~~~ - -{% include {{page.version.version}}/app/cc-free-tier-params.md %} - -{% include {{page.version.version}}/app/for-a-complete-example-java.md %} - -
- -
- -{% include_cached copy-clipboard.html %} -~~~ python -import psycopg2 - -conn = psycopg2.connect( - database='bank', - user='{username}', - password='{password}' - sslmode='verify-full', - sslrootcert='{path to the CA certificate}', - port=26257, - host='{globalhost}', - options="--cluster={cluster_name}" -) -~~~ - -{% include {{page.version.version}}/app/cc-free-tier-params.md %} - -{% include {{page.version.version}}/app/for-a-complete-example-python.md %} - -
- -{% include cockroachcloud/cc-no-user-certs.md %} - -## What's next? - - - -- [Design a Database Schema](schema-design-overview.html) -- [Insert Data](insert-data.html) -- [Query Data](query-data.html) -- [Update Data](update-data.html) -- [Delete Data](delete-data.html) - -You might also be interested in the following pages: - -- [Connection Pooling](connection-pooling.html) -- [Connection Parameters][connection_params] -- [Manual Deployments][manual] -- [Orchestrated Deployments][orchestrated] -- [Start a Local Cluster][local_secure] -- [Error Handling and Troubleshooting](error-handling-and-troubleshooting.html) -- [Optimize Statement Performance](make-queries-fast.html) -- [Hello World example apps](example-apps.html) - - - -[manual]: manual-deployment.html -[orchestrated]: orchestration.html -[local_secure]: secure-a-cluster.html -[connection_params]: connection-parameters.html diff --git a/src/current/v21.1/connect-to-the-database.md b/src/current/v21.1/connect-to-the-database.md deleted file mode 100644 index 94d4fba8460..00000000000 --- a/src/current/v21.1/connect-to-the-database.md +++ /dev/null @@ -1,147 +0,0 @@ ---- -title: Connect to the Database -summary: How to connect to a CockroachDB cluster from your application -toc: true ---- - -
- - -
- -This page has instructions for connecting to a CockroachDB cluster from your application using various programming languages. Each example shows a [connection string][connection_params] for a secure cluster to a `bank` database. Depending on your cluster's configuration, you may need to edit this connection string. - -The connection strings listed on this page set the required authentication options to connect to [local](authentication.html) clusters. Local clusters use self-signed SSL certificates. - -For a reference that lists all of the supported cluster connection parameters, see [Connection Parameters][connection_params]. - -## Before you begin - -Do the following: - -- Set up a local CockroachDB cluster: - - [Install CockroachDB](install-cockroachdb.html). - - [Start a local cluster](secure-a-cluster.html). -- [Install a client driver or ORM framework](install-client-drivers.html). - -## Connect - -
- - - - -
- -
- -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --certs-dir=certs --host=localhost:26257 -~~~ - -For more information about how to use the built-in SQL client, see the [`cockroach sql`](cockroach-sql.html) reference docs. - -
- -
- -{% include_cached copy-clipboard.html %} -~~~ go -import ( - "database/sql" - "fmt" - "log" - _ "github.com/lib/pq" -) - -db, err := sql.Open("postgres", - "postgresql://maxroach@localhost:26257/bank?ssl=true&sslmode=require&sslrootcert=certs/ca.crt&sslkey=certs/client.maxroach.key&sslcert=certs/client.maxroach.crt") -if err != nil { - log.Fatal("error connecting to the database: ", err) -} -defer db.Close() -~~~ - -{% include {{page.version.version}}/app/for-a-complete-example-go.md %} - -
- -
- -{% include_cached copy-clipboard.html %} -~~~ java -import java.sql.*; -import javax.sql.DataSource; - -PGSimpleDataSource ds = new PGSimpleDataSource(); -ds.setServerName("localhost"); -ds.setPortNumber(26257); -ds.setDatabaseName("bank"); -ds.setUser("maxroach"); -ds.setPassword(null); -ds.setSsl(true); -ds.setSslMode("require"); -ds.setSslCert("certs/client.maxroach.crt"); -ds.setSslKey("certs/client.maxroach.key.pk8"); -ds.setReWriteBatchedInserts(true); // add `rewriteBatchedInserts=true` to pg connection string -ds.setApplicationName("BasicExample"); -~~~ - -{{site.data.alerts.callout_success}} -{% include {{page.version.version}}/app/pkcs8-gen.md %} -{{site.data.alerts.end}} - -{% include {{page.version.version}}/app/for-a-complete-example-java.md %} - -
- -
- -{% include_cached copy-clipboard.html %} -~~~ python -import psycopg2 - -conn = psycopg2.connect( - database='bank', - user='maxroach', - sslmode='require', - sslrootcert='certs/ca.crt', - sslkey='certs/client.maxroach.key', - sslcert='certs/client.maxroach.crt', - port=26257, - host='localhost' -) -~~~ - -{% include {{page.version.version}}/app/for-a-complete-example-python.md %} - -
- -## What's next? - - - -- [Design a Database Schema](schema-design-overview.html) -- [Insert Data](insert-data.html) -- [Query Data](query-data.html) -- [Update Data](update-data.html) -- [Delete Data](delete-data.html) - -You might also be interested in the following pages: - -- [Connection Pooling](connection-pooling.html) -- [Connection Parameters][connection_params] -- [Manual Deployments][manual] -- [Orchestrated Deployments][orchestrated] -- [Start a Local Cluster][local_secure] -- [Error Handling and Troubleshooting](error-handling-and-troubleshooting.html) -- [Optimize Statement Performance](make-queries-fast.html) -- [Hello World example apps](example-apps.html) - - - -[manual]: manual-deployment.html -[orchestrated]: orchestration.html -[local_secure]: secure-a-cluster.html -[connection_params]: connection-parameters.html diff --git a/src/current/v21.1/connection-parameters.md b/src/current/v21.1/connection-parameters.md deleted file mode 100644 index d703f0af935..00000000000 --- a/src/current/v21.1/connection-parameters.md +++ /dev/null @@ -1,267 +0,0 @@ ---- -title: Client Connection Parameters -summary: This page describes the parameters used to establish a client connection. -toc: true ---- - -Client applications, including [`cockroach` client -commands](cockroach-commands.html), work by establishing a network -connection to a CockroachDB cluster. The client connection parameters -determine which CockroachDB cluster they connect to, and how to -establish this network connection. - -## Supported connection parameters - -Most client apps, including `cockroach` client commands, determine -which CockroachDB server to connect to using a [PostgreSQL connection -URL](#connect-using-a-url). When using a URL, a client can also -specify additional SQL-level parameters. This mode provides the most -configuration flexibility. - -In addition, all `cockroach` client commands also accept [discrete -connection parameters](#connect-using-discrete-parameters) that can -specify the connection parameters separately from a URL. - -## When to use a URL and when to use discrete parameters - -Specifying client parameters using a URL may be more convenient during -experimentation, as it facilitates copy-pasting the connection -parameters (the URL) between different tools: the output of `cockroach -start`, other `cockroach` commands, GUI database visualizer, -programming tools, etc. - - -Discrete parameters may be more convenient in automation, where the -components of the configuration are filled in separately from -different variables in a script or a service manager. - -## Connect using a URL - -A connection URL has the following format: - -{% include_cached copy-clipboard.html %} -~~~ -postgres://:@:/? -~~~ - -`cockroach` client commands also support [UNIX domain socket URIs](https://en.wikipedia.org/wiki/Unix_domain_socket) of the following form: - -{% include_cached copy-clipboard.html %} -~~~ -postgres://:@?host=&port=& -~~~ - - Component | Description | Required -----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------- - `` | The [SQL user](create-user.html) that will own the client session. | ✗ - `` | The user's password. It is not recommended to pass the password in the URL directly.

Note that CockroachDB currently does not allow passwords with special characters to be passed in a URL. For details, see [tracking issue](https://github.com/cockroachdb/cockroach/issues/35998).

[Find more detail about how CockroachDB handles passwords](authentication.html#client-authentication). | ✗ - `` | The host name or address of a CockroachDB node or load balancer. | Required by most client drivers. - `` | The port number of the SQL interface of the CockroachDB node or load balancer. The default port number for CockroachDB is 26257. Use this value when in doubt. | Required by most client drivers. - `` | A database name to use as [current database](sql-name-resolution.html#current-database). Defaults to `defaultdb`. | ✗ - `` | The directory path to the client listening for a socket connection. | Required when specifying a Unix domain socket URI. - `` | [Additional connection parameters](#additional-connection-parameters), including SSL/TLS certificate settings. | [`options=--cluster=`](#supported-options-parameters) is required for free CockroachDB {{ site.data.products.cloud }} clusters. - - -{{site.data.alerts.callout_info}} -For cockroach commands that accept a URL, you can specify the URL with the command-line flag `--url`. -If `--url` is not specified but -the environment variable `COCKROACH_URL` is defined, the environment -variable is used. Otherwise, the `cockroach` command will use -[discrete connection parameters](#connect-using-discrete-parameters) -as described below. -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -The `` part is not used for [`cockroach` -commands](cockroach-commands.html) other than [`cockroach -sql`](cockroach-sql.html). A warning -is currently printed if it is mistakenly specified, and -future versions of CockroachDB may return an error in that case. -{{site.data.alerts.end}} - -### Additional connection parameters - -The following additional parameters can be passed after the `?` character in the URL. After the first parameter is specified, any additional parameters must be separated by an ampersand (`&`). - -Parameter | Description | Default value -----------|-------------|--------------- -`application_name` | An initial value for the [`application_name` session variable](set-vars.html).

Note: For [Java JDBC](build-a-java-app-with-cockroachdb.html), use `ApplicationName`. | Empty string. -`sslmode` | Which type of secure connection to use: `disable`, `allow`, `prefer`, `require`, `verify-ca` or `verify-full`. See [Secure Connections With URLs](#secure-connections-with-urls) for details. | `disable` -`sslrootcert` | Path to the [CA certificate](cockroach-cert.html), when `sslmode` is not `disable`. | Empty string. -`sslcert` | Path to the [client certificate](cockroach-cert.html), when `sslmode` is not `disable`. | Empty string. -`sslkey` | Path to the [client private key](cockroach-cert.html), when `sslmode` is not `disable`. | Empty string. -`options` | [Additional options](#supported-options-parameters) to be passed to the server. | Empty string - -#### Supported `options` parameters - -CockroachDB supports the following `options` parameters. After the first `options` parameter is specified, any additional parameters in the same connection string must be separated by a space. - -Parameter | Description -----------|------------- -`--cluster=` | Specifies the cluster name when connecting to [CockroachDB {{ site.data.products.serverless }} clusters](connect-to-the-database-cockroachcloud.html#connect). -`-c =` | **New in v21.1:** Sets a [session variable](set-vars.html) for the SQL session. - -{{site.data.alerts.callout_info}} -Note that some drivers require certain characters to be properly encoded in URL connection strings. For example, spaces in [a JDBC connection string](https://jdbc.postgresql.org/documentation/use/#connection-parameters) must specified as `%20`. -{{site.data.alerts.end}} - -### Secure connections with URLs - -The following values are supported for `sslmode`, although only the first and the last are recommended for use. - -Parameter | Description | Recommended for use -----------|-------------|-------------------- -`sslmode=disable` | Do not use an encrypted, secure connection at all. | Use during development. -`sslmode=allow` | Enable a secure connection only if the server requires it.

**Not supported in all clients.** | -`sslmode=prefer` | Try to establish a secure connection, but accept an insecure connection if the server does not support secure connections.

**Not supported in all clients.** | -`sslmode=require` | Force a secure connection. An error occurs if the secure connection cannot be established. | -`sslmode=verify-ca` | Force a secure connection and verify that the server certificate is signed by a known CA. | -`sslmode=verify-full` | Force a secure connection, verify that the server certificate is signed by a known CA, and verify that the server address matches that specified in the certificate. | Use for [secure deployments](secure-a-cluster.html). - -{{site.data.alerts.callout_danger}} -Some client drivers and the `cockroach` commands do not support -`sslmode=allow` and `sslmode=prefer`. Check the documentation of your -SQL driver to determine whether these options are supported. -{{site.data.alerts.end}} - -### Example URL for an insecure connection - -The following URL is suitable to connect to a CockroachDB node using an insecure connection: - -{% include_cached copy-clipboard.html %} -~~~ -postgres://root@servername:26257/mydb?sslmode=disable -~~~ - -This specifies a connection for the `root` user to server `servername` -on port 26257 (the default CockroachDB SQL port), with `mydb` set as -current database. `sslmode=disable` makes the connection insecure. - -### Example URL for a secure connection - -The following URL is suitable to connect to a CockroachDB node using a secure connection: - -{% include_cached copy-clipboard.html %} -~~~ -postgres://root@servername:26257/mydb?sslmode=verify-full&sslrootcert=path/to/ca.crt&sslcert=path/to/client.username.crt&sslkey=path/to/client.username.key -~~~ - -This uses the following components: - -- User `root` -- Host name `servername`, port number 26257 (the default CockroachDB SQL port) -- Current database `mydb` -- SSL/TLS mode `verify-full`: - - Root CA certificate `path/to/ca.crt` - - Client certificate `path/to/client.username.crt` - - Client key `path/to/client.username.key` - -For details about how to create and manage SSL/TLS certificates, see -[Create Security Certificates](cockroach-cert.html) and -[Rotate Certificates](rotate-certificates.html). - -### Example URI for a Unix domain socket - -The following URI is suitable to connect to a CockroachDB cluster listening for Unix domain socket connections at `/path/to/client`: - -{% include_cached copy-clipboard.html %} -~~~ -postgres://root@?host=/path/to/client&port=26257 -~~~ - -This specifies a connection for the `root` user to an insecure cluster listening for a socket connection (e.g., a cluster started with the [`--socket-dir` flag](cockroach-start.html#networking)) at `/path/to/client`, and on port 26257. - -## Connect using discrete parameters - -Most [`cockroach` commands](cockroach-commands.html) accept connection -parameters as separate, discrete command-line flags, in addition (or -in replacement) to `--url` which [specifies all parameters as a -URL](#connect-using-a-url). - -For each command-line flag that directs a connection parameter, -CockroachDB also recognizes an environment variable. The environment -variable is used when the command-line flag is not specified. - -{% include {{ page.version.version }}/sql/connection-parameters.md %} - -### Example command-line flags for an insecure connection - -The following command-line flags establish an insecure connection: - -{% include_cached copy-clipboard.html %} -~~~ ---user=root \ ---host= ---insecure -~~~ - -This specifies a connection for the `root` user to server `servername` -on port 26257 (the default CockroachDB SQL port). `--insecure` makes -the connection insecure. - -### Example command-line flags for a secure connection - -The following command-line flags establish a secure connection: - -{% include_cached copy-clipboard.html %} -~~~ ---user=root \ ---host= ---certs-dir=path/to/certs -~~~ - -This uses the following components: - -- User `root` -- Host name `servername`, port number 26257 (the default CockroachDB SQL port) -- SSL/TLS enabled, with settings: - - Root CA certificate `path/to/certs/ca.crt` - - Client certificate `path/to/client..crt` (`path/to/certs/client.root.crt` with `--user root`) - - Client key `path/to/client..key` (`path/to/certs/client.root.key` with `--user root`) - -{{site.data.alerts.callout_info}} -When using discrete connection parameters, the file names of the CA -and client certificates and client key are derived automatically from -the value of `--certs-dir`. -{{site.data.alerts.end}} - -## Using both URL and client parameters - -Most `cockroach` commands accept both a URL and client parameters. -The information contained therein is combined in the order it appears -in the command line. - -This combination is useful so that discrete command-line flags can -override settings not otherwise set in the URL. - -### Example override of the current database - -The `cockroach start` command prints out the following connection URL, which connects to the `defaultdb` database: - -{% include_cached copy-clipboard.html %} -~~~ -postgres://root@servername:26257/?sslmode=disable -~~~ - -To specify `mydb` as the current database using [`cockroach sql`](cockroach-sql.html), run the following command: - -{% include_cached copy-clipboard.html %} -~~~ -cockroach sql \ ---url "postgres://root@servername:26257/?sslmode=disable" \ ---database mydb -~~~ - -This is equivalent to: - -{% include_cached copy-clipboard.html %} -~~~ -cockroach sql --url "postgres://root@servername:26257/mydb?sslmode=disable" -~~~ - -## See also - -- [`cockroach` commands](cockroach-commands.html) -- [Create Security Certificates](cockroach-cert.html) -- [Secure a Cluster](secure-a-cluster.html) -- [Create and Manage Users](authorization.html#create-and-manage-users) diff --git a/src/current/v21.1/connection-pooling.md b/src/current/v21.1/connection-pooling.md deleted file mode 100644 index 2634f3e6944..00000000000 --- a/src/current/v21.1/connection-pooling.md +++ /dev/null @@ -1,117 +0,0 @@ ---- -title: Connection Pooling -summary: How to plan, configure, and use connection pools when using drivers or frameworks with CockroachDB. -toc: true ---- - -This page has information on planning, configuring, and using connection pools when using drivers or frameworks with CockroachDB. - -## About connection pools - -A typical database operation consists of several steps. - -1. The driver uses a configuration to start a connection to the database server. -1. A network socket is opened on the client that connects to the database server. -1. Data is read or written through the network socket. -1. The connection is closed down. -1. The network socket is closed down and its resources are freed. - -For simple database operations, these steps are not expensive, but as an application scales up, the performance of the application will suffer as each connection is created and destroyed. One pattern for improving performance is a connection pool, a group of already configured and established network connections between the client and database that can be reused for data operations within an application. - -Each time an application reads or writes data, it will request one of the connections in the pool. After the data operation completes, the connection is returned to the pool so other operations can use it. - -Connection pooling can be a enabled as a feature of the driver, a separate library used in conjunction with a driver, a feature of an application server, or a proxy server that acts as a gateway to the database server. - -## Sizing connection pools - -Idle connections in CockroachDB do not consume many resources compared to PostgreSQL. Cockroach Labs estimates the memory overhead of idle connections in CockroachDB is 20 kB to 30 kB per connection. - -Creating the appropriate size pool of connections is critical to gaining maximum performance in an application. Too few connections in the pool will result in high latency as each operation waits for a connection to open up. But adding too many connections to the pool can also result in high latency as each connection thread is being run in parallel by the system. The time it takes for many threads to complete in parallel is typically higher than the time it takes a smaller number of threads to run sequentially. - -Each processor core can only execute one thread at a time. When there are more threads than processor cores, the system will use context switching to [time-slice](https://en.wikipedia.org/wiki/Preemption_(computing)#Time_slice) the thread execution. For example, if you have a system with a single core and two threads, processing threads 1 and 2 in parallel results in the system context switching to pause execution of thread 1 and begin executing thread 2, and then pause execution of thread 2 to resume executing thread 1. Executing thread 1 completely and then executing thread 2 will be faster because the system doesn't need to context switch, even though thread 2 had to wait until thread 1 fully completed to begin executing. - -Storage and network performance also will affect the ability of a thread to fully execute. If a thread is blocked by network or storage latency, adding connections to the pool is a good idea so other threads can execute while the original thread is being blocked. - -Cockroach Labs performed lab testing of various customer workloads and found no improvement in scalability beyond: - -**connections = (number of cores * 4)** - -Many workloads perform best when the maximum number of active connections is between 2 and 4 times the number of CPU cores in the cluster. - -If you have a large number of services connecting to the same cluster, make sure the number of concurrent active connections across all the services does not exceed this recommendation by a large amount. If each service has its own connection pool, then you should make sure the sum of all the pool sizes is close to our maximum connections recommendation. Each workload and application is different, so you should conduct testing to determine the best-performing pool sizes for each service in your architecture. - -In addition to setting a maximum connection pool size, set the maximum number of idle connections if possible. Cockroach Labs recommends setting the maximum number of idle connections to the maximum pool size. While this uses more memory, it allows many connections when concurrency is high without having to create a new connection for every new operation. - -## Validating connections in a pool - -After a connection pool initializes connections to CockroachDB clusters, those connections can occasionally break. This could be due to changes in the cluster topography, or rolling upgrades and restarts, or network disruptions. CockroachDB {{ site.data.products.cloud }} clusters periodically are restarted for minor version updates, for example, so previously established connections would be invalid after the restart. - -Validating connections is typically handled automatically by the connection pool. For example, in HikariCP the connection is validated whenever you request a connection from the pool, and the `keepaliveTime` property allows you to configure an interval to periodically check if the connections in the pool are valid. Whatever connection pool you use, make sure connection validation is enabled when running your application. - -## Example - -
- - -
- -
- -In this example, a Java application similar to the [basic JDBC example](build-a-java-app-with-cockroachdb.html) uses the [PostgreSQL JDBC driver](https://jdbc.postgresql.org/) and [HikariCP](https://github.com/brettwooldridge/HikariCP) as the connection pool layer to connect to a CockroachDB cluster. The database is being run on 10 cores across the cluster. - -Using the connection pool formula above: - -**connections = (10 [processor cores] * 4)** - -The connection pool size should be 40. - -{% include_cached copy-clipboard.html %} -~~~ java -HikariConfig config = new HikariConfig(); -config.setJdbcUrl("jdbc:postgresql://localhost:26257/bank"); -config.setUsername("maxroach"); -config.setPassword("password"); -config.addDataSourceProperty("ssl", "true"); -config.addDataSourceProperty("sslmode", "require"); -config.addDataSourceProperty("reWriteBatchedInserts", "true"); -config.setAutoCommit(false); -config.setMaximumPoolSize(40); -config.setKeepaliveTime(150000); - -HikariDataSource ds = new HikariDataSource(config); - -Connection conn = ds.getConnection(); -~~~ - -
- -
- -In this example, a Go application similar to the [basic pgx example](build-a-go-app-with-cockroachdb.html) uses the [pgxpool library](https://pkg.go.dev/github.com/jackc/pgx/v4/pgxpool) to create a connection pool on a CockroachDB cluster. The database is being run on 10 cores across the cluster. - -Using the connection pool formula above: - -**connections = (10 [processor cores] * 4)** - -The connection pool size should be 40. - -~~~ go -// Set connection pool configuration, with maximum connection pool size. -config, err := pgxpool.ParseConfig("postgres://max:roach@127.0.0.1:26257/bank?sslmode=require&pool_max_conns=40") - if err != nil { - log.Fatal("error configuring the database: ", err) - } - -// Create a connection pool to the "bank" database. -dbpool, err := pgxpool.ConnectConfig(context.Background(), config) -if err != nil { - log.Fatal("error connecting to the database: ", err) -} -defer dbpool.Close() -~~~ - -This example uses the `pool_max_conns` parameter to set the maximum number of connections in the connection pool to 30. - -For a full list of connection pool configuration parameters for pgxpool, see [the pgxpool documentation](https://pkg.go.dev/github.com/jackc/pgx/v4/pgxpool#Config). - -
diff --git a/src/current/v21.1/constraints.md b/src/current/v21.1/constraints.md deleted file mode 100644 index 537b4f2546f..00000000000 --- a/src/current/v21.1/constraints.md +++ /dev/null @@ -1,125 +0,0 @@ ---- -title: Constraints -summary: Constraints offer additional data integrity by enforcing conditions on the data within a column. -toc: true ---- - -Constraints offer additional data integrity by enforcing conditions on the data within a column. Whenever values are manipulated (inserted, deleted, or updated), constraints are checked and modifications that violate constraints are rejected. - -For example, the `UNIQUE` constraint requires that all values in a column be unique from one another (except *NULL* values). If you attempt to write a duplicate value, the constraint rejects the entire statement. - - -## Supported constraints - - Constraint | Description -------------|------------- - [`CHECK`](check.html) | Values must return `TRUE` or `NULL` for a Boolean expression. - [`DEFAULT` value](default-value.html) | If a value is not defined for the constrained column in an `INSERT` statement, the `DEFAULT` value is written to the column. - [`FOREIGN KEY`](foreign-key.html) | Values must exactly match existing values from the column it references. - [`NOT NULL`](not-null.html) | Values may not be *NULL*. - [`PRIMARY KEY`](primary-key.html) | Values must uniquely identify each row *(one per table)*. This behaves as if the `NOT NULL` and `UNIQUE` constraints are applied, as well as automatically creates an [index](indexes.html) for the table using the constrained columns. - [`UNIQUE`](unique.html) | Each non-*NULL* value must be unique. This also automatically creates an [index](indexes.html) for the table using the constrained columns. - -## Using constraints - -### Add constraints - -How you add constraints depends on the number of columns you want to constrain, as well as whether or not the table is new. - -- **One column of a new table** has its constraints defined after the column's data type. For example, this statement applies the `PRIMARY KEY` constraint to `foo.a`: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE TABLE foo (a INT PRIMARY KEY); - ~~~ -- **Multiple columns of a new table** have their constraints defined after the table's columns. For example, this statement applies the `PRIMARY KEY` constraint to `foo`'s columns `a` and `b`: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE TABLE bar (a INT, b INT, PRIMARY KEY (a,b)); - ~~~ - - {{site.data.alerts.callout_info}} - The `DEFAULT` and `NOT NULL` constraints cannot be applied to multiple columns. - {{site.data.alerts.end}} - -- **Existing tables** can have the following constraints added: - - `CHECK`, `FOREIGN KEY`, and `UNIQUE` constraints can be added through [`ALTER TABLE...ADD CONSTRAINT`](add-constraint.html). For example, this statement adds the `UNIQUE` constraint to `baz.id`: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER TABLE baz ADD CONSTRAINT id_unique UNIQUE (id); - ~~~ - - - `DEFAULT` values and `NOT NULL` constraints can be added through [`ALTER TABLE...ALTER COLUMN`](alter-column.html#set-or-change-a-default-value). For example, this statement adds the Default Value constraint to `baz.bool`: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER TABLE baz ALTER COLUMN bool SET DEFAULT true; - ~~~ - - - [`PRIMARY KEY`](primary-key.html) constraints can be added with [`ADD CONSTRAINT`](add-constraint.html)/[`ADD PRIMARY KEY`](alter-table.html) in the following circumstances: - - - A [`DROP CONSTRAINT`](drop-constraint.html) statement precedes the `ADD CONSTRAINT`/`ADD PRIMARY KEY` statement in the same transaction. For examples, see the [`ADD CONSTRAINT`](add-constraint.html#examples) and [`DROP CONSTRAINT`](drop-constraint.html#examples) pages. - - The current [primary key is on `rowid`](indexes.html#creation), the default primary key created if none is explicitly defined at table creation. - - The `ADD CONSTRAINT`/`ADD PRIMARY KEY` is in the same transaction as a `CREATE TABLE` statement with no primary key defined. - -#### Order of constraints - -The order in which you list constraints is not important because constraints are applied to every modification of their respective tables or columns. - -#### Name constraints on new tables - -You can name constraints applied to new tables using the `CONSTRAINT` clause before defining the constraint: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE foo (a INT CONSTRAINT another_name PRIMARY KEY); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE bar (a INT, b INT, CONSTRAINT yet_another_name PRIMARY KEY (a,b)); -~~~ - -### View constraints - -To view a table's constraints, use [`SHOW CONSTRAINTS`](show-constraints.html) or [`SHOW CREATE`](show-create.html). - -### Remove constraints - -The procedure for removing a constraint depends on its type: - -Constraint Type | Procedure ------------------|----------- -[`CHECK`](check.html) | Use [`DROP CONSTRAINT`](drop-constraint.html). -[`DEFAULT` value](default-value.html) | Use [`ALTER COLUMN`](alter-column.html#remove-default-constraint). -[`FOREIGN KEY`](foreign-key.html) | Use [`DROP CONSTRAINT`](drop-constraint.html). -[`NOT NULL`](not-null.html) | Use [`ALTER COLUMN`](alter-column.html#remove-not-null-constraint). -[`PRIMARY KEY`](primary-key.html) | Primary key constraints can be dropped with [`DROP CONSTRAINT`](drop-constraint.html) if an [`ADD CONSTRAINT`](add-constraint.html) statement follows the `DROP CONSTRAINT` statement in the same transaction. -[`UNIQUE`](unique.html) | The `UNIQUE` constraint cannot be dropped directly. To remove the constraint, [drop the index](drop-index.html) that was created by the constraint, e.g., `DROP INDEX my_unique_constraint`. - -### Change constraints - -The procedure for changing a constraint depends on its type: - -Constraint Type | Procedure ------------------|----------- -[`CHECK`](check.html) | [Issue a transaction](transactions.html#syntax) that adds a new `CHECK` constraint ([`ADD CONSTRAINT`](add-constraint.html)), and then remove the existing one ([`DROP CONSTRAINT`](drop-constraint.html)). -[`DEFAULT` value](default-value.html) | The `DEFAULT` value can be changed through [`ALTER COLUMN`](alter-column.html). -[`FOREIGN KEY`](foreign-key.html) | [Issue a transaction](transactions.html#syntax) that adds a new `FOREIGN KEY` constraint ([`ADD CONSTRAINT`](add-constraint.html)), and then remove the existing one ([`DROP CONSTRAINT`](drop-constraint.html)). -[`NOT NULL`](not-null.html) | The `NOT NULL` constraint cannot be changed, only added and removed with [`ALTER COLUMN`](alter-column.html). -[`PRIMARY KEY`](primary-key.html) | To change a primary key, use an [`ALTER TABLE ... ALTER PRIMARY KEY`](alter-primary-key.html) statement.

When you change a primary key with [`ALTER PRIMARY KEY`](alter-primary-key.html), the old primary key index becomes a secondary index. If you do not want the old primary key to become a secondary index, use [`DROP CONSTRAINT`](drop-constraint.html)/[`ADD CONSTRAINT`](add-constraint.html) to change the primary key. -[`UNIQUE`](unique.html) | [Issue a transaction](transactions.html#syntax) that adds a new `UNIQUE` constraint ([`ADD CONSTRAINT`](add-constraint.html)), and then remove the existing one ([`DROP CONSTRAINT`](drop-constraint.html)). - - -## See also - -- [`CREATE TABLE`](create-table.html) -- [`ADD CONSTRAINT`](add-constraint.html) -- [`DROP CONSTRAINT`](drop-constraint.html) -- [`SHOW CONSTRAINTS`](show-constraints.html) -- [`SHOW CREATE`](show-create.html) -- [`ALTER PRIMARY KEY`](alter-primary-key.html) -- [`ALTER TABLE`](alter-table.html) -- [`ALTER COLUMN`](alter-column.html) diff --git a/src/current/v21.1/convert-to-schema.md b/src/current/v21.1/convert-to-schema.md deleted file mode 100644 index e564e2489df..00000000000 --- a/src/current/v21.1/convert-to-schema.md +++ /dev/null @@ -1,137 +0,0 @@ ---- -title: CONVERT TO SCHEMA -summary: The CONVERT TO SCHEMA statement converts a database to a schema. -toc: true ---- - - The `CONVERT TO SCHEMA` [statement](sql-statements.html) converts a database to a new, user-defined [schema](sql-name-resolution.html). When you convert a database to a schema, all [tables](create-table.html), [sequences](create-sequence.html), and [user-defined types](enum.html) in the database become child objects of the new schema, and the database is deleted. - -In CockroachDB versions < v20.2, [user-defined schemas](create-schema.html) are not supported, and all stored objects in a given database use the `public` schema. To provide a [multi-level structure for stored objects](sql-name-resolution.html) in earlier versions of CockroachDB, we've recommended using [database](create-database.html) namespaces instead of schema namespaces. The `CONVERT TO SCHEMA` statement is meant to help users who are upgrading to v20.2 and want to use schema namespaces in a way that is more similar to [PostgreSQL](https://www.postgresql.org/docs/current/ddl-schemas.html). - -{{site.data.alerts.callout_info}} -`CONVERT TO SCHEMA` is a subcommand of [`ALTER DATABASE`](alter-database.html). -{{site.data.alerts.end}} - -## Required privileges - -Only members of the `admin` role can convert databases to schemas. By default, the `root` user belongs to the `admin` role. - -## Syntax - -~~~ -ALTER DATABASE CONVERT TO SCHEMA WITH PARENT -~~~ - -## Parameters - -Parameter | Description -----------|------------ -`name` | The name of the database to convert. -`parent_name` | The name of the parent database to which the new schema will belong. - -## Limitations - -A database cannot be converted to a schema if: - -- The database is the [current database](sql-name-resolution.html#current-database). -- The database has a child schema other than the `public` schema. -- The database contains dependent objects (e.g., [views](views.html)). - -## Example - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### Convert a database to a schema - -By default, tables are stored in the `public` schema: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM public; -~~~ - -~~~ - schema_name | table_name | type | estimated_row_count ---------------+----------------------------+-------+---------------------- - public | promo_codes | table | 1000 - public | rides | table | 500 - public | user_promo_codes | table | 0 - public | users | table | 50 - public | vehicle_location_histories | table | 1000 - public | vehicles | table | 15 -(6 rows) -~~~ - -Suppose that you want to convert `movr` to a schema, with a new database named `cockroach_labs` as its parent. - -First, create the new database: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE DATABASE cockroach_labs; -~~~ - -Then, set the new database as the current database (recall that you cannot convert the current database to a schema): - -{% include_cached copy-clipboard.html %} -~~~ sql -> USE cockroach_labs; -~~~ - -Convert the `movr` database to a schema, with `cockroach_labs` as its parent database: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER DATABASE movr CONVERT TO SCHEMA WITH PARENT cockroach_labs; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SCHEMAS; -~~~ - -~~~ - schema_name ----------------------- - crdb_internal - information_schema - movr - pg_catalog - pg_extension - public -(6 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES; -~~~ - -~~~ - schema_name | table_name | type | estimated_row_count ---------------+----------------------------+-------+---------------------- - movr | promo_codes | table | 0 - movr | rides | table | 0 - movr | user_promo_codes | table | 0 - movr | users | table | 0 - movr | vehicle_location_histories | table | 0 - movr | vehicles | table | 0 -(6 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM public; -~~~ - -~~~ - schema_name | table_name | type | estimated_row_count ---------------+------------+------+---------------------- -(0 rows) -~~~ - -## See also - -- [`CREATE SCHEMA`](create-schema.html) -- [`SHOW SCHEMAS`](show-schemas.html) -- [`Name Resolution`](sql-name-resolution.html) diff --git a/src/current/v21.1/copy-from.md b/src/current/v21.1/copy-from.md deleted file mode 100644 index 3e5912dcdff..00000000000 --- a/src/current/v21.1/copy-from.md +++ /dev/null @@ -1,124 +0,0 @@ ---- -title: COPY FROM -summary: Copy data from a third-party client to CockroachDB. -toc: true ---- - -The `COPY FROM` statement copies data from third-party clients to tables in your cluster. - -{{site.data.alerts.callout_info}} -CockroachDB currently only supports `COPY FROM` statements issued from third-party clients, for compatibility with PostgreSQL drivers and ORMs. `COPY FROM` statements cannot be issued from the [`cockroach` SQL shell](cockroach-sql.html). To copy data from a file to your cluster, we recommend using an [`IMPORT`](import.html) statement instead. -{{site.data.alerts.end}} - -## Syntax - -
-{% include {{ page.version.version }}/sql/generated/diagrams/copy_from.html %} -
- -### Parameters - -Parameter | Description ------------|------------- -`table_name` | The name of the table to which to copy data. -`opt_column_list` | The column name, or list of column names, to which to copy data. -`WITH copy_options` | Optionally specify one or more [copy options](#options). - -### Options - -Option | Description ------------|------------- -`DELIMITER 'value'` | **New in v21.1:** The value that delimits the rows of input data, passed as a string. -`NULL 'value'` | **New in v21.1:** The string that represents a `NULL` value in the input data. -`BINARY` | Copy data from binary format. If `BINARY` is specified, no other format can be specified.
If no format is specified, CockroachDB copies in plaintext format. -`CSV` | **New in v21.1:** Copy data from CSV format. If `CSV` is specified, no other format can be specified.
If no format is specified, CockroachDB copies in plaintext format. - -## Required privileges - -Only members of the `admin` role can run `COPY` statements. By default, the `root` user belongs to the `admin` role. - -## Known limitations - -### `COPY FROM` statements are not supported in the CockroachDB SQL shell - -{% include {{ page.version.version }}/known-limitations/copy-from-clients.md %} - -### `COPY` syntax not supported by CockroachDB - -{% include {{ page.version.version }}/known-limitations/copy-syntax.md %} - -## Example - -The following example copies data from the PostgreSQL [`psql` client](https://www.postgresql.org/docs/current/app-psql.html) into a demo CockroachDB cluster. To follow along, make sure that you have [PostgreSQL](https://www.postgresql.org/download/) installed. - -Run [`cockroach demo`](cockroach-demo.html) to start a temporary, in-memory cluster with the [`movr` database](movr.html) preloaded: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach demo -~~~ - -Take note of the `(sql)` connection string listed under `Connection parameters` in the welcome message of the demo cluster's SQL shell: - -~~~ -# Connection parameters: -... -# (sql) postgres://demo:demo11762@127.0.0.1:26257?sslmode=require -~~~ - -Open a new terminal window, and connect to your demo cluster with `psql`, using the connection string provided for the demo cluster, with the `movr` database specified: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ psql postgres://demo:demo11762@127.0.0.1:26257?sslmode=require -~~~ - -In the `psql` shell, run the following command to start copying data from `psql` to the `users` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> movr=# COPY users FROM STDIN; -~~~ - -The following prompt should appear: - -~~~ -Enter data to be copied followed by a newline. -End with a backslash and a period on a line by itself, or an EOF signal. -~~~ - -Enter some tab-delimited data that you want copied to the `users` table: - -{% include_cached copy-clipboard.html %} -~~~ ->> 8a3d70a3-d70a-4000-8000-00000000001d seattle Hannah 400 Broad St 0987654321 ->> 9eb851eb-851e-4800-8000-00000000001e new york Carl 53 W 23rd St 5678901234 ->> \. -~~~ - -~~~ -COPY 2 -~~~ - -In the demo cluster's shell, query the `users` table for the rows that you just inserted: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM users WHERE id IN ('8a3d70a3-d70a-4000-8000-00000000001d', '9eb851eb-851e-4800-8000-00000000001e'); -~~~ - -~~~ - id | city | name | address | credit_card ----------------------------------------+----------+--------+--------------+-------------- - 8a3d70a3-d70a-4000-8000-00000000001d | seattle | Hannah | 400 Broad St | 0987654321 - 9eb851eb-851e-4800-8000-00000000001e | new york | Carl | 53 W 23rd St | 5678901234 -(2 rows) -~~~ - -## See also - -- [`IMPORT`](import.html) -- [`IMPORT INTO`](import-into.html) -- [`EXPORT`](export.html) -- [Migrate from Postgres](migrate-from-postgres.html) -- [Migration Overview](migration-overview.html) diff --git a/src/current/v21.1/cost-based-optimizer.md b/src/current/v21.1/cost-based-optimizer.md deleted file mode 100644 index 748eadac421..00000000000 --- a/src/current/v21.1/cost-based-optimizer.md +++ /dev/null @@ -1,212 +0,0 @@ ---- -title: Cost-Based Optimizer -summary: The cost-based optimizer seeks the lowest cost for a query, usually related to time. -toc: true -keywords: gin, gin index, gin indexes, inverted index, inverted indexes, accelerated index, accelerated indexes ---- - -The cost-based optimizer seeks the lowest cost for a query, usually related to time. - -## How is cost calculated? - -A given SQL query can have thousands of equivalent query plans with vastly different execution times. The cost-based optimizer enumerates these plans and chooses the lowest cost plan. - -Cost is roughly calculated by: - -- Estimating how much time each node in the query plan will use to process all results -- Modeling how data flows through the query plan - -The most important factor in determining the quality of a plan is cardinality (i.e., the number of rows); the fewer rows each SQL operator needs to process, the faster the query will run. - -## Table statistics - -The cost-based optimizer can often find more performant query plans if it has access to statistical data on the contents of your tables. This data needs to be generated from scratch for new tables, and regenerated periodically for existing tables. - -By default, CockroachDB automatically generates table statistics when tables are [created](create-table.html), and as they are [updated](update.html). It does this using a [background job](create-statistics.html#view-statistics-jobs) that automatically determines which columns to get statistics on — specifically, it chooses: - -- Columns that are part of the primary key or an index (in other words, all indexed columns). -- Up to 100 non-indexed columns. - -By default, CockroachDB also automatically collects [multi-column statistics](create-statistics.html#create-statistics-on-multiple-columns) on columns that prefix an index. - -{{site.data.alerts.callout_info}} -[Schema changes](online-schema-changes.html) trigger automatic statistics collection for the affected table(s). -{{site.data.alerts.end}} - -### Control automatic statistics - -For best query performance, most users should leave automatic statistics enabled with the default settings. The information provided in this section is useful for troubleshooting or performance tuning by advanced users. - -#### Control statistics refresh rate - -Statistics are refreshed in the following cases: - -- When there are no statistics. -- When it's been a long time since the last refresh, where "long time" is defined according to a moving average of the time across the last several refreshes. -- After a successful [`IMPORT`](import.html) or [`RESTORE`](restore.html) into the table. -- After any schema change affecting the table. -- After each mutation operation ([`INSERT`](insert.html), [`UPDATE`](update.html), or [`DELETE`](delete.html)), the probability of a refresh is calculated using a formula that takes the [cluster settings](cluster-settings.html) shown in the following table as inputs. These settings define the target number of rows in a table that should be stale before statistics on that table are refreshed. Increasing either setting will reduce the frequency of refreshes. In particular, `min_stale_rows` impacts the frequency of refreshes for small tables, while `fraction_stale_rows` has more of an impact on larger tables. - - - | Setting | Default Value | Details | - |------------------------------------------------------+---------------+--------------------------------------------------------------------------------------| - | `sql.stats.automatic_collection.fraction_stale_rows` | 0.2 | Target fraction of stale rows per table that will trigger a statistics refresh | - | `sql.stats.automatic_collection.min_stale_rows` | 500 | Target minimum number of stale rows per table that will trigger a statistics refresh | - - {{site.data.alerts.callout_info}} - Because the formula for statistics refreshes is probabilistic, you will not see statistics update immediately after changing these settings, or immediately after exactly 500 rows have been updated. - {{site.data.alerts.end}} - -#### Turn off statistics - -To turn off automatic statistics collection, follow these steps: - -1. Run the following statement to disable the automatic statistics [cluster setting](cluster-settings.html): - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING sql.stats.automatic_collection.enabled = false; - ~~~ - -1. Use the [`SHOW STATISTICS`](show-statistics.html) statement to view automatically generated statistics. - -1. Delete the automatically generated statistics using the following statement: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > DELETE FROM system.table_statistics WHERE true; - ~~~ - -1. Restart the nodes in your cluster to clear the statistics caches. - -To see how to manually generate statistics, see the [`CREATE STATISTICS` examples](create-statistics.html#examples). - -#### Control histogram collection - -By default, the optimizer collects histograms for all index columns (specifically the first column in each index) during automatic statistics collection. If a single column statistic is explicitly requested using manual invocation of [`CREATE STATISTICS`](create-statistics.html), a histogram will be collected, regardless of whether or not the column is part of an index. - -{{site.data.alerts.callout_info}} -- CockroachDB does not support histograms on [`ARRAY`-typed](array.html) columns. As a result, statistics created on `ARRAY`-typed columns do not include histograms. -- CockroachDB does not support multi-column histograms. -{{site.data.alerts.end}} - -If you are an advanced user and need to disable histogram collection for troubleshooting or performance tuning reasons, change the [`sql.stats.histogram_collection.enabled` cluster setting](cluster-settings.html) by running [`SET CLUSTER SETTING`](set-cluster-setting.html) as follows: - -{% include_cached copy-clipboard.html %} -~~~ sql -SET CLUSTER SETTING sql.stats.histogram_collection.enabled = false; -~~~ - -When `sql.stats.histogram_collection.enabled` is set to `false`, histograms are never collected, either as part of automatic statistics collection or by manually invoking [`CREATE STATISTICS`](create-statistics.html). - -## Query plan cache - -CockroachDB uses a cache for the query plans generated by the optimizer. This can lead to faster query execution since the database can reuse a query plan that was previously calculated, rather than computing a new plan each time a query is executed. - -The query plan cache is enabled by default. To disable it, execute the following statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET CLUSTER SETTING sql.query_cache.enabled = false; -~~~ - -Only the following statements use the plan cache: - -- [`SELECT`](select-clause.html) -- [`INSERT`](insert.html) -- [`UPDATE`](update.html) -- [`UPSERT`](upsert.html) -- [`DELETE`](delete.html) - -## Join reordering - -For a query involving multiple joins, the cost-based optimizer will explore additional [join orderings](joins.html) in an attempt to find the lowest-cost execution plan, which can lead to significantly better performance in some cases. - -Because this process leads to an exponential increase in the number of possible execution plans for such queries, it's only used to reorder subtrees containing 4 or fewer joins by default. - -To change this setting, which is controlled by the `reorder_joins_limit` [session variable](set-vars.html), run the following statement. To disable this feature, set the variable to `0`. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET reorder_joins_limit = 6; -~~~ - -{{site.data.alerts.callout_danger}} -We strongly recommend not setting this value higher than 8 to avoid performance degradation. If set too high, the cost of generating and costing execution plans can end up dominating the total execution time of the query. -{{site.data.alerts.end}} - -For more information about the difficulty of selecting an optimal join ordering, see our blog post [An Introduction to Join Ordering](https://www.cockroachlabs.com/blog/join-ordering-pt1/). - -## Join hints - -To force the use of a specific join algorithm even if the optimizer determines that a different plan would have a lower cost, you can use a _join hint_. You specify a join hint as ` JOIN`. For example: - -- `INNER HASH JOIN` -- `OUTER MERGE JOIN` -- `LEFT LOOKUP JOIN` -- `CROSS HASH JOIN` -- `INNER INVERTED JOIN` -- `LEFT INVERTED JOIN` - -{{site.data.alerts.callout_info}} -Due to SQL's implicit `AS` syntax, you cannot specify a join hint with only the join algorithm keyword (e.g., `MERGE`). For example, `a MERGE JOIN b` will be interpreted as having an implicit `AS` and be executed as `a AS MERGE JOIN b`, which is equivalent to `a JOIN b`. Because the resulting query might execute without returning any hint-related error (because it is valid SQL), it will seem like the join hint "worked", but actually it didn't affect which join algorithm was used. The correct syntax is `a INNER MERGE JOIN b`. -{{site.data.alerts.end}} - -For a join hint example, see [Use the right join type](make-queries-fast.html#rule-3-use-the-right-join-type). - -### Supported join algorithms - -- `HASH`: Forces a hash join; in other words, it disables merge and lookup joins. A hash join is always possible, even if there are no equality columns - CockroachDB considers the nested loop join with no index a degenerate case of the hash join (i.e., a hash table with one bucket). - -- `MERGE`: Forces a merge join, even if it requires re-sorting both sides of the join. - -- `LOOKUP`: Forces a lookup join into the right side; the right side must be a table with a suitable index. Note that `LOOKUP` can only be used with `INNER` and `LEFT` joins. - -- `INVERTED`: Forces an inverted join into the right side; the right side must be a table with a suitable [GIN index](inverted-indexes.html). Note that `INVERTED` can only be used with `INNER` and `LEFT` joins. - - {{site.data.alerts.callout_info}} - You cannot use inverted joins on [partial GIN indexes](inverted-indexes.html#partial-gin-indexes). - {{site.data.alerts.end}} - -If it is not possible to use the algorithm specified in the hint, an error is signaled. - -{{site.data.alerts.callout_info}} -To make the optimizer prefer lookup joins to merge joins when performing foreign key checks, set the `prefer_lookup_joins_for_fks` [session variable](set-vars.html) to `on`. -{{site.data.alerts.end}} - -### Additional considerations - -- This syntax is consistent with the [SQL Server syntax for join hints](https://docs.microsoft.com/en-us/sql/t-sql/queries/hints-transact-sql-join?view=sql-server-2017), except that: - - - SQL Server uses `LOOP` instead of `LOOKUP`. - - - CockroachDB does not support `LOOP` and instead supports `LOOKUP` for the specific case of nested loop joins with an index. - -- When you specify a join hint, the two tables will not be reordered by the optimizer. The reordering behavior has the following characteristics, which can be affected by hints: - - - Given `a JOIN b`, CockroachDB will not try to commute to `b JOIN a`. This means that you will need to pay attention to this ordering, which is especially important for lookup joins. Without a hint, `a JOIN b` might be executed as `b INNER LOOKUP JOIN a` using an index into `a`, whereas `a INNER LOOKUP JOIN b` requires an index into `b`. - - - `(a JOIN b) JOIN c` might be changed to `a JOIN (b JOIN c)`, but this does not happen if `a JOIN b` uses a hint; the hint forces that particular join to happen as written in the query. - -- You should reconsider hint usage with each new release of CockroachDB. Due to improvements in the optimizer, hints specified to work with an older version may cause decreased performance in a newer version. - -## Inverted join examples - -{% include {{ page.version.version }}/sql/inverted-joins.md %} - -## Known Limitations - -* {% include {{page.version.version}}/known-limitations/old-multi-col-stats.md %} -* {% include {{page.version.version}}/known-limitations/single-col-stats-deletion.md %} -* {% include {{page.version.version}}/known-limitations/stats-refresh-upgrade.md %} - -## See also - -- [`JOIN` expressions](joins.html) -- [`SET (session variable)`](set-vars.html) -- [`SET CLUSTER SETTING`](set-cluster-setting.html) -- [`RESET CLUSTER SETTING`](reset-cluster-setting.html) -- [`SHOW (session variable)`](show-vars.html) -- [`CREATE STATISTICS`](create-statistics.html) -- [`SHOW STATISTICS`](show-statistics.html) -- [`EXPLAIN`](explain.html) diff --git a/src/current/v21.1/crdb-internal.md b/src/current/v21.1/crdb-internal.md deleted file mode 100644 index 0e31ee42690..00000000000 --- a/src/current/v21.1/crdb-internal.md +++ /dev/null @@ -1,130 +0,0 @@ ---- -title: crdb_internal -summary: The crdb_internal schema contains read-only views that you can use for introspection into CockroachDB internals. -toc: true ---- - -The `crdb_internal` [system catalog](system-catalogs.html) is a schema that contains information about internal objects, processes, and metrics related to a specific database. - -{{site.data.alerts.callout_danger}} -We do not recommend using `crdb_internal` tables in production environments for the following reasons: - -- The contents of `crdb_internal` are unstable, and subject to change in new releases of CockroachDB, without prior notice. -- There are memory and latency costs associated with each table in `crdb_internal`. Accessing the tables in the schema can impact cluster stability and performance. -{{site.data.alerts.end}} - -## Data exposed by `crdb_internal` - -Each table in `crdb_internal` corresponds to an internal object, process, or metric, for a specific database. `crdb_internal` tables are read-only. - -In CockroachDB {{ page.version.version }}, `crdb_internal` includes the following tables: - -Table | Description --------|------- -`backward_dependencies` | Contains information about backward dependencies. -`builtin_functions` | Contains information about supported [functions](functions-and-operators.html). -`cluster_contention_events` | Contains information about contention in your cluster. -`cluster_database_privileges` | Contains information about the [database privileges](authorization.html#privileges) on your cluster. -`cluster_queries` | Contains information about queries running on your cluster. -`cluster_sessions` | Contains information about cluster sessions, including current and past queries. -`cluster_settings` | Contains information about [cluster settings](cluster-settings.html). -`cluster_transactions` | Contains information about transactions running on your cluster. -`create_statements` | Contains information about tables and indexes in your database. -`create_type_statements` | Contains information about [user-defined types](enum.html) in your database. -`cross_db_references` | Contains information about cross-database references in your cluster. -`databases` | Contains information about the databases in your cluster. -`feature_usage` | Contains information about feature usage on your cluster. -`forward_dependencies` | Contains information about forward dependencies. -`gossip_alerts` | Contains information about gossip alerts. -`gossip_liveness` | Contains information about your cluster's gossip liveness. -`gossip_network` | Contains information about your cluster's gossip network. -`gossip_nodes` | Contains information about nodes in your cluster's gossip network. -`index_columns` | Contains information about indexed columns in your cluster. -`interleaved_tables` | Contains information about interleaved tables in your cluster. -`interleaved_views` | Contains information about interleaved views in your cluster. -`invalid_objects` | Contains information about invalid objects in your cluster. -`jobs` | Contains information about [jobs](show-jobs.html) running on your cluster. -`kv_node_status` | Contains information about node status at the [key-value layer](architecture/storage-layer.html). -`kv_store_status` | Contains information about the key-value store for your cluster. -`leases` | Contains information about [leases](architecture/replication-layer.html#leases) in your cluster. -`node_build_info` | Contains information about nodes in your cluster. -`node_contention_events` | Contains information about contention on the gateway node of your cluster. -`node_inflight_trace_spans` | Contains information about currently in-flight spans in the current node. -`node_metrics` | Contains metrics for nodes in your cluster. -`node_queries` | Contains information about queries running on nodes in your cluster. -`node_runtime_info` | Contains runtime information about nodes in your cluster. -`node_sessions` | Contains information about sessions to nodes in your cluster. -`node_statement_statistics` | Contains statement statistics for nodes in your cluster. -`node_transaction_statistics` | Contains transaction statistics for nodes in your cluster. -`node_transactions` | Contains information about transactions for nodes in your cluster. -`node_txn_stats` | Contains transaction statistics for nodes in your cluster. -`partitions` | Contains information about [partitions](partitioning.html) in your cluster. -`predefined_comments` | Contains predefined comments about your cluster. -`ranges` | Contains information about ranges in your cluster. -`ranges_no_leases` | Contains information about ranges in your cluster, without leases. -`schema_changes` | Contains information about schema changes in your cluster. -`session_trace` | Contains session trace information for your cluster. -`session_variables` | Contains information about [session variables](set-vars.html) in your cluster. -`table_columns` | Contains information about table columns in your cluster. -`table_indexes` | Contains information about table indexes in your cluster. -`table_row_statistics` | Contains row count statistics for tables in the current database. -`tables` | Contains information about tables in your cluster. -`zones` | Contains information about [zone configurations](configure-replication-zones.html) in your cluster. - -To list the `crdb_internal` tables for the [current database](sql-name-resolution.html#current-database), use the following [`SHOW TABLES`](show-tables.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM crdb_internal; -~~~ - -~~~ - schema_name | table_name | type | owner | estimated_row_count -----------------+-----------------------------+-------+-------+---------------------- - crdb_internal | backward_dependencies | table | NULL | NULL - crdb_internal | builtin_functions | table | NULL | NULL - ... -~~~ - -## Querying `crdb_internal` tables - -To get detailed information about objects, processes, or metrics related to your database, you can read from the `crdb_internal` table that corresponds to the item of interest. - -{{site.data.alerts.callout_success}} -To ensure that you can view all of the tables in `crdb_internal`, query the tables as a user with [`admin` privileges](authorization.html#admin-role). -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -Unless specified otherwise, queries to `crdb_internal` assume the [current database](sql-name-resolution.html#current-database). -{{site.data.alerts.end}} - -For example, to return the `crdb_internal` table for the ranges of the [`movr`](movr.html) database, you can use the following statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM movr.crdb_internal.ranges; -~~~ - -~~~ - range_id | start_key | start_pretty | end_key | end_pretty | database_name | table_name | index_name | replicas | replica_localities | learner_replicas | split_enforced_until | lease_holder | range_size ------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------+---------------+---------------------------------+------------+----------+--------------------------+------------------+----------------------------------+--------------+------------- - 1 | | /Min | \004\000liveness- | /System/NodeLiveness | | | | {1} | {"region=us-east1,az=b"} | {} | NULL | 1 | 8323 - 2 | \004\000liveness- | /System/NodeLiveness | \004\000liveness. | /System/NodeLivenessMax | | | | {1} | {"region=us-east1,az=b"} | {} | NULL | 1 | 9278 - 3 | \004\000liveness. | /System/NodeLivenessMax | \004tsd | /System/tsd | | | | {1} | {"region=us-east1,az=b"} | {} | NULL | 1 | 32377 - 4 | \004tsd | /System/tsd | \004tse | /System/"tse" | | | | {1} | {"region=us-east1,az=b"} | {} | NULL | 1 | 4067446 -(65 rows) -~~~ - -## See also - -- [`SHOW`](show-vars.html) -- [`SHOW COLUMNS`](show-columns.html) -- [`SHOW CONSTRAINTS`](show-constraints.html) -- [`SHOW CREATE`](show-create.html) -- [`SHOW DATABASES`](show-databases.html) -- [`SHOW GRANTS`](show-grants.html) -- [`SHOW INDEX`](show-index.html) -- [`SHOW SCHEMAS`](show-schemas.html) -- [`SHOW TABLES`](show-tables.html) -- [SQL Name Resolution](sql-name-resolution.html) -- [System Catalogs](system-catalogs.html) diff --git a/src/current/v21.1/create-changefeed.md b/src/current/v21.1/create-changefeed.md deleted file mode 100644 index cfffa089ca5..00000000000 --- a/src/current/v21.1/create-changefeed.md +++ /dev/null @@ -1,425 +0,0 @@ ---- -title: CREATE CHANGEFEED -summary: The CREATE CHANGEFEED statement creates a changefeed of row-level change subscriptions in a configurable format to a configurable sink. -toc: true ---- - -{{site.data.alerts.callout_info}} -`CREATE CHANGEFEED` is an [Enterprise-only](enterprise-licensing.html) feature. For the core version, see [`EXPERIMENTAL CHANGEFEED FOR`](changefeed-for.html). -{{site.data.alerts.end}} - -The `CREATE CHANGEFEED` [statement](sql-statements.html) creates a new Enterprise changefeed, which targets an allowlist of tables, called "watched rows". Every change to a watched row is emitted as a record in a configurable format (`JSON` or Avro) to a configurable sink ([Kafka](https://kafka.apache.org/) or a [cloud storage sink](#cloud-storage-sink)). You can [create](#create-a-changefeed-connected-to-kafka), [pause](#pause-a-changefeed), [resume](#resume-a-paused-changefeed), or [cancel](#cancel-a-changefeed) an Enterprise changefeed. - -For more information, see [Stream Data Out of CockroachDB Using Changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html). - -## Required privileges - -To create a changefeed, the user must be a member of the `admin` role or have the [`CREATECHANGEFEED`](create-user.html#create-a-user-that-can-control-changefeeds) parameter set. - -## Synopsis - -
-{% include {{ page.version.version }}/sql/generated/diagrams/create_changefeed.html %} -
- -## Parameters - -Parameter | Description -----------|------------ -`table_name` | The name of the table (or tables in a comma separated list) to create a changefeed for.

**Note:** Changefeeds do not share internal buffers, so each running changefeed will increase total memory usage. To watch multiple tables, we recommend creating a changefeed with a comma-separated list of tables. -`sink` | The location of the configurable sink. The scheme of the URI indicates the type. For more information, see [Sink URI](#sink-uri) below. -`option` / `value` | For a list of available options and their values, see [Options](#options) below. - - - -### Sink URI - -The sink URI follows the basic format of: - -~~~ -'{scheme}://{host}:{port}?{query_parameters}' -~~~ - -URI Component | Description --------------------+------------------------------------------------------------------ -`scheme` | The type of sink: [`kafka`](#kafka) or any [cloud storage sink](#cloud-storage-sink). -`host` | The sink's hostname or IP address. -`port` | The sink's port. -`query_parameters` | The sink's [query parameters](#query-parameters). - -#### Kafka - -Example of a Kafka sink URI: - -~~~ -'kafka://broker.address.com:9092?topic_prefix=bar_&tls_enabled=true&ca_cert=LS0tLS1CRUdJTiBDRVJUSUZ&sasl_enabled=true&sasl_user={sasl user}&sasl_password={url-encoded password}&sasl_mechanism=SASL-SCRAM-SHA-256' -~~~ - -#### Cloud storage sink - -Use a cloud storage sink to deliver changefeed data to OLAP or big data systems without requiring transport via Kafka. - -{{site.data.alerts.callout_info}} -Currently, cloud storage sinks only work with `JSON` and emits newline-delimited `JSON` files. -{{site.data.alerts.end}} - -Example of a cloud storage sink URI: - -~~~ -`experimental-s3://acme-co/employees?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456` -~~~ - -Cloud storage sink URIs must be pre-pended with `experimental-` when working with changefeeds. For more information on the sink URI structure, see [Use Cloud Storage for Bulk Operations](use-cloud-storage-for-bulk-operations.html#example-file-urls). - -#### Query parameters - -{% include {{ page.version.version }}/cdc/url-encoding.md %} - -Query parameters include: - -Parameter |
Sink Type
|
Type
| Description --------------------+-----------------------------------------------+-------------------------------------+------------------------------------------------------------ -`topic_prefix` | [Kafka](#kafka), [cloud](#cloud-storage-sink) | [`STRING`](string.html) | Adds a prefix to all topic names.

For example, `CREATE CHANGEFEED FOR TABLE foo INTO 'kafka://...?topic_prefix=bar_'` would emit rows under the topic `bar_foo` instead of `foo`. -`tls_enabled` | [Kafka](#kafka) | [`BOOL`](bool.html) | If `true`, enable Transport Layer Security (TLS) on the connection to Kafka. This can be used with a `ca_cert` (see below).

**Default:** `false` -`ca_cert` | [Kafka](#kafka) | [`STRING`](string.html) | The base64-encoded `ca_cert` file.

Note: To encode your `ca.cert`, run `base64 -w 0 ca.cert`. -`client_cert` | [Kafka](#kafka) | [`STRING`](string.html) | The base64-encoded Privacy Enhanced Mail (PEM) certificate. This is used with `client_key`. -`client_key` | [Kafka](#kafka) | [`STRING`](string.html) | The base64-encoded private key for the PEM certificate. This is used with `client_cert`.

{% include {{ page.version.version }}/cdc/client-key-encryption.md %} -`sasl_enabled` | [Kafka](#kafka) | [`BOOL`](bool.html) | If `true`, the authentication protocol can be set to SCRAM or PLAIN using the `sasl_mechanism` parameter. You must have `tls_enabled` set to `true` to use SASL.

**Default:** `false` -`sasl_mechanism` | [Kafka](#kafka) | [`STRING`](string.html) | Can be set to [`SASL-SCRAM-SHA-256`](https://docs.confluent.io/platform/current/kafka/authentication_sasl/authentication_sasl_scram.html), [`SASL-SCRAM-SHA-512`](https://docs.confluent.io/platform/current/kafka/authentication_sasl/authentication_sasl_scram.html), or [`SASL-PLAIN`](https://docs.confluent.io/current/kafka/authentication_sasl/authentication_sasl_plain.html). A `sasl_user` and `sasl_password` are required.

**Default:** `SASL-PLAIN` -`sasl_user` | [Kafka](#kafka) | [`STRING`](string.html) | Your SASL username. -`sasl_password` | [Kafka](#kafka) | [`STRING`](string.html) | Your SASL password. **Note:** Passwords should be [URL encoded](https://en.wikipedia.org/wiki/Percent-encoding) since the value can contain characters that would cause authentication to fail. -`file_size` | [cloud](#cloud-storage-sink) | [`STRING`](string.html) | The file will be flushed (i.e., written to the sink) when it exceeds the specified file size. This can be used with the [`WITH resolved` option](#options), which flushes on a specified cadence.

**Default:** `16MB` -`insecure_tls_skip_verify` | [Kafka](#kafka) | [`BOOL`](bool.html) | If `true`, disable client-side validation of responses. **Warning:** Use this query parameter with caution, as it creates [MITM](https://en.wikipedia.org/wiki/Man-in-the-middle_attack) vulnerabilities unless combined with another method of authentication.

**Default:** `false` - -### Options - -Option | Value | Description --------|-------|------------ -`updated` | N/A | Include updated timestamps with each row.

If a `cursor` is provided, the "updated" timestamps will match the [MVCC](architecture/storage-layer.html#mvcc) timestamps of the emitted rows, and there is no initial scan. If a `cursor` is not provided, the changefeed will perform an initial scan (as of the time the changefeed was created), and the "updated" timestamp for each change record emitted in the initial scan will be the timestamp of the initial scan. Similarly, when a [backfill is performed for a schema change](stream-data-out-of-cockroachdb-using-changefeeds.html#schema-changes-with-column-backfill), the "updated" timestamp is set to the first timestamp for when the new schema is valid. -`resolved` | [Duration string](https://pkg.go.dev/time#ParseDuration) | Emits [resolved timestamp](stream-data-out-of-cockroachdb-using-changefeeds.html#resolved-def) events per changefeed in a format dependent on the connected sink. Resolved timestamp events do not emit until all ranges in the changefeed have progressed to a specific point in time.

Set an optional minimal duration between emitting resolved timestamps. Example: `resolved='10s'`. This option will **only** emit a resolved timestamp event if the timestamp has advanced and at least the optional duration has elapsed. If unspecified, all resolved timestamps are emitted as the high-water mark advances. -`envelope` | `key_only` / `wrapped` | Use `key_only` to emit only the key and no value, which is faster if you only want to know when the key changes.

Default: `envelope=wrapped` -`cursor` | [Timestamp](as-of-system-time.html#parameters) | Emit any changes after the given timestamp, but does not output the current state of the table first. If `cursor` is not specified, the changefeed starts by doing an initial scan of all the watched rows and emits the current value, then moves to emitting any changes that happen after the scan.

When starting a changefeed at a specific `cursor`, the `cursor` cannot be before the configured garbage collection window (see [`gc.ttlseconds`](configure-replication-zones.html#replication-zone-variables)) for the table you're trying to follow; otherwise, the changefeed will error. With default garbage collection settings, this means you cannot create a changefeed that starts more than 25 hours in the past.

`cursor` can be used to [start a new changefeed where a previous changefeed ended.](#start-a-new-changefeed-where-another-ended)

Example: `CURSOR='1536242855577149065.0000000000'` -`format` | `json` / `experimental_avro` | Format of the emitted record. Currently, support for [Avro is limited and experimental](#avro-limitations). For mappings of CockroachDB types to Avro types, [see the table below](#avro-types).

Default: `format=json`. -`confluent_schema_registry` | Schema Registry address | The [Schema Registry](https://docs.confluent.io/current/schema-registry/docs/index.html#sr) address is required to use `experimental_avro`. -`key_in_value` | N/A | Make the [primary key](primary-key.html) of a deleted row recoverable in sinks where each message has a value but not a key (most have a key and value in each message). `key_in_value` is automatically used for these sinks (currently only [cloud storage sinks](#cloud-storage-sink)). -`diff` | N/A | Publish a `before` field with each message, which includes the value of the row before the update was applied. -`compression` | `gzip` | Compress changefeed data files written to a [cloud storage sink](#cloud-storage-sink). Currently, only [Gzip](https://www.gnu.org/software/gzip/) is supported for compression. -`protect_data_from_gc_on_pause` | N/A | When a [changefeed is paused](pause-job.html), ensure that the data needed to [resume the changefeed](resume-job.html) is not garbage collected.

Note: If you use this option, changefeeds left paused can prevent garbage collection for long periods of time. -`schema_change_events` | `default` / `column_changes` | The type of schema change event that triggers the behavior specified by the `schema_change_policy` option:
  • `default`: Include all [`ADD COLUMN`](add-column.html) events for columns that have a non-`NULL` [`DEFAULT` value](default-value.html) or are [computed](computed-columns.html), and all [`DROP COLUMN`](drop-column.html) events.
  • `column_changes`: Include all schema change events that add or remove any column.

Default: `schema_change_events=default` -`schema_change_policy` | `backfill` / `nobackfill` / `stop` | The behavior to take when an event specified by the `schema_change_events` option occurs:
  • `backfill`: When [schema changes with column backfill](stream-data-out-of-cockroachdb-using-changefeeds.html#schema-changes-with-column-backfill) are finished, output all watched rows using the new schema.
  • `nobackfill`: For [schema changes with column backfill](stream-data-out-of-cockroachdb-using-changefeeds.html#schema-changes-with-column-backfill), perform no logical backfills. The changefeed will still emit any duplicate records for the table being altered, but will not emit the new schema records.
  • `stop`: [schema changes with column backfill](stream-data-out-of-cockroachdb-using-changefeeds.html#schema-changes-with-column-backfill), wait for all data preceding the schema change to be resolved before exiting with an error indicating the timestamp at which the schema change occurred. An `error: schema change occurred at ` will display in the `cockroach.log` file.

Default: `schema_change_policy=backfill` -`initial_scan` / `no_initial_scan` | N/A | Control whether or not an initial scan will occur at the start time of a changefeed. `initial_scan` and `no_initial_scan` cannot be used simultaneously. If neither `initial_scan` nor `no_initial_scan` is specified, an initial scan will occur if there is no `cursor`, and will not occur if there is one. This preserves the behavior from previous releases.

Default: `initial_scan`
If used in conjunction with `cursor`, an initial scan will be performed at the cursor timestamp. If no `cursor` is specified, the initial scan is performed at `now()`. -`full_table_name` | N/A | **New in v21.1:** Use fully-qualified table name in topics, subjects, schemas, and record output instead of the default table name. This can prevent unintended behavior when the same table name is present in multiple databases.

Example: `CREATE CHANGEFEED FOR foo... WITH full_table_name` will create the topic name `defaultdb.public.foo` instead of `foo`. -`avro_schema_prefix` | Schema prefix name | **New in v21.1:** Provide a namespace for the schema of a table in addition to the default, the table name. This allows multiple databases or clusters to share the same schema registry when the same table name is present in multiple databases.

Example: `CREATE CHANGEFEED FOR foo WITH format=avro, confluent_schema_registry='registry_url', avro_schema_prefix='super'` will register subjects as `superfoo-key` and `superfoo-value` with the namespace `super`. -`kafka_sink_config` | [`STRING`](string.html) | Set fields to configure the required level of message acknowledgement from the Kafka server, the version of the server, and batching parameters for Kafka sinks. See [Advanced Configuration](#kafka-sink-configuration) for more detail on configuring all the available fields for this option.

Example: `CREATE CHANGEFEED FOR table INTO 'kafka://localhost:9092' WITH kafka_sink_config='{"Flush": {"MaxMessages": 1, "Frequency": "1s"}, "RequiredAcks": "ONE"}'` - -{{site.data.alerts.callout_info}} - Using the `format=experimental_avro`, `envelope=key_only`, and `updated` options together is rejected. `envelope=key_only` prevents any rows with updated fields from being emitted, which makes the `updated` option meaningless. -{{site.data.alerts.end}} - -#### Avro limitations - -Currently, support for Avro is limited and experimental. - -Below are clarifications for particular SQL types and values for Avro changefeeds: - - - [Decimals](decimal.html) must have precision specified. - - [`BIT`](bit.html) and [`VARBIT`](bit.html) types are encoded as arrays of 64-bit integers. - - {% include {{ page.version.version }}/cdc/avro-bit-varbit.md %} - -#### Avro types - -Below is a mapping of CockroachDB types to Avro types: - -CockroachDB Type | Avro Type | Avro Logical Type ------------------+-----------+--------------------- -[`INT`](int.html) | [`LONG`](https://avro.apache.org/docs/1.8.1/spec.html#schema_primitive) | -[`BOOL`](bool.html) | [`BOOLEAN`](https://avro.apache.org/docs/1.8.1/spec.html#schema_primitive) | -[`FLOAT`](float.html) | [`DOUBLE`](https://avro.apache.org/docs/1.8.1/spec.html#schema_primitive) | -[`STRING`](string.html) | [`STRING`](https://avro.apache.org/docs/1.8.1/spec.html#schema_primitive) | -[`DATE`](date.html) | [`INT`](https://avro.apache.org/docs/1.8.1/spec.html#schema_primitive) | [`DATE`](https://avro.apache.org/docs/1.8.1/spec.html#Date) -[`TIME`](time.html) | [`LONG`](https://avro.apache.org/docs/1.8.1/spec.html#schema_primitive) | [`TIME-MICROS`](https://avro.apache.org/docs/1.8.1/spec.html#Time+%28microsecond+precision%29) -[`TIMESTAMP`](timestamp.html) | [`LONG`](https://avro.apache.org/docs/1.8.1/spec.html#schema_primitive) | [`TIME-MICROS`](https://avro.apache.org/docs/1.8.1/spec.html#Time+%28microsecond+precision%29) -[`TIMESTAMPTZ`](timestamp.html) | [`LONG`](https://avro.apache.org/docs/1.8.1/spec.html#schema_primitive) | [`TIME-MICROS`](https://avro.apache.org/docs/1.8.1/spec.html#Time+%28microsecond+precision%29) -[`DECIMAL`](decimal.html) | [`BYTES`](https://avro.apache.org/docs/1.8.1/spec.html#schema_primitive) | [`DECIMAL`](https://avro.apache.org/docs/1.8.1/spec.html#Decimal) -[`UUID`](uuid.html) | [`STRING`](https://avro.apache.org/docs/1.8.1/spec.html#schema_primitive) | -[`INET`](inet.html) | [`STRING`](https://avro.apache.org/docs/1.8.1/spec.html#schema_primitive) | -[`JSONB`](jsonb.html) | [`STRING`](https://avro.apache.org/docs/1.8.1/spec.html#schema_primitive) | -[`ENUMS`](enum.html) | [`STRING`](https://avro.apache.org/docs/1.8.1/spec.html#schema_primitive) | -[`INTERVAL`](interval.html) | [`STRING`](https://avro.apache.org/docs/1.8.1/spec.html#schema_primitive) | -[`ARRAY`](array.html) | [`ARRAY`](https://avro.apache.org/docs/1.8.1/spec.html#schema_complex) | -[`BIT`](bit.html) | Array of [`LONG`](https://avro.apache.org/docs/1.8.1/spec.html#schema_primitive) | -[`VARBIT`](bit.html)| Array of [`LONG`](https://avro.apache.org/docs/1.8.1/spec.html#schema_primitive) | -[`COLLATE`](collate.html) | [`STRING`](https://avro.apache.org/docs/1.8.1/spec.html#schema_primitive) | - -#### Topic Naming - -By default, a Kafka topic has the same name as the table that a changefeed was created on. If a changefeed was created on multiple tables, the changefeed will write to multiple topics corresponding to those table names. You can specify a [topic prefix](#query-parameters) or use the [`full_table_name` option](#options) to modify this. - -You can either manually create a topic in your Kafka cluster before starting the changefeed, or the topic will be automatically created when the changefeed connects to your Kafka cluster. - -{{site.data.alerts.callout_info}} -You must have the Kafka cluster setting [`auto.create.topics.enable`](https://kafka.apache.org/documentation/#brokerconfigs_auto.create.topics.enable) set to `true` for automatic topic creation. This will create the topic when the changefeed sends its first message. If you create the consumer before that, you will also need the Kafka consumer configuration [`allow.auto.create.topics`](https://kafka.apache.org/documentation/#consumerconfigs_allow.auto.create.topics) to be set to `true`. -{{site.data.alerts.end}} - -Kafka has the following topic limitations: - -- [Legal characters](https://github.com/apache/kafka/blob/0.10.2/core/src/main/scala/kafka/common/Topic.scala#L29) are numbers, letters, and `[._-]`. -- The maximum character length of a topic name is 249. -- Topics with a period (`.`) and underscore (`_`) can collide on internal Kafka data structures, so you should use either but not both. -- Characters not accepted by Kafka will be automatically encoded as unicode characters by CockroachDB. - -## Advanced changefeed configuration - -{{site.data.alerts.callout_danger}} -The configurations and settings explained in these sections will have significant impact on a changefeed's behavior. -{{site.data.alerts.end}} - -The following sections describe settings, configurations, and details to tune changefeeds for particular use cases: - -* [High durability delivery](#tuning-for-high-durability-delivery) -* [High throughput](#tuning-for-high-throughput) -* [Kafka sinks](#kafka-sink-configuration) - -### Tuning for high durability delivery - -When designing a system that relies on high durability of message delivery — that is, not missing any message acknowledgement at the downstream sink — consider the following settings and configuration. Before tuning these settings we recommend reading details on our [changefeed at-least-once-delivery guarantee](stream-data-out-of-cockroachdb-using-changefeeds.html#ordering-guarantees). - -* Increase the number of seconds before [garbage collection](architecture/storage-layer.html#garbage-collection) with the [`gc.ttlseconds`](configure-replication-zones.html#gc-ttlseconds) setting to provide a higher recoverability window for data if a changefeed fails. For example, if a sink is unavailable, changes are queued until the sink is available again. While the sink is unavailable, changes will be retried until the garbage collection window is reached and then the data is removed. - * You can also use the [`protect_data_from_gc_on_pause`](#protect-pause) option to protect the changes from garbage collection. -* Determine what a successful write to Kafka is with the [`kafka_sink_config: {"RequiredAcks": "ALL"}`](#kafka-required-acks) option, which provides the highest consistency level. -* Set the [`changefeed.memory.per_changefeed_limit`](cluster-settings.html) cluster setting to a higher limit to give more memory for buffering per single changefeed. Increasing this limit is useful to avoid errors in situations where the sink is unreachable and the memory buffer fills up. -* Use [Kafka](#kafka) or [cloud storage](#cloud-storage-sink) sinks when tuning for high durability in changefeeds. -* Ensure that data is ingested downstream in its new format after a schema change by using the [`schema_change_events`](#schema-events) and [`schema_schange_policy`](#schema-policy) options. For example, setting `schema_change_events=column_changes` and `schema_change_policy=stop` will trigger an error to the `cockroach.log` file on a [schema change](stream-data-out-of-cockroachdb-using-changefeeds.html#schema-changes-with-column-backfill) and the changefeed to fail. - -### Tuning for high throughput - -When designing a system that needs to emit a lot of changefeed messages, whether it be steady traffic or a burst in traffic, consider the following settings and configuration: - -* Set the [`resolved`](#resolved-option) option to a higher duration (e.g. 10 minutes or more) to reduce emitted messages. -* Batch messages to your sink. See the [`Flush`](#kafka-flush) parameter for the `kafka_sink_config` option. When using cloud storage sinks, use the [`file_size`](#file-size) parameter to flush a file when it exceeds the specified size. -* Set the [`changefeed.memory.per_changefeed_limit`](cluster-settings.html) cluster setting to a higher limit to give more memory for buffering for a changefeed. This is useful in situations of heavy traffic to avoid an error due to the memory buffer overfilling. -* Use `avro` as the emitted message [format](#format) option with Kafka sinks; JSON encoding can potentially create a slowdown. -* Use the [`compression` option](#compression-opt) in cloud storage sinks with JSON to compress the changefeed data files. -* Increase the [`changefeed.backfill.concurrent_scan_requests` setting](cluster-settings.html), which controls the number of concurrent scan requests per node issued during a backfill event. The default behavior, when this setting is at `0`, is that the number of scan requests will be 3 times the number of nodes in the cluster (to a maximum of 100). While increasing this number will allow for higher throughput, it **will increase the cluster load overall**, including CPU and IO usage. - -### Kafka Sink Configuration - -The `kafka_sink_config` option allows configuration of a changefeed's message delivery, Kafka server version, and batching parameters. - -{{site.data.alerts.callout_danger}} -Each of the following settings have significant impact on a changefeed's behavior, such as latency. For example, it is possible to configure batching parameters to be very high, which would negatively impact changefeed latency. As a result it would take a long time to see messages coming through to the sink. Also, large batches may be rejected by the Kafka server unless it's separately configured to accept a high [`max.message.bytes`](https://kafka.apache.org/documentation/#brokerconfigs_message.max.bytes). -{{site.data.alerts.end}} - -~~~ -kafka_sink_config='{"Flush": {"MaxMessages": 1, "Frequency": "1s"}, "Version": "0.8.2.0", "RequiredAcks": "ONE" }' -~~~ - -The configurable fields include: - - - -* `"Flush"."MaxMessages"` and `"Flush"."Frequency"`: These are configurable batching parameters depending on latency and throughput needs. For example, if `"MaxMessages"` is set to 1000 and `"Frequency"` to 1 second, it will flush to Kafka either after 1 second or after 1000 messages are batched, whichever comes first. It's important to consider that if there are not many messages then a `"1s"` frequency will add 1 second latency. However, if there is a larger influx of messages these will be flushed quicker. - * `"MaxMessages"`: sets the maximum number of messages the producer will send in a single broker request. Increasing this value allows all messages to be sent in a batch. Default: `1`. Type: `INT`. - * `"Frequency"`: sets the maximum time that messages will be batched. Default: `"0s"`. Type: `INTERVAL`. - -* `"Version"`: sets the appropriate Kafka cluster version, which can be used to connect to [Kafka versions < v1.0](https://docs.confluent.io/platform/current/installation/versions-interoperability.html) (`kafka_sink_config='{"Version": "0.8.2.0"}'`). Default: `"1.0.0.0"` Type: `STRING`. - - - -* `"RequiredAcks"`: specifies what a successful write to Kafka is. CockroachDB [guarantees at least once delivery of messages](stream-data-out-of-cockroachdb-using-changefeeds.html#ordering-guarantees) — this value defines the _delivery_. Type: `STRING`. The possible values are: - * `"ONE"`: a write to Kafka is successful once the leader node has committed and acknowledged the write. Note that this has the potential risk of dropped messages; if the leader node acknowledges before replicating to a quorum of other Kafka nodes, but then fails. **This is the default value.** - * `"NONE"`: no Kafka brokers are required to acknowledge that they have committed the message. This will decrease latency and increase throughput, but comes at the cost of lower consistency. - * `"ALL"`: a quorum must be reached (that is, most Kafka brokers have committed the message) before the leader can acknowledge. This is the highest consistency level. - -## Responses - -### Messages - -The messages (i.e., keys and values) emitted to a sink are specific to the [`envelope`](#options). The default format is `wrapped`, and the output messages are composed of the following: - -- **Key**: An array always composed of the row's `PRIMARY KEY` field(s) (e.g., `[1]` for `JSON` or `{"id":{"long":1}}` for Avro). -- **Value**: - - One of three possible top-level fields: - - `after`, which contains the state of the row after the update (or `null`' for `DELETE`s). - - `updated`, which contains the updated timestamp. - - `resolved`, which is emitted for records representing resolved timestamps. These records do not include an "after" value since they only function as checkpoints. - - For [`INSERT`](insert.html) and [`UPDATE`](update.html), the current state of the row inserted or updated. - - For [`DELETE`](delete.html), `null`. - -For example: - -Statement | Response ------------------------------------------------+----------------------------------------------------------------------- -`INSERT INTO office_dogs VALUES (1, 'Petee');` | JSON: `[1] {"after": {"id": 1, "name": "Petee"}}`
Avro: `{"id":{"long":1}} {"after":{"office_dogs":{"id":{"long":1},"name":{"string":"Petee"}}}}` -`DELETE FROM office_dogs WHERE name = 'Petee'` | JSON: `[1] {"after": null}`
Avro: `{"id":{"long":1}} {"after":null}` - -### Files - -The files emitted to a sink use the following naming conventions: - -- [General file format](#general-file-format) -- [Resolved file format](#resolved-file-format) - -{{site.data.alerts.callout_info}} -The timestamp format is `YYYYMMDDHHMMSSNNNNNNNNNLLLLLLLLLL`. -{{site.data.alerts.end}} - -#### General file format - -~~~ -/[date]/[timestamp]-[uniquer]-[topic]-[schema-id] -~~~ - -For example: - -~~~ -/2020-04-02/202004022058072107140000000000000-56087568dba1e6b8-1-72-00000000-test_table-1.ndjson -~~~ - -#### Resolved file format - -~~~ -/[date]/[timestamp].RESOLVED -~~~ - -For example: - -~~~ -/2020-04-04/202004042351304139680000000000000.RESOLVED -~~~ - -## Examples - -### Create a changefeed connected to Kafka - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE CHANGEFEED FOR TABLE name, name2, name3 - INTO 'kafka://host:port' - WITH updated, resolved; -~~~ -~~~ -+--------------------+ -| job_id | -+--------------------+ -| 360645287206223873 | -+--------------------+ -(1 row) -~~~ - -For more information on how to create a changefeed connected to Kafka, see [Stream Data Out of CockroachDB Using Changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html#create-a-changefeed-connected-to-kafka). - -### Create a changefeed connected to Kafka using Avro - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE CHANGEFEED FOR TABLE name, name2, name3 - INTO 'kafka://host:port' - WITH format = experimental_avro, confluent_schema_registry = ; -~~~ -~~~ -+--------------------+ -| job_id | -+--------------------+ -| 360645287206223873 | -+--------------------+ -(1 row) -~~~ - -For more information on how to create a changefeed that emits an [Avro](https://avro.apache.org/docs/1.8.2/spec.html) record, see [Stream Data Out of CockroachDB Using Changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html#create-a-changefeed-connected-to-kafka-using-avro). - -### Create a changefeed connected to a cloud storage sink - -{{site.data.alerts.callout_danger}} -**This is an experimental feature.** The interface and output are subject to change. -{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE CHANGEFEED FOR TABLE name, name2, name3 - INTO 'experimental-scheme://host?parameters' - WITH updated, resolved; -~~~ -~~~ -+--------------------+ -| job_id | -+--------------------+ -| 360645287206223873 | -+--------------------+ -(1 row) -~~~ - -For more information on how to create a changefeed connected to a cloud storage sink, see [Stream Data Out of CockroachDB Using Changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html#create-a-changefeed-connected-to-a-cloud-storage-sink). - -### Manage a changefeed - -Use the following SQL statements to pause, resume, and cancel a changefeed. - -{{site.data.alerts.callout_info}} -Changefeed-specific SQL statements (e.g., `CANCEL CHANGEFEED`) will be added in the future. -{{site.data.alerts.end}} - -#### Pause a changefeed - -{% include_cached copy-clipboard.html %} -~~~ sql -> PAUSE JOB job_id; -~~~ - -For more information, see [`PAUSE JOB`](pause-job.html). - -#### Resume a paused changefeed - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESUME JOB job_id; -~~~ - -For more information, see [`RESUME JOB`](resume-job.html). - -#### Cancel a changefeed - -{% include_cached copy-clipboard.html %} -~~~ sql -> CANCEL JOB job_id; -~~~ - -For more information, see [`CANCEL JOB`](cancel-job.html). - -#### Configuring all changefeeds - -{% include {{ page.version.version }}/cdc/configure-all-changefeed.md %} - -### Start a new changefeed where another ended - -Find the [high-water timestamp](stream-data-out-of-cockroachdb-using-changefeeds.html#monitor-a-changefeed) for the ended changefeed: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM crdb_internal.jobs WHERE job_id = ; -~~~ -~~~ - job_id | job_type | ... | high_water_timestamp | error | coordinator_id -+--------------------+------------+ ... +--------------------------------+-------+----------------+ - 383870400694353921 | CHANGEFEED | ... | 1537279405671006870.0000000000 | | 1 -(1 row) -~~~ - -Use the `high_water_timestamp` to start the new changefeed: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE CHANGEFEED FOR TABLE name, name2, name3 - INTO 'kafka//host:port' - WITH cursor = ''; -~~~ - -Note that because the cursor is provided, the initial scan is not performed. - -## See also - -- [Stream Data Out of CockroachDB Using Changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html) -- [Other SQL Statements](sql-statements.html) -- [Changefeed Dashboard](ui-cdc-dashboard.html) diff --git a/src/current/v21.1/create-database.md b/src/current/v21.1/create-database.md deleted file mode 100644 index b170b28f8fe..00000000000 --- a/src/current/v21.1/create-database.md +++ /dev/null @@ -1,171 +0,0 @@ ---- -title: CREATE DATABASE -summary: The CREATE DATABASE statement creates a new CockroachDB database. -toc: true ---- - -The `CREATE DATABASE` [statement](sql-statements.html) creates a new CockroachDB database. - -{% include {{{ page.version.version }}/misc/schema-change-stmt-note.md %} - -## Required privileges - -To create a database, the user must be a member of the `admin` role or must have the [`CREATEDB`](create-role.html#create-a-role-that-can-create-and-rename-databases) parameter set. - -## Synopsis - -
-{% include {{ page.version.version }}/sql/generated/diagrams/create_database.html %} -
- -## Parameters - -Parameter | Description -----------|------------ -`IF NOT EXISTS` | Create a new database only if a database of the same name does not already exist; if one does exist, do not return an error. -`name` | The name of the database to create, which [must be unique](#create-fails-name-already-in-use) and follow these [identifier rules](keywords-and-identifiers.html#identifiers). -`encoding` | The `CREATE DATABASE` statement accepts an optional `ENCODING` clause for compatibility with PostgreSQL, but `UTF-8` is the only supported encoding. The aliases `UTF8` and `UNICODE` are also accepted. Values should be enclosed in single quotes and are case-insensitive.

Example: `CREATE DATABASE bank ENCODING = 'UTF-8'`. -`CONNECTION LIMIT` | Supported for compatibility with PostgreSQL. A value of `-1` indicates no connection limit. Values other than `-1` are currently not supported. By default, `CONNECTION LIMIT = -1`. -`PRIMARY REGION region_name` | **New in v21.1:** Create a [multi-region database](multiregion-overview.html) with `region_name` as [the primary region](multiregion-overview.html#database-regions).
Allowed values include any region returned by [`SHOW REGIONS FROM CLUSTER`](show-regions.html). -`REGIONS region_name_list` | **New in v21.1:** Create a [multi-region database](multiregion-overview.html) with `region_name_list` as [database regions](multiregion-overview.html#database-regions).
Allowed values include any region returned by [`SHOW REGIONS FROM CLUSTER`](show-regions.html).
To set database regions at database creation, a primary region must be specified in the same `CREATE DATABASE` statement. -`SURVIVE ZONE FAILURE` (*Default*)
`SURVIVE REGION FAILURE` | **New in v21.1:** Create a [multi-region database](multiregion-overview.html) with regional failure or zone failure [survival goals](multiregion-overview.html#survival-goals).
To set the regional failure survival goal, the database must have at least 3 [database regions](multiregion-overview.html#database-regions).
Surviving zone failures is the default setting for multi-region databases. - -## Example - -### Create a database - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE DATABASE bank; -~~~ - -~~~ -CREATE DATABASE -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW DATABASES; -~~~ - -~~~ - database_name | owner | primary_region | regions | survival_goal -----------------+-------+----------------+---------+---------------- - bank | demo | NULL | {} | NULL - defaultdb | root | NULL | {} | NULL - postgres | root | NULL | {} | NULL - system | node | NULL | {} | NULL -(4 rows) -~~~ - -### Create fails (name already in use) - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE DATABASE bank; -~~~ - -~~~ -ERROR: database "bank" already exists -SQLSTATE: 42P04 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE DATABASE IF NOT EXISTS bank; -~~~ - -~~~ -CREATE DATABASE -~~~ - -SQL does not generate an error, but instead responds `CREATE DATABASE` even though a new database wasn't created. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW DATABASES; -~~~ - -~~~ - database_name | owner | primary_region | regions | survival_goal -----------------+-------+----------------+---------+---------------- - bank | demo | NULL | {} | NULL - defaultdb | root | NULL | {} | NULL - postgres | root | NULL | {} | NULL - system | node | NULL | {} | NULL -(4 rows) -~~~ - -### Create a multi-region database - -{% include enterprise-feature.md %} - -Suppose you start a cluster with region and zone [localities specified at startup](cockroach-start.html#locality). - -For this example, let's use a [demo cluster](cockroach-demo.html), with the [`--demo-locality` flag](cockroach-demo.html#general) to simulate a multi-region cluster: - -{% include_cached copy-clipboard.html %} -~~~ shell -cockroach211 demo --nodes=6 --demo-locality=region=us-east1,zone=us-east1-a:region=us-east1,zone=us-east1-b:region=us-central1,zone=us-central1-a:region=us-central1,zone=us-central1-b:region=us-west1,zone=us-west1-a:region=us-west1,zone=us-west1-b --no-example-database -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW REGIONS; -~~~ - -~~~ - region | zones | database_names | primary_region_of ---------------+-------------------------------+----------------+-------------------- - us-central1 | {us-central1-a,us-central1-b} | {} | {} - us-east1 | {us-east1-a,us-east1-b} | {} | {} - us-west1 | {us-west1-a,us-west1-b} | {} | {} -(3 rows) -~~~ - -If regions are set at cluster start-up, you can create multi-region databases in the cluster that use the cluster regions. - -Use the following command to specify regions and survival goals at database creation: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE DATABASE bank PRIMARY REGION "us-east1" REGIONS "us-east1", "us-central1", "us-west1" SURVIVE REGION FAILURE; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW DATABASES; -~~~ - -~~~ - database_name | owner | primary_region | regions | survival_goal -----------------+-------+----------------+---------------------------------+---------------- - bank | demo | us-east1 | {us-central1,us-east1,us-west1} | region - defaultdb | root | NULL | {} | NULL - postgres | root | NULL | {} | NULL - system | node | NULL | {} | NULL -(4 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW REGIONS FROM DATABASE bank; -~~~ - -~~~ - database | region | primary | zones ------------+-------------+---------+-------------------------------- - bank | us-east1 | true | {us-east1-a,us-east1-b} - bank | us-central1 | false | {us-central1-a,us-central1-b} - bank | us-west1 | false | {us-west1-a,us-west1-b} -(3 rows) -~~~ - -## See also - -- [`SHOW DATABASES`](show-databases.html) -- [`RENAME DATABASE`](rename-database.html) -- [`SET DATABASE`](set-vars.html) -- [`DROP DATABASE`](drop-database.html) -- [Other SQL Statements](sql-statements.html) -- [Online Schema Changes](online-schema-changes.html) diff --git a/src/current/v21.1/create-index.md b/src/current/v21.1/create-index.md deleted file mode 100644 index f1194dad963..00000000000 --- a/src/current/v21.1/create-index.md +++ /dev/null @@ -1,290 +0,0 @@ ---- -title: CREATE INDEX -summary: The CREATE INDEX statement creates an index for a table. Indexes improve your database's performance by helping SQL quickly locate data. -toc: true -keywords: gin, gin index, gin indexes, inverted index, inverted indexes, accelerated index, accelerated indexes ---- - -The `CREATE INDEX` [statement](sql-statements.html) creates an index for a table. [Indexes](indexes.html) improve your database's performance by helping SQL locate data without having to look through every row of a table. - -The following types cannot be included in an index key, but can be stored (and used in a covered query) using the [`STORING` or `COVERING`](create-index.html#store-columns) clause: - -- [`JSONB`](jsonb.html) -- [`ARRAY`](array.html) -- The computed [`TUPLE`](scalar-expressions.html#tuple-constructor) type, even if it is constructed from indexed fields - -To create an index on the schemaless data in a [`JSONB`](jsonb.html) column, or on the data in an [`ARRAY`](array.html), use a [GIN index](inverted-indexes.html). - -{{site.data.alerts.callout_info}} -Indexes are automatically created for a table's [`PRIMARY KEY`](primary-key.html) and [`UNIQUE`](unique.html) columns. When querying a table, CockroachDB uses the fastest index. For more information about that process, see [Index Selection in CockroachDB](https://www.cockroachlabs.com/blog/index-selection-cockroachdb-2/). -{{site.data.alerts.end}} - -{% include {{{ page.version.version }}/misc/schema-change-stmt-note.md %} - -## Required privileges - -The user must have the `CREATE` [privilege](authorization.html#assign-privileges) on the table. - -## Synopsis - -**Standard index:** - -
{% include {{ page.version.version }}/sql/generated/diagrams/create_index.html %}
- -**GIN index:** - -
{% include {{ page.version.version }}/sql/generated/diagrams/create_inverted_index.html %}
- -## Parameters - -Parameter | Description -----------|------------ -`UNIQUE` | Apply the [`UNIQUE` constraint](unique.html) to the indexed columns.

This causes the system to check for existing duplicate values on index creation. It also applies the `UNIQUE` constraint at the table level, so the system checks for duplicate values when inserting or updating data. -`INVERTED` | Create a [GIN index](inverted-indexes.html) on the schemaless data in the specified [`JSONB`](jsonb.html) column.

You can also use the PostgreSQL-compatible syntax `USING GIN`. For more details, see [GIN Indexes](inverted-indexes.html#creation). -`IF NOT EXISTS` | Create a new index only if an index of the same name does not already exist; if one does exist, do not return an error. -`opt_index_name`
`index_name` | The name of the index to create, which must be unique to its table and follow these [identifier rules](keywords-and-identifiers.html#identifiers).

If you do not specify a name, CockroachDB uses the format `__key/idx`. `key` indicates the index applies the `UNIQUE` constraint; `idx` indicates it does not. Example: `accounts_balance_idx` -`table_name` | The name of the table you want to create the index on. -`USING name` | An optional clause for compatibility with third-party tools. Accepted values for `name` are `btree`, `gin`, and `gist`, with `btree` for a standard secondary index, `gin` as the PostgreSQL-compatible syntax for a [GIN index](#create-gin-indexes), and `gist` for a [spatial index](spatial-indexes.html). -`name` | The name of the column you want to index. For [multi-region tables](multiregion-overview.html#table-locality), you can use the `crdb_region` column within the index in the event the original index may contain non-unique entries across multiple, unique regions. -`ASC` or `DESC`| Sort the column in ascending (`ASC`) or descending (`DESC`) order in the index. How columns are sorted affects query results, particularly when using `LIMIT`.

__Default:__ `ASC` -`STORING ...`| Store (but do not sort) each column whose name you include.

For information on when to use `STORING`, see [Store Columns](#store-columns). Note that columns that are part of a table's [`PRIMARY KEY`](primary-key.html) cannot be specified as `STORING` columns in secondary indexes on the table.

`COVERING` and `INCLUDE` are aliases for `STORING` and work identically. -`opt_partition_by` | An [Enterprise-only](enterprise-licensing.html) option that lets you [define index partitions at the row level](partitioning.html). As of CockroachDB v21.1 and later, most users should use [`REGIONAL BY ROW` tables](multiregion-overview.html#regional-by-row-tables). Indexes against regional by row tables are automatically partitioned, so explicit index partitioning is not required. -`opt_where_clause` | An optional `WHERE` clause that defines the predicate boolean expression of a [partial index](partial-indexes.html). -`USING HASH WITH BUCKET COUNT` | Creates a [hash-sharded index](hash-sharded-indexes.html) with `n_buckets` number of buckets.
{{site.data.alerts.callout_info}}To enable hash-sharded indexes, set the `experimental_enable_hash_sharded_indexes` [session variable](set-vars.html) to `on`.{{site.data.alerts.end}} -`WITH storage_parameter` | A comma-separated list of [spatial index tuning parameters](spatial-indexes.html#index-tuning-parameters). Supported parameters include `fillfactor`, `s2_max_level`, `s2_level_mod`, `s2_max_cells`, `geometry_min_x`, `geometry_max_x`, `geometry_min_y`, and `geometry_max_y`. The `fillfactor` parameter is a no-op, allowed for PostgreSQL-compatibility.

For details, see [Spatial index tuning parameters](spatial-indexes.html#index-tuning-parameters). For an example, see [Create a spatial index that uses all of the tuning parameters](spatial-indexes.html#create-a-spatial-index-that-uses-all-of-the-tuning-parameters). -`CONCURRENTLY` | Optional, no-op syntax for PostgreSQL compatibility. All indexes are created concurrently in CockroachDB. -`opt_interleave` | [Interleave index into parent object](interleave-in-parent.html).
{% include {{ page.version.version }}/misc/interleave-deprecation-note.md %} - -## Viewing schema changes - -{% include {{ page.version.version }}/misc/schema-change-view-job.md %} - -## Examples - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### Create standard indexes - -To create the most efficient indexes, we recommend reviewing: - -- [Indexes: Best Practices](indexes.html#best-practices) -- [Index Selection in CockroachDB](https://www.cockroachlabs.com/blog/index-selection-cockroachdb-2/) - -#### Single-column indexes - -Single-column indexes sort the values of a single column. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE INDEX ON users (name); -~~~ - -Because each query can only use one index, single-column indexes are not typically as useful as multiple-column indexes. - -#### Multiple-column indexes - -Multiple-column indexes sort columns in the order you list them. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE INDEX ON users (name, city); -~~~ - -To create the most useful multiple-column indexes, we recommend reviewing our [best practices](schema-design-indexes.html). - -#### Unique indexes - -Unique indexes do not allow duplicate values among their columns. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE UNIQUE INDEX ON users (name, id); -~~~ - -This also applies the [`UNIQUE` constraint](unique.html) at the table level, similarly to [`ALTER TABLE`](alter-table.html). The above example is equivalent to: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE users ADD CONSTRAINT users_name_id_key UNIQUE (name, id); -~~~ - -### Create GIN indexes - -[GIN indexes](inverted-indexes.html) can be created on schemaless data in a [`JSONB`](jsonb.html) column. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE INVERTED INDEX ON promo_codes (rules); -~~~ - -The above example is equivalent to the following PostgreSQL-compatible syntax: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE INDEX ON promo_codes USING GIN (rules); -~~~ - -### Create spatial indexes - -[Spatial indexes](spatial-indexes.html) can be created on `GEOMETRY` and `GEOGRAPHY` columns. Spatial indexes are a special type of [GIN index](inverted-indexes.html). - -To create a spatial index on a `GEOMETRY` column: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE INDEX geom_idx_1 ON some_spatial_table USING GIST(geom); -~~~ - -Unlike GIN indexes, spatial indexes do not support an alternate `CREATE INVERTED INDEX ...` syntax. Only the syntax shown here is supported. - -For advanced users, there are a number of [spatial index tuning parameters](spatial-indexes.html#create-a-spatial-index-that-uses-all-of-the-tuning-parameters) that can be passed in using the syntax `WITH (var1=val1, var2=val2)` as follows: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE INDEX geom_idx_2 - ON some_spatial_table USING GIST(geom) - WITH (s2_max_cells = 20, s2_max_level = 12, s2_level_mod = 3); -~~~ - -{{site.data.alerts.callout_danger}} -Most users should not change the default spatial index settings. There is a risk that you will get worse performance by changing the default settings. For more information , see [Spatial indexes](spatial-indexes.html). -{{site.data.alerts.end}} - -### Store columns - -Storing a column improves the performance of queries that retrieve (but do not filter) its values. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE INDEX ON users (city) STORING (name); -~~~ - -However, to use stored columns, queries must filter another column in the same index. For example, SQL can retrieve `name` values from the above index only when a query's `WHERE` clause filters `city`. - -{{site.data.alerts.callout_info}} -{% include {{page.version.version}}/sql/covering-index.md %} -{{site.data.alerts.end}} - -### Change column sort order - -To sort columns in descending order, you must explicitly set the option when creating the index. (Ascending order is the default.) - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE INDEX ON users (city DESC, name); -~~~ - -Note that how a column is ordered in the index will affect the ordering of the index keys, and may affect the efficiency of queries that include an `ORDER BY` clause. - -### Query specific indexes - -Normally, CockroachDB selects the index that it calculates will scan the fewest rows. However, you can override that selection and specify the name of the index you want to use. To find the name, use [`SHOW INDEX`](show-index.html). - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW INDEX FROM users; -~~~ - -~~~ - table_name | index_name | non_unique | seq_in_index | column_name | direction | storing | implicit -+------------+----------------+------------+--------------+-------------+-----------+---------+----------+ - users | primary | false | 1 | city | ASC | false | false - users | primary | false | 2 | id | ASC | false | false - users | users_name_idx | true | 1 | name | ASC | false | false - users | users_name_idx | true | 2 | city | ASC | false | true - users | users_name_idx | true | 3 | id | ASC | false | true -(5 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT name FROM users@users_name_idx WHERE city='new york'; -~~~ - -~~~ - name -+------------------+ - Catherine Nelson - Devin Jordan - James Hamilton - Judy White - Robert Murphy -(5 rows) -~~~ - -### Create a hash-sharded secondary index - -{% include {{page.version.version}}/performance/use-hash-sharded-indexes.md %} - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE events ( - product_id INT8, - owner UUID, - serial_number VARCHAR, - event_id UUID, - ts TIMESTAMP, - data JSONB, - PRIMARY KEY (product_id, owner, serial_number, ts, event_id) -); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET experimental_enable_hash_sharded_indexes=on; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE INDEX ON events(ts) USING HASH WITH BUCKET_COUNT=8; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW INDEX FROM events; -~~~ - -~~~ - table_name | index_name | non_unique | seq_in_index | column_name | direction | storing | implicit --------------+---------------+------------+--------------+--------------------------+-----------+---------+----------- - events | events_ts_idx | true | 1 | crdb_internal_ts_shard_8 | ASC | false | false - events | events_ts_idx | true | 2 | ts | ASC | false | false - events | events_ts_idx | true | 3 | product_id | ASC | false | true - events | events_ts_idx | true | 4 | owner | ASC | false | true - events | events_ts_idx | true | 5 | serial_number | ASC | false | true - events | events_ts_idx | true | 6 | event_id | ASC | false | true - events | primary | false | 1 | product_id | ASC | false | false - events | primary | false | 2 | owner | ASC | false | false - events | primary | false | 3 | serial_number | ASC | false | false - events | primary | false | 4 | ts | ASC | false | false - events | primary | false | 5 | event_id | ASC | false | false -(11 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM events; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden ----------------------------+-----------+-------------+----------------+-------------------------------------------------+-------------------------+------------ - product_id | INT8 | false | NULL | | {events_ts_idx,primary} | false - owner | UUID | false | NULL | | {events_ts_idx,primary} | false - serial_number | VARCHAR | false | NULL | | {events_ts_idx,primary} | false - event_id | UUID | false | NULL | | {events_ts_idx,primary} | false - ts | TIMESTAMP | false | NULL | | {events_ts_idx,primary} | false - data | JSONB | true | NULL | | {} | false - crdb_internal_ts_shard_8 | INT4 | false | NULL | mod(fnv32(COALESCE(CAST(ts AS STRING), '')), 8) | {events_ts_idx} | true -(7 rows) -~~~ - -## See also - -- [Indexes](indexes.html) -- [`SHOW INDEX`](show-index.html) -- [`DROP INDEX`](drop-index.html) -- [`RENAME INDEX`](rename-index.html) -- [`SHOW JOBS`](show-jobs.html) -- [Other SQL Statements](sql-statements.html) -- [Online Schema Changes](online-schema-changes.html) diff --git a/src/current/v21.1/create-role.md b/src/current/v21.1/create-role.md deleted file mode 100644 index 1716b1be82f..00000000000 --- a/src/current/v21.1/create-role.md +++ /dev/null @@ -1,319 +0,0 @@ ---- -title: CREATE ROLE -summary: The CREATE ROLE statement creates SQL roles, which are groups containing any number of roles and users as members. -toc: true ---- - -The `CREATE ROLE` [statement](sql-statements.html) creates SQL [roles](authorization.html#create-and-manage-roles), which are groups containing any number of roles and users as members. You can assign [privileges](authorization.html#privileges) to roles, and all members of the role (regardless of whether if they are direct or indirect members) will inherit the role's privileges. - -{{site.data.alerts.callout_info}} - CREATE ROLE is no longer an Enterprise feature and is now freely available in the core version of CockroachDB. Also, since the keywords `ROLE` and `USER` can now be used interchangeably in SQL statements for enhanced Postgres compatibility. - - `CREATE USER` is equivalent to the statement `CREATE ROLE`, with one exception: `CREATE ROLE` sets the [`NOLOGIN`](#parameters) option by default, preventing the new role from being used to log in to the database. You can use `CREATE ROLE` and specify the [`LOGIN`](#parameters) option to achieve the same result as `CREATE USER`. -{{site.data.alerts.end}} - -## Considerations - -- Role names: - - Are case-insensitive - - Must start with either a letter or underscore - - Must contain only letters, numbers, periods, or underscores - - Must be between 1 and 63 characters. -- After creating roles, you must [grant them privileges to databases and tables](grant.html). -- Roles and users can be members of roles. -- Roles and users share the same namespace and must be unique. -- All [privileges](authorization.html#privileges) of a role are inherited by all of its members. -- Role options of a role are NOT inherited by any of its members. -- There is no limit to the number of members in a role. -- Membership loops are not allowed (direct: `A is a member of B is a member of A` or indirect: `A is a member of B is a member of C ... is a member of A`). - -## Required privileges - -Unless a role is a member of the admin role, additional [privileges](#parameters) are required to manage other roles. - -- To create other roles, a role must have the [`CREATEROLE`](#create-a-role-that-can-create-other-roles-and-manage-authentication-methods-for-the-new-roles) role option set. -- To add the `LOGIN` capability for other roles so that they may log in as users, a role must also have the [`CREATELOGIN`](#create-a-role-that-can-create-other-roles-and-manage-authentication-methods-for-the-new-roles) role option set. -- To be able to grant or revoke membership to a role for additional roles, a member of the role must be set as a [role admin](authorization.html#role-admin) for that role. - -## Synopsis - -
{% include {{ page.version.version }}/sql/generated/diagrams/create_role.html %}
- -## Parameters - -| Parameter | Description | -------------|-------------- -`name` | The name of the role you want to create. Role names are case-insensitive; must start with either a letter or underscore; must contain only letters, numbers, periods, or underscores; and must be between 1 and 63 characters.

Note that roles and [users](create-user.html) share the same namespace and must be unique. -`CREATELOGIN`/`NOCREATELOGIN` | Allow or disallow the role to manage authentication using the `WITH PASSWORD`, `VALID UNTIL`, and `LOGIN/NOLOGIN` role options.

By default, the role option is set to `NOCREATELOGIN` for all non-admin roles. -`LOGIN`/`NOLOGIN` | The `LOGIN` role option allows a role to login with one of the [client authentication methods](authentication.html#client-authentication). Setting the role option to `NOLOGIN` prevents the role from logging in using any authentication method. -`password` | Let the role [authenticate their access to a secure cluster](authentication.html#client-authentication) using this password. Passwords should be entered as a [string literal](sql-constants.html#string-literals). For compatibility with PostgreSQL, a password can also be entered as an identifier.

To prevent a role from using [password authentication](authentication.html#client-authentication) and to mandate [certificate-based client authentication](authentication.html#client-authentication), [set the password as `NULL`](#prevent-a-role-from-using-password-authentication). -`VALID UNTIL` | The date and time (in the [`timestamp`](timestamp.html) format) after which the password is not valid. -`CREATEROLE`/`NOCREATEROLE` | Allow or disallow the new role to create, [alter](alter-role.html), and [drop](drop-role.html) other non-admin roles.

By default, the role option is set to `NOCREATEROLE` for all non-admin users. -`CREATEDB`/`NOCREATEDB` | Allow or disallow the role to [create](create-database.html) or [rename](rename-database.html) a database. The role is assigned as the owner of the database.

By default, the role option is set to `NOCREATEDB` for all non-admin roles. -`CONTROLJOB`/`NOCONTROLJOB` | Allow or disallow the role to [pause](pause-job.html), [resume](resume-job.html), and [cancel](cancel-job.html) jobs. Non-admin roles cannot control jobs created by admins.

By default, the role option is set to `NOCONTROLJOB` for all non-admin roles. -`CANCELQUERY`/`NOCANCELQUERY` | Allow or disallow the role to cancel [queries](cancel-query.html) and [sessions](cancel-session.html) of other roles. Without this role option, roles can only cancel their own queries and sessions. Even with this role option, non-admins cannot cancel admin queries or sessions. This option should usually be combined with `VIEWACTIVITY` so that the role can view other roles' query and session information.

By default, the role option is set to `NOCANCELQUERY` for all non-admin roles. -`VIEWACTIVITY`/`NOVIEWACTIVITY` | Allow or disallow a role to see other roles' [queries](show-statements.html) and [sessions](show-sessions.html) using `SHOW STATEMENTS`, `SHOW SESSIONS`, and the [**Statements**](ui-statements-page.html) and **Transactions** pages in the DB Console. Without this role option, the `SHOW` commands only show the role's own data and the DB Console pages are unavailable.

By default, the role option is set to `NOVIEWACTIVITY` for all non-admin roles. -`CONTROLCHANGEFEED`/`NOCONTROLCHANGEFEED` | Allow or disallow the role to run [`CREATE CHANGEFEED`](create-changefeed.html) on tables they have `SELECT` privileges on.

By default, the role option is set to `NOCONTROLCHANGEFEED` for all non-admin roles. -`MODIFYCLUSTERSETTING`/`NOMODIFYCLUSTERSETTING` | Allow or disallow the role to modify the [cluster settings](cluster-settings.html) with the `sql.defaults` prefix.

By default, the role option is set to `NOMODIFYCLUSTERSETTING` for all non-admin roles. - -## Examples - -To run the following examples, [start a secure single-node cluster](cockroach-start-single-node.html) and use the built-in SQL shell: - -~~~ shell -$ cockroach sql --certs-dir=certs -~~~ - -~~~ sql -> SHOW ROLES; -~~~ - -~~~ -username | options | member_of ----------+---------+------------ -admin | | {} -root | | {admin} -(2 rows) -~~~ - -{{site.data.alerts.callout_info}} -The following statements are run by the `root` user that is a member of the `admin` role and has `ALL` privileges. -{{site.data.alerts.end}} - -### Create a role - -Role names are case-insensitive; must start with a letter, number, or underscore; must contain only letters, numbers, periods, or underscores; and must be between 1 and 63 characters. - -~~~ sql -root@:26257/defaultdb> CREATE ROLE no_options; -~~~ - -~~~ sql -root@:26257/defaultdb> SHOW ROLES; -~~~ - -~~~ - username | options | member_of - ----------+---------+------------ -admin | | {} -no_options | NOLOGIN | {} -root | | {admin} -(3 rows) -~~~ - -After creating roles, you must [grant them privileges to databases](grant.html). - -### Create a role that can log in to the database - -~~~ sql -root@:26257/defaultdb> CREATE ROLE can_login WITH LOGIN PASSWORD '$tr0nGpassW0rD' VALID UNTIL '2021-10-10'; -~~~ - -~~~ sql -root@:26257/defaultdb> SHOW ROLES; -~~~ - -~~~ - username | options | member_of ------------+---------------------------------------+------------ -admin | | {} -can_login | VALID UNTIL=2021-10-10 00:00:00+00:00 | {} -no_options | NOLOGIN | {} -root | | {admin} -(4 rows) -~~~ - -### Prevent a role from using password authentication - -The following statement prevents the role from using password authentication and mandates certificate-based client authentication: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE ROLE no_password WITH PASSWORD NULL; -~~~ - -~~~ sql -root@:26257/defaultdb> SHOW ROLES; -~~~ - -~~~ - username | options | member_of ------------+---------------------------------------+------------ -admin | | {} -can_login | VALID UNTIL=2021-10-10 00:00:00+00:00 | {} -no_options | NOLOGIN | {} -no_password| NOLOGIN | {} -root | | {admin} -(5 rows) -~~~ - -### Create a role that can create other roles and manage authentication methods for the new roles - -The following example allows the role to [create other users](create-user.html) and [manage authentication methods](authentication.html#client-authentication) for them: - -~~~ sql -root@:26257/defaultdb> CREATE ROLE can_create_role WITH CREATEROLE CREATELOGIN; -~~~ - -~~~ sql -root@:26257/defaultdb> SHOW ROLES; -~~~ - -~~~ - username | options | member_of -----------------+---------------------------------------+------------ -admin | | {} -can_create_role | CREATELOGIN, CREATEROLE, NOLOGIN | {} -can_login | VALID UNTIL=2021-10-10 00:00:00+00:00 | {} -no_options | NOLOGIN | {} -no_password | NOLOGIN | {} -root | | {admin} -(6 rows) -~~~ - -### Create a role that can create and rename databases - -The following example allows the role to [create](create-database.html) or [rename](rename-database.html) databases: - -~~~ sql -root@:26257/defaultdb> CREATE ROLE can_create_db WITH CREATEDB; -~~~ - -~~~ sql -root@:26257/defaultdb> SHOW ROLES; -~~~ - -~~~ - username | options | member_of -----------------------+---------------------------------------+------------ -admin | | {} -can_create_db | CREATEDB, NOLOGIN | {} -can_create_role | CREATELOGIN, CREATEROLE, NOLOGIN | {} -can_login | VALID UNTIL=2021-10-10 00:00:00+00:00 | {} -no_options | NOLOGIN | {} -no_password | NOLOGIN | {} -root | | {admin} -(7 rows) -~~~ - -### Create a role that can pause, resume, and cancel non-admin jobs - -The following example allows the role to [pause](pause-job.html), [resume](resume-job.html), and [cancel](cancel-job.html) jobs: - -~~~ sql -root@:26257/defaultdb> CREATE ROLE can_control_job WITH CONTROLJOB; -~~~ - -~~~ sql -root@:26257/defaultdb> SHOW ROLES; -~~~ - -~~~ - username | options | member_of -----------------------+---------------------------------------+------------ -admin | | {} -can_control_job | CONTROLJOB, NOLOGIN | {} -can_create_db | CREATEDB, NOLOGIN | {} -can_create_role | CREATELOGIN, CREATEROLE, NOLOGIN | {} -can_login | VALID UNTIL=2021-10-10 00:00:00+00:00 | {} -manage_auth_for_roles | CREATELOGIN, NOLOGIN | {} -no_options | NOLOGIN | {} -no_password | NOLOGIN | {} -root | | {admin} -(8 rows) -~~~ - -### Create a role that can see and cancel non-admin queries and sessions - -The following example allows the role to cancel [queries](cancel-query.html) and [sessions](cancel-session.html) for other non-admin roles: - -~~~ sql -root@:26257/defaultdb> CREATE ROLE can_manage_queries WITH CANCELQUERY VIEWACTIVITY; -~~~ - -~~~ sql -root@:26257/defaultdb> SHOW ROLES; -~~~ - -~~~ - username | options | member_of -----------------------+---------------------------------------+------------ -admin | | {} -can_control_job | CONTROLJOB, NOLOGIN | {} -can_create_db | CREATEDB, NOLOGIN | {} -can_create_role | CREATELOGIN, CREATEROLE, NOLOGIN | {} -can_login | VALID UNTIL=2021-10-10 00:00:00+00:00 | {} -can_manage_queries | CANCELQUERY, NOLOGIN, VIEWACTIVITY | {} -no_options | NOLOGIN | {} -no_password | NOLOGIN | {} -root | | {admin} -(9 rows) -~~~ - -### Create a role that can control changefeeds - -The following example allows the role to run [`CREATE CHANGEFEED`](create-changefeed.html): - -~~~ sql -root@:26257/defaultdb> CREATE ROLE can_control_changefeed WITH CONTROLCHANGEFEED; -~~~ - -~~~ sql -root@:26257/defaultdb> SHOW ROLES; -~~~ - -~~~ - username | options | member_of ------------------------+---------------------------------------+------------ -admin | | {} -can_control_changefeed | CONTROLCHANGEFEED, NOLOGIN | {} -can_control_job | CONTROLJOB, NOLOGIN | {} -can_create_db | CREATEDB, NOLOGIN | {} -can_create_role | CREATELOGIN, CREATEROLE, NOLOGIN | {} -can_login | VALID UNTIL=2021-10-10 00:00:00+00:00 | {} -can_manage_queries | CANCELQUERY, NOLOGIN, VIEWACTIVITY | {} -no_options | NOLOGIN | {} -no_password | NOLOGIN | {} -root | | {admin} -(10 rows) -~~~ - -### Create a role that can modify cluster settings - -The following example allows the role to modify [cluster settings](cluster-settings.html): - -~~~ sql -root@:26257/defaultdb> CREATE ROLE can_modify_cluster_setting WITH MODIFYCLUSTERSETTING; -~~~ - -~~~ sql -root@:26257/defaultdb> SHOW ROLES; -~~~ - -~~~ - username | options | member_of ----------------------------+---------------------------------------+------------ -admin | | {} -can_control_changefeed | CONTROLCHANGEFEED, NOLOGIN | {} -can_control_job | CONTROLJOB, NOLOGIN | {} -can_create_db | CREATEDB, NOLOGIN | {} -can_create_role | CREATELOGIN, CREATEROLE, NOLOGIN | {} -can_login | VALID UNTIL=2021-10-10 00:00:00+00:00 | {} -can_manage_queries | CANCELQUERY, NOLOGIN, VIEWACTIVITY | {} -can_modify_cluster_setting | MODIFYCLUSTERSETTING, NOLOGIN | {} -no_options | NOLOGIN | {} -no_password | NOLOGIN | {} -root | | {admin} -(11 rows) -~~~ - -## See also - -- [Authorization](authorization.html) -- [`DROP ROLE`](drop-role.html) -- [`GRANT`](grant.html) -- [`REVOKE`](revoke.html) -- [`SHOW ROLES`](show-roles.html) -- [`SHOW USERS`](show-users.html) -- [`SHOW GRANTS`](show-grants.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/create-schedule-for-backup.md b/src/current/v21.1/create-schedule-for-backup.md deleted file mode 100644 index c3418c46b5a..00000000000 --- a/src/current/v21.1/create-schedule-for-backup.md +++ /dev/null @@ -1,242 +0,0 @@ ---- -title: CREATE SCHEDULE FOR BACKUP -summary: The CREATE SCHEDULE FOR BACKUP statement creates a schedule for periodic backups. -toc: true ---- - - The `CREATE SCHEDULE FOR BACKUP` [statement](sql-statements.html) creates a schedule for periodic [backups](backup.html). - -For more information about creating, managing, monitoring, and restoring from a scheduled backup, see [Manage a Backup Schedule](manage-a-backup-schedule.html). - -{{site.data.alerts.callout_info}} -Core users can only use backup scheduling for [full backups](#create-a-schedule-for-full-backups-only-core) of clusters, databases, or tables. - -To use the other backup features, you need an [Enterprise license](enterprise-licensing.html). -{{site.data.alerts.end}} - -## Required privileges - -- Only members of the [`admin` role](authorization.html#default-roles) can run `CREATE SCHEDULE FOR BACKUP`. By default, the `root` user belongs to the `admin` role. -- `BACKUP` requires full read and write (including delete and overwrite) permissions to its target destination. - -## Synopsis - -~~~ -CREATE SCHEDULE
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Fault Tolerance Goals3 nodes5 nodes9 nodes
1 NodeRF = 3RF = 3RF = 3
1 AZRF = 3RF = 3RF = 3
2 NodesNot possibleRF = 5RF = 5
AZ + NodeNot possibleNot possibleRF = 9
2 AZNot possibleNot possibleNot possible
- -To be able to survive 2+ availability zones failing, scale to a [multi-region](#multi-region-survivability-planning) deployment. - -### Single-region recovery - -For hardware failures in a single-region cluster, the recovery actions vary and depend on the type of infrastructure used. - -For example, consider a cloud-deployed CockroachDB cluster with the following setup: - -- Single-region -- 3 nodes -- A node in each availability zone (i.e., 3 AZs) -- Replication factor of 3 - -The table below describes what actions to take to recover from various hardware failures in this example cluster: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
FailureAvailabilityConsequenceAction to Take
1 Disk√Fewer resources are available. Some data will be under-replicated until the failed nodes are marked dead.

Once marked dead, data is replicated to other nodes and the cluster remains healthy. - 		Restart the node with a new disk.
1 Node√If the node or AZ becomes unavailable, check the Overview dashboard on the DB Console: -
1 AZ√
2 NodesXCluster is unavailable.Restart 1 of the 2 nodes that are down to regain quorum.

If you can’t recover at least 1 node, contact Cockroach Labs support for assistance.
1 AZ + 1 NodeXCluster is unavailable.Restart the node that is down to regain quorum. When the AZ comes back online, try restarting the node.

If you can’t recover at least 1 node, contact Cockroach Labs support for assistance.
2 AZXCluster is unavailable.When the AZ comes back online, try restarting at least 1 of the nodes.

You can also contact Cockroach Labs support for assistance.
3 NodesXCluster is unavailable.Restart 2 of the 3 nodes that are down to regain quorum.

If you can’t recover 2 of the 3 failed nodes, contact Cockroach Labs support for assistance.
1 RegionXCluster is unavailable.

Potential data loss between last backup and time of outage if the region and nodes did not come back online.		When the region comes back online, try restarting the nodes in the cluster.

If the region does not come back online and nodes are lost or destroyed, try restoring the latest cluster backup into a new cluster.

You can also contact Cockroach Labs support for assistance.
- -{{site.data.alerts.callout_info}} -When using Kubernetes, recovery actions happen automatically in many cases and no action needs to be taken. -{{site.data.alerts.end}} - -### Multi-region survivability planning - -{{site.data.alerts.callout_success}} -{% include_cached new-in.html version="v21.1" %} By default, every [multi-region database](multiregion-overview.html) has a [zone-level survival goal](multiregion-overview.html#survival-goals) associated with it. The survival goal setting provides an abstraction that handles the low-level details of replica placement to ensure your desired fault tolerance. The information below is still useful for legacy deployments. -{{site.data.alerts.end}} - -The table below shows the replication factor (RF) needed to achieve the listed fault tolerance (e.g., survive 1 failed node) for a multi-region, cloud-deployed cluster with 3 availability zones (AZ) per region and one node in each AZ: - -{{site.data.alerts.callout_danger}} -The chart below describes the CockroachDB default behavior when locality flags are correctly set. It does not use geo-partitioning or a specific [topology pattern](topology-patterns.html). For a multi-region cluster in production, we do not recommend using the default behavior, as the cluster's performance will be negatively affected. -{{site.data.alerts.end}} - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Fault Tolerance Goals3 Regions
(9 Nodes Total)		4 Regions
(12 Nodes Total)		5 Regions
(15 Nodes Total)
1 NodeRF = 3RF = 3RF = 3
1 AZRF = 3RF = 3RF = 3
1 RegionRF = 3RF = 3RF = 3
2 NodesRF = 5RF = 5RF = 5
1 Region + 1 NodeRF = 9RF = 7RF = 5
2 RegionsNot possibleNot possibleRF = 5
2 Regions + 1 NodeNot possibleNot possibleRF = 15
- -### Multi-region recovery - -For hardware failures in a multi-region cluster, the actions taken to recover vary and depend on the type of infrastructure used. - -For example, consider a cloud-deployed CockroachDB cluster with the following setup: - -- 3 regions -- 3 AZs per region -- 9 nodes (1 node per AZ) -- Replication factor of 3 - -The table below describes what actions to take to recover from various hardware failures in this example cluster: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
FailureAvailabilityConsequenceAction to Take
1 Disk√Under-replicated data. Fewer resources for workload.Restart the node with a new disk.
1 Node√If the node or AZ becomes unavailable check the Overview dashboard on the DB Console: - -
1 AZ√
1 Region√Check the Overview dashboard on the DB Console. If nodes are marked Dead, decommission the nodes and add 3 new nodes in a new region. Ensure that locality flags are set correctly upon node startup.
2 or More RegionsXCluster is unavailable.

Potential data loss between last backup and time of outage if the region and nodes did not come back online.		When the regions come back online, try restarting the nodes in the cluster.

If the regions do not come back online and nodes are lost or destroyed, try restoring the latest cluster backup into a new cluster.

You can also contact Cockroach Labs support for assistance.
- -{{site.data.alerts.callout_info}} -When using Kubernetes, recovery actions happen automatically in many cases and no action needs to be taken. -{{site.data.alerts.end}} - -## Data failure - -When dealing with data failure due to bad actors, rogue applications, or data corruption, domain expertise is required to identify the affected rows and determine how to remedy the situation (e.g., remove the incorrectly inserted rows, insert deleted rows, etc.). However, there are a few actions that you can take for short-term remediation: - -- If you are within the garbage collection window, [run differentials](#run-differentials). -- If you have a backup file, [restore to a point in time](#restore-to-a-point-in-time). -- If your cluster is running and you do not have a backup with the data you need, [create a new backup](#create-a-new-backup). -- To [recover from corrupted data in a database or table](#recover-from-corrupted-data-in-a-database-or-table), restore the corrupted object. - -{{site.data.alerts.callout_success}} -To give yourself more time to recover and clean up the corrupted data, put your application in “read only” mode and only run [`AS OF SYSTEM TIME`](as-of-system-time.html) queries from the application. -{{site.data.alerts.end}} - -### Run differentials - -If you are within the [garbage collection window](configure-replication-zones.html#replication-zone-variables) (default is 25 hours), run [`AS OF SYSTEM TIME`](as-of-system-time.html) queries and use [`CREATE TABLE AS … SELECT * FROM`](create-table-as.html) to create comparison data and run differentials to find the offending rows to fix. - -If you are outside of the garbage collection window, you will need to use a [backup](backup.html) to run comparisons. - -### Restore to a point in time - -- If you are a core user, use a [backup](backup.html) that was taken with [`AS OF SYSTEM TIME`](as-of-system-time.html) to restore to a specific point. -- If you are an Enterprise user, use your [backup](backup.html) file to [restore to a point in time](take-backups-with-revision-history-and-restore-from-a-point-in-time.html) where you are certain there was no corruption. Note that the backup must have been taken with [revision history](backup.html#with-revision-history). - -### Create a new backup - -If your cluster is running, you do not have a backup that encapsulates the time you want to [restore](restore.html) to, and the data you want to recover is still in the [garbage collection window](configure-replication-zones.html#replication-zone-variables), there are two actions you can take: - -- If you are a core user, trigger a [backup](backup.html) using [`AS OF SYSTEM TIME`](as-of-system-time.html) to create a new backup that encapsulates the specific time. The `AS OF SYSTEM TIME` must be within the [garbage collection window](configure-replication-zones.html#replication-zone-variables) (default is 25 hours). -- If you are an Enterprise user, trigger a new [backup `with_revision_history`](take-backups-with-revision-history-and-restore-from-a-point-in-time.html) and you will have a backup you can use to restore to the desired point in time within the [garbage collection window](configure-replication-zones.html#replication-zone-variables) (default is 25 hours). - -### Recover from corrupted data in a database or table - -If you have corrupted data in a database or table, [restore](restore.html) the object from a prior [backup](backup.html). If revision history is in the backup, you can restore from a [point in time](take-backups-with-revision-history-and-restore-from-a-point-in-time.html). - -Instead of dropping the corrupted table or database, we recommend [renaming the table](rename-table.html) or [renaming the database](rename-database.html) so you have historical data to compare to later. If you drop a database, the database cannot be referenced with `AS OF SYSTEM TIME` queries (see [#51380](https://github.com/cockroachdb/cockroach/issues/51380) for more information), and you will need to take a backup that is backdated to the system time when the database still existed. - -{{site.data.alerts.callout_info}} -If the table you are restoring has foreign keys, [careful consideration](restore.html#remove-the-foreign-key-before-restore) should be applied to make sure data integrity is maintained during the restore process. -{{site.data.alerts.end}} - -## Compromised security keys - -CockroachDB maintains a secure environment for your data. However, there are bad actors who may find ways to gain access or expose important security information. In the event that this happens, there are a few things you can do to get ahead of a security issue: - -- If you have [changefeeds to cloud storage sinks](#changefeeds-to-cloud-storage), cancel the changefeed job and restart it with new access credentials. -- If you are using [encryption at rest](#encryption-at-rest), rotate the store key(s). -- If you are using [wire encryption / TLS](#wire-encryption-tls), rotate your keys. - -### Changefeeds to cloud storage - -1. [Cancel the changefeed job](cancel-job.html) immediately and [record the high water timestamp](stream-data-out-of-cockroachdb-using-changefeeds.html#monitor-a-changefeed) for where the changefeed was stopped. -2. Remove the access keys from the identity management system of your cloud provider and replace with a new set of access keys. -3. [Create a new changefeed](create-changefeed.html#start-a-new-changefeed-where-another-ended) with the new access credentials using the last high water timestamp. - -### Encryption at rest - -If you believe the user-defined store keys have been compromised, quickly attempt to rotate your store keys that are being used for your encryption at rest setup. If this key has already been compromised and the store keys were rotated by a bad actor, the cluster should be wiped if possible and [restored](restore.html) from a prior backup. - -If the compromised keys were not rotated by a bad actor, quickly attempt to [rotate the store key](encryption.html#rotating-keys) by restarting each of the nodes with the old key and the new key. For an example on how to do this, see [Encryption](encryption.html#changing-encryption-algorithm-or-keys). - -Once all of the nodes are restarted with the new key, put in a request to revoke the old key from the Certificate Authority. - -{{site.data.alerts.callout_info}} -CockroachDB does not allow prior store keys to be used again. -{{site.data.alerts.end}} - -### Wire Encryption / TLS - -As a best practice, [keys should be rotated](rotate-certificates.html). In the event that keys have been compromised, quickly attempt to rotate your keys. This can include rotating node certificates, client certificates, and the CA certificate. - -## See also - -- [Fault Tolerance & Recovery](demo-fault-tolerance-and-recovery.html) -- [Back up and Restore Data](take-full-and-incremental-backups.html) -- [Topology Patterns](topology-patterns.html) -- [Production Checklist](recommended-production-settings.html) diff --git a/src/current/v21.1/drop-column.md b/src/current/v21.1/drop-column.md deleted file mode 100644 index d172af9ae48..00000000000 --- a/src/current/v21.1/drop-column.md +++ /dev/null @@ -1,209 +0,0 @@ ---- -title: DROP COLUMN -summary: Use the ALTER COLUMN statement to remove columns from tables. -toc: true ---- - -The `DROP COLUMN` [statement](sql-statements.html) is part of `ALTER TABLE` and removes columns from a table. - -{{site.data.alerts.callout_danger}} -When used in an explicit transaction combined with other schema changes to the same table, `DROP COLUMN` can result in data loss if one of the other schema changes fails or is canceled. To work around this, move the `DROP COLUMN` statement to its own explicit transaction or run it in a single statement outside the existing transaction. -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} - By default, `DROP COLUMN` drops any [indexes](indexes.html) on the column being dropped, and any indexes that reference the column, including [partial indexes](partial-indexes.html) with predicates that reference the column and indexes with [`STORING` clauses](create-index.html#store-columns) that reference the column. -{{site.data.alerts.end}} - -{% include {{ page.version.version }}/sql/combine-alter-table-commands.md %} - -## Synopsis - -
-{% include {{ page.version.version }}/sql/generated/diagrams/drop_column.html %} -
- -## Required privileges - -The user must have the `CREATE` [privilege](authorization.html#assign-privileges) on the table. - -## Parameters - - Parameter | Description ------------|------------- - `table_name` | The name of the table with the column you want to drop. - `name` | The name of the column you want to drop.

When a column with a `CHECK` constraint is dropped, the `CHECK` constraint is also dropped. - `CASCADE` | Drop the column even if objects (such as [views](views.html)) depend on it; drop the dependent objects, as well. `CASCADE` will drop a column with a foreign key constraint if it is the only column in the reference.

`CASCADE` does not list the objects it drops, so should be used cautiously.

`CASCADE` is not required to drop an indexed column, or a column that is referenced by an index. By default, `DROP COLUMN` drops any [indexes](indexes.html) on the column being dropped, and any indexes that reference the column, including [partial indexes](partial-indexes.html) with predicates that reference the column and indexes with [`STORING` clauses](create-index.html#store-columns) that reference the column. - `RESTRICT` | *(Default)* Do not drop the column if any objects (such as [views](views.html)) depend on it. - -## Viewing schema changes - -{% include {{ page.version.version }}/misc/schema-change-view-job.md %} - -## Examples - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### Drop a column - -If you no longer want a column in a table, you can drop it. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM users; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden ---------------+-----------+-------------+----------------+-----------------------+-----------+------------ - id | UUID | false | NULL | | {primary} | false - city | VARCHAR | false | NULL | | {primary} | false - name | VARCHAR | true | NULL | | {} | false - address | VARCHAR | true | NULL | | {} | false - credit_card | VARCHAR | true | NULL | | {} | false -(5 rows) -~~~ - -If there is data in the table, the `sql_safe_updates` [session variable](set-vars.html) must be set to `false`. - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE users DROP COLUMN credit_card; -~~~ - -~~~ -ERROR: rejected (sql_safe_updates = true): ALTER TABLE DROP COLUMN will remove all data in that column -SQLSTATE: 01000 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET sql_safe_updates = false; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE users DROP COLUMN credit_card; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM users; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden ---------------+-----------+-------------+----------------+-----------------------+-----------+------------ - id | UUID | false | NULL | | {primary} | false - city | VARCHAR | false | NULL | | {primary} | false - name | VARCHAR | true | NULL | | {} | false - address | VARCHAR | true | NULL | | {} | false -(4 rows) -~~~ - -### Prevent dropping columns with dependent objects (`RESTRICT`) - -If the column has dependent objects, such as [views](views.html), CockroachDB will not drop the column by default. However, if you want to be sure of the behavior you can include the `RESTRICT` clause. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE VIEW expensive_rides AS SELECT id, city FROM rides WHERE revenue > 90; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE rides DROP COLUMN revenue RESTRICT; -~~~ - -~~~ -ERROR: cannot drop column "revenue" because view "expensive_rides" depends on it -SQLSTATE: 2BP01 -HINT: you can drop expensive_rides instead. -~~~ - -### Drop a column and its dependent objects (`CASCADE`) - -If you want to drop the column and all of its dependent options, include the `CASCADE` clause. - -{{site.data.alerts.callout_danger}} -CASCADE does not list objects it drops, so should be used cautiously. -{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE expensive_rides; -~~~ - -~~~ - table_name | create_statement -------------------+------------------------------------------------------------------------------------------------------------- - expensive_rides | CREATE VIEW public.expensive_rides (id, city) AS SELECT id, city FROM movr.public.rides WHERE revenue > 90 -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE rides DROP COLUMN revenue CASCADE; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE expensive_rides; -~~~ - -~~~ -ERROR: relation "expensive_rides" does not exist -SQLSTATE: 42P01 -~~~ - -### Drop an indexed column - - `DROP COLUMN` drops a column and any indexes on the column being dropped. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE INDEX start_end_idx ON rides(start_time, end_time); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW INDEXES FROM rides] WHERE index_name='start_end_idx'; -~~~ - -~~~ - table_name | index_name | non_unique | seq_in_index | column_name | direction | storing | implicit --------------+---------------+------------+--------------+-------------+-----------+---------+----------- - rides | start_end_idx | true | 1 | start_time | ASC | false | false - rides | start_end_idx | true | 2 | end_time | ASC | false | false - rides | start_end_idx | true | 3 | city | ASC | false | true - rides | start_end_idx | true | 4 | id | ASC | false | true -(4 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE rides DROP COLUMN start_time; -~~~ - -~~~ -NOTICE: the data for dropped indexes is reclaimed asynchronously -HINT: The reclamation delay can be customized in the zone configuration for the table. -ALTER TABLE -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW INDEXES FROM rides] WHERE index_name='start_end_idx'; -~~~ - -~~~ - table_name | index_name | non_unique | seq_in_index | column_name | direction | storing | implicit --------------+------------+------------+--------------+-------------+-----------+---------+----------- -(0 rows) -~~~ - -## See also - -- [`DROP CONSTRAINT`](drop-constraint.html) -- [`DROP INDEX`](drop-index.html) -- [`ALTER TABLE`](alter-table.html) -- [`SHOW JOBS`](show-jobs.html) diff --git a/src/current/v21.1/drop-constraint.md b/src/current/v21.1/drop-constraint.md deleted file mode 100644 index 59fb3989a02..00000000000 --- a/src/current/v21.1/drop-constraint.md +++ /dev/null @@ -1,155 +0,0 @@ ---- -title: DROP CONSTRAINT -summary: Use the ALTER CONSTRAINT statement to remove constraints from columns. -toc: true ---- - -The `DROP CONSTRAINT` [statement](sql-statements.html) is part of [`ALTER TABLE`](alter-table.html) and removes [`CHECK`](check.html) and [`FOREIGN KEY`](foreign-key.html) constraints from columns. - - [`PRIMARY KEY`](primary-key.html) constraints can be dropped with `DROP CONSTRAINT` if an [`ADD CONSTRAINT`](add-constraint.html) statement follows the `DROP CONSTRAINT` statement in the same transaction. - -{{site.data.alerts.callout_success}} -When you change a primary key with [`ALTER TABLE ... ALTER PRIMARY KEY`](alter-primary-key.html), the old primary key index becomes a secondary index. If you do not want the old primary key to become a secondary index, use `DROP CONSTRAINT`/[`ADD CONSTRAINT`](add-constraint.html) to change the primary key. -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -For information about removing other constraints, see [Constraints: Remove Constraints](constraints.html#remove-constraints). -{{site.data.alerts.end}} - -{% include {{ page.version.version }}/sql/combine-alter-table-commands.md %} - -## Synopsis - -
{% include {{ page.version.version }}/sql/generated/diagrams/drop_constraint.html %}
- -## Required privileges - -The user must have the `CREATE` [privilege](authorization.html#assign-privileges) on the table. - -## Parameters - - Parameter | Description ------------|------------- - `table_name` | The name of the table with the constraint you want to drop. - `name` | The name of the constraint you want to drop. - -## Viewing schema changes - -{% include {{ page.version.version }}/misc/schema-change-view-job.md %} - -## Examples - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### Drop a foreign key constraint - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CONSTRAINTS FROM vehicles; -~~~ - -~~~ - table_name | constraint_name | constraint_type | details | validated --------------+-------------------+-----------------+---------------------------------------------------------+------------ - vehicles | fk_city_ref_users | FOREIGN KEY | FOREIGN KEY (city, owner_id) REFERENCES users(city, id) | true - vehicles | primary | PRIMARY KEY | PRIMARY KEY (city ASC, id ASC) | true -(2 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE vehicles DROP CONSTRAINT fk_city_ref_users; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CONSTRAINTS FROM vehicles; -~~~ - -~~~ - table_name | constraint_name | constraint_type | details | validated --------------+-----------------+-----------------+--------------------------------+------------ - vehicles | primary | PRIMARY KEY | PRIMARY KEY (city ASC, id ASC) | true -(1 row) -~~~ - -### Drop and add a primary key constraint - -When you change a primary key with [`ALTER TABLE ... ALTER PRIMARY KEY`](alter-primary-key.html), the old primary key index becomes a secondary index. If you do not want the old primary key to become a secondary index when changing a primary key, you can use `DROP CONSTRAINT`/[`ADD CONSTRAINT`](add-constraint.html) instead. - -Suppose that you want to add `name` to the composite primary key of the `users` table. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE users; -~~~ - -~~~ - table_name | create_statement --------------+-------------------------------------------------------------- - users | CREATE TABLE users ( - | id UUID NOT NULL, - | city VARCHAR NOT NULL, - | name VARCHAR NULL, - | address VARCHAR NULL, - | credit_card VARCHAR NULL, - | CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - | FAMILY "primary" (id, city, name, address, credit_card) - | ) -(1 row) -~~~ - -First, add a [`NOT NULL`](not-null.html) constraint to the `name` column with [`ALTER COLUMN`](alter-column.html). - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE users ALTER COLUMN name SET NOT NULL; -~~~ - -Then, in the same transaction, `DROP` the old `"primary"` constraint and [`ADD`](add-constraint.html) the new one: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; -> ALTER TABLE users DROP CONSTRAINT "primary"; -> ALTER TABLE users ADD CONSTRAINT "primary" PRIMARY KEY (city, name, id); -> COMMIT; -~~~ - -~~~ -NOTICE: primary key changes are finalized asynchronously; further schema changes on this table may be restricted until the job completes -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE users; -~~~ - -~~~ - table_name | create_statement --------------+--------------------------------------------------------------------- - users | CREATE TABLE users ( - | id UUID NOT NULL, - | city VARCHAR NOT NULL, - | name VARCHAR NOT NULL, - | address VARCHAR NULL, - | credit_card VARCHAR NULL, - | CONSTRAINT "primary" PRIMARY KEY (city ASC, name ASC, id ASC), - | FAMILY "primary" (id, city, name, address, credit_card) - | ) -(1 row) -~~~ - -Using [`ALTER PRIMARY KEY`](alter-primary-key.html) would have created a `UNIQUE` secondary index called `users_city_id_key`. Instead, there is just one index for the primary key constraint. - -## See also - -- [`ADD CONSTRAINT`](add-constraint.html) -- [`SHOW CONSTRAINTS`](show-constraints.html) -- [`RENAME CONSTRAINT`](rename-constraint.html) -- [`VALIDATE CONSTRAINT`](validate-constraint.html) -- [`DROP COLUMN`](drop-column.html) -- [`DROP INDEX`](drop-index.html) -- [`ALTER TABLE`](alter-table.html) -- [`SHOW JOBS`](show-jobs.html) -- ['ALTER PRIMARY KEY'](alter-primary-key.html) diff --git a/src/current/v21.1/drop-database.md b/src/current/v21.1/drop-database.md deleted file mode 100644 index 47484f1eb17..00000000000 --- a/src/current/v21.1/drop-database.md +++ /dev/null @@ -1,143 +0,0 @@ ---- -title: DROP DATABASE -summary: The DROP DATABASE statement removes a database and all its objects from a CockroachDB cluster. -toc: true ---- - -The `DROP DATABASE` [statement](sql-statements.html) removes a database and all its objects from a CockroachDB cluster. - -{% include {{{ page.version.version }}/misc/schema-change-stmt-note.md %} - -## Required privileges - -The user must have the `DROP` [privilege](authorization.html#assign-privileges) on the database and on all tables in the database. - -## Synopsis - -
{% include {{ page.version.version }}/sql/generated/diagrams/drop_database.html %}
- -## Parameters - -Parameter | Description -----------|------------ -`IF EXISTS` | Drop the database if it exists; if it does not exist, do not return an error. -`name` | The name of the database you want to drop. You cannot drop a database if it is set as the [current database](sql-name-resolution.html#current-database) or if [`sql_safe_updates = true`](set-vars.html). -`CASCADE` | _(Default)_ Drop all tables and views in the database as well as all objects (such as [constraints](constraints.html) and [views](views.html)) that depend on those tables.

`CASCADE` does not list objects it drops, so should be used cautiously. -`RESTRICT` | Do not drop the database if it contains any [tables](create-table.html) or [views](create-view.html). - -## Viewing schema changes - -{% include {{ page.version.version }}/misc/schema-change-view-job.md %} - -## Examples - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### Drop a database and its objects (`CASCADE`) - -For non-interactive sessions (e.g., client applications), `DROP DATABASE` applies the `CASCADE` option by default, which drops all tables and views in the database as well as all objects (such as [constraints](constraints.html) and [views](views.html)) that depend on those tables. - -For interactive sessions from the [built-in SQL client](cockroach-sql.html), either the `CASCADE` option must be set explicitly or the `--unsafe-updates` flag must be set when starting the shell. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM movr; -~~~ - -~~~ - schema_name | table_name | type | estimated_row_count ---------------+----------------------------+-------+---------------------- - public | promo_codes | table | 1000 - public | rides | table | 500 - public | user_promo_codes | table | 0 - public | users | table | 50 - public | vehicle_location_histories | table | 1000 - public | vehicles | table | 15 -(6 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP DATABASE movr; -~~~ - -~~~ -ERROR: rejected (sql_safe_updates = true): DROP DATABASE on current database -SQLSTATE: 01000 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> USE defaultdb; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP DATABASE movr; -~~~ - -~~~ -ERROR: rejected (sql_safe_updates = true): DROP DATABASE on non-empty database without explicit CASCADE -SQLSTATE: 01000 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP DATABASE movr CASCADE; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM movr; -~~~ - -~~~ -ERROR: target database or schema does not exist -SQLSTATE: 3F000 -~~~ - -### Prevent dropping a non-empty database (`RESTRICT`) - -When a database is not empty, the `RESTRICT` option prevents the database from being dropped: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM movr; -~~~ - -~~~ - schema_name | table_name | type | estimated_row_count ---------------+----------------------------+-------+---------------------- - public | promo_codes | table | 1000 - public | rides | table | 500 - public | user_promo_codes | table | 0 - public | users | table | 50 - public | vehicle_location_histories | table | 1000 - public | vehicles | table | 15 -(6 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> USE defaultdb; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP DATABASE movr RESTRICT; -~~~ - -~~~ -ERROR: database "movr" is not empty and RESTRICT was specified -SQLSTATE: 2BP01 -~~~ - -## See also - -- [`CREATE DATABASE`](create-database.html) -- [`SHOW DATABASES`](show-databases.html) -- [`RENAME DATABASE`](rename-database.html) -- [`SET DATABASE`](set-vars.html) -- [`SHOW JOBS`](show-jobs.html) -- [Other SQL Statements](sql-statements.html) -- [Online Schema Changes](online-schema-changes.html) diff --git a/src/current/v21.1/drop-index.md b/src/current/v21.1/drop-index.md deleted file mode 100644 index 730c44eda5a..00000000000 --- a/src/current/v21.1/drop-index.md +++ /dev/null @@ -1,180 +0,0 @@ ---- -title: DROP INDEX -summary: The DROP INDEX statement removes indexes from tables. -toc: true ---- - -The `DROP INDEX` [statement](sql-statements.html) removes indexes from tables. - -{% include {{{ page.version.version }}/misc/schema-change-stmt-note.md %} - -## Synopsis - -
{% include {{ page.version.version }}/sql/generated/diagrams/drop_index.html %}
- -## Required privileges - -The user must have the `CREATE` [privilege](authorization.html#assign-privileges) on each specified table. - -## Parameters - - Parameter | Description ------------|------------- - `IF EXISTS` | Drop the named indexes if they exist; if they do not exist, do not return an error. - `table_name` | The name of the table with the index you want to drop. Find table names with [`SHOW TABLES`](show-tables.html). - `index_name` | The name of the index you want to drop. Find index names with [`SHOW INDEX`](show-index.html).

You cannot drop a table's `primary` index. - `CASCADE` | Drop all objects (such as [constraints](constraints.html)) that depend on the indexes. `CASCADE` does not list objects it drops, so should be used cautiously.

To drop an index created with [`CREATE UNIQUE INDEX`](create-index.html#unique-indexes), you do not need to use `CASCADE`. - `RESTRICT` | _(Default)_ Do not drop the indexes if any objects (such as [constraints](constraints.html)) depend on them. - `CONCURRENTLY` | Optional, no-op syntax for PostgreSQL compatibility. All indexes are dropped concurrently in CockroachDB. - -## Viewing schema changes - -{% include {{ page.version.version }}/misc/schema-change-view-job.md %} - -## Examples - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### Remove an index with no dependencies - -{{site.data.alerts.callout_danger}} -{% include {{ page.version.version }}/known-limitations/drop-unique-index-from-create-table.md %} -{{site.data.alerts.end}} - -Suppose you create an index on the `name` and `city` columns of the `users` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE INDEX ON users (name, city); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW INDEXES FROM users; -~~~ - -~~~ - table_name | index_name | non_unique | seq_in_index | column_name | direction | storing | implicit --------------+---------------------+------------+--------------+-------------+-----------+---------+----------- - users | primary | false | 1 | city | ASC | false | false - users | primary | false | 2 | id | ASC | false | false - users | users_name_city_idx | true | 1 | name | ASC | false | false - users | users_name_city_idx | true | 2 | city | ASC | false | false - users | users_name_city_idx | true | 3 | id | ASC | false | true -(5 rows) -~~~ - -You can drop this index with the `DROP INDEX` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP INDEX users@users_name_city_idx; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW INDEXES FROM users; -~~~ - -~~~ - table_name | index_name | non_unique | seq_in_index | column_name | direction | storing | implicit --------------+------------+------------+--------------+-------------+-----------+---------+----------- - users | primary | false | 1 | city | ASC | false | false - users | primary | false | 2 | id | ASC | false | false -(2 rows) -~~~ - -### Remove an index and dependent objects with `CASCADE` - -{{site.data.alerts.callout_danger}} -CASCADE drops all dependent objects without listing them, which can lead to inadvertent and difficult-to-recover losses. To avoid potential harm, we recommend dropping objects individually in most cases. -{{site.data.alerts.end}} - -Suppose you create a [`UNIQUE`](unique.html) constraint on the `id` and `name` columns of the `users` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE users ADD CONSTRAINT id_name_unique UNIQUE (id, name); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CONSTRAINTS from users; -~~~ - -~~~ - table_name | constraint_name | constraint_type | details | validated --------------+-----------------+-----------------+--------------------------------+------------ - users | id_name_unique | UNIQUE | UNIQUE (id ASC, name ASC) | true - users | primary | PRIMARY KEY | PRIMARY KEY (city ASC, id ASC) | true -(2 rows) -~~~ - -If no index exists on `id` and `name`, CockroachDB automatically creates an index: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW INDEXES from users; -~~~ - -~~~ - table_name | index_name | non_unique | seq_in_index | column_name | direction | storing | implicit --------------+----------------+------------+--------------+-------------+-----------+---------+----------- - users | primary | false | 1 | city | ASC | false | false - users | primary | false | 2 | id | ASC | false | false - users | id_name_unique | false | 1 | id | ASC | false | false - users | id_name_unique | false | 2 | name | ASC | false | false - users | id_name_unique | false | 3 | city | ASC | false | true -(5 rows) -~~~ - -The `UNIQUE` constraint is dependent on the `id_name_unique` index, so you cannot drop the index with a simple `DROP INDEX` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP INDEX id_name_unique; -~~~ - -~~~ -ERROR: index "id_name_unique" is in use as unique constraint -SQLSTATE: 2BP01 -HINT: use CASCADE if you really want to drop it. -~~~ - -To drop an index and its dependent objects, you can use `CASCADE`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP INDEX id_name_unique CASCADE; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW INDEXES from users; -~~~ - -~~~ - table_name | index_name | non_unique | seq_in_index | column_name | direction | storing | implicit --------------+------------+------------+--------------+-------------+-----------+---------+----------- - users | primary | false | 1 | city | ASC | false | false - users | primary | false | 2 | id | ASC | false | false -(2 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CONSTRAINTS from users; -~~~ - -~~~ - table_name | constraint_name | constraint_type | details | validated --------------+-----------------+-----------------+--------------------------------+------------ - users | primary | PRIMARY KEY | PRIMARY KEY (city ASC, id ASC) | true -(1 row) -~~~ - -## See Also - -- [Indexes](indexes.html) -- [Online Schema Changes](online-schema-changes.html) -- [`SHOW JOBS`](show-jobs.html) diff --git a/src/current/v21.1/drop-region.md b/src/current/v21.1/drop-region.md deleted file mode 100644 index 004322ca267..00000000000 --- a/src/current/v21.1/drop-region.md +++ /dev/null @@ -1,223 +0,0 @@ ---- -title: DROP REGION -summary: The DROP REGION statement drops a region from a multi-region database. -toc: true ---- - -{% include_cached new-in.html version="v21.1" %} The `ALTER DATABASE .. DROP REGION` [statement](sql-statements.html) drops a [region](multiregion-overview.html#database-regions) from a [multi-region database](multiregion-overview.html). While CockroachDB processes an index modification or changing a table to or from a [`REGIONAL BY ROW` table](multiregion-overview.html#regional-by-row-tables), attempting to drop a region from the database containing that `REGIONAL BY ROW` table will produce an error. Similarly, while this statement is running, all index modifications and locality changes on [`REGIONAL BY ROW`](multiregion-overview.html#regional-by-row-tables) tables will be blocked. - -{% include enterprise-feature.md %} - -{{site.data.alerts.callout_info}} -`DROP REGION` is a subcommand of [`ALTER DATABASE`](alter-database.html). -{{site.data.alerts.end}} - -## Synopsis - -
-{% include {{ page.version.version }}/sql/generated/diagrams/alter_database_drop_region.html %} -
- -## Parameters - -| Parameter | Description | -|-----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `database_name` | The database from which you are dropping a [region](multiregion-overview.html#database-regions). | -| `region_name` | The [region](multiregion-overview.html#database-regions) being dropped from this database. Allowed values include any region present in `SHOW REGIONS FROM DATABASE database_name`.
You can only drop the primary region from a multi-region database if it's the last remaining region. | - -## Required privileges - -To drop a region from a database, the user must have one of the following: - -- Membership to the [`admin`](authorization.html#roles) role for the cluster. -- Membership to the [owner](authorization.html#object-ownership) role, or the [`CREATE` privilege](authorization.html#supported-privileges), for the database and all [`REGIONAL BY ROW`](multiregion-overview.html#regional-by-row-tables) tables in the database. - -## Examples - -{% include {{page.version.version}}/sql/multiregion-example-setup.md %} - -### Set the primary region - -Suppose you have a database `foo` in your cluster, and you want to make it a multi-region database. - -To add the first region to the database, or to set an already-added region as the primary region, use a [`SET PRIMARY REGION`](set-primary-region.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE foo SET PRIMARY REGION "us-east1"; -~~~ - -~~~ -ALTER DATABASE PRIMARY REGION -~~~ - -### Add regions to a database - -To add more regions to a database that already has at least one region, use an [`ADD REGION`](add-region.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER database foo ADD region "us-west1"; -~~~ - -~~~ -ALTER DATABASE ADD REGION -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER database foo ADD region "europe-west1"; -~~~ - -~~~ -ALTER DATABASE ADD REGION -~~~ - -### View a database's regions - -To view the regions associated with a multi-region database, use a [`SHOW REGIONS FROM DATABASE`](show-regions.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW REGIONS FROM DATABASE foo; -~~~ - -~~~ - database | region | primary | zones ------------+--------------+---------+---------- - foo | us-east1 | true | {b,c,d} - foo | europe-west1 | false | {b,c,d} - foo | us-west1 | false | {a,b,c} -(3 rows) -~~~ - -### Drop regions from a database - -To drop a region from a multi-region database, use a `DROP REGION` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE foo DROP REGION "us-west1"; -~~~ - -~~~ -ALTER DATABASE DROP REGION -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW REGIONS FROM DATABASE foo; -~~~ - -~~~ - database | region | primary | zones ------------+--------------+---------+---------- - foo | us-east1 | true | {b,c,d} - foo | europe-west1 | false | {b,c,d} -(2 rows) -~~~ - -You can only drop the primary region from a multi-region database if it's the last remaining region. - -If you try to drop the primary region when there is more than one region, CockroachDB will return an error: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE foo DROP REGION "us-east1"; -~~~ - -~~~ -ERROR: cannot drop region "us-east1" -SQLSTATE: 42P12 -HINT: You must designate another region as the primary region using ALTER DATABASE foo PRIMARY REGION or remove all other regions before attempting to drop region "us-east1" -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE foo DROP REGION "europe-west1"; -~~~ - -~~~ -ALTER DATABASE DROP REGION -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW REGIONS FROM DATABASE foo; -~~~ - -~~~ - database | region | primary | zones ------------+----------+---------+---------- - foo | us-east1 | true | {b,c,d} -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE foo DROP REGION "us-east1"; -~~~ - -~~~ -ALTER DATABASE DROP REGION -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW REGIONS FROM DATABASE foo; -~~~ - -~~~ - database | region | primary | zones ------------+--------+---------+-------- -(0 rows) -~~~ - -You cannot drop a region from a database if the databases uses [`REGION` survival goal](multiregion-overview.html#surviving-region-failures) and there are only three regions configured on the database: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE foo SET PRIMARY REGION "us-east1"; -~~~ - -~~~ -ALTER DATABASE PRIMARY REGION -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE foo ADD REGION "us-west1"; -~~~ - -~~~ -ALTER DATABASE ADD REGION -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE foo ADD REGION "europe-west1"; -~~~ - -~~~ -ALTER DATABASE ADD REGION -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE foo DROP REGION "us-west1"; -~~~ - -~~~ -ERROR: at least 3 regions are required for surviving a region failure -SQLSTATE: 22023 -HINT: you must add additional regions to the database or change the survivability goal -~~~ - -## See also - -- [Multi-region overview](multiregion-overview.html) -- [`SET PRIMARY REGION`](set-primary-region.html) -- [`ADD REGION`](add-region.html) -- [`SHOW REGIONS`](show-regions.html) -- [`ALTER TABLE`](alter-table.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/drop-role.md b/src/current/v21.1/drop-role.md deleted file mode 100644 index 45f7b3cda62..00000000000 --- a/src/current/v21.1/drop-role.md +++ /dev/null @@ -1,68 +0,0 @@ ---- -title: DROP ROLE -summary: The DROP ROLE statement removes one or more SQL roles. -toc: true ---- - -The `DROP ROLE` [statement](sql-statements.html) removes one or more SQL roles. - -{{site.data.alerts.callout_info}} - DROP ROLE is no longer an Enterprise feature and is now freely available in the core version of CockroachDB. Also, since the keywords `ROLE` and `USER` can now be used interchangeably in SQL statements for enhanced Postgres compatibility, `DROP ROLE` is now an alias for [`DROP USER`](drop-user.html). -{{site.data.alerts.end}} - -## Considerations - -- The `admin` role cannot be dropped, and `root` must always be a member of `admin`. -- A role cannot be dropped if it has privileges. Use [`REVOKE`](revoke.html) to remove privileges. -- Roles that [own objects](authorization.html#object-ownership) (such as databases, tables, schemas, and types) cannot be dropped until the [ownership is transferred to another role](owner-to.html#change-a-databases-owner). - -## Required privileges - -Non-admin roles cannot drop admin roles. To drop non-admin roles, the role must be a member of the `admin` role or have the [`CREATEROLE`](create-role.html#create-a-role-that-can-create-other-roles-and-manage-authentication-methods-for-the-new-roles) parameter set. - -## Synopsis - -
{% include {{ page.version.version }}/sql/generated/diagrams/drop_role.html %}
- -## Parameters - - Parameter | Description -------------|-------------- -`name` | The name of the role to remove. To remove multiple roles, use a comma-separate list of roles.

You can use [`SHOW ROLES`](show-roles.html) to find the names of roles. - -## Example - -In this example, first check a role's privileges. Then, revoke the role's privileges and remove the role. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON documents FOR dev_ops; -~~~ -~~~ -+------------+--------+-----------+---------+------------+ -| Database | Schema | Table | User | Privileges | -+------------+--------+-----------+---------+------------+ -| jsonb_test | public | documents | dev_ops | INSERT | -+------------+--------+-----------+---------+------------+ -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> REVOKE INSERT ON documents FROM dev_ops; -~~~ - -{{site.data.alerts.callout_info}}All of a role's privileges must be revoked before the role can be dropped.{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP ROLE dev_ops; -~~~ - -## See also - -- [Authorization](authorization.html) -- [`CREATE ROLE`](create-role.html) -- [`SHOW ROLES`](show-roles.html) -- [`GRANT`](grant.html) -- [`SHOW GRANTS`](show-grants.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/drop-schedules.md b/src/current/v21.1/drop-schedules.md deleted file mode 100644 index 84f76dad605..00000000000 --- a/src/current/v21.1/drop-schedules.md +++ /dev/null @@ -1,74 +0,0 @@ ---- -title: DROP SCHEDULES -summary: The DROP SCHEDULES statement lets you remove specified backup schedules. -toc: true ---- - - The `DROP SCHEDULES` [statement](sql-statements.html) can be used to remove [backup schedules](create-schedule-for-backup.html). - -{{site.data.alerts.callout_danger}} -`DROP SCHEDULE` does **not** cancel any in-progress jobs started by the schedule. Before you drop a schedule, [cancel any in-progress jobs](cancel-job.html) first, as you will not be able to look up the job ID once the schedule is dropped. -{{site.data.alerts.end}} - -## Required privileges - -Only members of the [`admin` role](authorization.html#default-roles) can drop a schedule. By default, the `root` user belongs to the `admin` role. - -## Synopsis - -~~~ -DROP SCHEDULES - select clause: select statement returning schedule id to pause. -DROP SCHEDULE -~~~ - -## Parameters - - Parameter | Description ----------------+------------ -`selectclause` | A [selection query](selection-queries.html) that returns `id`(s) to drop. -`scheduleID` | The `id` of the schedule you want to drop, which can be found with [`SHOW SCHEDULES`](show-schedules.html). - -## Examples - -### Drop a schedule - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP SCHEDULE 589963390487363585; -~~~ - -~~~ -DROP SCHEDULES 1 -~~~ - -### Drop multiple schedules - -To drop multiple schedules, nest a [`SELECT` clause](select-clause.html) that retrieves `id`(s) inside the `DROP SCHEDULES` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP SCHEDULES SELECT id FROM [SHOW SCHEDULES] WHERE label = 'schedule_database'; -~~~ - -~~~ -DROP SCHEDULES 4 -~~~ - -In this example, all schedules with the label `schedule_database` are dropped. - -## See also - -- [Manage a Backup Schedule](manage-a-backup-schedule.html) -- [`BACKUP`](backup.html) -- [`RESTORE`](restore.html) -- [`SHOW BACKUP`](show-backup.html) -- [`SHOW SCHEDULES`](show-schedules.html) -- [`PAUSE SCHEDULES`](pause-schedules.html) -- [`RESUME SCHEDULES`](resume-schedules.html) -- [`PAUSE JOB`](pause-job.html) -- [`RESUME JOB`](pause-job.html) -- [`CANCEL JOB`](cancel-job.html) -- [Take Full and Incremental Backups](take-full-and-incremental-backups.html) -- [Use the Built-in SQL Client](cockroach-sql.html) -- [Other Cockroach Commands](cockroach-commands.html) diff --git a/src/current/v21.1/drop-schema.md b/src/current/v21.1/drop-schema.md deleted file mode 100644 index b306339faa3..00000000000 --- a/src/current/v21.1/drop-schema.md +++ /dev/null @@ -1,162 +0,0 @@ ---- -title: DROP SCHEMA -summary: The DROP SCHEMA statement removes a schema and all its objects from a CockroachDB cluster. -toc: true ---- - - The `DROP SCHEMA` [statement](sql-statements.html) removes a user-defined [schema](sql-name-resolution.html#naming-hierarchy). - -## Required privileges - -The user must have the `DROP` [privilege](authorization.html#assign-privileges) on the schema and on all tables in the schema. If the user is the owner of the schema, `DROP` privileges are not necessary. - -## Syntax - -
-{% include {{ page.version.version }}/sql/generated/diagrams/drop_schema.html %} -
- -### Parameters - -Parameter | Description -----------|------------ -`IF EXISTS` | Drop the schema if it exists. If it does not exist, do not return an error. -`schema_name_list` | The schema, or a list of schemas, that you want to drop.
To drop a schema in a database other than the current database, specify the name of the database and the name of the schema, separated by a "`.`" (e.g., `DROP SCHEMA IF EXISTS database.schema;`). -`CASCADE` | Drop all tables and views in the schema as well as all objects (such as [constraints](constraints.html) and [views](views.html)) that depend on those tables.

`CASCADE` does not list objects it drops, so should be used cautiously. -`RESTRICT` | _(Default)_ Do not drop the schema if it contains any [tables](create-table.html) or [views](create-view.html). - -## Examples - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### Drop a schema - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE SCHEMA org_one; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SCHEMAS; -~~~ - -~~~ - schema_name ----------------------- - crdb_internal - information_schema - org_one - pg_catalog - pg_extension - public -(6 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP SCHEMA org_one; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SCHEMAS; -~~~ - -~~~ - schema_name ----------------------- - crdb_internal - information_schema - pg_catalog - pg_extension - public -(5 rows) -~~~ - -### Drop a schema with tables - -To drop a schema that contains tables, you need to use the `CASCADE` keyword. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE SCHEMA org_two; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SCHEMAS; -~~~ - -~~~ - schema_name ----------------------- - crdb_internal - information_schema - org_two - pg_catalog - pg_extension - public -(6 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE org_two.users ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - city STRING, - name STRING, - address STRING -); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM org_two; -~~~ - -~~~ - schema_name | table_name | type | estimated_row_count ---------------+------------+-------+---------------------- - org_two | users | table | 0 -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP SCHEMA org_two; -~~~ - -~~~ -ERROR: schema "org_two" is not empty and CASCADE was not specified -SQLSTATE: 2BP01 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP SCHEMA org_two CASCADE; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SCHEMAS; -~~~ - -~~~ - schema_name ----------------------- - crdb_internal - information_schema - pg_catalog - pg_extension - public -(5 rows) -~~~ - -## See also - -- [`CREATE SCHEMA`](create-schema.html) -- [`SHOW SCHEMAS`](show-schemas.html) -- [`SHOW JOBS`](show-jobs.html) -- [Other SQL Statements](sql-statements.html) -- [Online Schema Changes](online-schema-changes.html) diff --git a/src/current/v21.1/drop-sequence.md b/src/current/v21.1/drop-sequence.md deleted file mode 100644 index 69d2666e8c9..00000000000 --- a/src/current/v21.1/drop-sequence.md +++ /dev/null @@ -1,94 +0,0 @@ ---- -title: DROP SEQUENCE -summary: The DROP SEQUENCE statement removes a sequence from a database. -toc: true ---- - -The `DROP SEQUENCE` [statement](sql-statements.html) removes a sequence from a database. - -{% include {{{ page.version.version }}/misc/schema-change-stmt-note.md %} - -## Required privileges - -The user must have the `DROP` [privilege](authorization.html#assign-privileges) on the specified sequence(s). - -## Synopsis - -
{% include {{ page.version.version }}/sql/generated/diagrams/drop_sequence.html %}
- -## Parameters - - - - Parameter | Description ------------|------------ -`IF EXISTS` | Drop the sequence only if it exists; if it does not exist, do not return an error. -`sequence_name` | The name of the sequence you want to drop. Find the sequence name with `SHOW CREATE` on the table that uses the sequence. -`RESTRICT` | _(Default)_ Do not drop the sequence if any objects (such as [constraints](constraints.html) and tables) use it. -`CASCADE` | Not yet implemented. Currently, you can only drop a sequence if nothing depends on it. - - - -## Examples - -### Remove a sequence (no dependencies) - -In this example, other objects do not depend on the sequence being dropped. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE SEQUENCE even_numbers INCREMENT 2 START 2; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SEQUENCES; -~~~ - -~~~ - sequence_schema | sequence_name -------------------+---------------- - public | even_numbers -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP SEQUENCE even_numbers; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SEQUENCES; -~~~ - -~~~ - sequence_schema | sequence_name -------------------+---------------- -(0 rows) -~~~ - - - -## See also -- [`CREATE SEQUENCE`](create-sequence.html) -- [`ALTER SEQUENCE`](alter-sequence.html) -- [`SHOW SEQUENCES`](show-sequences.html) -- [Functions and Operators](functions-and-operators.html) -- [Other SQL Statements](sql-statements.html) -- [Online Schema Changes](online-schema-changes.html) diff --git a/src/current/v21.1/drop-table.md b/src/current/v21.1/drop-table.md deleted file mode 100644 index 83bff508f25..00000000000 --- a/src/current/v21.1/drop-table.md +++ /dev/null @@ -1,204 +0,0 @@ ---- -title: DROP TABLE -summary: The DROP TABLE statement removes a table and all its indexes from a database. -toc: true ---- - -The `DROP TABLE` [statement](sql-statements.html) removes a table and all its indexes from a database. - -{% include {{{ page.version.version }}/misc/schema-change-stmt-note.md %} - -## Required privileges - -The user must have the `DROP` [privilege](authorization.html#assign-privileges) on the specified table(s). If `CASCADE` is used, the user must have the privileges required to drop each dependent object as well. - -## Synopsis - -
{% include {{ page.version.version }}/sql/generated/diagrams/drop_table.html %}
- -## Parameters - -Parameter | Description -----------|------------ -`IF EXISTS` | Drop the table if it exists; if it does not exist, do not return an error. -`table_name` | A comma-separated list of table names. To find table names, use [`SHOW TABLES`](show-tables.html). -`CASCADE` | Drop all objects (such as [constraints](constraints.html) and [views](views.html)) that depend on the table.

`CASCADE` does not list objects it drops, so should be used cautiously. -`RESTRICT` | _(Default)_ Do not drop the table if any objects (such as [constraints](constraints.html) and [views](views.html)) depend on it. - -## Viewing schema changes - -{% include {{ page.version.version }}/misc/schema-change-view-job.md %} - -## Examples - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### Remove a table (no dependencies) - -In this example, other objects do not depend on the table being dropped. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM movr; -~~~ - -~~~ - schema_name | table_name | type | estimated_row_count ---------------+----------------------------+-------+---------------------- - public | promo_codes | table | 1000 - public | rides | table | 500 - public | user_promo_codes | table | 0 - public | users | table | 50 - public | vehicle_location_histories | table | 1000 - public | vehicles | table | 15 -(6 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP TABLE promo_codes; -~~~ - -~~~ -DROP TABLE -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM movr; -~~~ - -~~~ - schema_name | table_name | type | estimated_row_count ---------------+----------------------------+-------+---------------------- - public | rides | table | 500 - public | user_promo_codes | table | 0 - public | users | table | 50 - public | vehicle_location_histories | table | 1000 - public | vehicles | table | 15 -(5 rows) -~~~ - -### Remove a table and dependent objects with `CASCADE` - -In this example, a [foreign key](foreign-key.html) from a different table references the table being dropped. Therefore, it's only possible to drop the table while simultaneously dropping the dependent foreign key constraint using `CASCADE`. - -{{site.data.alerts.callout_danger}}CASCADE drops all dependent objects without listing them, which can lead to inadvertent and difficult-to-recover losses. To avoid potential harm, we recommend dropping objects individually in most cases.{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM movr; -~~~ - -~~~ - schema_name | table_name | type | estimated_row_count ---------------+----------------------------+-------+---------------------- - public | rides | table | 500 - public | user_promo_codes | table | 0 - public | users | table | 50 - public | vehicle_location_histories | table | 1000 - public | vehicles | table | 15 -(5 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP TABLE users; -~~~ - -~~~ -pq: "users" is referenced by foreign key from table "vehicles" -~~~ - -To see how `users` is referenced from `vehicles`, you can use the [`SHOW CREATE`](show-create.html) statement. `SHOW CREATE` shows how the columns in a table are created, including data types, default values, indexes, and constraints. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE vehicles; -~~~ - -~~~ - table_name | create_statement --------------+--------------------------------------------------------------------------------------------------- - vehicles | CREATE TABLE public.vehicles ( - | id UUID NOT NULL, - | city VARCHAR NOT NULL, - | type VARCHAR NULL, - | owner_id UUID NULL, - | creation_time TIMESTAMP NULL, - | status VARCHAR NULL, - | current_location VARCHAR NULL, - | ext JSONB NULL, - | CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - | CONSTRAINT fk_city_ref_users FOREIGN KEY (city, owner_id) REFERENCES public.users(city, id), - | INDEX vehicles_auto_index_fk_city_ref_users (city ASC, owner_id ASC), - | FAMILY "primary" (id, city, type, owner_id, creation_time, status, current_location, ext) - | ) -(1 row) -~~~ - - -{% include_cached copy-clipboard.html %} -~~~sql -> DROP TABLE users CASCADE; -~~~ - -~~~ -DROP TABLE -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM movr; -~~~ - -~~~ - schema_name | table_name | type | estimated_row_count ---------------+----------------------------+-------+---------------------- - public | rides | table | 500 - public | user_promo_codes | table | 0 - public | vehicle_location_histories | table | 1000 - public | vehicles | table | 15 -(4 rows) -~~~ - -Use a `SHOW CREATE TABLE` statement to verify that the foreign key constraint has been removed from `vehicles`. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE vehicles; -~~~ - -~~~ - table_name | create_statement --------------+------------------------------------------------------------------------------------------------ - vehicles | CREATE TABLE public.vehicles ( - | id UUID NOT NULL, - | city VARCHAR NOT NULL, - | type VARCHAR NULL, - | owner_id UUID NULL, - | creation_time TIMESTAMP NULL, - | status VARCHAR NULL, - | current_location VARCHAR NULL, - | ext JSONB NULL, - | CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - | INDEX vehicles_auto_index_fk_city_ref_users (city ASC, owner_id ASC), - | FAMILY "primary" (id, city, type, owner_id, creation_time, status, current_location, ext) - | ) -(1 row) -~~~ - -## See also - -- [`ALTER TABLE`](alter-table.html) -- [`CREATE TABLE`](create-table.html) -- [`INSERT`](insert.html) -- [`RENAME TABLE`](rename-table.html) -- [`SHOW COLUMNS`](show-columns.html) -- [`SHOW TABLES`](show-tables.html) -- [`UPDATE`](update.html) -- [`DELETE`](delete.html) -- [`DROP INDEX`](drop-index.html) -- [`DROP VIEW`](drop-view.html) -- [`SHOW JOBS`](show-jobs.html) -- [Online Schema Changes](online-schema-changes.html) diff --git a/src/current/v21.1/drop-type.md b/src/current/v21.1/drop-type.md deleted file mode 100644 index d3d33880e47..00000000000 --- a/src/current/v21.1/drop-type.md +++ /dev/null @@ -1,140 +0,0 @@ ---- -title: DROP TYPE -summary: The DROP TYPE statement drops an enumerated data type from the database. -toc: true ---- - - The `DROP TYPE` [statement](sql-statements.html) drops a specified [enumerated data type](enum.html) from the current database. - -## Synopsis - -
-{% include {{ page.version.version }}/sql/generated/diagrams/drop_type.html %} -
- -## Parameters - -Parameter | Description -----------|------------ -`IF EXISTS` | Drop the type if it exists. If it does not exist, do not return an error. -`type_name_list` | A type name or a comma-separated list of type names to drop. - -## Required privileges - -The user must be the owner of the type. - -## Details - -- You cannot drop a type or view that is in use by a table. -- You can only drop a user-defined type from the database that contains the type. - -## Example - -### Drop a single type - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TYPE status AS ENUM ('open', 'closed', 'inactive'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW ENUMS; -~~~ - -~~~ - schema | name | value ----------+--------+----------------------- - public | status | open|closed|inactive -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE accounts ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - balance DECIMAL, - status status -); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP TYPE status; -~~~ - -~~~ -ERROR: cannot drop type "status" because other objects ([bank.public.accounts]) still depend on it -SQLSTATE: 2BP01 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP TABLE accounts; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP TYPE status; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW ENUMS; -~~~ - -~~~ - schema | name | value ----------+------+-------- -(0 rows) -~~~ - -### Drop multiple types - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TYPE weekday AS ENUM ('monday', 'tuesday', 'wednesday', 'thursday', 'friday'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TYPE weekend AS ENUM ('sunday', 'saturday'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW ENUMS; -~~~ - -~~~ - schema | name | value ----------+---------+------------------------------------------- - public | weekday | monday|tuesday|wednesday|thursday|friday - public | weekend | sunday|saturday -(2 rows) -~~~ - - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP TYPE weekday, weekend; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW ENUMS; -~~~ - -~~~ - schema | name | value ----------+------+-------- -(0 rows) -~~~ - -## See also - -- [`ENUM`](enum.html) -- [Data types](data-types.html) -- [`CREATE TYPE`](create-type.html) -- [`ALTER TYPE`](alter-type.html) -- [`SHOW ENUMS`](show-enums.html) diff --git a/src/current/v21.1/drop-user.md b/src/current/v21.1/drop-user.md deleted file mode 100644 index 8a970ef2259..00000000000 --- a/src/current/v21.1/drop-user.md +++ /dev/null @@ -1,71 +0,0 @@ ---- -title: DROP USER -summary: The DROP USER statement removes one or more SQL users. -toc: true ---- - -The `DROP USER` [statement](sql-statements.html) removes one or more SQL users. - -{{site.data.alerts.callout_info}} - Since the keywords `ROLE` and `USER` can now be used interchangeably in SQL statements for enhanced Postgres compatibility, `DROP USER` is now an alias for [`DROP ROLE`](drop-role.html). -{{site.data.alerts.end}} - -## Consideration - -Users that [own objects](authorization.html#object-ownership) (such as databases, tables, schemas, and types) cannot be dropped until the [ownership is transferred to another user](owner-to.html#change-a-databases-owner). - -## Required privileges - -Non-admin users cannot drop admin users. To drop non-admin users, the user must be a member of the `admin` role or have the [`CREATEROLE`](create-user.html#create-a-user-that-can-create-other-users-and-manage-authentication-methods-for-the-new-users) parameter set. - -## Synopsis - -
{% include {{ page.version.version }}/sql/generated/diagrams/drop_user.html %}
- -## Parameters - - Parameter | Description ------------|------------- -`user_name` | The username of the user to remove. To remove multiple users, use a comma-separate list of usernames.

You can use [`SHOW USERS`](show-users.html) to find usernames. - -## Example - -All of a user's privileges must be revoked before the user can be dropped. - -In this example, first check a user's privileges. Then, revoke the user's privileges before removing the user. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON test.customers FOR mroach; -~~~ - -~~~ -+-----------+--------+------------+ -| Table | User | Privileges | -+-----------+--------+------------+ -| customers | mroach | CREATE | -| customers | mroach | INSERT | -| customers | mroach | UPDATE | -+-----------+--------+------------+ -(3 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> REVOKE CREATE,INSERT,UPDATE ON test.customers FROM mroach; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP USER mroach; -~~~ - -## See also - -- [`CREATE USER`](create-user.html) -- [`ALTER USER`](alter-user.html) -- [`SHOW USERS`](show-users.html) -- [`GRANT`](grant.html) -- [`SHOW GRANTS`](show-grants.html) -- [Create Security Certificates](cockroach-cert.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/drop-view.md b/src/current/v21.1/drop-view.md deleted file mode 100644 index e972d375e70..00000000000 --- a/src/current/v21.1/drop-view.md +++ /dev/null @@ -1,132 +0,0 @@ ---- -title: DROP VIEW -summary: The DROP VIEW statement removes a view from a database. -toc: true ---- - -The `DROP VIEW` [statement](sql-statements.html) removes a [view](views.html) from a database. - -{% include {{{ page.version.version }}/misc/schema-change-stmt-note.md %} - -## Required privileges - -The user must have the `DROP` [privilege](authorization.html#assign-privileges) on the specified view(s). If `CASCADE` is used to drop dependent views, the user must have the `DROP` privilege on each dependent view as well. - -## Synopsis - -
{% include {{ page.version.version }}/sql/generated/diagrams/drop_view.html %}
- -## Parameters - - Parameter | Description -----------|------------- -`MATERIALIZED` | Drop a [materialized view](views.html#materialized-views). - `IF EXISTS` | Drop the view if it exists; if it does not exist, do not return an error. - `table_name` | A comma-separated list of view names. To find view names, use:

`SELECT * FROM information_schema.tables WHERE table_type = 'VIEW';` - `CASCADE` | Drop other views that depend on the view being dropped.

`CASCADE` does not list views it drops, so should be used cautiously. - `RESTRICT` | _(Default)_ Do not drop the view if other views depend on it. - -## Examples - -### Remove a view (no dependencies) - -In this example, other views do not depend on the view being dropped. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM information_schema.tables WHERE table_type = 'VIEW'; -~~~ - -~~~ -+---------------+-------------------+--------------------+------------+---------+ -| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | TABLE_TYPE | VERSION | -+---------------+-------------------+--------------------+------------+---------+ -| def | bank | user_accounts | VIEW | 1 | -| def | bank | user_emails | VIEW | 1 | -+---------------+-------------------+--------------------+------------+---------+ -(2 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP VIEW bank.user_emails; -~~~ - -~~~ -DROP VIEW -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM information_schema.tables WHERE table_type = 'VIEW'; -~~~ - -~~~ -+---------------+-------------------+--------------------+------------+---------+ -| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | TABLE_TYPE | VERSION | -+---------------+-------------------+--------------------+------------+---------+ -| def | bank | user_accounts | VIEW | 1 | -+---------------+-------------------+--------------------+------------+---------+ -(1 row) -~~~ - -### Remove a view (with dependencies) - -In this example, another view depends on the view being dropped. Therefore, it's only possible to drop the view while simultaneously dropping the dependent view using `CASCADE`. - -{{site.data.alerts.callout_danger}}CASCADE drops all dependent views without listing them, which can lead to inadvertent and difficult-to-recover losses. To avoid potential harm, we recommend dropping objects individually in most cases.{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM information_schema.tables WHERE table_type = 'VIEW'; -~~~ - -~~~ -+---------------+-------------------+--------------------+------------+---------+ -| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | TABLE_TYPE | VERSION | -+---------------+-------------------+--------------------+------------+---------+ -| def | bank | user_accounts | VIEW | 1 | -| def | bank | user_emails | VIEW | 1 | -+---------------+-------------------+--------------------+------------+---------+ -(2 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP VIEW bank.user_accounts; -~~~ - -~~~ -pq: cannot drop view "user_accounts" because view "user_emails" depends on it -~~~ - -{% include_cached copy-clipboard.html %} -~~~sql -> DROP VIEW bank.user_accounts CASCADE; -~~~ - -~~~ -DROP VIEW -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM information_schema.tables WHERE table_type = 'VIEW'; -~~~ - -~~~ -+---------------+-------------------+--------------------+------------+---------+ -| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | TABLE_TYPE | VERSION | -+---------------+-------------------+--------------------+------------+---------+ -| def | bank | create_test | VIEW | 1 | -+---------------+-------------------+--------------------+------------+---------+ -(1 row) -~~~ - -## See also - -- [Views](views.html) -- [`CREATE VIEW`](create-view.html) -- [`SHOW CREATE`](show-create.html) -- [`ALTER VIEW`](alter-view.html) -- [Online Schema Changes](online-schema-changes.html) diff --git a/src/current/v21.1/enable-node-map.md b/src/current/v21.1/enable-node-map.md deleted file mode 100644 index 953af89bf15..00000000000 --- a/src/current/v21.1/enable-node-map.md +++ /dev/null @@ -1,198 +0,0 @@ ---- -title: Enable the Node Map -summary: Learn how to enable the node map in the DB Console. -toc: true ---- - -The **Node Map** is useful for: - -- Visualizing the geographic configuration of a multi-region cluster on a world map. -- Viewing real-time cluster metrics. -- Drilling down to individual nodes for monitoring health and performance. - -This page guides you through the process of setting up and enabling the Node Map. - -{{site.data.alerts.callout_info}} -On a secure cluster, this area of the DB Console can only be accessed by an `admin` user. See [DB Console access](ui-overview.html#db-console-access). -{{site.data.alerts.end}} - -{% include enterprise-feature.md %} - -DB Console - -## Set up and enable the Node Map - -To enable the Node Map, you need to start the cluster with the correct [`--locality`](cockroach-start.html#locality) flags and assign the latitude and longitude for each locality. - -{{site.data.alerts.callout_info}} -The Node Map will not be displayed until *all* nodes are started with the correct `--locality` flags and all localities are assigned the corresponding latitude and longitude. -{{site.data.alerts.end}} - -Consider a four-node geo-distributed cluster with the following configuration: - -| Node | Region | Datacenter | -| ------ | ------ | ------ | -| Node1 | us-east-1 | us-east-1a | -| Node2 | us-east-1 | us-east-1b | -| Node3 | us-west-1 | us-west-1a | -| Node4 | eu-west-1 | eu-west-1a | - -### Step 1. Start the nodes with the correct `--locality` flags - -To start a new cluster with the correct `--locality` flags: - -Start Node 1: - -{% include_cached copy-clipboard.html %} -~~~ -$ cockroach start \ ---insecure \ ---locality=region=us-east-1,datacenter=us-east-1a \ ---advertise-addr= \ ---cache=.25 \ ---max-sql-memory=.25 \ ---join=,,, -~~~ - -Start Node 2: - -{% include_cached copy-clipboard.html %} -~~~ -$ cockroach start \ ---insecure \ ---locality=region=us-east-1,datacenter=us-east-1b \ ---advertise-addr= \ ---cache=.25 \ ---max-sql-memory=.25 \ ---join=,,, -~~~ - -Start Node 3: - -{% include_cached copy-clipboard.html %} -~~~ -$ cockroach start \ ---insecure \ ---locality=region=us-west-1,datacenter=us-west-1a \ ---advertise-addr= \ ---cache=.25 \ ---max-sql-memory=.25 \ ---join=,,, -~~~ - -Start Node 4: - -{% include_cached copy-clipboard.html %} -~~~ -$ cockroach start \ ---insecure \ ---locality=region=eu-west-1,datacenter=eu-west-1a \ ---advertise-addr= \ ---cache=.25 \ ---max-sql-memory=.25 \ ---join=,,, -~~~ - -Use the [`cockroach init`](cockroach-init.html) command to perform a one-time initialization of the cluster: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach init --insecure --host=
-~~~ - -[Access the DB Console](ui-overview.html#db-console-access). The following page is displayed: - -DB Console - -### Step 2. Set the Enterprise license and refresh the DB Console - -After [setting the Enterprise license](enterprise-licensing.html), the Node Map should now be displaying the highest-level localities you defined: - -DB Console - -{{site.data.alerts.callout_info}} -To be displayed on the world map, localities must be assigned a corresponding latitude and longitude. -{{site.data.alerts.end}} - -### Step 3. Set the latitudes and longitudes for the localities - -Launch the built-in SQL client: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --insecure --host=
-~~~ - -Insert the approximate latitude and longitude of each region into the `system.locations` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO system.locations VALUES - ('region', 'us-east-1', 37.478397, -76.453077), - ('region', 'us-west-1', 38.837522, -120.895824), - ('region', 'eu-west-1', 53.142367, -7.692054); -~~~ - -For the latitudes and longitudes of AWS, Azure, and Google Cloud regions, see [Location Coordinates for Reference](#location-coordinates). - -### Step 4. Refresh the Node Map - -Refresh the DB Console to see the updated Node Map: - -DB Console - -### Step 5. Navigate the Node Map - -Let's say you want to navigate to Node 2, which is in datacenter `us-east-1a` in the `us-east-1` region: - -1. Click on the map component marked as **region=us-east-1** on the Node Map. The [locality component](ui-cluster-overview-page.html#locality-component) for the datacenter is displayed. - - DB Console - -1. Click on the datacenter component marked as **datacenter=us-east-1a**. The individual [node components](ui-cluster-overview-page.html#node-component) are displayed. - - DB Console - -1. To navigate back to the cluster view, either click on **Cluster** in the breadcrumb trail at the top of the Node Map, or click **Up to REGION=US-EAST-1** and then click **Up to CLUSTER** in the lower left-hand side of the Node Map. - -## Troubleshoot the Node Map - -### Node Map not displayed - -The Node Map requires an [Enterprise license](enterprise-licensing.html). - -All nodes in the cluster must be assigned [localities](cockroach-start.html#locality). To be displayed on the world map, localities must be [assigned a corresponding latitude and longitude](#step-3-set-the-latitudes-and-longitudes-for-the-localities). - -To verify both of the above, navigate to the Localities debug page (`https://
:8080/#/reports/localities`) in the DB Console. - -DB Console - -The Localities debug page displays the following: - -- Localities configuration that you set up while starting the nodes with the `--locality` flags. -- Nodes corresponding to each locality. -- Latitude and longitude coordinates for each locality/node. - -### World Map not displayed for all locality levels - -The world map is displayed only when [localities are assigned latitude/longitude coordinates](#step-3-set-the-latitudes-and-longitudes-for-the-localities). - -If a locality (e.g., region) is not assigned latitude/longitude coordinates, it is displayed using the latitude/longitude of any lower-level localities it contains (e.g., datacenter). If no coordinates are available, localities are plotted in a circular layout. - -### Displayed Used Capacity value is more than configured capacity - -{% include {{ page.version.version }}/misc/available-capacity-metric.md %} - -## Location coordinates - -### AWS locations - -{% include {{ page.version.version }}/misc/aws-locations.md %} - -### Azure locations - -{% include {{ page.version.version }}/misc/azure-locations.md %} - -### Google Cloud locations - -{% include {{ page.version.version }}/misc/gce-locations.md %} diff --git a/src/current/v21.1/encryption.md b/src/current/v21.1/encryption.md deleted file mode 100644 index 54ebb4b609e..00000000000 --- a/src/current/v21.1/encryption.md +++ /dev/null @@ -1,226 +0,0 @@ ---- -title: Encryption -summary: Learn about the encryption features for secure CockroachDB clusters. -toc: true ---- - -Data encryption and decryption is the process of transforming plaintext data to cipher-text and vice versa using a key or password. - -## Encryption in flight - -CockroachDB uses TLS 1.2 for inter-node and client-node [authentication](authentication.html) as well as setting up a secure communication channel. Once the secure channel is set up, all inter-node and client-node network communication is encrypted using a [shared encryption key](https://en.wikipedia.org/wiki/Transport_Layer_Security) as per the TLS 1.2 protocol. This feature is enabled by default for all secure clusters and needs no additional configuration. - -## Encryption at Rest (Enterprise) - -Encryption at Rest provides transparent encryption of a node's data on the local disk. It allows encryption of all files on disk using [AES](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) in [counter mode](https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Counter_(CTR)), with all key -sizes allowed. - -Encryption is performed in the [storage layer](architecture/storage-layer.html) and configured per store. -All files used by the store, regardless of contents, are encrypted with the desired algorithm. - -To allow arbitrary rotation schedules and ensure security of the keys, we use two layers of keys: - -- **Store keys** are provided by the user in a file. They are used to encrypt the list of data keys (see below). This is known as a **key encryption key**: its only purpose is to encrypt other keys. Store keys are never persisted by CockroachDB. Since very little data is encrypted using this key, it can have a very long lifetime without risk of reuse. - -- **Data keys** are automatically generated by CockroachDB. They are used to encrypt all files on disk. This is known as a **data encryption key**. Data keys are persisted in a key registry file, encrypted using the store key. The key has a short lifetime to avoid reuse. - -Store keys are specified at node startup by passing a path to a locally readable file. The file must contain 32 bytes (the key ID) followed by the key (16, 24, or 32 bytes). The size of the key dictates the version of AES to use (AES-128, AES-192, or AES-256). For an example showing how to create a store key, see [Generating key files](#generating-store-key-files) below. - -Also during node startup, CockroachDB uses a data key with the same length as the store key. If encryption has just been enabled, -the key size has changed, or the data key is too old (default lifetime is one week), CockroachDB generates a new data key. - -Any new file created by the store uses the currently-active data key. All data keys (both active and previous) are stored in a key registry file and encrypted with the active store key. - -After startup, if the active data key is too old, CockroachDB generates a new data key and marks it as active, using it for all further encryption. - -CockroachDB does not currently force re-encryption of older files but instead relies on normal [storage engine](architecture/storage-layer.html) churn to slowly rewrite all data with the desired encryption. - -### Rotating keys - -Key rotation is necessary for Encryption at Rest for multiple reasons: - -- To prevent key reuse with the same encryption parameters (after encrypting many files). -- To reduce the risk of key exposure. - -Store keys are specified by the user and must be rotated by specifying different keys. This is done by restarting each node and setting the `key` parameter of the `--enterprise-encryption` flag to the path to the new key, and `old-key` to the previously used key. For an example, see [Changing encryption algorithm or keys](#changing-encryption-algorithm-or-keys). - -Data keys will automatically be rotated at startup if any of the following conditions are met: - -- The active store key has changed. -- The encryption type has changed (different key size, or plaintext to/from encryption). -- The current data key is `rotation-period` old or more. - -Data keys will automatically be rotated at runtime if the current data key is `rotation-period` old or more. - -Once rotated, an old store key cannot be made the active key again. - -Upon store key rotation the data keys registry is decrypted using the old key and encrypted with the new -key. The newly-generated data key is used to encrypt all new data from this point on. - -### Changing encryption type - -The user can change the encryption type from plaintext to encryption, between different encryption algorithms (using various key sizes), or from encryption to plaintext. - -When changing the encryption type to plaintext, the data key registry is no longer encrypted and all previous data keys are readable by anyone. All data on the store is effectively readable. - -When changing from plaintext to encryption, it will take some time for all data to eventually be re-written and encrypted. - -### Recommendations - -There are a number of considerations to keep in mind when running with encryption. - -#### Deployment configuration - -To prevent key leakage, production deployments should: - -* Use encrypted swap, or disable swap entirely. -* Disable core files. - -CockroachDB attempts to disable core files at startup when encryption is requested, but it may fail. - -#### Key handling - -Key management is the most dangerous aspect of encryption. The following rules should be kept in mind: - -* Make sure that only the UNIX user running the `cockroach` process has access to the keys. -* Do not store the keys on the same partition/drive as the CockroachDB data. It is best to load keys at run time from a separate system (e.g., [Keywhiz](https://square.github.io/keywhiz/), Vault). -* Rotate store keys frequently (every few weeks to months). -* Keep the data key rotation period low (default is one week). - -#### Other recommendations - -A few other recommendations apply for best security practices: - -* Do not switch from encrypted to plaintext, this leaks data keys. When plaintext is selected, all previously encrypted data must be considered reachable. -* Do not copy the encrypted files, as the data keys are not easily available. -* If encryption is desired, start a node with it enabled from the first run, without ever running in plaintext. - -{{site.data.alerts.callout_danger}} -Note that backups taken with the [`BACKUP`](backup.html) statement **are not encrypted** even if Encryption at Rest is enabled. Encryption at Rest only applies to the CockroachDB node's data on the local disk. If you want encrypted backups, you will need to encrypt your backup files using your preferred encryption method. -{{site.data.alerts.end}} - -### Examples - -#### Generating store key files - -Cockroach determines which encryption algorithm to use based on the size of the key file. -The key file must contain random data making up the key ID (32 bytes) and the actual key (16, 24, or 32 -bytes depending on the encryption algorithm). - -| Algorithm | Key size | Key file size | -|-|-|-| -| AES-128 | 128 bits (16 bytes) | 48 bytes | -| AES-192 | 192 bits (24 bytes) | 56 bytes | -| AES-256 | 256 bits (32 bytes) | 64 bytes | - -Generating a key file can be done using the `cockroach` CLI: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach gen encryption-key -s 128 /path/to/my/aes-128.key -~~~ - -Or the equivalent [openssl](https://www.openssl.org/docs/man1.1.1/man1/openssl.html) CLI command: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ openssl rand -out /path/to/my/aes-128.key 48 -~~~ - -#### Starting a node with encryption - -Encryption at Rest is configured at node start time using the `--enterprise-encryption` command line flag. -The flag specifies the encryption options for one of the stores on the node. If multiple stores exist, -the flag must be specified for each store. - -The flag takes the form: `--enterprise-encryption=path=,key=,old-key=,rotation-period=`. - -The allowed components in the flag are: - -| Component | Requirement | Description | -|-|-|-| -| `path` | Required | Path of the store to apply encryption to. | -| `key` | Required | Path to the key file to encrypt data with, or `plain` for plaintext. | -| `old-key` | Required | Path to the key file the data is encrypted with, or `plain` for plaintext. | -| `rotation-period` | Optional | How often data keys should be automatically rotated. Default: one week. | - -The `key` and `old-key` components must **always** be specified. They allow for transitions between -encryption algorithms, and between plaintext and encrypted. - -Starting a node for the first time using AES-128 encryption can be done using: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start --store=cockroach-data --enterprise-encryption=path=cockroach-data,key=/path/to/my/aes-128.key,old-key=plain -~~~ - -{{site.data.alerts.callout_danger}} -Once specified for a given store, the `--enterprise-encryption` flag must always be present. -{{site.data.alerts.end}} - -#### Checking encryption status - -Encryption status can be seen on the node's stores report, reachable through: `http(s)://nodeaddress:8080/#/reports/stores/local` (or replace `local` with the node ID). For example, if you are running a [local cluster](secure-a-cluster.html), you can see the node's stores report at `https://localhost:8080/#/reports/stores/local`. - -The report shows encryption status for all stores on the selected node, including: - -* Encryption algorithm. -* Active store key information. -* Active data key information. -* The fraction of files/bytes encrypted using the active data key. - -CockroachDB relies on [storage layer](architecture/storage-layer.html) compactions to write new files using the latest encryption key. It may take several days for all files to be replaced. Some files are only rewritten at startup, and some keep older copies around, requiring multiple restarts. You can force storage compaction with the `cockroach debug compact` command (the node must first be [stopped](cockroach-quit.html)). - -Information about keys is written to [the logs](logging-overview.html), including: - -* Active/old key information at startup. -* New key information after data key rotation. - -Alternatively, you can use the [`cockroach debug encryption-active-key`](cockroach-debug-encryption-active-key.html) command to view information about a store's encryption algorithm and store key. - -#### Changing encryption algorithm or keys - -Encryption type and keys can be changed at any time by restarting the node. -To change keys or encryption type, the `key` component of the `--enterprise-encryption` flag is set to the new key, -while the key previously used must be specified in the `old-key` component. - -For example, we can switch from AES-128 to AES-256 using: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start --store=cockroach-data --enterprise-encryption=path=cockroach-data,key=/path/to/my/aes-256.key,old-key=/path/to/my/aes-128.key -~~~ - -Upon starting, the node will read the existing data keys using the old encryption key (`aes-128.key`), then rewrite -the data keys using the new key (`aes-256.key`). A new data key will be generated to match the desired AES-256 algorithm. - -To check that the new key is active, use the stores report page in the DB Console to [check the encryption status](#checking-encryption-status). - -To disable encryption, specify `key=plain`. The data keys will be stored in plaintext and new data will not be encrypted. - -To rotate keys, specify `key=/path/to/my/new-aes-128.key` and `old-key=/path/to/my/old-aes-128.key`. The data keys -will be decrypted using the old key and then encrypted using the new key. A new data key will also be generated. - -## Encrypted backups (Enterprise) - -{% include {{ page.version.version }}/backups/encrypted-backup-description.md %} - -## Encryption caveats - -### Higher CPU utilization - -Enabling Encryption at Rest might result in a higher CPU utilization. We estimate a 5-10% increase in CPU utilization. - -### Encryption for touchpoints with other services - -- S3 backup encryption -- Encrypted comms with Kafka - - -## See also - -- [Client Connection Parameters](connection-parameters.html) -- [Manual Deployment](manual-deployment.html) -- [Orchestrated Deployment](orchestration.html) -- [Local Deployment](secure-a-cluster.html) -- [Other Cockroach Commands](cockroach-commands.html) diff --git a/src/current/v21.1/enterprise-licensing.md b/src/current/v21.1/enterprise-licensing.md deleted file mode 100644 index 5f3b0742216..00000000000 --- a/src/current/v21.1/enterprise-licensing.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -title: Enterprise Features -summary: Learn about CockroachDB features that require an Enterprise license key. -toc: true ---- - -CockroachDB distributes a single binary that contains both core and Enterprise features. You can use core features without any license key. However, to use the Enterprise features, you need either a trial or an Enterprise license key. - -This page lists Enterprise features. For information on how to obtain and set trial and Enterprise license keys for CockroachDB, see the [Licensing FAQs](licensing-faqs.html#obtain-a-license). - -{% include {{ page.version.version }}/misc/enterprise-features.md %} - -## See also - -- [`SET CLUSTER SETTING`](set-cluster-setting.html) -- [`SHOW CLUSTER SETTING`](show-cluster-setting.html) -- [Enterprise Trial –– Get Started](get-started-with-enterprise-trial.html) diff --git a/src/current/v21.1/enum.md b/src/current/v21.1/enum.md deleted file mode 100644 index f71a7c722e6..00000000000 --- a/src/current/v21.1/enum.md +++ /dev/null @@ -1,218 +0,0 @@ ---- -title: ENUM -summary: CockroachDB's ENUM data types comprise a set of values. -toc: true ---- - -A user-defined `ENUM` [data type](data-types.html) consists of a set of enumerated, static values. - -## Syntax - -To declare a new `ENUM` data type, use [`CREATE TYPE`](create-type.html): - -~~~ sql -> CREATE TYPE AS ENUM ('', '', ...); -~~~ - -where `` is the name of the new type, and `, , ...` are string literals that make up the type's set of static values. - -{{site.data.alerts.callout_info}} -You can qualify the `` of an enumerated type with a [database and schema name](sql-name-resolution.html) (e.g., `db.typename`). After the type is created, it can only be referenced from the database that contains the type. -{{site.data.alerts.end}} - -To show all `ENUM` types in the database, including all `ENUMS` created implicitly for [multi-region databases](movr-flask-overview.html), use [`SHOW ENUMS`](show-enums.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW ENUMS; -~~~ - -To modify an `ENUM` type, use [`ALTER TYPE`](alter-type.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TYPE ADD VALUE ''; -~~~ - -where `` is a string literal to add to the existing list of type values. You can also use `ALTER TYPE` to rename types, rename type values, set a type's schema, or change the type owner's [role specification](grant.html). - -To drop the type, use [`DROP TYPE`](drop-type.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP TYPE ; -~~~ - -## Required privileges - -- To [create a type](create-type.html) in a database, a user must have the `CREATE` [privilege](authorization.html#assign-privileges) on the database. -- To [drop a type](drop-type.html), a user must be the owner of the type. -- To [alter a type](alter-type.html), a user must be the owner of the type. -- To [grant privileges](grant.html) on a type, a user must have the `GRANT` privilege and the privilege that they want to grant. -- To create an object that depends on a type, a user must have the `USAGE` privilege on the type. - -## Example - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TYPE status AS ENUM ('open', 'closed', 'inactive'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW ENUMS; -~~~ - -~~~ - schema | name | value ----------+--------+----------------------- - public | status | open|closed|inactive -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE accounts ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - balance DECIMAL, - status status -); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO accounts(balance,status) VALUES (500.50,'open'), (0.00,'closed'), (1.25,'inactive'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM accounts; -~~~ - -~~~ - id | balance | status ----------------------------------------+---------+----------- - 3848e36d-ebd4-44c6-8925-8bf24bba957e | 500.50 | open - 60928059-ef75-47b1-81e3-25ec1fb6ff10 | 0.00 | closed - 71ae151d-99c3-4505-8e33-9cda15fce302 | 1.25 | inactive -(3 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE accounts; -~~~ - -~~~ - table_name | create_statement --------------+-------------------------------------------------- - accounts | CREATE TABLE public.accounts ( - | id UUID NOT NULL DEFAULT gen_random_uuid(), - | balance DECIMAL NULL, - | status public.status NULL, - | CONSTRAINT "primary" PRIMARY KEY (id ASC), - | FAMILY "primary" (id, balance, status) - | ) -(1 row) -~~~ - - -## Supported casting and conversion - -`ENUM` data type values can be [cast](data-types.html#data-type-conversions-and-casts) to [`STRING`s](string.html). - -Values can be cast explicitly or implicitly. For example, the following [`SELECT`](select-clause.html) statements are equivalent: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM accounts WHERE status::STRING='open'; -~~~ - -~~~ - id | balance | status ----------------------------------------+---------+--------- - 3848e36d-ebd4-44c6-8925-8bf24bba957e | 500.50 | open -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM accounts WHERE status='open'; -~~~ - -~~~ - id | balance | status ----------------------------------------+---------+--------- - 3848e36d-ebd4-44c6-8925-8bf24bba957e | 500.50 | open -(1 row) -~~~ - -### Comparing enumerated types - -To compare two enumerated types, you must explicitly cast both types as `STRING`s. For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TYPE inaccessible AS ENUM ('closed', 'inactive'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE notifications ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - status inaccessible, - message STRING -); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO notifications(status, message) VALUES ('closed', 'This account has been closed.'),('inactive', 'This account is on hold.'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT - accounts.id, notifications.message - FROM accounts JOIN notifications ON accounts.status = notifications.status; -~~~ - -~~~ -ERROR: unsupported comparison operator: = -SQLSTATE: 22023 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT - accounts.id, notifications.message - FROM accounts JOIN notifications ON accounts.status::STRING = notifications.status; -~~~ - -~~~ -ERROR: unsupported comparison operator: = -SQLSTATE: 22023 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT - accounts.id, notifications.message - FROM accounts JOIN notifications ON accounts.status::STRING = notifications.status::STRING; -~~~ - -~~~ - id | message ----------------------------------------+-------------------------------- - 285336c4-ca1f-490d-b0df-146aae94f5aa | This account is on hold. - 583157d5-4f34-43e5-a4d4-51db77feb391 | This account has been closed. -(2 rows) -~~~ - -## See also - -- [Data Types](data-types.html) -- [`CREATE TYPE`](create-type.html) -- [`ALTER TYPE`](alter-type.html) -- [`SHOW ENUMS`](show-enums.html) -- [`DROP TYPE`](drop-type.html) diff --git a/src/current/v21.1/error-handling-and-troubleshooting.md b/src/current/v21.1/error-handling-and-troubleshooting.md deleted file mode 100644 index cad4ac46444..00000000000 --- a/src/current/v21.1/error-handling-and-troubleshooting.md +++ /dev/null @@ -1,98 +0,0 @@ ---- -title: Error Handling and Troubleshooting -summary: How to troubleshoot problems and handle transaction retry errors during application development -toc: true ---- - -This page has instructions for handling errors and troubleshooting problems that may arise during application development. - -## Troubleshooting query problems - -If you are not satisfied with your SQL query performance, follow the instructions in [Optimize Statement Performance][fast] to be sure you are avoiding common performance problems like full table scans, missing indexes, etc. - -If you have already optimized your SQL queries as described in [Optimize Statement Performance][fast] and are still having issues such as: - -- Hanging or "stuck" queries -- Queries that are slow some of the time (but not always) -- Low throughput of queries - -Take a look at [Troubleshoot SQL Behavior](query-behavior-troubleshooting.html). - -{{site.data.alerts.callout_info}} -If you aren't sure whether SQL query performance needs to be improved on your cluster, see [Identify slow queries](query-behavior-troubleshooting.html#identify-slow-statements). -{{site.data.alerts.end}} - -## Transaction retry errors - -Messages with the Postgres error code `40001` indicate that a transaction failed because it [conflicted with another concurrent or recent transaction accessing the same data](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention). The transaction needs to be retried by the client. - -If your language's client driver or ORM implements transaction retry logic internally (e.g., if you are using Python and [SQLAlchemy with the CockroachDB dialect](build-a-python-app-with-cockroachdb-sqlalchemy.html)), then you do not need to handle this logic from your application. - -If your driver or ORM does not implement this logic, then you will need to implement a retry loop in your application. - -{% include {{page.version.version}}/misc/client-side-intervention-example.md %} - -{{site.data.alerts.callout_info}} -If a consistently high percentage of your transactions are resulting in transaction retry errors, then you may need to evaluate your schema design and data access patterns to find and remove sources of contention. For more information about contention, see [Understanding and Avoiding Transaction Contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention). - -For more information about what is causing a specific transaction retry error code, see the [Transaction Retry Error Reference](transaction-retry-error-reference.html). -{{site.data.alerts.end}} - -For more information about transaction retry errors, see [Transaction retries](transactions.html#client-side-intervention). - -## Unsupported SQL features - -CockroachDB has support for [most SQL features](sql-feature-support.html). - -Additionally, CockroachDB supports [the PostgreSQL wire protocol and the majority of its syntax](postgresql-compatibility.html). This means that existing applications can often be migrated to CockroachDB without changing application code. - -However, you may encounter features of SQL or the Postgres dialect that are not supported by CockroachDB. For example, the following Postgres features are not supported: - -{% include {{page.version.version}}/sql/unsupported-postgres-features.md %} - -For more information about the differences between CockroachDB and Postgres feature support, see [PostgreSQL Compatibility](postgresql-compatibility.html). - -For more information about the SQL standard features supported by CockroachDB, see [SQL Feature Support](sql-feature-support.html) - -## Troubleshooting cluster problems - -As a developer, you will mostly be working with the CockroachDB [SQL API](sql-statements.html). - -However, you may need to access the underlying cluster to troubleshoot issues where the root cause is not your SQL, but something happening at the cluster level. Symptoms of cluster-level issues can include: - -- Cannot join a node to an existing cluster -- Networking, client connection, or authentication issues -- Clock sync, replication, or node liveness issues -- Capacity planning, storage, or memory issues -- Node decommissioning failures - -For more information about how to troubleshoot cluster-level issues, see [Troubleshoot Cluster Setup](cluster-setup-troubleshooting.html). - -## See also - -Reference information related to this page: - -- [Troubleshoot Query Behavior](query-behavior-troubleshooting.html) -- [Troubleshoot Cluster Setup](cluster-setup-troubleshooting.html) -- [Common errors](common-errors.html) -- [Transactions](transactions.html) -- [Transaction retries](transactions.html#client-side-intervention) -- [Understanding and Avoiding Transaction Contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention) -- [SQL Layer][sql] - -Other common tasks: - -- [Connect to the Database](connect-to-the-database.html) -- [Insert Data](insert-data.html) -- [Query Data](query-data.html) -- [Update Data](update-data.html) -- [Delete Data](delete-data.html) -- [Run Multi-Statement Transactions](run-multi-statement-transactions.html) -- [Identify slow queries](query-behavior-troubleshooting.html#identify-slow-statements) -- [Optimize Statement Performance][fast] -- [Hello World Example apps](example-apps.html) - - - -[sql]: architecture/sql-layer.html -[fast]: make-queries-fast.html diff --git a/src/current/v21.1/eventlog.md b/src/current/v21.1/eventlog.md deleted file mode 100644 index ad6f007ee1f..00000000000 --- a/src/current/v21.1/eventlog.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: Notable Event Types -summary: Reference documentation for notable event types in logs. -toc: true ---- - -{% remote_include https://raw.githubusercontent.com/cockroachdb/cockroach/{{ page.release_info.crdb_branch_name }}/docs/generated/eventlog.md %} diff --git a/src/current/v21.1/example-apps.md b/src/current/v21.1/example-apps.md deleted file mode 100644 index 23228337ca5..00000000000 --- a/src/current/v21.1/example-apps.md +++ /dev/null @@ -1,98 +0,0 @@ ---- -title: Example Apps -summary: Examples that show you how to build simple applications with CockroachDB -tags: golang, python, java -toc: true -key: build-an-app-with-cockroachdb.html ---- - -The examples in this section show you how to build simple applications using CockroachDB. - -Click the links in the tables below to see simple but complete example applications for each supported language and library combination. - -If you are looking to do a specific task such as connect to the database, insert data, or run multi-statement transactions, see [this list of tasks](#tasks). - -{{site.data.alerts.callout_info}} -Applications may encounter incompatibilities when using advanced or obscure features of a driver or ORM with **beta-level** support. If you encounter problems, please [open an issue](https://github.com/cockroachdb/cockroach/issues/new) with details to help us make progress toward full support. - -Note that tools with [**community-level** support](community-tooling.html) have been tested or developed by the CockroachDB community, but are not officially supported by Cockroach Labs. If you encounter problems with using these tools, please contact the maintainer of the tool with details. -{{site.data.alerts.end}} - -## JavaScript/TypeScript - -| Driver/ORM Framework | Support level | Example apps | -|---------------------------------------------------------+----------------+--------------------------------------------------------| -| [node-postgres](https://www.npmjs.com/package/pg) | Full | [Simple CRUD](build-a-nodejs-app-with-cockroachdb.html) -| [Sequelize](https://www.npmjs.com/package/sequelize) | Full | [Simple CRUD](build-a-nodejs-app-with-cockroachdb-sequelize.html) -| [Knex.js](https://knexjs.org/) | Full | [Simple CRUD](build-a-nodejs-app-with-cockroachdb-knexjs.html) -| [Prisma](https://prisma.io) | Full | [Simple CRUD](build-a-nodejs-app-with-cockroachdb-prisma.html) -| [TypeORM](https://www.npmjs.com/package/typeorm) | Full | [Simple CRUD](build-a-typescript-app-with-cockroachdb.html) - -## Python - -| Driver/ORM Framework | Support level | Example apps | -|-----------------------------------------------------------------+----------------+--------------------------------------------------------| -| [psycopg2](https://www.psycopg.org/docs/install.html) | Full | [Simple CRUD](build-a-python-app-with-cockroachdb.html)
[AWS Lambda](deploy-lambda-function.html) -| [SQLAlchemy](https://www.sqlalchemy.org/) | Full | [Simple CRUD](build-a-python-app-with-cockroachdb-sqlalchemy.html)
[MovR-Flask (Global Web App)](movr-flask-overview.html) -| [Django](https://pypi.org/project/Django/) | Full | [Simple CRUD](build-a-python-app-with-cockroachdb-django.html) -| [PonyORM](https://ponyorm.org/) | Full | [Simple CRUD](build-a-python-app-with-cockroachdb-pony.html) - -## Go - -| Driver/ORM Framework | Support level | Example apps | -|--------------------------------------------------+----------------+--------------------------------------------------------| -| [pgx](https://github.com/jackc/pgx/releases) | Full | [Simple CRUD](build-a-go-app-with-cockroachdb.html) -| [GORM](https://github.com/jinzhu/gorm/releases) | Full | [Simple CRUD](build-a-go-app-with-cockroachdb-gorm.html) -| [pq](https://github.com/lib/pq) | Full | [Simple CRUD](build-a-go-app-with-cockroachdb-pq.html) -| [upper/db](https://github.com/upper/db) | Full | [Simple CRUD](build-a-go-app-with-cockroachdb-upperdb.html) - -## Java - -| Driver/ORM Framework | Support level | Example apps | -|--------------------------------------------+----------------+--------------------------------------------------------| -| [JDBC](https://jdbc.postgresql.org/) | Full | [Simple CRUD](build-a-java-app-with-cockroachdb.html)
[Roach Data (Spring Boot App)](build-a-spring-app-with-cockroachdb-jdbc.html) -| [Hibernate](https://hibernate.org/orm/) | Full | [Simple CRUD](build-a-java-app-with-cockroachdb-hibernate.html)
[Roach Data (Spring Boot App)](build-a-spring-app-with-cockroachdb-jpa.html) -| [jOOQ](https://www.jooq.org/) | Full | [Simple CRUD](build-a-java-app-with-cockroachdb-jooq.html) - -## Ruby - -| Driver/ORM Framework | Support level | Example apps | -|-----------------------------------------------------------+----------------+--------------------------------------------------------| -| [pg](https://rubygems.org/gems/pg) | Full | [Simple CRUD](build-a-ruby-app-with-cockroachdb.html) -| [ActiveRecord](https://rubygems.org/gems/activerecord) | Full | [Simple CRUD](build-a-ruby-app-with-cockroachdb-activerecord.html) - -## C# - -| Driver/ORM Framework | Support level | Example apps | -|-----------------------------------------------------------+----------------+--------------------------------------------------------| -| [Npgsql](https://www.npgsql.org/) | Beta | [Simple CRUD](build-a-csharp-app-with-cockroachdb.html) - -## Rust - -| Driver/ORM Framework | Support level | Example apps | -|------------------------------------------------+----------------+--------------------------------------------------------| -| [Rust-Postgres](https://github.com/sfackler/rust-postgres) | Beta | [Simple CRUD](build-a-rust-app-with-cockroachdb.html) - -## See also - -Reference information: - -- [Client drivers](install-client-drivers.html) -- [Third-Party Tools Supported by Cockroach Labs](third-party-database-tools.html) -- [Third-Party Tools Supported by the Community](community-tooling.html) -- [Connection parameters](connection-parameters.html) -- [Transactions](transactions.html) -- [Performance best practices](performance-best-practices-overview.html) - - - -Specific tasks: - -- [Connect to the Database](connect-to-the-database.html) -- [Insert Data](insert-data.html) -- [Query Data](query-data.html) -- [Update Data](update-data.html) -- [Delete Data](delete-data.html) -- [Optimize Statement Performance](make-queries-fast.html) -- [Run Multi-Statement Transactions](run-multi-statement-transactions.html) -- [Error Handling and Troubleshooting](error-handling-and-troubleshooting.html) diff --git a/src/current/v21.1/experimental-audit.md b/src/current/v21.1/experimental-audit.md deleted file mode 100644 index 4a837108560..00000000000 --- a/src/current/v21.1/experimental-audit.md +++ /dev/null @@ -1,125 +0,0 @@ ---- -title: EXPERIMENTAL_AUDIT -summary: Use the EXPERIMENTAL_AUDIT subcommand to turn SQL audit logging on or off for a table. -toc: true ---- - -`EXPERIMENTAL_AUDIT` is a subcommand of [`ALTER TABLE`](alter-table.html). When applied to a table, it enables or disables the recording of SQL audit events to the [`SENSITIVE_ACCESS`](logging.html#sensitive_access) logging channel for that table. - -{{site.data.alerts.callout_info}} -The `SENSITIVE_ACCESS` log output is also called the SQL audit log. See [SQL Audit Logging](sql-audit-logging.html) for a detailed example. -{{site.data.alerts.end}} - -SQL audit logs contain detailed information about queries being executed against your system, including: - -- Full text of the query (which may include personally identifiable information (PII)) -- Date/Time -- Client address -- Application name - -{{site.data.alerts.callout_success}} -For descriptions of all SQL audit event types and their fields, see [Notable Event Types](eventlog.html#sql-access-audit-events). -{{site.data.alerts.end}} - -CockroachDB stores audit log information in a way that ensures durability, but negatively impacts performance. As a result, we recommend using SQL audit logs for security purposes only. For more information, see [Performance considerations](#performance-considerations). - -{% include {{ page.version.version }}/misc/experimental-warning.md %} - -{% include {{ page.version.version }}/sql/combine-alter-table-commands.md %} - -## Synopsis - -
-{% include {{ page.version.version }}/sql/generated/diagrams/experimental_audit.html %} -
- -## Required privileges - -Only members of the `admin` role can enable audit logs on a table. By default, the `root` user belongs to the `admin` role. - -## Parameters - - Parameter | Description ---------------+---------------------------------------------------------- - `table_name` | The name of the table you want to create audit logs for. - `READ` | Log all table reads to the audit log file. - `WRITE` | Log all table writes to the audit log file. - `OFF` | Turn off audit logging. - -{{site.data.alerts.callout_info}} -This command logs all reads and writes, and both the READ and WRITE parameters are required (as shown in the examples below). Logging for only reads or only writes is not supported. -{{site.data.alerts.end}} - -## Audit log file format - -Audit log messages, like all [log messages](logging-overview.html), consist of two sections: - -- A payload that contains notable events structured in JSON. These can include information such as the application name, full text of the query (which may contain PII), user account that triggered the event, number of rows produced (e.g., for `SELECT`) or processed (e.g., for `INSERT` or `UPDATE`), status of the query, and more. For more information on the possible event types logged to the `SENSITIVE_ACCESS` channel, see [Notable Event Types](eventlog.html#sql-access-audit-events). -- An envelope that contains event metadata (e.g., severity, date, timestamp, channel). Depending on the log format you specify when [configuring logs](configure-logs.html), the envelope can be formatted either as JSON or as a flat prefix to the message. - -## Audit log file storage location - -By [default](configure-logs.html#default-logging-configuration), audit logs are prefixed `cockroach-sql-audit` and are stored in the [same directory](configure-logs.html#logging-directory) as the other logs generated by CockroachDB. - -To store the audit log files in a specific directory, [configure the `SENSITIVE_ACCESS` channel](configure-logs.html#output-to-files) with a custom `dir` path. - -{{site.data.alerts.callout_success}} -If your deployment requires particular lifecycle and access policies for audit log files, point `SENSITIVE_ACCESS` to a directory that has permissions set so that only CockroachDB can create/delete files. -{{site.data.alerts.end}} - -## Viewing schema changes - -{% include {{ page.version.version }}/misc/schema-change-view-job.md %} - -## Performance considerations - -To ensure [non-repudiation](https://en.wikipedia.org/wiki/Non-repudiation) in audit logs, we recommend [enabling `auditable`](configure-logs.html#configure-log-sinks) for the `SENSITIVE_ACCESS` channel. CockroachDB will then synchronously log all of the activity of every user on a cluster in a way that is durable to system failures. Note that every query that causes a logging event must access the disk of the node on which audit logging is enabled. As a result, enabling `auditable` on a logging channel negatively impacts performance, and we recommend using this setting for security purposes only. - -For debugging and troubleshooting on production clusters, the most performant way to log all queries is to enable the `SQL_EXEC` logging channel. For details, see [Logging Use Cases](logging-use-cases.html#sql_exec). - -## Examples - -### Turn on audit logging - -Let's say you have a `customers` table that contains personally identifiable information (PII). To turn on audit logs for that table, run the following command: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER TABLE customers EXPERIMENTAL_AUDIT SET READ WRITE; -~~~ - -Now, every access of customer data is logged to the `SENSITIVE_ACCESS` channel in a [`sensitive_table_access`](eventlog.html#sensitive_table_access) event that looks like the following: - -~~~ -I210323 18:50:10.951550 1182 8@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:49851›,hostnossl,user=root] 4 ={"Timestamp":1616525410949087000,"EventType":"sensitive_table_access","Statement":"‹SELECT * FROM \"\".\"\".customers›","User":"‹root›","DescriptorID":52,"ApplicationName":"‹$ cockroach sql›","ExecMode":"exec","NumRows":2,"Age":2.514,"FullTableScan":true,"TxnCounter":38,"TableName":"‹defaultdb.public.customers›","AccessMode":"r"} -~~~ - -{{site.data.alerts.callout_info}} -The above example shows the default [`crdb-v2`](log-formats.html#format-crdb-v2) log format. This can be changed to a different format (e.g., JSON). For details, see [Configure Logs](configure-logs.html#file-logging-format). -{{site.data.alerts.end}} - -{{site.data.alerts.callout_success}} -For descriptions of all SQL audit event types and their fields, see [Notable Event Types](eventlog.html#sql-access-audit-events). -{{site.data.alerts.end}} - -To turn on auditing for more than one table, issue a separate `ALTER` statement for each table. - -{{site.data.alerts.callout_success}} -For a more detailed example, see [SQL Audit Logging](sql-audit-logging.html). -{{site.data.alerts.end}} - -### Turn off audit logging - -To turn off logging, issue the following command: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER TABLE customers EXPERIMENTAL_AUDIT SET OFF; -~~~ - -## See also - -- [SQL Audit Logging](sql-audit-logging.html) -- [Logging Overview](logging-overview.html) -- [`ALTER TABLE`](alter-table.html) -- [`SHOW JOBS`](show-jobs.html) \ No newline at end of file diff --git a/src/current/v21.1/experimental-features.md b/src/current/v21.1/experimental-features.md deleted file mode 100644 index 1c9e5691a12..00000000000 --- a/src/current/v21.1/experimental-features.md +++ /dev/null @@ -1,176 +0,0 @@ ---- -title: Experimental Features -summary: Learn about the experimental features available in CockroachDB -toc: true ---- - -This page lists the experimental features that are available in CockroachDB {{ page.version.version }}. - -{{site.data.alerts.callout_danger}} -**This page describes experimental features.** Their interfaces and outputs are subject to change, and there may be bugs. -
-
-If you encounter a bug, please [file an issue](file-an-issue.html). -{{site.data.alerts.end}} - -## Session variables - -The table below lists the experimental session settings that are available. For a complete list of session variables, see [`SHOW` (session settings)](show-vars.html). - -| Variable | Default Value | Description | -|-------------------------------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `enable_experimental_alter_column_type_general` | `'false'` | If set to `'true'`, enables [column type altering](#alter-column-types) for general cases, with some limitations. | -| `experimental_enable_hash_sharded_indexes` | `'off'` | If set to `'on'`, enables [hash-sharded indexes](#hash-sharded-indexes) with `USING HASH`. | -| `experimental_enable_temp_tables` | `'off'` | If set to `'on'`, enables [temporary objects](#temporary-objects), including [temporary tables](temporary-tables.html), [temporary views](views.html#temporary-views), and [temporary sequences](create-sequence.html#temporary-sequences). | - -## SQL statements - -### Keep SQL audit logs - -Log all queries against a table to a file, for security purposes. For more information, see [`ALTER TABLE ... EXPERIMENTAL_AUDIT`](experimental-audit.html). - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE t EXPERIMENTAL_AUDIT SET READ WRITE; -~~~ - -### Relocate leases and replicas - -You have the following options for controlling lease and replica location: - -1. Relocate leases and replicas using `EXPERIMENTAL_RELOCATE` -2. Relocate just leases using `EXPERIMENTAL_RELOCATE LEASE` - -For example, to distribute leases and ranges for N primary keys across N stores in the cluster, run a statement with the following structure: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE t EXPERIMENTAL_RELOCATE SELECT ARRAY[, , ..., ], , , ..., ; -~~~ - -To relocate just the lease without moving the replicas, run a statement like the one shown below, which moves the lease for the range containing primary key 'foo' to store 1. - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE t EXPERIMENTAL_RELOCATE LEASE SELECT 1, 'foo'; -~~~ - -### Show table fingerprints - -Table fingerprints are used to compute an identification string of an entire table, for the purpose of gauging whether two tables have the same data. This is useful, for example, when restoring a table from backup. - -Example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW EXPERIMENTAL_FINGERPRINTS FROM TABLE t; -~~~ - -~~~ - index_name | fingerprint -------------+--------------------- - primary | 1999042440040364641 -(1 row) -~~~ - -### Turn on KV event tracing - -Use session tracing (via [`SHOW TRACE FOR SESSION`](show-trace.html)) to report the replicas of all KV events that occur during its execution. - -Example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET tracing = on; -> SELECT * from t; -> SET tracing = off; -> SHOW EXPERIMENTAL_REPLICA TRACE FOR SESSION; -~~~ - -~~~ - timestamp | node_id | store_id | replica_id -----------------------------------+---------+----------+------------ - 2018-10-18 15:50:13.345879+00:00 | 3 | 3 | 7 - 2018-10-18 15:50:20.628383+00:00 | 2 | 2 | 26 -~~~ - -### Check for constraint violations with `SCRUB` - -Checks the consistency of [`UNIQUE`](unique.html) indexes, [`CHECK`](check.html) constraints, and more. Partially implemented; see [cockroachdb/cockroach#10425](https://github.com/cockroachdb/cockroach/issues/10425) for details. - -{{site.data.alerts.callout_info}} -This example uses the `users` table from our open-source, fictional peer-to-peer vehicle-sharing application, [MovR](movr.html). -{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPERIMENTAL SCRUB table movr.users; -~~~ - -~~~ - job_uuid | error_type | database | table | primary_key | timestamp | repaired | details -----------+--------------------------+----------+-------+----------------------------------------------------------+---------------------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - | index_key_decoding_error | movr | users | ('boston','0009eeb5-d779-4bf8-b1bd-8566533b105c') | 2018-10-18 16:00:38.65916 | f | {"error_message": "key ordering did not match datum ordering. IndexDescriptor=ASC", "index_name": "primary", "row_data": {"address": "e'06484 Christine Villages\\nGrantport, TN 01572'", "city": "'boston'", "credit_card": "'4634253150884'", "id": "'0009eeb5-d779-4bf8-b1bd-8566533b105c'", "name": "'Jessica Webb'"}} - | index_key_decoding_error | movr | users | ('los angeles','0001252c-fc16-4006-b6dc-c6b1a0fd1f5b') | 2018-10-18 16:00:38.65916 | f | {"error_message": "key ordering did not match datum ordering. IndexDescriptor=ASC", "index_name": "primary", "row_data": {"address": "e'91309 Warner Springs\\nLake Danielmouth, PR 33400'", "city": "'los angeles'", "credit_card": "'3584736360686445'", "id": "'0001252c-fc16-4006-b6dc-c6b1a0fd1f5b'", "name": "'Rebecca Gibson'"}} - | index_key_decoding_error | movr | users | ('new york','000169a5-e337-4441-b664-dae63e682980') | 2018-10-18 16:00:38.65916 | f | {"error_message": "key ordering did not match datum ordering. IndexDescriptor=ASC", "index_name": "primary", "row_data": {"address": "e'0787 Christopher Highway Apt. 363\\nHamptonmouth, TX 91864-2620'", "city": "'new york'", "credit_card": "'4578562547256688'", "id": "'000169a5-e337-4441-b664-dae63e682980'", "name": "'Christopher Johnson'"}} - | index_key_decoding_error | movr | users | ('paris','00089fc4-e5b1-48f6-9f0b-409905f228c4') | 2018-10-18 16:00:38.65916 | f | {"error_message": "key ordering did not match datum ordering. IndexDescriptor=ASC", "index_name": "primary", "row_data": {"address": "e'46735 Martin Summit\\nMichaelview, OH 10906-5889'", "city": "'paris'", "credit_card": "'5102207609888778'", "id": "'00089fc4-e5b1-48f6-9f0b-409905f228c4'", "name": "'Nicole Fuller'"}} - | index_key_decoding_error | movr | users | ('rome','000209fc-69a1-4dd5-8053-3b5e5769876d') | 2018-10-18 16:00:38.65916 | f | {"error_message": "key ordering did not match datum ordering. IndexDescriptor=ASC", "index_name": "primary", "row_data": {"address": "e'473 Barrera Vista Apt. 890\\nYeseniaburgh, CO 78087'", "city": "'rome'", "credit_card": "'3534605564661093'", "id": "'000209fc-69a1-4dd5-8053-3b5e5769876d'", "name": "'Sheryl Shea'"}} - | index_key_decoding_error | movr | users | ('san francisco','00058767-1e83-4e18-999f-13b5a74d7225') | 2018-10-18 16:00:38.65916 | f | {"error_message": "key ordering did not match datum ordering. IndexDescriptor=ASC", "index_name": "primary", "row_data": {"address": "e'5664 Acevedo Drive Suite 829\\nHernandezview, MI 13516'", "city": "'san francisco'", "credit_card": "'376185496850202'", "id": "'00058767-1e83-4e18-999f-13b5a74d7225'", "name": "'Kevin Turner'"}} - | index_key_decoding_error | movr | users | ('seattle','0002e904-1256-4528-8b5f-abad16e695ff') | 2018-10-18 16:00:38.65916 | f | {"error_message": "key ordering did not match datum ordering. IndexDescriptor=ASC", "index_name": "primary", "row_data": {"address": "e'81499 Samuel Crescent Suite 631\\nLake Christopherborough, PR 50401'", "city": "'seattle'", "credit_card": "'38743493725890'", "id": "'0002e904-1256-4528-8b5f-abad16e695ff'", "name": "'Mark Williams'"}} - | index_key_decoding_error | movr | users | ('washington dc','00007caf-2014-4696-85b0-840e7d8b6db9') | 2018-10-18 16:00:38.65916 | f | {"error_message": "key ordering did not match datum ordering. IndexDescriptor=ASC", "index_name": "primary", "row_data": {"address": "e'4578 Holder Trafficway\\nReynoldsside, IL 23520-7418'", "city": "'washington dc'", "credit_card": "'30454993082943'", "id": "'00007caf-2014-4696-85b0-840e7d8b6db9'", "name": "'Marie Miller'"}} -(8 rows) -~~~ - -### Show range information for a specific row - -The [`SHOW RANGE ... FOR ROW`](show-range-for-row.html) statement shows information about a [range](architecture/overview.html#glossary) for a particular row of data. This information is useful for verifying how SQL data maps to underlying ranges, and where the replicas for a range are located. - -## Functions and Operators - -The table below lists the experimental SQL functions and operators available in CockroachDB. For more information, see each function's documentation at [Functions and Operators](functions-and-operators.html). - -| Function | Description | -|----------------------------------------------------------------------------------+-------------------------------------------------| -| [`experimental_strftime`](functions-and-operators.html#date-and-time-functions) | Format time using standard `strftime` notation. | -| [`experimental_strptime`](functions-and-operators.html#date-and-time-functions) | Format time using standard `strptime` notation. | -| [`experimental_uuid_v4()`](functions-and-operators.html#id-generation-functions) | Return a UUID. | - -## Alter column types - - CockroachDB supports altering the column types of existing tables, with certain limitations. For more information, see [Altering column data types](alter-column.html#altering-column-data-types). - -## Temporary objects - - Support for [temporary tables](temporary-tables.html), [temporary views](views.html#temporary-views), and [temporary sequences](create-sequence.html#temporary-sequences) is currently experimental in CockroachDB. If you create too many temporary objects in a session, the performance of DDL operations will degrade. Performance limitations could persist long after creating the temporary objects. For more details, see [cockroachdb/cockroach#46260](https://github.com/cockroachdb/cockroach/issues/46260). - -## Hash-sharded indexes - - CockroachDB supports hash-sharded indexes with the [`USING HASH`](create-index.html#parameters) keywords. Hash-sharded indexes distribute sequential traffic uniformly across ranges, eliminating single-range hotspots and improving write performance on sequentially-keyed indexes at a small cost to read performance. For more information, see [Hash-sharded indexes](hash-sharded-indexes.html). - -## Password authentication without TLS - - For deployments where transport security is already handled at the infrastructure level (e.g., IPSec with DMZ), and TLS-based transport security is not possible or not desirable, CockroachDB now supports delegating transport security to the infrastructure with the new experimental flag `--accept-sql-without-tls` for [`cockroach start`](cockroach-start.html#security). - - With this flag, SQL clients can establish a session over TCP without a TLS handshake. They still need to present valid authentication credentials, for example a password in the default configuration. Different authentication schemes can be further configured as per `server.host_based_authentication.configuration`. - - Example: - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --user=jpointsman --insecure - ~~~ - - ~~~ - # Welcome to the CockroachDB SQL shell. - # All statements must be terminated by a semicolon. - # To exit, type: \q. - # - Enter password: - ~~~ - -## See Also - -- [`SHOW` (session)](show-vars.html) -- [Functions and Operators](functions-and-operators.html) -- [`ALTER TABLE ... EXPERIMENTAL_AUDIT`](experimental-audit.html) -- [`SHOW TRACE FOR SESSION`](show-trace.html) -- [`SHOW RANGE ... FOR ROW`](show-range-for-row.html) diff --git a/src/current/v21.1/explain-analyze.md b/src/current/v21.1/explain-analyze.md deleted file mode 100644 index 7370a796830..00000000000 --- a/src/current/v21.1/explain-analyze.md +++ /dev/null @@ -1,303 +0,0 @@ ---- -title: EXPLAIN ANALYZE -summary: The EXPLAIN ANALYZE statement executes a query and generates a physical statement plan with execution statistics. -toc: true ---- - -The `EXPLAIN ANALYZE` [statement](sql-statements.html) **executes a SQL query** and generates a statement plan with execution statistics. The `(DEBUG)` option generates a URL to download a bundle with more details about the statement plan for advanced debugging. Statement plans provide information around SQL execution, which can be used to troubleshoot slow queries by figuring out where time is being spent, how long a processor (i.e., a component that takes streams of input rows and processes them according to a specification) is not doing work, etc. For more information about distributed SQL queries, see the [DistSQL section of our SQL Layer Architecture docs](architecture/sql-layer.html#distsql). - -{{site.data.alerts.callout_info}} -{% include {{ page.version.version }}/sql/physical-plan-url.md %} -{{site.data.alerts.end}} - -## Aliases - -In CockroachDB, the following are aliases for `EXPLAIN ANALYZE`: - -- `EXPLAIN ANALYSE` - -## Synopsis - -
{% include {{ page.version.version }}/sql/generated/diagrams/explain_analyze.html %}
- -## Parameters - -Parameter | Description --------------------|----------- -`PLAN` | **New in v21.1:**
_(Default)_ Executes the statement and returns CockroachDB's statement plan with planning and execution time for an [explainable statement](sql-grammar.html#preparable_stmt). For more information, see [Default option](#default-option). -`DISTSQL` | Return the statement plan and performance statistics as well as a generated link to a graphical distributed SQL physical statement plan tree. For more information, see [`DISTSQL` option](#distsql-option). -`DEBUG` | Generate a ZIP file containing files with detailed information about the query and the database objects referenced in the query. For more information, see [`DEBUG` option](#debug-option). -`preparable_stmt` | The [statement](sql-grammar.html#preparable_stmt) you want to execute and analyze. All preparable statements are explainable. - -## Required privileges - -The user requires the appropriate [privileges](authorization.html#assign-privileges) for the statement being explained. - -## Success responses - -Successful `EXPLAIN ANALYZE` statements return tables with the following details in the `info` column: - - Detail | Description ---------|------------ -[Global properties](#global-properties) | The properties and statistics that apply to the entire statement plan. -[Statement plan tree](#statement-plan-tree-properties) | A tree representation of the hierarchy of the statement plan. -Node details | The properties, columns, and ordering details for the current statement plan node in the tree. -Time | The time details for the statement. The total time is the planning and execution time of the statement. The execution time is the time it took for the final statement plan to complete. The network time is the amount of time it took to distribute the statement across the relevant nodes in the cluster. Some statements do not need to be distributed, so the network time is 0ms. - -If you use the `DISTSQL` option, the statement will also return a URL generated for a physical statement plan that provides high level information about how a statement will be executed. For details about reading the physical statement plan, see [DistSQL Plan Viewer](#distsql-plan-viewer).

{{site.data.alerts.callout_info}}{% include {{ page.version.version }}/sql/physical-plan-url.md %} {{site.data.alerts.end}} - -If you use the [`DEBUG` option](#debug-option), the statement will return a single `text` column with a URL and instructions to download the `DEBUG` bundle, which includes the physical statement plan. - -### Global properties - -Global property | Description -----------------|------------ -planning time | The total time the planner took to create a statement plan. -execution time | The time it took for the final statement plan to complete. -distribution | Shows whether the statement was distributed or local. If `distribution` is `full`, execution of the statement is performed by multiple nodes in parallel, then the results are returned by the gateway node. If `local`, the execution plan is performed only on the gateway node. Even if the execution plan is `local`, row data may be fetched from remote nodes, but the processing of the data is performed by the local node. -vectorized | Indicates whether the [vectorized execution engine](vectorized-execution.html) was used in this statement. -rows read from KV | The number of rows read from the [Storage layer](architecture/storage-layer.html). -cumulative time spent in KV | The total amount of time spent in the [Storage layer](architecture/storage-layer.html). -cumulative time spent due to contention | The total amount of time this statement was in contention with another transaction during execution. -maximum memory usage | The maximum amount of memory used by this statement anytime during its execution. -network usage | The amount of data transferred over the network while the statement was executed. If this value is 0 B, the statement was executed on a single node and didn't use the network. - -### Statement plan tree properties - -The statement plan tree shows - -Statement plan tree properties | Description --------------------------------|------------ -processor | Each processor in the statement plan hierarchy has a node with details about that phase of the statement. For example, a statement with a `GROUP BY` clause has a `group` processor with details about the cluster nodes, rows, and operations related to the `GROUP BY` operation. -nodes | The names of the CockroachDB cluster nodes affected by this phase of the statement. -regions | The [regions](show-regions.html) where the affected nodes were located. -actual row count | The actual number of rows affected by this processor during execution. -estimated row count | The estimated number of rows affected by this processor according to the statement planner. -KV rows read | During scans, the number of rows in the [Storage layer](architecture/storage-layer.html) read by this phase of the statement. -KV bytes read | During scans, the amount of data read from the [Storage layer](architecture/storage-layer.html) during this phase of the statement. -table | The table and index used in a scan operation in a statement, in the form `{table name}@{index name}`. -spans | The interval of the key space read by the processor. If `spans` is `FULL SCAN` the table is scanned on all key ranges of the index. If `spans` is `[/1 - /1]` only the key with value `1` is read by the processor. - -## Default option - -{% include_cached new-in.html version="v21.1" %} By default, `EXPLAIN ANALYZE` uses the `PLAN` option. `EXPLAIN ANALYZE` and `EXPLAIN ANALYZE (PLAN)` produce the same output. - -## `DISTSQL` option - -`EXPLAIN ANALYZE (DISTSQL)` generates a physical statement plan diagram in the [DistSQL Plan Viewer](#distsql-plan-viewer). The DistSQL Plan Viewer displays the physical statement plan, as well as execution statistics. The statistics listed depend on the query type and the [execution engine used](vectorized-execution.html). There will be multiple diagrams if the query contains subqueries or post-queries. - -{{site.data.alerts.callout_info}} -{% include_cached new-in.html version="v21.1" %} `EXPLAIN ANALYZE (DISTSQL)` can only be used as the top-level statement in a query. -{{site.data.alerts.end}} - -## `DEBUG` option - - `EXPLAIN ANALYZE (DEBUG)` executes a query and generates a link to a ZIP file that contains the [physical statement plan](#distsql-plan-viewer), execution statistics, statement tracing, and other information about the query. - - File | Description ---------------------+------------------- -`stats-.sql` | Contains [statistics](create-statistics.html) for a table in the query. -`schema.sql` | Contains [`CREATE`](create-table.html) statements for objects in the query. -`env.sql` | Contains information about the CockroachDB environment. -`trace.txt` | Contains [statement traces](show-trace.html) in plaintext format. -`trace.json` | Contains [statement traces](show-trace.html) in JSON format. -`trace-jaeger.json` | Contains [statement traces](show-trace.html) in JSON format that can be [imported to Jaeger](query-behavior-troubleshooting.html#visualize-statement-traces-in-jaeger). -`distsql.html` | The query's [physical statement plan](#distsql-plan-viewer). This diagram is identical to the one generated by [`EXPLAIN(DISTSQL)`](explain.html#distsql-option). -`plan.txt` | The query execution plan. This is identical to the output of [`EXPLAIN (VERBOSE)`](explain.html#verbose-option). -`opt.txt` | The statement plan tree generated by the [cost-based optimizer](cost-based-optimizer.html). This is identical to the output of [`EXPLAIN (OPT)`](explain.html#opt-option). -`opt-v.txt` | The statement plan tree generated by the [cost-based optimizer](cost-based-optimizer.html), with cost details. This is identical to the output of [`EXPLAIN (OPT, VERBOSE)`](explain.html#opt-option). -`opt-vv.txt` | The statement plan tree generated by the [cost-based optimizer](cost-based-optimizer.html), with cost details and input column data types. This is identical to the output of [`EXPLAIN (OPT, TYPES)`](explain.html#opt-option). -`vec.txt` | The statement plan tree generated by the [vectorized execution](vectorized-execution.html) engine. This is identical to the output of [`EXPLAIN (VEC)`](explain.html#vec-option). -`vec-v.txt` | The statement plan tree generated by the [vectorized execution](vectorized-execution.html) engine. This is identical to the output of [`EXPLAIN (VEC, VERBOSE)`](explain.html#vec-option). -`statement.txt` | The SQL statement for the query. - -You can obtain this ZIP file by following the link provided in the `EXPLAIN ANALYZE (DEBUG)` output, or by activating [statement diagnostics](ui-statements-page.html#diagnostics) in the DB Console. - -{% include {{ page.version.version }}/sql/statement-bundle-warning.md %} - -## DistSQL plan viewer - -The graphical diagram when using the `DISTSQL` option displays the processors and operations that make up the statement plan. While the text output from `PLAN` shows the statement plan across the cluster, `DISTSQL` shows details on each node involved in the query. - -Field | Description | Execution engine -------+-------------+---------------- -<Processor>/<id> | The processor and processor ID used to read data into the SQL execution engine.

A processor is a component that takes streams of input rows, processes them according to a specification, and outputs one stream of rows. For example, a "TableReader" processor reads in data, and an "Aggregator" aggregates input rows. | Both -<table>@<index> | The index used by the processor. | Both -Spans | The interval of the key space read by the processor. For example, `[/1 - /1]` indicates that only the key with value `1` is read by the processor. | Both -Out | The output columns. | Both -cluster nodes | The names of the CockroachDB cluster nodes involved in the execution of this processor. | Both -batches output | The number of batches of columnar data output. | Vectorized engine only -rows output | The number of rows output. | Vectorized engine only -IO time | How long the TableReader processor spent reading data from disk. | Vectorized engine only -stall time | How long the processor spent not doing work. This is aggregated into the stall time numbers as the query progresses down the tree (i.e., stall time is added up and overlaps with previous time). | Row-oriented engine only -bytes read | The size of the data read by the processor. | Both -rows read | The number of rows read by the processor. | Both -KV time | The total time this phase of the query was in the [Storage layer](architecture/storage-layer.html). | Both -KV contention time | The time the [Storage layer](architecture/storage-layer.html) was in contention during this phase of the query. | Both -KV rows read | During scans, the number of rows in the [Storage layer](architecture/storage-layer.html) read by this phase of the query. | Both -KV bytes read | During scans, the amount of data read from the [Storage layer](architecture/storage-layer.html) during this phase of the query. | Both -@<n> | The index of the column relative to the input. | Both -max memory used | How much memory (if any) is used to buffer rows. | Row-oriented engine only -max disk used | How much disk (if any) is used to buffer data. Routers and processors will spill to disk buffering if there is not enough memory to buffer the data. | Row-oriented engine only -execution time | How long the engine spent executing the processor. | Vectorized engine only -max vectorized memory allocated | How much memory is allocated to the processor to buffer batches of columnar data. | Vectorized engine only -max vectorized disk used | How much disk (if any) is used to buffer columnar data. Processors will spill to disk buffering if there is not enough memory to buffer the data. | Vectorized engine only -left(@<n>)=right(@<n>) | The equality columns used in the join. | Both -stored side | The smaller table that was stored as an in-memory hash table. | Both -rows routed | How many rows were sent by routers, which can be used to understand network usage. | Row-oriented engine only -network latency | The latency time in nanoseconds between nodes in a stream. | Vectorized engine only -bytes sent | The number of actual bytes sent (i.e., encoding of the rows). This is only relevant when doing network communication. | Both -Render | The stage that renders the output. | Both -by hash | _(Orange box)_ The router, which is a component that takes one stream of input rows and sends them to a node according to a routing algorithm.

For example, a hash router hashes columns of a row and sends the results to the node that is aggregating the result rows. | Both -unordered / ordered | _(Blue box)_ A synchronizer that takes one or more output streams and merges them to be consumable by a processor. An ordered synchronizer is used to merge ordered streams and keeps the rows in sorted order. | Both -<data type> | If [`EXPLAIN (DISTSQL, TYPES)`](explain.html#distsql-option) is specified, lists the data types of the input columns. | Both -Response | The response back to the client. | Both - -## Examples - -To run the examples, initialize a demo cluster with the MovR workload. - -{% include {{ page.version.version }}/demo_movr.md %} - -### `EXPLAIN ANALYZE` - -Use `EXPLAIN ANALYZE` without an option, or equivalently with the `PLAN` option, to execute a query and display the physical statement plan with execution statistics. - -For example, the following `EXPLAIN ANALYZE` statement executes a simple query against the [MovR database](movr.html) and then displays the physical statement plan with execution statistics: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN ANALYZE SELECT city, AVG(revenue) FROM rides GROUP BY city; -~~~ - -~~~ - info ------------------------------------------------------------------------------------------- - planning time: 527µs - execution time: 66ms - distribution: full - vectorized: true - rows read from KV: 125,000 (21 MiB) - cumulative time spent in KV: 142ms - maximum memory usage: 963 KiB - network usage: 2.5 KiB (24 messages) - regions: us-east1 - - • group - │ nodes: n1, n2, n3 - │ regions: us-east1 - │ actual row count: 9 - │ estimated row count: 9 - │ group by: city - │ ordered: +city - │ - └── • scan - nodes: n1, n2, n3 - regions: us-east1 - actual row count: 125,000 - KV rows read: 125,000 - KV bytes read: 21 MiB - estimated row count: 125,000 (100% of the table; stats collected 11 minutes ago) - table: rides@primary - spans: FULL SCAN -(27 rows) - -Time: 68ms total (execution 67ms / network 1ms) -~~~ - -### `EXPLAIN ANALYZE (DISTSQL)` - -Use `EXPLAIN ANALYZE (DISTSQL)` to execute a query, display the physical statement plan with execution statistics, and generate a link to a graphical DistSQL statement plan. - -~~~ sql -EXPLAIN ANALYZE (DISTSQL) SELECT city, AVG(revenue) FROM rides GROUP BY city; -~~~ - -~~~ - info -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - planning time: 476µs - execution time: 107ms - distribution: full - vectorized: true - rows read from KV: 125,000 (21 MiB) - cumulative time spent in KV: 214ms - maximum memory usage: 963 KiB - network usage: 2.5 KiB (24 messages) - regions: us-east1 - - • group - │ nodes: n1, n2, n3 - │ regions: us-east1 - │ actual row count: 9 - │ estimated row count: 9 - │ group by: city - │ ordered: +city - │ - └── • scan - nodes: n1, n2, n3 - regions: us-east1 - actual row count: 125,000 - KV rows read: 125,000 - KV bytes read: 21 MiB - estimated row count: 125,000 (100% of the table; stats collected 12 minutes ago) - table: rides@primary - spans: FULL SCAN - - Diagram: https://cockroachdb.github.io/distsqlplan/decode.html#eJzEV11u4zYQfu8pCD5lUSUWqT9LT94stkWwib3IT4FFFQSMNHWE2KKXpBO5QW5SoKfoBXqUnqSQZG9sWaLlrJ19MUyOZjTzDb9vqCcsv45wgC8-nn78cImiRM0MxB6GBwIeIJ3CO_TL-eAMiSQGiX49H1x9RsdfisewgVMeQ5-NQeLgd0ywgSk2sIWvDTwRPAIpuchNT8WDJ3GGA9PASTqZqnz72sARF4CDJ6wSNQIc4Et2O4JzYDGIjokNHINiyagIX2TQm4hkzET-7osJS2WADjshZmOpQMRsHOJOiMMwi7wwzIh_FIYZc_LfP_xeGGamGYZZ11z8q_6Qp9zVDDFiaYwI4uoOBDbwYKoC1KNGj-QpffoNqWQMAfKdsSzXEU8VpCrh6dxk_vvP3CT4o0QCWBwgxzEc2y63b2cKFvv-EUVnyTE28C1T0R1IxKdqkr_TcbCBiwgvO0WM62cDl1tzHKViQ8ABeTbaY_1-OBQwZIqLjrUKdS9v5UDEICAOULF63_9y0x9c3vSvTk8PeuRd3oCrs4Mezf99GFz1L-f_IYNougSFV4BUrYysFVap6SXN2xm6Y_JuLcPr55e6aWPdL3F4WU81zs9lIA047veBc3F1dnOSw2Plq3NIYxC5n4F6tNOzaiDz7fL4bATNajwIVgtApmkdJLVo9Pkhn3T8VSCqadP6rO1q1n5j1vZK1qS9VNBWUtEJ8S2XiqeFTHT_-_uvMMygG4ZZ7B9vlgdzVJpxLjkpPKIZF_cLxSHR10GuHqBGJy1Cub0QNyuLQ7dTFuoZnuevK4t95DUoC-1WuzKP0U5ZNrRmiTzO_pTFaqcsVZJ8j7KQ3SpL962VxTFbKgtpqSy0PUdJW47uepof7or37a4FrrsdeW1iuJ63Tl7vyGwgr73WrnmMduTd0LOl82nvj7z2218L6G7J6701ebsNA5ZuAq3NtaCm1nOQE55KqFwP6iObOVoQD6FEV_KpiOCz4FHxmnI5KPyKjRikKq2kXJykpSlPcNmZVJ3JsjNdcSZFNqr2ckLMcqaOWYbGMOZihthoxCOm8nZZJvpUUC03y0jkCKM4kffLD5mojoz10rlNEX5jESmoRy7u0YgpSKNZgDzHK8_AwvLIEvWtwnIgxiBBJGyU_MmWL2f-quNcgCJIHgoAlkwLEfpm89yi8oV9DFKyIVTd2w3ibYBx2wPTdZzXAmNpgSE6YAjdCTA1J4ZqOWOtAlN1tvSEM_WMs7Xejt7ZeSVd1xrqu1ZjQ905Ceobqj_o2n5Sc1_91OOyhQL4HmnExXb2hMv-zrkel2YBaPj2_CHq7m7D1dee-Q0i5r26udTbW3M9LTBdvZJ0d6YkdvPM7Po6xlBHiyrVUcbZDap0HVU9MM1Ssvb5rKMM3SFlaua-voYt5r7vNVPGcnXN1Y99U9PbjUOC1qBiVlEx11HxdyYkmiOvFxLP1sLiaGBxnY3XRLvt18v180__BwAA__92ERda -(29 rows) - -Time: 110ms total (execution 109ms / network 1ms) -~~~ - -To view the [DistSQL Plan Viewer](#distsql-plan-viewer), point your browser to the URL provided. - -### `EXPLAIN ANALYZE (DEBUG)` - -Use the [`DEBUG`](#debug-option) option to generate a ZIP file containing files with information about the query and the database objects referenced in the query. For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN ANALYZE (DEBUG) SELECT city, AVG(revenue) FROM rides GROUP BY city; -~~~ - -~~~ - info --------------------------------------------------------------------------------- - Statement diagnostics bundle generated. Download from the Admin UI (Advanced - Debug -> Statement Diagnostics History), via the direct link below, or using - the command line. - Admin UI: http://127.0.0.1:8080 - Direct link: http://127.0.0.1:8080/_admin/v1/stmtbundle/645706706591449089 - Command line: cockroach statement-diag list / download -(6 rows) - -Time: 150ms total (execution 150ms / network 0ms) -~~~ - -Navigating to the URL will automatically download the ZIP file. You can also obtain the bundle by activating [statement diagnostics](ui-statements-page.html#diagnostics) in the DB Console. - -## See also - -- [`ALTER TABLE`](alter-table.html) -- [`ALTER SEQUENCE`](alter-sequence.html) -- [`BACKUP`](backup.html) -- [`CANCEL JOB`](cancel-job.html) -- [`CREATE DATABASE`](create-database.html) -- [`DROP DATABASE`](drop-database.html) -- [`EXPLAIN`](explain.html) -- [`EXECUTE`](sql-grammar.html#execute_stmt) -- [`IMPORT`](import.html) -- [Indexes](indexes.html) -- [`INSERT`](insert.html) -- [`PAUSE JOB`](pause-job.html) -- [`RESET`](reset-vars.html) -- [`RESTORE`](restore.html) -- [`RESUME JOB`](resume-job.html) -- [`SELECT`](select-clause.html) -- [Selection Queries](selection-queries.html) -- [`SET`](set-vars.html) -- [`SET CLUSTER SETTING`](set-cluster-setting.html) -- [`SHOW COLUMNS`](show-columns.html) -- [`UPDATE`](update.html) -- [`UPSERT`](upsert.html) diff --git a/src/current/v21.1/explain.md b/src/current/v21.1/explain.md deleted file mode 100644 index b18891e7526..00000000000 --- a/src/current/v21.1/explain.md +++ /dev/null @@ -1,870 +0,0 @@ ---- -title: EXPLAIN -summary: The EXPLAIN statement provides information you can use to optimize SQL queries. -toc: true ---- - -The `EXPLAIN` [statement](sql-statements.html) returns CockroachDB's statement plan for an [explainable statement](sql-grammar.html#preparable_stmt). You can then use this information to optimize the query. - -{{site.data.alerts.callout_success}} -To actually execute a statement and return a physical statement plan with execution statistics, use [`EXPLAIN ANALYZE`](explain-analyze.html). -{{site.data.alerts.end}} - -## Query optimization - -Using `EXPLAIN`'s output, you can optimize your queries by taking the following points into consideration: - -- Queries with fewer levels execute more quickly. Restructuring queries to require fewer levels of processing will generally improve performance. - -- Avoid scanning an entire table, which is the slowest way to access data. You can avoid this by [creating indexes](indexes.html) that contain at least one of the columns that the query is filtering in its `WHERE` clause. - - You can disable statement plans that perform full table scans with the `disallow_full_table_scans` [session variable](set-vars.html). When `disallow_full_table_scans=on`, attempting to execute a query with a plan that includes a full table scan will return an error. - - The statement planner uses the [cost-based optimizer](cost-based-optimizer.html) to create statement plans. Even after adding secondary indexes, the optimizer may decide that a full table scan will be faster. - - For example, if you add a secondary index to a table with a large number of rows and see that a statement plan isn't using the secondary index, it is likely that performing a full table scan using the primary key is faster than doing a secondary index scan plus an [index join](indexes.html). - -- By default, the [vectorized execution](vectorized-execution.html) engine is enabled for all [supported operations](vectorized-execution.html#disk-spilling-operations). If you are querying a table with a small number of rows, it might be more efficient to use row-oriented execution. The `vectorize_row_count_threshold` [cluster setting](cluster-settings.html) specifies the minimum number of rows required to use the vectorized engine to execute a statement plan. - -You can find out if your queries are performing entire table scans by using `EXPLAIN` to see which: - -- Indexes the query uses; shown as the value of `table` property. - -- Key values in the index are being scanned; shown as the value of the `spans` property. - - You can also see the estimated number of rows that a scan will perform in the `estimated row count` property. - -For more information about indexing and table scans, see [Find the Indexes and Key Ranges a Query Uses](#find-the-indexes-and-key-ranges-a-query-uses). - -## Synopsis - -
{% include {{ page.version.version }}/sql/generated/diagrams/explain.html %}
- -## Required privileges - -The user requires the appropriate [privileges](authorization.html#assign-privileges) for the statement being explained. - -## Parameters - - Parameter | Description ---------------------+------------ - `VERBOSE` | Show as much information as possible about the statement plan. - `TYPES` | Include the intermediate [data types](data-types.html) CockroachDB chooses to evaluate intermediate SQL expressions. - `OPT` | Display the statement plan tree generated by the [cost-based optimizer](cost-based-optimizer.html).

To include cost details used by the optimizer in planning the query, use `OPT, VERBOSE`. To include cost and type details, use `OPT, TYPES`. To include all details used by the optimizer, including statistics, use `OPT, ENV`. - `VEC` | Show detailed information about the [vectorized execution](vectorized-execution.html) plan for a query. - `preparable_stmt` | The [statement](sql-grammar.html#preparable_stmt) you want details about. All preparable statements are explainable. - `DISTSQL` | Generate a URL to a [distributed SQL physical statement plan tree](explain-analyze.html#distsql-plan-viewer).

{% include {{ page.version.version }}/sql/physical-plan-url.md %} - -{{site.data.alerts.callout_danger}} -`EXPLAIN` also includes other modes besides statement plans that are useful only to CockroachDB developers, which are not documented here. -{{site.data.alerts.end}} - -## Success responses - -Successful `EXPLAIN` statements return tables with the following details in the `info` column: - - Detail | Description ------------|------------- -Global properties | Properties that apply to the entire query plan. Global properties include `distribution` and `vectorized`. -Statement plan tree | A tree representation of the hierarchy of the statement plan. -Node details | The properties, columns, and ordering details for the current statement plan node in the tree. -Number of rows | The number of rows affected by the query. -Time | The time details for the query. The total time is the planning and execution time of the query. The execution time is the time it took for the final statement plan to complete. The network time is the amount of time it took to distribute the query across the relevant nodes in the cluster. Some queries do not need to be distributed, so the network time is 0ms. - -## Examples - -The following examples use the [`movr` example dataset](cockroach-demo.html#datasets). - -{% include {{ page.version.version }}/demo_movr.md %} - -### Default statement plans - -By default, `EXPLAIN` includes the least detail about the statement plan but can be useful to find out which indexes and index key ranges are used by a query. For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN SELECT * FROM rides WHERE revenue > 90 ORDER BY revenue ASC; -~~~ - -~~~ - info ----------------------------------------------------------------------------------------------- - distribution: full - vectorized: true - - • sort - │ estimated row count: 12,317 - │ order: +revenue - │ - └── • filter - │ estimated row count: 12,317 - │ filter: revenue > 90 - │ - └── • scan - estimated row count: 125,000 (100% of the table; stats collected 19 minutes ago) - table: rides@primary - spans: FULL SCAN -(15 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -The output shows the tree structure of the statement plan, in this case a `sort`, a `filter`, and then a `scan`. - -The output also describes a set of properties, some global to the query, and some specific to an operation listed in the true structure (in this case, `sort`, `filter`, or `scan`): - -- `distribution`:`full` -
The planner chose a distributed execution plan, where execution of the query is performed by multiple nodes in parallel, then the results are returned by the gateway node. An execution plan with `full` distribution doesn't process on all nodes in the cluster. It is executed simultaneously on multiple nodes. -
An execution plan with `local` distribution, on the other hand, is performed only on the gateway node. Even if the execution plan is `local`, row data may be fetched from remote nodes, but the processing of the data is performed by the local node. -- `vectorized`:`true` -
The plan will be executed with the [vectorized execution engine](vectorized-execution.html). -- `order`:`+revenue` -
The sort will be ordered ascending on the `revenue` column. -- `filter`: `revenue > 90` -
The scan filters on the `revenue` column. -- `estimated row count`:`125,000 (100% of the table; stats collected 19 minutes ago)` -
The estimated number of rows scanned by the query, in this case, `125,000` rows of data; the percentage of the table the query spans, in this case 100%; and when the statistics for the table were last collected, in this case 19 minutes ago. -- `table`:`rides@primary` -
The table is scanned on the `primary` index. -- `spans`:`FULL SCAN` -
The table is scanned on all key ranges of the `primary` index (i.e., a full table scan). For more information on indexes and key ranges, see the [example](#find-the-indexes-and-key-ranges-a-query-uses) below. - -### Join queries - -If you run `EXPLAIN` on a [join](joins.html) query, the output will display which type of join will be executed. For example, the following `EXPLAIN` output shows that the query will perform a [hash join](joins.html#hash-joins): - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN SELECT * FROM rides AS r -JOIN users AS u ON r.rider_id = u.id; -~~~ - -~~~ - info ------------------------------------------------------------------------------------------- - distribution: full - vectorized: true - - • hash join - │ estimated row count: 124,482 - │ equality: (rider_id) = (id) - │ - ├── • scan - │ estimated row count: 125,000 (100% of the table; stats collected 24 minutes ago) - │ table: rides@primary - │ spans: FULL SCAN - │ - └── • scan - estimated row count: 12,500 (100% of the table; stats collected 35 minutes ago) - table: users@primary - spans: FULL SCAN -(16 rows) - -Time: 1ms total (execution 1ms / network 0ms -~~~ - -The following output shows that the query will perform a cross join: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN SELECT * FROM rides AS r -JOIN users AS u ON r.city = 'new york'; -~~~ - -~~~ - info ------------------------------------------------------------------------------------------ - distribution: full - vectorized: true - - • cross join - │ estimated row count: 176,093,135 - │ - ├── • scan - │ estimated row count: 14,087 (11% of the table; stats collected 25 minutes ago) - │ table: rides@primary - │ spans: [/'new york' - /'new york'] - │ - └── • scan - estimated row count: 12,500 (100% of the table; stats collected 36 minutes ago) - table: users@primary - spans: FULL SCAN -(15 rows) - -Time: 3ms total (execution 1ms / network 2ms) -~~~ - -### Insert queries - -`EXPLAIN` output for [`INSERT`](insert.html) queries is similar to the output for standard `SELECT` queries. For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN INSERT INTO users(id, city, name) VALUES ('c28f5c28-f5c2-4000-8000-000000000026', 'new york', 'Petee'); -~~~ - -~~~ - info -------------------------------------------------------- - distribution: local - vectorized: true - - • insert - │ into: users(id, city, name, address, credit_card) - │ auto commit - │ - └── • values - size: 4 columns, 1 row -(9 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -The output for this `INSERT` lists the primary operation (in this case, `insert`), and the table and columns affected by the operation in the `into` field (in this case, the `id`, `city`, `name`, `address`, and `credit_card` columns of the `users` table). The output also includes the size of the `INSERT` in the `size` field (in this case, 4 columns in a single row). - -`EXPLAIN` output can include more information, for more complex types of `INSERT` queries. - -For example, suppose that you create a `UNIQUE` index on the `users` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE UNIQUE INDEX ON users(city, id, name); -~~~ - -To display the `EXPLAIN` output for an [`INSERT ... ON CONFLICT` statement](insert.html#on-conflict-clause) that inserts some data that might conflict with the `UNIQUE` constraint imposed on the `name`, `city`, and `id` columns: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN INSERT INTO users(id, city, name) VALUES ('c28f5c28-f5c2-4000-8000-000000000026', 'new york', 'Petee') ON CONFLICT DO NOTHING; -~~~ - -~~~ - info ----------------------------------------------------------------------------------------------------------------------------------- - distribution: local - vectorized: true - - • insert - │ into: users(id, city, name, address, credit_card) - │ auto commit - │ arbiter indexes: primary, users_city_id_name_key - │ - └── • lookup join (anti) - │ estimated row count: 0 - │ table: users@users_city_id_name_key - │ equality: (column2, column1, column3) = (city,id,name) - │ equality cols are key - │ - └── • cross join (anti) - │ estimated row count: 0 - │ - ├── • values - │ size: 4 columns, 1 row - │ - └── • scan - estimated row count: 1 (<0.01% of the table; stats collected 38 minutes ago) - table: users@users_city_id_name_key - spans: [/'new york'/'c28f5c28-f5c2-4000-8000-000000000026' - /'new york'/'c28f5c28-f5c2-4000-8000-000000000026'] -(24 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -Because the `INSERT` includes an `ON CONFLICT` clause, the query requires more than a simple `insert` operation. CockroachDB must check the provided values against the values in the database, to ensure that the `UNIQUE` constraint on `name`, `city`, and `id` is not violated. The output also lists the indexes available to detect conflicts (the `arbiter indexes`), including the `users_city_id_name_key` index. - -### Options - -#### `VERBOSE` option - -The `VERBOSE` option includes: - -- SQL expressions that are involved in each processing stage, providing more granular detail about which portion of your query is represented at each level. -- Detail about which columns are being used by each level, as well as properties of the result set on that level. - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN (VERBOSE) SELECT * FROM rides AS r -JOIN users AS u ON r.rider_id = u.id -WHERE r.city = 'new york' -ORDER BY r.revenue ASC; -~~~ - -~~~ - info ------------------------------------------------------------------------------------------------------------------------------------------------------------------- - distribution: full - vectorized: true - - • sort - │ columns: (id, city, vehicle_city, rider_id, vehicle_id, start_address, end_address, start_time, end_time, revenue, id, city, name, address, credit_card) - │ ordering: +revenue - │ estimated row count: 14,087 - │ order: +revenue - │ - └── • hash join (inner) - │ columns: (id, city, vehicle_city, rider_id, vehicle_id, start_address, end_address, start_time, end_time, revenue, id, city, name, address, credit_card) - │ estimated row count: 14,087 - │ equality: (rider_id) = (id) - │ - ├── • scan - │ columns: (id, city, vehicle_city, rider_id, vehicle_id, start_address, end_address, start_time, end_time, revenue) - │ estimated row count: 14,087 (11% of the table; stats collected 29 minutes ago) - │ table: rides@primary - │ spans: /"new york"-/"new york"/PrefixEnd - │ - └── • scan - columns: (id, city, name, address, credit_card) - estimated row count: 12,500 (100% of the table; stats collected 42 seconds ago) - table: users@primary - spans: FULL SCAN -(25 rows) - -Time: 2ms total (execution 2ms / network 0ms) -~~~ - -#### `TYPES` option - -The `TYPES` option includes - -- The types of the values used in the statement plan. -- The SQL expressions that were involved in each processing stage, and includes the columns used by each level. - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN (TYPES) SELECT * FROM rides WHERE revenue > 90 ORDER BY revenue ASC; -~~~ - -~~~ - info ----------------------------------------------------------------------------------------------------- - - distribution: full - vectorized: true - - • sort - │ columns: (id uuid, city varchar, vehicle_city varchar, rider_id uuid, vehicle_id uuid, start_address varchar, end_address varchar, start_time timestamp, end_time timestamp, revenue decimal) - │ ordering: +revenue - │ estimated row count: 12,317 - │ order: +revenue - │ - └── • filter - │ columns: (id uuid, city varchar, vehicle_city varchar, rider_id uuid, vehicle_id uuid, start_address varchar, end_address varchar, start_time timestamp, end_time timestamp, revenue decimal) - │ estimated row count: 12,317 - │ filter: ((revenue)[decimal] > (90)[decimal])[bool] - │ - └── • scan - columns: (id uuid, city varchar, vehicle_city varchar, rider_id uuid, vehicle_id uuid, start_address varchar, end_address varchar, start_time timestamp, end_time timestamp, revenue decimal) - estimated row count: 125,000 (100% of the table; stats collected 29 minutes ago) - table: rides@primary - spans: FULL SCAN -(19 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -#### `OPT` option - -To display the statement plan tree generated by the [cost-based optimizer](cost-based-optimizer.html), use the `OPT` option . For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN (OPT) SELECT * FROM rides WHERE revenue > 90 ORDER BY revenue ASC; -~~~ - -~~~ - info -------------------------------- - sort - └── select - ├── scan rides - └── filters - └── revenue > 90 -(5 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -`OPT` has four suboptions: [`VERBOSE`](#opt-verbose-option), [`TYPES`](#opt-types-option), [`ENV`](#opt-env-option), [`MEMO`](#opt-memo-option). - -##### `OPT, VERBOSE` option - -To include cost details used by the optimizer in planning the query, use the `OPT, VERBOSE` option: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN (OPT, VERBOSE) SELECT * FROM rides WHERE revenue > 90 ORDER BY revenue ASC; -~~~ - -~~~ - info ----------------------------------------------------------------------------------------------------- ... - sort - ├── columns: id:1 city:2 vehicle_city:3 rider_id:4 vehicle_id:5 start_address:6 end_address:7 start_time:8 end_time:9 revenue:10 - ├── immutable - ├── stats: [rows=12316.644, distinct(10)=9.90909091, null(10)=0] - │ histogram(10)= 0 0 11130 1187 - │ <--- 90 ------- 99 - ├── cost: 156091.288 - ├── key: (1,2) - ├── fd: (1,2)-->(3-10) - ├── ordering: +10 - ├── prune: (1-9) - ├── interesting orderings: (+2,+1) (+2,+4,+1) (+3,+5,+2,+1) (+8,+2,+1) (+4,+2,+1) - └── select - ├── columns: id:1 city:2 vehicle_city:3 rider_id:4 vehicle_id:5 start_address:6 end_address:7 start_time:8 end_time:9 revenue:10 - ├── immutable - ├── stats: [rows=12316.644, distinct(10)=9.90909091, null(10)=0] - │ histogram(10)= 0 0 11130 1187 - │ <--- 90 ------- 99 - ├── cost: 151266.03 - ├── key: (1,2) - ├── fd: (1,2)-->(3-10) - ├── prune: (1-9) - ├── interesting orderings: (+2,+1) (+2,+4,+1) (+3,+5,+2,+1) (+8,+2,+1) (+4,+2,+1) - ├── scan rides - │ ├── columns: id:1 city:2 vehicle_city:3 rider_id:4 vehicle_id:5 start_address:6 end_address:7 start_time:8 end_time:9 revenue:10 - │ ├── stats: [rows=125000, distinct(1)=125000, null(1)=0, distinct(2)=9, null(2)=0, distinct(10)=100, null(10)=0] - │ │ histogram(1)= 0 12 612 12 612 12 612 - <--- '00064a9c-dc44-4915-8000-00000000000c' ----- '0162f166-e008-49b0-8000-0000000002a5' ----- '02834d26-fa3f-4ca0-8000-0000000004cb' ----- '03c85c24-c404-4720- - │ │ histogram(2)= 0 14512 0 13637 0 14512 0 14087 0 13837 0 13737 0 13550 0 13412 0 13712 - │ │ <--- 'amsterdam' --- 'boston' --- 'los angeles' --- 'new york' --- 'paris' --- 'rome' --- 'san francisco' --- 'seattle' --- 'washington dc' - │ │ histogram(10)= 0 1387 1.2242e+05 1187 - │ │ <--- 0 ------------- 99 - │ ├── cost: 150016.01 - │ ├── key: (1,2) - │ ├── fd: (1,2)-->(3-10) - │ ├── prune: (1-10) - │ └── interesting orderings: (+2,+1) (+2,+4,+1) (+3,+5,+2,+1) (+8,+2,+1) (+4,+2,+1) - └── filters - └── revenue:10 > 90 [outer=(10), immutable, constraints=(/10: (/90 - ]; tight)] -(39 rows) - -Time: 4ms total (execution 3ms / network 1ms) -~~~ - -##### `OPT, TYPES` option - -To include cost and type details, use the `OPT, TYPES` option: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN (OPT, TYPES) SELECT * FROM rides WHERE revenue > 90 ORDER BY revenue ASC; -~~~ - -~~~ - info ----------------------------------------------------------------------------------------------------- ... - sort - ├── columns: id:1(uuid!null) city:2(varchar!null) vehicle_city:3(varchar) rider_id:4(uuid) vehicle_id:5(uuid) start_address:6(varchar) end_address:7(varchar) start_time:8(timestamp) end_time:9(timestamp) revenue:10(decimal!null) - ├── immutable - ├── stats: [rows=12316.644, distinct(10)=9.90909091, null(10)=0] - │ histogram(10)= 0 0 11130 1187 - │ <--- 90 ------- 99 - ├── cost: 156091.288 - ├── key: (1,2) - ├── fd: (1,2)-->(3-10) - ├── ordering: +10 - ├── prune: (1-9) - ├── interesting orderings: (+2,+1) (+2,+4,+1) (+3,+5,+2,+1) (+8,+2,+1) (+4,+2,+1) - └── select - ├── columns: id:1(uuid!null) city:2(varchar!null) vehicle_city:3(varchar) rider_id:4(uuid) vehicle_id:5(uuid) start_address:6(varchar) end_address:7(varchar) start_time:8(timestamp) end_time:9(timestamp) revenue:10(decimal!null) - ├── immutable - ├── stats: [rows=12316.644, distinct(10)=9.90909091, null(10)=0] - │ histogram(10)= 0 0 11130 1187 - │ <--- 90 ------- 99 - ├── cost: 151266.03 - ├── key: (1,2) - ├── fd: (1,2)-->(3-10) - ├── prune: (1-9) - ├── interesting orderings: (+2,+1) (+2,+4,+1) (+3,+5,+2,+1) (+8,+2,+1) (+4,+2,+1) - ├── scan rides - │ ├── columns: id:1(uuid!null) city:2(varchar!null) vehicle_city:3(varchar) rider_id:4(uuid) vehicle_id:5(uuid) start_address:6(varchar) end_address:7(varchar) start_time:8(timestamp) end_time:9(timestamp) revenue:10(decimal) - │ ├── stats: [rows=125000, distinct(1)=125000, null(1)=0, distinct(2)=9, null(2)=0, distinct(10)=100, null(10)=0] - │ │ histogram(1)= 0 12 612 12 612 12 612 - │ │ <--- '00064a9c-dc44-4915-8000-00000000000c' ----- '0162f166-e008-49b0-8000-0000000002a5' ----- '02834d26-fa3f-4ca0-8000-0000000004cb' ----- '03c85c24-c404-4720- - │ │ histogram(2)= 0 14512 0 13637 0 14512 0 14087 0 13837 0 13737 0 13550 0 13412 0 13712 - │ │ <--- 'amsterdam' --- 'boston' --- 'los angeles' --- 'new york' --- 'paris' --- 'rome' --- 'san francisco' --- 'seattle' --- 'washington dc' - │ │ histogram(10)= 0 1387 1.2242e+05 1187 - │ │ <--- 0 ------------- 99 - │ ├── cost: 150016.01 - │ ├── key: (1,2) - │ ├── fd: (1,2)-->(3-10) - │ ├── prune: (1-10) - │ └── interesting orderings: (+2,+1) (+2,+4,+1) (+3,+5,+2,+1) (+8,+2,+1) (+4,+2,+1) - └── filters - └── gt [type=bool, outer=(10), immutable, constraints=(/10: (/90 - ]; tight)] - ├── variable: revenue:10 [type=decimal] - └── const: 90 [type=decimal] -(41 rows) - -Time: 4ms total (execution 3ms / network 1ms) -~~~ - -##### `OPT, ENV` option - -To include all details used by the optimizer, including statistics, use the `OPT, ENV` option. - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN (OPT, ENV) SELECT * FROM rides WHERE revenue > 90 ORDER BY revenue ASC; -~~~ - -The output of `EXPLAIN (OPT, ENV)` is a URL with the data encoded in the fragment portion. Encoding the data makes it easier to share debugging information across different systems without encountering formatting issues. Opening the URL shows a page with the decoded data. The data is processed in the local browser session and is never sent out over the network. Keep in mind that if you are using any browser extensions, they may be able to access the data locally. - -~~~ - info ------------------------------------------------------------------ ... - https://cockroachdb.github.io/text/decode.html#eJzsm9Fum0gXx6_L ... -(1 row) - -Time: 32ms total (execution 32ms / network 0ms) -~~~ - -When you open the URL you should see the following output in your browser. - -~~~ --- Version: CockroachDB CCL - --- reorder_joins_limit has the default value: 8 --- enable_zigzag_join has the default value: on --- optimizer_use_histograms has the default value: on --- optimizer_use_multicol_stats has the default value: on --- locality_optimized_partitioned_index_scan has the default value: on --- distsql has the default value: auto --- vectorize has the default value: on - -CREATE TABLE public.rides ( - id UUID NOT NULL, - city VARCHAR NOT NULL, - vehicle_city VARCHAR NULL, - rider_id UUID NULL, - vehicle_id UUID NULL, - start_address VARCHAR NULL, - end_address VARCHAR NULL, - start_time TIMESTAMP NULL, - end_time TIMESTAMP NULL, - revenue DECIMAL(10,2) NULL, - CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - CONSTRAINT fk_city_ref_users FOREIGN KEY (city, rider_id) REFERENCES public.users(city, id), - CONSTRAINT fk_vehicle_city_ref_vehicles FOREIGN KEY (vehicle_city, vehicle_id) REFERENCES public.vehicles(city, id), - INDEX rides_auto_index_fk_city_ref_users (city ASC, rider_id ASC), - INDEX rides_auto_index_fk_vehicle_city_ref_vehicles (vehicle_city ASC, vehicle_id ASC), - INDEX rides_start_time_idx (start_time ASC) STORING (rider_id), - INDEX rides_rider_id_idx (rider_id ASC), - FAMILY "primary" (id, city, vehicle_city, rider_id, vehicle_id, start_address, end_address, start_time, end_time, revenue), - CONSTRAINT check_vehicle_city_city CHECK (vehicle_city = city) -); - -ALTER TABLE movr.public.rides INJECT STATISTICS '[ - { - "columns": [ - "city" - ], - "created_at": "2021-03-16 17:27:01.301903", - "distinct_count": 9, - "histo_col_type": "STRING", - "name": "__auto__", - "null_count": 0, - "row_count": 125000 - }, - { - "columns": [ - "id" - ], - "created_at": "2021-03-16 17:27:01.301903", - "distinct_count": 125617, - "histo_col_type": "UUID", - "name": "__auto__", - "null_count": 0, - "row_count": 125000 - }, - { - "columns": [ - "city", - "id" - ], - "created_at": "2021-03-16 17:27:01.301903", - "distinct_count": 124937, - "histo_col_type": "", - "name": "__auto__", - "null_count": 0, - "row_count": 125000 - }, - ... -]'; - -EXPLAIN (OPT, ENV) SELECT * FROM rides WHERE revenue > 90 ORDER BY revenue ASC; ----- -sort - └── select - ├── scan rides - └── filters - └── revenue > 90 -~~~ - -##### `OPT, MEMO` option - -The `MEMO` suboption prints a representation of the optimizer memo with the best plan. You can use the `MEMO` flag in combination with other flags. For example, `EXPLAIN (OPT, MEMO, VERBOSE)` prints the memo along with verbose output for the best plan. - - -~~~sql -EXPLAIN (OPT, MEMO) SELECT * FROM rides WHERE revenue > 90 ORDER BY revenue ASC; -~~~ - -~~~ - info ----------------------------------------------------------------------------------------------------- ... - memo (optimized, ~5KB, required=[presentation: info:13]) - ├── G1: (explain G2 [presentation: id:1,city:2,vehicle_city:3,rider_id:4,vehicle_id:5,start_address:6,end_address:7,start_time:8,end_time:9,revenue:10] [ordering: +10]) - │ └── [presentation: info:13] - │ ├── best: (explain G2="[presentation: id:1,city:2,vehicle_city:3,rider_id:4,vehicle_id:5,start_address:6,end_address:7,start_time:8,end_time:9,revenue:10] [ordering: +10]" [presentation: id:1,city:2,vehicle_city:3,rider_id:4,vehicle_id:5,start_address:6,end_address:7,start_time:8,end_time:9,revenue:10] [ordering: +10]) - │ └── cost: 2939.68 - ├── G2: (select G3 G4) - │ ├── [presentation: id:1,city:2,vehicle_city:3,rider_id:4,vehicle_id:5,start_address:6,end_address:7,start_time:8,end_time:9,revenue:10] [ordering: +10] - │ │ ├── best: (sort G2) - │ │ └── cost: 2939.66 - │ └── [] - │ ├── best: (select G3 G4) - │ └── cost: 2883.30 - ├── G3: (scan rides,cols=(1-10)) - │ ├── [ordering: +10] - │ │ ├── best: (sort G3) - │ │ └── cost: 3551.50 - │ └── [] - │ ├── best: (scan rides,cols=(1-10)) - │ └── cost: 2863.02 - ├── G4: (filters G5) - ├── G5: (gt G6 G7) - ├── G6: (variable revenue) - └── G7: (const 90) - sort - └── select - ├── scan rides - └── filters - └── revenue > 90 -(28 rows) - - -Time: 2ms total (execution 2ms / network 1ms) -~~~ - - -#### `VEC` option - -To view details about the [vectorized execution plan](vectorized-execution.html#how-vectorized-execution-works) for the query, use the `VEC` option. - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN (VEC) SELECT * FROM rides WHERE revenue > 90 ORDER BY revenue ASC; -~~~ - -The output shows the different internal functions that will be used to process each batch of column-oriented data. - -~~~ - info ------------------------------------------------- - │ - └ Node 1 - └ *colexec.sortOp - └ *colexecsel.selGTDecimalDecimalConstOp - └ *colfetcher.ColBatchScan -(5 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -#### `DISTSQL` option - -The `DISTSQL` option generates a URL for a physical statement plan that provides high level information about how a query will be executed. For details about reading the physical statement plan, see [DistSQL Plan Viewer](explain-analyze.html#distsql-plan-viewer). For more information about distributed SQL queries, see the [DistSQL section of our SQL Layer Architecture docs](architecture/sql-layer.html#distsql). - -{{site.data.alerts.callout_info}} -{% include {{ page.version.version }}/sql/physical-plan-url.md %} -{{site.data.alerts.end}} - -For example, the following `EXPLAIN(DISTSQL)` statement generates a physical plan for a simple query against the [TPC-H database](http://www.tpc.org/tpch/) loaded to a 3-node CockroachDB cluster: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN (DISTSQL) SELECT l_shipmode, AVG(l_extendedprice) FROM lineitem GROUP BY l_shipmode; -~~~ - -~~~ - automatic | url ------------+---------------------------------------------- - true | https://cockroachdb.github.io/distsqlplan... -~~~ - -To view the [DistSQL Plan Viewer](explain-analyze.html#distsql-plan-viewer), point your browser to the URL provided: - -EXPLAIN (DISTSQL) - - To include the data types of the input columns in the physical plan, use `EXPLAIN(DISTSQL, TYPES)`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN (DISTSQL, TYPES) SELECT l_shipmode, AVG(l_extendedprice) FROM lineitem GROUP BY l_shipmode; -~~~ - -~~~ - automatic | url ------------+---------------------------------------------- - true | https://cockroachdb.github.io/distsqlplan... -~~~ - -To view the [DistSQL Plan Viewer](explain-analyze.html#distsql-plan-viewer), point your browser to the URL provided: - -EXPLAIN (DISTSQL) - - -### Find the indexes and key ranges a query uses - -You can use `EXPLAIN` to understand which indexes and key ranges queries use, which can help you ensure a query isn't performing a full table scan. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE kv (k INT PRIMARY KEY, v INT); -~~~ - -Because column `v` is not indexed, queries filtering on it alone scan the entire table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN SELECT * FROM kv WHERE v BETWEEN 4 AND 5; -~~~ - -~~~ - info ------------------------------------ - distribution: full - vectorized: true - - • filter - │ filter: (v >= 4) AND (v <= 5) - │ - └── • scan - missing stats - table: kv@primary - spans: FULL SCAN -(10 rows) - -Time: 50ms total (execution 50ms / network 0ms) -~~~ - - You can disable statement plans that perform full table scans with the `disallow_full_table_scans` [session variable](set-vars.html). - -When `disallow_full_table_scans=on`, attempting to execute a query with a plan that includes a full table scan will return an error: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET disallow_full_table_scans=on; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM kv WHERE v BETWEEN 4 AND 5; -~~~ - -~~~ -ERROR: query `SELECT * FROM kv WHERE v BETWEEN 4 AND 5` contains a full table/index scan which is explicitly disallowed -SQLSTATE: P0003 -HINT: try overriding the `disallow_full_table_scans` cluster/session setting -~~~ - -If there were an index on `v`, CockroachDB would be able to avoid scanning the entire table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE INDEX v ON kv (v); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN SELECT * FROM kv WHERE v BETWEEN 4 AND 5; -~~~ - -~~~ - info --------------------------------------------------------------------------------- - distribution: local - vectorized: true - - • scan - estimated row count: 1 (100% of the table; stats collected 11 seconds ago) - table: kv@v - spans: [/4 - /5] -(7 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -Now, only part of the index `v` is getting scanned, specifically the key range starting at (and including) 4 and stopping before 6. Also note that this statement plan is not distributed across nodes on the cluster. - -### Find out if a statement is using `SELECT FOR UPDATE` locking - - CockroachDB has support for ordering transactions by controlling concurrent access to one or more rows of a table using locks. This "`SELECT FOR UPDATE` locking" can result in improved performance for contended operations. It applies to the following statements: - -- [`SELECT FOR UPDATE`](select-for-update.html) -- [`UPDATE`](update.html) - -To see whether a SQL query using one of these statements is using this feature, check the output of `EXPLAIN` for a `locking strength` field as shown below. If the `locking strength` field does not appear, then the statement is not using this feature. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE IF NOT EXISTS kv (k INT PRIMARY KEY, v INT); -UPSERT INTO kv (k, v) VALUES (1, 5), (2, 10), (3, 15); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN UPDATE kv SET v = 100 WHERE k = 1; -~~~ - -~~~ - info ------------------------------------------- - distribution: local - vectorized: true - - • update - │ table: kv - │ set: v - │ auto commit - │ - └── • render - │ - └── • scan - missing stats - table: kv@primary - spans: [/1 - /1] - locking strength: for update -(15 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -By default, `SELECT FOR UPDATE` locking is enabled for the initial row scan of `UPDATE` and `UPSERT` statements. To disable it, toggle the [`enable_implicit_select_for_update` session setting](show-vars.html#enable-implicit-select-for-update). - -## See also - -- [`ALTER TABLE`](alter-table.html) -- [`ALTER SEQUENCE`](alter-sequence.html) -- [`BACKUP`](backup.html) -- [`CANCEL JOB`](cancel-job.html) -- [`CREATE DATABASE`](create-database.html) -- [`DROP DATABASE`](drop-database.html) -- [`EXECUTE`](sql-grammar.html#execute_stmt) -- [`EXPLAIN ANALYZE`](explain-analyze.html) -- [`IMPORT`](import.html) -- [Indexes](indexes.html) -- [`INSERT`](insert.html) -- [`PAUSE JOB`](pause-job.html) -- [`RESET`](reset-vars.html) -- [`RESTORE`](restore.html) -- [`RESUME JOB`](resume-job.html) -- [`SELECT`](select-clause.html) -- [Selection Queries](selection-queries.html) -- [`SET`](set-vars.html) -- [`SET CLUSTER SETTING`](set-cluster-setting.html) -- [`SHOW COLUMNS`](show-columns.html) -- [`UPDATE`](update.html) -- [`UPSERT`](upsert.html) diff --git a/src/current/v21.1/export-spatial-data.md b/src/current/v21.1/export-spatial-data.md deleted file mode 100644 index cdc8be8b88b..00000000000 --- a/src/current/v21.1/export-spatial-data.md +++ /dev/null @@ -1,128 +0,0 @@ ---- -title: Export Spatial Data -summary: Learn how to export spatial data from CockroachDB into various formats. -toc: true ---- - - CockroachDB supports efficiently storing and querying spatial data. - -This page has instructions for exporting spatial data from CockroachDB and converting it to other spatial formats using the [`ogr2ogr`](https://gdal.org/programs/ogr2ogr.html) command. - -{% include {{page.version.version}}/spatial/ogr2ogr-supported-version.md %} - -## Step 1. Export data to CSV - -First, use the [`EXPORT`](export.html) statement to export your data to a CSV file. - -In the example statement below, we export the tornadoes database used in [Working with spatial data](spatial-data.html). - -The statement will place the CSV file in the node's [store directory](cockroach-start.html#store), in a subdirectory named `extern/tornadoes`. The file's name is automatically generated, and will be displayed as output in the [SQL shell](cockroach-sql.html). - -{% include_cached copy-clipboard.html %} -~~~ sql -EXPORT INTO CSV 'nodelocal://self/tornadoes' WITH nullas = '' FROM SELECT * from "1950-2018-torn-initpoint"; -~~~ - -~~~ - filename | rows | bytes ---------------------------------------------------+-------+----------- - export16467a35d30d25700000000000000001-n1.0.csv | 63645 | 16557064 -(1 row) -~~~ - -{{site.data.alerts.callout_info}} -This example uses local file storage. For more information about other locations where you can export your data (such as cloud storage), see [`EXPORT`](export.html). -{{site.data.alerts.end}} - -## Step 2. Combine multiple CSV files into one, as needed - -You should now have one or more CSV files in the `extern/tornadoes` subdirectory of your node's [store directory](cockroach-start.html#store). Depending on the size of the data set, there may be more than one CSV file. - -To combine multiple CSVs into one file: - -1. Open the CSV file where you will be storing the combined output in a text editor. You will need to manually add the CSV header columns to that file so that the `ogr2ogr` output we generate below will have the proper column names. Start by running the statement below on the table you are exporting to get the necessary column names: - - {% include_cached copy-clipboard.html %} - ~~~ sql - SELECT string_agg(column_name, ',') FROM [SHOW COLUMNS FROM "1950-2018-torn-initpoint"]; - ~~~ - - ~~~ - string_agg - ------------------------------------------------------------------------------------------------------ - gid,om,yr,mo,dy,date,time,tz,st,stf,stn,mag,inj,fat,loss,closs,slat,slon,elat,elon,len,wid,fc,geom - ~~~ - -2. Add the column names output above to your target output CSV file (e.g., `tornadoes.csv`) as header columns. For the tornadoes database, they should look like the following: - - ~~~ - gid, om, yr, mo, dy, date, time, tz, st, stf, stn, mag, inj, fat, loss, closs, slat, slon, elat, elon, len, wid, fc, geom - ~~~ - -2. Concatenate the non-header data from all of the exported CSV files, and append the output to the target CSV file as shown below. The node's store directory on this machine is `/tmp/node0`. - - {% include_cached copy-clipboard.html %} - ~~~ shell - cat /tmp/node0/extern/tornadoes/*.csv >> tornadoes.csv - ~~~ - -## Step 3. Convert CSV to other formats using `ogr2ogr` - -Now that you have your data in CSV format, you can convert it to other spatial formats using [`ogr2ogr`](https://gdal.org/programs/ogr2ogr.html). - -For example, to convert the data to SQL, run the following command: - -{% include_cached copy-clipboard.html %} -~~~ shell -ogr2ogr -f PGDUMP tornadoes.sql -lco LAUNDER=NO -lco DROP_TABLE=OFF -oo GEOM_POSSIBLE_NAMES=geom -oo KEEP_GEOM_COLUMNS=off tornadoes.csv -~~~ - -Note that the options `-oo GEOM_POSSIBLE_NAMES= -oo KEEP_GEOM_COLUMNS=off` are required no matter what output format you are converting into. - -For more information about the formats supported by `ogr2ogr`, see the [`ogr2ogr` documentation](https://gdal.org/programs/ogr2ogr.html). - -{% include {{page.version.version}}/spatial/ogr2ogr-supported-version.md %} - -Finally, note that SQL type information is lost in the conversion to CSV, such that the `tornadoes.sql` file output by the `ogr2ogr` command above lists every non-geometry field as a [`VARCHAR`](string.html). - -This can be addressed in one of the following ways: - -- Modify the data definitions in the SQL output file to use the correct types. - -- Run [`ALTER TYPE`](alter-type.html) statements to restore the data's SQL types after loading this data into another database (including another CockroachDB instance). - -## See also - -- [`EXPORT`](export.html) -- [Migrate from Shapefiles](migrate-from-shapefiles.html) -- [Migrate from GeoJSON](migrate-from-geojson.html) -- [Migrate from GeoPackage](migrate-from-geopackage.html) -- [Migrate from OpenStreetMap](migrate-from-openstreetmap.html) -- [Spatial features](spatial-features.html) -- [Spatial indexes](spatial-indexes.html) -- [Working with Spatial Data](spatial-data.html) -- [Spatial and GIS Glossary of Terms](spatial-glossary.html) -- [Known Limitations](known-limitations.html#spatial-support-limitations) -- [Spatial functions](functions-and-operators.html#spatial-functions) -- [POINT](point.html) -- [LINESTRING](linestring.html) -- [POLYGON](polygon.html) -- [MULTIPOINT](multipoint.html) -- [MULTILINESTRING](multilinestring.html) -- [MULTIPOLYGON](multipolygon.html) -- [GEOMETRYCOLLECTION](geometrycollection.html) -- [Well known text](well-known-text.html) -- [Well known binary](well-known-binary.html) -- [GeoJSON](geojson.html) -- [SRID 4326 - longitude and latitude](srid-4326.html) -- [`ST_Contains`](st_contains.html) -- [`ST_ConvexHull`](st_convexhull.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Covers`](st_covers.html) -- [`ST_Disjoint`](st_disjoint.html) -- [`ST_Equals`](st_equals.html) -- [`ST_Intersects`](st_intersects.html) -- [`ST_Overlaps`](st_overlaps.html) -- [`ST_Touches`](st_touches.html) -- [`ST_Union`](st_union.html) -- [`ST_Within`](st_within.html) diff --git a/src/current/v21.1/export.md b/src/current/v21.1/export.md deleted file mode 100644 index 0bf0b057e68..00000000000 --- a/src/current/v21.1/export.md +++ /dev/null @@ -1,150 +0,0 @@ ---- -title: EXPORT -summary: Export tabular data from a CockroachDB cluster in CSV format. -toc: true ---- - -The `EXPORT` [statement](sql-statements.html) exports tabular data or the results of arbitrary `SELECT` statements to CSV files. - -Using the [CockroachDB distributed execution engine](architecture/sql-layer.html#distsql), `EXPORT` parallelizes CSV creation across all nodes in the cluster, making it possible to quickly get large sets of data out of CockroachDB in a format that can be ingested by downstream systems. If you do not need distributed exports, you can [export tabular data in CSV format](#non-distributed-export-using-the-sql-client). - -{{site.data.alerts.callout_info}} -`EXPORT` no longer requires an Enterprise license. -{{site.data.alerts.end}} - -## Cancelling export - -After the export has been initiated, you can cancel it with [`CANCEL QUERY`](cancel-query.html). - -## Synopsis - -
{% include {{ page.version.version }}/sql/generated/diagrams/export.html %}
- -{{site.data.alerts.callout_info}}The EXPORT statement cannot be used within a transaction.{{site.data.alerts.end}} - -## Required privileges - -{% include_cached new-in.html version="v21.1" %} The user must have the `SELECT` [privilege](authorization.html#assign-privileges) on the table being exported, unless the [destination URI requires `admin` privileges](import.html#source-privileges). - -## Parameters - - Parameter | Description ------------|------------- - `file_location` | Specify the [URL of the file location](#export-file-url) where you want to store the exported CSV data.

Note: It is best practice to use a unique destination for each export, to avoid mixing files from different exports in one directory. - `WITH kv_option` | Control your export's behavior with [these options](#export-options). - `select_stmt` | Specify the query whose result you want to export to CSV format. - `table_name` | Specify the name of the table you want to export to CSV format. - -### Export file URL - -You can specify the base directory where you want to store the exported .csv files. CockroachDB will create the export file(s) in the specified directory with programmatically generated names (e.g., `exportabc123-n1.1.csv`, `exportabc123-n1.2.csv`, `exportabc123-n2.1.csv`, ...). Each export should use a unique destination directory to avoid collision with other exports. - -The `EXPORT` command [returns](#success-responses) the list of files to which the data was exported. You may wish to record these for use in subsequent imports. - -{{site.data.alerts.callout_info}} -A hexadecimal hash code (`abc123...` in the file names above) uniquely identifies each export _run_; files sharing the same hash are part of the same export. If you see multiple hash codes within a single destination directory, then the directory contains multiple exports, which will likely cause confusion (duplication) on import. We recommend that you manually clean up the directory, to ensure that it contains only a single export run. -{{site.data.alerts.end}} - -For more information, see the following: - -- [Use Cloud Storage for Bulk Operations](use-cloud-storage-for-bulk-operations.html) -- [Use a Local File Server for Bulk Operations](use-a-local-file-server-for-bulk-operations.html) - -### Export options - -You can control the [`EXPORT`](export.html) process's behavior using any of the following key-value pairs as a `kv_option`. - -Key |
Context
| Value | ---------------------+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------- -`delimiter` | `CSV DATA` | The ASCII character that delimits columns in your rows. If not using comma as your column delimiter, you can specify another ASCII character as the delimiter. **Default:** `,`.

To use tab-delimited values: `WITH delimiter = e'\t'` -`nullas` | `CSV DATA`, `DELIMITED DATA` | The string that should be used to represent _NULL_ values. To avoid collisions, it is important to pick `nullas` values that do not appear in the exported data.

To use empty columns as _NULL_: `WITH nullas = ''` -`compression` | `CSV DATA` | This instructs export to write `gzip` compressed CSV files to the specified destination.

See the [example](#export-gzip-compressed-csv-files) below. -`chunk_rows` | `CSV DATA` | The number of rows to be converted to CSV and written to a single CSV file. **Default:** `100000`.
For example, `WITH chunk_rows = '5000'` for a table with 10,000 rows would produce two CSV files.

**Note**:`EXPORT` will stop and upload the CSV file whether the configured limit for `chunk_rows` or `chunk_size` is reached first. -`chunk_size` | `CSV DATA` | A target size per CSV file that you can specify during an `EXPORT`. Once the target size is reached, the CSV file is uploaded before processing further rows. **Default:** `32MB`.
For example, to set the size of each CSV file uploaded during the export to 10MB: `WITH chunk_size = '10MB'`.

**Note**:`EXPORT` will stop and upload the CSV file whether the configured limit for `chunk_rows` or `chunk_size` is reached first. - -## Success responses - -Successful `EXPORT` returns a table of (perhaps multiple) files to which the data was exported: - -| Response | Description | -|-----------|-------------| -`filename` | The file to which the data was exported. -`rows` | The number of rows exported to this file. -`bytes` | The file size in bytes. - -## Examples - -{% include {{ page.version.version }}/backups/bulk-auth-options.md %} - -Each of these examples use the `bank` database and the `customers` table; `customer-export-data` is the demonstration path to which we're exporting our customers' data in this example. - -### Export a table - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPORT INTO CSV - 's3://{BUCKET NAME}/{customer-export-data}?AWS_ACCESS_KEY_ID={ACCESS KEY}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}' - WITH delimiter = '|' FROM TABLE bank.customers; -~~~ - -### Export using a `SELECT` statement - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPORT INTO CSV - 's3://{BUCKET NAME}/{customer-export-data}?AWS_ACCESS_KEY_ID={ACCESS KEY}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}' - FROM SELECT * FROM bank.customers WHERE id >= 100; -~~~ - -For more information, see [selection queries](selection-queries.html). - -### Non-distributed export using the SQL client - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql -e "SELECT * from bank.customers WHERE id>=100;" --format=csv > my.csv -~~~ - -For more information, about the SQL client, see [`cockroach sql`](cockroach-sql.html). - -### Export gzip compressed CSV files - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPORT INTO CSV - 's3://{BUCKET NAME}/{customer-export-data}?AWS_ACCESS_KEY_ID={ACCESS KEY}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}' - WITH compression = 'gzip' FROM TABLE bank.customers; -~~~ - -~~~ -filename | rows | bytes ----------------------------------------------------+------+-------- -export16808a04292505c80000000000000001-n1.0.csv.gz | 17 | 824 -(1 row) -~~~ - -### View a running export - -View running exports by using [`SHOW STATEMENTS`](show-statements.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW STATEMENTS; -~~~ - -### Cancel a running export - -Use [`SHOW STATEMENTS`](show-statements.html) to get a running export's `query_id`, which can be used to [cancel the export](cancel-query.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> CANCEL QUERY '14dacc1f9a781e3d0000000000000001'; -~~~ - -## Known limitation - -`EXPORT` may fail with an error if the SQL statements are incompatible with DistSQL. In that case, [export tabular data in CSV format](#non-distributed-export-using-the-sql-client). - -## See also - -- [Use a Local File Server for Bulk Operations](use-a-local-file-server-for-bulk-operations.html) diff --git a/src/current/v21.1/file-an-issue.md b/src/current/v21.1/file-an-issue.md deleted file mode 100644 index c9e0026820e..00000000000 --- a/src/current/v21.1/file-an-issue.md +++ /dev/null @@ -1,66 +0,0 @@ ---- -title: File an Issue -summary: Learn how to file a GitHub issue with CockroachDB. -toc: false ---- - -If you've tried to [troubleshoot](troubleshooting-overview.html) an issue yourself, have [reached out for help](support-resources.html), and are still stumped, you can file an issue in GitHub. - -To file an issue in GitHub, we need the following information: - -1. A summary of the issue. - -2. The steps to reproduce the issue. - -3. The result you expected. - -4. The result that actually occurred. - -5. The first few lines of the log file from each node in the cluster in a timeframe as close as possible to reproducing the issue. On most Unix-based systems running with defaults, you can get this information using the following command: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ grep -F '[config]' cockroach-data/logs/cockroach.log - ~~~ - {{site.data.alerts.callout_info}}You might need to replace cockroach-data/logs with the location of your logs.{{site.data.alerts.end}} - If the logs are not available, please include the output of `cockroach version` for each node in the cluster. - -### Template - -You can use this as a template for [filing an issue in GitHub](https://github.com/cockroachdb/cockroach/issues/new): - -~~~ - -## Summary - - - -## Steps to reproduce - -1. -2. -3. - -## Expected Result - - - -## Actual Result - - - -## Log files/version - -### Node 1 - - - -### Node 2 - - - -### Node 3 - - - -~~~ diff --git a/src/current/v21.1/float.md b/src/current/v21.1/float.md deleted file mode 100644 index f0dc9de7d1d..00000000000 --- a/src/current/v21.1/float.md +++ /dev/null @@ -1,106 +0,0 @@ ---- -title: FLOAT -summary: The FLOAT data type stores inexact, floating-point numbers with up to 17 digits in total and at least one digit to the right of the decimal point. -toc: true ---- - -CockroachDB supports various inexact, floating-point number [data types](data-types.html) with up to 17 digits of decimal precision. - -They are handled internally using the [standard double-precision (64-bit binary-encoded) IEEE754 format](https://en.wikipedia.org/wiki/IEEE_floating_point). - - -## Names and Aliases - -Name | Aliases ------|-------- -`FLOAT` | None -`REAL` | `FLOAT4` -`DOUBLE PRECISION` | `FLOAT8` - -## Syntax - -A constant value of type `FLOAT` can be entered as a [numeric literal](sql-constants.html#numeric-literals). -For example: `1.414` or `-1234`. - -The special IEEE754 values for positive infinity, negative infinity -and [NaN (Not-a-Number)](https://en.wikipedia.org/wiki/NaN) cannot be -entered using numeric literals directly and must be converted using an -[interpreted literal](sql-constants.html#interpreted-literals) or an -[explicit conversion](scalar-expressions.html#explicit-type-coercions) -from a string literal instead. - -The following values are recognized: - - Syntax | Value -----------------------------------------|------------------------------------------------ - `inf`, `infinity`, `+inf`, `+infinity` | +∞ - `-inf`, `-infinity` | -∞ - `nan` | [NaN (Not-a-Number)](https://en.wikipedia.org/wiki/NaN) - -For example: - -- `FLOAT '+Inf'` -- `'-Inf'::FLOAT` -- `CAST('NaN' AS FLOAT)` - -## Size - -A `FLOAT` column supports values up to 8 bytes in width, but the total storage size is likely to be larger due to CockroachDB metadata. - -## Examples - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE floats (a FLOAT PRIMARY KEY, b REAL, c DOUBLE PRECISION); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM floats; -~~~ - -~~~ -+-------------+------------------+-------------+----------------+-----------------------+-------------+ -| column_name | data_type | is_nullable | column_default | generation_expression | indices | -+-------------+------------------+-------------+----------------+-----------------------+-------------+ -| a | FLOAT | false | NULL | | {"primary"} | -| b | REAL | true | NULL | | {} | -| c | DOUBLE PRECISION | true | NULL | | {} | -+-------------+------------------+-------------+----------------+-----------------------+-------------+ -(3 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO floats VALUES (1.012345678901, 2.01234567890123456789, CAST('+Inf' AS FLOAT)); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM floats; -~~~ - -~~~ -+----------------+--------------------+------+ -| a | b | c | -+----------------+--------------------+------+ -| 1.012345678901 | 2.0123456789012346 | +Inf | -+----------------+--------------------+------+ -(1 row) -# Note that the value in "b" has been limited to 17 digits. -~~~ - -## Supported casting and conversion - -`FLOAT` values can be [cast](data-types.html#data-type-conversions-and-casts) to any of the following data types: - -Type | Details ------|-------- -`INT` | Truncates decimal precision and requires values to be between -2^63 and 2^63-1 -`DECIMAL` | Causes an error to be reported if the value is NaN or +/- Inf. -`BOOL` | **0** converts to `false`; all other values convert to `true` -`STRING` | -- - -## See also - -[Data Types](data-types.html) diff --git a/src/current/v21.1/flyway.md b/src/current/v21.1/flyway.md deleted file mode 100644 index f7e1ec148b5..00000000000 --- a/src/current/v21.1/flyway.md +++ /dev/null @@ -1,189 +0,0 @@ ---- -title: Migrate CockroachDB Schemas with Flyway -summary: This tutorial guides you through a series of simple database schema changes using Flyway, an open-source schema migration tool. -toc: true ---- - -This page guides you through a series of simple database schema changes using Flyway, an open-source schema migration tool. For detailed information about using Flyway, see the [Flyway documentation site](https://flywaydb.org/documentation/). - -## Watch the demo - -{% include_cached youtube.html video_id="xz4j5tU0ZRU" %} - -## Before You Begin - -Before you begin, do the following: - -1. [Install CockroachDB](install-cockroachdb.html) and [start a secure cluster](secure-a-cluster.html). -1. Download the latest version of the [Flyway command-line tool](https://flywaydb.org/documentation/commandline/#download-and-installation). CockroachDB v21.1 is fully compatible with Flyway versions 7.1.0 and greater. - -## Step 1. Configure Flyway connect to CockroachDB - -1. Extract the Flyway TAR file that you downloaded, and change directories to the extracted `flyway-x.x.x` folder. For example: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ tar -xvf flyway-commandline-6.4.2-macosx-x64.tar.gz - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cd flyway-6.4.2 - ~~~ - -1. Edit the `flyway-x.x.x/conf/flyway.conf` configuration file to specify the correct [connection parameters](connection-parameters.html) for your running, secure cluster. For example: - - {% include_cached copy-clipboard.html %} - ~~~ conf - ... - flyway.url=jdbc:postgresql://localhost:26257/bank?ssl=true&sslmode=require&sslrootcert=certs/ca.crt&sslkey=certs/client.max.key&sslcert=certs/client.max.crt - flyway.user=max - flyway.password=roach - ... - ~~~ - - {{site.data.alerts.callout_info}} - The SSL connection parameters in the connection URL must specify the full path to the certificates that you generated when you [started the secure cluster](secure-a-cluster.html). Also, the user that you specify (e.g., `max`) must also have [admin privileges](grant.html) on the database whose schema you want to change (e.g., `bank`). - {{site.data.alerts.end}} - -## Step 2. Create a schema migration - -Flyway executes SQL statements defined in `.sql` files located in the `flyway-x.x.x/sql` subdirectory. The schema changes defined in these `.sql` files are known as *migrations*. - -1. Create a `.sql` file with a name that follows the [Flyway naming conventions](https://flywaydb.org/documentation/migrations#naming). For example: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ touch sql/V1__Add_accounts_table.sql - ~~~ - -1. Edit the `.sql` file, adding a [`CREATE TABLE IF NOT EXISTS`](create-table.html) statement for the table that you want to create, and a simple [`INSERT`](insert.html) statement to initialize the table with some data. For example: - - {% include_cached copy-clipboard.html %} - ~~~ sql - /* Create accounts table */ - CREATE TABLE IF NOT EXISTS accounts ( - id INT PRIMARY KEY, - balance INT - ); - - /* Add initial data to accounts table */ - INSERT INTO accounts (id, balance) VALUES (1, 1000), (2, 250); - ~~~ - -## Step 3. Execute a schema migration - -To execute the migration, run the following command from the top of the `flyway-x.x.x` directory: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ ./flyway migrate -~~~ - -You should see output similar to the following: - -~~~ -Database: jdbc:postgresql://localhost:26257/bank (PostgreSQL 9.5) -Successfully validated 1 migration (execution time 00:00.011s) -Creating Schema History table "bank"."flyway_schema_history" ... -Current version of schema "bank": << Empty Schema >> -Migrating schema "bank" to version 1 - Add accounts table [non-transactional] -Successfully applied 1 migration to schema "bank" (execution time 00:00.081s) -~~~ - -The schema `"bank"` is now on version 1. - -## Step 4. Add additional migrations - -Suppose that you want to change the primary key of the `accounts` table from a simple, incrementing [integer](int.html) (in this case, `id`) to an auto-generated [UUID](uuid.html), to follow some [CockroachDB best practices](performance-best-practices-overview.html#unique-id-best-practices). You can make these changes to the schema by creating and executing an additional migration: - -1. Create a second `.sql` schema migration file, and name the file following the [Flyway naming conventions](https://flywaydb.org/documentation/migrations#naming), to specify a new migration version. For example: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ touch sql/V2__Alter_accounts_pk.sql - ~~~ - - This file will create a version 2 of the `"bank"` schema. - -1. Edit the `V2__Alter_accounts_pk.sql` migration file, adding some SQL statements that will add a new column to the `accounts` table, and alter the table's primary key. For example: - - {% include_cached copy-clipboard.html %} - ~~~ sql - /* Add new UUID-typed column */ - ALTER TABLE accounts ADD COLUMN unique_id UUID NOT NULL DEFAULT gen_random_uuid(); - - /* Change primary key */ - ALTER TABLE accounts ALTER PRIMARY KEY USING COLUMNS (unique_id); - ~~~ - -1. Execute the migration by running the `flyway migrate` command from the top of the `flyway-x.x.x` directory: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ ./flyway migrate - ~~~ - - You should see output similar to the following: - - ~~~ - Flyway Community Edition 6.4.2 by Redgate - Database: jdbc:postgresql://localhost:26257/bank (PostgreSQL 9.5) - Successfully validated 2 migrations (execution time 00:00.016s) - Current version of schema "bank": 1 - Migrating schema "bank" to version 2 - Alter accounts pk [non-transactional] - DB: primary key changes are finalized asynchronously; further schema changes on this table may be restricted until the job completes - Successfully applied 1 migration to schema "bank" (execution time 00:00.508s) - ~~~ - - The schema `"bank"` is now on version 2. - -1. Check the complete and pending Flyway migrations with the `flyway info` command: - - ~~~ shell - $ ./flyway info - ~~~ - - ~~~ - Flyway Community Edition 6.4.2 by Redgate - Database: jdbc:postgresql://localhost:26257/bank (PostgreSQL 9.5) - Schema version: 2 - - +-----------+---------+--------------------+------+---------------------+---------+ - | Category | Version | Description | Type | Installed On | State | - +-----------+---------+--------------------+------+---------------------+---------+ - | Versioned | 1 | Add accounts table | SQL | 2020-05-13 17:16:54 | Success | - | Versioned | 2 | Alter accounts pk | SQL | 2020-05-14 13:27:27 | Success | - +-----------+---------+--------------------+------+---------------------+---------+ - ~~~ - -## Flyway and Transactions - -When used with most databases, [Flyway wraps the statements in a migration within a single transaction](https://flywaydb.org/documentation/migrations#transactions). When used with CockroachDB, Flyway does *not* wrap schema migrations in transactions. [Transaction boundaries](transactions.html) are instead handled by CockroachDB. - -### Transaction retries - -When multiple, concurrent transactions or statements are issued to a single CockroachDB cluster, [transaction contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention) can cause schema migrations to fail. In the event of transaction contention, CockroachDB returns a `40001 SQLSTATE` (i.e., a serialization failure), and Flyway automatically retries the migration. For more information about client-side transaction retries in CockroachDB, see [Transaction Retries](transactions.html#transaction-retries). - -### Transactional schema changes - -Support for [transactional schema changes](online-schema-changes.html) is limited in CockroachDB. As a result, if a migration with multiple schema changes fails at any point, the partial migration could leave the database schema in an incomplete state. If this happens, manual intervention will be required to determine the state of the schema, in addition to any possible fixes. - -Note that this limitation also applies to single [`ALTER TABLE`](alter-table.html) statements that include multiple schema changes (e.g., `ALTER TABLE ... ALTER COLUMN ... RENAME ..., ADD COLUMN ...`). - -## Report Issues with Flyway and CockroachDB - -If you run into problems, please file an issue on the [Flyway issue tracker](https://github.com/flyway/flyway/issues), including the following details about the environment where you encountered the issue: - -- CockroachDB version ([`cockroach version`](cockroach-version.html)) -- Flyway version -- Operating system -- Steps to reproduce the behavior - -## See Also - -+ [Flyway documentation](https://flywaydb.org/documentation/) -+ [Flyway issue tracker](https://github.com/flyway/flyway/issues) -+ [Client connection parameters](connection-parameters.html) -+ [Third-Party Database Tools](third-party-database-tools.html) -+ [Learn CockroachDB SQL](learn-cockroachdb-sql.html) diff --git a/src/current/v21.1/follower-reads.md b/src/current/v21.1/follower-reads.md deleted file mode 100644 index 8757398a4eb..00000000000 --- a/src/current/v21.1/follower-reads.md +++ /dev/null @@ -1,117 +0,0 @@ ---- -title: Follower Reads -summary: To reduce latency for read queries, you can choose to have the closest node serve the request using the follower reads feature. -toc: true ---- - -Follower reads are a mechanism that CockroachDB uses to provide faster reads in situations where you can afford to read data that may be slightly less than current (using [`AS OF SYSTEM TIME`](as-of-system-time.html)). Normally, reads have to be serviced by a replica's [leaseholder](architecture/overview.html#architecture-leaseholder). This can be slow, since the leaseholder may be geographically distant from the gateway node that is issuing the query. - -A follower read is a read taken from the closest [replica](architecture/overview.html#architecture-replica), regardless of the replica's leaseholder status. This can result in much better latency in [geo-distributed, multi-region deployments](topology-patterns.html#multi-region). - - The shortest interval at which [`AS OF SYSTEM TIME`](as-of-system-time.html) can serve follower reads is 4.8 seconds. In prior versions of CockroachDB, the interval was 48 seconds. - -For instructions showing how to use follower reads to get low latency, historical reads in multi-region deployments, see the [Follower Reads Topology Pattern](topology-follower-reads.html). - -{{site.data.alerts.callout_info}} -This is an [Enterprise feature](enterprise-licensing.html). -{{site.data.alerts.end}} - -## Watch the demo - -{% include_cached youtube.html video_id="V--skgN_JMo" %} - -## How follower reads work - -Each CockroachDB range tracks a property called its [_closed timestamp_](architecture/transaction-layer.html#closed-timestamps), which means that no new writes can ever be introduced below that timestamp. The closed timestamp advances forward by some target interval behind the current time. If the range receives a write at a timestamp less than its closed timestamp, it rejects the write. - -With [follower reads enabled](#enable-disable-follower-reads), any replica on a node can serve a read for a key as long as the time at which the operation is performed (i.e., the [`AS OF SYSTEM TIME`](as-of-system-time.html) value) is less or equal to the node's closed timestamp. - -When a gateway node in a cluster with follower reads enabled receives a request to read a key with a sufficiently old [`AS OF SYSTEM TIME`](as-of-system-time.html) value, it forwards the request to the closest node that contains a replica of the data–– whether it be a follower or the leaseholder. - -## When to use follower reads - -As long as your [`SELECT` operations](select-clause.html) can tolerate reading data that was current as of the [follower read timestamp](#run-queries-that-use-follower-reads), follower reads can reduce read latencies and increase throughput. - -You should not use follower reads when your application cannot tolerate reading data that was current as of the [follower read timestamp](#run-queries-that-use-follower-reads), since the results of follower reads may not reflect the latest writes against the tables you are querying. - -In addition, follower reads are "read-only" operations; they cannot be used in any way in read-write transactions. - -## Using follower reads - -### Enable/disable follower reads - -Use [`SET CLUSTER SETTING`](set-cluster-setting.html) to set `kv.closed_timestamp.follower_reads_enabled` to: - -- `true` to enable follower reads _(default)_ -- `false` to disable follower reads - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET CLUSTER SETTING kv.closed_timestamp.follower_reads_enabled = false; -~~~ - -If you have follower reads enabled, you may want to [verify that follower reads are happening](#verify-that-follower-reads-are-happening). - -{{site.data.alerts.callout_info}} -If follower reads are enabled, but the time-travel query is not using [`AS OF SYSTEM TIME`](as-of-system-time.html) far enough in the past (as defined by the [follower read timestamp](#run-queries-that-use-follower-reads)), CockroachDB does not perform a follower read. Instead, the read accesses the [leaseholder replica](architecture/overview.html#architecture-leaseholder). This adds network latency if the leaseholder is not the closest replica to the gateway node. -{{site.data.alerts.end}} - -### Verify that follower reads are happening - -To verify that your cluster is performing follower reads: - -1. Make sure that [follower reads are enabled](#enable-disable-follower-reads). -2. Go to the [Custom Chart Debug Page in the DB Console](ui-custom-chart-debug-page.html) and add the metric `follower_read.success_count` to the time series graph you see there. The number of follower reads performed by your cluster will be shown. - -### Run queries that use follower reads - -Any [`SELECT` statement](select-clause.html) with an [`AS OF SYSTEM TIME`](as-of-system-time.html) value at least 4.8 seconds in the past can be a follower read (i.e., served by any replica). - -To simplify this calculation, we've added the convenience [function](functions-and-operators.html) `follower_read_timestamp()`. `follower_read_timestamp()` returns the [`TIMESTAMP`](timestamp.html) `statement_timestamp() - 4.8s` (known as the [follower read timestamp](follower-reads.html#run-queries-that-use-follower-reads)). Using this function in an `AS OF SYSTEM TIME` statement will set the time as close as possible to the present time while remaining safe for [follower reads](follower-reads.html): - -``` sql -SELECT ... FROM ... AS OF SYSTEM TIME follower_read_timestamp(); -``` - -### Run read-only transactions that use follower reads - -You can set the [`AS OF SYSTEM TIME`](as-of-system-time.html) value for all operations in a read-only transaction: - -```sql -BEGIN; - -SET TRANSACTION AS OF SYSTEM TIME follower_read_timestamp(); -SELECT ... -SELECT ... - -COMMIT; -``` - -Note that follower reads are "read-only" operations; they cannot be used in any way in read-write transactions. - -{{site.data.alerts.callout_success}} -Using the [`SET TRANSACTION`](set-transaction.html#use-the-as-of-system-time-option) statement as shown in the example above will make it easier to use the follower reads feature from [drivers and ORMs](install-client-drivers.html). - -{% include_cached new-in.html version="v21.1" %} To set `AS OF SYSTEM TIME follower_read_timestamp()` on all implicit and explicit read-only transactions by default, set the `default_transaction_use_follower_reads` [session variable](set-vars.html) to `on`. When `default_transaction_use_follower_reads=on` and follower reads are enabled, all read-only transactions use follower reads. -{{site.data.alerts.end}} - -## Follower reads and long-running writes - -Long-running write transactions will create write intents with a timestamp near when the transaction began. When a follower read encounters a write intent, it will often end up in a "transaction wait queue", waiting for the operation to complete; however, this runs counter to the benefit follower reads provides. - -To counteract this, you can issue all follower reads in explicit transactions set with `HIGH` priority: - -```sql -BEGIN PRIORITY HIGH AS OF SYSTEM TIME follower_read_timestamp(); -SELECT ... -SELECT ... -COMMIT; -``` - -## See Also - -- [Cluster Settings Overview](cluster-settings.html) -- [Load-Based Splitting](load-based-splitting.html) -- [Network Latency Page](ui-network-latency-page.html) -- [Enterprise Features](enterprise-licensing.html) -- [Follower Reads Topology Pattern](topology-follower-reads.html) diff --git a/src/current/v21.1/foreign-key.md b/src/current/v21.1/foreign-key.md deleted file mode 100644 index 4eb29ef14f2..00000000000 --- a/src/current/v21.1/foreign-key.md +++ /dev/null @@ -1,919 +0,0 @@ ---- -title: Foreign Key Constraint -summary: The `FOREIGN KEY` constraint specifies a column can contain only values exactly matching existing values from the column it references. -toc: true ---- - -A foreign key is a column (or combination of columns) in a table whose values must match values of a column in some other table. `FOREIGN KEY` constraints enforce [referential integrity](https://en.wikipedia.org/wiki/Referential_integrity), which essentially says that if column value A refers to column value B, then column value B must exist. - -For example, given an `orders` table and a `customers` table, if you create a column `orders.customer_id` that references the `customers.id` primary key: - -- Each value inserted or updated in `orders.customer_id` must exactly match a value in `customers.id`, or be `NULL`. -- Values in `customers.id` that are referenced by `orders.customer_id` cannot be deleted or updated, unless you have [cascading actions](#use-a-foreign-key-constraint-with-cascade). However, values of `customers.id` that are _not_ present in `orders.customer_id` can be deleted or updated. - -To learn more about the basics of foreign keys, watch the video below: - -{% include_cached youtube.html video_id="5kiMg7GXAsY" widescreen=true %} - -{{site.data.alerts.callout_success}} -To read more about how foreign keys work, see our [What is a Foreign Key? (With SQL Examples)](https://www.cockroachlabs.com/blog/what-is-a-foreign-key/) blog post. -{{site.data.alerts.end}} - -## Details - -### Rules for creating foreign keys - -**Foreign Key Columns** - -- Foreign key columns must use their referenced column's [type](data-types.html). -- A foreign key column cannot be a virtual [computed column](computed-columns.html), but it can be a stored computed column. -- A single column can have multiple foreign key constraints. For an example, see [Add multiple foreign key constraints to a single column](#add-multiple-foreign-key-constraints-to-a-single-column). - -**Referenced Columns** - -- Referenced columns must contain only unique sets of values. This means the `REFERENCES` clause must use exactly the same columns as a [`UNIQUE`](unique.html) or [`PRIMARY KEY`](primary-key.html) constraint on the referenced table, in the same order as they are defined in the referenced table. For example, the clause `REFERENCES tbl (C, D)` requires `tbl` to have either the constraint `UNIQUE (C, D)` or `PRIMARY KEY (C, D)`. -- In the `REFERENCES` clause, if you specify a table but no columns, CockroachDB references the table's primary key. In these cases, the `FOREIGN KEY` constraint and the referenced table's primary key must contain the same number of columns. -- By default, referenced columns must be in the same database as the referencing foreign key column. To enable cross-database foreign key references, set the `sql.cross_db_fks.enabled` [cluster setting](cluster-settings.html) to `true`. - -### Null values - -Single-column foreign keys accept null values. - -Multiple-column (composite) foreign keys only accept null values in the following scenarios: - -- The write contains null values for all foreign key columns (if `MATCH FULL` is specified). -- The write contains null values for at least one foreign key column (if `MATCH SIMPLE` is specified). - -For more information about composite foreign keys, see the [composite foreign key matching](#composite-foreign-key-matching) section. - -Note that allowing null values in either your foreign key or referenced columns can degrade their referential integrity, since any key with a null value is never checked against the referenced table. To avoid this, you can use a [`NOT NULL` constraint](not-null.html) on foreign keys when [creating your tables](create-table.html). - -{{site.data.alerts.callout_info}} -A `NOT NULL` constraint cannot be added to existing tables. -{{site.data.alerts.end}} - -### Composite foreign key matching - -By default, composite foreign keys are matched using the `MATCH SIMPLE` algorithm (which is the same default as Postgres). `MATCH FULL` is available if specified. You can specify both `MATCH FULL` and `MATCH SIMPLE`. - -All composite key matches defined prior to version 19.1 use the `MATCH SIMPLE` comparison method. If you had a composite foreign key constraint and have just upgraded to version 19.1, then please check that `MATCH SIMPLE` works for your schema and consider replacing that foreign key constraint with a `MATCH FULL` one. - -#### How it works - -For matching purposes, composite foreign keys can be in one of three states: - -- **Valid**: Keys that can be used for matching foreign key relationships. - -- **Invalid**: Keys that will not be used for matching (including for any cascading operations). - -- **Unacceptable**: Keys that cannot be inserted at all (an error is signalled). - -`MATCH SIMPLE` stipulates that: - -- **Valid** keys may not contain any null values. - -- **Invalid** keys contain one or more null values. - -- **Unacceptable** keys do not exist from the point of view of `MATCH SIMPLE`; all composite keys are acceptable. - -`MATCH FULL` stipulates that: - -- **Valid** keys may not contain any null values. - -- **Invalid** keys must have all null values. - -- **Unacceptable** keys have any combination of both null and non-null values. In other words, `MATCH FULL` requires that if any column of a composite key is `NULL`, then all columns of the key must be `NULL`. - -For examples showing how these key matching algorithms work, see [Match composite foreign keys with `MATCH SIMPLE` and `MATCH FULL`](#match-composite-foreign-keys-with-match-simple-and-match-full). - -{{site.data.alerts.callout_info}} -CockroachDB does not support `MATCH PARTIAL`. For more information, see issue [#20305](https://github.com/cockroachdb/cockroach/issues/20305). -{{site.data.alerts.end}} - -### Foreign key actions - -When you set a foreign key constraint, you can control what happens to the constrained column when the column it's referencing (the foreign key) is deleted or updated. - -Parameter | Description -----------|------------ -`ON DELETE NO ACTION` | _Default action._ If there are any existing references to the key being deleted, the transaction will fail at the end of the statement. The key can be updated, depending on the `ON UPDATE` action.

Alias: `ON DELETE RESTRICT` -`ON UPDATE NO ACTION` | _Default action._ If there are any existing references to the key being updated, the transaction will fail at the end of the statement. The key can be deleted, depending on the `ON DELETE` action.

Alias: `ON UPDATE RESTRICT` -`ON DELETE RESTRICT` / `ON UPDATE RESTRICT` | `RESTRICT` and `NO ACTION` are currently equivalent until options for deferring constraint checking are added. To set an existing foreign key action to `RESTRICT`, the foreign key constraint must be dropped and recreated. -`ON DELETE CASCADE` / `ON UPDATE CASCADE` | When a referenced foreign key is deleted or updated, all rows referencing that key are deleted or updated, respectively. If there are other alterations to the row, such as a `SET NULL` or `SET DEFAULT`, the delete will take precedence.

Note that `CASCADE` does not list objects it drops or updates, so it should be used cautiously. -`ON DELETE SET NULL` / `ON UPDATE SET NULL` | When a referenced foreign key is deleted or updated, respectively, the columns of all rows referencing that key will be set to `NULL`. The column must allow `NULL` or this update will fail. -`ON DELETE SET DEFAULT` / `ON UPDATE SET DEFAULT` | When a referenced foreign key is deleted or updated, the columns of all rows referencing that key are set to the default value for that column.

If the default value for the column is null, or if no default value is provided and the column does not have a [`NOT NULL`](not-null.html) constraint, this will have the same effect as `ON DELETE SET NULL` or `ON UPDATE SET NULL`. The default value must still conform with all other constraints, such as `UNIQUE`. - -{{site.data.alerts.callout_info}} - If a foreign key column has multiple constraints that reference the same column, the foreign key action that is specified by the first foreign key takes precedence. For an example, see [Add multiple foreign key constraints to a single column](#add-multiple-foreign-key-constraints-to-a-single-column). -{{site.data.alerts.end}} - -### Performance - -Because the foreign key constraint requires per-row checks on two tables, statements involving foreign key or referenced columns can take longer to execute. - -To improve query performance, we recommend doing the following: - -- Create a secondary index on all referencing foreign key columns that are not already indexed. - -- For bulk inserts into new tables with foreign key or referenced columns, use the [`IMPORT`](import.html) statement instead of [`INSERT`](insert.html). - - {{site.data.alerts.callout_danger}} - Using [`IMPORT INTO`](import-into.html) will invalidate foreign keys without a [`VALIDATE CONSTRAINT`](validate-constraint.html) statement. - {{site.data.alerts.end}} - -## Syntax - -Foreign key constraints can be defined at the [table level](#table-level). However, if you only want the constraint to apply to a single column, it can be applied at the [column level](#column-level). - -{{site.data.alerts.callout_info}} -You can also add the `FOREIGN KEY` constraint to existing tables through [`ADD CONSTRAINT`](add-constraint.html#add-the-foreign-key-constraint-with-cascade). -{{site.data.alerts.end}} - -### Column level - -
{% include {{ page.version.version }}/sql/generated/diagrams/foreign_key_column_level.html %}
- -| Parameter | Description | -|-----------|-------------| -| `table_name` | The name of the table you're creating. | -| `column_name` | The name of the foreign key column. | -| `column_type` | The foreign key column's [data type](data-types.html). | -| `parent_table` | The name of the table the foreign key references. | -| `ref_column_name` | The name of the column the foreign key references.

If you do not include the `ref_column_name` you want to reference from the `parent_table`, CockroachDB uses the first column of `parent_table`'s primary key. -| `column_constraints` | Any other column-level [constraints](constraints.html) you want to apply to this column. | -| `column_def` | Definitions for any other columns in the table. | -| `table_constraints` | Any table-level [constraints](constraints.html) you want to apply. | - -**Example** - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE IF NOT EXISTS orders ( - id INT PRIMARY KEY, - customer INT NOT NULL REFERENCES customers (id) ON DELETE CASCADE, - orderTotal DECIMAL(9,2), - INDEX (customer) - ); -~~~ -{{site.data.alerts.callout_danger}} -`CASCADE` does not list objects it drops or updates, so it should be used cautiously. -{{site.data.alerts.end}} - -### Table level - -
{% include {{ page.version.version }}/sql/generated/diagrams/foreign_key_table_level.html %}
- -| Parameter | Description | -|-----------|-------------| -| `table_name` | The name of the table you're creating. | -| `column_def` | Definitions for the table's columns. | -| `name` | The name of the constraint. | -| `fk_column_name` | The name of the foreign key column. | -| `parent_table` | The name of the table the foreign key references. | -| `ref_column_name` | The name of the column the foreign key references.

If you do not include the `column_name` you want to reference from the `parent_table`, CockroachDB uses the first column of `parent_table`'s primary key. -| `table_constraints` | Any other table-level [constraints](constraints.html) you want to apply. | - -**Example** - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE packages ( - customer INT, - "order" INT, - id INT, - address STRING(50), - delivered BOOL, - delivery_date DATE, - PRIMARY KEY (customer, "order", id), - CONSTRAINT fk_order FOREIGN KEY (customer, "order") REFERENCES orders - ); -~~~ - -## Usage examples - -### Use a foreign key constraint with default actions - -In this example, we'll create a table with a foreign key constraint with the default [actions](#foreign-key-actions) (`ON UPDATE NO ACTION ON DELETE NO ACTION`). - -First, create the referenced table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE customers (id INT PRIMARY KEY, email STRING UNIQUE); -~~~ - -Next, create the referencing table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE IF NOT EXISTS orders ( - id INT PRIMARY KEY, - customer INT NOT NULL REFERENCES customers (id), - orderTotal DECIMAL(9,2), - INDEX (customer) - ); -~~~ - -Let's insert a record into each table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO customers VALUES (1001, 'a@co.tld'), (1234, 'info@cockroachlabs.com'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO orders VALUES (1, 1002, 29.99); -~~~ -~~~ -pq: foreign key violation: value [1002] not found in customers@primary [id] -~~~ - -The second record insertion returns an error because the customer `1002` doesn't exist in the referenced table. - -Let's insert a record into the referencing table and try to update the referenced table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO orders VALUES (1, 1001, 29.99); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE customers SET id = 1002 WHERE id = 1001; -~~~ -~~~ -pq: foreign key violation: value(s) [1001] in columns [id] referenced in table "orders" -~~~ - -The update to the referenced table returns an error because `id = 1001` is referenced and the default [foreign key action](#foreign-key-actions) is enabled (`ON UPDATE NO ACTION`). However, `id = 1234` is not referenced and can be updated: - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE customers SET id = 1111 WHERE id = 1234; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM customers; -~~~ -~~~ - id | email -+------+------------------------+ - 1001 | a@co.tld - 1111 | info@cockroachlabs.com -(2 rows) -~~~ - -Now let's try to delete a referenced row: - -{% include_cached copy-clipboard.html %} -~~~ sql -> DELETE FROM customers WHERE id = 1001; -~~~ -~~~ -pq: foreign key violation: value(s) [1001] in columns [id] referenced in table "orders" -~~~ - -Similarly, the deletion returns an error because `id = 1001` is referenced and the default [foreign key action](#foreign-key-actions) is enabled (`ON DELETE NO ACTION`). However, `id = 1111` is not referenced and can be deleted: - -{% include_cached copy-clipboard.html %} -~~~ sql -> DELETE FROM customers WHERE id = 1111; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM customers; -~~~ -~~~ - id | email -+------+----------+ - 1001 | a@co.tld -(1 row) -~~~ - -### Use a Foreign Key Constraint with `CASCADE` - -In this example, we'll create a table with a foreign key constraint with the [foreign key actions](#foreign-key-actions) `ON UPDATE CASCADE` and `ON DELETE CASCADE`. - -First, create the referenced table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE customers_2 ( - id INT PRIMARY KEY - ); -~~~ - -Then, create the referencing table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE orders_2 ( - id INT PRIMARY KEY, - customer_id INT REFERENCES customers_2(id) ON UPDATE CASCADE ON DELETE CASCADE - ); -~~~ - -Insert a few records into the referenced table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO customers_2 VALUES (1), (2), (3); -~~~ - -Insert some records into the referencing table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO orders_2 VALUES (100,1), (101,2), (102,3), (103,1); -~~~ - -Now, let's update an `id` in the referenced table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE customers_2 SET id = 23 WHERE id = 1; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM customers_2; -~~~ -~~~ - id -+----+ - 2 - 3 - 23 -(3 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM orders_2; -~~~ -~~~ - id | customer_id -+-----+-------------+ - 100 | 23 - 101 | 2 - 102 | 3 - 103 | 23 -(4 rows) -~~~ - -When `id = 1` was updated to `id = 23` in `customers_2`, the update propagated to the referencing table `orders_2`. - -Similarly, a deletion will cascade. Let's delete `id = 23` from `customers_2`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> DELETE FROM customers_2 WHERE id = 23; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM customers_2; -~~~ -~~~ - id -+----+ - 2 - 3 -(2 rows) -~~~ - -Let's check to make sure the rows in `orders_2` where `customers_id = 23` were also deleted: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM orders_2; -~~~ -~~~ - id | customer_id -+-----+-------------+ - 101 | 2 - 102 | 3 -(2 rows) -~~~ - -### Use a Foreign Key Constraint with `SET NULL` - -In this example, we'll create a table with a foreign key constraint with the [foreign key actions](#foreign-key-actions) `ON UPDATE SET NULL` and `ON DELETE SET NULL`. - -First, create the referenced table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE customers_3 ( - id INT PRIMARY KEY - ); -~~~ - -Then, create the referencing table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE orders_3 ( - id INT PRIMARY KEY, - customer_id INT REFERENCES customers_3(id) ON UPDATE SET NULL ON DELETE SET NULL - ); -~~~ - -Insert a few records into the referenced table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO customers_3 VALUES (1), (2), (3); -~~~ - -Insert some records into the referencing table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO orders_3 VALUES (100,1), (101,2), (102,3), (103,1); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM orders_3; -~~~ -~~~ - id | customer_id -+-----+-------------+ - 100 | 1 - 101 | 2 - 102 | 3 - 103 | 1 -(4 rows) -~~~ - -Now, let's update an `id` in the referenced table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE customers_3 SET id = 23 WHERE id = 1; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM customers_3; -~~~ -~~~ - id -+----+ - 2 - 3 - 23 -(3 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM orders_3; -~~~ -~~~ - id | customer_id -+-----+-------------+ - 100 | NULL - 101 | 2 - 102 | 3 - 103 | NULL -(4 rows) -~~~ - -When `id = 1` was updated to `id = 23` in `customers_3`, the referencing `customer_id` was set to `NULL`. - -Similarly, a deletion will set the referencing `customer_id` to `NULL`. Let's delete `id = 2` from `customers_3`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> DELETE FROM customers_3 WHERE id = 2; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM customers_3; -~~~ -~~~ - id -+----+ - 3 - 23 -(2 rows) -~~~ - -Let's check to make sure the row in `orders_3` where `customers_id = 2` was updated to `NULL`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM orders_3; -~~~ -~~~ - id | customer_id -+-----+-------------+ - 100 | NULL - 101 | NULL - 102 | 3 - 103 | NULL -(4 rows) -~~~ - -### Use a Foreign Key Constraint with `SET DEFAULT` - -In this example, we'll create a table with a `FOREIGN` constraint with the [foreign key actions](#foreign-key-actions) `ON UPDATE SET DEFAULT` and `ON DELETE SET DEFAULT`. - -First, create the referenced table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE customers_4 ( - id INT PRIMARY KEY - ); -~~~ - -Then, create the referencing table with the `DEFAULT` value for `customer_id` set to `9999`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE orders_4 ( - id INT PRIMARY KEY, - customer_id INT DEFAULT 9999 REFERENCES customers_4(id) ON UPDATE SET DEFAULT ON DELETE SET DEFAULT - ); -~~~ - -Insert a few records into the referenced table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO customers_4 VALUES (1), (2), (3), (9999); -~~~ - -Insert some records into the referencing table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO orders_4 VALUES (100,1), (101,2), (102,3), (103,1); -~~~ - - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM orders_4; -~~~ -~~~ - id | customer_id -+-----+-------------+ - 100 | 1 - 101 | 2 - 102 | 3 - 103 | 1 -(4 rows) -~~~ - -Now, let's update an `id` in the referenced table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE customers_4 SET id = 23 WHERE id = 1; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM customers_4; -~~~ -~~~ - id -+------+ - 2 - 3 - 23 - 9999 -(4 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM orders_4; -~~~ -~~~ - id | customer_id -+-----+-------------+ - 100 | 9999 - 101 | 2 - 102 | 3 - 103 | 9999 -(4 rows) -~~~ - -When `id = 1` was updated to `id = 23` in `customers_4`, the referencing `customer_id` was set to `DEFAULT` (i.e., `9999`). You can see this in the first and last rows of `orders_4`, where `id = 100` and the `customer_id` is now `9999` - -Similarly, a deletion will set the referencing `customer_id` to the `DEFAULT` value. Let's delete `id = 2` from `customers_4`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> DELETE FROM customers_4 WHERE id = 2; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM customers_4; -~~~ -~~~ - id -+------+ - 3 - 23 - 9999 -(3 rows) -~~~ - -Let's check to make sure the corresponding `customer_id` value to `id = 101`, was updated to the `DEFAULT` value (i.e., `9999`) in `orders_4`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM orders_4; -~~~ -~~~ - id | customer_id -+-----+-------------+ - 100 | 9999 - 101 | 9999 - 102 | 3 - 103 | 9999 -(4 rows) -~~~ - -If the default value for the `customer_id` column is not set, and the column does not have a [`NOT NULL`](not-null.html) constraint, `ON UPDATE SET DEFAULT` and `ON DELETE SET DEFAULT` actions set referenced column values to `NULL`. - -For example, let's create a new `customers_5` table and insert some values: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE customers_5 ( - id INT PRIMARY KEY - ); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO customers_5 VALUES (1), (2), (3), (4); -~~~ - -Then we can create a new `orders_5` table that references the `customers_5` table, but with no default value specified for the `ON UPDATE SET DEFAULT` and `ON DELETE SET DEFAULT` actions: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE orders_5 ( - id INT PRIMARY KEY, - customer_id INT REFERENCES customers_5(id) ON UPDATE SET DEFAULT ON DELETE SET DEFAULT - ); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO orders_5 VALUES (200,1), (201,2), (202,3), (203,4); -~~~ - -Deleting and updating values in the `customers_5` table sets the referenced values in `orders_5` to `NULL`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> DELETE FROM customers_5 WHERE id = 3; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE customers_5 SET id = 0 WHERE id = 1; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM orders_5; -~~~ -~~~ - id | customer_id -+-----+-------------+ - 200 | NULL - 201 | 2 - 202 | NULL - 203 | 4 -(4 rows) -~~~ - -### Add multiple foreign key constraints to a single column - - You can add more than one foreign key constraint to a single column. - -For example, if you create the following tables: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE customers ( - id INT PRIMARY KEY, - name STRING, - email STRING -); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE orders ( - id INT PRIMARY KEY, - customer_id INT UNIQUE, - item_number INT - ); -~~~ - -You can create a table with a column that references columns in both the `customers` and `orders` tables: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE shipments ( - tracking_number UUID DEFAULT gen_random_uuid() PRIMARY KEY, - carrier STRING, - status STRING, - customer_id INT, - CONSTRAINT fk_customers FOREIGN KEY (customer_id) REFERENCES customers(id), - CONSTRAINT fk_orders FOREIGN KEY (customer_id) REFERENCES orders(customer_id) - ); -~~~ - -Inserts into the `shipments` table must fulfill both foreign key constraints on `customer_id` (`fk_customers` and `fk_customers_2`). - -Let's insert a record into each table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO customers VALUES (1001, 'Alexa', 'a@co.tld'), (1234, 'Evan', 'info@cockroachlabs.com'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO orders VALUES (1, 1001, 25), (2, 1234, 15), (3, 2000, 5); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO shipments (carrier, status, customer_id) VALUES ('USPS', 'Out for delivery', 1001); -~~~ - -The last statement succeeds because `1001` matches a unique `id` value in the `customers` table and a unique `customer_id` value in the `orders` table. If `1001` was in neither of the referenced columns, or in just one of them, the statement would return an error. - -For instance, the following statement fulfills just one of the foreign key constraints and returns an error: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO shipments (carrier, status, customer_id) VALUES ('DHL', 'At facility', 2000); -~~~ - -~~~ -ERROR: insert on table "shipments" violates foreign key constraint "fk_customers" -SQLSTATE: 23503 -DETAIL: Key (customer_id)=(2000) is not present in table "customers". -~~~ - -CockroachDB allows you to add multiple foreign key constraints on the same column, that reference the same column: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE shipments ADD CONSTRAINT fk_customers_2 FOREIGN KEY (customer_id) REFERENCES customers(id) ON DELETE CASCADE; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CONSTRAINTS FROM shipments; -~~~ - -~~~ - table_name | constraint_name | constraint_type | details | validated --------------+-----------------+-----------------+----------------------------------------------------------------------+------------ - shipments | fk_customers | FOREIGN KEY | FOREIGN KEY (customer_id) REFERENCES customers(id) | true - shipments | fk_customers_2 | FOREIGN KEY | FOREIGN KEY (customer_id) REFERENCES customers(id) ON DELETE CASCADE | true - shipments | fk_orders | FOREIGN KEY | FOREIGN KEY (customer_id) REFERENCES orders(customer_id) | true - shipments | primary | PRIMARY KEY | PRIMARY KEY (tracking_number ASC) | true -(4 rows) -~~~ - -There are now two foreign key constraints on `customer_id` that reference the `customers(id)` column (i.e., `fk_customers` and `fk_customers_2`). - -In the event of a `DELETE` or `UPDATE` to the referenced column (`customers(id)`), the action for the first foreign key specified takes precedence. In this case, that will be the default [action](#foreign-key-actions) (`ON UPDATE NO ACTION ON DELETE NO ACTION`) on the first foreign key constraint (`fk_customers`). This means that `DELETE`s on referenced columns will fail, even though the second foreign key constraint (`fk_customer_2`) is defined with the `ON DELETE CASCADE` action. - -{% include_cached copy-clipboard.html %} -~~~ sql -> DELETE FROM orders WHERE customer_id = 1001; -~~~ - -~~~ -ERROR: delete on table "orders" violates foreign key constraint "fk_orders" on table "shipments" -SQLSTATE: 23503 -DETAIL: Key (customer_id)=(1001) is still referenced from table "shipments". -~~~ - -### Match composite foreign keys with `MATCH SIMPLE` and `MATCH FULL` - -The examples in this section show how composite foreign key matching works for both the `MATCH SIMPLE` and `MATCH FULL` algorithms. For a conceptual overview, see [Composite foreign key matching](#composite-foreign-key-matching). - -First, let's create some tables. `parent` is a table with a composite key: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE parent (x INT, y INT, z INT, UNIQUE (x, y, z)); -~~~ - -`full_test` has a foreign key on `parent` that uses the `MATCH FULL` algorithm: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE full_test ( - x INT, - y INT, - z INT, - FOREIGN KEY (x, y, z) REFERENCES parent (x, y, z) MATCH FULL ON DELETE CASCADE ON UPDATE CASCADE - ); -~~~ - -`simple_test` has a foreign key on `parent` that uses the `MATCH SIMPLE` algorithm (the default): - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE simple_test ( - x INT, - y INT, - z INT, - FOREIGN KEY (x, y, z) REFERENCES parent (x, y, z) ON DELETE CASCADE ON UPDATE CASCADE - ); -~~~ - -Next, we populate `parent` with some values: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT - INTO parent - VALUES (1, 1, 1), - (2, 1, 1), - (1, 2, 1), - (1, 1, 2), - (NULL, NULL, NULL), - (1, NULL, NULL), - (NULL, 1, NULL), - (NULL, NULL, 1), - (1, 1, NULL), - (1, NULL, 1), - (NULL, 1, 1); -~~~ - -Now let's look at some `INSERT` statements to see how the different key matching algorithms work. - -- [MATCH SIMPLE](#match-simple) -- [MATCH FULL](#match-full) - -#### MATCH SIMPLE - -Inserting values into the table using the `MATCH SIMPLE` algorithm (described [above](#composite-foreign-key-matching)) gives the following results: - -| Statement | Can insert? | Throws error? | Notes | -|---------------------------------------------------+-------------+---------------+-------------------------------| -| `INSERT INTO simple_test VALUES (1,1,1)` | Yes | No | References `parent (1,1,1)`. | -| `INSERT INTO simple_test VALUES (NULL,NULL,NULL)` | Yes | No | Does not reference `parent`. | -| `INSERT INTO simple_test VALUES (1,NULL,NULL)` | Yes | No | Does not reference `parent`. | -| `INSERT INTO simple_test VALUES (NULL,1,NULL)` | Yes | No | Does not reference `parent`. | -| `INSERT INTO simple_test VALUES (NULL,NULL,1)` | Yes | No | Does not reference `parent`. | -| `INSERT INTO simple_test VALUES (1,1,NULL)` | Yes | No | Does not reference `parent`. | -| `INSERT INTO simple_test VALUES (1,NULL,1)` | Yes | No | Does not reference `parent`. | -| `INSERT INTO simple_test VALUES (NULL,1,1)` | Yes | No | Does not reference `parent`. | -| `INSERT INTO simple_test VALUES (2,2,NULL)` | Yes | No | Does not reference `parent`. | -| `INSERT INTO simple_test VALUES (2,2,2)` | No | Yes | No `parent` reference exists. | - -#### MATCH FULL - -Inserting values into the table using the `MATCH FULL` algorithm (described [above](#composite-foreign-key-matching)) gives the following results: - -| Statement | Can insert? | Throws error? | Notes | -|-------------------------------------------------+-------------+---------------+-----------------------------------------------------| -| `INSERT INTO full_test VALUES (1,1,1)` | Yes | No | References `parent(1,1,1)`. | -| `INSERT INTO full_test VALUES (NULL,NULL,NULL)` | Yes | No | Does not reference `parent`. | -| `INSERT INTO full_test VALUES (1,NULL,NULL)` | No | Yes | Can't mix null and non-null values in `MATCH FULL`. | -| `INSERT INTO full_test VALUES (NULL,1,NULL)` | No | Yes | Can't mix null and non-null values in `MATCH FULL`. | -| `INSERT INTO full_test VALUES (NULL,NULL,1)` | No | Yes | Can't mix null and non-null values in `MATCH FULL`. | -| `INSERT INTO full_test VALUES (1,1,NULL)` | No | Yes | Can't mix null and non-null values in `MATCH FULL`. | -| `INSERT INTO full_test VALUES (1,NULL,1)` | No | Yes | Can't mix null and non-null values in `MATCH FULL`. | -| `INSERT INTO full_test VALUES (NULL,1,1)` | No | Yes | Can't mix null and non-null values in `MATCH FULL`. | -| `INSERT INTO full_test VALUES (2,2,NULL)` | No | Yes | Can't mix null and non-null values in `MATCH FULL`. | -| `INSERT INTO full_test VALUES (2,2,2)` | No | Yes | No `parent` reference exists. | - -## See also - -- [Constraints](constraints.html) -- [`DROP CONSTRAINT`](drop-constraint.html) -- [`ADD CONSTRAINT`](add-constraint.html) -- [`CHECK` constraint](check.html) -- [`DEFAULT` constraint](default-value.html) -- [`NOT NULL` constraint](not-null.html) -- [`PRIMARY KEY` constraint](primary-key.html) -- [`UNIQUE` constraint](unique.html) -- [`SHOW CONSTRAINTS`](show-constraints.html) -- [What is a Foreign Key? (With SQL Examples)](https://www.cockroachlabs.com/blog/what-is-a-foreign-key/) diff --git a/src/current/v21.1/frequently-asked-questions.md b/src/current/v21.1/frequently-asked-questions.md deleted file mode 100644 index 8d8d4fec4bd..00000000000 --- a/src/current/v21.1/frequently-asked-questions.md +++ /dev/null @@ -1,186 +0,0 @@ ---- -title: CockroachDB FAQs -summary: CockroachDB frequently asked questions - What is CockroachDB? How does it work? What makes it different from other databases? -tags: postgres, cassandra, google cloud spanner -toc: true ---- - -## What is CockroachDB? - -CockroachDB is a [distributed SQL](https://www.cockroachlabs.com/blog/what-is-distributed-sql/) database built on a transactional and strongly-consistent key-value store. It **scales** horizontally; **survives** disk, machine, rack, and even datacenter failures with minimal latency disruption and no manual intervention; supports **strongly-consistent** ACID transactions; and provides a familiar **SQL** API for structuring, manipulating, and querying data. - -CockroachDB is inspired by Google's [Spanner](http://research.google.com/archive/spanner.html) and [F1](http://research.google.com/pubs/pub38125.html) technologies, and the [source code](https://github.com/cockroachdb/cockroach) is freely available. - -## When is CockroachDB a good choice? - -CockroachDB is well suited for applications that require reliable, available, and correct data, and millisecond response times, regardless of scale. It is built to automatically replicate, rebalance, and recover with minimal configuration and operational overhead. Specific use cases include: - -- Distributed or replicated OLTP -- Multi-datacenter deployments -- Multi-region deployments -- Cloud migrations -- Infrastructure initiatives built for the cloud - - - -CockroachDB returns single-row reads in 2ms or less and single-row writes in 4ms or less, and supports a variety of [SQL and operational tuning practices](performance-best-practices-overview.html) for optimizing query performance. However, CockroachDB is not yet suitable for heavy analytics / OLAP. - -## How easy is it to install CockroachDB? - -It's as easy as downloading a binary or running our official Kubernetes configurations or Docker image. There are other simple install methods as well, such as running our Homebrew recipe on OS X or building from source files on both OS X and Linux. - -For more details, see [Install CockroachDB](install-cockroachdb.html). - -## How does CockroachDB scale? - -CockroachDB scales horizontally with minimal operator overhead. You can run it on your local computer, a single server, a corporate development cluster, or a private or public cloud. [Adding capacity](cockroach-start.html) is as easy as pointing a new node at the running cluster. - -At the key-value level, CockroachDB starts off with a single, empty range. As you put data in, this single range eventually reaches a threshold size (512 MiB by default). When that happens, the data splits into two ranges, each covering a contiguous segment of the entire key-value space. This process continues indefinitely; as new data flows in, existing ranges continue to split into new ranges, aiming to keep a relatively small and consistent range size. - -When your cluster spans multiple nodes (physical machines, virtual machines, or containers), newly split ranges are automatically rebalanced to nodes with more capacity. CockroachDB communicates opportunities for rebalancing using a peer-to-peer [gossip protocol](https://en.wikipedia.org/wiki/Gossip_protocol) by which nodes exchange network addresses, store capacity, and other information. - -## How does CockroachDB survive failures? - -CockroachDB is designed to survive software and hardware failures, from server restarts to datacenter outages. This is accomplished without confusing artifacts typical of other distributed systems (e.g., stale reads) using strongly-consistent replication as well as automated repair after failures. - -**Replication** - -CockroachDB replicates your data for availability and guarantees consistency between replicas using the [Raft consensus algorithm](https://raft.github.io/), a popular alternative to Paxos. You can [define the location of replicas](configure-replication-zones.html) in various ways, depending on the types of failures you want to secure against and your network topology. You can locate replicas on: - -- Different servers within a rack to tolerate server failures -- Different servers on different racks within a datacenter to tolerate rack power/network failures -- Different servers in different datacenters to tolerate large scale network or power outages - -In a CockroachDB cluster spread across multiple geographic regions, the round-trip latency between regions will have a direct effect on your database's performance. In such cases, it is important to think about the latency requirements of each table and then use the appropriate [data topologies](topology-patterns.html) to locate data for optimal performance and resiliency. For a step-by-step demonstration, see [Low Latency Multi-Region Deployment](demo-low-latency-multi-region-deployment.html). - -**Automated Repair** - -For short-term failures, such as a server restart, CockroachDB uses Raft to continue seamlessly as long as a majority of replicas remain available. Raft makes sure that a new “leader” for each group of replicas is elected if the former leader fails, so that transactions can continue and affected replicas can rejoin their group once they’re back online. For longer-term failures, such as a server/rack going down for an extended period of time or a datacenter outage, CockroachDB automatically rebalances replicas from the missing nodes, using the unaffected replicas as sources. Using capacity information from the gossip network, new locations in the cluster are identified and the missing replicas are re-replicated in a distributed fashion using all available nodes and the aggregate disk and network bandwidth of the cluster. - -## How is CockroachDB strongly-consistent? - -CockroachDB guarantees [serializable SQL transactions](demo-serializable.html), the highest isolation level defined by the SQL standard. It does so by combining the Raft consensus algorithm for writes and a custom time-based synchronization algorithms for reads. - -- Stored data is versioned with MVCC, so [reads simply limit their scope to the data visible at the time the read transaction started](architecture/transaction-layer.html#time-and-hybrid-logical-clocks). - -- Writes are serviced using the [Raft consensus algorithm](https://raft.github.io/), a popular alternative to Paxos. A consensus algorithm guarantees that any majority of replicas together always agree on whether an update was committed successfully. Updates (writes) must reach a majority of replicas (2 out of 3 by default) before they are considered committed. - - To ensure that a write transaction does not interfere with read transactions that start after it, CockroachDB also uses a [timestamp cache](architecture/transaction-layer.html#timestamp-cache) which remembers when data was last read by ongoing transactions. - - This ensures that clients always observe serializable consistency with regards to other concurrent transactions. - -## How is CockroachDB both highly available and strongly consistent? - -The [CAP theorem](https://en.wikipedia.org/wiki/CAP_theorem) states that it is impossible for a distributed system to simultaneously provide more than two out of the following three guarantees: - -- Consistency -- Availability -- Partition Tolerance - -CockroachDB is a CP (consistent and partition tolerant) system. This means -that, in the presence of partitions, the system will become unavailable rather than do anything which might cause inconsistent results. For example, writes require acknowledgements from a majority of replicas, and reads require a lease, which can only be transferred to a different node when writes are possible. - -Separately, CockroachDB is also Highly Available, although "available" here means something different than the way it is used in the CAP theorem. In the CAP theorem, availability is a binary property, but for High Availability, we talk about availability as a spectrum (using terms like "five nines" for a system that is available 99.999% of the time). - -Being both CP and HA means that whenever a majority of replicas can talk to each other, they should be able to make progress. For example, if you deploy CockroachDB to three datacenters and the network link to one of them fails, the other two datacenters should be able to operate normally with only a few seconds' disruption. We do this by attempting to detect partitions and failures quickly and efficiently, transferring leadership to nodes that are able to communicate with the majority, and routing internal traffic away from nodes that are partitioned away. - -## Why is CockroachDB SQL? - -At the lowest level, CockroachDB is a distributed, strongly-consistent, transactional key-value store, but the external API is Standard SQL with extensions. This provides developers familiar relational concepts such as schemas, tables, columns, and indexes and the ability to structure, manipulate, and query data using well-established and time-proven tools and processes. Also, since CockroachDB supports the PostgreSQL wire protocol, it’s simple to get your application talking to Cockroach; just find your [PostgreSQL language-specific driver](install-client-drivers.html) and start building. - -For more details, learn our [basic CockroachDB SQL statements](learn-cockroachdb-sql.html), explore the [full SQL grammar](sql-grammar.html), and try it out via our [built-in SQL client](cockroach-sql.html). Also, to understand how CockroachDB maps SQL table data to key-value storage and how CockroachDB chooses the best index for running a query, see [SQL in CockroachDB](https://www.cockroachlabs.com/blog/sql-in-cockroachdb-mapping-table-data-to-key-value-storage/) and [Index Selection in CockroachDB](https://www.cockroachlabs.com/blog/index-selection-cockroachdb-2/). - -## Does CockroachDB support distributed transactions? - -Yes. CockroachDB distributes transactions across your cluster, whether it’s a few servers in a single location or many servers across multiple datacenters. Unlike with sharded setups, you do not need to know the precise location of data; you just talk to any node in your cluster and CockroachDB gets your transaction to the right place seamlessly. Distributed transactions proceed without downtime or additional latency while rebalancing is underway. You can even move tables – or entire databases – between data centers or cloud infrastructure providers while the cluster is under load. - -## Do transactions in CockroachDB guarantee ACID semantics? - -Yes. Every [transaction](transactions.html) in CockroachDB guarantees [ACID semantics](https://en.wikipedia.org/wiki/ACID) spanning arbitrary tables and rows, even when data is distributed. - -- **Atomicity:** Transactions in CockroachDB are “all or nothing.” If any part of a transaction fails, the entire transaction is aborted, and the database is left unchanged. If a transaction succeeds, all mutations are applied together with virtual simultaneity. For a detailed discussion of atomicity in CockroachDB transactions, see [How CockroachDB Distributes Atomic Transactions](https://www.cockroachlabs.com/blog/how-cockroachdb-distributes-atomic-transactions/). -- **Consistency:** SQL operations never see any intermediate states and move the database from one valid state to another, keeping indexes up to date. Operations always see the results of previously completed statements on overlapping data and maintain specified constraints such as unique columns. For a detailed look at how we've tested CockroachDB for correctness and consistency, see [CockroachDB Beta Passes Jepsen Testing](https://www.cockroachlabs.com/blog/cockroachdb-beta-passes-jepsen-testing/). -- **Isolation:** Transactions in CockroachDB implement the strongest ANSI isolation level: serializable (`SERIALIZABLE`). This means that transactions will never result in anomalies. For more information about transaction isolation in CockroachDB, see [Transactions: Isolation Levels](transactions.html#isolation-levels). -- **Durability:** In CockroachDB, every acknowledged write has been persisted consistently on a majority of replicas (by default, at least 2) via the [Raft consensus algorithm](https://raft.github.io/). Power or disk failures that affect only a minority of replicas (typically 1) do not prevent the cluster from operating and do not lose any data. - -## Since CockroachDB is inspired by Spanner, does it require atomic clocks to synchronize time? - -No. CockroachDB was designed to work without atomic clocks or GPS clocks. It’s a database intended to be run on arbitrary collections of nodes, from physical servers in a corp development cluster to public cloud infrastructure using the flavor-of-the-month virtualization layer. It’d be a showstopper to require an external dependency on specialized hardware for clock synchronization. However, CockroachDB does require moderate levels of clock synchronization for correctness. If clocks drift past a maximum threshold, nodes will be taken offline. It's therefore highly recommended to run [NTP](http://www.ntp.org/) or other clock synchronization software on each node. - -For more details on how CockroachDB handles unsynchronized clocks, see [Clock Synchronization](recommended-production-settings.html#clock-synchronization). And for a broader discussion of clocks, and the differences between clocks in Spanner and CockroachDB, see [Living Without Atomic Clocks](https://www.cockroachlabs.com/blog/living-without-atomic-clocks/). - -## What languages can I use to work with CockroachDB? - -CockroachDB supports the PostgreSQL wire protocol, so you can use any available PostgreSQL client drivers. We've tested it from the following languages: - -- Go -- Python -- Ruby -- Java -- JavaScript (node.js) -- C++/C -- Clojure -- PHP -- Rust - -See [Install Client Drivers](install-client-drivers.html) for more details. - -## Why does CockroachDB use the PostgreSQL wire protocol instead of the MySQL protocol? - -CockroachDB uses the PostgreSQL wire protocol because it is better documented than the MySQL protocol, and because PostgreSQL has a liberal Open Source license, similar to BSD or MIT licenses, whereas MySQL has the more restrictive GNU General Public License. - -Note, however, that the protocol used doesn't significantly impact how easy it is to port applications. Swapping out SQL network drivers is rather straightforward in nearly every language. What makes it hard to move from one database to another is the dialect of SQL in use. CockroachDB's dialect is based on PostgreSQL as well. - -## What is CockroachDB’s security model? - -You can run a secure or insecure CockroachDB cluster. When secure, client/node and inter-node communication is encrypted, and SSL certificates authenticate the identity of both clients and nodes. When insecure, there's no encryption or authentication. - -Also, CockroachDB supports common SQL privileges on databases and tables. The `root` user has privileges for all databases, while unique users can be granted privileges for specific statements at the database and table-levels. - -For more details, see our [Security Overview](security-overview.html). - -## How does CockroachDB compare to MySQL or PostgreSQL? - -While all of these databases support SQL syntax, CockroachDB is the only one that scales easily (without the manual complexity of sharding), rebalances and repairs itself automatically, and distributes transactions seamlessly across your cluster. - -For more insight, see [CockroachDB in Comparison](cockroachdb-in-comparison.html). - -## How does CockroachDB compare to Cassandra, HBase, MongoDB, or Riak? - -While all of these are distributed databases, only CockroachDB supports distributed transactions and provides strong consistency. Also, these other databases provide custom APIs, whereas CockroachDB offers standard SQL with extensions. - -For more insight, see [CockroachDB in Comparison](cockroachdb-in-comparison.html). - -## Can a PostgreSQL or MySQL application be migrated to CockroachDB? - -Yes. Most users should be able to follow the instructions in [Migrate from Postgres](migrate-from-postgres.html) or [Migrate from MySQL](migrate-from-mysql.html). Due to differences in available features and syntax, some features supported by these databases may require manual effort to port to CockroachDB. Check those pages for details. - -We also fully support [importing your data via CSV](migrate-from-csv.html). - -## Does Cockroach Labs offer a cloud database as a service? - -Yes. The CockroachDB {{ site.data.products.dedicated }} offering is currently in Limited Availability and accepting customers on a qualified basis. The offering provides a running CockroachDB cluster suitable to your needs, fully managed by Cockroach Labs on GCP or AWS. Benefits include: - -- No provisioning or deployment efforts for you -- Daily full backups and hourly incremental backups of your data -- Upgrades to the latest stable release of CockroachDB -- Monitoring to provide SLA-level support - -For more details, see the [CockroachDB {{ site.data.products.dedicated }}](../cockroachcloud/quickstart-trial-cluster.html) docs. - -## Why did Cockroach Labs change the license for CockroachDB? - -Our past outlook on the right business model relied on a crucial norm in the OSS world: that companies could build a business around a strong open source core product without a much larger technology platform company coming along and offering the same product as a service. - -Recently, however, OSS companies have seen the rise of highly-integrated providers take advantage of their unique position to offer “as-a-service” versions of OSS products, and offer a superior user experience as a consequence of their integrations. We’ve most recently seen it happen with Amazon’s forked version of ElasticSearch. - -To respond to this breed of competitor, we changed our software licensing terms. To learn more about our motivations, see the [Licensing FAQs](licensing-faqs.html) as well as our [blog post](https://www.cockroachlabs.com/blog/oss-relicensing-cockroachdb/) about the license change. - -## Have questions that weren’t answered? - -Try searching the rest of our docs for answers or using our other [support resources](support-resources.html), including: - -- [CockroachDB Community Forum](https://forum.cockroachlabs.com) -- [CockroachDB Community Slack](https://cockroachdb.slack.com) -- [StackOverflow](http://stackoverflow.com/questions/tagged/cockroachdb) -- [CockroachDB Support Portal](https://support.cockroachlabs.com) diff --git a/src/current/v21.1/functions-and-operators.md b/src/current/v21.1/functions-and-operators.md deleted file mode 100644 index d765e8bed2d..00000000000 --- a/src/current/v21.1/functions-and-operators.md +++ /dev/null @@ -1,123 +0,0 @@ ---- -title: Functions and Operators -summary: CockroachDB supports many built-in functions, aggregate functions, and operators. -toc: true ---- - -CockroachDB supports the following SQL functions and operators for use in [scalar expressions](scalar-expressions.html). - -{{site.data.alerts.callout_success}}In the built-in SQL shell, use \hf [function] to get inline help about a specific function.{{site.data.alerts.end}} - -## Special syntax forms - -The following syntax forms are recognized for compatibility with the -SQL standard and PostgreSQL, but are equivalent to regular built-in -functions: - -{% include {{ page.version.version }}/sql/function-special-forms.md %} - -## Conditional and function-like operators - -The following table lists the operators that look like built-in -functions but have special evaluation rules: - - Operator | Description -----------|------------- - `ANNOTATE_TYPE(...)` | [Explicitly Typed Expression](scalar-expressions.html#explicitly-typed-expressions) - `ARRAY(...)` | [Conversion of Subquery Results to An Array](scalar-expressions.html#conversion-of-subquery-results-to-an-array) - `ARRAY[...]` | [Conversion of Scalar Expressions to An Array](scalar-expressions.html#array-constructors) - `CAST(...)` | [Type Cast](scalar-expressions.html#explicit-type-coercions) - `COALESCE(...)` | [First non-NULL expression with Short Circuit](scalar-expressions.html#coalesce-and-ifnull-expressions) - `EXISTS(...)` | [Existence Test on the Result of Subqueries](scalar-expressions.html#existence-test-on-the-result-of-subqueries) - `IF(...)` | [Conditional Evaluation](scalar-expressions.html#if-expressions) - `IFNULL(...)` | Alias for `COALESCE` restricted to two operands - `NULLIF(...)` | [Return `NULL` conditionally](scalar-expressions.html#nullif-expressions) - `ROW(...)` | [Tuple Constructor](scalar-expressions.html#tuple-constructor) - -## Built-in functions - -{% remote_include https://raw.githubusercontent.com/cockroachdb/cockroach/{{ page.release_info.crdb_branch_name }}/docs/generated/sql/functions.md %} - -## Aggregate functions - -For examples showing how to use aggregate functions, see [the `SELECT` clause documentation](select-clause.html#aggregate-functions). - -{{site.data.alerts.callout_info}} -Non-commutative aggregate functions are sensitive to the order in which the rows are processed in the surrounding [`SELECT` clause](select-clause.html#aggregate-functions). To specify the order in which input rows are processed, you can add an [`ORDER BY`](order-by.html) clause within the function argument list. For examples, see the [`SELECT` clause](select-clause.html#order-aggregate-function-input-rows-by-column) documentation. -{{site.data.alerts.end}} - -{% remote_include https://raw.githubusercontent.com/cockroachdb/cockroach/{{ page.release_info.crdb_branch_name }}/docs/generated/sql/aggregates.md %} - -## Window functions - -{% remote_include https://raw.githubusercontent.com/cockroachdb/cockroach/{{ page.release_info.crdb_branch_name }}/docs/generated/sql/window_functions.md %} - -## Operators - -The following table lists all CockroachDB operators from highest to lowest precedence, i.e., the order in which they will be evaluated within a statement. Operators with the same precedence are left associative. This means that those operators are grouped together starting from the left and moving right. - -| Order of Precedence | Operator | Name | Operator Arity | -| ------------------- | -------- | ---- | -------------- | -| 1 | `.` | Member field access operator | binary | -| 2 | `::` | [Type cast](scalar-expressions.html#explicit-type-coercions) | binary | -| 3 | `-` | Unary minus | unary (prefix) | -| | `~` | Bitwise not | unary (prefix) | -| 4 | `^` | Exponentiation | binary | -| 5 | `*` | Multiplication | binary | -| | `/` | Division | binary | -| | `//` | Floor division | binary | -| | `%` | Modulo | binary | -| 6 | `+` | Addition | binary | -| | `-` | Subtraction | binary | -| 7 | `<<` | Bitwise left-shift | binary | -| | `>>` | Bitwise right-shift | binary | -| 8 | `&` | Bitwise AND | binary | -| 9 | `#` | Bitwise XOR | binary | -| 10 | | | Bitwise OR | binary | -| 11 | || | Concatenation | binary | -| | `< ANY`, ` SOME`, ` ALL` | [Multi-valued] "less than" comparison | binary | -| | `> ANY`, ` SOME`, ` ALL` | [Multi-valued] "greater than" comparison | binary | -| | `= ANY`, ` SOME`, ` ALL` | [Multi-valued] "equal" comparison | binary | -| | `<= ANY`, ` SOME`, ` ALL` | [Multi-valued] "less than or equal" comparison | binary | -| | `>= ANY`, ` SOME`, ` ALL` | [Multi-valued] "greater than or equal" comparison | binary | -| | `<> ANY` / `!= ANY`, `<> SOME` / `!= SOME`, `<> ALL` / `!= ALL` | [Multi-valued] "not equal" comparison | binary | -| | `[NOT] LIKE ANY`, `[NOT] LIKE SOME`, `[NOT] LIKE ALL` | [Multi-valued] `LIKE` comparison | binary | -| | `[NOT] ILIKE ANY`, `[NOT] ILIKE SOME`, `[NOT] ILIKE ALL` | [Multi-valued] `ILIKE` comparison | binary | -| 12 | `[NOT] BETWEEN` | Value is [not] within the range specified | binary | -| | `[NOT] BETWEEN SYMMETRIC` | Like `[NOT] BETWEEN`, but in non-sorted order. For example, whereas `a BETWEEN b AND c` means `b <= a <= c`, `a BETWEEN SYMMETRIC b AND c` means `(b <= a <= c) OR (c <= a <= b)`. | binary | -| | `[NOT] IN` | Value is [not] in the set of values specified | binary | -| | `[NOT] LIKE` | Matches [or not] LIKE expression, case sensitive | binary | -| | `[NOT] ILIKE` | Matches [or not] LIKE expression, case insensitive | binary | -| | `[NOT] SIMILAR` | Matches [or not] SIMILAR TO regular expression | binary | -| | `~` | Matches regular expression, case sensitive | binary | -| | `!~` | Does not match regular expression, case sensitive | binary | -| | `~*` | Matches regular expression, case insensitive | binary | -| | `!~*` | Does not match regular expression, case insensitive | binary | -| 13 | `=` | Equal | binary | -| | `<` | Less than | binary | -| | `>` | Greater than | binary | -| | `<=` | Less than or equal to | binary | -| | `>=` | Greater than or equal to | binary | -| | `!=`, `<>` | Not equal | binary | -| 14 | `IS [DISTINCT FROM]` | Equal, considering `NULL` as value | binary | -| | `IS NOT [DISTINCT FROM]` | `a IS NOT b` equivalent to `NOT (a IS b)` | binary | -| | `ISNULL`, `IS UNKNOWN` , `NOTNULL`, `IS NOT UNKNOWN` | Equivalent to `IS NULL` / `IS NOT NULL` | unary (postfix) | -| | `IS NAN`, `IS NOT NAN` | [Comparison with the floating-point NaN value](scalar-expressions.html#comparison-with-nan) | unary (postfix) | -| | `IS OF(...)` | Type predicate | unary (postfix) -| 15 | `NOT` | [Logical NOT](scalar-expressions.html#logical-operators) | unary | -| 16 | `AND` | [Logical AND](scalar-expressions.html#logical-operators) | binary | -| 17 | `OR` | [Logical OR](scalar-expressions.html#logical-operators) | binary | - -[Multi-valued]: scalar-expressions.html#multi-valued-comparisons - -### Supported operations - -{% remote_include https://raw.githubusercontent.com/cockroachdb/cockroach/{{ page.release_info.crdb_branch_name }}/docs/generated/sql/operators.md %} - - diff --git a/src/current/v21.1/geojson.md b/src/current/v21.1/geojson.md deleted file mode 100644 index 6c35b2144ad..00000000000 --- a/src/current/v21.1/geojson.md +++ /dev/null @@ -1,203 +0,0 @@ ---- -title: GeoJSON -summary: The GeoJSON data format for representing spatial information is based on JavaScript Object Notation (JSON). -toc: true ---- - -GeoJSON is a textual data format for representing spatial information. It is based on [JavaScript Object Notation (JSON)](https://www.json.org). - -GeoJSON can be used to represent the following spatial objects, which also have [Well Known Text (WKT)](well-known-text.html) and [Well Known Binary (WKB)](well-known-binary.html) representations: - -- Point -- LineString -- Polygon -- MultiPoint -- MultiLineString -- MultiPolygon -- GeometryCollection - -GeoJSON introduces the following additional concepts, which are not part of WKT or WKB: - -- A "Feature" object that can contain a geometric shape and some additional properties that describe that shape. This is useful, for example, when drawing maps on the internet in color, such as on [geojson.io](http://geojson.io). For an example showing how to add color to a GeoJSON feature, [see below](#geojson-features-example). -- Features can additionally be grouped together into a "FeatureCollection". - -{{site.data.alerts.callout_success}} -For more detailed information, see the [GeoJSON RFC](https://www.rfc-editor.org/rfc/rfc7946.txt). -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -GeoJSON should only be used for spatial data that uses the [WGS84](spatial-glossary.html) geographic spatial reference system. For more information, see [SRID 4326](srid-4326.html). -{{site.data.alerts.end}} - -## Example - -In the example below, we will convert a shape represented in [Well Known Text](well-known-text.html) to GeoJSON using the `ST_AsGeoJSON` [function](functions-and-operators.html#spatial-functions). - -Here is the WKT: - -~~~ -SRID=4326;POLYGON((-87.906471 43.038902, -95.992775 36.153980, -75.704722 36.076944, -87.906471 43.038902), (-87.623177 41.881832, -90.199402 38.627003, -82.446732 38.413651, -87.623177 41.881832)) -~~~ - -Convert it to GeoJSON using the `ST_AsGeoJSON` function: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT ST_AsGeoJSON('SRID=4326;POLYGON((-87.906471 43.038902, -95.992775 36.153980, -75.704722 36.076944, -87.906471 43.038902), (-87.623177 41.881832, -90.199402 38.627003, -82.446732 38.413651, -87.623177 41.881832))'); -~~~ - -This is the JSON output of the above, but formatted: - -{% include_cached copy-clipboard.html %} -~~~ json -{ - "type": "Polygon", - "coordinates": [ - [ - [ - -87.906471, - 43.038902 - ], - [ - -95.992775, - 36.15398 - ], - [ - -75.704722, - 36.076944 - ], - [ - -87.906471, - 43.038902 - ] - ], - [ - [ - -87.623177, - 41.881832 - ], - [ - -90.199402, - 38.627003 - ], - [ - -82.446732, - 38.413651 - ], - [ - -87.623177, - 41.881832 - ] - ] - ] -} -~~~ - - - -The JSON below is modified from the output above: it is grouped into a GeoJSON `FeatureCollection` in which each `Feature` has additional styling information (in the `properties` field) that can be used in visualization tools such as [geojson.io](http://geojson.io): - -{% include_cached copy-clipboard.html %} -~~~ json -{ - "type": "FeatureCollection", - "features": [ - { - "properties": { - "fill-opacity": 0.3, - "stroke": "#30D5C8", - "stroke-width": 5, - "fill": "#30D5C8" - }, - "geometry": { - "coordinates": [ - [ - [ - -87.906471, - 43.038902 - ], - [ - -95.992775, - 36.15398 - ], - [ - -75.704722, - 36.076944 - ], - [ - -87.906471, - 43.038902 - ] - ], - [ - [ - -87.623177, - 41.881832 - ], - [ - -90.199402, - 38.627003 - ], - [ - -82.446732, - 38.413651 - ], - [ - -87.623177, - 41.881832 - ] - ] - ], - "type": "Polygon" - }, - "type": "Feature" - }, - { - "properties": { - "stroke": "yellow", - "fill-opacity": 0.3, - "stroke-width": 9, - "fill": "yellow" - }, - "geometry": { - "type": "LineString", - "coordinates": [ - [ - -87.623177, - 41.881832 - ], - [ - -90.199402, - 38.627003 - ], - [ - -82.446732, - 38.413651 - ], - [ - -87.623177, - 41.881832 - ] - ] - }, - "type": "Feature" - } - ] -} -~~~ - -Here is the geometry described above as shown on [geojson.io](http://geojson.io): - -GeoJSON.io output - -## See also - -- [GeoJSON RFC](https://www.rfc-editor.org/rfc/rfc7946.txt) -- [Spatial features](spatial-features.html) -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial indexes](spatial-indexes.html) -- [Spatial and GIS Glossary of Terms](spatial-glossary.html) -- [Well known text](well-known-text.html) -- [Well known binary](well-known-binary.html) -- [SRID 4326 - longitude and latitude](srid-4326.html) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/geometrycollection.md b/src/current/v21.1/geometrycollection.md deleted file mode 100644 index 190fcbf42b4..00000000000 --- a/src/current/v21.1/geometrycollection.md +++ /dev/null @@ -1,37 +0,0 @@ ---- -title: GEOMETRYCOLLECTION -summary: A GEOMETRYCOLLECTION is used for gathering one or more of the spatial object types into a group. -toc: true ---- - -A `GEOMETRYCOLLECTION` is a collection of heterogeneous [spatial objects](spatial-features.html#spatial-objects), such as [Points](point.html), [LineStrings](linestring.html), [Polygons](polygon.html), or other `GEOMETRYCOLLECTION`s. It provides a way of referring to a group of spatial objects as one "thing" so that you can operate on it/them more conveniently using various SQL functions. - -{% include {{page.version.version}}/spatial/zmcoords.md %} - -## Examples - -A GeometryCollection can be created from SQL by calling the `st_geomfromtext` function on a GeometryCollection definition expressed in the [Well Known Text (WKT)](spatial-glossary.html#wkt) format as shown below. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT ST_GeomFromText('GEOMETRYCOLLECTION(POINT(0 0), LINESTRING(0 0, 1440 900), POLYGON((0 0, 0 1024, 1024 1024, 1024 0, 0 0)))'); -~~~ - -~~~ - st_geomfromtext --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - 0107000000030000000101000000000000000000000000000000000000000102000000020000000000000000000000000000000000000000000000008096400000000000208C40010300000001000000050000000000000000000000000000000000000000000000000000000000000000009040000000000000904000000000000090400000000000009040000000000000000000000000000000000000000000000000 -(1 row) -~~~ - -## See also - -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial objects](spatial-features.html#spatial-objects) -- [POINT](point.html) -- [LINESTRING](linestring.html) -- [POLYGON](polygon.html) -- [MULTIPOINT](multipoint.html) -- [MULTILINESTRING](multilinestring.html) -- [MULTIPOLYGON](multipolygon.html) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/geoserver.md b/src/current/v21.1/geoserver.md deleted file mode 100644 index 420c06b1b51..00000000000 --- a/src/current/v21.1/geoserver.md +++ /dev/null @@ -1,204 +0,0 @@ ---- -title: Using GeoServer with CockroachDB -summary: Tutorial for configuring GeoServer to use CockroachDB. -toc: true -toc_not_nested: true ---- - -This page has instructions for configuring [GeoServer](http://geoserver.org) to use CockroachDB as the underlying database. - -The instructions here reuse parts of the data set described in the [Spatial Data tutorial](spatial-tutorial.html), specifically the `tutorial.roads` table, which contains the [U.S. National Atlas data set](https://www.sciencebase.gov/catalog/file/get/581d052be4b08da350d524ce?f=__disk__60%2F6b%2F4e%2F606b4e564884da8cca57ffeb229cd817006616e0&transform=1&allowOpen=true). - -Many of the instructions on this page come from the following GeoServer documentation pages: - -- [Using the web administration interface](https://docs.geoserver.org/stable/en/user/gettingstarted/web-admin-quickstart/index.html) -- [Publishing a PostGIS table](https://docs.geoserver.org/stable/en/user/gettingstarted/postgis-quickstart/index.html). - -## Prerequisites - -1. CockroachDB [installed on the local machine](install-cockroachdb.html) -1. GeoServer [installed on the local machine](https://docs.geoserver.org/stable/en/user/installation/index.html#installation). - -These instructions assume you are running on a UNIX-like system. - -{{site.data.alerts.callout_success}} -Mac users who use [Homebrew](https://brew.sh) can install GeoServer by typing `brew install geoserver`. -{{site.data.alerts.end}} - -## Step 1. Start CockroachDB and connect to your cluster - -Start a CockroachDB cluster by following the instructions in [Start a Local Cluster](start-a-local-cluster.html). - -## Step 2. Load spatial data - -Connect to the running cluster from the [SQL client](cockroach-sql.html) and enter the statements below. - -First, [create](create-database.html) the `tutorial` database: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE DATABASE tutorial; -~~~ - -Next, switch to the `tutorial` database: - -{% include_cached copy-clipboard.html %} -~~~ sql -USE tutorial; -~~~ - -Finally, load the spatial data set: - -{% include_cached copy-clipboard.html %} -~~~ sql -IMPORT PGDUMP ('https://spatial-tutorial.s3.us-east-2.amazonaws.com/bookstores-and-roads-20210125.sql') WITH ignore_unsupported_statements; -~~~ - -## Step 3. Turn on CockroachDB's experimental box comparison operators - -CockroachDB's support for GeoServer is still in development. To use CockroachDB with GeoServer, you will need to enable the use of certain experimental box2d comparison operators by changing the following [cluster setting](cluster-settings.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -SET CLUSTER SETTING sql.spatial.experimental_box2d_comparison_operators.enabled = ON; -~~~ - -The reasons the box2d comparison operators are experimental in CockroachDB are as follows: - -- PostGIS uses the `&&`, `~`, and `@` operators to do bounding box comparisons. These comparisons can always be index-accelerated by PostgreSQL since it uses [R-tree based indexing](https://en.wikipedia.org/wiki/R-tree) to generate its coverings. -- CockroachDB [uses a different indexing strategy based on space-filling curves](spatial-indexes.html) since this is necessary for [scaling horizontally](frequently-asked-questions.html#how-does-cockroachdb-scale). - - This means that the coverings generated by CockroachDB's `&&`, `~`, and `@` operators for index-accelerated lookups are not the same as the bounding box coverings generated by PostGIS. - - In practice, CockroachDB may return a smaller set of results for the same query, as the space-filling curve covering is often more exact than a bounding box covering, and will exclude from the result set any geometries that have an intersecting bounding box but where no part of the geometry hits the actual bounding box. - - Note that the behavior described above only applies to index-accelerated lookups. - -## Step 4. Start GeoServer - -The easiest place to create the GeoServer data directory is in your user's home directory. - -In the UNIX shell, run the following command: - -{% include_cached copy-clipboard.html %} -~~~ shell -mkdir -p $HOME/geoserver -~~~ - -Next, start GeoServer by running the following command: - -{% include_cached copy-clipboard.html %} -~~~ shell -geoserver $HOME/geoserver -~~~ - -You should see some log output that looks like the following: - -~~~ -2021-06-23 11:44:22.617:INFO::main: Logging initialized @721ms to org.eclipse.jetty.util.log.StdErrLog -2021-06-23 11:44:22.944:INFO:oejs.Server:main: jetty-9.4.36.v20210114; built: 2021-01-14T16:44:28.689Z; git: 238ec6997c7806b055319a6d11f8ae7564adc0de; jvm 1.8.0_282-b08 -2021-06-23 11:44:22.969:INFO:oejdp.ScanningAppProvider:main: Deployment monitor [file:///usr/local/Cellar/geoserver/2.19.1/libexec/webapps/] at interval 1 -2021-06-23 11:44:23.732:INFO:oejw.StandardDescriptorProcessor:main: NO JSP Support for /geoserver, did not find org.eclipse.jetty.jsp.JettyJspServlet -2021-06-23 11:44:24.687:INFO:oejs.session:main: DefaultSessionIdManager workerName=node0 -2021-06-23 11:44:24.687:INFO:oejs.session:main: No SessionScavenger set, using defaults -... -~~~ - -## Step 5. Log in to GeoServer - -In this and the following steps we will set up GeoServer so it can access the spatial data we loaded in [Step 2](#step-2-load-spatial-data). - -Open your web browser and navigate to your [locally running GeoServer instance](http://localhost:8080/geoserver/web/). Log in using the default credentials: username `admin`, password `geoserver`. - -## Step 6. Set up a GeoServer Workspace - -In the left-hand navigation menu, click **Data > Workspaces**. The **Workspaces** page will load. Click the **Add new workspace** button. - -On the **New Workspace** page, enter the following information: - -- In the **Name** field, enter the text "spatial-tutorial". -- In the **Namespace URI** field, enter the URL for the spatial tutorial where this data set is used: https://www.cockroachlabs.com/docs/stable/spatial-data.html. - -Press the **Save** button. - -You will be redirected to the **Workspaces** page, and you should see a workspace called **spatial-tutorial** in the list. - -## Step 7. Configure GeoServer to use CockroachDB - -In the left-hand navigation menu, click **Data > Stores**. The **Stores** page will load. Click the **Add new Store** button. - -You will be taken to the **New data source** page. Under the list of **Vector Data Sources**, click **PostGIS**. - -This opens the **New Vector Data Source** page, where you need to enter the following information: - -1. Under **Basic Store Info**, fill in the **Data Source Name** field with the text: `CockroachDB` - -1. Under **Connection Parameters**, edit **port** to the default CockroachDB port: `26257` - -1. Edit the **database** field to add the text: `tutorial` - -1. Fill in the **user** field with the text: `root` - -1. Delete the contents of the **passwd** field, if any - -Click **Save**, and you will be redirected to the **New Layer** page, with the following unpublished layers: - -- bookstore_routes -- bookstores -- roads - -Click the **Publish** button to the right of the `roads` layer. - -This will bring you to the **Edit Layer** page, where you need to enter the following information: - -1. In the **Bounding Boxes** section, for the **Native Bounding Box** settings, click the **Compute from data** button, which will fill in the form fields. -1. Also in the **Bounding Boxes** section, for the **Lat/Lon Bounding Box** setting, click the **Compute from native bounds** button, which will fill in the form fields. - -Click **Save**, and you will be redirected to the **Layers** page. - -## Step 8. View the `roads` layer - -In the left-hand navigation menu, click **Data > Layer Preview**. - -You will be redirected to the **Layer Preview** page. - -In the row for the `roads` layer, click the **OpenLayers** button under the **Common Formats** column. - -Your browser should open a new tab with the title **OpenLayers map preview**. It should show a map view that looks like the following: - -GeoServer U.S. National Atlas preview - -## See also - -- [Install CockroachDB](install-cockroachdb.html) -- [Working with Spatial Data](spatial-data.html) -- [Spatial Features](spatial-features.html) -- [Spatial Indexes](spatial-indexes.html) -- [Spatial & GIS Glossary of Terms](spatial-glossary.html) -- [Working with Spatial Data](spatial-data.html) -- [Migrate from Shapefiles](migrate-from-shapefiles.html) -- [Migrate from GeoJSON](migrate-from-geojson.html) -- [Migrate from GeoPackage](migrate-from-geopackage.html) -- [Migrate from OpenStreetMap](migrate-from-openstreetmap.html) -- [Spatial Functions](functions-and-operators.html#spatial-functions) -- [POINT](point.html) -- [LINESTRING](linestring.html) -- [POLYGON](polygon.html) -- [MULTIPOINT](multipoint.html) -- [MULTILINESTRING](multilinestring.html) -- [MULTIPOLYGON](multipolygon.html) -- [GEOMETRYCOLLECTION](geometrycollection.html) -- [Well Known Text](well-known-text.html) -- [Well Known Binary](well-known-binary.html) -- [GeoJSON](geojson.html) -- [SRID 4326 - Longitude and Latitude](srid-4326.html) -- [`ST_Contains`](st_contains.html) -- [`ST_ConvexHull`](st_convexhull.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Covers`](st_covers.html) -- [`ST_Disjoint`](st_disjoint.html) -- [`ST_Equals`](st_equals.html) -- [`ST_Intersects`](st_intersects.html) -- [`ST_Overlaps`](st_overlaps.html) -- [`ST_Touches`](st_touches.html) -- [`ST_Union`](st_union.html) -- [`ST_Within`](st_within.html) -- [Troubleshooting Overview](troubleshooting-overview.html) -- [Support Resources](support-resources.html) diff --git a/src/current/v21.1/get-started-with-enterprise-trial.md b/src/current/v21.1/get-started-with-enterprise-trial.md deleted file mode 100644 index 23859bb9624..00000000000 --- a/src/current/v21.1/get-started-with-enterprise-trial.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -title: Enterprise Trial –– Get Started -summary: Check out this page to get started with your CockroachDB Enterprise Trial -toc: true -license: true ---- - -Congratulations on starting your CockroachDB Enterprise Trial! With it, you'll not only get access to CockroachDB's core capabilities like [high availability](frequently-asked-questions.html#how-does-cockroachdb-survive-failures) and [`SERIALIZABLE` isolation](frequently-asked-questions.html#how-is-cockroachdb-strongly-consistent), but also our Enterprise-only features like distributed [`BACKUP`](backup.html) & [`RESTORE`](restore.html), [multi-region capabilities](multiregion-overview.html), and [cluster visualization](enable-node-map.html). - -## Install CockroachDB - -If you haven't already, you'll need to [locally install](install-cockroachdb.html), [remotely deploy](manual-deployment.html), or [orchestrate](orchestration.html) CockroachDB. - -## Enable Enterprise features - -{% include {{ page.version.version }}/misc/set-enterprise-license.md %} - -You can then use the [`SHOW CLUSTER SETTING`](set-cluster-setting.html) command to verify your license: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CLUSTER SETTING cluster.organization; -~~~ - -## Use Enterprise features - -Your cluster now has access to all of CockroachDB's Enterprise features for the length of the trial: - -{% include {{ page.version.version }}/misc/enterprise-features.md %} - -## Getting help - -If you or your team need any help during your trial, our engineers are available on [CockroachDB Community Slack](https://cockroachdb.slack.com), [our forum](https://forum.cockroachlabs.com/), or [GitHub](https://github.com/cockroachdb/cockroach). - -Also consider checking out [Cockroach University](https://university.cockroachlabs.com/) for free online courses that help you get the most out of CockroachDB. - -## See also - -- [Licensing FAQs](enterprise-licensing.html) -- [`SET CLUSTER SETTING`](set-cluster-setting.html) -- [`SHOW CLUSTER SETTING`](show-cluster-setting.html) -- [Cockroach University](https://university.cockroachlabs.com/) diff --git a/src/current/v21.1/global-tables.md b/src/current/v21.1/global-tables.md deleted file mode 100644 index 450d8660592..00000000000 --- a/src/current/v21.1/global-tables.md +++ /dev/null @@ -1,98 +0,0 @@ ---- -title: Global Tables -summary: Guidance on using Global Tables in a multi-region deployment. -toc: true ---- - -In a [multi-region deployment](multiregion-overview.html), the [`GLOBAL` table locality](multiregion-overview.html#global-tables) is a good choice for tables with the following requirements: - -- Read latency must be low, but write latency can be much higher. -- Reads must be up-to-date for business reasons or because the table is referenced by [foreign keys](foreign-key.html). -- Rows in the table, and all latency-sensitive reads, **cannot** be tied to specific regions. - -In general, this pattern is suited well for reference tables that are rarely updated. - -Tables with the `GLOBAL` locality can survive zone or region failures, depending on the database-level [survival goal](multiregion-overview.html#survival-goals) setting. - -{{site.data.alerts.callout_success}} -{% include {{page.version.version}}/misc/multiregion-max-offset.md %} -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -`GLOBAL` tables (and the other [multi-region capabilities](multiregion-overview.html)) require an [Enterprise license](https://www.cockroachlabs.com/get-started-cockroachdb/). -{{site.data.alerts.end}} - -## Prerequisites - -### Fundamentals - -{% include {{ page.version.version }}/topology-patterns/multiregion-fundamentals.md %} - -### Cluster setup - -{% include {{ page.version.version }}/topology-patterns/multi-region-cluster-setup.md %} - -## Configuration - -### Summary - -To use this pattern, you tell CockroachDB to set the [table locality](multiregion-overview.html#table-locality) to `GLOBAL`. - -{% include {{page.version.version}}/sql/global-table-description.md %} - -### Steps - -{% include {{page.version.version}}/topology-patterns/multiregion-db-setup.md %} - -Next, create a [`GLOBAL` table](multiregion-overview.html#global-tables) by issuing the following statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE postal_codes ( - id INT PRIMARY KEY, - code STRING -) LOCALITY GLOBAL; -~~~ - -Alternatively, you can set an existing table's locality to `GLOBAL` using [`ALTER TABLE ... SET LOCALITY`](set-locality.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE postal_codes SET LOCALITY GLOBAL; -~~~ - -{{site.data.alerts.callout_success}} -A good way to check that your [table locality settings](multiregion-overview.html#table-locality) are having the expected effect is by monitoring how the performance metrics of a workload change as the settings are applied to a running cluster. For a tutorial showing how table localities can improve performance metrics across a multi-region cluster, see [Low Latency Reads and Writes in a Multi-Region Cluster](demo-low-latency-multi-region-deployment.html). -{{site.data.alerts.end}} - -## Characteristics - -### Latency - -Global tables support low-latency, global reads of read-mostly data using an extension to CockroachDB's standard transaction protocol called [non-blocking transactions](architecture/transaction-layer.html#non-blocking-transactions). - -#### Reads - -Thanks to the [non-blocking transaction](architecture/transaction-layer.html#non-blocking-transactions) protocol extension, reads against `GLOBAL` tables access a consistent local replica and therefore never leave the region. This keeps read latency low. - -#### Writes - -Writes incur higher latencies than reads, since they require a "commit-wait" step to ensure consistency. For more information about how this works, see [non-blocking transactions](architecture/transaction-layer.html#non-blocking-transactions). - -### Resiliency - -Because the `test` database does not specify a [survival goal](multiregion-overview.html#survival-goals), it uses the default [`ZONE` survival goal](multiregion-overview.html#surviving-zone-failures). With the default settings, an entire zone can fail without interrupting access to the database. - -For more information about how to choose a database survival goal, see [When to use `ZONE` vs. `REGION` survival goals](when-to-use-zone-vs-region-survival-goals.html). - -## Alternatives - -- If rows in the table, and all latency-sensitive queries, can be tied to specific geographies, consider the [`REGIONAL` Table Locality Pattern](regional-tables.html) pattern. - -## Tutorial - -For a step-by-step demonstration showing how CockroachDB's multi-region capabilities (including `GLOBAL` and `REGIONAL` tables) give you low-latency reads in a distributed cluster, see the tutorial on [Low Latency Reads and Writes in a Multi-Region Cluster](demo-low-latency-multi-region-deployment.html). - -## See also - -{% include {{ page.version.version }}/topology-patterns/see-also.md %} diff --git a/src/current/v21.1/grant.md b/src/current/v21.1/grant.md deleted file mode 100644 index e1a3a7f30d9..00000000000 --- a/src/current/v21.1/grant.md +++ /dev/null @@ -1,289 +0,0 @@ ---- -title: GRANT -summary: The GRANT statement grants user privileges for interacting with specific database objects and adds roles or users as a member of a role. -toc: true ---- - -`GRANT` [statement](sql-statements.html) controls each [role](authorization.html#create-and-manage-roles) or [user's](authorization.html#create-and-manage-users) SQL [privileges](authorization.html#assign-privileges) for interacting with specific [databases](create-database.html), [schemas](create-schema.html), [tables](create-table.html), or [user-defined types](enum.html). For privileges required by specific statements, see the documentation for the respective [SQL statement](sql-statements.html). - -You can use `GRANT` to directly grant privileges to a role or user, or you can grant membership to an existing role, which grants that role's privileges to the grantee. - -## Syntax - -
-{% include {{ page.version.version }}/sql/generated/diagrams/grant.html %} -
- -### Parameters - -Parameter | Description ---------------------------|------------ -`ALL`
`ALL PRIVILEGES` | Grant all [privileges](#supported-privileges). -`targets` | A comma-separated list of database, schema, table, or user-defined type names, followed by the name of the object (e.g., `DATABASE mydatabase`).
{{site.data.alerts.callout_info}}To grant privileges on all tables in a database or schema, you can use `GRANT ... ON TABLE *`. For an example, see [Grant privileges on all tables in a database or schema](#grant-privileges-on-all-tables-in-a-database-or-schema).{{site.data.alerts.end}} -`name_list` | A comma-separated list of [users](authorization.html#create-and-manage-users) and/or [roles](authorization.html#create-and-manage-roles) to whom to grant privileges. -`privilege_list ON ...` | Specify a comma-separated list of [privileges](authorization.html#assign-privileges) to grant. -`privilege_list TO ...` | Specify a comma-separated list of [roles](authorization.html#create-and-manage-roles) whose membership to grant. -`WITH ADMIN OPTION` | Designate the user as a role admin. Role admins can grant or revoke membership for the specified role. - -## Supported privileges - -Roles and users can be granted the following privileges: - -{% include {{ page.version.version }}/sql/privileges.md %} - -## Required privileges - -- To grant privileges, the user granting the privileges must also have the privilege being granted on the target database or tables. For example, a user granting the `SELECT` privilege on a table to another user must have the `GRANT` and `SELECT` privileges on that table. - -- To grant roles, the user granting role membership must be a role admin (i.e., members with the `WITH ADMIN OPTION`) or a member of the `admin` role. To grant membership to the `admin` role, the user must have `WITH ADMIN OPTION` on the `admin` role. - -## Details - -### Granting privileges - -- When a role or user is granted privileges for a database, new tables created in the database will inherit the privileges, but the privileges can then be changed. - - {{site.data.alerts.callout_info}} - The user does not get privileges to existing tables in the database. To grant privileges to a user on all existing tables in a database, see [Grant privileges on all tables in a database](#grant-privileges-on-all-tables-in-a-database-or-schema) - {{site.data.alerts.end}} - -- When a role or user is granted privileges for a table, the privileges are limited to the table. -- The `root` user automatically belongs to the `admin` role and has the `ALL` privilege for new databases. -- For privileges required by specific statements, see the documentation for the respective [SQL statement](sql-statements.html). - -### Granting roles - -- Users and roles can be members of roles. -- The `root` user is automatically created as an `admin` role and assigned the `ALL` privilege for new databases. -- All privileges of a role are inherited by all its members. -- Membership loops are not allowed (direct: `A is a member of B is a member of A` or indirect: `A is a member of B is a member of C ... is a member of A`). - -## Examples - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### Grant privileges on databases - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE USER max WITH PASSWORD roach; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT ALL ON DATABASE movr TO max; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON DATABASE movr; -~~~ - -~~~ - database_name | grantee | privilege_type -----------------+---------+----------------- - movr | admin | ALL - movr | max | ALL - movr | root | ALL -(3 rows) -~~~ - -### Grant privileges on specific tables in a database - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT DELETE ON TABLE rides TO max; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON TABLE rides; -~~~ - -~~~ - database_name | schema_name | table_name | grantee | privilege_type -----------------+-------------+------------+---------+----------------- - movr | public | rides | admin | ALL - movr | public | rides | max | DELETE - movr | public | rides | root | ALL -(3 rows) -~~~ - -### Grant privileges on all tables in a database or schema - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT SELECT ON TABLE movr.public.* TO max; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON TABLE movr.public.*; -~~~ - -~~~ - database_name | schema_name | table_name | grantee | privilege_type -----------------+-------------+----------------------------+---------+----------------- - movr | public | promo_codes | admin | ALL - movr | public | promo_codes | max | SELECT - movr | public | promo_codes | root | ALL - movr | public | rides | admin | ALL - movr | public | rides | max | DELETE - movr | public | rides | max | SELECT - movr | public | rides | root | ALL - movr | public | user_promo_codes | admin | ALL - movr | public | user_promo_codes | max | SELECT - movr | public | user_promo_codes | root | ALL - movr | public | users | admin | ALL - movr | public | users | max | SELECT - movr | public | users | root | ALL - movr | public | vehicle_location_histories | admin | ALL - movr | public | vehicle_location_histories | max | SELECT - movr | public | vehicle_location_histories | root | ALL - movr | public | vehicles | admin | ALL - movr | public | vehicles | max | SELECT - movr | public | vehicles | root | ALL -(19 rows) -~~~ - -### Make a table readable to every user in the system - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT SELECT ON TABLE vehicles TO public; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON TABLE vehicles; -~~~ - -~~~ - database_name | schema_name | table_name | grantee | privilege_type -----------------+-------------+------------+---------+----------------- - movr | public | vehicles | admin | ALL - movr | public | vehicles | max | SELECT - movr | public | vehicles | public | SELECT - movr | public | vehicles | root | ALL -(4 rows) -~~~ - -### Grant privileges on schemas - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE SCHEMA cockroach_labs; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT ALL ON SCHEMA cockroach_labs TO max; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON SCHEMA cockroach_labs; -~~~ - -~~~ - database_name | schema_name | grantee | privilege_type -----------------+----------------+---------+----------------- - movr | cockroach_labs | admin | ALL - movr | cockroach_labs | max | ALL - movr | cockroach_labs | root | ALL -(3 rows) -~~~ - -### Grant privileges on user-defined types - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TYPE status AS ENUM ('available', 'unavailable'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT ALL ON TYPE status TO max; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON TYPE status; -~~~ - -~~~ - database_name | schema_name | type_name | grantee | privilege_type -----------------+-------------+-----------+---------+----------------- - movr | public | status | admin | ALL - movr | public | status | demo | ALL - movr | public | status | max | ALL - movr | public | status | public | USAGE - movr | public | status | root | ALL -(5 rows) -~~~ - -### Grant the privilege to manage the replication zones for a database or table - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT ZONECONFIG ON TABLE rides TO max; -~~~ - -The user `max` can then use the [`CONFIGURE ZONE`](configure-zone.html) statement to add, modify, reset, or remove replication zones for the table `rides`. - -### Grant role membership - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE ROLE developer WITH CREATEDB; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE USER abbey WITH PASSWORD lincoln; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT developer TO abbey; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON ROLE developer; -~~~ - -~~~ - role_name | member | is_admin -------------+--------+----------- - developer | abbey | false -(1 row) -~~~ - -### Grant the admin option - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT developer TO abbey WITH ADMIN OPTION; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON ROLE developer; -~~~ - -~~~ - role_name | member | is_admin -------------+--------+----------- - developer | abbey | true -(1 row) -~~~ - -## See also - -- [Authorization](authorization.html) -- [`REVOKE`](revoke.html) -- [`SHOW GRANTS`](show-grants.html) -- [`SHOW ROLES`](show-roles.html) -- [`CONFIGURE ZONE`](configure-zone.html) -- [Manage Users](authorization.html#create-and-manage-users) diff --git a/src/current/v21.1/gssapi_authentication.md b/src/current/v21.1/gssapi_authentication.md deleted file mode 100644 index 037f0f964f6..00000000000 --- a/src/current/v21.1/gssapi_authentication.md +++ /dev/null @@ -1,244 +0,0 @@ ---- -title: GSSAPI Authentication (Enterprise) -summary: Learn about the GSSAPI authentication features for secure CockroachDB clusters. -toc: true ---- - -CockroachDB supports the Generic Security Services API (GSSAPI) with Kerberos authentication. - -{% include enterprise-feature.md %} - -## Requirements - -- A working Active Directory or Kerberos environment -- A Service Principal -- A GSSAPI-compatible Postgres Client (psql, etc.) -- A client machine with a Kerberos client installed and configured - -## Configuring KDC for CockroachDB - -To use Kerberos authentication with CockroachDB, configure a Kerberos service principal name (SPN) for CockroachDB and generate a valid keytab file with the following specifications: - -- Set the SPN to the name specified by your client driver. For example, if you use the psql client, set SPN to `postgres`. -- Create SPNs for all DNS addresses that a user would use to connect to your CockroachDB cluster (including any TCP load balancers between the user and the CockroachDB node) and ensure that the keytab contains the keys for every SPN you create. - -### Active Directory - -For Active Directory, the client syntax for generating a keytab that maps a service principal to the SPN is as follows: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ ktpass -out {keytab_filename} -princ {Client_SPN}/{NODE/LB_FQDN}@{DOMAIN} -mapUser {Service_Principal}@{DOMAIN} -mapOp set -pType KRB5_NT_PRINCIPAL +rndPass -crypto AES256-SHA1 -~~~ - -Example: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ ktpass -out postgres.keytab -princ postgres/loadbalancer1.cockroach.industries@COCKROACH.INDUSTRIES -mapUser pguser@COCKROACH.INDUSTRIES -mapOp set -pType KRB5_NT_PRINCIPAL +rndPass -crypto AES256-SHA1 -~~~ - -Copy the resulting keytab to the database nodes. If clients are connecting to multiple addresses (more than one load balancer, or clients connecting directly to nodes), you will need to generate a keytab for each client endpoint. You may want to merge your keytabs together for easier management. You can do this using the `ktpass` command, using the following syntax: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ ktpass -out {new_keytab_filename} -in {old_keytab_filename} -princ {Client_SPN}/{NODE/LB_FQDN}@{DOMAIN} -mapUser {Service_Principal}@{DOMAIN} -mapOp add -pType KRB5_NT_PRINCIPAL +rndPass -crypto AES256-SHA1 -~~~ - -Example (adds `loadbalancer2` to the above example): - -{% include_cached copy-clipboard.html %} -~~~ shell -$ ktpass -out postgres_2lb.keytab -in postgres.keytab -princ postgres/loadbalancer2.cockroach.industries@COCKROACH.INDUSTRIES -mapUser pguser@COCKROACH.INDUSTRIES -mapOp add -pType KRB5_NT_PRINCIPAL +rndPass -crypto AES256-SHA1 -~~~ - -### MIT KDC - -In MIT KDC, you cannot map a service principal to an SPN with a different username, so you will need to create a service principal that includes the SPN for your client. - -{% include_cached copy-clipboard.html %} -~~~ shell -$ create-user: kadmin.local -q "addprinc {SPN}/{CLIENT_FQDN}@{DOMAIN}" -pw "{initial_password}" -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ create-keytab: kadmin.local -q "ktadd -k keytab {SPN}/{CLIENT_FQDN}@{DOMAIN}" -~~~ - -Example: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ kadmin.local -q "addprinc postgres/client2.cockroach.industries@COCKROACH.INDUSTRIES" -pw "testing12345!" -$ kadmin.local -q "ktadd -k keytab postgres/client2.cockroach.industries@COCKROACH.INDUSTRIES" -~~~ - -Copy the resulting keytab to the database nodes. If clients are connecting to multiple addresses (more than one load balancer, or clients connecting directly to nodes), you will need to generate a keytab for each client endpoint. You may want to merge your keytabs together for easier management. The `ktutil` command can be used to read multiple keytab files and output them into a single output [here](https://web.mit.edu/kerberos/krb5-devel/doc/admin/admin_commands/ktutil.html). - - -## Configuring the CockroachDB node -1. Copy the keytab file to a location accessible by the `cockroach` binary. - -2. [Create certificates](cockroach-cert.html) for inter-node and `root` user authentication: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ mkdir certs my-safe-directory - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach cert create-ca \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach cert create-node \ - localhost \ - $(hostname) \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach cert create-client \ - root \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key - ~~~ - -3. Provide the path to the keytab in the `KRB5_KTNAME` environment variable. - - Example: `export KRB5_KTNAME=/home/cockroach/postgres.keytab` - -4. Start a CockroachDB node: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --certs-dir=certs \ - --listen-addr=0.0.0.0 - ~~~ - -5. Connect to CockroachDB as `root` using the `root` client certificate generated above: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --certs-dir=certs - ~~~ - -6. [Enable an Enterprise license](licensing-faqs.html#obtain-a-license). - {{site.data.alerts.callout_info}} You need the Enterprise license if you want to use the GSSAPI feature. However, if you only want to test that the GSSAPI setup is working, you do not need to enable an Enterprise license. {{site.data.alerts.end}} - -7. Enable GSSAPI authentication: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET cluster setting server.host_based_authentication.configuration = 'host all all all gss include_realm=0'; - ~~~ - - Setting the `server.host_based_authentication.configuration` [cluster setting](cluster-settings.html) to this particular value makes it mandatory for all non-`root` users to authenticate using GSSAPI. The `root` user is always an exception and remains able to authenticate using a valid client cert or a user password. - - The `include_realm=0` option is required to tell CockroachDB to remove the `@DOMAIN.COM` realm information from the username. We do not support any advanced mapping of GSSAPI usernames to CockroachDB usernames right now. If you want to limit which realms' users can connect, you can also add one or more `krb_realm` parameters to the end of the line as an allowlist, as follows: `host all all all gss include_realm=0 krb_realm=domain.com krb_realm=corp.domain.com` - - The syntax is based on the `pg_hba.conf` standard for PostgreSQL which is documented [here](https://www.postgresql.org/docs/current/auth-pg-hba-conf.html). It can be used to exclude other users from Kerberos authentication. - -8. Create CockroachDB users for every Kerberos user. Ensure the username does not have the `DOMAIN.COM` realm information. For example, if one of your Kerberos users has a username `carl@realm.com`, then you need to create a CockroachDB user with the username `carl`: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE USER carl; - ~~~ - - Grant privileges to the user: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > GRANT ALL ON DATABASE defaultdb TO carl; - ~~~ - -## Configuring the client - -{{site.data.alerts.callout_info}} -The `cockroach sql` shell does not yet support GSSAPI authentication. You need to use a GSSAPI-compatible Postgres client, such as Postgres's `psql` client. -{{site.data.alerts.end}} - -1. Install and configure your Kerberos client: - - For CentOS/RHEL systems, run: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ yum install krb5-user - ~~~ - - For Ubuntu/Debian systems, run: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ apt-get install krb5-user - ~~~ - - Edit the `/etc/krb5.conf` file to include: - - {% include_cached copy-clipboard.html %} - ~~~ - [libdefaults] - default_realm = {REALM} - - [realms] - {REALM} = { - kdc = {fqdn-kdc-server or ad-server} - admin_server = {fqdn-kdc-server or ad-server} - default_domain = {realm-lower-case} - } - ~~~ - - Example: - - {% include_cached copy-clipboard.html %} - ~~~ - - [libdefaults] - default_realm = COCKROACH.INDUSTRIES - - [realms] - COCKROACH.INDUSTRIES = { - kdc = ad.cockroach.industries - admin_server = ad.cockroach.industries - default_domain = cockroach.industries - } - ~~~ - -2. Get a ticket for the db user: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kinit carl - ~~~ - -3. Verify if a valid ticket has been generated: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ klist - ~~~ - -4. Install the Postgres client (for example, postgresql-client-10 Debian package from postgresql.org). -5. Use the `psql` client, which supports GSSAPI authentication, to connect to CockroachDB: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ psql "postgresql://localhost:26257/defaultdb?sslmode=verify-full&sslrootcert=/certs/ca.crt" -U carl - ~~~ - -4. If you specified an Enterprise license earlier, you should now have a Postgres shell in CockroachDB, indicating that the GSSAPI authentication was successful. If you did not specify an Enterprise license, you'll see a message like this: `psql: ERROR: use of GSS authentication requires an Enterprise license.` If you see this message, GSSAPI authentication is set up correctly. - -## See also - -- [Authentication](authentication.html) -- [Create Security Certificates](cockroach-cert.html) diff --git a/src/current/v21.1/hash-sharded-indexes.md b/src/current/v21.1/hash-sharded-indexes.md deleted file mode 100644 index cbb5c20f2a0..00000000000 --- a/src/current/v21.1/hash-sharded-indexes.md +++ /dev/null @@ -1,34 +0,0 @@ ---- -title: Hash-sharded Indexes -summary: Hash-sharded indexes can eliminate single-range hotspots and improve write performance on sequentially-keyed indexes at a small cost to read performance -toc: true ---- - -If you are working with a table that must be indexed on sequential keys, you should use **hash-sharded indexes**. Hash-sharded indexes distribute sequential traffic uniformly across ranges, eliminating single-range hotspots and improving write performance on sequentially-keyed indexes at a small cost to read performance. - -{% include {{ page.version.version }}/misc/experimental-warning.md %} - -## How hash-sharded indexes work - - CockroachDB automatically splits ranges of data in [the key-value store](architecture/storage-layer.html) based on [the size of the range](architecture/distribution-layer.html#range-splits), and on [the load streaming to the range](load-based-splitting.html). To split a range based on load, the system looks for a point in the range that evenly divides incoming traffic. If the range is indexed on a column of data that is sequential in nature (e.g., [an ordered sequence](sql-faqs.html#what-are-the-differences-between-uuid-sequences-and-unique_rowid), or a series of increasing, non-repeating [`TIMESTAMP`s](timestamp.html)), then all incoming writes to the range will be the last (or first) item in the index and appended to the end of the range. As a result, the system cannot find a point in the range that evenly divides the traffic, and the range cannot benefit from load-based splitting, creating a hotspot on the single range. - -For details about the mechanics and performance improvements of hash-sharded indexes in CockroachDB, see our [Hash Sharded Indexes Unlock Linear Scaling for Sequential Workloads](https://www.cockroachlabs.com/blog/hash-sharded-indexes-unlock-linear-scaling-for-sequential-workloads/) blog post. - -## How to create a hash-sharded index - -To create a hash-sharded index, set the `experimental_enable_hash_sharded_indexes` [session variable](set-vars.html) to `on`. Then, add the optional [`USING HASH WITH BUCKET_COUNT = n_buckets` clause](sql-grammar.html#opt_hash_sharded) to a [`CREATE INDEX`](create-index.html) statement, to an [`INDEX` definition](sql-grammar.html#index_def) in a [`CREATE TABLE`](create-table.html) statement, or to an [`ALTER PRIMARY KEY`](alter-primary-key.html) statement. When this clause is used, CockroachDB creates `n_buckets` computed columns, shards the index into `n_buckets` shards, and then stores each index shard in the underlying key-value store with one of the computed column's hash as its prefix. - -To change the bucket size of an existing hash-sharded primary key index, use an [`ALTER PRIMARY KEY`](alter-primary-key.html) statement with a [`USING HASH WITH BUCKET_COUNT = n_buckets` clause](sql-grammar.html#opt_hash_sharded) that specifies the new bucket size and the existing primary key columns. - -{{site.data.alerts.callout_info}} -Hash-sharded indexes cannot be [interleaved](interleave-in-parent.html). -{{site.data.alerts.end}} - -## Examples - -For an example of a hash-sharded index, see [Create a hash-sharded secondary index](create-index.html#create-a-hash-sharded-secondary-index). - -## See also - -- [Indexes](indexes.html) -- [`CREATE INDEX`](create-index.html) diff --git a/src/current/v21.1/import-into.md b/src/current/v21.1/import-into.md deleted file mode 100644 index 2ea93c69121..00000000000 --- a/src/current/v21.1/import-into.md +++ /dev/null @@ -1,283 +0,0 @@ ---- -title: IMPORT INTO -summary: Import CSV data into an existing CockroachDB table. -toc: true ---- - -The `IMPORT INTO` [statement](sql-statements.html) imports CSV, Avro, or delimited data into an [existing table](create-table.html), by appending new rows into the table. - -## Considerations - -- `IMPORT INTO` works for existing tables. To import data into new tables, read the following [Import into a new table from a CSV file](#import-into-a-new-table-from-a-csv-file) example. -- `IMPORT INTO` takes the table **offline** before importing the data. The table will be online again once the job has completed successfully. -- `IMPORT INTO` cannot be used during a [rolling upgrade](upgrade-cockroach-version.html). -- `IMPORT INTO` is a blocking statement. To run an `IMPORT INTO` job asynchronously, use the [`DETACHED`](#options-detached) option. -- `IMPORT INTO` invalidates all [foreign keys](foreign-key.html) on the target table. To validate the foreign key(s), use the [`VALIDATE CONSTRAINT`](validate-constraint.html) statement. -- `IMPORT INTO` is an insert-only statement; it cannot be used to update existing rows—see [`UPDATE`](update.html). Imported rows cannot conflict with primary keys in the existing table, or any other [`UNIQUE`](unique.html) constraint on the table. -- `IMPORT INTO` does not offer `SELECT` or `WHERE` clauses to specify subsets of rows. To do this, use [`INSERT`](insert.html#insert-from-a-select-statement). -- `IMPORT INTO` will cause any [changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html) running on the targeted table to fail. -- {% include {{page.version.version}}/sql/import-into-regional-by-row-table.md %} - - -{{site.data.alerts.callout_info}} -Optimize import operations in your applications by following our [Import Performance Best Practices](import-performance-best-practices.html). -{{site.data.alerts.end}} - -## Required privileges - -#### Table privileges - -The user must have the `INSERT` and `DROP` [privileges](authorization.html#assign-privileges) on the specified table. (`DROP` is required because the table is taken offline during the `IMPORT INTO`.) - -#### Source privileges - -The source file URL does _not_ require the `ADMIN` role in the following scenarios: - -- S3 and GS using `SPECIFIED` (and not `IMPLICIT`) credentials. Azure is always `SPECIFIED` by default. -- [Userfile](use-userfile-for-bulk-operations.html) - -The source file URL _does_ require the `ADMIN` role in the following scenarios: - -- S3 or GS using `IMPLICIT` credentials -- Use of a [custom endpoint](https://docs.aws.amazon.com/sdk-for-go/api/aws/endpoints/) on S3 -- [Nodelocal](cockroach-nodelocal-upload.html), [HTTP](use-a-local-file-server-for-bulk-operations.html), or [HTTPS](use-a-local-file-server-for-bulk-operations.html) - -Learn more about [cloud storage for bulk operations](use-cloud-storage-for-bulk-operations.html). - -## Synopsis - -
-{% include {{ page.version.version }}/sql/generated/diagrams/import_into.html %} -
- -{{site.data.alerts.callout_info}} -While importing into an existing table, the table is taken offline. -{{site.data.alerts.end}} - -## Parameters - -Parameter | Description -----------|------------ -`table_name` | The name of the table you want to import into. -`column_name` | The table columns you want to import.

Note: Currently, target columns are not enforced. -`file_location` | The [URL](#import-file-location) of a CSV or Avro file containing the table data. This can be a comma-separated list of URLs. For an example, see [Import into an existing table from multiple CSV files](#import-into-an-existing-table-from-multiple-csv-files) below. -`
DEFAULT VALUES`. Running this command on the `drivers` table results in an error because the `city` column in `drivers` cannot be *NULL*, and has no default value specified. - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO drivers DEFAULT VALUES; -~~~ - -~~~ -pq: null value in column "city" violates not-null constraint -~~~ - -### Insert and return values - -In this example, the `RETURNING` clause returns the `id` values of the rows inserted, which are generated server-side by the `gen_random_uuid()` function. The language-specific versions assume that you have installed the relevant [client drivers](install-client-drivers.html). - -{{site.data.alerts.callout_success}}This use of RETURNING mirrors the behavior of MySQL's last_insert_id() function.{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}}When a driver provides a query() method for statements that return results and an exec() method for statements that do not (e.g., Go), it's likely necessary to use the query() method for INSERT statements with RETURNING.{{site.data.alerts.end}} - -
- - - - - -
- -
- -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO drivers (id, city) - VALUES (DEFAULT, 'seattle'), (DEFAULT, 'chicago') - RETURNING id; -~~~ - -~~~ - id -+--------------------------------------+ - b7750a60-91f2-404e-9cd1-5a3c310c1c9d - c85e637e-2b03-4a52-bc54-1e1f6d7fd89b -(2 rows) -~~~ - -
- -
- -{% include_cached copy-clipboard.html %} -~~~ python -# Import the driver. -import psycopg2 - -# Connect to the "movr" database. -conn = psycopg2.connect( - database='movr', - user='root', - host='localhost', - port=26257 -) - -# Make each statement commit immediately. -conn.set_session(autocommit=True) - -# Open a cursor to perform database operations. -cur = conn.cursor() - -# Insert two rows into the "drivers" table -# and return the "id" values generated server-side. -cur.execute( - "INSERT INTO drivers (id, city) " - "VALUES (DEFAULT, 'seattle'), (DEFAULT, 'chicago') " - "RETURNING id" -) - -# Print out the returned values. -rows = cur.fetchall() -print('IDs:') -for row in rows: - print([str(cell) for cell in row]) - -# Close the database connection. -cur.close() -conn.close() -~~~ - -The printed values would look like: - -~~~ -IDs: -['cdd379e3-2d0b-4622-8ba8-4f0a1edfbc8e'] -['4224b360-b1b0-4e4d-aba2-a35c64cdf404'] -~~~ - -
- -
- -{% include_cached copy-clipboard.html %} -~~~ ruby -# Import the driver. -require 'pg' - -# Connect to the "movr" database. -conn = PG.connect( - user: 'root', - dbname: 'movr', - host: 'localhost', - port: 26257 -) - -# Insert two rows into the "drivers" table -# and return the "id" values generated server-side. -conn.exec( - "INSERT INTO drivers (id, city) "\ - "VALUES (DEFAULT, 'seattle'), (DEFAULT, 'chicago') "\ - "RETURNING id" -) do |res| - -# Print out the returned values. -puts "IDs:" - res.each do |row| - puts row - end -end - -# Close communication with the database. -conn.close() -~~~ - -The printed values would look like: - -~~~ -IDs: -{"id"=>"cdd379e3-2d0b-4622-8ba8-4f0a1edfbc8e"} -{"id"=>"4224b360-b1b0-4e4d-aba2-a35c64cdf404"} -~~~ - -
- -
- -{% include_cached copy-clipboard.html %} -~~~ go -package main - -import ( - "database/sql" - "fmt" - "log" - - _ "github.com/lib/pq" -) - -func main() { - //Connect to the "movr" database. - db, err := sql.Open( - "postgres", - "postgresql://root@localhost:26257/movr?sslmode=disable", - ) - if err != nil { - log.Fatal("error connecting to the database: ", err) - } - - // Insert two rows into the "drivers" table - // and return the "id" values generated server-side. - rows, err := db.Query( - "INSERT INTO drivers (id, city) " + - "VALUES (DEFAULT, 'seattle'), (DEFAULT, 'chicago') " + - "RETURNING id", - ) - if err != nil { - log.Fatal(err) - } - - // Print out the returned values. - defer rows.Close() - fmt.Println("IDs:") - for rows.Next() { - var id string - if err := rows.Scan(&id); err != nil { - log.Fatal(err) - } - fmt.Printf("%s\n", id) - } -} -~~~ - -The printed values would look like: - -~~~ -IDs: -cdd379e3-2d0b-4622-8ba8-4f0a1edfbc8e -4224b360-b1b0-4e4d-aba2-a35c64cdf404 -~~~ - -
- -
- -{% include_cached copy-clipboard.html %} -~~~ js -var async = require('async') -var pg = require('pg') - -// Config to connect to the "movr" database. -var config = { - user: 'root', - host: 'localhost', - database: 'movr', - port: 26257 - } - -// Create pool -var pool = new pg.Pool(config) - -pool.connect(function (err, client, done) { - - // Close communication with the database and exit. - var finish = function () { - done() - process.exit() - } - - if (err) { - console.error('could not connect to cockroachdb', err) - finish() - } - async.waterfall([ - function (next) { - // Insert two rows into the "drivers" table - // and return the "id" values generated server-side. - client.query( - `INSERT INTO drivers (id, city) - VALUES (DEFAULT, 'seattle'), (DEFAULT, 'chicago') - RETURNING id`, - next - ) - } - ], - function (err, results) { - if (err) { - console.error('error inserting into and selecting from drivers', err) - finish() - } - // Print out the returned values. - console.log('IDs:') - results.rows.forEach(function (row) { - console.log(row) - }) - - finish() - }) -}) -~~~ - -The printed values would look like: - -~~~ -IDs: -{ id: 'cdd379e3-2d0b-4622-8ba8-4f0a1edfbc8e' } -{ id: '4224b360-b1b0-4e4d-aba2-a35c64cdf404' } -~~~ - -
- -### Update values `ON CONFLICT` - -When a uniqueness conflict is detected, CockroachDB stores the row in a temporary table called `excluded`. This example demonstrates how you use the columns in the temporary `excluded` table to apply updates on conflict. - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO user_promo_codes (city, user_id, code, "timestamp", usage_count) - VALUES ('new york', '147ae147-ae14-4b00-8000-000000000004', 'promo_code', now(), 1) - ON CONFLICT (city, user_id, code) - DO UPDATE SET usage_count = excluded.usage_count; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM user_promo_codes WHERE code = 'promo_code'; -~~~ - -~~~ - city | user_id | code | timestamp | usage_count -+----------+--------------------------------------+------------+----------------------------------+-------------+ - new york | 147ae147-ae14-4b00-8000-000000000004 | promo_code | 2019-08-12 14:23:52.262849+00:00 | 1 -(1 row) -~~~ - -You can also update the row using an existing value: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO user_promo_codes (city, user_id, code, "timestamp", usage_count) - VALUES ('new york', '147ae147-ae14-4b00-8000-000000000004', 'promo_code', now(), 1) - ON CONFLICT (city, user_id, code) - DO UPDATE SET ("timestamp", usage_count) = (now(), user_promo_codes.usage_count + excluded.usage_count); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM user_promo_codes WHERE code = 'promo_code'; -~~~ - -~~~ - city | user_id | code | timestamp | usage_count -+----------+--------------------------------------+------------+----------------------------------+-------------+ - new york | 147ae147-ae14-4b00-8000-000000000004 | promo_code | 2019-08-12 14:26:50.697382+00:00 | 2 -(1 row) -~~~ - -You can also use a `WHERE` clause to apply the `DO UPDATE SET` expression conditionally: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO user_promo_codes (city, user_id, code, "timestamp", usage_count) - VALUES ('new york', '147ae147-ae14-4b00-8000-000000000004', 'promo_code', now(), 3) - ON CONFLICT (city, user_id, code) - DO UPDATE SET ("timestamp", usage_count) = (now(), user_promo_codes.usage_count + excluded.usage_count) - WHERE excluded.usage_count = 1; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM user_promo_codes WHERE code = 'promo_code'; -~~~ - -~~~ - city | user_id | code | timestamp | usage_count -+----------+--------------------------------------+------------+----------------------------------+-------------+ - new york | 147ae147-ae14-4b00-8000-000000000004 | promo_code | 2019-08-12 14:26:50.697382+00:00 | 2 -(1 row) -~~~ - -### Do not update values `ON CONFLICT` - -In this example, we get an error from a uniqueness conflict. - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO user_promo_codes (city, user_id, code, "timestamp", usage_count) - VALUES ('new york', '147ae147-ae14-4b00-8000-000000000004', 'promo_code', now(), 1); -~~~ - -~~~ -pq: duplicate key value (city,user_id,code)=('new york','147ae147-ae14-4b00-8000-000000000004','promo_code') violates unique constraint "primary" -~~~ - -In this example, we use `ON CONFLICT DO NOTHING` to ignore the uniqueness error and prevent the affected row from being updated: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO user_promo_codes (city, user_id, code, "timestamp", usage_count) - VALUES ('new york', '147ae147-ae14-4b00-8000-000000000004', 'promo_code', now(), 1) - ON CONFLICT (city, user_id, code) - DO NOTHING; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM user_promo_codes WHERE code = 'promo_code'; -~~~ - -~~~ - city | user_id | code | timestamp | usage_count -+----------+--------------------------------------+------------+----------------------------------+-------------+ - new york | 147ae147-ae14-4b00-8000-000000000004 | promo_code | 2019-08-12 14:26:50.697382+00:00 | 2 -(1 row) -~~~ - -In this example, `ON CONFLICT DO NOTHING` prevents the first row from updating while allowing the second row to be inserted: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO user_promo_codes (city, user_id, code, "timestamp", usage_count) - VALUES ('new york', '147ae147-ae14-4b00-8000-000000000004', 'promo_code', now(), 1), ('new york', '147ae147-ae14-4b00-8000-000000000004', 'new_promo', now(), 1) - ON CONFLICT (city, user_id, code) - DO NOTHING; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM user_promo_codes WHERE code in ('promo_code', 'new_promo'); -~~~ - -~~~ - city | user_id | code | timestamp | usage_count -+----------+--------------------------------------+------------+----------------------------------+-------------+ - new york | 147ae147-ae14-4b00-8000-000000000004 | new_promo | 2019-08-12 14:30:16.666848+00:00 | 1 - new york | 147ae147-ae14-4b00-8000-000000000004 | promo_code | 2019-08-12 14:26:50.697382+00:00 | 2 -(2 rows) -~~~ - -### Import data containing duplicate rows using `ON CONFLICT` and `DISTINCT ON` - -If the input data for `INSERT ON CONFLICT` contains duplicate rows, -you must use [`DISTINCT -ON`](select-clause.html#eliminate-duplicate-rows) to remove these -duplicates. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> WITH inputrows (city, user_id, code, "timestamp", usage_count) - AS (VALUES ('new york', '147ae147-ae14-4b00-8000-000000000004'::uuid, 'promo_code', now()::timestamp, 0), ('new york', '147ae147-ae14-4b00-8000-000000000004'::uuid, 'new_promo', now()::timestamp, 2)) - INSERT INTO user_promo_codes (city, user_id, code, "timestamp", usage_count) - (SELECT DISTINCT ON(city, user_id, code) * FROM inputrows) - ON CONFLICT (city, user_id, code) - DO NOTHING; -~~~ - -The `DISTINCT ON` clause does not guarantee which of the duplicates is -considered. To force the selection of a particular duplicate, use an -`ORDER BY` clause: - -{% include_cached copy-clipboard.html %} -~~~ sql -> WITH inputrows (city, user_id, code, "timestamp", usage_count) - AS (VALUES ('new york', '147ae147-ae14-4b00-8000-000000000004'::uuid, 'promo_code', now()::timestamp, 0), ('new york', '147ae147-ae14-4b00-8000-000000000004'::uuid, 'new_promo', now()::timestamp, 2)) - INSERT INTO user_promo_codes (city, user_id, code, "timestamp", usage_count) - (SELECT DISTINCT ON(city, user_id, code) * FROM inputrows ORDER BY (city, user_id, code, usage_count)) - ON CONFLICT (city, user_id, code) - DO NOTHING; -~~~ - -{{site.data.alerts.callout_info}} -Using `DISTINCT ON` incurs a performance cost to search and eliminate duplicates. -For best performance, avoid using it when the input is known to not contain duplicates. -{{site.data.alerts.end}} - -## See also - -- [Ordering of rows in DML statements](order-by.html#ordering-rows-in-dml-statements) -- [Selection Queries](selection-queries.html) -- [`DELETE`](delete.html) -- [`UPDATE`](update.html) -- [`UPSERT`](upsert.html) -- [`TRUNCATE`](truncate.html) -- [`ALTER TABLE`](alter-table.html) -- [`DROP TABLE`](drop-table.html) -- [`DROP DATABASE`](drop-database.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/install-client-drivers.md b/src/current/v21.1/install-client-drivers.md deleted file mode 100644 index 83caee74a97..00000000000 --- a/src/current/v21.1/install-client-drivers.md +++ /dev/null @@ -1,398 +0,0 @@ ---- -title: Install a Driver or ORM Framework -summary: CockroachDB supports both native drivers and the PostgreSQL wire protocol, so you can use most available PostgreSQL client drivers and ORM frameworks. -toc: true ---- - -CockroachDB supports both native drivers and the PostgreSQL wire protocol, so most available PostgreSQL client drivers and ORM frameworks should work with CockroachDB. Choose a language for supported clients, and follow the installation steps. After you install a client library, you can [connect to the database](connect-to-the-database.html). - -{{site.data.alerts.callout_info}} -Applications may encounter incompatibilities when using advanced or obscure features of a driver or ORM framework with **beta-level** support. If you encounter problems, please [open an issue](https://github.com/cockroachdb/cockroach/issues/new) with details to help us make progress toward full support. -{{site.data.alerts.end}} - -
- - - - - - - -
- -
- -## Python Drivers - -### psycopg2 - -**Support level:** Full - -To install the Python psycopg2 driver: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ pip install psycopg2 -~~~ - -For other ways to install psycopg2, see the [official documentation](http://initd.org/psycopg/docs/install.html). - -For a simple but complete example app, see [Build a Python App with CockroachDB and psycopg2](build-a-python-app-with-cockroachdb.html). - -## Python ORM frameworks - -### SQLAlchemy - -**Support level:** Full - -To install SQLAlchemy and a [CockroachDB Python package](https://github.com/cockroachdb/sqlalchemy-cockroachdb) that accounts for some differences between CockroachDB and PostgreSQL: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ pip install sqlalchemy sqlalchemy-cockroachdb psycopg2 -~~~ - -{{site.data.alerts.callout_success}} -You can substitute psycopg2 for other alternatives that include the psycopg python package. -{{site.data.alerts.end}} - -For other ways to install SQLAlchemy, see the [official documentation](http://docs.sqlalchemy.org/en/latest/intro.html#installation-guide). - -For a simple but complete example app, see [Build a Python App with CockroachDB and SQLAlchemy](build-a-python-app-with-cockroachdb-sqlalchemy.html). - -### Django - -**Support level:** Full - -CockroachDB supports Django versions 3.1+. - -To install [Django](https://docs.djangoproject.com/en/4.0/topics/install/): - -{% include_cached copy-clipboard.html %} -~~~ shell -$ pip install django==3.1.* -~~~ - -Before installing the [CockroachDB backend for Django](https://github.com/cockroachdb/django-cockroachdb), you must install one of the following psycopg2 prerequisites: - -- [psycopg2](https://pypi.org/project/psycopg2/), which has some - [prerequisites](https://www.psycopg.org/docs/install.html#prerequisites) of - its own. This package is recommended for production environments. - -- [psycopg2-binary](https://pypi.org/project/psycopg2-binary/). This package is recommended for development and testing. - -After you install the psycopg2 prerequisite, you can install the CockroachDB Django backend: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ pip install django-cockroachdb==3.1.* -~~~ - -{{site.data.alerts.callout_info}} -The major version of `django-cockroachdb` must correspond to the major version of `django`. The minor release numbers do not need to match. -{{site.data.alerts.end}} - -For a simple but complete example app, see [Build a Python App with CockroachDB and Django](build-a-python-app-with-cockroachdb-django.html). - -### PonyORM - -**Support level:** Full - -To install PonyORM: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ pip install pony -~~~ - -For a simple but complete example app, see [Build a Python App with CockroachDB and PonyORM](build-a-python-app-with-cockroachdb-pony.html). - -### peewee - -**Support level:** Full - -To install peewee: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ pip install peewee -~~~ - -For instructions on using peewee with CockroachDB, see the [CockroachDatabase peewee extension documentation](http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#cockroach-database). - -
- -
- -{% include {{page.version.version}}/app/java-version-note.md %} - -{% include {{page.version.version}}/app/java-tls-note.md %} - -## Java Drivers - -### JDBC - -**Support level:** Full - -Download and set up the Java JDBC driver as described in the [official documentation](https://jdbc.postgresql.org/documentation/setup/). We recommend using the PostgreSQL JDBC 42.2.9 driver. - -For a simple but complete example app, see [Build a Java App with CockroachDB and JDBC](build-a-java-app-with-cockroachdb.html). - -## Java ORM frameworks - -### Hibernate - -**Support level:** Full - -You can use [Gradle](https://gradle.org/install) or [Maven](https://maven.apache.org/install.html) to get all dependencies for your application, including Hibernate. Only Hibernate versions 5.4.19 and later support the Hibernate CockroachDB dialect. - -If you are using Gradle, add the following to your `dependencies`: - -~~~ groovy -implementation 'org.hibernate:hibernate-core:5.4.19.Final' -implementation 'org.postgresql:postgresql:42.2.9' -~~~ - -For a simple but complete example app that uses Gradle for dependency management, see [Build a Java App with CockroachDB and Hibernate](build-a-java-app-with-cockroachdb-hibernate.html). - -If you are using Maven, add the following to your ``: - -~~~ xml - - org.hibernate - hibernate-core - 5.4.19.Final - - - org.postgresql - postgresql - -~~~ - -For a complete example app that uses Maven for dependency management, see [Build a Spring App with CockroachDB and Spring Data JPA (Hibernate)](build-a-spring-app-with-cockroachdb-jpa.html). - -You will also need to specify the CockroachDB dialect in your [Hibernate configuration file](https://www.tutorialspoint.com/hibernate/hibernate_configuration.htm). Versions of the Hibernate CockroachDB dialect correspond to the version of CockroachDB installed on your machine. For example, `org.hibernate.dialect.CockroachDB201Dialect` corresponds to CockroachDB v20.1 and later, and `org.hibernate.dialect.CockroachDB192Dialect` corresponds to CockroachDB v19.2 and later. - -All dialect versions are forward-compatible (e.g., CockroachDB v20.1 is compatible with `CockroachDB192Dialect`), as long as your application is not affected by any backward-incompatible changes listed in your CockroachDB version's [release notes](../releases/index.html). In the event of a CockroachDB version upgrade, using a previous version of the CockroachDB dialect will not break an application, but, to enable all features available in your version of CockroachDB, we recommend keeping the dialect version in sync with the installed version of CockroachDB. - -Not all versions of CockroachDB have a corresponding dialect yet. Use the dialect number that is closest to your installed version of CockroachDB. For example, use `CockroachDB201Dialect` when using CockroachDB v21.1. - -### jOOQ - -**Support level:** Full - -You can use [Gradle](https://gradle.org/install) or [Maven](https://maven.apache.org/install.html) to get all dependencies for your application, including jOOQ. - -For a simple but complete example app that uses Maven for dependency management, see [Build a Java App with CockroachDB and jOOQ](build-a-java-app-with-cockroachdb-jooq.html). - -
- -
- -## Go Drivers - -### pgx - -**Support level:** Full - -To install the [Go pgx driver](https://pkg.go.dev/github.com/jackc/pgx): - -{% include_cached copy-clipboard.html %} -~~~ shell -$ go get -u github.com/jackc/pgx -~~~ - -For a simple but complete example app, see [Build a Go App with CockroachDB and the Go pgx Driver](build-a-go-app-with-cockroachdb.html). - -### pq - -**Support level:** Full - -To install the [Go pq driver](https://godoc.org/github.com/lib/pq): - -{% include_cached copy-clipboard.html %} -~~~ shell -$ go get -u github.com/lib/pq -~~~ - -For a simple but complete example app, see [Build a Go App with CockroachDB and the Go pq Driver](build-a-go-app-with-cockroachdb.html). - -## Go ORM frameworks - -### GORM - -**Support level:** Full - -To install [GORM](http://gorm.io): - -{% include_cached copy-clipboard.html %} -~~~ shell -$ go get -u github.com/lib/pq # dependency -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ go get -u github.com/jinzhu/gorm -~~~ - -For a simple but complete example app, see [Build a Go App with CockroachDB and GORM](build-a-go-app-with-cockroachdb-gorm.html). - -
- -
- -## Ruby Drivers - -### pg - -**Support level:** Full - -To install the [Ruby pg driver](https://rubygems.org/gems/pg): - -{% include_cached copy-clipboard.html %} -~~~ shell -$ gem install pg -~~~ - -For a simple but complete example app, see [Build a Ruby App with CockroachDB and the Ruby pg Driver](build-a-ruby-app-with-cockroachdb.html). - -## Ruby ORM frameworks - -### ActiveRecord - -**Support level:** Full (5.2), Beta (6.0) - -To install ActiveRecord, the [pg driver](https://rubygems.org/gems/pg), and a [CockroachDB Ruby package](https://github.com/cockroachdb/activerecord-cockroachdb-adapter) that accounts for some minor differences between CockroachDB and PostgreSQL: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ gem install activerecord pg activerecord-cockroachdb-adapter -~~~ - -{{site.data.alerts.callout_info}} -The exact command above will vary depending on the desired version of ActiveRecord. Specifically, version 5.1.x of ActiveRecord requires version 0.2.x of the adapter; version 5.2.x of ActiveRecord requires version 5.2.x of the adapter; version 6.0.x of ActiveRecord requires version 6.0.0betax of the adapter. -{{site.data.alerts.end}} - -For a simple but complete example app, see [Build a Ruby App with CockroachDB and ActiveRecord](build-a-ruby-app-with-cockroachdb-activerecord.html). - -
- -
- -## JavaScript Drivers - -### pg - -**Support level:** Beta - -To install the [Node.js pg driver](https://www.npmjs.com/package/pg): - -{% include_cached copy-clipboard.html %} -~~~ shell -$ npm install pg -~~~ - -Some apps might also require [`async`](https://www.npmjs.com/package/async): - -{% include_cached copy-clipboard.html %} -~~~ shell -$ npm install async -~~~ - -For a simple but complete example app, see [Build a Node.js App with CockroachDB and the Node.js pg Driver](build-a-nodejs-app-with-cockroachdb.html). - -## JavaScript/TypeScript ORM frameworks - -### Sequelize - -**Support level:** Beta - -To install Sequelize and a [CockroachDB Node.js package](https://github.com/cockroachdb/sequelize-cockroachdb) that accounts for some minor differences between CockroachDB and PostgreSQL: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ npm install sequelize sequelize-cockroachdb -~~~ - -For a simple but complete example app, see [Build a Node.js App with CockroachDB and Sequelize](build-a-nodejs-app-with-cockroachdb-sequelize.html). - -### TypeORM - -**Support level:** Full - -Install TypeORM as described in the [official documentation](https://typeorm.io/#/). - -For a simple but complete example app, see [Build a TypeScript App with CockroachDB and TypeORM](build-a-typescript-app-with-cockroachdb.html). - -
- -
- -## C Drivers - -### libpq - -**Support level:** Beta - -Install the C libpq driver as described in the [official documentation](https://www.postgresql.org/docs/current/libpq.html). - -
- -
- -## C# Drivers - -### Npgsql - -**Support level:** Beta - -1. Create a .NET project: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ dotnet new console -o cockroachdb-test-app - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cd cockroachdb-test-app - ~~~ - - The `dotnet` command creates a new app of type `console`. The `-o` parameter creates a directory named `cockroachdb-test-app` where your app will be stored and populates it with the required files. The `cd cockroachdb-test-app` command puts you into the newly created app directory. - -2. Install the latest version of the [Npgsql driver](https://www.nuget.org/packages/Npgsql/) into the .NET project using the built-in nuget package manager: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ dotnet add package Npgsql - ~~~ - -For a simple but complete example app, see [Build a C# App with CockroachDB and the .NET Npgsql Driver](build-a-csharp-app-with-cockroachdb.html). - -
- -
- -## Rust Drivers - -### rust-postgres - -**Support level:** Beta - -Install the Rust Postgres driver as described in the [official documentation](https://crates.io/crates/postgres). - -For a simple but complete example app, see [Build a Rust App with CockroachDB and the Rust Postgres Driver](build-a-rust-app-with-cockroachdb.html). - -
- -## What's next? - -- [Connect to CockroachDB](connect-to-the-database.html) -- [Design a Database Schema](schema-design-overview.html) - -You might also be interested in the following pages: - -- [Third party database tools](third-party-database-tools.html) -- [Connection parameters](connection-parameters.html) -- [Transactions](transactions.html) -- [Performance best practices](performance-best-practices-overview.html) diff --git a/src/current/v21.1/install-cockroachdb-linux.md b/src/current/v21.1/install-cockroachdb-linux.md deleted file mode 100644 index 252d227b3a7..00000000000 --- a/src/current/v21.1/install-cockroachdb-linux.md +++ /dev/null @@ -1,226 +0,0 @@ ---- -title: Install CockroachDB on Linux -summary: Install CockroachDB on Mac, Linux, or Windows. Sign up for product release notes. -tags: download, binary, homebrew -toc: true -key: install-cockroachdb.html ---- - -
- - - -
- -

See Release Notes for what's new in the latest release, {{ page.release_info.version }}. To upgrade to this release from an older version, see Cluster Upgrade.

- -{% include cockroachcloud/use-cockroachcloud-instead.md %} - -

Install options

- -Use one of the options below to install CockroachDB. - -
-

Download the binary

- {% include {{ page.version.version }}/misc/linux-binary-prereqs.md %} -
    -
  1. -

    Download the CockroachDB archive for Linux and the supporting libraries that are used to provide spatial features, and copy the binary into your PATH so you can execute cockroach commands from any shell:

    -
    - icon/buttons/copy - -
    - - 
    $ curl https://binaries.cockroachdb.com/cockroach-{{ page.release_info.version }}.linux-amd64.tgz | tar -xz && sudo cp -i cockroach-{{ page.release_info.version }}.linux-amd64/cockroach /usr/local/bin/
    -

    If you get a permissions error, prefix the command with sudo.

    -
    2. -
    Note:
    -

    If you plan to use CockroachDB's spatial features, you must complete the following steps. Otherwise, your installation is now complete.

    -
    -
  2. -

    CockroachDB uses custom-built versions of the GEOS libraries. Copy these libraries to one of the locations where CockroachDB expects to find them.

    -

    By default, CockroachDB looks for external libraries in /usr/local/lib/cockroach or a lib subdirectory of the CockroachDB binary's current directory. If you place these libraries in another location, you must pass the location in the --spatial-libs flag to cockroach start. The instructions below assume the /usr/local/lib/cockroach location.

    -
      -
    1. Create the directory where the external libraries will be stored:

      -
      - icon/buttons/copy - -
      - 
      mkdir -p /usr/local/lib/cockroach
      -
      2. -
    2. Copy the library files to the directory:

      -
      - icon/buttons/copy - -
      - 
      cp -i cockroach-{{ page.release_info.version }}.linux-amd64/lib/libgeos.so /usr/local/lib/cockroach/
      -
      - icon/buttons/copy - -
      - 
      cp -i cockroach-{{ page.release_info.version }}.linux-amd64/lib/libgeos_c.so /usr/local/lib/cockroach/
      -

      If you get a permissions error, prefix the command with sudo.

      -
      3. -
    -
    3. -

  3. Verify that CockroachDB can execute spatial queries.

    -
      -

    1. Make sure the cockroach binary you just installed is the one that runs when you type cockroach in your shell:

      -
      - icon/buttons/copy - -
      - - 
      which cockroach
      - 
      /usr/local/bin/cockroach
      -
      2. -

    2. Start a temporary, in-memory cluster using cockroach demo:

      -
      - icon/buttons/copy - -
      - - 
      cockroach demo
      - -

    3. In the demo cluster's interactive SQL shell, run the following command to test that the spatial libraries have loaded properly:

      -
      - icon/buttons/copy - -
      - - 
      > SELECT ST_IsValid(ST_MakePoint(1,2));
      -

      You should see the following output:

      - 
        st_isvalid
---------------
-true
-(1 row)
      -

      If your cockroach binary is not properly accessing the dynamically linked C libraries in /usr/local/lib/cockroach, it will output an error message like the one below.

      - 
      ERROR: st_isvalid(): geos: error during GEOS init: geos: cannot load GEOS from dir "/usr/local/lib/cockroach": failed to execute dlopen
-          Failed running "sql"
      -
      4. -
    -
  4. -

    Keep up-to-date with CockroachDB releases and best practices:

    -{% include marketo-install.html uid="1" %} -
    5. -
-
- -
-

Use Kubernetes

- -

To orchestrate CockroachDB using Kubernetes, either with configuration files or the Helm package manager, use the following tutorials:

- - -
- -
-

Use Docker

- - {{site.data.alerts.callout_danger}}Running a stateful application like CockroachDB in Docker is more complex and error-prone than most uses of Docker. Unless you are very experienced with Docker, we recommend starting with a different installation and deployment method.{{site.data.alerts.end}} - -
    -
  1. -

    Install Docker for Linux. Please carefully check that you meet all prerequisites.

    -
    2. -
  2. -

    Confirm that the Docker daemon is running in the background:

    - -
    - icon/buttons/copy - -
    - 
    $ docker version
    -

    If you do not see the server listed, start the Docker daemon.

    - - {{site.data.alerts.callout_info}}On Linux, Docker needs sudo privileges.{{site.data.alerts.end}} -
    3. -
  3. -

    Pull the image for the {{page.release_info.version}} release of CockroachDB from Docker Hub:

    - -
    - icon/buttons/copy - -
    - 
    $ sudo docker pull {{page.release_info.docker_image}}:{{page.release_info.version}}
    -
    -
    4. -
  4. -

    Keep up-to-date with CockroachDB releases and best practices:

    -{% include marketo-install.html uid="2" %} -
    5. -
-
- -
-

Build from Source

-
    -
  1. -

    Install the following prerequisites, as necessary:

    - -
- - - - - - - - - - - - - - - - - - - - -
C++ compilerMust support C++ 11. GCC prior to 6.0 does not work due to this issue. On macOS, Xcode should suffice.
GoVersion 1.15.11+ is required. Older versions might work via make build IGNORE_GOVERS=1.
BashVersions 4+ are preferred, but later releases from the 3.x series are also known to work.
CMakeVersions 3.17.0+ are known to work.
AutoconfVersion 2.68+ is required.
-

A 64-bit system is strongly recommended. Building or running CockroachDB on 32-bit systems has not been tested. You'll also need at least 2GB of RAM. If you plan to run our test suite, you'll need closer to 4GB of RAM.

- -
  • -

    Download the CockroachDB {{ page.release_info.version }} source archive, and extract the sources:

    - -
    - icon/buttons/copy - -
    - 
    $ curl https://binaries.cockroachdb.com/cockroach-{{page.release_info.version}}.src.tgz | tar -xz
    -
    • -

  • In the extracted directory, run make build:

    - - {% include_cached copy-clipboard.html %}
    cd cockroach-{{ page.release_info.version }}
    - - {% include_cached copy-clipboard.html %}
    make build
    - -

    The build process can take 10+ minutes, so please be patient.

    -
    • -
  • - -

    Install the cockroach binary into /usr/local/bin/ so it's easy to execute cockroach commands from any directory:

    - - {% include_cached copy-clipboard.html %}
    make install
    -

    If you get a permissions error, prefix the command with sudo.

    - -

    You can also execute the cockroach binary directly from its built location, ./src/github.com/cockroachdb/cockroach/cockroach, but the rest of the documentation assumes you have the binary on your PATH.

    -
    • -
  • -

    Keep up-to-date with CockroachDB releases and best practices:

    -{% include marketo-install.html uid="3" %} -
    • - - - -

    What's next?

    - -{% include {{ page.version.version }}/misc/install-next-steps.html %} - -{% include {{ page.version.version }}/misc/diagnostics-callout.html %} diff --git a/src/current/v21.1/install-cockroachdb-mac.md b/src/current/v21.1/install-cockroachdb-mac.md deleted file mode 100644 index f388c3fe489..00000000000 --- a/src/current/v21.1/install-cockroachdb-mac.md +++ /dev/null @@ -1,240 +0,0 @@ ---- -title: Install CockroachDB on Mac -summary: Install CockroachDB on Mac, Linux, or Windows. Sign up for product release notes. -tags: download, binary, homebrew -toc: true -key: install-cockroachdb.html ---- - -
    - - - -
    - -

    See Release Notes for what's new in the latest release, {{ page.release_info.version }}. To upgrade to this release from an older version, see Cluster Upgrade.

    - -{% include cockroachcloud/use-cockroachcloud-instead.md %} - -

    Install options

    - -Use one of the options below to install CockroachDB. - - - -
    -

    Download the binary

    -
      -
    1. -

      Download the CockroachDB archive for OS X and the supporting libraries that are used to provide spatial features, and copy the binary into your PATH so you can execute cockroach commands from any shell:

      -
      - icon/buttons/copy - -
      - 
      $ curl https://binaries.cockroachdb.com/cockroach-{{ page.release_info.version }}.darwin-10.9-amd64.tgz | tar -xJ && cp -i cockroach-{{ page.release_info.version }}.darwin-10.9-amd64/cockroach /usr/local/bin/
      -

      If you get a permissions error, prefix the command with sudo.

      -
      2. -
      Note:
      -

      If you plan to use CockroachDB's spatial features, you must complete the following steps. Otherwise, your installation is now complete.

      -
      -
    2. -

      CockroachDB uses custom-built versions of the GEOS libraries. Copy these libraries to one of the locations where CockroachDB expects to find them.

      -

      By default, CockroachDB looks for external libraries in /usr/local/lib/cockroach or a lib subdirectory of the CockroachDB binary's current directory. If you place these libraries in another location, you must pass the location in the --spatial-libs flag to cockroach start. The instructions below assume the /usr/local/lib/cockroach location.

      -
        -

      1. Create the directory where the external libraries will be stored:

        -
        - icon/buttons/copy - -
        - 
        mkdir -p /usr/local/lib/cockroach
        -
        2. -

      2. Copy the library files to the directory:

        -
        - icon/buttons/copy - -
        - 
        cp -i cockroach-{{ page.release_info.version }}.darwin-10.9-amd64/lib/libgeos.dylib /usr/local/lib/cockroach/
        -
        - icon/buttons/copy - -
        - 
        cp -i cockroach-{{ page.release_info.version }}.darwin-10.9-amd64/lib/libgeos_c.dylib /usr/local/lib/cockroach/
        -

        If you get a permissions error, prefix the command with sudo.

        -
        3. -
      -
      3. -

    3. Verify that CockroachDB can execute spatial queries.

      -
        -

      1. Make sure the cockroach binary you just installed is the one that runs when you type cockroach in your shell:

        -
        - icon/buttons/copy - -
        - - 
        which cockroach
        - 
        /usr/local/bin/cockroach
        -
        2. -

      2. Start a temporary, in-memory cluster using cockroach demo:

        -
        - icon/buttons/copy - -
        - - 
        cockroach demo
        - -

      3. In the demo cluster's interactive SQL shell, run the following command to test that the spatial libraries have loaded properly:

        -
        - icon/buttons/copy - -
        - - 
        >SELECT ST_IsValid(ST_MakePoint(1,2));
        -

        You should see the following output:

        - 
          st_isvalid
---------------
-true
-(1 row)
        -

        If your cockroach binary is not properly accessing the dynamically linked C libraries in /usr/local/lib/cockroach, it will output an error message like the one below.

        - 
        ERROR: st_isvalid(): geos: error during GEOS init: geos: cannot load GEOS from dir "/usr/local/lib/cockroach": failed to execute dlopen
-          Failed running "sql"
        -
        4. -
      -
    4. -

      Keep up-to-date with CockroachDB releases and best practices:

      -{% include marketo-install.html uid="1" %} -
      5. -
    -
    - -
    -

    Use Kubernetes

    - -

    To orchestrate CockroachDB locally using Kubernetes, either with configuration files or the Helm package manager, see Orchestrate CockroachDB Locally with Minikube.

    -
    - -
    -

    Use Docker

    - - {{site.data.alerts.callout_danger}}Running a stateful application like CockroachDB in Docker is more complex and error-prone than most uses of Docker. Unless you are very experienced with Docker, we recommend starting with a different installation and deployment method.{{site.data.alerts.end}} - -
      -
    1. -

      Install Docker for Mac. Please carefully check that you meet all prerequisites.

      -
      2. -
    2. -

      Confirm that the Docker daemon is running in the background:

      - -
      - icon/buttons/copy - -
      - 
      $ docker version
      -

      If you do not see the server listed, start the Docker daemon.

      -
      3. -
    3. -

      Pull the image for the {{page.release_info.version}} release of CockroachDB from Docker Hub:

      - -
      - icon/buttons/copy - -
      - 
      $ docker pull {{page.release_info.docker_image}}:{{page.release_info.version}}
      -
      -
      4. -
    4. -

      Keep up-to-date with CockroachDB releases and best practices:

      -{% include marketo-install.html uid="2" %} -
      5. -
    -
    - -
    -

    Build from source

    -
      -
    1. -

      Install the following prerequisites, as necessary:

      - - - - - - - - - - - - - - - - - - - - - - -
      C++ compilerMust support C++ 11. GCC prior to 6.0 does not work due to this issue. On macOS, Xcode should suffice.
      GoVersion 1.15.11+ is required. Older versions might work via make build IGNORE_GOVERS=1.
      BashVersions 4+ are preferred, but later releases from the 3.x series are also known to work.
      CMakeVersions 3.17.0+ are known to work.
      AutoconfVersion 2.68+ is required.
      -

      A 64-bit system is strongly recommended. Building or running CockroachDB on 32-bit systems has not been tested. You'll also need at least 2GB of RAM. If you plan to run our test suite, you'll need closer to 4GB of RAM.

      -
      2. -
    2. -

      Download the CockroachDB {{ page.release_info.version }} source archive, and extract the sources:

      - -
      - icon/buttons/copy - -
      - 
      $ curl https://binaries.cockroachdb.com/cockroach-{{ page.release_info.version }}.src.tgz | tar -xz
      -
      3. -

    3. In the extracted directory, run make build:

      - - {% include_cached copy-clipboard.html %}
      cd cockroach-{{ page.release_info.version }}
      - - {% include_cached copy-clipboard.html %}
      make build
      - -

      The build process can take 10+ minutes, so please be patient.

      -
      4. -
    4. -

      Install the cockroach binary into /usr/local/bin/ so it's easy to execute cockroach commands from any directory:

      - - {% include_cached copy-clipboard.html %}
      make install
      -

      If you get a permissions error, prefix the command with sudo.

      - -

      You can also execute the cockroach binary directly from its built location, ./src/github.com/cockroachdb/cockroach/cockroach, but the rest of the documentation assumes you have the binary on your PATH.

      -
      5. -
    5. -

      Keep up-to-date with CockroachDB releases and best practices:

      -{% include marketo-install.html uid="3" %} -
      6. -
    -
    - -

    What's next?

    - -{% include {{ page.version.version }}/misc/install-next-steps.html %} - -{% include {{ page.version.version }}/misc/diagnostics-callout.html %} diff --git a/src/current/v21.1/install-cockroachdb-windows.md b/src/current/v21.1/install-cockroachdb-windows.md deleted file mode 100644 index fbe54cd9932..00000000000 --- a/src/current/v21.1/install-cockroachdb-windows.md +++ /dev/null @@ -1,103 +0,0 @@ ---- -title: Install CockroachDB on Windows -summary: Install CockroachDB on Mac, Linux, or Windows. Sign up for product release notes. -tags: download, binary, homebrew -toc: true -key: install-cockroachdb.html ---- - -
    - - - -
    - -

    See Release Notes for what's new in the latest release, {{ page.release_info.version }}. To upgrade to this release from an older version, see Cluster Upgrade.

    - -{% include cockroachcloud/use-cockroachcloud-instead.md %} - -

    Install options

    - -Use one of the options below to install CockroachDB. - -
    -

    Download the executable

    - - {% include windows_warning.md %} - -
      -
    1. -

      Use PowerShell to download the CockroachDB {{ page.release_info.version }} archive for Windows and copy the binary into your PATH:

      -

      - icon/buttons/copy - -
      - 
      PS $ErrorActionPreference = "Stop"; [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12;$ProgressPreference = 'SilentlyContinue'; $null = New-Item -Type Directory -Force $env:appdata/cockroach; Invoke-WebRequest -Uri https://binaries.cockroachdb.com/cockroach-{{ page.release_info.version }}.windows-6.2-amd64.zip -OutFile cockroach.zip; Expand-Archive -Force -Path cockroach.zip; Copy-Item -Force "cockroach/cockroach-{{ page.release_info.version }}.windows-6.2-amd64/cockroach.exe" -Destination $env:appdata/cockroach; $Env:PATH += ";$env:appdata/cockroach"
      -

      We recommend adding ;$env:appdata/cockroach to the PATH variable for your system environment so you can execute cockroach commands from any shell. See Microsoft's environment variable documentation for more information.

      -
      2. -
    2. -

      Run the following command to make sure the installation succeeded:

      -

      - icon/buttons/copy - -
      - 
      PS cockroach version
      -
      3. -
    3. -

      Keep up-to-date with CockroachDB releases and best practices:

      -{% include marketo-install.html uid="1" %} -
      4. -
    -
    - -
    -

    Use Kubernetes

    - -

    To orchestrate CockroachDB locally using Kubernetes, either with configuration files or the Helm package manager, see Orchestrate CockroachDB Locally with Minikube.

    -
    - -
    -

    Use Docker

    - - {{site.data.alerts.callout_danger}}Running a stateful application like CockroachDB in Docker is more complex and error-prone than most uses of Docker. Unless you are very experienced with Docker, we recommend starting with a different installation and deployment method.{{site.data.alerts.end}} - -
      -
    1. -

      Install Docker for Windows.

      -
      Docker for Windows requires 64bit Windows 10 Pro and Microsoft Hyper-V. Please see the official documentation for more details. Note that if your system does not satisfy the stated requirements, you can try using Docker Toolbox.
      -
      2. -
    2. -

      Open PowerShell and confirm that the Docker daemon is running in the background:

      - -
      - icon/buttons/copy - -
      - 
      PS C:\Users\username> docker version
      - -

      If you do not see the server listed, start Docker for Windows.

      -
      3. -
    3. -

      Share your local drives. This makes it possible to mount local directories as data volumes to persist node data after containers are stopped or deleted.

      -
      4. -
    4. -

      Pull the image for the {{page.release_info.version}} release of CockroachDB from Docker Hub:

      - -
      - icon/buttons/copy - -
      - 
      PS C:\Users\username> docker pull {{page.release_info.docker_image}}:{{page.release_info.version}}
      -
      5. -
    5. -

      Keep up-to-date with CockroachDB releases and best practices:

      -{% include marketo-install.html uid="2" %} -
      6. -
    -
    - -

    What's next?

    - -{% include {{ page.version.version }}/misc/install-next-steps.html %} - -{% include {{ page.version.version }}/misc/diagnostics-callout.html %} diff --git a/src/current/v21.1/install-cockroachdb.md b/src/current/v21.1/install-cockroachdb.md deleted file mode 100644 index 6e742bb3164..00000000000 --- a/src/current/v21.1/install-cockroachdb.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -title: Install CockroachDB -summary: Install CockroachDB on Mac, Linux, or Windows. Sign up for product release notes. -toc: false -feedback: false ---- - - diff --git a/src/current/v21.1/int.md b/src/current/v21.1/int.md deleted file mode 100644 index 1aeec8f5037..00000000000 --- a/src/current/v21.1/int.md +++ /dev/null @@ -1,109 +0,0 @@ ---- -title: INT -summary: CockroachDB supports various signed integer data types. -toc: true ---- - -CockroachDB supports various signed integer [data types](data-types.html). - -{{site.data.alerts.callout_info}} -For instructions showing how to auto-generate integer values (e.g., to auto-number rows in a table), see [this FAQ entry](sql-faqs.html#how-do-i-auto-generate-unique-row-ids-in-cockroachdb). -{{site.data.alerts.end}} - -## Names and Aliases - -| Name | Allowed Width | Aliases | Range | -|--------+---------------+--------------------------------------------------+----------------------------------------------| -| `INT` | 64-bit | `INTEGER`
    `INT8`
    `INT64`
    `BIGINT` | -9223372036854775807 to +9223372036854775807 | -| `INT2` | 16-bit | `SMALLINT` | -32768 to +32767 | -| `INT4` | 32-bit | None | -2147483648 to +2147483647 | -| `INT8` | 64-bit | `INT` | -9223372036854775807 to +9223372036854775807 | - -## Syntax - -A constant value of type `INT` can be entered as a [numeric literal](sql-constants.html#numeric-literals). -For example: `42`, `-1234`, or `0xCAFE`. - -## Size - -The different integer types place different constraints on the range of allowable values, but all integers are stored in the same way regardless of type. Smaller values take up less space than larger ones (based on the numeric value, not the data type). - -### Considerations for 64-bit signed integers - -By default, `INT` is an alias for `INT8`, which creates 64-bit signed integers. This differs from the Postgres default for `INT`, [which is 32 bits](https://www.postgresql.org/docs/9.6/datatype-numeric.html), and may cause issues for your application if it is not written to handle 64-bit integers, whether due to the language your application is written in, or the ORM/framework it uses to generate SQL (if any). - -For example, JavaScript language runtimes represent numbers as 64-bit floats, which means that the JS runtime [can only represent 53 bits of numeric accuracy](http://2ality.com/2012/04/number-encoding.html) and thus has a max safe value of 253, or 9007199254740992. This means that the maximum size of a default `INT` in CockroachDB is much larger than JavaScript can represent as an integer. Visually, the size difference is as follows: - -``` -9223372036854775807 # INT default max value - 9007199254740991 # JS integer max value -``` - -Given the above, if a table contains a column with a default-sized `INT` value, and you are reading from it or writing to it via JavaScript, you will not be able to read and write values to that column correctly. This issue can pop up in a surprising way if you are using a framework that autogenerates both frontend and backend code (such as [twirp](https://github.com/twitchtv/twirp)). In such cases, you may find that your backend code can handle 64-bit signed integers, but the generated client/frontend code cannot. - -If your application needs to use an integer size that is different than the CockroachDB default (for these or other reasons), you can change one or both of the settings below. For example, you can set either of the below to `4` to cause `INT` and `SERIAL` to become aliases for `INT4` and `SERIAL4`, which use 32-bit integers. - -1. The `default_int_size` [session variable](set-vars.html). -2. The `sql.defaults.default_int_size` [cluster setting](cluster-settings.html). - -{{site.data.alerts.callout_success}} -If your application requires arbitrary precision numbers, use the [`DECIMAL`](decimal.html) data type. -{{site.data.alerts.end}} - -## Examples - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE ints (a INT PRIMARY KEY, b SMALLINT); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM ints; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden ---------------+-----------+-------------+----------------+-----------------------+-----------+------------ - a | INT8 | false | NULL | | {primary} | false - b | INT2 | true | NULL | | {} | false -(2 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO ints VALUES (1, 32); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM ints; -~~~ - -~~~ - a | b -----+----- - 1 | 32 -(1 row) -~~~ - -## Supported casting and conversion - -`INT` values can be [cast](data-types.html#data-type-conversions-and-casts) to any of the following data types: - -Type | Details ------|-------- -`DECIMAL` | –– -`FLOAT` | Loses precision if the `INT` value is larger than 2^53 in magnitude. -`BIT` | Converts to the binary representation of the integer value. If the value is negative, the sign bit is replicated on the left to fill the entire bit array. -`BOOL` | **0** converts to `false`; all other values convert to `true`. -`DATE` | Converts to days since the Unix epoch (Jan. 1, 1970). -`TIMESTAMP` | Converts to seconds since the Unix epoch (Jan. 1, 1970). -`INTERVAL` | Converts to seconds. -`STRING` | –– - -## See also - -- [Data Types](data-types.html) -- [`FLOAT`](float.html) -- [`DECIMAL`](decimal.html) diff --git a/src/current/v21.1/intellij-idea.md b/src/current/v21.1/intellij-idea.md deleted file mode 100644 index 7580dbe8182..00000000000 --- a/src/current/v21.1/intellij-idea.md +++ /dev/null @@ -1,94 +0,0 @@ ---- -title: Use IntelliJ IDEA with CockroachDB -summary: Learn how to use IntelliJ IDEA with a CockroachDB cluster. -toc: true ---- - -You can use CockroachDB in [IntelliJ IDEA](https://www.jetbrains.com/idea/) as a [database data source](https://www.jetbrains.com/help/idea/managing-data-sources.html#data_sources), which lets you accomplish tasks like managing your database's schema from within your IDE. - -## Support - -As of CockroachDB {{page.version.version}}, IntelliJ IDEA only has **partial support**. This means that the application is mostly functional, but its integration still has a few rough edges. - -### Versions - -The level of support in this document was tested as of the following versions: - -- CockroachDB v19.1.0-beta.20190225 -- IntelliJ IDEA Ultimate 18.1.3 -- PostgreSQL JDBC 41.1 - -{{site.data.alerts.callout_info}} -This feature should also work with other JetBrains IDEs, such as PyCharm, but Cockroach Labs has not yet tested its integration. -{{site.data.alerts.end}} - -### Warnings & Errors - -Users can expect to encounter the following behaviors when using CockroachDB within IntelliJ IDEA. - -- **Warnings** do not require any action on the user's end and can be ignored. Note that even if a message indicates that it is an "error", it can still be treated as a warning by this definition. -- **Errors** require the user to take action to resolve the problem and cannot be ignored. - -#### Warnings - -##### [XXUUU] ERROR: could not decorrelate subquery... - -DBeaver - Select CockroachDB - -Displays once per load of schema. - -
    - -##### [42883] ERROR: unknown function: pg_function_is_visible() Failed to retrieve... - -DBeaver - Select CockroachDB - -Display periodically. Does not impact functionality. - -#### Errors - -##### [42703] org.postgresql.util.PSQLException: ERROR: column "n.xmin" does not exist - -DBeaver - Select CockroachDB - -Requires setting **Introspect using JDBC metadata** ([details below](#set-cockroachdb-as-a-data-source-in-intellij)). - -
    - -## Set CockroachDB as a Data Source in IntelliJ - -1. Launch the **Database** tool window. (**View** > **Tool Windows** > **Database**) DBeaver - Select CockroachDB -1. Add a PostgreSQL data source. (**New (+)** > **Data Source** > **PostgreSQL**)DBeaver - Select CockroachDB -1. On the **General** tab, enter your database's connection string: - - Field | Value - ------|------- - **Host** | Your CockroachDB cluster's hostname - **Port** | Your CockroachDB cluster's port. By default, CockroachDB uses port **26257**. - **Database** | The database you want to connect to. Note that CockroachDB's notion of database differs from PostgreSQL's; you can see your cluster's databases through the [`SHOW DATABASES`](show-databases.html) command. - **User** | The user to connect as. By default, you can use **root**. - **Password** | If your cluster uses password authentication, enter the password. - **Driver** | Select or install **PostgreSQL** using a version greater than or equal to 41.1. (Older drivers have not been tested.) - - DBeaver - Select CockroachDB -1. Install or select a **PostgreSQL** driver. We recommend a version greater than or equal to 41.1. -1. If your cluster uses SSL authentication, go to the **SSH/SSL** tab, select **Use SSL** and provide the location of your certificate files. -1. Go to the **Options** tab, and then select **Introspect using JDBC metadata**.DBeaver - Select CockroachDB -1. Click **OK**. - -You can now use IntelliJ's [database tool window](https://www.jetbrains.com/help/idea/working-with-the-database-tool-window.html) to interact with your CockroachDB cluster. - -## Report Issues with IntelliJ IDEA & CockroachDB - -If you encounter issues other than those outlined above, please [file an issue on the `cockroachdb/cockroach` GitHub repo](https://github.com/cockroachdb/cockroach/issues/new?template=bug_report.md), including the following details about the environment where you encountered the issue: - -- CockroachDB version ([`cockroach version`](cockroach-version.html)) -- IntelliJ IDEA version -- Operating system -- Steps to reproduce the behavior -- If possible, a trace of the SQL statements sent to CockroachDB while the error is being reproduced using [SQL query logging](logging-use-cases.html#sql_exec). - -## See Also - -+ [Client connection parameters](connection-parameters.html) -+ [Third-Party Database Tools](third-party-database-tools.html) diff --git a/src/current/v21.1/interleave-in-parent.md b/src/current/v21.1/interleave-in-parent.md deleted file mode 100644 index 444fab968a2..00000000000 --- a/src/current/v21.1/interleave-in-parent.md +++ /dev/null @@ -1,181 +0,0 @@ ---- -title: INTERLEAVE IN PARENT -summary: Interleaving tables improves query performance by optimizing the key-value structure of closely related table's data. -toc: true -toc_not_nested: true ---- - -{{site.data.alerts.callout_danger}} -`INTERLEAVE IN PARENT` was deprecated in v20.2, disabled by default in v21.1, and permanently removed in v21.2. We do not recommend interleaving tables or indexes in new clusters. - -For details, see [Deprecation](#deprecation). -{{site.data.alerts.end}} - -## Deprecation - -Interleaving tables and indexes was deprecated in CockroachDB v20.2 for the following reasons: - -- Scans over tables or indexes with interleaved, child objects (i.e., interleaved tables or indexes) are much slower than scans over tables and indexes with no child objects, as the scans must traverse the parent object and all of its child objects. -- Database schema changes are slower for interleaved objects and their parents than they are for non-interleaved objects and objects with no interleaved children. For example, if you add or remove a column to a parent or child table, CockroachDB must rewrite the entire interleaved hierarchy for that table and its parents/children. -- [Internal benchmarks](https://github.com/cockroachdb/cockroach/issues/53455) have shown the performance benefits of interleaving tables and indexes are limited to a small number of use cases. - -In CockroachDB v21.1, interleaving is disabled with the `sql.defaults.interleaved_tables.enabled` [cluster setting](cluster-settings.html) set to `false` by default. - -For more details, see the [GitHub tracking issue](https://github.com/cockroachdb/cockroach/issues/52009). - -After [upgrading to v21.1](upgrade-cockroach-version.html), we recommend that you do the following: - -- [Convert any existing interleaved tables to non-interleaved tables](#convert-interleaved-tables). -- [Replace any existing interleaved secondary indexes with non-interleaved indexes](#replace-interleaved-indexes). - -{{site.data.alerts.callout_success}} -Test your [schema changes](online-schema-changes.html) in a non-production environment before implementing them in production. -{{site.data.alerts.end}} - -### Convert interleaved tables - -To convert an interleaved table to a non-interleaved table, issue an [`ALTER PRIMARY KEY`](alter-primary-key.html) statement on the table, specifying the existing primary key column(s) for the table, and no `INTERLEAVE IN PARENT` clause. - -When converting interleaved tables with `ALTER PRIMARY KEY`, note the following: - -- CockroachDB executes `ALTER PRIMARY KEY` statements as [online schema changes](online-schema-changes.html). This means that you can convert your interleaved tables to non-interleaved tables without experiencing any downtime. -- `ALTER PRIMARY KEY` statements can only convert a child table if that table is not a parent. If your cluster has child tables that are also parents, you must start from the bottom of the interleaving hierarchy and work your way up (i.e., start with child tables that are not parents). - -You can identify interleaved objects by querying the `crdb_internal.interleaved_tables` and `crdb_internal.interleaved_views` system tables. - -For example, suppose you created an interleaved hierarchy between the `customers`, `orders`, and `packages` tables, using the following [`CREATE TABLE`](create-table.html) statements: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE customers ( - id INT PRIMARY KEY, - name STRING(50) - ); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE orders ( - customer INT, - id INT, - total DECIMAL(20, 5), - PRIMARY KEY (customer, id), - CONSTRAINT fk_customer FOREIGN KEY (customer) REFERENCES customers - ) INTERLEAVE IN PARENT customers (customer); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE packages ( - customer INT, - "order" INT, - id INT, - address STRING(50), - delivered BOOL, - delivery_date DATE, - PRIMARY KEY (customer, "order", id), - CONSTRAINT fk_order FOREIGN KEY (customer, "order") REFERENCES orders - ) INTERLEAVE IN PARENT orders (customer, "order"); -~~~ - -The `INTERLEAVE IN PARENT` clauses will appear in `SHOW CREATE` statements for the `packages` and `orders` tables: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE orders; -~~~ - -~~~ - table_name | create_statement --------------+------------------------------------------------------------------------------------- - orders | CREATE TABLE public.orders ( - | customer INT8 NOT NULL, - | id INT8 NOT NULL, - | total DECIMAL(20,5) NULL, - | CONSTRAINT "primary" PRIMARY KEY (customer ASC, id ASC), - | CONSTRAINT fk_customer FOREIGN KEY (customer) REFERENCES public.customers(id), - | FAMILY "primary" (customer, id, total) - | ) INTERLEAVE IN PARENT public.customers (customer) -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE packages; -~~~ - -~~~ - table_name | create_statement --------------+-------------------------------------------------------------------------------------------------- - packages | CREATE TABLE public.packages ( - | customer INT8 NOT NULL, - | "order" INT8 NOT NULL, - | id INT8 NOT NULL, - | address STRING(50) NULL, - | delivered BOOL NULL, - | delivery_date DATE NULL, - | CONSTRAINT "primary" PRIMARY KEY (customer ASC, "order" ASC, id ASC), - | CONSTRAINT fk_order FOREIGN KEY (customer, "order") REFERENCES public.orders(customer, id), - | FAMILY "primary" (customer, "order", id, address, delivered, delivery_date) - | ) INTERLEAVE IN PARENT public.orders (customer, "order") -(1 row) -~~~ - -To convert these tables to non-interleaved tables, use `ALTER PRIMARY KEY` statements, starting at the bottom of the hierarchy (i.e., with `packages`): - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE packages ALTER PRIMARY KEY USING COLUMNS (customer, "order", id); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE orders ALTER PRIMARY KEY USING COLUMNS (customer, id); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE orders; -~~~ - -~~~ - table_name | create_statement --------------+------------------------------------------------------------------------------------- - orders | CREATE TABLE public.orders ( - | customer INT8 NOT NULL, - | id INT8 NOT NULL, - | total DECIMAL(20,5) NULL, - | CONSTRAINT "primary" PRIMARY KEY (customer ASC, id ASC), - | CONSTRAINT fk_customer FOREIGN KEY (customer) REFERENCES public.customers(id), - | UNIQUE INDEX orders_customer_id_key (customer ASC, id ASC), - | FAMILY "primary" (customer, id, total) - | ) -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE packages; -~~~ - -~~~ - table_name | create_statement --------------+-------------------------------------------------------------------------------------------------- - packages | CREATE TABLE public.packages ( - | customer INT8 NOT NULL, - | "order" INT8 NOT NULL, - | id INT8 NOT NULL, - | address STRING(50) NULL, - | delivered BOOL NULL, - | delivery_date DATE NULL, - | CONSTRAINT "primary" PRIMARY KEY (customer ASC, "order" ASC, id ASC), - | CONSTRAINT fk_order FOREIGN KEY (customer, "order") REFERENCES public.orders(customer, id), - | UNIQUE INDEX packages_customer_order_id_key (customer ASC, "order" ASC, id ASC), - | FAMILY "primary" (customer, "order", id, address, delivered, delivery_date) - | ) -(1 row) -~~~ - -### Replace interleaved indexes - -Interleaved [secondary indexes](indexes.html) cannot be converted to non-interleaved indexes. You must [drop the existing index](drop-index.html), and then [create a new index](create-index.html) without an `INTERLEAVE IN PARENT` clause. diff --git a/src/current/v21.1/internal/version-switcher-page-data.json b/src/current/v21.1/internal/version-switcher-page-data.json deleted file mode 100644 index 5ec30bf893f..00000000000 --- a/src/current/v21.1/internal/version-switcher-page-data.json +++ /dev/null @@ -1,17 +0,0 @@ ---- -layout: none ---- - -{%- capture page_folder -%}/{{ page.version.version }}/{%- endcapture -%} -{%- assign pages = site.pages | where_exp: "pages", "pages.url contains page_folder" | where_exp: "pages", "pages.name != '404.md'" -%} -{ -{%- for x in pages -%} -{%- assign key = x.url | replace: page_folder, "" -%} -{%- if x.key -%} - {%- assign key = x.key -%} -{%- endif %} - {{ key | jsonify }}: { - "url": {{ x.url | jsonify }} - }{% unless forloop.last %},{% endunless -%} -{% endfor %} -} \ No newline at end of file diff --git a/src/current/v21.1/interval.md b/src/current/v21.1/interval.md deleted file mode 100644 index 474cbb941c6..00000000000 --- a/src/current/v21.1/interval.md +++ /dev/null @@ -1,130 +0,0 @@ ---- -title: INTERVAL -summary: The INTERVAL data type stores a value that represents a span of time. -toc: true ---- - -The `INTERVAL` [data type](data-types.html) stores a value that represents a span of time. - -## Aliases - -CockroachDB supports using uninterpreted [string literals](sql-constants.html#string-literals) in contexts where an `INTERVAL` value is otherwise expected. - -## Syntax - -A constant value of type `INTERVAL` can be expressed using an -[interpreted literal](sql-constants.html#interpreted-literals), or a -string literal -[annotated with](scalar-expressions.html#explicitly-typed-expressions) -type `INTERVAL` or -[coerced to](scalar-expressions.html#explicit-type-coercions) type -`INTERVAL`. - -`INTERVAL` constants can be expressed using the following formats: - -Format | Description --------|-------- -SQL Standard | `INTERVAL 'Y-M D H:M:S'`

    Seconds and days can be expressed as integers or floats. All other input values must be expressed as integers.

    For more details, see [Details on SQL Standard input](#details-on-sql-standard-input). -ISO 8601 | `INTERVAL 'P1Y2M3DT4H5M6S'` -Traditional PostgreSQL | `INTERVAL '1 year 2 months 3 days 4 hours 5 minutes 6 seconds'` -Abbreviated PostgreSQL | `INTERVAL '1 yr 2 mons 3 d 4 hrs 5 mins 6 secs'` - -CockroachDB also supports using uninterpreted -[string literals](sql-constants.html#string-literals) in contexts -where an `INTERVAL` value is otherwise expected. - -### Details on SQL Standard input - -Without a [precision](#precision) or [duration field](#duration-fields) specified, expect the following behavior from SQL Standard input (`Y-M D H:M:S`): - -- Using a single value defines seconds only. -
    For example, `INTERVAL '1'` is parsed as `00:00:01`. -- Using two colon-separated integers defines hours and minutes. -
    For example, `INTERVAL '1:2'` is parsed as `01:02:00`. -- If the second of two colon-separated values is a float, the interval is parsed as minutes and seconds (`M:S.fff`). -
    For example, `INTERVAL '1:2.345'` is parsed as `00:01:02.345`. -- If the first element of the input directly preceding a colon is specified as a float, the interval is parsed as `D H:M`. -
    For example, `INTERVAL '1.2:03:04'` is parsed as `1 day 07:52:00`. -- If the day is omitted, no day value will be stored. -
    For example, `INTERVAL '1-2 3:4:5'` is parsed as `1 year 2 mons 03:04:05`, and `INTERVAL 1-2` is parsed as `1 year 2 mons`. -- If the year and month are omitted, no year or month value will be stored. -
    For example, `INTERVAL '1 2:3:4` is parsed as `1 day 02:03:04`. - -## Size - -An `INTERVAL` column supports values up to 24 bytes in width, but the total storage size is likely to be larger due to CockroachDB metadata. Intervals are stored internally as months, days, and microseconds. - -## Precision - - CockroachDB supports precision levels from 0 (seconds) to 6 (microseconds) for `INTERVAL` values. Precision in time values specifies the number of fractional digits retained in the seconds field. By default, `INTERVAL` values have a precision of 6 (microseconds). - -For example, specifying an `INTERVAL` value as `INTERVAL(3)` truncates the time precision to milliseconds. - -## Duration fields - - CockroachDB supports duration fields for `INTERVAL` values. You can specify `SECOND`, `MINUTE`, `HOUR`, or `DAY` units of duration in the form `INTERVAL ... ` or `INTERVAL ... TO `. - -Specifying a single duration field truncates the interval at the unit specified, defining the interval as having the duration field unit as its least-significant unit. For example, `INTERVAL '1 2:03:04' HOUR` truncates the input to an exact hour, and parses the interval as `1 day 02:00:00`. - -A single duration field can also resolve ambiguity in the input. For example, `INTERVAL '1'` parses the interval as `00:00:01` (1 second). `INTERVAL '1' MINUTE` parses the interval as `00:01:00` (1 minute). - -If the interval input is ambiguous, specifying two duration fields stores the interval in the units specified. For example, `INTERVAL '02:03' MINUTE TO SECOND` parses the interval as `00:02:03` (in minutes and seconds). Without `MINUTE TO SECOND`, the input would be parsed as `02:03:00` (in hours and minutes). - -## Example - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE intervals (a INT PRIMARY KEY, b INTERVAL); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM intervals; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden ---------------+-----------+-------------+----------------+-----------------------+-----------+------------ - a | INT8 | false | NULL | | {primary} | false - b | INTERVAL | true | NULL | | {} | false -(2 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO - intervals - VALUES (1, INTERVAL '1 year 2 months 3 days 4 hours 5 minutes 6 seconds'), - (2, INTERVAL '1-2 3 4:5:6'), - (3, '1-2 3 4:5:6'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM intervals; -~~~ - -~~~ - a | b -----+-------------------------------- - 1 | 1 year 2 mons 3 days 04:05:06 - 2 | 1 year 2 mons 3 days 04:05:06 - 3 | 1 year 2 mons 3 days 04:05:06 -(3 rows) -~~~ - -## Supported casting and conversion - -`INTERVAL` values can be [cast](data-types.html#data-type-conversions-and-casts) to any of the following data types: - -Type | Details ------|-------- -`INT` | Converts to number of seconds (second precision) -`DECIMAL` | Converts to number of seconds (microsecond precision) -`FLOAT` | Converts to number of seconds (microsecond precision) -`STRING` | Converts to `h-m-s` format (microsecond precision) -`TIME` | Converts to `HH:MM:SS.SSSSSS`, the time equivalent to the interval after midnight (microsecond precision) - -## See also - -[Data Types](data-types.html) diff --git a/src/current/v21.1/inverted-indexes.md b/src/current/v21.1/inverted-indexes.md deleted file mode 100644 index 9c900951195..00000000000 --- a/src/current/v21.1/inverted-indexes.md +++ /dev/null @@ -1,377 +0,0 @@ ---- -title: Generalized Inverted Indexes -summary: Generalized inverted indexes (GIN) improve your database's performance and usefulness by helping SQL locate data in JSONB or ARRAY columns. -toc: true -keywords: gin, gin index, gin indexes, inverted index, inverted indexes, accelerated index, accelerated indexes ---- - -Generalized inverted indexes, or GIN indexes, store mappings from values within a container column (such as a [`JSONB`](jsonb.html) document) to the row that holds that value. They are used to speed up containment searches, e.g., "show me all of the rows from this table which have a JSON column that contains the key-value pair `{"location":"NYC"}`". GIN indexes are commonly used in [document retrieval systems](https://en.wikipedia.org/wiki/Document_retrieval). - -CockroachDB stores the contents of the following data types in GIN indexes: - -- [JSONB](jsonb.html) -- [Arrays](array.html) -- [Spatial data (`GEOMETRY` and `GEOGRAPHY` types)](spatial-indexes.html) - -{{site.data.alerts.callout_success}}For a hands-on demonstration of using a GIN index to improve query performance on a JSONB column, see the JSON tutorial.{{site.data.alerts.end}} - -## How do GIN indexes work? - -Standard [indexes](indexes.html) work well for searches based on prefixes of sorted data. However, data types like [`JSONB`](jsonb.html) or [arrays](array.html) cannot be queried without a full table scan, since they do not adhere to ordinary value prefix comparison operators. `JSONB` in particular needs to be indexed in a more detailed way than what a standard index provides. This is where GIN indexes prove useful. - -GIN indexes filter on components of tokenizable data. The `JSONB` data type is built on two structures that can be tokenized: - -- **Objects** - Collections of key-value pairs where each key-value pair is a token. -- **Arrays** - Ordered lists of values where every value in the array is a token. - -For example, take the following `JSONB` value in column `person`: - -~~~ json -{ - "firstName": "John", - "lastName": "Smith", - "age": 25, - "address": { - "state": "NY", - "postalCode": "10021" - }, - "cars": [ - "Subaru", - "Honda" - ] -} -~~~ - -A GIN index for this object would have an entry per component, mapping it back to the original object: - -~~~ -"firstName": "John" -"lastName": "Smith" -"age": 25 -"address": "state": "NY" -"address": "postalCode": "10021" -"cars" : "Subaru" -"cars" : "Honda" -~~~ - -This lets you search based on subcomponents. - -### Creation - -You can use GIN indexes to improve the performance of queries using `JSONB` or `ARRAY` columns. You can create them: - -- At the same time as the table with the `INVERTED INDEX` clause of [`CREATE TABLE`](create-table.html#create-a-table-with-secondary-and-gin-indexes). -- For existing tables with [`CREATE INVERTED INDEX`](create-index.html). -- Using the following PostgreSQL-compatible syntax: - - ~~~ sql - > CREATE INDEX ON USING GIN (); - ~~~ - -### Selection - -If a query contains a filter against an indexed `JSONB` or `ARRAY` column that uses any of the supported operators, the GIN index is added to the set of index candidates. - -Because each query can use only a single index, CockroachDB selects the index it calculates will scan the fewest rows (i.e., the fastest). For more detail, check out our blog post [Index Selection in CockroachDB](https://www.cockroachlabs.com/blog/index-selection-cockroachdb-2/). - -To override CockroachDB's index selection, you can also force [queries to use a specific index](table-expressions.html#force-index-selection) (also known as "index hinting") or use an [inverted join hint](cost-based-optimizer.html#supported-join-algorithms). - -### Storage - -CockroachDB stores indexes directly in your key-value store. You can find more information in our blog post [Mapping Table Data to Key-Value Storage](https://www.cockroachlabs.com/blog/sql-in-cockroachdb-mapping-table-data-to-key-value-storage/). - -### Locking - -Tables are not locked during index creation thanks to CockroachDB's [schema change procedure](https://www.cockroachlabs.com/blog/how-online-schema-changes-are-possible-in-cockroachdb/). - -### Performance - -Indexes create a trade-off: they greatly improve the speed of queries, but slightly slow down writes (because new values have to be copied and sorted). The first index you create has the largest impact, but additional indexes only introduce marginal overhead. - -### Comparisons - -#### JSONB - -GIN indexes on `JSONB` columns support the following comparison operators: - -- "is contained by": [`<@`](functions-and-operators.html#supported-operations) -- "contains": [`@>`](functions-and-operators.html#supported-operations) -- "equals": [`=`](functions-and-operators.html#supported-operations), but only when you've reached into the JSON document with the [`->`](functions-and-operators.html#supported-operations) operator. For example: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SELECT * FROM a WHERE j ->'foo' = '"1"'; - ~~~ - - This is equivalent to using `@>`: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SELECT * FROM a WHERE j @> '{"foo": "1"}'; - ~~~ - -If you require comparisons using [`<`](functions-and-operators.html#supported-operations), [`<=`](functions-and-operators.html#supported-operations), etc., you can create an index on a computed column using your JSON payload, and then create a regular index on that. So if you wanted to write a query where the value of "foo" is greater than three, you would: - -1. Create your table with a computed column: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE TABLE test ( - id INT, - data JSONB, - foo INT AS ((data->>'foo')::INT) STORED - ); - ~~~ - -2. Create an index on your computed column: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE INDEX test_idx ON test (foo); - ~~~ - -3. Execute your query with your comparison: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SELECT * FROM test where foo > 3; - ~~~ - -#### Arrays - -GIN indexes on [`ARRAY`](array.html) columns support the following comparison operators: - -- "is contained by": [`<@`](functions-and-operators.html#supported-operations) -- "contains": [`@>`](functions-and-operators.html#supported-operations) - -## Partial GIN indexes - -{% include_cached new-in.html version="v21.1" %} You can create a [partial](partial-indexes.html) GIN index, a GIN index on a subset of `JSON`, `ARRAY`, or geospatial container column data. Just like partial indexes that use non-container data types, create a partial GIN index by including a clause that evaluates to true on a boolean predicate, like a `WHERE` clause. - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE test ( - id INT, - data JSONB, - INVERTED INDEX idx_data(data) WHERE id > 10 -); -~~~ - -## GIN indexes on `REGIONAL BY ROW` tables in multi-region databases - -{% include {{page.version.version}}/sql/indexes-regional-by-row.md %} - -For an example that uses unique indexes but applies to all indexes on `REGIONAL BY ROW` tables, see [Add a unique index to a `REGIONAL BY ROW` table](add-constraint.html#add-a-unique-index-to-a-regional-by-row-table). - -## Multi-column GIN indexes - -{% include_cached new-in.html version="v21.1" %} You can create a GIN index with multiple columns. The last indexed column must be one of the inverted types such as `JSON`, `ARRAY`, `GEOMETRY`, and `GEOGRAPHY`. All preceding columns must have types that are indexable. These indexes may be used for queries that constrain all index columns. - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE users ( - profile_id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - user_type STRING, - user_profile JSONB, - INVERTED INDEX (user_type, user_profile) -); -~~~ - -## Examples - -### Create a table with GIN index on a JSONB column - -In this example, let's create a table with a `JSONB` column and a GIN index: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE users ( - profile_id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - last_updated TIMESTAMP DEFAULT now(), - user_profile JSONB, - INVERTED INDEX user_details (user_profile) - ); -~~~ - -Then, insert a few rows of data: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO users (user_profile) VALUES - ('{"first_name": "Lola", "last_name": "Dog", "location": "NYC", "online" : true, "friends" : 547}'), - ('{"first_name": "Ernie", "status": "Looking for treats", "location" : "Brooklyn"}'), - ('{"first_name": "Carl", "last_name": "Kimball", "location": "NYC", "breed": "Boston Terrier"}' - ); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT *, jsonb_pretty(user_profile) FROM users; -~~~ -~~~ -+--------------------------------------+----------------------------------+--------------------------------------------------------------------------+------------------------------------+ -| profile_id | last_updated | user_profile | jsonb_pretty | -+--------------------------------------+----------------------------------+--------------------------------------------------------------------------+------------------------------------+ -| 81330a51-80b2-44aa-b793-1b8d84ba69c9 | 2018-03-13 18:26:24.521541+00:00 | {"breed": "Boston Terrier", "first_name": "Carl", "last_name": | { | -| | | "Kimball", "location": "NYC"} | | -| | | | "breed": "Boston Terrier", | -| | | | "first_name": "Carl", | -| | | | "last_name": "Kimball", | -| | | | "location": "NYC" | -| | | | } | -| 81c87adc-a49c-4bed-a59c-3ac417756d09 | 2018-03-13 18:26:24.521541+00:00 | {"first_name": "Ernie", "location": "Brooklyn", "status": "Looking for | { | -| | | treats"} | | -| | | | "first_name": "Ernie", | -| | | | "location": "Brooklyn", | -| | | | "status": "Looking for treats" | -| | | | } | -| ec0a4942-b0aa-4a04-80ae-591b3f57721e | 2018-03-13 18:26:24.521541+00:00 | {"first_name": "Lola", "friends": 547, "last_name": "Dog", "location": | { | -| | | "NYC", "online": true} | | -| | | | "first_name": "Lola", | -| | | | "friends": 547, | -| | | | "last_name": "Dog", | -| | | | "location": "NYC", | -| | | | "online": true | -| | | | } | -+--------------------------------------+----------------------------------+--------------------------------------------------------------------------+------------------------------------+ -(3 rows) -~~~ - -Now, run a query that filters on the `JSONB` column: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM users where user_profile @> '{"location":"NYC"}'; -~~~ -~~~ -+--------------------------------------+----------------------------------+--------------------------------------------------------------------------+ -| profile_id | last_updated | user_profile | -+--------------------------------------+----------------------------------+--------------------------------------------------------------------------+ -| 81330a51-80b2-44aa-b793-1b8d84ba69c9 | 2018-03-13 18:26:24.521541+00:00 | {"breed": "Boston Terrier", "first_name": "Carl", "last_name": | -| | | "Kimball", "location": "NYC"} | -| ec0a4942-b0aa-4a04-80ae-591b3f57721e | 2018-03-13 18:26:24.521541+00:00 | {"first_name": "Lola", "friends": 547, "last_name": "Dog", "location": | -| | | "NYC", "online": true} | -+--------------------------------------+----------------------------------+--------------------------------------------------------------------------+ -(2 rows) -~~~ - -### Add a GIN index to a table with an array column - -In this example, let's create a table with an `ARRAY` column first, and add the GIN index later: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE students ( - student_id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - marks INT ARRAY - ); -~~~ - -Insert a few rows of data: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO students (marks) VALUES - (ARRAY[10,20,50]), - (ARRAY[20,40,100]), - (ARRAY[100,20,70] - ); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM students; -~~~ - -~~~ -+--------------------------------------+--------------+ -| student_id | marks | -+--------------------------------------+--------------+ -| 11cdc77c-2f12-48d4-8bb4-ddee7c705e00 | {10,20,50} | -| | | -| 2526c746-0b32-4f6b-a2b4-7ce6d411c1c2 | {20,40,100} | -| | | -| eefdc32e-4485-45ca-9df1-80c0f42d73c0 | {100,20,70} | -| | | -+--------------------------------------+--------------+ -(3 rows) -~~~ - -Now, let’s add a GIN index to the table and run a query that filters on the `ARRAY`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE INVERTED INDEX student_marks ON students (marks); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM students where marks @> ARRAY[100]; -~~~ - -~~~ -+--------------------------------------+--------------+ -| student_id | marks | -+--------------------------------------+--------------+ -| 2526c746-0b32-4f6b-a2b4-7ce6d411c1c2 | {20,40,100} | -| | | -| eefdc32e-4485-45ca-9df1-80c0f42d73c0 | {100,20,70} | -| | | -+--------------------------------------+--------------+ -(2 rows) -~~~ - -### Create a table with a partial GIN index on a JSONB column - -In the same [`users` table in the previous example](#create-a-table-with-gin-index-on-a-jsonb-column) create a partial GIN index for online users. - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE INVERTED INDEX idx_online_users ON users(user_profile) WHERE user_profile -> 'online' = 'true'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT * FROM users WHERE user_profile -> 'online' = 'true'; -~~~ - -~~~ - profile_id | last_updated | user_profile ----------------------------------------+-------------------------------------+------------------------------------------------------------------------------------------------ - b6df0cae-d619-4a08-ab4f-2815da7b981f | 2021-04-13 20:54:35.660734+00:00:00 | {"first_name": "Lola", "friends": 547, "last_name": "Dog", "location": "NYC", "online": true} -(1 row) - -Time: 2ms total (execution 2ms / network 0ms) -~~~ - -Now, use index hinting with the partial GIN index. - -~~~ sql -SELECT * FROM users@idx_online_users WHERE user_profile->'online' = 'true' AND user_profile->'location' = '"NYC"'; -~~~ - -~~~ - profile_id | last_updated | user_profile ----------------------------------------+-------------------------------------+------------------------------------------------------------------------------------------------ - ea1db57e-51c3-449d-b928-adab11191085 | 2021-04-14 20:45:39.960443+00:00:00 | {"first_name": "Lola", "friends": 547, "last_name": "Dog", "location": "NYC", "online": true} -(1 row) - -Time: 2ms total (execution 2ms / network 0ms) -~~~ - -### Inverted join examples - -{% include {{ page.version.version }}/sql/inverted-joins.md %} - -## See also - -- [`JSONB`](jsonb.html) -- [`ARRAY`](array.html) -- [JSON tutorial](demo-json-support.html) -- [Computed Columns](computed-columns.html) -- [`CREATE INDEX`](create-index.html) -- [`DROP INDEX`](drop-index.html) -- [`RENAME INDEX`](rename-index.html) -- [`SHOW INDEX`](show-index.html) -- [Indexes](indexes.html) -- [SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/joins.md b/src/current/v21.1/joins.md deleted file mode 100644 index 91e95d8ccba..00000000000 --- a/src/current/v21.1/joins.md +++ /dev/null @@ -1,197 +0,0 @@ ---- -title: JOIN expressions -summary: JOIN expressions combine data from two or more table expressions. -toc: true -keywords: gin, gin index, gin indexes, inverted index, inverted indexes, accelerated index, accelerated indexes ---- - -`JOIN` expressions, also called "joins", combine the results of two or more table expressions based on conditions on the values of particular columns (i.e., equality columns). - -`JOIN` expressions define a data source in the `FROM` sub-clause of [simple `SELECT` clauses](select-clause.html), or as parameter to [`TABLE`](selection-queries.html#table-clause). Joins are a particular kind of [table expression](table-expressions.html). - -{{site.data.alerts.callout_success}} -The [cost-based optimizer](cost-based-optimizer.html) supports hint syntax to force the use of a specific join algorithm. For more information, see [Join hints](cost-based-optimizer.html#join-hints). -{{site.data.alerts.end}} - -## Synopsis - -
    {% include {{ page.version.version }}/sql/generated/diagrams/joined_table.html %}
    - -
    - -## Parameters - -Parameter | Description -----------|------------ -`joined_table` | Another join expression. -`table_ref` | A [table expression](table-expressions.html). -`opt_join_hint` | A [join hint](cost-based-optimizer.html#join-hints). -`a_expr` | A [scalar expression](scalar-expressions.html) to use as [`ON` join condition](#supported-join-conditions). -`name` | A column name to use as [`USING` join condition](#supported-join-conditions) - -## Supported join types - -CockroachDB supports the following join types: - -- [Inner joins](#inner-joins) -- [Left outer joins](#left-outer-joins) -- [Right outer joins](#right-outer-joins) -- [Full outer joins](#full-outer-joins) - -### Inner joins - -Only the rows from the left and right operand that match the condition are returned. - -~~~ -
    [ INNER ] JOIN
    ON -
    [ INNER ] JOIN
    USING(, , ...) -
    NATURAL [ INNER ] JOIN
    -
    CROSS JOIN
    -~~~ - -### Left outer joins - -For every left row where there is no match on the right, `NULL` values are returned for the columns on the right. - -~~~ -
    LEFT [ OUTER ] JOIN
    ON -
    LEFT [ OUTER ] JOIN
    USING(, , ...) -
    NATURAL LEFT [ OUTER ] JOIN
    -~~~ - -### Right outer joins - -For every right row where there is no match on the left, `NULL` values are returned for the columns on the left. - -~~~ -
    RIGHT [ OUTER ] JOIN
    ON -
    RIGHT [ OUTER ] JOIN
    USING(, , ...) -
    NATURAL RIGHT [ OUTER ] JOIN
    -~~~ - -### Full outer joins - -For every row on one side of the join where there is no match on the other side, `NULL` values are returned for the columns on the non-matching side. - -~~~ -
    FULL [ OUTER ] JOIN
    ON -
    FULL [ OUTER ] JOIN
    USING(, , ...) -
    NATURAL FULL [ OUTER ] JOIN
    -~~~ - -## Supported join conditions - -CockroachDB supports the following conditions to match rows in a join: - -- No condition with `CROSS JOIN`: each row on the left is considered - to match every row on the right. -- `ON` predicates: a Boolean [scalar expression](scalar-expressions.html) - is evaluated to determine whether the operand rows match. -- `USING`: the named columns are compared pairwise from the left and - right rows; left and right rows are considered to match if the - columns are equal pairwise. -- `NATURAL`: generates an implicit `USING` condition using all the - column names that are present in both the left and right table - expressions. - -
    {{site.data.alerts.callout_danger}}NATURAL is supported for compatibility with PostgreSQL; its use in new applications is discouraged because its results can silently change in unpredictable ways when new columns are added to one of the join operands.{{site.data.alerts.end}}
    - -## Join algorithms - -CockroachDB supports the following algorithms for performing a join: - -- [Merge joins](#merge-joins) -- [Hash joins](#hash-joins) -- [Lookup joins](#lookup-joins) -- [Inverted joins](#inverted-joins) - -### Merge joins - -To perform a [merge join](https://en.wikipedia.org/wiki/Sort-merge_join) of two tables, both tables must be indexed on the equality columns, and any indexes must have the same ordering. Merge joins offer better computational performance and more efficient memory usage than [hash joins](#hash-joins). When tables and indexes are ordered for a merge, CockroachDB chooses to use merge joins over hash joins, by default. When merge conditions are not met, CockroachDB resorts to the slower hash joins. Merge joins can be used only with [distributed query processing](https://www.cockroachlabs.com/blog/local-and-distributed-processing-in-cockroachdb/). - -Merge joins are performed on the indexed columns of two tables as follows: - -1. CockroachDB checks for indexes on the equality columns and that they are ordered the same (i.e., `ASC` or `DESC`). -2. CockroachDB takes one row from each table and compares them. - - For inner joins: - - If the rows are equal, CockroachDB returns the rows. - - If there are multiple matches, the cartesian product of the matches is returned. - - If the rows are not equal, CockroachDB discards the lower-value row and repeats the process with the next row until all rows are processed. - - For outer joins: - - If the rows are equal, CockroachDB returns the rows. - - If there are multiple matches, the cartesian product of the matches is returned. - - If the rows are not equal, CockroachDB returns `NULL` for the non-matching column and repeats the process with the next row until all rows are processed. - -### Hash joins - -If a merge join cannot be used, CockroachDB uses a [hash join](https://en.wikipedia.org/wiki/Hash_join). Hash joins are computationally expensive and require additional memory. - -Hash joins are performed on two tables as follows: - -1. CockroachDB reads both tables and attempts to pick the smaller table. -2. CockroachDB creates an in-memory [hash table](https://en.wikipedia.org/wiki/Hash_table) on the smaller table. If the hash table is too large, it will spill over to disk storage (which could affect performance). -3. CockroachDB then scans the large table, looking up each row in the hash table. - -### Lookup joins - -The [cost-based optimizer](cost-based-optimizer.html) decides when it would be beneficial to use a lookup join. Lookup joins are used when there is a large imbalance in size between the two tables, as it only reads the smaller table and then looks up matches in the larger table. A lookup join requires that the right-hand (i.e., larger) table be indexed on the equality column. A [partial index](partial-indexes.html) can only be used if it contains the subset of rows being looked up. - -Lookup joins are performed on two tables as follows: - -1. CockroachDB reads each row in the small table. -2. CockroachDB then scans (or "looks up") the larger table for matches to the smaller table and outputs the matching rows. - -You can override the use of lookup joins using [join hints](cost-based-optimizer.html#join-hints). - -{{site.data.alerts.callout_info}} -To make the optimizer prefer lookup joins to merge joins when performing foreign key checks, set the `prefer_lookup_joins_for_fks` [session variable](set-vars.html) to `on`. -{{site.data.alerts.end}} - -The output of [`EXPLAIN (VERBOSE)`](explain.html#verbose-option) shows whether `equality cols are key` for lookup joins, which means that the lookup columns form a key in the target table such that each lookup has at most one result. - -### Inverted joins - -{% include_cached new-in.html version="v21.1" %} Inverted joins force the optimizer to use a join using a [GIN index](inverted-indexes.html) on the right side of the join. Inverted joins can only be used with `INNER` and `LEFT` joins. - -~~~ -
    INNER INVERTED JOIN
    ON -
    LEFT INVERTED JOIN
    ON -~~~ - -See the [cost-based optimizer examples](cost-based-optimizer.html#inverted-join-examples) for statements that use inverted joins. - -## `LATERAL` joins - -CockroachDB supports `LATERAL` subquery joins for `INNER` and `LEFT` cross joins. For more information about `LATERAL` subqueries, see [Lateral subqueries](subqueries.html#lateral-subqueries). - -## Apply joins - -Apply join is the operator that executes a lateral join if the optimizer is not able to de-correlate it (i.e., rewrite the query to use a regular join). Most of the time, the optimizer can de-correlate most queries. However, there are some cases where the optimizer cannot perform this rewrite, and `apply-join` would show up in the [`EXPLAIN`](explain.html#join-queries) output for the query. The optimizer also replaces correlated subqueries with apply joins, and therefore `apply-join` may appear in the `EXPLAIN` output even if `LATERAL` was not used. - -Apply joins are inefficient because they must be executed one row at a time. The left side row must be used to construct the right side row, and only then can the execution engine determine if the two rows should be output by the join. This corresponds to an `O(n*m)` time complexity. - -Other types of joins supported by CockroachDB (e.g., [hash join](#hash-joins), [merge join](#merge-joins), and [lookup join](#lookup-joins)) are generally much more efficient. For example, with a hash join, a hash table is constructed using rows from the smaller side of the join, and then the larger side of the join is used to probe into the hash table using the `ON` conditions of the join. This corresponds to an `O(n+m)` time complexity. - -If you see an `apply-join`, it means the optimizer was not able to perform de-correlation, and you should probably try to rewrite your query in a different way in order to get better performance. - -## Performance best practices - -{{site.data.alerts.callout_info}}CockroachDB is currently undergoing major changes to evolve and improve the performance of queries using joins. The restrictions and workarounds listed in this section will be lifted or made unnecessary over time.{{site.data.alerts.end}} - -- When no indexes can be used to satisfy a join, CockroachDB may load all the rows in memory that satisfy the condition one of the join operands before starting to return result rows. This may cause joins to fail if the join condition or other `WHERE` clauses are insufficiently selective. -- Outer joins (i.e., [left outer joins](#left-outer-joins), [right outer joins](#right-outer-joins), and [full outer joins](#full-outer-joins)) are generally processed less efficiently than [inner joins](#inner-joins). Use inner joins whenever possible. Full outer joins are the least optimized. -- Use [`EXPLAIN`](explain.html) over queries containing joins to verify that indexes are used. -- Use [indexes](indexes.html) for faster joins. - -## See also - -- [Cost-based Optimizer: Join Hints](cost-based-optimizer.html#join-hints) -- [Scalar Expressions](scalar-expressions.html) -- [Table Expressions](table-expressions.html) -- [Simple `SELECT` Clause](select-clause.html) -- [Selection Queries](selection-queries.html) -- [`EXPLAIN`](explain.html) -- [Performance Best Practices - Overview](performance-best-practices-overview.html) -- [SQL join operation (Wikipedia)](https://en.wikipedia.org/wiki/Join_(SQL)) -- [Modesty in Simplicity: CockroachDB's JOIN (CockroachDB Blog)](https://www.cockroachlabs.com/blog/cockroachdbs-first-join/) -- [On the Way to Better SQL Joins in CockroachDB (CockroachDB Blog)](https://www.cockroachlabs.com/blog/better-sql-joins-in-cockroachdb/) diff --git a/src/current/v21.1/jsonb.md b/src/current/v21.1/jsonb.md deleted file mode 100644 index ac16a058d93..00000000000 --- a/src/current/v21.1/jsonb.md +++ /dev/null @@ -1,364 +0,0 @@ ---- -title: JSONB -summary: The JSONB data type stores JSON (JavaScript Object Notation) data. -toc: true -keywords: gin, gin index, gin indexes, inverted index, inverted indexes, accelerated index, accelerated indexes ---- - -The `JSONB` [data type](data-types.html) stores JSON (JavaScript Object Notation) data as a binary representation of the `JSONB` value, which eliminates whitespace, duplicate keys, and key ordering. `JSONB` supports [GIN indexes](inverted-indexes.html). - -{{site.data.alerts.callout_success}}For a hands-on demonstration of storing and querying JSON data from a third-party API, see the JSON tutorial.{{site.data.alerts.end}} - -## Alias - -In CockroachDB, `JSON` is an alias for `JSONB`. - -{{site.data.alerts.callout_info}}In PostgreSQL, JSONB and JSON are two different data types. In CockroachDB, the JSONB / JSON data type is similar in behavior to the JSONB data type in PostgreSQL. -{{site.data.alerts.end}} - -## Considerations - -- The [primary key](primary-key.html), [foreign key](foreign-key.html), and [unique](unique.html) [constraints](constraints.html) cannot be used on `JSONB` values. -- A standard [index](indexes.html) cannot be created on a `JSONB` column; you must use a [GIN index](inverted-indexes.html). -- CockroachDB does not currently key-encode JSON values. As a result, tables cannot be [ordered by](order-by.html) `JSONB`/`JSON`-typed columns. For details, see [tracking issue](https://github.com/cockroachdb/cockroach/issues/35706). - -## Syntax - -The syntax for the `JSONB` data type follows the format specified in [RFC8259](https://tools.ietf.org/html/rfc8259). A constant value of type `JSONB` can be expressed using an -[interpreted literal](sql-constants.html#interpreted-literals) or a -string literal -[annotated with](scalar-expressions.html#explicitly-typed-expressions) -type `JSONB`. - -There are six types of `JSONB` values: - -- `null` -- Boolean -- String -- Number (i.e., [`decimal`](decimal.html), **not** the standard `int64`) -- Array (i.e., an ordered sequence of `JSONB` values) -- Object (i.e., a mapping from strings to `JSONB` values) - -Examples: - -- `'{"type": "account creation", "username": "harvestboy93"}'` -- `'{"first_name": "Ernie", "status": "Looking for treats", "location" : "Brooklyn"}'` - -{{site.data.alerts.callout_info}}If duplicate keys are included in the input, only the last value is kept.{{site.data.alerts.end}} - -## Size - -The size of a `JSONB` value is variable, but it's recommended to keep values under 1 MB to ensure performance. Above that threshold, [write amplification](https://en.wikipedia.org/wiki/Write_amplification) and other considerations may cause significant performance degradation. - -## Functions - -Function | Description ----------|------------ -`jsonb_array_elements()` | Expands a `JSONB` array to a set of `JSONB` values. -`jsonb_build_object(...)` | Builds a `JSONB` object out of a variadic argument list that alternates between keys and values. -`jsonb_each()` | Expands the outermost `JSONB` object into a set of key-value pairs. -`jsonb_object_keys()` | Returns sorted set of keys in the outermost `JSONB` object. -`jsonb_pretty()` | Returns the given `JSONB` value as a `STRING` indented and with newlines. See the [example](#retrieve-formatted-jsonb-data) below. - -For the full list of supported `JSONB` functions, see [Functions and Operators](functions-and-operators.html#jsonb-functions). - -## Operators - -Operator | Description | Example | ----------|-------------|---------| -`->` | Access a `JSONB` field, returning a `JSONB` value. | `SELECT '[{"foo":"bar"}]'::JSONB->0->'foo' = '"bar"'::JSONB;` -`->>` | Access a `JSONB` field, returning a string. | `SELECT '{"foo":"bar"}'::JSONB->>'foo' = 'bar'::STRING;` -`@>` | Tests whether the left `JSONB` field contains the right `JSONB` field. | `SELECT ('{"foo": {"baz": 3}, "bar": 2}'::JSONB @> '{"foo": {"baz":3}}'::JSONB ) = true;` - -For the full list of supported `JSONB` operators, see [Functions and Operators](functions-and-operators.html). - -## Known limitations - -If the execution of a [join](joins.html) query exceeds the limit set for [memory-buffering operations](vectorized-execution.html#disk-spilling-operations) (i.e., the value set for the `sql.distsql.temp_storage.workmem` [cluster setting](cluster-settings.html)), CockroachDB will spill the intermediate results of computation to disk. If the join operation spills to disk, and at least one of the columns is of type `JSON`, CockroachDB returns the error `unable to encode table key: *tree.DJSON`. If the memory limit is not reached, then the query will be processed without error. - -For details, see [tracking issue](https://github.com/cockroachdb/cockroach/issues/35706). - -## Examples - -### Create a Table with a `JSONB` Column - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE users ( - profile_id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - last_updated TIMESTAMP DEFAULT now(), - user_profile JSONB - ); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM users; -~~~ - -~~~ -+--------------+-----------+-------------+-------------------+-----------------------+-------------+ -| column_name | data_type | is_nullable | column_default | generation_expression | indices | -+--------------+-----------+-------------+-------------------+-----------------------+-------------+ -| profile_id | UUID | false | gen_random_uuid() | | {"primary"} | -| last_updated | TIMESTAMP | true | now() | | {} | -| user_profile | JSON | true | NULL | | {} | -+--------------+-----------+-------------+-------------------+-----------------------+-------------+ -(3 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO users (user_profile) VALUES - ('{"first_name": "Lola", "last_name": "Dog", "location": "NYC", "online" : true, "friends" : 547}'), - ('{"first_name": "Ernie", "status": "Looking for treats", "location" : "Brooklyn"}'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM users; -~~~ -~~~ -+--------------------------------------+----------------------------------+--------------------------------------------------------------------------+ -| profile_id | last_updated | user_profile | -+--------------------------------------+----------------------------------+--------------------------------------------------------------------------+ -| 33c0a5d8-b93a-4161-a294-6121ee1ade93 | 2018-02-27 16:39:28.155024+00:00 | {"first_name": "Lola", "friends": 547, "last_name": "Dog", "location": | -| | | "NYC", "online": true} | -| 6a7c15c9-462e-4551-9e93-f389cf63918a | 2018-02-27 16:39:28.155024+00:00 | {"first_name": "Ernie", "location": "Brooklyn", "status": "Looking for | -| | | treats"} | -+--------------------------------------+----------------------------------+--------------------------------------------------------------------------+ -~~~ - -### Retrieve formatted `JSONB` data - -To retrieve `JSONB` data with easier-to-read formatting, use the `jsonb_pretty()` function. For example, retrieve data from the table you created in the [first example](#create-a-table-with-a-jsonb-column): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT profile_id, last_updated, jsonb_pretty(user_profile) FROM users; -~~~ -~~~ -+--------------------------------------+----------------------------------+------------------------------------+ -| profile_id | last_updated | jsonb_pretty | -+--------------------------------------+----------------------------------+------------------------------------+ -| 33c0a5d8-b93a-4161-a294-6121ee1ade93 | 2018-02-27 16:39:28.155024+00:00 | { | -| | | "first_name": "Lola", | -| | | "friends": 547, | -| | | "last_name": "Dog", | -| | | "location": "NYC", | -| | | "online": true | -| | | } | -| 6a7c15c9-462e-4551-9e93-f389cf63918a | 2018-02-27 16:39:28.155024+00:00 | { | -| | | "first_name": "Ernie", | -| | | "location": "Brooklyn", | -| | | "status": "Looking for treats" | -| | | } | -+--------------------------------------+----------------------------------+------------------------------------+ -~~~ - -### Retrieve specific fields from a `JSONB` value - -To retrieve a specific field from a `JSONB` value, use the `->` operator. For example, retrieve a field from the table you created in the [first example](#create-a-table-with-a-jsonb-column): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT user_profile->'first_name',user_profile->'location' FROM users; -~~~ -~~~ -+----------------------------+--------------------------+ -| user_profile->'first_name' | user_profile->'location' | -+----------------------------+--------------------------+ -| "Lola" | "NYC" | -| "Ernie" | "Brooklyn" | -+----------------------------+--------------------------+ -~~~ - -You can also use the `->>` operator to return `JSONB` field values as `STRING` values: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT user_profile->>'first_name', user_profile->>'location' FROM users; -~~~ -~~~ -+-----------------------------+---------------------------+ -| user_profile->>'first_name' | user_profile->>'location' | -+-----------------------------+---------------------------+ -| Lola | NYC | -| Ernie | Brooklyn | -+-----------------------------+---------------------------+ -~~~ - -You can use the `@>` operator to filter the values in key-value pairs to return `JSONB` field values: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT user_profile->'first_name', user_profile->'location' FROM users WHERE user_profile @> '{"location":"NYC"}'; -~~~ -~~~ -+-----------------------------+---------------------------+ -| user_profile->>'first_name' | user_profile->>'location' | -+-----------------------------+---------------------------+ -| Lola | NYC | -+-----------------------------+---------------------------+ -~~~ - -For the full list of functions and operators we support, see [Functions and Operators](functions-and-operators.html). - -### Group and order `JSONB` values - -To organize your `JSONB` field values, use the `GROUP BY` and `ORDER BY` clauses with the `->>` operator. For example, organize the `first_name` values from the table you created in the [first example](#create-a-table-with-a-jsonb-column): - -For this example, we will add a few more records to the existing table. This will help us see clearly how the data is grouped. - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO users (user_profile) VALUES - ('{"first_name": "Lola", "last_name": "Kim", "location": "Seoul", "online": false, "friends": 600}'), - ('{"first_name": "Parvati", "last_name": "Patil", "location": "London", "online": false, "friends": 500}'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT user_profile->>'first_name' AS first_name, user_profile->>'location' AS location FROM users; -~~~ - -~~~ - first_name | location --------------+----------- - Ernie | Brooklyn - Lola | NYC - Parvati | London - Lola | Seoul -~~~ - -Now let’s group and order the data. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT user_profile->>'first_name' first_name, count(*) total FROM users group by user_profile->>'first_name' order by total; -~~~ - -~~~ - first_name | total --------------+------- - Ernie | 1 - Parvati | 1 - Lola | 2 -~~~ - -The `->>` operator returns `STRING` and uses string comparison rules to order the data. If you want numeric ordering, cast the resulting data to `FLOAT`. - -For the full list of functions and operators we support, see [Functions and Operators](functions-and-operators.html). - - -### Create a table with a `JSONB` column and a computed column - -{% include {{ page.version.version }}/computed-columns/jsonb.md %} - -### Create a table with a `JSONB` column and a virtual computed column - -{% include {{ page.version.version }}/computed-columns/virtual.md %} - -## Supported casting and conversion - -All `JSONB` values can be [cast](data-types.html#data-type-conversions-and-casts) to the following data type: - -- [`STRING`](string.html) - - Numeric `JSONB` values can be cast to the following numeric data types: - -- [`DECIMAL`](decimal.html) -- [`FLOAT`](float.html) -- [`INT`](int.html) - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT '100'::jsonb::int; -~~~ - -~~~ - int8 --------- - 100 -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT '100000'::jsonb::float; -~~~ - -~~~ - float8 ----------- - 100000 -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT '100.50'::jsonb::decimal; -~~~ - -~~~ - numeric ------------ - 100.50 -(1 row) -~~~ - -The [`parse_timestamp` function](functions-and-operators.html) is used to parse strings in `TIMESTAMP` format. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT parse_timestamp ('2021-09-28T10:53:25.160Z'); -~~~ - -~~~ - parse_timestamp --------------------------- -2021-09-28 10:53:25.16 -(1 row) -~~~ - -The `parse_timestamp` function can be used to retrieve string representations of timestamp data within `JSONB` columns in `TIMESTAMP` format. - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE events ( - raw JSONB, - event_created TIMESTAMP AS (parse_timestamp(raw->'event'->>'created')) VIRTUAL -); -INSERT INTO events (raw) VALUES ('{"event":{"created":"2021-09-28T10:53:25.160Z"}}'); -SELECT event_created FROM events; -~~~ - -~~~ -CREATE TABLE - - -Time: 6ms total (execution 6ms / network 0ms) - -INSERT 1 - - -Time: 9ms total (execution 9ms / network 0ms) - - event_created --------------------------- - 2021-09-28 10:53:25.16 -(1 row) - - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -## See also - -- [JSON tutorial](demo-json-support.html) -- [GIN Indexes](inverted-indexes.html) -- [Computed Columns](computed-columns.html) -- [Data Types](data-types.html) -- [Functions and Operators](functions-and-operators.html) diff --git a/src/current/v21.1/keywords-and-identifiers.md b/src/current/v21.1/keywords-and-identifiers.md deleted file mode 100644 index 4618f2da7df..00000000000 --- a/src/current/v21.1/keywords-and-identifiers.md +++ /dev/null @@ -1,48 +0,0 @@ ---- -title: Keywords & Identifiers -summary: Learn about SQL keywords and identifiers in CockroachDB. -toc: true ---- - -SQL statements consist of two fundamental components: - -- [__Keywords__](#keywords): Words with specific meaning in SQL like `CREATE`, `INDEX`, and `BOOL` -- [__Identifiers__](#identifiers): Names for things like databases and some functions - -## Keywords - -Keywords make up SQL's vocabulary and can have specific meaning in statements. Each SQL keyword that CockroachDB supports is on one of four lists: - -- [Reserved Keywords](sql-grammar.html#reserved_keyword) -- [Type Function Name Keywords](sql-grammar.html#type_func_name_keyword) -- [Column Name Keywords](sql-grammar.html#col_name_keyword) -- [Unreserved Keywords](sql-grammar.html#unreserved_keyword) - -Reserved keywords have fixed meanings and are not typically allowed as identifiers. All other types of keywords are considered non-reserved; they have special meanings in certain contexts and can be used as identifiers in other contexts. - -### Keyword uses - -Most users asking about keywords want to know more about them in terms of: - -- __Names of objects__, covered on this page in [Identifiers](#identifiers) -- __Syntax__, covered in our pages [SQL Statements](sql-statements.html) and [SQL Grammar](sql-grammar.html) - -## Identifiers - -Identifiers are most commonly used as names of objects like databases, tables, or columns—because of this, the terms "name" and "identifier" are often used interchangeably. However, identifiers also have less-common uses, such as changing column labels with `SELECT`. - -### Rules for Identifiers - -In our [SQL grammar](sql-grammar.html), all values that accept an `identifier` must: - -- Begin with a Unicode letter or an underscore (_). Subsequent characters can be letters, underscores, digits (0-9), or dollar signs ($). -- Not equal any [SQL keyword](#keywords) unless the keyword is accepted by the element's syntax. For example, [`name`](sql-grammar.html#name) accepts Unreserved or Column Name keywords. - -To bypass either of these rules, simply surround the identifier with double-quotes ("). You can also use double-quotes to preserve case-sensitivity in database, table, view, and column names. However, all references to such identifiers must also include double-quotes. - -{{site.data.alerts.callout_info}}Some statements have additional requirements for identifiers. For example, each table in a database must have a unique name. These requirements are documented on each statement's page.{{site.data.alerts.end}} - -## See also - -- [SQL Statements](sql-statements.html) -- [Full SQL Grammar](sql-grammar.html) diff --git a/src/current/v21.1/kibana.md b/src/current/v21.1/kibana.md deleted file mode 100644 index 257f28c3d78..00000000000 --- a/src/current/v21.1/kibana.md +++ /dev/null @@ -1,113 +0,0 @@ ---- -title: Monitor CockroachDB with Kibana -summary: The CockroachDB module for Metricbeat enables data visualization of CockroachDB metrics with Kibana. -toc: true ---- - -[Kibana](https://www.elastic.co/kibana/) is a platform that visualizes data on the [Elastic Stack](https://www.elastic.co/elastic-stack/). Using the [CockroachDB module for Metricbeat](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-cockroachdb.html), metrics exposed by the CockroachDB [Prometheus endpoint](monitoring-and-alerting.html#prometheus-endpoint) can be collected by Elasticsearch and visualized with Kibana. - -In this tutorial, you will enable the CockroachDB module for Metricbeat and visualize the data in Kibana. - -{{site.data.alerts.callout_success}} -For more information about using the CockroachDB module for Metricbeat, see the [Elastic documentation](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-cockroachdb.html). - -If you run into problems with this integration, please file an issue on the [Beats issue tracker](https://github.com/elastic/beats). -{{site.data.alerts.end}} - -## Prerequisites - -Either of the following: - -- Hosted [Elasticsearch Service](https://www.elastic.co/guide/en/kibana/current/get-started.html#set-up-on-cloud) with [Metricbeat configured](https://www.elastic.co/guide/en/beats/metricbeat/current/configure-cloud-id.html) -- Self-managed [Elastic Stack](https://www.elastic.co/guide/en/elastic-stack-get-started/current/get-started-elastic-stack.html) with [Metricbeat installed](https://www.elastic.co/guide/en/beats/metricbeat/7.13/metricbeat-installation-configuration.html) - -{{site.data.alerts.callout_info}} -This tutorial assumes that you have [started a secure CockroachDB cluster](secure-a-cluster.html). Note that [CockroachDB {{ site.data.products.serverless }}](../cockroachcloud/index.html) does not currently expose a compatible monitoring endpoint. -{{site.data.alerts.end}} - -## Step 1. Enable CockroachDB module - -From your Metricbeat installation directory, run: - -{% include_cached copy-clipboard.html %} -~~~ shell -./metricbeat modules enable cockroachdb -~~~ - -### Configure security certificates - -Open `modules.d/cockroachdb.yml` in your Metricbeat installation directory. - -Follow the steps in the [Elastic documentation](https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-ssl.html) to enable SSL on the CockroachDB module. - -For example, if you used [`cockroach cert`](cockroach-cert.html) to [secure your cluster](secure-a-cluster.html#step-1-generate-certificates), the YAML should look like: - -~~~ yaml -- module: cockroachdb - metricsets: ['status'] - period: 10s - hosts: ['localhost:8080'] - enabled: true - ssl.verification_mode: full - ssl.certificate_authorities: "/custom/dir/path/ca.crt" - ssl.certificate: "/custom/dir/path/client.root.crt" - ssl.key: "/custom/dir/path/client.root.key" -~~~ - -`ssl.certificate_authorities`, `ssl.certificate`, and `ssl.key` should specify the full file paths to your CA certificate, client certificate, and client key, respectively. - -## Step 2. Start Metricbeat - -Load the Kibana dashboards (this may take a few moments): - -{% include_cached copy-clipboard.html %} -~~~ shell -./metricbeat setup -~~~ - -Launch Metricbeat: - -{% include_cached copy-clipboard.html %} -~~~ shell -./metricbeat -e -~~~ - -## Step 3. View CockroachDB dashboards on Kibana - -Open the Kibana web interface and click **Dashboard**. - -Search for the CockroachDB dashboard: - -CockroachDB dashboard selection for Metricbeat - -Click the dashboard title to open the dashboard, which presents metrics on replicas and query performance: - -CockroachDB Overview dashboard for Metricbeat - -## Step 4. Run a sample workload - -To test the dashboard functionality, use [`cockroach workload`](cockroach-workload.html) to run a sample workload on the cluster. - -Initialize the workload for MovR, a fictional vehicle-sharing company: - -{% include_cached copy-clipboard.html %} -~~~ shell -cockroach workload init movr 'postgresql://root@localhost:26257?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' -~~~ - -Run the MovR workload for 5 minutes: - -{% include_cached copy-clipboard.html %} -~~~ shell -cockroach workload run movr --duration=5m 'postgresql://root@localhost:26257?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' -~~~ - -Click **Refresh**. The query metrics will appear on the dashboard: - -CockroachDB Overview dashboard for Metricbeat with SQL metrics - -## See Also - -- [Monitoring and Alerting](monitoring-and-alerting.html) -- [DB Console Overview](ui-overview.html) -- [Logging Overview](logging-overview.html) \ No newline at end of file diff --git a/src/current/v21.1/known-limitations.md b/src/current/v21.1/known-limitations.md deleted file mode 100644 index 7b0472a4c16..00000000000 --- a/src/current/v21.1/known-limitations.md +++ /dev/null @@ -1,660 +0,0 @@ ---- -title: Known Limitations in CockroachDB v21.1 -summary: Learn about newly identified limitations in CockroachDB as well as unresolved limitations identified in earlier releases. -toc: true -keywords: gin, gin index, gin indexes, inverted index, inverted indexes, accelerated index, accelerated indexes ---- - -This page describes newly identified limitations in the CockroachDB {{page.release_info.version}} release as well as unresolved limitations identified in earlier releases. - -## New limitations - -### CockroachDB does not properly optimize some left and anti joins with GIN indexes - -[Left joins](joins.html#left-outer-joins) and anti joins involving [`JSONB`](jsonb.html), [`ARRAY`](array.html), or [spatial-typed](spatial-data.html) columns with a multi-column or [partitioned](partition-by.html) [GIN index](inverted-indexes.html) will not take advantage of the index if the prefix columns of the index are unconstrained, or if they are constrained to multiple, constant values. - -To work around this limitation, make sure that the prefix columns of the index are either constrained to single constant values, or are part of an equality condition with an input column (e.g., `col1 = col2`, where `col1` is a prefix column and `col2` is an input column). - -For example, suppose you have the following [multi-region database](multiregion-overview.html) and tables: - -``` sql -CREATE DATABASE multi_region_test_db PRIMARY REGION "europe-west1" REGIONS "us-west1", "us-east1" SURVIVE REGION FAILURE; -USE multi_region_test_db; - -CREATE TABLE t1 ( - k INT PRIMARY KEY, - geom GEOMETRY -); - -CREATE TABLE t2 ( - k INT PRIMARY KEY, - geom GEOMETRY, - INVERTED INDEX geom_idx (geom) -) LOCALITY REGIONAL BY ROW; -``` - -And you [insert](insert.html) some data into the tables: - -``` sql -INSERT INTO t1 SELECT generate_series(1, 1000), 'POINT(1.0 1.0)'; -INSERT INTO t2 (crdb_region, k, geom) SELECT 'us-east1', generate_series(1, 1000), 'POINT(1.0 1.0)'; -INSERT INTO t2 (crdb_region, k, geom) SELECT 'us-west1', generate_series(1001, 2000), 'POINT(2.0 2.0)'; -INSERT INTO t2 (crdb_region, k, geom) SELECT 'europe-west1', generate_series(2001, 3000), 'POINT(3.0 3.0)'; -``` - -If you attempt a left join between `t1` and `t2` on only the geometry columns, CockroachDB will not be able to plan an [inverted join](joins.html#inverted-joins): - -``` -> EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON st_contains(t1.geom, t2.geom); - info ------------------------------------- - distribution: full - vectorized: true - - • cross join (right outer) - │ pred: st_contains(geom, geom) - │ - ├── • scan - │ estimated row count: 3,000 - │ table: t2@primary - │ spans: FULL SCAN - │ - └── • scan - estimated row count: 1,000 - table: t1@primary - spans: FULL SCAN -(15 rows) -``` - -However, if you constrain the `crdb_region` column to a single value, CockroachDB can plan an inverted join: - -``` -> EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON st_contains(t1.geom, t2.geom) AND t2.crdb_region = 'us-east1'; - info --------------------------------------------------- - distribution: full - vectorized: true - - • lookup join (left outer) - │ table: t2@primary - │ equality: (crdb_region, k) = (crdb_region,k) - │ equality cols are key - │ pred: st_contains(geom, geom) - │ - └── • inverted join (left outer) - │ table: t2@geom_idx - │ - └── • render - │ - └── • scan - estimated row count: 1,000 - table: t1@primary - spans: FULL SCAN -(18 rows) -``` - -If you do not know which region to use, you can combine queries with [`UNION ALL`](selection-queries.html#union-combine-two-queries): - -``` -> EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON st_contains(t1.geom, t2.geom) AND t2.crdb_region = 'us-east1' -UNION ALL SELECT * FROM t1 LEFT JOIN t2 ON st_contains(t1.geom, t2.geom) AND t2.crdb_region = 'us-west1' -UNION ALL SELECT * FROM t1 LEFT JOIN t2 ON st_contains(t1.geom, t2.geom) AND t2.crdb_region = 'europe-west1'; - info ----------------------------------------------------------- - distribution: full - vectorized: true - - • union all - │ - ├── • union all - │ │ - │ ├── • lookup join (left outer) - │ │ │ table: t2@primary - │ │ │ equality: (crdb_region, k) = (crdb_region,k) - │ │ │ equality cols are key - │ │ │ pred: st_contains(geom, geom) - │ │ │ - │ │ └── • inverted join (left outer) - │ │ │ table: t2@geom_idx - │ │ │ - │ │ └── • render - │ │ │ - │ │ └── • scan - │ │ estimated row count: 1,000 - │ │ table: t1@primary - │ │ spans: FULL SCAN - │ │ - │ └── • lookup join (left outer) - │ │ table: t2@primary - │ │ equality: (crdb_region, k) = (crdb_region,k) - │ │ equality cols are key - │ │ pred: st_contains(geom, geom) - │ │ - │ └── • inverted join (left outer) - │ │ table: t2@geom_idx - │ │ - │ └── • render - │ │ - │ └── • scan - │ estimated row count: 1,000 - │ table: t1@primary - │ spans: FULL SCAN - │ - └── • lookup join (left outer) - │ table: t2@primary - │ equality: (crdb_region, k) = (crdb_region,k) - │ equality cols are key - │ pred: st_contains(geom, geom) - │ - └── • inverted join (left outer) - │ table: t2@geom_idx - │ - └── • render - │ - └── • scan - estimated row count: 1,000 - table: t1@primary - spans: FULL SCAN -(54 rows) -``` - -[Tracking GitHub Issue](https://github.com/cockroachdb/cockroach/issues/59649) - -### `SET` does not `ROLLBACK` in a transaction - -{% include {{page.version.version}}/known-limitations/set-transaction-no-rollback.md %} - -## Unresolved limitations - -### Optimizer stale statistics deletion when columns are dropped - -* {% include {{page.version.version}}/known-limitations/old-multi-col-stats.md %} - -* {% include {{page.version.version}}/known-limitations/single-col-stats-deletion.md %} - -### Automatic statistics refresher may not refresh after upgrade - -{% include {{page.version.version}}/known-limitations/stats-refresh-upgrade.md %} - -### `IMPORT` into a `REGIONAL BY ROW` table - -CockroachDB does not currently support [`IMPORT`s](import.html) into [`REGIONAL BY ROW`](set-locality.html#regional-by-row) tables that are part of [multi-region databases](multiregion-overview.html). - -[Tracking GitHub Issue](https://github.com/cockroachdb/cockroach/issues/61133) - -To work around this limitation, you will need to take the following steps: - -1. In the source database, export the [`crdb_region` column](set-locality.html#crdb_region) separately when exporting your data. - - {% include_cached copy-clipboard.html %} - ~~~ sql - EXPORT INTO CSV 'nodelocal://0/src_rbr' FROM SELECT crdb_region, i from src_rbr; - ~~~ - - For more information about the syntax, see [`EXPORT`](export.html). - -1. In the destination database, create a table that has a `crdb_region` column of the right type as shown below. - - {% include_cached copy-clipboard.html %} - ~~~ sql - CREATE TABLE dest_rbr (crdb_region public.crdb_internal_region NOT NULL, i INT); - ~~~ - -1. Import the data (including the `crdb_region` column explicitly) using [`IMPORT INTO`](import-into.html): - - {% include_cached copy-clipboard.html %} - ~~~ sql - IMPORT INTO dest_rbr (crdb_region, i) CSV DATA ('nodelocal://0/src_rbr/export*.csv') - ~~~ - -1. Convert the destination table to `REGIONAL BY ROW` using [`ALTER TABLE ... ALTER COLUMN`](alter-column.html) and [`ALTER TABLE ... SET LOCALITY`](set-locality.html): - - {% include_cached copy-clipboard.html %} - ~~~ sql - ALTER TABLE dest_rbr ALTER COLUMN crdb_region SET DEFAULT default_to_database_primary_region(gateway_region())::public.crdb_internal_region; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - ALTER TABLE dest_rbr SET LOCALITY REGIONAL BY ROW AS crdb_region; - ~~~ - -### `BACKUP` of multi-region tables - -{% include {{page.version.version}}/backups/no-multiregion-table-backups.md %} - -### Differences in syntax and behavior between CockroachDB and PostgreSQL - -CockroachDB supports the [PostgreSQL wire protocol](https://www.postgresql.org/docs/current/protocol.html) and the majority of its syntax. However, CockroachDB does not support some of the PostgreSQL features or behaves differently from PostgreSQL because not all features can be easily implemented in a distributed system. - -For a list of known differences in syntax and behavior between CockroachDB and PostgreSQL, see [Features that differ from PostgreSQL](postgresql-compatibility.html#features-that-differ-from-postgresql). - -### Multiple arbiter indexes for `INSERT ON CONFLICT DO UPDATE` - -CockroachDB does not currently support multiple arbiter indexes for [`INSERT ON CONFLICT DO UPDATE`](insert.html#on-conflict-clause), and will return an error if there are multiple unique or exclusion constraints matching the `ON CONFLICT DO UPDATE` specification. - -[Tracking GitHub Issue](https://github.com/cockroachdb/cockroach/issues/53170) - -### `IMPORT` into a table with partial indexes - -CockroachDB does not currently support [`IMPORT`s](import.html) into tables with partial indexes. - -To work around this limitation: - -1. Drop any partial indexes defined on the table. -2. Perform the `IMPORT`. -3. Recreate the partial indexes. - -If you are [performing an `IMPORT` of a `PGDUMP`](migrate-from-postgres.html) with partial indexes: - -1. Drop the partial indexes on the PostgreSQL server. -2. Recreate the `PGDUMP`. -3. `IMPORT` the `PGDUMP`. -4. Add partial indexes on the CockroachDB server. - -[Tracking GitHub Issue](https://github.com/cockroachdb/cockroach/issues/50225) - -### Spatial support limitations - -CockroachDB supports efficiently storing and querying [spatial data](spatial-data.html), with the following limitations: - -- Not all [PostGIS spatial functions](https://postgis.net/docs/reference.html) are supported. - - [Tracking GitHub Issue](https://github.com/cockroachdb/cockroach/issues/49203) - -- The `AddGeometryColumn` [spatial function](functions-and-operators.html#spatial-functions) only allows constant arguments. - - [Tracking GitHub Issue](https://github.com/cockroachdb/cockroach/issues/49402) - -- The `AddGeometryColumn` spatial function only allows the `true` value for its `use_typmod` parameter. - - [Tracking GitHub Issue](https://github.com/cockroachdb/cockroach/issues/49448) - -- CockroachDB does not support the `@` operator. Instead of using `@` in spatial expressions, we recommend using the inverse, with `~`. For example, instead of `a @ b`, use `b ~ a`. - - [Tracking GitHub Issue](https://github.com/cockroachdb/cockroach/issues/56124) - -- CockroachDB does not yet support [`INSERT`](insert.html)s into the [`spatial_ref_sys` table](spatial-glossary.html#spatial-system-tables). This limitation also blocks the [`ogr2ogr -f PostgreSQL` file conversion command](https://gdal.org/programs/ogr2ogr.html#cmdoption-ogr2ogr-f). - - [Tracking GitHub Issue](https://github.com/cockroachdb/cockroach/issues/55903) - -- CockroachDB does not yet support `DECLARE CURSOR`, which prevents the `ogr2ogr` conversion tool from exporting from CockroachDB to certain formats and prevents [QGIS](https://qgis.org/en/site/) from working with CockroachDB. To work around this limitation, [export data first to CSV](export-spatial-data.html) or GeoJSON format. - - [Tracking GitHub Issue](https://github.com/cockroachdb/cockroach/issues/41412) - -- CockroachDB does not yet support Triangle or [`TIN`](https://en.wikipedia.org/wiki/Triangulated_irregular_network) spatial shapes. - - [Tracking GitHub Issue](https://github.com/cockroachdb/cockroach/issues/56196) - -- CockroachDB does not yet support Curve, MultiCurve, or CircularString spatial shapes. - - [Tracking GitHub Issue](https://github.com/cockroachdb/cockroach/issues/56199) - -- CockroachDB does not yet support [k-nearest neighbors](https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm). - - [Tracking GitHub Issue](https://github.com/cockroachdb/cockroach/issues/55227) - -- CockroachDB does not support using [schema name prefixes](sql-name-resolution.html#how-name-resolution-works) to refer to [data types](data-types.html) with type modifiers (e.g., `public.geometry(linestring, 4326)`). Instead, use fully-unqualified names to refer to data types with type modifiers (e.g., `geometry(linestring,4326)`). - - Note that, in [`IMPORT PGDUMP`](migrate-from-postgres.html) output, [`GEOMETRY` and `GEOGRAPHY`](spatial-data.html) data type names are prefixed by `public.`. If the type has a type modifier, you must remove the `public.` from the type name in order for the statements to work in CockroachDB. - - [Tracking GitHub Issue](https://github.com/cockroachdb/cockroach/issues/56492) - -### Subqueries in `SET` statements - -It is not currently possible to use a subquery in a [`SET`](set-vars.html) or [`SET CLUSTER SETTING`](set-cluster-setting.html) statement. For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET application_name = (SELECT 'a' || 'b'); -~~~ - -~~~ -ERROR: invalid value for parameter "application_name": "(SELECT 'a' || 'b')" -SQLSTATE: 22023 -DETAIL: subqueries are not allowed in SET -~~~ - -[Tracking GitHub Issue](https://github.com/cockroachdb/cockroach/issues/42896) - -### Enterprise `BACKUP` does not capture database/table/column comments - -The [`COMMENT ON`](comment-on.html) statement associates comments to databases, tables, or columns. However, the internal table (`system.comments`) in which these comments are stored is not captured by a [`BACKUP`](backup.html) of a table or database. - -As a workaround, take a cluster backup instead, as the `system.comments` table is included in cluster backups. - -[Tracking GitHub Issue](https://github.com/cockroachdb/cockroach/issues/44396) - -### Change data capture - -Change data capture (CDC) provides efficient, distributed, row-level change feeds into Apache Kafka for downstream processing such as reporting, caching, or full-text indexing. It has the following known limitations: - -{% include {{ page.version.version }}/known-limitations/cdc.md %} - -### DB Console may become inaccessible for secure clusters - -Accessing the DB Console for a secure cluster now requires login information (i.e., username and password). This login information is stored in a system table that is replicated like other data in the cluster. If a majority of the nodes with the replicas of the system table data go down, users will be locked out of the DB Console. - -### `AS OF SYSTEM TIME` in `SELECT` statements - -`AS OF SYSTEM TIME` can only be used in a top-level `SELECT` statement. That is, we do not support statements like `INSERT INTO t SELECT * FROM t2 AS OF SYSTEM TIME
    `](show-columns.html) statement or the `\d
    ` [shell command](cockroach-sql.html#commands): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM drivers; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden -+-------------+-----------+-------------+----------------+-----------------------+-----------+-----------+ - id | UUID | false | NULL | | {primary} | false - city | STRING | false | NULL | | {primary} | false - name | STRING | true | NULL | | {} | false - dl | STRING | true | NULL | | {} | false - address | STRING | true | NULL | | {} | false -(5 rows) -~~~ - - -## Insert rows - -To insert a row into a table, use [`INSERT INTO`](insert.html) followed by the table name and then the column values listed in the order in which the columns appear in the table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO drivers VALUES - ('c28f5c28-f5c2-4000-8000-000000000026', 'new york', 'Petee', 'ABC-1234', '101 5th Ave'); -~~~ - -If you want to pass column values in a different order, list the column names explicitly and provide the column values in the corresponding order: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO drivers (name, city, dl, address, id) VALUES - ('Adam Driver', 'chicago', 'DEF-5678', '201 E Randolph St', '1eb851eb-851e-4800-8000-000000000006'); -~~~ - -To insert multiple rows into a table, use a comma-separated list of parentheses, each containing column values for one row: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO drivers VALUES - ('8a3d70a3-d70a-4000-8000-00000000001b', 'seattle', 'Eric', 'GHI-9123', '400 Broad St'), - ('9eb851eb-851e-4800-8000-00000000001f', 'new york', 'Harry Potter', 'JKL-456', '214 W 43rd St'); -~~~ - -[Default values](default-value.html) are used when you leave specific columns out of your statement, or when you explicitly request default values. For example, both of the following statements create a row where the `name`, `dl`, and `address` entries each contain their default value, in this case `NULL`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO drivers (id, city) VALUES - ('70a3d70a-3d70-4400-8000-000000000016', 'chicago'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO drivers (id, city, name, dl, address) VALUES - ('b851eb85-1eb8-4000-8000-000000000024', 'seattle', DEFAULT, DEFAULT, DEFAULT); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM drivers WHERE id in ('70a3d70a-3d70-4400-8000-000000000016', 'b851eb85-1eb8-4000-8000-000000000024'); -~~~ - -~~~ - id | city | name | dl | address -+--------------------------------------+---------+------+------+---------+ - 70a3d70a-3d70-4400-8000-000000000016 | chicago | NULL | NULL | NULL - b851eb85-1eb8-4000-8000-000000000024 | seattle | NULL | NULL | NULL -(2 rows) -~~~ - -## Create an index - -[Indexes](indexes.html) help locate data without having to look through every row of a table. They're automatically created for the [primary key](primary-key.html) of a table and any columns with a [`UNIQUE` constraint](unique.html). - -To create an index for non-unique columns, use [`CREATE INDEX`](create-index.html) followed by an optional index name and an `ON` clause identifying the table and column(s) to index. For each column, you can choose whether to sort ascending (`ASC`) or descending (`DESC`). - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE INDEX name_idx ON users (name DESC); -~~~ - -You can create indexes during table creation as well; just include the `INDEX` keyword followed by an optional index name and the column(s) to index: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE IF NOT EXISTS drivers ( - id UUID NOT NULL, - city STRING NOT NULL, - name STRING, - dl STRING, - address STRING, - INDEX name_idx (name), - CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC) -); -~~~ - -## Show indexes - -To show the indexes on a table, use [`SHOW INDEX FROM`](show-index.html) followed by the name of the table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW INDEX FROM users; -~~~ - -~~~ - table_name | index_name | non_unique | seq_in_index | column_name | direction | storing | implicit -+------------+------------+------------+--------------+-------------+-----------+---------+----------+ - users | primary | false | 1 | city | ASC | false | false - users | primary | false | 2 | id | ASC | false | false - users | name_idx | true | 1 | name | DESC | false | false - users | name_idx | true | 2 | city | ASC | false | true - users | name_idx | true | 3 | id | ASC | false | true -(5 rows) -~~~ - - -## Query a table - -To query a table, use [`SELECT`](select-clause.html) followed by a comma-separated list of the columns to be returned and the table from which to retrieve the data. You can also use the [`LIMIT`](limit-offset.html) clause to restrict the number of rows retrieved: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT name FROM users LIMIT 10; -~~~ - -~~~ - name -+-------------------+ - William Wood - Victoria Jennings - Tyler Dalton - Tony Ortiz - Tina Miller - Taylor Cunningham - Susan Morse - Steven Lara - Stephen Diaz - Sarah Wang DDS -(10 rows) -~~~ - -To retrieve all columns, use the `*` wildcard: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM users LIMIT 10; -~~~ - -~~~ - id | city | name | address | credit_card -+--------------------------------------+-----------+--------------------+--------------------------------+-------------+ - c28f5c28-f5c2-4000-8000-000000000026 | amsterdam | Maria Weber | 14729 Karen Radial | 5844236997 - c7ae147a-e147-4000-8000-000000000027 | amsterdam | Tina Miller | 97521 Mark Extensions | 8880478663 - cccccccc-cccc-4000-8000-000000000028 | amsterdam | Taylor Cunningham | 89214 Jennifer Well | 5130593761 - d1eb851e-b851-4800-8000-000000000029 | amsterdam | Kimberly Alexander | 48474 Alfred Hollow | 4059628542 - 19999999-9999-4a00-8000-000000000005 | boston | Nicole Mcmahon | 11540 Patton Extensions | 0303726947 - 1eb851eb-851e-4800-8000-000000000006 | boston | Brian Campbell | 92025 Yang Village | 9016427332 - 23d70a3d-70a3-4800-8000-000000000007 | boston | Carl Mcguire | 60124 Palmer Mews Apt. 49 | 4566257702 - 28f5c28f-5c28-4600-8000-000000000008 | boston | Jennifer Sanders | 19121 Padilla Brooks Apt. 12 | 1350968125 - 80000000-0000-4000-8000-000000000019 | chicago | Matthew Clay | 49220 Lisa Junctions | 9132291015 - 851eb851-eb85-4000-8000-00000000001a | chicago | Samantha Coffey | 6423 Jessica Underpass Apt. 87 | 9437219051 -(10 rows) -~~~ - -To filter the results, add a `WHERE` clause identifying the columns and values to filter on: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT id, name FROM users WHERE city = 'san francisco'; -~~~ - -~~~ - id | name ----------------------------------------+------------------- - 75c28f5c-28f5-4400-8000-000000000017 | William Wood - 7ae147ae-147a-4000-8000-000000000018 | Alfred Garcia - 80000000-0000-4000-8000-000000000019 | Matthew Clay - 851eb851-eb85-4000-8000-00000000001a | Samantha Coffey - 8a3d70a3-d70a-4000-8000-00000000001b | Jessica Martinez -(5 rows) -~~~ - -To sort the results, add an `ORDER BY` clause identifying the columns to sort by. For each column, you can choose whether to sort ascending (`ASC`) or descending (`DESC`). - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT city, type, current_location FROM vehicles ORDER BY city, type DESC; -~~~ - -~~~ - city | type | current_location -+---------------+------------+--------------------------------+ - amsterdam | skateboard | 19202 Edward Pass - boston | scooter | 19659 Christina Ville - chicago | skateboard | 69721 Noah River - detroit | scooter | 43051 Jonathan Fords Suite 36 - los angeles | skateboard | 49164 Anna Mission Apt. 38 - minneapolis | scooter | 62609 Stephanie Route - minneapolis | scooter | 57637 Mitchell Shoals Suite 59 - new york | skateboard | 64110 Richard Crescent - new york | scooter | 86667 Edwards Valley - paris | skateboard | 2505 Harrison Parkway Apt. 89 - rome | bike | 64935 Matthew Flats Suite 55 - san francisco | skateboard | 81472 Morris Run - san francisco | scooter | 91427 Steven Spurs Apt. 49 - seattle | bike | 37754 Farmer Extension - washington dc | scooter | 47259 Natasha Cliffs -(15 rows) -~~~ - -## Update rows - -To update rows in a table, use [`UPDATE`](update.html) followed by the table name, a `SET` clause identifying the columns to update and their new values, and a `WHERE` clause identifying the rows to update: - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE promo_codes SET (description, rules) = ('EXPIRED', '{"type": "percent_discount", "value": "0%"}') WHERE expiration_time < '2019-01-22 03:04:05+00:00'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT code, description, rules FROM promo_codes LIMIT 10; -~~~ - -~~~ - code | description | rules ------------------------------+-------------+---------------------------------------------- - 0_explain_theory_something | EXPIRED | {"type": "percent_discount", "value": "0%"} - 100_address_garden_certain | EXPIRED | {"type": "percent_discount", "value": "0%"} - 101_system_skin_night | EXPIRED | {"type": "percent_discount", "value": "0%"} - 102_card_professional_kid | EXPIRED | {"type": "percent_discount", "value": "0%"} - 103_now_project_focus | EXPIRED | {"type": "percent_discount", "value": "0%"} - 104_long_become_prove | EXPIRED | {"type": "percent_discount", "value": "0%"} - 105_republican_guess_arm | EXPIRED | {"type": "percent_discount", "value": "0%"} - 106_court_especially_plan | EXPIRED | {"type": "percent_discount", "value": "0%"} - 107_she_matter_ten | EXPIRED | {"type": "percent_discount", "value": "0%"} - 108_wind_marriage_for | EXPIRED | {"type": "percent_discount", "value": "0%"} -(10 rows) -~~~ - -If a table has a primary key, you can use that in the `WHERE` clause to reliably update specific rows; otherwise, each row matching the `WHERE` clause is updated. When there's no `WHERE` clause, all rows in the table are updated. - -## Delete rows - -To delete rows from a table, use [`DELETE FROM`](delete.html) followed by the table name and a `WHERE` clause identifying the rows to delete: - -{% include_cached copy-clipboard.html %} -~~~ sql -> DELETE FROM promo_codes WHERE description = 'EXPIRED'; -~~~ -~~~ -DELETE 669 -~~~ - -Just as with the `UPDATE` statement, if a table has a primary key, you can use that in the `WHERE` clause to reliably delete specific rows; otherwise, each row matching the `WHERE` clause is deleted. When there's no `WHERE` clause, all rows in the table are deleted. - -## Remove a table - -When you no longer need a table, use [`DROP TABLE`](drop-table.html) followed by the table name to remove the table and all its data: - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP TABLE drivers; -~~~ - -## What's next? - -- Explore all [SQL Statements](sql-statements.html) -- [Use the built-in SQL client](cockroach-sql.html) to execute statements from a shell or directly from the command line -- [Install the client driver](install-client-drivers.html) for your preferred language and [build an app](example-apps.html) -- [Explore core CockroachDB features](demo-replication-and-rebalancing.html) like automatic replication, rebalancing, and fault tolerance diff --git a/src/current/v21.1/licensing-faqs.md b/src/current/v21.1/licensing-faqs.md deleted file mode 100644 index f2aa4dae8b2..00000000000 --- a/src/current/v21.1/licensing-faqs.md +++ /dev/null @@ -1,159 +0,0 @@ ---- -title: Licensing FAQs -summary: Frequently asked questions about CockroachDB licensing. -toc: true ---- - -Current CockroachDB code is primarily licensed in two ways: - -- [Business Source License (BSL)](#bsl) -- [Cockroach Community License (CCL)](#ccl) - -CockroachDB core is free to use. Most [core features](#feature-licensing) are licensed under the BSL, but some core features are subject to the CCL or third-party licenses. - -Non-CCL core features from version 19.1 and earlier are licensed under [Apache 2.0](#apache); however, some features remain under third-party licenses. Beginning in version 19.2, these non-CCL features are licensed under the BSL for three years before [converting](#license-conversion-timeline) to the Apache 2.0 license. - -CockroachDB [Enterprise features](enterprise-licensing.html) require a [paid license](#obtain-a-license) from Cockroach and are licensed under the Cockroach Community License. - -{{site.data.alerts.callout_info}} -You can find any feature's license by checking the code's file header in the [CockroachDB repository](https://github.com/cockroachdb/cockroach). -{{site.data.alerts.end}} - -## Types of licenses - -Type | Description --------------|------------ -**Apache 2.0 License** | Core features under the Apache License are free to use and fully open-source. BSL features convert to this license three years after their release. For license conversion dates, see the [table below](#license-conversion-timeline). -**Business Source License**| BSL features are free to use and the source code is available, but users may not use CockroachDB [as a service](#what-constitutes-hosting-cockroachdb-as-a-service) without an agreement with Cockroach Labs. The BSL is not certified as an open-source license, but most of the [Open Source Initiative](https://en.wikipedia.org/wiki/Open_Source_Initiative) (OSI) criteria are met. -**Cockroach
    Community License** |
    • CCL (Free) features are free to use. The source code is available to view and modify, but it cannot be reused without an agreement with Cockroach Labs.
    • CCL (Paid) features require an Enterprise license key to access. The source code is available to view and modify, but it cannot be used without an agreement with Cockroach Labs.
    - -For additional custom licensing options, [contact us](https://support.cockroachlabs.com/hc/en-us). - -For each BSL release all associated alpha, beta, major, and minor (point) releases will become Apache 2.0 on the same day three years after the major release date. Once a release is published under the BSL, the license cannot be changed to prevent code from becoming open-source at the specified change date. The following table lists the current license for non-CCL features for each published release: - -### License conversion timeline - -CockroachDB version | License | Converts to Apache 2.0 ---------------------|---------|---------------------------- -21.1 | Business Source License | May 18, 2024 -20.2 | Business Source License | Nov 10, 2023 -20.1 | Business Source License | May 12, 2023 -19.2 | Business Source License | Oct 01, 2022 -19.1 | Apache 2.0 | - -2.1 | Apache 2.0 | - -2.0 | Apache 2.0 | - - -## Feature licensing - -The table below shows how certain core and Enterprise features are licensed: - -Feature | BSL | CCL (free) | CCL (paid) ------------------|:-----:|:-----------------:|:---------------: -**[Import](import.html)** | | ✓ | -**[Export](export.html)** | ✓ | | -**[Restore](restore.html)** | | ✓ | -**[Full backups](take-full-and-incremental-backups.html#full-backups)** | | ✓ | -**[Incremental backups](take-full-and-incremental-backups.html#incremental-backups)** | | | ✓ -**[Other advanced backup features](backup.html)** | | | ✓ -**[Core changefeed](stream-data-out-of-cockroachdb-using-changefeeds.html#create-a-core-changefeed)** | | ✓ | -**[Enterprise changefeed](stream-data-out-of-cockroachdb-using-changefeeds.html#configure-a-changefeed-enterprise)** | | | ✓ -**[Table-level zone configuration](configure-replication-zones.html#replication-zone-levels)** | ✓ | | -**[Multi-Region Capabilities](multiregion-overview.html)** | | | ✓ -**[Follower reads](follower-reads.html)** | | | ✓ -**[Node map](enable-node-map.html)** | | | ✓ -**[Encryption at rest](encryption.html#encryption-at-rest-enterprise)** | | | ✓ -**[Role-based access management](authorization.html#roles)** | ✓ | | -**[Password and certificate authentication](authentication.html)** | ✓ | | -**[GSSAPI with Kerberos authentication](gssapi_authentication.html)** | | | ✓ -**[All other core features](https://www.cockroachlabs.com/compare)** | ✓ | | - -{{site.data.alerts.callout_info}} -Individual feature licensing may change with each release of CockroachDB. You can use the dropdown menu at the top of the page to view documentation for other versions of CockroachDB. -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -More information about all Enterprise features can be found [here](enterprise-licensing.html). -{{site.data.alerts.end}} - -## Obtain a license - -All CockroachDB code is included in the same binary. No license key is required to access BSL and CCL (Free) features. To access CCL (Paid) features, users have two options: - -- An **Enterprise License** enables you to use CockroachDB Enterprise features for longer periods (one year or more). To upgrade to an Enterprise license, contact Sales. -- A **Trial License** enables you to try out CockroachDB Enterprise features for 30 days for free. To obtain a trial license, fill out [the registration form](https://www.cockroachlabs.com/get-cockroachdb/enterprise/) and receive your trial license via email within a few minutes. - -{{site.data.alerts.callout_success}} -For quick local testing of Enterprise features, you can use the [`cockroach demo`](cockroach-demo.html) command, which starts a temporary, in-memory cluster with a SQL shell open and a trial license applied automatically. -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -Cockroach Labs is willing to offer self-hosted CockroachDB Enterprise features free of charge and discounted prices for CockroachDB {{ site.data.products.serverless }} to select non-profit organizations and non-commercial academic projects. To learn more, please [contact us](https://support.cockroachlabs.com/hc/en-us). -{{site.data.alerts.end}} - -## Set a license - -{% include {{ page.version.version }}/misc/set-enterprise-license.md %} - -## Verify a license - -To verify a license, open the [built-in SQL shell](cockroach-sql.html) and use the [`SHOW CLUSTER SETTING`](show-cluster-setting.html) command to check the organization name and license key: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CLUSTER SETTING cluster.organization; -~~~ -~~~ - cluster.organization -+----------------------+ - Acme Company -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CLUSTER SETTING enterprise.license; -~~~ -~~~ - enterprise.license -+-------------------------------------------+ - crl-0-ChB1x... -(1 row) -~~~ - -The license setting is also logged in the cockroach.log on the node where the command is run: - -{% include_cached copy-clipboard.html %} -~~~ sql -$ cat cockroach.log | grep license -~~~ -~~~ -I171116 18:11:48.279604 1514 sql/event_log.go:102 [client=[::1]:56357,user=root,n1] Event: "set_cluster_setting", target: 0, info: {SettingName:enterprise.license Value:xxxxxxxxxxxx User:root} -~~~ - -## Monitoring for license expiry - -You can monitor the time until your license expires with [Prometheus](monitor-cockroachdb-with-prometheus.html). The `seconds_until_enterprise_license_expiry` metric reports the number of seconds until the Enterprise license on a cluster expires. It will report 0 if there is no license or a negative number if the license has already expired. For more information, see [Monitoring and Alerting](monitoring-and-alerting.html). - -## Renew an expired license - -After your license expires, the Enterprise features stop working, but your production setup is unaffected. For example, the backup and restore features would not work until the license is renewed, but you would be able to continue using all other features of CockroachDB without interruption. - -To renew an expired license, contact Sales and then [set](#set-a-license) the new license. - -## FAQs - -### Can I host CockroachDB as a service for internal use at my organization? - -Yes, employees and contractors can use your internal CockroachDB instance as a service, but no people outside of your organization will be able to use it without purchasing a license. Use of Enterprise features will always require a license. - -### What constitutes hosting CockroachDB as a service? - -Hosting CockroachDB as a service means creating an offering that allows third parties (other than your employees and contractors) to operate a database. Specifically, third parties cannot modify table schemas. - -### I would like to reuse a single component from the CockroachDB project in my own software, which uses the AGPL or another open-source license. Is this possible? - -The CockroachDB team is committed to supporting the open-source community and willing to consider extracting specific internal components that are generally useful as a separate project with its own license, for example APL. For more details, feel free to [contact us](https://support.cockroachlabs.com/hc/en-us). - -### Can I fork the CockroachDB project pre-BSL and create my own CockroachDB derivative with a different license? - -You can fork any historical version of CockroachDB in your own project, as allowed by the license available for that version, and modify it for your purpose. Note however that only the copyright holder (Cockroach Labs) can relicense the components that you forked from: your derivative will need to keep the original license at the time of the fork. Any component you copy from a BSL-licensed CockroachDB into your project will make the BSL apply to your project as well. diff --git a/src/current/v21.1/limit-offset.md b/src/current/v21.1/limit-offset.md deleted file mode 100644 index 87daaa42578..00000000000 --- a/src/current/v21.1/limit-offset.md +++ /dev/null @@ -1,45 +0,0 @@ ---- -title: Limiting Query Results -summary: LIMIT and OFFSET restrict an operation to a few row. -toc: true ---- - -The `LIMIT` and `OFFSET` clauses restrict the operation of: - -- A [selection query](selection-queries.html), including when it occurs as part of [`INSERT`](insert.html) or [`UPSERT`](upsert.html). -- [`UPDATE`](update.html) and [`DELETE`](delete.html) statements. - - -`OFFSET` instructs the operation to skip a specified number of rows. It is often used in conjunction with `LIMIT` to "paginate" through retrieved rows. - -{{site.data.alerts.callout_danger}} -Using `LIMIT`/`OFFSET` to implement pagination can be very slow for large tables. We recommend using [keyset pagination](pagination.html) instead. -{{site.data.alerts.end}} - -## Syntax - -### LIMIT - -~~~ -LIMIT { | ALL } -~~~ - -For PostgreSQL compatibility, CockroachDB also supports `FETCH FIRST limit_value ROWS ONLY` and `FETCH NEXT limit_value ROWS ONLY` as aliases for `LIMIT`. If `limit_value` is omitted, then one row is fetched. - -### OFFSET - -~~~ -OFFSET [ ROW | ROWS ] -~~~ - -## Examples - -For example uses with `SELECT`, see [Limiting Row Count](selection-queries.html#limiting-row-count). - -## See also - -- [`DELETE`](delete.html) -- [`UPDATE`](delete.html) -- [`INSERT`](insert.html) -- [`UPSERT`](upsert.html) -- [Selection Queries](selection-queries.html) \ No newline at end of file diff --git a/src/current/v21.1/linestring.md b/src/current/v21.1/linestring.md deleted file mode 100644 index d5c62560938..00000000000 --- a/src/current/v21.1/linestring.md +++ /dev/null @@ -1,41 +0,0 @@ ---- -title: LINESTRING -summary: A LINESTRING is a collection of POINTs joined into a line. -toc: true ---- - -A `LINESTRING` is a collection of [Points](point.html) that are "strung together" into one geometric object. A LineString can be used to represent an arbitrary curve, such as a [Bézier curve](https://en.wikipedia.org/wiki/Bézier_curve). In practice, this means that LineStrings are useful for representing real-world objects such as roads and rivers. - -The coordinates of each Point that makes up the LineString are translated according to the current [spatial reference system](spatial-glossary.html#spatial-reference-system) (denoted by an [SRID](spatial-glossary.html#srid)) to determine what the Point "is", or what it "means" relative to the [other spatial objects](spatial-features.html#spatial-objects) (if any) in the data set. - -{% include {{page.version.version}}/spatial/zmcoords.md %} - -## Examples - -A LineString can be created from SQL by calling the `st_geomfromtext` function on a LineString definition expressed in the [Well Known Text (WKT)](spatial-glossary.html#wkt) format as shown below. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT ST_GeomFromText('LINESTRING(0 0, 1440 900)'); -~~~ - -~~~ - st_geomfromtext --------------------------------------------------------------------------------------- - 0102000000020000000000000000000000000000000000000000000000008096400000000000208C40 -(1 row) -~~~ - -You can also make a LineString using the [aggregate function form](functions-and-operators.html#aggregate-functions) of `st_makeline`. - -## See also - -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial objects](spatial-features.html#spatial-objects) -- [POINT](point.html) -- [POLYGON](polygon.html) -- [MULTIPOINT](multipoint.html) -- [MULTILINESTRING](multilinestring.html) -- [MULTIPOLYGON](multipolygon.html) -- [GEOMETRYCOLLECTION](geometrycollection.html) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/liquibase.md b/src/current/v21.1/liquibase.md deleted file mode 100644 index 29f8a42977f..00000000000 --- a/src/current/v21.1/liquibase.md +++ /dev/null @@ -1,516 +0,0 @@ ---- -title: Migrate CockroachDB Schemas with Liquibase -summary: Learn how to use Liquibase with a CockroachDB cluster. -toc: true ---- - -This page guides you through a series of simple database schema changes using the [Liquibase](https://www.liquibase.org/get-started/how-liquibase-works) command-line tool and the [CockroachDB SQL shell](cockroach-sql.html). - -For detailed information about using Liquibase, see the [Liquibase documentation site](https://docs.liquibase.com/home.html). - -## Before you begin - -Before you begin the tutorial, do the following: - -1. [Install CockroachDB](install-cockroachdb.html), and [start a secure cluster](secure-a-cluster.html). When starting your cluster, make sure that you generate cluster certificates, create the `bank` database, and create the `max` user. -1. Download and install a Java Development Kit. Liquibase supports JDK versions 8+. In this tutorial, we use [AdoptOpenJDK](https://adoptopenjdk.net/) 8, but you can follow along with any JDK version 8+. - -## Step 1. Download and install Liquibase - -To install the Liquibase binary on your machine: - -1. Download the latest version of the [Liquibase command-line tool](https://www.liquibase.org/download). CockroachDB is fully compatible with Liquibase versions 4.2.0 and greater. We use the binary download of Liquibase 4.20, for macOS. - - {{site.data.alerts.callout_info}} - In this tutorial, we go through a manual installation, using a download of the binary version of the Liquibase command-line tool. If you are new to Liquibase, you can also use the [Liquibase Installer](https://www.liquibase.org/get-started/using-the-liquibase-installer) to get started. The installer comes with some example properties and changelog files, an example H2 database, and a distribution of AdoptOpenJDK. - {{site.data.alerts.end}} - -1. Make a new directory for your Liquibase installation: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ mkdir liquibase-4.2.0-bin - ~~~ - -1. Extract the Liquibase download to the new directory: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ tar -xvf liquibase-4.2.0.tar.gz -C liquibase-4.2.0-bin - ~~~ - -1. Append the full path of the `liquibase` binary (now located in the `liquibase-4.2.0-bin` folder) to your machine's `PATH` environment variable: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ echo "export PATH=$PATH:/full-path/liquibase-4.2.0-bin" >> ~/.bash_profile - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ source ~/.bash_profile - ~~~ - - {{site.data.alerts.callout_info}} - If your terminal does not run `.bash_profile` at start-up, you can alternatively append the `liquibase` path to the `PATH` definition in `.bashrc` or `.profile`. - {{site.data.alerts.end}} - -1. To verify that the installation was successful, run the following command: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ liquibase --version - ~~~ - - You should get output similar to the following: - - ~~~ - #################################################### - ## _ _ _ _ ## - ## | | (_) (_) | ## - ## | | _ __ _ _ _ _| |__ __ _ ___ ___ ## - ## | | | |/ _` | | | | | '_ \ / _` / __|/ _ \ ## - ## | |___| | (_| | |_| | | |_) | (_| \__ \ __/ ## - ## \_____/_|\__, |\__,_|_|_.__/ \__,_|___/\___| ## - ## | | ## - ## |_| ## - ## ## - ## Get documentation at docs.liquibase.com ## - ## Get certified courses at learn.liquibase.com ## - ## Get advanced features and support at ## - ## liquibase.com/support ## - ## ## - #################################################### - Starting Liquibase at 13:38:36 (version 4.2.0 #18 built at 2020-11-13 16:49+0000) - Liquibase Version: 4.2.0 - Liquibase Community 4.2.0 by Datical - Running Java under /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre (Version 1.8.0_242) - ~~~ - -## Step 2: Download the PostgreSQL JDBC driver - -The Liquibase command-line tool uses the PostgreSQL JDBC driver to connect to CockroachDB as a Java application. - -To install the driver for Liquibase: - -1. Download the JDBC driver from [the PostgreSQL website](https://jdbc.postgresql.org/download/). -1. Place the driver in the `lib` directory of the Liquibase binary. For example: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cp ~/Downloads/postgresql-42.2.9.jar liquibase-4.2.0-bin/lib/ - ~~~ - -{{site.data.alerts.callout_success}} -If you are using Liquibase in the context of a separate Java application, we recommend that you use a dependency management tool, like [Maven](https://docs.liquibase.com/tools-integrations/maven/home.html), to download the driver. -{{site.data.alerts.end}} - -## Step 3. Generate TLS certificates for the `max` user - -When you [started a secure CockroachDB cluster](secure-a-cluster.html), you should have created a user `max`. You should have also given this user the [`admin` role](authorization.html#admin-role), which grants all privileges to all databases on the cluster. In this tutorial, Liquibase runs schema changes as the `max` user. - -To authenticate connection requests to CockroachDB from the Liquibase client, you need to generate some certificates for `max`. Use [`cockroach cert`](cockroach-cert.html#synopsis) to generate the certificates: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach cert create-client max --certs-dir=certs --ca-key=my-safe-directory/ca.key --also-generate-pkcs8-key -~~~ - -The [`--also-generate-pkcs8-key` flag](cockroach-cert.html#flag-pkcs8) generates a key in [PKCS#8 format](https://tools.ietf.org/html/rfc5208), which is the standard key encoding format in Java. In this case, the generated PKCS8 key will be named `client.max.key.pk8`. - -## Step 4: Create a changelog - -Liquibase uses [changelog files](https://docs.liquibase.com/concepts/basic/changelog.html) to manage database schema changes. Changelog files include a list of instructions, known as [changesets](https://docs.liquibase.com/concepts/basic/changeset.html), that are executed against the database in a specified order. Liquibase supports XML, YAML, and SQL formats for changelogs and changesets. - -Let's define a changelog with the [XML format](https://docs.liquibase.com/concepts/basic/xml-format.html): - -1. Create a file named `changelog-main.xml`: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ touch changelog-main.xml - ~~~ - -1. Add the following to the blank `changelog-main.xml` file: - - {% include_cached copy-clipboard.html %} - ~~~ xml - - - - ANY - - - - - ~~~ - - This first changeset uses [the `sqlFile` tag](https://docs.liquibase.com/change-types/community/sql-file.html), which tells Liquibase that an external `.sql` file contains some [SQL statements](https://docs.liquibase.com/concepts/basic/sql-format.html) to execute. - - {{site.data.alerts.callout_success}} - CockroachDB has [limited support for online schema changes in transactions](online-schema-changes.html#limited-support-for-schema-changes-within-transactions). To avoid running into issues with incomplete transactions, we recommend setting the [`runInTransaction` attribute](https://docs.liquibase.com/concepts/basic/changeset.html#available-attributes) to `"false"` on all changesets. - {{site.data.alerts.end}} - - -1. In the same directory, create the SQL file specified by the first changeset: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ touch create.sql - ~~~ - -1. Add the following [`CREATE TABLE`](create-table.html) statement to the `create.sql` file: - - {% include_cached copy-clipboard.html %} - ~~~ sql - create table account - ( - id int not null primary key default unique_rowid(), - balance numeric(19, 2) not null, - name varchar(128) not null, - type varchar(25) not null - ); - ~~~ - - When Liquibase runs, the first changeset will execute the statements in `create.sql`, creating a table named `account`. - -1. Now let's use the [XML format](https://docs.liquibase.com/concepts/basic/xml-format.html) to define the second changeset. Directly after the first `changeSet` element in `changelog-main.xml`, add the following: - - {% include_cached copy-clipboard.html %} - ~~~ xml - - - 1 - Alice - - asset - - - 2 - Bob - - expense - - - 3 - Bobby Tables - - asset - - - 4 - Doris - - expense - - - ~~~ - - This second changeset uses the [Liquibase XML syntax](https://docs.liquibase.com/concepts/basic/xml-format.html) to specify a series of sequential `INSERT` statements that initialize the `account` table with some values. - -When the application is started, all of the queries specified by the changesets are executed in the order specified by their `changeset` `id` values. - -{{site.data.alerts.callout_success}} -When possible, we recommend limiting each changeset to a single statement, per the **one change per changeset** [Liquibase best practice](https://docs.liquibase.com/concepts/bestpractices.html). This is especially important for [online schema changes](online-schema-changes.html). For more information, see [Liquibase and transactions](#liquibase-and-transactions). -{{site.data.alerts.end}} - -## Step 5. Configure a Liquibase properties file - -Liquibase properties are defined in a file named [`liquibase.properties`](https://docs.liquibase.com/workflows/liquibase-community/creating-config-properties.html). These properties define the database connection information. - -{{site.data.alerts.callout_info}} -You can also set Liquibase properties with the `liquibase` command-line tool. -{{site.data.alerts.end}} - -To configure Liquibase properties: - -1. In the same directory as `changelog-main.xml`, create a `liquibase.properties` file: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ touch liquibase.properties - ~~~ - -1. Add the following property definitions to the file: - - {% include_cached copy-clipboard.html %} - ~~~ yml - changeLogFile: changelog-main.xml - driver: org.postgresql.Driver - url: jdbc:postgresql://localhost:26257/bank?sslmode=verify-full&sslrootcert=/full-path/certs/ca.crt&sslkey=/full-path/certs/client.max.key.pk8&sslcert=/full-path/certs/client.max.crt - username: max - ~~~ - - {{site.data.alerts.callout_info}} - For `url`, the SSL connection parameters must specify the full paths of the certificates that you generated. - {{site.data.alerts.end}} - -## Step 6. Run Liquibase - -To run Liquibase from the command line, execute the following command from the directory containing your `liquibase.properties` and `changelog-main.xml` files: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ liquibase update -~~~ - -You should see output similar to the following: - -~~~ -Liquibase Community 4.2.0 by Datical -#################################################### -## _ _ _ _ ## -## | | (_) (_) | ## -## | | _ __ _ _ _ _| |__ __ _ ___ ___ ## -## | | | |/ _` | | | | | '_ \ / _` / __|/ _ \ ## -## | |___| | (_| | |_| | | |_) | (_| \__ \ __/ ## -## \_____/_|\__, |\__,_|_|_.__/ \__,_|___/\___| ## -## | | ## -## |_| ## -## ## -## Get documentation at docs.liquibase.com ## -## Get certified courses at learn.liquibase.com ## -## Get advanced features and support at ## -## liquibase.com/support ## -## ## -#################################################### -Starting Liquibase at 13:59:37 (version 4.2.0 #18 built at 2020-11-13 16:49+0000) -Liquibase: Update has been successful. -~~~ - -When the changelog is first executed, Liquibase also creates a table called [`databasechangelog`](https://docs.liquibase.com/concepts/databasechangelog-table.html) in the database where it performs changes. This table's rows log all completed changesets. - -To see the completed changesets, open a new terminal, start the [built-in SQL shell](cockroach-sql.html), and query the `databasechangelog` table: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --certs-dir=certs -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM bank.databasechangelog; -~~~ - -~~~ - id | author | filename | dateexecuted | orderexecuted | exectype | md5sum | description | comments | tag | liquibase | contexts | labels | deployment_id ------+--------+--------------------+----------------------------------+---------------+----------+------------------------------------+--------------------------------------------------------------------------------------------------------+----------+------+-----------+----------+--------+---------------- - 1 | max | changelog-main.xml | 2020-11-30 13:59:38.40272+00:00 | 1 | EXECUTED | 8:567321cdb0100cbe76731a7ed414674b | sqlFile | | NULL | 4.2.0 | NULL | NULL | 6762778263 - 2 | max | changelog-main.xml | 2020-11-30 13:59:38.542547+00:00 | 2 | EXECUTED | 8:c2945f2a445cf60b4b203e1a91d14a89 | insert tableName=account; insert tableName=account; insert tableName=account; insert tableName=account | | NULL | 4.2.0 | NULL | NULL | 6762778263 -(2 rows) -~~~ - -You can also query the `account` table directly to see the latest changes reflected in the table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM bank.account; -~~~ - -~~~ - id | balance | name | type ------+---------+--------------+---------- - 1 | 500.00 | Alice | asset - 2 | 500.00 | Bob | expense - 3 | 500.00 | Bobby Tables | asset - 4 | 500.00 | Doris | expense -(4 rows) -~~~ - -{{site.data.alerts.callout_info}} -Liquibase does not [retry transactions](transactions.html#transaction-retries) automatically. If a changeset fails at startup, you might need to restart the application manually to complete the changeset. -{{site.data.alerts.end}} - -## Step 7. Add additional changesets - -Suppose that you want to change the primary key of the `accounts` table from a simple, incrementing [integer](int.html) (in this case, `id`) to an auto-generated [UUID](uuid.html), to follow some [CockroachDB best practices](performance-best-practices-overview.html#unique-id-best-practices). You can make these changes to the schema by creating and executing an additional changeset: - -1. Create a SQL file to add a new UUID-typed column to the table: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ touch add_uuid.sql - ~~~ - - {{site.data.alerts.callout_success}} - Using [SQL files](https://docs.liquibase.com/concepts/basic/sql-format.html) to define statements can be helpful when you want to execute statements that use syntax specific to CockroachDB. - {{site.data.alerts.end}} - -1. Add the following SQL statement to `add_uuid.sql`: - - {% include_cached copy-clipboard.html %} - ~~~ - /* Add new UUID-typed column */ - ALTER TABLE account ADD COLUMN unique_id UUID NOT NULL DEFAULT gen_random_uuid(); - ~~~ - - This statement adds [a new `unique_id` column](add-column.html) to the `accounts` table, with the default value as [a randomly-generated UUID](performance-best-practices-overview.html#use-uuid-to-generate-unique-ids). - -1. In the `changelog-main.xml` file, add the following after the second `changeSet` element: - - ~~~ xml - - - - ~~~ - -1. Now create a SQL file to update the primary key for the table with the new column: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ touch update_pk.sql - ~~~ - -1. Add the following SQL statement to `update_pk.sql`: - - {% include_cached copy-clipboard.html %} - ~~~ - /* Change primary key */ - ALTER TABLE account ALTER PRIMARY KEY USING COLUMNS (unique_id); - ~~~ - - This statement [alters the `accounts` primary key](alter-primary-key.html) to use the `unique_id` column. - -1. In the `changelog-main.xml` file, add the following after the third `changeSet` element: - - ~~~ xml - - - - ~~~ - -1. To update the table, run `liquibase update` again: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ liquibase update - ~~~ - - You should see output similar to the following: - - ~~~ - Liquibase Community 4.2.0 by Datical - #################################################### - ## _ _ _ _ ## - ## | | (_) (_) | ## - ## | | _ __ _ _ _ _| |__ __ _ ___ ___ ## - ## | | | |/ _` | | | | | '_ \ / _` / __|/ _ \ ## - ## | |___| | (_| | |_| | | |_) | (_| \__ \ __/ ## - ## \_____/_|\__, |\__,_|_|_.__/ \__,_|___/\___| ## - ## | | ## - ## |_| ## - ## ## - ## Get documentation at docs.liquibase.com ## - ## Get certified courses at learn.liquibase.com ## - ## Get advanced features and support at ## - ## liquibase.com/support ## - ## ## - #################################################### - Starting Liquibase at 14:26:50 (version 4.2.0 #18 built at 2020-11-13 16:49+0000) - Liquibase: Update has been successful. - ~~~ - -To see the completed changesets, open a new terminal, start the [built-in SQL shell](cockroach-sql.html), and query the `databasechangelog` table: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql --certs-dir=certs -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM bank.databasechangelog; -~~~ - -~~~ - id | author | filename | dateexecuted | orderexecuted | exectype | md5sum | description | comments | tag | liquibase | contexts | labels | deployment_id ------+--------+--------------------+----------------------------------+---------------+----------+------------------------------------+--------------------------------------------------------------------------------------------------------+----------+------+-----------+----------+--------+---------------- - 1 | max | changelog-main.xml | 2020-11-30 13:59:38.40272+00:00 | 1 | EXECUTED | 8:567321cdb0100cbe76731a7ed414674b | sqlFile | | NULL | 4.2.0 | NULL | NULL | 6762778263 - 2 | max | changelog-main.xml | 2020-11-30 13:59:38.542547+00:00 | 2 | EXECUTED | 8:c2945f2a445cf60b4b203e1a91d14a89 | insert tableName=account; insert tableName=account; insert tableName=account; insert tableName=account | | NULL | 4.2.0 | NULL | NULL | 6762778263 - 3 | max | changelog-main.xml | 2020-11-30 14:26:51.916768+00:00 | 3 | EXECUTED | 8:7b76f0ae200b1ae1d9f0c0f78979348b | sqlFile | | NULL | 4.2.0 | NULL | NULL | 6764411427 - 4 | max | changelog-main.xml | 2020-11-30 14:26:52.609161+00:00 | 4 | EXECUTED | 8:fcaa0dca049c34c6372847af7a2646d9 | sqlFile | | NULL | 4.2.0 | NULL | NULL | 6764411427 -(4 rows) -~~~ - -You can also query the `account` table directly to see the latest changes reflected in the table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM bank.account; -~~~ - -~~~ - id | balance | name | type | unique_id ------+---------+--------------+---------+--------------------------------------- - 1 | 500.00 | Alice | asset | 3d2b7da4-0876-4ddd-8626-b980cef3323e - 2 | 500.00 | Bob | expense | 8917ce09-c7d2-42a0-9ee4-8cb9cb3515ec - 3 | 500.00 | Bobby Tables | asset | b5dccde6-25fe-4c73-b3a2-501225d8b235 - 4 | 500.00 | Doris | expense | f37dc62e-a2d5-4f63-801a-3eaa3fc68806 -(4 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE bank.account; -~~~ - -~~~ - table_name | create_statement -----------------------+------------------------------------------------------------ - bank.public.account | CREATE TABLE account ( - | id INT8 NOT NULL DEFAULT unique_rowid(), - | balance DECIMAL(19,2) NOT NULL, - | name VARCHAR(128) NOT NULL, - | type VARCHAR(25) NOT NULL, - | unique_id UUID NOT NULL DEFAULT gen_random_uuid(), - | CONSTRAINT "primary" PRIMARY KEY (unique_id ASC), - | UNIQUE INDEX account_id_key (id ASC), - | FAMILY "primary" (id, balance, name, type, unique_id) - | ) -(1 row) -~~~ - -## Liquibase and transactions - -By default, Liquibase wraps each changeset within a single transaction. If the transaction fails to successfully commit, Liquibase rolls back the transaction. - -CockroachDB has [limited support for online schema changes within transactions](online-schema-changes.html#limited-support-for-schema-changes-within-transactions). If a schema change fails, automatic rollbacks can lead to unexpected results. To avoid running into issues with incomplete transactions, we recommend setting the `runInTransaction` attribute on each of your changesets to `"false"`, as demonstrated throughout this tutorial. - -{{site.data.alerts.callout_info}} -If `runInTransaction="false"` for a changeset, and an error occurs while Liquid is running the changeset, the `databasechangelog` table might be left in an invalid state and need to be fixed manually. -{{site.data.alerts.end}} - -### Transaction retries - -When multiple, concurrent transactions or statements are issued to a single CockroachDB cluster, [transaction contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention) can cause schema migrations to fail. In the event of transaction contention, CockroachDB returns a `40001 SQLSTATE` (i.e., a serialization failure). - -Liquibase does not automatically retry transactions. To handle transaction failures, we recommend writing client-side transaction retry logic. For more information about client-side transaction retries in CockroachDB, see [Transaction Retries](transactions.html#transaction-retries). - -## Liquibase integrations - -You can run Liquibase in the context of a Java application framework, like Spring Boot. For examples of using Liquibase for schema management in a Spring Boot application built on CockroachDB, see [Build a Spring App with CockroachDB and JDBC](build-a-spring-app-with-cockroachdb-jdbc.html) and [Build a Spring App with CockroachDB and JPA](build-a-spring-app-with-cockroachdb-jpa.html). - -For documentation on running Liquibase with other tooling, see [the Liquibase documentation site](https://docs.liquibase.com/workflows/liquibase-community/running.html). - -## Report Issues with Liquibase and CockroachDB - -If you run into problems, please file an issue on the [Liquibase issue tracker](https://github.com/liquibase/liquibase/issues), including the following details about the environment where you encountered the issue: - -- CockroachDB version ([`cockroach version`](cockroach-version.html)) -- Liquibase version -- Operating system -- Steps to reproduce the behavior - -## See Also - -+ [Liquibase documentation](https://docs.liquibase.com/home.html) -+ [Liquibase issue tracker](https://github.com/liquibase/liquibase/issues) -+ [Client connection parameters](connection-parameters.html) -+ [Third-Party Database Tools](third-party-database-tools.html) -+ [Learn CockroachDB SQL](learn-cockroachdb-sql.html) -+ [Build a Spring App with CockroachDB and JDBC](build-a-spring-app-with-cockroachdb-jdbc.html) -+ [Build a Spring App with CockroachDB and JPA](build-a-spring-app-with-cockroachdb-jpa.html) diff --git a/src/current/v21.1/load-based-splitting.md b/src/current/v21.1/load-based-splitting.md deleted file mode 100644 index acde8a8f6af..00000000000 --- a/src/current/v21.1/load-based-splitting.md +++ /dev/null @@ -1,78 +0,0 @@ ---- -title: Load-Based Splitting -summary: To optimize your cluster's performance, CockroachDB can split frequently accessed keys into their own ranges. -toc: true ---- - -To optimize your cluster's performance, CockroachDB can split frequently accessed keys into smaller ranges. In conjunction with [load-based rebalancing](architecture/replication-layer.html#load-based-replica-rebalancing), load-based splitting distributes load evenly across your cluster. - -## Settings - -### Enable/disable load-based splitting - -Use [`SET CLUSTER SETTING`](set-cluster-setting.html) to set `kv.range_split.by_load_enabled` to: - -- `true` to enable load-based splitting _(default)_ -- `false` to disable load-based splitting - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET CLUSTER SETTING kv.range_split.by_load_enabled = true; -~~~ - -#### When to enable load-based splitting - -Load-based splitting is on by default and beneficial in almost all situations. - -#### When to disable load-based splitting - -You might want to disable load-based splitting when troubleshooting range-related issues under the guidance of our developers. - -### Control load-based splitting threshold - -Use [`SET CLUSTER SETTING`](set-cluster-setting.html) to set `kv.range_split.load_qps_threshold` to the queries-per-second (QPS) at which you want to consider splitting a range (defaults to `2500`): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET CLUSTER SETTING kv.range_split.load_qps_threshold = 2000; -~~~ - -#### When to modify the load-based splitting threshold - -Some workloads might find splitting ranges more aggressively (i.e., a lower QPS threshold) can improve performance. - -On the other hand, some workloads with very large machines might want to increase the QPS threshold to split more conservatively. - -## How load-based splitting works - -Whenever a range exceeds the cluster's setting for `kv.range_split.load_qps_threshold`, the range becomes eligible for load-based splitting. - -At that point, we begin gathering per-key metrics to determine whether or not a split would benefit the cluster's performance based on the following heuristics: - -- **Balance factor:** If we perform a split, would there be a balance of load on both sides of the split? For example, if 99% of the traffic is for a single key, splitting it apart from the others in the range will not have a substantial impact on performance. However, if 60% of the queries are for a single key, it is likely to benefit the cluster to move the key to its own range. - -- **Split crossing:** If we perform a split, how many queries would have to cross this new range boundary? Because involving multiple ranges in a query incurs greater overhead than a single range, splitting a range could actually degrade performance. For example, if the range is involved in many `SELECT COUNT(*)...` operations, splitting the range in two could negatively impact performance. - -### Determining a split point - -Using the per-key metrics gathered once a node exceeds the `kv.range_split.load_qps_threshold` value, we estimate a good place to perform a split by determining the total operations over the first set of keys that exceed the `kv.range_split.load_qps_threshold`. - -For example, if the operations on a single key exceed the `kv.range_split.load_qps_threshold` value, it's a good candidate to split the range at that point. - -Another example is that if the range has equal access among all keys, whose total operation exceeds `kv.range_split.load_qps_threshold`. By splitting the range at the first set of keys whose total operations exceed the threshold, we can reduce the load on the node. If the "other" range still exceeds the threshold, it will eventually be split again. - -In both of these examples, the split would only occur if the balance factor and split crossing heuristics determined the split would produce better results. - -## Why load-based splitting works - -CockroachDB creates a relatively even distribution of leaseholders throughout your cluster. ([Leaseholders](architecture/replication-layer.html#leases) are a single replica of a range that both serve reads and coordinate writes operations.) - -However, without load-based splitting this distribution is created without considering the load present on any set of keys. This means that even with an equitable distribution of leases throughout the cluster, some leases will generate more traffic for the node that houses them than others. Because each node can only provide so much throughput, a single node can become a bottleneck for providing access to a subset of data in your cluster. - -However, by leveraging load-based splitting, the cluster can understand load on a per-range basis and split up ranges that generate a significant amount of load into multiple ranges, and therefore multiple leaseholders. This lets the cluster express its load as a function of leases; so the roughly equal distribution of leases also generates a roughly equal distribution of traffic, preventing individual nodes from becoming bottlenecks for your cluster. - -This benefit is further amplified by [load-based rebalancing](architecture/replication-layer.html#load-based-replica-rebalancing), which ensures that all nodes contain replicas with a roughly equal load. By evenly distributing load throughout your cluster, it's easier to prevent bottlenecks from arising, as well as simplifying hardware forecasting. - -## See also - -- [`SET CLUSTER SETTING`](set-cluster-setting.html) diff --git a/src/current/v21.1/local-testing.md b/src/current/v21.1/local-testing.md deleted file mode 100644 index c1c8b155cf0..00000000000 --- a/src/current/v21.1/local-testing.md +++ /dev/null @@ -1,49 +0,0 @@ ---- -title: Test Your Application Locally -summary: Best practices for locally testing an application built on CockroachDB -toc: true ---- - -This page documents best practices for unit testing applications built on CockroachDB in a local environment. - -If you are deploying a self-hosted cluster, see the [Production Checklist](recommended-production-settings.html) for information about preparing your cluster for production. - -## Use a local, single-node cluster with in-memory storage - -The [`cockroach start-single-node`](cockroach-start-single-node.html) command starts a single-node, insecure cluster with [in-memory storage](cockroach-start-single-node.html#store): - -{% include_cached copy-clipboard.html %} -~~~ shell -cockroach start-single-node --insecure --store=type=mem,size=0.25 --advertise-addr=localhost -~~~ - -Using in-memory storage improves the speed of the cluster for local testing purposes. - -## Log test output to a file - -By default, `cockroach start-single-node` logs cluster activity to a file with the [default logging configuration](configure-logs.html#default-logging-configuration). When you specify the `--store=type=mem` flag, the command prints cluster activity directly to the console instead. - -To customize logging behavior for local clusters, use the [`--log` flag](cockroach-start-single-node.html#logging): - -{% include_cached copy-clipboard.html %} -~~~ shell -cockroach start-single-node --insecure --store=type=mem,size=0.25 --advertise-addr=localhost --log="{file-defaults: {dir: /path/to/logs}, sinks: {stderr: {filter: NONE}}}" -~~~ - -The `log` flag has two suboptions: - -- `file-defaults`, which specifies the path of the file in which to log events (`/path/to/logs`). -- `sinks`, which provides a secondary destination to which to log events (`stderr`). - -For more information about logging, see [Configure logs](configure-logs.html). - -## Use a local file server for bulk operations - -To test bulk operations like [`IMPORT`](import.html), [`BACKUP`](backup.html), or [`RESTORE`](restore.html), we recommend using a local file server. - -For more details, see [Use a Local File Server for Bulk Operations](use-a-local-file-server-for-bulk-operations.html). - -## See also - -- [Error Handling and Troubleshooting](error-handling-and-troubleshooting.html) -- [Optimize Statement Performance](make-queries-fast.html) \ No newline at end of file diff --git a/src/current/v21.1/log-formats.md b/src/current/v21.1/log-formats.md deleted file mode 100644 index 82de0494468..00000000000 --- a/src/current/v21.1/log-formats.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: Log Formats -summary: Reference documentation for structured and unstructured log formats. -toc: true ---- - -{% remote_include https://raw.githubusercontent.com/cockroachdb/cockroach/master/docs/generated/logformats.md %} diff --git a/src/current/v21.1/logging-overview.md b/src/current/v21.1/logging-overview.md deleted file mode 100644 index 358f35305d8..00000000000 --- a/src/current/v21.1/logging-overview.md +++ /dev/null @@ -1,79 +0,0 @@ ---- -title: Logging -summary: Overview of the logging system and logging channels. -toc: true ---- - -If you need to monitor your cluster, tune performance, or [troubleshoot](troubleshooting-overview.html) issues, you can check the CockroachDB logs, which include details about notable cluster, node, and range-level events. - -{{site.data.alerts.callout_info}} -This logging system has been introduced in v21.1 and is backward-compatible. For a summary of what has changed, see [Changes to logging system](#changes-to-logging-system). -{{site.data.alerts.end}} - -## Details - -When a node processes a [`cockroach` command](cockroach-commands.html), it produces a stream of messages about the command's activities. Each message is composed of: - -- A payload that contains events either structured in JSON or conveyed in an arbitrary string. For details on structured event types and their fields, see [Notable Event Types](eventlog.html). -- An envelope that contains event metadata (e.g., severity, date, timestamp, channel). Depending on the log format you specify when [configuring logs](configure-logs.html#file-logging-format), the envelope can be formatted either in JSON or as a flat prefix to the message. - -Messages are organized into appropriate [logging channels](#logging-channels) and then routed through log sinks. Each sink further processes and filters the messages before emitting them to destinations outside CockroachDB. The mapping of channels to sinks, as well as the processing and filtering done by each sink, is configurable and is intended to support multiple [use cases](logging-use-cases.html). For more information on configuring sinks and severity levels, see [Configure Logs](configure-logs.html). - -{{site.data.alerts.callout_success}} -All [`cockroach` commands](cockroach-commands.html) support logging. However, note that most messages related to cluster operation are generated by [`cockroach start`](cockroach-start.html) or [`cockroach start-single-node`](cockroach-start-single-node.html). Other commands generate messages related to their own execution, which are mainly useful when troubleshooting the behaviors of those commands. -{{site.data.alerts.end}} - -## Logging channels - -Log messages in CockroachDB are directed into logging channels, which can in turn be assigned to output to one or more [log sinks](configure-logs.html#configure-log-sinks). - -This allows you to group channels that log related information (e.g., operational, security, or SQL events) into their own sinks. Each sink can output to a predetermined destination where the logs can be collected and parsed. For usage examples, see [Logging Use Cases](logging-use-cases.html). - -| Channel | Description | -|-----------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [`DEV`](logging.html#dev) | Uncategorized and debug messages. | -| [`OPS`](logging.html#ops) | Process starts, stops, shutdowns, and crashes (if they can be logged); changes to cluster topology, such as node additions, removals, and decommissions. | -| [`HEALTH`](logging.html#health) | Resource usage; node-node connection events, including connection errors; up- and down-replication and range unavailability. | -| [`STORAGE`](logging.html#storage) | Low-level storage logs from the [Pebble storage engine](architecture/storage-layer.html#pebble). | -| [`SESSIONS`](logging.html#sessions) | Client connections and disconnections (when enabled via the `server.auth_log.sql_connections.enabled` [cluster setting](cluster-settings.html)); SQL authentication logins/attempts and session/query terminations (when enabled via the `server.auth_log.sql_sessions.enabled` [cluster setting](cluster-settings.html)). | -| [`SQL_SCHEMA`](logging.html#sql_schema) | Database, schema, table, sequence, view, and type creation; changes to table columns and sequence parameters. | -| [`USER_ADMIN`](logging.html#user_admin) | Changes to users, roles, and authentication credentials. | -| [`PRIVILEGES`](logging.html#privileges) | Changes to privileges and object ownership. | -| [`SENSITIVE_ACCESS`](logging.html#sensitive_access) | SQL audit events (when enabled via [`ALTER TABLE ... EXPERIMENTAL_AUDIT`](experimental-audit.html)). | -| [`SQL_EXEC`](logging.html#sql_exec) | SQL statement executions (when enabled via the `sql.trace.log_statement_execute`) [cluster setting](cluster-settings.html) and uncaught Go panic errors during SQL statement execution. | -| [`SQL_PERF`](logging.html#sql_perf) | SQL executions that impact performance, such as slow queries (when enabled via the `sql.log.slow_query.latency_threshold` and/or `sql.log.slow_query.experimental_full_table_scans.enabled` [cluster settings](cluster-settings.html)). | - - -Logging channels are analogous to [logging facilities in Syslog](https://en.wikipedia.org/wiki/Syslog) or [logging services in Datadog](https://docs.datadoghq.com/logs/log_collection/?tab=http#reserved-attributes). For more details on the contents of each logging channel, see the [Logging reference](logging.html#logging-channels). - -## Changes to logging system - -Prior to v21.1, logs were separated into a general CockroachDB log and secondary SQL and storage logs. These were output to correspondingly named log files. - -The events collected by [logging channels](#logging-channels) are now redirected to file sinks as follows: - -| Filename (legacy) | Description | Channel | -|---------------------------+-------------------------+--------------------| -| `cockroach.log` | General CockroachDB log | `DEV` | -| `cockroach-pebble.log` | Pebble log | `STORAGE` | -| `cockroach-sql-audit.log` | SQL audit log | `SENSITIVE_ACCESS` | -| `cockroach-sql-exec.log` | SQL execution log | `SQL_EXEC` | -| `cockroach-auth.log` | SQL authentication log | `SESSIONS` | -| `cockroach-sql-slow.log` | Slow query log | `SQL_PERF` | - -Notable events that were previously collected in the general CockroachDB log are now directed into several new logging channels: - -- `OPS` -- `HEALTH` -- `SQL_SCHEMA` -- `USER_ADMIN` -- `PRIVILEGES` - -{% include_cached new-in.html version="v21.1" %} Logging is now configurable via YAML. The YAML configuration allows you to customize which kinds of events are output to different logging destinations, along with many other parameters. As a result, the logging flags previously used with `cockroach` commands are now deprecated in favor of the YAML configuration. For details, see [Configure Logs](configure-logs.html). - -## See also - -- [Configure logs](configure-logs.html) -- [Logging Levels and Channels](logging.html) -- [Log formats](log-formats.html) -- [Notable Event Types](eventlog.html) diff --git a/src/current/v21.1/logging-use-cases.md b/src/current/v21.1/logging-use-cases.md deleted file mode 100644 index 06738b60f61..00000000000 --- a/src/current/v21.1/logging-use-cases.md +++ /dev/null @@ -1,458 +0,0 @@ ---- -title: Logging use cases -summary: Examples of common logging use cases and possible CockroachDB logging sink configurations. -toc: true ---- - -This page describes some common logging use cases, their relevant [logging channels](logging-overview.html#logging-channels), and examples of notable events to be found in the logs: - -- [Operational monitoring](#operational-monitoring) (for operators) -- [Security and audit monitoring](#security-and-audit-monitoring) (for security engineers) -- [Performance tuning](#performance-tuning) (for application developers) -- [Network logging](#network-logging) (for operators) - -We provide an example [file sink configuration](configure-logs.html#output-to-files) for each use case. These configurations are entirely optional and are intended to highlight the contents of each logging channel. A sink can include any combination of logging channels. Moreover, a single logging channel can be used in more than one sink in your logging configuration. - -Your deployment may use an external service (e.g., [Elasticsearch](https://www.elastic.co/elastic-stack), [Splunk](https://www.splunk.com/)) to collect and programmatically read logging data. - -{{site.data.alerts.callout_info}} -All log examples on this page use the default `crdb-v2` format, except for the [network logging](#network-logging) configuration, which uses the default `json-fluent-compact` format for network output. Most log entries for non-`DEV` channels record *structured* events, which use a standardized format that can be reliably parsed by an external collector. All structured event types and their fields are detailed in the [Notable events reference](eventlog.html). - -Logging channels may also contain events that are *unstructured*. Unstructured events can routinely change between CockroachDB versions, including minor patch revisions, so they are not officially documented. -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -`‹` and `›` are placed around values in log messages that may contain sensitive data (PII). To customize this behavior, see [Redact logs](configure-logs.html#redact-logs). -{{site.data.alerts.end}} - -## Operational monitoring - -A database operator can use the `OPS`, `HEALTH`, and `SQL_SCHEMA` channels to monitor [operational events](#ops) initiated by users or automatic processes, [DDL changes](#sql_schema) from applications, and overall [cluster health](#health). - -In this example configuration, the channels are grouped into a file sink called `ops`. The combined logging output will be found in a `cockroach-ops.log` file at the configured [logging directory](configure-logs.html#logging-directory). - -~~~ yaml -sinks: - file-groups: - ops: - channels: [OPS, HEALTH, SQL_SCHEMA] -~~~ - -{{site.data.alerts.callout_success}} -When monitoring your cluster, consider using these logs in conjunction with [Prometheus](monitor-cockroachdb-with-prometheus.html), which can be set up to track node-level metrics. -{{site.data.alerts.end}} - -### OPS - -The [`OPS`](logging.html#ops) channel logs operational events initiated by users or automation. These can include node additions and removals, process starts and shutdowns, [gossip](https://en.wikipedia.org/wiki/Gossip_protocol) connection events, and [zone configuration changes](configure-replication-zones.html) on the SQL schema or system ranges. - -#### Example: Node decommissioning - -This [`node_decommissioning`](eventlog.html#node_decommissioning) event shows that a node is in the [decommissioning](remove-nodes.html#how-it-works) state: - -~~~ -I210401 23:30:49.319360 5943 1@util/log/event_log.go:32 ⋮ [-] 42 ={"Timestamp":1617319848793433000,"EventType":"node_decommissioning","RequestingNodeID":4,"TargetNodeID":4} -~~~ - -- Preceding the `=` character is the `crdb-v2` event metadata. See the [reference documentation](log-formats.html#format-crdb-v2) for details on the fields. -- `TargetNodeID` shows that the decommissioning node is `4`. -- `RequestingNodeID` shows that decommissioning was also started by node `4`. You will see this when using the `--self` flag with [`cockroach node decommission`](cockroach-node.html). - -#### Example: Node restart - -This [`node_restart`](eventlog.html#node_restart) event shows that a node has rejoined the cluster after being offline (e.g., by being [restarted](cockroach-start.html) after being fully decommissioned): - -~~~ -I210323 20:53:44.765068 611 1@util/log/event_log.go:32 ⋮ [n1] 20 ={"Timestamp":1616532824096394000,"EventType":"node_restart","NodeID":1,"StartedAt":1616532823668899000,"LastUp":1616532816150919000} -~~~ - -- Preceding the `=` character is the `crdb-v2` event metadata. See the [reference documentation](log-formats.html#format-crdb-v2) for details on the fields. -- `NodeID` shows that the restarted node is `1`. -- `StartedAt` shows the timestamp when the node was most recently restarted. -- `LastUp` shows the timestamp when the node was up before being restarted. - -{{site.data.alerts.callout_info}} -All possible `OPS` event types are detailed in the [reference documentation](eventlog.html). -{{site.data.alerts.end}} - -### HEALTH - -The [`HEALTH`](logging.html#health) channel logs operational events initiated by CockroachDB or reported by automatic processes. These can include resource usage details, connection errors, [gossip](https://en.wikipedia.org/wiki/Gossip_protocol) status, [replication](architecture/replication-layer.html) events, and runtime statistics. - -#### Example: Runtime stats - -A [`runtime_stats`](eventlog.html) event is recorded every 10 seconds to reflect server health: - -~~~ -I210517 17:38:20.403619 586 2@util/log/event_log.go:32 ⋮ [n1] 168 ={"Timestamp":1621273100403617000,"EventType":"runtime_stats","MemRSSBytes":119361536,"GoroutineCount":262,"MemStackSysBytes":4063232,"GoAllocBytes":40047584,"GoTotalBytes":68232200,"GoStatsStaleness":0.008556,"HeapFragmentBytes":6114336,"HeapReservedBytes":6324224,"HeapReleasedBytes":10559488,"CGoAllocBytes":8006304,"CGoTotalBytes":11997184,"CGoCallRate":0.6999931,"CPUUserPercent":5.4999456,"CPUSysPercent":6.2399383,"GCRunCount":12,"NetHostRecvBytes":16315,"NetHostSendBytes":21347} -~~~ - -- Preceding the `=` character is the `crdb-v2` event metadata. See the [reference documentation](log-formats.html#format-crdb-v2) for details on the fields. - -{{site.data.alerts.callout_info}} -`runtime_stats` events are typically used for troubleshooting. To monitor your cluster's health, see [Monitoring and Alerting](monitoring-and-alerting.html). -{{site.data.alerts.end}} - -### SQL_SCHEMA - -The [`SQL_SCHEMA`](logging.html#sql_schema) channel logs changes to the SQL logical schema resulting from DDL operations. - -#### Example: Schema change initiated - -This [`alter_table`](eventlog.html#alter_table) event shows an [`ALTER TABLE ... ADD FOREIGN KEY`](add-constraint.html) schema change being initiated on a `movr.public.vehicles` table: - -~~~ -I210323 20:21:04.621132 113397 5@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:50812›,hostnossl,user=root] 14 ={"Timestamp":1616530864502127000,"EventType":"alter_table","Statement":"‹ALTER TABLE movr.public.vehicles ADD FOREIGN KEY (city, owner_id) REFERENCES movr.public.users (city, id)›","User":"‹root›","DescriptorID":59,"ApplicationName":"‹movr›","TableName":"‹movr.public.vehicles›","MutationID":1} -~~~ - -- Preceding the `=` character is the `crdb-v2` event metadata. See the [reference documentation](log-formats.html#format-crdb-v2) for details on the fields. -- `ApplicationName` shows that the events originated from an application named `movr`. You can use this field to filter the logging output by application. -- `DescriptorID` identifies the object descriptor (e.g., `movr.public.vehicles`) undergoing the schema change. -- `MutationID` identifies the job that is processing the schema change. - -#### Example: Schema change completed - -This [`finish_schema_change`](eventlog.html#finish_schema_change) event shows that the above schema change has completed: - -~~~ -I210323 20:21:05.916626 114212 5@util/log/event_log.go:32 ⋮ [n1,job=643761650092900353,scExec,id=59,mutation=1] 15 ={"Timestamp":1616530865791439000,"EventType":"finish_schema_change","InstanceID":1,"DescriptorID":59,"MutationID":1} -~~~ - -- Preceding the `=` character is the `crdb-v2` event metadata. See the [reference documentation](log-formats.html#format-crdb-v2) for details on the fields. -- `DescriptorID` identifies the object descriptor (e.g., `movr.public.vehicles`) affected by the schema change. -- `MutationID` identifies the job that processed the schema change. - -Note that the `DescriptorID` and `MutationID` values match in both of the above log entries, indicating that they are related. - -{{site.data.alerts.callout_info}} -All possible `SQL_SCHEMA` event types are detailed in the [reference documentation](eventlog.html#sql-logical-schema-changes). -{{site.data.alerts.end}} - -## Security and audit monitoring - -A security engineer can use the `SESSIONS`, `USER_ADMIN`, `PRIVILEGES`, and `SENSITIVE_ACCESS` channels to monitor [connection and authentication events](#sessions), changes to [user/role administration](#user_admin) and [privileges](#privileges), and any queries on [audited tables](#sensitive_access). - -In this example configuration, the channels are grouped into a file sink called `security`. The combined logging output will be found in a `cockroach-security.log` file at the configured [logging directory](configure-logs.html#logging-directory). - -In addition, the `security` channels are configured as `auditable`. This feature guarantees non-repudiability by enabling `exit-on-error` (stops nodes when they encounter a logging error) and disabling `buffered-writes` (flushes each log entry and synchronizes writes). This setting can incur a performance overhead and higher disk IOPS consumption, so it should only be used when necessary (e.g., for security purposes). - -~~~ yaml -sinks: - file-groups: - security: - channels: [SESSIONS, USER_ADMIN, PRIVILEGES, SENSITIVE_ACCESS] - auditable: true -~~~ - -### SESSIONS - -The [`SESSIONS`](logging.html#sessions) channel logs SQL session events. This includes client connection and session authentication events, for which logging must be enabled separately. For complete logging of client connections, we recommend enabling both types of events. - -{{site.data.alerts.callout_info}} -These logs perform one disk I/O per event. Enabling each setting will impact performance. -{{site.data.alerts.end}} - -{% include {{ page.version.version }}/misc/experimental-warning.md %} - -#### Example: Client connection events - -To log SQL client connection events to the `SESSIONS` channel, enable the `server.auth_log.sql_connections.enabled` [cluster setting](cluster-settings.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET CLUSTER SETTING server.auth_log.sql_connections.enabled = true; -~~~ - -{{site.data.alerts.callout_info}} -In addition to SQL sessions, connection events can include SQL-based liveness probe attempts. -{{site.data.alerts.end}} - -These logs show a [`client_connection_start`](eventlog.html#client_connection_start) (client connection established) and a [`client_connection_end`](eventlog.html#client_connection_end) (client connection terminated) event over a `hostssl` (TLS transport over TCP) connection: - -~~~ -I210323 21:53:58.300180 53298 4@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:52632›] 49 ={"Timestamp":1616536438300176000,"EventType":"client_connection_start","InstanceID":1,"Network":"tcp","RemoteAddress":"‹[::1]:52632›"} -I210323 21:53:58.305074 53298 4@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:52632›,hostssl] 54 ={"Timestamp":1616536438305072000,"EventType":"client_connection_end","InstanceID":1,"Network":"tcp","RemoteAddress":"‹[::1]:52632›","Duration":4896000} -~~~ - -- Preceding the `=` character is the `crdb-v2` event metadata. See the [reference documentation](log-formats.html#format-crdb-v2) for details on the fields. -- `Network` shows the network protocol of the connection. -- `RemoteAddress` shows the address of the SQL client, proxy, or other intermediate server. - -#### Example: Session authentication events - -To log SQL session authentication events to the `SESSIONS` channel, enable the `server.auth_log.sql_sessions.enabled` [cluster setting](cluster-settings.html) on every cluster: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET CLUSTER SETTING server.auth_log.sql_sessions.enabled = true; -~~~ - -These logs show certificate authentication success over a `hostssl` (TLS transport over TCP) connection: - -~~~ -I210323 23:35:19.458098 122619 4@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:53884›,hostssl,user=‹roach›] 62 ={"Timestamp":1616542519458095000,"EventType":"client_authentication_info","InstanceID":1,"Network":"tcp","RemoteAddress":"‹[::1]:53884›","Transport":"hostssl","User":"‹roach›","Method":"cert-password","Info":"‹HBA rule: host all all all cert-password # built-in CockroachDB default›"} -I210323 23:35:19.458136 122619 4@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:53884›,hostssl,user=‹roach›] 63 ={"Timestamp":1616542519458135000,"EventType":"client_authentication_info","InstanceID":1,"Network":"tcp","RemoteAddress":"‹[::1]:53884›","Transport":"hostssl","User":"‹roach›","Method":"cert-password","Info":"‹client presented certificate, proceeding with certificate validation›"} -I210323 23:35:19.458154 122619 4@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:53884›,hostssl,user=‹roach›] 64 ={"Timestamp":1616542519458154000,"EventType":"client_authentication_ok","InstanceID":1,"Network":"tcp","RemoteAddress":"‹[::1]:53884›","Transport":"hostssl","User":"‹roach›","Method":"cert-password"} -~~~ - -- Preceding the `=` character is the `crdb-v2` event metadata. See the [reference documentation](log-formats.html#format-crdb-v2) for details on the fields. -- The two [`client_authentication_info`](eventlog.html#client_authentication_info) events show the progress of certificate authentication. The `Info` fields show the progress of certificate validation. -- The [`client_authentication_ok`](eventlog.html#client_authentication_ok) event shows that certificate authentication was successful. -- `User` shows that the SQL session is authenticated for user `roach`. - -These logs show password authentication failure over a `hostssl` (TLS transport over TCP) connection: - -~~~ -I210323 21:53:58.304573 53299 4@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:52632›,hostssl,user=‹roach›] 50 ={"Timestamp":1616536438304572000,"EventType":"client_authentication_info","InstanceID":1,"Network":"tcp","RemoteAddress":"‹[::1]:52632›","Transport":"hostssl","User":"‹roach›","Method":"cert-password","Info":"‹HBA rule: host all all all cert-password # built-in CockroachDB default›"} -I210323 21:53:58.304648 53299 4@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:52632›,hostssl,user=‹roach›] 51 ={"Timestamp":1616536438304647000,"EventType":"client_authentication_info","InstanceID":1,"Network":"tcp","RemoteAddress":"‹[::1]:52632›","Transport":"hostssl","User":"‹roach›","Method":"cert-password","Info":"‹no client certificate, proceeding with password authentication›"} -I210323 21:53:58.304797 53299 4@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:52632›,hostssl,user=‹roach›] 52 ={"Timestamp":1616536438304796000,"EventType":"client_authentication_failed","InstanceID":1,"Network":"tcp","RemoteAddress":"‹[::1]:52632›","Transport":"hostssl","User":"‹roach›","Reason":6,"Detail":"‹password authentication failed for user roach›","Method":"cert-password"} -I210323 21:53:58.305016 53298 4@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:52632›,hostssl,user=‹roach›] 53 ={"Timestamp":1616536438305014000,"EventType":"client_session_end","InstanceID":1,"Network":"tcp","RemoteAddress":"‹[::1]:52632›","Duration":2273000} -~~~ - -- Preceding the `=` character is the `crdb-v2` event metadata. See the [reference documentation](log-formats.html#format-crdb-v2) for details on the fields. -- The two [`client_authentication_info`](eventlog.html#client_authentication_info) events show the progress of certificate authentication. The `Info` fields show that password authentication was attempted, in the absence of a client certificate. -- The [`client_authentication_failed`](eventlog.html#client_authentication_failed) event shows that password authentication was unsuccessful. The `Detail` field shows the related error. -- The [`client_session_end`](eventlog.html#client_session_end) event shows that the SQL session was terminated. This would typically be followed by a [`client_connection_end`](eventlog.html#client_connection_end) event. -- `User` shows that the SQL session authentication was attempted for user `roach`. - -{{site.data.alerts.callout_info}} -All possible `SESSIONS` event types are detailed in the [reference documentation](eventlog.html#sql-session-events). For more details on certificate and password authentication, see [Authentication](authentication.html). -{{site.data.alerts.end}} - -### SENSITIVE_ACCESS - -The [`SENSITIVE_ACCESS`](logging.html#sensitive_access) channel logs SQL audit events. These include all queries being run against [audited tables](experimental-audit.html), when enabled, as well as queries executed by users with the [`admin`](authorization.html#admin-role) role. - -{{site.data.alerts.callout_info}} -Enabling these logs can negatively impact performance. We recommend using `SENSITIVE_ACCESS` for security purposes only. -{{site.data.alerts.end}} - -{% include {{ page.version.version }}/misc/experimental-warning.md %} - -To log all queries against a specific table, enable auditing on the table with [`ALTER TABLE ... EXPERIMENTAL_AUDIT`](experimental-audit.html). - -#### Example: Audit events - -This command enables auditing on a `customers` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE customers EXPERIMENTAL_AUDIT SET READ WRITE; -~~~ - -This [`sensitive_table_access`](eventlog.html#sensitive_table_access) event shows that the audited table `customers` was accessed by user `root` issuing an [`INSERT`](insert.html) statement: - -~~~ -I210323 18:50:04.518707 1182 8@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:49851›,hostnossl,user=root] 2 ={"Timestamp":1616525404415644000,"EventType":"sensitive_table_access","Statement":"‹INSERT INTO \"\".\"\".customers(name, address, national_id, telephone, email) VALUES ('Pritchard M. Cleveland', '23 Crooked Lane, Garden City, NY USA 11536', 778124477, 12125552000, 'pritchmeister@aol.com')›","User":"‹root›","DescriptorID":52,"ApplicationName":"‹$ cockroach sql›","ExecMode":"exec","NumRows":1,"Age":103.066,"TxnCounter":28,"TableName":"‹defaultdb.public.customers›","AccessMode":"rw"} -~~~ - -- Preceding the `=` character is the `crdb-v2` event metadata. See the [reference documentation](log-formats.html#format-crdb-v2) for details on the fields. -- `AccessMode` shows that the table was accessed with a read/write (`rw`) operation. -- `ApplicationName` shows that the event originated from the [`cockroach sql`](cockroach-sql.html) shell. You can use this field to filter the logging output by application. - -{{site.data.alerts.callout_info}} -All possible `SENSITIVE_ACCESS` event types are detailed in the [reference documentation](eventlog.html#sql-access-audit-events). For a detailed tutorial on table auditing, see [SQL Audit Logging](sql-audit-logging.html). -{{site.data.alerts.end}} - -### PRIVILEGES - -The [`PRIVILEGES`](logging.html#privileges) channel logs SQL privilege changes. These include DDL operations performed by SQL operations that [modify the privileges](authorization.html#assign-privileges) granted to [roles and users](authorization.html#users-and-roles) on databases, schemas, tables, and [user-defined types](enum.html). - -#### Example: Database privileges - -This [`change_database_privilege`](eventlog.html#change_database_privilege) event shows that user `root` granted all privileges to user `roach` on the database `movr`: - -~~~ -I210329 22:54:48.888312 1742207 7@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:52487›,hostssl,user=root] 1 ={"Timestamp":1617058488747117000,"EventType":"change_database_privilege","Statement":"‹GRANT ALL ON DATABASE movr TO roach›","User":"‹root›","DescriptorID":57,"ApplicationName":"‹$ cockroach sql›","Grantee":"‹roach›","GrantedPrivileges":["ALL"],"DatabaseName":"‹movr›"} -~~~ - -- Preceding the `=` character is the `crdb-v2` event metadata. See the [reference documentation](log-formats.html#format-crdb-v2) for details on the fields. -- `ApplicationName` shows that the event originated from the [`cockroach sql`](cockroach-sql.html) shell. You can use this field to filter the logging output by application. -- `GrantedPrivileges` shows the privileges that were granted. - -{{site.data.alerts.callout_info}} -All possible `PRIVILEGE` event types are detailed in the [reference documentation](eventlog.html#sql-privilege-changes). -{{site.data.alerts.end}} - -### USER_ADMIN - -The [`USER_ADMIN`](logging.html#user_admin) channel logs changes to users and roles. This includes user and role [creation and assignment](authorization.html#create-and-manage-roles) and changes to [privileges](authorization.html#assign-privileges), [options](create-role.html#parameters), and [passwords](authentication.html#client-authentication). - -#### Example: SQL user creation - -This [`create_role`](eventlog.html#create_role) event shows that a user `roach` was created and assigned a password by user `root`. Note that the password in the SQL statement is pre-redacted even if `redact` is set to `false` for the logging sink. For more details on redaction behavior, see [Redact logs](configure-logs.html#redact-logs). - -~~~ -I210323 20:54:53.122681 1943 6@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:51676›,hostssl,user=root] 1 ={"Timestamp":1616532892887402000,"EventType":"create_role","Statement":"‹CREATE USER 'roach' WITH PASSWORD *****›","User":"‹root›","ApplicationName":"‹$ cockroach sql›","RoleName":"‹roach›"} -~~~ - -- Preceding the `=` character is the `crdb-v2` event metadata. See the [reference documentation](log-formats.html#format-crdb-v2) for details on the fields. -- `ApplicationName` shows that the event originated from the [`cockroach sql`](cockroach-sql.html) shell. You can use this field to filter the logging output by application. -- `RoleName` shows the name of the user/role. For details on user and role terminology, see [Users and roles](authorization.html#users-and-roles). - -{{site.data.alerts.callout_info}} -All possible `USER_ADMIN` event types are detailed in the [reference documentation](eventlog.html#sql-user-and-role-operations). -{{site.data.alerts.end}} - -## Performance tuning - -An application developer can use the `SQL_EXEC` and `SQL_PERF` channels to [examine SQL queries](#sql_exec) and filter [slow queries](#sql_perf) in order to optimize or troubleshoot performance. - -In this example configuration, the channels are grouped into a file sink called `performance`. The combined logging output will be found in a `cockroach-performance.log` file at the configured [logging directory](configure-logs.html#logging-directory). - -~~~ yaml -sinks: - file-groups: - performance: - channels: [SQL_EXEC, SQL_PERF] -~~~ - -### SQL_EXEC - -The [`SQL_EXEC`](logging.html#sql_exec) channel reports all SQL executions on the cluster, when enabled. - -To log cluster-wide executions, enable the `sql.trace.log_statement_execute` [cluster setting](cluster-settings.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET CLUSTER SETTING sql.trace.log_statement_execute = true; -~~~ - -Each node of the cluster will write all SQL queries it executes to the `SQL_EXEC` channel. These are recorded as [`query_execute`](eventlog.html#query_execute) events. - -#### Example: SQL query - -This event details a `SELECT` statement that was issued by user `root`: - -~~~ -I210401 22:57:20.047235 5475 9@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:59116›,hostnossl,user=root] 900 ={"Timestamp":1617317840045704000,"EventType":"query_execute","Statement":"‹SELECT * FROM \"\".\"\".users WHERE name = 'Cheyenne Smith'›","User":"‹root›","ApplicationName":"‹$ cockroach sql›","ExecMode":"exec","NumRows":1,"Age":1.583,"FullTableScan":true,"TxnCounter":12} -~~~ - -Note the `FullTableScan` value in the logged event, which shows that this query performed a full table scan and likely caused a performance hit. To learn more about when this issue appears and how it can be resolved, see [SQL Tuning with EXPLAIN](sql-tuning-with-explain.html#issue-full-table-scans). - -- Preceding the `=` character is the `crdb-v2` event metadata. See the [reference documentation](log-formats.html#format-crdb-v2) for details on the fields. -- `ApplicationName` shows that the event originated from the [`cockroach sql`](cockroach-sql.html) shell. You can use this field to filter the logging output by application. - -#### Example: Internal SQL query - -Internal queries are also logged to the `SQL_EXEC` channel. For example, this event details a statement issued on the internal `system.jobs` table: - -~~~ -I210330 16:09:04.966129 1885738 9@util/log/event_log.go:32 ⋮ [n1,intExec=‹find-scheduled-jobs›] 13 ={"Timestamp":1617120544952784000,"EventType":"query_execute","Statement":"‹SELECT (SELECT count(*) FROM \"\".system.jobs AS j WHERE ((j.created_by_type = 'crdb_schedule') AND (j.created_by_id = s.schedule_id)) AND (j.status NOT IN ('succeeded', 'canceled', 'failed'))) AS num_running, s.* FROM \"\".system.scheduled_jobs AS s WHERE next_run < current_timestamp() ORDER BY random() LIMIT 10 FOR UPDATE›","User":"‹root›","ApplicationName":"‹$ internal-find-scheduled-jobs›","ExecMode":"exec-internal","Age":2.934,"FullTableScan":true} -~~~ - -If you no longer need to log queries across the cluster, you can disable the setting: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET CLUSTER SETTING sql.trace.log_statement_execute = false; -~~~ - -{{site.data.alerts.callout_info}} -All possible `SQL_EXEC` event types are detailed in the [reference documentation](eventlog.html#sql-execution-log). -{{site.data.alerts.end}} - -### SQL_PERF - -The [`SQL_PERF`](logging.html#sql_perf) channel reports slow SQL queries, when enabled. This includes queries whose latency exceeds a configured threshold, as well as queries that perform a full table or index scan. - -To enable slow query logging, enable the `sql.log.slow_query.latency_threshold` [cluster setting](cluster-settings.html) by setting it to a non-zero value. This will log queries whose service latency exceeds a specified threshold value. The threshold value must be specified with a unit of time (e.g., `500ms` for 500 milliseconds, `5us` for 5 nanoseconds, or `5s` for 5 seconds). A threshold of `0s` disables the slow query log. - -{{site.data.alerts.callout_info}} -Setting `sql.log.slow_query.latency_threshold` to a non-zero time enables tracing on all queries, which impacts performance. After debugging, set the value back to `0s` to disable the log. -{{site.data.alerts.end}} - -To log all queries that perform full table or index scans to `SQL_PERF`, regardless of query latency, set the `sql.log.slow_query.experimental_full_table_scans.enabled` [cluster setting](cluster-settings.html) to `true`. - -#### Example: Slow SQL query - -For example, to enable the slow query log for all queries with a latency above 100 milliseconds: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET CLUSTER SETTING sql.log.slow_query.latency_threshold = '100ms'; -~~~ - -Each gateway node will now record queries that take longer than 100 milliseconds to the `SQL_PERF` channel as [`slow_query` events](eventlog.html#slow_query). - -This `slow_query` event was logged with a service latency (`age`) of 100.205 milliseconds: - -~~~ -I210323 20:02:16.857133 59270 10@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:50595›,hostnossl,user=root] 573 ={"Timestamp":1616529736756959000,"EventType":"slow_query","Statement":"‹UPDATE \"\".\"\".bank SET balance = CASE id WHEN $1 THEN balance - $3 WHEN $2 THEN balance + $3 END WHERE id IN ($1, $2)›","User":"‹root›","ApplicationName":"‹bank›","PlaceholderValues":["‹158›","‹210›","‹257›"],"ExecMode":"exec","NumRows":2,"Age":100.205,"TxnCounter":97} -~~~ - -- `ApplicationName` shows that the events originated from an application named `bank`. You can use this field to filter the logging output by application. - -The following query was logged with a service latency (`age`) of 9329.26 milliseconds, a very high latency that resulted from a [transaction retry error](transaction-retry-error-reference.html#retry_write_too_old): - -~~~ -I210323 20:02:12.095253 59168 10@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:50621›,hostnossl,user=root] 361 ={"Timestamp":1616529731816553000,"EventType":"slow_query","Statement":"‹UPDATE \"\".\"\".bank SET balance = CASE id WHEN $1 THEN balance - $3 WHEN $2 THEN balance + $3 END WHERE id IN ($1, $2)›","User":"‹root›","ApplicationName":"‹bank›","PlaceholderValues":["‹351›","‹412›","‹206›"],"ExecMode":"exec","SQLSTATE":"40001","ErrorText":"‹TransactionRetryWithProtoRefreshError: WriteTooOldError: write at timestamp 1616529731.152644000,2 too old; wrote at 1616529731.816553000,1: \"sql txn\" meta={id=6c8f776f pri=0.02076160 epo=1 ts=1616529731.816553000,1 min=1616529722.766004000,0 seq=0} lock=true stat=PENDING rts=1616529731.152644000,2 wto=false gul=1616529723.266004000,0›","Age":9329.26,"NumRetries":1,"TxnCounter":1} -~~~ - -- Preceding the `=` character is the `crdb-v2` event metadata. See the [reference documentation](log-formats.html#format-crdb-v2) for details on the fields. -- `ApplicationName` shows that the events originated from an application named `bank`. You can use this field to filter the logging output by application. -- `ErrorText` shows that this query encountered a type of [transaction retry error](transaction-retry-error-reference.html#retry_write_too_old). For details on transaction retry errors and how to resolve them, see the [Transaction retry error reference](transaction-retry-error-reference.html). -- `NumRetries` shows that the transaction was retried once before succeeding. - -{{site.data.alerts.callout_info}} -All possible `SQL_PERF` event types are detailed in the [reference documentation](eventlog.html#sql-slow-query-log). -{{site.data.alerts.end}} - -## Network logging - -A database operator can send logs over the network to a [Fluentd](https://www.fluentd.org/) server. - -{{site.data.alerts.callout_danger}} -TLS is not supported yet: the connection to the log collector is neither authenticated nor encrypted. Given that logging events may contain sensitive information, care should be taken to keep the log collector and the CockroachDB node close together on a private network, or connect them using a secure VPN. TLS support may be added at a later date. -{{site.data.alerts.end}} - -In this example configuration, [operational](#operational-monitoring) and [security](#security-and-audit-monitoring) logs are grouped into separate `ops` and `security` network sinks. The logs from both sinks are sent to a Fluentd server, which can then route them to a compatible log collector (e.g., [Elasticsearch](https://www.elastic.co/elastic-stack), [Splunk](https://www.splunk.com/)). - -{{site.data.alerts.callout_info}} -A network sink can be listed more than once with different `address` values. This routes the same logs to different Fluentd servers. -{{site.data.alerts.end}} - -~~~ yaml -sinks: - fluent-servers: - ops: - channels: [OPS, HEALTH, SQL_SCHEMA] - address: 127.0.0.1:5170 - net: tcp - redact: true - security: - channels: [SESSIONS, USER_ADMIN, PRIVILEGES, SENSITIVE_ACCESS] - address: 127.0.0.1:5170 - net: tcp - auditable: true -~~~ - -In this case, defining separate `ops` and `security` network sinks allows us to: - - - [Enable redaction](configure-logs.html#redact-logs) on the `ops` logs. - - Configure the `security` logs as [`auditable`](#security-and-audit-monitoring). - -Otherwise, it is generally more flexible to [configure Fluentd to route logs](https://docs.fluentd.org/configuration/routing-examples) to different destinations. - -By default, `fluent-servers` log messages use the [`json-fluent-compact`](log-formats.html#format-json-fluent-compact) format for ease of processing over a stream. - -For example, this JSON message found in the `OPS` logging channel contains a [`node_restart`](eventlog.html#node_restart) event. The event shows that a node has rejoined the cluster after being offline (e.g., by being [restarted](cockroach-start.html) after being fully decommissioned): - -~~~ json -{"tag":"cockroach.ops","c":1,"t":"1625766470.804899000","s":1,"sev":"I","g":7,"f":"util/log/event_log.go","l":32,"n":17,"r":1,"tags":{"n":"1"},"event":{"Timestamp":1625766470804896000,"EventType":"node_restart","NodeID":1,"StartedAt":1625766470561283000,"LastUp":1617319541533204000}} -~~~ - -- `tag` is a field required by the [Fluentd protocol](https://docs.fluentd.org/configuration/config-file). -- `sev` shows that the message has the `INFO` [severity level](logging.html#logging-levels-severities). -- `event` contains the fields for the structured [`node_restart`](eventlog.html#node_restart) event. - - `NodeID` shows that the restarted node is `1`. - - `StartedAt` shows the timestamp when the node was most recently restarted. - - `LastUp` shows the timestamp when the node was up before being restarted. - -See the [reference documentation](log-formats.html#format-json-fluent-compact) for details on the remaining JSON fields. - -## See also - -- [Notable Event Types](eventlog.html) -- [SQL Audit Logging](sql-audit-logging.html) -- [Use Prometheus and Alertmanager](monitor-cockroachdb-with-prometheus.html) diff --git a/src/current/v21.1/logging.md b/src/current/v21.1/logging.md deleted file mode 100644 index f02d1b1c332..00000000000 --- a/src/current/v21.1/logging.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: Logging Levels and Channels -summary: Reference documentation for logging levels and logging channels. -toc: true ---- - -{% remote_include https://raw.githubusercontent.com/cockroachdb/cockroach/master/docs/generated/logging.md %} diff --git a/src/current/v21.1/make-queries-fast.md b/src/current/v21.1/make-queries-fast.md deleted file mode 100644 index b7709a4e9ee..00000000000 --- a/src/current/v21.1/make-queries-fast.md +++ /dev/null @@ -1,510 +0,0 @@ ---- -title: Optimize Statement Performance -summary: How to make your statements run faster during application development -toc: true ---- - -This page provides an overview for optimizing statement performance in CockroachDB. To get good performance, you need to look at how you're accessing the database through several lenses: - -- [SQL statement performance](#sql-statement-performance): This is the most common cause of performance problems, and where most developers should start. -- [Schema design](#schema-design): Depending on your SQL schema and the data access patterns of your workload, you may need to make changes to avoid creating "hotspots". -- [Cluster topology](#cluster-topology): As a distributed system, CockroachDB requires you to trade off latency vs. resiliency. This requires choosing the right cluster topology for your needs. - -{{site.data.alerts.callout_info}} -If you aren't sure whether SQL statement performance needs to be improved on your cluster, see [Identify slow statements](query-behavior-troubleshooting.html#identify-slow-statements). -{{site.data.alerts.end}} - -## SQL statement performance - -To get good SQL statement performance, follow the rules below (in approximate order of importance): - -{{site.data.alerts.callout_info}} -These rules apply to an environment where thousands of [OLTP](https://en.wikipedia.org/wiki/Online_transaction_processing) statements are being run per second, and each statement needs to run in milliseconds. These rules are not intended to apply to analytical, or [OLAP](https://en.wikipedia.org/wiki/Online_analytical_processing), statements. -{{site.data.alerts.end}} - -- [Rule 1. Scan as few rows as possible](#rule-1-scan-as-few-rows-as-possible). If your application is scanning more rows than necessary for a given statement, it's going to be difficult to scale. -- [Rule 2. Use the right index](#rule-2-use-the-right-index): Your statement should use an index on the columns in the `WHERE` clause. You want to avoid the performance hit of a full table scan. -- [Rule 3. Use the right join type](#rule-3-use-the-right-join-type): Depending on the relative sizes of the tables you are querying, the type of [join][joins] may be important. This should rarely be necessary because the [cost-based optimizer](cost-based-optimizer.html) should pick the best-performing join type if you add the right indexes as described in Rule 2. - -{{site.data.alerts.callout_info}} -To identify poorly performing statements, use the [DB Console and slow query log](query-behavior-troubleshooting.html#identify-slow-statements). -{{site.data.alerts.end}} - -To show each of these rules in action, we will optimize a statement against the [MovR data set](movr.html) as follows: - -{% include {{ page.version.version }}/demo_movr.md %} - -It's common to offer users promo codes to increase usage and customer loyalty. In this scenario, we want to find the 10 users who have taken the highest number of rides on a given date, and offer them promo codes that provide a 10% discount. - -To phrase it in the form of a question: "Who are the top 10 users by number of rides on a given date?" - -{% comment %} -This is a test -{% endcomment %} - -### Rule 1. Scan as few rows as possible - -First, let's study the schema so we understand the relationships between the tables. - -Start a SQL shell: - -{% include_cached copy-clipboard.html %} -~~~ shell -cockroach sql --insecure -~~~ - -Next, set `movr` as the current database and run [`SHOW TABLES`](show-tables.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -USE movr; -SHOW TABLES; -~~~ - -~~~ - schema_name | table_name | type | estimated_row_count ---------------+----------------------------+-------+---------------------- - public | promo_codes | table | 250000 - public | rides | table | 125000 - public | user_promo_codes | table | 0 - public | users | table | 12500 - public | vehicle_location_histories | table | 250000 - public | vehicles | table | 3750 -(6 rows) - -Time: 17ms total (execution 17ms / network 0ms) -~~~ - -Let's look at the schema for the `users` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW CREATE TABLE users; -~~~ - -~~~ - table_name | create_statement --------------+-------------------------------------------------------------- - users | CREATE TABLE public.users ( - | id UUID NOT NULL, - | city VARCHAR NOT NULL, - | name VARCHAR NULL, - | address VARCHAR NULL, - | credit_card VARCHAR NULL, - | CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - | FAMILY "primary" (id, city, name, address, credit_card) - | ) -(1 row) - -Time: 9ms total (execution 9ms / network 0ms) -~~~ - -There's no information about the number of rides taken here, nor anything about the days on which rides occurred. Luckily, there is also a `rides` table. Let's look at it: - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW CREATE TABLE rides; -~~~ - -~~~ - table_name | create_statement --------------+---------------------------------------------------------------------------------------------------------------------------------- - rides | CREATE TABLE public.rides ( - | id UUID NOT NULL, - | city VARCHAR NOT NULL, - | vehicle_city VARCHAR NULL, - | rider_id UUID NULL, - | vehicle_id UUID NULL, - | start_address VARCHAR NULL, - | end_address VARCHAR NULL, - | start_time TIMESTAMP NULL, - | end_time TIMESTAMP NULL, - | revenue DECIMAL(10,2) NULL, - | CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - | CONSTRAINT fk_city_ref_users FOREIGN KEY (city, rider_id) REFERENCES public.users(city, id), - | CONSTRAINT fk_vehicle_city_ref_vehicles FOREIGN KEY (vehicle_city, vehicle_id) REFERENCES public.vehicles(city, id), - | INDEX rides_auto_index_fk_city_ref_users (city ASC, rider_id ASC), - | INDEX rides_auto_index_fk_vehicle_city_ref_vehicles (vehicle_city ASC, vehicle_id ASC), - | FAMILY "primary" (id, city, vehicle_city, rider_id, vehicle_id, start_address, end_address, start_time, end_time, revenue), - | CONSTRAINT check_vehicle_city_city CHECK (vehicle_city = city) - | ) -(1 row) - -Time: 9ms total (execution 8ms / network 1ms) -~~~ - -There is a `rider_id` field that we can use to match each ride to a user. There is also a `start_time` field that we can use to filter the rides by date. - -This means that to get the information we want, we'll need to do a [join][joins] on the `users` and `rides` tables. - -Next, let's get the row counts for the tables that we'll be using in this query. We need to understand which tables are large, and which are small by comparison. We will need this later if we need to verify we are [using the right join type](#rule-3-use-the-right-join-type). - -As specified above by our [`cockroach workload`](cockroach-workload.html) command, the `users` table has 12,500 records, and the `rides` table has 125,000 records. Because it's so large, we want to avoid scanning the entire `rides` table in our query. In this case, we can avoid scanning `rides` using an index, as shown in the next section. - -### Rule 2. Use the right index - -Below is a query that fetches the right answer to our question: "Who are the top 10 users by number of rides on a given date?" - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT - name, count(rides.id) AS sum -FROM - users JOIN rides ON users.id = rides.rider_id -WHERE - rides.start_time BETWEEN '2018-12-31 00:00:00' AND '2020-01-01 00:00:00' -GROUP BY - name -ORDER BY - sum DESC -LIMIT - 10; -~~~ - -~~~ - name | sum --------------------+------ - William Brown | 14 - William Mitchell | 10 - Joseph Smith | 10 - Paul Nelson | 9 - Christina Smith | 9 - Jennifer Johnson | 8 - Jeffrey Walker | 8 - Joseph Jones | 7 - James Williams | 7 - Thomas Smith | 7 -(10 rows) - -Time: 111ms total (execution 111ms / network 0ms) -~~~ - -Unfortunately, this query is a bit slow. 111 milliseconds puts us [over the limit where a user feels the system is reacting instantaneously](https://www.nngroup.com/articles/response-times-3-important-limits/), and we're still down in the database layer. This data still needs to be shipped back out to your application and displayed to the user. - -We can see why if we look at the output of [`EXPLAIN`](explain.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -EXPLAIN SELECT - name, count(rides.id) AS sum -FROM - users JOIN rides ON users.id = rides.rider_id -WHERE - rides.start_time BETWEEN '2018-12-31 00:00:00' AND '2019-01-01 00:00:00' -GROUP BY - name -ORDER BY - sum DESC -LIMIT - 10; -~~~ - -~~~ - info -------------------------------------------------------------------------------------------------------------- - distribution: full - vectorized: true - - • limit - │ estimated row count: 10 - │ count: 10 - │ - └── • sort - │ estimated row count: 3,392 - │ order: -count_rows - │ - └── • group - │ estimated row count: 3,392 - │ group by: name - │ - └── • hash join - │ estimated row count: 4,013 - │ equality: (id) = (rider_id) - │ - ├── • scan - │ estimated row count: 12,500 (100% of the table; stats collected 4 minutes ago) - │ table: users@primary - │ spans: FULL SCAN - │ - └── • filter - │ estimated row count: 4,013 - │ filter: (start_time >= '2018-12-31 00:00:00') AND (start_time <= '2019-01-01 00:00:00') - │ - └── • scan - estimated row count: 125,000 (100% of the table; stats collected 3 minutes ago) - table: rides@primary - spans: FULL SCAN -(32 rows) - -Time: 2ms total (execution 2ms / network 0ms) -~~~ - -The main problem is that we are doing full table scans on both the `users` and `rides` tables (see `spans: FULL SCAN`). This tells us that we do not have indexes on the columns in our `WHERE` clause, which is [an indexing best practice](indexes.html#best-practices). - -Therefore, we need to create an index on the column in our `WHERE` clause, in this case: `rides.start_time`. - -It's also possible that there is not an index on the `rider_id` column that we are doing a join against, which will also hurt performance. - -Before creating any more indexes, let's see what indexes already exist on the `rides` table by running [`SHOW INDEXES`](show-index.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW INDEXES FROM rides; -~~~ - -~~~ - table_name | index_name | non_unique | seq_in_index | column_name | direction | storing | implicit --------------+-----------------------------------------------+------------+--------------+--------------+-----------+---------+----------- - rides | primary | false | 1 | city | ASC | false | false - rides | primary | false | 2 | id | ASC | false | false - rides | rides_auto_index_fk_city_ref_users | true | 1 | city | ASC | false | false - rides | rides_auto_index_fk_city_ref_users | true | 2 | rider_id | ASC | false | false - rides | rides_auto_index_fk_city_ref_users | true | 3 | id | ASC | false | true - rides | rides_auto_index_fk_vehicle_city_ref_vehicles | true | 1 | vehicle_city | ASC | false | false - rides | rides_auto_index_fk_vehicle_city_ref_vehicles | true | 2 | vehicle_id | ASC | false | false - rides | rides_auto_index_fk_vehicle_city_ref_vehicles | true | 3 | id | ASC | false | true - rides | rides_auto_index_fk_vehicle_city_ref_vehicles | true | 4 | city | ASC | false | true -(9 rows) - -Time: 5ms total (execution 5ms / network 0ms) -~~~ - -As we suspected, there are no indexes on `start_time` or `rider_id`, so we'll need to create indexes on those columns. - -Because another performance best practice is to [create an index on the `WHERE` condition storing the join key](sql-tuning-with-explain.html#solution-create-a-secondary-index-on-the-where-condition-storing-the-join-key), we will create an index on `start_time` that stores the join key `rider_id`: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE INDEX ON rides (start_time) storing (rider_id); -~~~ - -Now that we have an index on the column in our `WHERE` clause that stores the join key, let's run the query again: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT - name, count(rides.id) AS sum -FROM - users JOIN rides ON users.id = rides.rider_id -WHERE - rides.start_time BETWEEN '2018-12-31 00:00:00' AND '2019-01-01 00:00:00' -GROUP BY - name -ORDER BY - sum DESC -LIMIT - 10; -~~~ - -~~~ - name | sum --------------------+------ - William Brown | 6 - Laura Marsh | 5 - Joseph Smith | 5 - David Martinez | 4 - Michael Garcia | 4 - David Mitchell | 4 - Arthur Nielsen | 4 - Michael Bradford | 4 - William Mitchell | 4 - Jennifer Johnson | 4 -(10 rows) - -Time: 22ms total (execution 22ms / network 0ms) -~~~ - -This query is now running much faster than it was before we added the indexes (111ms vs. 22ms). This means we have an extra 89 milliseconds we can budget towards other areas of our application. - -To see what changed, let's look at the [`EXPLAIN`](explain.html) output: - -{% include_cached copy-clipboard.html %} -~~~ sql -EXPLAIN SELECT - name, count(rides.id) AS sum -FROM - users JOIN rides ON users.id = rides.rider_id -WHERE - rides.start_time BETWEEN '2018-12-31 00:00:00' AND '2019-01-01 00:00:00' -GROUP BY - name -ORDER BY - sum DESC -LIMIT - 10; -~~~ - -As you can see, this query is no longer scanning the entire (larger) `rides` table. Instead, it is now doing a much smaller range scan against only the values in `rides` that match the index we just created on the `start_time` column (4,013 rows instead of 125,000). - -~~~ - info ----------------------------------------------------------------------------------------------------- - distribution: full - vectorized: true - - • limit - │ estimated row count: 10 - │ count: 10 - │ - └── • sort - │ estimated row count: 3,392 - │ order: -count_rows - │ - └── • group - │ estimated row count: 3,392 - │ group by: name - │ - └── • hash join - │ estimated row count: 4,013 - │ equality: (id) = (rider_id) - │ - ├── • scan - │ estimated row count: 12,500 (100% of the table; stats collected 8 minutes ago) - │ table: users@primary - │ spans: FULL SCAN - │ - └── • scan - estimated row count: 4,013 (3.2% of the table; stats collected 7 minutes ago) - table: rides@rides_start_time_idx - spans: [/'2018-12-31 00:00:00' - /'2019-01-01 00:00:00'] -(28 rows) - -Time: 2ms total (execution 1ms / network 0ms) -~~~ - - -### Rule 3. Use the right join type - -Out of the box, the [cost-based optimizer](cost-based-optimizer.html) will select the right join type for your statement in the majority of cases. Therefore, you should only provide [join hints](cost-based-optimizer.html#join-hints) in your query if you can **prove** to yourself through experimentation that the optimizer should be using a different [join type](joins.html#join-algorithms) than it is selecting. - -We can confirm that in this case the optimizer has already found the right join type for this statement by using a hint to force another join type. - -For example, we might think that a [lookup join](joins.html#lookup-joins) could perform better in this instance, since one of the tables in the join is 10x smaller than the other. - -In order to get CockroachDB to plan a lookup join in this case, we will need to add an explicit index on the join key for the right-hand-side table, in this case, `rides`. - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE INDEX ON rides (rider_id); -~~~ - -Next, we can specify the lookup join with a join hint: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT - name, count(rides.id) AS sum -FROM - users INNER LOOKUP JOIN rides ON users.id = rides.rider_id -WHERE - (rides.start_time BETWEEN '2018-12-31 00:00:00' AND '2019-01-01 00:00:00') -GROUP BY - name -ORDER BY - sum DESC -LIMIT - 10; -~~~ - -~~~ - name | sum --------------------+------ - William Brown | 6 - Laura Marsh | 5 - Joseph Smith | 5 - Michael Garcia | 4 - David Mitchell | 4 - David Martinez | 4 - Arthur Nielsen | 4 - Jennifer Johnson | 4 - William Mitchell | 4 - Michael Bradford | 4 -(10 rows) - -Time: 1.548s total (execution 1.548s / network 0.000s) -~~~ - -The results, however, are not good. The query is much slower using a lookup join than what CockroachDB planned for us earlier. - -The query is a little faster when we force CockroachDB to use a merge join: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT - name, count(rides.id) AS sum -FROM - users INNER MERGE JOIN rides ON users.id = rides.rider_id -WHERE - (rides.start_time BETWEEN '2018-12-31 00:00:00' AND '2019-01-01 00:00:00') -GROUP BY - name -ORDER BY - sum DESC -LIMIT - 10; -~~~ - -~~~ - name | sum -+-------------------+-----+ - William Brown | 6 - Laura Marsh | 5 - Joseph Smith | 5 - Jennifer Ford | 4 - David Mitchell | 4 - William Mitchell | 4 - Christopher Allen | 4 - Michael Bradford | 4 - Michael Garcia | 4 - Jennifer Johnson | 4 -(10 rows) - -Time: 22ms total (execution 22ms / network 0ms) -~~~ - -The results are consistently about 20-26ms with a merge join vs. 16-23ms when we let CockroachDB choose the join type as shown in the previous section. In other words, forcing the merge join is slightly slower than if we had done nothing. - -## Schema design - -If you are following the instructions in [the SQL performance section](#sql-statement-performance) and still not getting the performance you want, you may need to look at your schema design and data access patterns to make sure you are not creating "hotspots" in your cluster that will lead to performance problems due to transaction contention. - -You can avoid contention with the following strategies: - -- Use index keys with a more random distribution of values, as described in [Unique ID best practices](performance-best-practices-overview.html#unique-id-best-practices). -- Make transactions smaller by operating on less data per transaction. This will offer fewer opportunities for transactions' data access to overlap. -- [Split the table across multiple ranges](split-at.html) to distribute its data across multiple nodes for better load balancing of some write-heavy workloads. - -For more information about how to avoid performance problems caused by contention, see [Understanding and Avoiding Transaction Contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention). - -## Cluster topology - -It's very important to make sure that the cluster topology you are using is the right one for your use case. Because CockroachDB is a distributed system that involves nodes communicating over the network, you need to choose the cluster topology that results in the right latency vs. resiliency tradeoff. - -For more information about how to choose the cluster topology that is right for your application, see [this list of topology patterns](topology-patterns.html). - -## See also - -Reference information: - -- [SQL Performance Best Practices](performance-best-practices-overview.html) -- [SQL Tuning with `EXPLAIN`](sql-tuning-with-explain.html) -- [Joins](joins.html) -- [CockroachDB Performance](performance.html) -- [Understanding and Avoiding Transaction Contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention) -- [Topology Patterns](topology-patterns.html) - -Specific tasks: - -- [Connect to the Database](connect-to-the-database.html) -- [Insert Data](insert-data.html) -- [Query Data](query-data.html) -- [Update Data](update-data.html) -- [Delete Data](delete-data.html) -- [Run Multi-Statement Transactions](run-multi-statement-transactions.html) -- [Identify slow queries](query-behavior-troubleshooting.html#identify-slow-statements) -- [Error Handling and Troubleshooting](error-handling-and-troubleshooting.html) -- [Hello World Example apps](example-apps.html) - - - -[joins]: joins.html diff --git a/src/current/v21.1/manage-a-backup-schedule.md b/src/current/v21.1/manage-a-backup-schedule.md deleted file mode 100644 index ffe140a163d..00000000000 --- a/src/current/v21.1/manage-a-backup-schedule.md +++ /dev/null @@ -1,281 +0,0 @@ ---- -title: Manage a Backup Schedule -summary: Learn about how to create, manage, and restore from a schedule of periodic backups. -toc: true ---- - - You can create schedules in CockroachDB for periodic backups. Once a [backup schedule is created](#create-a-new-backup-schedule), you can do the following: - -- [Set up monitoring for the backup schedule](#set-up-monitoring-for-the-backup-schedule) -- [View scheduled backup details](#view-scheduled-backup-details) -- [View and control the backup schedule](#view-and-control-the-backup-schedule) -- [View and control a backup initiated by a schedule](#view-and-control-a-backup-initiated-by-a-schedule) -- [Restore from a scheduled backup](#restore-from-a-scheduled-backup) - -## Create a new backup schedule - -To create a new backup schedule, use the [`CREATE SCHEDULE FOR BACKUP`](create-schedule-for-backup.html) statement. For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE SCHEDULE schedule_label - FOR BACKUP INTO 's3://test/backups/test_schedule_1?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}' - WITH revision_history - RECURRING '@daily' - WITH SCHEDULE OPTIONS first_run = 'now'; -~~~ - -In this example, a schedule labeled `schedule_label` is created to take daily (incremental) backups with revision history in AWS S3, with the first backup being taken now. A second [schedule for weekly full backups](create-schedule-for-backup.html#full-backup-clause) is also created by default. Both schedules have the same `label` (i.e., `schedule_label`). - -For more information about the different options available when creating a backup schedule, see [`CREATE SCHEDULE FOR BACKUP`](create-schedule-for-backup.html). - -{{site.data.alerts.callout_info}} -Further guidance on connecting to Amazon S3, Google Cloud Storage, Azure Storage, and other storage options is outlined in [Use Cloud Storage for Bulk Operations](use-cloud-storage-for-bulk-operations.html). -{{site.data.alerts.end}} - -## Set up monitoring for the backup schedule - -We recommend that you [monitor your backup schedule with Prometheus](monitoring-and-alerting.html#prometheus-endpoint), and alert when there are anomalies such as backups that have failed or no backups succeeding over a certain amount of time— at which point, you can inspect schedules by running [`SHOW SCHEDULES`](show-schedules.html). - -Metrics for scheduled backups fall into two categories: - -- Backup schedule-specific metrics, aggregated across all schedules: - - - `schedules_BACKUP_started`: A counter for the total number of backups started by a schedule - - `schedules_BACKUP_succeeded`: A counter for the number of backups started by a schedule that succeeded - - `schedules_BACKUP_failed`: A counter for the number of backups started by a schedule that failed - - When `schedules_BACKUP_failed` increments, run [`SHOW SCHEDULES`](show-schedules.html) to check which schedule is affected and to inspect the error in the `status` column. - -- Scheduler-specific metrics: - - - `schedules.round.reschedule-wait`: The number of schedules that were rescheduled due to a currently running job. A value greater than 0 indicates that a previous backup was still running when a new scheduled backup was supposed to start. This corresponds to the [`on_previous_running=wait`](create-schedule-for-backup.html#on-previous-running-option) schedule option. - - - `schedules.round.reschedule-skip`: The number of schedules that were skipped due to a currently running job. A value greater than 0 indicates that a previous backup was still running when a new scheduled backup was supposed to start. This corresponds to the [`on_previous_running=skip`](create-schedule-for-backup.html#on-previous-running-option) schedule option. - -{{site.data.alerts.callout_info}} -`schedules.round.reschedule-wait` and `schedules.round.reschedule-skip` are gauge metrics and can be graphed. A continual positive value for either of these metrics may indicate a misconfigured backup cadence, and you should consider adjusting the cadence to avoid waiting for or skipping the next backup. -{{site.data.alerts.end}} - -For a tutorial on how to use Prometheus to set up monitoring and alerting, see [Monitor CockroachDB with Prometheus](monitor-cockroachdb-with-prometheus.html). - -## View scheduled backup details - - When a [backup is created by a schedule](create-schedule-for-backup.html), it is stored within a collection of backups in the given location. To view details for a backup created by a schedule, you can use the following: - -- `SHOW BACKUPS IN collectionURI` statement to [view a list of the full backup's subdirectories](show-backup.html#view-a-list-of-the-available-full-backup-subdirectories). -- `SHOW BACKUP subdirectory IN collectionURI` statement to [view a list of the full and incremental backups that are stored in a specific full backup's subdirectory](show-backup.html#view-a-list-of-the-full-and-incremental-backups-in-a-specific-full-backup-subdirectory). - -For more details, see [`SHOW BACKUP`](show-backup.html). - -## View and control the backup schedule - -Once a backup schedule is successfully created, you can [view the schedule](#view-the-schedule), [pause the schedule](#pause-the-schedule), [resume the schedule](#resume-the-schedule), or [drop the schedule](#drop-the-schedule). - -### View the schedule - -{% include_cached copy-clipboard.html %} -~~~~ -> SHOW SCHEDULES; -~~~~ - -For more information, see [`SHOW SCHEDULES`](show-schedules.html). - -### Pause the schedule - -To pause a schedule, you can either specify the schedule's `id`: - -{% include_cached copy-clipboard.html %} -~~~~ sql -> PAUSE SCHEDULE 589963390487363585; -~~~~ - -Or nest a [`SELECT` clause](select-clause.html) that retrieves `id`(s) inside the `PAUSE SCHEDULES` statement: - -{% include_cached copy-clipboard.html %} -~~~~ sql -> PAUSE SCHEDULES SELECT id FROM [SHOW SCHEDULES] WHERE label = 'schedule_database'; -~~~~ - -For more information, see [`PAUSE SCHEDULES`](pause-schedules.html). - -### Resume the schedule - -To resume a paused schedule, you can either specify the schedule's `id`: - -{% include_cached copy-clipboard.html %} -~~~~ sql -> RESUME SCHEDULE 589963390487363585; -~~~~ - -Or nest a [`SELECT` clause](select-clause.html) that retrieves `id`(s) inside the `RESUME SCHEDULES` statement: - -{% include_cached copy-clipboard.html %} -~~~~ sql -> RESUME SCHEDULES SELECT id FROM [SHOW SCHEDULES] WHERE label = 'schedule_database'; -~~~~ - -For more information, see [`RESUME SCHEDULES`](resume-schedules.html). - -### Drop the schedule - -To drop a schedule, you can either specify the schedule's `id`: - -{% include_cached copy-clipboard.html %} -~~~~ sql -> DROP SCHEDULE 589963390487363585; -~~~~ - -Or nest a [`SELECT` clause](select-clause.html) that retrieves `id`(s) inside the `DROP SCHEDULES` statement: - -{% include_cached copy-clipboard.html %} -~~~~ sql -> DROP SCHEDULES SELECT id FROM [SHOW SCHEDULES] WHERE label = 'schedule_database'; -~~~~ - -For more information, see [`DROP SCHEDULES`](drop-schedules.html). - -{{site.data.alerts.callout_danger}} -`DROP SCHEDULE` does **not** cancel any in-progress jobs started by the schedule. Before you drop a schedule, [cancel any in-progress jobs](cancel-job.html) first, as you will not be able to look up the job ID once the schedule is dropped. -{{site.data.alerts.end}} - -## View and control a backup initiated by a schedule - -After CockroachDB successfully initiates a scheduled backup, it registers the backup as a job. You can [view](#view-the-backup-job), [pause](#pause-the-backup-job), [resume](#resume-the-backup-job), or [cancel](#cancel-the-backup-job) each individual backup job. - -### View the backup job - -To view jobs for a specific [backup schedule](create-schedule-for-backup.html), use the schedule's `id`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW JOBS FOR SCHEDULE 590204387299262465; -~~~ -~~~ - job_id | job_type | description | statement | user_name | status | running_status | created | started | finished | modified | fraction_completed | error | coordinator_id ----------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+-----------+---------+----------------+----------------------------------+---------+----------+----------------------------------+--------------------+-------+----------------- - 590205481558802434 | BACKUP | BACKUP INTO '/2020/09/15-161444.99' IN 's3://test/scheduled-backup-0915?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=redacted' AS OF SYSTEM TIME '2020-09-15 16:20:00+00:00' WITH revision_history, detached | | root | running | NULL | 2020-09-15 16:20:18.347383+00:00 | NULL | NULL | 2020-09-15 16:20:18.347383+00:00 | 0 | | 0 -(1 row) -~~~ - -You can also view multiple schedules by nesting a [`SELECT` clause](select-clause.html) that retrieves `id`(s) inside the `SHOW JOBS` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW JOBS FOR SCHEDULES SELECT id FROM [SHOW SCHEDULES] WHERE label = 'test_schedule'; -~~~ - -~~~ - job_id | job_type | description | statement | user_name | status | running_status | created | started | finished | modified | fraction_completed | error | coordinator_id ----------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+-----------+-----------+----------------+----------------------------------+---------+----------------------------------+----------------------------------+--------------------+-------+----------------- - 590204496007299074 | BACKUP | BACKUP INTO '/2020/09/15-161444.99' IN 's3://test/scheduled-backup-0915?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=redacted' AS OF SYSTEM TIME '2020-09-15 16:14:44.991631+00:00' WITH revision_history, detached | | root | succeeded | NULL | 2020-09-15 16:15:17.720725+00:00 | NULL | 2020-09-15 16:15:20.913789+00:00 | 2020-09-15 16:15:20.910594+00:00 | 1 | | 0 - 590205481558802434 | BACKUP | BACKUP INTO '/2020/09/15-161444.99' IN 's3://test/scheduled-backup-0915?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=redacted' AS OF SYSTEM TIME '2020-09-15 16:20:00+00:00' WITH revision_history, detached | | root | succeeded | NULL | 2020-09-15 16:20:18.347383+00:00 | NULL | 2020-09-15 16:20:48.37873+00:00 | 2020-09-15 16:20:48.374256+00:00 | 1 | | 0 -(2 rows) -~~~ - -For more information, see [`SHOW JOBS`](show-jobs.html). - -### Pause the backup job - -To pause jobs for a specific [backup schedule](create-schedule-for-backup.html), use the schedule's `id`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> PAUSE JOBS FOR SCHEDULE 590204387299262465; -~~~ -~~~ -PAUSE JOBS FOR SCHEDULES 1 -~~~ - -You can also pause multiple schedules by nesting a [`SELECT` clause](select-clause.html) that retrieves `id`(s) inside the `PAUSE JOBS` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> PAUSE JOBS FOR SCHEDULES SELECT id FROM [SHOW SCHEDULES] WHERE label = 'test_schedule'; -~~~ - -~~~ -PAUSE JOBS FOR SCHEDULES 2 -~~~ - -For more information, see [`PAUSE JOB`](pause-job.html). - -### Resume the backup job - -To resume jobs for a specific [backup schedule](create-schedule-for-backup.html), use the schedule's `id`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESUME JOBS FOR SCHEDULE 590204387299262465; -~~~ -~~~ -RESUME JOBS FOR SCHEDULES 1 -~~~ - -You can also resume multiple schedules by nesting a [`SELECT` clause](select-clause.html) that retrieves `id`(s) inside the `PAUSE JOBS` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESUME JOBS FOR SCHEDULES SELECT id FROM [SHOW SCHEDULES] WHERE label = 'test_schedule'; -~~~ - -~~~ -RESUME JOBS FOR SCHEDULES 2 -~~~ - -For more information, see [`RESUME JOB`](resume-job.html). - -### Cancel the backup job - -To cancel jobs for a specific [backup schedule](create-schedule-for-backup.html), use the schedule's `id`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CANCEL JOBS FOR SCHEDULE 590204387299262465; -~~~ -~~~ -CANCEL JOBS FOR SCHEDULES 1 -~~~ - -You can also CANCEL multiple schedules by nesting a [`SELECT` clause](select-clause.html) that retrieves `id`(s) inside the `CANCEL JOBS` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CANCEL JOBS FOR SCHEDULES SELECT id FROM [SHOW SCHEDULES] WHERE label = 'test_schedule'; -~~~ - -~~~ -CANCEL JOBS FOR SCHEDULES 2 -~~~ - -For more information, see [`CANCEL JOB`](cancel-job.html). - -## Restore from a scheduled backup - -To restore from a scheduled backup, use the [`RESTORE`](restore.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESTORE - FROM '2020/08/19-035600.00' IN 's3://test/backups/test_schedule_1?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}' - AS OF SYSTEM TIME '2020-08-19 03:50:00+00:00'; -~~~ - -To view the backups stored within a collection, use the [`SHOW BACKUP`](#view-scheduled-backup-details) statement. - -## See also - -- [`CREATE SCHEDULE FOR BACKUP`](create-schedule-for-backup.html) -- [`BACKUP`](backup.html) -- [`RESTORE`](restore.html) -- [`SHOW BACKUP`](show-backup.html) -- [`SHOW SCHEDULES`](show-schedules.html) -- [`PAUSE SCHEDULES`](pause-schedules.html) -- [`RESUME SCHEDULES`](resume-schedules.html) -- [`DROP SCHEDULES`](drop-schedules.html) -- [`PAUSE JOB`](pause-job.html) -- [`RESUME JOB`](pause-job.html) -- [`CANCEL JOB`](cancel-job.html) -- [Take Full and Incremental Backups](take-full-and-incremental-backups.html) -- [Use the Built-in SQL Client](cockroach-sql.html) -- [Other Cockroach Commands](cockroach-commands.html) diff --git a/src/current/v21.1/manage-long-running-queries.md b/src/current/v21.1/manage-long-running-queries.md deleted file mode 100644 index 7352f7174c6..00000000000 --- a/src/current/v21.1/manage-long-running-queries.md +++ /dev/null @@ -1,72 +0,0 @@ ---- -title: Manage Long-Running Queries -summary: Learn how to identify and cancel long-running queries. -toc: true ---- - -This page shows you how to identify and, if necessary, cancel SQL queries that are taking longer than expected to process. - -{{site.data.alerts.callout_success}} Schema changes are treated differently than other SQL queries. You can use SHOW JOBS to monitor the progress of schema changes and CANCEL JOB to cancel schema changes that are taking longer than expected. {{site.data.alerts.end}} - - -## Identify long-running queries - -Use the [`SHOW STATEMENTS`](show-statements.html) statement to list details about currently active SQL queries, including each query's `start` timestamp: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW CLUSTER STATEMENTS] - WHERE application_name != '$ cockroach sql'; -~~~ - -~~~ - query_id | node_id | session_id | user_name | start | query | client_address | application_name | distributed | phase -+----------------------------------+---------+----------------------------------+-----------+----------------------------------+-----------------------------------------------------------------------+-----------------+------------------+-------------+-----------+ - 15f92c0dd24bec200000000000000003 | 3 | 15f92b0e4ea399680000000000000003 | root | 2020-03-04 18:06:21.871708+00:00 | SELECT city, id FROM vehicles WHERE city = $1 | 127.0.0.1:65088 | | false | executing - 15f92c0dd26655d80000000000000001 | 1 | 15f92be36964ac800000000000000001 | root | 2020-03-04 18:06:21.873515+00:00 | UPSERT INTO vehicle_location_histories VALUES ($1, $2, now(), $3, $4) | 127.0.0.1:65240 | | false | executing - 15f92c0dd25882c80000000000000001 | 1 | 15f92aefb240d2980000000000000001 | root | 2020-03-04 18:06:21.872608+00:00 | UPSERT INTO vehicle_location_histories VALUES ($1, $2, now(), $3, $4) | 127.0.0.1:65044 | | false | executing - 15f92c0dd262cb980000000000000002 | 2 | 15f92b7dc85b7ba80000000000000002 | maxroach | 2020-03-04 18:06:21.873286+00:00 | SELECT city, id FROM vehicles WHERE city = $1 | 127.0.0.1:65196 | | false | executing -~~~ - -You can also filter for queries that have been running for a certain amount of time. For example, to find queries that have been running for more than 3 hours, you would run the following: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW CLUSTER STATEMENTS] - WHERE start < (now() - INTERVAL '3 hours'); -~~~ - -## Cancel long-running queries - -Once you've identified a long-running query via [`SHOW STATEMENTS`](show-statements.html), note the `query_id` and use it with the [`CANCEL QUERY`](cancel-query.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CANCEL QUERY '15f92c0dd24bec200000000000000003'; -~~~ - -When a query is successfully cancelled, CockroachDB sends a `query execution canceled` error to the client that issued the query. - -- If the canceled query was a single, stand-alone statement, no further action is required by the client. -- If the canceled query was part of a larger, multi-statement [transaction](transactions.html), the client should then issue a [`ROLLBACK`](rollback-transaction.html) statement. - -You can cancel all queries from a particular application by using a subquery. - -{% include_cached copy-clipboard.html %} -~~~ sql -CANCEL QUERIES (SELECT query_id FROM [SHOW CLUSTER QUERIES] - WHERE application_name = 'test_app'); -~~~ - -## Improve query performance - -After cancelling a long-running query, use the [`EXPLAIN`](explain.html) statement to examine it. It's possible that the query was slow because it performs a full-table scan. In these cases, you can likely improve the query's performance by [adding an index](create-index.html). - -*(More guidance around query performance optimization forthcoming.)* - -## See also - -- [`SHOW STATEMENTS`](show-statements.html) -- [`CANCEL QUERY`](cancel-query.html) -- [`EXPLAIN`](explain.html) -- [Query Behavior Troubleshooting](query-behavior-troubleshooting.html) diff --git a/src/current/v21.1/manual-deployment.md b/src/current/v21.1/manual-deployment.md deleted file mode 100644 index ff1295a2652..00000000000 --- a/src/current/v21.1/manual-deployment.md +++ /dev/null @@ -1,22 +0,0 @@ ---- -title: Manual Deployment -summary: Learn how to deploy CockroachDB manually on-premises or on popular cloud platforms. -toc: false ---- - -Use the following guides to deploy CockroachDB manually on-premises or on popular cloud platforms: - -- [On-Premises](deploy-cockroachdb-on-premises.html) -- [Amazon Web Services (AWS)](deploy-cockroachdb-on-aws.html) -- [Digital Ocean](deploy-cockroachdb-on-digital-ocean.html) -- [Google Cloud Platform (GCE)](deploy-cockroachdb-on-google-cloud-platform.html) -- [Microsoft Azure](deploy-cockroachdb-on-microsoft-azure.html) - -{% include cockroachcloud/use-cockroachcloud-instead.md %} - -## See also - -- [Production Checklist](recommended-production-settings.html) -- [Orchestrated Deployment](orchestration.html) -- [Monitoring and Alerting](monitoring-and-alerting.html) -- [Local Deployment](start-a-local-cluster.html) diff --git a/src/current/v21.1/migrate-from-avro.md b/src/current/v21.1/migrate-from-avro.md deleted file mode 100644 index 44927a7b340..00000000000 --- a/src/current/v21.1/migrate-from-avro.md +++ /dev/null @@ -1,225 +0,0 @@ ---- -title: Migrate from Avro -summary: Learn how to migrate data from Avro files into a CockroachDB cluster. -toc: true ---- - -This page has instructions for migrating data from Avro files into CockroachDB using [`IMPORT`][import]. - -{% include {{ page.version.version }}/misc/import-perf.md %} - -## Step 1. Export data to Avro - -Please refer to the documentation of your database for instructions on exporting data to Avro. - -You will need to export one file per table, with the following requirements: - -- Files must be self-contained [object container file (OFC)](#import-an-object-container-file) or a [binary or JSON file](#import-binary-or-json-records) containing one Avro record per line. -- Files must be UTF-8 encoded. - -### Data type mapping - -Avro data types will be flexibly mapped to the target schema; that is, Avro and CockroachDB SQL types do not need to match exactly. By default, CockroachDB ignores unknown Avro fields and sets any columns to `NULL` if they were not set in the Avro record. - -Use the table below for data type mappings: - - Avro Data Type | CockroachDB Data Type -----------------+------------------------------------------------ -`BOOL` | [`BOOL`](bool.html), [`STRING`](string.html) -`INT` | [`INT`](int.html), [`STRING`](string.html) -`FLOAT` | [`FLOAT`](float.html), [`STRING`](string.html) -`STRING` | [`STRING`](string.html) -`BYTES` | [`BYTES`](bytes.html), [`STRING`](string.html) -`ARRAY` | [`ARRAY`](array.html), [`STRING`](string.html) -`UUID` | [`STRING`](string.html) -`DATE` | [`STRING`](string.html) -`TIME` | [`STRING`](string.html) -`INTERVAL` | [`STRING`](string.html) -`TIMESTAMP` | [`STRING`](string.html) -`JSON` | [`STRING`](string.html) -`BIT` | [`STRING`](string.html) -`DECIMAL` | [`STRING`](string.html) - -{{site.data.alerts.callout_info}} -CockroachDB will attempt to convert the Avro data type to the CockroachDB data type; otherwise, it will report an error. -{{site.data.alerts.end}} - -## Step 2. Host the files where the cluster can access them - -Each node in the CockroachDB cluster needs to have access to the files being imported. There are several ways for the cluster to access the data; for more information on the types of storage [`IMPORT`][import] can pull from, see the following: - -- [Use Cloud Storage for Bulk Operations](use-cloud-storage-for-bulk-operations.html) -- [Use a Local File Server for Bulk Operations](use-a-local-file-server-for-bulk-operations.html) - -{{site.data.alerts.callout_success}} -We strongly recommend using cloud storage such as Amazon S3 or Google Cloud to host the data files you want to import. -{{site.data.alerts.end}} - -## Step 3. Import the Avro - -To import Avro data: - -- [Import an object container file](#import-an-object-container-file). -- [Import binary or JSON records](#import-binary-or-json-records). - -### Import an object container file - -An [object container file (OCF)](https://avro.apache.org/docs/current/spec.html#Object+Container+Files) is a self-contained Avro file and includes both the table schema and records. For Avro OCF, there are two available import [options][option]: - -- `strict_validation`, which rejects Avro records that do not have a [one-to-one data type mapping][datatypes] to the target schema. By default, CockroachDB ignores unknown Avro fields and sets missing SQL fields to `NULL`. -- {% include_cached new-in.html version="v21.1" %} `row_limit`, which is useful for finding errors quickly before executing a more time- and resource-consuming import. Setting `row_limit = 'n'` will import the first *n* rows of a table. - -{{site.data.alerts.callout_info}} -The following example uses [sample data from Teradata](https://github.com/Teradata/kylo/tree/master/samples/sample-data/avro). -{{site.data.alerts.end}} - -For example, to import the data from `userdata1.avro` into an `employees` table, issue the following [`IMPORT`][import] statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> IMPORT TABLE employees ( - registration_dttm STRING, - id INT, - first_name STRING, - last_name STRING, - email STRING, - gender STRING, - ip_address STRING, - cc INT, - country STRING, - birthdate STRING, - salary FLOAT, - title STRING, - comments STRING - ) - AVRO DATA ( - 's3://[bucket-placeholder]/userdata1.avro?AWS_ACCESS_KEY_ID=[placeholder]&AWS_SECRET_ACCESS_KEY=[placeholder]' - ); -~~~ - -~~~ - job_id | status | fraction_completed | rows | index_entries | bytes ----------------------+-----------+--------------------+------+---------------+--------- - 535041064396062721 | succeeded | 1 | 1000 | 0 | 162825 -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM employees LIMIT 5; -~~~ - -~~~ - registration_dttm | id | first_name | last_name | email | gender | ip_address | cc | country | birthdate | salary | title | comments ------------------------+----+------------+-----------+--------------------------+--------+----------------+------------------+------------------------+------------+-----------+--------------------------+----------- - 2016-02-03T07:55:29Z | 1 | Amanda | Jordan | ajordan0@com.com | Female | 1.197.201.2 | 6759521864920116 | Indonesia | 3/8/1971 | 49756.53 | Internal Auditor | 1E+02 - 2016-02-03T17:04:03Z | 2 | Albert | Freeman | afreeman1@is.gd | Male | 218.111.175.34 | NULL | Canada | 1/16/1968 | 150280.17 | Accountant IV | - 2016-02-03T01:09:31Z | 3 | Evelyn | Morgan | emorgan2@altervista.org | Female | 7.161.136.94 | 6767119071901597 | Russia | 2/1/1960 | 144972.51 | Structural Engineer | - 2016-02-03T12:36:21Z | 4 | Denise | Riley | driley3@gmpg.org | Female | 140.35.109.83 | 3576031598965625 | China | 4/8/1997 | 90263.05 | Senior Cost Accountant | - 2016-02-03T05:05:31Z | 5 | Carlos | Burns | cburns4@miitbeian.gov.cn | | 169.113.235.40 | 5602256255204850 | South Africa | | NULL | | -(5 rows) -~~~ - -Repeat this process for each OCF you want to import. - -{% include {{ page.version.version }}/sql/use-import-into.md %} - -{{site.data.alerts.callout_info}} -You will need to run [`ALTER TABLE ... ADD CONSTRAINT`](add-constraint.html) to add any foreign key relationships. -{{site.data.alerts.end}} - -### Import binary or JSON records - -In addition to importing [Avro OCF](#import-an-object-container-file), you can also import a binary or JSON file containing Avro records: - -- To import a binary file, use the `data_as_binary_records` [option][option]. -- To import a JSON file, use the `data_as_json_records` [option][option]. - -The schema is not included in these types of files, so you need to specify the schema. You need to either: - -- Specify the schema inline with the `schema` [option][option]. -- Specify the schema by pointing to the file with the `schema_uri` [option][option]. - -There are additional import [options][option] you can use when importing binary and JSON files: - -- `strict_validation`, which rejects Avro records that do not have a [one-to-one data type mapping][datatypes] to the target schema. By default, CockroachDB ignores unknown Avro fields and sets missing SQL fields to `NULL`. -- `records_terminated_by`, which specifies the unicode character used to indicate new lines in the input binary or JSON file (default: `\n`). -- {% include_cached new-in.html version="v21.1" %} `row_limit`, which is useful for finding errors quickly before executing a more time- and resource-consuming import. Setting `row_limit = 'n'` will import the first *n* rows of a table. - -{{site.data.alerts.callout_info}} -The following example uses sample data generated by [Avro tools](https://github.com/cockroachdb/cockroach/tree/master/pkg/ccl/importccl/testdata/avro). -{{site.data.alerts.end}} - -For example, to import the data from `simple-schema.json` into an `simple` table, issue the following [`IMPORT`][import] statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> IMPORT TABLE simple - CREATE USING - 's3://[bucket-placeholder]/simple-schema.sql?AWS_ACCESS_KEY_ID=[placeholder]&AWS_SECRET_ACCESS_KEY=[placeholder]' - AVRO DATA ( - 's3://[bucket-placeholder]/simple-sorted.json?AWS_ACCESS_KEY_ID=[placeholder]&AWS_SECRET_ACCESS_KEY=[placeholder]' - ) - WITH - data_as_json_records, - schema = '{ "type": "record", - "name": "simple", - "fields": - [ - { "name": "i", "type": "int" }, - { "name": "s", "type": "string" }, - { "name": "b", "type": ["null", "bytes"] } - ] -}'; -~~~ - -~~~ - job_id | status | fraction_completed | rows | index_entries | bytes ----------------------+-----------+--------------------+------+---------------+-------- - 535294613223669761 | succeeded | 1 | 1000 | 0 | 50372 -(1 row) -~~~ - -~~~ -> SELECT * FROM simple LIMIT 5; -~~~ - -~~~ - i | s | b ---------------+-----------------------------------+---------------------------------------------... - -2135825688 | dcpamywjlvtohbbtbtpypubccu | \303\204\303\264\303\216\027\303\221\017\303... - -2135463332 | rmspluxnumigrpbrkfmuktphnmfskt | \303\232\017i>{b.~\302\277\177A\302\264\303\... - -2132354298 | mebfxrhurtngsqvlyjechuglymuxfjpvv | \303\2541E\302\277\302\2714\302\257\303\201\... - -2131856455 | hfrgfefflpopvtemrspaixitncghwqfrr | NULL - -2116408431 | thuosfwm | \016s\026\303\264\303\247\302\201\302\264o\3... -(5 rows) -~~~ - -Repeat this process for each binary or JSON file you want to import. - -{% include {{ page.version.version }}/sql/use-import-into.md %} - -{{site.data.alerts.callout_info}} -You will need to run [`ALTER TABLE ... ADD CONSTRAINT`](add-constraint.html) to add any foreign key relationships. -{{site.data.alerts.end}} - -## See also - -- [`IMPORT`][import] -- [Import Performance Best Practices](import-performance-best-practices.html) -- [Migrate from CSV][csv] -- [Migrate from MySQL][mysql] -- [Migrate from Postgres][postgres] -- [SQL Dump (Export)](cockroach-dump.html) -- [Back Up and Restore Data](take-full-and-incremental-backups.html) -- [Use the Built-in SQL Client](cockroach-sql.html) -- [Other Cockroach Commands](cockroach-commands.html) - - - -[csv]: migrate-from-csv.html -[postgres]: migrate-from-postgres.html -[mysql]: migrate-from-mysql.html -[import]: import.html -[option]: import.html#import-options -[datatypes]: migrate-from-avro.html#data-type-mapping diff --git a/src/current/v21.1/migrate-from-csv.md b/src/current/v21.1/migrate-from-csv.md deleted file mode 100644 index e0906307074..00000000000 --- a/src/current/v21.1/migrate-from-csv.md +++ /dev/null @@ -1,229 +0,0 @@ ---- -title: Migrate from CSV -summary: Learn how to migrate data from CSV files into a CockroachDB cluster. -toc: true ---- - -This page has instructions for migrating data from CSV files into CockroachDB using [`IMPORT`][import]. - -The examples below use the [employees data set](https://github.com/datacharmer/test_db) that is also used in the [MySQL docs](https://dev.mysql.com/doc/employee/en/). - -The examples below pull real data from [Amazon S3](https://aws.amazon.com/s3/). They use the [employees data set](https://github.com/datacharmer/test_db) that is also used in the [MySQL docs](https://dev.mysql.com/doc/employee/en/), dumped as a set of CSV files. - -{% include {{ page.version.version }}/misc/import-perf.md %} - -## Step 1. Export data to CSV - -Please refer to the documentation of your database for instructions on exporting data to CSV. - -You will need to export one CSV file per table, with the following requirements: - -- Files must be in [valid CSV format](https://tools.ietf.org/html/rfc4180), with the caveat that the delimiter must be a single character. To use a character other than comma (such as a tab), set a custom delimiter using the [`delimiter` option](import.html#delimiter). -- Files must be UTF-8 encoded. -- If one of the following characters appears in a field, the field must be enclosed by double quotes: - - delimiter (`,` by default) - - double quote (`"`) - - newline (`\n`) - - carriage return (`\r`) -- If double quotes are used to enclose fields, then a double quote appearing inside a field must be escaped by preceding it with another double quote. For example: `"aaa","b""bb","ccc"` -- If a column is of type [`BYTES`](bytes.html), it can either be a valid UTF-8 string or a [hex-encoded byte literal](sql-constants.html#hexadecimal-encoded-byte-array-literals) beginning with `\x`. For example, a field whose value should be the bytes `1`, `2` would be written as `\x0102`. - -## Step 2. Host the files where the cluster can access them - -Each node in the CockroachDB cluster needs to have access to the files being imported. There are several ways for the cluster to access the data; for more information on the types of storage [`IMPORT`][import] can pull from, see the following: - -- [Use Cloud Storage for Bulk Operations](use-cloud-storage-for-bulk-operations.html) -- [Use Userfile for Bulk Operations](use-userfile-for-bulk-operations.html) -- [Use a Local File Server for Bulk Operations](use-a-local-file-server-for-bulk-operations.html) - -{{site.data.alerts.callout_success}} -We strongly recommend using cloud storage such as Amazon S3 or Google Cloud to host the data files you want to import. -{{site.data.alerts.end}} - -## Step 3. Import the CSV - -You will need to write an [`IMPORT TABLE`][import] statement that matches the schema of the table data you're importing. - -For example, to import the data from `employees.csv` into an `employees` table, issue the following statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> IMPORT TABLE employees ( - emp_no INT PRIMARY KEY, - birth_date DATE NOT NULL, - first_name STRING NOT NULL, - last_name STRING NOT NULL, - gender STRING NOT NULL, - hire_date DATE NOT NULL - ) CSV DATA ('https://s3-us-west-1.amazonaws.com/cockroachdb-movr/datasets/employees-db/csv/employees.csv.gz'); -~~~ - -~~~ - job_id | status | fraction_completed | rows | index_entries | system_records | bytes ---------------------+-----------+--------------------+--------+---------------+----------------+---------- - 381866942129111041 | succeeded | 1 | 300024 | 0 | 0 | 13258389 -(1 row) -~~~ - -Repeat the above for each CSV file you want to import. - -{% include {{ page.version.version }}/sql/use-import-into.md %} - -{% include {{ page.version.version }}/misc/csv-import-callout.md %} - -{{site.data.alerts.callout_info}} -You will need to run [`ALTER TABLE ... ADD CONSTRAINT`](add-constraint.html) to add any foreign key relationships. -{{site.data.alerts.end}} - -## Configuration Options - -The following options are available to [`IMPORT ... CSV`][import]: - -+ [Column delimiter](#column-delimiter) -+ [Comment syntax](#comment-syntax) -+ [Skip header rows](#skip-header-rows) -+ {% include_cached new-in.html version="v21.1" %} [Row limit](#row-limit) -+ [Null strings](#null-strings) -+ [File compression](#file-compression) - -### Column delimiter - -The `delimiter` option is used to set the Unicode character that marks where each column ends. **Default: `,`**. - -Example usage: - -{% include_cached copy-clipboard.html %} -~~~ sql -> IMPORT TABLE employees ( - emp_no INT PRIMARY KEY, - birth_date DATE NOT NULL, - first_name STRING NOT NULL, - last_name STRING NOT NULL, - gender STRING NOT NULL, - hire_date DATE NOT NULL - ) - CSV DATA ('s3://acme-co/employees.csv?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456') - WITH delimiter = e'\t'; -~~~ - -### Comment syntax - -The `comment` option determines which Unicode character marks the rows in the data to be skipped. - -Example usage: - -{% include_cached copy-clipboard.html %} -~~~ sql -> IMPORT TABLE employees ( - emp_no INT PRIMARY KEY, - birth_date DATE NOT NULL, - first_name STRING NOT NULL, - last_name STRING NOT NULL, - gender STRING NOT NULL, - hire_date DATE NOT NULL - ) - CSV DATA ('s3://acme-co/employees.csv?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456') - WITH comment = '#'; -~~~ - -### Skip header rows - -The `skip` option determines the number of header rows to skip when importing a file. - -Example usage: - -{% include_cached copy-clipboard.html %} -~~~ sql -> IMPORT TABLE employees ( - emp_no INT PRIMARY KEY, - birth_date DATE NOT NULL, - first_name STRING NOT NULL, - last_name STRING NOT NULL, - gender STRING NOT NULL, - hire_date DATE NOT NULL - ) - CSV DATA ('s3://acme-co/employees.csv?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456') - WITH skip = '2'; -~~~ - -### Row limit - -{% include_cached new-in.html version="v21.1" %} The `row_limit` option determines the number of rows to import from a table. It is useful for finding errors quickly before executing a more time- and resource-consuming import. - -Example usage: - -{% include_cached copy-clipboard.html %} -~~~ sql -> IMPORT TABLE employees ( - emp_no INT PRIMARY KEY, - birth_date DATE NOT NULL, - first_name STRING NOT NULL, - last_name STRING NOT NULL, - gender STRING NOT NULL, - hire_date DATE NOT NULL - ) - CSV DATA ('s3://acme-co/employees.csv?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456') - WITH row_limit = '10'; -~~~ - -### Null strings - -The `nullif` option defines which string should be converted to `NULL`. - -Example usage: - -{% include_cached copy-clipboard.html %} -~~~ sql -> IMPORT TABLE employees ( - emp_no INT PRIMARY KEY, - birth_date DATE NOT NULL, - first_name STRING NOT NULL, - last_name STRING NOT NULL, - gender STRING NOT NULL, - hire_date DATE NOT NULL - ) - CSV DATA ('s3://acme-co/employees.csv?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456') - WITH nullif = ''; -~~~ - -### File compression - -The `compress` option defines which decompression codec should be used on the CSV file to be imported. Options include: - -+ `gzip`: Uses the [gzip](https://en.wikipedia.org/wiki/Gzip) algorithm to decompress the file. -+ `bzip`: Uses the [bzip](https://en.wikipedia.org/wiki/Bzip2) algorithm to decompress the file. -+ `none`: Disables decompression. -+ `auto`: **Default**. Guesses based on file extension ('none' for `.csv`, 'gzip' for `.gz`, 'bzip' for `.bz` and `.bz2`). - -Example usage: - -{% include_cached copy-clipboard.html %} -~~~ sql -> IMPORT TABLE employees ( - emp_no INT PRIMARY KEY, - birth_date DATE NOT NULL, - first_name STRING NOT NULL, - last_name STRING NOT NULL, - gender STRING NOT NULL, - hire_date DATE NOT NULL - ) - CSV DATA ('s3://acme-co/employees.csv.gz?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456') - WITH compress = 'gzip'; -~~~ - -## See also - -- [`IMPORT`][import] -- [Import Performance Best Practices](import-performance-best-practices.html) -- [Migrate from MySQL][mysql] -- [Migrate from Postgres][postgres] -- [SQL Dump (Export)](cockroach-dump.html) -- [Back Up and Restore Data](take-full-and-incremental-backups.html) -- [Use the Built-in SQL Client](cockroach-sql.html) -- [Other Cockroach Commands](cockroach-commands.html) - - - -[postgres]: migrate-from-postgres.html -[mysql]: migrate-from-mysql.html -[import]: import.html diff --git a/src/current/v21.1/migrate-from-geojson.md b/src/current/v21.1/migrate-from-geojson.md deleted file mode 100644 index bfc9cdddc20..00000000000 --- a/src/current/v21.1/migrate-from-geojson.md +++ /dev/null @@ -1,108 +0,0 @@ ---- -title: Migrate from GeoJSON -summary: Learn how to migrate data from GeoJSON into a CockroachDB cluster. -toc: true ---- - - CockroachDB supports efficiently storing and querying [spatial data](spatial-data.html). - -This page has instructions for migrating data from the [GeoJSON](https://en.wikipedia.org/wiki/GeoJSON) format into CockroachDB using [`ogr2ogr`](https://gdal.org/programs/ogr2ogr.html) and [`IMPORT`][import]. - -In the example below we will import a data set with [the locations of underground storage tanks in the state of Vermont (USA)](https://anrweb.vt.gov/DEC/ERT/UST.aspx?ustfacilityid=96). - -## Before You Begin - -To follow along with the example below, you will need the following prerequisites: - -- CockroachDB [installed](install-cockroachdb.html) and [running](start-a-local-cluster.html) -- [`ogr2ogr`](https://gdal.org/programs/ogr2ogr.html) -- [Python 3](https://www.python.org) - -{% include {{page.version.version}}/spatial/ogr2ogr-supported-version.md %} - -## Step 1. Download the GeoJSON data - -First, download the storage tank GeoJSON data: - -{% include_cached copy-clipboard.html %} -~~~ shell -wget -O tanks.geojson https://geodata.vermont.gov/datasets/986155613c5743239e7b1980b45bbf36_162.geojson -~~~ - -## Step 2. Convert the GeoJSON data to SQL - -Next, convert the GeoJSON data to SQL using the following `ogr2ogr` command: - -{% include_cached copy-clipboard.html %} -~~~ shell -ogr2ogr -f PGDUMP tanks.sql -lco LAUNDER=NO -lco DROP_TABLE=OFF tanks.geojson -~~~ - -{% include {{page.version.version}}/spatial/ogr2ogr-supported-version.md %} - -## Step 3. Host the files where the cluster can access them - -Each node in the CockroachDB cluster needs to have access to the files being imported. There are several ways for the cluster to access the data; for a complete list of the types of storage [`IMPORT`][import] can pull from, see [import file locations](import.html#import-file-location). - -For local testing, you can [start a local file server](use-a-local-file-server-for-bulk-operations.html). The following command will start a local file server listening on port 3000: - -{% include_cached copy-clipboard.html %} -~~~ shell -python3 -m http.server 3000 -~~~ - -## Step 4. Prepare the database - -Next, create a database to hold the storage tank data: - -{% include_cached copy-clipboard.html %} -~~~ shell -cockroach sql --insecure -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE DATABASE IF NOT EXISTS tanks; -USE tanks; -~~~ - -## Step 5. Import the SQL - -Since the file is being served from a local server and is formatted as Postgres-compatible SQL, we can import the data using the following [`IMPORT PGDUMP`](import.html#import-a-postgresql-database-dump) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -IMPORT PGDUMP ('http://localhost:3000/tanks.sql'); -~~~ - -~~~ - job_id | status | fraction_completed | rows | index_entries | bytes ----------------------+-----------+--------------------+------+---------------+--------- - 588555793549328385 | succeeded | 1 | 2709 | 2709 | 822504 -(1 row) -~~~ - -## See also - -- [`IMPORT`][import] -- [Export Spatial Data](export-spatial-data.html) -- [Spatial tutorial](spatial-tutorial.html) -- [Working with Spatial Data](spatial-data.html) -- [Migrate from OpenStreetMap](migrate-from-openstreetmap.html) -- [Migrate from Shapefiles](migrate-from-shapefiles.html) -- [Migrate from GeoPackage](migrate-from-geopackage.html) -- [Spatial indexes](spatial-indexes.html) -- [Migration Overview](migration-overview.html) -- [Migrate from MySQL][mysql] -- [Migrate from Postgres][postgres] -- [SQL Dump (Export)](cockroach-dump.html) -- [Back Up and Restore Data](take-full-and-incremental-backups.html) -- [Use the Built-in SQL Client](cockroach-sql.html) -- [Other Cockroach Commands](cockroach-commands.html) -- [Using GeoServer with CockroachDB](geoserver.html) - - - -[postgres]: migrate-from-postgres.html -[mysql]: migrate-from-mysql.html -[import]: import.html diff --git a/src/current/v21.1/migrate-from-geopackage.md b/src/current/v21.1/migrate-from-geopackage.md deleted file mode 100644 index f69e1a42939..00000000000 --- a/src/current/v21.1/migrate-from-geopackage.md +++ /dev/null @@ -1,120 +0,0 @@ ---- -title: Migrate from GeoPackages -summary: Learn how to migrate data from GeoPackages into a CockroachDB cluster. -toc: true ---- - - CockroachDB supports efficiently storing and querying [spatial data](spatial-data.html). - -This page has instructions for migrating data from the [GeoPackage](https://www.geopackage.org/) format into CockroachDB using [`ogr2ogr`](https://gdal.org/programs/ogr2ogr.html) and [`IMPORT`][import]. - -In the example below we will import a data set with the locations of natural springs in the state of Minnesota (USA) that is made available via gisdata.mn.gov. - -## Before You Begin - -To follow along with the example below, you will need the following prerequisites: - -- CockroachDB [installed](install-cockroachdb.html) and [running](start-a-local-cluster.html) -- [`ogr2ogr`](https://gdal.org/programs/ogr2ogr.html) -- [Python 3](https://www.python.org) - -{% include {{page.version.version}}/spatial/ogr2ogr-supported-version.md %} - -## Step 1. Download the GeoPackage data - -First, download the zip file containing the spring location data: - -{% include_cached copy-clipboard.html %} -~~~ shell -wget https://resources.gisdata.mn.gov/pub/gdrs/data/pub/us_mn_state_dnr/env_mn_springs_inventory/gpkg_env_mn_springs_inventory.zip -~~~ - -Next, unzip the file: - -{% include_cached copy-clipboard.html %} -~~~ shell -unzip gpkg_env_mn_springs_inventory.zip -~~~ - -## Step 2. Convert the GeoPackage data to SQL - -To load the GeoPackage into CockroachDB, we must first convert it to SQL using the `ogr2ogr` tool. - -{% include_cached copy-clipboard.html %} -~~~ shell -ogr2ogr -f PGDUMP springs.sql -lco LAUNDER=NO -lco DROP_TABLE=OFF env_mn_springs_inventory.gpkg -~~~ - -This particular data set emits a warning due to some date formatting. - -~~~ -Warning 1: Non-conformant content for record 1 in column field_ch_1, 2017/05/04, successfully parsed -~~~ - -{% include {{page.version.version}}/spatial/ogr2ogr-supported-version.md %} - -## Step 3. Host the files where the cluster can access them - -Each node in the CockroachDB cluster needs to have access to the files being imported. There are several ways for the cluster to access the data; for a complete list of the types of storage [`IMPORT`][import] can pull from, see [import file locations](import.html#import-file-location). - -For local testing, you can [start a local file server](use-a-local-file-server-for-bulk-operations.html). The following command will start a local file server listening on port 3000: - -{% include_cached copy-clipboard.html %} -~~~ shell -python3 -m http.server 3000 -~~~ - -## Step 4. Prepare the database - -Next, create a database to hold the natural spring location data: - -{% include_cached copy-clipboard.html %} -~~~ shell -cockroach sql --insecure -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE DATABASE springs; -USE springs; -~~~ - -## Step 5. Import the SQL - -Since the file is being served from a local server and is formatted as Postgres-compatible SQL, we can import the data using the following [`IMPORT PGDUMP`](import.html#import-a-postgresql-database-dump) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -IMPORT PGDUMP ('http://localhost:3000/springs.sql'); -~~~ - -~~~ - job_id | status | fraction_completed | rows | index_entries | bytes ----------------------+-----------+--------------------+------+---------------+---------- - 589053379352526849 | succeeded | 1 | 5124 | 5124 | 2449139 -(1 row) -~~~ - -## See also - -- [`IMPORT`][import] -- [Export Spatial Data](export-spatial-data.html) -- [Working with Spatial Data](spatial-data.html) -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial indexes](spatial-indexes.html) -- [Migrate from OpenStreetMap](migrate-from-openstreetmap.html) -- [Migrate from Shapefiles](migrate-from-shapefiles.html) -- [Migration Overview](migration-overview.html) -- [Migrate from MySQL][mysql] -- [Migrate from Postgres][postgres] -- [SQL Dump (Export)](cockroach-dump.html) -- [Back Up and Restore Data](take-full-and-incremental-backups.html) -- [Use the Built-in SQL Client](cockroach-sql.html) -- [Other Cockroach Commands](cockroach-commands.html) -- [Using GeoServer with CockroachDB](geoserver.html) - - - -[postgres]: migrate-from-postgres.html -[mysql]: migrate-from-mysql.html -[import]: import.html diff --git a/src/current/v21.1/migrate-from-mysql.md b/src/current/v21.1/migrate-from-mysql.md deleted file mode 100644 index 90ae4073079..00000000000 --- a/src/current/v21.1/migrate-from-mysql.md +++ /dev/null @@ -1,230 +0,0 @@ ---- -title: Migrate from MySQL -summary: Learn how to migrate data from MySQL into a CockroachDB cluster. -toc: true ---- - -This page has instructions for migrating data from MySQL to CockroachDB using [`IMPORT`](import.html)'s support for reading [`mysqldump`][mysqldump] files. - -The examples below use the [employees data set](https://github.com/datacharmer/test_db) that is also used in the [MySQL docs](https://dev.mysql.com/doc/employee/en/). - -{% include {{ page.version.version }}/misc/import-perf.md %} - -## Considerations - -In addition to the general considerations listed in the [Migration Overview](migration-overview.html), there is also the following MySQL-specific information to consider as you prepare your migration. - -### String case sensitivity - -MySQL strings are case-insensitive by default, but strings in CockroachDB are case-sensitive. This means that you may need to edit your MySQL dump file to get the results you expect from CockroachDB. For example, you may have been doing string comparisons in MySQL that will need to be changed to work with CockroachDB. - -For more information about the case sensitivity of strings in MySQL, see [Case Sensitivity in String Searches](https://dev.mysql.com/doc/refman/8.0/en/case-sensitivity.html) from the MySQL documentation. For more information about CockroachDB strings, see [`STRING`](string.html). - -### `FIELD` function - -The MYSQL `FIELD` function is not supported in CockroachDB. Instead, you can use the [`array_position`](functions-and-operators.html#array-functions) function, which returns the index of the first occurrence of element in the array. - -Example usage: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT array_position(ARRAY[4,1,3,2],1); -~~~ - -~~~ - array_position ------------------- - 2 -(1 row) -~~~ - -While MYSQL returns 0 when the element is not found, CockroachDB returns `NULL`. So if you are using the `ORDER BY` clause in a statement with the `array_position` function, the caveat is that sort is applied even when the element is not found. As a workaround, you can use the [`COALESCE`](functions-and-operators.html#conditional-and-function-like-operators) operator. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM table_a ORDER BY COALESCE(array_position(ARRAY[4,1,3,2],5),999); -~~~ - -## Step 1. Dump the MySQL database - -There are several ways to dump data from MySQL to be imported into CockroachDB: - -- [Dump the entire database](#dump-the-entire-database) -- [Dump one table at a time](#dump-one-table-at-a-time) - -### Dump the entire database - -Most users will want to import their entire MySQL database all at once, as shown below in [Import a full database dump](#import-a-full-database-dump). To dump the entire database, run the [`mysqldump`][mysqldump] command shown below: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ mysqldump -uroot employees > /tmp/employees-full.sql -~~~ - -If you only want to import one table from a database dump, see [Import a table from a full database dump](#import-a-table-from-a-full-database-dump) below. - -### Dump one table at a time - -To dump the `employees` table from a MySQL database also named `employees`, run the [`mysqldump`][mysqldump] command shown below. You can import this table using the instructions in [Import a table from a table dump](#import-a-table-from-a-table-dump) below. - -{% include_cached copy-clipboard.html %} -~~~ shell -$ mysqldump -uroot employees employees > employees.sql -~~~ - -## Step 2. Host the files where the cluster can access them - -Each node in the CockroachDB cluster needs to have access to the files being imported. There are several ways for the cluster to access the data; for more information on the types of storage [`IMPORT`][import] can pull from, see the following: - -- [Use Cloud Storage for Bulk Operations](use-cloud-storage-for-bulk-operations.html) -- [Use a Local File Server for Bulk Operations](use-a-local-file-server-for-bulk-operations.html) - -{{site.data.alerts.callout_success}} -We strongly recommend using cloud storage such as Amazon S3 or Google Cloud to host the data files you want to import. -{{site.data.alerts.end}} - -## Step 3. Import the MySQL dump file - -You can choose from several variants of the [`IMPORT`][import] statement, depending on whether you want to import an entire database or just one table: - -- [Import a full database dump](#import-a-full-database-dump) -- [Import a table from a full database dump](#import-a-table-from-a-full-database-dump) -- [Import a table from a table dump](#import-a-table-from-a-table-dump) - -All of the [`IMPORT`][import] statements in this section pull real data from [Amazon S3](https://aws.amazon.com/s3/) and will kick off background import jobs that you can monitor with [`SHOW JOBS`](show-jobs.html). - -{% include {{ page.version.version }}/sql/use-import-into.md %} - -### Import a full database dump - -This example assumes you [dumped the entire database](#dump-the-entire-database). - -The [`IMPORT`][import] statement below reads the data and [DDL](https://en.wikipedia.org/wiki/Data_definition_language) statements (including `CREATE TABLE` and [foreign key constraints](foreign-key.html)) from the full database dump. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE DATABASE IF NOT EXISTS employees; -> USE employees; -> IMPORT MYSQLDUMP 'https://s3-us-west-1.amazonaws.com/cockroachdb-movr/datasets/employees-db/mysqldump/employees-full.sql.gz'; -~~~ - -~~~ - job_id | status | fraction_completed | rows | index_entries | system_records | bytes ---------------------+-----------+--------------------+---------+---------------+----------------+----------- - 382716507639906305 | succeeded | 1 | 3919015 | 331636 | 0 | 110104816 -(1 row) -~~~ - -### Import a table from a full database dump - -This example assumes you [dumped the entire database](#dump-the-entire-database). - -[`IMPORT`][import] can import one table's data from a full database dump. It reads the data and applies any `CREATE TABLE` statements from the dump file. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE DATABASE IF NOT EXISTS employees; -> USE employees; -> IMPORT TABLE employees FROM MYSQLDUMP 'https://s3-us-west-1.amazonaws.com/cockroachdb-movr/datasets/employees-db/mysqldump/employees.sql.gz'; -~~~ - -~~~ - job_id | status | fraction_completed | rows | index_entries | system_records | bytes ---------------------+-----------+--------------------+--------+---------------+----------------+---------- - 383839294913871873 | succeeded | 1 | 300024 | 0 | 0 | 11534293 -(1 row) -~~~ - -### Import a table from a table dump - -The examples below assume you [dumped one table](#dump-one-table-at-a-time). - -The simplest way to import a table dump is to run [`IMPORT TABLE`][import] as shown below. It reads the table data and any `CREATE TABLE` statements from the dump file. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE DATABASE IF NOT EXISTS employees; -> USE employees; -> IMPORT TABLE employees FROM MYSQLDUMP 'https://s3-us-west-1.amazonaws.com/cockroachdb-movr/datasets/employees-db/mysqldump/employees.sql.gz'; -~~~ - -~~~ - job_id | status | fraction_completed | rows | index_entries | system_records | bytes ---------------------+-----------+--------------------+--------+---------------+----------------+---------- - 383855569817436161 | succeeded | 1 | 300024 | 0 | 0 | 11534293 -(1 row) -~~~ - -If you need to specify the table's columns for some reason, you can use an [`IMPORT TABLE`][import] statement like the one below, which will import data but ignore any `CREATE TABLE` statements in the dump file, instead relying on the columns you specify. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE DATABASE IF NOT EXISTS employees; -> USE employees; -> IMPORT TABLE employees ( - emp_no INT PRIMARY KEY, - birth_date DATE NOT NULL, - first_name STRING NOT NULL, - last_name STRING NOT NULL, - gender STRING NOT NULL, - hire_date DATE NOT NULL - ) - MYSQLDUMP DATA ('https://s3-us-west-1.amazonaws.com/cockroachdb-movr/datasets/employees-db/mysqldump/employees.sql.gz'); -~~~ - -## Configuration Options - -The following options are available to `IMPORT ... MYSQLDUMP`: - -+ {% include_cached new-in.html version="v21.1" %} [Row limit](#row-limit) -+ [Skip foreign keys](#skip-foreign-keys) - -### Row limit - -{% include_cached new-in.html version="v21.1" %} The `row_limit` option determines the number of rows to import. If you are importing one table, setting `row_limit = 'n'` will import the first *n* rows of the table. If you are importing an entire database, this option will import the first *n* rows from each table in the dump file. It is useful for finding errors quickly before executing a more time- and resource-consuming import. - -Example usage: - -{% include_cached copy-clipboard.html %} -~~~ sql -> IMPORT MYSQLDUMP 's3://your-external-storage/employees.sql?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456' WITH row_limit = '10'; -~~~ - -### Skip foreign keys - -By default, [`IMPORT ... MYSQLDUMP`][import] supports foreign keys. **Default: false**. Add the `skip_foreign_keys` option to speed up data import by ignoring foreign key constraints in the dump file's DDL. It will also enable you to import individual tables that would otherwise fail due to dependencies on other tables. - -{{site.data.alerts.callout_info}} -The most common dependency issues are caused by unsatisfied foreign key relationships. You can avoid these issues by adding the `skip_foreign_keys` option to your `IMPORT` statement as needed. For more information, see the list of [import options](import.html#import-options). - -For example, if you get the error message `pq: there is no unique constraint matching given keys for referenced table tablename`, use `IMPORT ... WITH skip_foreign_keys`. -{{site.data.alerts.end}} - -Example usage: - -{% include_cached copy-clipboard.html %} -~~~ sql -> IMPORT MYSQLDUMP 's3://your-external-storage/employees.sql?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456' WITH skip_foreign_keys; -~~~ - -[Foreign key constraints](foreign-key.html) can be added by using [`ALTER TABLE ... ADD CONSTRAINT`](add-constraint.html) commands after importing the data. - -## See also - -- [`IMPORT`](import.html) -- [Import Performance Best Practices](import-performance-best-practices.html) -- [Migrate from CSV][csv] -- [Migrate from Postgres][postgres] -- [Can a Postgres or MySQL application be migrated to CockroachDB?](frequently-asked-questions.html#can-a-postgresql-or-mysql-application-be-migrated-to-cockroachdb) -- [SQL Dump (Export)](cockroach-dump.html) -- [Back up Data](take-full-and-incremental-backups.html) -- [Restore Data](take-full-and-incremental-backups.html) -- [Use the Built-in SQL Client](cockroach-sql.html) -- [Other Cockroach Commands](cockroach-commands.html) - - - -[postgres]: migrate-from-postgres.html -[csv]: migrate-from-csv.html -[import]: import.html -[mysqldump]: https://dev.mysql.com/doc/refman/8.0/en/mysqldump-sql-format.html diff --git a/src/current/v21.1/migrate-from-openstreetmap.md b/src/current/v21.1/migrate-from-openstreetmap.md deleted file mode 100644 index b4fe225f3a2..00000000000 --- a/src/current/v21.1/migrate-from-openstreetmap.md +++ /dev/null @@ -1,146 +0,0 @@ ---- -title: Migrate from OpenStreetMap -summary: Learn how to migrate data from the OpenStreetMap pbf format into a CockroachDB cluster. -toc: true ---- - - CockroachDB supports efficiently storing and querying [spatial data](spatial-data.html). - -This page has instructions for migrating data from [OpenStreetMap](https://www.openstreetmap.org) `.pbf` data files into CockroachDB using [`osm2pgsql`](https://github.com/openstreetmap/osm2pgsql/) and [`IMPORT`][import]. - -In the example below we will import the [OSM data for Australia](https://download.geofabrik.de/australia-oceania/australia.html) that is available from [GeoFabrik GmbH](http://www.geofabrik.de/data/shapefiles.html). - -## Before You Begin - -To follow along with the example below, you will need the following prerequisites: - -- CockroachDB [installed](install-cockroachdb.html) and [running](start-a-local-cluster.html) -- [`osm2pgsql`](https://github.com/openstreetmap/osm2pgsql/) - -## Step 1. Download the OpenStreetMap data - -First, download the OSM data: - -{% include_cached copy-clipboard.html %} -~~~ shell -wget https://download.geofabrik.de/australia-oceania/australia-latest.osm.pbf -~~~ - -## Step 2. Prepare the database - -{% include_cached copy-clipboard.html %} -~~~ shell -cockroach sql --insecure -~~~ - -Next, create a database to hold the Australia map data: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE DATABASE IF NOT EXISTS australia; -USE australia; -~~~ - -## Step 3. Import the OpenStreetMap data - -Run the `osm2pgsql` command shown below to convert the OSM data and import it into the `australia` database. The arguments to `osm2pgsql` shown below assume a [locally running insecure cluster](start-a-local-cluster.html) and may need to be changed depending on your system. You may also need to tweak the cache setting (`-C`) depending on your system's hardware. - -{% include_cached copy-clipboard.html %} -~~~ shell -osm2pgsql -C 2048 -d australia -U root -H localhost -P 26257 australia-latest.osm.pbf -~~~ - -This will take a few (30+) minutes to run on a laptop, and there will be a lot of output. A successful run will look something like the following: - -~~~ -osm2pgsql version 1.3.0 - -Allocating memory for dense node cache -Allocating dense node cache in one big chunk -Allocating memory for sparse node cache -Sharing dense sparse -Node-cache: cache=2048MB, maxblocks=32768*65536, allocation method=3 -Using built-in tag processing pipeline -Using projection SRS 3857 (Spherical Mercator) -WARNING: setting session var "synchronous_commit" is a no-op -WARNING: setting session var "synchronous_commit" is a no-op -Setting up table: planet_osm_point -NOTICE: UNLOGGED TABLE will behave as a regular table in CockroachDB -NOTICE: storage parameter "autovacuum_enabled = 'off'" is ignored -WARNING: setting session var "synchronous_commit" is a no-op -Setting up table: planet_osm_line -NOTICE: UNLOGGED TABLE will behave as a regular table in CockroachDB -NOTICE: storage parameter "autovacuum_enabled = 'off'" is ignored -WARNING: setting session var "synchronous_commit" is a no-op -Setting up table: planet_osm_polygon -NOTICE: UNLOGGED TABLE will behave as a regular table in CockroachDB -NOTICE: storage parameter "autovacuum_enabled = 'off'" is ignored -WARNING: setting session var "synchronous_commit" is a no-op -Setting up table: planet_osm_roads -NOTICE: UNLOGGED TABLE will behave as a regular table in CockroachDB -NOTICE: storage parameter "autovacuum_enabled = 'off'" is ignored - -Reading in file: australia-latest.osm.pbf -Using PBF parser. -Processing: Node(66994k 411.0k/s) Way(4640k 7.13k/s) Relation(124777 1313.4/s) parse time: 909s -Node stats: total(66994811), max(7888181047) in 163s -Way stats: total(4640490), max(845495883) in 651s -Relation stats: total(124777), max(11596803) in 95s -Sorting data and creating indexes for planet_osm_point -node cache: stored: 66994811(100.00%), storage efficiency: 50.93% (dense blocks: 800, sparse nodes: 62492547), hit rate: 100.00% -Sorting data and creating indexes for planet_osm_line -Sorting data and creating indexes for planet_osm_polygon -Sorting data and creating indexes for planet_osm_roads -Using native order for clustering -Using native order for clustering -Using native order for clustering -Using native order for clustering -Copying planet_osm_roads to cluster by geometry finished -Creating geometry index on planet_osm_roads -Copying planet_osm_point to cluster by geometry finished -Creating geometry index on planet_osm_point -Creating indexes on planet_osm_roads finished -Copying planet_osm_polygon to cluster by geometry finished -Creating geometry index on planet_osm_polygon -All indexes on planet_osm_roads created in 318s -Completed planet_osm_roads -Copying planet_osm_line to cluster by geometry finished -Creating geometry index on planet_osm_line -Creating indexes on planet_osm_point finished -All indexes on planet_osm_point created in 1084s -Completed planet_osm_point -Creating indexes on planet_osm_polygon finished -All indexes on planet_osm_polygon created in 1897s -Completed planet_osm_polygon -Creating indexes on planet_osm_line finished -All indexes on planet_osm_line created in 1961s -Completed planet_osm_line - -Osm2pgsql took 2879s overall -~~~ - -## See also - -- [`IMPORT`][import] -- [Export Spatial Data](export-spatial-data.html) -- [Working with Spatial Data](spatial-data.html) -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial indexes](spatial-indexes.html) -- [Migrate from GeoPackages](migrate-from-geopackage.html) -- [Migrate from GeoJSON](migrate-from-geojson.html) -- [Migrate from Shapefiles](migrate-from-shapefiles.html) -- [Migration Overview](migration-overview.html) -- [Migrate from MySQL][mysql] -- [Migrate from Postgres][postgres] -- [SQL Dump (Export)](cockroach-dump.html) -- [Back Up and Restore Data](take-full-and-incremental-backups.html) -- [Use the Built-in SQL Client](cockroach-sql.html) -- [Other Cockroach Commands](cockroach-commands.html) -- [Using GeoServer with CockroachDB](geoserver.html) - - - -[postgres]: migrate-from-postgres.html -[mysql]: migrate-from-mysql.html -[import]: import.html - diff --git a/src/current/v21.1/migrate-from-oracle.md b/src/current/v21.1/migrate-from-oracle.md deleted file mode 100644 index f99cde0f988..00000000000 --- a/src/current/v21.1/migrate-from-oracle.md +++ /dev/null @@ -1,392 +0,0 @@ ---- -title: Migrate from Oracle -summary: Learn how to migrate data from Oracle into a CockroachDB cluster. -toc: true ---- - -This page has instructions for migrating data from Oracle into CockroachDB by [importing](import.html) CSV files. Note that `IMPORT` only works for creating new tables. For information on how to add CSV data to existing tables, see [`IMPORT INTO`](import-into.html). - -To illustrate this process, we use the following sample data and tools: - -- [Swingbench OrderEntry data set](http://www.dominicgiles.com/swingbench.html), which is based on the `oe` schema that ships with Oracle Database 11g and Oracle Database 12c. -- [Oracle Data Pump](https://docs.oracle.com/en/database/oracle/oracle-database/12.2/sutil/oracle-data-pump.html), which enables the movement of data and metadata from one database to another, and comes with all Oracle installations. -- [SQL*Plus](https://docs.oracle.com/database/121/SQPUG/ch_three.htm), the interactive and batch query tool that comes with every Oracle Database installation. - -{% include {{ page.version.version }}/misc/import-perf.md %} - -## Step 1. Export the Oracle schema - -Using [Oracle's Data Pump Export utility](https://docs.oracle.com/en/database/oracle/oracle-database/12.2/sutil/oracle-data-pump-export-utility.html), export the schema: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ expdp user/password directory=datapump dumpfile=oracle_example.dmp content=metadata_only logfile=example.log -~~~ - -The schema is stored in an Oracle-specific format (e.g., `oracle_example.dmp`). - -## Step 2. Convert the Oracle schema to SQL - -Using [Oracle's Data Pump Import utility](https://docs.oracle.com/en/database/oracle/oracle-database/12.2/sutil/datapump-import-utility.html), load the exported DMP file to convert it to a SQL file: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ impdp user/password directory=datapump dumpfile=oracle_example.dmp sqlfile=example_sql.sql TRANSFORM=SEGMENT_ATTRIBUTES:N:table PARTITION_OPTIONS=MERGE -~~~ - -This SQL output will be used later, in [Step 7](#step-7-map-oracle-to-cockroachdb-data-types). - -## Step 3. Export table data - -You need to extract each table's data into a data list file (`.lst`). We wrote a simple SQL script (`spool.sql`) to do this: - -~~~ shell -$ cat spool.sql - -SET ECHO OFF -SET TERMOUT OFF -SET FEEDBACK OFF -SET PAGESIZE 0 -SET TRIMSPOOL ON -SET WRAP OFF -set linesize 30000 -SET RECSEP OFF -SET VERIFY OFF -SET ARRAYSIZE 10000 -SET COLSEP '|' - -SPOOL '&1' - -ALTER SESSION SET nls_date_format = 'YYYY-MM-DD HH24:MI:SS'; - # Used to set a properly formatted date for CockroachDB - -SELECT * from &1; - -SPOOL OFF - -SET PAGESIZE 24 -SET FEEDBACK ON -SET TERMOUT ON -~~~ - -{{site.data.alerts.callout_info}} -In the example SQL script, `|` is used as a delimiter. Choose a delimiter that will not also occur in the rows themselves. For more information, see [`IMPORT`](import.html#delimiter). -{{site.data.alerts.end}} - -To extract the data, we ran the script for each table in SQL*Plus: - -{% include_cached copy-clipboard.html %} -~~~ sql -$ sqlplus user/password -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> @spool CUSTOMERS - @spool ADDRESSES - @spool CARD_DETAILS - @spool WAREHOUSES - @spool ORDER_ITEMS - @spool ORDERS - @spool INVENTORIES - @spool PRODUCT_INFORMATION - @spool LOGON - @spool PRODUCT_DESCRIPTIONS - @spool ORDERENTRY_METADATA -~~~ - -A data list file (`.lst`) with leading and trailing spaces is created for each table. - -Exit SQL*Plus: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXIT -~~~ - -## Step 4. Configure and convert the table data to CSV - -Each table's data list file needs to be converted to CSV and formatted for CockroachDB. We wrote a simple Python script (`fix-example.py`) to do this: - -~~~ python -$ cat fix-example.py - -import csv -import string -import sys - -for lstfile in sys.argv[1:]: - filename = lstfile.split(".")[0] - - with open(lstfile) as f: - reader = csv.reader(f, delimiter="|") - with open(filename+".csv", "w") as fo: - writer = csv.writer(fo) - for rec in reader: - writer.writerow(map(string.strip, rec)) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ python3 fix-example.py CUSTOMERS.lst ADDRESSES.lst CARD_DETAILS.lst WAREHOUSES.lst ORDER_ITEMS.lst ORDERS.lst INVENTORIES.lst PRODUCT_INFORMATION.lst LOGON.lst PRODUCT_DESCRIPTIONS.lst ORDERENTRY_METADATA.lst -~~~ - -Format the generated CSV files to meet the CockroachDB's [CSV requirements](#csv-requirements). - -### CSV requirements - -You will need to export one CSV file per table, with the following requirements: - -- Files must be in [valid CSV format](https://tools.ietf.org/html/rfc4180). -- Files must be UTF-8 encoded. -- If one of the following characters appears in a field, the field must be enclosed by double quotes: - - delimiter (`,` by default) - - double quote (`"`) - - newline (`\n`) - - carriage return (`\r`) -- If double quotes are used to enclose fields, then a double quote appearing inside a field must be escaped by preceding it with another double quote. For example: `"aaa","b""bb","ccc"` -- If a column is of type [`BYTES`](bytes.html), it can either be a valid UTF-8 string or a [hex-encoded byte literal](sql-constants.html#hexadecimal-encoded-byte-array-literals) beginning with `\x`. For example, a field whose value should be the bytes `1`, `2` would be written as `\x0102`. - -### CSV configuration options - -The following options are available to [`IMPORT ... CSV`](import.html): - -- [Column delimiter](migrate-from-csv.html#column-delimiter) -- [Comment syntax](migrate-from-csv.html#comment-syntax) -- [Skip header rows](migrate-from-csv.html#skip-header-rows) -- {% include_cached new-in.html version="v21.1" %} [Row limit](migrate-from-csv.html#row-limit) -- [Null strings](migrate-from-csv.html#null-strings) -- [File compression](migrate-from-csv.html#file-compression) - -For usage examples, see [Migrate from CSV - Configuration Options](migrate-from-csv.html#configuration-options). - -## Step 5. Compress the CSV files - -Compress the CSV files for a faster import: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ gzip CUSTOMERS.csv ADDRESSES.csv CARD_DETAILS.csv WAREHOUSES.csv ORDER_ITEMS.csv ORDERS.csv INVENTORIES.csv PRODUCT_INFORMATION.csv LOGON.csv PRODUCT_DESCRIPTIONS.csv ORDERENTRY_METADATA.csv -~~~ - -These compressed CSV files will be used to import your data into CockroachDB. - -## Step 6. Host the files where the cluster can access them - -Each node in the CockroachDB cluster needs to have access to the files being imported. There are several ways for the cluster to access the data; for more information on the types of storage [`IMPORT`](import.html) can pull from, see the following: - -- [Use Cloud Storage for Bulk Operations](use-cloud-storage-for-bulk-operations.html) -- [Use a Local File Server for Bulk Operations](use-a-local-file-server-for-bulk-operations.html) - -{{site.data.alerts.callout_success}} -We strongly recommend using cloud storage such as Amazon S3 or Google Cloud to host the data files you want to import. -{{site.data.alerts.end}} - -## Step 7. Map Oracle to CockroachDB data types - -Using the SQL file created in [Step 2](#step-2-convert-the-oracle-schema-to-sql), write [`IMPORT TABLE`](import.html) statements that match the schemas of the table data you're importing. - -Remove all Oracle-specific attributes, remap all Oracle data types, refactor all [`CREATE TABLE`](create-table.html) statements to include [primary keys](primary-key.html). - -### Data type mapping - -Use the table below for data type mappings: - - Oracle Data Type | CockroachDB Data Type -------------------+----------------------- -`BLOB` | [`BYTES`](bytes.html) [1](#considerations) -`CHAR(n)`, `CHARACTER(n)`
    n < 256 | [`CHAR(n)`, `CHARACTER(n)`](string.html) -`CLOB` | [`STRING`](string.html) [1](#considerations) -`DATE` | [`DATE`](date.html) -`FLOAT(n)` | [`DECIMAL(n)`](decimal.html) -`INTERVAL YEAR(p) TO MONTH `| [`VARCHAR`](string.html), [`INTERVAL`](interval.html) -`INTERVAL DAY(p) TO SECOND(s)` | [`VARCHAR`](string.html), [`INTERVAL`](interval.html) -`JSON` | [`JSON`](jsonb.html) [2](#considerations) -`LONG` | [`STRING`](string.html) -`LONG RAW` | [`BYTES`](bytes.html) -`NCHAR(n)`
    n < 256 | [`CHAR(n)`](string.html ) -`NCHAR(n)`
    n > 255 | [`VARCHAR`, `STRING`](string.html) -`NCLOB` | [`STRING`](string.html) -`NUMBER(p,0)`, `NUMBER(p)`
    1 <= p < 5 | [`INT2`](int.html) [3](#considerations) -`NUMBER(p,0)`, `NUMBER(p)`
    5 <= p < 9 | [`INT4`](int.html) [3](#considerations) -`NUMBER(p,0)`, `NUMBER(p)`
    9 <= p < 19 | [`INT8`](int.html) [3](#considerations) -`NUMBER(p,0)`, `NUMBER(p)`
    19 <= p <= 38 | [`DECIMAL(p)`](decimal.html) -`NUMBER(p,s)`
    s > 0 | [`DECIMAL(p,s)`](decimal.html) -`NUMBER`, `NUMBER(\*)` | [`DECIMAL`](decimal.html) -`NVARCHAR2(n)` | [`VARCHAR(n)`](string.html) -`RAW(n)` | [`BYTES`](bytes.html) -`TIMESTAMP(p)` | [`TIMESTAMP`](timestamp.html) -`TIMESTAMP(p) WITH TIME ZONE` | [`TIMESTAMP WITH TIMEZONE`](timestamp.html) -`VARCHAR(n)`, `VARCHAR2(n)` | [`VARCHAR(n)`](string.html) -`XML` | [`JSON`](jsonb.html) [2](#considerations) - - - -- 1 `BLOBS` and `CLOBS` should be converted to [`BYTES`](bytes.html), or [`STRING`](string.html) where the size is variable, but it's recommended to keep values under 1 MB to ensure performance. Anything above 1 MB would require refactoring into an object store with a pointer embedded in the table in place of the object. -- 2 `JSON` and `XML` types can be converted to [`JSONB`](jsonb.html) using any XML to JSON conversion. `XML` must be converted to `JSONB` before importing into CockroachDB. -- 3 When converting `NUMBER(p,0)`, consider `NUMBER` types with Base-10 limits map to the Base-10 Limits for CockroachDB [`INT`](int.html) types. Optionally, `NUMBERS` can be converted to [`DECIMAL`](decimal.html). - -When moving from Oracle to CockroachDB data types, consider the following: - -- [Schema changes within transactions](known-limitations.html#schema-changes-within-transactions) -- [Schema changes between executions of prepared statements](online-schema-changes.html#no-schema-changes-between-executions-of-prepared-statements) -- If [`JSON`](jsonb.html) columns are used only for payload, consider switching to [`BYTES`](bytes.html). -- Max size of a single column family (512 MiB by default). - -For more information, see [Known Limitations](known-limitations.html), [Online Schema Changes](online-schema-changes.html), and [Transactions](transactions.html). - -### NULLs - -For information on how CockroachDB handles `NULL`s, see [NULL Handling](null-handling.html) and [NOT NULL Constraint](not-null.html). - -### Primary key, constraints, and secondary indexes - -Cockroach distributes a table by the [primary key](primary-key.html) or by a default `ROWID` when a primary key is not provided. This also requires the primary key creation to be part of the table creation. Using the above [data type mapping](#data-type-mapping), refactor each table DDL to include the [primary key](primary-key.html), [constraints](constraints.html), and [secondary indexes](indexes.html). - -For more information and examples, refer to the following: - -- [`CREATE TABLE` - Create a table](create-table.html) -- [Define Table Partitions](partitioning.html) -- [Constraints - Supported constraints](constraints.html#supported-constraints) - -### Privileges for users and roles - -The Oracle privileges for [users](create-user.html) and [roles](create-role.html) must be rewritten for CockroachDB. Once the CockroachDB cluster is [secured](security-overview.html), CockroachDB follows the same [role-based access control](authorization.html) methodology as Oracle. - - -## Step 8. Import the CSV - -For example, to import the data from `CUSTOMERS.csv.gz` into a new `CUSTOMERS` table, issue the following statement in the CockroachDB SQL shell: - -{% include_cached copy-clipboard.html %} -~~~ sql -> IMPORT TABLE customers ( - customer_id DECIMAL - NOT NULL - PRIMARY KEY, - cust_first_name VARCHAR(40) NOT NULL, - cust_last_name VARCHAR(40) NOT NULL, - nls_language VARCHAR(3), - nls_territory VARCHAR(30), - credit_limit DECIMAL(9,2), - cust_email VARCHAR(100), - account_mgr_id DECIMAL, - customer_since DATE, - customer_class VARCHAR(40), - suggestions VARCHAR(40), - dob DATE, - mailshot VARCHAR(1), - partner_mailshot VARCHAR(1), - preferred_address DECIMAL, - preferred_card DECIMAL, - INDEX cust_email_ix (cust_email), - INDEX cust_dob_ix (dob), - INDEX cust_account_manager_ix ( - account_mgr_id - ) - ) - CSV DATA ( - 'https://your-bucket-name.s3.us-east-2.amazonaws.com/CUSTOMERS.csv.gz' - ) - WITH delimiter = e'\t', - "nullif" = '', - decompress = 'gzip'; -~~~ - -~~~ - job_id | status | fraction_completed | rows | index_entries | system_records | bytes ---------------------+-----------+--------------------+--------+---------------+----------------+---------- - 381866942129111041 | succeeded | 1 | 300024 | 0 | 0 | 13258389 -(1 row) -~~~ - -{% include {{ page.version.version }}/sql/use-import-into.md %} - -Then add the [computed columns](computed-columns.html), [constraints](add-constraint.html), and [function-based indexes](create-index.html). For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE CUSTOMERS SET credit_limit = 50000 WHERE credit_limit > 50000; - ALTER TABLE CUSTOMERS ADD CONSTRAINT CUSTOMER_CREDIT_LIMIT_MAX CHECK (credit_limit <= 50000); - ALTER TABLE CUSTOMERS ADD COLUMN LOW_CUST_LAST_NAME STRING AS (lower(CUST_LAST_NAME)) STORED; - ALTER TABLE CUSTOMERS ADD COLUMN LOW_CUST_FIRST_NAME STRING AS (lower(CUST_FIRST_NAME)) STORED; - CREATE INDEX CUST_FUNC_LOWER_NAME_IX on CUSTOMERS (LOW_CUST_LAST_NAME,CUST_FIRST_NAME); -~~~ - -Repeat the above for each CSV file you want to import. - -## Step 9. Refactor application SQL - -The last phase of the migration process is to change the [transactional behavior](#transactions-locking-and-concurrency-control) and [SQL dialect](#sql-dialect) of your application. - -### Transactions, locking, and concurrency control - -Both Oracle and CockroachDB support [multi-statement transactions](transactions.html), which are atomic and guarantee ACID semantics. However, CockroachDB operates in a serializable isolation mode while Oracle defaults to read committed, which can create both non-repeatable reads and phantom reads when a transaction reads data twice. It is typical that Oracle developers will use `SELECT FOR UPDATE` to work around read committed issues. The [`SELECT FOR UPDATE`](select-for-update.html) statement is also supported in CockroachDB. - -Regarding locks, Cockroach utilizes a [lightweight latch](architecture/transaction-layer.html#latch-manager) to serialize access to common keys across concurrent transactions. Oracle and CockroachDB transaction control flows only have a few minor differences; for more details, refer to [Transactions - SQL statements](transactions.html#sql-statements). - -As CockroachDB does not allow serializable anomalies, [transactions](begin-transaction.html) may experience deadlocks or [read/write contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention). This is expected during concurrency on the same keys. These can be addressed with either [automatic retries](transactions.html#automatic-retries) or [client-side intervention techniques](transactions.html#client-side-intervention). - -### SQL dialect - -Cockroach is ANSI SQL compliant with a Postgres dialect, which allows you to use [native drivers](install-client-drivers.html) to connect applications and ORMs to CockroachDB. CockroachDB’s [SQL Layer](architecture/sql-layer.html#sql-api) supports full relational schema and SQL (similar to Oracle). - -You will have to refactor Oracle SQL and functions that do not comply with [ANSI SQL-92](https://en.wikipedia.org/wiki/SQL-92) in order to work with CockroachDB. For more information about the [Cockroach SQL Grammar](sql-grammar.html) and a [SQL comparison](sql-feature-support.html), see below: - -- [SQL best practices](performance-best-practices-overview.html) -- [Common table expressions (CTE)](common-table-expressions.html) -- `DUAL` table - - Oracle requires use of the `DUAL` table, as Oracle requires a `SELECT ... FROM`. In CockroachDB, all reference to the `DUAL` table should be eliminated. - -- [Function call syntax](functions-and-operators.html#string-and-byte-functions) -- [Hints](cost-based-optimizer.html#join-hints) - - See also: [Table Expressions - Force index selection](table-expressions.html#force-index-selection) - -- [Joins](joins.html) - - CockroachDB supports [`HASH`](joins.html#hash-joins), [`MERGE`](joins.html#merge-joins), and [`LOOKUP`](joins.html#lookup-joins) joins. Oracle uses the `+` operator for `LEFT` and `RIGHT` joins, but CockroachDB uses the ANSI join syntax. - -- [Sequences](create-sequence.html) - - Sequences in CockroachDB do not require a trigger to self-increment; place the sequence in the table DDL: - - ~~~ - > CREATE TABLE customer_list ( - id INT PRIMARY KEY DEFAULT nextval('customer_seq'), - customer string, - address string - ); - ~~~ - -- [Subqueries](subqueries.html) -- `SYSDATE` - - CockroachDB does not support `SYSDATE`; however, it does support date and time with the following: - - ~~~ - > SELECT transaction_timestamp(), clock_timestamp(); - ~~~ - - ~~~ - > SELECT current_timestamp - ~~~ - - ~~~ - > SELECT now(); - ~~~ - -- [Window functions](window-functions.html) - -## See also - -- [`IMPORT`](import.html) -- [Import Performance Best Practices](import-performance-best-practices.html) -- [Migrate from CSV](migrate-from-csv.html) -- [Migrate from MySQL](migrate-from-mysql.html) -- [Migrate from Postgres](migrate-from-postgres.html) -- [SQL Dump (Export)](cockroach-dump.html) -- [Back Up and Restore Data](take-full-and-incremental-backups.html) -- [Use the Built-in SQL Client](cockroach-sql.html) -- [Other Cockroach Commands](cockroach-commands.html) diff --git a/src/current/v21.1/migrate-from-postgres.md b/src/current/v21.1/migrate-from-postgres.md deleted file mode 100644 index 75e13afa753..00000000000 --- a/src/current/v21.1/migrate-from-postgres.md +++ /dev/null @@ -1,278 +0,0 @@ ---- -title: Migrate from Postgres -summary: Learn how to migrate data from Postgres into a CockroachDB cluster. -toc: true ---- - -This page has instructions for migrating data from Postgres to CockroachDB using [`IMPORT`][import]'s support for reading [`pg_dump`][pgdump] files. - -The examples below pull real data from [Amazon S3](https://aws.amazon.com/s3/). They use the [employees data set](https://github.com/datacharmer/test_db) that is also used in the [MySQL docs](https://dev.mysql.com/doc/employee/en/). The data was imported to Postgres using [pgloader][pgloader], and then modified for use here as explained below. - -{% include {{ page.version.version }}/misc/import-perf.md %} - -## Step 1. Dump the Postgres database - -There are several ways to dump data from Postgres to be imported into CockroachDB: - -- [Dump the entire database](#dump-the-entire-database) -- [Dump one table at a time](#dump-one-table-at-a-time) - -The import will fail if the dump file contains functions or type definitions. In addition to calling [`pg_dump`][pgdump] as shown below, you may need to edit the dump file to remove functions and data types. - -Also, note that CockroachDB's [`IMPORT`][import] does not support automatically importing data from Postgres' non-public [schemas][pgschema]. As a workaround, you can edit the dump file to change the table and schema names in the `CREATE TABLE` statements. - -### Dump the entire database - -Most users will want to import their entire Postgres database all at once, as shown below in [Import a full database dump](#import-a-full-database-dump). - -To dump the entire database, run the [`pg_dump`][pgdump] command shown below. - -{% include_cached copy-clipboard.html %} -~~~ shell -$ pg_dump employees > /tmp/employees-full.sql -~~~ - -For this data set, the Postgres dump file required the following edits, which have already been performed on the files used in the examples below: - -- The type of the `employees.gender` column in the `CREATE TABLE` statement had to be changed from `employees.employees_gender` to [`STRING`](string.html) since Postgres represented the employee's gender using a [`CREATE TYPE`](https://www.postgresql.org/docs/10/static/sql-createtype.html) statement that is not supported by CockroachDB. - -- A `CREATE TYPE employee ...` statement needed to be removed. - -If you only want to import one table from a database dump, see [Import a table from a full database dump](#import-a-table-from-a-full-database-dump) below. - -### Dump one table at a time - -To dump the `employees` table from a Postgres database also named `employees`, run the [`pg_dump`][pgdump] command shown below. You can import this table using the instructions in [Import a table from a table dump](#import-a-table-from-a-table-dump) below. - -{% include_cached copy-clipboard.html %} -~~~ shell -$ pg_dump -t employees employees > /tmp/employees.sql -~~~ - -For this data set, the Postgres dump file required the following edits, which have already been performed on the files used in the examples below. - -- The type of the `employees.gender` column in the `CREATE TABLE` statement had to be changed from `employees.employees_gender` to [`STRING`](string.html) since Postgres represented the employee's gender using a [`CREATE TYPE`](https://www.postgresql.org/docs/10/static/sql-createtype.html) statement that is not supported by CockroachDB. - -## Step 2. Host the files where the cluster can access them - -Each node in the CockroachDB cluster needs to have access to the files being imported. There are several ways for the cluster to access the data; for more information on the types of storage [`IMPORT`](import.html) can pull from, see the following: - -- [Use Cloud Storage for Bulk Operations](use-cloud-storage-for-bulk-operations.html) -- [Use a Local File Server for Bulk Operations](use-a-local-file-server-for-bulk-operations.html) - -{{site.data.alerts.callout_success}} -We strongly recommend using cloud storage such as Amazon S3 or Google Cloud to host the data files you want to import. -{{site.data.alerts.end}} - -## Step 3. Import the Postgres dump file - -You can choose from several variants of the [`IMPORT`][import] statement, depending on whether you want to import a full database or a single table: - -- [Import a full database dump](#import-a-full-database-dump) -- [Import a table from a full database dump](#import-a-table-from-a-full-database-dump) -- [Import a table from a table dump](#import-a-table-from-a-table-dump) - -Note that all of the [`IMPORT`][import] statements in this section pull real data from [Amazon S3](https://aws.amazon.com/s3/) and will kick off background import jobs that you can monitor with [`SHOW JOBS`](show-jobs.html). - -{% include {{ page.version.version }}/sql/use-import-into.md %} - -### Import a full database dump - -This example assumes you [dumped the entire database](#dump-the-entire-database). - -The [`IMPORT`][import] statement below reads the data and [DDL](https://en.wikipedia.org/wiki/Data_definition_language) statements (including existing foreign key relationships) from the full database dump file. - -{% include_cached copy-clipboard.html %} -~~~ sql -> IMPORT PGDUMP 'https://s3-us-west-1.amazonaws.com/cockroachdb-movr/datasets/employees-db/pg_dump/employees-full.sql.gz' WITH ignore_unsupported_statements; -~~~ - -~~~ - job_id | status | fraction_completed | rows | index_entries | system_records | bytes ---------------------+-----------+--------------------+--------+---------------+----------------+---------- - 381845110403104769 | succeeded | 1 | 300024 | 0 | 0 | 11534293 -(1 row) -~~~ - -### Import a table from a full database dump - -This example assumes you [dumped the entire database](#dump-the-entire-database). - -[`IMPORT`][import] can import one table's data from a full database dump. It reads the data and applies any `CREATE TABLE` statements from the file. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE DATABASE IF NOT EXISTS employees; -> USE employees; -> IMPORT TABLE employees FROM PGDUMP 'https://s3-us-west-1.amazonaws.com/cockroachdb-movr/datasets/employees-db/pg_dump/employees-full.sql.gz' WITH ignore_unsupported_statements; -~~~ - -~~~ - job_id | status | fraction_completed | rows | index_entries | system_records | bytes ---------------------+-----------+--------------------+--------+---------------+----------------+---------- - 383839294913871873 | succeeded | 1 | 300024 | 0 | 0 | 11534293 -(1 row) -~~~ - -### Import a table from a table dump - -The examples below assume you [dumped one table](#dump-one-table-at-a-time). - -The simplest way to import a table dump is to run [`IMPORT TABLE`][import] as shown below. It reads the table data and any `CREATE TABLE` statements from the file. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE DATABASE IF NOT EXISTS employees; -> USE employees; -> IMPORT PGDUMP 'https://s3-us-west-1.amazonaws.com/cockroachdb-movr/datasets/employees-db/pg_dump/employees.sql.gz' WITH ignore_unsupported_statements; -~~~ - -~~~ - job_id | status | fraction_completed | rows | index_entries | system_records | bytes ---------------------+-----------+--------------------+--------+---------------+----------------+---------- - 383855569817436161 | succeeded | 1 | 300024 | 0 | 0 | 11534293 -(1 row) -~~~ - -If you need to specify the table's columns for some reason, you can use an [`IMPORT TABLE`][import] statement like the one below, which will import data but ignore any `CREATE TABLE` statements in the file, instead relying on the columns you specify. - -{% include_cached copy-clipboard.html %} -~~~ sql -> IMPORT TABLE employees ( - emp_no INT PRIMARY KEY, - birth_date DATE NOT NULL, - first_name STRING NOT NULL, - last_name STRING NOT NULL, - gender STRING NOT NULL, - hire_date DATE NOT NULL - ) - PGDUMP DATA ('https://s3-us-west-1.amazonaws.com/cockroachdb-movr/datasets/employees-db/pg_dump/employees.sql.gz') WITH ignore_unsupported_statements; -~~~ - -## Configuration Options - -The following options are available to `IMPORT ... PGDUMP`: - -+ [Max row size](#max-row-size) -+ {% include_cached new-in.html version="v21.1" %} [Row limit](#row-limit) -+ {% include_cached new-in.html version="v21.1" %} [Ignore unsupported statements](#ignore-unsupported-statements) -+ {% include_cached new-in.html version="v21.1" %} [Log unsupported statements](#log-unsupported-statements) -+ [Skip foreign keys](#skip-foreign-keys) - -### Max row size - -The `max_row_size` option is used to override limits on line size. **Default: 0.5MB**. This setting may need to be tweaked if your Postgres dump file has extremely long lines, for example as part of a `COPY` statement. - -Example usage: - -{% include_cached copy-clipboard.html %} -~~~ sql -> IMPORT TABLE employees ( - emp_no INT PRIMARY KEY, - birth_date DATE NOT NULL, - first_name STRING NOT NULL, - last_name STRING NOT NULL, - gender STRING NOT NULL, - hire_date DATE NOT NULL - ) - PGDUMP DATA ('s3://your-external-storage/employees.sql?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456') WITH max_row_size = '5MB'; -~~~ - -### Row limit - -{% include_cached new-in.html version="v21.1" %} The `row_limit` option determines the number of rows to import. If you are importing one table, setting `row_limit = 'n'` will import the first *n* rows of the table. If you are importing an entire database, this option will import the first *n* rows from each table in the dump file. It is useful for finding errors quickly before executing a more time- and resource-consuming import. - -Example usage: - -{% include_cached copy-clipboard.html %} -~~~ sql -> IMPORT PGDUMP 's3://your-external-storage/employees.sql?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456' WITH row_limit = '10'; -~~~ - -### Ignore unsupported statements - -{% include_cached new-in.html version="v21.1" %} The [`ignore_unsupported_statements` option](import.html#import-options) specifies whether the import will ignore unsupported statements in the `PGDUMP` file. **Default: false**. - -If `ignore_unsupported_statements` is omitted, the import will fail if it encounters a statement that is unsupported by CockroachDB. Use `ignore_unsupported_statements` with `log_ignored_statements` to log unsupported statements. - -Example usage: - -{% include_cached copy-clipboard.html %} -~~~ sql -> IMPORT PGDUMP 's3://your-external-storage/employees.sql?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456' WITH ignore_unsupported_statements; -~~~ - -### Log unsupported statements - -{% include_cached new-in.html version="v21.1" %} The `log_ignored_statements` option is used with the `ignore_unsupported_statements` option to log unsupported statements in the `PGDUMP` file to specified a destination file. - -Example usage: - -{% include_cached copy-clipboard.html %} -~~~ sql -> IMPORT PGDUMP 's3://your-external-storage/employees.sql?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456' WITH ignore_unsupported_statements, log_ignored_statements='userfile://defaultdb.public.userfiles_root/unsupported-statements.log'; -~~~ - -### Skip foreign keys - -By default, [`IMPORT ... PGDUMP`][import] supports foreign keys. **Default: false**. Add the `skip_foreign_keys` option to speed up data import by ignoring foreign key constraints in the dump file's DDL. It will also enable you to import individual tables that would otherwise fail due to dependencies on other tables. - -{{site.data.alerts.callout_info}} -The most common dependency issues are caused by unsatisfied foreign key relationships. You can avoid these issues by adding the `skip_foreign_keys` option to your `IMPORT` statement as needed. For more information, see the list of [import options](import.html#import-options). - -For example, if you get the error message `pq: there is no unique constraint matching given keys for referenced table tablename`, use `IMPORT ... WITH skip_foreign_keys`. -{{site.data.alerts.end}} - -Example usage: - -{% include_cached copy-clipboard.html %} -~~~ sql -> IMPORT TABLE employees ( - emp_no INTEGER PRIMARY KEY, - birth_date DATE NOT NULL, - first_name STRING NOT NULL, - last_name STRING NOT NULL, - gender STRING NOT NULL, - hire_date DATE NOT NULL - ) PGDUMP DATA ('s3://your-external-storage/employees.sql?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456') WITH skip_foreign_keys; -~~~ - -[Foreign key constraints](foreign-key.html) can be added by using [`ALTER TABLE ... ADD CONSTRAINT`](add-constraint.html) commands after importing the data. - -## See also - -- [`IMPORT`][import] -- [Import Performance Best Practices](import-performance-best-practices.html) -- [Migrate from CSV][csv] -- [Migrate from MySQL][mysql] -- [Can a Postgres or MySQL application be migrated to CockroachDB?](frequently-asked-questions.html#can-a-postgresql-or-mysql-application-be-migrated-to-cockroachdb) -- [SQL Dump (Export)](cockroach-dump.html) -- [Back up Data](take-full-and-incremental-backups.html) -- [Restore Data](take-full-and-incremental-backups.html) -- [Use the Built-in SQL Client](cockroach-sql.html) -- [Other Cockroach Commands](cockroach-commands.html) - - - -[csv]: migrate-from-csv.html -[mysql]: migrate-from-mysql.html -[import]: import.html -[pgdump]: https://www.postgresql.org/docs/current/static/app-pgdump.html -[postgres]: migrate-from-postgres.html -[pgschema]: https://www.postgresql.org/docs/current/static/ddl-schemas.html -[pgloader]: https://pgloader.io/ - - diff --git a/src/current/v21.1/migrate-from-shapefiles.md b/src/current/v21.1/migrate-from-shapefiles.md deleted file mode 100644 index 9c220f2b3b8..00000000000 --- a/src/current/v21.1/migrate-from-shapefiles.md +++ /dev/null @@ -1,125 +0,0 @@ ---- -title: Migrate from Shapefiles -summary: Learn how to migrate data from ESRI Shapefiles into a CockroachDB cluster. -toc: true ---- - - CockroachDB supports efficiently storing and querying [spatial data](spatial-data.html). - -This page has instructions for migrating data from ESRI [Shapefiles](spatial-glossary.html#shapefile) into CockroachDB using [`shp2pgsql`](https://manpages.debian.org/stretch/postgis/shp2pgsql.1.en.html) and [`IMPORT`][import]. - -{{site.data.alerts.callout_success}} -We are using `shp2pgsql` in the example below, but [`ogr2ogr`](https://gdal.org/programs/ogr2ogr.html) could also be used, e.g. -`ogr2ogr -f PGDUMP file.sql -lco LAUNDER=NO -lco DROP_TABLE=OFF file.shp` -{{site.data.alerts.end}} - -{% include {{page.version.version}}/spatial/ogr2ogr-supported-version.md %} - -In the example below we will import a [tornadoes data set](http://web.archive.org/web/20201018170120/https://www.spc.noaa.gov/gis/svrgis/zipped/1950-2018-torn-initpoint.zip) that is [available from the US National Weather Service](https://www.spc.noaa.gov/gis/svrgis/) (NWS). - -{{site.data.alerts.callout_info}} -Please refer to the documentation of your GIS software for instructions on exporting GIS data to Shapefiles. -{{site.data.alerts.end}} - -## Before You Begin - -To follow along with the example below, you will need the following prerequisites: - -- CockroachDB [installed](install-cockroachdb.html) and [running](start-a-local-cluster.html) -- [`shp2pgsql`](https://manpages.debian.org/stretch/postgis/shp2pgsql.1.en.html) -- [Python 3](https://www.python.org) - -## Step 1. Download the Shapefile data - -First, download and unzip the tornado data: - -{% include_cached copy-clipboard.html %} -~~~ shell -wget http://web.archive.org/web/20201018170120/https://www.spc.noaa.gov/gis/svrgis/zipped/1950-2018-torn-initpoint.zip -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -unzip 1950-2018-torn-initpoint.zip -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -cd 1950-2018-torn-initpoint/ -~~~ - -## Step 2. Convert the Shapefile data to SQL - -To load the tornado Shapefile into CockroachDB, we must first convert it to SQL using the `shp2pgsql` tool: - -{% include_cached copy-clipboard.html %} -~~~ shell -shp2pgsql 1950-2018-torn-initpoint.shp > tornado-points.sql & -~~~ - -## Step 3. Host the files where the cluster can access them - -Each node in the CockroachDB cluster needs to have access to the files being imported. There are several ways for the cluster to access the data; for a complete list of the types of storage [`IMPORT`][import] can pull from, see [import file locations](import.html#import-file-location). - -For local testing, you can [start a local file server](use-a-local-file-server-for-bulk-operations.html). The following command will start a local file server listening on port 3000: - -{% include_cached copy-clipboard.html %} -~~~ shell -python3 -m http.server 3000 -~~~ - -## Step 4. Prepare the database - -Next, create a `tornadoes` database to store the data in, and switch to it: - -{% include_cached copy-clipboard.html %} -~~~ shell -cockroach sql --insecure -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE DATABASE IF NOT EXISTS tornadoes; -USE tornadoes; -~~~ - -## Step 5. Import the SQL - -Since the file is being served from a local server and is formatted as Postgres-compatible SQL, we can import the data using the following [`IMPORT PGDUMP`](import.html#import-a-postgresql-database-dump) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -IMPORT PGDUMP ('http://localhost:3000/tornado-points.sql') WITH ignore_unsupported_statements; -~~~ - -~~~ - job_id | status | fraction_completed | rows | index_entries | bytes ----------------------+-----------+--------------------+-------+---------------+----------- - 584874782851497985 | succeeded | 1 | 63645 | 0 | 18287213 -(1 row) -~~~ - -## See also - -- [`IMPORT`][import] -- [Export Spatial Data](export-spatial-data.html) -- [Working with Spatial Data](spatial-data.html) -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial indexes](spatial-indexes.html) -- [Using GeoServer with CockroachDB](geoserver.html) -- [Migrate from OpenStreetMap](migrate-from-openstreetmap.html) -- [Migrate from GeoJSON](migrate-from-geojson.html) -- [Migrate from GeoPackage](migrate-from-geopackage.html) -- [Migration Overview](migration-overview.html) -- [Migrate from MySQL][mysql] -- [Migrate from Postgres][postgres] -- [SQL Dump (Export)](cockroach-dump.html) -- [Back Up and Restore Data](take-full-and-incremental-backups.html) -- [Use the Built-in SQL Client](cockroach-sql.html) -- [Other Cockroach Commands](cockroach-commands.html) - - - -[postgres]: migrate-from-postgres.html -[mysql]: migrate-from-mysql.html -[import]: import.html diff --git a/src/current/v21.1/migrate-to-multiregion-sql.md b/src/current/v21.1/migrate-to-multiregion-sql.md deleted file mode 100644 index 99a01f751aa..00000000000 --- a/src/current/v21.1/migrate-to-multiregion-sql.md +++ /dev/null @@ -1,307 +0,0 @@ ---- -title: Migrate to Multi-region SQL -summary: Learn how to migrate to CockroachDB's improved multi-region SQL user experience. -toc: true ---- - -## Overview - -{{site.data.alerts.callout_info}} -If you are already using [multi-region SQL statements](multiregion-overview.html) to control your multi-region cluster, you can ignore this page. -{{site.data.alerts.end}} - -CockroachDB v21.1 added support for [improved multi-region capabilities that make it easier to run global applications](multiregion-overview.html). Using high-level SQL statements, you can control where your data is stored and how it is accessed to provide good performance and tuneable latency for your application's users. - -Prior to v21.1, the only way to accomplish these goals in a multi-region cluster involved using lower-level mechanisms called [replication zones](configure-replication-zones.html) in specific patterns called _Duplicate Indexes_, _Geo-partitioned Replicas_, and _Geo-partitioned leaseholders_. - -These patterns and the use of replication zones are still fully supported. However, for most users, they are harder to use and in some cases can result in worse performance than the multi-region SQL abstractions. - -This page describes how to migrate a multi-region cluster from using replication zones to using multi-region SQL abstractions. It contains: - -- Mappings from each of the legacy replication-zone-based patterns to the multi-region SQL abstractions that are designed to replace them. - -- Instructions for migrating from replication-zone-based workflows to multi-region SQL abstractions. - -- A review of the underlying zone configuration changes that result from running the multi-region SQL statements. - -## Replication zone patterns and multi-region SQL abstractions - -{% include {{page.version.version}}/sql/replication-zone-patterns-to-multiregion-sql-mapping.md %} - -{{site.data.alerts.callout_info}} -CockroachDB will no longer provide the [Follow-the-Workload](topology-follow-the-workload.html) pattern's behavior for a database if you use the [multi-region SQL statements](multiregion-overview.html) with that database. In other words, the multi-region SQL statements do not provide a behavior that is analogous to Follow-the-Workload. -{{site.data.alerts.end}} - -For more information about how to use `ZONE` vs. `REGION` survival goals, see [When to use `ZONE` vs `REGION` survival goals](when-to-use-zone-vs-region-survival-goals.html). - -For more information about when to use `GLOBAL` vs. `REGIONAL` tables, see [When to use `REGIONAL` vs `GLOBAL` tables](when-to-use-regional-vs-global-tables.html). - -## How to migrate a database to the multi-region SQL abstractions - -The following instructions assume that you will re-use your existing [cluster regions](multiregion-overview.html#cluster-regions) (that is, regions added during cluster setup using [`cockroach start --locality`](cockroach-start.html#locality)). - -### Performance considerations - -Expect performance to be temporarily affected by this process. When you run the commands described below, the cluster may need to do a significant amount of work to meet the new replica placement constraints it has just been given. Therefore, we recommend running this procedure at a time when you would normally perform scheduled maintenance. - -Depending on how long you are willing to wait between steps for the replica rebalancing process to settle down, data movement may still be occurring when the next set of constraints is added using the multi-region SQL abstractions; this may lead to further data movement. - -In other words, the process described here may result in a significant increase in CPU usage, IOPS, and network traffic while the cluster rebalances replicas to meet the final set of constraints you have provided it with. Until this process completes, the cluster may not be able to handle its normal workload. - -Note that the process of dropping the old zone configs that occurs when the old configurations are removed **must be complete** before you can [add regions to the database](add-region.html) and set [table localities](set-locality.html). You can ensure this process is complete by waiting for the [`ALTER DATABASE ... CONFIGURE ZONE DISCARD`](configure-zone.html#remove-a-replication-zone) statement shown below to finish successfully. - -For a tutorial that shows how to transition a database to using multi-region SQL statements, see [Low Latency Reads and Writes in a Multi-Region Cluster](demo-low-latency-multi-region-deployment.html). - -{{site.data.alerts.callout_info}} -As part of this migration, data may move temporarily out of the geography where you have constrained it to be placed. For more information about CockroachDB's support for data domiciling, see [Data Domiciling with CockroachDB](data-domiciling.html). -{{site.data.alerts.end}} - -### Step 1. Remove the old replication zone configurations - -Depending on which [legacy multi-region topology pattern](../v20.2/topology-patterns.html#multi-region-patterns) you are migrating from, the procedure will vary. For instructions showing how to remove the existing zone configuration for each pattern, see below. - -- [Duplicate indexes](#duplicate-indexes) -- [Geo-partitioned leaseholders](#geo-partitioned-leaseholders) -- [Geo-partitioned replicas](#geo-partitioned-replicas) - -{{site.data.alerts.callout_success}} -You can check the state of any schema object's replication zone configuration at any time using [`SHOW ZONE CONFIGURATIONS`](show-zone-configurations.html). -{{site.data.alerts.end}} - -#### Duplicate indexes - -If you used the [duplicate indexes pattern][dupe_index], the steps for backing out the old configuration are: - -1. Remove the replication zone configurations you added using the [`ALTER DATABASE ... CONFIGURE ZONE DISCARD`](configure-zone.html#remove-a-replication-zone) statement. Note that this will remove all zone configurations from the table. If you had any additional customizations beyond what are required for the [duplicate indexes][dupe_index] pattern, you will have to reapply them. - - {% include_cached copy-clipboard.html %} - ~~~ sql - ALTER TABLE postal_codes CONFIGURE ZONE DISCARD; - ~~~ - -1. Drop the extra indexes you added. This will have the side effect of also deleting the zone configurations you added to those indexes. - - {% include_cached copy-clipboard.html %} - ~~~ sql - DROP INDEX postal_codes@idx_central; - DROP INDEX postal_codes@idx_east; - ~~~ - -The latency and resiliency benefits of the duplicate indexes pattern can be replaced by making `postal_codes` a [`GLOBAL` table](global-tables.html) with a [survival goal](multiregion-overview.html#survival-goals) that meets your needs. - -#### Geo-partitioned replicas - -If you applied the [geo-partitioned replicas][geo_replicas] pattern, the steps for backing out the old configuration are: - -1. Remove the manually created table partition. This will also automatically remove the replication zone configurations that were applied to the partition as part of the instructions. - - {% include_cached copy-clipboard.html %} - ~~~ sql - ALTER TABLE users PARTITION BY NOTHING; - ~~~ - -1. Remove the manually created partition on the secondary indexes. This will also automatically remove the replication zone configurations that were applied to the partition as part of the instructions. - - {% include_cached copy-clipboard.html %} - ~~~ sql - ALTER INDEX users_last_name_index PARTITION BY NOTHING; - ~~~ - -The latency and resiliency benefits of the geo-partitioned replicas pattern can be replaced by making `users` a [`REGIONAL BY ROW` table](regional-tables.html#regional-by-row-tables) with a [`ZONE` survival goal](multiregion-overview.html#surviving-zone-failures). - -{{site.data.alerts.callout_info}} -The multi-region SQL abstractions use a hidden [`crdb_region`](set-locality.html#crdb_region) column to represent the row's home region. You may need to modify your existing schema to take this into account. For example, if you already have a column you are using to denote each row's home region, you can use that name instead of `crdb_region` by following the instructions on the [`ALTER TABLE ... SET LOCALITY`](set-locality.html#rename-crdb_region) page. -{{site.data.alerts.end}} - -#### Geo-partitioned leaseholders - -If you applied the [geo-partitioned leaseholders][geo_leaseholders] pattern, the steps for backing out the old configuration are: - -1. Remove the manually created table partition. This will also automatically remove the replication zone configurations that were applied to the partition as part of the instructions. - - {% include_cached copy-clipboard.html %} - ~~~ sql - ALTER TABLE users PARTITION BY NOTHING; - ~~~ - -1. Remove the manually created partition on the secondary indexes. This will also automatically remove the replication zone configurations that were applied to the partition as part of the instructions. - - {% include_cached copy-clipboard.html %} - ~~~ sql - ALTER INDEX users_last_name_index PARTITION BY NOTHING; - ~~~ - -The latency and resiliency benefits of the geo-partitioned leaseholders pattern can be replaced by making `users` a [`REGIONAL BY ROW` table](regional-tables.html#regional-by-row-tables) with a [`ZONE` survival goal](multiregion-overview.html#surviving-zone-failures). - -### Step 2. Add a primary region to your database - -The steps from this point forward assume that you have cleared your prior replication zone configurations and will be using the [multi-region SQL abstractions](multiregion-overview.html) to work with a cluster that has existing [cluster regions](multiregion-overview.html#cluster-regions). - -Every multi-region database needs to have a primary region. To set the primary region, issue the [SET PRIMARY REGION](set-primary-region.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE foo SET PRIMARY REGION = "us-west1" -~~~ - -### Step 3. Add more regions to the database - -To add another region to the database, issue the [`ADD REGION`](add-region.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER database foo ADD REGION "us-central1"; -~~~ - -### Step 4. (Optional) Configure your database survival goal - -Depending on your desired database [survival goal](multiregion-overview.html#survival-goals), you can choose from the following settings: - -- [`ZONE` survival](multiregion-overview.html#surviving-zone-failures) (Default): the database will remain fully available for reads and writes if one zone in a region goes down. More than one zone going down concurrently may affect availability. -- [`REGION` survival](multiregion-overview.html#surviving-region-failures): the database will remain fully available for reads and writes if one region goes down. More than one region going down concurrently may affect availability. - -For example, to set a region survival goal, issue the following SQL statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE foo SURVIVE REGION FAILURE; -~~~ - -For more information about when to use `ZONE` vs. `REGION` survival goals, see [When to use `ZONE` vs `REGION` survival goals](when-to-use-zone-vs-region-survival-goals.html). - -### Step 5. Configure table localities - -For each table in your database, apply the [table locality](multiregion-overview.html#table-locality) that provides the latency and resiliency requirements you need for that table. - -As described above, the mapping from legacy replication zone patterns to multi-region SQL abstractions is: - -{% include {{page.version.version}}/sql/replication-zone-patterns-to-multiregion-sql-mapping.md %} - -For example, to configure the `postal_codes` table from the [duplicate indexes example above](#duplicate-indexes) to use [multi-region SQL](multiregion-overview.html), you would enter the following statements to make the `postal_codes` table a [`GLOBAL` table](global-tables.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER TABLE postal_codes SET LOCALITY GLOBAL; -~~~ - -For more information about when to use `GLOBAL` vs. `REGIONAL` tables, see [When to use `REGIONAL` vs `GLOBAL` tables](when-to-use-regional-vs-global-tables.html). - -### Step 6. (Optional) View the updated zone configurations - -The multi-region SQL statements operate on the same replication zone configurations that you have access to via the [`ALTER TABLE ... CONFIGURE ZONE`](configure-zone.html) statement. If you are interested in seeing how they work with the lower-level zone config mechanisms, you can use the [`SHOW ZONE CONFIGURATIONS`](show-zone-configurations.html) statement to view the zone configurations. - -For example, given a multi-region demo cluster set up using the instructions in [Low latency Reads and Writes in a Multi-Region Cluster](demo-low-latency-multi-region-deployment.html), here is what the zone configs for several tables in the [MovR schema](movr.html) look like. - -#### Regional by row tables - -A [`REGIONAL BY ROW`](multiregion-overview.html#regional-by-row-tables) table differs from the default by setting the following zone configuration settings: - -- [`num_replicas`](configure-replication-zones.html#num_replicas) -- [`num_voters`](configure-replication-zones.html#num_voters) -- [`constraints`](configure-replication-zones.html#constraints) -- [`voter_constraints`](configure-replication-zones.html#voter_constraints) -- [`lease_preferences`](configure-replication-zones.html#lease_preferences) - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW ZONE CONFIGURATION FROM TABLE users; -~~~ - -~~~ - target | raw_config_sql -----------------+------------------------------------------------------------------------------------------- - DATABASE movr | ALTER DATABASE movr CONFIGURE ZONE USING - | range_min_bytes = 134217728, - | range_max_bytes = 536870912, - | gc.ttlseconds = 90000, - | num_replicas = 5, - | num_voters = 3, - | constraints = '{+region=europe-west1: 1, +region=us-east1: 1, +region=us-west1: 1}', - | voter_constraints = '[+region=us-east1]', - | lease_preferences = '[[+region=us-east1]]' -(1 row) -~~~ - -The same table, loaded in a demo cluster with no zone configuration changes, looks like this: - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW ZONE CONFIGURATION FROM TABLE users; -~~~ - -~~~ - target | raw_config_sql -----------------+------------------------------------------- - RANGE default | ALTER RANGE default CONFIGURE ZONE USING - | range_min_bytes = 134217728, - | range_max_bytes = 536870912, - | gc.ttlseconds = 90000, - | num_replicas = 3, - | constraints = '[]', - | lease_preferences = '[]' -(1 row) - -~~~ - -#### Global tables - -A [`GLOBAL`](global-tables.html) table differs from the default by setting the following zone configuration settings: - -- [`global_reads`](configure-replication-zones.html#global_reads) -- [`num_replicas`](configure-replication-zones.html#num_replicas) -- [`num_voters`](configure-replication-zones.html#num_voters) -- [`constraints`](configure-replication-zones.html#constraints) -- [`voter_constraints`](configure-replication-zones.html#voter_constraints) -- [`lease_preferences`](configure-replication-zones.html#lease_preferences) - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW ZONE CONFIGURATION FROM TABLE promo_codes; -~~~ - -~~~ - target | raw_config_sql ---------------------+------------------------------------------------------------------------------------------- - TABLE promo_codes | ALTER TABLE promo_codes CONFIGURE ZONE USING - | range_min_bytes = 134217728, - | range_max_bytes = 536870912, - | gc.ttlseconds = 90000, - | global_reads = true, - | num_replicas = 5, - | num_voters = 3, - | constraints = '{+region=europe-west1: 1, +region=us-east1: 1, +region=us-west1: 1}', - | voter_constraints = '[+region=us-east1]', - | lease_preferences = '[[+region=us-east1]]' -(1 row) -~~~ - -The same table, loaded in a demo cluster with no zone configuration changes, looks like this: - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW ZONE CONFIGURATION FROM TABLE promo_codes; -~~~ - -~~~ - target | raw_config_sql -----------------+------------------------------------------- - RANGE default | ALTER RANGE default CONFIGURE ZONE USING - | range_min_bytes = 134217728, - | range_max_bytes = 536870912, - | gc.ttlseconds = 90000, - | num_replicas = 3, - | constraints = '[]', - | lease_preferences = '[]' -(1 row) -~~~ - -## See also - -- [Multi-region overview](multiregion-overview.html) -- [When to use `REGIONAL` vs. `GLOBAL` tables](when-to-use-regional-vs-global-tables.html) -- [When to use `ZONE` vs. `REGION` survival goals](when-to-use-zone-vs-region-survival-goals.html) -- [Topology Patterns](topology-patterns.html) -- [Disaster Recovery](disaster-recovery.html) -- [Multi-region SQL performance](demo-low-latency-multi-region-deployment.html) -- [Configure replication zones](configure-replication-zones.html) -- [Non-voting replicas](architecture/replication-layer.html#non-voting-replicas) diff --git a/src/current/v21.1/migration-overview.md b/src/current/v21.1/migration-overview.md deleted file mode 100644 index b1d345494ea..00000000000 --- a/src/current/v21.1/migration-overview.md +++ /dev/null @@ -1,80 +0,0 @@ ---- -title: Migration Overview -summary: Learn how to migrate data into a CockroachDB cluster. -toc: true ---- - -CockroachDB supports [importing](import.html) data from the following databases: - -- MySQL -- Oracle (using CSV) -- Postgres (and PostGIS) - -and from the following data formats: - -- CSV/TSV -- Avro -- ESRI Shapefiles (`.shp`) (using `shp2pgsql`) -- OpenStreetMap data files (`.pbf`) (using `osm2pgsql`) -- GeoPackage data files (`.gpkg`) (using `ogr2ogr`) -- GeoJSON data files (`.geojson`) (using `ogr2ogr`) - -This page lists general considerations to be aware of as you plan your migration to CockroachDB. - -In addition to the information listed below, see the following pages for specific instructions and considerations that apply to the database (or data format) you're migrating from: - -- [Migrate from Oracle][oracle] -- [Migrate from Postgres][postgres] -- [Migrate from MySQL][mysql] -- [Migrate from CSV][csv] -- [Migrate from Avro][avro] -- [Migrate from Shapefiles][shp] -- [Migrate from OpenStreetMap][pbf] -- [Migrate from GeoPackage][gpkg] -- [Migrate from GeoJSON][geojson] - -{% include {{ page.version.version }}/misc/import-perf.md %} - -## File storage during import - -During migration, all of the features of [`IMPORT`][import] that interact with external file storage assume that every node has the exact same view of that storage. In other words, in order to import from a file, every node needs to have the same access to that file. - -## Schema and application changes - -In general, you are likely to have to make changes to your schema, and how your app interacts with the database. We **strongly recommend testing your application against CockroachDB** to ensure that: - -1. The state of your data is what you expect post-migration. -2. Performance is as expected for your application's workloads. You may need to apply some [best practices for optimizing SQL performance in CockroachDB](performance-best-practices-overview.html). - -## Data type sizes - -Above a certain size, many data types such as [`STRING`](string.html)s, [`DECIMAL`](decimal.html)s, [`ARRAY`](array.html), [`BYTES`](bytes.html), and [`JSONB`](jsonb.html) may run into performance issues due to [write amplification](https://en.wikipedia.org/wiki/Write_amplification). See each data type's documentation for its recommended size limits. - -## See also - -- [`IMPORT`][import] -- [Import Performance Best Practices](import-performance-best-practices.html) -- [Migrate from Oracle][oracle] -- [Migrate from CSV][csv] -- [Migrate from MySQL][mysql] -- [Migrate from Postgres][postgres] -- [Migrate from Avro][avro] -- [Can a Postgres or MySQL application be migrated to CockroachDB?](frequently-asked-questions.html#can-a-postgresql-or-mysql-application-be-migrated-to-cockroachdb) -- [PostgreSQL Compatibility](postgresql-compatibility.html) -- [SQL Dump (Export)](cockroach-dump.html) -- [Back Up and Restore](take-full-and-incremental-backups.html) -- [Use the Built-in SQL Client](cockroach-sql.html) -- [Other Cockroach Commands](cockroach-commands.html) - - - -[oracle]: migrate-from-oracle.html -[postgres]: migrate-from-postgres.html -[mysql]: migrate-from-mysql.html -[csv]: migrate-from-csv.html -[import]: import.html -[avro]: migrate-from-avro.html -[shp]: migrate-from-shapefiles.html -[pbf]: migrate-from-openstreetmap.html -[gpkg]: migrate-from-geopackage.html -[geojson]: migrate-from-geojson.html diff --git a/src/current/v21.1/monitor-cockroachdb-kubernetes.md b/src/current/v21.1/monitor-cockroachdb-kubernetes.md deleted file mode 100644 index f7d282ef0f9..00000000000 --- a/src/current/v21.1/monitor-cockroachdb-kubernetes.md +++ /dev/null @@ -1,274 +0,0 @@ ---- -title: Monitor CockroachDB on Kubernetes -summary: How to monitor a secure 3-node CockroachDB cluster with Kubernetes. -toc: true -toc_not_nested: true ---- - -{{site.data.alerts.callout_info}} -This article assumes you have already [deployed CockroachDB on a single Kubernetes cluster](deploy-cockroachdb-with-kubernetes.html). -{{site.data.alerts.end}} - -Despite CockroachDB's various [built-in safeguards against failure](architecture/replication-layer.html), it is critical to actively monitor the overall health and performance of a cluster running in production and to create alerting rules that promptly send notifications when there are events that require investigation or intervention. - -
    - - - -
    - -## Configure Prometheus - -Every node of a CockroachDB cluster exports granular timeseries metrics formatted for easy integration with [Prometheus](https://prometheus.io/), an open source tool for storing, aggregating, and querying timeseries data. This section shows you how to orchestrate Prometheus as part of your Kubernetes cluster and pull these metrics into Prometheus for external monitoring. - -This guidance is based on [CoreOS's Prometheus Operator](https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md), which allows a Prometheus instance to be managed using built-in Kubernetes concepts. - -{{site.data.alerts.callout_info}} -If you're on Hosted GKE, before starting, make sure the email address associated with your Google Cloud account is part of the `cluster-admin` RBAC group, as shown in [Deploy CockroachDB with Kubernetes](deploy-cockroachdb-with-kubernetes.html#hosted-gke). -{{site.data.alerts.end}} - -1. From your local workstation, edit the `cockroachdb` service to add the `prometheus: cockroachdb` label: - -
    - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl label svc cockroachdb prometheus=cockroachdb - ~~~ - - ~~~ - service/cockroachdb labeled - ~~~ - - This ensures that only the `cockroachdb` (not the `cockroach-public` service) is being monitored by a Prometheus job. - -
    - -
    - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl label svc cockroachdb prometheus=cockroachdb - ~~~ - - ~~~ - service/cockroachdb labeled - ~~~ - - This ensures that only the `cockroachdb` (not the `cockroach-public` service) is being monitored by a Prometheus job. - -
    - -
    - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl label svc my-release-cockroachdb prometheus=cockroachdb - ~~~ - - ~~~ - service/my-release-cockroachdb labeled - ~~~ - - This ensures that there is a Prometheus job and monitoring data only for the `my-release-cockroachdb` service, not for the `my-release-cockroach-public` service. - -
    - -1. Determine the latest version of [CoreOS's Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator/releases/) and run the following to download and apply the latest `bundle.yaml` definition file: - - {{site.data.alerts.callout_info}} - Be sure to specify the latest [CoreOS Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator/releases/) version in the following command, in place of this example's use of version `v0.58.0`. - {{site.data.alerts.end}} - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl apply \ - -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.58.0/bundle.yaml \ - --server-side - ~~~ - - ~~~ - customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com serverside-applied - customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com serverside-applied - customresourcedefinition.apiextensions.k8s.io/probes.monitoring.coreos.com serverside-applied - customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com serverside-applied - customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com serverside-applied - customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com serverside-applied - customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com serverside-applied - clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator serverside-applied - clusterrole.rbac.authorization.k8s.io/prometheus-operator serverside-applied - deployment.apps/prometheus-operator serverside-applied - serviceaccount/prometheus-operator serverside-applied - service/prometheus-operator serverside-applied - ~~~ - -1. Confirm that the `prometheus-operator` has started: - - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get deploy prometheus-operator - ~~~ - - ~~~ - NAME READY UP-TO-DATE AVAILABLE AGE - prometheus-operator 1/1 1 1 27s - ~~~ - -1. Download our Prometheus manifest: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ curl -O https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/prometheus/prometheus.yaml - ~~~ - - {{site.data.alerts.callout_info}} - By default, this manifest uses the secret name generated by the CockroachDB Kubernetes Operator. If you generated your own certificates and keys when [starting CockroachDB](deploy-cockroachdb-with-kubernetes.html#step-2-start-cockroachdb), be sure that `ca.secret.name` matches the name of the node secret you created. - {{site.data.alerts.end}} - -1. Apply the Prometheus manifest. This creates the various objects necessary to run a Prometheus instance: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl apply -f prometheus.yaml - ~~~ - - ~~~ - serviceaccount/prometheus created - clusterrole.rbac.authorization.k8s.io/prometheus created - clusterrolebinding.rbac.authorization.k8s.io/prometheus created - servicemonitor.monitoring.coreos.com/cockroachdb created - prometheus.monitoring.coreos.com/cockroachdb created - ~~~ - -1. Access the Prometheus UI locally and verify that CockroachDB is feeding data into Prometheus: - - 1. Port-forward from your local machine to the pod running Prometheus: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl port-forward prometheus-cockroachdb-0 9090 - ~~~ - - 1. Go to http://localhost:9090 in your browser. - - 1. To verify that each CockroachDB node is connected to Prometheus, go to **Status > Targets**. The screen should look like this: - - Prometheus targets - - 1. To verify that data is being collected, go to **Graph**, enter the `sys_uptime` variable in the field, click **Execute**, and then click the **Graph** tab. The screen should like this: - - Prometheus graph - - {{site.data.alerts.callout_success}} - Prometheus auto-completes CockroachDB time series metrics for you, but if you want to see a full listing, with descriptions, port-forward as described in {% if page.secure == true %}[Access the DB Console](deploy-cockroachdb-with-kubernetes.html#step-4-access-the-db-console){% else %}[Access the DB Console](deploy-cockroachdb-with-kubernetes.html#step-4-access-the-db-console){% endif %} and then point your browser to http://localhost:8080/_status/vars. - - For more details on using the Prometheus UI, see their [official documentation](https://prometheus.io/docs/introduction/getting_started/). - {{site.data.alerts.end}} - -## Configure Alertmanager - -Active monitoring helps you spot problems early, but it is also essential to send notifications when there are events that require investigation or intervention. This section shows you how to use [Alertmanager](https://prometheus.io/docs/alerting/alertmanager/) and CockroachDB's starter [alerting rules](https://github.com/cockroachdb/cockroach/blob/master/cloud/kubernetes/prometheus/alert-rules.yaml) to do this. - -1. Download our alertmanager-config.yaml configuration file: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ curl -O \ - https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/prometheus/alertmanager-config.yaml - ~~~ - -1. Edit the `alertmanager-config.yaml` file to [specify the desired receivers for notifications](https://prometheus.io/docs/alerting/configuration/#receiver). Initially, the file contains a placeholder web hook. - -1. Add this configuration to the Kubernetes cluster as a secret, renaming it to `alertmanager.yaml` and labelling it to make it easier to find: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl create secret generic alertmanager-cockroachdb \ - --from-file=alertmanager.yaml=alertmanager-config.yaml - ~~~ - - ~~~ - secret/alertmanager-cockroachdb created - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl label secret alertmanager-cockroachdb app=cockroachdb - ~~~ - - ~~~ - secret/alertmanager-cockroachdb labeled - ~~~ - - {{site.data.alerts.callout_danger}} - The name of the secret, `alertmanager-cockroachdb`, must match the name used in the `alertmanager.yaml` file. If they differ, the Alertmanager instance will start without configuration, and nothing will happen. - {{site.data.alerts.end}} - -1. Use our [`alertmanager.yaml`](https://github.com/cockroachdb/cockroach/blob/master/cloud/kubernetes/prometheus/alertmanager.yaml) file to create the various objects necessary to run an Alertmanager instance, including a ClusterIP service so that Prometheus can forward alerts: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl apply \ - -f https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/prometheus/alertmanager.yaml - ~~~ - - ~~~ - alertmanager.monitoring.coreos.com/cockroachdb created - service/alertmanager-cockroachdb created - ~~~ - -1. Verify that Alertmanager is running: - - 1. Port-forward from your local machine to the pod running Alertmanager: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl port-forward alertmanager-cockroachdb-0 9093 - ~~~ - - 1. Go to http://localhost:9093 in your browser. The screen should look like this: - - Alertmanager - -1. Ensure that the Alertmanagers are visible to Prometheus by opening http://localhost:9090/status. The screen should look like this: - - Alertmanager - -1. Add CockroachDB's starter [alerting rules](https://github.com/cockroachdb/cockroach/blob/master/cloud/kubernetes/prometheus/alert-rules.yaml): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl apply \ - -f https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/prometheus/alert-rules.yaml - ~~~ - - ~~~ - prometheusrule.monitoring.coreos.com/prometheus-cockroachdb-rules created - ~~~ - -1. Ensure that the rules are visible to Prometheus by opening http://localhost:9090/rules. The screen should look like this: - - Alertmanager - -1. Verify that the `TestAlertManager` example alert is firing by opening http://localhost:9090/alerts. The screen should look like this: - - Alertmanager - -1. To remove the example alert: - - 1. Use the `kubectl edit` command to open the rules for editing: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl edit prometheusrules prometheus-cockroachdb-rules - ~~~ - - 1. Remove the `dummy.rules` block and save the file: - - ~~~ - - name: rules/dummy.rules - rules: - - alert: TestAlertManager - expr: vector(1) - ~~~ diff --git a/src/current/v21.1/monitor-cockroachdb-with-prometheus.md b/src/current/v21.1/monitor-cockroachdb-with-prometheus.md deleted file mode 100644 index b5001a431d6..00000000000 --- a/src/current/v21.1/monitor-cockroachdb-with-prometheus.md +++ /dev/null @@ -1,188 +0,0 @@ ---- -title: Monitor CockroachDB with Prometheus -summary: How to pull CockroachDB's time series metrics into Prometheus. -toc: true ---- - -CockroachDB generates detailed time series metrics for each node in a cluster. This page shows you how to pull these metrics into [Prometheus](https://prometheus.io/), an open source tool for storing, aggregating, and querying time series data. It also shows you how to connect [Grafana](https://grafana.com/) and [Alertmanager](https://prometheus.io/docs/alerting/alertmanager/) to Prometheus for flexible data visualizations and notifications. - -{{site.data.alerts.callout_success}}For details about other monitoring options, see Monitoring and Alerting. {{site.data.alerts.end}} - - -## Before you begin - -- Make sure you have already started a CockroachDB cluster, either [locally](start-a-local-cluster.html) or in a [production environment](manual-deployment.html). - -- Note that all files used in this tutorial can be found in the [`monitoring`](https://github.com/cockroachdb/cockroach/tree/master/monitoring) directory of the CockroachDB repository. - -## Step 1. Install Prometheus - -1. Download the [2.x Prometheus tarball](https://prometheus.io/download/) for your OS. - -2. Extract the binary and add it to your `PATH`. This makes it easy to start Prometheus from any shell. - -3. Make sure Prometheus installed successfully: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ prometheus --version - ~~~ - - ~~~ - prometheus, version 2.2.1 (branch: HEAD, revision: bc6058c81272a8d938c05e75607371284236aadc) - build user: root@149e5b3f0829 - build date: 20180314-14:21:40 - go version: go1.10 - ~~~ - -## Step 2. Configure Prometheus - -1. Download the starter [Prometheus configuration file](https://github.com/cockroachdb/cockroach/blob/master/monitoring/prometheus.yml) for CockroachDB: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ wget https://raw.githubusercontent.com/cockroachdb/cockroach/master/monitoring/prometheus.yml \ - -O prometheus.yml - ~~~ - - When you examine the configuration file, you'll see that it is set up to scrape the time series metrics of a single, insecure local node every 10 seconds: - - `scrape_interval: 10s` defines the scrape interval. - - `metrics_path: '/_status/vars'` defines the Prometheus-specific CockroachDB endpoint for scraping time series metrics. - - `scheme: 'http'` specifies that the cluster being scraped is insecure. - - `targets: ['localhost:8080']` specifies the hostname and `http-port` of the Cockroach node to collect time series metrics on. - -2. Edit the configuration file to match your deployment scenario: - - Scenario | Config Change - ---------|-------------- - Multi-node local cluster | Expand the `targets` field to include `'localhost:'` for each additional node. - Production cluster | Change the `targets` field to include `':'` for each node in the cluster. Also, be sure your network configuration allows TCP communication on the specified ports. - Secure cluster | Uncomment `scheme: 'https'` and comment out `scheme: 'http'`. - -4. Create a `rules` directory and download the [aggregation rules](https://github.com/cockroachdb/cockroach/blob/master/monitoring/rules/aggregation.rules.yml) and [alerting rules](https://github.com/cockroachdb/cockroach/blob/master/monitoring/rules/alerts.rules.yml) for CockroachDB into it: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ mkdir rules - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cd rules - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ wget -P rules https://raw.githubusercontent.com/cockroachdb/cockroach/master/monitoring/rules/aggregation.rules.yml - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ wget -P rules https://raw.githubusercontent.com/cockroachdb/cockroach/master/monitoring/rules/alerts.rules.yml - ~~~ - -## Step 3. Start Prometheus - -1. Start the Prometheus server, with the `--config.file` flag pointing to the configuration file: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ prometheus --config.file=prometheus.yml - ~~~ - - ~~~ - INFO[0000] Starting prometheus (version=1.4.1, branch=master, revision=2a89e8733f240d3cd57a6520b52c36ac4744ce12) source=main.go:77 - INFO[0000] Build context (go=go1.7.3, user=root@e685d23d8809, date=20161128-10:02:41) source=main.go:78 - INFO[0000] Loading configuration file prometheus.yml source=main.go:250 - INFO[0000] Loading series map and head chunks... source=storage.go:354 - INFO[0000] 0 series loaded. source=storage.go:359 - INFO[0000] Listening on :9090 source=web.go:248 - INFO[0000] Starting target manager... source=targetmanager.go:63 - ~~~ - -2. Point your browser to `http://:9090`, where you can use the Prometheus UI to query, aggregate, and graph CockroachDB time series metrics. - - Prometheus auto-completes CockroachDB time series metrics for you, but if you want to see a full listing, with descriptions, point your browser to `http://:8080/_status/vars`. - - For more details on using the Prometheus UI, see their [official documentation](https://prometheus.io/docs/introduction/getting_started/). - -## Step 4. Send notifications with Alertmanager - -Active monitoring helps you spot problems early, but it is also essential to send notifications when there are events that require investigation or intervention. In step 2, you already downloaded CockroachDB's starter [alerting rules](https://github.com/cockroachdb/cockroach/blob/master/monitoring/rules/alerts.rules.yml). Now, download, configure, and start [Alertmanager](https://prometheus.io/docs/alerting/alertmanager/). - -1. Download the [latest Alertmanager tarball](https://prometheus.io/download/#alertmanager) for your OS. - -2. Extract the binary and add it to your `PATH`. This makes it easy to start Alertmanager from any shell. - -3. Make sure Alertmanager installed successfully: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ alertmanager --version - ~~~ - - ~~~ - alertmanager, version 0.15.0-rc.1 (branch: HEAD, revision: acb111e812530bec1ac6d908bc14725793e07cf3) - build user: root@f278953f13ef - build date: 20180323-13:07:06 - go version: go1.10 - ~~~ - -4. [Edit the Alertmanager configuration file](https://prometheus.io/docs/alerting/configuration/) that came with the binary, `simple.yml`, to specify the desired receivers for notifications. - -5. Start the Alertmanager server, with the `--config.file` flag pointing to the configuration file: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ alertmanager --config.file=simple.yml - ~~~ - -6. Point your browser to `http://:9093`, where you can use the Alertmanager UI to define rules for [silencing alerts](https://prometheus.io/docs/alerting/alertmanager/#silences). - -## Step 5. Visualize metrics in Grafana - -Although Prometheus lets you graph metrics, [Grafana](https://grafana.com/) is a much more powerful visualization tool that integrates with Prometheus easily. - -1. [Install and start Grafana for your OS](https://grafana.com/grafana/download). - -2. Point your browser to `http://:3000` and log into the Grafana UI with the default username/password, `admin/admin`, or create your own account. - -3. [Add Prometheus as a datasource](http://docs.grafana.org/datasources/prometheus/), and configure the datasource as follows: - - Field | Definition - ------|----------- - Name | Prometheus - Default | True - Type | Prometheus - Url | `http://:9090` - Access | Direct - -4. Download the starter [Grafana dashboards](https://github.com/cockroachdb/cockroach/tree/master/monitoring/grafana-dashboards) for CockroachDB: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ wget https://raw.githubusercontent.com/cockroachdb/cockroach/master/monitoring/grafana-dashboards/runtime.json - # runtime dashboard: node status, including uptime, memory, and cpu. - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ wget https://raw.githubusercontent.com/cockroachdb/cockroach/master/monitoring/grafana-dashboards/storage.json - # storage dashboard: storage availability. - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ wget https://raw.githubusercontent.com/cockroachdb/cockroach/master/monitoring/grafana-dashboards/sql.json - # sql dashboard: sql queries/transactions. - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ wget https://raw.githubusercontent.com/cockroachdb/cockroach/master/monitoring/grafana-dashboards/replication.json - # replicas dashboard: replica information and operations. - ~~~ - -5. [Add the dashboards to Grafana](http://docs.grafana.org/reference/export_import/#importing-a-dashboard). - -## See also - -- [Monitoring and Alerting](monitoring-and-alerting.html) diff --git a/src/current/v21.1/monitoring-and-alerting.md b/src/current/v21.1/monitoring-and-alerting.md deleted file mode 100644 index 7ad23e9388f..00000000000 --- a/src/current/v21.1/monitoring-and-alerting.md +++ /dev/null @@ -1,193 +0,0 @@ ---- -title: Monitoring and Alerting -summary: Monitor the health and performance of a cluster and alert on critical events and metrics. -toc: true ---- - -Despite CockroachDB's various [built-in safeguards against failure](frequently-asked-questions.html#how-does-cockroachdb-survive-failures), it is critical to actively monitor the overall health and performance of a cluster running in production and to create alerting rules that promptly send notifications when there are events that require investigation or intervention. - -This page explains available monitoring tools and critical events and metrics to alert on. - -## Monitoring tools - -### DB Console - -The [DB Console](ui-overview.html) displays essential metrics about a cluster's health, such as node status, number of unavailable ranges, and queries per second and service latency across the cluster. This tool is designed to help you optimize cluster performance and troubleshoot issues. - -The DB Console is accessible from every node at `http://:`, or `http://:8080` by default. For more information on accessing the DB Console, see [DB Console access](ui-overview.html#db-console-access). - -{{site.data.alerts.callout_danger}} -Because the DB Console is built into CockroachDB, if a cluster becomes unavailable, most of the DB Console becomes unavailable as well. Therefore, it's essential to plan additional methods of monitoring cluster health as described below. -{{site.data.alerts.end}} - -### Cluster API - -The [Cluster API](cluster-api.html) is a REST API that provides much of the same information about your cluster and nodes as is available from the DB Console. - -The API is accessible from each node at the same address and port as the DB Console. - -For more information, see the Cluster API [overview](cluster-api.html) and [reference](https://www.cockroachlabs.com/docs/api/cluster/v2). - -### Prometheus endpoint - -Every node of a CockroachDB cluster exports granular timeseries metrics at `http://:/_status/vars`. The metrics are formatted for easy integration with [Prometheus](https://prometheus.io/), an open source tool for storing, aggregating, and querying timeseries data, but the format is **easy-to-parse** and can be massaged to work with other third-party monitoring systems (e.g., [Sysdig](https://sysdig.atlassian.net/wiki/plugins/servlet/mobile?contentId=64946336#content/view/64946336) and [Stackdriver](https://github.com/GoogleCloudPlatform/k8s-stackdriver/tree/master/prometheus-to-sd)). - -For a tutorial on using Prometheus, see [Monitor CockroachDB with Prometheus](monitor-cockroachdb-with-prometheus.html). - -{% include_cached copy-clipboard.html %} -~~~ shell -$ curl http://localhost:8080/_status/vars -~~~ - -~~~ -# HELP gossip_infos_received Number of received gossip Info objects -# TYPE gossip_infos_received counter -gossip_infos_received 0 -# HELP sys_cgocalls Total number of cgo calls -# TYPE sys_cgocalls gauge -sys_cgocalls 3501 -# HELP sys_cpu_sys_percent Current system cpu percentage -# TYPE sys_cpu_sys_percent gauge -sys_cpu_sys_percent 1.098855319644276e-10 -# HELP replicas_quiescent Number of quiesced replicas -# TYPE replicas_quiescent gauge -replicas_quiescent{store="1"} 20 -... -~~~ - -{{site.data.alerts.callout_info}}In addition to using the exported timeseries data to monitor a cluster via an external system, you can write alerting rules against them to make sure you are promptly notified of critical events or issues that may require intervention or investigation. See Events to Alert On for more details.{{site.data.alerts.end}} - -### Health endpoints - -CockroachDB provides two HTTP endpoints for checking the health of individual nodes. - -Note that these are also available as part of the [Cluster API](cluster-api.html) under `/v2/health/`. - -#### /health - -If a node is down, the `http://:/health` endpoint returns a `Connection refused` error: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ curl http://localhost:8080/health -~~~ - -~~~ -curl: (7) Failed to connect to localhost port 8080: Connection refused -~~~ - -Otherwise, it returns an HTTP `200 OK` status response code with an empty body: - -~~~ -{ - -} -~~~ - -The `/health` endpoint no longer returns details about the node such as its private IP address. These details could be considered privileged information in some deployments. If you need to retrieve node details, you can use the `/_status/details` endpoint along with a valid authentication cookie. - -#### /health?ready=1 - -The `http://:/health?ready=1` endpoint returns an HTTP `503 Service Unavailable` status response code with an error in the following scenarios: - -- The node is draining open SQL connections and rejecting new SQL connections because it is in the process of shutting down (e.g., after being [decommissioned](remove-nodes.html#how-it-works)). This is especially useful for making sure load balancers do not direct traffic to nodes that are live but not "ready", which is a necessary check during [rolling upgrades](upgrade-cockroach-version.html). - - {{site.data.alerts.callout_success}} - If you find that your load balancer's health check is not always recognizing a node as unready before the node shuts down, you can increase the `server.shutdown.drain_wait` [cluster setting](cluster-settings.html) to cause a node to return `503 Service Unavailable` even before it has started shutting down. - {{site.data.alerts.end}} - -- The node is unable to communicate with a majority of the other nodes in the cluster, likely because the cluster is unavailable due to too many nodes being down. - -{% include_cached copy-clipboard.html %} -~~~ shell -$ curl http://localhost:8080/health?ready=1 -~~~ - -~~~ -{ - "error": "node is not ready", - "code": 14 -} -~~~ - -Otherwise, it returns an HTTP `200 OK` status response code with an empty body: - -~~~ -{ - -} -~~~ - -### Raw status endpoints - -{{site.data.alerts.callout_info}} -These endpoints are deprecated in favor of the [Cluster API](#cluster-api). -{{site.data.alerts.end}} - -Several endpoints return raw status metrics in JSON at `http://:/#/debug`. Feel free to investigate and use these endpoints, but note that they are subject to change. - -Raw Status Endpoints - -### Node status command - -The [`cockroach node status`](cockroach-node.html) command gives you metrics about the health and status of each node. - -- With the `--ranges` flag, you get granular range and replica details, including unavailability and under-replication. -- With the `--stats` flag, you get granular disk usage details. -- With the `--decommission` flag, you get details about the [node decommissioning](remove-nodes.html) process. -- With the `--all` flag, you get all of the above. - -## Events to alert on - -Active monitoring helps you spot problems early, but it is also essential to create alerting rules that promptly send notifications when there are events that require investigation or intervention. This section identifies the most important events to create alerting rules for, with the [Prometheus Endpoint](#prometheus-endpoint) metrics to use for detecting the events. - -{{site.data.alerts.callout_success}}If you use Prometheus for monitoring, you can also use our pre-defined alerting rules with Alertmanager. See Monitor CockroachDB with Prometheus for guidance.{{site.data.alerts.end}} - -### Node is down - -- **Rule:** Send an alert when a node has been down for 5 minutes or more. - -- **How to detect:** If a node is down, its `_status/vars` endpoint will return a `Connection refused` error. Otherwise, the `liveness_livenodes` metric will be the total number of live nodes in the cluster. - -### Node is restarting too frequently - -- **Rule:** Send an alert if a node has restarted more than 5 times in 10 minutes. - -- **How to detect:** Calculate this using the number of times the `sys_uptime` metric in the node's `_status/vars` output was reset back to zero. The `sys_uptime` metric gives you the length of time, in seconds, that the `cockroach` process has been running. - -### Node is running low on disk space - -- **Rule:** Send an alert when a node has less than 15% of free space remaining. - -- **How to detect:** Divide the `capacity` metric by the `capacity_available` metric in the node's `_status/vars` output. - -### Node is not executing SQL - -- **Rule:** Send an alert when a node is not executing SQL despite having connections. - -- **How to detect:** The `sql_conns` metric in the node's `_status/vars` output will be greater than `0` while the `sql_query_count` metric will be `0`. You can also break this down by statement type using `sql_select_count`, `sql_insert_count`, `sql_update_count`, and `sql_delete_count`. - -### CA certificate expires soon - -- **Rule:** Send an alert when the CA certificate on a node will expire in less than a year. - -- **How to detect:** Calculate this using the `security_certificate_expiration_ca` metric in the node's `_status/vars` output. - -### Node certificate expires soon - -- **Rule:** Send an alert when a node's certificate will expire in less than a year. - -- **How to detect:** Calculate this using the `security_certificate_expiration_node` metric in the node's `_status/vars` output. - -### Changefeed is experiencing high latency - -- **Rule:** Send an alert when the latency of any changefeed running on any node is higher than the set threshold, which depends on the [`gc.ttlseconds`](configure-replication-zones.html#replication-zone-variables) variable set in the cluster. - -- **How to detect:** Calculate this using a threshold, where the threshold is less than the value of the [`gc.ttlseconds`](configure-replication-zones.html#replication-zone-variables) variable. For example, `changefeed.max_behind_nanos > [some threshold]`. - -## See also - -- [Production Checklist](recommended-production-settings.html) -- [Manual Deployment](manual-deployment.html) -- [Orchestrated Deployment](orchestration.html) -- [Local Deployment](start-a-local-cluster.html) diff --git a/src/current/v21.1/movr-flask-application.md b/src/current/v21.1/movr-flask-application.md deleted file mode 100644 index 921a16395e0..00000000000 --- a/src/current/v21.1/movr-flask-application.md +++ /dev/null @@ -1,389 +0,0 @@ ---- -title: Develop a Global Web Application -summary: This page includes instructions for building a multi-region web application on CockroachDB, using Flask and SQLAlchemy. -toc: true ---- - -This page guides you through developing a globally-available web application. It is the fourth section of the [Develop and Deploy a Global Application](movr-flask-overview.html) tutorial. - - - -## Before you begin - -Before you begin this section, complete the previous section of the tutorial, [Set Up a Virtual Environment for Developing Global Applications](movr-flask-setup.html). - -## Project structure - -The application needs to handle requests from clients, namely web browsers. To translate these kinds of requests into database transactions, the application stack consists of the following components: - -- A multi-region database schema that defines the tables and indexes for user, vehicle, and ride data. - The database schema is covered on a separate page, [Create a Multi-Region Database Schema](movr-flask-database.html). -- A multi-node, geo-distributed CockroachDB cluster, with each node's locality corresponding to cloud provider regions. - Database deployment is covered on two separate pages: - - For instructions on setting up a demo cluster, see [Set Up a Virtual Environment for Developing Multi-Region Applications](movr-flask-setup.html). - - For instructions on setting up a multi-region cluster, see [Deploy a Global Application](movr-flask-deployment.html). -- Python class definitions that map to the tables in our database. - For details, see [Mappings](#mappings) below. -- Functions that wrap database transactions. - For details, see [Transactions](#transactions) below. -- A backend API that defines the application's connection to the database. - For details, see [Database connection](#database-connection) below. -- A [Flask](https://palletsprojects.com/p/flask/) server that handles requests from clients. - For details, see [Web application](#web-application) below. -- HTML files that the Flask server can render into web pages. - For details, see [User interface](#user-interface) below. - -In the sections that follow, we go over each of the files and folders in the project, with a focus on the backend and database components. Here is the application's project directory structure: - -~~~ shell -movr -├── Dockerfile ## Defines the Docker container build -├── LICENSE ## A license file for your application -├── Pipfile ## Lists PyPi dependencies for pipenv -├── Pipfile.lock -├── README.md ## Contains instructions on running the application -├── __init__.py -├── dbinit.sql ## Initializes the database, and partitions the tables by region -├── init.sh ## Initializes the environment variables for debugging -├── movr -│   ├── __init__.py -│   ├── models.py ## Defines classes that map to tables in the movr database -│ ├── movr.py ## Defines the primary backend API -│   └── transactions.py ## Defines transaction callback functions -├── requirements.txt ## Lists PyPi dependencies for Docker container -├── server.py ## Defines a Flask web application to handle requests from clients and render HTML files -├── static ## Static resources for the web frontend -│   ├── css -│   ├── img -│   └── js -├── templates ## HTML templates for the web UI -│   ├── layouts -│   ├── _nav.html -│   ├── login.html -│   ├── register.html -│   ├── rides.html -│   ├── user.html -│   ├── users.html -│   ├── vehicles-add.html -│   └── vehicles.html -└── web - ├── __init__.py - ├── config.py ## Contains Flask configuration settings - ├── forms.py ## Defines FlaskForm classes - └── gunicorn.py ## Contains gunicorn configuration settings -~~~ - - -## SQLAlchemy with CockroachDB - -Object Relational Mappers (ORMs) map classes to tables, class instances to rows, and class methods to transactions on the rows of a table. The `sqlalchemy` package includes some base classes and methods that you can use to connect to your database's server from a Python application, and then map tables in that database to Python classes. - -In our example, we use SQLAlchemy's [Declarative](https://docs.sqlalchemy.org/en/13/orm/extensions/declarative/) extension, which is built on the `mapper()` and `Table` data structures. We also use the [`sqlalchemy-cockroachdb`](https://github.com/cockroachdb/sqlalchemy-cockroachdb) Python package, which defines the CockroachDB SQLAlchemy [dialect](https://docs.sqlalchemy.org/en/13/dialects/). The package includes some functions that help you handle [transactions](transactions.html) in a running CockroachDB cluster. - -### Mappings - -After completing the [Create a Multi-Region Database Schema](movr-flask-database.html) section, you should be familiar with the `movr` database and each of the tables in the database (`users`, `vehicles`, and `rides`). - -Open [`movr/models.py`](https://github.com/cockroachlabs/movr-flask/blob/master/movr/models.py), and look at the first 10 lines of the file: - -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/movr-flask/v2-doc-includes/movr/models.py ||# START front ||# END front %} -~~~ - -The first data structure that the file imports is `declarative_base`, a constructor for the [declarative base class](https://docs.sqlalchemy.org/en/13/orm/extensions/declarative/api.html#sqlalchemy.ext.declarative.declarative_base), built on SQLAlchemy's Declarative extension. All mapped objects inherit from this object. We assign a declarative base object to the `Base` variable right below the imports. - -The `models.py` file also imports some other standard SQLAlchemy data structures that represent database objects (like columns), data types, and constraints, in addition to a standard Python library to help with default values (i.e., `datetime`). - -Since the application handles user logins and authentication, we also need to import some security libraries from the Flask ecosystem. We'll cover the web development libraries in more detail in the [Web application](#web-application) section below. - -#### The `User` class - -Recall that each instance of a table class represents a row in the table, so we name our table classes as if they were individual rows of their parent table, since that's what they'll represent when we construct class objects. - -Take a look at the `User` class definition: - -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/movr-flask/v2-doc-includes/movr/models.py ||# START User ||# END User %} -~~~ - -The `User` class has the following attributes: - -- `__tablename__`, which holds the stored name of the table in the database. SQLAlchemy requires this attribute for all classes that map to tables. -- All of the other attributes of the `User` class (`id`, `city`, `first_name`, etc.), stored as `Column` objects. These attributes represent columns of the `users` table. The constructor for each `Column` takes the column data type as its first argument, and then any additional arguments, such as `primary_key`. -- To help define column objects, SQLAlchemy also includes classes for SQL data types and column constraints. For the columns in this table, we use `UUID` and `String` data types. -- The `__repr__` function, which defines the string representation of the object. - -#### The `Vehicle` class - -Next, look at the `Vehicle` class definition: - -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/movr-flask/v2-doc-includes/movr/models.py ||# START Vehicle ||# END Vehicle %} -~~~ - -Recall that the `vehicles` table contains more columns and data types than the `users` table. It also contains a [foreign key constraint](foreign-key.html) (on the `users` table), and a default value. These differences are reflected in the `Vehicle` class. - -#### The `Ride` class - -Lastly, there's the `Ride` class: - -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/movr-flask/v2-doc-includes/movr/models.py ||# START Ride ||# END Ride %} -~~~ - -The `rides` table has three foreign key constraints, one on the `users` table and two on the `vehicles` table. - -### Transactions - -After you create a class for each table in the database, you can start defining functions that bundle together common SQL operations as atomic [transactions](transactions.html). - -#### The SQLAlchemy CockroachDB dialect - -The [`sqlalchemy-cockroachdb`](https://github.com/cockroachdb/sqlalchemy-cockroachdb/tree/master/cockroachdb) Python library handles transactions in SQLAlchemy with the `run_transaction()` function. This function takes a `transactor`, which can be an [`Engine`](https://docs.sqlalchemy.org/en/13/core/connections.html#sqlalchemy.engine.Engine), [`Connection`](https://docs.sqlalchemy.org/en/13/core/connections.html#sqlalchemy.engine.Connection), or [`sessionmaker`](https://docs.sqlalchemy.org/en/13/orm/session_api.html#sqlalchemy.orm.session.sessionmaker) object, and a callback function. It then uses the `transactor` to connect to the database, and executes the callback as a database transaction. For some examples of `run_transaction()` usage, see [Transaction callback functions](#transaction-callback-functions) below. - -`run_transaction()` abstracts the details of [transaction retries](transactions.html#client-side-intervention) away from your application code. Transactions may require retries if they experience deadlock or read/write contention with other concurrent transactions that cannot be resolved without allowing potential serializable anomalies. As a result, a CockroachDB transaction may have to be tried more than once before it can commit. This is part of how we ensure that our transaction ordering guarantees meet the ANSI [SERIALIZABLE](https://en.wikipedia.org/wiki/Isolation_(database_systems)#Serializable) isolation level. - -`run_transaction()` has the following additional benefits: - -- When passed a [`sqlalchemy.orm.session.sessionmaker`](https://docs.sqlalchemy.org/en/latest/orm/session_api.html#session-and-sessionmaker) object, it ensures that a new session is created exclusively for use by the callback, which protects you from accidentally reusing objects via any sessions created outside the transaction. Note that a `sessionmaker` objects is different from a `session` object, which is not an allowable `transactor` for `run_transaction()`. -- By abstracting transaction retry logic away from your application, it keeps your application code portable across different databases. - - Because all callback functions are passed to `run_transaction()`, the `Session` method calls within those callback functions are written a little differently than the typical SQLAlchemy application. Most importantly, those functions must not change the session and/or transaction state. This is in line with the recommendations of the [SQLAlchemy FAQs](https://docs.sqlalchemy.org/en/latest/orm/session_basics.html#session-frequently-asked-questions), which state (with emphasis added by the original author) that - -As a general rule, the application should manage the lifecycle of the session *externally* to functions that deal with specific data. This is a fundamental separation of concerns which keeps data-specific operations agnostic of the context in which they access and manipulate that data. - - and - -Keep the lifecycle of the session (and usually the transaction) **separate and external**. - - In keeping with the above recommendations from the official docs, we strongly recommend avoiding any explicit mutations of the transaction state inside the callback passed to `run_transaction()`. Specifically, we do not make calls to the following functions from inside `run_transaction()`: - - - [`sqlalchemy.orm.Session.commit()`](https://docs.sqlalchemy.org/en/latest/orm/session_api.html?highlight=commit#sqlalchemy.orm.session.Session.commit) (or other variants of `commit()`) - - This is not necessary because `run_transaction()` handles the savepoint/commit logic for you. - - - [`sqlalchemy.orm.Session.rollback()`](https://docs.sqlalchemy.org/en/latest/orm/session_api.html?highlight=rollback#sqlalchemy.orm.session.Session.rollback) (or other variants of `rollback()`) - - This is not necessary because `run_transaction()` handles the commit/rollback logic for you. - - - `Session.flush()` - - This will not work as expected with CockroachDB because [CockroachDB does not support nested transactions](savepoint.html), which are necessary for `Session.flush()` to work properly. If the call to `Session.flush()` encounters an error and aborts, it will try to rollback. This will not be allowed by the currently-executing CockroachDB transaction created by `run_transaction()`, and will result in an error message like the following: `sqlalchemy.orm.exc.DetachedInstanceError: Instance is not bound to a Session; attribute refresh operation cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)`. - - In the example application, all calls to `run_transaction()` are found within the methods of the `MovR` class (defined in [`movr/movr.py`](https://github.com/cockroachlabs/movr-flask/blob/master/movr/movr.py)), which represents the connection to the running database. Requests to the web application frontend (defined in [`server.py`](https://github.com/cockroachlabs/movr-flask/blob/master/server.py)), are routed to the `MovR` class methods. - -#### Transaction callback functions - - To separate concerns, we define all callback functions passed to `run_transaction()` calls in a separate file, [`movr/transactions.py`](https://github.com/cockroachlabs/movr-flask/blob/master/movr/transactions.py). These callback functions wrap `Session` method calls, like [`session.query()`](https://docs.sqlalchemy.org/en/13/orm/session_api.html#sqlalchemy.orm.session.Session.query) and [`session.add()`](https://docs.sqlalchemy.org/en/13/orm/session_api.html#sqlalchemy.orm.session.Session.add), to perform database operations within a transaction. - -{{site.data.alerts.callout_success}} -We recommend that you use a `sessionmaker` object, bound to an existing `Engine`, as the `transactor` that you pass to `run_transaction()`. This protects you from accidentally reusing objects via any sessions created outside the transaction. Every time `run_transaction()` is called, it uses the `sessionmaker` object to create a new [`Session`](https://docs.sqlalchemy.org/en/13/orm/session.html) object for the callback. If the `sessionmaker` is bound to an existing `Engine`, the same database connection can be reused. -{{site.data.alerts.end}} - -`transactions.py` imports all of the table classes that we defined in [`movr/models.py`](https://github.com/cockroachlabs/movr-flask/blob/master/movr/models.py), in addition to some standard Python data structures needed to generate correctly-typed row values that the ORM can write to the database. - -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/movr-flask/v2-doc-includes/movr/transactions.py ||# START front ||# END front %} -~~~ - -##### Reading - -A common query that a client might want to run is a read of the `rides` table. The transaction callback function for this query is defined here as `get_rides_txn()`: - -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/movr-flask/v2-doc-includes/movr/transactions.py ||# START get_rides_txn ||# END get_rides_txn %} -~~~ - -The `get_rides_txn()` function takes a `Session` object and a `rider_id` string as its inputs, and it outputs a list of dictionaries containing the columns-value pairs of a row in the `rides` table. To retrieve the data from the database bound to a particular `Session` object, we use `Session.query()`, a method of the `Session` class. This method returns a `Query` object, with methods for filtering and ordering query results. - -Note that `get_rides_txn()` gets all rides for a specific rider, which might be located across multiple regions. As discussed in [MovR: A Global Application Use-Case](movr-flask-use-case.html#latency-in-global-applications), running queries on data in multiple regions of a multi-region deployment can lead to latency problems. Because this function defines [a read operation](architecture/reads-and-writes-overview.html#read-scenario), it requires fewer trips between replicated nodes than a write operation, but will likely be slower than a read operation on data constrained to a single region. - -Another common query would be to read the registered vehicles in a particular city, to see which vehicles are available for riding. Unlike `get_rides_txn()`, the `get_vehicles_txn()` function takes the `city` string as an input. - -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/movr-flask/v2-doc-includes/movr/transactions.py ||# START get_vehicles_txn ||# END get_vehicles_txn %} -~~~ - -This function filters the query on the `city` column. `vehicle` rows with the same value for `city` are inserted from the same region, making `city` values implicitly correspond to a specific region. Because the `vehicles` table has a `REGIONAL BY ROW` locality, CockroachDB can locality-optimize queries from nodes with a locality matching the [hidden `crdb_region` column](movr-flask-database.html#table-localities). This limits latency, as the query only needs to travel to database deployments in a single region. - -##### Writing - -There are two basic types of write operations: creating new rows and updating existing rows. In SQL terminology, these are [`INSERT`](insert.html)/[`UPSERT`](insert.html) statements and [`UPDATE`](update.html) statements. All transaction callback functions that update existing rows include a `session.query()` call. All functions adding new rows call `session.add()`. Some functions do both. - -For example, `start_ride_txn()`, which is called when a user starts a ride, adds a new row to the `rides` table, and then updates a row in the `vehicles` table. - -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/movr-flask/v2-doc-includes/movr/transactions.py ||# START start_ride_txn ||# END start_ride_txn %} -~~~ - -The function takes the `city` string, `rider_id` UUID, `rider_city` string, and `vehicle_id` UUID as inputs. It queries the `vehicles` table for all vehicles of a specific ID. It also creates a `Ride` object, representing a row of the `rides` table. To add the ride to the table in the database bound to the `Session`, the function calls `Session.add()`. To update a row in the `vehicles` table, it modifies the object attribute. `start_ride_txn()` is called by `run_transaction()`, which commits the transaction to the database. - -Be sure to review the other callback functions in [`movr/transactions.py`](https://github.com/cockroachlabs/movr-flask/blob/master/movr/transactions.py) before moving on to the next section. - -Now that we've covered the table classes and some transaction functions, we can look at the interface that connects web requests to a running CockroachDB cluster. - -### Database connection - -The `MovR` class, defined in [`movr/movr.py`](https://github.com/cockroachlabs/movr-flask/blob/master/movr/movr.py), handles connections to CockroachDB using SQLAlchemy's [`Engine`](https://docs.sqlalchemy.org/en/13/core/connections.html#sqlalchemy.engine.Engine) class. - -Let's start with the beginning of the file: - -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/movr-flask/v2-doc-includes/movr/movr.py ||# START front ||# END front %} -~~~ - -`movr.py` first imports the transaction callback functions that we defined in [`movr/transactions.py`](https://github.com/cockroachlabs/movr-flask/blob/master/movr/transactions.py). It then imports the `run_transaction()` function. The `MovR` class methods call `run_transaction()` to execute the transaction callback functions as transactions. - -The file also imports some `sqlalchemy` libraries to create instances of the `Engine` and `Session` classes, and to [register the CockroachDB as a dialect](https://docs.sqlalchemy.org/en/13/core/connections.html#registering-new-dialects). - -When called, the `MovR` class constructor creates an instance of the `Engine` class using an input connection string. This `Engine` object represents the connection to the database, and is used by the `sessionmaker()` function to create `Session` objects for each transaction. - -### Backend API - -The `MovR` class methods function as the "backend API" for the application. [Frontend requests](#routing) get routed to the these methods. - -We've already defined the transaction logic in the transaction callback functions, in [`movr/transactions.py`](https://github.com/cockroachlabs/movr-flask/blob/master/movr/transactions.py). We can now wrap calls to the `run_transaction()` function in the `MovR` class methods. - -For example, look at the `start_ride()` method, the method to which frontend requests to start a ride are routed: - -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/movr-flask/v2-doc-includes/movr/movr.py ||# START start_ride ||# END start_ride %} -~~~ - -This method takes some keyword arguments, and then returns a `run_transaction()` call. It passes `sessionmaker(bind=self.engine)` as the first argument to `run_transaction()`, which creates a new `Session` object that binds to the `Engine` instance initialized by the `MovR` constructor. It also passes `city`, `rider_id`, `rider_city`, and `vehicle_id`, values passed from the frontend request, as inputs. These are the same keyword arguments, and should be of the same types, as the inputs for the [transaction callback function](#writing) `start_ride_txn()`. - -Be sure to review the other functions in [`movr/movr.py`](https://github.com/cockroachlabs/movr-flask/blob/master/movr/movr.py) before moving on to the next section. - -## Web application - -The application needs a frontend and a user interface. We use Flask for the web server, routing, and forms. For the web UI, we use some basic, Bootstrapped HTML templates and just a little CSS and JS. - -Most of the web application components are found in the `web` and `templates` folders, and in the `server.py` file. - -### Configuration - -We store the Flask configuration settings in [a `Config` class](https://flask.palletsprojects.com/en/1.1.x/config/#development-production), defined in [`web/config.py`](https://github.com/cockroachlabs/movr-flask/blob/master/web/config.py): - -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/movr-flask/v2-doc-includes/web/config.py %} -~~~ - -This file imports the `os` library, which is used to read from environment variables. When debugging a local deployment, these environment variables are set by the `.env` file that `Pipenv` reads. In a multi-region deployment, the environment variables are set by the `Dockerfile`, and by the [managed cloud deployment service](movr-flask-deployment.html). - -The application sets a global Flask configuration variable (`DEBUG`), which it checks to determine whether or not to run against a demo cluster, or a real multi-region deployment. - -### Web forms - -Forms make up an important part of most web application frontends. We define these in [`web/forms.py`](https://github.com/cockroachlabs/movr-flask/blob/master/web/forms.py), using some data structures from the [`flask_wtf`](https://flask-wtf.readthedocs.io/en/latest/) and [`wtforms`](https://wtforms.readthedocs.io/en/stable/) libraries. - -We will not go into much detail about these forms. The important thing to know is that they help handle `POST` requests in the web UI. - -The `CredentialForm` class, for example, defines the fields of the login form that users interface with to send a login request to the server: - -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/movr-flask/v2-doc-includes/web/forms.py ||# START CredentialForm ||# END CredentialForm %} -~~~ - -Most of the forms defined on this page take the inputs for database reads and writes. - -`VehicleForm`, for example, defines the fields of the vehicle registration form. Users enter information about a vehicle they would like to register, and the data is routed to the `add_vehicle()` method defined in [`movr/movr.py`](https://github.com/cockroachlabs/movr-flask/blob/master/movr/movr.py): - -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/movr-flask/v2-doc-includes/web/forms.py ||# START VehicleForm ||# END VehicleForm %} -~~~ - -### Initialization - -[`server.py`](https://github.com/cockroachlabs/movr-flask/blob/master/server.py) defines the main process of the application: the web server. After initializing the database, we run `python server.py` to start up the web server. - -Let's look at the first ten lines of `server.py`: - -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/movr-flask/v2-doc-includes/server.py ||# START front ||# END front %} -~~~ - -The first line imports standard Flask libraries for connecting, routing, and rendering web pages. The next three lines import some libraries from the Flask ecosystem, for [bootstrapping](https://pythonhosted.org/Flask-Bootstrap/), [authentication](https://flask-login.readthedocs.io/en/latest/), and [security](https://github.com/pallets/werkzeug/blob/master/src/werkzeug/security.py). - -The next few lines import other web resources that we've defined separately in our project: - -- The [`MovR`](https://github.com/cockroachlabs/movr-flask/blob/master/movr/movr.py#L10) class, which handles the connection and interaction with the running CockroachDB cluster. -- Several [`FlaskForm`](https://flask-wtf.readthedocs.io/en/latest/api.html#flask_wtf.FlaskForm) superclasses, which define the structure of web forms. -- The [`Config`](https://github.com/cockroachlabs/movr-flask/blob/master/web/config.py) class, which holds configuration information for the Flask application. - -Finally, we import the `DBAPIError` type from `sqlalchemy`, for error handling. - -The next five or so lines initialize the application: - -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/movr-flask/v2-doc-includes/server.py ||# START init ||# END init %} -~~~ - -Calling the `Flask()` constructor initializes our Flask web server. By assigning this to a variable (`app`), we can then configure the application, store variables in its attributes, and call it with functions from other libraries, to add features and functionality to the application. - -For example, we can bootstrap the application for enhanced HTML and form generation with [`Bootstrap(app)`](https://pythonhosted.org/Flask-Bootstrap/basic-usage.html#basic-usage). We can also add authentication with [`LoginManager(app)`](https://flask-login.readthedocs.io/en/latest/#configuring-your-application), and then control the default routes, based on authentication, with the `DEFAULT_ROUTE_AUTHENTICATED` and `DEFAULT_ROUTE_AUTHENTICATED` constants. - -Note that we also define a `protocol` variable that the application later uses to determine its protocol scheme. You should always use HTTPS for secure connections when building an application that accepts user login information. - -After initializing the application, we can connect to the database: - -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/movr-flask/v2-doc-includes/server.py ||# START connect ||# END connect %} -~~~ - -These two lines connect the application to the database. First the application retrieves the connection string that is stored as a configuration variable of the `app`. Then it calls the `MovR()` constructor to establish a connection to the database at the location we provided to the `Config` object. - -### User authentication - -User authentication is handled with the [`Flask-Login`](https://flask-login.readthedocs.io/en/latest/) library. This library manages user logins with the [`LoginManager`](https://flask-login.readthedocs.io/en/latest/#flask_login.LoginManager), and some other functions that help verify if a user has been authenticated or not. - -To control whether certain routes are accessible to a client session, we define a [`user_loader()` function](https://flask-login.readthedocs.io/en/latest/#flask_login.LoginManager.user_loader): - -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/movr-flask/v2-doc-includes/server.py ||# START user_loader ||# END user_loader %} -~~~ - -To restrict access to a certain page to users that are logged in with the [`LoginManager`](https://flask-login.readthedocs.io/en/latest/#flask_login.LoginManager), we add the `@login_required` decorator function to the route. We'll go over some examples in the [Routing](#routing) section below. - -### Routing - -We define all Flask routing functions directly in `server.py`. - -Flask applications use [`@app.route()` decorators](https://flask.palletsprojects.com/en/1.1.x/patterns/viewdecorators/) to handle client requests to specific URLs. When a request is sent to a URL served by the Flask app instance, the server calls the function defined within the decorator (the routing function). - -Flask provides a few useful callbacks to use within a routing function definition: - -- [`redirect()`](https://flask.palletsprojects.com/en/1.1.x/api/#flask.redirect), which redirects a request to a different URL. -- [`render_template()`](https://flask.palletsprojects.com/en/1.1.x/api/#flask.render_template), which renders an HTML page, with Jinja2 templating, into a static webpage. -- [`flash()`](https://flask.palletsprojects.com/en/1.1.x/api/#flask.flash), which sends messages from the application output to the webpage. - -In addition to calling these functions, and some other standard Flask and Python libraries, the application's routing functions need to call some of the methods that we defined in [`movr/movr.py`](https://github.com/cockroachlabs/movr-flask/blob/master/movr/movr.py) (the "backend API"). - -For example, look at the `login()` route: - -~~~ python -{% remote_include https://raw.githubusercontent.com/cockroachlabs/movr-flask/v2-doc-includes/server.py ||# START login_page ||# END login_page %} -~~~ - -### Client Location - -To optimize for latency in a global application, when a user arrives at the website, their request needs to be routed to the application deployment closest to the location from which they made the request. This step is handled outside of the application logic, by a [cloud-hosted, global load balancer](https://en.wikipedia.org/wiki/Cloud_load_balancing). In our [example multi-region deployment](movr-flask-deployment.html), we use a [GCP external load balancer](https://cloud.google.com/load-balancing/docs/https) that distributes traffic based on the location of the request. - -Note that the application uses a `region` variable to keep track of the region in which the application is deployed. This variable is read in from the `REGION` environment variable, which is set during application deployment. The application uses this `region` variable to limit the cities in which a user can look for vehicles to ride to the supported cities in the region. - -### User interface - -For the example application, we limit the web UI to some static HTML web pages, rendered using Flask's built-in [Jinja-2 engine](https://flask.palletsprojects.com/en/1.1.x/templating/). We will not spend much time covering the web UI. Just note that the forms take input from the user, and that input is usually passed to the backend where it is translated into and executed as a database transaction. - -We've also added some Bootstrap syntax and Google Maps, for UX purposes. As you can see, the Google Maps API requires a key. For debugging, you can define this key in the `.env` file. If you decide to use an embedded service like Google Maps in production, you should restrict your Google Maps API key to a specific hostname or IP address from within the cloud provider's console, as the API key could be publicly visible in the HTML. In [Deploying a Multi-Region Web Application](movr-flask-deployment.html), we use GCP secrets to the store the API keys. - -## Next Steps - -After you finish developing and debugging your application, you can start [deploying the application](movr-flask-deployment.html). - -## See also - - -- [SQLAlchemy documentation](https://docs.sqlalchemy.org/en/latest/) -- [Transactions](transactions.html) -- [Flask documentation](https://flask.palletsprojects.com/en/1.1.x/) -- [Build a Python App with CockroachDB and SQLAlchemy](build-a-python-app-with-cockroachdb-sqlalchemy.html) diff --git a/src/current/v21.1/movr-flask-database.md b/src/current/v21.1/movr-flask-database.md deleted file mode 100644 index c3678654cc0..00000000000 --- a/src/current/v21.1/movr-flask-database.md +++ /dev/null @@ -1,110 +0,0 @@ ---- -title: Create a Multi-region Database Schema -summary: This page documents the database schema for the multi-region Flask application built on CockroachDB. -toc: true ---- - -This page guides you through creating a database schema for an example global application. It is the second section of the [Develop and Deploy a Global Application](movr-flask-overview.html) tutorial. - -## Before you begin - -Before you begin this section, complete the previous section of the tutorial, [MovR: A Global Application Use-Case](movr-flask-use-case.html). - -We also recommend reviewing [CockroachDB's multi-region capabilities](multiregion-overview.html), if you have not done so already. - -## The `movr` database - -The example MovR application is built on a multi-region deployment of CockroachDB, loaded with the `movr` database. This database contains the following tables: - -- `users` -- `vehicles` -- `rides` - -These tables store information about the users and vehicles registered with MovR, and the rides associated with those users and vehicles. - -Initialization statements for `movr` are defined in [`dbinit.sql`](https://github.com/cockroachlabs/movr-flask/blob/master/dbinit.sql), a SQL file that you use later in this tutorial to load the database to a running cluster. - -{{site.data.alerts.callout_info}} -The database schema used in this application is a slightly simplified version of the [`movr` database schema](movr.html) that is built into the `cockroach` binary. The schemas are similar, but they are not the same. -{{site.data.alerts.end}} - -## Multi-region in CockroachDB - -A distributed CockroachDB deployment consists of multiple, regional instances of CockroachDB that communicate as a single, logical entity. In [CockroachDB terminology](architecture/overview.html#terms), each instance is called a *node*. Together, the nodes form a *cluster*. - -To keep track of geographic information about nodes in a cluster, CockroachDB uses [*cluster regions*](multiregion-overview.html#cluster-regions), [*database regions*](multiregion-overview.html#database-regions), and [*table localities*](multiregion-overview.html#table-locality). - -### Cluster and database regions - -When you [add a node to a cluster](cockroach-start.html), you can assign the node a specific [locality](cockroach-start.html#locality). Localities represent a geographic region or zone, and are meant to correspond directly to the cloud provider region or zone in which the node is deployed. - -Each unique regional locality is stored in CockroachDB as a [cluster region](multiregion-overview.html#cluster-regions). After a [cluster is deployed](movr-flask-deployment.html), you can assign regions to new and existing databases in the cluster. - -{{site.data.alerts.callout_info}} -Only cluster regions specified at node startup can be used as [database regions](multiregion-overview.html#database-regions). You can view regions available to databases in the cluster with [`SHOW REGIONS FROM CLUSTER`](show-regions.html). -{{site.data.alerts.end}} - -Here is the [`CREATE DATABASE`](create-database.html) statement for the `movr` database: - -{% include_cached copy-clipboard.html %} -~~~ sql -> {% remote_include https://raw.githubusercontent.com/cockroachlabs/movr-flask/v2-doc-includes/dbinit.sql ||-- START database ||-- END database %} -~~~ - -Note that `movr` has the following [database regions](multiregion-overview.html#database-regions), which correspond to regions in Google Cloud: - -- `gcp-us-east1` (primary) -- `gcp-europe-west1` -- `gcp-us-west1` - -### Table localities - -After you have added regions to a database, you can control where the data in each table in the database is stored, using [table localities](multiregion-overview.html#table-locality). - -By default, CockroachDB uses the table locality setting [`REGIONAL BY TABLE IN PRIMARY REGION`](multiregion-overview.html#regional-tables) for all new tables added to a multi-region database. The `REGIONAL BY TABLE` table locality optimizes read and write access to the data in a table from a single region (in this case, the primary region `gcp-us-east1`). - -The `movr` database contains tables with rows of data that need to be accessed by users in more than one region. As a result, none of the tables benefit from using a `REGIONAL BY TABLE` locality. Instead, all three tables in the `movr` database schema should use a [`REGIONAL BY ROW` locality](multiregion-overview.html#regional-by-row-tables). For `REGIONAL BY ROW` tables, CockroachDB automatically assigns each row to a region based on the locality of the node from which the row is inserted. It then optimizes subsequent read and write queries executed from nodes located in the region assigned to the rows being queried. - -{{site.data.alerts.callout_info}} -As shown in the `CREATE TABLE` statements below, the `REGIONAL BY ROW` clauses do not identify a column to track the region for each row. To assign rows to regions, CockroachDB creates and manages a hidden [`crdb_region` column](set-locality.html#crdb_region), of [`ENUM`](enum.html) type `crdb_internal_region`. The values of `crdb_region` are populated using the regional locality of the node from which the query creating the row originates. -{{site.data.alerts.end}} - -## The `users` table - -Here is the `CREATE TABLE` statement for the `users` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> {% remote_include https://raw.githubusercontent.com/cockroachlabs/movr-flask/v2-doc-includes/dbinit.sql ||-- START users ||-- END users %} -~~~ - -## The `vehicles` table - -{% include_cached copy-clipboard.html %} -~~~ sql -> {% remote_include https://raw.githubusercontent.com/cockroachlabs/movr-flask/v2-doc-includes/dbinit.sql ||-- START vehicles ||-- END vehicles %} -~~~ - -Note that the `vehicles` table has a [foreign key constraint](foreign-key.html) on the `users` table, for the `city` and `owner_id` columns. This guarantees that a vehicle is registered to a particular user (i.e., an "owner") in the city where that user is registered. - -## The `rides` table - -{% include_cached copy-clipboard.html %} -~~~ sql -> {% remote_include https://raw.githubusercontent.com/cockroachlabs/movr-flask/v2-doc-includes/dbinit.sql ||-- START rides ||-- END rides %} -~~~ - -Note that, like the `vehicles` table, the `rides` table has foreign key constraints. These constraints are on the `users` and the `vehicles` tables. - -## Next steps - -Now that you are familiar with the `movr` schema, you can [set up a development environment for the application](movr-flask-setup.html). - -## See also - -- [`movr-flask` on GitHub](https://github.com/cockroachlabs/movr-flask) -- [CockroachDB terminology](architecture/overview.html#terms) -- [Configure Replication Zones](configure-replication-zones.html) -- [`CONFIGURE ZONE`](configure-zone.html) -- [Define Table Partitions](partitioning.html) -- [`PARTITION BY`](partition-by.html) diff --git a/src/current/v21.1/movr-flask-deployment.md b/src/current/v21.1/movr-flask-deployment.md deleted file mode 100644 index 4d622fdfa01..00000000000 --- a/src/current/v21.1/movr-flask-deployment.md +++ /dev/null @@ -1,240 +0,0 @@ ---- -title: Deploy a Global, Serverless Application -summary: This page includes instructions for deploying a multi-region web application using CockroachDB Cloud and Google Cloud services. -toc: true ---- - -This page guides you through deploying an application and database in multiple regions. It is the fifth and final section of the [Develop and Deploy a Global Application](movr-flask-overview.html) tutorial. - - - -## Before you begin - -Before you begin this section, complete the previous section of the tutorial, [Develop a Multi-Region Web Application](movr-flask-application.html). After you finish developing and debugging your multi-region application in a local development environment, you are ready to deploy the application and database in multiple regions. - -In addition to the requirements listed in [Setting Up a Virtual Environment for Developing Multi-Region Applications](movr-flask-setup.html), make sure that you have the following: - -- [A Google Cloud account](https://cloud.google.com/) -- [The Google Cloud SDK installed on your local machine](https://cloud.google.com/sdk/install) -- [Docker installed on your local machine](https://docs.docker.com/v17.12/docker-for-mac/install/) - -## Multi-region database deployment - -In production, you want to start a secure CockroachDB cluster, with nodes on machines located in different areas of the world. To deploy CockroachDB in multiple regions, we recommend using [CockroachDB {{ site.data.products.dedicated }}](../cockroachcloud/quickstart-trial-cluster.html). - -{{site.data.alerts.callout_info}} -You can also deploy CockroachDB manually. For instructions, see the [Manual Deployment](manual-deployment.html) page of the Cockroach Labs documentation site. -{{site.data.alerts.end}} - -### Create a multi-region CockroachDB {{ site.data.products.cloud }} cluster - -1. Sign up for a CockroachDB {{ site.data.products.cloud }} account. - -1. [Log in](https://cockroachlabs.cloud/) to your CockroachDB {{ site.data.products.cloud }} account. - -1. On the **Overview** page, select **Create Cluster**. - -1. On the **Create new cluster** page: - - For **Plan**, select CockroachDB {{ site.data.products.dedicated }}. You won't be charged for the first 30 days of service. - - For **Cloud Provider**, select Google Cloud. - - For **Regions & nodes**, add "us-east1", "us-west1", and "europe-west1", with 3 nodes in each region. - - Leave the **Hardware** and **Cluster name** as their default values. - - For **Additional settings**, turn on VPC peering, with the default IP range. - -1. Select **Next**, and on the **Summary** page, enter your credit card details. - - {{site.data.alerts.callout_info}} - You will not be charged until after your free trial expires in 30 days. - {{site.data.alerts.end}} - -1. Select **Create cluster**. - - Your cluster will be created in approximately 20-30 minutes. Watch [this video](https://www.youtube.com/watch?v=XJZD1rorEQE) while you wait to get a preview of how you'll connect to your cluster. - - Once your cluster is created, you will be redirected to the **Cluster Overview** page. - -1. On the **Cluster Overview** page, select **SQL Users** from the side panel. - -1. Select **Add user**, give the user a name and a password, and then add the user. You can use any user name except "root". - -1. Select **Networking** from the side panel, and select **Add network**. - -1. From the **Network** dropdown, select **Current Network** to auto-populate your local machine's IP address. To allow the network to access the cluster's DB Console and to use the CockroachDB client to access the databases, select the **DB Console to monitor the cluster** and **CockroachDB Client to access the databases** checkboxes. - - {{site.data.alerts.callout_info}} - After setting up your application's GCP project and GCP VPC network, you will need to return to the **Networking** page to configure VPC peering. - {{site.data.alerts.end}} - -### Initialize the `movr` database - -1. Open a new terminal window on your local machine, and navigate to the `movr-flask` project directory. - -1. In the CockroachDB {{ site.data.products.cloud }} cluster console, select **Connect** at the top-right corner of the page. - -1. Select the **SQL User** that you created and the **Region** closest to you, and then select **Next**. - -1. From the **Connection info** page, copy the `curl` command to download the cluster's root certificate, and execute the command in your terminal window. This root certificate is the same for all regions in a cluster. - -1. From the **Connection info** page, copy the `cockroach sql` command. In the terminal window, execute the command with an `-f dbinit.sql` option at the end of the command. For example: - - ~~~ shell - $ cockroach sql --url 'postgresql://user:password@cluster.gcp-us-east1.cockroachlabs.cloud:26257/defaultdb?sslmode=verify-full&sslrootcert='$HOME'/Library/CockroachCloud/certs/root.crt' -f dbinit.sql - ~~~ - - This command will initialize the `movr` database on your CockroachDB {{ site.data.products.cloud }} cluster. - -## Global application deployment - -To optimize the latency of requests from the application to the database, you need to deploy the application in multiple regions. - -{{site.data.alerts.callout_danger}} -This example deploys a secure web application. To take HTTPS requests, the web application must be accessible using a public domain name. SSL certificates are not assigned to IP addresses. - -We do not recommend deploying insecure web applications on public networks. -{{site.data.alerts.end}} - -### Set up a GCP project - -1. From the [GCP console](https://console.cloud.google.com/), create a Google Cloud project for the application. - - {{site.data.alerts.callout_info}} - Google Cloud automatically creates a `default` VPC network for each project. - {{site.data.alerts.end}} - -1. In the [API Library](https://console.cloud.google.com/apis/library), enable the following APIs for your project: - - Container Registry API - - Cloud Run Admin API - - Serverless VPC Access API - -1. **Optional:** Enable the [Google Maps Embed API](https://console.cloud.google.com/apis/library), create an API key, restrict the API key to all URLS on your domain name (e.g., `https://site.com/*`), and retrieve the API key. - - {{site.data.alerts.callout_info}} - The example HTML templates include maps. Not providing an API key to the application will prevent the maps from loading, but will not break the rest of the application. - {{site.data.alerts.end}} - -1. Configure/authorize the `gcloud` command-line tool to use your project and region. - - {{site.data.alerts.callout_info}} - `gcloud` is included with the [Google Cloud SDK](https://cloud.google.com/sdk) installation. - {{site.data.alerts.end}} - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ gcloud init - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ gcloud auth login - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ gcloud auth application-default login - ~~~ - -### Set up VPC peering - -1. Open the CockroachDB {{ site.data.products.cloud }} Console for your cluster, select the **Networking**, and then select **VPC peering**. - -1. Set up a new VPC peering connection, with your Google Cloud project's ID, the `default` VPC network created for your Google Cloud project, and a new VPC connection name (e.g., `movr-app-vpc`). - -1. Copy the `gcloud` command provided by the console, and execute the command in a new terminal window. - - {{site.data.alerts.callout_info}} - The connection strings used to connect to CockroachDB {{ site.data.products.cloud }} from an application in a VPC network differ slightly from connection strings to allow-listed IP addresses. - {{site.data.alerts.end}} - -1. Obtain the VPC connection string for each region: - - 1. Open CockroachDB {{ site.data.products.cloud }} Console, and select **Connect** at the top right of the page. - 1. Select **VPC Peering** for **Network Security**, your SQL username for the **SQL User**, one of the cluster regions for **Region**, `movr` for the **Database**. - 1. Select **Next**. On the following page, select the **Connection string** tab, and then copy the connection string provided. - - Take note of each region's connection string. You will need to provide connection information to the Google Cloud Run service in each region. - -1. Open the Google Cloud console for your project. Under the **VPC network** services, select the **Serverless VPC access** service. - -1. For each Cloud Run service, create a new VPC connector: - - For **Name**, use a name specific to the region. - - For **Region**, use the region of the Cloud Run service. - - For **Network**, use `default`. - - For **Subnet**, specify a custom IP range. This range must be an unused `/28` CIDR IP range (`10.8.0.0`, `10.8.1.0`, `10.8.2.0`, etc.). - -### Containerize the application - -1. Create a new folder named `certs` at the top level of the `movr-flask` project, and then copy the root certificate that you downloaded for your cluster to the new folder. The `Dockerfile` for the application contains instructions to mount this folder and certificate onto the container. - -1. Build and run the Docker image locally. - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ docker build -t gcr.io//movr-app:v1 . - ~~~ - - If there are no errors, the container built successfully. - -1. Push the Docker image to the Google Cloud project's container registry. - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ docker push gcr.io//movr-app:v1 - ~~~ - -### Deploy the application - -1. Create a [Google Cloud Run](https://console.cloud.google.com/run/) service for the application, in one of the regions in which the database is deployed (e.g., `gcp-us-east1`): - - Select the container image URL for the image that you just pushed to the container registry. - - Under **Advanced settings**->**Variables & Secrets**, do the following: - - Set an environment variable named `DB_URI` to the VPC connection string for a node on the CockroachDB {{ site.data.products.cloud }} cluster, in the region in which this first Cloud Run service is located. - - Verify that the `DB_URI` value: - 1. Specifies `cockroachdb` as the database protocol. - 1. Includes both the username and password. - 1. Includes the path to the container-mounted root certificate (e.g., `certs/root.crt`). - - For example: `cockroachdb://user:password@internal-cluster.gcp-us-east1.cockroachlabs.cloud:26257/movr?sslmode=verify-full&sslrootcert=certs/root.crt` - - Set an environment variable named `REGION` to the CockroachDB {{ site.data.products.cloud }} region (e.g., `gcp-us-east1`). - - **Optional:** Create a secret for your Google Maps API key and use it to set the environment variable `API_KEY`. You may need to enable the Secret Manager API to do this. - -1. Repeat the Google Cloud Run set-up steps for all regions. - - {{site.data.alerts.callout_info}} - Each region has the same root certificate, but different connection strings. - {{site.data.alerts.end}} - -1. Create an [external HTTPS load balancer](https://console.cloud.google.com/net-services/loadbalancing) to route requests to the right service: - - For the backend configuration, create separate network endpoint groups (NEGs) for the Google Cloud Run services in each region. - - For the frontend configuration, make sure that you use the HTTPS protocol. You can create a [managed IP address](https://console.cloud.google.com/networking/addresses) and a [managed SSL cert](https://console.cloud.google.com/net-services/loadbalancing/advanced/sslCertificates) for the load balancer directly from the load balancer console. For the SSL cert, you will need to specify a domain name. - - For detailed instructions on setting up a load balancer for a multi-region Google Cloud Run backend, see the [GCP Load Balancer docs](https://cloud.google.com/load-balancing/docs/https/setting-up-https-serverless#ip-address). - -1. In the **Cloud DNS** console (found under **Network Services**), create a new zone. You can name the zone whatever you want. Enter the same domain name for which you created a certificate earlier. - -1. Select your zone, and copy the nameserver addresses (under "**Data**") for the recordset labeled "**NS**". - -1. Outside of the GCP console, through your domain name provider, add the nameserver addresses to the authoritative nameserver list for your domain name. - - {{site.data.alerts.callout_info}} - It can take up to 48 hours for changes to the authoritative nameserver list to take effect. - {{site.data.alerts.end}} - -1. Navigate to the domain name and test out your application. - -## Next steps - -### Develop your own application - -This tutorial demonstrates how to develop and deploy an example multi-region application. Most of the development instructions are specific to Python, Flask, and SQLAlchemy, and most of the deployment instructions are specific to Google Cloud Platform (GCP). CockroachDB supports [many more drivers and ORM's for development](example-apps.html), and you can deploy applications using a number of cloud provider orchestration tools and networking services. We encourage you to modify the code and deployments to fit your framework and use case. - -### Upgrade your deployment - -Some time after you have deployed your application, you will likely need to push changes that you've made locally. When pushing changes, be aware that you defined the database separate from the application. If you change a data type, for example, in your application, you will also need to modify the database schema to be compatible with your application's requests. For information about making online changes to database schemas, see [Online Schema Changes](online-schema-changes.html). - -## See also - - -- [CockroachDB {{ site.data.products.dedicated }} documentation](../cockroachcloud/quickstart-trial-cluster.html) -- [Google Cloud Platform documentation](https://cloud.google.com/docs/) -- [Docker documentation](https://docs.docker.com/) -- [Kubernetes documentation](https://kubernetes.io/docs/home/) diff --git a/src/current/v21.1/movr-flask-overview.md b/src/current/v21.1/movr-flask-overview.md deleted file mode 100644 index b4e2367809c..00000000000 --- a/src/current/v21.1/movr-flask-overview.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -title: Develop and Deploy a Global Application -summary: Learn how to build and deploy an application built on CockroachDB, using Flask, SQLAlchemy, CockroachDB Cloud, and Google Cloud services. -toc: true ---- - -This tutorial guides you through developing and deploying a global application built on CockroachDB, using Flask, SQLAlchemy, CockroachDB {{ site.data.products.cloud }}, and [Google Cloud](https://cloud.google.com/) services. - -The following sections make up the tutorial: - -1. [MovR: A Global Application Use-case](movr-flask-use-case.html) -1. [Create a Multi-region Database Schema](movr-flask-database.html) -1. [Set up a Virtual Environment for Developing Global Applications](movr-flask-setup.html) -1. [Develop a Global Application](movr-flask-application.html) -1. [Deploy a Global Application](movr-flask-deployment.html) - -Throughout the tutorial, we reference the source code for an example web application for the fictional vehicle-sharing company [MovR](movr.html). The source code for this application is open source and available on GitHub, in the [`movr-flask` repository ](https://github.com/cockroachlabs/movr-flask). The code is well-commented, with docstrings defined at the beginning of each class and function definition. - -The repo's [README](https://github.com/cockroachlabs/movr-flask/blob/master/README.md) also includes instructions on debugging and deploying the application using Google Cloud services. Those instructions are reproduced in [Set Up a Virtual Environment for Developing Global Applications](movr-flask-setup.html) and [Deploy a Global Application](movr-flask-deployment.html). - - diff --git a/src/current/v21.1/movr-flask-setup.md b/src/current/v21.1/movr-flask-setup.md deleted file mode 100644 index 610fb92b6b0..00000000000 --- a/src/current/v21.1/movr-flask-setup.md +++ /dev/null @@ -1,117 +0,0 @@ ---- -title: Set up a Virtual Environment for Developing Global Applications -summary: This page guides you through setting up a demo multi-region CockroachDB cluster, and a virtual development environment. -toc: true ---- - -This page guides you through setting up a virtual environment for developing and debugging a global application. It is the third section of the [Develop and Deploy a Global Application](movr-flask-overview.html) tutorial. In this section, you will set up a demo CockroachDB cluster, initialize the database, and set up a virtual development environment. - -## Before you begin - -1. Complete the previous section of the tutorial, [Create a Multi-Region Database Schema](movr-flask-database.html). - -1. Make sure that you have the following installed on your local machine: - - [CockroachDB](install-cockroachdb-mac.html) - - [Python 3](https://www.python.org/downloads/) - - [Pipenv](https://pipenv.readthedocs.io/en/latest/) - -1. Clone the [`movr-flask`](https://github.com/cockroachlabs/movr-flask) repository. We'll reference the source code in this repository throughout the tutorial. - -## Set up a demo multi-region CockroachDB cluster - -For debugging and development purposes, you can use the [`cockroach demo`](cockroach-demo.html) command. This command starts up a secure, nine-node demo cluster. - -1. To set up the demo multi-region cluster, run `cockroach demo`, with the `--nodes` and `--demo-locality` flags. The localities specified below assume GCP region names. - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach demo \ - --nodes=9 \ - --demo-locality=region=gcp-us-east1:region=gcp-us-east1:region=gcp-us-east1:\ - region=gcp-us-west1:region=gcp-us-west1:region=gcp-us-west1:\ - region=gcp-europe-west1:region=gcp-europe-west1:region=gcp-europe-west1 \ - --empty - ~~~ - - Keep this terminal window open. Closing it will shut down the demo cluster. - -1. Open another terminal window. In the new window, run the following command to load `dbinit.sql` to the demo database. This file contains the `movr` database schema that we covered in [Create a Multi-Region Database Schema](movr-flask-database.html). - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --url='postgres://demo:@127.0.0.1:26257?sslmode=require' -f dbinit.sql - ~~~ - - -1. In the demo cluster terminal, verify that the database schema loaded properly: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW TABLES; - ~~~ - - ~~~ - table_name - +------------+ - rides - users - vehicles - (3 rows) - ~~~ - -{{site.data.alerts.callout_info}} -In production, you want to start a secure CockroachDB cluster, with nodes on machines located in different areas of the world. For instructions on deploying a multi-region CockroachDB cluster for this application, using CockroachDB {{ site.data.products.dedicated }}, see [Deploy a Global, Serverless Application](movr-flask-deployment.html). -{{site.data.alerts.end}} - - -## Set up a virtual development environment - -For debugging, use [`pipenv`](https://docs.pipenv.org/), a tool that manages dependencies with `pip` and creates virtual environments with `virtualenv`. - -1. Run the following command to initialize the project's virtual environment: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ pipenv --three - ~~~ - - `pipenv` creates a `Pipfile` in the current directory, with the requirements needed for the app. - -1. Run the following command to install the packages listed in the `Pipfile`: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ pipenv install - ~~~ - -1. Activate the virtual environment: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ pipenv shell - ~~~ - - From this shell, you can run any Python 3 application with the required dependencies that you listed in the `Pipfile`, and the environment variables that you listed in the `.env` file. You can exit the shell subprocess at any time with a simple `exit` command. - -1. To test out the application, you can run the server file: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ python server.py - ~~~ - -1. Navigate to the URL provided to test out the application. By default, this should be http://127.0.0.1:5000/. - -{{site.data.alerts.callout_info}} -In production, you want to [containerize](https://www.docker.com/resources/what-container) your application and deploy it with a deployment orchestration tool, like [Kubernetes](https://kubernetes.io/), or with a serverless deployment service, like [Google Cloud Run](https://cloud.google.com/run). We cover deploying the application with Google Cloud Run in [Deploy a Global Application](movr-flask-deployment.html). -{{site.data.alerts.end}} - -## Next steps - -Now that you've set up a development environment, you can start [developing and debugging the application](movr-flask-application.html). - -## See also - -- [`movr-flask` on GitHub](https://github.com/cockroachlabs/movr-flask) -- [`cockroach demo`](cockroach-demo.html) -- [Pipenv](https://pipenv.readthedocs.io/en/latest/) diff --git a/src/current/v21.1/movr-flask-use-case.md b/src/current/v21.1/movr-flask-use-case.md deleted file mode 100644 index 9d714b4a5da..00000000000 --- a/src/current/v21.1/movr-flask-use-case.md +++ /dev/null @@ -1,62 +0,0 @@ ---- -title: MovR: A Global Application Use-Case -summary: This page covers an example use-case for global applications. -toc: true ---- - -This page guides you through an example use-case for global application development and deployment. It is the first section of the [Develop and Deploy a Global Application](movr-flask-overview.html) tutorial. - -## Overview - -MovR is a fictional company that offers users a platform for sharing vehicles, like scooters, bicycles, and skateboards, in select cities across the United States and Europe. To serve users in two continents, the folks at MovR need an application that is globally available, [resilient to system failures](#resiliency-and-distributed-deployments), and [optimized for latency](#latency-in-global-applications). - -## Resiliency and distributed deployments - -For an application to be resilient to system failures, the application and database need to be deployed on multiple machines (i.e., part of a distributed deployment). In distributed CockroachDB deployments, all data is replicated and distributed across the instances of the database that make up the deployment. For more information about data replication and distribution in CockroachDB, see [The Raft Protocol in CockroachDB](https://www.youtube.com/watch?v=k5BR9m8o9ec&feature=youtu.be). - -The replication and distribution of data across multiple machines in a *single region* makes the deployment resilient to individual node failures within the region. Replication and distribution across *multiple regions* makes the deployment resilient to entire regional failures. To achieve the highest level of resiliency, we use a multi-region deployment for MovR database and application. - -{{site.data.alerts.callout_info}} -In the [example deployment](movr-flask-deployment.html), the application and the database deployments are separate and not co-located on the same machine. -{{site.data.alerts.end}} - -## Latency in global applications - -If the MovR application and database are deployed in a single region, latency can become a serious problem when users are located in cities outside the deployment region. Deploying the application and database in multiple regions is not guaranteed to improve latency if client requests are sent to any regional deployment, without consideration for the client's location. - -Limiting latency improves the user experience, and it can also help you avoid problems with data integrity, like [transaction contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention). - -For the purpose of this tutorial, we'll focus on two types of latency: - -- [*Database latency*](#reducing-database-latency), which we define as the time required to complete read/write operations on the database, issued by an application. -- [*Application latency*](#reducing-application-latency), which we define as the time required to make requests to an application from a client. - -### Reducing database latency - -If you are building an application, it's likely that the end user will not be making requests to the database directly. Instead, the user makes requests to the application, and the application makes requests to the database on behalf of the user. - -To limit the latency between the application and the database: - -- Deploy the application so that each instance of the application communicates with the closest database instance. This can be configured during [application deployment](movr-flask-deployment.html). - -- Design the application's persistence layer to limit the number of queries on data stored outside of the region closest to the application. You can use CockroachDB's [multi-region features](multiregion-overview.html) to control where data is stored and from where it is accessed. - - Note that querying the data closest to the application deployment is not always possible, as some data cannot be assigned to a specific region. In the event that a query requires CockroachDB to scan data spread across multiple regions, CockroachDB will first look for the data in the closest region. - -### Reducing application latency - -To limit the latency between client and application server requests, you need to deploy your application such that requests are routed to the application deployment closest to the client. This requires a global load balancer that can redirect traffic to application deployments based on client location. - -We cover setting up an external load balancer in [Deploy a Global, Serverless Application](movr-flask-deployment.html). - -## Next steps - -You should now be ready to start [creating a multi-region database schema](movr-flask-database.html). - -## See also - -- [`movr-flask` on GitHub](https://github.com/cockroachlabs/movr-flask) -- [CockroachDB terminology](architecture/overview.html#terms) -- [Configure Replication Zones](configure-replication-zones.html) -- [Define Table Partitions](partitioning.html) -- [Topology Patterns](topology-patterns.html) diff --git a/src/current/v21.1/movr.md b/src/current/v21.1/movr.md deleted file mode 100644 index 4dd92df68da..00000000000 --- a/src/current/v21.1/movr.md +++ /dev/null @@ -1,103 +0,0 @@ ---- -title: MovR -summary: The MovR application uses CockroachDB to store information about vehicles, users, and rides. -toc: true ---- - -MovR is a fictional vehicle-sharing company created to demonstrate CockroachDB's features. - -## Overview - -The MovR example consists of the following: - -- The `movr` dataset, which contains rows of data that populate tables in the `movr` database. The `movr` dataset is built into [`cockroach demo`](cockroach-demo.html) and [`cockroach workload`](cockroach-workload.html). -- The MovR application, a fully-functional vehicle-sharing application, written in Python. All of MovR application source code is open-source, and available on the [movr](https://github.com/cockroachdb/movr) GitHub repository. - -## The `movr` database - -{% include {{ page.version.version }}/misc/movr-schema.md %} - -## Generating schemas and data for MovR - -You can use the `cockroach demo` and `cockroach workload` commands to load the `movr` database and dataset into a CockroachDB cluster. - -[`cockroach demo`](cockroach-demo.html) opens a SQL shell to a temporary, in-memory cluster. To open a SQL shell to a demo cluster with the `movr` database preloaded and set as the [current database](sql-name-resolution.html#current-database), use the following command: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach demo movr -~~~ - -[`cockroach workload`](cockroach-workload.html) loads sample datasets and workloads into running clusters. To load the `movr` database and some sample data into a running cluster, do the following: - -1. Start a [secure](secure-a-cluster.html) or [insecure](start-a-local-cluster.html) local cluster. -1. Use `cockroach workload` to load the `movr` dataset: - -
    - - -
    - -
    - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload init movr 'postgresql://root@localhost:26257?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' - ~~~ - -
    - -
    - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload init movr 'postgresql://root@localhost:26257?sslmode=disable' - ~~~ - -
    - -1. Use [`cockroach sql`](cockroach-sql.html) to open an interactive SQL shell and set `movr` as the [current database](sql-name-resolution.html#current-database): - -
    - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --certs-dir=certs --host=localhost:26257 - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > USE movr; - ~~~ - -
    - -
    - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --insecure --host=localhost:26257 - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > USE movr; - ~~~ - -
    - -## How the MovR application works - -{% include {{ page.version.version }}/misc/movr-workflow.md %} - -## Extended examples - -For a tutorial on running MovR against a multi-region cluster, using two important multi-region [data topologies](topology-patterns.html) to get very low latency reads and writes, see [Low Latency, Multi-Region Deployment](demo-low-latency-multi-region-deployment.html). - -For a tutorial on developing and deploying a globally-available web application for MovR, see [Develop and Deploy a Global Application](movr-flask-overview.html). - -## See also - -- [Learn CockroachDB SQL](learn-cockroachdb-sql.html) -- [Build an App with CockroachDB](example-apps.html) -- [Experimental Features](experimental-features.html) diff --git a/src/current/v21.1/multi-active-availability.md b/src/current/v21.1/multi-active-availability.md deleted file mode 100644 index 94c92ef1f72..00000000000 --- a/src/current/v21.1/multi-active-availability.md +++ /dev/null @@ -1,66 +0,0 @@ ---- -title: Availability FAQs -summary: Learn about CockroachDB's high availability model, known as Multi-Active Availability. -toc: true ---- - -CockroachDB's availability model is described as "Multi-Active Availability." In essence, multi-active availability provides benefits similar to traditional notions of high availability, but also lets you read and write from every node in your cluster without generating any conflicts. - - -## What is high availability? - -High availability lets an application continue running even if a system hosting one of its services fails. This is achieved by scaling the application's services horizontally, i.e., replicating the service across many machines or systems. If any one of them fails, the others can simply step in and perform the same service. - -Before diving into the details of CockroachDB's multi-active availability, we'll review the two most common high availability designs: [Active-Passive](#active-passive) and [Active-Active](#active-active) systems. - -### Active-Passive - -In active-passive systems, all traffic is routed to a single, "active" replica. Changes to the replica's state are then copied to a backup "passive" replica, in an attempt to always mirror the active replica as closely as possible. - -However, this design has downsides: - -- If you use asynchronous replication, you cannot guarantee that any data is ever successfully replicated to passive followers––meaning you can easily lose data. Depending on your industry, this could have pretty dire consequences. -- If you use synchronous replication and any passive replicas fail, you have to either sacrifice availability for the entire application or risk inconsistencies. - -### Active-Active - -In active-active systems, multiple replicas run identical services, and traffic is routed to all of them. If any replica fails, the others simply handle the traffic that would've been routed to it. - -For databases, though, active-active replication is incredibly difficult to instrument for most workloads. For example, if you let multiple replicas handle writes for the same keys, how do you keep them consistent? - -#### Example: conflicts with active-active replication - -For this example, we have 2 replicas (**A**, **B**) in an active-active high availability cluster. - -1. **A** receives a write for key `xyz` of `'123'`, and then immediately fails. -2. **B** receives a read of key `xyz`, and returns a `NULL` because it cannot find the key. -3. **B** then receives a write for key `xyz` of `'456'`. -4. **A** is restarted and attempts to rejoin **B**––but what do you do about key `xyz`? There's an inconsistency in the system without a clear way to resolve it. - -{{site.data.alerts.callout_info}}In this example, the cluster remained active the entire time. But in terms of the CAP theorem, this is an AP system; it favored being available instead of consistent when partitions occur.{{site.data.alerts.end}} - -## What is multi-active availability? - -Multi-active availability is CockroachDB's version of high availability (keeping your application online in the face of partial failures), which we've designed to avoid the downsides of both active-passive and traditional active-active systems. - -Like active-active designs, all replicas can handle traffic, including both reads and writes. However, CockroachDB improves upon that design by also ensuring that data remains consistent across them, which we achieve by using "consensus replication." In this design, replication requests are sent to at least 3 replicas, and are only considered committed when a majority of replicas acknowledge that they've received it. This means that you can still have failures without compromising availability. - -To prevent conflicts and guarantee your data's consistency, clusters that lose a majority of replicas stop responding because they've lost the ability to reach a consensus on the state of your data. When a majority of replicas are restarted, your database resumes operation. - -### Consistency example - -For this example, we have 3 CockroachDB nodes (**A**, **B**, **C**) in a multi-active availability cluster. - -1. **A** receives a write on `xyz` of `'123'`. It communicates this write to nodes **B** and **C**, who confirm that they've received the write, as well. Once **A** receives the first confirmation, the change is committed. -2. **A** fails. -3. **B** receives a read of key `xyz`, and returns the result `'123'`. -4. **C** then receives an update for key `xyz` to the values `'456'`. It communicates this write to node **B**, who confirms that its received the write, as well. After receiving the confirmation, the change is committed. -5. **A** is restarted and rejoins the cluster. It receives an update that the key `xyz` had its value changed to `'456'`. - -{{site.data.alerts.callout_info}}In this example, if nodes B or C failed at any time, the cluster would have stopped responding. In terms of the CAP theorem, this is a CP system; it favored being consistent instead of available when partitions occur.{{site.data.alerts.end}} - -## What's next? - -To get a greater understanding of how CockroachDB is a survivable system that enforces strong consistency, check out our [architecture documentation](architecture/overview.html). - -To see Multi-Active Availability in action, see this [availability demo](demo-fault-tolerance-and-recovery.html). diff --git a/src/current/v21.1/multilinestring.md b/src/current/v21.1/multilinestring.md deleted file mode 100644 index 9bb00b4addf..00000000000 --- a/src/current/v21.1/multilinestring.md +++ /dev/null @@ -1,77 +0,0 @@ ---- -title: MULTILINESTRING -summary: The spatial MULTILINESTRING object is a collection of LINESTRINGs. -toc: true ---- - -A `MULTILINESTRING` is a collection of [LineStrings](linestring.html). MultiLineStrings are useful for gathering a group of LineStrings into one geometry. For example, you may want to gather the LineStrings denoting all of the roads in a particular municipality. - -{% include {{page.version.version}}/spatial/zmcoords.md %} - -## Examples - -### Well known text - -A MultiLineString can be created from SQL by calling the `st_geomfromtext` function on a MultiLineString definition expressed in the [Well Known Text (WKT)](spatial-glossary.html#wkt) format. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT ST_GeomFromText('MULTILINESTRING((0 0, 1440 900), (800 600, 200 400))'); -~~~ - -~~~ - st_geomfromtext ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - 0105000000020000000102000000020000000000000000000000000000000000000000000000008096400000000000208C4001020000000200000000000000000089400000000000C0824000000000000069400000000000007940 -(1 row) -~~~ - -### SQL - -A MultiLineString can be created from SQL by calling an aggregate function such as `ST_Collect` or [`ST_Union`](st_union.html) on a column that contains [LineString](linestring.html) geometries. In the example below, we will build a MultiLineString from several LineStrings. - -First, insert the LineStrings: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE tmp_linestrings (id INT8 default unique_rowid(), geom GEOMETRY); - -INSERT INTO tmp_linestrings (geom) -VALUES -(st_geomfromtext('SRID=4326;LINESTRING(-88.243385 40.116421, -87.906471 43.038902, -95.992775 36.153980)')), -(st_geomfromtext('SRID=4326;LINESTRING(-75.704722 36.076944, -95.992775 36.153980, -87.906471 43.038902)')), -(st_geomfromtext('SRID=4326;LINESTRING(-76.8261 42.1727, -75.6608 41.4102,-73.5422 41.052, -73.929 41.707, -76.8261 42.1727)')); -~~~ - -Next, build a MultiLineString from the individual [LineStrings](linestring.html) using `ST_Collect`, and check the output with `ST_GeometryType` to verify that it is indeed a MultiLineString: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT ST_GeometryType(st_collect(geom)) AS output FROM tmp_linestrings; -~~~ - -~~~ - output ----------------------- - ST_MultiLineString -(1 row) -~~~ - -Finally, drop the temporary table: - -{% include_cached copy-clipboard.html %} -~~~ sql -DROP TABLE tmp_linestrings; -~~~ - -## See also - -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial objects](spatial-features.html#spatial-objects) -- [POINT](point.html) -- [LINESTRING](linestring.html) -- [POLYGON](polygon.html) -- [MULTIPOINT](multipoint.html) -- [MULTIPOLYGON](multipolygon.html) -- [GEOMETRYCOLLECTION](geometrycollection.html) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/multipoint.md b/src/current/v21.1/multipoint.md deleted file mode 100644 index 19fc9d6187c..00000000000 --- a/src/current/v21.1/multipoint.md +++ /dev/null @@ -1,79 +0,0 @@ ---- -title: MULTIPOINT -summary: The spatial MULTIPOINT object is a collection of POINTs. -toc: true ---- - -A `MULTIPOINT` is a collection of [Points](point.html). MultiPoints are useful for gathering a group of Points into one geometry. For example, you may want to gather the points denoting all of the State Capitols in the U.S. into a single geometry. - -{% include {{page.version.version}}/spatial/zmcoords.md %} - -## Examples - -### SQL - -A MultiPoint can be created from SQL by calling an aggregate function such as `ST_Collect` or [`ST_Union`](st_union.html) on a column that contains [Point](point.html) geometries. In the example below, we will build a MultiPoint from several Points. - -First, insert the Points: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE tmp_points (id INT8 default unique_rowid(), geom GEOMETRY); - -INSERT INTO tmp_points (geom) -VALUES -(st_geomfromtext('POINT (-88.243357000000003 40.117404000000001)')), -(st_geomfromtext('POINT (-94.598371 39.050068000000003)')), -(st_geomfromtext('POINT (-73.962090000000003 40.609226)')); -~~~ - -Next, build a MultiPoint from the individual [Points](point.html) using `ST_Collect`, and check the output with `ST_GeometryType` to verify that it is indeed a MultiPoint: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT ST_GeometryType(st_collect(geom)) AS output FROM tmp_points; -~~~ - -~~~ - output ------------------ - ST_MultiPoint -(1 row) -~~~ - -Finally, drop the temporary table: - -{% include_cached copy-clipboard.html %} -~~~ sql -DROP TABLE tmp_points; -~~~ - -### Well known text - -A MultiPoint can be created from SQL by calling the `st_geomfromtext` function on a MultiPoint definition expressed in the [Well Known Text (WKT)](spatial-glossary.html#wkt) format. - -For example, the MultiPoint in the example below includes the locations of [independent bookstores in Chicago, Illinois USA](https://www.bookweb.org/member_directory/search/ABAmember/results/0/Chicago/IL/0): - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT ST_GeomFromText('MULTIPOINT (-87.738258999999999 42.010930999999999, -87.716257999999996 41.981231000000001, -87.708889999999997 41.975000000000001, -87.707705000000004 41.929195999999997, -87.707192000000006 41.926580000000001, -87.704300000000003 41.928013999999997, -87.698012000000006 41.939076, -87.682384999999996 41.943232000000002, -87.681599000000006 41.705936999999999, -87.677763999999996 41.916998, -87.674808999999996 41.9086, -87.668653000000006 41.977356999999998, -87.668611999999996 41.904580000000003, -87.664944000000006 41.921931999999998, -87.655131999999995 41.881686000000002, -87.654752999999999 41.881632000000003, -87.654584 41.944774000000002, -87.653409999999994 41.857928000000001, -87.650779999999997 41.926853000000001, -87.644745999999998 41.941915999999999, -87.644356999999999 41.899109000000003, -87.634562000000003 41.897446000000002, -87.630498000000003 41.899751000000002, -87.629164000000003 41.873215999999999, -87.627983999999998 41.883583999999999, -87.627189999999999 41.890832000000003, -87.624488999999997 41.885147000000003, -87.624283000000005 41.876899000000002, -87.624251999999998 41.874115000000003, -87.622851999999995 41.894931999999997, -87.619151000000002 41.864832999999997, -87.597796000000002 41.789636000000002, -87.596547999999999 41.790515999999997, -87.594948000000002 41.791434000000002, -87.591048999999998 41.808132999999998, -87.590436999999994 41.783611000000001, -87.590277 41.800938000000002)'); -~~~ - -~~~ - st_geomfromtext ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - 0104000000250000000101000000923EADA23FEF55C09CC1DF2F660145400101000000E55FCB2BD7ED55C0236937FA98FD44400101000000889D29745EED55C0CDCCCCCCCCFC44400101000000CE8DE9094BED55C079C9FFE4EFF6444001010000008BFF3BA242ED55C05890662C9AF644400101000000A9A44E4013ED55C0751DAA29C9F644400101000000CC0D863AACEC55C0B03A72A433F844400101000000FB912232ACEB55C0354580D3BBF844400101000000207F69519FEB55C0A704C4245CDA44400101000000FE99417C60EB55C0AB3FC23060F544400101000000F982161230EB55C0A3923A014DF444400101000000D4D7F335CBEA55C022C2BF081AFD44400101000000A46DFC89CAEA55C0CF4E0647C9F344400101000000F96A47718EEA55C0649126DE01F64440010100000009A4C4AEEDE955C0A8AB3B16DBF0444001010000004E7D2079E7E955C0B58D3F51D9F044400101000000081F4AB4E4E955C0390EBC5AEEF84440010100000047382D78D1E955C04E29AF95D0ED4440010100000004392861A6E955C089997D1EA3F644400101000000840EBA8443E955C021CA17B490F844400101000000B77C24253DE955C00745F30016F3444001010000003352EFA99CE855C088F6B182DFF244400101000000618C48145AE855C08BC56F0A2BF34440010100000084F4143944E855C0062CB98AC5EF44400101000000529ACDE330E855C06AA2CF4719F144400101000000359886E123E855C07A1D71C806F2444001010000008DEDB5A0F7E755C08693347F4CF144400101000000B91CAF40F4E755C09372F7393EF0444001010000009B1DA9BEF3E755C0B6F81400E3EF44400101000000E28FA2CEDCE755C06A12BC218DF2444001010000004912842BA0E755C033C005D9B2EE444001010000007F6B274A42E655C044DFDDCA12E544400101000000A19FA9D72DE655C07C7BD7A02FE54440010100000085B4C6A013E655C0A37895B54DE54440010100000058552FBFD3E555C0C0E8F2E670E7444001010000004B5645B8C9E555C097E4805D4DE4444001010000002FA52E19C7E555C0D40FEA2285E64440 -(1 row) -~~~ - -## See also - -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial objects](spatial-features.html#spatial-objects) -- [POINT](point.html) -- [LINESTRING](linestring.html) -- [POLYGON](polygon.html) -- [MULTILINESTRING](multilinestring.html) -- [MULTIPOLYGON](multipolygon.html) -- [GEOMETRYCOLLECTION](geometrycollection.html) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/multipolygon.md b/src/current/v21.1/multipolygon.md deleted file mode 100644 index adcbc1de77d..00000000000 --- a/src/current/v21.1/multipolygon.md +++ /dev/null @@ -1,37 +0,0 @@ ---- -title: MULTIPOLYGON -summary: The spatial MULTIPOLYGON object is a collection of POLYGONs. -toc: true ---- - -A `MULTIPOLYGON` is a collection of [Polygons](polygon.html). MultiPolygons are useful for gathering a group of Polygons into one geometry. For example, you may want to gather the Polygons denoting a group of properties in a particular municipality. Another use of MultiPolygons is to represent states or countries that include islands, or that are otherwise made up of non-overlapping shapes. - -{% include {{page.version.version}}/spatial/zmcoords.md %} - -## Examples - -A MultiPolygon can be created from SQL by calling the `st_geomfromtext` function on a MultiPolygon definition expressed in the [Well Known Text (WKT)](spatial-glossary.html#wkt) format. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT ST_GeomFromText('SRID=4326;MULTIPOLYGON(((-87.906471 43.038902, -95.992775 36.153980, -75.704722 36.076944, -87.906471 43.038902), (-87.623177 41.881832, -90.199402 38.627003, -82.446732 38.413651, -87.623177 41.881832), (-84.191605 39.758949, -75.165222 39.952583, -78.878738 42.880230, -84.191605 39.758949)))'); -~~~ - -~~~ - st_geomfromtext ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - 0106000020E610000001000000010300000003000000040000006FF1F09E03FA55C0DFDFA0BDFA844540545227A089FF57C0791EDC9DB513424064B14D2A1AED52C0CCCF0D4DD90942406FF1F09E03FA55C0DFDFA0BDFA84454004000000A4A7C821E2E755C07C48F8DEDFF0444073309B00C38C56C038BF61A241504340E884D041979C54C0967A1684F2344340A4A7C821E2E755C07C48F8DEDFF044400400000001309E41430C55C07C2AA73D25E143401AA54BFF92CA52C0DFDC5F3DEEF9434028F04E3E3DB853C0A27A6B60AB70454001309E41430C55C07C2AA73D25E14340 -(1 row) -~~~ - -## See also - -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial objects](spatial-features.html#spatial-objects) -- [POINT](point.html) -- [LINESTRING](linestring.html) -- [POLYGON](polygon.html) -- [MULTIPOINT](multipoint.html) -- [MULTILINESTRING](multilinestring.html) -- [GEOMETRYCOLLECTION](geometrycollection.html) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/multiregion-overview.md b/src/current/v21.1/multiregion-overview.md deleted file mode 100644 index 8d118d9475f..00000000000 --- a/src/current/v21.1/multiregion-overview.md +++ /dev/null @@ -1,180 +0,0 @@ ---- -title: Multi-region Capabilities Overview -summary: Learn how to use CockroachDB's improved multi-region capabilities. -toc: true -keywords: gin, gin index, gin indexes, inverted index, inverted indexes, accelerated index, accelerated indexes ---- - -## Overview - -{% include_cached new-in.html version="v21.1" %} CockroachDB has improved multi-region capabilities that make it easier to run global applications. - -To take advantage of these improved capabilities, you will need to understand the following concepts: - -- [_Cluster Regions_](#cluster-regions) are geographic regions that a user specifies at node start time. -- [_Database Regions_](#database-regions) are geographic regions that a given database operates within. A database region must be chosen from the list of available cluster regions. -- [_Survival Goals_](#survival-goals), which dictate how many simultaneous failure(s) that a database can survive. -- [_Table Localities_](#table-locality), which tell CockroachDB how to optimize access to a table's data. - - - -At a high level, the simplest process for running a multi-region cluster is: - -1. Set region information for each node in the cluster at startup using [node startup locality options](cockroach-start.html#locality). -2. Add one or more regions to a database, making it a "multi-region" database. One of these regions must be the _primary region_. -3. (*Optional*) Change table localities (global, regional by table, regional by row). This step is optional because by default the tables in a database will be homed in the database's primary region (as set during Step 2). -4. (*Optional*) Change the database's survival goals (zone or region). This step is optional because by default multi-region databases will be configured to survive zone failures. - -The steps above describe the simplest case, where you accept all of the default settings. If your application has performance or availability needs that are different than what the default settings provide, there are customization options to explore. - -For more information about CockroachDB's multi-region capabilities and the customization options that are available, see below. - -{{site.data.alerts.callout_success}} -{% include {{page.version.version}}/misc/multiregion-max-offset.md %} -{{site.data.alerts.end}} - -{% include enterprise-feature.md %} - -## Cluster Regions - -Regions and zones are defined at the node level using the following [node startup locality options](cockroach-start.html#locality): - -- Regions are specified using the `region` key. -- Zones through the use of the `zone` key. - -For example, the command below adds `us-east-1` to the list of cluster regions, and `us-east-1b` to the list of zones: - -{% include_cached copy-clipboard.html %} -~~~ shell -cockroach start --locality=region=us-east-1,zone=us-east-1b # ... other required flags go here -~~~ - -To show all of a cluster's regions, execute the following SQL statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW REGIONS FROM CLUSTER; -~~~ - -## Database Regions - - _Database regions_ are a high-level abstraction for a geographic region. Each region is broken into multiple zones. These terms are meant to correspond directly to the region and zone terminology used by cloud providers. - -The regions added during node startup become _Database Regions_ when they are added to a database. To add the first region, use the [`ALTER DATABASE ... PRIMARY REGION` statement](set-primary-region.html). - -While the database has only one region assigned to it, it is considered a "multi-region database." This means that all data in that database is stored within its assigned regions, and CockroachDB optimizes access to the database's data from the primary region. - -To add another database region, use the [`ALTER DATABASE ... ADD REGION` statement](add-region.html). - -To show all of a database's regions, execute the [`SHOW REGIONS FROM DATABASE` statement](show-regions.html). - -{{site.data.alerts.callout_info}} -If the default _Survival Goals_ and _Table Localities_ meet your needs, there is nothing else you need to do once you have set a database's primary region. -{{site.data.alerts.end}} - -## Survival goals - -All tables within the same database operate with the same survival goal. Each database is allowed to have its own survival goal setting. - -The following survival goals are available: - -- Zone failure -- Region failure - -Surviving zone failures is the default. You can upgrade a database to survive region failures at the cost of slower write performance (due to network hops) using the following statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE SURVIVE REGION FAILURE; -~~~ - -For more information about the survival goals supported by CockroachDB, see the following sections: - -- [Surviving zone failures](#surviving-zone-failures) -- [Surviving region failures](#surviving-region-failures) - -### Surviving zone failures - -With the zone level survival goal, the database will remain fully available for reads and writes, even if a zone goes down. However, the database may not remain fully available if multiple zones fail in the same region. - -You can configure a database to survive zone failures using the [`ALTER DATABASE ... SURVIVE ZONE FAILURE` statement](survive-failure.html). - -{{site.data.alerts.callout_info}} -Surviving zone failures is the default setting for multi-region databases. - -If your application has performance or availability needs that are different than what the default settings provide, you can explore the other customization options described on this page. -{{site.data.alerts.end}} - -### Surviving region failures - -The region level survival goal has the property that the database will remain fully available for reads and writes, even if an entire region goes down. This added survival comes at a cost: write latency will be increased by at least as much as the round-trip time to the nearest region. Read performance will be unaffected. In other words, you are adding network hops and making writes slower in exchange for robustness. - -You can upgrade a database to survive region failures using the [`ALTER DATABASE ... SURVIVE REGION FAILURE` statement](survive-failure.html). - -Setting this goal on a database in a cluster with 3 [cluster regions](#cluster-regions) will automatically increase the [replication factor](configure-replication-zones.html#num_replicas) of the [ranges](architecture/overview.html#architecture-range) underlying the database from 3 (the default) to 5. This ensures that there will be 5 replicas of each range spread across the 3 regions (2+2+1=5). This is how CockroachDB is able to provide region level resiliency while maintaining good read performance in the leaseholder's region. For writes, CockroachDB will need to coordinate across 2 of the 3 regions, so you will pay additional write latency in exchange for the increased resiliency. - -{{site.data.alerts.callout_info}} -In order to survive region failures, you must have added at least 3 [database regions](#database-regions) -{{site.data.alerts.end}} - -## Table locality - -Every table in a multi-region database has a "table locality setting" applied to it. CockroachDB uses the table locality setting to determine how to optimize access to the table's data from that locality. - -By default, all tables in a multi-region database are _regional_ tables -- that is, CockroachDB optimizes access to the table's data from a single region (by default, the database's primary region). - -For more information about the table localities CockroachDB supports, see the following sections: - -- [Regional tables](#regional-tables) provide low-latency reads and writes for an entire table from a single region. -- [Regional by row tables](#regional-by-row-tables) provide low-latency reads and writes for one or more rows of a table from a single region. Different rows in the table can be optimized for access from different regions. -- [Global tables](#global-tables) are optimized for low-latency reads from all regions. - -{{site.data.alerts.callout_success}} -Regardless of table locality, you can access all data in a multi-region database with low latency from any [database region](#database-regions) by using stale [follower reads](follower-reads.html). -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -Table locality settings are used for optimizing latency under different read/write patterns. If you are optimizing for read/write access to all of your tables from a single region (the primary region), there is nothing else you need to do once you set your [database's primary region](#database-regions). -{{site.data.alerts.end}} - -### Regional tables - -{% include {{page.version.version}}/sql/regional-table-description.md %} - -### Regional by row tables - -{% include {{page.version.version}}/sql/regional-by-row-table-description.md %} - -### Global tables - -{% include {{page.version.version}}/sql/global-table-description.md %} - -## Additional Features - -The features listed in this section are designed to make working with multi-region clusters easier. - -### Indexes on `REGIONAL BY ROW` tables - -{% include {{page.version.version}}/sql/indexes-regional-by-row.md %} - -Note that the behavior described above also applies to [GIN indexes](inverted-indexes.html). - -For an example that uses unique indexes but applies to all indexes on `REGIONAL BY ROW` tables, see [Add a unique index to a `REGIONAL BY ROW` table](add-constraint.html#add-a-unique-index-to-a-regional-by-row-table). - -## Schema changes in multi-region clusters - -{% include {{ page.version.version }}/performance/lease-preference-system-database.md %} - -## Next steps - -+ [Choosing a multi-region configuration](choosing-a-multi-region-configuration.html) - -## See also - -- [When to use `ZONE` vs. `REGION` survival goals](when-to-use-zone-vs-region-survival-goals.html) -- [When to use `REGIONAL` vs. `GLOBAL` tables](when-to-use-regional-vs-global-tables.html) -- [Topology Patterns](topology-patterns.html) -- [Disaster Recovery](disaster-recovery.html) -- [Develop and Deploy a Global Application](movr-flask-overview.html) -- [Multi-region SQL performance](demo-low-latency-multi-region-deployment.html) -- [Migrate to Multi-region SQL](migrate-to-multiregion-sql.html) diff --git a/src/current/v21.1/multiregion-scale-application.md b/src/current/v21.1/multiregion-scale-application.md deleted file mode 100644 index 4b49c67cf7f..00000000000 --- a/src/current/v21.1/multiregion-scale-application.md +++ /dev/null @@ -1,149 +0,0 @@ ---- -title: Scale to Multiple Regions -summary: Learn how to scale a single-region application to multiple regions. -toc: true ---- - -This page provides guidance for scaling a single-region application to multiple regions. - -Before reading this page, we recommend reviewing [CockroachDB's multi-region capabilities](multiregion-overview.html). - -## Overview - -Scaling an application from a single region to multiple regions consists of: - -- [Scaling the database](#scale-the-database), which includes adding new nodes to a CockroachDB cluster in different regions, adding regions to the database schema, and optionally transforming the database schema to leverage multi-region table localities. - -- [Scaling the application](#scale-the-application), which includes deploying the application in new regions and, if necessary, updating the application code to work with the multi-region database schema. - -## Scale the database - -### Step 1. Prep the database - -Use an [`ALTER DATABASE ... SET PRIMARY REGION`](set-primary-region.html) statement to set the database's [primary region](multiregion-overview.html#database-regions) to a region in which the cluster is deployed. This region must have been specified as a [regional locality](cockroach-start.html#locality) at cluster startup. - -Setting the primary region before adding new regional nodes to the cluster prevents CockroachDB from [rebalancing row replications](architecture/replication-layer.html#leaseholder-rebalancing) across all regions each time a node is added in a new region. - -{{site.data.alerts.callout_info}} -Executing `ALTER` statements performs a [schema migration](online-schema-changes.html) on the cluster. If you are using a schema migration tool, you will need to execute these statements as raw SQL, as the [multi-region SQL syntax](multiregion-overview.html) is specific to CockroachDB. - -Here are some simple tutorials on executing schema migrations against CockroachDB clusters: - -- [Migrate CockroachDB Schemas with Liquibase](liquibase.html) -- [Migrate CockroachDB Schemas with Flyway](flyway.html) -- [Migrate CockroachDB Schemas with Alembic](alembic.html) -- [Execute SQL statements from a file](cockroach-sql.html#execute-sql-statements-from-a-file) and [Change and Remove Objects in a Database Schema](schema-design-update.html) -{{site.data.alerts.end}} - -### Step 2. Scale the cluster deployment - -Scale the cluster by adding nodes to the cluster in new regions. - -For instructions on adding nodes to an existing cluster, see one of the following pages: - -- For managed CockroachDB {{ site.data.products.dedicated }} deployments, see [Cluster Management](../cockroachcloud/cluster-management.html). -- For orchestrated deployments, see [Orchestrate CockroachDB Across Multiple Kubernetes Clusters](orchestrate-cockroachdb-with-kubernetes-multi-cluster.html). -- For manual deployments, see [`cockroach start`](cockroach-start.html) and [Manual Deployment](manual-deployment.html). - -{{site.data.alerts.callout_info}} -For orchestrated and manual deployments, you must specify a [regional locality](cockroach-start.html#locality) for each node at startup. These regional localities are represented as [cluster regions](multiregion-overview.html#cluster-regions) in the cluster. -{{site.data.alerts.end}} - -### Step 3. Scale the database schema - -Use an [`ALTER DATABASE ... ADD REGIONS`](add-region.html) statement to add the new regions to your database. Only cluster regions (i.e., regional localities specified at cluster startup) can be added as [database regions](multiregion-overview.html#database-regions). - -After you add new regions to the database schema, you can optionally configure the [survival goals](multiregion-overview.html#survival-goals) and [table localities](multiregion-overview.html#table-locality) of the multi-region database: - -- Add [`ALTER DATABASE ... SURVIVE ... FAILURE`](survive-failure.html) statements to set your database's [survival goals](multiregion-overview.html#survival-goals). - -- Add [`ALTER TABLE ... SET LOCALITY`](set-locality.html) statements to set [table localities](multiregion-overview.html#table-locality) for each table. - -## Scale the application - -### Step 1. Scale application deployments - -Scaling application deployments in multiple regions can greatly improve latency for the end-user of the application. - -For guidance on connecting to CockroachDB from an application deployment, see one of the following pages: - -- For connecting to CockroachDB {{ site.data.products.dedicated }} deployments, see [Connect to Your CockroachDB {{ site.data.products.dedicated }} Cluster](../cockroachcloud/connect-to-your-cluster.html) and [Connect to the Database (CockroachDB {{ site.data.products.dedicated }})](connect-to-the-database-cockroachcloud.html). -- For connecting to a standard CockroachDB deployment, see [`cockroach sql`](cockroach-sql.html) and [Connect to the Database](connect-to-the-database.html). - -To limit the latency between the application and the database, each deployment of the application should communicate with the closest database deployment. For details on configuring database connections for individual application deployments, consult your cloud provider's documentation. For an example using Google Cloud services, see [Multi-region Application Deployment](movr-flask-deployment.html). - -{{site.data.alerts.callout_info}} -A multi-region application deployment does not require a multi-region database deployment. Deploying a global application in multiple regions can yield significant latency benefits for the end user, even if you have not yet scaled your database in multiple regions. For an example, see [Reducing Multi-Region Latency with Follower Reads](https://www.cockroachlabs.com/blog/follower-reads/#:~:text=Deployment%202%3A%20Global%20Application%20Deployment%2C%20No%20Follower%20reads). - -If you do scale the application first, make sure that you reconfigure each application deployment to communicate with the closest database deployment after deploying the database in multiple regions. -{{site.data.alerts.end}} - -### Step 2. *(Optional)* Update the application code for multi-region - -For most table localities, including the default locality `LOCALITY REGIONAL BY TABLE IN PRIMARY REGION`, *you do not need to update your application code after migrating your database schema for multi-region*. CockroachDB automatically optimizes queries against multi-region databases, based on the regional locality of the node executing the query, and on the multi-region configuration of the database. For more details, see [Regional Tables](regional-tables.html#regional-by-row-tables). For an extended example, see [Develop and Deploy a Global Application: Create a Multi-region Database Schema](movr-flask-database.html). - -However, there are some scenarios in which you might need to update the SQL operations in your application. For example: - -- If a table has a `REGIONAL BY ROW AS ` table locality, and you want to explicitly insert regional values into a table, as shown in [Low Latency Reads and Writes in a Multi-Region Cluster](demo-low-latency-multi-region-deployment.html#configure-regional-by-row-tables). -- If a table has a `REGIONAL BY ROW` locality, and you want to update the `crdb_region` value of existing rows in the table based on some other column value, as shown in [Set the table locality to `REGIONAL BY ROW`](set-locality.html#set-the-table-locality-to-regional-by-row). -- If a table has a `REGIONAL BY ROW` locality, and you want to filter a [selection query](select-clause.html#filter-rows) based on the `crdb_region` value. - -In all of these scenarios, statements reference the column that tracks the region for each row in a `REGIONAL BY ROW` locality. This column can be a custom column of the built-in `ENUM` type `crdb_internal_region`, or it can be the default, hidden [`crdb_region` column](set-locality.html#crdb_region). - -If you need to explicitly reference the region-tracking column in a SQL operation in your application code, you should do the following: - -- Verify that the region-tracking column is visible to the ORM. - - To make a hidden column visible, use an [`ALTER TABLE ... ALTER COLUMN ... SET VISIBLE` statement](alter-column.html). By default, the `crdb_region` column created by CockroachDB is hidden. -- Using your ORM framework, sync the mapping objects in your application to reflect the latest database schema with the region-tracking column(s). -- Reference the region-tracking column in read/write operations as needed. - -For example, suppose that you have a single-region table called `users` that has just been transformed into a multi-region table with a `REGIONAL BY ROW` locality. When the application was first deployed, this table had no region-tracking column. During the multi-region database schema transformation, CockroachDB automatically created a hidden `crdb_region` column to track the region of each row. - -In the absence of an explicit, back-filling computed column for the hidden `crdb_region` column, there is no way for CockroachDB to determine the region for old rows of data. The following steps update the `crdb_region` values in rows that were inserted before the multi-region transformation, based on the values of a `city` column: - -1. Make `crdb_region` visible in the relevant `REGIONAL BY ROW` table(s): - - {% include_cached copy-clipboard.html %} - ~~~ sql - ALTER TABLE users ALTER COLUMN crdb_region SET VISIBLE; - ~~~ - -1. Update the table mappings in the application code (written in Python, with [SQLAlchemy](https://www.sqlalchemy.org/)): - - {% include_cached copy-clipboard.html %} - ~~~ python - from models import Base - - ... - - Base.metadata.reflect(bind=self.engine, extend_existing=True, autoload_replace=True) - ~~~ - - {{site.data.alerts.callout_info}} - SQLAlchemy allows you to update all table mappings to reflect the database with the `sqlalchemy.schema.MetaData` class method [`reflect()`](https://docs.sqlalchemy.org/en/14/core/metadata.html#sqlalchemy.schema.MetaData.reflect). If your ORM framework does not support updating mapping objects dynamically, you might need to add the column to the table-mapping class definition as a `String`-typed column and reinstantiate the object. - {{site.data.alerts.end}} - -1. Reference the column value as needed. - - Here is an example function that updates the region value in a given table, using the values of the `city` column: - - {% include_cached copy-clipboard.html %} - ~~~ python - from sqlalchemy_cockroachdb import run_transaction - ... - - def update_region(engine, table, region, cities): - - def update_region_helper(session, table, region, cities): - query = table.update().where(column('city').in_(cities)).values({'crdb_region': region}) - session.execute(query) - - run_transaction(sessionmaker(bind=engine), - lambda session: update_region_helper(session, table, region, cities)) - ~~~ - -## See also - -- [Multi-region Capabilities Overview](multiregion-overview.html) -- [Low Latency Reads and Writes in a Multi-Region Cluster](demo-low-latency-multi-region-deployment.html) diff --git a/src/current/v21.1/not-null.md b/src/current/v21.1/not-null.md deleted file mode 100644 index ae30563aec4..00000000000 --- a/src/current/v21.1/not-null.md +++ /dev/null @@ -1,81 +0,0 @@ ---- -title: NOT NULL constraint -summary: The NOT NULL constraint specifies the column may not contain NULL values. -toc: true ---- - -The `NOT NULL` [constraint](constraints.html) specifies a column may not contain [`NULL`](null-handling.html) values. - -## Details - -- `INSERT` or `UPDATE` statements containing `NULL` values are rejected. This includes `INSERT` statements that do not include values for any columns that do not have a [`DEFAULT` value constraint](default-value.html). - - For example, if the table `foo` has columns `a` and `b` (and `b` *does not* have a `DEFAULT VALUE`), when you run the following command: - - ~~~ sql - > INSERT INTO foo (a) VALUES (1); - ~~~ - - CockroachDB tries to write a `NULL` value into column `b`. If that column has the `NOT NULL` constraint, the `INSERT` statement is rejected. - -- To add the `NOT NULL` constraint to an existing table column, use the [`ALTER COLUMN`](alter-column.html#set-not-null-constraint) statement. - -- For more information about `NULL`, see [NULL handling](null-handling.html). - -## Syntax - -You can only apply the `NOT NULL` constraint to individual columns. - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/not_null_column_level.html %} -
    - - Parameter | Description ------------|------------- - `table_name` | The name of the table you're creating. - `column_name` | The name of the constrained column. - `column_type` | The constrained column's [data type](data-types.html). - `column_constraints` | Any other column-level [constraints](constraints.html) you want to apply to this column. - `column_def` | Definitions for any other columns in the table. - `table_constraints` | Any table-level [constraints](constraints.html) you want to apply. - -## Usage example - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE IF NOT EXISTS customers ( - customer_id INT PRIMARY KEY, - cust_name STRING(30) NULL, - cust_email STRING(100) NOT NULL - ); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO customers (customer_id, cust_name, cust_email) VALUES (1, 'Smith', NULL); -~~~ - -~~~ -pq: null value in column "cust_email" violates not-null constraint -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO customers (customer_id, cust_name) VALUES (1, 'Smith'); -~~~ - -~~~ -pq: null value in column "cust_email" violates not-null constraint -~~~ - -## See also - -- [Constraints](constraints.html) -- [`DROP CONSTRAINT`](drop-constraint.html) -- [`CHECK` constraint](check.html) -- [`DEFAULT` constraint](default-value.html) -- [`REFERENCES` constraint (Foreign Key)](foreign-key.html) -- [`PRIMARY KEY` constraint](primary-key.html) -- [`UNIQUE` constraint](unique.html) -- [`SHOW CONSTRAINTS`](show-constraints.html) -- [`NULL HANDLING`](null-handling.html) diff --git a/src/current/v21.1/null-handling.md b/src/current/v21.1/null-handling.md deleted file mode 100644 index 12f615becd2..00000000000 --- a/src/current/v21.1/null-handling.md +++ /dev/null @@ -1,555 +0,0 @@ ---- -title: NULL Handling -summary: Learn how NULL values are handled in CockroachDB SQL. -toc: true ---- - -`NULL` is the term used to represent a missing value. A `NULL` value in a table is a value in a field that appears to be blank. A field with a `NULL` value is a field with no value. - -This page summarizes how `NULL` values are handled in CockroachDB SQL. Each topic is demonstrated via the [built-in SQL client](cockroach-sql.html). - -When using the built-in client, `NULL` values are displayed using the word `NULL`. This distinguishes them from a character field that contains an empty string (`""`). - -## NULLs and simple comparisons - -Any simple comparison between a value and `NULL` results in -`NULL`. The remaining cases are described in the next section. - -This behavior is consistent with PostgreSQL as well as all other major RDBMS's. - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO customers (customer_id, cust_name, cust_email) VALUES (1, 'Smith', NULL); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE t1( - a INT, - b INT, - c INT -); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO t1 VALUES(1, 0, 0); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO t1 VALUES(2, 0, 1); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO t1 VALUES(3, 1, 0); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO t1 VALUES(4, 1, 1); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO t1 VALUES(5, NULL, 0); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO t1 VALUES(6, NULL, 1); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO t1 VALUES(7, NULL, NULL); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM t1; -~~~ - -~~~ -+---+------+------+ -| a | b | c | -+---+------+------+ -| 1 | 0 | 0 | -| 2 | 0 | 1 | -| 3 | 1 | 0 | -| 4 | 1 | 1 | -| 5 | NULL | 0 | -| 6 | NULL | 1 | -| 7 | NULL | NULL | -+---+------+------+ -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM t1 WHERE b < 10; -~~~ - -~~~ -+---+---+---+ -| a | b | c | -+---+---+---+ -| 1 | 0 | 0 | -| 2 | 0 | 1 | -| 3 | 1 | 0 | -| 4 | 1 | 1 | -+---+---+---+ -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM t1 WHERE NOT b > 10; -~~~ - -~~~ -+---+---+---+ -| a | b | c | -+---+---+---+ -| 1 | 0 | 0 | -| 2 | 0 | 1 | -| 3 | 1 | 0 | -| 4 | 1 | 1 | -+---+---+---+ -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM t1 WHERE b < 10 OR c = 1; -~~~ - -~~~ -+---+------+---+ -| a | b | c | -+---+------+---+ -| 1 | 0 | 0 | -| 2 | 0 | 1 | -| 3 | 1 | 0 | -| 4 | 1 | 1 | -| 6 | NULL | 1 | -+---+------+---+ -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM t1 WHERE b < 10 AND c = 1; -~~~ - -~~~ -+---+---+---+ -| a | b | c | -+---+---+---+ -| 2 | 0 | 1 | -| 4 | 1 | 1 | -+---+---+---+ -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM t1 WHERE NOT (b < 10 AND c = 1); -~~~ - -~~~ -+---+------+---+ -| a | b | c | -+---+------+---+ -| 1 | 0 | 0 | -| 3 | 1 | 0 | -| 5 | NULL | 0 | -+---+------+---+ -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM t1 WHERE NOT (c = 1 AND b < 10); -~~~ - -~~~ -+---+------+---+ -| a | b | c | -+---+------+---+ -| 1 | 0 | 0 | -| 3 | 1 | 0 | -| 5 | NULL | 0 | -+---+------+---+ -~~~ - -Use the `IS NULL` or `IS NOT NULL` clauses when checking for `NULL` values. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM t1 WHERE b IS NULL AND c IS NOT NULL; -~~~ - -~~~ -+---+------+---+ -| a | b | c | -+---+------+---+ -| 5 | NULL | 0 | -| 6 | NULL | 1 | -+---+------+---+ -~~~ - -## NULLs and conditional operators - -The [conditional -operators](scalar-expressions.html#conditional-expressions) -(including `IF`, `COALESCE`, `IFNULL`) only evaluate some -operands depending on the value of a condition operand, so their -result is not always `NULL` depending on the given operands. - -For example, `COALESCE(1, NULL)` will always return `1` even though -the second operand is `NULL`. - -## NULLs and ternary logic - -`AND`, `OR` and `IS` implement ternary logic, as follows. - -| Expression | Result | --------------------|--------- - `FALSE AND FALSE` | `FALSE` - `FALSE AND TRUE` | `FALSE` - `FALSE AND NULL` | `FALSE` - `TRUE AND FALSE` | `FALSE` - `TRUE AND TRUE` | `TRUE` - `TRUE AND NULL` | `NULL` - `NULL AND FALSE` | `FALSE` - `NULL AND TRUE` | `NULL` - `NULL AND NULL` | `NULL` - -| Expression | Result | -------------------|--------- - `FALSE OR FALSE` | `FALSE` - `FALSE OR TRUE` | `TRUE` - `FALSE OR NULL` | `NULL` - `TRUE OR FALSE` | `TRUE` - `TRUE OR TRUE` | `TRUE` - `TRUE OR NULL` | `TRUE` - `NULL OR FALSE` | `NULL` - `NULL OR TRUE` | `TRUE` - `NULL OR NULL` | `NULL` - -| Expression | Result | -------------------|--------- - `FALSE IS FALSE` | `TRUE` - `FALSE IS TRUE` | `FALSE` - `FALSE IS NULL` | `FALSE` - `TRUE IS FALSE` | `FALSE` - `TRUE IS TRUE` | `TRUE` - `TRUE IS NULL` | `FALSE` - `NULL IS FALSE` | `FALSE` - `NULL IS TRUE` | `FALSE` - `NULL IS NULL` | `TRUE` - -## NULLs and arithmetic - -Arithmetic operations involving a `NULL` value will yield a `NULL` result. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT a, b, c, b*0, b*c, b+c FROM t1; -~~~ - -~~~ -+---+------+------+-------+-------+-------+ -| a | b | c | b * 0 | b * c | b + c | -+---+------+------+-------+-------+-------+ -| 1 | 0 | 0 | 0 | 0 | 0 | -| 2 | 0 | 1 | 0 | 0 | 1 | -| 3 | 1 | 0 | 0 | 0 | 1 | -| 4 | 1 | 1 | 0 | 1 | 2 | -| 5 | NULL | 0 | NULL | NULL | NULL | -| 6 | NULL | 1 | NULL | NULL | NULL | -| 7 | NULL | NULL | NULL | NULL | NULL | -+---+------+------+-------+-------+-------+ -~~~ - -## NULLs and aggregate functions - -Aggregate [functions](functions-and-operators.html) are those that operate on a set of rows and return a single value. The example data has been repeated here to make it easier to understand the results. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM t1; -~~~ - -~~~ -+---+------+------+ -| a | b | c | -+---+------+------+ -| 1 | 0 | 0 | -| 2 | 0 | 1 | -| 3 | 1 | 0 | -| 4 | 1 | 1 | -| 5 | NULL | 0 | -| 6 | NULL | 1 | -| 7 | NULL | NULL | -+---+------+------+ -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT COUNT(*), COUNT(b), SUM(b), AVG(b), MIN(b), MAX(b) FROM t1; -~~~ - -~~~ -+----------+----------+--------+--------------------+--------+--------+ -| COUNT(*) | COUNT(b) | SUM(b) | AVG(b) | MIN(b) | MAX(b) | -+----------+----------+--------+--------------------+--------+--------+ -| 7 | 4 | 2 | 0.5000000000000000 | 0 | 1 | -+----------+----------+--------+--------------------+--------+--------+ -~~~ - -Note the following: - -- `NULL` values are not included in the `COUNT()` of a column. `COUNT(*)` returns 7 while `COUNT(b)` returns 4. - -- `NULL` values are not considered as high or low values in `MIN()` or `MAX()`. - -- `AVG(b)` returns `SUM(b)/COUNT(b)`, which is different than `AVG(*)` as `NULL` values are not considered in the `COUNT(b)` of rows. See [NULLs as Other Values](#nulls-as-other-values) for more details. - - -## NULL as a distinct value - -`NULL` values are considered distinct from other values and are included in the list of distinct values from a column. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT DISTINCT b FROM t1; -~~~ - -~~~ -+------+ -| b | -+------+ -| 0 | -| 1 | -| NULL | -+------+ -~~~ - -However, counting the number of distinct values excludes `NULL`s, which is consistent with the `COUNT()` function. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT COUNT(DISTINCT b) FROM t1; -~~~ - -~~~ -+-------------------+ -| count(DISTINCT b) | -+-------------------+ -| 2 | -+-------------------+ -~~~ - -## NULLs as other values - -In some cases, you may want to include `NULL` values in arithmetic or aggregate function calculations. To do so, use the `IFNULL()` function to substitute a value for `NULL` during calculations. - -For example, let's say you want to calculate the average value of column `b` as being the `SUM()` of all numbers in `b` divided by the total number of rows, regardless of whether `b`'s value is `NULL`. In this case, you would use `AVG(IFNULL(b, 0))`, where `IFNULL(b, 0)` substitutes a value of zero (0) for `NULL`s during the calculation. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT COUNT(*), COUNT(b), SUM(b), AVG(b), AVG(IFNULL(b, 0)), MIN(b), MAX(b) FROM t1; -~~~ - -~~~ -+----------+----------+--------+--------------------+--------------------+--------+--------+ -| COUNT(*) | COUNT(b) | SUM(b) | AVG(b) | AVG(IFNULL(b, 0)) | MIN(b) | MAX(b) | -+----------+----------+--------+--------------------+--------------------+--------+--------+ -| 7 | 4 | 2 | 0.5000000000000000 | 0.2857142857142857 | 0 | 1 | -+----------+----------+--------+--------------------+--------------------+--------+--------+ -~~~ - -## NULLs and set operations - -`NULL` values are considered as part of a `UNION` [set operation](selection-queries.html#set-operations). - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT b FROM t1 UNION SELECT b FROM t1; -~~~ - -~~~ -+------+ -| b | -+------+ -| 0 | -| 1 | -| NULL | -+------+ -~~~ - - -## NULLs and sorting - -When [sorting a column](order-by.html) containing `NULL` values, CockroachDB sorts `NULL` values first with `ASC` and last with `DESC`. This differs from PostgreSQL, which sorts `NULL` values last with `ASC` and first with `DESC`. - -CockroachDB supports `NULLS FIRST` and `NULLS LAST` in [`ORDER BY`](order-by.html) clauses. However, in some cases the support is syntax-only—an error is returned if `NULLS FIRST` or `NULLS LAST` specification doesn't "match" the internal structure of the used index. If the index is ascending, the error is returned for `NULLS LAST`; if the index is descending, the error is returned for `NULLS FIRST`. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM t1 ORDER BY b ASC; -~~~ - -~~~ -+---+------+------+ -| a | b | c | -+---+------+------+ -| 6 | NULL | 1 | -| 5 | NULL | 0 | -| 7 | NULL | NULL | -| 1 | 0 | 0 | -| 2 | 0 | 1 | -| 4 | 1 | 1 | -| 3 | 1 | 0 | -+---+------+------+ -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM t1 ORDER BY b DESC; -~~~ - -~~~ -+---+------+------+ -| a | b | c | -+---+------+------+ -| 4 | 1 | 1 | -| 3 | 1 | 0 | -| 2 | 0 | 1 | -| 1 | 0 | 0 | -| 7 | NULL | NULL | -| 6 | NULL | 1 | -| 5 | NULL | 0 | -+---+------+------+ -~~~ - -## NULLs and unique constraints - -`NULL` values are not considered unique. Therefore, if a table has a Unique constraint on one or more columns that are optional (nullable), it is possible to insert multiple rows with `NULL` values in those columns, as shown in the example below. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE t2(a INT, b INT UNIQUE); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO t2 VALUES(1, 1); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO t2 VALUES(2, NULL); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO t2 VALUES(3, NULL); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM t2; -~~~ - -~~~ -+---+------+ -| a | b | -+---+------+ -| 1 | 1 | -| 2 | NULL | -| 3 | NULL | -+---+------+ -~~~ - -## NULLs and CHECK Constraints - -A [`CHECK` constraint](check.html) expression that evaluates to `NULL` is considered to pass, allowing for concise expressions like `discount < price` without worrying about adding `OR discount IS NULL` clauses. When non-null validation is desired, the usual `NOT NULL` constraint can be used along side a Check constraint. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE products (id STRING PRIMARY KEY, price INT NOT NULL CHECK (price > 0), discount INT, CHECK (discount <= price)); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO products (id, price) VALUES ('ncc-1701-d', 100); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO products (id, price, discount) VALUES ('ncc-1701-a', 100, 50); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM products; -~~~ - -~~~ -+----------+-------+----------+ -| id | price | discount | -+----------+-------+----------+ -| ncc1701a | 100 | 50 | -| ncc1701d | 100 | NULL | -+----------+-------+----------+ -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO products (id, price) VALUES ('ncc-1701-b', -5); -~~~ - -~~~ -failed to satisfy CHECK constraint (price > 0) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO products (id, price, discount) VALUES ('ncc-1701-b', 100, 150); -~~~ - -~~~ -failed to satisfy CHECK constraint (discount <= price) -~~~ - -## NULLs and concatenation with other types - -{% include_cached new-in.html version="v21.1" %} Concatenation between a non-`NULL` value and a `NULL` value results in a `NULL` value. - -{{site.data.alerts.callout_info}} -In CockroachDB v20.2 and earlier, for all values other than [`STRING`](string.html), concatenation between a non-`NULL` value and a `NULL` value results in an [`ARRAY`](array.html) of the non-`NULL` value's type. To return an `ARRAY` of a specific type from a `NULL` concatenation in CockroachDB v21.1 and later, [cast](data-types.html#data-type-conversions-and-casts) the `NULL` value to an `ARRAY`. -{{site.data.alerts.end}} - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT NULL || 1; -~~~ - -~~~ - ?column? ------------- - NULL -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT NULL || `item`; -~~~ - -~~~ - ?column? ------------- - NULL -(1 row) -~~~ diff --git a/src/current/v21.1/online-schema-changes.md b/src/current/v21.1/online-schema-changes.md deleted file mode 100644 index c9294a5d8a1..00000000000 --- a/src/current/v21.1/online-schema-changes.md +++ /dev/null @@ -1,266 +0,0 @@ ---- -title: Online Schema Changes -summary: Update table schemas without external tools or downtime. -toc: true ---- - -CockroachDB's online schema changes provide a simple way to update a table schema without imposing any negative consequences on an application — including downtime. The schema change engine is a built-in feature requiring no additional tools, resources, or ad hoc sequencing of operations. - -Benefits of online schema changes include: - -- Changes to your table schema happen while the database is running. -- The schema change runs as a [background job][show-jobs] without holding locks on the underlying table data. -- Your application's queries can run normally, with no effect on read/write latency. The schema is cached for performance. -- Your data is kept in a safe, [consistent][consistent] state throughout the entire schema change process. - -{{site.data.alerts.callout_danger}} -Schema changes consume additional resources, and if they are run when the cluster is near peak capacity, latency spikes can occur. This is especially true for any schema change that adds columns, drops columns, or adds an index. We do not recommend doing more than one schema change at a time while in production. -{{site.data.alerts.end}} - -{{site.data.alerts.callout_success}} -Support for schema changes within [transactions][txns] is [limited](#limitations). We recommend doing schema changes outside transactions where possible. When a schema management tool uses transactions on your behalf, we recommend only doing one schema change operation per transaction. -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -You cannot start an online schema change on a table if a [primary key change](alter-primary-key.html) is currently in progress on the same table. -{{site.data.alerts.end}} - -## How online schema changes work - -At a high level, online schema changes are accomplished by using a bridging strategy involving concurrent uses of multiple versions of the schema. The process is as follows: - -1. A user initiates a schema change by executing [`ALTER TABLE`][alter-table], [`CREATE INDEX`][create-index], [`TRUNCATE`][truncate], etc. - -2. The schema change engine converts the original schema to the new schema in discrete steps while ensuring that the underlying table data is always in a consistent state. These changes are executed as a [background job][show-jobs], and can be [paused](pause-job.html), [resumed](resume-job.html), and [canceled](cancel-job.html). - -This approach allows the schema change engine to roll out a new schema while the previous version is still in use. It then backfills or deletes the underlying table data as needed in the background, while the cluster is still running and servicing reads and writes from your application. - -During the backfilling process, the schema change engine updates the underlying table data to make sure all instances of the table are stored according to the requirements of the new schema. - -Once backfilling is complete, all nodes will switch over to the new schema, and will allow reads and writes of the table using the new schema. - -For more technical details, see [How online schema changes are possible in CockroachDB][blog]. - -{{site.data.alerts.callout_info}} -If a schema change fails, the schema change job will be cleaned up automatically. However, there are limitations with rolling back schema changes within a transaction; for more information, [see below](#schema-change-ddl-statements-inside-a-multi-statement-transaction-can-fail-while-other-statements-succeed). -{{site.data.alerts.end}} - -## Best practices for online schema changes - -### Schema changes in multi-region clusters - -{% include {{ page.version.version }}/performance/lease-preference-system-database.md %} - -## Examples - -{{site.data.alerts.callout_success}} -For more examples of schema change statements, see the [`ALTER TABLE`][alter-table] subcommands. -{{site.data.alerts.end}} - -### Run schema changes inside a transaction with `CREATE TABLE` - -As noted in [Limitations](#limitations), you cannot run schema changes inside transactions in general. - -However, as of version v2.1, you can run schema changes inside the same transaction as a [`CREATE TABLE`][create-table] statement. For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; - SAVEPOINT cockroach_restart; - CREATE TABLE fruits ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - name STRING, - color STRING - ); - INSERT INTO fruits (name, color) VALUES ('apple', 'red'); - ALTER TABLE fruits ADD COLUMN inventory_count INTEGER DEFAULT 5; - ALTER TABLE fruits ADD CONSTRAINT name CHECK (name IN ('apple', 'banana', 'orange')); - SELECT name, color, inventory_count FROM fruits; - RELEASE SAVEPOINT cockroach_restart; - COMMIT; -~~~ - -The transaction succeeds with the following output: - -~~~ -BEGIN -SAVEPOINT -CREATE TABLE -INSERT 0 1 -ALTER TABLE -ALTER TABLE -+-------+-------+-----------------+ -| name | color | inventory_count | -+-------+-------+-----------------+ -| apple | red | 5 | -+-------+-------+-----------------+ -(1 row) -COMMIT -COMMIT -~~~ - -### Run multiple schema changes in a single `ALTER TABLE` statement - -As of v19.1, some schema changes can be used in combination in a single `ALTER TABLE` statement. For a list of commands that can be combined, see [`ALTER TABLE`](alter-table.html). For a demonstration, see [Add and rename columns atomically](rename-column.html#add-and-rename-columns-atomically). - -### Show all schema change jobs - -You can check on the status of the schema change jobs on your system at any time using the [`SHOW JOBS`][show-jobs] statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW JOBS] WHERE job_type = 'SCHEMA CHANGE'; -~~~ - -~~~ -+--------------------+---------------+-----------------------------------------------------------------------------+-----------+-----------+----------------------------+----------------------------+----------------------------+----------------------------+--------------------+-------+----------------+ -| job_id | job_type | description | user_name | status | created | started | finished | modified | fraction_completed | error | coordinator_id | -|--------------------+---------------+-----------------------------------------------------------------------------+-----------+-----------+----------------------------+----------------------------+----------------------------+----------------------------+--------------------+-------+----------------| -| 368863345707909121 | SCHEMA CHANGE | ALTER TABLE test.public.fruits ADD COLUMN inventory_count INTEGER DEFAULT 5 | root | succeeded | 2018-07-26 20:55:59.698793 | 2018-07-26 20:55:59.739032 | 2018-07-26 20:55:59.816007 | 2018-07-26 20:55:59.816008 | 1 | | NULL | -| 370556465994989569 | SCHEMA CHANGE | ALTER TABLE test.public.foo ADD COLUMN bar VARCHAR | root | pending | 2018-08-01 20:27:38.708813 | NULL | NULL | 2018-08-01 20:27:38.708813 | 0 | | NULL | -| 370556522386751489 | SCHEMA CHANGE | ALTER TABLE test.public.foo ADD COLUMN bar VARCHAR | root | pending | 2018-08-01 20:27:55.830832 | NULL | NULL | 2018-08-01 20:27:55.830832 | 0 | | NULL | -+--------------------+---------------+-----------------------------------------------------------------------------+-----------+-----------+----------------------------+----------------------------+----------------------------+----------------------------+--------------------+-------+----------------+ -(1 row) -~~~ - -All schema change jobs can be [paused](pause-job.html), [resumed](resume-job.html), and [canceled](cancel-job.html). - -## Undoing a schema change - -Prior to [garbage collection](architecture/storage-layer.html#garbage-collection), it's possible to recover data that may have been lost prior to schema changes by using the [`AS OF SYSTEM TIME`](as-of-system-time.html) parameter. However, this solution is limited in terms of time, and doesn't work beyond the designated garbage collection window. - -For more long-term recovery solutions, consider taking either a [full or incremental backup](take-full-and-incremental-backups.html) of your cluster. - -## Limitations - -### Overview - -Schema changes keep your data consistent at all times, but they do not run inside [transactions][txns] in the general case. This is necessary so the cluster can remain online and continue to service application reads and writes. - -Specifically, this behavior is necessary because making schema changes transactional would mean requiring a given schema change to propagate across all the nodes of a cluster. This would block all user-initiated transactions being run by your application, since the schema change would have to commit before any other transactions could make progress. This would prevent the cluster from servicing reads and writes during the schema change, requiring application downtime. - -### Limited support for schema changes within transactions - -{% include {{ page.version.version }}/known-limitations/schema-changes-within-transactions.md %} - -### Schema change DDL statements inside a multi-statement transaction can fail while other statements succeed - -{% include {{ page.version.version }}/known-limitations/schema-change-ddl-inside-multi-statement-transactions.md %} - -### No schema changes between executions of prepared statements - -{% include {{ page.version.version }}/known-limitations/schema-changes-between-prepared-statements.md %} - -### Examples of statements that fail - -The following statements fail due to [limited support for schema changes within transactions](#limited-support-for-schema-changes-within-transactions). - -#### Create an index and then run a select against that index inside a transaction - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE foo (id INT PRIMARY KEY, name VARCHAR); - BEGIN; - SAVEPOINT cockroach_restart; - CREATE INDEX foo_idx ON foo (id, name); - SELECT * from foo@foo_idx; - RELEASE SAVEPOINT cockroach_restart; - COMMIT; -~~~ - -~~~ -CREATE TABLE -BEGIN -SAVEPOINT -CREATE INDEX -ERROR: relation "foo_idx" does not exist -ERROR: current transaction is aborted, commands ignored until end of transaction block -ROLLBACK -~~~ - -#### Add a column and then add a constraint against that column inside a transaction - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE foo (); - BEGIN; - SAVEPOINT cockroach_restart; - ALTER TABLE foo ADD COLUMN bar VARCHAR; - ALTER TABLE foo ADD CONSTRAINT bar CHECK (foo IN ('a', 'b', 'c', 'd')); - RELEASE SAVEPOINT cockroach_restart; - COMMIT; -~~~ - -~~~ -CREATE TABLE -BEGIN -SAVEPOINT -ALTER TABLE -ERROR: column "foo" not found for constraint "foo" -ERROR: current transaction is aborted, commands ignored until end of transaction block -ROLLBACK -~~~ - -#### Add a column and then select against that column inside a transaction - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE foo (); - BEGIN; - SAVEPOINT cockroach_restart; - ALTER TABLE foo ADD COLUMN bar VARCHAR; - SELECT bar FROM foo; - RELEASE SAVEPOINT cockroach_restart; - COMMIT; -~~~ - -~~~ -CREATE TABLE -BEGIN -SAVEPOINT -ALTER TABLE -ERROR: column name "bar" not found -ERROR: current transaction is aborted, commands ignored until end of transaction block -ROLLBACK -~~~ - -### `ALTER TYPE` schema changes cannot be cancelled - -You can only [cancel](cancel-job.html) [`ALTER TYPE`](alter-type.html) schema change jobs that drop values. All other `ALTER TYPE` schema change jobs are non-cancellable. - -## See also - -+ [How online schema changes are possible in CockroachDB][blog]: Blog post with more technical details about how our schema change engine works. -+ [`ALTER DATABASE`](alter-database.html) -+ [`ALTER INDEX`](alter-index.html) -+ [`ALTER RANGE`](alter-range.html) -+ [`ALTER SEQUENCE`](alter-sequence.html) -+ [`ALTER TABLE`][alter-table] -+ [`ALTER VIEW`](alter-view.html) -+ [`CREATE DATABASE`](create-database.html) -+ [`CREATE INDEX`][create-index] -+ [`CREATE SEQUENCE`](create-sequence.html) -+ [`CREATE TABLE`](create-table.html) -+ [`CREATE VIEW`](create-view.html) -+ [`DROP DATABASE`](drop-database.html) -+ [`DROP INDEX`][drop-index] -+ [`DROP SEQUENCE`](drop-sequence.html) -+ [`DROP TABLE`](drop-table.html) -+ [`DROP VIEW`](drop-view.html) -+ [`TRUNCATE`][truncate] - - - - - -[alter-table]: alter-table.html -[blog]: https://cockroachlabs.com/blog/how-online-schema-changes-are-possible-in-cockroachdb/ -[consistent]: frequently-asked-questions.html#how-is-cockroachdb-strongly-consistent -[create-index]: create-index.html -[drop-index]: drop-index.html -[create-table]: create-table.html -[select]: selection-queries.html -[show-jobs]: show-jobs.html -[sql-client]: cockroach-sql.html -[txns]: transactions.html -[truncate]: truncate.html diff --git a/src/current/v21.1/operate-cockroachdb-kubernetes.md b/src/current/v21.1/operate-cockroachdb-kubernetes.md deleted file mode 100644 index d09161c4fbf..00000000000 --- a/src/current/v21.1/operate-cockroachdb-kubernetes.md +++ /dev/null @@ -1,1257 +0,0 @@ ---- -title: Operate CockroachDB on Kubernetes -summary: How to operate and manage a secure 3-node CockroachDB cluster with Kubernetes. -toc: true -toc_not_nested: true -secure: true ---- - -{% capture latest_operator_version %}{% include_cached latest_operator_version.md %}{% endcapture %} - -{{site.data.alerts.callout_info}} -This article assumes you have already [deployed CockroachDB securely on a single Kubernetes cluster](deploy-cockroachdb-with-kubernetes.html). However, it's possible to configure these settings before starting CockroachDB on Kubernetes. -{{site.data.alerts.end}} - -You can configure, scale, and upgrade a CockroachDB deployment on Kubernetes by updating its StatefulSet values. This page describes how to: - -
    -- [Allocate CPU and memory resources](#allocate-resources) -- [Use a custom CA](#use-a-custom-ca) -- [Rotate security certificates](#rotate-security-certificates) -- [Secure webhooks](#secure-the-webhooks) -- [Provision and expand volumes](#provision-volumes) -- [Scale the cluster](#scale-the-cluster) -- [Configure ports](#configure-ports) -- [Perform rolling upgrades](#upgrade-the-cluster) -
    - -
    -- [Allocate CPU and memory resources](#allocate-resources) -- [Provision and expand volumes](#provision-volumes) -- [Scale the cluster](#scale-the-cluster) -- [Perform rolling upgrades](#upgrade-the-cluster) -
    - -
    -- [Allocate CPU and memory resources](#allocate-resources) -- [Use a custom CA](#use-a-custom-ca) -- [Rotate security certificates](#rotate-security-certificates) -- [Provision and expand volumes](#provision-volumes) -- [Scale the cluster](#scale-the-cluster) -- [Perform rolling upgrades](#upgrade-the-cluster) -
    - -
    - - - -
    - -
    -{% include {{ page.version.version }}/orchestration/operator-check-namespace.md %} - -{{site.data.alerts.callout_success}} -If you [deployed CockroachDB on Red Hat OpenShift](deploy-cockroachdb-with-kubernetes-openshift.html), substitute `kubectl` with `oc` in the following commands. -{{site.data.alerts.end}} -
    - -
    -## Apply settings - -Cluster parameters are configured in a `CrdbCluster` custom resource object. This tells the Operator how to configure the Kubernetes cluster. We provide a custom resource template called [`example.yaml`](https://raw.github.com/cockroachdb/cockroach-operator/v{{ latest_operator_version }}/examples/example.yaml): - -~~~ yaml -{% remote_include https://raw.githubusercontent.com/cockroachdb/cockroach-operator/v{{ latest_operator_version }}/examples/example.yaml %} -~~~ - -It's simplest to download and customize a local copy of the custom resource manifest. After you modify its parameters, run this command to apply the new values to the cluster: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ kubectl apply -f example.yaml -~~~ - -You will see: - -~~~ -crdbcluster.crdb.cockroachlabs.com/{cluster-name} configured -~~~ - -The Operator will trigger a rolling restart of the pods to effect the change, if necessary. You can observe its progress by running `kubectl get pods`. -
    - -
    -## Apply settings - -Cluster parameters are configured in the StatefulSet manifest. We provide a [StatefulSet template](https://github.com/cockroachdb/cockroach/blob/master/cloud/kubernetes/bring-your-own-certs/cockroachdb-statefulset.yaml) for use in our [deployment tutorial](deploy-cockroachdb-with-kubernetes.html). - -It's simplest to download and customize a local copy of the manifest file. After you modify its parameters, run this command to apply the new values to the cluster: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ kubectl apply -f {manifest-filename}.yaml -~~~ - -You will see: - -~~~ -crdbcluster.crdb.cockroachlabs.com/{cluster-name} configured -~~~ -
    - -## Allocate resources - -On a production cluster, the resources you allocate to CockroachDB should be proportionate to your machine types and workload. We recommend that you determine and set these values before deploying the cluster, but you can also update the values on a running cluster. - -{{site.data.alerts.callout_success}} -Run `kubectl describe nodes` to see the available resources on the instances that you have provisioned. -{{site.data.alerts.end}} - -### Memory and CPU - -You can set the CPU and memory resources allocated to the CockroachDB container on each pod. - -{{site.data.alerts.callout_info}} -1 CPU in Kubernetes is equivalent to 1 vCPU or 1 hyperthread. For best practices on provisioning CPU and memory for CockroachDB, see the [Production Checklist](recommended-production-settings.html#hardware). -{{site.data.alerts.end}} - -
    -Specify CPU and memory values in `resources.requests` and `resources.limits` in the custom resource: - -~~~ yaml -spec: - resources: - requests: - cpu: "4" - memory: "16Gi" - limits: - cpu: "4" - memory: "16Gi" -~~~ - -Then [apply](#apply-settings) the new values to the cluster. -
    - -
    -Specify CPU and memory values in `resources.requests` and `resources.limits` in the StatefulSet manifest: - -~~~ yaml -spec: - template: - containers: - - name: cockroachdb - resources: - requests: - cpu: "4" - memory: "16Gi" - limits: - cpu: "4" - memory: "16Gi" -~~~ - -Then [apply](#apply-settings) the new values to the cluster. -
    - -
    -Specify CPU and memory values in `resources.requests` and `resources.limits` in a custom [values file](deploy-cockroachdb-with-kubernetes.html?filters=helm#step-2-start-cockroachdb): - -~~~ yaml -statefulset: - resources: - limits: - cpu: "4" - memory: "16Gi" - requests: - cpu: "4" - memory: "16Gi" -~~~ - -Apply the custom values to override the default Helm chart [values](https://github.com/cockroachdb/helm-charts/blob/master/cockroachdb/values.yaml): - -{% include_cached copy-clipboard.html %} -~~~ shell -$ helm upgrade {release-name} --values {custom-values}.yaml cockroachdb/cockroachdb -~~~ -
    - -We recommend using identical values for `resources.requests` and `resources.limits`. When setting the new values, note that not all of a pod's resources will be available to the CockroachDB container. This is because a fraction of the CPU and memory is reserved for Kubernetes. For more information on how Kubernetes handles resource requests and limits, see the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/). - -{{site.data.alerts.callout_info}} -If no resource limits are specified, the pods will be able to consume the maximum available CPUs and memory. However, to avoid overallocating resources when another memory-intensive workload is on the same instance, always set resource requests and limits explicitly. -{{site.data.alerts.end}} - -
    -### Cache and SQL memory size - -Each CockroachDB node reserves a portion of its available memory for its cache and for storing temporary data for SQL queries. For more information on these settings, see the [Production Checklist](recommended-production-settings.html#cache-and-sql-memory-size). - -Our Kubernetes manifests dynamically set cache size and SQL memory size each to 1/4 (the recommended fraction) of the available memory, which depends on the memory request and limit you [specified](#memory-and-cpu) for your configuration. If you want to customize these values, set them explicitly. - -Specify `cache` and `maxSQLMemory` in the custom resource: - -~~~ yaml -spec: - cache: "4Gi" - maxSQLMemory: "4Gi" -~~~ - -Then [apply](#apply-settings) the new values to the cluster. - -{{site.data.alerts.callout_info}} -Specifying these values is equivalent to using the `--cache` and `--max-sql-memory` flags with [`cockroach start`](cockroach-start.html#flags). -{{site.data.alerts.end}} -
    - -For more information on resources, see the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/). - -## Provision volumes - -When you start your cluster, Kubernetes dynamically provisions and mounts a persistent volume into each pod. For more information on persistent volumes, see the [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/persistent-volumes/). - -
    -The storage capacity of each volume is set in `pvc.spec.resources` in the custom resource: - -~~~ yaml -spec: - dataStore: - pvc: - spec: - resources: - limits: - storage: "60Gi" - requests: - storage: "60Gi" -~~~ -
    - -
    -The storage capacity of each volume is initially set in `volumeClaimTemplates.spec.resources` in the StatefulSet manifest: - -~~~ yaml -volumeClaimTemplates: - spec: - resources: - requests: - storage: 100Gi -~~~ -
    - -
    -The storage capacity of each volume is initially set in the Helm chart's [`values.yaml`](https://github.com/cockroachdb/helm-charts/blob/master/cockroachdb/values.yaml): - -~~~ yaml -persistentVolume: - size: 100Gi -~~~ -
    - -You should provision an appropriate amount of disk storage for your workload. For recommendations on this, see the [Production Checklist](recommended-production-settings.html#storage). - -### Expand disk size - -If you discover that you need more capacity, you can expand the persistent volumes on a running cluster. Increasing disk size is often [beneficial for CockroachDB performance](kubernetes-performance.html#disk-size). - -
    -Specify a new volume size in `resources.requests` and `resources.limits` in the custom resource: - -~~~ yaml -spec: - dataStore: - pvc: - spec: - resources: - limits: - storage: "100Gi" - requests: - storage: "100Gi" -~~~ - -Then [apply](#apply-settings) the new values to the cluster. The Operator updates the StatefulSet and triggers a rolling restart of the pods with the new storage capacity. - -To verify that the storage capacity has been updated, run `kubectl get pvc` to view the persistent volume claims (PVCs). It will take a few minutes before the PVCs are completely updated. -
    - -
    -{% include {{ page.version.version }}/orchestration/kubernetes-expand-disk-manual.md %} -
    - -
    -{% include {{ page.version.version }}/orchestration/kubernetes-expand-disk-helm.md %} -
    - -
    -## Use a custom CA - -By default, the Operator will generate and sign 1 client and 1 node certificate to secure the cluster. - -To use your own certificate authority instead, add `clientTLSSecret` and `nodeTLSSecret` to the custom resource. These should specify the names of Kubernetes secrets that contain your generated certificates and keys. For details on creating Kubernetes secrets, see the [Kubernetes documentation](https://kubernetes.io/docs/tasks/configmap-secret/managing-secret-using-kubectl/). - -{{site.data.alerts.callout_info}} -Currently, the Operator requires that the client and node secrets each contain the filenames `tls.crt` and `tls.key`. For an example of working with this, see [Authenticating with `cockroach cert`](#example-authenticating-with-cockroach-cert). -{{site.data.alerts.end}} - -~~~ yaml -spec: - nodeTLSSecret: {node secret name} - clientTLSSecret: {client secret name} -~~~ - -Then [apply](#apply-settings) the new values to the cluster. - -### Example: Authenticating with `cockroach cert` - -These steps demonstrate how certificates and keys generated by [`cockroach cert`](cockroach-cert.html) can be used by the Operator. - -1. Create two directories: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ mkdir certs my-safe-directory - ~~~ - - Directory | Description - ----------|------------ - `certs` | You'll generate your CA certificate and all node and client certificates and keys in this directory. - `my-safe-directory` | You'll generate your CA key in this directory and then reference the key when generating node and client certificates. - -1. Create the CA certificate and key pair: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach cert create-ca \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key - ~~~ - -1. Create a client certificate and key pair for the root user: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach cert create-client \ - root \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key - ~~~ - -1. Upload the client certificate and key to the Kubernetes cluster as a secret, renaming them to the filenames required by the Operator: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl create secret \ - generic cockroachdb.client.root \ - --from-file=tls.key=certs/client.root.key \ - --from-file=tls.crt=certs/client.root.crt \ - --from-file=ca.crt=certs/ca.crt - ~~~ - - ~~~ - secret/cockroachdb.client.root created - ~~~ - -1. Create the certificate and key pair for your CockroachDB nodes: - - {% include_cached copy-clipboard.html %} - ~~~ shell - cockroach cert create-node \ - localhost 127.0.0.1 \ - cockroachdb-public \ - cockroachdb-public.default \ - cockroachdb-public.default.svc.cluster.local \ - *.cockroachdb \ - *.cockroachdb.default \ - *.cockroachdb.default.svc.cluster.local \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key - ~~~ - -1. Upload the node certificate and key to the Kubernetes cluster as a secret, renaming them to the filenames required by the Operator: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl create secret \ - generic cockroachdb.node \ - --from-file=tls.key=certs/node.key \ - --from-file=tls.crt=certs/node.crt \ - --from-file=ca.crt=certs/ca.crt - ~~~ - - ~~~ - secret/cockroachdb.node created - ~~~ - -1. Check that the secrets were created on the cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get secrets - ~~~ - - ~~~ - NAME TYPE DATA AGE - cockroachdb.client.root Opaque 3 13s - cockroachdb.node Opaque 3 3s - default-token-6js7b kubernetes.io/service-account-token 3 9h - ~~~ - -1. Add `nodeTLSSecret` and `clientTLSSecret` to the custom resource, specifying the generated secret names: - - ~~~ yaml - spec: - clientTLSSecret: cockroachdb.client.root - nodeTLSSecret: cockroachdb.node - ~~~ - - Then [apply](#apply-settings) the new values to the cluster. - -## Rotate security certificates - -You may need to rotate the node, client, or CA certificates in the following scenarios: - -- The node, client, or CA certificates are expiring soon. -- Your organization's compliance policy requires periodic certificate rotation. -- The key (for a node, client, or CA) is compromised. -- You need to modify the contents of a certificate, for example, to add another DNS name or the IP address of a load balancer through which a node can be reached. In this case, you would need to rotate only the node certificates. - -### Example: Rotating certificates signed with `cockroach cert` - -If you previously [authenticated with `cockroach cert`](#example-authenticating-with-cockroach-cert), follow these steps to rotate the certificates using the same CA: - -1. Create a new client certificate and key pair for the root user, overwriting the previous certificate and key: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach cert create-client \ - root \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key - --overwrite - ~~~ - -1. Delete the existing client secret: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl delete secret cockroachdb.client.root - ~~~ - - ~~~ - secret "cockroachdb.client.root" deleted - ~~~ - -1. Upload the new client certificate and key to the Kubernetes cluster as a secret, renaming them to the filenames required by the Operator: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl create secret \ - generic cockroachdb.client.root \ - --from-file=tls.key=certs/client.root.key \ - --from-file=tls.crt=certs/client.root.crt \ - --from-file=ca.crt=certs/ca.crt - ~~~ - - ~~~ - secret/cockroachdb.client.root created - ~~~ - -1. Create a new certificate and key pair for your CockroachDB nodes, overwriting the previous certificate and key: - - {% include_cached copy-clipboard.html %} - ~~~ shell - cockroach cert create-node \ - localhost 127.0.0.1 \ - cockroachdb-public \ - cockroachdb-public.default \ - cockroachdb-public.default.svc.cluster.local \ - *.cockroachdb \ - *.cockroachdb.default \ - *.cockroachdb.default.svc.cluster.local \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key - --overwrite - ~~~ - -1. Delete the existing node secret: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl delete secret cockroachdb.node - ~~~ - - ~~~ - secret "cockroachdb.node" deleted - ~~~ - -1. Upload the new node certificate and key to the Kubernetes cluster as a secret, renaming them to the filenames required by the Operator: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl create secret \ - generic cockroachdb.node \ - --from-file=tls.key=certs/node.key \ - --from-file=tls.crt=certs/node.crt \ - --from-file=ca.crt=certs/ca.crt - ~~~ - - ~~~ - secret/cockroachdb.node created - ~~~ - -1. Check that the secrets were created on the cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get secrets - ~~~ - - ~~~ - NAME TYPE DATA AGE - cockroachdb.client.root Opaque 3 4s - cockroachdb.node Opaque 3 1s - default-token-6js7b kubernetes.io/service-account-token 3 9h - ~~~ - - {{site.data.alerts.callout_info}} - Remember that `nodeTLSSecret` and `clientTLSSecret` in the custom resource must specify these secret names. For details, see [Use a custom CA](#use-a-custom-ca). - {{site.data.alerts.end}} - -1. Trigger a rolling restart of the pods by annotating the cluster (named `cockroachdb` in our [example](#apply-settings)): - - {% include_cached copy-clipboard.html %} - ~~~ shell - kubectl annotate {cluster-name} cockroachdb crdb.io/restarttype='rolling' - ~~~ - - {{site.data.alerts.callout_success}} - If you used a different CA to sign the new certificates, trigger a full restart of the cluster instead: `kubectl annotate {cluster-name} cockroachdb crdb.io/restarttype='fullcluster'`. - - **Note:** A full restart will cause a temporary database outage. - {{site.data.alerts.end}} - - ~~~ - crdbcluster.crdb.cockroachlabs.com/cockroachdb annotated - ~~~ - - The pods will terminate and restart one at a time, using the new certificates. - -1. You can observe this process: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get pods - ~~~ - - ~~~ - NAME READY STATUS RESTARTS AGE - cockroach-operator-655fbf7847-lvz6x 1/1 Running 0 4h29m - cockroachdb-0 1/1 Running 0 4h16m - cockroachdb-1 1/1 Terminating 0 4h16m - cockroachdb-2 1/1 Running 0 43s - ~~~ - -## Secure the webhooks - -The Operator ships with both [mutating](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook) and [validating](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook) webhooks. Communication between the Kubernetes API server and the webhook service must be secured with TLS. - -By default, the Operator searches for the TLS secret `cockroach-operator-webhook-ca`, which contains a CA certificate. If the secret is not found, the Operator auto-generates `cockroach-operator-webhook-ca` with a CA certificate for future runs. - -The Operator then generates a one-time server certificate for the webhook server that is signed with `cockroach-operator-webhook-ca`. Finally, the CA bundle for both mutating and validating webhook configurations is patched with the CA certificate. - -You can also use your own certificate authority rather than `cockroach-operator-webhook-ca`. Both the certificate and key files you generate must be PEM-encoded. See the following [example](#example-using-openssl-to-secure-the-webhooks). - -### Example: Using OpenSSL to secure the webhooks - -These steps demonstrate how to use the [`openssl genrsa`](https://www.openssl.org/docs/manmaster/man1/genrsa.html) and [`openssl req`](https://www.openssl.org/docs/manmaster/man1/req.html) subcommands to secure the webhooks on a running Kubernetes cluster: - -1. Generate a 4096-bit RSA private key: - - {% include_cached copy-clipboard.html %} - ~~~ shell - openssl genrsa -out tls.key 4096 - ~~~ - -1. Generate an X.509 certificate, valid for 10 years. You will be prompted for the certificate field values. - - {% include_cached copy-clipboard.html %} - ~~~ shell - openssl req -x509 -new -nodes -key tls.key -sha256 -days 3650 -out tls.crt - ~~~ - -1. Create the secret, making sure that [you are in the correct namespace](deploy-cockroachdb-with-kubernetes.html#install-the-operator): - - {% include_cached copy-clipboard.html %} - ~~~ shell - kubectl create secret tls cockroach-operator-webhook-ca --cert=tls.crt --key=tls.key - ~~~ - - ~~~ - secret/cockroach-operator-webhook-ca created - ~~~ - -1. Remove the certificate and key from your local environment: - - {% include_cached copy-clipboard.html %} - ~~~ shell - rm tls.crt tls.key - ~~~ - -1. Roll the Operator deployment to ensure a new server certificate is generated: - - {% include_cached copy-clipboard.html %} - ~~~ shell - kubectl rollout restart deploy/cockroach-operator-manager - ~~~ - - ~~~ - deployment.apps/cockroach-operator-manager restarted - ~~~ -
    - -
    -## Use a custom CA - -By default on secure deployments, the Helm chart will generate and sign 1 client and 1 node certificate to secure the cluster. - -{{site.data.alerts.callout_danger}} -If you are running a secure Helm deployment on Kubernetes 1.22 and later, you must migrate away from using the Kubernetes CA for cluster authentication. For details, see [Migration to self-signer](#migration-to-self-signer). -{{site.data.alerts.end}} - -To use your own certificate authority instead, specify the following in a custom [values file](deploy-cockroachdb-with-kubernetes.html?filters=helm#step-2-start-cockroachdb): - -{% include_cached copy-clipboard.html %} -~~~ yaml -tls: - enabled: true - certs: - provided: true - clientRootSecret: {client secret name} - nodeSecret: {node secret name} -~~~ - -`clientRootSecret` and `nodeSecret` should specify the names of Kubernetes secrets that contain your generated certificates and keys: - -- `clientRootSecret` specifies the client secret name. -- `nodeSecret` specifies the node secret name. - -Apply the custom values to override the default Helm chart [values](https://github.com/cockroachdb/helm-charts/blob/master/cockroachdb/values.yaml): - -{% include_cached copy-clipboard.html %} -~~~ shell -$ helm upgrade {release-name} --values {custom-values}.yaml cockroachdb/cockroachdb -~~~ - -### Example: Authenticating with `cockroach cert` - -{{site.data.alerts.callout_info}} -The below steps use [`cockroach cert` commands](cockroach-cert.html) to quickly generate and sign the CockroachDB node and client certificates. Read our [Authentication](authentication.html#using-digital-certificates-with-cockroachdb) docs to learn about other methods of signing certificates. -{{site.data.alerts.end}} - -1. Create two directories: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ mkdir certs my-safe-directory - ~~~ - - Directory | Description - ----------|------------ - `certs` | You'll generate your CA certificate and all node and client certificates and keys in this directory. - `my-safe-directory` | You'll generate your CA key in this directory and then reference the key when generating node and client certificates. - -1. Create the CA certificate and key pair: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach cert create-ca \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key - ~~~ - -1. Create a client certificate and key pair for the root user: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach cert create-client \ - root \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key - ~~~ - -1. Upload the client certificate and key to the Kubernetes cluster as a secret: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl create secret \ - generic cockroachdb.client.root \ - --from-file=certs - ~~~ - - ~~~ - secret/cockroachdb.client.root created - ~~~ - -1. Create the certificate and key pair for your CockroachDB nodes: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach cert create-node \ - localhost 127.0.0.1 \ - my-release-cockroachdb-public \ - my-release-cockroachdb-public.default \ - my-release-cockroachdb-public.default.svc.cluster.local \ - *.my-release-cockroachdb \ - *.my-release-cockroachdb.default \ - *.my-release-cockroachdb.default.svc.cluster.local \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key - ~~~ - - {{site.data.alerts.callout_info}} - This example assumes that you followed our [deployment example](deploy-cockroachdb-with-kubernetes.html?filters=helm), which uses `my-release` as the release name. If you used a different value, be sure to adjust the release name in this command. - {{site.data.alerts.end}} - -1. Upload the node certificate and key to the Kubernetes cluster as a secret: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl create secret \ - generic cockroachdb.node \ - --from-file=certs - ~~~ - - ~~~ - secret/cockroachdb.node created - ~~~ - -1. Check that the secrets were created on the cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get secrets - ~~~ - - ~~~ - NAME TYPE DATA AGE - cockroachdb.client.root Opaque 3 41m - cockroachdb.node Opaque 5 14s - default-token-6qjdb kubernetes.io/service-account-token 3 4m - ~~~ - -1. Specify the following in a custom [values file](deploy-cockroachdb-with-kubernetes.html?filters=helm#step-2-start-cockroachdb), using the generated secret names: - - {% include_cached copy-clipboard.html %} - ~~~ - tls: - enabled: true - certs: - provided: true - clientRootSecret: cockroachdb.client.root - nodeSecret: cockroachdb.node - ~~~ - -1. Apply the custom values to override the default Helm chart [values](https://github.com/cockroachdb/helm-charts/blob/master/cockroachdb/values.yaml): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ helm upgrade {release-name} --values {custom-values}.yaml cockroachdb/cockroachdb - ~~~ - -## Rotate security certificates - -You may need to rotate the node, client, or CA certificates in the following scenarios: - -- The node, client, or CA certificates are expiring soon. -- Your organization's compliance policy requires periodic certificate rotation. -- The key (for a node, client, or CA) is compromised. -- You need to modify the contents of a certificate, for example, to add another DNS name or the IP address of a load balancer through which a node can be reached. In this case, you would need to rotate only the node certificates. - -### Example: Rotating certificates signed with `cockroach cert` - -The Helm chart includes [values](https://github.com/cockroachdb/helm-charts/blob/master/cockroachdb/values.yaml) to configure a Kubernetes [cron job](https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/) that regularly rotates certificates before they expire. - -If you previously [authenticated with `cockroach cert`](#example-authenticating-with-cockroach-cert), follow these steps to ensure the certificates are rotated: - -1. Upload the CA certificate that you previously created to the Kubernetes cluster as a secret: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl create secret \ - generic cockroachdb.ca \ - --from-file=certs/ca.crt - ~~~ - - ~~~ - secret/cockroachdb.ca created - ~~~ - -1. Specify the following in a custom [values file](deploy-cockroachdb-with-kubernetes.html?filters=helm#step-2-start-cockroachdb), using the generated secret name: - - {% include_cached copy-clipboard.html %} - ~~~ yaml - selfSigner: - enabled: true - caProvided: true - caSecret: cockroachdb.ca - rotateCerts: true - ~~~ - - {{site.data.alerts.callout_info}} - `selfSigner.enabled` and `selfSigner.rotateCerts` are `true` by default in the Helm chart [values](https://github.com/cockroachdb/helm-charts/blob/master/cockroachdb/values.yaml). - {{site.data.alerts.end}} - -1. Customize the following `selfSigner` fields to set the frequency of certificate rotation. These should correspond to the durations of the CA, client, and node certificates. - - {% include_cached copy-clipboard.html %} - ~~~ yaml - selfSigner: - minimumCertDuration: 624h - caCertDuration: 43800h - caCertExpiryWindow: 648h - clientCertDuration: 672h - clientCertExpiryWindow: 48h - nodeCertDuration: 8760h - nodeCertExpiryWindow: 168h - ~~~ - -
      -
    • caCertDuration, clientCertDuration, and nodeCertDuration specify the duration in hours of the CA, client, and node certificates, respectively.
      • -
    • caCertExpiryWindow, clientCertExpiryWindow, and nodeCertExpiryWindow specify the timeframe in hours during which the CA, client, and node certificates, respectively, should be rotated before they expire.
      • -
    • minimumCertDuration specifies the minimum duration in hours for all certificates. This is to ensure that the client and node certificates are rotated within the duration of the CA certificate. This value must be less than: -
      • cacertExpiryWindow
        • -
      • The difference of clientCertDuration and clientExpiryWindow
        • -
      • The difference of nodeCertDuration and nodeCertExpiryWindow
        • -
      -
    - - Certificate duration is configured when running [`cockroach cert`](cockroach-cert.html#general). You can check the expiration dates of the `cockroach cert` certificates by running: - - {% include_cached copy-clipboard.html %} - ~~~ shell - cockroach cert list --certs-dir=certs - ~~~ - - ~~~ - Certificate directory: certs - Usage | Certificate File | Key File | Expires | Notes | Error - ---------+------------------+-----------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------- - CA | ca.crt | | 2031/09/07 | num certs: 1 | - Node | node.crt | node.key | 2026/09/03 | addresses: localhost,my-release-cockroachdb-public,my-release-cockroachdb-public.default,my-release-cockroachdb-public.default.svc.cluster.local,*.my-release-cockroachdb,*.my-release-cockroachdb.default,*.my-release-cockroachdb.default.svc.cluster.local,127.0.0.1 | - Client | client.root.crt | client.root.key | 2026/09/03 | user: root | - ~~~ - -1. Apply the custom values to override the default Helm chart [values](https://github.com/cockroachdb/helm-charts/blob/master/cockroachdb/values.yaml): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ helm upgrade {release-name} --values {custom-values}.yaml cockroachdb/cockroachdb - ~~~ - - The certificates will be rotated during the specified expiry windows. - -## Migration to self-signer - -Previous versions of the Helm chart used the Kubernetes CA to sign certificates. However, the Kubernetes CA is deprecated from Kubernetes 1.22 and later. The Helm chart now uses a self-signer for cluster authentication. - -To migrate your Helm deployment to use the self-signer: - -1. Set the cluster's upgrade strategy to `OnDelete`, which specifies that only pods deleted by the user will be upgraded. - - {% include_cached copy-clipboard.html %} - ~~~ shell - helm upgrade {release-name} cockroachdb/cockroachdb --set statefulset.updateStrategy.type="OnDelete" - ~~~ - -1. Confirm that the `init` pod has been created: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get pods - ~~~ - - ~~~ - NAME READY STATUS RESTARTS AGE - my-release-cockroachdb-0 1/1 Running 0 6m - my-release-cockroachdb-1 1/1 Running 0 6m - my-release-cockroachdb-2 1/1 Running 0 6m - my-release-cockroachdb-init-59ndf 0/1 Completed 0 8s - ~~~ - -1. Delete the cluster pods to start the upgrade process. - - {% include_cached copy-clipboard.html %} - ~~~ shell - kubectl delete pods -l app.kubernetes.io/component=cockroachdb - ~~~ - - ~~~ - pod "my-release-cockroachdb-0" deleted - pod "my-release-cockroachdb-1" deleted - pod "my-release-cockroachdb-2" deleted - ~~~ - - All pods will be restarted with new certificates generated by the self-signer. Note that this is not a rolling upgrade, so the cluster will experience some downtime. You can monitor this process: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get pods - ~~~ - - ~~~ - NAME READY STATUS RESTARTS AGE - my-release-cockroachdb-0 0/1 ContainerCreating 0 14s - my-release-cockroachdb-1 0/1 ContainerCreating 0 4s - my-release-cockroachdb-2 0/1 Terminating 0 7m16s - ~~~ -
    - -
    -## Configure ports - -The Operator separates traffic into three ports: - -| Protocol | Default | Description | Custom Resource Field | -|----------|---------|---------------------------------------------------------------------|-----------------------| -| gRPC | 26258 | Used for node connections | `grpcPort` | -| HTTP | 8080 | Used to [access the DB Console](ui-overview.html#db-console-access) | `httpPort` | -| SQL | 26257 | Used for SQL shell access | `sqlPort` | - -Specify alternate port numbers in the custom resource (for example, to match the default port `5432` on PostgreSQL): - -~~~ yaml -spec: - sqlPort: 5432 -~~~ - -Then [apply](#apply-settings) the new values to the cluster. The Operator updates the StatefulSet and triggers a rolling restart of the pods with the new port settings. - -{{site.data.alerts.callout_danger}} -Currently, only the pods are updated with new ports. To connect to the cluster, you need to ensure that the `public` service is also updated to use the new port. You can do this by deleting the service with `kubectl delete service {cluster-name}-public`. When service is recreated by the Operator, it will use the new port. This is a known limitation that will be fixed in an Operator update. -{{site.data.alerts.end}} -
    - -## Scale the cluster - -### Add nodes - -
    -Before scaling up CockroachDB, note the following [topology recommendations](recommended-production-settings.html#topology): - -- Each CockroachDB node (running in its own pod) should run on a separate Kubernetes worker node. -- Each availability zone should have the same number of CockroachDB nodes. - -If your cluster has 3 CockroachDB nodes distributed across 3 availability zones (as in our [deployment example](deploy-cockroachdb-with-kubernetes.html)), we recommend scaling up by a multiple of 3 to retain an even distribution of nodes. You should therefore scale up to a minimum of 6 CockroachDB nodes, with 2 nodes in each zone. - -1. Run `kubectl get nodes` to list the worker nodes in your Kubernetes cluster. There should be at least as many worker nodes as pods you plan to add. This ensures that no more than one pod will be placed on each worker node. - -1. If you need to add worker nodes, resize your GKE cluster by specifying the desired number of worker nodes in each zone: - - {% include_cached copy-clipboard.html %} - ~~~ shell - gcloud container clusters resize {cluster-name} --region {region-name} --num-nodes 2 - ~~~ - - This example distributes 2 worker nodes across the default 3 zones, raising the total to 6 worker nodes. - - 1. If you are adding nodes after previously [scaling down](#remove-nodes), and have not enabled [automatic PVC pruning](#automatic-pvc-pruning), you must first manually delete any persistent volumes that were orphaned by node removal. - - {{site.data.alerts.callout_info}} - Due to a [known issue](https://github.com/cockroachdb/cockroach-operator/issues/542), automatic pruning of PVCs is currently disabled by default. This means that after decommissioning and removing a node, the Operator will not remove the persistent volume that was mounted to its pod. - {{site.data.alerts.end}} - - View the PVCs on the cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - kubectl get pvc - ~~~ - - ~~~ - NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE - datadir-cockroachdb-0 Bound pvc-f1ce6ed2-ceda-40d2-8149-9e5b59faa9df 60Gi RWO standard 24m - datadir-cockroachdb-1 Bound pvc-308da33c-ec77-46c7-bcdf-c6e610ad4fea 60Gi RWO standard 24m - datadir-cockroachdb-2 Bound pvc-6816123f-29a9-4b86-a4e2-b67f7bb1a52c 60Gi RWO standard 24m - datadir-cockroachdb-3 Bound pvc-63ce836a-1258-4c58-8b37-d966ed12d50a 60Gi RWO standard 24m - datadir-cockroachdb-4 Bound pvc-1ylabv86-6512-6n12-bw3g-i0dh2zxvfhd0 60Gi RWO standard 24m - datadir-cockroachdb-5 Bound pvc-2vka2c9x-7824-41m5-jk45-mt7dzq90q97x 60Gi RWO standard 24m - ~~~ - - 1. The PVC names correspond to the pods they are bound to. For example, if the pods `cockroachdb-3`, `cockroachdb-4`, and `cockroachdb-5` had been removed by [scaling the cluster down](#remove-nodes) from 6 to 3 nodes, `datadir-cockroachdb-3`, `datadir-cockroachdb-4`, and `datadir-cockroachdb-5` would be the PVCs for the orphaned persistent volumes. To verify that a PVC is not currently bound to a pod: - - {% include_cached copy-clipboard.html %} - ~~~ shell - kubectl describe pvc datadir-cockroachdb-5 - ~~~ - - The output will include the following line: - - ~~~ - Mounted By: - ~~~ - - If the PVC is bound to a pod, it will specify the pod name. - - 1. Remove the orphaned persistent volumes by deleting their PVCs: - - {{site.data.alerts.callout_danger}} - Before deleting any persistent volumes, be sure you have a backup copy of your data. Data **cannot** be recovered once the persistent volumes are deleted. For more information, see the [Kubernetes documentation](https://kubernetes.io/docs/tasks/run-application/delete-stateful-set/#persistent-volumes). - {{site.data.alerts.end}} - - {% include_cached copy-clipboard.html %} - ~~~ shell - kubectl delete pvc datadir-cockroachdb-3 datadir-cockroachdb-4 datadir-cockroachdb-5 - ~~~ - - ~~~ - persistentvolumeclaim "datadir-cockroachdb-3" deleted - persistentvolumeclaim "datadir-cockroachdb-4" deleted - persistentvolumeclaim "datadir-cockroachdb-5" deleted - ~~~ - -1. Update `nodes` in the custom resource with the target size of the CockroachDB cluster. This value refers to the number of CockroachDB nodes, each running in one pod: - - ~~~ - nodes: 6 - ~~~ - - {{site.data.alerts.callout_info}} - Note that you must scale by updating the `nodes` value in the custom resource. Using `kubectl scale statefulset --replicas=4` will result in new pods immediately being terminated. - {{site.data.alerts.end}} - -1. [Apply](#apply-settings) the new value. - -1. Verify that the new pods were successfully started: - - {% include_cached copy-clipboard.html %} - ~~~ shell - kubectl get pods - ~~~ - - ~~~ - NAME READY STATUS RESTARTS AGE - cockroach-operator-655fbf7847-zn9v8 1/1 Running 0 30m - cockroachdb-0 1/1 Running 0 24m - cockroachdb-1 1/1 Running 0 24m - cockroachdb-2 1/1 Running 0 24m - cockroachdb-3 1/1 Running 0 30s - cockroachdb-4 1/1 Running 0 30s - cockroachdb-5 1/1 Running 0 30s - ~~~ - - Each pod should be running in one of the 6 worker nodes. -
    - -
    -{% include {{ page.version.version }}/orchestration/kubernetes-scale-cluster-manual.md %} -
    - -
    -{% include {{ page.version.version }}/orchestration/kubernetes-scale-cluster-helm.md %} -
    - -### Remove nodes - -{{site.data.alerts.callout_danger}} -Do **not** scale down to fewer than 3 nodes. This is considered an anti-pattern on CockroachDB and will cause errors. -{{site.data.alerts.end}} - -
    -{{site.data.alerts.callout_danger}} -Due to a [known issue](https://github.com/cockroachdb/cockroach-operator/issues/542), automatic pruning of PVCs is currently disabled by default. This means that after decommissioning and removing a node, the Operator will not remove the persistent volume that was mounted to its pod. - -If you plan to eventually [scale up](#add-nodes) the cluster after scaling down, you will need to manually delete any PVCs that were orphaned by node removal before scaling up. For more information, see [Add nodes](#add-nodes). -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -If you want to enable the Operator to automatically prune PVCs when scaling down, see [Automatic PVC pruning](#automatic-pvc-pruning). However, note that this workflow is currently unsupported. -{{site.data.alerts.end}} - -Before scaling down CockroachDB, note the following [topology recommendation](recommended-production-settings.html#topology): - -- Each availability zone should have the same number of CockroachDB nodes. - -If your nodes are distributed across 3 availability zones (as in our [deployment example](deploy-cockroachdb-with-kubernetes.html)), we recommend scaling down by a multiple of 3 to retain an even distribution. If your cluster has 6 CockroachDB nodes, you should therefore scale down to 3, with 1 node in each zone. - -1. Update `nodes` in the custom resource with the target size of the CockroachDB cluster. For instance, to scale down to 3 nodes: - - ~~~ - nodes: 3 - ~~~ - - {{site.data.alerts.callout_info}} - Before removing a node, the Operator first decommissions the node. This lets a node finish in-flight requests, rejects any new requests, and transfers all range replicas and range leases off the node. - {{site.data.alerts.end}} - -1. [Apply](#apply-settings) the new value. - - The Operator will remove nodes from the cluster one at a time, starting from the pod with the highest number in its address. - -1. Verify that the pods were successfully removed: - - {% include_cached copy-clipboard.html %} - ~~~ shell - kubectl get pods - ~~~ - - ~~~ - NAME READY STATUS RESTARTS AGE - cockroach-operator-655fbf7847-zn9v8 1/1 Running 0 32m - cockroachdb-0 1/1 Running 0 26m - cockroachdb-1 1/1 Running 0 26m - cockroachdb-2 1/1 Running 0 26m - ~~~ - -#### Automatic PVC pruning - -To enable the Operator to automatically remove persistent volumes when [scaling down](#remove-nodes) a cluster, turn on automatic PVC pruning through a feature gate. - -{{site.data.alerts.callout_danger}} -This workflow is unsupported and should be enabled at your own risk. -{{site.data.alerts.end}} - -1. Download the Operator manifest: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ curl -0 https://raw.githubusercontent.com/cockroachdb/cockroach-operator/v{{ latest_operator_version }}/install/operator.yaml - ~~~ - -1. Uncomment the following lines in the Operator manifest: - - ~~~ yaml - - feature-gates - - AutoPrunePVC=true - ~~~ - -1. Reapply the Operator manifest: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl apply -f operator.yaml - ~~~ - -1. Validate that the Operator is running: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get pods - ~~~ - - ~~~ - NAME READY STATUS RESTARTS AGE - cockroach-operator-6f7b86ffc4-9ppkv 1/1 Running 0 22s - ... - ~~~ -
    - -
    -{% include {{ page.version.version }}/orchestration/kubernetes-remove-nodes-manual.md %} -
    - -
    -{% include {{ page.version.version }}/orchestration/kubernetes-remove-nodes-helm.md %} -
    - -## Upgrade the cluster - -We strongly recommend that you regularly upgrade your CockroachDB version in order to pick up bug fixes, performance improvements, and new features. - -The upgrade process on Kubernetes is a [staged update](https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#staging-an-update) in which the Docker image is applied to the pods one at a time, with each pod being stopped and restarted in turn. This is to ensure that the cluster remains available during the upgrade. - -
    -1. Verify that you can upgrade. - - To upgrade to a new major version, you must first be on a production release of the previous version. The release does not need to be the latest production release of the previous version, but it must be a production [release](../releases/index.html) and not a testing release (alpha/beta). - - Therefore, in order to upgrade to v21.1, you must be on a production release of v20.2. - - 1. If you are upgrading to v21.1 from a production release earlier than v20.2, or from a testing release (alpha/beta), first [upgrade to a production release of v20.2](../v20.2/orchestrate-cockroachdb-with-kubernetes.html#upgrade-the-cluster). Be sure to complete all the steps. - - 1. Then return to this page and perform a second upgrade to v21.1. - - 1. If you are upgrading from a production release of v20.2, or from any earlier v21.1 patch release, you do not have to go through intermediate releases; continue to step 2. - -1. Verify the overall health of your cluster using the [DB Console](ui-overview.html). On the **Overview**: - - Under **Node Status**, make sure all nodes that should be live are listed as such. If any nodes are unexpectedly listed as suspect or dead, identify why the nodes are offline and either restart them or [decommission](#remove-nodes) them before beginning your upgrade. If there are dead and non-decommissioned nodes in your cluster, it will not be possible to finalize the upgrade (either automatically or manually). - - Under **Replication Status**, make sure there are 0 under-replicated and unavailable ranges. Otherwise, performing a rolling upgrade increases the risk that ranges will lose a majority of their replicas and cause cluster unavailability. Therefore, it's important to [identify and resolve the cause of range under-replication and/or unavailability](cluster-setup-troubleshooting.html#replication-issues) before beginning your upgrade. - - In the **Node List**: - - Make sure all nodes are on the same version. If not all nodes are on the same version, upgrade them to the cluster's highest current version first, and then start this process over. - - Make sure capacity and memory usage are reasonable for each node. Nodes must be able to tolerate some increase in case the new version uses more resources for your workload. Also go to **Metrics > Dashboard: Hardware** and make sure CPU percent is reasonable across the cluster. If there's not enough headroom on any of these metrics, consider [adding nodes](#add-nodes) to your cluster before beginning your upgrade. - -1. Review the [backward-incompatible changes in v21.1](../releases/v21.1.html#v21-1-0-backward-incompatible-changes) and [deprecated features](../releases/v21.1.html#v21-1-0-deprecations). If any affect your deployment, make the necessary changes before starting the rolling upgrade to v21.1. - -1. Change the desired Docker image in the custom resource: - - ~~~ - image: - name: cockroachdb/cockroach:{{page.release_info.version}} - ~~~ - -1. [Apply](#apply-settings) the new value. The Operator will perform the staged update. - - {{site.data.alerts.callout_info}} - The Operator automatically sets the `cluster.preserve_downgrade_option` [cluster setting](cluster-settings.html) to the version you are upgrading from. This disables auto-finalization of the upgrade so that you can monitor the stability and performance of the upgraded cluster before manually finalizing the upgrade. This will enable certain [features and performance improvements introduced in v21.1](upgrade-cockroach-version.html#features-that-require-upgrade-finalization). - - Note that after finalization, it will no longer be possible to perform a downgrade to v20.2. In the event of a catastrophic failure or corruption, the only option will be to start a new cluster using the old binary and then restore from a [backup](take-full-and-incremental-backups.html) created prior to performing the upgrade. - - Finalization only applies when performing a major version upgrade (for example, from v20.2.x to v21.1). Patch version upgrades (for example, within the v21.1.x series) can always be downgraded. - {{site.data.alerts.end}} - -1. To check the status of the rolling upgrade, run `kubectl get pods`. The pods are restarted one at a time with the new image. - -1. Verify that all pods have been upgraded by running: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get pods \ - -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}' - ~~~ - - You can also check the CockroachDB version of each node in the [DB Console](ui-cluster-overview-page.html#node-details). - -1. Monitor the stability and performance of your cluster until you are comfortable with the upgrade (generally at least a day). - - If you decide to roll back the upgrade, revert the image name in the custom resource and apply the new value. - - {{site.data.alerts.callout_info}} - This is only possible when performing a major version upgrade (for example, from v20.2.x to v21.1). Patch version upgrades (for example, within the v21.1.x series) are auto-finalized. - {{site.data.alerts.end}} - - To finalize the upgrade, re-enable auto-finalization: - - 1. Start the CockroachDB [built-in SQL client](cockroach-sql.html). For example, if you followed the steps in [Deploy CockroachDB with Kubernetes](deploy-cockroachdb-with-kubernetes.html#step-3-use-the-built-in-sql-client) to launch a secure client pod, get a shell into the `cockroachdb-client-secure` pod: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl exec -it cockroachdb-client-secure \-- ./cockroach sql \ - --certs-dir=/cockroach/cockroach-certs \ - --host={cluster-name}-public - ~~~ - - 1. Re-enable auto-finalization: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > RESET CLUSTER SETTING cluster.preserve_downgrade_option; - ~~~ - - 1. Exit the SQL shell and pod: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > \q - ~~~ - -
    - -
    -{% include {{ page.version.version }}/orchestration/kubernetes-upgrade-cluster-manual.md %} -
    - -
    -{% include {{ page.version.version }}/orchestration/kubernetes-upgrade-cluster-helm.md %} -
    - -## See also - -- [Kubernetes Multi-Cluster Deployment](orchestrate-cockroachdb-with-kubernetes-multi-cluster.html) -- [Kubernetes Performance Guide](kubernetes-performance.html) -{% include {{ page.version.version }}/prod-deployment/prod-see-also.md %} diff --git a/src/current/v21.1/operational-faqs.md b/src/current/v21.1/operational-faqs.md deleted file mode 100644 index 515cf8bacb9..00000000000 --- a/src/current/v21.1/operational-faqs.md +++ /dev/null @@ -1,143 +0,0 @@ ---- -title: Operational FAQs -summary: Get answers to frequently asked questions about operating CockroachDB. -toc: true -toc_not_nested: true ---- - -## Why is my process hanging when I try to start nodes with the `--background` flag? - -Check whether you have previously run a multi-node cluster using the same data directory. If you have not, refer to [Troubleshoot Cluster Setup](cluster-setup-troubleshooting.html). - -If you have previously started and stopped a multi-node cluster, and are now trying to bring it back up, note the following: - -The [`--background`](cockroach-start.html#general) flag of [`cockroach start`](cockroach-start.html) causes the `start` command to wait until the node has fully initialized and is able to start serving queries. In addition, to keep your data consistent, CockroachDB waits until a majority of nodes are running. This means that if only one node of a three-node cluster is running, that one node will not be operational. - -As a result, starting nodes with the `--background` flag will cause `cockroach start` to hang until a majority of nodes are fully initialized. - -To restart your cluster, you should either: - -- Use multiple terminals to start multiple nodes at once. -- Start each node in the background using your shell's functionality (e.g., `cockroach start &`) instead of using the `--background` flag. - -## Why is memory usage increasing despite lack of traffic? - -Like most databases, CockroachDB caches the most recently accessed data in memory so that it can provide faster reads, and [its periodic writes of time-series data](#why-is-disk-usage-increasing-despite-lack-of-writes) cause that cache size to increase until it hits its configured limit. For information about manually controlling the cache size, see [Recommended Production Settings](recommended-production-settings.html#cache-and-sql-memory-size). - -## Why is disk usage increasing despite lack of writes? - -The time-series data used in the [DB Console](ui-overview-dashboard.html) is stored within the cluster and accumulates for 10 days before it starts to be truncated. As a result, for the first 10 days or so of a cluster's life, you will see a steady increase in disk usage and the number of ranges even if you are not writing data to the cluster. - -## Can I reduce or disable the storage of time-series data? - -Yes, you can either [reduce the interval for time-series storage](#reduce-the-interval-for-time-series-storage) or [disable time-series storage entirely](#disable-time-series-storage-entirely). - -{{site.data.alerts.callout_info}} -After reducing or disabling time-series storage, it can take up to 24 hours for time-series data to be deleted and for the change to be reflected in DB Console metrics. -{{site.data.alerts.end}} - -### Reduce the interval for time-series storage - -By default, CockroachDB stores time-series data at 10s resolution for 10 days. This data is aggregated into time-series data at 30m resolution, which is stored for 90 days. - -To reduce the interval for storage of time-series data: - -- For data stored at 10s resolution, change the `timeseries.storage.resolution_10s.ttl` cluster setting to an [`INTERVAL`](interval.html) value less than `240h0m0s` (10 days). - - For example, to change the storage interval for time-series data at 10s resolution to 5 days, run the following [`SET CLUSTER SETTING`](set-cluster-setting.html) command: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING timeseries.storage.resolution_10s.ttl = '120h0m0s'; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW CLUSTER SETTING timeseries.storage.resolution_10s.ttl; - ~~~ - - ~~~ - timeseries.storage.resolution_10s.ttl - +---------------------------------------+ - 120:00:00 - (1 row) - ~~~ - - Note that this data is still aggregated into data at 30m resolution, which is stored for 90 days by default. - -- For data stored at 30m resolution, change the `timeseries.storage.resolution_30m.ttl` cluster setting to an [`INTERVAL`](interval.html) value less than `2160h0m0s` (90 days). - -### Disable time-series storage entirely - -{{site.data.alerts.callout_info}} -Disabling time-series storage is recommended only if you exclusively use a third-party tool such as [Prometheus](monitor-cockroachdb-with-prometheus.html) for time-series monitoring. Prometheus and other such tools do not rely on CockroachDB-stored time-series data; instead, they ingest metrics exported by CockroachDB from memory and then store the data themselves. -{{site.data.alerts.end}} - -To disable the storage of time-series data entirely, run the following command: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET CLUSTER SETTING timeseries.storage.enabled = false; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CLUSTER SETTING timeseries.storage.enabled; -~~~ - -~~~ - timeseries.storage.enabled -+----------------------------+ - false -(1 row) -~~~ - -If you want all existing time-series data to be deleted, also change both the `timeseries.storage.resolution_10s.ttl` and `timeseries.storage.resolution_30m.ttl` cluster settings: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET CLUSTER SETTING timeseries.storage.resolution_10s.ttl = '0s'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET CLUSTER SETTING timeseries.storage.resolution_30m.ttl = '0s'; -~~~ - -## What happens when a node runs out of disk space? - -When a node runs out of disk space, it shuts down and cannot be restarted until space is freed up. To prepare for this case, place a [ballast file](cockroach-debug-ballast.html) in each node's storage directory that can be deleted to free up enough space to be able to restart the node. If you did not create a ballast file, look for other files that can be deleted, such as [log files](configure-logs.html#logging-directory). - -{{site.data.alerts.callout_info}} -In addition to using ballast files, it is important to actively [monitor remaining disk space](monitoring-and-alerting.html#events-to-alert-on). -{{site.data.alerts.end}} - -## Why would increasing the number of nodes not result in more operations per second? - -If queries operate on different data, then increasing the number of nodes should improve the overall throughput (transactions/second or QPS). - -However, if your queries operate on the same data, you may be observing transaction contention. For details, see [SQL Performance Best Practices](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention). - -## Why does CockroachDB collect anonymized cluster usage details by default? - -Cockroach Labs collects information about CockroachDB's real-world usage to help prioritize the development of product features. We choose our default as "opt-in" to strengthen the information collected, and are careful to send only anonymous, aggregate usage statistics. For details on what information is collected and how to opt out, see [Diagnostics Reporting](diagnostics-reporting.html). - -## What happens when node clocks are not properly synchronized? - -{% include {{ page.version.version }}/faq/clock-synchronization-effects.md %} - -## How can I tell how well node clocks are synchronized? - -{% include {{ page.version.version }}/faq/clock-synchronization-monitoring.html %} - -You can also see these metrics in [the Clock Offset graph](ui-runtime-dashboard.html#clock-offset) on the DB Console. - -## How do I prepare for planned node maintenance? - -{% include {{ page.version.version }}/faq/planned-maintenance.md %} - -## See also - -- [Production Checklist](recommended-production-settings.html) -- [Product FAQs](frequently-asked-questions.html) -- [SQL FAQs](sql-faqs.html) \ No newline at end of file diff --git a/src/current/v21.1/orchestrate-a-local-cluster-with-kubernetes-insecure.md b/src/current/v21.1/orchestrate-a-local-cluster-with-kubernetes-insecure.md deleted file mode 100644 index 0b83149f743..00000000000 --- a/src/current/v21.1/orchestrate-a-local-cluster-with-kubernetes-insecure.md +++ /dev/null @@ -1,150 +0,0 @@ ---- -title: Orchestration with Kubernetes (Insecure) -summary: Orchestrate the deployment and management of a local cluster using Kubernetes. -toc: true ---- - -
    - - -
    - -On top of CockroachDB's built-in automation, you can use a third-party [orchestration](orchestration.html) system to simplify and automate even more of your operations, from deployment to scaling to overall cluster management. - -This page demonstrates a basic integration with the open-source [Kubernetes](http://kubernetes.io/) orchestration system. Using either the CockroachDB [Helm](https://helm.sh/) chart or a few configuration files, you'll quickly create a 3-node local cluster. You'll run some SQL commands against the cluster and then simulate node failure, watching how Kubernetes auto-restarts without the need for any manual intervention. You'll then scale the cluster with a single command before shutting the cluster down, again with a single command. - -{{site.data.alerts.callout_info}} -To orchestrate a physically distributed cluster in production, see [Orchestrated Deployments](orchestration.html). To deploy a 30-day free CockroachDB {{ site.data.products.cloud }} cluster instead of running CockroachDB yourself, see the [Quickstart](../cockroachcloud/quickstart.html). -{{site.data.alerts.end}} - -{% include {{ page.version.version }}/orchestration/local-start-kubernetes.md %} - -## Step 2. Start CockroachDB - -To start your CockroachDB cluster, you can either use our StatefulSet configuration and related files directly, or you can use the [Helm](https://helm.sh/) package manager for Kubernetes to simplify the process. - -
    - - -
    - -
    -{% include {{ page.version.version }}/orchestration/start-cockroachdb-local-insecure.md %} -
    - -
    -{% include {{ page.version.version }}/orchestration/start-cockroachdb-local-helm-insecure.md %} -
    - -## Step 3. Use the built-in SQL client - -{% include {{ page.version.version }}/orchestration/test-cluster-insecure.md %} - -## Step 4. Access the DB Console - -{% include {{ page.version.version }}/orchestration/monitor-cluster.md %} - -## Step 5. Simulate node failure - -{% include {{ page.version.version }}/orchestration/kubernetes-simulate-failure.md %} - -## Step 6. Add nodes - -1. Use the `kubectl scale` command to add a pod for another CockroachDB node: - -
    - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl scale statefulset cockroachdb --replicas=4 - ~~~ - - ~~~ - statefulset "cockroachdb" scaled - ~~~ - -
    - -
    - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl scale statefulset my-release-cockroachdb --replicas=4 - ~~~ - - ~~~ - statefulset "my-release-cockroachdb" scaled - ~~~ - -
    - -2. Verify that the pod for a fourth node, `cockroachdb-3`, was added successfully: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get pods - ~~~ - -
    - ~~~ - NAME READY STATUS RESTARTS AGE - cockroachdb-0 1/1 Running 0 28m - cockroachdb-1 1/1 Running 0 27m - cockroachdb-2 1/1 Running 0 10m - cockroachdb-3 1/1 Running 0 5s - example-545f866f5-2gsrs 1/1 Running 0 25m - ~~~ -
    - -
    - ~~~ - NAME READY STATUS RESTARTS AGE - my-release-cockroachdb-0 1/1 Running 0 28m - my-release-cockroachdb-1 1/1 Running 0 27m - my-release-cockroachdb-2 1/1 Running 0 10m - my-release-cockroachdb-3 1/1 Running 0 5s - example-545f866f5-2gsrs 1/1 Running 0 25m - ~~~ -
    - -## Step 7. Remove nodes - -{% include {{ page.version.version }}/orchestration/kubernetes-remove-nodes-insecure.md %} - -## Step 8. Stop the cluster - -- **If you plan to restart the cluster**, use the `minikube stop` command. This shuts down the minikube virtual machine but preserves all the resources you created: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ minikube stop - ~~~ - - ~~~ - Stopping local Kubernetes cluster... - Machine stopped. - ~~~ - - You can restore the cluster to its previous state with `minikube start`. - -- **If you do not plan to restart the cluster**, use the `minikube delete` command. This shuts down and deletes the minikube virtual machine and all the resources you created, including persistent volumes: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ minikube delete - ~~~ - - ~~~ - Deleting local Kubernetes cluster... - Machine deleted. - ~~~ - - {{site.data.alerts.callout_success}}To retain logs, copy them from each pod's stderr before deleting the cluster and all its resources. To access a pod's standard error stream, run kubectl logs <podname>.{{site.data.alerts.end}} - -## See also - -Explore other core CockroachDB benefits and features: - -{% include {{ page.version.version }}/misc/explore-benefits-see-also.md %} - -You might also want to learn how to [orchestrate a production deployment of CockroachDB with Kubernetes](deploy-cockroachdb-with-kubernetes.html). diff --git a/src/current/v21.1/orchestrate-a-local-cluster-with-kubernetes.md b/src/current/v21.1/orchestrate-a-local-cluster-with-kubernetes.md deleted file mode 100644 index 171894d58f4..00000000000 --- a/src/current/v21.1/orchestrate-a-local-cluster-with-kubernetes.md +++ /dev/null @@ -1,93 +0,0 @@ ---- -title: Deploy a Local Cluster with Kubernetes -summary: Deploy a local 3-node CockroachDB cluster with Kubernetes. -toc: true -secure: true ---- - -
    - - -
    - -On top of CockroachDB's built-in automation, you can use a third-party [orchestration](orchestration.html) system to simplify and automate even more of your operations, from deployment to scaling to overall cluster management. - -This page demonstrates a basic integration with the open-source [Kubernetes](http://kubernetes.io/) orchestration system. Using either the CockroachDB [Helm](https://helm.sh/) chart or a few configuration files, you'll quickly create a 3-node local cluster. You'll run some SQL commands against the cluster and then simulate node failure, watching how Kubernetes auto-restarts without the need for any manual intervention. You'll then scale the cluster with a single command before shutting the cluster down, again with a single command. - -{{site.data.alerts.callout_info}} -To orchestrate a physically distributed cluster in production, see [Orchestrated Deployments](orchestration.html). To deploy a 30-day free CockroachDB {{ site.data.products.cloud }} cluster instead of running CockroachDB yourself, see the [Quickstart](../cockroachcloud/quickstart.html). -{{site.data.alerts.end}} - -{% include {{ page.version.version }}/orchestration/local-start-kubernetes.md %} - -## Step 2. Start CockroachDB - -Choose a way to deploy and maintain the CockroachDB cluster: - -- [CockroachDB Kubernetes Operator](https://github.com/cockroachdb/cockroach-operator) (recommended) -- [Helm](https://helm.sh/) package manager -- Manually apply our StatefulSet configuration and related files - -
    - - - -
    - -
    -{% include {{ page.version.version }}/orchestration/start-cockroachdb-operator-secure.md %} -
    - -
    -{% include {{ page.version.version }}/orchestration/start-cockroachdb-secure.md %} -
    - -
    -{% include {{ page.version.version }}/orchestration/start-cockroachdb-helm-secure.md %} -
    - -## Step 3. Use the built-in SQL client - -{% include {{ page.version.version }}/orchestration/test-cluster-secure.md %} - -## Step 4. Access the DB Console - -{% include {{ page.version.version }}/orchestration/monitor-cluster.md %} - -## Step 8. Stop the cluster - -- **If you plan to restart the cluster**, use the `minikube stop` command. This shuts down the minikube virtual machine but preserves all the resources you created: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ minikube stop - ~~~ - - ~~~ - Stopping local Kubernetes cluster... - Machine stopped. - ~~~ - - You can restore the cluster to its previous state with `minikube start`. - -- **If you do not plan to restart the cluster**, use the `minikube delete` command. This shuts down and deletes the minikube virtual machine and all the resources you created, including persistent volumes: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ minikube delete - ~~~ - - ~~~ - Deleting local Kubernetes cluster... - Machine deleted. - ~~~ - - {{site.data.alerts.callout_success}}To retain logs, copy them from each pod's stderr before deleting the cluster and all its resources. To access a pod's standard error stream, run kubectl logs <podname>.{{site.data.alerts.end}} - -## See also - -Explore other core CockroachDB benefits and features: - -{% include {{ page.version.version }}/misc/explore-benefits-see-also.md %} - -You might also want to learn how to [orchestrate a production deployment of CockroachDB with Kubernetes](deploy-cockroachdb-with-kubernetes.html). diff --git a/src/current/v21.1/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md b/src/current/v21.1/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md deleted file mode 100644 index db1ea530163..00000000000 --- a/src/current/v21.1/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md +++ /dev/null @@ -1,1313 +0,0 @@ ---- -title: Orchestrate CockroachDB Across Multiple Kubernetes Clusters -summary: Orchestrate the deployment, management, and monitoring of CockroachDB across multiple Kubernetes clusters in different regions. -toc: true -toc_not_nested: true ---- - -
    - - -
    - -This page shows you how to orchestrate a secure CockroachDB deployment across three [Kubernetes](http://kubernetes.io/) clusters, each in a different geographic region, using [StatefulSets](http://kubernetes.io/docs/concepts/abstractions/controllers/statefulsets/) to manage the containers within each cluster and linking them together via DNS. This will result in a single, multi-region CockroachDB cluster running on Kubernetes. - -{{site.data.alerts.callout_success}} -To deploy CockroachDB in a single Kubernetes cluster instead, see [Kubernetes Single-Cluster Deployment](deploy-cockroachdb-with-kubernetes.html). Also, for details about potential performance bottlenecks to be aware of when running CockroachDB in Kubernetes and guidance on how to optimize your deployment for better performance, see [CockroachDB Performance on Kubernetes](kubernetes-performance.html). -{{site.data.alerts.end}} - -## Before you begin - -Before getting started, it's helpful to review some Kubernetes-specific terminology and current limitations. - -- [Kubernetes terminology](#kubernetes-terminology) -- [UX differences from running in a single cluster](#ux-differences-from-running-in-a-single-cluster) -- [Limitations](#limitations) - -### Kubernetes terminology - -Feature | Description ---------|------------ -[node](https://kubernetes.io/docs/concepts/architecture/nodes/) | A physical or virtual machine. In this tutorial, you'll run GKE or EKS instances and join them as worker nodes in three independent Kubernetes clusters, each in a different region. -[pod](http://kubernetes.io/docs/user-guide/pods/) | A pod is a group of one or more Docker containers. In this tutorial, each pod will run on a separate GKE or EKS instance and include one Docker container running a single CockroachDB node. You'll start with 3 pods in each region and grow to 4. -[StatefulSet](http://kubernetes.io/docs/concepts/abstractions/controllers/statefulsets/) | A group of pods treated as stateful units, where each pod has distinguishable network identity and always binds back to the same persistent storage on restart. StatefulSets are considered stable as of Kubernetes version 1.9 after reaching beta in version 1.5. -[persistent volume](http://kubernetes.io/docs/user-guide/persistent-volumes/) | A piece of networked storage (Persistent Disk on GCE, Elastic Block Store on AWS) mounted into a pod. The lifetime of a persistent volume is decoupled from the lifetime of the pod that's using it, ensuring that each CockroachDB node binds back to the same storage on restart.

    This tutorial assumes that dynamic volume provisioning is available. When that is not the case, [persistent volume claims](http://kubernetes.io/docs/user-guide/persistent-volumes/#persistentvolumeclaims) need to be created manually. -[CSR](https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/) | A CSR, or certificate signing request, is a request to have a TLS certificate verified by a certificate authority (CA). A CSR is issued for the CockroachDB node running in each pod, as well as each client as it connects to the Kubernetes cluster. -[RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) | RBAC, or role-based access control, is the system Kubernetes uses to manage permissions within the cluster. In order to take an action (e.g., `get` or `create`) on an API resource (e.g., a `pod`), the client must have a `Role` that allows it to do so. -[namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) | A namespace provides a scope for resources and names within a Kubernetes cluster. Names of resources must be unique within a namespace, but not across namespaces. Most Kubernetes client commands will use the `default` namespace by default, but can operate on resources in other namespaces as well. In this tutorial, CockroachDB pods will be deployed in their own namespace in each Kubernetes cluster. -[kubectl](https://kubernetes.io/docs/reference/kubectl/overview/) | `kubectl` is the command-line interface for running commands against Kubernetes clusters. -[kubectl context](https://kubernetes.io/docs/reference/kubectl/cheatsheet/#kubectl-context-and-configuration) | When multiple Kubernetes clusters are deployed on your account, `kubectl` "context" specifies a cluster to connect to. - -### UX differences from running in a single cluster - -These instructions create 3 StatefulSets that each run 3 CockroachDB pods in a separate Kubernetes cluster deployed in its own region. If you haven't often worked with multiple Kubernetes clusters, remember that `kubectl` commands are run against a cluster in a specific context. Either run `kubectl use-context ` frequently to switch contexts between commands, or append `--context ` to the commands you run to ensure they are run against the correct cluster. - -
    -Each Kubernetes cluster's DNS server is pointed at the other clusters' DNS servers so that DNS lookups for certain zone-scoped suffixes (e.g., `*.us-west1-a.svc.cluster.local`) can be deferred to the appropriate cluster's DNS server. To make this work, we create the StatefulSets in namespaces named after the region in which each Kubernetes cluster is deployed. To run a command against one of the pods, append `--namespace ` to your commands. Alternatively, run `kubectl config set-context --namespace ` to set the default namespace for a context. - -Because the CockroachDB pods run in a non-default namespace, client applications wanting to talk to CockroachDB from the default namespace would need to use a zone-scoped service name (e.g., `cockroachdb-public.us-west1-a`) rather than `cockroachdb-public`, as in a single-cluster setting. However, the setup script used by these instructions sets up an additional [`ExternalName` service](https://kubernetes.io/docs/concepts/services-networking/service/#externalname) in the default namespace such that the clients in the default namespace can simply talk to the `cockroachdb-public` address. -
    - -
    -To enable the pods to communicate across regions, we peer the VPCs in all 3 regions with each other and configure a CoreDNS service in each region to route DNS traffic to the appropriate pods. To make this work, we create the StatefulSets in namespaces named after the region in which each Kubernetes cluster is deployed. To run a command against one of the pods, append `--namespace ` to your commands. Alternatively, run `kubectl config set-context --namespace ` to set the default namespace for a context. -
    - -### Limitations - -{% include {{ page.version.version }}/orchestration/kubernetes-limitations.md %} - -
    -#### Exposing DNS servers - -{{site.data.alerts.callout_danger}} -Until December 2019, Google Cloud Platform restricted clients to using a load balancer in the same region. In the approach documented below, the DNS servers from each Kubernetes cluster are hooked together by exposing them via a load balanced IP address that is visible to the public Internet. None of the services in your Kubernetes cluster will be accessible publicly, but their names could leak out to a motivated attacker. - -You can now [enable global access](https://cloud.google.com/load-balancing/docs/internal/setting-up-internal#ilb-global-access) on Google Cloud Platform to allow clients from one region to use an internal load balancer in another. We will update this tutorial accordingly. -{{site.data.alerts.end}} -
    - -## Step 1. Start Kubernetes clusters - -Our multi-region deployment approach relies on pod IP addresses being routable across three distinct Kubernetes clusters and regions. Both the hosted Google Kubernetes Engine (GKE) and Amazon Elastic Kubernetes Service (EKS) satisfy this requirement. - -If you want to run on another cloud or on-premises, use this [basic network test](https://github.com/cockroachdb/cockroach/tree/master/cloud/kubernetes/multiregion#pod-to-pod-connectivity) to see if it will work. - -
    -1. Complete the **Before You Begin** steps described in the [Google Kubernetes Engine Quickstart](https://cloud.google.com/kubernetes-engine/docs/quickstart) documentation. - - This includes installing `gcloud`, which is used to create and delete Kubernetes Engine clusters, and `kubectl`, which is the command-line tool used to manage Kubernetes from your workstation. - - {{site.data.alerts.callout_success}}The documentation offers the choice of using Google's Cloud Shell product or using a local shell on your machine. Choose to use a local shell if you want to be able to view the DB Console using the steps in this guide.{{site.data.alerts.end}} - -1. From your local workstation, start the first Kubernetes cluster, specifying the [zone](https://cloud.google.com/compute/docs/regions-zones/) it should run in: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ gcloud container clusters create cockroachdb1 --zone= - ~~~ - - ~~~ - Creating cluster cockroachdb1...done. - ~~~ - - This creates GKE instances in the zone specified and joins them into a single Kubernetes cluster named `cockroachdb1`. - - {{site.data.alerts.callout_info}} - The process can take a few minutes, so do not move on to the next step until you see a `Creating cluster cockroachdb1...done` message and details about your cluster. - {{site.data.alerts.end}} - -1. Start the second Kubernetes cluster, specifying the [zone](https://cloud.google.com/compute/docs/regions-zones/) it should run in: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ gcloud container clusters create cockroachdb2 --zone= - ~~~ - - ~~~ - Creating cluster cockroachdb2...done. - ~~~ - -1. Start the third Kubernetes cluster, specifying the [zone](https://cloud.google.com/compute/docs/regions-zones/) it should run in: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ gcloud container clusters create cockroachdb3 --zone= - ~~~ - - ~~~ - Creating cluster cockroachdb3...done. - ~~~ - -1. Get the `kubectl` "contexts" for your clusters: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl config get-contexts - ~~~ - - ~~~ - CURRENT NAME CLUSTER AUTHINFO NAMESPACE - * gke_cockroach-shared_us-east1-b_cockroachdb1 gke_cockroach-shared_us-east1-b_cockroachdb1 gke_cockroach-shared_us-east1-b_cockroachdb1 - gke_cockroach-shared_us-west1-a_cockroachdb2 gke_cockroach-shared_us-west1-a_cockroachdb2 gke_cockroach-shared_us-west1-a_cockroachdb2 - gke_cockroach-shared_us-central1-a_cockroachdb3 gke_cockroach-shared_us-central1-a_cockroachdb3 gke_cockroach-shared_us-central1-a_cockroachdb3 - ~~~ - - {{site.data.alerts.callout_info}} - `kubectl` commands are run against the `CURRENT` context by default. You can change the current context with this command: - - {% include_cached copy-clipboard.html %} - ~~~ shell - kubectl config use-context - ~~~ - - When sending commands to another context, you need to use the `--context` flag to specify the context. For clarity, every `kubectl` command in this tutorial uses the `--context` flag to indicate the proper context. - {{site.data.alerts.end}} - -1. Get the email address associated with your Google Cloud account: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ gcloud info | grep Account - ~~~ - - ~~~ - Account: [your.google.cloud.email@example.org] - ~~~ - - {{site.data.alerts.callout_danger}} - This command returns your email address in all lowercase. However, in the next step, you must enter the address using the accurate capitalization. For example, if your address is YourName@example.com, you must use YourName@example.com and not yourname@example.com. - {{site.data.alerts.end}} - -1. For each Kubernetes cluster, [create the RBAC roles](https://cloud.google.com/kubernetes-engine/docs/how-to/role-based-access-control#prerequisites_for_using_role-based_access_control) CockroachDB needs for running on GKE, using the email address and relevant "context" name from the previous steps: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl create clusterrolebinding $USER-cluster-admin-binding --clusterrole=cluster-admin --user= --context - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl create clusterrolebinding $USER-cluster-admin-binding --clusterrole=cluster-admin --user= --context - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl create clusterrolebinding $USER-cluster-admin-binding --clusterrole=cluster-admin --user= --context - ~~~ -
    - -
    -1. Complete the steps described in the [EKS Getting Started](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html) documentation. - - This includes installing and configuring the AWS CLI and `eksctl`, which is the command-line tool used to create and delete Kubernetes clusters on EKS, and `kubectl`, which is the command-line tool used to manage Kubernetes from your workstation. - -1. From your local workstation, create three Kubernetes clusters. For each cluster, specify a unique region and a **non-overlapping** IP range for the VPC in CIDR notation (e.g., 10.0.0.0/16). Refer to the AWS documentation for valid [regions](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-available-regions) and [CIDR blocks](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Subnets.html#VPC_Sizing). - - {{site.data.alerts.callout_danger}} - In order to enable VPC peering between the regions, the CIDR blocks of the VPCs **must not** overlap. This value cannot change once the cluster has been created, so be sure that your IP ranges do not overlap. - {{site.data.alerts.end}} - - {{site.data.alerts.callout_success}} - To ensure that all 3 nodes can be placed into a different availability zone, you may want to first [confirm that at least 3 zones are available in the region](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#availability-zones-describe) for your account. - {{site.data.alerts.end}} - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ eksctl create cluster \ - --name cockroachdb1 \ - --nodegroup-name standard-workers \ - --node-type m5.xlarge \ - --nodes 3 \ - --node-ami auto \ - --region \ - --vpc-cidr - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ eksctl create cluster \ - --name cockroachdb2 \ - --nodegroup-name standard-workers \ - --node-type m5.xlarge \ - --nodes 3 \ - --node-ami auto \ - --region \ - --vpc-cidr - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ eksctl create cluster \ - --name cockroachdb3 \ - --nodegroup-name standard-workers \ - --node-type m5.xlarge \ - --nodes 3 \ - --node-ami auto \ - --region \ - --vpc-cidr - ~~~ - - Each command creates three EKS instances in a region, one for each CockroachDB node you will deploy. Note that each instance is assigned to a different availability zone in the region. - - In each region, the EKS instances are joined into a separate Kubernetes cluster: `cockroachdb1`, `cockroachdb2`, and `cockroachdb3`. The `--node-type` flag tells the node pool to use the [`m5.xlarge`](https://aws.amazon.com/ec2/instance-types/) instance type (4 vCPUs, 16 GB memory), which meets our [recommended CPU and memory configuration](recommended-production-settings.html#basic-hardware-recommendations). - - {{site.data.alerts.callout_info}} - Cluster provisioning usually takes between 10 and 15 minutes. Do not move on to the next step until you see a message like `[✔] EKS cluster "cockroachdb1" in "us-east-1" region is ready` for each cluster. - {{site.data.alerts.end}} - -1. Open the [Amazon EC2 console](https://console.aws.amazon.com/ec2/) and verify that three instances, using the node group name you specified, are running in each region. You will need to toggle between each region in the console. - -1. Get the context name for each of the 3 regions. When running `kubectl` commands against each region's cluster, you will need to specify the context name for that region. - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl config get-contexts - ~~~ - - ~~~ - CURRENT NAME CLUSTER AUTHINFO NAMESPACE - * maxroach@cockroachdb1.eu-central-1.eksctl.io cockroachdb1.eu-central-1.eksctl.io maxroach@cockroachdb1.eu-central-1.eksctl.io - maxroach@cockroachdb2.ca-central-1.eksctl.io cockroachdb2.ca-central-1.eksctl.io maxroach@cockroachdb2.ca-central-1.eksctl.io - maxroach@cockroachdb3.eu-north-1.eksctl.io cockroachdb3.eu-north-1.eksctl.io maxroach@cockroachdb3.eu-north-1.eksctl.io - ~~~ - - {{site.data.alerts.callout_info}} - `kubectl` commands are run against the "current" context by default. You can change the current context with `kubectl config use-context `. - - When running commands on another region, you need to use the `--context` flag to specify that region's context. For clarity, every `kubectl` command in this tutorial uses the `--context` flag to indicate the proper context. - {{site.data.alerts.end}} - -1. Create three namespaces, one corresponding to each region. The CockroachDB cluster in each region will run in this namespace. - - {% include_cached copy-clipboard.html %} - ~~~ shell - kubectl create namespace --context - ~~~ - - It's simplest for the namespace and region to have the same name, e.g.: - - ~~~ shell - kubectl create namespace eu-central-1 --context maxroach@cockroachdb1.eu-central-1.eksctl.io - ~~~ - - {{site.data.alerts.callout_info}} - `kubectl` commands are run against the namespace named `default` by default. You can change the default namespace for a given context with `kubectl config set-context --namespace `. - - For clarity, every `kubectl` command in this tutorial uses the `--namespace` flag to indicate the proper namespace. - {{site.data.alerts.end}} - -## Step 2. Configure your network - -### Set up VPC peering - -For pods to communicate across three separate Kubernetes clusters, the VPCs in all regions need to be peered. Network traffic can then be routed between the VPCs. For more information about VPC peering, see the [AWS documentation](https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html). - -1. Open the [Amazon VPC console](https://console.aws.amazon.com/vpc/) and note the ID of the VPC in each region. The VPC ID is found in the section called Your VPCs. - -1. Navigate to the Peering Connections section and [create a VPC peering connection](https://docs.aws.amazon.com/vpc/latest/peering/create-vpc-peering-connection.html#create-vpc-peering-connection-local) between each of the 3 regions. When you create a peering connection, you will first select a requester VPC in the current region and then an accepter VPC in another region, specified by pasting the VPC ID. - - {{site.data.alerts.callout_success}} - You need to create a total of 3 VPC peering connections between your 3 VPCs, which means switching regions at least once in the console. For example, if you are deploying in `eu-central-1`, `eu-north-1`, and `ca-central-1`, you can select `eu-central-1` in the console and create VPC peering connections to both `eu-north-1` and `ca-central-1`. Then switch to either `eu-north-1` or `ca-central-1` to create the VPC peering connection between those two regions. - {{site.data.alerts.end}} - -1. To complete the VPC peering connections, switch to each destination region and [accept the pending connection](https://docs.aws.amazon.com/vpc/latest/peering/create-vpc-peering-connection.html#accept-vpc-peering-connection) in the VPC console. - -1. For all 3 regions, navigate to the Route Tables section and find `PublicRouteTable`. [Update this route table](https://docs.aws.amazon.com/vpc/latest/peering/vpc-peering-routing.html) with 2 new entries that point traffic to the other 2 peered VPCs. The **Destination** should be the CIDR block of a destination region, and the **Target** should be the VPC peering connection between the current and the destination region. - -### Create inbound rules - -For each region, navigate to the Security Groups section of the [Amazon EC2 console](https://console.aws.amazon.com/ec2/) and locate the security group that enables communication between nodes in the cluster. It should have a name like `ClusterSharedNodeSecurityGroup`. [Add Custom TCP inbound rules](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html#adding-security-group-rule) to this security group to allow TCP communication on two ports: - -- `26257` for inter-node and client-node communication. This enables the nodes to work as a cluster, the load balancer to route traffic to the nodes, and applications to connect to the load balancer. -- `8080` for exposing the DB Console to the user, and for routing the load balancer to the health check endpoint. - -{{site.data.alerts.callout_info}} -Remember to create these inbound rules in all 3 regions. This enables CockroachDB to communicate across each Kubernetes cluster. -{{site.data.alerts.end}} - -#### Inter-region communication - - Field | Value --------|------------------- -Port Range | **26257** -Source | The IP range of each region's VPC in CIDR notation (e.g., 10.12.0.0/16) - -This important rule enables node communication between Kubernetes clusters in different regions. You need to create a separate rule for each region in your deployment. - -{% include {{ page.version.version }}/prod-deployment/aws-inbound-rules.md %} - -### Set up load balancing - -The Kubernetes cluster in each region needs to have a [Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html) pointed at its CoreDNS service, which you will configure in the next step. - -1. Upload our load balancer manifest [`dns-lb-eks.yaml`](https://github.com/cockroachdb/cockroach/blob/master/cloud/kubernetes/multiregion/eks/dns-lb-eks.yaml) to the Kubernetes clusters in all 3 regions: - - {% include_cached copy-clipboard.html %} - ~~~ shell - kubectl apply -f https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/multiregion/eks/dns-lb-eks.yaml --context - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - kubectl apply -f https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/multiregion/eks/dns-lb-eks.yaml --context - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - kubectl apply -f https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/multiregion/eks/dns-lb-eks.yaml --context - ~~~ - - You should see the load balancer appear in the Load Balancers section of the EC2 console in each region. This load balancer will route traffic to CoreDNS in the region. - -1. For each region, navigate to the Load Balancer section of the [EC2 console](https://console.aws.amazon.com/ec2/) and get the DNS name of the Network Load Balancer you created in the previous step. - -1. For each region's load balancer, look up the IP addresses mapped to the load balancer's DNS name: - - {% include_cached copy-clipboard.html %} - ~~~ shell - dig - ~~~ - - ~~~ - ... - - ;; ANSWER SECTION: - ac63a423fbf6231cba51235e1e51e6ec-132cf6c423e67123.elb.eu-central-1.amazonaws.com. 60 IN A ip1 - ac63a423fbf6231cba51235e1e51e6ec-132cf6c423e67123.elb.eu-central-1.amazonaws.com. 60 IN A ip2 - ac63a423fbf6231cba51235e1e51e6ec-132cf6c423e67123.elb.eu-central-1.amazonaws.com. 60 IN A ip3 - - ... - ~~~ - - `ip1`, `ip2`, and `ip3` correspond to the 3 availability zones in the region where Network Load Balancers have been set up. You will need these IP addresses when configuring your network in the next step. - -### Configure CoreDNS - -Each Kubernetes cluster has a [CoreDNS](https://coredns.io/) service that responds to DNS requests for pods in its region. CoreDNS can also forward DNS requests to pods in other regions. - -To enable traffic forwarding to CockroachDB pods in all 3 regions, you need to [modify the ConfigMap](https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/#coredns-configmap-options) for the CoreDNS Corefile in each region. - -1. Download and open our ConfigMap template [`configmap.yaml`](https://github.com/cockroachdb/cockroach/blob/master/cloud/kubernetes/multiregion/eks/configmap.yaml): - - {% include_cached copy-clipboard.html %} - ~~~ shell - curl -O https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/multiregion/eks/configmap.yaml - ~~~ - -1. After [obtaining the IP addresses of EKS instances](#set-up-load-balancing) in all 3 regions, you can use this information to define a **separate ConfigMap for each region**. Each unique ConfigMap lists the forwarding addresses for the pods in the 2 other regions. - - ~~~ - ... - region2.svc.cluster.local:53 { # <---- Modify - log - errors - ready - cache 10 - forward . ip1 ip2 ip3 { # <---- Modify - force_tcp - } - } - region3.svc.cluster.local:53 { # <---- Modify - log - errors - ready - cache 10 - forward . ip1 ip2 ip3 { # <---- Modify - force_tcp - } - } - ~~~ - - For each region, modify `configmap.yaml` by replacing: - -
    • region2 and region3 with the namespaces in which the CockroachDB pods will run in the other 2 regions. You defined these namespaces after starting the Kubernetes clusters.
    • ip1, ip2, and ip3 with the IP addresses of the Network Load Balancers in the region, which you looked up in the previous step.
    - - You will end up with 3 different ConfigMaps. Give each ConfigMap a unique filename like `configmap-1.yaml`. - - {{site.data.alerts.callout_info}} - If you configure your Network Load Balancer to support UDP traffic, you can drop the `force_tcp` line. - {{site.data.alerts.end}} - -1. For each region, first back up the existing ConfigMap: - - {% include_cached copy-clipboard.html %} - ~~~ shell - kubectl -n kube-system get configmap coredns -o yaml > - ~~~ - - Then apply the new ConfigMap: - - {% include_cached copy-clipboard.html %} - ~~~ - kubectl apply -f --context - ~~~ - -1. For each region, check that your CoreDNS settings were applied: - - {% include_cached copy-clipboard.html %} - ~~~ shell - kubectl get -n kube-system cm/coredns --export -o yaml --context - ~~~ - -### Exclude VPCs from SNAT - -You will need to tell AWS to exclude your VPCs from [source network address translation (SNAT)](https://docs.aws.amazon.com/eks/latest/userguide/external-snat.html). This ensures that cross-VPC traffic is handled correctly by AWS while still allowing access to the public internet from the pods. - -Set `AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS` to recognize the values of your 3 CIDR blocks. Do this for all 3 regions: - -{% include_cached copy-clipboard.html %} -~~~ -kubectl set env ds aws-node -n kube-system AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS="cidr1,cidr2,cidr3" --context -~~~ - -Remember that these are the CIDR blocks you chose when [starting your Kubernetes clusters](#step-1-start-kubernetes-clusters). You can also get the IP range of a VPC by opening the [Amazon VPC console](https://console.aws.amazon.com/vpc/) and finding the VPC listed in the section called Your VPCs. - -{{site.data.alerts.callout_info}} -If you plan to run your instances exclusively on private subnets, set the following environment variable instead on each region: kubectl set env ds aws-node -n kube-system AWS_VPC_K8S_CNI_EXTERNALSNAT=true --context \ -{{site.data.alerts.end}} -
    - -
    -## Step 2. Start CockroachDB - -1. Create a directory and download the required script and configuration files into it: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ mkdir multiregion - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cd multiregion - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ curl -OOOOOOOOO \ - https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/multiregion/{README.md,client-secure.yaml,cluster-init-secure.yaml,cockroachdb-statefulset-secure.yaml,dns-lb.yaml,example-app-secure.yaml,external-name-svc.yaml,setup.py,teardown.py} - ~~~ - -2. Retrieve the `kubectl` "contexts" for your clusters: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl config get-contexts - ~~~ - - At the top of the `setup.py` script, fill in the `contexts` map with the zones of your clusters and their "context" names, e.g.: - - ~~~ - contexts = { - 'us-east1-b': 'gke_cockroach-shared_us-east1-b_cockroachdb1', - 'us-west1-a': 'gke_cockroach-shared_us-west1-a_cockroachdb2', - 'us-central1-a': 'gke_cockroach-shared_us-central1-a_cockroachdb3', - } - ~~~ - -3. In the `setup.py` script, fill in the `regions` map with the zones and corresponding regions of your clusters, for example: - - ~~~ - $ regions = { - 'us-east1-b': 'us-east1', - 'us-west1-a': 'us-west1', - 'us-central1-a': 'us-central1', - } - ~~~ - - Setting `regions` is optional, but recommended, because it improves CockroachDB's ability to diversify data placement if you use more than one zone in the same region. If you aren't specifying regions, just leave the map empty. - -4. If you haven't already, [install CockroachDB locally and add it to your `PATH`](install-cockroachdb.html). The `cockroach` binary will be used to generate certificates. - - If the `cockroach` binary is not on your `PATH`, in the `setup.py` script, set the `cockroach_path` variable to the path to the binary. - -5. Optionally, to optimize your deployment for better performance, review [CockroachDB Performance on Kubernetes](kubernetes-performance.html) and make the desired modifications to the `cockroachdb-statefulset-secure.yaml` file. - -6. Run the `setup.py` script: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ python setup.py - ~~~ - - As the script creates various resources and creates and initializes the CockroachDB cluster, you'll see a lot of output, eventually ending with `job "cluster-init-secure" created`. - -7. Confirm that the CockroachDB pods in each cluster say `1/1` in the `READY` column, indicating that they've successfully joined the cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get pods --selector app=cockroachdb --all-namespaces --context - ~~~ - - ~~~ - NAMESPACE NAME READY STATUS RESTARTS AGE - us-east1-b cockroachdb-0 1/1 Running 0 14m - us-east1-b cockroachdb-1 1/1 Running 0 14m - us-east1-b cockroachdb-2 1/1 Running 0 14m - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get pods --selector app=cockroachdb --all-namespaces --context - ~~~ - - ~~~ - NAMESPACE NAME READY STATUS RESTARTS AGE - us-central1-a cockroachdb-0 1/1 Running 0 14m - us-central1-a cockroachdb-1 1/1 Running 0 14m - us-central1-a cockroachdb-2 1/1 Running 0 14m - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get pods --selector app=cockroachdb --all-namespaces --context - ~~~ - - ~~~ - NAMESPACE NAME READY STATUS RESTARTS AGE - us-west1-a cockroachdb-0 1/1 Running 0 14m - us-west1-a cockroachdb-1 1/1 Running 0 14m - us-west1-a cockroachdb-2 1/1 Running 0 14m - ~~~ - - If you notice that only one of the Kubernetes clusters' pods are marked as `READY`, you likely also need to configure a network firewall rule that will allow the pods in the different clusters to talk to each other. You can run the following command to create a firewall rule allowing traffic on port 26257 (the port used by CockroachDB for inter-node traffic) within your private GCE network. It will not allow any traffic in from outside your private network: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ gcloud compute firewall-rules create allow-cockroach-internal --allow=tcp:26257 --source-ranges=10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 - ~~~ - - ~~~ - Creating firewall...done. - NAME NETWORK DIRECTION PRIORITY ALLOW DENY - allow-cockroach-internal default INGRESS 1000 tcp:26257 - ~~~ -
    - -
    -## Step 3. Start CockroachDB - -### Generate certificates - -{{site.data.alerts.callout_info}} -The below steps use [`cockroach cert` commands](cockroach-cert.html) to quickly generate and sign the CockroachDB node and client certificates. Read our [Authentication](authentication.html#using-digital-certificates-with-cockroachdb) docs to learn about other methods of signing certificates. -{{site.data.alerts.end}} - -1. Create two directories: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ mkdir certs my-safe-directory - ~~~ - - Directory | Description - ----------|------------ - `certs` | You'll generate your CA certificate and all node and client certificates and keys in this directory. - `my-safe-directory` | You'll generate your CA key in this directory and then reference the key when generating node and client certificates. - -1. Create the CA certificate and key pair: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach cert create-ca \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key - ~~~ - -1. Create a client certificate and key pair for the root user: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach cert create-client \ - root \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key - ~~~ - -1. For all 3 regions, upload the client certificate and key to the Kubernetes cluster as a secret. - - {{site.data.alerts.callout_success}} - Specify the namespace in which the CockroachDB pods will run. You defined these namespaces after [starting your Kubernetes clusters](#step-1-start-kubernetes-clusters). - {{site.data.alerts.end}} - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl create secret \ - generic cockroachdb.client.root \ - --from-file=certs \ - --context \ - --namespace - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl create secret \ - generic cockroachdb.client.root \ - --from-file=certs \ - --context \ - --namespace - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl create secret \ - generic cockroachdb.client.root \ - --from-file=certs \ - --context \ - --namespace - ~~~ - -1. Create the certificate and key pair for your CockroachDB nodes in one region, substituting `` in this command with the appropriate namespace: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach cert create-node \ - localhost 127.0.0.1 \ - cockroachdb-public \ - cockroachdb-public. \ - cockroachdb-public..svc.cluster.local \ - *.cockroachdb \ - *.cockroachdb. \ - *.cockroachdb..svc.cluster.local \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key - ~~~ - -1. Upload the node certificate and key to the Kubernetes cluster as a secret, specifying the appropriate context and namespace: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl create secret \ - generic cockroachdb.node \ - --from-file=certs - --context \ - --namespace - ~~~ - -1. Repeat the previous 2 steps for your 2 remaining regions. You may need to delete the local `node.crt` and `node.key` in your `certs` directory before generating a new node certificate and key pair. - -1. For all 3 regions, check that the secrets were created on the cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get secrets --context - ~~~ - - ~~~ - NAME TYPE DATA AGE - cockroachdb-token-db5wp kubernetes.io/service-account-token 3 1m - cockroachdb.client.root Opaque 5 1m - cockroachdb.node Opaque 6 1m - default-token-65489 kubernetes.io/service-account-token 3 5m - ~~~ - -### Create StatefulSets - -1. Download and open our [multi-region StatefulSet configuration](https://github.com/cockroachdb/cockroach/blob/master/cloud/kubernetes/multiregion/eks/cockroachdb-statefulset-secure-eks.yaml). You'll save three versions of this file locally, one for each set of 3 CockroachDB nodes per region. - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ curl -O https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/multiregion/eks/cockroachdb-statefulset-secure-eks.yaml - ~~~ - - Look for **TODO** comments in the file. These highlight fields you need to define before deploying your StatefulSet. - -1. Fill in the namespace where CockroachDB nodes will run in region 1. - - ~~~ - namespace: - ~~~ - -1. Allocate a memory request and limit to CockroachDB on each pod, using the `resources` object in the CockroachDB `containers` spec. - - {{site.data.alerts.callout_success}} - These values should be appropriate for the instances that you have provisioned. Run `kubectl describe nodes` to see the available resources. - {{site.data.alerts.end}} - - For example, to allocate 8Gi of memory to CockroachDB in each pod: - - ~~~ - resources: - requests: - memory: "8Gi" - limits: - memory: "8Gi" - ~~~ - - {{site.data.alerts.callout_danger}} - If you do not specify a memory request, no memory will be allocated to CockroachDB. If you do not specify a memory limit, the Kubernetes scheduler will allocate the maximum possible amount. - {{site.data.alerts.end}} - -1. The StatefulSet configuration includes a [`cockroach start`](cockroach-start.html) command that creates the nodes on the Kubernetes pods. - - In the `--locality` flag, name `region` after region 1. This can technically be an arbitrary value, but it's simplest to use the CockroachDB namespace in region 1. - - ~~~ - --locality=region=,az=$(cat /etc/cockroach-env/zone),dns=$(hostname -f) - ~~~ - - In the `--join` flag, update the Cockroach node addresses in all 3 regions with their corresponding namespaces. - - ~~~ - --join cockroachdb-0.cockroachdb.,cockroachdb-1.cockroachdb.,cockroachdb-2.cockroachdb.,cockroachdb-0.cockroachdb.,cockroachdb-1.cockroachdb.,cockroachdb-2.cockroachdb.,cockroachdb-0.cockroachdb.,cockroachdb-1.cockroachdb.,cockroachdb-2.cockroachdb. - ~~~ - -1. Save this StatefulSet configuration, giving it a filename like `cockroachdb-statefulset-secure-eks-1.yaml`. - -1. Create and save a StatefulSet configuration for each of the other 2 regions in the same way, being sure to use the correct namespaces for those regions in steps 2 and 4. - -1. Deploy the StatefulSets in each of your 3 regions: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl create -f --context --namespace - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl create -f --context --namespace - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl create -f --context --namespace - ~~~ - -1. Run `cockroach init` on one of the pods to complete the node startup process and have them join together as a cluster: - - {% include_cached copy-clipboard.html %} - ~~~ - kubectl exec \ - --context \ - --namespace \ - -it cockroachdb-0 \ - -- /cockroach/cockroach init \ - --certs-dir=/cockroach/cockroach-certs - ~~~ - - ~~~ - Cluster successfully initialized - ~~~ - -1. Confirm that cluster initialization has completed successfully in each region. The job should be considered successful and the Kubernetes pods should soon be considered `Ready`: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get pods --context --namespace - ~~~ - - ~~~ - NAMESPACE NAME READY STATUS RESTARTS AGE - eu-central-1 cockroachdb-0 1/1 Running 0 12m - eu-central-1 cockroachdb-1 1/1 Running 0 12m - eu-central-1 cockroachdb-2 1/1 Running 0 12m - ~~~ -
    - -{{site.data.alerts.callout_success}} -In each Kubernetes cluster, the StatefulSet configuration sets all CockroachDB nodes to write to `stderr`, so if you ever need access to a pod/node's logs to troubleshoot, use `kubectl logs --context --namespace ` rather than checking the log on the persistent volume. -{{site.data.alerts.end}} - -
    -## Step 3. Use the built-in SQL client - -1. Use the `client-secure.yaml` file to launch a pod and keep it running indefinitely, specifying the context of the Kubernetes cluster to run it in: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl create -f client-secure.yaml --context - ~~~ - - ~~~ - pod "cockroachdb-client-secure" created - ~~~ - - The pod uses the `root` client certificate created earlier by the `setup.py` script. Note that this will work from any of the three Kubernetes clusters as long as you use the correct namespace and context combination. -
    - -
    -## Step 4. Use the built-in SQL client - -1. Use the `client-secure.yaml` file to launch a pod and keep it running indefinitely, specifying the context of the Kubernetes cluster and namespace of the CockroachDB pods to run it in: - - {% include_cached copy-clipboard.html %} - ~~~ shell - kubectl create -f https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/multiregion/client-secure.yaml --context --namespace - ~~~ - - ~~~ - pod "cockroachdb-client-secure" created - ~~~ - - The pod uses the `root` client certificate you [generated earlier](#generate-certificates). Note that this will work from any of the three Kubernetes clusters as long as you use the correct namespace and context combination. -
    - -1. Get a shell into the pod and start the CockroachDB [built-in SQL client](cockroach-sql.html), again specifying the namespace and context of the Kubernetes cluster where the pod is running: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl exec -it cockroachdb-client-secure --context --namespace -- ./cockroach sql --certs-dir=/cockroach-certs --host=cockroachdb-public - ~~~ - - ~~~ - # Welcome to the cockroach SQL interface. - # All statements must be terminated by a semicolon. - # To exit: CTRL + D. - # - # Server version: CockroachDB CCL v2.0.5 (x86_64-unknown-linux-gnu, built 2018/08/13 17:59:42, go1.10) (same version as client) - # Cluster ID: 99346e82-9817-4f62-b79b-fdd5d57f8bda - # - # Enter \? for a brief introduction. - # - warning: no current database set. Use SET database = to change, CREATE DATABASE to make a new database. - root@cockroachdb-public:26257/> - ~~~ - -1. Run some basic [CockroachDB SQL statements](learn-cockroachdb-sql.html): - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE DATABASE bank; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE TABLE bank.accounts (id INT PRIMARY KEY, balance DECIMAL); - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > INSERT INTO bank.accounts VALUES (1, 1000.50); - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SELECT * FROM bank.accounts; - ~~~ - - ~~~ - +----+---------+ - | id | balance | - +----+---------+ - | 1 | 1000.5 | - +----+---------+ - (1 row) - ~~~ - -1. [Create a user with a password](create-user.html#create-a-user-with-a-password): - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE USER roach WITH PASSWORD 'Q7gc8rEdS'; - ~~~ - - You will need this username and password to access the DB Console in the next step. - -1. Exit the SQL shell and pod: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > \q - ~~~ - - The pod will continue running indefinitely, so any time you need to reopen the built-in SQL client or run any other [`cockroach` client commands](cockroach-commands.html) (e.g., `cockroach node`), repeat step 2 using the appropriate command. - - If you'd prefer to delete the pod and recreate it when needed, run: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl delete pod cockroachdb-client-secure --context - ~~~ - -
    -## Step 4. Access the DB Console -
    - -
    -## Step 5. Access the DB Console -
    - -To access the cluster's [DB Console](ui-overview.html): - -1. On secure clusters, [certain pages of the DB Console](ui-overview.html#db-console-access) can only be accessed by `admin` users. - - Get a shell into the pod with the `cockroach` binary created earlier and start the CockroachDB [built-in SQL client](cockroach-sql.html): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl exec -it cockroachdb-client-secure --context --namespace -- ./cockroach sql --certs-dir=/cockroach-certs --host=cockroachdb-public - ~~~ - -1. Assign `roach` to the `admin` role (you only need to do this once): - - {% include_cached copy-clipboard.html %} - ~~~ sql - > GRANT admin TO roach; - ~~~ - -1. Exit the SQL shell and pod: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > \q - ~~~ - -1. Port-forward from your local machine to a pod in one of your Kubernetes clusters: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl port-forward cockroachdb-0 8080 --context --namespace - ~~~ - - ~~~ - Forwarding from 127.0.0.1:8080 -> 8080 - ~~~ - - {{site.data.alerts.callout_info}} - The `port-forward` command must be run on the same machine as the web browser in which you want to view the DB Console. If you have been running these commands from a cloud instance or other non-local shell, you will not be able to view the UI without configuring `kubectl` locally and running the above `port-forward` command on your local machine. - {{site.data.alerts.end}} - -1. Go to https://localhost:8080 and log in with the username and password created in the [Use the built-in SQL client](#step-3-use-the-built-in-sql-client) step. - -1. In the UI, check the [**Node List**](ui-cluster-overview-page.html#node-list) to verify that all nodes are running, open the [**Databases** page](ui-databases-page.html) to verify that `bank` is listed, and open the [**Network Latency** page](ui-network-latency-page.html) to see the performance of your CockroachDB cluster across 3 regions. - -
    -## Step 5. Simulate datacenter failure -
    - -
    -## Step 6. Simulate datacenter failure -
    - -One of the major benefits of running a multi-region CockroachDB cluster is that an entire datacenter or region can go down without affecting the availability of the cluster as a whole. - -To see this in action: - -1. Scale down one of the StatefulSets to zero pods, specifying the namespace and context of the Kubernetes cluster where it's running: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl scale statefulset cockroachdb --replicas=0 --context --namespace - ~~~ - - ~~~ - statefulset "cockroachdb" scaled - ~~~ - -1. In the DB Console, the **Cluster Overview** will soon show the three nodes from that region as **Suspect**. If you wait for 5 minutes or more, they will be listed as **Dead**. Note that even though there are three dead nodes, the other nodes are all healthy, and any clients using the database in the other regions will continue to work just fine. - -1. When you're done verifying that the cluster still fully functions with one of the regions down, you can bring the region back up by running: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl scale statefulset cockroachdb --replicas=3 --context --namespace - ~~~ - - ~~~ - statefulset "cockroachdb" scaled - ~~~ - -
    -## Step 6. Maintain the cluster -
    - -
    -## Step 7. Maintain the cluster -
    - -- [Scale the cluster](#scale-the-cluster) -- [Upgrade the cluster](#upgrade-the-cluster) -- [Stop the cluster](#stop-the-cluster) - -### Scale the cluster - -Each of your Kubernetes clusters contains 3 instances that can run CockroachDB pods. It's easy to scale a cluster to run more pods. To ensure that you do not have two CockroachDB pods on the same instance (as recommended in our [production best practices](recommended-production-settings.html)), you need to first add a new instance and then edit your StatefulSet configuration to add another pod. - -
    -1. [Resize your Kubernetes cluster.](https://cloud.google.com/kubernetes-engine/docs/how-to/resizing-a-cluster) -
    - -
    -1. [Resize your Kubernetes cluster.](https://eksctl.io/usage/managing-nodegroups/#scaling) -
    - -1. Use the `kubectl scale` command to add a pod to the StatefulSet in the Kubernetes cluster where you want to add a CockroachDB node: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl scale statefulset cockroachdb --replicas=4 --context --namespace - ~~~ - - ~~~ - statefulset "cockroachdb" scaled - ~~~ - -1. Verify that a fourth pod was added successfully: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get pods --context --namespace - ~~~ - - ~~~ - NAME READY STATUS RESTARTS AGE - cockroachdb-0 1/1 Running 0 1h - cockroachdb-1 1/1 Running 0 1h - cockroachdb-2 1/1 Running 0 7m - cockroachdb-3 1/1 Running 0 44s - cockroachdb-client-secure 1/1 Running 0 26m - ~~~ - -### Upgrade the cluster - -We strongly recommend that you regularly upgrade your CockroachDB version in order to pick up bug fixes, performance improvements, and new features. - -The upgrade process on Kubernetes is a [staged update](https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#staging-an-update) in which the Docker image is applied to the pods one at a time, with each pod being stopped and restarted in turn. This is to ensure that the cluster remains available during the upgrade. - -1. Verify that you can upgrade. - - To upgrade to a new major version, you must first be on a production release of the previous version. The release does not need to be the latest production release of the previous version, but it must be a production [release](../releases/index.html) and not a testing release (alpha/beta). - - Therefore, in order to upgrade to v21.1, you must be on a production release of v20.2. - - 1. If you are upgrading to v21.1 from a production release earlier than v20.2, or from a testing release (alpha/beta), first [upgrade to a production release of v20.2](../v20.2/orchestrate-cockroachdb-with-kubernetes-multi-cluster.html#upgrade-the-cluster). Be sure to complete all the steps. - - 1. Then return to this page and perform a second upgrade to v21.1. - - 1. If you are upgrading from any production release of v20.2, or from any earlier v21.1 release, you do not have to go through intermediate releases; continue to step 2. - -1. Verify the overall health of your cluster using the [DB Console](ui-overview.html). On the **Overview**: - - Under **Node Status**, make sure all nodes that should be live are listed as such. If any nodes are unexpectedly listed as suspect or dead, identify why the nodes are offline and either restart them or decommission them before beginning your upgrade. If there are dead and non-decommissioned nodes in your cluster, it will not be possible to finalize the upgrade (either automatically or manually). - - Under **Replication Status**, make sure there are 0 under-replicated and unavailable ranges. Otherwise, performing a rolling upgrade increases the risk that ranges will lose a majority of their replicas and cause cluster unavailability. Therefore, it's important to identify and resolve the cause of range under-replication and/or unavailability before beginning your upgrade. - - In the **Node List**: - - Make sure all nodes are on the same version. If any nodes are behind, upgrade them to the cluster's current version first, and then start this process over. - - Make sure capacity and memory usage are reasonable for each node. Nodes must be able to tolerate some increase in case the new version uses more resources for your workload. Also go to **Metrics > Dashboard: Hardware** and make sure CPU percent is reasonable across the cluster. If there's not enough headroom on any of these metrics, consider [adding nodes](#scale-the-cluster) to your cluster before beginning your upgrade. - -1. Review the [backward-incompatible changes in v21.1](../releases/v21.1.html#v21-1-0-backward-incompatible-changes) and [deprecated features](../releases/v21.1.html#v21-1-0-deprecations). If any affect your deployment, make the necessary changes before starting the rolling upgrade to v21.1. - -1. Decide how the upgrade will be finalized. - - By default, after all nodes are running the new version, the upgrade process will be **auto-finalized**. This will enable certain [features and performance improvements introduced in v21.2](upgrade-cockroach-version.html#features-that-require-upgrade-finalization). After finalization, however, it will no longer be possible to perform a downgrade to v21.1. In the event of a catastrophic failure or corruption, the only option is to start a new cluster using the old binary and then restore from a [backup](take-full-and-incremental-backups.html) created prior to the upgrade. For this reason, **we recommend disabling auto-finalization** so you can monitor the stability and performance of the upgraded cluster before finalizing the upgrade, but note that you will need to follow all of the subsequent directions, including the manual finalization in a later step. - - {{site.data.alerts.callout_info}} - Finalization only applies when performing a major version upgrade (for example, from v20.2.x to v21.1). Patch version upgrades (for example, within the v21.1.x series) can always be downgraded. - {{site.data.alerts.end}} - - 1. Get a shell into the pod with the `cockroach` binary created earlier and start the CockroachDB [built-in SQL client](cockroach-sql.html): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl exec -it cockroachdb-client-secure --context --namespace -- ./cockroach sql --certs-dir=/cockroach-certs --host=cockroachdb-public - ~~~ - - 2. Set the `cluster.preserve_downgrade_option` [cluster setting](cluster-settings.html): - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING cluster.preserve_downgrade_option = '20.2'; - ~~~ - -2. For each Kubernetes cluster, kick off the upgrade process by changing the desired Docker image. To do so, pick the version that you want to upgrade to, then run the following command, replacing "VERSION" with your desired new version and specifying the relevant namespace and "context" name for the Kubernetes cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl patch statefulset cockroachdb --namespace --context --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"cockroachdb/cockroach:VERSION"}]' - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl patch statefulset cockroachdb --namespace --context --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"cockroachdb/cockroach:VERSION"}]' - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl patch statefulset cockroachdb --namespace --context --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"cockroachdb/cockroach:VERSION"}]' - ~~~ - -3. If you then check the status of the pods in each Kubernetes cluster, you should see one of them being restarted: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get pods --selector app=cockroachdb --all-namespaces --context - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get pods --selector app=cockroachdb --all-namespaces --context - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get pods --selector app=cockroachdb --all-namespaces --context - ~~~ - - This will continue until all of the pods have restarted and are running the new image. - -4. If you disabled auto-finalization earlier, monitor the stability and performance of your cluster until you are comfortable with the upgrade (generally at least a day). - - If you decide to roll back the upgrade, repeat the rolling restart procedure with the old binary. - - {{site.data.alerts.callout_info}} - This is only possible when performing a major version upgrade (for example, from v20.2.x to v21.1). Patch version upgrades (for example, within the v21.1.x series) are auto-finalized. - {{site.data.alerts.end}} - - To finalize the upgrade, re-enable auto-finalization: - - 1. Get a shell into the pod with the `cockroach` binary created earlier and start the CockroachDB [built-in SQL client](cockroach-sql.html): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl exec -it cockroachdb-client-secure --context --namespace -- ./cockroach sql --certs-dir=/cockroach-certs --host=cockroachdb-public - ~~~ - - 2. Re-enable auto-finalization: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > RESET CLUSTER SETTING cluster.preserve_downgrade_option; - ~~~ - -### Stop the cluster - -
    -1. To delete all of the resources created in your clusters, copy the `contexts` map from `setup.py` into `teardown.py`, and then run `teardown.py`: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ python teardown.py - ~~~ - - ~~~ - namespace "us-east1-b" deleted - service "kube-dns-lb" deleted - configmap "kube-dns" deleted - pod "kube-dns-5dcfcbf5fb-l4xwt" deleted - pod "kube-dns-5dcfcbf5fb-tddp2" deleted - namespace "us-west1-a" deleted - service "kube-dns-lb" deleted - configmap "kube-dns" deleted - pod "kube-dns-5dcfcbf5fb-8csc9" deleted - pod "kube-dns-5dcfcbf5fb-zlzn7" deleted - namespace "us-central1-a" deleted - service "kube-dns-lb" deleted - configmap "kube-dns" deleted - pod "kube-dns-5dcfcbf5fb-6ngmw" deleted - pod "kube-dns-5dcfcbf5fb-lcfxd" deleted - ~~~ - -2. Stop each Kubernetes cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ gcloud container clusters delete cockroachdb1 --zone= - ~~~ - - ~~~ - Deleting cluster cockroachdb1...done. - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ gcloud container clusters delete cockroachdb2 --zone= - ~~~ - - ~~~ - Deleting cluster cockroachdb2...done. - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ gcloud container clusters delete cockroachdb3 --zone= - ~~~ - - ~~~ - Deleting cluster cockroachdb3...done. - ~~~ -
    - -
    -1. In each region, delete all of the resources associated with the `cockroachdb` label, including the logs, and remote persistent volumes: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl delete pods,statefulsets,services,persistentvolumeclaims,persistentvolumes,poddisruptionbudget,jobs,rolebinding,clusterrolebinding,role,clusterrole,serviceaccount -l app=cockroachdb --context - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl delete pods,statefulsets,services,persistentvolumeclaims,persistentvolumes,poddisruptionbudget,jobs,rolebinding,clusterrolebinding,role,clusterrole,serviceaccount -l app=cockroachdb --context - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl delete pods,statefulsets,services,persistentvolumeclaims,persistentvolumes,poddisruptionbudget,jobs,rolebinding,clusterrolebinding,role,clusterrole,serviceaccount -l app=cockroachdb --context - ~~~ - - ~~~ - pod "cockroachdb-0" deleted - pod "cockroachdb-1" deleted - pod "cockroachdb-2" deleted - service "cockroachdb" deleted - service "cockroachdb-public" deleted - persistentvolumeclaim "datadir-cockroachdb-0" deleted - persistentvolumeclaim "datadir-cockroachdb-1" deleted - persistentvolumeclaim "datadir-cockroachdb-2" deleted - poddisruptionbudget.policy "cockroachdb-budget" deleted - rolebinding.rbac.authorization.k8s.io "cockroachdb" deleted - clusterrolebinding.rbac.authorization.k8s.io "cockroachdb" deleted - role.rbac.authorization.k8s.io "cockroachdb" deleted - clusterrole.rbac.authorization.k8s.io "cockroachdb" deleted - serviceaccount "cockroachdb" deleted - ~~~ - -1. Delete the pod created for `cockroach` client commands, if you didn't do so earlier: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl delete pod cockroachdb-client-secure --context - ~~~ - - ~~~ - pod "cockroachdb-client-secure" deleted - ~~~ - -1. Get the names of the secrets you created on each cluster. These should be identical in all 3 regions: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl get secrets --context - ~~~ - - ~~~ - NAME TYPE DATA AGE - cockroachdb.client.root Opaque 5 1d - cockroachdb.node Opaque 6 1d - ... - ~~~ - -1. Delete the secrets that you created: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl delete secrets cockroachdb.client.root cockroachdb.node --context - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl delete secrets cockroachdb.client.root cockroachdb.node --context - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ kubectl delete secrets cockroachdb.client.root cockroachdb.node --context - ~~~ - -1. Stop Kubernetes in each region: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ eksctl delete cluster --name cockroachdb1 --region - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ eksctl delete cluster --name cockroachdb2 --region - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ eksctl delete cluster --name cockroachdb3 --region - ~~~ - -1. Open the [AWS CloudFormation console](https://console.aws.amazon.com/cloudformation/home) to verify that the stacks were successfully deleted in each region. - -{{site.data.alerts.callout_danger}} -If you stop Kubernetes without first deleting the persistent volumes, they will still exist in your cloud project. -{{site.data.alerts.end}} -
    - -## See also - -- [Kubernetes Single-Cluster Deployment](deploy-cockroachdb-with-kubernetes.html) -- [Kubernetes Performance Guide](kubernetes-performance.html) -{% include {{ page.version.version }}/prod-deployment/prod-see-also.md %} diff --git a/src/current/v21.1/orchestration.md b/src/current/v21.1/orchestration.md deleted file mode 100644 index e2a0275de90..00000000000 --- a/src/current/v21.1/orchestration.md +++ /dev/null @@ -1,26 +0,0 @@ ---- -title: Orchestration -summary: Learn how to run CockroachDB with popular open-source orchestration systems. -toc: false -canonical: /stable/kubernetes-overview ---- - -Orchestration systems automate the deployment, scaling, and management of containerized applications. - -Use the following guides to run CockroachDB with popular open-source orchestration systems: - -- [Kubernetes Single-Cluster Deployment](deploy-cockroachdb-with-kubernetes.html) -- [Operate CockroachDB on Kubernetes](operate-cockroachdb-kubernetes.html) -- [Monitor CockroachDB on Kubernetes](monitor-cockroachdb-kubernetes.html) -- [OpenShift Single-Cluster Deployment](deploy-cockroachdb-with-kubernetes-openshift.html) -- [Kubernetes Multi-Cluster Deployment](orchestrate-cockroachdb-with-kubernetes-multi-cluster.html) -- [Kubernetes Performance Optimization](kubernetes-performance.html) - -{% include cockroachcloud/use-cockroachcloud-instead.md %} - -## See also - -- [Production Checklist](recommended-production-settings.html) -- [Manual Deployment](manual-deployment.html) -- [Monitoring and Alerting](monitoring-and-alerting.html) -- [Local Deployment](start-a-local-cluster.html) diff --git a/src/current/v21.1/order-by.md b/src/current/v21.1/order-by.md deleted file mode 100644 index fbb17ce0d1b..00000000000 --- a/src/current/v21.1/order-by.md +++ /dev/null @@ -1,283 +0,0 @@ ---- -title: ORDER BY -summary: The ORDER BY clause controls the order in which rows are returned or processed. -toc: true ---- - -The `ORDER BY` clause controls the order in which rows are returned or -processed. It can be used in any [selection -query](selection-queries.html), including -as operand of [`INSERT`](insert.html) or [`UPSERT`](upsert.html), as -well as with [`DELETE`](delete.html) and [`UPDATE`](update.html) -statements. - - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/sort_clause.html %} -
    - -## Parameters - -The `ORDER BY` clause takes a comma-separated list of ordering specifications. -Each ordering specification is composed of a column selection followed optionally -by the keyword `ASC` or `DESC`. - -Each **column selection** can take one of the following forms: - -- A simple column selection, determined as follows: - 1. The name of a column label configured with `AS` earlier in the [`SELECT` clause](select-clause.html). This uses the value computed by the `SELECT` clause as the sorting key. - 2. A positive integer number, designating one of the columns in the data source, either the `FROM` clause of the `SELECT` clause where it happens or the table being written to by `DELETE` or `UPDATE`. This uses the corresponding input value from the data source to use as the sorting key. - 3. An arbitrary [scalar expression](scalar-expressions.html). This uses the result of evaluating that expression as the sorting key. -- The notation `PRIMARY KEY `. This uses the primary key column(s) of the given table as sorting key. This table must be part of the data source. -- The notation `INDEX @`. This uses the columns indexed by the given index as sorting key. This table must be part of the data source. - -The optional keyword `ASC` after a column selection indicates to use -the sorting key as-is, and thus is meaningless. - -The optional keyword `DESC` inverts the direction of the column(s) -selected by the selection that immediately precedes. - -CockroachDB supports `NULLS FIRST` and `NULLS LAST` in [`ORDER BY`](order-by.html) clauses. However, in some cases the support is syntax-only—an error is returned if `NULLS FIRST` or `NULLS LAST` specification doesn't "match" the internal structure of the used index. If the index is ascending, the error is returned for `NULLS LAST`; if the index is descending, the error is returned for `NULLS FIRST`. - -## Order preservation - -In general, the order of the intermediate results of a query is not guaranteed, -even if `ORDER BY` is specified. In other words, the `ORDER BY` clause is only -effective at the top-level statement. For example, it is *ignored* by the query -planner when present in a sub-query in a `FROM` clause as follows: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM a, b ORDER BY a.x; -- valid, effective -> SELECT * FROM (SELECT * FROM a ORDER BY a.x), b; -- ignored, ineffective -~~~ - -However, when combining queries together with -[sub-queries](table-expressions.html#subqueries-as-table-expressions), -some combinations will make the `ORDER BY` clause in a sub-query -significant: - -1. The ordering of the operand of a `WITH ORDINALITY` clause - (within the `FROM` operand of a `SELECT` clause) is preserved, - to control the numbering of the rows. -2. The ordering of the operand of a stand-alone `LIMIT` or `OFFSET` clause (within - a `FROM` operand of a `SELECT` clause) is preserved, to determine - which rows are kept in the result. -3. The ordering of the data source for an [`INSERT`](insert.html) - statement or an [`UPSERT`](upsert.html) statement that also uses - `LIMIT` is preserved, to determine [which rows are processed, but not their order](#ordering-rows-in-dml-statements). -4. The ordering indicated for an [`UPDATE`](update.html) or [`DELETE`](delete.html) - statement that also uses `LIMIT` is used to determine - [which rows are processed, but not their order](#ordering-rows-in-dml-statements). - (This is a CockroachDB extension.) -5. The ordering of a sub-query used in a scalar expression - is preserved. - -For example, using `WITH ORDINALITY`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM (SELECT * FROM a ORDER BY a.x) WITH ORDINALITY; - -- ensures that the rows are numbered in the order of column a.x. -~~~ - -For example, using a stand-alone `LIMIT` clause in `FROM`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM a, ((SELECT * FROM b ORDER BY b.x) LIMIT 1); - -- ensures that only the first row of b in the order of column b.x - -- is used in the cross join. -~~~ - -For example, using a sub-query in scalar context: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT ARRAY(SELECT a.x FROM a ORDER BY a.x); - -- ensures that the array is constructed using the values of a.x in sorted order. -> SELECT (1, 2, 3) = (SELECT a.x FROM a ORDER BY a.x); - -- ensures that the values on the right-hand side are compared in the order of column a.x. -~~~ - -## Ordering of rows without `ORDER BY` - -Without `ORDER BY`, rows are processed or returned in a -non-deterministic order. "Non-deterministic" means that the actual order -can depend on the logical plan, the order of data on disk, the topology -of the CockroachDB cluster, and is generally variable over time. - -## Sorting using simple column selections - -Considering the following table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE a(a INT); -> INSERT INTO a VALUES (1), (3), (2); -~~~ - -The following statements are equivalent: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT a AS b FROM a ORDER BY b; -- first form: refers to an AS alias. -> SELECT a FROM a ORDER BY 1; -- second form: refers to a column position. -> SELECT a FROM a ORDER BY a; -- third form: refers to a column in the data source. -~~~ - -~~~ -+---------+ -| a | -+---------+ -| 1 | -| 2 | -| 3 | -+---------+ -(3 rows) -~~~ - -Note that the order of the rules matter. If there is ambiguity, the `AS` aliases -take priority over the data source columns, for example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE ab(a INT, b INT); -> SELECT a AS b, b AS c FROM ab ORDER BY b; -- orders by column a, renamed to b -> SELECT a, b FROM ab ORDER BY b; -- orders by column b -~~~ - -It is also possible to sort using an arbitrary scalar expression computed for each row, for example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT a, b FROM ab ORDER BY a + b; -- orders by the result of computing a+b. -~~~ - -## Sorting using multiple columns - -When more than one ordering specification is given, the later specifications are used -to order rows that are equal over the earlier specifications, for example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE ab(a INT, b INT); -> SELECT a, b FROM ab ORDER BY b, a; -~~~ - -This sorts the results by column `b`, and then if there are multiple -rows that have the same value in column `b`, it will then order these -rows by column `a`. - -## Inverting the sort order - -The keyword `DESC` ("descending") can be added after an ordering specification to -invert its order. This can be specified separately for each specification, for example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE ab(a INT, b INT); -> SELECT a, b FROM ab ORDER BY b DESC, a; -- sorts on b descending, then a ascending. -~~~ - -## Sorting in primary key order - -The `ORDER BY PRIMARY KEY` notation guarantees that the results are -presented in primary key order. - -The particular advantage is that for queries using the primary index, -this guarantees the order while also guaranteeing there will not be an -additional sorting computation to achieve it, for example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE kv(k INT PRIMARY KEY, v INT); -> SELECT k, v FROM kv ORDER BY PRIMARY KEY kv; -- guarantees ordering by column k. -~~~ - -If a primary key uses the keyword `DESC` already, then its meaning -will be flipped (cancelled) if the `ORDER BY` clause also uses -`DESC`, for example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE ab(a INT, b INT, PRIMARY KEY (b DESC, a ASC)); -> SELECT * FROM ab ORDER BY b DESC; -- orders by b descending, then a ascending. - -- The primary index may be used to optimize. - -> SELECT * FROM ab ORDER BY PRIMARY KEY ab DESC; -- orders by b ascending, then a descending. - -- The index order is inverted. -~~~ - -## Sorting in index order - -The `ORDER BY INDEX` notation guarantees that the results are presented -in the order of a given index. - -The particular advantage is that for queries using that index, this -guarantees the order while also guaranteeing there will not be an -additional sorting computation to achieve it, for example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE kv(k INT PRIMARY KEY, v INT, INDEX v_idx(v)); -> SELECT k, v FROM kv ORDER BY INDEX kv@v_idx; -- guarantees ordering by column v. -~~~ - -If an index uses the keyword `DESC` already, then its meaning -will be flipped (cancelled) if the `ORDER BY` clause also uses -`DESC`, for example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE ab(a INT, b INT, INDEX b_idx (b DESC, a ASC)); -> SELECT * FROM ab ORDER BY b DESC; -- orders by b descending, then a ascending. - -- The index b_idx may be used to optimize. - -> SELECT * FROM ab ORDER BY INDEX ab@b_idx DESC; -- orders by b ascending, then a descending. - -- The index order is inverted. -~~~ - -## Ordering rows in DML statements - -When using `ORDER BY` with an [`INSERT`](insert.html), -[`UPSERT`](upsert.html), [`UPDATE`](update.html) or -[`DELETE`](delete.html) (i.e., a DML statement), the `ORDER BY` clause is -ignored if it is not used in combination with [`LIMIT` and/or -`OFFSET`](limit-offset.html). - -The combination of both `ORDER BY` and `LIMIT`/`OFFSET` determines -which rows of the input are used to insert, update or delete the table -data, but *it does not determine in which order the mutation takes -place*. - -For example, using `LIMIT` in `INSERT`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO a SELECT * FROM b ORDER BY b.x LIMIT 1; - -- ensures that only the first row of b in the order of column b.x - -- is inserted into a. -~~~ - -The reason why `ORDER BY` does not control the final order of the rows -in the table is that the ordering of rows in the target table is -determined by its [primary and secondary indexes](indexes.html). - -To order the result of the `RETURNING` clause, see [Sorting the output -of deletes](#sorting-the-output-of-deletes). - -## Sorting the output of deletes - -{% include {{page.version.version}}/misc/sorting-delete-output.md %} - -## See also - -- [Selection Queries](selection-queries.html) -- [Scalar Expressions](scalar-expressions.html) -- [`INSERT`](insert.html) -- [`UPSERT`](upsert.html) -- [`DELETE`](delete.html) -- [`UPDATE`](delete.html) diff --git a/src/current/v21.1/owner-to.md b/src/current/v21.1/owner-to.md deleted file mode 100644 index 19c9b404fe6..00000000000 --- a/src/current/v21.1/owner-to.md +++ /dev/null @@ -1,105 +0,0 @@ ---- -title: OWNER TO -summary: The OWNER TO subcommand changes the owner of an object. -toc: true ---- - - `OWNER TO` is a subcommand of [`ALTER DATABASE`](alter-database.html), [`ALTER TABLE`](alter-table.html), [`ALTER SCHEMA`](alter-schema.html), and [`ALTER TYPE`](alter-type.html), and is used to change the owner of an object in a cluster. - -{{site.data.alerts.callout_info}} -This page documents `ALTER DATABASE ... OWNER TO` and `ALTER TABLE ... OWNER TO`. For details on the `ALTER SCHEMA ... OWNER TO` and `ALTER TYPE ... OWNER TO`, see the [`ALTER SCHEMA`](alter-schema.html) and [`ALTER TYPE`](alter-type.html) pages. -{{site.data.alerts.end}} - -## Required privileges - -- To change the owner of a database, the user must be an `admin` user, or the current owner of the database and a member of the new owner [role](authorization.html#roles). The user must also have the `CREATEDB` [privilege](authorization.html#assign-privileges). -- To change the owner of a table, the user must be an `admin` user, or the current owner of the table and a member of the new owner [role](authorization.html#roles). The new owner role must also have the `CREATE` [privilege](authorization.html#assign-privileges) on the schema to which the table belongs. - -## Syntax - -### Databases - -~~~ -ALTER DATABASE OWNER TO -~~~ - -### Tables - -~~~ -ALTER TABLE OWNER TO -~~~ - -## Parameters - -Parameter | Description -----------|------------ -`name` | The name of the table or database. -`newowner` | The name of the new owner. - -## Examples - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### Change a database's owner - -Suppose that the current owner of the `movr` database is `root` and you want to change the owner to a new user named `max`. - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER DATABASE movr OWNER TO max; -~~~ - -To verify that the owner is now `max`, query the `pg_catalog.pg_database` and `pg_catalog.pg_roles` tables: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT rolname FROM pg_catalog.pg_database d JOIN pg_catalog.pg_roles r ON d.datdba = r.oid WHERE datname = 'movr'; -~~~ - -~~~ - rolname ------------ - max -(1 row) -~~~ - -{{site.data.alerts.callout_info}} -If the user running the command is not an admin user, they must own the database and be a member of the new owning role. They must also have the `CREATEDB` [privilege](authorization.html#assign-privileges). -{{site.data.alerts.end}} - -### Change a table's owner - -Suppose that the current owner of the `rides` table is `root` and you want to change the owner to a new user named `max`. - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE promo_codes OWNER TO max; -~~~ - -To verify that the owner is now `max`, query the `pg_catalog.pg_tables` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT tableowner FROM pg_catalog.pg_tables WHERE tablename = 'promo_codes'; -~~~ - -~~~ - tableowner --------------- - max -(1 row) -~~~ - -{{site.data.alerts.callout_info}} -If the user running the command is not an admin user, they must own the table and be a member of the new owning role. Also, the new owner role must also have the `CREATE` [privilege](authorization.html#assign-privileges) on the schema to which the table belongs. -{{site.data.alerts.end}} - - - -## See also - -- [`CREATE DATABASE`](create-database.html) -- [`SHOW DATABASES`](show-databases.html) -- [`CREATE TABLE`](create-table.html) -- [`SHOW TABLES`](show-tables.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/pagination.md b/src/current/v21.1/pagination.md deleted file mode 100644 index 0402d2f01a3..00000000000 --- a/src/current/v21.1/pagination.md +++ /dev/null @@ -1,188 +0,0 @@ ---- -title: Paginate Results -summary: Paginate results from queries against your cluster -toc: true ---- - -To iterate through a table one "page" of results at a time (also known as pagination) there are two options, only one of which is recommended: - -- Keyset pagination (**fast, recommended**) -- `LIMIT` / `OFFSET` pagination (slow, not recommended) - -## Keyset pagination - -Keyset pagination (also known as the "seek method") is used to fetch a subset of records from a table quickly. It does this by restricting the set of records returned with a combination of `WHERE` and [`LIMIT`](limit-offset.html) clauses. To get the next page, you check the value of the column in the `WHERE` clause against the last row returned in the previous page of results. - -The general pattern for keyset pagination queries is: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT * FROM t AS OF SYSTEM TIME ${time} - WHERE key > ${value} - ORDER BY key - LIMIT ${amount} -~~~ - -This is faster than using `LIMIT`/`OFFSET` because, instead of doing a full table scan up to the value of the `OFFSET`, a keyset pagination query looks at a fixed-size set of records for each iteration. This can be done quickly provided that the key used in the `WHERE` clause to implement the pagination is [indexed](indexes.html#best-practices) and [unique](unique.html). A [primary key](primary-key.html) meets both of these criteria. - -{{site.data.alerts.callout_info}} -CockroachDB does not have cursors. To support a cursor-like use case, namely "operate on a snapshot of the database at the moment the cursor is opened", use the [`AS OF SYSTEM TIME`](as-of-system-time.html) clause as shown in the examples below. -{{site.data.alerts.end}} - -## Example - -The examples in this section use the [employees data set](https://github.com/datacharmer/test_db), which you can load into CockroachDB as follows: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE DATABASE IF NOT EXISTS employees; -USE employees; -IMPORT PGDUMP 'https://s3-us-west-1.amazonaws.com/cockroachdb-movr/datasets/employees-db/pg_dump/employees-full.sql.gz' WITH ignore_unsupported_statements; -~~~ - -To get the first page of results using keyset pagination, run the statement below. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT * FROM employees AS OF SYSTEM TIME '-1m' WHERE emp_no > 10000 ORDER BY emp_no LIMIT 25; -~~~ - -~~~ -emp_no | birth_date | first_name | last_name | gender | hire_date ----------+---------------------------+------------+-------------+--------+---------------------------- - 10001 | 1953-09-02 00:00:00+00:00 | Georgi | Facello | M | 1986-06-26 00:00:00+00:00 - 10002 | 1964-06-02 00:00:00+00:00 | Bezalel | Simmel | F | 1985-11-21 00:00:00+00:00 - 10003 | 1959-12-03 00:00:00+00:00 | Parto | Bamford | M | 1986-08-28 00:00:00+00:00 - 10004 | 1954-05-01 00:00:00+00:00 | Chirstian | Koblick | M | 1986-12-01 00:00:00+00:00 - ... -(25 rows) - -Time: 4ms total (execution 3ms / network 0ms) -~~~ - -{{site.data.alerts.callout_success}} -When writing your own queries of this type, use a known minimum value for the key's data type. If you do not know what the minimum value of the key is, you can use `SELECT min(key) FROM table`. -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -We use [`AS OF SYSTEM TIME`](as-of-system-time.html) in these examples to ensure that we are operating on a consistent snapshot of the database as of the specified timestamp. This reduces the chance that there will be any concurrent updates to the data the query is accessing, and thus no missing or duplicated rows during the pagination. It also reduces the risk of [transaction retries](transactions.html#client-side-intervention) due to concurrent data access. The value of `-1m` passed to `AS OF SYSTEM TIME` may need to be updated depending on your application's data access patterns. -{{site.data.alerts.end}} - -To get the second page of results, run: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT * FROM employees AS OF SYSTEM TIME '-1m' WHERE emp_no > 10025 ORDER BY emp_no LIMIT 25; -~~~ - -~~~ - emp_no | birth_date | first_name | last_name | gender | hire_date ----------+---------------------------+------------+------------+--------+---------------------------- - 10026 | 1953-04-03 00:00:00+00:00 | Yongqiao | Berztiss | M | 1995-03-20 00:00:00+00:00 - 10027 | 1962-07-10 00:00:00+00:00 | Divier | Reistad | F | 1989-07-07 00:00:00+00:00 - 10028 | 1963-11-26 00:00:00+00:00 | Domenick | Tempesti | M | 1991-10-22 00:00:00+00:00 - 10029 | 1956-12-13 00:00:00+00:00 | Otmar | Herbst | M | 1985-11-20 00:00:00+00:00 -... -(25 rows) - -Time: 2ms total (execution 1ms / network 1ms) -~~~ - -To get an arbitrary page of results showing employees whose IDs (`emp_no`) are in a much higher range, run the following query. Note that it takes about the same amount of time to run as the previous queries. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT * FROM employees AS OF SYSTEM TIME '-1m' WHERE emp_no > 300025 ORDER BY emp_no LIMIT 25; -~~~ - -~~~ - emp_no | birth_date | first_name | last_name | gender | hire_date ----------+---------------------------+------------+--------------+--------+---------------------------- - 400000 | 1963-11-29 00:00:00+00:00 | Mitsuyuki | Reinhart | M | 1985-08-27 00:00:00+00:00 - 400001 | 1962-06-02 00:00:00+00:00 | Rosalie | Chinin | M | 1986-11-28 00:00:00+00:00 - 400002 | 1964-08-16 00:00:00+00:00 | Quingbo | Birnbaum | F | 1986-04-23 00:00:00+00:00 - 400003 | 1958-04-30 00:00:00+00:00 | Jianwen | Sidhu | M | 1986-02-01 00:00:00+00:00 - 400004 | 1958-04-30 00:00:00+00:00 | Sedat | Suppi | M | 1995-12-18 00:00:00+00:00 -.... -(25 rows) - -Time: 2ms total (execution 1ms / network 0ms) -~~~ - -Compare the execution speed of the previous keyset pagination queries with the query below that uses `LIMIT` / `OFFSET` to get the same page of results: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT * FROM employees AS OF SYSTEM TIME '-1m' LIMIT 25 OFFSET 200024; -~~~ - -~~~ - emp_no | birth_date | first_name | last_name | gender | hire_date ----------+---------------------------+------------+--------------+--------+---------------------------- - 400000 | 1963-11-29 00:00:00+00:00 | Mitsuyuki | Reinhart | M | 1985-08-27 00:00:00+00:00 - 400001 | 1962-06-02 00:00:00+00:00 | Rosalie | Chinin | M | 1986-11-28 00:00:00+00:00 - 400002 | 1964-08-16 00:00:00+00:00 | Quingbo | Birnbaum | F | 1986-04-23 00:00:00+00:00 - 400003 | 1958-04-30 00:00:00+00:00 | Jianwen | Sidhu | M | 1986-02-01 00:00:00+00:00 -... -(25 rows) - -Time: 158ms total (execution 156ms / network 1ms) -~~~ - -The query using `LIMIT`/`OFFSET` for pagination is almost 100 times slower. To see why, let's use [`EXPLAIN`](explain.html). - -{% include_cached copy-clipboard.html %} -~~~ sql -EXPLAIN SELECT * FROM employees LIMIT 25 OFFSET 200024; -~~~ - -~~~ - info ----------------------------------------------------------------------------------------- - distribution: full - vectorized: true - - • limit - │ estimated row count: 25 - │ offset: 200024 - │ - └── • scan - estimated row count: 200,049 (67% of the table; stats collected 5 minutes ago) - table: employees@idx_17110_primary - spans: LIMITED SCAN - limit: 200049 -(12 rows) - -Time: 4ms total (execution 3ms / network 0ms) -~~~ - -The culprit is this: because we used `LIMIT`/`OFFSET`, we are performing a limited scan of the entire table (see `spans: LIMITED SCAN` above) from the first record all the way up to the value of the offset. In other words, we are iterating over a big array of rows from 1 to *n*, where *n* is 200049. The `estimated row count` row shows this. - -Meanwhile, the keyset pagination queries are looking at a much smaller range of table spans, which is much faster (see `spans: [/300026 - ]` and `limit: 25` below). Because [there is an index on every column in the `WHERE` clause](indexes.html#best-practices), these queries are doing an index lookup to jump to the start of the page of results, and then getting an additional 25 rows from there. This is much faster. - -{% include_cached copy-clipboard.html %} -~~~ sql -EXPLAIN SELECT * FROM employees WHERE emp_no > 300025 ORDER BY emp_no LIMIT 25; -~~~ - -~~~ - info ----------------------------------------------------------------------------------- - distribution: local - vectorized: true - - • scan - estimated row count: 25 (<0.01% of the table; stats collected 6 minutes ago) - table: employees@idx_17110_primary - spans: [/300026 - ] - limit: 25 -(8 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -As shown by the `estimated row count` row, this query scans only 25 rows, far fewer than the 200049 scanned by the `LIMIT`/`OFFSET` query. - -{{site.data.alerts.callout_danger}} -Using a sequential (i.e., non-[UUID](uuid.html)) primary key creates hot spots in the database for write-heavy workloads, since concurrent [`INSERT`](insert.html)s to the table will attempt to write to the same (or nearby) underlying [ranges](architecture/overview.html#architecture-range). This can be mitigated by designing your schema with [multi-column primary keys which include a monotonically increasing column](performance-best-practices-overview.html#use-multi-column-primary-keys). -{{site.data.alerts.end}} diff --git a/src/current/v21.1/partial-indexes.md b/src/current/v21.1/partial-indexes.md deleted file mode 100644 index f3c54cb9d90..00000000000 --- a/src/current/v21.1/partial-indexes.md +++ /dev/null @@ -1,434 +0,0 @@ ---- -title: Partial Indexes -summary: Partial indexes allow you to specify a subset of rows and columns to add to an index. -toc: true -keywords: gin, gin index, gin indexes, inverted index, inverted indexes, accelerated index, accelerated indexes ---- - - Partial indexes allow you to specify a subset of rows and columns to add to an [index](indexes.html). Partial indexes include the subset of rows in a table that evaluate to true on a boolean *predicate expression* (i.e., a `WHERE` filter) defined at [index creation](#creation). - -## How do partial indexes work? - -When you create a partial index, CockroachDB "indexes" the columns and rows that evaluate to true on the index's boolean predicate expression, creating a sorted copy of the subset of row values, without modifying the values in the table itself. - -CockroachDB can use a partial index to efficiently execute queries on any subset of rows implied by the partial index. When possible, the [cost-based optimizer](cost-based-optimizer.html) creates a plan that limits table scans on rows implied by the partial index to just the rows in the index. It also limits index rewrites to fewer rows. - -Partial indexes can improve cluster performance in a number of ways: - -- They contain fewer rows than full indexes, making them less expensive to create and store on a cluster. -- Read queries on rows included in a partial index only scan the rows in the partial index. This contrasts with queries on columns in full indexes, which must scan all rows in the indexed column. -- Write queries on tables with a partial index only perform an index write when the rows inserted satisfy the partial index predicate. This contrasts with write queries on tables with full indexes, which incur the overhead of a full index write when the rows inserted modify an indexed column. - -{{site.data.alerts.callout_info}} -When a query on a table with a partial index has a filter expression, the [cost-based optimizer](cost-based-optimizer.html) attempts to prove that the filter implies the partial index predicate. It is not guaranteed that the optimizer can prove the implication of arbitrarily complex expressions. Although unlikely, it is possible that a filter implies a predicate, but the optimizer cannot prove the implication. -{{site.data.alerts.end}} - -## Creation - -To create a partial index, use a [`CREATE INDEX`](create-index.html) statement, with a standard `WHERE` clause defining a predicate expression. - -For example, to define a partial index on columns `a` and `b` of table `t`, filtering on rows in column `c` greater than 5: - -~~~ sql -> CREATE INDEX ON t (a, b) WHERE c > 5; -~~~ - -The following queries use the partial index: - -~~~ sql -> SELECT a, b FROM t WHERE c > 5; -~~~ - -~~~ sql -> SELECT * FROM t WHERE c = 10; -~~~ - -The following queries do *not* use the partial index: - -~~~ sql -> SELECT a, b FROM t; -~~~ - -~~~ sql -> SELECT * FROM t WHERE c = 3; -~~~ - -When defining the predicate expression, note that: - -- The predicate expression must result in a [boolean](bool.html). -- The predicate expression can only refer to columns in the table being indexed. -- [Functions](functions-and-operators.html) used in predicates must be immutable. For example, the `now()` function is not allowed in predicates because its value depends on more than its arguments. - -## Unique partial indexes - -You can enforce [uniqueness](unique.html) on a subset of rows with `CREATE UNIQUE INDEX ... WHERE ...`. - -For example, to define a unique partial index on columns `a` and `b` for table `t`, filtering on rows in column `d` equal to `'x'`: - -~~~ sql -> CREATE UNIQUE INDEX ON t (a, b) WHERE d = 'x'; -~~~ - -This creates a partial index and a `UNIQUE` constraint on the subset of rows where `d` is equal to `'x'`. - -For another example, see [Create a partial index that enforces uniqueness on a subset of rows](#create-a-partial-index-that-enforces-uniqueness-on-a-subset-of-rows). - -{{site.data.alerts.callout_success}} -When [inserted values](insert.html) conflict with a `UNIQUE` constraint on one or more columns, CockroachDB normally returns an error. We recommend adding an [`ON CONFLICT`](insert.html#on-conflict-clause) clause to all `INSERT` statements that might conflict with rows in the unique index. -{{site.data.alerts.end}} - -## Partial GIN indexes - -{% include_cached new-in.html version="v21.1" %} You can create partial [GIN indexes](inverted-indexes.html#partial-gin-indexes), which are indexes on a subset of `JSON`, `ARRAY`, or geospatial container column data. - -## Index hints - -You can force queries [to use a specific partial index](table-expressions.html#force-index-selection) (also known as "index hinting"), like you can with full indexes. However, unlike full indexes, partial indexes cannot be used to satisfy all queries. If a query's filter implies the partial index predicate expression, the partial index will be used in the query plan. If not, an error will be returned. - -## Known limitations - -- CockroachDB does not currently support [`IMPORT`](import.html) statements on tables with partial indexes. See [tracking issue](https://github.com/cockroachdb/cockroach/issues/50225). -- CockroachDB does not currently support multiple arbiter indexes for `INSERT ON CONFLICT DO UPDATE`, and will return an error if there are multiple unique or exclusion constraints matching the `ON CONFLICT DO UPDATE` specification. See [tracking issue](https://github.com/cockroachdb/cockroach/issues/53170). - -## Examples - -### Setup - -The following examples use MovR, a fictional vehicle-sharing application, to demonstrate CockroachDB SQL statements. For more information about the MovR example application and dataset, see [MovR: A Global Vehicle-sharing App](movr.html). - -To follow along, run [`cockroach demo`](cockroach-demo.html) to start a temporary, in-memory cluster with the `movr` workload: - -{% include {{ page.version.version }}/demo_movr.md %} - -### Create an index on a subset of rows - -Suppose that you want to query the subset of `rides` with a `revenue` greater than 90. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW TABLES] WHERE table_name='rides'; -~~~ - -~~~ - schema_name | table_name | type | owner | estimated_row_count | locality ---------------+------------+-------+-------+---------------------+----------- - public | rides | table | demo | 125000 | NULL -(1 row) - -Time: 21ms total (execution 21ms / network 0ms) -~~~ - -Without a partial index, querying the `rides` table with a `WHERE revenue > 90` clause will scan the entire table. To see the plan for such a query, you can use an [`EXPLAIN` statement](explain.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN SELECT * FROM rides WHERE revenue > 90; -~~~ - -~~~ - info ------------------------------------------------------------------------------------------- - distribution: full - vectorized: true - - • filter - │ estimated row count: 12,472 - │ filter: revenue > 90 - │ - └── • scan - estimated row count: 125,000 (100% of the table; stats collected 12 seconds ago) - table: rides@primary - spans: FULL SCAN -(11 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -The `estimated row count` in the scan node lists the number of rows that the query plan will scan (in this case, the entire table row count of 125,000). The `table` property lists the index used in the scan (in this case, the [primary key index](primary-key.html)). - -To limit the number of rows scanned to just the rows that you are querying, you can create a partial index: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE INDEX ON rides (city, revenue) WHERE revenue > 90; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW INDEXES FROM rides; -~~~ - -~~~ - table_name | index_name | non_unique | seq_in_index | column_name | direction | storing | implicit --------------+-----------------------------------------------+------------+--------------+--------------+-----------+---------+----------- - rides | primary | false | 1 | city | ASC | false | false - rides | primary | false | 2 | id | ASC | false | false -... - rides | rides_city_revenue_idx | true | 1 | city | ASC | false | false - rides | rides_city_revenue_idx | true | 2 | revenue | ASC | false | false - rides | rides_city_revenue_idx | true | 3 | id | ASC | false | true -(12 rows) - -Time: 8ms total (execution 8ms / network 0ms) -~~~ - -Another `EXPLAIN` statement shows that the number of rows scanned by the original query decreases significantly with a partial index on the `rides` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN SELECT * FROM rides WHERE revenue > 90; -~~~ - -~~~ - info ------------------------------------------------------------------------------------------- - distribution: full - vectorized: true - - • index join - │ estimated row count: 12,472 - │ table: rides@primary - │ - └── • scan - estimated row count: 12,472 (10.0% of the table; stats collected 36 seconds ago) - table: rides@rides_city_revenue_idx (partial index) - spans: FULL SCAN -(11 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -Note that the query's `SELECT` statement queries all columns in the `rides` table, not just the indexed columns. As a result, an "index join" is required on both the primary index and the partial index. - -Querying only the columns in the index will make the query more efficient by removing the index join from the query plan: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN SELECT city, revenue FROM rides WHERE revenue > 90; -~~~ - -~~~ - info ------------------------------------------------------------------------------------- - distribution: full - vectorized: true - - • scan - estimated row count: 11,463 (9.2% of the table; stats collected 4 minutes ago) - table: rides@rides_city_revenue_idx (partial index) - spans: FULL SCAN -(7 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -Querying a subset of the rows implied by the partial index predicate expression (in this case, `revenue > 90`) will also use the partial index: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN SELECT city, revenue FROM rides WHERE revenue > 95; -~~~ - -~~~ - info ----------------------------------------------------------------------------------------- - distribution: full - vectorized: true - - • filter - │ estimated row count: 5,037 - │ filter: revenue > 95 - │ - └── • scan - estimated row count: 11,463 (9.2% of the table; stats collected 5 minutes ago) - table: rides@rides_city_revenue_idx (partial index) - spans: FULL SCAN -(11 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -The number of rows scanned is the same, and an additional filter is applied to the query plan so that only the subset specified by the filter is returned. - -So far, all the query scans in this example have spanned the entire partial index (i.e., performed a `FULL SCAN` of the index). This is because the `WHERE` clause does not filter on the first column in the index prefix (`city`). Filtering the query on both columns in the partial index will limit the scan to just the rows that match the filter: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN SELECT city, revenue FROM rides WHERE city = 'new york' AND revenue > 90; -~~~ - -~~~ - info ------------------------------------------------------------------------------------ - distribution: local - vectorized: true - - • scan - estimated row count: 1,301 (1.0% of the table; stats collected 6 minutes ago) - table: rides@rides_city_revenue_idx (partial index) - spans: [/'new york' - /'new york'] -(7 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -Refining the `revenue` filter expression to match just a subset of the partial index will lower the scanned row count even more: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN SELECT city, revenue FROM rides WHERE city = 'new york' AND revenue >= 90 AND revenue < 95; -~~~ - -~~~ - info ---------------------------------------------------------------------------------------- - distribution: local - vectorized: true - - • filter - │ estimated row count: 746 - │ filter: (revenue >= 90) AND (revenue < 95) - │ - └── • scan - estimated row count: 14,187 (11% of the table; stats collected 6 minutes ago) - table: rides@primary - spans: [/'new york' - /'new york'] -(11 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -### Create an index that excludes values - -Suppose that you have a number of rows in a table with values that you regularly filter out of selection queries (e.g., `NULL` values). - -A selection query on these values will require a full table scan, using the primary index, as shown by the [`EXPLAIN` statement](explain.html) below: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN SELECT * FROM rides WHERE end_time IS NOT NULL; -~~~ - -~~~ - info ------------------------------------------------------------------------------------------ - distribution: full - vectorized: true - - • filter - │ estimated row count: 125,000 - │ filter: end_time IS NOT NULL - │ - └── • scan - estimated row count: 125,000 (100% of the table; stats collected 7 minutes ago) - table: rides@primary - spans: FULL SCAN -(11 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -You can create a partial index that excludes these rows, making queries that filter out the non-`NULL` values more efficient. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE INDEX ON rides (city, revenue) WHERE end_time IS NOT NULL; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW INDEXES FROM rides; -~~~ - -~~~ - table_name | index_name | non_unique | seq_in_index | column_name | direction | storing | implicit --------------+-----------------------------------------------+------------+--------------+--------------+-----------+---------+----------- - rides | primary | false | 1 | city | ASC | false | false - rides | primary | false | 2 | id | ASC | false | false -... - rides | rides_city_revenue_idx | true | 1 | city | ASC | false | false - rides | rides_city_revenue_idx | true | 2 | revenue | ASC | false | false - rides | rides_city_revenue_idx | true | 3 | id | ASC | false | true -(12 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN SELECT (city, revenue) FROM rides WHERE end_time IS NOT NULL; -~~~ - -~~~ - info ------------------------------------------------------------------------------------------ - distribution: full - vectorized: true - - • render - │ estimated row count: 125,000 - │ - └── • scan - estimated row count: 125,000 (100% of the table; stats collected 8 minutes ago) - table: rides@rides_city_revenue_idx1 (partial index) - spans: FULL SCAN -(10 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - - -### Create a partial index that enforces uniqueness on a subset of rows - -Suppose that you want to constrain a subset of the rows in a table, such that all values for a particular column in the subset are unique. For example, let's say that every user in New York City must have a unique name. - -You can do this efficiently with a [unique partial index](#unique-partial-indexes): - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE UNIQUE INDEX ON users (name) WHERE city='new york'; -~~~ - -This creates a partial index and a [`UNIQUE` constraint](unique.html) on just the subset of rows where `city='new york'`. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT id, name FROM users WHERE city='new york' LIMIT 3; -~~~ - -~~~ - id | name ----------------------------------------+------------------- - 8647a7cf-4af0-4c82-9344-224097f87b1a | Andre Sanchez - 598eaab2-5200-40cb-8e19-244d49f3f63a | Austin Meyer - 147ae147-ae14-4b00-8000-000000000004 | Catherine Nelson -(3 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO users(id, city, name) VALUES (gen_random_uuid(), 'new york', 'Andre Sanchez'); -~~~ - -~~~ -ERROR: duplicate key value (name)=('Andre Sanchez') violates unique constraint "users_name_key" -SQLSTATE: 23505 -~~~ - -Because the unique partial index predicate only implies the rows where `city='new york'`, the `UNIQUE` constraint does not apply to all rows in the table. - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO users(id, city, name) VALUES (gen_random_uuid(), 'seattle', 'Andre Sanchez'); -~~~ - -~~~ -INSERT 1 -~~~ - -## See also - -- [Indexes](indexes.html) -- [`CREATE INDEX`](create-index.html) -- [`DROP INDEX`](drop-index.html) -- [`RENAME INDEX`](rename-index.html) -- [`SHOW INDEX`](show-index.html) -- [SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/partition-by.md b/src/current/v21.1/partition-by.md deleted file mode 100644 index 4c525cbb6fb..00000000000 --- a/src/current/v21.1/partition-by.md +++ /dev/null @@ -1,171 +0,0 @@ ---- -title: PARTITION BY -summary: Use the PARTITION BY statement to partition, re-partition, or un-partition a table or secondary index. -toc: true ---- - -`PARTITION BY` is a subcommand of [`ALTER TABLE`](alter-table.html) and [`ALTER INDEX`](alter-index.html) that is used to partition, re-partition, or un-partition a table or secondary index. After defining partitions, [`CONFIGURE ZONE`](configure-zone.html) is used to control the replication and placement of partitions. - -{% include {{page.version.version}}/sql/use-multiregion-instead-of-partitioning.md %} - -{{site.data.alerts.callout_info}} -[Partitioning](partitioning.html) is an [Enterprise-only](enterprise-licensing.html) feature. If you are looking for the `PARTITION BY` used in SQL window functions, see [Window Functions](window-functions.html). -{{site.data.alerts.end}} - -{% include {{ page.version.version }}/sql/combine-alter-table-commands.md %} - -## Primary key requirements - -The [primary key required for partitioning](partitioning.html#partition-using-primary-key) is different from the conventional primary key: The unique identifier in the primary key must be prefixed with all columns you want to partition and subpartition the table on, in the order in which you want to nest your subpartitions. - -If the primary key in your existing table does not meet the requirements, you can change the primary key with an [`ALTER TABLE ... PRIMARY KEY` statement](alter-primary-key.html). - -## Synopsis - -**alter_table_partition_by_stmt ::=** - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/alter_table_partition_by.html %} -
    - -**alter_index_partition_by_stmt ::=** - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/alter_index_partition_by.html %} -
    - -## Parameters - -Parameter | Description | ------------|-------------| -`table_name` | The name of the table you want to define partitions for. -`index_name` | The name of the index you want to define partitions for. -`name_list` | List of columns you want to define partitions on (in the order they are defined in the primary key). -`list_partitions` | Name of list partition followed by the list of values to be included in the partition. -`range_partitions` | Name of range partition followed by the range of values to be included in the partition. - -## Required privileges - -The user must have the `CREATE` [privilege](authorization.html#assign-privileges) on the table. - -## Viewing schema changes - -{% include {{ page.version.version }}/misc/schema-change-view-job.md %} - - -{% include {{page.version.version}}/sql/querying-partitions.md %} - -## Examples - -### Define a list partition on a table or secondary index - -Suppose we have a table called `students_by_list`, and secondary index on the table called `name_idx`, in a global online learning portal, and the primary key of the table is defined as `(country, id)`. We can define partitions on the table and index by list: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE students_by_list PARTITION BY LIST (country) ( - PARTITION north_america VALUES IN ('CA','US'), - PARTITION australia VALUES IN ('AU','NZ'), - PARTITION DEFAULT VALUES IN (default) - ); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER INDEX students_by_list@name_idx PARTITION BY LIST (country) ( - PARTITION north_america VALUES IN ('CA','US'), - PARTITION australia VALUES IN ('AU','NZ'), - PARTITION DEFAULT VALUES IN (default) - ); -~~~ - -### Define a range partition on a table or secondary index - -Suppose we have another table called `students_by_range`, also with a secondary index called `name_idx`, and the primary key of the table is defined as `(expected_graduation_date, id)`. We can define partitions on the table and index by range: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE students_by_range PARTITION BY RANGE (expected_graduation_date) ( - PARTITION graduated VALUES FROM (MINVALUE) TO ('2017-08-15'), - PARTITION current VALUES FROM ('2017-08-15') TO (MAXVALUE) - ); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER INDEX students_by_range@name_idx PARTITION BY RANGE (expected_graduation_date) ( - PARTITION graduated VALUES FROM (MINVALUE) TO ('2017-08-15'), - PARTITION current VALUES FROM ('2017-08-15') TO (MAXVALUE) - ); -~~~ - -### Define subpartitions on a table or secondary index - -Suppose we have an yet another table named `students`, again with a secondary index called `name_idx`, and the primary key is defined as `(country, expected_graduation_date, id)`. We can define partitions and subpartitions on the table and index: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE students PARTITION BY LIST (country) ( - PARTITION australia VALUES IN ('AU','NZ') PARTITION BY RANGE (expected_graduation_date) ( - PARTITION graduated_au VALUES FROM (MINVALUE) TO ('2017-08-15'), - PARTITION current_au VALUES FROM ('2017-08-15') TO (MAXVALUE) - ), - PARTITION north_america VALUES IN ('US','CA') PARTITION BY RANGE (expected_graduation_date) ( - PARTITION graduated_us VALUES FROM (MINVALUE) TO ('2017-08-15'), - PARTITION current_us VALUES FROM ('2017-08-15') TO (MAXVALUE) - ) - ); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER INDEX students@name_idx PARTITION BY LIST (country) ( - PARTITION australia VALUES IN ('AU','NZ') PARTITION BY RANGE (expected_graduation_date) ( - PARTITION graduated_au VALUES FROM (MINVALUE) TO ('2017-08-15'), - PARTITION current_au VALUES FROM ('2017-08-15') TO (MAXVALUE) - ), - PARTITION north_america VALUES IN ('US','CA') PARTITION BY RANGE (expected_graduation_date) ( - PARTITION graduated_us VALUES FROM (MINVALUE) TO ('2017-08-15'), - PARTITION current_us VALUES FROM ('2017-08-15') TO (MAXVALUE) - ) - ); -~~~ - -### Repartition a table or secondary index - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE students_by_range PARTITION BY RANGE (expected_graduation_date) ( - PARTITION graduated VALUES FROM (MINVALUE) TO ('2018-08-15'), - PARTITION current VALUES FROM ('2018-08-15') TO (MAXVALUE) - ); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER INDEX students_by_range@name_idx PARTITION BY RANGE (expected_graduation_date) ( - PARTITION graduated VALUES FROM (MINVALUE) TO ('2018-08-15'), - PARTITION current VALUES FROM ('2018-08-15') TO (MAXVALUE) - ); -~~~ - -### Unpartition a table or secondary index - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE students PARTITION BY NOTHING; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER INDEX students@name_idx PARTITION BY NOTHING; -~~~ - -## See also - -- [`CREATE TABLE`](create-table.html) -- [`ALTER TABLE`](alter-table.html) -- [`ALTER INDEX`](alter-index.html) -- [Define Table Partitions](partitioning.html) -- [`SHOW JOBS`](show-jobs.html) -- [`SHOW PARTITIONS`](show-partitions.html) diff --git a/src/current/v21.1/partitioning.md b/src/current/v21.1/partitioning.md deleted file mode 100644 index 7de71de734a..00000000000 --- a/src/current/v21.1/partitioning.md +++ /dev/null @@ -1,723 +0,0 @@ ---- -title: Define Table Partitions -summary: Partitioning is an Enterprise feature that gives you row-level control of how and where your data is stored. -toc: true ---- - -CockroachDB allows you to define table partitions, thus giving you row-level control of how and where your data is stored. Partitioning enables you to reduce latencies and costs and can assist in meeting regulatory requirements for your data. - -{% include enterprise-feature.md %} - -## Why use table partitioning - -{% include {{page.version.version}}/sql/use-multiregion-instead-of-partitioning.md %} - -Table partitioning helps you reduce latency and cost: - -- **Geo-partitioning** allows you to keep user data close to the user, which reduces the distance that the data needs to travel, thereby **reducing latency**. To geo-partition a table, define location-based partitions while creating a table, create location-specific zone configurations, and apply the zone configurations to the corresponding partitions. -- **Archival-partitioning** allows you to store infrequently-accessed data on slower and cheaper storage, thereby **reducing costs**. To archival-partition a table, define frequency-based partitions while creating a table, create frequency-specific zone configurations with appropriate storage devices constraints, and apply the zone configurations to the corresponding partitions. - -## How it works - -Table partitioning involves a combination of CockroachDB features: - -- [Node attributes](#node-attributes) -- [Enterprise license](#enterprise-license) -- [Table creation](#table-creation) -- [Replication zones](#replication-zones) - -### Node attributes - -To store partitions in specific locations (e.g., geo-partitioning), or on machines with specific attributes (e.g., archival-partitioning), the nodes of your cluster must be [started](cockroach-start.html) with the relevant flags: - -- Use the `--locality` flag to assign key-value pairs that describe the location of a node, for example, `--locality=region=east,az=us-east-1`. -- Use the `--attrs` flag to specify node capability, which might include specialized hardware or number of cores, for example, `--attrs=ram:64gb`. -- Use the `attrs` field of the `--store` flag to specify disk type or capability, for example,`--store=path=/mnt/ssd01,attrs=ssd`. - -For more details about these flags, see the [`cockroach start`](cockroach-start.html) documentation. - -### Enterprise license - -You must have a valid Enterprise license to use table partitioning features. For details about requesting and setting a trial or full Enterprise license, see [Enterprise Licensing](enterprise-licensing.html). - -Note that the following features do not work with an **expired license**: - -- Creating new table partitions or adding new zone configurations for partitions -- Changing the partitioning scheme on any table or index -- Changing the zone config for a partition - -However, the following features continue to work even with an expired Enterprise license: - -- Querying a partitioned table (for example, `SELECT foo PARTITION`) -- Inserting or updating data in a partitioned table -- Dropping a partitioned table -- Unpartitioning a partitioned table -- Making non-partitioning changes to a partitioned table (for example, adding a column/index/foreign key/check constraint) - -### Table creation - -You can define partitions and subpartitions over one or more columns of a table. During [table creation](create-table.html), you declare which values belong to each partition in one of two ways: - -- **List partitioning**: Enumerate all possible values for each partition. List partitioning is a good choice when the number of possible values is small. List partitioning is well-suited for geo-partitioning. -- **Range partitioning**: Specify a contiguous range of values for each partition by specifying lower and upper bounds. Range partitioning is a good choice when the number of possible values is too large to explicitly list out. Range partitioning is well-suited for archival-partitioning. - -#### Partition by list - -[`PARTITION BY LIST`](partition-by.html) lets you map one or more tuples to a partition. - -To partition a table by list, use the [`PARTITION BY LIST`](partition-by.html) syntax while creating the table. While defining a list partition, you can also set the `DEFAULT` partition that acts as a catch-all if none of the rows match the requirements for the defined partitions. - -See [Partition by List](#define-table-partitions-by-list) example below for more details. - -#### Partition by range - -[`PARTITION BY RANGE`](partition-by.html) lets you map ranges of tuples to a partition. - -To define a table partition by range, use the [`PARTITION BY RANGE`](partition-by.html) syntax while creating the table. While defining a range partition, you can use CockroachDB-defined `MINVALUE` and `MAXVALUE` parameters to define the lower and upper bounds of the ranges respectively. - -{{site.data.alerts.callout_info}}The lower bound of a range partition is inclusive, while the upper bound is exclusive. For range partitions, NULL is considered less than any other data, which is consistent with our key encoding ordering and ORDER BY behavior.{{site.data.alerts.end}} - -Partition values can be any SQL expression, but it’s only evaluated once. If you create a partition with value `< (now() - '1d')` on 2017-01-30, it would be contain all values less than 2017-01-29. It would not update the next day, it would continue to contain values less than 2017-01-29. - -See [Partition by Range](#define-table-partitions-by-range) example below for more details. - -#### Partition using primary key - -The primary key required for partitioning is different from the conventional primary key. To define the primary key for partitioning, prefix the unique identifier(s) in the primary key with all columns you want to partition and subpartition the table on, in the order in which you want to nest your subpartitions. - -For instance, consider the database of a global online learning portal that has a table for students of all the courses across the world. If you want to geo-partition the table based on the countries of the students, then the primary key needs to be defined as: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE students ( - id INT DEFAULT unique_rowid(), - name STRING, - email STRING, - country STRING, - expected_graduation_date DATE, - PRIMARY KEY (country, id)); -~~~ - -##### Primary key considerations - -- The order in which the columns are defined in the primary key is important. The partitions and subpartitions need to follow that order. In the example of the online learning portal, if you define the primary key as `(country, expected_graduation_date, id)`, the primary partition is by `country`, and then subpartition is by `expected_graduation_date`. You cannot skip `country` and partition by `expected_graduation_date`. - -#### Partition using secondary index - -The primary key discussed above has two drawbacks: - -- It does not enforce that the identifier column is globally unique. -- It does not provide fast lookups on the identifier. - -To ensure uniqueness or fast lookups, create a [secondary index](indexes.html) on the identifier. - -Indexes are not required to be partitioned, but creating a non-partitioned index on a partitioned table may not perform well. - - When you create a non-partitioned index on a partitioned table, CockroachDB sends a [`NOTICE` message](https://www.postgresql.org/docs/current/plpgsql-errors-and-messages.html) to the client stating that creating a non-partitioned index on a partitioned table may not perform well. - -### Replication zones - -On their own, partitions are inert and simply apply a label to the rows of the table that satisfy the criteria of the defined partitions. Applying functionality to a partition requires creating and applying [replication zone](configure-replication-zones.html) to the corresponding partitions. - -CockroachDB uses the most granular zone config available. Zone configs that target a partition are considered more granular than those that target a table or index, which in turn are considered more granular than those that target a database. - - -{% include {{page.version.version}}/sql/querying-partitions.md %} - -## Examples - -### Define table partitions by list - -Consider a global online learning portal, RoachLearn, that has a database containing a table of students across the world. Suppose we have three availability zone: one in the United States, one in Germany, and another in Australia. To reduce latency, we want to keep the students' data closer to their locations: - -- We want to keep the data of the students located in the United States and Canada in the United States availability zone. -- We want to keep the data of students located in Germany and Switzerland in the German availability zone. -- We want to keep the data of students located in Australia and New Zealand in the Australian availability zone. - -#### Step 1. Identify the partitioning method - -We want to geo-partition the table to keep the students' data closer to their locations. We can achieve this by partitioning on `country` and using the `PARTITION BY LIST` syntax. - -#### Step 2. Start each node with its availability zone location specified in the `--locality` flag - -1. Start 3 nodes in the US availability zone: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --insecure \ - --listen-addr=localhost \ - --locality=region=us \ - --listen-addr=localhost:26257 \ - --http-addr=localhost:8080 \ - --store=node1 \ - --background \ - --join=localhost:26257,localhost:26258,localhost:26259 - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --insecure \ - --listen-addr=localhost \ - --locality=region=us \ - --listen-addr=localhost:26258 \ - --http-addr=localhost:8081 \ - --store=node2 \ - --background \ - --join=localhost:26257,localhost:26258,localhost:26259 - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --insecure \ - --listen-addr=localhost \ - --locality=region=us \ - --listen-addr=localhost:26259 \ - --http-addr=localhost:8082 \ - --store=node3 \ - --background \ - --join=localhost:26257,localhost:26258,localhost:26259 - ~~~ - -2. Initialize the cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach init \ - --insecure \ - --host=localhost:26257 - ~~~ - -3. Add 3 nodes in the German availability zone: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --insecure \ - --listen-addr=localhost \ - --locality=region=de \ - --listen-addr=localhost:26260 \ - --http-addr=localhost:8083 \ - --store=node4 \ - --background \ - --join=localhost:26257,localhost:26258,localhost:26259 - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --insecure \ - --listen-addr=localhost \ - --locality=region=de \ - --listen-addr=localhost:26261 \ - --http-addr=localhost:8084 \ - --store=node5 \ - --background \ - --join=localhost:26257,localhost:26258,localhost:26259 - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --insecure \ - --listen-addr=localhost \ - --locality=region=de \ - --listen-addr=localhost:26262 \ - --http-addr=localhost:8085 \ - --store=node6 \ - --background \ - --join=localhost:26257,localhost:26258,localhost:26259 - ~~~ - -3. Add 3 nodes in the Australian availability zone: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --insecure \ - --listen-addr=localhost \ - --locality=region=aus \ - --listen-addr=localhost:26263 \ - --http-addr=localhost:8086 \ - --store=node7 \ - --background \ - --join=localhost:26257,localhost:26258,localhost:26259 - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --insecure \ - --listen-addr=localhost \ - --locality=region=aus \ - --listen-addr=localhost:26264 \ - --http-addr=localhost:8087 \ - --store=node8 \ - --background \ - --join=localhost:26257,localhost:26258,localhost:26259 - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --insecure \ - --listen-addr=localhost \ - --locality=region=aus \ - --listen-addr=localhost:26265 \ - --http-addr=localhost:8088 \ - --store=node9 \ - --background \ - --join=localhost:26257,localhost:26258,localhost:26259 - ~~~ - - -#### Step 3. Request and set a trial Enterprise license - -See [Set the Trial or Enterprise License Key](licensing-faqs.html#set-a-license). - -#### Step 4. Create the `roachlearn` database and `students` table - -1. Open the CockroachDB SQL shell: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --insecure --host=localhost:26257 - ~~~ - -2. Create the database and set it as current: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE DATABASE roachlearn; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET DATABASE = roachlearn; - ~~~ - -3. Create the table with the appropriate partitions: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE TABLE students ( - id INT DEFAULT unique_rowid(), - name STRING, - email STRING, - country STRING, - expected_graduation_date DATE, - PRIMARY KEY (country, id)) - PARTITION BY LIST (country) ( - PARTITION north_america VALUES IN ('CA','US'), - PARTITION europe VALUES IN ('DE', 'CH'), - PARTITION australia VALUES IN ('AU','NZ'), - PARTITION DEFAULT VALUES IN (default) - ); - ~~~ - - Alternatively, you can create and partition the table as separate steps: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE TABLE students ( - id INT DEFAULT unique_rowid(), - name STRING, - email STRING, - country STRING, - expected_graduation_date DATE, - PRIMARY KEY (country, id)); - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER TABLE students - PARTITION BY LIST (country) ( - PARTITION north_america VALUES IN ('CA','US'), - PARTITION europe VALUES IN ('DE', 'CH'), - PARTITION australia VALUES IN ('AU','NZ'), - PARTITION DEFAULT VALUES IN (default) - ); - ~~~ - -#### Step 5. Create and apply corresponding replication zones - -To create replication zone and apply them to corresponding partitions, use the [`ALTER PARTITION ... CONFIGURE ZONE`](configure-zone.html) statement: - -1. Create a replication zone for the `north_america` partition and constrain its data to the US availability zone: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER PARTITION north_america OF TABLE students - CONFIGURE ZONE USING constraints='[+region=us]'; - ~~~ - -2. Create a replication zone for the `europe` partition and constrain its data to the German availability zone: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER PARTITION europe OF TABLE students - CONFIGURE ZONE USING constraints='[+region=de]'; - ~~~ - -3. Create a replication zone for the `australia` partition and constrain its data to the Australian availability zone: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > ALTER PARTITION australia OF TABLE students - CONFIGURE ZONE USING constraints='[+region=aus]'; - ~~~ - -4. After creating these replication zones, you can view them using the [`SHOW ZONE CONFIGURATION`](show-zone-configurations.html) statement: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW ZONE CONFIGURATION FROM PARTITION north_america OF TABLE students; - ~~~ - - ~~~ - target | raw_config_sql -+-------------------------------------------+----------------------------------------------------------------------+ - PARTITION north_america OF TABLE students | ALTER PARTITION north_america OF TABLE students CONFIGURE ZONE USING - | range_min_bytes = 134217728, - | range_max_bytes = 536870912, - | gc.ttlseconds = 90000, - | num_replicas = 3, - | constraints = '[+region=us]', - | lease_preferences = '[]' -(1 row) - ~~~ - - -#### Step 6. Check replica distribution - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW RANGES FROM TABLE roachlearn.students] WHERE "start_key" IS NOT NULL AND "start_key" NOT LIKE '%Prefix%'; -~~~ - -You should see the following output: - -~~~ - start_key | end_key | range_id | replicas | lease_holder -+-----------+-----------------+----------+----------+--------------+ - /"AU" | /"AU"/PrefixEnd | 35 | {7,8,9} | 8 - /"CA" | /"CA"/PrefixEnd | 31 | {1,2,3} | 1 - /"CH" | /"CH"/PrefixEnd | 51 | {4,5,6} | 5 - /"DE" | /"DE"/PrefixEnd | 53 | {4,5,6} | 5 - /"NZ" | /"NZ"/PrefixEnd | 55 | {7,8,9} | 8 - /"US" | /"US"/PrefixEnd | 33 | {1,2,3} | 1 -(6 rows) -~~~ - -For reference, here's how the nodes map to zones: - -Node IDs | Zone ----------|----- -1-3 | `north_america` -4-6 | `europe` -7-9 | `australia` - -We can see that, after partitioning, the replicas for `US` and `CA`-based students are located on nodes 1-3 in `north_america`, the replicas for `DE` and `CH`-based students are located on nodes 4-6 in `europe`, and the replicas for `AU` and `NZ`-based students are located on nodes 7-9 in `australia`. - -### Show partitions and zone constraints - -To retrieve table partitions, you can use the [`SHOW PARTITIONS`](show-partitions.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW PARTITIONS FROM TABLE students; -~~~ - -~~~ - database_name | table_name | partition_name | parent_partition | column_names | index_name | partition_value | zone_constraints -+---------------+------------+----------------+------------------+--------------+------------------+-----------------+------------------+ - roachlearn | students | north_america | NULL | country | students@primary | ('CA'), ('US') | [+region=us] - roachlearn | students | europe | NULL | country | students@primary | ('DE'), ('CH') | [+region=de] - roachlearn | students | australia | NULL | country | students@primary | ('AU'), ('NZ') | [+region=aus] - roachlearn | students | default | NULL | country | students@primary | (DEFAULT) | NULL -(4 rows) -~~~ - -You can also view partitions by [database](show-partitions.html#show-partitions-by-database) and [index](show-partitions.html#show-partitions-by-index). - -{% include {{page.version.version}}/sql/crdb-internal-partitions.md %} - -### Define table partitions by range - -Suppose we want to store the data of current students on fast and expensive storage devices (e.g., SSD) and store the data of the graduated students on slower, cheaper storage devices (e.g., HDD). - -#### Step 1. Identify the partitioning method - -We want to archival-partition the table to keep newer data on faster devices and older data on slower devices. We can achieve this by partitioning the table by date and using the `PARTITION BY RANGE` syntax. - -#### Step 2. Set the Enterprise license - -To set the Enterprise license, see [Set the Trial or Enterprise License Key](licensing-faqs.html#set-a-license). - -#### Step 3. Start each node with the appropriate storage device specified in the `--store` flag - -1. Start the first node: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start --insecure \ - --store=path=/mnt/1,attrs=ssd \ - --advertise-addr= \ - --join=, - ~~~ - -2. Start the second node: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start --insecure \ - --store=path=/mnt/2,attrs=hdd \ - --advertise-addr= \ - --join=, - ~~~ - -3. Initialize the cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach init \ - --insecure \ - --host=
    - ~~~ - -#### Step 4. Create a table with the appropriate partitions - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE students_by_range ( - id INT DEFAULT unique_rowid(), - name STRING, - email STRING, - country STRING, - expected_graduation_date DATE, - PRIMARY KEY (expected_graduation_date, id)) - PARTITION BY RANGE (expected_graduation_date) - (PARTITION graduated VALUES FROM (MINVALUE) TO ('2017-08-15'), - PARTITION current VALUES FROM ('2017-08-15') TO (MAXVALUE)); -~~~ - -#### Step 5. Create and apply corresponding zone configurations - -To create zone configurations and apply them to corresponding partitions, use the [`ALTER PARTITION ... CONFIGURE ZONE`](configure-zone.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER PARTITION current OF TABLE students_by_range - CONFIGURE ZONE USING constraints='[+ssd]'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER PARTITION graduated OF TABLE students_by_range - CONFIGURE ZONE USING constraints='[+hdd]'; -~~~ - -#### Step 6. Check replica distribution - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW RANGES FROM TABLE students_by_range; -~~~ - -You should see the following output: - -~~~ - start_key | end_key | range_id | replicas | lease_holder | locality -+-----------+---------+----------+----------+--------------+-----------+ - NULL | /17393 | 52 | {2,5,8} | 5 | region=de - /17393 | NULL | 53 | {6,7,10} | 6 | region=de -(2 rows) -~~~ - -### Define subpartitions on a table - -A list partition can itself be partitioned, forming a subpartition. There is no limit on the number of levels of subpartitioning; that is, list partitions can be infinitely nested. - -{{site.data.alerts.callout_info}}Range partitions cannot be subpartitioned.{{site.data.alerts.end}} - -Going back to RoachLearn's scenario, suppose we want to do all of the following: - -- Keep the students' data close to their location. -- Store the current students' data on faster storage devices. -- Store the graduated students' data on slower, cheaper storage devices (example: HDD). - -#### Step 1. Identify the Partitioning method - -We want to geo-partition as well as archival-partition the table. We can achieve this by partitioning the table first by location and then by date. - -#### Step 2. Start each node with the appropriate storage device specified in the `--store` flag - -Start a node in the US availability zone: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start \ ---insecure \ ---advertise-addr= \ ---locality=az=us1 \ ---store=path=/mnt/1,attrs=ssd \ ---store=path=/mnt/2,attrs=hdd \ ---join=, -~~~ - -Start a node in the AUS availability zone: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start \ ---insecure \ ---advertise-addr= \ ---locality=az=aus1 \ ---store=path=/mnt/3,attrs=ssd \ ---store=path=/mnt/4,attrs=hdd \ ---join=, -~~~ - -Initialize the cluster: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach init --insecure --host=
    -~~~ - -#### Step 3. Set the Enterprise license - -To set the Enterprise license, see [Set the Trial or Enterprise License Key](licensing-faqs.html#set-a-license). - -#### Step 4. Create a table with the appropriate partitions - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE students ( - id INT DEFAULT unique_rowid(), - name STRING, - email STRING, - country STRING, - expected_graduation_date DATE, - PRIMARY KEY (country, expected_graduation_date, id)) - PARTITION BY LIST (country)( - PARTITION australia VALUES IN ('AU','NZ') PARTITION BY RANGE (expected_graduation_date)(PARTITION graduated_au VALUES FROM (MINVALUE) TO ('2017-08-15'), PARTITION current_au VALUES FROM ('2017-08-15') TO (MAXVALUE)), - PARTITION north_america VALUES IN ('US','CA') PARTITION BY RANGE (expected_graduation_date)(PARTITION graduated_us VALUES FROM (MINVALUE) TO ('2017-08-15'), PARTITION current_us VALUES FROM ('2017-08-15') TO (MAXVALUE)) - ); -~~~ - -Subpartition names must be unique within a table. In our example, even though `graduated` and `current` are sub-partitions of distinct partitions, they still need to be uniquely named. Hence the names `graduated_au`, `graduated_us`, and `current_au` and `current_us`. - -#### Step 5. Create and apply corresponding zone configurations - -To create zone configurations and apply them to corresponding partitions, use the [`ALTER PARTITION ... CONFIGURE ZONE`](configure-zone.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER PARTITION current_us OF TABLE students - CONFIGURE ZONE USING constraints='[+ssd,+az=us1]'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER PARTITION graduated_us OF TABLE students CONFIGURE ZONE - USING constraints='[+hdd,+az=us1]'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER PARTITION current_au OF TABLE students - CONFIGURE ZONE USING constraints='[+ssd,+az=aus1]'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER PARTITION graduated_au OF TABLE students CONFIGURE ZONE - USING constraints='[+hdd,+az=aus1]'; -~~~ - -#### Step 6. Verify table partitions - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW RANGES FROM TABLE students; -~~~ - -You should see the following output: - -~~~ -+-----------------+-----------------+----------+----------+--------------+ -| start_key | end_key | range_id | replicas | lease_holder | -+-----------------+-----------------+----------+----------+--------------+ -| NULL | /"AU" | 260 | {1,2,3} | 1 | -| /"AU" | /"AU"/17393 | 268 | {1,2,3} | 1 | -| /"AU"/17393 | /"AU"/PrefixEnd | 266 | {1,2,3} | 1 | -| /"AU"/PrefixEnd | /"CA" | 267 | {1,2,3} | 1 | -| /"CA" | /"CA"/17393 | 265 | {1,2,3} | 1 | -| /"CA"/17393 | /"CA"/PrefixEnd | 261 | {1,2,3} | 1 | -| /"CA"/PrefixEnd | /"NZ" | 262 | {1,2,3} | 3 | -| /"NZ" | /"NZ"/17393 | 284 | {1,2,3} | 3 | -| /"NZ"/17393 | /"NZ"/PrefixEnd | 282 | {1,2,3} | 3 | -| /"NZ"/PrefixEnd | /"US" | 283 | {1,2,3} | 3 | -| /"US" | /"US"/17393 | 281 | {1,2,3} | 3 | -| /"US"/17393 | /"US"/PrefixEnd | 263 | {1,2,3} | 1 | -| /"US"/PrefixEnd | NULL | 264 | {1,2,3} | 1 | -+-----------------+-----------------+----------+----------+--------------+ -(13 rows) - -Time: 11.586626ms -~~~ - -### Repartition a table - -Consider the partitioned table of students of RoachLearn. Suppose the table has been partitioned on range to store the current students on fast and expensive storage devices (example: SSD) and store the data of the graduated students on slower, cheaper storage devices(example: HDD). Now suppose we want to change the date after which the students will be considered current to `2018-08-15`. We can achieve this by using the [`PARTITION BY`](partition-by.html) subcommand of the [`ALTER TABLE`](alter-table.html) command. - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE students_by_range PARTITION BY RANGE (expected_graduation_date) ( - PARTITION graduated VALUES FROM (MINVALUE) TO ('2018-08-15'), - PARTITION current VALUES FROM ('2018-08-15') TO (MAXVALUE)); -~~~ - -### Unpartition a table - -You can remove the partitions on a table by using the [`PARTITION BY NOTHING`](partition-by.html) syntax: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE students PARTITION BY NOTHING; -~~~ - -### Show the replication zone for a partition - -To view the replication zone for a partition, use the [`SHOW ZONE CONFIGURATION`](show-zone-configurations.html) statement. - -## Locality–resilience tradeoff - -There is a tradeoff between making reads/writes fast and surviving failures. Consider a partition with three replicas of `roachlearn.students` for Australian students. - -- If only one replica is pinned to an Australian availability zone, then reads may be fast (via leases follow the workload) but writes will be slow. -- If two replicas are pinned to an Australian availability zone, then reads and writes will be fast (as long as the cross-ocean link has enough bandwidth that the third replica doesn’t fall behind). If those two replicas are in the same availability zone, then the loss of one availability zone can lead to data unavailability, so some deployments may want two separate Australian availability zones. -- If all three replicas are in Australian availability zones, then three Australian availability zones are needed to be resilient to an availability zone loss. - -## How CockroachDB's partitioning differs from other databases - -Other databases use partitioning for three additional use cases: secondary indexes, sharding, and bulk loading/deleting. CockroachDB addresses these use-cases not by using partitioning, but in the following ways: - -- **Changes to secondary indexes:** CockroachDB solves these changes through online schema changes. Online schema changes are a superior feature to partitioning because they require zero-downtime and eliminate the potential for consistency problems. -- **Sharding:** CockroachDB automatically shards data as a part of its distributed database architecture. -- **Bulk Loading & Deleting:** CockroachDB does not have a feature that supports this use case as of now. -- **Logical structure of partitions:** CockroachDB uses the partitioning concept to allow users to logically subdivide a single physical table to enable independent configuration of those partitions using zone configurations. Other databases sometimes implement partitioning as the logical union of physically separate tables. This difference means that CockroachDB is able to permit inexpensive repartitioning in contrast to other databases. Consequently, CockroachDB is unable to provide the same bulk data deletion operations over table partitions that other databases achieve by physically dropping the underlying table represented by the partition. - -## Known limitations - -- {% include {{ page.version.version }}/known-limitations/partitioning-with-placeholders.md %} -- {% include {{ page.version.version }}/known-limitations/drop-single-partition.md %} - -## See also - -- [`CREATE TABLE`](create-table.html) -- [`ALTER TABLE`](alter-table.html) -- [Computed Columns](computed-columns.html) diff --git a/src/current/v21.1/pause-job.md b/src/current/v21.1/pause-job.md deleted file mode 100644 index 61d8777b6ef..00000000000 --- a/src/current/v21.1/pause-job.md +++ /dev/null @@ -1,127 +0,0 @@ ---- -title: PAUSE JOB -summary: The PAUSE JOB statement lets you temporarily halt the process of potentially long-running jobs. -toc: true ---- - -The `PAUSE JOB` [statement](sql-statements.html) lets you pause the following types of jobs: - -- [`IMPORT`](import.html) jobs -- [`BACKUP`](backup.html) and [`RESTORE`](restore.html) jobs -- [User-created table statistics](create-statistics.html) jobs -- [Automatic table statistics](cost-based-optimizer.html#table-statistics) jobs -- [Changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html) -- [Schema change](online-schema-changes.html) jobs -- [Scheduled backup](manage-a-backup-schedule.html) jobs - -After pausing jobs, you can resume them with [`RESUME JOB`](resume-job.html). - -## Required privileges - -To pause a job, the user must be a member of the `admin` role or must have the [`CONTROLJOB`](create-user.html#create-a-user-that-can-pause-resume-and-cancel-non-admin-jobs) parameter set. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/pause_job.html %} -
    - -## Parameters - -Parameter | Description -----------|------------ -`job_id` | The ID of the job you want to pause, which can be found with [`SHOW JOBS`](show-jobs.html). -`select_stmt` | A [selection query](selection-queries.html) that returns `job_id`(s) to pause. -`for_schedules_clause` | The schedule you want to pause jobs for. You can pause jobs for a specific schedule (`FOR SCHEDULE id`) or pause jobs for multiple schedules by nesting a [`SELECT` clause](select-clause.html) in the statement (`FOR SCHEDULES `). See the [examples](#pause-jobs-for-a-schedule) below. - -## Examples - -### Pause a single job - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW JOBS; -~~~ - -~~~ - job_id | job_type | description |... ------------------+-----------+-------------------------------------------+... - 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> PAUSE JOB 27536791415282; -~~~ - -### Pause multiple jobs - -To pause multiple jobs, nest a [`SELECT` clause](select-clause.html) that retrieves `job_id`(s) inside the `PAUSE JOBS` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> PAUSE JOBS (SELECT job_id FROM [SHOW JOBS] - WHERE user_name = 'maxroach'); -~~~ - -All jobs created by `maxroach` will be paused. - -### Pause automatic table statistics jobs - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW AUTOMATIC JOBS; -~~~ - -~~~ - job_id | job_type | description |... ----------------------+---------------------+-----------------------------------------------------+... - 438235476849557505 | AUTO CREATE STATS | Table statistics refresh for defaultdb.public.users |... -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> PAUSE JOB 438235476849557505; -~~~ - -To permanently disable automatic table statistics jobs, disable the `sql.stats.automatic_collection.enabled` [cluster setting](cluster-settings.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET CLUSTER SETTING sql.stats.automatic_collection.enabled = false; -~~~ - -### Pause jobs for a schedule - - To pause jobs for a specific [backup schedule](create-schedule-for-backup.html), use the schedule's `id`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> PAUSE JOBS FOR SCHEDULE 590204387299262465; -~~~ - -~~~ -PAUSE JOBS FOR SCHEDULES 1 -~~~ - -You can also pause multiple schedules by nesting a [`SELECT` clause](select-clause.html) that retrieves `id`(s) inside the `PAUSE JOBS` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> PAUSE JOBS FOR SCHEDULES SELECT id FROM [SHOW SCHEDULES] WHERE label = 'test_schedule'; -~~~ - -~~~ -PAUSE JOBS FOR SCHEDULES 2 -~~~ - -## See also - -- [`RESUME JOB`](resume-job.html) -- [`SHOW JOBS`](show-jobs.html) -- [`CANCEL JOB`](cancel-job.html) -- [`BACKUP`](backup.html) -- [`RESTORE`](restore.html) -- [`IMPORT`](import.html) -- [`CREATE CHANGEFEED`](create-changefeed.html) diff --git a/src/current/v21.1/pause-schedules.md b/src/current/v21.1/pause-schedules.md deleted file mode 100644 index e61ac0b2412..00000000000 --- a/src/current/v21.1/pause-schedules.md +++ /dev/null @@ -1,72 +0,0 @@ ---- -title: PAUSE SCHEDULES -summary: The PAUSE SCHEDULES statement lets you temporarily halt the process of a backup schedule. -toc: true ---- - - The `PAUSE SCHEDULES` [statement](sql-statements.html) can be used to pause [backup schedules](create-schedule-for-backup.html). - -After pausing a schedule, you can resume it with [`RESUME SCHEDULES`](resume-schedules.html). - -## Required privileges - -Only members of the [`admin` role](authorization.html#default-roles) can pause a schedule. By default, the `root` user belongs to the `admin` role. - -## Synopsis - -~~~ -PAUSE SCHEDULES - select clause: select statement returning schedule id to pause. -PAUSE SCHEDULE -~~~ - -## Parameters - - Parameter | Description ----------------+------------ -`selectclause` | A [selection query](selection-queries.html) that returns `id`(s) to pause. -`scheduleID` | The `id` of the schedule you want to pause, which can be found with [`SHOW SCHEDULES`](show-schedules.html). - -## Examples - -### Pause a single schedule - -{% include_cached copy-clipboard.html %} -~~~ sql -> PAUSE SCHEDULE 589963390487363585; -~~~ - -~~~ -PAUSE SCHEDULES 1 -~~~ - -### Pause multiple schedules - -To pause multiple schedules, nest a [`SELECT` clause](select-clause.html) that retrieves `id`(s) inside the `PAUSE SCHEDULES` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> PAUSE SCHEDULES SELECT id FROM [SHOW SCHEDULES] WHERE label = 'schedule_database'; -~~~ - -~~~ -PAUSE SCHEDULES 4 -~~~ - -In this example, all schedules with the label `schedule_database` are paused. - -## See also - -- [Manage a Backup Schedule](manage-a-backup-schedule.html) -- [`BACKUP`](backup.html) -- [`RESTORE`](restore.html) -- [`SHOW BACKUP`](show-backup.html) -- [`SHOW SCHEDULES`](show-schedules.html) -- [`RESUME SCHEDULES`](resume-schedules.html) -- [`DROP SCHEDULES`](drop-schedules.html) -- [`PAUSE JOB`](pause-job.html) -- [`RESUME JOB`](pause-job.html) -- [`CANCEL JOB`](cancel-job.html) -- [Take Full and Incremental Backups](take-full-and-incremental-backups.html) -- [Use the Built-in SQL Client](cockroach-sql.html) -- [Other Cockroach Commands](cockroach-commands.html) diff --git a/src/current/v21.1/performance-benchmarking-with-tpcc-large.md b/src/current/v21.1/performance-benchmarking-with-tpcc-large.md deleted file mode 100644 index c81ce40da61..00000000000 --- a/src/current/v21.1/performance-benchmarking-with-tpcc-large.md +++ /dev/null @@ -1,491 +0,0 @@ ---- -title: Performance Benchmarking with TPC-C -summary: Learn how to benchmark CockroachDB against TPC-C with 81 nodes on `c5d.9xlarge` machines -toc: true -toc_not_nested: true -key: performance-benchmarking-with-tpc-c-100k-warehouses.html ---- - -This page shows you how to reproduce [CockroachDB's TPC-C performance benchmarking results](performance.html#scale) on commodity AWS hardware. Across all scales, CockroachDB can process tpmC (new order transactions per minute) at near maximum efficiency. Start by choosing the scale you're interested in: - -
    - - - - -
    - -| Workload | Cluster size | Warehouses | Data size | -|----------+------------------------------------+------------+-----------| -| Local | 3 nodes on your laptop | 10 | 2 GB | -| Small | 3 nodes on `c5d.4xlarge` machines | 2500 | 200 GB | -| Medium | 15 nodes on `c5d.4xlarge` machines | 13,000 | 1.04 TB | -| Large | 81 nodes on `c5d.9xlarge` machines | 140,000 | 11.2 TB | - -## Before you begin - -- [Review TPC-C concepts](#review-tpc-c-concepts) -- [Request a trial license](#request-a-trial-license) - -### Review TPC-C concepts - -TPC-C provides the most realistic and objective measure for OLTP performance at various scale factors. Before you get started, consider reviewing [what TPC-C is and how it is measured](performance.html#tpc-c). - -### Request a trial license - -Reproducing these TPC-C results involves using CockroachDB's [partitioning](partitioning.html) feature to ensure replicas for any given section of data are located on the same nodes that will be queried by the load generator for that section of data. Partitioning helps distribute the workload evenly across the cluster. - -The partitioning feature requires an Enterprise license, so [request a 30-day trial license](https://www.cockroachlabs.com/get-cockroachdb/enterprise/) before you get started. - -You should receive your trial license via email within a few minutes. You'll enable your license once your cluster is up-and-running. - -## Step 1. Set up the environment - -- [Provision VMs](#provision-vms) -- [Configure your network](#configure-your-network) - -### Provision VMs - -1. [Create 86 VM instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/LaunchingAndUsingInstances.html), 81 for CockroachDB nodes and 5 for the TPC-C workload. - - Create all instances in the same region and the same security group. - - Use the `c5d.9xlarge` machine type. - - Use [local SSD instance store volumes](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html#instance-store-volumes). Local SSDs are low latency disks attached to each VM, which maximizes performance. This configuration best resembles what a bare metal deployment would look like, with machines directly connected to one physical disk each. We do not recommend using network-attached block storage. - -2. Note the internal IP address of each instance. You'll need these addresses when starting the CockroachDB nodes. - -{{site.data.alerts.callout_danger}} -This configuration is intended for performance benchmarking only. For production deployments, there are other important considerations, such as security, load balancing, and data location techniques to minimize network latency. For more details, see the [Production Checklist](recommended-production-settings.html). -{{site.data.alerts.end}} - -### Configure your network - -CockroachDB requires TCP communication on two ports: - -- `26257` for inter-node communication (i.e., working as a cluster) and for the TPC-C workload to connect to nodes -- `8080` for exposing your DB Console - -[Create inbound rules](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html#adding-security-group-rule) for your security group: - -#### Inter-node and TPCC-to-node communication - - Field | Recommended Value --------|------------------- - Type | Custom TCP Rule - Protocol | TCP - Port Range | **26257** - Source | The name of your security group (e.g., *sg-07ab277a*) - -#### DB Console - - Field | Recommended Value --------|------------------- - Type | Custom TCP Rule - Protocol | TCP - Port Range | **8080** - Source | Your network's IP ranges - -## Step 2. Start CockroachDB - -{% include {{ page.version.version }}/prod-deployment/insecure-flag.md %} - -1. SSH to the first VM where you want to run a CockroachDB node. - -2. Download the [CockroachDB archive](https://binaries.cockroachdb.com/cockroach-{{ page.release_info.version }}.linux-amd64.tgz) for Linux, extract the binary, and copy it into the `PATH`: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ curl https://binaries.cockroachdb.com/cockroach-{{ page.release_info.version }}.linux-amd64.tgz \ - | tar -xz - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cp -i cockroach-{{ page.release_info.version }}.linux-amd64/cockroach /usr/local/bin/ - ~~~ - - If you get a permissions error, prefix the command with `sudo`. - -3. Run the [`cockroach start`](cockroach-start.html) command: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --insecure \ - --advertise-addr= \ - --join=,, \ - --cache=.25 \ - --locality=rack=0 \ - --background - ~~~ - - Each node will start with a [locality](cockroach-start.html#locality) that includes an artificial "rack number" (e.g., `--locality=rack=0`). Use 81 racks for 81 nodes so that 1 node will be assigned to each rack. - -4. Repeat steps 1 - 3 for the other 80 VMs for CockroachDB nodes. Each time, be sure to: - - Adjust the `--advertise-addr` flag. - - Set the [`--locality`](cockroach-start.html#locality) flag to the appropriate "rack number", as described above. - -5. On any of the VMs with the `cockroach` binary, run the one-time [`cockroach init`](cockroach-init.html) command to join the first nodes into a cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach init --insecure --host=
    - ~~~ - -## Step 3. Configure the cluster - -You'll be importing a large TPC-C data set. To speed that up, you can temporarily disable replication and tweak some cluster settings. You'll also need to enable the Enterprise license you requested earlier. - -1. SSH to any VM with the `cockroach` binary. - -2. Launch the [built-in SQL shell](cockroach-sql.html): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --insecure --host=
    - ~~~ - -3. Adjust some [cluster settings](cluster-settings.html): - - {% include_cached copy-clipboard.html %} - ~~~ sql - SET CLUSTER SETTING kv.dist_sender.concurrency_limit = 2016; - SET CLUSTER SETTING kv.snapshot_rebalance.max_rate = '256 MiB'; - SET CLUSTER SETTING kv.snapshot_recovery.max_rate = '256 MiB'; - SET CLUSTER SETTING sql.stats.automatic_collection.enabled = false; - SET CLUSTER SETTING schemachanger.backfiller.max_buffer_size = '5 GiB'; - SET CLUSTER SETTING rocksdb.min_wal_sync_interval = '500us'; - SET CLUSTER SETTING kv.range_merge.queue_enabled = false - ~~~ - -4. Change the default [GC TTL](configure-replication-zones.html#gc-ttlseconds) to the following value: - - {% include_cached copy-clipboard.html %} - ~~~ sql - ALTER RANGE default CONFIGURE ZONE USING gc.ttlseconds = 600; - ~~~ - -5. Enable the trial license you requested earlier: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING cluster.organization = ''; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING enterprise.license = ''; - ~~~ - -6. Exit the SQL shell: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > \q - ~~~ - -## Step 4. Import the TPC-C dataset - -CockroachDB comes with a number of [built-in workloads](cockroach-workload.html) for simulating client traffic. This step features CockroachDB's version of the [TPC-C](http://www.tpc.org/tpcc/) workload. - -1. SSH to the VM where you want to run TPC-C. - -1. Download the [CockroachDB archive](https://binaries.cockroachdb.com/cockroach-{{ page.release_info.version }}.linux-amd64.tgz) for Linux, extract the binary, and copy it into the `PATH`: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ curl https://binaries.cockroachdb.com/cockroach-{{ page.release_info.version }}.linux-amd64.tgz \ - | tar -xz - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cp -i cockroach-{{ page.release_info.version }}.linux-amd64/cockroach /usr/local/bin/ - ~~~ - - If you get a permissions error, prefix the command with `sudo`. - -1. Import the TPC-C dataset: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload fixtures import tpcc \ - --partitions=81 \ - --warehouses=140000 \ - --replicate-static-columns \ - --partition-strategy=leases \ - 'postgres://root@
    :26257?sslmode=disable' - ~~~ - - This will load 11.2 TB of data for 140,000 "warehouses". This can take up to 8 hours to complete. - - You can monitor progress on the **Jobs** screen of the DB Console. Open the [DB Console](ui-overview.html) by pointing a browser to the address in the `admin` field in the standard output of any node on startup. - -## Step 5. Partition the database - -Next, [partition your database](partitioning.html) to divide all of the TPC-C tables and indexes into 81 partitions, one per rack, and then use [zone configurations](configure-replication-zones.html) to pin those partitions to a particular rack. - -Wait for up-replication and partitioning to finish. You will know when they have finished because both the number of *lease transfers* and *snapshots* will go down to `0` and stay there. Note that this will likely take 10s of minutes. - -- To monitor the number of Lease transfers, open the [DB Console](ui-overview.html), select the **Replication** dashboard, hover over the **Range Operations** graph, and check the **Lease Transfers** data point. - -- To check the number of snapshots, open the [DB Console](ui-overview.html), select the **Replication** dashboard, and hover over the **Snapshots** graph. - -TPC-C 140k replication and partitioning dashboards - -## Step 7. Allocate partitions - -Before running the benchmark, it's important to allocate partitions to workload binaries properly to ensure that the cluster is balanced. - -1. Create an `addrs` file containing connection strings to all 81 CockroachDB nodes: - - ~~~ - postgres://root@:26257?sslmode=disable postgres://root@:26257?sslmode=disable postgres://root@:26257?sslmode=disable postgres://root@:26257?sslmode=disable ... - ~~~ - -2. Upload the `addrs` file to the 5 VMs with the `workload` binary: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ scp addrs @:. - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ scp addrs @:. - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ scp addrs @:. - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ scp addrs @:. - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ scp addrs @:. - ~~~ - -3. SSH to each VM with `workload` and allocate partitions: - - {% include_cached copy-clipboard.html %} - ~~~ shell - ulimit -n 500000 && cockroach workload run tpcc --partitions=81 \ - --warehouses=140000 \ - --partition-affinity=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16 \ - --ramp=30m \ - --duration=1ms \ - --histograms=workload1.histogram.ndjson \ - $(cat addrs) - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - ulimit -n 500000 && cockroach workload run tpcc \ - --partitions 81 \ - --warehouses 140000 \ - --partition-affinity=17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32 \ - --ramp=30m \ - --duration=1ms \ - --histograms=workload2.histogram.ndjson \ - $(cat addrs) - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - ulimit -n 500000 && cockroach workload run tpcc \ - --partitions=81 \ - --warehouses=140000 \ - --partition-affinity=33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48 \ - --ramp=30m \ - --duration=1ms \ - --histograms=workload3.histogram.ndjson \ - $(cat addrs) - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - ulimit -n 500000 && cockroach workload run tpcc \ - --partitions=81 \ - --warehouses=140000 \ - --partition-affinity=49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64 \ - --ramp=30m \ - --duration=1ms \ - --histograms=workload4.histogram.ndjson \ - $(cat addrs) - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - ulimit -n 500000 && cockroach workload run tpcc \ - --partitions=81 \ - --warehouses=140000 \ - --partition-affinity=65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80 \ - --ramp=30m \ - --duration=1ms \ - --histograms=workload5.histogram.ndjson \ - $(cat addrs) - ~~~ - -## Step 8. Run the benchmark - -Once the allocations finish, run TPC-C for 30 minutes on each VM with `workload`: - -{{site.data.alerts.callout_info}} -It is critical to run the benchmark from the workload nodes in parallel, so start them as simultaneously as possible. -{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ shell -ulimit -n 500000 && cockroach workload run tpcc \ ---partitions=81 \ ---warehouses=140000 \ ---partition-affinity=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16 \ ---ramp=4m \ ---duration=30m \ ---histograms=workload1.histogram.ndjson \ -$(cat addrs) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -ulimit -n 500000 && cockroach workload run tpcc \ ---partitions=81 \ ---warehouses=140000 \ ---partition-affinity=17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32 \ ---ramp=4m \ ---duration=30m \ ---histograms=workload2.histogram.ndjson \ -$(cat addrs) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -ulimit -n 500000 && cockroach workload run tpcc \ ---partitions=81 \ ---warehouses=140000 \ ---partition-affinity=33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48 \ ---ramp=4m \ ---duration=30m \ ---histograms=workload3.histogram.ndjson \ -$(cat addrs) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -ulimit -n 500000 && cockroach workload run tpcc \ ---partitions=81 \ ---warehouses=140000 \ ---partition-affinity=49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64 \ ---ramp=4m \ ---duration=30m \ ---histograms=workload4.histogram.ndjson \ -$(cat addrs) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -ulimit -n 500000 && cockroach workload run tpcc \ ---partition=81 \ ---warehouses=140000 \ ---partition-affinity=65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80 \ ---ramp=4m \ ---duration=30m \ ---histograms=workload5.histogram.ndjson \ -$(cat addrs) -~~~ - -## Step 9. Interpret the results - -1. Collect the result files from each VM with `workload`: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ scp @:workload1.histogram.ndjson . - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ scp @:workload2.histogram.ndjson . - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ scp @:workload3.histogram.ndjson . - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ scp @:workload4.histogram.ndjson . - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ scp @:workload5.histogram.ndjson . - ~~~ - -2. Upload the result files to one of the VMs with the `workload` binary: - - {{site.data.alerts.callout_info}} - The commands below assume you're uploading to the VM with the `workload1.histogram.ndjson` file. - {{site.data.alerts.end}} - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ scp workload2.histogram.ndjson @:. - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ scp workload3.histogram.ndjson @:. - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ scp workload4.histogram.ndjson @:. - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ scp workload5.histogram.ndjson @:. - ~~~ - -3. SSH to the VM where you uploaded the results files. - -4. Run the `workload debug tpcc-merge-results` command to synthesize the results: - - {% include_cached copy-clipboard.html %} - ~~~ shell - cockroach workload debug tpcc-merge-results \ - --warehouses=140000 \ - workload*.histogram.ndjson - ~~~ - - You'll should see results similar to the following, with **1.68M tpmC with 140,000 warehouses, resulting in an efficiency score of 95%**: - - ~~~ - Duration: 30m1., Warehouses: 140000, Efficiency: 95.45, tpmC: 1684437.21 - _elapsed___ops/sec(cum)__p50(ms)__p90(ms)__p95(ms)__p99(ms)_pMax(ms) - 1801.1s 2824.0 302.0 1140.9 2415.9 9126.8 55834.6 delivery - 1801.1s 28074.0 402.7 1409.3 2684.4 9126.8 45097.2 newOrder - 1801.1s 2826.0 6.8 62.9 125.8 4160.7 33286.0 orderStatus - 1801.1s 28237.4 251.7 1006.6 2415.9 15032.4 103079.2 payment - 1801.1s 2823.5 39.8 469.8 906.0 5905.6 38654.7 stockLevel - ~~~ - -## See also - -- [Performance Overview](performance.html) - -- Hardware - - CockroachDB works well on commodity hardware in public cloud, private cloud, on-prem, and hybrid environments. For hardware recommendations, see our [Production Checklist](recommended-production-settings.html#hardware). - - Also note that CockroachDB creates a yearly cloud report focused on evaluating hardware performance. For more information, see the [2020 Cloud Report](https://www.cockroachlabs.com/blog/2020-cloud-report/). - -- Performance Tuning - - For guidance on tuning a real workload's performance, see [SQL Best Practices](performance-best-practices-overview.html), and for guidance on techniques to minimize network latency in multi-region or global clusters, see [Multi-Region Overview](multiregion-overview.html). diff --git a/src/current/v21.1/performance-benchmarking-with-tpcc-local.md b/src/current/v21.1/performance-benchmarking-with-tpcc-local.md deleted file mode 100644 index 0a574857576..00000000000 --- a/src/current/v21.1/performance-benchmarking-with-tpcc-local.md +++ /dev/null @@ -1,171 +0,0 @@ ---- -title: Performance Benchmarking with TPC-C -summary: Learn how to benchmark CockroachDB against TPC-C 13k on 3 nodes on your laptop -toc: true -toc_not_nested: true -key: performance-benchmarking-with-tpc-c-10-warehouses.html ---- - -This page shows you how to reproduce [CockroachDB's TPC-C performance benchmarking results](performance.html#scale) on commodity AWS hardware. Across all scales, CockroachDB can process tpmC (new order transactions per minute) at near maximum efficiency. Start by choosing the scale you're interested in: - -
    - - - - -
    - -| Workload | Cluster size | Warehouses | Data size | -|----------+------------------------------------+------------+-----------| -| Local | 3 nodes on your laptop | 10 | 2 GB | -| Small | 3 nodes on `c5d.4xlarge` machines | 2500 | 200 GB | -| Medium | 15 nodes on `c5d.4xlarge` machines | 13,000 | 1.04 TB | -| Large | 81 nodes on `c5d.9xlarge` machines | 140,000 | 11.2 TB | - -## Before you begin - -- TPC-C provides the most realistic and objective measure for OLTP performance at various scale factors. Before you get started, consider reviewing [what TPC-C is and how it is measured](performance.html#tpc-c). - -- Make sure you have already [installed CockroachDB](install-cockroachdb.html). - -## Step 1. Start CockroachDB - -{% include {{ page.version.version }}/prod-deployment/insecure-flag.md %} - -1. Use the [`cockroach start`](cockroach-start.html) command to start 3 nodes: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --insecure \ - --store=tpcc-local1 \ - --listen-addr=localhost:26257 \ - --http-addr=localhost:8080 \ - --join=localhost:26257,localhost:26258,localhost:26259 \ - --background - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --insecure \ - --store=tpcc-local2 \ - --listen-addr=localhost:26258 \ - --http-addr=localhost:8081 \ - --join=localhost:26257,localhost:26258,localhost:26259 \ - --background - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --insecure \ - --store=tpcc-local3 \ - --listen-addr=localhost:26259 \ - --http-addr=localhost:8082 \ - --join=localhost:26257,localhost:26258,localhost:26259 \ - --background - ~~~ - -2. Use the [`cockroach init`](cockroach-init.html) command to perform a one-time initialization of the cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach init \ - --insecure \ - --host=localhost:26257 - ~~~ - -## Step 2. Import the TPC-C dataset - -CockroachDB comes with a number of [built-in workloads](cockroach-workload.html) for simulating client traffic. This step features CockroachDB's version of the [TPC-C](http://www.tpc.org/tpcc/) workload. - -Use [`cockroach workload`](cockroach-workload.html) to load the initial schema and data: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach workload fixtures import tpcc \ ---warehouses=10 \ -'postgresql://root@localhost:26257?sslmode=disable' -~~~ - -This will load 2 GB of data for 10 "warehouses". - -## Step 3. Run the benchmark - -Run the workload for ten "warehouses" of data for ten minutes: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach workload run tpcc \ ---warehouses=10 \ ---ramp=3m \ ---duration=10m \ -'postgresql://root@localhost:26257?sslmode=disable' -~~~ - -You'll see per-operation statistics every second: - -~~~ -Initializing 20 connections... -Initializing 100 workers and preparing statements... -_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms) - 1.0s 0 0.0 0.0 0.0 0.0 0.0 0.0 delivery - 1.0s 0 0.0 0.0 0.0 0.0 0.0 0.0 newOrder -... - 105.0s 0 0.0 0.2 0.0 0.0 0.0 0.0 delivery - 105.0s 0 4.0 1.8 44.0 46.1 46.1 46.1 newOrder - 105.0s 0 0.0 0.2 0.0 0.0 0.0 0.0 orderStatus - 105.0s 0 1.0 2.0 14.7 14.7 14.7 14.7 payment - 105.0s 0 0.0 0.2 0.0 0.0 0.0 0.0 stockLevel -... -~~~ - -{{site.data.alerts.callout_success}} -For more `tpcc` options, use `cockroach workload run tpcc --help`. For details about other built-in load generators, use `cockroach workload run --help`. -{{site.data.alerts.end}} - -## Step 4. Interpret the results - -Once the `workload` has finished running, you'll see a final output line: - -~~~ -_elapsed_______tpmC____efc__avg(ms)__p50(ms)__p90(ms)__p95(ms)__p99(ms)_pMax(ms) - 300.0s 121.6 94.6% 41.0 39.8 54.5 71.3 96.5 130.0 -~~~ - -You will also see some audit checks and latency statistics for each individual query. For this run, some of those checks might indicate that they were `SKIPPED` due to insufficient data. For a more comprehensive test, run `workload` for a longer duration (e.g., two hours). The `tpmC` (new order transactions/minute) number is the headline number and `efc` ("efficiency") tells you how close CockroachDB gets to theoretical maximum `tpmC`. - -The [TPC-C specification](http://www.tpc.org/tpc_documents_current_versions/pdf/tpc-c_v5.11.0.pdf) has p90 latency requirements in the order of seconds, but as you see here, CockroachDB far surpasses that requirement with p90 latencies in the tens of milliseconds. - -## Step 5. Clean up - -1. When you're done with your test cluster, use the [`cockroach quit`](cockroach-quit.html) command to gracefully shut down each node. - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach quit --insecure --host=localhost:26257 - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach quit --insecure --host=localhost:26258 - ~~~ - - {{site.data.alerts.callout_info}} - For the last node, the shutdown process will take longer (about a minute each) and will eventually force the node to stop. This is because, with only 1 of 3 nodes left, all ranges no longer have a majority of replicas available, and so the cluster is no longer operational. - {{site.data.alerts.end}} - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach quit --insecure --host=localhost:26259 - ~~~ - -2. To restart the cluster at a later time, run the same `cockroach start` commands as earlier from the directory containing the nodes' data stores. - - If you do not plan to restart the cluster, you may want to remove the nodes' data stores: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ rm -rf tpcc-local1 tpcc-local2 tpcc-local3 - ~~~ diff --git a/src/current/v21.1/performance-benchmarking-with-tpcc-medium.md b/src/current/v21.1/performance-benchmarking-with-tpcc-medium.md deleted file mode 100644 index 1d6b5f19577..00000000000 --- a/src/current/v21.1/performance-benchmarking-with-tpcc-medium.md +++ /dev/null @@ -1,274 +0,0 @@ ---- -title: Performance Benchmarking with TPC-C -summary: Benchmark CockroachDB against TPC-C with 15 nodes on `c5d.4xlarge` machines -toc: true -toc_not_nested: true -key: performance-benchmarking-with-tpc-c-10k-warehouses.html ---- - -This page shows you how to reproduce [CockroachDB's TPC-C performance benchmarking results](performance.html#scale) on commodity AWS hardware. Across all scales, CockroachDB can process tpmC (new order transactions per minute) at near maximum efficiency. Start by choosing the scale you're interested in: - -
    - - - - -
    - -| Workload | Cluster size | Warehouses | Data size | -|----------+------------------------------------+------------+-----------| -| Local | 3 nodes on your laptop | 10 | 2 GB | -| Small | 3 nodes on `c5d.4xlarge` machines | 2500 | 200 GB | -| Medium | 15 nodes on `c5d.4xlarge` machines | 13,000 | 1.04 TB | -| Large | 81 nodes on `c5d.9xlarge` machines | 140,000 | 11.2 TB | - -## Before you begin - -- [Review TPC-C concepts](#review-tpc-c-concepts) -- [Request a trial license](#request-a-trial-license) - -### Review TPC-C concepts - -TPC-C provides the most realistic and objective measure for OLTP performance at various scale factors. Before you get started, consider reviewing [what TPC-C is and how it is measured](performance.html#tpc-c). - -### Request a trial license - -Reproducing these TPC-C results involves using CockroachDB's [partitioning](partitioning.html) feature to ensure replicas for any given section of data are located on the same nodes that will be queried by the load generator for that section of data. Partitioning helps distribute the workload evenly across the cluster. - -The partitioning feature requires an Enterprise license, so [request a 30-day trial license](https://www.cockroachlabs.com/get-cockroachdb/enterprise/) before you get started. - -You should receive your trial license via email within a few minutes. You'll enable your license once your cluster is up-and-running. - -## Step 1. Set up the environment - -- [Provision VMs](#provision-vms) -- [Configure your network](#configure-your-network) - -### Provision VMs - -1. [Create 16 VM instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/LaunchingAndUsingInstances.html), 15 for CockroachDB nodes and 1 for the TPC-C workload. - - Create all instances in the same region and the same security group. - - Use the `c5d.4xlarge` machine type. - - Use [local SSD instance store volumes](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html#instance-store-volumes). Local SSDs are low latency disks attached to each VM, which maximizes performance. This configuration best resembles what a bare metal deployment would look like, with machines directly connected to one physical disk each. We do not recommend using network-attached block storage. - -2. Note the internal IP address of each instance. You'll need these addresses when starting the CockroachDB nodes. - -{{site.data.alerts.callout_danger}} -This configuration is intended for performance benchmarking only. For production deployments, there are other important considerations, such as security, load balancing, and data location techniques to minimize network latency. For more details, see the [Production Checklist](recommended-production-settings.html). -{{site.data.alerts.end}} - -### Configure your network - -CockroachDB requires TCP communication on two ports: - -- `26257` for inter-node communication (i.e., working as a cluster) and for the TPC-C workload to connect to nodes -- `8080` for exposing your DB Console - -[Create inbound rules](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html#adding-security-group-rule) for your security group: - -#### Inter-node and TPCC-to-node communication - - Field | Recommended Value --------|------------------- - Type | Custom TCP Rule - Protocol | TCP - Port Range | **26257** - Source | The name of your security group (e.g., *sg-07ab277a*) - -#### DB Console - - Field | Recommended Value --------|------------------- - Type | Custom TCP Rule - Protocol | TCP - Port Range | **8080** - Source | Your network's IP ranges - -## Step 2. Start CockroachDB - -{% include {{ page.version.version }}/prod-deployment/insecure-flag.md %} - -1. SSH to the first VM where you want to run a CockroachDB node. - -2. Download the [CockroachDB archive](https://binaries.cockroachdb.com/cockroach-{{ page.release_info.version }}.linux-amd64.tgz) for Linux, extract the binary, and copy it into the `PATH`: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ curl https://binaries.cockroachdb.com/cockroach-{{ page.release_info.version }}.linux-amd64.tgz \ - | tar -xz - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cp -i cockroach-{{ page.release_info.version }}.linux-amd64/cockroach /usr/local/bin/ - ~~~ - - If you get a permissions error, prefix the command with `sudo`. - -3. Run the [`cockroach start`](cockroach-start.html) command: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --insecure \ - --advertise-addr= \ - --join=,, \ - --cache=.25 \ - --locality=rack=0 \ - --background - ~~~ - - Each node will start with a [locality](cockroach-start.html#locality) that includes an artificial "rack number" (e.g., `--locality=rack=0`). Use 5 racks for 15 nodes so that 3 nodes will be assigned to each rack. - -4. Repeat steps 1 - 3 for the other 14 VMs for CockroachDB nodes. Each time, be sure to: - - Adjust the `--advertise-addr` flag. - - Set the [`--locality`](cockroach-start.html#locality) flag to the appropriate "rack number", as described above. - -5. On any of the VMs with the `cockroach` binary, run the one-time [`cockroach init`](cockroach-init.html) command to join the first nodes into a cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach init --insecure --host=
    - ~~~ - -## Step 3. Configure the cluster - -You'll be importing a large TPC-C data set. To speed that up, you can tweak some cluster settings. You'll also need to enable the Enterprise license you requested earlier. - -1. SSH to any VM with the `cockroach` binary. - -2. Launch the [built-in SQL shell](cockroach-sql.html): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --insecure --host=
    - ~~~ - -3. Adjust some [cluster settings](cluster-settings.html): - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING rocksdb.ingest_backpressure.l0_file_count_threshold = 100; - SET CLUSTER SETTING rocksdb.ingest_backpressure.pending_compaction_threshold = '5 GiB'; - SET CLUSTER SETTING schemachanger.backfiller.max_buffer_size = '5 GiB'; - SET CLUSTER SETTING kv.snapshot_rebalance.max_rate = '128 MiB'; - SET CLUSTER SETTING rocksdb.min_wal_sync_interval = '500us'; - SET CLUSTER SETTING kv.range_merge.queue_enabled = false; - ~~~ - -4. Enable the trial license you requested earlier: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING cluster.organization = ''; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING enterprise.license = ''; - ~~~ - -5. Exit the SQL shell: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > \q - ~~~ - -## Step 4. Import the TPC-C dataset - -CockroachDB comes with a number of [built-in workloads](cockroach-workload.html) for simulating client traffic. This step features CockroachDB's version of the [TPC-C](http://www.tpc.org/tpcc/) workload. - -1. SSH to the VM where you want to run TPC-C. - -1. Download the [CockroachDB archive](https://binaries.cockroachdb.com/cockroach-{{ page.release_info.version }}.linux-amd64.tgz) for Linux, extract the binary, and copy it into the `PATH`: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ curl https://binaries.cockroachdb.com/cockroach-{{ page.release_info.version }}.linux-amd64.tgz \ - | tar -xz - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cp -i cockroach-{{ page.release_info.version }}.linux-amd64/cockroach /usr/local/bin/ - ~~~ - - If you get a permissions error, prefix the command with `sudo`. - -1. Import the TPC-C dataset: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload fixtures import tpcc \ - --partitions=5 \ - --warehouses=13000 \ - 'postgres://root@
    :26257?sslmode=disable' - ~~~ - - This will load 1.04 TB of data for 13,000 "warehouses". This can take around 1 hour to complete. - - You can monitor progress on the **Jobs** screen of the DB Console. Open the [DB Console](ui-overview.html) by pointing a browser to the address in the `admin` field in the standard output of any node on startup. - -## Step 5. Allow the cluster to rebalance - -Next, [partition your database](partitioning.html) to divide all of the TPC-C tables and indexes into 5 partitions, one per rack, and then use [zone configurations](configure-replication-zones.html) to pin those partitions to a particular rack. - -1. Still on the same VM, briefly run TPC-C to let the cluster balance and the leases settle. Bump the file descriptor limits with `ulimit` to the high value shown below, since the workload generators create a lot of database connections. - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ ulimit -n 100000 && cockroach workload run tpcc \ - --partitions=5 \ - --warehouses=13000 \ - --duration=1m \ - --ramp=1ms \ - 'postgres://root@
    :26257?sslmode=disable' - ~~~ - -2. Wait for range rebalancing to finish. - - This will likely take 10s of minutes. To watch the progress, go to the **Metrics > Queues > Replication Queue** graph in the DB Console. Once the **Replication Queue** gets to `0` for all actions and stays there, you can move on to the next step. - -## Step 6. Run the benchmark - -1. Back on the VM with the `workload` binary, create an `addrs` file containing connection strings to all 15 CockroachDB nodes: - - ~~~ - postgres://root@:26257?sslmode=disable postgres://root@:26257?sslmode=disable postgres://root@:26257?sslmode=disable postgres://root@:26257?sslmode=disable ... - ~~~ - -2. Run TPC-C for 30 minutes: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ ulimit -n 100000 && cockroach workload run tpcc \ - --partitions=5 \ - --warehouses=13000 \ - --ramp=1m \ - --duration=30m \ - $(cat addrs) - ~~~ - -## Step 7. Interpret the results - -Once the workload has finished running, you will see a result similar to the following. The efficiency and latency can be combined to determine whether this was a passing run. You should expect to see an efficiency number above 95%, well above the required minimum of 85%, and p95 latencies well below the required maximum of 10 seconds. - -~~~ -_elapsed_______tpmC____efc__avg(ms)__p50(ms)__p90(ms)__p95(ms)__p99(ms)_pMax(ms) - 1800.0s 160420.0 96.0% 303.1 285.2 570.4 671.1 939.5 4160.7 -~~~ - -## See also - -- [Performance Overview](performance.html) - -- Hardware - - CockroachDB works well on commodity hardware in public cloud, private cloud, on-prem, and hybrid environments. For hardware recommendations, see our [Production Checklist](recommended-production-settings.html#hardware). - - Also note that CockroachDB creates a yearly cloud report focused on evaluating hardware performance. For more information, see the [2020 Cloud Report](https://www.cockroachlabs.com/blog/2020-cloud-report/). - -- Performance Tuning - - For guidance on tuning a real workload's performance, see [SQL Best Practices](performance-best-practices-overview.html), and for guidance on techniques to minimize network latency in multi-region or global clusters, see [Multi-Region Overview](multiregion-overview.html). diff --git a/src/current/v21.1/performance-benchmarking-with-tpcc-small.md b/src/current/v21.1/performance-benchmarking-with-tpcc-small.md deleted file mode 100644 index d9014ded50b..00000000000 --- a/src/current/v21.1/performance-benchmarking-with-tpcc-small.md +++ /dev/null @@ -1,192 +0,0 @@ ---- -title: Performance Benchmarking with TPC-C -summary: Learn how to benchmark CockroachDB against TPC-C with 3 nodes on `c5d.4xlarge` machines -toc: true -toc_not_nested: true -key: performance-benchmarking-with-tpc-c-1k-warehouses.html ---- - -This page shows you how to reproduce [CockroachDB's TPC-C performance benchmarking results](performance.html#scale) on commodity AWS hardware. Across all scales, CockroachDB can process tpmC (new order transactions per minute) at near maximum efficiency. Start by choosing the scale you're interested in: - -
    - - - - -
    - -| Workload | Cluster size | Warehouses | Data size | -|----------+------------------------------------+------------+-----------| -| Local | 3 nodes on your laptop | 10 | 2 GB | -| Small | 3 nodes on `c5d.4xlarge` machines | 2500 | 200 GB | -| Medium | 15 nodes on `c5d.4xlarge` machines | 13,000 | 1.04 TB | -| Large | 81 nodes on `c5d.9xlarge` machines | 140,000 | 11.2 TB | - -## Before you begin - -### Review TPC-C concepts - -TPC-C provides the most realistic and objective measure for OLTP performance at various scale factors. Before you get started, consider reviewing [what TPC-C is and how it is measured](performance.html#tpc-c). - -## Step 1. Set up the environment - -- [Provision VMs](#provision-vms) -- [Configure your network](#configure-your-network) - -### Provision VMs - -1. [Create 4 VM instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/LaunchingAndUsingInstances.html), 3 for CockroachDB nodes and 1 for the TPC-C workload. - - Create all instances in the same region and the same security group. - - Use the `c5d.4xlarge` machine type. - - Use [local SSD instance store volumes](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html#instance-store-volumes). Local SSDs are low latency disks attached to each VM, which maximizes performance. This configuration best resembles what a bare metal deployment would look like, with machines directly connected to one physical disk each. We do not recommend using network-attached block storage. - -2. Note the internal IP address of each instance. You'll need these addresses when starting the CockroachDB nodes. - -{{site.data.alerts.callout_danger}} -This configuration is intended for performance benchmarking only. For production deployments, there are other important considerations, such as security, load balancing, and data location techniques to minimize network latency. For more details, see the [Production Checklist](recommended-production-settings.html). -{{site.data.alerts.end}} - -### Configure your network - -CockroachDB requires TCP communication on two ports: - -- `26257` for inter-node communication (i.e., working as a cluster) and for the TPC-C workload to connect to nodes -- `8080` for exposing your DB Console - -[Create inbound rules](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html#adding-security-group-rule) for your security group: - -#### Inter-node and TPCC-to-node communication - - Field | Recommended Value --------|------------------- - Type | Custom TCP Rule - Protocol | TCP - Port Range | **26257** - Source | The name of your security group (e.g., *sg-07ab277a*) - -#### DB Console - - Field | Recommended Value --------|------------------- - Type | Custom TCP Rule - Protocol | TCP - Port Range | **8080** - Source | Your network's IP ranges - -## Step 2. Start CockroachDB - -{% include {{ page.version.version }}/prod-deployment/insecure-flag.md %} - -1. SSH to the first VM where you want to run a CockroachDB node. - -2. Download the [CockroachDB archive](https://binaries.cockroachdb.com/cockroach-{{ page.release_info.version }}.linux-amd64.tgz) for Linux, extract the binary, and copy it into the `PATH`: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ curl https://binaries.cockroachdb.com/cockroach-{{ page.release_info.version }}.linux-amd64.tgz \ - | tar -xz - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cp -i cockroach-{{ page.release_info.version }}.linux-amd64/cockroach /usr/local/bin/ - ~~~ - - If you get a permissions error, prefix the command with `sudo`. - -3. Run the [`cockroach start`](cockroach-start.html) command: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --insecure \ - --advertise-addr= \ - --join=,, \ - --cache=.25 \ - --background - ~~~ - -4. Repeat steps 1 - 3 for the other 2 VMs for CockroachDB nodes. Each time, be sure to adjust the `--advertise-addr` flag. - -5. On any of the VMs with the `cockroach` binary, run the one-time [`cockroach init`](cockroach-init.html) command to join the first nodes into a cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach init --insecure --host=
    - ~~~ - -## Step 3. Import the TPC-C dataset - -CockroachDB comes with a number of [built-in workloads](cockroach-workload.html) for simulating client traffic. This step features CockroachDB's version of the [TPC-C](http://www.tpc.org/tpcc/) workload. - -1. SSH to the VM where you want to run TPC-C. - -1. Download the [CockroachDB archive](https://binaries.cockroachdb.com/cockroach-{{ page.release_info.version }}.linux-amd64.tgz) for Linux, extract the binary, and copy it into the `PATH`: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ curl https://binaries.cockroachdb.com/cockroach-{{ page.release_info.version }}.linux-amd64.tgz \ - | tar -xz - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cp -i cockroach-{{ page.release_info.version }}.linux-amd64/cockroach /usr/local/bin/ - ~~~ - - If you get a permissions error, prefix the command with `sudo`. - -1. Import the TPC-C dataset: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload fixtures import tpcc \ - --warehouses=2500 \ - 'postgres://root@
    :26257?sslmode=disable' - ~~~ - - This will load 200 GB of data for 2500 "warehouses". This can take a while to complete. - - You can monitor progress on the **Jobs** screen of the DB Console. Open the [DB Console](ui-overview.html) by pointing a browser to the address in the `admin` field in the standard output of any node on startup. - -## Step 4. Run the benchmark - -1. Still on the same VM, create an `addrs` file containing connection strings to the 3 CockroachDB nodes: - - ~~~ - postgres://root@:26257?sslmode=disable postgres://root@:26257?sslmode=disable postgres://root@:26257?sslmode=disable - ~~~ - -1. Run TPC-C for 30 minutes: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload run tpcc \ - --warehouses=2500 \ - --ramp=1m \ - --duration=30m \ - $(cat addrs) - ~~~ - -## Step 5. Interpret the results - -Once the workload has finished running, you will see a result similar to the following. The efficiency and latency can be combined to determine whether this was a passing run. You should expect to see an efficiency number above 95%, well above the required minimum of 85%, and p95 latencies well below the required maximum of 10 seconds. - -~~~ -_elapsed_______tpmC____efc__avg(ms)__p50(ms)__p90(ms)__p95(ms)__p99(ms)_pMax(ms) - 1800.0s 31064.6 96.6% 107.4 88.1 243.3 302.0 402.7 973.1 -~~~ - -## See also - -- [Performance Overview](performance.html) - -- Hardware - - CockroachDB works well on commodity hardware in public cloud, private cloud, on-prem, and hybrid environments. For hardware recommendations, see our [Production Checklist](recommended-production-settings.html#hardware). - - Also note that CockroachDB creates a yearly cloud report focused on evaluating hardware performance. For more information, see the [2020 Cloud Report](https://www.cockroachlabs.com/blog/2020-cloud-report/). - -- Performance Tuning - - For guidance on tuning a real workload's performance, see [SQL Best Practices](performance-best-practices-overview.html), and for guidance on techniques to minimize network latency in multi-region or global clusters, see [Multi-Region Overview](multiregion-overview.html). diff --git a/src/current/v21.1/performance-best-practices-overview.md b/src/current/v21.1/performance-best-practices-overview.md deleted file mode 100644 index 85e68bda2a7..00000000000 --- a/src/current/v21.1/performance-best-practices-overview.md +++ /dev/null @@ -1,395 +0,0 @@ ---- -title: SQL Performance Best Practices -summary: Best practices for optimizing SQL performance in CockroachDB. -toc: true ---- - -This page provides best practices for optimizing query performance in CockroachDB. - -{{site.data.alerts.callout_success}} -For guidance on deployment and data location techniques to minimize network latency, see [Topology Patterns](topology-patterns.html). -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -If you aren't sure whether SQL query performance needs to be improved on your cluster, see [Identify slow queries](query-behavior-troubleshooting.html#identify-slow-statements). -{{site.data.alerts.end}} - -## DML best practices - -### Use multi-row statements instead of multiple single-row statements - -For `INSERT`, `UPSERT`, and `DELETE` statements, a single multi-row statement is faster than multiple single-row statements. Whenever possible, use multi-row statements for DML queries instead of multiple single-row statements. - -For more information, see: - -- [Insert Data](insert-data.html) -- [Update Data](update-data.html) -- [Delete Data](delete-data.html) -- [How to improve IoT application performance with multi-row DML](https://www.cockroachlabs.com/blog/multi-row-dml/) - -### Use `UPSERT` instead of `INSERT ON CONFLICT` on tables with no secondary indexes - -When inserting/updating all columns of a table, and the table has no secondary -indexes, we recommend using an [`UPSERT`](upsert.html) statement instead of the -equivalent [`INSERT ON CONFLICT`](insert.html) statement. Whereas `INSERT ON -CONFLICT` always performs a read to determine the necessary writes, the `UPSERT` -statement writes without reading, making it faster. For tables with secondary -indexes, there is no performance difference between `UPSERT` and `INSERT ON -CONFLICT`. - -This issue is particularly relevant when using a simple SQL table of two columns -to [simulate direct KV access](sql-faqs.html#can-i-use-cockroachdb-as-a-key-value-store). -In this case, be sure to use the `UPSERT` statement. - -## Bulk-insert best practices - -### Use multi-row `INSERT` statements for bulk-inserts into existing tables - -To bulk-insert data into an existing table, batch multiple rows in one multi-row `INSERT` statement. Experimentally determine the optimal batch size for your application by monitoring the performance for different batch sizes (10 rows, 100 rows, 1000 rows). Do not include bulk `INSERT` statements within an explicit transaction. - -{{site.data.alerts.callout_success}} -You can also use the [`IMPORT INTO`](import-into.html) statement to bulk-insert CSV data into an existing table. -{{site.data.alerts.end}} - -For more information, see [Insert Multiple Rows](insert.html#insert-multiple-rows-into-an-existing-table). - -{{site.data.alerts.callout_info}} -Large multi-row `INSERT` queries can lead to long-running transactions that result in [transaction retry errors](transaction-retry-error-reference.html). If a multi-row `INSERT` query results in an error code [`40001` with the message `"transaction deadline exceeded"`](transaction-retry-error-reference.html#retry_commit_deadline_exceeded), we recommend breaking up the query up into smaller batches of rows. -{{site.data.alerts.end}} - -### Use `IMPORT` instead of `INSERT` for bulk-inserts into new tables - -To bulk-insert data into a brand new table, the [`IMPORT`](import.html) statement performs better than `INSERT`. - -## Bulk-update best practices - -### Use batch updates to delete a large number of rows - -To delete a large number of rows, we recommend iteratively deleting batches of rows until all of the unwanted rows are deleted. For an example, see [Bulk-update Data](bulk-update-data.html). - -## Bulk-delete best practices - -### Use `TRUNCATE` instead of `DELETE` to delete all rows in a table - -The [`TRUNCATE`](truncate.html) statement removes all rows from a table by dropping the table and recreating a new table with the same name. This performs better than using `DELETE`, which performs multiple transactions to delete all rows. - -### Use batch deletes to delete a large number of rows - -To delete a large number of rows, we recommend iteratively deleting batches of rows until all of the unwanted rows are deleted. For an example, see [Bulk-delete Data](bulk-delete-data.html). - -### Batch delete "expired" data - -CockroachDB does not support Time to Live (TTL) on table rows. To delete "expired" rows, we recommend automating a batch delete process with a job scheduler like `cron`. For an example, see [Batch-delete "expired" data](bulk-delete-data.html#batch-delete-expired-data). - -## Assign column families - -A column family is a group of columns in a table that is stored as a single key-value pair in the underlying key-value store. - -When a table is created, all columns are stored as a single column family. This default approach ensures efficient key-value storage and performance in most cases. However, when frequently updated columns are grouped with seldom updated columns, the seldom updated columns are nonetheless rewritten on every update. Especially when the seldom updated columns are large, it's therefore more performant to [assign them to a distinct column family](column-families.html). - -## Unique ID best practices - -The best practices for generating unique IDs in a distributed database like CockroachDB are very different than for a legacy single-node database. Traditional approaches for generating unique IDs for legacy single-node databases include: - -1. Using the [`SERIAL`](serial.html) pseudo-type for a column to generate random unique IDs. This can result in a performance bottleneck because IDs generated temporally near each other have similar values and are located physically near each other in a table's storage. -2. Generating monotonically increasing [`INT`](int.html) IDs by using transactions with roundtrip [`SELECT`](select-clause.html)s, e.g., `INSERT INTO tbl (id, …) VALUES ((SELECT max(id)+1 FROM tbl), …)`. This has a **very high performance cost** since it makes all [`INSERT`](insert.html) transactions wait for their turn to insert the next ID. You should only do this if your application really does require strict ID ordering. In some cases, using [Change Data Capture (CDC)](stream-data-out-of-cockroachdb-using-changefeeds.html) can help avoid the requirement for strict ID ordering. If you can avoid the requirement for strict ID ordering, you can use one of the higher performance ID strategies outlined below. - -The approaches described above are likely to create hotspots for both reads and writes in CockroachDB. To avoid this issue, we recommend the following approaches (listed in order from best to worst performance). - -| Approach | Pros | Cons | -|--------------------------------------------------------------------------------------+--------------------------------------------------+-----------------------------------------------------------------------------------------| -| 1. [Use multi-column primary keys](#use-multi-column-primary-keys) | Potentially fastest, if done right | Complex, requires up-front design and testing to ensure performance | -| 2. [Use `UUID` to generate unique IDs](#use-uuid-to-generate-unique-ids) | Good performance; spreads load well; easy choice | May leave some performance on the table; requires other columns to be useful in queries | -| 3. [Use `INSERT` with the `RETURNING` clause](#use-insert-with-the-returning-clause-to-generate-unique-ids) | Easy to query against; familiar design | Slower performance than the other options; higher chance of transaction contention | - -### Use multi-column primary keys - -A well-designed multi-column primary key can yield even better performance than a [UUID primary key](#use-uuid-to-generate-unique-ids), but it requires more up-front schema design work. To get the best performance, ensure that any monotonically increasing field is located **after** the first column of the primary key. When done right, such a composite primary key should result in: - -- Enough randomness in your primary key to spread the table data / query load relatively evenly across the cluster, which will avoid hotspots. By "enough randomness" we mean that the prefix of the primary key should be relatively uniformly distributed over its domain. Its domain should have at least as many elements as you have nodes. -- A monotonically increasing column that is part of the primary key (and thus indexed) which is also useful in your queries. - -For example, consider a social media website. Social media posts are written by users, and on login the user's last 10 posts are displayed. A good choice for a primary key might be `(username, post_timestamp)`. For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE posts ( - username STRING, - post_timestamp TIMESTAMP, - post_id INT, - post_content STRING, - CONSTRAINT posts_pk PRIMARY KEY(username, post_timestamp) -); -~~~ - -This would make the following query efficient. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM posts - WHERE username = 'alyssa' - ORDER BY post_timestamp DESC - LIMIT 10; -~~~ - -~~~ - username | post_timestamp | post_id | post_content -+----------+---------------------------+---------+--------------+ - alyssa | 2019-07-31 18:01:00+00:00 | ... | ... - alyssa | 2019-07-30 10:22:00+00:00 | ... | ... - alyssa | 2019-07-30 09:12:00+00:00 | ... | ... - alyssa | 2019-07-29 13:48:00+00:00 | ... | ... - alyssa | 2019-07-29 13:47:00+00:00 | ... | ... - alyssa | 2019-07-29 13:46:00+00:00 | ... | ... - alyssa | 2019-07-29 13:43:00+00:00 | ... | ... - ... - -Time: 924µs -~~~ - -To see why, let's look at the [`EXPLAIN`](explain.html) output. It shows that the query is fast because it does a point lookup on the indexed column `username` (as shown by the line `spans | /"alyssa"-...`). Furthermore, the column `post_timestamp` is already in an index, and sorted (since it's a monotonically increasing part of the primary key). - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN (VERBOSE) - SELECT * FROM posts - WHERE username = 'alyssa' - ORDER BY post_timestamp DESC - LIMIT 10; -~~~ - -~~~ - info ----------------------------------------------------------------- - distribution: local - vectorized: true - - • revscan - columns: (username, post_timestamp, post_id, post_content) - ordering: -post_timestamp - estimated row count: 10 (missing stats) - table: posts@posts_pk - spans: /"alyssa"-/"alyssa"/PrefixEnd - limit: 10 -(10 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -Note that the above query also follows the [indexing best practice](indexes.html#best-practices) of indexing all columns in the `WHERE` clause. - -### Use `UUID` to generate unique IDs - -To auto-generate unique row identifiers, use the [`UUID`](uuid.html) column with the `gen_random_uuid()` [function](functions-and-operators.html#id-generation-functions) as the [default value](default-value.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE t1 (id UUID PRIMARY KEY DEFAULT gen_random_uuid(), name STRING); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO t1 (name) VALUES ('a'), ('b'), ('c'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM t1; -~~~ - -~~~ -+--------------------------------------+------+ -| id | name | -+--------------------------------------+------+ -| 60853a85-681d-4620-9677-946bbfdc8fbc | c | -| 77c9bc2e-76a5-4ebc-80c3-7ad3159466a1 | b | -| bd3a56e1-c75e-476c-b221-0da9d74d66eb | a | -+--------------------------------------+------+ -(3 rows) -~~~ - -### Use `INSERT` with the `RETURNING` clause to generate unique IDs - -If something prevents you from using [multi-column primary keys](#use-multi-column-primary-keys) or [`UUID`s](#use-uuid-to-generate-unique-ids) to generate unique IDs, you might resort to using `INSERT`s with `SELECT`s to return IDs. Instead, [use the `RETURNING` clause with the `INSERT` statement](insert.html#insert-and-return-values) as shown below for improved performance. - -#### Generate monotonically-increasing unique IDs - -Suppose the table schema is as follows: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE X ( - ID1 INT, - ID2 INT, - ID3 INT DEFAULT 1, - PRIMARY KEY (ID1,ID2) - ); -~~~ - -The common approach would be to use a transaction with an `INSERT` followed by a `SELECT`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; - -> INSERT INTO X VALUES (1,1,1) - ON CONFLICT (ID1,ID2) - DO UPDATE SET ID3=X.ID3+1; - -> SELECT * FROM X WHERE ID1=1 AND ID2=1; - -> COMMIT; -~~~ - -However, the performance best practice is to use a `RETURNING` clause with `INSERT` instead of the transaction: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO X VALUES (1,1,1),(2,2,2),(3,3,3) - ON CONFLICT (ID1,ID2) - DO UPDATE SET ID3=X.ID3 + 1 - RETURNING ID1,ID2,ID3; -~~~ - -#### Generate random unique IDs - -Suppose the table schema is as follows: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE X ( - ID1 INT, - ID2 INT, - ID3 INT DEFAULT unique_rowid(), - PRIMARY KEY (ID1,ID2) - ); -~~~ - -The common approach to generate random Unique IDs is a transaction using a `SELECT` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; - -> INSERT INTO X VALUES (1,1); - -> SELECT * FROM X WHERE ID1=1 AND ID2=1; - -> COMMIT; -~~~ - -However, the performance best practice is to use a `RETURNING` clause with `INSERT` instead of the transaction: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO X VALUES (1,1),(2,2),(3,3) - RETURNING ID1,ID2,ID3; -~~~ - -## Secondary index best practices - -See [Secondary Index Best Practices](schema-design-indexes.html#best-practices). - -## Join best practices - -See [Join Performance Best Practices](joins.html#performance-best-practices). - -## Subquery best practices - -See [Subquery Performance Best Practices](subqueries.html#performance-best-practices). - -## Table scans best practices - -### Avoid `SELECT *` for large tables - -For large tables, avoid table scans (that is, reading the entire table data) whenever possible. Instead, define the required fields in a `SELECT` statement. - -#### Example - -Suppose the table schema is as follows: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE accounts ( - id INT, - customer STRING, - address STRING, - balance INT - nominee STRING - ); -~~~ - -Now if we want to find the account balances of all customers, an inefficient table scan would be: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM ACCOUNTS; -~~~ - -This query retrieves all data stored in the table. A more efficient query would be: - -{% include_cached copy-clipboard.html %} -~~~ sql - > SELECT CUSTOMER, BALANCE FROM ACCOUNTS; -~~~ - -This query returns the account balances of the customers. - -### Avoid `SELECT DISTINCT` for large tables - -`SELECT DISTINCT` allows you to obtain unique entries from a query by removing duplicate entries. However, `SELECT DISTINCT` is computationally expensive. As a performance best practice, use [`SELECT` with the `WHERE` clause](select-clause.html#filter-rows) instead. - -### Use `AS OF SYSTEM TIME` to decrease conflicts with long-running queries - -If you have long-running queries (such as analytics queries that perform full table scans) that can tolerate slightly out-of-date reads, consider using the [`... AS OF SYSTEM TIME` clause](select-clause.html#select-historical-data-time-travel). Using this, your query returns data as it appeared at a distinct point in the past and will not cause [conflicts](architecture/transaction-layer.html#transaction-conflicts) with other concurrent transactions, which can increase your application's performance. - -However, because `AS OF SYSTEM TIME` returns historical data, your reads might be stale. - -## Understanding and avoiding transaction contention - -Transaction contention occurs when the following three conditions are met: - -- There are multiple concurrent transactions or statements (sent by multiple clients connected simultaneously to a single CockroachDB cluster). -- They operate on the same data, specifically over table rows with the same index key values (either on [primary keys](primary-key.html) or secondary [indexes](indexes.html)) or using index key values that are close to each other, and thus place the indexed data on the same [data ranges](architecture/overview.html). -- At least some of the transactions write or modify the data. - -A set of transactions that all contend on the same keys will be limited in performance to the maximum processing speed of a single node (limited horizontal scalability). Non-contended transactions are not affected in this way. - -There are two levels of contention: - -- Transactions that operate on the same range but different index key values will be limited by the overall hardware capacity of a single node (the range lease holder). - -- Transactions that operate on the same index key values (specifically, that operate on the same [column family](column-families.html) for a given index key) will be more strictly serialized to obey transaction isolation semantics. - -Transaction contention can also increase the rate of transaction restarts, and thus make the proper implementation of [client-side transaction retries](transactions.html#client-side-intervention) more critical. - -### Find contention - -{% include {{ page.version.version }}/performance/statement-contention.md %} - -### Avoid contention - -To avoid contention, multiple strategies can be applied: - -- Use index key values with a more random distribution of values, so that transactions over different rows are more likely to operate on separate data ranges. See the [SQL FAQs](sql-faqs.html) on row IDs for suggestions. - -- Make transactions smaller, so that each transaction has less work to do. In particular, avoid multiple client-server exchanges per transaction. For example, use [common table expressions](common-table-expressions.html) to group multiple [`SELECT`](select-clause.html) and [`INSERT`](insert.html)/[`UPDATE`](update.html)/[`DELETE`](delete.html)/[`UPSERT`](upsert.html) clauses together in a single SQL statement. - - - For an example showing how to break up large transactions in an application, see [Break up large transactions into smaller units of work](build-a-python-app-with-cockroachdb-sqlalchemy.html#break-up-large-transactions-into-smaller-units-of-work). - - If you are experiencing contention (retries) when doing bulk deletes, see [Bulk-delete data](bulk-delete-data.html). - -- In combination with the above, if you are able to [send all of the statements in your transaction in a single batch](transactions.html#batched-statements), CockroachDB can automatically retry the transaction for you. - -- Use the [`SELECT FOR UPDATE`](select-for-update.html) statement in scenarios where a transaction performs a read and then updates the row(s) it just read. It orders transactions by controlling concurrent access to one or more rows of a table. It works by locking the rows returned by a [selection query](selection-queries.html), such that other transactions trying to access those rows are forced to wait for the transaction that locked the rows to finish. These other transactions are effectively put into a queue that is ordered based on when they try to read the value of the locked row(s). - -- When replacing values in a row, use [`UPSERT`](upsert.html) and specify values for all columns in the inserted rows. This will usually have the best performance under contention, compared to combinations of [`SELECT`](select-clause.html), [`INSERT`](insert.html), and [`UPDATE`](update.html). - -- Increase [normalization](https://en.wikipedia.org/wiki/Database_normalization) of the data to place parts of the same records that are modified by different transactions in different tables. Increasing normalization has drawbacks as well as benefits, because denormalization can also increase performance by creating multiple copies of often-referenced data in separate ranges. - -- If the application strictly requires operating on very few different index key values, consider using [`ALTER ... SPLIT AT`](split-at.html) so that each index key value can be served by a separate group of nodes in the cluster. - -It is always best to avoid contention as much as possible via the design of the schema and application. However, sometimes contention is unavoidable. To maximize performance in the presence of contention, you'll need to maximize the performance of a single range. To achieve this, multiple strategies can be applied: - -- Minimize the network distance between the replicas of a range, possibly using zone configs and partitioning. -- Use the fastest storage devices available. -- If the contending transactions operate on different keys within the same range, add more CPU power (more cores) per node. Note however that this is less likely to provide an improvement if the transactions all operate on the same key. diff --git a/src/current/v21.1/performance-recipes-solutions.md b/src/current/v21.1/performance-recipes-solutions.md deleted file mode 100644 index 1f3f416ac99..00000000000 --- a/src/current/v21.1/performance-recipes-solutions.md +++ /dev/null @@ -1,71 +0,0 @@ ---- -title: Performance tuning recipe solutions -summary: Identifying possible fixes for common performance problems -toc: true -toc_not_nested: true ---- - -This page provides solutions for common performance issues in your clusters. See [the recipes](performance-recipes.html) to identify performance problems in your workload. - -
    - - - -
    - -
    -## Indicators that your workoad is experiencing contention - -* Your application is experiencing degraded performance with serialization errors like `SQLSTATE: 40001`, `RETRY_WRITE_TOO_OLD`, and `RETRY_SERIALIZABLE`. -* The [SQL Statement Contention graph](ui-sql-dashboard.html#sql-statement-contention) graph is showing spikes over time. -SQL Statement Contention graph in the DB Console -* The [KV Transaction Restarts graph](ui-overview.html) graph is showing spikes in retries over time. - -## Fixing contention problems - -{% include {{ page.version.version }}/performance/statement-contention.md %} -
    - -
    - -## Indicators that your workload has statements with full table scans - -* The following query returns statements with full table scans in their statement plan: - - {% include_cached copy-clipboard.html %} - ~~~ sql - SHOW FULL TABLE SCANS; - ~~~ -* The following query against the `crdb_internal.node_statement_statistics` table returns results: - - {% include_cached copy-clipboard.html %} - ~~~ sql - SELECT count(*) as total_full_scans - FROM crdb_internal.node_statement_statistics - WHERE FullTableScan = 'True'; - ~~~ -* Viewing the statement plan on the [Statement details page](ui-statements-page.html#statement-details-page) of the DB Console indicates that the plan contains full table scans. -* The statement plans returned by the [`EXPLAIN`](sql-tuning-with-explain.html) and [`EXPLAIN ANALYZE` commands](explain-analyze.html) indicate that there are full table scans. - -## Fixing full table scans in statements - -Full table scans often result in poor statement performance. Not every full table scan is an indicator of poor performance, however. The [cost-based optimizer](cost-based-optimizer.html) may decide on a full table scan when other [index](indexes.html) or [join scans](joins.html) would result in longer execution time. - -[Examine the statements](sql-tuning-with-explain.html) that result in full table scans and consider adding [secondary indexes](schema-design-indexes.html#create-a-secondary-index). - -
    - -
    - -## Indicators that your tables are using suboptimal primary keys - -* The [Hardware metrics dashboard](ui-hardware-dashboard.html) in the DB Console shows high resource usage per node. -* The Problem Ranges report on the [Advanced Debug page](ui-debug-pages.html) of the DB Console indicates a high number of queries per second on a subset of ranges or nodes. - -## Fixing suboptimal primary keys - -Evaluate the schema of your table to see if you can redistribute data more evenly across multiple ranges. Specifically, make sure you have followed [best practices when selecting your primary key](schema-design-table.html#primary-key-best-practices). - -If your workload with a small dataset (for example, a dataset that contains few index key values) is experiencing resource contention, consider splitting your tables and indexes to [distribute ranges across multiple nodes](split-at.html#split-a-table) to reduce resource contention. - -
    \ No newline at end of file diff --git a/src/current/v21.1/performance-recipes.md b/src/current/v21.1/performance-recipes.md deleted file mode 100644 index 7060fe1a98e..00000000000 --- a/src/current/v21.1/performance-recipes.md +++ /dev/null @@ -1,56 +0,0 @@ ---- -title: Performance tuning recipes -summary: Identifying and fixing common performance problems -toc: true -toc_not_nested: true ---- - -This page provides recipes for fixing performance issues in your clusters. - -{{site.data.alerts.callout_info}} -If you aren't sure whether SQL query performance needs to be improved on your cluster, see [Identify slow queries](query-behavior-troubleshooting.html#identify-slow-statements). -{{site.data.alerts.end}} - - -
    - - - - - - - - - - - - - - - - - - - - -
    ProblemPossible solution
    Your application is experiencing high latency.Use the correct topology pattern for your cluster to minimize network latency.
      -
    • Your application is experiencing degraded performance with the following serialization errors: -
        -
      • SQLSTATE: 40001
        • -
      • RETRY_WRITE_TOO_OLD
        • -
      • RETRY_SERIALIZABLE
        • -
      -
    • The SQL Statement Contention dashboard in the DB Console is showing spikes over time.
      • -
    • The SQL Statement Errors graph in the DB Console is showing spikes in retries over time.
      • -
    -     		Your workload is experiencing contention
      -
    • The statement plan produced by EXPLAIN or EXPLAIN ANALYZE indicates that the statement uses a full table scan.
      • -
    • Querying the crdb_internal.node_statement_statistics table indicates that you have full table scans in some statement's plans.
      • -
    • Viewing the statement plan on the Statement details page of the DB Console indicates that the plan contains full table scans.
      • -
    • Running the SHOW FULL TABLE SCANS statement returns results.
      • -
    -     		Poor quality statement plans retrieve more rows than are required, leading to longer execution times
      -
    • The Hardware metrics dashboard in the DB Console shows high resource usage per node.
      • -
    • The Problem Ranges report on the Advanced Debug page of the DB Console indicates a high number of queries per second on a subset of ranges or nodes.
      • -
    -     		Your tables may be using a suboptimal primary key, causing resource contention
    diff --git a/src/current/v21.1/performance.md b/src/current/v21.1/performance.md deleted file mode 100644 index d69d74345a1..00000000000 --- a/src/current/v21.1/performance.md +++ /dev/null @@ -1,100 +0,0 @@ ---- -title: CockroachDB Performance -summary: An overview of the performance profiles you can expect from CockroachDB. -toc: true -toc_not_nested: true ---- - -CockroachDB delivers predictable throughput and latency at all scales on commodity hardware. This page provides an overview of the performance profiles you can expect, based on Cockroach Labs's extensive testing using the TPC-C industry-standard benchmark. - -For instructions to reproduce the TPC-C results listed here, see [Performance Benchmarking with TPC-C](performance-benchmarking-with-tpcc-large.html). If you fail to achieve similar results, there is likely a problem in either the hardware, workload, or test design. - -{{site.data.alerts.callout_success}} -This document is about CockroachDB’s performance on benchmarks. For guidance on tuning real workloads, see [SQL Best Practices](performance-best-practices-overview.html), and for guidance on data location techniques to minimize network latency, see [Topology Patterns](topology-patterns.html). -{{site.data.alerts.end}} - -## Scale - -TPC-C provides the most realistic and objective measure for OLTP performance at various scale factors, and CockroachDB can process **1.68M tpmC with 140,000 warehouses, resulting in an efficiency score of 95%.** As shown in the chart below, this is a 40% improvement over the results from CockroachDB 19.2. - -For a refresher on what exactly TPC-C is and how it is measured, see [Benchmark details](#benchmark-details) below. - -CockroachDB achieves this performance in [`SERIALIZABLE` isolation](demo-serializable.html), the strongest isolation level in the SQL standard. - -TPC-C 140,000 - -| Metric | CockroachDB 19.2 | CockroachDB 21.1 | -|-------------------------------------------------+------------------+------------------| -| Max warehouses with max efficiency (warehouses) | 100,000 | 140,000 | -| Max throughput (tpmC) | 1,245,462 | 1,684,437 | -| Efficiency (%) | 98.81 | 95.45 | -| Max number of rows (billion) | 49.8 | 69.7 | -| Max unreplicated data (TB) | 8 | 11.2 | -| p95 latency for New Order transactions (ms) | 486.50 | 2684.4 | -| Number of nodes | 81 | 81 | - -### Linear scaling - -CockroachDB has **no theoretical scaling limit** and, in practice, can achieve near-linear performance at 256 nodes. Because the TPC-C results above reflect leaps in scale, to test linear scaling, Cockroach Labs ran a simple benchmark named KV 95 (95% point reads, 5% point writes, all uniformly distributed) on AWS `c5d.4xlarge` machines: - -CRDB Linear Scale - -This chart shows that adding nodes increases throughput linearly while holding p50 and p99 latency constant. The concurrency for each scale was chosen to optimize throughput while maintaining an acceptable latency and can be observed in the table below. - -| Number of Nodes | Workers | Concurrency | -|-----------------+---------+-------------| -| 16 | 2 | 512 | -| 32 | 4 | 512 | -| 64 | 4 | 1024 | -| 128 | 8 | 1024 | -| 256 | 8 | 2048 | - -## Throughput - -Cockroach Labs believes TPC-C provides the most realistic and objective measure for OLTP throughput. In the real world, applications generate transactional workloads that consist of a combination of reads and writes, possibly with concurrency and likely without all data being loaded into memory. Approach benchmark results quoted in QPS with caution, because anything as simple as a “query” is unlikely to be representative of the workload you need to run in practice. - -## Latency - -CockroachDB returns single-row **reads in 1 ms** and processes single-row **writes in 2 ms** within a single availability zone. As you expand out to multiple availability zones or multiple regions, latency can increase due to distance and the limitation of the speed of light. - -For benchmarking latency, again, Cockroach Labs believes TPC-C provides the most realistic and objective measure, since it encompasses the latency distribution, including tail performance. - -CockroachDB provides a number of important tuning practices for both single-region and multi-region deployments, including [secondary indexes](indexes.html) and various [data topologies](topology-patterns.html) to achieve low latency. - -## Benchmark details - -### TPC-C - -Cockroach Labs measures performance through many diverse tests, including the [industry-standard OLTP benchmark TPC-C](http://www.tpc.org/tpcc/), which simulates an e-commerce or retail company. Created in 1992, TPC-C has withstood the test of time and remains the most mature industry benchmark for OLTP workloads, and **the only objective comparison for evaluating OLTP performance**. In its own words, TPC-C: - ->“…involves a mix of five concurrent transactions of different types and complexity either executed on-line or queued for deferred execution. The database is comprised of nine types of tables with a wide range of record and population sizes. While the benchmark portrays the activity of a wholesale supplier, TPC-C is not limited to the activity of any particular business segment, but, rather represents any industry that must manage, sell, or distribute a product or service.” - -As a result, TPC-C includes create, read, update, and delete (e.g., CRUD) queries, basic joins, and other SQL statements used to administer mission-critical transactional workloads. It includes detailed specifications for concurrency and workload contention. - -#### How TPC-C works - -TPC-C measures the throughput and latency for processing sales through a customer warehouse using a “business throughput” metric called **tpmC** that measures the number of order transactions performed per minute throughout the system. This metric is considerably more realistic than TPS (transactions per second) or QPS (queries per second) alone because it summarizes multiple transactions per order and accounts for failed transactions. TPC-C also has several latency requirements that apply to median, p90, and max latencies. - -TPC-C specifies restrictions on the maximum throughput achievable per warehouse. This is done to ensure that as a system becomes progressively more capable of throughput, it must also deal with progressively more data. This is how things work in the real world, and it makes little sense to say that your database can process a bazillion transactions per second if it’s processing the same data over and over again. - -Because TPC-C is constrained to a maximum amount of throughput per warehouse, we often discuss TPC-C performance as the **maximum number of warehouses for which a database can maintain the maximum throughput per minute.** For a full description of the benchmark, please consult the [official documentation](http://www.tpc.org/tpc_documents_current_versions/pdf/tpc-c_v5.11.0.pdf). - -## Performance limitations - -CockroachDB has no theoretical limitations to scaling, throughput, latency, or concurrency other than the speed of light. Practically, we will be improving bottlenecks and addressing challenges over the next several releases. - -## See also - -- Hardware - - CockroachDB works well on commodity hardware in public cloud, private cloud, on-prem, and hybrid environments. For hardware recommendations, see our [Production Checklist](recommended-production-settings.html#hardware). - - Also note that CockroachDB creates a yearly cloud report focused on evaluating hardware performance. For more information, see the [2020 Cloud Report](https://www.cockroachlabs.com/blog/2020-cloud-report/). - -- Performance Tuning - - For guidance on tuning a real workload's performance, see [SQL Best Practices](performance-best-practices-overview.html), and for guidance on techniques to minimize network latency in multi-region or global clusters, see [Multi-Region Overview](multiregion-overview.html). - -- TPC-C Replication Instructions - - For instructions showing how to replicate the TPC-C results described above, see [Performance Benchmarking with TPC-C](performance-benchmarking-with-tpcc-large.html) diff --git a/src/current/v21.1/pg-catalog.md b/src/current/v21.1/pg-catalog.md deleted file mode 100644 index 038a80d3716..00000000000 --- a/src/current/v21.1/pg-catalog.md +++ /dev/null @@ -1,170 +0,0 @@ ---- -title: pg_catalog -summary: The pg_catalog schema contains read-only views that you can use for introspection into your database. -toc: true ---- - -For PostgreSQL compatibility, CockroachDB includes a [system catalog schema](system-catalogs.html) called `pg_catalog`. The tables in the `pg_catalog` schema roughly correspond to the [system catalogs in PostgreSQL](https://www.postgresql.org/docs/13/catalogs.html). `pg_catalog` tables are read-only. - -## Data exposed by `pg_catalog` - -The tables in CockroachDB's `pg_catalog` schema correspond to a subset of the virtual tables and views that make up the PostgreSQL system catalogs. Not all PostgreSQL system catalogs have a corresponding table in `pg_catalog`, and some of the `pg_catalog` tables are empty. See the following table for a detailed comparison between PostgreSQL 13 system catalogs and `pg_catalog` tables. - -PostgreSQL 13 system catalog | `pg_catalog` table ------------------------------|-------------- -`pg_aggregate` | `pg_aggregate` -`pg_am` | `pg_am` -`pg_amop` | `pg_amop` (empty) -`pg_amproc` | `pg_amproc` (empty) -`pg_attrdef` | `pg_attrdef` -`pg_attribute` | `pg_attribute` -`pg_auth_members` | `pg_auth_members` -`pg_authid` | `pg_authid` -`pg_available_extension_versions` | `pg_available_extension_versions` (empty) -`pg_available_extensions` | `pg_available_extensions` -`pg_cast` | `pg_cast` -`pg_class` | `pg_class` -`pg_collation` | `pg_collation` -`pg_config` | `pg_config` (empty) -`pg_constraint` | `pg_constraint` -`pg_conversion` | `pg_conversion` -`pg_cursors` | `pg_cursors` (empty) -`pg_database` | `pg_database` -`pg_db_role_setting` | `pg_db_role_setting` (empty) -`pg_default_acl` | `pg_default_acl` -`pg_depend` | `pg_depend` -`pg_description` | `pg_description` -`pg_enum` | `pg_enum` -`pg_event_trigger` | `pg_event_trigger` -`pg_extension` | `pg_extension` -`pg_file_settings` | `pg_file_settings` (empty) -`pg_foreign_data_wrapper` | `pg_foreign_data_wrapper` -`pg_foreign_server` | `pg_foreign_server` -`pg_foreign_table` | `pg_foreign_table` -`pg_group` | `pg_group` (empty) -`pg_hba_file_rules` | `pg_hba_file_rules` (empty) -`pg_index` | `pg_index` -`pg_indexes` | `pg_indexes` -`pg_inherits` | `pg_inherits` -`pg_init_privs` | None -`pg_language` | `pg_language` (empty) -`pg_largeobject` | `pg_largeobject` (empty) -`pg_largeobject_metadata` | None -`pg_locks` | `pg_locks` -`pg_matviews` | `pg_matviews` -`pg_namespace` | `pg_namespace` -`pg_opclass` | `pg_opclass` (empty) -`pg_operator` | `pg_operator` -`pg_opfamily` | `pg_opfamily` (empty) -`pg_partitioned_table` | None -`pg_policies` | `pg_policies` (empty) -`pg_policy` | None -`pg_prepared_statements` | `pg_prepared_statements` (empty) -`pg_prepared_xacts` | `pg_prepared_xacts` (empty) -`pg_proc` | `pg_proc` -`pg_publication` | `pg_publication` (empty) -`pg_publication_rel` | `pg_publication_rel` (empty) -`pg_publication_tables` | `pg_publication_tables` (empty) -`pg_range` | `pg_range` -`pg_replication_origin` | `pg_replication_origin` (empty) -`pg_replication_origin_status` | None -`pg_replication_slots` | None -`pg_rewrite` | `pg_rewrite` -`pg_roles` | `pg_roles` -`pg_rules` | `pg_rules` (empty) -`pg_seclabel` | `pg_seclabel` -`pg_seclabels` | `pg_seclabels` -`pg_sequence` | `pg_sequence` -`pg_sequences` | None -`pg_settings` | `pg_settings` -`pg_shadow` | `pg_shadow` (empty) -`pg_shdepend` | `pg_shdepend` (empty) -`pg_shdescription` | `pg_shdescription` -`pg_shmem_allocations` | `pg_shmem_allocations` (empty) -`pg_shseclabel` | `pg_shseclabel` -`pg_stat_activity` | `pg_stat_activity` -`pg_statistic` | None -`pg_statistic_ext` | `pg_statistic_ext` (empty) -`pg_statistic_ext_data` | None -`pg_stats` | None -`pg_stats_ext` | None -`pg_subscription` | `pg_subscription` (empty) -`pg_subscription_rel` | None -`pg_tables` | `pg_tables` -`pg_tablespace` | `pg_tablespace` -`pg_timezone_abbrevs` | `pg_timezone_abbrevs` (empty) -`pg_timezone_names` | `pg_timezone_names` (empty) -`pg_transform` | `pg_transform` (empty) -`pg_trigger` | `pg_trigger` -`pg_ts_config` | `pg_ts_config` (empty) -`pg_ts_config_map` | `pg_ts_config_map` (empty) -`pg_ts_dict` | `pg_ts_dict` (empty) -`pg_ts_parser` | `pg_ts_parser` (empty) -`pg_ts_template` | `pg_ts_template` (empty) -`pg_type` | `pg_type` -`pg_user` | `pg_user` -`pg_user_mapping` | `pg_user_mapping` -`pg_user_mappings` | `pg_user_mappings` (empty) -`pg_views` | `pg_views` - -To list the tables in `pg_catalog` for the [current database](sql-name-resolution.html#current-database), use the following [`SHOW TABLES`](show-tables.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM pg_catalog; -~~~ - -~~~ - schema_name | table_name | type | owner | estimated_row_count ---------------+-------------------------+-------+-------+---------------------- - pg_catalog | pg_aggregate | table | NULL | NULL - pg_catalog | pg_am | table | NULL | NULL - ... -~~~ - -{{site.data.alerts.callout_info}} -To prohibit queries against empty tables, set the `stub_catalog_tables` [session variable](set-vars.html) to `off`. -{{site.data.alerts.end}} - -## Query `pg_catalog` tables - -You can run [`SELECT` queries](selection-queries.html) on the tables in `pg_catalog`. - -{{site.data.alerts.callout_success}} -To ensure that you can view all of the tables in `pg_catalog`, query the tables as a user with [`admin` privileges](authorization.html#admin-role). -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -Unless specified otherwise, queries to `pg_catalog` assume the [current database](sql-name-resolution.html#current-database). -{{site.data.alerts.end}} - -For example, to return the `pg_catalog` table with additional information about indexes in [`movr` database](movr.html), you can query the `pg_catalog.pg_indexes` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM movr.pg_catalog.pg_indexes; -~~~ - -~~~ - crdb_oid | schemaname | tablename | indexname | tablespace | indexdef --------------+------------+----------------------------+-----------------------------------------------+------------+--------------------------------------------------------------------------------------------------------------------------------- - 2055313241 | public | users | primary | NULL | CREATE UNIQUE INDEX "primary" ON movr.public.users USING btree (city ASC, id ASC) - 1795576970 | public | vehicles | primary | NULL | CREATE UNIQUE INDEX "primary" ON movr.public.vehicles USING btree (city ASC, id ASC) - 1795576969 | public | vehicles | vehicles_auto_index_fk_city_ref_users | NULL | CREATE INDEX vehicles_auto_index_fk_city_ref_users ON movr.public.vehicles USING btree (city ASC, owner_id ASC) - 450499963 | public | rides | primary | NULL | CREATE UNIQUE INDEX "primary" ON movr.public.rides USING btree (city ASC, id ASC) - 450499960 | public | rides | rides_auto_index_fk_city_ref_users | NULL | CREATE INDEX rides_auto_index_fk_city_ref_users ON movr.public.rides USING btree (city ASC, rider_id ASC) - 450499961 | public | rides | rides_auto_index_fk_vehicle_city_ref_vehicles | NULL | CREATE INDEX rides_auto_index_fk_vehicle_city_ref_vehicles ON movr.public.rides USING btree (vehicle_city ASC, vehicle_id ASC) - 2315049508 | public | vehicle_location_histories | primary | NULL | CREATE UNIQUE INDEX "primary" ON movr.public.vehicle_location_histories USING btree (city ASC, ride_id ASC, "timestamp" ASC) - 969972501 | public | promo_codes | primary | NULL | CREATE UNIQUE INDEX "primary" ON movr.public.promo_codes USING btree (code ASC) - 710236230 | public | user_promo_codes | primary | NULL | CREATE UNIQUE INDEX "primary" ON movr.public.user_promo_codes USING btree (city ASC, user_id ASC, code ASC) -(9 rows) -~~~ - -## See also - -- [`SHOW`](show-vars.html) -- [`SHOW DATABASES`](show-databases.html) -- [`SHOW SCHEMAS`](show-schemas.html) -- [`SHOW TABLES`](show-tables.html) -- [SQL Name Resolution](sql-name-resolution.html) -- [System Catalogs](system-catalogs.html) diff --git a/src/current/v21.1/pg-extension.md b/src/current/v21.1/pg-extension.md deleted file mode 100644 index 79c6e933be7..00000000000 --- a/src/current/v21.1/pg-extension.md +++ /dev/null @@ -1,78 +0,0 @@ ---- -title: pg_extension -summary: The pg_extension schema contains information about CockroachDB extensions. -toc: true ---- - -The `pg_extension` [system catalogs](system-catalogs.html) provides information about CockroachDB extensions. - -## Data exposed by `pg_extension` - -In CockroachDB {{ page.version.version }}, `pg_extension` contains the following tables, all of which provide information about CockroachDB's [spatial extension](spatial-features.html): - -- `geography_columns` -- `geometry_columns` -- `spatial_ref_sys` - -{{site.data.alerts.callout_info}} -`pg_extension` tables are read-only. -{{site.data.alerts.end}} - -To see the list of tables in `pg_extension` for the [current database](sql-name-resolution.html#current-database), use the following [`SHOW TABLES`](show-tables.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM pg_extension; -~~~ - -~~~ - schema_name | table_name | type | owner | estimated_row_count ----------------+-------------------+-------+-------+---------------------- - pg_extension | geography_columns | table | NULL | NULL - pg_extension | geometry_columns | table | NULL | NULL - pg_extension | spatial_ref_sys | table | NULL | NULL -(3 rows) -~~~ - -## Querying `pg_extension` tables - -You can run [`SELECT` queries](selection-queries.html) on the tables in `pg_extension`. - -{{site.data.alerts.callout_success}} -To ensure that you can view all of the tables in `pg_extension`, query the tables as a user with [`admin` privileges](authorization.html#admin-role). -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -Unless specified otherwise, queries to `pg_extension` assume the [current database](sql-name-resolution.html#current-database). -{{site.data.alerts.end}} - -For example, to return the `pg_extension` table with additional information about indexes in the `movr` database, you can query the `pg_extension.pg_indexes` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM movr.pg_extension.pg_indexes; -~~~ - -~~~ - crdb_oid | schemaname | tablename | indexname | tablespace | indexdef --------------+------------+----------------------------+-----------------------------------------------+------------+--------------------------------------------------------------------------------------------------------------------------------- - 2055313241 | public | users | primary | NULL | CREATE UNIQUE INDEX "primary" ON movr.public.users USING btree (city ASC, id ASC) - 1795576970 | public | vehicles | primary | NULL | CREATE UNIQUE INDEX "primary" ON movr.public.vehicles USING btree (city ASC, id ASC) - 1795576969 | public | vehicles | vehicles_auto_index_fk_city_ref_users | NULL | CREATE INDEX vehicles_auto_index_fk_city_ref_users ON movr.public.vehicles USING btree (city ASC, owner_id ASC) - 450499963 | public | rides | primary | NULL | CREATE UNIQUE INDEX "primary" ON movr.public.rides USING btree (city ASC, id ASC) - 450499960 | public | rides | rides_auto_index_fk_city_ref_users | NULL | CREATE INDEX rides_auto_index_fk_city_ref_users ON movr.public.rides USING btree (city ASC, rider_id ASC) - 450499961 | public | rides | rides_auto_index_fk_vehicle_city_ref_vehicles | NULL | CREATE INDEX rides_auto_index_fk_vehicle_city_ref_vehicles ON movr.public.rides USING btree (vehicle_city ASC, vehicle_id ASC) - 2315049508 | public | vehicle_location_histories | primary | NULL | CREATE UNIQUE INDEX "primary" ON movr.public.vehicle_location_histories USING btree (city ASC, ride_id ASC, "timestamp" ASC) - 969972501 | public | promo_codes | primary | NULL | CREATE UNIQUE INDEX "primary" ON movr.public.promo_codes USING btree (code ASC) - 710236230 | public | user_promo_codes | primary | NULL | CREATE UNIQUE INDEX "primary" ON movr.public.user_promo_codes USING btree (city ASC, user_id ASC, code ASC) -(9 rows) -~~~ - -## See also - -- [`SHOW`](show-vars.html) -- [`SHOW DATABASES`](show-databases.html) -- [`SHOW SCHEMAS`](show-schemas.html) -- [`SHOW TABLES`](show-tables.html) -- [SQL Name Resolution](sql-name-resolution.html) -- [System Catalogs](system-catalogs.html) diff --git a/src/current/v21.1/point.md b/src/current/v21.1/point.md deleted file mode 100644 index af3f98ef9b6..00000000000 --- a/src/current/v21.1/point.md +++ /dev/null @@ -1,57 +0,0 @@ ---- -title: POINT -summary: A POINT is a geometry object; a sizeless location identified by its X and Y coordinates. -toc: true ---- - -A `POINT` is a sizeless location identified by its X and Y coordinates. These coordinates are then translated according to the current [spatial reference system](spatial-glossary.html#spatial-reference-system) (denoted by an [SRID](spatial-glossary.html#srid)) to determine what the Point "is", or what it "means" relative to the [other spatial objects](spatial-features.html#spatial-objects) (if any) in the data set. - -{% include {{page.version.version}}/spatial/zmcoords.md %} - -## Examples - -### SQL - -A Point can be created in SQL by the `st_point` function. - -The statement below creates a Point (using the common [SRID 4326](spatial-glossary.html#srid)). - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT ST_SetSRID(ST_Makepoint(0,0), 4326); -~~~ - -~~~ - st_setsrid ------------------------------------------------------- - 0101000020E610000000000000000000000000000000000000 -(1 row) -~~~ - -### Well known text - -A Point can be created from SQL by calling the `st_geomfromtext` function on a Point definition expressed in the [Well Known Text (WKT)](spatial-glossary.html#wkt) format as shown below. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT ST_GeomFromText('POINT(0 0)'); -~~~ - -~~~ - st_geomfromtext ----------------------------------------------- - 010100000000000000000000000000000000000000 -(1 row) -~~~ - -## See also - -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial objects](spatial-features.html#spatial-objects) -- [LINESTRING](linestring.html) -- [POLYGON](polygon.html) -- [MULTIPOINT](multipoint.html) -- [MULTILINESTRING](multilinestring.html) -- [MULTIPOLYGON](multipolygon.html) -- [GEOMETRYCOLLECTION](geometrycollection.html) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/polygon.md b/src/current/v21.1/polygon.md deleted file mode 100644 index 876a8b2bd1b..00000000000 --- a/src/current/v21.1/polygon.md +++ /dev/null @@ -1,72 +0,0 @@ ---- -title: POLYGON -summary: A POLYGON is a shape with a closed exterior that is made up of lines. -toc: true ---- - -A `POLYGON` is a shape with a closed exterior that is made up of lines. Polygons can also contain holes. Polygons are often used to represent areas such as countries, states, or municipalities. - -The coordinates of each Point and line that make up the Polygon are translated according to the current [spatial reference system](spatial-glossary.html#spatial-reference-system) (denoted by an [SRID](spatial-glossary.html#srid)) to determine what the point "is", or what it "means" relative to the [other spatial objects](spatial-features.html#spatial-objects) (if any) in the data set. - -{% include {{page.version.version}}/spatial/zmcoords.md %} - -## Examples - -### Well known text - -A Polygon can be created from SQL by calling the `st_geomfromtext` function on a LineString definition expressed in the [Well Known Text (WKT)](spatial-glossary.html#wkt) format as shown below. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT ST_GeomFromText('POLYGON((0 0, 0 1024, 1024 1024, 1024 0, 0 0))'); -~~~ - -~~~ - st_geomfromtext ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - 010300000001000000050000000000000000000000000000000000000000000000000000000000000000009040000000000000904000000000000090400000000000009040000000000000000000000000000000000000000000000000 -(1 row) -~~~ - -### Polygons with holes - -To represent a polygon with holes in [WKT](spatial-glossary.html#wkt), add one or more additional lists of coordinates that define the boundaries of the holes as shown below: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT ST_GeomFromText('POLYGON((-87.906471 43.038902, -95.992775 36.153980, -75.704722 36.076944, -87.906471 43.038902), (-87.623177 41.881832, -90.199402 38.627003, -82.446732 38.413651, -87.623177 41.881832))'); -~~~ - -~~~ - st_geomfromtext ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - 010300000002000000040000006FF1F09E03FA55C0DFDFA0BDFA844540545227A089FF57C0791EDC9DB513424064B14D2A1AED52C0CCCF0D4DD90942406FF1F09E03FA55C0DFDFA0BDFA84454004000000A4A7C821E2E755C07C48F8DEDFF0444073309B00C38C56C038BF61A241504340E884D041979C54C0967A1684F2344340A4A7C821E2E755C07C48F8DEDFF04440 -(1 row) -~~~ - -### Using a SQL function - -You can also use the `st_makepolygon` function on a LineString that defines the outer boundary of the Polygon, e.g.: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT ST_MakePolygon('LINESTRING(0 0, 0 1024, 1024 1024, 1024 0, 0 0)'); -~~~ - -~~~ ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - 010300000001000000050000000000000000000000000000000000000000000000000000000000000000009040000000000000904000000000000090400000000000009040000000000000000000000000000000000000000000000000 -(1 row) -~~~ - -## See also - -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial objects](spatial-features.html#spatial-objects) -- [POINT](point.html) -- [LINESTRING](linestring.html) -- [MULTIPOINT](multipoint.html) -- [MULTILINESTRING](multilinestring.html) -- [MULTIPOLYGON](multipolygon.html) -- [GEOMETRYCOLLECTION](geometrycollection.html) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/postgresql-compatibility.md b/src/current/v21.1/postgresql-compatibility.md deleted file mode 100644 index dd815ee4e42..00000000000 --- a/src/current/v21.1/postgresql-compatibility.md +++ /dev/null @@ -1,203 +0,0 @@ ---- -title: PostgreSQL Compatibility -summary: A summary of CockroachDB's compatibility with PostgreSQL -toc: true ---- - -CockroachDB supports the [PostgreSQL wire protocol](https://www.postgresql.org/docs/current/protocol.html) and the majority of PostgreSQL syntax. This means that existing applications built on PostgreSQL can often be migrated to CockroachDB without changing application code. - -CockroachDB is compatible with version 3.0 of the Postgres wire protocol (pgwire) and works with the majority of PostgreSQL database tools such as [DBeaver](dbeaver.html), [Intellij](intellij-idea.html), and so on. Consult this link for a full list of supported [third-party database tools](third-party-database-tools.html). CockroachDB also works with most [PostgreSQL drivers and ORMs](example-apps.html). - -However, CockroachDB does not support some of the PostgreSQL features or behaves differently from PostgreSQL because not all features can be easily implemented in a distributed system. This page documents the known list of differences between PostgreSQL and CockroachDB for identical input. That is, a SQL statement of the type listed here will behave differently than in PostgreSQL. Porting an existing application to CockroachDB will require changing these expressions. - -{{site.data.alerts.callout_info}} -This document currently only covers unsupported SQL and how to rewrite SQL expressions. It does not discuss strategies for porting applications that use SQL features CockroachDB does not currently support. -{{site.data.alerts.end}} - -## Unsupported Features - -### Unsupported PostgreSQL features - -The following PostgreSQL features are not supported in CockroachDB {{ page.version.version }}: - -{% include {{page.version.version}}/sql/unsupported-postgres-features.md %} - -### Unsupported PostgreSQL wire protocol features - -The following features of the PostgreSQL wire protocol are not supported in CockroachDB {{ page.version.version }}: - -- [Multiple active portals](https://github.com/cockroachdb/cockroach/issues/40195) -- [Query cancellation](https://github.com/cockroachdb/cockroach/issues/41335) - -## Features that differ from PostgreSQL - -Note, some of the differences below only apply to rare inputs, and so no change will be needed, even if the listed feature is being used. In these cases, it is safe to ignore the porting instructions. - -### Overflow of `float` - -In PostgreSQL, the `float` type returns an error when it overflows or an expression would return Infinity: - -~~~ -postgres=# select 1e300::float * 1e10::float; -ERROR: value out of range: overflow -postgres=# select pow(0::float, -1::float); -ERROR: zero raised to a negative power is undefined -~~~ - -In CockroachDB, these expressions instead return Infinity: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT 1e300::float * 1e10::float; -~~~ - -~~~ - ?column? ------------- - +Inf -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT pow(0::float, -1::float); -~~~ - -~~~ - pow --------- - +Inf -(1 row) -~~~ - -### Precedence of unary `~` - -In PostgreSQL, the unary `~` (bitwise not) operator has a low precedence. For example, the following query is parsed as `~ (1 + 2)` because `~` has a lower precedence than `+`: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT ~1 + 2; -~~~ - -~~~ - ?column? ------------- - 0 -(1 row) -~~~ - -In CockroachDB, unary `~` has the same (high) precedence as unary `-`, so the above expression will be parsed as `(~1) + 2`. - -**Porting instructions:** Manually add parentheses around expressions that depend on the PostgreSQL behavior. - -### Precedence of bitwise operators - -In PostgreSQL, the operators `|` (bitwise OR), `#` (bitwise XOR), and `&` (bitwise AND) all have the same precedence. - -In CockroachDB, the precedence from highest to lowest is: `&`, `#`, `|`. - -**Porting instructions:** Manually add parentheses around expressions that depend on the PostgreSQL behavior. - -### Integer division - -In PostgreSQL, division of integers results in an integer. For example, the following query returns `1`, since the `1 / 2` is truncated to `0`: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT 1 + 1 / 2; -~~~ - -~~~ - ?column? ------------- - 1.5 -(1 row) -~~~ - -In CockroachDB, integer division results in a `decimal`. CockroachDB instead provides the `//` operator to perform floor division. - -**Porting instructions:** Change `/` to `//` in integer division where the result must be an integer. - -### Shift argument modulo - -In PostgreSQL, the shift operators (`<<`, `>>`) sometimes modulo their second argument to the bit size of the underlying type. For example, the following query results in a `1` because the int type is 32 bits, and `32 % 32` is `0`, so this is the equivalent of `1 << 0`: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT 1::int << 32; -~~~ - -~~~ - ?column? --------------- - 4294967296 -(1 row) -~~~ - -In CockroachDB, no such modulo is performed. - -**Porting instructions:** Manually add a modulo to the second argument. Also note that CockroachDB's [`INT`](int.html) type is always 64 bits. For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT 1::int << (x % 64); -~~~ - -### Locking and `FOR UPDATE` - -CockroachDB supports the `SELECT FOR UPDATE` statement, which is used to order transactions by controlling concurrent access to one or more rows of a table. - -For more information, see [`SELECT FOR UPDATE`](select-for-update.html). - -### `CHECK` constraint validation for `INSERT ON CONFLICT` - -CockroachDB validates [`CHECK`](check.html) constraints on the results of [`INSERT ON CONFLICT`](insert.html#on-conflict-clause) statements, preventing new or changed rows from violating the constraint. Unlike PostgreSQL, CockroachDB does not also validate `CHECK` constraints on the input rows of `INSERT ON CONFLICT` statements. - -If this difference matters to your client, you can `INSERT ON CONFLICT` from a `SELECT` statement and check the inserted value as part of the `SELECT`. For example, instead of defining `CHECK (x > 0)` on `t.x` and using `INSERT INTO t(x) VALUES (3) ON CONFLICT (x) DO UPDATE SET x = excluded.x`, you could do the following: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO t (x) - SELECT if (x <= 0, crdb_internal.force_error('23514', 'check constraint violated'), x) - FROM (values (3)) AS v(x) - ON CONFLICT (x) - DO UPDATE SET x = excluded.x; -~~~ - -An `x` value less than `1` would result in the following error: - -~~~ -pq: check constraint violated -~~~ - -[Tracking GitHub Issue](https://github.com/cockroachdb/cockroach/issues/35370) - -### Column name from an outer column inside a subquery - -CockroachDB returns the column name from an outer column inside a subquery as `?column?`, unlike PostgreSQL. For example: - -~~~ sql -> SELECT (SELECT t.*) FROM (VALUES (1)) t(x); -~~~ - -CockroachDB: - -~~~ - ?column? ------------- - 1 -~~~ - -PostgreSQL: - -~~~ - x ---- - 1 -~~~ - -[Tracking GitHub Issue](https://github.com/cockroachdb/cockroach/issues/46563) - -### SQL Compatibility - -Click the following link to find a full list of [CockroachDB supported SQL Features](sql-feature-support.html). diff --git a/src/current/v21.1/primary-key.md b/src/current/v21.1/primary-key.md deleted file mode 100644 index 926ccf53ee5..00000000000 --- a/src/current/v21.1/primary-key.md +++ /dev/null @@ -1,136 +0,0 @@ ---- -title: Primary Key Constraint -summary: A primary key constraint specifies columns that can be used to uniquely identify rows in a table. -toc: true ---- - -The `PRIMARY KEY` [constraint](constraints.html) specifies that the constrained columns' values must uniquely identify each row. A table can only have one primary key, but it can have multiple [unique constraints](unique.html). - -You should explicitly define a table's primary key in the [`CREATE TABLE`](create-table.html) statement. If you don't define a primary key at table creation time, CockroachDB will create a `rowid` column that is `NOT VISIBLE`, use the [`unique_rowid()` function](functions-and-operators.html#id-generation-functions) as its default value, and use that as the table's primary key. - -You can [change the primary key](#changing-primary-key-columns) of an existing table with an [`ALTER TABLE ... ALTER PRIMARY KEY`](alter-primary-key.html) statement, or by using [`DROP CONSTRAINT`](drop-constraint.html) and then [`ADD CONSTRAINT`](add-constraint.html) in the same transaction. You cannot fully drop the `PRIMARY KEY` constraint on a table without replacing it as it provides an intrinsic structure to the table's data. - -## Syntax - -`PRIMARY KEY` constraints can be defined at the [table level](#table-level). However, if you only want the constraint to apply to a single column, it can be applied at the [column level](#column-level). - -### Column level - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/primary_key_column_level.html %} -
    - - Parameter | Description ------------|------------- - `table_name` | The name of the table you're creating. - `column_name` | The name of the Primary Key column. For [multi-region tables](multiregion-overview.html#table-locality), you can use the `crdb_region` column within a composite primary key in the event the original primary key may contain non-unique entries across multiple, unique regions. - `column_type` | The Primary Key column's [data type](data-types.html). - `column_constraints` | Any other column-level [constraints](constraints.html) you want to apply to this column. - `column_def` | Definitions for any other columns in the table. - `table_constraints` | Any table-level [constraints](constraints.html) you want to apply. - -**Example** - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE orders ( - order_id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - order_date TIMESTAMP NOT NULL, - order_mode STRING(8), - customer_id INT, - order_status INT - ); -~~~ - -### Table level - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/primary_key_table_level.html %} -
    - - Parameter | Description ------------|------------- - `table_name` | The name of the table you're creating. - `column_def` | Definitions for any other columns in the table. - `name` | The name you want to use for the constraint, which must be unique to its table and follow these [identifier rules](keywords-and-identifiers.html#identifiers). - `column_name` | The name of the column you want to use as the `PRIMARY KEY`.

    The order in which you list columns here affects the structure of the `primary` index. - `table_constraints` | Any other table-level [constraints](constraints.html) you want to apply. - -**Example** - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE IF NOT EXISTS inventories ( - product_id INT, - warehouse_id INT, - quantity_on_hand INT NOT NULL, - PRIMARY KEY (product_id, warehouse_id) - ); -~~~ - -## Details - -The columns in the `PRIMARY KEY` constraint are used to create its `primary` [index](indexes.html), which CockroachDB uses by default to access the table's data. This index does not take up additional disk space (unlike secondary indexes, which do) because CockroachDB uses the `primary` index to structure the table's data in the key-value layer. For more information, see our blog post [SQL in CockroachDB: Mapping Table Data to Key-Value Storage](https://www.cockroachlabs.com/blog/sql-in-cockroachdb-mapping-table-data-to-key-value-storage/). - -To ensure each row has a unique identifier, the `PRIMARY KEY` constraint combines the properties of both the [`UNIQUE`](unique.html) and [`NOT NULL`](not-null.html) constraints. The properties of both constraints are necessary to make sure each row's primary key columns contain distinct sets of values. The properties of the `UNIQUE` constraint ensure that each value is distinct from all other values. However, because *NULL* values never equal other *NULL* values, the `UNIQUE` constraint is not enough (two rows can appear the same if one of the values is *NULL*). To prevent the appearance of duplicated values, the `PRIMARY KEY` constraint also enforces the properties of the `NOT NULL` constraint. - -For best practices, see [Schema Design: Select primary key columns](schema-design-table.html#select-primary-key-columns). - -## Example - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE IF NOT EXISTS inventories ( - product_id INT, - warehouse_id INT, - quantity_on_hand INT NOT NULL, - PRIMARY KEY (product_id, warehouse_id) - ); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO inventories VALUES (1, 1, 100); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO inventories VALUES (1, 1, 200); -~~~ - -~~~ -pq: duplicate key value (product_id,warehouse_id)=(1,1) violates unique constraint "primary" -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO inventories VALUES (1, NULL, 100); -~~~ - -~~~ -pq: null value in column "warehouse_id" violates not-null constraint -~~~ - -## Changing primary key columns - -You can change the primary key of an existing table by doing one of the following: - -- Issuing an [`ALTER TABLE ... ALTER PRIMARY KEY`](alter-primary-key.html) statement. When you change a primary key with `ALTER PRIMARY KEY`, the old primary key index becomes a secondary index. This helps optimize the performance of queries that still filter on the old primary key column. -- Issuing an [`ALTER TABLE ... DROP CONSTRAINT ... PRIMARY KEY`](drop-constraint.html) statement to drop the primary key, followed by an [`ALTER TABLE ... ADD CONSTRAINT ... PRIMARY KEY`](add-constraint.html) statement, in the same transaction, to add a new primary key. This replaces the existing primary key without creating a secondary index from the old primary key. For examples, see the [`ADD CONSTRAINT`](add-constraint.html#examples) and [`DROP CONSTRAINT`](drop-constraint.html#examples) pages. - -{{site.data.alerts.callout_info}} -You can use an [`ADD CONSTRAINT ... PRIMARY KEY`](add-constraint.html) statement without a [`DROP CONSTRAINT ... PRIMARY KEY`](drop-constraint.html) if the primary key was not explicitly defined at [table creation](create-table.html), and the current [primary key is on `rowid`](indexes.html#creation). -{{site.data.alerts.end}} - -## See also - -- [Constraints](constraints.html) -- [`CHECK` constraint](check.html) -- [`DEFAULT` constraint](default-value.html) -- [`REFERENCES` constraint (Foreign Key)](foreign-key.html) -- [`PRIMARY KEY` constraint](primary-key.html) -- [`NOT NULL` constraint](not-null.html) -- [`UNIQUE` constraint](unique.html) -- [`SHOW CONSTRAINTS`](show-constraints.html) -- [`ALTER PRIMARY KEY`](alter-primary-key.html) -- [`ALTER TABLE`](alter-table.html) diff --git a/src/current/v21.1/query-behavior-troubleshooting.md b/src/current/v21.1/query-behavior-troubleshooting.md deleted file mode 100644 index 26d1702f575..00000000000 --- a/src/current/v21.1/query-behavior-troubleshooting.md +++ /dev/null @@ -1,189 +0,0 @@ ---- -title: Troubleshoot SQL Behavior -summary: Learn how to troubleshoot issues with specific SQL statements with CockroachDB -toc: true ---- - -If a [SQL statement](sql-statements.html) returns an unexpected result or takes longer than expected to process, this page will help you troubleshoot the issue. - -{{site.data.alerts.callout_success}} -For a developer-centric tutorial of optimizing SQL query performance, see [Optimize Statement Performance](make-queries-fast.html). -{{site.data.alerts.end}} - -## Identify slow statements - -Use the [slow query log](logging-use-cases.html#sql_perf) or DB Console to detect slow queries in your cluster. - -High latency SQL statements are displayed on the [**Statements page**](ui-statements-page.html) of the DB Console. To view the Statements page, [access the DB Console](ui-overview.html#db-console-access) and click **Statements** on the left. - -You can also check the [service latency graph](ui-sql-dashboard.html#service-latency-sql-99th-percentile) and the [CPU graph](ui-hardware-dashboard.html#cpu-percent) on the SQL and Hardware Dashboards, respectively. If the graphs show latency spikes or CPU usage spikes, these might indicate slow queries in your cluster. - -## Visualize statement traces in Jaeger - -You can look more closely at the behavior of a statement by visualizing a statement trace in [Jaeger](https://www.jaegertracing.io/). A [statement trace](show-trace.html) contains messages and timing information from all nodes involved in the execution. - -1. Activate [statement diagnostics](ui-statements-page.html#diagnostics) on the DB Console Statements Page or run [`EXPLAIN ANALYZE (DEBUG)`](explain-analyze.html#debug-option) to obtain a diagnostics bundle for the statement. - -1. Start Jaeger: - - {% include_cached copy-clipboard.html %} - ~~~ shell - docker run -d --name jaeger -p 16686:16686 jaegertracing/all-in-one:1.17 - ~~~ - -1. Access the Jaeger UI at `http://localhost:16686/search`. - -1. Click on **JSON File** in the Jaeger UI and upload `trace-jaeger.json` from the diagnostics bundle. The trace will appear in the list on the right. - - Jaeger Trace Upload JSON - -1. Click on the trace to view its details. It is visualized as a collection of spans with timestamps. These may include operations executed by different nodes. - - Jaeger Trace Spans - - The full timeline displays the execution time and [execution phases](architecture/sql-layer.html#sql-parser-planner-executor) for the statement. - -1. Click on a span to view details for that span and log messages. - - Jaeger Trace Log Messages - -1. You can troubleshoot [transaction contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention), for example, by gathering [diagnostics](ui-statements-page.html#diagnostics) on statements with high latency and looking through the log messages in `trace-jaeger.json` for jumps in latency. - - In the example below, the trace shows that there is significant latency between a push attempt on a transaction that is holding a [lock](architecture/transaction-layer.html#writing) (56.85ms) and that transaction being committed (131.37ms). - - Jaeger Trace Log Messages - -## `SELECT` statement performance issues - -The common reasons for a sub-optimal `SELECT` performance are inefficient scans, full scans, and incorrect use of indexes. To improve the performance of `SELECT` statements, refer to the following documents: - -- [Table scan best practices](performance-best-practices-overview.html#table-scans-best-practices) - -- [Indexes best practices](schema-design-indexes.html#best-practices) - -## Query is always slow - -If you have consistently slow queries in your cluster, use the [Statement Details](ui-statements-page.html#statement-details-page) page to drill down to an individual statement and [collect diagnostics](ui-statements-page.html#diagnostics) for the statement. A diagnostics bundle contains a record of transaction events across nodes for the SQL statement. - -You can also use an [`EXPLAIN ANALYZE`](explain-analyze.html) statement, which executes a SQL query and returns a physical query plan with execution statistics. Query plans can be used to troubleshoot slow queries by indicating where time is being spent, how long a processor (i.e., a component that takes streams of input rows and processes them according to a specification) is not doing work, etc. - -We recommend sending either the diagnostics bundle or the `EXPLAIN ANALYZE` output to our [support team](support-resources.html) for analysis. - -## Query is sometimes slow - -If the query performance is irregular: - -1. Run [`SHOW TRACE`](show-trace.html) for the query twice: once when the query is performing as expected and once when the query is slow. - -2. [Contact us](support-resources.html) to analyze the outputs of the `SHOW TRACE` command. - -## Cancelling running queries - -See [Cancel query](manage-long-running-queries.html#cancel-long-running-queries) - -## Low throughput - -Throughput is affected by the disk I/O, CPU usage, and network latency. Use the DB Console to check the following metrics: - -- Disk I/O: [Disk IOPS in progress](ui-hardware-dashboard.html#disk-iops-in-progress) - -- CPU usage: [CPU percent](ui-hardware-dashboard.html#cpu-percent) - -- Network latency: [Network Latency page](ui-network-latency-page.html) - -## Single hot node - -A hot node is one that has much higher resource usage than other nodes. To determine if you have a hot node in your cluster, [access the DB Console](ui-overview.html#db-console-access), click **Metrics** on the left, and navigate to the following graphs. Hover over each of the following graphs to see the per-node values of the metrics. If one of the nodes has a higher value, you have a hot node in your cluster. - -- Replication dashboard > Average queries per store graph. - -- Overview Dashboard > Service Latency graph - -- Hardware Dashboard > CPU percent graph - -- SQL Dashboard > SQL Connections graph - -- Hardware Dashboard > Disk IOPS in Progress graph - -**Solution:** - -- If you have a small table that fits into one range, then only one of the nodes will be used. This is expected behavior. However, you can [split your range](split-at.html) to distribute the table across multiple nodes. - -- If the **SQL Connections** graph shows that one node has a higher number of SQL connections and other nodes have zero connections, check if your app is set to talk to only one node. - -- Check load balancer settings. - -- Check for [transaction contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention). - -- If you have a monotonically increasing index column or Primary Key, then your index or Primary Key should be redesigned. See [Unique ID best practices](performance-best-practices-overview.html#unique-id-best-practices) for more information. - -## INSERT/UPDATE statements are slow - -Use the [Statements page](ui-statements-page.html) to identify the slow [SQL statements](sql-statements.html). To view the Statements page, [access the DB Console](ui-overview.html#db-console-access) and then click Statements on the left. - -Refer to the following documents to improve `INSERT` / `UPDATE` performance: - -- [Multi-row DML](performance-best-practices-overview.html#dml-best-practices) - -- [Bulk-Insert best practices](performance-best-practices-overview.html#bulk-insert-best-practices) - -## Per-node queries per second (QPS) is high - -If a cluster is not idle, it is useful to monitor the per-node queries per second. Cockroach will automatically distribute load throughout the cluster. If one or more nodes is not performing any queries there is likely something to investigate. See `exec_success` and `exec_errors` which track operations at the KV layer and `sql_{select,insert,update,delete}_count` which track operations at the SQL layer. - -## Increasing number of nodes does not improve performance - -See [Why would increasing the number of nodes not result in more operations per second?](operational-faqs.html#why-would-increasing-the-number-of-nodes-not-result-in-more-operations-per-second) - -## `bad connection` & `closed` responses - -If you receive a response of `bad connection` or `closed`, this normally indicates that the node you connected to died. You can check this by connecting to another node in the cluster and running [`cockroach node status`](cockroach-node.html#show-the-status-of-all-nodes). - -Once you find the downed node, you can check its [logs](logging.html) (stored in `cockroach-data/logs` by [default](configure-logs.html#default-logging-configuration)). - -Because this kind of behavior is entirely unexpected, you should [file an issue](file-an-issue.html). - -## Local query testing - -If you are testing CockroachDB locally and want to log queries executed just by a specific node, you can either pass a CLI flag at node startup or execute a SQL function on a running node. - -Using the CLI to start a new node, use the `--vmodule` flag with the [`cockroach start`](cockroach-start.html) command. For example, to start a single node locally and log all client-generated SQL queries it executes, you'd run: - -~~~ shell -$ cockroach start --insecure --listen-addr=localhost --vmodule=exec_log=2 --join= -~~~ - -{{site.data.alerts.callout_success}} -To log CockroachDB-generated SQL queries as well, use `--vmodule=exec_log=3`. -{{site.data.alerts.end}} - -From the SQL prompt on a running node, execute the `crdb_internal.set_vmodule()` [function](functions-and-operators.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT crdb_internal.set_vmodule('exec_log=2'); -~~~ - -This will result in the following output: - -~~~ - crdb_internal.set_vmodule -+---------------------------+ - 0 -(1 row) -~~~ - -Once the logging is enabled, all client-generated SQL queries executed by the node will be written to the `DEV` [logging channel](logging.html#dev), which outputs by [default](configure-logs.html#default-logging-configuration) to the primary `cockroach` log file in `/cockroach-data/logs`. Use the symlink `cockroach.log` to open the most recent log. - -~~~ -I180402 19:12:28.112957 394661 sql/exec_log.go:173 [n1,client=127.0.0.1:50155,user=root] exec "psql" {} "SELECT version()" {} 0.795 1 "" -~~~ - -## Something else? - -Try searching the rest of our docs for answers or using our other [support resources](support-resources.html), including: - -- [CockroachDB Community Forum](https://forum.cockroachlabs.com) -- [CockroachDB Community Slack](https://cockroachdb.slack.com) -- [StackOverflow](http://stackoverflow.com/questions/tagged/cockroachdb) -- [CockroachDB Support Portal](https://support.cockroachlabs.com) diff --git a/src/current/v21.1/query-data.md b/src/current/v21.1/query-data.md deleted file mode 100644 index 712c7d687f6..00000000000 --- a/src/current/v21.1/query-data.md +++ /dev/null @@ -1,210 +0,0 @@ ---- -title: Query Data -summary: How to send SQL queries to CockroachDB from various programming languages -toc: true ---- - -This page has instructions for making SQL [selection queries][selection] against CockroachDB from various programming languages. - -## Before you begin - -Before reading this page, do the following: - -- [Install CockroachDB](install-cockroachdb.html). -- [Start a local cluster](secure-a-cluster.html), or [create a CockroachDB {{ site.data.products.cloud }} cluster](../cockroachcloud/create-your-cluster.html). -- [Install a Postgres client](install-client-drivers.html). -- [Connect to the database](connect-to-the-database.html). -- [Insert data](insert-data.html) that you now want to run queries against. - -{% include {{page.version.version}}/app/retry-errors.md %} - -## Simple selects - -
    - - - - -
    - -
    - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT id, balance from accounts; -~~~ - -For more information about how to use the built-in SQL client, see the [`cockroach sql`](cockroach-sql.html) reference docs. - -
    - -
    - -{% include_cached copy-clipboard.html %} -~~~ go -// 'db' is an open database connection - -rows, err := db.Query("SELECT id, balance FROM accounts") -if err != nil { - log.Fatal(err) -} -defer rows.Close() -fmt.Println("Initial balances:") -for rows.Next() { - var id, balance int - if err := rows.Scan(&id, &balance); err != nil { - log.Fatal(err) - } - fmt.Printf("%d %d\n", id, balance) -} -~~~ - -{% include {{page.version.version}}/app/for-a-complete-example-go.md %} - -
    - -
    - -{% include_cached copy-clipboard.html %} -~~~ java -// ds is an org.postgresql.ds.PGSimpleDataSource - -try (Connection connection = ds.getConnection()) { - Statement stmt = connection.createStatement(); - ResultSet rs = stmt.executeQuery("SELECT id, balance FROM accounts"); - - while (rs.next()) { - int id = rs.getInt(1); - int bal = rs.getInt(2); - System.out.printf("ID: %10s\nBalance: %5s\n", id, bal); - } - rs.close(); - -} catch (SQLException e) { - System.out.printf("sql state = [%s]\ncause = [%s]\nmessage = [%s]\n", - e.getSQLState(), e.getCause(), e.getMessage()); -} -~~~ - -{% include {{page.version.version}}/app/for-a-complete-example-java.md %} - -
    - -
    - -{% include_cached copy-clipboard.html %} -~~~ python -# conn is a psycopg2 connection - -with conn.cursor() as cur: - cur.execute("SELECT id, balance FROM accounts") - rows = cur.fetchall() - for row in rows: - print([str(cell) for cell in row]) -~~~ - -{% include {{page.version.version}}/app/for-a-complete-example-python.md %} - -
    - -## Order results - -To order the results of a query, use an `ORDER BY` clause. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT * FROM bank ORDER BY balance; -~~~ - -~~~ - id | balance | payload ------+---------+------------------------------------------------------------------------------------------------------- - 0 | -500 | initial-dTqnRurXztAPkykhZWvsCmeJkMwRNcJAvTlNbgUEYfagEQJaHmfPsquKZUBOGwpAjPtATpGXFJkrtQCEJODSlmQctvyh - 1 | -499 | initial-PCLGABqTvrtRNyhAyOhQdyLfVtCmRykQJSsdwqUFABkPOMQayVEhiAwzZKHpJUiNmVaWYZnReMKfONZvRKbTETaIDccE - 2 | -498 | initial-VNfyUJHfCmMeAUoTgoSVvnByDyvpHNPHDfVoNWdXBFQpwMOBgNVtNijyTjmecvFqyeLHlDbIBRrbCzSeiHWSLmWbhIvh - 3 | -497 | initial-llflzsVuQYUlfwlyoaqjdwKUNgNFVgvlnINeOUUVyfxyvmOiAelxqkTBfpBBziYVHgQLLEuCazSXmURnXBlCCfsOqeji - 4 | -496 | initial-rmGzVVucMqbYnBaccWilErbWvcatqBsWSXvrbxYUUEhmOnccXzvqcsGuMVJNBjmzKErJzEzzfCzNTmLQqhkrDUxdgqDD -(5 rows) -~~~ - -For reference documentation and more examples, see the [`ORDER BY`](order-by.html) syntax page. - -## Limit results - -To limit the results of a query, use a `LIMIT` clause. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT * FROM bank LIMIT 5; -~~~ - -~~~ - id | balance | payload ------+---------+------------------------------------------------------------------------------------------------------- - 0 | 0 | initial-dTqnRurXztAPkykhZWvsCmeJkMwRNcJAvTlNbgUEYfagEQJaHmfPsquKZUBOGwpAjPtATpGXFJkrtQCEJODSlmQctvyh - 1 | 0 | initial-PCLGABqTvrtRNyhAyOhQdyLfVtCmRykQJSsdwqUFABkPOMQayVEhiAwzZKHpJUiNmVaWYZnReMKfONZvRKbTETaIDccE - 2 | 0 | initial-VNfyUJHfCmMeAUoTgoSVvnByDyvpHNPHDfVoNWdXBFQpwMOBgNVtNijyTjmecvFqyeLHlDbIBRrbCzSeiHWSLmWbhIvh - 3 | 0 | initial-llflzsVuQYUlfwlyoaqjdwKUNgNFVgvlnINeOUUVyfxyvmOiAelxqkTBfpBBziYVHgQLLEuCazSXmURnXBlCCfsOqeji - 4 | 0 | initial-rmGzVVucMqbYnBaccWilErbWvcatqBsWSXvrbxYUUEhmOnccXzvqcsGuMVJNBjmzKErJzEzzfCzNTmLQqhkrDUxdgqDD -(5 rows) -~~~ - -For reference documentation and more examples, see the [`LIMIT`/`OFFSET`](limit-offset.html) syntax page. - -## Joins - -The syntax for a [selection query][selection] with a two-way [join][joins] is shown below. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT - a.col1, b.col1 -FROM - some_table AS a - JOIN - some_other_table AS b - ON - a.id = b.id -WHERE - a.col2 > 100 AND a.col3 > now() -ORDER BY - a.col2 DESC -LIMIT - 25; -~~~ - -Join performance can be a big factor in your application's performance. For more information about how to make sure your SQL performs well, see [Optimize Statement Performance][fast]. - -## See also - -Reference information related to this task: - -- [Selection queries][selection] -- [`SELECT`](select-clause.html) -- [Joins][joins] -- [Paginate through limited results][paginate] -- [Understanding and Avoiding Transaction Contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention) - -Other common tasks: - -- [Connect to the Database](connect-to-the-database.html) -- [Insert Data](insert-data.html) -- [Update Data](update-data.html) -- [Delete Data](delete-data.html) -- [Run Multi-Statement Transactions](run-multi-statement-transactions.html) -- [Error Handling and Troubleshooting](error-handling-and-troubleshooting.html) -- [Optimize Statement Performance][fast] -- [Hello World Example apps](example-apps.html) - - - -[selection]: selection-queries.html -[manual]: manual-deployment.html -[orchestrated]: orchestration.html -[fast]: make-queries-fast.html -[paginate]: pagination.html -[joins]: joins.html diff --git a/src/current/v21.1/query-replication-reports.md b/src/current/v21.1/query-replication-reports.md deleted file mode 100644 index bc51e14fd10..00000000000 --- a/src/current/v21.1/query-replication-reports.md +++ /dev/null @@ -1,518 +0,0 @@ ---- -title: Replication Reports -summary: Verify that your cluster's data replication, data placement, and zone configurations are working as expected. -keywords: availability zone, zone config, zone configs, zone configuration, constraint, constraints -toc: true ---- - -Several new and updated tables (listed below) are available to help you query the status of your cluster's data replication, data placement, and zone constraint conformance. For example, you can: - -- See what data is under-replicated or unavailable. -- Show which of your localities (if any) are critical. A locality is "critical" for a range if all of the nodes in that locality becoming unreachable would cause the range to become unavailable. In other words, the locality contains a majority of the range's replicas. -- See if any of your cluster's data placement constraints are being violated. - -{{site.data.alerts.callout_info}} -The information on this page assumes you are familiar with [replication zones](configure-replication-zones.html) and [partitioning](partitioning.html). -{{site.data.alerts.end}} - -{{site.data.alerts.callout_danger}} -**This is an experimental feature.** The interface and output are subject to change. - -In particular, the direct access to `system` tables shown here will not be a supported way to inspect CockroachDB in future versions. We're committed to adding stable ways to inspect these replication reports in the future, likely via `SHOW` statements and/or [views](views.html) and [built-in functions](functions-and-operators.html) in the `crdb_internal` schema. -{{site.data.alerts.end}} - -## Conformance reporting tables - -The following new and updated tables are available for verifying constraint conformance. - -- [`system.reports_meta`](#system-reports_meta) contains metadata about the replication report data listed in the `system.replication_*` tables. Currently, the only metadata available is the report generation time. -- [`system.replication_stats`](#system-replication_stats) shows information about whether data is under-replicated, over-replicated, or unavailable. -- [`system.replication_constraint_stats`](#system-replication_constraint_stats) shows the status of any data placement constraints you've configured. -- [`system.replication_critical_localities`](#system-replication_critical_localities) shows which localities in your cluster (if any) are critical. A locality is "critical" for a range if all of the nodes in that locality becoming unreachable would cause the range to become unavailable. In other words, the locality contains a majority of the range's replicas. -- [`crdb_internal.zones`](#crdb_internal-zones) can be used with the tables above to figure out the databases and table names where the non-conforming or at-risk data is located. - -To configure how often the conformance reports are run, adjust the `kv.replication_reports.interval` [cluster setting](cluster-settings.html), which accepts an [`INTERVAL`](interval.html). For example, to run it every five minutes: - -{% include_cached copy-clipboard.html %} -~~~ sql -SET CLUSTER setting kv.replication_reports.interval = '5m'; -~~~ - -Only members of the `admin` role can access these tables. By default, the `root` user belongs to the `admin` role. For more information about users and roles, see [Manage Users](authorization.html#create-and-manage-users) and [Manage Roles](authorization.html#create-and-manage-roles). - - - -{{site.data.alerts.callout_info}} -The replication reports are only generated for zones that meet the following criteria: - -- The zone overrides some replication attributes compared to their parent zone. Ranges in zones for which a report is not generated are counted in the report of the first ancestor zone for which a report is generated. - -The attributes that must be overridden to trigger each report to run are: - -| Report | Field(s) | -|-----------------------------------+--------------------------------| -| `replication_constraint_stats` | `constraints` | -| `replication_critical_localities` | `constraints`, `num_replicas` | -| `replication_stats` | `constraints`, `num_replicas` | - -In addition to the above, the system's `default` zone always gets a report. -{{site.data.alerts.end}} - -### system.replication_stats - -The `system.replication_stats` report contains information about whether data is under-replicated, over-replicated, or unavailable. - -For an example using this table, see [Find out which databases and tables have under-replicated ranges](#find-out-which-databases-and-tables-have-under-replicated-ranges). - - Ranges are considered under-replicated when one of the replicas is unresponsive. This includes the case when the node where the replica is stored is not running. - - This report considers a node to be dead (for the purposes of calculating the `unavailable_ranges` and `under_replicated_ranges` columns) if its [liveness record](cluster-setup-troubleshooting.html#check-node-liveness-record-last-update) is expired, which occurs if the node is unresponsive for more than a few seconds. In versions of CockroachDB prior to 20.1, this report used the value of the [cluster setting](cluster-settings.html) `server.time_until_store_dead`, which defaults to 5 minutes. - -#### Columns - -| Column name | Data type | Description | -|-------------------------+--------------------+---------------------------------------------------------------------------------------------------------------------------------------| -| zone_id | [`INT8`](int.html) | The ID of the [replication zone](configure-zone.html). | -| subzone_id | [`INT8`](int.html) | The ID of the subzone (i.e., [partition](partition-by.html)). | -| report_id | [`INT8`](int.html) | The ID of the [report](#system-reports_meta) that generated all of the rows in this table. | -| total_ranges | [`INT8`](int.html) | Total [ranges](architecture/overview.html#architecture-range) in the zone this report entry is referring to. | -| unavailable_ranges | [`INT8`](int.html) | Unavailable ranges in the zone this report entry is referring to. | -| under_replicated_ranges | [`INT8`](int.html) | [Under-replicated ranges](ui-replication-dashboard.html#under-replicated-ranges) in the zone this report entry is referring to. | -| over_replicated_ranges | [`INT8`](int.html) | Over-replicated ranges in the zone this report entry is referring to. | - -### system.replication_critical_localities - -The `system.replication_critical_localities` report contains which of your localities (if any) are critical. A locality is "critical" for a range if all of the nodes in that locality becoming unreachable would cause the range to become unavailable. In other words, the locality contains a majority of the range's replicas. - -That said, a locality being critical is not necessarily a bad thing as long as you are aware of it. What matters is that [you configure the topology of your cluster to get the resiliency you expect](topology-patterns.html). - -As described in [Configure Replication Zones](configure-replication-zones.html#descriptive-attributes-assigned-to-nodes), localities are key-value pairs defined at [node startup time](cockroach-start.html#locality), and are ordered into _locality tiers_ that range from most inclusive to least inclusive (e.g., region before datacenter as in `region=eu,dc=paris`). - -For an example using this table, see [Find out which databases and tables have ranges in critical localities](#find-out-which-databases-and-tables-have-ranges-in-critical-localities). - - This report considers a node to be dead (for the purposes of calculating the `at_risk_ranges` column) if its [liveness record](cluster-setup-troubleshooting.html#check-node-liveness-record-last-update) is expired, which occurs if the node is unresponsive for more than a few seconds. In versions of CockroachDB prior to 20.1, this report used the value of the [cluster setting](cluster-settings.html) `server.time_until_store_dead`, which defaults to 5 minutes. - -#### Columns - -| Column name | Data type | Description | -|----------------+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------| -| zone_id | [`INT8`](int.html) | The ID of the [replication zone](configure-zone.html). | -| subzone_id | [`INT8`](int.html) | The ID of the subzone (i.e., [partition](partition-by.html)). | -| locality | [`STRING`](string.html) | The name of the critical [locality](configure-replication-zones.html#zone-config-node-locality). | -| report_id | [`INT8`](int.html) | The ID of the [report](#system-reports_meta) that generated all of the rows in this table. | -| at_risk_ranges | [`INT8`](int.html) | The [ranges](architecture/overview.html#architecture-range) that are at risk of becoming unavailable as of the time of this report. | - -{{site.data.alerts.callout_info}} -If you have not [defined any localities](configure-replication-zones.html#zone-config-node-locality), this report will not return any results. It only reports on localities that have been explicitly defined. -{{site.data.alerts.end}} - -### system.replication_constraint_stats - -The `system.replication_constraint_stats` report lists violations to any data placement requirements you've configured. - -For an example using this table, see [Find out which of your tables have a constraint violation](#find-out-which-of-your-tables-have-a-constraint-violation). - -#### Columns - -| Column name | Data type | Description | -|------------------+---------------------------------+---------------------------------------------------------------------------------------------------------| -| zone_id | [`INT8`](int.html) | The ID of the [replication zone](configure-zone.html). | -| subzone_id | [`INT8`](int.html) | The ID of the subzone (i.e., [partition](partition-by.html)). | -| type | [`STRING`](string.html) | The type of zone configuration that was violated, e.g., `constraint`. | -| config | [`STRING`](string.html) | The YAML key-value pair used to configure the zone, e.g., `+region=europe-west1`. | -| report_id | [`INT8`](int.html) | The ID of the [report](#system-reports_meta) that generated all of the rows in this table. | -| violation_start | [`TIMESTAMPTZ`](timestamp.html) | The time when the violation was detected. Will return `NULL` if the number of `violating_ranges` is 0. | -| violating_ranges | [`INT8`](int.html) | The [ranges](architecture/overview.html#architecture-range) that are in violation of the configuration. | - -### system.reports_meta - -The `system.reports_meta` report contains metadata about when the replication reports were last run. Each report contains a number of report entries, one per zone. - -Replication reports are run at the interval specified by the `kv.replication_reports.interval` [cluster setting](cluster-settings.html). - -#### Columns - -| Column name | Data type | Description | -|-------------+---------------------------------+---------------------------------------------------------| -| id | [`INT8`](int.html) | The ID of the report that this report entry is part of. | -| generated | [`TIMESTAMPTZ`](timestamp.html) | When the report was generated. | - -### crdb_internal.zones - -The `crdb_internal.zones` table is useful for: - -- Viewing your cluster's zone configurations in various formats: YAML, SQL, etc. -- Matching up data returned from the various replication reports with the names of the databases and tables, indexes, and partitions where that data lives. - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW COLUMNS FROM crdb_internal.zones; -~~~ - -| column_name | data_type | description | -|---------------------+-------------------------+---------------------------------------------------------------------------------------------------------------------------------| -| zone_id | [`INT8`](int.html) | The ID of the [replication zone](configure-zone.html). | -| subzone_id | [`INT8`](int.html) | The ID of the subzone (i.e., [partition](partition-by.html)). | -| target | [`STRING`](string.html) | The "object" that the constraint is being applied to, e.g., `PARTITION us_west OF INDEX movr.public.users@primary`. | -| range_name | [`STRING`](string.html) | The zone's name. | -| database_name | [`STRING`](string.html) | The [database](show-databases.html) where the `target`'s data is located. | -| table_name | [`STRING`](string.html) | The [table](show-tables.html) where the `target`'s data is located. | -| index_name | [`STRING`](string.html) | The [index](show-index.html) where the `target`'s data is located. | -| partition_name | [`STRING`](string.html) | The [partition](show-partitions.html) where the `target`'s data is located. | -| full_config_yaml | [`STRING`](string.html) | The YAML you used to [configure this replication zone](configure-replication-zones.html). | -| full_config_sql | [`STRING`](string.html) | The SQL you used to [configure this replication zone](configure-replication-zones.html). | -| raw_config_yaml | [`STRING`](string.html) | The YAML for this [replication zone](configure-replication-zones.html), showing only values the user changed from the defaults. | -| raw_config_sql | [`STRING`](string.html) | The SQL for this [replication zone](configure-replication-zones.html), showing only values the user changed from the defaults. | -| raw_config_protobuf | [`BYTES`](bytes.html) | A protobuf representation of the configuration for this [replication zone](configure-replication-zones.html). | - -## Examples - -{% include {{page.version.version}}/sql/movr-statements-geo-partitioned-replicas.md %} - -### Find out which of your tables have a constraint violation - -By default, this geo-distributed demo cluster will not have any constraint violations. - -To introduce a violation that we can then query for, we'll modify the zone configuration of the `users` table. - -First, let's see what existing zone configurations are attached to the `users` table, so we know what to modify. - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW CREATE TABLE users; -~~~ - -~~~ - table_name | create_statement -+------------+-------------------------------------------------------------------------------------+ - users | CREATE TABLE users ( - | id UUID NOT NULL, - | city VARCHAR NOT NULL, - | name VARCHAR NULL, - | address VARCHAR NULL, - | credit_card VARCHAR NULL, - | CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - | FAMILY "primary" (id, city, name, address, credit_card) - | ) PARTITION BY LIST (city) ( - | PARTITION us_west VALUES IN (('seattle'), ('san francisco'), ('los angeles')), - | PARTITION us_east VALUES IN (('new york'), ('boston'), ('washington dc')), - | PARTITION europe_west VALUES IN (('amsterdam'), ('paris'), ('rome')) - | ); - | ALTER PARTITION europe_west OF INDEX movr.public.users@primary CONFIGURE ZONE USING - | constraints = '[+region=europe-west1]'; - | ALTER PARTITION us_east OF INDEX movr.public.users@primary CONFIGURE ZONE USING - | constraints = '[+region=us-east1]'; - | ALTER PARTITION us_west OF INDEX movr.public.users@primary CONFIGURE ZONE USING - | constraints = '[+region=us-west1]' -(1 row) -~~~ - -To create a constraint violation, let's tell the ranges in the `europe_west` partition that they are explicitly supposed to *not* be in the `region=europe-west1` locality by issuing the following statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER PARTITION europe_west of INDEX movr.public.users@primary CONFIGURE ZONE USING constraints = '[-region=europe-west1]'; -~~~ - -Once the statement above executes, the ranges currently stored in that locality will now be in a state where they are explicitly not supposed to be in that locality, and are thus in violation of a constraint. - -In other words, we are telling the ranges "where you are now is exactly where you are *not* supposed to be". This will cause the cluster to rebalance the ranges, which will take some time. During the time it takes for the rebalancing to occur, the ranges will be in violation. - -By default, the system constraint conformance report runs once every minute. You can change that interval by modifying the `kv.replication_reports.interval` [cluster setting](cluster-settings.html). - -After the internal constraint conformance report has run again, the following query should report a violation: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT * FROM system.replication_constraint_stats WHERE violating_ranges > 0; -~~~ - -~~~ - zone_id | subzone_id | type | config | report_id | violation_start | violating_ranges -+---------+------------+------------+----------------------+-----------+---------------------------------+------------------+ - 53 | 2 | constraint | +region=us-east1 | 1 | 2019-10-21 20:28:40.79508+00:00 | 2 - 53 | 3 | constraint | -region=europe-west1 | 1 | 2019-10-21 20:28:40.79508+00:00 | 2 - 54 | 2 | constraint | +region=us-west1 | 1 | 2019-10-21 20:28:40.79508+00:00 | 1 - 54 | 4 | constraint | +region=us-east1 | 1 | 2019-10-21 20:28:40.79508+00:00 | 2 - 55 | 6 | constraint | +region=us-east1 | 1 | 2019-10-21 20:28:40.79508+00:00 | 4 - 55 | 9 | constraint | +region=europe-west1 | 1 | 2019-10-21 20:28:40.79508+00:00 | 6 - 56 | 2 | constraint | +region=us-east1 | 1 | 2019-10-21 20:28:40.79508+00:00 | 2 - 56 | 3 | constraint | +region=europe-west1 | 1 | 2019-10-21 20:28:40.79508+00:00 | 1 - 58 | 2 | constraint | +region=us-east1 | 1 | 2019-10-21 20:28:40.79508+00:00 | 2 -(9 rows) -~~~ - -To be more useful, we'd like to find out the database and table names where these constraint-violating ranges live. To get that information we'll need to join the output of `system.replication_constraint_stats` report with the `crdb_internal.zones` table using a query like the following: - -{% include_cached copy-clipboard.html %} -~~~ sql -WITH - partition_violations - AS ( - SELECT - * - FROM - system.replication_constraint_stats - WHERE - violating_ranges > 0 - ), - report - AS ( - SELECT - crdb_internal.zones.zone_id, - crdb_internal.zones.subzone_id, - target, - database_name, - table_name, - index_name, - partition_violations.type, - partition_violations.config, - partition_violations.violation_start, - partition_violations.violating_ranges - FROM - crdb_internal.zones, partition_violations - WHERE - crdb_internal.zones.zone_id - = partition_violations.zone_id - ) -SELECT * FROM report; -~~~ - -~~~ - zone_id | subzone_id | target | database_name | table_name | index_name | type | config | violation_start | violating_ranges -+---------+------------+------------------------------------------------------------------------------------------------+---------------+------------+-----------------------------------------------+------------+----------------------+---------------------------------+------------------+ - 53 | 1 | PARTITION us_west OF INDEX movr.public.users@primary | movr | users | primary | constraint | -region=europe-west1 | 2019-10-21 20:28:40.79508+00:00 | 1 - 53 | 2 | PARTITION us_east OF INDEX movr.public.users@primary | movr | users | primary | constraint | -region=europe-west1 | 2019-10-21 20:28:40.79508+00:00 | 1 - 53 | 3 | PARTITION europe_west OF INDEX movr.public.users@primary | movr | users | primary | constraint | -region=europe-west1 | 2019-10-21 20:28:40.79508+00:00 | 1 - 54 | 1 | PARTITION us_west OF INDEX movr.public.vehicles@primary | movr | vehicles | primary | constraint | +region=us-east1 | 2019-10-21 20:28:40.79508+00:00 | 1 - 54 | 2 | PARTITION us_west OF INDEX movr.public.vehicles@vehicles_auto_index_fk_city_ref_users | movr | vehicles | vehicles_auto_index_fk_city_ref_users | constraint | +region=us-east1 | 2019-10-21 20:28:40.79508+00:00 | 1 - 54 | 3 | PARTITION us_east OF INDEX movr.public.vehicles@primary | movr | vehicles | primary | constraint | +region=us-east1 | 2019-10-21 20:28:40.79508+00:00 | 1 - 54 | 4 | PARTITION us_east OF INDEX movr.public.vehicles@vehicles_auto_index_fk_city_ref_users | movr | vehicles | vehicles_auto_index_fk_city_ref_users | constraint | +region=us-east1 | 2019-10-21 20:28:40.79508+00:00 | 1 - 54 | 5 | PARTITION europe_west OF INDEX movr.public.vehicles@primary | movr | vehicles | primary | constraint | +region=us-east1 | 2019-10-21 20:28:40.79508+00:00 | 1 - 54 | 6 | PARTITION europe_west OF INDEX movr.public.vehicles@vehicles_auto_index_fk_city_ref_users | movr | vehicles | vehicles_auto_index_fk_city_ref_users | constraint | +region=us-east1 | 2019-10-21 20:28:40.79508+00:00 | 1 - 55 | 1 | PARTITION us_west OF INDEX movr.public.rides@primary | movr | rides | primary | constraint | +region=us-east1 | 2019-10-21 20:28:40.79508+00:00 | 1 - 55 | 2 | PARTITION us_west OF INDEX movr.public.rides@rides_auto_index_fk_city_ref_users | movr | rides | rides_auto_index_fk_city_ref_users | constraint | +region=us-east1 | 2019-10-21 20:28:40.79508+00:00 | 1 - 55 | 3 | PARTITION us_west OF INDEX movr.public.rides@rides_auto_index_fk_vehicle_city_ref_vehicles | movr | rides | rides_auto_index_fk_vehicle_city_ref_vehicles | constraint | +region=us-east1 | 2019-10-21 20:28:40.79508+00:00 | 1 - 55 | 4 | PARTITION us_east OF INDEX movr.public.rides@primary | movr | rides | primary | constraint | +region=us-east1 | 2019-10-21 20:28:40.79508+00:00 | 1 - 55 | 5 | PARTITION us_east OF INDEX movr.public.rides@rides_auto_index_fk_city_ref_users | movr | rides | rides_auto_index_fk_city_ref_users | constraint | +region=us-east1 | 2019-10-21 20:28:40.79508+00:00 | 1 - 55 | 6 | PARTITION us_east OF INDEX movr.public.rides@rides_auto_index_fk_vehicle_city_ref_vehicles | movr | rides | rides_auto_index_fk_vehicle_city_ref_vehicles | constraint | +region=us-east1 | 2019-10-21 20:28:40.79508+00:00 | 1 - 55 | 7 | PARTITION europe_west OF INDEX movr.public.rides@primary | movr | rides | primary | constraint | +region=us-east1 | 2019-10-21 20:28:40.79508+00:00 | 1 - 55 | 8 | PARTITION europe_west OF INDEX movr.public.rides@rides_auto_index_fk_city_ref_users | movr | rides | rides_auto_index_fk_city_ref_users | constraint | +region=us-east1 | 2019-10-21 20:28:40.79508+00:00 | 1 - 55 | 9 | PARTITION europe_west OF INDEX movr.public.rides@rides_auto_index_fk_vehicle_city_ref_vehicles | movr | rides | rides_auto_index_fk_vehicle_city_ref_vehicles | constraint | +region=us-east1 | 2019-10-21 20:28:40.79508+00:00 | 1 -(18 rows) -~~~ - -If you were to repeat this query at 60-second intervals, you would see that the number of results returned decreases and eventually falls to zero as the cluster rebalances the ranges to their new homes. Eventually you will see this output, which will tell you that the rebalancing has finished. - -~~~ - zone_id | subzone_id | target | database_name | table_name | index_name | type | config | violation_start | violating_ranges -+---------+------------+--------+---------------+------------+------------+------+--------+-----------------+------------------+ -(0 rows) -~~~ - -### Find out which databases and tables have under-replicated ranges - -By default, this geo-distributed demo cluster will not have any [under-replicated ranges](ui-replication-dashboard.html#under-replicated-ranges). - -To force it into a state where some ranges are under-replicated, issue the following statement, which tells it to store 9 copies of each range underlying the `rides` table (by default [it stores 3](configure-replication-zones.html#num_replicas)). - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER TABLE rides CONFIGURE ZONE USING num_replicas=9; -~~~ - -Once the statement above executes, the cluster will rebalance so that it's storing 9 copies of each range underlying the `rides` table. During the time it takes for the rebalancing to occur, these ranges will be considered "under-replicated", since there are not yet as many copies (9) of each range as you have just specified. - -By default, the internal constraint conformance report runs once every minute. You can change that interval by modifying the `kv.replication_reports.interval` [cluster setting](cluster-settings.html). - -After the system constraint conformance report has run again, the following query should report under-replicated ranges: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT * FROM system.replication_stats WHERE under_replicated_ranges > 0; -~~~ - -~~~ - zone_id | subzone_id | report_id | total_ranges | unavailable_ranges | under_replicated_ranges | over_replicated_ranges -+---------+------------+-----------+--------------+--------------------+-------------------------+------------------------+ - 55 | 0 | 3 | 28 | 0 | 6 | 0 - 55 | 3 | 3 | 9 | 0 | 9 | 0 - 55 | 6 | 3 | 9 | 0 | 9 | 0 - 55 | 9 | 3 | 9 | 0 | 9 | 0 -(4 rows) -~~~ - -To be more useful, we'd like to find out the database and table names where these under-replicated ranges live. To get that information we'll need to join the output of `system.replication_stats` report with the `crdb_internal.zones` table using a query like the following: - -{% include_cached copy-clipboard.html %} -~~~ sql -WITH - under_replicated_zones - AS ( - SELECT - zone_id, under_replicated_ranges - FROM - system.replication_stats - WHERE - under_replicated_ranges > 0 - ), - report - AS ( - SELECT - crdb_internal.zones.zone_id, - target, - range_name, - database_name, - table_name, - index_name, - under_replicated_zones.under_replicated_ranges - FROM - crdb_internal.zones, under_replicated_zones - WHERE - crdb_internal.zones.zone_id - = under_replicated_zones.zone_id - ) -SELECT * FROM report; -~~~ - -~~~ - zone_id | target | range_name | database_name | table_name | index_name | under_replicated_ranges -+---------+------------------------------------------------------------------------------------------------+------------+---------------+------------+-----------------------------------------------+-------------------------+ - 55 | TABLE movr.public.rides | NULL | movr | rides | NULL | 9 - 55 | TABLE movr.public.rides | NULL | movr | rides | NULL | 9 - 55 | TABLE movr.public.rides | NULL | movr | rides | NULL | 9 - 55 | PARTITION us_west OF INDEX movr.public.rides@primary | NULL | movr | rides | primary | 9 - 55 | PARTITION us_west OF INDEX movr.public.rides@primary | NULL | movr | rides | primary | 9 - 55 | PARTITION us_west OF INDEX movr.public.rides@primary | NULL | movr | rides | primary | 9 - 55 | PARTITION us_west OF INDEX movr.public.rides@rides_auto_index_fk_city_ref_users | NULL | movr | rides | rides_auto_index_fk_city_ref_users | 9 - 55 | PARTITION us_west OF INDEX movr.public.rides@rides_auto_index_fk_city_ref_users | NULL | movr | rides | rides_auto_index_fk_city_ref_users | 9 - 55 | PARTITION us_west OF INDEX movr.public.rides@rides_auto_index_fk_city_ref_users | NULL | movr | rides | rides_auto_index_fk_city_ref_users | 9 - 55 | PARTITION us_west OF INDEX movr.public.rides@rides_auto_index_fk_vehicle_city_ref_vehicles | NULL | movr | rides | rides_auto_index_fk_vehicle_city_ref_vehicles | 9 - 55 | PARTITION us_west OF INDEX movr.public.rides@rides_auto_index_fk_vehicle_city_ref_vehicles | NULL | movr | rides | rides_auto_index_fk_vehicle_city_ref_vehicles | 9 - 55 | PARTITION us_west OF INDEX movr.public.rides@rides_auto_index_fk_vehicle_city_ref_vehicles | NULL | movr | rides | rides_auto_index_fk_vehicle_city_ref_vehicles | 9 - 55 | PARTITION us_east OF INDEX movr.public.rides@primary | NULL | movr | rides | primary | 9 - 55 | PARTITION us_east OF INDEX movr.public.rides@primary | NULL | movr | rides | primary | 9 - 55 | PARTITION us_east OF INDEX movr.public.rides@primary | NULL | movr | rides | primary | 9 - 55 | PARTITION us_east OF INDEX movr.public.rides@rides_auto_index_fk_city_ref_users | NULL | movr | rides | rides_auto_index_fk_city_ref_users | 9 - 55 | PARTITION us_east OF INDEX movr.public.rides@rides_auto_index_fk_city_ref_users | NULL | movr | rides | rides_auto_index_fk_city_ref_users | 9 - 55 | PARTITION us_east OF INDEX movr.public.rides@rides_auto_index_fk_city_ref_users | NULL | movr | rides | rides_auto_index_fk_city_ref_users | 9 - 55 | PARTITION us_east OF INDEX movr.public.rides@rides_auto_index_fk_vehicle_city_ref_vehicles | NULL | movr | rides | rides_auto_index_fk_vehicle_city_ref_vehicles | 9 - 55 | PARTITION us_east OF INDEX movr.public.rides@rides_auto_index_fk_vehicle_city_ref_vehicles | NULL | movr | rides | rides_auto_index_fk_vehicle_city_ref_vehicles | 9 - 55 | PARTITION us_east OF INDEX movr.public.rides@rides_auto_index_fk_vehicle_city_ref_vehicles | NULL | movr | rides | rides_auto_index_fk_vehicle_city_ref_vehicles | 9 - 55 | PARTITION europe_west OF INDEX movr.public.rides@primary | NULL | movr | rides | primary | 9 - 55 | PARTITION europe_west OF INDEX movr.public.rides@primary | NULL | movr | rides | primary | 9 - 55 | PARTITION europe_west OF INDEX movr.public.rides@primary | NULL | movr | rides | primary | 9 - 55 | PARTITION europe_west OF INDEX movr.public.rides@rides_auto_index_fk_city_ref_users | NULL | movr | rides | rides_auto_index_fk_city_ref_users | 9 - 55 | PARTITION europe_west OF INDEX movr.public.rides@rides_auto_index_fk_city_ref_users | NULL | movr | rides | rides_auto_index_fk_city_ref_users | 9 - 55 | PARTITION europe_west OF INDEX movr.public.rides@rides_auto_index_fk_city_ref_users | NULL | movr | rides | rides_auto_index_fk_city_ref_users | 9 - 55 | PARTITION europe_west OF INDEX movr.public.rides@rides_auto_index_fk_vehicle_city_ref_vehicles | NULL | movr | rides | rides_auto_index_fk_vehicle_city_ref_vehicles | 9 - 55 | PARTITION europe_west OF INDEX movr.public.rides@rides_auto_index_fk_vehicle_city_ref_vehicles | NULL | movr | rides | rides_auto_index_fk_vehicle_city_ref_vehicles | 9 - 55 | PARTITION europe_west OF INDEX movr.public.rides@rides_auto_index_fk_vehicle_city_ref_vehicles | NULL | movr | rides | rides_auto_index_fk_vehicle_city_ref_vehicles | 9 -(30 rows) -~~~ - -### Find out which databases and tables have ranges in critical localities - -The `system.replication_critical_localities` report contains which of your localities (if any) are critical. A locality is "critical" for a range if all of the nodes in that locality becoming unreachable would cause the range to become unavailable. In other words, the locality contains a majority of the range's replicas. - -That said, a locality being critical is not necessarily a bad thing as long as you are aware of it. What matters is that [you configure the topology of your cluster to get the resiliency you expect](topology-patterns.html). - -By default, the [movr demo cluster](cockroach-demo.html) has some ranges in critical localities. This is expected because it ties data for latency-sensitive queries to specific geographies at the cost of data unavailability during a region-wide failure. - -{% include {{page.version.version}}/sql/use-multiregion-instead-of-partitioning.md %} - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT * FROM system.replication_critical_localities WHERE at_risk_ranges > 0; -~~~ - -~~~ - zone_id | subzone_id | locality | report_id | at_risk_ranges -+---------+------------+---------------------+-----------+----------------+ - 53 | 1 | region=us-west1 | 2 | 3 - 53 | 2 | region=us-east1 | 2 | 3 - 53 | 3 | region=europe-west1 | 2 | 3 - 54 | 2 | region=us-west1 | 2 | 6 - 54 | 4 | region=us-east1 | 2 | 6 - 54 | 6 | region=europe-west1 | 2 | 6 - 55 | 3 | region=us-west1 | 2 | 9 - 55 | 6 | region=us-east1 | 2 | 9 - 55 | 9 | region=europe-west1 | 2 | 9 - 56 | 1 | region=us-west1 | 2 | 3 - 56 | 2 | region=us-east1 | 2 | 3 - 56 | 3 | region=europe-west1 | 2 | 3 - 58 | 1 | region=us-west1 | 2 | 3 - 58 | 2 | region=us-east1 | 2 | 3 - 58 | 3 | region=europe-west1 | 2 | 3 -(15 rows) -~~~ - -To be more useful, we'd like to find out the database and table names where these ranges live that are at risk of unavailability in the event of a locality becoming unreachable. To get that information we'll need to join the output of `system.replication_critical_localities` report with the `crdb_internal.zones` table using a query like the following: - -{% include_cached copy-clipboard.html %} -~~~ sql -WITH - at_risk_zones AS ( - SELECT - zone_id, locality, at_risk_ranges - FROM - system.replication_critical_localities - WHERE - at_risk_ranges > 0 - ), - report AS ( - SELECT - crdb_internal.zones.zone_id, - target, - database_name, - table_name, - index_name, - at_risk_zones.at_risk_ranges - FROM - crdb_internal.zones, at_risk_zones - WHERE - crdb_internal.zones.zone_id - = at_risk_zones.zone_id - ) -SELECT DISTINCT * FROM report; -~~~ - -~~~ - zone_id | target | database_name | table_name | index_name | at_risk_ranges -+---------+------------------------------------------------------------------------------------------------+---------------+----------------------------+-----------------------------------------------+----------------+ - 53 | PARTITION us_west OF INDEX movr.public.users@primary | movr | users | primary | 3 - 53 | PARTITION us_east OF INDEX movr.public.users@primary | movr | users | primary | 3 - 53 | PARTITION europe_west OF INDEX movr.public.users@primary | movr | users | primary | 3 - 54 | PARTITION us_west OF INDEX movr.public.vehicles@primary | movr | vehicles | primary | 6 - 54 | PARTITION us_west OF INDEX movr.public.vehicles@vehicles_auto_index_fk_city_ref_users | movr | vehicles | vehicles_auto_index_fk_city_ref_users | 6 - 54 | PARTITION us_east OF INDEX movr.public.vehicles@primary | movr | vehicles | primary | 6 - 54 | PARTITION us_east OF INDEX movr.public.vehicles@vehicles_auto_index_fk_city_ref_users | movr | vehicles | vehicles_auto_index_fk_city_ref_users | 6 - 54 | PARTITION europe_west OF INDEX movr.public.vehicles@primary | movr | vehicles | primary | 6 - 54 | PARTITION europe_west OF INDEX movr.public.vehicles@vehicles_auto_index_fk_city_ref_users | movr | vehicles | vehicles_auto_index_fk_city_ref_users | 6 - 55 | PARTITION us_west OF INDEX movr.public.rides@primary | movr | rides | primary | 9 - 55 | PARTITION us_west OF INDEX movr.public.rides@rides_auto_index_fk_city_ref_users | movr | rides | rides_auto_index_fk_city_ref_users | 9 - 55 | PARTITION us_west OF INDEX movr.public.rides@rides_auto_index_fk_vehicle_city_ref_vehicles | movr | rides | rides_auto_index_fk_vehicle_city_ref_vehicles | 9 - 55 | PARTITION us_east OF INDEX movr.public.rides@primary | movr | rides | primary | 9 - 55 | PARTITION us_east OF INDEX movr.public.rides@rides_auto_index_fk_city_ref_users | movr | rides | rides_auto_index_fk_city_ref_users | 9 - 55 | PARTITION us_east OF INDEX movr.public.rides@rides_auto_index_fk_vehicle_city_ref_vehicles | movr | rides | rides_auto_index_fk_vehicle_city_ref_vehicles | 9 - 55 | PARTITION europe_west OF INDEX movr.public.rides@primary | movr | rides | primary | 9 - 55 | PARTITION europe_west OF INDEX movr.public.rides@rides_auto_index_fk_city_ref_users | movr | rides | rides_auto_index_fk_city_ref_users | 9 - 55 | PARTITION europe_west OF INDEX movr.public.rides@rides_auto_index_fk_vehicle_city_ref_vehicles | movr | rides | rides_auto_index_fk_vehicle_city_ref_vehicles | 9 - 56 | PARTITION us_west OF INDEX movr.public.vehicle_location_histories@primary | movr | vehicle_location_histories | primary | 3 - 56 | PARTITION us_east OF INDEX movr.public.vehicle_location_histories@primary | movr | vehicle_location_histories | primary | 3 - 56 | PARTITION europe_west OF INDEX movr.public.vehicle_location_histories@primary | movr | vehicle_location_histories | primary | 3 - 58 | PARTITION us_west OF INDEX movr.public.user_promo_codes@primary | movr | user_promo_codes | primary | 3 - 58 | PARTITION us_east OF INDEX movr.public.user_promo_codes@primary | movr | user_promo_codes | primary | 3 - 58 | PARTITION europe_west OF INDEX movr.public.user_promo_codes@primary | movr | user_promo_codes | primary | 3 -(24 rows) -~~~ - -To give another example, let's say your cluster were similar to the one shown above, but configured with [tiered localities](cockroach-start.html#locality) such that you had split `us-east1` into `{region=us-east1,dc=dc1, region=us-east1,dc=dc2, region=us-east1,dc=dc3}`. In that case, you wouldn't expect any DC to be critical, because the cluster would "diversify" each range's location as much as possible across data centers. In such a situation, if you were to see a DC identified as a critical locality, you'd be surprised and you'd take some action. For example, perhaps the diversification process is failing because some localities are filled to capacity. If there is no disk space free in a locality, your cluster cannot move replicas there. - -## See also - -- [Configure Replication Zones](configure-replication-zones.html) -- [Partitioning](partitioning.html) -- [`PARTITION BY`](partition-by.html) -- [`CONFIGURE ZONE`](configure-zone.html) -- [Start a node](cockroach-start.html) diff --git a/src/current/v21.1/query-spatial-data.md b/src/current/v21.1/query-spatial-data.md deleted file mode 100644 index 092d2411c10..00000000000 --- a/src/current/v21.1/query-spatial-data.md +++ /dev/null @@ -1,150 +0,0 @@ ---- -title: Query Spatial Data -summary: How to query a spatial dataset imported into CockroachDB. -toc: true ---- - -This page has instructions for querying spatial data imported into CockroachDB. On this page, we use a sample [Shapefile dataset](spatial-glossary.html#file-formats) from the National Oceanic and Atmospheric Administration. - -## Before you begin - -- Install a build of CockroachDB with support for spatial data by following the instructions at [Install CockroachDB](install-cockroachdb.html). - -- [Start a local insecure cluster](start-a-local-cluster.html) and connect to that cluster from a [SQL client](cockroach-sql.html): - - {% include_cached copy-clipboard.html %} - ~~~ shell - cockroach sql --insecure --host=localhost --port=26257 - ~~~ - - Leave this shell open for use in the example below. - -- Import some sample spatial data into CockroachDB by following the instructions at [Migrate from Shapefiles](migrate-from-shapefiles.html). - -## Query spatial data - -After you have imported [the sample tornado data into CockroachDB](migrate-from-shapefiles.html), you can query the spatial data from SQL. - -For example, we may be interested in the [1999 Oklahoma tornado outbreak](https://en.wikipedia.org/wiki/1999_Oklahoma_tornado_outbreak), which is described by Wikipedia as: - -> a significant tornado outbreak that affected much of the Central and parts of the Eastern United States, with the highest record-breaking wind speeds of 302 ± 22 mph (486 ± 35 km/h). During this week-long event, 154 tornadoes touched down (including one in Canada), more than half of them on May 3 and 4 when activity reached its peak over Oklahoma, Kansas, Nebraska, Texas, and Arkansas. - -According to the wiki page linked above, there were 152 tornadoes confirmed between May 2-8, 1999 (one in Canada). - -We can try to verify this number against the NWS's tornado data set with the following query: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT COUNT(*) FROM "1950-2018-torn-initpoint" WHERE yr = 1999 AND mo = 5 AND dy >= 02 AND dy <= 08; -~~~ - -~~~ - count ---------- - 150 -(1 row) -~~~ - -It might be interesting to look into why these numbers are different! - -Next, let's get a list of starting points for all of the tornadoes in the outbreak that started in Oklahoma: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT ST_AsText(geom) FROM "1950-2018-torn-initpoint" WHERE yr = 1999 AND st = 'OK' AND mo = 5 AND dy > 02 AND dy <= 08; -~~~ - -~~~ - st_astext --------------------------------------------------- - POINT (-98.379999999999995 34.770000000000003) - POINT (-98.329999999999998 34.780000000000001) - POINT (-98.319999999999993 34.880000000000003) - POINT (-98.230000000000004 34.920000000000002) - POINT (-99.019999999999996 34.799999999999997) - POINT (-98.25 35.030000000000001) - POINT (-98.120000000000005 34.969999999999999) - POINT (-98.030000000000001 35.049999999999997) - POINT (-97.980000000000004 35.079999999999998) - POINT (-98.569999999999993 34.950000000000003) - POINT (-97.849999999999994 35.130000000000003) - POINT (-98.430000000000007 34.979999999999997) - POINT (-98.329999999999998 35.07) - POINT (-98.019999999999996 35.719999999999999) - POINT (-97.980000000000004 35.719999999999999) - POINT (-97.599999999999994 35.299999999999997) - POINT (-98.280000000000001 35.119999999999997) - POINT (-98.200000000000003 35.170000000000002) - POINT (-97.400000000000006 35.399999999999999) - POINT (-98.099999999999994 35.18) - POINT (-98.069999999999993 35.270000000000003) - POINT (-98.129999999999995 35.270000000000003) - POINT (-98.019999999999996 35.32) - POINT (-97.299999999999997 35.469999999999999) - POINT (-98 35.270000000000003) - POINT (-97.969999999999999 35.399999999999999) - POINT (-97.219999999999999 35.549999999999997) - POINT (-97.920000000000002 35.420000000000002) - POINT (-97.900000000000006 35.43) - POINT (-97.230000000000004 35.579999999999998) - POINT (-98.370000000000005 35.880000000000003) - POINT (-97.920000000000002 35.520000000000003) - POINT (-98.280000000000001 35.649999999999999) - POINT (-97.849999999999994 35.530000000000001) - POINT (-97.200000000000003 35.130000000000003) - POINT (-97.780000000000001 35.649999999999999) - POINT (-98.030000000000001 35.850000000000001) - POINT (-97.719999999999999 35.700000000000003) - POINT (-98.030000000000001 35.880000000000003) - POINT (-97 35.369999999999997) - POINT (-97.680000000000007 35.780000000000001) - POINT (-97.950000000000003 35.93) - POINT (-98.170000000000002 35.850000000000001) - POINT (-97.680000000000007 35.880000000000003) - POINT (-97.879999999999995 36.020000000000003) - POINT (-97.950000000000003 36.020000000000003) - POINT (-98 35.5) - POINT (-97.879999999999995 36.100000000000001) - POINT (-97.969999999999999 35.549999999999997) - POINT (-96.799999999999997 35.649999999999999) - POINT (-97.650000000000006 36.119999999999997) - POINT (-98.25 36.299999999999997) - POINT (-97.719999999999999 35.780000000000001) - POINT (-97.780000000000001 35.850000000000001) - POINT (-97.599999999999994 35.920000000000002) - POINT (-97.420000000000002 36.030000000000001) - POINT (-96.129999999999995 35.979999999999997) - POINT (-96.069999999999993 36.020000000000003) - POINT (-95.650000000000006 35.630000000000003) - POINT (-95.180000000000007 35.950000000000003) - POINT (-94.730000000000004 36) - POINT (-97.400000000000006 35.32) - POINT (-96.400000000000006 36.469999999999999) - POINT (-95.579999999999998 34.579999999999998) - POINT (-95.219999999999999 34.880000000000003) - POINT (-95 35.130000000000003) - POINT (-94.780000000000001 35.299999999999997) - POINT (-94.700000000000003 35.43) - POINT (-94.549999999999997 35.57) -(69 rows) -~~~ - -We can see that almost half of all of the tornadoes in this outbreak began in Oklahoma. - -It might be interesting to draw these points on a map. The image below shows the points from the query above drawn as a simple polygon on a map of Oklahoma. The boxes around the polygon show the [spatial index](spatial-indexes.html) coverings for the polygon. - -1999 Oklahoma tornado outbreak map view - -(Map data © 2020 Google) - -## See also - -- [Spatial Features](spatial-features.html) -- [Spatial indexes](spatial-indexes.html) -- [Spatial & GIS Glossary of Terms](spatial-glossary.html) -- [Working with Spatial Data](spatial-data.html) -- [Migrate from Shapefiles](migrate-from-shapefiles.html) -- [Migrate from GeoJSON](migrate-from-geojson.html) -- [Migrate from GeoPackage](migrate-from-geopackage.html) -- [Migrate from OpenStreetMap](migrate-from-openstreetmap.html) -- [Spatial functions](functions-and-operators.html#spatial-functions) diff --git a/src/current/v21.1/reassign-owned.md b/src/current/v21.1/reassign-owned.md deleted file mode 100644 index 6cd41aa0284..00000000000 --- a/src/current/v21.1/reassign-owned.md +++ /dev/null @@ -1,112 +0,0 @@ ---- -title: REASSIGN OWNED -summary: The REASSIGN OWNED statement changes the ownership of all objects in the current database that are owned by a specific role or user. -toc: true ---- - -{% include_cached new-in.html version="v21.1" %} The `REASSIGN OWNED` statement changes the [ownership](authorization.html#object-ownership) of all database objects (i.e., tables, types, or schemas) in the current database that are currently owned by a specific [role](authorization.html#roles) or [user](authorization.html#sql-users). - -{{site.data.alerts.callout_success}} -To change the ownership of any single object (e.g., a table or a database), use the [`OWNER TO`](owner-to.html) subcommand of the object's [`ALTER` statement](sql-statements.html). -{{site.data.alerts.end}} - -## Required privileges - -- The user executing the `REASSIGN OWNED` statement must be a member of the [`admin` role](authorization.html#admin-role), or must be a member of the target role and have the `CREATE` [privilege](authorization.html#assign-privileges) on the current database. -- The target role (i.e., the desired role of the objects) must have the `CREATE` privilege on the current database. - -## Syntax - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/reassign_owned_by.html %} -
    - -## Parameters - -Parameter | Description -----------|------------ -`role_spec_list` | The source role, or a comma-separated list of source roles. -`role_spec` | The target role. - -## Example - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### Change the owner of all tables in a database - -Suppose that the current owner of the `users`, `vehicles`, and `rides` tables in the `movr` database is a role named `cockroachlabs`. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE ROLE cockroachlabs; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT CREATE ON DATABASE movr TO cockroachlabs; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE users OWNER TO cockroachlabs; -> ALTER TABLE vehicles OWNER TO cockroachlabs; -> ALTER TABLE rides OWNER TO cockroachlabs; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES; -~~~ - -~~~ - schema_name | table_name | type | owner | estimated_row_count | locality ---------------+----------------------------+-------+---------------+---------------------+----------- - public | promo_codes | table | demo | 1000 | NULL - public | rides | table | cockroachlabs | 500 | NULL - public | user_promo_codes | table | demo | 0 | NULL - public | users | table | cockroachlabs | 50 | NULL - public | vehicle_location_histories | table | demo | 1000 | NULL - public | vehicles | table | cockroachlabs | 15 | NULL -(6 rows) -~~~ - -Now suppose you want to change the owner for all of the tables owned by `cockroachlabs` to a new role named `movrlabs`. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE ROLE movrlabs; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT CREATE ON DATABASE movr TO movrlabs; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> REASSIGN OWNED BY cockroachlabs TO movrlabs; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES; -~~~ - -~~~ - schema_name | table_name | type | owner | estimated_row_count | locality ---------------+----------------------------+-------+----------+---------------------+----------- - public | promo_codes | table | demo | 1000 | NULL - public | rides | table | movrlabs | 500 | NULL - public | user_promo_codes | table | demo | 0 | NULL - public | users | table | movrlabs | 50 | NULL - public | vehicle_location_histories | table | demo | 1000 | NULL - public | vehicles | table | movrlabs | 15 | NULL -(6 rows) -~~~ - -## See also - -- [Authorization](authorization.html) -- [`OWNER TO`](owner-to.html) -- [`SHOW TABLES`](show-tables.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/recommended-production-settings.md b/src/current/v21.1/recommended-production-settings.md deleted file mode 100644 index eed1e969334..00000000000 --- a/src/current/v21.1/recommended-production-settings.md +++ /dev/null @@ -1,583 +0,0 @@ ---- -title: Production Checklist -summary: Recommended settings for production deployments of CockroachDB. -toc: true -toc_not_nested: true ---- - -This page provides important recommendations for production deployments of CockroachDB. - -## Topology - -When planning your deployment, it's important to carefully review and choose the [topology patterns](topology-patterns.html) that best meet your latency and resiliency requirements. This is especially crucial for multi-region deployments. - -Also keep in mind some basic topology recommendations: - -{% include {{ page.version.version }}/prod-deployment/topology-recommendations.md %} - -## Software - -We recommend running a [glibc](https://www.gnu.org/software/libc/)-based Linux distribution and Linux kernel version from the last 5 years, such as [Ubuntu](https://ubuntu.com/), [Red Hat Enterprise Linux (RHEL)](https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux), [CentOS](https://www.centos.org/), or [Container-Optimized OS](https://cloud.google.com/container-optimized-os/docs). - -## Hardware - -{{site.data.alerts.callout_info}} -Mentions of "CPU resources" refer to vCPUs, which are also known as hyperthreads. -{{site.data.alerts.end}} - -### Basic hardware recommendations - -This hardware guidance is meant to be platform agnostic and can apply to bare-metal, containerized, and orchestrated deployments. Also see our [cloud-specific](#cloud-specific-recommendations) recommendations. - -| Value | Recommendation | Reference -|-------|----------------|---------- -| RAM per vCPU | 4 GiB | [CPU and memory](#cpu-and-memory) -| Capacity per vCPU | 150 GiB | [Storage](#storage) -| IOPS per vCPU | 500 | [Disk I/O](#disk-i-o) -| MB/s per vCPU | 30 | [Disk I/O](#disk-i-o) - -Before deploying to production, test and tune your hardware setup for your application workload. For example, read-heavy and write-heavy workloads will place different emphases on [CPU](#cpu-and-memory), [RAM](#cpu-and-memory), [storage](#storage), [I/O](#disk-i-o), and [network](#networking) capacity. - -#### CPU and memory - -Each node should have **at least 4 vCPUs**. For best performance, we recommend at least 8 vCPUs per node. Provision **4 GiB of RAM per vCPU**. - -- To optimize for throughput, use larger nodes with up to 32 vCPUs. To further increase throughput, add more nodes to the cluster instead of increasing node size. - - {{site.data.alerts.callout_info}} - Based on internal testing, 32 vCPUs is a sweet spot for OLTP workloads. It is not a hard limit, especially for deployments using physical hardware rather than cloud instances. - {{site.data.alerts.end}} - - {{site.data.alerts.callout_info}} - The benefits to having more RAM decrease as the number of vCPUs increases. - {{site.data.alerts.end}} - -- To optimize for resiliency, use many smaller nodes instead of fewer larger nodes. Recovery from a failed node is faster when data is spread across more nodes. - -- To optimize for the support of large numbers of tables, increase the amount of RAM. For more information, see [Quantity of tables and other schema objects](schema-design-overview.html#quantity-of-tables-and-other-schema-objects). Supporting a large number of rows is a matter of [Storage](#storage). - -- Avoid "burstable" or "shared-core" virtual machines that limit the load on CPU resources. - -- To ensure consistent SQL performance, make sure all nodes have a uniform configuration. - -- Disable Linux memory swapping. CockroachDB manages its own [memory caches](#cache-and-sql-memory-size) (configured via the `--cache` and `--max-sql-memory` flags), independent of the operating system. Over-allocating memory on production machines can lead to unexpected performance issues when pages have to be read back into memory. - -{{site.data.alerts.callout_info}} -Under-provisioning RAM results in reduced performance (due to reduced caching and increased spilling to disk), and in some cases can cause OOM crashes. Under-provisioning CPU generally results in poor performance, and in extreme cases can lead to cluster unavailability. For more information, see [capacity planning issues](cluster-setup-troubleshooting.html#capacity-planning-issues) and [memory issues](cluster-setup-troubleshooting.html#memory-issues). -{{site.data.alerts.end}} - -#### Storage - -We recommend provisioning volumes with **150 GiB per vCPU**. It's fine to have less storage per vCPU if your workload does not have significant capacity needs. - -- The maximum recommended storage capacity per node is 2.5 TiB, regardless of the number of vCPUs. - -- Use dedicated volumes for the CockroachDB [store](architecture/storage-layer.html). Do not share the store volume with any other I/O activity. - - We suggest storing CockroachDB [log files](logging-overview.html) in a separate volume from CockroachDB data so that logging is not impacted by I/O throttling. - -- The recommended Linux filesystems are [ext4](https://ext4.wiki.kernel.org/index.php/Main_Page) and [XFS](https://xfs.wiki.kernel.org/). - -- Always keep some of your disk capacity free on production. Doing so accommodates fluctuations in routine database operations and supports continuous data growth. - - We strongly recommend [monitoring](monitoring-and-alerting.html#node-is-running-low-on-disk-space) your storage utilization and rate of growth, and taking action to add capacity well before you hit the limit. - -- Place a [ballast file](cockroach-debug-ballast.html) in each node's storage directory. In the unlikely case that a node runs out of disk space and shuts down, you can delete the ballast file to free up enough space to be able to restart the node. - -- Use [zone configs](configure-replication-zones.html) to increase the replication factor from 3 (the default) to 5 (across at least 5 nodes). - - This is especially recommended if you are using local disks with no RAID protection rather than a cloud provider's network-attached disks that are often replicated under the hood, because local disks have a greater risk of failure. You can do this for the [entire cluster](configure-replication-zones.html#edit-the-default-replication-zone) or for specific [databases](configure-replication-zones.html#create-a-replication-zone-for-a-database), [tables](configure-replication-zones.html#create-a-replication-zone-for-a-table), or [rows](configure-replication-zones.html#create-a-replication-zone-for-a-partition) (Enterprise-only). - -{{site.data.alerts.callout_info}} -Underprovisioning storage leads to node crashes when the disks fill up. Once this has happened, it is difficult to recover from. To prevent your disks from filling up, provision enough storage for your workload, monitor your disk usage, and use a ballast file as described above. For more information, see [capacity planning issues](cluster-setup-troubleshooting.html#capacity-planning-issues) and [storage issues](cluster-setup-troubleshooting.html#storage-issues). -{{site.data.alerts.end}} - -##### Disk I/O - -Disks must be able to achieve **500 IOPS and 30 MB/s per vCPU**. - -- Monitor IOPS for higher service times. If they exceed 1-5 ms, you will need to add more devices or expand the cluster to reduce the disk latency. To monitor IOPS, use tools such as `iostat` (part of `sysstat`). - -- To calculate IOPS, use [sysbench](https://github.com/akopytov/sysbench). If IOPS decrease, add more nodes to your cluster to increase IOPS. - -- The optimal configuration for striping more than one device is [RAID 10](https://en.wikipedia.org/wiki/Nested_RAID_levels#RAID_10_(RAID_1+0)). RAID 0 and 1 are also acceptable from a performance perspective. - -{{site.data.alerts.callout_info}} -Disk I/O especially affects performance on write-heavy workloads. For more context, see [Reads and Writes in CockroachDB](architecture/reads-and-writes-overview.html#write-scenario). -{{site.data.alerts.end}} - -##### Node density testing configuration - -In a narrowly-scoped test, we were able to successfully store 4.32 TiB of logical data per node. The results of this test may not be applicable to your specific situation; testing with your workload is _strongly_ recommended before using it in a production environment. - -These results were achieved using the ["bank" workload](cockroach-workload.html#bank-workload) running on AWS using 6x c5d.4xlarge nodes, each with 5 TiB of gp2 EBS storage. - -Results: - -| Value | Result | -|-----------------------+-----------------| -| vCPU per Node | 16 | -| Logical Data per Node | 4.32 TiB | -| RAM per Node | 32 GiB | -| Data per Core | ~270 GiB / vCPU | -| Data per RAM | ~135 GiB / GiB | - -### Cloud-specific recommendations - -Cockroach Labs recommends the following cloud-specific configurations based on our own [internal testing](https://www.cockroachlabs.com/blog/2018_cloud_report/). Before using configurations not recommended here, be sure to test them exhaustively. - -#### AWS - -- Use `m` (general purpose) or `c` (compute-optimized) [instances](https://aws.amazon.com/ec2/instance-types/). - - For example, Cockroach Labs has used `c5d.4xlarge` (16 vCPUs and 32 GiB of RAM per instance, EBS) for [performance benchmarking](performance-benchmarking-with-tpcc-small.html). Note that the instance type depends on whether EBS is used or not. If you're using EBS, use a `c5` instance. - - {{site.data.alerts.callout_danger}} - Do not use ["burstable" `t` instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-performance-instances.html), which limit the load on CPU resources. - {{site.data.alerts.end}} - -- Use `c5` instances with EBS as a primary AWS configuration. To simulate bare-metal deployments, use `c5d` with [SSD Instance Store volumes](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ssd-instance-store.html). - - [Provisioned IOPS SSD-backed (`io1`) EBS volumes](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html#EBSVolumeTypes_piops) need to have IOPS provisioned, which can be very expensive. Cheaper `gp2` volumes can be used instead, if your performance needs are less demanding. Allocating more disk space than you will use can improve performance of `gp2` volumes. - -- A typical deployment will use [EC2](https://aws.amazon.com/ec2/) together with [key pairs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html), [load balancers](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/load-balancer-types.html), and [security groups](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/working-with-security-groups.html). For an example, see [Deploy CockroachDB on AWS EC2](deploy-cockroachdb-on-aws.html). - -#### Azure - -- Use compute-optimized [F-series](https://docs.microsoft.com/en-us/azure/virtual-machines/fsv2-series) VMs. - For example, Cockroach Labs has used `Standard_F16s_v2` VMs (16 vCPUs and 32 GiB of RAM per VM) for internal testing. - - {{site.data.alerts.callout_danger}} - Do not use ["burstable" B-series](https://docs.microsoft.com/en-us/azure/virtual-machines/linux/b-series-burstable) VMs, which limit the load on CPU resources. Also, Cockroach Labs has experienced data corruption issues on A-series VMs and irregular disk performance on D-series VMs, so we recommend avoiding those as well. - {{site.data.alerts.end}} - -- Use [Premium Storage](https://docs.microsoft.com/en-us/azure/virtual-machines/windows/premium-storage) or local SSD storage with a Linux filesystem such as `ext4` (not the Windows `ntfs` filesystem). Note that [the size of a Premium Storage disk affects its IOPS](https://docs.microsoft.com/en-us/azure/virtual-machines/windows/premium-storage#premium-storage-disk-limits). - -- If you choose local SSD storage, on reboot, the VM can come back with the `ntfs` filesystem. Be sure your automation monitors for this and reformats the disk to the Linux filesystem you chose initially. - -#### Digital Ocean - -- Use any [droplets](https://www.digitalocean.com/pricing/) except standard droplets with only 1 GiB of RAM, which is below our minimum requirement. All Digital Ocean droplets use SSD storage. - -#### GCP - -- Use `n2-standard` or `n2-highcpu` [predefined VMs](https://cloud.google.com/compute/pricing#predefined_machine_types), or [custom VMs](https://cloud.google.com/compute/pricing#custommachinetypepricing). - - For example, Cockroach Labs has used `n2-standard-16` (16 vCPUs and 64 GiB of RAM per VM, local SSD) for performance benchmarking. - - {{site.data.alerts.callout_danger}} - Do not use `f1` or `g1` [shared-core machines](https://cloud.google.com/compute/docs/machine-types#sharedcore), which limit the load on CPU resources. - {{site.data.alerts.end}} - -- Use [Local SSDs](https://cloud.google.com/compute/docs/disks/#localssds) or [SSD persistent disks](https://cloud.google.com/compute/docs/disks/#pdspecs). Note that [the IOPS of SSD persistent disks depends both on the disk size and number of vCPUs on the machine](https://cloud.google.com/compute/docs/disks/performance#optimizessdperformance). -- `nobarrier` can be used with SSDs, but only if it has battery-backed write cache. Without one, data can be corrupted in the event of a crash. - - Cockroach Labs conducts most of our [internal performance tests](https://www.cockroachlabs.com/blog/2018_cloud_report/) using `nobarrier` to demonstrate the best possible performance, but understand that not all use cases can support this option. - -## Security - -An insecure cluster comes with serious risks: - -- Your cluster is open to any client that can access any node's IP addresses. -- Any user, even `root`, can log in without providing a password. -- Any user, connecting as `root`, can read or write any data in your cluster. -- There is no network encryption or authentication, and thus no confidentiality. - -Therefore, to deploy CockroachDB in production, it is strongly recommended to [use TLS certificates to authenticate](authentication.html) the identity of nodes and clients and to encrypt in-flight data between nodes and clients. You can use either the built-in [`cockroach cert` commands](cockroach-cert.html) or [`openssl` commands](create-security-certificates-openssl.html) to generate security certificates for your deployment. Regardless of which option you choose, you'll need the following files: - -- A certificate authority (CA) certificate and key, used to sign all of the other certificates. -- A separate certificate and key for each node in your deployment, with the common name `node`. -- A separate certificate and key for each client and user you want to connect to your nodes, with the common name set to the username. The default user is `root`. - - Alternatively, CockroachDB supports [password authentication](authentication.html#client-authentication), although we typically recommend using client certificates instead. - -For more information, see the [Security Overview](security-overview.html). - -## Networking - -### Networking flags - -When [starting a node](cockroach-start.html), two main flags are used to control its network connections: - -- `--listen-addr` determines which address(es) to listen on for connections from other nodes and clients. -- `--advertise-addr` determines which address to tell other nodes to use. - -The effect depends on how these two flags are used in combination: - -| | `--listen-addr` not specified | `--listen-addr` specified | -|-|----------------------------|------------------------| -| **`--advertise-addr` not specified** | Node listens on all of its IP addresses on port `26257` and advertises its canonical hostname to other nodes. | Node listens on the IP address/hostname and port specified in `--listen-addr` and advertises this value to other nodes. -| **`--advertise-addr` specified** | Node listens on all of its IP addresses on port `26257` and advertises the value specified in `--advertise-addr` to other nodes. **Recommended for most cases.** | Node listens on the IP address/hostname and port specified in `--listen-addr` and advertises the value specified in `--advertise-addr` to other nodes. If the `--advertise-addr` port number is different than the one used in `--listen-addr`, port forwarding is required. - -{{site.data.alerts.callout_success}} -When using hostnames, make sure they resolve properly (e.g., via DNS or `etc/hosts`). In particular, be careful about the value advertised to other nodes, either via `--advertise-addr` or via `--listen-addr` when `--advertise-addr` is not specified. -{{site.data.alerts.end}} - -### Cluster on a single network - -When running a cluster on a single network, the setup depends on whether the network is private. In a private network, machines have addresses restricted to the network, not accessible to the public internet. Using these addresses is more secure and usually provides lower latency than public addresses. - -Private? | Recommended setup ----------|------------------ -Yes | Start each node with `--listen-addr` set to its private IP address and do not specify `--advertise-addr`. This will tell other nodes to use the private IP address advertised. Load balancers/clients in the private network must use it as well. -No | Start each node with `--advertise-addr` set to a stable public IP address that routes to the node and do not specify `--listen-addr`. This will tell other nodes to use the specific IP address advertised, but load balancers/clients will be able to use any address that routes to the node.

    If load balancers/clients are outside the network, also configure firewalls to allow external traffic to reach the cluster. - -### Cluster spanning multiple networks - -When running a cluster across multiple networks, the setup depends on whether nodes can reach each other across the networks. - -Nodes reachable across networks? | Recommended setup ----------------------------------|------------------ -Yes | This is typical when all networks are on the same cloud. In this case, use the relevant [single network setup](#cluster-on-a-single-network) above. -No | This is typical when networks are on different clouds. In this case, set up a [VPN](https://en.wikipedia.org/wiki/Virtual_private_network), [VPC](https://en.wikipedia.org/wiki/Virtual_private_cloud), [NAT](https://en.wikipedia.org/wiki/Network_address_translation), or another such solution to provide unified routing across the networks. Then start each node with `--advertise-addr` set to the address that is reachable from other networks and do not specify `--listen-addr`. This will tell other nodes to use the specific IP address advertised, but load balancers/clients will be able to use any address that routes to the node.

    Also, if a node is reachable from other nodes in its network on a private or local address, set [`--locality-advertise-addr`](cockroach-start.html#networking) to that address. This will tell nodes within the same network to prefer the private or local address to improve performance. Note that this feature requires that each node is started with the [`--locality`](cockroach-start.html#locality) flag. For more details, see this [example](cockroach-start.html#start-a-multi-node-cluster-across-private-networks). - -## Load balancing - -Each CockroachDB node is an equally suitable SQL gateway to a cluster, but to ensure client performance and reliability, it's important to use load balancing: - -- **Performance:** Load balancers spread client traffic across nodes. This prevents any one node from being overwhelmed by requests and improves overall cluster performance (queries per second). - -- **Reliability:** Load balancers decouple client health from the health of a single CockroachDB node. To ensure that traffic is not directed to failed nodes or nodes that are not ready to receive requests, load balancers should use [CockroachDB's readiness health check](monitoring-and-alerting.html#health-ready-1). - {{site.data.alerts.callout_success}} - With a single load balancer, client connections are resilient to node failure, but the load balancer itself is a point of failure. It's therefore best to make load balancing resilient as well by using multiple load balancing instances, with a mechanism like floating IPs or DNS to select load balancers for clients. - {{site.data.alerts.end}} - -For guidance on load balancing, see the tutorial for your deployment environment: - -Environment | Featured Approach -------------|--------------------- -[On-Premises](deploy-cockroachdb-on-premises.html#step-6-set-up-load-balancing) | Use HAProxy. -[AWS](deploy-cockroachdb-on-aws.html#step-4-set-up-load-balancing) | Use Amazon's managed load balancing service. -[Azure](deploy-cockroachdb-on-microsoft-azure.html#step-4-set-up-load-balancing) | Use Azure's managed load balancing service. -[Digital Ocean](deploy-cockroachdb-on-digital-ocean.html#step-3-set-up-load-balancing) | Use Digital Ocean's managed load balancing service. -[GCP](deploy-cockroachdb-on-google-cloud-platform.html#step-4-set-up-load-balancing) | Use GCP's managed TCP proxy load balancing service. - -## Connection pooling - -Creating the appropriate size pool of connections is critical to gaining maximum performance in an application. Too few connections in the pool will result in high latency as each operation waits for a connection to open up. But adding too many connections to the pool can also result in high latency as each connection thread is being run in parallel by the system. The time it takes for many threads to complete in parallel is typically higher than the time it takes a smaller number of threads to run sequentially. - -For guidance on sizing, validating, and using connection pools with CockroachDB, see [Use Connection Pools](connection-pooling.html). - -## Monitoring and alerting - -{% include {{ page.version.version }}/prod-deployment/monitor-cluster.md %} - -## Clock synchronization - -{% include {{ page.version.version }}/faq/clock-synchronization-effects.md %} - -## Cache and SQL memory size - -Each node has a default cache size of `128MiB` that is passively consumed. The default was chosen to facilitate development and testing, where users are likely to run multiple CockroachDB nodes on a single machine. When running a production cluster with one node per host, we recommend increasing this value. - -Each node has a default SQL memory size of `25%`. This memory is used as-needed by active operations to store temporary data for SQL queries. - -- Increasing a node's **cache size** will improve the node's read performance. -- Increasing a node's **SQL memory size** will increase the number of simultaneous client connections it allows, as well as the node's capacity for in-memory processing of rows when using `ORDER BY`, `GROUP BY`, `DISTINCT`, joins, and window functions. - -To manually increase a node's cache size and SQL memory size, start the node using the [`--cache`](cockroach-start.html#flags) and [`--max-sql-memory`](cockroach-start.html#flags) flags: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach start --cache=.25 --max-sql-memory=.25 -~~~ - -{{site.data.alerts.callout_danger}} -Avoid setting `--cache` and `--max-sql-memory` to a combined value of more than 75% of a machine's total RAM. Doing so increases the risk of memory-related failures. Also, since CockroachDB manages its own memory caches, disable Linux memory swapping to avoid over-allocating memory. -{{site.data.alerts.end}} - -## Dependencies - -The [CockroachDB binary for Linux](install-cockroachdb-linux.html) depends on the following libraries: - -Library | Description ------------|------------ -[`glibc`](https://www.gnu.org/software/libc/libc.html) | The standard C library.

    If you build CockroachDB from source, rather than use the official CockroachDB binary for Linux, you can use any C standard library, including MUSL, the C standard library used on Alpine. -`libncurses` | Required by the [built-in SQL shell](cockroach-sql.html). -[`tzdata`](https://www.iana.org/time-zones) | Required by certain features of CockroachDB that use time zone data, for example, to support using location-based names as time zone identifiers. This library is sometimes called `tz` or `zoneinfo`.

    **New in v21.1:** The CockroachDB binary now includes Go's copy of the `tzdata` library. If CockroachDB cannot find the `tzdata` library externally, it will use this built-in copy. - -These libraries are found by default on nearly all Linux distributions, with Alpine as the notable exception, but it's nevertheless important to confirm that they are installed and kept up-to-date. For the time zone data in particular, it's important for all nodes to have the same version; when updating the library, do so as quickly as possible across all nodes. - -{{site.data.alerts.callout_info}} -In Docker-based deployments of CockroachDB, these dependencies do not need to be manually addressed. The Docker image for CockroachDB includes them and keeps them up to date with each release of CockroachDB. -{{site.data.alerts.end}} - -## File descriptors limit - -CockroachDB can use a large number of open file descriptors, often more than is available by default. Therefore, please note the following recommendations. - -For each CockroachDB node: - -- At a **minimum**, the file descriptors limit must be `1956` (1700 per store plus 256 for networking). If the limit is below this threshold, the node will not start. -- It is **recommended** to set the file descriptors limit to `unlimited`; otherwise, the recommended limit is at least `15000` (10000 per store plus 5000 for networking). This higher limit ensures performance and accommodates cluster growth. -- When the file descriptors limit is not high enough to allocate the recommended amounts, CockroachDB allocates 10000 per store and the rest for networking; if this would result in networking getting less than 256, CockroachDB instead allocates 256 for networking and evenly splits the rest across stores. - -### Increase the file descriptors limit - - - -
    - - - -
    - -
    - -- [Yosemite and later](#yosemite-and-later) -- [Older versions](#older-versions) - -#### Yosemite and later - -To adjust the file descriptors limit for a single process in Mac OS X Yosemite and later, you must create a property list configuration file with the hard limit set to the recommendation mentioned [above](#file-descriptors-limit). Note that CockroachDB always uses the hard limit, so it's not technically necessary to adjust the soft limit, although we do so in the steps below. - -For example, for a node with 3 stores, we would set the hard limit to at least 35000 (10000 per store and 5000 for networking) as follows: - -1. Check the current limits: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ launchctl limit maxfiles - ~~~ - ~~~ - maxfiles 10240 10240 - ~~~ - - The last two columns are the soft and hard limits, respectively. If `unlimited` is listed as the hard limit, note that the hidden default limit for a single process is actually 10240. - -2. Create `/Library/LaunchDaemons/limit.maxfiles.plist` and add the following contents, with the final strings in the `ProgramArguments` array set to 35000: - - ~~~ xml - - - - - Label - limit.maxfiles - ProgramArguments - - launchctl - limit - maxfiles - 35000 - 35000 - - RunAtLoad - - ServiceIPC - - - - ~~~ - - Make sure the plist file is owned by `root:wheel` and has permissions `-rw-r--r--`. These permissions should be in place by default. - -3. Restart the system for the new limits to take effect. - -4. Check the current limits: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ launchctl limit maxfiles - ~~~ - ~~~ - maxfiles 35000 35000 - ~~~ - -#### Older versions - -To adjust the file descriptors limit for a single process in OS X versions earlier than Yosemite, edit `/etc/launchd.conf` and increase the hard limit to the recommendation mentioned [above](#file-descriptors-limit). Note that CockroachDB always uses the hard limit, so it's not technically necessary to adjust the soft limit, although we do so in the steps below. - -For example, for a node with 3 stores, we would set the hard limit to at least 35000 (10000 per store and 5000 for networking) as follows: - -1. Check the current limits: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ launchctl limit maxfiles - ~~~ - ~~~ - maxfiles 10240 10240 - ~~~ - - The last two columns are the soft and hard limits, respectively. If `unlimited` is listed as the hard limit, note that the hidden default limit for a single process is actually 10240. - -2. Edit (or create) `/etc/launchd.conf` and add a line that looks like the following, with the last value set to the new hard limit: - - ~~~ - limit maxfiles 35000 35000 - ~~~ - -3. Save the file, and restart the system for the new limits to take effect. - -4. Verify the new limits: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ launchctl limit maxfiles - ~~~ - ~~~ - maxfiles 35000 35000 - ~~~ - -
    - -
    - -- [Per-Process Limit](#per-process-limit) -- [System-Wide Limit](#system-wide-limit) - -#### Per-Process Limit - -To adjust the file descriptors limit for a single process on Linux, enable PAM user limits and set the hard limit to the recommendation mentioned [above](#file-descriptors-limit). Note that CockroachDB always uses the hard limit, so it's not technically necessary to adjust the soft limit, although we do so in the steps below. - -For example, for a node with 3 stores, we would set the hard limit to at least 35000 (10000 per store and 5000 for networking) as follows: - -1. Make sure the following line is present in both `/etc/pam.d/common-session` and `/etc/pam.d/common-session-noninteractive`: - - ~~~ shell - session required pam_limits.so - ~~~ - -2. Edit `/etc/security/limits.conf` and append the following lines to the file: - - ~~~ - * soft nofile 35000 - * hard nofile 35000 - ~~~ - - Note that `*` can be replaced with the username that will be running the CockroachDB server. - -4. Save and close the file. - -5. Restart the system for the new limits to take effect. - -6. Verify the new limits: - - ~~~ shell - $ ulimit -a - ~~~ - -Alternately, if you're using [Systemd](https://en.wikipedia.org/wiki/Systemd): - -1. Edit the service definition to configure the maximum number of open files: - - ~~~ ini - [Service] - ... - LimitNOFILE=35000 - ~~~ - - {{site.data.alerts.callout_success}} - To set the file descriptor limit to "unlimited" in the Systemd service definition file, use `LimitNOFILE=infinity`. - {{site.data.alerts.end}} - -2. Reload Systemd for the new limit to take effect: - - ~~~ shell - $ systemctl daemon-reload - ~~~ - -#### System-Wide Limit - -You should also confirm that the file descriptors limit for the entire Linux system is at least 10 times higher than the per-process limit documented above (e.g., at least 150000). - -1. Check the system-wide limit: - - ~~~ shell - $ cat /proc/sys/fs/file-max - ~~~ - -2. If necessary, increase the system-wide limit in the `proc` file system: - - ~~~ shell - $ echo 150000 > /proc/sys/fs/file-max - ~~~ - -
    -
    - -CockroachDB does not yet provide a Windows binary. Once that's available, we will also provide documentation on adjusting the file descriptors limit on Windows. - -
    - -#### Attributions - -This section, "File Descriptors Limit", is in part derivative of the chapter *Open File Limits* From the Riak LV 2.1.4 documentation, used under Creative Commons Attribution 3.0 Unported License. - -## Orchestration / Kubernetes - -When running CockroachDB on Kubernetes, making the following minimal customizations will result in better, more reliable performance: - -* Use [SSDs instead of traditional HDDs](kubernetes-performance.html#disk-type). -* Configure CPU and memory [resource requests and limits](kubernetes-performance.html#resource-requests-and-limits). - -For more information and additional customization suggestions, see our full detailed guide to [CockroachDB Performance on Kubernetes](kubernetes-performance.html). - -## Transaction Retries - -When several transactions are trying to modify the same underlying data concurrently, they may experience [contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention) that leads to [transaction retries](transactions.html#transaction-retries). In order to avoid failures in production, your application should be engineered to handle transaction retries using [client-side retry handling](transactions.html#client-side-intervention). diff --git a/src/current/v21.1/refresh.md b/src/current/v21.1/refresh.md deleted file mode 100644 index 521046dfabc..00000000000 --- a/src/current/v21.1/refresh.md +++ /dev/null @@ -1,129 +0,0 @@ ---- -title: REFRESH -summary: The REFRESH statement updates the stored query results of a materialized view. -toc: true ---- - - Stored query results in [materialized view](views.html#materialized-views) are not automatically updated to reflect the latest state of the table(s) they query. The `REFRESH` [statement](sql-statements.html) updates the stored query results of a materialized view. - -{{site.data.alerts.callout_info}} -CockroachDB does not support materialized views that are refreshed on [transaction commit](commit-transaction.html). -{{site.data.alerts.end}} - -## Required privileges - -The user must be the [owner](owner-to.html) of the materialized view or have [admin](authorization.html#admin-role) privileges. - -## Syntax - -~~~ -REFRESH MATERIALIZED VIEW [CONCURRENTLY] view_name [WITH [NO] DATA] -~~~ - -## Parameters - - Parameter | Description ------------|------------- -`CONCURRENTLY` | (*Default behavior*) This keyword is a no-op, added for PostgreSQL compatibility. All materialized views are refreshed concurrently with other jobs. -`view_name` | The name of the materialized view to refresh. -`WITH NO DATA` | Drop the query results of the materialized view from storage. -`WITH DATA` | (*Default behavior*) Refresh the stored query results. - -## Example - -The following example uses the [sample `bank` database](cockroach-workload.html#bank-workload), populated with some workload values. - -Suppose that you create a materialized view on the `bank` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE MATERIALIZED VIEW overdrawn_accounts - AS SELECT id, balance - FROM bank - WHERE balance < 0; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM overdrawn_accounts; -~~~ - -~~~ - id | balance -------+---------- - 1 | -17643 - 3 | -5928 - 13 | -3700 -... -(402 rows) -~~~ - -Now suppose that you update the `balance` values of the `bank` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE bank SET balance = 0 WHERE balance < 0; -~~~ - -~~~ -UPDATE 402 -~~~ - -The changes can be seen in the table with a simple `SELECT` statement against the table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT id, balance -FROM bank -WHERE balance < 0; -~~~ - -~~~ - id | balance ------+---------- -(0 rows) -~~~ - - -Recall that materialized views do not automatically update their stored results. Selecting from `overdrawn_accounts` returns stored results, which are outdated: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM overdrawn_accounts; -~~~ - -~~~ - id | balance -------+---------- - 1 | -17643 - 3 | -5928 - 13 | -3700 -... -(402 rows) -~~~ - -To update the materialized view's results, use a [`REFRESH`](refresh.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> REFRESH MATERIALIZED VIEW overdrawn_accounts; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM overdrawn_accounts; -~~~ - -~~~ - id | balance ------+---------- -(0 rows) -~~~ - -## See also - -- [Materialized views](views.html#materialized-views) -- [`CREATE VIEW`](create-view.html) -- [`SHOW TABLES`](show-tables.html) -- [`ALTER VIEW`](alter-view.html) -- [`DROP VIEW`](drop-view.html) diff --git a/src/current/v21.1/regional-tables.md b/src/current/v21.1/regional-tables.md deleted file mode 100644 index c42569a7d5a..00000000000 --- a/src/current/v21.1/regional-tables.md +++ /dev/null @@ -1,144 +0,0 @@ ---- -title: Regional Tables -summary: Guidance on using Regional Tables in a multi-region deployment. -toc: true ---- - -In a [multi-region deployment](multiregion-overview.html), the [Regional Table Locality Pattern](multiregion-overview.html#table-locality) is a good choice for tables with the following requirements: - -- Read and write latency must be low. -- Rows in the table, and all latency-sensitive queries, can be tied to specific regions. - -Tables with the Regional Table Locality Pattern can survive zone or region failures, depending on the database-level [survival goal](multiregion-overview.html#survival-goals) setting. - -{{site.data.alerts.callout_success}} -{% include {{page.version.version}}/misc/multiregion-max-offset.md %} -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -Regional tables (and the other [multi-region capabilities](multiregion-overview.html)) require an [Enterprise license](https://www.cockroachlabs.com/get-started-cockroachdb/). -{{site.data.alerts.end}} - -## Prerequisites - -### Fundamentals - -{% include {{ page.version.version }}/topology-patterns/multiregion-fundamentals.md %} - -### Cluster setup - -{% include {{ page.version.version }}/topology-patterns/multi-region-cluster-setup.md %} - -## Configuration - -### Summary - -To use this pattern, you tell CockroachDB to set the [table locality](multiregion-overview.html#table-locality) to either [`REGIONAL BY TABLE`](#regional-tables) or [`REGIONAL BY ROW`](#regional-by-row-tables). - -#### Regional tables - -{% include {{page.version.version}}/sql/regional-table-description.md %} - -#### Regional by row tables - -{% include {{page.version.version}}/sql/regional-by-row-table-description.md %} - -### Steps - -{{site.data.alerts.callout_info}} -By default, all tables in a multi-region database are [Regional tables](#regional-tables). Therefore, the steps below show how to set up [Regional by row tables](#regional-by-row-tables). -{{site.data.alerts.end}} - -{% include {{page.version.version}}/topology-patterns/multiregion-db-setup.md %} - -Next, create a `users` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE users ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - city STRING NOT NULL, - first_name STRING NOT NULL, - last_name STRING NOT NULL, - address STRING NOT NULL -); -~~~ - -By default, all tables in a multi-region cluster default to the [`REGIONAL BY TABLE`](#regional-tables) locality setting. To verify this, issue a [`SHOW CREATE`](show-create.html) on the `users` table you just created: - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW CREATE TABLE users; -~~~ - -~~~ - table_name | create_statement --------------+------------------------------------------------------------------ - users | CREATE TABLE public.users ( - | id UUID NOT NULL DEFAULT gen_random_uuid(), - | city STRING NOT NULL, - | first_name STRING NOT NULL, - | last_name STRING NOT NULL, - | address STRING NOT NULL, - | CONSTRAINT "primary" PRIMARY KEY (id ASC), - | FAMILY "primary" (id, city, first_name, last_name, address) - | ) LOCALITY REGIONAL BY TABLE IN PRIMARY REGION -~~~ - -Next, set the table's locality to [`REGIONAL BY ROW`](#regional-by-row-tables) using the [`ALTER TABLE ... SET LOCALITY`](set-locality.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER TABLE users SET LOCALITY REGIONAL BY ROW; -~~~ - -~~~ -NOTICE: LOCALITY changes will be finalized asynchronously; further schema changes on this table may be restricted until the job completes -ALTER TABLE SET LOCALITY -~~~ - -Now that the table is regional by row, we need to tell CockroachDB which rows need to be optimized for access from which regions. We do this by issuing [`UPDATE`](update.html) statements that modify the automatically created [`crdb_region`](set-locality.html#crdb_region) column. - -Issue the statements below to associate each row with a home region that depends on its `city` column: - -{% include_cached copy-clipboard.html %} -~~~ sql -UPDATE users SET crdb_region = 'us-central' WHERE city IN ('chicago', 'milwaukee', 'dallas'); -UPDATE users SET crdb_region = 'us-east' WHERE city IN ('washington dc', 'boston', 'new york'); -UPDATE users SET crdb_region = 'us-west' WHERE city IN ('los angeles', 'san francisco', 'seattle'); -~~~ - -By default, the region column will get auto-assigned on insert; this is also known as "auto-homing". For more information about how the `crdb_region` column works, see [`ALTER TABLE ... SET LOCALITY REGIONAL BY ROW`](set-locality.html#regional-by-row). - -{% include {{page.version.version}}/sql/locality-optimized-search.md %} - -{{site.data.alerts.callout_success}} -A good way to check that your [table locality settings](multiregion-overview.html#table-locality) are having the expected effect is by monitoring how the performance metrics of a workload change as the settings are applied to a running cluster. For a tutorial showing how table localities can improve performance metrics across a multi-region cluster, see [Low Latency Reads and Writes in a Multi-Region Cluster](demo-low-latency-multi-region-deployment.html). -{{site.data.alerts.end}} - -## Characteristics - -### Latency - -For [`REGIONAL BY TABLE` tables](#regional-tables), you get low latency for single-region writes and reads, as well as multi-region stale reads. - -For [`REGIONAL BY ROW` tables](#regional-by-row-tables), you get low-latency consistent multi-region reads & writes for rows which are homed in specific regions, and low-latency multi-region stale reads from all other regions. - -### Resiliency - -Because the `test` database does not specify a [survival goal](multiregion-overview.html#survival-goals), it uses the default [`ZONE` survival goal](multiregion-overview.html#surviving-zone-failures). With the default settings, an entire availability zone (AZ) can fail without interrupting access to the database. - -For more information about how to choose a database survival goal, see [When to use `ZONE` vs. `REGION` survival goals](when-to-use-zone-vs-region-survival-goals.html). - -## Alternatives - -- If rows in the table **cannot** be tied to specific geographies, reads must be up-to-date for business reasons or because the table is referenced by [foreign keys](foreign-key.html), and the table is rarely modified, consider the [`GLOBAL` Table Locality Pattern](global-tables.html). -- If your application can tolerate historical reads in some cases, consider the [Follower Reads](topology-follower-reads.html) pattern. - -## Tutorial - -For a step-by-step demonstration showing how CockroachDB's multi-region capabilities (including [`REGIONAL BY ROW` tables](#regional-by-row-tables)) give you low-latency reads in a distributed cluster, see the tutorial on [Low Latency Reads and Writes in a Multi-Region Cluster](demo-low-latency-multi-region-deployment.html). - -## See also - -{% include {{ page.version.version }}/topology-patterns/see-also.md %} diff --git a/src/current/v21.1/release-savepoint.md b/src/current/v21.1/release-savepoint.md deleted file mode 100644 index 584b678751b..00000000000 --- a/src/current/v21.1/release-savepoint.md +++ /dev/null @@ -1,90 +0,0 @@ ---- -title: RELEASE SAVEPOINT -summary: The RELEASE SAVEPOINT statement commits the nested transaction starting at the corresponding SAVEPOINT statement using the same savepoint name. -toc: true ---- - - The `RELEASE SAVEPOINT` statement commits the [nested transaction](transactions.html#nested-transactions) starting at the corresponding `SAVEPOINT` statement using the same savepoint name, including all its nested sub-transactions. This is in addition to continued support for working with [retry savepoints](savepoint.html#savepoints-for-client-side-transaction-retries). - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/release_savepoint.html %} -
    - -## Required privileges - -No [privileges](authorization.html#assign-privileges) are required to release a savepoint. However, privileges are required for each statement within a transaction. - -## Parameters - -Parameter | Description ---------- | ----------- -name | The name of the savepoint. [Retry savepoints](savepoint.html#savepoints-for-client-side-transaction-retries) default to using the name `cockroach_restart`, but this can be customized using a session variable. For more information, see [Customizing the retry savepoint name](savepoint.html#customizing-the-retry-savepoint-name). - -## Handling errors - -The `RELEASE SAVEPOINT` statement is invalid after the nested transaction has encountered an error. After an error, the following statements can be used: - -- [`ROLLBACK TO SAVEPOINT`](rollback-transaction.html#rollback-a-nested-transaction) to roll back to the previous savepoint. -- [`ROLLBACK` or `ABORT`](rollback-transaction.html#rollback-a-transaction) to roll back the entire surrounding transaction. -- [`COMMIT`](commit-transaction.html) to commit the entire surrounding transaction. In case of error, `COMMIT` is synonymous with [`ROLLBACK`/`ABORT`](rollback-transaction.html) and also rolls back the entire transaction. - -When a (sub-)transaction encounters a retry error, the client should repeat `ROLLBACK TO SAVEPOINT` and the statements in the transaction until the statements complete without error, then issue `RELEASE`. - -To completely remove the marker of a nested transaction after it encounters an error and begin other work in the outer transaction, use [`ROLLBACK TO SAVEPOINT`](rollback-transaction.html#rollback-a-nested-transaction) immediately followed by `RELEASE`. - -## Examples - -### Commit a nested transaction by releasing a savepoint - -{{site.data.alerts.callout_info}} -This example uses the [MovR data set](movr.html). -{{site.data.alerts.end}} - -In the example below, we roll back the inner [nested transaction](transactions.html#nested-transactions) (marked by the savepoint `lower`) and release (commit) the outer savepoint `higher`, which raises the promo code discount to 15% using CockroachDB's [JSONB functions](jsonb.html#functions). - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; -SAVEPOINT higher; -UPDATE promo_codes SET rules = jsonb_set(rules, '{value}', '"15%"') WHERE rules @> '{"type": "percent_discount"}'; -SAVEPOINT lower; -UPDATE promo_codes SET rules = jsonb_set(rules, '{value}', '"7.5%"') WHERE rules @> '{"type": "percent_discount"}'; -ROLLBACK TO SAVEPOINT lower; -RELEASE SAVEPOINT higher; -COMMIT; -~~~ - -~~~ -COMMIT -~~~ - -### Commit a transaction by releasing a retry savepoint - -{% include {{page.version.version}}/sql/retry-savepoints.md %} - -After declaring a retry savepoint, commit the transaction with `RELEASE SAVEPOINT` and then prepare the connection for the next transaction with [`COMMIT`](commit-transaction.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; -SAVEPOINT cockroach_restart; -UPDATE products SET inventory = 0 WHERE sku = '8675309'; -INSERT INTO orders (customer, sku, status) VALUES (1001, '8675309', 'new'); -RELEASE SAVEPOINT cockroach_restart; -COMMIT; -~~~ - -Applications using `SAVEPOINT` for client-side transaction retries must also include functions to execute retries with [`ROLLBACK TO SAVEPOINT `](rollback-transaction.html#retry-a-transaction). - -Note that you can [customize the retry savepoint name](savepoint.html#customizing-the-retry-savepoint-name) to something other than `cockroach_restart` with a session variable if you need to. - -## See also - -- [Transactions](transactions.html) -- [`SAVEPOINT`](savepoint.html) -- [`SHOW SAVEPOINT STATUS`](show-savepoint-status.html) -- [`ROLLBACK`](rollback-transaction.html) -- [`BEGIN`](begin-transaction.html) -- [`COMMIT`](commit-transaction.html) diff --git a/src/current/v21.1/remove-nodes.md b/src/current/v21.1/remove-nodes.md deleted file mode 100644 index f1a4568abbf..00000000000 --- a/src/current/v21.1/remove-nodes.md +++ /dev/null @@ -1,402 +0,0 @@ ---- -title: Decommission Nodes -summary: Remove one or more nodes from a CockroachDB cluster. -toc: true ---- - -This page shows you how to decommission one or more nodes. Decommissioning a node removes it from the CockroachDB cluster. - -You might do this, for example, when downsizing a cluster or reacting to hardware failures. - -{{site.data.alerts.callout_info}} -Node decommissioning should not be performed when [upgrading your cluster's version of CockroachDB](upgrade-cockroach-version.html) or performing planned maintenance (e.g., upgrading system software). In these scenarios, you will want to temporarily [stop the node](cockroach-quit.html) and restart it later. -{{site.data.alerts.end}} - -## Overview - -### How it works - -A node is considered to be decommissioned when it meets two criteria: - -1. The node has completed the decommissioning process. -2. The node has been stopped and has not [updated its liveness record](architecture/replication-layer.html#epoch-based-leases-table-data) for the duration configured via [`server.time_until_store_dead`](cluster-settings.html), which defaults to 5 minutes. - -The decommissioning process transfers all range replicas on the node to other nodes. During and after this process, the node is considered "decommissioning" and continues to accept new SQL connections. Even without replicas, the node can still function as a gateway to route connections to relevant data. For this reason, the [`/health?ready=1` monitoring endpoint](monitoring-and-alerting.html#health-ready-1) continues to consider the node "ready" so load balancers can continue directing traffic to the node. - -After all range replicas have been transferred, a graceful shutdown is initiated by sending `SIGTERM`, during which the node is drained of SQL clients, [distributed SQL](architecture/sql-layer.html#distsql) queries, and range leases. Meanwhile, the [`/health?ready=1` monitoring endpoint](monitoring-and-alerting.html#health-ready-1) starts returning a `503 Service Unavailable` status response code so that load balancers stop directing traffic to the node. Once draining completes and the process is terminated, the node stops updating its liveness record, and after the duration configured via [`server.time_until_store_dead`](cluster-settings.html) is considered to be decommissioned. - -You can [check the status of node decommissioning](#check-the-status-of-decommissioning-nodes) with the CLI. - -### Considerations - -Before decommissioning a node, make sure other nodes are available to take over the range replicas from the node. If no other nodes are available, the decommissioning process will hang indefinitely. See the [Examples](#examples) below for more details. - -### Examples - -#### 3-node cluster with 3-way replication - -In this scenario, each range is replicated 3 times, with each replica on a different node: - -
    Decommission Scenario 1
    - -If you try to decommission a node, the process will hang indefinitely because the cluster cannot move the decommissioning node's replicas to the other 2 nodes, which already have a replica of each range: - -
    Decommission Scenario 1
    - -To successfully decommission a node in this cluster, you need to add a 4th node. The decommissioning process can then complete: - -
    Decommission Scenario 1
    - -#### 5-node cluster with 3-way replication - -In this scenario, like in the scenario above, each range is replicated 3 times, with each replica on a different node: - -
    Decommission Scenario 1
    - -If you decommission a node, the process will run successfully because the cluster will be able to move the node's replicas to other nodes without doubling up any range replicas: - -
    Decommission Scenario 1
    - -#### 5-node cluster with 5-way replication for a specific table - -In this scenario, a [custom replication zone](configure-replication-zones.html#create-a-replication-zone-for-a-table) has been set to replicate a specific table 5 times (range 6), while all other data is replicated 3 times: - -
    Decommission Scenario 1
    - -If you try to decommission a node, the cluster will successfully rebalance all ranges but range 6. Since range 6 requires 5 replicas (based on the table-specific replication zone), and since CockroachDB will not allow more than a single replica of any range on a single node, the decommissioning process will hang indefinitely: - -
    Decommission Scenario 1
    - -To successfully decommission a node in this cluster, you need to add a 6th node. The decommissioning process can then complete: - -
    Decommission Scenario 1
    - -## Remove a single node (live) - -
    - - -
    - -### Before decommissioning a node - -To ensure your cluster can adequately handle decommissioning nodes: - -- Before decommissioning each node verify that there are no [underreplicated or unavailable ranges](ui-replication-dashboard.html). -- If you have a decommissioning node that appears to be hung, you can [recommission](#recommission-nodes) the node. If you notice any issues persisting, [contact our support team](support-resources.html). - - If possible, keep the node running instead of stopping it, because a hung decommissioning process might be a symptom of a problem that could result in data loss. -- Confirm that there are enough nodes to take over the replicas from the node you want to remove. See some [Example scenarios](#examples) above. - -### Step 1. Check the node before decommissioning - -Open the DB Console, click **Metrics** on the left, select the **Replication** dashboard, and hover over the **Replicas per Store** and **Leaseholders per Store** graphs: - -
    Decommission a single live node
    - -
    Decommission a single live node
    - -### Step 2. Start the decommissioning process on the node - -Run the [`cockroach node decommission`](cockroach-node.html) command against the address of the node to decommission: - -
    -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach node decommission --self --certs-dir=certs --host=
    -~~~ -
    - -
    -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach node decommission --self --insecure --host=
    -~~~ -
    - -You'll then see the decommissioning status print to `stderr` as it changes: - -~~~ - id | is_live | replicas | is_decommissioning | is_draining -+---+---------+----------+--------------------+-------------+ - 4 | true | 45 | true | false -(1 row) -~~~ - -Once the node has completed the decommissioning process, you'll see a confirmation: - -~~~ - id | is_live | replicas | is_decommissioning | is_draining -+---+---------+----------+--------------------+-------------+ - 4 | true | 0 | true | false -(1 row) - -No more data reported on target nodes. Please verify cluster health before removing the nodes. -~~~ - -Note that `is_decommissioning` will remain `true` after all replicas have been transferred from the node. - -### Step 3. Check the node and cluster after the decommissioning process - -In the DB Console **Replication** dashboard, again hover over the **Replicas per Store** and **Leaseholders per Store** graphs. For the decommissioning node, the counts should be 0: - -
    Decommission a single live node
    - -
    Decommission a single live node
    - -Return to the [**Node List**](ui-cluster-overview-page.html#node-list) on the Overview page. The `DECOMMISSIONING` node should have 0 replicas, and all other nodes should be healthy (`LIVE`): - -
    Decommission a single live node
    - -{{site.data.alerts.callout_success}} -Even with zero replicas on a node, its [status](ui-cluster-overview-page.html#node-status) on the Node List will be `DECOMMISSIONING` until you stop the node. It is also counted as a "Suspect" node in the [Cluster Overview panel](ui-cluster-overview-page.html#cluster-overview-panel) until being shut down. -{{site.data.alerts.end}} - -### Step 4. Stop the decommissioning node - -Drain and stop the node using one of the following methods: - -{% include {{ page.version.version }}/prod-deployment/node-shutdown.md %} - -After the duration configured via [`server.time_until_store_dead`](cluster-settings.html), you'll see the stopped node listed under **Recently Decommissioned Nodes**: - -
    Decommission a single live node
    - -At this point, the node is `DECOMMISSIONED` and will no longer appear in timeseries graphs unless you view a time range during which the node was live. However, it will never disappear from the historical list of decommissioned nodes, linked beneath Recently Decommissioned Nodes. - -## Remove a single node (dead) - -
    - - -
    - -After a node has been dead for 5 minutes, CockroachDB transfers the range replicas and range leases on the node to available live nodes. However, if the dead node is restarted, the cluster will rebalance replicas and leases to the node. - -To prevent the cluster from rebalancing data to a dead node if it comes back online, do the following: - -{{site.data.alerts.callout_success}} -You can check that a node is dead and find its internal ID by either running [`cockroach node status`](cockroach-node.html) or opening the DB Console and scrolling to the [**Node List**](ui-cluster-overview-page.html#node-list) on the **Overview** page. -{{site.data.alerts.end}} - -### Step 1. Mark the dead node as decommissioned - -Run the [`cockroach node decommission`](cockroach-node.html) command against the address of any live node, specifying the ID of the dead node: - -
    -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach node decommission --certs-dir=certs --host=
    -~~~ -
    - -
    -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach node decommission --insecure --host=
    -~~~ -
    - -~~~ - id | is_live | replicas | is_decommissioning | is_draining -+---+---------+----------+--------------------+-------------+ - 4 | false | 0 | true | true -(1 row) - -No more data reported on target nodes. Please verify cluster health before removing the nodes. -~~~ - -Within 5 minutes, you'll see the node move from the Node List to the **Recently Decommissioned Nodes** list in the DB Console, with its status changed from `DEAD` to `DECOMMISSIONED`. - -
    Decommission a single live node
    - -At this point, the node is `DECOMMISSIONED` and will no longer appear in timeseries graphs unless you view a time range during which the node was live. However, it will never disappear from the historical list of decommissioned nodes, linked beneath Recently Decommissioned Nodes. - -## Remove multiple nodes - -
    - - -
    - -### Before decommissioning nodes - -- Before decommissioning each node verify that there are no [underreplicated or unavailable ranges](ui-replication-dashboard.html). -- If you have a decommissioning node that appears to be hung, you can [recommission](#recommission-nodes) the node. If you notice any issues persisting, [contact our support team](support-resources.html). - - If possible, keep the node running instead of stopping it, because a hung decommissioning process might be a symptom of a problem that could result in data loss. -- Confirm that there are enough nodes to take over the replicas from the node you want to remove. See some [Example scenarios](#examples) above. - -### Step 1. Check the nodes before decommissioning - -Open the DB Console, click **Metrics** on the left, select the **Replication** dashboard, and hover over the **Replicas per Store** and **Leaseholders per Store** graphs: - -
    Decommission multiple nodes
    - -
    Decommission multiple nodes
    - -### Step 2. Start the decommissioning process on the nodes - -Run the [`cockroach node decommission`](cockroach-node.html) command against the address of each node to decommission: - -
    -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach node decommission --self --certs-dir=certs --host=
    -~~~ - -~~~ shell -$ cockroach node decommission --self --certs-dir=certs --host=
    -~~~ -
    - -
    -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach node decommission --self --insecure --host=
    -~~~ - -~~~ shell -$ cockroach node decommission --self --insecure --host=
    -~~~ -
    - -You'll then see the decommissioning status print to `stderr` as it changes: - -~~~ - id | is_live | replicas | is_decommissioning | is_draining -+---+---------+----------+--------------------+-------------+ - 4 | true | 18 | true | false - 5 | true | 16 | true | false -(2 rows) -~~~ - -Once the nodes have been fully decommissioned, you'll see a confirmation: - -~~~ - id | is_live | replicas | is_decommissioning | is_draining -+---+---------+----------+--------------------+-------------+ - 4 | true | 0 | true | false - 5 | true | 0 | true | false -(2 rows) - -No more data reported on target nodes. Please verify cluster health before removing the nodes. -~~~ - -Note that `is_decommissioning` will remain `true` after all replicas have been transferred from each node. - -### Step 3. Check the nodes and cluster after the decommissioning process - -In the DB Console **Replication** dashboard, again hover over the **Replicas per Store** and **Leaseholders per Store** graphs. For the decommissioning nodes, the counts should be 0: - -
    Decommission multiple nodes
    - -
    Decommission multiple nodes
    - -Return to the [**Node List**](ui-cluster-overview-page.html#node-list) on the Overview page. The `DECOMMISSIONING` nodes should each have 0 replicas, and all other nodes should be healthy (`LIVE`): - -
    Decommission multiple nodes
    - -{{site.data.alerts.callout_success}} -Even with zero replicas on a node, its [status](ui-cluster-overview-page.html#node-status) on the Node List will be `DECOMMISSIONING` until you stop the node. It is also counted as a "Suspect" node in the [Cluster Overview panel](ui-cluster-overview-page.html#cluster-overview-panel) until being shut down. -{{site.data.alerts.end}} - -### Step 4. Stop the decommissioning nodes - -Drain and stop each node using one of the following methods: - -{% include {{ page.version.version }}/prod-deployment/node-shutdown.md %} - -After the duration configured via [`server.time_until_store_dead`](cluster-settings.html), you'll see the stopped nodes listed under **Recently Decommissioned Nodes**: - -
    Decommission multiple nodes
    - -At this point, the nodes are `DECOMMISSIONED` and will no longer appear in timeseries graphs unless you view a time range during which the nodes were live. However, they will never disappear from the historical list of decommissioned nodes, linked beneath Recently Decommissioned Nodes. - -## Recommission nodes - -
    - - -
    - -If you accidentally started decommissioning a node, or have a node with a hung decommissioning process, you can recommission the node. This cancels the process of transferring replicas on the node to other nodes. - -{{site.data.alerts.callout_info}} -Recommissioning is intended to cancel an active decommissioning process. If all ranges have been removed from a node, you must start a new node. As of v20.2, a fully decommissioned node is permanently decommissioned, and cannot be recommissioned. -{{site.data.alerts.end}} - -### Step 1. Cancel the decommissioning process - -Press `ctrl-c` in each terminal with an ongoing decommissioning process that you want to cancel. - -### Step 2. Recommission the decommissioning nodes - -Execute the [`cockroach node recommission`](cockroach-node.html) command against the address of the node to recommission: - -
    -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach node recommission --self --certs-dir=certs --host=
    -~~~ -
    - -
    -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach node recommission --self --insecure --host=
    -~~~ -
    - -The value of `is_decommissioning` will change back to `false`: - -~~~ - id | is_live | replicas | is_decommissioning | is_draining -+---+---------+----------+--------------------+-------------+ - 4 | false | 0 | false | false -(1 row) -~~~ - -On the [**Node List**](ui-cluster-overview-page.html#node-list), you should soon see the recommissioned nodes listed as `LIVE`. After a few minutes, you should see replicas rebalanced to the nodes. - -## Check the status of decommissioning nodes - -To check the progress of decommissioning nodes, run the [`cockroach node status`](cockroach-node.html) command with the `--decommission` flag: - -
    - - -

    - -
    -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach node status --decommission --certs-dir=certs --host=
    -~~~ -
    - -
    -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach node status --decommission --insecure --host=
    -~~~ -
    - -~~~ - id | address | build | started_at | updated_at | is_available | is_live | gossiped_replicas | is_decommissioning | is_draining -+---+------------------------+---------+----------------------------------+----------------------------------+--------------+---------+-------------------+--------------------+-------------+ - 1 | 165.227.60.76:26257 | 91a299d | 2018-10-01 16:53:10.946245+00:00 | 2018-10-02 14:04:39.280249+00:00 | true | true | 26 | false | false - 2 | 192.241.239.201:26257 | 91a299d | 2018-10-01 16:53:24.22346+00:00 | 2018-10-02 14:04:39.415235+00:00 | true | true | 26 | false | false - 3 | 67.207.91.36:26257 | 91a299d | 2018-10-01 17:34:21.041926+00:00 | 2018-10-02 14:04:39.233882+00:00 | true | true | 25 | false | false - 4 | 138.197.12.74:26257 | 91a299d | 2018-10-01 17:09:11.734093+00:00 | 2018-10-02 14:04:37.558204+00:00 | true | true | 25 | false | false - 5 | 174.138.50.192:26257 | 91a299d | 2018-10-01 17:14:01.480725+00:00 | 2018-10-02 14:04:39.293121+00:00 | true | true | 0 | true | false -(5 rows) -~~~ - -- `is_decommissioning == true` implies that replicas are being or have been transferred to other nodes. The node is either undergoing or has completed the decommissioning process. -- `is_draining == true` implies that the node is no longer accepting SQL connections. The node is either in the process of shutting down or has already done so. - -## See also - -- [`cockroach start`](cockroach-start.html) -- [Node status](ui-cluster-overview-page.html#node-status) diff --git a/src/current/v21.1/rename-column.md b/src/current/v21.1/rename-column.md deleted file mode 100644 index a5bb4aba4f3..00000000000 --- a/src/current/v21.1/rename-column.md +++ /dev/null @@ -1,118 +0,0 @@ ---- -title: RENAME COLUMN -summary: The RENAME COLUMN statement changes the name of a column in a table. -toc: true ---- - -The `RENAME COLUMN` [statement](sql-statements.html) changes the name of a column in a table. - -{{site.data.alerts.callout_info}} -It is not possible to rename a column referenced by a view. For more details, see [View Dependencies](views.html#view-dependencies). -{{site.data.alerts.end}} - -{% include {{ page.version.version }}/sql/combine-alter-table-commands.md %} - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/rename_column.html %} -
    - -## Required privileges - -The user must have the `CREATE` [privilege](authorization.html#assign-privileges) on the table. - -## Parameters - - Parameter | Description ------------|------------- - `IF EXISTS` | Rename the column only if a table of `table_name` exists; if one does not exist, do not return an error. - `table_name` | The name of the table with the column you want to use. - `current_name` | The current name of the column. - `name` | The [`name`](sql-grammar.html#name) you want to use for the column, which must be unique to its table and follow these [identifier rules](keywords-and-identifiers.html#identifiers). - -## Viewing schema changes - -{% include {{ page.version.version }}/misc/schema-change-view-job.md %} - -## Examples - -### Rename a column - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE users ( - id INT PRIMARY KEY, - first_name STRING, - family_name STRING - ); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE users RENAME COLUMN family_name TO last_name; -~~~ - -~~~ - table_name | create_statement -+------------+--------------------------------------------------+ - users | CREATE TABLE users ( - | id INT8 NOT NULL, - | first_name STRING NULL, - | last_name STRING NULL, - | CONSTRAINT "primary" PRIMARY KEY (id ASC), - | FAMILY "primary" (id, first_name, last_name) - | ) -(1 row) -~~~ - -### Add and rename columns atomically - -Some subcommands can be used in combination in a single [`ALTER TABLE`](alter-table.html) statement. For example, let's say you create a `users` table with 2 columns, an `id` column for the primary key and a `name` column for each user's last name: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE users ( - id INT PRIMARY KEY, - name STRING - ); -~~~ - -Then you decide you want distinct columns for each user's first name, last name, and full name, so you execute a single `ALTER TABLE` statement renaming `name` to `last_name`, adding `first_name`, and adding a [computed column](computed-columns.html) called `name` that concatenates `first_name` and `last_name`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE users - RENAME COLUMN name TO last_name, - ADD COLUMN first_name STRING, - ADD COLUMN name STRING - AS (CONCAT(first_name, ' ', last_name)) STORED; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE users; -~~~ - -~~~ - table_name | create_statement -+------------+----------------------------------------------------------------------+ - users | CREATE TABLE users ( - | id INT8 NOT NULL, - | last_name STRING NULL, - | first_name STRING NULL, - | name STRING NULL AS (concat(first_name, ' ', last_name)) STORED, - | CONSTRAINT "primary" PRIMARY KEY (id ASC), - | FAMILY "primary" (id, last_name, first_name, name) - | ) -(1 row) -~~~ - -## See also - -- [`ALTER TABLE`](alter-table.html) -- [`ADD CONSTRAINT`](add-constraint.html) -- [`RENAME DATABASE`](rename-database.html) -- [`RENAME TABLE`](rename-table.html) -- [`RENAME CONSTRAINT`](rename-constraint.html) -- [`SHOW JOBS`](show-jobs.html) diff --git a/src/current/v21.1/rename-constraint.md b/src/current/v21.1/rename-constraint.md deleted file mode 100644 index 8613c5809ad..00000000000 --- a/src/current/v21.1/rename-constraint.md +++ /dev/null @@ -1,92 +0,0 @@ ---- -title: RENAME CONSTRAINT -summary: The RENAME CONSTRAINT statement changes the name of a constraint on a column. -toc: true ---- - -The `RENAME CONSTRAINT` [statement](sql-statements.html) changes the name of a constraint on a column. - -{{site.data.alerts.callout_info}} -It is not possible to rename a constraint for a column referenced by a view. For more details, see [View Dependencies](views.html#view-dependencies). -{{site.data.alerts.end}} - -{% include {{ page.version.version }}/sql/combine-alter-table-commands.md %} - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/rename_constraint.html %} -
    - -## Required privileges - -The user must have the `CREATE` [privilege](authorization.html#assign-privileges) on the table. - -## Parameters - - Parameter | Description ------------|------------- - `IF EXISTS` | Rename the constraint only if a constraint of `current_name` exists; if one does not exist, do not return an error. - `table_name` | The name of the table with the constraint you want to rename. - `current_name` | The current name of the constraint. - `name` | The new [`name`](sql-grammar.html#name) you want to use for the constraint, which must be unique to its table and follow these [identifier rules](keywords-and-identifiers.html#identifiers). - -## Viewing schema changes - -{% include {{ page.version.version }}/misc/schema-change-view-job.md %} - -## Example - -### Rename a constraint - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE logon ( - login_id INT PRIMARY KEY, - customer_id INT NOT NULL, - sales_id INT, - UNIQUE (customer_id, sales_id) - ); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CONSTRAINTS FROM logon; -~~~ - -~~~ - table_name | constraint_name | constraint_type | details | validated -+------------+--------------------------------+-----------------+----------------------------------------+-----------+ - logon | logon_customer_id_sales_id_key | UNIQUE | UNIQUE (customer_id ASC, sales_id ASC) | true - logon | primary | PRIMARY KEY | PRIMARY KEY (login_id ASC) | true -(2 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE logon RENAME CONSTRAINT logon_customer_id_sales_id_key TO unique_customer_id_sales_id; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CONSTRAINTS FROM logon; -~~~ - -~~~ - table_name | constraint_name | constraint_type | details | validated -+------------+-----------------------------+-----------------+----------------------------------------+-----------+ - logon | primary | PRIMARY KEY | PRIMARY KEY (login_id ASC) | true - logon | unique_customer_id_sales_id | UNIQUE | UNIQUE (customer_id ASC, sales_id ASC) | true -(2 rows) -~~~ - -## See also - -- [`ALTER TABLE`](alter-table.html) -- [`ADD CONSTRAINT`](add-constraint.html) -- [`SHOW CONSTRAINTS`](show-constraints.html) -- [`DROP CONSTRAINT`](drop-constraint.html) -- [`VALIDATE CONSTRAINT`](validate-constraint.html) -- [`RENAME DATABASE`](rename-database.html) -- [`RENAME TABLE`](rename-table.html) -- [`RENAME COLUMN`](rename-column.html) diff --git a/src/current/v21.1/rename-database.md b/src/current/v21.1/rename-database.md deleted file mode 100644 index a7107137ab9..00000000000 --- a/src/current/v21.1/rename-database.md +++ /dev/null @@ -1,99 +0,0 @@ ---- -title: ALTER DATABASE ... RENAME TO -summary: The ALTER DATABASE ... RENAME TO statement changes the name of a database. -toc: true ---- - -The `RENAME TO` clause is part of [`ALTER DATABASE`](alter-database.html), and changes the name of a database. - -{% include {{ page.version.version }}/misc/schema-change-stmt-note.md %} - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/rename_database.html %} -
    - -## Required privileges - -To rename a database, the user must be a member of the `admin` role or must have the [`CREATEDB`](create-role.html#create-a-role-that-can-create-and-rename-databases) parameter set. - -## Parameters - -Parameter | Description -----------|------------ -`name` | The first instance of `name` is the current name of the database. The second instance is the new name for the database. The new name [must be unique](#rename-fails-new-name-already-in-use) and follow these [identifier rules](keywords-and-identifiers.html#identifiers). You cannot rename a database if it is set as the [current database](sql-name-resolution.html#current-database) or if [`sql_safe_updates = true`](set-vars.html). - -## Viewing schema changes - -{% include {{ page.version.version }}/misc/schema-change-view-job.md %} - -## Limitations - -- It is not possible to rename a database if the database is referenced by a [view](views.html). For more details, see [View Dependencies](views.html#view-dependencies). - -## Examples - -### Rename a database - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE DATABASE db1; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW DATABASES; -~~~ - -~~~ - database_name ------------------ - db1 - defaultdb - movr - postgres - system -(5 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER DATABASE db1 RENAME TO db2; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW DATABASES; -~~~ - -~~~ - database_name ------------------ - db2 - defaultdb - movr - postgres - system -(5 rows) -~~~ - -### Rename fails (new name already in use) - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER DATABASE db2 RENAME TO movr; -~~~ - -~~~ -ERROR: the new database name "movr" already exists -SQLSTATE: 42P04 -~~~ - -## See also - -- [`CREATE DATABASE`](create-database.html) -- [`SHOW DATABASES`](show-databases.html) -- [`SET DATABASE`](set-vars.html) -- [`DROP DATABASE`](drop-database.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/rename-index.md b/src/current/v21.1/rename-index.md deleted file mode 100644 index 22adce7ef68..00000000000 --- a/src/current/v21.1/rename-index.md +++ /dev/null @@ -1,80 +0,0 @@ ---- -title: RENAME INDEX -summary: The RENAME INDEX statement changes the name of an index for a table. -toc: true ---- - -The `RENAME INDEX` [statement](sql-statements.html) changes the name of an index for a table. - -{{site.data.alerts.callout_info}}It is not possible to rename an index referenced by a view. For more details, see View Dependencies.{{site.data.alerts.end}} - -{% include {{ page.version.version }}/misc/schema-change-view-job.md %} - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/rename_index.html %} -
    - -## Required privileges - -The user must have the `CREATE` [privilege](authorization.html#assign-privileges) on the table. - -## Parameters - - Parameter | Description ------------|------------- - `IF EXISTS` | Rename the index only if an index `current_name` exists; if one does not exist, do not return an error. - `table_name` | The name of the table with the index you want to use - `index_name` | The current name of the index - `name` | The [`name`](sql-grammar.html#name) you want to use for the index, which must be unique to its table and follow these [identifier rules](keywords-and-identifiers.html#identifiers). - -## Example - -### Rename an Index - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW INDEXES FROM users; -~~~ - -~~~ -+------------+------------+------------+--------------+-------------+-----------+---------+----------+ -| table_name | index_name | non_unique | seq_in_index | column_name | direction | storing | implicit | -+------------+------------+------------+--------------+-------------+-----------+---------+----------+ -| users | primary | false | 1 | id | ASC | false | false | -| users | name_idx | true | 1 | name | ASC | false | false | -| users | name_idx | true | 2 | id | ASC | false | true | -+------------+------------+------------+--------------+-------------+-----------+---------+----------+ -(3 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER INDEX users@name_idx RENAME TO users_name_idx; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW INDEXES FROM users; -~~~ - -~~~ -+------------+----------------+------------+--------------+-------------+-----------+---------+----------+ -| table_name | index_name | non_unique | seq_in_index | column_name | direction | storing | implicit | -+------------+----------------+------------+--------------+-------------+-----------+---------+----------+ -| users | primary | false | 1 | id | ASC | false | false | -| users | users_name_idx | true | 1 | name | ASC | false | false | -| users | users_name_idx | true | 2 | id | ASC | false | true | -+------------+----------------+------------+--------------+-------------+-----------+---------+----------+ -(3 rows) -~~~ - -## See also - -- [Indexes](indexes.html) -- [`CREATE INDEX`](create-index.html) -- [`RENAME COLUMN`](rename-column.html) -- [`RENAME DATABASE`](rename-database.html) -- [`RENAME TABLE`](rename-table.html) -- [`SHOW JOBS`](show-jobs.html) diff --git a/src/current/v21.1/rename-table.md b/src/current/v21.1/rename-table.md deleted file mode 100644 index 2bea857aaeb..00000000000 --- a/src/current/v21.1/rename-table.md +++ /dev/null @@ -1,167 +0,0 @@ ---- -title: ALTER TABLE ... RENAME TO -summary: The ALTER TABLE ... RENAME TO statement changes the name of a table. -toc: true ---- - -The `RENAME TO` [statement](sql-statements.html) is part of [`ALTER TABLE`](alter-table.html), and changes the name of a table. - -{{site.data.alerts.callout_info}} -`ALTER TABLE ... RENAME TO` can be used to move a table from one database to another, but it cannot be used to move a table from one schema to another. To change a table's schema, use [`SET SCHEMA`](set-schema.html). - -Note that, in a future release, `ALTER TABLE ... RENAME TO` will be limited to changing the name of a table, and will not have to the ability to change a table's database. -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -It is not possible to rename a table referenced by a view. For more details, see View Dependencies. -{{site.data.alerts.end}} - -{% include {{{ page.version.version }}/misc/schema-change-stmt-note.md %} - -## Required privileges - -The user must have the `DROP` [privilege](authorization.html#assign-privileges) on the table and the `CREATE` on the parent database. When moving a table from one database to another, the user must have the `CREATE` privilege on both the source and target databases. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/rename_table.html %} -
    - -## Parameters - - Parameter | Description ------------|------------- - `IF EXISTS` | Rename the table only if a table with the current name exists; if one does not exist, do not return an error. - `current_name` | The current name of the table. - `new_name` | The new name of the table, which must be unique within its database and follow these [identifier rules](keywords-and-identifiers.html#identifiers). When the parent database is not set as the default, the name must be formatted as `database.name`.

    The [`UPSERT`](upsert.html) and [`INSERT ON CONFLICT`](insert.html) statements use a temporary table called `excluded` to handle uniqueness conflicts during execution. It's therefore not recommended to use the name `excluded` for any of your tables. - -## Viewing schema changes - -{% include {{ page.version.version }}/misc/schema-change-view-job.md %} - -## Examples - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### Rename a table - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES; -~~~ - -~~~ - schema_name | table_name | type | estimated_row_count ---------------+----------------------------+-------+---------------------- - public | promo_codes | table | 1000 - public | rides | table | 500 - public | user_promo_codes | table | 0 - public | users | table | 50 - public | vehicle_location_histories | table | 1000 - public | vehicles | table | 15 -(6 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE users RENAME TO riders; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES; -~~~ - -~~~ - schema_name | table_name | type | estimated_row_count ---------------+----------------------------+-------+---------------------- - public | promo_codes | table | 1000 - public | rides | table | 500 - public | user_promo_codes | table | 0 - public | riders | table | 50 - public | vehicle_location_histories | table | 1000 - public | vehicles | table | 15 -(6 rows) -~~~ - -To avoid an error in case the table does not exist, you can include `IF EXISTS`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE IF EXISTS customers RENAME TO clients; -~~~ - -### Move a table - -To move a table from one database to another, use the above syntax but specify the source database after `ALTER TABLE` and the target database after `RENAME TO`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM movr; -~~~ - -~~~ - schema_name | table_name | type | estimated_row_count ---------------+----------------------------+-------+---------------------- - public | promo_codes | table | 1000 - public | rides | table | 500 - public | user_promo_codes | table | 0 - public | riders | table | 50 - public | vehicle_location_histories | table | 1000 - public | vehicles | table | 15 -(6 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM defaultdb; -~~~ - -~~~ - schema_name | table_name | type | estimated_row_count ---------------+------------+------+---------------------- -(0 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE movr.promo_codes RENAME TO defaultdb.promos; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM movr; -~~~ - -~~~ - schema_name | table_name | type | estimated_row_count ---------------+----------------------------+-------+---------------------- - public | rides | table | 500 - public | user_promo_codes | table | 0 - public | riders | table | 50 - public | vehicle_location_histories | table | 1000 - public | vehicles | table | 15 -(5 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM defaultdb; -~~~ - -~~~ - schema_name | table_name | type | estimated_row_count ---------------+------------+-------+---------------------- - public | promos | table | 1000 -(1 row) -~~~ - -## See also - -- [`CREATE TABLE`](create-table.html) -- [`ALTER TABLE`](alter-table.html) -- [`SHOW TABLES`](show-tables.html) -- [`DROP TABLE`](drop-table.html) -- [`SHOW JOBS`](show-jobs.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/reset-cluster-setting.md b/src/current/v21.1/reset-cluster-setting.md deleted file mode 100644 index c713b303607..00000000000 --- a/src/current/v21.1/reset-cluster-setting.md +++ /dev/null @@ -1,72 +0,0 @@ ---- -title: RESET CLUSTER SETTING -summary: The RESET CLUSTER SETTING statement resets a cluster setting to its default value for the client session. -toc: true ---- - -The `RESET` [statement](sql-statements.html) resets a [cluster setting](set-cluster-setting.html) to its default value for the client session.. - - -## Required privileges - -Only members of the `admin` role can modify cluster settings. By default, the `root` user belongs to the `admin` role. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/reset_csetting.html %} -
    - -## Parameters - - Parameter | Description ------------|------------- - `var_name` | The name of the [cluster setting](cluster-settings.html) (case-insensitive). - -## Example - -{{site.data.alerts.callout_success}}You can use SET CLUSTER SETTING .. TO DEFAULT to reset a cluster setting as well.{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET CLUSTER SETTING sql.metrics.statement_details.enabled = false; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CLUSTER SETTING sql.metrics.statement_details.enabled; -~~~ - -~~~ -+---------------------------------------+ -| sql.metrics.statement_details.enabled | -+---------------------------------------+ -| false | -+---------------------------------------+ -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESET CLUSTER SETTING sql.metrics.statement_details.enabled; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CLUSTER SETTING sql.metrics.statement_details.enabled; -~~~ - -~~~ -+---------------------------------------+ -| sql.metrics.statement_details.enabled | -+---------------------------------------+ -| true | -+---------------------------------------+ -(1 row) -~~~ - -## See also - -- [`SET CLUSTER SETTING`](set-cluster-setting.html) -- [`SHOW CLUSTER SETTING`](show-cluster-setting.html) -- [Cluster settings](cluster-settings.html) diff --git a/src/current/v21.1/reset-vars.md b/src/current/v21.1/reset-vars.md deleted file mode 100644 index a7b65bbb4d2..00000000000 --- a/src/current/v21.1/reset-vars.md +++ /dev/null @@ -1,89 +0,0 @@ ---- -title: RESET (session variable) -summary: The SET statement resets a session variable to its default value. -toc: true ---- - -The `RESET` [statement](sql-statements.html) resets a [session variable](set-vars.html) to its default value for the client session. - - -## Required privileges - -No [privileges](authorization.html#assign-privileges) are required to reset a session setting. - -## Synopsis - -
    {% include {{ page.version.version }}/sql/generated/diagrams/reset_session.html %}
    - -## Parameters - - Parameter | Description ------------|------------- - `session_var` | The name of the [session variable](set-vars.html#supported-variables). - -## Example - -{{site.data.alerts.callout_success}}You can use SET .. TO DEFAULT to reset a session variable as well.{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET extra_float_digits = -10; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW extra_float_digits; -~~~ - -~~~ - extra_float_digits --------------------- - -10 -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT random(); -~~~ - -~~~ - random ---------- - 0.20286 -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESET extra_float_digits; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW extra_float_digits; -~~~ - -~~~ - extra_float_digits --------------------- - 0 -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT random(); -~~~ - -~~~ - random -------------------- - 0.561354028296755 -(1 row) -~~~ - -## See also - -- [`SET` (session variable)](set-vars.html) -- [`SHOW` (session variables)](show-vars.html) diff --git a/src/current/v21.1/restore.md b/src/current/v21.1/restore.md deleted file mode 100644 index f7b49996964..00000000000 --- a/src/current/v21.1/restore.md +++ /dev/null @@ -1,421 +0,0 @@ ---- -title: RESTORE -summary: Restore your CockroachDB cluster to a cloud storage services such as AWS S3, Google Cloud Storage, or other NFS. -toc: true ---- - -The `RESTORE` [statement](sql-statements.html) restores your cluster's schemas and data from [a `BACKUP`](backup.html) stored on services such as AWS S3, Google Cloud Storage, or NFS. - -Because CockroachDB is designed with high fault tolerance, restores are designed primarily for disaster recovery, i.e., restarting your cluster if it loses a majority of its nodes. Isolated issues (such as small-scale node outages) do not require any intervention. - -You can restore: - -- [A full cluster](#full-cluster) -- [Databases](#databases) -- [Tables](#tables) - -{{site.data.alerts.callout_info}} -The [`BACKUP ... TO`](../v20.2/backup.html) and [`RESTORE ... FROM`](../v20.2/restore.html) syntax is **deprecated** as of v22.1 and will be removed in a future release. - -We recommend using the `BACKUP ... INTO {collection}` syntax, which creates or adds to a [backup collection](take-full-and-incremental-backups.html#backup-collections) in your storage location. For restoring backups, we recommend using `RESTORE FROM {backup} IN {collection}` with `{backup}` being [`LATEST`](#restore-the-most-recent-backup) or a specific [subdirectory](#subdir-param). - -For guidance on the syntax for backups and restores, see the [`BACKUP`](backup.html#examples) and [`RESTORE`](restore.html#examples) examples. -{{site.data.alerts.end}} - -## Considerations - -- `RESTORE` cannot restore backups made by newer versions of CockroachDB. -- `RESTORE` is a blocking statement. To run a restore job asynchronously, use the `DETACHED` option. See [Options](#options) for more usage detail. -- `RESTORE` no longer requires an Enterprise license, regardless of the options passed to it or to the backup it is restoring. -- You cannot restore a backup of a multi-region database into a single-region database. - -## Required privileges - -- [Full cluster restores](#full-cluster) can only be run by members of the [`ADMIN` role](authorization.html#admin-role). By default, the `root` user belongs to the `admin` role. -- For all other restores, the user must have [write access](authorization.html#assign-privileges) (`CREATE` or `INSERT`) on all objects affected. -- [Zone configurations](configure-zone.html) present on the destination cluster prior to a restore will be **overwritten** during a [cluster restore](restore.html#full-cluster) with the zone configurations from the [backed up cluster](backup.html#backup-a-cluster). If there were no customized zone configurations on the cluster when the backup was taken, then after the restore the destination cluster will use the zone configuration from the [`RANGE DEFAULT` configuration](configure-replication-zones.html#view-the-default-replication-zone). - -### Source privileges - -{% include {{ page.version.version }}/misc/non-http-source-privileges.md %} - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/restore.html %} -
    - -## Parameters - - Parameter | Description ------------|------------- - `table_pattern` | The table or [view](views.html) you want to restore. - `database_name` | The name of the database you want to restore (i.e., restore all tables and views in the database). You can restore an entire database only if you had backed up the entire database. - `destination` | The URL where the [full backup](take-full-and-incremental-backups.html#full-backups) (and appended [incremental backups](take-full-and-incremental-backups.html#incremental-backups), if applicable) is stored.

    For information about this URL structure, see [Backup File URLs](#backup-file-urls). - `LATEST` | Restore the most recent backup in the given location. See the [Restore from the most recent backup](#restore-the-most-recent-backup) example. - `subdirectory` | Restore from a specific subdirectory in the given collection URI. See the [Restore a specific backup](#restore-a-specific-backup) example. - `partitioned_backup_location` | The URL where a [locality-aware backup](take-and-restore-locality-aware-backups.html) is stored. When restoring from an incremental locality-aware backup, you need to include _every_ locality ever used, even if it was only used once.

    For information about this URL structure, see [Backup File URLs](#backup-file-urls). - `AS OF SYSTEM TIME timestamp` | Restore data as it existed as of [`timestamp`](as-of-system-time.html). You can restore point-in-time data only if you had taken full or incremental backup [with revision history](take-backups-with-revision-history-and-restore-from-a-point-in-time.html). - `restore_options_list` | Control your backup's behavior with [these options](#options). - -### Options - -You can control `RESTORE` behavior using any of the following in the `restore_options_list`. To set multiple `RESTORE` options, use a comma-separated list: - - Option |
    Value
    | Description - -------------------------------------------------------------------+---------------+------------------------------------------------------- -`into_db` | Database name | Use to [change the target database](#restore-tables-into-a-different-database) for table restores. The target database must exist before a restore with `into_db`. (Does not apply to database or cluster restores.)

    Example: `WITH into_db = 'newdb'` -`skip_missing_foreign_keys` | N/A | Use to remove the missing [foreign key](foreign-key.html) constraints before restoring.

    Example: `WITH skip_missing_foreign_keys` -`skip_missing_sequences` | N/A | Use to ignore [sequence](show-sequences.html) dependencies (i.e., the `DEFAULT` expression that uses the sequence).

    Example: `WITH skip_missing_sequences` -`skip_missing_sequence_owners` | N/A | Must be used when restoring either a table that was previously a [sequence owner](create-sequence.html#owned-by) or a sequence that was previously owned by a table.

    Example: `WITH skip_missing_sequence_owners` -`skip_missing_views` | N/A | Use to skip restoring [views](views.html) that cannot be restored because their dependencies are not being restored at the same time.

    Example: `WITH skip_missing_views` -`encryption_passphrase` | Passphrase used to create the [encrypted backup](take-and-restore-encrypted-backups.html) | The passphrase used to decrypt the file(s) that were encrypted by the [`BACKUP`](take-and-restore-encrypted-backups.html) statement. -`DETACHED` | N/A | When `RESTORE` runs with `DETACHED`, the job will execute asynchronously. The job ID is returned after the restore job creation completes. Note that with `DETACHED` specified, further job information and the job completion status will not be returned. For more on the differences between the returned job data, see the [example](restore.html#restore-a-backup-asynchronously) below. To check on the job status, use the [`SHOW JOBS`](show-jobs.html) statement.

    To run a restore within a [transaction](transactions.html), use the `DETACHED` option. - -### Backup file URLs - -CockroachDB uses the URL provided to construct a secure API call to the service you specify. The URL structure depends on the type of file storage you are using. For more information, see the following: - -- [Use Cloud Storage for Bulk Operations](use-cloud-storage-for-bulk-operations.html) - - {{site.data.alerts.callout_info}} - HTTP storage is not supported for `BACKUP` and `RESTORE`. - {{site.data.alerts.end}} - -- [Use a Local File Server for Bulk Operations](use-a-local-file-server-for-bulk-operations.html) - -## Functional details - -You can restore: - -- [A full cluster](#full-cluster) -- [Databases](#databases) -- [Tables](#tables) - -#### Full cluster - - A full cluster restore can only be run on a target cluster with no user-created databases or tables. Restoring a full cluster includes: - -- All user tables -- Relevant system tables -- All [databases](create-database.html) -- All [tables](create-table.html) (which automatically includes their [indexes](indexes.html)) -- All [views](views.html) - -`RESTORE` will only restore the latest data in an object (table, database, cluster), or the latest data as per an [`AS OF SYSTEM TIME` restore](#as-of-system-time). That is, a restore will not include historical data even if you ran your backup with [`revision_history`](backup.html#with-revision-history). This means that if you issue an `AS OF SYSTEM TIME` query on a restored object, the query will fail or the response will be incorrect because there is no historical data to query. For example, if you restore a table at `2022-07-13 10:38:00`, it is not then possible to read or [back up](backup.html) that table at `2022-07-13 10:37:00` or earlier. This is also the case for backups with `revision_history` that might try to initiate a revision start time earlier than `2022-07-13 10:38:00`. - -{{site.data.alerts.callout_info}} -When you restore a full cluster with an Enterprise license, it will restore the [Enterprise license](enterprise-licensing.html) of the cluster you are restoring from. If you want to use a different license in the new cluster, make sure to [update the license](licensing-faqs.html#set-a-license) _after_ the restore is complete. -{{site.data.alerts.end}} - -#### Databases - -**The database cannot already exist in the target cluster.** Restoring a database will create a new database and restore all of its tables and views. The created database will have the name of the database in the backup. - -~~~ sql -RESTORE DATABASE backup_database_name FROM 'subdirectory' IN 'your_backup_location'; -~~~ - -{{site.data.alerts.callout_success}} -If [dropping](drop-database.html) or [renaming](rename-database.html) an existing database is not an option, you can use [_table_ restore](#restore-a-table) to restore all tables into the existing database by using the [`WITH into_db` option](#options). -{{site.data.alerts.end}} - -#### Tables - -You can also restore individual tables (which automatically includes their indexes) or [views](views.html) from a backup. This process uses the data stored in the backup to create entirely new tables or views in the target database. - -By default, tables and views are restored into a target database matching the name of the database from which they were backed up. If the target database does not exist, you must [create it](create-database.html). You can choose to change the target database with the [`into_db` option](#into_db). - -The target database must not have tables or views with the same name as the tables or views you're restoring. If any of the restore target's names are being used, you can: - -- [`DROP TABLE`](drop-table.html), [`DROP VIEW`](drop-view.html), or [`DROP SEQUENCE`](drop-sequence.html) and then restore them. Note that a sequence cannot be dropped while it is being used in a column's `DEFAULT` expression, so those expressions must be dropped before the sequence is dropped, and recreated after the sequence is recreated. The `setval` [function](functions-and-operators.html#sequence-functions) can be used to set the value of the sequence to what it was previously. -- [Restore the table or view into a different database](#into_db). - -{{site.data.alerts.callout_info}} -`RESTORE` only offers table-level granularity; it _does not_ support restoring subsets of a table. -{{site.data.alerts.end}} - -When restoring an individual table that references a user-defined type (e.g., [`ENUM`](enum.html)), CockroachDB will first check to see if the type already exists. The restore will attempt the following for each user-defined type within a table backup: - -- If there is _not_ an existing type in the cluster with the same name, CockroachDB will create the user-defined type as it exists in the backup. -- If there is an existing type in the cluster with the same name that is compatible with the type in the backup, CockroachDB will map the type in the backup to the type in the cluster. -- If there is an existing type in the cluster with the same name but it is _not_ compatible with the type in the backup, the restore will not succeed and you will be asked to resolve the naming conflict. You can do this by either [dropping](drop-type.html) or [renaming](alter-type.html) the existing user-defined type. - -In general, two types are compatible if they are the same kind (e.g., an enum is only compatible with other enums). Additionally, enums are only compatible if they have the same ordered set of elements that have also been [created in the same way](https://github.com/cockroachdb/cockroach/blob/master/docs/RFCS/20200331_enums.md#physical-layout). For example: - -- `CREATE TYPE t1 AS ENUM ('yes', 'no')` and `CREATE TYPE t2 AS ENUM ('yes', 'no')` are compatible. -- `CREATE TYPE t1 AS ENUM ('yes', 'no')` and `CREATE TYPE t2 AS ENUM ('no', 'yes')` are not compatible. -- `CREATE TYPE t1 AS ENUM ('yes', 'no')` and `CREATE TYPE t2 AS ENUM ('yes'); ALTER TYPE t2 ADD VALUE ('no')` are not compatible because they were not created in the same way. - -### Object dependencies - -Dependent objects must be restored at the same time as the objects they depend on. - -Object | Depends On --------|----------- -Table with [foreign key](foreign-key.html) constraints | The table it `REFERENCES` (however, this dependency can be [removed during the restore](#skip_missing_foreign_keys)). -Table with a [sequence](create-sequence.html) | The sequence. -[Views](views.html) | The tables used in the view's `SELECT` statement. -[Interleaved tables](interleave-in-parent.html) | The parent table in the [interleaved hierarchy](interleave-in-parent.html). - -### Users and privileges - -To restore your users and privilege [grants](grant.html), you can do a cluster backup and restore the cluster to a fresh cluster with no user data. - -If you are not doing a full cluster restore, the table-level privileges need to be granted to the users after the restore is complete. (By default, the user restoring will become the owner of the restored objects.) To grant table-level privileges after a restore, backup the `system.users` table, [restore users and their passwords](restore.html#restoring-users-from-system-users-backup), and then [grant](grant.html) the table-level privileges. - -### Restore types - -You can either restore from a full backup or from a full backup with incremental backups, based on the backup files you include: - -Restore Type | Parameters --------------|---------- -Full backup | Include the path to the full backup destination and the [subdirectory](#view-the-backup-subdirectories) of the backup. See the [Examples](#examples) section for syntax of [cluster](#restore-a-cluster), [database](#restore-a-database), and [table](#restore-a-table) restores. -Full backup +
    incremental backups | Include the path that contains the backup collection and the [subdirectory](#view-the-backup-subdirectories) containing the incremental backup. See [Restore from incremental backups](#restore-from-incremental-backups) for an example. - -## Performance - -The `RESTORE` process minimizes its impact to the cluster's performance by distributing work to all nodes. Subsets of the restored data (known as ranges) are evenly distributed among randomly selected nodes, with each range initially restored to only one node. Once the range is restored, the node begins replicating it others. - -{{site.data.alerts.callout_info}} -When a `RESTORE` fails or is canceled, partially restored data is properly cleaned up. This can have a minor, temporary impact on cluster performance. -{{site.data.alerts.end}} - -## Viewing and controlling restore jobs - -After CockroachDB successfully initiates a restore, it registers the restore as a job, which you can view with [`SHOW JOBS`](show-jobs.html). - -After the restore has been initiated, you can control it with [`PAUSE JOB`](pause-job.html), [`RESUME JOB`](resume-job.html), and [`CANCEL JOB`](cancel-job.html). - -{{site.data.alerts.callout_info}} -If initiated correctly, the statement returns when the restore is finished or if it encounters an error. In some cases, the restore can continue after an error has been returned (the error message will tell you that the restore has resumed in background). -{{site.data.alerts.end}} - -## Examples - -{% include {{ page.version.version }}/backups/bulk-auth-options.md %} - -
    - -{{site.data.alerts.callout_info}} -The examples in this section use the **default** `AUTH=specified` parameter. For more detail on how to use `implicit` authentication with Amazon S3 buckets, read [Use Cloud Storage for Bulk Operations — Authentication](use-cloud-storage-for-bulk-operations.html#authentication). -{{site.data.alerts.end}} - -### View the backup subdirectories - -`BACKUP ... INTO` adds a backup to a collection within the backup destination. The path to the backup is created using a date-based naming scheme. To view the backup paths in a given destination, use [`SHOW BACKUPS`](show-backup.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW BACKUPS IN 's3://{bucket_name}?AWS_ACCESS_KEY_ID={key_id}&AWS_SECRET_ACCESS_KEY={access_key}'; -~~~ - -~~~ - path -------------------------- -/2021/12/14-190909.83 -/2021/12/20-155249.37 -/2021/12/21-142943.73 -(3 rows) -~~~ - -When you want to [restore a specific backup](#restore-a-specific-backup), add the backup's subdirectory path (e.g., `/2021/12/21-142943.73`) to the `RESTORE` statement. For details on viewing the most recent backup, see [`SHOW BACKUP LATEST`](show-backup.html#show-the-most-recent-backup). - -Incremental backups will be appended to the full backup with `BACKUP ... INTO LATEST IN {destination}`. Your storage location will contain the incremental as a date-based subdirectory within the full backup. - -In the following example `/2021/12/21-142943.73` contains the full backup. The incremental backups (`144748.08/` and `144639.97/`) are appended as subdirectories to the full backup: - -~~~ -2021 -|—— 12 - |—— 21-142943.73/ - |—— 20211221/ - |—— 144748.08/ - |—— 144639.97/ -~~~ - -To output more detail about the backups contained within a directory, see [View a list of the full and incremental backups in a specific full backup subdirectory](show-backup.html#view-a-list-of-the-full-and-incremental-backups-in-a-specific-full-backup-subdirectory). - -See [Incremental backups with explicitly specified destinations](take-full-and-incremental-backups.html#incremental-backups-with-explicitly-specified-destinations) to control where your backups go. - -### Restore the most recent backup - -{% include_cached new-in.html version="v21.1.14" %} To restore from the most recent backup in the collection's location, use the `LATEST syntax`: - -{% include_cached copy-clipboard.html %} -~~~ sql -RESTORE FROM LATEST IN 's3://{bucket_name}?AWS_ACCESS_KEY_ID={key_id}&AWS_SECRET_ACCESS_KEY={access_key}'; -~~~ - -### Restore a specific backup - -To restore a specific backup, use the backup's [subdirectory](#subdir-param) in the collection's location: - -{% include_cached copy-clipboard.html %} -~~~ sql -RESTORE FROM '2021/03/23-213101.37' IN 's3://{bucket_name}?AWS_ACCESS_KEY_ID={key_id}&AWS_SECRET_ACCESS_KEY={access_key}'; -~~~ - -To view the available subdirectories, use [`SHOW BACKUPS`](#view-the-backup-subdirectories). - -### Restore a cluster - -To restore a full cluster: - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESTORE FROM LATEST IN 's3://{bucket name}?AWS_ACCESS_KEY_ID={key_id}&AWS_SECRET_ACCESS_KEY={access_key}'; -~~~ - -To view the available subdirectories, use [`SHOW BACKUPS`](#view-the-backup-subdirectories). - -### Restore a database - -To restore a database: - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESTORE DATABASE bank FROM LATEST IN 's3://{bucket name}?AWS_ACCESS_KEY_ID={key_id}&AWS_SECRET_ACCESS_KEY={access_key}'; -~~~ - -To view the available subdirectories, use [`SHOW BACKUPS`](#view-the-backup-subdirectories). - -{{site.data.alerts.callout_info}} -`RESTORE DATABASE` can only be used if the entire database was backed up. -{{site.data.alerts.end}} - -### Restore a table - -To restore a single table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESTORE TABLE bank.customers FROM LATEST IN 's3://{bucket name}?AWS_ACCESS_KEY_ID={key_id}&AWS_SECRET_ACCESS_KEY={access_key}'; -~~~ - -To restore multiple tables: - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESTORE TABLE bank.customers, bank.accounts FROM LATEST IN 's3://{bucket name}?AWS_ACCESS_KEY_ID={key_id}&AWS_SECRET_ACCESS_KEY={access_key}'; -~~~ - -To view the available subdirectories, use [`SHOW BACKUPS`](#view-the-backup-subdirectories). - -### Restore from incremental backups - -To restore the most recent [incremental backup](take-full-and-incremental-backups.html#incremental-backups) from a location containing the full and incremental backup: - -{% include_cached copy-clipboard.html %} -~~~ sql -RESTORE DATABASE bank FROM LATEST IN 's3://{bucket_name}/{path/to/backup-collection}?AWS_ACCESS_KEY_ID={key_id}&AWS_SECRET_ACCESS_KEY={access_key}'; -~~~ - -{{site.data.alerts.callout_info}} -{% include_cached new-in.html version="v21.1" %} `RESTORE` will re-validate [indexes](indexes.html) when [incremental backups](take-full-and-incremental-backups.html) are created from an older version (v20.2.2 and earlier or v20.1.4 and earlier), but restored by a newer version (v21.1.0+). These earlier releases may have included incomplete data for indexes that were in the process of being created. -{{site.data.alerts.end}} - -### Restore a backup asynchronously - -Use the `DETACHED` [option](#options) to execute the restore [job](show-jobs.html) asynchronously: - -{% include_cached copy-clipboard.html %} -~~~ sql -RESTORE TABLE bank.customers FROM LATEST IN 's3://{bucket name}?AWS_ACCESS_KEY_ID={key_id}&AWS_SECRET_ACCESS_KEY={access_key}' WITH DETACHED; -~~~ - -The job ID is returned after the restore job creation completes: - -~~~ - job_id ----------------------- - 592786066399264769 -(1 row) -~~~ - -**Without** the `DETACHED` option, `RESTORE` will block the SQL connection until the job completes. Once finished, the job status and more detailed job data is returned: - -~~~ -job_id | status | fraction_completed | rows | index_entries | bytes --------------------+-----------+--------------------+------+---------------+-------- -652471804772712449 | succeeded | 1 | 50 | 0 | 4911 -(1 row) -~~~ - -### Other restore usages - -#### Restore tables into a different database - -By default, tables and views are restored to the database they originally belonged to. However, using the [`into_db` option](#into_db), you can control the target database. Note that the target database must exist prior to the restore. - -First, create the new database that you'll restore the table or view into: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE DATABASE newdb; -~~~ - -Next, restore the table into the newly created database with `into_db`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESTORE bank.customers \ -FROM '2021/09/29-153014.47' IN 's3://{bucket name}?AWS_ACCESS_KEY_ID={key_id}&AWS_SECRET_ACCESS_KEY={access_key}' \ -WITH into_db = 'newdb'; -~~~ - -#### Remove the foreign key before restore - -By default, tables with [foreign key](foreign-key.html) constraints must be restored at the same time as the tables they reference. However, using the [`skip_missing_foreign_keys`](restore.html#skip_missing_foreign_keys) option you can remove the foreign key constraint from the table and then restore it. - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESTORE bank.accounts \ -FROM '2021/09/29-153014.47' IN 's3://{bucket name}?AWS_ACCESS_KEY_ID={key_id}&AWS_SECRET_ACCESS_KEY={access_key}' \ -WITH skip_missing_foreign_keys; -~~~ - -#### Restoring users from `system.users` backup - -The `system.users` table stores your cluster's usernames and their hashed passwords. To restore them, you must restore the `system.users` table into a new database because you cannot drop the existing `system.users` table. - -After it's restored into a new database, you can write the restored `users` table data to the cluster's existing `system.users` table. - -First, create the new database that you'll restore the `system.users` table into: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE DATABASE newdb; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESTORE system.users \ -FROM '2021/09/29-153014.47' IN 's3://{bucket name}?AWS_ACCESS_KEY_ID={key_id}&AWS_SECRET_ACCESS_KEY={access_key}' \ -WITH into_db = 'newdb'; -~~~ - -After the restore completes, add the `users` to the existing `system.users` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO system.users SELECT * FROM newdb.users; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP TABLE newdb.users; -~~~ - -## See also - -- [`BACKUP`](backup.html) -- [Take Full and Incremental Backups](take-full-and-incremental-backups.html) -- [Take and Restore Encrypted Backups](take-and-restore-encrypted-backups.html) -- [Take and Restore Locality-aware Backups](take-and-restore-locality-aware-backups.html) -- [Take Backups with Revision History and Restore from a Point-in-time](take-backups-with-revision-history-and-restore-from-a-point-in-time.html) -- [Manage a Backup Schedule](manage-a-backup-schedule.html) -- [Configure Replication Zones](configure-replication-zones.html) -- [`ENUM`](enum.html) -- [`CREATE TYPE`](create-type.html) -- [`DROP TYPE`](drop-type.html) diff --git a/src/current/v21.1/resume-job.md b/src/current/v21.1/resume-job.md deleted file mode 100644 index 08a9ad1f66e..00000000000 --- a/src/current/v21.1/resume-job.md +++ /dev/null @@ -1,105 +0,0 @@ ---- -title: RESUME JOB -summary: The RESUME JOB statement lets you resume jobs that were previously paused with PAUSE JOB. -toc: true ---- - - The `RESUME JOB` [statement](sql-statements.html) lets you resume the following types of jobs: - - - [`IMPORT`](import.html) jobs - - [`BACKUP`](backup.html) and [`RESTORE`](restore.html) jobs - - [User-created table statistics](create-statistics.html) jobs - - [Automatic table statistics](cost-based-optimizer.html#table-statistics) jobs - - [Changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html) - - [Schema change](online-schema-changes.html) jobs - - [Scheduled backup](manage-a-backup-schedule.html) jobs - -## Required privileges - -To resume a job, the user must be a member of the `admin` role or must have the [`CONTROLJOB`](create-user.html#create-a-user-that-can-pause-resume-and-cancel-non-admin-jobs) parameter set. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/resume_job.html %} -
    - -## Parameters - -Parameter | Description -----------|------------ -`job_id` | The ID of the job you want to resume, which can be found with [`SHOW JOBS`](show-jobs.html). -`select_stmt` | A [selection query](selection-queries.html) that returns `job_id`(s) to resume. -`for_schedules_clause` | The schedule you want to resume jobs for. You can resume jobs for a specific schedule (`FOR SCHEDULE id`) or resume jobs for multiple schedules by nesting a [`SELECT` clause](select-clause.html) in the statement (`FOR SCHEDULES `). See the [examples](#resume-jobs-for-a-schedule) below. - -## Examples - -### Pause a job - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW JOBS; -~~~ - -~~~ - job_id | job_type | description |... ------------------+-----------+-------------------------------------------+... - 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> PAUSE JOB 27536791415282; -~~~ - -### Resume a single job - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESUME JOB 27536791415282; -~~~ - -### Resume multiple jobs - -To resume multiple jobs, nest a [`SELECT` clause](select-clause.html) that retrieves `job_id`(s) inside the `RESUME JOBS` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESUME JOBS (SELECT job_id FROM [SHOW JOBS] - WHERE user_name = 'maxroach'); -~~~ - -All jobs created by `maxroach` will be resumed. - -### Resume jobs for a schedule - - To resume jobs for a specific [backup schedule](create-schedule-for-backup.html), use the schedule's `id`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESUME JOBS FOR SCHEDULE 590204387299262465; -~~~ -~~~ -RESUME JOBS FOR SCHEDULES 1 -~~~ - -You can also resume multiple schedules by nesting a [`SELECT` clause](select-clause.html) that retrieves `id`(s) inside the `PAUSE JOBS` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESUME JOBS FOR SCHEDULES SELECT id FROM [SHOW SCHEDULES] WHERE label = 'test_schedule'; -~~~ - -~~~ -RESUME JOBS FOR SCHEDULES 2 -~~~ - -## See also - -- [`PAUSE JOB`](pause-job.html) -- [`SHOW JOBS`](show-jobs.html) -- [`CANCEL JOB`](cancel-job.html) -- [`BACKUP`](backup.html) -- [`RESTORE`](restore.html) -- [`IMPORT`](import.html) -- [`CREATE CHANGEFEED`](create-changefeed.html) diff --git a/src/current/v21.1/resume-schedules.md b/src/current/v21.1/resume-schedules.md deleted file mode 100644 index e6c3406a891..00000000000 --- a/src/current/v21.1/resume-schedules.md +++ /dev/null @@ -1,82 +0,0 @@ ---- -title: RESUME SCHEDULES -summary: The RESUME SCHEDULES statement lets you resume jobs that were previously paused with PAUSE SCHEDULE. -toc: true ---- - - The `RESUME SCHEDULES` [statement](sql-statements.html) can be used to resume [paused backup schedules](pause-schedules.html). When a schedule is resumed, the `next_run` will be set to the next `TIMESTAMP` that conforms to the schedule. - -## Required privileges - -Only members of the [`admin` role](authorization.html#default-roles) can resume a schedule. By default, the `root` user belongs to the `admin` role. - -## Synopsis - -~~~ -RESUME SCHEDULES - selectclause: select statement returning schedule IDs to resume. - -RESUME SCHEDULE -~~~ - -## Parameters - - Parameter | Description ----------------+------------ -`selectclause` | A [selection query](selection-queries.html) that returns `id`(s) to resume. -`scheduleID` | The `id` of the schedule you want to resume, which can be found with [`SHOW SCHEDULES`](show-schedules.html). - -## Examples - -### Pause a schedule - -{% include_cached copy-clipboard.html %} -~~~ sql -> PAUSE SCHEDULE 589963390487363585; -~~~ - -~~~ -PAUSE SCHEDULES 1 -~~~ - -### Resume a single schedule - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESUME SCHEDULE 589963390487363585; -~~~ - -~~~ -RESUME SCHEDULES 1 -~~~ - -### Resume multiple schedules - -To resume multiple schedules, nest a [`SELECT` clause](select-clause.html) that retrieves `id`(s) inside the `RESUME SCHEDULES` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESUME SCHEDULES SELECT id FROM [SHOW SCHEDULES] WHERE label = 'schedule_database'; -~~~ - -~~~ -RESUME SCHEDULES 4 -~~~ - -All schedules with the label `schedule_database` are resumed. - -## See also - -- [Manage a Backup Schedule](manage-a-backup-schedule.html) -- [`BACKUP`](backup.html) -- [`RESTORE`](restore.html) -- [`SHOW BACKUP`](show-backup.html) -- [`SHOW SCHEDULES`](show-schedules.html) -- [`PAUSE SCHEDULES`](pause-schedules.html) -- [`DROP SCHEDULES`](drop-schedules.html) -- [`PAUSE JOB`](pause-job.html) -- [`RESUME JOB`](pause-job.html) -- [`CANCEL JOB`](cancel-job.html) -- [Take Full and Incremental Backups](take-full-and-incremental-backups.html) -- [Use the Built-in SQL Client](cockroach-sql.html) -- [Other Cockroach Commands](cockroach-commands.html) diff --git a/src/current/v21.1/revoke.md b/src/current/v21.1/revoke.md deleted file mode 100644 index b505652e46a..00000000000 --- a/src/current/v21.1/revoke.md +++ /dev/null @@ -1,383 +0,0 @@ ---- -title: REVOKE -summary: The REVOKE statement revokes privileges from users and/or roles, or revokes privileges from users and/or roles. -toc: true ---- - -The `REVOKE` [statement](sql-statements.html) revokes [privileges](authorization.html#assign-privileges) from [users](authorization.html#create-and-manage-users) and/or [roles](authorization.html#create-and-manage-roles). For the list of privileges that can be granted to and revoked from users and roles, see [`GRANT`](grant.html). - -You can use `REVOKE` to directly revoke privileges from a role or user, or you can revoke membership to an existing role, which effectively revokes that role's privileges. - -## Syntax - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/revoke.html %} -
    - -### Parameters - -Parameter | Description ---------------------------------------|------------ -`ALL`
    `ALL PRIVILEGES` | Revoke all [privileges](#supported-privileges). -`targets` | A comma-separated list of database, schema, table, or user-defined type names, followed by the name of the object (e.g., `DATABASE mydatabase`).
    {{site.data.alerts.callout_info}}To revoke privileges on all tables in a database or schema, you can use `REVOKE ... ON TABLE *`. For an example, see [Revoke privileges on all tables in a database or schema](#revoke-privileges-on-all-tables-in-a-database-or-schema).{{site.data.alerts.end}} -`name_list` | A comma-separated list of [users](authorization.html#create-and-manage-users) and/or [roles](authorization.html#create-and-manage-roles) from whom to revoke privileges. -`privilege_list ON targets FROM ...` | Specify a comma-separated list of [privileges](authorization.html#assign-privileges) to revoke. -`privilege_list FROM ...` | Specify a comma-separated list of [roles](authorization.html#create-and-manage-roles) whose membership to revoke. -`ADMIN OPTION FOR privilege_list` | Revoke the user's role admin status. - -## Supported privileges - -The following privileges can be revoked: - -{% include {{ page.version.version }}/sql/privileges.md %} - -## Required privileges - -- To revoke privileges, user revoking privileges must have the `GRANT` privilege on the target [database](create-database.html), [schema](create-schema.html), [table](create-table.html), or [user-defined type](enum.html). In addition to the `GRANT` privilege, the user revoking privileges must have the privilege being revoked on the target object. For example, a user revoking the `SELECT` privilege on a table to another user must have the `GRANT` and `SELECT` privileges on that table. - -- To revoke role membership, the user revoking role membership must be a role admin (i.e., members with the `WITH ADMIN OPTION`) or a member of the `admin` role. To remove membership to the `admin` role, the user must have `WITH ADMIN OPTION` on the `admin` role. - -## Considerations - -- The `root` user cannot be revoked from the `admin` role. - -## Examples - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### Revoke privileges on databases - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE USER max WITH PASSWORD roach; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT CREATE ON DATABASE movr TO max; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON DATABASE movr; -~~~ - -~~~ - database_name | grantee | privilege_type -----------------+---------+----------------- - movr | admin | ALL - movr | max | CREATE - movr | root | ALL -(3 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> REVOKE CREATE ON DATABASE movr FROM max; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON DATABASE movr; -~~~ - -~~~ - database_name | grantee | privilege_type -----------------+---------+----------------- - movr | admin | ALL - movr | root | ALL -(2 rows) -~~~ - -{{site.data.alerts.callout_info}} -Any tables that previously inherited the database-level privileges retain the privileges. -{{site.data.alerts.end}} - -### Revoke privileges on specific tables in a database - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT DELETE ON TABLE rides TO max; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON TABLE rides; -~~~ - -~~~ - database_name | schema_name | table_name | grantee | privilege_type -----------------+-------------+------------+---------+----------------- - movr | public | rides | admin | ALL - movr | public | rides | max | DELETE - movr | public | rides | root | ALL -(3 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> REVOKE DELETE ON TABLE rides FROM max; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON TABLE rides; -~~~ - -~~~ - database_name | schema_name | table_name | grantee | privilege_type -----------------+-------------+------------+---------+----------------- - movr | public | rides | admin | ALL - movr | public | rides | root | ALL -(2 rows) -~~~ - -### Revoke privileges on all tables in a database or schema - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT CREATE, SELECT, DELETE ON TABLE rides, users TO max; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON TABLE movr.*; -~~~ - -~~~ - database_name | schema_name | table_name | grantee | privilege_type -----------------+-------------+----------------------------+---------+----------------- - movr | public | promo_codes | admin | ALL - movr | public | promo_codes | root | ALL - movr | public | rides | admin | ALL - movr | public | rides | max | CREATE - movr | public | rides | max | DELETE - movr | public | rides | max | SELECT - movr | public | rides | root | ALL - movr | public | user_promo_codes | admin | ALL - movr | public | user_promo_codes | root | ALL - movr | public | users | admin | ALL - movr | public | users | max | CREATE - movr | public | users | max | DELETE - movr | public | users | max | SELECT - movr | public | users | root | ALL - movr | public | vehicle_location_histories | admin | ALL - movr | public | vehicle_location_histories | root | ALL - movr | public | vehicles | admin | ALL - movr | public | vehicles | root | ALL -(18 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> REVOKE DELETE ON movr.* FROM max; -~~~ - -~~~ - database_name | schema_name | table_name | grantee | privilege_type -----------------+-------------+----------------------------+---------+----------------- - movr | public | promo_codes | admin | ALL - movr | public | promo_codes | root | ALL - movr | public | rides | admin | ALL - movr | public | rides | max | CREATE - movr | public | rides | max | SELECT - movr | public | rides | root | ALL - movr | public | user_promo_codes | admin | ALL - movr | public | user_promo_codes | root | ALL - movr | public | users | admin | ALL - movr | public | users | max | CREATE - movr | public | users | max | SELECT - movr | public | users | root | ALL - movr | public | vehicle_location_histories | admin | ALL - movr | public | vehicle_location_histories | root | ALL - movr | public | vehicles | admin | ALL - movr | public | vehicles | root | ALL -(16 rows) -~~~ - - -### Revoke privileges on schemas - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE SCHEMA cockroach_labs; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT ALL ON SCHEMA cockroach_labs TO max; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON SCHEMA cockroach_labs; -~~~ - -~~~ - database_name | schema_name | grantee | privilege_type -----------------+----------------+---------+----------------- - movr | cockroach_labs | admin | ALL - movr | cockroach_labs | max | ALL - movr | cockroach_labs | root | ALL -(3 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> REVOKE CREATE ON SCHEMA cockroach_labs FROM max; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON SCHEMA cockroach_labs; -~~~ - -~~~ - database_name | schema_name | grantee | privilege_type -----------------+----------------+---------+----------------- - movr | cockroach_labs | admin | ALL - movr | cockroach_labs | max | GRANT - movr | cockroach_labs | max | USAGE - movr | cockroach_labs | root | ALL -(4 rows) -~~~ - -### Revoke privileges on user-defined types - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TYPE status AS ENUM ('available', 'unavailable'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT ALL ON TYPE status TO max; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON TYPE status; -~~~ - -~~~ - database_name | schema_name | type_name | grantee | privilege_type -----------------+-------------+-----------+---------+----------------- - movr | public | status | admin | ALL - movr | public | status | max | ALL - movr | public | status | public | USAGE - movr | public | status | root | ALL -(4 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> REVOKE GRANT ON TYPE status FROM max; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON TYPE status; -~~~ - -~~~ - database_name | schema_name | type_name | grantee | privilege_type -----------------+-------------+-----------+---------+----------------- - movr | public | status | admin | ALL - movr | public | status | max | USAGE - movr | public | status | public | USAGE - movr | public | status | root | ALL -(4 rows) -~~~ - -### Revoke role membership - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE ROLE developer WITH CREATEDB; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE USER abbey WITH PASSWORD lincoln; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT developer TO abbey; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON ROLE developer; -~~~ - -~~~ - role_name | member | is_admin -------------+--------+----------- - developer | abbey | false -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> REVOKE developer FROM abbey; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON ROLE developer; -~~~ - -~~~ - role_name | member | is_admin -------------+--------+----------- -(0 rows) -~~~ - -### Revoke the admin option - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT developer TO abbey WITH ADMIN OPTION; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON ROLE developer; -~~~ - -~~~ - role_name | member | is_admin -------------+--------+----------- - developer | abbey | true -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> REVOKE ADMIN OPTION FOR developer FROM abbey; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON ROLE developer; -~~~ - -~~~ - role_name | member | is_admin -------------+--------+----------- - developer | abbey | false -(1 row) -~~~ - - - -## See also - -- [Authorization](authorization.html) -- [`GRANT`](grant.html) -- [`SHOW GRANTS`](show-grants.html) -- [`SHOW ROLES`](show-roles.html) -- [`CREATE USER`](create-user.html) -- [`DROP USER`](drop-user.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/rollback-transaction.md b/src/current/v21.1/rollback-transaction.md deleted file mode 100644 index 4fac90c435f..00000000000 --- a/src/current/v21.1/rollback-transaction.md +++ /dev/null @@ -1,116 +0,0 @@ ---- -title: ROLLBACK -summary: Rolls back the current transaction and all of its nested sub-transactions, discarding all transactional updates made by statements inside the transaction. -toc: true ---- - -The `ROLLBACK` [statement](sql-statements.html) aborts the current [transaction](transactions.html) and all of its [nested transactions](transactions.html#nested-transactions), discarding all transactional updates made by statements included in the transaction. - -There are two ways to use `ROLLBACK`: - -- The `ROLLBACK` statement [rolls back the entire transaction](#rollback-a-transaction). - -- The `ROLLBACK TO SAVEPOINT` statement [rolls back and restarts the nested transaction](#rollback-a-nested-transaction) started at the corresponding `SAVEPOINT` statement, for working with [standard savepoints](savepoint.html#savepoints-for-nested-transactions). This is in addition to the existing support for working with [client-side transaction retries](transactions.html#client-side-intervention). For examples of each usage, see: - - - [Rollback a nested transaction](#rollback-a-nested-transaction) - - [Retry a transaction](#retry-a-transaction) - -{% include {{page.version.version}}/sql/savepoint-ddl-rollbacks.md %} - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/rollback_transaction.html %} -
    - -## Required privileges - -No [privileges](authorization.html#assign-privileges) are required to rollback a transaction. However, privileges are required for each statement within a transaction. - -## Parameters - - Parameter | Description ------------|------------- - `TO SAVEPOINT cockroach_restart` | If using [advanced client-side transaction retries](advanced-client-side-transaction-retries.html), retry the transaction. You should execute this statement when a transaction returns a `40001` / `retry transaction` error. - `TO SAVEPOINT ` | If using [nested transactions](savepoint.html#savepoints-for-nested-transactions), roll back and restart the [nested transaction](transactions.html#nested-transactions) started at the corresponding `SAVEPOINT` statement. - -## Savepoints and row locks - -{% include {{page.version.version}}/sql/savepoints-and-row-locks.md %} - -## Savepoints and high priority transactions - -{% include {{page.version.version}}/sql/savepoints-and-high-priority-transactions.md %} - -## Examples - -### Rollback a transaction - -Typically, an application conditionally executes rollbacks, but we can see their behavior by using `ROLLBACK` instead of `COMMIT` directly through SQL: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM accounts; -~~~ - -~~~ -+----------+---------+ -| name | balance | -+----------+---------+ -| Marciela | 1000 | -+----------+---------+ -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE accounts SET balance = 2500 WHERE name = 'Marciela'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ROLLBACK; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM accounts; -~~~ - -~~~ -+----------+---------+ -| name | balance | -+----------+---------+ -| Marciela | 1000 | -+----------+---------+ -~~~ - -### Rollback a nested transaction - -The `ROLLBACK TO SAVEPOINT` statement rolls back and restarts the [nested transaction](transactions.html#nested-transactions) started at the corresponding `SAVEPOINT` statement. - -For examples showing how to use `ROLLBACK TO SAVEPOINT` to rollback a nested transaction, see [the `SAVEPOINT` documentation on nested savepoints](savepoint.html#savepoints-for-nested-transactions). - -### Retry a transaction - -When using [advanced client-side transaction retries](advanced-client-side-transaction-retries.html), use `ROLLBACK TO SAVEPOINT` to handle a transaction that needs to be retried (identified via the `40001` error code or `restart transaction` string in the error message), and then re-execute the statements you want the transaction to contain. - -{% include_cached copy-clipboard.html %} -~~~ sql -> ROLLBACK TO SAVEPOINT cockroach_restart; -~~~ - -For examples of retrying transactions in an application, check out the transaction code samples in our [Build an App with CockroachDB](example-apps.html) tutorials. - -## See also - -- [`SAVEPOINT`](savepoint.html) -- [Transactions](transactions.html) -- [`BEGIN`](begin-transaction.html) -- [`COMMIT`](commit-transaction.html) -- [`RELEASE SAVEPOINT`](release-savepoint.html) -- [`SHOW SAVEPOINT STATUS`](show-savepoint-status.html) diff --git a/src/current/v21.1/rotate-certificates.md b/src/current/v21.1/rotate-certificates.md deleted file mode 100644 index f6873975451..00000000000 --- a/src/current/v21.1/rotate-certificates.md +++ /dev/null @@ -1,151 +0,0 @@ ---- -title: Rotate Security Certificates -summary: Rotate the security certificates of a secure CockroachDB cluster by creating and reloading new certificates. -toc: true ---- - -CockroachDB allows you to rotate security certificates without restarting nodes. - -{{site.data.alerts.callout_success}}For an introduction to how security certificates work in a secure CockroachDB cluster, see Create Security Certificates.{{site.data.alerts.end}} - - -## When to rotate certificates - -You may need to rotate the node, client, or CA certificates in the following scenarios: - -- The node, client, or CA certificates are expiring soon. -- Your organization's compliance policy requires periodical certificate rotation. -- The key (for a node, client, or CA) is compromised. -- You need to modify the contents of a certificate, for example, to add another DNS name or the IP address of a load balancer through which a node can be reached. In this case, you would need to rotate only the node certificates. - -### Rotate client certificates - -1. Create a new client certificate and key: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach cert create-client \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key - ~~~ - -2. Upload the new client certificate and key to the client using your preferred method. - -3. Have the client use the new client certificate. - - This step is application-specific and may require restarting the client. - -### Rotate node certificates - -To rotate a node certificate, you create a new node certificate and key and reload them on the node. - -1. Create a new node certificate and key: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach cert create-node \ - \ - \ - \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key \ - --overwrite - ~~~ - - Since you must create the new certificate and key in the same directory as the existing certificate and key, use the `--overwrite` flag to overwrite the existing files. Also, be sure to specify all addresses at which node can be reached. - -2. Upload the node certificate and key to the node: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ scp certs/node.crt \ - certs/node.key \ - @:~/certs - ~~~ - -3. Reload the node certificate without restarting the node by issuing a `SIGHUP` signal to the `cockroach` process: - - {% include_cached copy-clipboard.html %} - ~~~ shell - pkill -SIGHUP -x cockroach - ~~~ - - The `SIGHUP` signal must be sent by the same user running the process (e.g., run with `sudo` if the `cockroach` process is running under user `root`). - -4. Verify that certificate rotation was successful using the **Local Node Certificates** page in the DB Console: `https://
    :8080/#/reports/certificates/local`. - - Scroll to the node certificate details and confirm that the **Valid Until** field shows the new certificate expiration time. - -### Rotate the CA certificate - -To rotate the CA certificate, you create a new CA key and a combined CA certificate that contains the new CA certificate followed by the old CA certificate, and then you reload the new combined CA certificate on the nodes and clients. Once all nodes and clients have the combined CA certificate, you then create new node and client certificates signed with the new CA certificate and reload those certificates on the nodes and clients as well. - -For more background, see [Why CockroachDB creates a combined CA certificate](rotate-certificates.html#why-cockroachdb-creates-a-combined-ca-certificate) and [Why rotate CA certificate in advance](rotate-certificates.html#why-rotate-ca-certificates-in-advance). - -1. Rename the existing CA key: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ mv my-safe-directory/ca.key my-safe-directory/ca.old.key - ~~~ - -2. Create a new CA certificate and key, using the `--overwrite` flag to overwrite the old CA certificate: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach cert create-ca \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key \ - --overwrite - ~~~ - - This results in the [combined CA certificate](rotate-certificates.html#why-cockroachdb-creates-a-combined-ca-certificate), `ca.crt`, which contains the new certificate followed by the old certificate. - - {{site.data.alerts.callout_danger}}The CA key is never loaded automatically by cockroach commands, so it should be created in a separate directory, identified by the --ca-key flag.{{site.data.alerts.end}} - -2. Upload the new CA certificate to each node: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ scp certs/ca.crt - @:~/certs - ~~~ - -3. Upload the new CA certificate to each client using your preferred method. - -4. On each node, reload the CA certificate without restarting the node by issuing a `SIGHUP` signal to the `cockroach` process: - - {% include_cached copy-clipboard.html %} - ~~~ shell - pkill -SIGHUP -x cockroach - ~~~ - - The `SIGHUP` signal must be sent by the same user running the process (e.g., run with `sudo` if the `cockroach` process is running under user `root`). - -5. Reload the CA certificate on each client. - - This step is application-specific and may require restarting the client. - -6. Verify that certificate rotation was successful using the **Local Node Certificates** page in the DB Console: `https://
    :8080/#/reports/certificates/local`. - - The details of the old as well as new CA certificates should be shown. Confirm that the **Valid Until** field of the new CA certificate shows the new certificate expiration time. - -7. Once you are confident that all nodes and clients have the new CA certificate, [rotate the node certificates](#rotate-node-certificates) and [rotate the client certificates](#rotate-client-certificates). - -## Why CockroachDB creates a combined CA certificate - -On rotating the CA certificate, the nodes have the new CA certificate after certs directory is rescanned, and the clients have the new CA certificates when they are restarted. However, until the node and client certificates are rotated, the nodes and client certificates are still signed with the old CA certificate. Thus the nodes and clients are unable to verify each other's identity using the new CA certificate. - -To overcome the issue, we take advantage of the fact that multiple CA certificates can be active at the same time. While verifying the identity of another node or a client, they can check with multiple CA certificates uploaded to them. Thus instead of creating only the new certificate while rotating the CA certificates, CockroachDB creates a combined CA certificate that contains the new CA certificate followed by the old CA certificate. When node and client certificates are rotated, the combined CA certificate is used to verify old as well as new node and client certificates. - -## Why rotate CA certificates in advance - -On rotating node and client certificates after rotating the CA certificate, the node and client certificates are signed using new CA certificates. The nodes use the new node and CA certificates as soon as the certs directory on the node is rescanned. However, the clients use the new CA and client certificates only when the clients are restarted. Thus node certificates signed by the new CA certificate are not accepted by clients that do not have the new CA certificate yet. To ensure all nodes and clients have the latest CA certificate, rotate CA certificates on a completely different schedule; ideally, months before changing the node and client certificates. - -## See also - -- [Create Security Certificates](cockroach-cert.html) -- [Manual Deployment](manual-deployment.html) -- [Orchestrated Deployment](orchestration.html) -- [Local Deployment](secure-a-cluster.html) -- [Other Cockroach Commands](cockroach-commands.html) diff --git a/src/current/v21.1/run-multi-statement-transactions.md b/src/current/v21.1/run-multi-statement-transactions.md deleted file mode 100644 index af785750b16..00000000000 --- a/src/current/v21.1/run-multi-statement-transactions.md +++ /dev/null @@ -1,101 +0,0 @@ ---- -title: Run Multi-Statement Transactions -summary: How to use multi-statement transactions during application development -toc: true ---- - -This page has instructions for running [multi-statement transactions](transactions.html#batched-statements) against CockroachDB from various programming languages. - -## Before you begin - -Make sure you have already: - -- Set up a [local cluster](secure-a-cluster.html). -- [Installed a Postgres client](install-client-drivers.html). -- [Connected to the database](connect-to-the-database.html). -- [Inserted data](insert-data.html) that you now want to run queries against. - -{% include {{page.version.version}}/app/retry-errors.md %} - - -## Run a transaction - -
    - - - - -
    - -
    - -{% include_cached copy-clipboard.html %} -~~~ sql -BEGIN; -DELETE FROM customers WHERE id = 1; -DELETE orders WHERE customer = 1; -COMMIT; -~~~ - -For more information about how to use the built-in SQL client, see the [`cockroach sql`](cockroach-sql.html) reference docs. - -
    - -
    - -The best way to run a multi-statement transaction from Go code is to use one of the following approaches: - -- Use the [`crdb` transaction wrapper](https://github.com/cockroachdb/cockroach-go/tree/master/crdb) which automatically handles transaction retry errors if they occur, as shown in [Build a Go App with CockroachDB](build-a-go-app-with-cockroachdb.html). - -- Write your own retry loop wrapper, as shown in [Build a Go App with CockroachDB and GORM](build-a-go-app-with-cockroachdb-gorm.html) - -
    - -
    - -The best way to run a multi-statement transaction from Java is to write a wrapper method that automatically handles transaction retry errors. - -For complete examples showing how to write and use such wrapper methods, see [Build a Java App with CockroachDB](build-a-java-app-with-cockroachdb.html). - -
    - -
    - -The best way to run a multi-statement transaction from Python code is to use one of the following approaches: - -- Use the [sqlalchemy-cockroachdb](https://github.com/cockroachdb/sqlalchemy-cockroachdb) SQLAlchemy dialect, which automatically handles transaction retry errors if they occur, as shown in [Build a Python App with CockroachDB and SQLAlchemy](build-a-python-app-with-cockroachdb-sqlalchemy.html). - -- Write your own retry loop wrapper, as shown in [Build a Python App with CockroachDB](build-a-python-app-with-cockroachdb.html). - -
    - -## See also - -Reference information related to this task: - -- [Transactions](transactions.html) -- [Transaction retries](transactions.html#client-side-intervention) -- [Batched statements](transactions.html#batched-statements) -- [Understanding and Avoiding Transaction Contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention) -- [`BEGIN`](begin-transaction.html) -- [`COMMIT`](commit-transaction.html) - -Other common tasks: - -- [Connect to the Database](connect-to-the-database.html) -- [Insert Data](insert-data.html) -- [Query Data](query-data.html) -- [Update Data](update-data.html) -- [Delete Data](delete-data.html) -- [Optimize Statement Performance][fast] -- [Error Handling and Troubleshooting](error-handling-and-troubleshooting.html) -- [Hello World Example apps](example-apps.html) - - - -[selection]: selection-queries.html -[manual]: manual-deployment.html -[orchestrated]: orchestration.html -[fast]: make-queries-fast.html -[paginate]: pagination.html -[joins]: joins.html diff --git a/src/current/v21.1/savepoint.md b/src/current/v21.1/savepoint.md deleted file mode 100644 index 7a5b3389433..00000000000 --- a/src/current/v21.1/savepoint.md +++ /dev/null @@ -1,300 +0,0 @@ ---- -title: SAVEPOINT -summary: A savepoint is a marker that defines the beginning of a nested transaction. -toc: true ---- - -A savepoint is a marker that defines the beginning of a [nested transaction](transactions.html#nested-transactions). This marker can be later used to commit or roll back just the effects of the nested transaction without affecting the progress of the enclosing transaction. - - CockroachDB supports [general purpose savepoints for nested transactions](#savepoints-for-nested-transactions), in addition to continued support for [special-purpose retry savepoints](#savepoints-for-client-side-transaction-retries). - -{% include {{page.version.version}}/sql/savepoint-ddl-rollbacks.md %} - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/savepoint.html %} -
    - -## Required privileges - -No [privileges](authorization.html#assign-privileges) are required to create a savepoint. However, privileges are required for each statement within a transaction. - -## Parameters - -Parameter | Description ---------- | ----------- -name | The name of the savepoint. [Nested transactions](savepoint.html#savepoints-for-nested-transactions) can use any name for the savepoint. [Retry savepoints](savepoint.html#savepoints-for-client-side-transaction-retries) default to using the name `cockroach_restart`, but this can be customized using a session variable. For more information, see [Customizing the retry savepoint name](savepoint.html#customizing-the-retry-savepoint-name). - -## Savepoints and row locks - -{% include {{page.version.version}}/sql/savepoints-and-row-locks.md %} - -## Savepoints and high priority transactions - -{% include {{page.version.version}}/sql/savepoints-and-high-priority-transactions.md %} - -## Examples - -The examples below use the following table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE kv (k INT PRIMARY KEY, v INT); -~~~ - -### Basic usage - -To establish a savepoint inside a transaction: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SAVEPOINT foo; -~~~ - -{{site.data.alerts.callout_info}} -Due to the [rules for identifiers in our SQL grammar](keywords-and-identifiers.html#identifiers), `SAVEPOINT foo` and `SAVEPOINT Foo` define the same savepoint, whereas `SAVEPOINT "Foo"` defines another. -{{site.data.alerts.end}} - -To roll back a transaction partially to a previously established savepoint: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ROLLBACK TO SAVEPOINT foo; -~~~ - -To forget a savepoint, and keep the effects of statements executed after the savepoint was established, use [`RELEASE SAVEPOINT`](release-savepoint.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> RELEASE SAVEPOINT foo; -~~~ - -For example, the transaction below will insert the values `(1,1)` and `(3,3)` into the table, but not `(2,2)`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; -INSERT INTO kv VALUES (1,1); -SAVEPOINT my_savepoint; -INSERT INTO kv VALUES (2,2); -ROLLBACK TO SAVEPOINT my_savepoint; -INSERT INTO kv VALUES (3,3); -COMMIT; -~~~ - -### Savepoints for nested transactions - -Transactions can be nested using named savepoints. [`RELEASE SAVEPOINT`](release-savepoint.html) and [`ROLLBACK TO SAVEPOINT`](rollback-transaction.html) can both refer to a savepoint "higher" in the nesting hierarchy. When this occurs, all of the savepoints "under" the nesting are automatically released / rolled back too. Specifically: - -- When a previous savepoint is rolled back, the statements entered after that savepoint are also rolled back. - -- When a previous savepoint is released, it commits; the statements entered after that savepoint are also committed. - -For more information about nested transactions, see [Nested transactions](transactions.html#nested-transactions). - -### Multi-level rollback with `ROLLBACK TO SAVEPOINT` - -Savepoints can be arbitrarily nested, and rolled back to the outermost level so that every subsequent statement is rolled back. - -For example, this transaction does not insert anything into the table. Both `INSERT`s are rolled back: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; -SAVEPOINT foo; -INSERT INTO kv VALUES (5,5); -SAVEPOINT bar; -INSERT INTO kv VALUES (6,6); -ROLLBACK TO SAVEPOINT foo; -COMMIT; -~~~ - -### Multi-level commit with `RELEASE SAVEPOINT` - -Changes committed by releasing a savepoint commit all of the statements entered after that savepoint. - -For example, the following transaction inserts both `(2,2)` and `(4,4)` into the table when it releases the outermost savepoint: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; -SAVEPOINT foo; -INSERT INTO kv VALUES (2,2); -SAVEPOINT bar; -INSERT INTO kv VALUES (4,4); -RELEASE SAVEPOINT foo; -COMMIT; -~~~ - -### Multi-level rollback and commit in the same transaction - -Changes partially committed by a savepoint release can be rolled back by an outer savepoint. - -For example, the following transaction inserts only value `(5, 5)`. The values `(6,6)` and `(7,7)` are rolled back. - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; -INSERT INTO kv VALUES (5,5); -SAVEPOINT foo; -INSERT INTO kv VALUES (6,6); -SAVEPOINT bar; -INSERT INTO kv VALUES (7,7); -RELEASE SAVEPOINT bar; -ROLLBACK TO SAVEPOINT foo; -COMMIT; -~~~ - -### Error recovery in nested transactions with `ROLLBACK TO SAVEPOINT` - -If `ROLLBACK TO SAVEPOINT` is used after a database error, it can also cancel the error state of the transaction. Database errors move a transaction (or nested transaction) into an "Aborted" state. In this state, the transaction will not execute any further SQL statements. - -You can use `ROLLBACK TO SAVEPOINT` to recover from a logical error in a nested transaction. Logical errors include: - -- Unique index error (duplicate row) -- Failed foreign key constraint check (row does not exist in referenced table) -- Mistakes in queries (reference a column that does not exist) - -In addition, you can check the status of a nested transaction using the `SHOW TRANSACTION STATUS` statement as shown below. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; -SAVEPOINT error1; -INSERT INTO kv VALUES (5,5); -- Duplicate key error -~~~ - -~~~ -ERROR: duplicate key value (k)=(5) violates unique constraint "primary" -SQLSTATE: 23505 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW TRANSACTION STATUS; -~~~ - -~~~ - TRANSACTION STATUS ----------------------- - Aborted -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -ROLLBACK TO SAVEPOINT error1; -INSERT INTO kv VALUES (6,6); -COMMIT; -~~~ - -### Savepoint name visibility - -The name of a savepoint that was rolled back over is no longer visible afterward. - -For example, in the transaction below, the name "bar" is not visible after it was rolled back over: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; -SAVEPOINT foo; -SAVEPOINT bar; -ROLLBACK TO SAVEPOINT foo; -RELEASE SAVEPOINT bar; -COMMIT; -~~~ - -~~~ -ERROR: savepoint bar does not exist -SQLSTATE: 3B001 -~~~ - -The [SQL client](cockroach-sql.html) prompt will now display an error state, which you can clear by entering [`ROLLBACK`](rollback-transaction.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -? ERROR> ROLLBACK; -~~~ - -~~~ -ROLLBACK -~~~ - -#### Savepoints and prepared statements - -Prepared statements (`PREPARE` / `EXECUTE`) are not transactional. Therefore, prepared statements are not invalidated upon savepoint rollback. As a result, the prepared statement was saved and executed inside the transaction, despite the rollback to the prior savepoint: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; -SAVEPOINT foo; -PREPARE bar AS SELECT 1; -ROLLBACK TO SAVEPOINT foo; -EXECUTE bar; -COMMIT; -~~~ - -~~~ - ?column? ------------- - 1 -(1 row) -~~~ - -### Savepoints for client-side transaction retries - -{% include {{page.version.version}}/sql/retry-savepoints.md %} - -The example below shows basic usage of a retry savepoint. - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; -SAVEPOINT cockroach_restart; -UPDATE products SET inventory = 0 WHERE sku = '8675309'; -INSERT INTO orders (customer, sku, status) VALUES (1001, '8675309', 'new'); -RELEASE SAVEPOINT cockroach_restart; -COMMIT; -~~~ - -Applications using `SAVEPOINT` for client-side transaction retries must also include functions to execute retries with [`ROLLBACK TO SAVEPOINT `](rollback-transaction.html#retry-a-transaction). - -Note that you can [customize the retry savepoint name](#customizing-the-retry-savepoint-name) to something other than `cockroach_restart` with a session variable if you need to. - -#### Customizing the retry savepoint name - -{% include {{page.version.version}}/misc/customizing-the-savepoint-name.md %} - -### Showing savepoint status - -Use the [`SHOW SAVEPOINT STATUS`](show-savepoint-status.html) statement to see how many savepoints are active in the current transaction: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SAVEPOINT STATUS; -~~~ - -~~~ - savepoint_name | is_initial_savepoint ------------------+----------------------- - foo | true - bar | false - baz | false -(3 rows) -~~~ - -Note that the `is_initial_savepoint` column will be true if the savepoint is the outermost savepoint in the transaction. - -## See also - -- [`SHOW SAVEPOINT STATUS`](show-savepoint-status.html) -- [`RELEASE SAVEPOINT`](release-savepoint.html) -- [`ROLLBACK`](rollback-transaction.html) -- [`BEGIN`](begin-transaction.html) -- [`COMMIT`](commit-transaction.html) -- [Transactions](transactions.html) -- [Retryable transaction example code in Java using JDBC](build-a-java-app-with-cockroachdb.html) -- [CockroachDB Architecture: Transaction Layer](architecture/transaction-layer.html) diff --git a/src/current/v21.1/scalar-expressions.md b/src/current/v21.1/scalar-expressions.md deleted file mode 100644 index 873eeee4d14..00000000000 --- a/src/current/v21.1/scalar-expressions.md +++ /dev/null @@ -1,890 +0,0 @@ ---- -title: Scalar Expressions -summary: Scalar expressions allow the computation of new values from basic parts. -toc: true -key: sql-expressions.html ---- - -Most SQL statements can contain *scalar expressions* that compute new -values from data. For example, in the query `SELECT ceil(price) FROM -items`, the expression `ceil(price)` computes the rounded-up value of -the values from the `price` column. - -Scalar expressions produce values suitable to store in a single table -cell (one column of one row). They can be contrasted with -[table expressions](table-expressions.html) and [selection queries](selection-queries.html), which produce results -structured as a table. - -The following sections provide details on each of these options. - - -## Constants - -Constant expressions represent a simple value that doesn't change. -They are described further in section [SQL Constants](sql-constants.html). - -## Column references - -An expression in a query can refer to columns in the current data source in two ways: - -- Using the name of the column, e.g., `price` in `SELECT price FROM - items`. - - - If the name of a column is also a - [SQL keyword](keywords-and-identifiers.html#keywords), the name - must be appropriately quoted. For example: `SELECT "Default" FROM - configuration`. - - - If the name is ambiguous (e.g., when joining across multiple - tables), it is possible to disambiguate by prefixing the column - name by the table name. For example, `SELECT items.price FROM - items`. - -- Using the ordinal position of the column. For example, `SELECT @1 - FROM items` selects the first column in `items`. - - *This is a CockroachDB SQL extension.* - - {{site.data.alerts.callout_danger}} - Ordinal references should be used with care in production - code! During schema updates, column ordinal positions can change and - invalidate existing queries that use ordinal positions based on a - previous version of the schema. - {{site.data.alerts.end}} - -## Unary and binary operations - -An expression prefixed by a unary operator, or two expressions -separated by a binary operator, form a new expression. - -For a full list of CockroachDB operators, with details about their order of precedence and which data types are valid operands for each operator, see [Functions and Operators](functions-and-operators.html#operators). - -### Value comparisons - -The standard operators `<` (smaller than), `>` (greater than), `<=` -(lower than or equal to), `>=` (greater than or equal to), `=` -(equals), `<>` and `!=` (not equal to), `IS` (identical to), and `IS -NOT` (not identical to) can be applied to any pair of values from a -single data type, as well as some pairs of values from different data -types. - -See also [this section over which data types are valid operands -for each operator](functions-and-operators.html#operators). - -The following special rules apply: - -- `NULL` is always ordered smaller than every other value, even itself. -- `NULL` is never equal to anything via `=`, even `NULL`. To check - whether a value is `NULL`, use the `IS` operator or the conditional - expression `IFNULL(..)`. - -See also [NULLs and Ternary Logic](null-handling.html#nulls-and-ternary-logic). - -#### Typing rule - -All comparisons accept any combination of argument types and result in type `BOOL`. - -#### Comparison with `NaN` - -CockroachDB recognizes the special value `NaN` -([Not-a-Number](https://en.wikipedia.org/wiki/NaN)) for scalars of -type [`FLOAT`](float.html) or [`DECIMAL`](decimal.html). - -As per the [IEEE 754](https://en.wikipedia.org/wiki/IEEE_754) -standard, `NaN` is considered to be different from every other numeric -value in comparisons. - -There are two exceptions however, made for compatibility with PostgreSQL: - -- `NaN` is considered to be equal with itself in comparisons. IEEE 754 - specifies that `NaN` is different from itself. -- `NaN` is considered to be smaller than every other value, including - `-INFINITY`. IEEE 754 specifies that `NaN` does not order with any - other value, i.e., `x <= NaN` and `x >= NaN` are both false for every - value of `x` including infinities. - -These exceptions exist so that the value `NaN` can be used in `WHERE` -clauses and indexes. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT FLOAT 'NaN' < 1, 1 < FLOAT 'NaN', FLOAT 'NaN' < FLOAT 'NaN'; -~~~ - -~~~ - ?column? | ?column? | ?column? ------------+----------+----------- - true | false | false -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT FLOAT 'NaN' = FLOAT 'NaN' AS result; -~~~ - -~~~ - result ----------- - true -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT FLOAT 'NaN' < FLOAT '-INFINITY' AS result; -~~~ - -~~~ - result ----------- - true -(1 row) -~~~ - -### Multi-valued comparisons - -Syntax: - -~~~ - ANY - SOME - ALL -~~~ - -The value comparison operators `<`, `>`, `=`, `<=`, `>=`, `<>` and -`!=`, as well as the pattern matching operators `[NOT] LIKE` and -`[NOT] ILIKE`, can be applied to compare a single value on the left to -multiple values on the right. - -This is done by combining the operator using the keywords `ANY`/`SOME` or `ALL`. - -The right operand can be either an array, a tuple or [subquery](subqueries.html). - -The result of the comparison is true if and only if: - -- For `ANY`/`SOME`, the comparison of the left value is true for any - element on the right. -- For `ALL`, the comparison of the left value is true for every - element on the right. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT 12 = ANY (10, 12, 13); -~~~ - -~~~ - ?column? ------------- - true -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT 12 = ALL (10, 12, 13); -~~~ - -~~~ - ?column? ------------- - false -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT 1 = ANY ARRAY[2, 3, 1]; -~~~ - -~~~ - ?column? ------------- - true -(1 row) -~~~ - - -#### Typing rule - -The comparison between the type on the left and the element type of -the right operand must be possible. - -### Set membership - -Syntax: - -~~~ - IN - IN ( ... subquery ... ) - - NOT IN - NOT IN ( ... subquery ... ) -~~~ - -Returns `TRUE` if and only if the value of the left operand is part of -the result of evaluating the right operand. In the subquery form, any -[selection query](selection-queries.html) can be used. - -For example: - -{% include_cached copy-clipboard.html %} -~~~sql -> SELECT a IN (1, 2, 3) FROM sometable; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT a IN (SELECT * FROM allowedvalues) FROM sometable; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT ('x', 123) IN (SELECT * FROM rows); -~~~ - -{{site.data.alerts.callout_info}}See Subqueries for more details and performance best practices.{{site.data.alerts.end}} - -#### Typing rule - -`IN` requires its right operand to be a homogeneous tuple type and its left operand -to match the tuple element type. The result has type `BOOL`. - -### String pattern matching - -Syntax: - -~~~ - LIKE - ILIKE - NOT LIKE - NOT ILIKE -~~~ - -Evaluates both expressions as strings, then tests whether the string on the left -matches the pattern given on the right. Returns `TRUE` if a match is found -or `FALSE` otherwise, or the inverted value for the `NOT` variants. - -Patterns can contain `_` to match any single -character, or `%` to match any sequence of zero or more characters. -`ILIKE` causes the match to be tested case-insensitively. - -For example: - -{% include_cached copy-clipboard.html %} -~~~sql -> SELECT 'monday' LIKE '%day' AS a, 'tuesday' LIKE 'tue_day' AS b, 'wednesday' ILIKE 'W%' AS c; -~~~ - -~~~ - a | b | c --------+------+------- - true | true | true -(1 row) -~~~ - -#### Typing rule - -The operands must be either both `STRING` or both `BYTES`. The result has type `BOOL`. - -### String matching using POSIX regular expressions - -Syntax: - -~~~ - ~ - ~* - !~ - !~* -~~~ - -Evaluates both expressions as strings, then tests whether the string -on the left matches the pattern given on the right. Returns `TRUE` if -a match is found or `FALSE` otherwise, or the inverted value for the -`!` variants. - -The variants with an asterisk `*` use case-insensitive matching; -otherwise the matching is case-sensitive. - -The pattern is expressed using -[POSIX regular expression syntax](https://en.wikipedia.org/wiki/Regular_expression). Unlike -`LIKE` patterns, a regular expression is allowed to match anywhere -inside a string, not only at the beginning. - -For example: - -{% include_cached copy-clipboard.html %} -~~~sql -> SELECT 'monday' ~ 'onday' AS a, 'tuEsday' ~ 't[uU][eE]sday' AS b, 'wednesday' ~* 'W.*y' AS c; -~~~ - -~~~ - a | b | c --------+------+------- - true | true | true -(1 row) -~~~ - -#### Typing rule - -The operands must be either both `STRING` or both `BYTES`. The result has type `BOOL`. - -### String matching using SQL regular expressions - -Syntax: - -~~~ - SIMILAR TO - NOT SIMILAR TO -~~~ - -Evaluates both expressions as strings, then tests whether the string on the left -matches the pattern given on the right. Returns `TRUE` if a match is found -or `FALSE` otherwise, or the inverted value for the `NOT` variant. - -The pattern is expressed using the SQL standard's definition of a regular expression. -This is a mix of SQL `LIKE` patterns and POSIX regular expressions: - -- `_` and `%` denote any character or any string, respectively. -- `.` matches specifically the period character, unlike in POSIX where it is a wildcard. -- Most of the other POSIX syntax applies as usual. -- The pattern matches the entire string (as in `LIKE`, unlike POSIX regular expressions). - -For example: - -{% include_cached copy-clipboard.html %} -~~~sql -> SELECT 'monday' SIMILAR TO '_onday' AS a, 'tuEsday' SIMILAR TO 't[uU][eE]sday' AS b, 'wednesday' SIMILAR TO 'w%y' AS c; -~~~ - -~~~ - a | b | c --------+------+------- - true | true | true -(1 row) -~~~ - -#### Typing rule - -The operands must be either both `STRING` or both `BYTES`. The result has type `BOOL`. - -## Function calls and SQL special forms - -General syntax: - -~~~ - ( ) -~~~ - -A built-in function name followed by an opening parenthesis, followed -by a comma-separated list of expressions, followed by a closing -parenthesis. - -This applies the named function to the arguments between -parentheses. When the function's namespace is not prefixed, the -[name resolution rules](sql-name-resolution.html) determine which -function is called. - -See also [the separate section on supported built-in functions](functions-and-operators.html). - -In addition, the following SQL special forms are also supported: - -{% include {{ page.version.version }}/sql/function-special-forms.md %} - -#### Typing rule - -In general, a function call requires the arguments to be of the types -accepted by the function, and returns a value of the type determined -by the function. - -However, the typing of function calls is complicated by the fact -SQL supports function overloading. [See our blog post for more details](https://www.cockroachlabs.com/blog/revisiting-sql-typing-in-cockroachdb/). - -## Subscripted expressions - -It is possible to access one item in an array value using the `[` ... `]` operator. - -For example, if the name `a` refers to an array of 10 -values, `a[3]` will retrieve the 3rd value. The first value has index -1. - -If the index is smaller or equal to 0, or larger than the size of the array, then -the result of the subscripted expression is `NULL`. - -#### Typing rule - -The subscripted expression must have an array type; the index expression -must have type `INT`. The result has the element type of the -subscripted expression. - -## Conditional expressions - -Expressions can test a conditional expression and, depending on whether -or which condition is satisfied, evaluate to one or more additional -operands. - -These expression formats share the following property: some of their -operands are only evaluated if a condition is true. This matters -especially when an operand would be invalid otherwise. For example, -`IF(a=0, 0, x/a)` returns 0 if `a` is 0, and `x/a` otherwise. - -### `IF` expressions - -Syntax: - -~~~ -IF ( , , ) -~~~ - -Evaluates ``, then evaluates `` if the condition is true, -or `` otherwise. - -The expression corresponding to the case when the condition is false -is not evaluated. - -#### Typing rule - -The condition must have type `BOOL`, and the two remaining expressions -must have the same type. The result has the same type as the -expression that was evaluated. - -### Simple `CASE` expressions - -Syntax: - -~~~ -CASE - WHEN THEN - [ WHEN THEN ] ... - [ ELSE ] -END -~~~ - -Evaluates ``, then picks the `WHEN` branch where `` is -equal to ``, then evaluates and returns the corresponding `THEN` -expression. If no `WHEN` branch matches, the `ELSE` expression is -evaluated and returned, if any. Otherwise, `NULL` is returned. - -Conditions and result expressions after the first match are not evaluated. - -#### Typing rule - -The condition and the `WHEN` expressions must have the same type. -The `THEN` expressions and the `ELSE` expression, if any, must have the same type. -The result has the same type as the `THEN`/`ELSE` expressions. - -### Searched `CASE` expressions - -Syntax: - -~~~ -CASE WHEN THEN - [ WHEN THEN ] ... - [ ELSE ] -END -~~~ - -In order, evaluates each `` expression; at the first `` -expression that evaluates to `TRUE`, returns the result of evaluating the -corresponding `THEN` expression. If none of the `` expressions -evaluates to true, then evaluates and returns the value of the `ELSE` -expression, if any, or `NULL` otherwise. - -Conditions and result expressions after the first match are not evaluated. - -#### Typing rule - -All the `WHEN` expressions must have type `BOOL`. -The `THEN` expressions and the `ELSE` expression, if any, must have the same type. -The result has the same type as the `THEN`/`ELSE` expressions. - -### `NULLIF` expressions - -Syntax: - -~~~ -NULLIF ( , ) -~~~ - -Equivalent to: `IF ( = , NULL, )` - -#### Typing rule - -Both operands must have the same type, which is also the type of the result. - -### `COALESCE` and `IFNULL` expressions - -Syntax: - -~~~ -IFNULL ( , ) -COALESCE ( [, [, ] ...] ) -~~~ - -`COALESCE` evaluates the first expression first. If its value is not -`NULL`, its value is returned directly. Otherwise, it returns the -result of applying `COALESCE` on the remaining expressions. If all the -expressions are `NULL`, `NULL` is returned. - -Arguments to the right of the first non-null argument are not evaluated. - -`IFNULL(a, b)` is equivalent to `COALESCE(a, b)`. - -#### Typing rule - -The operands must have the same type, which is also the type of the result. - -## Logical operators - -The Boolean operators `AND`, `OR` and `NOT` are available. - -Syntax: - -~~~ -NOT - AND - OR -~~~ - -`AND` and `OR` are commutative. Moreover, the input to `AND` -and `OR` is not evaluated in any particular order. Some operand may -not even be evaluated at all if the result can be fully ascertained using -only the other operand. - -{{site.data.alerts.callout_info}}This is different from the left-to-right "short-circuit logic" found in other programming languages. When it is essential to force evaluation order, use a conditional expression.{{site.data.alerts.end}} - -See also [NULLs and Ternary Logic](null-handling.html#nulls-and-ternary-logic). - -### Typing rule - -The operands must have type `BOOL`. The result has type `BOOL`. - -## Aggregate expressions - -An aggregate expression has the same syntax as a function call, with a special -case for `COUNT`: - -~~~ - ( ) -COUNT ( * ) -~~~ - -The difference between aggregate expressions and function calls is -that the former use -[aggregate functions](functions-and-operators.html#aggregate-functions) -and can only appear in the list of rendered expressions in a -[`SELECT` clause](select-clause.html). - -An aggregate expression computes a combined value, depending on -which aggregate function is used, across all the rows currently -selected. - -#### Typing rule - -[The operand and return types are determined like for regular function calls](#function-calls-and-sql-special-forms). - -## Window function calls - -A window function call has the syntax of a function call followed by an `OVER` clause: - -~~~ - ( ) OVER - ( * ) OVER -~~~ - -It represents the application of a window or aggregate function over a -subset ("window") of the rows selected by a query. - -#### Typing rule - -[The operand and return types are determined like for regular function calls](#function-calls-and-sql-special-forms). - -## Explicit type coercions - -Syntax: - -~~~ - :: -CAST ( AS ) -~~~ - -Evaluates the expression and converts the resulting value to the -specified type. An error is reported if the conversion is invalid. - -For example: `CAST(now() AS DATE)` - -Note that in many cases a type annotation is preferrable to a type -coercion. See the section on -[type annotations](#explicitly-typed-expressions) below for more -details. - -#### Typing rule - -The operand can have any type. -The result has the type specified in the `CAST` expression. - -As a special case, if the operand is a literal, a constant expression -or a placeholder, the `CAST` type is used to guide the typing of the -operand. [See our blog post for more details](https://www.cockroachlabs.com/blog/revisiting-sql-typing-in-cockroachdb/). - -## Collation expressions - -Syntax: - -~~~ - COLLATE -~~~ - -Evaluates the expression and converts its result to a collated string -with the specified collation. - -For example: `'a' COLLATE de` - -#### Typing rule - -The operand must have type `STRING`. The result has type `COLLATEDSTRING`. - -## Array constructors - -Syntax: - -~~~ -ARRAY[ , , ... ] -~~~ - -Evaluates to an array containing the specified values. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT ARRAY[1,2,3] AS a; -~~~ - -~~~ - a ------------ - {1,2,3} -(1 row) -~~~ - -The data type of the array is inferred from the values of the provided -expressions. All the positions in the array must have the same data type. - -If there are no expressions specified (empty array), or -all the values are `NULL`, then the type of the array must be -specified explicitly using a type annotation. For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT ARRAY[]:::int[]; -~~~ - -{{site.data.alerts.callout_success}}To convert the results of a subquery to an array, use ARRAY(...) instead.{{site.data.alerts.end}} - -{{site.data.alerts.callout_success}}CockroachDB also recognizes the syntax ARRAY(a, b, c) as an alias for ARRAY[a, b, c]. This is an experimental, CockroachDB-specific SQL extension and may be removed in a later version of CockroachDB.{{site.data.alerts.end}} - -#### Typing rule - -The operands must all have the same type. -The result has the array type with the operand type as element type. - -## Tuple constructor - -Syntax: - -~~~ -(, , ...) -ROW (, , ...) -~~~ - -Evaluates to a tuple containing the values of the provided expressions. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT ('x', 123, 12.3) AS a; -~~~ - -~~~ - a ----------------- - (x,123,12.3) -(1 row) -~~~ - -The data type of the resulting tuple is inferred from the values. -Each position in a tuple can have a distinct data type. - -CockroachDB supports accessing the Nth element in a tuple as a single table cell using the syntax `(...).@N`. For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT (t).@2 FROM (SELECT (1,'b',2.3) AS t); -~~~ - -~~~ - ?column? ------------- - b -(1 row) -~~~ - -CockroachDB also supports expanding all elements of a tuple as a single row in a table with the `().*` notation. This notation works as the inverse of the tuple-creating notation `(.*)`. For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> WITH tuples AS (SELECT (t.*) AS tuple FROM (SELECT 1,'b',2.3) AS t(x,y,z)) -- Build the tuples, with labels - SELECT (tuple).* FROM tuples; -- Expands the tuples and restore the column labels -~~~ - -~~~ - x | y | z -----+---+------ - 1 | b | 2.3 -(1 row) -~~~ - - -#### Typing rule - -The operands can have any type. -The result has a tuple type whose item types are the types of the operands. - -## Explicitly typed expressions - -Syntax: - -~~~ -::: -ANNOTATE_TYPE(, ) -~~~ - -Evaluates to the given expression, requiring the expression to have -the given type. If the expression doesn't have the given type, an -error is returned. - -Type annotations are specially useful to guide the arithmetic on -numeric values. For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT (1 / 0):::FLOAT; -~~~ - -~~~ -ERROR: division by zero -SQLSTATE: 22012 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT (1 / 0); -~~~ - -~~~ -ERROR: division by zero -SQLSTATE: 22012 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT (1 / 0)::FLOAT; -~~~ - -~~~ -ERROR: division by zero -SQLSTATE: 22012 -~~~ - -Type annotations are also different from cast expressions (see above) in -that they do not cause the value to be converted. For example, -`now()::DATE` converts the current timestamp to a date value (and -discards the current time), whereas `now():::DATE` triggers an error -message (that `now()` does not have type `DATE`). - -Check our blog for -[more information about context-dependent typing](https://www.cockroachlabs.com/blog/revisiting-sql-typing-in-cockroachdb/). - -#### Typing rule - -The operand must be implicitly coercible to the given type. -The result has the given type. - -## Subquery expressions - -### Scalar subqueries - -Syntax: - -~~~ -( ... subquery ... ) -~~~ - -Evaluates the subquery, asserts that it returns a single row and single column, -and then evaluates to the value of that single cell. Any [selection query](selection-queries.html) -can be used as subquery. - -For example, the following query returns `TRUE` if there are more rows in table `users` than in table -`admins`: - -{% include_cached copy-clipboard.html %} -~~~sql -> SELECT (SELECT COUNT(*) FROM users) > (SELECT COUNT(*) FROM admins); -~~~ - -{{site.data.alerts.callout_info}}See Subqueries for more details and performance best practices.{{site.data.alerts.end}} - -#### Typing rule - -The operand must have a table type with only one column. -The result has the type of that single column. - -### Existence test on the result of subqueries - -Syntax: - -~~~ -EXISTS ( ... subquery ... ) -NOT EXISTS ( ... subquery ... ) -~~~ - -Evaluates the subquery and then returns `TRUE` or `FALSE` depending on -whether the subquery returned any row (for `EXISTS`) or didn't return -any row (for `NOT EXISTS`). Any [selection query](selection-queries.html) -can be used as subquery. - -{{site.data.alerts.callout_info}}See Subqueries for more details and performance best practices.{{site.data.alerts.end}} - -#### Typing rule - -The operand can have any table type. The result has type `BOOL`. - -### Conversion of subquery results to an array - -Syntax: - -~~~ -ARRAY( ... subquery ... ) -~~~ - -Evaluates the subquery and converts its results to an array. Any -[selection query](selection-queries.html) can be used as subquery. - -{{site.data.alerts.callout_info}}See Subqueries for more details and performance best practices.{{site.data.alerts.end}} - -{{site.data.alerts.callout_success}}To convert a list of scalar expressions to an array, use ARRAY[...] instead.{{site.data.alerts.end}} - -## See also - -- [Constants](sql-constants.html) -- [Selection Queries](selection-queries.html) -- [Table Expressions](table-expressions.html) -- [Data Types](data-types.html) -- [Functions and Operators](functions-and-operators.html) -- [Subqueries](subqueries.html) diff --git a/src/current/v21.1/schema-design-database.md b/src/current/v21.1/schema-design-database.md deleted file mode 100644 index 3ab8808fb38..00000000000 --- a/src/current/v21.1/schema-design-database.md +++ /dev/null @@ -1,103 +0,0 @@ ---- -title: Create a Database -summary: Best practices for creating databases in CockroachDB. -toc: true ---- - -This page provides best-practice guidance on creating databases, with a couple examples based on Cockroach Labs' fictional vehicle-sharing company, [MovR](movr.html). - -{{site.data.alerts.callout_success}} -For reference documentation on the `CREATE DATABASE` statement, including additional examples, see the [`CREATE DATABASE` syntax page](create-database.html). -{{site.data.alerts.end}} - -## Before you begin - -Before reading this page, do the following: - -- [Install CockroachDB](install-cockroachdb.html). -- [Start a local cluster](secure-a-cluster.html), or [create a CockroachDB {{ site.data.products.cloud }} cluster](../cockroachcloud/create-your-cluster.html). -- [Review the database schema objects](schema-design-overview.html). - -## Create a database - -Database objects make up the first level of the [CockroachDB naming hierarchy](sql-name-resolution.html#naming-hierarchy). - -To create a database, use a [`CREATE DATABASE` statement](create-database.html), following [the database best practices](#database-best-practices). After reviewing the best practices, see the examples we provide [below](#example). - -### Database best practices - -Here are some best practices to follow when creating and using databases: - -- Do not use the preloaded `defaultdb` database. Instead, create your own database with a `CREATE DATABASE` statement, and change it to the SQL session's [current database](sql-name-resolution.html#current-database) by executing a `USE {databasename};` statement, by passing the `--database={databasename}` flag to the [`cockroach sql` command](cockroach-sql.html#general), or by specifying the `database` parameter in the [connection string](connection-parameters.html#connect-using-a-url) passed to your database schema migration tool. - -- Create databases and [user-defined schemas](schema-design-schema.html) as a member of [the `admin` role](authorization.html#admin-role) (e.g., as the [`root` user](authorization.html#root-user)), and create all other lower-level objects as a [different user](schema-design-overview.html#controlling-access-to-objects), with fewer privileges, following [authorization best practices](authorization.html#authorization-best-practices). - -- Limit the number of databases you create. If you need to create multiple tables with the same name in your cluster, do so in different [user-defined schemas](schema-design-schema.html), in the same database. - -- {% include {{page.version.version}}/sql/dev-schema-changes.md %} - -### Example - -Create an empty file with the `.sql` file extension at the end of the filename. This file will initialize the database that will store all of the data for the MovR application. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ touch dbinit.sql -~~~ - -Open `dbinit.sql` in a text editor, and, at the top of the file, add a `CREATE DATABASE` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE DATABASE IF NOT EXISTS movr; -~~~ - -This statement will create a database named `movr`, if one does not already exist. - -To execute the statement in the `dbinit.sql` file as the `root` user, run the following command: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql \ ---certs-dir={certs-directory} \ ---user=root \ --f dbinit.sql -~~~ - -To view the database in the cluster, execute a [`SHOW DATABASES`](show-databases.html) statement from the command line: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql \ ---certs-dir={certs-directory} \ ---user=root \ ---execute="SHOW DATABASES;" -~~~ - -~~~ - database_name | owner | primary_region | regions | survival_goal -----------------+-------+----------------+---------+---------------- - defaultdb | root | NULL | {} | NULL - movr | root | NULL | {} | NULL - postgres | root | NULL | {} | NULL - system | node | NULL | {} | NULL -(4 rows) -~~~ - -You're now ready to start adding user-defined schemas to the `movr` database. - -For guidance on creating user-defined schemas, see at [Create a User-defined Schema](schema-design-schema.html). - -## What's next? - -- [Create a User-defined Schema](schema-design-schema.html) -- [Create a Table](schema-design-table.html) - -You might also be interested in the following pages: - -- [`CREATE DATABASE`](create-database.html) -- [Cockroach Commands](cockroach-commands.html) -- [Schema Design Overview](schema-design-overview.html) -- [CockroachDB naming hierarchy](sql-name-resolution.html#naming-hierarchy) diff --git a/src/current/v21.1/schema-design-indexes.md b/src/current/v21.1/schema-design-indexes.md deleted file mode 100644 index 856f87692af..00000000000 --- a/src/current/v21.1/schema-design-indexes.md +++ /dev/null @@ -1,279 +0,0 @@ ---- -title: Add Secondary Indexes -summary: Best practices for working with secondary indexes in CockroachDB. -toc: true -keywords: gin, gin index, gin indexes, inverted index, inverted indexes, accelerated index, accelerated indexes ---- - -This page provides best-practice guidance on creating indexes, with a simple example based on Cockroach Labs' fictional vehicle-sharing company, [MovR](movr.html). - -{{site.data.alerts.callout_success}} -For detailed reference documentation on the `CREATE INDEX` statement, including additional examples, see the [`CREATE INDEX` syntax page](create-index.html). -{{site.data.alerts.end}} - -## Before you begin - -Before reading this page, do the following: - -- [Install CockroachDB](install-cockroachdb.html). -- [Start a local cluster](secure-a-cluster.html), or [create a CockroachDB {{ site.data.products.cloud }} cluster](../cockroachcloud/create-your-cluster.html). -- [Review the database schema objects](schema-design-overview.html). -- [Create a database](schema-design-database.html). -- [Create a user-defined schema](schema-design-schema.html). -- [Create a table](schema-design-table.html). - -## Create a secondary index - -Indexes are [logical objects in a cluster](schema-design-overview.html#database-schema-objects) that help [CockroachDB queries](query-data.html) find data more efficiently. When you create an index, CockroachDB creates a copy of the columns selected for the index, and then sorts the rows of data by indexed column values, without sorting the values in the table itself. - -CockroachDB automatically creates an index on the table's [primary key](primary-key.html) columns. This index is called the *primary index*. The primary index helps CockroachDB more efficiently scan rows, as sorted by the table's primary key columns, but it does not help find values as identified by any other columns. - -*Secondary indexes* (i.e., all indexes that are not the primary index) improve the performance of queries that identify rows with columns that are not in a table's primary key. CockroachDB automatically creates secondary indexes for columns with a [`UNIQUE` constraint](unique.html). - -To add a secondary index to a table, do one of the following, following the [best practices listed below](#best-practices): - -- Add an `INDEX` clause to the end of a [`CREATE TABLE`](create-table.html#create-a-table-with-secondary-and-gin-indexes) statement. - - `INDEX` clauses generally take the following form: - - ~~~ - INDEX {index_name} ({column_names}); - ~~~ - - Parameter | Description - ----------|------------ - `{index_name}` | The name of the index. - `{column_names}` | The name of the column to index, or a comma-separated list of names of the columns to index. - -- Use a [`CREATE INDEX`](create-index.html) statement. - - `CREATE INDEX` statements generally take the following form: - - ~~~ - CREATE INDEX {index_name} ON {table_name} ({column_names}); - ~~~ - - Parameter | Description - ----------|------------ - `{index_name}` | The name of the index. - `{table_name}` | The name of the table. - `{column_names}` | The name of the column to index, or a comma-separated list of names of the columns to index. - -For an example, see [below](#example). - -{{site.data.alerts.callout_info}} -If you do not specify a name for an index, CockroachDB will generate a name. - -After creation, the notation for referring to indexes in CockroachDB is `{table_name}@{index_name}`. -{{site.data.alerts.end}} - -### Best practices - -Here are some best practices for creating indexes: - -- Index all columns that you plan to use for [sorting](order-by.html) or [filtering](select-clause.html#filter-rows) data. - - {% include {{page.version.version}}/sql/covering-index.md %} - - Note that columns listed in a filtering [`WHERE` clause](select-clause.html#parameters) with the equality operators (`=` or `IN`) should come first in the index, before those referenced with inequality operators (`<`, `>`). - - Columns with a higher cardinality (higher number of distinct values) should be placed in the index before columns with a lower cardinality. If the cardinality of the columns you wish to add to the index are similar, test multiple column arrangements in a non-production environment to determine the most performant arrangement. - -- Avoid indexing on sequential values. - - Writes to indexes with sequential keys can result in range hotspots that negatively affect performance. Instead, use [randomly generated unique IDs](performance-best-practices-overview.html#unique-id-best-practices), or [multi-column keys](performance-best-practices-overview.html#use-multi-column-primary-keys). - - If you are working with a table that *must* be indexed on sequential keys, use [hash-sharded indexes](hash-sharded-indexes.html). For details about the mechanics and performance improvements of hash-sharded indexes in CockroachDB, see our [Hash Sharded Indexes Unlock Linear Scaling for Sequential Workloads](https://www.cockroachlabs.com/blog/hash-sharded-indexes-unlock-linear-scaling-for-sequential-workloads/) blog post. - -- Avoid creating secondary indexes that you do not need, and [drop unused indexes](drop-index.html) whenever possible. Secondary indexes can slow down write performance and take up node memory. - - Note that queries can benefit from an index even if they only filter a prefix of its columns. For example, if you create an index of columns `(A, B, C)`, queries filtering `(A)` or `(A, B)` can still use the index. However, queries that do not filter `(A)` will not benefit from the index. This feature also lets you avoid using single-column indexes. Instead, use the column as the first column in a multiple-column index, which is useful to more queries. - - Also note that [`ALTER PRIMARY KEY`](alter-primary-key.html) creates a secondary index from the old primary key. If you need to [change a primary key](constraints.html#change-constraints), and you do not plan to filter queries on the old primary key column(s), do not use `ALTER PRIMARY KEY`. Instead, use [`DROP CONSTRAINT ... PRIMARY KEY`/`ADD CONSTRAINT ... PRIMARY KEY`](add-constraint.html#changing-primary-keys-with-add-constraint-primary-key), which does not create a secondary index. - -- Limit creation and deletion of secondary indexes to off-peak hours. Performance impacts are likely if done during peak business hours. - - - -- Use a [`STORING` clause](create-index.html#parameters) to store columns of data that you want returned by common queries, but that you do not plan to use in query filters. Note that the synonym `COVERING` is also supported. - - The `STORING` clause specifies columns that are not part of the index key but should be stored in the index, without being sorted. If a column is specified in a query, and the column is neither indexed nor stored in an index, CockroachDB will perform a full scan of the table, which can result in poor performance. For an example, see [below](#example). - -- Review the [specialized indexes that CockroachDB supports](schema-design-overview.html#specialized-indexes), and decide if you need to create a specialized index instead of a standard index. - -- Do not create indexes as the `root` user. Instead, create indexes as a [different user](schema-design-overview.html#controlling-access-to-objects), with fewer privileges, following [authorization best practices](authorization.html#authorization-best-practices). This will likely be the same user that created the table to which the index belongs. - -- {% include {{page.version.version}}/sql/dev-schema-changes.md %} - -- {% include {{page.version.version}}/sql/dev-schema-change-limits.md %} - -### Example - -Suppose you want the MovR application to display all of the bikes available to the users of the MovR platform. - -Recall that the `vehicles` table that you created in [Create a Table](schema-design-table.html) stores rows of data for each vehicle registered with MovR. Your application will need to read any data about vehicles into the application's persistence layer from this table. To display available bikes, the reads will need to filter on the `available` and `type` columns. - -Open `max_init.sql`, and, under the `CREATE TABLE` statement for the `vehicles` table, add a `CREATE INDEX` statement for an index on the `type` and `available` columns of the `vehicles` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE INDEX type_available_idx ON movr.vehicles (type, available)); -~~~ - -This statement creates a secondary index named `type_available_idx`, on the `vehicles` table. - -The MovR app might also need to display the vehicle's location and ID, but the app will not be filtering or sorting on those values. If any of the columns referenced in or returned by a query are not in a primary or secondary index key, CockroachDB will need to perform [a full scan of the table](sql-tuning-with-explain.html#issue-full-table-scans) to find the value. Full table scans can be costly, and should be avoided whenever possible. - -To help avoid unnecessary full table scans, add a `STORING` clause to the index: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE INDEX type_available_idx ON movr.vehicles (type, available) STORING (last_location); -~~~ - -The index will now store the values in `last_location`, which will improve the performance of reads from the `vehicles` table that return `type`, `available`, `id`, and `last_location` values and do not filter or sort on the `last_location` column. - -The `max_init.sql` file should now look similar to the following: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE movr.max_schema.users ( - first_name STRING, - last_name STRING, - email STRING UNIQUE, - CONSTRAINT "primary" PRIMARY KEY (first_name, last_name) - ); - -CREATE TYPE movr.max_schema.vtype AS ENUM ('bike', 'scooter', 'skateboard'); - -CREATE TABLE movr.max_schema.vehicles ( - id UUID DEFAULT gen_random_uuid() PRIMARY KEY, - type movr.max_schema.vtype, - creation_time TIMESTAMPTZ DEFAULT now(), - available BOOL, - last_location STRING - ); - -CREATE INDEX type_available_idx ON movr.max_schema.vehicles (type, available) STORING (last_location); - -CREATE TABLE movr.max_schema.rides ( - id UUID DEFAULT gen_random_uuid() PRIMARY KEY, - vehicle_id UUID REFERENCES movr.max_schema.vehicles(id), - start_address STRING, - end_address STRING, - start_time TIMESTAMPTZ DEFAULT now(), - end_time TIMESTAMPTZ - ); -~~~ - -If you executed this file when following the [Create a Table](schema-design-table.html) example, then all of these objects already exist. To clear the database and re-initialize the schemas, first execute the statements in the `dbinit.sql` file as the `root` user: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql \ ---certs-dir={certs-directory} \ ---user=root \ --f dbinit.sql -~~~ - -Then, execute the statements in the `max_init.sql` and `abbey_init.sql` files: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql \ ---certs-dir={certs-directory} \ ---user=max \ ---database=movr \ --f max_init.sql -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql \ ---certs-dir={certs-directory} \ ---user=abbey \ ---database=movr \ --f abbey_init.sql -~~~ - -After the statements have been executed, you can see the new index in the [CockroachDB SQL shell](cockroach-sql.html#sql-shell). - -Open the SQL shell to your cluster: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql \ ---certs-dir={certs-directory} \ ---user=max \ ---database=movr -~~~ - -To view the indexes in the `vehicles` table, issue a [`SHOW INDEXES`](show-index.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW INDEXES FROM movr.max_schema.vehicles; -~~~ - -~~~ - table_name | index_name | non_unique | seq_in_index | column_name | direction | storing | implicit --------------+--------------------+------------+--------------+---------------+-----------+---------+----------- - vehicles | primary | false | 1 | id | ASC | false | false - vehicles | type_available_idx | true | 1 | type | ASC | false | false - vehicles | type_available_idx | true | 2 | available | ASC | false | false - vehicles | type_available_idx | true | 3 | last_location | N/A | true | false - vehicles | type_available_idx | true | 4 | id | ASC | false | true -(5 rows) -~~~ - -The output from this `SHOW` statement displays the names and columns of the two indexes on the table (i.e., `primary` and `type_available_idx`). - -Note that the `last_location` column's `storing` value is `true` in the `type_available_idx` index, and is therefore not sorted. Also note that the primary key column `id` is implicit in the index, meaning the `id` column is implicitly indexed in `type_available_idx`. - -To see an index definition, use a [`SHOW CREATE`](show-create.html) statement on the table that contains the index: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE movr.max_schema.vehicles; -~~~ - -~~~ - table_name | create_statement ----------------------------+---------------------------------------------------------------------------------- - movr.max_schema.vehicles | CREATE TABLE max_schema.vehicles ( - | id UUID NOT NULL DEFAULT gen_random_uuid(), - | type max_schema.vtype NULL, - | creation_time TIMESTAMPTZ NULL DEFAULT now():::TIMESTAMPTZ, - | available BOOL NULL, - | last_location STRING NULL, - | CONSTRAINT "primary" PRIMARY KEY (id ASC), - | INDEX type_available_idx (available ASC, type ASC) STORING (last_location), - | FAMILY "primary" (id, type, creation_time, available, last_location) - | ) -(1 row) -~~~ - -After creating a database, a user-defined schema, some tables, and secondary indexes, the database schema should be ready for your application to [write](insert-data.html) and [read data](query-data.html). - -It's likely that you will need to update your database schema at some point. For an overview on how to update a database schema, see [Change and Remove Objects in a Database Schema](schema-design-update.html). We also recommend reading about [how online schema changes work in CockroachDB](online-schema-changes.html). - -## What's next? - -- [Change and Remove Objects in a Database Schema](schema-design-update.html) -- Read about [how schema changes work](online-schema-changes.html) -- [Insert Data](schema-design-indexes.html) -- [Query Data](online-schema-changes.html) - -You might also be interested in the following pages: - -- [`CREATE INDEX`](create-index.html) -- [Indexes](indexes.html) -- [Index a Subset of Rows](partial-indexes.html) -- [Index Sequential Keys](hash-sharded-indexes.html) -- [Index JSON and Array Data](inverted-indexes.html) -- [Index Spatial Data](spatial-indexes.html) -- [Cockroach Commands](cockroach-commands.html) -- [Create a User-defined Schema](schema-design-schema.html) -- [Create a Database](schema-design-database.html) -- [Schema Design Overview](schema-design-overview.html) -- [CockroachDB naming hierarchy](sql-name-resolution.html#naming-hierarchy) diff --git a/src/current/v21.1/schema-design-overview.md b/src/current/v21.1/schema-design-overview.md deleted file mode 100644 index 6b85762f081..00000000000 --- a/src/current/v21.1/schema-design-overview.md +++ /dev/null @@ -1,144 +0,0 @@ ---- -title: Database Schemas -summary: An overview of the objects that make up a logical schema -toc: true -keywords: gin, gin index, gin indexes, inverted index, inverted indexes, accelerated index, accelerated indexes ---- - -This page provides an overview of database schemas in CockroachDB. - -In later **Design a Database Schema** sections, we provide best practices for designing a database schema that optimizes performance and storage resources. - -## Database schema objects - -The sections below provide a brief overview of the logical objects that make up a database schema in CockroachDB, for the purpose of orienting application developers. - -For detailed documentation on object name resolution, see [Name Resolution](sql-name-resolution.html). - -### Databases - -Database objects make up the first level of the [CockroachDB naming hierarchy](sql-name-resolution.html#naming-hierarchy). They contain [schemas](#schemas). - -All CockroachDB clusters include a preloaded database named `defaultdb`. Rather than using the `defaultdb` database, we recommend creating your own database. - -For guidance on creating databases, see [Create a Database](schema-design-database.html). - -{% include {{page.version.version}}/sql/db-terms.md %} - -### Schemas - -Schemas make up the second level of the naming hierarchy. Each schema belongs to a single database. Schemas contain [tables](#tables) and other objects, like [views](#views) and [sequences](#sequences). - -All CockroachDB clusters include a preloaded schema named `public`. CockroachDB also supports creating your own *user-defined schema*. - -For guidance on creating user-defined schemas, see [Create a User-defined Schema](schema-design-schema.html). - -{% include {{page.version.version}}/sql/schema-terms.md %} - -### Tables - -Tables, belong to the third and lowest level of the naming hierarchy. Each table can belong to a single [user-defined schema](#schemas). - -Tables contain *rows* of data. Each value in a row of data belongs to a particular *column*. Each column allows values of data of a single data type. Columns can be further qualified with [column-level constraints](constraints.html), or computed with [scalar expressions](computed-columns.html). - -For guidance on defining tables, see [Tables](schema-design-table.html). - -### Indexes - -An index is a copy of the rows in a single table, sorted by a column or set of columns. CockroachDB queries use indexes to find data more efficiently in a table, given the values of a particular column. Each index belongs to a single table. - -The two main types of indexes are the `primary` index, an index on the row-identifying [primary key columns](primary-key.html), and the secondary index, an index created on non-primary-key columns of your choice. - -For guidance on defining primary keys, see [Select Primary Key Columns](schema-design-table.html#select-primary-key-columns). For guidance on defining secondary indexes, see [Add a Secondary Index](schema-design-indexes.html). - -#### Specialized indexes - -CockroachDB supports some specialized types of indexes, designed to improve query performance in specific use cases. For guidance on specialized indexes, see the following pages: - -- [Index a Subset of Rows](partial-indexes.html) -- [Index Sequential Keys](hash-sharded-indexes.html) -- [Index JSON and Array Data](inverted-indexes.html) -- [Index Spatial Data](spatial-indexes.html) - -### Other objects - -CockroachDB supports several other objects at the third level of the naming hierarchy, including reusable [views](#views), [sequences](#sequences), and [temporary objects](#temporary-objects). - -#### Views - -A view is a stored and named selection query. - -For guidance on using views, see [Views](views.html). - -#### Sequences - -Sequences create and store sequential data. - -For guidance on using sequences, see [the `CREATE SEQUENCE` syntax page](create-sequence.html). - -#### Temporary objects - -A temporary object is an object, such as a table, view, or sequence, that is not stored to persistent memory. - -For guidance on using temporary objects, see [Temporary Tables](temporary-tables.html). - -## Controlling access to objects - -CockroachDB supports both user-based and role-based access control. With roles, or with direct assignment, you can grant a [SQL user](authorization.html#sql-users) the [privileges](authorization.html#privileges) required to view, modify, and delete database schema objects. - -By default, the user that creates an object is that object's *owner*. [Object owners](authorization.html#object-ownership) have all privileges required to view, modify, or delete that object and the data stored within it. - -For more information about ownership, privileges, and authorization, see [Authorization](authorization.html). - -## Executing database schema changes - -We do not recommend using client drivers or ORM frameworks to execute database schema changes. As a best practice, we recommend creating database schemas and performing database schema changes with one of the following methods: - -- Use a database schema migration tool. - - CockroachDB is compatible with most PostgreSQL database schema migration tools, including [Flyway](https://flywaydb.org/) and [Liquibase](https://www.liquibase.com). For a tutorial on performing database schema changes using Liquibase, see our [Liquibase tutorial](liquibase.html). For a tutorial on performing schema changes with Flyway, see our [Flyway tutorial](flyway.html). - -- Use the [CockroachDB SQL client](cockroach-sql.html#execute-sql-statements-from-a-file). - - The CockroachDB SQL client allows you to execute commands from the command line, or through the [CockroachDB SQL shell](cockroach-sql.html#sql-shell) interface. From the command line, you can pass a string to the client for execution, or you can pass a `.sql` file populated with SQL commands. From the SQL shell, you can execute SQL commands directly. Throughout the guide, we pass a `.sql` file to the SQL client to perform most database schema changes. - -## Object size and scaling considerations - -CockroachDB does not place hard limits on most database objects. Extremely large attributes are not supported in certain scenarios as described in this section. - -### Hard limits - -The following table lists specific limits imposed by CockroachDB. - -| Object | Limit | Comments | -| ------ | ----- | -------- | -| Role names | 63 bytes | Other [restrictions](create-role.html#considerations) apply. | -| User names | 63 bytes | These are [equivalent](create-user.html) to role names. | -| Identifier length | 128 bytes | This limit is specified in the `max_identifier_length` variable for compatibility with other databases, but is not currently enforced. It may be enforced in future versions of CockroachDB, so we recommended remaining within this limit. | - -### Quantity of tables and other schema objects - -CockroachDB has been shown to perform well with clusters containing 2,500 tables. Greater numbers are possible, depending on the complexity of the tables (number of columns and indexes) and hardware specifications. - -As you scale to a large number of tables, note that: - -- The amount of RAM per node is the limiting factor for the number of tables and other schema objects the cluster can support. This includes columns, indexes, GIN indexes, constraints, and partitions. Increasing RAM is likely to have the greatest impact on the number of these objects that a cluster can support, while increasing the number of nodes will not have a substantial effect. -- The number of databases or schemas on the cluster has minimal impact on the total number of tables that it can support. - -See the [Hardware](recommended-production-settings.html#hardware) section for additional recommendations based on your expected workloads. - -### Quantity of rows - -CockroachDB can support any number of rows by adding additional nodes and [storage](recommended-production-settings.html#storage). - -## What's next? - -- [Create a Database](schema-design-database.html) -- [Create a User-defined Schema](schema-design-database.html) - -You might also be interested in the following pages: - -- [CockroachDB naming hierarchy](sql-name-resolution.html#naming-hierarchy) -- [Authorization](authorization.html) -- [Liquibase](liquibase.html) -- [Flyway](flyway.html) diff --git a/src/current/v21.1/schema-design-schema.md b/src/current/v21.1/schema-design-schema.md deleted file mode 100644 index f00f8c9509e..00000000000 --- a/src/current/v21.1/schema-design-schema.md +++ /dev/null @@ -1,168 +0,0 @@ ---- -title: Create a User-defined Schema -summary: Create a user-defined schema for your CockroachDB cluster -toc: true ---- - -This page provides best-practice guidance on creating user-defined schemas, with some simple examples based on Cockroach Labs' fictional vehicle-sharing company, [MovR](movr.html). - -{{site.data.alerts.callout_success}} -For detailed reference documentation on the `CREATE SCHEMA` statement, including additional examples, see the [`CREATE SCHEMA` syntax page](create-schema.html). -{{site.data.alerts.end}} - -## Before you begin - -Before reading this page, do the following: - -- [Install CockroachDB](install-cockroachdb.html). -- [Start a local cluster](secure-a-cluster.html), or [create a CockroachDB {{ site.data.products.cloud }} cluster](../cockroachcloud/create-your-cluster.html). -- [Review the database schema objects](schema-design-overview.html). -- [Create a database](schema-design-database.html). - -## Create a user-defined schema - -User-defined schemas belong to the second level of the [CockroachDB naming hierarchy](sql-name-resolution.html#naming-hierarchy). - -To create a user-defined schema, use a [`CREATE SCHEMA` statement](create-schema.html), following [the user-defined schema best practices](#user-defined-schema-best-practices). After reviewing the best practices, see the example we provide [below](#examples). - -### User-defined schema best practices - -Here are some best practices to follow when creating and using user-defined schemas: - -- If you want to create multiple objects (e.g., tables or views) with the same name in your cluster, do so in different user-defined schemas in the same database. - -- If you want to separate lower-level objects (e.g., a set of [tables](schema-design-table.html) or [views](views.html)) for access or organizational purposes, do not create those objects in the preloaded [`public` schema](sql-name-resolution.html#naming-hierarchy). Instead, create user-defined schemas, and then create the objects in the user-defined schemas. - -- Create user-defined schemas as a member of [the `admin` role](authorization.html#admin-role) (e.g., as the [`root` user](authorization.html#root-user)), and then give ownership of them to a [different user](schema-design-overview.html#controlling-access-to-objects), with fewer privileges across the database, following [authorization best practices](authorization.html#authorization-best-practices). - -- When you create a user-defined schema, take note of the [object's owner](authorization.html#object-ownership). You can specify the owner in a `CREATE SCHEMA` statement with the [`AUTHORIZATION` keyword](create-schema.html#parameters). If `AUTHORIZATION` is not specified, the owner will be the user creating the user-defined schema. - -- Do not create user-defined schemas in the preloaded `defaultdb` database. Instead, use a database [you have created](schema-design-database.html). If you do not specify a database in the `CREATE SCHEMA` statement, the user-defined schema will be created in your SQL session's [current database](sql-name-resolution.html#current-database). - -- When referring to a lower-level object in a database (e.g., a table), include the object's schema name (e.g., `schema_name.table_name`). Specifying the schema name in a lower-level object reference can prevent users from attempting to access the wrong object, if there are multiple objects with the same name in a database. - -- {% include {{page.version.version}}/sql/dev-schema-changes.md %} - -### Examples - -Suppose you want to separate the tables and indexes in your cluster such that one user manages a group of tables, and another user manages a different group of tables. You can do this with two different user-defined schemas, owned by two different SQL users, with fewer privileges than the `root` user. - -Open the `dbinit.sql` file that you created in the [Create a Database](schema-design-database.html) example, and add the following statements under the `CREATE DATABASE` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -USE movr; - -CREATE USER IF NOT EXISTS max; -GRANT CREATE ON DATABASE movr TO max; - -CREATE USER IF NOT EXISTS abbey; -GRANT CREATE ON DATABASE movr TO abbey; -~~~ - -The first statement sets the `movr` database as the [current database](sql-name-resolution.html#current-database). The next two sets of statements create SQL users named `max` and `abbey` in the `movr` database, with [`CREATE` privileges on the database](authorization.html#supported-privileges). `CREATE` privileges will allow each user to create tables in the database. - -Now, under the `CREATE USER` statements, add `DROP SCHEMA` and `CREATE SCHEMA` statements for each user's user-defined schema: - -{% include_cached copy-clipboard.html %} -~~~ sql -DROP SCHEMA IF EXISTS max_schema CASCADE; -CREATE SCHEMA max_schema AUTHORIZATION max; - -DROP SCHEMA IF EXISTS abbey_schema CASCADE; -CREATE SCHEMA abbey_schema AUTHORIZATION abbey; -~~~ - -The first set of statement clears the database of any existing schema named `max_schema`, and then creates a schema named `max_schema` with the owner `max`. The next set of statements does the same, but for `abbey_schema`, with `abbey` as the owner. - -It might also be a good idea to [grant the `USAGE` privilege](grant.html#supported-privileges) on each schema to the other user in the database. This will allow the other user to access objects in the schema, but it will not let them delete the schema, or create objects inside of it. - -Under the `CREATE SCHEMA` statements for each user-defined schema, add a `GRANT` statement granting `USAGE` privileges on the schema to the other user. - -The `dbinit.sql` file should now look something link this: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE DATABASE IF NOT EXISTS movr; - -USE movr; - -CREATE USER IF NOT EXISTS max; -GRANT CREATE ON DATABASE movr TO max; - -CREATE USER IF NOT EXISTS abbey; -GRANT CREATE ON DATABASE movr TO abbey; - -DROP SCHEMA IF EXISTS max_schema CASCADE; -CREATE SCHEMA max_schema AUTHORIZATION max; -GRANT USAGE ON SCHEMA max_schema TO abbey; - -DROP SCHEMA IF EXISTS abbey_schema CASCADE; -CREATE SCHEMA abbey_schema AUTHORIZATION abbey; -GRANT USAGE ON SCHEMA abbey_schema TO max; -~~~ - -To execute the statements in the `dbinit.sql` file as the `root` user, run the following command: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql \ ---certs-dir={certs-directory} \ ---user=root \ --f dbinit.sql -~~~ - -Before the new users can connect to the cluster and start creating objects, they each need a [user certificate](authentication.html#client-authentication). To create a user certificate for `max`, open a new terminal, and run the following [`cockroach cert`](cockroach-cert.html) command: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach cert create-client max --certs-dir={certs-directory} --ca-key={my-safe-directory}/ca.key -~~~ - -Create a user certificate for `abbey` as well: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach cert create-client abbey --certs-dir={certs-directory} --ca-key={my-safe-directory}/ca.key -~~~ - -As one of the new users, use a [`SHOW SCHEMAS` statement](show-schemas.html) to show the preloaded and user-defined schemas in the `movr` database: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql \ ---certs-dir={certs-directory} \ ---user=abbey \ ---database=movr \ ---execute="SHOW SCHEMAS;" -~~~ - -~~~ - schema_name | owner ----------------------+-------- - abbey_schema | abbey - crdb_internal | NULL - information_schema | NULL - max_schema | max - pg_catalog | NULL - pg_extension | NULL - public | admin -(7 rows) -~~~ - -You're now ready to start adding tables to the `max_schema` user-defined schema as the `max` user, and to the `abbey_schema` user-defined schema as the `abbey` user. - -For guidance on creating tables, see at [Create a Table](schema-design-table.html). - -## What's next? - -- [Create a Table](schema-design-table.html) -- [Add Secondary Indexes](schema-design-indexes.html) - -You might also be interested in the following pages: - -- [`CREATE SCHEMA`](create-schema.html) -- [Cockroach Commands](cockroach-commands.html) -- [Create a Database](schema-design-database.html) -- [Schema Design Overview](schema-design-overview.html) -- [CockroachDB naming hierarchy](sql-name-resolution.html#naming-hierarchy) diff --git a/src/current/v21.1/schema-design-table.md b/src/current/v21.1/schema-design-table.md deleted file mode 100644 index 6003d648202..00000000000 --- a/src/current/v21.1/schema-design-table.md +++ /dev/null @@ -1,604 +0,0 @@ ---- -title: Create a Table -summary: Best practices for working with tables in CockroachDB. -toc: true ---- - -This page provides best-practice guidance on creating tables, with some simple examples based on Cockroach Labs' fictional vehicle-sharing company, [MovR](movr.html). - -{{site.data.alerts.callout_success}} -For detailed reference documentation on the `CREATE TABLE` statement, including additional examples, see the [`CREATE TABLE` syntax page](create-table.html). -{{site.data.alerts.end}} - -## Before you begin - -Before reading this page, do the following: - -- [Install CockroachDB](install-cockroachdb.html). -- [Start a local cluster](secure-a-cluster.html), or [create a CockroachDB {{ site.data.products.cloud }} cluster](../cockroachcloud/create-your-cluster.html). -- [Review the database schema objects](schema-design-overview.html). -- [Create a database](schema-design-database.html). -- [Create a user-defined schema](schema-design-schema.html). - -## Create a table - -Tables are the [logical objects in a cluster](schema-design-overview.html#database-schema-objects) that store data sent from your application's persistence layer. Tables organize records of data in rows and columns. - -To create a table, use a [`CREATE TABLE` statement](create-table.html), following the best practices that we have listed in the following sections: - -- [Name a table](#name-a-table) -- [Define columns](#define-columns) -- [Select primary key columns](#select-primary-key-columns) -- [Add additional constraints](#add-additional-constraints) -- [Execute `CREATE TABLE` statements](#execute-create-table-statements) - -After reviewing the best practices in each section, see the example provided in that section. - -### Name a table - -Naming a table is the first step in table creation. - -`CREATE TABLE` statements generally take the form: - -~~~ -CREATE TABLE {schema_name}.{table_name} ( - {elements} - ); -~~~ - -Parameter | Description -----------|------------ -`{schema_name}` | The name of the user-defined schema. -`{table_name}` | The name of the table. -`{elements}` | A comma-separated list of table elements, such as column definitions. - -For an example, see [below](#table-naming-example). - -#### Table naming best practices - -Here are some best practices to follow when naming tables: - -- Use a [fully-qualified name](sql-name-resolution.html#how-name-resolution-works) (i.e., `CREATE TABLE database_name.schema_name.table_name`). If you do not specify the database name, CockroachDB will use the SQL session's [current database](sql-name-resolution.html#current-database) (`defaultdb`, by default). If you do not specify the user-defined schema in the table name, CockroachDB will create the table in the preloaded `public` schema. - -- Use a table name that reflects the contents of the table. For example, for a table containing information about orders, you could use the name `orders` (as opposed to naming the table something like `table1`). - -#### Table naming example - -Suppose you want to create a table to store information about users of the [MovR](movr.html) platform, and you want the SQL user `max` to manage that table. - -Create an empty `.sql` file for `max`: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ touch max_init.sql -~~~ - -This file will initialize the objects in the `max_schema` user-defined schema that you created in [Create a Schema](schema-design-schema.html), starting with a `users` table. - -In a text editor, open `max_init.sql`, and add an empty `CREATE TABLE` statement for the `users` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE movr.max_schema.users ( -); -~~~ - -Next, [define the table's columns](#define-columns). - -### Define columns - -Column definitions give structure to a table by separating the values in each row into columns of a single data type. - -Column definitions generally take the following form: - -~~~ -{column_name} {DATA_TYPE} {column_qualification} -~~~ - -Parameter | Description -----------|------------ -`{column_name}` | The name of the column. -`{DATA_TYPE}` | The [data type](data-types.html) of the row values in the column. -`{column_qualification}` | Some column qualification, such as a [column-level constraint](#add-additional-constraints), or a [computed column clause](computed-columns.html). - -For examples, see [below](#column-definition-examples). - -#### Column definition best practices - -Here are some best practices to follow when defining table columns: - -- Review the supported column [data types](data-types.html), and select the appropriate type for the data you plan to store in a column, following the best practices listed on the data type's reference page. - -- Use column data types with a fixed size limit, or set a maximum size limit on column data types of variable size (e.g., [`VARBIT(n)`](bit.html#size)). Values exceeding 1MB can lead to [write amplification](https://en.wikipedia.org/wiki/Write_amplification) and cause significant performance degradation. - -- Review the [primary key best practices](#select-primary-key-columns) and [examples](#primary-key-examples), decide if you need to define any dedicated primary key columns. - -- Review the best practices and examples for [adding additional constraints](#add-additional-constraints), and decide if you need to add any additional constraints to your columns. - -#### Column definition examples - -In the `max_init.sql` file, add a few column definitions to the `users` table's `CREATE TABLE` statement, for user names and email addresses: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE movr.max_schema.users ( - first_name STRING, - last_name STRING, - email STRING -); -~~~ - -All of the columns shown above use the [`STRING`](string.html) data type, meaning that any value in any of the columns must be of the data type `STRING`. - -CockroachDB supports a number of other column data types, including [`DECIMAL`](decimal.html), [`INT`](int.html), [`TIMESTAMP`](timestamp.html), [`UUID`](uuid.html), and [enumerated data types](enum.html) and [spatial data types](spatial-data.html). We recommend that you review the [supported types](data-types.html), and create columns with data types that correspond to the types of data that you intend to persist to the cluster from your application. - -Let's add another example table to our `max_schema` schema, with more column data types. - -As a vehicle-sharing platform, MovR needs to store data about its vehicles. In `max_init.sql`, add a `CREATE TABLE` statement for a `vehicles` table, under the `CREATE TABLE` statement for `users`. This table should probably include information about the type of vehicle, when it was created, what its availability is, and where it is located: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE movr.max_schema.vehicles ( - id UUID, - type STRING, - creation_time TIMESTAMPTZ, - available BOOL, - last_location STRING - ); -~~~ - -This table includes a few more data types than the `users` table: - -- [`UUID`](uuid.html), which we recommend for columns with [values that uniquely identify rows](https://en.wikipedia.org/wiki/Unique_key) (like an "id" column). -- [`TIMESTAMPTZ`](timestamp.html), which we recommend for [timestamp values](https://en.wikipedia.org/wiki/Timestamp). -- [`BOOL`](bool.html), which we recommend for columns that will only take one of two possible values. - -The rest of the columns are `STRING`-typed. - -Note that values in the `type` column will likely only be `STRING` values from a fixed list of values. Specifically, the vehicle type can only be one of the vehicle types supported by the MovR platform (e.g., a `bike`, a `scooter`, or a `skateboard`). For values like this, we recommend using a [user-defined, enumerated type](enum.html). - -To create a user-defined type, use a `CREATE TYPE` statement. For example, above the `CREATE TABLE` statement for the `vehicles` table, add the following statements: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TYPE movr.max_schema.vtype AS ENUM ('bike', 'scooter', 'skateboard'); -~~~ - -{{site.data.alerts.callout_success}} -For detailed reference documentation on the `CREATE TYPE` statement, including additional examples, see the [`CREATE TYPE` syntax page](create-type.html).
    For detailed reference documentation on enumerated data types, including additional examples, see [`ENUM`](enum.html). -{{site.data.alerts.end}} - -You can then use `movr.max_schema.vtype` as the `type` column's data type: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE movr.max_schema.vehicles ( - id UUID, - type movr.max_schema.vtype, - creation_time TIMESTAMPTZ, - available BOOL, - last_location STRING - ); -~~~ - -Only values in the set of `movr.max_schema.vtype` values will be allowed in the `type` column. - -The `users` and `vehicles` tables now have syntactically valid column definitions. As a best practice, you should explicitly [select primary key columns](#select-primary-key-columns) and add any [additional constraints](#add-additional-constraints) before executing the `CREATE TABLE` statements. - -### Select primary key columns - -A primary key is a column, or set of columns, whose values uniquely identify rows of data. Every table requires a primary key. - -Primary keys are defined in `CREATE TABLE` statements with the `PRIMARY KEY` [column constraint](constraints.html). The `PRIMARY KEY` constraint requires that all the constrained column(s) contain only unique and non-`NULL` values. - -When a table is created, CockroachDB creates an index (called the `primary` index) on the column(s) constrained by the `PRIMARY KEY` constraint. CockroachDB uses this [index](indexes.html) to find rows in a table more efficiently. - -To add a single column to a primary key, add the `PRIMARY KEY` keyword to the end of the column definition. To add multiple columns to a primary key (i.e., to create a [composite primary key](https://en.wikipedia.org/wiki/Composite_key)), add a separate `CONSTRAINT "primary" PRIMARY KEY` clause after the column definitions in the `CREATE TABLE` statement. - -For examples, see [below](#primary-key-examples). - -{{site.data.alerts.callout_success}} -For detailed reference documentation on the `PRIMARY KEY` constraint, including additional examples, see the [`PRIMARY KEY` constraint page](primary-key.html). -{{site.data.alerts.end}} - -#### Primary key best practices - -Here are some best practices to follow when selecting primary key columns: - -- Avoid defining primary keys over a single column of sequential data. - - Querying a table with a primary key on a single sequential column (e.g., an auto-incrementing [`INT`](int.html) column, or a [`TIMESTAMP`](timestamp.html) value) can result in single-range hotspots that negatively affect performance, or cause [transaction contention](transactions.html#transaction-contention). - - If you are working with a table that *must* be indexed on sequential keys, use [hash-sharded indexes](hash-sharded-indexes.html). For details about the mechanics and performance improvements of hash-sharded indexes in CockroachDB, see our [Hash Sharded Indexes Unlock Linear Scaling for Sequential Workloads](https://www.cockroachlabs.com/blog/hash-sharded-indexes-unlock-linear-scaling-for-sequential-workloads/) blog post. - -- Define a primary key for every table. - - If you create a table without defining a primary key, CockroachDB will automatically create a primary key over a hidden, [`INT`](int.html)-typed column named `rowid`. By default, sequential, unique identifiers are generated for each row in the `rowid` column with the [`unique_rowid()` function](functions-and-operators.html#built-in-functions). The sequential nature of the `rowid` values can lead to a poor distribution of the data across a cluster, which can negatively affect performance. Furthermore, because you cannot meaningfully use the `rowid` column to filter table data, the primary key index on `rowid` does not offer any performance optimization. This means you will always have improved performance by defining a primary key for a table. - -- When possible, define primary key constraints over multiple columns (i.e., use [composite primary keys](https://en.wikipedia.org/wiki/Composite_key)). - - When defining composite primary keys, make sure the data in the first column of the primary key prefix is well-distributed across the nodes in the cluster. To improve the performance of [ordered queries](order-by.html), you can add monotonically increasing primary key columns after the first column of the primary key prefix. For an example, see [below](#primary-key-examples). - -- For single-column primary keys, use [`UUID`](uuid.html)-typed columns with default values randomly-generated, using the `gen_random_uuid()` [SQL function](functions-and-operators.html#id-generation-functions). - - Randomly generating `UUID` values ensures that the primary key values will be unique and well-distributed across a cluster. For an example, see [below](#primary-key-examples). - -#### Primary key examples - -To follow a [primary key best practice](#primary-key-best-practices), the `CREATE TABLE` statements in `max_init.sql` for the `users` and `vehicles` tables need to explicitly define a primary key. - -In the `max_init.sql` file, add a composite primary key on the `first_name` and `last_name` columns of the `users` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE movr.max_schema.users ( - first_name STRING, - last_name STRING, - email STRING, - CONSTRAINT "primary" PRIMARY KEY (first_name, last_name) -); -~~~ - -This primary key will uniquely identify rows of user data. - -Because `PRIMARY KEY` constraints imply `UNIQUE` indexes, only one unique combination of `first_name` and `last_name` will be allowed in rows of the `users` table. Additionally, because `PRIMARY KEY` implies the `NOT NULL` constraint, all rows of data inserted into the `users` table must include values for `first_name` and `last_name`. - -Primary key columns can also be single columns, if those columns are guaranteed to uniquely identify the row. Their values should also be well-distributed across the cluster. - -In the `vehicles` table definition, add a `PRIMARY KEY` constraint on the `id` column: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE movr.max_schema.vehicles ( - id UUID DEFAULT gen_random_uuid() PRIMARY KEY, - type movr.max_schema.vtype, - creation_time TIMESTAMPTZ, - available BOOL, - last_location STRING - ); -~~~ - -Note that, in addition to the `PRIMARY KEY` constraint, the `id` column has a `DEFAULT` constraint. This constraint sets a default value for the column to a value generated by the `gen_random_uuid()` [function](functions-and-operators.html), following [`UUID`](uuid.html) best practices. The values generated by this function are guaranteed to be unique and well-distributed across the cluster. We discuss the `DEFAULT` constraint more [below](#populate-with-default-values). - -### Add additional constraints - -In addition to the `PRIMARY KEY` constraint, CockroachDB supports a number of other [column-level constraints](constraints.html), including [`CHECK`](check.html), [`DEFAULT`](#populate-with-default-values), [`FOREIGN KEY`](#reference-other-tables), [`UNIQUE`](#prevent-duplicates), and [`NOT NULL`](#prevent-null-values). Using constraints can simplify table queries, improve query performance, and ensure that data remains semantically valid. - -To constrain a single column, add a constraint keyword to the column's definition, as shown in the [single-column `PRIMARY KEY` example above](#primary-key-examples). To constrain more than one column, add the entire constraint's definition after the list of columns in the `CREATE TABLE` statement, also shown in the [composite `PRIMARY KEY` example above](#primary-key-examples). - -For guidance and examples for each constraint, see the sections below. - -{{site.data.alerts.callout_success}} -For detailed reference documentation for each supported constraint, see [the constraint's syntax page](constraints.html). -{{site.data.alerts.end}} - -#### Populate with default values - -To set default values on columns, use the `DEFAULT` constraint. Default values enable you to write queries without the need to specify values for every column. - -When combined with [supported SQL functions](functions-and-operators.html), default values can save resources in your application's persistence layer by offloading computation onto CockroachDB. For example, rather than using an application library to generate unique `UUID` values, you can set a default value to be an automatically-generated `UUID` value with the `gen_random_uuid()` SQL function. Similarly, you could use a default value to populate a `TIMESTAMP` column with the current time of day, using the `now()` function. - -For example, in the `vehicles` table definition in `max_init.sql`, you added a `DEFAULT gen_random_uuid()` clause to the `id` column definition. This set the default value to a generated `UUID` value. Now, add a default value to the `creation_time` column: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE movr.max_schema.vehicles ( - id UUID DEFAULT gen_random_uuid() PRIMARY KEY, - type movr.max_schema.vtype, - creation_time TIMESTAMPTZ DEFAULT now(), - available BOOL, - last_location STRING - ); -~~~ - -When a row is inserted into the `vehicles` table, CockroachDB generates a random default value for the vehicle `id`, and uses the current timestamp for the vehicle's `creation_time`. Rows inserted into the `vehicles` table do not need to include an explicit value for `id` or `creation_time`. - -{{site.data.alerts.callout_success}} -For detailed reference documentation on the `DEFAULT` constraint, including additional examples, see [the `DEFAULT` syntax page](default-value.html). -{{site.data.alerts.end}} - -#### Reference other tables - -To reference values in another table, use a `FOREIGN KEY` constraint. `FOREIGN KEY` constraints enforce [referential integrity](https://en.wikipedia.org/wiki/Referential_integrity), which means that a column can only refer to an existing column. - -For example, suppose you want to add a new table that contains data about the rides that MovR users are taking on vehicles. This table should probably include information about the location and duration of the ride, as well as information about the vehicle used for the ride. - -In `max_init.sql`, under the `CREATE TABLE` statement for `vehicles`, add a definition for a `rides` table, with a foreign key dependency on the `vehicles` table. To define a foreign key constraint, use the `REFERENCES` keyword: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE movr.max_schema.rides ( - id UUID DEFAULT gen_random_uuid() PRIMARY KEY, - vehicle_id UUID REFERENCES movr.max_schema.vehicles(id), - start_address STRING, - end_address STRING, - start_time TIMESTAMPTZ DEFAULT now(), - end_time TIMESTAMPTZ - ); -~~~ - -The `vehicle_id` column will be identical to the `id` column in the `vehicles` table. Any queries that insert a `vehicle_id` that does not exist in the `id` column of the `vehicles` table will return an error. - -Foreign keys cannot reference tables in a different database. They can, however reference tables in a different schema. - -Suppose that you want to introduce promotional codes for users on the MovR platform, but you want the user promo code data to be managed by the `abbey` user that you created in [Create a Database](schema-design-database.html). - -Create an empty `.sql` initialization file for `abbey`. - -{% include_cached copy-clipboard.html %} -~~~ shell -$ touch abbey_init.sql -~~~ - -This file will initialize the objects in the `abbey_schema` user-defined schema that you created in [Create a Schema](schema-design-schema.html). - -In a text editor, open `abbey_init.sql`, and add a `CREATE TABLE` statement for a table called `user_promo_codes`: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE movr.abbey_schema.user_promo_codes ( - code STRING, - user_email STRING REFERENCES movr.max_schema.users(email), - valid BOOL, - CONSTRAINT "primary" PRIMARY KEY (code, user_email) - ); -~~~ - -This new table references the `email` column of the `users` table in `max_schema`. Because the `user_promo_codes` table depends on the `users` table, you'll need to execute `max_init.sql` before `abbey_init.sql`. - -{{site.data.alerts.callout_info}} -Foreign key dependencies can significantly impact query performance, as queries involving tables with foreign keys, or tables referenced by foreign keys, require CockroachDB to check two separate tables. We recommend using them sparingly. -{{site.data.alerts.end}} - -{{site.data.alerts.callout_success}} -For detailed reference documentation on the `FOREIGN KEY` constraint, including additional examples, see [the `FOREIGN KEY` syntax page](foreign-key.html). -{{site.data.alerts.end}} - -#### Prevent duplicates - -To prevent duplicate values in a column, use the `UNIQUE` constraint. - -For example, suppose that you want to ensure that the email addresses of all users are different, to prevent users from registering for two accounts with the same email address. Add a `UNIQUE` constraint to the `email` column of the `users` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE movr.max_schema.users ( - first_name STRING, - last_name STRING, - email STRING UNIQUE, - CONSTRAINT "primary" PRIMARY KEY (first_name, last_name) -); -~~~ - -Attempting to insert `email` values that already exist in the `users` table will return an error. - -{{site.data.alerts.callout_info}} -When you add a `UNIQUE` constraint to a column, CockroachDB creates a secondary index on that column, to help speed up checks on a column value's uniqueness. - -Also note that the `UNIQUE` constraint is implied by the `PRIMARY KEY` constraint. -{{site.data.alerts.end}} - -{{site.data.alerts.callout_success}} -For detailed reference documentation on the `UNIQUE` constraint, including additional examples, see [the `UNIQUE` syntax page](unique.html). -{{site.data.alerts.end}} - -#### Prevent `NULL` values - -To prevent `NULL` values in a column, use the `NOT NULL` constraint. If you specify a `NOT NULL` constraint, all queries against the table with that constraint must specify a value for that column, or have a default value specified with a `DEFAULT` constraint. - -For example, if you require all users of the MovR platform to have an email on file, you can add a `NOT NULL` constraint to the `email` column of the `users` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE movr.max_schema.users ( - first_name STRING, - last_name STRING, - email STRING UNIQUE NOT NULL, - CONSTRAINT "primary" PRIMARY KEY (first_name, last_name) -); -~~~ - -{{site.data.alerts.callout_info}} -Note that the `NOT NULL` constraint is implied by the `PRIMARY KEY` constraint. -{{site.data.alerts.end}} - -{{site.data.alerts.callout_success}} -For detailed reference documentation on the `NOT NULL` constraint, including additional examples, see [the `NOT NULL` syntax page](not-null.html). -{{site.data.alerts.end}} - -### Execute `CREATE TABLE` statements - -After you have defined `CREATE TABLE` statements for your tables, you can execute the statements. - -#### `CREATE TABLE` execution best practices - -Here are some general best practices to follow when executing `CREATE TABLE` statements: - -- Do not create tables as the `root` user. Instead, create tables as a [different user](schema-design-overview.html#controlling-access-to-objects), with fewer privileges, following [authorization best practices](authorization.html#authorization-best-practices). The user that creates an object becomes that [object's owner](authorization.html#object-ownership). - -- {% include {{page.version.version}}/sql/dev-schema-changes.md %} - -- {% include {{page.version.version}}/sql/dev-schema-change-limits.md %} - -#### Execute the example `CREATE TABLE` statements - -After following the examples provided in the sections above, the `max_init.sql` file should look similar to the following: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE movr.max_schema.users ( - first_name STRING, - last_name STRING, - email STRING UNIQUE NOT NULL, - CONSTRAINT "primary" PRIMARY KEY (first_name, last_name) - ); - -CREATE TYPE movr.max_schema.vtype AS ENUM ('bike', 'scooter', 'skateboard'); - -CREATE TABLE movr.max_schema.vehicles ( - id UUID DEFAULT gen_random_uuid() PRIMARY KEY, - type movr.max_schema.vtype, - creation_time TIMESTAMPTZ DEFAULT now(), - available BOOL, - last_location STRING - ); - -CREATE TABLE movr.max_schema.rides ( - id UUID DEFAULT gen_random_uuid() PRIMARY KEY, - vehicle_id UUID REFERENCES movr.max_schema.vehicles(id), - start_address STRING, - end_address STRING, - start_time TIMESTAMPTZ DEFAULT now(), - end_time TIMESTAMPTZ - ); -~~~ - -To execute the statements in the `max_init.sql` file, run the following command: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql \ ---certs-dir={certs-directory} \ ---user=max \ ---database=movr \ --f max_init.sql -~~~ - -The SQL client will execute any statements in `max_init.sql`, with `movr` as the database and `max` as the user. `max` is now the owner of all objects created by the statements in the `max_init.sql` file. - -After the statements have been executed, you can see the tables in the [CockroachDB SQL shell](cockroach-sql.html#sql-shell). - -Open the SQL shell to your cluster, with `movr` as the database and `max` as the user: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql \ ---certs-dir={certs-directory} \ ---user=max \ ---database=movr -~~~ - -To view the tables in the `max_schema` user-defined schema, issue a [`SHOW TABLES`](show-tables.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM max_schema; -~~~ - -~~~ - schema_name | table_name | type | owner | estimated_row_count ---------------+------------+-------+-------+---------------------- - max_schema | rides | table | max | 0 - max_schema | users | table | max | 0 - max_schema | vehicles | table | max | 0 -(3 rows) -~~~ - -To see the individual `CREATE TABLE` statements for each table, use a [`SHOW CREATE`](show-create.html) statement. For example, to see the `vehicles` `CREATE TABLE` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE movr.max_schema.vehicles; -~~~ - -~~~ - table_name | create_statement ----------------------------+--------------------------------------------------------------------------- - movr.max_schema.vehicles | CREATE TABLE max_schema.vehicles ( - | id UUID NOT NULL DEFAULT gen_random_uuid(), - | type max_schema.vtype NULL, - | creation_time TIMESTAMPTZ NULL DEFAULT now():::TIMESTAMPTZ, - | available BOOL NULL, - | last_location STRING NULL, - | CONSTRAINT "primary" PRIMARY KEY (id ASC), - | FAMILY "primary" (id, type, creation_time, available, last_location) - | ) -(1 row) -~~~ - -After following the examples provided in the sections above, the `abbey_init.sql` file should look similar to the following: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE movr.abbey_schema.user_promo_codes ( - code STRING, - user_email STRING REFERENCES movr.max_schema.users(email), - valid BOOL, - CONSTRAINT "primary" PRIMARY KEY (code, user_email) - ); -~~~ - -To execute the statement in the `abbey_init.sql` file, run the following command: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql \ ---certs-dir={certs-directory} \ ---user=abbey \ ---database=movr \ --f abbey_init.sql -~~~ - -After the statements have been executed, you can see the table in the [CockroachDB SQL shell](cockroach-sql.html#sql-shell). - -Open the SQL shell to your cluster, with `movr` as the database and `abbey` as the user, and view the table: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql \ ---certs-dir={certs-directory} \ ---user=abbey \ ---database=movr -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM abbey_schema; -~~~ - -~~~ - schema_name | table_name | type | owner | estimated_row_count ----------------+------------------+-------+-------+---------------------- - abbey_schema | user_promo_codes | table | abbey | 0 -(1 row -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE movr.abbey_schema.user_promo_codes; -~~~ - -~~~ - table_name | create_statement --------------------------------------+------------------------------------------------------------------------------------------------------ - movr.abbey_schema.user_promo_codes | CREATE TABLE abbey_schema.user_promo_codes ( - | code STRING NOT NULL, - | user_email STRING NOT NULL, - | valid BOOL NULL, - | CONSTRAINT "primary" PRIMARY KEY (code ASC, user_email ASC), - | CONSTRAINT fk_user_email_ref_users FOREIGN KEY (user_email) REFERENCES max_schema.users(email), - | FAMILY "primary" (code, user_email, usage_count) - | ) -(1 row) -~~~ - -Note that none of the tables that you have created thus far have secondary indexes. For guidance on adding secondary indexes, see at [Add Secondary Indexes](schema-design-indexes.html). - -## What's next? - -- [Add Secondary Indexes](schema-design-indexes.html) -- [Online Schema Changes](online-schema-changes.html) - -You might also be interested in the following pages: - -- [Computed Columns](computed-columns.html) -- [Column Families](column-families.html) -- [`CREATE TABLE`](create-table.html) -- [Data Types](data-types.html) -- [`PRIMARY KEY`](primary-key.html) -- [Constraints](constraints.html) -- [Cockroach Commands](cockroach-commands.html) -- [Create a User-defined Schema](schema-design-schema.html) -- [Create a Database](schema-design-database.html) -- [Schema Design Overview](schema-design-overview.html) -- [CockroachDB naming hierarchy](sql-name-resolution.html#naming-hierarchy) diff --git a/src/current/v21.1/schema-design-update.md b/src/current/v21.1/schema-design-update.md deleted file mode 100644 index fa6c0e09c36..00000000000 --- a/src/current/v21.1/schema-design-update.md +++ /dev/null @@ -1,340 +0,0 @@ ---- -title: Change and Remove Objects in a Database Schema -summary: How to change and remove objects in a CockroachDB database schema. -toc: true ---- - -This page provides an overview on changing and removing the objects in a database schema, with some simple examples based on Cockroach Labs' fictional vehicle-sharing company, [MovR](movr.html). - -## Before you begin - -Before reading this page, do the following: - -- [Install CockroachDB](install-cockroachdb.html). -- [Start a local cluster](secure-a-cluster.html), or [create a CockroachDB {{ site.data.products.cloud }} cluster](../cockroachcloud/create-your-cluster.html). -- [Review the database schema objects](schema-design-overview.html). -- [Create a database](schema-design-database.html). -- [Create a user-defined schema](schema-design-schema.html). -- [Create a table](schema-design-table.html). -- [Add secondary indexes](schema-design-indexes.html). - -## Alter database schema objects - -To change an existing object in a database schema, use an `ALTER` statement. - -`ALTER` statements generally take the following form: - -~~~ -ALTER {OBJECT_TYPE} {object_name} {SUBCOMMAND}; -~~~ - -Parameter | Description -----------|------------ -`{OBJECT_TYPE}` | The type of the object. -`{object_name}` | The name of the object. -`{SUBCOMMAND}` | The subcommand for the change that you would like to make. - -For examples, see [below](#altering-objects-examples). - -CockroachDB supports the following `ALTER` statements: - -- [`ALTER DATABASE`](alter-database.html) -- [`ALTER SCHEMA`](alter-schema.html) -- [`ALTER TABLE`](alter-table.html) -- [`ALTER INDEX`](alter-index.html) -- [`ALTER VIEW`](alter-view.html) -- [`ALTER SEQUENCE`](alter-sequence.html) -- [`ALTER TYPE`](alter-type.html) -- [`ALTER USER/ROLE`](alter-user.html) - -### Best practices for altering objects - -- After you initialize a database schema, make any additional database schema changes in a separate set of changes (e.g., for the Cockroach SQL client, a separate `.sql` file; for Liquibase, a separate *changeset*). - -- For `ALTER TABLE` statements, combine multiple subcommands in a single `ALTER TABLE` statement, where possible. - -- {% include {{page.version.version}}/sql/dev-schema-changes.md %} - -- {% include {{page.version.version}}/sql/dev-schema-change-limits.md %} - -### Altering objects examples - -Suppose you want to make some changes to the `users` table that you created in [Create a Table](schema-design-table.html). In specific, you want to do the following: - -- Add a new `username` column. -- Change the columns in the table's primary key to `username` column and `email`. -- Move the table to the `abbey_schema` user-defined schema. -- Change the owner of the table to `abbey`. - -The `ALTER TABLE` statement has subcommands for all of these changes: - -- To add a new column, use the [`ADD COLUMN` subcommand](add-column.html). -- To change the primary key columns of a table, use the [`ALTER PRIMARY KEY` subcommand](alter-primary-key.html). -- To move a table to a different schema, use the [`SET SCHEMA`](set-schema.html) subcommand. -- To change the owner of a table, use the [`OWNER TO`](owner-to.html) subcommand. - -Create a new `.sql` file for the changes that you plan to make to the table: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ touch update_users_table.sql -~~~ - -Open `update_users_table.sql` in a text editor, and add the `ALTER TABLE` statement for adding the `username` column: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER TABLE IF EXISTS movr.max_schema.users ADD COLUMN username STRING; -~~~ - -Under that first `ALTER TABLE` statement, add another `ALTER TABLE` statement for changing the primary key columns to `username` and `email`: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER TABLE IF EXISTS movr.max_schema.users ALTER PRIMARY KEY USING COLUMNS (username, email); -~~~ - -In order to add a column to an existing table's primary key index, the column must have an existing [`NOT NULL` constraint](not-null.html). Neither the `username` nor the `email` columns have `NOT NULL` constraints. - -Add a `NOT NULL` constraint to the `ADD COLUMN` subcommand for `username`. In the same `ALTER TABLE` statement, add an [`ALTER COLUMN` subcommand](alter-column.html) to set the `NOT NULL` constraint on the `email` column: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER TABLE IF EXISTS movr.max_schema.users - ADD COLUMN username STRING NOT NULL, - ALTER COLUMN email SET NOT NULL; -~~~ - -The file should now look something like this: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER TABLE IF EXISTS movr.max_schema.users - ADD COLUMN username STRING NOT NULL, - ALTER COLUMN email SET NOT NULL; - -ALTER TABLE IF EXISTS movr.max_schema.users ALTER PRIMARY KEY USING COLUMNS (username, email); -~~~ - -The remaining changes that you want to make will require `ALTER TABLE` statements with the `SET SCHEMA` and `OWNER TO` subcommands. An `ALTER TABLE ... SET SCHEMA` statement will change the contents of two schemas, and an `ALTER TABLE ... OWNER TO` statement will change the privileges of two users. To follow [authorization best practices](authorization.html#authorization-best-practices), you should execute any statements that change databases, user-defined schemas, or user privileges as a member of the `admin` role (e.g., as `root`). - -Create a new `.sql` file for the remaining `ALTER TABLE` statements, to be executed by `root`: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ touch update_users_owner.sql -~~~ - -Add the following statements to the file: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER TABLE IF EXISTS movr.max_schema.users SET SCHEMA abbey_schema; - -ALTER TABLE IF EXISTS movr.abbey_schema.users OWNER TO abbey; -~~~ - -To execute the statements in the `update_users_table.sql` file as `max`, run the following command: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql \ ---certs-dir={certs-directory} \ ---user=max \ ---database=movr \ --f update_users_table.sql -~~~ - -To execute the statements in the `update_users_owner.sql` file as `root`, run the following command: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql \ ---certs-dir={certs-directory} \ ---user=root \ ---database=movr \ --f update_users_owner.sql -~~~ - -The `users` table should now have a new column, a different primary key, a different schema, and a different owner. - -You can verify with some `SHOW` statements: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql \ ---certs-dir={certs-directory} \ ---user=abbey \ ---database=movr \ ---execute="SHOW SCHEMAS; SHOW TABLES; SHOW CREATE TABLE movr.abbey_schema.users;" -~~~ - -~~~ - schema_name | owner ----------------------+-------- - abbey_schema | abbey - crdb_internal | NULL - information_schema | NULL - max_schema | max - pg_catalog | NULL - pg_extension | NULL - public | admin -(7 rows) - - schema_name | table_name | type | owner | estimated_row_count ----------------+------------------+-------+-------+---------------------- - abbey_schema | user_promo_codes | table | abbey | 0 - abbey_schema | users | table | abbey | 0 - max_schema | rides | table | max | 0 - max_schema | vehicles | table | max | 0 -(4 rows) - - table_name | create_statement ---------------------------+----------------------------------------------------------------------------------- - movr.abbey_schema.users | CREATE TABLE abbey_schema.users ( - | first_name STRING NOT NULL, - | last_name STRING NOT NULL, - | email STRING NOT NULL, - | username STRING NOT NULL, - | CONSTRAINT "primary" PRIMARY KEY (username ASC, email ASC), - | UNIQUE INDEX users_first_name_last_name_key (first_name ASC, last_name ASC), - | UNIQUE INDEX users_email_key (email ASC), - | FAMILY "primary" (first_name, last_name, email, username) - | ) -(1 row) -~~~ - -## Drop database schema objects - -To drop an object from a database schema, use a `DROP` statement. - -`DROP` statements generally take the following form: - -~~~ -DROP {OBJECT_TYPE} {object_name} CASCADE; -~~~ - -Parameter | Description -----------|------------ -`{OBJECT_TYPE}` | The type of the object. -`{object_name}` | The name of the object. -`{CASCADE}` | An optional keyword that will drop all objects dependent on the object being dropped. - -For examples, see [below](#altering-objects-examples). - -CockroachDB supports the following `DROP` statements: - -- [`DROP DATABASE`](drop-database.html) -- [`DROP SCHEMA`](drop-schema.html) -- [`DROP TABLE`](drop-table.html) -- [`DROP INDEX`](drop-index.html) -- [`DROP SEQUENCE`](drop-sequence.html) -- [`DROP VIEW`](drop-view.html) -- [`DROP USER/ROLE`](drop-user.html) - -{{site.data.alerts.callout_info}} -To drop columns and column constraints from a table, use the `DROP COLUMN` and `DROP CONSTRAINT` subcommands of the [`ALTER TABLE`](alter-table.html) statement. -{{site.data.alerts.end}} - -### Drop best practices - -- Check the contents and dependencies of the object that you want to drop before using the `CASCADE` option. `CASCADE` drops all of the contents of an object, and should be used sparingly after a schema has been initialized. - -### Drop example - -Suppose that you want to drop an index that isn't being used very much. In particular, you want to drop the index on `first_name` and `last_name` from the `users` table. - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql \ ---certs-dir={certs-directory} \ ---user=abbey \ ---database=movr \ ---execute="SHOW INDEXES FROM movr.abbey_schema.users; SHOW CREATE TABLE movr.abbey_schema.users;" -~~~ - -~~~ - table_name | index_name | non_unique | seq_in_index | column_name | direction | storing | implicit --------------+--------------------------------+------------+--------------+-------------+-----------+---------+----------- - users | primary | false | 1 | username | ASC | false | false - users | primary | false | 2 | email | ASC | false | false - users | users_first_name_last_name_key | false | 1 | first_name | ASC | false | false - users | users_first_name_last_name_key | false | 2 | last_name | ASC | false | false - users | users_first_name_last_name_key | false | 3 | username | ASC | false | true - users | users_first_name_last_name_key | false | 4 | email | ASC | false | true - users | users_email_key | false | 1 | email | ASC | false | false - users | users_email_key | false | 2 | username | ASC | false | true -(8 rows) - - table_name | create_statement ---------------------------+----------------------------------------------------------------------------------- - movr.abbey_schema.users | CREATE TABLE abbey_schema.users ( - | first_name STRING NOT NULL, - | last_name STRING NOT NULL, - | email STRING NOT NULL, - | username STRING NOT NULL, - | CONSTRAINT "primary" PRIMARY KEY (username ASC, email ASC), - | UNIQUE INDEX users_first_name_last_name_key (first_name ASC, last_name ASC), - | UNIQUE INDEX users_email_key (email ASC), - | FAMILY "primary" (first_name, last_name, email, username) - | ) -(1 row) -~~~ - -Note that `users_first_name_last_name_key` is a `UNIQUE` index, which means that it implies a dependent, `UNIQUE` constraint. To [drop indexes with dependencies](drop-index.html#remove-an-index-and-dependent-objects-with-cascade), you can use the `CASCADE` keyword. - -Create a new file, and add the `DROP` statement: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ touch drop_unique_users_idx.sql -~~~ - -{{site.data.alerts.callout_info}} -After creation, the notation for referring to indexes in CockroachDB is `[table_name]@[index_name]`. -{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ -DROP INDEX movr.abbey_schema.users@users_first_name_last_name_key CASCADE; -~~~ - -To drop the index, execute the file: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql \ ---certs-dir={certs-directory} \ ---user=abbey \ ---database=movr \ --f drop_unique_users_idx.sql -~~~ - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach sql \ ---certs-dir={certs-directory} \ ---user=abbey \ ---database=movr \ ---execute="SHOW INDEXES FROM movr.abbey_schema.users;" -~~~ - -~~~ - table_name | index_name | non_unique | seq_in_index | column_name | direction | storing | implicit --------------+-----------------+------------+--------------+-------------+-----------+---------+----------- - users | primary | false | 1 | username | ASC | false | false - users | primary | false | 2 | email | ASC | false | false - users | users_email_key | false | 1 | email | ASC | false | false - users | users_email_key | false | 2 | username | ASC | false | true -(4 rows) -~~~ - -## What's next? - -- Read about [Online Schema Changes in CockroachDB](online-schema-changes.html). -- [Write data](insert-data.html) -- [Read data](query-data.html) - -You might also be interested in the following pages: - -- [Cockroach Commands](cockroach-commands.html) diff --git a/src/current/v21.1/secure-a-cluster.md b/src/current/v21.1/secure-a-cluster.md deleted file mode 100644 index b913c69666a..00000000000 --- a/src/current/v21.1/secure-a-cluster.md +++ /dev/null @@ -1,447 +0,0 @@ ---- -title: Start a Local Cluster (Secure) -summary: Run a secure multi-node CockroachDB cluster locally, using TLS certificates to encrypt network communication. -toc: true ---- - -
    - - -
    - -Once you've [installed CockroachDB](install-cockroachdb.html), it's simple to run a secure multi-node cluster locally, using [TLS certificates](cockroach-cert.html) to encrypt network communication. - -{% include cockroachcloud/use-cockroachcloud-instead.md %} - -## Before you begin - -- Make sure you have already [installed CockroachDB](install-cockroachdb.html). -- For quick SQL testing or app development, consider [running a single-node cluster](cockroach-start-single-node.html) instead. -- Note that running multiple nodes on a single host is useful for testing CockroachDB, but it's not suitable for production. To run a physically distributed cluster, see [Manual Deployment](manual-deployment.html) or [Orchestrated Deployment](orchestration.html), and review the [Production Checklist](recommended-production-settings.html). - -## Step 1. Generate certificates - -You can use either [`cockroach cert`](cockroach-cert.html) commands or [`openssl` commands](create-security-certificates-openssl.html) to generate security certificates. This section features the `cockroach cert` commands. - -1. Create two directories: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ mkdir certs my-safe-directory - ~~~ - - Directory | Description - ----------|------------ - `certs` | You'll generate your CA certificate and all node and client certificates and keys in this directory. - `my-safe-directory` | You'll generate your CA key in this directory and then reference the key when generating node and client certificates. - -2. Create the CA (Certificate Authority) certificate and key pair: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach cert create-ca \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key - ~~~ - -3. Create the certificate and key pair for your nodes: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach cert create-node \ - localhost \ - $(hostname) \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key - ~~~ - - Because you're running a local cluster and all nodes use the same hostname (`localhost`), you only need a single node certificate. Note that this is different than running a production cluster, where you would need to generate a certificate and key for each node, issued to all common names and IP addresses you might use to refer to the node as well as to any load balancer instances. - -4. Create a client certificate and key pair for the `root` user: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach cert create-client \ - root \ - --certs-dir=certs \ - --ca-key=my-safe-directory/ca.key - ~~~ - -## Step 2. Start the cluster - -1. Use the [`cockroach start`](cockroach-start.html) command to start the first node: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --certs-dir=certs \ - --store=node1 \ - --listen-addr=localhost:26257 \ - --http-addr=localhost:8080 \ - --join=localhost:26257,localhost:26258,localhost:26259 \ - --background - ~~~ - - You'll see a message like the following: - - ~~~ - * - * INFO: initial startup completed. - * Node will now attempt to join a running cluster, or wait for `cockroach init`. - * Client connections will be accepted after this completes successfully. - * Check the log file(s) for progress. - * - ~~~ - -2. Take a moment to understand the [flags](cockroach-start.html#flags) you used: - - The `--certs-dir` directory points to the directory holding certificates and keys. - - Since this is a purely local cluster, `--listen-addr=localhost:26257` and `--http-addr=localhost:8080` tell the node to listen only on `localhost`, with port `26257` used for internal and client traffic and port `8080` used for HTTP requests from the DB Console. - - The `--store` flag indicates the location where the node's data and logs are stored. - - The `--join` flag specifies the addresses and ports of the nodes that will initially comprise your cluster. You'll use this exact `--join` flag when starting other nodes as well. - - {% include {{ page.version.version }}/prod-deployment/join-flag-single-region.md %} - - The `--background` flag starts the `cockroach` process in the background so you can continue using the same terminal for other operations. - -3. Start two more nodes: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --certs-dir=certs \ - --store=node2 \ - --listen-addr=localhost:26258 \ - --http-addr=localhost:8081 \ - --join=localhost:26257,localhost:26258,localhost:26259 \ - --background - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --certs-dir=certs \ - --store=node3 \ - --listen-addr=localhost:26259 \ - --http-addr=localhost:8082 \ - --join=localhost:26257,localhost:26258,localhost:26259 \ - --background - ~~~ - - These commands are the same as before but with unique `--store`, `--listen-addr`, and `--http-addr` flags. - -4. Use the [`cockroach init`](cockroach-init.html) command to perform a one-time initialization of the cluster, sending the request to any node on the `--join` list: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach init --certs-dir=certs --host=localhost:26257 - ~~~ - - You'll see the following message: - - ~~~ - Cluster successfully initialized - ~~~ - - At this point, each node also prints helpful [startup details](cockroach-start.html#standard-output) to its log. For example, the following command retrieves node 1's startup details: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ grep 'node starting' node1/logs/cockroach.log -A 11 - ~~~ - - The output will look something like this: - - ~~~ - CockroachDB node starting at {{ now | date: "%Y-%m-%d %H:%M:%S.%6 +0000 UTC" }} - build: CCL {{page.release_info.version}} @ {{page.release_info.build_time}} (go1.12.6) - webui: https://localhost:8080 - sql: postgresql://root@localhost:26257?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt - RPC client flags: cockroach --host=localhost:26257 --certs-dir=certs - logs: /Users//node1/logs - temp dir: /Users//node1/cockroach-temp966687937 - external I/O path: /Users//node1/extern - store[0]: path=/Users//node1 - status: initialized new cluster - clusterID: b2537de3-166f-42c4-aae1-742e094b8349 - nodeID: 1 - ~~~ - -## Step 3. Use the built-in SQL client - -Now that your cluster is live, you can use any node as a SQL gateway. To test this out, let's use CockroachDB's built-in SQL client. - -1. Run the [`cockroach sql`](cockroach-sql.html) command against node 1: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --certs-dir=certs --host=localhost:26257 - ~~~ - -2. Run some basic [CockroachDB SQL statements](learn-cockroachdb-sql.html): - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE DATABASE bank; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE TABLE bank.accounts (id INT PRIMARY KEY, balance DECIMAL); - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > INSERT INTO bank.accounts VALUES (1, 1000.50); - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SELECT * FROM bank.accounts; - ~~~ - - ~~~ - id | balance - +----+---------+ - 1 | 1000.50 - (1 row) - ~~~ - -3. Now exit the SQL shell on node 1 and open a new shell on node 2: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > \q - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --certs-dir=certs --host=localhost:26258 - ~~~ - - {{site.data.alerts.callout_info}} - In a real deployment, all nodes would likely use the default port `26257`, and so you wouldn't need to set the port portion of `--host`. - {{site.data.alerts.end}} - -4. Run the same `SELECT` query as before: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SELECT * FROM bank.accounts; - ~~~ - - ~~~ - id | balance - +----+---------+ - 1 | 1000.50 - (1 row) - ~~~ - - As you can see, node 1 and node 2 behaved identically as SQL gateways. - -5. Now [create a user with a password](create-user.html#create-a-user-with-a-password), which you will need to [access the DB Console](#step-5-access-the-db-console): - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE USER max WITH PASSWORD 'roach'; - ~~~ - -6. Exit the SQL shell on node 2: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > \q - ~~~ - -## Step 4. Run a sample workload - -CockroachDB also comes with a number of [built-in workloads](cockroach-workload.html) for simulating client traffic. Let's run the workload based on CockroachDB's sample vehicle-sharing application, [MovR](movr.html). - -1. Load the initial dataset: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload init movr \ - 'postgresql://root@localhost:26257?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' - ~~~ - - ~~~ - I190926 16:50:35.663708 1 workload/workloadsql/dataload.go:135 imported users (0s, 50 rows) - I190926 16:50:35.682583 1 workload/workloadsql/dataload.go:135 imported vehicles (0s, 15 rows) - I190926 16:50:35.769572 1 workload/workloadsql/dataload.go:135 imported rides (0s, 500 rows) - I190926 16:50:35.836619 1 workload/workloadsql/dataload.go:135 imported vehicle_location_histories (0s, 1000 rows) - I190926 16:50:35.915498 1 workload/workloadsql/dataload.go:135 imported promo_codes (0s, 1000 rows) - ~~~ - -2. Run the workload for 5 minutes: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload run movr \ - --duration=5m \ - 'postgresql://root@localhost:26257?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt' - ~~~ - -## Step 5. Access the DB Console - -The CockroachDB [DB Console](ui-overview.html) gives you insight into the overall health of your cluster as well as the performance of the client workload. - -1. On secure clusters, [certain pages of the DB Console](ui-overview.html#db-console-access) can only be accessed by `admin` users. - - Run the [`cockroach sql`](cockroach-sql.html) command against node 1: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --certs-dir=certs --host=localhost:26257 - ~~~ - -2. Assign `max` to the `admin` role (you only need to do this once): - - {% include_cached copy-clipboard.html %} - ~~~ sql - > GRANT admin TO max; - ~~~ - -3. Exit the SQL shell: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > \q - ~~~ - -4. Go to https://localhost:8080. Note that your browser will consider the CockroachDB-created certificate invalid; you'll need to click through a warning message to get to the UI. - - {% include {{ page.version.version }}/misc/chrome-localhost.md %} - -5. Log in with the username and password you created earlier (`max`/`roach`). - -6. On the [**Cluster Overview**](ui-cluster-overview-page.html), notice that three nodes are live, with an identical replica count on each node: - - DB Console - - This demonstrates CockroachDB's [automated replication](demo-replication-and-rebalancing.html) of data via the Raft consensus protocol. - - {{site.data.alerts.callout_info}} - Capacity metrics can be incorrect when running multiple nodes on a single machine. For more details, see this [limitation](known-limitations.html#available-capacity-metric-in-the-db-console). - {{site.data.alerts.end}} - -7. Click [**Metrics**](ui-overview-dashboard.html) to access a variety of time series dashboards, including graphs of SQL queries and service latency over time: - - DB Console - -8. Use the [**Databases**](ui-databases-page.html), [**Statements**](ui-statements-page.html), and [**Jobs**](ui-jobs-page.html) pages to view details about your databases and tables, to assess the performance of specific queries, and to monitor the status of long-running operations like schema changes, respectively. - -## Step 6. Simulate node failure - -1. In a new terminal, run the [`cockroach quit`](cockroach-quit.html) command against a node to simulate a node failure: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach quit --certs-dir=certs --host=localhost:26259 - ~~~ - -2. Back in the DB Console, despite one node being "suspect", notice the continued SQL traffic: - - DB Console - - This demonstrates CockroachDB's use of the Raft consensus protocol to [maintain availability and consistency in the face of failure](demo-fault-tolerance-and-recovery.html); as long as a majority of replicas remain online, the cluster and client traffic continue uninterrupted. - -4. Restart node 3: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --certs-dir=certs \ - --store=node3 \ - --listen-addr=localhost:26259 \ - --http-addr=localhost:8082 \ - --join=localhost:26257,localhost:26258,localhost:26259 \ - --background - ~~~ - -## Step 7. Scale the cluster - -Adding capacity is as simple as starting more nodes with `cockroach start`. - -1. Start 2 more nodes: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --certs-dir=certs \ - --store=node4 \ - --listen-addr=localhost:26260 \ - --http-addr=localhost:8083 \ - --join=localhost:26257,localhost:26258,localhost:26259 \ - --background - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --certs-dir=certs \ - --store=node5 \ - --listen-addr=localhost:26261 \ - --http-addr=localhost:8084 \ - --join=localhost:26257,localhost:26258,localhost:26259 \ - --background - ~~~ - - Again, these commands are the same as before but with unique `--store`, `--listen-addr`, and `--http-addr` flags. - -2. Back on the **Cluster Overview** in the DB Console, you'll now see 5 nodes listed: - - DB Console - - At first, the replica count will be lower for nodes 4 and 5. Very soon, however, you'll see those numbers even out across all nodes, indicating that data is being [automatically rebalanced](demo-replication-and-rebalancing.html) to utilize the additional capacity of the new nodes. - -## Step 8. Stop the cluster - -1. When you're done with your test cluster, use the [`cockroach quit`](cockroach-quit.html) command to gracefully shut down each node. - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach quit --certs-dir=certs --host=localhost:26257 - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach quit --certs-dir=certs --host=localhost:26258 - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach quit --certs-dir=certs --host=localhost:26259 - ~~~ - - {{site.data.alerts.callout_info}} - For nodes 4 and 5, the shutdown process will take longer (about a minute each) and will eventually force the nodes to stop. This is because, with only 2 of 5 nodes left, a majority of replicas are not available, and so the cluster is no longer operational. - {{site.data.alerts.end}} - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach quit --certs-dir=certs --host=localhost:26260 - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach quit --certs-dir=certs --host=localhost:26261 - ~~~ - -2. To restart the cluster at a later time, run the same `cockroach start` commands as earlier from the directory containing the nodes' data stores. - - If you do not plan to restart the cluster, you may want to remove the nodes' data stores and the certificate directories: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ rm -rf node1 node2 node3 node4 node5 certs my-safe-directory - ~~~ - -## What's next? - -- [Install the client driver](install-client-drivers.html) for your preferred language -- Learn more about [CockroachDB SQL](learn-cockroachdb-sql.html) and the [built-in SQL client](cockroach-sql.html) -- Further explore CockroachDB capabilities like [fault tolerance and automated repair](demo-fault-tolerance-and-recovery.html), [multi-region performance](demo-low-latency-multi-region-deployment.html), [serializable transactions](demo-serializable.html), and [JSON support](demo-json-support.html) - -You might also be interested in the following pages: - -- [CockroachDB SQL client](cockroach-sql.html) -- [Hello World Example Apps](example-apps.html) diff --git a/src/current/v21.1/security-overview.md b/src/current/v21.1/security-overview.md deleted file mode 100644 index affdfcb4016..00000000000 --- a/src/current/v21.1/security-overview.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -title: CockroachDB Security -summary: Learn about the authentication, encryption, authorization, and audit log features for secure CockroachDB clusters. -toc: true ---- - -An insecure CockroachDB cluster comes with serious risks: - -- Your cluster is open to any client that can access any node's IP addresses. -- Any user, even `root`, can log in without providing a password. -- Any user, connecting as `root`, can read or write any data in your cluster. -- There is no network encryption or authentication, and thus no confidentiality. - -To avoid these security risks, CockroachDB provides authentication, encryption, authorization, and audit logging features to deploy secure clusters. Before we deep-dive into how these features work, let's discuss why we need each of these features. - -## Security overview - -It all starts with the desire for two parties to communicate securely over an insecure computer network. A conventional solution to ensure secure communication is **symmetric encryption** that involves encrypting and decrypting a plaintext message using a shared key. This seems like the logical solution until you realize that you need a secure communication channel to share the encryption key. This is a Catch-22 situation: How do you establish a secure channel to share the encryption key? - -An elegant solution is to use **Public Key Cryptography (PKI)** (also called **asymmetric encryption**) to establish a secure communication channel, and then sharing the symmetric encryption key over the secure channel. - -Asymmetric encryption involves a pair of keys instead of a single key. The two keys are called the **public key** and the **private key**. The keys consist of very long numbers linked mathematically in a way such that a message encrypted using a public key can only be decrypted using the private key and vice versa. The message cannot be decrypted using the same key that was used to encrypt the message. - -CockroachDB uses the [TLS 1.2](https://en.wikipedia.org/wiki/Transport_Layer_Security) security protocol that takes advantage of both symmetric as well as asymmetric encryption. The TLS 1.2 protocol uses asymmetric encryption to establish a secure channel as well as **authenticate** the communicating parties. It then uses symmetric encryption to protect data in flight. - -However, it's not enough to protect data in flight; you also need to protect data at rest. That's where CockroachDB's **Encryption at Rest** feature comes into the picture. Encryption at Rest is an Enterprise feature that allows encryption of all files on disk using [AES](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) in [counter mode](https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Counter_(CTR)), with all key -sizes allowed. - -Along with authentication and encryption, we also need to allow CockroachDB to restrict access to **authorized** clients (or nodes acting as clients). CockroachDB allows you to create, manage, and remove your cluster's [users](authorization.html#create-and-manage-users) and assign SQL-level [privileges](authorization.html#assign-privileges) to the users. Additionally, you can use [role-based access management (RBAC)](authorization.html#create-and-manage-roles) for simplified user management. - -Finally, CockroachDB's **SQL audit logging** gives you detailed information about queries being executed against your system. This feature is especially useful when you want to log all queries that are run against a table containing personally identifiable information (PII). - -The following section summarizes the CockroachDB security features and provides links to detailed documentation for each feature. - -## Security features in CockroachDB - -Security feature | Description --------------|------------ -[Authentication](authentication.html) |
    • Inter-node and node identity authentication using TLS 1.2
    • Client identity authentication using TLS 1.2 or username/password
    -[Encryption](encryption.html) |
    • Encryption in flight using TLS 1.2
    • Encryption at Rest using AES in counter mode (Enterprise feature)
    -[Authorization](authorization.html) |
    • Users and privileges
    • Role-based access control
    -[Audit logging](sql-audit-logging.html) | `ALTER TABLE...EXPERIMENTAL AUDIT` to get detailed information about queries being executed against your system diff --git a/src/current/v21.1/select-clause.md b/src/current/v21.1/select-clause.md deleted file mode 100644 index b365236830c..00000000000 --- a/src/current/v21.1/select-clause.md +++ /dev/null @@ -1,569 +0,0 @@ ---- -title: Simple SELECT Clause -summary: The Simple SELECT clause loads or computes data from various sources. -toc: true -key: select.html ---- - -The simple `SELECT` clause is the main SQL syntax to read and process -existing data. - -When used as a stand-alone statement, the simple `SELECT` clause is -also called "the `SELECT` statement". However, it is also a -[selection clause](selection-queries.html#selection-clauses) that can be combined -with other constructs to form more complex [selection queries](selection-queries.html). - - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/simple_select_clause.html %} -
    - - -{{site.data.alerts.callout_success}} -The simple `SELECT` clause also has other applications not covered here, such as executing [functions](functions-and-operators.html) like `SELECT current_timestamp();`. -{{site.data.alerts.end}} - -## Required privileges - -The user must have the `SELECT` [privilege](authorization.html#assign-privileges) on the tables used as operands. - -## Parameters - -Parameter | Description -----------|------------- -`DISTINCT` or `ALL` | See [Eliminate Duplicate Rows](#eliminate-duplicate-rows). -`DISTINCT ON ( a_expr [, ...] )` | `DISTINCT ON` followed by a list of [scalar expressions](scalar-expressions.html) within parentheses. See [Eliminate Duplicate Rows](#eliminate-duplicate-rows). -`target_elem` | A [scalar expression](scalar-expressions.html) to compute a column in each result row, or `*` to automatically retrieve all columns from the `FROM` clause.

    If `target_elem` contains an [aggregate function](functions-and-operators.html#aggregate-functions), a `GROUP BY` clause can be used to further control the aggregation. -`table_ref` | The [table expression](table-expressions.html) you want to retrieve data from.

    Using two or more table expressions in the `FROM` sub-clause, separated with a comma, is equivalent to a [`CROSS JOIN`](joins.html) expression. -`AS OF SYSTEM TIME timestamp` | Retrieve data as it existed [as of `timestamp`](as-of-system-time.html).

    **Note**: Because `AS OF SYSTEM TIME` returns historical data, your reads might be stale. -`WHERE a_expr` | Only retrieve rows that return `TRUE` for `a_expr`, which must be a [scalar expression](scalar-expressions.html) that returns Boolean values using columns (e.g., ` = `). -`GROUP BY a_expr` | Group results on one or more columns.

    When an [aggregate function](functions-and-operators.html#aggregate-functions) follows `SELECT` as a `target_elem`, or `HAVING` as an `a_expr`, you can [create aggregate groups](#create-aggregate-groups) on column groupings listed after `GROUP BY`.
    You can group columns by an alias (i.e., a label assigned to the column with an `AS` clause) rather than the column name.
    If aggregate groups are created on a full primary key, any column in the table can be selected as a `target_elem`, or specified in a `HAVING` clause.
    If a selected column is in a [subquery](subqueries.html), and the column references a higher scope, the column does not need to be included in the `GROUP BY` clause (if one exists).

    Using a `GROUP BY` clause in a statement without an aggregate function is equivalent to using a [`DISTINCT ON`](#eliminate-duplicate-rows) clause on the grouping columns. -`HAVING a_expr` | Only retrieve aggregate function groups that return `TRUE` for `a_expr`, which must be a [scalar expression](scalar-expressions.html) that returns Boolean values using an aggregate function (e.g., ` = `).

    `HAVING` works like the `WHERE` clause, but for aggregate functions. -`WINDOW window_definition_list` | A list of [window definitions](window-functions.html#window-definitions). - -## Eliminate duplicate rows - -The `DISTINCT` subclause specifies to remove duplicate rows. - -By default, or when `ALL` is specified, `SELECT` returns all the rows -selected, without removing duplicates. When `DISTINCT` is specified, -duplicate rows are eliminated. - -Without `ON`, two rows are considered duplicates if they are equal on -all the results computed by `SELECT`. - -With `ON`, two rows are considered duplicates if they are equal only -using the [scalar expressions](scalar-expressions.html) listed with `ON`. When two rows are considered duplicates according to `DISTINCT ON`, the values from the first `FROM` row in the order specified by [`ORDER BY`](order-by.html) are used to compute the remaining target expressions. If `ORDER BY` is not specified, CockroachDB will pick any one of the duplicate rows as first row, non-deterministically. - -## Examples - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### Choose columns - -#### Retrieve specific columns - -Retrieve specific columns by naming them in a comma-separated list: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT id, city, name FROM users LIMIT 10; -~~~ - -~~~ - id | city | name -+--------------------------------------+---------------+------------------+ - 7ae147ae-147a-4000-8000-000000000018 | los angeles | Alfred Garcia - 570a3d70-a3d7-4c00-8000-000000000011 | san francisco | Amy Cobb - 428f5c28-f5c2-4000-8000-00000000000d | seattle | Anita Atkinson - 1eb851eb-851e-4800-8000-000000000006 | boston | Brian Campbell - 23d70a3d-70a3-4800-8000-000000000007 | boston | Carl Mcguire - a8f5c28f-5c28-4800-8000-000000000021 | detroit | Carl Russell - 147ae147-ae14-4b00-8000-000000000004 | new york | Catherine Nelson - 99999999-9999-4800-8000-00000000001e | detroit | Charles Montoya - e147ae14-7ae1-4800-8000-00000000002c | paris | Cheyenne Smith - 2e147ae1-47ae-4400-8000-000000000009 | washington dc | Cindy Medina -(10 rows) -~~~ - -#### Retrieve all columns - -Retrieve all columns by using `*`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM users LIMIT 10; -~~~ - -~~~ - id | city | name | address | credit_card -+--------------------------------------+-----------+--------------------+--------------------------------+-------------+ - c28f5c28-f5c2-4000-8000-000000000026 | amsterdam | Maria Weber | 14729 Karen Radial | 5844236997 - c7ae147a-e147-4000-8000-000000000027 | amsterdam | Tina Miller | 97521 Mark Extensions | 8880478663 - cccccccc-cccc-4000-8000-000000000028 | amsterdam | Taylor Cunningham | 89214 Jennifer Well | 5130593761 - d1eb851e-b851-4800-8000-000000000029 | amsterdam | Kimberly Alexander | 48474 Alfred Hollow | 4059628542 - 19999999-9999-4a00-8000-000000000005 | boston | Nicole Mcmahon | 11540 Patton Extensions | 0303726947 - 1eb851eb-851e-4800-8000-000000000006 | boston | Brian Campbell | 92025 Yang Village | 9016427332 - 23d70a3d-70a3-4800-8000-000000000007 | boston | Carl Mcguire | 60124 Palmer Mews Apt. 49 | 4566257702 - 28f5c28f-5c28-4600-8000-000000000008 | boston | Jennifer Sanders | 19121 Padilla Brooks Apt. 12 | 1350968125 - 80000000-0000-4000-8000-000000000019 | chicago | Matthew Clay | 49220 Lisa Junctions | 9132291015 - 851eb851-eb85-4000-8000-00000000001a | chicago | Samantha Coffey | 6423 Jessica Underpass Apt. 87 | 9437219051 -(10 rows) -~~~ - -### Filter rows - -#### Filter on a single condition - -Filter rows with expressions that use columns and return Boolean values in the `WHERE` clause: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT name, id FROM users WHERE city='seattle'; -~~~ - -~~~ - name | id -+------------------+--------------------------------------+ - Anita Atkinson | 428f5c28-f5c2-4000-8000-00000000000d - Patricia Herrera | 47ae147a-e147-4000-8000-00000000000e - Holly Williams | 4ccccccc-cccc-4c00-8000-00000000000f - Ryan Hickman | 51eb851e-b851-4c00-8000-000000000010 -(4 rows) -~~~ - -#### Filter on multiple conditions - -To use multiple `WHERE` filters join them with `AND` or `OR`. You can also create negative filters with `NOT`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM vehicles WHERE city = 'seattle' AND status = 'available'; -~~~ - -~~~ - id | city | type | owner_id | creation_time | status | current_location | ext -+--------------------------------------+---------+------+--------------------------------------+---------------------------+-----------+------------------------+----------------------------------------+ - 44444444-4444-4400-8000-000000000004 | seattle | bike | 428f5c28-f5c2-4000-8000-00000000000d | 2019-01-02 03:04:05+00:00 | available | 37754 Farmer Extension | {"brand": "Merida", "color": "yellow"} -(1 row) -~~~ - -#### Filter values with a list - -Using `WHERE IN ()` performs an `OR` search for listed values in the specified column: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT name, id FROM users WHERE city IN ('new york', 'chicago', 'seattle'); -~~~ - -~~~ - name | id -+------------------+--------------------------------------+ - Matthew Clay | 80000000-0000-4000-8000-000000000019 - Samantha Coffey | 851eb851-eb85-4000-8000-00000000001a - Jessica Martinez | 8a3d70a3-d70a-4000-8000-00000000001b - John Hines | 8f5c28f5-c28f-4000-8000-00000000001c - Kenneth Barnes | 947ae147-ae14-4800-8000-00000000001d - Robert Murphy | 00000000-0000-4000-8000-000000000000 - James Hamilton | 051eb851-eb85-4ec0-8000-000000000001 - Judy White | 0a3d70a3-d70a-4d80-8000-000000000002 - Devin Jordan | 0f5c28f5-c28f-4c00-8000-000000000003 - Catherine Nelson | 147ae147-ae14-4b00-8000-000000000004 - Anita Atkinson | 428f5c28-f5c2-4000-8000-00000000000d - Patricia Herrera | 47ae147a-e147-4000-8000-00000000000e - Holly Williams | 4ccccccc-cccc-4c00-8000-00000000000f - Ryan Hickman | 51eb851e-b851-4c00-8000-000000000010 -(14 rows) -~~~ - -#### Select distinct rows - -Columns without the [Primary Key](primary-key.html) or [Unique](unique.html) constraints can have multiple instances of the same value: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT name FROM users WHERE city in ('los angeles', 'washington dc'); -~~~ - -~~~ - name -+---------------------+ - Ricky Beck - Michael Brown - William Wood - Alfred Garcia - Cindy Medina - Daniel Hernandez MD - Sarah Wang DDS - Michael Brown -(8 rows) -~~~ - -Using `DISTINCT`, you can remove all but one instance of duplicate values from your retrieved data: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT DISTINCT name FROM users WHERE city in ('los angeles', 'washington dc'); -~~~ - -~~~ - name -+---------------------+ - Ricky Beck - Michael Brown - William Wood - Alfred Garcia - Cindy Medina - Daniel Hernandez MD - Sarah Wang DDS -(7 rows) -~~~ - -### Rename columns in output - -Instead of outputting a column's name in the retrieved table, you can change its label using `AS`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT current_location AS ny_address, id, type, status FROM vehicles WHERE city = 'new york'; -~~~ - -~~~ - ny_address | id | type | status -+------------------------+--------------------------------------+------------+--------+ - 64110 Richard Crescent | 00000000-0000-4000-8000-000000000000 | skateboard | in_use - 86667 Edwards Valley | 11111111-1111-4100-8000-000000000001 | scooter | in_use -(2 rows) -~~~ - -This *does not* change the name of the column in the table. To do that, use [`RENAME COLUMN`](rename-column.html). - -### Search for string values - -Search for partial [string](string.html) matches in columns using `LIKE`, which supports the following wildcard operators: - -- `%` matches 0 or more characters. -- `_` matches exactly 1 character. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT city, status, id FROM vehicles WHERE type LIKE 'scoot%'; -~~~ - -~~~ - city | status | id -+---------------+-----------+--------------------------------------+ - boston | in_use | 22222222-2222-4200-8000-000000000002 - detroit | in_use | 99999999-9999-4800-8000-000000000009 - minneapolis | in_use | aaaaaaaa-aaaa-4800-8000-00000000000a - minneapolis | available | bbbbbbbb-bbbb-4800-8000-00000000000b - new york | in_use | 11111111-1111-4100-8000-000000000001 - san francisco | available | 55555555-5555-4400-8000-000000000005 - washington dc | in_use | 33333333-3333-4400-8000-000000000003 -(7 rows) -~~~ - -### Aggregate functions - -[Aggregate functions](functions-and-operators.html#aggregate-functions) perform calculations on retrieved rows. - -#### Perform aggregate function on entire column - -By using an aggregate function as a `target_elem`, you can perform the calculation on the entire column. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT MIN(revenue) FROM rides; -~~~ - -~~~ - min -+------+ - 0.00 -(1 row) -~~~ - -You can also use the retrieved value as part of an expression. For example, you can use the result in the `WHERE` clause to select additional rows that were not part of the aggregate function itself: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT id, city, vehicle_id, rider_id -FROM rides -WHERE revenue = ( - SELECT - MIN(revenue) - FROM rides -); -~~~ - -~~~ - id | city | vehicle_id | rider_id -+--------------------------------------+-------------+--------------------------------------+--------------------------------------+ - 1f3b645a-1cac-4800-8000-00000000003d | boston | 22222222-2222-4200-8000-000000000002 | 19999999-9999-4a00-8000-000000000005 - 23d70a3d-70a3-4800-8000-000000000046 | boston | 22222222-2222-4200-8000-000000000002 | 19999999-9999-4a00-8000-000000000005 - 851eb851-eb85-4000-8000-000000000104 | chicago | 88888888-8888-4800-8000-000000000008 | 851eb851-eb85-4000-8000-00000000001a - 85a1cac0-8312-4000-8000-000000000105 | chicago | 88888888-8888-4800-8000-000000000008 | 947ae147-ae14-4800-8000-00000000001d - 722d0e56-0418-4400-8000-0000000000df | los angeles | 77777777-7777-4800-8000-000000000007 | 7ae147ae-147a-4000-8000-000000000018 - ae147ae1-47ae-4800-8000-000000000154 | minneapolis | aaaaaaaa-aaaa-4800-8000-00000000000a | b851eb85-1eb8-4000-8000-000000000024 - 0dd2f1a9-fbe7-4c80-8000-00000000001b | new york | 11111111-1111-4100-8000-000000000001 | 00000000-0000-4000-8000-000000000000 - f4bc6a7e-f9db-4000-8000-0000000001de | rome | eeeeeeee-eeee-4000-8000-00000000000e | f0a3d70a-3d70-4000-8000-00000000002f -(8 rows) -~~~ - -#### Perform aggregate function on retrieved rows - -By filtering the statement, you can perform the calculation only on retrieved rows: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT SUM(revenue) FROM rides WHERE city IN ('new york', 'chicago'); -~~~ - -~~~ - sum -+---------+ - 4079.00 -(1 row) -~~~ - -#### Filter columns fed into aggregate functions - -You can use `FILTER (WHERE )` in the `target_elem` to filter which rows are processed by an aggregate function; those that return `FALSE` or `NULL` for the `FILTER` clause's Boolean expression are not fed into the aggregate function: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT count(*) AS unfiltered, count(*) FILTER (WHERE revenue > 50) AS filtered FROM rides; -~~~ - -~~~ - unfiltered | filtered -+------------+----------+ - 500 | 252 -(1 row) -~~~ - -#### Create aggregate groups - -Instead of performing aggregate functions on an the entire set of retrieved rows, you can split the rows into groups and then perform the aggregate function on each of them. - -When creating aggregate groups, each column selected as a `target_elem` must be included in a `GROUP BY` clause. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT city, SUM(revenue) AS city_revenue FROM rides -WHERE city IN ('new york', 'chicago', 'seattle') GROUP BY city; -~~~ - -~~~ - city | city_revenue -+----------+--------------+ - chicago | 1990.00 - new york | 2089.00 - seattle | 2029.00 -(3 rows) -~~~ - -{{site.data.alerts.callout_info}} - If the group is created on a primary key column, any column in the table can be selected as a `target_elem`. If a selected column is in a [subquery](subqueries.html) that references a higher scope, a `GROUP BY` clause is not needed. -{{site.data.alerts.end}} - -#### Filter aggregate groups - -To filter aggregate groups, use `HAVING`, which is the equivalent of the `WHERE` clause for aggregate groups, which must evaluate to a Boolean value. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT city, AVG(revenue) as avg FROM rides GROUP BY city -HAVING AVG(revenue) BETWEEN 50 AND 60; -~~~ - -~~~ - city | avg -+---------------+-----------------------+ - amsterdam | 52.50 - boston | 52.666666666666666667 - los angeles | 55.951219512195121951 - minneapolis | 55.146341463414634146 - washington dc | 58.756097560975609756 -(5 rows) -~~~ - -#### Use aggregate functions in having clause - -Aggregate functions can also be used in the `HAVING` clause without needing to be included as a `target_elem`. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT vehicle_id, city FROM rides WHERE city IN ('new york', 'chicago', 'seattle') -GROUP BY vehicle_id, city HAVING COUNT(vehicle_id) > 20; -~~~ - -~~~ - vehicle_id | city -+--------------------------------------+----------+ - 88888888-8888-4800-8000-000000000008 | chicago - 11111111-1111-4100-8000-000000000001 | new york - 44444444-4444-4400-8000-000000000004 | seattle -(3 rows) -~~~ - -#### Order aggregate function input rows by column - -Non-commutative aggregate functions are sensitive to the order in which the rows are processed in the surrounding `SELECT` clause. To specify the order in which input rows are processed, you can add an [`ORDER BY`](order-by.html) clause within the function argument list. - -For example, suppose you want to create an array of `name` values, ordered alphabetically, and grouped by `city`. You can use the following statement to do so: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT city, array_agg(name ORDER BY name) AS users FROM users WHERE city IN ('new york', 'chicago', 'seattle') GROUP BY city; -~~~ - -~~~ - city | users -+----------+-------------------------------------------------------------------------------------+ - new york | {"Catherine Nelson","Devin Jordan","James Hamilton","Judy White","Robert Murphy"} - seattle | {"Anita Atkinson","Holly Williams","Patricia Herrera","Ryan Hickman"} - chicago | {"Jessica Martinez","John Hines","Kenneth Barnes","Matthew Clay","Samantha Coffey"} -(3 rows) -~~~ - -You can also order input rows using a column different than the input row column. The following statement returns an array of `revenue` values from high-revenue rides, ordered by ride `end_time`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT city, array_agg(revenue ORDER BY end_time) as revenues FROM rides WHERE revenue > 80 GROUP BY city; -~~~ - -~~~ - city | revenues -+---------------+---------------------------------------------------------------------------------+ - amsterdam | {87.00,95.00,87.00,85.00,87.00,85.00,88.00,95.00,86.00,97.00,98.00,87.00,82.00} - boston | {92.00,92.00,86.00,87.00,94.00} - detroit | {89.00,96.00,94.00,92.00,84.00} - minneapolis | {84.00,98.00,86.00,92.00,81.00,99.00,87.00,86.00,88.00,81.00} - new york | {83.00,94.00,86.00,95.00,81.00,91.00,94.00,81.00,81.00,90.00} - san francisco | {96.00,85.00,96.00,84.00,94.00,87.00,93.00} - chicago | {82.00,98.00,84.00,99.00,91.00,90.00,83.00,82.00,91.00} - los angeles | {92.00,98.00,92.00,99.00,93.00,87.00,98.00,91.00,89.00,81.00,87.00} - paris | {87.00,94.00,98.00,98.00,95.00,81.00,99.00,94.00,95.00,82.00} - rome | {83.00,96.00,90.00,98.00,95.00,87.00,86.00,97.00} - seattle | {88.00,88.00,82.00,86.00,91.00,81.00,99.00} - washington dc | {96.00,94.00,97.00,96.00,88.00,97.00,93.00} -(12 rows) -~~~ - -If you include multiple aggregate functions in a single `SELECT` clause, you can order the input rows of the multiple functions on different columns. For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT city, array_agg(revenue ORDER BY revenue) as revenues_by_revenue, array_agg(revenue ORDER BY end_time) as revenues_by_end_time FROM rides WHERE revenue > 90 GROUP BY city; -~~~ -~~~ - city | revenues_by_revenue | revenues_by_end_time -+---------------+---------------------------------------------+---------------------------------------------+ - amsterdam | {95.00,95.00,97.00,98.00} | {95.00,95.00,97.00,98.00} - boston | {92.00,92.00,94.00} | {92.00,92.00,94.00} - minneapolis | {92.00,98.00,99.00} | {98.00,92.00,99.00} - new york | {91.00,94.00,94.00,95.00} | {94.00,95.00,91.00,94.00} - paris | {94.00,94.00,95.00,95.00,98.00,98.00,99.00} | {94.00,98.00,98.00,95.00,99.00,94.00,95.00} - san francisco | {93.00,94.00,96.00,96.00} | {96.00,96.00,94.00,93.00} - chicago | {91.00,91.00,98.00,99.00} | {98.00,99.00,91.00,91.00} - detroit | {92.00,94.00,96.00} | {96.00,94.00,92.00} - los angeles | {91.00,92.00,92.00,93.00,98.00,98.00,99.00} | {92.00,98.00,92.00,99.00,93.00,98.00,91.00} - rome | {95.00,96.00,97.00,98.00} | {96.00,98.00,95.00,97.00} - seattle | {91.00,99.00} | {91.00,99.00} - washington dc | {93.00,94.00,96.00,96.00,97.00,97.00} | {96.00,94.00,97.00,96.00,97.00,93.00} -(12 rows) -~~~ - -### Group by an alias - - If a query includes an alias (i.e., a [label assigned to the column with an `AS` clause](#rename-columns-in-output)), you can group the aggregations in the query by the alias rather than by the column name. For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT city AS c, SUM(revenue) AS c_rev FROM rides GROUP BY c; -~~~ -~~~ - c | c_rev -----------------+---------- - amsterdam | 2966.00 - boston | 3019.00 - los angeles | 2772.00 - new york | 2923.00 - paris | 2849.00 - rome | 2653.00 - san francisco | 2857.00 - seattle | 2792.00 - washington dc | 2797.00 -(9 rows) -~~~ - -### Select from a specific index - -{% include {{page.version.version}}/misc/force-index-selection.md %} - -### Select historical data (time-travel) - -CockroachDB lets you find data as it was stored at a given point in -time using `AS OF SYSTEM TIME` with various [supported -formats](as-of-system-time.html). This can be also advantageous for -performance. For more details, see [`AS OF SYSTEM -TIME`](as-of-system-time.html). - -## Advanced uses of `SELECT` clauses - -CockroachDB supports numerous ways to combine results from `SELECT` -clauses together. - -See [Selection Queries](selection-queries.html) for -details. A few examples follow. - -### Sorting and limiting query results - -To order the results of a `SELECT` clause or limit the number of rows -in the result, you can combine it with `ORDER BY` or `LIMIT` / -`OFFSET` to form a [selection query](selection-queries.html) or -[subquery](table-expressions.html#subqueries-as-table-expressions). - -See [Ordering Query Results](order-by.html) and [Limiting Query -Results](limit-offset.html) for more details. - -{{site.data.alerts.callout_info}}When ORDER BY is not included in a query, rows are not sorted by any consistent criteria. Instead, CockroachDB returns them as the coordinating node receives them.

    Also, CockroachDB sorts NULL values first with ASC and last with DESC. This differs from PostgreSQL, which sorts NULL values last with ASC and first with DESC.{{site.data.alerts.end}} - -### Combining results from multiple queries - -Results from two or more queries can be combined together as follows: - -- Using [`JOIN` expressions](joins.html) to combine rows - according to conditions on specific columns. -- Using [set operations](selection-queries.html#set-operations) to combine rows - using inclusion/exclusion rules. - -### Row-level locking for concurrency control with `SELECT FOR UPDATE` - -{% include {{page.version.version}}/sql/select-for-update-overview.md %} - -For an example showing how to use it, see [`SELECT FOR UPDATE`](select-for-update.html). - -## See also - -- [Scalar Expressions](scalar-expressions.html) -- [Selection Clauses](selection-queries.html#selection-clauses) -- [`SELECT FOR UPDATE`](select-for-update.html) -- [Set Operations](selection-queries.html#set-operations) -- [Table Expressions](table-expressions.html) -- [Ordering Query Results](order-by.html) -- [Limiting Query Results](limit-offset.html) -- [SQL Performance Best Practices](performance-best-practices-overview.html) diff --git a/src/current/v21.1/select-for-update.md b/src/current/v21.1/select-for-update.md deleted file mode 100644 index 3b1092990e0..00000000000 --- a/src/current/v21.1/select-for-update.md +++ /dev/null @@ -1,164 +0,0 @@ ---- -title: SELECT FOR UPDATE -summary: The SELECT FOR UPDATE statement is used to order transactions under contention. -keywords: concurrency control, locking, transactions, update locking, update, contention -toc: true ---- - -{% include {{page.version.version}}/sql/select-for-update-overview.md %} - -## Syntax - -The following diagram shows the supported syntax for the optional `FOR` locking clause of a `SELECT` statement. - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/for_locking.html %} -
    - -For the full `SELECT` statement syntax documentation, see [Selection Queries](selection-queries.html). - -## Parameters - -### Locking strengths - -Locking strength dictates the row-level locking behavior on rows retrieved by a `SELECT` statement. - -Parameter | Description -----------|------------ -`FOR SHARE`/`FOR KEY SHARE` | This syntax is a no-op, allowed for PostgreSQL compatibility. Specifying `FOR SHARE`/`FOR KEY SHARE` does not cause CockroachDB to use shared locks over the rows retrieved by a statement.

    Note that CockroachDB always [ensures serializability](demo-serializable.html), regardless of the specified locking strength. -`FOR UPDATE`/`FOR NO KEY UPDATE` | Lock the rows returned by the [`SELECT`](selection-queries.html) statement, such that other transactions trying to access the rows must wait for the transaction to finish.

    Note that in CockroachDB, the `FOR NO KEY UPDATE` locking strength is identical to the `FOR UPDATE` locking strength. - -### Wait policies - - Wait policies determine how a `SELECT FOR UPDATE` statement handles conflicts with locks held by other active transactions. By default, `SELECT FOR UPDATE` queries on rows that are already locked by an active transaction must wait for the transaction to finish. - -Parameter | Description -----------|------------ -`SKIP LOCKED` | This syntax is not supported, as skipping locked rows is not yet supported by CockroachDB. -`NOWAIT` | Return an error if a row cannot be locked immediately. - -For documentation on all other parameters of a `SELECT` statement, see [Selection Queries](selection-queries.html). - -## Required privileges - -The user must have the `SELECT` and `UPDATE` [privileges](authorization.html#assign-privileges) on the tables used as operands. - -## Examples - -### Enforce transaction order when updating the same rows - -This example uses `SELECT FOR UPDATE` to lock a row inside a transaction, forcing other transactions that want to update the same row to wait for the first transaction to complete. The other transactions that want to update the same row are effectively put into a queue based on when they first try to read the value of the row. - -This example assumes you are running a [local unsecured cluster](start-a-local-cluster.html). - -First, connect to the running cluster (call this Terminal 1): - -{% include_cached copy-clipboard.html %} -~~~ shell -cockroach sql --insecure -~~~ - -Next, create a table and insert some rows: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE kv (k INT PRIMARY KEY, v INT); -INSERT INTO kv (k, v) VALUES (1, 5), (2, 10), (3, 15); -~~~ - -Next, we'll start a [transaction](transactions.html) and lock the row we want to operate on: - -{% include_cached copy-clipboard.html %} -~~~ sql -BEGIN; -SELECT * FROM kv WHERE k = 1 FOR UPDATE; -~~~ - -Hit enter twice in the [SQL client](cockroach-sql.html) to send the input so far to be evaluated. This will result in the following output: - -~~~ - k | v -+---+----+ - 1 | 5 -(1 row) -~~~ - -Now open another terminal and connect to the database from a second client (call this Terminal 2): - -{% include_cached copy-clipboard.html %} -~~~ shell -cockroach sql --insecure -~~~ - -From Terminal 2, start a transaction and try to lock the same row for updates that is already being accessed by the transaction we opened in Terminal 1: - -{% include_cached copy-clipboard.html %} -~~~ sql -BEGIN; -SELECT * FROM kv WHERE k = 1 FOR UPDATE; -~~~ - -Hit enter twice to send the input so far to be evaluated. Because Terminal 1 has already locked this row, the `SELECT FOR UPDATE` statement from Terminal 2 will appear to "wait". - -Back in Terminal 1, update the row and commit the transaction: - -{% include_cached copy-clipboard.html %} -~~~ sql -UPDATE kv SET v = v + 5 WHERE k = 1; -~~~ - -~~~ -UPDATE 1 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -COMMIT; -~~~ - -~~~ -COMMIT -~~~ - -Now that the transaction in Terminal 1 has committed, the transaction in Terminal 2 will be "unblocked", generating the following output, which shows the value left by the transaction in Terminal 1: - -~~~ - k | v -+---+----+ - 1 | 10 -(1 row) -~~~ - -The transaction in Terminal 2 can now receive input, so update the row in question again: - -{% include_cached copy-clipboard.html %} -~~~ sql -UPDATE kv SET v = v + 5 WHERE k = 1; -~~~ - -~~~ -UPDATE 1 -~~~ - -Finally, commit the transaction in Terminal 2: - -{% include_cached copy-clipboard.html %} -~~~ sql -COMMIT; -~~~ - -~~~ -COMMIT -~~~ - -## See also - -- [`SELECT`](select-clause.html) -- [Selection Queries](selection-queries.html) -- [Understanding and avoiding transaction contention][transaction_contention] - - - -[transaction_contention]: performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention -[retries]: transactions.html#client-side-intervention -[select]: select-clause.html diff --git a/src/current/v21.1/selection-queries.md b/src/current/v21.1/selection-queries.md deleted file mode 100644 index a1900b008ef..00000000000 --- a/src/current/v21.1/selection-queries.md +++ /dev/null @@ -1,451 +0,0 @@ ---- -title: Selection Queries -summary: Selection queries can read and process data in CockroachDB. -toc: true -key: selection-clauses.html ---- - -Selection queries read and process data in CockroachDB. They are more -general than [simple `SELECT` clauses](select-clause.html): they can -group one or more [selection clauses](#selection-clauses) with [set -operations](#set-operations) and can request a [specific -ordering](order-by.html) or [row limit](limit-offset.html). - -Selection queries can occur: - -- At the top level of a query like other [SQL statements](sql-statements.html). -- Between parentheses as a [subquery](table-expressions.html#subqueries-as-table-expressions). -- As [operand to other statements](#using-selection-queries-with-other-statements) that take tabular data as input, for example [`INSERT`](insert.html), [`UPSERT`](upsert.html), [`CREATE TABLE AS`](create-table-as.html) or [`ALTER ... SPLIT AT`](split-at.html). - - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/select.html %} -
    - -## Parameters - -Parameter | Description -----------|------------ -`common_table_expr` | See [Common Table Expressions](common-table-expressions.html). -`select_clause` | A valid [selection clause](#selection-clauses), either simple or using [set operations](#set-operations). -`sort_clause` | An optional `ORDER BY` clause. See [Ordering Query Results](order-by.html) for details. -`limit_clause` | An optional `LIMIT` clause. See [Limiting Query Results](limit-offset.html) for details. -`offset_clause` | An optional `OFFSET` clause. See [Limiting Query Results](limit-offset.html) for details. -`for_locking_clause` | The `FOR UPDATE` locking clause is used to order transactions by controlling concurrent access to one or more rows of a table. For more information, see [`SELECT FOR UPDATE`](select-for-update.html). - -The optional `LIMIT` and `OFFSET` clauses can appear in any order, but must appear after `ORDER BY`, if also present. - -{{site.data.alerts.callout_info}}Because the WITH, ORDER BY, LIMIT and OFFSET sub-clauses are all optional, any simple selection clause is also a valid selection query.{{site.data.alerts.end}} - -## Selection clauses - -Selection clauses are the main component of a selection query. They -define tabular data. There are four specific syntax forms collectively named selection clauses: - -Form | Usage ------|-------- -[`SELECT`](#select-clause) | Load or compute tabular data from various sources. This is the most common selection clause. -[`VALUES`](#values-clause) | List tabular data by the client. -[`TABLE`](#table-clause) | Load tabular data from the database. -[Set Operations](#set-operations) | Combine tabular data from two or more selection clauses. - -{{site.data.alerts.callout_info}}To perform joins or other relational operations over selection clauses, use a table expression and convert it back into a selection clause with TABLE or SELECT.{{site.data.alerts.end}} - -### Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/select_clause.html %} -
    - -### `VALUES` clause - -#### Syntax - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/values_clause.html %} -
    - -A `VALUES` clause defines tabular data defined by the expressions -listed within parentheses. Each parenthesis group defines a single row -in the resulting table. - -The columns of the resulting table data have automatically generated -names. [These names can be modified with -`AS`](table-expressions.html#aliased-table-expressions) when the -`VALUES` clause is used as a sub-query. - -#### Example - -{% include_cached copy-clipboard.html %} -~~~ sql -> VALUES (1, 2, 3), (4, 5, 6); -~~~ - -~~~ -+---------+---------+---------+ -| column1 | column2 | column3 | -+---------+---------+---------+ -| 1 | 2 | 3 | -| 4 | 5 | 6 | -+---------+---------+---------+ -~~~ - -### `TABLE` clause - -#### Syntax - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/table_clause.html %} -
    - -A `TABLE` clause reads tabular data from a specified table. The -columns of the resulting table data are named after the schema of the -table. - -In general, `TABLE x` is equivalent to `SELECT * FROM x`, but it is -shorter to type. - -{{site.data.alerts.callout_info}}Any table expression between parentheses is a valid operand for TABLE, not just simple table or view names.{{site.data.alerts.end}} - -#### Example - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE employee_copy AS TABLE employee; -~~~ - -This statement copies the content from table `employee` into a new -table. However, note that the `TABLE` clause does not preserve the indexing, -foreign key, or constraint and default information from the schema of the -table it reads from, so in this example, the new table `employee_copy` -will likely have a simpler schema than `employee`. - -Other examples: - -{% include_cached copy-clipboard.html %} -~~~ sql -> TABLE employee; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO employee_copy TABLE employee; -~~~ - -### `SELECT` clause - -See [Simple `SELECT` Clause](select-clause.html) for more -details. - -## Set operations - -Set operations combine data from two [selection -clauses](#selection-clauses). They are valid as operand to other -set operations or as main component in a selection query. - -### Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/set_operation.html %} -
    - -### Set operators - -SQL lets you compare the results of multiple [selection clauses](#selection-clauses). You can think of each of the set operators as representing a Boolean operator: - -- `UNION` = OR -- `INTERSECT` = AND -- `EXCEPT` = NOT - -By default, each of these comparisons displays only one copy of each value (similar to `SELECT DISTINCT`). However, each function also lets you add an `ALL` to the clause to display duplicate values. - -### Union: Combine two queries - -`UNION` combines the results of two queries into one result. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT name -FROM accounts -WHERE state_opened IN ('AZ', 'NY') -UNION -SELECT name -FROM mortgages -WHERE state_opened IN ('AZ', 'NY'); -~~~ -~~~ -+-----------------+ -| name | -+-----------------+ -| Naseem Joossens | -| Ricarda Caron | -| Carola Dahl | -| Aygün Sanna | -+-----------------+ -~~~ - -To show duplicate rows, you can use `ALL`. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT name -FROM accounts -WHERE state_opened IN ('AZ', 'NY') -UNION ALL -SELECT name -FROM mortgages -WHERE state_opened IN ('AZ', 'NY'); -~~~ -~~~ -+-----------------+ -| name | -+-----------------+ -| Naseem Joossens | -| Ricarda Caron | -| Carola Dahl | -| Naseem Joossens | -| Aygün Sanna | -| Carola Dahl | -+-----------------+ -~~~ - -### Intersect: Retrieve intersection of two queries - -`INTERSECT` finds only values that are present in both query operands. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT name -FROM accounts -WHERE state_opened IN ('NJ', 'VA') -INTERSECT -SELECT name -FROM mortgages; -~~~ -~~~ -+-----------------+ -| name | -+-----------------+ -| Danijel Whinery | -| Agar Archer | -+-----------------+ -~~~ - -### Except: Exclude one query's results from another - -`EXCEPT` finds values that are present in the first query operand but not the second. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT name -FROM mortgages -EXCEPT -SELECT name -FROM accounts; -~~~ -~~~ -+------------------+ -| name | -+------------------+ -| Günay García | -| Karla Goddard | -| Cybele Seaver | -+------------------+ -~~~ - -## Ordering results - -The following sections provide examples. For more details, see [Ordering Query Results](order-by.html). - -### Order retrieved rows by one column - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * -FROM accounts -WHERE balance BETWEEN 350 AND 500 -ORDER BY balance DESC; -~~~ -~~~ -+----+--------------------+---------+----------+--------------+ -| id | name | balance | type | state_opened | -+----+--------------------+---------+----------+--------------+ -| 12 | Raniya Žitnik | 500 | savings | CT | -| 59 | Annibale Karga | 500 | savings | ND | -| 27 | Adelbert Ventura | 500 | checking | IA | -| 86 | Theresa Slaski | 500 | checking | WY | -| 73 | Ruadh Draganov | 500 | checking | TN | -| 16 | Virginia Ruan | 400 | checking | HI | -| 43 | Tahirih Malinowski | 400 | checking | MS | -| 50 | Dusan Mallory | 350 | savings | NV | -+----+--------------------+---------+----------+--------------+ -~~~ - -### Order retrieved rows by multiple columns - -Columns are sorted in the order you list them in `sortby_list`. For example, `ORDER BY a, b` sorts the rows by column `a` and then sorts rows with the same `a` value by their column `b` values. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * -FROM accounts -WHERE balance BETWEEN 350 AND 500 -ORDER BY balance DESC, name ASC; -~~~ -~~~ -+----+--------------------+---------+----------+--------------+ -| id | name | balance | type | state_opened | -+----+--------------------+---------+----------+--------------+ -| 27 | Adelbert Ventura | 500 | checking | IA | -| 59 | Annibale Karga | 500 | savings | ND | -| 12 | Raniya Žitnik | 500 | savings | CT | -| 73 | Ruadh Draganov | 500 | checking | TN | -| 86 | Theresa Slaski | 500 | checking | WY | -| 43 | Tahirih Malinowski | 400 | checking | MS | -| 16 | Virginia Ruan | 400 | checking | HI | -| 50 | Dusan Mallory | 350 | savings | NV | -+----+--------------------+---------+----------+--------------+ -~~~ - -## Limiting row count - -You can reduce the number of results with `LIMIT`. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT id, name -FROM accounts -LIMIT 5; -~~~ -~~~ -+----+------------------+ -| id | name | -+----+------------------+ -| 1 | Bjorn Fairclough | -| 2 | Bjorn Fairclough | -| 3 | Arturo Nevin | -| 4 | Arturo Nevin | -| 5 | Naseem Joossens | -+----+------------------+ -~~~ - -## Row-level locking for concurrency control with `SELECT FOR UPDATE` - -{% include {{page.version.version}}/sql/select-for-update-overview.md %} - -For an example showing how to use it, see [`SELECT FOR UPDATE`](select-for-update.html). - -## Composability - -[Selection clauses](#selection-clauses) are defined in the context of selection queries. [Table expressions](table-expressions.html) are defined in the context of the `FROM` sub-clause of [`SELECT`](select-clause.html). Nevertheless, they can be integrated with one another to form more complex queries or statements. - -### Using any selection clause as a selection query - -Any [selection clause](#selection-clauses) can be used as a -selection query with no change. - -For example, the construct [`SELECT * FROM accounts`](select-clause.html) is a selection clause. It is also a valid selection query, and thus can be used as a stand-alone statement by appending a semicolon: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM accounts; -~~~ -~~~ -+----+-----------------------+---------+----------+--------------+ -| id | name | balance | type | state_opened | -+----+-----------------------+---------+----------+--------------+ -| 1 | Bjorn Fairclough | 1200 | checking | AL | -| 2 | Bjorn Fairclough | 2500 | savings | AL | -| 3 | Arturo Nevin | 250 | checking | AK | -[ truncated ] -+----+-----------------------+---------+----------+--------------+ -~~~ - -Likewise, the construct [`VALUES (1), (2), (3)`](#values-clause) is also a selection -clause and thus can also be used as a selection query on its own: - -{% include_cached copy-clipboard.html %} -~~~ sql -> VALUES (1), (2), (3); -~~~ -~~~ -+---------+ -| column1 | -+---------+ -| 1 | -| 2 | -| 3 | -+---------+ -(3 rows) -~~~ - -### Using any table expression as selection clause - -Any [table expression](table-expressions.html) can be used as a selection clause (and thus also a selection query) by prefixing it with `TABLE` or by using it as an operand to `SELECT * FROM`. - -For example, the [simple table name](table-expressions.html#table-or-view-names) `customers` is a table expression, which designates all rows in that table. The expressions [`TABLE accounts`](selection-queries.html#table-clause) and [`SELECT * FROM accounts`](select-clause.html) are valid selection clauses. - -Likewise, the [SQL join expression](joins.html) `customers c JOIN orders o ON c.id = o.customer_id` is a table expression. You can turn it into a valid selection clause, and thus a valid selection query as follows: - -{% include_cached copy-clipboard.html %} -~~~ sql -> TABLE (customers c JOIN orders o ON c.id = o.customer_id); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM customers c JOIN orders o ON c.id = o.customer_id; -~~~ - -### Using any selection query as table expression - -Any selection query (or [selection clause](#selection-clauses)) can be used as a [table -expression](table-expressions.html) by enclosing it between parentheses, which forms a -[subquery](table-expressions.html#subqueries-as-table-expressions). - -For example, the following construct is a selection query, but is not a valid table expression: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM customers ORDER BY name LIMIT 5 -~~~ - -To make it valid as operand to `FROM` or another table expression, you can enclose it between parentheses as follows: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT id FROM (SELECT * FROM customers ORDER BY name LIMIT 5); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT o.id - FROM orders o - JOIN (SELECT * FROM customers ORDER BY name LIMIT 5) AS c - ON o.customer_id = c.id; -~~~ - -### Using selection queries with other statements - -Selection queries are also valid as operand in contexts that require tabular data. - -For example: - - Statement | Example using `SELECT` | Example using `VALUES` | Example using `TABLE` | -|----------------|-----------------------------------|------------------------------------|------------------------------- - [`INSERT`](insert.html) | `INSERT INTO foo SELECT * FROM bar` | `INSERT INTO foo VALUES (1), (2), (3)` | `INSERT INTO foo TABLE bar` - [`UPSERT`](upsert.html) | `UPSERT INTO foo SELECT * FROM bar` | `UPSERT INTO foo VALUES (1), (2), (3)` | `UPSERT INTO foo TABLE bar` - [`CREATE TABLE AS`](create-table-as.html) | `CREATE TABLE foo AS SELECT * FROM bar` `CREATE TABLE foo AS VALUES (1),(2),(3)` | `CREATE TABLE foo AS TABLE bar` - [`ALTER ... SPLIT AT`](split-at.html) | `ALTER TABLE foo SPLIT AT SELECT * FROM bar` `ALTER TABLE foo SPLIT AT VALUES (1),(2),(3)` | `ALTER TABLE foo SPLIT AT TABLE bar` - Subquery in a [table expression](table-expressions.html) | `SELECT * FROM (SELECT * FROM bar)` | `SELECT * FROM (VALUES (1),(2),(3))` | `SELECT * FROM (TABLE bar)` - Subquery in a [scalar expression](scalar-expressions.html) | `SELECT * FROM foo WHERE x IN (SELECT * FROM bar)` | `SELECT * FROM foo WHERE x IN (VALUES (1),(2),(3))` | `SELECT * FROM foo WHERE x IN (TABLE bar)` - -## See also - -- [Simple `SELECT` Clause](select-clause.html) -- [`SELECT FOR UPDATE`](select-for-update.html) -- [Table Expressions](table-expressions.html) -- [Ordering Query Results](order-by.html) -- [Limiting Query Results](limit-offset.html) diff --git a/src/current/v21.1/serial.md b/src/current/v21.1/serial.md deleted file mode 100644 index e5accd3aaf5..00000000000 --- a/src/current/v21.1/serial.md +++ /dev/null @@ -1,278 +0,0 @@ ---- -title: SERIAL -summary: The SERIAL pseudo-type produces integer values automatically. -toc: true ---- - -The `SERIAL` pseudo [data type](data-types.html) is a keyword that can -be used *in lieu* of a real data type when defining table columns. It -is approximately equivalent to using an [integer type](int.html) with -a [`DEFAULT` expression](default-value.html) that generates different -values every time it is evaluated. This default expression in turn -ensures that inserts that do not specify this column will receive an -automatically generated value instead of `NULL`. - -{{site.data.alerts.callout_info}} -`SERIAL` is provided only for compatibility with PostgreSQL. New applications should use real data types and a suitable `DEFAULT` expression. - -In most cases, we recommend using the [`UUID`](uuid.html) data type with the `gen_random_uuid()` function as the default value, which generates 128-bit values (larger than `SERIAL`'s maximum of 64 bits) and more uniformly scatters them across all of a table's underlying key-value ranges. UUIDs ensure more effectively that multiple nodes share the insert load when a UUID column is used in an index or primary key. - -See [this FAQ entry](sql-faqs.html#how-do-i-auto-generate-unique-row-ids-in-cockroachdb) for more details. -{{site.data.alerts.end}} - -## Modes of operation - -The keyword `SERIAL` is recognized in `CREATE TABLE` and is -automatically translated to a real data type and a [`DEFAULT` -expression](default-value.html) during table creation. -The result of this translation is then used internally by CockroachDB, -and can be observed using [`SHOW CREATE`](show-create.html). - -The chosen `DEFAULT` expression ensures that different values are -automatically generated for the column during row insertion. These -are not guaranteed to increase monotonically, see [this section -below](#auto-incrementing-is-not-always-sequential) for details. - -There are three possible translation modes for `SERIAL`: - - Mode | Description ----------------------|------------------ - `rowid` (default) | `SERIAL` implies `DEFAULT unique_rowid()`. The real data type is always `INT8`, regardless of the `default_int_size` [session variable](set-vars.html) or `sql.defaults.default_int_size` [cluster setting](cluster-settings.html). - `virtual_sequence` | `SERIAL` creates a virtual sequence and implies `DEFAULT nextval()`. The real data type is always `INT8`, regardless of the `default_int_size` session variable or `sql.defaults.default_int_size` cluster setting. - `sql_sequence` | `SERIAL` creates a regular SQL sequence and implies `DEFAULT nextval()`. The real data type depends on `SERIAL` variant. - `sql_sequence_cached`| `SERIAL` creates a regular SQL sequence and implies `DEFAULT nextval()`, caching the results for reuse in the current session. The real data type depends on `SERIAL` variant. When the cache is empty, the underlying sequence will only be incremented once to populate the cache.
    When this mode is set, the `sql.defaults.serial_sequences_cache_size` [cluster setting](cluster-settings.html) controls the number of values to cache in a user's session, with a default of 256. - -These modes can be configured with the [session variable](set-vars.html) `serial_normalization`. - -{{site.data.alerts.callout_info}} -The particular choice of `DEFAULT` expression when clients use the -`SERIAL` keyword is subject to change in future versions of -CockroachDB. Applications that wish to use `unique_rowid()` -specifically must use the full explicit syntax `INT DEFAULT -unique_rowid()` and avoid `SERIAL` altogether. -{{site.data.alerts.end}} - -### Generated values for modes `rowid` and `virtual_sequence` - -In both modes `rowid` and `virtual_sequence`, a value is automatically -generated using the `unique_rowid()` function. -This produces a 64-bit integer (i.e., [`INT8`](int.html)) from the current timestamp and ID of -the node executing the [`INSERT`](insert.html) or [`UPSERT`](upsert.html) operation. -This behavior is statistically likely to be globally unique except in -extreme cases (see [this FAQ -entry](sql-faqs.html#how-do-i-auto-generate-unique-row-ids-in-cockroachdb) -for more details). - -Also, because value generation using `unique_rowid()` does not require -inter-node coordination, it is much faster than the other mode -`sql_sequence` discussed below when multiple SQL clients are writing to -the table from different nodes. - -{{site.data.alerts.callout_info}} -The difference between `rowid` and `virtual_sequence` is that the -latter setting also creates a virtual (pseudo) sequence in the -database. However in both cases the `unique_rowid()` function is -ultimately used to generate new values. -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -Values generated by the `unique_rowid()` function do not respect the `default_int_size` [session variable](set-vars.html) or `sql.defaults.default_int_size` [cluster setting](cluster-settings.html). -{{site.data.alerts.end}} - -### Generated values for mode `sql_sequence` and `sql_sequence_cached` - -In both modes, a regular [SQL sequence](create-sequence.html) is -automatically created alongside the table where `SERIAL` is specified. - -The actual data type is determined as follows: - -| `SERIAL` variant | Real data type | -|---|---| -| `SERIAL2`, `SMALLSERIAL` | `INT2` | -| `SERIAL4` | `INT4` | -| `SERIAL` | `INT` | -| `SERIAL8`, `BIGSERIAL` | `INT8` | - -Every insert or upsert into the table will then use `nextval()` to -increment the sequence and produce increasing values. - -Because SQL sequences persist the current sequence value in the -database, inter-node coordination is required when multiple clients -use the sequence concurrently via different nodes. This can cause -[contention](sql-faqs.html#what-is-transaction-contention) and impact -performance negatively. - -Therefore, applications should consider using `unique_rowid()` or -`gen_random_uuid()` as discussed in [this FAQ -entry](sql-faqs.html#how-do-i-auto-generate-unique-row-ids-in-cockroachdb) -instead of sequences when possible. - -{{site.data.alerts.callout_info}} -Note that `sql_sequence_cached` will perform fewer distributed calls to increment sequences, resulting in better performance than `sql_sequence`. However, cached sequences may result in large gaps between serial sequence numbers if a session terminates before using all the values in its cache. -{{site.data.alerts.end}} - - -## Examples - -### Use `SERIAL` to auto-generate primary keys - -In this example, we create a table with the `SERIAL` column as the primary key so we can auto-generate unique IDs on insert. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE serial (a SERIAL PRIMARY KEY, b STRING, c BOOL); -~~~ - -The [`SHOW COLUMNS`](show-columns.html) statement shows that the `SERIAL` type is just an alias for `INT` with `unique_rowid()` as the default. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM serial; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden ---------------+-----------+-------------+----------------+-----------------------+-----------+------------ - a | INT8 | false | unique_rowid() | | {primary} | false - b | STRING | true | NULL | | {} | false - c | BOOL | true | NULL | | {} | false -(3 rows) -~~~ - -When we insert rows without values in column `a` and display the new rows, we see that each row has defaulted to a unique value in column `a`. - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO serial (b,c) VALUES ('red', true), ('yellow', false), ('pink', true); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO serial (a,b,c) VALUES (123, 'white', false); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM serial; -~~~ - -~~~ - a | b | c ----------------------+--------+-------- - 123 | white | false - 645993277462937601 | red | true - 645993277462970369 | yellow | false - 645993277463003137 | pink | true -(4 rows) -~~~ - -## Auto-incrementing is not always sequential - -It's a common misconception that the auto-incrementing types in PostgreSQL and MySQL generate strictly sequential values. However, there can be gaps and the order is not completely guaranteed: - -- Each insert increases the sequence by one, even when the insert is not committed. This means that auto-incrementing types may leave gaps in a sequence. -- Two concurrent transactions can commit in a different order than their use of sequences, and thus "observe" the values to decrease relative to each other. This effect is amplified by automatic transaction retries. - -These are fundamental properties of a transactional system with non-transactional sequences. PostgreSQL, MySQL, and CockroachDB do not increase sequences transactionally with other SQL statements, so these effects can happen in any case. - -To experience this for yourself, run through the following example in PostgreSQL: - -1. Create a table with a `SERIAL` column: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE TABLE increment (a SERIAL PRIMARY KEY); - ~~~ - -2. Run four transactions for inserting rows: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > BEGIN; - > INSERT INTO increment DEFAULT VALUES; - > ROLLBACK; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > BEGIN; - > INSERT INTO increment DEFAULT VALUES; - > COMMIT; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > BEGIN; - > INSERT INTO increment DEFAULT VALUES; - > ROLLBACK; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > BEGIN; - > INSERT INTO increment DEFAULT VALUES; - > COMMIT; - ~~~ - -3. View the rows created: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SELECT * from increment; - ~~~ - - ~~~ - a - ---------------------- - 645994218934534145 - 645994218978082817 - (2 rows) - ~~~ - - Since each insert increased the sequence in column `a` by one, the first committed insert got the value `645994218934534145`, and the second committed insert got the value `645994218978082817`. As you can see, the values aren't strictly sequential, and the last value doesn't give an accurate count of rows in the table. - -In summary, the `SERIAL` type in PostgreSQL and CockroachDB, and the `AUTO_INCREMENT` type in MySQL, all behave the same in that they do not create strict sequences. CockroachDB will likely create more gaps than these other databases, but will generate these values much faster. An alternative is to use [`SEQUENCE`](create-sequence.html). - -### Additional examples - -If two transactions occur concurrently, CockroachDB cannot guarantee monotonically increasing IDs (i.e., first commit is smaller than second commit). Here are three more scenarios that demonstrate this: - -Scenario 1: - -- At time 1, transaction `T1` `BEGIN`s. -- At time 2, transaction `T2` `BEGIN`s on the same node (from a different client). -- At time 3, transaction `T1` creates a `SERIAL` value, `x`. -- At time 3 + 2 microseconds, transaction `T2` creates a `SERIAL` value, `y`. -- At time 4, transaction `T1` `COMMIT`s. -- At time 5, transaction `T2` `COMMIT`s. - -If this happens, CockroachDB cannot guarantee whether `x < y` or `x > y`, despite the fact `T1` and `T2` began and were committed in different times. In this particular example, it's even likely that `x = y` because there is less than a 10-microsecond difference and the `SERIAL` values are constructed from the number of microseconds in the current time. - -Scenario 2: - -- At time 1, transaction `T1` `BEGIN`s. -- At time 1, transaction `T2` `BEGIN`s somewhere else, on a different node. -- At time 2, transaction `T1` creates a `SERIAL` value, `x`. -- At time 3, transaction `T2` creates a `SERIAL` value, `y`. -- At time 4, transaction `T1` `COMMIT`s. -- At time 4, transaction `T2` `COMMIT`s. - -If this happens, CockroachDB cannot guarantee whether `x < y` or `x > y`. Both can happen, even though the transactions began and committed at the same time. However it's sure that `x != y` because the values were generated on different nodes. - -Scenario 3: - -- At time 1, transaction `T1` `BEGIN`s. -- At time 2, transaction `T1` creates a `SERIAL` value, `x`. -- At time 3, transaction `T1` `COMMIT`s. -- At time 4, transaction `T2` `BEGIN`s somewhere else, on a different node. -- At time 5, transaction `T2` creates a `SERIAL` value, `y`. -- At time 6, transaction `T2` `COMMIT`s. - -There is less than a 250-microsecond difference between the system clocks of the two nodes. - -If this happens, CockroachDB cannot guarantee whether `x < y` or `x > y`. Even though the transactions "clearly" occurred one "after" the other, perhaps there was a clock skew between the two nodes and the system time of the second node is set earlier than the first node. - -## See also - -- [FAQ: How do I auto-generate unique row IDs in CockroachDB?](sql-faqs.html#how-do-i-auto-generate-unique-row-ids-in-cockroachdb) -- [Data Types](data-types.html) diff --git a/src/current/v21.1/set-cluster-setting.md b/src/current/v21.1/set-cluster-setting.md deleted file mode 100644 index 02f1ebd5ea2..00000000000 --- a/src/current/v21.1/set-cluster-setting.md +++ /dev/null @@ -1,119 +0,0 @@ ---- -title: SET CLUSTER SETTING -summary: The SET CLUSTER SETTING statement configures one cluster setting. -toc: true ---- - -The `SET CLUSTER SETTING` [statement](sql-statements.html) modifies a [cluster-wide setting](cluster-settings.html). - -{{site.data.alerts.callout_danger}}Many cluster settings are intended for tuning CockroachDB internals. Before changing these settings, we strongly encourage you to discuss your goals with CockroachDB; otherwise, you use them at your own risk.{{site.data.alerts.end}} - - -## Required privileges - -Only members of the `admin` role can modify cluster settings. By default, the `root` user belongs to the `admin` role. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/set_cluster_setting.html %} -
    - -{{site.data.alerts.callout_info}}The SET CLUSTER SETTING statement is unrelated to the other SET TRANSACTION and SET (session variable) statements.{{site.data.alerts.end}} - -## Parameters - -| Parameter | Description | -|-----------|-------------| -| `var_name` | The name of the [cluster setting](cluster-settings.html) (case-insensitive). | -| `var_value` | The value for the [cluster setting](cluster-settings.html). | -| `DEFAULT` | Reset the [cluster setting](cluster-settings.html) to its default value.

    The [`RESET CLUSTER SETTING`](reset-cluster-setting.html) resets a cluster setting as well. | - -## Examples - -### Change the default distributed execution parameter - -To configure a cluster so that new sessions automatically try to run queries [in a distributed fashion](https://www.cockroachlabs.com/blog/local-and-distributed-processing-in-cockroachdb/): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET CLUSTER SETTING sql.defaults.distsql = 1; -~~~ - -To disable distributed execution for all new sessions: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET CLUSTER SETTING sql.defaults.distsql = 0; -~~~ - -### Disable automatic diagnostic reporting - -To opt out of [automatic diagnostic reporting](diagnostics-reporting.html) of usage data to Cockroach Labs: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET CLUSTER SETTING diagnostics.reporting.enabled = false; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CLUSTER SETTING diagnostics.reporting.enabled; -~~~ - -~~~ -+-------------------------------+ -| diagnostics.reporting.enabled | -+-------------------------------+ -| false | -+-------------------------------+ -(1 row) -~~~ - -### Reset a setting to its default value - -{{site.data.alerts.callout_success}}You can use RESET CLUSTER SETTING to reset a cluster setting as well.{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET CLUSTER SETTING sql.metrics.statement_details.enabled = false; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CLUSTER SETTING sql.metrics.statement_details.enabled; -~~~ - -~~~ -+---------------------------------------+ -| sql.metrics.statement_details.enabled | -+---------------------------------------+ -| false | -+---------------------------------------+ -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET CLUSTER SETTING sql.metrics.statement_details.enabled = DEFAULT; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CLUSTER SETTING sql.metrics.statement_details.enabled; -~~~ - -~~~ -+---------------------------------------+ -| sql.metrics.statement_details.enabled | -+---------------------------------------+ -| true | -+---------------------------------------+ -(1 row) -~~~ - -## See also - -- [`SET` (session variable)](set-vars.html) -- [`SHOW CLUSTER SETTING`](show-cluster-setting.html) -- [Cluster settings](cluster-settings.html) diff --git a/src/current/v21.1/set-locality.md b/src/current/v21.1/set-locality.md deleted file mode 100644 index ed8f2fc7fe2..00000000000 --- a/src/current/v21.1/set-locality.md +++ /dev/null @@ -1,160 +0,0 @@ ---- -title: SET LOCALITY -summary: The SET LOCALITY statement changes the locality of a table. -toc: true ---- - -{% include_cached new-in.html version="v21.1" %} The `ALTER TABLE .. SET LOCALITY` [statement](sql-statements.html) changes the [table locality](multiregion-overview.html#table-locality) of a [table](create-table.html) in a [multi-region database](multiregion-overview.html). - -While CockroachDB is processing an `ALTER TABLE .. SET LOCALITY` statement that enables or disables `REGIONAL BY ROW` on a table within a database, any [`ADD REGION`](add-region.html) and [`DROP REGION`](drop-region.html) statements on that database will fail. - -{{site.data.alerts.callout_info}} -`SET LOCALITY` is a subcommand of [`ALTER TABLE`](alter-table.html). -{{site.data.alerts.end}} - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/alter_table_locality.html %} -
    - -## Parameters - -| Parameter | Description | -|--------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `table_name` | The table whose [locality](multiregion-overview.html#table-locality) you are configuring. | -| `locality` | The [locality](multiregion-overview.html#table-locality) to apply to this table. Allowed values:
    • [`REGIONAL BY TABLE`](#regional-by-table) (default)
    • [`REGIONAL BY ROW`](#regional-by-row)
    • [`GLOBAL`](#global)
    | - -{{site.data.alerts.callout_info}} -For more information about which table locality is right for your use case, see the following pages: -
      -
    • [Multi-region table localities](multiregion-overview.html#table-locality)
      • -
    -{{site.data.alerts.end}} - -## Required privileges - -The user must be a member of the [`admin`](authorization.html#roles) or [owner](authorization.html#object-ownership) roles, or have the [`CREATE` privilege](authorization.html#supported-privileges) on the table. - -## Examples - - - -### Set the table locality to `REGIONAL BY TABLE` - -To optimize read and write access to the data in a table from the primary region, use the following statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER TABLE {table} SET LOCALITY REGIONAL BY TABLE IN PRIMARY REGION; -~~~ - -To optimize read and write access to the data in a table from the `us-east-1` region, use the following statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER TABLE {table} SET LOCALITY REGIONAL BY TABLE IN "us-east-1"; -~~~ - -{{site.data.alerts.callout_info}} -If no region is supplied, `REGIONAL BY TABLE` defaults to the primary region. -{{site.data.alerts.end}} - -For more information about how table localities work, see [Regional tables](multiregion-overview.html#regional-tables). - - - -### Set the table locality to `REGIONAL BY ROW` - -{{site.data.alerts.callout_info}} -Before setting the locality to `REGIONAL BY ROW` on a table targeted by a changefeed, read the considerations in [Changefeeds on regional by row tables](stream-data-out-of-cockroachdb-using-changefeeds.html#changefeeds-on-regional-by-row-tables). -{{site.data.alerts.end}} - -To make an existing table a _regional by row_ table, use the following statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER TABLE {table} SET LOCALITY REGIONAL BY ROW; -~~~ - - - -Every row in a regional by row table has a hidden `crdb_region` column that represents the row's home region. To see a row's region, issue a statement like the following: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT crdb_region, id FROM {table}; -~~~ - - To update an existing row's home region, use an [`UPDATE`](update.html) statement like the following: - -{% include_cached copy-clipboard.html %} -~~~ sql -UPDATE {table} SET crdb_region = 'eu-west' WHERE id IN (...) -~~~ - -To add a new row to a regional by row table, you must choose one of the following options. - -- Let CockroachDB set the row's home region automatically. It will use the region of the [gateway node](architecture/life-of-a-distributed-transaction.html#gateway) from which the row is inserted. - -- Set the home region explicitly using an [`INSERT`](insert.html) statement like the following: - - {% include_cached copy-clipboard.html %} - ~~~ sql - INSERT INTO {table} (crdb_region, ...) VALUES ('us-east-1', ...); - ~~~ - -This is necessary because every row in a regional by row table must have a home region. - -If you do not set a home region for a row in a regional by row table, it defaults to the value returned by the built-in function `gateway_region()`. If the value returned by `gateway_region()` does not belong to the multi-region database the table is a part of, the home region defaults to the database's primary region. - -For more information about how this table locality works, see [Regional by row tables](multiregion-overview.html#regional-by-row-tables). - - - -Note that you can use a name other than `crdb_region` for the hidden column by using the following statements: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER TABLE foo SET LOCALITY REGIONAL BY ROW AS bar; -SELECT bar, id FROM foo; -INSERT INTO foo (bar, ...) VALUES ('us-east-1', ...); -~~~ - -In fact, you can specify any column definition you like for the `REGIONAL BY ROW AS` column, as long as the column is of type `crdb_internal_region` and is not nullable. For example, you could modify the [movr schema](movr.html#the-movr-database) to have a region column generated as: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER TABLE rides ADD COLUMN region crdb_internal_region AS ( - CASE - WHEN city IN ('new york', 'boston', 'washington dc', 'chicago', 'detroit', 'minneapolis') THEN 'us-east-1' - WHEN city IN ('san francisco', 'seattle', 'los angeles') THEN 'us-west-1' - WHEN city IN ('amsterdam', 'paris', 'rome') THEN 'eu-west-1' - END -) STORED; -~~~ - -{% include {{page.version.version}}/sql/locality-optimized-search.md %} - - - -### Set the table locality to `GLOBAL` - -To optimize read access to the data in a table from any region (that is, globally), use the following statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER TABLE {table} SET LOCALITY GLOBAL; -~~~ - -~~~ -ALTER TABLE SET LOCALITY -~~~ - -For more information about how this table locality works, see [Global tables](multiregion-overview.html#global-tables). - -## See also - -- [Multi-region overview](multiregion-overview.html) -- [`ALTER TABLE`](alter-table.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/set-primary-region.md b/src/current/v21.1/set-primary-region.md deleted file mode 100644 index b071b57204f..00000000000 --- a/src/current/v21.1/set-primary-region.md +++ /dev/null @@ -1,235 +0,0 @@ ---- -title: SET PRIMARY REGION -summary: The SET PRIMARY REGION statement sets the primary region of a multi-region database. -toc: true ---- - -{% include_cached new-in.html version="v21.1" %} The `ALTER DATABASE .. SET PRIMARY REGION` [statement](sql-statements.html) sets the primary [region](multiregion-overview.html#database-regions) of a [multi-region database](multiregion-overview.html). - -{% include enterprise-feature.md %} - -{{site.data.alerts.callout_info}} -`SET PRIMARY REGION` is a subcommand of [`ALTER DATABASE`](alter-database.html). -{{site.data.alerts.end}} - -{{site.data.alerts.callout_danger}} -If a database's [zone configuration](configure-replication-zones.html) has been directly set with an [`ALTER DATABASE ... CONFIGURE ZONE`](configure-zone.html) statement, CockroachDB will block all `ALTER DATABASE ... SET PRIMARY REGION` statements on the database. - -To remove existing, manually-configured zones from a database (and unblock `SET PRIMARY REGION` statements on the database), use an [`ALTER DATABASE ... CONFIGURE ZONE DISCARD`](configure-zone.html#remove-a-replication-zone) statement. -{{site.data.alerts.end}} - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/alter_database_primary_region.html %} -
    - -## Parameters - -| Parameter | Description | -|-----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `database_name` | The database whose primary region to set. | -| `region_name` | The region to set as the database's primary region.
    Allowed values include any region present in `SHOW REGIONS FROM CLUSTER`. | - -## Required privileges - -To add a primary region to a database with no existing regions, the user must have one of the following: - -- Membership to the [`admin`](authorization.html#roles) role for the cluster. -- Membership to the [owner](authorization.html#object-ownership) role, or the [`CREATE` privilege](authorization.html#supported-privileges), for the database and all tables in the database. - -To switch primary regions to a region that has already been added to a database, the user must have membership to the [owner](authorization.html#object-ownership) role for the database, or have the [`CREATE` privilege](authorization.html#supported-privileges) on the database. - -## Examples - -{% include {{page.version.version}}/sql/multiregion-example-setup.md %} - -### Set the primary region - -Suppose you have a database `foo` in your cluster, and you want to make it a multi-region database. - -To add the first region to the database, or to set an already-added region as the primary region, use a `SET PRIMARY REGION` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE foo SET PRIMARY REGION "us-east1"; -~~~ - -~~~ -ALTER DATABASE PRIMARY REGION -~~~ - -Given a cluster with multiple regions, any databases in that cluster that have not yet had their primary regions set will have their replicas spread as broadly as possible for resiliency. When a primary region is added to one of these databases: - -- All tables will be [`REGIONAL BY TABLE`](set-locality.html#regional-by-table) in the primary region by default. -- This means that all such tables will have all of their voting replicas and leaseholders moved to the primary region. This process is known as [rebalancing](architecture/replication-layer.html#leaseholder-rebalancing). - -### Add more regions to the database - -To add more regions to the database, use an [`ADD REGION`](add-region.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER database foo ADD region "europe-west1"; -~~~ - -~~~ -ALTER DATABASE ADD REGION -~~~ - -To view the database's regions, and to see which region is the primary region, use a [`SHOW REGIONS FROM DATABASE`](show-regions.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW REGIONS FROM DATABASE foo; -~~~ - -~~~ - database | region | primary | zones ------------+--------------+---------+---------- - foo | us-east1 | true | {b,c,d} - foo | europe-west1 | false | {b,c,d} -(2 rows) -~~~ - -### Change an existing primary region - -To change the primary region to another region in the database, use a `SET PRIMARY REGION` statement. - -You can only change an existing primary region to a region that has already been added to the database. If you try to change the primary region to a region that is not already associated with a database, CockroachDB will return an error: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE foo SET PRIMARY REGION "us-west1"; -~~~ - -~~~ -ERROR: region "us-west1" has not been added to the database -SQLSTATE: 42602 -HINT: you must add the region to the database before setting it as primary region, using ALTER DATABASE foo ADD REGION "us-west1" -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER database foo ADD region "us-west1"; -~~~ - -~~~ -ALTER DATABASE ADD REGION -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE foo SET PRIMARY REGION "us-west1"; -~~~ - -~~~ -ALTER DATABASE PRIMARY REGION -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW REGIONS FROM DATABASE foo; -~~~ - -~~~ - database | region | primary | zones ------------+--------------+---------+---------- - foo | us-west1 | true | {a,b,c} - foo | europe-west1 | false | {b,c,d} - foo | us-east1 | false | {b,c,d} -(3 rows) -~~~ - -### Drop a region from a database - -To drop a region from a multi-region database, use a [`DROP REGION`](drop-region.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE foo DROP REGION "us-west1"; -~~~ - -~~~ -ALTER DATABASE DROP REGION -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW REGIONS FROM DATABASE foo; -~~~ - -~~~ - database | region | primary | zones ------------+--------------+---------+---------- - foo | us-east1 | true | {b,c,d} - foo | europe-west1 | false | {b,c,d} -(2 rows) -~~~ - -{{site.data.alerts.callout_info}} -You can only drop the primary region from a multi-region database if it's the last remaining region. -{{site.data.alerts.end}} - -If you try to drop the primary region when there is more than one region, CockroachDB will return an error: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE foo DROP REGION "us-east1"; -~~~ - -~~~ -ERROR: cannot drop region "us-east1" -SQLSTATE: 42P12 -HINT: You must designate another region as the primary region using ALTER DATABASE foo PRIMARY REGION or remove all other regions before attempting to drop region "us-east1" -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE foo DROP REGION "europe-west1"; -~~~ - -~~~ -ALTER DATABASE DROP REGION -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW REGIONS FROM DATABASE foo; -~~~ - -~~~ - database | region | primary | zones ------------+----------+---------+---------- - foo | us-east1 | true | {b,c,d} -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE foo DROP REGION "us-east1"; -~~~ - -~~~ -ALTER DATABASE DROP REGION -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW REGIONS FROM DATABASE foo; -~~~ - -~~~ - database | region | primary | zones ------------+--------+---------+-------- -(0 rows) -~~~ - -## See also - -- [Multi-region overview](multiregion-overview.html) -- [`ADD REGION`](add-region.html) -- [`DROP REGION`](drop-region.html) -- [`SHOW REGIONS`](show-regions.html) -- [`ALTER TABLE`](alter-table.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/set-schema.md b/src/current/v21.1/set-schema.md deleted file mode 100644 index cdca8493fc0..00000000000 --- a/src/current/v21.1/set-schema.md +++ /dev/null @@ -1,97 +0,0 @@ ---- -title: SET SCHEMA -summary: The SET SCHEMA statement changes the schema of a table. -toc: true ---- - - The `SET SCHEMA` [statement](sql-statements.html) changes the [schema](sql-name-resolution.html) of a [table](create-table.html). - -{{site.data.alerts.callout_info}} -`SET SCHEMA` is a subcommand of [`ALTER TABLE`](alter-table.html). - -CockroachDB also supports `SET SCHEMA` as an [alias for setting the `search_path` session variable](set-vars.html#supported-variables). -{{site.data.alerts.end}} - -## Required privileges - -The user must have the `DROP` [privilege](authorization.html#assign-privileges) on the table, and the `CREATE` privilege on the schema. - -## Syntax - -### Tables - -~~~ -ALTER TABLE [IF EXISTS] SET SCHEMA -~~~ - -## Parameters - - Parameter | Description ------------|------------- - `name` | The name of the table to alter. - `newschemaname` | The name of the table's new schema. - -## Examples - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### Change the schema of a table - -Suppose you want to add the `promo_codes` table to a new schema called `cockroach_labs`. - -By default, [unqualified tables](sql-name-resolution.html#lookup-with-unqualified-names) created in the database belong to the `public` schema: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES; -~~~ - -~~~ - schema_name | table_name | type | estimated_row_count ---------------+----------------------------+-------+---------------------- - public | promo_codes | table | 1000 - public | rides | table | 500 - public | user_promo_codes | table | 0 - public | users | table | 50 - public | vehicle_location_histories | table | 1000 - public | vehicles | table | 15 -(6 rows) -~~~ - -If the new schema does not already exist, [create it](create-schema.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE SCHEMA IF NOT EXISTS cockroach_labs; -~~~ - -Then, change the table's schema: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE promo_codes SET SCHEMA cockroach_labs; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES; -~~~ - -~~~ - schema_name | table_name | type | estimated_row_count ------------------+----------------------------+-------+---------------------- - cockroach_labs | promo_codes | table | 1000 - public | rides | table | 500 - public | user_promo_codes | table | 0 - public | users | table | 50 - public | vehicle_location_histories | table | 1000 - public | vehicles | table | 15 -(6 rows) -~~~ - -## See also - -- [`ALTER TABLE`](alter-table.html) -- [`CREATE SCHEMA`](drop-table.html) -- [`SHOW SCHEMAS`](show-schemas.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/set-transaction.md b/src/current/v21.1/set-transaction.md deleted file mode 100644 index d1a85c4dd31..00000000000 --- a/src/current/v21.1/set-transaction.md +++ /dev/null @@ -1,105 +0,0 @@ ---- -title: SET TRANSACTION -summary: The SET TRANSACTION statement sets the transaction priority for the current session or an individual transaction. -toc: true ---- - -The `SET TRANSACTION` [statement](sql-statements.html) sets the transaction priority, access mode, and "as of" timestamp after you [`BEGIN`](begin-transaction.html) it but before executing the first statement that manipulates a database. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/set_transaction.html %} -
    - -## Required privileges - -No [privileges](authorization.html#assign-privileges) are required to set the transaction priority. However, privileges are required for each statement within a transaction. - -## Parameters - -Parameter | Description -----------|------------ -`PRIORITY` | If you do not want the transaction to run with `NORMAL` priority, you can set it to `LOW` or `HIGH`. Transactions with higher priority are less likely to need to be retried. For more information, see [Transactions: Priorities](transactions.html#transaction-priorities).

    The current priority is also exposed as the read-only [session variable](show-vars.html) `transaction_priority`.

    **Default**: `NORMAL` -`READ` | Set the transaction access mode to `READ ONLY` or `READ WRITE`. The current transaction access mode is also exposed as the [session variable](show-vars.html) `transaction_read_only`.

    **Default**: `READ WRITE` -`AS OF SYSTEM TIME` | Execute the transaction using the database contents "as of" a specified time in the past.

    The `AS OF SYSTEM TIME` clause can be used only when the transaction is read-only. If the transaction contains any writes, or if the `READ WRITE` mode is specified, an error will be returned.

    For more information, see [AS OF SYSTEM TIME](as-of-system-time.html). -`NOT DEFERRABLE`
    `DEFERRABLE` | This clause is supported for compatibility with PostgreSQL. `NOT DEFERRABLE` is a no-op and the default behavior for CockroachDB. `DEFERRABLE` returns an `unimplemented` error. - -CockroachDB now only supports `SERIALIZABLE` isolation, so transactions can no longer be meaningfully set to any other `ISOLATION LEVEL`. In previous versions of CockroachDB, you could set transactions to `SNAPSHOT` isolation, but that feature has been removed. - -## Examples - -### Set priority - -{{site.data.alerts.callout_danger}}This example assumes you're using client-side intervention to handle transaction retries.{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET TRANSACTION PRIORITY HIGH; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SAVEPOINT cockroach_restart; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE products SET inventory = 0 WHERE sku = '8675309'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO orders (customer, sku, status) VALUES (1001, '8675309', 'new'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> RELEASE SAVEPOINT cockroach_restart; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> COMMIT; -~~~ - -### Use the `AS OF SYSTEM TIME` option - -You can execute the transaction using the database contents "as of" a specified time in the past. - -{% include {{ page.version.version }}/sql/set-transaction-as-of-system-time-example.md %} - -### Set the default transaction priority for a session - - To set the default transaction priority for all transactions in a session, use the `default_transaction_priority` [session variable](set-vars.html). For example: - -~~~ sql -> SET default_transaction_priority 'high'; -~~~ - -~~~ sql -> SHOW transaction_priority; -~~~ - -~~~ - transaction_priority ------------------------- - high -~~~ - -Note that `transaction_priority` is a read-only [session variable](show-vars.html) that cannot be set directly. - -## See also - -- [`SET`](set-vars.html) -- [Transactions: Priority levels](transactions.html#transaction-priorities) -- [`BEGIN`](begin-transaction.html) -- [`COMMIT`](commit-transaction.html) -- [`SAVEPOINT`](savepoint.html) -- [`RELEASE SAVEPOINT`](release-savepoint.html) -- [`ROLLBACK`](rollback-transaction.html) diff --git a/src/current/v21.1/set-vars.md b/src/current/v21.1/set-vars.md deleted file mode 100644 index 991bcfb77f6..00000000000 --- a/src/current/v21.1/set-vars.md +++ /dev/null @@ -1,242 +0,0 @@ ---- -title: SET (session variable) -summary: The SET statement modifies the current configuration variables for the client session. -toc: true ---- - -The `SET` [statement](sql-statements.html) can modify one of the session configuration variables. These can also be queried via [`SHOW`](show-vars.html). - -{{site.data.alerts.callout_danger}}In some cases, client drivers can drop and restart the connection to the server. When this happens, any session configurations made with SET statements are lost. It is therefore more reliable to configure the session in the client's connection string. For examples in different languages, see the Build an App with CockroachDB tutorials.{{site.data.alerts.end}} - - -## Required privileges - -No [privileges](authorization.html#assign-privileges) are required to modify the session settings. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/set_var.html %} -
    - -{{site.data.alerts.callout_info}}The SET statement for session settings is unrelated to the other SET TRANSACTION and SET CLUSTER SETTING statements.{{site.data.alerts.end}} - -## Parameters - -The `SET ` statement accepts two parameters: the -variable name and the value to use to modify the variable. - -The variable name is case insensitive. The value can be a list of one or more items. For example, the variable `search_path` is multi-valued. - -### Supported variables - -{% include {{ page.version.version }}/misc/session-vars.html %} - -Special syntax cases: - - Syntax | Equivalent to | Notes ---------|---------------|------- - `USE ...` | `SET database = ...` | This is provided as convenience for users with a MySQL/MSSQL background. - `SET NAMES ...` | `SET client_encoding = ...` | This is provided for compatibility with PostgreSQL clients. - `SET SCHEMA ` | `SET search_path = ` | This is provided for better compatibility with PostgreSQL. - `SET SESSION CHARACTERISTICS AS TRANSACTION ISOLATION LEVEL ...` | `SET default_transaction_isolation = ...` | This is provided for compatibility with standard SQL. - `SET TIME ZONE ...` | `SET timezone = ...` | This is provided for compatibility with PostgreSQL clients. - -## Examples - -### Set simple variables - -The following demonstrates how `SET` can be used to configure the -default database for the current session: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET database = movr; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW database; -~~~ - -~~~ - database -+----------+ - movr -(1 row) -~~~ - -### Set variables to values containing spaces - -The following demonstrates how to use quoting to use values containing spaces: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET database = "database name with spaces"; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW database; -~~~ - -~~~ - database -+----------+ - database name with spaces -(1 row) -~~~ - -### Set variables to a list of values - -The following demonstrates how to assign a list of values: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET search_path = pg_catalog,public; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW search_path; -~~~ - -~~~ - search_path -+--------------------+ - pg_catalog, public -(1 row) -~~~ - -### Reset a variable to its default value - -{{site.data.alerts.callout_success}}You can use RESET to reset a session variable as well.{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW search_path; -~~~ - -~~~ - search_path -+-------------+ - public -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET search_path = 'app'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW search_path; -~~~ - -~~~ - search_path -+-------------+ - app -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET search_path = DEFAULT; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW search_path; -~~~ - -~~~ - search_path -+-------------+ - public -(1 row) -~~~ - -## `SET TIME ZONE` - -{{site.data.alerts.callout_danger}} -As a best practice, we recommend not using this setting and avoid setting a session time for your database. We instead recommend converting UTC values to the appropriate time zone on the client side. -{{site.data.alerts.end}} - -You can control the default time zone for a session with `SET TIME ZONE`. This will apply an offset to all [`TIMESTAMPTZ`/`TIMESTAMP WITH TIME ZONE`](timestamp.html) and [`TIMETZ`/`TIME WITH TIME ZONE`](time.html) values in the session. By default, CockroachDB uses UTC as the time zone for `SET TIME ZONE` offsets. - -### Parameters - -The input passed to `SET TIME ZONE` indicates the time zone for the current session. This value can be a string representation of a local system-defined time zone (e.g., `'EST'`, `'America/New_York'`) or a positive or negative numeric offset from UTC (e.g., `-7`, `+7`, or `UTC-7`, `UTC+7`) or GMT (e.g., `GMT-7`, `GMT+7`). The numeric offset input can also be colon-delimited (e.g., `-7:00`, `GMT+7:00`). - -When setting a time zone, note the following: - -- Timezone abbreviations are case-insensitive. - -- Timezone abbreviations must be part of the [tz database](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones), as recognized by the [`tzdata` Golang package](https://golang.org/pkg/time/tzdata/). - -- `DEFAULT`, `LOCAL`, or `0` sets the session time zone to `UTC`. - -- Only offsets specified by integers (e.g., `-7`, `7`) use the [ISO 8601 time offset](https://en.wikipedia.org/wiki/ISO_8601#Time_offsets_from_UTC) (i.e., the offset input is parsed as hours *east* of UTC). If you explicitly specify `UTC` or `GMT` for the time zone offset (e.g., `UTC-7`,`GMT+7`), or if the numeric input is colon-delimited (e.g., `-7:00`, `GMT+7:00`), CockroachDB uses the [POSIX time offset](https://www.postgresql.org/docs/current/datetime-posix-timezone-specs.html) instead (i.e., hours *west* of the specified time zone). This means that specifying an offset of `-7` (i.e., -7 *east* of UTC) is equivalent to specifying `GMT+7` (i.e., 7 *west* of UTC). - -### Example: Set the default time zone via `SET TIME ZONE` - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET TIME ZONE 'EST'; -- same as SET "timezone" = 'EST' -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TIME ZONE; -~~~ - -~~~ - timezone -+----------+ - EST -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET TIME ZONE DEFAULT; -- same as SET "timezone" = DEFAULT -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TIME ZONE; -~~~ - -~~~ - timezone -+----------+ - UTC -(1 row) -~~~ - -## `SET TRACING` - -`SET TRACING` changes the trace recording state of the current session. A trace recording can be inspected with the [`SHOW TRACE FOR SESSION`](show-trace.html) statement. - - Value | Description --------|------------ -`off` | Trace recording is disabled. -`cluster` | Trace recording is enabled; distributed traces are collected. -`on` | Same as `cluster`. -`kv` | Same as `cluster` except that "kv messages" are collected instead of regular trace messages. See [`SHOW TRACE FOR SESSION`](show-trace.html). -`results` | Result rows and row counts are copied to the session trace. This must be specified in order for the output of a query to be printed in the session trace.

    Example: `SET tracing = kv, results;` - -## Known Limitations - -{% include {{page.version.version}}/known-limitations/set-transaction-no-rollback.md %} - -## See also - -- [`RESET`](reset-vars.html) -- [`SET TRANSACTION`](set-transaction.html) -- [`SET CLUSTER SETTING`](set-cluster-setting.html) -- [`SHOW` (session variable)](show-vars.html) -- [The `TIMESTAMP` and `TIMESTAMPTZ` data types.](timestamp.html) -- [`SHOW TRACE FOR SESSION`](show-trace.html) diff --git a/src/current/v21.1/show-backup.md b/src/current/v21.1/show-backup.md deleted file mode 100644 index 9f16c0a8eb7..00000000000 --- a/src/current/v21.1/show-backup.md +++ /dev/null @@ -1,378 +0,0 @@ ---- -title: SHOW BACKUP -summary: The SHOW BACKUP statement lists the contents of a backup. -toc: true ---- - -The `SHOW BACKUP` [statement](sql-statements.html) lists the contents of a backup created with the [`BACKUP`](backup.html) statement. - -## Required privileges - -`SHOW BACKUP` requires read permissions to its target destination. - -{% include {{ page.version.version }}/misc/non-http-source-privileges.md %} - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_backup.html %} -
    - -## Parameters - -Parameter | Description -----------|------------ -`SHOW BACKUPS IN location` | List the backup paths in the given [`location`](backup.html#backup-file-urls). [See the example](#view-a-list-of-the-available-full-backup-subdirectories). -`SHOW BACKUP location` | Show the details of the backup in the given [`location`](backup.html#backup-file-urls). [See the example](#show-a-backup). -`SHOW BACKUP LATEST IN location` | Show the most recent backup added in the given [`location`](backup.html#backup-file-urls). [See the example](#show-the-most-recent-backup). -`SHOW BACKUP SCHEMAS location` | Show the schema details of the backup in the given [`location`](backup.html#backup-file-urls). [See the example](#show-a-backup-with-schemas). -`SHOW BACKUP subdirectory IN location` | List the full and incremental backups that are stored in the given full backup's `subdirectory` within a [`location`](backup.html#backup-file-urls). [See the example](#show-details-for-scheduled-backups). -`kv_option_list` | Control the behavior of `SHOW BACKUP` with a comma-separated list of [these options](#options). - -### Options - -Option | Value | Description --------------+-------+----------------------------------------------------- -`privileges` | N/A | List which users and roles had which privileges on each table in the backup. Displays original ownership of the backup. -`encryption_passphrase` | [`STRING`](string.html) | The passphrase used to [encrypt the files](take-and-restore-encrypted-backups.html) that the `BACKUP` statement generates (the data files and its manifest, containing the backup's metadata). -`as_json` | N/A | [Display the backup's internal metadata](#show-a-backups-internal-metadata) as JSON in the response. - -## Response - -The following fields are returned: - -Field | Description -------|------------ -`database_name` | The database name. -`parent_schema_name` | The name of the parent schema. -`object_name` | The name of the [database](create-database.html), [table](create-table.html), [type](create-type.html), or schema. -`object_type` | The type of object: [database](create-database.html), [table](create-table.html), [type](create-type.html), or schema. -`start_time` | The time of the earliest data encapsulated in the backup. Note that this only displays for incremental backups. For a full backup, this is `NULL`. -`end_time` | The time to which data can be restored. This is equivalent to the [`AS OF SYSTEM TIME`](as-of-system-time.html) of the backup. If the backup was _not_ taken with [revision history](take-backups-with-revision-history-and-restore-from-a-point-in-time.html), the `end_time` is the _only_ time the data can be restored to. If the backup was taken with revision history, the `end_time` is the latest time the data can be restored to. -`size_bytes` | The size of the backup, in bytes. -`create_statement` | The `CREATE` statement used to create [table(s)](create-table.html), [view(s)](create-view.html), or [sequence(s)](create-sequence.html) that are stored within the backup. This displays when `SHOW BACKUP SCHEMAS` is used. Note that tables with references to [foreign keys](foreign-key.html) will only display foreign key constraints if the table to which the constraint relates to is also included in the backup. -`is_full_cluster` | Whether the backup is of a full cluster or not. -`path` | The list of the [full backup](take-full-and-incremental-backups.html#full-backups)'s subdirectories. This field is returned for `SHOW BACKUPS IN location` only. The path format is `//-`. - -## Example - -### Show a backup - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW BACKUP 's3://test/backup-test?AWS_ACCESS_KEY_ID=[placeholder]&AWS_SECRET_ACCESS_KEY=[placeholder]'; -~~~ - -~~~ - database_name | parent_schema_name | object_name | object_type | start_time | end_time | size_bytes | rows | is_full_cluster -----------------+--------------------+----------------------------+-------------+------------+----------------------------------+------------+------+------------------ - NULL | NULL | system | database | NULL | 2020-09-24 19:05:40.542168+00:00 | NULL | NULL | true - system | public | users | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 144 | 3 | true - system | public | zones | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 201 | 7 | true - system | public | settings | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 431 | 6 | true - system | public | ui | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 0 | 0 | true - system | public | jobs | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 434302 | 62 | true - system | public | locations | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 261 | 5 | true - system | public | role_members | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 184 | 2 | true - system | public | comments | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 0 | 0 | true - system | public | scheduled_jobs | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 875 | 2 | true - NULL | NULL | defaultdb | database | NULL | 2020-09-24 19:05:40.542168+00:00 | NULL | NULL | true - NULL | NULL | postgres | database | NULL | 2020-09-24 19:05:40.542168+00:00 | NULL | NULL | true - NULL | NULL | movr | database | NULL | 2020-09-24 19:05:40.542168+00:00 | NULL | NULL | true - movr | public | users | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 4911 | 50 | true - movr | public | vehicles | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 3182 | 15 | true - movr | public | rides | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 156387 | 500 | true - movr | public | vehicle_location_histories | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 73918 | 1000 | true - movr | public | promo_codes | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 216083 | 1000 | true - movr | public | user_promo_codes | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 0 | 0 | true - defaultdb | NULL | org_one | schema | NULL | 2020-09-24 19:05:40.542168+00:00 | NULL | NULL | true -(20 rows) -~~~ - -You will receive an error if there is a collection of backups in the storage location that you pass to `SHOW BACKUP`. It is necessary to run `SHOW BACKUP` with the specific backup directory rather than the backup collection's top-level directory. Use [`SHOW BACKUPS IN`](#show-backups-in) with your storage location to list the backup directories it contains, which can then be run with `SHOW BACKUP` to inspect the metadata. - -### View a list of the available full backup subdirectories - -To view a list of the available [full backups](take-full-and-incremental-backups.html#full-backups) subdirectories, use the following command: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW BACKUPS IN 's3://test/backup-test?AWS_ACCESS_KEY_ID=[placeholder]&AWS_SECRET_ACCESS_KEY=[placeholder]'; -~~~ - -~~~ - path ------------------------- - 2020/09/24-204152.88 - 2020/09/24-204623.44 - 2020/09/24-205612.40 - 2020/09/24-207328.36 -(4 rows) -~~~ - -The path format is `//-`. - -### Show the most recent backup - -{% include_cached new-in.html version="v21.1" %} To view the most recent backup, use the `LATEST` syntax: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW BACKUP LATEST IN 's3://{bucket name}?AWS_ACCESS_KEY_ID=[placeholder]&AWS_SECRET_ACCESS_KEY=[placeholder]'; -~~~ - -~~~ -database_name | parent_schema_name | object_name | object_type | backup_type | start_time | end_time | size_bytes | rows | is_full_cluster ---------------+--------------------+-------------+-------------+-------------+------------+----------------------------+------------+------+------------------ -NULL | NULL | movr | database | full | NULL | 2022-03-25 16:53:48.001825 | NULL | NULL | false -movr | NULL | public | schema | full | NULL | 2022-03-25 16:53:48.001825 | NULL | NULL | false -movr | public | users | table | full | NULL | 2022-03-25 16:53:48.001825 | 135144 | 1474 | false -(3 rows) -~~~ - -### View a list of the full and incremental backups in a specific full backup subdirectory - -To view a list of the [full](take-full-and-incremental-backups.html#full-backups) and [incremental](take-full-and-incremental-backups.html#incremental-backups) backups in a specific subdirectory, use the following command: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW BACKUP '2020/09/24-204152.88' IN 's3://test/backup-test?AWS_ACCESS_KEY_ID=[placeholder]&AWS_SECRET_ACCESS_KEY=[placeholder]'; -~~~ - -~~~ - database_name | parent_schema_name | object_name | object_type | start_time | end_time | size_bytes | rows | is_full_cluster -----------------+--------------------+----------------------------+-------------+----------------------------------+----------------------------------+------------+------+------------------ - NULL | NULL | system | database | NULL | 2020-09-24 20:41:52.880553+00:00 | NULL | NULL | true - system | public | users | table | NULL | 2020-09-24 20:41:52.880553+00:00 | 144 | 3 | true - system | public | zones | table | NULL | 2020-09-24 20:41:52.880553+00:00 | 201 | 7 | true - system | public | settings | table | NULL | 2020-09-24 20:41:52.880553+00:00 | 875 | 6 | true - system | public | ui | table | NULL | 2020-09-24 20:41:52.880553+00:00 | 0 | 0 | true - system | public | jobs | table | NULL | 2020-09-24 20:41:52.880553+00:00 | 795117 | 80 | true - system | public | locations | table | NULL | 2020-09-24 20:41:52.880553+00:00 | 261 | 5 | true - system | public | role_members | table | NULL | 2020-09-24 20:41:52.880553+00:00 | 184 | 2 | true - system | public | comments | table | NULL | 2020-09-24 20:41:52.880553+00:00 | 0 | 0 | true - system | public | scheduled_jobs | table | NULL | 2020-09-24 20:41:52.880553+00:00 | 1013 | 2 | true - NULL | NULL | defaultdb | database | NULL | 2020-09-24 20:41:52.880553+00:00 | NULL | NULL | true - NULL | NULL | postgres | database | NULL | 2020-09-24 20:41:52.880553+00:00 | NULL | NULL | true - NULL | NULL | movr | database | NULL | 2020-09-24 20:41:52.880553+00:00 | NULL | NULL | true - movr | public | users | table | NULL | 2020-09-24 20:41:52.880553+00:00 | 4911 | 50 | true - movr | public | vehicles | table | NULL | 2020-09-24 20:41:52.880553+00:00 | 3182 | 15 | true - movr | public | rides | table | NULL | 2020-09-24 20:41:52.880553+00:00 | 156387 | 500 | true - movr | public | vehicle_location_histories | table | NULL | 2020-09-24 20:41:52.880553+00:00 | 73918 | 1000 | true - movr | public | promo_codes | table | NULL | 2020-09-24 20:41:52.880553+00:00 | 216083 | 1000 | true - movr | public | user_promo_codes | table | NULL | 2020-09-24 20:41:52.880553+00:00 | 0 | 0 | true - defaultdb | NULL | org_one | schema | NULL | 2020-09-24 20:41:52.880553+00:00 | NULL | NULL | true - NULL | NULL | system | database | 2020-09-24 20:41:52.880553+00:00 | 2020-09-24 20:50:00+00:00 | NULL | NULL | true - system | public | users | table | 2020-09-24 20:41:52.880553+00:00 | 2020-09-24 20:50:00+00:00 | 0 | 0 | true - system | public | zones | table | 2020-09-24 20:41:52.880553+00:00 | 2020-09-24 20:50:00+00:00 | 0 | 0 | true - system | public | settings | table | 2020-09-24 20:41:52.880553+00:00 | 2020-09-24 20:50:00+00:00 | 0 | 0 | true - system | public | ui | table | 2020-09-24 20:41:52.880553+00:00 | 2020-09-24 20:50:00+00:00 | 0 | 0 | true - system | public | jobs | table | 2020-09-24 20:41:52.880553+00:00 | 2020-09-24 20:50:00+00:00 | 102381 | 1 | true - system | public | locations | table | 2020-09-24 20:41:52.880553+00:00 | 2020-09-24 20:50:00+00:00 | 0 | 0 | true - system | public | role_members | table | 2020-09-24 20:41:52.880553+00:00 | 2020-09-24 20:50:00+00:00 | 0 | 0 | true - system | public | comments | table | 2020-09-24 20:41:52.880553+00:00 | 2020-09-24 20:50:00+00:00 | 0 | 0 | true - system | public | scheduled_jobs | table | 2020-09-24 20:41:52.880553+00:00 | 2020-09-24 20:50:00+00:00 | 1347 | 2 | true - NULL | NULL | defaultdb | database | 2020-09-24 20:41:52.880553+00:00 | 2020-09-24 20:50:00+00:00 | NULL | NULL | true - NULL | NULL | postgres | database | 2020-09-24 20:41:52.880553+00:00 | 2020-09-24 20:50:00+00:00 | NULL | NULL | true - NULL | NULL | movr | database | 2020-09-24 20:41:52.880553+00:00 | 2020-09-24 20:50:00+00:00 | NULL | NULL | true - movr | public | users | table | 2020-09-24 20:41:52.880553+00:00 | 2020-09-24 20:50:00+00:00 | 0 | 0 | true - movr | public | vehicles | table | 2020-09-24 20:41:52.880553+00:00 | 2020-09-24 20:50:00+00:00 | 0 | 0 | true - movr | public | rides | table | 2020-09-24 20:41:52.880553+00:00 | 2020-09-24 20:50:00+00:00 | 0 | 0 | true - movr | public | vehicle_location_histories | table | 2020-09-24 20:41:52.880553+00:00 | 2020-09-24 20:50:00+00:00 | 0 | 0 | true - movr | public | promo_codes | table | 2020-09-24 20:41:52.880553+00:00 | 2020-09-24 20:50:00+00:00 | 0 | 0 | true - movr | public | user_promo_codes | table | 2020-09-24 20:41:52.880553+00:00 | 2020-09-24 20:50:00+00:00 | 0 | 0 | true - defaultdb | NULL | org_one | schema | 2020-09-24 20:41:52.880553+00:00 | 2020-09-24 20:50:00+00:00 | NULL | NULL | true -(40 rows) - -~~~ - -### Show a backup with schemas - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW BACKUP SCHEMAS 's3://test/backup-test?AWS_ACCESS_KEY_ID=[placeholder]&AWS_SECRET_ACCESS_KEY=[placeholder]'; -~~~ - -~~~ - database_name | parent_schema_name | object_name | object_type | start_time | end_time | size_bytes | rows | is_full_cluster | create_statement -----------------+--------------------+----------------------------+-------------+------------+----------------------------------+------------+------+-----------------+---------------------------------------------------------------------------------------------------------------------------------- - NULL | NULL | system | database | NULL | 2020-09-24 19:05:40.542168+00:00 | NULL | NULL | true | NULL - system | public | users | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 144 | 3 | true | CREATE TABLE users ( - | | | | | | | | | username STRING NOT NULL, - | | | | | | | | | "hashedPassword" BYTES NULL, - | | | | | | | | | "isRole" BOOL NOT NULL DEFAULT false, - | | | | | | | | | CONSTRAINT "primary" PRIMARY KEY (username ASC), - | | | | | | | | | FAMILY "primary" (username), - | | | | | | | | | FAMILY "fam_2_hashedPassword" ("hashedPassword"), - | | | | | | | | | FAMILY "fam_3_isRole" ("isRole") - | | | | | | | | | ) -... - system | public | jobs | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 434302 | 62 | true | CREATE TABLE jobs ( - | | | | | | | | | id INT8 NOT NULL DEFAULT unique_rowid(), - | | | | | | | | | status STRING NOT NULL, - | | | | | | | | | created TIMESTAMP NOT NULL DEFAULT now():::TIMESTAMP, - | | | | | | | | | payload BYTES NOT NULL, - | | | | | | | | | progress BYTES NULL, - | | | | | | | | | created_by_type STRING NULL, - | | | | | | | | | created_by_id INT8 NULL, - | | | | | | | | | claim_session_id BYTES NULL, - | | | | | | | | | claim_instance_id INT8 NULL, - | | | | | | | | | CONSTRAINT "primary" PRIMARY KEY (id ASC), - | | | | | | | | | INDEX jobs_status_created_idx (status ASC, created ASC), - | | | | | | | | | INDEX jobs_created_by_type_created_by_id_idx (created_by_type ASC, created_by_id ASC) STORING (status), - | | | | | | | | | FAMILY fam_0_id_status_created_payload (id, status, created, payload, created_by_type, created_by_id), - | | | | | | | | | FAMILY progress (progress), - | | | | | | | | | FAMILY claim (claim_session_id, claim_instance_id) - | | | | | | | | | ) - system | public | locations | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 261 | 5 | true | CREATE TABLE locations ( - | | | | | | | | | "localityKey" STRING NOT NULL, - | | | | | | | | | "localityValue" STRING NOT NULL, - | | | | | | | | | latitude DECIMAL(18,15) NOT NULL, - | | | | | | | | | longitude DECIMAL(18,15) NOT NULL, - | | | | | | | | | CONSTRAINT "primary" PRIMARY KEY ("localityKey" ASC, "localityValue" ASC), - | | | | | | | | | FAMILY "fam_0_localityKey_localityValue_latitude_longitude" ("localityKey", "localityValue", latitude, longitude) - | | | | | | | | | ) -... -~~~ - -### Show a backup with privileges - -Use the `WITH privileges` [option](#options) to view a list of which users and roles had which privileges on each database and table in the backup. This parameter also displays the original owner of objects in the backup: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW BACKUP 's3://test/backup-test?AWS_ACCESS_KEY_ID=[placeholder]&AWS_SECRET_ACCESS_KEY=[placeholder]' WITH privileges; -~~~ - -~~~ - database_name | parent_schema_name | object_name | object_type | start_time | end_time | size_bytes | rows | is_full_cluster | privileges | owner -----------------+--------------------+----------------------------+-------------+------------+----------------------------------+------------+------+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------- - NULL | NULL | system | database | NULL | 2020-09-24 19:05:40.542168+00:00 | NULL | NULL | true | GRANT GRANT, SELECT ON system TO admin; GRANT GRANT, SELECT ON system TO root; | root - system | public | users | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 144 | 3 | true | GRANT DELETE, GRANT, INSERT, SELECT, UPDATE ON users TO admin; GRANT DELETE, GRANT, INSERT, SELECT, UPDATE ON users TO root; | root - system | public | zones | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 201 | 7 | true | GRANT DELETE, GRANT, INSERT, SELECT, UPDATE ON zones TO admin; GRANT DELETE, GRANT, INSERT, SELECT, UPDATE ON zones TO root; | root - system | public | settings | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 431 | 6 | true | GRANT DELETE, GRANT, INSERT, SELECT, UPDATE ON settings TO admin; GRANT DELETE, GRANT, INSERT, SELECT, UPDATE ON settings TO root; | root - system | public | ui | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 0 | 0 | true | GRANT DELETE, GRANT, INSERT, SELECT, UPDATE ON ui TO admin; GRANT DELETE, GRANT, INSERT, SELECT, UPDATE ON ui TO root; | root - system | public | jobs | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 434302 | 62 | true | GRANT DELETE, GRANT, INSERT, SELECT, UPDATE ON jobs TO admin; GRANT DELETE, GRANT, INSERT, SELECT, UPDATE ON jobs TO root; | root - system | public | locations | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 261 | 5 | true | GRANT DELETE, GRANT, INSERT, SELECT, UPDATE ON locations TO admin; GRANT DELETE, GRANT, INSERT, SELECT, UPDATE ON locations TO root; | root - system | public | role_members | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 184 | 2 | true | GRANT DELETE, GRANT, INSERT, SELECT, UPDATE ON role_members TO admin; GRANT DELETE, GRANT, INSERT, SELECT, UPDATE ON role_members TO root; | root - system | public | comments | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 0 | 0 | true | GRANT DELETE, GRANT, INSERT, SELECT, UPDATE ON comments TO admin; GRANT SELECT ON comments TO public; GRANT DELETE, GRANT, INSERT, SELECT, UPDATE ON comments TO root; | root - system | public | scheduled_jobs | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 875 | 2 | true | GRANT DELETE, GRANT, INSERT, SELECT, UPDATE ON scheduled_jobs TO admin; GRANT DELETE, GRANT, INSERT, SELECT, UPDATE ON scheduled_jobs TO root; | root - NULL | NULL | defaultdb | database | NULL | 2020-09-24 19:05:40.542168+00:00 | NULL | NULL | true | GRANT ALL ON defaultdb TO admin; GRANT CREATE ON defaultdb TO max; GRANT ALL ON defaultdb TO root; | root - NULL | NULL | postgres | database | NULL | 2020-09-24 19:05:40.542168+00:00 | NULL | NULL | true | GRANT ALL ON postgres TO admin; GRANT ALL ON postgres TO root; | root - NULL | NULL | movr | database | NULL | 2020-09-24 19:05:40.542168+00:00 | NULL | NULL | true | GRANT ALL ON movr TO admin; GRANT ALL ON movr TO root; | root - movr | public | users | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 4911 | 50 | true | GRANT ALL ON users TO admin; GRANT ALL ON users TO root; | root - movr | public | vehicles | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 3182 | 15 | true | GRANT ALL ON vehicles TO admin; GRANT ALL ON vehicles TO root; | root - movr | public | rides | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 156387 | 500 | true | GRANT ALL ON rides TO admin; GRANT ALL ON rides TO root; | root - movr | public | vehicle_location_histories | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 73918 | 1000 | true | GRANT ALL ON vehicle_location_histories TO admin; GRANT ALL ON vehicle_location_histories TO root; | root - movr | public | promo_codes | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 216083 | 1000 | true | GRANT ALL ON promo_codes TO admin; GRANT ALL ON promo_codes TO root; | root - movr | public | user_promo_codes | table | NULL | 2020-09-24 19:05:40.542168+00:00 | 0 | 0 | true | GRANT ALL ON user_promo_codes TO admin; GRANT ALL ON user_promo_codes TO root; | root - defaultdb | NULL | org_one | schema | NULL | 2020-09-24 19:05:40.542168+00:00 | NULL | NULL | true | | root -(20 rows) -~~~ - -### Show details for scheduled backups - - When a [backup is created by a schedule](create-schedule-for-backup.html), it is stored within a collection of backups in the given location. To view details for a backup created by a schedule, you can use the following: - -- `SHOW BACKUPS IN y` statement to [view a list of the available full backup subdirectories](#view-a-list-of-the-available-full-backup-subdirectories). -- `SHOW BACKUP x IN y` statement to [view a list of the full and incremental backups that are stored in a specific full backup's subdirectory](#view-a-list-of-the-full-and-incremental-backups-in-a-specific-full-backup-subdirectory). - -### Show an encrypted backup - -Depending on how the backup was [encrypted](take-and-restore-encrypted-backups.html), use the [`encryption_passphrase` option](backup.html#with-encryption-passphrase) and the same passphrase that was used to create the backup: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW BACKUP 's3://test/backup-test?AWS_ACCESS_KEY_ID=[placeholder]&AWS_SECRET_ACCESS_KEY=[placeholder]' - WITH encryption_passphrase = 'password123'; -~~~ - -Or, use the `kms` option and the same KMS URI that was used to create the backup: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW BACKUP 's3://test/backups/test_explicit_kms?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=123' - WITH kms = 'aws:///arn:aws:kms:us-east-1:123456789:key/1234-abcd-5678-efgh-90ij?AWS_ACCESS_KEY_ID=123456&AWS_SECRET_ACCESS_KEY=123456®ION=us-east-1'; -~~~ - -~~~ - database_name | parent_schema_name | object_name | object_type | start_time | end_time | size_bytes | rows | is_full_cluster -----------------+--------------------+----------------------------+-------------+------------+----------------------------------+------------+------+------------------ - NULL | NULL | system | database | NULL | 2020-09-29 18:24:55.784364+00:00 | NULL | NULL | true - system | public | users | table | NULL | 2020-09-29 18:24:55.784364+00:00 | 144 | 3 | true - system | public | zones | table | NULL | 2020-09-29 18:24:55.784364+00:00 | 201 | 7 | true - system | public | settings | table | NULL | 2020-09-29 18:24:55.784364+00:00 | 431 | 6 | true - system | public | ui | table | NULL | 2020-09-29 18:24:55.784364+00:00 | 0 | 0 | true - system | public | jobs | table | NULL | 2020-09-29 18:24:55.784364+00:00 | 962415 | 97 | true - system | public | locations | table | NULL | 2020-09-29 18:24:55.784364+00:00 | 261 | 5 | true - system | public | role_members | table | NULL | 2020-09-29 18:24:55.784364+00:00 | 184 | 2 | true - system | public | comments | table | NULL | 2020-09-29 18:24:55.784364+00:00 | 0 | 0 | true - system | public | scheduled_jobs | table | NULL | 2020-09-29 18:24:55.784364+00:00 | 1991 | 4 | true - NULL | NULL | defaultdb | database | NULL | 2020-09-29 18:24:55.784364+00:00 | NULL | NULL | true - NULL | NULL | postgres | database | NULL | 2020-09-29 18:24:55.784364+00:00 | NULL | NULL | true - NULL | NULL | movr | database | NULL | 2020-09-29 18:24:55.784364+00:00 | NULL | NULL | true - movr | public | users | table | NULL | 2020-09-29 18:24:55.784364+00:00 | 4911 | 50 | true - movr | public | vehicles | table | NULL | 2020-09-29 18:24:55.784364+00:00 | 3182 | 15 | true - movr | public | rides | table | NULL | 2020-09-29 18:24:55.784364+00:00 | 156387 | 500 | true - movr | public | vehicle_location_histories | table | NULL | 2020-09-29 18:24:55.784364+00:00 | 73918 | 1000 | true - movr | public | promo_codes | table | NULL | 2020-09-29 18:24:55.784364+00:00 | 216083 | 1000 | true - movr | public | user_promo_codes | table | NULL | 2020-09-29 18:24:55.784364+00:00 | 0 | 0 | true - defaultdb | NULL | org_one | schema | NULL | 2020-09-29 18:24:55.784364+00:00 | NULL | NULL | true -(20 rows) -~~~ - -### Show a backup's internal metadata - -Use the `WITH as_json` option to output a backup's internal metadata, contained in its manifest file, as a JSON value: - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW BACKUP '/2021/11/15-150703.21' IN 's3://test/backup-test?AWS_ACCESS_KEY_ID=[placeholder]&AWS_SECRET_ACCESS_KEY=[placeholder]' WITH as_json; -~~~ - -The response will include a `manifest` column with the file's contents as the JSON value. Use [JSONB functions](functions-and-operators.html#jsonb-functions) to query particular data or edit the format of the response. - -{{site.data.alerts.callout_info}} -The response returned from `SHOW BACKUP ... WITH as_json` is a backup's internal metadata. This content is subject to change from version to version of CockroachDB and does not offer the same stability guarantees as the other `SHOW BACKUP` [options](#options) and their [responses](#response). As a result, `as_json` should **only** be used for debugging or general inspection purposes. -{{site.data.alerts.end}} - -For example, to return a specific entry from the JSON response as a [`string`](string.html) indented and with newlines use the [`jsonb_pretty()`](functions-and-operators.html#jsonb-functions) function: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT jsonb_pretty(manifest->'entryCounts') AS f FROM [SHOW BACKUP '/2021/11/15-150703.21' IN 's3://test/backup-test?AWS_ACCESS_KEY_ID=[placeholder]&AWS_SECRET_ACCESS_KEY=[placeholder]' with as_json]; -~~~ - -~~~ json - { - "dataSize": "458371", - "indexEntries": "1015", - "rows": "2565" - } -~~~ - -To query for particular data, use the [`jsonb_array_elements()` function](functions-and-operators.html#jsonb-functions) to expand the desired elements from the JSON response. The following query returns the paths to each of the data files within the backup: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT f->>'path' FROM (SELECT jsonb_array_elements(manifest->'files') AS f FROM [SHOW BACKUP '/2021/11/15-150703.21' IN 's3://test/backup-test?AWS_ACCESS_KEY_ID=[placeholder]&AWS_SECRET_ACCESS_KEY=[placeholder]' WITH as_json]); -~~~ - -~~~ - ?column? -------------------------------- - data/710798326337404929.sst - data/710798326337404929.sst - data/710798328891998209.sst - data/710798326337404929.sst - data/710798326337404929.sst - data/710798328434982913.sst - data/710798328891998209.sst - data/710798326337404929.sst - data/710798326337404929.sst -~~~ - -## See also - -- [`BACKUP`](backup.html) -- [`RESTORE`](restore.html) diff --git a/src/current/v21.1/show-cluster-setting.md b/src/current/v21.1/show-cluster-setting.md deleted file mode 100644 index 7e945d96ba1..00000000000 --- a/src/current/v21.1/show-cluster-setting.md +++ /dev/null @@ -1,135 +0,0 @@ ---- -title: SHOW CLUSTER SETTING -summary: The SHOW CLUSTER SETTING statement displays the current cluster settings. -toc: true ---- - -The `SHOW CLUSTER SETTING` [statement](sql-statements.html) displays the values of [cluster settings](cluster-settings.html). - -To configure cluster settings, use [`SET CLUSTER SETTING`](set-cluster-setting.html). - -{{site.data.alerts.callout_info}} -The `SHOW` statement for cluster settings is unrelated to the other `SHOW` statements: SHOW (session variable), SHOW CREATE, SHOW USERS, SHOW DATABASES, SHOW COLUMNS, SHOW GRANTS, and SHOW CONSTRAINTS. -{{site.data.alerts.end}} - -## Details - -- To display the value of a specific cluster setting, use the following syntax: - - ~~~ sql - SHOW CLUSTER SETTING ; - ~~~ - -- To display the values of *public* cluster settings (i.e., cluster settings that are documented and for which tuning is supported), use one of the following: - - ~~~ sql - SHOW CLUSTER SETTINGS; - ~~~ - ~~~ sql - SHOW PUBLIC CLUSTER SETTINGS; - ~~~ - -- To display the values of all cluster settings use one of the following: - - ~~~ sql - SHOW ALL CLUSTER SETTINGS; - ~~~ - ~~~ sql - SHOW CLUSTER SETTING ALL; - ~~~ - - When you display all cluster settings, the table output includes a `public` column that denotes whether a setting is public or not. - -## Required privileges - -Only members of the `admin` role can display cluster settings. By default, the `root` user belongs to the `admin` role. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_cluster_setting.html %} -
    - -## Parameters - -Parameter | Description -----------|------------ -`var_name` | The name of the [cluster setting](cluster-settings.html) (case-insensitive). -`ALL` | Display all cluster settings. -`PUBLIC` | Display only the public cluster settings.
    By default, only public settings are listed by `SHOW CLUSTER SETTINGS`. `SHOW PUBLIC CLUSTER SETTINGS` and `SHOW CLUSTER SETTINGS` are equivalent. - -## Response - -When you query multiple cluster settings (e.g., with `SHOW CLUSTER SETTINGS`, or with `SHOW ALL CLUSTER SETTINGS`), the following fields are returned: - -Field | Description -------|------------ -`variable` | The name of the cluster setting. -`value` | The value of the cluster setting. -`setting_type` | The type of the cluster setting.
    Possible values for `setting_type` include:
    • `b` (`true` or `false`)
    • `z` (size, in bytes)
    • `d` (duration)
    • `e` (one of a set of possible values)
    • `f` (floating-point value)
    • `i` (integer)
    • `s` (string)
    -`description` | A brief description of the cluster setting, including possible values. -`public` | `true` if the cluster setting is public.
    This field is only included only if all cluster settings are displayed. - -## Examples - -### Show the value of a single cluster setting - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CLUSTER SETTING diagnostics.reporting.enabled; -~~~ - -~~~ - diagnostics.reporting.enabled ---------------------------------- - true -(1 row) -~~~ - -### Show the values of all public cluster settings - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CLUSTER SETTINGS; -~~~ - -~~~ - variable | value | setting_type | description -------------------------------+----------------+--------------+------------------------------------------------------------------------------------------------------------------- - cloudstorage.gs.default.key | | s | if set, JSON key to use during Google Cloud Storage operations - cloudstorage.http.custom_ca | | s | custom root CA (appended to system's default CAs) for verifying certificates when interacting with HTTPS storage - cloudstorage.timeout | 10m0s | d | the timeout for import/export storage operations - cluster.organization | Cockroach Demo | s | organization name - ... -~~~ - -### Show the values of all cluster settings - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW ALL CLUSTER SETTINGS; -~~~ - -~~~ - variable | value | setting_type | public | description -----------------------------------------+-------+--------------+--------+------------------------------------------------------------------------------------------------------------------- - changefeed.experimental_poll_interval | 1s | d | false | polling interval for the table descriptors - cloudstorage.gs.default.key | | s | true | if set, JSON key to use during Google Cloud Storage operations - cloudstorage.http.custom_ca | | s | true | custom root CA (appended to system's default CAs) for verifying certificates when interacting with HTTPS storage - cloudstorage.timeout | 10m0s | d | true | the timeout for import/export storage operations - ... -~~~ - -## See also - -- [`SET CLUSTER SETTING`](set-cluster-setting.html) -- [`RESET CLUSTER SETTING`](reset-cluster-setting.html) -- [Cluster settings](cluster-settings.html) -- [`SHOW` (session variable)](show-vars.html) -- [`SHOW COLUMNS`](show-columns.html) -- [`SHOW CONSTRAINTS`](show-constraints.html) -- [`SHOW CREATE`](show-create.html) -- [`SHOW DATABASES`](show-databases.html) -- [`SHOW GRANTS`](show-grants.html) -- [`SHOW INDEX`](show-index.html) -- [`SHOW USERS`](show-users.html) diff --git a/src/current/v21.1/show-columns.md b/src/current/v21.1/show-columns.md deleted file mode 100644 index 3a0d098fbec..00000000000 --- a/src/current/v21.1/show-columns.md +++ /dev/null @@ -1,109 +0,0 @@ ---- -title: SHOW COLUMNS -summary: The SHOW COLUMNS statement shows details about columns in a table, including each column's name, type, default value, and whether or not it's nullable. -toc: true ---- - -The `SHOW COLUMNS` [statement](sql-statements.html) shows details about columns in a table, including each column's name, type, default value, and whether or not it's nullable. - -## Required privileges - -The user must have any [privilege](authorization.html#assign-privileges) on the target table. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_columns.html %} -
    - -## Parameters - -Parameter | Description -----------|------------ -`table_name` | The name of the table for which to show columns. - -## Response - -The following fields are returned for each column. - -Field | Description -------|------------ -`column_name` | The name of the column. -`data_type` | The [data type](data-types.html) of the column. -`is_nullable` | Whether or not the column accepts `NULL`. Possible values: `true` or `false`. -`column_default` | The default value for the column, or an expression that evaluates to a default value. -`generation_expression` | The expression used for a [computed column](computed-columns.html). -`indices` | The list of [indexes](indexes.html) that the column is involved in, as an array. -`is_hidden` | Whether or not the column is hidden. Possible values: `true` or `false`. - -## Examples - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### Show columns in a table - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM users; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden -+-------------+-----------+-------------+----------------+-----------------------+-----------+-----------+ - id | UUID | false | NULL | | {primary} | false - city | VARCHAR | false | NULL | | {primary} | false - name | VARCHAR | true | NULL | | {} | false - address | VARCHAR | true | NULL | | {} | false - credit_card | VARCHAR | true | NULL | | {} | false -(5 rows) -~~~ - -Alternatively, within the built-in SQL shell, you can use the `\d
    ` [shell command](cockroach-sql.html#commands): - -{% include_cached copy-clipboard.html %} -~~~ sql -> \d users -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden -+-------------+-----------+-------------+----------------+-----------------------+-----------+-----------+ - id | UUID | false | NULL | | {primary} | false - city | VARCHAR | false | NULL | | {primary} | false - name | VARCHAR | true | NULL | | {} | false - address | VARCHAR | true | NULL | | {} | false - credit_card | VARCHAR | true | NULL | | {} | false -(5 rows) -~~~ - -### Show columns with comments - -You can use [`COMMENT ON`](comment-on.html) to add comments on a column. - -{% include_cached copy-clipboard.html %} -~~~ sql -> COMMENT ON COLUMN users.credit_card IS 'This column contains user payment information.'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM users WITH COMMENT; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden | comment -+-------------+-----------+-------------+----------------+-----------------------+-----------+-----------+------------------------------------------------+ - id | UUID | false | NULL | | {primary} | false | NULL - city | VARCHAR | false | NULL | | {primary} | false | NULL - name | VARCHAR | true | NULL | | {} | false | NULL - address | VARCHAR | true | NULL | | {} | false | NULL - credit_card | VARCHAR | true | NULL | | {} | false | This column contains user payment information. -(5 rows) -~~~ - -## See also - -- [`CREATE TABLE`](create-table.html) -- [Information Schema](information-schema.html) -- [Other SQL Statements](sql-statements.html) -- [`COMMENT ON`](comment-on.html) diff --git a/src/current/v21.1/show-constraints.md b/src/current/v21.1/show-constraints.md deleted file mode 100644 index 803c82cc1b7..00000000000 --- a/src/current/v21.1/show-constraints.md +++ /dev/null @@ -1,84 +0,0 @@ ---- -title: SHOW CONSTRAINTS -summary: The SHOW CONSTRAINTS statement lists the constraints on a table. -toc: true ---- - -The `SHOW CONSTRAINTS` [statement](sql-statements.html) lists all named [constraints](constraints.html) as well as any unnamed [`CHECK`](check.html) constraints on a table. - -## Required privileges - -The user must have any [privilege](authorization.html#assign-privileges) on the target table. - -## Aliases - -`SHOW CONSTRAINT` is an alias for `SHOW CONSTRAINTS`. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_constraints.html %} -
    - -## Parameters - -Parameter | Description -----------|------------ -`table_name` | The name of the table for which to show constraints. - -## Response - -The following fields are returned for each constraint. - -Field | Description -------|------------ -`table_name` | The name of the table. -`constraint_name` | The name of the constraint. -`constraint_type` | The type of constraint. -`details` | The definition of the constraint, including the column(s) to which it applies. -`validated` | Whether values in the column(s) match the constraint. - -## Example - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE orders ( - id INT PRIMARY KEY, - date TIMESTAMP NOT NULL, - priority INT DEFAULT 1, - customer_id INT UNIQUE, - status STRING DEFAULT 'open', - CHECK (priority BETWEEN 1 AND 5), - CHECK (status in ('open', 'in progress', 'done', 'cancelled')), - FAMILY (id, date, priority, customer_id, status) -); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CONSTRAINTS FROM orders; -~~~ - -~~~ -+------------+------------------------+-----------------+--------------------------------------------------------------------------+-----------+ -| table_name | constraint_name | constraint_type | details | validated | -+------------+------------------------+-----------------+--------------------------------------------------------------------------+-----------+ -| orders | check_priority | CHECK | CHECK (priority BETWEEN 1 AND 5) | true | -| orders | check_status | CHECK | CHECK (status IN ('open':::STRING, 'in progress':::STRING, | true | -| | | | 'done':::STRING, 'cancelled':::STRING)) | | -| orders | orders_customer_id_key | UNIQUE | UNIQUE (customer_id ASC) | true | -| orders | primary | PRIMARY KEY | PRIMARY KEY (id ASC) | true | -+------------+------------------------+-----------------+--------------------------------------------------------------------------+-----------+ -(4 rows) -~~~ - -## See also - -- [Constraints](constraints.html) -- [`ADD CONSTRAINT`](add-constraint.html) -- [`RENAME CONSTRAINT`](rename-constraint.html) -- [`DROP CONSTRAINT`](drop-constraint.html) -- [`VALIDATE CONSTRAINT`](validate-constraint.html) -- [`CREATE TABLE`](create-table.html) -- [Information Schema](information-schema.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/show-create.md b/src/current/v21.1/show-create.md deleted file mode 100644 index e90c97b7af9..00000000000 --- a/src/current/v21.1/show-create.md +++ /dev/null @@ -1,401 +0,0 @@ ---- -title: SHOW CREATE -summary: The SHOW CREATE statement shows the CREATE statement for an existing table, view, or sequence. -toc: true ---- - -The `SHOW CREATE` [statement](sql-statements.html) shows the `CREATE` statement for an existing [table](create-table.html), [view](create-view.html), or [sequence](create-sequence.html). - -## Required privileges - -The user must have any [privilege](authorization.html#assign-privileges) on the target table, view, or sequence. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_create.html %} -
    - -## Parameters - -Parameter | Description -----------|------------ -`object_name` | The name of the table, view, or sequence for which to show the `CREATE` statement. -`ALL TABLES` | **New in v21.1:** Show the `CREATE` statements for all tables, views, and sequences in the current database.
    This option is intended to provide the statements required to recreate the objects in the current database. As a result, `SHOW CREATE ALL TABLES` also returns the [`ALTER` statements](alter-table.html) that add, modify, and validate an object's [constraints](constraints.html). The `ALTER` statements follow the `CREATE` statements to guarantee that all objects are added before their references. - -## Response - -Field | Description -------|------------ -`table_name` | The name of the table, view, or sequence. -`create_statement` | The `CREATE` statement for the table, view, or sequence. - -## Example - -{% include {{page.version.version}}/sql/movr-statements-nodes.md %} - -### Show the `CREATE TABLE` statement for a table - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE drivers ( - id UUID NOT NULL, - city STRING NOT NULL, - name STRING, - dl STRING UNIQUE, - address STRING, - CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC) -); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE drivers; -~~~ - -~~~ - table_name | create_statement --------------+----------------------------------------------------------- - drivers | CREATE TABLE public.drivers ( - | id UUID NOT NULL, - | city STRING NOT NULL, - | name STRING NULL, - | dl STRING NULL, - | address STRING NULL, - | CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - | UNIQUE INDEX drivers_dl_key (dl ASC), - | FAMILY "primary" (id, city, name, dl, address) - | ) -(1 row) -~~~ - -To return just the `create_statement` value: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT create_statement FROM [SHOW CREATE TABLE drivers]; -~~~ - -~~~ - create_statement ------------------------------------------------------------- - CREATE TABLE public.drivers ( - id UUID NOT NULL, - city STRING NOT NULL, - name STRING NULL, - dl STRING NULL, - address STRING NULL, - CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - UNIQUE INDEX drivers_dl_key (dl ASC), - FAMILY "primary" (id, city, name, dl, address) - ) -(1 row) -~~~ - -{{site.data.alerts.callout_info}} -`SHOW CREATE TABLE` also lists any partitions and zone configurations defined on primary and secondary indexes of a table. If partitions are defined, but no zones are configured, the `SHOW CREATE TABLE` output includes a warning. -{{site.data.alerts.end}} - -### Show the `CREATE VIEW` statement for a view - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE VIEW user_view (city, name) AS SELECT city, name FROM users; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE user_view; -~~~ - -~~~ - table_name | create_statement --------------+---------------------------------------------------------------------------------------- - user_view | CREATE VIEW public.user_view (city, name) AS SELECT city, name FROM movr.public.users -(1 row) -~~~ - -To return just the `create_statement` value: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT create_statement FROM [SHOW CREATE VIEW user_view]; -~~~ - -~~~ - create_statement ------------------------------------------------------------------------------------------ - CREATE VIEW public.user_view (city, name) AS SELECT city, name FROM movr.public.users -(1 row) -~~~ - -### Show just a view's `SELECT` statement - -To get just a view's `SELECT` statement, you can query the `views` table in the built-in `information_schema` database and filter on the view name: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT view_definition - FROM information_schema.views - WHERE table_name = 'user_view'; -~~~ - -~~~ - view_definition --------------------------------------------- - SELECT city, name FROM movr.public.users -(1 row) -~~~ - -### Show the `CREATE SEQUENCE` statement for a sequence - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE SEQUENCE desc_customer_list START -1 INCREMENT -2; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE desc_customer_list; -~~~ - -~~~ - table_name | create_statement ----------------------+------------------------------------------------------------------------------------------------------------ - desc_customer_list | CREATE SEQUENCE public.desc_customer_list MINVALUE -9223372036854775808 MAXVALUE -1 INCREMENT -2 START -1 -(1 row) -~~~ - -To return just the `create_statement` value: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT create_statement FROM [SHOW CREATE desc_customer_list]; -~~~ - -~~~ - create_statement -------------------------------------------------------------------------------------------------------------- - CREATE SEQUENCE public.desc_customer_list MINVALUE -9223372036854775808 MAXVALUE -1 INCREMENT -2 START -1 -(1 row) -~~~ - -### Show the `CREATE TABLE` statement for a table with a comment - -If you [add a comment](comment-on.html) on a table, `SHOW CREATE TABLE` will display the comment. - -{% include_cached copy-clipboard.html %} -~~~ sql -> COMMENT ON TABLE users IS 'This table contains information about users.'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE users; -~~~ - -~~~ - table_name | create_statement --------------+---------------------------------------------------------------------------------- - users | CREATE TABLE public.users ( - | id UUID NOT NULL, - | city VARCHAR NOT NULL, - | name VARCHAR NULL, - | address VARCHAR NULL, - | credit_card VARCHAR NULL, - | CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - | FAMILY "primary" (id, city, name, address, credit_card) - | ); - | COMMENT ON TABLE public.users IS 'This table contains information about users.' -(1 row) -~~~ - -To return just the `create_statement` value: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT create_statement FROM [SHOW CREATE TABLE users]; -~~~ - -~~~ - create_statement ------------------------------------------------------------------------------------ - CREATE TABLE public.users ( - id UUID NOT NULL, - city VARCHAR NOT NULL, - name VARCHAR NULL, - address VARCHAR NULL, - credit_card VARCHAR NULL, - CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - FAMILY "primary" (id, city, name, address, credit_card) - ); - COMMENT ON TABLE public.users IS 'This table contains information about users.' -(1 row) -~~~ - -For more information, see [`COMMENT ON`](comment-on.html). - -### Show the `CREATE TABLE` statement for a table with a multi-region locality - -{% include_cached new-in.html version="v21.1" %} Use the `SHOW CREATE TABLE` command to view [multi-region-defined](multiregion-overview.html) table localities. - -{% include enterprise-feature.md %} - -To add the first region to the database, or to set an already-added region as the primary region, use a [`SET PRIMARY REGION`](set-primary-region.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER DATABASE movr SET PRIMARY REGION "us-east"; -~~~ - -~~~ -ALTER DATABASE PRIMARY REGION - - -Time: 49ms total (execution 48ms / network 0ms) -~~~ - -All tables will be [`REGIONAL BY TABLE`](set-locality.html#regional-by-table) in `us-east` by default. To configure the `users` table to be [`REGIONAL BY ROW`](set-locality.html#regional-by-row) instead: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE users SET LOCALITY REGIONAL BY ROW; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE users; -~~~ - -~~~ - table_name | create_statement --------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------- - users | CREATE TABLE public.users ( - | id UUID NOT NULL, - | city VARCHAR NOT NULL, - | name VARCHAR NULL, - | address VARCHAR NULL, - | credit_card VARCHAR NULL, - | crdb_region public.crdb_internal_region NOT VISIBLE NOT NULL DEFAULT default_to_database_primary_region(gateway_region())::public.crdb_internal_region, - | CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - | FAMILY "primary" (id, city, name, address, credit_card, crdb_region) - | ) LOCALITY REGIONAL BY ROW; - | COMMENT ON TABLE public.users IS 'This table contains information about users.' -(1 row) -~~~ - -### Show the statements needed to recreate all tables, views, and sequences in the current database - -{% include_cached new-in.html version="v21.1" %} To return the `CREATE` statements for all of the tables, views, and sequences in the current database, use `SHOW CREATE ALL TABLES`. - -Note that this statement also returns the [`ALTER` statements](alter-table.html) that add, modify, and validate an object's [constraints](constraints.html). - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE ALL TABLES; -~~~ - -~~~ - create_statement ---------------------------------------------------------------------------------------------------------------------------------------------------------------- - CREATE TABLE public.users ( - id UUID NOT NULL, - city VARCHAR NOT NULL, - name VARCHAR NULL, - address VARCHAR NULL, - credit_card VARCHAR NULL, - crdb_region public.crdb_internal_region NOT VISIBLE NOT NULL DEFAULT default_to_database_primary_region(gateway_region())::public.crdb_internal_region, - CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - FAMILY "primary" (id, city, name, address, credit_card, crdb_region) - ) LOCALITY REGIONAL BY ROW; - COMMENT ON TABLE public.users IS 'This table contains information about users.'; - CREATE TABLE public.vehicles ( - id UUID NOT NULL, - city VARCHAR NOT NULL, - type VARCHAR NULL, - owner_id UUID NULL, - creation_time TIMESTAMP NULL, - status VARCHAR NULL, - current_location VARCHAR NULL, - ext JSONB NULL, - CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - INDEX vehicles_auto_index_fk_city_ref_users (city ASC, owner_id ASC), - FAMILY "primary" (id, city, type, owner_id, creation_time, status, current_location, ext) - ) LOCALITY REGIONAL BY TABLE IN PRIMARY REGION; - CREATE TABLE public.rides ( - id UUID NOT NULL, - city VARCHAR NOT NULL, - vehicle_city VARCHAR NULL, - rider_id UUID NULL, - vehicle_id UUID NULL, - start_address VARCHAR NULL, - end_address VARCHAR NULL, - start_time TIMESTAMP NULL, - end_time TIMESTAMP NULL, - revenue DECIMAL(10,2) NULL, - CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - INDEX rides_auto_index_fk_city_ref_users (city ASC, rider_id ASC), - INDEX rides_auto_index_fk_vehicle_city_ref_vehicles (vehicle_city ASC, vehicle_id ASC), - FAMILY "primary" (id, city, vehicle_city, rider_id, vehicle_id, start_address, end_address, start_time, end_time, revenue), - CONSTRAINT check_vehicle_city_city CHECK (vehicle_city = city) - ) LOCALITY REGIONAL BY TABLE IN PRIMARY REGION; - CREATE TABLE public.vehicle_location_histories ( - city VARCHAR NOT NULL, - ride_id UUID NOT NULL, - "timestamp" TIMESTAMP NOT NULL, - lat FLOAT8 NULL, - long FLOAT8 NULL, - CONSTRAINT "primary" PRIMARY KEY (city ASC, ride_id ASC, "timestamp" ASC), - FAMILY "primary" (city, ride_id, "timestamp", lat, long) - ) LOCALITY REGIONAL BY TABLE IN PRIMARY REGION; - CREATE TABLE public.promo_codes ( - code VARCHAR NOT NULL, - description VARCHAR NULL, - creation_time TIMESTAMP NULL, - expiration_time TIMESTAMP NULL, - rules JSONB NULL, - CONSTRAINT "primary" PRIMARY KEY (code ASC), - FAMILY "primary" (code, description, creation_time, expiration_time, rules) - ) LOCALITY REGIONAL BY TABLE IN PRIMARY REGION; - CREATE TABLE public.user_promo_codes ( - city VARCHAR NOT NULL, - user_id UUID NOT NULL, - code VARCHAR NOT NULL, - "timestamp" TIMESTAMP NULL, - usage_count INT8 NULL, - CONSTRAINT "primary" PRIMARY KEY (city ASC, user_id ASC, code ASC), - FAMILY "primary" (city, user_id, code, "timestamp", usage_count) - ) LOCALITY REGIONAL BY TABLE IN PRIMARY REGION; - CREATE TABLE public.drivers ( - id UUID NOT NULL, - city STRING NOT NULL, - name STRING NULL, - dl STRING NULL, - address STRING NULL, - CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - UNIQUE INDEX drivers_dl_key (dl ASC), - FAMILY "primary" (id, city, name, dl, address) - ) LOCALITY REGIONAL BY TABLE IN PRIMARY REGION; - CREATE VIEW public.user_view (city, name) AS SELECT city, name FROM movr.public.users; - CREATE SEQUENCE public.desc_customer_list MINVALUE -9223372036854775808 MAXVALUE -1 INCREMENT -2 START -1; - ALTER TABLE public.vehicles ADD CONSTRAINT fk_city_ref_users FOREIGN KEY (city, owner_id) REFERENCES public.users(city, id); - ALTER TABLE public.rides ADD CONSTRAINT fk_city_ref_users FOREIGN KEY (city, rider_id) REFERENCES public.users(city, id); - ALTER TABLE public.rides ADD CONSTRAINT fk_vehicle_city_ref_vehicles FOREIGN KEY (vehicle_city, vehicle_id) REFERENCES public.vehicles(city, id); - ALTER TABLE public.vehicle_location_histories ADD CONSTRAINT fk_city_ref_rides FOREIGN KEY (city, ride_id) REFERENCES public.rides(city, id); - ALTER TABLE public.user_promo_codes ADD CONSTRAINT fk_city_ref_users FOREIGN KEY (city, user_id) REFERENCES public.users(city, id); - -- Validate foreign key constraints. These can fail if there was unvalidated data during the SHOW CREATE ALL TABLES - ALTER TABLE public.vehicles VALIDATE CONSTRAINT fk_city_ref_users; - ALTER TABLE public.rides VALIDATE CONSTRAINT fk_city_ref_users; - ALTER TABLE public.rides VALIDATE CONSTRAINT fk_vehicle_city_ref_vehicles; - ALTER TABLE public.vehicle_location_histories VALIDATE CONSTRAINT fk_city_ref_rides; - ALTER TABLE public.user_promo_codes VALIDATE CONSTRAINT fk_city_ref_users; -(20 rows) -~~~ - -## See also - -- [`CREATE TABLE`](create-table.html) -- [`CREATE VIEW`](create-view.html) -- [`CREATE TABLE`](create-sequence.html) -- [Information Schema](information-schema.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/show-databases.md b/src/current/v21.1/show-databases.md deleted file mode 100644 index d49a9d20799..00000000000 --- a/src/current/v21.1/show-databases.md +++ /dev/null @@ -1,96 +0,0 @@ ---- -title: SHOW DATABASES -summary: The SHOW DATABASES statement lists all database in the CockroachDB cluster. -keywords: reflection -toc: true ---- - -The `SHOW DATABASES` [statement](sql-statements.html) lists all databases in the CockroachDB cluster. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_databases.html %} -
    - -## Required privileges - -The user must be granted the `SELECT` [privilege](authorization.html#assign-privileges) to specific databases in order to list those databases in the CockroachDB cluster. - -## Example - -### Show databases - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW DATABASES; -~~~ - -~~~ - database_name -+---------------+ - defaultdb - movr - postgres - startrek - system -(5 rows) -~~~ - -Alternatively, within the built-in SQL shell, you can use the `\l` [shell command](cockroach-sql.html#commands) to list all databases: - -{% include_cached copy-clipboard.html %} -~~~ sql -> \l -~~~ - -~~~ - database_name -+---------------+ - defaultdb - movr - postgres - startrek - system -(5 rows) -~~~ - -### Show databases with comments - -You can use [`COMMENT ON`](comment-on.html) to add comments on a database. - -{% include_cached copy-clipboard.html %} -~~~ sql -> COMMENT ON DATABASE movr IS 'This database holds information about users, vehicles, and rides.'; -~~~ - -To view a database's comments: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW DATABASES WITH COMMENT; -~~~ - -~~~ - database_name | comment -+---------------+-------------------------------------------------------------------+ - defaultdb | NULL - movr | This database holds information about users, vehicles, and rides. - postgres | NULL - startrek | NULL - system | NULL -(5 rows) -~~~ - -For more information, see [`COMMENT ON`](comment-on.html). - -## Preloaded databases - -{% include {{ page.version.version }}/sql/preloaded-databases.md %} - -## See also - -- [`COMMENT ON`](comment-on.html) -- [`SHOW SCHEMAS`](show-schemas.html) -- [Information Schema](information-schema.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/show-enums.md b/src/current/v21.1/show-enums.md deleted file mode 100644 index 2e48b03a0e3..00000000000 --- a/src/current/v21.1/show-enums.md +++ /dev/null @@ -1,66 +0,0 @@ ---- -title: SHOW ENUMS -summary: The SHOW ENUMS statement lists the enumerated data types in a database. -toc: true ---- - - The `SHOW ENUMS` statement lists the [enumerated data types](enum.html) in the current database. - -## Syntax - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_enums.html %} -
    - -## Parameters - -Parameter | Description -----------|------------ -`name`
    `name.name` | The name of the [schema](create-schema.html) from which to show enumerated data types, or the name of the [database](create-database.html) and the [schema](create-schema.html), separated by a "`.`". - -## Examples - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TYPE weekday AS ENUM ('monday', 'tuesday', 'wednesday', 'thursday', 'friday'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TYPE weekend AS ENUM ('sunday', 'saturday'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW ENUMS; -~~~ - -~~~ - schema | name | values | owner ----------+---------+--------------------------------------------+-------- - public | weekday | {monday,tuesday,wednesday,thursday,friday} | demo - public | weekend | {sunday,saturday} | demo -(2 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW ENUMS FROM movr.public; -~~~ - -~~~ - schema | name | values | owner ----------+---------+--------------------------------------------+-------- - public | weekday | {monday,tuesday,wednesday,thursday,friday} | demo - public | weekend | {sunday,saturday} | demo -(2 rows) -~~~ - - -## See also - -- [`ENUM`](enum.html) -- [Data types](data-types.html) -- [`CREATE TYPE`](create-type.html) -- [`ALTER TYPE`](alter-type.html) -- [`DROP TYPE`](drop-type.html) diff --git a/src/current/v21.1/show-full-table-scans.md b/src/current/v21.1/show-full-table-scans.md deleted file mode 100644 index e1fc30b9e7d..00000000000 --- a/src/current/v21.1/show-full-table-scans.md +++ /dev/null @@ -1,101 +0,0 @@ ---- -title: SHOW FULL TABLE SCANS -summary: The SHOW FULL TABLE SCANS statement lists recent queries that used a full table scan. -toc: true ---- - -{% include_cached new-in.html version="v21.1" %} The `SHOW FULL TABLE SCANS` [statement](sql-statements.html) lists recent queries for which CockroachDB performed a full table scan during query execution. - -Limiting the number of queries that require full table scans can help you optimize query execution performance. For more information on query performance optimization, see [Optimize Statement Performance](make-queries-fast.html) and [SQL Tuning with `EXPLAIN`](sql-tuning-with-explain.html). - -## Syntax - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_full_scans.html %} -
    - -## Required privileges - -The [`admin` role](authorization.html#admin-role) is required to run `SHOW FULL TABLE SCANS`. - -## Examples - -To follow along, run [`cockroach demo`](cockroach-demo.html) to start a temporary, in-memory cluster with the sample [`movr` dataset](movr.html) preloaded: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach demo -~~~ - -Now, suppose that you want to query the `rides` table for all rides that cost above 90 (i.e., `WHERE revenue > 90`): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM rides WHERE revenue > 90; -~~~ - -~~~ - id | city | vehicle_city | rider_id | vehicle_id | start_address | end_address | start_time | end_time | revenue ----------------------------------------+---------------+---------------+--------------------------------------+--------------------------------------+-----------------------------------+------------------------------------+---------------------+---------------------+---------- - b4bc6a7e-f9db-4000-8000-000000000161 | amsterdam | amsterdam | ae147ae1-47ae-4800-8000-000000000022 | bbbbbbbb-bbbb-4800-8000-00000000000b | 73310 Young Harbor | 31482 Omar Street | 2018-12-13 03:04:05 | 2018-12-13 07:04:05 | 92.00 - b9db22d0-e560-4000-8000-00000000016b | amsterdam | amsterdam | c28f5c28-f5c2-4000-8000-000000000026 | aaaaaaaa-aaaa-4800-8000-00000000000a | 66748 Carroll Ports Apt. 42 | 65413 Miller Point Suite 62 | 2018-12-08 03:04:05 | 2018-12-10 05:04:05 | 98.00 - - ... - - 4fdf3b64-5a1c-4c00-8000-00000000009c | washington dc | washington dc | 47ae147a-e147-4000-8000-00000000000e | 44444444-4444-4400-8000-000000000004 | 44086 Barbara Ville | 88493 Michael Flat Apt. 97 | 2018-12-27 03:04:05 | 2018-12-28 05:04:05 | 91.00 -(51 rows) -~~~ - -This `SELECT` statement requires a full table scan at execution. As a result, the query will show up in the `SHOW FULL TABLE SCANS` output, with all of the other queries that performed full table scans: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW FULL TABLE SCANS] WHERE query LIKE 'SELECT * FROM rides WHERE revenue > %'; -~~~ - -~~~ - query | count | rows_read_avg | bytes_read_avg | service_lat_avg | contention_time_avg | max_mem_usage_avg | network_bytes_avg | max_retries -----------------------------------------+-------+---------------+----------------+-----------------+---------------------+-------------------+-------------------+-------------- - SELECT * FROM rides WHERE revenue > _ | 1 | 500 | 88497 | 0.00456 | 0 | 450560 | 0 | 0 -(1 row) -~~~ - -To limit the number of rows scanned by `SELECT` queries that filter on the `revenue` column, you can add a secondary index to the `rides` table, on the `revenue` column: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE INDEX ON rides (revenue); -~~~ - -Now, if you execute a similar query, the query will not perform a full table scan. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM rides WHERE revenue < 10; -~~~ - -~~~ - id | city | vehicle_city | rider_id | vehicle_id | start_address | end_address | start_time | end_time | revenue ----------------------------------------+---------------+---------------+--------------------------------------+--------------------------------------+---------------------------------+---------------------------------+---------------------+---------------------+---------- - ac083126-e978-4800-8000-000000000150 | amsterdam | amsterdam | c28f5c28-f5c2-4000-8000-000000000026 | aaaaaaaa-aaaa-4800-8000-00000000000a | 50217 Victoria Fields Apt. 44 | 56217 Wilson Spring | 2018-12-07 03:04:05 | 2018-12-07 10:04:05 | 9.00 - ae147ae1-47ae-4800-8000-000000000154 | amsterdam | amsterdam | bd70a3d7-0a3d-4000-8000-000000000025 | aaaaaaaa-aaaa-4800-8000-00000000000a | 63503 Lisa Summit Suite 28 | 26800 Brown Station | 2018-12-25 03:04:05 | 2018-12-26 22:04:05 | 0.00 - ... -(32 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW FULL TABLE SCANS] WHERE query LIKE 'SELECT * FROM rides WHERE revenue < %'; -~~~ - -~~~ - query | count | rows_read_avg | bytes_read_avg | service_lat_avg | contention_time_avg | max_mem_usage_avg | network_bytes_avg | max_retries ---------+-------+---------------+----------------+-----------------+---------------------+-------------------+-------------------+-------------- -(0 rows) -~~~ - -## See also - -- [`EXPLAIN`](explain.html) -- [SQL Tuning with `EXPLAIN`](sql-tuning-with-explain.html) -- [Selection queries](selection-queries.html) diff --git a/src/current/v21.1/show-grants.md b/src/current/v21.1/show-grants.md deleted file mode 100644 index 983b79c947e..00000000000 --- a/src/current/v21.1/show-grants.md +++ /dev/null @@ -1,398 +0,0 @@ ---- -title: SHOW GRANTS -summary: The SHOW GRANTS statement lists the privileges granted to users. -keywords: reflection -toc: true ---- - -The `SHOW GRANTS` [statement](sql-statements.html) lists one of the following: - -- The [roles](authorization.html#sql-users) granted to [users](authorization.html#sql-users) in a cluster. -- The [privileges](authorization.html#assign-privileges) [granted](grant.html) to [users](authorization.html#sql-users) on [databases](create-database.html), [schemas](create-schema.html), [tables](create-table.html), or [user-defined types](enum.html). - -## Syntax - -### Show privilege grants - -Use the following syntax to show the privileges granted to users on database objects: - -~~~ -SHOW GRANTS [ON [DATABASE | SCHEMA | TABLE | TYPE] ] [FOR ] -~~~ - -When `DATABASE` is omitted, the schema, tables, and types in the [current database](sql-name-resolution.html#current-database) are listed. - -### Show role grants - -Use the following syntax to show the role grants for users in a cluster. - -~~~ -SHOW GRANTS ON ROLE [] [FOR ] -~~~ - -## Parameters - -Parameter | Description --------------|----------------------------------------------------------------------------------------------------- -`targets` | A comma-separated list of database, schema, table, or user-defined type names.

    {{site.data.alerts.callout_info}}To list the privilege grants for all tables in the current database, you can use `SHOW GRANTS ON TABLE *`.{{site.data.alerts.end}} -`users` | A comma-separated list of the [users](authorization.html#sql-users) whose privileges or roles you want to show. -`roles` | A comma-separated list of the roles whose grants you want to show. - -## Response - -### Privilege grants - -The `SHOW GRANTS ON [DATABASE | SCHEMA | TABLE | TYPE]` statement can return the following fields, depending on the target object specified: - -Field | Description ------------------|----------------------------------------------------------------------------------------------------- -`database_name` | The name of the database. -`schema_name` | The name of the schema. -`table_name` | The name of the table. -`type_name` | The name of the user-defined type. -`grantee` | The name of the user or role that was granted the [privilege](authorization.html#assign-privileges). -`privilege_type` | The name of the privilege. - -### Role grants - -The `SHOW GRANTS ON ROLE` statement returns the following fields: - -Field | Description --------------|----------------------------------------------------------------------------------------------------- -`role_name` | The name of the role. -`member` | The users in the role. -`is_admin` | If `true`, the role is an [admin](authorization.html#role-admin) role. - -## Required privileges - -- No [privileges](authorization.html#assign-privileges) are required to view privileges granted to users. - -- For `SHOW GRANTS ON ROLES`, the user must have the [`SELECT`](select-clause.html) [privilege](authorization.html#assign-privileges) on the system table. - -## Examples - -### Show all grants - -To list all grants for all users and roles on the current database and its tables: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS; -~~~ - -~~~ - database_name | schema_name | relation_name | grantee | privilege_type -----------------+--------------------+-----------------------------------+---------+----------------- - movr | crdb_internal | NULL | admin | ALL - movr | crdb_internal | NULL | root | ALL - movr | crdb_internal | backward_dependencies | public | SELECT - movr | crdb_internal | builtin_functions | public | SELECT -... -(365 rows) -~~~ - -### Show a specific user or role's grants - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE USER max WITH PASSWORD roach; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT ALL ON DATABASE movr TO max; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS FOR max; -~~~ - -~~~ - database_name | schema_name | relation_name | grantee | privilege_type -----------------+--------------------+---------------+---------+----------------- - movr | crdb_internal | NULL | max | ALL - movr | information_schema | NULL | max | ALL - movr | pg_catalog | NULL | max | ALL - movr | pg_extension | NULL | max | ALL - movr | public | NULL | max | ALL -(5 rows) -~~~ - -### Show grants on databases - -**Specific database, all users and roles:** - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON DATABASE movr; -~~~ - -~~~ - database_name | schema_name | grantee | privilege_type -----------------+--------------------+---------+----------------- - movr | crdb_internal | admin | ALL - movr | crdb_internal | max | ALL - movr | crdb_internal | root | ALL - movr | information_schema | admin | ALL - movr | information_schema | max | ALL - movr | information_schema | root | ALL - movr | pg_catalog | admin | ALL - movr | pg_catalog | max | ALL - movr | pg_catalog | root | ALL - movr | pg_extension | admin | ALL - movr | pg_extension | max | ALL - movr | pg_extension | root | ALL - movr | public | admin | ALL - movr | public | max | ALL - movr | public | root | ALL -(15 rows) -~~~ - -**Specific database, specific user or role:** - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON DATABASE movr FOR max; -~~~ - -~~~ - database_name | schema_name | grantee | privilege_type -----------------+--------------------+---------+----------------- - movr | crdb_internal | max | ALL - movr | information_schema | max | ALL - movr | pg_catalog | max | ALL - movr | pg_extension | max | ALL - movr | public | max | ALL -(5 rows) -~~~ - -### Show grants on tables - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT ALL ON TABLE users TO max; -~~~ - -**Specific table, all users and roles:** - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON TABLE users; -~~~ - -~~~ - database_name | schema_name | table_name | grantee | privilege_type -----------------+-------------+------------+---------+----------------- - movr | public | users | admin | ALL - movr | public | users | max | ALL - movr | public | users | root | ALL -(3 rows) -~~~ - -**Specific table, specific role or user:** - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON TABLE users FOR max; -~~~ - -~~~ - database_name | schema_name | table_name | grantee | privilege_type -----------------+-------------+------------+---------+----------------- - movr | public | users | max | ALL -(1 row) -~~~ - -**All tables, all users and roles:** - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON TABLE *; -~~~ - -~~~ - database_name | schema_name | table_name | grantee | privilege_type -----------------+-------------+----------------------------+---------+----------------- - movr | public | promo_codes | admin | ALL - movr | public | promo_codes | root | ALL - movr | public | rides | admin | ALL - movr | public | rides | root | ALL - movr | public | user_promo_codes | admin | ALL - movr | public | user_promo_codes | root | ALL - movr | public | users | admin | ALL - movr | public | users | max | ALL - movr | public | users | root | ALL - movr | public | vehicle_location_histories | admin | ALL - movr | public | vehicle_location_histories | root | ALL - movr | public | vehicles | admin | ALL - movr | public | vehicles | root | ALL -(13 rows) -~~~ - -**All tables, specific users or roles:** - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON TABLE * FOR max; -~~~ - -~~~ - database_name | schema_name | table_name | grantee | privilege_type -----------------+-------------+------------+---------+----------------- - movr | public | users | max | ALL -(1 row) -~~~ - -### Show grants on schemas - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE SCHEMA cockroach_labs; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT ALL ON SCHEMA cockroach_labs TO max; -~~~ - -**Specific schema, all users or roles:** - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON SCHEMA cockroach_labs; -~~~ - -~~~ - database_name | schema_name | grantee | privilege_type -----------------+----------------+---------+----------------- - movr | cockroach_labs | admin | ALL - movr | cockroach_labs | max | ALL - movr | cockroach_labs | root | ALL -(3 rows) -~~~ - -**Specific schema, specific users or roles:** - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON SCHEMA cockroach_labs FOR max; -~~~ - -~~~ - database_name | schema_name | grantee | privilege_type -----------------+----------------+---------+----------------- - movr | cockroach_labs | max | ALL -(1 row) -~~~ - -### Show grants on user-defined types - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TYPE status AS ENUM ('available', 'unavailable'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT ALL ON TYPE status TO max; -~~~ - -**Specific type, all users or roles:** - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON TYPE status; -~~~ - -~~~ - database_name | schema_name | type_name | grantee | privilege_type -----------------+-------------+-----------+---------+----------------- - movr | public | status | admin | ALL - movr | public | status | max | ALL - movr | public | status | public | USAGE - movr | public | status | root | ALL -(4 rows) -~~~ - -**Specific type, specific users or roles:** - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON TYPE status FOR max; -~~~ - -~~~ - database_name | schema_name | type_name | grantee | privilege_type -----------------+-------------+-----------+---------+----------------- - movr | public | status | max | ALL -(1 row) -~~~ - -### Show role memberships - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE ROLE moderator; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT moderator TO max; -~~~ - -**All members of all roles:** - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON ROLE; -~~~ - -~~~ - role_name | member | is_admin -------------+--------+----------- - admin | root | true - moderator | max | false -(2 rows) -~~~ - -**Members of a specific role:** - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON ROLE moderator; -~~~ - -~~~ - role_name | member | is_admin -------------+--------+----------- - moderator | max | false -(1 row) -~~~ - -**Roles of a specific user or role:** - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON ROLE FOR max; -~~~ - -~~~ - role_name | member | is_admin -------------+--------+----------- - moderator | max | false -(1 row) -~~~ - -## See also - -- [Authorization](authorization.html) -- [`CREATE ROLE`](create-role.html) -- [`DROP ROLE`](drop-role.html) -- [`SHOW ROLES`](show-roles.html) -- [`GRANT`](grant.html) -- [`REVOKE`](revoke.html) -- [`SHOW GRANTS`](show-grants.html) -- [Manage Users](authorization.html#create-and-manage-users) -- [Other Cockroach Commands](cockroach-commands.html) -- [Information Schema](information-schema.html) diff --git a/src/current/v21.1/show-index.md b/src/current/v21.1/show-index.md deleted file mode 100644 index cf8f3b9c617..00000000000 --- a/src/current/v21.1/show-index.md +++ /dev/null @@ -1,119 +0,0 @@ ---- -title: SHOW INDEX -summary: The SHOW INDEX statement returns index information for a table or database. -toc: true ---- - -The `SHOW INDEX` [statement](sql-statements.html) returns index information for a table or database. - - -## Required privileges - -The user must have any [privilege](authorization.html#assign-privileges) on the target table or database. - -## Aliases - -In CockroachDB, the following are aliases for `SHOW INDEX`: - -- `SHOW INDEXES` -- `SHOW KEYS` - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_indexes.html %} -
    - -## Parameters - -Parameter | Description -----------|------------ -`table_name` | The name of the table for which you want to show indexes. -`database_name` | The name of the database for which you want to show indexes. - -## Response - -The following fields are returned for each column in each index. - -Field | Description -----------|------------ -`table_name` | The name of the table. -`index_name` | The name of the index. -`non_unique` | Whether or not values in the indexed column are unique. Possible values: `true` or `false`. -`seq_in_index` | The position of the column in the index, starting with 1. -`column_name` | The indexed column. -`direction` | How the column is sorted in the index. Possible values: `ASC` or `DESC` for indexed columns; `N/A` for stored columns. -`storing` | Whether or not the `STORING` clause was used to index the column during [index creation](create-index.html). Possible values: `true` or `false`. -`implicit` | Whether or not the column is part of the index despite not being explicitly included during [index creation](create-index.html). Possible values: `true` or `false`

    At this time, [primary key](primary-key.html) columns are the only columns that get implicitly included in secondary indexes. The inclusion of primary key columns improves performance when retrieving columns not in the index. - -## Example - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### Show indexes for a table - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE INDEX ON users (name); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW INDEX FROM users; -~~~ - -~~~ - table_name | index_name | non_unique | seq_in_index | column_name | direction | storing | implicit -+------------+----------------+------------+--------------+-------------+-----------+---------+----------+ - users | primary | false | 1 | city | ASC | false | false - users | primary | false | 2 | id | ASC | false | false - users | users_name_idx | true | 1 | name | ASC | false | false - users | users_name_idx | true | 2 | city | ASC | false | true - users | users_name_idx | true | 3 | id | ASC | false | true -(5 rows) -~~~ - -### Show indexes for a database - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW INDEXES FROM DATABASE movr; -~~~ - -~~~ - table_name | index_name | non_unique | seq_in_index | column_name | direction | storing | implicit -+----------------------------+-----------------------------------------------+------------+--------------+--------------+-----------+---------+----------+ - users | primary | false | 1 | city | ASC | false | false - users | primary | false | 2 | id | ASC | false | false - vehicles | primary | false | 1 | city | ASC | false | false - vehicles | primary | false | 2 | id | ASC | false | false - vehicles | vehicles_auto_index_fk_city_ref_users | true | 1 | city | ASC | false | false - vehicles | vehicles_auto_index_fk_city_ref_users | true | 2 | owner_id | ASC | false | false - vehicles | vehicles_auto_index_fk_city_ref_users | true | 3 | id | ASC | false | true - rides | primary | false | 1 | city | ASC | false | false - rides | primary | false | 2 | id | ASC | false | false - rides | rides_auto_index_fk_city_ref_users | true | 1 | city | ASC | false | false - rides | rides_auto_index_fk_city_ref_users | true | 2 | rider_id | ASC | false | false - rides | rides_auto_index_fk_city_ref_users | true | 3 | id | ASC | false | true - rides | rides_auto_index_fk_vehicle_city_ref_vehicles | true | 1 | vehicle_city | ASC | false | false - rides | rides_auto_index_fk_vehicle_city_ref_vehicles | true | 2 | vehicle_id | ASC | false | false - rides | rides_auto_index_fk_vehicle_city_ref_vehicles | true | 3 | city | ASC | false | true - rides | rides_auto_index_fk_vehicle_city_ref_vehicles | true | 4 | id | ASC | false | true - vehicle_location_histories | primary | false | 1 | city | ASC | false | false - vehicle_location_histories | primary | false | 2 | ride_id | ASC | false | false - vehicle_location_histories | primary | false | 3 | timestamp | ASC | false | false - promo_codes | primary | false | 1 | code | ASC | false | false - user_promo_codes | primary | false | 1 | city | ASC | false | false - user_promo_codes | primary | false | 2 | user_id | ASC | false | false - user_promo_codes | primary | false | 3 | code | ASC | false | false -(23 rows) -~~~ - -## See also - -- [`CREATE INDEX`](create-index.html) -- [`COMMENT ON`](comment-on.html) -- [`DROP INDEX`](drop-index.html) -- [`RENAME INDEX`](rename-index.html) -- [Information Schema](information-schema.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/show-jobs.md b/src/current/v21.1/show-jobs.md deleted file mode 100644 index 4350e1b6285..00000000000 --- a/src/current/v21.1/show-jobs.md +++ /dev/null @@ -1,201 +0,0 @@ ---- -title: SHOW JOBS -summary: The SHOW JOBS statement lists all currently active schema changes and backup/restore jobs. -toc: true ---- - -The `SHOW JOBS` [statement](sql-statements.html) lists all of the types of long-running tasks your cluster has performed in the last 12 hours, including: - -{% include {{ page.version.version }}/sql/schema-changes.md %} -- [`IMPORT`](import.html) -- Enterprise [`BACKUP`](backup.html) and [`RESTORE`](restore.html) -- [User-created table statistics](create-statistics.html) created for use by the [cost-based optimizer](cost-based-optimizer.html) -- [Scheduled backups](manage-a-backup-schedule.html) - -These details can help you understand the status of crucial tasks that can impact the performance of your cluster, as well as help you control them. - -To view the [automatic table statistics](cost-based-optimizer.html#table-statistics), use [`SHOW AUTOMATIC JOBS`](#show-automatic-jobs). - -To block a call to `SHOW JOBS` that returns after all specified job ID(s) have a terminal state, use [`SHOW JOBS WHEN COMPLETE`](#show-job-when-complete). The statement will return a row per job ID, which provides details of the job execution. Note that while this statement is blocking, it will time out after 24 hours. - -## Considerations - -- The `SHOW JOBS` statement shows only long-running tasks. -- For jobs older than 12 hours, query the `crdb_internal.jobs` table. -- Jobs are deleted after 14 days. This interval can be changed via the `jobs.retention_time` [cluster setting](cluster-settings.html). -- While the `SHOW JOBS WHEN COMPLETE` statement is blocking, it will time out after 24 hours. -- Garbage collection jobs are created for [dropped tables](drop-table.html) and [dropped indexes](drop-index.html), and will execute after the [GC TTL](configure-replication-zones.html#replication-zone-variables) has elapsed (default is 25 hours). These jobs cannot be canceled. - -## Required privileges - -By default, only the `root` user can execute `SHOW JOBS`. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_jobs.html %} -
    - -## Parameters - - Parameter | Description ------------|------------- -`SHOW AUTOMATIC JOBS` | Show [automatic table statistics](cost-based-optimizer.html#table-statistics). For an example, see [Show automatic jobs](#show-automatic-jobs). -`SHOW JOBS WHEN COMPLETE` | Block `SHOW JOB` until the provided job ID reaches a terminal state. For an example, see [Show job when complete](#show-job-when-complete). -`select_stmt` | A [selection query](selection-queries.html) that specifies the `job_id`(s) to view. -`job_id` | The ID of the job you want to view. -`for_schedules_clause` | The schedule you want to view jobs for. You can view jobs for a specific schedule (`FOR SCHEDULE id`) or view jobs for multiple schedules by nesting a [`SELECT` clause](select-clause.html) in the statement (`FOR SCHEDULES `). See the [examples](#show-jobs-for-a-schedule) below. - -## Response - -The output of `SHOW JOBS` lists ongoing jobs first, then completed jobs within the last 12 hours. The list of ongoing jobs is sorted by starting time, whereas the list of completed jobs is sorted by finished time. - -The following fields are returned for each job: - -Field | Description -------|------------ -`job_id` | A unique ID to identify each job. This value is used if you want to control jobs (i.e., [pause](pause-job.html), [resume](resume-job.html), or [cancel](cancel-job.html) it). -`job_type` | The type of job. Possible values: `SCHEMA CHANGE`, [`BACKUP`](backup.html), [`RESTORE`](restore.html), [`IMPORT`](import.html), and [`CREATE STATS`](create-statistics.html).

    For `SHOW AUTOMATIC JOBS`, the possible value is [`AUTO CREATE STATS`](cost-based-optimizer.html#table-statistics). -`description` | The statement that started the job, or a textual description of the job. -`statement` | When `description` is a textual description of the job, the statement that started the job is returned in this column. Currently, this field is populated only for the automatic table statistics jobs. -`user_name` | The name of the [user](authorization.html#create-and-manage-users) who started the job. -`status` | The job's current state. Possible values: `pending`, `running`, `paused`, `failed`, `succeeded`, or `canceled`. -`running_status` | The job's detailed running status, which provides visibility into the progress of the dropping or truncating of tables (i.e., [`DROP TABLE`](drop-table.html), [`DROP DATABASE`](drop-database.html), or [`TRUNCATE`](truncate.html)). For dropping or truncating jobs, the detailed running status is determined by the status of the table at the earliest stage of the schema change. The job is completed when the GC TTL expires and both the table data and ID is deleted for each of the tables involved. Possible values: `draining names`, `waiting for GC TTL`, `RocksDB compaction`, or `NULL` (when the status cannot be determined).

    For the `SHOW AUTOMATIC JOBS` statement, the value of this field is `NULL`. -`created` | The `TIMESTAMP` when the job was created. -`started` | The `TIMESTAMP` when the job began running first. -`finished` | The `TIMESTAMP` when the job was `succeeded`, `failed`, or `canceled`. -`modified` | The `TIMESTAMP` when the job had anything modified. -`fraction_completed` | The fraction (between `0.00` and `1.00`) of the job that's been completed. -`error` | If the job `failed`, the error generated by the failure. -`coordinator_id` | The ID of the node running the job. - -## Examples - -### Show jobs - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW JOBS; -~~~ - -~~~ - job_id | job_type | description |... -+---------------+-----------+-------------------------------------------+... - 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... -~~~ - -### Filter jobs - -You can filter jobs by using `SHOW JOBS` as the data source for a [`SELECT`](select-clause.html) statement, and then filtering the values with the `WHERE` clause. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW JOBS] WHERE job_type = 'RESTORE' AND status IN ('running', 'failed') ORDER BY created DESC; -~~~ - -~~~ - job_id | job_type | description |... -+---------------+-----------+-------------------------------------------+... - 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... - -~~~ - -### Show automatic jobs - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW AUTOMATIC JOBS; -~~~ - -~~~ - job_id | job_type | description |... -+--------------------+---------------------+-----------------------------------------------------+... - 438235476849557505 | AUTO CREATE STATS | Table statistics refresh for defaultdb.public.users |... -(1 row) -~~~ - -### Filter automatic jobs - -You can filter jobs by using `SHOW AUTOMATIC JOBS` as the data source for a [`SELECT`](select-clause.html) statement, and then filtering the values with the `WHERE` clause. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW AUTOMATIC JOBS] WHERE status = ('succeeded') ORDER BY created DESC; -~~~ - -~~~ - job_id | job_type | description | ... -+--------------------+---------------------+-----------------------------------------------------+ ... - 438235476849557505 | AUTO CREATE STATS | Table statistics refresh for defaultdb.public.users | ... -(1 row) -~~~ - -### Show schema changes - -You can show just schema change jobs by using `SHOW JOBS` as the data source for a [`SELECT`](select-clause.html) statement, and then filtering the `job_type` value with the `WHERE` clause: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW JOBS] WHERE job_type = 'SCHEMA CHANGE'; -~~~ - -~~~ - job_id | job_type | description |... -+---------------+-----------------+----------------------------------------------------+... - 27536791415282 | SCHEMA CHANGE | ALTER TABLE test.public.foo ADD COLUMN bar VARCHAR |... -~~~ - - [Scheme change](online-schema-changes.html) jobs can be [paused](pause-job.html), [resumed](resume-job.html), and [canceled](cancel-job.html). - -### Show job when complete - -To block `SHOW JOB` until the provided job ID reaches a terminal state, use `SHOW JOB WHEN COMPLETE`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW JOB WHEN COMPLETE 27536791415282; -~~~ -~~~ - job_id | job_type | description |... -+---------------+-----------+-------------------------------------------+... - 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |... -~~~ - -### Show jobs for a schedule - - To view jobs for a specific [backup schedule](create-schedule-for-backup.html), use the schedule's `id`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW JOBS FOR SCHEDULE 590204387299262465; -~~~ -~~~ - job_id | job_type | description | statement | user_name | status | running_status | created | started | finished | modified | fraction_completed | error | coordinator_id ----------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+-----------+---------+----------------+----------------------------------+---------+----------+----------------------------------+--------------------+-------+----------------- - 590205481558802434 | BACKUP | BACKUP INTO '/2020/09/15-161444.99' IN 's3://test/scheduled-backup-0915?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=redacted' AS OF SYSTEM TIME '2020-09-15 16:20:00+00:00' WITH revision_history, detached | | root | running | NULL | 2020-09-15 16:20:18.347383+00:00 | NULL | NULL | 2020-09-15 16:20:18.347383+00:00 | 0 | | 0 -(1 row) -~~~ - -You can also view multiple schedules by nesting a [`SELECT` clause](select-clause.html) that retrieves `id`(s) inside the `SHOW JOBS` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW JOBS FOR SCHEDULES SELECT id FROM [SHOW SCHEDULES] WHERE label = 'test_schedule'; -~~~ - -~~~ - job_id | job_type | description | statement | user_name | status | running_status | created | started | finished | modified | fraction_completed | error | coordinator_id ----------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+-----------+-----------+----------------+----------------------------------+---------+----------------------------------+----------------------------------+--------------------+-------+----------------- - 590204496007299074 | BACKUP | BACKUP INTO '/2020/09/15-161444.99' IN 's3://test/scheduled-backup-0915?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=redacted' AS OF SYSTEM TIME '2020-09-15 16:14:44.991631+00:00' WITH revision_history, detached | | root | succeeded | NULL | 2020-09-15 16:15:17.720725+00:00 | NULL | 2020-09-15 16:15:20.913789+00:00 | 2020-09-15 16:15:20.910594+00:00 | 1 | | 0 - 590205481558802434 | BACKUP | BACKUP INTO '/2020/09/15-161444.99' IN 's3://test/scheduled-backup-0915?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=redacted' AS OF SYSTEM TIME '2020-09-15 16:20:00+00:00' WITH revision_history, detached | | root | succeeded | NULL | 2020-09-15 16:20:18.347383+00:00 | NULL | 2020-09-15 16:20:48.37873+00:00 | 2020-09-15 16:20:48.374256+00:00 | 1 | | 0 -(2 rows) -~~~ - -## See also - -- [`PAUSE JOB`](pause-job.html) -- [`RESUME JOB`](pause-job.html) -- [`CANCEL JOB`](cancel-job.html) -- [`ALTER TABLE`](alter-table.html) -- [`BACKUP`](backup.html) -- [`RESTORE`](restore.html) diff --git a/src/current/v21.1/show-locality.md b/src/current/v21.1/show-locality.md deleted file mode 100644 index e49175d4f0e..00000000000 --- a/src/current/v21.1/show-locality.md +++ /dev/null @@ -1,84 +0,0 @@ ---- -title: SHOW LOCALITY -summary: The SHOW LOCALITY statement returns the locality of the current node. -toc: true ---- - -The `SHOW LOCALITY` [statement](sql-statements.html) returns the [locality](cockroach-start.html#locality) of the current node. - -If locality was not specified on node startup, the statement returns an empty row. - -## Required privileges - -No [privileges](authorization.html#assign-privileges) are required to list the locality of the current node. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_locality.html %} -
    - -## Example - -### Setup - -The following example uses MovR, a fictional vehicle-sharing application, to demonstrate CockroachDB SQL statements. For more information about the MovR example application and dataset, see [MovR: A Global Vehicle-sharing App](movr.html). - -To follow along, run [`cockroach demo movr`](cockroach-demo.html) with the `--nodes` and `--demo-locality` tags. This command opens an interactive SQL shell to a temporary, multi-node in-memory cluster with the `movr` database preloaded and set as the [current database](sql-name-resolution.html#current-database). - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach demo movr --nodes=3 --demo-locality=region=us-east,az=a:region=us-central,az=b:region=us-west1,az=c -~~~ - -### Show locality - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW LOCALITY; -~~~ - -~~~ - locality -+---------------------+ - region=us-east,az=a -(1 row) -~~~ - -### Show locality with a built-in function - -If you know the locality key, you can use the [`crdb_internal.locality_value`](functions-and-operators.html#system-info-functions) built-in function to return the locality value for the current node: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM crdb_internal.locality_value('region'); -~~~ - -~~~ - crdb_internal.locality_value -+------------------------------+ - us-east -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM crdb_internal.locality_value('az'); -~~~ - -~~~ - crdb_internal.locality_value -+------------------------------+ - a -(1 row) -~~~ - -For a more extensive example, see [Create a table with node locality information](cockroach-start.html#create-a-table-with-node-locality-information). - - -## See also - -- [Multi-Region Performance](demo-low-latency-multi-region-deployment.html) -- [Locality](cockroach-start.html#locality) -- [Orchestrated Deployment](orchestration.html) -- [Manual Deployment](manual-deployment.html) diff --git a/src/current/v21.1/show-partitions.md b/src/current/v21.1/show-partitions.md deleted file mode 100644 index d544628e30b..00000000000 --- a/src/current/v21.1/show-partitions.md +++ /dev/null @@ -1,239 +0,0 @@ ---- -title: SHOW PARTITIONS -summary: Use the SHOW PARTITIONS statement to list details about existing partitions. -toc: true ---- - -Use the `SHOW PARTITIONS` [statement](sql-statements.html) to view details about existing [partitions](partitioning.html). - -{% include {{page.version.version}}/sql/use-multiregion-instead-of-partitioning.md %} - -{% include enterprise-feature.md %} - -{% include {{page.version.version}}/sql/crdb-internal-partitions.md %} - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_partitions.html %} -
    - -## Required privileges - -No [privileges](authorization.html#assign-privileges) are required to list partitions. - -## Parameters - -Parameter | Description -----------|------------ -`database_name` | The name of the [database](create-database.html) for which to show [partitions](partitioning.html). -`table_name` | The name of the [table](create-table.html) for which to show [partitions](partitioning.html). -`table_index_name` | The name of the [index](create-index.html) for which to show [partitions](partitioning.html). - -## Response - -The following fields are returned for each partition: - -Field | Description -------|------------ -`database_name` | The name of the database that contains the partition. -`table_name` | The name of the table that contains the partition. -`partition_name` | The name of the partition. -`parent_partition` | The name of the parent partition, if the partition is a [subpartition](partitioning.html#define-subpartitions-on-a-table). -`column_names` | The names of the columns in the partition definition expression. -`index_name` | The name of the index for the partition. -`partition_value` | The value that defines the partition. -`zone_constraints` | The [zone constraints](configure-replication-zones.html), if replication zones are configured for the partition. - -## Examples - -{% include {{page.version.version}}/sql/movr-statements-geo-partitioned-replicas.md %} - -The `movr` database in this example is pre-partitioned. For information about partitioning tables, see [Define Table Partitions](partitioning.html) or [`PARTITION BY`](partition-by.html). - -### Show table partitions - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW PARTITIONS FROM TABLE users; -~~~ - -~~~ - database_name | table_name | partition_name | parent_partition | column_names | index_name | partition_value | zone_config | full_zone_config -+---------------+------------+----------------+------------------+--------------+---------------+-------------------------------------------------+----------------------------------------+-----------------------------------------+ - movr | users | us_west | NULL | city | users@primary | ('seattle'), ('san francisco'), ('los angeles') | constraints = '[+region=us-west1]' | range_min_bytes = 134217728, - | | | | | | | | range_max_bytes = 536870912, - | | | | | | | | gc.ttlseconds = 90000, - | | | | | | | | num_replicas = 3, - | | | | | | | | constraints = '[+region=us-west1]', - | | | | | | | | lease_preferences = '[]' - movr | users | us_east | NULL | city | users@primary | ('new york'), ('boston'), ('washington dc') | constraints = '[+region=us-east1]' | range_min_bytes = 134217728, - | | | | | | | | range_max_bytes = 536870912, - | | | | | | | | gc.ttlseconds = 90000, - | | | | | | | | num_replicas = 3, - | | | | | | | | constraints = '[+region=us-east1]', - | | | | | | | | lease_preferences = '[]' - movr | users | europe_west | NULL | city | users@primary | ('amsterdam'), ('paris'), ('rome') | constraints = '[+region=europe-west1]' | range_min_bytes = 134217728, - | | | | | | | | range_max_bytes = 536870912, - | | | | | | | | gc.ttlseconds = 90000, - | | | | | | | | num_replicas = 3, - | | | | | | | | constraints = '[+region=europe-west1]', - | | | | | | | | lease_preferences = '[]' -(3 rows) -~~~ - -You can also use [`SHOW CREATE TABLE`](show-create.html) to view partitions on a table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE users; -~~~ - -~~~ - table_name | create_statement -+------------+-------------------------------------------------------------------------------------+ - users | CREATE TABLE users ( - | id UUID NOT NULL, - | city VARCHAR NOT NULL, - | name VARCHAR NULL, - | address VARCHAR NULL, - | credit_card VARCHAR NULL, - | CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - | FAMILY "primary" (id, city, name, address, credit_card) - | ) PARTITION BY LIST (city) ( - | PARTITION us_west VALUES IN (('seattle'), ('san francisco'), ('los angeles')), - | PARTITION us_east VALUES IN (('new york'), ('boston'), ('washington dc')), - | PARTITION europe_west VALUES IN (('amsterdam'), ('paris'), ('rome')) - | ); - | ALTER PARTITION europe_west OF INDEX movr.public.users@primary CONFIGURE ZONE USING - | constraints = '[+region=europe-west1]'; - | ALTER PARTITION us_east OF INDEX movr.public.users@primary CONFIGURE ZONE USING - | constraints = '[+region=us-east1]'; - | ALTER PARTITION us_west OF INDEX movr.public.users@primary CONFIGURE ZONE USING - | constraints = '[+region=us-west1]' -(1 row) -~~~ - -If a partitioned table has no zones configured, the `SHOW CREATE TABLE` output includes a warning. - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER PARTITION us_west OF TABLE users CONFIGURE ZONE DISCARD; - ALTER PARTITION us_east OF TABLE users CONFIGURE ZONE DISCARD; - ALTER PARTITION europe_west OF TABLE users CONFIGURE ZONE DISCARD; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE users; -~~~ - -~~~ - table_name | create_statement -+------------+-------------------------------------------------------------------------------------+ - users | CREATE TABLE users ( - | id UUID NOT NULL, - | city VARCHAR NOT NULL, - | name VARCHAR NULL, - | address VARCHAR NULL, - | credit_card VARCHAR NULL, - | CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - | FAMILY "primary" (id, city, name, address, credit_card) - | ) PARTITION BY LIST (city) ( - | PARTITION us_west VALUES IN (('seattle'), ('san francisco'), ('los angeles')), - | PARTITION us_east VALUES IN (('new york'), ('boston'), ('washington dc')), - | PARTITION europe_west VALUES IN (('amsterdam'), ('paris'), ('rome')) - | ) - | -- Warning: Partitioned table with no zone configurations. -(1 row) -~~~ - - -### Show partitions by index - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW PARTITIONS FROM INDEX vehicles@vehicles_auto_index_fk_city_ref_users; -~~~ - -~~~ - database_name | table_name | partition_name | parent_partition | column_names | index_name | partition_value | zone_config | full_zone_config -+---------------+------------+----------------+------------------+--------------+------------------------------------------------+-------------------------------------------------+----------------------------------------+-----------------------------------------+ - movr | vehicles | us_west | NULL | city | vehicles@vehicles_auto_index_fk_city_ref_users | ('seattle'), ('san francisco'), ('los angeles') | constraints = '[+region=us-west1]' | range_min_bytes = 134217728, - | | | | | | | | range_max_bytes = 536870912, - | | | | | | | | gc.ttlseconds = 90000, - | | | | | | | | num_replicas = 3, - | | | | | | | | constraints = '[+region=us-west1]', - | | | | | | | | lease_preferences = '[]' - movr | vehicles | us_east | NULL | city | vehicles@vehicles_auto_index_fk_city_ref_users | ('new york'), ('boston'), ('washington dc') | constraints = '[+region=us-east1]' | range_min_bytes = 134217728, - | | | | | | | | range_max_bytes = 536870912, - | | | | | | | | gc.ttlseconds = 90000, - | | | | | | | | num_replicas = 3, - | | | | | | | | constraints = '[+region=us-east1]', - | | | | | | | | lease_preferences = '[]' - movr | vehicles | europe_west | NULL | city | vehicles@vehicles_auto_index_fk_city_ref_users | ('amsterdam'), ('paris'), ('rome') | constraints = '[+region=europe-west1]' | range_min_bytes = 134217728, - | | | | | | | | range_max_bytes = 536870912, - | | | | | | | | gc.ttlseconds = 90000, - | | | | | | | | num_replicas = 3, - | | | | | | | | constraints = '[+region=europe-west1]', - | | | | | | | | lease_preferences = '[]' -(3 rows) -~~~ - -### Show partitions by database - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW PARTITIONS FROM DATABASE movr; -~~~ - -~~~ - database_name | table_name | partition_name | parent_partition | column_names | index_name | partition_value | zone_config | full_zone_config -+---------------+------------+----------------+------------------+--------------+------------------+-------------------------------------------------+----------------------------------------+-----------------------------------------+ - movr | users | us_west | NULL | city | users@primary | ('seattle'), ('san francisco'), ('los angeles') | NULL | range_min_bytes = 134217728, - | | | | | | | | range_max_bytes = 536870912, - | | | | | | | | gc.ttlseconds = 90000, - | | | | | | | | num_replicas = 3, - | | | | | | | | constraints = '[]', - | | | | | | | | lease_preferences = '[]' - movr | users | us_east | NULL | city | users@primary | ('new york'), ('boston'), ('washington dc') | NULL | range_min_bytes = 134217728, - | | | | | | | | range_max_bytes = 536870912, - | | | | | | | | gc.ttlseconds = 90000, - | | | | | | | | num_replicas = 3, - | | | | | | | | constraints = '[]', - | | | | | | | | lease_preferences = '[]' - movr | users | europe_west | NULL | city | users@primary | ('amsterdam'), ('paris'), ('rome') | NULL | range_min_bytes = 134217728, - | | | | | | | | range_max_bytes = 536870912, - | | | | | | | | gc.ttlseconds = 90000, - | | | | | | | | num_replicas = 3, - | | | | | | | | constraints = '[]', - | | | | | | | | lease_preferences = '[]' - movr | vehicles | us_west | NULL | city | vehicles@primary | ('seattle'), ('san francisco'), ('los angeles') | constraints = '[+region=us-west1]' | range_min_bytes = 134217728, - | | | | | | | | range_max_bytes = 536870912, - | | | | | | | | gc.ttlseconds = 90000, - | | | | | | | | num_replicas = 3, - | | | | | | | | constraints = '[+region=us-west1]', - | | | | | | | | lease_preferences = '[]' - movr | vehicles | us_east | NULL | city | vehicles@primary | ('new york'), ('boston'), ('washington dc') | constraints = '[+region=us-east1]' | range_min_bytes = 134217728, - | | | | | | | | range_max_bytes = 536870912, - | | | | | | | | gc.ttlseconds = 90000, - | | | | | | | | num_replicas = 3, - | | | | | | | | constraints = '[+region=us-east1]', - | | | | | | | | lease_preferences = '[]' - movr | vehicles | europe_west | NULL | city | vehicles@primary | ('amsterdam'), ('paris'), ('rome') | constraints = '[+region=europe-west1]' | range_min_bytes = 134217728, - | | | | | | | | range_max_bytes = 536870912, - | | | | | | | | gc.ttlseconds = 90000, - | | | | | | | | num_replicas = 3, - | | | | | | | | constraints = '[+region=europe-west1]', - | | | | | | | | lease_preferences = '[]' -... -(24 rows) -~~~ - -{% include {{page.version.version}}/sql/crdb-internal-partitions-example.md %} - -## See also - -- [Define Table Partitions](partitioning.html) -- [SQL Statements](sql-statements.html) -- [Multi-Region Performance](demo-low-latency-multi-region-deployment.html) diff --git a/src/current/v21.1/show-range-for-row.md b/src/current/v21.1/show-range-for-row.md deleted file mode 100644 index 2ab51c49422..00000000000 --- a/src/current/v21.1/show-range-for-row.md +++ /dev/null @@ -1,162 +0,0 @@ ---- -title: SHOW RANGE FOR ROW -summary: The SHOW RANGE FOR ROW statement shows information about the range for a single row. -toc: true ---- - -The `SHOW RANGE ... FOR ROW` [statement](sql-statements.html) shows information about a [range](architecture/overview.html#glossary) for a single row in a table or index. This information is useful for verifying how SQL data maps to underlying ranges, and where the replicas for a range are located. - -{% include {{ page.version.version }}/misc/experimental-warning.md %} - -{{site.data.alerts.callout_info}} -To show information about the ranges for all data in a table, index, or database, use the [`SHOW RANGES`](show-ranges.html) statement. -{{site.data.alerts.end}} - -## Syntax - -~~~ -SHOW RANGE FROM TABLE FOR ROW (value1, value2, ...) -SHOW RANGE FROM INDEX [ @ ] FOR ROW (value1, value2, ...) -~~~ - -## Required privileges - -The user must have the `SELECT` [privilege](authorization.html#assign-privileges) on the target table. - -## Parameters - -Parameter | Description -----------|------------ -`tablename` | The name of the table that contains the row that you want range information about. -`indexname` | The name of the index that contains the row that you want range information about. -`(value1, value2, ...)` | The values of the indexed columns of the row that you want range information about, as a tuple. In previous releases, this statement required the values of all columns of a row. - -## Response - -The following fields are returned: - -Field | Description -------|------------ -`start_key` | The start key for the range. -`end_key` | The end key for the range. -`range_id` | The range ID. -`lease_holder` | The node that contains the range's [leaseholder](architecture/overview.html#glossary). -`lease_holder_locality` | The [locality](cockroach-start.html#locality) of the leaseholder. -`replicas` | The nodes that contain the range [replicas](architecture/overview.html#glossary). -`replica_localities` | The [locality](cockroach-start.html#locality) of the range. - -## Examples - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### Show range information for a row in a table - -To show information about a row in a table, you must know the values of the columns in the row's primary key: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW INDEX FROM vehicles; -~~~ - -~~~ - table_name | index_name | non_unique | seq_in_index | column_name | direction | storing | implicit --------------+---------------------------------------+------------+--------------+-------------+-----------+---------+----------- - vehicles | primary | false | 1 | city | ASC | false | false - vehicles | primary | false | 2 | id | ASC | false | false - vehicles | vehicles_auto_index_fk_city_ref_users | true | 1 | city | ASC | false | false - vehicles | vehicles_auto_index_fk_city_ref_users | true | 2 | owner_id | ASC | false | false - vehicles | vehicles_auto_index_fk_city_ref_users | true | 3 | id | ASC | false | true -(5 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT city, id FROM vehicles LIMIT 5; -~~~ - -~~~ - city | id ---------------+--------------------------------------- - amsterdam | bbbbbbbb-bbbb-4800-8000-00000000000b - amsterdam | aaaaaaaa-aaaa-4800-8000-00000000000a - boston | 22222222-2222-4200-8000-000000000002 - boston | 33333333-3333-4400-8000-000000000003 - los angeles | 99999999-9999-4800-8000-000000000009 -(5 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW RANGE FROM TABLE vehicles FOR ROW ( - 'boston', - '22222222-2222-4200-8000-000000000002' - ); -~~~ - -~~~ - start_key | end_key | range_id | lease_holder | lease_holder_locality | replicas | replica_localities -----------------------------------------------------------------+---------------------------------------------------------+----------+--------------+-----------------------+----------+--------------------------- - /"boston"/"\"\"\"\"\"\"B\x00\x80\x00\x00\x00\x00\x00\x00\x02" | /"boston"/"333333D\x00\x80\x00\x00\x00\x00\x00\x00\x03" | 57 | 1 | region=us-east1,az=b | {1} | {"region=us-east1,az=b"} -(1 row) -~~~ - -### Show range information for a row by a secondary index - -To show information about a row in a secondary index, you must know the values of the indexed columns: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW INDEX FROM vehicles; -~~~ - -~~~ - table_name | index_name | non_unique | seq_in_index | column_name | direction | storing | implicit --------------+---------------------------------------+------------+--------------+-------------+-----------+---------+----------- - vehicles | primary | false | 1 | city | ASC | false | false - vehicles | primary | false | 2 | id | ASC | false | false - vehicles | vehicles_auto_index_fk_city_ref_users | true | 1 | city | ASC | false | false - vehicles | vehicles_auto_index_fk_city_ref_users | true | 2 | owner_id | ASC | false | false - vehicles | vehicles_auto_index_fk_city_ref_users | true | 3 | id | ASC | false | true -(5 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT city, owner_id, id FROM vehicles@vehicles_auto_index_fk_city_ref_users LIMIT 5; -~~~ - -~~~ - city | owner_id | id ---------------+--------------------------------------+--------------------------------------- - amsterdam | bd70a3d7-0a3d-4000-8000-000000000025 | bbbbbbbb-bbbb-4800-8000-00000000000b - amsterdam | c28f5c28-f5c2-4000-8000-000000000026 | aaaaaaaa-aaaa-4800-8000-00000000000a - boston | 2e147ae1-47ae-4400-8000-000000000009 | 22222222-2222-4200-8000-000000000002 - boston | 33333333-3333-4400-8000-00000000000a | 33333333-3333-4400-8000-000000000003 - los angeles | 9eb851eb-851e-4800-8000-00000000001f | 99999999-9999-4800-8000-000000000009 -(5 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW RANGE FROM INDEX vehicles@vehicles_auto_index_fk_city_ref_users FOR ROW ( - 'boston', - '2e147ae1-47ae-4400-8000-000000000009', - '22222222-2222-4200-8000-000000000002' - ); -~~~ - -~~~ - start_key | end_key | range_id | lease_holder | lease_holder_locality | replicas | replica_localities -------------+---------+----------+--------------+-----------------------+----------+--------------------------- - NULL | NULL | 53 | 1 | region=us-east1,az=b | {1} | {"region=us-east1,az=b"} -~~~ - -## See also - -- [`SHOW RANGES`](show-ranges.html) -- [`SPLIT AT`](split-at.html) -- [`CREATE TABLE`](create-table.html) -- [`CREATE INDEX`](create-index.html) -- [Indexes](indexes.html) -- [Partitioning tables](partitioning.html) -- [Architecture Overview](architecture/overview.html) diff --git a/src/current/v21.1/show-ranges.md b/src/current/v21.1/show-ranges.md deleted file mode 100644 index c5c3fb7f46a..00000000000 --- a/src/current/v21.1/show-ranges.md +++ /dev/null @@ -1,127 +0,0 @@ ---- -title: SHOW RANGES -summary: The SHOW RANGES statement shows information about the ranges that comprise the data for a table, index, or entire database. -toc: true ---- - -The `SHOW RANGES` [statement](sql-statements.html) shows information about the [ranges](architecture/overview.html#glossary) that comprise the data for a table, index, or entire database. This information is useful for verifying how SQL data maps to underlying ranges, and where the replicas for ranges are located. If `SHOW RANGES` displays `NULL` for both the start and end keys of a range, the range is empty and has no splits. - -{{site.data.alerts.callout_info}} -To show range information for a specific row in a table or index, use the [`SHOW RANGE ... FOR ROW`](show-range-for-row.html) statement. -{{site.data.alerts.end}} - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_ranges.html %} -
    - -## Required privileges - -Only members of the [`admin` role](authorization.html#admin-role) can run `SHOW RANGES`. By default, the `root` user belongs to the `admin` role. - -## Parameters - -Parameter | Description -----------|------------ -[`table_name`](sql-grammar.html#table_name) | The name of the table you want range information about. -[`table_index_name`](sql-grammar.html#table_index_name) | The name of the index you want range information about. -[`database_name`](sql-grammar.html#database_name) | The name of the database you want range information about. - -## Response - -The following fields are returned for each partition: - -Field | Description -------|------------ -`table_name` | The name of the table. -`start_key` | The start key for the range. -`end_key` | The end key for the range. -`range_id` | The range ID. -`range_size_mb` | The size of the range. -`lease_holder` | The node that contains the range's [leaseholder](architecture/overview.html#glossary). -`lease_holder_locality` | The [locality](cockroach-start.html#locality) of the leaseholder. -`replicas` | The nodes that contain the range [replicas](architecture/overview.html#glossary). -`replica_localities` | The [locality](cockroach-start.html#locality) of the range. - -{{site.data.alerts.callout_info}} -If both `start_key` and `end_key` show `NULL`, the range is empty and has no splits. -{{site.data.alerts.end}} - -## Examples - -{% include {{page.version.version}}/sql/movr-statements-geo-partitioned-replicas.md %} - -### Show ranges for a table (primary index) - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW RANGES FROM TABLE vehicles] WHERE "start_key" NOT LIKE '%Prefix%'; -~~~ -~~~ - start_key | end_key | range_id | range_size_mb | lease_holder | lease_holder_locality | replicas | replica_localities -+------------------+----------------------------+----------+---------------+--------------+--------------------------+----------+------------------------------------------------------------------------------------+ - /"new york" | /"new york"/PrefixEnd | 58 | 0.000304 | 2 | region=us-east1,az=c | {1,2,5} | {"region=us-east1,az=b","region=us-east1,az=c","region=us-west1,az=b"} - /"washington dc" | /"washington dc"/PrefixEnd | 102 | 0.000173 | 2 | region=us-east1,az=c | {1,2,3} | {"region=us-east1,az=b","region=us-east1,az=c","region=us-east1,az=d"} - /"boston" | /"boston"/PrefixEnd | 63 | 0.000288 | 3 | region=us-east1,az=d | {1,2,3} | {"region=us-east1,az=b","region=us-east1,az=c","region=us-east1,az=d"} - /"seattle" | /"seattle"/PrefixEnd | 97 | 0.000295 | 4 | region=us-west1,az=a | {4,5,6} | {"region=us-west1,az=a","region=us-west1,az=b","region=us-west1,az=c"} - /"los angeles" | /"los angeles"/PrefixEnd | 55 | 0.000156 | 5 | region=us-west1,az=b | {4,5,6} | {"region=us-west1,az=a","region=us-west1,az=b","region=us-west1,az=c"} - /"san francisco" | /"san francisco"/PrefixEnd | 71 | 0.000309 | 6 | region=us-west1,az=c | {1,5,6} | {"region=us-east1,az=b","region=us-west1,az=b","region=us-west1,az=c"} - /"amsterdam" | /"amsterdam"/PrefixEnd | 59 | 0.000305 | 9 | region=europe-west1,az=d | {7,8,9} | {"region=europe-west1,az=b","region=europe-west1,az=c","region=europe-west1,az=d"} - /"paris" | /"paris"/PrefixEnd | 62 | 0.000299 | 9 | region=europe-west1,az=d | {7,8,9} | {"region=europe-west1,az=b","region=europe-west1,az=c","region=europe-west1,az=d"} - /"rome" | /"rome"/PrefixEnd | 67 | 0.000168 | 9 | region=europe-west1,az=d | {7,8,9} | {"region=europe-west1,az=b","region=europe-west1,az=c","region=europe-west1,az=d"} -(9 rows) -~~~ - -### Show ranges for an index - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW RANGES FROM INDEX vehicles_auto_index_fk_city_ref_users] WHERE "start_key" NOT LIKE '%Prefix%'; -~~~ -~~~ - start_key | end_key | range_id | range_size_mb | lease_holder | lease_holder_locality | replicas | replica_localities -+------------------+----------------------------+----------+---------------+--------------+--------------------------+----------+------------------------------------------------------------------------------------+ - /"washington dc" | /"washington dc"/PrefixEnd | 188 | 0.000089 | 2 | region=us-east1,az=c | {1,2,3} | {"region=us-east1,az=b","region=us-east1,az=c","region=us-east1,az=d"} - /"boston" | /"boston"/PrefixEnd | 141 | 0.000164 | 3 | region=us-east1,az=d | {1,2,3} | {"region=us-east1,az=b","region=us-east1,az=c","region=us-east1,az=d"} - /"new york" | /"new york"/PrefixEnd | 168 | 0.000174 | 3 | region=us-east1,az=d | {1,2,3} | {"region=us-east1,az=b","region=us-east1,az=c","region=us-east1,az=d"} - /"los angeles" | /"los angeles"/PrefixEnd | 165 | 0.000087 | 6 | region=us-west1,az=c | {4,5,6} | {"region=us-west1,az=a","region=us-west1,az=b","region=us-west1,az=c"} - /"san francisco" | /"san francisco"/PrefixEnd | 174 | 0.000183 | 6 | region=us-west1,az=c | {4,5,6} | {"region=us-west1,az=a","region=us-west1,az=b","region=us-west1,az=c"} - /"seattle" | /"seattle"/PrefixEnd | 186 | 0.000166 | 6 | region=us-west1,az=c | {4,5,6} | {"region=us-west1,az=a","region=us-west1,az=b","region=us-west1,az=c"} - /"amsterdam" | /"amsterdam"/PrefixEnd | 137 | 0.00017 | 9 | region=europe-west1,az=d | {7,8,9} | {"region=europe-west1,az=b","region=europe-west1,az=c","region=europe-west1,az=d"} - /"paris" | /"paris"/PrefixEnd | 170 | 0.000162 | 9 | region=europe-west1,az=d | {7,8,9} | {"region=europe-west1,az=b","region=europe-west1,az=c","region=europe-west1,az=d"} - /"rome" | /"rome"/PrefixEnd | 172 | 0.00008 | 9 | region=europe-west1,az=d | {7,8,9} | {"region=europe-west1,az=b","region=europe-west1,az=c","region=europe-west1,az=d"} -(9 rows) -~~~ - -### Show ranges for a database - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW RANGES FROM database movr] WHERE "start_key" NOT LIKE '%Prefix%'; -~~~ -~~~ - table_name | start_key | end_key | range_id | range_size_mb | lease_holder | lease_holder_locality | replicas | replica_localities -+----------------------------+------------------+----------------------------+----------+---------------+--------------+--------------------------+----------+------------------------------------------------------------------------------------+ - users | /"amsterdam" | /"amsterdam"/PrefixEnd | 47 | 0.000562 | 7 | region=europe-west1,az=b | {7,8,9} | {"region=europe-west1,az=b","region=europe-west1,az=c","region=europe-west1,az=d"} - users | /"boston" | /"boston"/PrefixEnd | 51 | 0.000665 | 3 | region=us-east1,az=d | {1,2,3} | {"region=us-east1,az=b","region=us-east1,az=c","region=us-east1,az=d"} - users | /"chicago" | /"los angeles" | 83 | 0 | 4 | region=us-west1,az=a | {2,4,8} | {"region=us-east1,az=c","region=us-west1,az=a","region=europe-west1,az=c"} - users | /"los angeles" | /"los angeles"/PrefixEnd | 45 | 0.000697 | 4 | region=us-west1,az=a | {4,5,6} | {"region=us-west1,az=a","region=us-west1,az=b","region=us-west1,az=c"} - users | /"new york" | /"new york"/PrefixEnd | 48 | 0.000664 | 1 | region=us-east1,az=b | {1,2,3} | {"region=us-east1,az=b","region=us-east1,az=c","region=us-east1,az=d"} - users | /"paris" | /"paris"/PrefixEnd | 52 | 0.000628 | 8 | region=europe-west1,az=c | {7,8,9} | {"region=europe-west1,az=b","region=europe-west1,az=c","region=europe-west1,az=d"} - - ... - - user_promo_codes | /"washington dc" | /"washington dc"/PrefixEnd | 144 | 0 | 2 | region=us-east1,az=c | {1,2,3} | {"region=us-east1,az=b","region=us-east1,az=c","region=us-east1,az=d"} -(73 rows) -~~~ - -## See also - -- [`SHOW RANGE ... FOR ROW`](show-range-for-row.html) -- [`SPLIT AT`](split-at.html) -- [`CREATE TABLE`](create-table.html) -- [`CREATE INDEX`](create-index.html) -- [Indexes](indexes.html) -- [Partitioning tables](partitioning.html) -- [Architecture Overview](architecture/overview.html) diff --git a/src/current/v21.1/show-regions.md b/src/current/v21.1/show-regions.md deleted file mode 100644 index 69037eb24dc..00000000000 --- a/src/current/v21.1/show-regions.md +++ /dev/null @@ -1,196 +0,0 @@ ---- -title: SHOW REGIONS -summary: The SHOW REGIONS statement shows the cluster regions or database regions in a multi-region cluster. -toc: true ---- - -{% include_cached new-in.html version="v21.1" %} The `SHOW REGIONS` [statement](sql-statements.html) lists the [cluster regions](multiregion-overview.html#cluster-regions) for a multi-region cluster, or the [database regions](multiregion-overview.html#database-regions) for the databases in a multi-region cluster. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_regions.html %} -
    - -## Required privileges - -Only members of the [`admin` role](authorization.html#admin-role) can run `SHOW REGIONS`. By default, the `root` user belongs to the `admin` role. - -## Parameters - -Parameter | Description -----------|------------ -`FROM CLUSTER`| Show the [cluster regions](multiregion-overview.html#cluster-regions) for a cluster.
    Cluster regions are specified [at cluster startup](cockroach-start.html#locality). -`FROM DATABASE` | Show all [database regions](multiregion-overview.html#database-regions) for the current database.
    Database regions can be added [at database creation](create-database.html), or [after a database has been created](alter-database.html). -`FROM DATABASE database_name` | Show all database regions from the database named `database_name`. -`FROM ALL DATABASES` | Show the database regions for all databases in the cluster. - -## Response - -`SHOW REGIONS`, `SHOW REGIONS FROM CLUSTER`, and `SHOW REGIONS FROM DATABASE` return the following fields for each region: - -Field | Description | `SHOW REGIONS` | `SHOW REGIONS FROM CLUSTER` | `SHOW REGIONS FROM DATABASE` -------|--------------|----------------|-----------------------------|------------------------ -`region` | The name of the region. | ✓ | ✓ | ✓ -`zones` | The availability zones for the region. | ✓ | ✓ | ✓ -`database_names` | A set of database names that use the region. | ✓ | | -`primary_region_of` | A set of database names for which the region is the primary region. | ✓ | | -`database`| The name of the database that uses the region. | | | ✓ -`primary` | If `true`, indicates that the region is the primary region. | | | ✓ - -`SHOW REGIONS FROM ALL DATABASES` returns the following fields for each database: - -Field | Description -------|-------------- -`database_name` | The name of the database. -`regions` | A set of region names in use by the database. -`primary_region` | The primary region of the database. - -## Examples - -{% include {{page.version.version}}/sql/movr-statements-nodes.md %} - -### View the regions in a cluster - -After cluster startup, you can view all of the cluster regions available in the cluster with `SHOW REGIONS FROM CLUSTER`: - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW REGIONS FROM CLUSTER; -~~~ - -~~~ - region | zones --------------+------------------------------ - us-central | {us-central-a,us-central-b} - us-east | {us-east-a,us-east-b} - us-west | {us-west-a,us-west-b} -(3 rows) -~~~ - -### View the regions in a single database - -`SHOW REGIONS FROM DATABASE` returns the database regions for a specific database. - -[Add an available region](add-region.html) as the primary region for the `movr` database: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE movr PRIMARY REGION "us-east"; -~~~ - -~~~ -ALTER DATABASE PRIMARY REGION -~~~ - -{{site.data.alerts.callout_info}} -Only [cluster regions](multiregion-overview.html#cluster-regions) (i.e., regions that are defined at [node startup time](cockroach-start.html#locality)) can be added to a multi-region database. -{{site.data.alerts.end}} - -Then, add more regions to the database: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE movr ADD REGION "us-west"; -~~~ - -~~~ -ALTER DATABASE ADD REGION -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE movr ADD REGION "us-central"; -~~~ - -~~~ -ALTER DATABASE ADD REGION -~~~ - -To view the regions associated with the database: - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW REGIONS FROM DATABASE movr; -~~~ - -~~~ - database | region | primary | zones ------------+------------+---------+------------------------------ - movr | us-east | true | {us-east-a,us-east-b} - movr | us-central | false | {us-central-a,us-central-b} - movr | us-west | false | {us-west-a,us-west-b} -(3 rows) -~~~ - -With `movr` set as the current database, the following statement returns the same results: - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW REGIONS FROM DATABASE; -~~~ - -~~~ - database | region | primary | zones ------------+------------+---------+------------------------------ - movr | us-east | true | {us-east-a,us-east-b} - movr | us-central | false | {us-central-a,us-central-b} - movr | us-west | false | {us-west-a,us-west-b} -(3 rows) -~~~ - -### View the regions for all databases in a cluster - -Create another database in the cluster with a primary region: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE DATABASE cockroachlabs PRIMARY REGION "us-east"; -~~~ - -Then, add another region to the database: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE cockroachlabs ADD REGION "us-west"; -~~~ - -To show the regions in use by all the databases in a cluster, use `SHOW REGIONS`: - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW REGIONS; -~~~ - -~~~ - region | zones | database_names | primary_region_of --------------+-----------------------------+----------------------+----------------------- - us-central | {us-central-a,us-central-b} | {movr} | {} - us-east | {us-east-a,us-east-b} | {cockroachlabs,movr} | {cockroachlabs,movr} - us-west | {us-west-a,us-west-b} | {cockroachlabs,movr} | {} -(3 rows) -~~~ - -To show the region information for each database in the cluster, use `SHOW REGIONS FROM ALL DATABASES`: - -{% include_cached copy-clipboard.html %} -~~~ sql -SHOW REGIONS FROM ALL DATABASES; -~~~ - -~~~ - database_name | regions | primary_region -----------------+------------------------------+----------------- - cockroachlabs | {us-east,us-west} | us-east - defaultdb | {} | NULL - movr | {us-central,us-east,us-west} | us-east - postgres | {} | NULL - system | {} | NULL -(5 rows) -~~~ - -## See also - -- [Multi-region overview](multiregion-overview.html) -- [`ADD REGION`](add-region.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/show-roles.md b/src/current/v21.1/show-roles.md deleted file mode 100644 index ffdb563dbd4..00000000000 --- a/src/current/v21.1/show-roles.md +++ /dev/null @@ -1,47 +0,0 @@ ---- -title: SHOW ROLES -summary: The SHOW ROLES statement lists the roles for all databases. -toc: true ---- - -The `SHOW ROLES` [statement](sql-statements.html) lists the roles for all databases. - -{{site.data.alerts.callout_info}} - Since the keywords `ROLES` and `USERS` can now be used interchangeably in SQL statements for enhanced Postgres compatibility, `SHOW ROLES` is now an alias for [`SHOW USERS`](show-users.html). -{{site.data.alerts.end}} - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_roles.html %} -
    - -## Required privileges - -The role must have the [`SELECT`](select-clause.html) [privilege](authorization.html#assign-privileges) on the `system.users` and `system.role_members` tables. - -## Example - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW ROLES; -~~~ - -~~~ - username | options | member_of ------------+------------+------------ - admin | CREATEROLE | {} - carl | NOLOGIN | {} - petee | | {} - root | CREATEROLE | {admin} -(4 rows) -~~~ - -## See also - -- [Authorization](authorization.html) -- [`CREATE ROLE`](create-role.html) -- [`DROP ROLE`](drop-role.html) -- [`GRANT`](grant.html) -- [`REVOKE`](revoke.html) -- [Manage Users](authorization.html#create-and-manage-users) diff --git a/src/current/v21.1/show-savepoint-status.md b/src/current/v21.1/show-savepoint-status.md deleted file mode 100644 index 85446fd0345..00000000000 --- a/src/current/v21.1/show-savepoint-status.md +++ /dev/null @@ -1,71 +0,0 @@ ---- -title: SHOW SAVEPOINT STATUS -summary: The SHOW SAVEPOINT STATUS statement lists the active savepoints in the current transaction. -toc: true ---- - -The `SHOW SAVEPOINT STATUS` [statement](sql-statements.html) lists the active [savepoints](savepoint.html) in the current [transaction](transactions.html). - -## Required privileges - -No [privileges](authorization.html#assign-privileges) are required to create or show a savepoint. However, privileges are required for each statement within a transaction. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_savepoint_status.html %} -
    - -## Response - -The following fields are returned for each savepoint. - -Field | Description -------|------------ -`savepoint_name` | The name of the savepoint. -`is_initial_savepoint` | Whether the savepoint is the outermost savepoint in the transaction. - -## Example - -First, open a [transaction](transactions.html) using [`BEGIN`](begin-transaction.html), and create a [nested transaction](transactions.html#nested-transactions) using a [savepoint](savepoint.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; -SAVEPOINT foo; -~~~ - -Next, use the `SHOW SAVEPOINT STATUS` statement to list the active savepoints in the current nested transaction. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SAVEPOINT STATUS; -~~~ - -~~~ - savepoint_name | is_initial_savepoint ------------------+----------------------- - foo | true -(1 row) -~~~ - -Currently, there is only one savepoint. - -We can commit this nested transaction by issuing the [`RELEASE SAVEPOINT`](release-savepoint.html) statement. Then, we clear the connection for the next transaction by issuing a [`COMMIT`](commit-transaction.html) statement. - -{% include_cached copy-clipboard.html %} -~~~ sql -> RELEASE SAVEPOINT foo; -COMMIT; -~~~ - -If we did not want to commit this nested transaction, but restart it instead, we would have issued a [`ROLLBACK TO SAVEPOINT`](rollback-transaction.html#rollback-a-nested-transaction). - -## See also - -- [`SAVEPOINT`](savepoint.html) -- [`RELEASE SAVEPOINT`](release-savepoint.html) -- [`ROLLBACK`](rollback-transaction.html) -- [`BEGIN`](begin-transaction.html) -- [`COMMIT`](commit-transaction.html) -- [Transactions](transactions.html) diff --git a/src/current/v21.1/show-schedules.md b/src/current/v21.1/show-schedules.md deleted file mode 100644 index 661db0e83a5..00000000000 --- a/src/current/v21.1/show-schedules.md +++ /dev/null @@ -1,129 +0,0 @@ ---- -title: SHOW SCHEDULES -summary: The SHOW SCHEDULES statement lists all currently active backup schedules. -toc: true ---- - - The `SHOW SCHEDULES` [statement](sql-statements.html) lists all of the currently active [backup schedules](create-schedule-for-backup.html). - -## Required privileges - -Only members of the [`admin` role](authorization.html#default-roles) can resume a schedule. By default, the `root` user belongs to the `admin` role. - -## Synopsis - -~~~ -SHOW [RUNNING | PAUSED] SCHEDULES [FOR BACKUP]; -SHOW SCHEDULE ; -~~~ - -## Parameters - - Parameter | Description -----------------+------------- -`schedule_id` | The ID of the schedule you want to view. - -## Response - -The output of `SHOW SCHEDULES` is sorted by creation time. The following fields are returned for each schedule: - -Field | Description -------+------------ -`id` | A unique ID to identify each schedule. This value is used if you want to control schedules (i.e., [pause](pause-schedules.html), [resume](resume-schedules.html), or [drop](drop-schedules.html) it). -`label` | The name used to identify the backup schedule, given at the time of schedule creation. -`schedule_status` | The schedule's current status. -`next_run` | The [`TIMESTAMP`](timestamp.html) at which the next backup in the schedule is slated to run. -`state` | Displays last-known errors or messages about the backup schedule. Cleared on retry of a scheduled job. -`recurrence` | How often the backup is taken, which is set at the time of schedule creation. -`jobsrunning` | The number of [jobs](show-jobs.html) currently running for the schedule. -`owner` | The [user](create-user.html) who created the backup schedule. Users with active schedules cannot be dropped. -`created` | The [`TIMESTAMP`](timestamp.html) when the job was created. -`command` | The command that the schedule will run to take the backup. This is derived from the [`CREATE SCHEDULE FOR BACKUP`](create-schedule-for-backup.html) statement used. - -## Examples - -### Show schedules - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SCHEDULES; -~~~ - -~~~ - id | label | schedule_status | next_run | state | recurrence | jobsrunning | owner | created | command ----------------------+---------------------+-----------------+---------------------------+----------------------------------------+------------+-------------+-------+----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - 588796190000218113 | schedule_name | ACTIVE | 2020-09-15 00:00:00+00:00 | NULL | @daily | 0 | root | 2020-09-10 16:52:16.846944+00:00 | {"backup_statement": "BACKUP INTO LATEST IN 's3://test/schedule-test2?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=redacted' WITH revision_history, detached", "backup_type": 1} - 588819866656997377 | schedule_database | ACTIVE | 2020-09-15 00:01:00+00:00 | NULL | 1 0 * * * | 0 | root | 2020-09-10 18:52:42.339026+00:00 | {"backup_statement": "BACKUP DATABASE movr INTO LATEST IN 's3://test/schedule-database?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=redacted' WITH revision_history, detached", "backup_type": 1} - 588820615348027393 | schedule_database | ACTIVE | 2020-09-14 22:00:00+00:00 | NULL | @hourly | 0 | root | 2020-09-10 18:56:30.919388+00:00 | {"backup_statement": "BACKUP TABLE movr.vehicles INTO LATEST IN 's3://test/schedule-table?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=redacted' WITH revision_history, detached", "backup_type": 1} - 589963390457741313 | scheduled_first_run | PAUSED | NULL | Waiting for initial backup to complete | @hourly | 0 | root | 2020-09-14 19:48:58.042612+00:00 | {"backup_statement": "BACKUP TABLE movr.vehicles INTO LATEST IN 's3://test/scheduled-first-run?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=redacted' WITH revision_history, detached", "backup_type": 1} - 588796190012702721 | schedule_name | ACTIVE | 2020-09-20 00:00:00+00:00 | NULL | @weekly | 0 | root | 2020-09-10 16:52:16.846944+00:00 | {"backup_statement": "BACKUP INTO 's3://test/schedule-test2?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=redacted' WITH revision_history, detached"} - 588819866674233345 | schedule_database | ACTIVE | 2020-09-20 00:00:00+00:00 | NULL | @weekly | 0 | root | 2020-09-10 18:52:42.339026+00:00 | {"backup_statement": "BACKUP DATABASE movr INTO 's3://test/schedule-database?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=redacted' WITH revision_history, detached"} - 588820615382302721 | schedule_database | ACTIVE | 2020-09-15 00:00:00+00:00 | NULL | @daily | 0 | root | 2020-09-10 18:56:30.919388+00:00 | {"backup_statement": "BACKUP TABLE movr.vehicles INTO 's3://test/schedule-table?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=redacted' WITH revision_history, detached"} - 589963390487363585 | scheduled_first_run | ACTIVE | 2020-09-15 00:00:00+00:00 | NULL | @daily | 0 | root | 2020-09-14 19:48:58.042612+00:00 | {"backup_statement": "BACKUP TABLE movr.vehicles INTO 's3://test/scheduled-first-run?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=redacted' WITH revision_history, detached", "unpause_on_success": 589963390457741313} -(8 rows) -~~~ - -### Show running schedules - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW RUNNING SCHEDULES; -~~~ - -~~~ - id | label | schedule_status | next_run | state | recurrence | jobsrunning | owner | created | command ----------------------+---------------------+-----------------+---------------------------+----------+------------+-------------+-------+----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------- - 588796190000218113 | schedule_name | ACTIVE | 2020-09-15 00:00:00+00:00 | NULL | @daily | 0 | root | 2020-09-10 16:52:16.846944+00:00 | BACKUP INTO LATEST IN 's3://test/schedule-test2?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=redacted' WITH revision_history, detached - 588819866656997377 | schedule_database | ACTIVE | 2020-09-15 00:01:00+00:00 | NULL | 1 0 * * * | 0 | root | 2020-09-10 18:52:42.339026+00:00 | BACKUP DATABASE movr INTO LATEST IN 's3://test/schedule-database?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=redacted' WITH revision_history, detached - 588820615348027393 | schedule_database | ACTIVE | 2020-09-14 22:00:00+00:00 | NULL | @hourly | 0 | root | 2020-09-10 18:56:30.919388+00:00 | BACKUP TABLE movr.vehicles INTO LATEST IN 's3://test/schedule-table?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=redacted' WITH revision_history, detached - 588796190012702721 | schedule_name | ACTIVE | 2020-09-20 00:00:00+00:00 | NULL | @weekly | 0 | root | 2020-09-10 16:52:16.846944+00:00 | BACKUP INTO 's3://test/schedule-test2?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=redacted' WITH revision_history, detached - 588819866674233345 | schedule_database | ACTIVE | 2020-09-20 00:00:00+00:00 | NULL | @weekly | 0 | root | 2020-09-10 18:52:42.339026+00:00 | BACKUP DATABASE movr INTO 's3://test/schedule-database?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=redacted' WITH revision_history, detached - 588820615382302721 | schedule_database | ACTIVE | 2020-09-15 00:00:00+00:00 | NULL | @daily | 0 | root | 2020-09-10 18:56:30.919388+00:00 | BACKUP TABLE movr.vehicles INTO 's3://test/schedule-table?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=redacted' WITH revision_history, detached - 589963390487363585 | scheduled_first_run | ACTIVE | 2020-09-15 00:00:00+00:00 | NULL | @daily | 0 | root | 2020-09-14 19:48:58.042612+00:00 | BACKUP TABLE movr.vehicles INTO 's3://test/scheduled-first-run?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=redacted' WITH revision_history, detached -(7 rows) -~~~ - -### Show paused schedules - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW PAUSED SCHEDULES; -~~~ -~~~ - id | label | schedule_status | next_run | state | recurrence | jobsrunning | owner | created | command ----------------------+---------------------+-----------------+----------+----------------------------------------+------------+-------------+-------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - 589963390457741313 | scheduled_first_run | PAUSED | NULL | Waiting for initial backup to complete | @hourly | 0 | root | 2020-09-14 19:48:58.042612+00:00 | {"backup_statement": "BACKUP TABLE movr.vehicles INTO LATEST IN 's3://test/scheduled-first-run?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=redacted' WITH revision_history, detached", "backup_type": 1} -(1 row) -~~~ - -### Show a specific schedule - -To view a specific schedule, use the schedule's `id`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SCHEDULE 588796190012702721; -~~~ - -~~~ - id | label | schedule_status | next_run | state | recurrence | jobsrunning | owner | created | command ----------------------+---------------+-----------------+---------------------------+---------+------------+-------------+-------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------ - 588796190012702721 | schedule_name | ACTIVE | 2020-09-20 00:00:00+00:00 | NULL | @weekly | 0 | root | 2020-09-10 16:52:16.846944+00:00 | {"backup_statement": "BACKUP INTO 's3://test/schedule-test2?AWS_ACCESS_KEY_ID=x&AWS_SECRET_ACCESS_KEY=redacted' WITH revision_history, detached"} -(1 row) -~~~ - -## See also - -- [Manage a Backup Schedule](manage-a-backup-schedule.html) -- [`BACKUP`](backup.html) -- [`RESTORE`](restore.html) -- [`SHOW BACKUP`](show-backup.html) -- [`PAUSE SCHEDULES`](pause-schedules.html) -- [`RESUME SCHEDULES`](resume-schedules.html) -- [`DROP SCHEDULES`](drop-schedules.html) -- [`PAUSE JOB`](pause-job.html) -- [`RESUME JOB`](pause-job.html) -- [`CANCEL JOB`](cancel-job.html) -- [Take Full and Incremental Backups](take-full-and-incremental-backups.html) -- [Use the Built-in SQL Client](cockroach-sql.html) -- [Other Cockroach Commands](cockroach-commands.html) diff --git a/src/current/v21.1/show-schemas.md b/src/current/v21.1/show-schemas.md deleted file mode 100644 index 0660806b039..00000000000 --- a/src/current/v21.1/show-schemas.md +++ /dev/null @@ -1,59 +0,0 @@ ---- -title: SHOW SCHEMAS -summary: The SHOW SCHEMAS statement lists the schemas in a database. -toc: true ---- - -The `SHOW SCHEMAS` [statement](sql-statements.html) lists all [schemas](sql-name-resolution.html#naming-hierarchy) in a database. - -## Required privileges - -The `SELECT` [privilege](authorization.html#assign-privileges) on the database is required to list the schemas in a database. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_schemas.html %} -
    - -## Parameters - -Parameter | Description -----------|------------ -`name` | The name of the database for which to show [schemas](sql-name-resolution.html#naming-hierarchy). When omitted, the schemas in the [current database](sql-name-resolution.html#current-database) are listed. - -## Example - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### Show schemas in the current database - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE SCHEMA org_one; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SCHEMAS; -~~~ - -~~~ - schema_name ----------------------- - crdb_internal - information_schema - org_one - pg_catalog - pg_extension - public -(6 rows) -~~~ - -## See also - -- [Logical Schemas and Namespaces](sql-name-resolution.html) -- [`CREATE SCHEMA`](create-schema.html) -- [`SHOW DATABASES`](show-databases.html) -- [Information Schema](information-schema.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/show-sequences.md b/src/current/v21.1/show-sequences.md deleted file mode 100644 index b6c84c62616..00000000000 --- a/src/current/v21.1/show-sequences.md +++ /dev/null @@ -1,48 +0,0 @@ ---- -title: SHOW SEQUENCES -summary: The SHOW SEQUENCES statement lists the sequences in a database. -toc: true ---- - -The `SHOW SEQUENCES` [statement](sql-statements.html) lists all sequences in a database. - -## Required privileges - -No [privileges](authorization.html#assign-privileges) are required to list the sequences in a database. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_sequences.html %} -
    - -## Parameters - -Parameter | Description -----------|------------ -`name` | The name of the database for which to list sequences. When omitted, the sequences in the [current database](sql-name-resolution.html#current-database) are listed. - -## Example - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE SEQUENCE sequence_test; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SEQUENCES; -~~~ - -~~~ - sequence_schema | sequence_name -------------------+---------------- - public | sequence_test -(1 row) -~~~ - -## See also - -- [`CREATE SEQUENCE`](create-sequence.html) -- [`DROP SEQUENCE`](drop-sequence.html) -- [`ALTER SEQUENCE`](alter-sequence.html) diff --git a/src/current/v21.1/show-sessions.md b/src/current/v21.1/show-sessions.md deleted file mode 100644 index c63a371a23a..00000000000 --- a/src/current/v21.1/show-sessions.md +++ /dev/null @@ -1,196 +0,0 @@ ---- -title: SHOW SESSIONS -summary: The SHOW SESSIONS statement lists all currently active sessions across the cluster or on the local node. -toc: true ---- - -The `SHOW SESSIONS` [statement](sql-statements.html) lists details about currently active sessions, including: - -- The address of the client that opened the session -- The node connected to -- How long the connection has been open -- Which queries are active in the session -- Which query has been running longest in the session - -These details let you monitor the overall state of client connections and identify those that may need further investigation or adjustment. - - -## Required privileges - -All users can see their own currently active sessions. All users belonging to the `admin` role can view see all users' currently active sessions. To view other non-admin users' sessions, the non-admin user must have the [`VIEWACTIVITY`](create-user.html#create-a-user-that-can-see-and-cancel-non-admin-queries-and-sessions) parameter set. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_sessions.html %} -
    - -- To list the active sessions across all nodes of the cluster, use `SHOW SESSIONS` or `SHOW CLUSTER SESSIONS`. -- To list the active sessions just on the local node, use `SHOW LOCAL SESSIONS`. - -## Response - -The following fields are returned for each session: - -Field | Description -------|------------ -`node_id` | The ID of the node connected to. -`session_id` | The ID of the connected session. -`user_name` | The username of the connected user. -`client_address` | The address and port of the connected client. -`application_name` | The [application name](set-vars.html#supported-variables) specified by the client, if any. For sessions from the [built-in SQL client](cockroach-sql.html), this will be `cockroach`. -`active_queries` | The SQL queries currently active in the session. -`last_active_query` | The most recently completed SQL query in the session. -`session_start` | The timestamp at which the session started. -`oldest_query_start` | The timestamp at which the oldest currently active SQL query in the session started. -`kv_txn` | The ID of the current key-value transaction for the session. - -## Examples - -### List active sessions across the cluster - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CLUSTER SESSIONS; -~~~ - -~~~ -+---------+----------------------------------+-----------+--------------------+------------------+---------------------------------------------+--------------------------------------------|----------------------------------+----------------------------------+--------------------------------------+ -| node_id | session_id | user_name | client_address | application_name | active_queries | last_active_query | session_start | oldest_query_start | kv_txn -+---------+----------------------------------+-----------+--------------------+------------------+---------------------------------------------+--------------------------------------------+----------------------------------+----------------------------------+--------------------------------------| -| 2 | 1530fd8813ad694b0000000000000001 | mroach | 192.168.0.72:56194 | test_app | UPSERT INTO test.kv(k, v) VALUES ($1, $2); | SELECT k, v FROM test.kv WHERE k = $1; | 2017-08-10 14:08:22.878113+00:00 | 2017-08-10 14:08:44.648985+00:00 | 81fbdd4d-394c-4784-b540-97cd73910dba -| 2 | 1530ce8813ad694b0000000000000001 | mroach | 192.168.0.72:56201 | test_app | UPSERT INTO test.kv(k, v) VALUES ($1, $2); | SELECT k, v FROM test.kv WHERE k = $1; | 2017-08-10 14:08:22.878306+00:00 | 2017-08-10 14:08:44.653135+00:00 | 5aa6f141-5cae-468f-b16a-dfe8d4fb4bea -| 2 | 1520ab8813ad694b0000000000000001 | mroach | 192.168.0.72:56198 | test_app | UPSERT INTO test.kv(k, v) VALUES ($1, $2); | SELECT k, v FROM test.kv WHERE k = $1; | 2017-08-10 14:08:22.878464+00:00 | 2017-08-10 14:08:44.643749+00:00 | d8fedb88-fc21-4720-aabe-cd43ec204d88 -| 3 | 1559f68813ad694b0000000000000001 | broach | 192.168.0.73:56199 | test_app | SELECT k, v FROM test.kv WHERE k = $1; | UPSERT INTO test.kv(k, v) VALUES ($1, $2); | 2017-08-10 14:08:22.878048+00:00 | 2017-08-10 14:08:44.655709+00:00 | NULL -| 3 | 1340rd8813ad694b0000000000000001 | broach | 192.168.0.73:56196 | test_app | UPSERT INTO test.kv(k, v) VALUES ($1, $2); | SELECT k, v FROM test.kv WHERE k = $1; | 2017-08-10 14:08:22.878166+00:00 | 2017-08-10 14:08:44.647464+00:00 | aded7717-94e1-4ac4-9d37-8765e3418e32 -| 1 | 1230ab8813ad694b0000000000000001 | lroach | 192.168.0.71:56180 | test_app | UPSERT INTO test.kv(k, v) VALUES ($1, $2); | SELECT k, v FROM test.kv WHERE k = $1; | 2017-08-10 14:08:22.87337+00:00 | 2017-08-10 14:08:44.64788+00:00 | f691c5dd-b29e-48ed-a1dd-6d7f71faa82e -| 1 | 15234d8813ad694b0000000000000001 | lroach | 192.168.0.71:56197 | test_app | UPSERT INTO test.kv(k, v) VALUES ($1, $2); | SELECT k, v FROM test.kv WHERE k = $1; | 2017-08-10 14:08:22.877932+00:00 | 2017-08-10 14:08:44.644786+00:00 | 86ae25ea-9abf-4f5e-ad96-0522178f4ce6 -| 1 | 14605d8813ad694b0000000000000001 | lroach | 192.168.0.71:56200 | test_app | UPSERT INTO test.kv(k, v) VALUES ($1, $2); | SELECT k, v FROM test.kv WHERE k = $1; | 2017-08-10 14:08:22.878534+00:00 | 2017-08-10 14:08:44.653524+00:00 | 8ad972b6-4347-4128-9e52-8553f3491963 -| 1 | 1965lh8813ad694b0000000000000001 | root | 127.0.0.1:56211 | cockroach | SHOW CLUSTER SESSIONS; | | 2017-08-10 14:08:27.666826+00:00 | 2017-08-10 14:08:44.653355+00:00 | NULL -+---------+----------------------------------+-----------+--------------------+------------------+---------------------------------------------+--------------------------------------------+----------------------------------+----------------------------------|--------------------------------------+ -(9 rows) -~~~ - -Alternatively, you can use `SHOW SESSIONS` to receive the same response. - -### List active sessions on the local node - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW LOCAL SESSIONS; -~~~ - -~~~ -+---------+----------------------------------+-----------+--------------------+------------------+---------------------------------------------+--------------------------------------------|----------------------------------+----------------------------------+--------------------------------------+ -| node_id | session_id | user_name | client_address | application_name | active_queries | last_active_query | session_start | oldest_query_start | kv_txn | -+---------+----------------------------------+-----------+--------------------+------------------+---------------------------------------------+--------------------------------------------+----------------------------------+----------------------------------+--------------------------------------| -| 1 | 1230ab8813ad694b0000000000000001 | lroach | 192.168.0.71:56180 | test_app | UPSERT INTO test.kv(k, v) VALUES ($1, $2); | SELECT k, v FROM test.kv WHERE k = $1; | 2017-08-10 14:08:22.87337+00:00 | 2017-08-10 14:08:44.64788+00:00 | f691c5dd-b29e-48ed-a1dd-6d7f71faa82e | -| 1 | 15234d8813ad694b0000000000000001 | lroach | 192.168.0.71:56197 | test_app | UPSERT INTO test.kv(k, v) VALUES ($1, $2); | SELECT k, v FROM test.kv WHERE k = $1; | 2017-08-10 14:08:22.877932+00:00 | 2017-08-10 14:08:44.644786+00:00 | 86ae25ea-9abf-4f5e-ad96-0522178f4ce6 | -| 1 | 14605d8813ad694b0000000000000001 | lroach | 192.168.0.71:56200 | test_app | UPSERT INTO test.kv(k, v) VALUES ($1, $2); | SELECT k, v FROM test.kv WHERE k = $1; | 2017-08-10 14:08:22.878534+00:00 | 2017-08-10 14:08:44.653524+00:00 | 8ad972b6-4347-4128-9e52-8553f3491963 | -| 1 | 1965lh8813ad694b0000000000000001 | root | 127.0.0.1:56211 | cockroach | SHOW CLUSTER SESSIONS; | | 2017-08-10 14:08:27.666826+00:00 | 2017-08-10 14:08:44.653355+00:00 | NULL | -+---------+----------------------------------+-----------+--------------------+------------------+---------------------------------------------+--------------------------------------------+----------------------------------+----------------------------------|--------------------------------------+ -(4 rows) -~~~ - -### Filter for specific sessions - -You can use a [`SELECT`](select-clause.html) statement to filter the list of currently active sessions by one or more of the [response fields](#response). - -#### Show sessions associated with a specific user - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW CLUSTER SESSIONS] WHERE user_name = 'mroach'; -~~~ - -~~~ -+---------+-----------+--------------------+------------------+---------------------------------------------+--------------------------------------------|----------------------------------+----------------------------------+--------------------------------------+ -| node_id | user_name | client_address | application_name | active_queries | last_active_query | session_start | oldest_query_start | kv_txn | -+---------+-----------+--------------------+------------------+---------------------------------------------+--------------------------------------------+----------------------------------+----------------------------------+--------------------------------------| -| 2 | mroach | 192.168.0.72:56194 | test_app | UPSERT INTO test.kv(k, v) VALUES ($1, $2); | SELECT k, v FROM test.kv WHERE k = $1; | 2017-08-10 14:08:22.878113+00:00 | 2017-08-10 14:08:44.648985+00:00 | 81fbdd4d-394c-4784-b540-97cd73910dba | -| 2 | mroach | 192.168.0.72:56201 | test_app | UPSERT INTO test.kv(k, v) VALUES ($1, $2); | SELECT k, v FROM test.kv WHERE k = $1; | 2017-08-10 14:08:22.878306+00:00 | 2017-08-10 14:08:44.653135+00:00 | 5aa6f141-5cae-468f-b16a-dfe8d4fb4bea | -| 2 | mroach | 192.168.0.72:56198 | test_app | UPSERT INTO test.kv(k, v) VALUES ($1, $2); | SELECT k, v FROM test.kv WHERE k = $1; | 2017-08-10 14:08:22.878464+00:00 | 2017-08-10 14:08:44.643749+00:00 | d8fedb88-fc21-4720-aabe-cd43ec204d88 | -+---------+-----------+--------------------+------------------+---------------------------------------------+--------------------------------------------+----------------------------------+----------------------------------|--------------------------------------+ -(3 rows) -~~~ - -#### Exclude sessions from the built-in SQL client - -To exclude sessions from the [built-in SQL client](cockroach-sql.html), filter for sessions that do not show `cockroach` as the `application_name`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW CLUSTER SESSIONS] - WHERE application_name != 'cockroach'; -~~~ - -~~~ -+---------+-----------+--------------------+------------------+---------------------------------------------+--------------------------------------------|----------------------------------+----------------------------------+--------------------------------------+ -| node_id | user_name | client_address | application_name | active_queries | last_active_query | session_start | oldest_query_start | kv_txn -+---------+-----------+--------------------+------------------+---------------------------------------------+--------------------------------------------+----------------------------------+----------------------------------+--------------------------------------| -| 2 | mroach | 192.168.0.72:56194 | test_app | UPSERT INTO test.kv(k, v) VALUES ($1, $2); | SELECT k, v FROM test.kv WHERE k = $1; | 2017-08-10 14:08:22.878113+00:00 | 2017-08-10 14:08:44.648985+00:00 | 81fbdd4d-394c-4784-b540-97cd73910dba -| 2 | mroach | 192.168.0.72:56201 | test_app | UPSERT INTO test.kv(k, v) VALUES ($1, $2); | SELECT k, v FROM test.kv WHERE k = $1; | 2017-08-10 14:08:22.878306+00:00 | 2017-08-10 14:08:44.653135+00:00 | 5aa6f141-5cae-468f-b16a-dfe8d4fb4bea -| 2 | mroach | 192.168.0.72:56198 | test_app | UPSERT INTO test.kv(k, v) VALUES ($1, $2); | SELECT k, v FROM test.kv WHERE k = $1; | 2017-08-10 14:08:22.878464+00:00 | 2017-08-10 14:08:44.643749+00:00 | d8fedb88-fc21-4720-aabe-cd43ec204d88 -| 3 | broach | 192.168.0.73:56199 | test_app | SELECT k, v FROM test.kv WHERE k = $1; | UPSERT INTO test.kv(k, v) VALUES ($1, $2); | 2017-08-10 14:08:22.878048+00:00 | 2017-08-10 14:08:44.655709+00:00 | NULL -| 3 | broach | 192.168.0.73:56196 | test_app | UPSERT INTO test.kv(k, v) VALUES ($1, $2); | SELECT k, v FROM test.kv WHERE k = $1; | 2017-08-10 14:08:22.878166+00:00 | 2017-08-10 14:08:44.647464+00:00 | aded7717-94e1-4ac4-9d37-8765e3418e32 -| 1 | lroach | 192.168.0.71:56180 | test_app | UPSERT INTO test.kv(k, v) VALUES ($1, $2); | SELECT k, v FROM test.kv WHERE k = $1; | 2017-08-10 14:08:22.87337+00:00 | 2017-08-10 14:08:44.64788+00:00 | f691c5dd-b29e-48ed-a1dd-6d7f71faa82e -| 1 | lroach | 192.168.0.71:56197 | test_app | UPSERT INTO test.kv(k, v) VALUES ($1, $2); | SELECT k, v FROM test.kv WHERE k = $1; | 2017-08-10 14:08:22.877932+00:00 | 2017-08-10 14:08:44.644786+00:00 | 86ae25ea-9abf-4f5e-ad96-0522178f4ce6 -| 1 | lroach | 192.168.0.71:56200 | test_app | UPSERT INTO test.kv(k, v) VALUES ($1, $2); | SELECT k, v FROM test.kv WHERE k = $1; | 2017-08-10 14:08:22.878534+00:00 | 2017-08-10 14:08:44.653524+00:00 | 8ad972b6-4347-4128-9e52-8553f3491963 -+---------+-----------+--------------------+------------------+---------------------------------------------+--------------------------------------------+----------------------------------+----------------------------------|--------------------------------------+ -(8 rows) -~~~ - -### Identify and cancel a problematic query - -If a session has been open for a long time and you are concerned that the oldest active SQL query may be problematic, you can use the [`SHOW STATEMENTS`](show-statements.html) statement to further investigate the query and then, if necessary, use the [`CANCEL QUERY`](cancel-query.html) statement to cancel it. - -For example, let's say you run `SHOW SESSIONS` and notice that the following session has been open for more than 2 hours: - -~~~ -+---------+----------+--------------------+------------------+------------------------------------+--------------------|----------------------------------+----------------------------------+--------+ -| node_id | user_name | client_address | application_name | active_queries | last_active_query | session_start | oldest_query_start | kv_txn -+---------+-----------+--------------------+------------------+------------------------------------+--------------------+----------------------------------+----------------------------------|--------+ -| 2 | mroach | 192.168.0.72:56194 | test_app | SELECT * FROM test.kv ORDER BY k; | | 2017-08-10 14:08:22.878113+00:00 | 2017-08-10 14:08:22.878113+00:00 | NULL -+---------+-----------+--------------------+------------------+------------------------------------+--------------------|----------------------------------+----------------------------------+--------+ -~~~ - -Since the `oldest_query_start` timestamp is the same as the `session_start` timestamp, you are concerned that the `SELECT` query shown in `active_queries` has been running for too long and may be consuming too many resources. So you use the [`SHOW STATEMENTS`](show-statements.html) statement to get more information about the query, filtering based on details you already have: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW CLUSTER STATEMENTS] - WHERE client_address = '192.168.0.72:56194' - AND user_name = 'mroach' - AND query = 'SELECT * FROM test.kv ORDER BY k'; -~~~ - -~~~ -+----------------------------------+---------+-----------+----------------------------------+----------------------------------+--------------------+------------------+-------------+-----------+ -| query_id | node_id | user_name | start | query | client_address | application_name | distributed | phase | -+----------------------------------+---------+-----------+----------------------------------+----------------------------------+--------------------+------------------+-------------+-----------+ -| 14dacc1f9a781e3d0000000000000001 | 2 | mroach | 2017-08-10 14:08:22.878113+00:00 | SELECT * FROM test.kv ORDER BY k | 192.168.0.72:56194 | test_app | false | executing | -+----------------------------------+---------+-----------+----------------------------------+----------------------------------+--------------------+------------------+-------------+-----------+ -~~~ - -Using the `start` field, you confirm that the query has been running since the start of the session and decide that is too long. So to cancel the query, and stop it from consuming resources, you note the `query_id` and use it with the [`CANCEL QUERY`](cancel-query.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CANCEL QUERY '14dacc1f9a781e3d0000000000000001'; -~~~ - -Alternatively, if you know that you want to cancel the query based on the details in `SHOW SESSIONS`, you could execute a single [`CANCEL QUERY`](cancel-query.html) statement with a nested `SELECT` statement that returns the `query_id`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CANCEL QUERY (SELECT query_id FROM [SHOW CLUSTER STATEMENTS] - WHERE client_address = '192.168.0.72:56194' - AND user_name = 'mroach' - AND query = 'SELECT * FROM test.kv ORDER BY k'); -~~~ - -## See also - -- [`SHOW STATEMENTS`](show-statements.html) -- [`CANCEL QUERY`](cancel-query.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/show-statements.md b/src/current/v21.1/show-statements.md deleted file mode 100644 index d53322cbaf3..00000000000 --- a/src/current/v21.1/show-statements.md +++ /dev/null @@ -1,178 +0,0 @@ ---- -title: SHOW STATEMENTS -summary: The SHOW STATEMENTS statement lists all currently active queries across the cluster or on the local node. -toc: true ---- - -The `SHOW STATEMENTS` [statement](sql-statements.html) lists details about currently active SQL queries, including: - -- The internal ID of the query -- The node executing the query -- The SQL query itself -- How long the query has been running -- The client address, application name, and user that issued the query -- The ID for the current session - -These details let you monitor the progress of active queries and, if necessary, identify those that may need to be [cancelled](cancel-query.html) to prevent unwanted resource consumption. - -{{site.data.alerts.callout_info}} -Schema changes and [`BACKUP`](backup.html)/[`RESTORE`](restore.html) statements are not executed as queries internally and so are not listed by `SHOW STATEMENTS`. To monitor such statements, use [`SHOW JOBS`](show-jobs.html) instead. -{{site.data.alerts.end}} - -## Aliases - -In CockroachDB, the following are aliases for `SHOW STATEMENTS`: - -- `SHOW QUERIES` - -## Required privileges - -All users can see their own currently active queries. All users belonging to the `admin` role can view all users' currently active queries. To view other non-admin users' queries, the non-admin user must have the [`VIEWACTIVITY`](create-user.html#create-a-user-that-can-see-and-cancel-non-admin-queries-and-sessions) parameter set. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_statements.html %} -
    - -- To list the active queries across all nodes of the cluster, use `SHOW STATEMENTS` or `SHOW CLUSTER STATEMENTS`. -- To list the active queries just on the local node, use `SHOW LOCAL STATEMENTS`. - -## Response - -The following fields are returned for each query: - -Field | Description -------|------------ -`query_id` | The ID of the query. -`node_id` | The ID of the node. -`session_id` | The ID of the session. -`user_name` | The username of the connected user. -`start` | The timestamp at which the query started. -`query` | The SQL query. -`client_address` | The address and port of the client that issued the SQL query. -`application_name` | The [application name](set-vars.html#supported-variables) specified by the client, if any. For queries from the [built-in SQL client](cockroach-sql.html), this will be `$ cockroach sql`. -`distributed` | If `true`, the query is being executed by the Distributed SQL (DistSQL) engine. If `false`, the query is being executed by the standard "local" SQL engine. If `NULL`, the query is being prepared and it's not yet known which execution engine will be used. -`phase` | The phase of the query's execution. If `preparing`, the statement is being parsed and planned. If `executing`, the statement is being executed. - -## Examples - -### List queries across the cluster - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CLUSTER STATEMENTS; -~~~ - -~~~ - query_id | node_id | session_id | user_name | start | query | client_address | application_name | distributed | phase -+----------------------------------+---------+----------------------------------+-----------+----------------------------------+-----------------------------------------------------------------------+-----------------+------------------+-------------+-----------+ - 15f92b12b2fb95b00000000000000002 | 2 | 15f92b10b92ed4080000000000000002 | root | 2020-03-04 17:48:23.309592+00:00 | SELECT city, id FROM vehicles WHERE city = $1 | 127.0.0.1:65092 | | false | executing - 15f92b12b2ea5f700000000000000001 | 1 | 15f92adefd48d8a00000000000000001 | root | 2020-03-04 17:48:23.3085+00:00 | SHOW CLUSTER STATEMENTS | 127.0.0.1:64970 | $ cockroach sql | false | executing - 15f92b12b2ffeb100000000000000003 | 3 | 15f92b0e4ea399680000000000000003 | root | 2020-03-04 17:48:23.30989+00:00 | UPSERT INTO vehicle_location_histories VALUES ($1, $2, now(), $3, $4) | 127.0.0.1:65088 | | NULL | preparing -(3 rows) -~~~ - -Alternatively, you can use `SHOW STATEMENTS` to receive the same response. - -### List queries on the local node - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW LOCAL STATEMENTS; -~~~ - -~~~ - query_id | node_id | session_id | user_name | start | query | client_address | application_name | distributed | phase -+----------------------------------+---------+----------------------------------+-----------+----------------------------------+------------------------------------------------------------+-----------------+------------------+-------------+-----------+ - 15f92b15bece88900000000000000001 | 1 | 15f92aefb240d2980000000000000001 | root | 2020-03-04 17:48:36.392919+00:00 | INSERT INTO user_promo_codes VALUES ($1, $2, $3, now(), 1) | 127.0.0.1:65044 | | false | executing - 15f92b15bed80a280000000000000001 | 1 | 15f92adefd48d8a00000000000000001 | root | 2020-03-04 17:48:36.393495+00:00 | SHOW LOCAL STATEMENTS | 127.0.0.1:64970 | $ cockroach sql | false | executing -(2 rows) -~~~ - -### Filter for specific queries - -You can use a [`SELECT`](select-clause.html) statement to filter the list of active queries by one or more of the [response fields](#response). - -#### Show all queries on node 2 - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW CLUSTER STATEMENTS] - WHERE node_id = 2; -~~~ - -~~~ - query_id | node_id | session_id | user_name | start | query | client_address | application_name | distributed | phase -+----------------------------------+---------+----------------------------------+-----------+----------------------------------+-----------------------------------------------------------------------+-----------------+------------------+-------------+-----------+ - 15f92b1cb931f6900000000000000002 | 2 | 15f92b10b92ed4080000000000000002 | root | 2020-03-04 17:49:06.363515+00:00 | UPSERT INTO vehicle_location_histories VALUES ($1, $2, now(), $3, $4) | 127.0.0.1:65092 | | NULL | preparing -(1 row) -~~~ - -#### Show all queries from a specific address and user - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW CLUSTER STATEMENTS] - WHERE client_address = '127.0.0.1:65196' AND user_name = 'maxroach'; -~~~ - -~~~ - query_id | node_id | session_id | user_name | start | query | client_address | application_name | distributed | phase -+----------------------------------+---------+----------------------------------+-----------+----------------------------------+-----------------------------------------------+-----------------+------------------+-------------+-----------+ - 15f92bf4b27f0b480000000000000002 | 2 | 15f92b7dc85b7ba80000000000000002 | maxroach | 2020-03-04 18:04:33.964083+00:00 | SELECT city, id FROM vehicles WHERE city = $1 | 127.0.0.1:65196 | | false | executing -(1 row) -~~~ - -#### Exclude queries from the built-in SQL client - -To exclude queries from the [built-in SQL client](cockroach-sql.html), filter for queries that do not show `$ cockroach sql` as the `application_name`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW CLUSTER STATEMENTS] - WHERE application_name != '$ cockroach sql'; -~~~ - -~~~ - query_id | node_id | session_id | user_name | start | query | client_address | application_name | distributed | phase -+----------------------------------+---------+----------------------------------+-----------+----------------------------------+-----------------------------------------------------------------------+-----------------+------------------+-------------+-----------+ - 15f92c0dd24bec200000000000000003 | 3 | 15f92b0e4ea399680000000000000003 | root | 2020-03-04 18:06:21.871708+00:00 | SELECT city, id FROM vehicles WHERE city = $1 | 127.0.0.1:65088 | | false | executing - 15f92c0dd26655d80000000000000001 | 1 | 15f92be36964ac800000000000000001 | root | 2020-03-04 18:06:21.873515+00:00 | UPSERT INTO vehicle_location_histories VALUES ($1, $2, now(), $3, $4) | 127.0.0.1:65240 | | false | executing - 15f92c0dd25882c80000000000000001 | 1 | 15f92aefb240d2980000000000000001 | root | 2020-03-04 18:06:21.872608+00:00 | UPSERT INTO vehicle_location_histories VALUES ($1, $2, now(), $3, $4) | 127.0.0.1:65044 | | false | executing - 15f92c0dd262cb980000000000000002 | 2 | 15f92b7dc85b7ba80000000000000002 | maxroach | 2020-03-04 18:06:21.873286+00:00 | SELECT city, id FROM vehicles WHERE city = $1 | 127.0.0.1:65196 | | false | executing -~~~ - -### Cancel a query - -When you see a query that is taking too long to complete, you can use the [`CANCEL QUERY`](cancel-query.html) statement to end it. - -For example, let's say you use `SHOW CLUSTER STATEMENTS` to find queries that have been running for more than 3 hours: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW CLUSTER STATEMENTS] - WHERE start < (now() - INTERVAL '3 hours'); -~~~ - -~~~ - query_id | node_id | session_id | user_name | start | query | client_address | application_name | distributed | phase -+----------------------------------+---------+----------------------------------+-----------+---------------------------------+---------------------+-----------------+------------------+-------------+-----------+ - 15f92c745fe203600000000000000001 | 1 | 15f92c63d4b393b80000000000000001 | root | 2020-03-04 18:13:42.33385+00:00 | SELECT * FROM rides | 127.0.0.1:65287 | $ cockroach sql | true | executing -(1 row) -~~~ - -To cancel this long-running query, and stop it from consuming resources, you note the `query_id` and use it with the `CANCEL QUERY` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CANCEL QUERY '15f92c745fe203600000000000000001'; -~~~ - -## See also - -- [Manage Long-Running Queries](manage-long-running-queries.html) -- [`CANCEL QUERY`](cancel-query.html) -- [`SHOW SESSIONS`](show-sessions.html) -- [`SHOW JOBS`](show-jobs.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/show-statistics.md b/src/current/v21.1/show-statistics.md deleted file mode 100644 index ce7c67b9e20..00000000000 --- a/src/current/v21.1/show-statistics.md +++ /dev/null @@ -1,71 +0,0 @@ ---- -title: SHOW STATISTICS -summary: The SHOW STATISTICS statement lists table statistics. -toc: true ---- -The `SHOW STATISTICS` [statement](sql-statements.html) lists [table statistics](create-statistics.html) used by the [cost-based optimizer](cost-based-optimizer.html). - -{{site.data.alerts.callout_info}} -[By default, CockroachDB automatically generates statistics](cost-based-optimizer.html#table-statistics) on all indexed columns, and up to 100 non-indexed columns. - - CockroachDB also automatically collects [multi-column statistics](create-statistics.html#create-statistics-on-multiple-columns) on the columns that prefix each index. -{{site.data.alerts.end}} - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_stats.html %} -
    - -## Required Privileges - -No [privileges](authorization.html#assign-privileges) are required to list table statistics. - -## Parameters - -Parameter | Description ----------------+--------------- -`table_name` | The name of the table you want to view statistics for. - -## Examples - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### List table statistics - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW STATISTICS FOR TABLE rides; -~~~ - -~~~ - statistics_name | column_names | created | row_count | distinct_count | null_count | histogram_id -------------------+---------------------------+----------------------------------+-----------+----------------+------------+--------------------- - __auto__ | {city} | 2020-08-26 16:55:24.725089+00:00 | 500 | 9 | 0 | 584550071425531905 - __auto__ | {id} | 2020-08-26 16:55:24.725089+00:00 | 500 | 500 | 0 | 584550071432740865 - __auto__ | {city,id} | 2020-08-26 16:55:24.725089+00:00 | 500 | 500 | 0 | NULL - __auto__ | {rider_id} | 2020-08-26 16:55:24.725089+00:00 | 500 | 50 | 0 | 584550071446732801 - __auto__ | {city,rider_id} | 2020-08-26 16:55:24.725089+00:00 | 500 | 50 | 0 | NULL - __auto__ | {vehicle_city} | 2020-08-26 16:55:24.725089+00:00 | 500 | 9 | 0 | 584550071461019649 - __auto__ | {vehicle_id} | 2020-08-26 16:55:24.725089+00:00 | 500 | 15 | 0 | 584550071467966465 - __auto__ | {vehicle_city,vehicle_id} | 2020-08-26 16:55:24.725089+00:00 | 500 | 15 | 0 | NULL - __auto__ | {start_address} | 2020-08-26 16:55:24.725089+00:00 | 500 | 500 | 0 | 584550071482122241 - __auto__ | {end_address} | 2020-08-26 16:55:24.725089+00:00 | 500 | 500 | 0 | 584550071489167361 - __auto__ | {start_time} | 2020-08-26 16:55:24.725089+00:00 | 500 | 30 | 0 | 584550071496671233 - __auto__ | {end_time} | 2020-08-26 16:55:24.725089+00:00 | 500 | 367 | 0 | 584550071504437249 - __auto__ | {revenue} | 2020-08-26 16:55:24.725089+00:00 | 500 | 100 | 0 | 584550071512137729 -(13 rows) -~~~ - -### Delete statistics - -{% include {{ page.version.version }}/misc/delete-statistics.md %} - -## See Also - -- [Cost-Based Optimizer](cost-based-optimizer.html) -- [`CREATE STATISTICS`](create-statistics.html) -- [`CREATE TABLE`](create-table.html) -- [`INSERT`](insert.html) -- [`IMPORT`](import.html) -- [SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/show-tables.md b/src/current/v21.1/show-tables.md deleted file mode 100644 index 1b3bf4f5fd2..00000000000 --- a/src/current/v21.1/show-tables.md +++ /dev/null @@ -1,273 +0,0 @@ ---- -title: SHOW TABLES -summary: The SHOW TABLES statement lists the tables in a schema or database. -keywords: reflection -toc: true ---- - -The `SHOW TABLES` [statement](sql-statements.html) lists the schema, table name, table type, owner, and estimated row count for the tables or [views](views.html) in a schema or database. - -{{site.data.alerts.callout_info}} -While a table or view is being [dropped](drop-table.html), `SHOW TABLES` will list the object with a `(dropped)` suffix. -{{site.data.alerts.end}} - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_tables.html %} -
    - -## Required privileges - -The `CONNECT` [privilege](authorization.html#assign-privileges) on the database of the concerned table is required to list it with SHOW TABLES. - -## Parameters - -Parameter | Description -----------|------------ -`database_name` | The name of the database for which to show tables. -`schema_name` | The name of the schema for which to show tables. - -When a `database_name` and `schema_name` are omitted, the tables of the [current schema](sql-name-resolution.html#current-schema) in the [current database](sql-name-resolution.html#current-database) are listed. - -`SHOW TABLES` will attempt to find a schema with the specified name first. If that fails, it will try to find a database with that name instead, and list the tables of its `public` schema. For more details, see [Name Resolution](sql-name-resolution.html). - -## Performance - -To optimize the performance of the `SHOW TABLES` statement, you can do the following: - -- Disable table row-count estimation by setting the `sql.show_tables.estimated_row_count.enabled` [cluster setting](cluster-settings.html) to `false` before executing a `SHOW TABLES` statement. -- Avoid running `SHOW TABLES` on databases with a large number of tables (e.g., more than 10,000 tables). - -## Examples - -{% include {{page.version.version}}/sql/movr-statements-nodes.md %} - -### Show tables in the current database - -`SHOW TABLES` uses the [current schema](sql-name-resolution.html#current-schema) `public` set by default in `search_path`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES; -~~~ - -~~~ - schema_name | table_name | type | owner | estimated_row_count | locality ---------------+----------------------------+-------+-------+---------------------+----------- - public | promo_codes | table | demo | 0 | NULL - public | rides | table | demo | 0 | NULL - public | user_promo_codes | table | demo | 0 | NULL - public | users | table | demo | 0 | NULL - public | vehicle_location_histories | table | demo | 0 | NULL - public | vehicles | table | demo | 0 | NULL -(6 rows) -~~~ - -Alternatively, within the built-in SQL shell, you can use the `\dt` [shell command](cockroach-sql.html#commands): - -{% include_cached copy-clipboard.html %} -~~~ sql -> \dt -~~~ - -~~~ - schema_name | table_name | type | owner | estimated_row_count | locality ---------------+----------------------------+-------+-------+---------------------+----------- - public | promo_codes | table | demo | 0 | NULL - public | rides | table | demo | 0 | NULL - public | user_promo_codes | table | demo | 0 | NULL - public | users | table | demo | 0 | NULL - public | vehicle_location_histories | table | demo | 0 | NULL - public | vehicles | table | demo | 0 | NULL -(6 rows) -~~~ - -### Show tables in a different schema - -You can show the tables in schemas other than the current schema. You can also show the schema by table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM movr.information_schema; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM information_schema; -~~~ - -Because `movr` is the current database, these statements return the same output: - -~~~ - schema_name | table_name | type | owner | estimated_row_count | locality ----------------------+---------------------------------------+-------+-------+---------------------+----------- - information_schema | administrable_role_authorizations | table | NULL | NULL | NULL - information_schema | applicable_roles | table | NULL | NULL | NULL - information_schema | character_sets | table | NULL | NULL | NULL - information_schema | check_constraints | table | NULL | NULL | NULL -... -(27 rows) -~~~ - -### Show tables in a different database - -You can also show tables from a different database. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM system.public; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM system; -~~~ - -Because `public` is the current schema, these statements return the same output: - -~~~ - schema_name | table_name | type | owner | estimated_row_count | locality ---------------+---------------------------------+-------+-------+---------------------+----------- - public | comments | table | NULL | 0 | NULL - public | descriptor | table | NULL | 0 | NULL - public | eventlog | table | NULL | 0 | NULL - public | jobs | table | NULL | 0 | NULL -... -(30 rows) -~~~ - -### Show user-defined tables with comments - -You can use [`COMMENT ON`](comment-on.html) to add comments on a table. - -{% include_cached copy-clipboard.html %} -~~~ sql -> COMMENT ON TABLE users IS 'This table contains information about users.'; -~~~ - -To view a table's comments: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM movr WITH COMMENT; -~~~ - -~~~ - schema_name | table_name | type | owner | estimated_row_count | locality | comment ---------------+----------------------------+-------+-------+---------------------+----------+----------------------------------------------- - public | promo_codes | table | demo | 1000 | NULL | - public | rides | table | demo | 500 | NULL | - public | user_promo_codes | table | demo | 0 | NULL | - public | users | table | demo | 50 | NULL | This table contains information about users. - public | vehicle_location_histories | table | demo | 1000 | NULL | - public | vehicles | table | demo | 15 | NULL | -(6 rows) -~~~ - - You can also view comments on a table with [`SHOW CREATE`](show-create.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE users; -~~~ - -~~~ - table_name | create_statement --------------+--------------------------------------------------------------------------- - users | CREATE TABLE users ( - | id UUID NOT NULL, - | city VARCHAR NOT NULL, - | name VARCHAR NULL, - | address VARCHAR NULL, - | credit_card VARCHAR NULL, - | CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - | FAMILY "primary" (id, city, name, address, credit_card) - | ); - | COMMENT ON TABLE users IS 'This table contains information about users.' -(1 row) -~~~ - -For more information, see [`COMMENT ON`](comment-on.html). - -### Show virtual tables with comments - -The virtual tables in the `pg_catalog`, `information_schema`, and `crdb_internal` schemas contain useful comments, often with links to further documentation. - -To view virtual tables with comments and documentation links, use `SHOW TABLES FROM WITH COMMENT`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM information_schema WITH COMMENT; -~~~ - -~~~ - schema_name | table_name | type | owner | estimated_row_count | locality | comment ----------------------+---------------------------------------+-------+-------+---------------------+----------+------------------------------------------------------------------------------------------------------------------------------------- - information_schema | administrable_role_authorizations | table | NULL | NULL | NULL | roles for which the current user has admin option - | | | | | | https://www.cockroachlabs.com/docs/v21.1/information-schema.html#administrable_role_authorizations - | | | | | | https://www.postgresql.org/docs/9.5/infoschema-administrable-role-authorizations.html - information_schema | applicable_roles | table | NULL | NULL | NULL | roles available to the current user - | | | | | | https://www.cockroachlabs.com/docs/v21.1/information-schema.html#applicable_roles - | | | | | | https://www.postgresql.org/docs/9.5/infoschema-applicable-roles.html - information_schema | character_sets | table | NULL | NULL | NULL | character sets available in the current database - | | | | | | https://www.cockroachlabs.com/docs/v21.1/information-schema.html#character_sets - | | | | | | https://www.postgresql.org/docs/9.5/infoschema-character-sets.html - information_schema | check_constraints | table | NULL | NULL | NULL | check constraints - | | | | | | https://www.cockroachlabs.com/docs/v21.1/information-schema.html#check_constraints - | | | | | | https://www.postgresql.org/docs/9.5/infoschema-check-constraints.html -... -(27 rows) -~~~ - -### Show locality of tables - -For [multi-region](multiregion-overview.html) tables, you can display the locality of each table using the `SHOW TABLES` command. - -{% include enterprise-feature.md %} - -First, [set the primary region](set-primary-region.html) on `movr` to `us-east`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER DATABASE movr SET PRIMARY REGION "us-east"; -~~~ - -All tables will be [`REGIONAL BY TABLE`](set-locality.html#set-the-table-locality-to-regional-by-table) in the primary region by default. - -Next, configure the `users` table to be [`REGIONAL BY ROW`](set-locality.html#set-the-table-locality-to-regional-by-row): - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE users SET LOCALITY REGIONAL BY ROW; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES; -~~~ - -~~~ - schema_name | table_name | type | owner | estimated_row_count | locality ---------------+----------------------------+-------+-------+---------------------+-------------------------------------- - public | promo_codes | table | demo | 1000 | REGIONAL BY TABLE IN PRIMARY REGION - public | rides | table | demo | 500 | REGIONAL BY TABLE IN PRIMARY REGION - public | user_promo_codes | table | demo | 0 | REGIONAL BY TABLE IN PRIMARY REGION - public | users | table | demo | 50 | REGIONAL BY ROW - public | vehicle_location_histories | table | demo | 1000 | REGIONAL BY TABLE IN PRIMARY REGION - public | vehicles | table | demo | 15 | REGIONAL BY TABLE IN PRIMARY REGION -(6 rows) -~~~ - -{{site.data.alerts.callout_info}} -Locality information for tables is also available in the `locality` column within the [`crdb_internal.tables`](crdb-internal.html) table. -{{site.data.alerts.end}} - -## See also - -- [`SHOW DATABASES`](show-databases.html) -- [`SHOW SCHEMAS`](show-schemas.html) -- [`CREATE TABLE`](create-table.html) -- [`CREATE VIEW`](create-view.html) -- [`COMMENT ON`](comment-on.html) -- [Information Schema](information-schema.html) diff --git a/src/current/v21.1/show-trace.md b/src/current/v21.1/show-trace.md deleted file mode 100644 index b9abe139fd9..00000000000 --- a/src/current/v21.1/show-trace.md +++ /dev/null @@ -1,216 +0,0 @@ ---- -title: SHOW TRACE FOR SESSION -summary: The SHOW TRACE FOR SESSION statement returns details about how CockroachDB executed a statement or series of statements recorded during a session. -toc: true ---- - -The `SHOW TRACE FOR SESSION` [statement](sql-statements.html) returns details about how CockroachDB executed a statement or series of statements recorded during a session. These details include messages and timing information from all nodes involved in the execution, providing visibility into the actions taken by CockroachDB across all of its software layers. - -You can use `SHOW TRACE FOR SESSION` to debug why a query is not performing as expected, to add more information to bug reports, or to generally learn more about how CockroachDB works. - -{{site.data.alerts.callout_info}} -Statement traces can be obtained in plaintext, JSON, and [Jaeger-compatible](query-behavior-troubleshooting.html#visualize-statement-traces-in-jaeger) formats by activating [statement diagnostics](ui-statements-page.html#diagnostics) on the DB Console Statements Page or running [`EXPLAIN ANALYZE (DEBUG)`](explain-analyze.html#debug-option). -{{site.data.alerts.end}} - -## Usage overview - -`SHOW TRACE FOR SESSION` returns messages and timing information for all statements recorded during a session. It's important to note the following: - -- `SHOW TRACE FOR SESSION` only returns the most recently recorded traces, or for a currently active recording of traces. - - To start recording traces during a session, enable the `tracing` session variable via [`SET tracing = on;`](set-vars.html#set-tracing). - - To stop recording traces during a session, disable the `tracing` session variable via [`SET tracing = off;`](set-vars.html#set-tracing). - -- Recording traces during a session does not effect the execution of any statements traced. This means that errors encountered by statements during a recording are returned to clients. CockroachDB will [automatically retry](transactions.html#automatic-retries) individual statements (considered implicit transactions) and multi-statement transactions sent as a single batch when [retry errors](transactions.html#error-handling) are encountered due to contention. Also, clients will receive retry errors required to handle [client-side retries](transactions.html#client-side-intervention). As a result, traces of all transaction retries will be captured during a recording. - - -## Required privileges - -For `SHOW TRACE FOR SESSION`, no privileges are required. - -## Syntax - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_trace.html %} -
    - -## Parameters - -Parameter | Description -----------|------------ -`COMPACT` | If specified, fewer columns are returned by the statement. See [Response](#response) for more details. -`KV` | If specified, the returned messages are restricted to those describing requests to and responses from the underlying key-value [storage layer](architecture/storage-layer.html), including per-result-row messages.

    For `SHOW KV TRACE FOR SESSION`, per-result-row messages are included only if the session was/is recording with `SET tracing = kv;`. - -## Trace description - -CockroachDB's definition of a "trace" is a specialization of [OpenTracing's](https://opentracing.io/docs/overview/what-is-tracing/#what-is-opentracing) definition. Internally, CockroachDB uses OpenTracing libraries for tracing, which also means that -it can be easily integrated with OpenTracing-compatible trace collectors; for example, [Jaeger](query-behavior-troubleshooting.html#visualize-statement-traces-in-jaeger), Lightstep, and Zipkin are already supported. - -Concept | Description ---------|------------ -**trace** | Information about the sub-operations performed as part of a high-level operation (a query or a transaction). This information is internally represented as a tree of "spans", with a special "root span" representing a whole SQL transaction in the case of `SHOW TRACE FOR SESSION`. -**span** | A named, timed operation that describes a contiguous segment of work in a trace. Each span links to "child spans", representing sub-operations; their children would be sub-sub-operations of the grandparent span, etc.

    Different spans can represent (sub-)operations that executed either sequentially or in parallel with respect to each other. (This possibly-parallel nature of execution is one of the important things that a trace is supposed to describe.) The operations described by a trace may be _distributed_, that is, different spans may describe operations executed by different nodes. -**message** | A string with timing information. Each span can contain a list of these. They are produced by CockroachDB's logging infrastructure and are the same messages that can be found in node [log files](logging-overview.html) except that a trace contains message across all severity levels, whereas log files, by default, do not. Thus, a trace is much more verbose than logs but only contains messages produced in the context of one particular traced operation. - -To further clarify these concepts, let's look at a visualization of a trace for one statement. This particular trace is [visualized by Jaeger](query-behavior-troubleshooting.html#visualize-statement-traces-in-jaeger). The image shows spans and log messages. You can see names of operations and sub-operations, along with parent-child relationships and timing information, and it's easy to see which operations are executed in parallel. - -Jaeger Trace Log Messages - -## Response - -{{site.data.alerts.callout_info}}The format of the SHOW TRACE FOR SESSION response may change in future versions.{{site.data.alerts.end}} - -CockroachDB outputs traces in linear tabular format. Each result row represents either a span start (identified by the `=== SPAN START: ===` message) or a log message from a span. Rows are generally listed in their timestamp order (i.e., the order in which the events they represent occurred) with the exception that messages from child spans are interleaved in the parent span according to their timing. Messages from sibling spans, however, are not interleaved with respect to one another. - -The following diagram shows the order in which messages from different spans would be interleaved in an example trace. Each box is a span; inner-boxes are child spans. The numbers indicate the order in which the log messages would appear in the virtual table. - -~~~ - +-----------------------+ - | 1 | - | +-------------------+ | - | | 2 | | - | | +----+ | | - | | | | +----+ | | - | | | 3 | | 4 | | | - | | | | | | 5 | | - | | | | | | ++ | | - | | +----+ | | | | - | | +----+ | | - | | 6 | | - | +-------------------+ | - | 7 | - +-----------------------+ -~~~ - -Each row contains the following columns: - -Column | Type | Description --------|------|------------ -`timestamp` | timestamptz | The absolute time when the message occurred. -`age` | interval | The age of the message relative to the beginning of the trace (i.e., the beginning of the statement execution in the case of `SHOW TRACE FOR ` and the beginning of the recording in the case of `SHOW TRACE FOR SESSION`. -`message` | string | The log message. -`tag` | string | Meta-information about the message's context. This is the same information that appears in the beginning of log file messages in between square brackets (e.g, `[client=[::1]:49985,user=root,n1]`). -`location` | string | The file:line location of the line of code that produced the message. Only some of the messages have this field set; it depends on specifically how the message was logged. The `--vmodule` flag passed to the node producing the message also affects what rows get this field populated. Generally, if `--vmodule==` is specified, messages produced by that file will have the field populated. -`operation` | string | The name of the operation (or sub-operation) on whose behalf the message was logged. -`span` | int | The index of the span within the virtual list of all spans if they were ordered by the span's start time. - -{{site.data.alerts.callout_info}}If the COMPACT keyword was specified, only the age, message, tag and operation columns are returned. In addition, the value of the location columns is prepended to message.{{site.data.alerts.end}} - -## Examples - -### Trace a session - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET tracing = on; -~~~ - -~~~ -SET TRACING -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TRACE FOR SESSION; -~~~ - -~~~ - timestamp | age | message | tag | location | operation | span ---------------------------------------+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------+--------------------------------------+-------------------+------- - 2021-04-13 19:56:00.201326+00:00:00 | 00:00:00 | === SPAN START: session recording === | | | session recording | 0 - 2021-04-13 19:56:00.20135+00:00:00 | 00:00:00.000024 | === SPAN START: sync === | | | sync | 1 - 2021-04-13 19:56:00.2014+00:00:00 | 00:00:00.000074 | [NoTxn pos:13] executing Sync | [n1,client=127.0.0.1:61235,hostssl,user=root] | sql/conn_executor.go:1478 | sync | 1 - 2021-04-13 19:56:00.20153+00:00:00 | 00:00:00.000204 | === SPAN START: exec stmt === | | | exec stmt | 2 - 2021-04-13 19:56:00.201562+00:00:00 | 00:00:00.000236 | [NoTxn pos:14] executing ExecStmt: SHOW LAST QUERY STATISTICS | [n1,client=127.0.0.1:61235,hostssl,user=root] | sql/conn_executor.go:1478 | exec stmt | 2 - 2021-04-13 19:56:00.201583+00:00:00 | 00:00:00.000257 | executing: SHOW LAST QUERY STATISTICS in state: NoTxn | [n1,client=127.0.0.1:61235,hostssl,user=root] | sql/conn_executor_exec.go:75 | exec stmt | 2 - ... - - (68 rows) -~~~ - -### Trace conflicting transactions - -In this example, we use two terminals concurrently to generate conflicting transactions. - -1. In terminal 1, create a table: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE TABLE t (k INT); - ~~~ - -2. Still in terminal 1, open a transaction and perform a write without closing the transaction: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > BEGIN; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > INSERT INTO t VALUES (1); - ~~~ - - Press enter one more time to send these statements to the server. - -3. In terminal 2, turn tracing on: - - ~~~ - > SET tracing = on; - ~~~ - -4. Still in terminal 2, execute a conflicting read: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SELECT * FROM t; - ~~~ - - You'll see that this statement is blocked until the transaction in terminal 1 finishes. - -4. Back in terminal 1, finish the transaction: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > COMMIT; - ~~~ - -5. In terminal 2, you'll see the completed read: - - ~~~ - k - +---+ - 1 - (1 row) - ~~~ - -6. Still in terminal 2, stop tracing and then view the completed trace: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET tracing = off; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW TRACE FOR SESSION; - ~~~ - - ~~~ - timestamp | age | message | tag | location | operation | span - --------------------------------------+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------+-----------------------------------------------------------+-----------------------------------+------- - 2021-04-13 18:55:43.094235+00:00:00 | 00:00:00 | === SPAN START: session recording === | | | session recording | 0 - 2021-04-13 18:55:43.094244+00:00:00 | 00:00:00.000009 | === SPAN START: sync === | | | sync | 1 - 2021-04-13 18:55:43.094286+00:00:00 | 00:00:00.000051 | [NoTxn pos:43] executing Sync | [n1,client=127.0.0.1:60668,hostssl,user=root] | sql/conn_executor.go:1478 | sync | 1 - 2021-04-13 18:55:43.094436+00:00:00 | 00:00:00.000201 | === SPAN START: exec stmt === | | | exec stmt | 2 - 2021-04-13 18:55:43.094473+00:00:00 | 00:00:00.000238 | [NoTxn pos:44] executing ExecStmt: SHOW LAST QUERY STATISTICS | [n1,client=127.0.0.1:60668,hostssl,user=root] | sql/conn_executor.go:1478 | exec stmt | 2 - 2021-04-13 18:55:43.094486+00:00:00 | 00:00:00.000251 | executing: SHOW LAST QUERY STATISTICS in state: NoTxn | [n1,client=127.0.0.1:60668,hostssl,user=root] | sql/conn_executor_exec.go:75 | exec stmt | 2 - 2021-04-13 18:55:43.094497+00:00:00 | 00:00:00.000262 | === SPAN START: sync === | | | sync | 3 - ... - (200 rows) - ~~~ - - -## See also - -- [`EXPLAIN`](explain.html) -- [`SET (session settings)`](set-vars.html) diff --git a/src/current/v21.1/show-transactions.md b/src/current/v21.1/show-transactions.md deleted file mode 100644 index ae1aac159f9..00000000000 --- a/src/current/v21.1/show-transactions.md +++ /dev/null @@ -1,115 +0,0 @@ ---- -title: SHOW TRANSACTIONS -summary: The SHOW TRANSACTIONS statement lists all currently active transactions across the cluster or on the gateway node. -toc: true ---- - - The `SHOW TRANSACTIONS` [statement](sql-statements.html) lists details about currently active transactions, including: - -- The node running the transaction -- The application that initiated the transaction -- The number of statements that have been executed on the transaction -- The number of times the transaction was retried (both manually and by the SQL executor) - -These details let you monitor the overall state of transactions and identify those that may need further investigation or adjustment. - -## Required privileges - -No [privileges](authorization.html#assign-privileges) are required to execute this statement. However, note that non-`admin` users see only their own currently active transactions, whereas the `admin` users see all users' currently active transactions. - -## Syntax - -~~~ -SHOW [ALL] [CLUSTER | LOCAL] TRANSACTIONS -~~~ - -- To list the active transactions across all nodes of the cluster, use `SHOW TRANSACTIONS` or `SHOW CLUSTER TRANSACTIONS`. -- To list the active transactions just on the gateway node, use `SHOW LOCAL TRANSACTIONS`. -- To list internal transactions (that are issued by the database itself), use `SHOW ALL .. TRANSACTIONS`. - -## Response - -The following fields are returned for each transaction: - - Field | Description --------------------+--------------------------------------- -`node_id` | The ID of the node running the transaction. -`txn_id` | The ID of the transaction. -`application_name` | The [application name](set-vars.html#supported-variables) specified by the client, if any. For transactions from the [built-in SQL client](cockroach-sql.html), this will be `$ cockroach sql`. -`num_stmts` | The number of statements that have been executed on the transaction. -`num_retries` | The number of times that the transaction was retried. -`num_auto_retries` | The number of times that the transaction was automatically retried by the SQL executor. - -## Examples - -### List active transactions across the cluster - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CLUSTER TRANSACTIONS; -~~~ - -~~~ - node_id | txn_id | application_name | num_stmts | num_retries | num_auto_retries -----------+--------------------------------------+------------------+-----------+-------------+------------------- - 1 | 1f02ca1c-46c8-491b-8048-daa6b38fc587 | $ cockroach sql | 1 | 0 | 0 - 2 | 3af3f2e3-d0ed-4fd4-886b-1f5fc67b397d | movr | 1 | 0 | 0 -(2 rows) -~~~ - -Alternatively, you can use `SHOW TRANSACTIONS` to receive the same response. - -### List active transactions on the gateway node - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW LOCAL TRANSACTIONS; -~~~ - -~~~ - node_id | txn_id | application_name | num_stmts | num_retries | num_auto_retries -----------+--------------------------------------+------------------+-----------+-------------+------------------- - 1 | 1f02ca1c-46c8-491b-8048-daa6b38fc587 | $ cockroach sql | 1 | 0 | 0 -(1 row) -~~~ - -### Filter for specific transactions - -You can use a [`SELECT`](select-clause.html) statement to filter the list of currently active transactions by one or more of the [response fields](#response). - -#### Show transactions associated with a specific application - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW CLUSTER TRANSACTIONS] WHERE application_name = 'movr'; - -~~~ - -~~~ - node_id | txn_id | application_name | num_stmts | num_retries | num_auto_retries -----------+--------------------------------------+------------------+-----------+-------------+------------------- - 1 | 59891000-260a-44f6-abf9-59c94fefc081 | movr | 1 | 0 | 0 -(1 row) -~~~ - -#### Exclude transactions from the built-in SQL client - -To exclude transactions from the [built-in SQL client](cockroach-sql.html), filter for transactions that do not show `$ cockroach sql` as the `application_name`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM [SHOW CLUSTER TRANSACTIONS] - WHERE application_name != '$ cockroach sql'; -~~~ - -~~~ - node_id | txn_id | application_name | num_stmts | num_retries | num_auto_retries -----------+--------------------------------------+------------------+-----------+-------------+------------------- - 1 | 8851060b-5e7c-4402-a8cd-487baa63e02e | movr | 1 | 0 | 0 -(1 row) -~~~ - -## See also - -- [Transactions](transactions.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/show-types.md b/src/current/v21.1/show-types.md deleted file mode 100644 index 3dff66a6162..00000000000 --- a/src/current/v21.1/show-types.md +++ /dev/null @@ -1,55 +0,0 @@ ---- -title: SHOW TYPES -summary: The SHOW TYPES statement lists the user-defined data types in a database. -toc: true ---- - - The `SHOW TYPES` statement lists the user-defined [data types](data-types.html) in the current database. - -{{site.data.alerts.callout_info}} -CockroachDB currently only supports [enumerated user-defined types](enum.html). As a result, [`SHOW ENUMS`](show-enums.html) and `SHOW TYPES` return the same results. -{{site.data.alerts.end}} - -## Syntax - -~~~ -SHOW TYPES -~~~ - -## Required privileges - -The `SELECT` [privilege](authorization.html#assign-privileges) on the database is required to list any user-defined types in the database. - -## Examples - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TYPE weekday AS ENUM ('monday', 'tuesday', 'wednesday', 'thursday', 'friday'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TYPE weekend AS ENUM ('sunday', 'saturday'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TYPES; -~~~ - -~~~ - schema | name | value ----------+---------+------------------------------------------- - public | weekday | monday|tuesday|wednesday|thursday|friday - public | weekend | sunday|saturday -(2 rows) -~~~ - - -## See also - -- [`ENUM`](enum.html) -- [Data types](data-types.html) -- [`CREATE TYPE`](create-type.html) -- [`ALTER TYPE`](alter-type.html) -- [`DROP TYPE`](drop-type.html) diff --git a/src/current/v21.1/show-users.md b/src/current/v21.1/show-users.md deleted file mode 100644 index fe431d6039d..00000000000 --- a/src/current/v21.1/show-users.md +++ /dev/null @@ -1,60 +0,0 @@ ---- -title: SHOW USERS -summary: The SHOW USERS statement lists the users for all databases. -toc: true ---- - -The `SHOW USERS` [statement](sql-statements.html) lists the users for all databases. - -{{site.data.alerts.callout_info}} - Since the keywords `ROLES` and `USERS` can now be used interchangeably in SQL statements for enhanced Postgres compatibility, `SHOW USERS` is now an alias for [`SHOW ROLES`](show-roles.html). -{{site.data.alerts.end}} - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_users.html %} -
    - -## Required privileges - -The user must have the [`SELECT`](select-clause.html) [privilege](authorization.html#assign-privileges) on the `system.users` and `system.role_members` tables. - -## Example - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW USERS; -~~~ - -~~~ - username | options | member_of ------------+------------+------------ - admin | CREATEROLE | {} - carl | NOLOGIN | {} - petee | | {} - root | CREATEROLE | {admin} -(4 rows) -~~~ - -Alternatively, within the built-in SQL shell, you can use the `\du` [shell command](cockroach-sql.html#commands): - -{% include_cached copy-clipboard.html %} -~~~ sql -> \du -~~~ - -~~~ - username | options | member_of ------------+------------+------------ - admin | CREATEROLE | {} - carl | NOLOGIN | {} - petee | | {} - root | CREATEROLE | {admin} -(4 rows) -~~~ - -## See also - -- [`CREATE USER`](create-user.html) -- [Manage Users](authorization.html#create-and-manage-users) diff --git a/src/current/v21.1/show-vars.md b/src/current/v21.1/show-vars.md deleted file mode 100644 index 4bd8d43986d..00000000000 --- a/src/current/v21.1/show-vars.md +++ /dev/null @@ -1,87 +0,0 @@ ---- -title: SHOW (session settings) -summary: The SHOW statement displays the current settings for the client session. -toc: true ---- - -The `SHOW` [statement](sql-statements.html) can display the value of either one or all of -the session setting variables. Some of these can also be configured via [`SET`](set-vars.html). - -## Required privileges - -No [privileges](authorization.html#assign-privileges) are required to display the session settings. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_var.html %} -
    - -{{site.data.alerts.callout_info}}The SHOW statement for session settings is unrelated to the other SHOW statements: SHOW CLUSTER SETTING, SHOW CREATE, SHOW USERS, SHOW DATABASES, SHOW COLUMNS, SHOW GRANTS, and SHOW CONSTRAINTS.{{site.data.alerts.end}} - -## Parameters - -The `SHOW ` statement accepts a single parameter: the variable name. - -The variable name is case insensitive. It may be enclosed in double quotes; this is useful if the variable name itself contains spaces. - -### Supported variables - -{% include {{ page.version.version }}/misc/session-vars.html %} - -For session variables on experimental features, see [Experimental Features](experimental-features.html). - -Special syntax cases supported for compatibility: - - Syntax | Equivalent to ---------|--------------- -`SHOW TRANSACTION PRIORITY` | `SHOW "transaction priority"` -`SHOW TRANSACTION ISOLATION LEVEL` | `SHOW "transaction isolation level"` -`SHOW TIME ZONE` | `SHOW "timezone"` -`SHOW TRANSACTION STATUS` | `SHOW "transaction status"` - -## Examples - -### Showing the value of a single session variable - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW DATABASE; -~~~ - -~~~ - database ------------- - movr -(1 row) -~~~ - -### Showing the value of all session variables - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW ALL; -~~~ - -~~~ - variable | value -----------------------+------------------- - application_name | $ cockroach demo - bytea_output | hex - client_encoding | UTF8 - client_min_messages | notice -... -~~~ - -## See also - -- [`SET` (session variable)](set-vars.html) -- [Transactions](transactions.html), including [Priority levels](transactions.html#transaction-priorities) -- [`SHOW CLUSTER SETTING`](show-cluster-setting.html) -- [`SHOW COLUMNS`](show-columns.html) -- [`SHOW CONSTRAINTS`](show-constraints.html) -- [`SHOW CREATE`](show-create.html) -- [`SHOW DATABASES`](show-databases.html) -- [`SHOW GRANTS`](show-grants.html) -- [`SHOW INDEX`](show-index.html) -- [`SHOW USERS`](show-users.html) diff --git a/src/current/v21.1/show-zone-configurations.md b/src/current/v21.1/show-zone-configurations.md deleted file mode 100644 index 7f450d5d744..00000000000 --- a/src/current/v21.1/show-zone-configurations.md +++ /dev/null @@ -1,96 +0,0 @@ ---- -title: SHOW ZONE CONFIGURATIONS -summary: Use the SHOW ZONE CONFIGURATIONS statement to list details about existing replication zones. -toc: true ---- - -Use the `SHOW ZONE CONFIGURATIONS` [statement](sql-statements.html) to view details about existing [replication zones](configure-replication-zones.html). - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/show_zone.html %} -
    - -## Required privileges - -No [privileges](authorization.html#assign-privileges) are required to list replication zones. - -## Parameters - -Parameter | Description -----------|------------ -`zone_name` | The name of the system [range](architecture/overview.html#glossary) for which to show [replication zone configurations](configure-replication-zones.html). -`database_name` | The name of the [database](create-database.html) for which to show [replication zone configurations](configure-replication-zones.html). -`table_name` | The name of the [table](create-table.html) for which to show [replication zone configurations](configure-replication-zones.html). -`partition_name` | The name of the [partition](partitioning.html) for which to show [replication zone configurations](configure-replication-zones.html). -`index_name` | The name of the [index](indexes.html) for which to show [replication zone configurations](configure-replication-zones.html). - -## Examples - -{% include {{ page.version.version }}/sql/movr-statements-geo-partitioned-replicas.md %} - -### View all replication zones - -{% include {{ page.version.version }}/zone-configs/view-all-replication-zones.md %} - -### View the default replication zone for the cluster - -{% include {{ page.version.version }}/zone-configs/view-the-default-replication-zone.md %} - -### View the replication zone for a database - -{% include {{ page.version.version }}/zone-configs/create-a-replication-zone-for-a-database.md %} - -### View the replication zone for a table - -{% include {{ page.version.version }}/zone-configs/create-a-replication-zone-for-a-table.md %} - -You can also use [`SHOW CREATE TABLE`](show-create.html) to view zone configurations for a table. If a table is partitioned, but no zones are configured, the `SHOW CREATE TABLE` output includes a warning. - -### View the replication zone for an index - -To control replication for a specific table, use the `ALTER INDEX ... CONFIGURE ZONE` statement to define the relevant values (other values will be inherited from the parent zone): - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER INDEX vehicles@vehicles_auto_index_fk_city_ref_users CONFIGURE ZONE USING num_replicas = 5, gc.ttlseconds = 100000; -~~~ - -~~~ -CONFIGURE ZONE 1 -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW ZONE CONFIGURATION FROM INDEX vehicles@vehicles_auto_index_fk_city_ref_users; -~~~ - -~~~ - target | raw_config_sql --------------------------------------------------------+---------------------------------------------------------------------------------- - INDEX vehicles@vehicles_auto_index_fk_city_ref_users | ALTER INDEX vehicles@vehicles_auto_index_fk_city_ref_users CONFIGURE ZONE USING - | range_min_bytes = 134217728, - | range_max_bytes = 536870912, - | gc.ttlseconds = 100000, - | num_replicas = 5, - | constraints = '[]', - | lease_preferences = '[]' -(1 row) -~~~ - -### View the replication zone for a partition - -{% include {{page.version.version}}/sql/use-multiregion-instead-of-partitioning.md %} - -{% include {{ page.version.version }}/zone-configs/create-a-replication-zone-for-a-table-partition.md %} - -## See also - -- [Configure Replication Zones](configure-replication-zones.html) -- [`CONFIGURE ZONE`](configure-zone.html) -- [`ALTER DATABASE`](alter-database.html) -- [`ALTER INDEX`](alter-index.html) -- [`ALTER RANGE`](alter-range.html) -- [`ALTER TABLE`](alter-table.html) -- [SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/simulate-a-multi-region-cluster-on-localhost.md b/src/current/v21.1/simulate-a-multi-region-cluster-on-localhost.md deleted file mode 100644 index e13d04cbf89..00000000000 --- a/src/current/v21.1/simulate-a-multi-region-cluster-on-localhost.md +++ /dev/null @@ -1,101 +0,0 @@ ---- -title: Simulate a Multi-Region Cluster on localhost (Insecure) -summary: Run a simulated multi-region CockroachDB cluster locally using cockroach demo. -toc: true ---- - -{% include_cached new-in.html version="v21.1" %} Once you've [installed CockroachDB](install-cockroachdb.html), it's simple to simulate multi-region cluster on your local machine using [`cockroach demo`](cockroach-demo.html). This is a useful way to start playing with the [improved multi-region abstractions](multiregion-overview.html) provided by CockroachDB. - -{{site.data.alerts.callout_info}} -[`cockroach demo`](cockroach-demo.html) is not suitable for production deployments. Additionally, simulating multiple geographically distributed nodes on a single host is not representative of the [performance you should expect](frequently-asked-questions.html#single-row-perf) of a production deployment. For instructions showing how to do production multi-region deployments, see [Orchestrate CockroachDB Across Multiple Kubernetes Clusters](orchestrate-cockroachdb-with-kubernetes-multi-cluster.html) and [Deploy a Global, Serverless Application](movr-flask-deployment.html). Also be sure to review the [Production Checklist](recommended-production-settings.html). -{{site.data.alerts.end}} - -## Before you begin - -- Make sure you have already [installed CockroachDB](install-cockroachdb.html). - -## Step 1. Start the cluster - -{% include {{page.version.version}}/sql/start-a-multi-region-demo-cluster.md %} - -## Step 2. Enter SQL statements at the prompt - -Now that your simulated multi-region cluster is running, you are presented with a SQL prompt: - -~~~ -root@127.0.0.1:26257/defaultdb> -~~~ - -You can run some basic [CockroachDB SQL statements](learn-cockroachdb-sql.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE DATABASE bank; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> USE bank; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE bank.accounts (id INT PRIMARY KEY, balance DECIMAL); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO bank.accounts VALUES (1, 1000.50); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM bank.accounts; -~~~ - -~~~ - id | balance -+----+---------+ - 1 | 1000.50 -(1 row) -~~~ - -## Step 3. View cluster health and performance in the DB Console - -The [DB Console](ui-overview.html) gives you insight into the overall health of your cluster, as well as the performance of any [client workloads](cockroach-workload.html). - -To verify that the 9 nodes you specified are up and running as expected, go to the [**Cluster Overview**](ui-cluster-overview-page.html) at http://localhost:8080/#/overview/list. - -{{site.data.alerts.callout_info}} -Capacity metrics can be incorrect when running multiple nodes on a single machine. For more details, see this [limitation](known-limitations.html#available-capacity-metric-in-the-db-console). -{{site.data.alerts.end}} - -To see which nodes are located in which regions, and to see the simulated latencies between them, go to the [**Network Diagnostics**](ui-network-latency-page.html) page at http://localhost:8080/#/reports/network/region - -To access a variety of time series charts, including graphs of [SQL queries](ui-sql-dashboard.html#sql-statements) and [SQL service latency](ui-sql-dashboard.html#service-latency-sql-99th-percentile), go to the [**Metrics**](ui-overview-dashboard.html) page at http://localhost:8080/#/metrics - -You may also find the following pages useful: - -- The [**Databases**](ui-databases-page.html) page at http://localhost:8080/#/databases shows details about your databases and tables. -- The [**Statements**](ui-statements-page.html) page at http://localhost:8080/#/statements is used to assess the performance of specific queries. -- The [**Jobs**](ui-jobs-page.html) page at http://localhost:8080/#/jobs is used to monitor the status of long-running operations like [schema changes](online-schema-changes.html). - -## Step 4. Wipe the cluster - -When you're done with your demo cluster, you can wipe the cluster by typing the command shown below at the SQL prompt. Note that because this is a temporary, in-memory cluster, this will wipe any data you have added to the cluster. For more information, see the [`cockroach demo`](cockroach-demo.html) documentation. - -{% include_cached copy-clipboard.html %} -~~~ sql -\quit -~~~ - -## What's next? - -- [Install the client driver](install-client-drivers.html) for your preferred language -- Learn more about [CockroachDB SQL](learn-cockroachdb-sql.html) and the [built-in SQL client](cockroach-sql.html) -- [Build an app with CockroachDB](example-apps.html) -- Further explore CockroachDB capabilities like: - - [Multi-region SQL performance](demo-low-latency-multi-region-deployment.html) - - [Fault tolerance and automated repair](demo-fault-tolerance-and-recovery.html) - - [Serializable transactions](demo-serializable.html) - - [JSON support](demo-json-support.html) diff --git a/src/current/v21.1/spatial-data.md b/src/current/v21.1/spatial-data.md deleted file mode 100644 index ec4d078935b..00000000000 --- a/src/current/v21.1/spatial-data.md +++ /dev/null @@ -1,184 +0,0 @@ ---- -title: Working with Spatial Data -summary: CockroachDB has special support for efficiently storing and querying spatial data. -toc: true -keywords: gin, gin index, gin indexes, inverted index, inverted indexes, accelerated index, accelerated indexes ---- - -This page provides information about working with spatial data in CockroachDB. - -## Supported data types - -Supported [spatial](spatial-features.html) data types include: - -- Geometric objects such as [points](point.html), [lines](linestring.html), and [polygons](polygon.html) in 2-dimensional space. These are projected onto the flat surface of a plane and are represented in SQL by the `GEOMETRY` data type. - -- Geographic objects, which are also made up of [points](point.html), [lines](linestring.html), [polygons](polygon.html), etc., in 2-dimensional space. They are projected onto the surface of a sphere and are represented in SQL by the `GEOGRAPHY` data type. (Technically, they are projected onto a spheroid: "a sphere with a bulge"). The spheroid projection means that: - - - The X and Y coordinates of 2-dimensional points are actually Longitude and Latitude values. - - The paths between geographic objects are not straight lines; they are curves, and so the distances between objects are calculated using [great circle math](https://en.wikipedia.org/wiki/Great-circle_distance). - -## Compatibility - -Just as CockroachDB strives for [Postgres compatibility](postgresql-compatibility.html), our spatial data support is designed to be as compatible as possible with the functionality provided by the [PostGIS](https://postgis.net) extension. - -However, we do not yet implement the full list of PostGIS built-in functions and operators. Also, our [spatial indexing works differently](spatial-indexes.html) (see the [Performance](#performance) section below). For a list of the spatial functions we support, see [Geospatial functions](functions-and-operators.html#spatial-functions). - -If your application needs support for functions that are not yet implemented, please check out [our meta-issue for built-in function support on GitHub](https://github.com/cockroachdb/cockroach/issues/49203), which describes how to find an issue for the built-in function(s) you need. - -For a list of other known limitations, see [Known Limitations](known-limitations.html#spatial-support-limitations). - -### ORM compatibility - -The following ORM spatial libraries are fully compatible with CockroachDB's spatial features: - -- [Hibernate Spatial](https://docs.jboss.org/hibernate/orm/current/userguide/html_single/Hibernate_User_Guide.html#spatial) - - [Hibernate 5.4.30.Final](https://in.relation.to/2021/03/19/hibernate-orm-5430-final-release/) added the `CockroachDB202SpatialDialect` dialect to the `hibernate-spatial` module. The `CockroachDB202SpatialDialect` dialect supports spatial features available in CockroachDB v20.2 and later. - -- [RGeo/RGeo-ActiveRecord](https://github.com/rgeo/rgeo-activerecord) - - CockroachDB's ActiveRecord adapter ([`activerecord-cockroachdb-adapter`](https://github.com/cockroachdb/activerecord-cockroachdb-adapter)) uses [RGeo](https://github.com/rgeo/rgeo) and [RGeo-ActiveRecord](https://github.com/rgeo/rgeo-activerecord) for spatial support with ActiveRecord v5.2.2+ and v6.0.0+. For information about using CockroachDB spatial features with ActiveRecord, see the [`activerecord-cockroachdb-adapter` README](https://github.com/cockroachdb/activerecord-cockroachdb-adapter#working-with-spatial-data). - -- [GeoDjango](https://github.com/cockroachdb/django-cockroachdb#gis-support) - - Starting with CockroachDB 20.2.x and [`django-cockroachdb`](https://github.com/cockroachdb/django-cockroachdb) 3.1.3, CockroachDB is compatible with [GeoDjango](https://docs.djangoproject.com/en/3.1/ref/contrib/gis/). For information about using CockroachDB spatial features with GeoDjango, see the [`django-cockroachdb` README](https://github.com/cockroachdb/django-cockroachdb#gis-support). - -{{site.data.alerts.callout_info}} -Most PostGIS-compatible client libraries are incompatible with CockroachDB's spatial features without an adapter. -{{site.data.alerts.end}} - -## Troubleshooting - -For general CockroachDB troubleshooting information, see [this troubleshooting overview](troubleshooting-overview.html). - -If you need help troubleshooting an issue with our spatial support, please get in touch using our [Support resources](support-resources.html). - -## Performance - -In order to avoid full table scans, make sure to add [indexes](spatial-indexes.html) to any columns that are accessed as part of a predicate in the [`WHERE`](select-clause.html#filter-on-a-single-condition) clause. For geospatial columns, the index will only be used if the column is accessed using an index-accelerated geospatial function from the list below (all of these functions work on `GEOMETRY` data types; a `*` means that a function also works on `GEOGRAPHY` data types): - -- [`ST_Covers`](st_covers.html) (*) -- [`ST_CoveredBy`](st_coveredby.html) (*) -- [`ST_Contains`](st_contains.html) -- `ST_ContainsProperly` -- `ST_Crosses` -- `ST_DWithin` (*) -- `ST_DFullyWithin` -- [`ST_Equals`](st_equals.html) -- [`ST_Intersects`](st_intersects.html) (*) -- [`ST_Overlaps`](st_overlaps.html) -- [`ST_Touches`](st_touches.html) -- [`ST_Within`](st_within.html) - -To use a version of a function from the list above that will explicitly *not* use the index, add an underscore (`_`) to the beginning of the function name, e.g., [`_ST_Covers`](st_covers.html). - -You can check which queries are using which indexes using the [`EXPLAIN`](explain.html) statement. For more information about general query tuning (including index usage), see [Optimize Statement Performance](make-queries-fast.html). - -The syntax for adding an [index](spatial-indexes.html) to a geometry column is `CREATE INDEX index_name ON table_name USING GIST (column_name)`. - -For example, to add an index to the `geom` column of the [sample `tornadoes` table](migrate-from-shapefiles.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE INDEX tornado_geom_idx ON tornadoes USING GIST (geom); -~~~ - -This creates a (spatial) [GIN index](inverted-indexes.html) on the `geom` column. - -Because CockroachDB is a scale-out, multi-node database, our spatial indexing strategy is based on a [space-filling curve](https://en.wikipedia.org/wiki/Space-filling_curve)/quad-tree design (also known as "divide the space"), rather than the [R-Tree](https://en.wikipedia.org/wiki/R-tree) data structure used by some other spatial databases (also known as "divide the objects"). Other databases that use a "divide the space" strategy include Microsoft SQL Server and MongoDB. - -For more detailed information about how CockroachDB's spatial indexes work, see [Spatial indexes](spatial-indexes.html). - -If you encounter behavior that you think is due to a performance issue, please get in touch using our [Support resources](support-resources.html). - -## Example: Load NYC data for the PostGIS tutorial - -Follow the steps below to load the SQL for the NYC data used in the [Introduction to PostGIS](https://postgis.net/workshops/postgis-intro/) tutorial. - -{{site.data.alerts.callout_info}} -CockroachDB can work with the tutorial up to Chapter 22, with the following exceptions: - -- Do not try to load Shapefiles via the GUI as shown in the tutorial. Instead, follow the steps below to load the SQL data directly into CockroachDB. (We have already converted the tutorial Shapefiles to SQL for you.) -- We do not support GML or KML data. -- We do not support SVG. -{{site.data.alerts.end}} - -### Before you begin - -- Install a build of CockroachDB with support for spatial data by following the instructions at [Install CockroachDB](install-cockroachdb.html). - -- [Start a local insecure cluster](start-a-local-cluster.html) and connect to that cluster from a [SQL client](cockroach-sql.html): - - {% include_cached copy-clipboard.html %} - ~~~ shell - cockroach sql --insecure --host=localhost --port=26257 - ~~~ - - Leave this shell open for use in the examples below. - -### Step 1. Load the NYC data - -Clone the data set: - -{% include_cached copy-clipboard.html %} -~~~ shell -git clone https://github.com/otan-cockroach/otan-scripts -~~~ - -Load the SQL files into your CockroachDB cluster: - -{% include_cached copy-clipboard.html %} -~~~ shell -cat otan-scripts/geospatial_sql/*.sql | cockroach sql --insecure --host=localhost --port=26257 -~~~ - -The command above will take a few minutes to run. - -### Step 2. Follow the PostGIS tutorial - -When the cluster is finished loading the data, open a SQL shell and start working through the [Introduction to PostGIS](https://postgis.net/workshops/postgis-intro/) tutorial: - -{% include_cached copy-clipboard.html %} -~~~ shell -cockroach sql --insecure --host=localhost --port=26257 -~~~ - -## See also - -- [Install CockroachDB](install-cockroachdb.html) -- [Spatial Features](spatial-features.html) -- [Spatial indexes](spatial-indexes.html) -- [Spatial & GIS Glossary of Terms](spatial-glossary.html) -- [Working with Spatial Data](spatial-data.html) -- [Migrate from Shapefiles](migrate-from-shapefiles.html) -- [Migrate from GeoJSON](migrate-from-geojson.html) -- [Migrate from GeoPackage](migrate-from-geopackage.html) -- [Migrate from OpenStreetMap](migrate-from-openstreetmap.html) -- [Spatial functions](functions-and-operators.html#spatial-functions) -- [POINT](point.html) -- [LINESTRING](linestring.html) -- [POLYGON](polygon.html) -- [MULTIPOINT](multipoint.html) -- [MULTILINESTRING](multilinestring.html) -- [MULTIPOLYGON](multipolygon.html) -- [GEOMETRYCOLLECTION](geometrycollection.html) -- [Well known text](well-known-text.html) -- [Well known binary](well-known-binary.html) -- [GeoJSON](geojson.html) -- [SRID 4326 - longitude and latitude](srid-4326.html) -- [`ST_Contains`](st_contains.html) -- [`ST_ConvexHull`](st_convexhull.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Covers`](st_covers.html) -- [`ST_Disjoint`](st_disjoint.html) -- [`ST_Equals`](st_equals.html) -- [`ST_Intersects`](st_intersects.html) -- [`ST_Overlaps`](st_overlaps.html) -- [`ST_Touches`](st_touches.html) -- [`ST_Union`](st_union.html) -- [`ST_Within`](st_within.html) -- [Introducing Distributed Spatial Data in Free, Open Source CockroachDB](https://www.cockroachlabs.com/blog/spatial-data/) (blog post) -- [Troubleshooting overview](troubleshooting-overview.html) -- [Support resources](support-resources.html) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/spatial-features.md b/src/current/v21.1/spatial-features.md deleted file mode 100644 index 0ad3c92565f..00000000000 --- a/src/current/v21.1/spatial-features.md +++ /dev/null @@ -1,69 +0,0 @@ ---- -title: Spatial Features -summary: A summary of CockroachDB's support for storing and querying spatial data. -toc: true ---- - - CockroachDB supports efficiently storing and querying spatial data. - -See the links below for more information about how to use CockroachDB for spatial use cases. - -## Getting Started - -- [Install CockroachDB](install-cockroachdb.html) -- [Working with Spatial Data](spatial-data.html) -- [Spatial tutorial](spatial-tutorial.html) - -## Migrating spatial data into and out of CockroachDB - -- [Migrate from Shapefiles](migrate-from-shapefiles.html) -- [Migrate from GeoJSON](migrate-from-geojson.html) -- [Migrate from GeoPackage](migrate-from-geopackage.html) -- [Migrate from OpenStreetMap](migrate-from-openstreetmap.html) -- [Export Spatial Data](export-spatial-data.html) - -## Reference - -- [Spatial indexes](spatial-indexes.html) -- [Spatial and GIS Glossary of Terms](spatial-glossary.html) -- [Known Limitations](known-limitations.html#spatial-support-limitations) -- [Spatial functions](functions-and-operators.html#spatial-functions) -- [Client library compatibility](spatial-data.html#compatibility) - -### Spatial objects - -- [POINT](point.html) -- [LINESTRING](linestring.html) -- [POLYGON](polygon.html) -- [MULTIPOINT](multipoint.html) -- [MULTILINESTRING](multilinestring.html) -- [MULTIPOLYGON](multipolygon.html) -- [GEOMETRYCOLLECTION](geometrycollection.html) - -### Data representations - -- [Well known text](well-known-text.html) -- [Well known binary](well-known-binary.html) -- [GeoJSON](geojson.html) -- [SRID 4326 - longitude and latitude](srid-4326.html) - -### Spatial functions - -In addition to the [generated reference documentation for spatial functions](functions-and-operators.html#spatial-functions), we have written additional documentation for the following functions, including examples: - -- [`ST_Contains`](st_contains.html) -- [`ST_ConvexHull`](st_convexhull.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Covers`](st_covers.html) -- [`ST_Disjoint`](st_disjoint.html) -- [`ST_Equals`](st_equals.html) -- [`ST_Intersects`](st_intersects.html) -- [`ST_Overlaps`](st_overlaps.html) -- [`ST_Touches`](st_touches.html) -- [`ST_Union`](st_union.html) -- [`ST_Within`](st_within.html) - -## See also - -- [Introducing Distributed Spatial Data in Free, Open Source CockroachDB](https://www.cockroachlabs.com/blog/spatial-data/) (blog post) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/spatial-glossary.md b/src/current/v21.1/spatial-glossary.md deleted file mode 100644 index 46d73e3d9ef..00000000000 --- a/src/current/v21.1/spatial-glossary.md +++ /dev/null @@ -1,238 +0,0 @@ ---- -title: Spatial and GIS Glossary of Terms -summary: A summary of CockroachDB's support for storing and querying spatial data. -toc: true ---- - -This page contains a glossary of terms common to spatial databases and geographic information systems (GIS). Where possible, we provide links to further information. - -{{site.data.alerts.callout_info}} -This page is provided for reference purposes only. The inclusion of a term in this glossary does not imply that CockroachDB has support for any feature(s) related to that term. For more information about the specific spatial and GIS features supported by CockroachDB, see [Working with Spatial Data](spatial-data.html). -{{site.data.alerts.end}} - -## Geometry Terms - - - -- _Bounding box_: Given a set of points, a bounding box is the smallest rectangle that encloses all of the points in the set. Due to edge cases in how geographic points are mapped to [cartographic projections](#cartographic-projection), a bounding box for a given set of points may be larger than expected. - - - -- _Spheroid_: A spheroid (also known as an ellipsoid) is essentially a "slightly squished" sphere. Spheroids are used to represent almost-but-not-quite spherical objects. For example, a spheroid is used to represent the Earth in the [World Geodetic System](#wgs84) standard. - - - -- _Cartographic projection_: A cartographic projection, or map projection, is the process used to represent 3-dimensional (or higher) data on a 2-dimensional surface. This is usually related to how we might display 3-dimensional shapes represented in a database by the [`GEOGRAPHY` data type](#geography) on the surface of a map, which is a flat plane. For more information, see the GIS Lounge article [What is a Map Projection?](https://www.gislounge.com/map-projection/) by Caitlin Dempsey. - - - -- _Covering_: The covering of a shape _A_ is a set of locations (in CockroachDB, [S2 cell IDs](#s2)) that comprise another shape _B_ such that no points of _A_ lie outside of _B_. - - - -- _Geocoder_: Takes an address or the name of a place, and returns latitude and longitude coordinates. For more information, see [the wikipedia article on Geocoding](https://en.wikipedia.org/wiki/Geocoding). - - - -- _Nearest-neighbor search_: Given a starting point on a map and a set of search criteria, find the specified number of points nearest the starting point that meet the criteria. For example, a nearest-neighbor search can be used to answer the question, "What are the 10 closest [Waffle House restaurants](https://www.wafflehouse.com) to my current location?" This is also sometimes referred to as "_k_ nearest-neighbor" search. - - - -- _SRID_: The Spatial Referencing System Identifier (a.k.a. SRID) is used to tell which spatial reference system will be used to interpret each spatial object. A [commonly used SRID is 4326](srid-4326.html), which represents spatial data using longitude and latitude coordinates on the Earth's surface as defined in the [WGS84](#wgs84) standard. - - - -- _Spatial reference system_: Used to define what a spatial object "means". For example, a spatial object could use [geographic](#geography) coordinates using latitude and longitude, or a [geometry](#geometry) projection using points with X,Y coordinates in a 2-dimensional plane. - -## Data types - - - -- `GEOMETRY`: Used to represent shapes relative to 2-, 3-, or higher-dimensional plane geometry. For more information about the spatial objects used to represent geometries, see: - - [`POINT`](point.html) - - [`LINESTRING`](linestring.html) - - [`POLYGON`](polygon.html) - - [`MULTIPOINT`](multipoint.html) - - [`MULTILINESTRING`](multilinestring.html) - - [`MULTIPOLYGON`](multipolygon.html) - - [`GEOMETRYCOLLECTION`](geometrycollection.html) - - - -- `GEOGRAPHY`: Used to represent shapes relative to locations on the Earth's [spheroidal](#spheroid) surface. - -## Data Formats - - - - - - -- _WKT_: The "Well Known Text" data format is a convenient human-readable notation for representing [spatial objects](#spatial-objects). For example a 2-dimensional point object with x- and y-coordinates is represented in WKT as `POINT(123,456)`. This format is defined by the [OGC](#ogc). For more information, see the [Well Known Text](well-known-text.html) documentation. - -- _EWKT_: The "Extended Well Known Text" data format extends [WKT](#wkt) by prepending an [SRID](#srid) to the shape's description. For more information, see the [Well Known Text](well-known-text.html#ewkt) documentation. - - - - - - -- _WKB_: The "Well Known Binary" data format is a convenient machine-readable binary representation for [spatial objects](#spatial-objects). For efficiency, an application may choose to use this data format, but humans may prefer to read [WKT](#wkt). This format is defined by the [OGC](#ogc). For more information, see [Well Known Binary](well-known-binary.html). - -- _EWKB_: The "Extended Well Known Binary" data format extends [WKB](#wkb) by prepending [SRID](#srid) information to the shape's description. For more information, see [Well Known Binary](well-known-binary.html#ewkb). - -## Organizations - - - -- _OSGeo_: The Open Source Geospatial Foundation. For more information, see . - - - -- _OGC_: The [Open Geospatial Consortium](https://www.ogc.org) was formerly known as the "Open GIS Consortium". The organization is still referred to colloquially as "OpenGIS" in many places online. The OGC is a consortium of businesses, government agencies, universities, etc., described as "a worldwide community committed to improving access to geospatial (location) information." - - - -- _MapBox_: A company providing a location data platform for mobile and web applications. For more information, see . - - - -- _Esri_: A company providing "location intelligence" services. Esri develops spatial and [GIS](#gis) software, including the popular [ArcGIS](#arcgis) package. For more information about Esri, see . - -## Industry Standards - - - -- _SQL/MM_: The SQL Multimedia Applications specification. The part of this standard that applies to SQL geospatial data types is defined in [part 3 of the ISO/IEC 13249 document](https://www.iso.org/standard/60343.html). For a freely available paper discussing the geospatial data types and functions defined by the standard, see the (PDF) paper [SQL/MM Spatial: The Standard to Manage Spatial Data in Relational Database Systems](http://doesen0.informatik.uni-leipzig.de/proceedings/paper/68.pdf), by Knut Stolze. - - - -- _WGS84_: The latest revision of the [World Geodetic System](http://wiki.gis.com/wiki/index.php/World_Geodetic_System) standard (from 1984 CE), which defines a standard spheroidal reference system for mapping the Earth. See also: [spheroid](#spheroid), [cartographic projection](#cartographic-projection). - - - -- _DE-9IM_: The [Dimensionally Extended nine-Intersection Model (DE-9IM)](https://en.wikipedia.org/wiki/DE-9IM) defines a method that uses a 3x3 matrix to determine whether two shapes (1) touch along a boundary, (2), intersect (overlap), or (3) are equal to each other - that is, they are the same shape that covers the same area. This notation is used by the `ST_Relate` built-in function. Almost all other spatial predicate functions can be logically implemented using this model. However, in practice, most are not, and `ST_Relate` is reserved for advanced use cases. - -## File Formats - - - -- _Shapefile_: A spatial data file format developed by [Esri](#esri) and used by [GIS](#gis) software for storing geospatial data. It can be automatically converted to SQL by tools like [shp2pgsql](https://manpages.debian.org/stretch/postgis/shp2pgsql.1.en.html) for use by a database that can run spatial queries. - - - -- _Vector file_: A file format that uses a non-pixel-based, abstract coordinate representation for geospatial data. Because it is abstract and not tied to pixels, the vector format is scalable. The motivation is similar to that behind the [Scalable Vector Graphics (SVG)](https://en.wikipedia.org/wiki/Scalable_Vector_Graphics) image format: scaling the image up or down does not reveal any "jaggedness" (due to loss of information) such as might be revealed by a pixel representation. However, vector files are usually much larger in size and more expensive (in terms of CPU, memory, and disk) to work with than [Raster files](#raster-file). - - - -- _Raster file_: A file format that uses a non-scalable, pixel-based representation for geospatial data. Raster files are smaller and generally faster to read, write, or generate than [Vector files](#vector-file). However, raster files have inferior image quality and/or accuracy when compared to vector files: they can appear "jagged" due to the reduced information available when compared to vector files. - - - -- _GeoJSON_: A format for encoding geometric and geographic data as [JSON](https://www.json.org). For more information, see [GeoJSON](geojson.html). - -## Software and Code Libraries - - - -- _GIS_: A "Geographic Information System" (or GIS) is used to store geographic information in a computer for processing and interaction by humans and/or other software. Some systems provide graphical "point and click" user interfaces, and some are embedded in programming languages or data query languages like SQL. For example, CockroachDB versions 20.2 and later provide support for [executing spatial queries from SQL](spatial-data.html). - - - -- _ArcGIS_: A commercial [GIS](#gis) software package developed by the location intelligence company [Esri](#esri). For more information, see Esri's [ArcGIS overview](https://www.esri.com/en-us/arcgis/about-arcgis/overview). - - - -- _PostGIS_: An extension to the [Postgres](https://postgresql.org/) database that adds support for geospatial queries. For more information, see [postgis.net](https://postgis.net). - - - -- _GEOS_: An open source geometry library used by CockroachDB, [PostGIS](#postgis), and other projects to provide the calculations underlying various spatial predicate functions and operators. For more information, see . - - - -- _GeographicLib_: A C++ library for performing various geographic and other calculations used by CockroachDB and other projects. For more information, see . - - - -- _GDAL_: A Geospatial Data Abstraction Library used to provide support for many types of [raster file formats](#raster-file). Used in [PostGIS](#postgis). For more information, see . - - - -- _PROJ_: A [cartographic projection](#cartographic-projection) library. Used by CockroachDB and other projects. For more information, see . - - - -- _GeoBuf_: A compact binary encoding developed by [MapBox](#mapbox) that provides nearly lossless compression of [GeoJSON](#geojson) data into [protocol buffers](https://developers.google.com/protocol-buffers/). For more information, see . - - - -- _CGAL_: The computational geometry algorithms library. For more information, see . - - - -- _SFCGAL_: A C++ wrapper library around [CGAL](#cgal). For more information, see . - - - -- _TIGER_: The "Topographically Integrated Geographic Encoding and Referencing System" released by the [U.S. Census Bureau](https://www.census.gov/programs-surveys/geography/guidance/tiger-data-products-guide.html). - - - -- _S2_: The [S2 Geometry Library](http://s2geometry.io) is a C++ code library for performing spherical geometry computations. It models a sphere using a [quadtree](https://en.wikipedia.org/wiki/Quadtree) "divide the space" approach, and is used by CockroachDB. - -## Spatial objects - -This section has information about the representation of geometric and geographic "shapes" according to the [SQL/MM](#sql-mm) standard. - - - -- _Point_: A point is a sizeless location identified by its X and Y coordinates. These coordinates are then translated according to the [spatial reference system](#spatial-reference-system) to determine what the point "is", or what it "means" relative to the other geometric objects (if any) in the data set. A point can be created in SQL by the `ST_Point()` function. - - - -- _LineString_: A linestring is a collection of [points](#point) that are "strung together" into one geometric object, like a necklace. If the "necklace" were "closed", it could also represent a [polygon](#polygon). A linestring can also be used to represent an arbitrary curve, such as a [Bézier curve](https://en.wikipedia.org/wiki/Bézier_curve). - - - -- _Polygon_: A polygon is a closed shape that can be made up of straight or curved lines. It can be thought of as a "closed" [linestring](#linestring). Irregular polygons can take on almost any arbitrary shape. Common regular polygons include: squares, rectangles, hexagons, and so forth. For more information about regular polygons, see the ['Regular polygon' Wikipedia article](https://en.wikipedia.org/wiki/Regular_polygon). - - - -- _GeometryCollection_: A geometry collection is a "box" or "bag" used for gathering 1 or more of the other types of objects defined above into a collection: namely, points, linestrings, or polygons. In the particular case of SQL, it provides a way of referring to a group of spatial objects as one "thing" so that you can operate on it/them more conveniently, using various SQL functions. - -## Spatial System tables - - - -- `pg_extension`: A table used by the Postgres database to store information about extensions to that database. Provided for compatibility by CockroachDB. For more information, see [the Postgres documentation](https://www.postgresql.org/docs/current/catalog-pg-extension.html). - - - -- `spatial_ref_sys`: A SQL table defined by the [OGC](#ogc) that holds the list of [SRID](#srid)s supported by a database, e.g., `SELECT count(*) FROM spatial_ref_sys;` - - - -- `geometry_columns`: Used to list all of the columns in a database with the [`GEOMETRY`](#geometry) data type, e.g., `SELECT * from geometry_columns`. - - - -- `geography_columns`: Used to list all of the columns in a database with the [`GEOGRAPHY`](#geography) data type, e.g., `SELECT * from geography_columns`. - -## See also - -- [Introducing Distributed Spatial Data in Free, Open Source CockroachDB](https://www.cockroachlabs.com/blog/spatial-data/) (blog post) -- [Spatial Features](spatial-features.html) -- [Spatial tutorial](spatial-tutorial.html) -- [Working with Spatial Data](spatial-data.html) -- [Spatial indexes](spatial-indexes.html) -- [Spatial functions](functions-and-operators.html#spatial-functions) -- [Migrate from Shapefiles](migrate-from-shapefiles.html) -- [Migrate from GeoJSON](migrate-from-geojson.html) -- [Migrate from GeoPackage](migrate-from-geopackage.html) -- [Migrate from OpenStreetMap](migrate-from-openstreetmap.html) -- [Well known text](well-known-text.html) -- [Well known binary](well-known-binary.html) -- [GeoJSON](geojson.html) -- [SRID 4326 - longitude and latitude](srid-4326.html) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/spatial-indexes.md b/src/current/v21.1/spatial-indexes.md deleted file mode 100644 index 80c95b9a462..00000000000 --- a/src/current/v21.1/spatial-indexes.md +++ /dev/null @@ -1,258 +0,0 @@ ---- -title: Spatial Indexes -summary: How CockroachDB uses spatial indexes for efficiently storing and querying spatial data. -toc: true -keywords: gin, gin index, gin indexes, inverted index, inverted indexes, accelerated index, accelerated indexes ---- - -This page describes CockroachDB's support for indexing [spatial data](spatial-data.html), including: - -- What spatial indexes are -- How they work -- How to create and tune spatial indexes in CockroachDB - -{{site.data.alerts.callout_info}} -CockroachDB does not support indexing geospatial types in default [primary keys](primary-key.html), nor in [unique secondary indexes](indexes.html#unique-secondary-indexes). -{{site.data.alerts.end}} - -## What is a spatial index? - -At a high level, a spatial index is just like any other [index](indexes.html). Its purpose is to improve your database's performance by helping SQL locate data without having to look through every row of a table. - -Spatial indexes store information about spatial objects, but they are used for the same tasks as any other index type, i.e.: - -- Fast filtering of lists of shapes based on spatial predicate functions such as `ST_Contains`. - -- Speeding up joins that involve spatial data columns. - -Spatial indexes differ from other types of indexes as follows: - -- They are specialized to operate on 2-dimensional `GEOMETRY` and `GEOGRAPHY` data types. They are stored by CockroachDB as a special type of [GIN index](inverted-indexes.html). For more details, [see below](#details). - -- If needed, they can be tuned to store looser or tighter coverings of the shapes being indexed, depending on the needs of your application. Tighter coverings are more expensive to generate, store, and update, but can perform better in some cases because they return fewer false positives during the initial index lookup. Tighter coverings can also lead to worse performance due to more rows in the index, and more scans at the storage layer. That's why we recommend that most users should not need to change the default settings. For more information, see [Tuning spatial indexes](#tuning-spatial-indexes) below. - -## How CockroachDB's spatial indexing works - -### Overview - -There are two main approaches to building geospatial indexes: - -- One approach is to "divide the objects" by inserting the objects into a tree (usually an [R-tree](https://en.wikipedia.org/wiki/R-tree)) whose shape depends on the data being indexed. This is the approach taken by [PostGIS](https://postgis.net). - -- The other approach is to "divide the space" by creating a decomposition of the space being indexed into buckets of various sizes. - -CockroachDB takes the "divide the space" approach to spatial indexing. This is necessary to preserve CockroachDB's ability to [scale horizontally](frequently-asked-questions.html#how-does-cockroachdb-scale) by [adding nodes to a running cluster](cockroach-start.html#add-a-node-to-a-cluster). - -CockroachDB uses the "divide the space" approach for the following reasons: - -- It's easy to scale horizontally. -- It requires no balancing operations, unlike [R-tree indexes](https://en.wikipedia.org/wiki/R-tree). -- Inserts under this approach require no locking. -- Bulk ingest is simpler to implement than under other approaches. -- It allows advanced users to make a per-object tradeoff between index size and false positives during index creation. (See [Tuning spatial indexes](#tuning-spatial-indexes) below.) - -Whichever approach to indexing is used, when an object is indexed, a "covering" shape (i.e., a bounding box) is constructed that completely encompasses the indexed object. Index queries work by looking for containment or intersection between the covering shape for the query object and the indexed covering shapes. This retrieves false positives but no false negatives. - -### Details - -Under the hood, CockroachDB uses the [S2 geometry library](https://s2geometry.io/) to divide the space being indexed into a [quadtree](https://en.wikipedia.org/wiki/Quadtree) data structure with a set number of levels and a data-independent shape. Each node in the quadtree (really, [S2 cell](https://s2geometry.io/devguide/s2cell_hierarchy.html)) represents some part of the indexed space and is divided once horizontally and once vertically to produce 4 child cells in the next level. The image below shows visually how a location (marked in red) is represented using levels of a quadtree: - -Quadtree - -Visually, you can think of the S2 library as enclosing a sphere in a cube. We map from points on each face of the cube to points on the face of the sphere. As you can see in the 2-dimensional picture below, there is a projection that occurs in this mapping: the lines entering from the left mark points on the cube face, and are "refracted" by the material of the cube face before touching the surface of the sphere. This projection reduces the distortion that would occur if the points on the cube face were projected straight onto the sphere. - -S2 Cubed Sphere - 2D - -Next, let's look at a 3-dimensional image that shows the cube and sphere more clearly. Each cube face is mapped to the quadtree data structure mentioned above, and each node in the quadtree is numbered using a [Hilbert space-filling curve](https://en.wikipedia.org/wiki/Hilbert_curve) which preserves locality of reference. In the image below, you can imagine the points of the Hilbert curve on the rear face of the cube being projected onto the sphere in the center. The use of a space-filling curve means that two shapes that are near each other on the sphere are very likely to be near each other on the line that makes up the Hilbert curve. This is good for performance. - -S2 Cubed Sphere - 3D - -When you index a spatial object, a covering is computed using some number of the cells in the quadtree. The number of covering cells can vary per indexed object by passing special arguments to `CREATE INDEX` that tell CockroachDB how many levels of S2 cells to use. The leaf nodes of the S2 quadtree are at level 30, and for `GEOGRAPHY` measure 1cm across the Earth's surface. By default, `GEOGRAPHY` indexes use up to level 30, and get this level of precision. We also use S2 cell coverings in a slightly different way for `GEOMETRY` indexes. The precision you get there is the bounding length of the `GEOMETRY` index divided by 4^30. For more information, see [Tuning spatial indexes](#tuning-spatial-indexes) below. - -CockroachDB stores spatial indexes as a special type of [GIN index](inverted-indexes.html). The spatial index maps from a location, which is a square cell in the quadtree, to one or more shapes whose [coverings](spatial-glossary.html#covering) include that location. Since a location can be used in the covering for multiple shapes, and each shape can have multiple locations in its covering, there is a many-to-many relationship between locations and shapes. - -## Tuning spatial indexes - -{{site.data.alerts.callout_danger}} -The information in this section is for advanced users doing performance optimization. Most users should not need to change the default settings. Without careful testing, you can get worse performance by changing the default settings. -{{site.data.alerts.end}} - -When an object is indexed, a "covering" shape (e.g., a bounding box) is constructed that completely encompasses the indexed object. Index queries work by looking for containment or intersection between the covering shape for the query object and the indexed covering shapes. This retrieves false positives but no false negatives. - -This leads to a tradeoff when creating spatial indexes. The number of cells used to represent an object in the index is tunable: fewer cells use less space, but create a looser covering. A looser covering retrieves a larger amount of positives from the index, which is expensive because the exact answer computation that's run after the index query can be expensive. However, at some point the benefits of retrieving fewer false positives is outweighed by how long it takes to scan a large index. - -Another consideration is that the larger index created for a tighter covering is more expensive to scan and to update, especially for write-heavy workloads. - -We strongly recommend leaving your spatial indexes at the default settings unless you can take at least the following steps: - -1. Verify the coverings that will be generated meet your accuracy expectations by using the `st_s2covering` function as shown in [Viewing an object's S2 covering](#viewing-an-objects-s2-covering) below. -2. Do extensive performance tests to make sure the indexes that use the non-default coverings actually provide a performance benefit. - -### Visualizing index coverings - -In what follows we will visualize how index coverings change as we change the `s2_max_cells` parameter, which determines how much work CockroachDB will perform to find a tight covering of a shape. - -We will generate coverings for the following geometry object, which describes a shape whose vertices land on some small cities in the Northeastern US: - -~~~ -'SRID=4326;LINESTRING(-76.4955 42.4405, -75.6608 41.4102,-73.5422 41.052, -73.929 41.707, -76.4955 42.4405)' -~~~ - -The animated image below shows the S2 coverings that are generated as we increase the `s2_max_cells` parameter (described below) from the 1 to 30 (minimum to maximum): - -Animated GIF of S2 Coverings - Levels 1 to 30 - -Below are the same images presented in a grid. You can see that as we turn up the `s2_max_cells` parameter, more work is done by CockroachDB to discover a tighter and tighter covering (that is, a covering using more and smaller cells). The covering for this particular shape reaches a reasonable level of accuracy when `s2_max_cells` reaches 10, and stops improving much past 12. - -Static image of S2 Coverings - Levels 1 to 30 - -### Index tuning parameters - -The following parameters are supported by both `CREATE INDEX` and the [built-in function](functions-and-operators.html#spatial-functions) `st_s2covering`. The latter is useful for seeing what kinds of coverings would be generated by an index with various tuning parameters if you were to create it. For an example showing how to use these parameters with `st_s2covering`, see [viewing an object's S2 covering](#viewing-an-objects-s2-covering) below. - -{{site.data.alerts.callout_info}} -If a shape falls outside the bounding coordinates determined by the `geometry_*` settings below, there will be a performance loss in that such shapes may not be eliminated by the index lookup, but the database will still return the correct answers. -{{site.data.alerts.end}} - -| Option | Default value | Description | -|------------------+-----------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `s2_max_level` | 30 | The maximum level of S2 cell used in the covering. Allowed values: 1-30. Setting it to less than the default means that CockroachDB will be forced to generate coverings using larger cells. | -| `s2_max_cells` | 4 | The maximum number of S2 cells used in the covering. Provides a limit on how much work is done exploring the possible coverings. Allowed values: 1-30. You may want to use higher values for odd-shaped regions such as skinny rectangles. | -| `geometry_min_x` | Derived from SRID bounds, else `-(1 << 31)` | Minimum X-value of the [spatial reference system](spatial-glossary.html#spatial-reference-system) for the object(s) being covered. This only needs to be set if the default bounds of the SRID are too large/small for the given data, or SRID = 0 and you wish to use a smaller range (unfortunately this is currently not exposed, but is viewable on ). By default, SRID = 0 assumes `[-min int32, max int32]` ranges. | -| `geometry_max_x` | Derived from SRID bounds, else `(1 << 31) -1` | Maximum X-value of the [spatial reference system](spatial-glossary.html#spatial-reference-system) for the object(s) being covered. This only needs to be set if you are using a custom [SRID](spatial-glossary.html#srid). | -| `geometry_min_y` | Derived from SRID bounds, else `-(1 << 31)` | Minimum Y-value of the [spatial reference system](spatial-glossary.html#spatial-reference-system) for the object(s) being covered. This only needs to be set if you are using a custom [SRID](spatial-glossary.html#srid). | -| `geometry_max_y` | Derived from SRID bounds, else `(1 << 31) -1` | Maximum Y-value of the [spatial reference system](spatial-glossary.html#spatial-reference-system) for the object(s) being covered. This only needs to be set if you are using a custom [SRID](spatial-glossary.html#srid). | - -## Examples - -### Viewing an object's S2 covering - -Here is an example showing how to pass the [spatial index tuning parameters](#index-tuning-parameters) to `st_s2covering`. It generates [GeoJSON](https://geojson.org) output showing both a shape and the S2 covering that would be generated for that shape in your index, if you passed the same parameters to `CREATE INDEX`. You can paste this output into [geojson.io](http://geojson.io) to see what it looks like. - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE tmp_viz (id INT8, geom GEOMETRY); - -INSERT INTO tmp_viz (id, geom) -VALUES (1, st_geomfromtext('LINESTRING(-76.8261 42.1727, -75.6608 41.4102,-73.5422 41.052, -73.929 41.707, -76.8261 42.1727)')); - -INSERT INTO tmp_viz (id, geom) VALUES (2, st_s2covering(st_geomfromtext('LINESTRING(-76.8261 42.1727, -75.6608 41.4102,-73.5422 41.052, -73.929 41.707, -76.8261 42.1727)'), 's2_max_cells=20,s2_max_level=12,s2_level_mod=3,geometry_min_x=-180,geometry_max_x=180,geometry_min_y=-180,geometry_max_y=180')); - -SELECT ST_AsGeoJSON(st_collect(geom)) FROM tmp_viz; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ json -{"type":"GeometryCollection","geometries":[{"type":"LineString","coordinates":[[-76.8261,42.1727],[-75.6608,41.4102],[-73.5422,41.052],[-73.929,41.707],[-76.8261,42.1727]]},{"type":"MultiPolygon","coordinates":[[[[-76.904296875,42.099609375],[-76.81640625,42.099609375],[-76.81640625,42.1875],[-76.904296875,42.1875],[-76.904296875,42.099609375]]],[[[-76.81640625,42.099609375],[-76.728515625,42.099609375],[-76.728515625,42.1875],[-76.81640625,42.1875],[-76.81640625,42.099609375]]],[[[-76.728515625,42.099609375],[-76.640625,42.099609375],[-76.640625,42.1875],[-76.728515625,42.1875],[-76.728515625,42.099609375]]],[[[-76.728515625,42.01171875],[-76.640625,42.01171875],[-76.640625,42.099609375],[-76.728515625,42.099609375],[-76.728515625,42.01171875]]],[[[-76.640625,41.484375],[-75.9375,41.484375],[-75.9375,42.1875],[-76.640625,42.1875],[-76.640625,41.484375]]],[[[-74.53125,40.78125],[-73.828125,40.78125],[-73.828125,41.484375],[-74.53125,41.484375],[-74.53125,40.78125]]],[[[-73.828125,40.78125],[-73.125,40.78125],[-73.125,41.484375],[-73.828125,41.484375],[-73.828125,40.78125]]],[[[-73.828125,41.484375],[-73.740234375,41.484375],[-73.740234375,41.572265625],[-73.828125,41.572265625],[-73.828125,41.484375]]],[[[-74.53125,41.484375],[-73.828125,41.484375],[-73.828125,42.1875],[-74.53125,42.1875],[-74.53125,41.484375]]],[[[-75.234375,41.484375],[-74.53125,41.484375],[-74.53125,42.1875],[-75.234375,42.1875],[-75.234375,41.484375]]],[[[-75.234375,40.78125],[-74.53125,40.78125],[-74.53125,41.484375],[-75.234375,41.484375],[-75.234375,40.78125]]],[[[-75.322265625,41.30859375],[-75.234375,41.30859375],[-75.234375,41.396484375],[-75.322265625,41.396484375],[-75.322265625,41.30859375]]],[[[-75.41015625,41.30859375],[-75.322265625,41.30859375],[-75.322265625,41.396484375],[-75.41015625,41.396484375],[-75.41015625,41.30859375]]],[[[-75.5859375,41.396484375],[-75.498046875,41.396484375],[-75.498046875,41.484375],[-75.5859375,41.484375],[-75.5859375,41.396484375]]],[[[-75.5859375,41.30859375],[-75.498046875,41.30859375],[-75.498046875,41.396484375],[-75.5859375,41.396484375],[-75.5859375,41.30859375]]],[[[-75.498046875,41.30859375],[-75.41015625,41.30859375],[-75.41015625,41.396484375],[-75.498046875,41.396484375],[-75.498046875,41.30859375]]],[[[-75.673828125,41.396484375],[-75.5859375,41.396484375],[-75.5859375,41.484375],[-75.673828125,41.484375],[-75.673828125,41.396484375]]],[[[-75.76171875,41.396484375],[-75.673828125,41.396484375],[-75.673828125,41.484375],[-75.76171875,41.484375],[-75.76171875,41.396484375]]],[[[-75.849609375,41.396484375],[-75.76171875,41.396484375],[-75.76171875,41.484375],[-75.849609375,41.484375],[-75.849609375,41.396484375]]],[[[-75.9375,41.484375],[-75.234375,41.484375],[-75.234375,42.1875],[-75.9375,42.1875],[-75.9375,41.484375]]]]}]} -~~~ - -When you paste the JSON output above into [geojson.io](http://geojson.io), it generates the picture below, which shows both the `LINESTRING` and its S2 covering based on the options you passed to `st_s2covering`. - -S2 LINESTRING example covering - -### Create a spatial index - -The example below shows how to create a spatial index on a `GEOMETRY` object using the default settings: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE INDEX geom_idx_1 ON some_spatial_table USING GIST(geom); -~~~ - -~~~ -CREATE INDEX -~~~ - -### Create a spatial index with non-default tuning parameters - -The examples below show how to create spatial indexes with non-default settings for some of the [spatial index tuning parameters](#index-tuning-parameters): - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE INDEX geom_idx_1 ON geo_table1 USING GIST(geom) WITH (s2_level_mod=3); -CREATE INDEX geom_idx_2 ON geo_table2 USING GIST(geom) WITH (geometry_min_x=0, s2_max_level=15) -CREATE INDEX geom_idx_3 ON geo_table3 USING GIST(geom) WITH (s2_max_level=10) -CREATE INDEX geom_idx_4 ON geo_table4 USING GIST(geom) WITH (geometry_min_x=0, s2_max_level=15); -~~~ - -~~~ -CREATE INDEX -~~~ - -### Create spatial indexes during table definition - -This example shows how to [create a table](create-table.html) with spatial indexes on `GEOMETRY` and `GEOGRAPHY` types that use non-default settings for some of the [spatial index tuning parameters](#index-tuning-parameters): - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE TABLE public.geo_table ( - id INT8 NOT NULL, - geog GEOGRAPHY(GEOMETRY,4326) NULL, - geom GEOMETRY(GEOMETRY,3857) NULL, - CONSTRAINT "primary" PRIMARY KEY (id ASC), - INVERTED INDEX geom_idx_1 (geom) WITH (s2_max_level=15, geometry_min_x=0), - INVERTED INDEX geom_idx_2 (geom) WITH (geometry_min_x=0), - INVERTED INDEX geom_idx_3 (geom) WITH (s2_max_level=10), - INVERTED INDEX geom_idx_4 (geom), - INVERTED INDEX geog_idx_1 (geog) WITH (s2_level_mod=2), - INVERTED INDEX geog_idx_2 (geog), - FAMILY fam_0_geog (geog), - FAMILY fam_1_geom (geom), - FAMILY fam_2_id (id) -) -~~~ - -~~~ -CREATE TABLE -~~~ - -### Create a spatial index that uses all of the tuning parameters - -This example shows how to set all of the [spatial index tuning parameters](#index-tuning-parameters) at the same time. It is extremely unlikely you will ever need to set all of the options at once; this example is being provided to show the syntax. - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE INDEX geom_idx - ON some_spatial_table USING GIST(geom) - WITH (s2_max_cells = 20, s2_max_level = 12, s2_level_mod = 3, - geometry_min_x = -180, geometry_max_x = 180, - geometry_min_y = -180, geometry_max_y = 180); -~~~ - -~~~ -CREATE INDEX -~~~ - -{{site.data.alerts.callout_danger}} -As noted above, most users should not change the default settings. There is a risk that you will get worse performance by changing the default settings. -{{site.data.alerts.end}} - -### Create a spatial index on a `GEOGRAPHY` object - -The example below shows how to create a spatial index on a `GEOGRAPHY` object using the default settings: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE INDEX geog_idx_3 ON geo_table USING GIST(geog); -~~~ - -~~~ -CREATE INDEX -~~~ - -## See also - -- [GIN Indexes](inverted-indexes.html) -- [S2 Geometry Library](https://s2geometry.io/) -- [Indexes](indexes.html) -- [Spatial Features](spatial-features.html) -- [Spatial tutorial](spatial-tutorial.html) -- [Working with Spatial Data](spatial-data.html) -- [Spatial and GIS Glossary of Terms](spatial-glossary.html) -- [Spatial functions](functions-and-operators.html#spatial-functions) -- [Migrate from Shapefiles](migrate-from-shapefiles.html) -- [Migrate from GeoJSON](migrate-from-geojson.html) -- [Migrate from GeoPackage](migrate-from-geopackage.html) -- [Migrate from OpenStreetMap](migrate-from-openstreetmap.html) -- [Introducing Distributed Spatial Data in Free, Open Source CockroachDB](https://www.cockroachlabs.com/blog/spatial-data/) (blog post) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/spatial-tutorial.md b/src/current/v21.1/spatial-tutorial.md deleted file mode 100644 index 853477405af..00000000000 --- a/src/current/v21.1/spatial-tutorial.md +++ /dev/null @@ -1,1857 +0,0 @@ ---- -title: Spatial Data -summary: Tutorial for working with spatial data in CockroachDB. -toc: true -toc_not_nested: true ---- - -In this tutorial, you will plan a vacation from New York City to the [Adirondack Mountains](https://visitadirondacks.com/about) in northern New York State to do some birdwatching while visiting local independent bookstores. In the process, you will explore several of CockroachDB's [spatial capabilities](spatial-features.html): - -+ Importing spatial data from SQL files (including how to build spatial geometries from data in CSV files). -+ Putting together separate spatial data sets to ask and answer potentially interesting questions. -+ Using various [built-in functions](functions-and-operators.html#spatial-functions) for operating on spatial data. -+ Creating [indexes](spatial-indexes.html) on spatial data. -+ Performing [joins](joins.html) on spatial data, and using [`EXPLAIN`](explain.html) to make sure indexes are effective. -+ Visualizing the output of your queries using free tools like - -## Step 1. Review the scenario - -You live in New York City and are an avid birdwatcher and reader of books. You are going on a vacation up to the [Adirondack Mountains](https://visitadirondacks.com/about) in northern New York State. Although you are interested in many bird species, you are most interested in seeing (and [hearing](https://macaulaylibrary.org/asset/107964)) the Common Loon, a bird that can be found near the lakes and ponds of the north woods. - -As a connoisseur of bookstores, you also want to make sure to visit as many of the local independent bookstores as possible on your trip, as long as they are nearby to Loon habitat. - -Therefore, in order to plan your vacation trip effectively and optimize for having the most fun, you want to make sure to visit: - -+ Loon habitat (at minimum -- ideally you would like to go on the "hunt" for other interesting bird species). -+ Independent bookstores that are located within the approximate home range of the loon. - -To accomplish the above, you have put together a data set that consists of what are normally several different data sets, namely: - -+ NY State bird sighting data from the years 2000-2019, taken from the [North American Breeding Bird Survey (NABBS)](https://www.sciencebase.gov/catalog/item/52b1dfa8e4b0d9b325230cd9). The full survey covers the US and Canada from the years 1966-2019 and is very large, so you have decided to import just the recent NY data since that is all you need to plan your trip. - -+ Road data from the [US National Atlas - Major Roads of the United States](https://www.sciencebase.gov/catalog/item/581d052be4b08da350d524ce) data set. This consists of a full road map of the United States. It was last modified in 2016, so it should be reasonably accurate. - -+ Bookstore information (name, address, website, etc.) you scraped from the [American Booksellers Association website's member directory](https://bookweb.org/member_directory/search/ABAmember). - -For more information about how this data set is put together, see the [Data set description](#data-set-description). - -## Step 2. Start CockroachDB - -This tutorial can be accomplished in any CockroachDB cluster running [v20.2](../releases/v20.2.html#v20-2-0) or later. - -The simplest way to get up and running is with [`cockroach demo`](cockroach-demo.html), which starts a temporary, in-memory CockroachDB cluster and opens an interactive SQL shell: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach demo -~~~ - -You will see a [SQL client prompt](cockroach-sql.html) that looks like the one shown below. You will use this prompt for the rest of the tutorial. - -~~~ -root@127.0.0.1:34839/movr> -~~~ - -## Step 3. Load the data set - -1. Create a `tutorial` database, and use it. - - {% include_cached copy-clipboard.html %} - ~~~ sql - CREATE DATABASE tutorial; - USE tutorial; - ~~~ - -2. [`IMPORT`](import.html) the parts of [the data set](#data-set-description) that live in the `tutorial` database. - - {% include_cached copy-clipboard.html %} - ~~~ sql - IMPORT PGDUMP ('https://spatial-tutorial.s3.us-east-2.amazonaws.com/bookstores-and-roads-20210125.sql') WITH ignore_unsupported_statements; - ~~~ - - ~~~ - job_id | status | fraction_completed | rows | index_entries | bytes - ---------------------+-----------+--------------------+--------+---------------+----------- - 629565276454256641 | succeeded | 1 | 228807 | 0 | 75952972 - (1 row) - - Time: 17.745s total (execution 17.744s / network 0.000s) - ~~~ - -3. Create a `birds` database, and use it. - - {% include_cached copy-clipboard.html %} - ~~~ sql - CREATE DATABASE birds; - USE birds; - ~~~ - -4. [`IMPORT`](import.html) the parts of [the data set](#data-set-description) that live in the `birds` database. - - {% include_cached copy-clipboard.html %} - ~~~ sql - IMPORT PGDUMP ('https://spatial-tutorial.s3.us-east-2.amazonaws.com/birds-20210125.sql') WITH ignore_unsupported_statements; - ~~~ - - ~~~ - job_id | status | fraction_completed | rows | index_entries | bytes - ---------------------+-----------+--------------------+-------+---------------+---------- - 629565605599412225 | succeeded | 1 | 86616 | 0 | 4096847 - (1 row) - ~~~ - -5. Switch back to the `tutorial` database. All of the queries in this tutorial assume you are in the `tutorial` database. - - {% include_cached copy-clipboard.html %} - ~~~ sql - USE tutorial; - ~~~ - -## Step 4. Scout loon locations - -### (1) Where has the Common Loon been sighted by the NABBS in the years 2000-2019 in NY state? - -As a first step, you'd like to know where exactly the Common Loon has been sighted in New York State. You know that many of the loon sightings will have taken place in the Adirondacks (our overall destination) due to the nature of the loon's preferred habitat of northern lakes and ponds, but it will be useful to know the precise locations for trip planning, and for asking further related questions about other bird habitats as well as nearby bookstores you'd like to visit. - -Because of the structure of [the `birds` database](#the-birds-database), you will wrap the results of a [subquery](subqueries.html) against the `birds` database in a [common table expression (CTE)](common-table-expressions.html) to provide a shorthand name for referring to this data. This step will be necessary every time you want to get information about bird sightings. Therefore, the general pattern for many of these queries will be something like: - -1. Get bird information (usually including location data) and store the results in a named CTE. -2. Using the named CTE, perform additional processing against the results of the CTE combined with other data you have (e.g., bookstores, roads). Depending on the complexity of the questions asked, you may even need to create multiple CTEs. - -In the query below, to answer the question "where are the loons?", take the following steps: - -1. Join `birds.birds`, `birds.routes`, and `birds.observations` on the bird ID and route IDs where the bird name is "Common Loon". -2. Collect the resulting birdwatcher route geometries (`routes.geom`) into one geometry (a [MultiPoint](multipoint.html)). -3. Give the resulting table a name, `loon_sightings`, and query against it. In this case the query is rather simple: since the geometries have been collected into one in step 2 above, output the geometry as [GeoJSON](geojson.html) so the result can be pasted into to generate a map of the sightings. - -{% include_cached copy-clipboard.html %} -~~~ sql -WITH - loon_sightings - AS ( - SELECT - st_collect(routes.geom) AS the_geom - FROM - birds.birds, - birds.observations, - birds.routes - WHERE - birds.name = 'Common Loon' - AND birds.id = observations.bird_id - AND observations.route_id = routes.id - ) -SELECT - st_asgeojson(the_geom) -FROM - loon_sightings; -~~~ - -~~~ - st_asgeojson --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - {"type":"MultiPoint","coordinates":[[-75.078218,43.662991],[-75.078218,43.662991],[-75.078218,43.662991],[-75.078218,43.662991],[-75.078218,43.662991],[-75.078218,43.662991],[-75.078218,43.662991],[-74.389678,43.652701],[-74.389678,43.652701],[-74.389678,43.652701],[-74.389678,43.652701],[-74.389678,43.652701],[-74.389678,43.652701],[-74.389678,43.652701],[-74.389678,43.652701],[-74.389678,43.652701],[-74.9492509,43.6625207],[-74.473788,43.726581],[-74.473788,43.726581],[-74.473788,43.726581],[-74.234828,43.971271],[-74.234828,43.971271],[-74.234828,43.971271],[-74.234828,43.971271],[-74.234828,43.971271],[-74.234828,43.971271],[-74.3975289,43.5654587],[-74.3975289,43.5654587],[-74.7918689,43.7217467],[-74.7918689,43.7217467],[-74.7918689,43.7217467],[-74.7918689,43.7217467],[-74.7918689,43.7217467],[-74.7918689,43.7217467],[-74.7918689,43.7217467],[-74.7918689,43.7217467],[-74.0622389,43.6884937],[-74.0622389,43.6884937],[-73.7371009,43.6098217],[-75.0557089,44.4354227],[-74.9230359,44.1319167],[-74.9230359,44.1319167],[-74.9230359,44.1319167],[-74.9230359,44.1319167],[-74.9230359,44.1319167],[-74.9230359,44.1319167],[-74.9230359,44.1319167],[-74.9230359,44.1319167],[-74.9230359,44.1319167],[-74.9230359,44.1319167],[-74.9230359,44.1319167],[-74.1807169,44.4319917],[-74.1807169,44.4319917],[-74.1807169,44.4319917],[-74.1807169,44.4319917],[-74.1807169,44.4319917],[-74.1807169,44.4319917],[-74.1807169,44.4319917],[-74.1807169,44.4319917],[-74.1807169,44.4319917],[-74.1807169,44.4319917],[-74.1807169,44.4319917],[-74.1807169,44.4319917],[-74.1807169,44.4319917],[-74.1807169,44.4319917],[-74.1807169,44.4319917],[-74.1807169,44.4319917],[-74.1807169,44.4319917],[-74.7210849,44.5376687],[-74.7210849,44.5376687],[-74.7210849,44.5376687],[-73.4887779,44.4433387],[-73.4887779,44.4433387],[-75.7014799,44.1487467],[-75.7014799,44.1487467],[-75.7014799,44.1487467],[-75.7014799,44.1487467],[-75.7014799,44.1487467],[-74.9852279,42.0781957],[-74.9852279,42.0781957],[-74.966888,43.679941],[-74.966888,43.679941],[-74.966888,43.679941],[-74.966888,43.679941],[-74.966888,43.679941],[-74.966888,43.679941],[-74.966888,43.679941],[-74.966888,43.679941],[-74.966888,43.679941],[-74.966888,43.679941],[-74.966888,43.679941],[-74.492148,44.522881],[-74.492148,44.522881],[-74.492148,44.522881],[-74.6305979,44.1294587],[-74.6305979,44.1294587],[-74.6305979,44.1294587],[-74.6305979,44.1294587],[-74.6305979,44.1294587],[-74.6305979,44.1294587],[-74.6305979,44.1294587],[-74.6305979,44.1294587],[-74.6305979,44.1294587],[-74.6305979,44.1294587],[-74.6305979,44.1294587]]} -(1 row) -~~~ - -Paste the result above into and you should see the following map, with gray markers for each loon sighting from the bird survey. - -Common Loon sightings in the years 2000-2019 in NY state - -### (2) What is the total area of Loon sightings? - -Now that you have some sense of how loon sightings are distributed across the state, you may wonder: What is the area of the loon's approximate habitat (per these sightings) in New York State? - -To find the answer: - -1. Collect the geometries of all loon sightings together in a CTE as one geometry. -2. Get the area of the [convex hull](st_convexhull.html) of the resulting geometry. -3. Because the `birds.routes` data uses [SRID 4326](srid-4326.html), the resulting area is measured in degrees, which can be converted to square miles by casting the data to a `GEOGRAPHY` type and dividing by 1609 (the number of meters in a mile) squared. - -{% include_cached copy-clipboard.html %} -~~~ sql -WITH - loon_sightings - AS ( - SELECT - st_collect(routes.geom) AS the_geom - FROM - birds.birds, - birds.observations, - birds.routes - WHERE - birds.name = 'Common Loon' - AND birds.id = observations.bird_id - AND observations.route_id = routes.id - ) -SELECT - st_area(st_convexhull(the_geom)::GEOGRAPHY) / 1609 ^ 2 -FROM - loon_sightings; -~~~ - -The result is an area of about 10,000 square miles, which can be visualized by a box with sides that are a little over 100 miles long. This answer can be verified by looking at the map output from [#1][q_01] and noting that most of the observation locations are well within 100 miles of each other. - -~~~ - ?column? ----------------------- - 10430.691899422598 -(1 row) -~~~ - -### (3) How many Loon sightings have there been in NY state in the years 2000-2019? - -In addition to the [loon observation point locations][q_01] and the [area of those observations][q_02] (which you can take as a proxy for the size of the loon's local habitat), you may want to know exactly how many times a loon has been sighted in the area. - -To find the answer: - -1. Join `birds.birds` and `birds.observations` on the bird ID where the bird name is "Common Loon". -2. Sum all of the sightings; the `GROUP BY` on bird names is necessary due to the use of the `sum` aggregate function. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT - birds.name, sum(observations.count) AS sightings -FROM - birds.birds, birds.observations -WHERE - birds.name = 'Common Loon' - AND birds.id = observations.bird_id -GROUP BY - birds.name; -~~~ - -~~~ - name | sightings ---------------+------------ - Common Loon | 269 -(1 row) -~~~ - -### (4) How many Loon sightings were there in NY state in just the year 2019? - -You might like to get a sense of how many of the loon sightings were more recent. For example, if there have been fewer sightings of loons in recent years, you might wonder if the population were declining in NY State. - -To determine how many sightings there were in NY state in 2019, re-use the query from [#3][q_03], with the additional constraint that the observation year is 2019. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT - birds.name, sum(observations.count) AS sightings -FROM - birds.birds, birds.observations -WHERE - birds.name = 'Common Loon' - AND birds.id = observations.bird_id - AND observations.year = 2019 -GROUP BY - birds.name; -~~~ - -If the sightings are evenly distributed, you might expect this to yield about 13 sightings. In fact, it's pretty close. This suggests that the loon population may have remained fairly stable over the years 2000-2019. - -~~~ - name | sightings ---------------+------------ - Common Loon | 12 -(1 row) -~~~ - -### (5) What is the density of Loon sightings per square mile in its range as defined by this data set in 2019? - -Since you are planning to do some hiking back into the woods to find the actual loons, you may get curious as to how densely the loon population is scattered over its habitat. From a practical perspective, the distribution is not actually even, since loons are most frequently located on lakes and ponds. Even so, it may serve as a useful proxy for the general areas in which to focus your birdwatching travels. - -To answer this question: - -1. Build a CTE that returns both the convex hull of loon habitat, as well as the sum of all loon observations in NY. -2. Query the result table of the CTE from step 1 to divide the number of sightings by the area of the loon's habitat (the convex hull). As in [#2][q_02], do some arithmetic to convert from the unit of degrees returned by [SRID 4326](srid-4326.html) to square miles. - -{% include_cached copy-clipboard.html %} -~~~ sql -WITH - loon_habitat - AS ( - SELECT - sum(observations.count) AS sightings, - st_convexhull(st_collect(routes.geom)) - AS the_hull - FROM - birds.birds, - birds.observations, - birds.routes - WHERE - birds.name = 'Common Loon' - AND birds.id = observations.bird_id - AND observations.route_id = routes.id - AND observations.year = 2019 - ) -SELECT - loon_habitat.sightings::FLOAT8 - / ( - SELECT - st_area(loon_habitat.the_hull::GEOGRAPHY) - / 1609 ^ 2 - ) -FROM - loon_habitat; -~~~ - -It turns out that this is a pretty small number. There are not very many loon sightings per square mile within its overall habitat area, since lakes and ponds make up a small portion of the physical space inside the convex hull of observations. - -~~~ - ?column? ------------------------- - 0.005078531189694916 -(1 row) -~~~ - -## Step 5. Find bookstores to visit - -### (6) What are the bookstores that lie within the Loon's habitat range in NY state? - -Now that you have asked (and answered) some exploratory questions that may inform your birdwatching activities while on vacation, it would be good to start thinking about what bookstores to visit as part of your travels. - -A natural question that arises is: given that you are looking for loon habitat, what are the bookstores located within that habitat? - -To answer this question: - -1. Build a CTE that returns the [convex hull](st_convexhull.html) of Common Loon habitat. -2. Join the results of the above CTE with a query against [the `bookstores` table](#the-bookstores-and-bookstore_routes-tables) that checks whether a bookstore's location is [contained](st_contains.html) by the loon habitat. Note that the query below [orders by](order-by.html) the store geometries so that stores in the list are clustered by location. This ordering may be useful if you want to travel between nearby stores. For more information about how this ordering is calculated, see [How CockroachDB's spatial indexing works](spatial-indexes.html#how-cockroachdbs-spatial-indexing-works). - -{% include_cached copy-clipboard.html %} -~~~ sql -WITH - loon_sightings - AS ( - SELECT - st_convexhull(st_collect(routes.geom)) - AS loon_hull - FROM - birds.birds, - birds.observations, - birds.routes - WHERE - birds.name = 'Common Loon' - AND birds.id = observations.bird_id - AND observations.route_id = routes.id - ) -SELECT - name, street, city, state, url, phone -FROM - tutorial.bookstores, loon_sightings -WHERE - st_contains(loon_hull, geom) -ORDER BY - geom; -~~~ - -~~~ - name | street | city | state | url | phone ----------------------------------------------------+-----------------------+----------------+-------+-------------------------------------------------+----------------- - Blacktree Books | 5006 State Highway 23 | Oneonta | NY | https://blacktreeoneonta.square.site/ | 6074321200 - The Green Toad Bookstore | 198 Main St | Oneonta | NY | http://www.greentoadbookstore.com | (607) 433-8898 - The Treehouse Reading and Arts Center | 587 Main St Ste 304 | New York Mills | NY | http://treehousebookshop.com | 315-765-6262 - Gansevoort House Books at Gems Along the Mohawk | 800 Mohawk St | Herkimer | NY | | - Gansevoort House Books at The Shoppes at 25 West | 25 W Mill Street | Little Falls | NY | http://www.gansevoorthouse.com/bookstore/ | - Mysteries On Main Street | 144 W Main St | Johnstown | NY | https://www.facebook.com/MysteriesOnMainStreet/ | (518) 736-2665 - The Bookstore Plus Music & Art | 2491 Main St | Lake Placid | NY | http://www.thebookstoreplus.com | (518) 523-2950 - The Book Nook (Saranac Lake, NY) | 7 Broadway | Saranac Lake | NY | https://www.facebook.com/slbooknook/ | 6315999511 -(8 rows) -~~~ - -### (7) How many different species of bird habitats contain the location of the Book Nook in Saranac Lake, NY? - -As a birdwatcher, you probably do not want to only see the Common Loon. As long as you are spending time outside, it would be nice to see what other species of bird are in the area. This may provide even more birdwatching fun. - -Since you know you will want to hit up [The Book Nook in Saranac Lake, NY](https://www.facebook.com/slbooknook/) at least once on your trip, you decide to see what bird species' habitats contain the store's location. That way you can get a sense of the diversity of bird species in the area. - -To answer this question: - -1. Build a CTE that returns some information about the bookstore you want to visit. -2. Build another CTE that returns information about the habitats of birds observed in NY state, and collects the habitat geometries together into one geometry. -3. Join the results of the above CTEs and query the count of birds whose habitats contain the location of the bookstore. - -{{site.data.alerts.callout_info}} -The final [`SELECT`](selection-queries.html) in the query below is doing a join that will not benefit from [spatial indexing](spatial-indexes.html), since both sides of the join are the results of [CTEs](common-table-expressions.html), and are therefore not indexed. -{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ sql -WITH - the_book_nook - AS ( - SELECT - bookstores.name, street, city, state, geom - FROM - tutorial.bookstores - WHERE - state = 'NY' AND city = 'Saranac Lake' - ), - local_birds - AS ( - SELECT - birds.name, - st_convexhull(st_collect(routes.geom)) - AS the_hull - FROM - birds.birds, - birds.observations, - birds.routes - WHERE - birds.id = observations.bird_id - AND observations.route_id = routes.id - GROUP BY - birds.name - ) -SELECT - count(local_birds.name) -FROM - local_birds, the_book_nook -WHERE - st_contains(local_birds.the_hull, the_book_nook.geom); -~~~ - -This is encouraging; there are over 120 species of birds for you to choose from whose habitats include the bookstore you want to visit! - -~~~ - count ---------- - 124 -(1 row) -~~~ - -You can verify the results by checking how many bird species have been sighted by the bird survey overall; you should expect that this number will be much larger than 124, and indeed it is: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT COUNT(name) FROM birds.birds; -~~~ - -~~~ - count ---------- - 756 -(1 row) -~~~ - -### (8) Which 25 birds were most often sighted within 10 miles of the Book Nook in Saranac Lake, NY during the 2000-2019 observation period? - -It's great that you [know how many bird species may be near a given bookstore][q_07]; however, that query didn't tell you which birds you have the best chance of seeing. Therefore, you'd like to figure out what birds are the most commonly seen in the area near a bookstore you want to visit: The Book Nook in Saranac Lake, NY. - -To answer this question: - -1. Build a CTE that returns some information about the bookstore you want to visit. -2. Join the results of the above CTE and a query against the `birds` database that lists the names and observation totals (sums) of birds whose habitats are within 10 miles of the location of the bookstore. - -{{site.data.alerts.callout_info}} -The query below can also be written using an explicit `ST_DWithin`, which is an [index-accelerated function](spatial-data.html#performance). CockroachDB optimizes `ST_Distance(...) < $some_value` to use `ST_DWithin` (see this query's [`EXPLAIN`](explain.html) output for details). -{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ sql -WITH - the_book_nook - AS ( - SELECT - bookstores.name, street, city, state, geom - FROM - tutorial.bookstores - WHERE - state = 'NY' AND city = 'Saranac Lake' - ) -SELECT - birds.name, sum(observations.count) AS sightings -FROM - birds.birds, - birds.observations, - birds.routes, - the_book_nook -WHERE - birds.id = observations.bird_id - AND observations.route_id = routes.id - AND st_distance( - the_book_nook.geom::GEOGRAPHY, - routes.geom::GEOGRAPHY - ) - < (1609 * 10) -GROUP BY - birds.name -ORDER BY - sightings DESC -LIMIT - 25; -~~~ - -Perhaps surprisingly, the Red-eyed Vireo is the "winner", followed by a number of other fairly common birds. If you want a birdwatching challenge, you can reverse the sort order of the above query to find the rarest birds. - -~~~ - name | sightings ------------------------------------------+------------ - Red-eyed Vireo | 2557 - White-throated Sparrow | 928 - Hermit Thrush | 924 - American Robin | 691 - Ovenbird | 650 - American Crow | 528 - (Myrtle Warbler) Yellow-rumped Warbler | 506 - Chipping Sparrow | 465 - Black-capped Chickadee | 390 - Yellow-bellied Sapsucker | 359 - Blue-headed Vireo | 357 - Blue Jay | 345 - Winter Wren | 344 - Cedar Waxwing | 272 - Blackburnian Warbler | 248 - Magnolia Warbler | 236 - Black-throated Green Warbler | 226 - Common Grackle | 218 - Red-breasted Nuthatch | 209 - Common Yellowthroat | 184 - Northern Parula | 175 - Nashville Warbler | 164 - Red-winged Blackbird | 152 - Least Flycatcher | 128 - American Redstart | 123 -(25 rows) -~~~ - -### (9) What does the shape of all bookstore locations that lie within the Loon's habitat look like? - -You [already discovered which bookstores are located within loon habitat][q_06]. However, you cannot really tell where these stores are located based on reading that query's output. You need to see them on the map. Therefore, for trip planning purposes, you decide you would like to look at the shape of the convex hull of the store locations. - -To answer this question: - -1. Build a CTE that returns the [convex hull](st_convexhull.html) of Common Loon habitat. -2. Join the results of the above CTE with a query against [the `bookstores` table](#the-bookstores-and-bookstore_routes-tables) that checks whether a bookstore's location is [contained](st_contains.html) by the loon habitat. -3. Collect the geometries that result from the step above into a single geometry, calculate its convex hull, and return the results as [GeoJSON](geojson.html). - -{% include_cached copy-clipboard.html %} -~~~ sql -WITH - loon_habitat - AS ( - SELECT - st_convexhull(st_collect(routes.geom)) - AS the_hull - FROM - birds.birds, - birds.observations, - birds.routes - WHERE - birds.name = 'Common Loon' - AND birds.id = observations.bird_id - AND observations.route_id = routes.id - ) -SELECT - st_asgeojson(st_convexhull(st_collect(geom))) -FROM - tutorial.bookstores, loon_habitat -WHERE - st_contains(the_hull, geom); -~~~ - -~~~ - st_asgeojson ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - {"type":"Polygon","coordinates":[[[-75.036815,42.448793],[-75.062138,42.454003],[-75.296575,43.098322],[-74.131673,44.326892],[-73.980251,44.279377],[-74.374175,43.006495],[-75.036815,42.448793]]]} -(1 row) -~~~ - -Paste the result above into and you should see the following map: - -Convex hull of bookstore locations within Common Loon habitat - -### (10) What is the area of the shape of all bookstore locations that are in the Loon's habitat range within NY state? - -You have already [visualized the convex hull][q_09], but now you would like to calculate its area in square miles. - -To answer this question: - -1. Build a CTE that returns the [convex hull](st_convexhull.html) of Common Loon habitat. -2. Join the results of the above CTE with a query against [the `bookstores` table](#the-bookstores-and-bookstore_routes-tables) that checks whether a bookstore's location is [contained](st_contains.html) by the loon habitat. -2. Get the area of the [convex hull](st_convexhull.html) of the resulting geometry. -3. Collect the geometries that result from the step above into a single geometry, calculate its convex hull, and calculate the area of the hull. As in previous examples, note that because the `birds.routes` data uses [SRID 4326](srid-4326.html), the resulting area is measured in degrees, which is converted to square miles by casting the data to a `GEOGRAPHY` type and dividing by 1609 (the number of meters in a mile) squared. - -{% include_cached copy-clipboard.html %} -~~~ sql -WITH - loon_habitat - AS ( - SELECT - st_convexhull(st_collect(routes.geom)) - AS the_hull - FROM - birds.birds, - birds.observations, - birds.routes - WHERE - birds.name = 'Common Loon' - AND birds.id = observations.bird_id - AND observations.route_id = routes.id - ) -SELECT - st_area(st_convexhull(st_collect(geom))::GEOGRAPHY) - / 1609 ^ 2 -FROM - tutorial.bookstores, loon_habitat -WHERE - st_contains(the_hull, geom); -~~~ - -The result is an area of about 3,500 square miles, which can be visualized by a box with sides that are approximately 60 miles long. This answer can be verified by looking at the map output from [#9][q_09] and noting that most of the points on the hull are within about 60 miles of each other. - -~~~ - ?column? ----------------------- - 3564.1722404508437 -(1 row) -~~~ - -## Step 6. Plan your route - -### (11) How long is the route from Mysteries on Main Street in Johnstown, NY to The Book Nook in Saranac Lake, NY? - -So far, you have written queries that ask questions about bird habitats, bookstore locations, or some combination of the two. Because you are planning a trip, you want to think about how to travel between the bookstores in the loon habitat area. Depending on how far apart the bookstores are, you may want to visit more than one of them in a day, perhaps doing some birdwatching along the way. - -In order to accomplish this, you start looking at [the `bookstore_routes` table](#the-bookstores-and-bookstore_routes-tables) to see how far the distance is in miles between two of the bookstores you'd like to visit. - -To answer this question: - -1. Issue subqueries that find the IDs of two of the bookstores you'd like to travel between, as start and end points. -2. Measure the length of the geometry that corresponds to those start and end IDs. Note that because the `bookstore_routes.geom` column has a SRID of 0 (which it inherited from the `roads` database from which it was created), you can convert to miles by casting the data to a `GEOGRAPHY` type, which uses meters, and then dividing by 1609 (the number of meters in a mile). - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT - st_length(geom::GEOGRAPHY) / 1609 -FROM - bookstore_routes -WHERE - end_store_id - = ( - SELECT - id - FROM - tutorial.bookstores - WHERE - city = 'Johnstown' AND state = 'NY' - ) - AND start_store_id - = ( - SELECT - id - FROM - tutorial.bookstores - WHERE - city = 'Saranac Lake' AND state = 'NY' - ); -~~~ - -The answer is: these two stores are about 132 miles apart, measured in road miles. - -~~~ - ?column? ---------------------- - 132.6837073049756 -(1 row) -~~~ - -### (12) What does the route from Mysteries on Main Street in Johnstown, NY to The Book Nook in Saranac Lake, NY look like? - -You have [determined how long the drive between these two stores will be][q_11], but you would like to see what it looks like on the map. - -To find this out, you can re-use the query from the previous question, using `ST_AsGeoJSON` instead of `ST_Length`, and skipping the distance math to generate output in miles. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT - st_asgeojson(geom) -FROM - bookstore_routes -WHERE - end_store_id - = ( - SELECT - id - FROM - tutorial.bookstores - WHERE - city = 'Johnstown' AND state = 'NY' - ) - AND start_store_id - = ( - SELECT - id - FROM - tutorial.bookstores - WHERE - city = 'Saranac Lake' AND state = 'NY' - ); -~~~ - -The result is a very large chunk of JSON: - -~~~ - st_asgeojson ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - {"type":"MultiLineString","coordinates":[[[-74.152265,44.317412001],[-74.150683244,44.320361245],[-74.149541,44.322491001]],[[-74.357329,44.244378001],[-74.356093,44.243859001],[-74.354178,44.242898001],[-74.348143,44.239648001],[-74.343626,44.237058001],[-74.342321,44.236058001],[-74.341749001,44.235410001],[-74.33975,44.232579001],[-74.339208,44.231977001],[-74.338667001,44.231599001],[-74.337461,44.231248001],[-74.326169,44.228478001],[-74.324681,44.228200001],[-74.322827,44.228066001],[-74.319936001,44.228040001],[-74.318920999,44.228200001],[-74.316251,44.228909001],[-74.315228,44.229337001],[-74.313824,44.230576001],[-74.310864,44.234299001],[-74.308766,44.236287001],[-74.30769,44.237137001],[-74.306668,44.237778001],[-74.305104,44.238369001],[-74.300305,44.239258001],[-74.298489,44.239849001],[-74.296765,44.240559001],[-74.293965,44.241978001],[-74.285755,44.246469001],[-74.284199,44.246998001],[-74.283496,44.247158001],[-74.278659999,44.246807001],[-74.277065,44.246487001],[-74.27103,44.244926001],[-74.269977,44.244766001],[-74.268862,44.244926001],[-74.267528001,44.245315001],[-74.264667,44.246425001],[-74.263103,44.246887001],[-74.262439,44.246956001],[-74.261516,44.246887001],[-74.258806999,44.246215001],[-74.258105,44.246128001],[-74.256106,44.246376001],[-74.249583,44.248748001],[-74.242563999,44.250486001],[-74.240374,44.251056001],[-74.237955,44.251956001],[-74.236117,44.252757001],[-74.234713,44.253264001],[-74.233477,44.253375001],[-74.232264001,44.253214001],[-74.230066,44.252714001],[-74.228067,44.252554001],[-74.224855,44.252695001],[-74.222374999,44.253027001],[-74.220468,44.253485001],[-74.219284999,44.253916001],[-74.218202,44.254465001],[-74.217285999,44.255606001],[-74.216554001,44.256914001],[-74.216272,44.258444001],[-74.216172,44.260206001],[-74.215837,44.261663001],[-74.215417,44.262537001],[-74.215105,44.262884001],[-74.210526,44.266176001],[-74.209443,44.267164001],[-74.205635,44.272016001],[-74.204712,44.272855001],[-74.203781,44.273633001],[-74.20137,44.275014001],[-74.199875,44.276155001],[-74.189735,44.286454001],[-74.1883,44.287625001],[-74.187415,44.288033001],[-74.186333,44.288353001],[-74.184364869,44.288747417],[-74.184364734,44.288747444],[-74.180854001,44.289451001],[-74.178374,44.290524001],[-74.177191,44.291213001],[-74.176085,44.292084001],[-74.175354,44.292725001],[-74.174521,44.293915001],[-74.173979,44.295353001],[-74.173468,44.297363001],[-74.172934001,44.298374001],[-74.166349,44.306545001],[-74.165144,44.308003001],[-74.160711,44.311824001],[-74.159916999,44.312461001],[-74.157789,44.313514001],[-74.154476999,44.315021001],[-74.153623,44.315642001],[-74.152265,44.317412001]],[[-74.464373,44.224200001],[-74.462122,44.224250001],[-74.461046,44.224318001],[-74.455468999,44.22478],[-74.442636406,44.225904612],[-74.437326,44.226370001],[-74.436273,44.226599001],[-74.416718001,44.232618001],[-74.398811,44.238371001],[-74.396683,44.238729001],[-74.390892,44.239401001],[-74.386916999,44.240198001],[-74.376418001,44.242708001],[-74.37461,44.243078001],[-74.369964,44.243669001],[-74.358573,44.244409001],[-74.357329,44.244378001]],[[-74.538494,44.124160001],[-74.537784999,44.125049001],[-74.534733,44.128341001],[-74.533054,44.129662001],[-74.531116,44.130992001],[-74.528957,44.132541001],[-74.527345325,44.134460927],[-74.527338703,44.134468817],[-74.527332393,44.134476334],[-74.525867,44.136222001],[-74.525164999,44.137359001],[-74.524524,44.138992001],[-74.524012999,44.140750001],[-74.522853,44.151859001],[-74.522876,44.154721001],[-74.523166,44.156341001],[-74.523501999,44.157512001],[-74.523822,44.158702001],[-74.523914,44.160320001],[-74.523753999,44.161079001],[-74.523371,44.161991001],[-74.521656,44.164352001],[-74.518795,44.167732001],[-74.517734,44.168800001],[-74.515415,44.170791001],[-74.512392,44.173229001],[-74.511211,44.174079001],[-74.507396,44.176181001],[-74.500621,44.180629001],[-74.492266,44.190040001],[-74.487185,44.193339001],[-74.485827,44.194141001],[-74.480456,44.196220001],[-74.478991,44.196551001],[-74.474985,44.197211001],[-74.472384001,44.197902001],[-74.471864933,44.198667258],[-74.471781,44.198791001],[-74.471609775,44.199419986],[-74.47160767,44.199427717],[-74.471357398,44.200347074],[-74.471238999,44.200782001],[-74.470742999,44.204081001],[-74.470324,44.205100001],[-74.470011,44.205630001],[-74.468932391,44.207118271],[-74.468927664,44.207124794],[-74.467852,44.208609001],[-74.464525,44.213050001],[-74.462816,44.215560001],[-74.462626,44.216022001],[-74.462687,44.217509001],[-74.46288977,44.21846686],[-74.46306,44.219271001],[-74.463861,44.220321001],[-74.464914999,44.221049001],[-74.465776999,44.221599001],[-74.466311,44.222030001],[-74.466249999,44.222491001],[-74.465655,44.223059001],[-74.464373,44.224200001]],[[-74.529117,44.099819001],[-74.531985,44.102500001],[-74.532741,44.103572001],[-74.533183,44.104492001],[-74.533435,44.106109001],[-74.533527,44.108600001],[-74.534061,44.111190001],[-74.534503,44.112152001],[-74.535045,44.113109001],[-74.536502999,44.115379001],[-74.537357,44.116381001],[-74.537456,44.116455001]],[[-74.472935039,44.000000001],[-74.479534023,44.002542684],[-74.479984937,44.002716429],[-74.480005378,44.002724305],[-74.480329,44.002849001],[-74.48677,44.005149001],[-74.49295,44.007480001],[-74.496628,44.009689001],[-74.498024,44.010719001],[-74.499611,44.012371001],[-74.501511,44.014541001],[-74.503151,44.016508001],[-74.508523,44.023811001],[-74.511529,44.027950001],[-74.514778999,44.032821001],[-74.515222,44.034469001],[-74.515473999,44.036991001],[-74.516015,44.039112001],[-74.517175,44.041202001],[-74.517816,44.042251001],[-74.519449,44.044540001],[-74.520181,44.045272001],[-74.521486,44.046760001],[-74.52189,44.047332001],[-74.522234,44.048221],[-74.522074,44.052341001],[-74.521654,44.055591001],[-74.520983,44.059932001],[-74.520051,44.065921001],[-74.519023544,44.072237853],[-74.519022043,44.072247084],[-74.518511,44.075389001],[-74.518381,44.076869001],[-74.518374,44.078979001],[-74.518503,44.079730001],[-74.518976,44.081382001],[-74.519572001,44.082641001],[-74.52122,44.085311001],[-74.525523,44.093051001],[-74.52682,44.095809001],[-74.528056,44.098220001],[-74.5291102,44.099808753],[-74.529115821,44.099817225],[-74.529117,44.099819001]],[[-74.419277,43.968522001],[-74.420169,43.969804001],[-74.420413,43.970281001],[-74.420604,43.971353001],[-74.420643889,43.971779185],[-74.420644773,43.971788634],[-74.42081,43.973554001],[-74.421245,43.974141001],[-74.421848,43.974301001],[-74.426426993,43.974695925],[-74.42644422,43.974697411],[-74.427425,43.974782001],[-74.42815,43.975010001],[-74.428618155,43.975309199],[-74.428626193,43.975314337],[-74.429577,43.975922001],[-74.43047,43.976590001],[-74.431927,43.977391001],[-74.435977,43.978982001],[-74.437787,43.979302001],[-74.439526,43.979302001],[-74.443196,43.979050001],[-74.444340001,43.979092001],[-74.445003999,43.979191001],[-74.446484,43.979640001],[-74.449117,43.980812001],[-74.451276,43.981953001],[-74.453779,43.983780001],[-74.456640001,43.986951001],[-74.460989,43.992531001],[-74.461897,43.993790001],[-74.4634,43.995430001],[-74.464407001,43.996460001],[-74.465362,43.997082001],[-74.472935039,44.000000001]],[[-74.430422635,43.855487245],[-74.430823844,43.859778886],[-74.432091,43.864603001],[-74.430741,43.872534001],[-74.430741,43.873312001],[-74.431092,43.874205001],[-74.431824999,43.875052001],[-74.432267,43.875433001],[-74.433274,43.876513001],[-74.433854,43.877634001],[-74.433946,43.878454001],[-74.433725001,43.879091001],[-74.432962,43.880034001],[-74.431573,43.881113001],[-74.427842,43.882784001],[-74.424844,43.884363001],[-74.424027,43.884882001],[-74.418244,43.889235],[-74.417732,43.889895001],[-74.417481,43.890444001],[-74.417428,43.891023001],[-74.417481,43.891981001],[-74.417992999,43.893305001],[-74.418686999,43.894175001],[-74.419221,43.894613001],[-74.420586,43.895502001],[-74.422838,43.896364001],[-74.424661,43.896803001],[-74.428651,43.896982001],[-74.434976,43.898191001],[-74.435762,43.898394001],[-74.436464,43.898992001],[-74.436815,43.899541001],[-74.437037001,43.900682001],[-74.437006,43.902284001],[-74.436844816,43.902957433],[-74.436815,43.903082001],[-74.436568249,43.903468348],[-74.436564,43.903475001],[-74.435687001,43.905462001],[-74.435648,43.906812001],[-74.435877,43.907381001],[-74.436884,43.908254001],[-74.437456,43.908434001],[-74.439768,43.909033001],[-74.440714,43.909444001],[-74.442675,43.910814001],[-74.443247,43.911241001],[-74.443949,43.912252001],[-74.444194,43.912843001],[-74.444163,43.913831001],[-74.443477,43.917234001],[-74.443286,43.917814001],[-74.442683,43.918993001],[-74.442408,43.919753002],[-74.441707,43.924302001],[-74.442027,43.925031001],[-74.44321,43.926824001],[-74.444118,43.927891001],[-74.444727999,43.928422001],[-74.455486001,43.936163001],[-74.457503769,43.936996535],[-74.457628577,43.938124354],[-74.456197485,43.939437358],[-74.452444415,43.94344539],[-74.450565344,43.945532689],[-74.449504442,43.946317113],[-74.446375524,43.948497454],[-74.446308501,43.948545699],[-74.4417465,43.951936699],[-74.441713582,43.95196156],[-74.441577106,43.952073562],[-74.44144825,43.95219425],[-74.441440603,43.952201926],[-74.440944602,43.952701927],[-74.440831562,43.952823108],[-74.440719559,43.952959582],[-74.440616724,43.953103091],[-74.440523492,43.953253017],[-74.440440267,43.95340872],[-74.440417621,43.953455787],[-74.440279357,43.953750403],[-74.439810963,43.954070359],[-74.438161986,43.954733199],[-74.435710574,43.955480195],[-74.435629767,43.955506216],[-74.435464532,43.955568406],[-74.435303719,43.955641268],[-74.435303325,43.955641463],[-74.434326325,43.956123464],[-74.434253446,43.956160804],[-74.429912445,43.958468803],[-74.429830017,43.958514493],[-74.429792382,43.958536629],[-74.428869382,43.95908963],[-74.428757089,43.959160725],[-74.428613582,43.959263561],[-74.428602976,43.959271742],[-74.425443975,43.961720742],[-74.425061746,43.963823434],[-74.423999,43.966344001],[-74.419277,43.968522001]],[[-74.430422635,43.855487245],[-74.422981,43.853304001],[-74.422035,43.852503001],[-74.421463,43.851863001],[-74.42086,43.850794001],[-74.420173,43.849055001],[-74.417701001,43.847792001],[-74.416725,43.847434001],[-74.411512999,43.845713001],[-74.410461,43.845374001],[-74.405379,43.845622001],[-74.404302999,43.845653001],[-74.402564,43.845443001],[-74.391691,43.843593001],[-74.389043999,43.842933001],[-74.386389,43.842044001],[-74.383202099,43.84067843],[-74.383194658,43.840675242],[-74.383008908,43.840595648],[-74.380788,43.839644001],[-74.378271,43.838885001],[-74.370183,43.837283001],[-74.369199,43.836986001],[-74.367238,43.835895001],[-74.366452,43.835223001],[-74.362378,43.829894001],[-74.361927,43.829215001],[-74.360920001,43.826236001],[-74.360073,43.825275001],[-74.359158,43.824615001],[-74.356823,43.823444001],[-74.351642,43.821704001],[-74.350788,43.821453001],[-74.345065,43.820084001],[-74.339946,43.819373001],[-74.338977001,43.819164001],[-74.33768,43.818595001],[-74.336580999,43.817832001],[-74.334681,43.816055001],[-74.333987,43.815483001],[-74.33301,43.815112001],[-74.332346,43.815025001],[-74.328997,43.815135001],[-74.328265,43.815044001],[-74.32606,43.814525001],[-74.324885,43.814014001],[-74.323565,43.813075001],[-74.317789,43.807224001],[-74.316904,43.805984001],[-74.315865999,43.804916001],[-74.314164,43.803356001],[-74.312616,43.802234001],[-74.309617,43.800743001],[-74.304787,43.798774001],[-74.303597,43.798156002],[-74.302544,43.797374001],[-74.302017,43.796764001],[-74.301766,43.795703001],[-74.301293,43.789993001],[-74.301239,43.788345001],[-74.30156,43.787366001],[-74.301964,43.786903001],[-74.303009,43.785984001],[-74.303856,43.785053001],[-74.303766,43.784523001],[-74.303169,43.783584001],[-74.30237657,43.782755241],[-74.302368102,43.782746385],[-74.301773,43.782124001],[-74.300392,43.781143001],[-74.299507001,43.780723001],[-74.298248001,43.780403001],[-74.296165,43.780174001],[-74.294173999,43.780174001],[-74.293197,43.780403001],[-74.289786,43.781475001],[-74.283652001,43.782913001],[-74.281074,43.783226001],[-74.276053,43.783065001],[-74.266203,43.782344001],[-74.265295,43.782203001]],[[-74.363938,43.502013001],[-74.366326,43.504158001],[-74.367432,43.504927001],[-74.368592001,43.505598001],[-74.3695,43.506236001],[-74.369966,43.506716001],[-74.371263,43.508746001],[-74.371728,43.510188002],[-74.371759,43.510878001],[-74.371072,43.513346001],[-74.371126,43.514128001],[-74.371263001,43.514609001],[-74.371606001,43.515086001],[-74.372988,43.516226001],[-74.37459,43.516986001],[-74.377199,43.518266001],[-74.385432,43.522406001],[-74.386088,43.523017001],[-74.386408999,43.523497001],[-74.386691,43.524646001],[-74.386532553,43.526017531],[-74.38653144,43.526027172],[-74.386442444,43.526797514],[-74.386409,43.527087001],[-74.385913,43.528948001],[-74.385119,43.531458001],[-74.383929,43.534228001],[-74.383212,43.535826001],[-74.383021,43.537089001],[-74.382983,43.538138001],[-74.383334,43.538897001],[-74.384273,43.540125001],[-74.385593,43.541586001],[-74.390468001,43.545298001],[-74.391071,43.546008002],[-74.391918,43.547678001],[-74.395001,43.555265001],[-74.395131,43.556048001],[-74.395283,43.557879001],[-74.395321,43.561217001],[-74.395543,43.562746001],[-74.395909,43.563776],[-74.396512,43.564509001],[-74.397519,43.565466001],[-74.398803582,43.566517366],[-74.398814559,43.56652635],[-74.399221,43.566859001],[-74.399824,43.567705001],[-74.399944999,43.568259001],[-74.399945,43.569647001],[-74.399221,43.572188001],[-74.39919,43.573626001],[-74.399662,43.574797001],[-74.40064,43.575918001],[-74.404486,43.579416001],[-74.405202999,43.580538001],[-74.405584,43.581227001],[-74.405916539,43.5819675],[-74.405920409,43.581976118],[-74.406069135,43.582307303],[-74.407255,43.584948001],[-74.407721,43.586386001],[-74.408263,43.588358001],[-74.408606,43.591215001],[-74.408484,43.592638001],[-74.408454,43.594439001],[-74.408896,43.595877001],[-74.409934,43.596907001],[-74.410712,43.597319001],[-74.414336,43.598826001],[-74.418052,43.600195001],[-74.418937,43.600676001],[-74.419784,43.601568001],[-74.420166,43.602099001],[-74.420356,43.602716001],[-74.420326,43.603445001],[-74.419976,43.604178001],[-74.417839,43.607267001],[-74.416397,43.610147001],[-74.414604,43.614607001],[-74.414423322,43.616825244],[-74.412448675,43.618905862],[-74.40718547,43.621568425],[-74.402912985,43.623921388],[-74.397525939,43.62776043],[-74.395234896,43.628936913],[-74.392262735,43.632218675],[-74.39075766,43.634708852],[-74.389909772,43.637358042],[-74.388159464,43.640096794],[-74.387494889,43.644231168],[-74.387252,43.646085001],[-74.387122,43.646905001],[-74.387222,43.647615001],[-74.387664,43.648465001],[-74.388618,43.649698001],[-74.388953999,43.650544001],[-74.389282339,43.651568593],[-74.389285046,43.651577041],[-74.389527445,43.652333453],[-74.389847852,43.655810221],[-74.388573718,43.658372116],[-74.385712295,43.660061454],[-74.378887271,43.663353982],[-74.375864153,43.663837969],[-74.373971827,43.663959237],[-74.371619282,43.665055508],[-74.369785751,43.66516015],[-74.365134387,43.665658299],[-74.363288,43.666715001],[-74.362213,43.667745001],[-74.360854,43.669385001],[-74.360518,43.669934001],[-74.360167,43.670827001],[-74.356002,43.676476001],[-74.349882999,43.683015001],[-74.344527,43.687794001],[-74.343077,43.689374001],[-74.342131,43.691105001],[-74.340582,43.694836001],[-74.339514,43.696484],[-74.335066,43.700436001],[-74.334242,43.701874001],[-74.333174,43.704007001],[-74.331496,43.707844001],[-74.330802,43.709084001],[-74.330076,43.709927001],[-74.329352001,43.710366001],[-74.328215,43.710705001],[-74.324652001,43.710957001],[-74.320875,43.711563001],[-74.319922,43.711865001],[-74.319266,43.712277001],[-74.313734,43.718174001],[-74.313069,43.719135001],[-74.312566999,43.720524001],[-74.311713,43.722103001],[-74.309912,43.724556001],[-74.302595001,43.732204001],[-74.299216,43.735523001],[-74.297856999,43.736393001],[-74.294798006,43.740837095],[-74.294792616,43.740844925],[-74.292921,43.743564001],[-74.290266,43.747505001],[-74.289922999,43.747887001],[-74.287932,43.748733001],[-74.287207,43.749325001],[-74.286703,43.750084001],[-74.285567,43.752563001],[-74.285345,43.753802001],[-74.284903,43.754875001],[-74.283926,43.756206001],[-74.282431,43.757225001],[-74.278173999,43.761955001],[-74.276846,43.763714001],[-74.27561,43.764866001],[-74.274252,43.765823001],[-74.270308,43.768245001],[-74.265844,43.770633001],[-74.264074,43.773136001],[-74.263722,43.773906001],[-74.263662,43.775093001],[-74.263884,43.776195001],[-74.264570001,43.777404001],[-74.264853,43.778395001],[-74.265265,43.781635001],[-74.265295,43.782203001]],[[-74.538405,43.399411001],[-74.534987,43.400719001],[-74.533636,43.401360001],[-74.531745,43.402501001],[-74.530805999,43.403229001],[-74.529921,43.404080001],[-74.527311,43.407731001],[-74.526495,43.409058001],[-74.524985001,43.412920001],[-74.524451,43.413998001],[-74.523793,43.414979001],[-74.523108,43.415730001],[-74.522154001,43.416398001],[-74.521093999,43.417008001],[-74.518043001,43.418500001],[-74.513304,43.420670001],[-74.508344,43.422768001],[-74.507361,43.423131001],[-74.499571001,43.426739001],[-74.497915,43.427628001],[-74.495504,43.429299001],[-74.488104,43.434572002],[-74.486913001,43.435559001],[-74.483991,43.437459001],[-74.480543,43.439400001],[-74.476689,43.440937001],[-74.474523,43.441449001],[-74.473745,43.441516001],[-74.472646,43.441288001],[-74.47054,43.440808001],[-74.468564,43.440540001],[-74.467183,43.440702001],[-74.465771,43.441201001],[-74.457173,43.447567001],[-74.453511,43.450928001],[-74.452411999,43.452049001],[-74.451405,43.452919001],[-74.445484001,43.457390001],[-74.440525,43.460407001],[-74.43845,43.461510001],[-74.435283,43.462418001],[-74.432674,43.463798001],[-74.430072,43.466109001],[-74.428562001,43.466999001],[-74.423259001,43.469658001],[-74.421505,43.470231001],[-74.420529,43.470428001],[-74.414973,43.471008001],[-74.412081,43.470958001],[-74.411082001,43.470890001],[-74.408251,43.470256001],[-74.407214,43.470119001],[-74.406328,43.470231001],[-74.405237,43.470639001],[-74.404421,43.471187001],[-74.403751968,43.471760018],[-74.40374812,43.471763313],[-74.402247,43.473049001],[-74.398798,43.475128001],[-74.397257,43.475996001],[-74.395281,43.476959001],[-74.394808,43.477138001],[-74.39409,43.477649001],[-74.392458,43.479358001],[-74.392168001,43.479747001],[-74.389879,43.481849001],[-74.388879,43.482417001],[-74.388063,43.482768001],[-74.384286001,43.483890001],[-74.38231,43.484687001],[-74.379548,43.485988001],[-74.378007,43.487247001],[-74.37459,43.490497001],[-74.371728001,43.493628001],[-74.368516,43.497519001],[-74.368081,43.498088001],[-74.366921,43.499526001],[-74.363938,43.502013001]],[[-74.517516,43.229490001],[-74.517511546,43.229491206],[-74.51715,43.229589001],[-74.516121,43.230180001],[-74.515517999,43.231168001],[-74.515166985,43.234115798],[-74.516126976,43.235651784],[-74.516352909,43.23601328],[-74.516358236,43.236021803],[-74.516627882,43.236453236],[-74.517407028,43.237621954],[-74.517309635,43.23869328],[-74.516725275,43.241225503],[-74.516335703,43.242783794],[-74.516451181,43.24314947],[-74.516596508,43.243609679],[-74.516599124,43.243617961],[-74.517504421,43.246484736],[-74.517504421,43.249114353],[-74.517789825,43.250826773],[-74.517791285,43.250835533],[-74.517991388,43.25203615],[-74.517699208,43.253497048],[-74.517991387,43.25447098],[-74.519257499,43.255152732],[-74.522958442,43.255931877],[-74.528899427,43.255931877],[-74.534353447,43.255055339],[-74.534409397,43.255065212],[-74.534613985,43.255101316],[-74.534616201,43.255101707],[-74.536009132,43.255347518],[-74.539125714,43.256224057],[-74.540781399,43.258464101],[-74.541755331,43.259340641],[-74.543216229,43.260119786],[-74.544190161,43.261483291],[-74.544287555,43.263236368],[-74.545346767,43.265596244],[-74.547662904,43.26791238],[-74.54885406,43.269831465],[-74.549383463,43.271419675],[-74.550640795,43.272809358],[-74.551104022,43.274265215],[-74.552295179,43.275654897],[-74.553486335,43.276846053],[-74.553817212,43.277706333],[-74.55342016,43.278831314],[-74.552824581,43.280088646],[-74.553023108,43.281015101],[-74.553684861,43.282338608],[-74.553817212,43.283926816],[-74.553883387,43.286242954],[-74.553618685,43.287764988],[-74.553287809,43.289353196],[-74.553221634,43.291272281],[-74.553817213,43.293654593],[-74.556202148,43.296330374],[-74.55620838,43.296337366],[-74.556530402,43.296698659],[-74.557439202,43.297789221],[-74.557853909,43.298286869],[-74.561625903,43.299875077],[-74.563942041,43.300603006],[-74.565265548,43.301198584],[-74.567250809,43.303316195],[-74.566593,43.306008001],[-74.565959,43.306329001],[-74.565021001,43.306668001],[-74.564342,43.306809001],[-74.562518,43.306870001],[-74.561672,43.307118001],[-74.560832,43.307580001],[-74.56026,43.308179001],[-74.559603999,43.309251001],[-74.559513,43.310868001],[-74.557223999,43.314487001],[-74.556812,43.314969001],[-74.556179,43.315419001],[-74.551799,43.317689001],[-74.550791999,43.318028001],[-74.550205,43.318139001],[-74.54932,43.318120001],[-74.548602,43.317998001],[-74.546553305,43.317435031],[-74.546540117,43.317431407],[-74.546535,43.317430001],[-74.545314,43.317448001],[-74.543769359,43.317982042],[-74.54346,43.318089001],[-74.541895334,43.319480577],[-74.541874268,43.319499312],[-74.541227786,43.320074278],[-74.537982,43.322961001],[-74.536966999,43.324059001],[-74.534404,43.326890001],[-74.534053,43.327348001],[-74.534023,43.328469001],[-74.534145,43.329949001],[-74.53445,43.331300001],[-74.534633,43.335969001],[-74.534572,43.336610001],[-74.533748,43.339700001],[-74.532902,43.341908001],[-74.532711,43.342562001],[-74.532711,43.344109001],[-74.533154001,43.347749001],[-74.533329,43.348180001],[-74.534245,43.348569001],[-74.535931,43.348920001],[-74.539784,43.352101001],[-74.540059001,43.352578001],[-74.5425,43.359578001],[-74.542431,43.360329001],[-74.544652,43.365231001],[-74.545285,43.366009001],[-74.546719999,43.368889001],[-74.546911,43.369328],[-74.546941,43.370099001],[-74.546910434,43.37081979],[-74.546910043,43.370828991],[-74.546743,43.374768001],[-74.54523859,43.380839868],[-74.545236318,43.380849039],[-74.544607,43.383389001],[-74.544165,43.384350001],[-74.543592,43.385129001],[-74.543226,43.386269001],[-74.543211,43.387047001],[-74.543211,43.393700001],[-74.543363,43.395100001],[-74.543692001,43.395955001]],[[-74.523337,43.190378001],[-74.523589,43.190599001],[-74.524406,43.191679001],[-74.524398,43.193460001],[-74.526046,43.199522001],[-74.526756,43.201952001],[-74.526946,43.202500001],[-74.527358,43.203001001],[-74.528816,43.204241001],[-74.529228,43.204851001],[-74.530654,43.209131001],[-74.530807,43.211530001],[-74.531021,43.219041001],[-74.53099,43.219652001],[-74.530456,43.220411001],[-74.530075,43.220682001],[-74.529297,43.220998001],[-74.527389,43.221368001],[-74.526168,43.222299],[-74.52523,43.223351001],[-74.524131,43.225301],[-74.523285,43.226121001],[-74.520644999,43.228219001],[-74.519776,43.228719001],[-74.518394,43.229248001],[-74.517552345,43.229479983],[-74.517516,43.229490001]],[[-74.489522,43.144732001],[-74.487653,43.146842001],[-74.486974,43.147891001],[-74.486386,43.149222001],[-74.486196,43.149931001],[-74.486257,43.151370001],[-74.486982,43.157683001],[-74.487302,43.158499001],[-74.487553068,43.15885108],[-74.487558704,43.158858983],[-74.487645,43.158980001],[-74.488584,43.159922001],[-74.488995999,43.160261001],[-74.492902,43.163321001],[-74.493688,43.163782001],[-74.494749,43.164259001],[-74.498129,43.165240001],[-74.499593,43.166041002],[-74.506094,43.169752001],[-74.50688,43.170550001],[-74.507994,43.172289001],[-74.508909,43.172949001],[-74.509619,43.173319001],[-74.511809,43.174372001],[-74.512953,43.174563001],[-74.514395,43.174440001],[-74.516836999,43.174101001],[-74.517744999,43.174101001],[-74.522367999,43.174429001],[-74.525397,43.174822001],[-74.526427,43.175142001],[-74.527365999,43.175741001],[-74.527617001,43.176222001],[-74.527709,43.177180001],[-74.526206,43.180349001],[-74.524421,43.182619001],[-74.523757,43.183832001],[-74.523505,43.184771001],[-74.523482798,43.184871668],[-74.523481108,43.184879335],[-74.523350626,43.185470982],[-74.522689,43.188471001],[-74.522567,43.189432001],[-74.522849,43.189939001],[-74.523337,43.190378001]],[[-74.481137,43.137126001],[-74.477955,43.136122001],[-74.474796,43.133918001],[-74.473491999,43.132907001],[-74.469212,43.127990001],[-74.467541,43.125544001],[-74.462276001,43.121722001],[-74.461239002,43.120872001],[-74.451899999,43.111972001],[-74.44482,43.104675001],[-74.442248,43.100520001],[-74.441287,43.099738001],[-74.438442511,43.098239949],[-74.43842793,43.098232271],[-74.433276,43.095519001],[-74.427227,43.092842001],[-74.424792,43.092453001],[-74.423205,43.091941001],[-74.422495,43.091514001],[-74.421168,43.090263001],[-74.420733,43.089435001],[-74.420069,43.087520001],[-74.419657,43.084262001],[-74.418360001,43.081066001],[-74.416842,43.078594001],[-74.416178,43.077903001],[-74.409875999,43.074226001],[-74.402719,43.070606001],[-74.401857,43.070297001],[-74.396333,43.069805001],[-74.389688,43.068142001],[-74.388277001,43.067131001],[-74.383843,43.061993001],[-74.382844,43.060459001],[-74.381737,43.057114001],[-74.380936,43.055908001],[-74.37941,43.054917001],[-74.374505,43.053871001],[-74.373430143,43.05371745],[-74.372559,43.053593001],[-74.367921,43.052548001],[-74.366035999,43.051289001]],[[-74.366035999,43.051289001],[-74.365227,43.050747001],[-74.3619,43.050244001],[-74.360909,43.050106001],[-74.359956,43.050152001],[-74.354088001,43.051007001],[-74.352554,43.051045001],[-74.351057999,43.050888001],[-74.348739,43.050381001],[-74.346245,43.050488001]],[[-74.352012,43.027703001],[-74.350357,43.028802001],[-74.34217,43.035565001],[-74.335830001,43.039841001]],[[-74.360901,43.013665001],[-74.358557,43.013592001],[-74.357467,43.013772001],[-74.356567,43.014214001],[-74.353729,43.016545001],[-74.351463001,43.017712001]],[[-74.385208,43.015484001],[-74.384799414,43.015297285],[-74.374367565,43.010530139],[-74.374315078,43.009100739],[-74.374314756,43.009091976],[-74.374214,43.006348001]]]} -(1 row) -~~~ - -Paste the result above into and you should see the following map: - -What does the route from Mysteries on Main Street in Johnstown, NY to The Book Nook in Saranac Lake, NY look like? - -### (13) What were the 25 most-commonly-sighted birds in 2019 within 10 miles of the route between Mysteries on Main Street in Johnstown, NY and The Bookstore Plus in Lake Placid, NY? - -A natural question that arises once you start traveling between stores is: if I decide to visit two of these stores in a day, and do some birdwatching in between, what are some of the bird species I may expect to find? In this case you are looking at what species are near the route between [Mysteries on Main Street in Johnstown, NY](https://www.facebook.com/MysteriesOnMainStreet), and [The Bookstore Plus in Lake Placid, NY](http://www.thebookstoreplus.com). - -To answer this question: - -1. Build a CTE that returns the route between the two stores of interest (a geometry). -2. Join the results of the above CTE with a query against [the `birds` database](#the-birds-database) that checks whether the distance between the route geometry and the location of the bird observation (`birds.routes.geom`) bookstore's location is less than the desired length of 10 miles. Note that because the call to `ST_Distance` is operating on shapes cast to `GEOGRAPHY` data type, the results are in meters, which then have to be converted to miles by dividing the result by 1609 (the number of meters in a mile). - -{{site.data.alerts.callout_info}} -The query below can also be written using an explicit `ST_DWithin`, which is an [index-accelerated function](spatial-data.html#performance). CockroachDB optimizes `ST_Distance(...) < $some_value` to use `ST_DWithin` (see this query's [`EXPLAIN`](explain.html) output for details). -{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ sql -WITH - bookstore_trip - AS ( - SELECT - geom - FROM - bookstore_routes - WHERE - start_store_id - = ( - SELECT - id - FROM - tutorial.bookstores - WHERE - city = 'Johnstown' - AND state = 'NY' - ) - AND end_store_id - = ( - SELECT - id - FROM - tutorial.bookstores - WHERE - city = 'Lake Placid' - AND state = 'NY' - ) - ) -SELECT - birds.birds.name, - sum(birds.observations.count) AS sightings -FROM - birds.birds, - birds.observations, - birds.routes, - bookstore_trip -WHERE - birds.birds.id = birds.observations.bird_id - AND observations.route_id = routes.id - AND observations.year = 2019 - AND st_distance( - bookstore_trip.geom::GEOGRAPHY, - birds.routes.geom::GEOGRAPHY - ) - < (1609 * 10) -GROUP BY - birds.name -ORDER BY - sightings DESC -LIMIT - 25; -~~~ - -~~~ - name | sightings ------------------------------------------+------------ - Red-eyed Vireo | 124 - Ovenbird | 98 - Blue Jay | 56 - Black-capped Chickadee | 44 - American Robin | 43 - American Crow | 31 - White-throated Sparrow | 26 - Common Yellowthroat | 25 - Red-breasted Nuthatch | 19 - Blue-headed Vireo | 18 - Red-winged Blackbird | 16 - Veery | 16 - Winter Wren | 16 - American Redstart | 15 - Blackburnian Warbler | 15 - Chipping Sparrow | 14 - Least Flycatcher | 13 - Northern Parula | 13 - American Goldfinch | 13 - Magnolia Warbler | 12 - Cedar Waxwing | 12 - Song Sparrow | 11 - Hermit Thrush | 11 - Common Raven | 10 - (Myrtle Warbler) Yellow-rumped Warbler | 10 -(25 rows) -~~~ - -### (14) What are the 25 least often sighted other species of birds in the loon's sightings range? - -Since you are already looking for loons, you would like to know if there are other, more challenging birdwatching opportunities available within the loon habitat you are already visiting. Specifically, you would like to know: What are the least-often-observed bird species in the loon's habitat? - -To answer this question: - -1. Build a CTE that returns the [convex hull](st_convexhull.html) of Common Loon habitat. -2. Join the results of the above CTE and a query against [the `birds` database](#the-birds-database) that lists the names and observation totals (sums) of birds whose habitats are contained within the convex hull of loon habitat. - -{% include_cached copy-clipboard.html %} -~~~ sql -WITH - loon_habitat - AS ( - SELECT - st_convexhull(st_collect(routes.geom)) - AS loon_hull - FROM - birds.birds, - birds.observations, - birds.routes - WHERE - birds.name = 'Common Loon' - AND birds.id = observations.bird_id - AND observations.route_id = routes.id - ) -SELECT - birds.name, sum(birds.observations.count) AS sightings -FROM - birds.birds, - birds.observations, - birds.routes, - loon_habitat -WHERE - birds.id = observations.bird_id - AND observations.route_id = routes.id - AND st_contains(loon_hull, routes.geom) -GROUP BY - birds.name -ORDER BY - sightings ASC -LIMIT - 25; -~~~ - -~~~ - name | sightings ----------------------------------------------------+------------ - Vesper Sparrow | 1 - Northern Saw-whet Owl | 1 - Sora | 1 - Least Bittern | 1 - Palm Warbler | 1 - Ring-necked Duck | 1 - American Three-toed Woodpecker | 1 - unid. Yellow-billed Cuckoo / Black-billed Cuckoo | 1 - unid. Accipiter hawk | 1 - Eastern Whip-poor-will | 1 - Golden-winged Warbler | 1 - Eastern Screech-Owl | 1 - Northern Goshawk | 1 - Upland Sandpiper | 1 - Hooded Warbler | 2 - Grasshopper Sparrow | 2 - Gadwall | 2 - Brewster Warbler (Golden-winged x Blue-winged) | 2 - Double-crested Cormorant | 3 - White-winged Crossbill | 3 - Marsh Wren | 3 - Pine Siskin | 3 - Great Horned Owl | 3 - Northern Mockingbird | 3 - Red-shouldered Hawk | 4 -(25 rows) -~~~ - -## Step 7. Get road and travel info - -### (15) What are the top 10 roads nearest to a Loon sighting location in NY? - -So far, you have learned [where the birds are](#step-4-scout-loon-locations), [where the overlaps are between birds and bookstores](#step-5-find-bookstores-to-visit), and [how to travel between bookstores while finding birds](#step-6-plan-your-route). - -Now you will start asking questions that involve looking at [the `roads` table](#the-roads-table) as well. - -Specifically, you would like to know: What are the 10 roads that are closest to a loon observation site? The answer to this question may help you figure out exactly which roads you should travel down to start the search for loons. The closer a road is to the sighting location, the better: it means there is probably less work to do to see the birds you are looking for. - -To answer this question: - -1. Build a CTE called `loon_habitat` that collects all of the Common Loon sightings into a single geometry. -2. Build another CTE called `nearby_roads` that joins the results of the subquery above with [the `roads` table](#the-roads-table) and pulls out the roads in NY state that are within a degree (about 60 nautical miles). Order the roads returned by their distance from a loon sighting. This will return some duplicate roads (since loons can be sighted multiple times along a single road), which is why you need to `LIMIT` to 20 here so you can get the list down to 10 later. Because the data in the `roads` table has an SRID of 0, you need to use `ST_SetSRID` to set its SRID to [4326](srid-4326.html). This step is necessary because `ST_Distance` cannot operate on geometries with differing SRIDs. -3. Finally, query the results of the `nearby_roads` subquery to get a list of 10 distinct road names that you can plan to visit. - -{{site.data.alerts.callout_info}} -The query below can also be written using an explicit `ST_DWithin`, which is an [index-accelerated function](spatial-data.html#performance). CockroachDB optimizes `ST_Distance(...) < $some_value` to use `ST_DWithin` (see this query's [`EXPLAIN`](explain.html) output for details). -{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ sql -WITH - loon_habitat - AS ( - SELECT - st_collect(routes.geom) AS geom - FROM - birds.birds, - birds.observations, - birds.routes - WHERE - birds.name = 'Common Loon' - AND birds.id = observations.bird_id - AND observations.route_id = routes.id - ), - nearby_roads - AS ( - SELECT - roads.prime_name AS road_name - FROM - roads, loon_habitat - WHERE - st_distance( - loon_habitat.geom, - st_setsrid(roads.geom, 4326) - ) - < 1 - ORDER BY - st_distance( - loon_habitat.geom, - st_setsrid(roads.geom, 4326) - ) - ASC - LIMIT - 20 - ) -SELECT - DISTINCT road_name -FROM - nearby_roads -LIMIT - 10; -~~~ - -~~~ - prime_name ------------------------ - US Route 9 - State Route 30 - State Route 22 Spur - State Route 22 - State Route 28 - State Route 86 - State Route 28N - Interstate 87 - US Route 11 - State Route 421 -(10 rows) - -Time: 1.447s total (execution 1.446s / network 0.000s) -~~~ - -Unfortunately, this query is a bit slower than you would like: about 1.5 seconds on a single-node [`cockroach demo`](cockroach-demo.html) cluster on a laptop. There are several reasons for this: - -1. You haven't created any indexes at all yet. The query is likely to be doing full table scans, which you will need to hunt down with [`EXPLAIN`](explain.html). -2. CockroachDB does not yet have built-in support for index-based nearest neighbor queries. If this feature is important to you, please comment with some information about your use case on [cockroachdb/cockroach#55227](https://github.com/cockroachdb/cockroach/issues/55227). - -Let's look at the `EXPLAIN` output to see if there is something that can be done to improve this query's performance: - -{% include_cached copy-clipboard.html %} -~~~ sql -EXPLAIN WITH loon_habitat AS (SELECT st_collect(routes.geom) AS geom FROM birds.birds, birds.observations, birds.routes WHERE birds.name = 'Common Loon' AND birds.id = observations.bird_id AND observations.route_id = routes.id), nearby_roads AS (SELECT roads.prime_name AS road_name FROM roads, loon_habitat WHERE st_distance(loon_habitat.geom, st_setsrid(roads.geom, 4326)) < 1 ORDER BY st_distance(loon_habitat.geom, st_setsrid(roads.geom, 4326)) ASC LIMIT 20) SELECT DISTINCT road_name FROM nearby_roads LIMIT 10; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ - tree | field | description --------------------------------------------------------------------+---------------------+--------------------------------------------------------- - | distribution | full - | vectorized | true - limit | | - │ | count | 10 - └── distinct | | - │ | distinct on | road_name - └── render | | - └── limit | | - │ | count | 20 - └── sort | | - │ | order | +column48 - └── render | | - └── cross join | | - │ | pred | st_dwithinexclusive(geom, st_setsrid(geom, 4326), 1.0) - ├── scan | | - │ | estimated row count | 225838 - │ | table | roads@primary - │ | spans | FULL SCAN - └── render | | - └── group (scalar) | | - └── hash join | | - │ | equality | (route_id) = (id) - │ | right cols are key | - ├── hash join | | - │ │ | equality | (bird_id) = (id) - │ │ | right cols are key | - │ ├── scan | | - │ │ | Estimated row count | 85731 - │ │ | table | observations@primary - │ │ | spans | FULL SCAN - │ └── filter | | - │ │ | filter | name = 'Common Loon' - │ └── scan | | - │ | estimated row count | 756 - │ | table | birds@primary - │ | spans | FULL SCAN - └── scan | | - | estimated row count | 129 - | table | routes@primary - | spans | FULL SCAN -(40 rows) -~~~ - -As you can see above, there are several full table scans being performed (see `FULL SCAN`). For each table, you have a hypothesis on what is causing the scan; usually the hypothesis is that the table is missing an index. - -- For [the `roads` table](#the-roads-table), there is no [spatial index](spatial-indexes.html) on the `road.geom` column. This is probably the worst performance offender of the bunch, since this table has almost a quarter of a million rows. It makes sense that it's quite large, since it represents a road map of the entire United States. - - In addition to adding an index to `roads.geom`, you will need to update the SRID of the `roads.geom` column to use [SRID 4326](srid-4326.html) so it matches the SRID of the loon habitat geometry. Updating the SRID to match will allow you to stop using `ST_SetSRID` on `roads.geom` as shown above, which will keep CockroachDB from taking advantage of the spatial index on that column. -- On [the `birds.observations` table](#the-birds-database), there is no index on the `birds.observations.route_id` or `birds.observations.bird_id` columns, which are used for all bird information lookups. -- On the [`birds.birds` table](#the-birds-database), there is no index on the `birds.birds.name` column which you're filtering on (this is also used by almost all of your other bird-related queries). - -Based on these hypotheses, you take the following steps: - -1. Add an index to the `roads.geom` column. Note that creating the index on `roads.geom` takes about a minute on a single-node [`cockroach demo`](cockroach-demo.html) cluster, since the table is relatively large. - - {% include_cached copy-clipboard.html %} - ~~~ sql - CREATE INDEX ON roads USING GIST(geom); - ~~~ - -2. Update the SRID of the `roads.geom` column from 0 to [4326](srid-4326.html). This will take a few seconds. - - {% include_cached copy-clipboard.html %} - ~~~ sql - UPDATE roads SET geom = st_transform(st_setsrid(geom, 4326), 4326) WHERE gid IS NOT NULL RETURNING NOTHING; - ~~~ - -3. Add indexes on the `birds.observations.route_id` and `birds.observations.bird_id` columns: - - {% include_cached copy-clipboard.html %} - ~~~ sql - CREATE INDEX ON birds.observations(bird_id); - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - CREATE INDEX ON birds.observations(route_id); - ~~~ - -4. Add an index on the `birds.birds.name` column: - - {% include_cached copy-clipboard.html %} - ~~~ sql - CREATE INDEX ON birds.birds(name); - ~~~ - -Did adding all of these indexes and updating the road geometry SRIDs make this query any faster? Let's check. Note that the query below has been modified to no longer use the `ST_SetSRID` function. - -{% include_cached copy-clipboard.html %} -~~~ sql -WITH - loon_habitat - AS ( - SELECT - st_collect(routes.geom) AS geom - FROM - birds.birds, - birds.observations, - birds.routes - WHERE - birds.name = 'Common Loon' - AND birds.id = observations.bird_id - AND observations.route_id = routes.id - ), - nearby_roads - AS ( - SELECT - roads.prime_name AS road_name - FROM - roads, loon_habitat - WHERE - st_distance(loon_habitat.geom, roads.geom) - < 1 - ORDER BY - st_distance(loon_habitat.geom, roads.geom) - ASC - LIMIT - 20 - ) -SELECT - DISTINCT road_name -FROM - nearby_roads -LIMIT - 10; -~~~ - -It looks like the answer is yes; you have sped up this query by about 30%: - -~~~ - prime_name ------------------------ - US Route 9 - State Route 30 - State Route 22 Spur - State Route 22 - State Route 28 - State Route 86 - State Route 28N - Interstate 87 - US Route 11 - State Route 421 -(10 rows) - -Time: 998ms total (execution 998ms / network 0ms) -~~~ - -To see why, look at the [`EXPLAIN`](explain.html) output: - -{% include_cached copy-clipboard.html %} -~~~ sql -EXPLAIN WITH loon_habitat AS (SELECT st_collect(routes.geom) AS geom FROM birds.birds, birds.observations, birds.routes WHERE birds.name = 'Common Loon' AND birds.id = observations.bird_id AND observations.route_id = routes.ID), nearby_roads AS (SELECT roads.prime_name AS road_name FROM roads, loon_habitat WHERE st_distance(loon_habitat.geom, roads.geom) < 1 ORDER BY st_distance(loon_habitat.geom, roads.geom) ASC LIMIT 20) SELECT DISTINCT road_name FROM nearby_roads LIMIT 10; -~~~ - -~~~ - tree | field | description ---------------------------------------------------------------------------+-----------------------+---------------------------------------- - | distribution | full - | vectorized | false - limit | | - │ | count | 10 - └── distinct | | - │ | distinct on | road_name - └── render | | - └── limit | | - │ | count | 20 - └── sort | | - │ | order | +column49 - └── render | | - └── lookup join | | - │ | table | roads@primary - │ | equality | (gid) = (gid) - │ | equality cols are key | - │ | pred | st_dwithinexclusive(geom, geom, 1.0) - └── inverted join | | - │ | table | roads@roads_geom_idx - └── render | | - └── group (scalar) | | - └── hash join | | - │ | equality | (route_id) = (id) - │ | right cols are key | - ├── lookup join | | - │ │ | table | observations@primary - │ │ | equality | (id) = (id) - │ │ | equality cols are key | - │ └── lookup join | | - │ │ | table | observations@observations_bird_id_idx - │ │ | equality | (id) = (bird_id) - │ └── scan | | - │ | estimated row count | 1 - │ | table | birds@birds_name_idx - │ | spans | [/'Common Loon' - /'Common Loon'] - └── scan | | - | estimated row count | 129 - | table | routes@primary - | spans | FULL SCAN -(39 rows) -~~~ - -Based on the output above, you notice the following improvements: - -- The [spatial index](spatial-indexes.html) on `roads.geom` is being used by the join between that column and the loon habitat geometry. This is the biggest reason for improved performance, since [the `roads` table](#the-roads-table) has ~225,000 rows. -- You no longer appear to be scanning all ~85,000 rows of [the `birds.observations` table](#the-birds-database) thanks to the index on the `birds.observations.bird_id` column. -- You are probably getting a small speedup from the index on [`birds.birds.name`](#the-birds-database) (although that table is not very large, only about 750 rows). - -### (16) How many miles of roads are contained within the portion of the Loon habitat that lies within NY state? - -It may not be immediately relevant to your birdwatching or book-buying travels, but a question of general interest that could arise is: How many miles of roads are contained within loon habitat? - -To answer this: - -1. Build a CTE that returns the [convex hull](st_convexhull.html) of Common Loon habitat. -2. Join the results of the above CTE and a query against [the `roads` table](#the-roads-table) that sums the mileage of all roads that are contained within the convex hull of loon habitat. - -{{site.data.alerts.callout_info}} -Because you are using `ST_Contains`, the query below only sums the road mileages of roads whose geometries lie entirely within the loon habitat. -{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ sql -WITH - loon_habitat - AS ( - SELECT - st_convexhull(st_collect(routes.geom)) - AS hull - FROM - birds.birds, - birds.observations, - birds.routes - WHERE - birds.name = 'Common Loon' - AND birds.id = observations.bird_id - AND observations.route_id = routes.id - ) -SELECT - sum(roads.miles) -FROM - roads, loon_habitat -WHERE - roads.state = 'NY' - AND st_contains( - loon_habitat.hull, - roads.geom - ); -~~~ - -The answer to your question is that there are about 1,700 miles of roads within the Common Loon's habitat. Further, you can change the `WHERE` clause of the above query slightly to say `AND (NOT (ST_Contains(...)))` to find out how many miles of roads in NY state are _not_ within loon habitat (it's about 14,600). - -~~~ - sum ------------------------ - 1704.65731711040306 -(1 row) -~~~ - -### (17) Which bookstore in the Loon's region in NY has the fewest miles of roads within a 10 mile radius of the store? - -As you are driving around the Adirondacks, searching for loons as well as your next great read, the question occurs to you: Which bookstore is the most remotely located? This is important since one of your goals for this vacation is to "get away from it all." You do not have population density data, but you do have road data. Therefore you decide to use "miles of roads near the store" as a rough proxy for which store location is the most remote: the fewer miles of roads near the store, the more remote you can assume it is. - -To answer this question: - -1. Build a CTE that returns the [convex hull](st_convexhull.html) of Common Loon habitat. -2. Build a CTE that joins [the `bookstores` table](#the-bookstores-and-bookstore_routes-tables) and the results of the above subquery to generate a set of bookstores inside the loon's habitat area. -3. Finally, generate a query that joins the results of the above subquery against [the `roads` table](#the-roads-table) based on which roads are within a 10 mile radius. This generates a list of bookstores and the number of miles of nearby roads, sorted in order of which store has the fewest miles of road nearby. - -{% include_cached copy-clipboard.html %} -~~~ sql -WITH - loon_habitat - AS ( - SELECT - st_convexhull(st_collect(routes.geom)) - AS geom - FROM - birds.birds, - birds.observations, - birds.routes - WHERE - birds.name = 'Common Loon' - AND birds.id = observations.bird_id - AND observations.route_id = routes.id - ), - loon_bookstores - AS ( - SELECT - bookstores.name, - address, - bookstores.geom AS geom - FROM - bookstores, loon_habitat - WHERE - st_contains( - loon_habitat.geom, - bookstores.geom - ) - ) -SELECT - loon_bookstores.name, - address, - sum(roads.miles)::INT AS nearby_road_miles -FROM - roads, loon_bookstores -WHERE - st_distance( - loon_bookstores.geom, - roads.geom - ) - < (10 / 69) -GROUP BY - loon_bookstores.name, address -ORDER BY - nearby_road_miles ASC; -~~~ - -You get an answer, but unfortunately it's rather slow: about 6 seconds! - -~~~ - name | address | nearby_road_miles ----------------------------------------------------+------------------------------------------------+-------------------- - The Bookstore Plus Music & Art | 2491 Main St, Lake Placid, NY, 12946 | 40 - The Book Nook (Saranac Lake, NY) | 7 Broadway, Saranac Lake, NY, 12983 | 81 - Blacktree Books | 5006 State Highway 23, Oneonta, NY, 13820 | 106 - The Green Toad Bookstore | 198 Main St, Oneonta, NY, 13820 | 107 - Gansevoort House Books at Gems Along the Mohawk | 800 Mohawk St, Herkimer, NY, 13350 | 136 - Gansevoort House Books at The Shoppes at 25 West | 25 W Mill Street, Little Falls, NY, 13365 | 155 - The Treehouse Reading and Arts Center | 587 Main St Ste 304, New York Mills, NY, 13417 | 156 - Mysteries On Main Street | 144 W Main St, Johnstown, NY, 12095 | 172 -(8 rows) - -Time: 6.214s total (execution 6.214s / network 0.000s) -~~~ - -Let's look at the `EXPLAIN` output to see if there is something that can be done to improve this query's performance: - -{% include_cached copy-clipboard.html %} -~~~ sql -EXPLAIN WITH loon_habitat AS (SELECT st_convexhull(st_collect(routes.geom)) AS geom FROM birds.birds, birds.observations, birds.routes WHERE birds.name = 'Common Loon' AND birds.id = observations.bird_id AND observations.route_id = routes.id), loon_bookstores AS (SELECT bookstores.name, address, bookstores.geom AS geom FROM bookstores, loon_habitat WHERE st_contains(loon_habitat.geom, bookstores.geom)) SELECT loon_bookstores.name, address, sum(roads.miles)::INT AS nearby_road_miles FROM roads, loon_bookstores WHERE st_distance(loon_bookstores.geom, roads.geom) < (10 / 69) GROUP BY loon_bookstores.name, address ORDER BY nearby_road_miles ASC; -~~~ - -~~~ - tree | field | description -----------------------------------------------------------------+-----------------------+------------------------------------------------------- - | distribution | full - | vectorized | true - sort | | - │ | order | +nearby_road_miles - └── render | | - └── group | | - │ | group by | name, address - └── cross join | | - │ | pred | st_dwithinexclusive(geom, geom, 0.14492753623188406) - ├── scan | | - │ | estimated row count | 225838 - │ | table | roads@primary - │ | spans | FULL SCAN - └── render | | - └── cross join | | - │ | pred | st_contains(geom, geom) - ├── scan | | - │ | estimated row count | 2913 - │ | table | bookstores@primary - │ | spans | FULL SCAN - └── render | | - └── group (scalar) | | - └── hash join | | - │ | equality | (route_id) = (id) - │ | right cols are key | - ├── lookup join | | - │ │ | table | observations@primary - │ │ | equality | (id) = (id) - │ │ | equality cols are key | - │ └── lookup join | | - │ │ | table | observations@observations_bird_id_idx - │ │ | equality | (id) = (bird_id) - │ └── scan | | - │ | estimated row count | 1 - │ | table | birds@birds_name_idx - │ | spans | [/'Common Loon' - /'Common Loon'] - └── scan | | - | estimated row count | 129 - | table | routes@primary - | spans | FULL SCAN -(40 rows) -~~~ - -Looking at these results from the bottom up, you can see that: - -1. There is a full table scan happening on the `birds.birds.routes` table. Since it's so small (129 rows), let's move on for now. -2. There is a full table scan on the `bookstores` table. Since the predicate is [`ST_Contains`](st_contains.html), you probably need to add a [spatial index](spatial-indexes.html) on `bookstores.geom`. -3. There is a full table scan happening on the `roads` table. This is a serious problem due to the size of that table: ~225,000 rows. You already have an index on the `roads.geom` column for the `ST_DWithin` predicate to use, so you need to find another way to reduce the number of rows scanned. You can do what you did in [query 16][q_16] and add a check that the `roads.state = 'NY'`, since you are only looking at roads inside New York State. - -Based on these observations, you add an index to the `bookstores.geom` column: - -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE INDEX ON bookstores USING GIST(geom); -~~~ - -After adding the index, you modify your query as shown below to filter on `roads.state`. This looks only at roads in New York (as in [query 16][q_16]): - -{% include_cached copy-clipboard.html %} -~~~ sql -WITH - loon_habitat - AS ( - SELECT - st_convexhull(st_collect(routes.geom)) - AS geom - FROM - birds.birds, - birds.observations, - birds.routes - WHERE - birds.name = 'Common Loon' - AND birds.id = observations.bird_id - AND observations.route_id = routes.id - ), - loon_bookstores - AS ( - SELECT - bookstores.name, - address, - bookstores.geom AS geom - FROM - bookstores, loon_habitat - WHERE - st_contains( - loon_habitat.geom, - bookstores.geom - ) - ) -SELECT - loon_bookstores.name, - address, - sum(roads.miles)::INT8 AS nearby_road_miles -FROM - roads, loon_bookstores -WHERE - roads.state = 'NY' - AND st_distance( - roads.geom, - loon_bookstores.geom - ) - < (10 / 69) -GROUP BY - loon_bookstores.name, address -ORDER BY - nearby_road_miles ASC; -~~~ - -The steps above brings your query execution down to about 400ms! - -~~~ - name | address | nearby_road_miles ----------------------------------------------------+------------------------------------------------+-------------------- - The Bookstore Plus Music & Art | 2491 Main St, Lake Placid, NY, 12946 | 40 - The Book Nook (Saranac Lake, NY) | 7 Broadway, Saranac Lake, NY, 12983 | 81 - Blacktree Books | 5006 State Highway 23, Oneonta, NY, 13820 | 106 - The Green Toad Bookstore | 198 Main St, Oneonta, NY, 13820 | 107 - Gansevoort House Books at Gems Along the Mohawk | 800 Mohawk St, Herkimer, NY, 13350 | 136 - Gansevoort House Books at The Shoppes at 25 West | 25 W Mill Street, Little Falls, NY, 13365 | 155 - The Treehouse Reading and Arts Center | 587 Main St Ste 304, New York Mills, NY, 13417 | 156 - Mysteries On Main Street | 144 W Main St, Johnstown, NY, 12095 | 172 -(8 rows) - -Time: 376ms total (execution 376ms / network 0ms) -~~~ - -When you look at `EXPLAIN` for the modified query, you see why: the filter on `roads.state='NY'` means you are scanning far fewer columns of the `roads` table (~8,900), and the index on `bookstores.geom` means you are using that index now as well: - -{% include_cached copy-clipboard.html %} -~~~ sql -EXPLAIN WITH loon_habitat AS (SELECT st_convexhull(st_collect(routes.geom)) AS geom FROM birds.birds, birds.observations, birds.routes WHERE birds.name = 'Common Loon' AND birds.id = observations.bird_id AND observations.route_id = routes.id), loon_bookstores AS (SELECT bookstores.name, address, bookstores.geom AS geom FROM bookstores, loon_habitat WHERE st_contains(loon_habitat.geom, bookstores.geom)) SELECT loon_bookstores.name, address, sum(roads.miles)::INT8 AS nearby_road_miles FROM roads, loon_bookstores WHERE roads.state = 'NY' AND st_distance(roads.geom, loon_bookstores.geom) < (10 / 69) GROUP BY loon_bookstores.name, address ORDER BY nearby_road_miles ASC; -~~~ - -~~~ - tree | field | description ----------------------------------------------------------------------+-----------------------+------------------------------------------------------------------------- - | distribution | full - | vectorized | true - sort | | - │ | order | +nearby_road_miles - └── render | | - └── group | | - │ | group by | name, address - └── cross join | | - │ | pred | st_dwithinexclusive(geom, geom, 0.14492753623188406) - ├── index join | | - │ │ | table | roads@primary - │ └── scan | | - │ | estimated row count | 8899 - │ | table | roads@roads_state_idx - │ | spans | [/'NY' - /'NY'] - └── render | | - └── lookup join | | - │ | table | bookstores@primary - │ | equality | (id) = (id) - │ | equality cols are key | - │ | pred | st_contains(geom, geom) - └── inverted join | | - │ | table | bookstores@bookstores_geom_idx - └── render | | - └── group (scalar) | | - └── hash join | | - │ | equality | (route_id) = (id) - │ | right cols are key | - ├── lookup join | | - │ │ | table | observations@primary - │ │ | equality | (id) = (id) - │ │ | equality cols are key | - │ └── lookup join | | - │ │ | table | observations@observations_bird_id_idx - │ │ | equality | (id) = (bird_id) - │ └── scan | | - │ | estimated row count | 1 - │ | table | birds@birds_name_idx - │ | spans | [/'Common Loon' - /'Common Loon'] - └── scan | | - | estimated row count | 129 - | table | routes@primary - | spans | FULL SCAN -~~~ - -## Step 8. Find other birds - -### (18) What are the hawks sighted in the same region as the Common Loon? - -If you get tired of looking for Common Loons (and it's understandable that you might), you may want to widen your search to other types of birds. You have already looked at [different bird species based on frequency of observation][q_13]. Now you turn your attention to a specific type of bird: hawks. - -Due to your long birdwatching experience, you recall that hawks are in the family "Accipitridae". You can use this knowledge to answer the question: What hawks live in the same region as the Common Loon in New York State? - -To answer this question: - -1. Build a CTE that returns the [convex hull](st_convexhull.html) of Common Loon habitat. -2. Join the results of the above subquery with [the `birds` database](#the-birds-database) to find the names and observation counts of the birds that: - 1. Are in the family "Accipitridae". - 2. Have sighting locations whose geometry is contained by the hull describing the Common Loon's habitat. -3. Order the birds in the list by how frequently they are sighted, since you may want to look for the most common hawks first. - -{% include_cached copy-clipboard.html %} -~~~ sql -WITH - loon_habitat - AS ( - SELECT - st_collect(routes.geom) AS geom - FROM - birds.birds, - birds.observations, - birds.routes - WHERE - birds.name = 'Common Loon' - AND birds.id = observations.bird_id - AND observations.route_id = routes.id - ) -SELECT - birds.name, sum(birds.observations.count) AS sightings -FROM - birds.birds, - birds.observations, - birds.routes, - loon_habitat -WHERE - birds.family = 'Accipitridae' - AND birds.id = observations.bird_id - AND observations.route_id = routes.id - AND st_contains(loon_habitat.geom, routes.geom) -GROUP BY - birds.name -ORDER BY - sightings DESC; -~~~ - -Based on this data, it looks like the most commonly sighted hawk by far in the area over the years 2000-2019 is the Broad-winged Hawk: - -~~~ - name | sightings ------------------------+------------ - Broad-winged Hawk | 115 - Red-tailed Hawk | 33 - Northern Harrier | 12 - Red-shouldered Hawk | 11 - Sharp-shinned Hawk | 9 - Bald Eagle | 9 - Cooper Hawk | 7 - Northern Goshawk | 5 - unid. Accipiter hawk | 1 -(9 rows) -~~~ - -### (19) What if you also want to look for owls as well as hawks? - -If you are a fan of owls as well as hawks, you can make several updates to [the previous query][q_18], which just looked for hawks. In particular, you want to group the hawks together and the owls together. - -To answer this question: - -1. Build a CTE that returns the [convex hull](st_convexhull.html) of Common Loon habitat. -2. Join the results of the above subquery with [the `birds` database](#the-birds-database) to find the names and observation counts of the birds that: - 1. Are in the family "Accipitridae" (hawks) _or_ the family "Strigidae" (owls). - 2. Have sighting locations whose geometry is contained by the hull describing the Common Loon's habitat. -3. Group the birds by name and family, and within each grouping order the birds by how frequently they are sighted, since you may want to look for the most common hawks or owls first. - -{% include_cached copy-clipboard.html %} -~~~ sql -WITH - loon_habitat - AS ( - SELECT - st_collect(routes.geom) AS geom - FROM - birds.birds, - birds.observations, - birds.routes - WHERE - birds.name = 'Common Loon' - AND birds.id = observations.bird_id - AND observations.route_id = routes.id - ) -SELECT - birds.name, - birds.family, - sum(birds.observations.count) AS sightings -FROM - birds.birds, - birds.observations, - birds.routes, - loon_habitat -WHERE - ( - birds.family = 'Accipitridae' - OR birds.family = 'Strigidae' - ) - AND birds.id = observations.bird_id - AND observations.route_id = routes.id - AND st_contains(loon_habitat.geom, routes.geom) -GROUP BY - birds.name, birds.family -ORDER BY - birds.family, sightings DESC; -~~~ - -~~~ - name | family | sightings -------------------------+--------------+------------ - Broad-winged Hawk | Accipitridae | 115 - Red-tailed Hawk | Accipitridae | 33 - Northern Harrier | Accipitridae | 12 - Red-shouldered Hawk | Accipitridae | 11 - Sharp-shinned Hawk | Accipitridae | 9 - Bald Eagle | Accipitridae | 9 - Cooper Hawk | Accipitridae | 7 - Northern Goshawk | Accipitridae | 5 - unid. Accipiter hawk | Accipitridae | 1 - Barred Owl | Strigidae | 44 - Eastern Screech-Owl | Strigidae | 3 - Great Horned Owl | Strigidae | 1 - Northern Saw-whet Owl | Strigidae | 1 -(13 rows) -~~~ - -## Next steps - -Now that you are familiar with writing and tuning spatial queries, you are ready for the following steps: - -- Learn more details about the [spatial features](spatial-features.html) supported by CockroachDB. - -- Learn how to migrate spatial data from [shapefiles](migrate-from-shapefiles.html), [GeoPackages](migrate-from-geopackage.html), [GeoJSON](migrate-from-geojson.html), or [OpenStreetMap](migrate-from-openstreetmap.html) into CockroachDB. - -- Learn how CockroachDB's [spatial indexes](spatial-indexes.html) work. - -## Appendix - -### Spatial features and corresponding queries - -The queries are presented above in a "narrative order" that corresponds roughly to the order in which you might ask questions about bookstores and loon habitat as you plan for your vacation. - -However, you may just want to see what queries are exercising a specific [spatial feature](spatial-features.html) supported by CockroachDB. The table below provides a mapping from a feature (such as 'spatial indexing' or 'spatial joins') to the queries that use that feature. - -| Feature | Queries | Description(s) | -|------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Spatial data set preparation | N/A | For information about how this data set was prepared, see the relevant portions of the [Data set description](#data-set-description). | -| Loading spatial data | N/A | For loading instructions on loading the data set, see [Loading the data set](#step-3-load-the-data-set). | -| Spatial joins | [#6][q_06], [#7][q_07], [#8][q_08], [#9][q_09], [#10][q_10], [#13][q_13], [#14][q_14], [#15][q_15], [#16][q_16], [#17][q_17], [#18][q_18], [#19][q_19] | Joins that involve comparing spatial features from two or more tables, usually with a predicate such as [`ST_Contains`](st_contains.html). | -| Spatial indexing | [#15][q_15], [#17][q_17] | These queries show explicit spatial index creation; many of the spatial join queries listed above could use these indexes, once created. | -| Spatial relationships | [#6][q_06], [#7][q_07], [#8][q_08], [#9][q_09], [#10][q_10], [#13][q_13], [#14][q_14], [#15][q_15], [#16][q_16], [#17][q_17], [#18][q_18], [#19][q_19] | Comparing spatial relationships with predicates such as [`ST_Contains`](st_contains.html) and `ST_DWithin`. | -| Nearest-neighbor searching | [#15][q_15] | This query shows how to execute a nearest-neighbor search using CockroachDB. | - -### Data set description - -This section contains an overview of the data sets and how they were constructed and put together. Because this data set is a "compound data set" that is built from several other data sets, there is some complexity to the schema that you will have to navigate. - -The data set contains two databases: - -- [The `tutorial` database](#the-tutorial-database) -- [The `birds` database](#the-birds-database) - -#### The `tutorial` database - -The `tutorial` database contains the following tables: - -- [The `bookstores` table](#the-bookstores-and-bookstore_routes-tables) stores the independent bookstore information such as name, address, geo-location, etc. -- [The `bookstore_routes` table](#the-bookstores-and-bookstore_routes-tables) is a small table containing just the routes between the local bookstores that lie within the loon's Adirondack habitat (this will be explained in more detail below). -- [The `roads` table](#the-roads-table) is where the U.S. National Atlas road data is kept. - -##### The `bookstores` and `bookstore_routes` tables - -Below is an entity-relationship diagram showing the `bookstores` and `bookstore_routes` tables (generated using [DBeaver](dbeaver.html)): - -tutorial.bookstores and tutorial.bookstore_routes ER diagrams - -As mentioned above, the `bookstores` table was created by scraping web data from the [American Booksellers Association website's member directory](https://bookweb.org/member_directory/search/ABAmember). In addition, the `geom` column was constructed by doing some [address geocoding](https://en.wikipedia.org/wiki/Address_geocoding) that converted each bookstore's address to a lon/lat pair and converted to a spatial object using `ST_MakePoint`. For each bookstore, the script did a bit of parsing and geocoding and ran essentially the following query: - -{% include_cached copy-clipboard.html %} -~~~ sql -INSERT -INTO - bookstores - ( - name, - url, - phone, - address, - street, - city, - state, - zip, - description, - geom - ) -VALUES - ( - '$name', - '$url', - '$phone', - '$address', - '$street', - '$city', - '$state', - '$zip', - '$description', - ( - SELECT - st_setsrid( - st_makepoint('$lon', '$lat'), - 4326 - ) - ) - ); -~~~ - -Similarly, the paths (specifically [MultiLinestrings](multilinestring.html)) between bookstores in the `bookstore_routes` table were also calculated using a script that made use of a simple greedy algorithm to find paths between the bookstores using the `roads` table (described below). The algorithm always looks for the next road that will bring you closer to the destination. It is fine for this small use case, but it is not robust enough for any kind of production use, since it does not perform any backtracking when it gets stuck in a local optimum. The algorithm works as follows, and assumes a starting bookstore _A_ and an ending bookstore _B_: - -1. (Start) Find the road nearest _A_; call this road _R_A. -2. Find the road nearest the destination _B_; call this road _R_B. -3. Do _R_A and _R_B intersect? If so, the algorithm is done. You have arrived at your destination. (This is the recursive base case.) -4. Otherwise, find the roads that intersect with _R_A. Of these roads, choose the one that passes closest to _R_B. Call it _R_A'. -5. Do _R_A' and _R_B intersect? If so, the algorithm is done. You have arrived at your destination. -6. If the algorithm has arrived at this step without finishing, it goes back to step 4 and applies that step to the current "closest road to destination" _R_A', and keeps operating steps 4-6 recursively until it has found a complete path or it hits some kind of heuristic timeout. - -{{site.data.alerts.callout_info}} -For more information about production quality map routing software that uses OpenStreetMap, see [the OpenStreetMap wiki page on Routing](https://wiki.openstreetmap.org/wiki/Routing). -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -There are multiple ways to do geocoding. You can use REST API-based services or do the geocoding yourself by processing [TIGER/LINE address data](https://www2.census.gov/geo/tiger/tigerua/) using a library for that purpose. For more information about some of the options available to you, see [this OpenStreetMap wiki page on geocoders](https://wiki.openstreetmap.org/wiki/Search_engines) or [this list of open source geocoding software](http://www.tsusiatsoftware.net/geocode.html). -{{site.data.alerts.end}} - -##### The `roads` table - -Meanwhile, the `roads` table has many columns; the most important ones used in this tutorial are `state`, `geom`, `miles`, and `prime_name` (the human-readable name of the road). - -tutorial.roads ER diagrams - -For more information about what the other columns in `roads` mean, see the [full data set description](https://www.sciencebase.gov/catalog/file/get/581d052be4b08da350d524ce?f=__disk__60%2F6b%2F4e%2F606b4e564884da8cca57ffeb229cd817006616e0&transform=1&allowOpen=true). - -{{site.data.alerts.callout_info}} -The `roads` table was imported from a [shapefile](https://prd-tnm.s3.amazonaws.com/StagedProducts/Small-scale/data/Transportation/roadtrl010g.shp_nt00920.tar.gz) using the method described in [Migrate from Shapefiles](migrate-from-shapefiles.html). -{{site.data.alerts.end}} - -#### The `birds` database - -The `birds` database contains several tables that you will have to [join](joins.html) together in most of the queries shown elsewhere on this page. The multi-table design reflects [the schema of the actual data set](https://www.sciencebase.gov/catalog/file/get/5ea04e9a82cefae35a129d65?f=__disk__b4%2F2f%2Fcf%2Fb42fcfe28a799db6e8c97200829ea1ebaccbf8ea&transform=1&allowOpen=true), which is split across files that map to these tables. - -The tables in the `birds` database are diagrammed below: - -- `birds` is a list of ~750 bird species. Most queries will use the `name` column. -- `routes` is a list of ~130 prescribed locations that the birdwatchers helping with the survey visit each year. The `geom` associated with each route is a [Point](point.html) marking the latitude and longitude of the route's starting point. For details, see the [schema](https://www.sciencebase.gov/catalog/file/get/5ea04e9a82cefae35a129d65?f=__disk__b4%2F2f%2Fcf%2Fb42fcfe28a799db6e8c97200829ea1ebaccbf8ea&transform=1&allowOpen=true) (search for the text "routes.csv"). -- `observations` describes the ~85,000 times and places in which birds of various species were actually seen. The `bird_id` is a [foreign key](foreign-key.html) to the ID in the `birds` table, and the `route_id` points to the ID of the `routes` table. - -birds.birds, birds.routes, and birds.observations ER diagrams - -Each of these tables were populated using a script that parsed [the CSV files available for download](https://www.sciencebase.gov/catalog/item/52b1dfa8e4b0d9b325230cd9) and added the data using [`INSERT`](insert.html) statements. For the `routes` table, once again the `ST_MakePoint` function was used to create a geometry from the lon/lat values in the CSV as follows: - -{% include_cached copy-clipboard.html %} -~~~ sql -INSERT -INTO - routes (id, name, country, state, geom) -VALUES - ( - '$route_id', - '$route_name', - '$country', - '$state', - ( - SELECT - st_setsrid( - st_makepoint('$lon', '$lat'), - 4326 - ) - ) - ); -~~~ - -{{site.data.alerts.callout_info}} -This data is stored in a separate `birds` database due to the fact that it is split into several tables. It could also have been added to the `tutorial` database by naming the tables something like `bird_species`, `bird_routes`, and `bird_observations`. -{{site.data.alerts.end}} - -## See also - -- [Install CockroachDB](install-cockroachdb.html) -- [Working with Spatial Data](spatial-data.html) -- [Spatial Features](spatial-features.html) -- [Spatial indexes](spatial-indexes.html) -- [Spatial & GIS Glossary of Terms](spatial-glossary.html) -- [Working with Spatial Data](spatial-data.html) -- [Migrate from Shapefiles](migrate-from-shapefiles.html) -- [Migrate from GeoJSON](migrate-from-geojson.html) -- [Migrate from GeoPackage](migrate-from-geopackage.html) -- [Migrate from OpenStreetMap](migrate-from-openstreetmap.html) -- [Spatial functions](functions-and-operators.html#spatial-functions) -- [POINT](point.html) -- [LINESTRING](linestring.html) -- [POLYGON](polygon.html) -- [MULTIPOINT](multipoint.html) -- [MULTILINESTRING](multilinestring.html) -- [MULTIPOLYGON](multipolygon.html) -- [GEOMETRYCOLLECTION](geometrycollection.html) -- [Well known text](well-known-text.html) -- [Well known binary](well-known-binary.html) -- [GeoJSON](geojson.html) -- [SRID 4326 - longitude and latitude](srid-4326.html) -- [`ST_Contains`](st_contains.html) -- [`ST_ConvexHull`](st_convexhull.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Covers`](st_covers.html) -- [`ST_Disjoint`](st_disjoint.html) -- [`ST_Equals`](st_equals.html) -- [`ST_Intersects`](st_intersects.html) -- [`ST_Overlaps`](st_overlaps.html) -- [`ST_Touches`](st_touches.html) -- [`ST_Union`](st_union.html) -- [`ST_Within`](st_within.html) -- [Troubleshooting overview](troubleshooting-overview.html) -- [Support resources](support-resources.html) - - - -[q_01]: #1-where-has-the-common-loon-been-sighted-by-the-nabbs-in-the-years-2000-2019-in-ny-state -[q_02]: #2-what-is-the-total-area-of-loon-sightings -[q_03]: #3-how-many-loon-sightings-have-there-been-in-ny-state-in-the-years-2000-2019 -[q_04]: #4-how-many-loon-sightings-were-there-in-ny-state-in-just-the-year-2019 -[q_05]: #5-what-is-the-density-of-loon-sightings-per-square-mile-in-its-range-as-defined-by-this-data-set-in-2019 -[q_06]: #6-what-are-the-bookstores-that-lie-within-the-loons-habitat-range-in-ny-state -[q_07]: #7-how-many-different-species-of-bird-habitats-contain-the-location-of-the-book-nook-in-saranac-lake-ny -[q_08]: #8-which-25-birds-were-most-often-sighted-within-10-miles-of-the-book-nook-in-saranac-lake-ny-during-the-2000-2019-observation-period -[q_09]: #9-what-does-the-shape-of-all-bookstore-locations-that-lie-within-the-loons-habitat-look-like -[q_10]: #10-what-is-the-area-of-the-shape-of-all-bookstore-locations-that-are-in-the-loons-habitat-range-within-ny-state -[q_11]: #11-how-long-is-the-route-from-mysteries-on-main-street-in-johnstown-ny-to-the-book-nook-in-saranac-lake-ny -[q_12]: #12-what-does-the-route-from-mysteries-on-main-street-in-johnstown-ny-to-the-book-nook-in-saranac-lake-ny-look-like -[q_13]: #13-what-were-the-25-most-commonly-sighted-birds-in-2019-within-10-miles-of-the-route-between-mysteries-on-main-street-in-johnstown-ny-and-the-bookstore-plus-in-lake-placid-ny -[q_14]: #14-what-are-the-25-least-often-sighted-other-species-of-birds-in-the-loons-sightings-range -[q_15]: #15-what-are-the-top-10-roads-nearest-to-a-loon-sighting-location-in-ny -[q_16]: #16-how-many-miles-of-roads-are-contained-within-the-portion-of-the-loon-habitat-that-lies-within-ny-state -[q_17]: #17-which-bookstore-in-the-loons-region-in-ny-has-the-fewest-miles-of-roads-within-a-10-mile-radius-of-the-store -[q_18]: #18-what-are-the-hawks-sighted-in-the-same-region-as-the-common-loon -[q_19]: #19-what-if-you-also-want-to-look-for-owls-as-well-as-hawks - - diff --git a/src/current/v21.1/split-at.md b/src/current/v21.1/split-at.md deleted file mode 100644 index b4244d7c529..00000000000 --- a/src/current/v21.1/split-at.md +++ /dev/null @@ -1,300 +0,0 @@ ---- -title: SPLIT AT -summary: The SPLIT AT statement forces a range split at the specified row in a table or index. -toc: true ---- - -The `SPLIT AT` [statement](sql-statements.html) forces a range split at the specified row in a table or index. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/split_table_at.html %} -
    - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/split_index_at.html %} -
    - -## Required privileges - -The user must have the `INSERT` [privilege](authorization.html#assign-privileges) on the table or index. - -## Parameters - - Parameter | Description ------------|------------- - `table_name`
    `table_name @ index_name` | The name of the table or index that should be split. - `select_stmt` | A [selection query](selection-queries.html) that produces one or more rows at which to split the table or index. - `a_expr` | The expiration of the split enforcement on the table or index. This can be a [`DECIMAL`](decimal.html), [`INTERVAL`](interval.html), [`TIMESTAMP`](timestamp.html), or [`TIMESTAMPZ`](timestamp.html). - -## Why manually split a range? - -CockroachDB breaks data into ranges. By default, CockroachDB attempts to keep ranges below -a size of 512 MiB. To do this, the system will automatically [split](architecture/distribution-layer.html#range-splits) -a range if it grows larger than this limit. For most use cases, this automatic -range splitting is sufficient, and you should never need to worry about -when or where the system decides to split ranges. - -However, there are reasons why you may want to perform manual splits on -the ranges that store tables or indexes: - -- When a table only consists of a single range, all writes and reads to the - table will be served by that range's [leaseholder](architecture/replication-layer.html#leases). - If a table only holds a small amount of data but is serving a large amount of traffic, - load distribution can become unbalanced. Splitting the table's ranges manually - can allow the load on the table to be more evenly distributed across multiple - nodes. For tables consisting of more than a few ranges, load will naturally - be distributed across multiple nodes and this will not be a concern. - -- When a table is created, it will only consist of a single range. If you know - that a new table will immediately receive significant write - traffic, you may want to preemptively split the table based on the expected - distribution of writes before applying the load. This can help avoid reduced - workload performance that results when automatic splits are unable to keep up - with write traffic. - -## Examples - -{% include {{page.version.version}}/sql/movr-statements-geo-partitioned-replicas.md %} - -### Split a table - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW RANGES FROM TABLE users; -~~~ - -~~~ - start_key | end_key | range_id | range_size_mb | lease_holder | lease_holder_locality | replicas | replica_localities ------------------------------------------------------------------------------------+----------------------------------------------------------------------------------+----------+---------------+--------------+--------------------------+----------+----------------------------------------------------------------------------- - NULL | /"amsterdam"/"\xb333333@\x00\x80\x00\x00\x00\x00\x00\x00#" | 37 | 0.000116 | 1 | region=us-east1,az=b | {1,6,8} | {"region=us-east1,az=b","region=us-west1,az=c","region=europe-west1,az=c"} - /"amsterdam"/"\xb333333@\x00\x80\x00\x00\x00\x00\x00\x00#" | /"boston"/"333333D\x00\x80\x00\x00\x00\x00\x00\x00\n" | 46 | 0.000886 | 8 | region=europe-west1,az=c | {1,6,8} | {"region=us-east1,az=b","region=us-west1,az=c","region=europe-west1,az=c"} - /"boston"/"333333D\x00\x80\x00\x00\x00\x00\x00\x00\n" | /"los angeles"/"\x99\x99\x99\x99\x99\x99H\x00\x80\x00\x00\x00\x00\x00\x00\x1e" | 45 | 0.00046 | 8 | region=europe-west1,az=c | {2,4,8} | {"region=us-east1,az=c","region=us-west1,az=a","region=europe-west1,az=c"} - /"los angeles"/"\x99\x99\x99\x99\x99\x99H\x00\x80\x00\x00\x00\x00\x00\x00\x1e" | /"new york"/"\x19\x99\x99\x99\x99\x99J\x00\x80\x00\x00\x00\x00\x00\x00\x05" | 44 | 0.001015 | 8 | region=europe-west1,az=c | {1,6,8} | {"region=us-east1,az=b","region=us-west1,az=c","region=europe-west1,az=c"} - /"new york"/"\x19\x99\x99\x99\x99\x99J\x00\x80\x00\x00\x00\x00\x00\x00\x05" | /"paris"/"\xcc\xcc\xcc\xcc\xcc\xcc@\x00\x80\x00\x00\x00\x00\x00\x00(" | 77 | 0.000214 | 8 | region=europe-west1,az=c | {1,6,8} | {"region=us-east1,az=b","region=us-west1,az=c","region=europe-west1,az=c"} - /"paris"/"\xcc\xcc\xcc\xcc\xcc\xcc@\x00\x80\x00\x00\x00\x00\x00\x00(" | /"san francisco"/"\x80\x00\x00\x00\x00\x00@\x00\x80\x00\x00\x00\x00\x00\x00\x19" | 43 | 0.001299 | 8 | region=europe-west1,az=c | {3,4,8} | {"region=us-east1,az=d","region=us-west1,az=a","region=europe-west1,az=c"} - /"san francisco"/"\x80\x00\x00\x00\x00\x00@\x00\x80\x00\x00\x00\x00\x00\x00\x19" | /"seattle"/"ffffffH\x00\x80\x00\x00\x00\x00\x00\x00\x14" | 61 | 0.000669 | 3 | region=us-east1,az=d | {3,4,8} | {"region=us-east1,az=d","region=us-west1,az=a","region=europe-west1,az=c"} - /"seattle"/"ffffffH\x00\x80\x00\x00\x00\x00\x00\x00\x14" | /"washington dc"/"L\xcc\xcc\xcc\xcc\xccL\x00\x80\x00\x00\x00\x00\x00\x00\x0f" | 57 | 0.000671 | 3 | region=us-east1,az=d | {3,4,7} | {"region=us-east1,az=d","region=us-west1,az=a","region=europe-west1,az=b"} - /"washington dc"/"L\xcc\xcc\xcc\xcc\xccL\x00\x80\x00\x00\x00\x00\x00\x00\x0f" | NULL | 87 | 0.000231 | 4 | region=us-west1,az=a | {3,4,7} | {"region=us-east1,az=d","region=us-west1,az=a","region=europe-west1,az=b"} -(9 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE users SPLIT AT VALUES ('chicago'), ('new york'), ('seattle'); -~~~ - -~~~ - key | pretty | split_enforced_until --------------------------------+-------------+-------------------------------------- - \275\211\022chicago\000\001 | /"chicago" | 2262-04-11 23:47:16.854776+00:00:00 - \275\211\022new york\000\001 | /"new york" | 2262-04-11 23:47:16.854776+00:00:00 - \275\211\022seattle\000\001 | /"seattle" | 2262-04-11 23:47:16.854776+00:00:00 -(3 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW RANGES FROM TABLE users; -~~~ - -~~~ - start_key | end_key | range_id | range_size_mb | lease_holder | lease_holder_locality | replicas | replica_localities ------------------------------------------------------------------------------------+----------------------------------------------------------------------------------+----------+---------------+--------------+--------------------------+----------+------------------------------------------------------------------------------------- - NULL | /"amsterdam"/"\xb333333@\x00\x80\x00\x00\x00\x00\x00\x00#" | 37 | 0.000116 | 1 | region=us-east1,az=b | {1,6,8} | {"region=us-east1,az=b","region=us-west1,az=c","region=europe-west1,az=c"} - /"amsterdam"/"\xb333333@\x00\x80\x00\x00\x00\x00\x00\x00#" | /"amsterdam"/PrefixEnd | 46 | 0.000446 | 8 | region=europe-west1,az=c | {7,8,9} | {"region=europe-west1,az=b","region=europe-west1,az=c","region=europe-west1,az=d"} - /"amsterdam"/PrefixEnd | /"boston" | 70 | 0 | 8 | region=europe-west1,az=c | {3,6,8} | {"region=us-east1,az=d","region=us-west1,az=c","region=europe-west1,az=c"} - /"boston" | /"boston"/"333333D\x00\x80\x00\x00\x00\x00\x00\x00\n" | 71 | 0.00044 | 1 | region=us-east1,az=b | {1,6,8} | {"region=us-east1,az=b","region=us-west1,az=c","region=europe-west1,az=c"} - /"boston"/"333333D\x00\x80\x00\x00\x00\x00\x00\x00\n" | /"boston"/PrefixEnd | 45 | 0.000225 | 2 | region=us-east1,az=c | {2,3,8} | {"region=us-east1,az=c","region=us-east1,az=d","region=europe-west1,az=c"} - /"boston"/PrefixEnd | /"chicago" | 72 | 0 | 8 | region=europe-west1,az=c | {2,4,8} | {"region=us-east1,az=c","region=us-west1,az=a","region=europe-west1,az=c"} - /"chicago" | /"los angeles" | 74 | 0 | 8 | region=europe-west1,az=c | {2,4,8} | {"region=us-east1,az=c","region=us-west1,az=a","region=europe-west1,az=c"} - /"los angeles" | /"los angeles"/"\x99\x99\x99\x99\x99\x99H\x00\x80\x00\x00\x00\x00\x00\x00\x1e" | 73 | 0.000235 | 8 | region=europe-west1,az=c | {2,4,8} | {"region=us-east1,az=c","region=us-west1,az=a","region=europe-west1,az=c"} - /"los angeles"/"\x99\x99\x99\x99\x99\x99H\x00\x80\x00\x00\x00\x00\x00\x00\x1e" | /"los angeles"/PrefixEnd | 44 | 0.000462 | 4 | region=us-west1,az=a | {4,6,8} | {"region=us-west1,az=a","region=us-west1,az=c","region=europe-west1,az=c"} - /"los angeles"/PrefixEnd | /"new york" | 68 | 0 | 8 | region=europe-west1,az=c | {2,6,8} | {"region=us-east1,az=c","region=us-west1,az=c","region=europe-west1,az=c"} - /"new york" | /"new york"/"\x19\x99\x99\x99\x99\x99J\x00\x80\x00\x00\x00\x00\x00\x00\x05" | 69 | 0.000553 | 8 | region=europe-west1,az=c | {1,3,8} | {"region=us-east1,az=b","region=us-east1,az=d","region=europe-west1,az=c"} - /"new york"/"\x19\x99\x99\x99\x99\x99J\x00\x80\x00\x00\x00\x00\x00\x00\x05" | /"new york"/PrefixEnd | 77 | 0.000111 | 1 | region=us-east1,az=b | {1,6,8} | {"region=us-east1,az=b","region=us-west1,az=c","region=europe-west1,az=c"} - /"new york"/PrefixEnd | /"paris" | 62 | 0 | 8 | region=europe-west1,az=c | {3,4,8} | {"region=us-east1,az=d","region=us-west1,az=a","region=europe-west1,az=c"} - /"paris" | /"paris"/"\xcc\xcc\xcc\xcc\xcc\xcc@\x00\x80\x00\x00\x00\x00\x00\x00(" | 63 | 0.000103 | 8 | region=europe-west1,az=c | {7,8,9} | {"region=europe-west1,az=b","region=europe-west1,az=c","region=europe-west1,az=d"} - /"paris"/"\xcc\xcc\xcc\xcc\xcc\xcc@\x00\x80\x00\x00\x00\x00\x00\x00(" | /"paris"/PrefixEnd | 43 | 0.000525 | 8 | region=europe-west1,az=c | {7,8,9} | {"region=europe-west1,az=b","region=europe-west1,az=c","region=europe-west1,az=d"} - /"paris"/PrefixEnd | /"rome" | 64 | 0 | 8 | region=europe-west1,az=c | {3,4,8} | {"region=us-east1,az=d","region=us-west1,az=a","region=europe-west1,az=c"} - /"rome" | /"rome"/PrefixEnd | 65 | 0.000539 | 8 | region=europe-west1,az=c | {7,8,9} | {"region=europe-west1,az=b","region=europe-west1,az=c","region=europe-west1,az=d"} - /"rome"/PrefixEnd | /"san francisco" | 66 | 0 | 8 | region=europe-west1,az=c | {3,4,8} | {"region=us-east1,az=d","region=us-west1,az=a","region=europe-west1,az=c"} - /"san francisco" | /"san francisco"/"\x80\x00\x00\x00\x00\x00@\x00\x80\x00\x00\x00\x00\x00\x00\x19" | 67 | 0.000235 | 4 | region=us-west1,az=a | {4,5,8} | {"region=us-west1,az=a","region=us-west1,az=b","region=europe-west1,az=c"} - /"san francisco"/"\x80\x00\x00\x00\x00\x00@\x00\x80\x00\x00\x00\x00\x00\x00\x19" | /"san francisco"/PrefixEnd | 61 | 0.000365 | 4 | region=us-west1,az=a | {4,5,6} | {"region=us-west1,az=a","region=us-west1,az=b","region=us-west1,az=c"} - /"san francisco"/PrefixEnd | /"seattle" | 88 | 0 | 3 | region=us-east1,az=d | {3,4,8} | {"region=us-east1,az=d","region=us-west1,az=a","region=europe-west1,az=c"} - /"seattle" | /"seattle"/"ffffffH\x00\x80\x00\x00\x00\x00\x00\x00\x14" | 89 | 0.000304 | 3 | region=us-east1,az=d | {3,4,8} | {"region=us-east1,az=d","region=us-west1,az=a","region=europe-west1,az=c"} - /"seattle"/"ffffffH\x00\x80\x00\x00\x00\x00\x00\x00\x14" | /"seattle"/PrefixEnd | 57 | 0.000327 | 3 | region=us-east1,az=d | {3,4,7} | {"region=us-east1,az=d","region=us-west1,az=a","region=europe-west1,az=b"} - /"seattle"/PrefixEnd | /"washington dc" | 90 | 0 | 3 | region=us-east1,az=d | {3,4,7} | {"region=us-east1,az=d","region=us-west1,az=a","region=europe-west1,az=b"} - /"washington dc" | /"washington dc"/"L\xcc\xcc\xcc\xcc\xccL\x00\x80\x00\x00\x00\x00\x00\x00\x0f" | 91 | 0.000344 | 3 | region=us-east1,az=d | {1,2,3} | {"region=us-east1,az=b","region=us-east1,az=c","region=us-east1,az=d"} - /"washington dc"/"L\xcc\xcc\xcc\xcc\xccL\x00\x80\x00\x00\x00\x00\x00\x00\x0f" | /"washington dc"/PrefixEnd | 87 | 0.000231 | 3 | region=us-east1,az=d | {1,2,3} | {"region=us-east1,az=b","region=us-east1,az=c","region=us-east1,az=d"} - /"washington dc"/PrefixEnd | NULL | 157 | 0 | 4 | region=us-west1,az=a | {3,4,7} | {"region=us-east1,az=d","region=us-west1,az=a","region=europe-west1,az=b"} -(27 rows) -~~~ - -### Split a table with a compound primary key - -You may want to split a table with a compound primary key. - -Suppose that you want MovR to offer ride-sharing services, in addition to vehicle-sharing services. Some users need to sign up to be drivers, so you need a `drivers` table to store driver information. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE drivers ( - id UUID DEFAULT gen_random_uuid(), - city STRING, - name STRING, - dl STRING DEFAULT left(md5(random()::text),8) UNIQUE CHECK (LENGTH(dl) < 9), - address STRING, - CONSTRAINT "primary" PRIMARY KEY (city ASC, dl ASC) -); -~~~ - -The table's compound primary key is on the `city` and `dl` columns. Note that the table automatically generates an `id` and a `dl` [using supported SQL functions](functions-and-operators.html), if they are not provided. - -Because this table has several columns in common with the `users` table, you can populate the table with values from the `users` table with an `INSERT` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO drivers (id, city, name, address) - SELECT id, city, name, address FROM users; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW RANGES FROM TABLE drivers; -~~~ - -~~~ - start_key | end_key | range_id | range_size_mb | lease_holder | lease_holder_locality | replicas | replica_localities -------------+---------+----------+---------------+--------------+--------------------------+----------+----------------------------------------------------------------------------- - NULL | NULL | 310 | 0.007218 | 7 | region=europe-west1,az=b | {1,4,7} | {"region=us-east1,az=b","region=us-west1,az=a","region=europe-west1,az=b"} -(1 row) -~~~ - -Now you can split the table based on the compound primary key. Note that you do not have to specify the entire value for the primary key, just the prefix. - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE drivers SPLIT AT VALUES ('new york', '3'), ('new york', '7'), ('chicago', '3'), ('chicago', '7'), ('seattle', '3'), ('seattle', '7'); -~~~ - -~~~ - key | pretty | split_enforced_until ---------------------------------------------+-----------------+-------------------------------------- - \303\211\022new york\000\001\0223\000\001 | /"new york"/"3" | 2262-04-11 23:47:16.854776+00:00:00 - \303\211\022new york\000\001\0227\000\001 | /"new york"/"7" | 2262-04-11 23:47:16.854776+00:00:00 - \303\211\022chicago\000\001\0223\000\001 | /"chicago"/"3" | 2262-04-11 23:47:16.854776+00:00:00 - \303\211\022chicago\000\001\0227\000\001 | /"chicago"/"7" | 2262-04-11 23:47:16.854776+00:00:00 - \303\211\022seattle\000\001\0223\000\001 | /"seattle"/"3" | 2262-04-11 23:47:16.854776+00:00:00 - \303\211\022seattle\000\001\0227\000\001 | /"seattle"/"7" | 2262-04-11 23:47:16.854776+00:00:00 -(6 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW RANGES FROM TABLE drivers; -~~~ - -~~~ - start_key | end_key | range_id | range_size_mb | lease_holder | lease_holder_locality | replicas | replica_localities -------------------+-----------------+----------+---------------+--------------+--------------------------+----------+----------------------------------------------------------------------------- - NULL | /"chicago"/"3" | 310 | 0.001117 | 7 | region=europe-west1,az=b | {3,4,7} | {"region=us-east1,az=d","region=us-west1,az=a","region=europe-west1,az=b"} - /"chicago"/"3" | /"chicago"/"7" | 314 | 0 | 7 | region=europe-west1,az=b | {3,4,7} | {"region=us-east1,az=d","region=us-west1,az=a","region=europe-west1,az=b"} - /"chicago"/"7" | /"new york"/"3" | 315 | 0.000933 | 7 | region=europe-west1,az=b | {3,4,7} | {"region=us-east1,az=d","region=us-west1,az=a","region=europe-west1,az=b"} - /"new york"/"3" | /"new york"/"7" | 311 | 0 | 7 | region=europe-west1,az=b | {1,4,7} | {"region=us-east1,az=b","region=us-west1,az=a","region=europe-west1,az=b"} - /"new york"/"7" | /"seattle"/"3" | 312 | 0.001905 | 7 | region=europe-west1,az=b | {1,4,7} | {"region=us-east1,az=b","region=us-west1,az=a","region=europe-west1,az=b"} - /"seattle"/"3" | /"seattle"/"7" | 316 | 0.000193 | 7 | region=europe-west1,az=b | {1,6,7} | {"region=us-east1,az=b","region=us-west1,az=c","region=europe-west1,az=b"} - /"seattle"/"7" | NULL | 317 | 0.00307 | 7 | region=europe-west1,az=b | {1,4,7} | {"region=us-east1,az=b","region=us-west1,az=a","region=europe-west1,az=b"} -(7 rows) -~~~ - -### Split an index - -Add a new secondary [index](indexes.html) to the `rides` table, on the `revenue` column: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE INDEX revenue_idx ON rides(revenue); -~~~ - -Then split the table ranges by secondary index values: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER INDEX rides@revenue_idx SPLIT AT VALUES (25.00), (50.00), (75.00); -~~~ -~~~ - key | pretty | split_enforced_until ---------------------+--------+-------------------------------------- - \277\214*2\000 | /25 | 2262-04-11 23:47:16.854776+00:00:00 - \277\214*d\000 | /5E+1 | 2262-04-11 23:47:16.854776+00:00:00 - \277\214*\226\000 | /75 | 2262-04-11 23:47:16.854776+00:00:00 -(3 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW RANGES FROM INDEX rides@revenue_idx; -~~~ -~~~ - start_key | end_key | range_id | range_size_mb | lease_holder | lease_holder_locality | replicas | replica_localities -------------+---------+----------+---------------+--------------+-----------------------+----------+----------------------------------------------------------------------------- - NULL | /25 | 249 | 0.007464 | 3 | region=us-east1,az=d | {3,5,7} | {"region=us-east1,az=d","region=us-west1,az=b","region=europe-west1,az=b"} - /25 | /5E+1 | 250 | 0.008995 | 3 | region=us-east1,az=d | {3,5,7} | {"region=us-east1,az=d","region=us-west1,az=b","region=europe-west1,az=b"} - /5E+1 | /75 | 251 | 0.008212 | 3 | region=us-east1,az=d | {3,5,7} | {"region=us-east1,az=d","region=us-west1,az=b","region=europe-west1,az=b"} - /75 | NULL | 252 | 0.009267 | 3 | region=us-east1,az=d | {3,5,7} | {"region=us-east1,az=d","region=us-west1,az=b","region=europe-west1,az=b"} -(4 rows) -~~~ - -### Set the expiration on a split enforcement - -You can specify the time at which a split enforcement expires by adding a `WITH EXPIRATION` clause to your `SPLIT` statement. Supported expiration values include [`DECIMAL`](decimal.html), [`INTERVAL`](interval.html), [`TIMESTAMP`](timestamp.html), and [`TIMESTAMPZ`](timestamp.html). - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE vehicles SPLIT AT VALUES ('chicago'), ('new york'), ('seattle') WITH EXPIRATION '2022-01-10 23:30:00+00:00'; -~~~ -~~~ - key | pretty | split_enforced_until --------------------------------+-------------+------------------------------- - \276\211\022chicago\000\001 | /"chicago" | 2022-01-10 23:30:00+00:00:00 - \276\211\022new york\000\001 | /"new york" | 2022-01-10 23:30:00+00:00:00 - \276\211\022seattle\000\001 | /"seattle" | 2022-01-10 23:30:00+00:00:00 -(3 rows) -~~~ - -You can see the split's expiration date in the `split_enforced_until` column. The `crdb_internal.ranges` table also contains information about ranges in your CockroachDB cluster, including the `split_enforced_until` column. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT range_id, start_pretty, end_pretty, split_enforced_until FROM crdb_internal.ranges WHERE table_name='vehicles'; -~~~ - -~~~ - range_id | start_pretty | end_pretty | split_enforced_until ------------+-------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------+-------------------------------------- - 38 | /Table/54 | /Table/54/1/"amsterdam" | NULL - 55 | /Table/54/1/"amsterdam" | /Table/54/1/"amsterdam"/PrefixEnd | NULL - 109 | /Table/54/1/"amsterdam"/PrefixEnd | /Table/54/1/"boston" | NULL - 114 | /Table/54/1/"boston" | /Table/54/1/"boston"/"\"\"\"\"\"\"B\x00\x80\x00\x00\x00\x00\x00\x00\x02" | NULL - 50 | /Table/54/1/"boston"/"\"\"\"\"\"\"B\x00\x80\x00\x00\x00\x00\x00\x00\x02" | /Table/54/1/"boston"/"333333D\x00\x80\x00\x00\x00\x00\x00\x00\x03" | 2262-04-11 23:47:16.854776+00:00:00 - 49 | /Table/54/1/"boston"/"333333D\x00\x80\x00\x00\x00\x00\x00\x00\x03" | /Table/54/1/"boston"/PrefixEnd | 2262-04-11 23:47:16.854776+00:00:00 - 129 | /Table/54/1/"boston"/PrefixEnd | /Table/54/1/"chicago" | NULL - 241 | /Table/54/1/"chicago" | /Table/54/1/"los angeles" | 2022-01-10 23:30:00+00:00:00 - 130 | /Table/54/1/"los angeles" | /Table/54/1/"los angeles"/PrefixEnd | NULL - 131 | /Table/54/1/"los angeles"/PrefixEnd | /Table/54/1/"new york" | NULL - 132 | /Table/54/1/"new york" | /Table/54/1/"new york"/"\x11\x11\x11\x11\x11\x11A\x00\x80\x00\x00\x00\x00\x00\x00\x01" | 2022-01-10 23:30:00+00:00:00 - 48 | /Table/54/1/"new york"/"\x11\x11\x11\x11\x11\x11A\x00\x80\x00\x00\x00\x00\x00\x00\x01" | /Table/54/1/"new york"/PrefixEnd | 2262-04-11 23:47:16.854776+00:00:00 -... -(46 rows) -~~~ - - -## See also - -- [Selection Queries](selection-queries.html) -- [Distribution Layer](architecture/distribution-layer.html) -- [Replication Layer](architecture/replication-layer.html) -- [`SHOW JOBS`](show-jobs.html) -- [`UNSPLIT AT`](unsplit-at.html) diff --git a/src/current/v21.1/sql-audit-logging.md b/src/current/v21.1/sql-audit-logging.md deleted file mode 100644 index 53983484bc5..00000000000 --- a/src/current/v21.1/sql-audit-logging.md +++ /dev/null @@ -1,203 +0,0 @@ ---- -title: SQL Audit Logging -summary: Use the EXPERIMENTAL_AUDIT setting to turn SQL audit logging on or off for a table. -toc: true ---- - -SQL audit logging gives you detailed information about queries being executed against your system. This feature is especially useful when you want to log all queries that are run against a table containing personally identifiable information (PII). - -It consists of using the [`ALTER TABLE ... EXPERIMENTAL_AUDIT`](experimental-audit.html) command to enable the [`SENSITIVE_ACCESS`](logging.html#sensitive_access) logging channel per table. - -This page provides an example of SQL audit logging in CockroachDB, including: - -- How to turn audit logging on and off. -- Where the audit log files live. -- What the audit log files look like. - -Note that enabling SQL audit logs can negatively impact performance. As a result, we recommend using SQL audit logs for security purposes only. For more details, see the [`ALTER TABLE ... EXPERIMENTAL_AUDIT`](experimental-audit.html#performance-considerations) reference page. - -{{site.data.alerts.callout_success}} -For the best visibility into security-related events on your cluster, we recommend configuring `SENSITIVE_ACCESS` together with the `USER_ADMIN`, `PRIVILEGES`, and `SESSIONS` logging channels. To learn more, see [Logging Use Cases](logging-use-cases.html#security-and-audit-monitoring). -{{site.data.alerts.end}} - -{% include {{ page.version.version }}/misc/experimental-warning.md %} - -## Step 1. Create sample tables - -Use the statements below to create: - -- A `customers` table which contains PII such as name, address, etc. -- An `orders` table with a foreign key into `customers`, which does not expose any PII - -Later, we'll show how to turn on auditing for the `customers` table. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE customers ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - name STRING NOT NULL, - address STRING NOT NULL, - national_id INT NOT NULL, - telephone INT NOT NULL, - email STRING UNIQUE NOT NULL -); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE orders ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - product_id INT NOT NULL, - delivery_status STRING check (delivery_status='processing' or delivery_status='in-transit' or delivery_status='delivered') NOT NULL, - customer_id UUID NOT NULL REFERENCES customers (id) -); -~~~ - -## Step 2. Turn on auditing for the `customers` table - -We turn on auditing for a table using the [`EXPERIMENTAL_AUDIT`](experimental-audit.html) subcommand of [`ALTER TABLE`](alter-table.html). - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE customers EXPERIMENTAL_AUDIT SET READ WRITE; -~~~ - -This directs [SQL audit events](eventlog.html#sql-access-audit-events) for the `customers` table into the [`SENSITIVE_ACCESS`](logging.html#logging-channels) logging channel. - -{{site.data.alerts.callout_info}} -To turn on auditing for more than one table, issue a separate `ALTER` statement for each table. -{{site.data.alerts.end}} - -## Step 3. Populate the `customers` table - -Now that we have auditing turned on, let's add some customer data: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO customers (name, address, national_id, telephone, email) VALUES ( - 'Pritchard M. Cleveland', - '23 Crooked Lane, Garden City, NY USA 11536', - 778124477, - 12125552000, - 'pritchmeister@aol.com' -); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO customers (name, address, national_id, telephone, email) VALUES ( - 'Vainglorious K. Snerptwiddle III', - '44 Straight Narrows, Garden City, NY USA 11536', - 899127890, - 16465552000, - 'snerp@snerpy.net' -); -~~~ - -Now let's verify that our customers were added successfully: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM customers; -~~~ - -~~~ - id | name | address | national_id | telephone | email ----------------------------------------+----------------------------------+------------------------------------------------+-------------+-------------+------------------------ - 859c6aa1-ae36-49c8-9f12-7a952b4e6915 | Vainglorious K. Snerptwiddle III | 44 Straight Narrows, Garden City, NY USA 11536 | 899127890 | 16465552000 | snerp@snerpy.net - 90810df2-d3c1-4038-8462-132f4df5112b | Pritchard M. Cleveland | 23 Crooked Lane, Garden City, NY USA 11536 | 778124477 | 12125552000 | pritchmeister@aol.com -(2 rows) -~~~ - -## Step 4. Check the audit log - -By [default](configure-logs.html#default-logging-configuration), events in the `SENSITIVE_ACCESS` channel are output to a log file that is prefixed `cockroach-sql-audit` and stored in the [same directory](configure-logs.html#logging-directory) as the other logs generated by CockroachDB. - -To store the audit log files in a specific directory, [configure the `SENSITIVE_ACCESS` channel](configure-logs.html#output-to-files) with a custom `dir` path. Like the other log files, it's rotated according to the [`max-file-size` setting](configure-logs.html#set-file-defaults). - -When we look at the audit log for this example, we see the following lines showing every command we've run so far, as expected. - -~~~ -I210323 18:49:57.612823 1182 8@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:49851›,hostnossl,user=root] 1 ={"Timestamp":1616525397608639000,"EventType":"sensitive_table_access","Statement":"‹ALTER TABLE defaultdb.public.customers EXPERIMENTAL_AUDIT SET READ WRITE›","User":"‹root›","DescriptorID":52,"ApplicationName":"‹$ cockroach sql›","ExecMode":"exec","Age":4.222,"TxnCounter":20,"TableName":"‹defaultdb.public.customers›","AccessMode":"rw"} -I210323 18:50:04.518707 1182 8@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:49851›,hostnossl,user=root] 2 ={"Timestamp":1616525404415644000,"EventType":"sensitive_table_access","Statement":"‹INSERT INTO \"\".\"\".customers(name, address, national_id, telephone, email) VALUES ('Pritchard M. Cleveland', '23 Crooked Lane, Garden City, NY USA 11536', 778124477, 12125552000, 'pritchmeister@aol.com')›","User":"‹root›","DescriptorID":52,"ApplicationName":"‹$ cockroach sql›","ExecMode":"exec","NumRows":1,"Age":103.066,"TxnCounter":28,"TableName":"‹defaultdb.public.customers›","AccessMode":"rw"} -I210323 18:50:07.591609 1182 8@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:49851›,hostnossl,user=root] 3 ={"Timestamp":1616525407566154000,"EventType":"sensitive_table_access","Statement":"‹INSERT INTO \"\".\"\".customers(name, address, national_id, telephone, email) VALUES ('Vainglorious K. Snerptwiddle III', '44 Straight Narrows, Garden City, NY USA 11536', 899127890, 16465552000, 'snerp@snerpy.net')›","User":"‹root›","DescriptorID":52,"ApplicationName":"‹$ cockroach sql›","ExecMode":"exec","NumRows":1,"Age":25.48,"TxnCounter":36,"TableName":"‹defaultdb.public.customers›","AccessMode":"rw"} -I210323 18:50:10.951550 1182 8@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:49851›,hostnossl,user=root] 4 ={"Timestamp":1616525410949087000,"EventType":"sensitive_table_access","Statement":"‹SELECT * FROM \"\".\"\".customers›","User":"‹root›","DescriptorID":52,"ApplicationName":"‹$ cockroach sql›","ExecMode":"exec","NumRows":2,"Age":2.514,"FullTableScan":true,"TxnCounter":38,"TableName":"‹defaultdb.public.customers›","AccessMode":"r"} -~~~ - -{{site.data.alerts.callout_info}} -The above example shows the default [`crdb-v2`](log-formats.html#format-crdb-v2) log format. This can be changed to another format (e.g., JSON). For details, see [Configure Logs](configure-logs.html#file-logging-format). -{{site.data.alerts.end}} - -{{site.data.alerts.callout_success}} -For descriptions of all SQL audit event types and their fields, see [Notable Event Types](eventlog.html#sql-access-audit-events). -{{site.data.alerts.end}} - -## Step 5. Populate the `orders` table - -Unlike the `customers` table, `orders` doesn't have any PII, just a Product ID and a delivery status. - -Let's populate the `orders` table with some placeholder data using [`CREATE SEQUENCE`](create-sequence.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE SEQUENCE product_ids_asc START 1 INCREMENT 1; -~~~ - -Evaluate the below a few times to generate data; note that this would error if [`SELECT`](select-clause.html) returned multiple results, but it doesn't in this case. - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO orders (product_id, delivery_status, customer_id) VALUES ( - nextval('product_ids_asc'), - 'processing', - (SELECT id FROM customers WHERE name ~ 'Cleve') -); -~~~ - -Let's verify that our orders were added successfully: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM orders ORDER BY product_id; -~~~ - -~~~ - id | product_id | delivery_status | customer_id ----------------------------------------+------------+-----------------+--------------------------------------- - 77fa8340-8a65-4ab2-8191-ed87fc049b33 | 1 | processing | 90810df2-d3c1-4038-8462-132f4df5112b - 36c8b00d-01f0-4956-bb0e-6e9219f49bae | 2 | processing | 90810df2-d3c1-4038-8462-132f4df5112b - 5eebf961-1e4c-41a4-b6c6-441c3d5ef595 | 3 | processing | 90810df2-d3c1-4038-8462-132f4df5112b -(3 rows) -~~~ - -## Step 6. Check the audit log again - -Because we used a `SELECT` against the `customers` table to generate the placeholder data for `orders`, those queries will also show up in the audit log. - -Note that two log entries are created for each query: one entry for the `SELECT` subquery, and one entry for the foreign key check on `customer_id`. Since the `customers` table is read twice with each query, the `TableName` and `TxnCounter` values will be duplicated across entries: - -~~~ -I210323 19:12:09.339065 1182 8@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:49851›,hostnossl,user=root] 5 ={"Timestamp":1616526729194157000,"EventType":"sensitive_table_access","Statement":"‹INSERT INTO \"\".\"\".orders(product_id, delivery_status, customer_id) VALUES (nextval('product_ids_asc'), 'processing', (SELECT id FROM \"\".\"\".customers WHERE name ~ 'Cleve'))›","User":"‹root›","DescriptorID":52,"ApplicationName":"‹$ cockroach sql›","ExecMode":"exec","NumRows":1,"Age":144.956,"FullTableScan":true,"FullIndexScan":true,"TxnCounter":46,"TableName":"‹defaultdb.public.customers›","AccessMode":"r"} -I210323 19:12:09.339204 1182 8@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:49851›,hostnossl,user=root] 6 ={"Timestamp":1616526729194157000,"EventType":"sensitive_table_access","Statement":"‹INSERT INTO \"\".\"\".orders(product_id, delivery_status, customer_id) VALUES (nextval('product_ids_asc'), 'processing', (SELECT id FROM \"\".\"\".customers WHERE name ~ 'Cleve'))›","User":"‹root›","DescriptorID":52,"ApplicationName":"‹$ cockroach sql›","ExecMode":"exec","NumRows":1,"Age":144.956,"FullTableScan":true,"FullIndexScan":true,"TxnCounter":46,"TableName":"‹defaultdb.public.customers›","AccessMode":"r"} -I210323 19:12:13.407107 1182 8@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:49851›,hostnossl,user=root] 7 ={"Timestamp":1616526733375741000,"EventType":"sensitive_table_access","Statement":"‹INSERT INTO \"\".\"\".orders(product_id, delivery_status, customer_id) VALUES (nextval('product_ids_asc'), 'processing', (SELECT id FROM \"\".\"\".customers WHERE name ~ 'Cleve'))›","User":"‹root›","DescriptorID":52,"ApplicationName":"‹$ cockroach sql›","ExecMode":"exec","NumRows":1,"Age":31.427,"FullTableScan":true,"FullIndexScan":true,"TxnCounter":52,"TableName":"‹defaultdb.public.customers›","AccessMode":"r"} -I210323 19:12:13.407177 1182 8@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:49851›,hostnossl,user=root] 8 ={"Timestamp":1616526733375741000,"EventType":"sensitive_table_access","Statement":"‹INSERT INTO \"\".\"\".orders(product_id, delivery_status, customer_id) VALUES (nextval('product_ids_asc'), 'processing', (SELECT id FROM \"\".\"\".customers WHERE name ~ 'Cleve'))›","User":"‹root›","DescriptorID":52,"ApplicationName":"‹$ cockroach sql›","ExecMode":"exec","NumRows":1,"Age":31.427,"FullTableScan":true,"FullIndexScan":true,"TxnCounter":52,"TableName":"‹defaultdb.public.customers›","AccessMode":"r"} -I210323 19:12:14.228906 1182 8@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:49851›,hostnossl,user=root] 9 ={"Timestamp":1616526734201401000,"EventType":"sensitive_table_access","Statement":"‹INSERT INTO \"\".\"\".orders(product_id, delivery_status, customer_id) VALUES (nextval('product_ids_asc'), 'processing', (SELECT id FROM \"\".\"\".customers WHERE name ~ 'Cleve'))›","User":"‹root›","DescriptorID":52,"ApplicationName":"‹$ cockroach sql›","ExecMode":"exec","NumRows":1,"Age":27.554,"FullTableScan":true,"FullIndexScan":true,"TxnCounter":58,"TableName":"‹defaultdb.public.customers›","AccessMode":"r"} -I210323 19:12:14.228964 1182 8@util/log/event_log.go:32 ⋮ [n1,client=‹[::1]:49851›,hostnossl,user=root] 10 ={"Timestamp":1616526734201401000,"EventType":"sensitive_table_access","Statement":"‹INSERT INTO \"\".\"\".orders(product_id, delivery_status, customer_id) VALUES (nextval('product_ids_asc'), 'processing', (SELECT id FROM \"\".\"\".customers WHERE name ~ 'Cleve'))›","User":"‹root›","DescriptorID":52,"ApplicationName":"‹$ cockroach sql›","ExecMode":"exec","NumRows":1,"Age":27.554,"FullTableScan":true,"FullIndexScan":true,"TxnCounter":58,"TableName":"‹defaultdb.public.customers›","AccessMode":"r"} -~~~ - -{{site.data.alerts.callout_info}} -The above example shows the default [`crdb-v2`](log-formats.html#format-crdb-v2) log format. This can be changed to another format (e.g., JSON). For details, see [Configure Logs](configure-logs.html#file-logging-format). -{{site.data.alerts.end}} - -{{site.data.alerts.callout_success}} -For descriptions of all SQL audit event types and their fields, see [Notable Event Types](eventlog.html#sql-access-audit-events). -{{site.data.alerts.end}} - -## See also - -- [`ALTER TABLE ... EXPERIMENTAL_AUDIT`](experimental-audit.html) -- [`cockroach start` logging flags](cockroach-start.html#logging) -- [Logging Use Cases](logging-use-cases.html) -- [SQL FAQ - generating unique row IDs](sql-faqs.html#how-do-i-auto-generate-unique-row-ids-in-cockroachdb) -- [`CREATE SEQUENCE`](create-sequence.html) -- [SQL Feature Support](sql-feature-support.html) diff --git a/src/current/v21.1/sql-constants.md b/src/current/v21.1/sql-constants.md deleted file mode 100644 index 8e6d340af12..00000000000 --- a/src/current/v21.1/sql-constants.md +++ /dev/null @@ -1,286 +0,0 @@ ---- -title: Constant Values -summary: SQL Constants represent a simple value that doesn't change. -toc: true ---- - -SQL Constants represent a simple value that doesn't change. - - -## Introduction - -There are five categories of constants in CockroachDB: - -- [String literals](#string-literals): these define string values but their actual data type will - be inferred from context, for example, `'hello'`. -- [Numeric literals](#numeric-literals): these define numeric values but their actual data - type will be inferred from context, for example, `-12.3`. -- [Bit array literals](#bit-array-literals): these define bit array values with data type - `BIT`, for example, `B'1001001'`. -- [Byte array literals](#byte-array-literals): these define byte array values with data type - `BYTES`, for example, `b'hello'`. -- [Interpreted literals](#interpreted-literals): these define arbitrary values with an explicit - type, for example, `INTERVAL '3 days'`. -- [Named constants](#named-constants): these have predefined values with a predefined - type, for example, `TRUE` or `NULL`. - -## String literals - -CockroachDB supports the following formats for string literals: - -- [Standard SQL string literals](#standard-sql-string-literals). -- [String literals with C escape sequences](#string-literals-with-character-escapes). -- [Dollar-quoted string literals](#dollar-quoted-string-literals) - -These formats also allow arbitrary Unicode characters encoded as UTF-8. - -In any case, the actual data type of a string literal is determined -using the context where it appears. - -For example: - - Expression | Data type of the string literal -------------|--------------------------------- - `length('hello')` | `STRING` - `now() + '3 day'` | `INTERVAL` - `INSERT INTO tb(date_col) VALUES ('2013-01-02')` | `DATE` - -In general, the data type of a string literal is that demanded by the -context if there is no ambiguity, or `STRING` otherwise. - -Check our blog for -[more information about the typing of string literals](https://www.cockroachlabs.com/blog/revisiting-sql-typing-in-cockroachdb/). - -### Standard SQL string literals - -SQL string literals are formed by an arbitrary sequence of characters -enclosed between single quotes (`'`), for example, `'hello world'`. - -To include a single quote in the string, use a double single quote. -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT 'hello' as a, 'it''s a beautiful day' as b; -~~~ - -~~~ -+-------+----------------------+ -| a | b | -+-------+----------------------+ -| hello | it's a beautiful day | -+-------+----------------------+ -~~~ - -For compatibility with the SQL standard, CockroachDB also recognizes -the following special syntax: two simple string literals separated by -a newline character are automatically concatenated together to form a -single constant. For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT 'hello' -' world!' as a; -~~~ - -~~~ -+--------------+ -| a | -+--------------+ -| hello world! | -+--------------+ -~~~ - -This special syntax only works if the two simple literals are -separated by a newline character. For example `'hello' ' world!'` -doesn't work. This is mandated by the SQL standard. - -### String literals with character escapes - -CockroachDB also supports string literals containing escape sequences -like in the programming language C. These are constructed by prefixing -the string literal with the letter `e`, for example, -`e'hello\nworld!'`. - -The following escape sequences are supported: - -Escape Sequence | Interpretation -----------------|--------------- -`\a` | ASCII code 7 (BEL) -`\b` | backspace (ASCII 8) -`\t` | tab (ASCII 9) -`\n` | newline (ASCII 10) -`\v` | vertical tab (ASCII 11) -`\f` | form feed (ASCII 12) -`\r` | carriage return (ASCII 13) -`\xHH` | hexadecimal byte value -`\ooo` | octal byte value -`\uXXXX` | 16-bit hexadecimal Unicode character value -`\UXXXXXXXX` | 32-bit hexadecimal Unicode character value - -For example, the `e'x61\141\u0061'` escape string represents the -hexadecimal byte, octal byte, and 16-bit hexadecimal Unicode character -values equivalent to the `'aaa'` string literal. - -### Dollar-quoted string literals - - To make it easier to write certain types of string constants in SQL code, CockroachDB supports dollar-quoted string literals. This is particularly useful for strings that need to contain lots of single quotes (`'`) or backslashes (`\`). - -At a high level, the dollar-quoting behavior works similarly to ["heredocs"](https://en.wikipedia.org/wiki/Here_document) as used in UNIX shells and some programming languages. - -Dollar-quoted strings have the form: `$` + (optional) tag + `$` + arbitrary text + `$` + (optional) tag + `$` - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT char_length($MyCoolString$ -You can put anything you want in this string -- for example, here's a Windows filesystem pathname: 'C:\Users\foo\Downloads\file.zip' - -You can even nest additional dollar-quoted strings inside each other. For example, here is a regular expression using backticks: $myRegex$[foo\tbar]$myRegex$ - -Finally, you can use $stand-alone dollar signs without the optional tag$. -$MyCoolString$); -~~~ - -~~~ - char_length ---------------- - 369 -(1 row) -~~~ - -## Numeric literals - -Numeric literals can have the following forms: - -~~~ -[+-]9999 -[+-]9999.[9999][e[+-]999] -[+-][9999].9999[e[+-]999] -[+-]9999e[+-]999 -[+-]0xAAAA -~~~ - -Some examples: - -~~~ -+4269 -3.1415 --.001 -6.626e-34 -50e6 -0xcafe111 -~~~ - -The actual data type of a numeric constant depends both on the context -where it is used, its literal format, and its numeric value. - - Syntax | Possible data types ---------|--------------------- - Contains a decimal separator | `FLOAT`, `DECIMAL` - Contains an exponent | `FLOAT`, `DECIMAL` - Contains a value outside of the range -2^63...(2^63)-1 | `FLOAT`, `DECIMAL` - Otherwise | `INT`, `DECIMAL`, `FLOAT` - -Of the possible data types, which one is actually used is then further -refined depending on context. - -Check our blog for -[more information about the typing of numeric literals](https://www.cockroachlabs.com/blog/revisiting-sql-typing-in-cockroachdb/). - -## Bit array literals - -Bit array literals consist of the `B` prefix followed by a string of -binary digits (bits) enclosed in single quotes. - -For example: `B'1001010101'` - -Bit array literals are acceptable both when values of types -[`BIT`](bit.html) or [`VARBIT`](bit.html) (`BIT VARYING`) are -expected. - -The number of bits is arbitrary. An empty bit array is denoted `B''`; -the number of bits need not be a multiple of 8, and bit arrays can -contain more than 64 bits. - -## Byte array literals - -CockroachDB supports two formats for byte array literals: - -- [Byte array literals with C escape sequences](#byte-array-literals-with-character-escapes) -- [Hexadecimal-encoded byte array literals](#hexadecimal-encoded-byte-array-literals) - -### Byte array literals with character escapes - -This uses the same syntax as [string literals containing character escapes](#string-literals-with-character-escapes), -using a `b` prefix instead of `e`. Any character escapes are interpreted like they -would be for string literals. - -For example: `b'hello,\x32world'` - -The two differences between byte array literals and string literals -with character escapes are as follows: - -- Byte array literals always have data type `BYTES`, whereas the data - type of a string literal depends on context. -- Byte array literals may contain invalid UTF-8 byte sequences, - whereas string literals must always contain valid UTF-8 sequences. - -### Hexadecimal-encoded byte array literals - -This is a CockroachDB-specific extension to express byte array -literals: the delimiter `x'` followed by an arbitrary sequence of -hexadecimal digits, followed by a closing `'`. - -For example, all the following formats are equivalent to `b'cat'`: - -- `x'636174'` -- `X'636174'` - -## Interpreted literals - -A constant of any data type can be formed using either of the following formats: - -~~~ -type 'string' -'string':::type -~~~ - -The value of the string part is used as input for the conversion function to the -specified data type, and the result is used as a constant with that data type. - -Examples: - -~~~ -DATE '2013-12-23' -BOOL 'FALSE' -'42.69':::INT -'TRUE':::BOOL -'3 days':::INTERVAL -~~~ - -Additionally, for compatibility with PostgreSQL, the notation -`'string'::type` and `CAST('string' AS type)` is also recognized as an -interpreted literal. These are special cases of -[cast expressions](scalar-expressions.html). - -For more information about the allowable format of interpreted -literals, refer to the "Syntax" section of the respective data types: -[`DATE`](date.html#syntax), [`INET`](inet.html#syntax), [`INTERVAL`](interval.html#syntax), [`TIME`](time.html#syntax), -[`TIMESTAMP`/`TIMESTAMPTZ`](timestamp.html#syntax). - -## Named constants - -CockroachDB recognizes the following SQL named constants: - -- `TRUE` and `FALSE`, the two possible values of data type `BOOL`. -- `NULL`, the special SQL symbol that indicates "no value present". - -Note that `NULL` is a valid constant for any type: its actual data -type during expression evaluation is determined based on context. - -## See also - -- [Scalar Expressions](scalar-expressions.html) -- [Data Types](data-types.html) diff --git a/src/current/v21.1/sql-faqs.md b/src/current/v21.1/sql-faqs.md deleted file mode 100644 index 4374c360402..00000000000 --- a/src/current/v21.1/sql-faqs.md +++ /dev/null @@ -1,141 +0,0 @@ ---- -title: SQL FAQs -summary: Get answers to frequently asked questions about CockroachDB SQL. -toc: true -toc_not_nested: true ---- - -## How do I bulk insert data into CockroachDB? - -- To bulk-insert data into an existing table, batch multiple rows in one [multi-row `INSERT`](insert.html#insert-multiple-rows-into-an-existing-table) statement and do not include the `INSERT` statements within a transaction. Experimentally determine the optimal batch size for your application by monitoring the performance for different batch sizes (10 rows, 100 rows, 1000 rows). - - {{site.data.alerts.callout_info}} - You can also use the [`IMPORT INTO`](import-into.html) statement to bulk-insert CSV data into an existing table. - {{site.data.alerts.end}} -- To bulk-insert data into a new table, the [`IMPORT`](import.html) statement performs better than `INSERT`. `IMPORT` can also be used to [migrate data from other databases](migration-overview.html) like MySQL, Oracle, and Postgres. - -## How do I auto-generate unique row IDs in CockroachDB? - -{% include {{ page.version.version }}/faq/auto-generate-unique-ids.html %} - -## How do I generate unique, slowly increasing sequential numbers in CockroachDB? - -{% include {{ page.version.version }}/faq/sequential-numbers.md %} - -## What are the differences between `UUID`, sequences, and `unique_rowid()`? - -{% include {{ page.version.version }}/faq/differences-between-numberings.md %} - -## How do I order writes to a table to closely follow time in CockroachDB? - -{% include {{ page.version.version }}/faq/sequential-transactions.md %} - -## How do I get the last ID/SERIAL value inserted into a table? - -There’s no function in CockroachDB for returning last inserted values, but you can use the [`RETURNING` clause](insert.html#insert-and-return-values) of the `INSERT` statement. - -For example, this is how you’d use `RETURNING` to return a value auto-generated via `unique_rowid()` or [`SERIAL`](serial.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE users (id INT DEFAULT unique_rowid(), name STRING); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO users (name) VALUES ('mike') RETURNING id; -~~~ - -## What is transaction contention? - -Transaction contention occurs when transactions issued from multiple -clients at the same time operate on the same data. -This can cause transactions to wait on each other and decrease -performance, like when many people try to check out with the same -cashier at a store. - -For more information about contention, see [Understanding and Avoiding -Transaction -Contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention). - -## Does CockroachDB support `JOIN`? - -[CockroachDB supports SQL joins](joins.html). - -## Does CockroachDB support JSON or Protobuf datatypes? - -Yes, the [`JSONB`](jsonb.html) data type is supported. - -## How do I know which index CockroachDB will select for a query? - -To see which indexes CockroachDB is using for a given query, you can use the [`EXPLAIN`](explain.html) statement, which will print out the query plan, including any indexes that are being used: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN SELECT col1 FROM tbl1; -~~~ - -If you'd like to tell the query planner which index to use, you can do so via some [special syntax for index hints](table-expressions.html#force-index-selection): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT col1 FROM tbl1@idx1; -~~~ - -## How do I log SQL queries? - -You can enable the CockroachDB [logging channels](logging-overview.html#logging-channels) that record SQL events. - -## Does CockroachDB support a UUID type? - -Yes. For more details, see [`UUID`](uuid.html). - -## How does CockroachDB sort results when `ORDER BY` is not used? - -When an [`ORDER BY`](order-by.html) clause is not used in a query, rows are processed or returned in a -non-deterministic order. "Non-deterministic" means that the actual order -can depend on the logical plan, the order of data on disk, the topology -of the CockroachDB cluster, and is generally variable over time. - -## Why are my `INT` columns returned as strings in JavaScript? - -In CockroachDB, all `INT`s are represented with 64 bits of precision, but JavaScript numbers only have 53 bits of precision. This means that large integers stored in CockroachDB are not exactly representable as JavaScript numbers. For example, JavaScript will round the integer `235191684988928001` to the nearest representable value, `235191684988928000`. Notice that the last digit is different. This is particularly problematic when using the `unique_rowid()` [function](functions-and-operators.html), since `unique_rowid()` nearly always returns integers that require more than 53 bits of precision to represent. - -To avoid this loss of precision, Node's [`pg` driver](https://github.com/brianc/node-postgres) will, by default, return all CockroachDB `INT`s as strings. - -{% include_cached copy-clipboard.html %} -~~~ javascript -// Schema: CREATE TABLE users (id INT DEFAULT unique_rowid(), name STRING); -pgClient.query("SELECT id FROM users WHERE name = 'Roach' LIMIT 1", function(err, res) { - var idString = res.rows[0].id; - // idString === '235191684988928001' - // typeof idString === 'string' -}); -~~~ - -To perform another query using the value of `idString`, you can simply use `idString` directly, even where an `INT` type is expected. The string will automatically be coerced into a CockroachDB `INT`. - -{% include_cached copy-clipboard.html %} -~~~ javascript -pgClient.query("UPDATE users SET name = 'Ms. Roach' WHERE id = $1", [idString], function(err, res) { - // All should be well! -}); -~~~ - -If you instead need to perform arithmetic on `INT`s in JavaScript, you will need to use a big integer library like [Long.js](https://www.npmjs.com/package/long). Do _not_ use the built-in `parseInt` function. - -{% include_cached copy-clipboard.html %} -~~~ javascript -parseInt(idString, 10) + 1; // WRONG: returns 235191684988928000 -require('long').fromString(idString).add(1).toString(); // GOOD: returns '235191684988928002' -~~~ - -## Can I use CockroachDB as a key-value store? - -{% include {{ page.version.version }}/faq/simulate-key-value-store.html %} - - -## See also - -- [Product FAQs](frequently-asked-questions.html) -- [Operational FAQS](operational-faqs.html) diff --git a/src/current/v21.1/sql-feature-support.md b/src/current/v21.1/sql-feature-support.md deleted file mode 100644 index 4ee7ebab7c8..00000000000 --- a/src/current/v21.1/sql-feature-support.md +++ /dev/null @@ -1,188 +0,0 @@ ---- -title: SQL Feature Support in CockroachDB v21.1 -summary: Summary of CockroachDB's conformance to the SQL standard and which common extensions it supports. -toc: true -keywords: gin, gin index, gin indexes, inverted index, inverted indexes, accelerated index, accelerated indexes ---- - -Making CockroachDB easy to use is a top priority for us, so we chose to implement SQL. However, even though SQL has a standard, no database implements all of it, nor do any of them have standard implementations of all features. - -To understand which standard SQL features we support (as well as common extensions to the standard), use the table below. - -- **Component** lists the components that are commonly considered part of SQL. -- **Supported** shows CockroachDB's level of support for the component. -- **Type** indicates whether the component is part of the SQL *Standard* or is an *Extension* created by ourselves or others. -- **Details** provides greater context about the component. - - - -## Features - -### Row values - - Component | Supported | Type | Details ------------|-----------|------|--------- - `ARRAY` | ✓ | Standard | [`ARRAY` documentation](array.html) - `AUTO INCREMENT`
    (Automatic key generation) | Alternative | Common Extension | [Automatic key generation FAQ](sql-faqs.html#how-do-i-auto-generate-unique-row-ids-in-cockroachdb) - `BIT` | ✓ | Standard | [`BIT` documentation](bit.html) - `BYTES` | ✓ | CockroachDB Extension | [`BYTES` documentation](bytes.html) - `BOOLEAN` | ✓ | Standard | [`BOOL` documentation](bool.html) - `COLLATE` | ✓ | Standard | [`COLLATE` documentation](collate.html) - `DATE` | ✓ | Standard | [`DATE` documentation](date.html) - `DECIMAL`, `NUMERIC` | ✓ | Standard | [`DECIMAL` documentation](decimal.html) - `ENUM` | ✓ | PostgreSQL Extension | [`ENUM` documentation](enum.html) - `FLOAT`, `REAL` | ✓ | Standard | [`FLOAT` documentation](float.html) - `INET` | ✓ | PostgreSQL Extension | [`INET` documentation](inet.html) - `INT` | ✓ | Standard | [`INT` documentation](int.html) - `INTERVAL` | ✓ | Standard | [`INTERVAL` documentation](interval.html) - `JSON`/`JSONB` | ✓ | Common Extension | [`JSONB` documentation](jsonb.html) - `NULL` | ✓ | Standard | [*NULL*-handling documentation](null-handling.html) - `SERIAL`| ✓ | PostgreSQL Extension | [`SERIAL` documentation](serial.html) - `SET`| ✗ | MySQL| Only allow rows to contain values from a defined set of terms. - `STRING`, `CHARACTER` | ✓ | Standard | [`STRING` documentation](string.html) - `TIME` | ✓ | Standard | [`TIME` documentation](time.html) - `TIMESTAMP`/`TIMESTAMPTZ` | ✓ | Standard | [`TIMESTAMP` documentation](timestamp.html) - `UNSIGNED INT` | ✗ | Common Extension | `UNSIGNED INT` causes numerous casting issues, so we do not plan to support it. - `UUID` | ✓ | PostgreSQL Extension | [`UUID` documentation](uuid.html) - Identifiers | ✓ | Standard | [Identifiers documentation](keywords-and-identifiers.html#identifiers) - Key-value pairs | Alternative | Extension | [Key-Value FAQ](sql-faqs.html#can-i-use-cockroachdb-as-a-key-value-store) - XML | ✗ | Standard | XML data can be stored as `BYTES`, but we do not offer XML parsing. - -### Constraints - - Component | Supported | Type | Details ------------|-----------|------|--------- - Not Null | ✓ | Standard | [Not Null documentation](not-null.html) - Unique | ✓ | Standard | [Unique documentation](unique.html) - Primary Key | ✓ | Standard | [Primary Key documentation](primary-key.html) - Check | ✓ | Standard | [Check documentation](check.html) - Foreign Key | ✓ | Standard | [Foreign Key documentation](foreign-key.html) - Default Value | ✓ | Standard | [Default Value documentation](default-value.html) - -### Transactions - - Component | Supported | Type | Details ------------|-----------|------|--------- - Transactions (ACID semantics) | ✓ | Standard | [Transactions documentation](transactions.html) - `BEGIN` | ✓ | Standard | [`BEGIN` documentation](begin-transaction.html) - `COMMIT` | ✓ | Standard | [`COMMIT` documentation](commit-transaction.html) - `ROLLBACK` | ✓ | Standard | [`ROLLBACK` documentation](rollback-transaction.html) - `SAVEPOINT` | ✓ | Standard with CockroachDB extensions | CockroachDB supports nested transactions using [`SAVEPOINT`](savepoint.html) - -### Indexes - - Component | Supported | Type | Details ------------|-----------|------|--------- - Indexes | ✓ | Common Extension | [Indexes documentation](indexes.html) - Multi-column indexes | ✓ | Common Extension | We do not limit on the number of columns indexes can include - Covering indexes | ✓ | Common Extension | [Storing Columns documentation](create-index.html#store-columns) - GIN indexes | ✓ | Common Extension | [GIN Indexes documentation](inverted-indexes.html) - Partial indexes | ✓ | Common Extension | [Partial indexes documentation](partial-indexes.html) - Spatial indexes | ✓ | Common Extension | [Spatial indexes documentation](spatial-indexes.html) - Multiple indexes per query | Partial | Common Extension | [Use multiple indexes for a single query](https://github.com/cockroachdb/cockroach/issues/2142) - Full-text indexes | Planned | Common Extension | [GitHub issue tracking full-text index support](https://github.com/cockroachdb/cockroach/issues/7821) - Prefix/Expression Indexes | Potential | Common Extension | Apply expressions (such as `LOWER()`) to values before indexing them - Hash indexes | ✗ | Common Extension | Improves performance of queries looking for single, exact values - -### Schema changes - - Component | Supported | Type | Details ------------|-----------|------|--------- - `ALTER TABLE` | ✓ | Standard | [`ALTER TABLE` documentation](alter-table.html) - Database renames | ✓ | Standard | [`RENAME DATABASE` documentation](rename-database.html) - Table renames | ✓ | Standard | [`RENAME TABLE` documentation](rename-table.html) - Column renames | ✓ | Standard | [`RENAME COLUMN` documentation](rename-column.html) - Altering a column's data type | ✓ | Standard | [`ALTER COLUMN` documentation](alter-column.html#altering-column-data-types) - Adding columns | ✓ | Standard | [`ADD COLUMN` documentation](add-column.html) - Removing columns | ✓ | Standard | [`DROP COLUMN` documentation](drop-column.html) - Adding constraints | ✓ | Standard | [`ADD CONSTRAINT` documentation](add-constraint.html) - Removing constraints | ✓ | Standard | [`DROP CONSTRAINT` documentation](drop-constraint.html) - Index renames | ✓ | Standard | [`RENAME INDEX` documentation](rename-index.html) - Adding indexes | ✓ | Standard | [`CREATE INDEX` documentation](create-index.html) - Removing indexes | ✓ | Standard | [`DROP INDEX` documentation](drop-index.html) - Altering a primary key | ✓ | Standard | [`ALTER PRIMARY KEY` documentation](alter-primary-key.html) - Adding user-defined schemas | ✓ | Standard | [`CREATE SCHEMA` documentation](create-schema.html) - Removing user-defined schemas | ✓ | Standard | [`DROP SCHEMA` documentation](drop-schema.html) - Altering user-defined schemas | ✓ | Standard | [`ALTER SCHEMA` documentation](create-schema.html) - Converting a database to a user-defined schema | ✓ | CockroachDB Extension | [`CONVERT TO SCHEMA` documentation](convert-to-schema.html) - -### Statements - - Component | Supported | Type | Details ------------|-----------|------|--------- - Common statements | ✓ | Standard, PostgreSQL/CockroachDB Extension | [SQL Statements documentation](sql-statements.html) - `UPSERT` | ✓ | PostgreSQL, MSSQL Extension | [`UPSERT` documentation](upsert.html) - `EXPLAIN` | ✓ | Common Extension | [`EXPLAIN` documentation](explain.html) - `SELECT INTO` | Alternative | Common Extension | You can replicate similar functionality using [`CREATE TABLE`](create-table.html) and then `INSERT INTO ... SELECT ...`. - `SELECT FOR UPDATE` | ✓ | Common Extension | [`SELECT FOR UPDATE` documentation](select-for-update.html) - -### Clauses - - Component | Supported | Type | Details ------------|-----------|------|--------- - Common clauses | ✓ | Standard | [SQL Grammar documentation](sql-grammar.html) - `LIMIT` | ✓ | Common Extension | Limit the number of rows a statement returns. - `LIMIT` with `OFFSET` | ✓ | Common Extension | Skip a number of rows, and then limit the size of the return set. - `RETURNING` | ✓ | Common Extension | Retrieve a table of rows statements affect. - -### Table expressions - - Component | Supported | Type | Details ------------|-----------|------|--------- - Table and View references | ✓ | Standard | [Table expressions documentation](table-expressions.html#table-or-view-names) - `AS` in table expressions | ✓ | Standard | [Aliased table expressions documentation](table-expressions.html#aliased-table-expressions) - `JOIN` (`INNER`, `LEFT`, `RIGHT`, `FULL`, `CROSS`) | [Functional](https://www.cockroachlabs.com/blog/better-sql-joins-in-cockroachdb/) | Standard | [`JOIN` expressions documentation](table-expressions.html#join-expressions) - Sub-queries as table expressions | Partial | Standard | Non-correlated subqueries are [supported](table-expressions.html#subqueries-as-table-expressions), as are most [correlated subqueries](subqueries.html#correlated-subqueries). - Table generator functions | Partial | PostgreSQL Extension | [Table generator functions documentation](table-expressions.html#table-generator-functions) - `WITH ORDINALITY` | ✓ | CockroachDB Extension | [Ordinality annotation documentation](table-expressions.html#ordinality-annotation) - -### Scalar expressions and Boolean formulas - - Component | Supported | Type | Details ------------|-----------|------|--------- - Common functions | ✓ | Standard | [Functions calls and SQL special forms documentation](scalar-expressions.html#function-calls-and-sql-special-forms) - Common operators | ✓ | Standard | [Operators documentation](scalar-expressions.html#unary-and-binary-operations) - `IF`/`CASE`/`NULLIF` | ✓ | Standard | [Conditional expressions documentation](scalar-expressions.html#conditional-expressions) - `COALESCE`/`IFNULL` | ✓ | Standard | [Conditional expressions documentation](scalar-expressions.html#conditional-expressions) -`AND`/`OR` | ✓ | Standard | [Conditional expressions documentation](scalar-expressions.html#conditional-expressions) - `LIKE`/`ILIKE` | ✓ | Standard | [String pattern matching documentation](scalar-expressions.html#string-pattern-matching) - `SIMILAR TO` | ✓ | Standard | [SQL regexp pattern matching documentation](scalar-expressions.html#string-matching-using-sql-regular-expressions) - Matching using POSIX regular expressions | ✓ | Common Extension | [POSIX regexp pattern matching documentation](scalar-expressions.html#string-matching-using-posix-regular-expressions) - `EXISTS` | Partial | Standard | Non-correlated subqueries are [supported](scalar-expressions.html#existence-test-on-the-result-of-subqueries), as are most [correlated subqueries](subqueries.html#correlated-subqueries). Works only with small data sets. - Scalar subqueries | Partial | Standard | Non-correlated subqueries are [supported](scalar-expressions.html#scalar-subqueries), as are most [correlated subqueries](subqueries.html#correlated-subqueries). Works only with small data sets. - Bitwise arithmetic | ✓ | Common Extension | [Operators documentation](scalar-expressions.html#unary-and-binary-operations) - Array constructors and subscripting | Partial | PostgreSQL Extension | Array expression documentation: [Constructor syntax](scalar-expressions.html#array-constructors) and [Subscripting](scalar-expressions.html#subscripted-expressions) - `COLLATE`| ✓ | Standard | [Collation expressions documentation](scalar-expressions.html#collation-expressions) - Column ordinal references | ✓ | CockroachDB Extension | [Column references documentation](scalar-expressions.html#column-references) - Type annotations | ✓ | CockroachDB Extension | [Type annotations documentation](scalar-expressions.html#explicitly-typed-expressions) - -### Permissions - - Component | Supported | Type | Details ------------|-----------|------|--------- - Users | ✓ | Standard | [Users documentation](authorization.html#sql-users) - Roles | ✓ | Standard | [Roles documentation](authorization.html#roles) - Object ownership | ✓ | Common Extension | [Ownership documentation](authorization.html#object-ownership) - Privileges | ✓ | Standard | [Privileges documentation](authorization.html#assign-privileges) - -### Miscellaneous - - Component | Supported | Type | Details ------------|-----------|------|--------- - Column families | ✓ | CockroachDB Extension | [Column Families documentation](column-families.html) - Computed columns (stored and virtual) | ✓ | Common Extension | [Computed Columns documentation](computed-columns.html) - Interleaved tables | ✓ | CockroachDB Extension | [Interleaved Tables documentation](interleave-in-parent.html)
    {% include {{ page.version.version }}/misc/interleave-deprecation-note.md %} - Multi-region capabilities | ✓ | CockroachDB Extension | [Multi-region documentation](multiregion-overview.html) - System catalog schemas | ✓ | Standard, PostgreSQL/CockroachDB Extension | [`crdb_internal`](crdb-internal.html) (CockroachDB Extension)
    [`information_schema`](information-schema.html) (Standard)
    [`pg_catalog`](pg-catalog.html) (PostgreSQL Extension)
    [`pg_extension`](pg-extension.html) (PostgreSQL Extension) - Sequences | ✓ | Common Extension | [`CREATE SEQUENCE` documentation](create-sequence.html) - Views | ✓ | Standard | [Views documentation](views.html) - Materialized views | ✓ | Common Extension | [Materialized views documentation](views.html#materialized-views) - Window functions | ✓ | Standard | [Window Functions documentation](window-functions.html) - Common table expressions | Partial | Common Extension | [Common Table Expressions documentation](common-table-expressions.html) - Stored procedures | ✗ | Common Extension | Execute a procedure explicitly. - Cursors | ✗ | Standard | Traverse a table's rows. - Triggers | ✗ | Standard | Execute a set of commands whenever a specified event occurs. diff --git a/src/current/v21.1/sql-grammar.md b/src/current/v21.1/sql-grammar.md deleted file mode 100644 index 819540e4806..00000000000 --- a/src/current/v21.1/sql-grammar.md +++ /dev/null @@ -1,43 +0,0 @@ ---- -title: SQL Grammar -summary: The full SQL grammar for CockroachDB, generated automatically from the CockroachDB code. -toc: true -back_to_top: true ---- - - - -{{site.data.alerts.callout_success}} -This page describes the full CockroachDB SQL grammar. However, as a starting point, it's best to reference our [SQL statements pages](sql-statements.html) first, which provide detailed explanations and examples. -{{site.data.alerts.end}} - -{% comment %} -TODO: clean up the SQL diagrams not to link to these missing nonterminals. -{% endcomment %} - - - - - - - - - - - - - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/stmt_block.html %} -
    diff --git a/src/current/v21.1/sql-name-resolution.md b/src/current/v21.1/sql-name-resolution.md deleted file mode 100644 index c9eea4d1b8b..00000000000 --- a/src/current/v21.1/sql-name-resolution.md +++ /dev/null @@ -1,278 +0,0 @@ ---- -title: Name Resolution -summary: Object names can exist in multiple places in the naming hierarchy. Resolution decides which one to use. -toc: true ---- - -This page documents **name resolution** in CockroachDB. - -To reference an object (e.g., a table) in a query, you can specify a database, a schema, both, or neither. To resolve which object a query references, CockroachDB scans the [appropriate namespaces](#naming-hierarchy), following [a set of rules outlined below](#how-name-resolution-works). - -## Naming hierarchy - -For compatibility with PostgreSQL, CockroachDB supports a **three-level structure for names**. This is called the "naming hierarchy". - -In the naming hierarchy, the path to a stored object has three components: - -- database name -- schema name -- object name - -A CockroachDB cluster can store multiple databases. Each database can store multiple schemas, and each schema can store multiple [tables](create-table.html), [views](views.html), [sequences](create-sequence.html), and [user-defined types](enum.html). - -When you first [start a cluster](start-a-local-cluster.html), a number of [preloaded databases](show-databases.html#preloaded-databases) and schemas are included, including the `defaultdb` database and the `public` schema. By default, objects (e.g., tables) are stored in the preloaded `public` schema, in the [current database](#current-database) (`defaultdb`, by default). - -In addition to the `public` schema, CockroachDB supports a fixed set of [system catalog schemas](system-catalogs.html), available in every database, that provide ancillary, non-stored data to client applications. For example, [`information_schema`](information-schema.html) is provided for compatibility with the SQL standard, and [`pg_catalog`](pg-catalog.html) and [`pg_extension`](spatial-glossary.html#spatial-system-tables) are provided for compatibility with PostgreSQL. - -To create a new database, use a [`CREATE DATABASE`](create-database.html) statement. To create a new schema, use a [`CREATE SCHEMA`](create-schema.html) statement. The list of all databases can be obtained with [`SHOW DATABASES`](show-databases.html). The list of all schemas for a given database can be obtained with [`SHOW SCHEMAS`](show-schemas.html). The list of all objects for a given schema can be obtained with other `SHOW` statements. - -### Migrating namespaces from previous versions of CockroachDB - -In CockroachDB versions < v20.2, [user-defined schemas](create-schema.html) are not supported, and all objects created in a given database use the `public` schema. To provide a multi-level structure for stored objects in earlier versions of CockroachDB, we have recommended using [database](create-database.html) namespaces instead of schema namespaces. - -In CockroachDB versions >= v20.2, we recommend using schema namespaces, not database namespaces, to create a naming structure that is more similar to [PostgreSQL](https://www.postgresql.org/docs/current/ddl-schemas.html). - -If you are upgrading to v20.2, take any combination of the following actions after the upgrade is complete: - -- Use the [`ALTER DATABASE ... CONVERT TO SCHEMA`](convert-to-schema.html) statement to convert databases into schemas. This statement is particularly useful if you created databases for the sole purpose of creating a multi-level naming structure. When you convert a database to a schema, all tables, [sequences](create-sequence.html), and [user-defined types](enum.html) in the database become child objects of a new schema, and the database is deleted. Note that you cannot convert databases that contain user-defined schemas or [views](views.html). - -- [Create new schemas](create-schema.html) in databases on your cluster. After the schemas are created, use [`ALTER TABLE ... RENAME`](rename-table.html), [`ALTER SEQUENCE ... RENAME`](alter-sequence.html), [`ALTER TYPE ... RENAME`](alter-type.html), or [`ALTER VIEW ... RENAME`](alter-view.html) statements to move objects between databases as needed. To move objects between schemas, use [`ALTER TABLE ... SET SCHEMA`](set-schema.html), [`ALTER SEQUENCE ... SET SCHEMA`](alter-sequence.html), or [`ALTER VIEW ... SET SCHEMA`](alter-view.html). - -- If your cluster contains cross-database references (e.g., a cross-database foreign key reference, or a cross-database view reference), use the relevant [`ALTER TABLE`](alter-table.html), [`ALTER SEQUENCE`](alter-sequence.html), [`ALTER TYPE`](alter-type.html), or [`ALTER VIEW `](alter-view.html) statements to move any cross-referencing objects to the same database, but different schemas. Cross-database object references were allowed in earlier versions of CockroachDB to make database-object naming hierarchies more flexible for users. In v20.2, creating cross-database references are disabled for [foreign keys](foreign-key.html), [views](views.html), and [sequence ownership](create-sequence.html). For details, see [tracking issue](https://github.com/cockroachdb/cockroach/issues/55791). - -## How name resolution works - -Name resolution occurs separately to **look up existing objects** and to -**decide the full name of a new object**. - -The rules to look up an existing object are as follows: - -1. If the name already fully specifies the database and schema, use that information. -2. If the name has a single-component prefix (e.g., a schema name), try to find a schema with the prefix name in the [current database](#current-database) and [current schema](#current-schema). If that fails, try to find the object in the `public` schema of a database with the prefix name. -3. If the name has no prefix, use the [search path](#search-path) with the [current database](#current-database). - -Similarly, the rules to decide the full name of a new object are as follows: - -1. If the name already fully specifies the database and schema, use that. -2. If the name has a single-component prefix (e.g., a schema name), try to find a schema with that name. If no such schema exists, use the `public` schema in the database with the prefix name. -3. If the name has no prefix, use the [current schema](#current-schema) in the [current database](#current-database). - -## Parameters for name resolution - -### Current database - -The current database is used when a name is unqualified or has only one component prefix. It is the current value of the `database` session variable. - -- You can view the current value of the `database` session variable with [`SHOW -database`](show-vars.html) and change it with [`SET database`](set-vars.html). - -- You can inspect the list of valid database names that can be specified in `database` with [`SHOW DATABASES`](show-databases.html). - -- For client apps that connect to CockroachDB using a URL of the form `postgres://...`, the initial value of the `database` session variable can be set using the path component of the URL. For example, `postgres://node/mydb` sets `database` to `mydb` when the connection is established. - -### Search path - -The search path is used when a name is unqualified (i.e., has no prefix). It lists the schemas where objects are looked up. Its first element is also the [current schema](#current-schema) where new objects are created. - -- You can set the current search path with [`SET search_path`](set-vars.html) and inspected it with [`SHOW -search_path`](show-vars.html). -- You can inspect the list of valid schemas that can be listed in `search_path` with [`SHOW SCHEMAS`](show-schemas.html). -- By default, the search path contains `$user`, `public`, `pg_catalog`, and `pg_extension`. For compatibility with PostgreSQL, `pg_catalog` is forced to be present in `search_path` at all times, even when not specified with `SET search_path`. -- To mimic the behavior in PostgreSQL, CockroachDB will attempt a resolution to `pg_extension` prior to attempting `public`. - -### Current schema - -The current schema is used as target schema when creating a new object if the name is unqualified (has no prefix). - -- The current schema is always the first value of `search_path`, for compatibility with PostgreSQL. - -- You can inspect the current schema using the special built-in function/identifier `current_schema()`. - -## Index name resolution - -CockroachDB supports the following ways to specify an index name for statements that require one (e.g., [`DROP INDEX`](drop-index.html), [`ALTER INDEX ... RENAME`](alter-index.html), etc.): - -1. Index names are resolved relative to a table name using the `@` character, e.g., `DROP INDEX tbl@idx;`. This is the default and most common syntax. -2. Index names are resolved by searching all tables in the current schema to find a table with an index named `idx`, e.g., `DROP INDEX idx;` or (with optional schema prefix) `DROP INDEX public.idx;`. This syntax is necessary for Postgres compatibility because Postgres index names live in the schema namespace such that e.g., `public.idx` will resolve to the index `idx` of some table in the public schema. This capability is used by some ORMs. - -The name resolution algorithm for index names supports both partial and complete qualification, using the same [name resolution rules](#how-name-resolution-works) as other objects. - -## Examples - -The examples below use the following logical schema as a starting point: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE DATABASE mydb; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE mydb.mytable(x INT); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET database = mydb; -~~~ - -### Lookup with unqualified names - -An unqualified name is a name with no prefix, that is, a simple identifier. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM mytable; -~~~ - -This uses the search path over the current database. The search path is `$user` by default, in the current database. If a `$user` schema does not exist, the search path resolves to the `public` schema. In this case, there is no `$user` schema, and the resolved name is `mydb.public.mytable`. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET database = system; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM mytable; -~~~ - -~~~ -pq: relation "mytable" does not exist -~~~ - -This uses the search path over the current database, which is now -`system`. No schema in the search path contain table `mytable`, so the -look up fails with an error. - -### Lookup with fully qualified names - -A fully qualified name is a name with two prefix components, that is, -three identifiers separated by periods. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM mydb.public.mytable; -~~~ - -Both the database and schema components are specified. The lookup -succeeds if and only if the object exists at that specific location. - -### Lookup with partially qualified names - -A partially qualified name is a name with one prefix component, that is, two identifiers separated by a period. When a name is partially qualified, CockroachDB will try to use the prefix as a schema name first; and if that fails, use it as a database name. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM public.mytable; -~~~ - -This looks up `mytable` in the `public` schema of the current -database. If the current database is `mydb`, the lookup succeeds. - -To ease development in multi-database scenarios, CockroachDB also allows queries to specify a database name in a partially qualified name. For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM mydb.mytable; -~~~ - -In that case, CockroachDB will first attempt to find a schema called -`mydb` in the current database. When no such schema exists (which is -the case with the starting point in this section), it then tries to -find a database called `mydb` and uses the `public` schema in that. In -this example, this rule applies and the fully resolved name is -`mydb.public.mytable`. - -### Using the search path to use tables across schemas - -Suppose that a client frequently accesses a stored table as well as a virtual table in the [Information Schema](information-schema.html). Because `information_schema` is not in the search path by default, all queries that need to access it must mention it explicitly. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM mydb.information_schema.schemata; -- valid -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM information_schema.schemata; -- valid; uses mydb implicitly -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM schemata; -- invalid; information_schema not in search_path -~~~ - -For clients that use `information_schema` often, you can add it to the -search path to simplify queries. For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET search_path = public, information_schema; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM schemata; -- now valid, uses search_path -~~~ - -## Databases with special names - -When resolving a partially qualified name with just one component -prefix, CockroachDB will look up a schema with the given prefix name -first, and only look up a database with that name if the schema lookup -fails. This matters in the (likely uncommon) case where you wish your -database to be called `information_schema`, `public`, `pg_catalog`, [`pg_extension`](spatial-glossary.html#spatial-system-tables), -or `crdb_internal`. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE DATABASE public; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET database = mydb; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE public.mypublictable (x INT); -~~~ - -The [`CREATE TABLE`](create-table.html) statement in this example uses a partially -qualified name. Because the `public` prefix designates a valid schema -in the current database, the full name of `mypublictable` becomes -`mydb.public.mypublictable`. The table is created in database `mydb`. - -To create the table in database `public`, one would instead use a -fully qualified name, as follows: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE DATABASE public; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE public.public.mypublictable (x INT); -~~~ - -### Preloaded databases - -{% include {{ page.version.version }}/sql/preloaded-databases.md %} - -## See also - -- [`CREATE SCHEMA`](create-schema.html) -- [`SET`](set-vars.html) -- [`SHOW`](show-vars.html) -- [`SHOW DATABASES`](show-databases.html) -- [`SHOW SCHEMAS`](show-schemas.html) -- [Information Schema](information-schema.html) diff --git a/src/current/v21.1/sql-statements.md b/src/current/v21.1/sql-statements.md deleted file mode 100644 index f8bf0190a39..00000000000 --- a/src/current/v21.1/sql-statements.md +++ /dev/null @@ -1,202 +0,0 @@ ---- -title: SQL Statements -summary: Overview of SQL statements supported by CockroachDB. -toc: true ---- - -CockroachDB supports the following SQL statements. Click a statement for more details. - -{{site.data.alerts.callout_success}} -In the [built-in SQL shell](cockroach-sql.html#help), use `\h [statement]` to get inline help about a specific statement. -{{site.data.alerts.end}} - -## Data manipulation statements - -Statement | Usage -----------|------------ -[`CREATE TABLE AS`](create-table-as.html) | Create a new table in a database using the results from a [selection query](selection-queries.html). -[`COPY FROM`](copy-from.html) | Copy data from a third-party client to a CockroachDB cluster.
    Note that CockroachDB currently only supports `COPY FROM` statements issued from third-party clients, for compatibility with PostgreSQL drivers and ORMs. `COPY FROM` statements cannot be issued from the [`cockroach` SQL shell](cockroach-sql.html). To import data from files, we use an [`IMPORT`](import.html) statement instead. -[`DELETE`](delete.html) | Delete specific rows from a table. -[`EXPORT`](export.html) | Export an entire table's data, or the results of a `SELECT` statement, to CSV files. Note that this statement requires an [Enterprise license](enterprise-licensing.html). -[`IMPORT`](import.html) | Bulk-insert CSV data into a new table. -[`IMPORT INTO`](import-into.html) | Bulk-insert CSV data into an existing table. -[`INSERT`](insert.html) | Insert rows into a table. -[`SELECT`](select-clause.html) | Select specific rows and columns from a table and optionally compute derived values. -[`SELECT FOR UPDATE`](select-for-update.html) | Order transactions by controlling concurrent access to one or more rows of a table. -[`TABLE`](selection-queries.html#table-clause) | Select all rows and columns from a table. -[`TRUNCATE`](truncate.html) | Delete all rows from specified tables. -[`UPDATE`](update.html) | Update rows in a table. -[`UPSERT`](upsert.html) | Insert rows that do not violate uniqueness constraints; update rows that do. -[`VALUES`](selection-queries.html#values-clause) | Return rows containing specific values. - -## Data definition statements - -Statement | Usage -----------|------------ -[`ADD COLUMN`](add-column.html) | Add columns to a table. -[`ADD REGION`](add-region.html) | **New in v21.1:** Add a [region](multiregion-overview.html#database-regions) to a database. Note that [multi-region features](multiregion-overview.html) require an [Enterprise license](enterprise-licensing.html). -[`ADD CONSTRAINT`](add-constraint.html) | Add a constraint to a column. -[`ALTER COLUMN`](alter-column.html) | Change a column's [Default constraint](default-value.html), [`NOT NULL` constraint](not-null.html), or [data type](data-types.html). -[`ALTER DATABASE`](alter-database.html) | Apply a schema change to a database. -[`ALTER INDEX`](alter-index.html) | Apply a schema change to an index. -[`ALTER PARTITION`](alter-partition.html) | Configure the replication zone for a partition. Note that [partitioning](partitioning.html) requires an [Enterprise license](enterprise-licensing.html). -[`ALTER PRIMARY KEY`](alter-primary-key.html) | Change the [primary key](primary-key.html) of a table. -[`ALTER RANGE`](alter-range.html) | Configure the replication zone for a system range. -[`ALTER SCHEMA`](alter-schema.html) | Alter a user-defined schema. -[`ALTER SEQUENCE`](alter-sequence.html) | Apply a schema change to a sequence. -[`ALTER TABLE`](alter-table.html) | Apply a schema change to a table. -[`ALTER TYPE`](alter-type.html) | Modify a user-defined, [enumerated data type](enum.html). -[`ALTER USER`](alter-user.html) | add, change, or remove a user's password and to change the login privileges for a role. -[`ALTER ROLE`](alter-role.html) | Add, change, or remove a [role's](create-role.html) password and to change the login privileges for a role. -[`ALTER VIEW`](alter-view.html) | Apply a schema change to a view. -[`COMMENT ON`](comment-on.html) | Associate a comment to a database, table, or column. -[`CONFIGURE ZONE`](configure-zone.html) | Add, modify, reset, or remove a [replication zone](configure-replication-zones.html) for a database, table, index, partition, or system range. -[`CONVERT TO SCHEMA`](convert-to-schema.html) | Convert a database to a schema. -[`CREATE DATABASE`](create-database.html) | Create a new database. -[`CREATE INDEX`](create-index.html) | Create an index for a table. -[`CREATE SCHEMA`](create-schema.html) | Create a user-defined schema. -[`CREATE SEQUENCE`](create-sequence.html) | Create a new sequence. -[`CREATE TABLE`](create-table.html) | Create a new table in a database. -[`CREATE TABLE AS`](create-table-as.html) | Create a new table in a database using the results from a [selection query](selection-queries.html). -[`CREATE TYPE`](create-type.html) | Create a user-defined, [enumerated data type](enum.html). -[`CREATE VIEW`](create-view.html) | Create a new [view](views.html) in a database. -[`DROP COLUMN`](drop-column.html) | Remove columns from a table. -[`DROP CONSTRAINT`](drop-constraint.html) | Remove constraints from a column. -[`DROP DATABASE`](drop-database.html) | Remove a database and all its objects. -[`DROP INDEX`](drop-index.html) | Remove an index for a table. -[`DROP REGION`](drop-region.html) | **New in v21.1:** Drop a [region](multiregion-overview.html#database-regions) from a database. Note that [multi-region features](multiregion-overview.html) require an [Enterprise license](enterprise-licensing.html). -[`DROP SCHEMA`](drop-schema.html) | Drop a user-defined schema. -[`DROP SEQUENCE`](drop-sequence.html) | Remove a sequence. -[`DROP TABLE`](drop-table.html) | Remove a table. -[`DROP TYPE`](drop-type.html) | Remove a user-defined, [enumerated data type](enum.html). -[`DROP VIEW`](drop-view.html)| Remove a view. -[`EXPERIMENTAL_AUDIT`](experimental-audit.html) | Turn SQL audit logging on or off for a table. -[`PARTITION BY`](partition-by.html) | Partition, re-partition, or un-partition a table or secondary index. Note that [partitioning](partitioning.html) requires an [Enterprise license](enterprise-licensing.html). -[`REFRESH`](refresh.html) | Refresh the stored query results of a [materialized view](views.html#materialized-views). -[`RENAME COLUMN`](rename-column.html) | Rename a column in a table. -[`RENAME CONSTRAINT`](rename-constraint.html) | Rename a constraint on a column. -[`RENAME DATABASE`](rename-database.html) | Rename a database. -[`RENAME INDEX`](rename-index.html) | Rename an index for a table. -[`RENAME TABLE`](rename-table.html) | Rename a table or move a table between databases. -[`SET SCHEMA`](set-schema.html) | Change the schema of a table. -[`SET PRIMARY REGION`](set-primary-region.html) | **New in v21.1:** Assign a [primary region](multiregion-overview.html#database-regions) to a multi-region database, or change an existing primary region. Note that [multi-region features](multiregion-overview.html) require an [Enterprise license](enterprise-licensing.html). -[`SHOW COLUMNS`](show-columns.html) | View details about columns in a table. -[`SHOW CONSTRAINTS`](show-constraints.html) | List constraints on a table. -[`SHOW CREATE`](show-create.html) | View the `CREATE` statement for a table, view, or sequence. -[`SHOW DATABASES`](show-databases.html) | List databases in the cluster. -[`SHOW ENUMS`](show-enums.html) | List user-defined, [enumerated data types](enum.html) in a database. -[`SHOW FULL TABLE SCANS`](show-full-table-scans.html) | **New in v21.1:** List recent queries that used a full table scan. -[`SHOW INDEX`](show-index.html) | View index information for a table or database. -[`SHOW LOCALITY`](show-locality.html) | View the locality of the current node. -[`SHOW PARTITIONS`](show-partitions.html) | List partitions in a database. Note that [partitioning](partitioning.html) requires an [Enterprise license](enterprise-licensing.html). -[`SHOW REGIONS`](show-regions.html) | **New in v21.1:** List the [cluster regions](multiregion-overview.html#cluster-regions) or [database regions](multiregion-overview.html#database-regions) in a [multi-region cluster](multiregion-overview.html). -[`SHOW SCHEMAS`](show-schemas.html) | List the schemas in a database. -[`SHOW SEQUENCES`](show-sequences.html) | List the sequences in a database. -[`SHOW TABLES`](show-tables.html) | List tables or views in a database or virtual schema. -[`SHOW TYPES`](show-types.html) | List user-defined [data types](data-types.html) in a database. -[`SHOW RANGES`](show-ranges.html) | Show range information for all data in a table or index. -[`SHOW RANGE FOR ROW`](show-range-for-row.html) | Show range information for a single row in a table or index. -[`SHOW ZONE CONFIGURATIONS`](show-zone-configurations.html) | List details about existing [replication zones](configure-replication-zones.html). -[`SPLIT AT`](split-at.html) | Force a range split at the specified row in the table or index. -[`UNSPLIT AT`](unsplit-at.html) | Remove a range split enforcement at a specified row in the table or index. -[`VALIDATE CONSTRAINT`](validate-constraint.html) | Check whether values in a column match a [constraint](constraints.html) on the column. - -## Transaction management statements - -Statement | Usage -----------|------------ -[`BEGIN`](begin-transaction.html)| Initiate a [transaction](transactions.html). -[`COMMIT`](commit-transaction.html) | Commit the current [transaction](transactions.html). -[`SAVEPOINT`](savepoint.html) | Start a [nested transaction](transactions.html#nested-transactions). -[`RELEASE SAVEPOINT`](release-savepoint.html) | Commit a [nested transaction](transactions.html#nested-transactions). -[`ROLLBACK TO SAVEPOINT`](rollback-transaction.html#rollback-a-nested-transaction) | Roll back and restart the [nested transaction](transactions.html#nested-transactions) started at the corresponding `SAVEPOINT` statement. -[`ROLLBACK`](rollback-transaction.html) | Roll back the current [transaction](transactions.html) and all of its [nested transaction](transactions.html#nested-transactions), discarding all transactional updates made by statements inside the transaction. -[`SET TRANSACTION`](set-transaction.html) | Set the priority for the session or for an individual [transaction](transactions.html). -[`SHOW`](show-vars.html) | View the current [transaction settings](transactions.html). -[`SHOW TRANSACTIONS`](show-transactions.html) | View all currently active transactions across the cluster or on the local node. - -## Access management statements - -Statement | Usage -----------|------------ -[`CREATE ROLE`](create-role.html) | Create SQL [roles](authorization.html#create-and-manage-roles), which are groups containing any number of roles and users as members. -[`CREATE USER`](create-user.html) | Create SQL users, which lets you control [privileges](authorization.html#assign-privileges) on your databases and tables. -[`DROP ROLE`](drop-role.html) | Remove one or more SQL [roles](authorization.html#create-and-manage-roles). -[`DROP USER`](drop-user.html) | Remove one or more SQL users. -[`GRANT`](grant.html) | Grant privileges to [users](authorization.html#create-and-manage-users) or [roles](authorization.html#create-and-manage-roles), or add a [role](authorization.html#create-and-manage-roles) or [user](authorization.html#create-and-manage-users) as a member to a role. -[`REASSIGN OWNED`](reassign-owned.html) | **New in v21.1:** Change the [ownership](authorization.html#object-ownership) of all database objects in the current database that are currently owned by a specific [role](authorization.html#roles) or [user](authorization.html#sql-users). -[`REVOKE`](revoke.html) | Revoke privileges from [users](authorization.html#create-and-manage-users) or [roles](authorization.html#create-and-manage-roles), or revoke a [role](authorization.html#create-and-manage-roles) or [user's](authorization.html#create-and-manage-users) membership to a role. -[`SHOW GRANTS`](show-grants.html) | View privileges granted to users. -[`SHOW ROLES`](show-roles.html) | Lists the roles for all databases. -[`SHOW USERS`](show-users.html) | Lists the users for all databases. - -## Session management statements - -Statement | Usage -----------|------------ -[`RESET`](reset-vars.html) | Reset a session variable to its default value. -[`SET`](set-vars.html) | Set a current session variable. -[`SET TRANSACTION`](set-transaction.html) | Set the priority for an individual [transaction](transactions.html). -[`SHOW TRACE FOR SESSION`](show-trace.html) | Return details about how CockroachDB executed a statement or series of statements recorded during a session. -[`SHOW`](show-vars.html) | List the current session or transaction settings. - -## Cluster management statements - -Statement | Usage -----------|------------ -[`RESET CLUSTER SETTING`](reset-cluster-setting.html) | Reset a cluster setting to its default value. -[`SET CLUSTER SETTING`](set-cluster-setting.html) | Set a cluster-wide setting. -[`SHOW ALL CLUSTER SETTINGS`](show-cluster-setting.html) | List the current cluster-wide settings. -[`SHOW SESSIONS`](show-sessions.html) | List details about currently active sessions. -[`CANCEL SESSION`](cancel-session.html) | Cancel a long-running session. - -## Query management statements - -Statement | Usage -----------|------------ -[`CANCEL QUERY`](cancel-query.html) | Cancel a running SQL query. -[`SHOW STATEMENTS`/`SHOW QUERIES`](show-statements.html) | List details about current active SQL queries. - -## Query planning statements - -Statement | Usage -----------|------------ -[`CREATE STATISTICS`](create-statistics.html) | Create table statistics for the [cost-based optimizer](cost-based-optimizer.html) to use. -[`EXPLAIN`](explain.html) | View debugging and analysis details for a statement that operates over tabular data. -[`EXPLAIN ANALYZE`](explain-analyze.html) | Execute the query and generate a physical query plan with execution statistics. -[`SHOW STATISTICS`](show-statistics.html) | List table statistics used by the [cost-based optimizer](cost-based-optimizer.html). - - -## Job management statements - -Jobs in CockroachDB represent tasks that might not complete immediately, such as schema changes or Enterprise backups or restores. - -Statement | Usage -----------|------------ -[`CANCEL JOB`](cancel-job.html) | Cancel a `BACKUP`, `RESTORE`, `IMPORT`, or `CHANGEFEED` job. -[`PAUSE JOB`](pause-job.html) | Pause a `BACKUP`, `RESTORE`, `IMPORT`, or `CHANGEFEED` job. -[`RESUME JOB`](resume-job.html) | Resume a paused `BACKUP`, `RESTORE`, `IMPORT`, or `CHANGEFEED` job. -[`SHOW JOBS`](show-jobs.html) | View information on jobs. - -## Backup and restore statements - -Statement | Usage -----------|------------ -[`BACKUP`](backup.html) | Create disaster recovery backups of databases and tables. -[`RESTORE`](restore.html) | Restore databases and tables using your backups. -[`SHOW BACKUP`](show-backup.html) | List the contents of a backup. -[`CREATE SCHEDULE FOR BACKUP`](create-schedule-for-backup.html) | Create a schedule for periodic backups. -[`SHOW SCHEDULES`](show-schedules.html) | View information on backup schedules. -[`PAUSE SCHEDULES`](pause-schedules.html) | Pause backup schedules. -[`RESUME SCHEDULES`](resume-schedules.html) | Resume paused backup schedules. -[`DROP SCHEDULES`](drop-schedules.html) | Drop backup schedules. - - -## Changefeed statements (Enterprise) - -[Change data capture](stream-data-out-of-cockroachdb-using-changefeeds.html) (CDC) provides an Enterprise and core version of row-level change subscriptions for downstream processing. - -Statement | Usage -----------|------------ -[`CREATE CHANGEFEED`](create-changefeed.html) | _(Enterprise)_ Create a new changefeed to stream row-level changes in a configurable format to a configurable sink (Kafka or a cloud storage sink). -[`EXPERIMENTAL CHANGEFEED FOR`](changefeed-for.html) | _(Core)_ Create a new changefeed to stream row-level changes to the client indefinitely until the underlying connection is closed or the changefeed is canceled. diff --git a/src/current/v21.1/sql-tuning-with-explain.md b/src/current/v21.1/sql-tuning-with-explain.md deleted file mode 100644 index ad22aa2c6ce..00000000000 --- a/src/current/v21.1/sql-tuning-with-explain.md +++ /dev/null @@ -1,425 +0,0 @@ ---- -title: SQL Tuning with EXPLAIN -summary: How to use `EXPLAIN to identify and resolve SQL performance issues -toc: true ---- - -This tutorial guides you through the common reasons for [slow SQL statements](query-behavior-troubleshooting.html#identify-slow-statements) and describes how to use [`EXPLAIN`](explain.html) to troubleshoot the issues. - -The following examples use [MovR](movr.html), a fictional vehicle-sharing application, to demonstrate CockroachDB SQL statements. Run [`cockroach demo movr`](cockroach-demo.html) to open an interactive SQL shell to a temporary, in-memory cluster with the `movr` database preloaded and set as the [current database](sql-name-resolution.html#current-database). - -## Issue: Full table scans - -The most common reason for slow queries is sub-optimal `SELECT` statements that include full table scans and incorrect use of indexes. - -You'll get generally poor performance when retrieving a single row based on a column that is not in the primary key or any secondary index: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM users WHERE name = 'Cheyenne Smith'; -~~~ - -~~~ - id | city | name | address | credit_card --------------------------------------+----------+----------------+-------------------+-------------- -00e6afcc-e1c5-4258-8000-00000000002c | new york | Cheyenne Smith | 8550 Kelsey Flats | 4374468739 -(1 row) - -Time: 14ms total (execution 14ms / network 0ms) -~~~ - -To understand why this query performs poorly, use [`EXPLAIN`](explain.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN SELECT * FROM users WHERE name = 'Cheyenne Smith'; -~~~ - -~~~ - info ------------------------------------------------------------------------------------------ - distribution: full - vectorized: true - - • filter - │ estimated row count: 1 - │ filter: name = 'Cheyenne Smith' - │ - └── • scan - estimated row count: 12,500 (100% of the table; stats collected 56 minutes ago) - table: users@primary - spans: FULL SCAN -(11 rows) - -Time: 2ms total (execution 1ms / network 2ms) -~~~ - -`table: users@primary` indicates the index used (`primary`) to scan the table (`users`). `spans: FULL SCAN` shows you that, without a secondary index on the `name` column, CockroachDB scans every row of the `users` table, ordered by the primary key (`city`/`id`), until it finds the row with the correct `name` value. - -### Solution: Filter by a secondary index - -To speed up this query, add a secondary index on `name`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE INDEX on users (name); -~~~ - -The query will now return much faster: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM users WHERE name = 'Cheyenne Smith'; -~~~ - -~~~ - id | city | name | address | credit_card ----------------------------------------+----------+----------------+-------------------+-------------- - 00e6afcc-e1c5-4258-8000-00000000002c | new york | Cheyenne Smith | 8550 Kelsey Flats | 4374468739 -(1 row) - -Time: 7ms total (execution 7ms / network 0ms) -~~~ - -To understand why the performance improved, use [`EXPLAIN`](explain.html) to see the new query plan: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN SELECT * FROM users WHERE name = 'Cheyenne Smith'; -~~~ - -~~~ - info --------------------------------------------------------------------------------------- - distribution: local - vectorized: true - - • index join - │ estimated row count: 1 - │ table: users@primary - │ - └── • scan - estimated row count: 1 (<0.01% of the table; stats collected 58 minutes ago) - table: users@users_name_idx - spans: [/'Cheyenne Smith' - /'Cheyenne Smith'] -(11 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -This shows you that CockroachDB starts with the secondary index (`users@users_name_idx`). Because it is sorted by `name`, the query can jump directly to the relevant value (`/'Cheyenne Smith' - /'Cheyenne Smith'`). However, the query needs to return values not in the secondary index, so CockroachDB grabs the primary key (`city`/`id`) stored with the `name` value (the primary key is always stored with entries in a secondary index), jumps to that value in the primary index, and then returns the full row. - -Because the `users` table is under 512 MiB, the primary index and all secondary indexes are contained in a single range with a single leaseholder. If the table were bigger, however, the primary index and secondary index could reside in separate ranges, each with its own leaseholder. In this case, if the leaseholders were on different nodes, the query would require more network hops, further increasing latency. - -### Solution: Filter by a secondary index storing additional columns - -When you have a query that filters by a specific column but retrieves a subset of the table's total columns, you can improve performance by [storing](indexes.html#storing-columns) those additional columns in the secondary index to prevent the query from needing to scan the primary index as well. - -For example, let's say you frequently retrieve a user's name and credit card number: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT name, credit_card FROM users WHERE name = 'Cheyenne Smith'; -~~~ - -~~~ - name | credit_card ------------------+-------------- - Cheyenne Smith | 4374468739 -(1 row) - -Time: 6ms total (execution 6ms / network 0ms) -~~~ - -With the current secondary index on `name`, CockroachDB still needs to scan the primary index to get the credit card number: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN SELECT name, credit_card FROM users WHERE name = 'Cheyenne Smith'; -~~~ - - -~~~ - info -------------------------------------------------------------------------------------- - distribution: local - vectorized: true - - • index join - │ estimated row count: 1 - │ table: users@primary - │ - └── • scan - estimated row count: 1 (<0.01% of the table; stats collected 2 minutes ago) - table: users@users_name_idx - spans: [/'Cheyenne Smith' - /'Cheyenne Smith'] -(11 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -Let's drop and recreate the index on `name`, this time storing the `credit_card` value in the index: - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP INDEX users_name_idx; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE INDEX ON users (name) STORING (credit_card); -~~~ - -Now that `credit_card` values are stored in the index on `name`, CockroachDB only needs to scan that index: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN SELECT name, credit_card FROM users WHERE name = 'Cheyenne Smith'; -~~~ - -~~~ - info ---------------------------------------------------------------------------------- - distribution: local - vectorized: true - - • scan - estimated row count: 1 (<0.01% of the table; stats collected 2 minutes ago) - table: users@users_name_idx - spans: [/'Cheyenne Smith' - /'Cheyenne Smith'] -(7 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -This results in even faster performance: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT name, credit_card FROM users WHERE name = 'Cheyenne Smith'; -~~~ - -~~~ -name | credit_card ------------------+-------------- -Cheyenne Smith | 4374468739 -(1 row) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -To reset the database for following examples, let's drop the index on `name`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP INDEX users_name_idx; -~~~ - -## Issue: Joining data from different tables - -Secondary indexes are crucial when [joining](joins.html) data from different tables as well. - -For example, let's say you want to count the number of users who started rides on a given day. To do this, you need to use a join to get the relevant rides from the `rides` table and then map the `rider_id` for each of those rides to the corresponding `id` in the `users` table, counting each mapping only once: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT count(DISTINCT users.id) FROM users INNER JOIN rides ON rides.rider_id = users.id WHERE start_time BETWEEN '2018-12-16 00:00:00' AND '2018-12-17 00:00:00'; -~~~ - -~~~ - count ---------- - 17 -(1 row) - -Time: 4ms total (execution 3ms / network 0ms) -~~~ - -To understand what's happening, use [`EXPLAIN`](explain.html) to see the query plan: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN SELECT count(DISTINCT users.id) FROM users INNER JOIN rides ON rides.rider_id = users.id WHERE start_time BETWEEN '2018-12-16 00:00:00' AND '2018-12-17 00:00:00'; -~~~ - -~~~ - info ---------------------------------------------------------------------------------------------------------- - distribution: full - vectorized: true - - • group (scalar) - │ estimated row count: 1 - │ - └── • distinct - │ estimated row count: 14 - │ distinct on: id - │ - └── • hash join - │ estimated row count: 16 - │ equality: (id) = (rider_id) - │ - ├── • scan - │ estimated row count: 50 (100% of the table; stats collected 2 minutes ago) - │ table: users@primary - │ spans: FULL SCAN - │ - └── • filter - │ estimated row count: 16 - │ filter: (start_time >= '2018-12-16 00:00:00') AND (start_time <= '2018-12-17 00:00:00') - │ - └── • scan - estimated row count: 500 (100% of the table; stats collected 2 minutes ago) - table: rides@primary - spans: FULL SCAN -(27 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -CockroachDB does a full table scan first on `rides` to get all rows with a `start_time` in the specified range and then does another full table scan on `users` to find matching rows and calculate the count. - -Given the `WHERE` condition of the join, the full table scan of `rides` is particularly wasteful. - -### Solution: Create a secondary index on the `WHERE` condition storing the join key - -To speed up the query, you can create a secondary index on the `WHERE` condition (`rides.start_time`) storing the join key (`rides.rider_id`): - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE INDEX ON rides (start_time) STORING (rider_id); -~~~ - -Adding the secondary index reduced the query time: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT count(DISTINCT users.id) FROM users INNER JOIN rides ON rides.rider_id = users.id WHERE start_time BETWEEN '2018-12-16 00:00:00' AND '2018-12-17 00:00:00'; -~~~ - -~~~ - count ---------- - 17 -(1 row) - -Time: 2ms total (execution 2ms / network 0ms) -~~~ - -To understand why performance improved, again use [`EXPLAIN`](explain.html) to see the new query plan: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN SELECT count(DISTINCT users.id) FROM users INNER JOIN rides ON rides.rider_id = users.id WHERE start_time BETWEEN '2020-09-16 00:00:00' AND '2020-09-17 00:00:00'; -~~~ - -~~~ - info --------------------------------------------------------------------------------------------- - distribution: full - vectorized: true - - • group (scalar) - │ estimated row count: 1 - │ - └── • distinct - │ estimated row count: 14 - │ distinct on: id - │ - └── • hash join - │ estimated row count: 16 - │ equality: (id) = (rider_id) - │ - ├── • scan - │ estimated row count: 50 (100% of the table; stats collected 4 minutes ago) - │ table: users@primary - │ spans: FULL SCAN - │ - └── • scan - estimated row count: 16 (3.2% of the table; stats collected 4 minutes ago) - table: rides@rides_start_time_idx - spans: [/'2018-12-16 00:00:00' - /'2018-12-17 00:00:00'] -(23 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -Notice that CockroachDB now starts by using `rides@rides_start_time_idx` secondary index to retrieve the relevant rides without needing to scan the full `rides` table. - -## Issue: Inefficient joins - -[Hash joins](joins.html#hash-joins) are more expensive and require more memory than [lookup joins](joins.html#lookup-joins). Hence the [cost-based optimizer](cost-based-optimizer.html) uses a lookup join whenever possible. - -For the following query, the cost-based optimizer can’t perform a lookup join because the query doesn’t have a prefix of the `rides` table’s primary key available and thus has to read the entire table and search for a match, resulting in a slow query: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN SELECT * FROM vehicles JOIN rides on rides.vehicle_id = vehicles.id limit 1; -~~~ - -~~~ ------------------------------------------------------------------------------------------- - distribution: full - vectorized: true - - • limit - │ estimated row count: 1 - │ count: 1 - │ - └── • hash join - │ estimated row count: 500 - │ equality: (vehicle_id) = (id) - │ - ├── • scan - │ estimated row count: 500 (100% of the table; stats collected 23 seconds ago) - │ table: rides@primary - │ spans: FULL SCAN - │ - └── • scan - estimated row count: 15 (100% of the table; stats collected 4 minutes ago) - table: vehicles@primary - spans: FULL SCAN -(20 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -### Solution: Provide primary key to allow lookup join - -To speed up the query, you can provide the primary key to allow the cost-based optimizer to perform a lookup join instead of a hash join: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN SELECT * FROM vehicles JOIN rides ON rides.vehicle_id = vehicles.id and rides.city = vehicles.city limit 1; -~~~ - -~~~ - info ----------------------------------------------------------------------------------------- - distribution: full - vectorized: true - - • limit - │ estimated row count: 1 - │ count: 1 - │ - └── • lookup join - │ estimated row count: 56 - │ table: vehicles@primary - │ equality: (city, vehicle_id) = (city,id) - │ equality cols are key - │ - └── • scan - estimated row count: 500 (100% of the table; stats collected 1 minute ago) - table: rides@primary - spans: FULL SCAN -(17 rows) - -Time: 1ms total (execution 1ms / network 0ms) -~~~ - -## See also - -- [SQL Best Practices](performance-best-practices-overview.html) -- [Troubleshoot SQL Behavior](query-behavior-troubleshooting.html) -- [Optimize Statement Performance](make-queries-fast.html) diff --git a/src/current/v21.1/srid-4326.md b/src/current/v21.1/srid-4326.md deleted file mode 100644 index 4d89dca495f..00000000000 --- a/src/current/v21.1/srid-4326.md +++ /dev/null @@ -1,129 +0,0 @@ ---- -title: SRID 4326 - longitude and latitude -summary: The Spatial Reference System Identifier (SRID) 4326 represents spatial data using latitude and longitude coordinates on the Earth's surface. -toc: true ---- - -Every geometric shape has a spatial reference system associated with it, and each such reference system has a Spatial Reference System ID ([SRID](spatial-glossary.html#srid)). The SRID is used to tell which spatial reference system will be used to interpret each spatial object. - -A common SRID in use is 4326, which represents spatial data using longitude and latitude coordinates on the Earth's surface as defined in the [WGS84](spatial-glossary.html#wgs84) standard, which is also used for the [Global Positioning System (GPS)](https://en.wikipedia.org/wiki/Global_Positioning_System). - -Here is an example shape that uses SRID 4326, a line stretching from Milwaukee, WI to Tulsa, OK: - -~~~ -SRID=4326;LINESTRING( -87.906471 43.038902, -95.992775 36.153980) -~~~ - -Most users should only have to care about SRIDs in the following situations: - -- When creating shapes. By default, shapes which do not have an SRID associated with them will use an SRID of 0, which means "this shape has no SRID". -- When comparing shapes. If you try to compare two shapes with different SRIDs using a spatial predicate, CockroachDB will signal an error. - -## Adding SRIDs to shapes - -### Add an SRID with WKT - -To add an SRID to a shape you are defining with [Well Known Text](well-known-text.html), prepend the text with `SRID=N;` as shown below: - -~~~ -SRID=4326;POLYGON((-79.976111 40.374444, -74.621157 40.323294, -76.609383 39.299236, -79.976111 40.374444)) -~~~ - -### Add an SRID with SQL - -To add an SRID to a shape in SQL, use the `ST_SetSRID` [function](functions-and-operators.html#spatial-functions): - -{% include_cached copy-clipboard.html %} -~~~ sql -select ST_SetSRID(ST_MakePoint(1,2), 4326); -~~~ - -~~~ - st_setsrid ------------------------------------------------------- - 0101000020E6100000000000000000F03F0000000000000040 -(1 row) -~~~ - -### Change a shape's SRID - -To convert a shape to use a different SRID while maintaining the same reference location, use the `ST_Transform` function. In the example below, we transform the reference system for a line from Albany, NY to Saranac Lake, NY from 4326 to use the [StatePlane New York East reference (SRID 3101)](https://epsg.io/102715). This can be useful (or at least interesting) in some situations, since the StatePlane systems give distances in feet. - -{% include_cached copy-clipboard.html %} -~~~ sql -select ST_Transform(ST_GeomFromText('SRID=4326;LINESTRING(-73.756233 42.652580, -74.130833 44.326111)'),3101); -~~~ - -~~~ - st_transform ----------------------------------------------------------------------------------------------- - 01020000201D0C000002000000487C82CDD54D4D41EC8A5DA9AD726B4181403E98D7B34C417A15F9897C116B41 -(1 row) -~~~ - -{{site.data.alerts.callout_danger}} -When setting a shape's SRID, you can only use SRIDs that are defined in the [`spatial_ref_sys`](spatial-glossary.html#spatial_ref_sys) table. For more information, see [Getting SRID information](#getting-srid-information). -{{site.data.alerts.end}} - -## Getting SRID information - -You can get more detailed information about the SRIDs supported in CockroachDB from the [SQL client](cockroach-sql.html) by querying the [`spatial_ref_sys`](spatial-glossary.html#spatial_ref_sys) table. - -To see how many SRIDs are supported, run the following query: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT count(*) FROM spatial_ref_sys; -~~~ - -~~~ - count ---------- - 6139 -(1 row) -~~~ - -To get more information about a specific SRID (in this case 4326), run the following query: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT * FROM spatial_ref_sys WHERE srid = 4326; -~~~ - -~~~ - srid | auth_name | auth_srid | srtext | proj4text --------+-----------+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------- - 4326 | EPSG | 4326 | GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]] | +proj=longlat +datum=WGS84 +no_defs -(1 row) -~~~ - -## Comparing shapes with different SRIDs - -If you try to compare two shapes with different SRIDs using a spatial predicate, CockroachDB will signal an error: - -{% include_cached copy-clipboard.html %} -~~~ sql -select ST_Contains(ST_MakePoint(1,2), ST_SetSRID(ST_MakePoint(1,2), 4326)); -~~~ - -~~~ -ERROR: st_contains(): operation on mixed SRIDs forbidden: (Point, 0) != (Point, 4326) -~~~ - -## Known limitations - -{{site.data.alerts.callout_info}} -Defining a custom SRID by inserting rows into [`spatial_ref_sys`](spatial-glossary.html#spatial_ref_sys) is not currently supported. For more information, please see the tracking issue [cockroachdb/cockroach#55903](https://github.com/cockroachdb/cockroach/issues/55903). -{{site.data.alerts.end}} - -## See also - -- [Spatial features](spatial-features.html) -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial indexes](spatial-indexes.html) -- [Spatial and GIS Glossary of Terms](spatial-glossary.html) -- [Well known text](well-known-text.html) -- [Well known binary](well-known-binary.html) -- [GeoJSON](geojson.html) -- [Introducing Distributed Spatial Data in Free, Open Source CockroachDB](https://www.cockroachlabs.com/blog/spatial-data/) (blog post) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/sso.md b/src/current/v21.1/sso.md deleted file mode 100644 index b66237ab5a9..00000000000 --- a/src/current/v21.1/sso.md +++ /dev/null @@ -1,149 +0,0 @@ ---- -title: Single Sign-on (Enterprise) -summary: Implement single sign-on (SSO) for DB Console access. -toc: true ---- - -Single sign-on (SSO) allows a CockroachDB user to access the DB Console in a secure cluster via an external identity provider. When SSO is configured and enabled, the [DB Console login page](ui-overview.html#db-console-access) will display an OAuth login button in addition to the password access option. - -CockroachDB supports SSO via [OpenID Connect (OIDC)](https://openid.net/connect/), an authentication layer built on top of OAuth 2.0. - -{{site.data.alerts.callout_info}} -SSO authentication is an [Enterprise-only](enterprise-licensing.html) feature. -{{site.data.alerts.end}} - -## Requirements - -- An external OAuth 2.0 identity provider - -## Login flow - -This SSO implementation uses the [authorization code grant type](https://tools.ietf.org/html/rfc6749#section-4.1) to authenticate a user. - -1. A user opens the DB Console and clicks on the OAuth login button. -1. The user is redirected to an external identity provider. -1. The user successfully authenticates with the provider, completing the OAuth flow. -1. The user is redirected to the CockroachDB cluster via a callback URL. -1. The authorization code in the callback query is exchanged for an [ID Token](https://openid.net/specs/openid-connect-core-1_0.html#IDToken). -1. CockroachDB matches the ID Token to a registered SQL user. -1. CockroachDB creates a web session for the SQL user. -1. The user is redirected to the [DB Console Cluster Overview](ui-cluster-overview-page.html). - -## Cluster settings - -The following OIDC [cluster settings](cluster-settings.html) are used to configure and enable SSO. - -| Cluster Setting | Example Value -|-----------------|------ -| `server.oidc_authentication.enabled` | true -| `server.oidc_authentication.client_id` | '32789079457-g3hdfw8cbw85obi5cb525hsceaqf69unn.apps.googleusercontent.com' -| `server.oidc_authentication.client_secret` | '6Q_jmRMgrPNOc_mN91boe-9EP' -| `server.oidc_authentication.redirect_url` | 'https://localhost:8080/oidc/v1/callback' -| `server.oidc_authentication.provider_url` | 'https://accounts.google.com' -| `server.oidc_authentication.scopes` | 'openid profile email' -| `server.oidc_authentication.claim_json_key` | 'email' -| `server.oidc_authentication.principal_regex` | '^([^@]+)@[^@]+$' -| `server.oidc_authentication.autologin` | true - -- `enabled` is a Boolean that enables or disables SSO. -- `client_id` and `client_secret` are generated by the external identity provider. -- `redirect_url` specifies the callback URL that redirects the user to CockroachDB after a successful authentication. This can be the address of a node in the cluster or the address of a load balancer that routes traffic to the nodes. The path must be appended with `/oidc/v1/callback`. -- `provider_url` specifies the OAuth issuer identifier. Ensure that the URL does not have a terminating `/`. For more information, see the [OIDC specification](https://openid.net/specs/openid-connect-discovery-1_0.html#ProviderConfig). Note that CockroachDB appends the required `/.well-known/openid-configuration` by default. You do not need to include it. -- `scopes` is a space-delimited list of the [OAuth scopes](https://openid.net/specs/openid-connect-core-1_0.html#ScopeClaims) being requested for an Access Token. The `openid` scope must be included. -- `claim_json_key` is the key of the field to be extracted from the external identity provider's [ID Token](https://openid.net/specs/openid-connect-core-1_0.html#IDToken) and mapped to a SQL user. This field is likely to be a [standard claim](https://openid.net/specs/openid-connect-core-1_0.html#StandardClaims) such as `email` from which a username is extracted via `principal_regex`. -- `principal_regex` is a regular expression applied to the field specified by `claim_json_key`. It is used to extract an identifier that can be mapped to a SQL user. For example: - - `^([^@]+)@[^@]+$` matches any email address (defined as a string containing one `@` sign) and extracts a username from the string to the left of `@`. - - `^(.+)$` maps the claim directly to a principal. -- `autologin` is a Boolean. When `true`, upon loading the DB Console login page, the browser will automatically initiate the OIDC authentication process by redirecting to `/oidc/v1/login` instead of waiting for the user to log in manually or click the OIDC login button. - -## Example (Google OAuth 2.0) - -These steps demonstrate how to enable SSO authentication for the DB Console on a [local secure cluster](secure-a-cluster.html) using [Google's OAuth 2.0](https://developers.google.com/identity/protocols/oauth2) implementation. - -1. Open the [Credentials page](https://console.developers.google.com/apis/credentials) for your account at Google APIs. - -1. Click **+ CREATE CREDENTIALS** and select **OAuth client ID**. Specify a **web application** from the pulldown menu. - -1. Note the *client ID* and *client secret* of the OAuth 2.0 client. You can find these in the dialog that appears after you create the client, or in the details view for the client: - - Google OAuth 2.0 client details - -1. Add the callback URL to the list of **Authorized redirect URIs**. On a local cluster, this will be `https://localhost:8080/oidc/v1/callback`. You will later set `server.oidc_authentication.redirect_url` to the same value. - -1. Open a SQL shell to the cluster on node 1: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --certs-dir=certs --host=localhost:26257 - ~~~ - -1. Specify the client ID and client secret you obtained earlier: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING server.oidc_authentication.client_id = '\'; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING server.oidc_authentication.client_secret = '\'; - ~~~ - -1. Specify the OAuth issuer identifier: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING server.oidc_authentication.provider_url = 'https://accounts.google.com'; - ~~~ - -1. Specify the callback URL to redirect the user to the CockroachDB cluster: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING server.oidc_authentication.redirect_url = 'https://localhost:8080/oidc/v1/callback'; - ~~~ - -1. Specify the following values to obtain an OIDC identifier that will be mapped to a SQL user. - - Request the `openid` and `email` scopes from the Access Token: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING server.oidc_authentication.scopes = 'openid email'; - ~~~ - - Specify the `email` field from the ID Token: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING server.oidc_authentication.claim_json_key = 'email'; - ~~~ - - Use a regular expression that will extract a username from `email` that you can match to a SQL user. For example, `'^([^@]+)@cockroachlabs\.com$'` extracts the characters that precede `@cockroachlabs.com` in the email address. - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING server.oidc_authentication.principal_regex = '^([^@]+)@cockroachlabs.com$'; - ~~~ - -1. [Create a SQL user](create-user.html#create-a-user) that will log into the DB Console. The SQL username you specify must match the identifier obtained in the previous step. For example, a user with the email address `maxroach@cockroachlabs.com` will need the SQL username `maxroach`: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE USER maxroach; - ~~~ - -1. Finally, enable OIDC authentication: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING server.oidc_authentication.enabled = true; - ~~~ - - When the user [accesses the DB Console](ui-overview.html#db-console-access), they will be able to log in with their Google account. - - DB Console Single Sign-on - -{{site.data.alerts.callout_info}} -You can optionally enable the [`server.oidc_authentication.autologin` cluster setting](#cluster-settings) to automatically log in an authenticated user who visits the DB Console. -{{site.data.alerts.end}} diff --git a/src/current/v21.1/st_contains.md b/src/current/v21.1/st_contains.md deleted file mode 100644 index 7748e67f8f6..00000000000 --- a/src/current/v21.1/st_contains.md +++ /dev/null @@ -1,103 +0,0 @@ ---- -title: ST_Contains -summary: ST_Contains(A, B) returns true if no point in shape B lies outside of shape A, and at least one point in the interior of B lies in the interior of A. -toc: true -has_prefixed_variant: true ---- - -Given two shapes _A_ and _B_, the predicate function `ST_Contains(A, B)` returns `true` if: - -- No point in _B_ lies outside of shape _A_, and -- At least one point in the interior of _B_ lies in the interior of _A_. - -In other words, the exterior of shape _A_ must not include any point in _B_, and one or more points of _B_'s interior must lie in the interior of _A_. - -This behavior is similar to [`ST_Covers`](st_covers.html), except that the criteria are more exacting, and therefore some pairs of shapes will be rejected by this function that would be accepted by `ST_Covers`. - -`ST_Contains` works on the following data types: - -- [`GEOMETRY`](spatial-glossary.html#geometry) - -{% if page.has_prefixed_variant %} -{{site.data.alerts.callout_info}} -`{{page.title}}` will attempt to use any available [spatial index](spatial-indexes.html) to speed up its operation. Use the prefixed variant `_{{page.title}}` if you do not want any spatial indexes to be used. -{{site.data.alerts.end}} -{% endif %} - -{{site.data.alerts.callout_info}} -This function is the inverse of [`ST_Within`](st_within.html). -{{site.data.alerts.end}} - -## Examples - -{% include {{page.version.version}}/misc/geojson_geometry_note.md %} - -### True - -In this example, `{{page.title}}` returns `true` because: - -- No point in the LineString _B_ lies outside of the Polygon _A_, and -- At least one point in the interior of _B_ lies in the interior of _A_. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT ST_Contains(st_geomfromtext('SRID=4326;POLYGON((-87.906471 43.038902, -95.992775 36.153980, -75.704722 36.076944, -87.906471 43.038902))'), st_geomfromtext('SRID=4326;LINESTRING(-88.243385 40.116421, -87.906471 43.038902, -95.992775 36.153980)')); -~~~ - -~~~ - st_contains ---------------- - true - -(1 row) -~~~ - -ST_Contains - true - -### False - -In this example, `{{page.title}}` returns `false` because: - -- At least one point in the interior of LineString _B_ does not lie in the interior of the Polygon _A_. - -Note that A query against these shapes with [`ST_Covers`](st_covers.html) will return `true`. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT st_contains(st_geomfromtext('SRID=4326;POLYGON((-87.906471 43.038902, -95.992775 36.153980, -75.704722 36.076944, -87.906471 43.038902))'), st_geomfromtext('SRID=4326;LINESTRING( -87.906471 43.038902, -95.992775 36.153980)')); -~~~ - -~~~ - st_contains ---------------- - false -(1 row) -~~~ - -ST_Contains - false - -## See also - -- [Working with Spatial Data](spatial-data.html) -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial and GIS Glossary of Terms](spatial-glossary.html) -- [Spatial indexes](spatial-indexes.html) -- [Spatial functions](functions-and-operators.html#spatial-functions) -- [`ST_Covers`](st_covers.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Within`](st_within.html) -- [`ST_Intersects`](st_intersects.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Covers`](st_covers.html) -- [`ST_Disjoint`](st_disjoint.html) -- [`ST_Equals`](st_equals.html) -- [`ST_Overlaps`](st_overlaps.html) -- [`ST_Touches`](st_touches.html) -- [`ST_ConvexHull`](st_convexhull.html) -- [`ST_Union`](st_union.html) -- [Migrate from Shapefiles](migrate-from-shapefiles.html) -- [Migrate from GeoJSON](migrate-from-geojson.html) -- [Migrate from GeoPackage](migrate-from-geopackage.html) -- [Migrate from OpenStreetMap](migrate-from-openstreetmap.html) -- [Introducing Distributed Spatial Data in Free, Open Source CockroachDB](https://www.cockroachlabs.com/blog/spatial-data/) (blog post) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/st_convexhull.md b/src/current/v21.1/st_convexhull.md deleted file mode 100644 index b26d2c5b23c..00000000000 --- a/src/current/v21.1/st_convexhull.md +++ /dev/null @@ -1,270 +0,0 @@ ---- -title: ST_ConvexHull -summary: ST_ConvexHull(A) returns another shape B that is the convex hull of A. -toc: true -has_prefixed_variant: false ---- - -Given a shape _A_, `ST_ConvexHull(A)` returns another shape _B_ that is the [convex hull](https://en.wikipedia.org/wiki/Convex_hull) of _A_. The convex hull of a shape is the smallest [convex set](https://en.wikipedia.org/wiki/Convex_set) of points that [contains](st_contains.html) every point in the set that comprises that shape. - -In other words, given a set of points _A_ in the plane, the convex hull is the shape _B_ created by stretching an imaginary rubber band around the outermost points in _A_. - -`ST_ConvexHull` works on the following data types: - -- [`GEOMETRY`](spatial-glossary.html#geometry) - -{{site.data.alerts.callout_info}} -`ST_ConvexHull` is not an aggregate function. It operates on a single `GEOMETRY` object. This means that in practice it is most often used in combination with an aggregate function such as [`ST_Union`](st_union.html). -{{site.data.alerts.end}} - -## Examples - -In this example, we will generate the convex hull of a single geometry. The geometry is made from many individual points using [`ST_Union`](st_union.html). - -1. Create a temporary table to hold all the points, which will be in [Well Known Text (WKT)](spatial-glossary.html#wkt) format: - - {% include_cached copy-clipboard.html %} - ~~~ sql - CREATE TABLE tmp (ID UUID DEFAULT gen_random_uuid(), geom_text STRING); - ~~~ - -2. Insert the points with the following statement: - - {% include_cached copy-clipboard.html %} - ~~~ sql - INSERT INTO tmp (geom_text) VALUES - ('POINT (-73.962090000000003 40.609226)'), - ('POINT (-74.007309000000006 40.619892)'), - ('POINT (-73.949724000000003 41.498103999999998)'), - ('POINT (-72.390808000000007 40.961891999999999)'), - ('POINT (-77.937476000000004 43.218451000000002)'), - ('POINT (-73.842978000000002 40.944997999999998)'), - ('POINT (-73.928291999999999 40.680906999999998)'), - ('POINT (-73.914484999999999 40.775554)'), - ('POINT (-73.993281999999994 40.722833000000001)'), - ('POINT (-73.900373999999999 41.691662000000001)'), - ('POINT (-73.787897999999998 42.250467)'), - ('POINT (-75.062138000000004 42.454003)'), - ('POINT (-73.992576 40.753692000000001)'), - ('POINT (-75.647559999999999 44.590586000000002)'), - ('POINT (-73.698974000000007 40.832500000000003)'), - ('POINT (-73.957140999999993 40.728864000000002)'), - ('POINT (-73.954773000000003 40.732667999999997)'), - ('POINT (-74.024456000000001 41.521116999999997)'), - ('POINT (-73.785302000000001 43.081800000000001)'), - ('POINT (-74.860436000000007 43.040708000000002)'), - ('POINT (-74.001165999999998 40.687367000000002)'), - ('POINT (-78.809790000000007 42.845362000000002)'), - ('POINT (-75.061370999999994 41.608192000000003)'), - ('POINT (-76.499137000000005 42.440623000000002)'), - ('POINT (-73.379208000000006 43.027712000000001)'), - ('POINT (-73.684466999999998 41.259498000000001)'), - ('POINT (-74.003300999999993 40.706865999999998)'), - ('POINT (-76.384369000000007 43.034204000000003)'), - ('POINT (-73.983408999999995 40.723255000000002)'), - ('POINT (-74.670794000000001 42.371471999999997)'), - ('POINT (-73.963970000000003 40.804766999999998)'), - ('POINT (-73.989228999999995 40.743865)'), - ('POINT (-73.487143000000003 42.587166000000003)'), - ('POINT (-78.608046999999999 42.767758999999998)'), - ('POINT (-78.824476000000004 42.754263999999999)'), - ('POINT (-72.185676999999998 40.963121999999998)'), - ('POINT (-78.639556999999996 43.001435000000001)'), - ('POINT (-77.428916999999998 43.212463)'), - ('POINT (-73.964771999999996 40.770071999999999)'), - ('POINT (-73.937073999999996 40.578499000000001)'), - ('POINT (-74.076237000000006 40.636749000000002)'), - ('POINT (-73.923480999999995 41.091296)'), - ('POINT (-78.00206 42.718395000000001)'), - ('POINT (-74.992647000000005 43.018610000000002)'), - ('POINT (-73.991900000000001 40.683962000000001)'), - ('POINT (-74.119761999999994 42.040447)'), - ('POINT (-73.955706000000006 40.785203000000003)'), - ('POINT (-73.698363000000001 42.709778)'), - ('POINT (-75.296575000000004 43.098322000000003)'), - ('POINT (-73.804747000000006 42.709560000000003)'), - ('POINT (-79.326158000000007 42.102457000000001)'), - ('POINT (-73.938518000000002 40.837735000000002)'), - ('POINT (-73.976410999999999 40.672561999999999)'), - ('POINT (-73.978993000000003 40.745123999999997)'), - ('POINT (-77.587306999999996 43.159300999999999)'), - ('POINT (-74.001251999999994 40.734977000000001)'), - ('POINT (-76.206723999999994 43.102012000000002)'), - ('POINT (-78.876722000000001 42.922820999999999)'), - ('POINT (-73.495369999999994 44.703318000000003)'), - ('POINT (-79.052965999999998 43.105055)'), - ('POINT (-73.551120999999995 41.189729)'), - ('POINT (-73.760520999999997 40.809649999999998)'), - ('POINT (-73.952360999999996 42.077744000000003)'), - ('POINT (-76.511525000000006 43.456606000000001)'), - ('POINT (-75.036815000000004 42.448793000000002)'), - ('POINT (-75.544407000000007 42.826815000000003)'), - ('POINT (-73.686974000000006 40.724902)'), - ('POINT (-77.582922999999994 43.154963000000002)'), - ('POINT (-78.836257000000003 42.960456999999998)'), - ('POINT (-73.919056999999995 41.091124999999998)'), - ('POINT (-73.998487999999995 40.733795999999998)'), - ('POINT (-73.939606999999995 40.815913000000002)'), - ('POINT (-74.058107000000007 41.083153000000003)'), - ('POINT (-73.991060000000004 40.733919)'), - ('POINT (-73.967905000000002 40.758032999999998)'), - ('POINT (-73.875281000000001 41.016382999999998)'), - ('POINT (-77.134634000000005 42.961303000000001)'), - ('POINT (-78.885418999999999 42.906629000000002)'), - ('POINT (-73.553145999999998 40.665826000000003)'), - ('POINT (-73.834867000000003 42.621578999999997)'), - ('POINT (-73.982307000000006 40.752575)'), - ('POINT (-77.816855000000004 42.795613000000003)'), - ('POINT (-73.962288000000001 40.671320999999999)'), - ('POINT (-72.354918999999995 41.083874999999999)'), - ('POINT (-73.965331000000006 40.806128999999999)'), - ('POINT (-74.080895999999996 41.844256000000001)'), - ('POINT (-73.942249000000004 40.812187000000002)'), - ('POINT (-73.974255999999997 40.783757999999999)'), - ('POINT (-73.990206999999998 40.754969000000003)'), - ('POINT (-73.601747000000003 41.562685000000002)'), - ('POINT (-74.117880999999997 42.040733000000003)'), - ('POINT (-73.979851999999994 40.664327999999998)'), - ('POINT (-73.960042999999999 40.657589000000002)'), - ('POINT (-74.009314000000003 40.715324000000003)'), - ('POINT (-73.903509 40.703226000000001)'), - ('POINT (-74.030330000000006 40.624713999999997)'), - ('POINT (-73.909406000000004 40.814979000000001)'), - ('POINT (-73.903160999999997 40.744929999999997)'), - ('POINT (-73.504622999999995 41.952860000000001)'), - ('POINT (-73.865554000000003 42.219304000000001)'), - ('POINT (-73.981960000000001 40.766441)'), - ('POINT (-73.913732999999993 40.698230000000002)'), - ('POINT (-73.596703000000005 40.874966000000001)'), - ('POINT (-73.974082999999993 40.686098000000001)'), - ('POINT (-73.996341999999999 40.725087000000002)'), - ('POINT (-73.595579999999998 42.363210000000002)'), - ('POINT (-73.952259999999995 40.674118999999997)'), - ('POINT (-73.993341000000001 40.730010999999998)'), - ('POINT (-73.989086999999998 40.721902)'), - ('POINT (-73.955855999999997 40.655028000000001)'), - ('POINT (-74.002930000000006 40.731848999999997)'), - ('POINT (-76.086753000000002 44.242874)'), - ('POINT (-73.839067 42.681877999999998)'), - ('POINT (-74.020105000000001 41.935105)'), - ('POINT (-73.980250999999996 44.279376999999997)'), - ('POINT (-74.085936000000004 41.747742000000002)'), - ('POINT (-77.227361999999999 43.063468)'), - ('POINT (-74.093547000000001 40.924697999999999)'), - ('POINT (-74.986999999999995 44.669899999999998)'), - ('POINT (-73.793597000000005 41.135866)'), - ('POINT (-78.889482000000001 42.924545000000002)'), - ('POINT (-73.914946 41.936036999999999)'), - ('POINT (-73.9071 41.721612999999998)'), - ('POINT (-73.993579999999994 40.736569000000003)'), - ('POINT (-73.900388000000007 41.691189000000001)'), - ('POINT (-73.989261999999997 40.663339000000001)'), - ('POINT (-73.918380999999997 41.090746000000003)'), - ('POINT (-73.862252999999995 40.839613)'), - ('POINT (-73.806933000000001 40.989787999999997)'), - ('POINT (-73.635006000000004 40.744909)'), - ('POINT (-73.794276999999994 41.008797000000001)'), - ('POINT (-73.759060000000005 41.022123000000001)'), - ('POINT (-73.711965000000006 40.968676000000002)'), - ('POINT (-74.668248000000006 42.372920000000001)'), - ('POINT (-73.992476999999994 40.739109999999997)'), - ('POINT (-74.131673000000006 44.326892000000001)'), - ('POINT (-73.974738000000002 40.732664)'), - ('POINT (-73.812349999999995 40.798867000000001)'), - ('POINT (-78.641949999999994 42.991985)'), - ('POINT (-73.962226999999999 40.778944000000003)'), - ('POINT (-74.374174999999994 43.006495000000001)'), - ('POINT (-77.095714000000001 43.041158000000003)'), - ('POINT (-73.425764000000001 40.871898000000002)'), - ('POINT (-73.466729000000001 44.708036)'), - ('POINT (-73.978020999999998 40.686546999999997)'), - ('POINT (-73.974082999999993 40.686098000000001)'), - ('POINT (-73.971849000000006 40.602485000000001)'), - ('POINT (-73.983153999999999 40.690553000000001)'), - ('POINT (-77.052417000000005 42.143089000000003)'), - ('POINT (-73.776431000000002 40.934483)'), - ('POINT (-74.002019000000004 40.730927999999999)'), - ('POINT (-73.772469999999998 42.264814000000001)'), - ('POINT (-73.772876999999994 41.158641000000003)'), - ('POINT (-72.296062000000006 41.002409)'), - ('POINT (-74.021124999999998 41.933720000000001)'), - ('POINT (-74.058107000000007 41.083153000000003)'), - ('POINT (-73.967922999999999 40.801301000000002)'), - ('POINT (-72.294278000000006 41.000610999999999)'), - ('POINT (-76.499340000000004 42.438741999999998)'), - ('POINT (-73.976597999999996 40.786610000000003)'), - ('POINT (-73.988552999999996 40.703600000000002)'), - ('POINT (-74.004855000000006 40.741959000000001)'), - ('POINT (-73.844444999999993 40.925654999999999)'), - ('POINT (-74.005101999999994 40.746654999999997)'), - ('POINT (-73.937281999999996 42.819218999999997)'), - ('POINT (-74.001677000000001 40.736243000000002)'), - ('POINT (-73.684112999999996 40.981757000000002)'), - ('POINT (-73.919064000000006 40.832836999999998)'), - ('POINT (-78.885268999999994 42.944516)'), - ('POINT (-72.299519000000004 40.994070999999998)'), - ('POINT (-73.833477999999999 40.940247999999997)'), - ('POINT (-73.695717000000002 41.783996999999999)'), - ('POINT (-73.926891999999995 40.807422000000003)'), - ('POINT (-74.085899999999995 41.747892999999998)'), - ('POINT (-73.957583 41.417974999999998)'); - ~~~ - -3. Run the query below, which gathers the points into a single geometry using [`ST_Union`](st_union.html) that can be operated on by `ST_ConvexHull`, and converts the output to [GeoJSON](spatial-glossary.html#geojson) so that we can view it with [geojson.io](http://geojson.io): - - {% include_cached copy-clipboard.html %} - ~~~ sql - WITH - the_geoms_table - AS ( - SELECT - st_union(st_geomfromtext(geom_text)) AS the_union - FROM - tmp - ) - SELECT - st_asgeojson(st_convexhull(the_union)) - FROM - the_geoms_table; - ~~~ - - ~~~ json - {"type":"Polygon","coordinates":[[[-73.937074,40.578499],[-79.326158,42.102457],[-79.052966,43.105055],[-75.64756,44.590586],[-74.987,44.6699],[-73.466729,44.708036],[-72.185677,40.963122],[-73.937074,40.578499]]]} - ~~~ - -4. Paste the JSON emitted in the previous step into [geojson.io](http://geojson.io) and you should see an image like the following, which shows the convex hull surrounding the locations of [most of the independent bookstores in New York State](https://www.bookweb.org/member_directory/search/ABAmember/results/0/0/ny/0): - - ST_ConvexHull example - -5. Finally, drop the temporary table if you no longer need it: - - {% include_cached copy-clipboard.html %} - ~~~ sql - DROP TABLE tmp; - ~~~ - -## See also - -- [Working with Spatial Data](spatial-data.html) -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial and GIS Glossary of Terms](spatial-glossary.html) -- [Spatial indexes](spatial-indexes.html) -- [Spatial functions](functions-and-operators.html#spatial-functions) -- [`ST_Covers`](st_covers.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Contains`](st_contains.html) -- [`ST_Within`](st_within.html) -- [`ST_Intersects`](st_intersects.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Covers`](st_covers.html) -- [`ST_Disjoint`](st_disjoint.html) -- [`ST_Equals`](st_equals.html) -- [`ST_Overlaps`](st_overlaps.html) -- [`ST_Touches`](st_touches.html) -- [`ST_Union`](st_union.html) -- [Migrate from Shapefiles](migrate-from-shapefiles.html) -- [Migrate from GeoJSON](migrate-from-geojson.html) -- [Migrate from GeoPackage](migrate-from-geopackage.html) -- [Migrate from OpenStreetMap](migrate-from-openstreetmap.html) -- [Introducing Distributed Spatial Data in Free, Open Source CockroachDB](https://www.cockroachlabs.com/blog/spatial-data/) (blog post) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/st_coveredby.md b/src/current/v21.1/st_coveredby.md deleted file mode 100644 index 6e1d6992071..00000000000 --- a/src/current/v21.1/st_coveredby.md +++ /dev/null @@ -1,96 +0,0 @@ ---- -title: ST_CoveredBy -summary: ST_CoveredBy(A, B) returns true if no point in shape A lies outside of shape B -toc: true -has_prefixed_variant: true ---- - -Given two shapes _A_ and _B_, the predicate function `ST_CoveredBy(A, B)` returns `true` if no point in _A_ lies outside of shape _B_. Otherwise, it returns `false`. - -In other words, shape _B_ must completely cover every point in _A_. - -`ST_CoveredBy` works on the following data types: - -- [`GEOMETRY`](spatial-glossary.html#geometry) -- [`GEOGRAPHY`](spatial-glossary.html#geography) - - -{% if page.has_prefixed_variant %} -{{site.data.alerts.callout_info}} -`{{page.title}}` will attempt to use any available [spatial index](spatial-indexes.html) to speed up its operation. Use the prefixed variant `_{{page.title}}` if you do not want any spatial indexes to be used. -{{site.data.alerts.end}} -{% endif %} - -{{site.data.alerts.callout_info}} -This function is the inverse of [`ST_Covers`](st_covers.html). -{{site.data.alerts.end}} - -## Examples - -{% include {{page.version.version}}/misc/geojson_geometry_note.md %} - -### True - -In this example, `{{page.title}}` returns `true` because: - -- No Point in the smaller Polygon _A_ lies outside of the larger Polygon _B_. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT ST_CoveredBy(st_geomfromtext('SRID=4326;POLYGON((-87.623177 41.881832, -90.199402 38.627003, -82.446732 38.413651, -87.623177 41.881832))'), st_geomfromtext('SRID=4326;POLYGON((-87.906471 43.038902, -95.992775 36.153980, -75.704722 36.076944, -87.906471 43.038902))')); -~~~ - -~~~ - st_coveredby ----------------- - true -(1 row) -~~~ - -ST_CoveredBy - true - -### False - -In this example, `{{page.title}}` returns `false` because: - -- Many Points in the smaller Polygon _A_ lie outside of the larger Polygon _B_. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT ST_CoveredBy(st_geomfromtext('SRID=4326;POLYGON((-87.906471 43.038902, -95.992775 36.153980, -75.704722 36.076944, -87.906471 43.038902))'), st_geomfromtext('SRID=4326;POLYGON((-84.191605 39.758949, -75.165222 39.952583, -78.878738 42.880230, -84.191605 39.758949))')); -~~~ - -~~~ - st_coveredby ----------------- - false -(1 row) -~~~ - -ST_CoveredBy - false - -## See also - -- [Working with Spatial Data](spatial-data.html) -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial and GIS Glossary of Terms](spatial-glossary.html) -- [Spatial indexes](spatial-indexes.html) -- [Spatial functions](functions-and-operators.html#spatial-functions) -- [`ST_Covers`](st_covers.html) -- [`ST_Contains`](st_contains.html) -- [`ST_Within`](st_within.html) -- [`ST_Intersects`](st_intersects.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Covers`](st_covers.html) -- [`ST_Disjoint`](st_disjoint.html) -- [`ST_Equals`](st_equals.html) -- [`ST_Overlaps`](st_overlaps.html) -- [`ST_Touches`](st_touches.html) -- [`ST_ConvexHull`](st_convexhull.html) -- [`ST_Union`](st_union.html) -- [Migrate from Shapefiles](migrate-from-shapefiles.html) -- [Migrate from GeoJSON](migrate-from-geojson.html) -- [Migrate from GeoPackage](migrate-from-geopackage.html) -- [Migrate from OpenStreetMap](migrate-from-openstreetmap.html) -- [Introducing Distributed Spatial Data in Free, Open Source CockroachDB](https://www.cockroachlabs.com/blog/spatial-data/) (blog post) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/st_covers.md b/src/current/v21.1/st_covers.md deleted file mode 100644 index 176f8c73ba3..00000000000 --- a/src/current/v21.1/st_covers.md +++ /dev/null @@ -1,97 +0,0 @@ ---- -title: ST_Covers -summary: ST_Covers(A, B) returns true if no point in shape B lies outside of shape A -toc: true -has_prefixed_variant: true ---- - -Given two shapes _A_ and _B_, predicate function `ST_Covers(A, B)` returns `true` if no point in _B_ lies outside of shape _A_. Otherwise, it returns `false`. - -In other words, shape _A_ must completely cover every point in _B_. - -This behavior is similar to [`ST_Contains`](st_contains.html), except that the criteria for `ST_Contains` are more exacting, and therefore some pairs of shapes will be accepted by this function that would be rejected by `ST_Contains`. - -`ST_Covers` works on the following data types: - -- [`GEOMETRY`](spatial-glossary.html#geometry) -- [`GEOGRAPHY`](spatial-glossary.html#geography) - -{% if page.has_prefixed_variant %} -{{site.data.alerts.callout_info}} -`{{page.title}}` will attempt to use any available [spatial index](spatial-indexes.html) to speed up its operation. Use the prefixed variant `_{{page.title}}` if you do not want any spatial indexes to be used. -{{site.data.alerts.end}} -{% endif %} - -{{site.data.alerts.callout_info}} -This function is the inverse of [`ST_CoveredBy`](st_coveredby.html). -{{site.data.alerts.end}} - -## Examples - -{% include {{page.version.version}}/misc/geojson_geometry_note.md %} - -### True - -In this example, `{{page.title}}` returns `true` because: - -- No Point in the smaller Polygon _B_ lies outside of the larger Polygon _A_. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT ST_Covers(st_geomfromtext('SRID=4326;POLYGON((-87.906471 43.038902, -95.992775 36.153980, -75.704722 36.076944, -87.906471 43.038902))'), st_geomfromtext('SRID=4326;POLYGON((-87.623177 41.881832, -90.199402 38.627003, -82.446732 38.413651, -87.623177 41.881832))')); -~~~ - -~~~ - st_covers ---------------- - true - -(1 row) -~~~ - -ST_Covers - true - -### False - -In this example, `{{page.title}}` returns `false` because: - -- Many Points in the smaller Polygon _B_ lie outside of the larger Polygon _A_. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT ST_Covers(st_geomfromtext('SRID=4326;POLYGON((-87.906471 43.038902, -95.992775 36.153980, -75.704722 36.076944, -87.906471 43.038902))'), st_geomfromtext('SRID=4326;POLYGON((-84.191605 39.758949, -75.165222 39.952583, -78.878738 42.880230, -84.191605 39.758949))')); -~~~ - -~~~ - st_covers ---------------- - false -(1 row) -~~~ - -ST_Covers - false - -## See also - -- [Working with Spatial Data](spatial-data.html) -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial and GIS Glossary of Terms](spatial-glossary.html) -- [Spatial indexes](spatial-indexes.html) -- [Spatial functions](functions-and-operators.html#spatial-functions) -+ [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Contains`](st_contains.html) -- [`ST_Within`](st_within.html) -- [`ST_Intersects`](st_intersects.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Disjoint`](st_disjoint.html) -- [`ST_Equals`](st_equals.html) -- [`ST_Overlaps`](st_overlaps.html) -- [`ST_Touches`](st_touches.html) -- [`ST_ConvexHull`](st_convexhull.html) -- [`ST_Union`](st_union.html) -- [Migrate from Shapefiles](migrate-from-shapefiles.html) -- [Migrate from GeoJSON](migrate-from-geojson.html) -- [Migrate from GeoPackage](migrate-from-geopackage.html) -- [Migrate from OpenStreetMap](migrate-from-openstreetmap.html) -- [Introducing Distributed Spatial Data in Free, Open Source CockroachDB](https://www.cockroachlabs.com/blog/spatial-data/) (blog post) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/st_disjoint.md b/src/current/v21.1/st_disjoint.md deleted file mode 100644 index 2b1a0563cf7..00000000000 --- a/src/current/v21.1/st_disjoint.md +++ /dev/null @@ -1,97 +0,0 @@ ---- -title: ST_Disjoint -summary: ST_Disjoint(A, B) returns true if no point in shape A lies within shape B. -toc: true -has_prefixed_variant: false ---- - -Given two shapes _A_ and _B_, `ST_Disjoint(A, B)` returns `true` if the shapes do not share any of the same space -- that is, if no point in the set that comprises _A_ is also a member of the set of points that make up _B_. - -`ST_Disjoint` works on the following spatial data types: - -- [`GEOMETRY`](spatial-glossary.html#geometry) - -{% if page.has_prefixed_variant %} -{{site.data.alerts.callout_info}} -`{{page.title}}` will attempt to use any available [spatial index](spatial-indexes.html) to speed up its operation. Use the prefixed variant `_{{page.title}}` if you do not want any spatial indexes to be used. -{{site.data.alerts.end}} -{% else %} -{{site.data.alerts.callout_info}} -`{{page.title}}` does not make use of [spatial indexes](spatial-indexes.html). -{{site.data.alerts.end}} -{% endif %} - -{{site.data.alerts.callout_info}} -This function is the inverse of [`ST_Intersects`](st_intersects.html). -{{site.data.alerts.end}} - -## Examples - -{% include {{page.version.version}}/misc/geojson_geometry_note.md %} - -### True - -In this example, `{{page.title}}` returns `true` because: - -- No Point in the set that comprises Polygon _A_ is also a member of the set of points that make up Polygon _B_. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT st_disjoint(st_geomfromtext('SRID=4326;POLYGON((-87.906471 43.038902, -95.992775 36.153980, -75.704722 36.076944, -87.906471 43.038902), (-87.623177 41.881832, -90.199402 38.627003, -82.446732 38.413651, -87.623177 41.881832))'), st_geomfromtext('SRID=4326;POLYGON((-87.356934 41.595161, -84.512016 39.103119, -86.529167 39.162222, -87.356934 41.595161))')); -~~~ - -~~~ - st_disjoint ---------------- - true - -(1 row) -~~~ - -ST_Disjoint - true - -### False - -In this example, `{{page.title}}` returns `false` because: - -- Many Points in the set that comprises Polygon _A_ are also members of the set of points that make up Polygon _B_. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT st_disjoint(st_geomfromtext('SRID=4326;POLYGON((-87.906471 43.038902, -95.992775 36.153980, -75.704722 36.076944, -87.906471 43.038902))'), st_geomfromtext('SRID=4326;POLYGON((-84.191605 39.758949, -75.165222 39.952583, -78.878738 42.880230, -84.191605 39.758949))')); -~~~ - -~~~ - st_disjoint ---------------- - false -(1 row) -~~~ - -ST_Disjoint - false - -## See also - -- [Working with Spatial Data](spatial-data.html) -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial and GIS Glossary of Terms](spatial-glossary.html) -- [Spatial indexes](spatial-indexes.html) -- [Spatial functions](functions-and-operators.html#spatial-functions) -- [`ST_Covers`](st_covers.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Contains`](st_contains.html) -- [`ST_Within`](st_within.html) -- [`ST_Intersects`](st_intersects.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Covers`](st_covers.html) -- [`ST_Equals`](st_equals.html) -- [`ST_Overlaps`](st_overlaps.html) -- [`ST_Touches`](st_touches.html) -- [`ST_ConvexHull`](st_convexhull.html) -- [`ST_Union`](st_union.html) -- [Migrate from Shapefiles](migrate-from-shapefiles.html) -- [Migrate from GeoJSON](migrate-from-geojson.html) -- [Migrate from GeoPackage](migrate-from-geopackage.html) -- [Migrate from OpenStreetMap](migrate-from-openstreetmap.html) -- [Introducing Distributed Spatial Data in Free, Open Source CockroachDB](https://www.cockroachlabs.com/blog/spatial-data/) (blog post) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/st_equals.md b/src/current/v21.1/st_equals.md deleted file mode 100644 index 84c70a92c00..00000000000 --- a/src/current/v21.1/st_equals.md +++ /dev/null @@ -1,93 +0,0 @@ ---- -title: ST_Equals -summary: ST_Equals(A, B) returns true if every point that makes up shape A is also part of shape B, and vice versa. -toc: true -has_prefixed_variant: true ---- - -Given two shapes _A_ and _B_, `ST_Equals(A, B)` returns `true` if every point in the set of points that make up _A_ is also in _B_, and if every point in the set of points that make up _B_ is also in _A_. The ordering of the points in _A_ and _B_ may differ, but they must be made up of equivalent sets. - -Another way of describing `ST_Equals(A, B)` is that it will return `true` if both `ST_Within(A, B)` and `ST_Within(B, A)` also return `true`. - -`ST_Equals` works on the following data types: - -- [`GEOMETRY`](spatial-glossary.html#geometry) - -{% if page.has_prefixed_variant %} -{{site.data.alerts.callout_info}} -`{{page.title}}` will attempt to use any available [spatial index](spatial-indexes.html) to speed up its operation. Use the prefixed variant `_{{page.title}}` if you do not want any spatial indexes to be used. -{{site.data.alerts.end}} -{% endif %} - -## Examples - -{% include {{page.version.version}}/misc/geojson_geometry_note.md %} - -### True - -In this example, `{{page.title}}` returns `true` because: - -- Every Point in the set of Points that make up Polygon _A_ is also in Polygon _B_, and every Point in the set of Points that make up Polygon _B_ is also in Polygon _A_ - -As mentioned above, the ordering of the points in _A_ and _B_ does not matter. Below, there are two triangles with the same vertices, which are ordered differently. In the screenshot, one triangle is drawn in yellow, and the other blue. The blue and yellow strokes of the two separate triangles' boundaries are visible. Because the images are overlaid on each other, the yellow and blue combine to make part of the exterior outline of the image a green color. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT st_equals(st_geomfromtext('SRID=4326;POLYGON((-87.906471 43.038902, -95.992775 36.153980, -75.704722 36.076944, -87.906471 43.038902))'), st_geomfromtext('SRID=4326;POLYGON((-95.992775 36.153980, -87.906471 43.038902, -75.704722 36.076944, -95.992775 36.153980))')); -~~~ - -~~~ - st_equals ---------------- - true - -(1 row) -~~~ - -ST_Equals - true - -### False - -In this example, `{{page.title}}` returns `false` because: - -- It's obvious that not every Point that makes up the set of Points in Polygon _A_ is not also in Polygon _B_. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT st_equals(st_geomfromtext('SRID=4326;POLYGON((-87.906471 43.038902, -95.992775 36.153980, -75.704722 36.076944, -87.906471 43.038902))'), st_geomfromtext('SRID=4326;POLYGON((-84.191605 39.758949, -75.165222 39.952583, -78.878738 42.880230, -84.191605 39.758949))')); -~~~ - -~~~ - st_equals ---------------- - false -(1 row) -~~~ - -ST_Equals - false - -## See also - -- [Working with Spatial Data](spatial-data.html) -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial and GIS Glossary of Terms](spatial-glossary.html) -- [Spatial indexes](spatial-indexes.html) -- [Spatial functions](functions-and-operators.html#spatial-functions) -- [`ST_Covers`](st_covers.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Contains`](st_contains.html) -- [`ST_Within`](st_within.html) -- [`ST_Intersects`](st_intersects.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Covers`](st_covers.html) -- [`ST_Disjoint`](st_disjoint.html) -- [`ST_Overlaps`](st_overlaps.html) -- [`ST_Touches`](st_touches.html) -- [`ST_ConvexHull`](st_convexhull.html) -- [`ST_Union`](st_union.html) -- [Migrate from Shapefiles](migrate-from-shapefiles.html) -- [Migrate from GeoJSON](migrate-from-geojson.html) -- [Migrate from GeoPackage](migrate-from-geopackage.html) -- [Migrate from OpenStreetMap](migrate-from-openstreetmap.html) -- [Introducing Distributed Spatial Data in Free, Open Source CockroachDB](https://www.cockroachlabs.com/blog/spatial-data/) (blog post) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/st_intersects.md b/src/current/v21.1/st_intersects.md deleted file mode 100644 index c5f1e29bd72..00000000000 --- a/src/current/v21.1/st_intersects.md +++ /dev/null @@ -1,94 +0,0 @@ ---- -title: ST_Intersects -summary: ST_Intersects(A, B) returns true if any point in shape A lies within shape B. -toc: true -has_prefixed_variant: true ---- - -Given two shapes _A_ and _B_, `ST_Intersects(A, B)` returns `true` if the shapes share any of the same space -- that is, if any point in the set that comprises _A_ is also a member of the set of points that make up _B_. - -`ST_Intersects` works on the following data types: - -- [`GEOMETRY`](spatial-glossary.html#geometry) -- [`GEOGRAPHY`](spatial-glossary.html#geography) - -{% if page.has_prefixed_variant %} -{{site.data.alerts.callout_info}} -`{{page.title}}` will attempt to use any available [spatial index](spatial-indexes.html) to speed up its operation. Use the prefixed variant `_{{page.title}}` if you do not want any spatial indexes to be used. -{{site.data.alerts.end}} -{% endif %} - -{{site.data.alerts.callout_info}} -This function is the inverse of [`ST_Disjoint`](st_disjoint.html). -{{site.data.alerts.end}} - -## Examples - -{% include {{page.version.version}}/misc/geojson_geometry_note.md %} - -### True - -In this example, `{{page.title}}` returns `true` because: - -- The shapes share some of the same space -- that is, there are Points in the set that comprises Polygon _A_ that are also members of Polygon _B_. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT st_intersects(st_geomfromtext('SRID=4326;POLYGON((-87.906471 43.038902, -95.992775 36.153980, -75.704722 36.076944, -87.906471 43.038902))'), st_geomfromtext('SRID=4326;POLYGON((-84.191605 39.758949, -75.165222 39.952583, -78.878738 42.880230, -84.191605 39.758949))')); -~~~ - -~~~ - st_intersects ---------------- - true - -(1 row) -~~~ - -ST_Intersects - true - -### False - -In this example, `{{page.title}}` returns `false` because: - -- The shapes do not share any of the same space -- that is, there are no Points in the set that comprises Polygon _A_ that are also members of Polygon _B_. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT st_intersects(st_geomfromtext('SRID=4326;POLYGON((-87.906471 43.038902, -95.992775 36.153980, -75.704722 36.076944, -87.906471 43.038902))'), st_geomfromtext('SRID=4326;POLYGON((-79.995888 40.440624,-74.666728 40.358244, -76.5 42.443333, -79.995888 40.440624))')); -~~~ - -~~~ - st_intersects ---------------- - false -(1 row) -~~~ - -ST_Intersects - false - -## See also - -- [Working with Spatial Data](spatial-data.html) -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial and GIS Glossary of Terms](spatial-glossary.html) -- [Spatial indexes](spatial-indexes.html) -- [Spatial functions](functions-and-operators.html#spatial-functions) -- [`ST_Covers`](st_covers.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Contains`](st_contains.html) -- [`ST_Within`](st_within.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Covers`](st_covers.html) -- [`ST_Disjoint`](st_disjoint.html) -- [`ST_Equals`](st_equals.html) -- [`ST_Overlaps`](st_overlaps.html) -- [`ST_Touches`](st_touches.html) -- [`ST_ConvexHull`](st_convexhull.html) -- [`ST_Union`](st_union.html) -- [Migrate from Shapefiles](migrate-from-shapefiles.html) -- [Migrate from GeoJSON](migrate-from-geojson.html) -- [Migrate from GeoPackage](migrate-from-geopackage.html) -- [Migrate from OpenStreetMap](migrate-from-openstreetmap.html) -- [Introducing Distributed Spatial Data in Free, Open Source CockroachDB](https://www.cockroachlabs.com/blog/spatial-data/) (blog post) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/st_overlaps.md b/src/current/v21.1/st_overlaps.md deleted file mode 100644 index 5e27ecb568e..00000000000 --- a/src/current/v21.1/st_overlaps.md +++ /dev/null @@ -1,95 +0,0 @@ ---- -title: ST_Overlaps -summary: ST_Overlaps(A, B) returns true if the shapes intersect, but neither is contained within the other. -toc: true -has_prefixed_variant: true ---- - -Given two shapes _A_ and _B_, `ST_Overlaps(A, B)` returns `true` if the following criteria are met: - -- The shapes share any of the same space -- that is, if any point in the set that comprises _A_ is also a member of the set of points that make up _B_. -- Neither of the shapes is contained by the other, in the [`ST_Contains`](st_contains.html) sense. - -In other words, `ST_Overlaps` returns `true` if the shapes intersect (in the [`ST_Intersects`](st_intersects.html) sense), but neither is contained within the other. - -`ST_Overlaps` works on the following spatial data types: - -- [`GEOMETRY`](spatial-glossary.html#geometry) - -{% if page.has_prefixed_variant %} -{{site.data.alerts.callout_info}} -`{{page.title}}` will attempt to use any available [spatial index](spatial-indexes.html) to speed up its operation. Use the prefixed variant `_{{page.title}}` if you do not want any spatial indexes to be used. -{{site.data.alerts.end}} -{% endif %} - -## Examples - -{% include {{page.version.version}}/misc/geojson_geometry_note.md %} - -### True - -In this example, `{{page.title}}` returns `true` because: - -- There is a Point in the set that comprises Polygon _A_ that is also a member of the set of Points that make up Polygon _B_. -- Neither of the shapes is contained by the other, in the [`ST_Contains`](st_contains.html) sense. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT st_overlaps(st_geomfromtext('SRID=4326;POLYGON((-87.906471 43.038902, -95.992775 36.153980, -75.704722 36.076944, -87.906471 43.038902))'), st_geomfromtext('SRID=4326;POLYGON((-84.191605 39.758949, -75.165222 39.952583, -78.878738 42.880230, -84.191605 39.758949))')); -~~~ - -~~~ - st_overlaps ---------------- - true - -(1 row) -~~~ - -ST_Overlaps - true - -### False - -In this example, `{{page.title}}` returns `false` because: - -- - There is not any Point in the set that comprises Polygon _A_ that is also a member of the set of Points that make up Polygon _B_. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT st_overlaps(st_geomfromtext('SRID=4326;POLYGON((-79.995888 40.440624,-74.666728 40.358244, -76.5 42.443333, -79.995888 40.440624))'), st_geomfromtext('SRID=4326;POLYGON((-79.976111 40.374444, -74.621157 40.323294, -76.609383 39.299236, -79.976111 40.374444))')); -~~~ - -~~~ - st_overlaps ---------------- - false -(1 row) -~~~ - -ST_Overlaps - false - -## See also - -- [Working with Spatial Data](spatial-data.html) -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial and GIS Glossary of Terms](spatial-glossary.html) -- [Spatial indexes](spatial-indexes.html) -- [Spatial functions](functions-and-operators.html#spatial-functions) -- [`ST_Covers`](st_covers.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Contains`](st_contains.html) -- [`ST_Within`](st_within.html) -- [`ST_Intersects`](st_intersects.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Covers`](st_covers.html) -- [`ST_Disjoint`](st_disjoint.html) -- [`ST_Equals`](st_equals.html) -- [`ST_Touches`](st_touches.html) -- [`ST_ConvexHull`](st_convexhull.html) -- [`ST_Union`](st_union.html) -- [Migrate from Shapefiles](migrate-from-shapefiles.html) -- [Migrate from GeoJSON](migrate-from-geojson.html) -- [Migrate from GeoPackage](migrate-from-geopackage.html) -- [Migrate from OpenStreetMap](migrate-from-openstreetmap.html) -- [Introducing Distributed Spatial Data in Free, Open Source CockroachDB](https://www.cockroachlabs.com/blog/spatial-data/) (blog post) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/st_touches.md b/src/current/v21.1/st_touches.md deleted file mode 100644 index e44fb04d551..00000000000 --- a/src/current/v21.1/st_touches.md +++ /dev/null @@ -1,95 +0,0 @@ ---- -title: ST_Touches -summary: ST_Touches(A, B) returns true if A and B have at least one point in common, but their interiors do not intersect. -toc: true -has_prefixed_variant: true ---- - -Given two shapes _A_ and _B_, `ST_Touches(A, B)` returns `true` if both the following are true: - -- At least one point in the set of points that comprises _A_ is also a member of the set of points that make up _B_. -- No points that make up the interior of _A_ are also part of the interior of _B_. - -In other words, _A_ and _B_ have a point along their boundaries in common (i.e., they "touch"), but none of their interior points intersect. This distinction between shapes touching along a boundary vs. intersecting is also made by the [DE-9IM](spatial-glossary.html#de-9IM) standard. - -`ST_Touches` works on the following data types: - -- [`GEOMETRY`](spatial-glossary.html#geometry) - -{% if page.has_prefixed_variant %} -{{site.data.alerts.callout_info}} -`{{page.title}}` will attempt to use any available [spatial index](spatial-indexes.html) to speed up its operation. Use the prefixed variant `_{{page.title}}` if you do not want any spatial indexes to be used. -{{site.data.alerts.end}} -{% endif %} - -## Examples - -{% include {{page.version.version}}/misc/geojson_geometry_note.md %} - -### True - -In this example, `{{page.title}}` returns `true` because both of the following are true: - -- At least one point in the set of Points that comprise Polygon _A_ is a member of the set of points that make up the LineString _B_. -- No points from the interior of _A_ are also part of the interior of _B_. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT st_touches(st_geomfromtext('SRID=4326;POLYGON((-87.906471 43.038902, -95.992775 36.153980, -75.704722 36.076944, -87.906471 43.038902), (-87.623177 41.881832, -90.199402 38.627003, -82.446732 38.413651, -87.623177 41.881832))'), st_geomfromtext('SRID=4326;LINESTRING(-87.623177 41.881832, -90.199402 38.627003, -82.446732 38.413651, -87.623177 41.881832)')); -~~~ - -~~~ - st_touches ---------------- - true - -(1 row) -~~~ - -ST_Touches - true - -### False - -In this example, `{{page.title}}` returns `false` because: - -- Some points from the interior of the LineString _B_ are also part of the interior of the Polygon _A_. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT st_touches(st_geomfromtext('SRID=4326;POLYGON((-87.906471 43.038902, -95.992775 36.153980, -75.704722 36.076944, -87.906471 43.038902))'), st_geomfromtext('SRID=4326;LINESTRING(-88.243385 40.116421, -87.906471 43.038902, -95.992775 36.153980, -95.235278 38.971667)')); -~~~ - -~~~ - st_touches ---------------- - false -(1 row) -~~~ - -ST_Touches - false - -## See also - -- [Working with Spatial Data](spatial-data.html) -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial and GIS Glossary of Terms](spatial-glossary.html) -- [Spatial indexes](spatial-indexes.html) -- [Spatial functions](functions-and-operators.html#spatial-functions) -- [`ST_Covers`](st_covers.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Contains`](st_contains.html) -- [`ST_Within`](st_within.html) -- [`ST_Intersects`](st_intersects.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Covers`](st_covers.html) -- [`ST_Disjoint`](st_disjoint.html) -- [`ST_Equals`](st_equals.html) -- [`ST_Overlaps`](st_overlaps.html) -- [`ST_ConvexHull`](st_convexhull.html) -- [`ST_Union`](st_union.html) -- [Migrate from Shapefiles](migrate-from-shapefiles.html) -- [Migrate from GeoJSON](migrate-from-geojson.html) -- [Migrate from GeoPackage](migrate-from-geopackage.html) -- [Migrate from OpenStreetMap](migrate-from-openstreetmap.html) -- [Introducing Distributed Spatial Data in Free, Open Source CockroachDB](https://www.cockroachlabs.com/blog/spatial-data/) (blog post) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/st_union.md b/src/current/v21.1/st_union.md deleted file mode 100644 index c719de12ee8..00000000000 --- a/src/current/v21.1/st_union.md +++ /dev/null @@ -1,271 +0,0 @@ ---- -title: ST_Union -summary: ST_Union is an aggregate function that combines a set of shapes into a single shape. -toc: true ---- - -Given a set of shapes (e.g., from a [selection query](selection-queries.html)), `ST_Union` combines that set of shapes into a single shape. The resulting shape can then be passed to functions that operate on a single shape, such as [`ST_ConvexHull`](st_convexhull.html). - -`ST_Union` works on the following data types: - -- [`GEOMETRY`](spatial-glossary.html#geometry) - -{{site.data.alerts.callout_info}} -The non-aggregate version of `ST_Union` is not yet implemented. For more information, see [cockroach#49064](https://github.com/cockroachdb/cockroach/issues/49064). -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -Unlike `ST_Collect`, which does not change the shapes it operates on and merely gathers them into a collection, `ST_Union` modifies the shapes it operates on, merging them together. -{{site.data.alerts.end}} - -## Examples - -In this example, we will generate a single geometry from many individual points using `ST_Union`. - -1. Create a temporary table to hold all the points, which will be in [Well Known Text (WKT)](spatial-glossary.html#wkt) format: - - {% include_cached copy-clipboard.html %} - ~~~ sql - CREATE TABLE tmp (ID UUID DEFAULT gen_random_uuid(), geom_text STRING); - ~~~ - -2. Insert the points with the following statement: - - {% include_cached copy-clipboard.html %} - ~~~ sql - INSERT INTO tmp (geom_text) VALUES - ('POINT (-73.962090000000003 40.609226)'), - ('POINT (-74.007309000000006 40.619892)'), - ('POINT (-73.949724000000003 41.498103999999998)'), - ('POINT (-72.390808000000007 40.961891999999999)'), - ('POINT (-77.937476000000004 43.218451000000002)'), - ('POINT (-73.842978000000002 40.944997999999998)'), - ('POINT (-73.928291999999999 40.680906999999998)'), - ('POINT (-73.914484999999999 40.775554)'), - ('POINT (-73.993281999999994 40.722833000000001)'), - ('POINT (-73.900373999999999 41.691662000000001)'), - ('POINT (-73.787897999999998 42.250467)'), - ('POINT (-75.062138000000004 42.454003)'), - ('POINT (-73.992576 40.753692000000001)'), - ('POINT (-75.647559999999999 44.590586000000002)'), - ('POINT (-73.698974000000007 40.832500000000003)'), - ('POINT (-73.957140999999993 40.728864000000002)'), - ('POINT (-73.954773000000003 40.732667999999997)'), - ('POINT (-74.024456000000001 41.521116999999997)'), - ('POINT (-73.785302000000001 43.081800000000001)'), - ('POINT (-74.860436000000007 43.040708000000002)'), - ('POINT (-74.001165999999998 40.687367000000002)'), - ('POINT (-78.809790000000007 42.845362000000002)'), - ('POINT (-75.061370999999994 41.608192000000003)'), - ('POINT (-76.499137000000005 42.440623000000002)'), - ('POINT (-73.379208000000006 43.027712000000001)'), - ('POINT (-73.684466999999998 41.259498000000001)'), - ('POINT (-74.003300999999993 40.706865999999998)'), - ('POINT (-76.384369000000007 43.034204000000003)'), - ('POINT (-73.983408999999995 40.723255000000002)'), - ('POINT (-74.670794000000001 42.371471999999997)'), - ('POINT (-73.963970000000003 40.804766999999998)'), - ('POINT (-73.989228999999995 40.743865)'), - ('POINT (-73.487143000000003 42.587166000000003)'), - ('POINT (-78.608046999999999 42.767758999999998)'), - ('POINT (-78.824476000000004 42.754263999999999)'), - ('POINT (-72.185676999999998 40.963121999999998)'), - ('POINT (-78.639556999999996 43.001435000000001)'), - ('POINT (-77.428916999999998 43.212463)'), - ('POINT (-73.964771999999996 40.770071999999999)'), - ('POINT (-73.937073999999996 40.578499000000001)'), - ('POINT (-74.076237000000006 40.636749000000002)'), - ('POINT (-73.923480999999995 41.091296)'), - ('POINT (-78.00206 42.718395000000001)'), - ('POINT (-74.992647000000005 43.018610000000002)'), - ('POINT (-73.991900000000001 40.683962000000001)'), - ('POINT (-74.119761999999994 42.040447)'), - ('POINT (-73.955706000000006 40.785203000000003)'), - ('POINT (-73.698363000000001 42.709778)'), - ('POINT (-75.296575000000004 43.098322000000003)'), - ('POINT (-73.804747000000006 42.709560000000003)'), - ('POINT (-79.326158000000007 42.102457000000001)'), - ('POINT (-73.938518000000002 40.837735000000002)'), - ('POINT (-73.976410999999999 40.672561999999999)'), - ('POINT (-73.978993000000003 40.745123999999997)'), - ('POINT (-77.587306999999996 43.159300999999999)'), - ('POINT (-74.001251999999994 40.734977000000001)'), - ('POINT (-76.206723999999994 43.102012000000002)'), - ('POINT (-78.876722000000001 42.922820999999999)'), - ('POINT (-73.495369999999994 44.703318000000003)'), - ('POINT (-79.052965999999998 43.105055)'), - ('POINT (-73.551120999999995 41.189729)'), - ('POINT (-73.760520999999997 40.809649999999998)'), - ('POINT (-73.952360999999996 42.077744000000003)'), - ('POINT (-76.511525000000006 43.456606000000001)'), - ('POINT (-75.036815000000004 42.448793000000002)'), - ('POINT (-75.544407000000007 42.826815000000003)'), - ('POINT (-73.686974000000006 40.724902)'), - ('POINT (-77.582922999999994 43.154963000000002)'), - ('POINT (-78.836257000000003 42.960456999999998)'), - ('POINT (-73.919056999999995 41.091124999999998)'), - ('POINT (-73.998487999999995 40.733795999999998)'), - ('POINT (-73.939606999999995 40.815913000000002)'), - ('POINT (-74.058107000000007 41.083153000000003)'), - ('POINT (-73.991060000000004 40.733919)'), - ('POINT (-73.967905000000002 40.758032999999998)'), - ('POINT (-73.875281000000001 41.016382999999998)'), - ('POINT (-77.134634000000005 42.961303000000001)'), - ('POINT (-78.885418999999999 42.906629000000002)'), - ('POINT (-73.553145999999998 40.665826000000003)'), - ('POINT (-73.834867000000003 42.621578999999997)'), - ('POINT (-73.982307000000006 40.752575)'), - ('POINT (-77.816855000000004 42.795613000000003)'), - ('POINT (-73.962288000000001 40.671320999999999)'), - ('POINT (-72.354918999999995 41.083874999999999)'), - ('POINT (-73.965331000000006 40.806128999999999)'), - ('POINT (-74.080895999999996 41.844256000000001)'), - ('POINT (-73.942249000000004 40.812187000000002)'), - ('POINT (-73.974255999999997 40.783757999999999)'), - ('POINT (-73.990206999999998 40.754969000000003)'), - ('POINT (-73.601747000000003 41.562685000000002)'), - ('POINT (-74.117880999999997 42.040733000000003)'), - ('POINT (-73.979851999999994 40.664327999999998)'), - ('POINT (-73.960042999999999 40.657589000000002)'), - ('POINT (-74.009314000000003 40.715324000000003)'), - ('POINT (-73.903509 40.703226000000001)'), - ('POINT (-74.030330000000006 40.624713999999997)'), - ('POINT (-73.909406000000004 40.814979000000001)'), - ('POINT (-73.903160999999997 40.744929999999997)'), - ('POINT (-73.504622999999995 41.952860000000001)'), - ('POINT (-73.865554000000003 42.219304000000001)'), - ('POINT (-73.981960000000001 40.766441)'), - ('POINT (-73.913732999999993 40.698230000000002)'), - ('POINT (-73.596703000000005 40.874966000000001)'), - ('POINT (-73.974082999999993 40.686098000000001)'), - ('POINT (-73.996341999999999 40.725087000000002)'), - ('POINT (-73.595579999999998 42.363210000000002)'), - ('POINT (-73.952259999999995 40.674118999999997)'), - ('POINT (-73.993341000000001 40.730010999999998)'), - ('POINT (-73.989086999999998 40.721902)'), - ('POINT (-73.955855999999997 40.655028000000001)'), - ('POINT (-74.002930000000006 40.731848999999997)'), - ('POINT (-76.086753000000002 44.242874)'), - ('POINT (-73.839067 42.681877999999998)'), - ('POINT (-74.020105000000001 41.935105)'), - ('POINT (-73.980250999999996 44.279376999999997)'), - ('POINT (-74.085936000000004 41.747742000000002)'), - ('POINT (-77.227361999999999 43.063468)'), - ('POINT (-74.093547000000001 40.924697999999999)'), - ('POINT (-74.986999999999995 44.669899999999998)'), - ('POINT (-73.793597000000005 41.135866)'), - ('POINT (-78.889482000000001 42.924545000000002)'), - ('POINT (-73.914946 41.936036999999999)'), - ('POINT (-73.9071 41.721612999999998)'), - ('POINT (-73.993579999999994 40.736569000000003)'), - ('POINT (-73.900388000000007 41.691189000000001)'), - ('POINT (-73.989261999999997 40.663339000000001)'), - ('POINT (-73.918380999999997 41.090746000000003)'), - ('POINT (-73.862252999999995 40.839613)'), - ('POINT (-73.806933000000001 40.989787999999997)'), - ('POINT (-73.635006000000004 40.744909)'), - ('POINT (-73.794276999999994 41.008797000000001)'), - ('POINT (-73.759060000000005 41.022123000000001)'), - ('POINT (-73.711965000000006 40.968676000000002)'), - ('POINT (-74.668248000000006 42.372920000000001)'), - ('POINT (-73.992476999999994 40.739109999999997)'), - ('POINT (-74.131673000000006 44.326892000000001)'), - ('POINT (-73.974738000000002 40.732664)'), - ('POINT (-73.812349999999995 40.798867000000001)'), - ('POINT (-78.641949999999994 42.991985)'), - ('POINT (-73.962226999999999 40.778944000000003)'), - ('POINT (-74.374174999999994 43.006495000000001)'), - ('POINT (-77.095714000000001 43.041158000000003)'), - ('POINT (-73.425764000000001 40.871898000000002)'), - ('POINT (-73.466729000000001 44.708036)'), - ('POINT (-73.978020999999998 40.686546999999997)'), - ('POINT (-73.974082999999993 40.686098000000001)'), - ('POINT (-73.971849000000006 40.602485000000001)'), - ('POINT (-73.983153999999999 40.690553000000001)'), - ('POINT (-77.052417000000005 42.143089000000003)'), - ('POINT (-73.776431000000002 40.934483)'), - ('POINT (-74.002019000000004 40.730927999999999)'), - ('POINT (-73.772469999999998 42.264814000000001)'), - ('POINT (-73.772876999999994 41.158641000000003)'), - ('POINT (-72.296062000000006 41.002409)'), - ('POINT (-74.021124999999998 41.933720000000001)'), - ('POINT (-74.058107000000007 41.083153000000003)'), - ('POINT (-73.967922999999999 40.801301000000002)'), - ('POINT (-72.294278000000006 41.000610999999999)'), - ('POINT (-76.499340000000004 42.438741999999998)'), - ('POINT (-73.976597999999996 40.786610000000003)'), - ('POINT (-73.988552999999996 40.703600000000002)'), - ('POINT (-74.004855000000006 40.741959000000001)'), - ('POINT (-73.844444999999993 40.925654999999999)'), - ('POINT (-74.005101999999994 40.746654999999997)'), - ('POINT (-73.937281999999996 42.819218999999997)'), - ('POINT (-74.001677000000001 40.736243000000002)'), - ('POINT (-73.684112999999996 40.981757000000002)'), - ('POINT (-73.919064000000006 40.832836999999998)'), - ('POINT (-78.885268999999994 42.944516)'), - ('POINT (-72.299519000000004 40.994070999999998)'), - ('POINT (-73.833477999999999 40.940247999999997)'), - ('POINT (-73.695717000000002 41.783996999999999)'), - ('POINT (-73.926891999999995 40.807422000000003)'), - ('POINT (-74.085899999999995 41.747892999999998)'), - ('POINT (-73.957583 41.417974999999998)'); - ~~~ - -3. Run the query below, which gathers the points into a single geometry using `ST_Union`, and converts the geometry to [GeoJSON](spatial-glossary.html#geojson) so that we can view it with [geojson.io](http://geojson.io): - - {% include_cached copy-clipboard.html %} - ~~~ sql - WITH - the_geoms_table - AS ( - SELECT - st_union(st_geomfromtext(geom_text)) AS the_union - FROM - tmp - ) - SELECT - st_asgeojson(the_union) - FROM - the_geoms_table; - ~~~ - - ~~~ json - {"type":"MultiPoint","coordinates":[[-79.326158,42.102457],[-79.052966,43.105055],[-78.889482,42.924545],[-78.885419,42.906629],[-78.885269,42.944516],[-78.876722,42.922821],[-78.836257,42.960457],[-78.824476,42.754264],[-78.80979,42.845362],[-78.64195,42.991985],[-78.639557,43.001435],[-78.608047,42.767759],[-78.00206,42.718395],[-77.937476,43.218451],[-77.816855,42.795613],[-77.587307,43.159301],[-77.582923,43.154963],[-77.428917,43.212463],[-77.227362,43.063468],[-77.134634,42.961303],[-77.095714,43.041158],[-77.052417,42.143089],[-76.511525,43.456606],[-76.49934,42.438742],[-76.499137,42.440623],[-76.384369,43.034204],[-76.206724,43.102012],[-76.086753,44.242874],[-75.64756,44.590586],[-75.544407,42.826815],[-75.296575,43.098322],[-75.062138,42.454003],[-75.061371,41.608192],[-75.036815,42.448793],[-74.992647,43.01861],[-74.987,44.6699],[-74.860436,43.040708],[-74.670794,42.371472],[-74.668248,42.37292],[-74.374175,43.006495],[-74.131673,44.326892],[-74.119762,42.040447],[-74.117881,42.040733],[-74.093547,40.924698],[-74.085936,41.747742],[-74.0859,41.747893],[-74.080896,41.844256],[-74.076237,40.636749],[-74.058107,41.083153],[-74.03033,40.624714],[-74.024456,41.521117],[-74.021125,41.93372],[-74.020105,41.935105],[-74.009314,40.715324],[-74.007309,40.619892],[-74.005102,40.746655],[-74.004855,40.741959],[-74.003301,40.706866],[-74.00293,40.731849],[-74.002019,40.730928],[-74.001677,40.736243],[-74.001252,40.734977],[-74.001166,40.687367],[-73.998488,40.733796],[-73.996342,40.725087],[-73.99358,40.736569],[-73.993341,40.730011],[-73.993282,40.722833],[-73.992576,40.753692],[-73.992477,40.73911],[-73.9919,40.683962],[-73.99106,40.733919],[-73.990207,40.754969],[-73.989262,40.663339],[-73.989229,40.743865],[-73.989087,40.721902],[-73.988553,40.7036],[-73.983409,40.723255],[-73.983154,40.690553],[-73.982307,40.752575],[-73.98196,40.766441],[-73.980251,44.279377],[-73.979852,40.664328],[-73.978993,40.745124],[-73.978021,40.686547],[-73.976598,40.78661],[-73.976411,40.672562],[-73.974738,40.732664],[-73.974256,40.783758],[-73.974083,40.686098],[-73.971849,40.602485],[-73.967923,40.801301],[-73.967905,40.758033],[-73.965331,40.806129],[-73.964772,40.770072],[-73.96397,40.804767],[-73.962288,40.671321],[-73.962227,40.778944],[-73.96209,40.609226],[-73.960043,40.657589],[-73.957583,41.417975],[-73.957141,40.728864],[-73.955856,40.655028],[-73.955706,40.785203],[-73.954773,40.732668],[-73.952361,42.077744],[-73.95226,40.674119],[-73.949724,41.498104],[-73.942249,40.812187],[-73.939607,40.815913],[-73.938518,40.837735],[-73.937282,42.819219],[-73.937074,40.578499],[-73.928292,40.680907],[-73.926892,40.807422],[-73.923481,41.091296],[-73.919064,40.832837],[-73.919057,41.091125],[-73.918381,41.090746],[-73.914946,41.936037],[-73.914485,40.775554],[-73.913733,40.69823],[-73.909406,40.814979],[-73.9071,41.721613],[-73.903509,40.703226],[-73.903161,40.74493],[-73.900388,41.691189],[-73.900374,41.691662],[-73.875281,41.016383],[-73.865554,42.219304],[-73.862253,40.839613],[-73.844445,40.925655],[-73.842978,40.944998],[-73.839067,42.681878],[-73.834867,42.621579],[-73.833478,40.940248],[-73.81235,40.798867],[-73.806933,40.989788],[-73.804747,42.70956],[-73.794277,41.008797],[-73.793597,41.135866],[-73.787898,42.250467],[-73.785302,43.0818],[-73.776431,40.934483],[-73.772877,41.158641],[-73.77247,42.264814],[-73.760521,40.80965],[-73.75906,41.022123],[-73.711965,40.968676],[-73.698974,40.8325],[-73.698363,42.709778],[-73.695717,41.783997],[-73.686974,40.724902],[-73.684467,41.259498],[-73.684113,40.981757],[-73.635006,40.744909],[-73.601747,41.562685],[-73.596703,40.874966],[-73.59558,42.36321],[-73.553146,40.665826],[-73.551121,41.189729],[-73.504623,41.95286],[-73.49537,44.703318],[-73.487143,42.587166],[-73.466729,44.708036],[-73.425764,40.871898],[-73.379208,43.027712],[-72.390808,40.961892],[-72.354919,41.083875],[-72.299519,40.994071],[-72.296062,41.002409],[-72.294278,41.000611],[-72.185677,40.963122]]} - ~~~ - -4. Paste the JSON emitted in the previous step into [geojson.io](http://geojson.io) and you should see an image like the following, which shows the location of [most of the independent bookstores in New York State](https://www.bookweb.org/member_directory/search/ABAmember/results/0/0/ny/0): - - ST_Union example - -5. Finally, drop the temporary table if you no longer need it: - - {% include_cached copy-clipboard.html %} - ~~~ sql - DROP TABLE tmp; - ~~~ - -## See also - -- [Working with Spatial Data](spatial-data.html) -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial and GIS Glossary of Terms](spatial-glossary.html) -- [Spatial indexes](spatial-indexes.html) -- [Spatial functions](functions-and-operators.html#spatial-functions) -- [`ST_Covers`](st_covers.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Contains`](st_contains.html) -- [`ST_Within`](st_within.html) -- [`ST_Intersects`](st_intersects.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Covers`](st_covers.html) -- [`ST_Disjoint`](st_disjoint.html) -- [`ST_Equals`](st_equals.html) -- [`ST_Overlaps`](st_overlaps.html) -- [`ST_Touches`](st_touches.html) -- [`ST_ConvexHull`](st_convexhull.html) -- [Migrate from Shapefiles](migrate-from-shapefiles.html) -- [Migrate from GeoJSON](migrate-from-geojson.html) -- [Migrate from GeoPackage](migrate-from-geopackage.html) -- [Migrate from OpenStreetMap](migrate-from-openstreetmap.html) -- [Introducing Distributed Spatial Data in Free, Open Source CockroachDB](https://www.cockroachlabs.com/blog/spatial-data/) (blog post) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/st_within.md b/src/current/v21.1/st_within.md deleted file mode 100644 index 23c5fd4a5aa..00000000000 --- a/src/current/v21.1/st_within.md +++ /dev/null @@ -1,101 +0,0 @@ ---- -title: ST_Within -summary: ST_Within(A, B) returns true if no point in shape A lies outside of shape B, and at least one point in the interior of A lies in the interior of B. -toc: true -has_prefixed_variant: true ---- - -Given two shapes _A_ and _B_, the predicate function `ST_Within(A, B)` returns `true` if the following criteria are met: - -- No point in _A_ lies outside of _B_. -- At least one point in the interior of _A_ lies in the interior of _B_. - -In other words, the exterior of shape _B_ must not include any point in _A_, and one or more points of _A_'s interior must lie to the interior of _B_. - -This behavior is similar to [`ST_CoveredBy`](st_coveredby.html), except that the criteria are more exacting, and therefore some pairs of shapes will be rejected by this function that would be accepted by `ST_CoveredBy`. - -`ST_Within` works on the following spatial data types: - -- [`GEOMETRY`](spatial-glossary.html#geometry) - -{% if page.has_prefixed_variant %} -{{site.data.alerts.callout_info}} -`{{page.title}}` will attempt to use any available [spatial index](spatial-indexes.html) to speed up its operation. Use the prefixed variant `_{{page.title}}` if you do not want any spatial indexes to be used. -{{site.data.alerts.end}} -{% endif %} - -{{site.data.alerts.callout_info}} -This function is the inverse of [`ST_Contains`](st_contains.html). -{{site.data.alerts.end}} - -## Examples - -{% include {{page.version.version}}/misc/geojson_geometry_note.md %} - -### True - -In this example, `{{page.title}}` returns `true` because: - -- No point in Polygon _A_ lies outside of Polygon _B_. -- At least one point in the interior of Polygon _A_ lies in the interior of Polygon _B_. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT ST_Within(st_geomfromtext('SRID=4326;POLYGON((-87.623177 41.881832, -90.199402 38.627003, -82.446732 38.413651, -87.623177 41.881832))'), st_geomfromtext('SRID=4326;POLYGON((-87.906471 43.038902, -95.992775 36.153980, -75.704722 36.076944, -87.906471 43.038902))')); -~~~ - -~~~ - st_within ---------------- - true - -(1 row) -~~~ - -ST_Within - true - -### False - -In this example, `{{page.title}}` returns `false` because: - -- All points in Polygon _A_ lie outside of Polygon _B_. - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT ST_Within(st_geomfromtext('SRID=4326;POLYGON((-87.906471 43.038902, -95.992775 36.153980, -75.704722 36.076944, -87.906471 43.038902), (-87.623177 41.881832, -90.199402 38.627003, -82.446732 38.413651, -87.623177 41.881832))'), st_geomfromtext('SRID=4326;POLYGON((-87.356934 41.595161, -84.512016 39.103119, -86.529167 39.162222, -87.356934 41.595161))')); -~~~ - -~~~ - st_within ---------------- - false -(1 row) -~~~ - -ST_Within - false - -## See also - -- [Working with Spatial Data](spatial-data.html) -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial and GIS Glossary of Terms](spatial-glossary.html) -- [Spatial indexes](spatial-indexes.html) -- [Spatial functions](functions-and-operators.html#spatial-functions) -- [`ST_Covers`](st_covers.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Contains`](st_contains.html) -- [`ST_Intersects`](st_intersects.html) -- [`ST_CoveredBy`](st_coveredby.html) -- [`ST_Covers`](st_covers.html) -- [`ST_Disjoint`](st_disjoint.html) -- [`ST_Equals`](st_equals.html) -- [`ST_Overlaps`](st_overlaps.html) -- [`ST_Touches`](st_touches.html) -- [`ST_ConvexHull`](st_convexhull.html) -- [`ST_Union`](st_union.html) -- [Migrate from Shapefiles](migrate-from-shapefiles.html) -- [Migrate from GeoJSON](migrate-from-geojson.html) -- [Migrate from GeoPackage](migrate-from-geopackage.html) -- [Migrate from OpenStreetMap](migrate-from-openstreetmap.html) -- [Introducing Distributed Spatial Data in Free, Open Source CockroachDB](https://www.cockroachlabs.com/blog/spatial-data/) (blog post) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/start-a-local-cluster-in-docker-linux.md b/src/current/v21.1/start-a-local-cluster-in-docker-linux.md deleted file mode 100644 index f59013e817d..00000000000 --- a/src/current/v21.1/start-a-local-cluster-in-docker-linux.md +++ /dev/null @@ -1,32 +0,0 @@ ---- -title: Start a Cluster in Docker (Insecure) -summary: Run an insecure multi-node CockroachDB cluster across multiple Docker containers on a single host. -toc: true ---- - -
    - - - -
    - -Once you've [installed the official CockroachDB Docker image](install-cockroachdb.html), it's simple to run an insecure multi-node cluster across multiple Docker containers on a single host, using Docker volumes to persist node data. - -{% include cockroachcloud/use-cockroachcloud-instead.md %} - -{% include {{ page.version.version }}/prod-deployment/insecure-flag.md %} - -## Before you begin - -- Make sure you have already [installed the official CockroachDB Docker image](install-cockroachdb.html). -- For quick SQL testing or app development, consider [running a single-node cluster](cockroach-start-single-node.html) instead. -- Note that running multiple nodes on a single host is useful for testing CockroachDB, but it's not suitable for production. To run a physically distributed cluster in containers, use an orchestration tool like Kubernetes. See [Orchestration](orchestration.html) for more details, and review the [Production Checklist](recommended-production-settings.html). - -{% include {{ page.version.version }}/start-in-docker/mac-linux-steps.md %} - -## What's next? - -- Learn more about [CockroachDB SQL](learn-cockroachdb-sql.html) and the [built-in SQL client](cockroach-sql.html) -- [Install the client driver](install-client-drivers.html) for your preferred language -- [Build an app with CockroachDB](example-apps.html) -- Further explore CockroachDB capabilities like [fault tolerance and automated repair](demo-fault-tolerance-and-recovery.html), [multi-region performance](demo-low-latency-multi-region-deployment.html), [serializable transactions](demo-serializable.html), and [JSON support](demo-json-support.html) diff --git a/src/current/v21.1/start-a-local-cluster-in-docker-mac.md b/src/current/v21.1/start-a-local-cluster-in-docker-mac.md deleted file mode 100644 index 364380f7740..00000000000 --- a/src/current/v21.1/start-a-local-cluster-in-docker-mac.md +++ /dev/null @@ -1,30 +0,0 @@ ---- -title: Start a Cluster in Docker (Insecure) -summary: Run an insecure multi-node CockroachDB cluster across multiple Docker containers on a single host. -toc: true ---- - -
    - - - -
    - -Once you've [installed the official CockroachDB Docker image](install-cockroachdb.html), it's simple to run an insecure multi-node cluster across multiple Docker containers on a single host, using Docker volumes to persist node data. - -{% include cockroachcloud/use-cockroachcloud-instead.md %} - -## Before you begin - -- Make sure you have already [installed the official CockroachDB Docker image](install-cockroachdb.html). -- For quick SQL testing or app development, consider [running a single-node cluster](cockroach-start-single-node.html) instead. -- Note that running multiple nodes on a single host is useful for testing CockroachDB, but it's not suitable for production. To run a physically distributed cluster in containers, use an orchestration tool like Kubernetes. See [Orchestration](orchestration.html) for more details, and review the [Production Checklist](recommended-production-settings.html). - -{% include {{ page.version.version }}/start-in-docker/mac-linux-steps.md %} - -## What's next? - -- Learn more about [CockroachDB SQL](learn-cockroachdb-sql.html) and the [built-in SQL client](cockroach-sql.html) -- [Install the client driver](install-client-drivers.html) for your preferred language -- [Build an app with CockroachDB](example-apps.html) -- Further explore CockroachDB capabilities like [fault tolerance and automated repair](demo-fault-tolerance-and-recovery.html), [multi-region performance](demo-low-latency-multi-region-deployment.html), [serializable transactions](demo-serializable.html), and [JSON support](demo-json-support.html) diff --git a/src/current/v21.1/start-a-local-cluster-in-docker-windows.md b/src/current/v21.1/start-a-local-cluster-in-docker-windows.md deleted file mode 100644 index 80107bdcc92..00000000000 --- a/src/current/v21.1/start-a-local-cluster-in-docker-windows.md +++ /dev/null @@ -1,238 +0,0 @@ ---- -title: Start a Cluster in Docker (Insecure) -summary: Run an insecure multi-node CockroachDB cluster across multiple Docker containers on a single host. -toc: true ---- - -
    - - - -
    - -Once you've [installed the official CockroachDB Docker image](install-cockroachdb.html), it's simple to run an insecure multi-node cluster across multiple Docker containers on a single host, using Docker volumes to persist node data. - -{% include cockroachcloud/use-cockroachcloud-instead.md %} - -{% include {{ page.version.version }}/prod-deployment/insecure-flag.md %} - -## Before you begin - -- Make sure you have already [installed the official CockroachDB Docker image](install-cockroachdb.html). -- For quick SQL testing or app development, consider [running a single-node cluster](cockroach-start-single-node.html) instead. -- Note that running multiple nodes on a single host is useful for testing CockroachDB, but it's not suitable for production. To run a physically distributed cluster in containers, use an orchestration tool like Kubernetes. See [Orchestration](orchestration.html) for more details, and review the [Production Checklist](recommended-production-settings.html). - -## Step 1. Create a bridge network - -Since you'll be running multiple Docker containers on a single host, with one CockroachDB node per container, you need to create what Docker refers to as a [bridge network](https://docs.docker.com/engine/userguide/networking/#/a-bridge-network). The bridge network will enable the containers to communicate as a single cluster while keeping them isolated from external networks. - -~~~ powershell -PS C:\Users\username> docker network create -d bridge roachnet -~~~ - -We've used `roachnet` as the network name here and in subsequent steps, but feel free to give your network any name you like. - -## Step 2. Start the cluster - -1. Start the first node: - - {{site.data.alerts.callout_info}}Be sure to replace <username> in the -v flag with your actual username.{{site.data.alerts.end}} - - ~~~ powershell - PS C:\Users\username> docker run -d ` - --name=roach1 ` - --hostname=roach1 ` - --net=roachnet ` - -p 26257:26257 -p 8080:8080 ` - -v "//c/Users//cockroach-data/roach1:/cockroach/cockroach-data" ` - {{page.release_info.docker_image}}:{{page.release_info.version}} start ` - --insecure ` - --join=roach1,roach2,roach3 - ~~~ - -2. This command creates a container and starts the first CockroachDB node inside it. Take a moment to understand each part: - - `docker run`: The Docker command to start a new container. - - `-d`: This flag runs the container in the background so you can continue the next steps in the same shell. - - `--name`: The name for the container. This is optional, but a custom name makes it significantly easier to reference the container in other commands, for example, when opening a Bash session in the container or stopping the container. - - `--hostname`: The hostname for the container. You will use this to join other containers/nodes to the cluster. - - `--net`: The bridge network for the container to join. See step 1 for more details. - - `-p 26257:26257 -p 8080:8080`: These flags map the default port for inter-node and client-node communication (`26257`) and the default port for HTTP requests to the DB Console (`8080`) from the container to the host. This enables inter-container communication and makes it possible to call up the DB Console from a browser. - - `-v "//c/Users//cockroach-data/roach1:/cockroach/cockroach-data"`: This flag mounts a host directory as a data volume. This means that data and logs for this node will be stored in `Users//cockroach-data/roach1` on the host and will persist after the container is stopped or deleted. For more details, see Docker's Bind Mounts topic. - - `{{page.release_info.docker_image}}:{{page.release_info.version}} start --insecure --join`: The CockroachDB command to [start a node](cockroach-start.html) in the container in insecure mode. The `--join` flag specifies the `hostname` of each node that will initially comprise your cluster. Otherwise, all [`cockroach start`](cockroach-start.html) defaults are accepted. Note that since each node is in a unique container, using identical default ports won’t cause conflicts. - -3. Start two more nodes: - - {{site.data.alerts.callout_info}}Again, be sure to replace <username> in the -v flag with your actual username.{{site.data.alerts.end}} - - ~~~ powershell - PS C:\Users\username> docker run -d ` - --name=roach2 ` - --hostname=roach2 ` - --net=roachnet ` - -v "//c/Users//cockroach-data/roach2:/cockroach/cockroach-data" ` - {{page.release_info.docker_image}}:{{page.release_info.version}} start ` - --insecure ` - --join=roach1,roach2,roach3 - ~~~ - - ~~~ powershell - PS C:\Users\username> docker run -d ` - --name=roach3 ` - --hostname=roach3 ` - --net=roachnet ` - -v "//c/Users//cockroach-data/roach3:/cockroach/cockroach-data" ` - {{page.release_info.docker_image}}:{{page.release_info.version}} start ` - --insecure ` - --join=roach1,roach2,roach3 - ~~~ - -4. Perform a one-time initialization of the cluster: - - ~~~ powershell - PS C:\Users\username> docker exec -it roach1 ./cockroach init --insecure - ~~~ - - You'll see the following message: - - ~~~ - Cluster successfully initialized - ~~~ - -## Step 3. Use the built-in SQL client - -Now that your cluster is live, you can use any node as a SQL gateway. To test this out, let's use the `docker exec` command to start the [built-in SQL shell](cockroach-sql.html) in the first container. - -1. Start the SQL shell in the first container: - - ~~~ powershell - PS C:\Users\username> docker exec -it roach1 ./cockroach sql --insecure - ~~~ - -2. Run some basic [CockroachDB SQL statements](learn-cockroachdb-sql.html): - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE DATABASE bank; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE TABLE bank.accounts (id INT PRIMARY KEY, balance DECIMAL); - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > INSERT INTO bank.accounts VALUES (1, 1000.50); - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SELECT * FROM bank.accounts; - ~~~ - - ~~~ - id | balance - +----+---------+ - 1 | 1000.50 - (1 row) - ~~~ - -3. Now exit the SQL shell on node 1 and open a new shell on node 2: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > \q - ~~~ - - ~~~ powershell - PS C:\Users\username> docker exec -it roach2 ./cockroach sql --insecure - ~~~ - -4. Run the same `SELECT` query as before: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SELECT * FROM bank.accounts; - ~~~ - - ~~~ - id | balance - +----+---------+ - 1 | 1000.50 - (1 row) - ~~~ - - As you can see, node 1 and node 2 behaved identically as SQL gateways. - -5. Exit the SQL shell on node 2: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > \q - ~~~ - -## Step 4. Run a sample workload - -CockroachDB also comes with a number of [built-in workloads](cockroach-workload.html) for simulating client traffic. Let's run the workload based on CockroachDB's sample vehicle-sharing application, [MovR](movr.html). - -1. Load the initial dataset: - - ~~~ powershell - PS C:\Users\username> docker exec -it roach1 ./cockroach workload init movr \ - 'postgresql://root@roach1:26257?sslmode=disable' - ~~~ - -2. Run the workload for 5 minutes: - - ~~~ powershell - PS C:\Users\username> docker exec -it roach1 ./cockroach workload run movr \ - --duration=5m \ - 'postgresql://root@roach1:26257?sslmode=disable' - ~~~ - -## Step 5. Access the DB Console - -The CockroachDB [DB Console](ui-overview.html) gives you insight into the overall health of your cluster as well as the performance of the client workload. - -1. When you started the first container/node, you mapped the node's default HTTP port `8080` to port `8080` on the host, so go to http://localhost:8080. - -2. On the [**Cluster Overview**](ui-cluster-overview-page.html), notice that three nodes are live, with an identical replica count on each node: - - DB Console - - This demonstrates CockroachDB's [automated replication](demo-replication-and-rebalancing.html) of data via the Raft consensus protocol. - - {{site.data.alerts.callout_info}} - Capacity metrics can be incorrect when running multiple nodes on a single machine. For more details, see this [limitation](known-limitations.html#available-capacity-metric-in-the-db-console). - {{site.data.alerts.end}} - -3. Click [**Metrics**](ui-overview-dashboard.html) to access a variety of time series dashboards, including graphs of SQL queries and service latency over time: - - DB Console - -4. Use the [**Databases**](ui-databases-page.html), [**Statements**](ui-statements-page.html), and [**Jobs**](ui-jobs-page.html) pages to view details about your databases and tables, to assess the performance of specific queries, and to monitor the status of long-running operations like schema changes, respectively. - -## Step 6. Stop the cluster - -Use the `docker stop` and `docker rm` commands to stop and remove the containers (and therefore the cluster): - -~~~ powershell -PS C:\Users\username> docker stop roach1 roach2 roach3 -~~~ - -~~~ powershell -PS C:\Users\username> docker rm roach1 roach2 roach3 -~~~ - -If you do not plan to restart the cluster, you may want to remove the nodes' data stores: - -~~~ powershell -PS C:\Users\username> Remove-Item cockroach-data -recurse -~~~ - -## What's next? - -- Learn more about [CockroachDB SQL](learn-cockroachdb-sql.html) and the [built-in SQL client](cockroach-sql.html) -- [Install the client driver](install-client-drivers.html) for your preferred language -- [Build an app with CockroachDB](example-apps.html) -- Further explore CockroachDB capabilities like [fault tolerance and automated repair](demo-fault-tolerance-and-recovery.html), [multi-region performance](demo-low-latency-multi-region-deployment.html), [serializable transactions](demo-serializable.html), and [JSON support](demo-json-support.html) diff --git a/src/current/v21.1/start-a-local-cluster.md b/src/current/v21.1/start-a-local-cluster.md deleted file mode 100644 index 468f8780f8e..00000000000 --- a/src/current/v21.1/start-a-local-cluster.md +++ /dev/null @@ -1,372 +0,0 @@ ---- -title: Start a Local Cluster (Insecure) -summary: Run an insecure multi-node CockroachDB cluster locally with each node listening on a different port. -toc: true -toc_not_nested: true ---- - -
    - - -
    - -Once you've [installed CockroachDB](install-cockroachdb.html), it's simple to run an insecure multi-node cluster locally. - -{% include cockroachcloud/use-cockroachcloud-instead.md %} - -## Before you begin - -- Make sure you have already [installed CockroachDB](install-cockroachdb.html). -- For quick SQL testing or app development, consider [running a single-node cluster](cockroach-start-single-node.html) instead. -- Note that running multiple nodes on a single host is useful for testing CockroachDB, but it's not suitable for production. To run a physically distributed cluster, see [Manual Deployment](manual-deployment.html) or [Orchestrated Deployment](orchestration.html), and review the [Production Checklist](recommended-production-settings.html). - -## Step 1. Start the cluster - -1. Use the [`cockroach start`](cockroach-start.html) command to start the first node: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --insecure \ - --store=node1 \ - --listen-addr=localhost:26257 \ - --http-addr=localhost:8080 \ - --join=localhost:26257,localhost:26258,localhost:26259 \ - --background - ~~~ - - You'll see a message like the following: - - ~~~ - * - * WARNING: RUNNING IN INSECURE MODE! - * - * - Your cluster is open for any client that can access localhost. - * - Any user, even root, can log in without providing a password. - * - Any user, connecting as root, can read or write any data in your cluster. - * - There is no network encryption nor authentication, and thus no confidentiality. - * - * Check out how to secure your cluster: https://www.cockroachlabs.com/docs/v19.2/secure-a-cluster.html - * - * - * INFO: initial startup completed. - * Node will now attempt to join a running cluster, or wait for `cockroach init`. - * Client connections will be accepted after this completes successfully. - * Check the log file(s) for progress. - * - ~~~ - -2. Take a moment to understand the [flags](cockroach-start.html#flags) you used: - - The `--insecure` flag makes communication unencrypted. - - Since this is a purely local cluster, `--listen-addr=localhost:26257` and `--http-addr=localhost:8080` tell the node to listen only on `localhost`, with port `26257` used for internal and client traffic and port `8080` used for HTTP requests from the DB Console. - - The `--store` flag indicates the location where the node's data and logs are stored. - - The `--join` flag specifies the addresses and ports of the nodes that will initially comprise your cluster. You'll use this exact `--join` flag when starting other nodes as well. - - {% include {{ page.version.version }}/prod-deployment/join-flag-single-region.md %} - - The `--background` flag starts the `cockroach` process in the background so you can continue using the same terminal for other operations. - -3. Start two more nodes: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --insecure \ - --store=node2 \ - --listen-addr=localhost:26258 \ - --http-addr=localhost:8081 \ - --join=localhost:26257,localhost:26258,localhost:26259 \ - --background - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --insecure \ - --store=node3 \ - --listen-addr=localhost:26259 \ - --http-addr=localhost:8082 \ - --join=localhost:26257,localhost:26258,localhost:26259 \ - --background - ~~~ - - These commands are the same as before but with unique `--store`, `--listen-addr`, and `--http-addr` flags. - -4. Use the [`cockroach init`](cockroach-init.html) command to perform a one-time initialization of the cluster, sending the request to any node on the `--join` list: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach init --insecure --host=localhost:26257 - ~~~ - - You'll see the following message: - - ~~~ - Cluster successfully initialized - ~~~ - - At this point, each node also prints helpful [startup details](cockroach-start.html#standard-output) to its log. For example, the following command retrieves node 1's startup details: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ grep 'node starting' node1/logs/cockroach.log -A 11 - ~~~ - - The output will look something like this: - - ~~~ - CockroachDB node starting at {{ now | date: "%Y-%m-%d %H:%M:%S.%6 +0000 UTC" }} - build: CCL {{page.release_info.version}} @ {{page.release_info.build_time}} (go1.12.6) - webui: http://localhost:8080 - sql: postgresql://root@localhost:26257?sslmode=disable - RPC client flags: cockroach --host=localhost:26257 --insecure - logs: /Users//node1/logs - temp dir: /Users//node1/cockroach-temp242232154 - external I/O path: /Users//node1/extern - store[0]: path=/Users//node1 - status: initialized new cluster - clusterID: 8a681a16-9623-4fc1-a537-77e9255daafd - nodeID: 1 - ~~~ - -## Step 2. Use the built-in SQL client - -Now that your cluster is live, you can use any node as a SQL gateway. To test this out, let's use CockroachDB's built-in SQL client. - -1. Run the [cockroach sql](cockroach-sql.html) command against node 1: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --insecure --host=localhost:26257 - ~~~ - -2. Run some basic [CockroachDB SQL statements](learn-cockroachdb-sql.html): - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE DATABASE bank; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE TABLE bank.accounts (id INT PRIMARY KEY, balance DECIMAL); - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > INSERT INTO bank.accounts VALUES (1, 1000.50); - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SELECT * FROM bank.accounts; - ~~~ - - ~~~ - id | balance - +----+---------+ - 1 | 1000.50 - (1 row) - ~~~ - -3. Now exit the SQL shell on node 1 and open a new shell on node 2: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > \q - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --insecure --host=localhost:26258 - ~~~ - - {{site.data.alerts.callout_info}} - In a real deployment, all nodes would likely use the default port `26257`, and so you wouldn't need to set the port portion of `--host`. - {{site.data.alerts.end}} - -4. Run the same `SELECT` query as before: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SELECT * FROM bank.accounts; - ~~~ - - ~~~ - id | balance - +----+---------+ - 1 | 1000.50 - (1 row) - ~~~ - - As you can see, node 1 and node 2 behaved identically as SQL gateways. - -5. Exit the SQL shell on node 2: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > \q - ~~~ - -## Step 3. Run a sample workload - -CockroachDB also comes with a number of [built-in workloads](cockroach-workload.html) for simulating client traffic. Let's run the workload based on CockroachDB's sample vehicle-sharing application, [MovR](movr.html). - -1. Load the initial dataset: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload init movr \ - 'postgresql://root@localhost:26257?sslmode=disable' - ~~~ - - ~~~ - I190926 16:50:35.663708 1 workload/workloadsql/dataload.go:135 imported users (0s, 50 rows) - I190926 16:50:35.682583 1 workload/workloadsql/dataload.go:135 imported vehicles (0s, 15 rows) - I190926 16:50:35.769572 1 workload/workloadsql/dataload.go:135 imported rides (0s, 500 rows) - I190926 16:50:35.836619 1 workload/workloadsql/dataload.go:135 imported vehicle_location_histories (0s, 1000 rows) - I190926 16:50:35.915498 1 workload/workloadsql/dataload.go:135 imported promo_codes (0s, 1000 rows) - ~~~ - -2. Run the workload for 5 minutes: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach workload run movr \ - --duration=5m \ - 'postgresql://root@localhost:26257?sslmode=disable' - ~~~ - -## Step 4. Access the DB Console - -The CockroachDB [DB Console](ui-overview.html) gives you insight into the overall health of your cluster as well as the performance of the client workload. - -1. Go to http://localhost:8080. - -2. On the [**Cluster Overview**](ui-cluster-overview-page.html), notice that three nodes are live, with an identical replica count on each node: - - DB Console - - This demonstrates CockroachDB's [automated replication](demo-replication-and-rebalancing.html) of data via the Raft consensus protocol. - - {{site.data.alerts.callout_info}} - Capacity metrics can be incorrect when running multiple nodes on a single machine. For more details, see this [limitation](known-limitations.html#available-capacity-metric-in-the-db-console). - {{site.data.alerts.end}} - -3. Click [**Metrics**](ui-overview-dashboard.html) to access a variety of time series dashboards, including graphs of SQL queries and service latency over time: - - DB Console - -4. Use the [**Databases**](ui-databases-page.html), [**Statements**](ui-statements-page.html), and [**Jobs**](ui-jobs-page.html) pages to view details about your databases and tables, to assess the performance of specific queries, and to monitor the status of long-running operations like schema changes, respectively. - -## Step 5. Simulate node failure - -1. In a new terminal, run the [`cockroach quit`](cockroach-quit.html) command against a node to simulate a node failure: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach quit --insecure --host=localhost:26259 - ~~~ - -2. Back in the DB Console, despite one node being "suspect", notice the continued SQL traffic: - - DB Console - - This demonstrates CockroachDB's use of the Raft consensus protocol to [maintain availability and consistency in the face of failure](demo-fault-tolerance-and-recovery.html); as long as a majority of replicas remain online, the cluster and client traffic continue uninterrupted. - -4. Restart node 3: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --insecure \ - --store=node3 \ - --listen-addr=localhost:26259 \ - --http-addr=localhost:8082 \ - --join=localhost:26257,localhost:26258,localhost:26259 \ - --background - ~~~ - -## Step 6. Scale the cluster - -Adding capacity is as simple as starting more nodes with `cockroach start`. - -1. Start 2 more nodes: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --insecure \ - --store=node4 \ - --listen-addr=localhost:26260 \ - --http-addr=localhost:8083 \ - --join=localhost:26257,localhost:26258,localhost:26259 \ - --background - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --insecure \ - --store=node5 \ - --listen-addr=localhost:26261 \ - --http-addr=localhost:8084 \ - --join=localhost:26257,localhost:26258,localhost:26259 \ - --background - ~~~ - - Again, these commands are the same as before but with unique `--store`, `--listen-addr`, and `--http-addr` flags. - -2. Back on the **Cluster Overview** in the DB Console, you'll now see 5 nodes listed: - - DB Console - - At first, the replica count will be lower for nodes 4 and 5. Very soon, however, you'll see those numbers even out across all nodes, indicating that data is being [automatically rebalanced](demo-replication-and-rebalancing.html) to utilize the additional capacity of the new nodes. - -## Step 7. Stop the cluster - -1. When you're done with your test cluster, use the [`cockroach quit`](cockroach-quit.html) command to gracefully shut down each node. - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach quit --insecure --host=localhost:26257 - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach quit --insecure --host=localhost:26258 - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach quit --insecure --host=localhost:26259 - ~~~ - - {{site.data.alerts.callout_info}} - For the last 2 nodes, the shutdown process will take longer (about a minute each) and will eventually force the nodes to stop. This is because, with only 2 of 5 nodes left, a majority of replicas are not available, and so the cluster is no longer operational. - {{site.data.alerts.end}} - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach quit --insecure --host=localhost:26260 - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach quit --insecure --host=localhost:26261 - ~~~ - -2. To restart the cluster at a later time, run the same `cockroach start` commands as earlier from the directory containing the nodes' data stores. - - If you do not plan to restart the cluster, you may want to remove the nodes' data stores: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ rm -rf node1 node2 node3 node4 node5 - ~~~ - -## What's next? - -- [Install the client driver](install-client-drivers.html) for your preferred language -- Learn more about [CockroachDB SQL](learn-cockroachdb-sql.html) and the [built-in SQL client](cockroach-sql.html) -- [Build an app with CockroachDB](example-apps.html) -- Further explore CockroachDB capabilities like [fault tolerance and automated repair](demo-fault-tolerance-and-recovery.html), [multi-region performance](demo-low-latency-multi-region-deployment.html), [serializable transactions](demo-serializable.html), and [JSON support](demo-json-support.html) diff --git a/src/current/v21.1/stream-data-out-of-cockroachdb-using-changefeeds.md b/src/current/v21.1/stream-data-out-of-cockroachdb-using-changefeeds.md deleted file mode 100644 index f22a91de23e..00000000000 --- a/src/current/v21.1/stream-data-out-of-cockroachdb-using-changefeeds.md +++ /dev/null @@ -1,871 +0,0 @@ ---- -title: Stream Data Out of CockroachDB Using Changefeeds -summary: Stream data out of CockroachDB with efficient, distributed, row-level change subscriptions (changefeeds). -toc: true ---- - -Change data capture (CDC) provides efficient, distributed, row-level change feeds into a configurable sink for downstream processing such as reporting, caching, or full-text indexing. - -## What is change data capture? - -While CockroachDB is an excellent system of record, it also needs to coexist with other systems. For example, you might want to keep your data mirrored in full-text indexes, analytics engines, or big data pipelines. - -The main feature of CDC is the changefeed, which targets an allowlist of tables, called the "watched rows". There are two implementations of changefeeds: - -| [Core changefeeds](#create-a-core-changefeed) | [Enterprise changefeeds](#configure-a-changefeed-enterprise) | ---------------------------------------------------|-----------------------------------------------------------------| -| Useful for prototyping or quick testing. | Recommended for production use. | -| Available in all products. | Available in CockroachDB {{ site.data.products.dedicated }} and CockroachDB {{ site.data.products.dedicated }} or with an [Enterprise license](enterprise-licensing.html) in CockroachDB. | -| Streams indefinitely until underlying SQL connection is closed. | Maintains connection to configured sink. | -| Create with [`EXPERIMENTAL CHANGEFEED FOR`](changefeed-for.html). | Create with [`CREATE CHANGEFEED`](create-changefeed.html). | -| Watches one or multiple tables in a comma-separated list. Emits every change to a "watched" row as a record. | Watches one or multiple tables in a comma-separated list. Emits every change to a "watched" row as a record in a
    configurable format (`JSON` or Avro) to a configurable sink ([Kafka](https://kafka.apache.org/)). | -| [`CREATE`](#create-a-changefeed-core) changefeed and cancel by closing the connection. | Manage changefeed with [`CREATE`](#create), [`PAUSE`](#pause), [`RESUME`](#resume), and [`CANCEL`](#cancel), as well as [monitor](#monitor-a-changefeed) and [debug](#debug-a-changefeed). | - -## Considerations - -- It is necessary to [enable rangefeeds](#enable-rangefeeds) for changefeeds to work. -- Changefeeds do not share internal buffers, so each running changefeed will increase total memory usage. To watch multiple tables, we recommend creating a changefeed with a comma-separated list of tables. -- Many DDL queries (including [`TRUNCATE`](truncate.html), [`DROP TABLE`](drop-table.html), and queries that add a column family) will cause errors on a changefeed watching the affected tables. You will need to [start a new changefeed](create-changefeed.html#start-a-new-changefeed-where-another-ended). -- Partial or intermittent sink unavailability may impact changefeed stability. If a sink is unavailable, messages can't send, which means that a changefeed's high-water mark timestamp is at risk of falling behind the cluster's [garbage collection window](configure-replication-zones.html#replication-zone-variables). Throughput and latency can be affected once the sink is available again. However, [ordering guarantees](stream-data-out-of-cockroachdb-using-changefeeds.html#ordering-guarantees) will still hold for as long as a changefeed [remains active](stream-data-out-of-cockroachdb-using-changefeeds.html#monitor-a-changefeed). -- When an [`IMPORT INTO`](import-into.html) statement is run, any current changefeed jobs targeting that table will fail. - -## Enable rangefeeds - -Changefeeds connect to a long-lived request (i.e., a rangefeed), which pushes changes as they happen. This reduces the latency of row changes, as well as reduces transaction restarts on tables being watched by a changefeed for some workloads. - -**Rangefeeds must be enabled for a changefeed to work.** To [enable the cluster setting](set-cluster-setting.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET CLUSTER SETTING kv.rangefeed.enabled = true; -~~~ - -Any created changefeed will error until this setting is enabled. Note that enabling rangefeeds currently has a small performance cost (about a 5-10% increase in latencies), whether or not the rangefeed is being used in a changefeed. - -The `kv.closed_timestamp.target_duration` [cluster setting](cluster-settings.html) can be used with changefeeds. Resolved timestamps will always be behind by at least this setting's duration; however, decreasing the duration leads to more transaction restarts in your cluster, which can affect performance. - -## Ordering guarantees - -- In most cases, each version of a row will be emitted once. However, some infrequent conditions (e.g., node failures, network partitions) will cause them to be repeated. This gives our changefeeds an **at-least-once delivery guarantee**. - -- Once a row has been emitted with some timestamp, no previously unseen versions of that row will be emitted with a lower timestamp. That is, you will never see a _new_ change for that row at an earlier timestamp. - - For example, if you ran the following: - - ~~~ sql - > CREATE TABLE foo (id INT PRIMARY KEY DEFAULT unique_rowid(), name STRING); - > CREATE CHANGEFEED FOR TABLE foo INTO 'kafka://localhost:9092' WITH UPDATED; - > INSERT INTO foo VALUES (1, 'Carl'); - > UPDATE foo SET name = 'Petee' WHERE id = 1; - ~~~ - - You'd expect the changefeed to emit: - - ~~~ shell - [1] {"__crdb__": {"updated": }, "id": 1, "name": "Carl"} - [1] {"__crdb__": {"updated": }, "id": 1, "name": "Petee"} - ~~~ - - It is also possible that the changefeed emits an out of order duplicate of an earlier value that you already saw: - - ~~~ shell - [1] {"__crdb__": {"updated": }, "id": 1, "name": "Carl"} - [1] {"__crdb__": {"updated": }, "id": 1, "name": "Petee"} - [1] {"__crdb__": {"updated": }, "id": 1, "name": "Carl"} - ~~~ - - However, you will **never** see an output like the following (i.e., an out of order row that you've never seen before): - - ~~~ shell - [1] {"__crdb__": {"updated": }, "id": 1, "name": "Petee"} - [1] {"__crdb__": {"updated": }, "id": 1, "name": "Carl"} - ~~~ - -- If a row is modified more than once in the same transaction, only the last change will be emitted. - -- Rows are sharded between Kafka partitions by the row’s [primary key](primary-key.html). - -- The `UPDATED` option adds an "updated" timestamp to each emitted row. You can also use the [`RESOLVED` option](create-changefeed.html#resolved-option) to emit "resolved" timestamp messages to each Kafka partition. A "resolved" timestamp is a guarantee that no (previously unseen) rows with a lower update timestamp will be emitted on that partition. - - For example: - - ~~~ shell - {"__crdb__": {"updated": "1532377312562986715.0000000000"}, "id": 1, "name": "Petee H"} - {"__crdb__": {"updated": "1532377306108205142.0000000000"}, "id": 2, "name": "Carl"} - {"__crdb__": {"updated": "1532377358501715562.0000000000"}, "id": 3, "name": "Ernie"} - {"__crdb__":{"resolved":"1532379887442299001.0000000000"}} - {"__crdb__":{"resolved":"1532379888444290910.0000000000"}} - {"__crdb__":{"resolved":"1532379889448662988.0000000000"}} - ... - {"__crdb__":{"resolved":"1532379922512859361.0000000000"}} - {"__crdb__": {"updated": "1532379923319195777.0000000000"}, "id": 4, "name": "Lucky"} - ~~~ - -- With duplicates removed, an individual row is emitted in the same order as the transactions that updated it. However, this is not true for updates to two different rows, even two rows in the same table. - - To compare two different rows for [happens-before](https://en.wikipedia.org/wiki/Happened-before), compare the "updated" timestamp. This works across anything in the same cluster (e.g., tables, nodes, etc.). - - Resolved timestamp notifications on every Kafka partition can be used to provide strong ordering and global consistency guarantees by buffering records in between timestamp closures. Use the "resolved" timestamp to see every row that changed at a certain time. - - The complexity with timestamps is necessary because CockroachDB supports transactions that can affect any part of the cluster, and it is not possible to horizontally divide the transaction log into independent changefeeds. For more information about this, [read our blog post on CDC](https://www.cockroachlabs.com/blog/change-data-capture/). - -## Delete messages - -Deleting a row will result in a changefeed outputting the primary key of the deleted row and a null value. For example, with default options, deleting the row with primary key `5` will output: - -~~~ shell -[5] {"after": null} -~~~ - -In some unusual situations you may receive a delete message for a row without first seeing an insert message. For example, if an attempt is made to delete a row that does not exist, you may or may not get a delete message because the changefeed behavior is undefined to allow for optimizations at the storage layer. Similarly, if there are multiple writes to a row within a single transaction, only the last one will propagate to a changefeed. This means that creating and deleting a row within the same transaction will never result in an insert message, but may result in a delete message. - -## Avro schema changes - -To ensure that the Avro schemas that CockroachDB publishes will work with the schema compatibility rules used by the Confluent schema registry, CockroachDB emits all fields in Avro as nullable unions. This ensures that Avro and Confluent consider the schemas to be both backward- and forward-compatible, since the Confluent Schema Registry has a different set of rules than Avro for schemas to be backward- and forward-compatible. - -Note that the original CockroachDB column definition is also included in the schema as a doc field, so it's still possible to distinguish between a `NOT NULL` CockroachDB column and a `NULL` CockroachDB column. - -## Schema changes with column backfill - -When schema changes with column backfill (e.g., adding a column with a default, adding a computed column, adding a `NOT NULL` column, dropping a column) are made to watched rows, the changefeed will emit some duplicates during the backfill. When it finishes, CockroachDB outputs all watched rows using the new schema. When using Avro, rows that have been backfilled by a schema change are always re-emitted. - -For an example of a schema change with column backfill, start with the changefeed created in the [example below](#create-a-changefeed-connected-to-kafka): - -~~~ -[1] {"id": 1, "name": "Petee H"} -[2] {"id": 2, "name": "Carl"} -[3] {"id": 3, "name": "Ernie"} -~~~ - -Add a column to the watched table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE office_dogs ADD COLUMN likes_treats BOOL DEFAULT TRUE; -~~~ - -The changefeed emits duplicate records 1, 2, and 3 before outputting the records using the new schema: - -~~~ -[1] {"id": 1, "name": "Petee H"} -[2] {"id": 2, "name": "Carl"} -[3] {"id": 3, "name": "Ernie"} -[1] {"id": 1, "name": "Petee H"} # Duplicate -[2] {"id": 2, "name": "Carl"} # Duplicate -[3] {"id": 3, "name": "Ernie"} # Duplicate -[1] {"id": 1, "likes_treats": true, "name": "Petee H"} -[2] {"id": 2, "likes_treats": true, "name": "Carl"} -[3] {"id": 3, "likes_treats": true, "name": "Ernie"} -~~~ - -When using the [`schema_change_policy = nobackfill` option](create-changefeed.html#schema-policy), the changefeed will still emit duplicate records for the table that is being altered. In the preceding output, the records marked as `# Duplicate` will still emit with this option, but not the new schema records. - -{{site.data.alerts.callout_info}} -Changefeeds will emit [`NULL` values](null-handling.html) for [`VIRTUAL` computed columns](computed-columns.html) and not the column's computed value. -{{site.data.alerts.end}} - -## Changefeeds on regional by row tables - -Changefeeds are supported on [regional by row tables](multiregion-overview.html#regional-by-row-tables). When working with changefeeds on regional by row tables, it is necessary to consider the following: - -- Setting a table's locality to [`REGIONAL BY ROW`](set-locality.html#regional-by-row) is equivalent to a [schema change](online-schema-changes.html) as the [`crdb_region` column](set-locality.html#crdb_region) becomes a hidden column for each of the rows in the table and is part of the [primary key](primary-key.html). Therefore, when existing tables targeted by changefeeds are made regional by row, it will trigger a backfill of the table through the changefeed. (See [Schema changes with a column backfill](stream-data-out-of-cockroachdb-using-changefeeds.html#schema-changes-with-column-backfill) for more details on the effects of schema changes on changefeeds.) - -{{site.data.alerts.callout_info}} -If the [`schema_change_policy`](create-changefeed.html#options) changefeed option is configured to `stop`, the backfill will cause the changefeed to fail. -{{site.data.alerts.end}} - -- Setting a table to `REGIONAL BY ROW` will have an impact on the changefeed's output as a result of the schema change. The backfill and future updated or inserted rows will emit output that includes the newly added `crdb_region` column as part of the schema. Therefore, it is necessary to ensure that programs consuming the changefeed can manage the new format of the primary keys. - -- [Changing a row's region](set-locality.html#update-a-rows-home-region) will appear as an insert and delete in the emitted changefeed output. For example, in the following output in which the region has been updated to `us-east1`, the insert messages are emitted followed by the [delete messages](#delete-messages): - - ~~~ - . . . - {"after": {"city": "washington dc", "crdb_region": "us-east1", "creation_time": "2019-01-02T03:04:05", "current_location": "52372 Katherine Plains", "ext": {"color": "black"}, "id": "54a69217-35ee-4000-8000-0000000001f0", "owner_id": "3dcc63f1-4120-4c00-8000-0000000004b7", "status": "in_use", "type": "scooter"}, "updated": "1632241564629087669.0000000000"} - {"after": {"city": "washington dc", "crdb_region": "us-east1", "creation_time": "2019-01-02T03:04:05", "current_location": "75024 Patrick Bridge", "ext": {"color": "black"}, "id": "54d242e6-bdc8-4400-8000-0000000001f1", "owner_id": "3ab9f559-b3d0-4c00-8000-00000000047b", "status": "in_use", "type": "scooter"}, "updated": "1632241564629087669.0000000000"} - {"after": {"city": "washington dc", "crdb_region": "us-east1", "creation_time": "2019-01-02T03:04:05", "current_location": "45597 Jackson Inlet", "ext": {"brand": "Schwinn", "color": "red"}, "id": "54fdf3b6-45a1-4c00-8000-0000000001f2", "owner_id": "4339c0eb-edfa-4400-8000-000000000521", "status": "in_use", "type": "bike"}, "updated": "1632241564629087669.0000000000"} - {"after": {"city": "washington dc", "crdb_region": "us-east1", "creation_time": "2019-01-02T03:04:05", "current_location": "18336 Katherine Port", "ext": {"color": "yellow"}, "id": "5529a485-cd7b-4000-8000-0000000001f3", "owner_id": "452bd3c3-6113-4000-8000-000000000547", "status": "in_use", "type": "scooter"}, "updated": "1632241564629087669.0000000000"} - {"after": null, "updated": "1632241564629087669.0000000000"} - {"after": null, "updated": "1632241564629087669.0000000000"} - {"after": null, "updated": "1632241564629087669.0000000000"} - {"after": null, "updated": "1632241564629087669.0000000000"} - . . . - ~~~ - -See the changefeed [responses](create-changefeed.html#responses) section for more general information on the messages emitted from a changefeed. - -## Create a changefeed (Core) - -A core changefeed streams row-level changes to the client indefinitely until the underlying connection is closed or the changefeed is canceled. - -To create a core changefeed: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPERIMENTAL CHANGEFEED FOR name; -~~~ - -For more information, see [`CHANGEFEED FOR`](changefeed-for.html). - -## Configure a changefeed (Enterprise) - -An Enterprise changefeed streams row-level changes in a configurable format to a configurable sink (i.e., Kafka or a cloud storage sink). You can [create](#create), [pause](#pause), [resume](#resume), [cancel](#cancel), [monitor](#monitor-a-changefeed), and [debug](#debug-a-changefeed) an Enterprise changefeed. - -### Create - -To create an Enterprise changefeed: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE CHANGEFEED FOR TABLE table_name, table_name2 INTO '{scheme}://{host}:{port}?{query_parameters}'; -~~~ - -{% include {{ page.version.version }}/cdc/url-encoding.md %} - -For more information, see [`CREATE CHANGEFEED`](create-changefeed.html). - -### Pause - -To pause an Enterprise changefeed: - -{% include_cached copy-clipboard.html %} -~~~ sql -> PAUSE JOB job_id; -~~~ - -For more information, see [`PAUSE JOB`](pause-job.html). - -### Resume - -To resume a paused Enterprise changefeed: - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESUME JOB job_id; -~~~ - -For more information, see [`RESUME JOB`](resume-job.html). - -### Cancel - -To cancel an Enterprise changefeed: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CANCEL JOB job_id; -~~~ - -For more information, see [`CANCEL JOB`](cancel-job.html). - -### Configuring all changefeeds - -{% include {{ page.version.version }}/cdc/configure-all-changefeed.md %} - -## Monitor a changefeed - -{{site.data.alerts.callout_info}} -Monitoring is only available for Enterprise changefeeds. -{{site.data.alerts.end}} - -Changefeed progress is exposed as a high-water timestamp that advances as the changefeed progresses. This is a guarantee that all changes before or at the timestamp have been emitted. You can monitor a changefeed: - -- On the [Changefeed Dashboard](ui-cdc-dashboard.html) of the DB Console. -- On the [Jobs page](ui-jobs-page.html) of the DB Console. Hover over the high-water timestamp to view the [system time](as-of-system-time.html). -- Using `crdb_internal.jobs`: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SELECT * FROM crdb_internal.jobs WHERE job_id = ; - ~~~ - ~~~ - job_id | job_type | description | ... | high_water_timestamp | error | coordinator_id - +--------------------+------------+------------------------------------------------------------------------+ ... +--------------------------------+-------+----------------+ - 383870400694353921 | CHANGEFEED | CREATE CHANGEFEED FOR TABLE office_dogs INTO 'kafka://localhost:9092' | ... | 1537279405671006870.0000000000 | | 1 - (1 row) - ~~~ - -- Setting up an alert on the `changefeed.max_behind_nanos` metric to track when a changefeed's high-water mark timestamp is at risk of falling behind the cluster's [garbage collection window](configure-replication-zones.html#replication-zone-variables). For more information, see [Monitoring and Alerting](monitoring-and-alerting.html#changefeed-is-experiencing-high-latency). - -{{site.data.alerts.callout_info}} -You can use the high-water timestamp to [start a new changefeed where another ended](create-changefeed.html#start-a-new-changefeed-where-another-ended). -{{site.data.alerts.end}} - -## Debug a changefeed - -### Using logs - -For Enterprise changefeeds, [use log information](logging-overview.html) to debug connection issues (i.e., `kafka: client has run out of available brokers to talk to (Is your cluster reachable?)`). Debug by looking for lines in the logs with `[kafka-producer]` in them: - -~~~ -I190312 18:56:53.535646 585 vendor/github.com/Shopify/sarama/client.go:123 [kafka-producer] Initializing new client -I190312 18:56:53.535714 585 vendor/github.com/Shopify/sarama/client.go:724 [kafka-producer] client/metadata fetching metadata for all topics from broker localhost:9092 -I190312 18:56:53.536730 569 vendor/github.com/Shopify/sarama/broker.go:148 [kafka-producer] Connected to broker at localhost:9092 (unregistered) -I190312 18:56:53.537661 585 vendor/github.com/Shopify/sarama/client.go:500 [kafka-producer] client/brokers registered new broker #0 at 172.16.94.87:9092 -I190312 18:56:53.537686 585 vendor/github.com/Shopify/sarama/client.go:170 [kafka-producer] Successfully initialized new client -~~~ - -### Using `SHOW JOBS` - -For Enterprise changefeeds, you can check the status by using: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT * FROM [SHOW JOBS] WHERE job_type='CHANGEFEED'; -~~~ - -Or: - -{% include_cached copy-clipboard.html %} -~~~ sql -SELECT * from crdb_internal.jobs WHERE job_type='CHANGEFEED'; -~~~ - -For more information, see [`SHOW JOBS`](show-jobs.html). - -### Using the DB Console - - On the [**Custom Chart** debug page](ui-custom-chart-debug-page.html) of the DB Console: - -1. To add a chart, click **Add Chart**. -2. Select `changefeed.error_retries` from the **Metric Name** dropdown menu. - - A graph of changefeed restarts due to retryable errors will display. - -## Usage examples - -### Create a core changefeed - -{% include {{ page.version.version }}/cdc/create-core-changefeed.md %} - -### Create a core changefeed using Avro - -{% include {{ page.version.version }}/cdc/create-core-changefeed-avro.md %} - -### Create a changefeed connected to Kafka - -{{site.data.alerts.callout_info}} -[`CREATE CHANGEFEED`](create-changefeed.html) is an [Enterprise-only](enterprise-licensing.html) feature. For the core version, see [the `CHANGEFEED FOR` example above](#create-a-core-changefeed). -{{site.data.alerts.end}} - -In this example, you'll set up a changefeed for a single-node cluster that is connected to a Kafka sink. The changefeed will watch two tables. - -1. If you do not already have one, [request a trial Enterprise license](enterprise-licensing.html). - -2. Use the [`cockroach start-single-node`](cockroach-start-single-node.html) command to start a single-node cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start-single-node --insecure --listen-addr=localhost --background - ~~~ - -3. Download and extract the [Confluent Open Source platform](https://www.confluent.io/download/) (which includes Kafka). - -4. Move into the extracted `confluent-` directory and start Confluent: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ ./bin/confluent local services start - ~~~ - - Only `zookeeper` and `kafka` are needed. To troubleshoot Confluent, see [their docs](https://docs.confluent.io/current/installation/installing_cp.html#zip-and-tar-archives) and the [Quick Start Guide](https://docs.confluent.io/platform/current/quickstart/ce-quickstart.html#ce-quickstart). - -5. Create two Kafka topics: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ ./bin/kafka-topics \ - --create \ - --zookeeper localhost:2181 \ - --replication-factor 1 \ - --partitions 1 \ - --topic office_dogs - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ ./bin/kafka-topics \ - --create \ - --zookeeper localhost:2181 \ - --replication-factor 1 \ - --partitions 1 \ - --topic employees - ~~~ - - {{site.data.alerts.callout_info}} - You are expected to create any Kafka topics with the necessary number of replications and partitions. [Topics can be created manually](https://kafka.apache.org/documentation/#basic_ops_add_topic) or [Kafka brokers can be configured to automatically create topics](https://kafka.apache.org/documentation/#topicconfigs) with a default partition count and replication factor. - {{site.data.alerts.end}} - -6. As the `root` user, open the [built-in SQL client](cockroach-sql.html): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --insecure - ~~~ - -7. Set your organization name and [Enterprise license](enterprise-licensing.html) key that you received via email: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING cluster.organization = ''; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING enterprise.license = ''; - ~~~ - -8. Enable the `kv.rangefeed.enabled` [cluster setting](cluster-settings.html): - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING kv.rangefeed.enabled = true; - ~~~ - -9. Create a database called `cdc_demo`: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE DATABASE cdc_demo; - ~~~ - -10. Set the database as the default: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET DATABASE = cdc_demo; - ~~~ - -11. Create a table and add data: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE TABLE office_dogs ( - id INT PRIMARY KEY, - name STRING); - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > INSERT INTO office_dogs VALUES - (1, 'Petee'), - (2, 'Carl'); - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > UPDATE office_dogs SET name = 'Petee H' WHERE id = 1; - ~~~ - -12. Create another table and add data: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE TABLE employees ( - dog_id INT REFERENCES office_dogs (id), - employee_name STRING); - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > INSERT INTO employees VALUES - (1, 'Lauren'), - (2, 'Spencer'); - ~~~ - -13. Start the changefeed: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE CHANGEFEED FOR TABLE office_dogs, employees INTO 'kafka://localhost:9092'; - ~~~ - ~~~ - - job_id - +--------------------+ - 360645287206223873 - (1 row) - ~~~ - - This will start up the changefeed in the background and return the `job_id`. The changefeed writes to Kafka. - -14. In a new terminal, move into the extracted `confluent-` directory and start watching the Kafka topics: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ ./bin/kafka-console-consumer \ - --bootstrap-server=localhost:9092 \ - --from-beginning \ - --whitelist 'office_dogs|employees' - ~~~ - - ~~~ shell - {"after": {"id": 1, "name": "Petee H"}} - {"after": {"id": 2, "name": "Carl"}} - {"after": {"id": 1, "name": "Lauren", "rowid": 528514320239329281}} - {"after": {"id": 2, "name": "Spencer", "rowid": 528514320239362049}} - ~~~ - - The initial scan displays the state of the tables as of when the changefeed started (therefore, the initial value of `"Petee"` is omitted). - - {% include {{ page.version.version }}/cdc/print-key.md %} - -15. Back in the SQL client, insert more data: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > INSERT INTO office_dogs VALUES (3, 'Ernie'); - ~~~ - -16. Back in the terminal where you're watching the Kafka topics, the following output has appeared: - - ~~~ shell - {"after": {"id": 3, "name": "Ernie"}} - ~~~ - -17. When you are done, exit the SQL shell (`\q`). - -18. To stop `cockroach`, run: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach quit --insecure - ~~~ - -19. To stop Kafka, move into the extracted `confluent-` directory and stop Confluent: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ ./bin/confluent local services stop - ~~~ - -### Create a changefeed connected to Kafka using Avro - -{{site.data.alerts.callout_info}} -[`CREATE CHANGEFEED`](create-changefeed.html) is an [Enterprise-only](enterprise-licensing.html) feature. For the core version, see [the `CHANGEFEED FOR` example above](#create-a-core-changefeed-using-avro). -{{site.data.alerts.end}} - -In this example, you'll set up a changefeed for a single-node cluster that is connected to a Kafka sink and emits [Avro](https://avro.apache.org/docs/1.8.2/spec.html) records. The changefeed will watch two tables. - -1. If you do not already have one, [request a trial Enterprise license](enterprise-licensing.html). - -2. Use the [`cockroach start-single-node`](cockroach-start-single-node.html) command to start a single-node cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start-single-node --insecure --listen-addr=localhost --background - ~~~ - -3. Download and extract the [Confluent Open Source platform](https://www.confluent.io/download/) (which includes Kafka). - -4. Move into the extracted `confluent-` directory and start Confluent: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ ./bin/confluent local services start - ~~~ - - Only `zookeeper`, `kafka`, and `schema-registry` are needed. To troubleshoot Confluent, see [their docs](https://docs.confluent.io/current/installation/installing_cp.html#zip-and-tar-archives) and the [Quick Start Guide](https://docs.confluent.io/platform/current/quickstart/ce-quickstart.html#ce-quickstart). - -5. Create two Kafka topics: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ ./bin/kafka-topics \ - --create \ - --zookeeper localhost:2181 \ - --replication-factor 1 \ - --partitions 1 \ - --topic office_dogs - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ ./bin/kafka-topics \ - --create \ - --zookeeper localhost:2181 \ - --replication-factor 1 \ - --partitions 1 \ - --topic employees - ~~~ - - {{site.data.alerts.callout_info}} - You are expected to create any Kafka topics with the necessary number of replications and partitions. [Topics can be created manually](https://kafka.apache.org/documentation/#basic_ops_add_topic) or [Kafka brokers can be configured to automatically create topics](https://kafka.apache.org/documentation/#topicconfigs) with a default partition count and replication factor. - {{site.data.alerts.end}} - -6. As the `root` user, open the [built-in SQL client](cockroach-sql.html): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --insecure - ~~~ - -7. Set your organization name and [Enterprise license](enterprise-licensing.html) key that you received via email: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING cluster.organization = ''; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING enterprise.license = ''; - ~~~ - -8. Enable the `kv.rangefeed.enabled` [cluster setting](cluster-settings.html): - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING kv.rangefeed.enabled = true; - ~~~ - -9. Create a database called `cdc_demo`: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE DATABASE cdc_demo; - ~~~ - -10. Set the database as the default: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET DATABASE = cdc_demo; - ~~~ - -11. Create a table and add data: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE TABLE office_dogs ( - id INT PRIMARY KEY, - name STRING); - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > INSERT INTO office_dogs VALUES - (1, 'Petee'), - (2, 'Carl'); - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > UPDATE office_dogs SET name = 'Petee H' WHERE id = 1; - ~~~ - -12. Create another table and add data: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE TABLE employees ( - dog_id INT REFERENCES office_dogs_avro (id), - employee_name STRING); - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > INSERT INTO employees VALUES - (1, 'Lauren'), - (2, 'Spencer'); - ~~~ - -13. Start the changefeed: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE CHANGEFEED FOR TABLE office_dogs, employees INTO 'kafka://localhost:9092' WITH format = experimental_avro, confluent_schema_registry = 'http://localhost:8081'; - ~~~ - - ~~~ - job_id - +--------------------+ - 360645287206223873 - (1 row) - ~~~ - - This will start up the changefeed in the background and return the `job_id`. The changefeed writes to Kafka. - -14. In a new terminal, move into the extracted `confluent-` directory and start watching the Kafka topics: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ ./bin/kafka-avro-console-consumer \ - --bootstrap-server=localhost:9092 \ - --from-beginning \ - --whitelist 'office_dogs|employees' - ~~~ - - ~~~ shell - {"after":{"office_dogs":{"id":{"long":1},"name":{"string":"Petee H"}}}} - {"after":{"office_dogs":{"id":{"long":2},"name":{"string":"Carl"}}}} - {"after":{"employees":{"dog_id":{"long":1},"employee_name":{"string":"Lauren"},"rowid":{"long":528537452042682369}}}} - {"after":{"employees":{"dog_id":{"long":2},"employee_name":{"string":"Spencer"},"rowid":{"long":528537452042747905}}}} - ~~~ - - The initial scan displays the state of the table as of when the changefeed started (therefore, the initial value of `"Petee"` is omitted). - - {% include {{ page.version.version }}/cdc/print-key.md %} - -15. Back in the SQL client, insert more data: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > INSERT INTO office_dogs VALUES (3, 'Ernie'); - ~~~ - -16. Back in the terminal where you're watching the Kafka topics, the following output has appeared: - - ~~~ shell - {"after":{"office_dogs":{"id":{"long":3},"name":{"string":"Ernie"}}}} - ~~~ - -17. When you are done, exit the SQL shell (`\q`). - -18. To stop `cockroach`, run: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach quit --insecure - ~~~ - -19. To stop Kafka, move into the extracted `confluent-` directory and stop Confluent: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ ./bin/confluent local services stop - ~~~ - -### Create a changefeed connected to a cloud storage sink - -{{site.data.alerts.callout_info}} -[`CREATE CHANGEFEED`](create-changefeed.html) is an [Enterprise-only](enterprise-licensing.html) feature. For the core version, see [the `CHANGEFEED FOR` example above](#create-a-core-changefeed). -{{site.data.alerts.end}} - -{% include {{ page.version.version }}/misc/experimental-warning.md %} - -In this example, you'll set up a changefeed for a single-node cluster that is connected to an AWS S3 sink. The changefeed watches two tables. Note that you can set up changefeeds for any of [these cloud storage providers](create-changefeed.html#cloud-storage-sink). - -1. If you do not already have one, [request a trial Enterprise license](enterprise-licensing.html). - -2. Use the [`cockroach start-single-node`](cockroach-start-single-node.html) command to start a single-node cluster: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start-single-node --insecure --listen-addr=localhost --background - ~~~ - -3. As the `root` user, open the [built-in SQL client](cockroach-sql.html): - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach sql --insecure - ~~~ - -4. Set your organization name and [Enterprise license](enterprise-licensing.html) key that you received via email: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING cluster.organization = ''; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING enterprise.license = ''; - ~~~ - -5. Enable the `kv.rangefeed.enabled` [cluster setting](cluster-settings.html): - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING kv.rangefeed.enabled = true; - ~~~ - -6. Create a database called `cdc_demo`: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE DATABASE cdc_demo; - ~~~ - -7. Set the database as the default: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET DATABASE = cdc_demo; - ~~~ - -8. Create a table and add data: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE TABLE office_dogs ( - id INT PRIMARY KEY, - name STRING); - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > INSERT INTO office_dogs VALUES - (1, 'Petee'), - (2, 'Carl'); - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > UPDATE office_dogs SET name = 'Petee H' WHERE id = 1; - ~~~ - -9. Create another table and add data: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE TABLE employees ( - dog_id INT REFERENCES office_dogs (id), - employee_name STRING); - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - > INSERT INTO employees VALUES - (1, 'Lauren'), - (2, 'Spencer'); - ~~~ - -10. Start the changefeed: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > CREATE CHANGEFEED FOR TABLE office_dogs, employees INTO 'experimental-s3://example-bucket-name/test?AWS_ACCESS_KEY_ID=enter_key-here&AWS_SECRET_ACCESS_KEY=enter_key_here' with updated, resolved='10s'; - ~~~ - - ~~~ - job_id - +--------------------+ - 360645287206223873 - (1 row) - ~~~ - - This will start up the changefeed in the background and return the `job_id`. The changefeed writes to AWS. - -11. Monitor your changefeed on the DB Console. For more information, see [Changefeeds Dashboard](ui-cdc-dashboard.html). - -12. When you are done, exit the SQL shell (`\q`). - -13. To stop `cockroach`, run: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach quit --insecure - ~~~ - -## Known limitations - -{% include {{ page.version.version }}/known-limitations/cdc.md %} - -## See also - -- [`CREATE CHANGEFEED`](create-changefeed.html) -- [`CHANGEFEED FOR`](changefeed-for.html) -- [`PAUSE JOB`](pause-job.html) -- [`CANCEL JOB`](cancel-job.html) -- [Other SQL Statements](sql-statements.html) -- [Changefeed Dashboard](ui-cdc-dashboard.html) diff --git a/src/current/v21.1/string.md b/src/current/v21.1/string.md deleted file mode 100644 index 6989d9203eb..00000000000 --- a/src/current/v21.1/string.md +++ /dev/null @@ -1,223 +0,0 @@ ---- -title: STRING -summary: The STRING data type stores a string of Unicode characters. -toc: true ---- - -The `STRING` [data type](data-types.html) stores a string of Unicode characters. - -{{site.data.alerts.callout_info}} -`STRING` is not a data type supported by PostgreSQL. For PostgreSQL compatibility, CockroachDB supports additional [aliases](#aliases) and [`STRING`-related types](#related-types). -{{site.data.alerts.end}} - -## Aliases - -CockroachDB supports the following alias for `STRING`: - -- `TEXT` - -## Related types - -For PostgreSQL compatibility, CockroachDB supports the following `STRING`-related types and their aliases: - -- `VARCHAR` (and alias `CHARACTER VARYING`) -- `CHAR` (and alias `CHARACTER`) - -These types are functionality identical to `STRING`. - -## Length - -To limit the length of a string column, use `STRING(n)`, where `n` is the maximum number of Unicode code points (normally thought of as "characters") allowed. This applies to all related types as well (e.g., to limit the length of a `VARCHAR` type, use `VARCHAR(n)`). To reduce performance issues caused by storing very large string values in indexes, Cockroach Labs recommends setting length limits on string-typed columns. - -When inserting a `STRING` value or a `STRING`-related-type value: - -- If the value is cast with a length limit (e.g., `CAST('hello world' AS STRING(5))`), CockroachDB truncates to the limit. This applies to `STRING(n)` and all related types. -- If the value exceeds the column's length limit, CockroachDB returns an error. This applies to `STRING(n)` and all related types. -- For `STRING(n)` and `VARCHAR(n)`/`CHARACTER VARYING(n)` types, if the value is under the column's length limit, CockroachDB does **not** add space padding to the end of the value. -- For `CHAR(n)`/`CHARACTER(n)` types, if the value is under the column's length limit, CockroachDB adds space padding from the end of the value to the length limit. - - Type | Length ---------------------------------------------------|------------------------------ -`CHARACTER`, `CHARACTER(n)`, `CHAR`, `CHAR(n)` | Fixed-length -`CHARACTER VARYING(n)`, `VARCHAR(n)`, `STRING(n)` | Variable-length, with a limit -`TEXT`, `VARCHAR`, `CHARACTER VARYING`, `STRING` | Variable-length, with no limit - -## Syntax - -A value of type `STRING` can be expressed using a variety of formats. -See [string literals](sql-constants.html#string-literals) for more details. - -When printing out a `STRING` value in the [SQL shell](cockroach-sql.html), the shell uses the simple -SQL string literal format if the value doesn't contain special character, -or the escaped format otherwise. - -### Collations - -`STRING` values accept [collations](collate.html), which lets you sort strings according to language- and country-specific rules. - -## Size - -The size of a `STRING` value is variable, but it's recommended to keep values under 64 kilobytes to ensure performance. Above that threshold, [write amplification](https://en.wikipedia.org/wiki/Write_amplification) and other considerations may cause significant performance degradation. - -## Examples - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE strings (a STRING PRIMARY KEY, b STRING(4), c TEXT); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM strings; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden -+-------------+-----------+-------------+----------------+-----------------------+-------------+-----------+ - a | STRING | false | NULL | | {"primary"} | false - b | STRING(4) | true | NULL | | {} | false - c | STRING | true | NULL | | {} | false -(3 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO strings VALUES ('a1b2c3d4', 'e5f6', 'g7h8i9'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM strings; -~~~ - -~~~ - a | b | c -+----------+------+--------+ - a1b2c3d4 | e5f6 | g7h8i9 -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE aliases (a STRING PRIMARY KEY, b VARCHAR, c CHAR); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM aliases; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden -+-------------+-----------+-------------+----------------+-----------------------+-------------+-----------+ - a | STRING | false | NULL | | {"primary"} | false - b | VARCHAR | true | NULL | | {} | false - c | CHAR | true | NULL | | {} | false -(3 rows) -~~~ - -## Supported casting and conversion - -`STRING` values can be [cast](data-types.html#data-type-conversions-and-casts) to any of the following data types: - -Type | Details ------|-------- -`ARRAY` | Requires supported [`ARRAY`](array.html) string format, e.g., `'{1,2,3}'`.
    Note that string literals can be implicitly cast to any supported `ARRAY` data type except [`BYTES`](bytes.html), [`ENUM`](enum.html), [`JSONB`](jsonb.html), [`SERIAL`](serial.html), and the [spatial data types](spatial-glossary.html#data-types) `Box2D`, `GEOGRAPHY`, and `GEOMETRY`. -`BIT` | Requires supported [`BIT`](bit.html) string format, e.g., `'101001'`. -`BOOL` | Requires supported [`BOOL`](bool.html) string format, e.g., `'true'`. -`BYTES` | For more details, [see here](bytes.html#supported-conversions). -`DATE` | Requires supported [`DATE`](date.html) string format, e.g., `'2016-01-25'`. -`DECIMAL` | Requires supported [`DECIMAL`](decimal.html) string format, e.g., `'1.1'`. -`FLOAT` | Requires supported [`FLOAT`](float.html) string format, e.g., `'1.1'`. -`INET` | Requires supported [`INET`](inet.html) string format, e.g, `'192.168.0.1'`. -`INT` | Requires supported [`INT`](int.html) string format, e.g., `'10'`. -`INTERVAL` | Requires supported [`INTERVAL`](interval.html) string format, e.g., `'1h2m3s4ms5us6ns'`. -`TIME` | Requires supported [`TIME`](time.html) string format, e.g., `'01:22:12'` (microsecond precision). -`TIMESTAMP` | Requires supported [`TIMESTAMP`](timestamp.html) string format, e.g., `'2016-01-25 10:10:10.555555'`. -`UUID` | Requires supported [`UUID`](uuid.html) string format, e.g., `'63616665-6630-3064-6465-616462656562'`. - -### `STRING` vs. `BYTES` - -While both `STRING` and `BYTES` can appear to have similar behavior in many situations, one should understand their nuance before casting one into the other. - -`STRING` treats all of its data as characters, or more specifically, Unicode code points. `BYTES` treats all of its data as a byte string. This difference in implementation can lead to dramatically different behavior. For example, let's take a complex Unicode character such as ☃ ([the snowman emoji](https://emojipedia.org/snowman/)): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT length('☃'::string); -~~~ - -~~~ - length -+--------+ - 1 -~~~ - -~~~ sql -> SELECT length('☃'::bytes); -~~~ -~~~ - length -+--------+ - 3 -~~~ - -In this case, [`LENGTH(string)`](functions-and-operators.html#string-and-byte-functions) measures the number of Unicode code points present in the string, whereas [`LENGTH(bytes)`](functions-and-operators.html#string-and-byte-functions) measures the number of bytes required to store that value. Each character (or Unicode code point) can be encoded using multiple bytes, hence the difference in output between the two. - -#### Translate literals to `STRING` vs. `BYTES` - -A literal entered through a SQL client will be translated into a different value based on the type: - -+ `BYTES` give a special meaning to the pair `\x` at the beginning, and translates the rest by substituting pairs of hexadecimal digits to a single byte. For example, `\xff` is equivalent to a single byte with the value of 255. For more information, see [SQL Constants: String literals with character escapes](sql-constants.html#string-literals-with-character-escapes). -+ `STRING` does not give a special meaning to `\x`, so all characters are treated as distinct Unicode code points. For example, `\xff` is treated as a `STRING` with length 4 (`\`, `x`, `f`, and `f`). - -### Concatenate `STRING` values with values of other types - -{% include_cached new-in.html version="v21.1" %} `STRING` values can be concatenated with any non-`ARRAY`, non-`NULL` type, resulting in a `STRING` value. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT 1 || 'item'; -~~~ - -~~~ - ?column? ------------- - 1item -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT true || 'item'; -~~~ - -~~~ - ?column? ------------- - titem -(1 row) -~~~ - -Concatenating a `STRING` value with a [`NULL` value](null-handling.html) results in a `NULL` value. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT NULL || 'item'; -~~~ - -~~~ - ?column? ------------- - NULL -(1 row) -~~~ - -## See also - -- [Data Types](data-types.html) -- [String literal syntax](sql-constants.html#string-literals) diff --git a/src/current/v21.1/subqueries.md b/src/current/v21.1/subqueries.md deleted file mode 100644 index 4ca7824357a..00000000000 --- a/src/current/v21.1/subqueries.md +++ /dev/null @@ -1,154 +0,0 @@ ---- -title: Subqueries -summary: Subqueries enable the use of the results from a query within another query. -toc: true ---- - -SQL subqueries enable reuse of the results from a [selection query](selection-queries.html) within another query. - - -## Overview - -CockroachDB supports two kinds of subqueries: - -- **Relational** subqueries which appear as operand in [selection queries](selection-queries.html) or [table expressions](table-expressions.html). -- **Scalar** subqueries which appear as operand in a [scalar expression](scalar-expressions.html). - -## Data writes in subqueries - -When a subquery contains a data-modifying statement (`INSERT`, -`DELETE`, etc.), the data modification is always executed to -completion even if the surrounding query only uses a subset of the -result rows. - -This is true both for subqueries defined using the `(...)` or `[...]` -notations, and those defined using -[`WITH`](common-table-expressions.html). - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * - FROM [INSERT INTO t(x) VALUES (1), (2), (3) RETURNING x] - LIMIT 1; -~~~ - -This query always inserts 3 rows into `t`, even though the surrounding -query only observes 1 row using [`LIMIT`](limit-offset.html). - -## Correlated subqueries - -CockroachDB's [cost-based optimizer](cost-based-optimizer.html) supports most [correlated subqueries](https://en.wikipedia.org/wiki/Correlated_subquery). - -A subquery is said to be "correlated" when it uses table or column names defined in the surrounding query. - -For example, to find every customer with at least one order, run: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT - c.name - FROM - customers AS c - WHERE - EXISTS( - SELECT * FROM orders AS o WHERE o.customer_id = c.id - ); -~~~ - -The subquery is correlated because it uses `c` defined in the surrounding query. - -### `LATERAL` subqueries - - CockroachDB supports `LATERAL` subqueries. A `LATERAL` subquery is a correlated subquery that references another query or subquery in its `SELECT` statement, usually in the context of a [`LEFT` join](joins.html#left-outer-joins) or an [`INNER` join](joins.html#inner-joins). Unlike other correlated subqueries, `LATERAL` subqueries iterate through each row in the referenced query for each row in the inner subquery, like a [for loop](https://en.wikipedia.org/wiki/For_loop). - -To create a `LATERAL` subquery, use the `LATERAL` keyword directly before the inner subquery's `SELECT` statement. - -For example, the following statement performs an `INNER` join of the `users` table and a subquery of the `rides` table that filters on values in the `users` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT name, address FROM users, LATERAL (SELECT * FROM rides WHERE rides.start_address = users.address AND city = 'new york'); -~~~ - -~~~ - name | address -+------------------+-----------------------------+ - Robert Murphy | 99176 Anderson Mills - James Hamilton | 73488 Sydney Ports Suite 57 - Judy White | 18580 Rosario Ville Apt. 61 - Devin Jordan | 81127 Angela Ferry Apt. 8 - Catherine Nelson | 1149 Lee Alley - Nicole Mcmahon | 11540 Patton Extensions -(6 rows) -~~~ - -`LATERAL` subquery joins are especially useful when the join table includes a [computed column](computed-columns.html). - -For example, the following query joins a subquery of the `rides` table with a computed column (`adjusted_revenue`), and a subquery of the `users` table that references columns in the `rides` subquery: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT - ride_id, - vehicle_id, - type, - adjusted_revenue -FROM - ( - SELECT - id AS ride_id, - vehicle_id, - revenue - 0.25*revenue AS adjusted_revenue - FROM - rides - ) - r - JOIN - LATERAL ( - SELECT - type - FROM - vehicles - WHERE - city = 'new york' - AND vehicles.id = r.vehicle_id - AND r.adjusted_revenue > 65 ) v - ON true; -~~~ - -~~~ - ride_id | vehicle_id | type | adjusted_revenue -+--------------------------------------+--------------------------------------+------------+------------------+ - 049ba5e3-53f7-4ec0-8000-000000000009 | 11111111-1111-4100-8000-000000000001 | scooter | 71.2500 - 0624dd2f-1a9f-4e80-8000-00000000000c | 00000000-0000-4000-8000-000000000000 | skateboard | 70.5000 - 08b43958-1062-4e00-8000-000000000011 | 11111111-1111-4100-8000-000000000001 | scooter | 70.5000 - 0bc6a7ef-9db2-4d00-8000-000000000017 | 00000000-0000-4000-8000-000000000000 | skateboard | 68.2500 - 0d4fdf3b-645a-4c80-8000-00000000001a | 00000000-0000-4000-8000-000000000000 | skateboard | 67.5000 - 1ba5e353-f7ce-4900-8000-000000000036 | 11111111-1111-4100-8000-000000000001 | scooter | 70.5000 -(6 rows) -~~~ - -{{site.data.alerts.callout_info}} -In a `LATERAL` subquery join, the rows returned by the inner subquery are added to the result of the join with the outer query. Without the `LATERAL` keyword, each subquery is evaluated independently and cannot refer to objects defined in separate queries. -{{site.data.alerts.end}} - -### Limitations - -The [cost-based optimizer](cost-based-optimizer.html) supports most correlated subqueries, with the exception of correlated subqueries that generate side effects inside a `CASE` statement. - -## Performance best practices - -{{site.data.alerts.callout_info}} -CockroachDB is currently undergoing major changes to evolve and improve the performance of subqueries. The restrictions and workarounds listed in this section will be lifted or made unnecessary over time. -{{site.data.alerts.end}} - -- The results of scalar subqueries are currently loaded entirely into memory when the execution of the surrounding query starts. To prevent execution errors due to memory exhaustion, ensure that subqueries return as few results as possible. - -## See also - -- [Selection Queries](selection-queries.html) -- [Scalar Expressions](scalar-expressions.html) -- [Table Expressions](table-expressions.html) -- [Performance Best Practices - Overview](performance-best-practices-overview.html) diff --git a/src/current/v21.1/support-resources.md b/src/current/v21.1/support-resources.md deleted file mode 100644 index c44f9406bbc..00000000000 --- a/src/current/v21.1/support-resources.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -title: Support Resources -summary: There are various ways to reach out for support from Cockroach Labs and our community. -toc: false ---- - -For each major release of CockroachDB, Cockroach Labs provides maintenance support for at least 365 days and assistance support for at least an additional 180 days. For more details, see the [Release Support Policy](../releases/release-support-policy.html). - -If you're having an issue with CockroachDB, you can reach out for support from Cockroach Labs and our community: - -- [Troubleshooting documentation](troubleshooting-overview.html) -- [CockroachDB Community Forum](https://forum.cockroachlabs.com) -- [CockroachDB Community Slack](https://cockroachdb.slack.com) -- [StackOverflow](http://stackoverflow.com/questions/tagged/cockroachdb) -- [File a GitHub issue](file-an-issue.html) -- [CockroachDB Support Portal](https://support.cockroachlabs.com) - -We also rely on contributions from users like you. If you know how to help users who might be struggling with a problem, we hope you will! diff --git a/src/current/v21.1/survive-failure.md b/src/current/v21.1/survive-failure.md deleted file mode 100644 index 7a769aa1492..00000000000 --- a/src/current/v21.1/survive-failure.md +++ /dev/null @@ -1,77 +0,0 @@ ---- -title: SURVIVE {ZONE, REGION} FAILURE -summary: The SURVIVE {ZONE,REGION} FAILURE statement configures a multi-region database. -toc: true ---- - -{% include_cached new-in.html version="v21.1" %} The `ALTER DATABASE ... SURVIVE {ZONE,REGION} FAILURE` [statement](sql-statements.html) sets the [survival goal](multiregion-overview.html#survival-goals) for a [multi-region database](multiregion-overview.html). - -{{site.data.alerts.callout_info}} -`SURVIVE {ZONE,REGION} FAILURE` is a subcommand of [`ALTER DATABASE`](alter-database.html). -{{site.data.alerts.end}} - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/alter_database_survival_goal.html %} -
    - -## Parameters - -| Parameter | Description | -|-----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `database_name` | The database whose [survival goal](multiregion-overview.html#survival-goals) you are configuring. | - -## Required privileges - -The user must be a member of the [`admin`](authorization.html#roles) or [owner](authorization.html#object-ownership) roles, or have the [`CREATE` privilege](authorization.html#supported-privileges) on the database. - -## Examples - -### Survive zone failures - -To change the survival goal of a multi-region database to survive zone failures, use the following statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE {db} SURVIVE ZONE FAILURE; -~~~ - -~~~ -ALTER DATABASE SURVIVE -~~~ - -{{site.data.alerts.callout_info}} -Surviving zone failures is the default setting for multi-region databases. -{{site.data.alerts.end}} - -For more information about the zone survival goal, see [Surviving zone failures](multiregion-overview.html#surviving-zone-failures). - -### Survive region failures - -To change the survival goal of a multi-region database to survive region failures, use the following statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -ALTER DATABASE {db} SURVIVE REGION FAILURE; -~~~ - -~~~ -ALTER DATABASE SURVIVE -~~~ - -If you try to change a database with less than 3 [database regions](multiregion-overview.html#database-regions) to survive region failures, the following error will be signalled: - -~~~ -ERROR: at least 3 regions are required for surviving a region failure -SQLSTATE: 42602 -HINT: you must add additional regions to the database using ALTER DATABASE mr ADD REGION -~~~ - -For more information about the region survival goal, see [Surviving region failures](multiregion-overview.html#surviving-region-failures). - -## See also - -- [Multi-region overview](multiregion-overview.html) -- [`ALTER TABLE`](alter-table.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/system-catalogs.md b/src/current/v21.1/system-catalogs.md deleted file mode 100644 index 3607963e257..00000000000 --- a/src/current/v21.1/system-catalogs.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -title: System Catalogs -summary: CockroachDB includes several virtual schemas that enable you to interface with CockroachDB. -toc: true ---- - -CockroachDB includes a set of system catalogs that provide non-stored data to client applications. - -The following system catalogs are available as schemas preloaded to every database: - -- [`information_schema`](information-schema.html), a schema provided for compatibility with the SQL standard. -- [`crdb_internal`](crdb-internal.html), a schema with information about CockroachDB internals. -- [`pg_catalog`](pg-catalog.html), a schema provided for compatibility with PostgreSQL. -- [`pg_extension`](pg-extension.html), a schema catalog with information about CockroachDB extensions. - -{{site.data.alerts.callout_danger}} -Tables in the system catalogs have varying levels of stability. Not all system catalog tables are meant for programmatic purposes. -{{site.data.alerts.end}} - -To see all of the system catalogs for the [current database](sql-name-resolution.html#current-database), you can use a [`SHOW SCHEMAS`](show-schemas.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SCHEMAS; -~~~ - -~~~ - schema_name | owner ----------------------+-------- - crdb_internal | NULL - information_schema | NULL - pg_catalog | NULL - pg_extension | NULL - public | admin -(5 rows) -~~~ - -## See also - -- [`SHOW`](show-vars.html) -- [`SHOW SCHEMAS`](show-schemas.html) -- [SQL Name Resolution](sql-name-resolution.html) diff --git a/src/current/v21.1/table-expressions.md b/src/current/v21.1/table-expressions.md deleted file mode 100644 index ac86ec630ab..00000000000 --- a/src/current/v21.1/table-expressions.md +++ /dev/null @@ -1,409 +0,0 @@ ---- -title: Table Expressions -summary: Table expressions define a data source in selection clauses. -toc: true ---- - -Table expressions define a data source in the `FROM` sub-clause of -[simple `SELECT` clauses](select-clause.html), or as parameter to -[`TABLE`](selection-queries.html#table-clause). - -[SQL Joins](joins.html) are a particular kind of table -expression. - - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/table_ref.html %} -
    - -## Parameters - -Parameter | Description -----------|------------ -`table_name` | A [table or view name](#table-or-view-names). -`table_alias_name` | A name to use in an [aliased table expression](#aliased-table-expressions). -`name` | One or more aliases for the column names, to use in an [aliased table expression](#aliased-table-expressions). -`index_name` | Optional syntax to [force index selection](#force-index-selection). -`func_application` | [Results from a function](#results-from-a-function). -`row_source_extension_stmt` | [Result rows](#using-the-output-of-other-statements) from a [supported statement](sql-grammar.html#row_source_extension_stmt). -`select_stmt` | A [selection query](selection-queries.html) to use as [subquery](#subqueries-as-table-expressions). -`joined_table` | A [join expression](joins.html). - -## Table expressions language - -The synopsis above really defines a mini-language to construct -complex table expressions from simpler parts. - -Construct | Description | Examples -----------|-------------|------------ -`table_name [@ scan_parameters]` | [Access a table or view](#access-a-table-or-view). | `accounts`, `accounts@name_idx` -`function_name ( exprs ... )` | Generate tabular data using a [scalar function](#scalar-function-as-data-source) or [table generator function](#table-generator-functions). | `sin(1.2)`, `generate_series(1,10)` -`
    [AS] name [( name [, ...] )]` | [Rename a table and optionally columns](#aliased-table-expressions). | `accounts a`, `accounts AS a`, `accounts AS a(id, b)` -`
    WITH ORDINALITY` | [Enumerate the result rows](#ordinality-annotation). | `accounts WITH ORDINALITY` -`
    JOIN
    ON ...` | [Join expression](joins.html). | `orders o JOIN customers c ON o.customer_id = c.id` -`(... subquery ...)` | A [selection query](selection-queries.html) used as [subquery](#subqueries-as-table-expressions). | `(SELECT * FROM customers c)` -`[... statement ...]` | [Use the result rows](#using-the-output-of-other-statements) of an [explainable statement](sql-grammar.html#preparable_stmt).

    This is a CockroachDB extension. | `[SHOW COLUMNS FROM accounts]` - -The following sections provide details on each of these options. - -## Table expressions that generate data - -The following sections describe primary table expressions that produce -data. - -### Access a table or view - -#### Table or view names - -Syntax: - -~~~ -identifier -identifier.identifier -identifier.identifier.identifier -~~~ - -A single SQL identifier in a table expression context designates -the contents of the table, [view](views.html), or sequence with that name -in the current database, as configured by [`SET DATABASE`](set-vars.html). - -If the name is composed of two or more identifiers, [name resolution](sql-name-resolution.html) rules apply. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM users; -- uses table `users` in the current database -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM mydb.users; -- uses table `users` in database `mydb` -~~~ - -#### Force index selection - -{% include {{page.version.version}}/misc/force-index-selection.md %} - -{{site.data.alerts.callout_info}} -You can also force index selection for [`DELETE`](delete.html#force-index-selection-for-deletes) and [`UPDATE`](update.html#force-index-selection-for-updates) statements. -{{site.data.alerts.end}} - -### Access a common table expression - -A single identifier in a table expression context can refer to a -[common table expression](common-table-expressions.html) defined -earlier. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> WITH a AS (SELECT * FROM users) - SELECT * FROM a; -- "a" refers to "WITH a AS .." -~~~ - -### Results from a function - -A table expression can use the results from a function application as -a data source. - -Syntax: - -~~~ -name ( arguments... ) -~~~ - -The name of a function, followed by an opening parenthesis, followed -by zero or more [scalar expressions](scalar-expressions.html), followed by -a closing parenthesis. - -The resolution of the function name follows the same rules as the -resolution of table names. See [Name -Resolution](sql-name-resolution.html) for more details. - -#### Scalar function as data source - -When a [function returning a single -value](scalar-expressions.html#function-calls-and-sql-special-forms) is -used as a table expression, it is interpreted as tabular data with a -single column and single row containing the function results. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM sin(3.2) -~~~ -~~~ -+-----------------------+ -| sin | -+-----------------------+ -| -0.058374143427580086 | -+-----------------------+ -~~~ - -{{site.data.alerts.callout_info}}CockroachDB only supports this syntax for compatibility with PostgreSQL. The canonical syntax to evaluate scalar functions is as a direct target of SELECT, for example SELECT sin(3.2).{{site.data.alerts.end}} - - -#### Table generator functions - -Some functions directly generate tabular data with multiple rows from -a single function application. This is also called a "set-returning -function". - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM generate_series(1, 3); -~~~ -~~~ -+-----------------+ -| generate_series | -+-----------------+ -| 1 | -| 2 | -| 3 | -+-----------------+ -~~~ - -Set-returning functions (SRFs) can now be accessed using `(SRF).x` where `x` is one of the following: - -- The name of a column returned from the function -- `*` to denote all columns. - -For example (note that the output of queries against [`information_schema`](information-schema.html) will vary per database): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT (i.keys).* FROM (SELECT information_schema._pg_expandarray(indkey) AS keys FROM pg_index) AS i; -~~~ - -~~~ - x | n ----+--- - 1 | 1 - 2 | 1 -(2 rows) -~~~ - -{{site.data.alerts.callout_info}} -Currently CockroachDB only supports a small set of generator functions compatible with [the PostgreSQL set-generating functions with the same -names](https://www.postgresql.org/docs/9.6/static/functions-srf.html). -{{site.data.alerts.end}} - -## Operators that extend a table expression - -The following sections describe table expressions that change the -metadata around tabular data, or add more data, without modifying the -data of the underlying operand. - -### Aliased table expressions - -Aliased table expressions rename tables and columns temporarily in -the context of the current query. - -Syntax: - -~~~ -
    AS -
    AS (, , ...) -~~~ - -In the first form, the table expression is equivalent to its left operand -with a new name for the entire table, and where columns retain their original name. - -In the second form, the columns are also renamed. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT c.x FROM (SELECT COUNT(*) AS x FROM users) AS c; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT c.x FROM (SELECT COUNT(*) FROM users) AS c(x); -~~~ - -### Ordinality annotation - -Syntax: - -~~~ -
    WITH ORDINALITY -~~~ - -Designates a data source equivalent to the table expression operand with -an extra "Ordinality" column that enumerates every row in the data source. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM (VALUES('a'),('b'),('c')); -~~~ -~~~ -+---------+ -| column1 | -+---------+ -| a | -| b | -| c | -+---------+ -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM (VALUES ('a'), ('b'), ('c')) WITH ORDINALITY; -~~~ -~~~ -+---------+------------+ -| column1 | ordinality | -+---------+------------+ -| a | 1 | -| b | 2 | -| c | 3 | -+---------+------------+ -~~~ - -{{site.data.alerts.callout_info}} -`WITH ORDINALITY` necessarily prevents some optimizations of the surrounding query. Use it sparingly if performance is a concern, and always check the output of [`EXPLAIN`](explain.html) in case of doubt. -{{site.data.alerts.end}} - -## `JOIN` expressions - -`JOIN` expressions combine the results of two or more table expressions -based on conditions on the values of particular columns. - -See [`JOIN` expressions](joins.html) for more details. - -## Using other queries as table expressions - -The following sections describe how to use the results produced by -another SQL query or statement as a table expression. - -### Subqueries as table expressions - -Any [selection -query](selection-queries.html) enclosed -between parentheses can be used as a table expression, including -[simple `SELECT` clauses](select-clause.html). This is called a -"[subquery](subqueries.html)". - -Syntax: - -~~~ -( ... subquery ... ) -~~~ - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT c+2 FROM (SELECT COUNT(*) AS c FROM users); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM (VALUES(1), (2), (3)); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT firstname || ' ' || lastname FROM (TABLE employees); -~~~ - -{{site.data.alerts.callout_info}} -- See also [Subqueries](subqueries.html) for more details and performance best practices. -- To use other statements that produce data in a table expression, for example `SHOW`, use the [square bracket notation](#using-the-output-of-other-statements). -{{site.data.alerts.end}} - -
    - -### Using the output of other statements - -Syntax: - -~~~ -SELECT .. FROM [ ] -~~~ - -A [statement](sql-grammar.html#row_source_extension_stmt) between square brackets in a table expression context designates the output of executing the statement as a row source. The following statements are supported as row sources for table expressions: - -- [`DELETE`](delete.html) -- [`EXPLAIN`](explain.html) -- [`INSERT`](insert.html) -- [`SELECT`](select-clause.html) -- [`SHOW`](sql-statements.html#data-definition-statements) -- [`UPDATE`](update.html) -- [`UPSERT`](upsert.html) - - `SELECT .. FROM [ ]` is equivalent to `WITH table_expr AS ( ) SELECT .. FROM table_expr` - -{{site.data.alerts.callout_info}} -This CockroachDB extension syntax complements the [subquery syntax using parentheses](#subqueries-as-table-expressions), which is restricted to [selection queries](selection-queries.html). It was introduced to enable the use of [statements](sql-grammar.html#row_source_extension_stmt) as [subqueries](subqueries.html). -{{site.data.alerts.end}} - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT "column_name" FROM [SHOW COLUMNS FROM customer]; -~~~ - -~~~ -+-------------+ -| column_name | -+-------------+ -| id | -| name | -| address | -+-------------+ -(3 rows) -~~~ - -The following statement inserts Albert in the `employee` table and -immediately creates a matching entry in the `management` table with the -auto-generated employee ID, without requiring a round trip with the SQL -client: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO management(manager, reportee) - VALUES ((SELECT id FROM employee WHERE name = 'Diana'), - (SELECT id FROM [INSERT INTO employee(name) VALUES ('Albert') RETURNING id])); -~~~ - -## Composability - -Table expressions are used in the [`SELECT`](select-clause.html) and -[`TABLE`](selection-queries.html#table-clause) variants of [selection -clauses](selection-queries.html#selection-clauses), and thus can appear everywhere where -a selection clause is possible. For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT ... FROM
    ,
    , ... -> TABLE
    -> INSERT INTO ... SELECT ... FROM
    ,
    , ... -> INSERT INTO ... TABLE
    -> CREATE TABLE ... AS SELECT ... FROM
    ,
    , ... -> UPSERT INTO ... SELECT ... FROM
    ,
    , ... -~~~ - -For more options to compose query results, see [Selection Queries](selection-queries.html). - -## See also - -- [Constants](sql-constants.html) -- [Selection Queries](selection-queries.html) - - [Selection Clauses](selection-queries.html#selection-clauses) -- [Explainable Statements](sql-grammar.html#preparable_stmt) -- [Scalar Expressions](scalar-expressions.html) -- [Data Types](data-types.html) -- [Subqueries](subqueries.html) diff --git a/src/current/v21.1/take-and-restore-encrypted-backups.md b/src/current/v21.1/take-and-restore-encrypted-backups.md deleted file mode 100644 index 81fb0ca4653..00000000000 --- a/src/current/v21.1/take-and-restore-encrypted-backups.md +++ /dev/null @@ -1,174 +0,0 @@ ---- -title: Take and Restore Encrypted Backups -summary: Learn about the advanced options you can use when you backup and restore a CockroachDB cluster. -toc: true ---- - -This doc provides information about how to take and restore encrypted backups in the following ways: - -- [Using AWS Key Management Service (KMS)](#use-aws-key-management-service) -- [Using a passphrase](#use-a-passphrase) - -{{site.data.alerts.callout_info}} -Encrypted [`BACKUP`](backup.html) is an [Enterprise-only](https://www.cockroachlabs.com/product/cockroachdb/) feature. However, you can take [full backups](take-full-and-incremental-backups.html) without an Enterprise license. -{{site.data.alerts.end}} - -## Use AWS Key Management Service - - You can encrypt full or incremental backups with AWS Key Management Service (KMS) by using the [`kms` option](backup.html#options). Files written by the backup (`BACKUP` manifest and data files) are encrypted using a 256-bit crypto-random generated data key. This data key is encrypted with the provided AWS KMS URI(s) and stored alongside the `BACKUP` data in an `ENCRYPTION_INFO` file, which is used when restoring the backed-up data. - -On [`RESTORE`](#restore-from-an-encrypted-backup-with-aws-kms), CockroachDB reads the `ENCRYPTION_INFO` file and attempts to decrypt the encrypted data key using the KMS URI provided in the `RESTORE` statement. Once CockroachDB successfully obtains the unencrypted data key, the `BACKUP` manifest and data files will be decrypted and the restoration will proceed. Similarly, the same KMS URI is needed to decrypt the file to list the contents of the backup when using [`SHOW BACKUP`](show-backup.html#show-an-encrypted-backup). - -When used with [incremental backups](take-full-and-incremental-backups.html#incremental-backups), the `kms` option is applied to all the [backup file URLs](backup.html#backup-file-urls), which means each incremental must include at least one of the KMS URIs used to take the full backup. It can be any subset of the original URIs, but you cannot include any new KMS URIs. Similarly, when used with [locality-aware backups](take-and-restore-locality-aware-backups.html), the KMS URI provided is applied to files in all localities. - -For more information about AWS KMS, see the [documentation](https://aws.amazon.com/kms/). - -### Generate a customer master key (CMK) - -Before you can use AWS KMS to encrypt a CockroachDB backup, you must first generate a **customer master key (CMK)**. This is the master key generated by AWS KMS and it never leaves the KMS. It contains key-related metadata and key material to encrypt/decrypt other data. The key material can never be exported, deleted, or extracted. CockroachDB expects the CMK to be symmetric (256 bit). - - There is also the option to [use multi-region encryption on your data](#take-a-backup-with-multi-region-encryption). At the time of `BACKUP`, you can [provide multiple AWS KMS URIs](#aws-kms-uri-format), each referencing a CMK in a different AWS region. This allows CockroachDB to save multiple versions of the encrypted data key used to encrypt the backup data, one per AWS KMS URI. With these encrypted versions of the data key stored alongside the encrypted backup data, a user can `RESTORE` the encrypted data using any one of the KMS URIs that were supplied during backup. In the case of a single KMS region outage, the data can be decrypted with any of the CMKs from the other regions. - -### AWS KMS URI format - -The AWS KMS URI must use the following format: - -~~~ -aws:///?AUTH=®ION= -~~~ - -The AWS URI **requires** the following: - - Component | Description -----------------------------+------------------------------------------------------------------------ -`aws:///` | The AWS scheme. Note the triple slash (`///`). -`` | The key identifiers used to reference the CMK that should be used to encrypt or decrypt. For information about the supported formats, see the [AWS KMS docs](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#key-id). -`AUTH=` | The user-specified credentials.
    • If the `AUTH` parameter is not provided, the AWS connections default to `specified` and the access keys must be provided in the URI parameters (e.g., `AWS_ACCESS_KEY_ID=&AWS_SECRET_ACCESS_KEY=`).
    • If `AUTH=implicit`, the access keys can be omitted and the [credentials will be loaded from the environment](https://docs.aws.amazon.com/sdk-for-go/api/aws/session/).
    -`REGION=` | The region of the CMK. - -See below for [examples](#examples). - -### Examples - -- [Take an encrypted backup with AWS KMS](#take-an-encrypted-backup-with-aws-kms) -- [Take a backup with multi-region encryption](#take-a-backup-with-multi-region-encryption) -- [Restore from an encrypted backup with AWS KMS](#restore-from-an-encrypted-backup-with-aws-kms) - -#### Take an encrypted backup with AWS KMS - -To take an encrypted backup with AWS KMS, use the `kms` [option](backup.html#options): - -{% include_cached copy-clipboard.html %} -~~~ sql -> BACKUP INTO 's3://{BUCKET NAME}/{PATH}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}' - WITH kms = 'aws:///?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}®ION=us-east-1'; -~~~ - -~~~ - job_id | status | fraction_completed | rows | index_entries | bytes ----------------------+-----------+--------------------+------+---------------+---------- - 594193600274956289 | succeeded | 1 | 2689 | 1217 | 1420108 -(1 row) -~~~ - -#### Take a backup with multi-region encryption - -To take a backup with [multi-region encryption](#multi-region), use the `kms` option to specify a comma-separated list of KMS URIs: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BACKUP INTO 's3://{BUCKET NAME}/{PATH}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}' - WITH KMS=( - 'aws:///?AUTH=implicit®ION=us-east-1', - 'aws:///?AUTH=implict®ION=us-west-1' - ); -~~~ - -~~~ - job_id | status | fraction_completed | rows | index_entries | bytes ----------------------+-----------+--------------------+------+---------------+-------- - 594471427115220993 | succeeded | 1 | 20 | 2 | 1026 -(1 row) -~~~ - -#### Restore from an encrypted backup with AWS KMS - -To decrypt an [encrypted backup](#take-an-encrypted-backup-with-aws-kms), use the `kms` option and any subset of the KMS URIs that was used to take the backup. - -For example, the encrypted backup created in the [first example](#take-an-encrypted-backup-with-aws-kms) can be restored with: - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESTORE FROM LATEST IN 's3://{BUCKET NAME}/{PATH}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}' - WITH kms = 'aws:///?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}®ION=us-east-1'; -~~~ - -~~~ - job_id | status | fraction_completed | rows | index_entries | bytes ----------------------+-----------+--------------------+------+---------------+---------- - 594193600274956291 | succeeded | 1 | 2689 | 1217 | 1420108 -(1 row) -~~~ - -To restore from a specific backup, use [`RESTORE FROM {subdirectory} IN ...`](restore.html#restore-a-specific-backup). - -## Use a passphrase - -{% include {{ page.version.version }}/backups/encrypted-backup-description.md %} - -### Examples - -{% include {{ page.version.version }}/backups/bulk-auth-options.md %} - -#### Take an encrypted backup using a passphrase - -To take an encrypted backup, use the [`encryption_passphrase` option](backup.html#with-encryption-passphrase): - -{% include_cached copy-clipboard.html %} -~~~ sql -> BACKUP INTO 's3://{BUCKET NAME}/{PATH}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}' WITH encryption_passphrase = 'password123'; -~~~ -~~~ - job_id | status | fraction_completed | rows | index_entries | bytes ----------------------+-----------+--------------------+------+---------------+--------- - 543214409874014209 | succeeded | 1 | 2597 | 1028 | 467701 -(1 row) -~~~ - -To [restore](restore.html), use the same `encryption_passphrase`. See the [example](#restore-from-an-encrypted-backup-using-a-passphrase) below for more details. - -#### Restore from an encrypted backup using a passphrase - -To decrypt an encrypted backup, use the [`encryption_passphrase` option](backup.html#with-encryption-passphrase) option and the same passphrase that was used to create the backup. - -For example, the encrypted backup created in the previous example can be restored with: - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESTORE FROM LATEST IN 's3://{BUCKET NAME}/{PATH}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}' WITH encryption_passphrase = 'password123'; -~~~ -~~~ - job_id | status | fraction_completed | rows | index_entries | bytes ----------------------+-----------+--------------------+------+---------------+--------- - 543217488273801217 | succeeded | 1 | 2597 | 1028 | 467701 -(1 row) -~~~ - -To restore from a specific backup, use [`RESTORE FROM {subdirectory} IN ...`](restore.html#restore-a-specific-backup). - -## See also - -- [`BACKUP`][backup] -- [`RESTORE`][restore] -- [Take Full and Incremental Backups](take-full-and-incremental-backups.html) -- [Take and Restore Locality-aware Backups](take-and-restore-locality-aware-backups.html) -- [Take Backups with Revision History and Restore from a Point-in-time](take-backups-with-revision-history-and-restore-from-a-point-in-time.html) -- [`SQL DUMP`](cockroach-dump.html) -- [`IMPORT`](migration-overview.html) -- [Use the Built-in SQL Client](cockroach-sql.html) -- [Other Cockroach Commands](cockroach-commands.html) - - - -[backup]: backup.html -[restore]: restore.html diff --git a/src/current/v21.1/take-and-restore-locality-aware-backups.md b/src/current/v21.1/take-and-restore-locality-aware-backups.md deleted file mode 100644 index a893f7841b0..00000000000 --- a/src/current/v21.1/take-and-restore-locality-aware-backups.md +++ /dev/null @@ -1,219 +0,0 @@ ---- -title: Take and Restore Locality-aware Backups -summary: Learn about the advanced options you can use when you backup and restore a CockroachDB cluster. -toc: true ---- - -This page provides information about how to take and restore locality-aware backups. - -{{site.data.alerts.callout_info}} -Locality-aware [`BACKUP`](backup.html) is an [Enterprise-only](https://www.cockroachlabs.com/product/cockroachdb/) feature. However, you can take [full backups](take-full-and-incremental-backups.html) without an Enterprise license. -{{site.data.alerts.end}} - -You can create locality-aware backups such that each node writes files only to the backup destination that matches the [node locality](configure-replication-zones.html#descriptive-attributes-assigned-to-nodes) configured at [node startup](cockroach-start.html). - -This is useful for: - -- Reducing cloud storage data transfer costs by keeping data within cloud regions. -- Helping you comply with data domiciling requirements. - -A locality-aware backup is specified by a list of URIs, each of which has a `COCKROACH_LOCALITY` URL parameter whose single value is either `default` or a single locality key-value pair such as `region=us-east`. At least one `COCKROACH_LOCALITY` must be the `default`. Given a list of URIs that together contain the locations of all of the files for a single locality-aware backup, [`RESTORE` can read in that backup](#restore-from-a-locality-aware-backup). - -{{site.data.alerts.callout_info}} -The locality query string parameters must be [URL-encoded](https://en.wikipedia.org/wiki/Percent-encoding). -{{site.data.alerts.end}} - -Since each node is responsible for backing up the ranges for which it is the [leaseholder](architecture/replication-layer.html#leases), the locality of the leaseholder node determines where the backup files will be placed in a locality-aware backup. The leaseholder node writes files to the backup storage location with a locality that matches its own localities (with an overall preference for more specific values in the locality hierarchy). If there is no match, the `default` locality is used. - -## Create a locality-aware backup - -For example, to create a locality-aware backup where nodes with the locality `region=us-west` write backup files to `s3://us-west-bucket`, and all other nodes write to `s3://us-east-bucket` by default, run: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BACKUP INTO - ('s3://us-east-bucket?COCKROACH_LOCALITY=default', 's3://us-west-bucket?COCKROACH_LOCALITY=region%3Dus-west'); -~~~ - -You can restore this backup by running: - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESTORE FROM LATEST IN ('s3://us-east-bucket', 's3://us-west-bucket'); -~~~ - -Note that the first URI in the list has to be the URI specified as the `default` URI when the backup was created. If you have moved your backups to a different location since the backup was originally taken, the first URI must be the new location of the files originally written to the `default` location. - -To restore from a specific backup, use [`RESTORE FROM {subdirectory} IN ...`](restore.html#restore-a-specific-backup). - -For guidance on how to identify the locality of a node to pass in a backup query, see [Show a node's locality](#show-a-nodes-locality). - -{{site.data.alerts.callout_info}} -For guidance on connecting to other storage options or using other authentication parameters, read [Use Cloud Storage for Bulk Operations](use-cloud-storage-for-bulk-operations.html). -{{site.data.alerts.end}} - -## Show a node's locality - -To determine the locality that a node was started with, run [`SHOW LOCALITY`](show-locality.html): - -{% include_cached copy-clipboard.html %} -~~~sql -SHOW LOCALITY; -~~~ - -~~~ - locality -+---------------------+ -region=us-east,az=az1 -(1 row) -~~~ - -The output shows the locality to which the node will write backup data. One of the single locality key-value pairs can be passed to `BACKUP` with the `COCKROACH_LOCALITY` parameter (e.g., `'s3://us-east-bucket?COCKROACH_LOCALITY=region%3Dus-east'`). - -{{site.data.alerts.callout_info}} -Specifying both locality tier pairs (e.g., `region=us-east,az=az1`) from the output will cause the backup query to fail with: `tier must be in the form "key=value"`. -{{site.data.alerts.end}} - -## Restore from a locality-aware backup - -Given a list of URIs that together contain the locations of all of the files for a single [locality-aware backup](#create-a-locality-aware-backup), [`RESTORE`](restore.html) can read in that backup. Note that the list of URIs passed to [`RESTORE`](restore.html) may be different from the URIs originally passed to [`BACKUP`](backup.html). This is because it's possible to move the contents of one of the parts of a locality-aware backup (i.e., the files written to that destination) to a different location, or even to consolidate all the files for a locality-aware backup into a single location. - -When restoring a [full backup](take-full-and-incremental-backups.html#full-backups), the cluster data is restored first, then the system table data "as is." This means that the restored zone configurations can point to regions that do not have active nodes in the new cluster. For example, if your full backup has the following [zone configurations](configure-zone.html): - -~~~ sql -> ALTER PARTITION europe_west OF INDEX movr.public.rides@primary \ - CONFIGURE ZONE USING constraints = '[+region=europe-west1]'; - -> ALTER PARTITION us_east OF INDEX movr.public.rides@primary \ - CONFIGURE ZONE USING constraints = '[+region=us-east1]'; - -> ALTER PARTITION us_west OF INDEX movr.public.rides@primary \ - CONFIGURE ZONE USING constraints = '[+region=us-west1]'; -~~~ - -And the restored cluster does not have [nodes with the locality](partitioning.html#node-attributes) `region=us-west1`, the restored cluster will still have a zone configuration for `us-west1`. This means that the cluster's data will **not** be reshuffled to `us-west1` because the region does not exist. The data will be distributed as if the zone configuration does not exist. For the data to be distributed correctly, you can [add node(s)](cockroach-start.html) with the missing region or [remove the zone configuration](configure-zone.html#remove-a-replication-zone). - -For example, use the following to create a locality-aware backup: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BACKUP INTO - ('s3://us-east-bucket?COCKROACH_LOCALITY=default', 's3://us-west-bucket?COCKROACH_LOCALITY=region%3Dus-west') -~~~ - -Restore a locality-aware backup with: - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESTORE FROM LATEST IN ('s3://us-east-bucket/', 's3://us-west-bucket/'); -~~~ - -To restore from a specific backup, use [`RESTORE FROM {subdirectory} IN ...`](restore.html#restore-a-specific-backup). - -{{site.data.alerts.callout_info}} -[`RESTORE`](restore.html) is not truly locality-aware; while restoring from backups, a node may read from a store that does not match its locality. This can happen in the cases that either the [`BACKUP`](backup.html) or [`RESTORE`](restore.html) was not of a [full cluster](take-full-and-incremental-backups.html#full-backups). Note that during a locality-aware restore, some data may be temporarily located on another node before it is eventually relocated to the appropriate node. To avoid this, you can [manually restore zone configurations from a locality-aware backup](#manually-restore-zone-configurations-from-a-locality-aware-backup). -{{site.data.alerts.end}} - -## Create an incremental locality-aware backup - -If you backup to a destination already containing a [full backup](take-full-and-incremental-backups.html#full-backups), an [incremental backup](take-full-and-incremental-backups.html#incremental-backups) will be appended to the full backup in a subdirectory. For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BACKUP INTO LATEST IN - ('s3://us-east-bucket?COCKROACH_LOCALITY=default', 's3://us-west-bucket?COCKROACH_LOCALITY=region%3Dus-west'); -~~~ - -{{site.data.alerts.callout_info}} -When [restoring from an incremental locality-aware backup](#restore-from-an-incremental-locality-aware-backup), you need to include **every** locality ever used, even if it was only used once. - -It is recommended that the same localities be included for every incremental backup in the series of backups; however, only the `default` locality is required. -{{site.data.alerts.end}} - -And if you want to explicitly control where your incremental backups go, specify the subdirectory in the `BACKUP` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BACKUP INTO {subdirectory} IN - ('s3://us-east-bucket?COCKROACH_LOCALITY=default', 's3://us-west-bucket?COCKROACH_LOCALITY=region%3Dus-west'); -~~~ - -To view the available subdirectories, use [`SHOW BACKUPS`](restore.html#view-the-backup-subdirectories). - -## Restore from an incremental locality-aware backup - -A locality-aware backup URI can also be used in place of any incremental backup URI in [`RESTORE`](restore.html). - -For example, an incremental locality-aware backup created with: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BACKUP INTO LATEST IN - ('s3://us-east-bucket?COCKROACH_LOCALITY=default', 's3://us-west-bucket?COCKROACH_LOCALITY=region%3Dus-west') -~~~ - -can be restored by running: - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESTORE FROM LATEST IN ('s3://us-east-bucket/', 's3://us-west-bucket/'); -~~~ - -To restore from a specific backup, use [`RESTORE FROM {subdirectory} IN ...`](restore.html#restore-a-specific-backup). - -{{site.data.alerts.callout_info}} -When [restoring from an incremental locality-aware backup](take-and-restore-locality-aware-backups.html#restore-from-an-incremental-locality-aware-backup), you need to include **every** locality ever used, even if it was only used once. -{{site.data.alerts.end}} - -## Manually restore zone configurations from a locality-aware backup - -During a [locality-aware restore](#restore-from-a-locality-aware-backup), some data may be temporarily located on another node before it is eventually relocated to the appropriate node. To avoid this, you need to manually restore [zone configurations](configure-replication-zones.html) first: - -Once the locality-aware restore has started, [pause the restore](pause-job.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> PAUSE JOB 27536791415282; -~~~ - -The `system.zones` table stores your cluster's [zone configurations](configure-replication-zones.html), which will prevent the data from rebalancing. To restore them, you must restore the `system.zones` table into a new database because you cannot drop the existing `system.zones` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESTORE TABLE system.zones FROM '2021/03/23-213101.37' IN - 'azure://acme-co-backup?AZURE_ACCOUNT_KEY=hash&AZURE_ACCOUNT_NAME=acme-co' - WITH into_db = 'newdb'; -~~~ - -After it's restored into a new database, you can write the restored `zones` table data to the cluster's existing `system.zones` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO system.zones SELECT * FROM newdb.zones; -~~~ - -Then drop the temporary table you created: - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP TABLE newdb.zones; -~~~ - -Then, [resume the restore](resume-job.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESUME JOB 27536791415282; -~~~ - -## See also - -- [`BACKUP`](backup.html) -- [`RESTORE`](restore.html) -- [Take Full and Incremental Backups](take-full-and-incremental-backups.html) -- [Take and Restore Encrypted Backups](take-and-restore-encrypted-backups.html) -- [Take Backups with Revision History and Restore from a Point-in-time](take-backups-with-revision-history-and-restore-from-a-point-in-time.html) -- [`SQL DUMP`](cockroach-dump.html) -- [`IMPORT`](migration-overview.html) -- [Use the Built-in SQL Client](cockroach-sql.html) -- [Other Cockroach Commands](cockroach-commands.html) diff --git a/src/current/v21.1/take-backups-with-revision-history-and-restore-from-a-point-in-time.md b/src/current/v21.1/take-backups-with-revision-history-and-restore-from-a-point-in-time.md deleted file mode 100644 index 37f05613c3f..00000000000 --- a/src/current/v21.1/take-backups-with-revision-history-and-restore-from-a-point-in-time.md +++ /dev/null @@ -1,59 +0,0 @@ ---- -title: Take Backups with Revision History and Restore from a Point-in-time -summary: Learn about the advanced options you can use when you backup and restore a CockroachDB cluster. -toc: true ---- - -This doc provides information about how to take backups with revision history and restore from a point-in-time. - -{{site.data.alerts.callout_info}} -[`BACKUP`](backup.html) with revision history is an [Enterprise-only](https://www.cockroachlabs.com/product/cockroachdb/) feature. However, you can take [full backups](take-full-and-incremental-backups.html) without an Enterprise license. -{{site.data.alerts.end}} - -You can create full or incremental backups [with revision history](backup.html#with-revision-history): - -- Taking full backups with revision history allows you to back up every change made within the garbage collection period leading up to and including the given timestamp. -- Taking incremental backups with revision history allows you to back up every change made since the last backup and within the garbage collection period leading up to and including the given timestamp. You can take incremental backups with revision history even when your previous full or incremental backups were taken without revision history. - -You can configure garbage collection periods using the `ttlseconds` [replication zone setting](configure-replication-zones.html). Taking backups with revision history allows for point-in-time restores within the revision history. - -## Create a backup with revision history - -{% include_cached copy-clipboard.html %} -~~~ sql -> BACKUP INTO '{destination}' AS OF SYSTEM TIME '-10s' WITH revision_history; -~~~ - -For guidance on connecting to Amazon S3, Google Cloud Storage, Azure Storage, and other storage options, read [Use Cloud Storage for Bulk Operations](use-cloud-storage-for-bulk-operations.html). - -## Point-in-time restore - -If the full or incremental backup was taken [with revision history](#create-a-backup-with-revision-history), you can restore the data as it existed at an arbitrary point-in-time within the revision history captured by that backup. Use the [`AS OF SYSTEM TIME`](as-of-system-time.html) clause to specify the point-in-time. - -Additionally, if you want to restore a specific incremental backup, you can do so by specifying the `end_time` of the backup by using the [`AS OF SYSTEM TIME`](as-of-system-time.html) clause. To find the incremental backup's `end_time`, use [`SHOW BACKUP`](show-backup.html). - -If you do not specify a point-in-time, the data will be restored to the backup timestamp; that is, the restore will work as if the data was backed up without revision history. - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESTORE FROM '/2021/12/13-211056.62' IN '(destination)' AS OF SYSTEM TIME '2021-12-13 10:00:00'; -~~~ - -To view the available backup subdirectories you can restore from, use [`SHOW BACKUPS`](restore.html#view-the-backup-subdirectories). - -## See also - -- [`BACKUP`][backup] -- [`RESTORE`][restore] -- [Take Full and Incremental Backups](take-full-and-incremental-backups.html) -- [Take and Restore Encrypted Backups](take-and-restore-encrypted-backups.html) -- [Take and Restore Locality-aware Backups](take-and-restore-locality-aware-backups.html) -- [`SQL DUMP`](cockroach-dump.html) -- [`IMPORT`](migration-overview.html) -- [Use the Built-in SQL Client](cockroach-sql.html) -- [Other Cockroach Commands](cockroach-commands.html) - - - -[backup]: backup.html -[restore]: restore.html diff --git a/src/current/v21.1/take-full-and-incremental-backups.md b/src/current/v21.1/take-full-and-incremental-backups.md deleted file mode 100644 index a219bdb331f..00000000000 --- a/src/current/v21.1/take-full-and-incremental-backups.md +++ /dev/null @@ -1,236 +0,0 @@ ---- -title: Take Full and Incremental Backups -summary: Learn how to back up and restore a CockroachDB cluster. -toc: true ---- - -Because CockroachDB is designed with high fault tolerance, backups are primarily needed for [disaster recovery](disaster-recovery.html) (i.e., if your cluster loses a majority of its nodes). Isolated issues (such as small-scale node outages) do not require any intervention. However, as an operational best practice, **we recommend taking regular backups of your data**. - -There are two main types of backups: - -- [Full backups](#full-backups) -- [Incremental backups](#incremental-backups) - -You can use the [`BACKUP`](backup.html) statement to efficiently back up your cluster's schemas and data to popular cloud services such as AWS S3, Google Cloud Storage, or NFS, and the [`RESTORE`](restore.html) statement to efficiently restore schema and data as necessary. For more information, see [Use Cloud Storage for Bulk Operations](use-cloud-storage-for-bulk-operations.html). - -{{site.data.alerts.callout_success}} - You can create [schedules for periodic backups](manage-a-backup-schedule.html) in CockroachDB. We recommend using scheduled backups to automate daily backups of your cluster. -{{site.data.alerts.end}} - -## Backup collections - -A _backup collection_ defines a set of backups and their metadata. The collection can contain multiple full backups and their subsequent [incremental backups](#incremental-backups). The path to a backup is created using a date-based naming scheme and stored at the URI passed with the `BACKUP` statement. - -In the following example, a user has taken weekly full backups and nightly incremental backups to their `collectionURI`: - -~~~ -Collection: -|—— 2022 - |—— 02 - |—— 09-155340.13/ - |—— Full backup files - |—— 20220210/ - |—— 155530.50/ - |—— Incremental backup files - |—— 20220211/ - |—— 155628.07/ - |—— Incremental backup files - [...] - |—— 16-143018.72/ - |—— Full backup files - |—— 20220217/ - |—— 155530.50/ - |—— Incremental backup files - |—— 20220218/ - |—— 155628.07/ - |—— Incremental backup files - [...] -~~~ - -[`SHOW BACKUPS IN {collectionURI}`](show-backup.html#view-a-list-of-the-available-full-backup-subdirectories) will display a list of the full backup subdirectories in the collection's storage location. - -A [locality-aware backup](take-and-restore-locality-aware-backups.html) is a specific case where part of the collection data is stored at a different URI. The backup collection will be stored according to the URIs passed with the `BACKUP` statement: `BACKUP INTO LATEST IN {collectionURI}, {localityURI}, {localityURI}`. Here, the `collectionURI` represents the default locality. - -In the examples on this page, `{collectionURI}` is a placeholder for the storage location that will contain the example backup. - -## Full backups - -Full backups are now available to both core and Enterprise users. - -Full backups contain an un-replicated copy of your data and can always be used to restore your cluster. These files are roughly the size of your data and require greater resources to produce than incremental backups. You can take full backups as of a given timestamp. Optionally, you can include the available [revision history](take-backups-with-revision-history-and-restore-from-a-point-in-time.html) in the backup. - -In most cases, **it's recommended to take nightly full backups of your cluster**. A cluster backup allows you to do the following: - -- Restore table(s) from the cluster -- Restore database(s) from the cluster -- Restore a full cluster - -Backups will export [Enterprise license keys](enterprise-licensing.html) during a [full cluster backup](backup.html#backup-a-cluster). When you [restore](restore.html) a full cluster with an Enterprise license, it will restore the Enterprise license of the cluster you are restoring from. - -### Take a full backup - -To perform a full cluster backup, use the [`BACKUP`](backup.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BACKUP INTO '{collectionURI}'; -~~~ - -To restore a backup, use the [`RESTORE`][restore.html] statement, specifying what you want to restore as well as the [collection's](#backup-collections) URI: - -- To restore the latest backup of a table: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > RESTORE TABLE bank.customers FROM LATEST IN '{collectionURI}'; - ~~~ - -- To restore the latest backup of a database: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > RESTORE DATABASE bank FROM LATEST IN '{collectionURI}'; - ~~~ - -- To restore the latest backup of your full cluster: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > RESTORE FROM LATEST IN '{collectionURI}'; - ~~~ - - {{site.data.alerts.callout_info}} - A full cluster restore can only be run on a target cluster that has **never** had user-created databases or tables. - {{site.data.alerts.end}} - -- To restore a backup from a specific subdirectory: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > RESTORE DATABASE bank FROM {subdirectory} IN '{collectionURI}'; - ~~~ - -To view the available backup subdirectories, use [`SHOW BACKUPS`](show-backup.html). - -## Incremental backups - -{{site.data.alerts.callout_info}} -To take incremental backups, you need an [Enterprise license](enterprise-licensing.html). -{{site.data.alerts.end}} - -If your cluster grows too large for nightly [full backups](#full-backups), you can take less frequent full backups (e.g., weekly) with nightly incremental backups. Incremental backups are storage efficient and faster than full backups for larger clusters. - -Incremental backups are smaller and faster to produce than full backups because they contain only the data that has changed since a base set of backups you specify (which must include one full backup, and can include many incremental backups). You can take incremental backups either as of a given timestamp or with full [revision history](take-backups-with-revision-history-and-restore-from-a-point-in-time.html). - -{{site.data.alerts.callout_danger}} -Incremental backups can only be created within the [garbage collection](architecture/storage-layer.html#garbage-collection) period of the base backup's most recent timestamp. This is because incremental backups are created by finding which data has been created or modified since the most recent timestamp in the base backup—that timestamp data, though, is deleted by the garbage collection process. - -You can configure garbage collection periods using the `ttlseconds` [replication zone setting](configure-replication-zones.html#gc-ttlseconds). - -If an incremental backup is created outside of the garbage collection period, you will receive a `protected ts verification error…`. To resolve this issue, see the [Common Errors](common-errors.html#protected-ts-verification-error) page. -{{site.data.alerts.end}} - -### Take an incremental backup - -Periodically run the [`BACKUP`][backup] command to take a full backup of your cluster: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BACKUP INTO '{collectionURI}'; -~~~ - -Then, create nightly incremental backups based off of the full backups you've already created. To append an incremental backup to the most recent full backup created in the given destination, use `LATEST`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BACKUP INTO LATEST IN '{collectionURI}'; -~~~ - -For an example on how to specify the destination of an incremental backup, see [Incremental backups with explicitly specified destinations](#incremental-backups-with-explicitly-specified-destinations). - -If it's ever necessary, you can then use the [`RESTORE`][restore.html] command to restore your cluster, database(s), and/or table(s). Restoring from incremental backups requires previous full and incremental backups. - -To restore from the most recent backup, run the following: - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESTORE FROM LATEST IN '{collectionURI}'; -~~~ - -To restore a specific backup, run `RESTORE` with the backup's subdirectory: - -{% include_cached copy-clipboard.html %} -~~~ sql -> RESTORE FROM '{subdirectory}' IN '{collectionURI}'; -~~~ - -{{site.data.alerts.callout_info}} -{% include_cached new-in.html version="v21.1" %} `RESTORE` will re-validate [indexes](indexes.html) when [incremental backups](take-full-and-incremental-backups.html) are created from an older version (v20.2.2 and earlier or v20.1.4 and earlier), but restored by a newer version (v21.1.0+). These earlier releases may have included incomplete data for indexes that were in the process of being created. -{{site.data.alerts.end}} - -## Incremental backups with explicitly specified destinations - -To explicitly control where your incremental backups go, use the [`INTO {subdirectory} IN (destination)`](backup.html#synopsis) syntax: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BACKUP DATABASE bank INTO '{subdirectory}' IN '{collectionURI}' \ - AS OF SYSTEM TIME '-10s' \ - WITH revision_history; -~~~ - -A full backup must be present in the `destination` for an incremental backup to be stored in a `subdirectory`. If there isn't a full backup present in the `destination` when taking an incremental backup, one will be taken and stored in the `destination`. - -{{site.data.alerts.callout_info}} -To take incremental backups, you need an [Enterprise license](enterprise-licensing.html). -{{site.data.alerts.end}} - -## Examples - -{% include {{ page.version.version }}/backups/bulk-auth-options.md %} - -### Automated full backups - -Both core and Enterprise users can use backup scheduling for full backups of clusters, databases, or tables. To create schedules that only take full backups, include the `FULL BACKUP ALWAYS` clause. For example, to create a schedule for taking full cluster backups: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE SCHEDULE core_schedule_label - FOR BACKUP INTO 's3://{BUCKET NAME}/{PATH}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}' - RECURRING '@daily' - FULL BACKUP ALWAYS - WITH SCHEDULE OPTIONS first_run = 'now'; -~~~ -~~~ - schedule_id | name | status | first_run | schedule | backup_stmt ----------------------+---------------------+--------+---------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - 588799238330220545 | core_schedule_label | ACTIVE | 2020-09-11 00:00:00+00:00 | @daily | BACKUP INTO 's3://{BUCKET NAME}/{PATH}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}' WITH detached -(1 row) -~~~ - -For more examples on how to schedule backups that take full and incremental backups, see [`CREATE SCHEDULE FOR BACKUP`](create-schedule-for-backup.html). - -### Advanced examples - -{% include {{ page.version.version }}/backups/advanced-examples-list.md %} - -{{site.data.alerts.callout_info}} -To take incremental backups, backups with revision history, locality-aware backups, and encrypted backups, you need an [Enterprise license](enterprise-licensing.html). -{{site.data.alerts.end}} - -## See also - -- [`BACKUP`][backup] -- [`RESTORE`][restore] -- [Take and Restore Encrypted Backups](take-and-restore-encrypted-backups.html) -- [Take and Restore Locality-aware Backups](take-and-restore-locality-aware-backups.html) -- [Take Backups with Revision History and Restore from a Point-in-time](take-backups-with-revision-history-and-restore-from-a-point-in-time.html) -- [`SQL DUMP`](cockroach-dump.html) -- [`IMPORT`](migration-overview.html) -- [Use the Built-in SQL Client](cockroach-sql.html) -- [Other Cockroach Commands](cockroach-commands.html) - - - -[backup]: backup.html -[restore]: restore.html diff --git a/src/current/v21.1/temporary-tables.md b/src/current/v21.1/temporary-tables.md deleted file mode 100644 index ca14b7666d3..00000000000 --- a/src/current/v21.1/temporary-tables.md +++ /dev/null @@ -1,222 +0,0 @@ ---- -title: Temporary Tables -summary: CockroachDB supports session-scoped temporary tables. -toc: true ---- - - CockroachDB supports session-scoped temporary tables (also called "temp tables"). Unlike [persistent tables](create-table.html), temp tables can only be accessed from the session in which they were created, and they are dropped at the end of the session. - -To create a temp table, add `TEMP`/`TEMPORARY` to a [`CREATE TABLE`](create-table.html) or [`CREATE TABLE AS`](create-table-as.html) statement. For full syntax details, see the [`CREATE TABLE`](create-table.html#synopsis) and [`CREATE TABLE AS`](create-table-as.html#synopsis) pages. For example usage, see [Examples](#examples). - -{{site.data.alerts.callout_danger}} -**This is an experimental feature**. The interface and output are subject to change. For details, see the tracking issue [cockroachdb/cockroach#46260](https://github.com/cockroachdb/cockroach/issues/46260). -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -By default, temp tables are disabled in CockroachDB. To enable temp tables, set the `experimental_enable_temp_tables` [session variable](set-vars.html) to `on`. -{{site.data.alerts.end}} - -CockroachDB also supports [temporary views](views.html#temporary-views) and [temporary sequences](create-sequence.html#temporary-sequences). - -## Details - -- Temp tables are automatically dropped at the end of the session. -- A temp table can only be accessed from the session in which it was created. -- Temp tables persist across transactions in the same session. -- Temp tables can reference persistent tables, but persistent tables cannot reference temp tables. -- Temp tables cannot be converted to persistent tables. -- For [PostgreSQL compatibility](https://www.postgresql.org/docs/current/sql-createtable.html), CockroachDB supports the clause `ON COMMIT PRESERVE ROWS` at the end of `CREATE TEMP TABLE` statements. CockroachDB only supports session-scoped temp tables, and does not support the clauses `ON COMMIT DELETE ROWS` and `ON COMMIT DROP`, which are used to define transaction-scoped temp tables in PostgreSQL. - -By default, every 30 minutes CockroachDB cleans up all temporary objects that are not tied to an active session. You can change how often the cleanup job runs with the `sql.temp_object_cleaner.cleanup_interval` [cluster setting](cluster-settings.html). - -## Temporary schemas - -Temp tables are not part of the `public` schema. Instead, when you create the first temp table for a session, CockroachDB generates a single temporary schema (`pg_temp_`) for all of the temp tables, [temporary views](views.html#temporary-views), and [temporary sequences](create-sequence.html#temporary-sequences) in the current session for a database. In a session, you can reference the session's temporary schema as `pg_temp`. - -{{site.data.alerts.callout_info}} -Because the [`SHOW TABLES`](show-tables.html) statement defaults to the `public` schema (which doesn't include temp tables), using `SHOW TABLES` without specifying a schema will not return any temp tables. -{{site.data.alerts.end}} - -## Examples - -To use temp tables, you need to set `experimental_enable_temp_tables` to `on`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET experimental_enable_temp_tables=on; -~~~ - -### Create a temp table - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TEMP TABLE users ( - id UUID, - city STRING, - name STRING, - address STRING, - CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC) -); -~~~ - -You can use [`SHOW CREATE`](show-create.html) to view temp tables: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE users; -~~~ - -~~~ - table_name | create_statement --------------+------------------------------------------------- - users | CREATE TEMP TABLE users ( - | id UUID NOT NULL, - | city STRING NULL, - | name STRING NULL, - | address STRING NULL, - | CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - | FAMILY "primary" (id, city, name, address) - | ) -(1 row) -~~~ - -To show the newly created `pg_temp` schema, use [`SHOW SCHEMAS`](show-schemas.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW SCHEMAS; -~~~ - -~~~ - schema_name ---------------------------------- - crdb_internal - information_schema - pg_catalog - pg_extension - pg_temp_1602087923187609000_1 - public -(6 rows) -~~~ - -### Create a temp table that references another temp table - -To create another temp table that references `users`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TEMP TABLE vehicles ( - id UUID NOT NULL, - city STRING NOT NULL, - type STRING, - owner_id UUID, - creation_time TIMESTAMP, - CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - CONSTRAINT fk_city_ref_users FOREIGN KEY (city, owner_id) REFERENCES users(city, id) -); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE vehicles; -~~~ - -~~~ - table_name | create_statement --------------+-------------------------------------------------------------------------------------------- - vehicles | CREATE TEMP TABLE vehicles ( - | id UUID NOT NULL, - | city STRING NOT NULL, - | type STRING NULL, - | owner_id UUID NULL, - | creation_time TIMESTAMP NULL, - | CONSTRAINT "primary" PRIMARY KEY (city ASC, id ASC), - | CONSTRAINT fk_city_ref_users FOREIGN KEY (city, owner_id) REFERENCES users(city, id), - | INDEX vehicles_auto_index_fk_city_ref_users (city ASC, owner_id ASC), - | FAMILY "primary" (id, city, type, owner_id, creation_time) - | ) -(1 row) -~~~ - -### Show all temp tables in a session - -To show all temp tables in a session's temporary schema, use `SHOW TABLES FROM pg_temp`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM pg_temp; -~~~ - -~~~ - schema_name | table_name | type | estimated_row_count ---------------------------------+------------+-------+---------------------- - pg_temp_1602087923187609000_1 | users | table | 0 - pg_temp_1602087923187609000_1 | vehicles | table | 0 -(2 rows) -~~~ - -You can also use the full name of the temporary schema in the `SHOW` statement (e.g., `SHOW TABLES FROM pg_temp_1602087923187609000_1`). - -### Show temp tables in `information_schema` - -Although temp tables are not included in the `public` schema, metadata for temp tables is included in the [`information_schema`](information-schema.html) and `pg_catalog` schemas. - -For example, the [`information_schema.tables`](information-schema.html#tables) table includes information about all tables in all schemas in all databases, including temp tables: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM information_schema.tables WHERE table_schema='pg_temp_1602087923187609000_1'; -~~~ - -~~~ - table_catalog | table_schema | table_name | table_type | is_insertable_into | version ------------------+-------------------------------+------------+-----------------+--------------------+---------- - defaultdb | pg_temp_1602087923187609000_1 | users | LOCAL TEMPORARY | YES | 2 - defaultdb | pg_temp_1602087923187609000_1 | vehicles | LOCAL TEMPORARY | YES | 2 -(2 rows) -~~~ - -### Cancel a session - -If you end the session, all temp tables are lost. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW session_id; -~~~ - -~~~ - session_id ------------------------------------- - 15fd69f9831c1ed00000000000000001 -(1 row) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> CANCEL SESSION '15fd69f9831c1ed00000000000000001'; -~~~ - -~~~ -ERROR: driver: bad connection -warning: connection lost! -opening new connection: all session settings will be lost -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE TABLE users; -~~~ - -~~~ -ERROR: relation "users" does not exist -SQLSTATE: 42P01 -~~~ - -## See also - -- [`CREATE TABLE`](create-table.html) -- [`CREATE TABLE AS`](create-table-as.html) -- [`SHOW CREATE TABLE`](show-create.html) -- [Temporary views](views.html#temporary-views) -- [Temporary sequences](create-sequence.html#temporary-sequences). diff --git a/src/current/v21.1/third-party-database-tools.md b/src/current/v21.1/third-party-database-tools.md deleted file mode 100644 index 71fdbbdfdd8..00000000000 --- a/src/current/v21.1/third-party-database-tools.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -title: Third-Party Tools Supported by Cockroach Labs -summary: Learn about third-party software that works with CockroachDB. -toc: true ---- - -CockroachDB's support of the PostgreSQL wire protocol makes most PostgreSQL drivers, ORM frameworks, and other types of third-party database tools designed for PostgreSQL compatible with CockroachDB. Cockroach Labs guarantees [official support](#support-levels) for a set of popular PostgreSQL tools, which we list on this page. - -{% include {{page.version.version}}/misc/tooling.md %} - -## See also - -- [Example Apps](example-apps.html) -- [Install a Postgres Client](install-client-drivers.html) -- [Third-Party Tools Supported by the Community](community-tooling.html) diff --git a/src/current/v21.1/third-party-monitoring-tools.md b/src/current/v21.1/third-party-monitoring-tools.md deleted file mode 100644 index da8307d3c78..00000000000 --- a/src/current/v21.1/third-party-monitoring-tools.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -title: Third-Party Monitoring Integrations -summary: Learn about third-party monitoring tools that are integrated with CockroachDB. -toc: true ---- - -CockroachDB is officially integrated with the following third-party monitoring platforms. These integrations enable external tools and interfaces to collect, visualize, and alert on CockroachDB metrics. - -| Platform | Integration | Metrics from | Tutorial | -|----------|----------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------|--------------------------------------------------| -| Datadog | [CockroachDB check for Datadog Agent](https://docs.datadoghq.com/integrations/cockroachdb/?tab=host) | [Prometheus endpoint](monitoring-and-alerting.html#prometheus-endpoint) | [Monitor CockroachDB with Datadog](datadog.html) | -| Kibana | [CockroachDB module for Metricbeat](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-cockroachdb.html) | [Prometheus endpoint](monitoring-and-alerting.html#prometheus-endpoint) | [Monitor CockroachDB with Kibana](kibana.html) | - -## See Also - -- [Monitoring and Alerting](monitoring-and-alerting.html) -- [DB Console Overview](ui-overview.html) -- [Logging Overview](logging-overview.html) \ No newline at end of file diff --git a/src/current/v21.1/time.md b/src/current/v21.1/time.md deleted file mode 100644 index 54d761bb369..00000000000 --- a/src/current/v21.1/time.md +++ /dev/null @@ -1,226 +0,0 @@ ---- -title: TIME / TIMETZ -summary: CockroachDB's TIME data type stores a time of day in UTC. -toc: true ---- - -The `TIME` [data type](data-types.html) stores the time of day in UTC. - - The `TIMETZ` data type stores a time of day with a time zone offset from UTC. - -## Variants - -`TIME` has two variants: - -- `TIME`, which presents all `TIME` values in UTC. - -- `TIMETZ`, which converts `TIME` values with a specified time zone offset from UTC. - - Ordering for `TIMETZ` is done in terms of [epoch](https://en.wikipedia.org/wiki/Epoch_(computing)). Time zones with lesser values are ranked higher if times are equal. For example, `'2:00-1' > '2:00+0'` and `'12:00-1' > '1:00+0'`. - - [Like Postgres](https://www.postgresql.org/docs/current/static/datatype-datetime.html), we implement the `TIMETZ` variant for [SQL standards compliance](sql-feature-support.html). We also implement the `TIMETZ` variant for compatibility with ORMs, like [Hibernate](build-a-java-app-with-cockroachdb-hibernate.html). - -You can use the [`timezone()`](functions-and-operators.html#date-and-time-functions) and [`AT TIME ZONE`](functions-and-operators.html#special-syntax-forms) functions to convert a `TIMETZ` into a `TIME` at a specified timezone, or to convert a `TIME` into a `TIMETZ` at a specified timezone. - -{{site.data.alerts.callout_success}} -We recommend always using `TIME` instead of `TIMETZ`. Convert UTC values to the appropriate time zone on the client side. -{{site.data.alerts.end}} - -## Aliases - -In CockroachDB, the following are aliases: - -Alias | Long Version ----------|------------- -`TIME` | `TIME WITHOUT TIME ZONE` -`TIMETZ` | `TIME WITH TIME ZONE` - -## Syntax - -### `TIME` - -A constant value of type `TIME` can be expressed using an [interpreted literal](sql-constants.html#interpreted-literals), or a string literal [annotated with](scalar-expressions.html#explicitly-typed-expressions) type `TIME` or [coerced to](scalar-expressions.html#explicit-type-coercions) type `TIME`. When it is unambiguous, a simple unannotated [string literal](sql-constants.html#string-literals) can also be automatically interpreted as type `TIME`. - -The string format for `TIME` is `HH:MM:SS.SSSSSS`. For example: `TIME '05:40:00.000001'`. The fractional portion is optional and is rounded to microseconds (i.e., six digits after the decimal) for compatibility with the [PostgreSQL wire protocol](https://www.postgresql.org/docs/current/static/protocol.html). - -{{site.data.alerts.callout_info}} -A date of `0000-01-01` is displayed for all `TIME`/`TIMETZ` values, but is not stored in the database. To print without a date, you can cast the type to a `STRING`. - -A time zone offset of `+00:00` is also displayed for all `TIME` and [`TIMESTAMP`](timestamp.html) values, but is not stored in the database. -{{site.data.alerts.end}} - -### `TIMETZ` - -To express a `TIMETZ` value with a time zone offset from UTC, you can add an offset to a `TIME` value. For example, `TIMETZ '10:10:10.555555-05:00'` offsets from UTC by -5. - -If no time zone is specified for a `TIMETZ` value, the `timezone` [session variable](show-vars.html#supported-variables) is used. For example, if you [set the `timezone`](set-vars.html#set-time-zone) for a session using `SET TIME ZONE -2`, and you define the `TIMETZ` as `TIMETZ '10:10:10.55'`, the value will be displayed with an offset of -2 from UTC. - -`TIMETZ` is not affected by session-scoped offsets (unlike [`TIMESTAMPTZ`](timestamp.html)). Time zone offsets only apply to values inserted after the offset has been set, and do not affect existing `TIMETZ` values, or `TIMETZ` values with a time zone offset specified. - -## Size - -A `TIME` column supports values up to 8 bytes in width, but the total storage size is likely to be larger due to CockroachDB metadata. - -A `TIMETZ` column supports values up to 12 bytes in width, but the total storage size is likely to be larger due to CockroachDB metadata. - -## Precision - - CockroachDB supports precision levels from 0 (seconds) to 6 (microseconds) for `TIME`/`TIMETZ` values. Precision in time values specifies the number of fractional digits retained in the seconds field. For example, specifying a `TIME` value as `TIME(3)` truncates the time precision to milliseconds. By default, `TIME`/`TIMETZ` values have a precision of 6 (microseconds). - -You can use an [`ALTER COLUMN ... SET DATA TYPE`](alter-column.html) statement to change the precision level of a `TIME`-typed column. If there is already a non-default precision level specified for the column, the precision level can only be changed to an equal or greater precision level. For an example, see [Create a table with a `TIME`-typed column, with precision](#create-a-table-with-a-time-typed-column-with-precision). - -## Examples - -### Create a table with a `TIME`-typed column - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE time (time_id INT PRIMARY KEY, time_val TIME); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM time; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden -+-------------+-----------+-------------+----------------+-----------------------+-----------+-----------+ - time_id | INT8 | false | NULL | | {primary} | false - time_val | TIME | true | NULL | | {} | false -(2 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO time VALUES (1, TIME '05:40:00'), (2, TIME '05:41:39'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM time; -~~~ - -~~~ - time_id | time_val -+---------+---------------------------+ - 1 | 0000-01-01 05:40:00+00:00 - 2 | 0000-01-01 05:41:39+00:00 -(2 rows) -~~~ - -{{site.data.alerts.callout_info}} -The SQL shell displays the date and time zone due to the Go SQL driver it uses. Other client drivers may behave similarly. In such cases, however, the date and time zone are not relevant and are not stored in the database. -{{site.data.alerts.end}} - -Comparing `TIME` values: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT (SELECT time_val FROM time WHERE time_id = 1) < (SELECT time_val FROM time WHERE time_id = 2); -~~~ - -~~~ -< (SELECT time_val FROM time WHERE time_id = 2); - ?column? -+----------+ - true -(1 row) -~~~ - -### Create a table with a `TIME`-typed column, with precision - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE time (time_id INT PRIMARY KEY, time_val TIME(4)); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM time; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden ---------------+-----------+-------------+----------------+-----------------------+-----------+------------ - time_id | INT8 | false | NULL | | {primary} | false - time_val | TIME(4) | true | NULL | | {} | false -(2 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO time VALUES (1, TIME '05:40:00.123456'), (2, TIME '05:41:39.12345'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM time; -~~~ - -~~~ - time_id | time_val -----------+--------------------------------- - 1 | 0000-01-01 05:40:00.1235+00:00 - 2 | 0000-01-01 05:41:39.1235+00:00 -(2 rows) -~~~ - -To change the precision level of a column, you can use an [`ALTER COLUMN ... SET DATA TYPE`](alter-column.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE time ALTER COLUMN time_val SET DATA TYPE TIME(5); -~~~ - -~~~ -ALTER TABLE -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM time; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden ---------------+-----------+-------------+----------------+-----------------------+-----------+------------ - time_id | INT8 | false | NULL | | {primary} | false - time_val | TIME(5) | true | NULL | | {} | false -(2 rows) -~~~ - -{{site.data.alerts.callout_info}} -If a non-default precision level has already been specified, you cannot change the precision to a lower level. -{{site.data.alerts.end}} - -In this case, the `time_val` column, which is of type `TIME(5)`, cannot be changed to a precision level below `5`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE time ALTER COLUMN time_val SET DATA TYPE TIME(3); -~~~ - -~~~ -ERROR: unimplemented: type conversion from TIME(5) to TIME(3) requires overwriting existing values which is not yet implemented -SQLSTATE: 0A000 -~~~ - - -## Supported casting & conversion - -`TIME`/`TIMETZ` values can be [cast](data-types.html#data-type-conversions-and-casts) to any of the following data types: - -Type | Details ------|-------- -`INTERVAL` | Converts to the span of time since midnight (00:00) -`STRING` | Converts to format `'HH:MM:SS.SSSSSS'` (microsecond precision) - -{{site.data.alerts.callout_info}} -CockroachDB displays `TIME '24:00:00'` and `TIMETZ '24:00:00'` as `0000-01-01 00:00:00`. To display the proper stored value (`24:00:00`), you can [cast the data type to a `STRING`](time.html#supported-casting-conversion). -{{site.data.alerts.end}} - -## See also - -- [Data Types](data-types.html) -- [SQL Feature Support](sql-feature-support.html) diff --git a/src/current/v21.1/timestamp.md b/src/current/v21.1/timestamp.md deleted file mode 100644 index b65f70b4090..00000000000 --- a/src/current/v21.1/timestamp.md +++ /dev/null @@ -1,246 +0,0 @@ ---- -title: TIMESTAMP / TIMESTAMPTZ -summary: The TIMESTAMP and TIMESTAMPTZ data types stores a date and time pair in UTC. -toc: true ---- - -The `TIMESTAMP` and `TIMESTAMPTZ` [data types](data-types.html) store a date and time pair in UTC. - -## Variants - -`TIMESTAMP` has two variants: - -- `TIMESTAMP` presents all `TIMESTAMP` values in UTC. - -- `TIMESTAMPTZ` converts `TIMESTAMP` values from UTC to the client's session time zone (unless another time zone is specified for the value). However, it is conceptually important to note that `TIMESTAMPTZ` **does not** store any time zone data. - - {{site.data.alerts.callout_info}} - The default session time zone is UTC, which means that by default `TIMESTAMPTZ` values display in UTC. - {{site.data.alerts.end}} - -The difference between these two variants is that `TIMESTAMPTZ` uses the client's session [time zone](set-vars.html#set-time-zone), while the other simply does not. This behavior extends to functions like `now()` and `extract()` on `TIMESTAMPTZ` values. - -You can use the [`timezone()`](functions-and-operators.html#date-and-time-functions) and [`AT TIME ZONE`](functions-and-operators.html#special-syntax-forms) functions to convert a `TIMESTAMPTZ` into a `TIMESTAMP` at a specified timezone, or to convert a `TIMESTAMP` into a `TIMESTAMPTZ` at a specified timezone. - -## Best practices - -We recommend always using the `TIMESTAMPTZ` variant because the `TIMESTAMP` variant can sometimes lead to unexpected behaviors when it ignores a session offset. However, we also recommend you avoid [setting a session time zone offset](set-vars.html#set-time-zone) for your database. - -## Aliases - -In CockroachDB, the following are aliases: - -- `TIMESTAMP`, `TIMESTAMP WITHOUT TIME ZONE` -- `TIMESTAMPTZ`, `TIMESTAMP WITH TIME ZONE` - -## Syntax - -A constant value of type `TIMESTAMP`/`TIMESTAMPTZ` can be expressed using an -[interpreted literal](sql-constants.html#interpreted-literals), or a -string literal -[annotated with](scalar-expressions.html#explicitly-typed-expressions) -type `TIMESTAMP`/`TIMESTAMPTZ` or -[coerced to](scalar-expressions.html#explicit-type-coercions) type -`TIMESTAMP`/`TIMESTAMPTZ`. - -`TIMESTAMP` constants can be expressed using the -following string literal formats: - -Format | Example --------|-------- -Date only | `TIMESTAMP '2016-01-25'` -Date and Time | `TIMESTAMP '2016-01-25 10:10:10.555555'` -ISO 8601 | `TIMESTAMP '2016-01-25T10:10:10.555555'` - -To express a `TIMESTAMPTZ` value (with time zone offset from UTC), use -the following format: `TIMESTAMPTZ '2016-01-25 10:10:10.555555-05:00'` - -When it is unambiguous, a simple unannotated [string literal](sql-constants.html#string-literals) can also -be automatically interpreted as type `TIMESTAMP` or `TIMESTAMPTZ`. - -Note that the fractional portion is optional and is rounded to -microseconds (6 digits after decimal) for compatibility with the -PostgreSQL wire protocol. - - For PostgreSQL compatibility, CockroachDB bounds `TIMESTAMP` values by the lowest and highest `TIMESTAMP` values supported by PostgreSQL. The minimum allowable `TIMESTAMP` value is `4714-11-24 00:00:00+00 BC`, and the highest allowable `TIMESTAMP` value is `294276-12-31 23:59:59.999999`. - -{{site.data.alerts.callout_info}} -A time zone offset of `+00:00` is displayed for all [`TIME`](time.html) and `TIMESTAMP` values, but is not stored in the database. -{{site.data.alerts.end}} - -## Size - -A `TIMESTAMP`/`TIMESTAMPTZ` column supports values up to 12 bytes in width, but the total storage size is likely to be larger due to CockroachDB metadata. - -## Precision - - CockroachDB supports precision levels from 0 (seconds) to 6 (microseconds) for `TIMESTAMP`/`TIMESTAMPTZ` values. Precision in time values specifies the number of fractional digits retained in the seconds field. For example, specifying a `TIMESTAMPTZ` value as `TIMESTAMPTZ(3)` truncates the time component to milliseconds. By default, `TIMESTAMP`/`TIMESTAMPTZ` values have a precision of 6 (microseconds). - -You can use an [`ALTER COLUMN ... SET DATA TYPE`](alter-column.html) statement to change the precision level of a `TIMESTAMP`/`TIMESTAMPTZ`-typed column. If there is already a non-default precision level specified for the column, the precision level can only be changed to an equal or greater precision level. For an example, see [Create a table with a `TIMESTAMP`-typed column, with precision](#create-a-table-with-a-timestamp-typed-column-with-precision). - -## Examples - -### Create a table with a `TIMESTAMPTZ`-typed column - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE timestamps (a INT PRIMARY KEY, b TIMESTAMPTZ); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM timestamps; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden -+-------------+-------------+-------------+----------------+-----------------------+-----------+-----------+ - a | INT8 | false | NULL | | {primary} | false - b | TIMESTAMPTZ | true | NULL | | {} | false -(2 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO timestamps VALUES (1, TIMESTAMPTZ '2016-03-26 10:10:10-05:00'), (2, TIMESTAMPTZ '2016-03-26'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM timestamps; -~~~ - -~~~ - a | b -+---+---------------------------+ - 1 | 2016-03-26 15:10:10+00:00 - 2 | 2016-03-26 00:00:00+00:00 -(2 rows) -~~~ - -### Create a table with a `TIMESTAMP`-typed column, with precision - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE timestamps (a INT PRIMARY KEY, b TIMESTAMP(3)); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM timestamps; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden ---------------+--------------+-------------+----------------+-----------------------+-----------+------------ - a | INT8 | false | NULL | | {primary} | false - b | TIMESTAMP(3) | true | NULL | | {} | false -(2 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO timestamps VALUES (1, TIMESTAMP '2020-03-25 12:00:00.123456'), (2, TIMESTAMP '2020-03-26 4:00:00.123456'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM timestamps; -~~~ - -~~~ - a | b -----+-------------------------------- - 1 | 2020-03-25 12:00:00.123+00:00 - 2 | 2020-03-26 04:00:00.123+00:00 -(2 rows) -~~~ - -To change the precision level of a column, you can use an [`ALTER COLUMN ... SET DATA TYPE`](alter-column.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE timestamps ALTER COLUMN b SET DATA TYPE TIMESTAMP(4); -~~~ - -~~~ -ALTER TABLE -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM timestamps; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden ---------------+--------------+-------------+----------------+-----------------------+-----------+------------ - a | INT8 | false | NULL | | {primary} | false - b | TIMESTAMP(4) | true | NULL | | {} | false -(2 rows) -~~~ - -When changing precision level, `TIMESTAMP` can be changed to `TIMESTAMPTZ`, and `TIMESTAMPTZ` can be changed to `TIMESTAMP`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE timestamps ALTER COLUMN b SET DATA TYPE TIMESTAMPTZ(5); -~~~ - -~~~ -ALTER TABLE -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM timestamps; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden ---------------+----------------+-------------+----------------+-----------------------+-----------+------------ - a | INT8 | false | NULL | | {primary} | false - b | TIMESTAMPTZ(5) | true | NULL | | {} | false -(2 rows) -~~~ - -{{site.data.alerts.callout_info}} -If a non-default precision level has already been specified, you cannot change the precision to a lower level. -{{site.data.alerts.end}} - -In this case, the `b` column, which is of type `TIMESTAMPTZ(5)`, cannot be changed to a precision level below `5`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE timestamps ALTER COLUMN b SET DATA TYPE TIMESTAMPTZ(3); -~~~ - -~~~ -ERROR: unimplemented: type conversion from TIMESTAMPTZ(5) to TIMESTAMPTZ(3) requires overwriting existing values which is not yet implemented -SQLSTATE: 0A000 -~~~ - -## Supported casting and conversion - -`TIMESTAMP` values can be [cast](data-types.html#data-type-conversions-and-casts) to any of the following data types: - -Type | Details ------|-------- -`DECIMAL` | Converts to number of seconds since the Unix epoch (Jan. 1, 1970). -`FLOAT` | Converts to number of seconds since the Unix epoch (Jan. 1, 1970). -`TIME` | Converts to the time portion (HH:MM:SS) of the timestamp. -`INT` | Converts to number of seconds since the Unix epoch (Jan. 1, 1970). -`DATE` | -- -`STRING` | Converts to the date and time portion (YYYY-MM-DD HH:MM:SS) of the timestamp and omits the time zone offset. - -### Infinity `TIMESTAMP` casts - -CockroachDB currently does not support an `infinity`/`-infinity` representation for `TIMESTAMP` casts. Instead, `infinity::TIMESTAMP` evaluates to `294276-12-31 23:59:59.999999+00:00`, the maximum `TIMESTAMP` value supported, and `-infinity::TIMESTAMP` evaluates to `-4713-11-24 00:00:00+00:00`, the minimum `TIMESTAMP` value supported. - -Note that this behavior differs from PostgreSQL, for which `infinity` is higher than any allowable `TIMESTAMP` value (including `294276-12-31 23:59:59.999999+00:00`), and `-infinity` is lower than any allowable `TIMESTAMP` value (including `-4713-11-24 00:00:00+00:00`). - -For more details, see [tracking issue](https://github.com/cockroachdb/cockroach/issues/41564). - -## See also - -[Data Types](data-types.html) diff --git a/src/current/v21.1/topology-basic-production.md b/src/current/v21.1/topology-basic-production.md deleted file mode 100644 index a3186dacd9d..00000000000 --- a/src/current/v21.1/topology-basic-production.md +++ /dev/null @@ -1,89 +0,0 @@ ---- -title: Basic Production Topology -summary: Guidance for a single-region production deployment. -toc: true ---- - -When you're ready to run CockroachDB in production in a single region, it's important to deploy at least 3 CockroachDB nodes to take advantage of CockroachDB's automatic replication, distribution, rebalancing, and resiliency capabilities. - -{{site.data.alerts.callout_success}} -If you haven't already, [review the full range of topology patterns](topology-patterns.html) to ensure you choose the right one for your use case. -{{site.data.alerts.end}} - -## Prerequisites - -{% include {{ page.version.version }}/topology-patterns/fundamentals.md %} - -## Configuration - -Basic production topology - -1. Provision hardware as follows: - - 1 region with 3 AZs - - 3+ VMs evenly distributed across AZs; add more VMs to increase throughput - - App and load balancer in same region as VMs for CockroachDB - - The load balancer redirects to CockroachDB nodes in the region - -2. Start each node on a separate VM, setting the [`--locality`](cockroach-start.html#locality) flag to the node's region and AZ combination. For example, the following command starts a node in the east1 availability zone of the us-east region: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --locality=region=us-east,zone=east1 \ - --certs-dir=certs \ - --advertise-addr= \ - --join=:26257,:26257,:26257 \ - --cache=.25 \ - --max-sql-memory=.25 \ - --background - ~~~ - -With the default 3-way replication factor and `--locality` set as described above, CockroachDB balances each range of table data across AZs, one replica per AZ. System data is replicated 5 times by default and also balanced across AZs, thus increasing the [resiliency of the cluster](configure-replication-zones.html#create-a-replication-zone-for-a-system-range) as a whole. - -## Characteristics - -### Latency - -#### Reads - -Since all ranges, including leaseholder replicas, are in a single region, read latency is very low. - -For example, in the animation below: - -1. The read request reaches the load balancer. -2. The load balancer routes the request to a gateway node. -3. The gateway node routes the request to the relevant leaseholder. -4. The leaseholder retrieves the results and returns to the gateway node. -5. The gateway node returns the results to the client. - -Basic production topology - -#### Writes - -Since all ranges are in a single region, writes achieve consensus without leaving the region and, thus, write latency is very low as well. - -For example, in the animation below: - -1. The write request reaches the load balancer. -2. The load balancer routes the request to a gateway node. -3. The gateway node routes the request to the leaseholder replicas for the relevant table and secondary index. -4. While each leaseholder appends the write to its Raft log, it notifies its follower replicas. -5. In each case, as soon as one follower has appended the write to its Raft log (and thus a majority of replicas agree based on identical Raft logs), it notifies the leaseholder and the write is committed on the agreeing replicas. -6. The leaseholders then return acknowledgement of the commit to the gateway node. -7. The gateway node returns the acknowledgement to the client. - -Leaseholder preferences topology - -### Resiliency - -Because each range is balanced across AZs, one AZ can fail without interrupting access to any data: - -Basic production topology - -However, if an additional AZ fails at the same time, the ranges that lose consensus become unavailable for reads and writes: - -Basic production topology - -## See also - -{% include {{ page.version.version }}/topology-patterns/see-also.md %} diff --git a/src/current/v21.1/topology-development.md b/src/current/v21.1/topology-development.md deleted file mode 100644 index ea2a5388b41..00000000000 --- a/src/current/v21.1/topology-development.md +++ /dev/null @@ -1,39 +0,0 @@ ---- -title: Development Topology -summary: Guidance for a single-node cluster for local development. -toc: true ---- - -While developing an application against CockroachDB, it's sufficient to deploy a single-node cluster close to your test application, whether that's on a single VM or on your laptop. - -{{site.data.alerts.callout_success}} -If you haven't already, [review the full range of topology patterns](topology-patterns.html) to ensure you choose the right one for your use case. -{{site.data.alerts.end}} - -## Prerequisites - -{% include {{ page.version.version }}/topology-patterns/fundamentals.md %} - -## Configuration - -Development topology - -For this pattern, you can either [run CockroachDB locally](start-a-local-cluster.html) or [deploy a single-node cluster on a cloud VM](manual-deployment.html). - -## Characteristics - -### Latency - -With the CockroachDB node in the same region as your client, and without the overhead of replication, both read and write latency are very low: - -Development topology - -### Resiliency - -In a single-node cluster, CockroachDB does not replicate data and, therefore, is not resilient to failures. If the machine where the node is running fails, or if the region or availability zone containing the machine fails, the cluster becomes unavailable: - -Development topology - -## See also - -{% include {{ page.version.version }}/topology-patterns/see-also.md %} diff --git a/src/current/v21.1/topology-follow-the-workload.md b/src/current/v21.1/topology-follow-the-workload.md deleted file mode 100644 index e3d7cb446f2..00000000000 --- a/src/current/v21.1/topology-follow-the-workload.md +++ /dev/null @@ -1,86 +0,0 @@ ---- -title: Follow-the-Workload Topology -summary: Guidance on using the follow-the-workload topology in a multi-region deployment. -toc: true ---- - -In a multi-region deployment, follow-the-workload is the default behavior for tables that do not have a [table locality](multiregion-overview.html#table-locality). In general, this is a good choice only for tables with the following requirements: - -- The table is active mostly in one region at a time, e.g., following the sun. -- In the active region, read latency must be low, but write latency can be higher. -- In non-active regions, both read and write latency can be higher. -- Table data must remain available during a region failure. - -{{site.data.alerts.callout_success}} -If read performance is your main focus for a table, but you want low-latency reads everywhere instead of just in the most active region, consider [Global Tables](global-tables.html) or [Follower Reads](topology-follower-reads.html). - -Note that if you start using the [multi-region SQL abstractions](multiregion-overview.html) for a database, CockroachDB will no longer provide the follow-the-workload behavior described on this page for that database. -{{site.data.alerts.end}} - -## Prerequisites - -### Fundamentals - -{% include {{ page.version.version }}/topology-patterns/fundamentals.md %} - -### Cluster setup - -{% include {{ page.version.version }}/topology-patterns/multi-region-cluster-setup.md %} - -## Configuration - -Aside from [deploying a cluster across three regions](#cluster-setup) properly, with each node started with the [`--locality`](cockroach-start.html#locality) flag specifying its region and zone combination, this behavior requires no extra configuration. CockroachDB will balance the replicas for a table across the three regions and will assign the range lease to the replica in the region with the greatest demand at any given time (the follow-the-workload feature). This means that read latency in the active region will be low while read latency in other regions will be higher due to having to leave the region to reach the leaseholder. Write latency will be higher as well due to always involving replicas in multiple regions. - -Follower reads topology - -{{site.data.alerts.callout_info}} -Follow-the-workload is also used by [system ranges containing important internal data](configure-replication-zones.html#create-a-replication-zone-for-a-system-range). -{{site.data.alerts.end}} - -## Characteristics - -### Latency - -#### Reads - -Reads in the region with the most demand will access the local leaseholder and, therefore, never leave the region. This makes read latency very low in the currently most active region. Reads in other regions, however, will be routed to the leaseholder in a different region and, thus, read latency will be higher. - -For example, in the animation below, the most active region is `us-east` and, thus, the table's leaseholder is in that region: - -1. The read request in `us-east` reaches the regional load balancer. -2. The load balancer routes the request to a gateway node. -3. The gateway node routes the request to the leaseholder replica. -4. The leaseholder retrieves the results and returns to the gateway node. -5. The gateway node returns the results to the client. In this case, reads in the `us-east` remain in the region and are lower-latency than reads in other regions. - -Follow-the-workload topology - -#### Writes - -The replicas for the table are spread across all 3 regions, so writes involve multiple network hops across regions to achieve consensus. This increases write latency significantly. - -For example, in the animation below, assuming the most active region is still `us-east`: - -1. The write request in `us-east` reaches the regional load balancer. -2. The load balancer routes the request to a gateway node. -3. The gateway node routes the request to the leaseholder replica. -4. While the leaseholder appends the write to its Raft log, it notifies its follower replicas. -5. As soon as one follower has appended the write to its Raft log (and thus a majority of replicas agree based on identical Raft logs), it notifies the leaseholder and the write is committed on the agreeing replicas. -6. The leaseholders then return acknowledgement of the commit to the gateway node. -7. The gateway node returns the acknowledgement to the client. - -Follow-the-workload topology - -### Resiliency - -Because this pattern balances the replicas for the table across regions, one entire region can fail without interrupting access to the table: - -Follow-the-workload topology - - - -## See also - -{% include {{ page.version.version }}/topology-patterns/see-also.md %} diff --git a/src/current/v21.1/topology-follower-reads.md b/src/current/v21.1/topology-follower-reads.md deleted file mode 100644 index b64ec0799ec..00000000000 --- a/src/current/v21.1/topology-follower-reads.md +++ /dev/null @@ -1,141 +0,0 @@ ---- -title: Follower Reads Topology -summary: Guidance on using the follower reads topology in a multi-region deployment. -toc: true ---- - -In a multi-region deployment, [Follower Reads](follower-reads.html) are a good choice for tables with the following requirements: - -- Read latency must be low, but write latency can be higher. -- Reads can be historical (4.8 seconds or more in the past). -- Rows in the table, and all latency-sensitive queries, **cannot** be tied to specific geographies (e.g., a reference table). -- Table data must remain available during a region failure. - -{{site.data.alerts.callout_success}} -If reads from a table must be exactly up-to-date, use [Global Tables](global-tables.html) or [Regional Tables](regional-tables.html) instead. Up-to-date reads are required by tables referenced by [foreign keys](foreign-key.html), for example. -{{site.data.alerts.end}} - -## Prerequisites - -### Fundamentals - -{% include {{ page.version.version }}/topology-patterns/fundamentals.md %} - -### Cluster setup - -{% include {{ page.version.version }}/topology-patterns/multi-region-cluster-setup.md %} - -## Configuration - -{{site.data.alerts.callout_info}} -Follower reads requires an [Enterprise license](https://www.cockroachlabs.com/get-started-cockroachdb/). -{{site.data.alerts.end}} - -### Summary - -You configure your application to use [follower reads](follower-reads.html) by adding an [`AS OF SYSTEM TIME`](as-of-system-time.html) clause when reading from the table. This tells CockroachDB to read slightly historical data from the closest replica so as to avoid being routed to the leaseholder, which may be in an entirely different region. Writes, however, will still leave the region to get consensus for the table. - -### Steps - -Follower reads topology - -Assuming you have a [cluster deployed across three regions](#cluster-setup) and a table like the following: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE postal_codes ( - id INT PRIMARY KEY, - code STRING -); -~~~ - -Insert some data: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO postal_codes (ID, code) VALUES (1, '10001'), (2, '10002'), (3, '10003'), (4,'60601'), (5,'60602'), (6,'60603'), (7,'90001'), (8,'90002'), (9,'90003'); -~~~ - -1. If you do not already have one, [request a trial Enterprise license](https://www.cockroachlabs.com/get-started-cockroachdb/). - -2. Configure your app to use `AS OF SYSTEM TIME follower_read_timestamp()` whenever reading from the table: - - {{site.data.alerts.callout_info}} - The `follower_read_timestamp()` [function](functions-and-operators.html) returns the [`TIMESTAMP`](timestamp.html) `statement_timestamp() - 4.8s`. - {{site.data.alerts.end}} - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SELECT code FROM postal_codes - AS OF SYSTEM TIME follower_read_timestamp() - WHERE id = 5; - ~~~ - - Alternately, instead of modifying individual read queries on the table, you can set the `AS OF SYSTEM TIME` value for all operations in a read-only transaction: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > BEGIN; - - SET TRANSACTION AS OF SYSTEM TIME follower_read_timestamp(); - - SELECT code FROM postal_codes - WHERE id = 5; - - SELECT code FROM postal_codes - WHERE id = 6; - - COMMIT; - ~~~ - -{{site.data.alerts.callout_success}} -Using the [`SET TRANSACTION`](set-transaction.html#use-the-as-of-system-time-option) statement as shown in the example above will make it easier to use the follower reads feature from [drivers and ORMs](install-client-drivers.html). -{{site.data.alerts.end}} - -## Characteristics - -### Latency - -#### Reads - -Reads retrieve historical data from the closest replica and, therefore, never leave the region. This makes read latency very low but slightly stale. - -For example, in the animation below: - -1. The read request in `us-central` reaches the regional load balancer. -2. The load balancer routes the request to a gateway node. -3. The gateway node routes the request to the closest replica for the table. In this case, the replica is *not* the leaseholder. -4. The replica retrieves the results as of 4.8 seconds in the past and returns to the gateway node. -5. The gateway node returns the results to the client. - -Follower reads topology - -#### Writes - -The replicas for the table are spread across all 3 regions, so writes involve multiple network hops across regions to achieve consensus. This increases write latency significantly. - -For example, in the animation below: - -1. The write request in `us-central` reaches the regional load balancer. -2. The load balancer routes the request to a gateway node. -3. The gateway node routes the request to the leaseholder replica for the table in `us-east`. -4. Once the leaseholder has appended the write to its Raft log, it notifies its follower replicas. -5. As soon as one follower has appended the write to its Raft log (and thus a majority of replicas agree based on identical Raft logs), it notifies the leaseholder and the write is committed on the agreeing replicas. -6. The leaseholder then returns acknowledgement of the commit to the gateway node. -7. The gateway node returns the acknowledgement to the client. - -Follower reads topology - -### Resiliency - -Because this pattern balances the replicas for the table across regions, one entire region can fail without interrupting access to the table: - -Follower reads topology - - - -## See also - -{% include {{ page.version.version }}/topology-patterns/see-also.md %} diff --git a/src/current/v21.1/topology-patterns.md b/src/current/v21.1/topology-patterns.md deleted file mode 100644 index 719a5ecdc4d..00000000000 --- a/src/current/v21.1/topology-patterns.md +++ /dev/null @@ -1,56 +0,0 @@ ---- -title: Topology Patterns Overview -summary: Recommended patterns for running CockroachDB in a cloud environment. -toc: true -key: cluster-topology-patterns.html ---- - -This section provides recommended patterns for running CockroachDB in a cloud environment. - -{{site.data.alerts.callout_info}} -You can observe latency for your cluster on the [Network Latency page](ui-network-latency-page.html) of the DB Console. -{{site.data.alerts.end}} - -## Single-region - -When your clients are in a single geographic region, choosing a pattern is straightforward. - -Deployment Type | Latency | Resiliency | Configuration ---------|---------|------------|-------------- -[Development](topology-development.html) |
    • Fast reads and writes
    |
    • None
    |
    • 1 node
    • No replication
    -[Basic Production](topology-basic-production.html) |
    • Fast reads and writes
    |
    • 1 zone failure
    |
    • 1 region
    • 3 zones
    • 3+ nodes across zones
    - -## Multi-region - -When your clients are in multiple geographic regions, it is important to deploy your cluster across regions properly and then carefully choose: - -1. The right [survival goal](multiregion-overview.html#survival-goals) for each database. -1. The right [table locality](multiregion-overview.html#table-locality) for each of your tables. - -Not doing so can result in unexpected latency and resiliency. For more information, see the [Multi-Region Capabilities Overview](multiregion-overview.html). - -{{site.data.alerts.callout_info}} -The multi-region patterns described below are almost always table-specific. For example, you might use [Regional Tables](regional-tables.html) for frequently updated tables that are tied to a specific region, and [Global Tables](global-tables.html) for reference tables that are not tied to a specific region, and that are read frequently but updated infrequently. -{{site.data.alerts.end}} - -| Pattern | Latency | -|----------------------------------------------------------+------------------------------------------------------------------------------------------------------------| -| [Regional Tables](regional-tables.html) | Low latency for single-region writes and multi-region stale reads. | -| [Global Tables](global-tables.html) | Low-latency multi-region reads from all regions, at the expense of higher latency cross-region writes. | -| [Follower Reads](topology-follower-reads.html) | Fast regional (historical) reads, slower cross-region writes. | -| [Follow-the-Workload](topology-follow-the-workload.html) | Fast regional reads in the active region; slower cross-region reads elsewhere. Slower cross-region writes. | - -{{site.data.alerts.callout_info}} -In [multi-region databases](multiregion-overview.html), the resiliency of each database depends on its [survival goal settings](multiregion-overview.html#survival-goals). -{{site.data.alerts.end}} - -## Anti-patterns - -The following anti-patterns are ineffective or risky: - -- Single-region deployments using 2 zones, or multi-region deployments using 2 regions. In these cases, the cluster would be unable to survive the loss of a single zone or a single region, respectively. -- Broadly distributed multi-region deployments (e.g., `us-west`, `asia`, and `europe`) using only the default [Follow-the-Workload](topology-follow-the-workload.html) behavior. In this case, latency will likely be unacceptably high. - -## See also - -{% include {{ page.version.version }}/topology-patterns/see-also.md %} diff --git a/src/current/v21.1/transaction-retry-error-reference.md b/src/current/v21.1/transaction-retry-error-reference.md deleted file mode 100644 index e43d941c1d9..00000000000 --- a/src/current/v21.1/transaction-retry-error-reference.md +++ /dev/null @@ -1,283 +0,0 @@ ---- -title: Transaction Retry Error Reference -summary: A list of the transaction retry (serialization) errors emitted by CockroachDB, including likely causes and user actions for mitigation. -toc: true ---- - -This page has a list of the transaction retry error codes emitted by CockroachDB. - -Transaction retry errors use the `SQLSTATE` error code `40001`, and emit error messages with the string `restart transaction`. The most common transaction retry errors are also known as serialization errors. Serialization errors indicate that a transaction failed because it could not be placed into a [serializable ordering](demo-serializable.html) among all of the currently-executing transactions. - -The failure to place a transaction into a serializable ordering usually happens due to a [transaction conflict](architecture/transaction-layer.html#transaction-conflicts) with another concurrent or recent transaction that is trying to write to the same data. When multiple transactions are trying to write to the same data, this state is also known as contention. When serialization errors occur due to contention, the transaction needs to be retried by the client as described in [client-side retry handling](transactions.html#client-side-intervention). - -In less common cases, transaction retry errors are not caused by contention, but by the internal state of the CockroachDB cluster. For example, the cluster could be overloaded. In such cases, other actions may need to be taken above and beyond client-side retries. - -See below for a complete list of retry error codes. For each error code, we describe: - -- Why the error is happening. -- What to do about it. - -This page is meant to provide information about specific transaction retry error codes to make troubleshooting easier. In most cases, the correct actions to take when these errors occur are: - -1. Update your app to retry on serialization errors (where `SQLSTATE` is `40001`), as described in [client-side retry handling](transactions.html#client-side-intervention). - -2. Design your schema and queries to reduce contention. For more information about how contention occurs and how to avoid it, see [Understanding and avoiding transaction contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention). In particular, if you are able to [send all of the statements in your transaction in a single batch](transactions.html#batched-statements), CockroachDB can usually automatically retry the entire transaction for you. - -3. Use historical reads with [`SELECT ... AS OF SYSTEM TIME`](as-of-system-time.html). - -4. In some cases, [`SELECT FOR UPDATE`](select-for-update.html) can also be used to reduce the occurrence of serialization errors. - -5. [High priority transactions](transactions.html#transaction-priorities) are less likely to experience serialization errors than low priority transactions. Adjusting transaction priorities usually does not change how many serialization errors occur, but it can help control which transactions experience them. - -{{site.data.alerts.callout_info}} -Note that your application's retry logic does not need to distinguish between the different types of serialization errors. They are listed here for reference during advanced troubleshooting. -{{site.data.alerts.end}} - -## Overview - -CockroachDB always attempts to find a [serializable ordering](demo-serializable.html) among all of the currently-executing transactions. - -Whenever possible, CockroachDB will [auto-retry a transaction internally](transactions.html#automatic-retries) without notifying the client. CockroachDB will only send a serialization error to the client when it cannot resolve the error automatically without client-side intervention. - -In other words, by the time a serialization error bubbles up to the client, CockroachDB has already tried to handle the error internally, and could not. - -The main reason why CockroachDB cannot auto-retry every serialization error without sending an error to the client is that the SQL language is "conversational" by design. The client can send arbitrary statements to the server during a transaction, receive some results, and then decide to issue other arbitrary statements inside the same transaction based on the server's response. - -Suppose that the client is [a Java application using JDBC](build-a-java-app-with-cockroachdb.html), or an analyst typing [`BEGIN`](begin-transaction.html) directly to [a SQL shell](cockroach-sql.html). In either case, the client is free to issue a `BEGIN`, wait an arbitrary amount of time, and issue additional statements. Meanwhile, other transactions are being processed by the system, potentially writing to the same data. - -This "conversational" design means that there is no way for the server to always retry the arbitrary statements sent so far inside an open transaction. If there are different results for any given statement than there were at an earlier point in the currently open transaction's lifetime (likely due to the operations of other, concurrently-executing transactions), CockroachDB must defer to the client to decide how to handle that situation. This is why we recommend [keeping transactions as small as possible](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention). - -## Error reference - -- [RETRY_WRITE_TOO_OLD](#retry_write_too_old) -- [RETRY_SERIALIZABLE](#retry_serializable) -- [RETRY_ASYNC_WRITE_FAILURE](#retry_async_write_failure) -- [ReadWithinUncertaintyInterval](#readwithinuncertaintyinterval) -- [RETRY_COMMIT_DEADLINE_EXCEEDED](#retry_commit_deadline_exceeded) -- [ABORT_REASON_ABORTED_RECORD_FOUND](#abort_reason_aborted_record_found) -- [ABORT_REASON_CLIENT_REJECT](#abort_reason_client_reject) -- [ABORT_REASON_PUSHER_ABORTED](#abort_reason_pusher_aborted) -- [ABORT_REASON_ABORT_SPAN](#abort_reason_abort_span) -- [ABORT_REASON_NEW_LEASE_PREVENTS_TXN](#abort_reason_new_lease_prevents_txn) -- [ABORT_REASON_TIMESTAMP_CACHE_REJECTED](#abort_reason_timestamp_cache_rejected) - -### RETRY_WRITE_TOO_OLD - -``` -TransactionRetryWithProtoRefreshError: ... RETRY_WRITE_TOO_OLD ... -``` - -_Description_: - -The `RETRY_WRITE_TOO_OLD` error occurs when a transaction _A_ tries to write to a row _R_, but another transaction _B_ that was supposed to be serialized after _A_ (i.e., had been assigned a higher timestamp), has already written to that row _R_, and has already committed. This is a common error when you have too much contention in your workload. - -_Action_: - -1. Retry transaction _A_ as described in [client-side retry handling](transactions.html#client-side-intervention). -2. Design your schema and queries to reduce contention. For more information about how contention occurs and how to avoid it, see [Understanding and avoiding transaction contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention). In particular, if you are able to [send all of the statements in your transaction in a single batch](transactions.html#batched-statements), CockroachDB can usually automatically retry the entire transaction for you. - -### RETRY_SERIALIZABLE - -``` -TransactionRetryWithProtoRefreshError: ... RETRY_SERIALIZABLE ... -``` - -_Description_: - -The `RETRY_SERIALIZABLE` error occurs in the following cases: - -1. When a transaction _A_ has its timestamp moved forward (also known as _A_ being "pushed") as CockroachDB attempts to find a serializable transaction ordering. Specifically, transaction _A_ tried to write a key that transaction _B_ had already read, and _B_ was supposed to be serialized after _A_ (i.e., _B_ had a higher timestamp than _A_). CockroachDB will try to serialize _A_ after _B_ by changing _A_'s timestamp, but it cannot do that when another transaction has subsequently written to some of the keys that _A_ has read and returned to the client. When that happens, the `RETRY_SERIALIZATION` error is signalled. For more information about how timestamp pushes work in our transaction model, see the [architecture docs on the transaction layer's timestamp cache](architecture/transaction-layer.html). - -2. When a [high-priority transaction](transactions.html#transaction-priorities) _A_ does a read that runs into a write intent from another lower-priority transaction _B_, and some other transaction _C_ writes to a key that _B_ has already read. Transaction _B_ will get this error when it tries to commit, because _A_ has already read some of the data touched by _B_ and returned results to the client, and _C_ has written data previously read by _B_. - -3. When a transaction _A_ is forced to refresh (i.e., change its timestamp) due to hitting the maximum [_closed timestamp_](architecture/transaction-layer.html#closed-timestamps) interval (closed timestamps enable [Follower Reads](follower-reads.html#how-follower-reads-work) and [Change Data Capture (CDC)](stream-data-out-of-cockroachdb-using-changefeeds.html)). This can happen when transaction _A_ is a long-running transaction, and there is a write by another transaction to data that _A_ has already read. If this is the cause of the error, the solution is to increase the `kv.closed_timestamp.target_duration` setting to a higher value. Unfortunately, there is no indication from this error code that a too-low closed timestamp setting is the issue. Therefore, you may need to rule out cases 1 and 2 (or experiment with increasing the closed timestamp interval, if that is possible for your application - see the note below). - -_Action_: - -1. If you encounter case 1 or 2 above, the solution is to: - 1. Retry transaction _A_ as described in [client-side retry handling](transactions.html#client-side-intervention). - 2. Design your schema and queries to reduce contention. For more information about how contention occurs and how to avoid it, see [Understanding and avoiding transaction contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention). In particular, if you are able to [send all of the statements in your transaction in a single batch](transactions.html#batched-statements), CockroachDB can usually automatically retry the entire transaction for you. - 3. Use historical reads with [`SELECT ... AS OF SYSTEM TIME`](as-of-system-time.html). - -2. If you encounter case 3 above, the solution is to: - 1. Increase the `kv.closed_timestamp.target_duration` setting to a higher value. As described above, this will impact the freshness of data available via [Follower Reads](follower-reads.html) and [CDC changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html). - 2. Retry transaction _A_ as described in [client-side retry handling](transactions.html#client-side-intervention). - 3. Design your schema and queries to reduce contention. For more information about how contention occurs and how to avoid it, see [Understanding and avoiding transaction contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention). In particular, if you are able to [send all of the statements in your transaction in a single batch](transactions.html#batched-statements), CockroachDB can usually automatically retry the entire transaction for you. - 3. Use historical reads with [`SELECT ... AS OF SYSTEM TIME`](as-of-system-time.html). - -{{site.data.alerts.callout_info}} -If you increase the `kv.closed_timestamp.target_duration` setting, it means that you are increasing the amount of time by which the data available in [Follower Reads](follower-reads.html) and [CDC changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html) lags behind the current state of the cluster. In other words, there is a trade-off here: if you absolutely must execute long-running transactions that execute concurrently with other transactions that are writing to the same data, you may have to settle for longer delays on Follower Reads and/or CDC to avoid frequent serialization errors. The anomaly that would be exhibited if these transactions were not retried is called [write skew](https://www.cockroachlabs.com/blog/what-write-skew-looks-like/). -{{site.data.alerts.end}} - -### RETRY_ASYNC_WRITE_FAILURE - -``` -TransactionRetryWithProtoRefreshError: ... RETRY_ASYNC_WRITE_FAILURE ... -``` - -_Description_: - -The `RETRY_ASYNC_WRITE_FAILURE` error occurs when some kind of problem with your cluster's operation occurs at the moment of a previous write in the transaction, causing CockroachDB to fail to replicate one of the transaction's writes. For example, this can happen if you have a networking partition that cuts off access to some nodes in your cluster. - -_Action_: - -1. Retry the transaction as described in [client-side retry handling](transactions.html#client-side-intervention). This is worth doing because the problem with the cluster is likely to be transient. -2. Investigate the problems with your cluster. For cluster troubleshooting information, see [Troubleshoot Cluster Setup](cluster-setup-troubleshooting.html). - -### ReadWithinUncertaintyInterval - -``` -TransactionRetryWithProtoRefreshError: ReadWithinUncertaintyIntervalError: - read at time 1591009232.376925064,0 encountered previous write with future timestamp 1591009232.493830170,0 within uncertainty interval `t <= 1591009232.587671686,0`; - observed timestamps: [{1 1591009232.587671686,0} {5 1591009232.376925064,0}] -``` - -_Description_: - -The `ReadWithinUncertaintyIntervalError` can occur when two transactions which start on different gateway nodes attempt to operate on the same data at close to the same time, and one of the operations is a write. The uncertainty comes from the fact that we cannot tell which one started first - the clocks on the two gateway nodes may not be perfectly in sync. - -For example, if the clock on node _A_ is ahead of the clock on node _B_, a transaction started on node _A_ may be able to commit a write with a timestamp that is still in the "future" from the perspective of node _B_. A later transaction that starts on node _B_ should be able to see the earlier write from node _A_, even if _B_'s clock has not caught up to _A_. The "read within uncertainty interval" occurs if we discover this situation in the middle of a transaction, when it is too late for the database to handle it automatically. When node _B_'s transaction retries, it will unambiguously occur after the transaction from node _A_. - -{{site.data.alerts.callout_info}} -This behavior is non-deterministic: it depends on which node is the [leaseholder](architecture/life-of-a-distributed-transaction.html#leaseholder-node) of the underlying data range. It’s generally a sign of contention. Uncertainty errors are always possible with near-realtime reads under contention. -{{site.data.alerts.end}} - -_Action_: - -The solution is to do one of the following: - -1. Be prepared to retry on uncertainty (and other) errors, as described in [client-side retry handling](transactions.html#client-side-intervention). -2. Use historical reads with [`SELECT ... AS OF SYSTEM TIME`](as-of-system-time.html). -3. Design your schema and queries to reduce contention. For more information about how contention occurs and how to avoid it, see [Understanding and avoiding transaction contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention). In particular, if you are able to [send all of the statements in your transaction in a single batch](transactions.html#batched-statements), CockroachDB can usually automatically retry the entire transaction for you. -4. If you [trust your clocks](operational-faqs.html#what-happens-when-node-clocks-are-not-properly-synchronized), you can try lowering the [`--max-offset` option to `cockroach start`](cockroach-start.html#flags), which provides an upper limit on how long a transaction can continue to restart due to uncertainty. - -{{site.data.alerts.callout_info}} -Uncertainty errors are a form of transaction conflict. For more information about transaction conflicts, see [Transaction conflicts](architecture/transaction-layer.html#transaction-conflicts). -{{site.data.alerts.end}} - -### RETRY_COMMIT_DEADLINE_EXCEEDED - -``` -TransactionRetryWithProtoRefreshError: TransactionPushError: transaction deadline exceeded ... -``` - -_Description_: - -The `RETRY_COMMIT_DEADLINE_EXCEEDED` error means that the transaction timed out due to being pushed by other concurrent transactions. This error is most likely to happen to long-running transactions. The conditions that trigger this error are very similar to the conditions that lead to a [`RETRY_SERIALIZABLE`](#retry_serializable) error, except that a transaction that hits this error got pushed for several minutes, but did not hit any of the conditions that trigger a `RETRY_SERIALIZABLE` error. In other words, the conditions that trigger this error are a subset of those that trigger `RETRY_SERIALIZABLE`, and that this transaction ran for too long (several minutes). - -{{site.data.alerts.callout_info}} -Read-only transactions do not get pushed, so they do not run into this error. -{{site.data.alerts.end}} - -This error occurs in the cases described below. - -1. When a transaction _A_ has its timestamp moved forward (also known as _A_ being "pushed") as CockroachDB attempts to find a serializable transaction ordering. Specifically, transaction _A_ tried to write a key that transaction _B_ had already read. _B_ was supposed to be serialized after _A_ (i.e., _B_ had a higher timestamp than _A_). CockroachDB will try to serialize _A_ after _B_ by changing _A_'s timestamp. - -2. When a [high-priority transaction](transactions.html#transaction-priorities) _A_ does a read that runs into a write intent from another lower-priority transaction _B_. Transaction _B_ may get this error when it tries to commit, because _A_ has already read some of the data touched by _B_ and returned results to the client. - -3. When a transaction _A_ is forced to refresh (change its timestamp) due to hitting the maximum [_closed timestamp_](architecture/transaction-layer.html#closed-timestamps) interval (closed timestamps enable [Follower Reads](follower-reads.html#how-follower-reads-work) and [Change Data Capture (CDC)](stream-data-out-of-cockroachdb-using-changefeeds.html)). This can happen when transaction _A_ is a long-running transaction, and there is a write by another transaction to data that _A_ has already read. - -_Action_: - -1. The `RETRY_COMMIT_DEADLINE_EXCEEDED` error is one case where the standard advice to add a retry loop to your application may not be advisable. A transaction that runs for long enough to get pushed beyond its deadline is quite likely to fail again on retry for the same reasons. Therefore, the best thing to do in this case is to shrink the running time of your transactions so they complete more quickly and do not hit the deadline. -2. If you encounter case 3 above, you can increase the `kv.closed_timestamp.target_duration` setting to a higher value. Unfortunately, there is no indication from this error code that a too-low closed timestamp setting is the issue. Therefore, you may need to rule out cases 1 and 2 (or experiment with increasing the closed timestamp interval, if that is possible for your application - see the note below). - -{{site.data.alerts.callout_info}} -If you increase the `kv.closed_timestamp.target_duration` setting, it means that you are increasing the amount of time by which the data available in [Follower Reads](follower-reads.html) and [CDC changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html) lags behind the current state of the cluster. In other words, there is a trade-off here: if you absolutely must execute long-running transactions that execute concurrently with other transactions that are writing to the same data, you may have to settle for longer delays on Follower Reads and/or CDC to avoid frequent serialization errors. The anomaly that would be exhibited if these transactions were not retried is called [write skew](https://www.cockroachlabs.com/blog/what-write-skew-looks-like/). -{{site.data.alerts.end}} - -### ABORT_REASON_ABORTED_RECORD_FOUND - -``` -TransactionRetryWithProtoRefreshError:TransactionAbortedError(ABORT_REASON_ABORTED_RECORD_FOUND) ... -``` - -_Description_: - -The `ABORT_REASON_ABORTED_RECORD_FOUND` error means that the client application is trying to use a transaction that has been aborted. This happens in one of the following cases: - -- Write-write conflict: Another [high-priority transaction](transactions.html#transaction-priorities) _B_ encountered a write intent by our transaction _A_, and tried to push _A_'s timestamp. -- Cluster overload: _B_ thinks that _A_'s transaction coordinator node is dead, because the coordinator node hasn't heartbeated the transaction record for a few seconds. -- Deadlock: Some transaction _B_ is trying to acquire conflicting locks in reverse order from transaction _A_. - -_Action_: - -If you are encountering deadlocks: - -- Avoid producing deadlocks in your application by making sure that transactions acquire locks in the same order. - -If you are using only default [transaction priorities](transactions.html#transaction-priorities): - -- This error means your cluster has problems. You are likely overloading it. Investigate the source of the overload, and do something about it. For more information, see [Node liveness issues](cluster-setup-troubleshooting.html#node-liveness-issues). - -If you are using [high- or low-priority transactions](transactions.html#transaction-priorities): - -- Update your app to retry on serialization errors (where `SQLSTATE` is `40001`), as described in [client-side retry handling](transactions.html#client-side-intervention). -- Design your schema and queries to reduce contention. For more information about how contention occurs and how to avoid it, see [Understanding and avoiding transaction contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention). In particular, if you are able to [send all of the statements in your transaction in a single batch](transactions.html#batched-statements), CockroachDB can usually automatically retry the entire transaction for you. - -### ABORT_REASON_CLIENT_REJECT - -``` -TransactionRetryWithProtoRefreshError:TransactionAbortedError(ABORT_REASON_CLIENT_REJECT) ... -``` - -_Description_: - -The `ABORT_REASON_CLIENT_REJECT` error is caused by the same conditions as the [`ABORT_REASON_ABORTED_RECORD_FOUND`](#abort_reason_aborted_record_found), and requires the same actions. The errors are fundamentally the same, except that they are discovered at different points in the process. - -### ABORT_REASON_PUSHER_ABORTED - -``` -TransactionRetryWithProtoRefreshError:TransactionAbortedError(ABORT_REASON_PUSHER_ABORTED) ... -``` - -_Description_: - -The `ABORT_REASON_PUSHER_ABORTED` error is caused by the same conditions as the [`ABORT_REASON_ABORTED_RECORD_FOUND`](#abort_reason_aborted_record_found), and requires the same actions. The errors are fundamentally the same, except that they are discovered at different points in the process. - -### ABORT_REASON_ABORT_SPAN - -``` -TransactionRetryWithProtoRefreshError:TransactionAbortedError(ABORT_REASON_ABORT_SPAN) ... -``` - -_Description_: - -The `ABORT_REASON_ABORT_SPAN` error is caused by the same conditions as the [`ABORT_REASON_ABORTED_RECORD_FOUND`](#abort_reason_aborted_record_found), and requires the same actions. The errors are fundamentally the same, except that they are discovered at different points in the process. - -### ABORT_REASON_NEW_LEASE_PREVENTS_TXN - -``` -TransactionRetryWithProtoRefreshError:TransactionAbortedError(ABORT_REASON_NEW_LEASE_PREVENTS_TXN) ... -``` - -_Description_: - -The `ABORT_REASON_NEW_LEASE_PREVENTS_TXN` error occurs because the timestamp cache will not allow transaction _A_ to create a transaction record. A new lease wipes the timestamp cache, so this could mean the [leaseholder](architecture/life-of-a-distributed-transaction.html#leaseholder-node) was moved and the duration of transaction _A_ was unlucky enough to happen across a lease acquisition. In other words, leaseholders got shuffled out from underneath transaction _A_ (due to no fault of the client application or schema design), and now it has to be retried. - -_Action_: - -Retry transaction _A_ as described in [client-side retry handling](transactions.html#client-side-intervention). - -### ABORT_REASON_TIMESTAMP_CACHE_REJECTED - -``` -TransactionRetryWithProtoRefreshError:TransactionAbortedError(ABORT_REASON_TIMESTAMP_CACHE_REJECTED) ... -``` - -_Description_: - -The `ABORT_REASON_TIMESTAMP_CACHE_REJECTED` error occurs when the timestamp cache will not allow transaction _A_ to create a transaction record. This can happen due to a [range merge](architecture/distribution-layer.html#range-merges) happening in the background, or because the timestamp cache is an in-memory cache, and has outgrown its memory limit (about 64 MB). - -_Action_: - -Retry transaction _A_ as described in [client-side retry handling](transactions.html#client-side-intervention). - -## See also - -- [Common Errors](common-errors.html) -- [Transactions](transactions.html) -- [Client-side retry handling](transactions.html#client-side-intervention) -- [Understanding and avoiding transaction contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention) -- [DB Console Transactions Page](ui-transactions-page.html) -- [Architecture - Transaction Layer](architecture/transaction-layer.html) diff --git a/src/current/v21.1/transactions.md b/src/current/v21.1/transactions.md deleted file mode 100644 index ab5d947e1bb..00000000000 --- a/src/current/v21.1/transactions.md +++ /dev/null @@ -1,326 +0,0 @@ ---- -title: Transactions -summary: CockroachDB supports bundling multiple SQL statements into a single all-or-nothing transaction. -toc: true ---- - -CockroachDB supports bundling multiple SQL statements into a single all-or-nothing transaction. Each transaction guarantees [ACID semantics](https://en.wikipedia.org/wiki/ACID) spanning arbitrary tables and rows, even when data is distributed. If a transaction succeeds, all mutations are applied together with virtual simultaneity. If any part of a transaction fails, the entire transaction is aborted, and the database is left unchanged. CockroachDB guarantees that while a transaction is pending, it is isolated from other concurrent transactions with serializable [isolation](#isolation-levels). - -For a detailed discussion of CockroachDB transaction semantics, see [How CockroachDB Does Distributed Atomic Transactions](https://www.cockroachlabs.com/blog/how-cockroachdb-distributes-atomic-transactions/) and [Serializable, Lockless, Distributed: Isolation in CockroachDB](https://www.cockroachlabs.com/blog/serializable-lockless-distributed-isolation-cockroachdb/). The explanation of the transaction model described in this blog post is slightly out of date. See the [Transaction Retries](#transaction-retries) section for more details. - -## SQL statements - -The following SQL statements control transactions. - -| Statement | Description | -|------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [`BEGIN`](begin-transaction.html) | Initiate a transaction, as well as control its [priority](#transaction-priorities). | -| [`SET TRANSACTION`](set-transaction.html) | Control a transaction's [priority](#transaction-priorities). | -| [`COMMIT`](commit-transaction.html) | Commit a regular transaction, or clear the connection after committing a transaction using the [advanced retry protocol](advanced-client-side-transaction-retries.html). | -| [`ROLLBACK`](rollback-transaction.html) | Abort a transaction and roll the database back to its state before the transaction began. | -| [`SHOW`](show-vars.html) | Display the current transaction settings. | -| [`SAVEPOINT`](savepoint.html) | Used for [nested transactions](#nested-transactions); also used to implement [advanced client-side transaction retries](advanced-client-side-transaction-retries.html). | -| [`RELEASE SAVEPOINT`](release-savepoint.html) | Commit a [nested transaction](#nested-transactions); also used for [retryable transactions](advanced-client-side-transaction-retries.html). | -| [`ROLLBACK TO SAVEPOINT`](rollback-transaction.html) | Roll back a [nested transaction](#nested-transactions); also used to handle [retryable transaction errors](advanced-client-side-transaction-retries.html). | - -{{site.data.alerts.callout_info}} -If you are using a framework or library that does not have [advanced retry logic](advanced-client-side-transaction-retries.html) built in, you should implement an application-level retry loop with exponential backoff. See [Client-side intervention](#client-side-intervention). -{{site.data.alerts.end}} - -## Syntax - -In CockroachDB, a transaction is set up by surrounding SQL statements with the [`BEGIN`](begin-transaction.html) and [`COMMIT`](commit-transaction.html) statements. - -To use [advanced client-side transaction retries](advanced-client-side-transaction-retries.html), you should also include the [`SAVEPOINT`](savepoint.html), [`ROLLBACK TO SAVEPOINT`](rollback-transaction.html) and [`RELEASE SAVEPOINT`](release-savepoint.html) statements. - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; - -> SAVEPOINT cockroach_restart; - - - -> RELEASE SAVEPOINT cockroach_restart; - -> COMMIT; -~~~ - -At any time before it's committed, you can abort the transaction by executing the [`ROLLBACK`](rollback-transaction.html) statement. - -Clients using transactions must also include logic to handle [retries](#transaction-retries). - -## Error handling - -To handle errors in transactions, you should check for the following types of server-side errors: - -Type | Description ------|------------ -**Retry Errors** | Errors with the code `40001` or string `retry transaction`, which indicate that a transaction failed because it could not be placed in a serializable ordering of transactions by CockroachDB. This is often due to contention: conflicts with another concurrent or recent transaction accessing the same data. In such cases, the transaction needs to be retried by the client as described in [client-side intervention](#client-side-intervention). For a reference listing all of the retry error codes emitted by CockroachDB, see the [Transaction Retry Error Reference](transaction-retry-error-reference.html). -**Ambiguous Errors** | Errors with the code `40003` which indicate that the state of the transaction is ambiguous, i.e., you cannot assume it either committed or failed. How you handle these errors depends on how you want to resolve the ambiguity. For information about how to handle ambiguous errors, see [here](common-errors.html#result-is-ambiguous). -**SQL Errors** | All other errors, which indicate that a statement in the transaction failed. For example, violating the `UNIQUE` constraint generates a `23505` error. After encountering these errors, you can either issue a [`COMMIT`](commit-transaction.html) or [`ROLLBACK`](rollback-transaction.html) to abort the transaction and revert the database to its state before the transaction began.

    If you want to attempt the same set of statements again, you must begin a completely new transaction. - -## Transaction retries - -Transactions may require retries if they experience deadlock or [read/write contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention) with other concurrent transactions which cannot be resolved without allowing potential [serializable anomalies](https://en.wikipedia.org/wiki/Serializability). - -To mitigate read-write contention and reduce the need for transaction retries, use the following techniques: - -- Perform reads using [`AS OF SYSTEM TIME`](performance-best-practices-overview.html#use-as-of-system-time-to-decrease-conflicts-with-long-running-queries). -- Use [`SELECT FOR UPDATE`](select-for-update.html) to order transactions by controlling concurrent access to one or more rows of a table. This reduces retries in scenarios where a transaction performs a read and then updates the same row it just read. - -There are two cases in which transaction retries can occur: - -- [Automatic retries](#automatic-retries), which CockroachDB processes for you. -- [Client-side intervention](#client-side-intervention), which your application must handle. - -### Automatic retries - -CockroachDB automatically retries individual statements (implicit transactions) and transactions sent from the client as a single batch, as long as the size of the results being produced for the client, including protocol overhead, is less than 16KiB by default. Once that buffer overflows, CockroachDB starts streaming results back to the client, at which point automatic retries cannot be performed any more. As long as the results of a single statement or batch of statements are known to stay clear of this limit, the client does not need to worry about transaction retries. - -You can change the results buffer size for all new sessions using the `sql.defaults.results_buffer.size` [cluster setting](cluster-settings.html), or for a specific session using the `results_buffer_size` [session variable](set-vars.html). Decreasing the buffer size can increase the number of transaction retry errors a client receives, whereas increasing the buffer size can increase the delay until the client receives the first result row. - -#### Individual statements - -Individual statements are treated as implicit transactions, and so they fall -under the rules described above. If the results are small enough, they will be -automatically retried. In particular, `INSERT/UPDATE/DELETE` statements without -a `RETURNING` clause are guaranteed to have minuscule result sizes. -For example, the following statement would be automatically retried by CockroachDB: - -~~~ sql -> DELETE FROM customers WHERE id = 1; -~~~ - -#### Batched statements - -Transactions can be sent from the client as a single batch. Batching implies that CockroachDB receives multiple statements without being asked to return results in between them; instead, CockroachDB returns results after executing all of the statements, except when the accumulated results overflow the buffer mentioned above, in which case they are returned sooner and automatic retries can no longer be performed. - -Batching is generally controlled by your driver or client's behavior. Technically, it can be achieved in two ways, both supporting automatic retries: - -1. When the client/driver is using the [PostgreSQL Extended Query protocol](https://www.postgresql.org/docs/10/static/protocol-flow.html#PROTOCOL-FLOW-EXT-QUERY), a batch is made up of all queries sent in between two `Sync` messages. Many drivers support such batches through explicit batching constructs. - -2. When the client/driver is using the [PostgreSQL Simple Query protocol](https://www.postgresql.org/docs/10/static/protocol-flow.html#id-1.10.5.7.4), a batch is made up of semicolon-separated strings sent as a unit to CockroachDB. For example, in Go, this code would send a single batch (which would be automatically retried): - - ~~~ go - db.Exec( - "BEGIN; - - DELETE FROM customers WHERE id = 1; - - DELETE orders WHERE customer = 1; - - COMMIT;" - ) - ~~~ - -Within a batch of statements, CockroachDB infers that the statements are not -conditional on the results of previous statements, so it can retry all of them. -Of course, if the transaction relies on conditional logic (e.g., statement 2 is -executed only for some results of statement 1), then the transaction cannot be -all sent to CockroachDB as a single batch. In these common cases, CockroachDB -cannot retry, say, statement 2 in isolation. Since results for statement 1 have -already been delivered to the client by the time statement 2 is forcing the -transaction to retry, the client needs to be involved in retrying the whole -transaction and so you should write your transactions to use -[client-side intervention](#client-side-intervention). - -### Client-side intervention - -Your application should include client-side retry handling when the statements are sent individually, such as: - -{% include_cached copy-clipboard.html %} -~~~ sql -> BEGIN; - -> UPDATE products SET inventory = 0 WHERE sku = '8675309'; - -> INSERT INTO orders (customer, status) VALUES (1, 'new'); - -> COMMIT; -~~~ - -To indicate that a transaction must be retried, CockroachDB signals an error with the code `40001` and an error message that begins with the string `"retry transaction"`. For a complete list of transaction retry error codes, see [Transaction retry error reference](transaction-retry-error-reference.html). - -To handle these types of errors, you have the following options: - -- If your database library or framework provides a method for retryable transactions (it will often be documented as a tool for handling deadlocks), use it. -- If you're building an application in the following languages, Cockroach Labs has created adapters that include automatic retry handling: - - **Go** developers using [GORM](https://github.com/jinzhu/gorm) or [pgx](https://github.com/jackc/pgx) can use the [`github.com/cockroachdb/cockroach-go/crdb`](https://github.com/cockroachdb/cockroach-go/tree/master/crdb) package. For an example, see [Build a Go App with CockroachDB](build-a-go-app-with-cockroachdb.html). - - **Python** developers using [SQLAlchemy](https://www.sqlalchemy.org) can use the [`sqlalchemy-cockroachdb` adapter](https://github.com/cockroachdb/sqlalchemy-cockroachdb). For an example, see [Build a Python App with CockroachDB and SQLAlchemy](build-a-python-app-with-cockroachdb-sqlalchemy.html). - - **Ruby (ActiveRecord)** developers can use the [`activerecord-cockroachdb-adapter`](https://rubygems.org/gems/activerecord-cockroachdb-adapter). For an example, see [Build a Ruby App with CockroachDB and ActiveRecord](build-a-ruby-app-with-cockroachdb-activerecord.html). -- If you're building an application with another driver or data access framework that is [supported by CockroachDB](third-party-database-tools.html), we recommend reusing the retry logic in our ["Simple CRUD" Example Apps](example-apps.html). For example, **Java** developers accessing the database with [JDBC](https://jdbc.postgresql.org) can reuse the example code implementing retry logic shown in [Build a Java app with CockroachDB](build-a-java-app-with-cockroachdb.html). -- If you're building an application with a language and framework for which we do not provide example retry logic, you might need to write your own retry logic. For an example, see the [Client-side intervention example](#client-side-intervention-example). -- **Advanced users, such as library authors**: See [Advanced Client-Side Transaction Retries](advanced-client-side-transaction-retries.html). - -#### Client-side intervention example - -{% include {{page.version.version}}/misc/client-side-intervention-example.md %} - -## Transaction contention - -Transactions in CockroachDB lock data resources that are written during their execution. When a pending write from one transaction conflicts with a write of a concurrent transaction, the concurrent transaction must wait for the earlier transaction to complete before proceeding. When a dependency cycle is detected between transactions, the transaction with the higher priority aborts the dependent transaction to avoid deadlock, which must be [retried](#client-side-intervention). - -For more details about transaction contention and best practices for avoiding contention, see [Understanding and Avoiding Transaction Contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention). - -## Nested transactions - -CockroachDB supports the nesting of transactions using [savepoints](savepoint.html). These nested transactions are also known as sub-transactions. Nested transactions can be rolled back without discarding the state of the entire surrounding transaction. - -This can be useful in applications that abstract database access using an application development framework or [ORM](install-client-drivers.html). Different components of the application can operate on different sub-transactions without having to know about each others' internal operations, while trusting that the database will maintain isolation between sub-transactions and preserve data integrity. - -Just as [`COMMIT`](commit-transaction.html) and [`ROLLBACK`](rollback-transaction.html) are used to commit and discard entire transactions, respectively, [`RELEASE SAVEPOINT`](release-savepoint.html) and [`ROLLBACK TO SAVEPOINT`](rollback-transaction.html#rollback-a-nested-transaction) are used to commit and discard nested transactions. This relationship is shown in the following table: - -| Statement | Effect | -|------------------------------------------------------------------------------------+-------------------------------------------------------| -| [`COMMIT`](commit-transaction.html) | Commit an entire transaction. | -| [`ROLLBACK`](rollback-transaction.html) | Discard an entire transaction. | -| [`RELEASE SAVEPOINT`](release-savepoint.html) | Commit (really, forget) the named nested transaction. | -| [`ROLLBACK TO SAVEPOINT`](rollback-transaction.html#rollback-a-nested-transaction) | Discard the changes in the named nested transaction. | - -For more information, including examples showing how to use savepoints to create nested transactions, see [the savepoints documentation](savepoint.html#examples). - -## Transaction priorities - -Every transaction in CockroachDB is assigned an initial **priority**. By default, the transaction priority is `NORMAL`. - -### Set transaction priority - -For transactions that should be given higher or lower preference in [high-contention scenarios](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention), you can set the priority in the [`BEGIN`](begin-transaction.html) statement: - -~~~ sql -> BEGIN PRIORITY ; -~~~ - -You can also set the priority immediately after a transaction is started: - -~~~ sql -> SET TRANSACTION PRIORITY ; -~~~ - -To set the default transaction priority for all transactions in a session, use the `default_transaction_priority` [session variable](set-vars.html). For example: - -~~~ sql -> SET default_transaction_priority = 'high'; -~~~ - -{{site.data.alerts.callout_info}} -When two transactions contend for the same resources indirectly, they may create a dependency cycle leading to a deadlock situation, where both transactions are waiting on the other to finish. In these cases, CockroachDB allows the transaction with higher priority to abort the other, which must then retry. On retry, the transaction inherits the higher priority. This means that each retry makes a transaction more likely to succeed in the event it again experiences deadlock. -{{site.data.alerts.end}} - -### View transaction priority - -`transaction_priority` is a read-only [session variable](show-vars.html). - -To view the current priority of a transaction, use `SHOW transaction_priority` or [`SHOW TRANSACTION PRIORITY`](show-vars.html): - -~~~ sql -> SHOW transaction_priority; -~~~ - -~~~ - transaction_priority ------------------------- - high -~~~ - -~~~ sql -> SHOW TRANSACTION PRIORITY; -~~~ - -~~~ - transaction_priority ------------------------- - high -~~~ - -## Isolation levels - -CockroachDB executes all transactions at the strongest ANSI transaction isolation level: -`SERIALIZABLE`. All other ANSI transaction isolation levels (e.g., `SNAPSHOT`, `READ UNCOMMITTED`, -`READ COMMITTED`, and `REPEATABLE READ`) are automatically upgraded to `SERIALIZABLE`. Weaker -isolation levels have historically been used to maximize transaction throughput. However, -[ACIDRain: Concurrency-Related Attacks on Database-Backed Web Applications](http://www.bailis.org/papers/acidrain-sigmod2017.pdf) has demonstrated that the use -of weak isolation levels results in substantial vulnerability to concurrency-based attacks. - -For a detailed discussion of isolation in CockroachDB transactions, see [Serializable, Lockless, Distributed: Isolation in CockroachDB](https://www.cockroachlabs.com/blog/serializable-lockless-distributed-isolation-cockroachdb/). - -#### Serializable isolation - -With `SERIALIZABLE` isolation, a transaction behaves as though it has the entire database all to itself for the duration of its execution. This means that no concurrent writers can affect the transaction unless they commit before it starts, and no concurrent readers can be affected by the transaction until it has successfully committed. This is the strongest level of isolation provided by CockroachDB and it's the default. - -`SERIALIZABLE` isolation permits no anomalies. To prevent [write skew](https://en.wikipedia.org/wiki/Snapshot_isolation) anomalies, `SERIALIZABLE` isolation may require transaction restarts. For a demonstration of `SERIALIZABLE` preventing write skew, see [Serializable Transactions](demo-serializable.html). - -### Comparison to ANSI SQL isolation levels - -CockroachDB uses slightly different isolation levels than [ANSI SQL isolation levels](https://en.wikipedia.org/wiki/Isolation_(database_systems)#Isolation_levels). - -#### Aliases - -`SNAPSHOT`, `READ UNCOMMITTED`, `READ COMMITTED`, and `REPEATABLE READ` are aliases for `SERIALIZABLE`. - -#### Comparison - -The CockroachDB `SERIALIZABLE` level is stronger than the ANSI SQL `READ UNCOMMITTED`, `READ COMMITTED`, and `REPEATABLE READ` levels and equivalent to the ANSI SQL `SERIALIZABLE` level. - -For more information about the relationship between these levels, see [A Critique of ANSI SQL Isolation Levels](https://arxiv.org/ftp/cs/papers/0701/0701157.pdf). - -## Limit the number of rows written or read in a transaction - -You can limit the number of rows written or read in a transaction at the cluster or session level. This allows you configure CockroachDB to log or reject statements that could destabilize a cluster or violate application best practices. - -Use the [cluster](cluster-settings.html) and [session](set-vars.html) settings `sql.defaults.transaction_rows_written_log`, -`sql.defaults.transaction_rows_written_err`, `sql.defaults.transaction_rows_read_log`, and -`sql.defaults.transaction_rows_read_err` to limit the number of rows written or read in a -transaction. When the `log` limit is exceeded, the transaction is logged to the `SQL_PERF` channel. -When the `err` limit is exceeded, the transaction is rejected. The limits are enforced after each -statement of a transaction has been fully executed. - -The "write" limits apply to `INSERT`, `INSERT INTO SELECT FROM`, `INSERT ON CONFLICT`, `UPSERT`, `UPDATE`, -and `DELETE` SQL statements. The "read" limits apply to the `SELECT` -statement in addition to the statements subject to the "write" limits. The limits **do not** -apply to `CREATE TABLE AS`, `SELECT`, `IMPORT`, `TRUNCATE`, `DROP`, `ALTER TABLE`, `BACKUP`, -`RESTORE`, or `CREATE STATISTICS` statements. - -{{site.data.alerts.callout_info}} -Enabling `transaction_rows_read_err` disables the auto commit optimization for mutation statements -in implicit transactions. For write limits CockroachDB can count how many rows have been modified -before using the auto commit optimization. However, for read limits CockroachDB doesn't have that -information on the write path, so must disable the auto commit. -{{site.data.alerts.end}} - -## Limit the number of rows written or read in a transaction - -You can limit the number of rows written or read in a transaction at the cluster or session level. This allows you configure CockroachDB to log or reject statements that could destabilize a cluster or violate application best practices. - -Use the [cluster settings](cluster-settings.html) `sql.defaults.transaction_rows_written_log`, -`sql.defaults.transaction_rows_written_err`, `sql.defaults.transaction_rows_read_log`, and -`sql.defaults.transaction_rows_read_err` and [session settings](set-vars.html) `transaction_rows_written_log`, -`transaction_rows_written_err`, `transaction_rows_read_log`, and -`transaction_rows_read_err` to limit the number of rows written or read in a -transaction. When the `log` limit is reached, the transaction is logged to the `SQL_PERF` channel. -When the `err` limit is reached, the transaction is rejected. The limits are enforced after each -statement of a transaction has been fully executed. - -The "write" limits apply to `INSERT`, `INSERT INTO SELECT FROM`, `INSERT ON CONFLICT`, `UPSERT`, `UPDATE`, -and `DELETE` SQL statements. The "read" limits apply to the `SELECT` -statement in addition to the statements subject to the "write" limits. The limits **do not** -apply to `CREATE TABLE AS`, `SELECT`, `IMPORT`, `TRUNCATE`, `DROP`, `ALTER TABLE`, `BACKUP`, -`RESTORE`, or `CREATE STATISTICS` statements. - -{{site.data.alerts.callout_info}} -Enabling `transaction_rows_read_err` disables a performance optimization for mutation statements in implicit transactions where CockroachDB can auto-commit without additional network round trips. -{{site.data.alerts.end}} - -## See also - -- [`BEGIN`](begin-transaction.html) -- [`COMMIT`](commit-transaction.html) -- [`ROLLBACK`](rollback-transaction.html) -- [`SAVEPOINT`](savepoint.html) -- [`RELEASE SAVEPOINT`](release-savepoint.html) -- [`SHOW`](show-vars.html) -- [Retryable transaction example code in Java using JDBC](build-a-java-app-with-cockroachdb.html) -- [DB Console Transactions Page](ui-transactions-page.html) -- [CockroachDB Architecture: Transaction Layer](architecture/transaction-layer.html) -- [Transaction retry error reference](transaction-retry-error-reference.html) diff --git a/src/current/v21.1/troubleshooting-overview.md b/src/current/v21.1/troubleshooting-overview.md deleted file mode 100644 index abeed628e47..00000000000 --- a/src/current/v21.1/troubleshooting-overview.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -title: Troubleshooting Overview -summary: Initial steps to take if you run in to issues with CockroachDB. -toc: false ---- - -If you run into issues with CockroachDB, there are a few initial steps you can always take: - -1. Check your [logs](logging-overview.html) for errors related to your issue. - - Logs are generated on a per-node basis, so you must either identify the node where the issue occurred or [collect the logs from all active nodes in your cluster](cockroach-debug-zip.html). - -2. Check our list of [common errors](common-errors.html) for a solution. If you are getting transaction retry errors, see [client-side retry handling](transactions.html#client-side-intervention) and the [Transaction Retry Error Reference](transaction-retry-error-reference.html). - -3. If the problem doesn't match a common error, try the following pages: - - [Troubleshoot Cluster Setup](cluster-setup-troubleshooting.html) helps start your cluster and scale it by adding nodes. - - [Troubleshoot Query Behavior](query-behavior-troubleshooting.html) helps with unexpected query results. - -4. If you cannot resolve the issue easily yourself, the following tools can help you get unstuck: - - [Support Resources](support-resources.html) identifies ways you can get help with troubleshooting. - - [File an Issue](file-an-issue.html) provides details about filing issues that you're unable to resolve. diff --git a/src/current/v21.1/truncate.md b/src/current/v21.1/truncate.md deleted file mode 100644 index 398a66512e4..00000000000 --- a/src/current/v21.1/truncate.md +++ /dev/null @@ -1,160 +0,0 @@ ---- -title: TRUNCATE -summary: The TRUNCATE statement deletes all rows from specified tables. -toc: true ---- - -The `TRUNCATE` [statement](sql-statements.html) removes all rows from a table. At a high level, it works by dropping the table and recreating a new table with the same name. - -{% include {{{ page.version.version }}/misc/schema-change-stmt-note.md %} - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/truncate.html %} -
    - -## Required privileges - -The user must have the `DROP` [privilege](authorization.html#assign-privileges) on the table. - -## Parameters - -Parameter | Description -----------|------------ -`table_name` | The name of the table to truncate. -`CASCADE` | Truncate all tables with [Foreign Key](foreign-key.html) dependencies on the table being truncated.

    `CASCADE` does not list dependent tables it truncates, so should be used cautiously. -`RESTRICT` | _(Default)_ Do not truncate the table if any other tables have [Foreign Key](foreign-key.html) dependencies on it. - -## Limitations - -`TRUNCATE` is a schema change, and as such is not transactional. For more information about how schema changes work, see [Online Schema Changes](online-schema-changes.html). - -## Viewing schema changes - -{% include {{ page.version.version }}/misc/schema-change-view-job.md %} - -## Examples - -### Truncate a table (no foreign key dependencies) - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM t1; -~~~ - -~~~ -+----+------+ -| id | name | -+----+------+ -| 1 | foo | -| 2 | bar | -+----+------+ -(2 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> TRUNCATE t1; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM t1; -~~~ - -~~~ -+----+------+ -| id | name | -+----+------+ -+----+------+ -(0 rows) -~~~ - -### Truncate a table and dependent tables - -In these examples, the `orders` table has a [Foreign Key](foreign-key.html) relationship to the `customers` table. Therefore, it's only possible to truncate the `customers` table while simultaneously truncating the dependent `orders` table, either using `CASCADE` or explicitly. - -#### Truncate dependent tables using `CASCADE` - -{{site.data.alerts.callout_danger}}CASCADE truncates all dependent tables without listing them, which can lead to inadvertent and difficult-to-recover losses. To avoid potential harm, we recommend truncating tables explicitly in most cases. See Truncate Dependent Tables Explicitly for more details.{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ sql -> TRUNCATE customers; -~~~ - -~~~ -pq: "customers" is referenced by foreign key from table "orders" -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> TRUNCATE customers CASCADE; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM customers; -~~~ - -~~~ -+----+-------+ -| id | email | -+----+-------+ -+----+-------+ -(0 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM orders; -~~~ - -~~~ -+----+----------+------------+ -| id | customer | orderTotal | -+----+----------+------------+ -+----+----------+------------+ -(0 rows) -~~~ - -#### Truncate dependent tables explicitly - -{% include_cached copy-clipboard.html %} -~~~ sql -> TRUNCATE customers, orders; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM customers; -~~~ - -~~~ -+----+-------+ -| id | email | -+----+-------+ -+----+-------+ -(0 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM orders; -~~~ - -~~~ -+----+----------+------------+ -| id | customer | orderTotal | -+----+----------+------------+ -+----+----------+------------+ -(0 rows) -~~~ - -## See also - -- [`DELETE`](delete.html) -- [`SHOW JOBS`](show-jobs.html) -- [Foreign Key constraint](foreign-key.html) -- [Online Schema Changes](online-schema-changes.html) diff --git a/src/current/v21.1/ui-cdc-dashboard.md b/src/current/v21.1/ui-cdc-dashboard.md deleted file mode 100644 index 989cfd7f5f3..00000000000 --- a/src/current/v21.1/ui-cdc-dashboard.md +++ /dev/null @@ -1,74 +0,0 @@ ---- -title: Changefeeds Dashboard -summary: The Changefeeds dashboard lets you monitor the changefeeds created across your cluster. -toc: true ---- - -The **Changefeeds** dashboard in the DB Console lets you monitor the [changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html) created across your cluster. To view this dashboard, [access the DB Console](ui-overview.html#db-console-access), click **Metrics** on the left-hand navigation bar, and then select **Dashboard** > **Changefeeds**. - -{% include {{ page.version.version }}/ui/ui-metrics-navigation.md %} - -The **Changefeeds** dashboard displays the following time series graphs: - -## Max Changefeed Latency - -This graph shows the maximum latency for resolved timestamps of any running changefeed. - -DB Console Max Changefeed Latency graph - -{{site.data.alerts.callout_info}} -The maximum latency for resolved timestamps is distinct from and slower than the commit-to-emit latency for individual change messages. For more information about resolved timestamps, see [Ordering guarantees](stream-data-out-of-cockroachdb-using-changefeeds.html#ordering-guarantees). -{{site.data.alerts.end}} - -## Sink Byte Traffic - -This graph shows the number of bytes emitted by CockroachDB into the sink for changefeeds. - -DB Console Sink Byte Traffic graph - -Metric | Description ---------|---- -**Emitted Bytes** | The number of bytes emitted by CockroachDB into the sink for changefeeds. - -## Sink Counts - -This graph shows: - -- The number of messages that CockroachDB sent to the sink. -- The number of flushes that the sink performed for changefeeds. - -DB Console Sink Counts graph - -Metric | Description ---------|---- -**Messages** | The number of messages that CockroachDB sent to the sink for changefeeds. -**Flushes** | The the number of flushes that the sink performed for changefeeds. - -## Sink Timings - -This graph shows: - -- The time in milliseconds per second required by CockroachDB to send messages to the sink. -- The time CockroachDB spent waiting for the sink to flush the messages for changefeeds. - -DB Console Sink Timings graph - -Metric | Description ---------|---- -**Message Emit Time** | The time in milliseconds per second required by CockroachDB to send messages to the sink for changefeeds. -**Flush Time** | The time in milliseconds per second that CockroachDB spent waiting for the sink to flush the messages for changefeeds. - -## Changefeed Restarts - -This graph displays the number of times changefeeds restarted due to retryable errors. - -DB Console Changefeed Restarts graph - -{% include {{ page.version.version }}/ui/ui-summary-events.md %} - -## See also - -- [Stream Data Out of CockroachDB Using Changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html) -- [Troubleshooting Overview](troubleshooting-overview.html) -- [Support Resources](support-resources.html) -- [Raw Status Endpoints](monitoring-and-alerting.html#raw-status-endpoints) diff --git a/src/current/v21.1/ui-cluster-overview-page.md b/src/current/v21.1/ui-cluster-overview-page.md deleted file mode 100644 index b247d304a34..00000000000 --- a/src/current/v21.1/ui-cluster-overview-page.md +++ /dev/null @@ -1,149 +0,0 @@ ---- -title: Cluster Overview Page -summary: The Cluster Overview page of the DB Console displays key metrics about your cluster and individual nodes. -toc: true ---- - -The **Cluster Overview** page of the DB Console displays key metrics about your cluster and individual nodes. These include: - -- Liveness status -- Replication status -- Uptime -- Hardware usage - -If you have an [Enterprise license](enterprise-licensing.html), you can enable the [Node Map](#node-map-enterprise) view for a visual representation of your cluster's geographic layout. - -{{site.data.alerts.callout_success}} -Enter your email in the banner at the top to receive updates on CockroachDB [releases](../releases/). -{{site.data.alerts.end}} - -## Cluster Overview panel - -Use the **Cluster Overview** panel to quickly assess the capacity and health of your cluster. - -DB Console cluster overview - -Metric | Description ---------|---- -Capacity Usage |
    • Used: The total disk space in use by CockroachDB data across all nodes. This excludes the disk space used by the Cockroach binary, operating system, and other system files.
    • Usable: The total disk space usable by CockroachDB data across all nodes. This cannot exceed the store size, if one has been set using [`--store`](cockroach-start.html#store).
      • **Note:** Due to a [known issue](known-limitations.html#available-capacity-metric-in-the-db-console) in CockroachDB v21.1, the **Usable Capacity** shown may not be accurate if running multiple CockroachDB nodes locally on a single node. This is fixed in CockroachDB v21.2 and later.
    See [Capacity metrics](#capacity-metrics) for details on how these values are calculated. -Node Status |
    • The number of `LIVE` nodes in the cluster.
    • The number of `SUSPECT` nodes in the cluster. A node is considered suspect if its [liveness status is unavailable](cluster-setup-troubleshooting.html#node-liveness-issues) or the node is in the process of [decommissioning](#decommissioned-nodes).
    • The number of `DEAD` nodes in the cluster.
      • -Replication Status |
      • The total number of [ranges](architecture/overview.html#glossary) in the cluster.
      • The number of [under-replicated ranges](ui-replication-dashboard.html#review-of-cockroachdb-terminology) in the cluster. A non-zero number indicates an unstable cluster.
      • The number of [unavailable ranges](ui-replication-dashboard.html#review-of-cockroachdb-terminology) in the cluster. A non-zero number indicates an unstable cluster.
        • - -### Capacity metrics - -The [Cluster Overview](#cluster-overview-panel), [Node List](#node-list), and [Node Map](#node-map-enterprise) display **Capacity Usage** by the CockroachDB [store](architecture/storage-layer.html) (the directory on each node where CockroachDB reads and writes its data) as a percentage of the disk space that is **usable** on the cluster, locality, or node. - -{{site.data.alerts.callout_info}} -Due to a [known issue](known-limitations.html#available-capacity-metric-in-the-db-console) in CockroachDB v21.1, the **Capacity Usage** shown may not be accurate if running multiple CockroachDB nodes locally on a single node. This is fixed in CockroachDB v21.2 and later. -{{site.data.alerts.end}} - -Usable disk space is constrained by the following: - -- The maximum store size, which may be specified using the [`--store`](cockroach-start.html#store) flag when starting nodes. If no store size has been explicitly set, the actual disk capacity is used as the limit. This value is displayed on the Capacity graph in the [Storage dashboard](ui-storage-dashboard.html#capacity). -- Any disk space occupied by non-CockroachDB data. This may include the operating system and other system files, as well as the Cockroach binary itself. - -The DB Console thus calculates **usable** disk space as the sum of empty disk space, up to the value of the maximum store size, and disk space that is already being **used** by CockroachDB data. - -{{site.data.alerts.callout_info}} -{% include {{ page.version.version }}/misc/available-capacity-metric.md %} -{{site.data.alerts.end}} - -## Node List - -The **Node List** groups nodes by locality. The lowest-level locality tier is used to organize the Node List. Hover over a locality to see all localities for the group of nodes. - -{{site.data.alerts.callout_success}} -We recommend [defining `--locality` flags when starting nodes](cockroach-start.html#locality). CockroachDB uses locality to distribute replicas and mitigate [network latency](ui-network-latency-page.html). Locality is also a prerequisite for enabling the [Node Map](#node-map-enterprise). -{{site.data.alerts.end}} - -### Node status - -Each locality and node is displayed with its current operational status. - -Locality Status | Description --------|------------ -`LIVE` | All nodes in the locality are live. -`WARNING` | Locality has 1 or more `SUSPECT`, `DECOMMISSIONING`, or `DEAD` nodes (red indicates a dead node). - -Node Status | Description --------|------------ -`LIVE` | Node is online and updating its liveness record. -`SUSPECT` | Node has an [unavailable liveness status](cluster-setup-troubleshooting.html#node-liveness-issues). -`DECOMMISSIONING` | Node is in the [process of decommissioning](remove-nodes.html#how-it-works). -`DECOMMISSIONED` | Node has completed decommissioning, has been stopped, and has not [updated its liveness record](cluster-setup-troubleshooting.html#node-liveness-issues) for 5 minutes. -`DEAD` | Node has not [updated its liveness record](cluster-setup-troubleshooting.html#node-liveness-issues) for 5 minutes. - -{{site.data.alerts.callout_info}} -Nodes are considered dead once they have not [updated their liveness record](cluster-setup-troubleshooting.html#node-liveness-issues) for a certain amount of time (5 minutes by default). At this point, the [automated repair process](cockroach-quit.html#how-it-works) starts, wherein CockroachDB rebalances replicas from dead nodes to live nodes, using the unaffected replicas as sources. -{{site.data.alerts.end}} - -### Node details - -The following details are also shown. - -Column | Description --------|------------ -Node Count | Number of nodes in the locality. -Nodes | Nodes are grouped by locality and displayed with their address and node ID (the ID is the number that is prepended by `n`). Click the address to view node statistics. Hover over a row and click **Logs** to see the node's log. -Uptime | Amount of time the node has been running. -Replicas | Number of replicas on the node or in the locality. -Capacity Usage | Percentage of usable disk space occupied by CockroachDB data on the node or in the locality. See [Capacity metrics](#capacity-metrics). -Memory Usage | Memory used by CockroachDB as a percentage of the total memory on the node or in the locality. -vCPUs | Number of vCPUs on the machine. -Version | Build tag of the CockroachDB version installed on the node. - -### Decommissioned Nodes - -Nodes that have been [decommissioned](remove-nodes.html#how-it-works) will be listed in the table of **Recently Decommissioned Nodes**, indicating that they are removed from the cluster. You can see the full history of decommissioned nodes by clicking "View all decommissioned nodes". - -DB Console node list - -{{site.data.alerts.callout_info}} -When you initiate the [decommissioning process](remove-nodes.html#how-it-works) on a node, CockroachDB transfers all range replicas and range leases off the node so that it can be safely shut down. -{{site.data.alerts.end}} - -## Node Map (Enterprise) - -The **Node Map** is an [enterprise](enterprise-licensing.html) feature that visualizes the geographical configuration of your cluster. It requires that [`--locality` flags have been defined](cockroach-start.html#locality) for your nodes. - -For guidance on enabling and configuring the node map, see [Enable the Node Map](enable-node-map.html). - -DB Console Summary Panel - -The Node Map uses the longitude and latitude of each locality to position the components on the map. The map is populated with [**locality components**](#locality-component) and [**node components**](#node-component). - -### Locality component - -A locality component represents capacity, CPU, and QPS metrics for a given locality. - -The map shows the components for the highest-level locality tier (e.g., region). You can click on the **Node Count** of a locality component to view any lower-level localities (e.g., availability zone). - -For details on how **Capacity Usage** is calculated, see [Capacity metrics](#capacity-metrics). - -DB Console Summary Panel - -{{site.data.alerts.callout_info}} -On multi-core systems, the displayed CPU usage can be greater than 100%. Full utilization of 1 core is considered as 100% CPU usage. If you have _n_ cores, then CPU usage can range from 0% (indicating an idle system) to (_n_ * 100)% (indicating full utilization). -{{site.data.alerts.end}} - -### Node component - -A node component represents capacity, CPU, and QPS metrics for a given node. - -Node components are accessed by clicking on the **Node Count** of the lowest-level [locality component](#locality-component). - -For details on how **Capacity Usage** is calculated, see [Capacity metrics](#capacity-metrics). - -DB Console Summary Panel - -{{site.data.alerts.callout_info}} -On multi-core systems, the displayed CPU usage can be greater than 100%. Full utilization of 1 core is considered as 100% CPU usage. If you have _n_ cores, then CPU usage can range from 0% (indicating an idle system) to (_n_ * 100)% (indicating full utilization). -{{site.data.alerts.end}} - -## See also - -- [Production Checklist](recommended-production-settings.html) -- [Locality](cockroach-start.html#locality) -- [Troubleshooting Overview](troubleshooting-overview.html) -- [Support Resources](support-resources.html) -- [Raw Status Endpoints](monitoring-and-alerting.html#raw-status-endpoints) diff --git a/src/current/v21.1/ui-custom-chart-debug-page.md b/src/current/v21.1/ui-custom-chart-debug-page.md deleted file mode 100644 index c1af4467614..00000000000 --- a/src/current/v21.1/ui-custom-chart-debug-page.md +++ /dev/null @@ -1,60 +0,0 @@ ---- -title: Custom Chart Debug Page -summary: The Custom Chart debug page can be used to create one or multiple custom charts showing any combination of over 200 available metrics. -toc: true ---- - -The **Custom Chart** debug page in the DB Console can be used to create one or multiple custom charts showing any combination of over [200 available metrics](#available-metrics). - -The definition of the customized dashboard is encoded in the URL. To share the dashboard with someone, send them the URL. Like any other URL, it can be bookmarked, sit in a pinned tab in your browser, etc. - - -## Accessing the Custom Chart page - -To access the **Custom Chart** debug page, [access the DB Console](ui-overview.html), and either: - -- Open http://localhost:8080/#/debug/chart in your browser (replacing `localhost` and `8080` with your node's host and port). - -- Click the gear icon on the left to access the **Advanced Debugging Page**. In the **Reports** section, click **Custom TimeSeries Chart**. - -## Using the Custom Chart page - -DB Console - -On the **Custom Chart** page, you can set the time span for all charts, add new custom charts, and customize each chart: - -- To set the time span for the page, use the dropdown menu above the charts and select the desired time span. - -- To add a chart, click **Add Chart** and customize the new chart. - -- To customize each chart, use the **Units** dropdown menu to set the units to display. Then use the table below the chart to select the metrics being queried, and how they'll be combined and displayed. Options include: -{% include {{page.version.version}}/ui-custom-chart-debug-page-00.html %} - -## Examples - -### Query user and system CPU usage - -DB Console - -To compare system vs. userspace CPU usage, select the following values under **Metric Name**: - -- `sys.cpu.sys.percent` -- `sys.cpu.user.percent` - -The Y-axis label is the **Count**. A count of 1 represents 100% utilization. The **Aggregator** of **Sum** can show the count to be above 1, which would mean CPU utilization is greater than 100%. - -Checking **Per Node** displays statistics for each node, which could show whether an individual node's CPU usage was higher or lower than the average. - -## Available metrics - -{{site.data.alerts.callout_info}} -This list is taken directly from the source code and is subject to change. Some of the metrics listed below are already visible in other areas of the [DB Console](ui-overview.html). -{{site.data.alerts.end}} - -{% include {{page.version.version}}/metric-names.md %} - -## See also - -- [Troubleshooting Overview](troubleshooting-overview.html) -- [Support Resources](support-resources.html) -- [Raw Status Endpoints](monitoring-and-alerting.html#raw-status-endpoints) diff --git a/src/current/v21.1/ui-databases-page.md b/src/current/v21.1/ui-databases-page.md deleted file mode 100644 index f4f6975d2eb..00000000000 --- a/src/current/v21.1/ui-databases-page.md +++ /dev/null @@ -1,62 +0,0 @@ ---- -title: Databases Page -summary: The Databases page provides details about databases configured, the tables in each database, and the grants assigned to each user. -toc: true ---- - -{{site.data.alerts.callout_info}} -On a secure cluster, this area of the DB Console can only be accessed by an `admin` user. See [DB Console access](ui-overview.html#db-console-access). -{{site.data.alerts.end}} - -The **Databases** page of the DB Console provides details of the following: - -- The databases configured. -- The tables in each database. -- The grants assigned to each user. - -To view this page, [access the DB Console](ui-overview.html#db-console-access) and click **Databases** in the left-hand navigation. - -## Tables view - -The **Tables** view shows details of the system table as well as the tables in your databases. - -To view [table details](#table-details), click on a table name. - -DB Console Database Tables View - -The following are displayed for each table: - -Parameter | Description ---------|---- -Table Name | The name of the table. -Size | Approximate disk size of all replicas of this table on the cluster. -Ranges | The number of ranges in the table. -\# of Columns | The number of columns in the table. -\# of Indices | The number of indices for the table. - -### Table details - -Click any table name in [**Tables**](#tables-view) view to display details for that table. - -DB Console Database Tables View - -- **Overview** displays the SQL statements used to [create and define the table](create-table.html), as well as [partitioning](partitioning.html) info, [zone configurations](configure-replication-zones.html), constraints, and lease preferences. In addition, the following metrics are displayed: - - **Size** is the approximate disk size of all replicas of this table on the cluster. - - **Ranges** is the number of [ranges](architecture/overview.html#terms) in this table. - - **Replicas** is the number of [replicas](architecture/replication-layer.html) of this table on the cluster. -- **Grants** displays the [grants](#grants-view) associated with the table. - -## Grants view - -The **Grants** view shows the [privileges](authorization.html#assign-privileges) granted to users for each database. - -For more details about grants and privileges, see [`GRANT`](grant.html). - -DB Console Database Grants View - -## See also - -- [Statements page](ui-statements-page.html) -- [Assign privileges](authorization.html#assign-privileges) -- [`GRANT`](grant.html) -- [Raw status endpoints](monitoring-and-alerting.html#raw-status-endpoints) diff --git a/src/current/v21.1/ui-debug-pages.md b/src/current/v21.1/ui-debug-pages.md deleted file mode 100644 index 589b72d382a..00000000000 --- a/src/current/v21.1/ui-debug-pages.md +++ /dev/null @@ -1,57 +0,0 @@ ---- -title: Advanced Debug Page -summary: The Advanced Debug page provides links to advanced monitoring and troubleshooting reports and cluster configuration details. -toc: true ---- - -{{site.data.alerts.callout_info}} -Most of the functionality on this page can only be accessed by an `admin` user. See [DB Console security](ui-overview.html#db-console-security). -{{site.data.alerts.end}} - -The **Advanced Debug** page of the DB Console provides links to advanced monitoring and troubleshooting reports and cluster configuration details. To view this page, [access the DB Console](ui-overview.html#db-console-access) and click **Advanced Debug** in the left-hand navigation. - -{{site.data.alerts.callout_info}} -These pages are experimental and undocumented. If you find an issue, let us know through [these channels](https://www.cockroachlabs.com/community/). - {{site.data.alerts.end}} - -## License and node information - -On the right-side of the page, the following information is displayed: - -- CockroachDB license type: Helps determine if you have access to Enterprise features. -- Current node ID: Helps identify the current node when viewing the DB Console through a load balancer. - -## Reports and Configuration - -The following debug reports and configuration views are useful for monitoring and troubleshooting CockroachDB: - -Report | Description | Access level ---------|-----|-------- -[Custom Time Series Chart](ui-custom-chart-debug-page.html) | Create a custom chart of time series data. | All users -Problem Ranges | View ranges in your cluster that are unavailable, underreplicated, slow, or have other problems. | [`admin` users only on secure clusters](ui-overview.html#db-console-access) -Network Latency | Check latencies between all nodes in your cluster. | All users -Data Distribution and Zone Configs | View the distribution of table data across nodes and verify zone configuration. | [`admin` users only on secure clusters](ui-overview.html#db-console-access) -Cluster Settings | View cluster settings and their configured values. | All users can view data according to their privileges -Localities | Check node localities for your cluster. | [`admin` users only on secure clusters](ui-overview.html#db-console-access) - -## Even More Advanced Debugging - -The **Even More Advanced Debugging** section of the page lists additional reports that are largely internal and intended for use by CockroachDB developers. You can ignore this section while monitoring and troubleshooting CockroachDB. Alternatively, if you want to learn how to use these pages, feel free to contact us through [these channels](https://www.cockroachlabs.com/community/). - -## Raw Status Endpoints (JSON) - -Depending on your [access level](ui-overview.html#db-console-access), the endpoints listed here provide access to: - -- [Log files](logging-overview.html) -- Node status -- Hot ranges -- Node-specific metrics -- Session data -- Cluster-wide range data -- Allocator runs - -## See also - -- [Troubleshooting Overview](troubleshooting-overview.html) -- [Support Resources](support-resources.html) -- [Raw Status Endpoints](monitoring-and-alerting.html#raw-status-endpoints) diff --git a/src/current/v21.1/ui-hardware-dashboard.md b/src/current/v21.1/ui-hardware-dashboard.md deleted file mode 100644 index 889e2f3c7f2..00000000000 --- a/src/current/v21.1/ui-hardware-dashboard.md +++ /dev/null @@ -1,128 +0,0 @@ ---- -title: Hardware Dashboard -summary: The Hardware dashboard lets you monitor CPU usage, disk throughput, network traffic, storage capacity, and memory. -toc: true ---- - -The **Hardware** dashboard lets you monitor the hardware utilization of your cluster. This includes CPU usage, disk throughput, network traffic, storage capacity, and memory. - -To view this dashboard, [access the DB Console](ui-overview.html#db-console-access), click **Metrics** in the left-hand navigation, and select **Dashboard** > **Hardware**. - -{% include {{ page.version.version }}/ui/ui-metrics-navigation.md %} - -The **Hardware** dashboard displays the following time series graphs: - -## CPU Percent - -DB Console CPU Percent graph - -{{site.data.alerts.callout_info}} -This graph shows the CPU consumption by the CockroachDB process only and is useful as long as there are no other processes consuming significant CPU on the node. In case you have other processes running on the node, use a separate monitoring tool to measure the total CPU consumption across all processes. -{{site.data.alerts.end}} - -- In the node view, the graph shows the percentage of CPU in use by the CockroachDB process for the selected node. - -- In the cluster view, the graph shows the percentage of CPU in use by the CockroachDB process across all nodes. - -{{site.data.alerts.callout_info}} -For multi-core systems, the percentage of CPU usage is calculated by normalizing the CPU usage across all cores, whereby 100% utilization indicates that all cores are fully utilized. -{{site.data.alerts.end}} - -## Memory Usage - -DB Console Memory Usage graph - -{{site.data.alerts.callout_info}} -This graph shows the memory consumption by the CockroachDB process only and is useful as long as there are no other processes consuming significant memory on the node. In case you have other processes running on the node, use a separate monitoring tool to measure the total memory consumption across all processes. -{{site.data.alerts.end}} - -- In the node view, the graph shows the memory in use by CockroachDB for the selected node. - -- In the cluster view, the graph shows the memory in use by CockroachDB across all nodes in the cluster. - -## Disk Read Bytes - -DB Console Disk Read Bytes graph - -- In the node view, the graph shows the 10-second average of the number of bytes read per second by all processes, including CockroachDB, for the selected node. - -- In the cluster view, the graph shows the 10-second average of the number of bytes read per second by all processes, including CockroachDB, across all nodes. - -## Disk Write Bytes - -DB Console Disk Write Bytes graph - -- In the node view, the graph shows the 10-second average of the number of bytes written per second by all processes, including CockroachDB, for the node. - -- In the cluster view, the graph shows the 10-second average of the number of bytes written per second by all processes, including CockroachDB, across all nodes. - -## Disk Read Ops - -DB Console Disk Read Ops graph - -- In the node view, the graph shows the 10-second average of the number of disk read ops per second for all processes, including CockroachDB, for the selected node. - -- In the cluster view, the graph shows the 10-second average of the number of disk read ops per second for all processes, including CockroachDB, across all nodes. - -## Disk Write Ops - -DB Console Disk Write Ops graph - -- In the node view, the graph shows the 10-second average of the number of disk write ops per second for all processes, including CockroachDB, for the node. - -- In the cluster view, the graph shows the 10-second average of the number of disk write ops per second for all processes, including CockroachDB, across all nodes. - -## Disk IOPS in Progress - -DB Console Disk IOPS in Progress graph - -- In the node view, the graph shows the number of disk reads and writes in queue for all processes, including CockroachDB, for the selected node. - -- In the cluster view, the graph shows the number of disk reads and writes in queue for all processes, including CockroachDB, across all nodes in the cluster. - -{{site.data.alerts.callout_info}} -For Mac OS, this graph is not populated and shows zero disk IOPS in progress. This is a [known limitation](https://github.com/cockroachdb/cockroach/issues/27927) that may be lifted in the future. -{{site.data.alerts.end}} - -## Available Disk Capacity - -DB Console Disk Capacity graph - -Metric | Description ---------|-------- -**Available Disk Capacity** | Free disk space available to CockroachDB data on each node. - -### Capacity metrics - -The **available** disk capacity equals the amount of empty disk space, up to the value of the maximum store size. The store size is determined as follows: - -- If a store size was specified using the [`--store`](cockroach-start.html#store) flag when starting nodes, this value is used as the limit for CockroachDB data. -- If no store size has been explicitly set, the actual disk capacity is used as the limit for CockroachDB data. - -The disk usage of the Cockroach binary, operating system, and other system files is not shown on the **Available Disk Capacity** graph. - -{{site.data.alerts.callout_info}} -{% include {{ page.version.version }}/misc/available-capacity-metric.md %} -{{site.data.alerts.end}} - -## Network Bytes Received - -DB Console Network Bytes Received graph - -- In the node view, the graph shows the 10-second average of the number of network bytes received per second for all processes, including CockroachDB, for the node. - -- In the cluster view, the graph shows the 10-second average of the number of network bytes received for all processes, including CockroachDB, per second across all nodes. - -## Network Bytes Sent - -DB Console Network Bytes Sent graph - -- In the node view, the graph shows the 10-second average of the number of network bytes sent per second by all processes, including CockroachDB, for the node. - -- In the cluster view, the graph shows the 10-second average of the number of network bytes sent per second by all processes, including CockroachDB, across all nodes. - -## See also - -- [Troubleshooting Overview](troubleshooting-overview.html) -- [Support Resources](support-resources.html) -- [Raw Status Endpoints](monitoring-and-alerting.html#raw-status-endpoints) diff --git a/src/current/v21.1/ui-jobs-page.md b/src/current/v21.1/ui-jobs-page.md deleted file mode 100644 index 94b98b2a671..00000000000 --- a/src/current/v21.1/ui-jobs-page.md +++ /dev/null @@ -1,81 +0,0 @@ ---- -title: Jobs Page -summary: The Jobs page of the DB Console provides details about long-running tasks performed by your cluster. -toc: true ---- - -The **Jobs** page of the DB Console provides details about long-running tasks performed by your cluster. These can include: - -{% include {{ page.version.version }}/sql/schema-changes.md %}. -- [`IMPORT`](import.html). -- Enterprise [`BACKUP`](backup.html) and [`RESTORE`](restore.html). -- [User-created table statistics](create-statistics.html) created for use by the [cost-based optimizer](cost-based-optimizer.html). -- [Automatic table statistics](cost-based-optimizer.html#table-statistics). -- [Changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html). - -{{site.data.alerts.callout_success}} -All users can see their own jobs, and `admin` users can view all jobs performed across all nodes in the cluster. -{{site.data.alerts.end}} - -To view these details, [access the DB Console](ui-overview.html#db-console-access) and click **Jobs** in the left-hand navigation. - -## Filter jobs - -Use the **Status** menu to filter jobs by [job status](#job-status). - -Use the **Type** menu to filter jobs by type. - -You can toggle between showing the latest 50 jobs or all jobs on the cluster. - -{{site.data.alerts.callout_info}} -Jobs are deleted every 14 days. This interval can be changed via the `jobs.retention_time` [cluster setting](cluster-settings.html). - -The Jobs list is designed for you to manage pending work. It is not intended to display the canonical record of all jobs that have run. If you need a historical record of all jobs you have run, you should log this information externally. -{{site.data.alerts.end}} - -## Jobs list - -Use the **Jobs** list to see your recently created and completed jobs. - -- For changefeeds, the table displays a [high-water timestamp that advances as the changefeed progresses](stream-data-out-of-cockroachdb-using-changefeeds.html#monitor-a-changefeed). This is a guarantee that all changes before or at the timestamp have been emitted. Hover over the high-water timestamp to view the [system time](as-of-system-time.html). - -- [Automatic table statistics](cost-based-optimizer.html#table-statistics) jobs are not displayed even when the **Type** menu is set to **All**. To view these jobs, set **Type** to **Automatic-Statistics Creation** as described [above](#filter-jobs). - -- To view [job details](#job-details), click on the job description. - -DB Console Jobs Page - -Parameter | Description -----------|------------ -Description | SQL statement that created the job. -Job ID | Unique job ID. This value is used to [pause](pause-job.html), [resume](resume-job.html), or [cancel](cancel-job.html) jobs. -Users | User that created the job. -Creation Time | Date and time the job was created. -Status | Current [job status](#job-status) or completion progress. - -### Job status - -Status | Description --------|------------ -`PENDING` | Job is created but has not started running. -`PAUSED` | Job is [paused](pause-job.html). -`FAILED` | Job failed to complete. -`SUCCEEDED` | Job successfully completed. -`CANCELED` | Job was [cancelled](cancel-job.html). - -A job that is currently running will be displayed with its percent completion and time remaining, rather than the `RUNNING` status. - -## Job details - -Click any description on the [jobs list](#jobs-list) to see the full SQL statement that created the job. - -The job ID, creation time, users, status, and error messages (if any) are also shown. - -DB Console Jobs Page - -## See also - -- [`SHOW JOBS`](show-jobs.html) -- [Troubleshooting Overview](troubleshooting-overview.html) -- [Support Resources](support-resources.html) -- [Raw Status Endpoints](monitoring-and-alerting.html#raw-status-endpoints) diff --git a/src/current/v21.1/ui-network-latency-page.md b/src/current/v21.1/ui-network-latency-page.md deleted file mode 100644 index b9c8ce39542..00000000000 --- a/src/current/v21.1/ui-network-latency-page.md +++ /dev/null @@ -1,58 +0,0 @@ ---- -title: Network Latency Page -summary: The Network Latency page displays round-trip latencies between all nodes in your cluster. -toc: true ---- - -The **Network Latency** page displays round-trip latencies between all nodes in your cluster. Latency is the time required to transmit a packet across a network, and is highly dependent on your network topology. Use this page to determine whether your latency is appropriate for your [topology pattern](topology-patterns.html), or to identify nodes with unexpected latencies. - -To view this page, [access the DB Console](ui-overview.html#db-console-access) and click **Network Latency** in the left-hand navigation. - -## Sort and filter network latency - -Use the **Sort By** menu to arrange the latency matrix by [locality](cockroach-start.html#locality) (e.g., cloud, region, availability zone, datacenter). - -Use the **Filter** menu to select specific nodes or localities to view. - -Select **Collapse Nodes** to display the mean latencies of each locality, depending on how the matrix is sorted. This is a way to quickly assess cross-regional or cross-cloud latency. - -## Understanding the Network Latency matrix - -Each cell in the matrix displays the round-trip latency in milliseconds between two nodes in your cluster. Round-trip latency includes the return time of a packet. Latencies are color-coded by their standard deviation from the mean latency on the network: green for lower values, and blue for higher. - -DB Console Network Latency matrix - -Rows represent origin nodes, and columns represent destination nodes. Hover over a cell to see round-trip latency and locality metadata for origin and destination nodes. - -On a [typical multi-region cluster](demo-low-latency-multi-region-deployment.html), you can expect much lower latencies between nodes in the same region/availability zone. Nodes in different regions/availability zones, meanwhile, will experience higher latencies that reflect their geographical distribution. - -For instance, the cluster shown above has nodes in `us-west1`, `us-east1`, and `europe-west2`. Latencies are highest between nodes in `us-west1` and `europe-west2`, which span the greatest distance. This is especially clear when sorting by region or availability zone and collapsing nodes: - -DB Console Network Latency collapsed nodes - -### No connections - -Nodes that have lost a connection are displayed in a separate color. This can help you locate a network partition in your cluster. - -{{site.data.alerts.callout_info}} -A network partition prevents nodes from communicating with each other in one or both directions. This can be due to a configuration problem with the network, such as when allowlisted IP addresses or hostnames change after a node is torn down and rebuilt. In a symmetric partition, node communication is broken in both directions. In an asymmetric partition, node communication works in one direction but not the other. - -The effect of a network partition depends on which nodes are partitioned, where the ranges are located, and to a large extent, whether [localities](cockroach-start.html#locality) are defined. If localities are not defined, a partition that cuts off at least (n-1)/2 nodes will cause data unavailability. -{{site.data.alerts.end}} - -Click the **NO CONNECTIONS** link to see lost connections between nodes or [localities](cockroach-start.html#locality), if any are defined. - -## Topology fundamentals - -{% include {{ page.version.version }}/topology-patterns/fundamentals.md %} - -{{site.data.alerts.callout_info}} -Network latency limits the performance of individual operations. You can use the [Statements](ui-statements-page.html) page to see the latencies of SQL statements on gateway nodes. -{{site.data.alerts.end}} - -## See also - -- [Topology Patterns](topology-patterns.html) -- [CockroachDB Performance](performance.html#latency) -- [Performance Tuning](performance-best-practices-overview.html) -- [Low Latency Reads and Writes in a Multi-Region Cluster](demo-low-latency-multi-region-deployment.html) diff --git a/src/current/v21.1/ui-overview-dashboard.md b/src/current/v21.1/ui-overview-dashboard.md deleted file mode 100644 index ca89e6c9a9c..00000000000 --- a/src/current/v21.1/ui-overview-dashboard.md +++ /dev/null @@ -1,78 +0,0 @@ ---- -title: Overview Dashboard -summary: The Overview dashboard lets you monitor important SQL performance, replication, and storage metrics. -toc: true ---- - -The **Overview** dashboard lets you monitor important SQL performance, replication, and storage metrics. To view this dashboard, [access the DB Console](ui-overview.html#db-console-access) and click **Metrics** on the left-hand navigation bar. The **Overview** dashboard is displayed by default. - -{% include {{ page.version.version }}/ui/ui-metrics-navigation.md %} - -{{site.data.alerts.callout_info}} -All timestamps in the DB Console are shown in Coordinated Universal Time (UTC). -{{site.data.alerts.end}} - -The **Overview** dashboard displays the following time series graphs: - -## SQL Statements - -- In the node view, the graph shows the 10-second average of the number of `SELECT`/`INSERT`/`UPDATE`/`DELETE` queries per second issued by SQL clients on the node. - -- In the cluster view, the graph shows the sum of the per-node averages, that is, an aggregate estimation of the current query load over the cluster, assuming the last 10 seconds of activity per node are representative of this load. - -See the [Statements page](ui-statements-page.html) for more details on the cluster's SQL statements. - -## Service Latency: SQL, 99th percentile - -{% include {{ page.version.version }}/ui/ui-sql-latency-99th-percentile.md %} - -## Replicas per Node - -DB Console Replicas per node graph - -Ranges are subsets of your data, which are replicated to ensure survivability. Ranges are replicated to a configurable number of CockroachDB nodes. - -- In the node view, the graph shows the number of range replicas on the selected node. - -- In the cluster view, the graph shows the number of range replicas on each node in the cluster. - -For details about how to control the number and location of replicas, see [Configure Replication Zones](configure-replication-zones.html). - -{{site.data.alerts.callout_info}} -The timeseries data used to power the graphs in the DB Console is stored within the cluster and accumulates for 30 days before it starts getting truncated. As a result, for the first 30 days or so of a cluster's life, you will see a steady increase in disk usage and the number of ranges even if you aren't writing data to the cluster yourself. For more details, see this [FAQ](operational-faqs.html#why-is-disk-usage-increasing-despite-lack-of-writes). -{{site.data.alerts.end}} - -## Capacity - -DB Console Capacity graph - -You can monitor the **Capacity** graph to determine when additional storage is needed (e.g., by [scaling your cluster](cockroach-start.html)). - -Metric | Description ---------|-------- -**Capacity** | The maximum store size. This value may be set per node using [`--store`](cockroach-start.html#store). If a store size has not been set, this metric displays the actual disk capacity. See [Capacity metrics](#capacity-metrics). -**Available** | The free disk space available to CockroachDB data. -**Used** | The disk space in use by CockroachDB data. This excludes the Cockroach binary, operating system, and other system files. - -### Capacity metrics - -The **Capacity** graph displays disk usage by CockroachDB data in relation to the maximum [store](architecture/storage-layer.html) size, which is determined as follows: - -- If a store size was specified using the [`--store`](cockroach-start.html#store) flag when starting nodes, this value is used as the limit for CockroachDB data. -- If no store size has been explicitly set, the actual disk capacity is used as the limit for CockroachDB data. - -The **available** capacity thus equals the amount of empty disk space, up to the value of the maximum store size. The **used** capacity refers only to disk space occupied by CockroachDB data, which resides in the store directory on each node. - -The disk usage of the Cockroach binary, operating system, and other system files is not shown on the **Capacity** graph. - -{{site.data.alerts.callout_info}} -{% include {{ page.version.version }}/misc/available-capacity-metric.md %} -{{site.data.alerts.end}} - -{% include {{ page.version.version }}/ui/ui-summary-events.md %} - -## See also - -- [Troubleshooting Overview](troubleshooting-overview.html) -- [Support Resources](support-resources.html) -- [Raw Status Endpoints](monitoring-and-alerting.html#raw-status-endpoints) diff --git a/src/current/v21.1/ui-overview.md b/src/current/v21.1/ui-overview.md deleted file mode 100644 index 8ee710c20ba..00000000000 --- a/src/current/v21.1/ui-overview.md +++ /dev/null @@ -1,74 +0,0 @@ ---- -title: DB Console Overview -summary: Use the DB Console to monitor and optimize cluster performance. -toc: true -key: explore-the-admin-ui.html ---- - -The DB Console provides details about your cluster and database configuration, and helps you optimize cluster performance. - -## DB Console areas - -Area | Description ---------|---- -[Cluster Overview](ui-cluster-overview-page.html) | Essential metrics about the cluster and nodes, including liveness status, replication status, uptime, and hardware usage. -[Node Map](enable-node-map.html) | Geographical configuration of your cluster and metrics at the locality and node levels, visualized on a map. -[Overview Dashboard](ui-overview-dashboard.html) | Metrics about SQL performance, replication, and storage. -[Hardware Dashboard](ui-hardware-dashboard.html) | Metrics about CPU usage, disk throughput, network traffic, storage capacity, and memory. -[Runtime Dashboard](ui-runtime-dashboard.html) | Metrics about node count, CPU time, and memory usage. -[SQL Dashboard](ui-sql-dashboard.html) | Metrics about SQL connections, byte traffic, queries, transactions, and service latency. -[Storage Dashboard](ui-storage-dashboard.html) | Metrics about storage capacity and file descriptors. -[Replication Dashboard](ui-replication-dashboard.html) | Metrics about how data is replicated across the cluster, e.g., range status, replicas per store, and replica quiescence. -[Changefeeds Dashboard](ui-cdc-dashboard.html) | Metrics about the [changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html) created across your cluster. -[Databases](ui-databases-page.html) | Details about the system and user databases in the cluster. -[Sessions](ui-sessions-page.html) | Details about open sessions in the cluster. -[Statements](ui-statements-page.html) | Frequently executed and high latency [SQL statements](sql-statements.html), with the option to collect statement diagnostics. -[Transactions](ui-transactions-page.html) | Details about transactions running on the cluster. -[Network Latency](ui-network-latency-page.html) | Latencies and lost connections between all nodes in your cluster. -[Jobs](ui-jobs-page.html) | Details of jobs running in the cluster. -[Advanced Debug](ui-debug-pages.html) | Advanced monitoring and troubleshooting reports. These include details about data distribution, the state of specific queues, and slow query metrics. These details are largely intended for use by CockroachDB developers. - -## DB Console access - -The DB Console is accessible from every node at `http://:`, or `http://:8080` by default. - -- If you included the [`--http-addr`](cockroach-start.html#networking) flag when starting nodes, use the IP address/hostname and port specified by that flag. -- If you didn't include the [`--http-addr`](cockroach-start.html#networking) flag when starting nodes, use the IP address/hostname specified by the [`--listen-addr`](cockroach-start.html#networking) flag and port `8080`. -- If you are running a [secure cluster](#db-console-security), use `https` instead of `http`. You will also need to [create a user with a password](create-user.html#create-a-user-with-a-password) to log in. - -{{site.data.alerts.callout_success}} -For guidance on accessing the DB Console in the context of cluster deployment, see [Start a Local Cluster](start-a-local-cluster.html) and [Manual Deployment](manual-deployment.html). -{{site.data.alerts.end}} - -### DB Console security - -On insecure clusters, all areas of the DB Console are accessible to all users. - -On secure clusters, for each user who should have access to the DB Console, you must [create a user with a password](create-user.html#create-a-user-with-a-password) and optionally [grant the user membership to the `admin` role](grant.html). - -{{site.data.alerts.callout_info}} -The default `root` user is a member of the `admin` role. Use the following command to [grant users membership to the `admin` role](grant.html): - -GRANT admin TO \; -{{site.data.alerts.end}} - -For security reasons, non-admin users access only the data over which they have privileges (e.g., their tables, jobs, and list of sessions), and data that does not require privileges (e.g., cluster health, node status, metrics). - -The following areas of the DB Console can only be accessed by [`admin` users](authorization.html#admin-role). These areas display information from privileged HTTP endpoints that operate with `admin` privilege. - -Secure area | Privileged information ------|----- -[Node Map](enable-node-map.html) | Database and table names -[Databases](ui-databases-page.html) | Stored table data -[Statements](ui-statements-page.html) | SQL statements -[Advanced Debug](ui-debug-pages.html) (some reports) | Stored table data, operational details, internal IP addresses, names, credentials, application data (depending on report) - -{{site.data.alerts.callout_info}} -By default, the DB Console shares anonymous usage details with Cockroach Labs. For information about the details shared and how to opt-out of reporting, see [Diagnostics Reporting](diagnostics-reporting.html). -{{site.data.alerts.end}} - -## See also - -- [Troubleshooting Overview](troubleshooting-overview.html) -- [Support Resources](support-resources.html) -- [Raw Status Endpoints](monitoring-and-alerting.html#raw-status-endpoints) diff --git a/src/current/v21.1/ui-replication-dashboard.md b/src/current/v21.1/ui-replication-dashboard.md deleted file mode 100644 index ec9dc2eafa0..00000000000 --- a/src/current/v21.1/ui-replication-dashboard.md +++ /dev/null @@ -1,113 +0,0 @@ ---- -title: Replication Dashboard -summary: The Replication dashboard lets you monitor the replication metrics for your cluster. -toc: true ---- - -The **Replication** dashboard in the DB Console enables you to monitor the replication metrics for your cluster. - -To view this dashboard, [access the DB Console](ui-overview.html#db-console-access), click **Metrics** in the left-hand navigation, and select **Dashboard** > **Replication**. - -## Review of CockroachDB terminology - -- **Range**: CockroachDB stores all user data and almost all system data in a giant sorted map of key-value pairs. This keyspace is divided into "ranges", contiguous chunks of the keyspace, so that every key can always be found in a single range. -- **Range Replica:** CockroachDB replicates each range (3 times by default) and stores each replica on a different node. -- **Range Lease:** For each range, one of the replicas holds the "range lease". This replica, referred to as the "leaseholder", is the one that receives and coordinates all read and write requests for the range. -- **Under-replicated Ranges:** When a cluster is first initialized, the few default starting ranges will only have a single replica, but as soon as other nodes are available, they will replicate to them until they've reached their desired replication factor, the default being 3. If a range does not have enough replicas, the range is said to be "under-replicated". -- **Unavailable Ranges:** If a majority of a range's replicas are on nodes that are unavailable, then the entire range is unavailable and will be unable to process queries. - -For more details, see [Scalable SQL Made Easy: How CockroachDB Automates Operations](https://www.cockroachlabs.com/blog/automated-rebalance-and-repair/) - -{% include {{ page.version.version }}/ui/ui-metrics-navigation.md %} - -The **Replication** dashboard displays the following time series graphs: - -## Ranges - -DB Console Replicas per Store - -The **Ranges** graph shows you various details about the status of ranges. - -- In the node view, the graph shows details about ranges on the node. - -- In the cluster view, the graph shows details about ranges across all nodes in the cluster. - -On hovering over the graph, the values for the following metrics are displayed: - -Metric | Description ---------|---- -Ranges | The number of ranges. -Leaders | The number of ranges with leaders. If the number does not match the number of ranges for a long time, troubleshoot your cluster. -Lease Holders | The number of ranges that have leases. -Leaders w/o Leases | The number of Raft leaders without leases. If the number if non-zero for a long time, troubleshoot your cluster. -Unavailable | The number of unavailable ranges. If the number if non-zero for a long time, troubleshoot your cluster. -Under-replicated | The number of under-replicated ranges. - -## Logical Bytes per Store - -DB Console Replicas per Store - -Metric | Description ---------|-------- -**Logical Bytes per Store** | Number of logical bytes stored in [key-value pairs](architecture/distribution-layer.html#table-data) on each node. This includes historical and deleted data. - -{{site.data.alerts.callout_info}} -{% include {{ page.version.version }}/ui/logical-bytes.md %} -{{site.data.alerts.end}} - -## Replicas Per Store - -DB Console Replicas per Store - -- In the node view, the graph shows the number of range replicas on the store. - -- In the cluster view, the graph shows the number of range replicas on each store. - -You can [Configure replication zones](configure-replication-zones.html) to set the number and location of replicas. You can monitor the configuration changes using the DB Console, as described in [Fault tolerance and recovery](demo-fault-tolerance-and-recovery.html). - -## Replica Quiescence - -DB Console Replica Quiescence - -- In the node view, the graph shows the number of replicas on the node. - -- In the cluster view, the graph shows the number of replicas across all nodes. - -On hovering over the graph, the values for the following metrics are displayed: - -Metric | Description ---------|---- -Replicas | The number of replicas. -Quiescent | The number of replicas that haven't been accessed for a while. - -## Snapshots - -DB Console Replica Snapshots - -Usually the nodes in a [Raft group](architecture/replication-layer.html#raft) stay synchronized by following along the log message by message. However, if a node is far enough behind the log (e.g., if it was offline or is a new node getting up to speed), rather than send all the individual messages that changed the range, the cluster can send it a snapshot of the range and it can start following along from there. Commonly this is done preemptively, when the cluster can predict that a node will need to catch up, but occasionally the Raft protocol itself will request the snapshot. - -Metric | Description --------|------------ -Generated | The number of snapshots created per second. -Applied (Raft-initiated) | The number of snapshots applied to nodes per second that were initiated within Raft. -Applied (Learner) | The number of snapshots applied to nodes per second that were anticipated ahead of time (e.g., because a node was about to be added to a Raft group). This metric replaces the `Applied (Preemptive)` metric in 19.2 and onwards. -Applied (Preemptive) | The number of snapshots applied to nodes per second that were anticipated ahead of time (e.g., because a node was about to be added to a Raft group). This metric was used in pre-v19.2 releases and will be removed in future releases. -Reserved | The number of slots reserved per second for incoming snapshots that will be sent to a node. - -## Other graphs - -The **Replication** dashboard shows other time series graphs that are important for CockroachDB developers: - -- Leaseholders per Store -- Average Queries per Store -- Range Operations - -For monitoring CockroachDB, it is sufficient to use the [**Ranges**](#ranges), [**Replicas per Store**](#replicas-per-store), and [**Replica Quiescence**](#replica-quiescence) graphs. - -{% include {{ page.version.version }}/ui/ui-summary-events.md %} - -## See also - -- [Troubleshooting Overview](troubleshooting-overview.html) -- [Support Resources](support-resources.html) -- [Raw Status Endpoints](monitoring-and-alerting.html#raw-status-endpoints) diff --git a/src/current/v21.1/ui-runtime-dashboard.md b/src/current/v21.1/ui-runtime-dashboard.md deleted file mode 100644 index 89e3f7cc4ad..00000000000 --- a/src/current/v21.1/ui-runtime-dashboard.md +++ /dev/null @@ -1,80 +0,0 @@ ---- -title: Runtime Dashboard -summary: The Runtime dashboard lets you monitor runtime metrics for you cluster, such as node count, memory usage, and CPU time. -toc: true ---- - -The **Runtime** dashboard in the DB Console lets you monitor runtime metrics for you cluster, such as node count, memory usage, and CPU time. To view this dashboard, [access the DB Console](ui-overview.html#db-console-access), click **Metrics** on the left-hand navigation bar, and then select **Dashboard** > **Runtime**. - -{% include {{ page.version.version }}/ui/ui-metrics-navigation.md %} - -The **Runtime** dashboard displays the following time series graphs: - -## Live Node Count - -DB Console Node Count - -In the node view as well as the cluster view, the graph shows the number of live nodes in the cluster. - -A dip in the graph indicates decommissioned nodes, dead nodes, or nodes that are not responding. To troubleshoot the dip in the graph, refer to the [Summary panel](#summary-panel). - -## Memory Usage - -DB Console Memory Usage - -- In the node view, the graph shows the memory in use for the selected node. - -- In the cluster view, the graph shows the memory in use across all nodes in the cluster. - -On hovering over the graph, the values for the following metrics are displayed: - -Metric | Description ---------|---- -RSS | Total memory in use by CockroachDB. -Go Allocated | Memory allocated by the Go layer. -Go Total | Total memory managed by the Go layer. -CGo Allocated | Memory allocated by the C layer. -CGo Total | Total memory managed by the C layer. - -{{site.data.alerts.callout_info}}If Go Total or CGO Total fluctuates or grows steadily over time, contact us.{{site.data.alerts.end}} - -## CPU Time - -DB Console CPU Time - - -- In the node view, the graph shows the [CPU time](https://en.wikipedia.org/wiki/CPU_time) used by CockroachDB user and system-level operations for the selected node. -- In the cluster view, the graph shows the [CPU time](https://en.wikipedia.org/wiki/CPU_time) used by CockroachDB user and system-level operations across all nodes in the cluster. - -On hovering over the CPU Time graph, the values for the following metrics are displayed: - -Metric | Description ---------|---- -User CPU Time | Total CPU seconds per second used by the CockroachDB process across all nodes. -Sys CPU Time | Total CPU seconds per second used for CockroachDB system-level operations across all nodes. - -## Clock Offset - -DB Console Clock Offset - -- In the node view, the graph shows the mean clock offset of the node against the rest of the cluster. -- In the cluster view, the graph shows the mean clock offset of each node against the rest of the cluster. - -## Other graphs - -The **Runtime** dashboard shows other time series graphs that are important for CockroachDB developers: - -- Goroutine Count -- Runnable Goroutines per CPU -- GC Runs -- GC Pause Time - -For monitoring CockroachDB, it is sufficient to use the [**Live Node Count**](#live-node-count), [**Memory Usage**](#memory-usage), [**CPU Time**](#cpu-time), and [**Clock Offset**](#clock-offset) graphs. - -{% include {{ page.version.version }}/ui/ui-summary-events.md %} - -## See also - -- [Troubleshooting Overview](troubleshooting-overview.html) -- [Support Resources](support-resources.html) -- [Raw Status Endpoints](monitoring-and-alerting.html#raw-status-endpoints) diff --git a/src/current/v21.1/ui-sessions-page.md b/src/current/v21.1/ui-sessions-page.md deleted file mode 100644 index 99eb986f3d4..00000000000 --- a/src/current/v21.1/ui-sessions-page.md +++ /dev/null @@ -1,83 +0,0 @@ ---- -title: Sessions Page -summary: The Sessions page provides details of all open sessions in the cluster. -toc: true ---- - -{{site.data.alerts.callout_info}} -On a secure cluster, this area of the DB Console can only be accessed by a SQL user with the [`VIEWACTIVITY`](authorization.html#create-and-manage-users) role option. Note that non-`admin` users will see only their own sessions, while `admin` users see sessions for all users. -{{site.data.alerts.end}} - - The **Sessions** page of the DB Console provides details of all open sessions in the cluster. - -To view this page, [access the DB Console](ui-overview.html#db-console-access) and click **Sessions** in the left-hand navigation. - -## Sessions list - -Use the **Sessions** list to see the open sessions in the cluster. This includes active and idle sessions. - -{{site.data.alerts.callout_info}} -A session is *active* if it has an open transaction (including implicit transactions, which are individual SQL statements), and *idle* if it has no open transaction. Active sessions consume hardware resources. -{{site.data.alerts.end}} - -- If a session is active, the most recent SQL statement is displayed in the **Statement** column. -- If a session is idle, **Transaction Duration**, **Statement Duration**, and **Statement** will display `N/A`. -- To view [details of a session](#session-details), click the **Session Duration**. - -{{site.data.alerts.callout_info}} -An active session can have an open transaction that is not currently running SQL. In this case, the **Statement** and **Statement Duration** columns will display `N/A` and **Transaction Duration** will display a value. Transactions that are held open can cause [contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention). -{{site.data.alerts.end}} - -DB Console Database Tables View - -The following are displayed for each active session: - -Parameter | Description ---------- | ----------- -Session Duration | Amount of time the session has been open. -Transaction Duration | Amount of time the transaction has been active, if there is an open transaction. -Statement Duration | Amount of time the SQL statement has been active, if there is an active statement. -Memory Usage | Amount of memory currently allocated to this session, followed by the maximum amount of memory this session has ever been allocated. -Statement | Active SQL statement. If more than one statement is active, the most recent statement is shown. -Actions | Options to terminate the active query and/or terminate the session. These require the [`CANCELQUERY` role option](authorization.html#create-and-manage-users).

        **Terminate Statement:** Ends the SQL statement. The session running this statement will receive an error.

        **Terminate Session:** Ends the session. The client that holds this session will receive a "connection terminated" event. - -{{site.data.alerts.callout_success}} -Sort by **Transaction Duration** to display all active sessions at the top. -{{site.data.alerts.end}} - -## Session details - -Click the **Session Duration** of any session to display details and possible actions for that session. - -DB Console Database Tables View - -- **Session** shows the ID of the connected session. - - **Session Start Time** shows the timestamp at which the session started. - - **Gateway Node** shows the node ID and IP address/port of the [gateway](architecture/life-of-a-distributed-transaction.html#gateway) node handling the client connection. - - **Client Address** shows the IP address/port of the client that opened the session. - - **Memory Usage** shows the amount of memory currently allocated to this session, followed by the maximum amount of memory this session has ever allocated. - - The **Terminate Session** button ends the session. The client that holds this session will receive a "connection terminated" event. - - The **Terminate Statement** button ends the SQL statement. The session running this statement will receive an error. -- **Transaction** will display the following information for an open transaction. - - **Transaction Start Time** shows the timestamp at which the transaction started. - - **Number of Statements Executed** shows the total number of SQL statements executed by the transaction. - - **Number of Retries** shows the total number of [retries](transactions.html#transaction-retries) for the transaction. - - **Number of Automatic Retries** shows the total number of [automatic retries](transactions.html#automatic-retries) run by CockroachDB for the transaction. - - **Priority** shows the [priority](transactions.html#transaction-priorities) for the transaction. - - **Read Only?** shows whether the transaction is read-only. - - **AS OF SYSTEM TIME?** shows whether the transaction uses [`AS OF SYSTEM TIME`](performance-best-practices-overview.html#use-as-of-system-time-to-decrease-conflicts-with-long-running-queries) to return historical data. - - **Memory Usage** shows the amount of memory currently allocated to this transaction, followed by the maximum amount of memory this transaction has ever allocated. -- **Statement** will display the following information for an active statement. - - The SQL statement is shown. - - **Execution Start Time** shows the timestamp at which the statement was run. - - **Distributed Execution?** shows whether the statement uses [Distributed SQL (DistSQL)](architecture/sql-layer.html#distsql) optimization. - - **View Statement Details** opens the [Statement Details](ui-statements-page.html#statement-details-page) page for the statement. - -## See also - -- [`SHOW SESSIONS`](show-sessions.html) -- [Statements page](ui-statements-page.html) -- [SQL Statements](sql-statements.html) -- [Transactions](transactions.html) -- [Transaction Error Retry Reference](transaction-retry-error-reference.html) -- [Production Checklist](recommended-production-settings.html#hardware) diff --git a/src/current/v21.1/ui-sql-dashboard.md b/src/current/v21.1/ui-sql-dashboard.md deleted file mode 100644 index b7cc7a07393..00000000000 --- a/src/current/v21.1/ui-sql-dashboard.md +++ /dev/null @@ -1,162 +0,0 @@ ---- -title: SQL Dashboard -summary: The SQL dashboard lets you monitor the performance of your SQL queries. -toc: true ---- - -The **SQL** dashboard in the DB Console lets you monitor the performance of your SQL queries. To view this dashboard, [access the DB Console](ui-overview.html#db-console-access), click **Metrics** on the left-hand navigation bar, and then select **Dashboard** > **SQL**. - -{% include {{ page.version.version }}/ui/ui-metrics-navigation.md %} - -For monitoring CockroachDB, it is sufficient to use the [**Open SQL Sessions**](#open-sql-sessions), [**SQL Byte Traffic**](#sql-byte-traffic), [**SQL Statements**](#sql-statements), [**Service Latency**](#service-latency-sql-99th-percentile), and [**Transactions**](#transactions) graphs. - -The **SQL** dashboard displays the following time series graphs: - -## Open SQL Sessions - -- In the node view, the graph shows the number of connections open between the client and the selected node. - -- In the cluster view, the graph shows the total number of SQL client connections to all nodes combined, with lines for each node. - - -## Open SQL Transactions - -- In the node view, the graph shows the total number of open SQL transactions on the node. - -- In the cluster view, the graph shows the total number of open SQL transactions across all nodes in the cluster. - -See the [Transactions page](ui-transactions-page.html) for more details on the transactions. - -## Active SQL Statements - -- In the node view, the graph shows the total number of SQL statements running on that node. - -- In the cluster view, the graph shows the total number of SQL statements running across all nodes in the cluster. - -See the [Statements page](ui-statements-page.html) for more details on the cluster's SQL statements. - -## SQL Byte Traffic - -The **SQL Byte Traffic** graph helps you correlate SQL query count to byte traffic, especially in bulk data inserts or analytic queries that return data in bulk. - -- In the node view, the graph shows the current byte throughput (bytes/second) between all the connected SQL clients and the node. There are lines for bytes in and bytes out. - -- In the cluster view, the graph shows the aggregate client throughput across all nodes. There are lines for bytes in and bytes out. - -## SQL Statements - -- In the node view, the graph shows the 10-second average of the number of `SELECT`/`INSERT`/`UPDATE`/`DELETE` statements per second issued by SQL clients on the node. - -- In the cluster view, the graph shows the sum of the per-node averages, that is, an aggregate estimation of the current statement load over the cluster, assuming the last 10 seconds of activity per node are representative of this load. - -## SQL Statement Errors - -- In the node view, the graph shows the 10-second average of the number of SQL statements issued to the node that returned a [planning](architecture/sql-layer.html#sql-parser-planner-executor), [runtime](architecture/sql-layer.html#sql-parser-planner-executor), or [retry error](transactions.html#error-handling). - -- In the cluster view, the graph shows the 10-second average of the number of SQL statements that returned a [planning](architecture/sql-layer.html#sql-parser-planner-executor), [runtime](architecture/sql-layer.html#sql-parser-planner-executor), or [retry error](transactions.html#error-handling) across all nodes. - -See the [Statements page](ui-statements-page.html) for more details on the cluster's SQL statements. - -## SQL Statement Contention - -- In the node view, the graph shows the total number of SQL statements that experienced [contention](transactions.html#transaction-contention) on that node. - -- In the cluster view, the graph shows the total number of SQL statements that experienced [contention](transactions.html#transaction-contention) across all nodes in the cluster. - -See the [Statements page](ui-statements-page.html) for more details on the cluster's SQL statements. - -## Active Flows for Distributed SQL Statements - -- In the node view, the graph shows the number of flows on that node contributing to the running [distributed SQL](architecture/sql-layer.html#distsql) statements. - -- In the cluster view, the graph shows the number of flows across all nodes in the cluster contributing to the running [distributed SQL](architecture/sql-layer.html#distsql) statements. - -## Service Latency: SQL, 99th percentile - -{% include {{ page.version.version }}/ui/ui-sql-latency-99th-percentile.md %} - -## Service Latency: SQL, 90th percentile - -Service latency is calculated as the time in nanoseconds between when the cluster [receives a query and finishes executing the query](architecture/sql-layer.html). This time does not include returning results to the client. - -- In the node view, the graph shows the 90th [percentile](https://en.wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of service latency for the node. Over the last minute this node executed 90% of queries within this time, not including network latency between the node and the client. - -- In the cluster view, the graph shows the 90th [percentile](https://en.wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of service latency across all nodes in the cluster. There are lines for each node in the cluster. Over the last minute the cluster executed 90% of queries within this time, not including network latency between the node and the client. - -## KV Execution Latency: 99th percentile - -KV execution latency is calculated as the time in milliseconds between when the [KV layer](architecture/overview.html) receives the request and delivers a response. - -- In the node view, the graph shows the 99th [percentile](https://en.wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of KV execution latency for the node. Over the last minute the node executed 99% of requests within this time. - -- In the cluster view, the graph shows the 99th [percentile](https://en.wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of KV execution latency for each node in the cluster. There are lines for each node in the cluster. Over the last minute the node executed 99% of requests within this time. - -## KV Execution Latency: 90th percentile - -KV execution latency is calculated as the time in milliseconds between when the [KV layer](architecture/overview.html) receives the request and delivers a response. - -- In the node view, the graph shows the 90th [percentile](https://en.wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of KV execution latency for the node. Over the last minute the node executed 90% of requests within this time. - -- In the cluster view, the graph shows the 90th [percentile](https://en.wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of KV execution latency for each node in the cluster. There are lines for each node in the cluster. Over the last minute the node executed 90% of requests within this time. - -## Transactions - -- In the node view, the graph shows the 10-second average of the number of opened (`Begin`), committed (`Commits`), rolled back (`Rollbacks`), and aborted (`Aborts`) [transactions](transactions.html) per second issued by SQL clients on the node. - -- In the cluster view, the graph shows the sum of the per-node averages, that is, an aggregate estimation of the current [transactions](transactions.html) load over the cluster, assuming the last 10 seconds of activity per node are representative of this load. - -If the graph shows excessive aborts or rollbacks, it might indicate issues with the SQL statements. In that case, re-examine [statements](ui-statements-page.html) to lower contention. - -See the [Transactions page](ui-transactions-page.html) for more details on the transactions. - -## Transaction Latency: 99th percentile - -Transaction latency is calculated as the total time in nanoseconds a [transaction](transactions.html) took to complete. - -- In the node view, the graph shows the 99th [percentile](https://en.wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of transaction time over a 1 minute period for the node. Over the last minute the node completed 99% of transactions within this time. - -- In the cluster view, the graph shows the 99th [percentile](https://en.wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of transaction time over a 1 minute period for each node in the cluster. Over the last minute the node completed 99% of transactions within this time. - -See the [Transactions page](ui-transactions-page.html) for more details on the transactions. - -## Transaction Latency: 90th percentile - -Transaction latency is calculated as the total time in nanoseconds a [transaction](transactions.html) took to complete. - -- In the node view, the graph shows the 90th [percentile](https://en.wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of transaction time over a 1 minute period for the node. Over the last minute the node completed 90% of transactions within this time. - -- In the cluster view, the graph shows the 90th [percentile](https://en.wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of transaction time over a 1 minute period for each node in the cluster. Over the last minute the node completed 90% of transactions within this time. - -See the [Transactions page](ui-transactions-page.html) for more details on the transactions. - -## SQL Memory - -- In the node view, the graph shows the current amount of memory in KiB allocated to the SQL layer on this node. This amount is what is compared against the node's [`--max-sql-memory` flag](cockroach-start.html#general). - -- In the cluster view, the graph shows the current amount of memory in KiB allocated to the SQL layer on all nodes in the cluster. This amount is what is compared against the node's [`--max-sql-memory` flag](cockroach-start.html#general). - -## Schema Changes - -- In the node view, the graph shows the total number of [DDL statements](online-schema-changes.html) per second on the node. - -- In the cluster view, the graph shows the total number of [DDL statements](online-schema-changes.html) per second across all nodes in the cluster. - -## Statement Denials: Cluster Settings - -Statement denials are statements that were denied due to a [cluster setting](cluster-settings.html) with the following format: - -``` -{feature}.{statement_type}.enabled = FALSE -``` - -- In the node view, the graph shows the total number of statements denied per second on this node. - -- In the cluster view, the graph shows the total number of statements denied per second across all nodes in the cluster. - -{% include {{ page.version.version }}/ui/ui-summary-events.md %} - -## See also - -- [Troubleshooting Overview](troubleshooting-overview.html) -- [Support Resources](support-resources.html) -- [Raw Status Endpoints](monitoring-and-alerting.html#raw-status-endpoints) diff --git a/src/current/v21.1/ui-statements-page.md b/src/current/v21.1/ui-statements-page.md deleted file mode 100644 index 051d56819d2..00000000000 --- a/src/current/v21.1/ui-statements-page.md +++ /dev/null @@ -1,179 +0,0 @@ ---- -title: Statements Page -summary: The Statements page helps you identify frequently executed or high latency SQL statements, view statement details, and download statement diagnostics. -toc: true ---- - -{{site.data.alerts.callout_info}} -On a secure cluster, this area of the DB Console can only be accessed by an `admin` user. See [DB Console access](ui-overview.html#db-console-access). -{{site.data.alerts.end}} - -The **Statements** page helps you: - -- Identify frequently executed or high latency [SQL statements](sql-statements.html). -- View SQL statement [details](#statement-details-page). -- Download SQL statement [diagnostics](#diagnostics) for troubleshooting. - -To view this page, [access the DB Console](ui-overview.html#db-console-access) and click **Statements** in the left-hand navigation. - -## Search and filter by application - -By default, this page shows SQL statements from all applications running on the cluster, including internal CockroachDB queries. - -To filter the statements by [`application_name`](connection-parameters.html#additional-connection-parameters), use the **App** menu. If you haven't set `application_name` in the client connection string, it appears as `unset`. - -CockroachDB's internal queries are displayed under the `(internal)` app. Queries from the SQL shell are displayed under the `$ cockroach sql` app. - -You can also search for statements using the search bar. - -## Understand the Statements page - -Use this page to identify SQL statements that you may want to [troubleshoot](query-behavior-troubleshooting.html). This might include statements that are experiencing high latencies, multiple [retries](transactions.html#transaction-retries), or execution failures. You can optionally create and retrieve [diagnostics](#diagnostics) for these statements. - -{{site.data.alerts.callout_success}} -If you haven't yet executed any queries in the cluster as a user, this page will initially be blank. -{{site.data.alerts.end}} - -Columns | Description ------|------------ -{% include {{ page.version.version }}/ui/statement_table.md %} -Diagnostics | Option to activate [diagnostics](#diagnostics) for this fingerprint. If activated, this displays the status of diagnostics collection (`WAITING FOR QUERY`, `READY`, OR `ERROR`). When `READY`, the most recent diagnostics bundle can be downloaded here. Access the full history of diagnostics for the fingerprint in the [**Statement Details**](#statement-details-page) page. - -### Time interval - -By default, the Statements page displays all SQL statements executed within a one-hour time interval. The display is cleared at the end of each interval. You can change the interval with the [`diagnostics.reporting.interval`](cluster-settings.html#settings) cluster setting. - -### SQL statement fingerprints - -The Statements page displays SQL statement *fingerprints*. - -A statement fingerprint represents one or more SQL statements by replacing literal values (e.g., numbers and strings) with underscores (`_`). This can help you quickly identify frequently executed SQL statements and their latencies. - -For multiple SQL statements to be represented by a fingerprint, they must be identical aside from their literal values: - -- INSERT INTO new_order(product_id, customer_id, transaction_id) VALUES (380, 11, 11098) -- INSERT INTO new_order(product_id, customer_id, transaction_id) VALUES (192, 891, 20) -- INSERT INTO new_order(product_id, customer_id, transaction_id) VALUES (784, 452, 78) - -The above SQL statements have the fingerprint: - -INSERT INTO new_order(product_id, customer_id, no_w_id) VALUES (_, _, _) - -The following statements cannot be represented by the same fingerprint: - -- INSERT INTO orders(product_id, customer_id, transaction_id) VALUES (380, 11, 11098) -- INSERT INTO new_order(product_id, customer_id, transaction_id) VALUES (380, 11, 11098) -- INSERT INTO new_order(product_id, customer_id, transaction_id) VALUES ($1, 11, 11098) -- INSERT INTO new_order(product_id, customer_id, transaction_id) VALUES ($1, $2, 11098) -- INSERT INTO new_order(product_id, customer_id, transaction_id) VALUES ($1, $2, $3) - -It is possible to see the same fingerprint listed multiple times in the following scenarios: - -- Statements with this fingerprint were executed by more than one [`application_name`](show-vars.html#supported-variables). -- Statements with this fingerprint were executed both successfully and unsuccessfully. - -## Statement Details page - -Click on a SQL statement fingerprint to open **Statement Details**. For each statement fingerprint, the details include: - -- [Overview](#overview) -- [Diagnostics](#diagnostics) -- [Logical plan](#logical-plan) -- [Execution stats](#execution-stats) - -### Overview - -The **Overview** section displays the SQL statement fingerprint and essential statistics on the right-hand side of the page: - -- **Statement Time** is the cumulative time taken to execute statements with this fingerprint within the [specified time interval](#time-interval). -- **Planning Time** is the cumulative time taken by the [planner](architecture/sql-layer.html#sql-parser-planner-executor) to create an execution plan for statements with this fingerprint within the specified time interval. -- **Execution time** is the cumulative time taken to execute statements with this fingerprint in the specified time interval. - -**Resource usage** displays statistics about storage, memory, and network usage for the SQL statement fingerprint. - -- **Mean rows/bytes read** displays the mean average number of rows and bytes [read from the storage layer](architecture/life-of-a-distributed-transaction.html#reads-from-the-storage-layer) for statements with this fingerprint within the last hour or specified [time interval](#time-interval). -- **Max memory usage** displays the maximum memory used by a statement with this fingerprint at any time during its execution within the last hour or specified time interval. -- **Network usage** displays the amount of [data transferred over the network](architecture/reads-and-writes-overview.html) (e.g., between regions and nodes) for statements with this fingerprint within the last hour or specified [time interval](#time-interval).

        If this value is 0, the statement was executed on a single node. -- **Max scratch disk usage** displays the maximum amount of data [spilled to temporary storage on disk](vectorized-execution.html#disk-spilling-operations) while executing statements with this fingerprint within the last hour or specified time interval. - -**Statement details** displays information about the execution of the statement. - -- **App** displays the name specified by the [`application_name`](show-vars.html#supported-variables) session setting. -- **Failed?** indicates whether the statement failed to execute. -- **Used cost-based optimizer?** indicates whether the execution used the [cost-based optimizer](cost-based-optimizer.html). -- **Distributed execution?** indicates whether the execution was distributed. -- **Vectorized execution?** indicates whether the execution used the [vectorized execution engine](vectorized-execution.html). -- **Transaction type** displays the type of transaction (implicit or explicit). - -**Execution counts** displays execution statistics for the SQL statement fingerprint. - -- **First Attempts** is the cumulative number of first attempts at executing statements with this fingerprint within the [specified time interval](#time-interval). -- **Total executions** is the total number of executions of statements with this fingerprint. It is calculated as the sum of first attempts and retries. -- **Retries** is the cumulative number of [retries](transactions.html#transaction-retries) of statements with this fingerprint within the [specified time interval](#time-interval). -- **Max Retries** is the highest number of retries of a single statement with this fingerprint within the [specified time interval](#time-interval). For example, if three statements with the same fingerprint had to be retried 0, 1, and 5 times, then the Max Retries value for the fingerprint is 5. - -**Rows Affected** displays statistics on rows returned for the SQL statement fingerprint. - -### Diagnostics - -The **Diagnostics** section of the Statement Details page allows you to activate and view diagnostics for the SQL statement fingerprint. - -When you activate diagnostics for a fingerprint, CockroachDB waits for the next SQL query that matches this fingerprint to be run on any node. On the next match, information about the SQL statement is written to a diagnostics bundle that you can download. This bundle consists of [statement traces](show-trace.html) in various formats (including a JSON file that can be [imported to Jaeger](query-behavior-troubleshooting.html#visualize-statement-traces-in-jaeger)), a physical query plan, execution statistics, and other information about the query. The bundle contents are identical to those produced by [`EXPLAIN ANALYZE (DEBUG)`](explain-analyze.html#debug-option). - -{{site.data.alerts.callout_success}} -Diagnostics will be collected a maximum of *N* times for a given activated fingerprint where *N* is the number of nodes in your cluster. -{{site.data.alerts.end}} - -{% include {{ page.version.version }}/sql/statement-bundle-warning.md %} - -DB Console Statements Page - -- Click the **Activate** button to begin collecting diagnostics for the fingerprint. This will open the list of **Statement diagnostics** with a status next to each activated diagnostic. - - `WAITING FOR QUERY` indicates that a SQL statement matching the fingerprint has not yet been recorded. - - `ERROR` indicates that the attempt at diagnostics collection failed. - - `READY` indicates that the diagnostics have run and can be downloaded. A download link will appear beside the status. -- For any row with a `READY` status, click **Bundle (.zip)** to retrieve the diagnostics. - -The information collected in the bundle can be used to diagnose problematic SQL statements, such as [slow queries](query-behavior-troubleshooting.html#query-is-always-slow). We recommend that you share the diagnostics bundle with our [support team](support-resources.html), which can help you interpret the results. You can also import `trace-jaeger.json` into Jaeger to [visualize the statement traces](query-behavior-troubleshooting.html#visualize-statement-traces-in-jaeger). - -Click **All statement diagnostics** to view a complete history of your collected diagnostics, each of which can be downloaded. Although fingerprints are periodically cleared from the Statements page, all diagnostics bundles are preserved. If you need to access diagnostics that were collected for a fingerprint not present in the past [interval](#time-interval), you can find the bundle here. - -### Logical Plan - -The **Logical Plan** section displays CockroachDB's query plan for an [explainable statement](sql-grammar.html#preparable_stmt). You can use this information to optimize the query. For more information about logical plans, see [`EXPLAIN`](explain.html). - -By default, the logical plan for each fingerprint is sampled every 5 minutes. You can change the interval with the [`sql.metrics.statement_details.plan_collection.period`](cluster-settings.html#settings) cluster setting. For example, to change the interval to 2 minutes, run the following [`SET CLUSTER SETTING`](set-cluster-setting.html) command: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET CLUSTER SETTING sql.metrics.statement_details.plan_collection.period = '2m0s'; -~~~ - -### Execution Stats - -**Execution Latency by Phase** displays the service latency of statements matching this fingerprint, broken down by [phase](architecture/sql-layer.html#sql-parser-planner-executor) (parse, plan, run, overhead), as well as the overall service latency. The gray bar indicates the mean latency. The blue bar indicates one standard deviation from the mean. - -{{site.data.alerts.callout_success}} -"Overhead" comprises the statements that remain after subtracting parse, plan, and run latencies from the overall latency. These might include fetching table descriptors that were not cached, or other background tasks required to execute the query. -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -Service latency can be affected by network latency, which is displayed for your cluster on the [Network Latency](ui-network-latency-page.html) page. -{{site.data.alerts.end}} - -**Other Execution Statistics** displays the following statistics. - -Statistic | Description -----------|------------ -Rows Read | The number of rows read by the statement. The gray bar indicates the mean number of rows read. The blue bar indicates one standard deviation from the mean. -Disk Bytes Read | The size of the data read by the statement. The gray bar indicates the mean number of bytes read. The blue bar indicates one standard deviation from the mean. - -The **Statistics by Node** table provides a breakdown of the number of statements of the selected fingerprint per gateway node. You can use this table to determine whether, for example, you are executing queries on a node that is far from the data you are requesting (see [Optimize Statement Performance](make-queries-fast.html#cluster-topology)). - -## See also - -- [Troubleshoot Query Behavior](query-behavior-troubleshooting.html) -- [Transaction retries](transactions.html#transaction-retries) -- [Optimize Statement Performance](make-queries-fast.html) -- [Support Resources](support-resources.html) -- [Raw Status Endpoints](monitoring-and-alerting.html#raw-status-endpoints) diff --git a/src/current/v21.1/ui-storage-dashboard.md b/src/current/v21.1/ui-storage-dashboard.md deleted file mode 100644 index fee1a4dd309..00000000000 --- a/src/current/v21.1/ui-storage-dashboard.md +++ /dev/null @@ -1,95 +0,0 @@ ---- -title: Storage Dashboard -summary: The Storage dashboard lets you monitor the storage utilization of your cluster. -toc: true ---- - -The **Storage** dashboard lets you monitor the storage utilization of your cluster. - -To view this dashboard, [access the DB Console](ui-overview.html#db-console-access), click **Metrics** in the left-hand navigation, and select **Dashboard** > **Storage**. - -{% include {{ page.version.version }}/ui/ui-metrics-navigation.md %} - -The **Storage** dashboard displays the following time series graphs: - -## Capacity - -You can monitor the **Capacity** graph to determine when additional storage is needed (e.g., by [scaling your cluster](cockroach-start.html)). - -DB Console Capacity graph - -Metric | Description ---------|-------- -**Capacity** | The maximum store size. This value may be set per node using [`--store`](cockroach-start.html#store). If a store size has not been set, this metric displays the actual disk capacity. See [Capacity metrics](#capacity-metrics). -**Available** | The free disk space available to CockroachDB data. -**Used** | The disk space in use by CockroachDB data. This excludes the Cockroach binary, operating system, and other system files. - -### Capacity metrics - -The **Capacity** graph displays disk usage by CockroachDB data in relation to the maximum [store](architecture/storage-layer.html) size, which is determined as follows: - -- If a store size was specified using the [`--store`](cockroach-start.html#store) flag when starting nodes, this value is used as the limit for CockroachDB data. -- If no store size has been explicitly set, the actual disk capacity is used as the limit for CockroachDB data. - -The **available** capacity thus equals the amount of empty disk space, up to the value of the maximum store size. The **used** capacity refers only to disk space occupied by CockroachDB data, which resides in the store directory on each node. - -The disk usage of the Cockroach binary, operating system, and other system files is not shown on the **Capacity** graph. - -{{site.data.alerts.callout_info}} -{% include {{ page.version.version }}/misc/available-capacity-metric.md %} -{{site.data.alerts.end}} - -## Live Bytes - -The **Live Bytes** graph displays the amount of data that can be read by applications and CockroachDB. - -DB Console Replicas per Store - -Metric | Description ---------|-------- -**Live** | Number of logical bytes stored in live [key-value pairs](architecture/distribution-layer.html#table-data). Live data excludes historical and deleted data. -**System** | Number of physical bytes stored in [system key-value pairs](architecture/distribution-layer.html#meta-ranges). This includes historical and deleted data that has not been [garbage collected](architecture/storage-layer.html#garbage-collection). - -{{site.data.alerts.callout_info}} -{% include {{ page.version.version }}/ui/logical-bytes.md %} -{{site.data.alerts.end}} - -## File Descriptors - -DB Console File Descriptors - -- In the node view, the graph shows the number of open file descriptors for that node, compared with the file descriptor limit. - -- In the cluster view, the graph shows the number of open file descriptors across all nodes, compared with the file descriptor limit. - -If the Open count is almost equal to the Limit count, increase [File Descriptors](recommended-production-settings.html#file-descriptors-limit). - -{{site.data.alerts.callout_info}} -If you are running multiple nodes on a single machine (not recommended), the actual number of open file descriptors are considered open on each node. Thus the limit count value displayed on the DB Console is the actual value of open file descriptors multiplied by the number of nodes, compared with the file descriptor limit. -{{site.data.alerts.end}} - -For Windows systems, you can ignore the File Descriptors graph because the concept of file descriptors is not applicable to Windows. - -## Other graphs - -The **Storage** dashboard shows other time series graphs that are important for CockroachDB developers: - -- Log Commit Latency: 99th Percentile -- Log Commit Latency: 50th Percentile -- Command Commit Latency: 99th Percentile -- Command Commit Latency: 50th Percentile -- Read Amplification -- SSTables -- Compactions/Flushes -- Time Series Writes -- Time Series Bytes Written - -For monitoring CockroachDB, it is sufficient to use the [**Capacity**](#capacity) and [**File Descriptors**](#file-descriptors) graphs. - -{% include {{ page.version.version }}/ui/ui-summary-events.md %} - -## See also - -- [Troubleshooting Overview](troubleshooting-overview.html) -- [Support Resources](support-resources.html) -- [Raw Status Endpoints](monitoring-and-alerting.html#raw-status-endpoints) diff --git a/src/current/v21.1/ui-transactions-page.md b/src/current/v21.1/ui-transactions-page.md deleted file mode 100644 index c823906a0f6..00000000000 --- a/src/current/v21.1/ui-transactions-page.md +++ /dev/null @@ -1,91 +0,0 @@ ---- -title: Transactions Page -summary: The Transactions page helps you identify frequently retried or high latency transactions and view transaction details. -toc: true ---- - -{{site.data.alerts.callout_info}} -On a secure cluster, this area of the DB Console can only be accessed by an `admin` user. See [DB Console access](ui-overview.html#db-console-access). -{{site.data.alerts.end}} - - The **Transactions** page helps you: - -- Identify frequently retried or high latency transactions. -- View transaction [details](#transaction-details-page). - -{{site.data.alerts.callout_success}} -In contrast with the [**Statements** page](ui-statements-page.html), which displays [SQL statement fingerprints](ui-statements-page.html#sql-statement-fingerprints), the **Transactions** page displays SQL statement fingerprints grouped by [transaction](transactions.html). -{{site.data.alerts.end}} - -To view this page, [access the DB Console](ui-overview.html#db-console-access) and click **Transactions** in the left-hand navigation. - -## Search and filter by application - -By default, this page shows transactions from all applications running on the cluster, and hides internal CockroachDB transactions. - -To filter the transactions by [`application_name`](connection-parameters.html#additional-connection-parameters), use the **App** pulldown in the **Filters** menu. If you haven't set `application_name` in the client connection string, it appears as `unset`. - -- CockroachDB's internal transactions are only displayed under the `$ internal` app. -- Transactions from the SQL shell are displayed under the `$ cockroach sql` app. - -You can also search for transactions using the search bar. - -## Filter by transaction latency - -You can filter transactions in which a SQL statement fingerprint exceeds a specified latency value. Use the pulldown in the **Filters** menu. - -## Understand the Transactions page - -Use this page to identify transactions that you may want to [troubleshoot](query-behavior-troubleshooting.html), such as transactions that are experiencing high latencies, multiple [retries](transactions.html#transaction-retries), or execution failures. - -{{site.data.alerts.callout_success}} -If you haven't yet run any transactions in the cluster as a user, this page will display a blank table. -{{site.data.alerts.end}} - -Column | Description ------|------------ -Transactions | The [SQL statement fingerprints](ui-statements-page.html#sql-statement-fingerprints) that make up the transaction.

        To view the transaction fingerprint and details, click this to open the [**Transaction Details** page](#transaction-details-page). -Execution Count | Cumulative number of executions of this transaction within the last hour or specified [time interval](#time-interval).

        The bar indicates the ratio of runtime success (gray) to [retries](transactions.html#transaction-retries) (red) for the transaction. -Rows Read | Average number of rows [read from disk](architecture/life-of-a-distributed-transaction.html#reads-from-the-storage-layer) while executing this transaction within the last hour or specified [time interval](#time-interval).

        The gray bar indicates the mean number of rows returned. The blue bar indicates one standard deviation from the mean. -Bytes Read | Aggregation of all bytes [read from disk](architecture/life-of-a-distributed-transaction.html#reads-from-the-storage-layer) across all operators for this transaction within the last hour or specified [time interval](#time-interval).

        The gray bar indicates the mean number of bytes read from disk. The blue bar indicates one standard deviation from the mean. -Statement Time | Average [planning and execution time](architecture/sql-layer.html#sql-parser-planner-executor) of this transaction within the last hour or specified [time interval](#time-interval).

        The gray bar indicates the mean latency. The blue bar indicates one standard deviation from the mean. -Contention | Average time this transaction was [in contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention) with other transactions within the last hour or specified [time interval](#time-interval).

        The gray bar indicates mean contention time. The blue bar indicates one standard deviation from the mean. -Max Memory | Maximum memory used by this transaction at any time during its execution within the last hour or specified [time interval](#time-interval).

        The gray bar indicates the average max memory usage. The blue bar indicates one standard deviation from the mean. -Network | Amount of [data transferred over the network](architecture/reads-and-writes-overview.html) (e.g., between regions and nodes) for this transaction within the last hour or specified [time interval](#time-interval).

        If this value is 0, the transaction was executed on a single node.

        The gray bar indicates the mean number of bytes sent over the network. The blue bar indicates one standard deviation from the mean. -Retries | Cumulative number of [retries](transactions.html#transaction-retries) of this transaction within the last hour or specified [time interval](#time-interval). -Statements | Number of SQL statements in the transaction. - -{{site.data.alerts.callout_info}} -Significant transactions on your database are likely to have a high execution count or number of rows read. -{{site.data.alerts.end}} - -### Time interval - -By default, the Transactions page displays all transactions executed within a one-hour time interval. The display is cleared at the end of each interval. You can change the interval with the [`diagnostics.reporting.interval`](cluster-settings.html#settings) cluster setting. - -## Transaction Details page - -Click on a transaction fingerprint to open **Transaction Details**. - -- The **transaction fingerprint** is displayed as a list of the individual [SQL statement fingerprints](ui-statements-page.html#sql-statement-fingerprints) in the transaction. -- The **mean transaction time** is the mean average time it took to execute the transaction within the last hour or specified [time interval](#time-interval). -- **Transaction resource** usage shows overall statistics about the transaction. - - **Mean rows/bytes read** shows the mean average number of rows and bytes [read from the storage layer](architecture/life-of-a-distributed-transaction.html#reads-from-the-storage-layer) during the execution of the transaction within the last hour or specified [time interval](#time-interval). - - **Bytes read over network** displays the amount of [data transferred over the network](architecture/reads-and-writes-overview.html) (e.g., between regions and nodes) for this transaction within the last hour or specified [time interval](#time-interval).

        If this value is 0, the statement was executed on a single node. - - **Max memory usage** is the maximum memory used by this transaction at any time during its execution within the last hour or specified time interval. - - **Max scratch disk usage** displays the maximum amount of data [spilled to temporary storage on disk](vectorized-execution.html#disk-spilling-operations) while executing this transaction within the last hour or specified time interval. - -The statement table gives details for each SQL statement in the transaction: - -Column | Description --------|------------ -{% include {{ page.version.version }}/ui/statement_table.md %} - -## See also - -- [Transactions](transactions.html) -- [Transaction Layer](architecture/transaction-layer.html) -- [Run Multi-Statement Transactions](run-multi-statement-transactions.html) -- [Transaction latency graphs](ui-sql-dashboard.html#transactions) -- [Transaction retries](transactions.html#transaction-retries) -- [DB Console Statements Page](ui-statements-page.html) diff --git a/src/current/v21.1/unique.md b/src/current/v21.1/unique.md deleted file mode 100644 index 61c8c3770fe..00000000000 --- a/src/current/v21.1/unique.md +++ /dev/null @@ -1,146 +0,0 @@ ---- -title: Unique Constraint -summary: The UNIQUE constraint specifies that each non-NULL value in the constrained column must be unique. -toc: true ---- - -The `UNIQUE` [constraint](constraints.html) specifies that each non-`NULL` value in the constrained column must be unique. - - -## Details - -- You can insert `NULL` values into columns with the `UNIQUE` constraint because `NULL` is the absence of a value, so it is never equal to other `NULL` values and not considered a duplicate value. This means that it's possible to insert rows that appear to be duplicates if one of the values is `NULL`. - - If you need to strictly enforce uniqueness, use the [`NOT NULL` constraint](not-null.html) in addition to the `UNIQUE` constraint. You can also achieve the same behavior through the table's [Primary Key](primary-key.html). - -- Columns with the `UNIQUE` constraint automatically have an [index](indexes.html) created with the name `
    __key`. To avoid having two identical indexes, you should not create indexes that exactly match the `UNIQUE` constraint's columns and order. - - The `UNIQUE` constraint depends on the automatically created index, so dropping the index also drops the `UNIQUE` constraint. - -- When using the `UNIQUE` constraint on multiple columns, the collective values of the columns must be unique. This *does not* mean that each value in each column must be unique, as if you had applied the `UNIQUE` constraint to each column individually. - -- You can define the `UNIQUE` constraint when [creating a table](#syntax), or you can add it to existing tables through [`ADD CONSTRAINT`](add-constraint.html#add-the-unique-constraint). - -{% include {{page.version.version}}/sql/indexes-regional-by-row.md %} - -For an example that uses unique indexes, see [Add a unique index to a `REGIONAL BY ROW` table](add-constraint.html#add-a-unique-index-to-a-regional-by-row-table). - -## Syntax - -`UNIQUE` constraints can be defined at the [table level](#table-level). However, if you only want the constraint to apply to a single column, it can be applied at the [column level](#column-level). - -### Column level - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/unique_column_level.html %} -
    - -Parameter | Description -----------|------------ -`table_name` | The name of the table you're creating. -`column_name` | The name of the constrained column. -`column_type` | The constrained column's [data type](data-types.html). -`column_constraints` | Any other column-level [constraints](constraints.html) you want to apply to this column. -`column_def` | Definitions for any other columns in the table. -`table_constraints` | Any table-level [constraints](constraints.html) you want to apply. - -**Example** - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE warehouses ( - warehouse_id INT PRIMARY KEY NOT NULL, - warehouse_name STRING(35) UNIQUE, - location_id INT - ); -~~~ - -### Table level - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/unique_table_level.html %} -
    - -Parameter | Description -----------|------------ -`table_name` | The name of the table you're creating. -`column_def` | Definitions for any other columns in the table. -`name` | The name you want to use for the constraint, which must be unique to its table and follow these [identifier rules](keywords-and-identifiers.html#identifiers). -`column_name` | The name of the column you want to constrain. -`table_constraints` | Any other table-level [constraints](constraints.html) you want to apply. - -**Example** - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE logon ( - login_id INT PRIMARY KEY, - customer_id INT, - logon_date TIMESTAMP, - UNIQUE (customer_id, logon_date) - ); -~~~ - -## Usage example - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE IF NOT EXISTS logon ( - login_id INT PRIMARY KEY, - customer_id INT NOT NULL, - sales_id INT, - UNIQUE (customer_id, sales_id) - ); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO logon (login_id, customer_id, sales_id) VALUES (1, 2, 1); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO logon (login_id, customer_id, sales_id) VALUES (2, 2, 1); -~~~ - -~~~ -duplicate key value (customer_id,sales_id)=(2,1) violates unique constraint "logon_customer_id_sales_id_key" -~~~ - -As mentioned in the [details](#details) above, it is possible when using the `UNIQUE` constraint alone to insert *NULL* values in a way that causes rows to appear to have rows with duplicate values. - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO logon (login_id, customer_id, sales_id) VALUES (3, 2, NULL); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO logon (login_id, customer_id, sales_id) VALUES (4, 2, NULL); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT customer_id, sales_id FROM logon; -~~~ - -~~~ -+-------------+----------+ -| customer_id | sales_id | -+-------------+----------+ -| 2 | 1 | -| 2 | NULL | -| 2 | NULL | -+-------------+----------+ -~~~ - -## See also - -- [Constraints](constraints.html) -- [`DROP CONSTRAINT`](drop-constraint.html) -- [`CHECK` constraint](check.html) -- [`DEFAULT` value constraint](default-value.html) -- [Foreign key constraint](foreign-key.html) -- [`NOT NULL` constraint](not-null.html) -- [`PRIMARY` key constraint](primary-key.html) -- [`SHOW CONSTRAINTS`](show-constraints.html) diff --git a/src/current/v21.1/unsplit-at.md b/src/current/v21.1/unsplit-at.md deleted file mode 100644 index 29ae3a78fd3..00000000000 --- a/src/current/v21.1/unsplit-at.md +++ /dev/null @@ -1,248 +0,0 @@ ---- -title: UNSPLIT AT -summary: The UNSPLIT AT statement removes a range split enforcement at a specified row in the table or index. -toc: true ---- - -The `UNSPLIT AT` [statement](sql-statements.html) removes a [split enforcement](split-at.html) on a [range split](architecture/distribution-layer.html#range-splits), at a specified row in a table or index. - -Removing a split enforcement from a table or index ("unsplitting") allows CockroachDB to merge ranges as needed, to help improve your cluster's performance. For more information, see [Range Merges](architecture/distribution-layer.html#range-merges). - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/unsplit_table_at.html %} -
    - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/unsplit_index_at.html %} -
    - -## Required privileges - -The user must have the `INSERT` [privilege](authorization.html#assign-privileges) on the table or index. - -## Parameters - - Parameter | Description ------------|------------- - `table_name` | The name of the table that you want to unsplit. - `table_name @ index_name`
    `standalone_index_name` | The name of the index that you want to unsplit. - `select_stmt` | A [selection query](selection-queries.html) that produces one or more rows at which to unsplit a table or index. - -## Examples - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### Unsplit a table - -Suppose that you want MovR to offer ride-sharing services, in addition to vehicle-sharing services. Some users need to sign up to be drivers, so you need a `drivers` table to store driver information. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE drivers ( - id UUID DEFAULT gen_random_uuid(), - city STRING, - name STRING, - dl STRING DEFAULT left(md5(random()::text),8) UNIQUE CHECK (LENGTH(dl) < 9), - address STRING, - CONSTRAINT "primary" PRIMARY KEY (city ASC, dl ASC) -); -~~~ - -The table's compound primary key is on the `city` and `dl` columns. Note that the table automatically generates an `id` and a `dl` [using supported SQL functions](functions-and-operators.html), if they are not provided. - -Because this table has several columns in common with the `users` table, you can populate the table with values from the `users` table with an `INSERT` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO drivers (id, city, name, address) - SELECT id, city, name, address FROM users; -~~~ - - At this point, just one range contains the data in the `drivers` table. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW RANGES FROM TABLE drivers; -~~~ - -~~~ - start_key | end_key | range_id | range_size_mb | lease_holder | lease_holder_locality | replicas | replica_localities -------------+---------+----------+---------------+--------------+-----------------------+----------+--------------------------- - NULL | NULL | 74 | 0.007218 | 1 | region=us-east1,az=b | {1} | {"region=us-east1,az=b"} -(1 row) -~~~ - -You can [split](split-at.html) the table based on the compound primary key. Note that you do not have to specify the entire value for the primary key, just the prefix. - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE drivers SPLIT AT VALUES ('new york', '3'), ('new york', '7'), ('chicago', '3'), ('chicago', '7'), ('seattle', '3'), ('seattle', '7'); -~~~ - -~~~ - key | pretty | split_enforced_until ---------------------------------------------+-----------------+----------------------------- - \xc3\x89\x12new york\x00\x01\x123\x00\x01 | /"new york"/"3" | 2262-04-11 23:47:16.854776 - \xc3\x89\x12new york\x00\x01\x127\x00\x01 | /"new york"/"7" | 2262-04-11 23:47:16.854776 - \xc3\x89\x12chicago\x00\x01\x123\x00\x01 | /"chicago"/"3" | 2262-04-11 23:47:16.854776 - \xc3\x89\x12chicago\x00\x01\x127\x00\x01 | /"chicago"/"7" | 2262-04-11 23:47:16.854776 - \xc3\x89\x12seattle\x00\x01\x123\x00\x01 | /"seattle"/"3" | 2262-04-11 23:47:16.854776 - \xc3\x89\x12seattle\x00\x01\x127\x00\x01 | /"seattle"/"7" | 2262-04-11 23:47:16.854776 -(6 rows) -~~~ - -The [`crdb_internal.ranges`](crdb-internal.html) view contains additional information about ranges in your CockroachDB cluster, including the expiration of the split enforcement. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT range_id, start_pretty, end_pretty, split_enforced_until FROM crdb_internal.ranges WHERE table_name='drivers'; -~~~ - -~~~ - range_id | start_pretty | end_pretty | split_enforced_until ------------+----------------------------+----------------------------+----------------------------- - 74 | /Table/59 | /Table/59/1/"chicago"/"3" | NULL - 77 | /Table/59/1/"chicago"/"3" | /Table/59/1/"chicago"/"7" | 2262-04-11 23:47:16.854776 - 78 | /Table/59/1/"chicago"/"7" | /Table/59/1/"new york"/"3" | 2262-04-11 23:47:16.854776 - 75 | /Table/59/1/"new york"/"3" | /Table/59/1/"new york"/"7" | 2262-04-11 23:47:16.854776 - 76 | /Table/59/1/"new york"/"7" | /Table/59/1/"seattle"/"3" | 2262-04-11 23:47:16.854776 - 79 | /Table/59/1/"seattle"/"3" | /Table/59/1/"seattle"/"7" | 2262-04-11 23:47:16.854776 - 80 | /Table/59/1/"seattle"/"7" | /Max | 2262-04-11 23:47:16.854776 -(7 rows) -~~~ - -Now unsplit the table to remove the split enforcements: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE drivers UNSPLIT AT VALUES ('new york', '3'), ('new york', '7'), ('chicago', '3'), ('chicago', '7'), ('seattle', '3'), ('seattle', '7'); -~~~ - -~~~ - key | pretty ---------------------------------------------+----------------------------- - \xc3\x89\x12new york\x00\x01\x123\x00\x01 | /Table/59/1/"new york"/"3" - \xc3\x89\x12new york\x00\x01\x127\x00\x01 | /Table/59/1/"new york"/"7" - \xc3\x89\x12chicago\x00\x01\x123\x00\x01 | /Table/59/1/"chicago"/"3" - \xc3\x89\x12chicago\x00\x01\x127\x00\x01 | /Table/59/1/"chicago"/"7" - \xc3\x89\x12seattle\x00\x01\x123\x00\x01 | /Table/59/1/"seattle"/"3" - \xc3\x89\x12seattle\x00\x01\x127\x00\x01 | /Table/59/1/"seattle"/"7" -(6 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT range_id, start_pretty, end_pretty, split_enforced_until FROM crdb_internal.ranges WHERE table_name='drivers'; -~~~ - -~~~ - range_id | start_pretty | end_pretty | split_enforced_until ------------+----------------------------+----------------------------+----------------------- - 74 | /Table/59 | /Table/59/1/"chicago"/"3" | NULL - 77 | /Table/59/1/"chicago"/"3" | /Table/59/1/"chicago"/"7" | NULL - 78 | /Table/59/1/"chicago"/"7" | /Table/59/1/"new york"/"3" | NULL - 75 | /Table/59/1/"new york"/"3" | /Table/59/1/"new york"/"7" | NULL - 76 | /Table/59/1/"new york"/"7" | /Table/59/1/"seattle"/"3" | NULL - 79 | /Table/59/1/"seattle"/"3" | /Table/59/1/"seattle"/"7" | NULL - 80 | /Table/59/1/"seattle"/"7" | /Max | NULL -(7 rows) - -~~~ - -The `drivers` table is still split into ranges at specific primary key column values, but the `split_enforced_until` column is now `NULL` for all ranges in the table. The split is no longer enforced, and CockroachDB can [merge the data](architecture/distribution-layer.html#range-merges) in the table as needed. - -### Unsplit an index - -Add a new secondary [index](indexes.html) to the `rides` table, on the `revenue` column, and then [split](split-at.html) the table ranges by secondary index values: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE INDEX revenue_idx ON rides(revenue); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER INDEX rides@revenue_idx SPLIT AT VALUES (25.00), (50.00), (75.00); -~~~ -~~~ - key | pretty | split_enforced_until ---------------------+--------+-------------------------------------- - \277\214*2\000 | /25 | 2262-04-11 23:47:16.854776+00:00:00 - \277\214*d\000 | /5E+1 | 2262-04-11 23:47:16.854776+00:00:00 - \277\214*\226\000 | /75 | 2262-04-11 23:47:16.854776+00:00:00 -(3 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT range_id, start_pretty, end_pretty, split_enforced_until FROM crdb_internal.ranges WHERE table_name='rides'; -~~~ -~~~ - range_id | start_pretty | end_pretty | split_enforced_until ------------+---------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------+-------------------------------------- - 39 | /Table/55 | /Table/55/1/"amsterdam"/"\xc5\x1e\xb8Q\xeb\x85@\x00\x80\x00\x00\x00\x00\x00\x01\x81" | NULL - 56 | /Table/55/1/"amsterdam"/"\xc5\x1e\xb8Q\xeb\x85@\x00\x80\x00\x00\x00\x00\x00\x01\x81" | /Table/55/1/"boston"/"8Q\xeb\x85\x1e\xb8B\x00\x80\x00\x00\x00\x00\x00\x00n" | 2262-04-11 23:47:16.854776+00:00:00 - 55 | /Table/55/1/"boston"/"8Q\xeb\x85\x1e\xb8B\x00\x80\x00\x00\x00\x00\x00\x00n" | /Table/55/1/"los angeles"/"\xa8\xf5\u008f\\(H\x00\x80\x00\x00\x00\x00\x00\x01J" | 2262-04-11 23:47:16.854776+00:00:00 - 53 | /Table/55/1/"los angeles"/"\xa8\xf5\u008f\\(H\x00\x80\x00\x00\x00\x00\x00\x01J" | /Table/55/1/"new york"/"\x1c(\xf5\u008f\\I\x00\x80\x00\x00\x00\x00\x00\x007" | 2262-04-11 23:47:16.854776+00:00:00 - 66 | /Table/55/1/"new york"/"\x1c(\xf5\u008f\\I\x00\x80\x00\x00\x00\x00\x00\x007" | /Table/55/1/"paris"/"\xe1G\xae\x14z\xe1H\x00\x80\x00\x00\x00\x00\x00\x01\xb8" | 2262-04-11 23:47:16.854776+00:00:00 - 52 | /Table/55/1/"paris"/"\xe1G\xae\x14z\xe1H\x00\x80\x00\x00\x00\x00\x00\x01\xb8" | /Table/55/1/"san francisco"/"\x8c\xcc\xcc\xcc\xcc\xcc@\x00\x80\x00\x00\x00\x00\x00\x01\x13" | 2262-04-11 23:47:16.854776+00:00:00 - 65 | /Table/55/1/"san francisco"/"\x8c\xcc\xcc\xcc\xcc\xcc@\x00\x80\x00\x00\x00\x00\x00\x01\x13" | /Table/55/1/"seattle"/"p\xa3\xd7\n=pD\x00\x80\x00\x00\x00\x00\x00\x00\xdc" | 2262-04-11 23:47:16.854776+00:00:00 - 64 | /Table/55/1/"seattle"/"p\xa3\xd7\n=pD\x00\x80\x00\x00\x00\x00\x00\x00\xdc" | /Table/55/1/"washington dc"/"Tz\xe1G\xae\x14L\x00\x80\x00\x00\x00\x00\x00\x00\xa5" | 2262-04-11 23:47:16.854776+00:00:00 - 54 | /Table/55/1/"washington dc"/"Tz\xe1G\xae\x14L\x00\x80\x00\x00\x00\x00\x00\x00\xa5" | /Table/55/4 | 2262-04-11 23:47:16.854776+00:00:00 - 68 | /Table/55/4 | /Table/55/4/25 | 2021-04-08 16:27:45.201336+00:00:00 - 69 | /Table/55/4/25 | /Table/55/4/5E+1 | 2262-04-11 23:47:16.854776+00:00:00 - 70 | /Table/55/4/5E+1 | /Table/55/4/75 | 2262-04-11 23:47:16.854776+00:00:00 - 71 | /Table/55/4/75 | /Table/56 | 2262-04-11 23:47:16.854776+00:00:00 -(13 rows) -~~~ - -Now unsplit the index to remove the split enforcements: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER INDEX rides@revenue_idx UNSPLIT AT VALUES (25.00), (50.00), (75.00); -~~~ -~~~ - key | pretty ---------------------+------------------- - \277\214*2\000 | /Table/55/4/25 - \277\214*d\000 | /Table/55/4/5E+1 - \277\214*\226\000 | /Table/55/4/75 -(3 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT range_id, start_pretty, end_pretty, split_enforced_until FROM crdb_internal.ranges WHERE table_name='rides'; -~~~ -~~~ - range_id | start_pretty | end_pretty | split_enforced_until ------------+---------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------+-------------------------------------- - 39 | /Table/55 | /Table/55/1/"amsterdam"/"\xc5\x1e\xb8Q\xeb\x85@\x00\x80\x00\x00\x00\x00\x00\x01\x81" | NULL - 56 | /Table/55/1/"amsterdam"/"\xc5\x1e\xb8Q\xeb\x85@\x00\x80\x00\x00\x00\x00\x00\x01\x81" | /Table/55/1/"boston"/"8Q\xeb\x85\x1e\xb8B\x00\x80\x00\x00\x00\x00\x00\x00n" | 2262-04-11 23:47:16.854776+00:00:00 - 55 | /Table/55/1/"boston"/"8Q\xeb\x85\x1e\xb8B\x00\x80\x00\x00\x00\x00\x00\x00n" | /Table/55/1/"los angeles"/"\xa8\xf5\u008f\\(H\x00\x80\x00\x00\x00\x00\x00\x01J" | 2262-04-11 23:47:16.854776+00:00:00 - 53 | /Table/55/1/"los angeles"/"\xa8\xf5\u008f\\(H\x00\x80\x00\x00\x00\x00\x00\x01J" | /Table/55/1/"new york"/"\x1c(\xf5\u008f\\I\x00\x80\x00\x00\x00\x00\x00\x007" | 2262-04-11 23:47:16.854776+00:00:00 - 66 | /Table/55/1/"new york"/"\x1c(\xf5\u008f\\I\x00\x80\x00\x00\x00\x00\x00\x007" | /Table/55/1/"paris"/"\xe1G\xae\x14z\xe1H\x00\x80\x00\x00\x00\x00\x00\x01\xb8" | 2262-04-11 23:47:16.854776+00:00:00 - 52 | /Table/55/1/"paris"/"\xe1G\xae\x14z\xe1H\x00\x80\x00\x00\x00\x00\x00\x01\xb8" | /Table/55/1/"san francisco"/"\x8c\xcc\xcc\xcc\xcc\xcc@\x00\x80\x00\x00\x00\x00\x00\x01\x13" | 2262-04-11 23:47:16.854776+00:00:00 - 65 | /Table/55/1/"san francisco"/"\x8c\xcc\xcc\xcc\xcc\xcc@\x00\x80\x00\x00\x00\x00\x00\x01\x13" | /Table/55/1/"seattle"/"p\xa3\xd7\n=pD\x00\x80\x00\x00\x00\x00\x00\x00\xdc" | 2262-04-11 23:47:16.854776+00:00:00 - 64 | /Table/55/1/"seattle"/"p\xa3\xd7\n=pD\x00\x80\x00\x00\x00\x00\x00\x00\xdc" | /Table/55/1/"washington dc"/"Tz\xe1G\xae\x14L\x00\x80\x00\x00\x00\x00\x00\x00\xa5" | 2262-04-11 23:47:16.854776+00:00:00 - 54 | /Table/55/1/"washington dc"/"Tz\xe1G\xae\x14L\x00\x80\x00\x00\x00\x00\x00\x00\xa5" | /Table/55/4 | 2262-04-11 23:47:16.854776+00:00:00 - 68 | /Table/55/4 | /Table/55/4/25 | 2021-04-08 16:27:45.201336+00:00:00 - 69 | /Table/55/4/25 | /Table/55/4/5E+1 | NULL - 70 | /Table/55/4/5E+1 | /Table/55/4/75 | NULL - 71 | /Table/55/4/75 | /Table/56 | NULL -(13 rows) -~~~ - -The table is still split into ranges at `25.00`, `50.00`, and `75.00`, but the `split_enforced_until` column is now `NULL` for all ranges in the table. The split is no longer enforced, and CockroachDB can [merge the data](architecture/distribution-layer.html#range-merges) in the table as needed. - -## See also - -- [`SPLIT AT`](split-at.html) -- [Selection Queries](selection-queries.html) -- [Distribution Layer](architecture/distribution-layer.html) -- [Range Merges](architecture/distribution-layer.html#range-merges) -- [Replication Layer](architecture/replication-layer.html) -- [`SHOW JOBS`](show-jobs.html) diff --git a/src/current/v21.1/update-data.md b/src/current/v21.1/update-data.md deleted file mode 100644 index 8c440ab5d88..00000000000 --- a/src/current/v21.1/update-data.md +++ /dev/null @@ -1,436 +0,0 @@ ---- -title: Update Data -summary: How to update the data in CockroachDB from various programming languages -toc: true ---- - -This page has instructions for updating existing rows of data in CockroachDB, using the following [SQL statements](sql-statements.html): - -- [`UPDATE`](#use-update), which updates existing rows in a table. -- [`UPSERT`](#use-upsert), which inserts new rows in a table, updating existing rows that conflict on a primary key. -- [`INSERT ... ON CONFLICT ... DO UPDATE`](#use-insert-on-conflict), which inserts new rows in a table, updating existing rows in the event of a conflict on specified, `UNIQUE`-constrained columns. - -## Before you begin - -Before reading this page, do the following: - -- [Install CockroachDB](install-cockroachdb.html). -- [Start a local cluster](secure-a-cluster.html), or [create a CockroachDB {{ site.data.products.cloud }} cluster](../cockroachcloud/create-your-cluster.html). -- [Install a Postgres client](install-client-drivers.html). -- [Connect to the database](connect-to-the-database.html). -- [Create a database schema](schema-design-overview.html). -- [Insert data](insert-data.html) that you now want to update. - - In the examples on this page, we use sample [`movr`](movr.html) data imported with the [`cockroach workload` command](cockroach-workload.html). - -## Use `UPDATE` - -To update existing rows in a table, use an [`UPDATE` statement](update.html) with a `WHERE` clause that filters on the columns that identify the rows that you want to update. - -{{site.data.alerts.callout_info}} -To update a large number of rows (i.e., tens of thousands of rows or more), we recommend iteratively updating subsets of the rows that you want to update, until all of the rows have been updated. You can write a script to do this, or you can write a loop into your application. - -For guidance and an example, see [Bulk-update Data](bulk-update-data.html). -{{site.data.alerts.end}} - -### `UPDATE` SQL syntax - -In SQL, `UPDATE` statements generally take the following form: - -~~~ -UPDATE {table} SET {update_column} = {update_value} WHERE {filter_column} = {filter_value} -~~~ - -Where: - -- `{table}` is the table to update. -- `{update_column}` is the column to update. -- `{update_value}` is the new value of the column in the matching row. -- `{filter_column}` is the column to filter on. -- `{filter_value}` is the matching value for the filter. - -{{site.data.alerts.callout_success}} -For detailed reference documentation on the `UPDATE` statement, including additional examples, see the [`UPDATE` syntax page](update.html). -{{site.data.alerts.end}} - -### `UPDATE` best practices - -Here are some best practices to follow when updating rows: - -- Always specify a `WHERE` clause in `UPDATE` queries. If no `WHERE` clause is specified, CockroachDB will update all of the rows in the specified table. -- To update a large number of rows (i.e., tens of thousands of rows or more), use a [batch-update loop](bulk-update-data.html). -- When executing `UPDATE` statements from an application, make sure that you wrap the SQL-executing functions in [a retry loop that handles transaction errors](error-handling-and-troubleshooting.html#transaction-retry-errors) that can occur under contention. - -### `UPDATE` example - -Suppose you want to change the status of all of the vehicles owned by a particular MovR user. To update all rows in the `vehicles` table with the `owner_id` equal to `bd70a3d7-0a3d-4000-8000-000000000025`: - -
    - - - - -
    - -
    - -{% include_cached copy-clipboard.html %} -~~~ sql -UPDATE vehicle SET status = 'unavailable' WHERE owner_id = 'bd70a3d7-0a3d-4000-8000-000000000025'; -~~~ - -
    - -
    - -{% include_cached copy-clipboard.html %} -~~~ go -// 'conn' is an open database connection - -ownerID := "bd70a3d7-0a3d-4000-8000-000000000025" - -err = conn.QueryRow(context.Background(), "UPDATE vehicle SET status = 'unavailable' WHERE owner_id = $1", ownerID) - if err != nil { - fmt.Fprintf(os.Stderr, "QueryRow failed: %v\n", err) - os.Exit(1) - } -~~~ - -
    - -
    - -{% include_cached copy-clipboard.html %} -~~~ java -// ds is an org.postgresql.ds.PGSimpleDataSource - -String ownerId = "bd70a3d7-0a3d-4000-8000-000000000025"; - -try (Connection connection = ds.getConnection()) { - PreparedStatement p = connection.prepareStatement("UPDATE vehicles SET status = 'unavailable' WHERE owner_id = ?"); - p.setString(1, ownerId); - p.executeUpdate(); - -} catch (SQLException e) { - System.out.printf("sql state = [%s]\ncause = [%s]\nmessage = [%s]\n", e.getSQLState(), e.getCause(), - e.getMessage()); -} -~~~ - -
    - -
    - -{% include_cached copy-clipboard.html %} -~~~ python -# conn is a psycopg2 connection - -ownerID = 'bd70a3d7-0a3d-4000-8000-000000000025' - -with conn.cursor() as cur: - cur.execute( - "UPDATE vehicles SET status = 'unavailable' WHERE owner_id = %s", (ownerID,)) -~~~ - -
    - -## Use `UPSERT` - -To insert new rows into a table, and update rows that conflict with the primary key value(s) of the new rows, use an [`UPSERT` statement](upsert.html). - -### `UPSERT` SQL syntax - -In SQL, `UPSERT` statements take the following form: - -~~~ -UPSERT INTO {table} ({upsert_columns}) VALUES ({upsert_values}); -~~~ - -Where: - -- `{table}` is the table to update. -- `{upsert_columns}` is a comma-separated list of the columns to which you want to insert values. -- `{upsert_values}` is a comma-separated list of values that you want to insert. - -{{site.data.alerts.callout_success}} -For detailed reference documentation on the `UPSERT` statement, including additional examples, see the [`UPSERT` syntax page](upsert.html). -{{site.data.alerts.end}} - -### `UPSERT` best practices - -Here are some best practices to follow when using `UPSERT`: - -- Limit the number of `UPSERT` statements that you execute. It's more efficient to insert multiple rows with a single statement than to execute multiple `UPSERT` statements that each insert a single row. -- When executing `UPSERT` statements from an application, make sure that you wrap the SQL-executing functions in [a retry loop that handles transaction errors](error-handling-and-troubleshooting.html#transaction-retry-errors) that can occur under contention. - -### `UPSERT` example - -Suppose you want to add some promo codes to the MovR platform, and overwrite any existing promos with the same code. To insert new rows into the `promo_codes` table, and update any rows that have the same primary key `code` value: - -
    - - - - -
    - -
    - -{% include_cached copy-clipboard.html %} -~~~ sql -UPSERT INTO promo_codes (code, description, rules) - VALUES ('0_explain_theory_something','Fifteen percent off.', '{"type": "percent_discount", "value": "15%"}'), - ('100_address_garden_certain','Twenty percent off.','{"type": "percent_discount", "value": "20%"}'), - ('1000_do_write_words','Twenty-five percent off.','{"type": "percent_discount", "value": "25%"}'); -~~~ - -
    - -
    - -{% include_cached copy-clipboard.html %} -~~~ go -// 'db' is an open database connection - -codeOne := "0_explain_theory_something" -descriptionOne := "Fifteen percent off." -rulesOne := `{"type": "percent_discount", "value\": "15%"}` -codeTwo := "100_address_garden_certain" -descriptionTwo := "Twenty percent off." -rulesTwo := `{"type": "percent_discount", "value": "20%"}` -codeThree := "1000_do_write_words" -descriptionThree := "Twenty-five percent off." -rulesThree := `{"type": "percent_discount", "value": "25%"}` - -if _, err := db.Exec("UPSERT INTO promo_codes (code, description, rules) "+ - "values ($1, $2, $3), ($4, $5, $6), ($7, $8, $9)", - codeOne, descriptionOne, rulesOne, codeTwo, descriptionTwo, - rulesTwo, codeThree, descriptionThree, rulesThree); err != nil { - return err -} -return nil -~~~ - -
    - -
    - -{% include_cached copy-clipboard.html %} -~~~ java -// ds is an org.postgresql.ds.PGSimpleDataSource - -String codeOne = "0_explain_theory_something"; -String descriptionOne = "Fifteen percent off."; -String rulesOne = "{\"type\": \"percent_discount\", \"value\": \"15%\"}"; -String codeTwo = "100_address_garden_certain"; -String descriptionTwo = "Twenty percent off."; -String rulesTwo = "{\"type\": \"percent_discount\", \"value\": \"20%\"}"; -String codeThree = "1000_do_write_words"; -String descriptionThree = "Twenty-five percent off."; -String rulesThree = "{\"type\": \"percent_discount\", \"value\": \"25%\"}"; - -try (Connection connection = ds.getConnection()) { - PreparedStatement p = connection.prepareStatement("UPSERT INTO promo_codes (code, description, rules) values (?, ?, ?), (?, ?, ?), (?, ?, ?)"); - p.setString(1, codeOne); - p.setString(2, descriptionOne); - p.setString(3, rulesOne); - p.setString(4, codeTwo); - p.setString(5, descriptionTwo); - p.setString(6, rulesTwo); - p.setString(7, codeThree); - p.setString(8, descriptionThree); - p.setString(9, rulesThree); - p.executeUpdate(); - - -} catch (SQLException e) { - System.out.printf("sql state = [%s]\ncause = [%s]\nmessage = [%s]\n", e.getSQLState(), e.getCause(), - e.getMessage()); -} -~~~ - -
    - -
    - -{% include_cached copy-clipboard.html %} -~~~ python -# conn is a psycopg2 connection - -codeOne = '0_explain_theory_something' -descriptionOne = 'Fifteen percent off.' -rulesOne = '{"type": "percent_discount", "value": "15%"}' -codeTwo = '100_address_garden_certain' -descriptionTwo = 'Twenty percent off.' -rulesTwo = '{"type": "percent_discount", "value": "20%"}' -codeThree = '1000_do_write_words' -descriptionThree = 'Twenty-five percent off.' -rulesThree = '{"type": "percent_discount", "value": "25%"}' - -with conn.cursor() as cur: - cur.execute("UPSERT INTO promo_codes (code, description, rules) values (%s,%s,%s), (%s,%s,%s), (%s,%s,%s)", - (codeOne, descriptionOne, rulesOne, codeTwo, descriptionTwo, rulesTwo, codeThree, descriptionThree, rulesThree)) -~~~ - -
    - -## Use `INSERT ON CONFLICT` - -To insert new rows into a table, and to update rows with `UNIQUE`-constrained values that conflict with any values of the new rows, use an `INSERT ... ON CONFLICT ... DO UPDATE` statement. - -`INSERT ... ON CONFLICT ... DO UPDATE` is semantically identical to `UPSERT`, when the conflicting values are in the primary key and the action to take on conflict is to update the conflicting rows with the new rows. `INSERT ... ON CONFLICT` is more flexible than `UPSERT`, and can be used to consider uniqueness for columns not in the primary key. With `INSERT ... ON CONFLICT`, you can also control how to update rows in the event of a conflict. This contrasts with the behavior of an `UPSERT` statement, which just overwrites conflicting rows with new rows. - -{{site.data.alerts.callout_info}} - Note that if you are inserting to/updating all columns of a table, and the table has no secondary indexes, `UPSERT` will be faster than the equivalent `INSERT ON CONFLICT` statement, as it will write without first reading. -{{site.data.alerts.end}} - -### `INSERT ON CONFLICT` SQL syntax - -In SQL, `INSERT ... ON CONFLICT ... DO UPDATE` statements take the following form: - -~~~ -INSERT INTO {table} ({insert_columns}) VALUES ({insert_values}) - ON CONFLICT ({conflict_columns}) DO UPDATE SET {update_column} = {update value}; -~~~ - -Where: - -- `{table}` is the table to update. -- `{insert_columns}` is a comma-separated list of the columns to which you want to insert values. -- `{insert_values}` is a comma-separated list of values that you want to insert. -- `{conflict_columns}` is a comma-separated list of the columns to evaluate for a conflict. -- `{update_column}` is the column to update, on conflict. -- `{update_values}` is the value to assign the update column. - -Note that the statement contains an `UPDATE` clause, which is semantically identical to an `UPDATE` statement. - -### `INSERT ON CONFLICT` best practices - -Here are some best practices to follow when using `INSERT ... ON CONFLICT ... DO UPDATE`: - -- Limit the number of `INSERT` statements that you execute. It's more efficient to insert multiple rows with a single statement than to execute multiple `INSERT` statements that each insert a single row. -- When executing `INSERT` statements from an application, make sure that you wrap the SQL-executing functions in [a retry loop that handles transaction errors](error-handling-and-troubleshooting.html#transaction-retry-errors) that can occur under contention. -- Follow the [performance best practices listed on the `INSERT`](insert.html#performance-best-practices) page. - -### `INSERT ON CONFLICT` example - -Suppose you want to record a particular user's promo code usage count. The `user_promo_codes` table keeps track of user promo usage. If no usage counter exists, you want to insert a new row, and if one does exist, you want to increment the `usage_count` column by 1. - -
    - - - - -
    - -
    - -{% include_cached copy-clipboard.html %} -~~~ sql -INSERT INTO user_promo_codes (city, user_id, code, "timestamp", usage_count) - VALUES ('new york', '147ae147-ae14-4b00-8000-000000000004', 'promo_code', now(), 1) - ON CONFLICT (city, user_id, code) - DO UPDATE SET usage_count = user_promo_codes.usage_count; -~~~ - -
    - -
    - -{% include_cached copy-clipboard.html %} -~~~ go -// 'db' is an open database connection - -city := "new york" -userId := "147ae147-ae14-4b00-8000-000000000004" -code := "promo_code" -ts := "now()" -usageCount := 1 - -if _, err := db.Exec("INSERT INTO user_promo_codes "+ - "VALUES ($1, $2, $3, $4, $5) ON CONFLICT (city, user_id, code) "+ - "DO UPDATE SET usage_count = user_promo_codes.usage_count + 1", - city, userId, code, ts, usageCount); err != nil { - return err -} -return nil -~~~ - -
    - -
    - -{% include_cached copy-clipboard.html %} -~~~ java -// ds is an org.postgresql.ds.PGSimpleDataSource - -String city = "new york"; -String userId = "147ae147-ae14-4b00-8000-000000000004"; -String code = "promo_code"; -String ts = "now()"; -int usageCount = 1; - -try (Connection connection = ds.getConnection()) { - PreparedStatement p = connection.prepareStatement("INSERT INTO user_promo_codes VALUES (?, ?, ?, ?, ?) ON CONFLICT (city, user_id, code) DO UPDATE SET usage_count = user_promo_codes.usage_count+1"); - p.setString(1, city); - p.setString(2, userId); - p.setString(3, code); - p.setString(4, ts); - p.setInt(5, usageCount); - p.executeUpdate(); - -} catch (SQLException e) { - System.out.printf("sql state = [%s]\ncause = [%s]\nmessage = [%s]\n", e.getSQLState(), e.getCause(), - e.getMessage()); -} -~~~ - -
    - -
    - -{% include_cached copy-clipboard.html %} -~~~ python -# conn is a psycopg2 connection - -city = 'new york' -userId = '147ae147-ae14-4b00-8000-000000000004' -code = 'promo_code' -ts = 'now()' -usageCount = 1 - -with conn.cursor() as cur: - cur.execute("INSERT INTO user_promo_codes VALUES (%s, %s, %s, %s, %s) ON CONFLICT (city, user_id, code)" - "DO UPDATE SET usage_count = user_promo_codes.usage_count+1", (city, userId, code, ts, usageCount)) -~~~ - -
    - -## See also - -Reference information related to this task: - -- [`UPDATE`](update.html) -- [Bulk-update Data](bulk-update-data.html) -- [`UPSERT`](update.html) -- [`INSERT ... ON CONFLICT`](insert.html#on-conflict-clause) -- [Understanding and Avoiding Transaction Contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention) - -Other common tasks: - -- [Connect to the Database](connect-to-the-database.html) -- [Insert Data](insert-data.html) -- [Query Data](query-data.html) -- [Delete Data](delete-data.html) -- [Bulk-update Data](bulk-update-data.html) -- [Run Multi-Statement Transactions](run-multi-statement-transactions.html) -- [Error Handling and Troubleshooting][error_handling] -- [Optimize Statement Performance](make-queries-fast.html) -- [Hello World Example apps](example-apps.html) - - - -[error_handling]: error-handling-and-troubleshooting.html -[manual]: manual-deployment.html -[orchestrated]: orchestration.html -[selection]: selection-queries.html diff --git a/src/current/v21.1/update.md b/src/current/v21.1/update.md deleted file mode 100644 index 65602b636bb..00000000000 --- a/src/current/v21.1/update.md +++ /dev/null @@ -1,608 +0,0 @@ ---- -title: UPDATE -summary: The UPDATE statement updates one or more rows in a table. -toc: true ---- - -The `UPDATE` [statement](sql-statements.html) updates rows in a table. - -{{site.data.alerts.callout_danger}} -If you update a row that contains a column referenced by a [foreign key constraint](foreign-key.html) and has an [`ON UPDATE` action](foreign-key.html#foreign-key-actions), all of the dependent rows will also be updated. -{{site.data.alerts.end}} - - -## Required privileges - -The user must have the `SELECT` and `UPDATE` [privileges](authorization.html#assign-privileges) on the table. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/update.html %} -
    - -## Parameters - -Parameter | Description -----------|------------ -`common_table_expr` | See [Common Table Expressions](common-table-expressions.html). -`table_name` | The name of the table that contains the rows you want to update. -`AS table_alias_name` | An alias for the table name. When an alias is provided, it completely hides the actual table name. -`column_name` | The name of the column whose values you want to update. -`a_expr` | The new value you want to use, the [aggregate function](functions-and-operators.html#aggregate-functions) you want to perform, or the [scalar expression](scalar-expressions.html) you want to use.

    To fill columns with their [default values](default-value.html), use `DEFAULT VALUES` in place of `a_expr`. To fill a specific column with its default value, leave the value out of the `a_expr` or use `DEFAULT` at the appropriate position. -`FROM table_ref` | Specify a table to reference, but not update, in `UPDATE` expressions, or in `RETURNING` and `WHERE` clauses. For more details, see [Reference other tables in an update](#reference-other-tables-in-an-update). -`select_stmt` | A [selection query](selection-queries.html). Each value must match the [data type](data-types.html) of its column on the left side of `=`. -`WHERE a_expr`| `a_expr` must be a [scalar expression](scalar-expressions.html) that returns Boolean values using columns (e.g., ` = `). Update rows that return `TRUE`.

    **Without a `WHERE` clause in your statement, `UPDATE` updates all rows in the table.** -`sort_clause` | An `ORDER BY` clause. See [Ordering Query Results](order-by.html) and [Ordering of rows in DML statements](order-by.html#ordering-rows-in-dml-statements) for more details. -`limit_clause` | A `LIMIT` clause. See [Limiting Query Results](limit-offset.html) for more details. -`RETURNING target_list` | Return values based on rows updated, where `target_list` can be specific column names from the table, `*` for all columns, or computations using [scalar expressions](scalar-expressions.html).

    To return nothing in the response, not even the number of rows updated, use `RETURNING NOTHING`. -`ONLY ... *` | Supported for compatibility with PostgreSQL table inheritance syntax. This clause is a no-op, as CockroachDB does not currently support table inheritance. - -## Force index selection for updates - -By using the explicit index annotation (also known as "index hinting"), you can override [CockroachDB's index selection](https://www.cockroachlabs.com/blog/index-selection-cockroachdb-2/) and use a specific [index](indexes.html) for updating rows of a named table. - -{{site.data.alerts.callout_info}} -Index selection can impact [performance](performance-best-practices-overview.html), but does not change the result of a query. -{{site.data.alerts.end}} - -The syntax to force an update for a specific index is: - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE table@my_idx SET ... -~~~ - -This is equivalent to the longer expression: - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE table@{FORCE_INDEX=my_idx} SET ... -~~~ - -To view how the index hint modifies the query plan that CockroachDB follows for updating rows, use an [`EXPLAIN (OPT)`](explain.html#opt-option) statement. To see all indexes available on a table, use [`SHOW INDEXES`](show-index.html). - -For examples, see [Update with index hints](#update-with-index-hints). - -## Reference other tables in an update - -To reference values from a table other than the table being updated, add a `FROM` clause that specifies one or more tables in the cluster. Values from tables specified in a `FROM` clause can be used in `UPDATE` expressions, and in `RETURNING` and `WHERE` clauses. - -When executing an `UPDATE` query with a `FROM` clause, CockroachDB [joins](joins.html) the target table (i.e., the table being updated) to the tables referenced in the `FROM` clause. The output of this join should have the same number of rows as the rows being updated in the target table, as CockroachDB uses a single row from the join output to update a given row in the target table. If the join produces more rows than the rows being updated in the target table, there is no way to predict which row from the join output will be used to update a row in the target table. - -For an example, see [Update using values from a different table](#update-using-values-from-a-different-table). - -## Bulk-update data - -To update a large number of rows (i.e., tens of thousands of rows or more), we recommend iteratively updating subsets of the rows that you want to update, until all of the rows have been updated. You can write a script to do this, or you can write a loop into your application. - -For guidance and an example, see [Bulk-update Data](bulk-update-data.html). - -## Examples - -{% include {{page.version.version}}/sql/movr-statements.md %} - -### Update a single column in a single row - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM users LIMIT 10; -~~~ - -~~~ - id | city | name | address | credit_card -+--------------------------------------+-----------+--------------------+--------------------------------+-------------+ - c28f5c28-f5c2-4000-8000-000000000026 | amsterdam | Maria Weber | 14729 Karen Radial | 5844236997 - c7ae147a-e147-4000-8000-000000000027 | amsterdam | Tina Miller | 97521 Mark Extensions | 8880478663 - cccccccc-cccc-4000-8000-000000000028 | amsterdam | Taylor Cunningham | 89214 Jennifer Well | 5130593761 - d1eb851e-b851-4800-8000-000000000029 | amsterdam | Kimberly Alexander | 48474 Alfred Hollow | 4059628542 - 19999999-9999-4a00-8000-000000000005 | boston | Nicole Mcmahon | 11540 Patton Extensions | 0303726947 - 1eb851eb-851e-4800-8000-000000000006 | boston | Brian Campbell | 92025 Yang Village | 9016427332 - 23d70a3d-70a3-4800-8000-000000000007 | boston | Carl Mcguire | 60124 Palmer Mews Apt. 49 | 4566257702 - 28f5c28f-5c28-4600-8000-000000000008 | boston | Jennifer Sanders | 19121 Padilla Brooks Apt. 12 | 1350968125 - 80000000-0000-4000-8000-000000000019 | chicago | Matthew Clay | 49220 Lisa Junctions | 9132291015 - 851eb851-eb85-4000-8000-00000000001a | chicago | Samantha Coffey | 6423 Jessica Underpass Apt. 87 | 9437219051 -(10 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE users SET address = '201 E Randolph St' WHERE id = '851eb851-eb85-4000-8000-00000000001a'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM users LIMIT 10; -~~~ - -~~~ - id | city | name | address | credit_card -+--------------------------------------+-----------+--------------------+------------------------------+-------------+ - c28f5c28-f5c2-4000-8000-000000000026 | amsterdam | Maria Weber | 14729 Karen Radial | 5844236997 - c7ae147a-e147-4000-8000-000000000027 | amsterdam | Tina Miller | 97521 Mark Extensions | 8880478663 - cccccccc-cccc-4000-8000-000000000028 | amsterdam | Taylor Cunningham | 89214 Jennifer Well | 5130593761 - d1eb851e-b851-4800-8000-000000000029 | amsterdam | Kimberly Alexander | 48474 Alfred Hollow | 4059628542 - 19999999-9999-4a00-8000-000000000005 | boston | Nicole Mcmahon | 11540 Patton Extensions | 0303726947 - 1eb851eb-851e-4800-8000-000000000006 | boston | Brian Campbell | 92025 Yang Village | 9016427332 - 23d70a3d-70a3-4800-8000-000000000007 | boston | Carl Mcguire | 60124 Palmer Mews Apt. 49 | 4566257702 - 28f5c28f-5c28-4600-8000-000000000008 | boston | Jennifer Sanders | 19121 Padilla Brooks Apt. 12 | 1350968125 - 80000000-0000-4000-8000-000000000019 | chicago | Matthew Clay | 49220 Lisa Junctions | 9132291015 - 851eb851-eb85-4000-8000-00000000001a | chicago | Samantha Coffey | 201 E Randolph St | 9437219051 -(10 rows) -~~~ - -### Update multiple columns in a single row - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE rides SET (end_address, revenue) = ('201 E Randolph St', 25.00) WHERE id = '851eb851-eb85-4000-8000-000000000104'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM rides WHERE rider_id = '851eb851-eb85-4000-8000-00000000001a'; -~~~ - -~~~ - id | city | vehicle_city | rider_id | vehicle_id | start_address | end_address | start_time | end_time | revenue -+--------------------------------------+---------+--------------+--------------------------------------+--------------------------------------+-------------------------------+-----------------------------+---------------------------+---------------------------+---------+ - 849ba5e3-53f7-4000-8000-000000000103 | chicago | chicago | 851eb851-eb85-4000-8000-00000000001a | 88888888-8888-4800-8000-000000000008 | 77630 Steven Road Suite 60 | 74140 Andrew Spur | 2018-12-30 03:04:05+00:00 | 2018-12-31 08:04:05+00:00 | 20.00 - 851eb851-eb85-4000-8000-000000000104 | chicago | chicago | 851eb851-eb85-4000-8000-00000000001a | 88888888-8888-4800-8000-000000000008 | 76707 Timothy Square | 201 E Randolph St | 2018-12-15 03:04:05+00:00 | 2018-12-17 07:04:05+00:00 | 25.00 - 86a7ef9d-b22d-4000-8000-000000000107 | chicago | chicago | 851eb851-eb85-4000-8000-00000000001a | 88888888-8888-4800-8000-000000000008 | 28532 Kevin Villages Suite 90 | 27493 Ortega Radial Apt. 60 | 2018-12-08 03:04:05+00:00 | 2018-12-09 03:04:05+00:00 | 36.00 - 92f1a9fb-e76c-4800-8000-00000000011f | chicago | chicago | 851eb851-eb85-4000-8000-00000000001a | 88888888-8888-4800-8000-000000000008 | 56955 Edward Walks | 53193 Jerry Village | 2019-01-01 03:04:05+00:00 | 2019-01-01 15:04:05+00:00 | 35.00 - 94fdf3b6-45a1-4800-8000-000000000123 | chicago | chicago | 851eb851-eb85-4000-8000-00000000001a | 88888888-8888-4800-8000-000000000008 | 63820 Robinson Fields | 89245 Eric Orchard | 2018-12-14 03:04:05+00:00 | 2018-12-16 10:04:05+00:00 | 80.00 -(5 rows) -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE rides SET end_address = '10000 W OHare Ave', revenue = 60.00 WHERE id = '94fdf3b6-45a1-4800-8000-000000000123'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM rides WHERE rider_id = '851eb851-eb85-4000-8000-00000000001a'; -~~~ - -~~~ - id | city | vehicle_city | rider_id | vehicle_id | start_address | end_address | start_time | end_time | revenue -+--------------------------------------+---------+--------------+--------------------------------------+--------------------------------------+-------------------------------+-----------------------------+---------------------------+---------------------------+---------+ - 849ba5e3-53f7-4000-8000-000000000103 | chicago | chicago | 851eb851-eb85-4000-8000-00000000001a | 88888888-8888-4800-8000-000000000008 | 77630 Steven Road Suite 60 | 74140 Andrew Spur | 2018-12-30 03:04:05+00:00 | 2018-12-31 08:04:05+00:00 | 20.00 - 851eb851-eb85-4000-8000-000000000104 | chicago | chicago | 851eb851-eb85-4000-8000-00000000001a | 88888888-8888-4800-8000-000000000008 | 76707 Timothy Square | 201 E Randolph St | 2018-12-15 03:04:05+00:00 | 2018-12-17 07:04:05+00:00 | 25.00 - 86a7ef9d-b22d-4000-8000-000000000107 | chicago | chicago | 851eb851-eb85-4000-8000-00000000001a | 88888888-8888-4800-8000-000000000008 | 28532 Kevin Villages Suite 90 | 27493 Ortega Radial Apt. 60 | 2018-12-08 03:04:05+00:00 | 2018-12-09 03:04:05+00:00 | 36.00 - 92f1a9fb-e76c-4800-8000-00000000011f | chicago | chicago | 851eb851-eb85-4000-8000-00000000001a | 88888888-8888-4800-8000-000000000008 | 56955 Edward Walks | 53193 Jerry Village | 2019-01-01 03:04:05+00:00 | 2019-01-01 15:04:05+00:00 | 35.00 - 94fdf3b6-45a1-4800-8000-000000000123 | chicago | chicago | 851eb851-eb85-4000-8000-00000000001a | 88888888-8888-4800-8000-000000000008 | 63820 Robinson Fields | 10000 W OHare Ave | 2018-12-14 03:04:05+00:00 | 2018-12-16 10:04:05+00:00 | 60.00 -(5 rows) -~~~ - -### Update using `SELECT` statement - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE rides SET (revenue, start_address) = - (SELECT revenue, end_address FROM rides WHERE id = '94fdf3b6-45a1-4800-8000-000000000123') - WHERE id = '851eb851-eb85-4000-8000-000000000104'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM rides WHERE rider_id = '851eb851-eb85-4000-8000-00000000001a'; -~~~ - -~~~ - id | city | vehicle_city | rider_id | vehicle_id | start_address | end_address | start_time | end_time | revenue -+--------------------------------------+---------+--------------+--------------------------------------+--------------------------------------+-------------------------------+-----------------------------+---------------------------+---------------------------+---------+ - 849ba5e3-53f7-4000-8000-000000000103 | chicago | chicago | 851eb851-eb85-4000-8000-00000000001a | 88888888-8888-4800-8000-000000000008 | 77630 Steven Road Suite 60 | 74140 Andrew Spur | 2018-12-30 03:04:05+00:00 | 2018-12-31 08:04:05+00:00 | 20.00 - 851eb851-eb85-4000-8000-000000000104 | chicago | chicago | 851eb851-eb85-4000-8000-00000000001a | 88888888-8888-4800-8000-000000000008 | 10000 W OHare Ave | 201 E Randolph St | 2018-12-15 03:04:05+00:00 | 2018-12-17 07:04:05+00:00 | 60.00 - 86a7ef9d-b22d-4000-8000-000000000107 | chicago | chicago | 851eb851-eb85-4000-8000-00000000001a | 88888888-8888-4800-8000-000000000008 | 28532 Kevin Villages Suite 90 | 27493 Ortega Radial Apt. 60 | 2018-12-08 03:04:05+00:00 | 2018-12-09 03:04:05+00:00 | 36.00 - 92f1a9fb-e76c-4800-8000-00000000011f | chicago | chicago | 851eb851-eb85-4000-8000-00000000001a | 88888888-8888-4800-8000-000000000008 | 56955 Edward Walks | 53193 Jerry Village | 2019-01-01 03:04:05+00:00 | 2019-01-01 15:04:05+00:00 | 35.00 - 94fdf3b6-45a1-4800-8000-000000000123 | chicago | chicago | 851eb851-eb85-4000-8000-00000000001a | 88888888-8888-4800-8000-000000000008 | 63820 Robinson Fields | 10000 W OHare Ave | 2018-12-14 03:04:05+00:00 | 2018-12-16 10:04:05+00:00 | 60.00 -(5 rows) -~~~ - -### Update with default values - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE users SET address = DEFAULT WHERE id = '19999999-9999-4a00-8000-000000000005'; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM users LIMIT 5; -~~~ - -~~~ - id | city | name | address | credit_card -+--------------------------------------+-----------+--------------------+-----------------------+-------------+ - c28f5c28-f5c2-4000-8000-000000000026 | amsterdam | Maria Weber | 14729 Karen Radial | 5844236997 - c7ae147a-e147-4000-8000-000000000027 | amsterdam | Tina Miller | 97521 Mark Extensions | 8880478663 - cccccccc-cccc-4000-8000-000000000028 | amsterdam | Taylor Cunningham | 89214 Jennifer Well | 5130593761 - d1eb851e-b851-4800-8000-000000000029 | amsterdam | Kimberly Alexander | 48474 Alfred Hollow | 4059628542 - 19999999-9999-4a00-8000-000000000005 | boston | Nicole Mcmahon | NULL | 0303726947 -(5 rows) -~~~ - -### Update using values from a different table - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE rides SET revenue = NULL FROM vehicles WHERE rides.rider_id=vehicles.owner_id AND rides.vehicle_id=vehicles.id; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM rides WHERE revenue IS NULL LIMIT 5; -~~~ - -~~~ - id | city | vehicle_city | rider_id | vehicle_id | start_address | end_address | start_time | end_time | revenue ----------------------------------------+-----------+--------------+--------------------------------------+--------------------------------------+--------------------------------+-----------------------------+---------------------------+---------------------------+---------- - ab020c49-ba5e-4800-8000-00000000014e | amsterdam | amsterdam | c28f5c28-f5c2-4000-8000-000000000026 | aaaaaaaa-aaaa-4800-8000-00000000000a | 1905 Christopher Locks Apt. 77 | 66037 Belinda Plaza Apt. 93 | 2018-12-13 03:04:05+00:00 | 2018-12-14 08:04:05+00:00 | NULL - ac083126-e978-4800-8000-000000000150 | amsterdam | amsterdam | c28f5c28-f5c2-4000-8000-000000000026 | aaaaaaaa-aaaa-4800-8000-00000000000a | 50217 Victoria Fields Apt. 44 | 56217 Wilson Spring | 2018-12-07 03:04:05+00:00 | 2018-12-07 10:04:05+00:00 | NULL - af9db22d-0e56-4800-8000-000000000157 | amsterdam | amsterdam | c28f5c28-f5c2-4000-8000-000000000026 | aaaaaaaa-aaaa-4800-8000-00000000000a | 20937 Gibson River | 50480 Steven Row | 2018-12-23 03:04:05+00:00 | 2018-12-25 11:04:05+00:00 | NULL - b22d0e56-0418-4000-8000-00000000015c | amsterdam | amsterdam | bd70a3d7-0a3d-4000-8000-000000000025 | bbbbbbbb-bbbb-4800-8000-00000000000b | 36054 Ward Crescent Suite 35 | 7745 John Run | 2018-12-09 03:04:05+00:00 | 2018-12-10 18:04:05+00:00 | NULL - b53f7ced-9168-4000-8000-000000000162 | amsterdam | amsterdam | bd70a3d7-0a3d-4000-8000-000000000025 | bbbbbbbb-bbbb-4800-8000-00000000000b | 86091 Mcdonald Motorway | 1652 Robert Ford | 2018-12-05 03:04:05+00:00 | 2018-12-05 06:04:05+00:00 | NULL -(5 rows) -~~~ - -### Update all rows - -{{site.data.alerts.callout_danger}} -If you do not use the `WHERE` clause to specify the rows to be updated, the values for all rows will be updated. -{{site.data.alerts.end}} -{{site.data.alerts.callout_info}} -If the [`sql_safe_updates`](cockroach-sql.html#allow-potentially-unsafe-sql-statements) session variable is set to `true`, the client will prevent the update. `sql_safe_updates` is set to `true` by default. -{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE rides SET revenue = 7.00; -~~~ - -~~~ -pq: rejected: UPDATE without WHERE clause (sql_safe_updates = true) -~~~ - -You can use a [`SET`](set-vars.html) statement to set session variables. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET sql_safe_updates = false; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE rides SET revenue = 7.00; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM rides LIMIT 5; -~~~ - -~~~ - id | city | vehicle_city | rider_id | vehicle_id | start_address | end_address | start_time | end_time | revenue -+--------------------------------------+-----------+--------------+--------------------------------------+--------------------------------------+--------------------------------+-----------------------------------+---------------------------+---------------------------+---------+ - c0000000-0000-4000-8000-000000000177 | amsterdam | amsterdam | c28f5c28-f5c2-4000-8000-000000000026 | cccccccc-cccc-4000-8000-00000000000c | 65738 Williams Summit | 72424 Thomas Field Suite 82 | 2018-12-31 03:04:05+00:00 | 2019-01-01 03:04:05+00:00 | 7.00 - c083126e-978d-4000-8000-000000000178 | amsterdam | amsterdam | cccccccc-cccc-4000-8000-000000000028 | cccccccc-cccc-4000-8000-00000000000c | 53613 Johnson Terrace | 12667 Monica Hollow | 2018-12-16 03:04:05+00:00 | 2018-12-17 15:04:05+00:00 | 7.00 - c10624dd-2f1a-4000-8000-000000000179 | amsterdam | amsterdam | c7ae147a-e147-4000-8000-000000000027 | cccccccc-cccc-4000-8000-00000000000c | 61921 Brittany Orchard Apt. 85 | 81157 Stephanie Court Suite 96 | 2018-12-30 03:04:05+00:00 | 2019-01-01 07:04:05+00:00 | 7.00 - c189374b-c6a7-4000-8000-00000000017a | amsterdam | amsterdam | cccccccc-cccc-4000-8000-000000000028 | cccccccc-cccc-4000-8000-00000000000c | 75456 Gray View | 69175 Christopher Shoals Suite 47 | 2018-12-23 03:04:05+00:00 | 2018-12-23 03:04:05+00:00 | 7.00 - c20c49ba-5e35-4000-8000-00000000017b | amsterdam | amsterdam | cccccccc-cccc-4000-8000-000000000028 | cccccccc-cccc-4000-8000-00000000000c | 38892 Joseph Summit Suite 86 | 89582 Melissa Streets | 2018-12-27 03:04:05+00:00 | 2018-12-28 18:04:05+00:00 | 7.00 -(5 rows) -~~~ - -### Update and return values - -In this example, the `RETURNING` clause returns the `id` value of the row updated. The language-specific versions assume that you have installed the relevant [client drivers](install-client-drivers.html). - -{{site.data.alerts.callout_success}}This use of RETURNING mirrors the behavior of MySQL's last_insert_id() function.{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}}When a driver provides a query() method for statements that return results and an exec() method for statements that do not (e.g., Go), it's likely necessary to use the query() method for UPDATE statements with RETURNING.{{site.data.alerts.end}} - -
    - - - - - -
    - -
    -

    - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE vehicles SET status = 'available' WHERE city = 'new york' RETURNING id; -~~~ - -~~~ - id -+--------------------------------------+ - 00000000-0000-4000-8000-000000000000 - 11111111-1111-4100-8000-000000000001 -(2 rows) -~~~ - -
    - -
    -

    - -{% include_cached copy-clipboard.html %} -~~~ python -# Import the driver. -import psycopg2 - -# Connect to the "bank" database. -conn = psycopg2.connect( - database='movr', - user='root', - host='localhost', - port=26257 -) - -# Make each statement commit immediately. -conn.set_session(autocommit=True) - -# Open a cursor to perform database operations. -cur = conn.cursor() - -# Update a row in the "vehicles" table -# and return the "id" value. -cur.execute( - "UPDATE vehicles SET status = 'available' WHERE city = 'new york' RETURNING id;" -) - -# Print out the returned value. -rows = cur.fetchall() -print('IDs:') -for row in rows: - print([str(cell) for cell in row]) - -# Close the database connection. -cur.close() -conn.close() -~~~ - -The printed value would look like: - -~~~ -IDs: -['00000000-0000-4000-8000-000000000000'] -['11111111-1111-4100-8000-000000000001'] -~~~ - -
    - -
    -

    - -{% include_cached copy-clipboard.html %} -~~~ ruby -# Import the driver. -require 'pg' - -# Connect to the "bank" database. -conn = PG.connect( - user: 'root', - dbname: 'movr', - host: 'localhost', - port: 26257 -) - -# Update a row in the "vehicles" table -# and return the "id" value. -conn.exec( - "UPDATE vehicles SET status = 'available' WHERE city = 'new york' RETURNING id;" -) do |res| - -# Print out the returned value. -puts "IDs:" - res.each do |row| - puts row - end -end - -# Close communication with the database. -conn.close() -~~~ - -The printed value would look like: - -~~~ -IDs: -{"id"=>"00000000-0000-4000-8000-000000000000"} -{"id"=>"11111111-1111-4100-8000-000000000001"} -~~~ - -
    - -
    -

    - -{% include_cached copy-clipboard.html %} -~~~ go -package main - -import ( - "database/sql" - "fmt" - "log" - - _ "github.com/lib/pq" -) - -func main() { - //Connect to the "bank" database. - db, err := sql.Open( - "postgres", - "postgresql://root@localhost:26257/movr?sslmode=disable", - ) - if err != nil { - log.Fatal("error connecting to the database: ", err) - } - - // Update a row in the "vehicles" table - // and return the "id" value. - rows, err := db.Query( - "UPDATE vehicles SET status = 'available' WHERE city = 'new york' RETURNING id;", - ) - if err != nil { - log.Fatal(err) - } - - // Print out the returned value. - defer rows.Close() - fmt.Println("IDs:") - for rows.Next() { - var id string - if err := rows.Scan(&id); err != nil { - log.Fatal(err) - } - fmt.Printf("%s\n", id) - } -} -~~~ - -The printed value would look like: - -~~~ -IDs: -00000000-0000-4000-8000-000000000000 -11111111-1111-4100-8000-000000000001 -~~~ - -
    - -
    -

    - -{% include_cached copy-clipboard.html %} -~~~ js -var async = require('async') -var pg = require('pg') - -// Config to connect to the "movr" database. -var config = { - user: 'root', - host: 'localhost', - database: 'movr', - port: 26257 - } - -// Create pool -var pool = new pg.Pool(config) - -pool.connect(function (err, client, done) { - - // Close communication with the database and exit. - var finish = function () { - done() - process.exit() - } - - if (err) { - console.error('could not connect to cockroachdb', err); - finish() - } - async.waterfall([function (next) { - client.query( - `UPDATE vehicles SET status = 'available' WHERE city = 'new york' RETURNING id;`, - next - ) - } - ], - function (err, results) { - if (err) { - console.error('error updating and selecting from users', err); - finish() - } - // Print out the returned value. - console.log('IDs:') - results.rows.forEach(function (row) { - console.log(row) - }) - - finish() - }) - }) -~~~ - -The printed value would like: - -~~~ -IDs: -{ id: '00000000-0000-4000-8000-000000000000' } -{ id: '11111111-1111-4100-8000-000000000001' } -~~~ - -
    - -### Update with index hints - -Suppose that you create a multi-column index on the `users` table with the `name` and `city` columns. - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE INDEX ON users (name, city); -~~~ - -Now suppose you want to update a couple rows in the table, based on their contents. You can use the [`EXPLAIN (OPT)`](explain.html#opt-option) command to see how the [cost-based optimizer](cost-based-optimizer.html) decides to perform the `UPDATE` statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN (opt) UPDATE users SET name='Patricia Smith (there are two)' WHERE name='Patricia Smith'; -~~~ - -~~~ - text ------------------------------------------------------------------------------------- - update users - └── project - ├── index-join users - │ └── scan users@users_name_city_idx - │ └── constraint: /10/9/8: [/'Patricia Smith' - /'Patricia Smith'] - └── projections - └── 'Patricia Smith (there are two)' -(7 rows) -~~~ - -The output of the `EXPLAIN` statement shows that the optimizer scans the newly-created `users_name_city_idx` index when performing the update. This makes sense, as you are performing an update based on the `name` column. - -Although `users_name_city_idx` is likely the most efficient index for the table scan, you may want to assess the performance difference between scanning on `users_name_city_idx` and scanning on the primary index. You can provide an index hint (i.e., force the index selection) to use the primary key of the `users` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> EXPLAIN (opt) UPDATE users@primary SET name='Patricia Smith (there are two)' WHERE name='Patricia Smith'; -~~~ - -~~~ - text ---------------------------------------------------- - update users - └── project - ├── select - │ ├── scan users - │ │ └── flags: force-index=primary - │ └── filters - │ └── name = 'Patricia Smith' - └── projections - └── 'Patricia Smith (there are two)' -(9 rows) -~~~ - -{% include {{page.version.version}}/sql/limit-row-size.md %} - -## See also - -- [`DELETE`](delete.html) -- [`INSERT`](insert.html) -- [`UPSERT`](upsert.html) -- [`TRUNCATE`](truncate.html) -- [`ALTER TABLE`](alter-table.html) -- [`DROP TABLE`](drop-table.html) -- [`DROP DATABASE`](drop-database.html) -- [Other SQL Statements](sql-statements.html) -- [Limiting Query Results](limit-offset.html) -- [Ordering of rows in DML statements](order-by.html#ordering-rows-in-dml-statements) diff --git a/src/current/v21.1/upgrade-cockroach-version.md b/src/current/v21.1/upgrade-cockroach-version.md deleted file mode 100644 index 09e8be78415..00000000000 --- a/src/current/v21.1/upgrade-cockroach-version.md +++ /dev/null @@ -1,326 +0,0 @@ ---- -title: Upgrade to CockroachDB v21.1 -summary: Learn how to upgrade your CockroachDB cluster to v21.1. -toc: true -keywords: gin, gin index, gin indexes, inverted index, inverted indexes, accelerated index, accelerated indexes ---- - -{% assign previous_version = site.data.versions | where_exp: "previous_version", "previous_version.major_version == page.version.version" | map: "previous_version" | first %} -{% assign earliest = site.data.releases | where_exp: "earliest", "earliest.major_version == page.version.version" | sort: "release_date" | first %} -{% assign latest = site.data.releases | where_exp: "latest", "latest.major_version == page.version.version" | sort: "release_date" | last %} -{% assign prior = site.data.releases | where_exp: "prior", "prior.major_version == page.version.version" | sort: "release_date" | pop | last %} -{% assign previous_latest_prod = site.data.releases | where_exp: "previous_latest_prod", "previous_latest_prod.major_version == previous_version" | where: "release_type", "Production" | sort: "release_date" | last %} -{% assign actual_latest_prod = site.data.releases | where: "major_version", site.versions["stable"] | where: "release_type", "Production" | sort: "release_date" | last %} - -Because of CockroachDB's [multi-active availability](multi-active-availability.html) design, you can perform a "rolling upgrade" of your CockroachDB cluster. This means that you can upgrade nodes one at a time without interrupting the cluster's overall health and operations. - -This page describes how to upgrade to the latest **{{ page.version.version }}** release, **{{ latest.release_name }}**. - -## Terminology - -Before upgrading, review the CockroachDB [release](../releases/) terminology: - -- A new *major release* is performed every 6 months. The major version number indicates the year of release followed by the release number, which will be either 1 or 2. For example, the latest major release is {{ actual_latest_prod.major_version }} (also written as {{ actual_latest_prod.major_version }}.0). -- Each [supported](../releases/release-support-policy.html) major release is maintained across *patch releases* that fix crashes, security issues, and data correctness issues. Each patch release increments the major version number with its corresponding patch number. For example, patch releases of {{ actual_latest_prod.major_version }} use the format {{ actual_latest_prod.major_version }}.x. -- All major and patch releases are suitable for production usage, and are therefore considered "production releases". For example, the latest production release is {{ actual_latest_prod.release_name }}. -- Prior to an upcoming major release, alpha and beta releases and release candidates are made available. These "testing releases" are not suitable for production usage. They are intended for users who need early access to a feature before it is available in a production release. These releases append the terms `alpha`, `beta`, or `rc` to the version number. - -{{site.data.alerts.callout_info}} -There are no "minor releases" of CockroachDB. -{{site.data.alerts.end}} - -## Step 1. Verify that you can upgrade - -Run [`cockroach sql`](cockroach-sql.html) against any node in the cluster to open the SQL shell. Then check your current cluster version: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CLUSTER SETTING version; -~~~ - -{{site.data.alerts.callout_info}} -**Before upgrading from v20.2 to v21.1**, you must ensure that any previously decommissioned nodes are fully decommissioned. Otherwise, they will block the upgrade. For instructions, see [Check decommissioned nodes](#check-decommissioned-nodes). -{{site.data.alerts.end}} - -To upgrade to {{ latest.release_name }}, you must be running{% if prior.release_name %} either{% endif %}: - -{% if prior.release_name %} -- **Any earlier {{ page.version.version }} release:** {{ earliest.release_name }} to {{ prior.release_name }}. -{% endif %} -- **A {{ previous_version }} production release:** {{ previous_version }}.0 to {{ previous_latest_prod.release_name }}. - -If you are running any other version, take the following steps **before** continuing on this page: - -| Version | Action(s) before upgrading to any {{ page.version.version }} release | -|------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Pre-{{ page.version.version }} testing release | Upgrade to a corresponding production release; then upgrade through each subsequent major release, [ending with a {{ previous_version }} production release](../{{ previous_version }}/upgrade-cockroach-version.html). | -| Pre-{{ previous_version }} production release | Upgrade through each subsequent major release, [ending with a {{ previous_version }} production release](../{{ previous_version }}/upgrade-cockroach-version.html). | -| {{ previous_version}} testing release | [Upgrade to a {{ previous_version }} production release](../{{ previous_version }}/upgrade-cockroach-version.html). | - -When you are ready to upgrade to {{ latest.release_name }}, continue to [step 2](#step-2-prepare-to-upgrade). - -## Step 2. Prepare to upgrade - -Before starting the upgrade, complete the following steps. - -### Check load balancing - -Make sure your cluster is behind a [load balancer](recommended-production-settings.html#load-balancing), or your clients are configured to talk to multiple nodes. If your application communicates with a single node, stopping that node to upgrade its CockroachDB binary will cause your application to fail. - -### Check cluster health - -Verify the overall health of your cluster using the [DB Console](ui-overview.html). On the **Overview**: - -- Under **Node Status**, make sure all nodes that should be live are listed as such. If any nodes are unexpectedly listed as suspect or dead, identify why the nodes are offline and either restart them or [decommission](remove-nodes.html) them before beginning your upgrade. If there are dead and non-decommissioned nodes in your cluster, it will not be possible to finalize the upgrade (either automatically or manually). - -- Under **Replication Status**, make sure there are 0 under-replicated and unavailable ranges. Otherwise, performing a rolling upgrade increases the risk that ranges will lose a majority of their replicas and cause cluster unavailability. Therefore, it's important to identify and resolve the cause of range under-replication and/or unavailability before beginning your upgrade. - -- In the **Node List**: - - Make sure all nodes are on the same version. If any nodes are behind, upgrade them to the cluster's current version first, and then start this process over. - - Make sure capacity and memory usage are reasonable for each node. Nodes must be able to tolerate some increase in case the new version uses more resources for your workload. Also go to **Metrics > Dashboard: Hardware** and make sure CPU percent is reasonable across the cluster. If there's not enough headroom on any of these metrics, consider [adding nodes](cockroach-start.html) to your cluster before beginning your upgrade. - -### Check decommissioned nodes - -If your cluster contains partially-decommissioned nodes, they will block an upgrade attempt. - -1. To check the status of decommissioned nodes, run the [`cockroach node status --decommission`](cockroach-node.html) command: - - {% include_cached copy-clipboard.html %} - ~~~ shell - cockroach node status --decommission - ~~~ - - In the output, verify that the value of the `membership` field of each node is `decommissioned`. If any node's `membership` value is `decommissioning`, that node is not fully decommissioned. - -1. If any node is not fully decommissioned, try the following: - - 1. **Before upgrading from v20.2 to v21.1**, you must manually change the status of any `decommissioning` nodes to `decommissioned`. To do this, [run `cockroach node decommission`](remove-nodes.html#step-2-start-the-decommissioning-process-on-the-nodes) on these nodes and confirm that they update to `decommissioned`. - 1. For a cluster running v21.1 and above: - 1. First, reissue the [decommission command](remove-nodes.html). The second command typically succeeds within a few minutes. - 1. If the second decommission command does not succeed, [recommission](remove-nodes.html#recommission-nodes) and then decommission it again. Before continuing the upgrade, the node must be marked as `decommissioned`. - -### Review breaking changes - -Review the [backward-incompatible changes in v21.1](../releases/v21.1.html#v21-1-0-backward-incompatible-changes) and [deprecated features](../releases/v21.1.html#v21-1-0-deprecations). If any affect your deployment, make the necessary changes before starting the rolling upgrade to v21.1. - -Two changes that are particularly important to note: - -- As of v21.1, CockroachDB always uses the [Pebble storage engine](https://github.com/cockroachdb/pebble). As such, `pebble` is the default and only option for the `--storage-engine` flag on the `cockroach start` command. RocksDB can no longer be used as the storage engine. - - If your cluster currently uses RocksDB as the storage engine, before you upgrade to v21.1, restart each of your nodes, removing `--storage-engine=rocksdb` from the `cockroach start` command. You can follow the same rolling process described in [step 4](#step-4-perform-the-rolling-upgrade) below, but do not change the binary; just remove the `--storage-engine=rocksdb` flag and restart. - -- [Interleaving data](interleave-in-parent.html) was deprecated in v20.2, disabled by default in v21.1, and removed from CockroachDB in v21.2. For migration steps, see the [interleave deprecation](interleave-in-parent.html#deprecation) notice. - - If your cluster includes interleaved data and you perform backups, first make sure you are running [v20.2.10+](../v20.2/upgrade-cockroach-version.html); then update your `BACKUP` commands to use the [`INCLUDE_DEPRECATED_INTERLEAVES` option](backup.html#include-deprecated-interleaves); and only then return to this page and upgrade to v21.1. Note that the `INCLUDE_DEPRECATED_INTERLEAVES` option is a no-op in v20.2.10, but this sequence is the only way to prevent backups including interleaved data from failing on v21.1. - -## Step 3. Decide how the upgrade will be finalized - -{{site.data.alerts.callout_info}} -This step is relevant only when upgrading from v20.2.x to v21.1. For upgrades within the v21.1.x series, skip this step. -{{site.data.alerts.end}} - -By default, after all nodes are running the new version, the upgrade process will be **auto-finalized**. This will enable certain [features and performance improvements introduced in v21.1](#features-that-require-upgrade-finalization). However, it will no longer be possible to perform a downgrade to v20.2. In the event of a catastrophic failure or corruption, the only option will be to start a new cluster using the old binary and then restore from one of the backups created prior to performing the upgrade. For this reason, **we recommend disabling auto-finalization** so you can monitor the stability and performance of the upgraded cluster before finalizing the upgrade, but note that you will need to follow all of the subsequent directions, including the manual finalization in [step 5](#step-5-finish-the-upgrade): - -1. [Upgrade to v20.2](../v20.2/upgrade-cockroach-version.html), if you haven't already. - -1. Start the [`cockroach sql`](cockroach-sql.html) shell against any node in the cluster. - -1. Set the `cluster.preserve_downgrade_option` [cluster setting](cluster-settings.html): - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SET CLUSTER SETTING cluster.preserve_downgrade_option = '20.2'; - ~~~ - - It is only possible to set this setting to the current cluster version. - -### Features that require upgrade finalization - -When upgrading from v20.2 to v21.1, certain features and performance improvements will be enabled only after finalizing the upgrade, including but not limited to: - -- **Improved multi-region features:** After finalization, it will be possible to use new and improved [multi-region features](multiregion-overview.html), such as the ability to set database regions, survival goals, and table localities. Internal capabilities supporting these features, such as [non-voting replicas](architecture/replication-layer.html#non-voting-replicas) and [non-blocking transactions](architecture/transaction-layer.html#non-blocking-transactions), will be available after finalization as well. - -- **Empty arrays in GIN indexes:** After finalization, newly created [GIN indexes](inverted-indexes.html) will contain rows containing empty arrays in [`ARRAY`](array.html) columns, which allows the indexes to be used for more queries. Note, however, that rows containing `NULL` values in an indexed column will still not be included in GIN indexes. - -- **Virtual computed columns:** After finalization, it will be possible to use the `VIRTUAL` keyword to define [virtual computed columns](computed-columns.html). - -- **Changefeed support for primary key changes:** After finalization, [changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html) will detect primary key changes. - -## Step 4. Perform the rolling upgrade - -For each node in your cluster, complete the following steps. Be sure to upgrade only one node at a time, and wait at least one minute after a node rejoins the cluster to upgrade the next node. Simultaneously upgrading more than one node increases the risk that ranges will lose a majority of their replicas and cause cluster unavailability. - -{{site.data.alerts.callout_success}} -We recommend creating scripts to perform these steps instead of performing them manually. Also, if you are running CockroachDB on Kubernetes, see our documentation on [single-cluster](operate-cockroachdb-kubernetes.html#upgrade-the-cluster) and/or [multi-cluster](orchestrate-cockroachdb-with-kubernetes-multi-cluster.html#upgrade-the-cluster) orchestrated deployments for upgrade guidance instead. -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -These steps perform an upgrade to the latest {{ page.version.version }} release, **{{ latest.release_name}}**. -{{site.data.alerts.end}} - -1. Drain and stop the node using one of the following methods: - - {% include {{ page.version.version }}/prod-deployment/node-shutdown.md %} - - Verify that the process has stopped: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ ps aux | grep cockroach - ~~~ - - Alternately, you can check the node's logs for the message `server drained and shutdown completed`. - -1. Download and install the CockroachDB binary you want to use: - -
    - - -
    -

    - -
    - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ curl https://binaries.cockroachdb.com/cockroach-{{latest.release_name}}.darwin-10.9-amd64.tgz|tar -xzf - - ~~~ - -
    - -
    - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ curl https://binaries.cockroachdb.com/cockroach-{{latest.release_name}}.linux-amd64.tgz|tar -xzf - - ~~~ - -
    - -1. If you use `cockroach` in your `$PATH`, rename the outdated `cockroach` binary, and then move the new one into its place: - -
    - - -
    -

    - -
    - - {% include_cached copy-clipboard.html %} - ~~~ shell - i="$(which cockroach)"; mv "$i" "$i"_old - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cp -i cockroach-{{latest.release_name}}.darwin-10.9-amd64/cockroach /usr/local/bin/cockroach - ~~~ - -
    - -
    - - {% include_cached copy-clipboard.html %} - ~~~ shell - i="$(which cockroach)"; mv "$i" "$i"_old - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cp -i cockroach-{{latest.release_name}}.linux-amd64/cockroach /usr/local/bin/cockroach - ~~~ - -
    - -1. Start the node to have it rejoin the cluster. - - {{site.data.alerts.callout_danger}} - For maximum availability, do not wait more than a few minutes before restarting the node with the new binary. See [this open issue](https://github.com/cockroachdb/cockroach/issues/37906) for context. - {{site.data.alerts.end}} - - Without a process manager like `systemd`, re-run the [`cockroach start`](cockroach-start.html) command that you used to start the node initially, for example: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ cockroach start \ - --certs-dir=certs \ - --advertise-addr= \ - --join=,, - ~~~ - - If you are using `systemd` as the process manager, run this command to start the node: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ systemctl start - ~~~ - -1. Verify the node has rejoined the cluster through its output to [`stdout`](cockroach-start.html#standard-output) or through the [DB Console](ui-cluster-overview-page.html#node-status). - -1. If you use `cockroach` in your `$PATH`, you can remove the old binary: - - {% include_cached copy-clipboard.html %} - ~~~ shell - $ rm /usr/local/bin/cockroach_old - ~~~ - - If you leave versioned binaries on your servers, you do not need to do anything. - -1. After the node has rejoined the cluster, ensure that the node is ready to accept a SQL connection. - - Unless there are tens of thousands of ranges on the node, it's usually sufficient to wait one minute. To be certain that the node is ready, run the following command: - - {% include_cached copy-clipboard.html %} - ~~~ shell - cockroach sql -e 'select 1' - ~~~ - - The command will automatically wait to complete until the node is ready. - -1. Repeat these steps for the next node. - -## Step 5. Finish the upgrade - -{{site.data.alerts.callout_info}} -This step is relevant only when upgrading from v20.2.x to v21.1. For upgrades within the v21.1.x series, skip this step. -{{site.data.alerts.end}} - -If you disabled auto-finalization in [step 3](#step-3-decide-how-the-upgrade-will-be-finalized), monitor the stability and performance of your cluster for as long as you require to feel comfortable with the upgrade (generally at least a day). If during this time you decide to roll back the upgrade, repeat the rolling restart procedure with the old binary. - -Once you are satisfied with the new version: - -1. Start the [`cockroach sql`](cockroach-sql.html) shell against any node in the cluster. - -1. Re-enable auto-finalization: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > RESET CLUSTER SETTING cluster.preserve_downgrade_option; - ~~~ - - {{site.data.alerts.callout_info}} - This statement can take up to a minute to complete, depending on the amount of data in the cluster, as it kicks off various internal maintenance and migration tasks. During this time, the cluster will experience a small amount of additional load. - {{site.data.alerts.end}} - -1. Check the cluster version to confirm that the finalize step has completed: - - {% include_cached copy-clipboard.html %} - ~~~ sql - > SHOW CLUSTER SETTING version; - ~~~ - -## Troubleshooting - -After the upgrade has finalized (whether manually or automatically), it is no longer possible to downgrade to the previous release. If you are experiencing problems, we therefore recommend that you: - -1. Run the [`cockroach debug zip`](cockroach-debug-zip.html) command against any node in the cluster to capture your cluster's state. - -1. [Reach out for support](support-resources.html) from Cockroach Labs, sharing your debug zip. - -In the event of catastrophic failure or corruption, the only option will be to start a new cluster using the old binary and then restore from one of the backups created prior to performing the upgrade. - -## See also - -- [View Node Details](cockroach-node.html) -- [Collect Debug Information](cockroach-debug-zip.html) -- [View Version Details](cockroach-version.html) -- [Release notes for our latest version](../releases/{{page.version.version}}.html) diff --git a/src/current/v21.1/upsert.md b/src/current/v21.1/upsert.md deleted file mode 100644 index e7dded31220..00000000000 --- a/src/current/v21.1/upsert.md +++ /dev/null @@ -1,386 +0,0 @@ ---- -title: UPSERT -summary: The UPSERT statement inserts rows when values do not violate uniqueness constraints, and it updates rows when values do violate uniqueness constraints. -toc: true ---- - -The `UPSERT` [statement](sql-statements.html) inserts rows in cases where specified values do not violate uniqueness constraints and updates rows in cases where values do violate uniqueness constraints. `UPSERT` considers uniqueness only for [primary key](primary-key.html) columns. - -## `UPSERT` vs. `INSERT ON CONFLICT` - -Assuming that columns `a` and `b` are the primary key, the following `UPSERT` and [`INSERT ON CONFLICT`](insert.html#on-conflict-clause) statements are equivalent: - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPSERT INTO t (a, b, c) VALUES (1, 2, 3); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO t (a, b, c) - VALUES (1, 2, 3) - ON CONFLICT (a, b) - DO UPDATE SET c = excluded.c; -~~~ - -If your statement considers uniqueness for columns other than primary key columns, you must use `INSERT ON CONFLICT`. For an example, see the [Upsert that fails (conflict on non-primary key)](#upsert-that-fails-conflict-on-non-primary-key). - -{% include {{page.version.version}}/sql/insert-vs-upsert.md %} - -To learn more about how to perform and when to use an upsert in CockroachDB, PostgreSQL, and MySQL, see [Upsert in SQL: What is an Upsert, and When Should You Use One?](https://www.cockroachlabs.com/blog/sql-upsert/). - -## Considerations - -- An `UPSERT` statement affecting a proper subset of columns behaves differently depending on whether or not you specify the target columns in the statement. - - - If you specify target columns (e.g., `UPSERT INTO accounts (id, name) VALUES (2, 'b2');`), the values of columns that do not have new values in the `UPSERT` statement will not be updated. - - If you do not specify the target columns (e.g., `UPSERT INTO accounts VALUES (2, 'b2');`), the value of columns that do not have new values in the `UPSERT` statement will be updated to their default values. - - For examples, see [Upsert a proper subset of columns](#upsert-a-proper-subset-of-columns). - -- A single [multi-row `UPSERT`](#upsert-multiple-rows) statement is faster than multiple single-row `UPSERT` statements. Whenever possible, use multi-row `UPSERT` instead of multiple single-row `UPSERT` statements. - -- If the input data contains duplicates, see [Import data containing duplicate rows using `DISTINCT ON`](#import-data-containing-duplicate-rows-using-distinct-on). - -## Required privileges - -The user must have the `INSERT`, `SELECT` and `UPDATE` [privileges](authorization.html#assign-privileges) on the table. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/upsert.html %} -
    - -## Parameters - -Parameter | Description -----------|------------ -`common_table_expr` | See [Common Table Expressions](common-table-expressions.html). -`table_name` | The name of the table. -`AS table_alias_name` | An alias for the table name. When an alias is provided, it completely hides the actual table name. -`column_name` | The name of a column to populate during the insert. -`select_stmt` | A [selection query](selection-queries.html). Each value must match the [data type](data-types.html) of its column. Also, if column names are listed after `INTO`, values must be in corresponding order; otherwise, they must follow the declared order of the columns in the table. -`DEFAULT VALUES` | To fill all columns with their [default values](default-value.html), use `DEFAULT VALUES` in place of `select_stmt`. To fill a specific column with its default value, leave the value out of the `select_stmt` or use `DEFAULT` at the appropriate position. -`RETURNING target_list` | Return values based on rows inserted, where `target_list` can be specific column names from the table, `*` for all columns, or computations using [scalar expressions](scalar-expressions.html).

    Within a [transaction](transactions.html), use `RETURNING NOTHING` to return nothing in the response, not even the number of rows affected. - -## How `UPSERT` transforms into `INSERT ON CONFLICT` - -`UPSERT` considers uniqueness only for [primary key](primary-key.html) columns. For example, assuming that columns `a` and `b` are the primary key, the following `UPSERT` and `INSERT ON CONFLICT` statements are equivalent: - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPSERT INTO t (a, b, c) VALUES (1, 2, 3); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO t (a, b, c) - VALUES (1, 2, 3) - ON CONFLICT (a, b) - DO UPDATE SET c = excluded.c; -~~~ - -`INSERT ON CONFLICT` is more flexible and can be used to consider uniqueness for columns not in the primary key. For more details, see the [Upsert that Fails (Conflict on Non-Primary Key)](#upsert-that-fails-conflict-on-non-primary-key) example below. - -## Examples - -### Upsert a row (no conflict) - -In this example, the `id` column is the primary key. Because the inserted `id` value does not conflict with the `id` value of any existing row, the `UPSERT` statement inserts a new row into the table. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM accounts; -~~~ - -~~~ -+----+----------+ -| id | balance | -+----+----------+ -| 1 | 10000.5 | -| 2 | 20000.75 | -+----+----------+ -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPSERT INTO accounts (id, balance) VALUES (3, 6325.20); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM accounts; -~~~ - -~~~ -+----+----------+ -| id | balance | -+----+----------+ -| 1 | 10000.5 | -| 2 | 20000.75 | -| 3 | 6325.2 | -+----+----------+ -~~~ - -### Upsert multiple rows - -In this example, the `UPSERT` statement inserts multiple rows into the table. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM accounts; -~~~ - -~~~ -+----+----------+ -| id | balance | -+----+----------+ -| 1 | 10000.5 | -| 2 | 20000.75 | -| 3 | 6325.2 | -+----+----------+ -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPSERT INTO accounts (id, balance) VALUES (4, 1970.4), (5, 2532.9), (6, 4473.0); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM accounts; -~~~ - -~~~ -+----+----------+ -| id | balance | -+----+----------+ -| 1 | 10000.5 | -| 2 | 20000.75 | -| 3 | 6325.2 | -| 4 | 1970.4 | -| 5 | 2532.9 | -| 6 | 4473.0 | -+----+----------+ -~~~ - -### Upsert that updates a row (conflict on primary key) - -In this example, the `id` column is the primary key. Because the inserted `id` value is not unique, the `UPSERT` statement updates the row with the new `balance`. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM accounts; -~~~ - -~~~ -+----+----------+ -| id | balance | -+----+----------+ -| 1 | 10000.5 | -| 2 | 20000.75 | -| 3 | 6325.2 | -| 4 | 1970.4 | -| 5 | 2532.9 | -| 6 | 4473.0 | -+----+----------+ -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPSERT INTO accounts (id, balance) VALUES (3, 7500.83); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM accounts; -~~~ - -~~~ -+----+----------+ -| id | balance | -+----+----------+ -| 1 | 10000.5 | -| 2 | 20000.75 | -| 3 | 7500.83 | -| 4 | 1970.4 | -| 5 | 2532.9 | -| 6 | 4473.0 | -+----+----------+ -~~~ - -### Upsert that fails (conflict on non-primary key) - -`UPSERT` will not update rows when the uniqueness conflict is on columns not in the primary key. In this example, the `a` column is the primary key, but the `b` column also has the [`UNIQUE` constraint](unique.html). Because the inserted `b` value is not unique, the `UPSERT` fails. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM unique_test; -~~~ - -~~~ -+---+---+ -| a | b | -+---+---+ -| 1 | 1 | -| 2 | 2 | -| 3 | 3 | -+---+---+ -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPSERT INTO unique_test VALUES (4, 1); -~~~ - -~~~ -pq: duplicate key value (b)=(1) violates unique constraint "unique_test_b_key" -~~~ - -In such a case, you would need to use the [`INSERT ON CONFLICT`](insert.html) statement to specify the `b` column as the column with the `UNIQUE` constraint. - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO unique_test VALUES (4, 1) ON CONFLICT (b) DO UPDATE SET a = excluded.a; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM unique_test; -~~~ - -~~~ -+---+---+ -| a | b | -+---+---+ -| 2 | 2 | -| 3 | 3 | -| 4 | 1 | -+---+---+ -~~~ - -### Upsert a proper subset of columns - -~~~ sql -> CREATE TABLE accounts ( - id INT PRIMARY KEY, - name STRING, - balance DECIMAL(10, 2) DEFAULT 0 -); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO accounts (id, name, balance) VALUES - (1, 'a1', 10000.5), - (2, 'b1', 20000.75), - (3, 'c1', 6325.2); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM accounts; -~~~ - -~~~ -+----+------+----------+ -| id | name | balance | -+----+------+----------+ -| 1 | a1 | 10000.50 | -| 2 | b1 | 20000.75 | -| 3 | c1 | 6325.20 | -+----+------+----------+ -~~~ - -Upserting a proper subset of columns without specifying the column names will write the default values of the unspecified columns when there is a conflict on the primary key. The account with `id` of `1` has a balance of `0` (the column's default value) after the `UPSERT`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPSERT INTO accounts VALUES (1, 'a2'); - -> SELECT * FROM accounts; -~~~ - -~~~ -+----+------+----------+ -| id | name | balance | -+----+------+----------+ -| 1 | a2 | 0.00 | -| 2 | b1 | 20000.75 | -| 3 | c1 | 6325.20 | -+----+------+----------+ -~~~ - -If the target column names are included in the `UPSERT`, then the subset of columns without values will not change when there is a conflict on the primary key. The `balance` of the account with `id` of `2` is unchanged after the `UPSERT`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPSERT INTO accounts (id, name) VALUES (2, 'b2'); - -> SELECT * FROM accounts; -~~~ - -~~~ -+----+------+----------+ -| id | name | balance | -+----+------+----------+ -| 1 | a2 | 0.00 | -| 2 | b2 | 20000.75 | -| 3 | c1 | 6325.20 | -+----+------+----------+ -~~~ - -### Import data containing duplicate rows using `DISTINCT ON` - -If the input data to insert/update contains duplicate rows, you must -use [`DISTINCT ON`](select-clause.html#eliminate-duplicate-rows) to -ensure there is only one row for each value of the primary key. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> WITH - -- the following data contains duplicates on the conflict column "id": - inputrows AS (VALUES (8, 130), (8, 140)) - - UPSERT INTO accounts (id, balance) - (SELECT DISTINCT ON(id) id, balance FROM inputrows); -- de-duplicate the input rows -~~~ - -The `DISTINCT ON` clause does not guarantee which of the duplicates is -considered. To force the selection of a particular duplicate, use an -`ORDER BY` clause: - -{% include_cached copy-clipboard.html %} -~~~ sql -> WITH - -- the following data contains duplicates on the conflict column "id": - inputrows AS (VALUES (8, 130), (8, 140)) - - UPSERT INTO accounts (id, balance) - (SELECT DISTINCT ON(id) id, balance - FROM inputrows - ORDER BY balance); -- pick the lowest balance as value to update in each account -~~~ - -{{site.data.alerts.callout_info}} -Using `DISTINCT ON` incurs a performance cost to search and eliminate duplicates. -For best performance, avoid using it when the input is known to not contain duplicates. -{{site.data.alerts.end}} - -{% include {{page.version.version}}/sql/limit-row-size.md %} - -## See also - -- [Ordering of rows in DML statements](order-by.html#ordering-rows-in-dml-statements) -- [Selection Queries](selection-queries.html) -- [`DELETE`](delete.html) -- [`INSERT`](insert.html) -- [`UPDATE`](update.html) -- [`TRUNCATE`](truncate.html) -- [`ALTER TABLE`](alter-table.html) -- [`DROP TABLE`](drop-table.html) -- [`DROP DATABASE`](drop-database.html) -- [Other SQL Statements](sql-statements.html) diff --git a/src/current/v21.1/use-a-local-file-server-for-bulk-operations.md b/src/current/v21.1/use-a-local-file-server-for-bulk-operations.md deleted file mode 100644 index eaa61628a2e..00000000000 --- a/src/current/v21.1/use-a-local-file-server-for-bulk-operations.md +++ /dev/null @@ -1,104 +0,0 @@ ---- -title: Use a Local File Server for Bulk Operations -summary: Learn how to create a simple file server for use with bulk operations within CockroachDB -toc: true ---- - -If you need a location to store files for the [`IMPORT`](import.html) process, but do not have access to (or cannot use) [cloud storage providers](use-cloud-storage-for-bulk-operations.html), you can run a local file server. You can then use this file server by leveraging support for our [HTTP Export Storage API](#http-export-storage-api). - -This is especially useful for: - -- Implementing a compatibility layer in front of custom or proprietary storage providers for which CockroachDB does not yet have built-in support -- Using on-premises storage - -{{site.data.alerts.callout_info}} -HTTP file servers are not supported as storage for [backups](take-full-and-incremental-backups.html). -{{site.data.alerts.end}} - -## HTTP export storage API - -A CockroachDB [`IMPORT`](import.html) process that requires reading or writing external files can use the HTTP Export Storage API by prefacing the address with `http`, e.g., `http://fileserver/mnt/cockroach-exports`. - -This API uses the `GET`, `PUT` and `DELETE` methods. This behaves like you would expect typical HTTP requests to work. After a `PUT` request to some path, a subsequent `GET` request should return the content sent in the `PUT` request body, at least until a `DELETE` request is received for that path. - -## Examples - -You can use any file server software that supports `GET`, `PUT` and `DELETE` methods, but we've included code samples for common ones: - -- [Using PHP with `IMPORT`](#using-php-with-import) -- [Using Python with `IMPORT`](#using-python-with-import) -- [Using Ruby with `IMPORT`](#using-ruby-with-import) -- [Using nginx as a file server](#using-nginx-as-a-file-server) - -{{site.data.alerts.callout_info}}We do not recommend using any machines running cockroach as file servers. Using machines that are running cockroach as file servers could negatively impact performance if I/O operations exceed capacity.{{site.data.alerts.end}} - -### Using PHP with `IMPORT` - -The PHP language has an HTTP server built in. You can serve local files using the commands below. For more information about how to import these locally served files, see the documentation for the [`IMPORT`][import] statement. - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cd /path/to/data -$ php -S 127.0.0.1:3000 # files available at e.g., 'http://localhost:3000/data.sql' -~~~ - -### Using Python with `IMPORT` - -The Python language has an HTTP server included in the standard library. You can serve local files using the commands below. For more information about how to import these locally served files, see the documentation for the [`IMPORT`][import] statement. - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cd /path/to/data -$ python -m SimpleHTTPServer 3000 # files available at e.g., 'http://localhost:3000/data.sql' -~~~ - -If you use Python 3, try: - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cd /path/to/data -$ python -m http.server 3000 -~~~ - -### Using Ruby with `IMPORT` - -The Ruby language has an HTTP server included in the standard library. You can serve local files using the commands below. For more information about how to import these locally served files, see the documentation for the [`IMPORT`][import] statement. - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cd /path/to/data -$ ruby -run -ehttpd . -p3000 # files available at e.g., 'http://localhost:3000/data.sql' -~~~ - -### Using nginx as a file server - -1. Install `nginx` with the `webdav` module (often included in `-full` or similarly named packages in various distributions). - -2. In the `nginx.conf` file, add a `dav_methods PUT DELETE` directive. For example: - - {% include_cached copy-clipboard.html %} - ~~~ nginx - events { - worker_connections 1024; - } - http { - server { - listen 20150; - location / { - dav_methods PUT DELETE; - root /mnt/cockroach-exports; - sendfile on; - sendfile_max_chunk 1m; - } - } - } - ~~~ - -## See also - -- [`IMPORT`][import] -- [Use Cloud Storage for Bulk Operations](use-cloud-storage-for-bulk-operations.html) - - - -[import]: import.html diff --git a/src/current/v21.1/use-cloud-storage-for-bulk-operations.md b/src/current/v21.1/use-cloud-storage-for-bulk-operations.md deleted file mode 100644 index 3a636bd682d..00000000000 --- a/src/current/v21.1/use-cloud-storage-for-bulk-operations.md +++ /dev/null @@ -1,241 +0,0 @@ ---- -title: Use Cloud Storage for Bulk Operations -summary: CockroachDB constructs a secure API call to the cloud storage specified in a URL passed to bulk operation statements. -toc: true ---- - -CockroachDB constructs a secure API call to the cloud storage specified in a URL passed to one of the following statements: - -- [`BACKUP`](backup.html) -- [`RESTORE`](restore.html) -- [`IMPORT`](import.html) -- [`EXPORT`](export.html) -- [`CREATE CHANGEFEED`](create-changefeed.html) - -{{site.data.alerts.callout_success}} -We strongly recommend using cloud/remote storage. -{{site.data.alerts.end}} - -## URL format - -URLs for the files you want to import must use the format shown below. For examples, see [Example file URLs](#example-file-urls). - -~~~ -[scheme]://[host]/[path]?[parameters] -~~~ - -Location | Scheme | Host | Parameters -------------------------------------------------------------+-------------+--------------------------------------------------+---------------------------------------------------------------------------- -Amazon | `s3` | Bucket name | `AUTH` (optional; can be `implicit` or `specified`), `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`

    For more information, see [Authentication — Amazon S3](#authentication). -Azure | `azure` | Storage container | `AZURE_ACCOUNT_KEY`, `AZURE_ACCOUNT_NAME` -Google Cloud | `gs` | Bucket name | `AUTH` (optional; can be `default`, `implicit`, or `specified`), `CREDENTIALS`

    **Deprecation notice:** In v21.1, we suggest you do not use the `cloudstorage.gs.default.key` [cluster setting](cluster-settings.html), as the default behavior will be changing in v21.2. For more information, see [Authentication - Google Cloud Storage](#authentication). -HTTP | `http` | Remote host | N/A

    For more information, see [Authentication — HTTP](#authentication). -NFS/Local [1](#considerations) | `nodelocal` | `nodeID` or `self` [2](#considerations) (see [Example file URLs](#example-file-urls)) | N/A -S3-compatible services | `s3` | Bucket name | **Warning:** Unlike Amazon S3, Google Cloud Storage, and Azure Storage options, the usage of S3-compatible services is not actively tested by Cockroach Labs.

    `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`, `AWS_REGION` [3](#considerations) (optional), `AWS_ENDPOINT`

    For more information, see [Authentication - S3-compatible services](#authentication). - -{{site.data.alerts.callout_success}} -The location parameters often contain special characters that need to be URI-encoded. Use Javascript's [encodeURIComponent](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent) function or Go language's [url.QueryEscape](https://golang.org/pkg/net/url/#QueryEscape) function to URI-encode the parameters. Other languages provide similar functions to URI-encode special characters. -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -You can disable the use of implicit credentials when accessing external cloud storage services for various bulk operations by using the [`--external-io-disable-implicit-credentials` flag](cockroach-start.html#security). -{{site.data.alerts.end}} - - - -1 The file system backup location on the NFS drive is relative to the path specified by the [`--external-io-dir`](cockroach-start.html#flags-external-io-dir) flag set while [starting the node](cockroach-start.html). If the flag is set to `disabled`, then imports from local directories and NFS drives are disabled. - -2 Using a `nodeID` is required and the data files will be in the `extern` directory of the specified node. In most cases (including single-node clusters), using `nodelocal://1/` is sufficient. Use `self` if you do not want to specify a `nodeID`, and the individual data files will be in the `extern` directories of arbitrary nodes; however, to work correctly, each node must have the [`--external-io-dir`](cockroach-start.html#flags-external-io-dir) flag point to the same NFS mount or other network-backed, shared storage. - -3 The `AWS_REGION` parameter is optional since it is not a required parameter for most S3-compatible services. Specify the parameter only if your S3-compatible service requires it. - -### Example file URLs - -Example URLs for [`BACKUP`](backup.html), [`RESTORE`](restore.html), [`EXPORT`](export.html), or [changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html) given a bucket or container name of `acme-co` and an `employees` subdirectory: - -Location | Example --------------+---------------------------------------------------------------------------------- -Amazon S3 | `s3://acme-co/employees?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456` -Azure | `azure://acme-co/employees?AZURE_ACCOUNT_NAME=acme-co&AZURE_ACCOUNT_KEY=url-encoded-123` -Google Cloud | `gs://acme-co/employees?AUTH=specified&CREDENTIALS=encoded-123` -NFS/Local | `nodelocal://1/path/employees`, `nodelocal://self/nfsmount/backups/employees` [2](#considerations) - -{{site.data.alerts.callout_info}} -URLs for [changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html) should be prepended with `experimental-`. -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -Currently, cloud storage sinks (for changefeeds) only work with `JSON` and emits newline-delimited `JSON` files. -{{site.data.alerts.end}} - -Example URLs for [`IMPORT`](import.html) given a bucket or container name of `acme-co` and a filename of `employees`: - -Location | Example --------------+---------------------------------------------------------------------------------- -Amazon S3 | `s3://acme-co/employees.sql?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456` -Azure | `azure://acme-co/employees.sql?AZURE_ACCOUNT_NAME=acme-co&AZURE_ACCOUNT_KEY=url-encoded-123` -Google Cloud | `gs://acme-co/employees.sql?AUTH=specified&CREDENTIALS=encoded-123` -HTTP | `http://localhost:8080/employees.sql` -NFS/Local | `nodelocal://1/path/employees`, `nodelocal://self/nfsmount/backups/employees` [2](#considerations) - -{{site.data.alerts.callout_info}} -HTTP storage can only be used for [`IMPORT`](import.html). -{{site.data.alerts.end}} - -## Encryption - -[Transport Layer Security (TLS)](https://en.wikipedia.org/wiki/Transport_Layer_Security) is used for encryption in transit when transmitting data to or from Amazon S3, Google Cloud Storage, and Azure. - -For encryption at rest, if your cloud provider offers transparent data encryption, you can use that to ensure that your backups are not stored on disk in cleartext. - -* [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.html) -* [Azure Storage](https://docs.microsoft.com/en-us/azure/storage/common/storage-service-encryption#about-azure-storage-encryption) -* [Google Cloud Storage](https://cloud.google.com/storage/docs/encryption) - -CockroachDB also provides client-side encryption of backup data, for more information, see [Take and Restore Encrypted Backups](take-and-restore-encrypted-backups.html). - -## Authentication - -When running bulk operations to and from a storage bucket, authentication setup can vary depending on the cloud provider. This section details the necessary steps to authenticate to each cloud provider. - -{{site.data.alerts.callout_info}} -`implicit` authentication **cannot** be used to run bulk operations from CockroachDB {{ site.data.products.cloud }} clusters—instead, use `AUTH=specified`. -{{site.data.alerts.end}} - -
    - - - - - -
    - -
    - -The `AUTH` parameter passed to the file URL must be set to either `specified` or `implicit`. The following sections describe how to set up each authentication method. - -### Specified authentication - -If the `AUTH` parameter is not provided, AWS connections default to `specified` and the access keys must be provided in the URI parameters. - -As an example: - -{% include_cached copy-clipboard.html %} -~~~sql -BACKUP DATABASE INTO 's3://{bucket name}/{path in bucket}/?AWS_ACCESS_KEY_ID={access key ID}&AWS_SECRET_ACCESS_KEY={secret access key}'; -~~~ - -### Implicit authentication - -If the `AUTH` parameter is `implicit`, the access keys can be omitted and [the credentials will be loaded from the environment](https://docs.aws.amazon.com/sdk-for-go/api/aws/session/), i.e. the machines running the backup. - -~~~sql -BACKUP DATABASE INTO 's3://{bucket name}/{path}?AUTH=implicit'; -~~~ - -You [can associate an EC2 instance with an IAM role](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html) to provide implicit access to S3 storage within the IAM role's policy. In the following command, the `instance example` EC2 instance is [associated](https://docs.aws.amazon.com/cli/latest/reference/ec2/associate-iam-instance-profile.html) with the `example profile` instance profile, giving the EC2 instance implicit access to any `example profile` S3 buckets. - -{% include_cached copy-clipboard.html %} -~~~shell -aws ec2 associate-iam-instance-profile --iam-instance-profile Name={example profile} --region={us-east-2} --instance-id {instance example} -~~~ - -
    - -
    - -The `AUTH` parameter passed to the file URL must be set to either `specified` or `implicit`. The following sections describe how to set up each authentication method. - -In v21.1 and earlier, if no `AUTH` parameter is provided with a Google Cloud Storage URI then authentication will default to `default`. This means that the connection will only use the key provided in the `cloudstorage.gs.default.key` [cluster setting](cluster-settings.html), and will error if not present. - -{% include {{ page.version.version }}/backups/gcs-default-deprec.md %} - -### Specified authentication - -To access the storage bucket with `specified` credentials, it's necessary to [create a service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts) and add the service account address to the permissions on the specific storage bucket. - -[The JSON credentials file for authentication](https://cloud.google.com/iam/docs/creating-managing-service-account-keys#iam-service-account-keys-create-console) can be downloaded from the **Service Accounts** page in the Google Cloud Console and then [base64-encoded](https://tools.ietf.org/html/rfc4648): - -{% include_cached copy-clipboard.html %} -~~~shell -cat gcs_key.json | base64 -~~~ - -Pass the encoded JSON object to the `CREDENTIALS` parameter: - -{% include_cached copy-clipboard.html %} -~~~sql -BACKUP DATABASE INTO 'gs://{bucket name}/{path}?AUTH=specified&CREDENTIALS={encoded key}'; -~~~ - -### Implicit authentication - -For CockroachDB instances that are running within a Google Cloud Environment, [environment data](https://cloud.google.com/docs/authentication/production#providing_credentials_to_your_application) can be used from the [service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts) to implicitly access resources within the storage bucket. - -For CockroachDB clusters running in other environments, `implicit` authentication access can still be set up manually with the following steps: - - 1. [Create a service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts) and add the service account address to the permissions on the specific storage bucket. - - 1. Download the [JSON credentials file](https://cloud.google.com/iam/docs/creating-managing-service-account-keys#iam-service-account-keys-create-console) from the **Service Accounts** page in the Google Cloud Console to the machines that CockroachDB is running on. (Since this file will be passed as an environment variable, it does **not** need to be base64-encoded.) Ensure that the file is located in a path that CockroachDB can access. - - 1. Create an environment variable instructing CockroachDB where the credentials file is located. The environment variable must be exported on each CockroachDB node: - - {% include_cached copy-clipboard.html %} - ~~~shell - export GOOGLE_APPLICATION_CREDENTIALS="/{cockroach}/gcs_key.json" - ~~~ - - Alternatively, to pass the credentials using [`systemd`](https://www.freedesktop.org/wiki/Software/systemd/), use `systemctl edit cockroach.service` to add the environment variable `Environment="GOOGLE_APPLICATION_CREDENTIALS=gcs-key.json"` under `[Service]` in the `cockroach.service` unit file. Then, run `systemctl daemon-reload` to reload the `systemd` process. Restart the `cockroach` process on each of the cluster's nodes with `systemctl restart cockroach`, which will reload the configuration files. - - To pass the credentials using code, see [Google's Authentication documentation](https://cloud.google.com/docs/authentication/production#passing_code). - - 1. Run a backup (or other bulk operation) to the storage bucket with the `AUTH` parameter set to `implicit`: - - {% include_cached copy-clipboard.html %} - ~~~sql - BACKUP DATABASE INTO 'gs://{bucket name}/{path}?AUTH=implicit'; - ~~~ - -{{site.data.alerts.callout_info}} -If the use of implicit credentials is disabled with [`--external-io-disable-implicit-credentials` flag](cockroach-start.html#security), an error will be returned when accessing external cloud storage services for various bulk operations when using `AUTH=implicit`. -{{site.data.alerts.end}} - -
    - -
    - -To access Azure storage containers, it is sometimes necessary to [url encode](https://en.wikipedia.org/wiki/Percent-encoding) the account key since it is base64-encoded and may contain `+`, `/`, `=` characters. For example: - -{% include_cached copy-clipboard.html %} -~~~sql -BACKUP DATABASE INTO 'azure://{container name}/{path}?AZURE_ACCOUNT_NAME={account name}&AZURE_ACCOUNT_KEY={url-encoded key}'; -~~~ - -
    - -
    - -If your environment requires an HTTP or HTTPS proxy server for outgoing connections, you can set the standard `HTTP_PROXY` and `HTTPS_PROXY` [environment variables](cockroach-commands.html#environment-variables) when starting CockroachDB. You can create your own HTTP server with [NGINX](use-a-local-file-server-for-bulk-operations.html). A custom root CA can be appended to the system's default CAs by setting the `cloudstorage.http.custom_ca` [cluster setting](cluster-settings.html), which will be used when verifying certificates from HTTPS URLs. - -If you cannot run a full proxy, you can disable external HTTP(S) access (as well as custom HTTP(S) endpoints) when importing by using the [`--external-io-disable-http` flag](cockroach-start.html#flags-external-io-disable-http). - -
    - -
    - -{{site.data.alerts.callout_danger}} -Unlike Amazon S3, Google Cloud Storage, and Azure Storage options, the usage of S3-compatible services is not actively tested by Cockroach Labs. -{{site.data.alerts.end}} - -A custom root CA can be appended to the system's default CAs by setting the `cloudstorage.http.custom_ca` [cluster setting](cluster-settings.html), which will be used when verifying certificates from an S3-compatible service. - -
    - -## See also - -- [`BACKUP`](backup.html) -- [`RESTORE`](restore.html) -- [`IMPORT`](import.html) -- [`EXPORT`](export.html) -- [`CREATE CHANGEFEED`](create-changefeed.html) -- [Cluster Settings](cluster-settings.html) diff --git a/src/current/v21.1/use-userfile-for-bulk-operations.md b/src/current/v21.1/use-userfile-for-bulk-operations.md deleted file mode 100644 index 12946f0c1a4..00000000000 --- a/src/current/v21.1/use-userfile-for-bulk-operations.md +++ /dev/null @@ -1,104 +0,0 @@ ---- -title: Use Userfile for Bulk Operations -summary: The IMPORT statement imports various types of data into CockroachDB.with user-scoped storage. -toc: true ---- - - To put files on your CockroachDB cluster without external servers, use `userfile`, a per-user bulk file storage. To interact with `userfile`, use the following [commands](cockroach-commands.html): - -- [`cockroach userfile upload`](#upload-a-file) -- [`cockroach userfile list`](#list-files) -- [`cockroach userfile get`](#get-files) -- [`cockroach userfile delete`](#delete-files) - -Once a userfile is uploaded, you can run [`IMPORT`](#import-from-userfile). - -{% include_cached new-in.html version="v21.1" %} For `PGDUMP` and `MYSQLDUMP` formats, you can use [`cockroach import`](#upload-and-import-from-a-dump-file) to upload a userfile, import its data, and delete the userfile in one command. - -## Upload a file - -{{site.data.alerts.callout_info}} -A userfile uses storage space in the cluster, and is replicated with the rest of the cluster's data. We recommend using [`cockroach userfile upload`](cockroach-userfile-upload.html) for quick uploads from your client (about 15MB or smaller). -{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach userfile upload /Users/maxroach/Desktop/test-data.csv /test-data.csv --certs-dir=certs -~~~ - -~~~ -successfully uploaded to userfile://defaultdb.public.userfiles_root/test-data.csv -~~~ - -For more information, see [`cockroach userfile upload`](cockroach-userfile-upload.html). - -## List files - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach userfile list '*.csv' --certs-dir=certs -~~~ - -~~~ -userfile://defaultdb.public.userfiles_root/test-data-2.csv -userfile://defaultdb.public.userfiles_root/test-data.csv -~~~ - -For more information, see [`cockroach userfile list`](cockroach-userfile-list.html). - -## Get files - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach userfile get test-data.csv --certs-dir=certs -~~~ - -For more information, see [`cockroach userfile get`](cockroach-userfile-get.html). - -## Delete files - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach userfile delete test-data.csv --certs-dir=certs -~~~ - -~~~ -deleted userfile://defaultdb.public.userfiles_root/test-data.csv -~~~ - -For more information, see [`cockroach userfile delete`](cockroach-userfile-delete.html). - -## Upload and import from a dump file - -{{site.data.alerts.callout_info}} -We recommend using [`cockroach import`](cockroach-import.html) for quick imports from your client (about 15MB or smaller). For larger imports, use the [IMPORT](import.html) statement. -{{site.data.alerts.end}} - -{% include_cached copy-clipboard.html %} -~~~ shell -$ cockroach import db mysqldump /Users/maxroach/Desktop/test-db.sql --certs-dir=certs -~~~ - -~~~ -successfully imported mysqldump file /Users/maxroach/Desktop/test-db.sql -~~~ - -For more information, see [`cockroach import`](cockroach-import.html). - -## Import from `userfile` - -{% include {{ page.version.version }}/userfile-examples/import-into-userfile.md %} - -## Backup and restore with `userfile` - -{% include {{ page.version.version }}/userfile-examples/backup-userfile.md %} - -## See also - -- [`cockroach userfile upload`](cockroach-userfile-upload.html) -- [`cockroach userfile list`](cockroach-userfile-list.html) -- [`cockroach userfile get`](cockroach-userfile-get.html) -- [`cockroach userfile delete`](cockroach-userfile-delete.html) -- [Other Cockroach Commands](cockroach-commands.html) -- [`IMPORT`](import.html) -- [`IMPORT INTO`](import-into.html) diff --git a/src/current/v21.1/uuid.md b/src/current/v21.1/uuid.md deleted file mode 100644 index 893ccecfd71..00000000000 --- a/src/current/v21.1/uuid.md +++ /dev/null @@ -1,137 +0,0 @@ ---- -title: UUID -summary: The UUID data type stores 128-bit Universal Unique Identifiers. -toc: true ---- - -The `UUID` (Universally Unique Identifier) [data type](data-types.html) stores a 128-bit value that is [unique across both space and time](https://www.ietf.org/rfc/rfc4122.txt). - -{{site.data.alerts.callout_success}} -To auto-generate unique row identifiers, use [`UUID`](uuid.html) with the `gen_random_uuid()` function as the default value. See the [example](#create-a-table-with-auto-generated-unique-row-ids) below for more details. -{{site.data.alerts.end}} - -## Syntax - -You can express `UUID` values using the following formats: - -Format | Description --------|------------- -Standard [RFC4122](http://www.ietf.org/rfc/rfc4122.txt) format | Hyphen-separated groups of 8, 4, 4, 4, and 12 hexadecimal digits.

    Example: `acde070d-8c4c-4f0d-9d8a-162843c10333` -`BYTES` | `UUID` value specified as a [`BYTES`](bytes.html) value.

    Example: `b'kafef00ddeadbeed'` -Uniform Resource Name | A [Uniform Resource Name (URN)](https://www.ietf.org/rfc/rfc2141.txt) specified as "urn:uuid:" followed by the [RFC4122](http://www.ietf.org/rfc/rfc4122.txt) format.

    Example: `urn:uuid:63616665-6630-3064-6465-616462656564` -Alternate RFC4122 formats | The [RFC4122](http://www.ietf.org/rfc/rfc4122.txt) format, surrounded by braces, expressed with upper-case digits, or with hyphens omitted.

    Examples: `{acde070d-8c4c-4f0d-9d8a-162843c10333}`, `ACDE070D-8C4C-4f0D-9d8A-162843c10333`, `acde070d8c4c4f0d9d8a162843c10333` - -CockroachDB displays all `UUID` values in the standard [RFC4122](http://www.ietf.org/rfc/rfc4122.txt) format. - -## Size - -A `UUID` value is 128 bits in width, but the total storage size is likely to be larger due to CockroachDB metadata. - -## Examples - -### Create a table with manually-entered `UUID` values - -#### Create a table with `UUID` in standard [RFC4122](http://www.ietf.org/rfc/rfc4122.txt)-specified format - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TABLE v (token uuid); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO v VALUES ('63616665-6630-3064-6465-616462656562'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM v; -~~~ - -~~~ - token ----------------------------------------- - 63616665-6630-3064-6465-616462656562 -(1 row) -~~~ - -#### Create a table with `UUID` in `BYTE` format - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO v VALUES (b'kafef00ddeadbeed'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM v; -~~~ - -~~~ - token ----------------------------------------- - 63616665-6630-3064-6465-616462656562 - 6b616665-6630-3064-6465-616462656564 -(2 rows) -~~~ - -#### Create a table with `UUID` used as URN - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO v VALUES ('urn:uuid:63616665-6630-3064-6465-616462656564'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM v; -~~~ - -~~~ - token ----------------------------------------- - 63616665-6630-3064-6465-616462656562 - 6b616665-6630-3064-6465-616462656564 - 63616665-6630-3064-6465-616462656564 -(3 rows) -~~~ - -#### Express UUIDs in alternate formats - -{% include_cached copy-clipboard.html %} -~~~ sql -> INSERT INTO v VALUES ('{acde070d-8c4c-4f0d-9d8a-162843c10333}'), ('ACDE070D-8C4C-4f0D-9d8A-162843c10333'), ('acde070d8c4c4f0d9d8a162843c10333'); -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM v; -~~~ - -~~~ - token ----------------------------------------- - 63616665-6630-3064-6465-616462656562 - 6b616665-6630-3064-6465-616462656564 - 63616665-6630-3064-6465-616462656564 - acde070d-8c4c-4f0d-9d8a-162843c10333 - acde070d-8c4c-4f0d-9d8a-162843c10333 - acde070d-8c4c-4f0d-9d8a-162843c10333 -(6 rows) -~~~ - -### Create a table with auto-generated unique row IDs - -{% include {{ page.version.version }}/faq/auto-generate-unique-ids.html %} - -## Supported casting and conversion - -`UUID` values can be [cast](data-types.html#data-type-conversions-and-casts) to the following data type: - -Type | Details ------|-------- -`BYTES` | Requires supported [`BYTES`](bytes.html) string format, e.g., `b'\141\061\142\062\143\063'`. - -## See also - -[Data Types](data-types.html) diff --git a/src/current/v21.1/validate-constraint.md b/src/current/v21.1/validate-constraint.md deleted file mode 100644 index 885b3f424e9..00000000000 --- a/src/current/v21.1/validate-constraint.md +++ /dev/null @@ -1,60 +0,0 @@ ---- -title: VALIDATE CONSTRAINT -summary: Use the ADD COLUMN statement to add columns to tables. -toc: true ---- - -The `VALIDATE CONSTRAINT` [statement](sql-statements.html) is part of `ALTER TABLE` and checks whether values in a column match a [constraint](constraints.html) on the column. This statement is especially useful after applying a constraint to an existing column via [`ADD CONSTRAINT`](add-constraint.html). In this case, `VALIDATE CONSTRAINT` can be used to find values already in the column that do not match the constraint. - -{% include {{ page.version.version }}/sql/combine-alter-table-commands.md %} - -## Required privileges - -The user must have the `CREATE` [privilege](authorization.html#assign-privileges) on the table. - -## Synopsis - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/validate_constraint.html %} -
    - -## Parameters - - Parameter | Description --------------------+----------------------------------------------------------------------------- - `table_name` | The name of the table in which the constraint you'd like to validate lives. - `constraint_name` | The name of the constraint on `table_name` you'd like to validate. - -## Viewing schema changes - -{% include {{ page.version.version }}/misc/schema-change-view-job.md %} - -## Examples - -In [`ADD CONSTRAINT`](add-constraint.html), we [added a foreign key constraint](add-constraint.html#add-the-foreign-key-constraint-with-cascade) like so: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE orders ADD CONSTRAINT customer_fk FOREIGN KEY (customer_id) REFERENCES customers (id) ON DELETE CASCADE; -~~~ - -In order to ensure that the data added to the `orders` table prior to the creation of the `customer_fk` constraint conforms to that constraint, run the following: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE orders VALIDATE CONSTRAINT customer_fk; -~~~ - -{{site.data.alerts.callout_info}} -If present in a [`CREATE TABLE`](create-table.html) statement, the table is considered validated because an empty table trivially meets its constraints. -{{site.data.alerts.end}} - -## See also - -- [Constraints](constraints.html) -- [`ADD CONSTRAINT`](add-constraint.html) -- [`SHOW CONSTRAINTS`](show-constraints.html) -- [`RENAME CONSTRAINT`](rename-constraint.html) -- [`DROP CONSTRAINT`](drop-constraint.html) -- [`CREATE TABLE`](create-table.html) -- [`SHOW JOBS`](show-jobs.html) diff --git a/src/current/v21.1/vectorized-execution.md b/src/current/v21.1/vectorized-execution.md deleted file mode 100644 index 31cde3f8bf8..00000000000 --- a/src/current/v21.1/vectorized-execution.md +++ /dev/null @@ -1,86 +0,0 @@ ---- -title: Vectorized Query Execution -summary: The CockroachDB vectorized SQL query execution engine processes query plans using a column-oriented model to improve performance. -toc: true ---- - -CockroachDB supports [column-oriented](https://en.wikipedia.org/wiki/Column-oriented_DBMS#Column-oriented_systems) ("vectorized") query execution on all [CockroachDB data types](data-types.html). - -Many SQL databases execute [query plans](https://en.wikipedia.org/wiki/Query_plan) one row of table data at a time. Row-oriented execution models can offer good performance for [online transaction processing (OLTP)](https://en.wikipedia.org/wiki/Online_transaction_processing) queries, but suboptimal performance for [online analytical processing (OLAP)](https://en.wikipedia.org/wiki/Online_analytical_processing) queries. The CockroachDB vectorized execution engine dramatically improves performance over [row-oriented execution](https://en.wikipedia.org/wiki/Column-oriented_DBMS#Row-oriented_systems) by processing each component of a query plan on type-specific batches of column data. - -## Configuring vectorized execution - - By default, vectorized execution is enabled in CockroachDB. - -You can configure vectorized execution with the `vectorize` [session variable](set-vars.html). The following options are supported: - -Option | Description -----------|------------ -`on` | Turns on vectorized execution for all queries on rows over the [`vectorize_row_count_threshold`](#setting-the-row-threshold-for-vectorized-execution) (0 rows, by default, meaning all queries will use the vectorized engine).

    **Default:** `vectorize=on` -`off` | Turns off vectorized execution for all queries. - -For information about setting session variables, see [`SET` <session variable>](set-vars.html). - -{{site.data.alerts.callout_success}} -To see if CockroachDB will use the vectorized execution engine for a query, run a simple [`EXPLAIN`](explain.html) statement on the query. If `vectorize` is `true`, the query will be executed with the vectorized engine. If it is `false`, the row-oriented execution engine is used instead. -{{site.data.alerts.end}} - -### Setting the row threshold for vectorized execution - -The efficiency of vectorized execution increases with the number of rows processed. If you are querying a table with a small number of rows, it is more efficient to use row-oriented execution. - -By default, vectorized execution is enabled for all queries. - -For performance tuning, you can change the minimum number of rows required to use the vectorized engine to execute a query plan in the current session with the `vectorize_row_count_threshold` [session variable](set-vars.html). - -## How vectorized execution works - -When you issue a query, the gateway node (i.e., the node from which you issue the query) [parses the query and creates a physical plan](architecture/sql-layer.html#sql-parser-planner-executor) for execution on each node that receives the plan. If vectorized execution is enabled, the physical plan is sent to each node to be executed by the vectorized execution engine. - -{{site.data.alerts.callout_success}} -To see a detailed view of the vectorized execution plan for a query, run the [`EXPLAIN(VEC)`](explain.html#vec-option) statement on the query. -{{site.data.alerts.end}} - -For information about vectorized execution in the context of the CockroachDB architecture, see [Query Execution](architecture/sql-layer.html#query-execution). - -For detailed examples of vectorized query execution for hash and merge joins, see the blog posts [40x faster hash joiner with vectorized execution](https://www.cockroachlabs.com/blog/vectorized-hash-joiner/) and [Vectorizing the merge joiner in CockroachDB](https://www.cockroachlabs.com/blog/vectorizing-the-merge-joiner-in-cockroachdb/). - -## Disk-spilling operations - -The following operations require [memory buffering](https://en.wikipedia.org/wiki/Data_buffer) during execution: - -- Global [sorts](order-by.html) -- [Unordered aggregations](order-by.html) -- [Hash joins](joins.html#hash-joins) -- [Merge joins](joins.html#merge-joins) on non-unique columns. Merge joins on columns that are guaranteed to have one row per value, also known as "key columns", can execute entirely in-memory. -- [Window functions](window-functions.html). Note that [support for window functions is limited in the vectorized execution engine](#window-functions). - -If there is not enough memory allocated for an operation, CockroachDB will spill the intermediate execution results to disk. By default, the memory limit allocated per operator is 64MiB. You can change this limit with the `sql.distsql.temp_storage.workmem` [cluster setting](cluster-settings.html). - -You can also configure a node's total budget for in-memory query processing at node startup with the [`--max-sql-memory` flag](cockroach-start.html#general). If the queries running on the node exceed the memory budget, the node spills intermediate execution results to disk. The [`--max-disk-temp-storage` flag](cockroach-start.html#general) sets the maximum on-disk storage capacity. If the maximum on-disk storage capacity is reached, the query will return an error during execution. - -## Known limitations - -### Unsupported queries - -The vectorized engine does not support queries containing: - -- A join filtered with an [`ON` expression](joins.html#supported-join-conditions). See [tracking issue](https://github.com/cockroachdb/cockroach/issues/38018). - -### Window functions - -Support for certain [window functions](window-functions.html) is limited in the vectorized execution engine. If a query includes an unsupported window function, the window function will be handled by the row-oriented execution engine. If the same query includes other, supported operations, those operations will be handled by the vectorized execution engine. See [tracking issue](https://github.com/cockroachdb/cockroach/issues/37040). - -### Spatial features - -The vectorized engine does not support [working with spatial data](spatial-data.html). Queries with [geospatial functions](functions-and-operators.html#spatial-functions) or [spatial data](spatial-data.html) will revert to the row-oriented execution engine. - -### Unordered distinct operations - -{% include {{ page.version.version }}/known-limitations/unordered-distinct-operations.md %} - -## See also - -- [SQL Layer](architecture/sql-layer.html) -- [`SET` <session variable>](set-vars.html) -- [`SHOW` <session variable>](show-vars.html) diff --git a/src/current/v21.1/views.md b/src/current/v21.1/views.md deleted file mode 100644 index 9a9a1a10e07..00000000000 --- a/src/current/v21.1/views.md +++ /dev/null @@ -1,621 +0,0 @@ ---- -title: Views -summary: Learn about CockroachDB's dematerialized and materialized views. -toc: true ---- - -A view is a stored and named [selection query](selection-queries.html). By default, CockroachDB's views are **dematerialized**: they do not store the results of the underlying queries. Instead, the underlying query is executed anew every time the view is used. - - CockroachDB also supports [**materialized views**](#materialized-views). Materialized views are views that store their selection query results. - -{{site.data.alerts.callout_info}} - By default, views created in a database cannot reference objects in a different database. To enable cross-database references for views, set the `sql.cross_db_views.enabled` [cluster setting](cluster-settings.html) to `true`. -{{site.data.alerts.end}} - -## Why use views? - -There are various reasons to use views, including: - -- [Hide query complexity](#hide-query-complexity) -- [Limit access to underlying data](#limit-access-to-underlying-data) - -### Hide query complexity - -When you have a complex query that, for example, joins several tables, or performs complex calculations, you can store the query as a view and then select from the view as you would from a standard table. - -#### Example - -Let's say you're using our [sample `startrek` database](cockroach-gen.html#generate-example-data), which contains two tables, `episodes` and `quotes`. - -{% include_cached copy-clipboard.html %} -~~~ sql -> USE startrek; -~~~ - -There's a foreign key constraint between the `episodes.id` column and the `quotes.episode` column. To count the number of famous quotes per season, you could run the following join: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT episodes.season, count(*) - FROM quotes - JOIN episodes - ON quotes.episode = episodes.id - GROUP BY episodes.season; -~~~ - -~~~ - season | count ----------+-------- - 1 | 78 - 2 | 76 - 3 | 46 -(3 rows) -~~~ - -Alternatively, to make it much easier to run this complex query, you could create a view: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE VIEW quotes_per_season (season, quotes) - AS SELECT episodes.season, count(*) - FROM quotes - JOIN episodes - ON quotes.episode = episodes.id - GROUP BY episodes.season - ORDER BY episodes.season; -~~~ - -Then, executing the query is as easy as `SELECT`ing from the view: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM quotes_per_season; -~~~ - -~~~ - season | quotes ----------+--------- - 1 | 78 - 2 | 76 - 3 | 46 -(3 rows) -~~~ - -### Limit access to underlying data - -When you do not want to grant a user access to all the data in one or more standard tables, you can create a view that contains only the columns and/or rows that the user should have access to and then grant the user permissions on the view. - -#### Example - -Let's say you have a `bank` database containing an `accounts` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> USE bank; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM accounts; -~~~ - -~~~ - id | type | balance | email ------+----------+---------+------------------ - 1 | checking | 1000 | max@roach.com - 2 | savings | 10000 | max@roach.com - 3 | checking | 15000 | betsy@roach.com - 4 | checking | 5000 | lilly@roach.com - 5 | savings | 50000 | ben@roach.com -(5 rows) -~~~ - -You want a particular user, `bob`, to be able to see the types of accounts each user has without seeing the balance in each account, so you create a view to expose just the `type` and `email` columns: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE VIEW user_accounts - AS SELECT type, email - FROM accounts; -~~~ - -You then make sure `bob` does not have privileges on the underlying `accounts` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW GRANTS ON accounts; -~~~ - -~~~ - database_name | schema_name | table_name | grantee | privilege_type -----------------+-------------+------------+---------+----------------- - bank | public | accounts | admin | ALL - bank | public | accounts | root | ALL -(2 rows) -~~~ - -Finally, you grant `bob` privileges on the `user_accounts` view: - -{% include_cached copy-clipboard.html %} -~~~ sql -> GRANT SELECT ON user_accounts TO bob; -~~~ - -Now, `bob` will get a permissions error when trying to access the underlying `accounts` table but will be allowed to query the `user_accounts` view: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM accounts; -~~~ - -~~~ -pq: user bob does not have SELECT privilege on table accounts -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM user_accounts; -~~~ - -~~~ - type | email ------------+------------------ - checking | max@roach.com - savings | max@roach.com - checking | betsy@roach.com - checking | lilly@roach.com - savings | ben@roach.com -(5 rows) -~~~ - -## How views work - -### Creating views - -To create a view, use the [`CREATE VIEW`](create-view.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE VIEW quotes_per_season (season, quotes) - AS SELECT episodes.season, count(*) - FROM quotes - JOIN episodes - ON quotes.episode = episodes.id - GROUP BY episodes.season - ORDER BY episodes.season; -~~~ - -{{site.data.alerts.callout_info}} -Any [selection query](selection-queries.html) is valid as operand to `CREATE VIEW`, not just [simple `SELECT` clauses](select-clause.html). -{{site.data.alerts.end}} - -### Listing views - -Once created, views are listed alongside regular tables in the database: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES FROM startrek; -~~~ - -~~~ - schema_name | table_name | type | estimated_row_count ---------------+-------------------+-------+---------------------- - public | episodes | table | 79 - public | quotes | table | 200 - public | quotes_per_season | view | 0 -(3 rows) -~~~ - -To list just views, you can query the `views` table in the [Information Schema](information-schema.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM information_schema.views; -~~~ - -~~~ - table_catalog | table_schema | table_name | view_definition | check_option | is_updatable | is_insertable_into | is_trigger_updatable | is_trigger_deletable | is_trigger_insertable_into -----------------+--------------+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+--------------+--------------------+----------------------+----------------------+----------------------------- - startrek | public | quotes_per_season | SELECT episodes.season, count(*) FROM startrek.public.quotes JOIN startrek.public.episodes ON quotes.episode = episodes.id GROUP BY episodes.season ORDER BY episodes.season | NULL | NO | NO | NO | NO | NO -(1 row) -~~~ - -### Querying views - -To query a view, target it with a [table expression](table-expressions.html#table-or-view-names), for example using a [`SELECT` clause](select-clause.html), just as you would with a stored table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM quotes_per_season; -~~~ - -~~~ - season | quotes ----------+--------- - 1 | 78 - 2 | 76 - 3 | 46 -(3 rows) -~~~ - -`SELECT`ing a view executes the view's stored `SELECT` statement, which returns the relevant data from the underlying table(s). To inspect the `SELECT` statement executed by the view, use the [`SHOW CREATE`](show-create.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW CREATE quotes_per_season; -~~~ - -~~~ - table_name | create_statement ---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - quotes_per_season | CREATE VIEW public.quotes_per_season (season, quotes) AS SELECT episodes.season, count(*) FROM startrek.public.quotes JOIN startrek.public.episodes ON quotes.episode = episodes.id GROUP BY episodes.season ORDER BY episodes.season -(1 row) -~~~ - -You can also inspect the `SELECT` statement executed by a view by querying the `views` table in the [Information Schema](information-schema.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT view_definition FROM information_schema.views WHERE table_name = 'quotes_per_season'; -~~~ - -~~~ - view_definition --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - SELECT episodes.season, count(*) FROM startrek.public.quotes JOIN startrek.public.episodes ON quotes.episode = episodes.id GROUP BY episodes.season ORDER BY episodes.season -(1 row) -~~~ - -### View dependencies - -A view depends on the objects targeted by its underlying query. Attempting to [rename an object](rename-table.html) referenced in a view's stored query therefore results in an error: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE quotes RENAME TO sayings; -~~~ - -~~~ -ERROR: cannot rename relation "startrek.public.quotes" because view "quotes_per_season" depends on it -SQLSTATE: 2BP01 -HINT: you can drop quotes_per_season instead. -~~~ - -Likewise, attempting to [drop an object](drop-table.html) referenced in a view's stored query results in an error: - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP TABLE quotes; -~~~ - -~~~ -ERROR: cannot drop relation "quotes" because view "quotes_per_season" depends on it -SQLSTATE: 2BP01 -HINT: you can drop quotes_per_season instead. -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE episodes DROP COLUMN season; -~~~ - -~~~ -ERROR: cannot drop column "season" because view "quotes_per_season" depends on it -SQLSTATE: 2BP01 -HINT: you can drop quotes_per_season instead. -~~~ - - You can [drop](drop-column.html) or [rename columns](rename-column.html) from a table on which a view is dependent, as long as the view does not depend on that column of the table. For example, because there is no view that depends on the `num` column of the `episodes` table, you can rename it to `number`: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE startrek.episodes RENAME COLUMN num TO number; -~~~ - -Similarly, because no view depends on the `title` column of the `episodes` table, you can drop it. Note that to drop a column with data in it, you must first set `sql_safe_updates = false`. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SET sql_safe_updates = false; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER TABLE startrek.episodes DROP COLUMN title; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW COLUMNS FROM startrek.episodes; -~~~ - -~~~ - column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden ---------------+-----------+-------------+----------------+-----------------------+-----------+------------ - id | INT8 | false | NULL | | {primary} | false - season | INT8 | true | NULL | | {} | false - number | INT8 | true | NULL | | {} | false - stardate | DECIMAL | true | NULL | | {} | false -(4 rows) -~~~ - -When [dropping a table](drop-table.html) or [dropping a view](drop-view.html), you can use the `CASCADE` keyword to drop all dependent objects as well: - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP TABLE quotes CASCADE; -~~~ - -~~~ -DROP TABLE -~~~ - -{{site.data.alerts.callout_danger}} -`CASCADE` drops **all** dependent objects without listing them, which can lead to inadvertent and difficult-to-recover losses. To avoid potential harm, we recommend dropping objects individually in most cases. -{{site.data.alerts.end}} - -### Renaming views - -To rename a view, use the [`ALTER VIEW`](alter-view.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER VIEW quotes_per_season RENAME TO quotes_count; -~~~ - -~~~ -RENAME VIEW -~~~ - -It is not possible to change the stored query executed by the view. Instead, you must drop the existing view and create a new view. - -### Replacing views - - To replace a view, use [`CREATE OR REPLACE VIEW`](create-view.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE OR REPLACE VIEW quotes_count (season, quotes, stardate) - AS SELECT episodes.season, count(*), episodes.stardate - FROM quotes - JOIN episodes - ON quotes.episode = episodes.id - GROUP BY episodes.season, episodes.stardate; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM quotes_count LIMIT 10; -~~~ - -~~~ - season | quotes | stardate ----------+--------+----------- - 1 | 5 | 1709.2 - 1 | 7 | 2821.5 - 1 | 2 | 3113.2 - 1 | 6 | 3134 - 1 | 3 | 2715.1 - 1 | 7 | 3012.4 - 1 | 2 | 3196.1 - 2 | 4 | 3468.1 - 2 | 1 | 3541.9 - 2 | 5 | 4211.4 -(10 rows) -~~~ - -### Removing views - -To remove a view, use the [`DROP VIEW`](drop-view.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP VIEW quotes_count; -~~~ - -~~~ -DROP VIEW -~~~ - -## Materialized views - - CockroachDB supports [materialized views](https://en.wikipedia.org/wiki/Materialized_view). Materialized views are views that store the results of their underlying queries. - -When you [select](selection-queries.html) from a materialized view, the stored query data that is returned might be out-of-date. This contrasts with a standard (i.e., "dematerialized") view, which runs its underlying query every time it is used, returning the latest results. In order to get the latest results from a materialized view, you must [refresh the view](refresh.html), and then select from it. - -Because materialized views store query results, they offer better performance than standard views, at the expense of the additional storage required to store query results and the guarantee that the results are up-to-date. - -### Usage - -Materialized views and standard views share similar syntax for [creating](create-view.html), [showing](show-tables.html), [renaming](alter-view.html), and [dropping](drop-view.html). - -To create a materialized view, use a [`CREATE MATERIALIZED VIEW`](create-view.html) statement. - -For example, suppose that you have the [sample `bank` database](cockroach-workload.html#bank-workload) loaded to a CockroachDB cluster, and populated with some workload values: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE MATERIALIZED VIEW overdrawn_accounts - AS SELECT id, balance - FROM bank - WHERE balance < 0; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM overdrawn_accounts; -~~~ - -~~~ - id | balance -------+---------- - 1 | -17643 - 3 | -5928 - 13 | -3700 -... -(402 rows) -~~~ - -To show existing materialized views, use a [`SHOW TABLES`](show-tables.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SHOW TABLES; -~~~ - -~~~ - schema_name | table_name | type | estimated_row_count ---------------+--------------------+-------------------+---------------------- - public | bank | table | 1000 - public | overdrawn_accounts | materialized view | 0 -(2 rows) -~~~ - -Now suppose you update the `balance` values of the `bank` table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> UPDATE bank SET balance = 0 WHERE balance < 0; -~~~ - -~~~ -UPDATE 402 -~~~ - -The changes can be seen in the table with a simple `SELECT` statement against the table: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT id, balance -FROM bank -WHERE balance < 0; -~~~ - -~~~ - id | balance ------+---------- -(0 rows) -~~~ - -Recall that materialized views do not automatically update their stored results. Selecting from `overdrawn_accounts` returns stored results, which are outdated: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM overdrawn_accounts; -~~~ - -~~~ - id | balance -------+---------- - 1 | -17643 - 3 | -5928 - 13 | -3700 -... -(402 rows) -~~~ - -To update the materialized view's results, use a [`REFRESH`](refresh.html) statement: - -{% include_cached copy-clipboard.html %} -~~~ sql -> REFRESH MATERIALIZED VIEW overdrawn_accounts; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM overdrawn_accounts; -~~~ - -~~~ - id | balance ------+---------- -(0 rows) -~~~ - -To rename the materialized view, use [`ALTER MATERIALIZED VIEW`](alter-view.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> ALTER MATERIALIZED VIEW overdrawn_accounts RENAME TO forgiven_accounts; -~~~ - -~~~ -RENAME VIEW -~~~ - -To remove the materialized view, use [`DROP MATERIALIZED VIEW`](drop-view.html): - -{% include_cached copy-clipboard.html %} -~~~ sql -> DROP MATERIALIZED VIEW forgiven_accounts; -~~~ - -~~~ -DROP VIEW -~~~ - -## Temporary views - -CockroachDB supports session-scoped temporary views. Unlike persistent views, temporary views can only be accessed from the session in which they were created, and they are dropped at the end of the session. You can create temporary views on both persistent tables and [temporary tables](temporary-tables.html). - -{{site.data.alerts.callout_danger}} -**This is an experimental feature**. The interface and output are subject to change. For details, see the tracking issue [cockroachdb/cockroach#46260](https://github.com/cockroachdb/cockroach/issues/46260). -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -Temporary tables must be enabled in order to use temporary views. By default, temporary tables are disabled in CockroachDB. To enable temporary tables, set the `experimental_enable_temp_tables` [session variable](set-vars.html) to `on`. -{{site.data.alerts.end}} - -### Details - -- Temporary views are automatically dropped at the end of the session. -- A temporary view can only be accessed from the session in which it was created. -- Temporary views persist across transactions in the same session. -- Temporary views cannot be converted to persistent views. -- Temporary views can be created on both persistent tables and [temporary tables](temporary-tables.html). -- When you create a view on a temporary table, the view automatically becomes temporary. - -{{site.data.alerts.callout_info}} -Like [temporary tables](temporary-tables.html), temporary views are not in the `public` schema. Instead, when you create the first temporary table, view, or sequence for a session, CockroachDB generates a single temporary schema (`pg_temp_`) for all of the temporary objects in the current session for a database. -{{site.data.alerts.end}} - -### Usage - -To create a temporary view, add [`TEMP`/`TEMPORARY`](sql-grammar.html#opt_temp) to a [`CREATE VIEW`](create-view.html) statement. - -For example: - -{% include_cached copy-clipboard.html %} -~~~ sql -> CREATE TEMP VIEW temp_view (season, quotes) - AS SELECT episodes.season, count(*) - FROM quotes - JOIN episodes - ON quotes.episode = episodes.id - GROUP BY episodes.season; -~~~ - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM temp_view; -~~~ - -~~~ - season | quotes ----------+--------- - 1 | 78 - 2 | 76 - 3 | 46 -(3 rows) -~~~ - -## See also - -- [Selection Queries](selection-queries.html) -- [Simple `SELECT` Clauses](select-clause.html) -- [`CREATE VIEW`](create-view.html) -- [`SHOW CREATE`](show-create.html) -- [`GRANT`](grant.html) -- [`ALTER VIEW`](alter-view.html) -- [`DROP VIEW`](drop-view.html) diff --git a/src/current/v21.1/well-known-binary.md b/src/current/v21.1/well-known-binary.md deleted file mode 100644 index dbe573dd4d5..00000000000 --- a/src/current/v21.1/well-known-binary.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -title: Well Known Binary -summary: The Well Known Binary format provides a representation of a geometric shape that is efficient for machine processing and storage. -toc: true ---- - -The Well Known Binary format (hereafter _WKB_) provides a non-textual, binary representation of a geometric shape. It is used to: - -- Allow shapes to be transferred between CockroachDB and a [SQL client](cockroach-sql.html) or [application](build-a-java-app-with-cockroachdb.html). -- Provide a portable binary format that can be read across different platforms. - -WKB is obtained by serializing a shape as a sequence of numbers. For more detailed information about the structure of the WKB format, see the diagrams showing WKB integer codes in the [OpenGIS Implementation Specification for Geographic information - Simple feature access - Part 1: Common architecture](https://portal.opengeospatial.org/files/?artifact_id=25355). - - - -The Extended Well Known Binary (_EWKB_) format is the same as WKB, with an [SRID](spatial-glossary.html#srid) representation prepended to the data structure. - -## See also - -- [Spatial features](spatial-features.html) -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial indexes](spatial-indexes.html) -- [Spatial and GIS Glossary of Terms](spatial-glossary.html) -- [OpenGIS Implementation Specification for Geographic information - Simple feature access - Part 1: Common architecture](https://portal.opengeospatial.org/files/?artifact_id=25355) -- [Well known text](well-known-text.html) -- [GeoJSON](geojson.html) -- [SRID 4326 - longitude and latitude](srid-4326.html) -- [Introducing Distributed Spatial Data in Free, Open Source CockroachDB](https://www.cockroachlabs.com/blog/spatial-data/) (blog post) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/well-known-text.md b/src/current/v21.1/well-known-text.md deleted file mode 100644 index 4995de2f813..00000000000 --- a/src/current/v21.1/well-known-text.md +++ /dev/null @@ -1,55 +0,0 @@ ---- -title: Well Known Text -summary: The Well Known Text format provides a text representation of a geometric shape. -toc: true ---- - -The Well Known Text format (hereafter _WKT_) provides a text representation of a geometric shape. It can be used to: - -- Build instances of various shapes (e.g., using `ST_MakePolygon` on a Linestring). -- Convert existing shapes to a text representation for ease of reading. - -Each shape in WKT has a _tag_ which says what it is, usually followed by a list of coordinates. For example, here are some common tags: - -- `POINT(...)` -- `LINESTRING(...)` -- `POLYGON((...))` - -WKT is technically case insensitive, but is sometimes displayed using "CamelCase" for easier reading. For example, one can represent a MultiPolygon as either of: - -- `MULTIPOLYGON(...)` -- `MultiPolygon(...)` - -When a shape is made up of homogeneous subcomponents, such as a polygon made up of various points, the subcomponents do not need to have their own tags. For example: - -`LINESTRING(-88.243385 40.116421, -87.906471 43.038902, -95.992775 36.153980)` - - - -Shapes expressed in WKT can have an [SRID](spatial-glossary.html) prepended to the shape and followed by a semicolon, in the form `SRID=123;TAG(...)`. The format is known as Extended Well Known Text (_EWKT_); it is the same as WKT, with an [SRID](spatial-glossary.html#srid) representation prepended to the data structure. - -For example, below is a polygon representing a geometry that uses [SRID 4326](srid-4326.html), which is used to represent latitude and longitude coordinates on the Earth as defined in the [WGS84](spatial-glossary.html#wgs84) standard: - -`SRID=4326;POLYGON((-87.906471 43.038902, -95.992775 36.153980, -75.704722 36.076944, -87.906471 43.038902))` - -{{site.data.alerts.callout_info}} -For more detailed information about the Well Known Text format, see [the OGC specification for WKT](http://docs.opengeospatial.org/is/18-010r7/18-010r7.html). -{{site.data.alerts.end}} - -{{site.data.alerts.callout_info}} -CockroachDB only supports 2-dimensional geometries. -{{site.data.alerts.end}} - -## See also - -- [Spatial features](spatial-features.html) -- [Spatial tutorial](spatial-tutorial.html) -- [Spatial indexes](spatial-indexes.html) -- [Spatial and GIS Glossary of Terms](spatial-glossary.html) -- [Geographic information — Well-known text representation of coordinate reference systems](http://www.opengis.net/doc/is/wkt-crs/2.0.6) -- [OpenGIS Implementation Specification for Geographic information - Simple feature access - Part 1: Common architecture](https://portal.opengeospatial.org/files/?artifact_id=25355) -- [Well known binary](well-known-binary.html) -- [GeoJSON](geojson.html) -- [SRID 4326 - longitude and latitude](srid-4326.html) -- [Introducing Distributed Spatial Data in Free, Open Source CockroachDB](https://www.cockroachlabs.com/blog/spatial-data/) (blog post) -- [Using GeoServer with CockroachDB](geoserver.html) diff --git a/src/current/v21.1/when-to-use-regional-vs-global-tables.md b/src/current/v21.1/when-to-use-regional-vs-global-tables.md deleted file mode 100644 index fb25b93c8e0..00000000000 --- a/src/current/v21.1/when-to-use-regional-vs-global-tables.md +++ /dev/null @@ -1,40 +0,0 @@ ---- -title: When to use REGIONAL vs. GLOBAL tables -summary: Learn how to use CockroachDB's improved multi-region user experience. -toc: false ---- - -{% include_cached new-in.html version="v21.1" %} [_Table Localities_](multiregion-overview.html#table-locality) tell CockroachDB how to optimize access to a table's data in a multi-region cluster. CockroachDB uses the table locality setting to determine how to optimize access to the table's data from that locality. - -The following table localities are available: - -- `REGIONAL` -- `GLOBAL` - -Use [`REGIONAL` tables](multiregion-overview.html#regional-by-row-tables) if: - -- Your application requires low-latency reads and writes from a single region (either at the [row level](multiregion-overview.html#regional-by-row-tables) or the [table level](multiregion-overview.html#regional-tables)). -- Access to the table's data can be slower (higher latency) from other regions. - -Use [`GLOBAL` tables](multiregion-overview.html#global-tables) if: - -- Your application has a "read-mostly" table of reference data that is rarely updated, and that needs to be available to all regions. -- You can accept that writes to the table will incur higher latencies from any given region, since writes use a novel [non-blocking transaction protocol](architecture/transaction-layer.html#non-blocking-transactions) that uses a timestamp "in the future". Note that the observed write latency is dependent on the [`--max-offset`](cockroach-start.html#flags-max-offset) setting. - -For more information about how to choose an overall multi-region configuration, see [Choosing a multi-region configuration](choosing-a-multi-region-configuration.html). - -{{site.data.alerts.callout_success}} -{% include {{page.version.version}}/misc/multiregion-max-offset.md %} -{{site.data.alerts.end}} - -{% include enterprise-feature.md %} - -## See also - -- [Multi-region Overview](multiregion-overview.html) -- [Choosing a multi-region configuration](choosing-a-multi-region-configuration.html) -- [When to use `ZONE` vs. `REGION` survival goals](when-to-use-zone-vs-region-survival-goals.html) -- [Multi-region SQL performance](demo-low-latency-multi-region-deployment.html) -- [Topology Patterns](topology-patterns.html) -- [Disaster Recovery](disaster-recovery.html) -- [Migrate to Multi-region SQL](migrate-to-multiregion-sql.html) diff --git a/src/current/v21.1/when-to-use-zone-vs-region-survival-goals.md b/src/current/v21.1/when-to-use-zone-vs-region-survival-goals.md deleted file mode 100644 index 596ba7c6b04..00000000000 --- a/src/current/v21.1/when-to-use-zone-vs-region-survival-goals.md +++ /dev/null @@ -1,38 +0,0 @@ ---- -title: When to use ZONE vs. REGION survival goals -summary: Learn how to use CockroachDB's improved multi-region user experience. -toc: false ---- - -{% include_cached new-in.html version="v21.1" %} [_Survival Goals_](multiregion-overview.html#survival-goals) dictate how many simultaneous failure(s) a [multi-region database](multiregion-overview.html) can survive. All tables within the same database operate with the same survival goal. Each database is allowed to have its own survival goal setting. - -Allowed survival goals include: - -- `ZONE` (default) -- `REGION` - -Set a [`ZONE` survival goal](multiregion-overview.html#surviving-zone-failures) if: - -- You can accept a single node failure up to an entire zone failure. If multiple zones fail in the same region, the database may become unavailable. - -Set a [`REGION` survival goal](multiregion-overview.html#surviving-region-failures) if: - -- The database must remain available, even if a region goes down. -- You can accept the performance cost: write latency will be increased by at least as much as the round-trip time to the nearest region. Read performance will be unaffected. -- The database can be or already is configured with 3 or more [database regions](multiregion-overview.html#database-regions). At least three database regions are required to survive region failures. - -{{site.data.alerts.callout_success}} -For more information about how to choose a multi-region configuration, see [Choosing a multi-region configuration](choosing-a-multi-region-configuration.html). -{{site.data.alerts.end}} - -{% include enterprise-feature.md %} - -## See also - -+ [Multi-region overview](multiregion-overview.html) -+ [Choosing a multi-region configuration](choosing-a-multi-region-configuration.html) -+ [When to use `REGIONAL` vs `GLOBAL` tables](when-to-use-regional-vs-global-tables.html) -- [Multi-region SQL performance](demo-low-latency-multi-region-deployment.html) -- [Topology Patterns](topology-patterns.html) -- [Disaster Recovery](disaster-recovery.html) -- [Migrate to Multi-region SQL](migrate-to-multiregion-sql.html) diff --git a/src/current/v21.1/window-functions.md b/src/current/v21.1/window-functions.md deleted file mode 100644 index e7789731317..00000000000 --- a/src/current/v21.1/window-functions.md +++ /dev/null @@ -1,388 +0,0 @@ ---- -title: Window Functions -summary: A window function performs a calculation across a set of table rows that are somehow related to the current row. -toc: true ---- - -CockroachDB supports the application of a function over a subset of the rows returned by a [selection query][selection-query]. Such a function is known as a _window function_, and it allows you to compute values by operating on more than one row at a time. The subset of rows a window function operates on is known as a _window frame_. - -For a complete list of supported window functions, see [Functions and Operators](functions-and-operators.html#window-functions). - -{{site.data.alerts.callout_success}} -All [aggregate functions][aggregate-functions] can also be used as [window functions][window-functions]. For more information, see the [Examples](#examples) below. -{{site.data.alerts.end}} - -## Window definitions - -Window frames are defined in [`OVER` clauses](sql-grammar.html#over_clause) or [`WINDOW` clauses](sql-grammar.html#window_clause). - -### Syntax - -
    - - -
    - -

    - -
    - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/window_definition.html %} -
    - -### Parameters - -Parameter | Description -----------|------------- -`window_name` | The name of the new window frame. -`opt_existing_window_name` | An optional name of an existing window frame, defined in a different window definition. -`opt_partition_clause` | An optional `PARTITION BY` clause. -`opt_sort_clause` | An optional `ORDER BY` clause. See [Ordering Query Results](order-by.html) for details. -`opt_frame_clause` | An optional frame clause, which contains a frame boundary and/or an `EXCLUDE` clause. - -
    - -
    - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/window_definition.html %} -
    - -**opt_frame_clause ::=** - -
    -{% include {{ page.version.version }}/sql/generated/diagrams/opt_frame_clause.html %} -
    - -### Parameters - -Parameter | Description -----------|------------- -`window_name` | The name of the new window frame. -`opt_existing_window_name` | An optional name of an existing window frame, defined in a different window definition. -`opt_partition_clause` | An optional `PARTITION BY` clause. -`opt_sort_clause` | An optional `ORDER BY` clause. See [Ordering Query Results](order-by.html) for details. -`frame_bound` | An optional frame boundary.
    Valid start boundaries include `UNBOUNDED PRECEDING`, ` PRECEDING`, and `CURRENT ROW`.
    Valid end boundaries include `UNBOUNDED FOLLOWING`, ` FOLLOWING`, and `CURRENT ROW`. -`opt_frame_exclusion` | An optional frame `EXCLUDE` clause.
    Valid exclusions include `CURRENT ROW`, `GROUP`, `TIES`, and `NO OTHERS`. - -
    - -## How window functions work - -At a high level, window functions work by: - -1. Creating a "virtual table" using a [selection query][selection-query]. -2. Splitting that table into window frames with [window definitions](#window-definitions). You can define window frames in an [`OVER` clause](sql-grammar.html#over_clause), directly after the window function, or in a [`WINDOW` clause](sql-grammar.html#window_clause), as a part of the selection query. -3. Applying the window function to each of the window frames. - -For example, consider a query where the window frames are defined for each window function call: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT DISTINCT(city), - SUM(revenue) OVER () AS total_revenue, - SUM(revenue) OVER (PARTITION BY city) AS city_revenue - FROM rides - ORDER BY city_revenue DESC; -~~~ - -Its operation can be described as follows (numbered steps listed here correspond to the numbers in the diagram below): - -1. The outer `SELECT DISTINCT(city) ... FROM rides` creates a "virtual table" on which the window functions will operate. -2. The window function `SUM(revenue) OVER ()` operates on a window frame containing all rows of the query output. -3. The window function `SUM(revenue) OVER (PARTITION BY city)` operates on several window frames in turn; each frame contains the `revenue` columns for a different city [partition](partitioning.html) (Amsterdam, Boston, L.A., etc.). - -Window function diagram - -### Caveats - -The most important part of working with window functions is understanding what data will be in the frame that the window function will be operating on. By default, the window frame includes all of the rows of the partition. If you order the partition, the default frame includes all rows from the first row in the partition to the current row. In other words, adding an `ORDER BY` clause when you create the window frame (e.g., `PARTITION BY x ORDER by y`) has the following effects: - -- It makes the rows inside the window frame ordered. -- It changes what rows the function is called on - no longer all of the rows in the window frame, but a subset between the "first" row and the current row. - -Another way of saying this is that you can run a window function on either: - -- All rows in the window frame created by the `PARTITION BY` clause, e.g., `SELECT f(x) OVER () FROM z`. -- A subset of the rows in the window frame if the frame is created with `SELECT f(x) OVER (PARTITION BY x ORDER BY y) FROM z`. - -Because of this, you should be aware of the behavior of any [aggregate function][aggregate-functions] you use as a [window function][window-functions]. If you are not seeing results you expect from a window function, this behavior may explain why. You may need to specify the frame boundaries explicitly in [the window definition](#window-definitions). - -{{site.data.alerts.callout_success}} -If you are running separate window functions over the same window frame, you can define the window frame once in a `WINDOW` clause, and then refer to the window by its name when you call the window function. For an example, see [Customers taking the most rides and generating the most revenue](#customers-taking-the-most-rides-and-generating-the-most-revenue). -{{site.data.alerts.end}} - -## Examples - -{% include {{page.version.version}}/sql/movr-statements-geo-partitioned-replicas.md %} - -### Customers taking the most rides - -To see which customers have taken the most rides, run: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM - (SELECT distinct(name) as "name", - COUNT(*) OVER (PARTITION BY name) AS "number of rides" - FROM users JOIN rides ON users.id = rides.rider_id) - ORDER BY "number of rides" DESC LIMIT 10; -~~~ - -~~~ - name | number of rides -+------------------+-----------------+ - Michael Brown | 19 - Richard Bullock | 17 - Judy White | 15 - Patricia Herrera | 15 - Tony Ortiz | 15 - Maria Weber | 15 - Cindy Medina | 14 - Samantha Coffey | 13 - Nicole Mcmahon | 13 - Amy Cobb | 13 -(10 rows) -~~~ - -### Customers generating the most revenue - -To see which customers have generated the most revenue, run: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT DISTINCT name, - SUM(revenue) OVER (PARTITION BY name) AS "total rider revenue" - FROM users JOIN rides ON users.id = rides.rider_id - ORDER BY "total rider revenue" DESC - LIMIT 10; -~~~ - -~~~ - name | total rider revenue -+------------------+---------------------+ - Richard Bullock | 952.00 - Patricia Herrera | 948.00 - Maria Weber | 903.00 - Michael Brown | 858.00 - Judy White | 818.00 - Tyler Dalton | 786.00 - Samantha Coffey | 758.00 - Cindy Medina | 740.00 - Tony Ortiz | 724.00 - Nicole Mcmahon | 696.00 -(10 rows) -~~~ - -### Add row numbers to query output - -To add row numbers to the output, kick the previous query down into a subquery and run the `row_number()` window function. - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT row_number() OVER (), * - FROM ( - SELECT DISTINCT - name, - sum(revenue) OVER ( - PARTITION BY name - ) AS "total rider revenue" - FROM users JOIN rides ON users.id = rides.rider_id - ORDER BY "total rider revenue" DESC - LIMIT 10 - ); -~~~ - -~~~ - row_number | name | total rider revenue -+------------+------------------+---------------------+ - 1 | Richard Bullock | 952.00 - 2 | Patricia Herrera | 948.00 - 3 | Maria Weber | 903.00 - 4 | Michael Brown | 858.00 - 5 | Judy White | 818.00 - 6 | Tyler Dalton | 786.00 - 7 | Samantha Coffey | 758.00 - 8 | Cindy Medina | 740.00 - 9 | Tony Ortiz | 724.00 - 10 | Nicole Mcmahon | 696.00 -(10 rows) -~~~ - -### Customers taking the most rides and generating the most revenue - -To see which customers have taken the most rides while generating the most revenue, run: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM ( - SELECT DISTINCT name, - COUNT(*) OVER w AS "number of rides", - (SUM(revenue) OVER w)::DECIMAL(100,2) AS "total rider revenue" - FROM users JOIN rides ON users.ID = rides.rider_id - WINDOW w AS (PARTITION BY name) - ) - ORDER BY "number of rides" DESC, - "total rider revenue" DESC - LIMIT 10; -~~~ - -~~~ - name | number of rides | total rider revenue -+------------------+-----------------+---------------------+ - Michael Brown | 19 | 858.00 - Richard Bullock | 17 | 952.00 - Patricia Herrera | 15 | 948.00 - Maria Weber | 15 | 903.00 - Judy White | 15 | 818.00 - Tony Ortiz | 15 | 724.00 - Cindy Medina | 14 | 740.00 - Tyler Dalton | 13 | 786.00 - Samantha Coffey | 13 | 758.00 - Nicole Mcmahon | 13 | 696.00 -(10 rows) -~~~ - -Note that in the query above, a `WINDOW` clause defines the window frame, and the `OVER` clauses reuse the window frame. - -### Customers with the highest average revenue per ride - -To see which customers have the highest average revenue per ride, run: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT DISTINCT name, - COUNT(*) OVER w AS "number of rides", - AVG(revenue) OVER w AS "average revenue per ride" - FROM users JOIN rides ON users.ID = rides.rider_id - WINDOW w AS (PARTITION BY name) - ORDER BY "average revenue per ride" DESC, "number of rides" ASC - LIMIT 10; -~~~ - -~~~ - name | number of rides | average revenue per ride -+---------------------+-----------------+--------------------------+ - Daniel Hernandez MD | 7 | 69.714285714285714286 - Pamela Morse | 5 | 68.40 - Mark Adams | 7 | 66.571428571428571429 - Sarah Wang DDS | 8 | 63.375 - Patricia Herrera | 15 | 63.20 - Taylor Cunningham | 10 | 62.60 - Tyler Dalton | 13 | 60.461538461538461538 - Maria Weber | 15 | 60.20 - James Hamilton | 8 | 60.00 - Deborah Carson | 10 | 59.70 -(10 rows) -~~~ - -### Customers with the highest average revenue per ride, given ten or more rides - -To see which customers have the highest average revenue per ride, given that they have taken at least 10 rides, run: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT * FROM ( - SELECT DISTINCT name, - COUNT(*) OVER w AS "number of rides", - (AVG(revenue) OVER w)::DECIMAL(100,2) AS "average revenue per ride" - FROM users JOIN rides ON users.ID = rides.rider_id - WINDOW w AS (PARTITION BY name) - ) - WHERE "number of rides" >= 10 - ORDER BY "average revenue per ride" DESC - LIMIT 10; -~~~ - -~~~ - name | number of rides | average revenue per ride -+-------------------+-----------------+--------------------------+ - Patricia Herrera | 15 | 63.20 - Taylor Cunningham | 10 | 62.60 - Tyler Dalton | 13 | 60.46 - Maria Weber | 15 | 60.20 - Deborah Carson | 10 | 59.70 - Carl Russell | 11 | 58.36 - Samantha Coffey | 13 | 58.31 - Matthew Clay | 11 | 56.09 - Richard Bullock | 17 | 56.00 - Ryan Hickman | 10 | 55.70 -(10 rows) -~~~ - -### Total number of riders, and total revenue - -To find out the total number of riders and total revenue generated thus far by the app, run: - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT - COUNT("id") AS "total # of riders", - SUM("total rider revenue") AS "total revenue" FROM ( - SELECT DISTINCT users.id, - SUM(revenue) OVER (PARTITION BY users.id) AS "total rider revenue" - FROM users JOIN rides ON users.id = rides.rider_id - ORDER BY "total rider revenue" DESC); -~~~ - -~~~ - total # of riders | total revenue ---------------------+---------------- - 50 | 25628.00 -(1 row) -~~~ - -### How many vehicles of each type - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT DISTINCT type, COUNT(*) OVER (PARTITION BY type) AS cnt FROM vehicles ORDER BY cnt DESC; -~~~ - -~~~ - type | cnt -+------------+-----+ - scooter | 7 - skateboard | 6 - bike | 2 -(3 rows) -~~~ - -### How much revenue per city - -{% include_cached copy-clipboard.html %} -~~~ sql -> SELECT DISTINCT(city), SUM(revenue) OVER (PARTITION BY city) AS city_revenue FROM rides ORDER BY city_revenue DESC; -~~~ - -~~~ - city | city_revenue -+---------------+--------------+ - boston | 3019.00 - amsterdam | 2966.00 - new york | 2923.00 - san francisco | 2857.00 - paris | 2849.00 - washington dc | 2797.00 - seattle | 2792.00 - los angeles | 2772.00 - rome | 2653.00 -(9 rows) -~~~ - -## See also - -- [Simple `SELECT` clause][simple-select] -- [Selection Queries][selection-query] -- [Aggregate functions][aggregate-functions] -- [Window Functions][window-functions] -- [CockroachDB 2.0 Demo][demo] - - - -[aggregate-functions]: functions-and-operators.html#aggregate-functions -[demo]: https://www.youtube.com/watch?v=v2QK5VgLx6E -[simple-select]: select-clause.html -[selection-query]: selection-queries.html -[window-functions]: functions-and-operators.html#window-functions diff --git a/src/current/v21.2/interleave-in-parent.md b/src/current/v21.2/interleave-in-parent.md index eb8bf2d4af8..4983f0858be 100644 --- a/src/current/v21.2/interleave-in-parent.md +++ b/src/current/v21.2/interleave-in-parent.md @@ -9,5 +9,5 @@ docs_area: {{site.data.alerts.callout_danger}} `INTERLEAVE IN PARENT` was permanently removed from CockroachDB in v21.2. -For details, see the [v21.1 interleaving deprecation notice](../v21.1/interleave-in-parent.html#deprecation). +For details about the deprecation process, see the archived documentation. {{site.data.alerts.end}} diff --git a/src/current/v21.2/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md b/src/current/v21.2/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md index 55b09d040b9..694fff8fff5 100644 --- a/src/current/v21.2/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md +++ b/src/current/v21.2/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md @@ -1052,7 +1052,7 @@ The upgrade process on Kubernetes is a [staged update](https://kubernetes.io/doc Therefore, in order to upgrade to v21.2, you must be on a production release of v21.1. - 1. If you are upgrading to v21.2 from a production release earlier than v21.1, or from a testing release (alpha/beta), first [upgrade to a production release of v21.1](../v21.1/orchestrate-cockroachdb-with-kubernetes-multi-cluster.html#upgrade-the-cluster). Be sure to complete all the steps. + 1. If you are upgrading to v21.2 from a production release earlier than v21.1, or from a testing release (alpha/beta), first upgrade to a production release of v21.1. Be sure to complete all the steps. 1. Then return to this page and perform a second upgrade to v21.2. diff --git a/src/current/v21.2/upgrade-cockroach-version.md b/src/current/v21.2/upgrade-cockroach-version.md index 28a8320789f..af81082931e 100644 --- a/src/current/v21.2/upgrade-cockroach-version.md +++ b/src/current/v21.2/upgrade-cockroach-version.md @@ -53,9 +53,9 @@ If you are running any other version, take the following steps **before** contin | Version | Action(s) before upgrading to any {{ page.version.version }} release | |------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Pre-{{ page.version.version }} testing release | Upgrade to a corresponding production release; then upgrade through each subsequent major release, [ending with a {{ previous_version }} production release](../{{ previous_version }}/upgrade-cockroach-version.html). | -| Pre-{{ previous_version }} production release | Upgrade through each subsequent major release, [ending with a {{ previous_version }} production release](../{{ previous_version }}/upgrade-cockroach-version.html). | -| {{ previous_version}} testing release | [Upgrade to a {{ previous_version }} production release](../{{ previous_version }}/upgrade-cockroach-version.html). | +| Pre-{{ page.version.version }} testing release | Upgrade to a corresponding production release; then upgrade through each subsequent major release, ending with a v21.1 production release. | +| Pre-v21.1 production release | Upgrade through each subsequent major release, ending with a v21.1 production release. | +| v21.1 testing release | Upgrade to a v21.1 production release. | When you are ready to upgrade to {{ latest.release_name }}, continue to [step 2](#step-2-prepare-to-upgrade). @@ -103,7 +103,7 @@ If your cluster contains partially-decommissioned nodes, they will block an upgr Review the [changes in v21.2](../releases/v21.2.html#v21-2-0). If any affect your deployment, make the necessary changes before starting the rolling upgrade to v21.2. -- Interleaved tables and interleaved indexes have been removed. Before upgrading to v21.2, [convert interleaved tables](../v21.1/interleave-in-parent.html#convert-interleaved-tables) and [replace interleaved indexes](../v21.1/interleave-in-parent.html#replace-interleaved-indexes). Clusters with interleaved tables and indexes cannot finalize the v21.2 upgrade. +- Interleaved tables and interleaved indexes have been removed. Before upgrading to v21.2, convert interleaved tables and replace interleaved indexes. Clusters with interleaved tables and indexes cannot finalize the v21.2 upgrade. - Previously, CockroachDB only supported the YMD format for parsing timestamps from strings. It now also supports the MDY format to better align with PostgreSQL. A timestamp such as `1-1-18`, which was previously interpreted as `2001-01-18`, will now be interpreted as `2018-01-01`. To continue interpreting the timestamp in the YMD format, the first number can be represented with 4 digits, `2001-1-18`. - The deprecated [cluster setting](cluster-settings.html) `cloudstorage.gs.default.key` has been removed, and the behavior of the `AUTH` parameter in Google Cloud Storage [`BACKUP`](backup.html) and `IMPORT` URIs has been changed. The default behavior is now that of `AUTH=specified`, which uses the credentials passed in the `CREDENTIALS` parameter, and the previous default behavior of using the node's implicit access (via its machine account or role) now requires explicitly passing `AUTH=implicit`. - We have switched types from `TEXT` to `"char"` for compatibility with PostgreSQL in the following columns: `pg_constraint` (`confdeltype`, `confmatchtype`, `confudptype`, `contype`) `pg_operator` (`oprkind`), `pg_prog` (`proargmodes`), `pg_rewrite` (`ev_enabled`, `ev_type`), and `pg_trigger` (`tgenabled`). @@ -116,7 +116,7 @@ This step is relevant only when upgrading from v21.1.x to v21.2. For upgrades wi By default, after all nodes are running the new version, the upgrade process will be **auto-finalized**. This will enable certain [features and performance improvements introduced in v21.2](#features-that-require-upgrade-finalization). However, it will no longer be possible to perform a downgrade to v21.1. In the event of a catastrophic failure or corruption, the only option will be to start a new cluster using the old binary and then restore from one of the backups created prior to performing the upgrade. For this reason, **we recommend disabling auto-finalization** so you can monitor the stability and performance of the upgraded cluster before finalizing the upgrade, but note that you will need to follow all of the subsequent directions, including the manual finalization in [step 5](#step-5-finish-the-upgrade): -1. [Upgrade to v21.1](../v21.1/upgrade-cockroach-version.html), if you haven't already. +1. Upgrade to v21.1, if you haven't already. 1. Start the [`cockroach sql`](cockroach-sql.html) shell against any node in the cluster. diff --git a/src/current/v22.1/upgrade-cockroachdb-kubernetes.md b/src/current/v22.1/upgrade-cockroachdb-kubernetes.md index 33df63f9029..79c8b17048f 100644 --- a/src/current/v22.1/upgrade-cockroachdb-kubernetes.md +++ b/src/current/v22.1/upgrade-cockroachdb-kubernetes.md @@ -36,7 +36,7 @@ If you [deployed CockroachDB on Red Hat OpenShift](deploy-cockroachdb-with-kuber Therefore, in order to upgrade to {{ page.version.version }}, you must be on a production release of {{ previous_version }}. - 1. If you are upgrading to {{ page.version.version }} from a production release earlier than {{ previous_version }}, or from a testing release (alpha/beta), first [upgrade to a production release of {{ previous_version }}](../v21.1/operate-cockroachdb-kubernetes.html#upgrade-the-cluster). Be sure to complete all the steps. + 1. If you are upgrading to {{ page.version.version }} from a production release earlier than {{ previous_version }}, or from a testing release (alpha/beta), first upgrade to a production release of {{ previous_version }}. Be sure to complete all the steps. 1. Then return to this page and perform a second upgrade to {{ page.version.version }}. diff --git a/src/current/v22.2/upgrade-cockroachdb-kubernetes.md b/src/current/v22.2/upgrade-cockroachdb-kubernetes.md index 21acc955168..f5c5ef96ba8 100644 --- a/src/current/v22.2/upgrade-cockroachdb-kubernetes.md +++ b/src/current/v22.2/upgrade-cockroachdb-kubernetes.md @@ -36,7 +36,7 @@ If you [deployed CockroachDB on Red Hat OpenShift](deploy-cockroachdb-with-kuber Therefore, in order to upgrade to {{ page.version.version }}, you must be on a production release of {{ previous_version }}. - 1. If you are upgrading to {{ page.version.version }} from a production release earlier than {{ previous_version }}, or from a testing release (alpha/beta), first [upgrade to a production release of {{ previous_version }}](../v21.1/operate-cockroachdb-kubernetes.html#upgrade-the-cluster). Be sure to complete all the steps. + 1. If you are upgrading to {{ page.version.version }} from a production release earlier than {{ previous_version }}, or from a testing release (alpha/beta), first upgrade to a production release of {{ previous_version }}. Be sure to complete all the steps. 1. Then return to this page and perform a second upgrade to {{ page.version.version }}.