Dataflow Templates 2025-12-02-00_RC00

Note: This release is being rolled out. It may not be in your region yet.

What's Changed

Improvements

• Fix JSON/JSONB datatypes in postgres-to-sql template by @yair-harel in #2960
• Add IcebergResourceManager by @tarun-google in #2985
• Add support for additional fields in terraform templates for Reverse and Live migration by @aasthabharill in #3005
• [DatastreamToSpanner] Integration test for CDC processing algorithm for Live migration template by @darshan-sj in #3018

Bug fixes

• Remove preemptive cleanup by @rahulkumar0630 in #2981
• chore: Added latency metrics to Write to source DoFn by @pratickchokhani in #3009
• [spanner-to-sourcedb] Custom Transformation fails when SQL PK col is dropped in Spanner by @aasthabharill in #3003
• [spanner-to-sourcedb] DELETEs: Error fetching shard id column by @aasthabharill in #3024
• Fix dependency conflict on beam-sdks-java-core in Datastream to Firestore by @baeminbo in #3026

New Contributors

• @tarun-google made their first contribution in #2985
• @nmemond made their first contribution in #2992
• @aasthabharill made their first contribution in #3005
• @pratickchokhani made their first contribution in #3009
• @baeminbo made their first contribution in #3026

Full Changelog: 2025-11-11-00_RC00...2025-12-02-00_RC00

Dataflow Templates 2025-11-25-00_RC00

[spanner-to-sourcedb] Custom Transformation fails when SQL PK col is …

Dataflow Templates 2025-11-18-00_RC00

[tests] Add integration tests for IDENTITY columns to bulk, live, and…

Dataflow Templates 2025-11-11-00_RC00

What's Changed

Package Upgrades

• chore(deps): update Python dependencies by @github-actions[bot] in #2979

Full Changelog: 2025-11-06-00_RC00...2025-11-11-00_RC00

Dataflow Templates 2025-11-06-00_RC00

What's Changed

Improvements

• Delete v2/datastream-to-postgres directory by @dhercher in #2959
• [SpannerToSourceDb] Improving error classification in Reverse replication template by @darshan-sj in #2964
• Update to beam 2.69 by @Abacn in #2966

Bug fixes

• fixing retry count in reverse replication by @shreyakhajanchi in #2949

Package Upgrades

• chore(deps): update Python dependencies by @github-actions[bot] in #2956

Full Changelog: 2025-10-28-01_RC00...2025-11-06-00_RC00

Dataflow Templates 2025-11-05-00_RC00

fix(build): update surefire report processing to handle time format (…

Dataflow Templates 2025-10-28-01_RC00

What's Changed

Improvements

• fix(kafka): handle domain-scoped project IDs in bootstrap server URL by @liferoad in #2908
• Yaml: Add yaml syntax UT by @derrickaw in #2872
• [SpannerToSourceDb] Fixing CDC events processing logic for Reverse replication template by @darshan-sj in #2919
• Apply GCP Application Default authentication when using a Managed Service for Apache Kafka cluster. by @fozzie15 in #2925

Bug fixes

• Remove special handling for only PK column tables by @manitgupta in #2929
• [yaml] - move syntax test verification to validate section by @derrickaw in #2936

Package Upgrades

• chore(deps): update Python dependencies by @github-actions[bot] in #2899
• build: update jetty dependencies to 12.0.12 to fix CVE-2024-6763 by @stankiewicz in #2921
• build: update org.apache.commons:commons-configuration to resolve CVE-2024-29131 by @stankiewicz in #2922

New Contributors

• @LucaDai made their first contribution in #2898

Full Changelog: 2025-10-21-00_RC00...2025-10-28-01_RC00

Dataflow Templates 2025-10-28-00_RC00

bump org.apache.commons:commons-configuration to resolve CVE-2024-291…

Dataflow Templates 2025-10-21-00_RC00

What's Changed

Improvements

• KafkaToBigQueryYaml - change hidden to false by @derrickaw in #2879
• Add option to unwrap messages replayed from DLQ to avoid nesting loops. by @claudevdm in #2888

Bug fixes

• fix dependabot labeling workflow name by @derrickaw in #2886
• fix label dependabot workflow by @derrickaw in #2890

Full Changelog: 2025-10-14-00_RC00...2025-10-21-00_RC00

Dataflow Templates 2025-10-14-00_RC00

What's Changed

Improvements

• feat(kafka-to-bigquery): Fix dynamic destination connection churn by @anpag in #2873

Bug fixes

• changed mysql event ordering in datastream to spanner by @bharadwaj-aditya in #2867

New Contributors

• @tianz101 made their first contribution in #2871
• @anpag made their first contribution in #2873

Full Changelog: 2025-10-07-00_RC00...2025-10-14-00_RC00