|
| 1 | +--- |
| 2 | +title: "September Bulletin #2" |
| 3 | +date: "2025-10-13" |
| 4 | +authors: ["Community Team"] |
| 5 | +excerpt: "Overview of all work happening in Vortex" |
| 6 | +published: true |
| 7 | +--- |
| 8 | + |
| 9 | +> Editor's note: Its been pointed out to us that while this post is published in October, it covers work done in September, so this will be the second September Bulletin. |
| 10 | +
|
| 11 | +This is the second monthly issue of all that has been happening in Vortex! |
| 12 | + |
| 13 | +Development keeps moving quickly, we released [0.54.0](https://github.com/vortex-data/vortex/releases/tag/0.54.0) with many new features and improvements to the Rust API. This month we accepted 227 commits from 14 different contributors, with the relentless renovate-bot taking the crown for most changes merged. |
| 14 | + |
| 15 | +## Core |
| 16 | + |
| 17 | +1. Published an [RFC](https://github.com/vortex-data/vortex/pull/4726) presenting our planned changes to how arrays and compute interact with each other, introducing what we call Operators. |
| 18 | +1. Started work on GPU-powered kernels. |
| 19 | +1. Merged a few big changes to the IO APIs ([#4557](https://github.com/vortex-data/vortex/pull/4608), [#4608](https://github.com/vortex-data/vortex/pull/4608)), which introduce new push-based write API (in addition to the existing stream-based one), and writes now return both the file's footer (which can now be cached, reused and even de/serialized [#4598](https://github.com/vortex-data/vortex/pull/4608)), and the file's total compressed size. |
| 20 | +1. Added a new encoding - [fastlanes](https://github.com/spiraldb/fastlanes)-based RLE [#4588](https://github.com/vortex-data/vortex/pull/4588), [#4789](https://github.com/vortex-data/vortex/pull/4789). |
| 21 | +1. The work on the new `FixedSizeList` keeps moving forward - [#4590](https://github.com/vortex-data/vortex/pull/4590), [#4601](https://github.com/vortex-data/vortex/pull/4601). |
| 22 | +1. Added a new canonical encoding for list types, see the [tracking issue](https://github.com/vortex-data/vortex/issues/4699) for full details and work. |
| 23 | +1. Fixed a long-standing issue where we used to underestimate the uncompressed size of arrays, causing inaccurate array stats and a consistent underestimation of the overall data size. [#4963](https://github.com/vortex-data/vortex/pull/4693). |
| 24 | +1. Improved performance for very wide tables (1K+ columns), improving a new compression [benchmark](https://github.com/vortex-data/vortex/pull/4867) by over 80%. [#4863](https://github.com/vortex-data/vortex/pull/4863) [#4868](https://github.com/vortex-data/vortex/pull/4868) [#4877](https://github.com/vortex-data/vortex/pull/4877) |
| 25 | +1. Allow users to write with different compression strategies from Python [#4825](https://github.com/vortex-data/vortex/pull/4825) |
| 26 | + |
| 27 | +## Integrations |
| 28 | + |
| 29 | +### DuckDB |
| 30 | + |
| 31 | +1. Landed a release of the Vortex extension for the most recent DuckDB release (1.4). As always, you can install and load the extension with: |
| 32 | + |
| 33 | + ```sql |
| 34 | + INSTALL vortex FROM community; |
| 35 | + LOAD vortex; |
| 36 | + ``` |
| 37 | + |
| 38 | +1. Improved the testing of our DuckDB extension, making it easier and faster to run tests, using a debug build of DuckDB. |
| 39 | +1. Added zero-copy exporting of arrays to DuckDB [#4812](https://github.com/vortex-data/vortex/pull/4812) [#4804](https://github.com/vortex-data/vortex/pull/4804) |
| 40 | + |
| 41 | +### Apache DataFusion |
| 42 | + |
| 43 | +1. Use the built-in `FilePruner` to prune file based on the full expression, even for expressions we can't push down yet like dynamic expressions. |
| 44 | +1. Support for tables with hive-style partitioning. |
| 45 | +1. Updated our Apache DataFusion integration to the most recent release (v50.1.0) [#4577](https://github.com/vortex-data/vortex/pull/4577) |
| 46 | + |
| 47 | +## Acknowledgments |
| 48 | + |
| 49 | +We want to thank to anyone who has tried Vortex, provided feedback, asked question and filed issues. |
| 50 | + |
| 51 | +Special thanks go for all the contributors who took the time and care to contribute to Vortex this month (in descending count of commits): |
| 52 | + |
| 53 | +```text |
| 54 | + 41 Adam Gutglick |
| 55 | + 32 Connor Tsui |
| 56 | + 22 Joe Isaacs |
| 57 | + 21 Robert Kruszewski |
| 58 | + 21 Alexander Droste |
| 59 | + 20 Onur Satici |
| 60 | + 20 Nicholas Gates |
| 61 | + 10 Dan King |
| 62 | + 4 Dmitrii Blaginin |
| 63 | + 2 Andrew Duffy |
| 64 | + 1 Will Manning |
| 65 | + 1 Maksim Dergousov |
| 66 | + 1 Evan Martin |
| 67 | + 1 Alfonso Subiotto Marqués |
| 68 | +``` |
0 commit comments