Skip to content

Conversation

@cxzl25
Copy link
Contributor

@cxzl25 cxzl25 commented Dec 12, 2025

Which issue does this PR close?

Closes #1716

Rationale for this change

set spark.auron.enable=true;
select cast(array(c1) as string ) from test_t1 ;
[]

In Spark3.1 and above, it should be [null]

What changes are included in this PR?

Are there any user-facing changes?

How was this patch tested?

Add UT

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a bug in array-to-string casting where null values within arrays were not being properly rendered. Previously, cast(array(c1) as string) would return [] for an array containing a null element, but in Spark 3.1+, it should return [null]. The implementation adds a custom Spark-compatible cast operation for List to Utf8 types.

Key Changes:

  • Added custom array-to-string cast implementation in Rust that properly handles null elements
  • Added comprehensive test coverage with 5 integration tests and 5 unit tests
  • Implementation follows existing Spark-compatible patterns for complex type casting

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
native-engine/datafusion-ext-commons/src/arrow/cast.rs Implements array-to-string casting with proper null handling, mirroring the existing struct-to-string pattern. Includes 5 unit tests covering edge cases (nulls, empty arrays, nested arrays, string arrays).
spark-extension-shims-spark/src/test/scala/org.apache.auron/AuronQuerySuite.scala Adds 5 integration tests to verify Spark compatibility for various array casting scenarios (basic arrays, nested arrays, arrays with nulls, string arrays, empty arrays).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

string cast array missing null value

1 participant