Skip to content

Conversation

@yew1eb
Copy link
Contributor

@yew1eb yew1eb commented Dec 5, 2025

Which issue does this PR close?

Closes #1697 .

Rationale for this change

Cache NativePlan nodes after conversion check and reuse in doExecuteNative

What changes are included in this PR?

  • Cache serialized NativePlan nodes as lazy vals right after the native support check.
  • Reuse the cached nodes in doExecuteNative when creating NativeRDD to avoid recomputation.

Are there any user-facing changes?

No.

How was this patch tested?

@github-actions github-actions bot added the spark label Dec 5, 2025
@yew1eb yew1eb force-pushed the lazy_cache_nativeplan branch from 7ed3566 to 1d906c9 Compare December 5, 2025 01:57
Copy link
Contributor

@ShreyeshArangath ShreyeshArangath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change LGTM, but are we seeing any benefits coming from the lazy-caching? Curious if there is a real-world use-case that drove this change.

@yew1eb
Copy link
Contributor Author

yew1eb commented Dec 8, 2025

Change LGTM, but are we seeing any benefits coming from the lazy-caching? Curious if there is a real-world use-case that drove this change.

This change is a minor optimization to avoid unnecessary repeated conversions of Spark Plan nodes (e.g., expressions) into Native Plan nodes (serialized into protobuf format).

@richox
Copy link
Contributor

richox commented Dec 11, 2025

this native conversions are designed to be eagerly computed in native plan construction, so we can fallback the plan when any conversion fails.

Copy link
Contributor

@richox richox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this native conversions are designed to be eagerly computed in native plan construction, so we can fallback the plan when any conversion fails.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Lazy-cache NativePlan nodes after native conversion check to avoid recomputation

3 participants