[AURON #2017] [BUILD] Add Spark 4.x support to dev/reformat script.#2018
[AURON #2017] [BUILD] Add Spark 4.x support to dev/reformat script.#2018slfan1989 wants to merge 3 commits intoapache:masterfrom
Conversation
There was a problem hiding this comment.
Pull request overview
This pull request adds support for Spark 4.0 and 4.1 to the dev/reformat script, enabling automated code formatting and style checks for the new Spark versions. The script is updated to automatically switch between JDK and Scala versions based on the Spark version being processed, as Spark 4.x requires JDK 17+ and Scala 2.13, while Spark 3.x uses JDK 8 and Scala 2.12.
Changes:
- Fixed Maven profile references for Flink and Iceberg to use correct versioned profile IDs
- Added Spark 4.0 and 4.1 to the version sweep array in the reformat script
- Implemented automatic JDK and Scala version switching logic for Spark 4.x vs 3.x
- Updated CI workflow to set up both JDK 8 and JDK 17
- Applied formatting changes to Scala files (indentation adjustments from running the reformatter)
Reviewed changes
Copilot reviewed 3 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| dev/reformat | Core changes: fixed Flink/Iceberg profiles, added Spark 4.0/4.1 to version array, added JDK/Scala switching logic |
| .github/workflows/style.yml | Added JDK 17 setup to support Spark 4.x formatting in CI |
| thirdparty/auron-uniffle/src/main/scala/org/apache/spark/sql/execution/auron/shuffle/uniffle/AuronUniffleShuffleReader.scala | Indentation formatting adjustment |
| spark-extension/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeParquetInsertIntoHiveTableBase.scala | Indentation formatting adjustment |
| spark-extension/src/main/scala/org/apache/spark/sql/auron/util/TaskContextHelper.scala | Indentation formatting adjustment |
| spark-extension/src/main/scala/org/apache/spark/sql/auron/NativeHelper.scala | Indentation formatting adjustment |
| spark-extension/src/main/scala/org/apache/spark/sql/auron/AuronSparkSessionExtension.scala | Indentation formatting adjustment |
| spark-extension/src/main/scala/org/apache/spark/sql/auron/AuronConverters.scala | Indentation formatting adjustments |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@cxzl25 Thanks for reviewing this pr! I’ll continue refining this pr. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 8 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
slfan1989
force-pushed
the
auron-2017
branch
from
428854b to
5ac544f
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 8 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…ipt. Signed-off-by: slfan1989 <[email protected]>
slfan1989
force-pushed
the
auron-2017
branch
from
a096ee8 to
c099533
Compare
…ipt. Signed-off-by: slfan1989 <[email protected]>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 8 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <[email protected]>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 8 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # Run spotless with a consistent JDK/version to avoid formatting drift. | ||
| run_spotless scala-2.12 spark-3.5 | ||
| run_spotless scala-2.13 spark-4.0 | ||
|
|
||
| # Check or format all code, including third-party code, with spark-3.5 | ||
| set_java_home 8 | ||
| sparkver=spark-3.5 | ||
| for celebornver in celeborn-0.5 celeborn-0.6 | ||
| do | ||
| run_maven -P"${sparkver}" -Pceleborn,"${celebornver}" -Puniffle,uniffle-0.10 -Ppaimon,paimon-1.2 -Pflink,flink-1.18 -Piceberg,iceberg-1.9 | ||
| run_maven -P"${sparkver}" -Pceleborn,"${celebornver}" -Puniffle,uniffle-0.10 -Ppaimon,paimon-1.2 -Pflink-1.18 -Piceberg-1.9 | ||
|
|
There was a problem hiding this comment.
run_spotless only runs for scala-2.12/spark-3.5 and scala-2.13/spark-4.0 and does not activate the third-party module profiles (celeborn/uniffle/paimon/flink/iceberg). Since Spotless was removed from run_maven, formatting problems inside those third-party modules are no longer enforced by ./dev/reformat --check (and may be silently auto-applied during compile due to the lifecycle-bound Spotless execution). Consider running run_spotless with the same third-party profiles used later in the script, or reintroducing spotless:check for those third-party runs.
Which issue does this PR close?
Closes #2017
Rationale for this change
With Spark 4.0 and 4.1 support added to the project, the
dev/reformatscript needs to be updated to handle formatting and style checks for these new versions. Spark 4.x requires JDK 17+ and Scala 2.13, while Spark 3.x uses JDK 8 and Scala 2.12. The script should automatically switch between these environments.What changes are included in this PR?
1. Fix Flink Maven profile
2.Add Spark 4.x support
3.Update CI workflow (.github/workflows/style.yml)
Are there any user-facing changes?
No.
How was this patch tested?
Verified automatic JDK switching works for Spark 3.x (JDK 8) and Spark 4.x (JDK 17)