diff --git a/meetings/notes/2025-04-22.rst b/meetings/notes/2025-04-22.rst new file mode 100644 index 0000000..1b60a1b --- /dev/null +++ b/meetings/notes/2025-04-22.rst @@ -0,0 +1,119 @@ +=========================== + Open Source WG: 04/22/2025 +=========================== + +Recording: A recording of the meeting is available in the Linux Foundation https://openprofile.dev/ profile. If you are +a member of the Working Group you can access this through your account. + +Attendees +========= + +* Deepika H V - Centre for Development of Advanced Computing + +* Rod Burns - Codeplay +* Kumudha Narasimhan - Codeplay +* Aaron Dron - Codeplay + +* Ragesh Hajela - Fujitsu + +* Kevan Ahmadi - Imagination Technologies +* Ed Tuke - Imagination Technologies + +* Maria Kraynyuk - Intel +* Vadim Pirogov - Intel +* Maria Petrova - Intel +* Vasudha Badri-Paul - Intel +* Alison Richards - Intel +* Maria Garzaran - Intel +* Timmie Smith - Intel +* John Melonakos - Intel +* Nikolay Petrov - Intel +* Konstantin Boyarinov - Intel +* Andrey Fedorov - Intel +* Mike Voss - Intel + +* Melissa Aranzamendez - The Linux Foundation + +Next Steps +========== + +* All projects to review and update their project CI documentation in the open source working group repo. +* Aaron and team to continue working on documentation for CI infrastructure. +* Rod and John to have a separate meeting about expanding the UXL Foundation GitHub organization's README and structure. +* All projects to check and ensure documentation links are easily findable on their project pages. +* Alison to collect and review marketing materials from all projects to identify gaps and plan next steps. +* All project members to mention UXL Foundation and oneAPI in conference talks and tag them in social media posts. +* John to arrange for Vasanth to present his CPU inclusivity proposal at the next meeting. +* Rod to ensure relevant member organization representatives are present for Vasanth's presentation on CPU inclusivity. + +Security Work Package Breakdown Discussion (`slides`_) +========================================== + +Aaron discussed the security work package breakdown, noting that most projects still have items in the in progress +column. He asked for feedback on any issues with Coverity or OSS-Fuzz, and mentioned that the oneAPI Construction Kit +and scikit-learn-intelex projects were behind on their work packages. Mike expressed that his team was not prioritizing +fuzz testing due to the high effort required. Aaron suggested that the security work packages could be closed if they +are not relevant, but also noted that some issues might not come up again. He proposed using the OpenSSF Scorecard as a +guidance for future directions and suggested that the security contacts and other stakeholders should regularly review +and discuss it. + +UXL Project Dashboard and Maintainer Selection (`slides`_) +============================================== + +Aaron provides an update on several topics related to the UXL project. He discusses a dashboard implemented by an +engineer to show GitHub organization scores, and recommends running scorecards within projects rather than relying on +external workflows. Aaron raises the question of establishing a default process for maintainer selection and suggests +using UXL membership to create a basic chain of trust. He also proposes creating a master list of GitHub accounts and +their organizational affiliations to improve user management. Regarding CI, Aaron reports that ARM runners are working +well, but Intel GPU Max runners are currently unstable due to a location change. He mentions that there may be some +downtime for these runners until the migration is complete. + +Standardizing Project Infrastructure and Documentation (`slides`_) +====================================================== + +Aaron discussed the need for standardization in project infrastructure and the importance of documenting testing +processes. He mentioned the development of two new runner runners and the need for more documentation. Rod suggested +reviewing the information in the Github repo to ensure it's up-to-date and accurate. Aaron also discussed the +possibility of using Conda Forge for binary releases and the need for a standard UXL release process. Timmie asked about +the level of detail required in the CI infrastructure documentation, and Aaron responded that while specific hardware +details may not be necessary, software versions and compatibility should be included. + +UXL Roadmap Progress and Future Plans +===================================== + +John discussed the progress of the UXL roadmap, noting that three projects had responded, while one project was not at +1.0 yet and lacked a defined roadmap. John also mentioned that some projects had shared their feedback in chat +messages. Maria clarified that they had listed milestones on Github but not specific issues to be addressed. Mike added +that they were starting to apply milestones to their RFCs, issues, and PRs. John suggested a separate meeting to discuss +the UXL Foundation Github organization and how to expand the roadmap. Rod agreed to this. John also mentioned the +potential merger of two working groups (Spec and Open-source). + +Intel Project Documentation and Marketing Strategy +================================================== + +In the meeting, the team discussed the organization of their projects, specifically focusing on the Intel projects that +have been moved to the UXL Foundation. They discussed the need for a unified way to direct users to the documentation +for each project. Rod suggested a review of each project to ensure that documentation is easily accessible. The team +also discussed the creation of a 'get started' page to link everything together. Alison proposed the creation of a +presentation or video to provide a quick understanding of each project, its value proposition, and how to get +started. The team agreed to collect all marketing materials for the projects in the next month to identify gaps and plan +for the future. + +Expanding CPU Support in UXL +============================ + +John presented an idea to improve CPU inclusivity in the UXL libraries, which was originally proposed by Vasanth to +Qualcomm and ARM. The current UXL specifications focus on accelerator offloading, but John suggests expanding support +for CPUs to address the fragmented CPU landscape with its many competing interfaces. Vadim expresses uncertainty about +the claim of CPU landscape fragmentation, prompting John to clarify. + +John proposed a strategy to add an abstraction layer that insulates the hardware community from rapid software +innovation. This layer would allow for years of stable API while including CPU backends as well as accelerator +backends. The proposal was to create a C API specification in the policy layer, which would decide whether to use the +C++ API or a plugin API. The idea was to support CPUs in a structured way in their libraries. However, the team was +unsure about the problem statement and the benefits of this proposal. + +John agreed to bring in Vasanth to explain the proposal in more detail. + + +.. _`slides`: ../presentations/2025-03-25-UXLCIPoC.pdf diff --git a/meetings/notes/README.rst b/meetings/notes/README.rst index 7895476..3359be1 100644 --- a/meetings/notes/README.rst +++ b/meetings/notes/README.rst @@ -2,6 +2,7 @@ Meeting Notes =============== +* `2025-04-22 <2025-04-22.rst>`__ * `2025-03-25 <2025-03-25.rst>`__ * `2025-02-25 <2025-02-25.rst>`__ * `2025-01-28 <2025-01-28.rst>`__ @@ -16,4 +17,3 @@ * `2024-03-26 <2024-03-26.rst>`__ * `2024-02-27 <2024-02-27.rst>`__ * `2024-01-24 <2024-01-24.rst>`__ - diff --git a/meetings/presentations/2025-04-22-UXLCIPoC.pdf b/meetings/presentations/2025-04-22-UXLCIPoC.pdf new file mode 100644 index 0000000..83b83d0 Binary files /dev/null and b/meetings/presentations/2025-04-22-UXLCIPoC.pdf differ diff --git a/project-infrastructure/project-ci-documentation.md b/project-infrastructure/project-ci-documentation.md index cbb4feb..9fd7820 100644 --- a/project-infrastructure/project-ci-documentation.md +++ b/project-infrastructure/project-ci-documentation.md @@ -78,11 +78,22 @@ Representative: Timmie Smith Support contact for CI: +| Maintainers | +| ------------------------- | +| Dan Hoeflinger @danhoeflinger | +| Dmitriy Sobolev @dmitriy-sobolev | +| Timmie Smith @timmiesmith | + *Existing public CI* -| Owner | Type | OS | Number | Active? | How to access logs | -| --- | --- | --- | --- | --- | --- | -| ? | ? | ? | ? | ? | ? | +The current CI infrastructure is setup for per-commit testing. It is run automatically on PRs in the oneDPL repository +when a new commit is made to the source branch of the PR. + +| Owner | Type | OS | How to access logs | +| --- | --- | --- | --- | +| GitHub | CPU x86 | Ubuntu | Via CI Testing Workflow view | +| GitHub | CPU x86 | Windows | Via CI Testing Workflow view | +| GitHub | AArch64 | Mac OS | Via CI Testing Workflow view | *Required Public CI Infrastruture Needed To Confidently Accept Contributions* @@ -90,36 +101,124 @@ Support contact for CI: | --- | --- | --- | --- | | x86 | Intel | CPU | Ubuntu | | AArch64 | Arm | CPU | Ubuntu | +| Xe, Xe2, Xe3 | Intel | GPU | Ubuntu, Windows | +| | NVIDIA | GPU | Ubuntu, Windows | +| | AMD | GPU | Ubuntu, Windows | + +There are no special paths for particular architectures for AMD and NVIDIA GPUs in oneDPL at this point. It is +sufficient for correctness to run functional testing on one GPU from a vendor. More information on the supported +platforms can be found in the links below. +* NVIDIA: https://developer.codeplay.com/products/oneapi/nvidia/latest/guides/get-started-guide-nvidia#supported-platforms +* AMD: https://developer.codeplay.com/products/oneapi/amd/latest/guides/get-started-guide-amd#supported-platforms + +oneDPL testing must cover all C++ standard execution policies as well as oneDPL device policies. + +* CPU: ``seq``, ``unseq``, ``par_unseq``, ``device_policy`` +* GPU: ``device_policy`` + +Testing on CPU platforms must exercise ``par`` and ``par_unseq`` execution policies with OpenMP and oneTBB to cover all +of the oneDPL backends. + +Minimum Software Versions: +| Software | Windows | Linux | MacOS (Arm CPU testing) | +| --- | --- | --- | --- | +| OpenMP | any | any | any | +| oneTBB | 2022.0 | 2022.0 | 2022.0 | +| CMake | 3.20 | 3.11 | 3.11 | +| git | any | any | any | +| python | any | any | any | +| DPC++ Compiler | 2024.2 | 2024.2 | - | +| clang++ compiler | 16 | 16 | 16 | +| GCC compiler | 10 | 10 | 10 | +| ninja | any | any | any | +| Microsoft Visual Studio* | 2022 | - | - | +| Intel General-Purpose GPU driver (for Intel HW testing) | 2423.32 (Rolling) and 2350.61 (LTS) | 2423.32 (Rolling) and 2350.61 (LTS) | - | -Software Versions: -* CMake -* glibc -* ... oneDAL ------ -Representative: Nikolay Petrov +Representative: [Nikolay Petrov](https://github.com/napetrov) -Support contact for CI: +Support contacts for CI: -*Existing public CI* +| Area | Owner | +| ------------------ | --------------------------------------------------------------------------------------------------- | +| AArch64 | [Hamza Butt](https://github.com/theComputeKid) [Rakshith G B](https://github.com/rakshithgb-fujitsu)| +| RISC-V | [Keeran Rothenfusser](https://github.com/keeranroth) | +| x64, new platforms | [Nikolay Petrov](https://github.com/napetrov) | -| Owner | Type | OS | Number | Active? | How to access logs | -| --- | --- | --- | --- | --- | --- | -| ? | ? | ? | ? | ? | ? | +More details available in [MAINTAINERS.md](https://github.com/uxlfoundation/oneDAL/blob/main/MAINTAINERS.md) -*Required Public CI Infrastruture Needed To Confidently Accept Contributions* +### *Existing public CI* + +oneDAL + +| Platform | Type | OS | Number | Active? | Comments | +| --- | --- | --- | --- | --- | --- | +| Github | CPU AArch64 | Linux | 2 | Yes | 2 Pipelines on Physical Arm systems Github hosted and Arm-Hosted. [ci-aarch64.yml](https://github.com/uxlfoundation/oneDAL/blob/main/.github/workflows/ci-aarch64.yml) | +| Github | Intel GPU (ICX compiler) | Linux | 1 | Yes* | Pipeline configured for GPU validation on uxlfoundation GPU runners, currently disabled due to issues with Tiber Cloud. [ci.yml](https://github.com/uxlfoundation/oneDAL/blob/main/.github/workflows/ci.yml) | +| Github | ABI conformance | Linux | 1 | Yes | ABI compatibility runs that compare PR to the main. [ci.yml](https://github.com/uxlfoundation/oneDAL/blob/main/.github/workflows/ci.yml) | +| Github | Docker validation | Linux | 1 | Yes | Validation of oneDAL development env docker file. [docker-validation-ci.yml](https://github.com/uxlfoundation/oneDAL/blob/main/.github/workflows/docker-validation-ci.yml) | +| Github | CPU Nightly | Linux, Windows | 2 | Yes | Nightly builds and broader validation for oneDAL. [nightly-build.yml](https://github.com/uxlfoundation/oneDAL/blob/main/.github/workflows/nightly-build.yml) | +| Github | Copyright headers check | Linux | 1 | Yes | Check for proper copyright headers. [skywalking-eyes.yml](https://github.com/uxlfoundation/oneDAL/blob/main/.github/workflows/skywalking-eyes.yml) | +| Github | PR checklist validation | Linux | 1 | Yes | Validation of PR conformance. [pr-checklist.yml](https://github.com/uxlfoundation/oneDAL/blob/main/.github/workflows/pr-checklist.yml) | +| Github | Documentation deployment to gh-pages | Linux | 1 | Yes | Automatic docs deployment with release tag creation. [docs-release.yml](https://github.com/uxlfoundation/oneDAL/blob/main/.github/workflows/docs-release.yml) | +| Mergify | Helper automation for merges/backporting | Linux | 1 | Yes | Automated labels assignment, removal of renovate branches. [.mergify.yml](https://github.com/uxlfoundation/oneDAL/blob/main/.github/.mergify.yml) | +| Renovate | Automated dependency updates | Linux | 1 | Yes | Dependabot alternative, manages dependency updates for components. [renovate.json](https://github.com/uxlfoundation/oneDAL/blob/main/.github/renovate.json) | +| Azure DevOps | Clang format check| Linux | 1 | Yes | Enforce coding standards. [ci.yml](https://github.com/uxlfoundation/oneDAL/blob/main/.ci/pipeline/ci.yml) | +| Azure DevOps | CI build/test for x86/ARM/RISC-V with OS compilers | Linux, Windows | 8 | Yes | Intel build natively, ARM and RISC-V with cross-platform build and QEMU emulation. Build with GCC/VC compilers. [ci.yml](https://github.com/uxlfoundation/oneDAL/blob/main/.ci/pipeline/ci.yml) | +| Azure DevOps | CI build/test with Bazel | Linux | 1 | Yes | Bazel-based build and validation. [ci.yml](https://github.com/uxlfoundation/oneDAL/blob/main/.ci/pipeline/ci.yml) | +| Azure DevOps | oneDAL documentation build | Linux | 1 | Yes | Build documentation for validation purposes in CI. [docs.yml](https://github.com/uxlfoundation/oneDAL/blob/main/.ci/pipeline/docs.yml) | +| Azure DevOps | sklearnex validation | Linux | 1 | Yes | Checking out sklearnex sources, doing build and validation with oneDAL PR changes. [ci.yml](https://github.com/uxlfoundation/oneDAL/blob/main/.ci/pipeline/ci.yml) | +| Codefactor | Codefactor checks | N/A | 1 | Yes | Enforcing code checks in PRs, Bandit, and other code quality checks. [Codefactor](https://www.codefactor.io/repository/github/uxlfoundation/onedal) | + +scikit-learn-intelex + +| Platform | Type | OS | Number | Active? | Comments | +| --- | --- | --- | --- | --- | --- | +| Mergify | Helper automation for merges/backporting | Linux | 1 | Yes | Automated labels assignment, removal of renovate branches. [.mergify.yml](https://github.com/uxlfoundation/scikit-learn-intelex/blob/main/.github/.mergify.yml) | +| Renovate | Automated dependency updates | Linux | 1 | Yes | Dependabot alternative, manages dependency updates for components. [renovate.json](https://github.com/uxlfoundation/scikit-learn-intelex/blob/main/.github/renovate.json) | +| Azure DevOps | CI conda based | Linux, Windows | 10 | Yes | CI build and testing for different scikit/python combinations [ci.yml](https://github.com/uxlfoundation/scikit-learn-intelex/blob/main/.ci/pipeline/ci.yml) | +| Azure DevOps | Documentation validation | Linux | 10| Yes | Documentation build validation [docs.yml](https://github.com/uxlfoundation/scikit-learn-intelex/blob/main/.ci/pipeline/docs.yml) | +| Azure DevOps | Linting | Linux | 1 | Yes | Linting enforcement through pre-commit [linting.yml](https://github.com/uxlfoundation/scikit-learn-intelex/blob/main/.ci/pipeline/linting.yml) | +| Azure DevOps | Nightly | Linux | 1 | Yes | Nightly validation against scikit-learn main branch [nightly.yml](https://github.com/uxlfoundation/scikit-learn-intelex/blob/main/.ci/pipeline/nightly.yml) | +| Azure DevOps | Coverity | Linux | 1 | Yes | [Coverity](https://scan.coverity.com/projects/daal4py) scans [nightly.yml](https://github.com/uxlfoundation/scikit-learn-intelex/blob/main/.ci/pipeline/nightly.yml) | +| Azure DevOps | Releases validation | Linux, Windows | 12 | Yes | Validation of already released versions in pypi and conda-forge [ci.yml](https://github.com/uxlfoundation/scikit-learn-intelex/blob/main/.ci/pipeline/release.yml) | +| Github | CI venv based | Linux, Windows | 6 | Yes | CI build and testing for different scikit/python combinations. [ci.yml](https://github.com/uxlfoundation/scikit-learn-intelex/blob/main/.github/workflows/ci.yml) | +| Github | Copyright headers check | Linux | 1 | Yes | Check for proper copyright headers. [skywalking-eyes.yml](https://github.com/uxlfoundation/scikit-learn-intelex/blob/main/.github/workflows/skywalking-eyes.yml) | +| Github | PR checklist validation | Linux | 1 | Yes | Validation of PR conformance. [pr-checklist.yml](https://github.com/uxlfoundation/scikit-learn-intelex/blob/main/.github/workflows/pr-checklist.yml) | +| Github | Documentation deployment to gh-pages | Linux | 1 | Yes | Automatic docs deployment with release tag creation. [docs-release.yml](https://github.com/uxlfoundation/scikit-learn-intelex/blob/main/.github/workflows/docs-release.yml) | +| Codefactor | Codefactor checks | N/A | 1 | Yes | Enforcing code checks in PRs, Bandit, and other code quality checks. [Codefactor](https://www.codefactor.io/repository/github/uxlfoundation/scikit-learn-intelex) | +| Codecov | Codecoverage | N/A | 1 | Yes | Enforcing coverage tracking and increments in PRs. [Codecov](https://app.codecov.io/gh/uxlfoundation/scikit-learn-intelex) | + + +### *Required Public CI Infrastructure Needed To Confidently Accept Contributions* + +Currently internal Intel CI validation is required for code verification. +Intel employees can start internal CI with comment "/intelci: run" and corresponding CI would be triggered. +It's not possible to view these logs without access to Intel network + +Migration to public validation is possible but will require more x86 systems available in public - currently public validation covers a small subset of validation scopes that we are planning to expand | Instruction set architecture | Hardware Vendor | Processor Type | Operating System | | --- | --- | --- | --- | -| x86 | Intel | CPU | Ubuntu | +| x86 | Intel | CPU | Ubuntu, Windows | | AArch64 | Arm | CPU | Ubuntu | +| Xe, Xe2, Xe3 | Intel | GPU | Ubuntu, Windows | Software Versions: -* CMake -* glibc -* ... +* C/C++ Compiler +* DPC++ Compiler and oneMKL if building with SYCL support +* BLAS and LAPACK libraries - both provided by oneMKL or openBLAS could be used +* Python version 3.9 or higher +* oneTBB library (repository contains script to download it) +* oneDPL library if building with SYCL support +* Microsoft Visual Studio* (Windows* only) +* MSYS2 (Windows* only) +* make and dos2unix tools +Full list of SW requirements and steps defined in [INSTALL.md](https://github.com/uxlfoundation/oneDAL/blob/main/INSTALL.md) + oneCCL ------ @@ -149,27 +248,29 @@ Software Versions: oneMath ------- -Representative: Maria Kraynyuk +Representative: [Maria Kraynyuk](https://github.com/mkrainiuk) -Support contact for CI: +Support contact for CI: [Alexey Srednitsky](https://github.com/toxicscum) *Existing public CI* | Owner | Type | OS | Number | Active? | How to access logs | | --- | --- | --- | --- | --- | --- | -| ? | ? | ? | ? | ? | ? | +| GitHub | CPU x86 | Ubuntu latest | N/A - GitHub-hosted runners | Yes | From workflow run | -*Required Public CI Infrastruture Needed To Confidently Accept Contributions* +*Required Public CI Infrastructure Needed To Confidently Accept Contributions* -| Instruction set architecture | Hardware Vendor | Processor Type | Operating System | -| --- | --- | --- | --- | -| x86 | Intel | CPU | Ubuntu | -| AArch64 | Arm | CPU | Ubuntu | +| Instruction set architecture | Hardware Vendor | Processor Type | Operating System | Comment | +| --- | --- | --- | --- | --- | +| x86 | Intel/AMD | CPU | Ubuntu | Already supported in public CI on x64 VM | +| AArch64 | Arm | CPU | Ubuntu | Arm Neoverse Processor Family: N1, V1, or V2 | +| Intel Data Center Max Series | Intel | GPU | Ubuntu | Or at least one from [Intel oneMKL supported list](https://www.intel.com/content/www/us/en/developer/articles/system-requirements/oneapi-math-kernel-library-system-requirements.html) on Linux | +| A100 or H100 | NVIDIA | GPU | Ubuntu | Or at least Compute Capability 7.5 or later (T4+), see [CUDA toolkit deprecated GPUs](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#deprecated-architectures) | +| MI210 | AMD | GPU | Ubuntu | Or at least one from [ROCm supported list](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html) | +| x86 | Intel/AMD | CPU | Windows | Can be supported with GitHub-hosted runners, but they have not enough processors for acceptable build time | +| Intel Flex or Arch Series | Intel | GPU | Windows | Or at least one from [Intel oneMKL supported list](https://www.intel.com/content/www/us/en/developer/articles/system-requirements/oneapi-math-kernel-library-system-requirements.html) on Windows | -Software Versions: -* CMake -* glibc -* ... +Software requirements: [link](https://github.com/uxlfoundation/oneMath/tree/develop?tab=readme-ov-file#software-requirements) oneTBB ------