Skip to content

Commit e18674d

Browse files
shahmishalshahmishal
authored
Merge pull request #298 from compnerd/dev
Windows: add an image for building the toolchain
2 parents a7a878a + 3536ad1 commit e18674d

File tree

5 files changed

+208
-0
lines changed

5 files changed

+208
-0
lines changed

‎swift-ci/README.md‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,113 @@ docker run \
3535
/bin/bash -lc "cp -r /source/* /home/build-user/; ./swift/utils/build-script --preset buildbot_linux install_destdir=/home/build-user/swift-install installable_package=/home/build-user/swift-DEVELOPMENT-SNAPSHOT-$(date +'%F')-a.tar.gz"
3636
```
3737

38+
## Quick Start for Windows Development
39+
40+
The Windows Docker image will setup an enviornment with Python, Visual Studio
41+
Build Tools, and Git. It is setup to assume that the sources will be available
42+
in `S:\SourceCache`.
43+
44+
### Building and Tagging the image
45+
46+
```powershell
47+
cd master\windows\10.0.19044.1706
48+
docker image build --compress -t swift:swiftci .
49+
```
50+
51+
### Running the image
52+
53+
```powershell
54+
docker run --rm -it -v %UserProfile%\data:S: swift:swiftci
55+
```
56+
57+
### Building the Toolchain
58+
59+
While we can build the toolchain in the containerized environment, the sources
60+
are expected to reside on the host and is passed into the docker container as a
61+
volume. The rest of automation expects that the Sources reside under a
62+
directory with the name `SourceCache`.
63+
64+
#### Clone the Sources
65+
66+
```cmd
67+
md %UserProfile%\data\SourceCache
68+
cd %UserProfile%\data\SourceCache
69+
70+
git clone -b stable/20220426 -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/llvm-project
71+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift
72+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-cmark cmark
73+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-experimental-string-processing
74+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-corelibs-libdispatch
75+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-corelibs-foundation
76+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-corelibs-xctest
77+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-argument-parser
78+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-crypto
79+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-driver
80+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-llbuild llbuild
81+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-package-manager
82+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-system
83+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-tools-support-core
84+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/swift-installer-scripts
85+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/indexstore-db
86+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/apple/sourcekit-lsp
87+
git clone -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/jpsim/Yams
88+
git clone -t curl-7_77_0 -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/curl/curl
89+
git clone -t v2.9.12 -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/gnome/libxml2
90+
git clone -t v1.2.11 -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/madler/zlib
91+
git clone -b maint/maint-69 -c core.autocrlf=input -c core.symlink=true -c core.useBuiltinFSMonitor=false https://github.com/unicode-org/icu
92+
```
93+
94+
#### Run Docker
95+
96+
```cmd
97+
docker run --rm -it -v %UserProfile%\data:S: swift:swiftci
98+
```
99+
100+
#### Build the Toolchain
101+
102+
This will build a full Swift toolchain distribution (llvm, clang, lld, lldb,
103+
swift, swift-package-manger, SourceKit-LSP) and the Windows SDK (x86, x64,
104+
ARM64).
105+
106+
```cmd
107+
S:
108+
S:\SourceCache\swift\utils\build.cmd
109+
```
110+
111+
#### Running Swift Tests
112+
113+
The toolchain tests require some modifications to the path to find some of the
114+
dependencies. The following will run the Swift test suite within the docker
115+
container:
116+
117+
```cmd
118+
path S:\b\1\bin;S:\b\1\tools\swift\libdispatch-windows-x86_64-prefix\bin;%Path%;%ProgramFiles%\Git\usr\bin
119+
ninja -C S:\b\1 check-swift
120+
```
121+
122+
#### Using the Toolchain
123+
124+
> **NOTE**: Running the test suite and using the toolchain near the production mode are mututally incompatible (due to the path changes).
125+
126+
The build will generate a toolchain image that is roughly similar to the
127+
installed version. The following can be run inside the docker container to use
128+
the toolchain:
129+
130+
```cmd
131+
set SDKROOT=S:\Library\Developer\Platforms\Windows.platform\Developer\SDKs\Windows.sdk
132+
path S:\Library\Developer\Platforms\Windows.platform\Developer\SDKs\Windows.sdk\usr\bin\x64;S:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin;%Path%
133+
```
134+
135+
Because the toolchain is built in the volume which is backed by the host, the
136+
toolchain can be used on the host (assuming the dependencies such as Visual
137+
Studio is installed and the module modules deployed). The adjusted paths below
138+
should enable that:
139+
140+
```cmd
141+
set SDKROOT=%UserProfile%\data\Library\Developer\Platforms\Windows.platform\Developer\SDKs\Windows.sdk
142+
path %UserProfile%\data\Library\Developer\Platforms\Windows.platform\Developer\SDKs\Windows.sdk\usr\bin\x64;%UserProfile%\data\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin;%Path%
143+
```
144+
38145
## Contributions
39146

40147
Contributions via pull requests are welcome and encouraged :)

‎swift-ci/master/windows/10.0.19042.1645/Dockerfile‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# escape=`
2+
3+
FROM mcr.microsoft.com/windows/servercore:10.0.19042.1645 AS windows
4+
5+
LABEL maintainer="Swift Infrastructure <swift-infrastructure@forums.swift.org>"
6+
LABEL description="Docker Container for the Swift programming language"
7+
8+
ARG GIT=https://github.com/git-for-windows/git/releases/download/v2.36.0.windows.1/Git-2.36.0-64-bit.exe
9+
ARG PYTHON=https://www.python.org/ftp/python/3.10.4/python-3.10.4-amd64.exe
10+
11+
# restore the default Windows shell for correct batch processing
12+
SHELL ["cmd", "/S", "/C"]
13+
14+
# Enable Developer Mode.
15+
RUN reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\AppModelUnlock" /t REG_DWORD /f /v "AllowDevelopmentWithoutDevLicense" /d "1"
16+
17+
# Enable Long Paths
18+
RUN reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem" /t REG_DWORD /f /v "LongPathsEnabled" /d "1"
19+
20+
# Install Git.
21+
# See: git-[version]-[bit].exe /SAVEINF=git.inf and /?
22+
COPY git.inf .
23+
RUN `
24+
curl -SLo git.exe %GIT% `
25+
&& (start /w git.exe /SP- /VERYSILENT /SUPPRESSMSGBOXES /NOCANCEL /NORESTART /CLOSEAPPLICATIONS /FORCECLOSEAPPLICATIONS /LOADINF=git.inf ) `
26+
&& del /q git.exe git.inf
27+
28+
# Install Python.
29+
# See: https://docs.python.org/3.10/using/windows.html
30+
# FIXME: it appears that `PYTHONHOME` and `PYTHONPATH` are unset
31+
COPY unattend.xml .
32+
RUN `
33+
curl -SLo python.exe %PYTHON% `
34+
&& (start /w python.exe /quiet ) `
35+
&& del /q python.exe unattend.xml
36+
37+
# Install Visual Studio Build Tools
38+
RUN `
39+
curl -SLo vs_buildtools.exe https://aka.ms/vs/17/release/vs_buildtools.exe `
40+
&& (start /w vs_buildtools.exe --quiet --wait --norestart --nocache `
41+
--add Microsoft.VisualStudio.Component.Windows11SDK.22000 `
42+
--add Microsoft.VisualStudio.Component.VC.ATL `
43+
--add Microsoft.VisualStudio.Component.VC.ATL.ARM `
44+
--add Microsoft.VisualStudio.Component.VC.ATL.ARM64 `
45+
--add Microsoft.VisualStudio.Component.VC.CMake.Project `
46+
--add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 `
47+
--add Microsoft.VisualStudio.Component.VC.Tools.ARM64 `
48+
--add Microsoft.VisualStudio.Component.VC.Tools.ARM `
49+
|| IF "%EXITCODE%"=="3010" EXIT 0) `
50+
&& del /q vs_buildtools.exe
51+
COPY InstallModules.cmd .
52+
RUN InstallModules.cmd && del /q InstallModules.cmd
53+
54+
# FIXME: we should use a non-Administrator user
55+
# USER ContainerUser
56+
57+
ENV PYTHONUTF8=1
58+
# Default to powershell
59+
CMD ["powershell.exe", "-nologo", "-ExecutionPolicy", "Bypass"]

‎swift-ci/master/windows/10.0.19042.1645/InstallModules.cmd‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
@echo off
2+
setlocal
3+
set vswhere=%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe
4+
FOR /F "tokens=* usebackq" %%r IN (`"%vswhere%" -nologo -latest -all -prerelease -products * -property installationPath`) DO SET VsDevCmd=%%r\Common7\Tools\VsDevCmd.bat
5+
CALL "%VsDevCmd%" -no_logo -host_arch=amd64 -arch=amd64
6+
mklink "%UniversalCRTSdkDir%\Include\%UCRTVersion%\ucrt\module.modulemap" S:\SourceCache\swift\stdlib\public\Platform\ucrt.modulemap
7+
mklink "%UniversalCRTSdkDir%\Include\%UCRTVersion%\um\module.modulemap" S:\SourceCache\swift\stdlib\public\Platform\winsdk.modulemap
8+
mklink "%VCToolsInstallDir%\include\module.modulemap" S:\SourceCache\swift\stdlib\public\Platform\visualc.modulemap
9+
mklink "%VCToolsInstallDir%\include\visualc.apinotes" S:\SourceCache\swift\stdlib\public\Platform\visualc.apinotes
10+
endlocal
11+
@echo on

‎swift-ci/master/windows/10.0.19042.1645/git.inf‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
[Setup]
2+
NoIcons=1
3+
Components=gitlfs
4+
EditorOption=VIM
5+
PathOption=Cmd
6+
SSHOption=OpenSSH
7+
TurtoiseOption=false
8+
CURLOption=WinSSL
9+
BashTerminalOption=ConHost
10+
PerformanceTweaksFSCache=Enabled
11+
EnableSymlinks=Enabled
12+
EnablePseudoConsoltSupport=Disabled
13+
EnableFSMonitor=Enabled

‎swift-ci/master/windows/10.0.19042.1645/unattend.xml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
<?xml version="1.0" encoding="utf-8"?>
2+
<Options>
3+
<Option Name="InstallAllUsers" Value="1" />
4+
<Option Name="AssociateFiles" Value="0" />
5+
<Option Name="PrependPath" Value="1" />
6+
<Option Name="Shortcuts" Value="0" />
7+
<Option Name="Include_doc" Value="0" />
8+
<Option Name="Include_debug" Value="0" />
9+
<Option Name="Include_dev" Value="1" />
10+
<Option Name="Include_exe" Value="1" />
11+
<Option Name="Include_launcher" Value="0" />
12+
<Option Name="InstallLauncherAllUsers" Value="0" />
13+
<Option Name="Include_lib" Value="1" />
14+
<Option Name="Include_symbols" Value="0" />
15+
<Option Name="Include_tcltk" Value="0" />
16+
<Option Name="Include_test" Value="0" />
17+
<Option Name="Include_tools" Value="0" />
18+
</Options>

0 commit comments

Comments
 (0)