Upgrade 4.22

[meiwen333]

Hi,

I know this is not particular a bug. But in 4.22 there are quiete a few changes in the UBT which are not yet in FastBuild. Some classes have changed completely. I will have a look into the code but it seems quite a bit of work which I am not sure I can handle on my own. Anybody worked through it so far?

[meiwen333]

I got some stuff fixed. Like the path that needs to be converted to a string. But some of the classes in the buildpipline have changed completely. So the sorting of actions do not work anymore. Unfortunately I am not that familiar with Fastbuild to get my head around that any time soon. Is there a better documentation that could point me in the right direction?

[wout276]

Hi @meiwen333
I've stumbled upon the same issue as you. For the path, I simply changed action.CommandPath.Contains() to action.CommandPath.FullName.Contains().
As far as I'm aware there are no other useful docs for this but the idea is pretty simple.
Fastbuild distributes all actions but most actions require items produced by other actions as prerequisite. I'm looking for another way to get those ProducingActions as well.
The best source for me is the XGE executor (which is from Incredibuild) in UnrealBuildTool/Executors and the changes to their workflow between 4.21 and 4.22.

I'll let you know if I make any progress.

[MrPatry]

We got it to work with some small changes (one of which (the regex) is really hacky). You're welcome to try it. It worked for us but I can't guarantee it's 100% error-free

(Update 12:17PM: New version attached, some error happened when compiling c files in LiveCoding)

FASTBuild.zip

[meiwen333]

Hi @MrPatry ,

thanks for the update.
Did you also experience all the missing precrocessor macros?
Seems similar to this thread:
https://answers.unrealengine.com/questions/813258/updated-to-420-now-i-cant-build-error-c4668.html

Anyway thanks for posting the solution to the upgrade :)

Cheers Pascal

[MrPatry]

No we did not, sorry

[yinweihn]

We got it to work with some small changes (one of which (the regex) is really hacky). You're welcome to try it. It worked for us but I can't guarantee it's 100% error-free

(Update 12:17PM: New version attached, some error happened when compiling c files in LiveCoding)

FASTBuild.zip

Hi @MrPatry I use your code in vs2017 but error occured
error described that:
opening file 'C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Redist\MSVC\14.13.26020\x64\Microsoft.VC141.CRT\msvcp140.dll' in Compiler ToolManifest

[MrPatry]

Hi @MrPatry I use your code in vs2017 but error occured
error described that:
opening file 'C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Redist\MSVC\14.13.26020\x64\Microsoft.VC141.CRT\msvcp140.dll' in Compiler ToolManifest

Make sure you're using at least VS2017 15.9 and have .Net Framework 4.6.2 targetting pack installed

[meiwen333]

In my case I changed the path to the correct version:
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Redist\MSVC\14.16.27012\x64\Microsoft.VC141.CRT
It is in line 617-618 or 624-625

if (VCEnv.Compiler == WindowsCompiler.VisualStudio2015)
				{
					AddText(string.Format("\t\t'{0}/VC/redist/x64/Microsoft.VC{1}.CRT/msvcp{2}.dll'\n", VCInstallDir.ToString(), platformVersionNumber, platformVersionNumber));
					AddText(string.Format("\t\t'{0}/VC/redist/x64/Microsoft.VC{1}.CRT/vccorlib{2}.dll'\n", VCInstallDir.ToString(), platformVersionNumber, platformVersionNumber));
				}
				else
				{
					//VS 2017 is really confusing in terms of version numbers and paths so these values might need to be modified depending on what version of the tool chain you
					// chose to install.
					AddText(string.Format("\t\t'{0}/VC/Redist/MSVC/14.13.26020/x64/Microsoft.VC141.CRT/msvcp{1}.dll'\n", VCInstallDir.ToString(), platformVersionNumber));
					AddText(string.Format("\t\t'{0}/VC/Redist/MSVC/14.13.26020/x64/Microsoft.VC141.CRT/vccorlib{1}.dll'\n", VCInstallDir.ToString(), platformVersionNumber));
				}

Just check which version you have and adjust the path.

[meiwen333]

Anyboday experiencing these errors:
C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared\intsafe.h(66): error C4668: '__midl' is not defined as a preprocessor macro, replacing with '0' for '#if/#elif'

I just found this post:
https://answers.unrealengine.com/questions/856910/error-c4668-cplusplus-not-defined.html?sort=oldest

And it seems that the win sdk is the culprit. Which version are you using @MrPatry ?

[wout276]

Thx @MrPatry ! My own fix for the command-line parsing issue was a lot less clean/more hacky.. The other small changes you made were what was missing on my end. I'm just hitting one small issue:
just before the build completely finishes, this happens:

1>5> -Build: 351 ms	D:\GitHub\UE4.22\UnrealEngine\Engine\Binaries\Win64\ShaderCompileWorker-DerivedDataCache.dll
1>9> -Build: 1158 ms	D:\GitHub\UE4.22\UnrealEngine\Engine\Binaries\Win64\ShaderCompileWorker-RHI.dll
1>11>-Build: 381 ms	D:\GitHub\UE4.22\UnrealEngine\Engine\Binaries\Win64\ShaderCompileWorker-Json.dll
1>12>-Build: 638 ms	D:\GitHub\UE4.22\UnrealEngine\Engine\Binaries\Win64\ShaderCompileWorker-ApplicationCore.dll
1>8> -Build: 535 ms	D:\GitHub\UE4.22\UnrealEngine\Engine\Binaries\Win64\ShaderCompileWorker-Projects.dll
1>7> -Build: 451 ms	D:\GitHub\UE4.22\UnrealEngine\Engine\Binaries\Win64\ShaderCompileWorker-ShaderCompilerCommon.dll
1>10>-Build: 423 ms	D:\GitHub\UE4.22\UnrealEngine\Engine\Binaries\Win64\ShaderCompileWorker-InputCore.dll
1>1> -Build: 1494 ms	D:\GitHub\UE4.22\UnrealEngine\Engine\Binaries\Win64\ShaderCompileWorker-VulkanShaderFormat.dll
1>3> -Build: 3792 ms	D:\GitHub\UE4.22\UnrealEngine\Engine\Binaries\Win64\ShaderCompileWorker-Core.dll
1>2> -Build: 3429 ms	D:\GitHub\UE4.22\UnrealEngine\Engine\Binaries\Win64\ShaderCompileWorker-Slate.dll
1>4> -Build: 4265 ms	D:\GitHub\UE4.22\UnrealEngine\Engine\Binaries\Win64\ShaderCompileWorker-CoreUObject.dll
1>6> -Build: 113871 ms	D:\GitHub\UE4.22\UnrealEngine\Engine\Binaries\Win64\UE4Editor-Engine.dll
1>Alias: all
1>Failed due to missing file: D:\GitHub\UE4.22\UnrealEngine\Engine\Source\Action_1744
1>Saving DepGraph 'D:\GitHub\UE4.22\UnrealEngine\Engine\Intermediate\Build\fbuild.fdb'
1>Saving DepGraph Complete in 0.068s

I looked a further into this issue and saw that this actions are not explicitly created in the bff file, just it's command line command is in there:

// "D:\GitHub\UE4.22\UnrealEngine\Engine\Build\Windows\cl-filter\cl-filter.exe" D:\GitHub\UE4.22\UnrealEngine\Engine\Intermediate\Build\Win64\UE4Editor\Development\LiveCoding\xxhash.c.txt -- "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX64\x64\cl.exe"  @"D:\GitHub\UE4.22\UnrealEngine\Engine\Intermediate\Build\Win64\UE4Editor\Development\LiveCoding\xxhash.c.obj.response" /showIncludes
ObjectList('Action_1743')
{
	.Compiler = 'UE4Compiler' 
	.CompilerInputFiles = "D:\GitHub\UE4.22\UnrealEngine\Engine\Source\Developer\Windows\LiveCoding\Private\External\xxhash.c"
	.CompilerOutputPath = "D:\GitHub\UE4.22\UnrealEngine\Engine\Intermediate\Build\Win64\UE4Editor\Development\LiveCoding"
	.CompilerOptions = '/Zc:inline /nologo /Oi /c /Gw /Gy /Zm850 /wd4819 /D_CRT_STDIO_LEGACY_WIDE_SPECIFIERS=1 /D_SILENCE_STDEXT_HASH_DEPRECATION_WARNINGS=1 /D_WINDLL /D_DISABLE_EXTENDED_ALIGNED_STORAGE /source-charset:utf-8 /execution-charset:utf-8 /Ox /Ot /GF /Ob2 /errorReport:prompt /EHsc /Z7 /MD /bigobj /fp:fast /Zo /Zp8 /wd4463 /wd4838 /I . /I Developer\Windows\LiveCoding\Private /I Runtime /I Runtime\Core\Public /I ..\Intermediate\Build\Win64\UE4Editor\Inc\CoreUObject /I Runtime\CoreUObject\Public /I Developer /I Developer\Settings\Public /I ..\Intermediate\Build\Win64\UE4Editor\Inc\LiveCoding /I Developer\Windows /I Developer\Windows\LiveCoding\Public /I "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\INCLUDE" /I "C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" /I "C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt" /I "C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared" /I "C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um" /I "C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt" /DIS_PROGRAM=0 /DUE_EDITOR=1 /DENABLE_PGO_PROFILE=0 /DUSE_VORBIS_FOR_STREAMING=1 /DWITH_DEV_AUTOMATION_TESTS=1 /DWITH_PERF_AUTOMATION_TESTS=1 /DUNICODE /D_UNICODE /D__UNREAL__ /DIS_MONOLITHIC=0 /DWITH_ENGINE=1 /DWITH_UNREAL_DEVELOPER_TOOLS=1 /DWITH_APPLICATION_CORE=1 /DWITH_COREUOBJECT=1 /DUSE_STATS_WITHOUT_ENGINE=0 /DWITH_PLUGIN_SUPPORT=0 /DWITH_PERFCOUNTERS=1 /DUSE_LOGGING_IN_SHIPPING=0 /DWITH_LOGGING_TO_MEMORY=0 /DUSE_CACHE_FREED_OS_ALLOCS=1 /DUSE_CHECKS_IN_SHIPPING=0 /DWITH_EDITOR=1 /DWITH_SERVER_CODE=1 /DWITH_CEF3=1 /DWITH_LIVE_CODING=1 /DWITH_XGE_CONTROLLER=1 /DUBT_MODULE_MANIFEST=\"UE4Editor.modules\" /DUBT_MODULE_MANIFEST_DEBUGGAME=\"UE4Editor-Win64-DebugGame.modules\" /DUBT_COMPILED_PLATFORM=Win64 /DUBT_COMPILED_TARGET=Editor /DUE_APP_NAME=\"UE4Editor\" /DWIN32=1 /D_WIN32_WINNT=0x0601 /DWINVER=0x0601 /DPLATFORM_WINDOWS=1 /DNDEBUG=1 /DUE_BUILD_DEVELOPMENT=1 /DUE_IS_ENGINE_MODULE=1 /DUE_ENABLE_ICU=1 /DWITH_VS_PERF_PROFILER=0 /DWITH_DIRECTXMATH=0 /DWITH_MALLOC_STOMP=1 /DCORE_API=DLLIMPORT /DCOREUOBJECT_API=DLLIMPORT /DSETTINGS_API=DLLIMPORT /DLIVECODING_API=DLLEXPORT /TC /W0  /Fo"%2" "%1" '
	.CompilerOutputExtension = '.c.obj' 
}

// "C:\WINDOWS\system32\cmd.exe" /C "copy /Y "D:\GitHub\UE4.22\UnrealEngine\Engine\Binaries\ThirdParty\Windows\DirectX\x64\dxcompiler.dll" "D:\GitHub\UE4.22\UnrealEngine\Engine\Binaries\Win64\dxcompiler.dll" 1>nul"
// "C:\WINDOWS\system32\cmd.exe" /C "copy /Y "D:\GitHub\UE4.22\UnrealEngine\Engine\Binaries\ThirdParty\Windows\DirectX\x64\dxil.dll" "D:\GitHub\UE4.22\UnrealEngine\Engine\Binaries\Win64\dxil.dll" 1>nul"
// "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX64\x64\link.exe" @"D:\GitHub\UE4.22\UnrealEngine\Engine\Intermediate\Build\Win64\ShaderCompileWorker\Development\ShaderFormatD3D\ShaderCompileWorker-ShaderFormatD3D.dll.response"
Copy('Action_1746_dummy')
{ 
	.Source = 'D:\GitHub\UE4.22\UnrealEngine\Engine\Intermediate\Build\Win64\ShaderCompileWorker\Development\ShaderFormatD3D\ShaderCompileWorker-ShaderFormatD3D.dll.response' 
	.Dest = 'D:\GitHub\UE4.22\UnrealEngine\Engine\Intermediate\Build\Win64\ShaderCompileWorker\Development\ShaderFormatD3D\ShaderCompileWorker-ShaderFormatD3D.dll.response.dummy' 
	.PreBuildDependencies = {
		'Action_200', ;SharedPCH.Core.NoUndef.cpp
		'Action_1108', ;Module.ShaderFormatD3D.cpp
		'Action_278', ;ShaderCompileWorker-Core.lib
		'Action_534', ;ShaderCompileWorker-RenderCore.lib
		'Action_763', ;ShaderCompileWorker-ShaderPreprocessor.lib
		'Action_765', ;ShaderCompileWorker-ShaderCompilerCommon.lib
	} 
}

Executable('Action_1746')
{ 
	.Linker = 'C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX64\x64\link.exe' 
	.Libraries = 'Action_1746_dummy' 
	.LinkerOptions = '/TLBOUT:"%1" /Out:"%2" @"D:\GitHub\UE4.22\UnrealEngine\Engine\Intermediate\Build\Win64\ShaderCompileWorker\Development\ShaderFormatD3D\ShaderCompileWorker-ShaderFormatD3D.dll.response" ' 
	.LinkerOutput = 'D:\GitHub\UE4.22\UnrealEngine\Engine\Binaries\Win64\ShaderCompileWorker-ShaderFormatD3D.dll' 
}

This action, along with some other actions (like 1745 in my case - as you can also see in the snippet above) are missing as well. Checking the ActionTypes, all of them are BuildProject or WriteMetaData steps. This is as expected from CreateBffFile().
Since this function hasn't really changed in comparison to our working setups for earlier engine versions, I'm in the dark about why these actions have suddenly become a problem.

[MrPatry]

@wout276
Try with this new version, it should fix this problem

FASTBuild.zip

[wout276]

@MrPatry Thx again for the quick answer. I just figured it out myself and coded it slightly differently. Here's my solution - with what I had to change to make caching work as well.

Edit 17:20 fixed cashing issue:
FASTBuild.zip

[MrPatry]

And it seems that the win sdk is the culprit. Which version are you using @MrPatry ?

@meiwen333 I'm using 10.0.16299.0

[msolemajor]

hello @meiwen333 / @wout276
with all this setup do you have buld failure, that are complaining of missing Lib files... ?

11>UnrealBuildTool : error : Failed to produce item: ....\Unreal-Stable\Engine\Plugins\Tests\RuntimeTests\Intermediate\Build\Win64\UE4Editor\Development\RuntimeTests\UE4Editor-RuntimeTests.lib

those kind?

[meiwen333]

Hi,
@msolemajor
I didn't have these errors.

I got it up and running with a change that was suggested in this thread:
#29