llvm
/
torch-mlir
mirror of https://github.com/llvm/torch-mlir
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
torch-mlir/build_tools/python_deploy/build_windows_ci.sh

22 lines
595 B
Bash
Raw Normal View History

ci: enable ccache on Windows (#1548) This patch makes a few small, but key, changes to enable ccache on Windows. First, it replaces the hendrikmuhs/ccache-action action with command line invocations to the ccache binary, since the action has two bugs, one of which causes CI to refer to different ccache artifacts before versus after the build on Windows whereas the other bug can sometimes cause the action to incorrectly infer that the cache is empty. Second, this patch slightly alters the cache key, so that our old cache artifacts, which have grown too big, are eventually discarded in favor of the new, smaller cache artifacts. Along the way, this patch also keeps the RollPyTorch's cache artifact separate from the regular build's cache artifact so as to keep these artifacts small, and also because the RollPyTorch action is off the critical path for most contributors. Finally, this patch makes small changes to the CMake file so that on Windows, the ccache binary is added as a prefix, as recommended on the [ccache Wiki](https://github.com/ccache/ccache/wiki/MS-Visual-Studio).
2022-11-04 01:17:22 +08:00
#!/usr/bin/env bash
CI: prepare CI for ccache updates for MSVC/Windows (#2120) This patch, by itself, doesn't fix caching on Windows, but once a new release of ccache is available, caching for Windows builds should start working again (validated by building ccache from source and using it with LLVM builds). Ccache rejects caching when either the `/Zi` or `/ZI` flags are used during compilation on Windows, since these flags tell the compiler to embed debug information in a PDB file (separate from the object file produced by the compiler). In particular, our CI builds add the `/Zi` flag, making ccache mark these compiler invocations as uncacheable. But what caused our CI to add debug flags, especially when we specified `-DCMAKE_BUILD_TYPE=Release`? On Windows, unless we specify the `--config Release` flag during the CMake build step, CMake assumes a debug build. So all this while, we had been producing debug builds of torch-mlir for every PR! No doubt it took so long to build the Windows binaries. The reason for having to specify the configuration during the _build_ step (as opposed to the _configure_ step) of CMake on Windows is that CMake's Visual Studio generators will produce _both_ Release and Debug profiles during the CMake configure step (thus requiring a build-time value that tells CMake whether to build in Release or Debug mode). Luckily, on Linux and macOS, the `--config` flag seems to be simply ignored, instead of causing build errors. Strangely, based on cursory tests, it seems like on Windows we need to specify the Relase configuration as both `-DCMAKE_BUILD_TYPE=Release` as well as `--config Release`. Dropping either made my build switch to a Debug configuration. Additionally, there is a bug in ccache v4.8 (although this is addressed in trunk) that causes ccache to reject caching if the compiler invocation includes any flag that starts with `/Z`, including /`Zc`, which is added by LLVM's HandleLLVMOptions.cmake and which isn't related to debug info or PDB files. The next release of ccache should include the fix, which is to reject caching only for `/Zi` and `/ZI` flags and not all flags that start with `/Z`. As a side note, debugging this problem was possible because of ccache's log file, which is enabled by: `ccache --set-config="log_file=log.txt"`.
2023-05-13 01:45:01 +08:00
set -eo pipefail
ci: enable ccache on Windows (#1548) This patch makes a few small, but key, changes to enable ccache on Windows. First, it replaces the hendrikmuhs/ccache-action action with command line invocations to the ccache binary, since the action has two bugs, one of which causes CI to refer to different ccache artifacts before versus after the build on Windows whereas the other bug can sometimes cause the action to incorrectly infer that the cache is empty. Second, this patch slightly alters the cache key, so that our old cache artifacts, which have grown too big, are eventually discarded in favor of the new, smaller cache artifacts. Along the way, this patch also keeps the RollPyTorch's cache artifact separate from the regular build's cache artifact so as to keep these artifacts small, and also because the RollPyTorch action is off the critical path for most contributors. Finally, this patch makes small changes to the CMake file so that on Windows, the ccache binary is added as a prefix, as recommended on the [ccache Wiki](https://github.com/ccache/ccache/wiki/MS-Visual-Studio).
2022-11-04 01:17:22 +08:00
echo "Building torch-mlir"
cmake -GNinja -Bbuild \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER_LAUNCHER=ccache \
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
-DLLVM_ENABLE_PROJECTS=mlir \
-DLLVM_TARGETS_TO_BUILD=host \
-DMLIR_ENABLE_BINDINGS_PYTHON=ON \
-DPython3_FIND_VIRTUALENV=ONLY \
Re-organize project structure to separate PyTorch dependencies from core project. (#2542) This is a first step towards the structure we discussed here: https://gist.github.com/stellaraccident/931b068aaf7fa56f34069426740ebf20 There are two primary goals: 1. Separate the core project (C++ dialects and conversions) from the hard PyTorch dependencies. We move all such things into projects/pt1 as a starting point since they are presently entangled with PT1-era APIs. Additional work can be done to disentangle components from that (specifically LTC is identified as likely ultimately living in a `projects/ltc`). 2. Create space for native PyTorch2 Dynamo-based infra to be upstreamed without needing to co-exist with the original TorchScript path. Very little changes in this path with respect to build layering or options. These can be updated in a followup without commingling directory structure changes. This also takes steps toward a couple of other layering enhancements: * Removes the llvm-external-projects/torch-mlir-dialects sub-project, collapsing it into the main tree. * Audits and fixes up the core C++ build to account for issues found while moving things. This is just an opportunistic pass through but roughly ~halves the number of build actions for the project from the high 4000's to the low 2000's. It deviates from the discussed plan by having a `projects/` tree instead of `compat/`. As I was thinking about it, this will better accommodate the follow-on code movement. Once things are roughly in place and the CI passing, followups will focus on more in-situ fixes and cleanups.
2023-11-03 10:45:55 +08:00
-DLLVM_EXTERNAL_PROJECTS="torch-mlir" \
ci: enable ccache on Windows (#1548) This patch makes a few small, but key, changes to enable ccache on Windows. First, it replaces the hendrikmuhs/ccache-action action with command line invocations to the ccache binary, since the action has two bugs, one of which causes CI to refer to different ccache artifacts before versus after the build on Windows whereas the other bug can sometimes cause the action to incorrectly infer that the cache is empty. Second, this patch slightly alters the cache key, so that our old cache artifacts, which have grown too big, are eventually discarded in favor of the new, smaller cache artifacts. Along the way, this patch also keeps the RollPyTorch's cache artifact separate from the regular build's cache artifact so as to keep these artifacts small, and also because the RollPyTorch action is off the critical path for most contributors. Finally, this patch makes small changes to the CMake file so that on Windows, the ccache binary is added as a prefix, as recommended on the [ccache Wiki](https://github.com/ccache/ccache/wiki/MS-Visual-Studio).
2022-11-04 01:17:22 +08:00
-DLLVM_EXTERNAL_TORCH_MLIR_SOURCE_DIR="$PWD" \
-DPython3_EXECUTABLE="$(which python)" \
$GITHUB_WORKSPACE/externals/llvm-project/llvm
CI: prepare CI for ccache updates for MSVC/Windows (#2120) This patch, by itself, doesn't fix caching on Windows, but once a new release of ccache is available, caching for Windows builds should start working again (validated by building ccache from source and using it with LLVM builds). Ccache rejects caching when either the `/Zi` or `/ZI` flags are used during compilation on Windows, since these flags tell the compiler to embed debug information in a PDB file (separate from the object file produced by the compiler). In particular, our CI builds add the `/Zi` flag, making ccache mark these compiler invocations as uncacheable. But what caused our CI to add debug flags, especially when we specified `-DCMAKE_BUILD_TYPE=Release`? On Windows, unless we specify the `--config Release` flag during the CMake build step, CMake assumes a debug build. So all this while, we had been producing debug builds of torch-mlir for every PR! No doubt it took so long to build the Windows binaries. The reason for having to specify the configuration during the _build_ step (as opposed to the _configure_ step) of CMake on Windows is that CMake's Visual Studio generators will produce _both_ Release and Debug profiles during the CMake configure step (thus requiring a build-time value that tells CMake whether to build in Release or Debug mode). Luckily, on Linux and macOS, the `--config` flag seems to be simply ignored, instead of causing build errors. Strangely, based on cursory tests, it seems like on Windows we need to specify the Relase configuration as both `-DCMAKE_BUILD_TYPE=Release` as well as `--config Release`. Dropping either made my build switch to a Debug configuration. Additionally, there is a bug in ccache v4.8 (although this is addressed in trunk) that causes ccache to reject caching if the compiler invocation includes any flag that starts with `/Z`, including /`Zc`, which is added by LLVM's HandleLLVMOptions.cmake and which isn't related to debug info or PDB files. The next release of ccache should include the fix, which is to reject caching only for `/Zi` and `/ZI` flags and not all flags that start with `/Z`. As a side note, debugging this problem was possible because of ccache's log file, which is enabled by: `ccache --set-config="log_file=log.txt"`.
2023-05-13 01:45:01 +08:00
cmake --build build --config Release
ci: enable ccache on Windows (#1548) This patch makes a few small, but key, changes to enable ccache on Windows. First, it replaces the hendrikmuhs/ccache-action action with command line invocations to the ccache binary, since the action has two bugs, one of which causes CI to refer to different ccache artifacts before versus after the build on Windows whereas the other bug can sometimes cause the action to incorrectly infer that the cache is empty. Second, this patch slightly alters the cache key, so that our old cache artifacts, which have grown too big, are eventually discarded in favor of the new, smaller cache artifacts. Along the way, this patch also keeps the RollPyTorch's cache artifact separate from the regular build's cache artifact so as to keep these artifacts small, and also because the RollPyTorch action is off the critical path for most contributors. Finally, this patch makes small changes to the CMake file so that on Windows, the ccache binary is added as a prefix, as recommended on the [ccache Wiki](https://github.com/ccache/ccache/wiki/MS-Visual-Studio).
2022-11-04 01:17:22 +08:00
echo "Build completed successfully"