The LTC backend has drifted from being able to pass tests on the stable
PyTorch version, so pinning to nightly on ARM.
Signed-Off By: Vivek Khandelwal <vivek@nod-labs.com>
The CI jobs that use stable PyTorch are currently not required to pass
in order for a patch to get merged in `main`. This commit makes sure
that if a CI job for stable PyTorch fails, it does not cancel the
other required jobs.
This patch removes the mock commands, so that once the Build And Test
workflow has successfully completed on the RollPyTorch action, the PR is
merged and the branch is deleted.
This patch adds a new workflow that runs when an update to the
rollpytorch branch by silvasean (in whose name the RollPyTorch action
runs) causes the regular CI build to complete without errors. Upon
execution, this workflow currently just prints the PR number(s) of the
PR created by the RollPyTorch action, but once this is working as
expected, we will add the step to merge the PR changes.
* CI: Skip (redundant) libtorch build when using stable PyTorch version
When we use PyTorch stable builds, there is no need to build libtorch
from source, making the stable-pytorch-with-torch-binary-OFF
configuration redundant with stable-pytorch-with-torch-binary-ON. This
patch drops the redundant configuration from CI.
* CI: Simplify guard conditions for creating and using libtorch cache
Whether libtorch is enabled or not is predicated on a host of conditions
such as the platform, in-tree versus out-of-tree build, and stable
versus nightly PyTorch builds. Instead of repeating these conditions to
guard whether to create or use the libtorch cache artifacts (and getting
them almost incorrect), this patch predicates the relevant pipeline
steps to whether libtorch is enabled, thus making the conditions far
simpler.
* feat: split pytorch requirements into stable and nightly
* fix: add true to tests to see full output
* refactor: add comments to explain true statement
* feat: move some tests to experimental mode
* refactor: refactor pipeline into more fine grained difference
* feat: add version differentiation for some tests
* feat: activate more configs
* refactor: change implementation to use less requirement files
* refactor: remove contraints used for testing
* fix: revert some requirement file names
* refactor: remove unnecessary ninja install
* fix: fix version parsing
* refactor: remove dependency on torchvision in main requirements file
* refactor: remove index url
* style: remove unnecesary line switch
* fix: readd index url
Creates a build_linux_arm64 job that builds the release on an arm64 self-hosted runner.
Drop Python 3.10 support
Pass TM_TORCH_VERSION to choose the Stable PyTorch version (since arm64 doesn't have nightly builds)
Borrows nightly / stable Pytorch switch from the WIP
https://github.com/llvm/torch-mlir/pull/2038
This patch adds a (default-true) input called `cache-enabled` to the
setup-build action, so that when the input is false, ccache is not setup
on the host machine. This patch also sets the input to be false for the
release builds.
Since PRs created by the GitHub action bot cannot trigger workflows (and
thus build tests), this patch uses the token for a GitHub app that was
specifically created for the RollPyTorch action.
We previously used a fork of the action/cache repository for the PyTorch
cache since the actions/cache repo did not support read-only caches.
Now that actions/cache supports separate read and write steps, this
patch switches back to the actions/cache repo.
This patch, by itself, doesn't fix caching on Windows, but once a new
release of ccache is available, caching for Windows builds should start
working again (validated by building ccache from source and using it
with LLVM builds).
Ccache rejects caching when either the `/Zi` or `/ZI` flags are used
during compilation on Windows, since these flags tell the compiler to
embed debug information in a PDB file (separate from the object file
produced by the compiler). In particular, our CI builds add the `/Zi`
flag, making ccache mark these compiler invocations as uncacheable.
But what caused our CI to add debug flags, especially when we specified
`-DCMAKE_BUILD_TYPE=Release`? On Windows, unless we specify the
`--config Release` flag during the CMake build step, CMake assumes a
debug build. So all this while, we had been producing debug builds of
torch-mlir for every PR! No doubt it took so long to build the Windows
binaries.
The reason for having to specify the configuration during the _build_
step (as opposed to the _configure_ step) of CMake on Windows is that
CMake's Visual Studio generators will produce _both_ Release and Debug
profiles during the CMake configure step (thus requiring a build-time
value that tells CMake whether to build in Release or Debug mode).
Luckily, on Linux and macOS, the `--config` flag seems to be simply
ignored, instead of causing build errors.
Strangely, based on cursory tests, it seems like on Windows we need to
specify the Relase configuration as both `-DCMAKE_BUILD_TYPE=Release` as
well as `--config Release`. Dropping either made my build switch to a
Debug configuration.
Additionally, there is a bug in ccache v4.8 (although this is addressed
in trunk) that causes ccache to reject caching if the compiler
invocation includes any flag that starts with `/Z`, including /`Zc`,
which is added by LLVM's HandleLLVMOptions.cmake and which isn't related
to debug info or PDB files. The next release of ccache should include
the fix, which is to reject caching only for `/Zi` and `/ZI` flags and
not all flags that start with `/Z`.
As a side note, debugging this problem was possible because of ccache's
log file, which is enabled by: `ccache --set-config="log_file=log.txt"`.
The GitHub action for creating the PR expects that either the changes
are not committed (in which case it commits them with the specified
commit message) or that the commit exists but that it is also pushed to
remote.
Prior to this patch, we created the commit but did not push it to
remote, causing failures. This patch leaves the changes uncommitted so
that they're committed and pushed to remote as part of the PR creation.
Currently, we run just the Linux in-tree tests when the RollPyTorch
workflow runs, but this is insufficient since WHL files for macOS or
Windows are sometimes not uploaded by PyTorch, causing the RollPyTorch
action to pass but all subsequent torch-mlir CI tests to fail because of
the broken build.
The easiest way to validate the RollPyTorch action on all platforms is
to run the standard set of tests that we run for each submitted PR, so
this patch makes the RollPyTorch action submit a PR instead of
committing the changes to the main branch directly. The PR is assigned
to a handful of folks for review, although this can be changed in the
future.
Despite using sudo to delete the workspace directory, we still
occasionally run into checkout errors. This patch thus drops the
deletion of the workspace prior to checkout. It also restricts the
number of parallel jobs in the submodule fetch step to just one, to try
and resolve the checkout issue ("index.lock: File exists.").
We have recently started seeing errors like:
```
Synchronizing submodule url for 'externals/llvm-project'
Synchronizing submodule url for 'externals/mlir-hlo'
/usr/bin/git -c protocol.version=2 submodule update --init --force --depth=1
Error: fatal: Unable to create '/home/anush/actions-runner/_work/torch-mlir/torch-mlir/.git/modules/externals/llvm-project/index.lock': File exists.
```
As a workaround, this patch removes the workspace directory before the
checkout step.
The RollPyTorch action needs the `unzip` command to peek into WHL files
for fetching metadata. This patch makes sure that the command is
installed before referencing it.
We want to ensure that pip packages required for building torch-mlir
should be included in the dependencies of torch-mlir, but we don't want
the pip packages required for _testing_ of torch-mlir to be included
among the dependencies. To be able to specify and install one set of
dependencies and not the other, this patch separates the pip packages
into two files: build-requirements.txt and test-requirements.txt.
This patch also updates references to the requirements.txt file so that
CI builds that run end-to-end tests install test-related pip
dependencies while everything else (including WHL builds) sticks to just
the build-related pip dependencies.
Despite this change, this patch should not affect a torch-mlir
developer's workflow. More precisely, since this patch makes the
top-level requirements.txt file refer to both build-requirements.txt and
test-requirements.txt files, a torch-mlir developer should be able to
continue referring to the requirements.txt file without any impact.
This patch replaces all MHLO operations with their StableHLO
counterparts and adds a validation pass to ensure that no MHLO operations
remain before translating all Stablehlo operations to the MHLO dialect
for further lowering to the Linalg dialect.
This patch also updates all lit tests so that they refer to the
`convert-torch-to-stablehlo` pass and so that they check for StableHLO
operations.
- Use v3 of actions/checkout, since the version we use (v2) uses
Node.js 12, which is deprecated by GitHub.
- Source the PowerShell venv sctipt (instead of the bash sript) since
the calling script is a PowerShell script. Without this, the build
doesn't use venv at all.
- Make the build dependencies in whl-requirements.txt (used by
setup.py) match those in requirements.txt. To that end, this patch
creates a build-requirements.txt that is referenced by
requirements.txt and whl-requirements.txt.
Now that the RollPyTorch tracker issue exists, we can automate the job
of notifying folks of failures instead of having to do it manually.
This patch adds a step to the workflow to post such a message.
There appear to be two problems with the caching layer in our CI runs:
(a) the sizes of some of the caches have grown to multiples of the
300 MB limit and (b) caching on Windows seems to be provide little to no
benefit.
To help understand the reasons for these problems, this patch adds a
line item to the list of steps run in CI to dump the ccache
configuration and statistics just prior to uploading the cache artifact.
The RollPyTorch action often takes more than 1.5 hours to finish.
During this time, if another PR is merged, then the RollPyTorch action
needs to first pull the merged changes before committing the updates to
the PyTorch commit hash and version files. This patch adds the required
`git pull` statement, without which, the subsequent `git push` statement
fails, causing the RollPyTorch action to fail as well.
* [custom op] Generalize shape library logic to work with dtypes
This commit generalizes the shape library logic, so that dtype rules
for ops can also be expressed using the same mechanism. In other
words, each op can now have a shape function and a dtype function
specified in Python that is imported during lowering to calculate the
shapes and dtypes throught a program. For more information about how
to specify a dtype function, see the updated
`docs/adding_a_shape_and_dtype_function.md`.
For those not familiar with how the shape library works, the file
`docs/calculations_lib.md` provides an overview.
Until recently, the metadata file in the torchvision package included
the nightly version of the torch package, but since that is no longer
the case, our RollPyTorch workflow is broken.
As a workaround, this patch uses the `pip download` command's ability to
fetch the dependent torch package for the specified version of
torchvision, before peeking into the WHL file for the torch package to
determine the release version and the commit hash.
The upload timestamp of the nightly torchvision package has drifted
beyond the scheduled time of the RollPyTorch action because of the time
change due to daylight saving. As a result, the RollPyTorch action now
picks the torchvision package from a day earlier instead of the most
recent package.
This patch schedules the RollPyTorch action to start one hour later than
before so that it continues to pick the most recent nightly package.
Vivek Khandelwal
Ashay Rane
powderluv
Ramiro Leal-Cavazos
Sambhav Jain
maxbartel
Maksim Levental
Daniel Ellis