This is a large change because prior to this point, Python files in the
project were not consistently formatted. This reformats them all with
black defaults.
Based on experience with prior projects, if you have a dev/long-term
branch with Python patches, you can minimize merge conflicts prior to
rebasing to include this commit by running `black` on your modified
Python files, squashing, and then rebasing/merging.
This is part 1 of ~3, formatting all miscellaneous text files and CPP files matched by a first run of pre-commit. These tend to be low change-traffic and are likely not disruptive.
Subsequent patches will format Python files and remaining CPP files.
https://github.com/llvm/torch-mlir/pull/3055 adds
`lib/Dialect/Torch/Transforms/ScalarizeShapes.cpp`, which depends on
`torch-mlir/Dialect/TorchConversion/Transforms/BackendTypeConversion.h`.
```
ERROR: /root/.cache/bazel/_bazel_root/b89349c08f7224396763d14fe35cba11/external/torch-mlir/BUILD.bazel:170:11: Compiling lib/Dialect/Torch/Transforms/ScalarizeShapes.cpp failed: (Exit 1): clang failed: error executing command /usr/lib/llvm-16/bin/clang -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -Wunused-but-set-parameter -Wno-free-nonheap-object -fcolor-diagnostics -fno-omit-frame-pointer ... (remaining 101 arguments skipped)
Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
external/torch-mlir/lib/Dialect/Torch/Transforms/ScalarizeShapes.cpp:18:10: fatal error: 'torch-mlir/Dialect/TorchConversion/Transforms/BackendTypeConversion.h' file not found
#include "torch-mlir/Dialect/TorchConversion/Transforms/BackendTypeConversion.h"
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
Target @torch-mlir//:torch-mlir-opt failed to build
```
This PR adds the dependency and brings bazel builds back to green.
CI:
https://github.com/sjain-stanford/torch-mlir/actions/runs/8445558053/job/23132941876
Fixes a broken bazel build from https://github.com/llvm/torch-mlir/pull/1887 causing the following build error:
```
ERROR: /root/.cache/bazel/_bazel_root/b89349c08f7224396763d14fe35cba11/external/torch-mlir/BUILD.bazel:819:10: Compiling tools/torch-mlir-opt/torch-mlir-opt.cpp failed: (Exit 1): clang failed: error executing command /usr/lib/llvm-16/bin/clang -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -Wunused-but-set-parameter -Wno-free-nonheap-object -fcolor-diagnostics -fno-omit-frame-pointer ... (remaining 366 arguments skipped)
Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
external/torch-mlir/tools/torch-mlir-opt/torch-mlir-opt.cpp:16:10: fatal error: 'stablehlo/dialect/Register.h' file not found
#include "stablehlo/dialect/Register.h"
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
Target @torch-mlir//:torch-mlir-opt failed to build
```
GHA workflow with the fix:
https://github.com/sjain-stanford/torch-mlir/actions/runs/4421486154
This patch replaces all MHLO operations with their StableHLO
counterparts and adds a validation pass to ensure that no MHLO operations
remain before translating all Stablehlo operations to the MHLO dialect
for further lowering to the Linalg dialect.
This patch also updates all lit tests so that they refer to the
`convert-torch-to-stablehlo` pass and so that they check for StableHLO
operations.
- Use v3 of actions/checkout, since the version we use (v2) uses
Node.js 12, which is deprecated by GitHub.
- Source the PowerShell venv sctipt (instead of the bash sript) since
the calling script is a PowerShell script. Without this, the build
doesn't use venv at all.
- Make the build dependencies in whl-requirements.txt (used by
setup.py) match those in requirements.txt. To that end, this patch
creates a build-requirements.txt that is referenced by
requirements.txt and whl-requirements.txt.
A circular dependency was introduced in e7edcc62fd.
Specifically, the `makeShapeLLVMCompatible` and `makeShapeTorchCompatible` utilities were being called from `lib/Dialect/Torch/IR/TorchTypes.cpp` and `lib/Dialect/Torch/IR/TorchOps.cpp` defined under the `:TorchMLIRTorchDialect` bazel target, leading it to take a dependency on `:TorchMLIRConversionUtils` which already depends on `:TorchMLIRTorchDialect`, hence creating a circular dependency.
This commit resolves the same by moving said utilities from `lib/Conversion/Utils/Utils.cpp` to `lib/Dialect/Torch/Utils/Utils.cpp`. Please LMK if there's a better way to fix this and I will update the code.
This commit also adds the required targets to support building the new conversions from Torch to ML Program dialect that was introduced in f416953600.
Bazel build GHA triggered manually to verify: https://github.com/sjain-stanford/torch-mlir/actions/runs/3645944517
Adds support to run `.mlir` LIT tests in bazel.
```
bazel test @torch-mlir//test/...
```
Follow-on PR will contain these updates:
- Add tests to GHA CI workflow
- Add `.py` LIT tests to bazel
This introduces a new pass LowerToBackendContract (better name very
welcome) which performs the bulk of the simplifications that we do,
such as
- shape refinement
- dtype refinement
- maximizing value semantics
- inlining global slots
- decomposing complex ops
The key difference from before is that it iterates the set of
transformations, which can help to break a number of "catch-22" issues
where one simplification depends on another, the latest example being
here:
https://github.com/llvm/torch-mlir/issues/1131
This also exposed that RefineTypes was sometimes crashing/asserting for
certain inputs. This commit hardens it a bit.
This PR adds:
- A minimal docker wrapper to the bazel GHA workflow to make it reproducible locally
- Bazel cache to speed up GHA workflows (down to ~5 minutes from ~40+minutes)
This is a no-op for non-bazel workflows and an incremental improvement.
The torch-mlir bazel build is [failing](https://github.com/llvm/torch-mlir/runs/7737425906?check_suite_focus=true) since [this commit](504de5e701) due to a linker failure (undefined symbol: `mlir::torch::Torch::createEraseModuleInitializerPass()`).
```
ERROR: /home/runner/.cache/bazel/_bazel_runner/db599744cd37f8c161e5034d9b9cd520/external/torch-mlir/BUILD.bazel:845:10: Linking external/torch-mlir/torch-mlir-opt failed: (Exit 1): clang failed: error executing command /usr/lib/llvm-11/bin/clang @bazel-out/k8-fastbuild/bin/external/torch-mlir/torch-mlir-opt-2.params
Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
ld.lld: error: undefined symbol: mlir::torch::Torch::createEraseModuleInitializerPass()
>>> referenced by Passes.cpp
>>> bazel-out/k8-fastbuild/bin/external/torch-mlir/_objs/TorchMLIRTorchPasses/Passes.pic.o:(mlir::torch::Torch::createTorchFunctionToTorchBackendPipeline(mlir::OpPassManager&, mlir::torch::Torch::TorchLoweringPipelineOptions const&))
>>> referenced by Passes.cpp
>>> bazel-out/k8-fastbuild/bin/external/torch-mlir/_objs/TorchMLIRTorchPasses/Passes.pic.o:((anonymous namespace)::registerEraseModuleInitializerPass()::'lambda'()::operator()() const)
clang: error: linker command failed with exit code 1 (use -v to see invocation)
```
This PR adds `lib/Dialect/Torch/Transforms/EraseModuleInitializer.cpp` to `TorchMLIRTorchPasses` library.
Sambhav Jain
penguin_wwy
Peiming Liu
Stella Laurenzo
Yuanqiang Liu
Aart Bik
Ahmed S. Taei
Ashay Rane
Vivek Khandelwal
Tanyo Kwok
Sean Silva
Ramana Radhakrishnan
Jacques Pienaar
武家伟