This moves the bulk of the Python code (including the Torch interop)
from `frontends/pytorch` into `torch-mlir/TorchPlugin`. This also
required reconciling a bunch of other Python-related stuff, like the
`torch` dialects.
As I did this, it was simpler to just remove all the old numpy/basicpy
stuff because we were going to delete it anyway and it was faster than
debugging an intermediate state that would only last O(days) anyway.
torch-mlir has two top-level python packages (built into the
`python_packages` directory):
- `torch_mlir_dialects`: `torch` dialect Python bindings (does not
depend on PyTorch). This also involves building the aggregate CAPI for
`torch-mlir`.
- `torch_mlir`: bindings to the part of the code that links against
PyTorch (or C++ code that transitively does).
Additionally, there remain two more Python packages in npcomp (but
outside `torch-mlir`):
- `npcomp_torch`: Contains the e2e test framework and testing configs
that plug into RefBackend and IREE.
- `npcomp_core`: Contains the low-level interfaces to RefBackend and
IREE that `npcomp_torch` uses, along with its own
`MLIR_PYTHON_PACKAGE_PREFIX=npcomp.` aggregation of the core MLIR
python bindings. (all other functionality has been stripped out)
After all the basicpy/numpy deletions, the `npcomp` C++ code is now very
tiny. It basically just contains RefBackend and the `TorchConversion`
dialect/passes (e.g. `TorchToLinalg.cpp`).
Correspondingly, there are now 4 main testing targets paralleling the
Python layering (which is reflective of the deeper underlying dependency
structure)
- `check-torch-mlir`: checks the `torch-mlir` pure MLIR C++ code.
- `check-torch-mlir-plugin`: checks the code in `TorchPlugin` (e.g.
TorchScript import)
- `check-frontends-pytorch`: Checks the little code we have in
`frontends/pytorch` -- mainly things related to the e2e framework
itself.
- `check-npcomp`: Checks the pure MLIR C++ code inside npcomp.
There is a target `check-npcomp-all` that runs all of them.
The `torch-mlir/build_standalone.sh` script does a standalone build of
`torch-mlir`.
The e2e tests (`tools/torchscript_e2e_test.sh`) are working too.
The update_torch_ods script now lives in
`torch-mlir/build_tools/update_torch_ods.sh` and expects a standalone
build.
This change also required a fix upstream related to cross-shlib Python
dependencies, so we also update llvm-project to
8dca953dd39c0cd8c80decbeb38753f58a4de580 to get
https://reviews.llvm.org/D109776 (no other fixes were needed for the
integrate, thankfully).
This completes most of the large source code changes. Next will be
bringing the CI/packaging/examples back to life.
* Adds a minimal setup.py for frontends/pytorch
* Makes npcomp-core export its headers and libraries
* Adds a script to build packages.
* Adds CI step to package and smoke test.
* Will need some more tweaks and coordination prior to deploying (version locking etc).
This includes IREE and RefBackend.
This includes a fixup to torchscript_e2e_test.sh for handling the
situation where PYTHONPATH was not already exported.
I'm seeing the following error:
```
CMake Error in frontends/pytorch/csrc/CMakeLists.txt:
Imported target "torch" includes non-existent path
"/usr/local/include/breakpad"
in its INTERFACE_INCLUDE_DIRECTORIES.
```
Reported upstream in: https://github.com/pytorch/pytorch/issues/60485
This temporarily works around the CMake error:
```
CMake Error in frontends/pytorch/csrc/CMakeLists.txt:
Imported target "torch" includes non-existent path
"/pytorch/torch/lib"
in its INTERFACE_INCLUDE_DIRECTORIES.
```
torchvision nightly has not bump to 0.10.0 alpha, so pip installs
torchvision==0.9.0 even with the --pre flag.
Signed-off-by: Bairen Yi <yibairen.byron@bytedance.com>
So CI build options are closer to those in `build_tools/install_mlir.sh`.
Also append hash of CI spec file to LLVM commit hash when caching builds.
Signed-off-by: Bairen Yi <yibairen.byron@bytedance.com>
* Incorporates source fixes.
* Uses upstream pybind11 detection logic.
* Patches CI.
* This may break the CI, which will need to be fixed manually in a followup.
Sean Silva
Stella Laurenzo
Bairen Yi
Bairen Yi
powderluv
Marius Brehler
stephenneuendorffer
stephenneuendorffer