torch-mlir/frontends/pytorch/python/CMakeLists.txt

################################################################################
# Manage python source files
# Collapse all local python sources to the project level python/ directory.
################################################################################

include(NpcompPython)

npcomp_python_create_symlinks(
  ${MLIR_NPCOMP_PYTHON_PACKAGES_DIR}/npcomp_torch ${CMAKE_CURRENT_SOURCE_DIR})
Create public API for torch_mlir python code. * Adds a trampoline/loader 'torch_mlir' module. * Plumbs through the MLIR python Context and Module creation, interoping with the MLIR Python API (resolves TODO on creating with own context and accessing the module being built). * Inter-module Python API interop is still a bit rough but workable via the capsule mechanism. Can be evolved later. * Exports the frontends/pytorch python sources to the project python/ build directory. * Requires D89294 to land. 2020-10-13 12:39:48 +08:00			`################################################################################`
			`# Manage python source files`
			`# Collapse all local python sources to the project level python/ directory.`
			`################################################################################`

[torch-mlir earthmoving (2/N)] Python code movement. This moves the bulk of the Python code (including the Torch interop) from `frontends/pytorch` into `torch-mlir/TorchPlugin`. This also required reconciling a bunch of other Python-related stuff, like the `torch` dialects. As I did this, it was simpler to just remove all the old numpy/basicpy stuff because we were going to delete it anyway and it was faster than debugging an intermediate state that would only last O(days) anyway. torch-mlir has two top-level python packages (built into the `python_packages` directory): - `torch_mlir_dialects`: `torch` dialect Python bindings (does not depend on PyTorch). This also involves building the aggregate CAPI for `torch-mlir`. - `torch_mlir`: bindings to the part of the code that links against PyTorch (or C++ code that transitively does). Additionally, there remain two more Python packages in npcomp (but outside `torch-mlir`): - `npcomp_torch`: Contains the e2e test framework and testing configs that plug into RefBackend and IREE. - `npcomp_core`: Contains the low-level interfaces to RefBackend and IREE that `npcomp_torch` uses, along with its own `MLIR_PYTHON_PACKAGE_PREFIX=npcomp.` aggregation of the core MLIR python bindings. (all other functionality has been stripped out) After all the basicpy/numpy deletions, the `npcomp` C++ code is now very tiny. It basically just contains RefBackend and the `TorchConversion` dialect/passes (e.g. `TorchToLinalg.cpp`). Correspondingly, there are now 4 main testing targets paralleling the Python layering (which is reflective of the deeper underlying dependency structure) - `check-torch-mlir`: checks the `torch-mlir` pure MLIR C++ code. - `check-torch-mlir-plugin`: checks the code in `TorchPlugin` (e.g. TorchScript import) - `check-frontends-pytorch`: Checks the little code we have in `frontends/pytorch` -- mainly things related to the e2e framework itself. - `check-npcomp`: Checks the pure MLIR C++ code inside npcomp. There is a target `check-npcomp-all` that runs all of them. The `torch-mlir/build_standalone.sh` script does a standalone build of `torch-mlir`. The e2e tests (`tools/torchscript_e2e_test.sh`) are working too. The update_torch_ods script now lives in `torch-mlir/build_tools/update_torch_ods.sh` and expects a standalone build. This change also required a fix upstream related to cross-shlib Python dependencies, so we also update llvm-project to 8dca953dd39c0cd8c80decbeb38753f58a4de580 to get https://reviews.llvm.org/D109776 (no other fixes were needed for the integrate, thankfully). This completes most of the large source code changes. Next will be bringing the CI/packaging/examples back to life. 2021-09-11 02:44:38 +08:00			`include(NpcompPython)`

Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI. 2021-07-28 07:10:10 +08:00			`npcomp_python_create_symlinks(`
			`${MLIR_NPCOMP_PYTHON_PACKAGES_DIR}/npcomp_torch ${CMAKE_CURRENT_SOURCE_DIR})`