llvm
/
torch-mlir
mirror of https://github.com/llvm/torch-mlir
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
torch-mlir/python/CMakeLists.txt

119 lines
3.9 KiB
CMake
Raw Normal View History

Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
include(AddMLIRPython)
include(MLIRDetectPythonEnv)
Merge npcomp and mlir python namespaces. * Now the parts of the MLIR API are directly exported under the npcomp module (i.e. `npcomp.ir`, etc). * Has required fixes for https://reviews.llvm.org/D108489 * Deletes npcomp.tracing vs fixing it because it was a very early experiment that will not be carried forward. * This makes the npcomp python distribution completely standalone and separate from an mlir installation. * Makes most of npcomp itself relocatable for future use as a library. * Most things are a namespace package now. In the future we can s/torch_mlir/npcomp.frontends.torch/ and have it layer properly.
2021-08-23 03:17:12 +08:00
add_compile_definitions("MLIR_PYTHON_PACKAGE_PREFIX=npcomp.")
Initial commit of python boiler-plate.
2020-04-27 06:50:23 +08:00
################################################################################
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
# Declare sources
Initial commit of python boiler-plate.
2020-04-27 06:50:23 +08:00
################################################################################
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
Build packages for npcomp-torch. * Adds a minimal setup.py for frontends/pytorch * Makes npcomp-core export its headers and libraries * Adds a script to build packages. * Adds CI step to package and smoke test. * Will need some more tweaks and coordination prior to deploying (version locking etc).
2021-07-30 01:43:18 +08:00
declare_mlir_python_sources(NPCOMPPythonSources)
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
Build packages for npcomp-torch. * Adds a minimal setup.py for frontends/pytorch * Makes npcomp-core export its headers and libraries * Adds a script to build packages. * Adds CI step to package and smoke test. * Will need some more tweaks and coordination prior to deploying (version locking etc).
2021-07-30 01:43:18 +08:00
declare_mlir_python_sources(NPCOMPPythonSources.Core
Merge npcomp and mlir python namespaces. * Now the parts of the MLIR API are directly exported under the npcomp module (i.e. `npcomp.ir`, etc). * Has required fixes for https://reviews.llvm.org/D108489 * Deletes npcomp.tracing vs fixing it because it was a very early experiment that will not be carried forward. * This makes the npcomp python distribution completely standalone and separate from an mlir installation. * Makes most of npcomp itself relocatable for future use as a library. * Most things are a namespace package now. In the future we can s/torch_mlir/npcomp.frontends.torch/ and have it layer properly.
2021-08-23 03:17:12 +08:00
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/npcomp"
Remove frontends/pytorch directory. It just contained the e2e testing framework. We now fold it into the main project to reduce complexity. - `frontends/pytorch/python/` -> `python/torch_support` - `frontends/pytorch/e2e_testing -> e2e_testing` - `frontends/pytorch/examples -> examples` - `frontends/pytorch/test` -> `python/test` - `torch_mlir_torchscript` python module -> `npcomp_torchscript` - `torch_mlir_torchscript_e2e_test_configs` python module -> `npcomp_torchscript_e2e_test_configs` This also changes the license of a handful of files from the "pytorch-style" license to the regular LLVM/npcomp license. The only people who committed to those files were myself and Yi.
2021-09-17 06:18:17 +08:00
ADD_TO_PARENT NPCOMPPythonSources
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
SOURCES_GLOB
Remove frontends/pytorch directory. It just contained the e2e testing framework. We now fold it into the main project to reduce complexity. - `frontends/pytorch/python/` -> `python/torch_support` - `frontends/pytorch/e2e_testing -> e2e_testing` - `frontends/pytorch/examples -> examples` - `frontends/pytorch/test` -> `python/test` - `torch_mlir_torchscript` python module -> `npcomp_torchscript` - `torch_mlir_torchscript_e2e_test_configs` python module -> `npcomp_torchscript_e2e_test_configs` This also changes the license of a handful of files from the "pytorch-style" license to the regular LLVM/npcomp license. The only people who committed to those files were myself and Yi.
2021-09-17 06:18:17 +08:00
*.py
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
)
Build packages for npcomp-torch. * Adds a minimal setup.py for frontends/pytorch * Makes npcomp-core export its headers and libraries * Adds a script to build packages. * Adds CI step to package and smoke test. * Will need some more tweaks and coordination prior to deploying (version locking etc).
2021-07-30 01:43:18 +08:00
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
declare_mlir_python_sources(NPCOMPPythonSources.Dialects
ADD_TO_PARENT NPCOMPPythonSources
)
declare_mlir_python_sources(NPCOMPPythonExtensions)
Build packages for npcomp-torch. * Adds a minimal setup.py for frontends/pytorch * Makes npcomp-core export its headers and libraries * Adds a script to build packages. * Adds CI step to package and smoke test. * Will need some more tweaks and coordination prior to deploying (version locking etc).
2021-07-30 01:43:18 +08:00
declare_mlir_python_sources(NPCOMPPythonCAPIHeaderSources
ROOT_DIR "${MLIR_NPCOMP_SOURCE_DIR}/include"
SOURCES_GLOB "npcomp-c/*.h"
DEST_PREFIX "_mlir_libs/include"
)
################################################################################
# Extensions
################################################################################
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
declare_mlir_python_extension(NPCOMPPythonExtensions.Core
MODULE_NAME _npcomp
ADD_TO_PARENT NPCOMPPythonExtensions
SOURCES
${CMAKE_CURRENT_SOURCE_DIR}/NpcompModule.cpp
EMBED_CAPI_LINK_LIBS
NPCOMPCAPI
PRIVATE_LINK_LIBS
LLVMSupport
)
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
################################################################################
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
# Declare dialects
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
################################################################################
[torch-mlir earthmoving (2/N)] Python code movement. This moves the bulk of the Python code (including the Torch interop) from `frontends/pytorch` into `torch-mlir/TorchPlugin`. This also required reconciling a bunch of other Python-related stuff, like the `torch` dialects. As I did this, it was simpler to just remove all the old numpy/basicpy stuff because we were going to delete it anyway and it was faster than debugging an intermediate state that would only last O(days) anyway. torch-mlir has two top-level python packages (built into the `python_packages` directory): - `torch_mlir_dialects`: `torch` dialect Python bindings (does not depend on PyTorch). This also involves building the aggregate CAPI for `torch-mlir`. - `torch_mlir`: bindings to the part of the code that links against PyTorch (or C++ code that transitively does). Additionally, there remain two more Python packages in npcomp (but outside `torch-mlir`): - `npcomp_torch`: Contains the e2e test framework and testing configs that plug into RefBackend and IREE. - `npcomp_core`: Contains the low-level interfaces to RefBackend and IREE that `npcomp_torch` uses, along with its own `MLIR_PYTHON_PACKAGE_PREFIX=npcomp.` aggregation of the core MLIR python bindings. (all other functionality has been stripped out) After all the basicpy/numpy deletions, the `npcomp` C++ code is now very tiny. It basically just contains RefBackend and the `TorchConversion` dialect/passes (e.g. `TorchToLinalg.cpp`). Correspondingly, there are now 4 main testing targets paralleling the Python layering (which is reflective of the deeper underlying dependency structure) - `check-torch-mlir`: checks the `torch-mlir` pure MLIR C++ code. - `check-torch-mlir-plugin`: checks the code in `TorchPlugin` (e.g. TorchScript import) - `check-frontends-pytorch`: Checks the little code we have in `frontends/pytorch` -- mainly things related to the e2e framework itself. - `check-npcomp`: Checks the pure MLIR C++ code inside npcomp. There is a target `check-npcomp-all` that runs all of them. The `torch-mlir/build_standalone.sh` script does a standalone build of `torch-mlir`. The e2e tests (`tools/torchscript_e2e_test.sh`) are working too. The update_torch_ods script now lives in `torch-mlir/build_tools/update_torch_ods.sh` and expects a standalone build. This change also required a fix upstream related to cross-shlib Python dependencies, so we also update llvm-project to 8dca953dd39c0cd8c80decbeb38753f58a4de580 to get https://reviews.llvm.org/D109776 (no other fixes were needed for the integrate, thankfully). This completes most of the large source code changes. Next will be bringing the CI/packaging/examples back to life.
2021-09-11 02:44:38 +08:00
# Declare dialects with `declare_mlir_dialect_python_bindings`.
# Currently there are none, because the interesting ones to bind have been
# removed or moved into torch-mlir.
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
################################################################################
# Build composite binaries
################################################################################
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
# Bundle our own, self-contained CAPI library with all of our deps.
add_mlir_python_common_capi_library(NPCOMPPythonCAPI
INSTALL_COMPONENT NPCOMPPythonModules
Merge npcomp and mlir python namespaces. * Now the parts of the MLIR API are directly exported under the npcomp module (i.e. `npcomp.ir`, etc). * Has required fixes for https://reviews.llvm.org/D108489 * Deletes npcomp.tracing vs fixing it because it was a very early experiment that will not be carried forward. * This makes the npcomp python distribution completely standalone and separate from an mlir installation. * Makes most of npcomp itself relocatable for future use as a library. * Most things are a namespace package now. In the future we can s/torch_mlir/npcomp.frontends.torch/ and have it layer properly.
2021-08-23 03:17:12 +08:00
INSTALL_DESTINATION python_packages/npcomp_core/npcomp/_mlir_libs
OUTPUT_DIRECTORY "${MLIR_NPCOMP_PYTHON_PACKAGES_DIR}/npcomp_core/npcomp/_mlir_libs"
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
RELATIVE_INSTALL_ROOT "../../../.."
DECLARED_SOURCES
Merge npcomp and mlir python namespaces. * Now the parts of the MLIR API are directly exported under the npcomp module (i.e. `npcomp.ir`, etc). * Has required fixes for https://reviews.llvm.org/D108489 * Deletes npcomp.tracing vs fixing it because it was a very early experiment that will not be carried forward. * This makes the npcomp python distribution completely standalone and separate from an mlir installation. * Makes most of npcomp itself relocatable for future use as a library. * Most things are a namespace package now. In the future we can s/torch_mlir/npcomp.frontends.torch/ and have it layer properly.
2021-08-23 03:17:12 +08:00
# TODO: Common MLIR deps can be reduced substantially.
MLIRPythonSources.Core
MLIRPythonSources.Dialects
MLIRPythonSources.ExecutionEngine
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
MLIRPythonExtension.AllPassesRegistration
NPCOMPPythonSources
NPCOMPPythonExtensions
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
)
Merge npcomp and mlir python namespaces. * Now the parts of the MLIR API are directly exported under the npcomp module (i.e. `npcomp.ir`, etc). * Has required fixes for https://reviews.llvm.org/D108489 * Deletes npcomp.tracing vs fixing it because it was a very early experiment that will not be carried forward. * This makes the npcomp python distribution completely standalone and separate from an mlir installation. * Makes most of npcomp itself relocatable for future use as a library. * Most things are a namespace package now. In the future we can s/torch_mlir/npcomp.frontends.torch/ and have it layer properly.
2021-08-23 03:17:12 +08:00
# Bundle MLIR and NPCOMP into the top-level npcomp package.
add_mlir_python_modules(NPCOMPPythonModules
ROOT_PREFIX "${MLIR_NPCOMP_PYTHON_PACKAGES_DIR}/npcomp_core/npcomp"
INSTALL_PREFIX "python_packages/npcomp_core/npcomp"
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
DECLARED_SOURCES
MLIRPythonSources
MLIRPythonExtension.AllPassesRegistration
Build packages for npcomp-torch. * Adds a minimal setup.py for frontends/pytorch * Makes npcomp-core export its headers and libraries * Adds a script to build packages. * Adds CI step to package and smoke test. * Will need some more tweaks and coordination prior to deploying (version locking etc).
2021-07-30 01:43:18 +08:00
MLIRPythonCAPIHeaderSources
Merge npcomp and mlir python namespaces. * Now the parts of the MLIR API are directly exported under the npcomp module (i.e. `npcomp.ir`, etc). * Has required fixes for https://reviews.llvm.org/D108489 * Deletes npcomp.tracing vs fixing it because it was a very early experiment that will not be carried forward. * This makes the npcomp python distribution completely standalone and separate from an mlir installation. * Makes most of npcomp itself relocatable for future use as a library. * Most things are a namespace package now. In the future we can s/torch_mlir/npcomp.frontends.torch/ and have it layer properly.
2021-08-23 03:17:12 +08:00
NPCOMPPythonSources
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
NPCOMPPythonExtensions
Build packages for npcomp-torch. * Adds a minimal setup.py for frontends/pytorch * Makes npcomp-core export its headers and libraries * Adds a script to build packages. * Adds CI step to package and smoke test. * Will need some more tweaks and coordination prior to deploying (version locking etc).
2021-07-30 01:43:18 +08:00
NPCOMPPythonCAPIHeaderSources
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
COMMON_CAPI_LINK_LIBS
NPCOMPPythonCAPI
)
Remove frontends/pytorch directory. It just contained the e2e testing framework. We now fold it into the main project to reduce complexity. - `frontends/pytorch/python/` -> `python/torch_support` - `frontends/pytorch/e2e_testing -> e2e_testing` - `frontends/pytorch/examples -> examples` - `frontends/pytorch/test` -> `python/test` - `torch_mlir_torchscript` python module -> `npcomp_torchscript` - `torch_mlir_torchscript_e2e_test_configs` python module -> `npcomp_torchscript_e2e_test_configs` This also changes the license of a handful of files from the "pytorch-style" license to the regular LLVM/npcomp license. The only people who committed to those files were myself and Yi.
2021-09-17 06:18:17 +08:00
################################################################################
# Torch support libraries.
################################################################################
# Torch-related modules that are too broad of a scope/opinion for torch-mlir
# belong there.
declare_mlir_python_sources(NPCOMPTorchSupportPythonSources
ADD_TO_PARENT NPCOMPPythonSources
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/torch_support"
SOURCES_GLOB
npcomp_torchscript/*.py
npcomp_torchscript_e2e_test_configs/*.py
)
add_mlir_python_modules(NPCOMPTorchSupportPythonModules
ROOT_PREFIX "${MLIR_NPCOMP_PYTHON_PACKAGES_DIR}/npcomp_core"
INSTALL_PREFIX "python_packages/npcomp_core"
DECLARED_SOURCES
NPCOMPTorchSupportPythonSources
)
add_dependencies(NPCOMPTorchSupportPythonModules
NPCOMPPythonModules
)
################################################################################
# Recurse into the tests.
################################################################################
Bring CI back to life. This brings back `check-npcomp-all` and the refbackend e2e tests coverage.
2021-09-17 02:39:53 +08:00
Remove frontends/pytorch directory. It just contained the e2e testing framework. We now fold it into the main project to reduce complexity. - `frontends/pytorch/python/` -> `python/torch_support` - `frontends/pytorch/e2e_testing -> e2e_testing` - `frontends/pytorch/examples -> examples` - `frontends/pytorch/test` -> `python/test` - `torch_mlir_torchscript` python module -> `npcomp_torchscript` - `torch_mlir_torchscript_e2e_test_configs` python module -> `npcomp_torchscript_e2e_test_configs` This also changes the license of a handful of files from the "pytorch-style" license to the regular LLVM/npcomp license. The only people who committed to those files were myself and Yi.
2021-09-17 06:18:17 +08:00
add_subdirectory(test)