llvm
/
torch-mlir
mirror of https://github.com/llvm/torch-mlir
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
torch-mlir/python/CMakeLists.txt

84 lines
3.3 KiB
CMake
Raw Normal View History

Wire through codegen and runtime dependencies. * Enables e2e test. * With what I've learned in upstream about test directory layout, I can consolidate most of the separate directories we have for these things. Will do that in a followup. * Not pleased with the LLVM global initialization depends but serviceable for now.
2020-07-11 13:45:30 +08:00
################################################################################
# Resources that must be packaged into the python tree
################################################################################
Fix precommit workflow (#13)
2020-08-07 14:51:05 +08:00
file(MAKE_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/npcomp/compiler/backend/refjit_resources")
Wire through codegen and runtime dependencies. * Enables e2e test. * With what I've learned in upstream about test directory layout, I can consolidate most of the separate directories we have for these things. Will do that in a followup. * Not pleased with the LLVM global initialization depends but serviceable for now.
2020-07-11 13:45:30 +08:00
add_custom_target(NPCOMPPythonResources)
add_custom_command(
Fix precommit workflow (#13)
2020-08-07 14:51:05 +08:00
TARGET NPCOMPPythonResources
COMMAND ${CMAKE_COMMAND} -E copy
Wire through codegen and runtime dependencies. * Enables e2e test. * With what I've learned in upstream about test directory layout, I can consolidate most of the separate directories we have for these things. Will do that in a followup. * Not pleased with the LLVM global initialization depends but serviceable for now.
2020-07-11 13:45:30 +08:00
# TODO: Make the runtime library work for windows.
Fix precommit workflow (#13)
2020-08-07 14:51:05 +08:00
${CMAKE_BINARY_DIR}/lib/libNPCOMPCompilerRuntimeShlib.so
Repurpose numpy-compiler compiler/runtime flow for PyTorch. * A bit gross because I took the chance to upgrade all of the backend bits to the new MLIR Python bindings and we still co-mingle the old and new for now. * Since the Python created PassManagers are configured for explicit nesting, I had to upgrade some of the pass pipelines to be explicit. * The demo in mul_maximum_e2e.py now compiles, runs through PyTorch and through the JIT, prints and asserts the same results. * I am not claiming that this is the prettiest API in this patch: consider that this is just directly using low-level APIs and there should be an intervening high level API.
2020-11-11 13:38:13 +08:00
${CMAKE_CURRENT_BINARY_DIR}/npcomp/compiler/generic/backend/libNPCOMPCompilerRuntimeShlib.so
Wire through codegen and runtime dependencies. * Enables e2e test. * With what I've learned in upstream about test directory layout, I can consolidate most of the separate directories we have for these things. Will do that in a followup. * Not pleased with the LLVM global initialization depends but serviceable for now.
2020-07-11 13:45:30 +08:00
)
add_dependencies(NPCOMPPythonResources
NPCOMPCompilerRuntimeShlib
)
Initial commit of python boiler-plate.
2020-04-27 06:50:23 +08:00
################################################################################
# Manage python source files
################################################################################
Create public API for torch_mlir python code. * Adds a trampoline/loader 'torch_mlir' module. * Plumbs through the MLIR python Context and Module creation, interoping with the MLIR Python API (resolves TODO on creating with own context and accessing the module being built). * Inter-module Python API interop is still a bit rough but workable via the capsule mechanism. Can be evolved later. * Exports the frontends/pytorch python sources to the project python/ build directory. * Requires D89294 to land.
2020-10-13 12:39:48 +08:00
npcomp_python_create_symlinks(${CMAKE_CURRENT_BINARY_DIR} ${CMAKE_CURRENT_SOURCE_DIR})
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
################################################################################
# Native extensions
################################################################################
# Normally on unix-like platforms, extensions are built as "MODULE" libraries
# and do not explicitly link to the python shared object. This allows for
# come greater deployment flexibility since the extension will bind to
# symbols in the python interpreter on load. However, it also keeps the
# linker from erroring on undefined symbols, leaving this to (usually obtuse)
# runtime errors. Building in "SHARED" mode with an explicit link to the
# python libraries allows us to build with the expectation of no undefined
# symbols, which is better for development.
# TODO(laurenzo): Windows requires linking against the PYTHON_LIBRARIES
# TODO(laurenzo): OSX requires allowing undefined (-undefined dynamic_lookup)
Misc fixes to enable building/testing on manylinux2014 images. * Since the manylinux images do not hard-link against python libs (resolving them at runtime), the module must be built without resolving undefined references. * For some reason, builds on this platform are stricter, exposing dependency ordering issues. * The CMake bits to build the extension are still somewhat of a mess. I have better versions both upstream and in IREE and will be reconciling. For now, don't look too closely.
2020-08-05 08:10:14 +08:00
if(NPCOMP_PYTHON_BINDINGS_VERSION_LOCKED)
set(NPCOMP_PYEXT_LINK_MODE SHARED)
Update PYTHON cmake variables to Python3 (#121) After the recent change of cmake variables from PYTHON_INCLUDE_DIRS to Python3_INCLUDE_DIRS and PYTHON_LIBRARIES to Python3_LIBRARIES, there were a few files that still had references to the old variables. This patch fixes that.
2020-11-18 08:04:14 +08:00
set(NPCOMP_PYEXT_LIBADD ${Python3_LIBRARIES})
Misc fixes to enable building/testing on manylinux2014 images. * Since the manylinux images do not hard-link against python libs (resolving them at runtime), the module must be built without resolving undefined references. * For some reason, builds on this platform are stricter, exposing dependency ordering issues. * The CMake bits to build the extension are still somewhat of a mess. I have better versions both upstream and in IREE and will be reconciling. For now, don't look too closely.
2020-08-05 08:10:14 +08:00
else()
set(NPCOMP_PYEXT_LINK_MODE MODULE)
set(NPCOMP_PYEXT_LIBADD)
endif()
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
if(NPCOMP_ENABLE_REFJIT)
list(APPEND NPCOMP_PYEXT_LIBADD
NPCOMPBackendRefJITPythonModule
)
endif()
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
add_library(NPCOMPNativePyExt ${NPCOMP_PYEXT_LINK_MODE}
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
NpcompModule.cpp
)
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
set_target_properties(NPCOMPNativePyExt PROPERTIES LIBRARY_OUTPUT_DIRECTORY
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
"${CMAKE_CURRENT_BINARY_DIR}")
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
set_target_properties(NPCOMPNativePyExt PROPERTIES OUTPUT_NAME _npcomp)
set_target_properties(NPCOMPNativePyExt PROPERTIES PREFIX
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
"${PYTHON_MODULE_PREFIX}")
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
set_target_properties(NPCOMPNativePyExt PROPERTIES SUFFIX
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
"${PYTHON_MODULE_EXTENSION}")
# pybind requires binding code to be compiled with -fvisibility=hidden
# Better code can be generated if the entire project compiles that way, but
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
# that is not enforced here.
set_target_properties(NPCOMPNativePyExt PROPERTIES CXX_VISIBILITY_PRESET "hidden")
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
get_property(dialect_libs GLOBAL PROPERTY MLIR_DIALECT_LIBS)
get_property(conversion_libs GLOBAL PROPERTY MLIR_CONVERSION_LIBS)
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
target_link_libraries(NPCOMPNativePyExt
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
PRIVATE
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
# Transitive dep on the shared library links most things from there.
NPCOMP
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
NPCOMPInitAll
NPCOMPPythonCommon
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
${NPCOMP_PYEXT_LIBADD}
Sever C++ level depend on IREE and rebase on exe and python interface. * IREE doesn't have proper install support, so there is some temporary hoaky hacking in our CMakeLists.txt to shuttle some symlinks around. * Reworked the original numpy e2e with IREE test to pipe through iree-translate. * Removed all of the C++-level dependencies. * Will generalize and apply to the PyTorch backend in a followup.
2020-11-14 10:06:59 +08:00
)
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
npcomp_python_target_compile_options(NPCOMPNativePyExt)
[cmake] Updates for basic shared library support (#7) Mostly this is CMake cleanup. Several library dependencies are missing, which is often revealed with shared library builds. Also, it's generally bad to link directly against LLVM libraries because it fails when using LLVM_LINK_LLVM_DYLIB. MLIR will pull in libLLVM.so, and there will be duplicate linkage with the the explicit libraries. There may need to be more refactoring here.
2020-08-06 05:49:18 +08:00
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
mlir_check_all_link_libraries(NPCOMPNativePyExt)