llvm
/
torch-mlir
mirror of https://github.com/llvm/torch-mlir
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
torch-mlir/python/CMakeLists.txt

86 lines
3.5 KiB
CMake
Raw Normal View History

Wire through codegen and runtime dependencies. * Enables e2e test. * With what I've learned in upstream about test directory layout, I can consolidate most of the separate directories we have for these things. Will do that in a followup. * Not pleased with the LLVM global initialization depends but serviceable for now.
2020-07-11 13:45:30 +08:00
################################################################################
# Resources that must be packaged into the python tree
################################################################################
Fix precommit workflow (#13)
2020-08-07 14:51:05 +08:00
file(MAKE_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/npcomp/compiler/backend/refjit_resources")
Wire through codegen and runtime dependencies. * Enables e2e test. * With what I've learned in upstream about test directory layout, I can consolidate most of the separate directories we have for these things. Will do that in a followup. * Not pleased with the LLVM global initialization depends but serviceable for now.
2020-07-11 13:45:30 +08:00
add_custom_target(NPCOMPPythonResources)
add_custom_command(
Fix precommit workflow (#13)
2020-08-07 14:51:05 +08:00
TARGET NPCOMPPythonResources
COMMAND ${CMAKE_COMMAND} -E copy
Wire through codegen and runtime dependencies. * Enables e2e test. * With what I've learned in upstream about test directory layout, I can consolidate most of the separate directories we have for these things. Will do that in a followup. * Not pleased with the LLVM global initialization depends but serviceable for now.
2020-07-11 13:45:30 +08:00
# TODO: Make the runtime library work for windows.
Fix OSX builds. (#143) --version_script doesn't work on OSX. Shared libs are .dylibs on OSX. TEST=Build on iMac Pro. M1 has other issues will be fixed later Change-Id: I2bda46349a878b8265e273c05d8db6b46c0df633
2020-12-28 17:30:45 +08:00
${CMAKE_BINARY_DIR}/lib/libNPCOMPCompilerRuntimeShlib${CMAKE_SHARED_LIBRARY_SUFFIX}
${CMAKE_CURRENT_BINARY_DIR}/npcomp/compiler/generic/backend/libNPCOMPCompilerRuntimeShlib${CMAKE_SHARED_LIBRARY_SUFFIX}
Wire through codegen and runtime dependencies. * Enables e2e test. * With what I've learned in upstream about test directory layout, I can consolidate most of the separate directories we have for these things. Will do that in a followup. * Not pleased with the LLVM global initialization depends but serviceable for now.
2020-07-11 13:45:30 +08:00
)
add_dependencies(NPCOMPPythonResources
NPCOMPCompilerRuntimeShlib
)
Initial commit of python boiler-plate.
2020-04-27 06:50:23 +08:00
################################################################################
# Manage python source files
################################################################################
Create public API for torch_mlir python code. * Adds a trampoline/loader 'torch_mlir' module. * Plumbs through the MLIR python Context and Module creation, interoping with the MLIR Python API (resolves TODO on creating with own context and accessing the module being built). * Inter-module Python API interop is still a bit rough but workable via the capsule mechanism. Can be evolved later. * Exports the frontends/pytorch python sources to the project python/ build directory. * Requires D89294 to land.
2020-10-13 12:39:48 +08:00
npcomp_python_create_symlinks(${CMAKE_CURRENT_BINARY_DIR} ${CMAKE_CURRENT_SOURCE_DIR})
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
################################################################################
# Native extensions
################################################################################
# Normally on unix-like platforms, extensions are built as "MODULE" libraries
# and do not explicitly link to the python shared object. This allows for
# come greater deployment flexibility since the extension will bind to
# symbols in the python interpreter on load. However, it also keeps the
# linker from erroring on undefined symbols, leaving this to (usually obtuse)
# runtime errors. Building in "SHARED" mode with an explicit link to the
# python libraries allows us to build with the expectation of no undefined
# symbols, which is better for development.
# TODO(laurenzo): Windows requires linking against the PYTHON_LIBRARIES
# TODO(laurenzo): OSX requires allowing undefined (-undefined dynamic_lookup)
Misc fixes to enable building/testing on manylinux2014 images. * Since the manylinux images do not hard-link against python libs (resolving them at runtime), the module must be built without resolving undefined references. * For some reason, builds on this platform are stricter, exposing dependency ordering issues. * The CMake bits to build the extension are still somewhat of a mess. I have better versions both upstream and in IREE and will be reconciling. For now, don't look too closely.
2020-08-05 08:10:14 +08:00
if(NPCOMP_PYTHON_BINDINGS_VERSION_LOCKED)
set(NPCOMP_PYEXT_LINK_MODE SHARED)
Update PYTHON cmake variables to Python3 (#121) After the recent change of cmake variables from PYTHON_INCLUDE_DIRS to Python3_INCLUDE_DIRS and PYTHON_LIBRARIES to Python3_LIBRARIES, there were a few files that still had references to the old variables. This patch fixes that.
2020-11-18 08:04:14 +08:00
set(NPCOMP_PYEXT_LIBADD ${Python3_LIBRARIES})
Misc fixes to enable building/testing on manylinux2014 images. * Since the manylinux images do not hard-link against python libs (resolving them at runtime), the module must be built without resolving undefined references. * For some reason, builds on this platform are stricter, exposing dependency ordering issues. * The CMake bits to build the extension are still somewhat of a mess. I have better versions both upstream and in IREE and will be reconciling. For now, don't look too closely.
2020-08-05 08:10:14 +08:00
else()
set(NPCOMP_PYEXT_LINK_MODE MODULE)
set(NPCOMP_PYEXT_LIBADD)
endif()
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
if(NPCOMP_ENABLE_REFJIT)
list(APPEND NPCOMP_PYEXT_LIBADD
NPCOMPBackendRefJITPythonModule
)
endif()
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
add_library(NPCOMPNativePyExt ${NPCOMP_PYEXT_LINK_MODE}
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
NpcompModule.cpp
)
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
set_target_properties(NPCOMPNativePyExt PROPERTIES LIBRARY_OUTPUT_DIRECTORY
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
"${CMAKE_CURRENT_BINARY_DIR}")
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
set_target_properties(NPCOMPNativePyExt PROPERTIES OUTPUT_NAME _npcomp)
set_target_properties(NPCOMPNativePyExt PROPERTIES PREFIX
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
"${PYTHON_MODULE_PREFIX}")
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
set_target_properties(NPCOMPNativePyExt PROPERTIES SUFFIX
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
"${PYTHON_MODULE_EXTENSION}")
# pybind requires binding code to be compiled with -fvisibility=hidden
# Better code can be generated if the entire project compiles that way, but
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
# that is not enforced here.
set_target_properties(NPCOMPNativePyExt PROPERTIES CXX_VISIBILITY_PRESET "hidden")
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
get_property(dialect_libs GLOBAL PROPERTY MLIR_DIALECT_LIBS)
get_property(conversion_libs GLOBAL PROPERTY MLIR_CONVERSION_LIBS)
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
target_link_libraries(NPCOMPNativePyExt
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
PRIVATE
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
# Transitive dep on the shared library links most things from there.
NPCOMP
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
NPCOMPInitAll
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
${NPCOMP_PYEXT_LIBADD}
Sever C++ level depend on IREE and rebase on exe and python interface. * IREE doesn't have proper install support, so there is some temporary hoaky hacking in our CMakeLists.txt to shuttle some symlinks around. * Reworked the original numpy e2e with IREE test to pipe through iree-translate. * Removed all of the C++-level dependencies. * Will generalize and apply to the PyTorch backend in a followup.
2020-11-14 10:06:59 +08:00
)
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
npcomp_python_target_compile_options(NPCOMPNativePyExt)
[cmake] Updates for basic shared library support (#7) Mostly this is CMake cleanup. Several library dependencies are missing, which is often revealed with shared library builds. Also, it's generally bad to link directly against LLVM libraries because it fails when using LLVM_LINK_LLVM_DYLIB. MLIR will pull in libLLVM.so, and there will be duplicate linkage with the the explicit libraries. There may need to be more refactoring here.
2020-08-06 05:49:18 +08:00
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
mlir_check_all_link_libraries(NPCOMPNativePyExt)
NFC: Delete npcomp python API and switch to upstream. * Most updates are mechanical except: * python/npcomp/__init__.py and python/NpcompModule.cpp: New init/registration bits to replace some automatic things being done in the old bindings. Also an annoying linkage hack that I'll need to triage next. * NpcompModule.cpp: New python helpers for custom types and other hard to reach items (for the new bindings). * PybindUtils.h: Extended type casting so that the local extension can directly exchange Mlir* C types. * python/npcomp/dialects/*: Build support and ODS bindings for local dialects. * mlir_utils.py: Defines an ImportContext to replace the old/bad "Helper" class that tracked locations, and insertion points. This has a number of methods on it that would be good candidates to think about better ways to do them upstream. * Also hoisted a few stand-alone samples to dedicated unit tests as they covered important things. * More cleanup can be done, but keeping this patch as mechanical as possible to stay in NFC land (this is big enough).
2020-12-30 05:22:18 +08:00
# Order dependent: Built artifacts add dependencies to the above targets.
add_subdirectory(npcomp/dialects)