llvm
/
torch-mlir
mirror of https://github.com/llvm/torch-mlir
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
torch-mlir/python/CMakeLists.txt

128 lines
4.6 KiB
CMake
Raw Normal View History

Wire through codegen and runtime dependencies. * Enables e2e test. * With what I've learned in upstream about test directory layout, I can consolidate most of the separate directories we have for these things. Will do that in a followup. * Not pleased with the LLVM global initialization depends but serviceable for now.
2020-07-11 13:45:30 +08:00
################################################################################
# Resources that must be packaged into the python tree
################################################################################
Fix precommit workflow (#13)
2020-08-07 14:51:05 +08:00
file(MAKE_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/npcomp/compiler/backend/refjit_resources")
Wire through codegen and runtime dependencies. * Enables e2e test. * With what I've learned in upstream about test directory layout, I can consolidate most of the separate directories we have for these things. Will do that in a followup. * Not pleased with the LLVM global initialization depends but serviceable for now.
2020-07-11 13:45:30 +08:00
add_custom_target(NPCOMPPythonResources)
add_custom_command(
Fix precommit workflow (#13)
2020-08-07 14:51:05 +08:00
TARGET NPCOMPPythonResources
COMMAND ${CMAKE_COMMAND} -E copy
Wire through codegen and runtime dependencies. * Enables e2e test. * With what I've learned in upstream about test directory layout, I can consolidate most of the separate directories we have for these things. Will do that in a followup. * Not pleased with the LLVM global initialization depends but serviceable for now.
2020-07-11 13:45:30 +08:00
# TODO: Make the runtime library work for windows.
Fix precommit workflow (#13)
2020-08-07 14:51:05 +08:00
${CMAKE_BINARY_DIR}/lib/libNPCOMPCompilerRuntimeShlib.so
Wire through codegen and runtime dependencies. * Enables e2e test. * With what I've learned in upstream about test directory layout, I can consolidate most of the separate directories we have for these things. Will do that in a followup. * Not pleased with the LLVM global initialization depends but serviceable for now.
2020-07-11 13:45:30 +08:00
${CMAKE_CURRENT_BINARY_DIR}/npcomp/compiler/backend/refjit_resources/libNPCOMPCompilerRuntimeShlib.so
)
add_dependencies(NPCOMPPythonResources
NPCOMPCompilerRuntimeShlib
)
Initial commit of python boiler-plate.
2020-04-27 06:50:23 +08:00
################################################################################
# Manage python source files
################################################################################
function (create_symlinks)
# Do nothing if building in-source
if (${CMAKE_CURRENT_BINARY_DIR} STREQUAL ${CMAKE_CURRENT_SOURCE_DIR})
return()
endif()
foreach (path_file ${ARGN})
get_filename_component(folder ${path_file} PATH)
# Create REAL folder
file(MAKE_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/${folder}")
# Delete symlink if it exists
file(REMOVE "${CMAKE_CURRENT_BINARY_DIR}/${path_file}")
# Get OS dependent path to use in `execute_process`
file(TO_NATIVE_PATH "${CMAKE_CURRENT_BINARY_DIR}/${path_file}" link)
file(TO_NATIVE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/${path_file}" target)
if (UNIX)
set(command ln -s ${target} ${link})
else()
set(command cmd.exe /c mklink ${link} ${target})
endif()
Wire through codegen and runtime dependencies. * Enables e2e test. * With what I've learned in upstream about test directory layout, I can consolidate most of the separate directories we have for these things. Will do that in a followup. * Not pleased with the LLVM global initialization depends but serviceable for now.
2020-07-11 13:45:30 +08:00
execute_process(COMMAND ${command}
Initial commit of python boiler-plate.
2020-04-27 06:50:23 +08:00
RESULT_VARIABLE result
ERROR_VARIABLE output)
if (NOT ${result} EQUAL 0)
message(FATAL_ERROR "Could not create symbolic link for: ${target} --> ${output}")
endif()
endforeach(path_file)
endfunction(create_symlinks)
file(GLOB_RECURSE python_files RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} *.py)
create_symlinks(${python_files})
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
################################################################################
# Native extensions
################################################################################
include(NpcompPython)
# Normally on unix-like platforms, extensions are built as "MODULE" libraries
# and do not explicitly link to the python shared object. This allows for
# come greater deployment flexibility since the extension will bind to
# symbols in the python interpreter on load. However, it also keeps the
# linker from erroring on undefined symbols, leaving this to (usually obtuse)
# runtime errors. Building in "SHARED" mode with an explicit link to the
# python libraries allows us to build with the expectation of no undefined
# symbols, which is better for development.
# TODO(laurenzo): Windows requires linking against the PYTHON_LIBRARIES
# TODO(laurenzo): OSX requires allowing undefined (-undefined dynamic_lookup)
Misc fixes to enable building/testing on manylinux2014 images. * Since the manylinux images do not hard-link against python libs (resolving them at runtime), the module must be built without resolving undefined references. * For some reason, builds on this platform are stricter, exposing dependency ordering issues. * The CMake bits to build the extension are still somewhat of a mess. I have better versions both upstream and in IREE and will be reconciling. For now, don't look too closely.
2020-08-05 08:10:14 +08:00
if(NPCOMP_PYTHON_BINDINGS_VERSION_LOCKED)
set(NPCOMP_PYEXT_LINK_MODE SHARED)
set(NPCOMP_PYEXT_LIBADD ${PYTHON_LIBRARIES})
else()
set(NPCOMP_PYEXT_LINK_MODE MODULE)
set(NPCOMP_PYEXT_LIBADD)
endif()
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
if(NPCOMP_ENABLE_IREE)
list(APPEND NPCOMP_PYEXT_LIBADD NPCOMPBackendIREEPythonModule)
endif()
if(NPCOMP_ENABLE_REFJIT)
list(APPEND NPCOMP_PYEXT_LIBADD
NPCOMPBackendRefJITPythonModule
)
endif()
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
add_library(NPCOMPNativePyExt ${NPCOMP_PYEXT_LINK_MODE}
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
NpcompModule.cpp
)
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
set_target_properties(NPCOMPNativePyExt PROPERTIES LIBRARY_OUTPUT_DIRECTORY
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
"${CMAKE_CURRENT_BINARY_DIR}")
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
set_target_properties(NPCOMPNativePyExt PROPERTIES OUTPUT_NAME _npcomp)
set_target_properties(NPCOMPNativePyExt PROPERTIES PREFIX
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
"${PYTHON_MODULE_PREFIX}")
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
set_target_properties(NPCOMPNativePyExt PROPERTIES SUFFIX
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
"${PYTHON_MODULE_EXTENSION}")
# pybind requires binding code to be compiled with -fvisibility=hidden
# Better code can be generated if the entire project compiles that way, but
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
# that is not enforced here.
set_target_properties(NPCOMPNativePyExt PROPERTIES CXX_VISIBILITY_PRESET "hidden")
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
get_property(dialect_libs GLOBAL PROPERTY MLIR_DIALECT_LIBS)
get_property(conversion_libs GLOBAL PROPERTY MLIR_CONVERSION_LIBS)
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
target_link_libraries(NPCOMPNativePyExt
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
PRIVATE
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
# Transitive dep on the shared library links most things from there.
NPCOMP
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
NPCOMPInitAll
NPCOMPPythonCommon
Collapse python_native/ into python/. * These were separated originally for layering reasons that no longer apply. * Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-04 08:46:34 +08:00
${NPCOMP_PYEXT_LIBADD}
)
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
npcomp_python_target_compile_options(NPCOMPNativePyExt)
[cmake] Updates for basic shared library support (#7) Mostly this is CMake cleanup. Several library dependencies are missing, which is often revealed with shared library builds. Also, it's generally bad to link directly against LLVM libraries because it fails when using LLVM_LINK_LLVM_DYLIB. MLIR will pull in libLLVM.so, and there will be duplicate linkage with the the explicit libraries. There may need to be more refactoring here.
2020-08-06 05:49:18 +08:00
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
mlir_check_all_link_libraries(NPCOMPNativePyExt)