llvm
/
torch-mlir
mirror of https://github.com/llvm/torch-mlir
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
torch-mlir/CMakeLists.txt

216 lines
8.6 KiB
CMake
Raw Normal View History

Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
cmake_minimum_required(VERSION 3.13.4)
Adapt to use installed MLIR
2020-04-27 07:26:45 +08:00
Deal with CMP0116
2021-08-12 15:10:54 +08:00
# CMP0116: Ninja generators transform `DEPFILE`s from `add_custom_command()`
# New in CMake 3.20. https://cmake.org/cmake/help/latest/policy/CMP0116.html
if(POLICY CMP0116)
cmake_policy(SET CMP0116 OLD)
endif()
Adapt to use installed MLIR
2020-04-27 07:26:45 +08:00
if(POLICY CMP0068)
cmake_policy(SET CMP0068 NEW)
set(CMAKE_BUILD_WITH_INSTALL_NAME_DIR ON)
endif()
if(POLICY CMP0075)
cmake_policy(SET CMP0075 NEW)
endif()
if(POLICY CMP0077)
cmake_policy(SET CMP0077 NEW)
endif()
Add initial support for taking a dep on IREE. * This won't work for most people until some upstream changes percolate. * Sequestered it behind a flag and a special configure script (cmake_configure_iree.sh) for now.
2020-06-12 07:40:31 +08:00
#-------------------------------------------------------------------------------
# Options and settings
#-------------------------------------------------------------------------------
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
set(NPCOMP_MINIMUM_PYTHON_VERSION 3.6)
Add initial support for taking a dep on IREE. * This won't work for most people until some upstream changes percolate. * Sequestered it behind a flag and a special configure script (cmake_configure_iree.sh) for now.
2020-06-12 07:40:31 +08:00
Tweak build flags for efficiency and document building without a container. * Enables -gsplit-dwarf for both LLVM and NPCOMP, reducing the occurrence of the ~GB scale binaries. * CMake shared linking seems incompatible with this, so shared objects are still "too big" but there are few of them. * Reduces disk thrash on clean/install of everything.
2020-11-04 05:46:46 +08:00
# Turn on -gsplit-dwarf if requested in debug builds.
if (NPCOMP_USE_SPLIT_DWARF AND
((CMAKE_BUILD_TYPE STREQUAL "Debug") OR
(CMAKE_BUILD_TYPE STREQUAL "RelWithDebInfo")))
# Limit to clang and gcc so far. Add compilers supporting this option.
if (CMAKE_CXX_COMPILER_ID MATCHES "Clang" OR
CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
add_compile_options(-gsplit-dwarf)
endif()
endif()
Some fixes to get npcomp building and passing on windows. There is more that can be done here, but this gets it minimally working.
2020-07-02 12:28:04 +08:00
#-------------------------------------------------------------------------------
# MSVC defaults
#-------------------------------------------------------------------------------
if(MSVC)
add_compile_options(
$<$<CONFIG:>:/MD>
$<$<CONFIG:Debug>:/MD>
$<$<CONFIG:Release>:/MD>
)
endif()
[torch-mlir earthmoving (2/N)] Python code movement. This moves the bulk of the Python code (including the Torch interop) from `frontends/pytorch` into `torch-mlir/TorchPlugin`. This also required reconciling a bunch of other Python-related stuff, like the `torch` dialects. As I did this, it was simpler to just remove all the old numpy/basicpy stuff because we were going to delete it anyway and it was faster than debugging an intermediate state that would only last O(days) anyway. torch-mlir has two top-level python packages (built into the `python_packages` directory): - `torch_mlir_dialects`: `torch` dialect Python bindings (does not depend on PyTorch). This also involves building the aggregate CAPI for `torch-mlir`. - `torch_mlir`: bindings to the part of the code that links against PyTorch (or C++ code that transitively does). Additionally, there remain two more Python packages in npcomp (but outside `torch-mlir`): - `npcomp_torch`: Contains the e2e test framework and testing configs that plug into RefBackend and IREE. - `npcomp_core`: Contains the low-level interfaces to RefBackend and IREE that `npcomp_torch` uses, along with its own `MLIR_PYTHON_PACKAGE_PREFIX=npcomp.` aggregation of the core MLIR python bindings. (all other functionality has been stripped out) After all the basicpy/numpy deletions, the `npcomp` C++ code is now very tiny. It basically just contains RefBackend and the `TorchConversion` dialect/passes (e.g. `TorchToLinalg.cpp`). Correspondingly, there are now 4 main testing targets paralleling the Python layering (which is reflective of the deeper underlying dependency structure) - `check-torch-mlir`: checks the `torch-mlir` pure MLIR C++ code. - `check-torch-mlir-plugin`: checks the code in `TorchPlugin` (e.g. TorchScript import) - `check-frontends-pytorch`: Checks the little code we have in `frontends/pytorch` -- mainly things related to the e2e framework itself. - `check-npcomp`: Checks the pure MLIR C++ code inside npcomp. There is a target `check-npcomp-all` that runs all of them. The `torch-mlir/build_standalone.sh` script does a standalone build of `torch-mlir`. The e2e tests (`tools/torchscript_e2e_test.sh`) are working too. The update_torch_ods script now lives in `torch-mlir/build_tools/update_torch_ods.sh` and expects a standalone build. This change also required a fix upstream related to cross-shlib Python dependencies, so we also update llvm-project to 8dca953dd39c0cd8c80decbeb38753f58a4de580 to get https://reviews.llvm.org/D109776 (no other fixes were needed for the integrate, thankfully). This completes most of the large source code changes. Next will be bringing the CI/packaging/examples back to life.
2021-09-11 02:44:38 +08:00
#-------------------------------------------------------------------------------
# Directory setup
#-------------------------------------------------------------------------------
set(MLIR_NPCOMP_SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR})
set(MLIR_NPCOMP_BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR})
set(MLIR_NPCOMP_PYTHON_PACKAGES_DIR ${CMAKE_CURRENT_BINARY_DIR}/python_packages)
set(TORCH_MLIR_PYTHON_PACKAGES_DIR "${MLIR_NPCOMP_PYTHON_PACKAGES_DIR}")
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
if(CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR)
Enable building using LLVM_EXTERNAL_PROJECTS. (#152) This allows building NPCOMP as an external project of LLVM, similar to how CIRCT can be built: https://github.com/llvm/circt/pull/227. The CMake options to use this build style look like this: ``` -DLLVM_EXTERNAL_PROJECTS=npcomp \ -DLLVM_EXTERNAL_NPCOMP_SOURCE_DIR=/path/to/mlir-npcomp \ ```
2021-01-27 02:43:43 +08:00
# If we are not building as a part of LLVM, build NPCOMP as a standalone
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
# project, including LLVM as a subdirectory. This gives us the most control
# and is used for standalone releases.
Enable building using LLVM_EXTERNAL_PROJECTS. (#152) This allows building NPCOMP as an external project of LLVM, similar to how CIRCT can be built: https://github.com/llvm/circt/pull/227. The CMake options to use this build style look like this: ``` -DLLVM_EXTERNAL_PROJECTS=npcomp \ -DLLVM_EXTERNAL_NPCOMP_SOURCE_DIR=/path/to/mlir-npcomp \ ```
2021-01-27 02:43:43 +08:00
#-------------------------------------------------------------------------------
# Project setup and globals
#-------------------------------------------------------------------------------
project(npcomp LANGUAGES CXX C)
set(CMAKE_C_STANDARD 11)
set(CMAKE_CXX_STANDARD 14)
Adapt to use installed MLIR
2020-04-27 07:26:45 +08:00
Enable building using LLVM_EXTERNAL_PROJECTS. (#152) This allows building NPCOMP as an external project of LLVM, similar to how CIRCT can be built: https://github.com/llvm/circt/pull/227. The CMake options to use this build style look like this: ``` -DLLVM_EXTERNAL_PROJECTS=npcomp \ -DLLVM_EXTERNAL_NPCOMP_SOURCE_DIR=/path/to/mlir-npcomp \ ```
2021-01-27 02:43:43 +08:00
#-------------------------------------------------------------------------------
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
# Default and required options.
Enable building using LLVM_EXTERNAL_PROJECTS. (#152) This allows building NPCOMP as an external project of LLVM, similar to how CIRCT can be built: https://github.com/llvm/circt/pull/227. The CMake options to use this build style look like this: ``` -DLLVM_EXTERNAL_PROJECTS=npcomp \ -DLLVM_EXTERNAL_NPCOMP_SOURCE_DIR=/path/to/mlir-npcomp \ ```
2021-01-27 02:43:43 +08:00
#-------------------------------------------------------------------------------
[cmake] Updates for basic shared library support (#7) Mostly this is CMake cleanup. Several library dependencies are missing, which is often revealed with shared library builds. Also, it's generally bad to link directly against LLVM libraries because it fails when using LLVM_LINK_LLVM_DYLIB. MLIR will pull in libLLVM.so, and there will be duplicate linkage with the the explicit libraries. There may need to be more refactoring here.
2020-08-06 05:49:18 +08:00
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
# CMake library generation settings.
set(BUILD_SHARED_LIBS OFF CACHE BOOL "We are actually building a static mondo-lib")
set(CMAKE_PLATFORM_NO_VERSIONED_SONAME ON CACHE BOOL "Python soname linked libraries are bad")
set(CMAKE_VISIBILITY_INLINES_HIDDEN ON CACHE BOOL "Hide inlines")
set(CMAKE_CXX_VISIBILITY_PRESET hidden)
# Improved LLVM defaults for a standalone build.
set(LLVM_ENABLE_PROJECTS mlir CACHE STRING "LLVM projects")
set(LLVM_ENABLE_Z3_SOLVER OFF CACHE BOOL "Disable Z3")
set(LLVM_ENABLE_ZLIB OFF CACHE BOOL "Disable ZLIB")
set(LLVM_TARGETS_TO_BUILD "host" CACHE STRING "Only build for the host")
set(LLVM_INCLUDE_EXAMPLES OFF CACHE BOOL "Disable examples")
# TODO: MLIR is a "tool"
set(LLVM_INCLUDE_TOOLS ON CACHE BOOL "Disable tools")
set(LLVM_INCLUDE_TESTS ON CACHE BOOL "Disable tests")
set(MLIR_BINDINGS_PYTHON_LOCK_VERSION ON CACHE BOOL "Link against libpython for development (should be disabled for production)")
# Required LLVM settings.
set(MLIR_ENABLE_BINDINGS_PYTHON ON CACHE BOOL "Enable MLIR python bindings" FORCE)
#-------------------------------------------------------------------------------
# MLIR/LLVM Build Setup
# TODO: It would be nice to have a better setup than this for sub including
# MLIR.
#-------------------------------------------------------------------------------
if(NOT LLVM_MAIN_SRC_DIR)
set(LLVM_MAIN_SRC_DIR "${CMAKE_CURRENT_SOURCE_DIR}/external/llvm-project/llvm")
endif()
if(NOT MLIR_MAIN_SRC_DIR)
set(MLIR_MAIN_SRC_DIR "${LLVM_MAIN_SRC_DIR}/../mlir")
endif()
[cmake] Updates for basic shared library support (#7) Mostly this is CMake cleanup. Several library dependencies are missing, which is often revealed with shared library builds. Also, it's generally bad to link directly against LLVM libraries because it fails when using LLVM_LINK_LLVM_DYLIB. MLIR will pull in libLLVM.so, and there will be duplicate linkage with the the explicit libraries. There may need to be more refactoring here.
2020-08-06 05:49:18 +08:00
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
set(LLVM_MAIN_BINARY_DIR "${CMAKE_CURRENT_BINARY_DIR}/llvm")
set(MLIR_MAIN_BINARY_DIR "${LLVM_MAIN_BINARY_DIR}/tools/mlir")
set(LLVM_INCLUDE_DIR "${LLVM_MAIN_SRC_DIR}/include")
set(LLVM_GENERATED_INCLUDE_DIR "${LLVM_MAIN_BINARY_DIR}/include")
set(MLIR_INCLUDE_DIR "${MLIR_MAIN_SRC_DIR}/include")
set(MLIR_GENERATED_INCLUDE_DIR "${MLIR_MAIN_BINARY_DIR}/include")
set(MLIR_TABLEGEN_EXE "$<TARGET_FILE:mlir-tblgen>")
include_directories(SYSTEM "${LLVM_INCLUDE_DIR}")
include_directories(SYSTEM "${LLVM_GENERATED_INCLUDE_DIR}")
include_directories(SYSTEM "${MLIR_INCLUDE_DIR}")
include_directories(SYSTEM "${MLIR_GENERATED_INCLUDE_DIR}")
list(APPEND CMAKE_MODULE_PATH "${MLIR_MAIN_SRC_DIR}/cmake/modules")
list(APPEND CMAKE_MODULE_PATH "${LLVM_MAIN_SRC_DIR}/cmake/modules")
# Pre-configure the Python environment using the MLIR macros so that they
# are in scope and subsequent include of LLVM will match them.
include(MLIRDetectPythonEnv)
find_package(Python3 ${NPCOMP_MINIMUM_PYTHON_VERSION}
COMPONENTS Interpreter Development NumPy REQUIRED)
message(STATUS "Found python include dirs: ${Python3_INCLUDE_DIRS}")
message(STATUS "Found python libraries: ${Python3_LIBRARIES}")
mlir_detect_pybind11_install()
find_package(pybind11 2.6 CONFIG REQUIRED)
message(STATUS "Found pybind11 v${pybind11_VERSION}: ${pybind11_INCLUDE_DIR}")
message(STATUS "Python prefix = '${PYTHON_MODULE_PREFIX}', "
"suffix = '${PYTHON_MODULE_SUFFIX}', "
"extension = '${PYTHON_MODULE_EXTENSION}")
[torch-mlir earthmoving (1/N)] C/C++ code movement. This creates the `external/torch-mlir` directory as an LLVM_EXTERNAL_PROJECTS-compatible project (analogous to `iree-dialects`) and completes movement/rename of all pure MLIR C/C++ compiler code into there. The next step will be to move all the Python code / code that links/includes PyTorch C++ code (which currently lives in `frontends/pytorch`) into a subdirectory here. I call this "earthmoving" because it is mostly mechanical changes and renames. As a quick summary (we can change this down the road easily) - C++ `mlir::NPCOMP::Torch -> mlir::torch::Torch` - CAPI `npcompTorchListTypeGet -> torchMlirTorchListTypeGet` - preprocessor `#ifndef NPCOMP_ -> #ifndef TORCHMLIR_` - CMake `NPCOMPFoo -> TorchMLIRFoo` The goal of this is to create a standalone project creating a center of mass for entry into the MLIR ecosystem from PyTorch, suitable in scope for eventual inclusion/ownership in PyTorch. The idea is that `external/torch-mlir` will some day be pulled out into its own repository, and then npcomp will simply pull it in as a submodule. Layering-wise, what lives in `torch-mlir` lowers code from PyTorch (currently TorchScript, but TorchFX or pytorch/xla-style tracing are possible extensions) down to what we have been calling the "Torch backend contract" which is cleaned up IR (inlining, simplifcation, conversion to value tensors, ...) entirely in the `torch` dialect. This is the branching off point for further lowering, of which npcomp takes one opinion (outside `torch-mlir` of course!), namely the `TorchConversion` dialect/transforms which lower to IR suitable for IREE and other linalg-on-tensors based lower-level compilers. Summary of changes: - move `{include,lib,test}/Dialect/Torch` into `torch-mlir` - move relevant parts of CAPI into `torch-mlir`. - leave a few things related to the `torch-mlir` Python build commented out, which should be resolved in a subsequent change.
2021-09-10 03:24:10 +08:00
# Include LLVM_EXTERNAL_PROJECTS.
Eliminate almost all mentions of IREE. A few remain in examples/docs that will be naturally be updated in due time. This regresses the list support and the general direction of more widely supported control flow, lists/dicts/globals that we were going for with the TorchScript path. The idea is that we are deferring that work to make torch-mlir a very clean standalone thing. We will reboot it, probably using some of the tools of iree_pydm to make it simpler, and in a more natural place (such as an iree-torch repo that depends on IREE and torch-mlir to build a working PyTorch frontend solution for IREE -- it was really weird that npcomp depended on IREE).
2021-09-23 06:48:39 +08:00
set(LLVM_EXTERNAL_PROJECTS "torch-mlir")
[torch-mlir earthmoving (1/N)] C/C++ code movement. This creates the `external/torch-mlir` directory as an LLVM_EXTERNAL_PROJECTS-compatible project (analogous to `iree-dialects`) and completes movement/rename of all pure MLIR C/C++ compiler code into there. The next step will be to move all the Python code / code that links/includes PyTorch C++ code (which currently lives in `frontends/pytorch`) into a subdirectory here. I call this "earthmoving" because it is mostly mechanical changes and renames. As a quick summary (we can change this down the road easily) - C++ `mlir::NPCOMP::Torch -> mlir::torch::Torch` - CAPI `npcompTorchListTypeGet -> torchMlirTorchListTypeGet` - preprocessor `#ifndef NPCOMP_ -> #ifndef TORCHMLIR_` - CMake `NPCOMPFoo -> TorchMLIRFoo` The goal of this is to create a standalone project creating a center of mass for entry into the MLIR ecosystem from PyTorch, suitable in scope for eventual inclusion/ownership in PyTorch. The idea is that `external/torch-mlir` will some day be pulled out into its own repository, and then npcomp will simply pull it in as a submodule. Layering-wise, what lives in `torch-mlir` lowers code from PyTorch (currently TorchScript, but TorchFX or pytorch/xla-style tracing are possible extensions) down to what we have been calling the "Torch backend contract" which is cleaned up IR (inlining, simplifcation, conversion to value tensors, ...) entirely in the `torch` dialect. This is the branching off point for further lowering, of which npcomp takes one opinion (outside `torch-mlir` of course!), namely the `TorchConversion` dialect/transforms which lower to IR suitable for IREE and other linalg-on-tensors based lower-level compilers. Summary of changes: - move `{include,lib,test}/Dialect/Torch` into `torch-mlir` - move relevant parts of CAPI into `torch-mlir`. - leave a few things related to the `torch-mlir` Python build commented out, which should be resolved in a subsequent change.
2021-09-10 03:24:10 +08:00
set(LLVM_EXTERNAL_TORCH_MLIR_SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/external/torch-mlir")
Add TorchToIREE and factor out TorchConversion dialect. This converts a basic list op (torch.prim.ListConstruct) to the IREE dialect. ``` def forward(self, x: float): return [x, x] ``` turns into: ``` builtin.func @forward(%arg0: !torch.float) -> !torch.list<!torch.float> { %0 = torch.prim.ListConstruct %arg0, %arg0 : (!torch.float, !torch.float) -> !torch.list<!torch.float> return %0 : !torch.list<!torch.float> } ``` which turns into: ``` builtin.func @forward(%arg0: f64) -> !iree.list<f64> { %c1 = constant 1 : index %c0 = constant 0 : index %c2 = constant 2 : index %0 = iree.list.create %c2 : !iree.list<f64> iree.list.set %0[%c0], %arg0 : !iree.list<f64>, f64 iree.list.set %0[%c1], %arg0 : !iree.list<f64>, f64 return %0 : !iree.list<f64> } ``` As part of doing this, I realized that it was time to formalize the IR form that we reach right before running TorchTo{Linalg,Std,...}. We now call it the "Torch backend contract". We then lower the "Torch backend contract" to the "npcomp backend contract", which involves the new TorchConversion (`torch_c`) dialect, which holds ops that need to operate on both the npcomp backend types (e.g. builtin tensors, i1, IREE list, etc.) and the `!torch` types. This made more sense, as I realized that if I didn't factor out `torch_c` then the Torch dialect would have a dependency on IREE dialect (we previously didn't notice this was an issue because we only depended on `builtin` types), which seemed wrong to me. Recommended review order: - TorchToIREE.cpp / `TorchToIREE/basic.mlir` - Look at the new structure of createTorchScriptToNpcompBackendPipeline. It now lives in TorchConversion/Transforms/Passes.cpp and cleanly calls into `Torch::createTorchScriptToTorchBackendPipeline` for the frontend lowering to the Torch backend contract. - Mechanical change extracting `torch_c.{to,from}_{i1,i64,f64,builtin_tensor,iree_list}` into a new TorchConversion dialect, and a few passes specific to the lowering from the Torch backend contract to the npcomp backend contract. - Minor fixes to TorchToLinalg.cpp to use unconverted operands (now that we convert lists as part of operand materialization, we need to use the original operands). Also added test for AtenMaxPool2dOp and fixed m_TorchConstantIntList. - TmpDeleteDeadIREELists pass. Temporary pass for deleting dead IREE lists that are created as part of operand materialization for conv/max pool/avg pool ops in TorchToLinalg.
2021-08-12 05:40:08 +08:00
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
# LLVM configuration.
message(STATUS "*** ADDING LLVM ***")
add_subdirectory(
"${CMAKE_CURRENT_SOURCE_DIR}/external/llvm-project/llvm"
"${LLVM_MAIN_BINARY_DIR}"
EXCLUDE_FROM_ALL)
message(STATUS "*** LLVM DONE ***")
set(LLVM_RUNTIME_OUTPUT_INTDIR ${CMAKE_BINARY_DIR}/bin)
Enable building using LLVM_EXTERNAL_PROJECTS. (#152) This allows building NPCOMP as an external project of LLVM, similar to how CIRCT can be built: https://github.com/llvm/circt/pull/227. The CMake options to use this build style look like this: ``` -DLLVM_EXTERNAL_PROJECTS=npcomp \ -DLLVM_EXTERNAL_NPCOMP_SOURCE_DIR=/path/to/mlir-npcomp \ ```
2021-01-27 02:43:43 +08:00
set(LLVM_LIBRARY_OUTPUT_INTDIR ${CMAKE_BINARY_DIR}/lib)
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
set(LLVM_EXTERNAL_LIT "${LLVM_MAIN_BINARY_DIR}/bin/llvm-lit")
set(LLVM_TOOLS_DIR "${LLVM_MAIN_BINARY_DIR}/bin")
Enable building using LLVM_EXTERNAL_PROJECTS. (#152) This allows building NPCOMP as an external project of LLVM, similar to how CIRCT can be built: https://github.com/llvm/circt/pull/227. The CMake options to use this build style look like this: ``` -DLLVM_EXTERNAL_PROJECTS=npcomp \ -DLLVM_EXTERNAL_NPCOMP_SOURCE_DIR=/path/to/mlir-npcomp \ ```
2021-01-27 02:43:43 +08:00
# Define the default arguments to use with 'lit', and an option for the user to
# override.
set(LIT_ARGS_DEFAULT "-sv")
if (MSVC_IDE OR XCODE)
set(LIT_ARGS_DEFAULT "${LIT_ARGS_DEFAULT} --no-progress-bar")
endif()
set(LLVM_LIT_ARGS "${LIT_ARGS_DEFAULT}" CACHE STRING "Default options for lit")
include(TableGen)
include(AddLLVM)
include(AddMLIR)
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
include(AddMLIRPython)
Enable building using LLVM_EXTERNAL_PROJECTS. (#152) This allows building NPCOMP as an external project of LLVM, similar to how CIRCT can be built: https://github.com/llvm/circt/pull/227. The CMake options to use this build style look like this: ``` -DLLVM_EXTERNAL_PROJECTS=npcomp \ -DLLVM_EXTERNAL_NPCOMP_SOURCE_DIR=/path/to/mlir-npcomp \ ```
2021-01-27 02:43:43 +08:00
include(HandleLLVMOptions)
set(NPCOMP_BUILT_STANDALONE 1)
else()
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
# TODO: RE-ENABLE EXTERNAL UNIFIED BUILD
message(FATAL_ERROR "External project builds of npcomp are currently not available")
Enable building using LLVM_EXTERNAL_PROJECTS. (#152) This allows building NPCOMP as an external project of LLVM, similar to how CIRCT can be built: https://github.com/llvm/circt/pull/227. The CMake options to use this build style look like this: ``` -DLLVM_EXTERNAL_PROJECTS=npcomp \ -DLLVM_EXTERNAL_NPCOMP_SOURCE_DIR=/path/to/mlir-npcomp \ ```
2021-01-27 02:43:43 +08:00
# Otherwise, we are building as a part of LLVM, and we need to set up some
# variables and includes.
Rework the python build to a static assembly of MLIR+NPCOMP (#251) * Adapt to python build system updates. * Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates) * Adds refback C-API. * Re-layers all python builds. * Rework CI.
2021-07-28 07:10:10 +08:00
# set(MLIR_MAIN_SRC_DIR ${LLVM_MAIN_SRC_DIR}/../mlir/include ) # --src-root
# set(MLIR_INCLUDE_DIR ${LLVM_MAIN_SRC_DIR}/../mlir/include ) # --includedir
# set(MLIR_TABLEGEN_OUTPUT_DIR ${CMAKE_BINARY_DIR}/tools/mlir/include)
# set(MLIR_TABLEGEN_EXE $<TARGET_FILE:mlir-tblgen>)
# include_directories(SYSTEM ${MLIR_INCLUDE_DIR})
# include_directories(SYSTEM ${MLIR_TABLEGEN_OUTPUT_DIR})
# set(BACKEND_PACKAGE_STRING "${PACKAGE_STRING}")
[cmake] Updates for basic shared library support (#7) Mostly this is CMake cleanup. Several library dependencies are missing, which is often revealed with shared library builds. Also, it's generally bad to link directly against LLVM libraries because it fails when using LLVM_LINK_LLVM_DYLIB. MLIR will pull in libLLVM.so, and there will be duplicate linkage with the the explicit libraries. There may need to be more refactoring here.
2020-08-06 05:49:18 +08:00
endif()
Enable building using LLVM_EXTERNAL_PROJECTS. (#152) This allows building NPCOMP as an external project of LLVM, similar to how CIRCT can be built: https://github.com/llvm/circt/pull/227. The CMake options to use this build style look like this: ``` -DLLVM_EXTERNAL_PROJECTS=npcomp \ -DLLVM_EXTERNAL_NPCOMP_SOURCE_DIR=/path/to/mlir-npcomp \ ```
2021-01-27 02:43:43 +08:00
list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake/modules")
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
include(AddNPCOMP)
Adapt to use installed MLIR
2020-04-27 07:26:45 +08:00
include_directories(${LLVM_INCLUDE_DIRS})
include_directories(${MLIR_INCLUDE_DIRS})
Enable building using LLVM_EXTERNAL_PROJECTS. (#152) This allows building NPCOMP as an external project of LLVM, similar to how CIRCT can be built: https://github.com/llvm/circt/pull/227. The CMake options to use this build style look like this: ``` -DLLVM_EXTERNAL_PROJECTS=npcomp \ -DLLVM_EXTERNAL_NPCOMP_SOURCE_DIR=/path/to/mlir-npcomp \ ```
2021-01-27 02:43:43 +08:00
include_directories(${CMAKE_CURRENT_SOURCE_DIR}/include)
include_directories(${CMAKE_CURRENT_BINARY_DIR}/include)
[torch-mlir earthmoving (1/N)] C/C++ code movement. This creates the `external/torch-mlir` directory as an LLVM_EXTERNAL_PROJECTS-compatible project (analogous to `iree-dialects`) and completes movement/rename of all pure MLIR C/C++ compiler code into there. The next step will be to move all the Python code / code that links/includes PyTorch C++ code (which currently lives in `frontends/pytorch`) into a subdirectory here. I call this "earthmoving" because it is mostly mechanical changes and renames. As a quick summary (we can change this down the road easily) - C++ `mlir::NPCOMP::Torch -> mlir::torch::Torch` - CAPI `npcompTorchListTypeGet -> torchMlirTorchListTypeGet` - preprocessor `#ifndef NPCOMP_ -> #ifndef TORCHMLIR_` - CMake `NPCOMPFoo -> TorchMLIRFoo` The goal of this is to create a standalone project creating a center of mass for entry into the MLIR ecosystem from PyTorch, suitable in scope for eventual inclusion/ownership in PyTorch. The idea is that `external/torch-mlir` will some day be pulled out into its own repository, and then npcomp will simply pull it in as a submodule. Layering-wise, what lives in `torch-mlir` lowers code from PyTorch (currently TorchScript, but TorchFX or pytorch/xla-style tracing are possible extensions) down to what we have been calling the "Torch backend contract" which is cleaned up IR (inlining, simplifcation, conversion to value tensors, ...) entirely in the `torch` dialect. This is the branching off point for further lowering, of which npcomp takes one opinion (outside `torch-mlir` of course!), namely the `TorchConversion` dialect/transforms which lower to IR suitable for IREE and other linalg-on-tensors based lower-level compilers. Summary of changes: - move `{include,lib,test}/Dialect/Torch` into `torch-mlir` - move relevant parts of CAPI into `torch-mlir`. - leave a few things related to the `torch-mlir` Python build commented out, which should be resolved in a subsequent change.
2021-09-10 03:24:10 +08:00
include_directories(${CMAKE_CURRENT_SOURCE_DIR}/external/torch-mlir/include)
include_directories(${CMAKE_CURRENT_BINARY_DIR}/llvm/tools/torch-mlir/include)
Adapt to use installed MLIR
2020-04-27 07:26:45 +08:00
link_directories(${LLVM_BUILD_LIBRARY_DIR})
add_definitions(${LLVM_DEFINITIONS})
Basicpy conversion to IREE+std skeleton and first conversions. * Conversions to std for numeric binary expressions, numeric to_boolean, and numeric comparisons. * Added folders to constant ops to comply with requirements of the pass system. * Extended the frontend with parameter/result annotation processing for primitives (can specify types for function arguments). * Added (empty) directory/sources for IREEVM conversions. These are only enabled if IREE is enabled.
2020-06-14 14:45:43 +08:00
set(NPCOMP_TABLEGEN_ARGS "")
Adapt to use installed MLIR
2020-04-27 07:26:45 +08:00
Initial python plumbing to interface with the refjit backend.
2020-07-11 08:36:32 +08:00
#-------------------------------------------------------------------------------
# Optional feature selection
#-------------------------------------------------------------------------------
Add initial support for taking a dep on IREE. * This won't work for most people until some upstream changes percolate. * Sequestered it behind a flag and a special configure script (cmake_configure_iree.sh) for now.
2020-06-12 07:40:31 +08:00
#-------------------------------------------------------------------------------
Remove NPCOMP_ENABLE_IREE CMake flag. Our new dependency management solution relies: - on the C++ side with the public iree-dialects project, which we include and are using as representative of some missing upstream ops (so we treat them "as if" they were upstream, with the hope of upstreaming them after some codevelopment has happened) - on the Python side, with simple PYTHONPATH manipulation or installed Python packages. No CMake stuff required.
2021-09-17 10:00:24 +08:00
# Subdirectories and aggregate testing targets.
Add initial support for taking a dep on IREE. * This won't work for most people until some upstream changes percolate. * Sequestered it behind a flag and a special configure script (cmake_configure_iree.sh) for now.
2020-06-12 07:40:31 +08:00
#-------------------------------------------------------------------------------
Aggregate all lit test targets under check-npcomp.
2020-06-08 05:35:58 +08:00
add_custom_target(check-npcomp)
Enable building using LLVM_EXTERNAL_PROJECTS. (#152) This allows building NPCOMP as an external project of LLVM, similar to how CIRCT can be built: https://github.com/llvm/circt/pull/227. The CMake options to use this build style look like this: ``` -DLLVM_EXTERNAL_PROJECTS=npcomp \ -DLLVM_EXTERNAL_NPCOMP_SOURCE_DIR=/path/to/mlir-npcomp \ ```
2021-01-27 02:43:43 +08:00
add_custom_target(check-npcomp-all)
Remove frontends/pytorch directory. It just contained the e2e testing framework. We now fold it into the main project to reduce complexity. - `frontends/pytorch/python/` -> `python/torch_support` - `frontends/pytorch/e2e_testing -> e2e_testing` - `frontends/pytorch/examples -> examples` - `frontends/pytorch/test` -> `python/test` - `torch_mlir_torchscript` python module -> `npcomp_torchscript` - `torch_mlir_torchscript_e2e_test_configs` python module -> `npcomp_torchscript_e2e_test_configs` This also changes the license of a handful of files from the "pytorch-style" license to the regular LLVM/npcomp license. The only people who committed to those files were myself and Yi.
2021-09-17 06:18:17 +08:00
add_dependencies(check-npcomp-all
check-npcomp
check-npcomp-python
check-torch-mlir
)
Aggregate all lit test targets under check-npcomp.
2020-06-08 05:35:58 +08:00
Initial commit of python boiler-plate.
2020-04-27 06:50:23 +08:00
add_subdirectory(include/npcomp)
add_subdirectory(lib)
Misc fixes to enable building/testing on manylinux2014 images. * Since the manylinux images do not hard-link against python libs (resolving them at runtime), the module must be built without resolving undefined references. * For some reason, builds on this platform are stricter, exposing dependency ordering issues. * The CMake bits to build the extension are still somewhat of a mess. I have better versions both upstream and in IREE and will be reconciling. For now, don't look too closely.
2020-08-05 08:10:14 +08:00
add_subdirectory(python)
Add npcomp-opt and lit runner.
2020-04-27 08:55:15 +08:00
add_subdirectory(test)
Start reworking towards a shared library build. * Need to have a dag of shared library deps in order to interop across python extensions (as presented in ODM). * Introduced add_npcomp_library and friends to mirror the MLIR setup. * Adds a libNPCOMP.so shared library. * Redirects tools and extensions to link against libNPCOMP.so (instead of static libs). * Moves all libraries to lib/, all binaries to bin/ and all python extensions to python/. The invariant is that the rpaths are setup to have a one level directory structure. * Reworks the _torch_mlir extension to build like the others (still need to come up with a consolidated rule to do this instead of open coded). * Includes an upstream version bump to pick up needed changes. Sizes with dynamic linking (stripped, release, asserts enabled): libNPCOMP.so: 43M (includes much of the underlying LLVM codegen deps) libMLIR.so: 31M _npcomp.so: 1.6M (python extension) _torch_mlir.so: 670K (python extension) npcomp-capi-ir-test: 6.3K npcomp-opt: 351K npcomp-run-mlir: 461K mnist-playground: 530K Still more can be done to normalize and optimize but this gets us structurally to the starting point.
2020-10-09 09:29:59 +08:00
add_subdirectory(tools)