torch-mlir/.github/workflows/buildAndTest.yml

88 lines
3.3 KiB
YAML
Raw Normal View History

name: Build and Test
on: [push, pull_request]
jobs:
build:
name: Build and Test (Release Asserts)
2020-08-08 01:32:02 +08:00
runs-on: ubuntu-20.04
steps:
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.9
- name: Get npcomp
uses: actions/checkout@v2
with:
submodules: 'true'
- name: Install python depends
2020-10-13 12:49:39 +08:00
run: |
python -m pip install -r $GITHUB_WORKSPACE/external/llvm-project/mlir/python/requirements.txt
- name: Install pytorch_nightly depends
run: |
2021-08-03 01:58:02 +08:00
python -m pip install --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html
- name: Install Ninja
uses: llvm/actions/install-ninja@55d844821959226fab4911f96f37071c1d4c3268
- name: Get Submodule Hash
id: get-submodule-hash
run: echo "::set-output name=hash::$(md5sum $(git submodule status))"
shell: bash
- name: Ccache for C++ compilation
uses: hendrikmuhs/ccache-action@4687d037e4d7cf725512d9b819137a3af34d39b3
with:
key: ${{ runner.os }}-clangreleaseasserts-${{ steps.get-submodule-hash.outputs.hash }}
- name: Build and Test npcomp (Assert)
run: |
cd $GITHUB_WORKSPACE
mkdir build
cd build
cmake .. -GNinja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_LINKER=lld \
-DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
-DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ \
-DPython3_EXECUTABLE=$(which python) \
-DLLVM_ENABLE_ASSERTIONS=ON \
[torch-mlir earthmoving (2/N)] Python code movement. This moves the bulk of the Python code (including the Torch interop) from `frontends/pytorch` into `torch-mlir/TorchPlugin`. This also required reconciling a bunch of other Python-related stuff, like the `torch` dialects. As I did this, it was simpler to just remove all the old numpy/basicpy stuff because we were going to delete it anyway and it was faster than debugging an intermediate state that would only last O(days) anyway. torch-mlir has two top-level python packages (built into the `python_packages` directory): - `torch_mlir_dialects`: `torch` dialect Python bindings (does not depend on PyTorch). This also involves building the aggregate CAPI for `torch-mlir`. - `torch_mlir`: bindings to the part of the code that links against PyTorch (or C++ code that transitively does). Additionally, there remain two more Python packages in npcomp (but outside `torch-mlir`): - `npcomp_torch`: Contains the e2e test framework and testing configs that plug into RefBackend and IREE. - `npcomp_core`: Contains the low-level interfaces to RefBackend and IREE that `npcomp_torch` uses, along with its own `MLIR_PYTHON_PACKAGE_PREFIX=npcomp.` aggregation of the core MLIR python bindings. (all other functionality has been stripped out) After all the basicpy/numpy deletions, the `npcomp` C++ code is now very tiny. It basically just contains RefBackend and the `TorchConversion` dialect/passes (e.g. `TorchToLinalg.cpp`). Correspondingly, there are now 4 main testing targets paralleling the Python layering (which is reflective of the deeper underlying dependency structure) - `check-torch-mlir`: checks the `torch-mlir` pure MLIR C++ code. - `check-torch-mlir-plugin`: checks the code in `TorchPlugin` (e.g. TorchScript import) - `check-frontends-pytorch`: Checks the little code we have in `frontends/pytorch` -- mainly things related to the e2e framework itself. - `check-npcomp`: Checks the pure MLIR C++ code inside npcomp. There is a target `check-npcomp-all` that runs all of them. The `torch-mlir/build_standalone.sh` script does a standalone build of `torch-mlir`. The e2e tests (`tools/torchscript_e2e_test.sh`) are working too. The update_torch_ods script now lives in `torch-mlir/build_tools/update_torch_ods.sh` and expects a standalone build. This change also required a fix upstream related to cross-shlib Python dependencies, so we also update llvm-project to 8dca953dd39c0cd8c80decbeb38753f58a4de580 to get https://reviews.llvm.org/D109776 (no other fixes were needed for the integrate, thankfully). This completes most of the large source code changes. Next will be bringing the CI/packaging/examples back to life.
2021-09-11 02:44:38 +08:00
-DLLVM_TARGETS_TO_BUILD=host
ninja check-npcomp-all
2021-07-30 13:50:52 +08:00
- name: Refbackend integration tests
run: |
cd $GITHUB_WORKSPACE
export PYTHONPATH="$GITHUB_WORKSPACE/build/python_packages/npcomp_core:$GITHUB_WORKSPACE/build/python_packages/npcomp_torch:$GITHUB_WORKSPACE/build/python_packages/torch_mlir:$GITHUB_WORKSPACE/build/python_packages/torch_mlir_dialects"
2021-07-30 13:50:52 +08:00
python -m frontends.pytorch.e2e_testing.torchscript.main --config=refbackend
# build_wheels:
# name: Build Wheels and Smoketest
# runs-on: ubuntu-20.04
# steps:
# - name: Set up Python
# uses: actions/setup-python@v2
# with:
# python-version: 3.9
# - name: Get npcomp
# uses: actions/checkout@v2
# with:
# submodules: 'true'
# - name: Install python dependencies
# run: |
# python -m pip install --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html
# python -m pip install wheel
# - name: Install Ninja
# uses: llvm/actions/install-ninja@55d844821959226fab4911f96f37071c1d4c3268
# - name: Get Submodule Hash
# id: get-submodule-hash
# run: echo "::set-output name=hash::$(md5sum $(git submodule status))"
# shell: bash
# - name: Ccache for C++ compilation
# uses: hendrikmuhs/ccache-action@4687d037e4d7cf725512d9b819137a3af34d39b3
# with:
# key: ${{ runner.os }}-wheel-${{ steps.get-submodule-hash.outputs.hash }}
# - name: Build wheels
# run: |
# cd $GITHUB_WORKSPACE
# ./build_tools/build_python_wheels.sh
# - uses: actions/upload-artifact@v2
# with:
# name: ubuntu-20.04-wheels
# path: wheelhouse/