mirror of https://github.com/llvm/torch-mlir
Remove build automation that is no longer used.
Most of this can't actually work given the token/security issues, so even if we re-introduce it, we are better off fetching from history vs carrying things that will lead folks astray.rm_obsolete_build_automation
Stella Laurenzo
parent
105aad6f57
commit
cb8ec30070
|
|
@ -1,146 +0,0 @@
|
||||||
# yamllint disable rule:line-length
|
|
||||||
name: Roll PyTorch
|
|
||||||
|
|
||||||
on:
|
|
||||||
schedule:
|
|
||||||
- cron: '0 13 * * *'
|
|
||||||
workflow_dispatch:
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
build_linux:
|
|
||||||
name: Manylinux Build
|
|
||||||
runs-on: torch-mlir-cpubuilder-manylinux-x86-64
|
|
||||||
|
|
||||||
# Don't run this in everyone's forks.
|
|
||||||
if: github.repository == 'llvm/torch-mlir'
|
|
||||||
|
|
||||||
steps:
|
|
||||||
- name: Prepare workspace
|
|
||||||
run: |
|
|
||||||
# Clear the workspace directory so that we don't run into errors about
|
|
||||||
# existing lock files.
|
|
||||||
sudo rm -rf $GITHUB_WORKSPACE/*
|
|
||||||
|
|
||||||
- name: Get torch-mlir
|
|
||||||
uses: actions/checkout@v3
|
|
||||||
with:
|
|
||||||
submodules: 'false'
|
|
||||||
token: ${{ secrets.WORKFLOW_INVOCATION_TOKEN }}
|
|
||||||
|
|
||||||
- name: Get LLVM and StableHlo submodules
|
|
||||||
run: |
|
|
||||||
set -eo pipefail
|
|
||||||
cd ${GITHUB_WORKSPACE}
|
|
||||||
|
|
||||||
# Fetching the submodules concurrently may cause problems, so we fetch
|
|
||||||
# them one after another.
|
|
||||||
rm -f .git/modules/externals/llvm-project/index.lock
|
|
||||||
rm -f .git/modules/externals/stablehlo/index.lock
|
|
||||||
git submodule update --init --recursive externals/llvm-project
|
|
||||||
git submodule update --init --recursive externals/stablehlo
|
|
||||||
|
|
||||||
- name: Setup ccache
|
|
||||||
uses: ./.github/actions/setup-build
|
|
||||||
with:
|
|
||||||
cache-suffix: 'rollPyTorch'
|
|
||||||
|
|
||||||
- name: Determine nightly PyTorch version
|
|
||||||
run: |
|
|
||||||
set -eo pipefail
|
|
||||||
|
|
||||||
cd ${GITHUB_WORKSPACE}
|
|
||||||
python -m pip install wheel
|
|
||||||
sudo apt-get install unzip
|
|
||||||
|
|
||||||
# Fetch the most recent nightly torchvision release
|
|
||||||
VISION_RELEASE=$(python -m pip index versions -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html --pre torchvision | grep "Available versions" | tr ' ' '\n' | grep "^[0-9]" | sort --version-sort --reverse | head -n1 | tr -d ',' | sed 's/\([^+]*\).*/\1/')
|
|
||||||
echo "Found torchvision release ${VISION_RELEASE}"
|
|
||||||
|
|
||||||
# Fetch the whl file associated with the nightly torchvision release
|
|
||||||
rm -f torch*.whl
|
|
||||||
python -m pip download -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html --pre "torchvision==${VISION_RELEASE}"
|
|
||||||
|
|
||||||
# Downloading the torchvision WHL also downloads the PyTorch WHL file
|
|
||||||
# Read the version from the downloaded whl file without extracting it
|
|
||||||
PT_RELEASE=$(unzip -p torch-*.whl 'torch-*/METADATA' | grep "^Version:" | awk '{ print $2 }' | sed 's/\([^+]*\).*/\1/')
|
|
||||||
echo "Found torch release ${PT_RELEASE}"
|
|
||||||
printf -- "-f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html\n--pre\ntorch==%s\n" "${PT_RELEASE}" > pytorch-requirements.txt
|
|
||||||
printf -- "-f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html\n--pre\ntorchvision==%s\n" "${VISION_RELEASE}" > torchvision-requirements.txt
|
|
||||||
|
|
||||||
# Read the commit hash from the downloaded whl file without extracting it
|
|
||||||
PT_HASH=$(unzip -p torch-"${PT_RELEASE}"*.whl torch/version.py | grep git_version | tail -1 | awk '{ print $3 }' | tr -d "'")
|
|
||||||
echo "Found torch commit hash ${PT_HASH}"
|
|
||||||
|
|
||||||
PT_HASH_CHANGED=0
|
|
||||||
echo "${PT_HASH}" | cmp - pytorch-hash.txt --quiet || PT_HASH_CHANGED=$?
|
|
||||||
echo "${PT_HASH}" > pytorch-hash.txt
|
|
||||||
rm torch-"${PT_RELEASE}"*.whl
|
|
||||||
|
|
||||||
# Write the release and hash to the environment file so that we can
|
|
||||||
# retrieve them when creating a PR
|
|
||||||
echo "PT_HASH=${PT_HASH}" >> ${GITHUB_ENV}
|
|
||||||
echo "PT_RELEASE=${PT_RELEASE}" >> ${GITHUB_ENV}
|
|
||||||
echo "PTVISION_RELEASE=${VISION_RELEASE}" >> ${GITHUB_ENV}
|
|
||||||
echo "PT_HASH_CHANGED=${PT_HASH_CHANGED}" >> ${GITHUB_ENV}
|
|
||||||
|
|
||||||
- name: Build and test (out-of-tree), also update ODS and abstract interpretation library
|
|
||||||
if: env.PT_HASH_CHANGED != '0'
|
|
||||||
run: |
|
|
||||||
cd ${GITHUB_WORKSPACE}
|
|
||||||
TM_PACKAGES="out-of-tree" TM_USE_PYTORCH_BINARY="OFF" \
|
|
||||||
TORCH_MLIR_SRC_PYTORCH_BRANCH="${{ env.PT_HASH }}" \
|
|
||||||
TORCH_MLIR_SRC_PYTORCH_RELEASE="${{ env.PT_RELEASE }}" \
|
|
||||||
TM_UPDATE_ODS_AND_ABSTRACT_INTERP_LIB="ON" \
|
|
||||||
./build_tools/python_deploy/build_linux_packages.sh
|
|
||||||
|
|
||||||
- name: Post issue comment on build failure
|
|
||||||
if: failure()
|
|
||||||
uses: peter-evans/create-or-update-comment@v2
|
|
||||||
with:
|
|
||||||
issue-number: 1690
|
|
||||||
body: |
|
|
||||||
The RollPyTorch action has failed. See [CI log](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}) for details.
|
|
||||||
|
|
||||||
The following information may come handy when fixing the code.
|
|
||||||
```
|
|
||||||
torch version: ${{ env.PT_RELEASE }}
|
|
||||||
torch commit hash: ${{ env.PT_HASH }}
|
|
||||||
torchvision version: ${{ env.PTVISION_RELEASE }}
|
|
||||||
```
|
|
||||||
|
|
||||||
- name: Update PyTorch Build Cache (if running on main branch)
|
|
||||||
if: github.ref_name == 'main'
|
|
||||||
id: cache-pytorch
|
|
||||||
uses: actions/cache@v3
|
|
||||||
with:
|
|
||||||
path: ${{ github.workspace }}/build_tools/python_deploy/wheelhouse
|
|
||||||
key: ${{ runner.os }}-pytorch-${{ env.PT_HASH }}
|
|
||||||
|
|
||||||
- name: Commit changes locally
|
|
||||||
if: env.PT_HASH_CHANGED != '0'
|
|
||||||
run: |
|
|
||||||
cd ${GITHUB_WORKSPACE}
|
|
||||||
git config user.email "torch-mlir@users.noreply.github.com"
|
|
||||||
git config user.name "Roll PyTorch Action"
|
|
||||||
git fetch --recurse-submodules=no
|
|
||||||
git checkout main
|
|
||||||
git pull origin main
|
|
||||||
|
|
||||||
- name: Create pull request
|
|
||||||
uses: peter-evans/create-pull-request@v5.0.1
|
|
||||||
with:
|
|
||||||
author: Roll PyTorch Action <torch-mlir@users.noreply.github.com>
|
|
||||||
branch: rollpytorch
|
|
||||||
body: |
|
|
||||||
torch version: ${{ env.PT_RELEASE }}
|
|
||||||
torch commit hash: ${{ env.PT_HASH }}
|
|
||||||
torchvision version: ${{ env.PTVISION_RELEASE }}
|
|
||||||
commit-message: |
|
|
||||||
update PyTorch version to ${{ env.PT_RELEASE }}
|
|
||||||
|
|
||||||
- torch version: ${{ env.PT_RELEASE }}
|
|
||||||
- torch commit hash: ${{ env.PT_HASH }}
|
|
||||||
- torchvision version: ${{ env.PTVISION_RELEASE }}
|
|
||||||
committer: Roll PyTorch Action <torch-mlir@users.noreply.github.com>
|
|
||||||
title: update PyTorch version to ${{ env.PT_RELEASE }}
|
|
||||||
token: ${{ secrets.ROLLPYTORCH_TOKEN0 }}
|
|
||||||
|
|
@ -1,158 +0,0 @@
|
||||||
# yamllint disable rule:line-length
|
|
||||||
name: Build and Test
|
|
||||||
|
|
||||||
on:
|
|
||||||
# pull_request:
|
|
||||||
# branches: [main]
|
|
||||||
# push:
|
|
||||||
# branches: [main]
|
|
||||||
# workflow_dispatch:
|
|
||||||
|
|
||||||
# Ensure that only a single job or workflow using the same
|
|
||||||
# concurrency group will run at a time. This would cancel
|
|
||||||
# any in-progress jobs in the same github workflow and github
|
|
||||||
# ref (e.g. refs/heads/main or refs/pull/<pr_number>/merge).
|
|
||||||
concurrency:
|
|
||||||
# A PR number if a pull request and otherwise the commit hash. This cancels
|
|
||||||
# queued and in-progress runs for the same PR (presubmit) or commit
|
|
||||||
# (postsubmit). The workflow name is prepended to avoid conflicts between
|
|
||||||
# different workflows.
|
|
||||||
group: ${{ github.workflow }}-${{ github.event.number || github.sha }}
|
|
||||||
cancel-in-progress: true
|
|
||||||
|
|
||||||
|
|
||||||
# Provisioned Jobs:
|
|
||||||
# ubuntu/docker - x86_64 - llvm in-tree - pytorch binary - build+test # most used dev flow and fastest signal
|
|
||||||
# ubuntu/docker - x86_64 - llvm out-of-tree - pytorch source - build+test # most elaborate build
|
|
||||||
# macos - arm64 - llvm in-tree - pytorch binary - build only # cross compile, can't test arm64
|
|
||||||
jobs:
|
|
||||||
build-test:
|
|
||||||
strategy:
|
|
||||||
fail-fast: true
|
|
||||||
matrix:
|
|
||||||
os-arch: [macos-arm64, windows-x86_64]
|
|
||||||
llvm-build: [in-tree, out-of-tree]
|
|
||||||
torch-binary: [ON]
|
|
||||||
torch-version: [nightly, stable]
|
|
||||||
exclude:
|
|
||||||
# Exclude llvm out-of-tree and pytorch stable (to save resources)
|
|
||||||
- llvm-build: out-of-tree
|
|
||||||
torch-version: stable
|
|
||||||
# Exclude macos-arm64 and llvm out-of-tree altogether
|
|
||||||
- os-arch: macos-arm64
|
|
||||||
llvm-build: out-of-tree
|
|
||||||
- os-arch: macos-arm64
|
|
||||||
torch-version: stable
|
|
||||||
- os-arch: windows-x86_64
|
|
||||||
llvm-build: out-of-tree
|
|
||||||
- os-arch: windows-x86_64
|
|
||||||
torch-version: stable
|
|
||||||
include:
|
|
||||||
# Specify OS versions
|
|
||||||
- os-arch: ubuntu-x86_64
|
|
||||||
os: a100
|
|
||||||
- os-arch: macos-arm64
|
|
||||||
os: macos-latest
|
|
||||||
- os-arch: windows-x86_64
|
|
||||||
os: windows-latest
|
|
||||||
runs-on: ${{ matrix.os }}
|
|
||||||
|
|
||||||
steps:
|
|
||||||
- name: Prepare workspace
|
|
||||||
if: ${{ matrix.os-arch == 'ubuntu-x86_64' }}
|
|
||||||
run: |
|
|
||||||
# Clear the workspace directory so that we don't run into errors about
|
|
||||||
# existing lock files.
|
|
||||||
sudo rm -rf $GITHUB_WORKSPACE/*
|
|
||||||
|
|
||||||
- name: Checkout torch-mlir
|
|
||||||
uses: actions/checkout@v3
|
|
||||||
with:
|
|
||||||
submodules: 'true'
|
|
||||||
fetch-depth: 0
|
|
||||||
|
|
||||||
- name: Fetch PyTorch commit hash
|
|
||||||
if: ${{ matrix.os-arch != 'windows-x86_64' }}
|
|
||||||
run: |
|
|
||||||
PT_HASH="$(cat ${GITHUB_WORKSPACE}/pytorch-hash.txt)"
|
|
||||||
echo "PT_HASH=${PT_HASH}" >> ${GITHUB_ENV}
|
|
||||||
|
|
||||||
- name: Setup ccache
|
|
||||||
uses: ./.github/actions/setup-build
|
|
||||||
with:
|
|
||||||
cache-suffix: 'build-${{ matrix.llvm-build }}-${{ matrix.torch-version }}'
|
|
||||||
torch-version: ${{ matrix.torch-version }}
|
|
||||||
|
|
||||||
- name: Set up Visual Studio shell
|
|
||||||
if: ${{ matrix.os-arch == 'windows-x86_64' }}
|
|
||||||
uses: egor-tensin/vs-shell@v2
|
|
||||||
with:
|
|
||||||
arch: x64
|
|
||||||
|
|
||||||
- name: Try to Restore PyTorch Build Cache
|
|
||||||
if: ${{ matrix.torch-binary == 'OFF' }}
|
|
||||||
id: cache-pytorch
|
|
||||||
uses: actions/cache/restore@v3
|
|
||||||
with:
|
|
||||||
path: ${{ github.workspace }}/build_tools/python_deploy/wheelhouse
|
|
||||||
key: ${{ runner.os }}-pytorch-${{ env.PT_HASH }}
|
|
||||||
|
|
||||||
- name: Build and Test os-arch='ubuntu-x86_64' llvm-build='${{ matrix.llvm-build }}' torch-binary='${{ matrix.torch-binary }}'
|
|
||||||
if: ${{ matrix.os-arch == 'ubuntu-x86_64' }}
|
|
||||||
run: |
|
|
||||||
cd $GITHUB_WORKSPACE
|
|
||||||
TORCH_MLIR_SRC_PYTORCH_BRANCH="$(cat pytorch-hash.txt)" \
|
|
||||||
TM_PACKAGES="${{ matrix.llvm-build }}" \
|
|
||||||
TM_USE_PYTORCH_BINARY="${{ matrix.torch-binary }}" \
|
|
||||||
TM_PYTORCH_INSTALL_WITHOUT_REBUILD="${{ steps.cache-pytorch.outputs.cache-hit }}" \
|
|
||||||
TM_TORCH_VERSION="${{ matrix.torch-version }}" \
|
|
||||||
./build_tools/python_deploy/build_linux_packages.sh
|
|
||||||
|
|
||||||
- name: Configure os-arch='macos-arm64' llvm-build='in-tree' torch-binary='${{ matrix.torch-binary }}'
|
|
||||||
# cross compile, can't test arm64
|
|
||||||
if: ${{ matrix.os-arch == 'macos-arm64' && matrix.llvm-build == 'in-tree' }}
|
|
||||||
run: |
|
|
||||||
# TODO: Reenable LTC after build on macOS-arm64 is fixed (https://github.com/llvm/torch-mlir/issues/1253)
|
|
||||||
cmake -GNinja -Bbuild_arm64 \
|
|
||||||
-DCMAKE_BUILD_TYPE=Release \
|
|
||||||
-DCMAKE_C_COMPILER=clang \
|
|
||||||
-DCMAKE_CXX_COMPILER=clang++ \
|
|
||||||
-DCMAKE_C_COMPILER_LAUNCHER=ccache \
|
|
||||||
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
|
|
||||||
-DCMAKE_LINKER=lld \
|
|
||||||
-DCMAKE_OSX_ARCHITECTURES=arm64 \
|
|
||||||
-DLLVM_ENABLE_ASSERTIONS=ON \
|
|
||||||
-DLLVM_ENABLE_PROJECTS=mlir \
|
|
||||||
-DLLVM_EXTERNAL_PROJECTS="torch-mlir" \
|
|
||||||
-DLLVM_EXTERNAL_TORCH_MLIR_SOURCE_DIR="$GITHUB_WORKSPACE" \
|
|
||||||
-DLLVM_TARGETS_TO_BUILD=AArch64 \
|
|
||||||
-DLLVM_USE_HOST_TOOLS=ON \
|
|
||||||
-DLLVM_ENABLE_ZSTD=OFF \
|
|
||||||
-DMLIR_ENABLE_BINDINGS_PYTHON=ON \
|
|
||||||
-DTORCH_MLIR_ENABLE_STABLEHLO=OFF \
|
|
||||||
-DTORCH_MLIR_ENABLE_LTC=OFF \
|
|
||||||
-DTORCH_MLIR_USE_INSTALLED_PYTORCH="${{ matrix.torch-binary }}" \
|
|
||||||
-DMACOSX_DEPLOYMENT_TARGET=12.0 \
|
|
||||||
-DPython3_EXECUTABLE="$(which python)" \
|
|
||||||
$GITHUB_WORKSPACE/externals/llvm-project/llvm
|
|
||||||
|
|
||||||
- name: Build torch-mlir (cross-compile)
|
|
||||||
if: ${{ matrix.os-arch == 'macos-arm64' }}
|
|
||||||
run: |
|
|
||||||
cmake --build build_arm64
|
|
||||||
|
|
||||||
- name: Build (Windows)
|
|
||||||
if: ${{ matrix.os-arch == 'windows-x86_64' }}
|
|
||||||
shell: bash
|
|
||||||
run: ./build_tools/python_deploy/build_windows_ci.sh
|
|
||||||
|
|
||||||
- name: Save PyTorch Build Cache
|
|
||||||
if: ${{ github.ref_name == 'main' && matrix.torch-binary == 'OFF' }}
|
|
||||||
uses: actions/cache/save@v3
|
|
||||||
with:
|
|
||||||
path: ${{ github.workspace }}/build_tools/python_deploy/wheelhouse
|
|
||||||
key: ${{ runner.os }}-pytorch-${{ env.PT_HASH }}
|
|
||||||
|
|
||||||
- name: Print ccache statistics
|
|
||||||
shell: bash
|
|
||||||
run: ccache --show-stats
|
|
||||||
|
|
@ -1,309 +0,0 @@
|
||||||
# yamllint disable rule:line-length
|
|
||||||
name: Release Build
|
|
||||||
|
|
||||||
on:
|
|
||||||
workflow_dispatch:
|
|
||||||
inputs:
|
|
||||||
release_id:
|
|
||||||
description: 'Release id to upload artifacts to'
|
|
||||||
default: ''
|
|
||||||
python_package_version:
|
|
||||||
description: 'Version to use for creating the Python package'
|
|
||||||
default: ''
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
build_linux:
|
|
||||||
name: Manylinux x86_64 Build
|
|
||||||
runs-on: a100
|
|
||||||
strategy:
|
|
||||||
matrix:
|
|
||||||
package: [torch-mlir]
|
|
||||||
py_version: [cp38-cp38, cp311-cp311]
|
|
||||||
|
|
||||||
steps:
|
|
||||||
- name: Prepare workspace
|
|
||||||
run: |
|
|
||||||
# Clear the workspace directory so that we don't run into errors about
|
|
||||||
# existing lock files.
|
|
||||||
sudo rm -rf $GITHUB_WORKSPACE/*
|
|
||||||
|
|
||||||
- name: Get torch-mlir
|
|
||||||
uses: actions/checkout@v3
|
|
||||||
with:
|
|
||||||
submodules: 'true'
|
|
||||||
fetch-depth: 0
|
|
||||||
|
|
||||||
- uses: ./.github/actions/setup-build
|
|
||||||
with:
|
|
||||||
cache-enabled: 'false'
|
|
||||||
- name: Build Python wheels and smoke test.
|
|
||||||
run: |
|
|
||||||
cd $GITHUB_WORKSPACE
|
|
||||||
TM_PACKAGE_VERSION=${{ github.event.inputs.python_package_version }}
|
|
||||||
printf "TORCH_MLIR_PYTHON_PACKAGE_VERSION=%s\n" $TM_PACKAGE_VERSION > ./torch_mlir_package_version
|
|
||||||
TM_PYTHON_VERSIONS=${{ matrix.py_version }} TM_PACKAGES=${{ matrix.package }} ./build_tools/python_deploy/build_linux_packages.sh
|
|
||||||
|
|
||||||
# If we were given a release_id, then upload the package we just built
|
|
||||||
# to the github releases page.
|
|
||||||
- name: Upload Release Assets (if requested)
|
|
||||||
if: github.event.inputs.release_id != ''
|
|
||||||
id: upload-release-assets
|
|
||||||
uses: dwenegar/upload-release-assets@v1
|
|
||||||
env:
|
|
||||||
GITHUB_TOKEN: ${{ secrets.WORKFLOW_INVOCATION_TOKEN }}
|
|
||||||
with:
|
|
||||||
release_id: ${{ github.event.inputs.release_id }}
|
|
||||||
assets_path: ./build_tools/python_deploy/wheelhouse/torch*.whl
|
|
||||||
# Publishing is necessary to make the release visible to `pip`
|
|
||||||
# on the github releases page.
|
|
||||||
- name: Publish Release (if requested)
|
|
||||||
if: github.event.inputs.release_id != ''
|
|
||||||
id: publish_release
|
|
||||||
uses: eregon/publish-release@v1
|
|
||||||
env:
|
|
||||||
GITHUB_TOKEN: ${{ secrets.WORKFLOW_INVOCATION_TOKEN }}
|
|
||||||
with:
|
|
||||||
release_id: ${{ github.event.inputs.release_id }}
|
|
||||||
- name: Create dist directory
|
|
||||||
if: github.event.inputs.release_id != ''
|
|
||||||
run: mkdir dist
|
|
||||||
- name: Copy releases to publish to dist directory
|
|
||||||
if: github.event.inputs.release_id != ''
|
|
||||||
run: cp build_tools/python_deploy/wheelhouse/torch_mlir*.whl dist/
|
|
||||||
|
|
||||||
# Wheels must be published from a linux environment.
|
|
||||||
#
|
|
||||||
# See https://github.com/pypa/gh-action-pypi-publish/discussions/15
|
|
||||||
- name: Store the binary wheel
|
|
||||||
uses: actions/upload-artifact@v2
|
|
||||||
with:
|
|
||||||
name: wheels
|
|
||||||
path: dist
|
|
||||||
|
|
||||||
build_linux_arm64:
|
|
||||||
name: Manylinux arm64 Build
|
|
||||||
runs-on: linux-arm64
|
|
||||||
strategy:
|
|
||||||
matrix:
|
|
||||||
package: [torch-mlir]
|
|
||||||
py_version: [cp311-cp311]
|
|
||||||
|
|
||||||
steps:
|
|
||||||
- name: Prepare workspace
|
|
||||||
run: |
|
|
||||||
# Clear the workspace directory so that we don't run into errors about
|
|
||||||
# existing lock files.
|
|
||||||
sudo rm -rf $GITHUB_WORKSPACE/*
|
|
||||||
|
|
||||||
- name: Get torch-mlir
|
|
||||||
uses: actions/checkout@v3
|
|
||||||
with:
|
|
||||||
submodules: 'true'
|
|
||||||
fetch-depth: 0
|
|
||||||
|
|
||||||
- uses: ./.github/actions/setup-build
|
|
||||||
with:
|
|
||||||
cache-enabled: 'false'
|
|
||||||
- name: Build Python wheels and smoke test.
|
|
||||||
run: |
|
|
||||||
cd $GITHUB_WORKSPACE
|
|
||||||
TM_PACKAGE_VERSION=${{ github.event.inputs.python_package_version }}
|
|
||||||
printf "TORCH_MLIR_PYTHON_PACKAGE_VERSION=%s\n" $TM_PACKAGE_VERSION > ./torch_mlir_package_version
|
|
||||||
TM_PYTHON_VERSIONS=${{ matrix.py_version }} TM_PACKAGES=${{ matrix.package }} TORCH_MLIR_ENABLE_LTC='0' ./build_tools/python_deploy/build_linux_packages.sh
|
|
||||||
|
|
||||||
# If we were given a release_id, then upload the package we just built
|
|
||||||
# to the github releases page.
|
|
||||||
- name: Upload Release Assets (if requested)
|
|
||||||
if: github.event.inputs.release_id != ''
|
|
||||||
id: upload-release-assets
|
|
||||||
uses: dwenegar/upload-release-assets@v1
|
|
||||||
env:
|
|
||||||
GITHUB_TOKEN: ${{ secrets.WORKFLOW_INVOCATION_TOKEN }}
|
|
||||||
with:
|
|
||||||
release_id: ${{ github.event.inputs.release_id }}
|
|
||||||
assets_path: ./build_tools/python_deploy/wheelhouse/torch*.whl
|
|
||||||
# Publishing is necessary to make the release visible to `pip`
|
|
||||||
# on the github releases page.
|
|
||||||
- name: Publish Release (if requested)
|
|
||||||
if: github.event.inputs.release_id != ''
|
|
||||||
id: publish_release
|
|
||||||
uses: eregon/publish-release@v1
|
|
||||||
env:
|
|
||||||
GITHUB_TOKEN: ${{ secrets.WORKFLOW_INVOCATION_TOKEN }}
|
|
||||||
with:
|
|
||||||
release_id: ${{ github.event.inputs.release_id }}
|
|
||||||
- name: Create dist directory
|
|
||||||
if: github.event.inputs.release_id != ''
|
|
||||||
run: mkdir dist
|
|
||||||
- name: Copy releases to publish to dist directory
|
|
||||||
if: github.event.inputs.release_id != ''
|
|
||||||
run: cp build_tools/python_deploy/wheelhouse/torch_mlir*.whl dist/
|
|
||||||
|
|
||||||
# Wheels must be published from a linux environment.
|
|
||||||
#
|
|
||||||
# See https://github.com/pypa/gh-action-pypi-publish/discussions/15
|
|
||||||
- name: Store the binary wheel
|
|
||||||
uses: actions/upload-artifact@v2
|
|
||||||
with:
|
|
||||||
name: wheels
|
|
||||||
path: dist
|
|
||||||
|
|
||||||
build_macos:
|
|
||||||
name: MacOS Build
|
|
||||||
runs-on: macos-latest
|
|
||||||
strategy:
|
|
||||||
matrix:
|
|
||||||
package: [torch-mlir]
|
|
||||||
steps:
|
|
||||||
- name: Get torch-mlir
|
|
||||||
uses: actions/checkout@v3
|
|
||||||
with:
|
|
||||||
submodules: 'true'
|
|
||||||
- uses: ./.github/actions/setup-build
|
|
||||||
with:
|
|
||||||
cache-enabled: 'false'
|
|
||||||
- name: Build Python wheels and smoke test.
|
|
||||||
run: |
|
|
||||||
cd $GITHUB_WORKSPACE
|
|
||||||
python -m pip install wheel
|
|
||||||
TM_PACKAGE_VERSION=${{ github.event.inputs.python_package_version }}
|
|
||||||
printf "TORCH_MLIR_PYTHON_PACKAGE_VERSION=%s\n" $TM_PACKAGE_VERSION > ./torch_mlir_package_version
|
|
||||||
sudo ./build_tools/python_deploy/install_macos_deps.sh
|
|
||||||
packages=${{ matrix.package }} TORCH_MLIR_PYTHON_VERSIONS="3.11" ./build_tools/python_deploy/build_macos_packages.sh
|
|
||||||
|
|
||||||
# If we were given a release_id, then upload the package we just built
|
|
||||||
# to the github releases page.
|
|
||||||
- name: Upload Release Assets (if requested)
|
|
||||||
if: github.event.inputs.release_id != ''
|
|
||||||
id: upload-release-assets
|
|
||||||
uses: dwenegar/upload-release-assets@v1
|
|
||||||
env:
|
|
||||||
GITHUB_TOKEN: ${{ secrets.WORKFLOW_INVOCATION_TOKEN }}
|
|
||||||
with:
|
|
||||||
release_id: ${{ github.event.inputs.release_id }}
|
|
||||||
assets_path: ./build_tools/python_deploy/wheelhouse/torch*.whl
|
|
||||||
# Publishing is necessary to make the release visible to `pip`
|
|
||||||
# on the github releases page.
|
|
||||||
- name: Publish Release (if requested)
|
|
||||||
if: github.event.inputs.release_id != ''
|
|
||||||
id: publish_release
|
|
||||||
uses: eregon/publish-release@v1
|
|
||||||
env:
|
|
||||||
GITHUB_TOKEN: ${{ secrets.WORKFLOW_INVOCATION_TOKEN }}
|
|
||||||
with:
|
|
||||||
release_id: ${{ github.event.inputs.release_id }}
|
|
||||||
- name: Create dist directory
|
|
||||||
if: github.event.inputs.release_id != ''
|
|
||||||
run: mkdir dist
|
|
||||||
- name: Copy releases to publish to dist directory
|
|
||||||
if: github.event.inputs.release_id != ''
|
|
||||||
run: cp build_tools/python_deploy/wheelhouse/torch_mlir*.whl dist/
|
|
||||||
|
|
||||||
# Wheels must be published from a linux environment.
|
|
||||||
#
|
|
||||||
# See https://github.com/pypa/gh-action-pypi-publish/discussions/15
|
|
||||||
- name: Store the binary wheel
|
|
||||||
uses: actions/upload-artifact@v2
|
|
||||||
with:
|
|
||||||
name: wheels
|
|
||||||
path: dist
|
|
||||||
|
|
||||||
build_windows:
|
|
||||||
name: Windows Build
|
|
||||||
runs-on: windows-latest
|
|
||||||
strategy:
|
|
||||||
matrix:
|
|
||||||
package: [torch-mlir]
|
|
||||||
steps:
|
|
||||||
- name: Get torch-mlir
|
|
||||||
uses: actions/checkout@v3
|
|
||||||
with:
|
|
||||||
submodules: 'true'
|
|
||||||
- uses: ./.github/actions/setup-build
|
|
||||||
with:
|
|
||||||
cache-enabled: 'false'
|
|
||||||
- name: Set up Visual Studio shell
|
|
||||||
uses: egor-tensin/vs-shell@v2
|
|
||||||
with:
|
|
||||||
arch: x64
|
|
||||||
- name: Build Python wheels and smoke test.
|
|
||||||
shell: pwsh
|
|
||||||
run: |
|
|
||||||
$env:TORCH_MLIR_ENABLE_JIT_IR_IMPORTER='1'
|
|
||||||
$env:TORCH_MLIR_ENABLE_ONLY_MLIR_PYTHON_BINDINGS='0'
|
|
||||||
$env:TORCH_MLIR_PYTHON_PACKAGE_VERSION = '${{ github.event.inputs.python_package_version }}'
|
|
||||||
./build_tools/python_deploy/build_windows.ps1
|
|
||||||
|
|
||||||
# If we were given a release_id, then upload the package we just built
|
|
||||||
# to the github releases page.
|
|
||||||
- name: Upload Release Assets (if requested)
|
|
||||||
if: github.event.inputs.release_id != ''
|
|
||||||
id: upload-release-assets
|
|
||||||
uses: dwenegar/upload-release-assets@v1
|
|
||||||
env:
|
|
||||||
GITHUB_TOKEN: ${{ secrets.WORKFLOW_INVOCATION_TOKEN }}
|
|
||||||
with:
|
|
||||||
release_id: ${{ github.event.inputs.release_id }}
|
|
||||||
assets_path: ./wheelhouse/torch*.whl
|
|
||||||
# Publishing is necessary to make the release visible to `pip`
|
|
||||||
# on the github releases page.
|
|
||||||
- name: Publish Release (if requested)
|
|
||||||
if: github.event.inputs.release_id != ''
|
|
||||||
id: publish_release
|
|
||||||
uses: eregon/publish-release@v1
|
|
||||||
env:
|
|
||||||
GITHUB_TOKEN: ${{ secrets.WORKFLOW_INVOCATION_TOKEN }}
|
|
||||||
with:
|
|
||||||
release_id: ${{ github.event.inputs.release_id }}
|
|
||||||
- name: Create dist directory
|
|
||||||
if: github.event.inputs.release_id != ''
|
|
||||||
run: mkdir dist
|
|
||||||
continue-on-error: true
|
|
||||||
- name: Copy releases to publish to dist directory
|
|
||||||
if: github.event.inputs.release_id != ''
|
|
||||||
run: cp ./wheelhouse/torch_mlir*.whl dist/
|
|
||||||
|
|
||||||
# Wheels must be published from a linux environment.
|
|
||||||
#
|
|
||||||
# See https://github.com/pypa/gh-action-pypi-publish/discussions/15
|
|
||||||
- name: Store the binary wheel
|
|
||||||
uses: actions/upload-artifact@v2
|
|
||||||
with:
|
|
||||||
name: wheels
|
|
||||||
path: dist
|
|
||||||
|
|
||||||
publish_releases:
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
needs:
|
|
||||||
- build_linux
|
|
||||||
- build_linux_arm64
|
|
||||||
- build_macos
|
|
||||||
- build_windows
|
|
||||||
|
|
||||||
# Publish even if one of the builds failed
|
|
||||||
if: ${{ always() }}
|
|
||||||
|
|
||||||
steps:
|
|
||||||
- name: Invoke Publish Releases Page
|
|
||||||
uses: benc-uk/workflow-dispatch@v1
|
|
||||||
with:
|
|
||||||
workflow: Publish releases page
|
|
||||||
token: ${{ secrets.WORKFLOW_INVOCATION_TOKEN }}
|
|
||||||
|
|
||||||
# Wheels must be published from a linux environment.
|
|
||||||
#
|
|
||||||
# See https://github.com/pypa/gh-action-pypi-publish/discussions/15
|
|
||||||
#
|
|
||||||
# We're temporarily disabling pypi publishing until we can fix audit wheel
|
|
||||||
# ODR torch issues. See https://github.com/llvm/torch-mlir/issues/1709
|
|
||||||
#
|
|
||||||
#- name: Download wheels for publishing to PyPI
|
|
||||||
# uses: actions/download-artifact@v3
|
|
||||||
# with:
|
|
||||||
# name: wheels
|
|
||||||
# path: dist
|
|
||||||
#- name: Publish to PyPI
|
|
||||||
# if: github.event.inputs.release_id != ''
|
|
||||||
# uses: pypa/gh-action-pypi-publish@v1.5.1
|
|
||||||
# with:
|
|
||||||
# password: ${{ secrets.PYPI_API_TOKEN }}
|
|
||||||
|
|
@ -1,34 +0,0 @@
|
||||||
# yamllint disable rule:line-length
|
|
||||||
name: RollPyTorch Merge
|
|
||||||
|
|
||||||
on:
|
|
||||||
workflow_run:
|
|
||||||
workflows: [Build and Test]
|
|
||||||
types: [completed]
|
|
||||||
branches: [rollpytorch]
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
merge-pr:
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
if: |
|
|
||||||
github.repository == 'llvm/torch-mlir' &&
|
|
||||||
github.event.workflow_run.actor.login == 'stellaraccident' &&
|
|
||||||
github.event.workflow_run.conclusion == 'success'
|
|
||||||
|
|
||||||
steps:
|
|
||||||
# Fetch the repo first so that the gh command knows where to look for the PR
|
|
||||||
- name: Fetch Repo
|
|
||||||
uses: actions/checkout@v3
|
|
||||||
with:
|
|
||||||
token: ${{ secrets.WORKFLOW_INVOCATION_TOKEN }}
|
|
||||||
|
|
||||||
- name: Merge RollPyTorch PR
|
|
||||||
run: |
|
|
||||||
for pr_id in ${{ join(github.event.workflow_run.pull_requests.*.number, ' ') }}
|
|
||||||
do
|
|
||||||
echo "Merging PR: $pr_id"
|
|
||||||
gh pr merge $pr_id --delete-branch --squash
|
|
||||||
done
|
|
||||||
shell: bash
|
|
||||||
env:
|
|
||||||
GH_TOKEN: ${{ secrets.ROLLPYTORCH_TOKEN1 }}
|
|
||||||
|
|
@ -1,70 +0,0 @@
|
||||||
# yamllint disable rule:line-length
|
|
||||||
name: Release oneshot snapshot package
|
|
||||||
|
|
||||||
on:
|
|
||||||
workflow_dispatch:
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
release_snapshot_package:
|
|
||||||
name: "Tag snapshot release"
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
# Don't run this in everyone's forks.
|
|
||||||
if: github.repository == 'llvm/torch-mlir'
|
|
||||||
steps:
|
|
||||||
- name: Prepare workspace
|
|
||||||
run: |
|
|
||||||
# Clear the workspace directory so that we don't run into errors about
|
|
||||||
# existing lock files.
|
|
||||||
sudo rm -rf $GITHUB_WORKSPACE/*
|
|
||||||
|
|
||||||
- name: Checking out repository
|
|
||||||
uses: actions/checkout@v3
|
|
||||||
with:
|
|
||||||
token: ${{ secrets.WORKFLOW_INVOCATION_TOKEN }}
|
|
||||||
|
|
||||||
- name: Compute version
|
|
||||||
run: |
|
|
||||||
git fetch --depth=1 origin +refs/tags/*:refs/tags/*
|
|
||||||
package_version="$(printf '%(%Y%m%d)T.${{ github.run_number }}')"
|
|
||||||
tag_name="oneshot-${package_version}"
|
|
||||||
echo "package_version=${package_version}" >> $GITHUB_ENV
|
|
||||||
echo "tag_name=${tag_name}" >> $GITHUB_ENV
|
|
||||||
|
|
||||||
- name: Updating snapshot tag
|
|
||||||
run: |
|
|
||||||
git tag "${tag_name}"
|
|
||||||
|
|
||||||
- name: Pushing changes
|
|
||||||
uses: ad-m/github-push-action@v0.6.0
|
|
||||||
with:
|
|
||||||
github_token: ${{ secrets.WORKFLOW_INVOCATION_TOKEN }}
|
|
||||||
branch: ${{ github.ref_name }}
|
|
||||||
tags: true
|
|
||||||
|
|
||||||
- name: Create Release
|
|
||||||
id: create_release
|
|
||||||
uses: actions/create-release@v1
|
|
||||||
env:
|
|
||||||
GITHUB_TOKEN: ${{ secrets.WORKFLOW_INVOCATION_TOKEN }}
|
|
||||||
with:
|
|
||||||
tag_name: ${{ env.tag_name }}
|
|
||||||
release_name: torch-mlir snapshot ${{ env.tag_name }}
|
|
||||||
body: |
|
|
||||||
Automatic snapshot release of torch-mlir.
|
|
||||||
draft: true
|
|
||||||
prerelease: false
|
|
||||||
|
|
||||||
- name: "Invoke workflow :: Build and Test"
|
|
||||||
uses: benc-uk/workflow-dispatch@v1
|
|
||||||
with:
|
|
||||||
workflow: Build and Test
|
|
||||||
token: ${{ secrets.WORKFLOW_INVOCATION_TOKEN }}
|
|
||||||
ref: "${{ env.tag_name }}"
|
|
||||||
|
|
||||||
- name: "Invoke workflow :: Release Build"
|
|
||||||
uses: benc-uk/workflow-dispatch@v1
|
|
||||||
with:
|
|
||||||
workflow: Release Build
|
|
||||||
token: ${{ secrets.WORKFLOW_INVOCATION_TOKEN }}
|
|
||||||
ref: "${{ env.tag_name }}"
|
|
||||||
inputs: '{"release_id": "${{ steps.create_release.outputs.id }}", "python_package_version": "${{ env.package_version }}"}'
|
|
||||||
|
|
@ -1,73 +0,0 @@
|
||||||
# yamllint disable rule:line-length
|
|
||||||
name: Release snapshot package
|
|
||||||
|
|
||||||
on:
|
|
||||||
# schedule:
|
|
||||||
# - cron: '0 11 * * *'
|
|
||||||
workflow_dispatch:
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
release_snapshot_package:
|
|
||||||
name: "Tag snapshot release"
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
# Don't run this in everyone's forks.
|
|
||||||
if: github.repository == 'llvm/torch-mlir'
|
|
||||||
steps:
|
|
||||||
|
|
||||||
- name: Prepare workspace
|
|
||||||
run: |
|
|
||||||
# Clear the workspace directory so that we don't run into errors about
|
|
||||||
# existing lock files.
|
|
||||||
sudo rm -rf $GITHUB_WORKSPACE/*
|
|
||||||
|
|
||||||
- name: Checking out repository
|
|
||||||
uses: actions/checkout@v3
|
|
||||||
with:
|
|
||||||
token: ${{ secrets.WORKFLOW_INVOCATION_TOKEN }}
|
|
||||||
|
|
||||||
- name: Compute version
|
|
||||||
run: |
|
|
||||||
git fetch --depth=1 origin +refs/tags/*:refs/tags/*
|
|
||||||
package_version="$(printf '%(%Y%m%d)T.${{ github.run_number }}')"
|
|
||||||
tag_name="snapshot-${package_version}"
|
|
||||||
echo "package_version=${package_version}" >> $GITHUB_ENV
|
|
||||||
echo "tag_name=${tag_name}" >> $GITHUB_ENV
|
|
||||||
|
|
||||||
- name: Updating snapshot tag
|
|
||||||
run: |
|
|
||||||
git tag "${tag_name}"
|
|
||||||
|
|
||||||
- name: Pushing changes
|
|
||||||
uses: ad-m/github-push-action@v0.6.0
|
|
||||||
with:
|
|
||||||
github_token: ${{ secrets.WORKFLOW_INVOCATION_TOKEN }}
|
|
||||||
branch: main
|
|
||||||
tags: true
|
|
||||||
|
|
||||||
- name: Create Release
|
|
||||||
id: create_release
|
|
||||||
uses: actions/create-release@v1
|
|
||||||
env:
|
|
||||||
GITHUB_TOKEN: ${{ secrets.WORKFLOW_INVOCATION_TOKEN }}
|
|
||||||
with:
|
|
||||||
tag_name: ${{ env.tag_name }}
|
|
||||||
release_name: torch-mlir snapshot ${{ env.tag_name }}
|
|
||||||
body: |
|
|
||||||
Automatic snapshot release of torch-mlir.
|
|
||||||
draft: true
|
|
||||||
prerelease: false
|
|
||||||
|
|
||||||
- name: "Invoke workflow :: Build and Test"
|
|
||||||
uses: benc-uk/workflow-dispatch@v1
|
|
||||||
with:
|
|
||||||
workflow: Build and Test
|
|
||||||
token: ${{ secrets.WORKFLOW_INVOCATION_TOKEN }}
|
|
||||||
ref: "${{ env.tag_name }}"
|
|
||||||
|
|
||||||
- name: "Invoke workflow :: Release Build"
|
|
||||||
uses: benc-uk/workflow-dispatch@v1
|
|
||||||
with:
|
|
||||||
workflow: Release Build
|
|
||||||
token: ${{ secrets.WORKFLOW_INVOCATION_TOKEN }}
|
|
||||||
ref: "${{ env.tag_name }}"
|
|
||||||
inputs: '{"release_id": "${{ steps.create_release.outputs.id }}", "python_package_version": "${{ env.package_version }}"}'
|
|
||||||
|
|
@ -1,35 +0,0 @@
|
||||||
#!/bin/bash
|
|
||||||
set -eu -o pipefail
|
|
||||||
|
|
||||||
if [ -z "$PYTHON" ]; then
|
|
||||||
PYTHON="$(which python)"
|
|
||||||
fi
|
|
||||||
version="$("$PYTHON" --version)"
|
|
||||||
echo "Using python: $PYTHON (version $version)"
|
|
||||||
|
|
||||||
repo_root="$(cd "$(dirname "$0")"/.. && pwd)"
|
|
||||||
wheelhouse="$repo_root/wheelhouse"
|
|
||||||
package_test_venv="$wheelhouse/package-test.venv"
|
|
||||||
mkdir -p "$wheelhouse"
|
|
||||||
cd "$wheelhouse"
|
|
||||||
|
|
||||||
echo "---- BUILDING torch-mlir ----"
|
|
||||||
CMAKE_GENERATOR=Ninja \
|
|
||||||
$PYTHON "${repo_root}/setup.py" bdist_wheel --dist-dir "$wheelhouse" -v
|
|
||||||
|
|
||||||
# Smoke test: create a venv, install the package, and run an example.
|
|
||||||
|
|
||||||
echo "---- CREATING VENV ----"
|
|
||||||
python -m venv "$package_test_venv"
|
|
||||||
VENV_PYTHON="$package_test_venv/bin/python"
|
|
||||||
|
|
||||||
# Install the Torch-MLIR package.
|
|
||||||
# Note that we also need to pass in the `-r requirements.txt` here to pick up
|
|
||||||
# the right --find-links flag for the nightly PyTorch wheel registry.
|
|
||||||
echo "---- INSTALLING torch-mlir and dependencies ----"
|
|
||||||
$VENV_PYTHON -m pip install -f "$wheelhouse" --force-reinstall torch_mlir -r "${repo_root}/requirements.txt"
|
|
||||||
echo "---- INSTALLING other deps for smoke test ----"
|
|
||||||
$VENV_PYTHON -m pip install requests pillow
|
|
||||||
|
|
||||||
echo "---- RUNNING SMOKE TEST ----"
|
|
||||||
$VENV_PYTHON "$repo_root/examples/torchscript_resnet18.py"
|
|
||||||
|
|
@ -1,29 +0,0 @@
|
||||||
#!/bin/bash
|
|
||||||
|
|
||||||
# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
|
|
||||||
# See https://llvm.org/LICENSE.txt for license information.
|
|
||||||
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
|
||||||
# Also available under a BSD-style license. See LICENSE.
|
|
||||||
|
|
||||||
# Simple script that does a CMake configure of this project as an external
|
|
||||||
# LLVM project so it can be tested in isolation to larger assemblies.
|
|
||||||
# This is meant for CI's and project maintainers.
|
|
||||||
|
|
||||||
set -eu -o errtrace
|
|
||||||
|
|
||||||
project_dir="$(cd "$(dirname "$0")"/.. && pwd)"
|
|
||||||
llvm_project_dir="$project_dir/externals/llvm-project"
|
|
||||||
build_dir="$project_dir/build"
|
|
||||||
|
|
||||||
cmake -GNinja -B"$build_dir" "$llvm_project_dir/llvm" \
|
|
||||||
-DCMAKE_BUILD_TYPE=Release \
|
|
||||||
-DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
|
|
||||||
-DLLVM_ENABLE_PROJECTS=mlir \
|
|
||||||
-DLLVM_EXTERNAL_PROJECTS="torch-mlir" \
|
|
||||||
-DLLVM_EXTERNAL_TORCH_MLIR_SOURCE_DIR="$project_dir" \
|
|
||||||
-DMLIR_ENABLE_BINDINGS_PYTHON=ON \
|
|
||||||
-DLLVM_ENABLE_ASSERTIONS=ON \
|
|
||||||
-DLLVM_TARGETS_TO_BUILD=host
|
|
||||||
|
|
||||||
cd "$build_dir"
|
|
||||||
ninja tools/torch-mlir/all check-torch-mlir-all
|
|
||||||
|
|
@ -1 +0,0 @@
|
||||||
manylinux2014-x64
|
|
||||||
|
|
@ -1,494 +0,0 @@
|
||||||
#!/bin/bash
|
|
||||||
# Copyright 2022 The IREE Authors
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License v2.0 with LLVM Exceptions.
|
|
||||||
# See https://llvm.org/LICENSE.txt for license information.
|
|
||||||
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
|
||||||
|
|
||||||
# build_linux_packages.sh
|
|
||||||
# One stop build of IREE Python packages for Linux. The Linux build is
|
|
||||||
# complicated because it has to be done via a docker container that has
|
|
||||||
# an LTS glibc version, all Python packages and other deps.
|
|
||||||
# This script handles all of those details.
|
|
||||||
#
|
|
||||||
# Usage:
|
|
||||||
# Build everything (all packages, all python versions):
|
|
||||||
# ./build_tools/python_deploy/build_linux_packages.sh
|
|
||||||
#
|
|
||||||
# Build specific Python versions and packages to custom directory:
|
|
||||||
# TM_PYTHON_VERSIONS="cp38-cp38 cp39-cp39" \
|
|
||||||
# TM_PACKAGES="torch-mlir" \
|
|
||||||
# TM_OUTPUT_DIR="/tmp/wheelhouse" \
|
|
||||||
# ./build_tools/python_deploy/build_linux_packages.sh
|
|
||||||
#
|
|
||||||
# Valid Python versions match a subdirectory under /opt/python in the docker
|
|
||||||
# image. Typically:
|
|
||||||
# cp39-cp39 cp310-cp310
|
|
||||||
#
|
|
||||||
# Valid packages:
|
|
||||||
# torch-mlir, in-tree, out-of-tree
|
|
||||||
#
|
|
||||||
# Note that this script is meant to be run on CI and it will pollute both the
|
|
||||||
# output directory and in-tree build/ directories with docker created, root owned builds.
|
|
||||||
# Sorry - there is no good way around it but TODO: move to using user UID/GID.
|
|
||||||
#
|
|
||||||
# It can be run on a workstation but recommend using a git worktree dedicated
|
|
||||||
# to packaging to avoid stomping on development artifacts.
|
|
||||||
set -eu -o errtrace
|
|
||||||
|
|
||||||
this_dir="$(cd "$(dirname "$0")" && pwd)"
|
|
||||||
repo_root="$(cd "$this_dir"/../../ && pwd)"
|
|
||||||
arch="$(uname -m)"
|
|
||||||
echo "Running on Arch: ${arch}"
|
|
||||||
# This needs to be a manylinux image so we can ship pip packages
|
|
||||||
TM_RELEASE_DOCKER_IMAGE="${TM_RELEASE_DOCKER_IMAGE:-quay.io/pypa/manylinux2014_${arch}}"
|
|
||||||
# This assumes an Ubuntu LTS like image. You can build your own with
|
|
||||||
# ./build_tools/docker/Dockerfile
|
|
||||||
TM_CI_DOCKER_IMAGE="${TM_CI_DOCKER_IMAGE:-powderluv/torch-mlir-ci:latest}"
|
|
||||||
# Version of Python to use in Release builds. Ignored in CIs.
|
|
||||||
TM_PYTHON_VERSIONS="${TM_PYTHON_VERSIONS:-cp38-cp38 cp310-cp310 cp311-cp311}"
|
|
||||||
# Location to store Release wheels
|
|
||||||
TM_OUTPUT_DIR="${TM_OUTPUT_DIR:-${this_dir}/wheelhouse}"
|
|
||||||
# What "packages to build"
|
|
||||||
TM_PACKAGES="${TM_PACKAGES:-torch-mlir torch-mlir-core}"
|
|
||||||
# Use pre-built Pytorch
|
|
||||||
TM_USE_PYTORCH_BINARY="${TM_USE_PYTORCH_BINARY:-ON}"
|
|
||||||
# Skip running tests if you want quick iteration
|
|
||||||
TM_SKIP_TESTS="${TM_SKIP_TESTS:-OFF}"
|
|
||||||
# Update ODS and abstract interpretation library files
|
|
||||||
TM_UPDATE_ODS_AND_ABSTRACT_INTERP_LIB="${TM_UPDATE_ODS_AND_ABSTRACT_INTERP_LIB:-OFF}"
|
|
||||||
# Determine if we use a stable or a nightly torch build
|
|
||||||
TM_TORCH_VERSION="${TM_TORCH_VERSION:-nightly}"
|
|
||||||
|
|
||||||
PKG_VER_FILE="${repo_root}"/torch_mlir_package_version ; [ -f "$PKG_VER_FILE" ] && . "$PKG_VER_FILE"
|
|
||||||
TORCH_MLIR_PYTHON_PACKAGE_VERSION="${TORCH_MLIR_PYTHON_PACKAGE_VERSION:-0.0.1}"
|
|
||||||
echo "Setting torch-mlir Python Package version to: ${TORCH_MLIR_PYTHON_PACKAGE_VERSION}"
|
|
||||||
|
|
||||||
export TORCH_MLIR_SRC_PYTORCH_REPO="${TORCH_MLIR_SRC_PYTORCH_REPO:-pytorch/pytorch}"
|
|
||||||
echo "Setting torch-mlir PyTorch Repo for source builds to: ${TORCH_MLIR_SRC_PYTORCH_REPO}"
|
|
||||||
export TORCH_MLIR_SRC_PYTORCH_BRANCH="${TORCH_MLIR_SRC_PYTORCH_BRANCH:-master}"
|
|
||||||
echo "Setting torch-mlir PyTorch version for source builds to: ${TORCH_MLIR_SRC_PYTORCH_BRANCH}"
|
|
||||||
|
|
||||||
# If using PyTorch source, install from the existing build instead of rebuilding
|
|
||||||
# all of PyTorch. This option is useful in CI, when it determines that the
|
|
||||||
# PyTorch version has not changed between consecutive runs.
|
|
||||||
export TM_PYTORCH_INSTALL_WITHOUT_REBUILD="${TM_PYTORCH_INSTALL_WITHOUT_REBUILD:-false}"
|
|
||||||
|
|
||||||
function run_on_host() {
|
|
||||||
echo "Running on host for $1:$@"
|
|
||||||
echo "Outputting to ${TM_OUTPUT_DIR}"
|
|
||||||
if [[ $TM_PYTORCH_INSTALL_WITHOUT_REBUILD != "true" ]]; then
|
|
||||||
# We want to use the cached files, so don't remove them.
|
|
||||||
rm -rf "${TM_OUTPUT_DIR}"
|
|
||||||
fi
|
|
||||||
mkdir -p "${TM_OUTPUT_DIR}"
|
|
||||||
case "$package" in
|
|
||||||
torch-mlir)
|
|
||||||
TM_CURRENT_DOCKER_IMAGE=${TM_RELEASE_DOCKER_IMAGE}
|
|
||||||
export USERID=0
|
|
||||||
export GROUPID=0
|
|
||||||
;;
|
|
||||||
torch-mlir-core)
|
|
||||||
TM_CURRENT_DOCKER_IMAGE=${TM_RELEASE_DOCKER_IMAGE}
|
|
||||||
export USERID=0
|
|
||||||
export GROUPID=0
|
|
||||||
;;
|
|
||||||
out-of-tree)
|
|
||||||
TM_CURRENT_DOCKER_IMAGE=${TM_CI_DOCKER_IMAGE}
|
|
||||||
# CI uses only Python3.10
|
|
||||||
TM_PYTHON_VERSIONS="cp310-cp310"
|
|
||||||
export USERID=$(id -u)
|
|
||||||
export GROUPID=$(id -g)
|
|
||||||
;;
|
|
||||||
in-tree)
|
|
||||||
TM_CURRENT_DOCKER_IMAGE=${TM_CI_DOCKER_IMAGE}
|
|
||||||
# CI uses only Python3.10
|
|
||||||
TM_PYTHON_VERSIONS="cp310-cp310"
|
|
||||||
export USERID=$(id -u)
|
|
||||||
export GROUPID=$(id -g)
|
|
||||||
;;
|
|
||||||
*)
|
|
||||||
echo "Unrecognized package '$package'"
|
|
||||||
exit 1
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
echo "Launching docker image ${TM_CURRENT_DOCKER_IMAGE} with UID:${USERID} GID:${GROUPID}"
|
|
||||||
docker run --rm \
|
|
||||||
-v "${repo_root}:/main_checkout/torch-mlir" \
|
|
||||||
-v "${TM_OUTPUT_DIR}:/wheelhouse" \
|
|
||||||
-v "${HOME}:/home/${USER}" \
|
|
||||||
--user ${USERID}:${GROUPID} \
|
|
||||||
--workdir="/home/$USER" \
|
|
||||||
--volume="/etc/group:/etc/group:ro" \
|
|
||||||
--volume="/etc/passwd:/etc/passwd:ro" \
|
|
||||||
--volume="/etc/shadow:/etc/shadow:ro" \
|
|
||||||
--ipc=host \
|
|
||||||
-e __MANYLINUX_BUILD_WHEELS_IN_DOCKER=1 \
|
|
||||||
-e "TORCH_MLIR_PYTHON_PACKAGE_VERSION=${TORCH_MLIR_PYTHON_PACKAGE_VERSION}" \
|
|
||||||
-e "TM_PYTHON_VERSIONS=${TM_PYTHON_VERSIONS}" \
|
|
||||||
-e "TM_PACKAGES=${package}" \
|
|
||||||
-e "TM_SKIP_TESTS=${TM_SKIP_TESTS}" \
|
|
||||||
-e "TM_UPDATE_ODS_AND_ABSTRACT_INTERP_LIB=${TM_UPDATE_ODS_AND_ABSTRACT_INTERP_LIB}" \
|
|
||||||
-e "TM_USE_PYTORCH_BINARY=${TM_USE_PYTORCH_BINARY}" \
|
|
||||||
-e "TORCH_MLIR_SRC_PYTORCH_REPO=${TORCH_MLIR_SRC_PYTORCH_REPO}" \
|
|
||||||
-e "TORCH_MLIR_SRC_PYTORCH_BRANCH=${TORCH_MLIR_SRC_PYTORCH_BRANCH}" \
|
|
||||||
-e "TM_PYTORCH_INSTALL_WITHOUT_REBUILD=${TM_PYTORCH_INSTALL_WITHOUT_REBUILD}" \
|
|
||||||
-e "TM_TORCH_VERSION=${TM_TORCH_VERSION}" \
|
|
||||||
-e "CCACHE_DIR=/main_checkout/torch-mlir/.ccache" \
|
|
||||||
"${TM_CURRENT_DOCKER_IMAGE}" \
|
|
||||||
/bin/bash /main_checkout/torch-mlir/build_tools/python_deploy/build_linux_packages.sh
|
|
||||||
}
|
|
||||||
|
|
||||||
function run_in_docker() {
|
|
||||||
echo "Running in docker"
|
|
||||||
echo "Using python versions: ${TM_PYTHON_VERSIONS}"
|
|
||||||
|
|
||||||
local orig_path="$PATH"
|
|
||||||
|
|
||||||
# Build phase.
|
|
||||||
for package in $TM_PACKAGES; do
|
|
||||||
echo "******************** BUILDING PACKAGE ${package} (docker) ************"
|
|
||||||
for python_version in $TM_PYTHON_VERSIONS; do
|
|
||||||
python_dir="/opt/python/$python_version"
|
|
||||||
if ! [ -x "$python_dir/bin/python" ]; then
|
|
||||||
echo "Could not find python: $python_dir (using system default Python3)"
|
|
||||||
python_dir=`which python3`
|
|
||||||
echo "Defaulting to $python_dir (expected for CI builds)"
|
|
||||||
fi
|
|
||||||
export PATH=$python_dir/bin:$orig_path
|
|
||||||
echo ":::: Python version $(python3 --version)"
|
|
||||||
case "$package" in
|
|
||||||
torch-mlir)
|
|
||||||
clean_wheels torch_mlir "$python_version"
|
|
||||||
build_torch_mlir "$TM_TORCH_VERSION"
|
|
||||||
|
|
||||||
# Disable audit wheel until we can fix ODR torch issues. See
|
|
||||||
# https://github.com/llvm/torch-mlir/issues/1709
|
|
||||||
#
|
|
||||||
#run_audit_wheel torch_mlir "$python_version"
|
|
||||||
|
|
||||||
clean_build torch_mlir "$python_version"
|
|
||||||
;;
|
|
||||||
torch-mlir-core)
|
|
||||||
clean_wheels torch_mlir_core "$python_version"
|
|
||||||
build_torch_mlir_core
|
|
||||||
run_audit_wheel torch_mlir_core "$python_version"
|
|
||||||
clean_build torch_mlir_core "$python_version"
|
|
||||||
;;
|
|
||||||
out-of-tree)
|
|
||||||
setup_venv "$python_version" "$TM_TORCH_VERSION"
|
|
||||||
build_out_of_tree "$TM_USE_PYTORCH_BINARY" "$python_version" "$TM_TORCH_VERSION"
|
|
||||||
if [ "${TM_UPDATE_ODS_AND_ABSTRACT_INTERP_LIB}" == "ON" ]; then
|
|
||||||
pushd /main_checkout/torch-mlir
|
|
||||||
TORCH_MLIR_BUILD_DIR=/main_checkout/torch-mlir/build_oot ./build_tools/update_torch_ods.sh
|
|
||||||
TORCH_MLIR_BUILD_DIR=/main_checkout/torch-mlir/build_oot ./build_tools/update_abstract_interp_lib.sh
|
|
||||||
popd
|
|
||||||
fi
|
|
||||||
if [ "${TM_SKIP_TESTS}" == "OFF" ]; then
|
|
||||||
test_out_of_tree
|
|
||||||
fi
|
|
||||||
;;
|
|
||||||
in-tree)
|
|
||||||
setup_venv "$python_version" "$TM_TORCH_VERSION"
|
|
||||||
build_in_tree "$TM_USE_PYTORCH_BINARY" "$python_version" "$TM_TORCH_VERSION"
|
|
||||||
if [ "${TM_UPDATE_ODS_AND_ABSTRACT_INTERP_LIB}" == "ON" ]; then
|
|
||||||
pushd /main_checkout/torch-mlir
|
|
||||||
./build_tools/update_torch_ods.sh
|
|
||||||
./build_tools/update_abstract_interp_lib.sh
|
|
||||||
popd
|
|
||||||
fi
|
|
||||||
if [ "${TM_SKIP_TESTS}" == "OFF" ]; then
|
|
||||||
test_in_tree "$TM_TORCH_VERSION";
|
|
||||||
fi
|
|
||||||
;;
|
|
||||||
*)
|
|
||||||
echo "Unrecognized package '$package'"
|
|
||||||
exit 1
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
done
|
|
||||||
done
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
function build_in_tree() {
|
|
||||||
local torch_from_bin="$1"
|
|
||||||
local python_version="$2"
|
|
||||||
|
|
||||||
local torch_version="$3"
|
|
||||||
local enable_ltc="ON"
|
|
||||||
if [[ "${torch_version}" == "stable" ]]
|
|
||||||
then
|
|
||||||
enable_ltc="OFF"
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo ":::: Build in-tree Torch from binary: $torch_from_bin with Python: $python_version"
|
|
||||||
cmake -GNinja -B/main_checkout/torch-mlir/build \
|
|
||||||
-DCMAKE_BUILD_TYPE=Release \
|
|
||||||
-DCMAKE_C_COMPILER=clang \
|
|
||||||
-DCMAKE_CXX_COMPILER=clang++ \
|
|
||||||
-DCMAKE_EXE_LINKER_FLAGS_INIT="-fuse-ld=lld" \
|
|
||||||
-DCMAKE_MODULE_LINKER_FLAGS_INIT="-fuse-ld=lld" \
|
|
||||||
-DCMAKE_SHARED_LINKER_FLAGS_INIT="-fuse-ld=lld" \
|
|
||||||
-DLLVM_ENABLE_ASSERTIONS=ON \
|
|
||||||
-DCMAKE_C_COMPILER_LAUNCHER=ccache \
|
|
||||||
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
|
|
||||||
-DLLVM_ENABLE_PROJECTS=mlir \
|
|
||||||
-DLLVM_EXTERNAL_PROJECTS="torch-mlir" \
|
|
||||||
-DLLVM_EXTERNAL_TORCH_MLIR_SOURCE_DIR="/main_checkout/torch-mlir" \
|
|
||||||
-DLLVM_TARGETS_TO_BUILD=host \
|
|
||||||
-DMLIR_ENABLE_BINDINGS_PYTHON=ON \
|
|
||||||
-DTORCH_MLIR_ENABLE_LTC=${enable_ltc} \
|
|
||||||
-DTORCH_MLIR_USE_INSTALLED_PYTORCH="$torch_from_bin" \
|
|
||||||
-DTORCH_MLIR_SRC_PYTORCH_REPO=${TORCH_MLIR_SRC_PYTORCH_REPO} \
|
|
||||||
-DTORCH_MLIR_SRC_PYTORCH_BRANCH=${TORCH_MLIR_SRC_PYTORCH_BRANCH} \
|
|
||||||
-DTM_PYTORCH_INSTALL_WITHOUT_REBUILD=${TM_PYTORCH_INSTALL_WITHOUT_REBUILD} \
|
|
||||||
-DPython3_EXECUTABLE="$(which python3)" \
|
|
||||||
/main_checkout/torch-mlir/externals/llvm-project/llvm
|
|
||||||
cmake --build /main_checkout/torch-mlir/build --target tools/torch-mlir/all
|
|
||||||
ccache -s
|
|
||||||
}
|
|
||||||
|
|
||||||
function _check_file_not_changed_by() {
|
|
||||||
# _check_file_not_changed_by <cmd> <file>
|
|
||||||
cmd="$1"
|
|
||||||
file="$2"
|
|
||||||
file_backup="$PWD/$(basename $file)"
|
|
||||||
file_new="$PWD/$(basename $file).new"
|
|
||||||
# Save the original file.
|
|
||||||
cp "$file" "$file_backup"
|
|
||||||
# Run the command to regenerate it.
|
|
||||||
"$1" || return 1
|
|
||||||
# Save the new generated file.
|
|
||||||
cp "$file" "$file_new"
|
|
||||||
# Restore the original file. We want this function to not change the user's
|
|
||||||
# working tree state.
|
|
||||||
mv "$file_backup" "$file"
|
|
||||||
# We use git-diff as "just a diff program" (no SCM stuff) because it has
|
|
||||||
# nicer output than regular `diff`.
|
|
||||||
if ! git diff --no-index --quiet "$file" "$file_new"; then
|
|
||||||
echo "#######################################################"
|
|
||||||
echo "Generated file '${file}' is not up to date (see diff below)"
|
|
||||||
echo ">>> Please run '${cmd}' to update it <<<"
|
|
||||||
echo "#######################################################"
|
|
||||||
git diff --no-index --color=always "$file" "$file_new"
|
|
||||||
# TODO: Is there a better cleanup strategy that doesn't require duplicating
|
|
||||||
# this inside and outside the `if`?
|
|
||||||
rm "$file_new"
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
rm "$file_new"
|
|
||||||
}
|
|
||||||
|
|
||||||
function test_in_tree() {
|
|
||||||
local torch_version="$1"
|
|
||||||
|
|
||||||
echo ":::: Test in-tree"
|
|
||||||
cmake --build /main_checkout/torch-mlir/build --target check-torch-mlir-all
|
|
||||||
|
|
||||||
cd /main_checkout/torch-mlir/
|
|
||||||
export PYTHONPATH="/main_checkout/torch-mlir/build/tools/torch-mlir/python_packages/torch_mlir:/main_checkout/torch-mlir/projects/pt1"
|
|
||||||
|
|
||||||
case $torch_version in
|
|
||||||
nightly)
|
|
||||||
echo ":::: Test with nightly torch"
|
|
||||||
|
|
||||||
echo ":::: Check that update_abstract_interp_lib.sh has been run"
|
|
||||||
_check_file_not_changed_by ./build_tools/update_abstract_interp_lib.sh lib/Dialect/Torch/Transforms/AbstractInterpLibrary.cpp
|
|
||||||
|
|
||||||
echo ":::: Check that update_torch_ods.sh has been run"
|
|
||||||
_check_file_not_changed_by ./build_tools/update_torch_ods.sh include/torch-mlir/Dialect/Torch/IR/GeneratedTorchOps.td
|
|
||||||
|
|
||||||
echo ":::: Run Lazy Tensor Core e2e integration tests"
|
|
||||||
python -m e2e_testing.main --config=lazy_tensor_core -v
|
|
||||||
|
|
||||||
echo ":::: Run Linalg e2e integration tests"
|
|
||||||
python -m e2e_testing.main --config=linalg -v
|
|
||||||
|
|
||||||
# Dynamo is changing a lot in nightly versions, and thus the implementation
|
|
||||||
# tends to become incompatible to the stable version.
|
|
||||||
echo ":::: Run TorchDynamo e2e integration tests"
|
|
||||||
python -m e2e_testing.main --config=torchdynamo -v
|
|
||||||
;;
|
|
||||||
stable)
|
|
||||||
echo ":::: Test with stable torch"
|
|
||||||
|
|
||||||
# Disabled until the next stable PyTorch release (v2.1) is available
|
|
||||||
# echo ":::: Run Lazy Tensor Core e2e integration tests in experimental mode"
|
|
||||||
# python -m e2e_testing.main --config=lazy_tensor_core -v --ignore_failures
|
|
||||||
;;
|
|
||||||
*)
|
|
||||||
echo "Unrecognized torch version '$torch_version'"
|
|
||||||
exit 1
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
|
|
||||||
echo ":::: Run make_fx + TOSA e2e integration tests"
|
|
||||||
python -m e2e_testing.main --config=make_fx_tosa -v
|
|
||||||
|
|
||||||
echo ":::: Run TOSA e2e integration tests"
|
|
||||||
python -m e2e_testing.main --config=tosa -v
|
|
||||||
}
|
|
||||||
|
|
||||||
function setup_venv() {
|
|
||||||
local python_version="$1"
|
|
||||||
local torch_version="$2"
|
|
||||||
|
|
||||||
echo ":::: Setting up VENV with Python: $python_version PyTorch $torch_version"
|
|
||||||
python3 -m venv /main_checkout/torch-mlir/docker_venv
|
|
||||||
source /main_checkout/torch-mlir/docker_venv/bin/activate
|
|
||||||
|
|
||||||
echo ":::: pip installing dependencies"
|
|
||||||
python3 -m pip install --no-cache-dir -r /main_checkout/torch-mlir/externals/llvm-project/mlir/python/requirements.txt
|
|
||||||
|
|
||||||
case $torch_version in
|
|
||||||
nightly)
|
|
||||||
echo ":::: Using nightly dependencies"
|
|
||||||
python3 -m pip install --no-cache-dir -r /main_checkout/torch-mlir/requirements.txt
|
|
||||||
python3 -m pip install --no-cache-dir -r /main_checkout/torch-mlir/torchvision-requirements.txt
|
|
||||||
;;
|
|
||||||
stable)
|
|
||||||
echo ":::: Using stable dependencies"
|
|
||||||
python3 -m pip install --no-cache-dir torch torchvision --index-url https://download.pytorch.org/whl/cpu
|
|
||||||
python3 -m pip install --no-cache-dir -r /main_checkout/torch-mlir/build-requirements.txt
|
|
||||||
;;
|
|
||||||
*)
|
|
||||||
echo "Unrecognized torch version '$torch_version'"
|
|
||||||
exit 1
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
|
|
||||||
python3 -m pip install --no-cache-dir -r /main_checkout/torch-mlir/test-requirements.txt
|
|
||||||
}
|
|
||||||
|
|
||||||
function build_out_of_tree() {
|
|
||||||
local torch_from_bin="$1"
|
|
||||||
local python_version="$2"
|
|
||||||
local torch_version="$3"
|
|
||||||
echo ":::: Build out-of-tree Torch from binary: $torch_from_bin with Python: $python_version ($torch_version)"
|
|
||||||
|
|
||||||
local enable_ltc="ON"
|
|
||||||
if [[ "${torch_version}" == "stable" ]]
|
|
||||||
then
|
|
||||||
enable_ltc="OFF"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [ ! -d "/main_checkout/torch-mlir/llvm-build/lib/cmake/mlir/" ]
|
|
||||||
then
|
|
||||||
echo ":::: LLVM / MLIR is not built so building it first.."
|
|
||||||
cmake -GNinja -B/main_checkout/torch-mlir/llvm-build \
|
|
||||||
-DCMAKE_BUILD_TYPE=Release \
|
|
||||||
-DCMAKE_C_COMPILER=clang \
|
|
||||||
-DCMAKE_CXX_COMPILER=clang++ \
|
|
||||||
-DCMAKE_C_COMPILER_LAUNCHER=ccache \
|
|
||||||
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
|
|
||||||
-DCMAKE_EXE_LINKER_FLAGS_INIT="-fuse-ld=lld" \
|
|
||||||
-DCMAKE_MODULE_LINKER_FLAGS_INIT="-fuse-ld=lld" \
|
|
||||||
-DCMAKE_SHARED_LINKER_FLAGS_INIT="-fuse-ld=lld" \
|
|
||||||
-DLLVM_ENABLE_ASSERTIONS=ON \
|
|
||||||
-DLLVM_ENABLE_PROJECTS=mlir \
|
|
||||||
-DLLVM_TARGETS_TO_BUILD=host \
|
|
||||||
-DMLIR_ENABLE_BINDINGS_PYTHON=ON \
|
|
||||||
-DPython3_EXECUTABLE="$(which python3)" \
|
|
||||||
/main_checkout/torch-mlir/externals/llvm-project/llvm
|
|
||||||
cmake --build /main_checkout/torch-mlir/llvm-build
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Incremental builds come here directly and can run cmake if required.
|
|
||||||
cmake -GNinja -B/main_checkout/torch-mlir/build_oot \
|
|
||||||
-DCMAKE_C_COMPILER=clang \
|
|
||||||
-DCMAKE_CXX_COMPILER=clang++ \
|
|
||||||
-DCMAKE_C_COMPILER_LAUNCHER=ccache \
|
|
||||||
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
|
|
||||||
-DCMAKE_EXE_LINKER_FLAGS_INIT="-fuse-ld=lld" \
|
|
||||||
-DCMAKE_MODULE_LINKER_FLAGS_INIT="-fuse-ld=lld" \
|
|
||||||
-DCMAKE_SHARED_LINKER_FLAGS_INIT="-fuse-ld=lld" \
|
|
||||||
-DLLVM_DIR="/main_checkout/torch-mlir/llvm-build/lib/cmake/llvm/" \
|
|
||||||
-DMLIR_DIR="/main_checkout/torch-mlir/llvm-build/lib/cmake/mlir/" \
|
|
||||||
-DMLIR_ENABLE_BINDINGS_PYTHON=OFF \
|
|
||||||
-DTORCH_MLIR_ENABLE_LTC=${enable_ltc} \
|
|
||||||
-DTORCH_MLIR_USE_INSTALLED_PYTORCH="$torch_from_bin" \
|
|
||||||
-DTORCH_MLIR_SRC_PYTORCH_REPO=${TORCH_MLIR_SRC_PYTORCH_REPO} \
|
|
||||||
-DTORCH_MLIR_SRC_PYTORCH_BRANCH=${TORCH_MLIR_SRC_PYTORCH_BRANCH} \
|
|
||||||
-DTM_PYTORCH_INSTALL_WITHOUT_REBUILD=${TM_PYTORCH_INSTALL_WITHOUT_REBUILD} \
|
|
||||||
-DPython3_EXECUTABLE="$(which python3)" \
|
|
||||||
/main_checkout/torch-mlir
|
|
||||||
cmake --build /main_checkout/torch-mlir/build_oot
|
|
||||||
ccache -s
|
|
||||||
}
|
|
||||||
|
|
||||||
function test_out_of_tree() {
|
|
||||||
echo ":::: Test out-of-tree"
|
|
||||||
cmake --build /main_checkout/torch-mlir/build_oot --target check-torch-mlir-all
|
|
||||||
}
|
|
||||||
|
|
||||||
function clean_build() {
|
|
||||||
# clean up for recursive runs
|
|
||||||
local package="$1"
|
|
||||||
local python_version="$2"
|
|
||||||
echo ":::: Clean build dir $package $python_version"
|
|
||||||
rm -rf /main_checkout/torch-mlir/build /main_checkout/torch-mlir/llvm-build /main_checkout/torch-mlir/docker_venv /main_checkout/torch-mlir/libtorch
|
|
||||||
}
|
|
||||||
|
|
||||||
function build_torch_mlir() {
|
|
||||||
local torch_version="$1"
|
|
||||||
case $torch_version in
|
|
||||||
nightly)
|
|
||||||
echo ":::: Using nightly dependencies"
|
|
||||||
python -m pip install --no-cache-dir -r /main_checkout/torch-mlir/requirements.txt \
|
|
||||||
--extra-index-url https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html
|
|
||||||
CMAKE_GENERATOR=Ninja \
|
|
||||||
TORCH_MLIR_PYTHON_PACKAGE_VERSION=${TORCH_MLIR_PYTHON_PACKAGE_VERSION} \
|
|
||||||
python -m pip wheel -v -w /wheelhouse /main_checkout/torch-mlir \
|
|
||||||
-f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html \
|
|
||||||
-r /main_checkout/torch-mlir/whl-requirements.txt
|
|
||||||
;;
|
|
||||||
stable)
|
|
||||||
echo ":::: Using stable dependencies"
|
|
||||||
python3 -m pip install --no-cache-dir torch torchvision
|
|
||||||
python3 -m pip install --no-cache-dir -r /main_checkout/torch-mlir/build-requirements.txt
|
|
||||||
CMAKE_GENERATOR=Ninja \
|
|
||||||
TORCH_MLIR_PYTHON_PACKAGE_VERSION=${TORCH_MLIR_PYTHON_PACKAGE_VERSION} \
|
|
||||||
python -m pip wheel -v -w /wheelhouse /main_checkout/torch-mlir
|
|
||||||
;;
|
|
||||||
*)
|
|
||||||
echo "Unrecognized torch version '$torch_version'"
|
|
||||||
exit 1
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
}
|
|
||||||
|
|
||||||
function run_audit_wheel() {
|
|
||||||
local wheel_basename="$1"
|
|
||||||
local python_version="$2"
|
|
||||||
generic_wheel="/wheelhouse/${wheel_basename}-${TORCH_MLIR_PYTHON_PACKAGE_VERSION}-${python_version}-linux_${arch}.whl"
|
|
||||||
echo ":::: Auditwheel $generic_wheel"
|
|
||||||
auditwheel repair -w /wheelhouse "$generic_wheel"
|
|
||||||
rm "$generic_wheel"
|
|
||||||
}
|
|
||||||
|
|
||||||
function build_torch_mlir_core() {
|
|
||||||
python -m pip install --no-cache-dir -r /main_checkout/torch-mlir/build-requirements.txt
|
|
||||||
CMAKE_GENERATOR=Ninja \
|
|
||||||
TORCH_MLIR_PYTHON_PACKAGE_VERSION=${TORCH_MLIR_PYTHON_PACKAGE_VERSION} \
|
|
||||||
TORCH_MLIR_ENABLE_JIT_IR_IMPORTER=0 \
|
|
||||||
TORCH_MLIR_ENABLE_ONLY_MLIR_PYTHON_BINDINGS=1 \
|
|
||||||
python -m pip wheel -v -w /wheelhouse /main_checkout/torch-mlir
|
|
||||||
}
|
|
||||||
|
|
||||||
function clean_wheels() {
|
|
||||||
local wheel_basename="$1"
|
|
||||||
local python_version="$2"
|
|
||||||
echo ":::: Clean wheels $wheel_basename $python_version"
|
|
||||||
rm -f /wheelhouse/"${wheel_basename}"-*-"${python_version}"-*.whl
|
|
||||||
}
|
|
||||||
|
|
||||||
# Trampoline to the docker container if running on the host.
|
|
||||||
if [ -z "${__MANYLINUX_BUILD_WHEELS_IN_DOCKER-}" ]; then
|
|
||||||
for package in $TM_PACKAGES; do
|
|
||||||
echo "******************** BUILDING PACKAGE ${package} (host) *************"
|
|
||||||
run_on_host "${package} $@"
|
|
||||||
done
|
|
||||||
else
|
|
||||||
run_in_docker "$@"
|
|
||||||
fi
|
|
||||||
|
|
@ -1,144 +0,0 @@
|
||||||
#!/bin/bash
|
|
||||||
# Copyright 2022 The IREE Authors
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License v2.0 with LLVM Exceptions.
|
|
||||||
# See https://llvm.org/LICENSE.txt for license information.
|
|
||||||
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
|
||||||
|
|
||||||
# build_macos_packages.sh
|
|
||||||
# One stop build of IREE Python packages for MacOS. This presumes that
|
|
||||||
# dependencies are installed from install_macos_deps.sh. This will build
|
|
||||||
# for a list of Python versions synchronized with that script and corresponding
|
|
||||||
# with directory names under:
|
|
||||||
# /Library/Frameworks/Python.framework/Versions
|
|
||||||
#
|
|
||||||
# MacOS convention is to refer to this as major.minor (i.e. "3.9", "3.10").
|
|
||||||
# Valid packages:
|
|
||||||
# torch-mlir
|
|
||||||
|
|
||||||
set -eu -o errtrace
|
|
||||||
|
|
||||||
this_dir="$(cd "$(dirname "$0")" && pwd)"
|
|
||||||
repo_root="$(cd "$this_dir"/../../ && pwd)"
|
|
||||||
python_versions="${TORCH_MLIR_PYTHON_VERSIONS:-3.9 3.10 3.11}"
|
|
||||||
output_dir="${output_dir:-${this_dir}/wheelhouse}"
|
|
||||||
packages="${packages:-torch-mlir}"
|
|
||||||
|
|
||||||
PKG_VER_FILE="${repo_root}"/torch_mlir_package_version ; [ -f "$PKG_VER_FILE" ] && . "$PKG_VER_FILE"
|
|
||||||
export TORCH_MLIR_PYTHON_PACKAGE_VERSION="${TORCH_MLIR_PYTHON_PACKAGE_VERSION:-0.0.1}"
|
|
||||||
echo "Setting torch-mlir Python Package version to: ${TORCH_MLIR_PYTHON_PACKAGE_VERSION}"
|
|
||||||
|
|
||||||
# Note that this typically is selected to match the version that the official
|
|
||||||
# Python distributed is built at.
|
|
||||||
export MACOSX_DEPLOYMENT_TARGET="${TORCH_MLIR_OSX_TARGET:-11.0}"
|
|
||||||
export CMAKE_OSX_ARCHITECTURES="${TORCH_MLIR_OSX_ARCH:-arm64;x86_64}"
|
|
||||||
echo "CMAKE_OSX_ARCHITECTURES: $CMAKE_OSX_ARCHITECTURES"
|
|
||||||
echo "MACOSX_DEPLOYMENT_TARGET $MACOSX_DEPLOYMENT_TARGET"
|
|
||||||
|
|
||||||
# Disable LTC build on MacOS to avoid linkage issues
|
|
||||||
# https://github.com/llvm/torch-mlir/issues/1253
|
|
||||||
export TORCH_MLIR_ENABLE_LTC=0
|
|
||||||
|
|
||||||
function run() {
|
|
||||||
echo "Using python versions: ${python_versions}"
|
|
||||||
|
|
||||||
local orig_path="$PATH"
|
|
||||||
|
|
||||||
# Build phase.
|
|
||||||
for package in $packages; do
|
|
||||||
echo "******************** BUILDING PACKAGE ${package} ********************"
|
|
||||||
for python_version in $python_versions; do
|
|
||||||
python_dir="/Library/Frameworks/Python.framework/Versions/$python_version"
|
|
||||||
if ! [ -x "$python_dir/bin/python3" ]; then
|
|
||||||
echo "ERROR: Could not find python3: $python_dir (skipping)"
|
|
||||||
continue
|
|
||||||
fi
|
|
||||||
export PATH=$python_dir/bin:$orig_path
|
|
||||||
echo ":::: Python version $(python3 --version)"
|
|
||||||
case "$package" in
|
|
||||||
torch-mlir)
|
|
||||||
clean_wheels torch_mlir "$python_version"
|
|
||||||
build_torch_mlir torch_mlir "$python_version"
|
|
||||||
run_audit_wheel torch_mlir "$python_version"
|
|
||||||
;;
|
|
||||||
torch-mlir-core)
|
|
||||||
clean_wheels torch_mlir_core "$python_version"
|
|
||||||
build_torch_mlir_core torch_mlir_core "$python_version"
|
|
||||||
run_audit_wheel torch_mlir_core "$python_version"
|
|
||||||
;;
|
|
||||||
*)
|
|
||||||
echo "Unrecognized package '$package'"
|
|
||||||
exit 1
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
done
|
|
||||||
done
|
|
||||||
}
|
|
||||||
|
|
||||||
function build_torch_mlir() {
|
|
||||||
local wheel_basename="$1"
|
|
||||||
local python_version="$2"
|
|
||||||
rm -rf "$output_dir"/build_venv
|
|
||||||
python"${python_version}" -m venv "$output_dir"/build_venv
|
|
||||||
source "$output_dir"/build_venv/bin/activate
|
|
||||||
python"${python_version}" -m pip install -U pip
|
|
||||||
python"${python_version}" -m pip install -r "$repo_root"/pytorch-requirements.txt --extra-index-url https://download.pytorch.org/whl/nightly/cpu
|
|
||||||
python"${python_version}" -m pip install -r "$repo_root"/build-requirements.txt
|
|
||||||
CMAKE_GENERATOR=Ninja \
|
|
||||||
TORCH_MLIR_PYTHON_PACKAGE_VERSION=${TORCH_MLIR_PYTHON_PACKAGE_VERSION} \
|
|
||||||
MACOSX_DEPLOYMENT_TARGET=$MACOSX_DEPLOYMENT_TARGET \
|
|
||||||
CMAKE_OSX_ARCHITECTURES=$CMAKE_OSX_ARCHITECTURES \
|
|
||||||
python"${python_version}" -m pip wheel -v -w "$output_dir" "$repo_root" --extra-index-url https://download.pytorch.org/whl/nightly/cpu
|
|
||||||
deactivate
|
|
||||||
rm -rf "$output_dir"/build_venv
|
|
||||||
}
|
|
||||||
|
|
||||||
function build_torch_mlir_core() {
|
|
||||||
local wheel_basename="$1"
|
|
||||||
local python_version="$2"
|
|
||||||
rm -rf "$output_dir"/build_venv
|
|
||||||
python"${python_version}" -m venv "$output_dir"/build_venv
|
|
||||||
source "$output_dir"/build_venv/bin/activate
|
|
||||||
python"${python_version}" -m pip install -U pip delocate
|
|
||||||
python"${python_version}" -m pip install -r "$repo_root"/build-requirements.txt
|
|
||||||
CMAKE_GENERATOR=Ninja \
|
|
||||||
TORCH_MLIR_PYTHON_PACKAGE_VERSION=${TORCH_MLIR_PYTHON_PACKAGE_VERSION} \
|
|
||||||
MACOSX_DEPLOYMENT_TARGET=$MACOSX_DEPLOYMENT_TARGET \
|
|
||||||
CMAKE_OSX_ARCHITECTURES=$CMAKE_OSX_ARCHITECTURES \
|
|
||||||
TORCH_MLIR_ENABLE_JIT_IR_IMPORTER=0 \
|
|
||||||
TORCH_MLIR_ENABLE_ONLY_MLIR_PYTHON_BINDINGS=1 \
|
|
||||||
python"${python_version}" -m pip wheel -v -w "$output_dir" "$repo_root"
|
|
||||||
deactivate
|
|
||||||
rm -rf "$output_dir"/build_venv
|
|
||||||
}
|
|
||||||
|
|
||||||
function clean_wheels() {
|
|
||||||
local wheel_basename="$1"
|
|
||||||
local python_version="$2"
|
|
||||||
echo ":::: Clean wheels $wheel_basename $python_version"
|
|
||||||
rm -rf "$repo_root"/build/
|
|
||||||
rm -f "$output_dir"/"${wheel_basename}"-*-"${python_version//./}"-*.whl
|
|
||||||
}
|
|
||||||
|
|
||||||
function run_audit_wheel() {
|
|
||||||
set +x
|
|
||||||
local wheel_basename="$1"
|
|
||||||
local python_version="$2"
|
|
||||||
generic_wheel=$(ls "$output_dir"/"${wheel_basename}"-* | grep "${python_version//./}")
|
|
||||||
echo "Looking for $generic_wheel"
|
|
||||||
if [ -f "$generic_wheel" ]; then
|
|
||||||
echo "$generic_wheel found. Delocating it.."
|
|
||||||
rm -rf "$output_dir"/test_venv
|
|
||||||
python"${python_version}" -m venv "$output_dir"/test_venv
|
|
||||||
source "$output_dir"/test_venv/bin/activate
|
|
||||||
python"${python_version}" -m pip install -U pip
|
|
||||||
python"${python_version}" -m pip install -r "$repo_root"/pytorch-requirements.txt --extra-index-url https://download.pytorch.org/whl/nightly/cpu
|
|
||||||
python"${python_version}" -m pip install -r "$repo_root"/build-requirements.txt
|
|
||||||
python"${python_version}" -m pip install "$generic_wheel" --extra-index-url https://download.pytorch.org/whl/nightly/cpu
|
|
||||||
DYLD_LIBRARY_PATH="$output_dir"/test_venv/lib/python"${python_version}"/site-packages/torch/lib delocate-wheel -v "$generic_wheel"
|
|
||||||
deactivate
|
|
||||||
rm -rf "$output_dir"/test_venv
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
run
|
|
||||||
|
|
@ -1,40 +0,0 @@
|
||||||
#!/bin/bash
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License v2.0 with LLVM Exceptions.
|
|
||||||
# See https://llvm.org/LICENSE.txt for license information.
|
|
||||||
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
|
||||||
|
|
||||||
# build_upload_m1_snapshot.sh
|
|
||||||
# This is a wrapper to build_macos_packages.sh to be run on Apple M1 systems
|
|
||||||
# since GH Actions don't support M1 runners yet and Universal builds
|
|
||||||
# don't work for Torch-MLIR since we don't have universal PyTorch binaries
|
|
||||||
# This presumes that dependencies are installed from install_macos_deps.sh and
|
|
||||||
# you have the gh credentials to upload to the release
|
|
||||||
|
|
||||||
set -eu -o errtrace
|
|
||||||
|
|
||||||
this_dir="$(cd "$(dirname "$0")" && pwd)"
|
|
||||||
repo_root="$(cd "$this_dir"/../../ && pwd)"
|
|
||||||
output_dir="${output_dir:-${this_dir}/wheelhouse}"
|
|
||||||
rm -rf "${output_dir}"
|
|
||||||
|
|
||||||
git fetch --all
|
|
||||||
latest_snapshot=$(git for-each-ref --sort=creatordate --format '%(refname:short)' refs/tags | tail -n 1)
|
|
||||||
git checkout "${latest_snapshot}"
|
|
||||||
git submodule update --init
|
|
||||||
package_version=${latest_snapshot#"snapshot-"}
|
|
||||||
echo "Latest snapshot tag is: ${latest_snapshot}"
|
|
||||||
echo "Latest version is: ${package_version}"
|
|
||||||
|
|
||||||
export TORCH_MLIR_PYTHON_VERSIONS="3.9 3.10"
|
|
||||||
echo "Using Python Versions: ${TORCH_MLIR_PYTHON_VERSIONS}"
|
|
||||||
export TORCH_MLIR_PYTHON_PACKAGE_VERSION="${package_version}"
|
|
||||||
echo "Setting torch-mlir Python Package version to: ${TORCH_MLIR_PYTHON_PACKAGE_VERSION}"
|
|
||||||
|
|
||||||
TORCH_MLIR_OSX_ARCH=arm64 \
|
|
||||||
TORCH_MLIR_OSX_TARGET=11.0 \
|
|
||||||
TORCH_MLIR_PYTHON_PACKAGE_VERSION="${package_version}" \
|
|
||||||
TORCH_MLIR_PYTHON_VERSIONS="${TORCH_MLIR_PYTHON_VERSIONS}" \
|
|
||||||
"${repo_root}"/build_tools/python_deploy/build_macos_packages.sh
|
|
||||||
|
|
||||||
gh release upload "${latest_snapshot}" "${repo_root}"/build_tools/python_deploy/wheelhouse/torch*.whl
|
|
||||||
|
|
@ -1,30 +0,0 @@
|
||||||
# Uncomment if you want to install Python. GHA provides this
|
|
||||||
|
|
||||||
#Write-Host "Installing python"
|
|
||||||
|
|
||||||
#Start-Process choco 'install python --version=3.10.8' -wait -NoNewWindow
|
|
||||||
|
|
||||||
#Write-Host "python installation completed successfully"
|
|
||||||
|
|
||||||
#Write-Host "Reload environment variables"
|
|
||||||
#$env:Path = [System.Environment]::GetEnvironmentVariable("Path","Machine") + ";" + [System.Environment]::GetEnvironmentVariable("Path","User")
|
|
||||||
#Write-Host "Reloaded environment variables"
|
|
||||||
|
|
||||||
Write-Host "Installing Build Dependencies"
|
|
||||||
python -m venv .\mlir_venv\
|
|
||||||
.\mlir_venv\Scripts\Activate.PS1
|
|
||||||
pip install -r .\pytorch-requirements.txt
|
|
||||||
pip install -r .\build-requirements.txt
|
|
||||||
pip install delvewheel
|
|
||||||
Write-Host "Build Deps installation completed successfully"
|
|
||||||
|
|
||||||
Write-Host "Building torch-mlir"
|
|
||||||
$env:CMAKE_GENERATOR='Ninja'
|
|
||||||
$env:TORCH_MLIR_ENABLE_LTC='0'
|
|
||||||
python -m pip wheel -v -w wheelhouse ./ -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html -r whl-requirements.txt
|
|
||||||
|
|
||||||
Write-Host "Build completed successfully"
|
|
||||||
|
|
||||||
Write-Host "Fixing up wheel dependencies"
|
|
||||||
delvewheel repair --add-path .\build\cmake_build\tools\torch-mlir\python_packages\torch_mlir\torch_mlir\_mlir_libs --add-dll TorchMLIRAggregateCAPI.dll --no-dll 'c10.dll;torch_python.dll;torch_cpu.dll' -v (get-item .\wheelhouse\torch_mlir*.whl).FullName
|
|
||||||
Write-Host "All Done."
|
|
||||||
|
|
@ -1,21 +0,0 @@
|
||||||
#!/usr/bin/env bash
|
|
||||||
set -eo pipefail
|
|
||||||
|
|
||||||
echo "Building torch-mlir"
|
|
||||||
|
|
||||||
cmake -GNinja -Bbuild \
|
|
||||||
-DCMAKE_BUILD_TYPE=Release \
|
|
||||||
-DCMAKE_C_COMPILER_LAUNCHER=ccache \
|
|
||||||
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
|
|
||||||
-DLLVM_ENABLE_PROJECTS=mlir \
|
|
||||||
-DLLVM_TARGETS_TO_BUILD=host \
|
|
||||||
-DMLIR_ENABLE_BINDINGS_PYTHON=ON \
|
|
||||||
-DPython3_FIND_VIRTUALENV=ONLY \
|
|
||||||
-DLLVM_EXTERNAL_PROJECTS="torch-mlir" \
|
|
||||||
-DLLVM_EXTERNAL_TORCH_MLIR_SOURCE_DIR="$PWD" \
|
|
||||||
-DPython3_EXECUTABLE="$(which python)" \
|
|
||||||
$GITHUB_WORKSPACE/externals/llvm-project/llvm
|
|
||||||
|
|
||||||
cmake --build build --config Release
|
|
||||||
|
|
||||||
echo "Build completed successfully"
|
|
||||||
|
|
@ -1,63 +0,0 @@
|
||||||
#!/bin/zsh
|
|
||||||
# Copyright 2022 The IREE Authors
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License v2.0 with LLVM Exceptions.
|
|
||||||
# See https://llvm.org/LICENSE.txt for license information.
|
|
||||||
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
|
||||||
|
|
||||||
# Installs dependencies on MacOS necessary to build IREE.
|
|
||||||
# Additional dependencies (i.e MoltenVK) may be needed to use all functionality.
|
|
||||||
#
|
|
||||||
# Usage:
|
|
||||||
# sudo install_macos_deps.sh
|
|
||||||
|
|
||||||
set -eu -o pipefail
|
|
||||||
|
|
||||||
if [[ "$(whoami)" != "root" ]]; then
|
|
||||||
echo "ERROR: Must setup deps as root"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
PYTHON_INSTALLER_URLS=(
|
|
||||||
"https://www.python.org/ftp/python/3.11.2/python-3.11.2-macos11.pkg"
|
|
||||||
"https://www.python.org/ftp/python/3.10.10/python-3.10.10-macos11.pkg"
|
|
||||||
"https://www.python.org/ftp/python/3.9.13/python-3.9.13-macos11.pkg"
|
|
||||||
)
|
|
||||||
|
|
||||||
PYTHON_SPECS=(
|
|
||||||
3.11@https://www.python.org/ftp/python/3.11.2/python-3.11.2-macos11.pkg
|
|
||||||
3.10@https://www.python.org/ftp/python/3.10.5/python-3.10.5-macos11.pkg
|
|
||||||
3.9@https://www.python.org/ftp/python/3.9.13/python-3.9.13-macos11.pkg
|
|
||||||
)
|
|
||||||
|
|
||||||
for python_spec in $PYTHON_SPECS; do
|
|
||||||
python_version="${python_spec%%@*}"
|
|
||||||
url="${python_spec##*@}"
|
|
||||||
echo "-- Installing Python $python_version from $url"
|
|
||||||
python_path="/Library/Frameworks/Python.framework/Versions/$python_version"
|
|
||||||
python_exe="$python_path/bin/python3"
|
|
||||||
|
|
||||||
# Install Python.
|
|
||||||
if ! [ -x "$python_exe" ]; then
|
|
||||||
package_basename="$(basename "$url")"
|
|
||||||
download_path="/tmp/torch_mlir_python_install/$package_basename"
|
|
||||||
mkdir -p "$(dirname "$download_path")"
|
|
||||||
echo "Downloading $url -> $download_path"
|
|
||||||
curl "$url" -o "$download_path"
|
|
||||||
|
|
||||||
echo "Installing $download_path"
|
|
||||||
installer -pkg "$download_path" -target /
|
|
||||||
else
|
|
||||||
echo ":: Python version already installed. Not reinstalling."
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo ":: Python version $python_version installed:"
|
|
||||||
$python_exe --version
|
|
||||||
$python_exe -m pip --version
|
|
||||||
|
|
||||||
echo ":: Installing system pip packages"
|
|
||||||
$python_exe -m pip install --upgrade pip
|
|
||||||
$python_exe -m pip install --upgrade delocate
|
|
||||||
done
|
|
||||||
|
|
||||||
echo "*** All done ***"
|
|
||||||
|
|
@ -1,36 +0,0 @@
|
||||||
"""Scrapes the github releases API to generate a static pip-install-able releases page.
|
|
||||||
|
|
||||||
See https://github.com/llvm/torch-mlir/issues/1374
|
|
||||||
"""
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
|
|
||||||
import requests
|
|
||||||
|
|
||||||
# Parse arguments
|
|
||||||
parser = argparse.ArgumentParser()
|
|
||||||
parser.add_argument('owner', type=str)
|
|
||||||
parser.add_argument('repo', type=str)
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
# Get releases
|
|
||||||
response = requests.get(
|
|
||||||
f"https://api.github.com/repos/{args.owner}/{args.repo}/releases")
|
|
||||||
body = json.loads(response.content)
|
|
||||||
|
|
||||||
# Parse releases
|
|
||||||
releases = []
|
|
||||||
for row in body:
|
|
||||||
for asset in row['assets']:
|
|
||||||
releases.append((asset["name"], asset["browser_download_url"]))
|
|
||||||
|
|
||||||
# Output HTML
|
|
||||||
html = """<!DOCTYPE html>
|
|
||||||
<html>
|
|
||||||
<body>
|
|
||||||
"""
|
|
||||||
for name, url in releases:
|
|
||||||
html += f" <a href='{url}'>{name}</a><br />\n"
|
|
||||||
html += """ </body>
|
|
||||||
</html>"""
|
|
||||||
print(html)
|
|
||||||
|
|
@ -225,106 +225,6 @@ is built as part of the LLVM project along with MLIR.
|
||||||
We mount a ccache and pip cache inside the docker container to speed up iterative builds. Iterative
|
We mount a ccache and pip cache inside the docker container to speed up iterative builds. Iterative
|
||||||
builds should be as fast as running without docker.
|
builds should be as fast as running without docker.
|
||||||
|
|
||||||
### In-Tree builds
|
|
||||||
|
|
||||||
Build MLIR and Torch-MLIR together as part of the LLVM repo.
|
|
||||||
|
|
||||||
```shell
|
|
||||||
TM_PACKAGES="in-tree" ./build_tools/python_deploy/build_linux_packages.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
### Out-of-Tree builds
|
|
||||||
|
|
||||||
Build LLVM/MLIR first and then build Torch-MLIR referencing that build
|
|
||||||
```shell
|
|
||||||
TM_PACKAGES="out-of-tree" ./build_tools/python_deploy/build_linux_packages.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
### Release builds
|
|
||||||
|
|
||||||
Build in a manylinux Docker image so we can upload artifacts to PyPI.
|
|
||||||
|
|
||||||
```shell
|
|
||||||
TM_PACKAGES="torch-mlir" ./build_tools/python_deploy/build_linux_packages.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
### Mimicing CI+Release builds
|
|
||||||
|
|
||||||
If you wanted to build all the CIs locally
|
|
||||||
|
|
||||||
```shell
|
|
||||||
TM_PACKAGES="out-of-tree in-tree" ./build_tools/python_deploy/build_linux_packages.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
If you wanted to build all the CIs and the Release builds (just with Python 3.10 since most other Python builds are redundant)
|
|
||||||
|
|
||||||
```shell
|
|
||||||
TM_PACKAGES="torch-mlir out-of-tree in-tree" TM_PYTHON_VERSIONS="cp310-cp310" ./build_tools/python_deploy/build_linux_packages.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
Note: The Release docker still runs as root so it may generate some files owned by root:root. We hope to move it to run as a user in the future.
|
|
||||||
|
|
||||||
### Cleaning up
|
|
||||||
|
|
||||||
Docker builds tend to leave a wide variety of files around. Luckily most are owned by the user but there are still some that need to be removed
|
|
||||||
as superuser.
|
|
||||||
|
|
||||||
```shell
|
|
||||||
rm -rf build build_oot llvm-build docker_venv externals/pytorch/build .ccache
|
|
||||||
```
|
|
||||||
|
|
||||||
## Building your own Docker image
|
|
||||||
|
|
||||||
If you would like to build your own docker image (usually not necessary). You can run:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
cd ./build_tools/docker
|
|
||||||
docker build -t your-name/torch-mlir-ci --no-cache .
|
|
||||||
```
|
|
||||||
|
|
||||||
### Other configurable environmental variables
|
|
||||||
|
|
||||||
The following additional environmental variables can be used to customize your docker build:
|
|
||||||
|
|
||||||
* Custom Release Docker image:
|
|
||||||
Defaults to `stellaraccident/manylinux2014_x86_64-bazel-5.1.0:latest`
|
|
||||||
```shell
|
|
||||||
TM_RELEASE_DOCKER_IMAGE="stellaraccident/manylinux2014_x86_64-bazel-5.1.0:latest"
|
|
||||||
```
|
|
||||||
* Custom CI Docker image:
|
|
||||||
Defaults to `powderluv/torch-mlir-ci:latest`. This assumes an Ubuntu LTS like image. You can build your own with `./build_tools/docker/Dockerfile`
|
|
||||||
```shell
|
|
||||||
TM_CI_DOCKER_IMAGE="powderluv/torch-mlir-ci:latest"
|
|
||||||
```
|
|
||||||
|
|
||||||
* Custom Python Versions for Release builds:
|
|
||||||
Version of Python to use in Release builds. Ignored in CIs. Defaults to `cp38-cp38 cp39-cp39 cp310-cp310`
|
|
||||||
```shell
|
|
||||||
TM_PYTHON_VERSIONS="cp38-cp38 cp39-cp39 cp310-cp310"
|
|
||||||
```
|
|
||||||
|
|
||||||
* Location to store Release build wheels
|
|
||||||
```shell
|
|
||||||
TM_OUTPUT_DIR="./build_tools/python_deploy/wheelhouse"
|
|
||||||
```
|
|
||||||
|
|
||||||
* What "packages" to build:
|
|
||||||
Defaults to torch-mlir. Options are `torch-mlir out-of-tree in-tree`
|
|
||||||
```shell
|
|
||||||
TM_PACKAGES="torch-mlir out-of-tree in-tree"
|
|
||||||
```
|
|
||||||
* Use pre-built Pytorch:
|
|
||||||
Defaults to using pre-built Pytorch. Setting it to `OFF` builds from source
|
|
||||||
```shell
|
|
||||||
TM_USE_PYTORCH_BINARY="OFF"
|
|
||||||
```
|
|
||||||
* Skip running tests
|
|
||||||
Skip running tests if you want quick build only iteration. Default set to `OFF`
|
|
||||||
```shell
|
|
||||||
TM_SKIP_TESTS="OFF"
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
## Build Python Packages
|
## Build Python Packages
|
||||||
|
|
||||||
We have preliminary support for building Python packages. This can be done
|
We have preliminary support for building Python packages. This can be done
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue