2020-04-27 06:50:23 +08:00
|
|
|
# npcomp - An aspirational MLIR based numpy compiler
|
|
|
|
|
|
2020-04-27 11:14:20 +08:00
|
|
|
This is a research prototype of MLIR dialects for representing
|
|
|
|
|
numpy programs, and a set of reference tracing/compiler tools.
|
2020-04-30 07:11:18 +08:00
|
|
|
The primary purpose at this point is to establish a solid modeling
|
|
|
|
|
of restricted Python programs and Numpy based computations in MLIR.
|
|
|
|
|
While this project will provide some reference implementations to prove
|
|
|
|
|
the design, the intention is to align this with the broader set of
|
|
|
|
|
tools that exist at this level of abstraction.
|
2020-04-27 06:50:23 +08:00
|
|
|
|
2020-04-30 01:55:35 +08:00
|
|
|
## Design Notes
|
|
|
|
|
|
|
|
|
|
As I work through things, I've been jotting down some design notes:
|
|
|
|
|
|
|
|
|
|
* [Type Extraction - April 15, 2020](https://gist.github.com/stellaraccident/ec1ab0f633cfca0a05866fd77705b4e4)
|
|
|
|
|
* [Ufunc modeling Part 1 - April 29, 2020](https://gist.github.com/stellaraccident/4fcd2a24a66b6588f92b22b2b8ab974f)
|
2020-05-06 04:49:05 +08:00
|
|
|
* [Array funcs and op granularity - May 5, 2020](https://gist.github.com/stellaraccident/2c11652cfdee1457921bc7c98807b462)
|
2020-04-30 01:55:35 +08:00
|
|
|
|
2020-04-27 07:32:10 +08:00
|
|
|
## Quick start
|
2020-04-27 06:50:23 +08:00
|
|
|
|
2020-04-27 07:32:10 +08:00
|
|
|
```
|
2020-04-30 07:33:48 +08:00
|
|
|
LLVM_VERSION=10
|
|
|
|
|
export CC=clang-$LLVM_VERSION
|
|
|
|
|
export CXX=clang++-$LLVM_VERSION
|
|
|
|
|
export LDFLAGS=-fuse-ld=$(which ld.lld-$LLVM_VERSION)
|
2020-04-27 07:32:10 +08:00
|
|
|
export LLVM_SRC_DIR=/path/to/llvm-project
|
2020-04-30 07:33:48 +08:00
|
|
|
|
2020-05-06 05:00:06 +08:00
|
|
|
# Check out last known good commit.
|
2020-05-22 04:09:06 +08:00
|
|
|
(cd $LLVM_SRC_DIR && git checkout 753a21928413f8a7e76978cb1354e09150e114e0)
|
2020-05-06 05:00:06 +08:00
|
|
|
|
2020-04-27 07:32:10 +08:00
|
|
|
./tools/install_mlir.sh
|
|
|
|
|
./tools/cmake_configure.sh
|
|
|
|
|
|
2020-05-08 11:28:40 +08:00
|
|
|
|
|
|
|
|
# ./tools/test_all.sh runs all of these commands.
|
2020-04-27 07:32:10 +08:00
|
|
|
cd build
|
|
|
|
|
ninja
|
2020-04-30 08:05:45 +08:00
|
|
|
ninja check-npcomp-opt
|
|
|
|
|
# Note: currently, python tests run separately
|
2020-04-27 07:32:10 +08:00
|
|
|
./python/run_tests.py
|
2020-04-27 06:50:23 +08:00
|
|
|
```
|
|
|
|
|
|
2020-05-07 13:44:12 +08:00
|
|
|
## Interactive Use
|
2020-05-03 10:54:49 +08:00
|
|
|
|
2020-05-07 13:44:12 +08:00
|
|
|
The cmake configuration populates symlinks in the `build/python` directory
|
|
|
|
|
mirroring the source layout. This allows edit-run without rebuilding (unless
|
|
|
|
|
if files are added/removed).
|
2020-04-27 06:50:23 +08:00
|
|
|
|
2020-05-07 13:44:12 +08:00
|
|
|
Configuring the `PYTHONPATH` should be sufficient to run any interactive
|
|
|
|
|
tooling (`python3`, Jupyter/Colab, etc).
|
2020-04-27 06:50:23 +08:00
|
|
|
|
2020-05-07 13:44:12 +08:00
|
|
|
```shell
|
|
|
|
|
export PYTHONPATH="$(realpath build/python):$(realpath build/python_native)"
|
|
|
|
|
```
|
2020-04-27 06:50:23 +08:00
|
|
|
|
2020-05-07 13:44:12 +08:00
|
|
|
The `run_tests.py` script is special in that it sets up the PYTHONPATH
|
|
|
|
|
correctly when run.
|
2020-04-27 06:50:23 +08:00
|
|
|
|
2020-05-07 14:25:04 +08:00
|
|
|
Note that running the `cmake_configure.sh` script will also output a `.env`
|
|
|
|
|
file in the workspace folder with the correct PYTHONPATH set. This allows
|
|
|
|
|
tools like VSCode to work by default for debugging.
|
|
|
|
|
|
2020-05-07 13:44:12 +08:00
|
|
|
### Things to look at:
|
2020-04-27 11:14:20 +08:00
|
|
|
|
2020-05-07 13:44:12 +08:00
|
|
|
* `python/npcomp/tracing/mlir_trace_test.py` : Simple test case of tracing a function to an MLIR module.
|
2020-04-27 06:50:23 +08:00
|
|
|
|
|
|
|
|
Notes:
|
|
|
|
|
|
|
|
|
|
* Python sources are symlinked to the output directory at configure time.
|
|
|
|
|
Adding sources will require a reconfigure. Editing should not.
|
|
|
|
|
* It is a very common issue to have both python 2.7 (aka. "python") and python
|
|
|
|
|
3.x (aka. "python3") on a system at a time (and we can only hope that one
|
|
|
|
|
day this ends). Since the native library at development time binds to a
|
|
|
|
|
specific version, if you try to run with a different python, you will get
|
|
|
|
|
an error about the "native" module not being found.
|
2020-04-27 11:14:20 +08:00
|
|
|
|
