Commit Graph

57 Commits (28a7738189b9d66c72fc530d27852002d0b32420)

Author SHA1 Message Date
Sean Silva 496051163f Rename npcomp-run-mlir to refback-run
This better represents its limited scope. This was causing confusion --
people were feeding it higher level ops that require frontend lowering.
2021-08-03 18:24:24 -07:00
Sean Silva 719f0cd709 Minor wordsmithing to README 2021-08-03 14:59:58 -07:00
Sean Silva a153cf4ef2 Refresh Dockerfile and instructions.
Related to https://github.com/llvm/mlir-npcomp/issues/266
2021-08-03 14:54:39 -07:00
Sean Silva 453e29ea05 Add E2E support for tests with heavy dependencies (heavydep tests).
The tests use the same (pure-Python) test framework as the
normal torchscript_e2e_test.sh, but the tests are added in
`build_tools/torchscript_e2e_heavydep_tests` instead of
`frontends/pytorch/e2e_testing/torchscript`. Any needed dependencies can
easily be configured in generate_serialized_tests.sh.

We add an initial machine translation model with a complex set of
dependencies to seed the curriculum there. I verified that this model
gets to the point of MLIR import (it fails there with a segfault due to
not being able to import the "Any" type).

This required moving a few files from the `torch_mlir` Python module
into multiple modules to isolate the code that depends on our C++
extensions (which now live in `torch_mlir` and
`torch_mlir_torchscript_e2e_test_configs`) from the pure Python code
(which now lives in `torch_mlir_torchscript`). This is an entirely
mechanical change, and lots of imports needed to be updated.

The dependency graph is:
```
       torch_mlir_torchscript_e2e_test_configs
                  /              |
                 /               |
                /                |
               V                 V
torch_mlir_torchscript       torch_mlir
```

The `torch_mlir_torchscript_e2e_test_configs` are then dependency-injected
into the `torch_mlir_torchscript` modules to successfully assemble a
working test harness (the code was already structured this way, but this
new file organization allows the isolation from C++ code to actually
happen).  This isolation is critical to allowing the serialized programs
to be transported across PyTorch versions and for the test harness to be
used seamlessly to generate the heavydep tests.

Also:
- Extend `_Tracer` class to support nested property (submodule) accesses.

Recommended review order:
- "user-level" docs in README.md
- code in `build_tools/torchscript_e2e_heavydep_tests`.
- changes in `torch_mlir_torchscript/e2e_test/framework.py`
- misc mechanical changes.
2021-08-03 14:09:56 -07:00
Stella Laurenzo 445472c51e Build packages for npcomp-torch.
* Adds a minimal setup.py for frontends/pytorch
* Makes npcomp-core export its headers and libraries
* Adds a script to build packages.
* Adds CI step to package and smoke test.
* Will need some more tweaks and coordination prior to deploying (version locking etc).
2021-07-29 19:58:59 -07:00
Yi Zhang 6fbf94f0b2 Update readme and scripts for setting the new PYTHONPATH
Add scripts for generating .env and update instructions in README.
2021-07-28 15:06:40 -04:00
Stella Laurenzo 2dbab50444
Rework the python build to a static assembly of MLIR+NPCOMP (#251)
* Adapt to python build system updates.

* Bump llvm to 310c9496d80961188e8d8f8ad306cdf44bd7541f (includes python build updates)
* Adds refback C-API.
* Re-layers all python builds.
* Rework CI.
2021-07-27 16:10:10 -07:00
Sean Silva c289d83407 Fix README.md
Use the new `tools/torchscript_e2e_test.sh`.

Also, fix a few whitespace/comment issues.
2021-07-02 10:57:21 -07:00
Sean Silva d5108b9dc1 Add IREE support in TorchScript e2e tests.
- Add support for "expected failures" in test reporting. The new error
  reports look like
  [this](https://gist.github.com/silvasean/6ffd95e1d55302b699673da201da210d).
  - We will now be able to put these tests into CI, since the harness
    understand which tests are expected to pass and fail.
- Refactor RefBackendTestConfig to NpcompBackendTestConfig which
  supports both RefBackend and IREE.
- Add instructions for installing IREE dependencies (both from packages
  and for local builds of IREE)
- Add `tools/torchscript_e2e_test.sh` for invoking the e2e test
  harness (this makes invoking a bit easier, as it doesn't rely on a
  loose Python invocation).
2021-06-30 16:19:25 -07:00
Sean Silva 390d39a96c
README: mention installing nightly pytorch. 2021-04-27 11:37:09 -07:00
Sean Silva 3f9760dc33
Add communication channels to README 2021-04-02 13:00:47 -07:00
stephenneuendorffer f96c05abd4
Update README.md
Fix more links
2021-01-26 10:02:25 -08:00
stephenneuendorffer 142de3bab3
Update README.md
Links were broken
2021-01-26 09:52:46 -08:00
Stella Laurenzo f6d7ee06ef Make torch_mlir compatible with binary PyTorch installations.
* This has been anticipated for a long time in that it is quite hard to keep C++ binary compatibility across a system landscape as diverse as PyTorch, LLVM, and this project. This is why we based the PyTorch extension on the MLIR and NPCOMP C APIs only: that is the only sane linkage story for the entire matrix.
* Removes the few LLVM'isms in torch_mlir that had snuck in, using either STL or PyTorch support utilities. The new rule here is that LLVM C++ includes are forbidden at this level and (as stated in the design), torch_mlir should use the PyTorch runtime and support libraries (not introduce an incidental C++ dependency on LLVM).
* Also deletes mnist-playground as it was proving impossible to keep the grid of PyTorch vs system ABI divisions functioning. I am open to a less drastic course here (optional/disabled by default?)
* This gets us pretty close to just using PyTorch's extension builder API, which will be nice for distribution (i.e. it integrates well with the PyTorch ecosystem for deployment). I ended up just simplifying the in-tree CMake support for now.
* Fixes #138
2020-12-14 09:51:00 -08:00
Sean Silva 45ca371129 cmake_configure.sh: Add mlir native modules to PYTHONPATH
Also, update README.md to use the canonical .env file written by
`cmake_configure.sh`.
2020-11-20 17:29:57 -08:00
Stella Laurenzo 6850295ec5 Teach cmake how to find the installed PyTorch.
* In most situations, this eliminates the need to explicitly set a path to the Torch cmake files.
* Also upgrades to new Python3 find package. (should eliminate 2.x mismatches)
* Since PyTorch is located by asking Python where it is, this eliminates a lot of causes of mismatch. (one source of truth)
2020-11-13 17:19:25 -08:00
Stella Laurenzo 59b7c559f4 Tweak build flags for efficiency and document building without a container.
* Enables -gsplit-dwarf for both LLVM and NPCOMP, reducing the occurrence of the ~GB scale binaries.
* CMake shared linking seems incompatible with this, so shared objects are still "too big" but there are few of them.
* Reduces disk thrash on clean/install of everything.
2020-11-03 13:46:46 -08:00
Stella Laurenzo 9e52f6235b More progress on PyTorch acap device capture.
* Now gets far enough to capture batch_norm.
* Has some issues still with in-place ops.
* Can materialize constants.
* Includes an upgrade to PyTorch nightly, which has important bug fixes for fallback and boxed kernel dispatch.
* Fixes #78, #79, #80.
* Will do more testing in a follow-up once further bugs are fixed that facilitate getting at the other features.
2020-10-15 21:43:21 -07:00
Sean Silva d6b05c507a Fix up the docker script / instructions after attempting to use it. 2020-10-09 10:27:06 -07:00
Stella Laurenzo 51d51241b4 Add scripts/documentation for VSCode setup with a docker dev image.
* Forks a subset of my shell functions into docker_shell_funcs.sh, specifically needed to create docker images that run as yourself.
* Extends the readme with the three command bootstrap to get a dev container running.
* Step by step instructions for configuring VSCode for Intellisense in either npcomp or LLVM.
* Changes LLVM config options to enable tests. This setup is now suitable for upstream changes as well without rebuilding.
2020-10-07 21:27:20 -07:00
Stella Laurenzo 47c3a9f461 Add docker image/instructions for building against pytorch 1.6. 2020-09-24 17:40:25 -07:00
Stella Laurenzo 361abebb51 Update README to reference published docker tag. 2020-09-16 23:12:05 -07:00
Stella Laurenzo 678989a321
Update docker, instructions and some fixes for the pytorch 1.3 build. (#45)
* Includes pybind11 directly (for some reason using the pytorch helper header for this depends on a source file not in the image).
* Installs nnpack into the image.
* Installs new-clang and LLD and configures environment to use it (otherwise, link time is terrible).
* Fixes a gcc compile error (in the off chance you build with default gcc compiler).
* Tests are failing based on some dialect registration stuff that must not have been factored correctly. Will followup with a fix.
2020-09-16 21:57:46 -07:00
Stella Laurenzo 4c37aed841 Update build instructions to use the submodule for llvm.
* Previous instructions were referring to the option to use an external llvm-project checkout with a stale version hash.
2020-08-28 16:20:55 -07:00
Stella Laurenzo 77b235f621
Create frontends/pytorch directory. (#31)
* Adds/updates readmes with some notes about code organization and direction.
* Meant to prepare a space for upcoming integration of #30.
2020-08-18 09:43:20 -07:00
Phoenix Meadowlark ac3d402cd6 Update broken links 2020-08-04 18:55:46 -07:00
Stella Laurenzo 38abe99805 Collapse python_native/ into python/.
* These were separated originally for layering reasons that no longer apply.
* Most of the python extension code is under lib/ with just the module setup in python/.
2020-08-03 17:46:34 -07:00
Stella Laurenzo 478f35620d
License and readme changes to align with inclusion in LLVM. (#1)
* Updates the LICENSE to the same verbiage as used in the circt project.
* Adds the incubator disclaimer to the README.
* Reworks the introduction of the README to more accurately reflect the eventual scope.
* There is a fair amount of further rework of the repo that needs to take place. This is just the minimal cosmetic changes now that it is part of LLVM.
2020-07-31 20:53:09 -07:00
Stella Laurenzo aeb422b030 Some fixes to get npcomp building and passing on windows.
There is more that can be done here, but this gets it minimally working.
2020-07-01 21:28:04 -07:00
Stella Laurenzo 046751254f Refactor old tracing tests and remove deprecated ops.
* Old doctests to run under lit.
* Old custom filecheck tests -> pytest directory (under lit).
* Rename some old ufunc ops in the tracer.
2020-06-29 16:19:03 -07:00
Stella Laurenzo 2242f48228 Fix broken link 2020-06-20 12:01:49 -07:00
Stella Laurenzo 12d8459d73 Rework README, add docs, and move some scripts to better locations accordingly. 2020-06-20 11:56:13 -07:00
Stella Laurenzo fc5f10c5c5 Bump revision and fix issues.
* llvm revision = 4836188ad9b3334b0c1e055d45ccaa54ed797e4b
* iree revision = 091482e8fdf599d6cb5c701d5b3ccb27fc66c014
2020-06-19 10:38:51 -07:00
Stella Laurenzo 308a54c3d0 Bump llvm-project to 52cae05e087b3d4fd02849fc37c387c720055ffb (2020/6/10).
* Fixes compile errors from upstream.
* XFAIL several tests that are now failing to legalize (will hand off to Sean).
2020-06-11 16:10:05 -07:00
Stella Laurenzo 8280b86c05 Aggregate all lit test targets under check-npcomp. 2020-06-07 14:35:58 -07:00
Stella Laurenzo 639826183d Update readme with test command. 2020-06-07 14:30:39 -07:00
Stella Laurenzo f67c5a42dc Add notes on AST extraction. 2020-06-06 21:42:34 -07:00
Stella Laurenzo bb871e7601 Bump llvm-project version to 232d348c6eff8493fc016b1ea8a99db0e5620d81. 2020-06-02 16:17:23 -07:00
Sean Silva 67b129af7a Add tools/bash_helpers.sh
This script (make sure to `source` it!) defines a number of handy
aliases which make compiler development more convenient.

Please feel free to add anything you find useful here.
2020-05-29 16:39:00 -07:00
Sean Silva ccd5754b88 Rename `check-npcomp-opt` to just `check-npcomp`.
It runs npcomp-run-mlir as well now, so having `-opt` in the name is
confusing.
2020-05-29 16:12:10 -07:00
Sean Silva 1fed1cb016 Update llvm-project to 753a21928413f8a7e76978cb1354e09150e114e0 2020-05-21 13:09:06 -07:00
Sean Silva 83db558db9 Update llvm-project to 310d32cb80a611e6384a921e85607fea05841f26 2020-05-11 15:12:47 -07:00
Stella Laurenzo 950ba12426 Bump llvm-project to 3af85fa8f06220b43f03f26de216a67be4568fe7. 2020-05-08 20:42:40 -07:00
Sean Silva f394e12d86 Add tools/test_all.sh
We should look into having a `ninja check-npcomp` that runs everything
with lit so that we get decent test multithreading.
We can look to how LLVM does its gtest tests ("unittests") for
inspiration.
2020-05-08 18:13:19 -07:00
Stella Laurenzo 0092b912ab Update all python imports to be absolute and use a .env file to set the path correctly.
This makes things just work for debugging in VSCode.
2020-05-06 23:25:04 -07:00
Stella Laurenzo 3611958b11 Move python native library to python_native/_npcomp...so.
This allows binary and source packages to exist at different physical paths.
2020-05-06 22:44:12 -07:00
Stella Laurenzo 714bc01c02 Bump llvm version to 0c4aab27b3da05dd1b0c0c39472525325fda5e23.
* Fixes some api-change breakages.
* Deletes edsc module now that the IR bindings are working (and since it broke with this update).
2020-05-05 21:00:49 -07:00
Stella Laurenzo f8cedd8656 Add known good llvm-project commit hash to readme. 2020-05-05 14:00:06 -07:00
Stella Laurenzo 9a0f498774 Add new gist to the design notes. 2020-05-05 13:49:05 -07:00
Stella Laurenzo 1f136f9dba Update readme 2020-05-02 19:54:49 -07:00