llvm
/
torch-mlir
mirror of https://github.com/llvm/torch-mlir
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
torch-mlir/e2e_testing/torchscript/xfail_sets.py

28 lines
1.2 KiB
Python
Raw Normal View History

Dual license the torch-mlir project. This commit (with approval from all contributors) dual licenses the torch-mlir project under both the standard LLVM license and the standard PyTorch license. This will facilitate moving code between torch-mlir and the two upstream projects. The standard file comment is now: ``` // This file is licensed under the Apache License v2.0 with LLVM Exceptions. // See https://llvm.org/LICENSE.txt for license information. // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception // Also available under a BSD-style license. See LICENSE. ``` See `LICENSE` in the project root for the terms of both licenses.
2021-09-30 00:03:40 +08:00
# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
# See https://llvm.org/LICENSE.txt for license information.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
# Also available under a BSD-style license. See LICENSE.
Add IREE support in TorchScript e2e tests. - Add support for "expected failures" in test reporting. The new error reports look like [this](https://gist.github.com/silvasean/6ffd95e1d55302b699673da201da210d). - We will now be able to put these tests into CI, since the harness understand which tests are expected to pass and fail. - Refactor RefBackendTestConfig to NpcompBackendTestConfig which supports both RefBackend and IREE. - Add instructions for installing IREE dependencies (both from packages and for local builds of IREE) - Add `tools/torchscript_e2e_test.sh` for invoking the e2e test harness (this makes invoking a bit easier, as it doesn't rely on a loose Python invocation).
2021-07-01 05:13:21 +08:00
# This file describes the sets of tests expected to fail for each config.
# This information is deliberately kept in a side table, rather than
# in-situ on the test, as a deliberate layering decision: tests should
# have unique keys to identify them and enable side tables of various kinds
# (this includes down into lower parts of the stack, where a side table
# might be used to keep more elaborate sets of testing configurations).
# Lists of tests that fail to even reach the backends.
Port the bulk of the remaining code to torch-mlir This leaves no real code outside torch-mlir. This also renames the "npcomp backend contract" to "linalg on tensors backend contract" as the name of the abstraction layer that RefBackend (IREE too) accepts.
2021-09-28 02:36:44 +08:00
# These represent further work needed in torch-mlir to lower them properly
Add IREE support in TorchScript e2e tests. - Add support for "expected failures" in test reporting. The new error reports look like [this](https://gist.github.com/silvasean/6ffd95e1d55302b699673da201da210d). - We will now be able to put these tests into CI, since the harness understand which tests are expected to pass and fail. - Refactor RefBackendTestConfig to NpcompBackendTestConfig which supports both RefBackend and IREE. - Add instructions for installing IREE dependencies (both from packages and for local builds of IREE) - Add `tools/torchscript_e2e_test.sh` for invoking the e2e test harness (this makes invoking a bit easier, as it doesn't rely on a loose Python invocation).
2021-07-01 05:13:21 +08:00
# to the backend contract.
Add --external-config option to tools/torchscript_e2e_test.sh This is a simple way for externals to plug their backends into the test suite. They just implement the `TestConfig` class for their backend and write a small script that exposes it. I have a pending PR for iree-samples that successfully integrates this.
2021-10-05 02:24:12 +08:00
COMMON_TORCH_MLIR_LOWERING_XFAILS = {
Add a basic TOSA E2E backend. We lower through linalg-on-tensors and use RefBackend to run it. This adds enough support for a "tanh" op. Adding more ops should be fairly mechanical now that things are wired up. Run with: ``` ./tools/torchscript_e2e_test.sh -c tosa ``` The backend structure is very similar to linalg-on-tensors based E2E backends and is a nice parallel (see `tosa_backend.py`). Actually, this forced a nice refactoring to the layering here. We removed `torchscript-module-to-linalg-on-tensors-backend-pipeline` and instead require separately running ``` torchscript-function-to-torch-backend-pipeline,torch-backend-to-linalg-on-tensors-backend-pipeline ``` This highlights the step that lowers to the "torch backend contract" of cleaned up `torch` dialect ops is a critical step in the lowering. Going forward, that is the key load-bearing contract of the torch-mlir project, not the linalg-on-tensors backend contract. Recommended review order: - `TorchToTosa.cpp` / `TorchToTosa/basic.mlir` - `python/torch_mlir_e2e_test/torchscript/configs/tosa_backend.py` and the new `utils.py` file there. - `python/torch_mlir_e2e_test/tosa_backends/linalg_on_tensors.py` and `abc.py` in that directory for the TOSA backend e2e interface. - other misc mechanical changes
2021-10-08 10:07:03 +08:00
"QuantizedMLP_basic",
Add basic support for lists. This plumbs through a vertical slice of support for lists. The main chunk of new code here is AnnotateABIPass which captures the program signature at the Torch backend contract layer, right before we start `TorchConversion`. The `TorchConversion` lowering process is lossy w.r.t. types, so it's necessary to do this for all targets in general. Like using `!iree.list` directly, we use IREE's ABI annotation representation for this, although there is nothing very IREE-specific about it (see https://github.com/google/iree/blob/main/docs/developers/design_docs/function_abi.md) We change `ListLiteralModule_basic` to use `!torch.int` because IREE doesn't support f64 yet (and we don't yet have a way for users to say that they want `!torch.float` to lower as f32). Recommended review order: - AnnotateABIPass and tests - Arg marshaling in npcomp_backend.py and `iree.py` - Updates to `list_programs.py` / `xfail_sets.py` - Moving DeleteDeadIREEListsPass to Backend/Common, so that backends that don't support lists can use it. RefBackend uses that pass, for example.
2021-09-03 03:47:59 +08:00
}
Add a basic TOSA E2E backend. We lower through linalg-on-tensors and use RefBackend to run it. This adds enough support for a "tanh" op. Adding more ops should be fairly mechanical now that things are wired up. Run with: ``` ./tools/torchscript_e2e_test.sh -c tosa ``` The backend structure is very similar to linalg-on-tensors based E2E backends and is a nice parallel (see `tosa_backend.py`). Actually, this forced a nice refactoring to the layering here. We removed `torchscript-module-to-linalg-on-tensors-backend-pipeline` and instead require separately running ``` torchscript-function-to-torch-backend-pipeline,torch-backend-to-linalg-on-tensors-backend-pipeline ``` This highlights the step that lowers to the "torch backend contract" of cleaned up `torch` dialect ops is a critical step in the lowering. Going forward, that is the key load-bearing contract of the torch-mlir project, not the linalg-on-tensors backend contract. Recommended review order: - `TorchToTosa.cpp` / `TorchToTosa/basic.mlir` - `python/torch_mlir_e2e_test/torchscript/configs/tosa_backend.py` and the new `utils.py` file there. - `python/torch_mlir_e2e_test/tosa_backends/linalg_on_tensors.py` and `abc.py` in that directory for the TOSA backend e2e interface. - other misc mechanical changes
2021-10-08 10:07:03 +08:00
REFBACKEND_XFAIL_SET = COMMON_TORCH_MLIR_LOWERING_XFAILS
E2E framework: Add support for list/dict/scalar values Most of the change is in the reporting code to give error messages that are useful, and adjusting TraceItem to be semantically correct w.r.t. Python's modeling of return values. This allows writing a test like `ListLiteralModule_basic` for list functionality, which we will soon be hooking up to IREE. The IR for that test currently gets this far: ``` builtin.func @forward(%arg0: f64) -> !torch.list<!torch.float> { %0 = torch.from_f64 %arg0 %1 = torch.prim.ListConstruct %0, %0 : (!torch.float, !torch.float) -> !torch.list<!torch.float> return %1 : !torch.list<!torch.float> } ``` It should be sufficient to just add a conversion of `torch.prim.ListConstruct` (+ relevant type conversion) to necessary IREE primitives. For lists of *tensors* (rather than scalar floats), it gets more complicated, as we need to deal with changing their element type to ValueTensorType first (by default, they will all be NonValueTensorType). It seems that IREE might have a type we can lower into for non-value tensors as well, TBD.
2021-08-11 07:10:31 +08:00
Add a basic TOSA E2E backend. We lower through linalg-on-tensors and use RefBackend to run it. This adds enough support for a "tanh" op. Adding more ops should be fairly mechanical now that things are wired up. Run with: ``` ./tools/torchscript_e2e_test.sh -c tosa ``` The backend structure is very similar to linalg-on-tensors based E2E backends and is a nice parallel (see `tosa_backend.py`). Actually, this forced a nice refactoring to the layering here. We removed `torchscript-module-to-linalg-on-tensors-backend-pipeline` and instead require separately running ``` torchscript-function-to-torch-backend-pipeline,torch-backend-to-linalg-on-tensors-backend-pipeline ``` This highlights the step that lowers to the "torch backend contract" of cleaned up `torch` dialect ops is a critical step in the lowering. Going forward, that is the key load-bearing contract of the torch-mlir project, not the linalg-on-tensors backend contract. Recommended review order: - `TorchToTosa.cpp` / `TorchToTosa/basic.mlir` - `python/torch_mlir_e2e_test/torchscript/configs/tosa_backend.py` and the new `utils.py` file there. - `python/torch_mlir_e2e_test/tosa_backends/linalg_on_tensors.py` and `abc.py` in that directory for the TOSA backend e2e interface. - other misc mechanical changes
2021-10-08 10:07:03 +08:00
# Write the TOSA set as a "passing" set as it is very early in development
# and very few tests work yet.
Bump llvm-project to f1b922188ead5ca492c8d8edd47921b013a22ae0. Includes a fix to use `add_mlir_public_c_api_library` for Torch-MLIR's CAPI library, which is now required (note: upstream sample has it the right way). Disabled a TOSA test per discussion: https://github.com/llvm/torch-mlir/issues/379
2021-10-26 02:43:21 +08:00
TOSA_PASS_SET = {
# See: https://github.com/llvm/torch-mlir/issues/379
#"ElementwiseUnaryModule_basic",
}