llvm
/
torch-mlir
mirror of https://github.com/llvm/torch-mlir
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
torch-mlir/e2e_testing/torchscript/xfail_sets.py

35 lines
1.3 KiB
Python
Raw Normal View History

Add IREE support in TorchScript e2e tests. - Add support for "expected failures" in test reporting. The new error reports look like [this](https://gist.github.com/silvasean/6ffd95e1d55302b699673da201da210d). - We will now be able to put these tests into CI, since the harness understand which tests are expected to pass and fail. - Refactor RefBackendTestConfig to NpcompBackendTestConfig which supports both RefBackend and IREE. - Add instructions for installing IREE dependencies (both from packages and for local builds of IREE) - Add `tools/torchscript_e2e_test.sh` for invoking the e2e test harness (this makes invoking a bit easier, as it doesn't rely on a loose Python invocation).
2021-07-01 05:13:21 +08:00
# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
# See https://llvm.org/LICENSE.txt for license information.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
# This file describes the sets of tests expected to fail for each config.
# This information is deliberately kept in a side table, rather than
# in-situ on the test, as a deliberate layering decision: tests should
# have unique keys to identify them and enable side tables of various kinds
# (this includes down into lower parts of the stack, where a side table
# might be used to keep more elaborate sets of testing configurations).
XFAIL_SETS = {}
# Lists of tests that fail to even reach the backends.
# These represent further work needed in npcomp to lower them properly
# to the backend contract.
_common_npcomp_lowering_xfails = {
'QuantizedMLP_basic',
Add basic support for lists. This plumbs through a vertical slice of support for lists. The main chunk of new code here is AnnotateABIPass which captures the program signature at the Torch backend contract layer, right before we start `TorchConversion`. The `TorchConversion` lowering process is lossy w.r.t. types, so it's necessary to do this for all targets in general. Like using `!iree.list` directly, we use IREE's ABI annotation representation for this, although there is nothing very IREE-specific about it (see https://github.com/google/iree/blob/main/docs/developers/design_docs/function_abi.md) We change `ListLiteralModule_basic` to use `!torch.int` because IREE doesn't support f64 yet (and we don't yet have a way for users to say that they want `!torch.float` to lower as f32). Recommended review order: - AnnotateABIPass and tests - Arg marshaling in npcomp_backend.py and `iree.py` - Updates to `list_programs.py` / `xfail_sets.py` - Moving DeleteDeadIREEListsPass to Backend/Common, so that backends that don't support lists can use it. RefBackend uses that pass, for example.
2021-09-03 03:47:59 +08:00
}
# Any test expected to fail on backends that don't support non-tensor types
# should be listed here.
_common_non_tensor_type_xfails = {
E2E framework: Add support for list/dict/scalar values Most of the change is in the reporting code to give error messages that are useful, and adjusting TraceItem to be semantically correct w.r.t. Python's modeling of return values. This allows writing a test like `ListLiteralModule_basic` for list functionality, which we will soon be hooking up to IREE. The IR for that test currently gets this far: ``` builtin.func @forward(%arg0: f64) -> !torch.list<!torch.float> { %0 = torch.from_f64 %arg0 %1 = torch.prim.ListConstruct %0, %0 : (!torch.float, !torch.float) -> !torch.list<!torch.float> return %1 : !torch.list<!torch.float> } ``` It should be sufficient to just add a conversion of `torch.prim.ListConstruct` (+ relevant type conversion) to necessary IREE primitives. For lists of *tensors* (rather than scalar floats), it gets more complicated, as we need to deal with changing their element type to ValueTensorType first (by default, they will all be NonValueTensorType). It seems that IREE might have a type we can lower into for non-value tensors as well, TBD.
2021-08-11 07:10:31 +08:00
'ListLiteralModule_basic',
Add IREE support in TorchScript e2e tests. - Add support for "expected failures" in test reporting. The new error reports look like [this](https://gist.github.com/silvasean/6ffd95e1d55302b699673da201da210d). - We will now be able to put these tests into CI, since the harness understand which tests are expected to pass and fail. - Refactor RefBackendTestConfig to NpcompBackendTestConfig which supports both RefBackend and IREE. - Add instructions for installing IREE dependencies (both from packages and for local builds of IREE) - Add `tools/torchscript_e2e_test.sh` for invoking the e2e test harness (this makes invoking a bit easier, as it doesn't rely on a loose Python invocation).
2021-07-01 05:13:21 +08:00
}
Add basic support for lists. This plumbs through a vertical slice of support for lists. The main chunk of new code here is AnnotateABIPass which captures the program signature at the Torch backend contract layer, right before we start `TorchConversion`. The `TorchConversion` lowering process is lossy w.r.t. types, so it's necessary to do this for all targets in general. Like using `!iree.list` directly, we use IREE's ABI annotation representation for this, although there is nothing very IREE-specific about it (see https://github.com/google/iree/blob/main/docs/developers/design_docs/function_abi.md) We change `ListLiteralModule_basic` to use `!torch.int` because IREE doesn't support f64 yet (and we don't yet have a way for users to say that they want `!torch.float` to lower as f32). Recommended review order: - AnnotateABIPass and tests - Arg marshaling in npcomp_backend.py and `iree.py` - Updates to `list_programs.py` / `xfail_sets.py` - Moving DeleteDeadIREEListsPass to Backend/Common, so that backends that don't support lists can use it. RefBackend uses that pass, for example.
2021-09-03 03:47:59 +08:00
XFAIL_SETS['refbackend'] = (_common_npcomp_lowering_xfails
| _common_non_tensor_type_xfails)
Add IREE support in TorchScript e2e tests. - Add support for "expected failures" in test reporting. The new error reports look like [this](https://gist.github.com/silvasean/6ffd95e1d55302b699673da201da210d). - We will now be able to put these tests into CI, since the harness understand which tests are expected to pass and fail. - Refactor RefBackendTestConfig to NpcompBackendTestConfig which supports both RefBackend and IREE. - Add instructions for installing IREE dependencies (both from packages and for local builds of IREE) - Add `tools/torchscript_e2e_test.sh` for invoking the e2e test harness (this makes invoking a bit easier, as it doesn't rely on a loose Python invocation).
2021-07-01 05:13:21 +08:00
Update IREE xfail sets. All tests pass after https://github.com/google/iree/pull/6666 :)
2021-08-12 02:11:13 +08:00
XFAIL_SETS['iree'] = _common_npcomp_lowering_xfails
E2E framework: Add support for list/dict/scalar values Most of the change is in the reporting code to give error messages that are useful, and adjusting TraceItem to be semantically correct w.r.t. Python's modeling of return values. This allows writing a test like `ListLiteralModule_basic` for list functionality, which we will soon be hooking up to IREE. The IR for that test currently gets this far: ``` builtin.func @forward(%arg0: f64) -> !torch.list<!torch.float> { %0 = torch.from_f64 %arg0 %1 = torch.prim.ListConstruct %0, %0 : (!torch.float, !torch.float) -> !torch.list<!torch.float> return %1 : !torch.list<!torch.float> } ``` It should be sufficient to just add a conversion of `torch.prim.ListConstruct` (+ relevant type conversion) to necessary IREE primitives. For lists of *tensors* (rather than scalar floats), it gets more complicated, as we need to deal with changing their element type to ValueTensorType first (by default, they will all be NonValueTensorType). It seems that IREE might have a type we can lower into for non-value tensors as well, TBD.
2021-08-11 07:10:31 +08:00
XFAIL_SETS['torchscript'] = {}
XFAIL_SETS['native_torch'] = {}