llvm
/
torch-mlir
mirror of https://github.com/llvm/torch-mlir
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
torch-mlir/e2e_testing/torchscript/xfail_sets.py

104 lines
3.9 KiB
Python
Raw Normal View History

Dual license the torch-mlir project. This commit (with approval from all contributors) dual licenses the torch-mlir project under both the standard LLVM license and the standard PyTorch license. This will facilitate moving code between torch-mlir and the two upstream projects. The standard file comment is now: ``` // This file is licensed under the Apache License v2.0 with LLVM Exceptions. // See https://llvm.org/LICENSE.txt for license information. // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception // Also available under a BSD-style license. See LICENSE. ``` See `LICENSE` in the project root for the terms of both licenses.
2021-09-30 00:03:40 +08:00
# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
# See https://llvm.org/LICENSE.txt for license information.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
# Also available under a BSD-style license. See LICENSE.
Add IREE support in TorchScript e2e tests. - Add support for "expected failures" in test reporting. The new error reports look like [this](https://gist.github.com/silvasean/6ffd95e1d55302b699673da201da210d). - We will now be able to put these tests into CI, since the harness understand which tests are expected to pass and fail. - Refactor RefBackendTestConfig to NpcompBackendTestConfig which supports both RefBackend and IREE. - Add instructions for installing IREE dependencies (both from packages and for local builds of IREE) - Add `tools/torchscript_e2e_test.sh` for invoking the e2e test harness (this makes invoking a bit easier, as it doesn't rely on a loose Python invocation).
2021-07-01 05:13:21 +08:00
# This file describes the sets of tests expected to fail for each config.
# This information is deliberately kept in a side table, rather than
# in-situ on the test, as a deliberate layering decision: tests should
# have unique keys to identify them and enable side tables of various kinds
# (this includes down into lower parts of the stack, where a side table
# might be used to keep more elaborate sets of testing configurations).
# Lists of tests that fail to even reach the backends.
Port the bulk of the remaining code to torch-mlir This leaves no real code outside torch-mlir. This also renames the "npcomp backend contract" to "linalg on tensors backend contract" as the name of the abstraction layer that RefBackend (IREE too) accepts.
2021-09-28 02:36:44 +08:00
# These represent further work needed in torch-mlir to lower them properly
Add IREE support in TorchScript e2e tests. - Add support for "expected failures" in test reporting. The new error reports look like [this](https://gist.github.com/silvasean/6ffd95e1d55302b699673da201da210d). - We will now be able to put these tests into CI, since the harness understand which tests are expected to pass and fail. - Refactor RefBackendTestConfig to NpcompBackendTestConfig which supports both RefBackend and IREE. - Add instructions for installing IREE dependencies (both from packages and for local builds of IREE) - Add `tools/torchscript_e2e_test.sh` for invoking the e2e test harness (this makes invoking a bit easier, as it doesn't rely on a loose Python invocation).
2021-07-01 05:13:21 +08:00
# to the backend contract.
Add --external-config option to tools/torchscript_e2e_test.sh This is a simple way for externals to plug their backends into the test suite. They just implement the `TestConfig` class for their backend and write a small script that exposes it. I have a pending PR for iree-samples that successfully integrates this.
2021-10-05 02:24:12 +08:00
COMMON_TORCH_MLIR_LOWERING_XFAILS = {
Add a basic TOSA E2E backend. We lower through linalg-on-tensors and use RefBackend to run it. This adds enough support for a "tanh" op. Adding more ops should be fairly mechanical now that things are wired up. Run with: ``` ./tools/torchscript_e2e_test.sh -c tosa ``` The backend structure is very similar to linalg-on-tensors based E2E backends and is a nice parallel (see `tosa_backend.py`). Actually, this forced a nice refactoring to the layering here. We removed `torchscript-module-to-linalg-on-tensors-backend-pipeline` and instead require separately running ``` torchscript-function-to-torch-backend-pipeline,torch-backend-to-linalg-on-tensors-backend-pipeline ``` This highlights the step that lowers to the "torch backend contract" of cleaned up `torch` dialect ops is a critical step in the lowering. Going forward, that is the key load-bearing contract of the torch-mlir project, not the linalg-on-tensors backend contract. Recommended review order: - `TorchToTosa.cpp` / `TorchToTosa/basic.mlir` - `python/torch_mlir_e2e_test/torchscript/configs/tosa_backend.py` and the new `utils.py` file there. - `python/torch_mlir_e2e_test/tosa_backends/linalg_on_tensors.py` and `abc.py` in that directory for the TOSA backend e2e interface. - other misc mechanical changes
2021-10-08 10:07:03 +08:00
"QuantizedMLP_basic",
Add min/max/clamp support. Part of #380 Also - BoolType is not considered as Scalar - e2e framework fixes for nan handling - `tu.rand(..., low=, high=)` support - delete unused variable (fix warning) - Add IouOfModule from #380 to e2e test suite (this is a common calculation in vision models) Your branch is ahead of 'origin/main' by 1 commit.
2021-10-27 11:44:01 +08:00
"IouOfModule_basic",
[TBE] Add a test module for table batch embedding This commit adds a test module specifically for table batch embedding algorithm. This test case is in reference to the FBGEMM table batch embedding: https://github.com/pytorch/FBGEMM/blob/main/fbgemm_gpu/bench/split_table_batched_embeddings_benchmark.py#L270 Signed-Off-by: Gaurav Shukla <gaurav@nod-labs.com>
2022-01-22 01:21:11 +08:00
"TableBatchEmbeddingModule_basic",
Add mobilenetv2 and mobilenetv3 to e2e test
2022-02-08 05:45:38 +08:00
"MobilenetV2Module_basic",
"MobilenetV3Module_basic",
Add basic support for lists. This plumbs through a vertical slice of support for lists. The main chunk of new code here is AnnotateABIPass which captures the program signature at the Torch backend contract layer, right before we start `TorchConversion`. The `TorchConversion` lowering process is lossy w.r.t. types, so it's necessary to do this for all targets in general. Like using `!iree.list` directly, we use IREE's ABI annotation representation for this, although there is nothing very IREE-specific about it (see https://github.com/google/iree/blob/main/docs/developers/design_docs/function_abi.md) We change `ListLiteralModule_basic` to use `!torch.int` because IREE doesn't support f64 yet (and we don't yet have a way for users to say that they want `!torch.float` to lower as f32). Recommended review order: - AnnotateABIPass and tests - Arg marshaling in npcomp_backend.py and `iree.py` - Updates to `list_programs.py` / `xfail_sets.py` - Moving DeleteDeadIREEListsPass to Backend/Common, so that backends that don't support lists can use it. RefBackend uses that pass, for example.
2021-09-03 03:47:59 +08:00
}
Small fixes for slice edge cases (#476)
2021-12-16 05:54:41 +08:00
REFBACKEND_XFAIL_SET = COMMON_TORCH_MLIR_LOWERING_XFAILS
E2E framework: Add support for list/dict/scalar values Most of the change is in the reporting code to give error messages that are useful, and adjusting TraceItem to be semantically correct w.r.t. Python's modeling of return values. This allows writing a test like `ListLiteralModule_basic` for list functionality, which we will soon be hooking up to IREE. The IR for that test currently gets this far: ``` builtin.func @forward(%arg0: f64) -> !torch.list<!torch.float> { %0 = torch.from_f64 %arg0 %1 = torch.prim.ListConstruct %0, %0 : (!torch.float, !torch.float) -> !torch.list<!torch.float> return %1 : !torch.list<!torch.float> } ``` It should be sufficient to just add a conversion of `torch.prim.ListConstruct` (+ relevant type conversion) to necessary IREE primitives. For lists of *tensors* (rather than scalar floats), it gets more complicated, as we need to deal with changing their element type to ValueTensorType first (by default, they will all be NonValueTensorType). It seems that IREE might have a type we can lower into for non-value tensors as well, TBD.
2021-08-11 07:10:31 +08:00
Add a basic TOSA E2E backend. We lower through linalg-on-tensors and use RefBackend to run it. This adds enough support for a "tanh" op. Adding more ops should be fairly mechanical now that things are wired up. Run with: ``` ./tools/torchscript_e2e_test.sh -c tosa ``` The backend structure is very similar to linalg-on-tensors based E2E backends and is a nice parallel (see `tosa_backend.py`). Actually, this forced a nice refactoring to the layering here. We removed `torchscript-module-to-linalg-on-tensors-backend-pipeline` and instead require separately running ``` torchscript-function-to-torch-backend-pipeline,torch-backend-to-linalg-on-tensors-backend-pipeline ``` This highlights the step that lowers to the "torch backend contract" of cleaned up `torch` dialect ops is a critical step in the lowering. Going forward, that is the key load-bearing contract of the torch-mlir project, not the linalg-on-tensors backend contract. Recommended review order: - `TorchToTosa.cpp` / `TorchToTosa/basic.mlir` - `python/torch_mlir_e2e_test/torchscript/configs/tosa_backend.py` and the new `utils.py` file there. - `python/torch_mlir_e2e_test/tosa_backends/linalg_on_tensors.py` and `abc.py` in that directory for the TOSA backend e2e interface. - other misc mechanical changes
2021-10-08 10:07:03 +08:00
# Write the TOSA set as a "passing" set as it is very early in development
# and very few tests work yet.
Bump llvm-project to f1b922188ead5ca492c8d8edd47921b013a22ae0. Includes a fix to use `add_mlir_public_c_api_library` for Torch-MLIR's CAPI library, which is now required (note: upstream sample has it the right way). Disabled a TOSA test per discussion: https://github.com/llvm/torch-mlir/issues/379
2021-10-26 02:43:21 +08:00
TOSA_PASS_SET = {
Rename `tosa-to-linalg-on-tensors` to `tosa-to-linalg` The pass name changed upstream.
2021-10-26 04:43:54 +08:00
"ElementwiseUnaryModule_basic",
Update external llvm to 966b72098363d44adf2882b9c34 The external llvm is updated to point to https://reviews.llvm.org/rG966b72098363d44adf2882b9c34fcdbe344ff913. Some of the changes wrt. NamedAttr has been addressed. Signed-off-by: Prashant Kumar <prashant@nod-labs.com>
2021-12-04 15:31:38 +08:00
"ElementwiseBinaryModule_basic",
[tosa] Add Torch.sigmoid fp32 to TOSA (#386) * [tosa] Add Torch.sigmoid fp32 to TOSA Signed-off-by: Suraj Sudhir <suraj.sudhir@arm.com>
2021-10-29 01:09:12 +08:00
"ElementwiseSigmoidModule_basic",
[tosa] Add structure for eltwise ops Add a bunch of op legalizations. Signed-off-by: Suraj Sudhir <suraj.sudhir@arm.com>
2021-11-11 11:03:36 +08:00
"ElementwiseReluModule_basic",
"ElementwiseFloorModule_basic",
"ElementwiseLogModule_basic",
Make broadcasting result shape more static This involes the following 2 parts: - Change refine type to propagate more static shape info. - Get as much static shape info as possible when creating the result tensor when converting to linalg.
2021-12-29 10:11:07 +08:00
"ElementwiseBinaryStaticShapeModule_basic",
[mlir][tosa] Refactor conversions to use templates (#416) - Remove use of conversion construction macros - Add mul and div op conversions - Add corresponding tests Signed-off-by: Suraj Sudhir <suraj.sudhir@arm.com>
2021-11-12 08:15:58 +08:00
"TanhBackward_basic",
[tosa] Add TosaMakeBroadcastable pass to torch-to-tosa pipeline. Fixes broken e2e test ElementwiseAddModule_basic Signed-off-by: Suraj Sudhir <suraj.sudhir@arm.com>
2021-11-24 14:25:59 +08:00
"ElementwiseAddModule_basic",
Add support for multiple return values This change is to unblock the work of some backprop ops returning more than one tensors. We will need to think of a more scalable approach in the future if more flexible return types combinations are needed.
2021-11-08 23:56:40 +08:00
"ReturnThreeTensorFloat32_basic",
lowered addcmul and addcdiv to linalg
2021-11-25 06:01:48 +08:00
"AddCMulModule_basic",
"AddCDivModule_basic",
Fix forward XPASS
2021-12-01 05:48:10 +08:00
"SqueezeModule_broadcast",
Add support for passing & returning memref of bool types Support for passing memref of bool types as a function argument and return is added in ref-backend. Signed-off-by: Prashant Kumar <prashant@nod-labs.com>
2021-12-08 22:05:02 +08:00
"BoolTensorReturnFalseModule_basic",
"BoolTensorReturnTrueModule_basic",
"BoolTensorReturnMixedModule_basic",
* [tosa] Support for Rsqrt legalization (#480) Signed-off-by: Anup Gangwar <anup.gangwar@arm.com> Co-authored-by: Anup Gangwar <anup.gangwar@arm.com>
2021-12-15 02:03:58 +08:00
"ElementwiseRsqrtModule_basic",
[tosa] Support for Torch.squeeze (#487)
2021-12-16 13:40:29 +08:00
"SqueezeModule_static",
"SqueezeModule_noUnitDim",
"SqueezeModule_allUnitDim",
Added aten::t() Op
2021-12-17 12:08:07 +08:00
"TModuleRank1_basic",
"TModuleRank0_basic",
[TORCH][MLIR] Fold trivial cases of `aten.to.dtype` and `aten.view` op - It folds `aten.to.dtype` when the input tensor type and result type are exactly same. - It folds `aten.view` when the rank of both the input tensor type and result type is unity. Signed-Off-by: Gaurav Shukla <gaurav@nod-labs.com>
2021-12-23 20:04:29 +08:00
"ElementwiseToDtypeIdentityModule_basic",
"View1DFoldModule_basic",
[TORCH][MLIR] Add E2E support for `aten._unsafe_view` op. This commit adds decomposition of `aten._unsafe_view` op into `aten.view` op. Signed-Off-By: Prateek Gupta<prateek@nod-labs.com>
2022-02-10 16:11:05 +08:00
"UnsafeView1DFoldModule_basic",
[tosa] Implement squeeze.dim support (#511) Templated variants for squeeze and squeeze.dim
2022-01-07 00:31:29 +08:00
"SqueezeDimModule_static",
"SqueezeDimModule_identity",
"SqueezeDimModule_unitDim",
Add support for two return values This commit adds support for two return values of type memref f32 and i64. Signed-Off By: Vivek Khandelwal <vivek@nod-labs.com>
2022-01-10 21:24:08 +08:00
"ReturnTwoTensorF32I64_basic",
[tosa] Add tosa-to-standard before tosa-to-linalg pass (#524) Signed-off-by: Suraj Sudhir <suraj.sudhir@arm.com>
2022-01-15 03:05:11 +08:00
"ElementwisePowModule_basic",
[tosa] Enable tosa-to-linalg-named so Matmul works again (#530) Signed-off-by: Suraj Sudhir <suraj.sudhir@arm.com>
2022-01-20 04:10:04 +08:00
"BmmModule_basic",
Add static type information support to `aten.mm` This commit adds static type information support to `aten.mm`. This is needed for the forward pass of Bert training.
2022-02-16 07:08:36 +08:00
"MmDagModule_basic",
[tosa] Enable tosa-to-linalg-named so Matmul works again (#530) Signed-off-by: Suraj Sudhir <suraj.sudhir@arm.com>
2022-01-20 04:10:04 +08:00
"Matmul_dot",
"Matmul_3d",
* [tosa] Support for AtenRsubScalarOp for scalar constants (#531) * [tosa] Support for AtenCeilOp and AtenReciprocalOp * [tosa] Support for comparator ops, Aten[Gt|Lt|Eq][Tensor|Scalar]Op with scalar constant * [tosa] Support for Scalar variants of Aten[Mul|Div|Add|Sub] Ops with scalar constants Signed-off-by: Anup Gangwar <anup.gangwar@arm.com> Co-authored-by: Anup Gangwar <anup.gangwar@arm.com>
2022-01-21 02:58:30 +08:00
"RsubModule_basic",
"RsubModule_noalpha_basic",
"ElementwiseGtFloatScalarModule_basic",
"ElementwiseGtIntScalarModule_basic",
"ElementwiseGtMixed2ScalarModule_basic",
"ElementwiseGtFloatTensorModule_basic",
"ElementwiseGtIntTensorModule_basic",
"ElementwiseLtFloatScalarModule_basic",
"ElementwiseLtIntScalarModule_basic",
"ElementwiseLtDiffWidthScalarModule_basic",
"ElementwiseLtFloatTensorModule_basic",
"ElementwiseLtIntTensorModule_basic",
"ElementwiseEqFloatScalarModule_basic",
"ElementwiseEqIntScalarModule_basic",
"ElementwiseEqDiffWidthScalarModule_basic",
"ElementwiseEqFloatTensorModule_basic",
"ElementwiseEqIntTensorModule_basic",
"ElementwiseMulScalarModule_int",
"ElementwiseMulScalarModule_float",
"ElementwiseMulTensorIntModule_basic",
"ElementwiseDivScalarModule_basic",
"ElementwiseSubScalarFloatModule_basic",
"ElementwiseAddScalarFloatModule_basic",
"ElementwiseMulScalarModule_float",
"ElementwiseCeilModule_basic",
"ElementwiseReciprocalModule_basic",
"TypePromotionAlphaWiderModule_basic",
[tosa] Implement conv2d support (#541) Signed-off-by: Suraj Sudhir <suraj.sudhir@arm.com>
2022-01-27 11:16:13 +08:00
"Conv2dWithPaddingDilationStrideStaticModule_basic",
* [tosa] Support for AtenReshapeOp (#543) * [tosa] Support for AtenBatchNormOp Signed-off-by: Anup Gangwar <anup.gangwar@arm.com> Co-authored-by: Anup Gangwar <anup.gangwar@arm.com>
2022-01-28 06:38:59 +08:00
"BatchNorm1DModule_basic",
"BatchNorm2DModule_basic",
"BatchNorm3DModule_basic",
* [tosa] Support for AtenFlattenUsingIntsOp (#548)
2022-01-29 13:38:56 +08:00
"FlattenStaticModule_basic",
"FlattenRank0Module_basic",
"ElementwiseFlattenBroadcastModule_basic",
[TORCH] Add aten.std e2e support
2022-01-30 01:10:50 +08:00
"SquareModule_basic",
[tosa] Add maxpool2d and adaptive_avgpool2d support (#550) Signed-off-by: Suraj Sudhir <suraj.sudhir@arm.com>
2022-02-01 05:34:09 +08:00
"MaxPool2dStaticModule_basic",
RefineTypes fixes for TOSA backend (#557) Handles Linear, Adaptive_AvgPool2D and FlattenUsintInts Adds ResNet18 static model for TOSA Signed-off-by: Suraj Sudhir <suraj.sudhir@arm.com>
2022-02-02 06:08:54 +08:00
"ResNet18StaticModule_basic",
* [tosa] Support for AtenNativeLayerNormOp * [tosa] Support for AtenPermuteOp Signed-off-by: Anup Gangwar <anup.gangwar@arm.com>
2022-02-04 06:08:19 +08:00
"NativeLayerNormModule4D_basic",
"LayerNormNormalizeOverAllDimsModule_basic",
"PermuteModule_basic",
"PermuteNegativeIndexModule_basic",
Bump llvm-project to f1b922188ead5ca492c8d8edd47921b013a22ae0. Includes a fix to use `add_mlir_public_c_api_library` for Torch-MLIR's CAPI library, which is now required (note: upstream sample has it the right way). Disabled a TOSA test per discussion: https://github.com/llvm/torch-mlir/issues/379
2021-10-26 02:43:21 +08:00
}