llvm
/
torch-mlir
mirror of https://github.com/llvm/torch-mlir
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
torch-mlir/e2e_testing/torchscript/xfail_sets.py

157 lines
5.7 KiB
Python
Raw Normal View History

Dual license the torch-mlir project. This commit (with approval from all contributors) dual licenses the torch-mlir project under both the standard LLVM license and the standard PyTorch license. This will facilitate moving code between torch-mlir and the two upstream projects. The standard file comment is now: ``` // This file is licensed under the Apache License v2.0 with LLVM Exceptions. // See https://llvm.org/LICENSE.txt for license information. // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception // Also available under a BSD-style license. See LICENSE. ``` See `LICENSE` in the project root for the terms of both licenses.
2021-09-30 00:03:40 +08:00
# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
# See https://llvm.org/LICENSE.txt for license information.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
# Also available under a BSD-style license. See LICENSE.
Add IREE support in TorchScript e2e tests. - Add support for "expected failures" in test reporting. The new error reports look like [this](https://gist.github.com/silvasean/6ffd95e1d55302b699673da201da210d). - We will now be able to put these tests into CI, since the harness understand which tests are expected to pass and fail. - Refactor RefBackendTestConfig to NpcompBackendTestConfig which supports both RefBackend and IREE. - Add instructions for installing IREE dependencies (both from packages and for local builds of IREE) - Add `tools/torchscript_e2e_test.sh` for invoking the e2e test harness (this makes invoking a bit easier, as it doesn't rely on a loose Python invocation).
2021-07-01 05:13:21 +08:00
# This file describes the sets of tests expected to fail for each config.
# This information is deliberately kept in a side table, rather than
# in-situ on the test, as a deliberate layering decision: tests should
# have unique keys to identify them and enable side tables of various kinds
# (this includes down into lower parts of the stack, where a side table
# might be used to keep more elaborate sets of testing configurations).
Move COMMON_TORCH_MLIR_LOWERING_XFAILS into test_suite That way, downstreams don't have to duplicate this list. Also, remove "external config" feature, since it is subsumed by just importing the test suite.
2022-04-20 03:35:56 +08:00
from torch_mlir_e2e_test.test_suite import COMMON_TORCH_MLIR_LOWERING_XFAILS
Small fixes for slice edge cases (#476)
2021-12-16 05:54:41 +08:00
REFBACKEND_XFAIL_SET = COMMON_TORCH_MLIR_LOWERING_XFAILS
E2E framework: Add support for list/dict/scalar values Most of the change is in the reporting code to give error messages that are useful, and adjusting TraceItem to be semantically correct w.r.t. Python's modeling of return values. This allows writing a test like `ListLiteralModule_basic` for list functionality, which we will soon be hooking up to IREE. The IR for that test currently gets this far: ``` builtin.func @forward(%arg0: f64) -> !torch.list<!torch.float> { %0 = torch.from_f64 %arg0 %1 = torch.prim.ListConstruct %0, %0 : (!torch.float, !torch.float) -> !torch.list<!torch.float> return %1 : !torch.list<!torch.float> } ``` It should be sufficient to just add a conversion of `torch.prim.ListConstruct` (+ relevant type conversion) to necessary IREE primitives. For lists of *tensors* (rather than scalar floats), it gets more complicated, as we need to deal with changing their element type to ValueTensorType first (by default, they will all be NonValueTensorType). It seems that IREE might have a type we can lower into for non-value tensors as well, TBD.
2021-08-11 07:10:31 +08:00
add ci tests (#754)
2022-05-26 03:59:59 +08:00
EAGER_MODE_XFAIL_SET = {
# RefBackend fails
"TableBatchEmbeddingModule_basic",
"QuantizedMLP_basic"
}
This PR implements an eager mode backend for PyTorch through the torch-mlir framework. This is accomplished by overriding the `__torch_dispatch__` class method on wrapper subclass `TorchMLIRTensor(torch.Tensor)`. Effectively, this mode works by compiling op by op as the NN is eagerly executed by PyTorch. Entailed in that compilation is building a representation of the op that can be `torch.jit.script`ed, importing using `ModuleBuilder`, and then executing (e.g., with `RefBackendLinalgOnTensorsBackend`). This mode includes a fallback to conventional PyTorch if anything in the torch-mlir compilation process fails (e.g., unsupported op). Currently, all e2e tests pass execpt for two that involve an upstream PyTorch bug (https://github.com/pytorch/pytorch/issues/74400). High priority next steps: 1. A compile cache in order to speed up reruns of the same NN. 2. Integration with IREE (though not in this repo). 3. Integration with `torch.distributed`.
2022-03-03 07:02:18 +08:00
Add a basic TOSA E2E backend. We lower through linalg-on-tensors and use RefBackend to run it. This adds enough support for a "tanh" op. Adding more ops should be fairly mechanical now that things are wired up. Run with: ``` ./tools/torchscript_e2e_test.sh -c tosa ``` The backend structure is very similar to linalg-on-tensors based E2E backends and is a nice parallel (see `tosa_backend.py`). Actually, this forced a nice refactoring to the layering here. We removed `torchscript-module-to-linalg-on-tensors-backend-pipeline` and instead require separately running ``` torchscript-function-to-torch-backend-pipeline,torch-backend-to-linalg-on-tensors-backend-pipeline ``` This highlights the step that lowers to the "torch backend contract" of cleaned up `torch` dialect ops is a critical step in the lowering. Going forward, that is the key load-bearing contract of the torch-mlir project, not the linalg-on-tensors backend contract. Recommended review order: - `TorchToTosa.cpp` / `TorchToTosa/basic.mlir` - `python/torch_mlir_e2e_test/torchscript/configs/tosa_backend.py` and the new `utils.py` file there. - `python/torch_mlir_e2e_test/tosa_backends/linalg_on_tensors.py` and `abc.py` in that directory for the TOSA backend e2e interface. - other misc mechanical changes
2021-10-08 10:07:03 +08:00
# Write the TOSA set as a "passing" set as it is very early in development
# and very few tests work yet.
Bump llvm-project to f1b922188ead5ca492c8d8edd47921b013a22ae0. Includes a fix to use `add_mlir_public_c_api_library` for Torch-MLIR's CAPI library, which is now required (note: upstream sample has it the right way). Disabled a TOSA test per discussion: https://github.com/llvm/torch-mlir/issues/379
2021-10-26 02:43:21 +08:00
TOSA_PASS_SET = {
Rename `tosa-to-linalg-on-tensors` to `tosa-to-linalg` The pass name changed upstream.
2021-10-26 04:43:54 +08:00
"ElementwiseUnaryModule_basic",
Update external llvm to 966b72098363d44adf2882b9c34 The external llvm is updated to point to https://reviews.llvm.org/rG966b72098363d44adf2882b9c34fcdbe344ff913. Some of the changes wrt. NamedAttr has been addressed. Signed-off-by: Prashant Kumar <prashant@nod-labs.com>
2021-12-04 15:31:38 +08:00
"ElementwiseBinaryModule_basic",
[tosa] Add Torch.sigmoid fp32 to TOSA (#386) * [tosa] Add Torch.sigmoid fp32 to TOSA Signed-off-by: Suraj Sudhir <suraj.sudhir@arm.com>
2021-10-29 01:09:12 +08:00
"ElementwiseSigmoidModule_basic",
Add non-default dtype support for a few elementwise math ops. (#687) * fix type inference * fix Torch2Linalg conversion * add test cases
2022-03-24 04:35:43 +08:00
"ElementwiseExpModule_basic",
[tosa] Add structure for eltwise ops Add a bunch of op legalizations. Signed-off-by: Suraj Sudhir <suraj.sudhir@arm.com>
2021-11-11 11:03:36 +08:00
"ElementwiseReluModule_basic",
"ElementwiseFloorModule_basic",
"ElementwiseLogModule_basic",
Make broadcasting result shape more static This involes the following 2 parts: - Change refine type to propagate more static shape info. - Get as much static shape info as possible when creating the result tensor when converting to linalg.
2021-12-29 10:11:07 +08:00
"ElementwiseBinaryStaticShapeModule_basic",
Add aten.hardtanh e2e support.
2022-02-09 04:57:23 +08:00
"ElementwiseMinimumModule_basic",
"ElementwiseMinimumIntModule_basic",
"ElementwiseMaximumModule_basic",
"ElementwiseMaximumIntModule_basic",
[mlir][tosa] Refactor conversions to use templates (#416) - Remove use of conversion construction macros - Add mul and div op conversions - Add corresponding tests Signed-off-by: Suraj Sudhir <suraj.sudhir@arm.com>
2021-11-12 08:15:58 +08:00
"TanhBackward_basic",
[tosa] Add TosaMakeBroadcastable pass to torch-to-tosa pipeline. Fixes broken e2e test ElementwiseAddModule_basic Signed-off-by: Suraj Sudhir <suraj.sudhir@arm.com>
2021-11-24 14:25:59 +08:00
"ElementwiseAddModule_basic",
Add support for multiple return values This change is to unblock the work of some backprop ops returning more than one tensors. We will need to think of a more scalable approach in the future if more flexible return types combinations are needed.
2021-11-08 23:56:40 +08:00
"ReturnThreeTensorFloat32_basic",
lowered addcmul and addcdiv to linalg
2021-11-25 06:01:48 +08:00
"AddCMulModule_basic",
"AddCDivModule_basic",
Fix forward XPASS
2021-12-01 05:48:10 +08:00
"SqueezeModule_broadcast",
Add support for passing & returning memref of bool types Support for passing memref of bool types as a function argument and return is added in ref-backend. Signed-off-by: Prashant Kumar <prashant@nod-labs.com>
2021-12-08 22:05:02 +08:00
"BoolTensorReturnFalseModule_basic",
"BoolTensorReturnTrueModule_basic",
"BoolTensorReturnMixedModule_basic",
Fixes https://github.com/llvm/torch-mlir/issues/751 where `torch.bool` is parsed as signless `i1`. (#752)
2022-04-14 01:28:27 +08:00
"BoolTensorHandleSignless_basic",
* [tosa] Support for Rsqrt legalization (#480) Signed-off-by: Anup Gangwar <anup.gangwar@arm.com> Co-authored-by: Anup Gangwar <anup.gangwar@arm.com>
2021-12-15 02:03:58 +08:00
"ElementwiseRsqrtModule_basic",
[tosa] Support for Torch.squeeze (#487)
2021-12-16 13:40:29 +08:00
"SqueezeModule_static",
"SqueezeModule_noUnitDim",
"SqueezeModule_allUnitDim",
Added aten::t() Op
2021-12-17 12:08:07 +08:00
"TModuleRank1_basic",
"TModuleRank0_basic",
[TORCH][MLIR] Fold trivial cases of `aten.to.dtype` and `aten.view` op - It folds `aten.to.dtype` when the input tensor type and result type are exactly same. - It folds `aten.view` when the rank of both the input tensor type and result type is unity. Signed-Off-by: Gaurav Shukla <gaurav@nod-labs.com>
2021-12-23 20:04:29 +08:00
"ElementwiseToDtypeIdentityModule_basic",
"View1DFoldModule_basic",
[TORCH][MLIR] Add E2E support for `aten._unsafe_view` op. This commit adds decomposition of `aten._unsafe_view` op into `aten.view` op. Signed-Off-By: Prateek Gupta<prateek@nod-labs.com>
2022-02-10 16:11:05 +08:00
"UnsafeView1DFoldModule_basic",
[tosa] Implement squeeze.dim support (#511) Templated variants for squeeze and squeeze.dim
2022-01-07 00:31:29 +08:00
"SqueezeDimModule_static",
"SqueezeDimModule_identity",
"SqueezeDimModule_unitDim",
Add support for two return values This commit adds support for two return values of type memref f32 and i64. Signed-Off By: Vivek Khandelwal <vivek@nod-labs.com>
2022-01-10 21:24:08 +08:00
"ReturnTwoTensorF32I64_basic",
[tosa] Add tosa-to-standard before tosa-to-linalg pass (#524) Signed-off-by: Suraj Sudhir <suraj.sudhir@arm.com>
2022-01-15 03:05:11 +08:00
"ElementwisePowModule_basic",
[tosa] Enable tosa-to-linalg-named so Matmul works again (#530) Signed-off-by: Suraj Sudhir <suraj.sudhir@arm.com>
2022-01-20 04:10:04 +08:00
"BmmModule_basic",
Add static type information support to `aten.mm` (#602) This commit adds static type information support to `aten.mm`. This is needed for the forward pass of Bert training.
2022-02-19 01:56:48 +08:00
"MmDagModule_basic",
[tosa] Enable tosa-to-linalg-named so Matmul works again (#530) Signed-off-by: Suraj Sudhir <suraj.sudhir@arm.com>
2022-01-20 04:10:04 +08:00
"Matmul_dot",
"Matmul_3d",
* [tosa] Support for AtenRsubScalarOp for scalar constants (#531) * [tosa] Support for AtenCeilOp and AtenReciprocalOp * [tosa] Support for comparator ops, Aten[Gt|Lt|Eq][Tensor|Scalar]Op with scalar constant * [tosa] Support for Scalar variants of Aten[Mul|Div|Add|Sub] Ops with scalar constants Signed-off-by: Anup Gangwar <anup.gangwar@arm.com> Co-authored-by: Anup Gangwar <anup.gangwar@arm.com>
2022-01-21 02:58:30 +08:00
"RsubModule_basic",
"RsubModule_noalpha_basic",
"ElementwiseGtFloatScalarModule_basic",
"ElementwiseGtIntScalarModule_basic",
"ElementwiseGtMixed2ScalarModule_basic",
"ElementwiseGtFloatTensorModule_basic",
"ElementwiseGtIntTensorModule_basic",
"ElementwiseLtFloatScalarModule_basic",
"ElementwiseLtIntScalarModule_basic",
"ElementwiseLtDiffWidthScalarModule_basic",
"ElementwiseLtFloatTensorModule_basic",
"ElementwiseLtIntTensorModule_basic",
"ElementwiseEqFloatScalarModule_basic",
"ElementwiseEqIntScalarModule_basic",
"ElementwiseEqDiffWidthScalarModule_basic",
"ElementwiseEqFloatTensorModule_basic",
"ElementwiseEqIntTensorModule_basic",
"ElementwiseMulScalarModule_int",
"ElementwiseMulScalarModule_float",
"ElementwiseMulTensorIntModule_basic",
"ElementwiseDivScalarModule_basic",
"ElementwiseSubScalarFloatModule_basic",
"ElementwiseAddScalarFloatModule_basic",
"ElementwiseMulScalarModule_float",
"ElementwiseCeilModule_basic",
"ElementwiseReciprocalModule_basic",
"TypePromotionAlphaWiderModule_basic",
[tosa] Implement conv2d support (#541) Signed-off-by: Suraj Sudhir <suraj.sudhir@arm.com>
2022-01-27 11:16:13 +08:00
"Conv2dWithPaddingDilationStrideStaticModule_basic",
Revert "[LINALG] Decompose `aten.batch_norm` into `aten.native_batch_norm`" This reverts commit 442ff4605c54f836afa17d2780bd9bddcc2a5ed8.
2022-02-25 03:41:55 +08:00
"BatchNorm1DModule_basic",
"BatchNorm2DModule_basic",
"BatchNorm3DModule_basic",
* [tosa] Support for AtenFlattenUsingIntsOp (#548)
2022-01-29 13:38:56 +08:00
"FlattenStaticModule_basic",
"FlattenRank0Module_basic",
"ElementwiseFlattenBroadcastModule_basic",
[TORCH] Add aten.std e2e support
2022-01-30 01:10:50 +08:00
"SquareModule_basic",
[tosa] Add maxpool2d and adaptive_avgpool2d support (#550) Signed-off-by: Suraj Sudhir <suraj.sudhir@arm.com>
2022-02-01 05:34:09 +08:00
"MaxPool2dStaticModule_basic",
Revert "[LINALG] Decompose `aten.batch_norm` into `aten.native_batch_norm`" This reverts commit 442ff4605c54f836afa17d2780bd9bddcc2a5ed8.
2022-02-25 03:41:55 +08:00
"ResNet18StaticModule_basic",
* [tosa] Support for AtenNativeLayerNormOp * [tosa] Support for AtenPermuteOp Signed-off-by: Anup Gangwar <anup.gangwar@arm.com>
2022-02-04 06:08:19 +08:00
"NativeLayerNormModule4D_basic",
"LayerNormNormalizeOverAllDimsModule_basic",
"PermuteModule_basic",
"PermuteNegativeIndexModule_basic",
[tosa] Support for some ops and fix for Issue #532 (#575) * [tosa] Support for AtenNe[Tensor|Scalar]Op, AtenLog2Op, AtenBitwiseAndTensorOp, AtenSquareOp and AtenThresholdOp * Fix for Issue #532 - Mixed input types for few ops and updated few tests to use i32 instead of i64 Signed-off-by: Anup Gangwar <anup.gangwar@arm.com> Co-authored-by: Anup Gangwar <anup.gangwar@arm.com>
2022-02-12 04:30:02 +08:00
"ElementwiseLog2Module_basic",
"Threshold1dIntI32Module_basic",
"Threshold1dFloatModule_basic",
"Threshold2dFloatModule_basic",
"Threshold3dFloatModule_basic",
"ElementwiseSubScalarIntModule_basic",
"ElementwiseAddScalarIntModule_basic",
"ElementwiseMulScalarModule_basic",
[tosa] Support for Aten[Zeros|Ones|Fill_Scalar] ops (#604) Signed-off-by: Anup Gangwar <anup.gangwar@arm.com> Co-authored-by: Anup Gangwar <anup.gangwar@arm.com>
2022-02-17 01:53:51 +08:00
"ZerosModuleDefaultDtype_basic",
"ZerosModuleInt2D_basic",
"ZerosModuleInt3D_basic",
"ZerosModuleFloat2D_basic",
"ZerosModuleFloat3D_basic",
"ZerosModuleFalsePinMemory_basic",
"OnesModuleDefaultDtype_basic",
"OnesModuleInt_basic",
"OnesModuleFloat_basic",
"OnesModuleFalsePinMemory_basic",
[MLIR][TORCH] Add E2E support for aten.new_zeros, aten.new_ones op This commit adds lowering of `aten.new_zeros` and `aten.new_ones` op Signed-Off By: Vivek Khandelwal <vivek@nod-labs.com>
2022-02-28 14:14:40 +08:00
"NewZerosModuleDefaultDtype_basic",
"NewZerosModuleInt2D_basic",
"NewZerosModuleInt3D_basic",
"NewZerosModuleFloat2D_basic",
"NewZerosModuleFloat3D_basic",
"NewZerosModuleFalsePinMemory_basic",
"NewOnesModuleDefaultDtype_basic",
"NewOnesModuleInt2D_basic",
"NewOnesModuleInt3D_basic",
"NewOnesModuleFloat2D_basic",
"NewOnesModuleFloat3D_basic",
"NewOnesModuleFalsePinMemory_basic",
Decompose aten.silu op Decomposition of aten.silu.op is added as silu(x) = x * sigmoid(x).
2022-03-02 01:30:58 +08:00
"SiluModule_basic",
[LINALG] Add decomposition of `aten.dropout` op - This commit adds decomposition of `aten.dropout` op. It also covers the training mode of the same op. - It also adds lowering of `aten.sub.float` op. Signed-Off-by: Gaurav Shukla <gaurav@nod-labs.com>
2022-02-17 00:34:03 +08:00
"DropoutEvalIntModule_basic",
"DropoutEvalFloatModule_basic",
[tosa] Support for Aten[Unsqueeze|Contiguous|Dropout|Reshape|View] ops (#700)
2022-03-26 05:15:07 +08:00
"ContiguousModule_basic",
"DropoutModule_basic",
"ViewExpandModule_basic",
"ViewCollapseInferredDimModule_basic",
"ViewExpandInferredDimModule_basic",
"ViewNoChangeStaticModule_basic",
"UnsafeViewExpandModule_basic",
"ReshapeCollapseModule_basic",
tosa] Support for Aten[Gelu|GeluBackward] ops (#720) Signed-off-by: Anup Gangwar <anup.gangwar@arm.com> Co-authored-by: Anup Gangwar <anup.gangwar@arm.com>
2022-03-31 08:00:55 +08:00
"ElementwiseGeluModule_basic",
"GeluBackwardModule_basic",
[LINALG] Add the lowering of `aten.ne.Scalar` op The lowering of `aten.ne.Scalar` op has been added to the linalg backend.
2022-04-04 00:19:01 +08:00
"ElementwiseNeIntScalarModule_basic",
"ElementwiseNeFloatTensorModule_basic",
Add 2D case for convolution (#693)
2022-04-08 12:47:57 +08:00
"ConvolutionModule2DStatic_basic",
test: add end-to-end test for aten.neg (#760)
2022-04-16 03:37:57 +08:00
"ElementwiseNegModule_basic",
[REFBACKEND] Add support for returning multiple different return types. Added the dynamic registration of return function to the execution engine. This makes sure that different/multiple return types are supported. Also, updated the .style.yapf indentation to 4.
2022-04-19 21:47:47 +08:00
"TestMultipleTensorReturn_basic",
[tosa] Support for AtenAvgPool2d op Signed-Off By: Vivek Khandelwal <vivek@nod-labs.com>
2022-05-19 20:35:59 +08:00
"AdaptiveAvgPool2dUnitOutputSizeStaticModule_basic",
Bump llvm-project to f1b922188ead5ca492c8d8edd47921b013a22ae0. Includes a fix to use `add_mlir_public_c_api_library` for Torch-MLIR's CAPI library, which is now required (note: upstream sample has it the right way). Disabled a TOSA test per discussion: https://github.com/llvm/torch-mlir/issues/379
2021-10-26 02:43:21 +08:00
}