mirror of https://github.com/llvm/torch-mlir
42ba541c68
The fx tracer does not support tracing "real" quantized tensors currently. A "real" quantized tensor here means a tensor that is created using a method like `torch.quantize_per_tensor()` and carries the quantization parameters (scale, zero_point, scheme) in the object. However, it seems like the DQ-Q type fake quantizatation is now commonly used as a high level representation of quantized operators and is only lowered to native quantized ops (if available) in the respective hardware backend. Quantization of floating point modules in PyTorch is recently also performed as a graph transformation after exporting/tracing the original module. ```python # Examples of "real"/native quantization tens = torch.randint(-127, 127, (1,), dtype=torch.int8) torch._make_per_tensor_quantized_tensor(tens, 1, 0) # tensor([90.], size=(1,), dtype=torch.qint8, # quantization_scheme=torch.per_tensor_affine, scale=1.0, zero_point=0) tens = torch.rand((1,)) torch.quantize_per_tensor(tens, 1, 0, torch.qint8) # tensor([1.], size=(1,), dtype=torch.qint8, # quantization_scheme=torch.per_tensor_affine, scale=1.0, zero_point=0) # Example of DQ/Q quantization import torch.ao.quantization.fx._decomposed tens = torch.rand((1,)) torch.ops.quantized_decomposed.quantize_per_tensor.default(tens, 1, 0, -128, 127, torch.int8) # tensor([1], dtype=torch.int8) ``` This means that a typical import flow for a quantized network into/through torch-mlir would look like this: `torch.export() -> quantization transformations on fx graph -> fx_importer` Where the tensors in the graph are normal float/int tensors and the quantization parameters are carried by the DQ/Q ops. These kinds of graphs can be traced without issues. Currently, our quantized convolution tests use the "real" quantized tensors. This means that with the retirement of the `jit_ir_importer`, these tests cannot be imported any longer. In summary, I see no reason to stick to the "real" quantization in these tests, as both PyTorch 2.0 is using DQ/Q quantization and our linalg backend is also using it. This patch updates our quantized convolution tests to use the DQ-Q quantization with the ops from `torch.ops.quantized_decomposed`. Note: For future reference, there seems to be an ongoing consolidation of the ops for the DQ/Q scheme on the PyTorch side (https://github.com/pytorch/ao/issues/986#issuecomment-2390296826). |
||
|---|---|---|
| .. | ||
| torch_mlir | ||
| CMakeLists.txt | ||
| TorchMLIRModule.cpp | ||