shape-lib: generate string as multiple lines to work with MSVC (#1370)

As @oroppas identified, literal strings that are over 16,380 characters
cause the MSVC compiler to throw an error (C2026), eventually causing
the Windows build of Torch-MLIR to fail because the length of the
generated MLIR for the shape library crosses the allowed threshold.

This patch fixes the problem by making the Python script generate one
literal string per line to satisfy the MSVC compiler.

Thanks to @oroppas for the bulk of the effort required to resolve this!
pull/1355/head
Ashay Rane 2022-09-16 15:16:01 -05:00 committed by GitHub
parent 51e3c3f1ed
commit 1895b581c4
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 7819 additions and 6974 deletions

File diff suppressed because it is too large Load Diff

View File

@ -1251,6 +1251,17 @@ def main(args):
# Put the `` back to a regular `.`.
asm = asm.replace("\\E3\\80\\87", ".")
# We're about to put quotes around the string, so escape the `"` characters.
asm = asm.replace("\"", "\\\"")
# Instead of dumping one big chunk of text that is several thousand lines
# long (and which causes MSVC to error out), split it into multiple lines.
# See MSVC Compiler Error C2026
# [https://docs.microsoft.com/en-us/cpp/error-messages/compiler-errors-1/compiler-error-c2026?view=msvc-170]
# for details.
multiple_lines = asm.replace("\n", "\\n\"\n\"")
asm = f"\"{multiple_lines}\""
# Write out the shape library .cpp file.
shape_lib_cpp_file = os.path.join(
args.torch_transforms_cpp_dir, "ShapeLibrary.cpp")
@ -1276,19 +1287,16 @@ f"""//===-------------------------------------------------------------*- C++-*-=
using namespace mlir;
StringRef mlir::torch::Torch::getShapeLibrary() {{
// TODO: Find a way to embed this string nicely.
// It is currently too long, and will probably break MSVC builds if anyone
// attempts that.
// We want to preserve the legibility of the shape library as a checked in file,
// since that is sometimes useful for debugging / diffing.
// Probably the ideal outcome is to have the shape library be a .mlir file
// that is checked in, and then we embed it as part of the build process.
#ifndef _MSC_VER
#pragma clang diagnostic push
#pragma clang diagnostic ignored "-Woverlength-strings"
constexpr StringLiteral shapeLib(R"mlir(
{asm})mlir");
#endif
// clang-format off
return {asm};
// clang-format on
#ifndef _MSC_VER
#pragma clang diagnostic pop
return shapeLib;
#endif
}}""")
def _create_argparse() -> argparse.ArgumentParser: