String serialization for errors very costly

[vchuravy]

I was looking at SpeedyWeather/SpeedyWeather.jl#876 today and during a 25min/1533s compile time (before I terminated the process, roughly 50% of the time was spent in julia_error and more precisely the string call to serialize the IR.

Ideas:

• Avoid string serialization and instead use bitcode (and only stringify on the other side)
• Memoize the string, it might be simply the case that for multiple possible errors we are serializing the same module multiple times.

[vchuravy]

diff --git a/src/errors.jl b/src/errors.jl
index 0c91f738..38a691e9 100644
--- a/src/errors.jl
+++ b/src/errors.jl
@@ -336,6 +336,8 @@ parent_scope(@nospecialize(val::LLVM.Value), depth = 0) = parent_scope(LLVM.pare
 parent_scope(val::LLVM.Argument, depth = 0) =
     parent_scope(LLVM.Function(LLVM.API.LLVMGetParamParent(val)), depth + 1)

+
+const ENZYME_FULL_ERROR = Ref(false)
 function julia_error(
     cstr::Cstring,
     val::LLVM.API.LLVMValueRef,
@@ -385,7 +387,9 @@ function julia_error(
         else
             # Need to convert function to string, since when the error is going to be printed
             # the module might have been destroyed
-            ir = string(parent_scope(val))
+            if ENZYME_FULL_ERROR[]
+                ir = string(parent_scope(val))
+            end
         end
     end

This is an option but feels unsatisfying.

I can look at memoizing after #2643

[vchuravy]

@wsmoses mentions that this should never be the parent_scope(val) isa LLVM.Module

[vchuravy]

Might be fixed by #2694