[WIP] HIVE-29641: Upgrade Calcite to 1.42.0

[rubenada]

What changes were proposed in this pull request?

Upgrade Calcite to 1.42.0.

Why are the changes needed?

Upgrade to latest Calcite version.

Does this PR introduce any user-facing change?

tbd

How was this patch tested?

tbd

Details

pom changes

• Change calcite version in parent pom (from 1.33.0 to 1.42.0)

Code adjustments due to Calcite breaking changes

• [CALCITE-5557] Add SAFE_CAST function (enabled in BigQuery library) (1.35)
  Use non-deprecated version of RexBuilder#makeAbstractCast in HiveSqlSumAggFunction, RexNodeConverter, ASTConverter
• ddb4200 Refactor: Add fields AggregateCall.rexList and RelBuilder.AggCall.preOperands (1.35)
  AggCall construction adjustments in HiveAggregate and HiveSqlSumAggFunction
• [CALCITE-3679] Allow lambda expressions in SQL queries (1.37)
  Add visitLambda and visitLambdaRef in RexVisitor implementation (HiveCalciteUtil#ConstantFinder)
• [CALCITE-6961] Support LogicalRepeatUnion in RelShuttle + [CALCITE-6959] Support LogicalAsofJoin in RelShuttle (1.40)
  Add visit(LogicalAsofJoin) and visit(LogicalRepeatUnion) to HiveRelShuttleImpl
• [CALCITE-5387] Type-mismatch on nullability in JoinPushTransitivePredicatesRule RelRule (1.40)
  Adjust RexPermuteInputsShuttle constructor in HiveJoinConstraintsRule
• [CALCITE-7029] Support DPhyp to handle various join types (1.41)
  Add visitNodeAndFieldIndex in RexVisitor implementation (HiveCalciteUtil#ConstantFinder)
• [CALCITE-7351] Make getMaxNumericScale() and getMaxNumericPrecision() final (1.42)
  Adjust HiveTypeSystemImpl

Other code changes

• Hive.java: update Update Calcite default charset system property (use UTF16 instead of the default ISO-8859-1), otherwise we get "charset pollution" on char literals in explained plans (e.g. _UTF-16LE'1' instead of simply '1') due to the modification in SqlImplemento#toSql(RexLiteral) via "[CALCITE-6006] RelToSqlConverter loses charset information" (introduced in 1.36)
  As a consequence of defining Calcite default charset system property, now RexLiteral#appendAsJava (which already contained the literal charset vs default charset verification, added to SqlImplementor in CALCITE-6006 1.36), literals printed via this method (i.e. used to print the charset) don't do it any more, e.g. _UTF-16LE'ten' => 'ten', which aligns them with the ones printed via SqlImplemento#toSql(RexLiteral)
• Due to "7fc3e1b Refactor: Add RelNode.stripped" (included in Calcite 1.35), which added a new check in HepRelVertex constructor (inner rel cannot be another HepRelVertex), we basically cannot apply a HepProgram on a tree containing already HepVertex. This was violated by this is violated by HiveHepExtractRelNodeRule, so all its caller now use an existing auxiliary method (based on a shuttle) for this unwrapping (HiveCalciteUtil.stripHepVertices).
  This was also violated by materialization rules in HiveMaterializedViewRule, which used its own private static class HiveHepExtractRelNodeRule in a HepProgram as unionRewritingPullProgram. As a workaround, it is proposed to extend the MaterializedView rules (as HiveMaterializedViewRule), override the rewriteQuery
  (where the UnionRewritingPullProgram is called), and unwrap the HepRelVertex tree ourselves (using (HiveCalciteUtil.stripHepVertices) before calling the super.rewriteQuery.
• The upgrade unveiled a bug in HiveRelFieldTrimmer#rewriteGBConstantKeys: the condition on its javadoc "none of the relnode above aggregate refers to these group keys" should not be checked via fieldsUsed.contains(aggregate.getGroupSet()) but rather fieldsUsed.intersects(aggregate.getGroupSet()).
  Also took the opportunity here to get rid of the unnecessary check aggregate.getIndicatorCount() > 0 which is always false (this method is deprecated and always returns zero)
• Disable simplifyValues in HiveRelBuilder config. Due to "[CALCITE-5717] RelBuilder.project of literals on a single-row Aggregate should create a Values" (introduced 1.35), cases like TestMiniLlapLocalCliDriver with file explainuser_1.q led to: java.lang.UnsupportedOperationException: Values with non-empty tuples are not supported. at org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.convert(ASTConverter.java:264). The reason is that RelBuilder#project_ contains the simplification (only applicable if config.simplifyValues is true): "If the expressions are all literals, and the input is a Values with N rows [...], replace with a Values with same tuple N times"; and with CALCITE-5717 that simplification was extended to not only "Values with N rows" but also "Aggregates with 1 row", and this case would lead to creating a non-empty HiveValues, which is not supported in our case. The easiest solution to prevent that from happening is disabling simplifyValues in HiveRelBuilder config.
• Due to a new assertion introduced in 1.41 via "[CALCITE-7070] FILTER_REDUCE_EXPRESSIONS crashes on expression BETWEEN ( NULL) AND X", now Calcite can throw an AssertionError in case of a wrong binary comparison predicate with null value java.lang.AssertionError: Comparison with NULL in pulledUpPredicates, so in order to avoid that, a few classes needed to be adjusted. Firstly HiveFunctionHelper#getExpression must not create such expressions (seen NULL = NULL in TestMiniLlapLocalCliDriver with subquery_null_agg.q); secondly avoid incorrect comparison ($snapshotIdInputRef <= NULL seen in TestIcebergCliDriver with mv_iceberg_orc.q) in HiveAugmentSnapshotMaterializationRule, use snapshotId -1 instead, and then switch it back in HivePushdownSnapshotFilterRule.