JSON AST with properties

datatype-parser/.gitignore

@@ -0,0 +1 @@
+build/

datatype-parser/CMakeLists.txt

@@ -0,0 +1,27 @@
+cmake_minimum_required(VERSION 3.16)
+project(chdt_datatype_parser CXX)
+
+# A self-contained library: no dependency on the ClickHouse source tree.
+set(CMAKE_CXX_STANDARD 20)
+set(CMAKE_CXX_STANDARD_REQUIRED ON)
+set(CMAKE_CXX_EXTENSIONS OFF)
+
+if(NOT CMAKE_BUILD_TYPE)
+    set(CMAKE_BUILD_TYPE Release)
+endif()
+
+add_compile_options(-Wall -Wextra)
+
+add_library(chdt_datatype_parser
+    src/lexer.cpp
+    src/parser.cpp
+    src/json.cpp
+)
+target_include_directories(chdt_datatype_parser PUBLIC include)
+
+# CLI: read a data-type string, print its JSON AST. Exit non-zero on error.
+add_executable(chdt-parse tool/main.cpp)
+target_link_libraries(chdt-parse PRIVATE chdt_datatype_parser)
+
+enable_testing()
+add_subdirectory(test)

datatype-parser/README.md

@@ -0,0 +1,107 @@
+# chdt — standalone ClickHouse data-type parser
+
+A small, self-contained C++ library that parses a ClickHouse **data-type
+string** (the kind sent in the types row of `RowBinaryWithNamesAndTypes`, e.g.
+`Array(Nullable(UInt64))`, `Tuple(a UInt8, b String)`, `Enum8('a' = 1)`,
+`Decimal(10, 2)`) into a JSON AST.
+
+It is extracted from the server's `ParserDataType`
+(`src/Parsers/ParserDataType.cpp`) but has **no dependency on the ClickHouse
+source tree** — only the C++20 standard library. The JSON it emits mirrors the
+data-type subtree of the frozen `EXPLAIN AST json = 1` document (format
+**version 2**; see `AST.md` in the ClickHouse repo), so its output is a drop-in
+match for what the server produces.
+
+## Why this exists
+
+The server's type parser is entangled with the lexer, the expression parsers,
+and the `IAST` / `Field` machinery. Vendoring all of that verbatim would pull
+in ~9–10k lines (`Field`, `ReadHelpers`, `Exception`, the formatting/hashing
+layer). Instead, this reimplements just the type grammar on a minimal AST:
+
+- a purpose-built tokenizer (`src/lexer.*`) covering the slice of SQL that type
+  strings use, in place of the full `Lexer` and its `UTF8Helpers` /
+  `find_symbols` dependencies;
+- a faithful port of `ParserDataType::parseImpl` (`src/parser.cpp`) — same
+  control flow: identifier + SQL-standard multi-word aliases, the Enum and
+  Tuple special cases, then the generic parametric-argument loop;
+- plain structs for the AST (`include/chdt/ast.h`) instead of `IAST` + `Field`.
+
+## Building
+
+```bash
+cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
+cmake --build build
+```
+
+This produces `libchdt_datatype_parser.a` and the `chdt-parse` CLI.
+
+## Usage
+
+Library:
+
+```cpp
+#include "chdt/parser.h"
+
+chdt::ParseResult r = chdt::parseDataType("Tuple(a UInt8, b String)");
+if (r.ok())
+    std::string json = chdt::toJSON(*r.ast);
+else
+    /* r.error->message, r.error->position */;
+```
+
+CLI:
+
+```bash
+./build/chdt-parse "Array(Nullable(UInt64))"
+echo "Enum8('a' = 1, 'b' = 2)" | ./build/chdt-parse
+```
+
+## Output shape
+
+Node types and slots match the server (format v2):
+
+| `type`           | slots                                                    |
+|------------------|----------------------------------------------------------|
+| `DataType`       | `name`, `arguments?` (present iff the type had `(...)`)  |
+| `EnumDataType`   | `name`, `values` (array of `{ name, value }`)            |
+| `TupleDataType`  | `name`, `arguments?`, `element_names?` (named tuples)    |
+| `NameTypePair`   | `name`, `data_type` (a `Nested(...)` element)            |
+| `Literal`        | `value_type`, `value` (64-bit ints as JSON strings)      |
+| `Function`       | `name`, `is_operator?`, `arguments` (e.g. `max_types=5`) |
+| `Identifier`     | `name`, `name_parts?`                                    |
+
+`EnumDataType.values` and `TupleDataType.element_names` are carried here exactly
+as the server emits them since format v2.
+
+## Coverage
+
+Supported: scalars, parametric types with literal args (`Decimal`,
+`FixedString`, `DateTime64`, …), nested type args (`Array`, `Map`, `Nullable`,
+`LowCardinality`, `Variant`, …), enums (explicit → `EnumDataType`;
+auto-assigned → generic `DataType`), named/unnamed/mixed tuples, `Nested`,
+`Dynamic(max_types = N)`, the legacy `Object('json')`, and the SQL-standard
+multi-word aliases (`DOUBLE PRECISION`, `CHAR VARYING`, `INT SIGNED`, …).
+
+**Deliberately not supported yet** (the parser returns a clear error):
+
+- `AggregateFunction` / `SimpleAggregateFunction` — needs the function-expression
+  parser the server reaches for here.
+- the new `JSON(...)` object-argument syntax (`JSON(a.b UInt32, SKIP x)`). Bare
+  `JSON` and legacy `Object('json')` parse fine.
+
+## Tests
+
+`ctest` runs two suites against the parser:
+
+- **oracle** (`test/oracle_compare.py`) — for each type in `test/cases.txt`,
+  compares the parser's JSON against the `data_type` subtree the real server
+  emits for `CREATE TABLE t (c <TYPE>) ENGINE = Null`. Needs a `clickhouse`
+  binary (default: `../build/programs/clickhouse`; override with
+  `-DCLICKHOUSE_BINARY=...`).
+- **unsupported** (`test/check_unsupported.py`) — asserts the deferred types in
+  `test/cases_unsupported.txt` are rejected.
+
+```bash
+ctest --test-dir build --output-on-failure
+```

datatype-parser/include/chdt/ast.h

@@ -0,0 +1,87 @@
+#pragma once
+
+/// Minimal, self-contained AST for ClickHouse data-type strings.
+///
+/// The node shapes mirror the frozen `EXPLAIN AST json = 1` document
+/// (format version 2; see ClickHouse `AST.md`) so that JSON produced here is
+/// a drop-in match for the data-type subtree the server emits — and a
+/// superset of it: `EnumDataType.values` and `TupleDataType.element_names`
+/// are carried here as they are in the server (since v2).
+///
+/// This header has no dependency on the ClickHouse source tree.
+
+#include <cstdint>
+#include <memory>
+#include <string>
+#include <vector>
+
+namespace chdt
+{
+
+enum class NodeKind
+{
+    DataType,       /// generic type: name + optional argument list
+    EnumDataType,   /// Enum / Enum8 / Enum16 with fully explicit values
+    TupleDataType,  /// Tuple, with optional element names
+    NameTypePair,   /// `name Type` element of a Nested(...)
+    Literal,        /// numeric / string argument (e.g. Decimal(10, 2))
+    Function,       /// operator/function argument (e.g. `max_types = 5`)
+    Identifier,     /// bare identifier argument
+};
+
+struct Node;
+using NodePtr = std::shared_ptr<Node>;
+
+struct EnumValue
+{
+    std::string name;
+    int64_t value = 0;
+};
+
+/// One node type for the whole tree. Only the fields relevant to `kind` are
+/// populated; serialization emits exactly the slots the server would.
+struct Node
+{
+    explicit Node(NodeKind kind_) : kind(kind_) {}
+
+    NodeKind kind;
+
+    /// DataType / EnumDataType / TupleDataType / Function / Identifier / NameTypePair
+    std::string name;
+
+    /// DataType / TupleDataType / Function argument list (children inlined in JSON).
+    std::vector<NodePtr> arguments;
+    /// DataType only: whether the type carried a parenthesised argument list at
+    /// all. `UInt8` omits the `arguments` slot; `Array(...)` emits it (possibly
+    /// empty). Tuple/Function always emit their list.
+    bool has_argument_list = false;
+
+    /// EnumDataType: explicit `'name' = value` pairs.
+    std::vector<EnumValue> values;
+
+    /// TupleDataType: element names. Empty => unnamed tuple (slot omitted).
+    std::vector<std::string> element_names;
+
+    /// NameTypePair: the element's type.
+    NodePtr data_type;
+
+    /// Literal: `value_type` is the Field type id ("UInt64", "Int64",
+    /// "Float64", "String"); `value` is the textual value.
+    std::string value_type;
+    std::string value;
+
+    /// Function: set for operators such as `equals`.
+    bool is_operator = false;
+
+    /// Identifier: populated when the identifier is compound (a.b).
+    std::vector<std::string> name_parts;
+
+    static NodePtr make(NodeKind kind_) { return std::make_shared<Node>(kind_); }
+};
+
+/// Serialize a node tree to JSON, matching the server's `formatASTAsJSON`
+/// shape for data types. `indent` < 0 produces compact output; >= 0 produces
+/// pretty output with that many spaces per level.
+std::string toJSON(const Node & node, int indent = 2);
+
+}

datatype-parser/include/chdt/parser.h

@@ -0,0 +1,44 @@
+#pragma once
+
+/// Public entry point: parse a ClickHouse data-type string into the AST in
+/// `ast.h`. Self-contained — no dependency on the ClickHouse source tree.
+///
+/// Coverage mirrors the server's `ParserDataType` (`src/Parsers/ParserDataType.cpp`)
+/// with two deliberate omissions, deferred for now:
+///   * AggregateFunction / SimpleAggregateFunction — would pull in the full
+///     function-expression parser; a clear error is returned instead.
+///   * the new JSON/Object path-typed arguments (`JSON(a.b UInt32, SKIP x)`).
+///     The bare `JSON` type and legacy `Object('json')` parse fine; the
+///     object-argument syntax returns an error.
+/// Everything else — nested types, parametric types, enums (explicit and
+/// auto-assigned), named/unnamed tuples, Nested, Dynamic(max_types=N), and the
+/// SQL-standard multi-word aliases — is supported.
+
+#include <optional>
+#include <string>
+
+#include "chdt/ast.h"
+
+namespace chdt
+
+{
+
+struct ParseError
+{
+    std::string message;   /// human-readable description
+    size_t position = 0;   /// byte offset into the input where parsing stuck
+};
+
+struct ParseResult
+{
+    NodePtr ast;                       /// non-null on success
+    std::optional<ParseError> error;   /// set on failure
+
+    bool ok() const { return ast != nullptr; }
+};
+
+/// Parse the whole string as a single data type. Trailing tokens after a
+/// complete type are an error (the entire input must be one type).
+ParseResult parseDataType(const std::string & input);
+
+}