Skip to content

boston-lot: metadata review loop - #227

Open
pointcloud-org wants to merge 3 commits into
mainfrom
boston-lot-metadata-review
Open

boston-lot: metadata review loop#227
pointcloud-org wants to merge 3 commits into
mainfrom
boston-lot-metadata-review

Conversation

@pointcloud-org

Copy link
Copy Markdown
Contributor

Exercising the new metadata inventory + explain verb against Boston Lot.

The manifest still has license: UNKNOWN and a TODO description on purpose — that is what we are trying to resolve with LLM-assisted enrichment.

@pointcloud-org

Copy link
Copy Markdown
Contributor Author

boston-lot

  • ✅ file existence: all referenced objects found
  • ⚪ pdal_filters: none declared -- nothing to check
  • ⚪ CRS consistency: not applicable (no items_dir)

@pointcloud-org

pointcloud-org commented Aug 21, 2026

Copy link
Copy Markdown
Contributor Author

📋 Metadata inventory — boston-lot

Everything in the tables below is parsed straight out of the submitted file's LAS public header and VLR records (a few kilobytes, read with a range request — the file itself is never downloaded). Nothing here is inferred or model-written.

What the file says about itself

boston-lot-2021-09-01-CRREL-RSGIS.copc.laz
Points 1,746,342,476
File size 18.5 GiB
Format LAS 1.4, point format 6, COPC
CRS NAD83 / UTM zone 18N
Linear units metre
XY extent 5633 × 6273
Z range -2159.6 – 742.7
Acquiring system empty
Generating software empty
File creation date unset
GPS time range none (0–0)
Extra dimensions 5
VLRs 4
Dimensions and VLRs — boston-lot-2021-09-01-CRREL-RSGIS.copc.laz

Extra dimensions (beyond the standard LAS point format)

Name Type Description
HeightAboveGround double
Synthetic uint8
Overlap uint8
Withheld uint8
KeyPoint uint8

Variable Length Records

# user_id record_id Bytes What it is
0 copc 1 160 COPC info
1 laszip encoded 22204 46 LASzip compression
2 LASF_Projection 2112 615 OGC WKT projection
3 LASF_Spec 4 960 Extra Bytes (extra dimensions) — HeightAboveGround (double), Synthetic (uint8), Overlap (uint8), Withheld (uint8), KeyPoint (uint8)

What the file does not tell us

These are the metadata a reviewer would otherwise assume came from the data. They did not — so if the manifest states them, they came from somewhere else.

  • LAS system_id is empty — the file names no acquiring system
  • LAS generating_software is empty — the file names no creating application
  • no file creation date (creation_year is unset, which PDAL reports as 1) — any acquisition date in the manifest comes from a filename or a human, not from the data
  • GPS time range is 0–0 — the points carry no usable timestamps, so the data cannot date itself
  • no pdal_metadata VLR — the file carries no record of the pipeline that produced it (PDAL only writes this when asked with pdal_metadata=true)
  • declared Z range spans 2902 m (-2159.6 to 742.7) — larger than almost any real survey, so the header bounds are probably set by outlier points rather than terrain; anything derived from them (elevation ranges, hero-image colour ramps) will be skewed until they are filtered

Manifest fields still needing an answer

  • license is UNKNOWN — a real SPDX identifier or proprietary is needed before publication. The model may quote what a source says about licensing, but a human decides it.
  • description is still a placeholder (TODO: write a one-line summary for Boston Lot)
  • no pointcloud_org.acknowledgement
  • no sci:citation or sci:doi — nothing for a user to cite

Reply with @pointcloud-org, please explain … to ask about any of this, or point the bot at an external metadata document and it will read it as part of answering.

@hobu

hobu commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

@pointcloud-org, please explain what we actually know about this dataset's provenance, and whether the 2021-09-01 date is supported by anything

@pointcloud-org

Copy link
Copy Markdown
Contributor Author

🤖 Request received from @hobu (admin access) via comment.

explain what we actually know about this dataset's provenance, and whether the 2021-09-01 date is supported by anything

Working on it…

@pointcloud-org

Copy link
Copy Markdown
Contributor Author

I couldn't tell which command you meant. Supported commands:

  • @pointcloud-org​, please reingest <DATASET_ID> — re-run ingest for one dataset. Opens a pull request for an admin to merge; the ingest happens on merge.
  • @pointcloud-org​, please refresh-stac — rebuild the archive-wide STAC catalog, collections manifest, and the single-file items.parquet.
  • @pointcloud-org​, please check-reachability — check every dataset's external URLs and report anything unreachable.

Dataset ids are directory names under manifests/ (for grouped datasets, just the leaf, e.g. MN_SEDriftless_3_2021).

@pointcloud-org

Copy link
Copy Markdown
Contributor Author

🤖 explain — asked by @hobu

What the file actually establishes

Everything below is from the LAS header/VLRs, which are ground truth:

  • Format and structure: LAS 1.4, point format 6, COPC, 1,746,342,476 points, 18.5 GiB.
  • CRS: NAD83 / UTM zone 18N, linear units metre — so Z values are metres, no compound-CRS unit trap here.
  • Footprint: 5633 × 6273 (metres) in XY.
  • Processing fingerprint: an Extra Bytes VLR carrying HeightAboveGround (double) plus Synthetic, Overlap, Withheld, KeyPoint as uint8. That combination is characteristic of a PDAL-based post-process (HAG computation, classification flags demoted to extra dimensions) — but it names no tool or version.

What the file does not establish — which is nearly all the provenance

  • system_id empty: the file names no acquiring sensor or platform.
  • generating_software empty: the file names no producing application.
  • No pdal_metadata VLR: no record of the pipeline, so the inference above about PDAL is circumstantial, not documented.
  • creation_year unset (PDAL reports 1): no file creation date.
  • GPS time range 0–0: the points carry no usable timestamps. The data cannot date itself at all.

So there is no in-file evidence of who flew it, with what, when, for whom, or under what license.

The 2021-09-01 date

Not supported by anything you have given me. Its only appearance is inside the filename boston-lot-2021-09-01-CRREL-RSGIS.copc.laz. The two header fields that could corroborate an acquisition date are both blank — creation date unset, GPS time 0–0 — so the date is a human's filename convention, not a measurement. Treat it as an unverified claim, and note that even if the string is faithful it could as easily be a processing or delivery date as a flight date.

The CRREL-RSGIS token is in the same position: a filename string, plausibly an organizational initialism, but nothing in the file corroborates any provider. I am not going to expand it into an institution name on that basis.

I also cannot compare any of this against the manifest — manifests/boston-lot/manifest.yaml could not be read at this commit, so I have no submitted claims for license, description, extent.temporal.interval, providers[], or sci:citation to check.

One substantive data concern, independent of provenance

Declared Z spans -2159.6 to 742.7 m, a 2902 m range. For a ~5.6 × 6.3 km footprint that is not terrain; the header bounds are almost certainly set by a handful of outlier points. Anything a user or the site derives from those bounds — elevation range in the collection extent, hero-image colour ramp — will be wrong until the outliers are filtered.

To close this out you would need

  1. The manifest itself (re-fetch at a readable commit).
  2. An external source for the flight window — a survey report, delivery metadata, or a project page from the originating organization — since the file can never supply it.
  3. A stated license from that same source; nothing in the file speaks to licensing, and I won't infer one.
  4. Confirmation of what CRREL-RSGIS denotes, from the submitter or a source document, before it becomes a providers[] entry.

claude-opus-5 · 2205 in / 1323 out tokens

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants