Skip to content

US Census 2025 Gazetteer & USGS Containment Dataset Pipeline#1919

Open
abhishekjaisw wants to merge 1 commit intodatacommonsorg:masterfrom
abhishekjaisw:us_census
Open

US Census 2025 Gazetteer & USGS Containment Dataset Pipeline#1919
abhishekjaisw wants to merge 1 commit intodatacommonsorg:masterfrom
abhishekjaisw:us_census

Conversation

@abhishekjaisw
Copy link
Contributor

An automated data pipeline to download, parse, and merge the newly released 2025 US Census Gazetteer files with USGS National Federal Codes. This update standardizes foundational geographic nodes (Places, Counties, Tracts, ZCTAs, Congressional Districts, etc.) by importing 2025 boundaries, geographic coordinates (LatLong), and LSADs. It features a custom pure-text "Last-Writer-Wins" merging strategy that seamlessly overwrites 2018–2022 data while safely injecting deleted historical names as 'alternateName' to ensure zero loss of historical search context in the Knowledge Graph. It also aggregates multi-county intersections to generate highly accurate 'containedInPlace' mappings.

@gemini-code-assist
Copy link
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request establishes a robust, automated data pipeline for integrating the 2025 US Census Gazetteer files and USGS National Federal Codes. The pipeline ensures the update of critical geographic data, including boundaries and coordinates, while intelligently preserving historical naming context and generating precise containment relationships for various US administrative divisions.

Highlights

  • 2025 US Census Gazetteer Integration: Implemented an automated data pipeline to download, parse, and merge the newly released 2025 US Census Gazetteer files.
  • Geographic Node Updates: Updated foundational geographic nodes (Places, Counties, Tracts, ZCTAs, Congressional Districts, etc.) with 2025 boundaries, geographic coordinates (LatLong), and Legal/Statistical Area Descriptions (LSADs).
  • Historical Context Preservation: Introduced a 'Last-Writer-Wins' merging strategy that overwrites 2018–2022 data while safely injecting deleted historical names as 'alternateName' to maintain search context.
  • USGS Containment Mapping: Aggregated multi-county intersections to generate highly accurate 'containedInPlace' mappings using USGS National Federal Codes.
  • Pipeline Scripts: Added new scripts for downloading raw data, parsing gazetteer data to MCF, merging with historical context, and mapping USGS county containment.
  • Test Data Updates: Renamed and updated test data files (S1251_output.mcf, S1251_output.tmcf, S1251_summary.json, column_map.json) to reflect 2024 data and adjusted property ordering.
Changelog
  • scripts/us_census/acs5yr/subject_tables/S1251/test/S1251_output.mcf
    • Updated statistical variable nodes by reordering maritalStatus and gender properties, and added age and occupancyStatus to several node identifiers and properties.
  • scripts/us_census/acs5yr/subject_tables/S1251/test/S1251_output.tmcf
    • Renamed from scripts/us_census/acs5yr/subject_tables/S1251/testdata/S1251_output.tmcf.
  • scripts/us_census/acs5yr/subject_tables/S1251/test/S1251_summary.json
    • Updated summary statistics and year from 2012 to 2024.
  • scripts/us_census/acs5yr/subject_tables/S1251/test/column_map.json
    • Updated the year to 2024, adjusted column keys to match new data, and reordered/added properties for statistical variables.
  • scripts/us_census/acs5yr/subject_tables/S1251/testdata/S1251_cleaned.csv
    • Removed.
  • scripts/us_census/gazetteer/README.md
    • Added a new README file detailing the 2025 Gazetteer and USGS Containment Dataset pipeline.
  • scripts/us_census/gazetteer/download_2025.sh
    • Added a new script to download 2025 US Census Gazetteer files.
  • scripts/us_census/gazetteer/gazetteer.py
    • Added a new Python script for parsing 2025 Gazetteer data into MCF format.
  • scripts/us_census/gazetteer/place_usgs.py
    • Added a new Python script to map USGS Federal Codes for place containment.
  • scripts/us_census/gazetteer/run_pipeline.py
    • Added a new Python script to manage the end-to-end processing and merging of Gazetteer data.
Activity
  • No human activity has been recorded on this pull request yet.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a comprehensive and well-documented data pipeline for the US Census 2025 Gazetteer and USGS containment data. The new scripts for downloading, parsing, and merging data are well-structured, and the strategy for preserving historical names is a thoughtful addition. However, I've identified a critical issue in run_pipeline.py where a local file path is hardcoded, which would prevent the script from running in other environments. I've provided a suggestion to resolve this. The updates to the test data for the S1251 subject table appear correct and consistent with the pipeline's evolution. Overall, this is a strong contribution that will be ready for merging after the hardcoded path is fixed.

@abhishekjaisw abhishekjaisw force-pushed the us_census branch 6 times, most recently from 488eb9f to 6e4e026 Compare March 16, 2026 14:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant