Enhancement: Add Enrich Host - Get Host Enrichment API action for Censys with automatic fallback by jayminp-crest · Pull Request #1068 · chronicle/content-hub

jayminp-crest · 2026-08-04T05:11:58Z

Enhancement: Add Enrich Host - Get Host Enrichment API action for Censys with automatic fallback

Description

What problem does this PR solve?

Censys added a new, purpose-built host enrichment endpoint (GET /v3/global/asset/enrichment/host/{ip}) that returns richer risk-context signals for an IP than the existing Enrich IPs endpoint provides — GreyNoise actor/classification, privacy/network risk flags (Tor, VPN, proxy, anonymous, relay, hosting, mobile, satellite), and third-party threat-actor intelligence (Mallory). There was no way to consume this from the Google SecOps SOAR integration.

How does this PR solve the problem?

Added a new action, Enrich Host - Get Host Enrichment API, that calls the new endpoint per-IP (the endpoint doesn't support batch calls or the At Time param).
Renamed the existing Enrich IPs action to Enrich Host - Get Host API for clearer naming — no functional change.
Added a new instance-level config toggle, Enable Get Host Enrichment API (default false), so the new action is opt-in per instance.
Added a new reusable block, Censys - Host Enrichment Block, that runs the new action first and automatically falls back to Enrich Host - Get Host API if the toggle is off or the API returns an account-level error (401/403/409/429).
Updated Censys - Entity Enrichment (v1.0 → v2.0) and Censys - Rescan (v2.0 → v3.0) to use the new block in place of their inline IP-enrichment steps.
Added unit tests (test_enrich_host.py) and supporting docs (ENRICH_HOST_CHANGE_ANALYSIS.md, User Guide v2 supplement).

Any other relevant information (design choices, tradeoffs, known issues):

Toggle defaults to off, so existing behavior is unchanged until an instance explicitly opts in.
The new action ends FAILED (not COMPLETED) on account-level API errors, matching the convention of the integration's other actions; the fallback condition step in the block has AutoSkipOnFailure set so it can still evaluate and route to the fallback action even when the enrichment step fails.
Enrich IPs (now Enrich Host - Get Host API) is otherwise untouched — no regression risk to existing playbooks/cases using it, beyond the display-name change.

Checklist

General Checks:

I have read and followed the project's contributing.md guide.
My code follows the project's coding style guidelines.
I have performed a self-review of my own code.
My changes do not introduce any new warnings.
My changes pass all existing tests.
I have added new tests where appropriate to cover my changes. (test_enrich_host.py)
I have updated the documentation where necessary. (User Guide v2 supplement, change-analysis doc, release notes for integration/playbooks/block)

Open-Source Specific Checks:

My changes do not introduce any Personally Identifiable Information (PII) or sensitive customer data.
My changes do not expose any internal-only code examples, configurations, or URLs.
All code examples, comments, and messages are generic and suitable for a public repository.
I understand that any internal context or sensitive details related to this work are handled separately in internal systems (Buganizer for Google team members).

For Google Team Members and Reviewers Only:

I have included the Buganizer ID in the PR title or description, if applicable.
I have ensured that all internal discussions and PII related to this work remain in Buganizer.
I have tagged the PR with one or more labels that reflect the pull request purpose.

gemini-code-assist · 2026-08-04T05:12:01Z

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

gemini-code-assist · 2026-08-04T05:53:21Z

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

Signed-off-by: jayminp-crest <jaymin.prajapati@crestdata.ai>

jayminp-crest · 2026-08-24T09:13:35Z

Hi @AmitJ98 - Could you please review this PR?

jayminp-crest · 2026-09-08T08:56:45Z

hi @KrishnaSharma06 / @chanduc-ux / @salvatoreasammito - Could you please review this PR or redirect to the relevant person for the review? PR has been open since last month.

jayminp-crest · 2026-09-17T05:25:11Z

Hi @adarshtwy - Could you please review this PR and Let me know if there any changes required?

adarshtwy · 2026-09-17T06:25:11Z

@@ -1 +1,12 @@
-[]
+[
+  {


Both content/parsers/third_party/PARTNER_INDEX.json and content/parsers/third_party/COMMUNITY_INDEX.json were modified in this branch by the fork's automated parser index workflow (Automated update of parser indexes [skip ci]). Please revert both files to origin/main (git checkout origin/main -- content/parsers/third_party/COMMUNITY_INDEX.json content/parsers/third_party/PARTNER_INDEX.json) so this PR only contains changes for the Censys integration and playbooks.

I have removed it via your suggested command but after commit it gets added via the aautomated workflow

adarshtwy · 2026-09-17T06:25:11Z

+    name: Branches
+    value: '[{"LogicalOperator":0,"Conditions":[{"Operator":0,"FieldName":"[Enrich
+        Host - Get Host Enrichment API.ScriptResult]","Type":7,"Value":"false","CustomOperatorName":"Core
+        Functions.Equal"}],"Order":1,"IsDefaultBranch":false,"Name":"Branch"},{"LogicalOperator":0,"Conditions":[],"Order":2,"IsDefaultBranch":true,"Name":"Branch"}]'


This condition routes to the fallback Enrich Host - Get Host API action whenever [Enrich Host - Get Host Enrichment API.ScriptResult] equals "false". However, in enrich_host.py (line 396), result_value is also False whenever all target IPs return 404 Not Found (not_found_entities), even when Enable Get Host Enrichment API is enabled and no account-level error occurred. That means every unindexed IP (404) will trigger a redundant fallback call to Enrich Host - Get Host API, consuming double API quota.

To ensure fallback only runs when the feature toggle is disabled or an account-level error (401/403/409/429) occurs, expose an explicit flag in JsonResult (e.g., {"fallback_required": true}) or check the execution state/output condition so routine 404 Not Found outcomes do not trigger the fallback action.

adarshtwy · 2026-09-17T06:25:12Z

+        reputation = self.host_data.get("reputation", {})
+        greynoise = self.host_data.get("greynoise", {})
+        privacy = self.host_data.get("privacy", [])
+        network = self.host_data.get("network", [])


Null-Safety Bug with JSON null Sub-Objects: In Censys v3 JSON payloads (as seen in enrich_host_JsonResult_example.json), unset fields/objects can be returned as explicit null (None in Python). When a key exists with value None (e.g., {"greynoise": null} or {"third_party": null}), .get("greynoise", {}) returns None rather than {}, causing greynoise.get("actor") to raise AttributeError: 'NoneType' object has no attribute 'get'. Use or {} / or [] on all nested dictionary and list lookups (including (raw_data.get("result") or {}).get("resource") or {} in __init__).

Method Length (119 Lines vs. 50-Line Maximum): get_enrichment_data() spans lines 392–510 (119 lines). Please split field extraction into smaller helper methods (e.g., _get_core_host_fields, _get_location_and_network_fields, _get_risk_and_privacy_fields, _get_mallory_fields), each <= 50 lines.

Python 3.11+ Type Annotations: Since from __future__ import annotations is enabled, prefer dict[str, Any], list[Any], and str | None over Dict, List, and Optional.

adarshtwy · 2026-09-17T06:25:12Z

+            or first_opinion.get("reference_source_slug"),
+        }
+
+        return {k: v for k, v in enrichment.items() if v is not None}


_get_top_values (and is_found) in HostEnrichmentDatamodel is an exact duplicate of HostDatamodel._get_top_values (lines 345–366). Move _get_top_values to BaseModel or have HostEnrichmentDatamodel inherit from HostDatamodel to eliminate the duplicated method.

adarshtwy · 2026-09-17T06:25:12Z

+            }
+        ],
+    )
+    def test_enrich_host_success(


test_enrich_host_success is 70 lines long (lines 25–94), exceeding the 50-line function limit because the mock response dictionary is defined inline. Extract the sample enrichment payload into a module-level constant (SAMPLE_ENRICH_HOST_RESPONSE) or a pytest fixture so the test function remains concise (<= 50 lines).

Fix retry logic re-raising RateLimitException instead of the original exception on exhausted retries, and null-safety in HostEnrichmentDatamodel where explicit JSON null sub-objects caused AttributeError. Replace datetime.utcnow() with timezone-aware datetime.now(timezone.utc). Consolidate duplicate error logging in enrich_host.py. Decompose main() and validate_response() into smaller helper functions to meet the 50-line/3-argument guideline, using EnrichmentSummary and EnrichmentContext dataclasses to replace the multi-list parameter clump. Extract repeated truncation logic into _format_entity_preview(). Extract inline test payload into SAMPLE_ENRICH_HOST_RESPONSE. Add explicit return type annotations and concrete generics to new conftest mock helpers. Modernize typing (Dict/List/Optional -> dict/list/X | None) in touched files. Fix release note action name and description. Verified with mp test (74/74 passed), mp check --static-type-check, and mp validate (integration + playbook), all clean.

…t actions Adds reputation.label, score, score_suppressed, class_probabilities, and evidence to both actions' entity enrichment and case wall output, plus three combined reputation data tables (Summary, Class Probabilities, Evidence) rendered via a shared ReputationExtractor. Also gitignores local-only working files (.mcp.json, Censys-new-support/, REVIEW-COMMENTS.MD). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Added New Host Enrichment Action & Playbook Update

6239716

jayminp-crest added 2 commits August 4, 2026 11:16

remove unnecessary files

2c0e813

priority change revert

151d978

jayminp-crest marked this pull request as ready for review August 4, 2026 05:53

jayminp-crest requested a review from a team as a code owner August 4, 2026 05:53

jayminp-crest and others added 2 commits August 5, 2026 14:57

Merge branch 'main' into feature/censys-integration-new-host-enrichment

54ce84c

Automated update of parser indexes [skip ci]

f4f73d8

github-actions Bot requested a review from a team as a code owner August 5, 2026 09:27

jayminp-crest and others added 7 commits August 5, 2026 14:59

Update version in constant

648eade

Merge branch 'main' into feature/censys-integration-new-host-enrichment

eab21ce

Signed-off-by: jayminp-crest <jaymin.prajapati@crestdata.ai>

Automated update of parser indexes [skip ci]

8cd73a2

Merge branch 'main' into feature/censys-integration-new-host-enrichment

bad4c59

Automated update of parser indexes [skip ci]

e06bf12

Merge branch 'main' into feature/censys-integration-new-host-enrichment

841d7ec

Automated update of parser indexes [skip ci]

8dae2d5

jayminp-crest and others added 5 commits August 24, 2026 14:43

Merge branch 'main' into feature/censys-integration-new-host-enrichment

51a67c3

Automated update of parser indexes [skip ci]

b8143a7

Merge branch 'main' into feature/censys-integration-new-host-enrichment

04a7687

Merge branch 'main' into feature/censys-integration-new-host-enrichment

b9ebf8d

Automated update of parser indexes [skip ci]

35696fe

Merge branch 'main' into feature/censys-integration-new-host-enrichment

70eb806

jayminp-crest requested review from a team and adarshtwy as code owners September 17, 2026 05:23

Automated update of parser indexes [skip ci]

ed3e8be

adarshtwy requested changes Sep 17, 2026

View reviewed changes

jayminp-crest added 2 commits September 21, 2026 12:42

remove parser changes

cbd652d

jayminp-crest force-pushed the feature/censys-integration-new-host-enrichment branch from 8f81478 to cbd652d Compare September 21, 2026 07:13

secops-parsers-helper-bot and others added 6 commits September 21, 2026 07:13

Automated update of parser indexes [skip ci]

ca36be2

Merge branch 'main' into feature/censys-integration-new-host-enrichment

62df0ca

Automated update of parser indexes [skip ci]

aead8f3

removed the parser files

d1c3128

Automated update of parser indexes [skip ci]

7b333e8

jayminp-crest requested a review from adarshtwy September 24, 2026 06:51

Navigation Menu

Enhancement: Add Enrich Host - Get Host Enrichment API action for Censys with automatic fallback - #1068

Enhancement: Add Enrich Host - Get Host Enrichment API action for Censys with automatic fallback#1068
jayminp-crest wants to merge 27 commits into
chronicle:mainfrom
jayminp-crest:feature/censys-integration-new-host-enrichment

jayminp-crest commented Aug 4, 2026

Uh oh!

gemini-code-assist Bot commented Aug 4, 2026

Uh oh!

gemini-code-assist Bot commented Aug 4, 2026

Uh oh!

jayminp-crest commented Aug 24, 2026

Uh oh!

jayminp-crest commented Sep 8, 2026

Uh oh!

jayminp-crest commented Sep 17, 2026

Uh oh!

adarshtwy Sep 17, 2026

Uh oh!

jayminp-crest Sep 21, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adarshtwy Sep 17, 2026

Uh oh!

adarshtwy Sep 17, 2026

Uh oh!

adarshtwy Sep 17, 2026

Uh oh!

Uh oh!

Uh oh!

adarshtwy Sep 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

@@ @@ -1 +1,12 @@ @@
-              []
+              [
+                {

Conversation

jayminp-crest commented Aug 4, 2026