Add HTML to Markdown functionality

[navarone-feekery]

Problem Description

Markdown is the de facto standard for LLMs, so having an HTML -> markdown conversion functionality would help users who want to build RAG applications.

Proposed Solution

Add an optional HTML -> Markdown functionality

• Configurable in the YAML file
• When configured, adds a single field to ingested document, containing the converted markdown content
• When not configured, field is not present on ingested documents at all

[seanstory]

Just a note for prioritization, that I'd rather we implement this outside of Crawler, so that other ingestion mechanisms that might bring HTML could also benefit. The feature request from and end-user perspective is a good one, I'm just haggling over implementation details of where it's implemented.

[navarone-feekery]

@seanstory I considered if this was a good use-case for the data extraction service, but I thought not because we don't have any other products ingesting HTML (right?). But you raise a good point about future-proofing.
I'm ultimately agnostic about where we implement this, I just think it's a feature we should have somewhere.

[seanstory]

Confluence connector is another that might be a good fit, since the formatting options you have for a Confluence Page are pretty rich, and one of the less-lossy ways to export a page is through HTML. But I'm also thinking that 3rd-party crawlers or a local dir of HTML files could benefit too.

[sketchybinary]

Sorry about that, I don't usually use GitHub and I thought I was opening that MR in my fork.

I have this working in my project and I'm using it locally because I need it. Would you be open to a MR for me to add this in, or would you prefer that I figure out how to bring the feature in somewhere else?