Extension md_in_html does not recognize tags with hyphens

[igordsm]

Web components are custom HTML components that are required to have - in their names. This breaks current HTML handling since these elements are not considered. IMHO they should be treated the same as <div> ("block" elements, if I'm not mistaken).

The following was tested in current main with the extension md_in_html active.

input

<a-b>

asdf

</a-b>

output:

<p><a-b></p>
<p>asdf</p>
<p></a-b></p>

expected:

<a-b>
<p>asdf</p>
</a-b>

I went through the code and might know how to add this, but I would like the maintainers' input before proceeding.

[waylan]

Web components are custom HTML components that are required to have - in their names.

Can you point us to a spec for this?

[igordsm]

The resource I use the most is MDN: https://developer.mozilla.org/en-US/docs/Web/Web_Components/Using_custom_elements

The actual specification of valid names is at https://html.spec.whatwg.org/#valid-custom-element-name

[waylan]

Thank you for the links. There are two things I need to mention here.

First of all, the way Python-Markdown handles raw HTML is to define a list of known block-level tags. Any content within those block-level tags gets special treatment. Anything outside those known block-level tags is just treated as regular Markdown content, including inline raw HTML elements, which explains the behavior of the sample provided above.

Second, I will note that to use custom elements, the HTML spec requires you to register the elements with the browser first. Without registering them, then the browser would have no knowledge of how to handle them. In fact, a custom element could be an inline element or a block-level element.

Given the above, I think that the logical way to support custom elements in Python-Markdown is to require the user to "register" the elements. That is, if you have a custom element which should be treated as a block-level element, you need to inform the Markdown class about it. This would probably make a good candidate for a third party extension (extension to register custom elements), although you can do this without an extension as demonstrated below.

>>> src = '''
... <a-b>
...
... asdf
...
... </a-b>
... '''
>>> md = markdown.Markdown()
>>> md.block_level_elements.append('a-b')
>>> md.convert(src)
'<a-b>\n\nasdf\n\n</a-b>'

That said, this does not currently work correctly with the md_in_html extension. Specifically, the extension fails to allow Markdown parsing within the element.

>>> md = markdown.Markdown(extensions=['md_in_html'])
>>> md.block_level_elements.append('a-b')
>>> md.convert(src)
'<a-b>\n\nasdf\n\n</a-b>'

This would appear to be because the extension compiles its lists of element types when the class instance is created and therefore does not see the changes made to the Markdown class latter (see relevant code here). Ideally, the extension would build its list of element types after all extensions are loaded. I'm open to a PR which makes this change only. However, I do not see any need to add explicit support for custom elements specifically.

[igordsm]

Thanks for the detailed feedback @waylan . I'll try and make a PR with the changes you outlined above this week.

[DSLituiev]

this is a big limitation. It also fails to parse "details" tag, and there seem to be no recourse to add it

[facelessuser]

It also fails to parse "details" tag, and there seem to be no recourse to add it

@DSLituiev This is simply not true. The Markdown parser handles the "details" tag just fine. Or are you arguing that it doesn't parse the content within the "details" tag automatically? You can enable md_in_html and attach the markdown attribute and then it will parse the content of the "details" tag.

<details markdown>
<summary>Summary</summary>

Markdown **content**.

</details>

[waylan]

I understand that some users do not like how Markdown handles raw HTML. However, we did not create the rules here. We are only implementing them (specifically these rules. The md_in_html extension was implemented to match what was the most common way at the time to allow Markdown processing in raw HTML. Since then, various other solutions have become more popular. Personally, I am surprised that no one has as-yet created an extension which parses raw HTML according to the Commonmark spec. To be clear, we are not a CommonMark implementation and there are thousands of users with many thousands of existing documents using the existing implementation, so the default behavior and the behavior of md_in_html need to remain in place to continue supporting those existing users. Any different way of handling raw HTML would need to be implemented in a separate extension. And we are already spread too thin with the maintenance burden of the existing built-in extensions. Therefore, any new extensions would need to be created and maintained by a third-party.

That said, we are still open to the limited change I mentioned in my previous comment on this issue. It's just that the maintainers of the project aren't using HTML components and therefore this is not a high priority for us. However, we will happily review any PR which is submitted.