Shortcode output wrapped in <p> tags under certain conditions

[josharcheruk]

Previous related issue 'Shortcode output wrapped in <p> tags' #1148 which was closed with 004fcdd.

Consistent incorrect wrapping of shortcode output under certain conditions

Reported in discuss forum here.

Salient points:

1. If there is just one shortcode, it doesn't get wrapped in <p> tags

2. If there are multiple shortcodes, the first doesn't get wrapped in <p> tags (probably related to above)

3. In order to ensure no shortcodes are wrapped in <p> tags, the final shortcode must be followed by a number of distinct <p> tag wrapped lines proportionate to the number of shortcodes in the document.

By that I mean if you have say 6 shortcodes one after the other, you need 16 lines following them in order for none to be wrapped in <p> tags. If you have 5 shortcodes one after the other you need 12 lines.

4. 'Lines following shortcode' don't have to come at the end of the document, if you have a series of shortcodes it seems the number of interceeding <p> tag wrapped lines are cumulatively ranked towards each consecutive shortcode so if you had 30 lines interspaced between shortcodes throughout the document, as long as each shortcode has at least 5 proceeding lines each, they'll all be spared wrapping.

5. Related to point 4, the <p> tags that get wrapped around shortcodes incorrectly also count towards the number of <p> tag wrapped lines needed to follow preceeding shortcodes - so enough shortcodes in a row and you seem to be about to start unwrapping some of the 'first' few in the list.

6. Length of shortcode title attribute also affects whether output wrapped in <p> tags though this has not been investigated in detail

NOTE - I may have miss counted the exact number of lines in between shortcodes, my eyes were actually bleeding...

[hnarayanan]

As another participant in this discussion on discuss.gohugo.io, I can second that I see the above behaviour. I'm also willing to help continue debugging this.

[tburt]

Am a new user... really appreciate the project, is incredibly elegant. Would really like to see bug fixed :)

[bep]

See 1ce184b

These test cases are deliberately made green, even if they (probably) should not.

Test case 1-3:

1. The group of a, b, c shortcodes gets wrapped in a P-tag. This kind of makes sense. Is this correct?
2. b,c i wrapped, d is not. Inconsistent, but what is correct?
3. A weird one. This is @PuffinBlue s variant with 5 shortcoes and 12 lines ...

[AnBauer]

We ran into this as well. We also had the problem, that a markdown list of shortcodes inserted a wrong closing p-tag after an image from within the shortcode.

Would it eventually make sense introduce a param or something like that, that prevents hugo from wrapping shortcode in p-tags?

[bep]

@AnBauer Hugo doesn't add any p-tags, we actually try to remove them ... This is coming from Blackfriday (the Markdown engine) ... so a flag wouldn't help ...

[AnBauer]

You meant p-tags, didn't you? OK, I thought "shortcode output gets wrapped in a P-tag" is a thing, hugo does. Thanks for clarification!

[bep]

You meant p-tags, didn't you?

Yes.

[Jos512]

I also have this issue, although its behaviour is better on Hugo 0.15 but does happen when multiple shortcodes are used on the same page (as already mentioned above). Would love to see Hugo fix the Blackfriday output that adds those

's is that's possible.

[bep]

First, a workaround:

Create a shortcode wrap.html:

{{ .Inner }}

Then wrap your shortcode sections in that:

{{< wrap >}}
{{< s >}}
{{< s >}}

{{< s >}}
{{< /wrap >}}

To be even more protective, wrap it in a div:

<div>
{{< wrap >}}
{{< s >}}
{{< s >}}

{{< s >}}
{{< /wrap >}}
</div>

I think Hugo should do two things:

1. Provide an internal wrapper shortcode (name?)
2. Provide a way to signal that a shortcode is "block level" and not needed in the Mardkdown evaluation (this is currently links; the ref and relref internal shortcodes). This will add a temporary div to the shortcode placeholders when sent to Blackfriday.
   1. The "block level" will be the default.

/cc @halostatue @spf13

[ahausamm]

Currently, i have the same problem and i for me the workaround does not work properly. Is there any progress from the hugo-team?

[bep]

@ahausamm what is the problem with my workaround?

[ahausamm]

@bep For me, it works only with the

before the shortcodes, the {{ wrap }} does not change anything. But i don't want to have some HTML in the markdown, do you have an example of your working workaround?

{{< go_table >}}
    {{< go_table_row >}}
        {{% go_table_cell %}}
## Cell 1|1
        {{% /go_table_cell %}}
        {{% go_table_cell %}}
## Cell 2|1
        {{% /go_table_cell %}}
        {{% go_table_cell %}}
## Cell 3|1
        {{% /go_table_cell %}}
    {{< /go_table_row >}}
    {{< go_table_row >}}
        {{% go_table_cell %}}
## Cell 1|2
        {{% /go_table_cell %}}
        {{% go_table_cell %}}
## Cell 2|2
        {{% /go_table_cell %}}
        {{% go_table_cell %}}
## Cell 3|2
        {{% /go_table_cell %}}
    {{< /go_table_row >}}
{{< /go_table >}}

{{< go_table >}}
    {{< go_table_row >}}
        {{% go_table_cell %}}
## Cell 1|1
        {{% /go_table_cell %}}
        {{% go_table_cell %}}
## Cell 2|1
        {{% /go_table_cell %}}
        {{% go_table_cell %}}
## Cell 3|1
        {{% /go_table_cell %}}
    {{< /go_table_row >}}
    {{< go_table_row >}}
        {{% go_table_cell %}}
## Cell 1|2
        {{% /go_table_cell %}}
        {{% go_table_cell %}}
## Cell 2|2
        {{% /go_table_cell %}}
        {{% go_table_cell %}}
## Cell 3|2
        {{% /go_table_cell %}}
    {{< /go_table_row >}}
{{< /go_table >}}

gives me this output (first table seems to be okay, second has the

around:

<table>
    <tr>
        <td><h2 id="cell-1-1:d680e8a854a7cbad6d490c445cba2eba">Cell 1|1</h2>
</td>
        <td><h2 id="cell-2-1:d680e8a854a7cbad6d490c445cba2eba">Cell 2|1</h2>
</td>
        <td><h2 id="cell-3-1:d680e8a854a7cbad6d490c445cba2eba">Cell 3|1</h2>
</td>
    </tr>
    <tr>
        <td><h2 id="cell-1-2:d680e8a854a7cbad6d490c445cba2eba">Cell 1|2</h2>
</td>
        <td><h2 id="cell-2-2:d680e8a854a7cbad6d490c445cba2eba">Cell 2|2</h2>
</td>
        <td><h2 id="cell-3-2:d680e8a854a7cbad6d490c445cba2eba">Cell 3|2</h2>
</td>
    </tr>
</table>

<p><table>
    <tr>
        <td><h2 id="cell-1-1:d680e8a854a7cbad6d490c445cba2eba">Cell 1|1</h2>
</td>
        <td><h2 id="cell-2-1:d680e8a854a7cbad6d490c445cba2eba">Cell 2|1</h2>
</td>
        <td><h2 id="cell-3-1:d680e8a854a7cbad6d490c445cba2eba">Cell 3|1</h2>
</td>
    </tr>
    <tr>
        <td><h2 id="cell-1-2:d680e8a854a7cbad6d490c445cba2eba">Cell 1|2</h2>
</td>
        <td><h2 id="cell-2-2:d680e8a854a7cbad6d490c445cba2eba">Cell 2|2</h2>
</td>
        <td><h2 id="cell-3-2:d680e8a854a7cbad6d490c445cba2eba">Cell 3|2</h2>
</td>
    </tr>
</table></p>

Any ideas?

[bep]

@ahausamm use plain HTML tables and the {{< -- your use case was not my intented target for my "workaround" ... and It looks like a hack.

[ahausamm]

@bep okay, thanks for your answer. For me, plain HTML isn't the right solution (for the moment) because i want to have some custom section with custom templates. So i will try another way or simply wait on the next update of hugo.

[AnBauer]

the workaround from @bep came unfortunately too late for my project, because we already have tons of mds with tons shortcodes in it. what I did was creating a grunt-task, which simply removes all wrong p-tags with a regex :-)

[maxxscho]

In my case a temporary workaround is to wrap each shortcode in a block-element for example a div

<div>
{{< shortcode >}}
</div>

FYI

[ahausamm]

I think, the workaround of @maxxscho would be the best workaround for this problem. I had another nice discussion about this problem in the gohugo-forum: https://discuss.gohugo.io/t/single-page-with-nested-content/2645

[AnBauer]

@maxxscho: nice idea. Unfortunately I have multiple shortcodes already consisting of block elements. And I didn't want html-code in the md files or needless nesting.
These are the patterns for the replace-job btw. (after htmlmin ran):

patterns: [
    {
        match: /<p><(div|section|md-content|h3|ul)(.*?)>/g,
        replacement: '<$1$2>'
    },
    {
        match: /<.(div|section|md-content|h3|ul)><.p>/g,
        replacement: '</$1>'
    }
]

[bep]

As I think I said before, these "workarounds" should be included in the Hugo core. If we have to add a div to trick Blackfriday, we will remove it afterwords.

I will do it ... once I get some spare time.

[maxxscho]

@AnBauer: nice one. Can you show me your gulp task, with handles the replacement? :)
Thx!

[ahausamm]

@maxxscho i think he uses gulp-replace-task (https://www.npmjs.com/package/gulp-replace-task). @AnBauer please correct me if i'm wrong.

[ahausamm]

@AnBauer actually im wondering how i can use gulp with hugo server -w to create valid html code and replace the "shortcoddmistakes"

[AnBauer]

I'm using grunt for that. But it should be nearly the same with gulp. Important is to htmlmin before replacing, so all line breaks are deleted. The config for grunt-replace looks like:

replace: {
    dist: {
        options: {
            patterns: [
                {
                    match: /<p><(div|section|md-content|h3|ul)(.*?)>/g,
                    replacement: '<$1$2>'
                },
                {
                    match: /<.(div|section|md-content|h3|ul)><.p>/g,
                    replacement: '</$1>'
                }
            ]
        }
    ,
        files: [
            {
                cwd: '<%= dist %>/',
                expand: true,
                flatten: false,
                filter: 'isFile',
                src: ['**/*.html'],
                dest: '<%= dist %>/'
            }
        ]
    }
}

The elements in the regex depend on the project, of course. E.g. I also use angular material, so I needed md-content.

@ahausamm: I have a custom "hugo" - task. When this one ran follows htmlmin and then replacing.

[bep]

This issue has been automatically marked as stale because it has not been commented on for at least four months.

The resources of the Hugo team are limited, and so we are asking for your help.

If this is a bug and you can still reproduce this error on the master branch, please reply with all of the information you have about it in order to keep the issue open.

If this is a feature request, and you feel that it is still valuable, please open a proposal at https://discuss.gohugo.io/.

This issue will automatically be closed in four months if no further activity occurs. Thank you for all your contributions.

[bep]

Note/Update: This issue is marked as stale, and I may have said something earlier about "opening a thread on the discussion forum". Please don't.

If this is a bug and you can still reproduce this error on the latest release or the master branch, please reply with all of the information you have about it in order to keep the issue open.

If this is a feature request, and you feel that it is still relevant and valuable, please tell us why.

[bashlund]

Thanks for Hugo,

I'm still a Hugo n00b, but I can confirm that I have this problem, about the p-wrapping.

Hugo version: Hugo Static Site Generator v0.20-DEV linux/amd64 BuildDate: 2017-03-13T09:34:48+01:00
Installed using: go get -v github.com/spf13/hugo

shortcodes/aaa.html:

<div class="aaa">
    This is the aaa shortcode
</div>

shortcodes/aab.html:

<div class="aab">
    This is the aab shortcode.
    {{ .Inner }}
</div>

page.md variant 1:

+++
+++

{{% aaa %}}
{{% aab %}}
My Markdown content goes here
{{% /aab %}}

Output 1:

<p><div class="aaa">
    This is the aaa shortcode
</div>

<div class="aab">
    This is the aab shortcode.
    <p>My Markdown content goes here</p>

</div>
</p>

page.md variant 2:

+++
+++

{{% aaa %}}

{{% aab %}}
My Markdown content goes here
{{% /aab %}}

Output 2:

<div class="aaa">
    This is the aaa shortcode
</div>


<p><div class="aab">
    This is the aab shortcode.
    <p>My Markdown content goes here</p>

</div>
</p>

As you can see by inserting newlines I can move the P wrapping around, but I can't get rid of it in a sane manner.

Again, thanks for working on Hugo, it's awesome.

[Jos512]

I can also confirm that this issue still happens on Hugo 0.19. Like Thomas mentioned above, the issue in my case also happens when 'irregular' text is used inside a shortcode. With that, I mean that this text without line endings gives no problem:

{{% note %}}Example text.{{% /note %}}

Renders as:

<div class="note">
    <p>Example text.</p>
</div>

Going beyond regular text in the shortcode introduces the additional <p> tags.

{{% note %}}Example text.
And another line.{{% /note %}}

Renders the invalid HTML (according to the w3.org validator) of:

<div class="note">
    <p><p>Example text.
And another line.</p>
</p>
</div>

Likewise, the shortcode use like shown below also generates the invalid <p> tags:

{{% note %}}Example text.
![Example image](/assets/example.png){{% /note %}}

<div class="note">
    <p><p>Example text.
<img src="http://localhost:1313/assets/example.png" alt="Example image" /></p>
</p>
</div>

From my understanding, the <p> tag issue seems to happen whenever there's a new line (Enter) used in the shortcode, because I have no trouble with <p> tags when I include a long paragraph of text inside the shortcode.

---

Hugo Static Site Generator v0.19 windows/amd64 BuildDate: 2017-02-27T12:36:30+01:00
GOOS="windows"
GOARCH="amd64"
GOVERSION="go1.8"

[rdwatters]

Hugo seems to see the line breaks as a trigger to create new paragraphs. If I uglify my shortcode and compress it down to the following...

That’s Markdown, not Hugo, no? If you want to use inline shortcodes, I would recommend using the <> rather than the {{}}, but this is starting to get forum-ish in nature...

[ChrisTucker2000]

Maybe it's the Markdown engine. When I say "Hugo", I really just mean "Hugo and all of its internals and dependencies".

I'm a newcomer to this issue and haven't been following it until now, but I did search around before and I came across old issues where a blocks/inline option was being discussed for the Markdown processor, and apparently in the end it was set up so that Hugo removes generated paragraphs from the Markdown output if the Markdown is one line. Perhaps I'm mistaken, but that's what I was referring to when I said "Hugo seems to see the line breaks as a trigger..." and "Hugo needs to ignore whitespace-only blocks when deciding...". I might be misunderstanding things though.

If by <> you mean {{< tags instead of {{%, that resolves the issue but then introduces a new issue in that Markdown no longer functions in the shortcode. Even inline shortcodes can need Markdown support.

[rdwatters]

I see @ChrisTucker2000 @Jos512 . It was only a question since I haven't been involved in Hugo much for the past half year 😄

If by <> you mean {{< tags instead of {{%, that resolves the issue but then introduces a new issue in that Markdown no longer functions in the shortcode. Even inline shortcodes can need Markdown support.

Maybe something like this for a param in the {{<>}} shortcode: .Get 1 | markdownify (positional) or .Get "mynamedparam" | markdownify (named). Sorry I can't be of more help.

[tdryer]

Here's another simple way to reproduce this issue (using hugo v0.42.2):

layouts/shortcodes/bug.html:

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

content/posts/bug.md:

{{< bug >}}

{{< bug >}}

Resulting html:

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx


<p>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
</p>

In this case, any of the following prevents the <p> tag from being inserted:

• Removing a single character from the shortcode, reducing it from 81 characters to 80 characters
• Using the shortcode in the post any number of times other than twice
• Wrapping each shortcode in the post with a <div>

[danbruen]

This bug is still present in 0.46 and none of the above workarounds seem to work for me. I am using sections in my shortcodes and placing them in p makes the markup invalid - which definitely is a showstopper. Does anyone have a pointer to the regex that is supposed to insert these p's?

[Jehu]

Complex designs are always messed up because of this.
I have to leave HUGO in favour of another static site builder.

[danbruen]

What a shame... but still, ancient template toolkit (TT2) never had problems like this. Might as well migrate back again.

[danbruen]

FYI: mmark shows the same behaviour. If it really is a bug in blackfriday and not hugo itself, it has been in there for a long time, as mmark started off as a fork of blackfriday and has not seen much development in the last couple of years.

[danbruen]

And, for the record: Same problem with using pandoc instead of blackfriday. At this point I am pretty sure the problem has its roots in hugo itself.

[geeknoid]

I was happy to see this bug fixed in 0.47. This corrected several cases of <p> elements being incorrectly generated in our web site. Sadly, after switching to 0.47, this problem is still occurring in one specific case.

To repro, grab https://github.com/istio.github.io at commit be7c374f297a712fb6e5cd1be98b656a3b52732f and build the site with Hugo 0.47. This is where the error occurs:

• ./public/docs/examples/multicluster/iks-icp/index.html
  • 7563:6: ERROR: Unexpected end tag : p (line 7563)

[geeknoid]

The problem with our content is that there was an extra period after a closing shortcode:

{{< text >}}
some preformatted text
{{< /text >}}.

where text is a shortcode that produces a pre/code block. The presence of this period caused a p block to be wrapped around the pre/code sequence:

<p>
<pre><code>some preformatted text
</code</pre>
</p>

which is invalid HTML. Removing the period in the markdown fixed the problem.

[TakahikoKawasaki]

In some cases reported above and in my own case, Hugo wraps the second shortcode with p. For example, if there is a markdown like the following,

{{< first-shortcode />}}

{{< second-shortcode />}}

{{< third-shortcode />}}

the markdown will be converted to the following:

The output from first-shortcode

<p>The output from second-shortcode</p>

The output from third-shortcode

A simple workaround for this case is (1) define an empty shortcode like this:

<!--
  This shortcode is used as a workaround for a Hugo's bug.

  [Issue #1642] Shortcode output wrapped in <p> tags under certain conditions
    https://github.com/gohugoio/hugo/issues/1642
-->{{ with .Inner }}{{ end -}}

and (2) put the shortcode as the second shortcode.

{{< first-shortcode />}}

{{< empty "Workaround for a Hugo's bug. Don't remove this." />}}

{{< second-shortcode />}}

{{< third-shortcode />}}

[clutterstack]

Following the trail of issues and discussions of shortcodes and <p> tags, I wind up here. I am not certain that this is the same issue, but I get an empty paragraph generated after a shortcode (such as highlight), but only with certain themes. I made a simple test site and the issue does not come up if I use the "simple-hugo-theme" theme. It looks like a problem somewhere in the templates of my custom theme, and I have not traced the root of it. Just a heads-up to anyone searching for this issue like me to check the theme.

[rdwatters]

@clutterstack. With closed issues, you're likely to get very little (if any) feedback. I would recommend taking your issue to the forums at https://discourse.gohugo.io and get feedback directly from other Hugo users.

[christophlieck]

@clutterstack pls. see this discussion: https://discourse.gohugo.io/t/solved-empty-p-before-and-after-plain-html-in-markdown-files/5559

You must handle your shortcodes like block elements.

[clutterstack]

@rdwatters @christophlieck Thanks for your comments -- in fact, I hadn't expected any feedback! I'm sure I've done something simple wrong in my own theme templates.

I commented because my comment would been helpful to me, had I seen one like it in my prior searches. It took me some time to discover that my problem was not with my shortcode or the way I was using it in markdown files.

[martinnaughton]

I just came across this when looking for a solution to remove

elements from around the {{ .Content }} when it is generated. it seem hugo or blackfriday assumes you need these. I am using hugo to generate a site.

I have BODY element wrapped in my markdown file at the start and finish in each md file. not really using hugo for mark down but for keeping html code in every file so it will generate a site with header and footers in every file.

To remove those annoying PARAGRAPH elements for a generated site i find a sed command on linux in my hugo project. This will remove the paragraph elements. I have to do this every time i generate the site. Here are the commands I use.

find public -name "*.html" -exec sed -i 's/<p><body>/<body>/' {} \;
find public -name "*.html" -exec sed -i 's/<\/body><\/p>/<\/body>/' {} \;

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.