Roundup Tracker - Issues

Issue 2551094

classification
Title: Markdown rendering should break on new line
Type: behavior Severity: normal
Components: Web interface Versions: 2.0.0
process
Status: fixed Resolution: fixed
Dependencies: Superseder:
Assigned To: rouilj Nosy List: ced, cmeerw, rouilj
Priority: normal Keywords: jinja2, patch

Created on 2020-10-22 12:21 by ced, last changed 2020-10-25 20:15 by rouilj.

Files
File name Uploaded Description Edit Remove
markdown-break-on-newline.patch ced, 2020-10-22 12:21
roundup-markdown-break-on-newline.patch ced, 2020-10-24 08:49 Add markdown configuration
roundup-markdown-break-on-newline.patch ced, 2020-10-24 15:37 Add markdown configuration (with template)
roundup-markdown-break-on-newline.patch ced, 2020-10-25 19:14 Add markdown configuration (with template and cleaned)
Messages
msg6970 Author: [hidden] (ced) Date: 2020-10-22 12:21
simplemde which is used by jinja2 template to enter markdown message, render the markdown by adding break on new line. But the markdown rendering on the server side does not. So the user does not get what he saw when editing the message.
I think we should render markdown on server side also with break on new line.
Here is a patch that implements for markdown2 and markdown (I could not find an option for mistune).
msg6973 Author: [hidden] (rouilj) Date: 2020-10-22 15:47
Hi Cédric:

I am including Christof who did the markdown work.

This is a mismatch between simplemde's formatting and the back end.
I am not sure if simplemde needs a fix or the roundup backend needs
a fix, or if some user education needs to happen:

If some other mde editor is used on the front end, then we can
have the same issue but in the opposite direction.

Maybe this code needs to be conditional with a config option?

I have developed a test for this patch (that skips testing with
mistune), so we can include it if the discussion points that way.

Also there has been discussion of this as a misfeature:

  https://github.com/sparksuite/simplemde-markdown-editor/issues/71

and markdown's break format (per 
https://daringfireball.net/projects/markdown/syntax#block) is:

    When you do want to insert a <br /> break tag using Markdown,
    you end a line with two or more spaces, then type return.

so it looks like simplemde should have a "hard return" option that
inserts the trailing spaces.
msg6974 Author: [hidden] (rouilj) Date: 2020-10-22 15:59
Also: https://github.com/sparksuite/simplemde-markdown-editor/issues/430 
for change to simplemde to disable github markdown.

This may be needed unless we are using a markdown in the backend that
can have the github markdown variant enabled.

Can simplemde do a rest query to get formatted output? If so then
msg6814 and msg6815 may provide a solution.

Also: https://www.npmjs.com/package/@amrn/simplemde mentions:

  renderingConfig: Adjust settings for parsing the Markdown during 
previewing (not editing).
    singleLineBreaks: If set to false, disable parsing GFM single line 
breaks. Defaults to true.

maybe this can be added to the client side if it does the right thing?
msg6976 Author: [hidden] (ced) Date: 2020-10-22 16:34
For me it was simpler to change on the server side the rendering because we already got messages created with simplemde syntax.
But if we change simplemde then we will have some messages with bad formatting.
msg6979 Author: [hidden] (cmeerw) Date: 2020-10-22 18:30
We should keep in mind that there are other WYSIWYG editors and simplemde 
is essentially just an example (although it's the only one shipped as 
part of a roundup template). The templates (and the WYSIWYG editor) are 
user replaceable, but the backend not so much, so we should try to keep 
the backend stable and instead configure the WYSIWYG editor to match the 
backend.

So I would prefer to add "singleLineBreaks: false" to the simplemde 
rendering config.
msg6982 Author: [hidden] (ced) Date: 2020-10-22 19:38
The singleLibeBreaks: false on simplemde does not provide a good
feadback when editing. I think this will confuse end users.
I think most of the markdown editor provides such feature.

Also as the default editor of roundup was historically simplemde, I
think it has defined the standard.
msg6983 Author: [hidden] (ced) Date: 2020-10-22 19:38
An alternative solution would be to provide a roundup configuration to define which markdown format to use.
msg6985 Author: [hidden] (rouilj) Date: 2020-10-23 00:55
Hi Christof:

In msg6979 you said:

   So I would prefer to add "singleLineBreaks: false" to the
   simplemde rendering config.

Is that something you can change and test? I don't have much of
a feel for how you incorporated the editor into the template.
msg6986 Author: [hidden] (rouilj) Date: 2020-10-23 01:16
Hi Cedric:

in msg6983 you said:

> An alternative solution would be to provide a roundup
> configuration to define which markdown format to use.

Are you thinking of a config option:

  MarkdownSettings

where we can specify a set of key:value or array of string values
to be added to be merged into the extras or extensions dict/array passed 
to the markdown implementation?

Something similar could be done for mistune but it uses actual 
classes/objects in python to augment its plugin array. So interfaces.py 
would probably need to be used to override _import_mistune in some way.
msg6987 Author: [hidden] (ced) Date: 2020-10-23 06:00
On 2020-10-23 01:16, John Rouillard wrote:
> Are you thinking of a config option:
> 
>   MarkdownSettings
> 
> where we can specify a set of key:value or array of string values
> to be added to be merged into the extras or extensions dict/array passed 
> to the markdown implementation?

I was thinking more about boolean configuration under a [markdown]
section which activate or not a subset of common setup between the three
implementations.
msg6988 Author: [hidden] (cmeerw) Date: 2020-10-23 17:37
the simplemde change would be:

diff -r 2f53d41ae71f 
share/roundup/templates/jinja2/html/issue.item.html
--- a/share/roundup/templates/jinja2/html/issue.item.html	Tue Jul 
28 21:41:59 2020 -0400
+++ b/share/roundup/templates/jinja2/html/issue.item.html	Fri Oct 
23 18:21:40 2020 +0100
@@ -32,7 +32,7 @@
       var node = $('#change_note')[0];
       var initSimpleMde = function () {
         node.parentNode.appendChild($('<input/>', { type: 'hidden', 
name: 'msg-1@type', value: 'text/markdown'})[0]);
-        var simplemde = new SimpleMDE({ element: node, status: false, 
styleSelectedText: false });
+        var simplemde = new SimpleMDE({ element: node, status: false, 
styleSelectedText: false, renderingConfig: { singleLineBreaks: false } 
});
         simplemde.render();
       };
       {% if context.id %}
msg6989 Author: [hidden] (rouilj) Date: 2020-10-23 22:48
Applied Christof's patch in rev 6276:177b186dd23a.

Closing.

Cédric you will have to keep applying your patch and revert the
patch to the template in future releases.

Getting support for markdown config options and support for 
"githubMarkdownNewline" or some such option for the markdown backends 
will remove he need for your local patches. (Note Mistune can support 
this via a new plugin to be written by somebody.)

Alternatively you can bulk edit the affected message files replacing
the trailing newline with with two spaces followed by a newline.
So they are using the standard markdown format for embedded linebreak

That's one of the advantages of storing the message files in the 
filesystem. Some roundup-admin fu to select the markdown format
messages and a little sed/perl should get you fixed up if you want
to go this way.

-- rouilj
msg6992 Author: [hidden] (ced) Date: 2020-10-24 08:49
As I prefer to not have to patch roundup on each update and because I think most end user will prefer the break-on-newline behavior. Here is a patch that add the option.
msg6993 Author: [hidden] (rouilj) Date: 2020-10-24 15:10
Hi Cédric:

In message <1603529364.7.0.732210980831.issue2551094@roundup.psfhosted.org>,
=?utf-8?q?C=C3=A9dric_Krier?= writes:
>
>As I prefer to not have to patch roundup on each update and because I
>think most end user will prefer the break-on-newline behavior. Here is
>a patch that adds

Very cool. Nice find on mistune's hard_wrap.

One missing piece is to toggle the setting in issue.index.html
template as well so that all 4 markdown (the 4th is simplemde)
renderers agree.

So:

  renderingConfig: {singleLineBreaks: false}

needs to be

  renderingConfig: {singleLineBreaks: true}

when the option is set to true. Christof, would:

   renderingConfig: {singleLineBreaks: {% config.MARKDOWN_BREAK_ON_NEWLINE %} }

work? I assume I am missing something as we want the strings "false"
and "true" here. I am not sure what value is produced by
config.MARKDOWN_BREAK_ON_NEWLINE maybe 0/1.  (If this technique works
it is nicer than how I have been modifying javascript with TAL.)

Cédric once you have verified a working solution please attach a
patch.

It's funny but your test code and mine use the same trick of counting
the number of '<br' in the returned markup.

Thanks for this work. It looks good so far. I am excited to get it
finished and committed.
msg6994 Author: [hidden] (ced) Date: 2020-10-24 15:37
> It's funny but your test code and mine use the same trick of counting
the number of '<br' in the returned markup.

Well I saw your test when I got issue with the test because mistune generate a <br> and markdow/2 a <br/>.

Here is the version of the patch with jinja2 template updated.
msg6995 Author: [hidden] (cmeerw) Date: 2020-10-24 23:53
> when the option is set to true. Christof, would:
> 
>    renderingConfig: {singleLineBreaks: {% 
config.MARKDOWN_BREAK_ON_NEWLINE %} }
> 
> work? I assume I am missing something as we want the strings "false"
> and "true" here. I am not sure what value is produced by
> config.MARKDOWN_BREAK_ON_NEWLINE maybe 0/1.  (If this technique works
> it is nicer than how I have been modifying javascript with TAL.)

there are a few ways you can do that with jinja2

  {% if config.MARKDOWN_BREAK_ON_NEWLINE %}true{% else %}false{% endif 
%}

is probably the most basic one, but you could also do something like

  {{ config.MARKDOWN_BREAK_ON_NEWLINE and 'true' or 'false' }}

looks like at the moment we get 0/1 for the string value of the option. 
Just wondering... maybe we shouldn't rely on the string value being '0' 
or '1'
msg6997 Author: [hidden] (rouilj) Date: 2020-10-25 17:19
In msg6995 Christof says:

> you could also do something like
>
>  {{ config.MARKDOWN_BREAK_ON_NEWLINE and 'true' or 'false' }}
>
>looks like at the moment we get 0/1 for the string value of the option. 

This is what Cedric used calling the Boolean() javascript constructor
to create a true boolean.

>Just wondering... maybe we shouldn't rely on the string value being
>'0' or '1'

In issue2550905 to discuss testing I mention in msg6744 that we have
no testing for chameleon or jinja2 templating engines and poor
coverage on the TAL engine. Testing for this type of return value
and enforcing it through tests could be a good first test case
for jinja2. Created issue2551095 to test/enforce this.

However even without that test, I am inclined to commit Cedric's
patch including the simplemede template fix.
msg6998 Author: [hidden] (rouilj) Date: 2020-10-25 17:38
Hi Cedric:

In your patch to templating.py you have:

  +            s = u2s(markdown(s2u(s), self._db.config))
           except Exception:  # when markdown formatting fails return 
markup
  +            raise
               return self.plain(escape=0, hyperlink=hyperlink)
           return s

why was raise added? Doesn't that defeat the ability to return plain
text if the markdown formatter raises an error? It certainly breaks
the markdown tests.

Also in test_tempating.py you have:

  + from operator import setitem

why is this import needed? I don't see any call to setitem.

Have a great day.

-- rouilj
msg6999 Author: [hidden] (ced) Date: 2020-10-25 19:14
Those are former debugging stuffs.
Here is the cleaned patch which also use the and/or syntax because I think it is more stable if MARKDOWN_BREAK_ON_NEWLINE become boolean instead of integer.
msg7000 Author: [hidden] (rouilj) Date: 2020-10-25 20:15
Committed with slight doc tweak in configuration.py in
hg 6277:957a0fc20021.
History
Date User Action Args
2020-10-25 20:15:41rouiljsetstatus: open -> fixed
messages: + msg7000
2020-10-25 19:14:34cedsetfiles: + roundup-markdown-break-on-newline.patch
messages: + msg6999
2020-10-25 17:38:06rouiljsetmessages: + msg6998
2020-10-25 17:19:59rouiljsetstatus: fixed -> open
messages: + msg6997
2020-10-24 23:53:01cmeerwsetmessages: + msg6995
2020-10-24 15:37:42cedsetfiles: + roundup-markdown-break-on-newline.patch
messages: + msg6994
2020-10-24 15:10:58rouiljsetmessages: + msg6993
2020-10-24 08:49:24cedsetfiles: + roundup-markdown-break-on-newline.patch
messages: + msg6992
2020-10-23 22:48:32rouiljsetpriority: normal
status: open -> fixed
resolution: fixed
messages: + msg6989
2020-10-23 17:37:18cmeerwsetmessages: + msg6988
2020-10-23 06:00:03cedsetmessages: + msg6987
2020-10-23 01:16:19rouiljsetmessages: + msg6986
2020-10-23 00:55:16rouiljsetmessages: + msg6985
2020-10-22 19:38:46cedsetmessages: + msg6983
2020-10-22 19:38:02cedsetmessages: + msg6982
2020-10-22 18:30:32cmeerwsetkeywords: + jinja2
messages: + msg6979
2020-10-22 16:34:44cedsetmessages: + msg6976
2020-10-22 15:59:49rouiljsetmessages: + msg6974
2020-10-22 15:47:14rouiljsetstatus: new -> open
assignee: rouilj
messages: + msg6973
nosy: + rouilj, cmeerw
2020-10-22 12:21:52cedcreate