It would  be nice to allow use of some markup language when creating
messages.

This allows people to set off blocks of text, add headers etc. to make
it easier to scan the message body.

Since messages have mime types, we can save the file as mime type:

text/x-rst

text/x-markdown; charset=UTF-8

etc.

Also this would be helpful in better representing html email.

See issue2550799.

Also this would be helpful with implementing:

  http://www.roundup-tracker.org/cgi-bin/moin.cgi/RoundupAsWiki

Hi John,
reading your last message on the html issue, I was thinking on similiar 
lines, I believe. My faviorite markup language for this endeavour is creole.

Creole 1 is standardized (which markdown, Mediawiki and Moinoin is not)
and minimal (see 
https://fsfe.org/activities/os/minimalisticstandards.en.html)
There are python parsers.


https://en.wikipedia.org/wiki/Creole_(markup)

With CPython switching to GitHub for repository management and code
review, while retaining Roundup for issue tracking, the lack of markup
support on the issue tracker is become a more noticeable limitation.

For another Python based project (https://beaker-project.org) we were
entirely happy with https://pypi.org/project/Markdown/ as an option for
adding basic markup support that enabled bulleted lists, italics, bold
text, monospace text, and pre-formatted 

There are also a couple of libraries that add support for GitHub's
markdown enhancements (like fenced code blocks that don't require
indentation changes):

* https://pypi.python.org/pypi/gfm/ (preprocessor based approach)
* https://pypi.python.org/pypi/py-gfm (markdown extension based approach)

(Note: I also filed an wishlist item for this directly against the
meta-tracker for the PSF's downstream Roundup instance:
psf.upfronthosting.co.za/roundup/meta/issue626 )

@ncoghlan
Thanks for request and the pointer to
  http://psf.upfronthosting.co.za/roundup/meta/issue626

I'm also still interested in this topic.
We should add a possibility to configure markup display.

First technical design question: how do we indicate markup?
My ideas:
 # Let us come up with something that Markdown or MoinMo.in do for
 inline markup see https://moinmo.in/ParserMarket/burndown for a
moinmo.in example

{{{
!#creole

}}}

 # go by Content-Type (for files), maybe additional detection my file
suffix.
 Research point: What are the best content types for 
  markup files like creole, markdown or reStructuredtext

For beaker-project.org, the approach we went with was to just render all
fields where markdown was permitted as markdown, and only emit plaintext
if the markdown rendering failed for some reason.

= Pondering ways to implement this:

== directly in cgi
https://sourceforge.net/p/roundup/code/ci/default/tree/roundup/cgi/templating.py#1460

  def hyperlinked
is used by default in the classic template to render the contents
of a message.

It would need a configuration option or another function instead of
hyperlinked.

== as extension
registering as a utility function just like
 
https://sourceforge.net/p/roundup/code/ci/default/tree/share/roundup/templates/responsive/extensions/local_replace.py
and then changeing the template like
https://sourceforge.net/p/roundup/code/ci/default/tree/share/roundup/templates/devel/html/task.item.html#228
in all relevant places.

== candidate libraries
There is https://pypi.python.org/pypi/CommonMark/0.7.5

Otherwise github-style makes sense which I guess is done best by using
https://pypi.org/project/Markdown/ and the guide to configure the extensions
https://facelessuser.github.io/pymdown-extensions/extensions/github/

I have recently started using Markdown in a project using a Mantis
tracker, and due to the syntax for quotes text in lines starting with
">" I would strongly recommend using this instead of reStructured text
or Creole.

One thing I missed in the Mantis implementation was a preview mode to
see how my entry will be rendered before submitting it. This is even
more important here, because Roundup does not (yet) allow editing messages.

I wrote it as an extension (local_markdown).

For the translation from markdown to HTML I use the commonmark library.

A way to include 'local_markdown' in Roundup is to call it from
'local_replace'.
The call must come after the replacement of 'local_replace' because
'local_markdown' could disturb the regular expressions.
It should also be noted that the strings to be replaced by
'local_replace' can now start and end with '*' or '_' to highlight them
with Markdown.


diff -r 1025affaa63a local_replace.py
--- a/local_replace.py  Thu Jan 10 17:01:30 2019 +0100
+++ b/local_replace.py  Thu Jan 10 17:02:30 2019 +0100
@@ -1,4 +1,5 @@
 import re
+import local_markdown
 
 hg_url_base = r'http://sourceforge.net/p/roundup/code/ci/'
 
@@ -35,6 +36,8 @@
 def local_replace(message):
     for cre, replacement in substitutions:
         message = cre.sub(replacement, message)
+
+    message = local_markdown.local_markdown(message)
     return message
 
 def init(instance):


The final implementation should check if 'local_markdown' can be
imported or not to start Roundup without 'local_markdown'.

The advantage of this is that the templates do not have to be edited.
The disadvantage is that 'local_replace' must be edited and it only
works with 'local_replace'.


The other possibility is to call 'local_markdown' in the templates, like
'local_replace').
But for this all templates have to be edited and I haven't found a good
way to implement the call there yet.


Whether 'local_markdown' is technically safe (executable code) still has
to be checked.

Hello Magnus:

In message <1547145956.07.0.476198900875.issue2550856@roundup.psfhosted.org>,
Magnus Schieder writes:
>Magnus Schieder added the comment:
>
>I wrote it as an extension (local_markdown).

Nice.

>For the translation from markdown to HTML I use the commonmark library.

Once you have validated the code are you planning on uploading the
local_markdown.py implementation to this ticket?

>A way to include 'local_markdown' in Roundup is to call it from
>'local_replace'.
[...]
>The final implementation should check if 'local_markdown' can be
>imported or not to start Roundup without 'local_markdown'.

Agreed. Also is there a .py version of commonmark that you are using
or are you shelling out to the C implementation?

>The advantage of this is that the templates do not have to be edited.
>The disadvantage is that 'local_replace' must be edited and it only
>works with 'local_replace'.

I don't see that as a disadvantage as local_replace is local to each
roundup tracker and meant to be customized.

>The other possibility is to call 'local_markdown' in the templates, like
>'local_replace').
>But for this all templates have to be edited and I haven't found a good
>way to implement the call there yet.

I agree that way sounds messy and makes updating the templates from
upstream more difficult than it is already.

>Whether 'local_markdown' is technically safe (executable code) still has
>to be checked.

You could add it as an attachment on the ticket. Also might I suggest
adding it to the wiki as a customization example. If you would like
wiki access, I can help you set that up. Also you may want to post
your implementation to the roundup-users mailing list.

Thanks for tackling this issue.

I have uploaded local_markdown.py and I would like to add it to the wiki
as a customization example. It would be nice if you would help me with
the wiki access.

Hi Magnus:

I took a look at local_markdown.py, the commonmark spec and the python
implementation. My one concern is that raw html is passed through the
parser.

I wonder if this could be used for some form of attack using html/iframe
or possibly letting a <script> tag skip through. The latter would be
really bad as without the correct restrictions in place it would be
executed.

I think using local_replace to change < into &lt; and > into &gt;
before passing the resulting text to local_markdown will shortcut this
attack, but I am not positive.

However doing so will break markdown email links: <address@example.com>
for example. It may also have some other unwanted side effects.

I didn't see an obvious way to patch/modify the commonmark library
to neuter html and wrap it in <pre> blocks. Alternatively prevent
some tags (e.g. iframe, script) from being recognized as a block.
Maybe this can be done by modifying reHtmlBlockOpen, or monkey patching
a class may work. My python knowledge isn't good enough to say.

Also have you considered implementing a document preview mode to allow
web entry of commonmark to be tested before it is committed?

-- rouilj

Hi rouilj:

>I wonder if this could be used for some form of attack using html/iframe
>or possibly letting a <script> tag skip through. The latter would be
>really bad as without the correct restrictions in place it would be
>executed.
>
>I think using local_replace to change < into &lt; and > into &gt;
>before passing the resulting text to local_markdown will shortcut this
>attack, but I am not positive.

Roundup passes the HTML to local_replace with the replacements ('< into
&lt;', '> into &gt;', etc).  So html/iframe and <script> tags are no
problems.

The link label ('[a safe side](http://a_bad_side.com)') doesn't work
either, because Roundup replaces the link with '<a
href="http://a_bad_side.com">http://a_bad_side.co</a>' and commonmark
doesn't recognize the link anymore.

The e-mail links work without restrictions.


>Also have you considered implementing a document preview mode to allow
>web entry of commonmark to be tested before it is committed?

Unfortunately I don't know Roundup so well yet and therefore I can't
estimate how complex the implementation is. So I'm not sure I'll
implement it.

ThomasAH:

I noticed you mentioned the lack of preview mode.

Since the newly added rest interface can be extended,
maybe a javascript solution would work.

1) Add a button that takes the comment text (@note) and posts it to

     .../rest/@method/md2html

2) the rest/@method/md2html endpoint is a tracker local
  addition that calls local_replace and markdown to process
  the text like viewing it would do. It should take POSTed text
  and return (possibly json wrapped) html.

3) the javascript takes the returned html and inserts it into a
   <div id="preview"> </div> added to the issue.item.html template.
   Alternately it brings up an overlay on the page with the preview.

Another way to do it (without js) would be for a preview button to trigger a
"md-preview" action on the roundup side that does similar processing
to the @note field. Then returns a page with rendered text and the original
fields.

Thoughts?

-- rouilj

Something I kicked together as a POC:

Add:

    @Routing.route("/@method/format", 'POST')
    def format_text(self,input):
        from docutils.core import publish_parts as ReStructuredText

        if 'note' in input:
            return ReStructuredText(input['note'].value, writer_name="html")['html_body']

to interfaces.py using the rest.txt documentation on how to extend the rest interface.

Then:

curl -s -u demo:demo -H "Content-Type: application/json"\
   -H "Referer: https://.../demo/" \
   -H "X-Requested-With: rest" \
   --data '{ "note": "hello\n+++++\nworld\n`example <https://example.com>`_\nlink." }' \
  https://.../demo/rest/@method/format

returns a json encoded string:

    "<div class=\"document\" id=\"hello\">\n<h1 class=\"title\">hello</h1>\n<p>world\n<a class=\"reference external\" href=\"https://example.com\">example</a>\nlink.</p>\n</div>\n"

So looks like basic concept makes sense at least.

Another option would be to use https://simplemde.com/ for the textarea 
fields.

I have now added a "markdown" method to the StringHTMLProperty.

I believe this can work quite well together with the use of 
https://simplemde.com/ for the edit field.

Hi Christof:

In message <1582227452.47.0.768303835374.issue2550856@roundup.psfhosted.org>,
Christof Meerwald writes:
>I have now added a "markdown" method to the StringHTMLProperty.
>
>I believe this can work quite well together with the use of 
>https://simplemde.com/ for the edit field.

Very nice. Are you considering making an example implementation in a
patch to the jinja2 template?

Also codecov is showing a couple of untested cases in
_hyper_repl_markdown. You can see them in:

https://codecov.io/gh/roundup-tracker/roundup/commit/171c916b91c36bc4ab245480d234f36fd6871fec

Do you have any ideas on trying to test those cases?