Roundup Tracker - Issues

Issue 2550856

classification
Title: Allow use of reStructured text (or maybe markdown or Creole) in messages
Type: rfe Severity: normal
Components: Web interface, User Interface Versions:
process
Status: new Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ThomasAH, ber, mschieder, ncoghlan, rouilj
Priority: normal Keywords: Effort-Medium, GSOC

Created on 2014-10-18 02:22 by rouilj, last changed 2019-01-17 14:39 by mschieder.

Files
File name Uploaded Description Edit Remove
local_markdown.py mschieder, 2019-01-10 18:45
Messages
msg5151 Author: [hidden] (rouilj) Date: 2014-10-18 02:22
It would  be nice to allow use of some markup language when creating
messages.

This allows people to set off blocks of text, add headers etc. to make
it easier to scan the message body.

Since messages have mime types, we can save the file as mime type:

text/x-rst

text/x-markdown; charset=UTF-8

etc.

Also this would be helpful in better representing html email.

See issue2550799.

Also this would be helpful with implementing:

  http://www.roundup-tracker.org/cgi-bin/moin.cgi/RoundupAsWiki
msg5155 Author: [hidden] (ber) Date: 2014-10-21 07:05
Hi John,
reading your last message on the html issue, I was thinking on similiar 
lines, I believe. My faviorite markup language for this endeavour is creole.

Creole 1 is standardized (which markdown, Mediawiki and Moinoin is not)
and minimal (see 
https://fsfe.org/activities/os/minimalisticstandards.en.html)
There are python parsers.


https://en.wikipedia.org/wiki/Creole_(markup)
msg5966 Author: [hidden] (ncoghlan) Date: 2017-04-26 05:31
With CPython switching to GitHub for repository management and code
review, while retaining Roundup for issue tracking, the lack of markup
support on the issue tracker is become a more noticeable limitation.

For another Python based project (https://beaker-project.org) we were
entirely happy with https://pypi.org/project/Markdown/ as an option for
adding basic markup support that enabled bulleted lists, italics, bold
text, monospace text, and pre-formatted 

There are also a couple of libraries that add support for GitHub's
markdown enhancements (like fenced code blocks that don't require
indentation changes):

* https://pypi.python.org/pypi/gfm/ (preprocessor based approach)
* https://pypi.python.org/pypi/py-gfm (markdown extension based approach)

(Note: I also filed an wishlist item for this directly against the
meta-tracker for the PSF's downstream Roundup instance:
psf.upfronthosting.co.za/roundup/meta/issue626 )
msg5967 Author: [hidden] (ber) Date: 2017-04-26 07:01
@ncoghlan
Thanks for request and the pointer to
  http://psf.upfronthosting.co.za/roundup/meta/issue626

I'm also still interested in this topic.
We should add a possibility to configure markup display.

First technical design question: how do we indicate markup?
My ideas:
 # Let us come up with something that Markdown or MoinMo.in do for
 inline markup see https://moinmo.in/ParserMarket/burndown for a
moinmo.in example

{{{
!#creole

}}}

 # go by Content-Type (for files), maybe additional detection my file
suffix.
 Research point: What are the best content types for 
  markup files like creole, markdown or reStructuredtext
msg5968 Author: [hidden] (ncoghlan) Date: 2017-04-26 12:18
For beaker-project.org, the approach we went with was to just render all
fields where markdown was permitted as markdown, and only emit plaintext
if the markdown rendering failed for some reason.
msg6068 Author: [hidden] (ber) Date: 2018-03-14 15:08
= Pondering ways to implement this:

== directly in cgi
https://sourceforge.net/p/roundup/code/ci/default/tree/roundup/cgi/templating.py#1460

  def hyperlinked
is used by default in the classic template to render the contents
of a message.

It would need a configuration option or another function instead of
hyperlinked.

== as extension
registering as a utility function just like
 
https://sourceforge.net/p/roundup/code/ci/default/tree/share/roundup/templates/responsive/extensions/local_replace.py
and then changeing the template like
https://sourceforge.net/p/roundup/code/ci/default/tree/share/roundup/templates/devel/html/task.item.html#228
in all relevant places.

== candidate libraries
There is https://pypi.python.org/pypi/CommonMark/0.7.5

Otherwise github-style makes sense which I guess is done best by using
https://pypi.org/project/Markdown/ and the guide to configure the extensions
https://facelessuser.github.io/pymdown-extensions/extensions/github/
msg6073 Author: [hidden] (ThomasAH) Date: 2018-04-23 07:33
I have recently started using Markdown in a project using a Mantis
tracker, and due to the syntax for quotes text in lines starting with
">" I would strongly recommend using this instead of reStructured text
or Creole.

One thing I missed in the Mantis implementation was a preview mode to
see how my entry will be rendered before submitting it. This is even
more important here, because Roundup does not (yet) allow editing messages.
msg6316 Author: [hidden] (mschieder) Date: 2019-01-10 18:45
I wrote it as an extension (local_markdown).

For the translation from markdown to HTML I use the commonmark library.

A way to include 'local_markdown' in Roundup is to call it from
'local_replace'.
The call must come after the replacement of 'local_replace' because
'local_markdown' could disturb the regular expressions.
It should also be noted that the strings to be replaced by
'local_replace' can now start and end with '*' or '_' to highlight them
with Markdown.


diff -r 1025affaa63a local_replace.py
--- a/local_replace.py  Thu Jan 10 17:01:30 2019 +0100
+++ b/local_replace.py  Thu Jan 10 17:02:30 2019 +0100
@@ -1,4 +1,5 @@
 import re
+import local_markdown
 
 hg_url_base = r'http://sourceforge.net/p/roundup/code/ci/'
 
@@ -35,6 +36,8 @@
 def local_replace(message):
     for cre, replacement in substitutions:
         message = cre.sub(replacement, message)
+
+    message = local_markdown.local_markdown(message)
     return message
 
 def init(instance):


The final implementation should check if 'local_markdown' can be
imported or not to start Roundup without 'local_markdown'.

The advantage of this is that the templates do not have to be edited.
The disadvantage is that 'local_replace' must be edited and it only
works with 'local_replace'.


The other possibility is to call 'local_markdown' in the templates, like
'local_replace').
But for this all templates have to be edited and I haven't found a good
way to implement the call there yet.


Whether 'local_markdown' is technically safe (executable code) still has
to be checked.
msg6317 Author: [hidden] (rouilj) Date: 2019-01-10 21:15
Hello Magnus:

In message <1547145956.07.0.476198900875.issue2550856@roundup.psfhosted.org>,
Magnus Schieder writes:
>Magnus Schieder added the comment:
>
>I wrote it as an extension (local_markdown).

Nice.

>For the translation from markdown to HTML I use the commonmark library.

Once you have validated the code are you planning on uploading the
local_markdown.py implementation to this ticket?

>A way to include 'local_markdown' in Roundup is to call it from
>'local_replace'.
[...]
>The final implementation should check if 'local_markdown' can be
>imported or not to start Roundup without 'local_markdown'.

Agreed. Also is there a .py version of commonmark that you are using
or are you shelling out to the C implementation?

>The advantage of this is that the templates do not have to be edited.
>The disadvantage is that 'local_replace' must be edited and it only
>works with 'local_replace'.

I don't see that as a disadvantage as local_replace is local to each
roundup tracker and meant to be customized.

>The other possibility is to call 'local_markdown' in the templates, like
>'local_replace').
>But for this all templates have to be edited and I haven't found a good
>way to implement the call there yet.

I agree that way sounds messy and makes updating the templates from
upstream more difficult than it is already.

>Whether 'local_markdown' is technically safe (executable code) still has
>to be checked.

You could add it as an attachment on the ticket. Also might I suggest
adding it to the wiki as a customization example. If you would like
wiki access, I can help you set that up. Also you may want to post
your implementation to the roundup-users mailing list.

Thanks for tackling this issue.
msg6318 Author: [hidden] (mschieder) Date: 2019-01-11 14:44
I have uploaded local_markdown.py and I would like to add it to the wiki
as a customization example. It would be nice if you would help me with
the wiki access.
msg6319 Author: [hidden] (rouilj) Date: 2019-01-12 01:41
Hi Magnus:

I took a look at local_markdown.py, the commonmark spec and the python
implementation. My one concern is that raw html is passed through the
parser.

I wonder if this could be used for some form of attack using html/iframe
or possibly letting a <script> tag skip through. The latter would be
really bad as without the correct restrictions in place it would be
executed.

I think using local_replace to change < into &lt; and > into &gt;
before passing the resulting text to local_markdown will shortcut this
attack, but I am not positive.

However doing so will break markdown email links: <address@example.com>
for example. It may also have some other unwanted side effects.

I didn't see an obvious way to patch/modify the commonmark library
to neuter html and wrap it in <pre> blocks. Alternatively prevent
some tags (e.g. iframe, script) from being recognized as a block.
Maybe this can be done by modifying reHtmlBlockOpen, or monkey patching
a class may work. My python knowledge isn't good enough to say.

Also have you considered implementing a document preview mode to allow
web entry of commonmark to be tested before it is committed?

-- rouilj
msg6320 Author: [hidden] (mschieder) Date: 2019-01-17 14:39
Hi rouilj:

>I wonder if this could be used for some form of attack using html/iframe
>or possibly letting a <script> tag skip through. The latter would be
>really bad as without the correct restrictions in place it would be
>executed.
>
>I think using local_replace to change < into &lt; and > into &gt;
>before passing the resulting text to local_markdown will shortcut this
>attack, but I am not positive.

Roundup passes the HTML to local_replace with the replacements ('< into
&lt;', '> into &gt;', etc).  So html/iframe and <script> tags are no
problems.

The link label ('[a safe side](http://a_bad_side.com)') doesn't work
either, because Roundup replaces the link with '<a
href="http://a_bad_side.com">http://a_bad_side.co</a>' and commonmark
doesn't recognize the link anymore.

The e-mail links work without restrictions.


>Also have you considered implementing a document preview mode to allow
>web entry of commonmark to be tested before it is committed?

Unfortunately I don't know Roundup so well yet and therefore I can't
estimate how complex the implementation is. So I'm not sure I'll
implement it.
History
Date User Action Args
2019-01-17 14:39:49mschiedersetmessages: + msg6320
2019-01-12 01:41:45rouiljsetmessages: + msg6319
2019-01-11 14:44:59mschiedersetmessages: + msg6318
2019-01-10 21:15:10rouiljsetmessages: + msg6317
2019-01-10 18:45:56mschiedersetfiles: + local_markdown.py
nosy: + mschieder
messages: + msg6316
2018-04-23 07:33:30ThomasAHsetnosy: + ThomasAH
messages: + msg6073
2018-03-14 15:08:41bersetmessages: + msg6068
2017-04-26 12:18:50ncoghlansetmessages: + msg5968
2017-04-26 07:01:43bersetmessages: + msg5967
2017-04-26 05:31:11ncoghlansetnosy: + ncoghlan
messages: + msg5966
2014-10-21 07:05:30bersetnosy: + ber
messages: + msg5155
title: Allow use of reStructured text (or maybe markdown) in messages -> Allow use of reStructured text (or maybe markdown or Creole) in messages
2014-10-18 02:23:10rouiljsettitle: Allow use of reStrucutred text (or maybe markdown) in messages -> Allow use of reStructured text (or maybe markdown) in messages
2014-10-18 02:22:51rouiljcreate