Once a roundup datebase grows larger and keywords are used a little 
bit more, the need arises for finding issues that below to two or 
several keywords. The current way in the URL is only match one OR
one of the other keywords. This wish is about the AND search.

An example: I would like to search for "server" and "crypto"
in my tracker.

Am Mittwoch, 5. Mai 2010 15:41:49 schrieb Ralf Schlatterbeck:
> One idea for AND-search would be to have *several* fields (more than 
one
> line) for items where would would want AND search. This is the 
current
> behaviour: when you fill out several fields in a search mask the 
search
> result is ANDed. When you provide more than one value in a field this 
is
> an OR-search.

Good idea.

> We do a similar thing currently for sorting and grouping: You can 
have
> several sort and group targets in a query (Don't know if the standard
> templates in roundup are using this).

Hi!

To allow logical expressions of keywords I've introduced
a reversed polish notation in the keyword indices. Attached
patch keyword-expressions.diff applies against rev. 4543

The idea:

Currently the keywords are represented like "keyword=4,5,42"
with an implicited OR operator of the listed keyword indices.

I defined three special indices (should be okay):

  -2: NOT
  -3: AND
  -4: OR

If the id sequence contains values lower than -1 it is
interpreted as an RPN expression with the special ids
as operators. e.g.

  4,5,-2,-3,42,-4 is read as (4 AND NOT 5) OR 42

If there are no ids below -1 the original behavior (ORed)
is used.

This notation is choosen to reduce the overall impact
to the code and it should be compatible with the old
behavior. Furthermore RPN allows complex expressions
without the need to introduce something like brackets.

The new feature is currently only available by modifying
the URLs. An editor inside Roundup would be nice.
I crafted a Javascript based demo expr-editor.html 
how this could work.
I've to look closer into the UI code of Roundup to
integrate this. If somebody could help me here ...
assistance is wellcome. :-)

Regards,
    Sascha

While I would very much appreciate any work in this direction, I believe
the design should start not in the frontend (and thus be concerned by
syntax), but instead in the middle-layer or backend. (The backend may
actually be easiest, especially if we focus on SQL-enabled backends.)

Using such an approach helps to define a robust and expressive grammar,
and reduces the danger of running into a wall, where it is hard or
impossible to formulate new compound expressions.

Stefan, I believe it is fine to start with the frontend as this is the 
exposed functionality in gernal. The point is that I got sick of 
waiting for the feature and I really need it badly right now, even if 
it is suboptimal. So I assigned one of our software engineers to it to 
give me some solution as a first step.

As a next step we can improve or redesign.

(As far as I have understood the sql vs non-sql debate is not yet 
conclude on the developers list.)

On 10/15/2010 12:04 PM, Bernhard Reiter wrote:
>
> Bernhard Reiter<bernhard@intevation.de>  added the comment:
>
> Stefan, I believe it is fine to start with the frontend as this is the
> exposed functionality in gernal.

Having just looked at the provided patch I realize that the change 
actually *is* to all layers (even though only a single backend is 
supported right now), so my earlier comment was mistaken.

> The point is that I got sick of
> waiting for the feature and I really need it badly right now, even if
> it is suboptimal. So I assigned one of our software engineers to it to
> give me some solution as a first step.
>
> As a next step we can improve or redesign.

OK, I agree. One additional type of expressions I would like to see is a 
compound filter involving multiple properties, such as "foo == 1 OR bar 
== 2". The current suggestion only allows expressions on a single 
property. But I agree, this is a very useful first step.

> (As far as I have understood the sql vs non-sql debate is not yet
> conclude on the developers list.)

True.

Thanks,
		Stefan

Hi,

here is a patch keyword-expressions-complete.diff 
against rev. 4543 with a complete integration into 
the search dialog of Roundup.

It's my first Roundup hack so it's maybe not perfect 
and needs some additional work but it seems to be usable.

Tested against Chromium 6.0.472.63, Opera 10.62, 
Firefox 3.6.10; No tests against members of the 
beloved IE family, yet. Only tested against the default 
backend. I will test it against IE and alternative
backends (SQLite and PostgreSQL) when i am back in the
Office on Monday.

Feel free to test and contribute.

Regard,
  Sascha

I've reworked the patch to be compatible with all rdbms now
(needs more testing).
keyword-expr-alldbm.diff is current now.
Templates for the non classic interfaces are still missing.

Argh! Forgot to add the new classic templates to my last patch.
IE 8 works fine, BTW.

keyword-expr-alldbm2.diff is current now.

@Stefan:
Thanks for your feedback.

I believe that going for a multi-property expression would be a
more radical change for the current url syntax, which actually is quite 
nice. So it is way beyond this issue, I guess it best should be brought 
up on the development list.

Sascha now wrote that he supports the other backends now as well.
So I am still faced with the question: Shall I committ it to trunk?

The patch introduces a couple of significant changes and additions. I
suggest a summary (discussing use-cases, syntax, as well as
implementation details) to be sent to the devel list, to gather feedback
from a wider audience.

Some questions I'd like to see discussed:

* The new expression editor uses javascript. Is there a fallback
mechanism for a js-free environment ?

* How are search expressions be spelled out in other frontends
(roundup-admin, xmlrpc) ? Assuming a straight-forward use of RPN, have
you thought of ways to improve this towards a more natural syntax ?


Personally, I quite like the patch, and am looking forward to seeing it
in trunk.

Thanks for working on this !

PostgreSQL backend works fine, too. :-)

Am Montag, 18. Oktober 2010 14:47:06 schrieb Stefan Seefeld:
> * The new expression editor uses javascript. Is there a fallback
> mechanism for a js-free environment ?

Not yet. It is not trivial. Would be another step.

> * How are search expressions be spelled out in other frontends
> (roundup-admin, xmlrpc) ? Assuming a straight-forward use of RPN, have
> you thought of ways to improve this towards a more natural syntax ?

Not yet researched.

* Bernhard Reiter <issues@roundup-tracker.org> [20101019 11:14]:
> Am Montag, 18. Oktober 2010 14:47:06 schrieb Stefan Seefeld:
> > * The new expression editor uses javascript. Is there a fallback
> > mechanism for a js-free environment ?
> 
> Not yet. It is not trivial. Would be another step.

What's already possible is adding pre-defined searches to the
dropdown and in the status_notresolved value in page.html.

> The patch introduces a couple of significant changes and additions. 
> I suggest a summary (discussing use-cases, syntax, as well as
> implementation details) to be sent to the devel list, to gather
> feedback from a wider audience.

I want to check one last thing before (expression evaluation
directly in SQL). Then I'll write a post to the devel list.

Okay, here is keyword-expr-alldbm3.diff (against rev. 4546)

This one evaluates the expression in database space
if the underlaying engine has decent support for subselects.
If this is not the case it falls back to the older in
program space solution. Tested with SQLite3 and PostgreSQL.

Committed after discussion on roundup-devel@.

rev4575

On Mon, Nov 08, 2010 at 04:26:19PM +0000, Bernhard Reiter wrote:
> 
> Committed after discussion on roundup-devel@.
> 
> rev4575

Is it possible that your commit misses a file?
Im now getting the traceback below and "svn blame"
of cgi/templating.py tells me

  4575        ber from KeywordsExpr import render_keywords_expression_editor

Traceback (most recent call last):
  File "/usr/local/bin/roundup-mailgw", line 6, in <module>
    run()
  File "/usr/local/lib/python2.5/site-packages/roundup/scripts/roundup_mailgw.py", line 212, in run
    sys.exit(main(sys.argv))
  File "/usr/local/lib/python2.5/site-packages/roundup/scripts/roundup_mailgw.py", line 143, in main
    import roundup.instance
  File "/usr/local/lib/python2.5/site-packages/roundup/instance.py", line 35, in <module>
    from roundup.cgi import client, templating
  File "/usr/local/lib/python2.5/site-packages/roundup/cgi/client.py", line 10, in <module>
    from roundup.cgi import templating, cgitb, TranslationService
  File "/usr/local/lib/python2.5/site-packages/roundup/cgi/templating.py", line 30, in <module>
    from KeywordsExpr import render_keywords_expression_editor
ImportError: No module named KeywordsExpr

Thanks, Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  www:   http://www.runtux.com
Reichergasse 131                        email: office@runtux.com
A-3411 Weidling       
osAlliance member                       email: rsc@osalliance.com

@schlatterbeck: Thanks for the catch!

Missing two files commited with r4576.

Thanks, works for me now (ran all roundup test + some of my own from a
tracker), this doesn't mean I've tested the new features, though.

BTW: I think the new features deserve some entry in doc/upgrading.txt
for existing tracker installations...

Seems to work (tested in classic with local modifications).
One thing I noticed is that the (expr) link on the search page vanishes
once an expression has been entered.

I have split out Issue2550698 (Document the logical expression search a 
bit) which does not share the same priority as the implemented feature 
which works right away on new trackers.

Allow me to re-open this issue, as I have discovered a couple of issues
that I think ought to be addressed before this can be considered completed.

To start with the high-order bit: I think in its current form (even
after addressing the immediate issues) this harness isn't quite ready to
be part of Roundup's core. It is too much tied to tracker-specific
aspects, such as a 'keyword' entity in the schema, or a particular HTML
layout.

I thus suggest this code to be moved into tracker-specific extensions.
(This won't affect the functionality at all, just the code layout.)

Now some specific problems:

1) After (re-)discovering what was already reported in msg4204 (the
disappearance of the "(edit)" link once the keyword value was set), I
attempted to validate the generated HTML, and noticed that it was
invalid. This may be the cause for the reported behavior.

2) The KeywordEditor module attempts to inject HTML by means of the
"innerHTML" DOM attribute. However, that only works with HTML 4, but not
XHTML. Since otherwise Roundup isn't tied to HTML 4 (in fact, it has a
config value to set whether XHTML should be generated), I think this
needs to be fixed, too.

Stefan,
thanks for the feedback, this is very valuable! :)

Moving the code to a different place would be okay to me of course.
As for being tied to "keyword" and certain HTML versions,
I suggest that we open up new issues. Otherwise this one will
get to cluttered. Would this be okay for you?

Rationale:
This is a large feature to be done in its full entity
and to me it turned out that we need to tackle it in steps
which are itself usable. I believe we have completed
a good and important step in this direction.
The restrictions towards "keyword" and html versions are there,
but were done deliberately to reach a useful step early
and not get stuck.
Maybe we could rename this issue to be more specific if this helps
to make it one step. 

What do you think?
Bernhard

On 2011-05-02 05:19, Bernhard Reiter wrote:

> Moving the code to a different place would be okay to me of course.
> As for being tied to "keyword" and certain HTML versions,
> I suggest that we open up new issues. Otherwise this one will
> get to cluttered. Would this be okay for you?

Yes, absolutely. I didn't mean to make this a never-ending story, just 
didn't want to loose the context.

Thanks,
		Stefan

I have now split out:
  Issue2550708 (Binary filter operators specific to "keyword" should be 
more generic)
  Issue2550709 (Classic template binary search "(edit)" or "(expr)" 
link vanishes once an expression was entered)
  Issue2550710 (Classic template binary search: Too HTML4 specific)

I tried this feature on the Roundup instance of bugs.python.org and it
didn't work out of the box.  The problem seemed to be caused by some
conflict between the words "keyword" and "keywords".
In the search page the <select> uses "keywords", but AFAIU the script
expects "keyword" in order to work, both in the "(expr)" link of the
popup and in the js code that tries to access the form using
getElementById('keyword').
I fixed it by replacing the occurrences of 'keyword' to 'keywords' where
necessary [0].  Note that in issue.search.html I just added back the 's'
that I had removed in the previous commit.  That made the popup appear
correctly, but then the search wasn't working anymore because Roundup
was expecting "keywords" (with the 's') on the other side.
My solution is probably not the most elegant (I basically hardcoded
'keyword' and 'keywords' in the right places), but it fixed the problem
for me.

Another issue I noticed is that all the keywords (included retired ones)
are listed in the popup.

[0]: http://svn.python.org/view?view=revision&revision=88877

As the latest comment illustrates that this is in fact not working
correctly, I'm reopening the issue to increase the likeliness of it not
getting ignored.

I addressed Ezio's issue with the popup editor including retired values.

See commit: 0142b4fb5a2d

It was a simple commit, but while committing the following question
came up:

   Supposed a keyword is retired and I want to search for an issue
   with that retired keyword?

Do we have a best policy document that says to remove retired keywords
from all places it could possibly be used? It could be argued that the
simple search dropdown is wrong and should allow selecting retired values.

There are no comments in hg or the code to tell me how the developer
interpreted the meaning of "Retired" and if the decision to leave
retired values in the popup was intentional.

-- rouilj

* John Rouillard <issues@roundup-tracker.org> [20160709 01:39]:
>    Supposed a keyword is retired and I want to search for an issue
>    with that retired keyword?
> 
> Do we have a best policy document that says to remove retired keywords
> from all places it could possibly be used? It could be argued that the
> simple search dropdown is wrong and should allow selecting retired values.
> 
> There are no comments in hg or the code to tell me how the developer
> interpreted the meaning of "Retired" and if the decision to leave
> retired values in the popup was intentional.

I understand "retired" as:
It is deleted, but since we do not want to really delete anything in
the database, it is just marked as deleted and can be undeleted
(unretired) later.

So retired keywords/users/messages/issues should not appear
anywhere. They currently do, which sometimes causes problems, e.g.
when editing an issue.

On Sat, Jul 09, 2016 at 01:15:30PM +0000, Thomas Arendsen Hein wrote:
> 
> * John Rouillard <issues@roundup-tracker.org> [20160709 01:39]:
> >    Supposed a keyword is retired and I want to search for an issue
> >    with that retired keyword?
> > 
> > Do we have a best policy document that says to remove retired keywords
> > from all places it could possibly be used? It could be argued that the
> > simple search dropdown is wrong and should allow selecting retired values.
> > 
> > There are no comments in hg or the code to tell me how the developer
> > interpreted the meaning of "Retired" and if the decision to leave
> > retired values in the popup was intentional.
> 
> I understand "retired" as:
> It is deleted, but since we do not want to really delete anything in
> the database, it is just marked as deleted and can be undeleted
> (unretired) later.
> 
> So retired keywords/users/messages/issues should not appear
> anywhere. They currently do, which sometimes causes problems, e.g.
> when editing an issue.

We're leaving retired keywords etc. in old issues. It's usually not a
good idea to modify history and it avoids practical issues with
referential integrity. So retired items in the database may be linked
from other items.

Concerning the search: When searching for no-longer-valid items is
required, I've implemented an 'is_valid' flag in some of my trackers
(not for keyword but for other customized properties). If an item is not
marked as valid, it can still be searched for but will not be available
in the menus for issue-editing. (So it will be in seen in index
templates but not in item templates).

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  www:   http://www.runtux.com
Reichergasse 131, A-3411 Weidling       email: office@runtux.com

Here is another place where it's not quite working right.

When editing a query from the query edit page, any conditions
created by the expression editor are lost.

E.G. create a keyword search "keyword1 and not keyword2".
Save the search. You can execute the search and it will work fine.
Now edit the search and set the owner to semothing.

Note that the edit screen doesn't display

  "keyword1 and not keyword2"

anymore. If you save the query, you loose the keyword expression
even though it was not edited.

I am not sure if this is an issue with the underlying mechanism
we use to store the query or just a problem with the form that is created.

-- rouilj

I committed: changeset:   6416:99d344aa825d
Now you can invoke the keyword editor multiple times.
The link to invoke the editor doesn't disappear.

If there is an existing expression, it will not be inherited
back into the editor. The user will have to recreate a new
keyword expression from scratch.