Roundup Tracker - Issues

Issue 2550648

classification
Title: Should be possible to search for matching several keywords (keyword1 AND keyword2)
Type: rfe Severity: major
Components: Web interface, User Interface, Infrastructure Versions: devel, 1.4
process
Status: open Resolution: accepted
Dependencies: Superseder:
Assigned To: ber Nosy List: ThomasAH, ber, cslotty, ezio.melotti, pefu, rouilj, schlatterbeck, slteichmann, stefan
Priority: high Keywords: patch

Created on 2010-05-04 21:13 by ber, last changed 2016-07-16 03:03 by rouilj.

Files
File name Uploaded Description Edit Remove
keyword-expressions.diff slteichmann, 2010-10-15 13:23
expr-editor.html slteichmann, 2010-10-15 13:23
keyword-expressions-complete.diff slteichmann, 2010-10-16 11:47
keyword-expr-alldbm.diff slteichmann, 2010-10-17 19:36
keyword-expr-alldbm2.diff slteichmann, 2010-10-18 10:25
keyword-expr-alldbm3.diff slteichmann, 2010-10-19 16:31
Messages
msg4046 Author: [hidden] (ber) Date: 2010-05-04 21:13
Once a roundup datebase grows larger and keywords are used a little 
bit more, the need arises for finding issues that below to two or 
several keywords. The current way in the URL is only match one OR
one of the other keywords. This wish is about the AND search.

An example: I would like to search for "server" and "crypto"
in my tracker.
msg4053 Author: [hidden] (ber) Date: 2010-05-10 07:22
Am Mittwoch, 5. Mai 2010 15:41:49 schrieb Ralf Schlatterbeck:
> One idea for AND-search would be to have *several* fields (more than 
one
> line) for items where would would want AND search. This is the 
current
> behaviour: when you fill out several fields in a search mask the 
search
> result is ANDed. When you provide more than one value in a field this 
is
> an OR-search.

Good idea.

> We do a similar thing currently for sorting and grouping: You can 
have
> several sort and group targets in a query (Don't know if the standard
> templates in roundup are using this).
msg4145 Author: [hidden] (slteichmann) Date: 2010-10-15 13:23
Hi!

To allow logical expressions of keywords I've introduced
a reversed polish notation in the keyword indices. Attached
patch keyword-expressions.diff applies against rev. 4543

The idea:

Currently the keywords are represented like "keyword=4,5,42"
with an implicited OR operator of the listed keyword indices.

I defined three special indices (should be okay):

  -2: NOT
  -3: AND
  -4: OR

If the id sequence contains values lower than -1 it is
interpreted as an RPN expression with the special ids
as operators. e.g.

  4,5,-2,-3,42,-4 is read as (4 AND NOT 5) OR 42

If there are no ids below -1 the original behavior (ORed)
is used.

This notation is choosen to reduce the overall impact
to the code and it should be compatible with the old
behavior. Furthermore RPN allows complex expressions
without the need to introduce something like brackets.

The new feature is currently only available by modifying
the URLs. An editor inside Roundup would be nice.
I crafted a Javascript based demo expr-editor.html 
how this could work.
I've to look closer into the UI code of Roundup to
integrate this. If somebody could help me here ...
assistance is wellcome. :-)

Regards,
    Sascha
msg4148 Author: [hidden] (stefan) Date: 2010-10-15 15:47
While I would very much appreciate any work in this direction, I believe
the design should start not in the frontend (and thus be concerned by
syntax), but instead in the middle-layer or backend. (The backend may
actually be easiest, especially if we focus on SQL-enabled backends.)

Using such an approach helps to define a robust and expressive grammar,
and reduces the danger of running into a wall, where it is hard or
impossible to formulate new compound expressions.
msg4149 Author: [hidden] (ber) Date: 2010-10-15 16:04
Stefan, I believe it is fine to start with the frontend as this is the 
exposed functionality in gernal. The point is that I got sick of 
waiting for the feature and I really need it badly right now, even if 
it is suboptimal. So I assigned one of our software engineers to it to 
give me some solution as a first step.

As a next step we can improve or redesign.

(As far as I have understood the sql vs non-sql debate is not yet 
conclude on the developers list.)
msg4151 Author: [hidden] (stefan) Date: 2010-10-15 16:25
On 10/15/2010 12:04 PM, Bernhard Reiter wrote:
>
> Bernhard Reiter<bernhard@intevation.de>  added the comment:
>
> Stefan, I believe it is fine to start with the frontend as this is the
> exposed functionality in gernal.

Having just looked at the provided patch I realize that the change 
actually *is* to all layers (even though only a single backend is 
supported right now), so my earlier comment was mistaken.

> The point is that I got sick of
> waiting for the feature and I really need it badly right now, even if
> it is suboptimal. So I assigned one of our software engineers to it to
> give me some solution as a first step.
>
> As a next step we can improve or redesign.

OK, I agree. One additional type of expressions I would like to see is a 
compound filter involving multiple properties, such as "foo == 1 OR bar 
== 2". The current suggestion only allows expressions on a single 
property. But I agree, this is a very useful first step.

> (As far as I have understood the sql vs non-sql debate is not yet
> conclude on the developers list.)

True.

Thanks,
		Stefan
msg4152 Author: [hidden] (slteichmann) Date: 2010-10-16 11:47
Hi,

here is a patch keyword-expressions-complete.diff 
against rev. 4543 with a complete integration into 
the search dialog of Roundup.

It's my first Roundup hack so it's maybe not perfect 
and needs some additional work but it seems to be usable.

Tested against Chromium 6.0.472.63, Opera 10.62, 
Firefox 3.6.10; No tests against members of the 
beloved IE family, yet. Only tested against the default 
backend. I will test it against IE and alternative
backends (SQLite and PostgreSQL) when i am back in the
Office on Monday.

Feel free to test and contribute.

Regard,
  Sascha
msg4153 Author: [hidden] (slteichmann) Date: 2010-10-17 19:36
I've reworked the patch to be compatible with all rdbms now
(needs more testing).
keyword-expr-alldbm.diff is current now.
Templates for the non classic interfaces are still missing.
msg4157 Author: [hidden] (slteichmann) Date: 2010-10-18 10:25
Argh! Forgot to add the new classic templates to my last patch.
IE 8 works fine, BTW.

keyword-expr-alldbm2.diff is current now.
msg4158 Author: [hidden] (ber) Date: 2010-10-18 12:36
@Stefan:
Thanks for your feedback.

I believe that going for a multi-property expression would be a
more radical change for the current url syntax, which actually is quite 
nice. So it is way beyond this issue, I guess it best should be brought 
up on the development list.

Sascha now wrote that he supports the other backends now as well.
So I am still faced with the question: Shall I committ it to trunk?
msg4159 Author: [hidden] (stefan) Date: 2010-10-18 12:47
The patch introduces a couple of significant changes and additions. I
suggest a summary (discussing use-cases, syntax, as well as
implementation details) to be sent to the devel list, to gather feedback
from a wider audience.

Some questions I'd like to see discussed:

* The new expression editor uses javascript. Is there a fallback
mechanism for a js-free environment ?

* How are search expressions be spelled out in other frontends
(roundup-admin, xmlrpc) ? Assuming a straight-forward use of RPN, have
you thought of ways to improve this towards a more natural syntax ?


Personally, I quite like the patch, and am looking forward to seeing it
in trunk.

Thanks for working on this !
msg4160 Author: [hidden] (slteichmann) Date: 2010-10-18 13:47
PostgreSQL backend works fine, too. :-)
msg4161 Author: [hidden] (ber) Date: 2010-10-19 09:14
Am Montag, 18. Oktober 2010 14:47:06 schrieb Stefan Seefeld:
> * The new expression editor uses javascript. Is there a fallback
> mechanism for a js-free environment ?

Not yet. It is not trivial. Would be another step.

> * How are search expressions be spelled out in other frontends
> (roundup-admin, xmlrpc) ? Assuming a straight-forward use of RPN, have
> you thought of ways to improve this towards a more natural syntax ?

Not yet researched.
msg4162 Author: [hidden] (ThomasAH) Date: 2010-10-19 09:30
* Bernhard Reiter <issues@roundup-tracker.org> [20101019 11:14]:
> Am Montag, 18. Oktober 2010 14:47:06 schrieb Stefan Seefeld:
> > * The new expression editor uses javascript. Is there a fallback
> > mechanism for a js-free environment ?
> 
> Not yet. It is not trivial. Would be another step.

What's already possible is adding pre-defined searches to the
dropdown and in the status_notresolved value in page.html.
msg4163 Author: [hidden] (slteichmann) Date: 2010-10-19 09:34
> The patch introduces a couple of significant changes and additions. 
> I suggest a summary (discussing use-cases, syntax, as well as
> implementation details) to be sent to the devel list, to gather
> feedback from a wider audience.

I want to check one last thing before (expression evaluation
directly in SQL). Then I'll write a post to the devel list.
msg4167 Author: [hidden] (slteichmann) Date: 2010-10-19 16:31
Okay, here is keyword-expr-alldbm3.diff (against rev. 4546)

This one evaluates the expression in database space
if the underlaying engine has decent support for subselects.
If this is not the case it falls back to the older in
program space solution. Tested with SQLite3 and PostgreSQL.
msg4199 Author: [hidden] (ber) Date: 2010-11-08 16:26
Committed after discussion on roundup-devel@.

rev4575
msg4200 Author: [hidden] (schlatterbeck) Date: 2010-11-09 14:05
On Mon, Nov 08, 2010 at 04:26:19PM +0000, Bernhard Reiter wrote:
> 
> Committed after discussion on roundup-devel@.
> 
> rev4575

Is it possible that your commit misses a file?
Im now getting the traceback below and "svn blame"
of cgi/templating.py tells me

  4575        ber from KeywordsExpr import render_keywords_expression_editor

Traceback (most recent call last):
  File "/usr/local/bin/roundup-mailgw", line 6, in <module>
    run()
  File "/usr/local/lib/python2.5/site-packages/roundup/scripts/roundup_mailgw.py", line 212, in run
    sys.exit(main(sys.argv))
  File "/usr/local/lib/python2.5/site-packages/roundup/scripts/roundup_mailgw.py", line 143, in main
    import roundup.instance
  File "/usr/local/lib/python2.5/site-packages/roundup/instance.py", line 35, in <module>
    from roundup.cgi import client, templating
  File "/usr/local/lib/python2.5/site-packages/roundup/cgi/client.py", line 10, in <module>
    from roundup.cgi import templating, cgitb, TranslationService
  File "/usr/local/lib/python2.5/site-packages/roundup/cgi/templating.py", line 30, in <module>
    from KeywordsExpr import render_keywords_expression_editor
ImportError: No module named KeywordsExpr

Thanks, Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  www:   http://www.runtux.com
Reichergasse 131                        email: office@runtux.com
A-3411 Weidling       
osAlliance member                       email: rsc@osalliance.com
msg4201 Author: [hidden] (ber) Date: 2010-11-09 15:07
@schlatterbeck: Thanks for the catch!

Missing two files commited with r4576.
msg4203 Author: [hidden] (schlatterbeck) Date: 2010-11-11 14:12
Thanks, works for me now (ran all roundup test + some of my own from a
tracker), this doesn't mean I've tested the new features, though.

BTW: I think the new features deserve some entry in doc/upgrading.txt
for existing tracker installations...
msg4204 Author: [hidden] (ThomasAH) Date: 2010-11-11 15:51
Seems to work (tested in classic with local modifications).
One thing I noticed is that the (expr) link on the search page vanishes
once an expression has been entered.
msg4288 Author: [hidden] (ber) Date: 2011-04-15 07:34
I have split out Issue2550698 (Document the logical expression search a 
bit) which does not share the same priority as the implemented feature 
which works right away on new trackers.
msg4302 Author: [hidden] (stefan) Date: 2011-04-21 15:25
Allow me to re-open this issue, as I have discovered a couple of issues
that I think ought to be addressed before this can be considered completed.

To start with the high-order bit: I think in its current form (even
after addressing the immediate issues) this harness isn't quite ready to
be part of Roundup's core. It is too much tied to tracker-specific
aspects, such as a 'keyword' entity in the schema, or a particular HTML
layout.

I thus suggest this code to be moved into tracker-specific extensions.
(This won't affect the functionality at all, just the code layout.)

Now some specific problems:

1) After (re-)discovering what was already reported in msg4204 (the
disappearance of the "(edit)" link once the keyword value was set), I
attempted to validate the generated HTML, and noticed that it was
invalid. This may be the cause for the reported behavior.

2) The KeywordEditor module attempts to inject HTML by means of the
"innerHTML" DOM attribute. However, that only works with HTML 4, but not
XHTML. Since otherwise Roundup isn't tied to HTML 4 (in fact, it has a
config value to set whether XHTML should be generated), I think this
needs to be fixed, too.
msg4304 Author: [hidden] (ber) Date: 2011-05-02 09:19
Stefan,
thanks for the feedback, this is very valuable! :)

Moving the code to a different place would be okay to me of course.
As for being tied to "keyword" and certain HTML versions,
I suggest that we open up new issues. Otherwise this one will
get to cluttered. Would this be okay for you?

Rationale:
This is a large feature to be done in its full entity
and to me it turned out that we need to tackle it in steps
which are itself usable. I believe we have completed
a good and important step in this direction.
The restrictions towards "keyword" and html versions are there,
but were done deliberately to reach a useful step early
and not get stuck.
Maybe we could rename this issue to be more specific if this helps
to make it one step. 

What do you think?
Bernhard
msg4305 Author: [hidden] (stefan) Date: 2011-05-02 11:16
On 2011-05-02 05:19, Bernhard Reiter wrote:

> Moving the code to a different place would be okay to me of course.
> As for being tied to "keyword" and certain HTML versions,
> I suggest that we open up new issues. Otherwise this one will
> get to cluttered. Would this be okay for you?

Yes, absolutely. I didn't mean to make this a never-ending story, just 
didn't want to loose the context.

Thanks,
		Stefan
msg4323 Author: [hidden] (ber) Date: 2011-07-01 15:29
I have now split out:
  Issue2550708 (Binary filter operators specific to "keyword" should be 
more generic)
  Issue2550709 (Classic template binary search "(edit)" or "(expr)" 
link vanishes once an expression was entered)
  Issue2550710 (Classic template binary search: Too HTML4 specific)
msg4360 Author: [hidden] (ezio.melotti) Date: 2011-08-08 21:53
I tried this feature on the Roundup instance of bugs.python.org and it
didn't work out of the box.  The problem seemed to be caused by some
conflict between the words "keyword" and "keywords".
In the search page the <select> uses "keywords", but AFAIU the script
expects "keyword" in order to work, both in the "(expr)" link of the
popup and in the js code that tries to access the form using
getElementById('keyword').
I fixed it by replacing the occurrences of 'keyword' to 'keywords' where
necessary [0].  Note that in issue.search.html I just added back the 's'
that I had removed in the previous commit.  That made the popup appear
correctly, but then the search wasn't working anymore because Roundup
was expecting "keywords" (with the 's') on the other side.
My solution is probably not the most elegant (I basically hardcoded
'keyword' and 'keywords' in the right places), but it fixed the problem
for me.

Another issue I noticed is that all the keywords (included retired ones)
are listed in the popup.

[0]: http://svn.python.org/view?view=revision&revision=88877
msg4362 Author: [hidden] (stefan) Date: 2011-08-08 22:25
As the latest comment illustrates that this is in fact not working
correctly, I'm reopening the issue to increase the likeliness of it not
getting ignored.
msg5787 Author: [hidden] (rouilj) Date: 2016-07-08 23:39
I addressed Ezio's issue with the popup editor including retired values.

See commit: 0142b4fb5a2d

It was a simple commit, but while committing the following question
came up:

   Supposed a keyword is retired and I want to search for an issue
   with that retired keyword?

Do we have a best policy document that says to remove retired keywords
from all places it could possibly be used? It could be argued that the
simple search dropdown is wrong and should allow selecting retired values.

There are no comments in hg or the code to tell me how the developer
interpreted the meaning of "Retired" and if the decision to leave
retired values in the popup was intentional.

-- rouilj
msg5791 Author: [hidden] (ThomasAH) Date: 2016-07-09 13:15
* John Rouillard <issues@roundup-tracker.org> [20160709 01:39]:
>    Supposed a keyword is retired and I want to search for an issue
>    with that retired keyword?
> 
> Do we have a best policy document that says to remove retired keywords
> from all places it could possibly be used? It could be argued that the
> simple search dropdown is wrong and should allow selecting retired values.
> 
> There are no comments in hg or the code to tell me how the developer
> interpreted the meaning of "Retired" and if the decision to leave
> retired values in the popup was intentional.

I understand "retired" as:
It is deleted, but since we do not want to really delete anything in
the database, it is just marked as deleted and can be undeleted
(unretired) later.

So retired keywords/users/messages/issues should not appear
anywhere. They currently do, which sometimes causes problems, e.g.
when editing an issue.
msg5840 Author: [hidden] (schlatterbeck) Date: 2016-07-12 08:43
On Sat, Jul 09, 2016 at 01:15:30PM +0000, Thomas Arendsen Hein wrote:
> 
> * John Rouillard <issues@roundup-tracker.org> [20160709 01:39]:
> >    Supposed a keyword is retired and I want to search for an issue
> >    with that retired keyword?
> > 
> > Do we have a best policy document that says to remove retired keywords
> > from all places it could possibly be used? It could be argued that the
> > simple search dropdown is wrong and should allow selecting retired values.
> > 
> > There are no comments in hg or the code to tell me how the developer
> > interpreted the meaning of "Retired" and if the decision to leave
> > retired values in the popup was intentional.
> 
> I understand "retired" as:
> It is deleted, but since we do not want to really delete anything in
> the database, it is just marked as deleted and can be undeleted
> (unretired) later.
> 
> So retired keywords/users/messages/issues should not appear
> anywhere. They currently do, which sometimes causes problems, e.g.
> when editing an issue.

We're leaving retired keywords etc. in old issues. It's usually not a
good idea to modify history and it avoids practical issues with
referential integrity. So retired items in the database may be linked
from other items.

Concerning the search: When searching for no-longer-valid items is
required, I've implemented an 'is_valid' flag in some of my trackers
(not for keyword but for other customized properties). If an item is not
marked as valid, it can still be searched for but will not be available
in the menus for issue-editing. (So it will be in seen in index
templates but not in item templates).

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  www:   http://www.runtux.com
Reichergasse 131, A-3411 Weidling       email: office@runtux.com
msg5861 Author: [hidden] (rouilj) Date: 2016-07-16 03:03
Here is another place where it's not quite working right.

When editing a query from the query edit page, any conditions
created by the expression editor are lost.

E.G. create a keyword search "keyword1 and not keyword2".
Save the search. You can execute the search and it will work fine.
Now edit the search and set the owner to semothing.

Note that the edit screen doesn't display

  "keyword1 and not keyword2"

anymore. If you save the query, you loose the keyword expression
even though it was not edited.

I am not sure if this is an issue with the underlying mechanism
we use to store the query or just a problem with the form that is created.

-- rouilj
History
Date User Action Args
2016-07-16 03:03:50rouiljsetmessages: + msg5861
2016-07-12 08:43:21schlatterbecksetmessages: + msg5840
2016-07-09 13:15:30ThomasAHsetmessages: + msg5791
2016-07-08 23:39:40rouiljsetnosy: + rouilj
messages: + msg5787
2013-09-24 20:06:29cslottysetnosy: + cslotty
2011-08-08 22:25:35stefansetstatus: closed -> open
resolution: fixed -> accepted
messages: + msg4362
2011-08-08 21:53:28ezio.melottisetnosy: + ezio.melotti
messages: + msg4360
2011-07-01 15:29:53bersetstatus: open -> closed
resolution: fixed
messages: + msg4323
2011-05-02 11:16:18stefansetmessages: + msg4305
2011-05-02 09:19:03bersetmessages: + msg4304
2011-04-21 15:25:05stefansetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg4302
2011-04-15 07:34:28bersetstatus: pending -> closed
messages: + msg4288
2010-11-11 15:51:32ThomasAHsetmessages: + msg4204
2010-11-11 14:12:26schlatterbecksetmessages: + msg4203
2010-11-09 15:07:48bersetstatus: new -> pending
resolution: fixed
messages: + msg4201
2010-11-09 14:05:42schlatterbecksetmessages: + msg4200
2010-11-08 16:26:19bersetmessages: + msg4199
2010-11-08 15:49:12bersetassignee: ber
2010-10-19 16:31:56slteichmannsetfiles: + keyword-expr-alldbm3.diff
messages: + msg4167
2010-10-19 09:34:48slteichmannsetmessages: + msg4163
2010-10-19 09:30:19ThomasAHsetmessages: + msg4162
2010-10-19 09:14:15bersetmessages: + msg4161
2010-10-18 13:47:32slteichmannsetmessages: + msg4160
2010-10-18 12:47:06stefansetmessages: + msg4159
2010-10-18 12:36:16bersetmessages: + msg4158
2010-10-18 10:25:09slteichmannsetfiles: + keyword-expr-alldbm2.diff
messages: + msg4157
2010-10-17 19:36:25slteichmannsetfiles: + keyword-expr-alldbm.diff
messages: + msg4153
2010-10-16 11:47:31slteichmannsetfiles: + keyword-expressions-complete.diff
messages: + msg4152
2010-10-15 16:25:47stefansetmessages: + msg4151
2010-10-15 16:04:01bersetmessages: + msg4149
2010-10-15 15:47:47stefansetnosy: + stefan
messages: + msg4148
2010-10-15 13:23:33slteichmannsetfiles: + expr-editor.html
2010-10-15 13:23:23slteichmannsetfiles: + keyword-expressions.diff
keywords: + patch
messages: + msg4145
nosy: + slteichmann
2010-05-10 07:23:00bersetnosy: + schlatterbeck
messages: + msg4053
2010-05-05 13:34:19ThomasAHsetnosy: + ThomasAH
2010-05-05 09:18:21pefusetnosy: + pefu
2010-05-04 21:13:16bercreate