Currently only Date properties allow searching ranges
in the web interface. This should be implemented for
numbers, too, e.g., that searching for 
1-200 (or similar syntax, see below)
for a number property will find values in the
(inclusive) range 1-200. A similar syntax for exclusive
ranges would be nice -- maybe use the mathematical
notation with an outward-facing angular bracket for an
exclusive range, e.g. 
[1,200] would find 1<=x<=200
]1,200[ would find 1<x<200
[1,200[ would find 1<=x<200
support for a list of several ranges in a single query
would be fine :-)
leaving out either the left or the right range number
should denote -infinity vs +infinity, respectively.

I currently have a requirement where searching for
numbers greater 0 would be very nice to have...

This patch add a very bare-bones approach to Number searching by
intervals. It allows searching for 'x', '>x' and '<x', where x is a
positive integer.

Support for fancy format specifications and float values should be easy
to implement on top/instead of this.

I didn't know what is apparently the ISO notation
(<http://en.wikipedia.org/wiki/Interval_(mathematics)>); I was familiar
with the "set builder notation":

[1,200] would find 1<=x<=200
(1,200) would find 1<x<200
[1,200) would find 1<=x<200

The "]a,b[" example looked contra-intuitive to me; some could read it as
"<= a or >= b".  Thus, the mathematical notation might not be the best
solution;  it doesn't conform with the Python notation anyway:

  range(1,200)
or
  [1:200]  # Python

is like
  [1,200[  # ISO
  [1,200)  # set builder
or
  [1,199]  # both ISO and set builder

What about accepting input like this:

1 <= x <= 200   # Python; accept any identifier
>=1 and <=200   # pythonic/english "and"
>=1 && <=200    # C, Bash, DOS-Shell

1 < x < 200     # Python; accept any identifier
>1 and <200     # pythonic "and"
>1 && <200      # C, Bash, DOS-Shell

1 <= x < 200    # Python; accept any identifier
>=1 and <200    # pythonic "and"
>=1 && <200     # C, Bash, DOS-Shell
 
We could create a callable object when parsing a non-empty expression. 
The concept could be extended to evaluate parentheses and even field
names (for a one-entryfield search function).  The comma could be
interpreted as "and".

We could generate boolean functions (accepting a single number) or,
perhaps better, SQL code (for the WHERE clause).

The expression syntax would be a superset of Daniel's solution :-)

On Wed, Apr 01, 2009 at 03:01:03PM +0000, Tobias wrote:
>
> I didn't know what is apparently the ISO notation
> (<http://en.wikipedia.org/wiki/Interval_(mathematics)>); I was familiar
> with the "set builder notation":
>
> [1,200] would find 1<=x<=200
> (1,200) would find 1<x<200
> [1,200) would find 1<=x<200

I can live with that, too.

> What about accepting input like this:
>
> 1 <= x <= 200   # Python; accept any identifier
> >=1 and <=200   # pythonic/english "and"
> >=1 && <=200    # C, Bash, DOS-Shell
>
> 1 < x < 200     # Python; accept any identifier
> >1 and <200     # pythonic "and"
> >1 && <200      # C, Bash, DOS-Shell
>
> 1 <= x < 200    # Python; accept any identifier
> >=1 and <200    # pythonic "and"
> >=1 && <200     # C, Bash, DOS-Shell

This looks too complicated to me and doesn't fit well with the
already-existing syntax for date ranges. If you want the added
complication of boolean expressions, these should be optional for
application like searching for multiple ranges, e.g.,

(5, 27) or [99,100)

alternatively:

(5, 27), [99,100)

> We could create a callable object when parsing a non-empty expression.
> The concept could be extended to evaluate parentheses and even field
> names (for a one-entryfield search function).  The comma could be
> interpreted as "and".
>
> We could generate boolean functions (accepting a single number) or,
> perhaps better, SQL code (for the WHERE clause).

You'll probably need both for the various backends. For SQL backends you
definitely want to generate SQL. For dbm you need some boolean
expressions. And our input syntax should cover a subset of what can be
expressed in SQL (semantically not syntactically).

and we should keep in mind that we may want to use the new syntax for
dates, too (optionally, I think we need to be backward compatible)

Ralf

The interval syntax has the drawback to require an upper value,
which is not true for specs like ">5".

I have implemented a module (attached) which supports:

- Daniel's simple conditions (">a", "<a")
- closed intervals ("[a,b]")
- ISO open intervals ("]a,b[")
- set builder open intervals ("(a,b)")
- boolean operators ("and" and "or", also accepted as "&&" and "||")

The expression is tokenized, and some operators are accepted in variants
known from other programming languages; then, some simple
transformations are performed to create a valid Python expression.

The comma is not accepted as a boolean operator because it is needed
for the interval notion.

I'm not sure whether it does really make sense to support open
intervals; however, it is possible (and done).  Of course, this could be
removed easily.

I'll upload a test module as well; with Python 2.5+, it yields a strange
message:
  Exception exceptions.RuntimeError: 'generator ignored GeneratorExit'
    in <generator object at 0x00AC6E68> ignored
However, all implemented tests succeed.

On Fri, Apr 03, 2009 at 08:13:10AM +0000, Tobias wrote:
> 
> The interval syntax has the drawback to require an upper value,
> which is not true for specs like ">5".

Why? My original proposal says you can leave out upper or lower value,
e.g. (123,) and of course the syntax should be backward compatible to
allow simple numbers, e.g.
123 or 124 or (127,130)

> 
> I have implemented a module (attached) which supports:
> 
> - Daniel's simple conditions (">a", "<a")
> - closed intervals ("[a,b]")
> - ISO open intervals ("]a,b[")
> - set builder open intervals ("(a,b)")
> - boolean operators ("and" and "or", also accepted as "&&" and "||")

Nice. I currently don't have the time to look into this.

Ralf
-- 
Ralf Schlatterbeck
email: ralf@zoo.priv.at FAX: +43/2243/26465/23

>> The interval syntax has the drawback to require an upper value,
>> which is not true for specs like ">5".

>Why? My original proposal says you can leave out upper or lower value,
>e.g. (123,)

Right; missed that.

Regarding the comma, IMO we should support specifications like
  "2, 4, 5"
or
  "(2, 4, 5)",

meaning something like
  "x in (2, 4, 5)"
or
  "x == 2 or x == 4 or x = 5".

This would make it difficult to support the comma in ranges as well (as
it is the case in my current implementation).  For date ranges, we use
the semicolon, don't we?

Thus, IMO we should use the semicolon ";" as a "range operator" (along
with "-", ".." and "..."), and the comma "," should be equivalent to a
logical "or".

BTW, meanwhile I found out that the Python 2.5+ warning in my
implementation seems to be related to PEP 342
(http://www.python.org/dev/peps/pep-0342/).  If anyone could squeeze
some sense out of the warning and tell me how to fix my code in a way
compatible to Python <= 2.4, I'd appreciate that.

I'll second this wish to make the Number type more usable.
(One additional issue to improve Numbers is Issue2550559 (Pretty 
printing / formatting for Number types.))

One comment on using "," as separator. Please do not do this.
"." and "," are typical decimal delimiter characters. 
Roundup should be able to be localised in languages using ","
like in Germany in the user interface.
So it is not good to use it for range indications.

Tobias,
I took a first look at your intrange.py and test_intrange.py files
from 2009-04-03:

* Which "errors" module are you refering to in main of intrange.py?
* The "warning" you are getting comes from the 
  test_invalidExpressions test.
  Found by running
    python test_intrange.py -v 
  So here is a run only showing the warning:
python test_intrange.py -v TestCase_intrange.test_invalidExpressions

Tobias,
I've simplified your warning case further, maybe it helps:
$ python
Python 2.5.2 (r252:60911, Jan  4 2009, 17:40:26)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from intrange import makefilter, ExpressionError
>>> makefilter("<8 and or >1")
Exception exceptions.RuntimeError: 'generator ignored GeneratorExit' in 
<generator object at 0xb7db2fec> ignored
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "workbench/numbers/intrange.py", line 235, in makefilter
    expr, varname = expressify(s)
  File "workbench/numbers/intrange.py", line 188, in expressify
    for newtype, token in completed(s):
  File "workbench/numbers/intrange.py", line 146, in completed
    % (token, newtype))
intrange.ExpressionError: 2nd token of same type (or, 2)
>>>

@Bernhard:

> Which "errors" module are you refering to ...

Well, this is my private error/warning counting module for commandline
tools (I thought everyone would have one).  Since it is not necessary
for intrange usage, I omitted it; it was not suitable yet for an
international audience.

Not very surprisingly, the "err" function writes a text to stderr and
increases a counter; "check_errors" checks this counter and terminates
the program.

(Ok, you got me; here it is. When called directly, it can tell about its
functionality.)

> The "warning" you are getting comes from the
> test_invalidExpressions test.

Well, of course I know (or knew at the time of writing ;-) the location
of the warning in my code.  I just don't understand it (it doesn't make
any sense to me, and in fact looks like some ill assumption about
iterators); and Python versions <= 2.4.x don't warn.

To me, my code looks right; nothing to complain about.  If anybody else
understands the warning, please explain it to me and tell me how to
avoid it (in a *compatible* way, of course).  Currently I consider this
warning a Python bug.

> One comment on using "," as separator. Please do not do this.
> "." and "," are typical decimal delimiter characters.

Point taken. *But* I consider it very unlikely ...
- to be confronted with non-integer numeric values here
  (date values can be handled in another place)
- formatted (thousands-separated) input values to be supported

The comma "," is the common punctuation mark for enumerations in most
languages (I couldn't tell a counterexample); the semicolon ";" divides
related sentences, right?

Thus, not supporting the comma would inconvenience most users.

Am Donnerstag, 9. Juli 2009 12:21:37 schrieb Tobias:
> > One comment on using "," as separator. Please do not do this.
> > "." and "," are typical decimal delimiter characters.
>
> Point taken. *But* I consider it very unlikely ...
> - to be confronted with non-integer numeric values here

Think about rates, scientific parameters or small prices;
they are quite common. And because roundup shall directly interface with 
users, it must accommodate for representing "12.5" like "12,5" in German.

> - formatted (thousands-separated) input values to be supported

I agree here, thousands-separated values can be left out.

In general I have the feeling that the syntax is too complicated
and something more simple would be adaquat. 
Ideally it should be considered to join the parser for Date and Number ranges
to avoid code duplication and easy learning.

From the current discussion I'd propose the following syntax --
simplifying what is allowed and not confusing users with syntax from
(other) programming languages, this takes into account the wish for
simplification *and* Tobias' proposal to use the semicolon as the range
operator (which also is used in date ranges currently).

use angular brackets for inclusive range, use round paranthesis for
exclusive range. Don't use the out-facing (ISO-Notation) angular
bracket, use ";" for ranges so that future implementations may use a
comma for decimal notation (german), e.g.

[1;2.6)
[1;2]
if there is only a single inclusive range we may leave out the brackets
making the syntax resemble the date ranges:
1;2
As in the date-range implementation we may want to allow "to" instead of
";".

For multiple ranges we may use the "or" or a "," for separating ranges
-- note that this time the comma doesn't collide with the numeric ranges
because we make brackets/parantheses mandatory, e.g.

[;0],[1;5] to (7;9),[11;]

Oh, and supporting "and" for intervals will confuse the typical user,
won't it?

What do you think ?

That also would require interval arithmetics, if the user specifies
several overlapping intervals with "or" we should silently join these
(or leave that to the sql engine).

Of course the same syntax should be generalized for date values, too.

> Think about rates, scientific parameters or small prices

Ok, you're right;  I was thinking along the lines of identifiers or at
least enumerable types (which floating point numbers are not).

With whitespace following, a comma could some day be accepted as an
operator (once we have a solution which accepts formatted input, e.g.
with decimal commas);  AFAIK, punctuation marks (as commas in
enumerations) are separated by whitespace from following text in all
natural languages using latin letters anyway.

> supporting "and" for intervals will confuse the typical user

Agreed.  Maybe we can some day come up with a suitable keyword or symbol
for 'intersection'.

> Of course the same syntax should be generalized for date values, too.

Agreed.

This arises the problem of compatibility to existing syntax which must
be handled somehow.  However, I don't insist in total compatibility,
since the existing syntax doesn't support enumerations, and IMO we need
this.

Thus, here is my new proposal:

expression                resulting set  remarks     
==========                =============  =======

1; 2; 3                   1, 2, 3        no parentheses nor 'to'
                                         `-> enumeration (';' is 'or')
1 to 5                    1, 2, 3, 4, 5  'to' -> range
1; 5                      1, 5           <> existing date syntax!
[1; 5]                    1, 2, 3, 4, 5  parentheses -> range
                                         (';' is 'to')
1 to 3; 6                 1, 2, 3, 6
[1; 5)                    1, 2, 3, 4     like current "1; 5" for dates
(1; 5)                    2, 3, 4        for consistency
[1; 3] or (6; 9)          1, 2, 3, 7, 8
[3;]                      3, 4, 5 ...
(3;)                      4, 5, 6 ...
(3;) or 1                 1, 4, 5, 6 ...
(3;);1                    1, 4, 5, 6 ...

With comma support (whitespace after semicolon is optional; whitespace
after comma is significant):

1, 2, 3                   1, 2, 3        natural language
1,2,3                     ERROR          possible ambiguity
1,5                       ERROR          possible ambiguity
(1,5)                     ERROR          possible ambiguity
(1, 5)                    2, 3, 4
[1,]                      1, 2, 3, ...

(all examples with ";" should work, with ";\s*" replaced by ", ";
commas raise errors unless followed by whitespace or parentheses)

Thus, the 'to' keyword would imply a closed range, *including* the
boundary values: "a to b" is the same as "[a; b]" (or "[a to b]").

Logic:
- parentheses are considered first:
  * every parenthesis is unambiguously opening ('(', '[', perhaps '{')
    or closing
  * assumptions about the possible content are implied (ranges, open
    or closed; perhaps '{' enforcing non-range)
- 'to' implies a range; unless the nature of the range is appointed
  by parentheses, it implies a *closed* range
- 'or' implies alternatives
- every ';' can be replaced by 'to' (range meaning) or 'or'

More examples:

expression                resulting set  remarks     
==========                =============  =======

1 to 3; 5                 1, 2, 3, 5
1 to 3 to 5               ERROR
[1; 3] or 5               1, 2, 3, 5
[1; 3]; 5                 1, 2, 3, 5
(1; 3) or 5               2, 5
1 or 3 or 5               1, 3, 5        {without parentheses,
1; 3 or 5                 1, 3, 5         ";" means 'or'}

To be discussed:

(1 to 3; 5)               1, 2, 3, 5     internally turned to
                                         ((1 to 3) or 5)
1 to 3 or 5               1, 2, 3, 5     like "(1 to 3) or 5"
[1 or 3]                  1, 3           because of "[1; 3]", which
                                         includes 1 and 3 as well
(1 or 3)                  1, 3           should be allowed
{1; 3}                    1, 3           non-range implying parentheses
{1 to 3}                  ERROR          contradiction
(3;)1                     ERROR          missing operator
(1; 3;)                   1, 3; or ERROR ';' before closing parenthesis
                                         could imply "no limit"
(;)                       all            as long as no 'and' is imple-
                                         mented, this will cause the
                                         entire expression to match
                                         every value
1 5                       1, 5
1;3 5                     1, 3, 5
(1;3 5)                   ERROR          ambiguity

Some thoughs for dates:
- the syntax is incompatible to the existing;  the current
  "a;b" date expression would need to be expressed as "[a; b)"
- "a to b" (or "[a; b]") would *contain* the records of day b
  (it doesn't currently, does it?), while "[a; b)" would not;
  as long as no hours or minutes (...) are specified, only the date
  parts of the values should be considered
- we should have a configuration switch for date expression
  compatibility:
  * first, the switch should be set by default to evaluate *date*
    expressions the traditional way (since they might be used by
    scripts, which is not true for non-date range exceptions)
  * after a reasonable time, the next major Roundup version switch
    (Roundup v1.5?) should switch this default to the new syntax
  * again after a reasonable time, the support for old date expressions
    could be removed

A function which takes an expression and returns a function,
could/should return a normalized version of the expression as well, e.g.
replacing all semicolons with 'or' or 'to', like this:

>>> evaluate("[3;8)")
("[3 to 8)", lambda x: 3 <= x < 8)

If this normalized version would be used in a "refine search" input
field of the result page -- or displayed/inserted by some AJAX
functionality -- it could make the whole feature quite self-explanatory.
The only difficult bit would be about "open" ("()") or "closed" ("[]")
ranges.

Longing for comments!

On 10/07/2009, at 8:57 PM, Ralf Schlatterbeck wrote:
> [;0],[1;5] to (7;9),[11;]

I have no idea what that could possibly mean :)

My module (attached) says:

"AmbiguityError: token 'to' is not allowed in enumerations" ;-)

Test module following.

On Mon, Jul 13, 2009 at 02:11:06AM +0000, Richard Jones wrote:
> 
> Richard Jones <richard@mechanicalcat.net> added the comment:
> 
> On 10/07/2009, at 8:57 PM, Ralf Schlatterbeck wrote:
> > [;0],[1;5] to (7;9),[11;]
> 
> I have no idea what that could possibly mean :)

Oops. That should be "or" instead of "to".
so it should really be

[;0],[1;5] or (7;9),[11;]
"or" can be used instead of ",".

meaning the numbers are in one of the following intervals:
- infinity to 0 (0 included)
- 1 to 5 (1 and 5 included)
- 7 to 9 (7 and 9 *not* included in the range)
- 11 to infinity (11 included)

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  Fax:   +43/2243/26465-23
Reichergasse 131                        www:   http://www.runtux.com
A-3411 Weidling                         email: office@runtux.com
osAlliance member                       email: rsc@osalliance.com

With the semicolons replaced with commas, my module yields

   lambda x: (x <= 0) or (1 <= x <= 5) or (7 < x < 9) or (11 <= x))

Btw, it is currently a little bit too liberal; some (very queer)
expressions yield wrong results. I'll apply a few changes to decrease
complexity and submit a new version.

Sorry, this has become quite long as I'm trying to show my syntax
proposal alongside yours. Some comments on your proposal to use two
distinct syntaxes are at the end.

My proposal boils down to 
- Use ';' or 'to' for ranges
- Use ',' or 'or' for enumeration of ranges (joining them)
- Require brackets/parentheses around ranges if several are used (needed
  only if decimal comma should be supported)
- Make inclusive ranges the default (brackets)
This is compatible with existing date range syntax except that current
date ranges use an implicit interval of [) where the end of a range is
*not* included. So maybe we want the default range to be [)

On Sun, Jul 12, 2009 at 03:04:52PM +0000, Tobias wrote:
> 
> Thus, here is my new proposal:
> 
> expression                resulting set  remarks     
> ==========                =============  =======
> 
> 1; 2; 3                   1, 2, 3        no parentheses nor 'to'

This breaks compatibility with existing date ranges where the ";"
represents the range operator. So this would be a syntax error in my
proposal, if you want to specify an 'or' of three numbers you'd have to
write [1], [2], [3] (except if we don't allow decimal comma, then we can
leave out the brackets yielding 1,2,3)

These two should be equivalent as in the existing date syntax.
The existing syntax uses the ";" as the range operator. And there currently
is no need to write explicit brackets. Think of stored queries by users!

I'd prefer this to mean the same thing as without brackets.

.. and *require* brackets if you want several ranges. This can
disabiguate the use of the ",", i.e, I'd write here
[1 to 3] , [6] Note that you require the brackets/parentheses only if
you want to support alien number formats using a decimal comma. They
wouldn't be needed if we can live with a decimal point (see comments at
the end).

BTW: Roundup represents number as floats and you're using integer
examples only. The real benefit of the round parantheses comes with real
numbers (e.g., "less than" instead of "less or equal")

> [1; 3] or (6; 9)          1, 2, 3, 7, 8
> [3;]                      3, 4, 5 ...
> (3;)                      4, 5, 6 ...
> (3;) or 1                 1, 4, 5, 6 ...
> (3;);1                    1, 4, 5, 6 ...
> 
> With comma support (whitespace after semicolon is optional; whitespace
> after comma is significant):

Uh, I don't think you can train users that whitespace is significant.
Lets use the comma for multiple ranges and the semicolon for ranges.
In that way the comma inside braces/parentheses is used for separation
of ranges while inside the braces/parentheses it can be used as a
decimal comma (if we really want to support that).

> 1, 2, 3                   1, 2, 3        natural language
> 1,2,3                     ERROR          possible ambiguity
> 1,5                       ERROR          possible ambiguity
> (1,5)                     ERROR          possible ambiguity
> (1, 5)                    2, 3, 4
> [1,]                      1, 2, 3, ...
> 
> (all examples with ";" should work, with ";\s*" replaced by ", ";
> commas raise errors unless followed by whitespace or parentheses)
> 
> Thus, the 'to' keyword would imply a closed range, *including* the
> boundary values: "a to b" is the same as "[a; b]" (or "[a to b]").

I'd prefer to make the exclusive range explicit with round parentheses
while making angular brackets (inclusive range) the default. And use the
"to" and the ";" interchangeably (as in the current date implementation).

Fine (execpt for introducing a third form with curly braces here).
A range opening with a round paranthesis may be closed by an agular
bracket and vice versa.

Fine.

Fine.

As said above, I prefer the ';' to mean the same as "to" as it is
currently implemented for date ranges. Otherwise you'll break all stored
queries by all users.

Should be [1 to 3] , [5]
brackets could be left out if we stick to the number format with decimal
point only.

definitely :-)

5 should be bracketed:
[1; 3] or [5]

As said above: I'd prefer to use the semicolon inside braces while using
the comma for enumerating ranges:
[1; 3] , [5]

(1; 3) or [5]

[1] or [3] or [5]
or
[1], [3], [5]

But see below: If we stick to decimal-point only we can leave out
brackets in most examples.

No, don't mix ranges and enumeration.

Require ranges to be bracketed/parenthesized if enumeration is used

Dont mix ranges and enumeration.

Dont mix ranges and enumeration.

Don't use curly braces, this makes the syntax even harder to understand.

Error. If you want no limit, use
[1] or [3;]

This is the same as currently implemented for date ranges. With or
without braces and no matter if inclusive or not would be an extension.

Don't add an additional syntax with spaces.

See my proposal:
- Use ';' for range separation, if several ranges are used, use 
  explicit brackets/parentheses. Use ',' or 'or' for combining ranges.
  This is completely backward compatible.

You're right, [a; b] *will* contain a *but not* b. You just don't see this
because usually you don't have anything that starts/ends at exactly 0:00
on the given date and specifying a search date up to the second is
seldomly done.

Currently if no hours and minutes are specified this means 0:00 and I
don't think this should/can be changed. I've implemented a time-tracker
using roundup and I'm relying on the current date semantics. So I
strongly vote against changing that.

Uh. No please don't. You don't want to maintain two semantics for the
future. Lets make the expression syntax compatible with the current
syntax.

I strongly vote against this proposal. It adds a maintenance headache
(think two sets of regression tests for all time to come).
And think of the user-experience: There are *lots* of stored queries
now (in the trackers I'm running at least). They need to be in one or
the other syntax. So you're forcing users to switch at some point.
I don't want to think about the other option of setting a syntax-flag
for queries containing date expressions ...

> A function which takes an expression and returns a function,
> could/should return a normalized version of the expression as well, e.g.
> replacing all semicolons with 'or' or 'to', like this:
> 
> >>> evaluate("[3;8)")
> ("[3 to 8)", lambda x: 3 <= x < 8)
> 
> If this normalized version would be used in a "refine search" input
> field of the result page -- or displayed/inserted by some AJAX
> functionality -- it could make the whole feature quite self-explanatory.
> The only difficult bit would be about "open" ("()") or "closed" ("[]")
> ranges.

I don't understand the purpose of the expression function.
In most backends you'll want to directly generate SQL. Only for the
non-sql backend(s) (there is only dbm left?) you need to evaluate the
expression.

And another thing for consideration: Although I'm german (living in
austria for quite some time) and here we have a decimal comma instead of
a decimal point, I *can* live with an expression syntax that uses
decimal points for numbers. In that case we can leave out the brackets
in many of my examples. But such a decision shouldn't be taken lightly.
Note that if you support several different number formats, switching of
the browser language will invalidate existing queries!

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  Fax:   +43/2243/26465-23
Reichergasse 131                        www:   http://www.runtux.com
A-3411 Weidling                         email: office@runtux.com
osAlliance member                       email: rsc@osalliance.com

First, what we agree about:

- Semicolon and 'to' should continue to work as range operator
- the optional 'from' could be supported as well
- for several ranges to be combined, parentheses are needed
- all existing search syntax expression syntax
- 'or' should be supported to combine range expressions
- the comma currently works as an 'or' operator, which I admit
  I wasn't aware of (I didn't use it, since whitespace works as well)
  and should continue to work.

And, of course, in real world usage I will yield SQL code (e.g. a
"while" clause without "while", but with dict comprehension placeholders
for the variable name).  My first goal was expression evaluation, and it
won't be difficult to create an expression for SQL instead of Python. 
The Python expression is a good first shot, and it can be tested very
easily.

I agree that some of my examples are somewhat outdated; as I wrote in
msg3772, the syntax is too liberal, and I will restrict it (see below).
As mentioned above, I was not aware of the current nature of the comma.

BTW, as you could have noticed, I uploaded a unit test for float ranges
as well (file412); and of course I know that enumerations don't make
much sense for them.

Then something where you are IMO wrong:

- The whitespace syntax is *not* new; it is supported e.g. when seeking
  issues by id.  Thus, it *must* continue to work.
- *Requiring* the comma (or 'or') as an enumeration operator thus
  would *break* compatibility (yes, it is supported; but it is
  currently not *required*).

I disagree about the decimal comma cause. You and me, we are IT experts
who are used to decimal points; but many people who are supposed to
simply *use* Roundup are not.  One major strength of Roundup is its huge
customizability which makes it suitable for non-geek usage (for people
who won't use Bugzilla because it is -- or looks -- too complicated).
Roundup should treat them well.

Of course it would be possible to recognise floats in both flavours,
with decimal points or decimal comma, regardless of the locale setting. 
It just occurred to me: we *have* a difference already between search
syntax for date and integers, which is derived from the value syntax;
and we know it doesn't make very much sense to support enumerations for
floats, because of the nature of the data.  Thus we could simply drop
enumeration support for floats (only open intervals and 'or'), and we
could support decimal commas right away.  Probably we should require at
least one digit behind the comma, and when enumeration syntax is used,
the expression generator could yield a useful message.

I strictly disagree about whether curly brackets would make the syntax
harder to understand; and I disagree about the "problem" of having both
ranges and enumerations in the same expression.

I consider it a quite common need to specify both a range and a
enumeration.  It is easy, it is implemented, and it should be possible.

What is difficult about  "[3; 7] or {2 3 5}"?  The most difficult bit is
the difference between open and closed interval edges.  The curly
brackets are a possibility to explicitly state you want an enumeration.
This is very "pythonic": explicit is better than implicit ;-)

There is a problem about the range "(3; 5)" and an enumeration "(3, 5)":
The range doesn't contain the edges (thus, for integers it would yield
4), while the enumeration *does* contain the edges.  This is
inconsistent.  The better solution would be to enforce the usage of
curly brackets (instead of others; see below) for enums, which make
clear that an enumeration is a completely different beast.

There is another problem.  Considering "(a;)" and "(;a)", logically
"(;)" must be a match-all expression, because a range spec contains
restrictions which are not present here.  An empty set "(,)" in contrast
can never ever match *all* but must match *nothing*, because the set
values are positive specifications.  It is A Good Thing to make this
difference visible, and to write the set "{}".

My updated proposal is:
- semicolons are for ranges, commas for enumerations/sets (your point)
- whitespace is for enumerations/sets (support required and convenient)
- "a; b"             -- a range
- "a b", "a, b"      -- equivalent sets
- "[a; b]"           -- the above range, but combinable
- "{a, b}", "{a b}"  -- the above set, but combinable
- 'or' can be used anywhere instead of commas
- 'to' can be used anywhere instead of semicolons
- commas (...) inside range bracketes are errors
- semicolons (...) inside set brackets are errors
- expressions like "a; b; c" (a range can have at most 2 edges)
  or "a; b, c" are errors (change of expression type requires brackets)
- a single value in the expression is a "set of itself"
  (and of course doesn't require a comma)
- when an error occurs, a nice message (including the position)
  explains it.

Now the compatibility topic.

We need a new syntax anyway; for numeric fields, the new syntax will be
the *only* possibility to specify ranges.  To stay compabible, it *must*
support whitespace enumerations for numbers.

For date values, whitespace can't be used this way, because Roundup's
date specs can contain whitespace.  There is nothing we can do about
this difference:  Whitespace was supported for integers, and so must
continue to be; whitespace could not be supported for date values, and
won't ever be.  For both cases (and for floats as well, of course),
there are two possibilities they have in common:
- the 'or' operator
- inside curly brackets, the semicolon divides values, but changes
  nothing about the nature of the enumeration-implying curly bracket.

I can't see a problem with my proposal of a configuration switch
(msg3765).  The current functionality for the old date search syntax is
there, and so are hopefully the unittests.  There will be unittests for
the new syntax as well.  What else do you need?!

If it is already true that date specs like '-1d' refer to yesterday,
0:00, this is fine; nobody wants to change this.  The 2nd edge seems to
be seldom used anyway; however, we should be as consistent between data
types as possible, and thus a specification of "yesterday;today" should
find *all* of yesterday and *all* of today (some date values can lie in
the future).

It is *possible* that there are scripts around that rely on the
[date1;date2) logic of the old syntax.  Given enough time, it won't be a
problem to update the scripts; this wouldn't be complicated.  Given a
configuration switch, every admin can decide himself if and when to
switch.  Stored queries could be updated easily be a script as well.  It
would be worth the afford.

Let's take the opportunity to do it right.  I'll implement it -- maybe
with a little help with the dates --, including the unit tests, and
everyone will like it; I promise ;-)

>> If this normalized version would be used in a "refine search" input
>> field of the result page -- or displayed/inserted by some AJAX
>> functionality -- it could make the whole feature quite
>> self-explanatory.

> I don't understand the purpose of the expression function.

Probably you misunderstood the idea of the *normalized* version (which
is not the same as the Python expression).  This 'normalized' method is
not active in my submitted module because of an obscure "SyntaxError"
(nothing obvious at the stated position, but the error is there on my
system in Python 2.3 ... 2.6, and it is not related to the .pyc file). 
It's purpose is to provide the best and most expressive version of the
input expression to support users and prevent them from thinking they
have executed a search which they in reality haven't.

As mentioned before, the purpose of the expression function is:
development, prove of concept, testablity;  a small step before the SQL
expression.  Furthermore, I'm not sure whether all of our database
backends support expressions like "a BETWEEN x AND y" or "a IN (1, 2,
5)", and IIRC we store integers as CHARs;  reasons enough to use Python
expressions for a first shot, I think.

> For date values, whitespace can't be used this way, because Roundup's
> date specs can contain whitespace.  (...) For both cases (and for
> floats as well, of course), there are two possibilities they have in
> common:
> - the 'or' operator
> - inside curly brackets, the semicolon divides values, but changes
>   nothing about the nature of the enumeration-implying curly bracket.

Correction: Of course, we'll use (consistently) commas or 'or' for date
enumerations.  As mentioned elsewhere, enumerations of float values
don't make much sense;  thus I suggest to drop them.

Just a note here:
- I'm considering enumerations of numbers (1,2,3) to be enumerations of
  degenerate ranges (with a single point). This makes the curly-brace
  syntax obsolete. We can specify it as
  [1] or [2] or [3]
  I agree that this is not a very nice syntax.
- To make the syntax easier we might want to find a way to combine
  intervals into sets *without* requiring brackets/parentheses. 
  - Space won't work as date expressions already can contain spaces
  - '+' won't work as it is also contained in date expressions (and
    might be used as a sign in numbers)
  - 'or' is fine, it doesn't collide with anything
- So maybe we *require* an 'or', then braces can be left out in the
  general case? Or find another unused symbol instead of ',' or space?

On Mon, Jul 13, 2009 at 05:57:46PM +0000, Tobias wrote:
> - the comma currently works as an 'or' operator, which I admit
>   I wasn't aware of (I didn't use it, since whitespace works as well)
>   and should continue to work.

This I don't understand, maybe it is a misunderstanding. As far as I
know, date ranges currently accept only a single range, not multiple
ranges. Neither separated by comma nor by space.

Or do you mean in integer fields ?

I currently don't have a schema containing an integer to test with.
I don't think you can search multiple integers currently.

> Then something where you are IMO wrong:
> 
> - The whitespace syntax is *not* new; it is supported e.g. when seeking
>   issues by id.  Thus, it *must* continue to work.

Ah, yes, searching ids works with whitespace.
Note that this is different from integers. IDs are *strings* in roundup.
That's why you can combine them with space!

It doesn't work when searching by name, e.g., if I'm searching for two
creators with their usernames, I can only use a comma, no space (and no
optional whitespace after the comma). No matter if I'm searching
numerically or by username.
It looks like form-processing is doing some additional magic here, a
space becomes a '+' in the request URL in my test.

> - *Requiring* the comma (or 'or') as an enumeration operator thus
>   would *break* compatibility (yes, it is supported; but it is
>   currently not *required*).

I haven't thought about using a space for separation of the ranges. Now
that I think of it, it may be a good for numbers. But what does the
form-processing create from a space? And as you point out later, date
expressions like '. -1d' can contain spaces. That rules out the space as
a separator.

> I disagree about the decimal comma cause. You and me, we are IT experts
> who are used to decimal points; but many people who are supposed to
> simply *use* Roundup are not.  One major strength of Roundup is its huge
> customizability which makes it suitable for non-geek usage (for people
> who won't use Bugzilla because it is -- or looks -- too complicated).
> Roundup should treat them well.

Fine with me. But the comma should *always* be accepted and should not
change with the browser language (otherwise changing browser language
settings will invalidate stored queries).

> Of course it would be possible to recognise floats in both flavours,
> with decimal points or decimal comma, regardless of the locale setting. 

See above, otherwise stored queries will be a problem.

> It just occurred to me: we *have* a difference already between search
> syntax for date and integers, which is derived from the value syntax;
> and we know it doesn't make very much sense to support enumerations for
> floats, because of the nature of the data.  Thus we could simply drop
> enumeration support for floats (only open intervals and 'or'), and we
> could support decimal commas right away.

See my examples, I've always seen enumeration of values a special case
of single-point intervals, i.e., 1,2,3 is [1],[2],[3]
So you can have enumeration of numbers when seeing them as single-point
intervals. So even if it doesn't make sense to enumerate floats (in most
cases there are exceptions if you know what you are doing) i wouldn't
want to restrict the syntax to integers.

No, many queries will use integer numbers.

See above for enumeration syntax. Don't make enumeration of numbers a
special case. Use enumeration of intervals only.

> I strictly disagree about whether curly brackets would make the syntax
> harder to understand; and I disagree about the "problem" of having both
> ranges and enumerations in the same expression.

So far I've not understood what curly braces are for.

> I consider it a quite common need to specify both a range and a
> enumeration.  It is easy, it is implemented, and it should be possible.

Ah, curly braces for enumerations?

> What is difficult about  "[3; 7] or {2 3 5}"?  The most difficult bit is
> the difference between open and closed interval edges.  The curly
> brackets are a possibility to explicitly state you want an enumeration.
> This is very "pythonic": explicit is better than implicit ;-)

In the example above 3 and 5 are already in the interval, so lets pick a
diffent example, curly braces
[3; 7] or {9 10 11} would be equivalent to
[3;7] or [9] or [10] or [11]
?
See above, maybe we can invent something shorter for the 'or' that works
for numbers and for dates so that we can leave out the brackets. It
can't be a space (as dates may contain spaces) nor a comma (as numbers
may contain commas).

> There is a problem about the range "(3; 5)" and an enumeration "(3, 5)":
> The range doesn't contain the edges (thus, for integers it would yield
> 4), while the enumeration *does* contain the edges.  This is
> inconsistent.  The better solution would be to enforce the usage of
> curly brackets (instead of others; see below) for enums, which make
> clear that an enumeration is a completely different beast.

I'm not seeing enumerations at all. We only have enumerations of
intervals. I had proposed the comma for this operation but that requires
brackets in all cases. Maybe we allow only the 'or' or come up with
something else not contained in numbers and dates.

So lets see a number as a degenerated interval with only one point. Once
you take that step you don't need curly braces and everything becomes
easier :-)

You don't ever want to match the empty set?
And for all other cases I'd consider it a degenerated interval:
(1;)
and a set is 
1 + 2

Maybe we want to drop the comma. Then we can leave off the brackets in
most cases.

Doesn't work for dates. And has never worked for integers. So please
lets drop whitespace for enumerations.

Fine

We don't have sets, just degenerate intervals [a], [b]
I don't see why we need a special syntax.
If we get rid of comma and space we don't need the brackets. Then we can
write:
a or b

Yes, but see above, we probably should leave out the
brackets/parenthesis in the general case. 

not needed.

Fine.

fine

fine.

We don't need set braces I think.

the latter case would mean
[a;b] or [c] if we see a simple number as a degenerate interval.

I don't think we need expressions in the generic sense. No multiple
parentheses. Just a set of intervals.

> - when an error occurs, a nice message (including the position)
>   explains it.
> 
> Now the compatibility topic.
> 
> We need a new syntax anyway; for numeric fields, the new syntax will be
> the *only* possibility to specify ranges.  To stay compabible, it *must*
> support whitespace enumerations for numbers.

Why would you need a new syntax for specifying ranges numbers????
There is no whitespace enumeration for numbers. Forget about IDs: They
are strings.

I don't think whitespace currently works for anything other than
strings.

Compatibility: When we treat a single number as a degenerate interval
we're backward compatible with numbers.

OK. Maybe we can find something different that doesn't break existing
syntax. Then we could leave out the braces in the general case
(especially for degenerate intervals or number-enumerations).

can you give a test-case/example of an expression with whitespace that
currently works? With integers?

OK.
So lets drop whitespace for numbers too. It has never worked.
So there are no stored queries with that syntax.

> For both cases (and for floats as well, of course),
> there are two possibilities they have in common:
> - the 'or' operator
> - inside curly brackets, the semicolon divides values, but changes
>   nothing about the nature of the enumeration-implying curly bracket.

Once you're using the semicolon for ranges I don't think we have a
compatibility problem. If a single number is treated as a degenerate
interval.

> I can't see a problem with my proposal of a configuration switch
> (msg3765).  The current functionality for the old date search syntax is
> there, and so are hopefully the unittests.  There will be unittests for
> the new syntax as well.  What else do you need?!

One syntax. Which passes the old unittests. Otherwise users will have to
change *all* their stored queries.
Roundup is deployed for large installations.
- Think about the python.org tracker
- Think about a customer of mine with >200 users and *lots* of stored
  queries.

But I think we can achieve that when using semicolon for ranges.

it's '. -1d' for yesterday at the same time as 'now'.

> 0:00, this is fine; nobody wants to change this.  The 2nd edge seems to
> be seldom used anyway; however, we should be as consistent between data
> types as possible, and thus a specification of "yesterday;today" should
> find *all* of yesterday and *all* of today (some date values can lie in
> the future).

There is no "yesterday" in roundup. And '.' is the current date/time
including seconds.
If you're referring to example date lets use a concrete example:
 2009-06-01;2009-06-02
includes all times after 2009-06-01 midnight (including midnight) up to
and *not* including 2009-06-02 midnight.
If we change that to an *inclusive range* we have only a problem with
midnight. I *think* we can live with that but lets ask Richard to
comment on this once we've agreed on a spec.

The alternative would be to make *all* ranges without explicit
parentheses/brackets compatible to the current date semantics, so that 

1;2
is the same as
[1;2)

I think so, too. But lets ask Richard. The proposal above (ranges
include the start but exclude the end of the interval if no explicit
parentheses/brackets are given) is also possible.

> Let's take the opportunity to do it right.  I'll implement it -- maybe
> with a little help with the dates --, including the unit tests, and
> everyone will like it; I promise ;-)

Fine. But lets start with a specification not an implementation.

I also want to do it right ...
Thats why I'm discussing it with you -- to get to a point where we can
present a proposal ...

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  Fax:   +43/2243/26465-23
Reichergasse 131                        www:   http://www.runtux.com
A-3411 Weidling                         email: office@runtux.com
osAlliance member                       email: rsc@osalliance.com

On Mon, Jul 13, 2009 at 06:37:47PM +0000, Tobias wrote:
> 
> As mentioned before, the purpose of the expression function is:
> development, prove of concept, testablity;  a small step before the SQL
> expression.  Furthermore, I'm not sure whether all of our database
> backends support expressions like "a BETWEEN x AND y" or "a IN (1, 2,
> 5)", and IIRC we store integers as CHARs;  reasons enough to use Python
> expressions for a first shot, I think.

Fine. You'll need a python expression for the non-SQL backends. I think
only anydbm is left of these.

On Mon, Jul 13, 2009 at 06:49:40PM +0000, Tobias wrote:
> 
> Tobias <tobias.herp@gmx.de> added the comment:
> 
> > For date values, whitespace can't be used this way, because Roundup's
> > date specs can contain whitespace.  (...) For both cases (and for
> > floats as well, of course), there are two possibilities they have in
> > common:
> > - the 'or' operator
> > - inside curly brackets, the semicolon divides values, but changes
> >   nothing about the nature of the enumeration-implying curly bracket.
> 
> Correction: Of course, we'll use (consistently) commas or 'or' for date
> enumerations.  As mentioned elsewhere, enumerations of float values
> don't make much sense;  thus I suggest to drop them.

See my comments: We only have enumeration of intervals. A number (be it
float or integer) is a degenerate interval that covers a single number.
It can be written as [n] but maybe we can come up with a syntax where we
can drop the brackets/parentheses in the general case.

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  Fax:   +43/2243/26465-23
Reichergasse 131                        www:   http://www.runtux.com
A-3411 Weidling                         email: office@runtux.com
osAlliance member                       email: rsc@osalliance.com

> I'm considering enumerations of numbers (1,2,3)
> to be enumerations of degenerate ranges (with a single point).

IMO, the more natural way would be to consider them sets (for our topic
here, I consider sets and enumerations equivalent).

We don't need to invent such a strange beast like a "degenerate range".
A set can have 0, 1, 2 or more members. It is a natural choice.

Btw, SQL "WHERE" supports:
- a BETWEEN x and y       (for [x; y] ranges)
- a IN (x, y, z, ...)     (for {x y z ...} sets

Sets do exist. We don't have to invent them. They are known to Python
and to SQL.  There is definitely no need to work around them.  This
would be as useful as a Brainfuck
(http://en.wikipedia.org/wiki/Brainfuck) implementation of Roundup.

> This makes the curly-brace syntax obsolete. We can specify it as
>   [1] or [2] or [3]
> I agree that this is not a very nice syntax.

Right. And unnecessary.

> IDs are *strings* in roundup.

Yes, but
- they *look* like integers
- they can only be strings which can be interpreted as integers
- they *should be* integers
- they are considered integers by users
- concerning search expressions, they should be considered integers
  by Roundup.

> That's why you can combine them with space!

That's not logic. Strings can *contain* space.

It would be very, very bad to have different expression syntax for
integers and ids.

> date expressions like '. -1d' can contain spaces.
> That rules out the space as a separator.

Wrong. That rules out the space as a separator *for dates*.
Roundup date expressions are very special beasts, and everyone will
understand the difference.

We *need* space as a separator for ids, and thus for integers.
We *would* consequently support it as a separator for floats if we
decided to support sets (and [x; y] ranges) of floats.

> Ah, curly braces for enumerations?

Yes, of course.  As far as search expressions are concerned,
enumerations are sets.  Have a look at
http://en.wikipedia.org/wiki/Set_theory :

"The union of {1, 2, 3} and {2, 3, 4} is the set {1, 2, 3, 4}."

Thus, we simply would support the mathematical notation of sets.
It will be very easily understood. Nobody will expect a C block or
Python dictionary in a search expression ;-)

For convenience, we would allow "{1 2 5}" as the equivalent notation for
"{1, 2, 5}" (which we must anyway because of the existing id search syntax).

Curly braces for sets are not only *a* solution; they are *the*
solution.  There is definitely no reason why we should try to avoid them.

Remember my proposal: All of
- "1 2 5"     (neede because of existing id search)
- "1, 2, 5"
- "{1 2 5}"
- "{1, 2, 5}" (the mathematical notation)
are equivalent.

Curly braces allow us to threat sets as sets, and to easily combine sets
with ranges:

  [3; 7] or {9, 10, 11}

is much more convenient and understandable than

  [3;7] or [9] or [10] or [11]

The idea is in the world. If we implement something new now and miss the
opportunity to support sets, complaints are sure to come.

To be continued in a next message, addressing the other topics...

Some recapitulation for the start:

- we agree to support ranges with "open interval" edges; otherwise
  a search for "[date1;date2)" (which is the current understanding of
  "date1;date2") won't be possible.
- thus, we consequently need to support
  "(a; b)"           <-->    a <  x <  b
  "[a; b]"           <-->    a <= x <= b
  "[a; b)"           <-->    a <= x <  b
  "(a; b]"           <-->    a <  x <= b
  and
  "(a;)"             <-->    a <  x
  "[a;)"             <-->    a <= x
  "(;b)"             <-->         x <  b
  "(;b]"             <-->         x <= b
  (maybe we can support things like "[;b]" as well, but I'm not sure
  whether we should)
- my proposal is to support commonly recognised set syntax for sets
  ("{1, 2, 5}"); *whereever possible*, separation of values can be
  done with whitespace.

Now for the compatibility and current date search issue.

I think we agree that, throughout *the new* search syntax, the default
nature of ranges (which are specified without braces and thus cannot be
combined with other ranges and sets) should be as consistent as possible.

But your proposal to make "[a; b)" the default behaviour would imply
that users searching for "1; 5" would get {1, 2, 3, 4}, but not 5.

For computer scientists, this might look logical, since they are used to
this kind of indexing e.g. for Python slices.  But for the usual guy
(and for this use case -- think of ids -- the usual computer scientist
as well) who issues a search request, it is not; we would expext to get
"5" as well, and rather assume that "5" doesn't exist.

Think of someone who is about to search for "2 5"; then he changes his
mind and decides, just to be sure, to check for the values inbetween as
well: he changes his search expression to "2; 5".  This is the kind of
things which should work as expected.

The current date syntax is inappropriate for numbers; the open-interval
edge default was fixed with past-only dates in mind.  Most date fields
won't ever be set to a future value, and so this was accepted.  But it
is not a good idea to be continued in the future and carried over to
numbers and ids.

Question: is it currently possible to search for "-1d; ." and get all
values in the 48-hour range of yesterday and today?  IMO, this search
should be possible.

We can leave it to the admin whether and when to switch date search
syntax to the new logic (we should), and we can provide a simple script
to convert all old search expressions in stored searches to new ones (we
should, as well).

Just to clarify, answering to what you have written:
> One syntax. Which passes the old unittests. Otherwise users
> will have to change *all* their stored queries.

- it is not necessary for the *new* syntax to pass the *old*
  unittests; the *old* syntax must pass the *new* unittest.
  Otherwise we couldn't introduce a new syntax at all
  (which is what we are about to do).
- the old syntax will continue to be valid, and perhaps --
  "." could keep its meaning of current date and time --
  would yield the same results.

> Roundup is deployed for large installations.
> - Think about the python.org tracker
> - Think about a customer of mine with >200 users
>   and *lots* of stored queries.

Yes; both cases are covered nicely by my proposal about a switch and a
conversion script.

> But I think we can achieve that when using semicolon for ranges.

Of course; that's what we do: we use semicolon for ranges.

We *shouldn't* tie us to a default understanding of ranges which was --
given past-only values! -- suitable for dates, and bugger up the numeric
and id search.  Days are ranges by nature (24-hour-ranges), and numbers
and ids are not.

> If you're referring to example date lets use a concrete example:
>  2009-06-01;2009-06-02
> includes all times after 2009-06-01 midnight (including midnight)
> up to and *not* including 2009-06-02 midnight.

If I understand this right, "midnight" is 0:00, and thus the specified
range spans 24 hours.

- What if someone seeks "2009-06-01"; wouldn't this
  find the same range? (IMO, it should!)
- Is this really an issue for stored queries?
  (while we can convert those:)
  I'd reckon most of these search for relatively specified
  ranges like "-3d;" or "-1d;." which would make the whole problem
  an non-issue.

We have the chance to introduce a more convenient semantics. "." will
continue to mean "now", including the time.

For ranges, we should consider days atomic (unless time specified).
Carrying over our range considerations from numbers to atomic days, I get:
- "[day1; ..."   means   *including* day1, thus "day1, 0:00" <= x
- "... ;day2]"   means   *including* day2, thus x <= "day2, 24:00"
                      (or, with databases in mind, x <= date(day2))
- "(day1; ..."   means   *excluding* day1, thus date(day1)   <  x
- "... ;day2)"   means   *excluding* day2, thus x  < "day2, 0:00"
                      (or, with databases in mind, x < date(day2))

Currenty, if I want to seek for records from day1 to day2, I *must*
specify day1 and day2+1; for scripts, this requires date calculations.

To seek for the records of 2009-06-01 and 2009-06-02, I must currently
specify
  "2009-06-01;2009-06-03"
This is frankly a PITA.  Given the choice, I'm sure almost everyone
(including the computer scientists) would prefer to use inclusive ranges
for dates.

For convenience, we could have a symbol 'now' which is equivalent to "."
(there is an SQL function "NOW()").  And we could have a symbol "*",
meaning (all of) today.

The we could seek for...
- ;.         (like current ";." -- different in theory,
             but not in practice)
- ;now       (the same; more explicitly saying "this includes time")
- ;*         (all up to today, 24:00)
- (;*)       (all before today, 0:00; remember atomic days)
- ;-1d       (the same, expressed as "all up to yesterday")

For sets of dates, there are several possibilities. I already called the
comma an 'or' operator; it can be used without any problem e.g. to make

  {1, 2, 5}, [9;)

the same as

  {1, 2, 5} or [9;)

Since a set can consist of a single element, date sets can be noted as
unions of sets:

  {date1} or {date2}
  {date1}, {date2}
or simply (dates can't contain commas, can they?):
  {date1, date2}

There is no problem with

  date1, date2

(no semicolon there, thus it can't be a range), and we can even support

  date1 or date2

... which some might prefer, being an even more human-readable variant.

(We don't necessarily need to support the 'or' inside the braces, but it
is logical: "{date1, date2}" -- or "{date1 or date2}" -- is the same as
"{date1} or {date2}")

>> Let's take the opportunity to do it right. (...)
>
> Fine. But lets start with a specification not an implementation.

That's why we are discussing the specification ;-)
(Of course, "it can't be done" would be a killer argument. But it
clearly *can* be done).

On Tue, Jul 14, 2009 at 08:47:52AM +0000, Tobias wrote:
> 
> Tobias <tobias.herp@gmx.de> added the comment:
> 
> > I'm considering enumerations of numbers (1,2,3)
> > to be enumerations of degenerate ranges (with a single point).
> 
> IMO, the more natural way would be to consider them sets (for our topic
> here, I consider sets and enumerations equivalent).
> 
> We don't need to invent such a strange beast like a "degenerate range".

Its already there. Try to specify a search for a single date into the
search for of dates. You'll find all occurrences of that date. These
*are* already such strange beasts.

Fine with me. Just think about sets of ranges, not just sets of numbers
and you understand what I mean. Then there is no conceptual difference
between

1 or 2 or 3

and

1 or 2 or 3 or 4;5

(I'm using "or" not comma here to show that we can leave out the
brackets in the general case if the syntax isn't ambiguous.)

or degenerate ranges
- a = x

so we can translate the thing above to
(a=1 or a=2 or a=3 or (a>=4 and a<=5))
or use the "between" operator or use "in" for separate ranges. Thats the
*backend*. I'm arguing about the frontend.

> Sets do exist. We don't have to invent them. They are known to Python
> and to SQL.  There is definitely no need to work around them.  This
> would be as useful as a Brainfuck
> (http://en.wikipedia.org/wiki/Brainfuck) implementation of Roundup.

I have no problem of using sets in python and/or sql to implement the
mini-query language for numbers and dates. I just don't want to force
that concept on the users of that mini-language.

No. And I've already mentioned that we can simplify it in the general
case if we can leave out the brackets.

> > IDs are *strings* in roundup.
> 
> Yes, but
> - they *look* like integers
> - they can only be strings which can be interpreted as integers
> - they *should be* integers
> - they are considered integers by users
> - concerning search expressions, they should be considered integers
>   by Roundup.
> 
> > That's why you can combine them with space!
> 
> That's not logic. Strings can *contain* space.

No. For the search engine we only search for words. These cannot
contain spaces.

Why? I'm more concerned with
- providing backward compatibility
- making it easy for users
- treating integers and dates alike

> > date expressions like '. -1d' can contain spaces.
> > That rules out the space as a separator.
> 
> Wrong. That rules out the space as a separator *for dates*.

See above. I don't want to invent two different syntaxes for dates and
numbers.

> Roundup date expressions are very special beasts, and everyone will
> understand the difference.

I don't think so. And I don't want two different code-bases for dates
and ints.

No.

I still can't understand why you want to support sets of numbers *and*
sets of ranges. Is there a good technical argument for that -- syntax
left aside for a moment? I think we can come up with a good syntax that
covers all.
Are there search terms that use your number-sets which can express
queries you cannot express in terms of sets of ranges? Mathematically,
leave aside the syntax for a moment. (and I think the answer to that
question is 'no').
So I ask for keep-it-simple: We don't need to introduce sets of numbers
if we already have sets of ranges.

Just for the record: I'm holding a Phd in computer science, so don't try
to teach me set theory. 
On the other hand I've found that the users of roundup I'm working with
will have a hard time understanding set theory. Thats why I argue that we
should not put this into our mini-query-language if it can be avoided.

> "The union of {1, 2, 3} and {2, 3, 4} is the set {1, 2, 3, 4}."
> 
> Thus, we simply would support the mathematical notation of sets.
> It will be very easily understood. Nobody will expect a C block or
> Python dictionary in a search expression ;-)

Note that I still think that set-queries of numbers are a special case.
You usually want ranges. Or sets of *ranges*. So lets make the number a
special case of the range and come up with a convenient syntax.

No we don't need to "anyway". And the more I'm thinkig about it I tend
to abolish the comma too. That would give us a form where we can leave
out the brackets/parentheses in the general case.

I don't think we need that. If we use the 'or' syntax we can leave out
the brackets and can rewrite that as

1 or 2 or 5

My syntax is more convenient:

3;7 or 9 or 10 or 11

See above. If we only allow 'or' (and not ',' or space) we can leave out
the brackets.

We already have sets of ranges. I'm arguing against a separate
set-of-nubmers concept.

Ralf

>> We don't need to invent such a strange beast
>>  like a "degenerate range".

> Its already there. Try to specify a search for a single date
> into the search for of dates. You'll find all occurrences of that
> date. These *are* already such strange beasts.

No, not really. Consider the SQL function DATE, which returns the date
part of a datetime field.  This date part is a *value* which can be
compared by equality; it doesn't need to be compared to a range.

SQL:
  WHERE DATE(thedatefield) = date1;     -- single value
  WHERE DATE(thedatefield)
               BETWEEN date1 AND date2; -- inclusive range

(IMO, we should employ such structures whenever possible, as it allows
us to benefit from database engine optimisations)

Of course, this date component is common to a *range* of datetime
values; nevertheless, a date can very well be considered a *value*, when
it comes to search expressions.

(Thanks a lot for this demur, which reminded me of the DATE function ;-)

> Just think about sets of ranges, not just sets of numbers
> and you understand what I mean.

I think I *do* understand very well what you mean:  You'd like exactly
*one* concept for search expressions, and you think this should be
ranges; you even go so far to treat single values of "degenerated
ranges", just to save this idea. IMHO, you go overboard here:

There *are* two fundamental concepts for searching numbers and dates:
- the search for ranges, like already implemented for dates
- the search for sets (or alternatives), like implemented for integers
  and ids.
(for strings, there are regular expressions as well, thus it is a
perfectly normal thing to have different search possibilities for
different types of data)

Everyone understands that.  Most will have felt the need to search for
both a set of known values and a range in one request.  Very few will
have thought of the known values as of "degenerated ranges".

> Then there is no conceptual difference between
> 
> 1 or 2 or 3
> (...)

... no problem with that; we have no differences about the
exchangeability of commas and 'or' keywords ...

> and
> 
> 1 or 2 or 3 or 4;5

I have implemented such a liberal thing, but I went from it.  As it
looks now, it looks nice; but consider

  1, 2, 3, 4; 5

or

  1, 2, 3; 4, 5

These expressions could be interpreted as well, but I consider it
clearly better to demand parentheses here, at least for the range.

The Zen of Python (http://www.python.org/dev/peps/pep-0020/) says:

 (2) Explicit is better than implicit.
 (7) Readability counts.
(10) Errors should never pass silently.

Explicit is better than implicit: better denote a set explicitly if you
have one (and you *do* have a set values -- {1, 2, 3} -- no matter what
you call them).

Readability counts: the curly braces of sets are a widely known syntax,
and it is perfectly readable; no need to mull over precedences.  Braces
are A Good Thing (tm) because they clarify things a lot; for Python
expressions, it has often been recommended to use braces to make the
excecution logic explicit, even when they don't change it.
(we are not talking about LISP here ;-)

Errors should never pass silently:  An explicit syntax would enable us
to tell the user about his defective search expression; if we don't, we
cause inconvenience, because it will happen much more often that a user
doesn't find what (s)he seeked just because the search expression worked
in another way than (s)he thought.

> No. And I've already mentioned that we can simplify it
> in the general case if we can leave out the brackets.

You are quite obsessive about avoiding brackets, are you?

Remember, we don't invent something new here; instead, we'll support a
widely known and easily understood syntax.


>> That's not logic. Strings can *contain* space.
>
> No. For the search engine we only search for words.
> These cannot contain spaces.

Do you really want ordinary users to think about how the roundup backend
implements indexes which certainly look like integers?!


> Why? I'm more concerned with
> - providing backward compatibility

For your "1 or 2 or 3 or 4;5" example, backward compatibility is an
non-issue: combinations of ranges and sets (or, like you'd put it,
several ranges) are currently not supported.

> - making it easy for users

I don't think you make it easy for users if you allow a syntax which can
easily understood wrong, and thus have them think e.g. the issue they
seeked doesn't exist.  Explicit is better than implicit; the 2nd most
important point in the list in the Zen of Python -- a language we all
love for expressiveness and readability!

> - treating integers and dates alike

You are obsessive about treating integers and dates alike; but you are
willing to treat integers and ids differently.  This Is Evil, because --
from a user's point of view -- ids *are* integers, and it is a Very Very
Bad Idea to treat them differently.

* For ids, separation of values by space is supported.
* This is required for backward compatibility (your 1st point)
* To treat integers and ids alike, space support *must* be added
  for integers as well.
* Since spaces *must* be supported for integers and *can't* be
  supported for dates, it is *impossible* to treat dates and integers
  perfectly alike (your 3rd point).

> I don't want to invent two different syntaxes for dates and
> numbers.

You don't need to; I have done it already ;-)

Calm down -- the difference is necessary but minimal.  And it's
well-grounded: by compatibility needs, and by the nature of the data.
(Don't blame me for Roundup's date expression syntax...)

> I still can't understand why you want to support sets of numbers
> *and* sets of ranges.

I want to support sets of numbers because they are simple, expressive,
useful and well-known, and because both Python and SQL support them
(that's 6 reasons).

As for "sets of ranges", see it the other way round; don't call them
"sets of ranges" but a bunch of ranges which are connected by 'or'
keywords (or by commas, if you like, meaning the same).

It is possible to consider the elements of a set or-connected as well:
when checking for

  x in {a, b, c},

this is equivalent to

  x==a or x==b or x==c

Thus, from the search expression point of view, there is really no
difference between a set and this number of or-connected comparison terms.

> So I ask for keep-it-simple: We don't need to introduce sets
> of numbers if we already have sets of ranges.

But people *know* sets of numbers!

> don't try to teach me set theory.

I didn't.  I submitted the link simply to demonstrate that the curly
braces are a well-known set syntax and thus a natural choice.

> My syntax is more convenient:
>
> 3;7 or 9 or 10 or 11

Well, that's of course a matter of taste.  

I like "[3;7] or {9, 10, 11}" much better, and I invoke the Zen of
Python (explicit is better than implicit):

When the range is written down explicitly, *including* the braces, it is
almost impossible to misunderstand the expression.

Some people love Perl for allowing very short (but barely readable)
programs.  I don't.
The Python way is to favour expressiveness to brevity.

Brevity is not everything. And, citing Einstein:
Make everything as simple as possible, but /not simpler/.

Binary numbers are "simpler" than decimals because they get along with
two symbols instead of 10.  There are kinds of simplicity which suit
computers better than humans.  For search expressions, we should suit
the needs of humans.

The curly braces don't impact simplicity very much, but they are
expressive and help avoid errors, and this is A Good Thing (tm).

We have or-connected expressions; think of sets of numbers as an
expressive shortcut for or-connected values. From the user's point of
view, there *are* are no different set concepts for numbers and ranges.

On 14/07/2009, at 6:35 AM, Ralf Schlatterbeck wrote:
> I think so, too. But lets ask Richard. The proposal above (ranges
> include the start but exclude the end of the interval if no explicit
> parentheses/brackets are given) is also possible.

FWIW I'm really not following the discussion closely enough to make  
any statement either way, except my last email (which inadvertently  
pointed out an error even though I was simply indicating mystification  
at the expression as a whole :)

I trust you two to nut something out and come up with something  
workable and agreeable between you. Perhaps a summary of the proposal  
sent to roundup-users might be a good idea, once you've solidified  
your intentions.

On 15/07/2009, at 4:23 PM, Richard Jones wrote:
> On 14/07/2009, at 6:35 AM, Ralf Schlatterbeck wrote:
>> I think so, too. But lets ask Richard. The proposal above (ranges
>> include the start but exclude the end of the interval if no explicit
>> parentheses/brackets are given) is also possible.
>
> FWIW I'm really not following the discussion closely enough to make
> any statement either way, except my last email (which inadvertently
> pointed out an error even though I was simply indicating mystification
> at the expression as a whole :)
>
> I trust you two to nut something out and come up with something
> workable and agreeable between you. Perhaps a summary of the proposal
> sent to roundup-users might be a good idea, once you've solidified
> your intentions.

Haha, that's what I get for not reading the entire backlog of mail  
before sending this! Ignore the second para :)

A recapitulation for all who just jump in. First, what I think we agree
about:

(1) All existing search syntax must continue to work
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This is:
  date1;date2
  date1;
  ;date2
  from date1 to date2
  date1 to date2
  from date1
  to date2
  # (date specs can contain spaces in Roundup; this is important)

--> The semicolon ";" is the "range operator";
    it can be replaced by 'to'

  1, 2, 5    # for integer values and ids
  1 2 5      # for ids

--> the comma ',' separates enumerated single values

As far as possible, all types of data should be handled likewise
in search expressions.

This is not possible to the full extent, since ids can be separated
by spaces, but dates -- which can /contain/ blanks -- can't.

From the user's point of view, ids *are* integers; therefore, they
should be treated identically when it comes to search expressions.

Ranges can be specified as "open" and "closed" intervals, following the
"set builder notation"
<http://en.wikipedia.org/wiki/Interval_(mathematics)#Excluding_the_endpoints>

  [date1;date2)   --> date1 <= x < date2

(this is what date ranges currently work like)

  [a; b]          --> a <= x <= b
  (a; b)          --> a <  x <  b
  (a; b]          --> a <  x <= b

It should be possible to combine range expressions:

  [;v1] or [v2;]
  [;v1], [v2;]

This would allow to seek values x with

  (x <= v1) or (x >= v2)

which can't be achieved with [v1; v2] if v1 > v2.

Ralf calls this a set of ranges.
I call it or-connected ranges.

Furthermore, it should be possible to specify in one expression
/both/ a number of known values /and/ ranges; how this is should look
like, is subject to our discussion.
Ralf calls this number of values "degenerated ranges" (which contain a
single element each only).
I call this enumeration of values a set.

Finally, before I make a point here to give Ralf the oppurtunity to
countersign the text so far:

We agree that it should be possible to specify float values with decimal
commas.  Many Roundup users like in countries live in countries which
use decimal commas, where the decimal point is known to computer geeks
only (Germany, for one).

Of course, this seems to conflict with the comma as an enumeration
indicator. This might can be solved when using a special tokeniser for
float searches (which is a small and perfectly testable piece of code),
which could require the comma /separator/ not to be followed by a digit.  
But the problem could be a non-issue anyway, since float arithmetics is
unprecise, and it might make sense to disallow both enumerations and
closed intervals ("[a;b]" --> a <= x <= b) for floats, since comparison
for equality is unrelyable for them.

I forgot to mention that we agree /not/ to support an 'and' operator,
because
- it is not necessary; everone can work out the result himself
- the popular meaning of 'and' contradicts the mathematical meaning.
Thus, an 'and' operator would cause confusion.

Now to the combination of sets ("sets of numbers", like Ralf calls it)
and ranges.

Ralf would like to support things like

  1 or 2 or 3 or 4;5

and calls it a "set of ranges".  I take into consideration that

- the 'or' keyword is interchangeable with the /optional/ comma
- we should /not/ rely on a blank /not/ following the semicolon;

thus, something like
  1 2 3 4; 5
would need to be supported as well.

My opinion is:
- the "1 or 2 or 3" part -- or "1 2 3", "1, 2, 3" -- is a /set/.
- a set is a totally different beast than a range:
  * a set has /members/ while a range has /limits/
  * thus, a set will always contain the listed values;
    a (a,b) range won't
  * a set can by its very nature contain 1, 2 or more elements;
    it doesn't need to "degenerate" to have one member only.

And, most important, I invoke the Zen of Python, "explicit is better
than implicit":

When sets and ranges are connected, it is better to demand parentheses.
This prevents users to request searches they didn't mean to, and to
trouble their heads about the results (or simply not find what they seeked).

Thus, my suggestion for this combination is:

  {1, 2, 3} or [4; 5]
  {1 2 3} or [4; 5]
  {1 2 3},[4;5]          # shortest version

Maybe I could live with

  1 or 2 or 3 or [4; 5]

as well; but in any case the /range/ must be made explicit in such
combinations.

Ralf fights grimly to avoid the set braces, because he considers it too
complicated, and he suspects them to impact simplicity.  To him, search
expressions consist of sets of ranges.

I say:
- sets are a concept known to everybody
- sets are supported by Python as well as SQL
- the curly-braces set notation is well-known (see
  <http://en.wikipedia.org/wiki/Set_theory>)
- sets are a natural choice for listed known values,
  including (at least when combined) single values

And, please, don't get confused about the examples from msg3765; they
reflect an obsolete stage of discussion. At the time I implemented this
module, I wasn't aware of the already existing usage of the comma; I
have always used blanks to separate ids seeked for.  At the most, take
my module as a proof of concept:  Yes, it can be done.

Now the date problem, and the default nature of ranges.

*Currently*, the only existing ranges have an /including/ (if any) lower
limit, and an /excluding/ (if any) upper limit.

This is the result of such ranges being supported for date values only,
and we simply compare to the 0:00 values of the given days:

  2009-07-13;2009-07-15

finds the records for /two/ days only instead of three, because the
field values x are simply compared to the 0:00 values:

  2009-07-13.00:00 <= x < 2009-07-15.00:00

Thus, to find the records of days date1 to date2, you currently must add
1 to the upper limit:

  date1;date2+1

(of course, you must calculate "date2+1" yourself before sending the
request).

This is inconvenient.
For numbers, this is not only inconvenient but inappropriate:

Consider a user who is about to seek

  8 13

(the set {8, 13}, like currently supported for ids)

Then (s)he thinks about it and decides to better /include/ the
intermediate values:

  8; 13

(inserting the range operator).

Of course, (s)he will expect the result to /include/ the specified upper
limit.  This is the kind of things which must work as expected.
Therefore, ranges must be inclusive by default.

This changes the result of some existing stored queries:  those which
specify a day (especially /without/ a time specification) as the upper
limit.  Therefore, *for dates* this changed default can't be activated
automatically:  We need

- a configuration switch to allow the tracker admin to choose
  whether and when to active the new default
- a conversion script to change the affected stored queries.

BTW, this is one more argument to demand explicit range brackets when
the existing range syntax is stepped beyond:
- the incompatibility affects the /default/ nature
  of /date/ ranges only
- even with the "magic lever" not thrown, it should be possible for
  users to use the explicit syntax already, e.g. to seek
  [date1;date2]
  and get all values between date1 and date2, /including/ date2

Finally, SQL supports /inclusive/ -- but not /exclusive/! --
date ranges very naturally:

  SELECT ... WHERE DATE(fieldname) BETWEEN date1 AND date2;
  SELECT ... DATE(fieldname) = date1

Proposal
~~~~~~~~

Here is my proposal; hopefully Ralf can live with it, too:

- the semicolon ";" (or 'to') is the range operator, like in
    date1;date2
- ranges can have inclusive or exclusive limits:
    [a; b]          --> a <= x <= b
    [a; b)          --> a <= x <  b
    (a; b)          --> a <  x <  b
    (a; b]          --> a <  x <= b
  (to continue support for existing date searches,
  and important for floats)

- a simple, single range can be specified without brackets:
    a; b

- ranges without brackets are by default /inclusive/ ("[a;b]")
  except (for compatibility) for date ranges before the magic
  lever is thrown

- the comma "," (or 'or') is the -- optional! -- set operator:
    a, b            --> x in (a, b,)
    a or b          --> x in (a, b,)     (same as "x==a or x==b")
    a b             --> x in (a, b,)

- when sets and ranges are combined, ranges *must* be put in braces.

  * Preferred syntax:
      {a, b} or [c; d]    --> x in (a, b) or c <= x <= d
  * Accepted syntax (examples):
      a or b or [c;d]
      a, b, [c;d]
      {a b} [c;d]         # (not for dates)
      a b [c;d]           # (not for dates)
    or, queer but possible:
      {a} {b} [c;d]

* search support for dates, integers, ids and floats works
  as similar as possible:
  - when it comes to search expressions,
    ids are treated exactly like integers
  - for dates, separation of alternatives by blanks can't be
    supported; but for all data types -- which /can't/ contain
    blanks -- it is.
  - comparison of floats for equality is /unreliable/ by nature;
    thus, sets and /inclusive/ ranges are disallowed for float fields.
    After a short while, floats might be the only case where
    /exclusive/ ranges are actually used:
      (0;)              --> 0 < x
    (Ralf's initial need, probably concerning floats)

* for convenience, float values can be specified using decimal commas
  as well as decimal points:
    (0.0;)            --> 0.0 < x
    (0,0;)            --> 0.0 < x
    (0.;)             --> 0.0 < x    # just to please geeks
    (0,;)             --> 0.0 < x    # (perhaps; not recommended)
    0,0               --> ERROR: sets or single values not supported

Philosophy (the "Zen of Searching" ;-):
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Support what works already.
- But don't stick with unappropriate defaults.
- As far as possible, do similar things similarly.
- But when necessary, threat different things differently.
- Predictibility is more important than simplicity.
- Better raise an error than perform an unintended search.
- Explicit is better than implicit.

On Tue, Jul 14, 2009 at 09:30:27PM +0000, Tobias wrote:
> 
> >> We don't need to invent such a strange beast
> >>  like a "degenerate range".
> 
> > Its already there. Try to specify a search for a single date
> > into the search for of dates. You'll find all occurrences of that
> > date. These *are* already such strange beasts.
> 
> No, not really. Consider the SQL function DATE, which returns the date
> part of a datetime field.  This date part is a *value* which can be
> compared by equality; it doesn't need to be compared to a range.

You're mixing up implementation is SQL and the input syntax. You
currently can specify a range
date1;date2
or a single date
date1

these are roundup.date.Range objects
The translation to SQL is something completely different.

> > Just think about sets of ranges, not just sets of numbers
> > and you understand what I mean.
> 
> I think I *do* understand very well what you mean:  You'd like exactly
> *one* concept for search expressions, and you think this should be
> ranges; you even go so far to treat single values of "degenerated
> ranges", just to save this idea. IMHO, you go overboard here:

I'm just taking the current implementation which works fine.
It's a matter of input syntax to make it usable.

> 
> There *are* two fundamental concepts for searching numbers and dates:
> - the search for ranges, like already implemented for dates
> - the search for sets (or alternatives), like implemented for integers
>   and ids.

I agree. I just don't want two diffenent sorts of set, sets of numbers
and sets of ranges.
Do you agree that we *need* sets of ranges?
Do you agree that your proposal generates two distinct types of sets (of
ranges and of integers)?

For SQL as far as I know there is no generic regex search in roundup.

> > Then there is no conceptual difference between
> > 
> > 1 or 2 or 3
> > (...)
> 
> ... no problem with that; we have no differences about the
> exchangeability of commas and 'or' keywords ...
> 
> > and
> > 
> > 1 or 2 or 3 or 4;5
> 
> I have implemented such a liberal thing, but I went from it.  As it
> looks now, it looks nice; but consider
> 
>   1, 2, 3, 4; 5
> 
> or
> 
>   1, 2, 3; 4, 5
> 
> These expressions could be interpreted as well, but I consider it
> clearly better to demand parentheses here, at least for the range.

If find these clear and concise.

> The Zen of Python (http://www.python.org/dev/peps/pep-0020/) says:
> 
>  (2) Explicit is better than implicit.
>  (7) Readability counts.

I thinks thats a heavy argument *against* curly braces.

> (10) Errors should never pass silently.

> 
> Readability counts: the curly braces of sets are a widely known syntax,
> and it is perfectly readable; no need to mull over precedences.  Braces
> are A Good Thing (tm) because they clarify things a lot; for Python
> expressions, it has often been recommended to use braces to make the
> excecution logic explicit, even when they don't change it.
> (we are not talking about LISP here ;-)

Why do you think python has left out the braces for statement blocks?

Curly braces not brackets, yes.

> Remember, we don't invent something new here; instead, we'll support a
> widely known and easily understood syntax.

I'm thinking of users of some trackers I maintain.

> > - making it easy for users
> 
> I don't think you make it easy for users if you allow a syntax which can
> easily understood wrong, and thus have them think e.g. the issue they
> seeked doesn't exist.

Hmm. So our tastes are different there. For me the example

1 or 2 or 3 or 4;5 or 6

is perfectly understandable. Note that meanwhile I think we should *not*
use commas or spaces instead of 'or'.

> > - treating integers and dates alike
> 
> You are obsessive about treating integers and dates alike; but you are
> willing to treat integers and ids differently.  This Is Evil, because --
> from a user's point of view -- ids *are* integers, and it is a Very Very
> Bad Idea to treat them differently.

You are arguing that "explicit is better than implicit" but want to make
a space and implicit indicator of "or". I can't follow that.

And, yes, I want to treat dates and integers (all things where range
searching makes sense) alike. Thats sound software engineering practice.

So we might want to fix that. But that would be major surgery because
you'd have to change the representation of ids -- something at the very
core of roundup. I also don't think that in the shortterm you will be
able to apply our little query-language for ranges to IDs. IDs are
strings in SQL currently.

No. For integers no spaces are currently allowed/used.

No.

No.

:-(

> Calm down -- the difference is necessary but minimal.  And it's
> well-grounded: by compatibility needs, and by the nature of the data.
> (Don't blame me for Roundup's date expression syntax...)

The difference is not minimal since you're obsoleting all stored queries
of all users.

> > I still can't understand why you want to support sets of numbers
> > *and* sets of ranges.
> 
> I want to support sets of numbers because they are simple, expressive,
> useful and well-known, and because both Python and SQL support them
> (that's 6 reasons).

You can still compile our expressions to something using sets of numbers
in python and/or SQL. My point is the input syntax should be easy to use
and understand with a minimum of different notations.

> As for "sets of ranges", see it the other way round; don't call them
> "sets of ranges" but a bunch of ranges which are connected by 'or'
> keywords (or by commas, if you like, meaning the same).

Huh? What makes it different from a set by calling it something else?

No.

> > My syntax is more convenient:
> >
> > 3;7 or 9 or 10 or 11
> 
> Well, that's of course a matter of taste.  

Yes. I thinks so.

Simple is better than complex
Beautiful is better than ugly
Readability counts
Special cases aren't special enough to break the rules
There should be one-- and preferably only one --obvious way to do it
If the implementation is hard to explain, it's a bad idea

all argue for an easier syntax.

Except if the concept of sets has to be mastered first.

But you're just trying to introduce more obscure symbols into the query
language, are you? Perl is full of these.

There aren't any expressions that can be formulated in your syntax that
can't be formulated in my syntax. At least you haven't come up with an
example.
There should be one-- and preferably only one --obvious way to do it

You didn't make it as simple as possible.

They impact simplicity very much in my opinion.

Then lets drop the sets of numbers when they're not needed from the user
perspective.

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  Fax:   +43/2243/26465-23
Reichergasse 131                        www:   http://www.runtux.com
A-3411 Weidling                         email: office@runtux.com
osAlliance member                       email: rsc@osalliance.com

We don't agree on these yet.
I think we should use
1 or 2 or 5 *explicitly* not use the space *and* the comma for denoting
"or".

I don't agree, since one of the requirements was that the comma should
be used as a decimal separator. That excludes it as a 'set separator'.
> 
> As far as possible, all types of data should be handled likewise
> in search expressions.
Yes.

Leave out IDs. They are strings. Changing that would be major surgery.

No, see above.

> Ranges can be specified as "open" and "closed" intervals, following the
> "set builder notation"
> <http://en.wikipedia.org/wiki/Interval_(mathematics)#Excluding_the_endpoints>
> 
>   [date1;date2)   --> date1 <= x < date2
> 
> (this is what date ranges currently work like)
> 
>   [a; b]          --> a <= x <= b
>   (a; b)          --> a <  x <  b
>   (a; b]          --> a <  x <= b
> 
> It should be possible to combine range expressions:
> 
>   [;v1] or [v2;]
>   [;v1], [v2;]
> 
> This would allow to seek values x with
> 
>   (x <= v1) or (x >= v2)
> 
> which can't be achieved with [v1; v2] if v1 > v2.
> 
> Ralf calls this a set of ranges.
> I call it or-connected ranges.
> 
> Furthermore, it should be possible to specify in one expression
> /both/ a number of known values /and/ ranges; how this is should look
> like, is subject to our discussion.
> Ralf calls this number of values "degenerated ranges" (which contain a
> single element each only).
> I call this enumeration of values a set.
> 
> Finally, before I make a point here to give Ralf the oppurtunity to
> countersign the text so far:
> 
> We agree that it should be possible to specify float values with decimal
> commas.  Many Roundup users like in countries live in countries which
> use decimal commas, where the decimal point is known to computer geeks
> only (Germany, for one).
> 
> Of course, this seems to conflict with the comma as an enumeration
> indicator. This might can be solved when using a special tokeniser for
> float searches (which is a small and perfectly testable piece of code),
> which could require the comma /separator/ not to be followed by a digit.  

I don't think we should introduce this ambiguity into the parser.

> But the problem could be a non-issue anyway, since float arithmetics is
> unprecise, and it might make sense to disallow both enumerations and
> closed intervals ("[a;b]" --> a <= x <= b) for floats, since comparison
> for equality is unrelyable for them.

You don't want to do special cases for special numbers here:
"Special cases aren't special enough to break the rules"

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  Fax:   +43/2243/26465-23
Reichergasse 131                        www:   http://www.runtux.com
A-3411 Weidling                         email: office@runtux.com
osAlliance member                       email: rsc@osalliance.com

On Wed, Jul 15, 2009 at 08:35:34AM +0000, Tobias wrote:
> 
> Tobias <tobias.herp@gmx.de> added the comment:
> 
> I forgot to mention that we agree /not/ to support an 'and' operator,
> because
> - it is not necessary; everone can work out the result himself
> - the popular meaning of 'and' contradicts the mathematical meaning.
> Thus, an 'and' operator would cause confusion.

Yes.

> Now to the combination of sets ("sets of numbers", like Ralf calls it)
> and ranges.
> 
> Ralf would like to support things like
> 
>   1 or 2 or 3 or 4;5
> 
> and calls it a "set of ranges".  I take into consideration that
> 
> - the 'or' keyword is interchangeable with the /optional/ comma

I've repeatedly said that -- since the comma should be a decimal comma
too, I DO NOT WANT the comma to mean "or".

Same for blank. I DO NOT WANT a space to implicitly mean "or".

Right.

No.

> My opinion is:
> - the "1 or 2 or 3" part -- or "1 2 3", "1, 2, 3" -- is a /set/.
> - a set is a totally different beast than a range:
>   * a set has /members/ while a range has /limits/
>   * thus, a set will always contain the listed values;
>     a (a,b) range won't
>   * a set can by its very nature contain 1, 2 or more elements;
>     it doesn't need to "degenerate" to have one member only.
> 
> And, most important, I invoke the Zen of Python, "explicit is better
> than implicit":

If you see ranges as things that can contain just one number (as it is
currently implemented in roundup.date.Range) you won't need the sets.

Zen of Python:
Beautiful is better than ugly
Simple is better than complex
Readability counts
Special cases aren't special enough to break the rules
There should be one-- and preferably only one --obvious way to do it
If the implementation is hard to explain, it's a bad idea

So lets drop sets of numbers.

Or to fix a misconception: We don't need sets of numbers for full
expressiveness.

> Thus, my suggestion for this combination is:
> 
>   {1, 2, 3} or [4; 5]
>   {1 2 3} or [4; 5]
>   {1 2 3},[4;5]          # shortest version
> 
> Maybe I could live with
> 
>   1 or 2 or 3 or [4; 5]
> 
> as well; but in any case the /range/ must be made explicit in such
> combinations.

I find that this syntax violates the above-mentioned "Zen of Python"
guidelines. All of them. We don't need explicitness for a special case.

Yes.

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  Fax:   +43/2243/26465-23
Reichergasse 131                        www:   http://www.runtux.com
A-3411 Weidling                         email: office@runtux.com
osAlliance member                       email: rsc@osalliance.com

On Wed, Jul 15, 2009 at 09:18:40AM +0000, Tobias wrote:
> 
> Now the date problem, and the default nature of ranges.
> 
> *Currently*, the only existing ranges have an /including/ (if any) lower
> limit, and an /excluding/ (if any) upper limit.
> 
> This is the result of such ranges being supported for date values only,
> and we simply compare to the 0:00 values of the given days:
> 
>   2009-07-13;2009-07-15

or use times explicitly
2009-01-13.10:55:20;2009-07-15.11:23:17 

And I'm currently not sure if we shouldn't make the default interval
(without brackets/parentheses) to model the current semantics of dates:
Include the lower but exclude the upper bound.

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  Fax:   +43/2243/26465-23
Reichergasse 131                        www:   http://www.runtux.com
A-3411 Weidling                         email: office@runtux.com
osAlliance member                       email: rsc@osalliance.com

Let me put my proposal here, too:

Proposal
~~~~~~~~

The following examples use either numbers or dates but we're using the
same search-syntax for both.

- search support for roundup Date and Number (Numbers in roundup are floats)
  no support for searching IDs (that would require major surgery in
  roundup)

- the semicolon ";" (or 'to') is the range operator, like in::

    date1;date2

- A more human-readable variant replaces the ';' with 'to' and allows an
  optional 'from' at the start, e.g.::

    from date1 to date2

- ranges can have inclusive or exclusive limits::

    [a; b]          --> a <= x <= b
    [a; b)          --> a <= x <  b
    (a; b)          --> a <  x <  b
    (a; b]          --> a <  x <= b

  (to continue support for existing date searches,
  and important for floats)

  The lower or upper bound can be left out to specify open (up to
  infinity) ranges::

    [a;]
    [;b]

  (and similar for round parentheses)

  A single number is a range including only that number::

    [a]

  which can be simplified (see below) to::

    a

- ranges can be specified without brackets::

    a;b

  or for the single-number case::

    a

- ranges without brackets are by default *inclusive* for the lower bound
  and *exclusive* for the upper bound: ``[a;b)`` to be compatible with the
  current date semantics. See note under "Cons" below.

- ranges can be combined using the 'or' operator::

   [1] or [5;7.25)

  or simpler::

   1 or 5;7.25

  Note that I'm *not* allowing comma and/or space to be written instead
  of the "or". See below.

- for convenience of foreign users (e.g., german), float values can be
  specified using decimal commas as well as decimal points, the
  following are different representations of the number 0::

    0
    0.0
    0,0
    0.
    0,
    .0
    ,0

Cons:
=====

The default range without explicit brackets/parentheses (include lower
bound but exclude upper bound) may be counter-intuitive and we might
want to change this.

Using the comma as a decimal separator is fine for non-english speaking
countries but means we can't use the comma for other purposes (like
replacing the 'or').

We might want to have a shorter notation for the 'or' similar to the
'to' vs. ';' in ranges.

Pros:
=====

A usable query syntax that can express all intended searches which isn't
complicated in the easy-search cases.

The proposed syntax is fully backward-compatible for current
implementation of roundup Number and Date types. No need to invalidate
and/or rewrite stored queries. No need to re-train users.

Compared with the other proposal: No two concepts of sets (of numbers
and of ranges) in fact a number is a range (containing just that number)
in my proposal.

Examples
========

Search for several ranges::

  [1;2) or [5;9)

or simpler::

  1;2 or 5;9

Search for a set of numbers (which are in fact intervals internally)::

  1 or 2 or 3 or 7.25

or combine both::

  1 or 2 or 3 or 7.25 or [8.5;9] or 10;11 or 12

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  Fax:   +43/2243/26465-23
Reichergasse 131                        www:   http://www.runtux.com
A-3411 Weidling                         email: office@runtux.com
osAlliance member                       email: rsc@osalliance.com

Oh, you drive me nuts...

Ralf Schlatterbeck schrieb:
> I've repeatedly said that -- since the comma should be a decimal comma
> too, I DO NOT WANT the comma to mean "or".

But you know that we *need* to support things like

  2, 5

because they are supported currently for ids and integers.

> Same for blank. I DO NOT WANT a space to implicitly mean "or".

You know that we *need* to support the space, because thinks like

  2 5

are supported for ids.

Both cases are important for compatibility, and you don't mention them
in your proposal.  In fact, what you intend to do is to *break*
compatibility for very common and convenient search expressions.

> If you see ranges as things that can contain just one number (as it is
> currently implemented in roundup.date.Range) you won't need the sets.

Why would I want to see ranges like that? Ranges have limits, not members!

There is a reason for this implementation:  Roundup supports non-SQL
backends.  If it wouldn't, we would have supported date searches right
away using

  WHERE DATE(fieldname) = date1

and inclusive range searches using

  WHERE DATE(fieldname) BETWEEN date1 AND date2

Instead of generally impacting Roundup's search facilities it would have
been better do date range searches the SQL way and teach the non-SQL
backend to extract the date component from a datetime value.

Maybe you are too deeply involved in the Roundup code to be able to take
up an ordinary user's viewpoint.

> Zen of Python:
> Beautiful is better than ugly

Well, concerning curly braces, this is surely a matter of taste.
I like them.

> Readability counts

Sets with curly brackets are /very/ readable!
And surely,

  {1, 2, 5} or [10;20]

is very much better readable than

  1 or 2 or 5 or 10; 20

> Special cases aren't special enough to break the rules

You fail to name the rules which you accuse me to fail.

> There should be one-- and preferably only one --obvious way to do it

/preferably/ only one /obvious/ way.

But we *must* already support several ways, e.g. because of the existing
date range syntax.

> If the implementation is hard to explain, it's a bad idea

I /have/ explained my concept, and it wasn't hard at all.

The implementation can get modularized nicely and be easy to explain as
well.

> Or to fix a misconception: We don't need sets of numbers for full
> expressiveness.

What you mean is, you can express every search without using sets.

But this is *not* the same as expressiveness.

I won't answer to you anymore in this issue until you explain how you
intend to

- stay compatible to the current search facilities
  /without/ treating the comma and the blank like 'or'
- have ranges inclusive by default for numbers and get away without
  a configuration switch for the default range of dates ("magic lever")
- meet user expectations /without/ treating ids, which certainly
  /look like/ numbers, exactly like numbers

Maybe you understand it that way:

Ids walk like integers and swim like integers and quack like integers;
they certainly /are/ integers.

Well, of course they are not /real/ integers; they don't do all this by
themselves but need someone to wind them up first.

But surely they look like integers, and the common user doesn't know the
difference; thus, (s)he should be able to treat them like integers.

There is no need to turn ids into /real/ integers; the backend can
happily continue to take care of the winding up.

End of transmission.
-- 
Tobias

On Wed, Jul 15, 2009 at 11:44:56AM +0000, Tobias wrote:

I'm trying to get to the facts of what currently is possible and implemented.
See below.

> Ralf Schlatterbeck schrieb:
> > I've repeatedly said that -- since the comma should be a decimal comma
> > too, I DO NOT WANT the comma to mean "or".
> 
> But you know that we *need* to support things like
> 
>   2, 5

NO WE DON'T.
I'm currently getting -- If I'm entering that into the query-form for an
integer AND PLEASE NOTE THAT AN ID IS NOT AN INTEGER:

psycopg.ProgrammingError: ERROR: invalid input syntax for type real: "0.5, 0.7" select
_user.id,lower(_user._lastname),(lower(_user._lastname) is not NULL) from _user where ((_user._status in ('1'))) and
_user._lunch_duration='0.5, 0.7' and _user.__retired__ <> 1 order by (lower(_user._lastname) is not
NULL),lower(_user._lastname),_user.id

When entering "0.5, 0.7" into the search form in one of the running
trackers I have (with heavily customized schema).

> because they are supported currently for ids and integers.

FOR IDs yes but FOR INTEGERS NO and my proposal states that IDs should
be left alone for the moment

So please accept that you were *WRONG* concerning what
backward-compatibility requirements we have.

> > Same for blank. I DO NOT WANT a space to implicitly mean "or".
> 
> You know that we *need* to support the space, because thinks like
> 
>   2 5
> 
> are supported for ids.

I've repeatedly stated that IDS ARE STRINGS IN THE CURRENT
IMPLEMENTATION. This IS CONSIDERED A BUG. But one that is very hard to
fix without breaking a lot of things for a lot of people.

> Both cases are important for compatibility, and you don't mention them
> in your proposal.  In fact, what you intend to do is to *break*
> compatibility for very common and convenient search expressions.

See above. YOU are the one who is wrong.

> > If you see ranges as things that can contain just one number (as it is
> > currently implemented in roundup.date.Range) you won't need the sets.
> 
> Why would I want to see ranges like that? Ranges have limits, not members!

come on. think mathematically. A range with a zero-length inclusive
interval is the same as a number.

> There is a reason for this implementation:  Roundup supports non-SQL
> backends.  If it wouldn't, we would have supported date searches right
> away using
> 
>   WHERE DATE(fieldname) = date1
> 
> and inclusive range searches using
> 
>   WHERE DATE(fieldname) BETWEEN date1 AND date2

One of the reasons why roundup has implemented its own query-syntax is
that its easier for users than writing sql. Otherwise we could have used
one of the many sql parsers out there to implement sql-queries for other
backends.

> Instead of generally impacting Roundup's search facilities it would have
> been better do date range searches the SQL way and teach the non-SQL
> backend to extract the date component from a datetime value.

My proposal does not limit the search facilities. It achieves the same
as your proposal with 
- less syntactic constructs
- easier expression syntax
- full backward compatibility
- easier to implement and maintain

> Maybe you are too deeply involved in the Roundup code to be able to take
> up an ordinary user's viewpoint.

Thats not an argument but a personal attack.

> > Zen of Python:
> > Beautiful is better than ugly
> 
> Well, concerning curly braces, this is surely a matter of taste.
> I like them.

I don't -- especially since I've proved that we don't need them.

> > Readability counts
> 
> Sets with curly brackets are /very/ readable!

Which, again, is a question of viewpoint.

> And surely,
> 
>   {1, 2, 5} or [10;20]
> 
> is very much better readable than
> 
>   1 or 2 or 5 or 10; 20

Huh?

See for example the next guideline.

> 
> > There should be one-- and preferably only one --obvious way to do it
> 
> /preferably/ only one /obvious/ way.
> 
> But we *must* already support several ways, e.g. because of the existing
> date range syntax.

No. See above about the backward compatibility arguments.

> > If the implementation is hard to explain, it's a bad idea
> 
> I /have/ explained my concept, and it wasn't hard at all.

explaining is something that requires understanding on the receiver
side, otherwise it fails.

Yes.

Huh?
If you mean by "expressiveness" that there are several ways to formulate
the same thing I'd answer with
"There should be one-- and preferably only one --obvious way to do it"

See above. Neither comma nor space is currently supported for integers
or dates.

I don't. Read my proposal and the discussion at the end.

That would be left for a future bugfix.

They are strings. I've done some patches to at least provide for proper
sorting (some years ago). But this remains a bug in roundup. Patches
that don't break a lot for a lot of people are welcome.
And note that I do *not* intend to apply the new range-search operations
to IDs for the moment.

patches are welcome.

patches are welcome.

I don't think so, otherwise searches in SQL are not possible. But if you
have a good idea provide a patch.

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  Fax:   +43/2243/26465-23
Reichergasse 131                        www:   http://www.runtux.com
A-3411 Weidling                         email: office@runtux.com
osAlliance member                       email: rsc@osalliance.com

On Wed, Jul 15, 2009 at 01:00:23PM +0000, Ralf Schlatterbeck wrote:
> Ralf Schlatterbeck <rsc@runtux.com> added the comment:
> On Wed, Jul 15, 2009 at 11:44:56AM +0000, Tobias wrote:
> > Ralf Schlatterbeck schrieb:
> > > I've repeatedly said that -- since the comma should be a decimal comma
> > > too, I DO NOT WANT the comma to mean "or".
> > 
> > But you know that we *need* to support things like
> > 
> >   2, 5
> 
> NO WE DON'T.
> I'm currently getting -- If I'm entering that into the query-form for an
> integer AND PLEASE NOTE THAT AN ID IS NOT AN INTEGER:

Update: I'm getting the same traceback when entering the comma without a
space (for Number). 
For IDs it seems that "1,2" works and "1 2" works but "1, 2" doesn't.

But as stated previously I think we should *not* include IDs.

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  Fax:   +43/2243/26465-23
Reichergasse 131                        www:   http://www.runtux.com
A-3411 Weidling                         email: office@runtux.com
osAlliance member                       email: rsc@osalliance.com

What's the current status on this?

Also will it work for both anydb and rdbms back ends?