Debian Squeeze recently pushed out an update which appears to break xapian 
indexing in roundup with the tip of the traceback reporting:

  File "/usr/local/lib/python2.6/dist-
packages/roundup/backends/indexer_xapian.py", line 123, in find
    for m in matches]
TypeError: 'MSetItem' object does not support indexing

The xapian version in squeeze is 1.2.3-2 and python-xapian is 1.2.3-3.

No relevant debian bugs appear to be reported against these packages 
suggesting an API change (in xapian or python-xapian) which is 
incompatible with roundup. A clue is a similar breakage in Debian's 
software-center package which also uses xapian via python.

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=596079 

The bottom line is that indexing and text search in roundup is broken in 
(my) squeeze hosted 1.4 tracker and I'm not sure whether anything needs to 
be done in roundup to fix it (ie. for an xapian API change) or whether 
it's just a bug in the xapian or python-xapian packages which will be 
fixed eventually.

Any thoughts?

I'm the DD packaging roundup, and implemented a Xapian related change to
roundup upon request by the DD responsible for Xapian. Please see

http://bugs.debian.org/579927

and the bug in this tracker mentioned therein,

http://issues.roundup-tracker.org/issue2550647

Forgot to say that I'm very much interested to try to push any required
patches to the release, if still possible... but really could use some
help with this. :|

Hmm, guess I should have looked at Debian's roundup package :-)

Needless to say I have an installation made from source (as we've 
modified it somewhat for our own purposes). I'm afraid I'm not a debian 
packaging expert so I'm not sure what's required to get this into the 
Squeeze release (but it would good for it to be there!).

Can you please reproduce the problem with the Debian version of the
package and tell me, whether you see the problem there, too?

Bruce, which version of roundup were you trying precisely?

Sorry for the delayed reply, I'm on a local fork of 1.4.6 but I'm
working to get back onto the trunk (1.4.17) soon.

Has your patch been applied upstream? If so at which release?

-b

On 10/25/10 20:56, Bernhard Reiter wrote:
> Bernhard Reiter <bernhard@intevation.de> added the comment:
>
> Bruce, which version of roundup were you trying precisely?
>
> ----------
> nosy: +ber
>
> ________________________________________________
> Roundup tracker <issues@roundup-tracker.org>
> <http://issues.roundup-tracker.org/issue2550676>
> ________________________________________________
>

So it appears this problem is not yet fixed in 1.4.16. Please see
the attached result of the test_indexer unit tests on my Debian
Squeeze system for roundup 1.4.16 (which I understand already
has the xapian changes made in issue 2550647.

Bernhard has applied the patch at SVN r4483, as per

http://issues.roundup-tracker.org/msg4069 (part of
http://issues.roundup-tracker.org/issue2550647).

The next step would be to determine whether the program or the test is
broken - I don't remember patching the tests...

The patch has been applied but the problem I originally reported when 
creating this issue remains in roundup as of r4577.

That is, all unit tests that exercise xapian report:

  TypeError: 'MSetItem' object does not support indexing

at line 126 in roundup/backends/indexer_xapian.py when run on Debian
Squeeze (python-xapian-1.2.3-3) and text indexed search fails to work
on any tracker run with this roundup revision in Debian Squeeze.

I've avoided this problem for now by disabling xapian on my Debian
Squeeze hosted roundup installation. So I lose relevance driven
keyword search but it seems to me that roundup derives little 
benefit from this anyway since it typically sorts the results
according to its own rules.

I'm no IR expert. Am I missing something here?

What is the benefit of xapian indexed search in roundup compared
with the old rdbms indexer?

Looks like the last two lines of roundup/backends/indexer_xapian.py need
to be changed from:

        return [tuple(m[xapian.MSET_DOCUMENT].get_data().split(':'))
            for m in matches]

to:

        return [tuple(m[xapian.MSET_DOCUMENT].get_data().split(':'))
            for m in matches.items]

(i.e. add ".items" to the last line).

That should work with both Xapian 1.0 and 1.2, but I've not tested - I
don't have a source tree for roundup currently setup, and I've got a
post-holiday mail mountain to get through.

Incidentally, looks like this code isn't exercised by the roundup
testsuite, since that passed with Xapian 1.2 when I tried it after the
previous fix.

tonimueller - this probably needs fixing in Debian squeeze...

Thanks Olly. Unfortunately your suggestion appears not to work.

The error now reported in tests that exercise xapian are:

  IndexError: tuple index out of range

so I guess we're moving in the right direction but we're not quite
there yet :-) Tested with the Debian Squeeze (latest as of writing)
on roundup SVN r4577.

> Incidentally, looks like this code isn't exercised by the roundup
> testsuite, since that passed with Xapian 1.2 when I tried it after
> the previous fix.

It is tested by run_tests.py IF xapian and its python bindings are 
installed on the system it's run on, otherwise the xapian code is
silently ignored by roundup. This is the basis of my workaround;
I've simply removed the xapian indexer from roundup for now but it
would be good to fix this problem.

Just been looking in to this, and it seems there are actually several
issues here (all in Xapian).

The first is that MSetItem's support for the tuple interface via
xapian.MSET_* wasn't documented apart from its use in an older version
of the simpleindex.py example program.  Despite not being explicitly
deprecated, this feature no longer works in Xapian 1.2.

The only working way to use xapian.MSET_* in 1.2 is via mset.items, and
that's the change I suggested in msg4220.  The reason that didn't
actually fix this is that the member of the tuple corresponding to
xapian.MSET_DOCUMENT isn't set (this is a bug since it is documented to
work, but actually it never has worked!)

Our (Xapian devs) The plan to sort out this mess is to note in the
deprecation document that xapian.MSET_* on MSetItem doesn't work in
1.2.x (essentially to retroactively deprecate it to match its removal -
not ideal, but we think the best option at this point), and that
xapian.MSET_DOCUMENT is no longer useful.  We'll also now mark
xapian.MSet.items and xapian.MSET_* as deprecated (but not remove them
for a release cycle).  The preferred API is to use the properties, which
is nicer and works in both 1.0 and 1.2.

This ticket has more details:

http://trac.xapian.org/ticket/531

I've attached a patch against roundup SVN HEAD which should fix this,
but which I haven't tested.  This should still retain compatibility with
Xapian 1.0 too.

Improve the title (there isn't anything Debian-specific about this).

Olly, thanks for looking into this and the patch.
(if someone has tested it, I could commit it, if nobody else is willing 
to.)

I have tested olly's patch and it fixes the issue for me.

(olly, it might also be sensible to submit this patch to Debian as a
squeeze update -- it's a pretty grave bug to have search not work.)

Thanks for testing.

The maintainer of the debian package is on the nosy list (tonimueller)
so should be already be aware of this patch.

Thanks Olly, the unit tests now all pass for me too. 
I will check with a few of our trackers shortly.

It's still not right AFAICT.

The patch fixes the unit test but text search does not work in
one of my trackers when xapian is used. That is, if I delete the
xapian indexer from the roundup installation and reload apache, 
the tracker text search works just fine. If xapian is present it
does not work (i.e. all searches result in zero record results).

Is there something else that needs to be done to enable xapian
accelerated searching?

For what it's worth, it really does work for me.

Asheesh, what version did you test precisely?
(Debian, Xapian, python)
Thanks for helping here!

Tested with Debian 6.0 (Squeeze), libxapian 1.2.3-2 and roundup 1.4.16 
(from the SVN trunk as of r4576 or thereabouts).

I tested:

Debian squeeze 6.0

Roundup (I applied the patch by hand):

ii  roundup             1.4.15-3            an issue-tracking system

paulproteus@linode:~$ dpkg -l libxapian*
Desired=Unknown/Install/Remove/Purge/Hold
|
Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                Version             Description
+++-===================-===================-======================================================
ii  libxapian-ruby1.8   1.2.3-3             Xapian search engine
interface for Ruby 1.8
ii  libxapian15         1.0.7-4             Search engine library
un  libxapian2          <none>              (no description available)
ii  libxapian22         1.2.3-2             Search engine library

paulproteus@linode:~$ dpkg -l python-xapian
Desired=Unknown/Install/Remove/Purge/Hold
|
Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                Version             Description
+++-===================-===================-======================================================
ii  python-xapian       1.2.3-3             Xapian search engine
interface for Python

bruce wrote:
> Is there something else that needs to be done to enable xapian
accelerated searching?

http://roundup.sourceforge.net/docs/installation.html says:

You may install Xapian at any time, even after a tracker has been 
installed and used. You will need to run the “roundup-admin reindex” 
command if the tracker has existing data.

I did apply roundup-xapian-1.2-fix.patch   olly, 2011-01-31 07:32
to rev4579.

Units tests for Debian Lenny and Debian sid parse for me now.
(Debian sid has ii  libxapian22    1.2.4-1  python-xapian  1.2.4-1)

Units pass and testing revealed that xapian is used
(demo/db/text-index/flintlock is indication this)
and searches return results.

Thanks again to Olly for the analysis and fix.