Roundup Tracker - Issues

Issue 2550790

classification
Title: Select menu generation in search template is slower than in item template
Type: rfe Severity: minor
Components: Web interface Versions: 1.4
process
Status: new Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ber, dhancock@arinc.com, rouilj
Priority: Keywords:

Created on 2013-01-23 18:22 by dhancock@arinc.com, last changed 2013-10-12 05:39 by rouilj.

Messages
msg4768 Author: [hidden] (dhancock@arinc.com) Date: 2013-01-23 18:22
(not sure what Type rfe is, sorry).

We have a tracker with about 3,000 "custcomp" (customer company) entries and about 35,000 
issues overall. The issue.item.html for creating or modifying an issue loads in about 1 second, 
but the issue.search.html takes 9 seconds. Most of that time is in rendering the custcomp 
pulldown for searching.

I'd like to know how to either make this faster or replace the search retrieval with the item 
retrieval.

In issue.item.html, the following is the line of interest:

<td tal:content="structure context/custcomp/menu">custcomp</td>

And the corresponding section of issue.search.html is:

<tr tal:define="name string:custcomp;                                                                                   
                db_klass string:custcomp;                                                                               
                db_content string:name;">
  <th>Company:</th>
  <td metal:use-macro="search_select"></td>
  <td metal:use-macro="column_input"></td>
  <td metal:use-macro="sort_input"></td>
  <td metal:use-macro="group_input"></td>
</tr>

If I simply replace search_select line with the context/custcomp/menu line, the select list 
renders very fast, but of course then the search doesn't work.
msg4769 Author: [hidden] (rouilj) Date: 2013-01-23 20:28
In message <1358965339.4.0.483830445644.issue2550790@psf.upfronthosting.co.za> 
<1358965339.4.0.483830445644.issue2550790@psf.upfronthosting.co.za>,
David Hancock writes:
>New submission from David Hancock:
>
>(not sure what Type rfe is, sorry).

Request For Enhancement (i.e. new feature request).

> We have a tracker with about 3,000 "custcomp" (customer company)
> entries and about 35,000 issues overall. The issue.item.html
> for creating or modifying an issue loads in about 1 second, 
> but the issue.search.html takes 9 seconds. Most of that time is
> in rendering the custcomp pulldown for searching.
>
>I'd like to know how to either make this faster or replace the search
>retrieval with the item retrieval.
>
>In issue.item.html, the following is the line of interest:
>
><td tal:content="structure context/custcomp/menu">custcomp</td>
>
>And the corresponding section of issue.search.html is:
>
><tr tal:define="name string:custcomp;                                       
>                                            
>                db_klass string:custcomp;                                   
>                                            
>                db_content string:name;">
>  <th>Company:</th>
>  <td metal:use-macro="search_select"></td>
>  <td metal:use-macro="column_input"></td>
>  <td metal:use-macro="sort_input"></td>
>  <td metal:use-macro="group_input"></td>
></tr>
>
>If I simply replace search_select line with the context/custcomp/menu line,
>the select list renders very fast, but of course then the search doesn't
> work.

One thing to look at is to see what the html generated with the
original template and the modified template look like.

That may give an indication of what component is missing that stops
the two from being tied together. Then you can use actual html rather
than the macros to replicate the original components.
msg4770 Author: [hidden] (rouilj) Date: 2013-01-24 01:15
I think this may also be another instance of issue2550702.
msg4774 Author: [hidden] (ber) Date: 2013-01-24 08:12
David,
thanks for creating the issue. What would be the easiest
way for someone like me to actually see your issue in a running
tracker that I can examine? Just start one with postgresql db
and then fill in 35K issues and 3K accounts?

John,
yes, issue2550702 could be related, but this one may be more specific.
msg4780 Author: [hidden] (dhancock@arinc.com) Date: 2013-01-27 14:49
Sorry to be late responding. Unfortunately, our tracker is behind our corporate firewall, so to test 
would require populating a few thousand customers and a similar number of issues. I don't even 
think the number of issues is important, because the problem manifests itself on the search 
page (with good response on the item page), and not on any of the indexes that list issues.

I haven't been successful trying to fix differences between the two menu-generation methods. 
but I can't fiddle too much as this tracker is in use 24x7.
msg4781 Author: [hidden] (ber) Date: 2013-01-28 19:36
David,
as I've said, if you are making this easier for a developer
to see on a vanilla tracker that is close enough to your problem, 
we can easier come up with a workaround.
So if you start with a vanilla tracker 1.4.21 or hg tip you say
and then show us a short way to show the problem. That is the next step.

Best,
Bernhard
msg4805 Author: [hidden] (rouilj) Date: 2013-02-24 05:38
Might the mechanism discussed in:

   http://www.roundup-tracker.org/cgi-bin/moin.cgi/UserMenu

be a workaround? Basically it uses a detector to generate
the html statically on user changes and defines a tal macro
that can just be substituted when the page is rendered.

So it takes the generation of the list by querying the database
out of the flow of displaying the page.

Not the best solution I agree but it is a basic caching
mechanism.
msg4836 Author: [hidden] (rouilj) Date: 2013-03-26 15:31
David another way to handle this is to pre-cache the user list
as described in:

  http://www.roundup-tracker.org/cgi-bin/moin.cgi/UserMenu

Would that solve your problem?
msg4837 Author: [hidden] (dhancock@arinc.com) Date: 2013-03-26 15:57
I appreciate all the help and ideas. I think the approach in UserMenu on
the Wiki will solve the problem. I will give it a try soon.

It's still interesting that the retrieval of essentially the same list is
so fast on the issue.item.html page and so slow on the issue.search.html
page.

Cheers! 
-- 
David Hancock | dhancock@arinc.com
msg4937 Author: [hidden] (rouilj) Date: 2013-10-12 05:39
There is some discussion on how to improve this situation occurring on
the roundup-devel list.

See:

  https://sourceforge.net/p/roundup/mailman/message/31511901/

titled: warm_cache call initial results and failures

for a prototype of a mechanism to replace many single
item loads with a batch item load that provides a speedup.

Also see:

  https://sourceforge.net/p/roundup/mailman/message/31511633/

titled: Re: load spikes due to user lookup

for tuning to make sure that the rdbms cache is correctly tuned for
your workload along with background discussion on the thread
that lead to the warm_cache implementation.
History
Date User Action Args
2013-10-12 05:39:43rouiljsetmessages: + msg4937
2013-03-26 15:57:36dhancock@arinc.comsetmessages: + msg4837
2013-03-26 15:31:08rouiljsetmessages: + msg4836
2013-02-24 05:38:53rouiljsettype: resource usage -> rfe
messages: + msg4805
severity: normal -> minor
2013-01-28 19:36:53bersetmessages: + msg4781
2013-01-27 14:49:36dhancock@arinc.comsetmessages: + msg4780
2013-01-24 08:12:40bersetnosy: + ber
messages: + msg4774
severity: minor -> normal
type: rfe -> resource usage
2013-01-24 01:15:03rouiljsetmessages: + msg4770
2013-01-23 20:28:27rouiljsetnosy: + rouilj
messages: + msg4769
2013-01-23 18:22:19dhancock@arinc.comcreate