Roundup Tracker - Issues

Issue 2550923

classification
Create new Computed property type
Type: rfe Severity: normal
Components: Database Versions: devel
process
Status: open
:
: rouilj : antmail, rouilj, schlatterbeck
Priority: normal : Effort-High, GSOC

Created on 2016-07-07 02:04 by rouilj, last changed 2021-03-12 23:06 by rouilj.

Messages
msg5773 Author: [hidden] (rouilj) Date: 2016-07-07 02:04
At: https://sourceforge.net/p/roundup/mailman/message/34268641/

is a discussion of computed properties for a class.

Use case: suppose you wanted to have a count of the number of messages
in an issue.

You can add a msg_count:

 issue = IssueClass(db,'issue',
   ....
   msg_count = Number()

to the issue, but then you need an auditor to keep the count up to
date, and it has to run on every msg update even if nobody ever asks
for the msg_count.

Suppose you could instead add:

   msg_count = Computed("countIssueMessages")

When somebody does a db.issue.get(id, 'msg_count') the function:

  countIssueMessages(InstanceOfIssueClass, db)

is invoked. Then in schema.py (or lib/) I define:

Class Computed:

 def countIssueMessages(Myissue, db):
        return Myissue['msgs'].count()

Use Case: this could be used for getting the user phone number from an
external address book by changing the user schema to read:

   user = Class(db, "user",
                ...
                phone=Computed("getPhoneFromLdap")

then:

  def getPhoneFromLdap(Myuser, db):
         return lookupPhoneGivenUser(Myuser['username'])

So this way there is no need to "sync" a bunch of data from an external
source into roundup.

Design etc. TBD. But I wanted to get this in the tracker as it seems it
may be a good summer of code idea for the future.

Note that there is no way to set the properties, they are strictly
read only at this time.
msg5858 Author: [hidden] (antmail) Date: 2016-07-15 11:13
> At: https://sourceforge.net/p/roundup/mailman/message/34268641/

> is a discussion of computed properties for a class.

I  think  this  is very interesting issue. I think we must discuss a
design and future direction of roundup-tracker.

There are my raw thoughts.
 I  think that a ReadOnly  properties is more suitable to roundup-tracker
 conception  than  Computed. Firstly it can be typified and then
 used in template as usual.
 The example of external address book will look like:

user = Class(db, "user",
                ...
                phone=ROString("getPhoneFromLdap")
                ...
then:

  def getPhoneFromLdap(MyUser):
         return lookupPhoneGivenUser(Myuser['username'])
msg7031 Author: [hidden] (rouilj) Date: 2020-11-18 15:19
I have an initial read only implementation of this setup.

It supports query/display in html, rest and xml interfaces.
You can specify a cache parameter, but using it raises 
NotImplementedError.

It does not support: search, sort or grouping by the computed field.

Doc is also done (bloating customizing.txt even more).

While the initial use case is read only, a developer I know pointed
out that this could be useful abstraction to model external data
sources into roundup. So in theory setting a property in a remote
db (oracle, mongo, ldap) could be done via this mechanism. I am not
sure this is better then using a reactor. It would keep data
retrieval together with data manipulation/setting code but....

I am still using the classname Computed, but I think it needs a better
name. Maybe: Operator, Agent, Broker (my fave so far), Proxy... 
Suggestions?

-- rouilj
msg7032 Author: [hidden] (schlatterbeck) Date: 2020-11-18 15:44
On Wed, Nov 18, 2020 at 03:19:16PM +0000, John Rouillard wrote:
> 
> I have an initial read only implementation of this setup.

Cool!

...
> I am still using the classname Computed, but I think it needs a better
> name. Maybe: Operator, Agent, Broker (my fave so far), Proxy... 
> Suggestions?

I'd keep Computed as the base class. Other implementation may chose to
derive from it.
It would be nice to make a derived class that can search, too, like
your implementation of the number of items: This could map to some SQL
code and thus be searchable, too. (Maybe this needs two different SQL
parts, one for search one for display, I haven't looked too closely yet)

Concerning read-only: My guess is that most computed properties will be
read-only.

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  www:   www.runtux.com
Reichergasse 131, A-3411 Weidling       email: office@runtux.com
msg7033 Author: [hidden] (rouilj) Date: 2020-11-18 18:06
Hi Ralf:

In message <20201118154402.cvygbqhrgspo7pwb@runtux.com>,
Ralf Schlatterbeck writes:
>On Wed, Nov 18, 2020 at 03:19:16PM +0000, John Rouillard wrote:
>> I am still using the classname Computed, but I think it needs a better
>> name. Maybe: Operator, Agent, Broker (my fave so far), Proxy... 
>> Suggestions?
>
>I'd keep Computed as the base class. Other implementation may chose
>to derive from it.

Well this is a base class like Integer, Number, Date ... I don't
expect them to be subclassed.

The only thing you can really set are new methods. New methods are
added by setattr() on the Computed class rather than subclassing.

>It would be nice to make a derived class that can search, too, like
>your implementation of the number of items: This could map to some SQL
>code and thus be searchable, too. (Maybe this needs two different SQL
>parts, one for search one for display, 

For data where the source of truth lives in roundup (e.g. message
count), it would probably be better implemented via a real property
with a detector/auditor that updates the data.

For data where the source of truth is outside of roundup, we need to
cache data locally for all instances of the class. Otherwise we have
to do expensive computation for say 1,000,000 issues of the
field. Ideally that would never be the only field searched, so the
number of computed fields would be a fraction of the total number of
issues but...

If we cache all 1,000,000 issues, they need to be rescanned and have
their values recalcuated into a sql/anydbm cache on a regular
basis. Alternatively the cache could be maintained by a (rest update)
request from the source of truth. (Another reason writing to a
computed field may be useful.)  Otherwise the search results will be
incomplete at best or wrong at worst.

> I haven't looked too closely yet)

I haven't checked it in yet. I can send you a patch offline if you
like. It doesn't have tests and there is no obvious existing test to
extend.

>Concerning read-only: My guess is that most computed properties will be
>read-only.

That was my expectation as well. The current implementation is only
for read only access, but we could have write use cases.
msg7038 Author: [hidden] (rouilj) Date: 2020-11-27 23:13
I pushed my changes onto the issue2550923_computed_property branch
starting with rev6292:1e5ed659e8ca. Ralf feel free to take a look
I am not sure it is worth merging in its current state but...
msg7101 Author: [hidden] (rouilj) Date: 2021-03-12 23:06
Ralf, Anthony:

Have you had a chance to look at the new Computed type?

-- rouilj
History
Date User Action Args
2021-03-12 23:06:33rouiljsetmessages: + msg7101
2020-11-27 23:13:45rouiljsetassignee: rouilj
messages: + msg7038
components: + Database
versions: + devel
2020-11-18 18:06:04rouiljsetmessages: + msg7033
2020-11-18 15:44:04schlatterbecksetmessages: + msg7032
2020-11-18 15:19:16rouiljsetstatus: new -> open
messages: + msg7031
2016-07-15 11:13:47antmailsetnosy: + antmail
messages: + msg5858
2016-07-12 15:34:14schlatterbecksetnosy: + schlatterbeck
2016-07-07 02:04:39rouiljcreate