Hello! I suggest you consider adding support for LDAP authentication to Roundup. I think this could be an important feature for tracker instances on internal networks or anywhere else where LDAP authentication could be critical.

I recently implemented LDAP support for Roundup as an extension that uses ldap3 module. The extension supports connections using unencrypted LDAP, LDAP over TLS and LDAPS, connection logging, updating user properties based on directory entry attributes and authorization based on LDAP group membership. It also supports configuration of connection settings, logging levels and authorization settings.

I think this might be a good starting point for implementing built-in LDAP authentication in Roundup. You can view the code on GitHub: <https://github.com/savchuk1985/roundup-ldap3>.

I licensed the extension under the MIT license, so it is fully compatible with the Roundup license.

If you have any questions or suggestions feel free to contact me.

Thanks for this. We have multiple LDAP implementations described in the wiki 
(https://wiki.roundup-tracker.org). It would be great to have one supported way
to do this. Also on our wish list is oauth/openid login (issue2551239).

A few questions:

  does ldap3 work with python2? It's ok if it doesn't but makes integration
     a little tricky.

  if it becomes a standard part of the trackers, init() should do nothing
     (leaving default login in place) unless the LDAP uri is configured.
     This way it can just hang out and do nothing by default.

  it looks like you only support the User and Admin role? IIUC the User role is added
      if the user is a member of a group listed in `user_groups` and similarly
      Admin role for admin_groups. Do you have ideas on handling more roles?

  it doesn't look like it falls back to native (db) based auth if ldap fails
     (server down, network issue). Am I correct?

  have you thought about authenticating API access (REST, XML-RPC) against LDAP?

The wiki LDAP (https://wiki.roundup-tracker.org/?
action=fullsearch&context=180&value=LDAP&titlesearch=Titles) implementations fall back
to local db auth (or use it first and check ldap on failure). They also
sync the password to the local db so API access and local db auth work.
The advisability of storing the password in the local db is another question.

Your suggestion forces me to consider if we need to register a stack of
validate_user possibilities. At this point we have 6 or more validators/tests that
hook into the login authentication chain:

  native db
  LDAP
  Oauth (someday 8-)
  TOTP/HOTP
  Allow login with email
  Captcha
  HIBP password validation (not verification)

Only native db is supported at this time for API access, but supporting LDAP
would be good as well. The rest make no sense for an API call. Sounds like it's time
to start looking at PAM again for design ideas.

>   does ldap3 work with python2? It's ok if it doesn't but makes integration a little tricky.

Yes, it does: "ldap3 can be used with any Python version starting from 2.6, including all Python 3 versions. It also works with PyPy and PyPy3". 

https://ldap3.readthedocs.io/en/latest/

I think the extension in general should work with python2, but some imports need to be fixed.

>  if it becomes a standard part of the trackers, init() should do nothing (leaving default login in place) unless the LDAP uri is configured. This way it can just hang out and do nothing by default.

I think another option needs to be added to the global configuration that explicitly enables LDAP authentication. If the administrator chooses LDAP authentication, they must also specify the LDAP server. If they don't specify it, then this is just a misconfiguration.

> it looks like you only support the User and Admin role? IIUC the User role is added if the user is a member of a group listed in `user_groups` and similarly Admin role for admin_groups. Do you have ideas on handling more roles?

I think there are two different ways to solve this problem. 

On the one hand, if a user has the Admin role, they can manually assign roles to other users. The User role in this case should be set only if the user does not have roles, i.e. this is a new user who is not a member of admin_groups.

But I don't like this idea because if we already have a directory server, all permissions need to be checked against the entries.

On the other hand, we can get a list of available roles and check the roles associated with LDAP groups. This is preferable, but it is not entirely clear what this should look like in the configuration file. Maybe you have some ideas?

> it doesn't look like it falls back to native (db) based auth if ldap fails (server down, network issue). Am I correct?

Yes, this was done intentionally. We should only have one valid user base. If something is not working properly, it's the administrator's job. We do not attempt to use AnyDBM if the PostgreSQL server is down.

> have you thought about authenticating API access (REST, XML-RPC) against LDAP?

No, unfortunately.

Hi Anton:

In message <1702585627.57.0.263140702446.issue2551307@roundup.psfhosted.org>,
Anton Savchuk writes:

>> does ldap3 work with python2?
>Yes, it does:

Nice.

>I think the extension in general should work with python2, but some
>imports need to be fixed.

Ok.

>> if it becomes a standard part of the trackers, init() should do
>> nothing (leaving default login in place) unless the LDAP uri is
>> configured. This way it can just hang out and do nothing by default.
>
>I think another option needs to be added to the global configuration
>that explicitly enables LDAP authentication. If the administrator
>chooses LDAP authentication, they must also specify the LDAP
>server. If they don't specify it, then this is just a
>misconfiguration.

This is how JWT enabling works currently. If jwt_secret in the main
config file isn't configured JWT's are disabled.  I don't like adding
a global config option that handles just this one case. There are too
many knobs in Roundup's main config.ini already. Plus there are too
many possible authentication methods that could be implemented to have
a flag for each one.

This goes back to my comment about needing a configurable stack a la
PAM. The config for PAMish support would be reasonable to add to the
global config. It probably would need something like:

  www_authn_stack =  recaptcha HIBP otp OIDC ldap local
  rest_authn_stack = jwt ldap local
  xmlrpc_authn_stack = jwt ldap local

with a registration function like:

  register_authn('otp', OtpLoginAction)
  register_authn('ldap', LdapLoginAction)
  register_authn('HIBP', HIBPLoginAction)

to be called from the extension's init() function.

Each extension could define a handler() or verify_user() method that
returns something...  "authorized", "not authorized", "passed",
"failed", "try again answer is hazy" 8-). Depending on whether the
element:

  1) identifies the user: "authorized" - let them in
                          "not authorized" - reject with error message
                          "try again" - go to next item in list

  2) checks validity of login attempt:
         passed: go on to next item in list
                (otp is valid)
         failed: go on to next item in list and report failure
            (e.g. this password has been exposed, please change it after login)
         not authorized - as above
            (e.g. Unable to verify you're a human did you pass the captcha?)

I'm not sure this is complete. Arguably items like the login rate
limiting could be added to the stack as extensions and removed from
the core.

>> it looks like you only support the User and Admin role? IIUC the
>> User role is added if the user is a member of a group listed in
>> `user_groups` and similarly Admin role for admin_groups. Do you
>> have ideas on handling more roles?
>
>I think there are two different ways to solve this problem. 
>
>On the one hand, if a user has the Admin role, they can manually
>assign roles to other users. The User role in this case should be set
>only if the user does not have roles, i.e. this is a new user who is
>not a member of admin_groups.

Defining the default role is what the:

 new_web_user_roles = User
 new_email_user_roles = User

settings do. So I doubly agree with you that this isn't a great idea
as the directory server could do authZ as well as authN although the
schema for the directory server may need to be enhanced..

>On the other hand, we can get a list of available roles

Uh, yeah..... Richard made a mistake when he originally designed the
Roles mechanism. It's a string not a list of enumerated objects
(typos have been known to cause hilarity 8-)).  That being said, it
should be accessible via:

  self.db.security.role

which IIRC is created on the fly by calls to addPermissionToRole().

>and check the roles associated with LDAP groups.

Do you mean there would be a group in ldap called "roundup_provisional
user" and membership in that group would add the "Provisional User"
role to the user to start? I assume on every login the Roles would be
recreated so that removal from the "roundup_provisional user" group
would be reflected in Roundup?

Also how would this work for multiple trackers working from the same
LDAP server where a user might have different roles on different
trackers? They could add groups "roundup1_admin", "roundup2_admin"
etc. I guess.

These sorts of questions I think are why the roles were not derived
from LDAP by prior LDAP implementations. IIRC they just added the new
user and used the new_web_user_roles setting.

>This is preferable,

But is more complex and I am not sure what the answer should look
like.

>but it is not entirely clear what this should
>look like in the configuration file. Maybe you have some ideas?

The only thing that comes to mind is:

  [ldap-role-mapping]
  admin = roundup1_admin
  provisional user = roundup1_provisional_user
  ....

so we have a section specifically set as a dictionary. But I'm not
happy about it. INI files are notoriously bad for this sort of stuff.
Then again I have been getting Roundup running with Kubernetes recently
and YAML is a PITA as well.

>> it doesn't look like it falls back to native (db) based auth if
>> ldap fails (server down, network issue). Am I correct?
>
>Yes, this was done intentionally. We should only have one valid user
>base. If something is not working properly, it's the administrator's
>job. We do not attempt to use AnyDBM if the PostgreSQL server is
>down.

Your point it taken but, ANYDBM wouldn't have any data to use if
Roundup is deployed with Postgres 8-). Surprisingly for SQLite you
could keep your authentication if there was a SQLite error. Session
info was maintained in anydbm files for a long time. Once you got in
I'm not sure you could do much without SQLite but your session was
still available 8-).

But the question is what happens for users who are not in LDAP? Any
email sent to Roundup creates a user. So somebody (sub contractor,
partner ...) could email the Roundup instance and get an account with
a random password. Then they could connect to the web interface and
trigger a password reset (or just login without a password with one
version of the LDAP implementation). I know my Roundup tracker at one
large company was set up to allow anybody at the company to make a
request (and get an account). However the AD subgroup/branch etc. that
we were located under only had the subset of people that were in the
division managing the tracker.

As you note, this LDAP setup doesn't handle complex LDAP trees, but
support for a fallback to local db (enabled via a config.ini
parameter) should be there.

>> have you thought about authenticating API access (REST, XML-RPC)
>> against LDAP?
>
>No, unfortunately.

Yeah that's a moderate sized issue. It's not a huge issue because:

  1) somebody using AJAX will send their session key derived from an
     LDAP login. This should (will) let them access REST from the HTML
     UI. This also handles the case where the user exports a session
     key for use outside of the web UI.

  2) JWT's bypass username/password auth. However the admin would need
     to provide some other way to obtain a JWT and handle refresh
     tokens etc.

This argues for my PAMish style authN stack idea and reworking login
internals to support the multiple use cases 8-(.

Thoughts?

> John Rouillard added the comment:
>
>
> This goes back to my comment about needing a configurable stack a la
> PAM. The config for PAMish support would be reasonable to add to the
> global config. It probably would need something like:
>
>   www_authn_stack =  recaptcha HIBP otp OIDC ldap local
>   rest_authn_stack = jwt ldap local
>   xmlrpc_authn_stack = jwt ldap local
>
> with a registration function like:
>
>   register_authn('otp', OtpLoginAction)
>   register_authn('ldap', LdapLoginAction)
>   register_authn('HIBP', HIBPLoginAction)
>
> to be called from the extension's init() function.

That sounds pretty reasonable. But I'm not entirely sure I can properly evaluate
the design or give any advice in this case.

> >and check the roles associated with LDAP groups.
>
> Do you mean there would be a group in ldap called "roundup_provisional
> user" and membership in that group would add the "Provisional User"
> role to the user to start? I assume on every login the Roles would be
> recreated so that removal from the "roundup_provisional user" group
> would be reflected in Roundup?

Almost like that. I think a mapping section or perhaps a mapping file is
needed. Some kind of dictionary. This dictionary can contain all the Roundup
roles, some of them, or none at all. If it does not contain roles, authorization
is not performed, only authentication. Users get roles from
NEW_WEB_USER_ROLES. If the dictionary contains roles mapped to LDAP groups,
membership will be checked and roles would be applied to the user. If User role
does not mapped to LDAP group, any new user get roles from NEW_WEB_USER_ROLES
without checking membership. If LDAP authorization is used, group membership
must of course be checked on every login.

> Also how would this work for multiple trackers working from the same
> LDAP server where a user might have different roles on different
> trackers? They could add groups "roundup1_admin", "roundup2_admin"
> etc. I guess.

Something like that. I think in real world it would something like
"cn=admins,ou=games,ou=groups,o=acme" and "cn=admins,ou=webapp,ou=groups,o=acme".
The same principle applies to developers, managers, customers and so on.

> >This is preferable,
>
> But is more complex and I am not sure what the answer should look
> like.

In theory, DIT can be very complex and use custom schemes. When I selected the
attributes that would be supported in my extension, I selected only the
attributes contained in the "orgPerson", "inetOrgPerson", "nsOrgPerson" and
"nsPerson" classes. These classes are most likely used for user entries in most
LDAP installations. I think we should follow the same principle regarding
roles. I believe that it is necessary to provide minimum reasonable support for
LDAP authorization.

> The only thing that comes to mind is:
>
>   [ldap-role-mapping]
>   admin = roundup1_admin
>   provisional user = roundup1_provisional_user
>   ....
>
> so we have a section specifically set as a dictionary. But I'm not
> happy about it. INI files are notoriously bad for this sort of stuff.

Perhaps it's better to use a mapping file rather than a mapping section?

> Then again I have been getting Roundup running with Kubernetes recently
> and YAML is a PITA as well.

TOML is probably the best choice. But native support for TOML parsing in Python
is available only from version 3.11.

> As you note, this LDAP setup doesn't handle complex LDAP trees, but
> support for a fallback to local db (enabled via a config.ini
> parameter) should be there.

Yes, I now agree that it should be possible to go back to using a database. I
was thinking about an internal tracker rather than mixed use with an external
users. This is my mistake.

> Thoughts?

I'm not sure I can help with the design of global authentication. This is very
interesting, but my skills are not enough here. I can only say that if the need
to make a new design is obvious, then it should be done =)

John, am I correct in understanding that you are proposing to use an authentication plugin system as something like the current Roundup extensions? That is, in the tracker home there will be a certain directory (e.g. 'auth') in which the auth plugins will be located?

In this case, the solution to support LDAP authentication would be to provide out-of-the-box support for only those roles that are provided by default for that schema. And if the tracker needs to be configured to add additional roles, the LDAP plugin should also be customized.

Another idea is that the authenticator could get all the roles and then iterate through them to check the user's membership in each of them. But we need to somehow define their order. We can get it from a mapping file, for example. Alternatively, perhaps the order (and even the mapping of LDAP groups) could be set in the schema.py file and stored in the database.

For now these are the only things that come to mind.

Hi Anton:

> you are proposing to use an authentication plugin system as
> something like the current Roundup extensions? That is, in the
> tracker home there will be a certain directory (e.g. 'auth') in
> which the auth plugins will be located?

I was going to put the auth extension in the extensions directory.
The difference would be that the init() function at the end of the
extension calls:

   instance.registerAuth('name', function, flow=None priority=100)

I haven't decided if the second arg is a 'function' or a class similar
to how registerAction works. If it's a class it should inherit from a
new authNbase (or some other name) class. I'm leaning toward the class
so it can have methods to return "passed", "failed", "not authorized"
....

Also I realized my earlier example shouldn't be Actions. Actions are
web interface thingies.

The 'flow' parameter would be used to define the stack where the auth
method should be applied. Maybe we can reuse the tokens from db.tx_Source.
For example to add LDAP to all WEB auths:

   instance.registerAuth('name', LdapLookup,
                         flow=( "web", "rest", "xmlrpc" ), priority=50)

the priority 50 would cause it to be run before the native stack of:

  WEB cookie priority - if enabled
  REMOTE_USER header - if enabled
  username/password against native db
  jwt - if enabled
  session lookup fallback

As part of implementing this, each of the 5 in the native stack should
be rewritten to be pre-existing registered auths. They could even have
predefined priorities (90, 100, 110...)  so that you could slot your
auth method between them.

For email you have the stack:

  from address lookup
  gpg signature validation

and can add:

   instance.registerAuth('name', SmimeValidate, flow=( "email",))

would allow you to look at the smime envelope for authentication.

I suspect the authenticator would have to return a tuple of:

  (finding, 'method')

where method could be used as db.tx_Source if provided. Finding would be
"passed"/"failed"/"not authorized"... and be used by the authentication loop to
determine if it rejects the user, allows the user in, needs to go to the
next authenticator.

> In this case, the solution to support LDAP authentication would be to provide
> out-of-the-box support for only those roles that are provided by default for that
> schema.

Correct, new_web_user_roles as defined in config.ini.

> And if the tracker needs to be configured to add additional roles, the LDAP
> plugin should also be customized.

Yeah probably.

> Another idea is that the authenticator could get all the roles and then iterate
> through them to check the user's membership in each of them. But we need to
> somehow define their order.

Roles are (maybe) order independent. Agent,User == User,Agent. Hmm, now that I have
said (well typed) that, I am not sure. The role order may be preserved
in the generated permission rules. If there is a negative permission (deny)
in the sequence, then the roles are not commutative.

> We can get it from a mapping file, for example.

We can create a simple mapping (and ordering) in extensions/config.ini. E.G.

  [ldap]
  role_order = Admin(roundup_admin), Agent(roundup_agent), User(Roundup_user:Roundup_user2), Anonymous(Anon)

The ldap code can then check the roundup_admin group in LDAP and see if the user is a
member. If so, append the role Admin to the new role list. If a user is a member of
roundup_agent and Roundup_user (or Roundup_user2) LDAP groups, the role will be
'Agent, User' and not 'User, Agent'.

> Alternatively, perhaps the order (and even the mapping of LDAP groups) could be
> set in the schema.py file and stored in the database.

That's another possibility, but it would be a departure from existing practice.
I prefer to keep things in the DB that are:

  1) user-facing
  2) need updating for business rules/workflow.

This is an admin setting, so not user-facing. Additional Roles can be added. But
defining and implementing them requires access to the underlying tracker which means
access to config.ini.

Quips, comments, evasions, questions or answers?