Roundup Tracker - Issues

Issue 2551201

classification
Postgres tests fail on debian oldstable
Type: Severity: normal
Components: Database, Test Versions: devel
process
Status: fixed
:
: : marcus.priesch, rouilj, schlatterbeck
Priority: normal :

Created on 2022-05-03 11:36 by schlatterbeck, last changed 2022-05-03 17:09 by rouilj.

Messages
msg7488 Author: [hidden] (schlatterbeck) Date: 2022-05-03 11:36
All (most?) of the postgres indexer tests and two tests in test/test_postgresql.py fail because older psycopg2 versions don't have an 'errors' module:

test/test_indexer.py:236: 
...
>   from psycopg2.errors import InFailedSqlTransaction, SyntaxError, \
                                UndefinedObject
E                               ModuleNotFoundError: No module named 'psycopg2.errors'

postgresqlDBTest.testUpgrade_6_to_7:
>       with self.assertRaises(psycopg2.errors.UndefinedTable) as ctx:
E       AttributeError: module 'psycopg2' has no attribute 'errors'

test/test_postgresql.py:113: AttributeError

postgresqlDBTest.testUpgrade_5_to_6
>       with self.assertRaises(psycopg2.errors.UndefinedTable) as ctx:
E       AttributeError: module 'psycopg2' has no attribute 'errors'

test/test_postgresql.py:113: AttributeError

I'll look into this if we can get the same results with older postgres versions.

Note that this happens for me only with python3 (which uses psycopg2 from the distribution) not with python2 where I seem to have installed a psycopg2 module by hand via pip (and not using the one shipped by debian oldstable).

So the question is if we want to fix this or recommend to use a newer psycopg2?
msg7493 Author: [hidden] (rouilj) Date: 2022-05-03 14:41
Hi Ralf:

In message <1651577795.25.0.549293703088.issue2551201@roundup.psfhosted.org>,
Ralf Schlatterbeck writes:
>All (most?) of the postgres indexer tests and two tests in
>test/test_postgresql.py fail because older psycopg2 versions don't
>have an 'errors' module:
> [...]
>I'll look into this if we can get the same results with older postgres
>versions.
>
>Note that this happens for me only with python3 (which uses psycopg2
>from the distribution) not with python2 where I seem to have
>installed a psycopg2 module by hand via pip (and not using the one
>shipped by debian oldstable).
>
>So the question is if we want to fix this or recommend to use a newer
>psycopg2?

From what I can find, psycopg2 version 2.8 released on Apr 4, 2019
introduced the errors module. So that's three years ago.
redhat/debian/ubuntu etc. have a 5 year (free/full support) cycle on
their (lts) releases. While it would be nice to support back to 2017
versions of psycopg2, the question is how much of a mess will it be
for how much gain.

For example:

  except SyntaxError:
    do syntax error stuff

I think has to become:

  except psycopg2.Error:
             if pgcode == psycopg2.errorcodes.SYNTAX_ERROR:
	        do syntax error stuff

Is there a way in python to make this work:

  except SyntaxError:
    do syntax error stuff
  except psycopg2.Error as e:
             if e.pgcode == psycopg2.errorcodes.SYNTAX_ERROR:
	        jump to SyntaxError exception above

I don't think there is. But this would allow us to easily identify and
remove the compatibility code in 2 years. Some type of mapping shim
that intercepted psycopg2.Error and regenerated it as (our own)
SyntaxError exception would also work. We simply remove the imported
shim in a couple of years.

By the time these changes get released (this July?), there will only
be a year and a half or so of life in them. Any compatibility code we
add won't be tested (exception handling code in general is not well
tested).

My take on this is to not change any code. Instead add a minimum
required version of psycopg2 to the notes.

We have a good workaround. Install psycopg2 version 2.8 or newer in a
virtual env via pip. This preserves the system version.

Thoughts?
msg7495 Author: [hidden] (schlatterbeck) Date: 2022-05-03 14:57
On Tue, May 03, 2022 at 02:41:47PM +0000, John Rouillard wrote:
> 
> From what I can find, psycopg2 version 2.8 released on Apr 4, 2019
> introduced the errors module. So that's three years ago.
> redhat/debian/ubuntu etc. have a 5 year (free/full support) cycle on
> their (lts) releases. While it would be nice to support back to 2017
> versions of psycopg2, the question is how much of a mess will it be
> for how much gain.
[...]
> My take on this is to not change any code. Instead add a minimum
> required version of psycopg2 to the notes.

Yes.
I've investigated the errors we use from the psycopg2.errors module. It
is almost impossible to find out with what to replace these errors in
the older version. Debian oldstable has 2.7.7 while stable has 2.8.6.
And the code would be awkward. With try/except in import or similar.

Note that we use the errors module only in the tests and in the postgres
indexer in indexer_postgresql_fts.py. Not in the database code itself.
The backend code works just fine with 2.7.7. (when not using the
postgres indexer). I do have several production versions running that
use the latest roundup version.

> We have a good workaround. Install psycopg2 version 2.8 or newer in a
> virtual env via pip. This preserves the system version.

Yes, I think documenting the version + the possible workaround should be
fine. I'll try to come up with and addition to the docs.

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  www:   www.runtux.com
Reichergasse 131, A-3411 Weidling       email: office@runtux.com
msg7497 Author: [hidden] (schlatterbeck) Date: 2022-05-03 15:22
On Tue, May 03, 2022 at 02:57:55PM +0000, Ralf Schlatterbeck wrote:
> Yes, I think documenting the version + the possible workaround should be
> fine. I'll try to come up with and addition to the docs.

I've added some documentation to doc/postgresql.txt.

Let me know if this is ok, if ok for you we can close this issue.

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  www:   www.runtux.com
Reichergasse 131, A-3411 Weidling       email: office@runtux.com
msg7498 Author: [hidden] (rouilj) Date: 2022-05-03 17:09
Hi Ralf:

In message <20220503152224.o7mpdzw5alyw47as@runtux.com>,
Ralf Schlatterbeck writes:
>On Tue, May 03, 2022 at 02:57:55PM +0000, Ralf Schlatterbeck wrote:
>> Yes, I think documenting the version + the possible workaround should be
>> fine. I'll try to come up with and addition to the docs.
>
>I've added some documentation to doc/postgresql.txt.
>
>Let me know if this is ok, if ok for you we can close this issue.

That looks good. I also updated installation.txt. Changed psycopg to
psycopg2 and recommended at least version 2.8 (version 1 was still
recommended). Also added links for postgresql.txt and mysql.txt (they
were text references before).

I guess we need to think about supporting psycopg3 at some point as
well.

Thanks Ralf. Closing.
History
Date User Action Args
2022-05-03 17:09:35rouiljsetstatus: new -> fixed
messages: + msg7498
2022-05-03 15:22:33schlatterbecksetmessages: + msg7497
2022-05-03 14:57:55schlatterbecksetmessages: + msg7495
2022-05-03 14:41:47rouiljsetmessages: + msg7493
2022-05-03 11:36:35schlatterbeckcreate