Issue 2551440
Created on 2026-11-19 21:29 by rouilj, last changed 2026-11-19 21:29 by rouilj.
| File name |
Uploaded |
Description |
Edit |
Remove |
|
marshal_.py
|
rouilj,
2026-11-19 21:29
|
|
|
|
| msg8589 |
Author: [hidden] (rouilj) |
Date: 2026-11-19 21:29 |
|
The marshal package is used to serialize roundup data when using a KV
store backend:
anydbm or redis. It shows up in back_anydbm.py, indexer_dbm.py,
session_redis.py, and
sessions_db.py.
However it should not be used for untrusted data. From: https://
docs.python.org/3/library/marshal.html
The marshal module is not intended to be secure against erroneous or
maliciously
constructed data. Never unmarshal data received from an untrusted or
unauthenticated source.
Pickle or shelve are suggested (but pickle also has the security
warning).
While the data we marshal and unmarshal is trusted (we hope), marshal is
not meant to
be portable between python versions.
Therefore, the Python maintainers reserve the right to modify the
marshal format in backward
incompatible ways should the need arise.
msgpack creates smaller serialized data and seems to have fewer security
issues with untrusted data. Using it in place of marshal also results in
smaller
data sizes. This is a win when using with a remote KV datastore.
However to convert we need to do one of two things:
1) convert all dbm database to re-serialise their values to use msgpack
(exporttables/importtables
may do this.)
2) do it dynamically by serializing all data using msgpack but reading
data in either msgpack or marshal format.
The attached marshall_.py can be placed in anypy. Other roundup code can
replace:
import marshal
with
import roundup.anypy.marshal_ as marshal
And the test cases work.
However to handle dbm full text indexes where the dict keys are tuples,
I have to set
strict_key_map=False. This opens an issue with untrusted data in the
msgpack module.
I claim the full-text search index tuples are trusted data. The keys
for the database items (serialized values) are defined from the class
property definitions and are therefore trusted. I am not sure if data is
marshaled from other places (one time keys, session ID's) that is
considered
untrusted. However we were using marshal for that data so....
I think this is a win in either case over marshal since:
the data format doesn't depend on the python version
the data is smaller
security wise I think it's more safe but I can't say how much
Speed wise, msgpack (compiled version) is the same order of magnitude
and differs
by less than .01. So .05x the x is different among marshal, pickle and
msgpack.
No real speed difference among the three.
Native json is 10x slower (.4x) when run under timeit as a comparison.
Couple of downsides:
the serialized values in the dbm files will be mixed msgpack and marshal
format.
This makes debugging the raw values more difficult. Full text indexes
can be
rebuilt using roundup-admin reindex, so it's not an issue there.
marshal data has to fail msgpack loading before it gets read. So this
slows it down a bit.
As items are changed, they are converted to msgpack which is faster.
However index pages
and other displays with a lot of older data will be kept as marshal. I
think an
exporttable/importtable fixes this issue by converting to msgpack on
import.
One last question is how widely installed is msgpack? msgpack has a
compiled
shared library version and a (slower) pure python version. I was able to
pip install msgpack with a compiled version for cygwin (unix emulation
on MS windows).
I suspect most distributions have a compiled version available in pip or
via
the distro package manager.
Thoughts?
|
|
| Date |
User |
Action |
Args |
| 2026-11-19 21:29:02 | rouilj | create | |
|