Consider replacing `sqlitedict`

[ewu63]

Description of feature

sqlitedict has not received any updates in 3 years, so we should probably look at other options. It also has some compatibility issues so we can look to other backends. However, it's not entirely clear to me what the requirements are, in terms of serialization, speed, stability, portability to other systems, and scalability to large datasets.

Potential solution

I've no idea. Just throwing some names:

• tinydb
• pickledb

[robfalck]

Just voicing support for this, there's a vulerability on sqlitedict. While untrusted data sources may not be a big deal in practice, having a security vulnerability might forbid pyoptsparse from being installed in some environments.

Another alternative is diskcache, but it is also somewhat out of date and has a similar vulnerability.

It may be possible to roll your own dict-like access using only python standard library sqlite3 capability if you avoid the use of pickle.

[ewu63]

I thought a bit more about this and came to two options

• the native sqlite3 option as mentioned by @robfalck, with some custom handling of numpy arrays via arr.tobytes() to avoid pickling (which we are currently doing)
• switch to a hdf5 backend, which is a heavier dep but less reliant on custom schema/reader etc. and anyone can inspect the files

Of course we will add a shim layer and some detection code to make new versions of the code able to read older .hst files. Any thoughts on this is welcome before we make any decision