load_compendium fills four pipelines (eq_id_to_id_db, id_to_eqids_db, id_to_type_db, info_content_db) and then flushes them serially — four .execute() calls one after another per block:
term2id_pipeline.execute()
id2eqids_pipeline.execute()
id2type_pipeline.execute()
info_content_pipeline.execute()
In production those four databases are four independent Redis instances, so the flushes could overlap (threads, or an async client once #381 / the redis-py upgrade lands) instead of blocking on each in turn. This would let a single Job keep several backend servers busy at once rather than round-robining them.
Context: load_compendium in node_normalizer/loader/loader.py (the per-block flush block).
load_compendiumfills four pipelines (eq_id_to_id_db,id_to_eqids_db,id_to_type_db,info_content_db) and then flushes them serially — four.execute()calls one after another per block:In production those four databases are four independent Redis instances, so the flushes could overlap (threads, or an async client once #381 / the redis-py upgrade lands) instead of blocking on each in turn. This would let a single Job keep several backend servers busy at once rather than round-robining them.
Context:
load_compendiuminnode_normalizer/loader/loader.py(the per-block flush block).