DOMINO fails in Singularity during local testing

[agitter]

Nistha reported that DOMINO failed on our departmental cluster when running in Singularity. I confirmed this when testing it with the SPRAS EGFR configuration:

Running Singularity containers
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Select jobs to execute...
Running slicer with arguments: slicer --network_file /spras/D263Q3O/network.txt --output_file /spras/X62MXDY/slices.txt
INFO:    Using cached SIF image

Running DOMINO with arguments: domino --active_genes_files /spras/36SYPK7/active_genes.txt --network_file /spras/D263Q3O/network.txt --slices_file /spras/X62MXDY/slices.txt --output_folder /spras/DU7TQ24/tps-egfr-domino-params-V3X4RW7 --use_cache false --parallelization 1 --visualization true --slice_threshold 0.3 --module_threshold 0.05
{'message': ["start running DOMINO...\ngenerating graph from /spras/D263Q3O/network.txt\nnetwork' pkl is saved: /spras/D263Q3O/network.txt.pkl\ndone building network\ngenerating subgraphs...\nBefore slicing: n of cc:30, n of nodes: 15679, n of edges, 161886\n# of modules after extraction: 201\nAfter slicing: n of cc:203, n of nodes: 15615, n of edges, 47086\nsubgraphs' pkl is saved\nnumber of slices: 203\n# of slices after perturbation TH: 35/203\n33 relevant slices were retained with threshold 0.3\nn of putative modules: 33\nn of final modules: 26 (n=[60, 22, 18, 35, 30, 22, 20, 16, 11, 18, 81, 8, 9, 10, 16, 6, 28, 20, 20, 13, 22, 15, 15, 4, 18, 12])\n26 final modules are reported at /spras/DU7TQ24/tps-egfr-domino-params-V3X4RW7/active_genes/modules.out\nTrue\nvisualizing modules...\n", 'INFO:    Using cached SIF image\nmultiprocessing.pool.RemoteTraceback: \n"""\nTraceback (most recent call last):\n  File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 121, in worker\n    result = (True, func(*args, **kwds))\n  File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar\n    return list(map(*args))\n  File "/usr/local/lib/python3.7/site-packages/src/utils/visualize_modules.py", line 199, in module_report\n    generate_report_from_template(cy, output_base_dir, str(module_index))\n  File "/usr/local/lib/python3.7/site-packages/src/utils/visualize_modules.py", line 169, in generate_report_from_template\n    report_file_name=format_script(os.path.join(os.path.dirname(os.path.abspath(__file__)),\'../data\', "graph.html"), NUM_OF_GENES=len([x for x in cy if not "source" in x["data"] and len(x["data"]["modules"])>0]), HG_REPORT=[], MODULES_SUMMARY=[], DISEASE_GENES=[], DATA=json.dumps(cy))\n  File "/usr/local/lib/python3.7/site-packages/src/utils/scripts.py", line 11, in format_script\n    open(exec_file_name, "w+").write(formatted_script)\nOSError: [Errno 30] Read-only file system: \'/usr/local/lib/python3.7/site-packages/src/utils/../data/graph_0.9355720314087165.html\'\n"""\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File "/usr/local/bin/domino", line 8, in <module>\n    sys.exit(main_domino())\n  File "/usr/local/lib/python3.7/site-packages/src/runner.py", line 53, in main_domino\n    visualize_modules(os.path.splitext(cur_ag.split(\'/\')[-1])[0], G_final_modules, None, network_file, report_folder)\n  File "/usr/local/lib/python3.7/site-packages/src/utils/visualize_modules.py", line 188, in visualize_modules\n    p.map(module_report, params)\n  File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 268, in map\n    return self._map_async(func, iterable, mapstar, chunksize).get()\n  File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 657, in get\n    raise self._value\nOSError: [Errno 30] Read-only file system: \'/usr/local/lib/python3.7/site-packages/src/utils/../data/graph_0.9355720314087165.html\'\n'], 'return_code': 1}

I believe the problem is this line of the source code: https://github.com/Shamir-Lab/DOMINO/blob/85dad1515717b425b17f58f92b13a063ccccb85d/src/utils/visualize_modules.py#L169

DOMINO tries to write a temporary output file to the directory where it is installed, which is not a writeable location inside a Singularity container. Testing the container interactively with Singularity confirms this directory is owned by root:

$ singularity exec docker://reedcompbio/domino /bin/bash
Apptainer> ls -l /usr/local/lib/python3.7/site-packages/src/utils/
total 20
-rw-r--r--. 1 root root    0 Jul 23 11:06 __init__.py
drwxr-xr-x. 2 root root  206 Jul 23 11:06 __pycache__
-rw-r--r--. 1 root root 2451 Jul 23 11:06 ensembl2gene_symbol.py
-rw-r--r--. 1 root root 4068 Jul 23 11:06 graph_influence_linear_th.py
-rw-r--r--. 1 root root  452 Jul 23 11:06 scripts.py
-rw-r--r--. 1 root root 9327 Jul 23 11:06 visualize_modules.py

Edit: See https://github.com/Shamir-Lab/DOMINO/issues/13

[tristan-f-r]

What would be a desirable path? Since the DOMINO maintainers are not responding, we could probably fix this with a similar approach as to how the OI2 offline error was addressed (#78) by PR #226.

[ntalluri]

When I was running long form runs of DOMINO locally using Docker, I was running into errors where DOMINO would randomly disconnect leading to incomplete outputs such as empty network files or pathway folders.

This is one error I would run into:
Error while fetching server API version: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60) None.

This specific error occurred unpredictably, typically manifesting after running 30-60 containerized jobs, regardless of core usage.

The way I was getting around this issue was restarting Docker and fully shutting down Docker Desktop but this workaround introduced unwanted downtime and constant manual intervention.

[tristan-f-r]

I can look into both of these issues. @ntalluri can you make that a separate issue so we can reference it better?

[ntalluri]

@tristan-f-r for this error to occur, you might need to run it on a lot of data and parameter combinations. I can send my config.yaml file I was using if needed and the data I was using is the pilot data from the synthetic data in the spras-benchmarking repo.

[tristan-f-r]

That would be great! 👍 I can attack the suite of DOMINO issues now since it seems to be quite the headache across the board.

[tristan-f-r]

I can attack the suite of DOMINO issues now since it seems to be quite the headache across the board.