Repository navigation
Question: support for workflows outside of spack? #164
Description
Hiya! Is there support (or planned support) for being able to define an execute workflows with other kinds of package managers? E.g., one very easy family would be something like conda/mamba to install a bunch of python machine learning libraries, which aren't handled as well under spack. Thanks!
Activity
douglasjacobsen commentedon Jun 9, 2023
Hey @vsoch!
The infrastructure should support workflows like this. We don't have plans to implement them, but the way we've written stuff so far it could easily extend this.
I think if we had a few examples, it might not take much to implement. But, in case this helps show where the hooks might exist...
lib/ramble/ramble/application_typeshouses specific types of applications. Right now, they are:At a minimum, an executable application could be written (with some semi-manual installation steps in the documentation) to run some ML workloads. But, a separate application class (i.e. conda.py) could be written to automate some of the conda installation in venvs or something.
The other bit that's specific to spack is
lib/ramble/ramble/spack_runner.pywhich handles calling out to execute spack commands and generate spack environments. Something similar would need to be implemented for conda too.vsoch commentedon Jun 9, 2023
Cool - thank you! Pinging @tpatki - I think we could probably do well to add some python-based ML type workflows or simulations. If you like this idea, I can go ahead and try adding.
Also pinging @wolfv - would this be a good place to sneak in pixi? Do you have Python bindings? It would be cool to test it out in a workflow context.
wolfv commentedon Jun 9, 2023
That would be cool indeed! Not sure if pixi is at the right abstraction level (since it is kinda opinionated), but building on top of the rattler-library (and the upcoming Python bindings) would be very interesting!
vsoch commentedon Jun 9, 2023
What about trying conda or mamba in the meantime? I think most workflows I have in mind use something like pytorch, so maybe (despite the TBA rattler / pixi being exciting) we could do that in the meantime. Snakemake, for example, just calls conda and I believe it's the most popular execution mode!
I'll also wait to hear from @tpatki on her thoughts!
wolfv commentedon Jun 9, 2023
Yes, doing that with
micromambashould be fairly straightforward, indeed!micromambamakes it much easier to set up one (or more) environments in an arbitrary location.douglasjacobsen commentedon Jun 9, 2023
Honestly, another pretty easy way to do it would be to use the spack package definition to install micromamba or miniconda, and then make the installation of your packages part of the executable steps.
I'd be happy to give some input if you have a concrete example for me to look at. :)
vsoch commentedon Jun 9, 2023
@douglasjacobsen haha I think that would make me run away :) I have a few things on my queue that I want to work on first, but I'll come back to this to work on soon!