diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c39069208..27519bbfb 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -6,7 +6,7 @@ Reporting bugs and asking questions You can post questions or issues or feedback through the following channels: -1. [Github Discussions](https://github.com/lithops-cloud/lithops/discussions): For discussions about development, questions about usage, and feature requests. +1. [GitHub Discussions](https://github.com/lithops-cloud/lithops/discussions): For discussions about development, questions about usage, and feature requests. 2. [GitHub Issues](https://github.com/lithops-cloud/lithops/issues): For bug reports and feature requests. @@ -17,7 +17,7 @@ To contribute a patch: harder to merge in a large change with a lot of disjoint features. 2. Submit the patch as a GitHub pull request against the master branch. 3. Make sure that your code passes the tests. -4. Make sure that your code passes the linter. Install `flake8` with `pip3 install flake8` and run the following command until you don't see any linting error: +4. Make sure that your code passes the linter. Install `flake8` with `pip3 install flake8` and run the following command until you see no linting errors: ```bash flake8 lithops --count --max-line-length=180 --statistics --ignore W605,W503 ``` @@ -27,7 +27,7 @@ To contribute a patch: Testing ------- -To test that all is working as expected, you must install `pytest`, navigate to the tests folder `lithops/tests/`, and execute: +To test that everything is working as expected, you must install `pytest`, navigate to the tests folder `lithops/tests/`, and execute: ```bash pytest -v ``` diff --git a/README.md b/README.md index db1be9154..501763e3c 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@

-    PyPI - Downloads Ask DeepWiki +    PyPI - Downloads Ask DeepWiki

Lithops is a Python multi-cloud distributed computing framework that lets you run unmodified Python code at massive scale across cloud, HPC, and on-premise platforms. It supports major cloud providers and Kubernetes platforms, running your code transparently without requiring you to manage deployment or infrastructure. @@ -14,7 +14,7 @@ Lithops is ideal for highly parallel workloads—such as Monte Carlo simulations ## Installation -1. Install Lithops from the PyPi repository: +1. Install Lithops from the PyPI repository: ```bash pip install lithops @@ -40,7 +40,7 @@ Lithops provides an extensible backend architecture (compute, storage) designed ## High-level API -Lithops is shipped with 2 different high-level Compute APIs, and 2 high-level Storage APIs +Lithops ships with 2 different high-level Compute APIs, and 2 high-level Storage APIs.
diff --git a/config/README.md b/config/README.md index 958a3faae..389ca2328 100644 --- a/config/README.md +++ b/config/README.md @@ -1,16 +1,16 @@ # Lithops configuration -By default Lithops works on Localhost if no configuration is provided. To run workloads on the Cloud, you must configure both a compute and a storage backend. Failing to configure them properly will prevent Lithops to submit workloads. Lithops configuration can be provided either in a configuration file or in runtime via a Python dictionary. +By default Lithops works on Localhost if no configuration is provided. To run workloads on the Cloud, you must configure both a compute and a storage backend. Failing to configure them properly will prevent Lithops from submitting workloads. Lithops configuration can be provided either in a configuration file or at runtime via a Python dictionary. ### Configuration file To configure Lithops through a [configuration file](config_template.yaml) you have multiple options: -1. Create a new file called `config` in the `~/.lithops` folder (i.e: `~/.lithops/config`). +1. Create a new file called `config` in the `~/.lithops` folder (i.e., `~/.lithops/config`). 2. Create a new file called `.lithops_config` in the root directory of your project from where you will execute your Lithops scripts. -3. Create a new file called `config` in the `/etc/lithops/` folder (i.e: `/etc/lithops/config`). Useful for sharing the config file on multi-user machines. +3. Create a new file called `config` in the `/etc/lithops/` folder (i.e., `/etc/lithops/config`). Useful for sharing the config file on multi-user machines. 4. Create the config file in any other location and configure the `LITHOPS_CONFIG_FILE` system environment variable: @@ -18,10 +18,10 @@ To configure Lithops through a [configuration file](config_template.yaml) you ha ### Configuration keys in runtime -An alternative mode of configuration is to use a python dictionary. This option allows to pass all the configuration details as part of the Lithops invocation in runtime. An entire list of sections and keys is [here](config_template.yaml) +An alternative mode of configuration is to use a Python dictionary. This option allows you to pass all the configuration details as part of the Lithops invocation at runtime. The full list of sections and keys is [here](config_template.yaml) ## Compute and Storage backends -Choose your compute and storage engines from the table below +Choose your compute and storage backends from the table below
@@ -49,7 +49,7 @@ Storage Backends Serverless (FaaS) Backends: - [AWS Lambda](../docs/source/compute_config/aws_lambda.md) -- [Google Cloud Functions](../docs/source/compute_config/gcp_functions.md) +- [Google Cloud Run functions](../docs/source/compute_config/gcp_functions.md) - [Azure Functions](../docs/source/compute_config/azure_functions.md) - [Aliyun Functions](../docs/source/compute_config/aliyun_functions.md) - [Oracle Cloud Functions](../docs/source/compute_config/oracle_functions.md) @@ -59,7 +59,7 @@ Storage Backends - [IBM Code Engine](../docs/source/compute_config/code_engine.md) - [AWS Batch](../docs/source/compute_config/aws_batch.md) - [Google Cloud Run](../docs/source/compute_config/gcp_cloudrun.md) -- [Azure Container APPs](../docs/source/compute_config/azure_containers.md) +- [Azure Container Apps](../docs/source/compute_config/azure_containers.md) - [Kubernetes](../docs/source/compute_config/kubernetes.md) - [Knative](../docs/source/compute_config/knative.md) - [Singularity](../docs/source/compute_config/singularity.md) diff --git a/docs/index.rst b/docs/index.rst index 4b46d3da3..e448aba9e 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -56,7 +56,7 @@ Object Storage Made Easy Lithops simplifies working with data lakes and object storage by providing: - 🔍 **Automatic data discovery**: Detects and lists files across nested directories. -- 📂 **Transparent data partitioning**: Splits large files (e.g., CSV, Parquet, JSON) into chunks for efficient parallel processing. +- 📂 **Transparent data partitioning**: Splits large text-based files (for example, CSV) into chunks for efficient parallel processing. - 🧰 **Unified, Pythonic API**: Interact with your data using a single interface, regardless of where it's stored. You write simple Python code — Lithops handles the complexity of parallel I/O, data distribution, and storage backends under the hood. diff --git a/docs/source/airflow.rst b/docs/source/airflow.rst index e0e5dd7a4..27794dd6c 100644 --- a/docs/source/airflow.rst +++ b/docs/source/airflow.rst @@ -6,7 +6,7 @@ The Airflow/Lithops integration allows Airflow users to keep all of their Lithop define task dependencies by moving data through Python functions. -Refer to the `integration repository `_ . +Refer to the `integration repository `_. Examples -------- @@ -18,7 +18,7 @@ Define a function in a separate file (``my_functions.py``): .. code:: python def add(x, y): - return x + y + return x + y Import the Lithops operator and the function, and create the DAG to execute: @@ -48,7 +48,7 @@ Import the Lithops operator and the function, and create the DAG to execute: mult_num_map = LithopsMapOperator( task_id='mult_num_map', - map_function=example_functions.add_num, + map_function=add, iterdata_from_task={'a': 'gen_list'}, extra_args={'b': 10}, dag=dag diff --git a/docs/source/api_stats.rst b/docs/source/api_stats.rst index fe4188ac8..b65a4bc6d 100644 --- a/docs/source/api_stats.rst +++ b/docs/source/api_stats.rst @@ -8,7 +8,7 @@ Lithops provides built-in internal tools to supply the user with technical stati Execution summary plots ----------------------- -The :code:`plot()` method from :code:`FunctionExecutor` creates a scatter plot and a histogram plot showing a summary of the tasks executed by a :code:`FunctionExecutor`. By default, lithops creates a :code:`plots/` directory in the working directory path containing both plots in PNG format. For more details refer to the `FunctionExecutor API reference `_. +The :code:`plot()` method from :code:`FunctionExecutor` creates a scatter plot and a histogram plot showing a summary of the tasks executed by a :code:`FunctionExecutor`. By default, Lithops creates a :code:`plots/` directory in the working directory path containing both plots in PNG format. For more details, refer to the `FunctionExecutor API reference `_. To get started, first install Lithops and the plotting dependencies with: @@ -115,7 +115,7 @@ The user can obtain these statistics through the future object: * - Stat - Description * - :code:`func_data_size_bytes` - - Size in bytes of the input data processed by this function. That is, the object size of the input list item processed by this function. Note that if the function processes data obtained from object storage, this value does not include the size of that data, only those that have been serialized and sent from the host process to the function. + - Size in bytes of the input data processed by this function. That is, the object size of the input list item processed by this function. Note that if the function processes data obtained from object storage, this value does not include the size of that data—only the data serialized and sent from the host process to the function. * - :code:`func_module_size_bytes` - Size in bytes of the dependencies (function and modules) serialized and uploaded by the host process. * - :code:`func_result_size` @@ -131,7 +131,7 @@ The user can obtain these statistics through the future object: * - :code:`host_job_serialize_time` - Total time taken by the host process to serialize the input data and dependencies (functions and modules). * - :code:`host_result_done_tstamp` - - Timestamp of when host received the function result from cloud object storage. + - Timestamp of when the host received the function result from cloud object storage. * - :code:`host_result_query_count` - Number of queries to the object storage to get the result object. * - :code:`host_status_done_tstamp` @@ -143,7 +143,7 @@ The user can obtain these statistics through the future object: * - :code:`worker_cold_start` - Indicates whether it was a "warm" invocation (the container was already running) or "cold" (the container had to be deployed). * - :code:`worker_end_tstamp` - - Timestamp in which the worker function had finished its execution. + - Timestamp when the worker function finished its execution. * - :code:`worker_exec_time` - Total execution time of the worker function (Lithops wrapper + user-defined function execution time). * - :code:`worker_func_cpu_usage` diff --git a/docs/source/api_storage_os.rst b/docs/source/api_storage_os.rst index 9fcca1453..151b73696 100644 --- a/docs/source/api_storage_os.rst +++ b/docs/source/api_storage_os.rst @@ -43,7 +43,7 @@ Cloud Proxy Storage API Similar to Python's built-in function `open() `__. -Manipulate an object stored in Cloud Object Storage. +Manipulate an object storaged in Cloud Object Storage. +-------------+-------------------------------------------------------------------------------------------------------------------+ | Parameter | Description | diff --git a/docs/source/cli.rst b/docs/source/cli.rst index 772059a62..cf94b46bc 100644 --- a/docs/source/cli.rst +++ b/docs/source/cli.rst @@ -28,7 +28,7 @@ Runs a *hello-world* function. | --debug, -d | Activate debug logs (Flag) | +-----------------+------------------------------+ -- **Usage example**: ``lithops hello -b ibm_cf -s ibm_cos`` +- **Usage example**: ``lithops hello -b aws_lambda -s aws_s3`` ``lithops test`` ~~~~~~~~~~~~~~~~ @@ -58,14 +58,14 @@ Runs the unit testing suite. For more instructions about testing, see the | --keep\_datasets, -k | Keeps datasets in storage after the test run (Flag) | +------------------------+----------------------------------------------------------------+ -- **Usage example**: ``lithops test -b ibm_cf -s ibm_cos`` +- **Usage example**: ``lithops test -b aws_lambda -s aws_s3`` ``lithops clean`` ~~~~~~~~~~~~~~~~~ Deletes all the information related to Lithops except the config file. It includes deployed runtimes and temporary data stored in the storage -backend. Run this command is like *start from scratch* with Lithops. In +backend. Running this command is like *starting from scratch* with Lithops. In some circumstances, when there is some inconsistency between the local machine and the cloud, it is convenient to run this command. @@ -85,7 +85,7 @@ machine and the cloud, it is convenient to run this command. | --all, -a | Delete all (Flag) | +-----------------+------------------------------+ -- **Usage example**: ``lithops clean -b ibm_cf -s ibm_cos`` +- **Usage example**: ``lithops clean -b aws_lambda -s aws_s3`` ``lithops attach`` ~~~~~~~~~~~~~~~~~~ @@ -177,7 +177,7 @@ the runtime to the compute backend. +-----------------+-----------------------------------+ - **Usage example**: - ``lithops runtime build -f Dockefile.pythonv39 -b ibm_cf lithopscloud/my-runtime-name-v39:01`` + ``lithops runtime build -f Dockerfile -b aws_lambda lithopscloud/my-runtime-name-v312:01`` ``lithops runtime deploy `` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -210,7 +210,7 @@ cases you can avoid using this command. +-----------------+------------------------------------------------+ - **Usage example**: - ``lithops runtime deploy -b ibm_cf lithopscloud/my-runtime-name-v39:01 -m 1024 -t 300`` + ``lithops runtime deploy -b aws_lambda lithopscloud/my-runtime-name-v312:01 -m 1024 -t 300`` ``lithops runtime update `` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -236,7 +236,7 @@ let Lithops create the runtime automatically with the new Lithops code. +-----------------+------------------------------+ - **Usage example**: - ``lithops runtime update -b ibm_cf lithopscloud/my-runtime-name-v39:01`` + ``lithops runtime update -b aws_lambda lithopscloud/my-runtime-name-v312:01`` ``lithops runtime list`` ~~~~~~~~~~~~~~~~~~~~~~~~ @@ -253,7 +253,7 @@ Lists all created/deployed runtimes of a specific compute backend. | --debug, -d | Activate debug logs (Flag) | +-----------------+------------------------------+ -- **Usage example**: ``lithops runtime list -b ibm_cf`` +- **Usage example**: ``lithops runtime list -b aws_lambda`` ``lithops runtime delete `` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -282,7 +282,7 @@ match the provided runtime-name, but all of them. +-----------------+----------------------------------------------+ - **Usage example**: - ``lithops runtime delete -b ibm_cf -s ibm_cos lithopscloud/my-runtime-name-v39:01`` + ``lithops runtime delete -b aws_lambda -s aws_s3 lithopscloud/my-runtime-name-v312:01`` VM Images management @@ -306,7 +306,7 @@ Build a new VM image. +-----------------+-----------------------------------+ | --region, -r | Compute backend region | +-----------------+-----------------------------------+ -| --overwrite, -o | Overwrite the VM image if exists | +| --overwrite, -o | Overwrite the VM image if it exists | +-----------------+-----------------------------------+ | --debug, -d | Activate debug logs (Flag) | +-----------------+-----------------------------------+ @@ -387,12 +387,12 @@ Uploads a local file to a bucket. +-----------------+--------------------------------------+ - **Usage example**: - ``lithops storage put -b ibm_cos test.txt cloudbucket`` + ``lithops storage put -b aws_s3 test.txt cloudbucket`` ``lithops storage get `` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Downloads a remote object stored in a bucket to a local file. +Downloads a remote object storaged in a bucket to a local file. +-----------------+------------------------------------+ | Parameter | Description | @@ -411,7 +411,7 @@ Downloads a remote object stored in a bucket to a local file. +-----------------+------------------------------------+ - **Usage example**: - ``lithops storage get -b ibm_cos cloudbucket test.txt`` + ``lithops storage get -b aws_s3 cloudbucket test.txt`` ``lithops storage delete `` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -437,13 +437,13 @@ Deletes objects from a given bucket. - **Usage examples**: - To delete a given object: - ``lithops storage delete -b ibm_cos cloudbucket test.txt`` + ``lithops storage delete -b aws_s3 cloudbucket test.txt`` - To delete all objects that start with a given prefix: - ``lithops storage delete -b ibm_cos cloudbucket -p test/`` + ``lithops storage delete -b aws_s3 cloudbucket -p test/`` - To delete all objects (empty the bucket): - ``lithops storage delete -b ibm_cos cloudbucket`` + ``lithops storage delete -b aws_s3 cloudbucket`` ``lithops storage list `` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -467,7 +467,7 @@ Lists objects from a given bucket. - **Usage examples**: - To list all objects in a bucket: - ``lithops storage list -b ibm_cos cloudbucket`` + ``lithops storage list -b aws_s3 cloudbucket`` - To list all objects that start with a given prefix: - ``lithops storage list -b ibm_cos cloudbucket -p test/`` + ``lithops storage list -b aws_s3 cloudbucket -p test/`` diff --git a/docs/source/comparing_lithops.rst b/docs/source/comparing_lithops.rst index f8d21e769..b7b1a0b5d 100644 --- a/docs/source/comparing_lithops.rst +++ b/docs/source/comparing_lithops.rst @@ -14,7 +14,7 @@ In 2018, the Lithops team forked PyWren to adapt it for **IBM Cloud Functions**, By September 2020, the IBM PyWren fork had diverged significantly. The maintainers rebranded the project as **Lithops**, reflecting its broader goals — including multi-cloud compatibility, improved developer experience, and support for modern Python environments and distributed computing patterns. -For more details, refer to the Middleware'18 industry paper: +For more details, refer to the Middleware 2018 industry paper: `Serverless Data Analytics in the IBM Cloud `_. Ray and Dask diff --git a/docs/source/compute_config/aliyun_functions.md b/docs/source/compute_config/aliyun_functions.md index 26f1a322a..a86ebbb05 100644 --- a/docs/source/compute_config/aliyun_functions.md +++ b/docs/source/compute_config/aliyun_functions.md @@ -18,7 +18,7 @@ python3 -m pip install lithops[aliyun] 2. [Access to your Function Compute dashboard](https://fc.console.aliyun.com/fc/overview), and choose your preferred region. -3. Access to the [Resource Access Management (RAM) Roles dashboard](https://ram.console.aliyun.com/roles/), and create a new Role that contains the `AliyunOSSFullAccess` permission. Alternatively you can use an already created Role that contains the `AliyunOSSFullAccess` permission. +3. Access the [Resource Access Management (RAM) Roles dashboard](https://ram.console.aliyun.com/roles/), and create a new Role that contains the `AliyunOSSFullAccess` permission. Alternatively you can use an already created Role that contains the `AliyunOSSFullAccess` permission. 4. Edit your Lithops config and add the following keys: @@ -65,7 +65,7 @@ aliyun_fc: ## Test Lithops -Once you have your compute and storage backends configured, you can run a hello world function with: +Once you have your compute and storage backends configured, you can run a Hello World function with: ```bash lithops hello -b aliyun_fc -s aliyun_oss diff --git a/docs/source/compute_config/aws_batch.md b/docs/source/compute_config/aws_batch.md index b5edc79a7..57c7292b8 100644 --- a/docs/source/compute_config/aws_batch.md +++ b/docs/source/compute_config/aws_batch.md @@ -62,7 +62,7 @@ In summary, you can use one of the following settings: 1. Provide the credentials via the `~/.aws/config` file, or set the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables. - You can run `aws configure` command if the AWS CLI is installed to setup the credentials. Then set in the Lithops config file: + If the AWS CLI is installed, you can run the `aws configure` command to set up the credentials. Then set them in the Lithops config file: ```yaml lithops: backend: aws_batch @@ -136,7 +136,7 @@ In summary, you can use one of the following settings: ## Test Lithops -Once you have your compute and storage backends configured, you can run a hello world function with: +Once you have your compute and storage backends configured, you can run a Hello World function with: ```bash lithops hello -b aws_batch -s aws_s3 diff --git a/docs/source/compute_config/aws_ec2.md b/docs/source/compute_config/aws_ec2.md index a107fd9c2..c30378d8a 100644 --- a/docs/source/compute_config/aws_ec2.md +++ b/docs/source/compute_config/aws_ec2.md @@ -68,7 +68,7 @@ In summary, you can use one of the following settings: 1. Provide the credentials via the `~/.aws/config` file, or set the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables. - You can run `aws configure` command if the AWS CLI is installed to setup the credentials. Then set in the Lithops config file: + If the AWS CLI is installed, you can run the `aws configure` command to set up the credentials. Then set them in the Lithops config file: ```yaml lithops: backend: aws_ec2 @@ -187,7 +187,7 @@ In summary, you can use one of the following settings: 1. Provide the credentials via the `~/.aws/config` file, or set the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables. - You can run `aws configure` command if the AWS CLI is installed to setup the credentials. Then set in the Lithops config file: + If the AWS CLI is installed, you can run the `aws configure` command to set up the credentials. Then set them in the Lithops config file: ```yaml lithops: backend: aws_ec2 @@ -240,7 +240,7 @@ In summary, you can use one of the following settings: ## Test Lithops -Once you have your compute and storage backends configured, you can run a hello world function with: +Once you have your compute and storage backends configured, you can run a Hello World function with: ```bash lithops hello -b aws_ec2 -s aws_s3 diff --git a/docs/source/compute_config/aws_lambda.md b/docs/source/compute_config/aws_lambda.md index f21f837c3..c08a84349 100644 --- a/docs/source/compute_config/aws_lambda.md +++ b/docs/source/compute_config/aws_lambda.md @@ -53,7 +53,7 @@ In summary, you can use one of the following settings: 1. Provide the credentials via the `~/.aws/config` file, or set the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables. - You can run `aws configure` command if the AWS CLI is installed to setup the credentials. Then set in the Lithops config file: + If the AWS CLI is installed, you can run the `aws configure` command to set up the credentials. Then set them in the Lithops config file: ```yaml lithops: backend: aws_lambda @@ -107,7 +107,7 @@ In summary, you can use one of the following settings: | aws_lambda | user_tags | {} | no | List of {name: ..., value: ...} pairs for Lambda instance user tags | | aws_lambda | env_vars | {} | no | List of {name: ..., value: ...} pairs for Lambda instance environment variables | | aws_lambda | namespace | | no | Virtual namespace. This can be useful to virtually group Lithops function workers. The functions deployed by lithops will be prefixed by this namespace. For example you can set it to differentiate between `prod`, `dev` and `stage` environments. | -| aws_lambda | runtime_include_function | False | no | If set to true, Lithops will automatically build a new runtime, including the function's code, instead of transferring it through the storage backend at invocation time. This is useful when the function's code size is large (in the order of 10s of MB) and the code does not change frequently | +| aws_lambda | runtime_include_function | False | no | If set to true, Lithops will automatically build a new runtime, including the function's code, instead of transferring it through the storage backend at invocation time. This is useful when the function's code size is large (on the order of tens of MB) and the code does not change frequently | ## Additional configuration @@ -163,7 +163,7 @@ For more information, check out [AWS documentation on attaching EFS volumes to L ## Test Lithops -Once you have your compute and storage backends configured, you can run a hello world function with: +Once you have your compute and storage backends configured, you can run a Hello World function with: ```bash lithops hello -b aws_lambda -s aws_s3 diff --git a/docs/source/compute_config/azure_containers.md b/docs/source/compute_config/azure_containers.md index cb1eeae80..0210145d0 100644 --- a/docs/source/compute_config/azure_containers.md +++ b/docs/source/compute_config/azure_containers.md @@ -22,9 +22,9 @@ az login Option 1: - 1. Access to the [Azure portal Resource Groups](https://portal.azure.com/#view/HubsExtension/BrowseResourceGroups) and create a new Resource group named **LithopsResourceGroup** (or similar) in your preferred region. If you already have a resource group, omit this step. + 1. Access the [Azure portal Resource Groups](https://portal.azure.com/#view/HubsExtension/BrowseResourceGroups) and create a new Resource group named **LithopsResourceGroup** (or similar) in your preferred region. If you already have a resource group, omit this step. - 2. Access to the [Azure portal Storage Accounts](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts) and create a new Storage Account with a unique name, for example: **lithops0sa25s1**. If you already have a storage account, omit this step. + 2. Access the [Azure portal Storage Accounts](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts) and create a new Storage Account with a unique name, for example: **lithops0sa25s1**. If you already have a storage account, omit this step. Option 2: @@ -53,11 +53,11 @@ az containerapp env create --name lithops --resource-group LithopsResourceGroup ## Configuration -1. Access to the [Storage Account](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts) +1. Access the [Storage Account](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts) 2. In the left menu, under the *Security + networking* section, click on *Access Keys* and copy the *Key 1* -3. Edit your lithops config and add the following keys: +3. Edit your Lithops config and add the following keys: ```yaml lithops: @@ -99,7 +99,7 @@ az containerapp env create --name lithops --resource-group LithopsResourceGroup |azure_containers| environment_id | |no | Full Azure resource ID of the Container Apps environment (for example: `/subscriptions//resourceGroups//providers/Microsoft.App/managedEnvironments/lithops`). If set, Lithops uses it directly and skips the `az containerapp env show` lookup | |azure_containers | docker_server | index.docker.io |no | Container registry URL | |azure_containers | docker_user | |no | Container registry user name | -|azure_containers | docker_password | |no | Container registry password/token. In case of Docker hub, login to your docker hub account and generate a new access token [here](https://hub.docker.com/settings/security)| +|azure_containers | docker_password | |no | Container registry password/token. For Docker Hub, log in to your Docker Hub account and generate a new access token [here](https://hub.docker.com/settings/security)| |azure_containers | max_workers | 1000 | no | Max number of parallel workers. Although Azure limits the number of parallel workers to 30, it is convenient to keep this value high| |azure_containers | worker_processes | 1 | no | Number of Lithops processes within a given worker. This can be used to parallelize function activations within a worker | |azure_containers| runtime | |no | Docker image name| @@ -107,12 +107,12 @@ az containerapp env create --name lithops --resource-group LithopsResourceGroup |azure_containers | runtime_timeout | 600 |no | Runtime timeout in seconds. Default 10 minutes | |azure_containers| trigger | pub/sub | no | Currently it supports pub/sub invocation| |azure_containers | invoke_pool_threads | 32 |no | Number of concurrent threads used for invocation | -|azure_containers | runtime_include_function | False | no | If set to true, Lithops will automatically build a new runtime, including the function's code, instead of transferring it through the storage backend at invocation time. This is useful when the function's code size is large (in the order of 10s of MB) and the code does not change frequently | +|azure_containers | runtime_include_function | False | no | If set to true, Lithops will automatically build a new runtime, including the function's code, instead of transferring it through the storage backend at invocation time. This is useful when the function's code size is large (on the order of tens of MB) and the code does not change frequently | ## Test Lithops -Once you have your compute and storage backends configured, you can run a hello world function with: +Once you have your compute and storage backends configured, you can run a Hello World function with: ```bash lithops hello -b azure_containers -s azure_storage diff --git a/docs/source/compute_config/azure_functions.md b/docs/source/compute_config/azure_functions.md index 7e6d673bd..9d3d1029d 100644 --- a/docs/source/compute_config/azure_functions.md +++ b/docs/source/compute_config/azure_functions.md @@ -22,9 +22,9 @@ az login Option 1: - 1. Access to the [Azure portal Resource Groups](https://portal.azure.com/#view/HubsExtension/BrowseResourceGroups) and create a new Resource group named **LithopsResourceGroup** (or similar) in your preferred region. If you already have a resource group, omit this step. + 1. Access the [Azure portal Resource Groups](https://portal.azure.com/#view/HubsExtension/BrowseResourceGroups) and create a new Resource group named **LithopsResourceGroup** (or similar) in your preferred region. If you already have a resource group, omit this step. - 2. Access to the [Azure portal Storage Accounts](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts) and create a new Storage Account with a unique name, for example: **lithops0sa25s1**. If you already have a storage account, omit this step. + 2. Access the [Azure portal Storage Accounts](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts) and create a new Storage Account with a unique name, for example: **lithops0sa25s1**. If you already have a storage account, omit this step. Option 2: @@ -45,11 +45,11 @@ az login ## Configuration -1. Access to the [Storage Account](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts) +1. Access the [Storage Account](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts) 2. In the left menu, under the *Security + networking* section, click on *Access Keys* and copy the *Key 1* -3. Edit your lithops config and add the following keys: +3. Edit your Lithops config and add the following keys: ```yaml lithops: @@ -97,7 +97,7 @@ az login ## Test Lithops -Once you have your compute and storage backends configured, you can run a hello world function with: +Once you have your compute and storage backends configured, you can run a Hello World function with: ```bash lithops hello -b azure_functions -s azure_storage diff --git a/docs/source/compute_config/azure_vms.md b/docs/source/compute_config/azure_vms.md index 2bfa7a5ed..3d974fbed 100644 --- a/docs/source/compute_config/azure_vms.md +++ b/docs/source/compute_config/azure_vms.md @@ -22,9 +22,9 @@ az login Option 1: - 1. Access to the [Azure portal Resource Groups](https://portal.azure.com/#view/HubsExtension/BrowseResourceGroups) and create a new Resource group named **LithopsResourceGroup** (or similar) in your preferred region. If you already have a resource group, omit this step. + 1. Access the [Azure portal Resource Groups](https://portal.azure.com/#view/HubsExtension/BrowseResourceGroups) and create a new Resource group named **LithopsResourceGroup** (or similar) in your preferred region. If you already have a resource group, omit this step. - 2. Access to the [Azure portal Storage Accounts](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts) and create a new Storage Account with a unique name, for example: **lithops0sa25s1**. If you already have a storage account, omit this step. + 2. Access the [Azure portal Storage Accounts](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts) and create a new Storage Account with a unique name, for example: **lithops0sa25s1**. If you already have a storage account, omit this step. Option 2: @@ -44,7 +44,7 @@ az login ``` ## Choose an operating system image for the VM -- Option 1: By default, Lithops uses an Ubuntu 22.04 image. In this case, no further action is required and you can continue to the next step. Lithops will install all required dependencies in the VM by itself. Note that this can take about 3 minutes to complete all installations. +- Option 1: By default, Lithops uses an Ubuntu 24.04 image. In this case, no further action is required and you can continue to the next step. Lithops will install all required dependencies in the VM by itself. Note that this can take about 3 minutes to complete all installations. - Option 2: Alternatively, you can use a pre-built custom image that will greatly improve VM creation time for Lithops jobs. To benefit from this approach, navigate to [runtime/azure_vms](https://github.com/lithops-cloud/lithops/tree/master/runtime/azure_vms), and follow the instructions. @@ -54,7 +54,7 @@ Alternatively, you can set the `reuse` mode to keep the started worker VMs runni ### Lithops configuration for the create or reuse modes -Edit your lithops config and add the relevant keys: +Edit your Lithops config and add the relevant keys: ```yaml lithops: @@ -107,7 +107,7 @@ In this mode, Lithops can start and stop an existing VM, and deploy an entire jo ### Lithops configuration for the consume mode -Edit your lithops config and add the relevant keys: +Edit your Lithops config and add the relevant keys: ```yaml lithops: @@ -150,7 +150,7 @@ Edit your lithops config and add the relevant keys: ## Test Lithops -Once you have your compute and storage backends configured, you can run a hello world function with: +Once you have your compute and storage backends configured, you can run a Hello World function with: ```bash lithops hello -b azure_vms -s azure_storage diff --git a/docs/source/compute_config/code_engine.md b/docs/source/compute_config/code_engine.md index 091f87f74..5c7d4a578 100644 --- a/docs/source/compute_config/code_engine.md +++ b/docs/source/compute_config/code_engine.md @@ -20,7 +20,7 @@ python3 -m pip install lithops[ibm] 4. Navigate to the [resource groups dashboard](https://cloud.ibm.com/account/resource-groups), and copy the desired resource group ID. -5. Edit your lithops config and add the following keys: +5. Edit your Lithops config and add the following keys: ```yaml lithops: @@ -52,7 +52,7 @@ python3 -m pip install lithops[ibm] |code_engine | region | | no | Cluster region. One of: `eu-gb`, `eu-de`, `us-south`, `us-east`, `br-sao`, `ca-tor`, `jp-tok`, `jp-osa`, `au-syd`. Lithops will use the `region` set under the `ibm` section if it is not set here | |code_engine | docker_server | docker.io |no | Container registry URL | |code_engine | docker_user | |no | Container registry user name | -|code_engine | docker_password | |no | Container registry password/token. In case of Docker hub, login to your docker hub account and generate a new access token [here](https://hub.docker.com/settings/security)| +|code_engine | docker_password | |no | Container registry password/token. For Docker Hub, log in to your Docker Hub account and generate a new access token [here](https://hub.docker.com/settings/security)| |code_engine | max_workers | 1000 | no | Max number of workers per `FunctionExecutor()`| |code_engine | worker_processes | 1 | no | Number of Lithops processes within a given worker. This can be used to parallelize function activations within a worker. It is recommended to set this value to the same number of CPUs as the container. | |code_engine | runtime | |no | Docker image name.| @@ -60,13 +60,13 @@ python3 -m pip install lithops[ibm] |code_engine | runtime_memory | 256 |no | Memory limit in MB. Default 256Mi. See [valid combinations](https://cloud.ibm.com/docs/codeengine?topic=codeengine-mem-cpu-combo) | |code_engine | runtime_timeout | 600 |no | Runtime timeout in seconds. Default 600 seconds | |code_engine | connection_retries | |no | If specified, number of job invoke retries in case of connection failure with error code 500 | -|code_engine | runtime_include_function | False | no | If set to true, Lithops will automatically build a new runtime, including the function's code, instead of transferring it through the storage backend at invocation time. This is useful when the function's code size is large (in the order of 10s of MB) and the code does not change frequently | +|code_engine | runtime_include_function | False | no | If set to true, Lithops will automatically build a new runtime, including the function's code, instead of transferring it through the storage backend at invocation time. This is useful when the function's code size is large (on the order of tens of MB) and the code does not change frequently | ## Runtime ### Use your own runtime -If a pre-built runtime is not provided, Lithops will automatically build the default runtime the first time you run a script. For this task it uses the **docker** command installed locally on your machine. To make this work, you need to: +If a pre-built runtime is not provided, Lithops will automatically build the default runtime the first time you run a script. For this task it uses the **Docker** command installed locally on your machine. To make this work, you need to: 1. [Install the Docker CE version](https://docs.docker.com/get-docker/). @@ -82,8 +82,8 @@ If you need to create a runtime with custom system packages and libraries, pleas ## Configure a private container registry for your runtime -### Configure Docker hub -To configure Lithops to access a private repository in your docker hub account, you need to extend the Code Engine config and add the following keys: +### Configure Docker Hub +To configure Lithops to access a private repository in your Docker Hub account, you need to extend the Code Engine config and add the following keys: ```yaml code_engine: @@ -108,7 +108,7 @@ code_engine: ## Test Lithops -Once you have your compute and storage backends configured, you can run a hello world function with: +Once you have your compute and storage backends configured, you can run a Hello World function with: ```bash lithops hello -b code_engine -s ibm_cos diff --git a/docs/source/compute_config/gcp_cloudrun.md b/docs/source/compute_config/gcp_cloudrun.md index 643659831..6540f3c6d 100644 --- a/docs/source/compute_config/gcp_cloudrun.md +++ b/docs/source/compute_config/gcp_cloudrun.md @@ -56,7 +56,7 @@ Replace `gcf-artifacts`, `us-east1`, `lithops-dev`, and the service account emai ## Configuration -1. Edit your lithops config and add the following keys: +1. Edit your Lithops config and add the following keys: ```yaml lithops: @@ -89,12 +89,12 @@ Replace `gcf-artifacts`, `us-east1`, `lithops-dev`, and the service account emai |gcp_cloudrun | runtime_timeout | 300 |no | Runtime timeout in seconds. Default 5 minutes | |gcp_cloudrun | trigger | https | no | Currently it supports 'https' trigger| |gcp_cloudrun | invoke_pool_threads | 100 |no | Number of concurrent threads used for invocation | -|gcp_cloudrun | runtime_include_function | False | no | If set to true, Lithops will automatically build a new runtime, including the function's code, instead of transferring it through the storage backend at invocation time. This is useful when the function's code size is large (in the order of 10s of MB) and the code does not change frequently | +|gcp_cloudrun | runtime_include_function | False | no | If set to true, Lithops will automatically build a new runtime, including the function's code, instead of transferring it through the storage backend at invocation time. This is useful when the function's code size is large (on the order of tens of MB) and the code does not change frequently | |gcp_cloudrun | docker_server | pkg.dev | no | Marker for [Artifact Registry](https://cloud.google.com/artifact-registry/docs/docker/names) default image names (`REGION-docker.pkg.dev/PROJECT/REPOSITORY/IMAGE`). | |gcp_cloudrun | artifact_registry_repository | lithops | no | Docker repository name in Artifact Registry (must exist under your project in the configured region). | ## Test Lithops -Once you have your compute and storage backends configured, you can run a hello world function with: +Once you have your compute and storage backends configured, you can run a Hello World function with: ```bash lithops hello -b gcp_cloudrun -s gcp_storage diff --git a/docs/source/compute_config/gcp_compute_engine.md b/docs/source/compute_config/gcp_compute_engine.md index 2f5ae07cc..1a2b6d242 100644 --- a/docs/source/compute_config/gcp_compute_engine.md +++ b/docs/source/compute_config/gcp_compute_engine.md @@ -143,7 +143,7 @@ gcp_compute_engine: ## Test Lithops -Once you have your compute and storage backends configured, you can run a hello world function with: +Once you have your compute and storage backends configured, you can run a Hello World function with: ```bash lithops hello -b gcp_compute_engine -s gcp_storage diff --git a/docs/source/compute_config/gcp_functions.md b/docs/source/compute_config/gcp_functions.md index 3506f09f8..ff961d5c4 100644 --- a/docs/source/compute_config/gcp_functions.md +++ b/docs/source/compute_config/gcp_functions.md @@ -54,7 +54,7 @@ gcloud services enable \ ## Configuration -1. Edit your lithops config and add the following keys: +1. Edit your Lithops config and add the following keys: ```yaml lithops: @@ -88,7 +88,7 @@ gcloud services enable \ ## Test Lithops -Once you have your compute and storage backends configured, you can run a hello world function with: +Once you have your compute and storage backends configured, you can run a Hello World function with: ```bash lithops hello -b gcp_functions -s gcp_storage diff --git a/docs/source/compute_config/ibm_cf.md b/docs/source/compute_config/ibm_cf.md index a07eccf5a..98bd4130f 100644 --- a/docs/source/compute_config/ibm_cf.md +++ b/docs/source/compute_config/ibm_cf.md @@ -24,7 +24,7 @@ python3 -m pip install lithops[ibm] 5. Navigate to the [resource groups dashboard](https://cloud.ibm.com/account/resource-groups), and copy the desired resource group ID. -6. Edit your lithops config and add the following keys: +6. Edit your Lithops config and add the following keys: ```yaml lithops: @@ -61,12 +61,12 @@ python3 -m pip install lithops[ibm] |ibm_cf | runtime_timeout | 600 |no | Runtime timeout in seconds. Default 600 seconds | |ibm_cf | invoke_pool_threads | 500 |no | Number of concurrent threads used for invocation | |ibm_cf | remote_invoker | False | no | Activate the remote invoker feature that uses one cloud function to spawn all the actual `map()` activations | -|ibm_cf | runtime_include_function | False | no | If set to true, Lithops will automatically build a new runtime, including the function's code, instead of transferring it through the storage backend at invocation time. This is useful when the function's code size is large (in the order of 10s of MB) and the code does not change frequently | +|ibm_cf | runtime_include_function | False | no | If set to true, Lithops will automatically build a new runtime, including the function's code, instead of transferring it through the storage backend at invocation time. This is useful when the function's code size is large (on the order of tens of MB) and the code does not change frequently | ## Test Lithops -Once you have your compute and storage backends configured, you can run a hello world function with: +Once you have your compute and storage backends configured, you can run a Hello World function with: ```bash lithops hello -b ibm_cf -s ibm_cos diff --git a/docs/source/compute_config/ibm_vpc.md b/docs/source/compute_config/ibm_vpc.md index 45e64701d..9d7070a60 100644 --- a/docs/source/compute_config/ibm_vpc.md +++ b/docs/source/compute_config/ibm_vpc.md @@ -30,7 +30,7 @@ Alternatively, you can set the `reuse` mode to keep the started worker VMs runni ### Lithops configuration for the *create* and *reuse* mode -Edit your lithops config and add the relevant keys: +Edit your Lithops config and add the relevant keys: ```yaml lithops: @@ -118,7 +118,7 @@ In this mode, Lithops can start and stop an existing VM, and deploy an entire jo ### Lithops configuration for the consume mode -Edit your lithops config and add the relevant keys: +Edit your Lithops config and add the relevant keys: ```yaml lithops: @@ -169,7 +169,7 @@ If you need to create a new VM, follow the steps below to create the VM and upda ## Test Lithops -Once you have your compute and storage backends configured, you can run a hello world function with: +Once you have your compute and storage backends configured, you can run a Hello World function with: ```bash lithops hello -b ibm_vpc -s ibm_cos diff --git a/docs/source/compute_config/knative.md b/docs/source/compute_config/knative.md index fb1f002e0..2e95338db 100644 --- a/docs/source/compute_config/knative.md +++ b/docs/source/compute_config/knative.md @@ -1,10 +1,10 @@ # Knative -Lithops with *Knative* as serverless compute backend. Lithops also supports vanilla Knative for running applications. The easiest way to make it working is to create an IBM Kubernetes (IKS) cluster through the [IBM dashboard](https://cloud.ibm.com/kubernetes/landing). Alternatively you can use your own kubernetes cluster or a kind/minikube installation. +Lithops with *Knative* as serverless compute backend. Lithops also supports vanilla Knative for running applications. The easiest way to make it work is to create an IBM Kubernetes (IKS) cluster through the [IBM dashboard](https://cloud.ibm.com/kubernetes/landing). Alternatively, you can use your own Kubernetes cluster or a kind/minikube installation. ## Installation -Note that Lithops automatically builds the default runtime the first time you run a script. For this task it uses the **docker** command installed locally in your machine. +Note that Lithops automatically builds the default runtime the first time you run a script. For this task it uses the **Docker** command installed locally on your machine. 1. Install Knative backend dependencies: @@ -32,7 +32,7 @@ python3 -m pip install lithops[knative] 7. Install a networking layer. Currently Lithops supports **Kourier**. [Follow these instructions to install Kourier.](https://knative.dev/docs/install/yaml-install/serving/install-serving-with-yaml/#install-a-networking-layer) -8. Edit your lithops config and add: +8. Edit your Lithops config and add: ```yaml knative: ingress_endpoint : http://127.0.0.1:80 @@ -45,7 +45,7 @@ python3 -m pip install lithops[knative] ### Option 2 - IBM IKS: -5. Access to the [IBM dashboard](https://cloud.ibm.com/kubernetes/landing) and create a new Kubernetes cluster. +5. Access the [IBM dashboard](https://cloud.ibm.com/kubernetes/landing) and create a new Kubernetes cluster. 6. Once the cluster is running, follow the instructions of the "Actions"--> "Connect via CLI" option of the dashboard to configure the *kubectl* client in your local machine. @@ -70,7 +70,7 @@ python3 -m pip install lithops[knative] export KUBECONFIG= ``` -9. Edit your lithops config and add the following keys: +9. Edit your Lithops config and add the following keys: ```yaml lithops: backend: knative @@ -78,15 +78,15 @@ python3 -m pip install lithops[knative] ### Configure a private container registry for your runtime -#### Configure Docker hub -To configure Lithops to access a private repository in your docker hub account, you need to extend the Knative config and add the following keys: +#### Configure Docker Hub +To configure Lithops to access a private repository in your Docker Hub account, you need to extend the Knative config and add the following keys: ```yaml knative: .... docker_server : docker.io - docker_user : - docker_password : + docker_user : + docker_password : ``` #### Configure IBM Container Registry @@ -110,7 +110,7 @@ knative: |knative | ingress_endpoint | |no | Ingress endpoint. Make sure to use http:// prefix | |knative | docker_server | docker.io |no | Container registry URL | |knative | docker_user | |no | Container registry user name | -|knative | docker_password | |no | Container registry password/token. In case of Docker hub, login to your docker hub account and generate a new access token [here](https://hub.docker.com/settings/security)| +|knative | docker_password | |no | Container registry password/token. For Docker Hub, log in to your Docker Hub account and generate a new access token [here](https://hub.docker.com/settings/security)| |knative | git_url | |no | Git repository to build the image | |knative | git_rev | |no | Git revision to build the image | |knative | max_workers | 100 | no | Max number of workers per `FunctionExecutor()`| @@ -135,10 +135,10 @@ knative: ## Test Lithops -Once you have your compute and storage backends configured, you can run a hello world function with: +Once you have your compute and storage backends configured, you can run a Hello World function with: ```bash -lithops hello -b knative -s ibm_cos +lithops hello -b knative -s aws_s3 ``` ## Viewing the execution logs diff --git a/docs/source/compute_config/kubernetes.md b/docs/source/compute_config/kubernetes.md index 6183e621e..081a082b8 100644 --- a/docs/source/compute_config/kubernetes.md +++ b/docs/source/compute_config/kubernetes.md @@ -35,15 +35,15 @@ python3 -m pip install lithops[kubernetes] ## Configure a private container registry for your runtime -### Configure Docker hub -To configure Lithops to access a private repository in your docker hub account, you need to extend the kubernetes config and add the following keys: +### Configure Docker Hub +To configure Lithops to access a private repository in your Docker Hub account, you need to extend the kubernetes config and add the following keys: ```yaml k8s: .... docker_server : docker.io - docker_user : - docker_password : + docker_user : + docker_password : ``` ### Configure IBM Container Registry @@ -67,7 +67,7 @@ k8s: |k8s | namespace | default |no | Kubernetes namespace to use for Lithops execution | |k8s | docker_server | docker.io |no | Container registry URL | |k8s | docker_user | |no | Container registry user name | -|k8s | docker_password | |no | Container registry password/token. In case of Docker hub, login to your docker hub account and generate a new access token [here](https://hub.docker.com/settings/security)| +|k8s | docker_password | |no | Container registry password/token. For Docker Hub, log in to your Docker Hub account and generate a new access token [here](https://hub.docker.com/settings/security)| |k8s | rabbitmq_executor | False | no | Alternative K8s backend that accelerates parallel function execution (map) by using RabbitMQ group calls and warm-state pods with higher granularity. For more information, see [here](./kubernetes_rabbitmq.md). | |k8s | max_workers | 100 | no | Max number of workers per `FunctionExecutor()`| |k8s | worker_processes | 1 | no | Number of Lithops processes within a given worker. This can be used to parallelize function activations within a worker. It is recommended to set this value to the same number of CPUs as the container. | @@ -106,10 +106,10 @@ Providing `container_security_context` fully replaces the defaults — copy the ## Test Lithops -Once you have your compute and storage backends configured, you can run a hello world function with: +Once you have your compute and storage backends configured, you can run a Hello World function with: ```bash -lithops hello -b k8s -s ibm_cos +lithops hello -b k8s -s aws_s3 ``` ## Viewing the execution logs diff --git a/docs/source/compute_config/kubernetes_rabbitmq.md b/docs/source/compute_config/kubernetes_rabbitmq.md index e2cd809a6..6cbd126bc 100644 --- a/docs/source/compute_config/kubernetes_rabbitmq.md +++ b/docs/source/compute_config/kubernetes_rabbitmq.md @@ -16,7 +16,7 @@ All of these changes are **ideal** for pipelines where launching **hundreds of p ## Installation -1. Install kubernetes backend dependencies: +1. Install Kubernetes backend dependencies: ```bash python3 -m pip install lithops[kubernetes] @@ -33,8 +33,8 @@ python3 -m pip install lithops[kubernetes] k8s: .... docker_server : docker.io - docker_user : - docker_password : + docker_user : + docker_password : .... rabbitmq_executor : True ``` @@ -107,7 +107,7 @@ In this scenario, it is evident that the invocation time is consistently reduced |k8s | namespace | default |no | Kubernetes namespace to use for lithops execution | |k8s | docker_server | docker.io |no | Container registry URL | |k8s | docker_user | |no | Container registry user name | -|k8s | docker_password | |no | Container registry password/token. In case of Docker hub, login to your docker hub account and generate a new access token [here](https://hub.docker.com/settings/security)| +|k8s | docker_password | |no | Container registry password/token. For Docker Hub, log in to your Docker Hub account and generate a new access token [here](https://hub.docker.com/settings/security)| |k8s | rabbitmq_executor | False | yes | Alternative K8s backend accelerating parallel function execution (map) thanks to rabbitmq group calls and warm-state pods of higher granularity.| |k8s | worker_processes | |no | CPUs per pod. This enables pod granularity. Default gets all CPUs of the nodes. | |k8s | runtime | |no | Docker image name.| @@ -116,10 +116,10 @@ In this scenario, it is evident that the invocation time is consistently reduced ## Test Lithops -Once you have your compute and storage backends configured, you can run a hello world function with: +Once you have your compute and storage backends configured, you can run a Hello World function with: ```bash -lithops hello -b k8s -s ibm_cos +lithops hello -b k8s -s aws_s3 ``` ## Viewing the execution logs diff --git a/docs/source/compute_config/localhost.md b/docs/source/compute_config/localhost.md index ce9076342..105a5f917 100644 --- a/docs/source/compute_config/localhost.md +++ b/docs/source/compute_config/localhost.md @@ -40,7 +40,7 @@ fexec = lithops.LocalhostExecutor() ### Container Environment -The Container environment runs the functions within a ``docker container``. In this case you must [install the Docker CE version](https://docs.docker.com/get-docker/) on your machine. This environment is automatically activated when you provide a Docker image as a runtime. For example, by adding the following keys in the config: +The Container environment runs the functions within a ``Docker container``. In this case you must [install the Docker CE version](https://docs.docker.com/get-docker/) on your machine. This environment is automatically activated when you provide a Docker image as a runtime. For example, by adding the following keys in the config: ```yaml localhost: @@ -71,7 +71,7 @@ In this mode of execution, you can use any Docker image that contains all the re ## Test Lithops -Once you have your compute and storage backends configured, you can run a hello world function with: +Once you have your compute and storage backends configured, you can run a Hello World function with: ```bash lithops hello -b localhost -s localhost diff --git a/docs/source/compute_config/openwhisk.md b/docs/source/compute_config/openwhisk.md index bbc2d5537..86027f501 100644 --- a/docs/source/compute_config/openwhisk.md +++ b/docs/source/compute_config/openwhisk.md @@ -80,11 +80,11 @@ Lithops with *OpenWhisk* as serverless compute backend. Lithops can also run fun |openwhisk | runtime_memory | 256 |no | Memory limit in MB. Default 256MB | |openwhisk | runtime_timeout | 600 |no | Runtime timeout in seconds. Default 10 minutes | |openwhisk | invoke_pool_threads | 500 |no | Number of concurrent threads used for invocation | -|openwhisk | runtime_include_function | False | no | If set to true, Lithops will automatically build a new runtime, including the function's code, instead of transferring it through the storage backend at invocation time. This is useful when the function's code size is large (in the order of 10s of MB) and the code does not change frequently | +|openwhisk | runtime_include_function | False | no | If set to true, Lithops will automatically build a new runtime, including the function's code, instead of transferring it through the storage backend at invocation time. This is useful when the function's code size is large (on the order of tens of MB) and the code does not change frequently | ## Test Lithops -Once you have your compute and storage backends configured, you can run a hello world function with: +Once you have your compute and storage backends configured, you can run a Hello World function with: ```bash lithops hello -b openwhisk -s ibm_cos diff --git a/docs/source/compute_config/oracle_functions.md b/docs/source/compute_config/oracle_functions.md index 8243dd3e6..2d9989069 100644 --- a/docs/source/compute_config/oracle_functions.md +++ b/docs/source/compute_config/oracle_functions.md @@ -2,7 +2,7 @@ Lithops with *Oracle Functions* as serverless compute backend. -**Note**: This is a beta backend. Please open an issue if you encounter any error/bug +**Note**: This is a beta backend. Please open an issue if you encounter any errors or bugs. ## Installation @@ -62,11 +62,12 @@ Now that the dynamic group is set up, you'll need to create a policy that allows ### Configure Lithops -Now, your Oracle Functions have the necessary permissions to manage resources in your Oracle Cloud Infrastructure tenancy. -1. Navigate to the [VCNs page](https://cloud.oracle.com/networking/vcns) and create a new VCN using the **VCN Wizard**. Then choose *create VCN with Internet Connectivity*. In the next page, you can uncheck `Use DNS hostnames in this VCN` and leave the rest of the parameters as provided by default. +Your Oracle Functions application now has the necessary permissions to manage resources in your Oracle Cloud Infrastructure tenancy. -2. The **VCN Wizard** will create all the necessary VCN resources, including the subnets. Now access the private subnet and copy the OCID to the `subnet_id` parameter under the `oracle_f` section of the configuration. +1. Navigate to the [VCNs page](https://cloud.oracle.com/networking/vcns) and create a new VCN using the **VCN Wizard**. Choose **Create VCN with Internet Connectivity**. On the next page, you can uncheck `Use DNS hostnames in this VCN` and leave the rest of the parameters at their defaults. + +2. The **VCN Wizard** creates all the necessary VCN resources, including the subnets. Open the private subnet and copy its OCID to the `subnet_id` parameter under the `oracle_f` section of the configuration. 3. Navigate to the [API keys page](https://cloud.oracle.com/identity/domains/my-profile/api-keys) and generate and download a new API signing key. Omit this step if you have already generated and downloaded a key. When you generate a new key, Oracle provides a sample config file with most of the parameters required by Lithops. Copy all the `key:value` pairs and configure Lithops as follows: @@ -88,7 +89,7 @@ oracle_f: docker_password: ``` -## Summary of configuration keys for Oracle: +## Summary of configuration keys for Oracle |Group|Key|Default|Mandatory|Additional info| |---|---|---|---|---| @@ -101,7 +102,7 @@ oracle_f: |oracle | tenancy_namespace | |no | Auto-generated Object Storage namespace string of the tenancy. You can find it [here](https://cloud.oracle.com/tenancy), under *Object storage namespace*| -## Summary of configuration keys for Oracle Functions : +## Summary of configuration keys for Oracle Functions |Group|Key|Default|Mandatory|Additional info| |---|---|---|---|---| @@ -115,11 +116,12 @@ oracle_f: |oracle_f | runtime | |no | Runtime name you built and deployed using the Lithops client| |oracle_f | runtime_memory | 256 |no | Memory limit in MB. Default 256MB | |oracle_f | runtime_timeout | 300 |no | Runtime timeout in seconds. Default 5 minutes | -|oracle_f | runtime_include_function | False | no | If set to true, Lithops will automatically build a new runtime, including the function's code, instead of transferring it through the storage backend at invocation time. This is useful when the function's code size is large (in the order of 10s of MB) and the code does not change frequently | +|oracle_f | runtime_include_function | False | no | If set to true, Lithops will automatically build a new runtime, including the function's code, instead of transferring it through the storage backend at invocation time. This is useful when the function's code size is large (on the order of tens of MB) and the code does not change frequently | ## Test Lithops -Once you have your compute and storage backends configured, you can run a hello world function with: + +Once you have your compute and storage backends configured, you can run a Hello World function with: ```bash lithops hello -b oracle_f -s oracle_oss diff --git a/docs/source/compute_config/singularity.md b/docs/source/compute_config/singularity.md index 513e7e687..8fc5f6732 100644 --- a/docs/source/compute_config/singularity.md +++ b/docs/source/compute_config/singularity.md @@ -53,7 +53,7 @@ Depending on your cluster setup, you might need to adjust permissions of the `.s ## Test Lithops -Once you have your compute and storage backends configured, you can run a hello world function with: +Once you have your compute and storage backends configured, you can run a Hello World function with: ```bash lithops hello -b singularity diff --git a/docs/source/compute_config/vm.md b/docs/source/compute_config/vm.md index c41b61726..00134089e 100644 --- a/docs/source/compute_config/vm.md +++ b/docs/source/compute_config/vm.md @@ -18,7 +18,7 @@ vm: ## Execution Environments -The virtual machine backend can run functions either using the default ``python3`` interpreter of the VM, or by using a ``docker container`` within the VM. The environment is automatically chosen depending on whether or not you provided a Docker image as a runtime. +The virtual machine backend can run functions either using the default ``python3`` interpreter of the VM, or by using a ``Docker container`` within the VM. The environment is automatically chosen depending on whether or not you provided a Docker image as a runtime. ### Default Environment The default environment runs the functions in the same ``python3`` interpreter that you ran the Lithops script. It does not require any extra configuration. You must ensure that your VM contains the same ``python3`` interpreter, and all the dependencies required by your Lithops app. So, once the backend is configured in the config file, you only need to create a ``FunctionExecutor`` to work with it: @@ -29,7 +29,7 @@ fexec = lithops.FunctionExecutor() ### Docker Environment -The Docker environment runs the functions within a ``docker container``. In this case you must [install the Docker CE version](https://docs.docker.com/get-docker/) in your VM. Note that in this case the container image must contain all the dependencies required by your Lithops app. This environment is automatically activated when you provide a docker image as a runtime. For example, by adding the following keys in the config: +The Docker environment runs the functions within a ``Docker container``. In this case you must [install the Docker CE version](https://docs.docker.com/get-docker/) in your VM. Note that in this case the container image must contain all the dependencies required by your Lithops app. This environment is automatically activated when you provide a Docker image as a runtime. For example, by adding the following keys in the config: ```yaml vm: @@ -42,7 +42,7 @@ or by using the ``runtime`` param in a function executor: fexec = lithops.FunctionExecutor(runtime='lithopscloud/ibmcf-python-v312') ``` -In this backend, you can use any docker image that contains all the required dependencies. For example, the IBM Cloud Functions and Knative runtimes are compatible with it. +In this backend, you can use any Docker image that contains all the required dependencies. For example, the IBM Cloud Functions and Knative runtimes are compatible with it. ## Summary of configuration keys for a single Virtual Machine: @@ -52,17 +52,17 @@ In this backend, you can use any docker image that contains all the required dep |vm | ssh_username | | yes | SSH username for accessing the VM | |vm | ssh_password | | no | SSH password for accessing the VM | |vm | ssh_key_filename | | no | Path to SSH key | -|vm | runtime | python3 |no | `python3` or a docker image name | +|vm | runtime | python3 |no | `python3` or a Docker image name | |vm | worker_processes | 1 | no | Number of Lithops processes within the VM. This can be used to parallelize function activations within the VM. It is recommended to set it to the same number of CPUs as the VM | |vm | extra_apt_packages | [] | no | Extra Debian/Ubuntu packages during Lithops setup on the VM (YAML list or space-separated string) | |vm | extra_python_packages | [] | no | Extra pip packages after Lithops on the VM (YAML list or space-separated string) | ## Test Lithops -Once you have your compute and storage backends configured, you can run a hello world function with: +Once you have your compute and storage backends configured, you can run a Hello World function with: ```bash -lithops hello -b vm -s ibm_cos +lithops hello -b vm -s aws_s3 ``` ## Viewing the execution logs diff --git a/docs/source/configuration.rst b/docs/source/configuration.rst index a0bbca4ac..f7b0ee032 100644 --- a/docs/source/configuration.rst +++ b/docs/source/configuration.rst @@ -7,13 +7,13 @@ To work with Lithops on the Cloud, you must configure at least one compute backe Lithops can work both with the leading cloud providers, and using on-premise or Kubernetes platforms. You have multiple options to choose compute and storage backends based on your needs. -Lithops configuration can be provided either using a **configuration file**, or in runtime via a **python dictionary.** +Lithops configuration can be provided either using a **configuration file**, or in runtime via a **Python dictionary.** Compute and Storage backends ---------------------------- -Choose your compute and storage engines from the table below: +Choose your compute and storage backends from the table below: +--------------------------------------------------------------------+--------------------------------------------------------------------+ | Compute backends | Storage Backends | @@ -23,7 +23,7 @@ Choose your compute and storage engines from the table below: || `IBM Code Engine `_ || `Google Cloud Storage `_ | || `AWS Lambda `_ || `Azure Blob Storage `_ | || `AWS Batch `_ || `Aliyun Object Storage Service `_ | -|| `Google Cloud Functions `_ || `Infinispan `_ | +|| `Google Cloud Run functions `_ || `Infinispan `_ | || `Google Cloud Run `_ || `Ceph `_ | || `Azure Functions `_ || `MinIO `_ | || `Azure Container Apps `_ || `Redis `_ | diff --git a/docs/source/contributing.rst b/docs/source/contributing.rst index 5a9f562c2..ba8eeb1c4 100644 --- a/docs/source/contributing.rst +++ b/docs/source/contributing.rst @@ -8,7 +8,7 @@ Reporting bugs and asking questions You can post questions or issues or feedback through the following channels: -1. `Github Discussions `_: For discussions about development, questions about usage, and feature requests. +1. `GitHub Discussions `_: For discussions about development, questions about usage, and feature requests. 2. `GitHub Issues `_: For bug reports and feature requests. diff --git a/docs/source/data_partitioning.rst b/docs/source/data_partitioning.rst index a99604dfc..cf54c25bb 100644 --- a/docs/source/data_partitioning.rst +++ b/docs/source/data_partitioning.rst @@ -28,13 +28,13 @@ stored in COS, split into chunks of a pre-determined size: size = int(6.7 * pow(2,20)) # ~6.7MiB - arbitrarily chosen chunk size in bytes fexec = lithops.FunctionExecutor() - fexec.map(line_counter_in_chunk, data_location,obj_chunk_size=size) + fexec.map(line_counter_in_chunk, data_location, obj_chunk_size=size) res = fexec.get_result() with open('logs/map_output', 'w') as f: f.write(str(res).replace('{','\n{')) -- To take full advantage of the test above (for the next topic), use a +- To take full advantage of the example above (for the next topic), use a file with a fixed number of rows repeated as a routine. You can create an example CSV file using the following function: @@ -129,8 +129,8 @@ implemented as follows: #. Since the first byte is, as a matter of fact, the last byte of the previous chunk, we inspect whether it is a newline (``\n``) or not. If - not, it means that the current chunk starts in the middle of a line - belonging in its entirety to the former chunk. In such a case, + not, it means that the current chunk starts in the middle of a line + belonging entirely to the former chunk. In such a case, position ``first_row_start_pos`` at the beginning of the next line. #. Because each chunk received an extra amount of bytes (the diff --git a/docs/source/dso.rst b/docs/source/dso.rst index b58714c99..1087d8611 100644 --- a/docs/source/dso.rst +++ b/docs/source/dso.rst @@ -13,7 +13,7 @@ Usage .. code:: lithops runtime build -f runtime/ibm_cf/Docker.dso id/runtime:tag - lithops runtime create id/runtime:tag + lithops runtime deploy id/runtime:tag 2. Create a DSO server in the cloud following the instructions available here: https://github.com/crucial-project/dso/tree/2.0 diff --git a/docs/source/execution_modes.rst b/docs/source/execution_modes.rst index 8ea31a0a8..6c18286fb 100644 --- a/docs/source/execution_modes.rst +++ b/docs/source/execution_modes.rst @@ -21,7 +21,7 @@ as it eliminates the need to deploy code to a cloud environment during the development phase. -.. note:: This is the preferable option for starting with Lithops, and for testing (debugging) your applications. +.. note:: This is the preferred option for starting with Lithops, and for testing (debugging) your applications. .. code:: python @@ -34,8 +34,8 @@ development phase. Serverless mode --------------- The "serverless mode" in Lithops is designed to execute functions using publicly -accessible serverless compute services, including IBM Cloud Functions, Amazon Lambda, -Google Cloud Functions, and more, enabling parallel task execution in isolated cloud +accessible serverless compute services, including IBM Cloud Functions, AWS Lambda, +Google Cloud Run functions, and more, enabling parallel task execution in isolated cloud environments. In serverless mode, Lithops leverages the power of these serverless platforms to execute @@ -77,7 +77,7 @@ By leveraging the automatic creation and configuration of VMs provided by Lithop developers can focus on writing their functions while Lithops takes care of the underlying infrastructure. -.. note:: This is the preferable option if your application (or a part) requires a more powerful environment than the ones provided by the Serverless backends (in terms of CPU and Memory). +.. note:: This is the preferred option if your application (or a part) requires a more powerful environment than the ones provided by the Serverless backends (in terms of CPU and Memory). .. code:: python diff --git a/docs/source/functions.md b/docs/source/functions.md index c7f1f4a46..19911d622 100644 --- a/docs/source/functions.md +++ b/docs/source/functions.md @@ -12,17 +12,17 @@ Reserved parameters are only accessible when using the [Futures API](./api_futur - **obj**: This parameter is used to activate the internal logic that allows you to process data objects stored in object storage or at public URLs in a transparent way. See the [data processing](data_processing.rst) documentation for more details and instructions on how to use this built-in data-processing logic. -- **storage**: To get a ready-to use lithops.storage.Storage() instance. This allows you to access your storage backend defined in configuration from any function in an easy way, for example: [storage_arg.py](https://github.com/lithops-cloud/lithops/blob/master/examples/storage_arg.py) +- **storage**: To get a ready-to-use `lithops.storage.Storage()` instance. This allows you to access your storage backend defined in configuration from any function in an easy way, for example: [storage_arg.py](https://github.com/lithops-cloud/lithops/blob/master/examples/storage_arg.py) -- **rabbitmq**: To get a ready-to use [pika.BlockingConnection()](https://pika.readthedocs.io/en/latest/modules/adapters/blocking.html) instance (AMQP URL must be set in the configuration to make it working). This allows you to access the RabbitMQ service from any function in an easy way, for example: [rabbitmq_arg.py](https://github.com/lithops-cloud/lithops/blob/master/examples/rabbitmq_arg.py) +- **rabbitmq**: To get a ready-to-use [pika.BlockingConnection()](https://pika.readthedocs.io/en/latest/modules/adapters/blocking.html) instance (AMQP URL must be set in the configuration to make it work). This allows you to access the RabbitMQ service from any function in an easy way, for example: [rabbitmq_arg.py](https://github.com/lithops-cloud/lithops/blob/master/examples/rabbitmq_arg.py) Parameters in the call_async() method ------------------------------------- -You can send multiple parameters to a single call function writing them into a list. The parameters will be mapped in -the order you wrote them. In the following example the x parameter will take the value 3 and the y parameter will +You can send multiple parameters to a single-call function by writing them into a list. The parameters will be mapped in +the order you wrote them. In the following example, the `x` parameter will take the value 3 and the `y` parameter will take the value 6. ```python @@ -109,7 +109,7 @@ To test all of the previous examples run the [multiple_args_call_async.py](https Parameters in the map() and map_reduce() methods ------------------------------------------------ -The 'iterdata' variable must be always a list []. In this case to send multiple parameters to the function, the parameters of +The 'iterdata' variable must always be a list []. In this case to send multiple parameters to the function, the parameters of each function must be enclosed within a tuple () as in the next example. The parameters will be mapped in the order you wrote them. @@ -212,7 +212,7 @@ print(fexec.get_result()) ``` -Common parameters across functions invocations +Common parameters across function invocations ---------------------------------------------- Sometimes, functions have common parameters for all the invocations. In this case you have two options to proceed: diff --git a/docs/source/monitoring.rst b/docs/source/monitoring.rst index e5a98ddf2..7247b0978 100644 --- a/docs/source/monitoring.rst +++ b/docs/source/monitoring.rst @@ -4,11 +4,11 @@ Monitoring By default, Lithops uses the storage backend to monitor function activations: each function activation stores a file named *{id}/status.json* in Object Storage when it finishes its execution. This file contains statistics about the execution, including whether the function activation ran successfully or not. Based on these files, the default -monitoring approach polls the Object Store every few seconds to determine which function activations have finished +monitoring approach polls the Object Storage every few seconds to determine which function activations have finished and which have not. Since this default approach can slow down total application execution time due to the number of requests it has to -make against the object store, Lithops also integrates a RabbitMQ service to monitor function activations in real time. +make against the object storage, Lithops also integrates a RabbitMQ service to monitor function activations in real time. With RabbitMQ, the content of the *{id}/status.json* file is sent through a queue. This speeds up total application execution time since Lithops only needs one connection to the messaging service to monitor all function activations. We currently support the AMQP protocol. diff --git a/docs/source/notebooks/function_chaining.ipynb b/docs/source/notebooks/function_chaining.ipynb index e3b5bd9f1..137b8b93a 100644 --- a/docs/source/notebooks/function_chaining.ipynb +++ b/docs/source/notebooks/function_chaining.ipynb @@ -12,14 +12,14 @@ "\n", "Function chaining is a pattern where multiple functions are called on the same executor consecutively. Using the\n", "same `lithops.FunctionExecutor` object reference, multiple functions can be invoked. It increases the readability of the\n", - "code and means less redundancy. This means we chain multiple functions together with the same element reference. It’s\n", + "code and means less redundancy. This means we chain multiple functions together with the same element reference. It\u2019s\n", "not necessary to attach the `lithops.FunctionExecutor` reference multiple times for each function call.\n", "\n", - "This patter is specially useful when the output of one invocation is the input of another invocation. In this case,\n", - "Lithops does not download the intermediate results to the local client, instead, the intermediate results are directly\n", - "read from the next function.\n", + "This pattern is especially useful when the output of one invocation is the input of another invocation. In this case,\n", + "Lithops does not download the intermediate results to the local client; instead, the intermediate results are read\n", + "directly by the next function.\n", "\n", - "It currently works with the Futures API, and you can chain the `map()`, `map_reuce()`, `wait()` and `get_result()`\n", + "It currently works with the Futures API, and you can chain the `map()`, `map_reduce()`, `wait()`, and `get_result()`\n", "methods. Note that the returning value of one function must match the signature of the next function when chaining\n", "multiple `map()` calls. View the next examples:\n", "\n", @@ -28,7 +28,6 @@ }, { "cell_type": "code", - "execution_count": 1, "metadata": { "collapsed": false, "jupyter": { @@ -38,15 +37,6 @@ "name": "#%%\n" } }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[2, 4, 6]\n" - ] - } - ], "source": [ "import lithops\n", "\n", @@ -60,6 +50,15 @@ "fexec = lithops.FunctionExecutor()\n", "res = fexec.map(my_func1, iterdata).get_result()\n", "print(res)" + ], + "execution_count": 1, + "outputs": [ + { + "output_type": "stream", + "text": [ + "[2, 4, 6]\n" + ] + } ] }, { @@ -75,7 +74,6 @@ }, { "cell_type": "code", - "execution_count": 2, "metadata": { "collapsed": false, "jupyter": { @@ -85,15 +83,6 @@ "name": "#%%\n" } }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[7, 9, 11]\n" - ] - } - ], "source": [ "import lithops\n", "\n", @@ -111,6 +100,15 @@ "fexec = lithops.FunctionExecutor()\n", "res = fexec.map(my_func1, iterdata).map(my_func2).get_result()\n", "print(res)" + ], + "execution_count": 2, + "outputs": [ + { + "output_type": "stream", + "text": [ + "[7, 9, 11]\n" + ] + } ] }, { @@ -126,7 +124,6 @@ }, { "cell_type": "code", - "execution_count": 3, "metadata": { "collapsed": false, "jupyter": { @@ -136,15 +133,6 @@ "name": "#%%\n" } }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[15, 16, 17]\n" - ] - } - ], "source": [ "def my_func1(x):\n", " return x + 2, 5\n", @@ -163,6 +151,15 @@ "fexec = lithops.FunctionExecutor()\n", "res = fexec.map(my_func1, iterdata).map(my_func2).map(my_func3).get_result()\n", "print(res)" + ], + "execution_count": 3, + "outputs": [ + { + "output_type": "stream", + "text": [ + "[15, 16, 17]\n" + ] + } ] }, { @@ -176,7 +173,6 @@ }, { "cell_type": "code", - "execution_count": 4, "metadata": { "collapsed": false, "jupyter": { @@ -186,15 +182,6 @@ "name": "#%%\n" } }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[15, 16, 17]\n" - ] - } - ], "source": [ "def my_func1(x):\n", " return x + 2, 5\n", @@ -217,6 +204,15 @@ "final_result = fexec.get_result()\n", "\n", "print(final_result)\n" + ], + "execution_count": 4, + "outputs": [ + { + "output_type": "stream", + "text": [ + "[15, 16, 17]\n" + ] + } ] } ], @@ -236,7 +232,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.8.10" + "version": "3.12.0" } }, "nbformat": 4, diff --git a/docs/source/sklearn_joblib.rst b/docs/source/sklearn_joblib.rst index d2824d008..fbe7ed3a1 100644 --- a/docs/source/sklearn_joblib.rst +++ b/docs/source/sklearn_joblib.rst @@ -10,14 +10,14 @@ To get started, first install Lithops and the joblib dependencies with: python3 -m pip install lithops[joblib] -Once installed, use ``from lithops.util.joblib import register_lithops`` and run ``register_lithops()``. This will register Lithops as a joblib backend for scikit-learn to use. Then run your original scikit-learn code inside with ``joblib.parallel_backend('lithops')``. +Once installed, use ``from lithops.util.joblib import register_lithops`` and run ``register_lithops()``. This will register Lithops as a joblib backend for scikit-learn to use. Then run your original scikit-learn code with ``joblib.parallel_backend('lithops')``. -Refer to the official `JobLib `_ and `SkLearn `_ documentation to operate with these libraries. +Refer to the official `joblib `_ and `scikit-learn `_ documentation to use these libraries. Examples -------- -- JobLib Lithops backend example +- joblib Lithops backend example .. code:: python @@ -39,7 +39,7 @@ Examples Parallel()(delayed(my_function)(i) for i in range(10)) -- SkLearn example with Lithops as backend for JobLib +- scikit-learn example with Lithops as backend for joblib .. code:: python diff --git a/docs/source/storage_config/aliyun_oss.md b/docs/source/storage_config/aliyun_oss.md index 51a1e1833..93cfad518 100644 --- a/docs/source/storage_config/aliyun_oss.md +++ b/docs/source/storage_config/aliyun_oss.md @@ -15,7 +15,7 @@ python3 -m pip install lithops[aliyun] 1. [Navigate to the Cloud Console](https://ram.console.aliyun.com/manage/ak) and create a new AccessKey (If you don't have one) -2. Edit your lithops config and add the following keys: +2. Edit your Lithops config and add the following keys: ```yaml lithops: diff --git a/docs/source/storage_config/aws_s3.md b/docs/source/storage_config/aws_s3.md index 7f2304170..96d137bc9 100644 --- a/docs/source/storage_config/aws_s3.md +++ b/docs/source/storage_config/aws_s3.md @@ -27,7 +27,7 @@ In summary, you can use one of the following settings: 1. Provide the credentials via the `~/.aws/config` file, or set the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables. - You can run `aws configure` command if the AWS CLI is installed to setup the credentials. Then set in the Lithops config file: + If the AWS CLI is installed, you can run the `aws configure` command to set up the credentials. Then set them in the Lithops config file: ```yaml lithops: storage: aws_s3 diff --git a/docs/source/storage_config/azure_blob.md b/docs/source/storage_config/azure_blob.md index 4e0fa2624..e26a524e5 100644 --- a/docs/source/storage_config/azure_blob.md +++ b/docs/source/storage_config/azure_blob.md @@ -15,7 +15,7 @@ $ python3 -m pip install lithops[azure] Option 1: - 1. Access to the [Azure portal Resource Groups](https://portal.azure.com/#blade/HubsExtension/BrowseResourceGroups) and create a new Resource group named **LithopsResourceGroup** in your preferred region. If you already have a resource group, omit this step. + 1. Access the [Azure portal Resource Groups](https://portal.azure.com/#blade/HubsExtension/BrowseResourceGroups) and create a new Resource group named **LithopsResourceGroup** in your preferred region. If you already have a resource group, omit this step. 2. Access the [Azure portal Storage Accounts](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts) and create a new Storage Account with a unique name, for example: **lithops0sa25s1**. If you already have a storage account, omit this step. @@ -46,11 +46,11 @@ $ python3 -m pip install lithops[azure] ## Configuration -1. Access to the [Storage Account](https://portal.azure.com/#blade/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts) +1. Access the [Storage Account](https://portal.azure.com/#blade/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts) 2. In the left menu, under the *Security + networking* section, click on *Access Keys* and copy the *Key 1* -3. Edit your lithops config and add the following keys: +3. Edit your Lithops config and add the following keys: ```yaml lithops: diff --git a/docs/source/storage_config/ceph.md b/docs/source/storage_config/ceph.md index b21588419..1abb3d22e 100644 --- a/docs/source/storage_config/ceph.md +++ b/docs/source/storage_config/ceph.md @@ -18,7 +18,7 @@ python3 -m pip install lithops[ceph] ## Configuration -1. Edit your lithops config file and add the following keys: +1. Edit your Lithops config file and add the following keys: ```yaml lithops: diff --git a/docs/source/storage_config/gcp_storage.md b/docs/source/storage_config/gcp_storage.md index 7d6e24a80..b88db9e40 100644 --- a/docs/source/storage_config/gcp_storage.md +++ b/docs/source/storage_config/gcp_storage.md @@ -28,7 +28,7 @@ python3 -m pip install lithops[gcp] 6. Click on *Continue*. Then, click on *Create key*. Select *JSON* and then *Create*. Download the JSON file to a secure location on your computer. Click *Done*. -7. Edit your lithops config file and add the following keys: +7. Edit your Lithops config file and add the following keys: ```yaml lithops: diff --git a/docs/source/storage_config/ibm_cos.md b/docs/source/storage_config/ibm_cos.md index 4ccfb8f3d..a8648202b 100644 --- a/docs/source/storage_config/ibm_cos.md +++ b/docs/source/storage_config/ibm_cos.md @@ -28,7 +28,7 @@ python3 -m pip install lithops[ibm] 5. Click `View credentials` and copy the *apikey* value. -6. Edit your lithops config file and add the following keys: +6. Edit your Lithops config file and add the following keys: ```yaml lithops: @@ -54,7 +54,7 @@ python3 -m pip install lithops[ibm] 7. When using HMAC credentials, you can omit providing a storage bucket, since Lithops will be able to create it automatically. -8. Edit your lithops config file and add the following keys: +8. Edit your Lithops config file and add the following keys: ```yaml lithops: @@ -75,7 +75,7 @@ python3 -m pip install lithops[ibm] 4. Copy the generated IAM API key (You can only see the key the first time you create it, so make sure to copy it). -5. Edit your lithops config file and add the following keys: +5. Edit your Lithops config file and add the following keys: ```yaml lithops: diff --git a/docs/source/storage_config/infinispan.md b/docs/source/storage_config/infinispan.md index e9a3d76cf..bdae6465c 100644 --- a/docs/source/storage_config/infinispan.md +++ b/docs/source/storage_config/infinispan.md @@ -11,7 +11,7 @@ Lithops with Infinispan as storage backend. Infinispan provides two different en ## Configuration -Edit your lithops config file and add the following keys: +Edit your Lithops config file and add the following keys: ### REST endpoint ```yaml diff --git a/docs/source/storage_config/minio.md b/docs/source/storage_config/minio.md index 6fb46c3d2..4b6fb1f61 100644 --- a/docs/source/storage_config/minio.md +++ b/docs/source/storage_config/minio.md @@ -17,7 +17,7 @@ python3 -m pip install lithops[minio] ## Configuration -Edit your lithops config file and add the following keys: +Edit your Lithops config file and add the following keys: ```yaml lithops: diff --git a/docs/source/storage_config/oracle_oss.md b/docs/source/storage_config/oracle_oss.md index 2a0ab147c..d00458279 100644 --- a/docs/source/storage_config/oracle_oss.md +++ b/docs/source/storage_config/oracle_oss.md @@ -2,7 +2,7 @@ Lithops with Oracle Object Storage as storage backend. -**Note**: This is a beta backend. Please open an issue if you encounter any error/bug +**Note**: This is a beta backend. Please open an issue if you encounter any errors or bugs. ## Installation diff --git a/docs/source/storage_config/redis.md b/docs/source/storage_config/redis.md index bbdf91487..662c784bb 100644 --- a/docs/source/storage_config/redis.md +++ b/docs/source/storage_config/redis.md @@ -18,7 +18,7 @@ python3 -m pip install lithops[redis] ## Configuration -Edit your lithops config file and add the following keys: +Edit your Lithops config file and add the following keys: ```yaml lithops: diff --git a/docs/source/storage_config/swift.md b/docs/source/storage_config/swift.md index dd70e53f9..236189c90 100644 --- a/docs/source/storage_config/swift.md +++ b/docs/source/storage_config/swift.md @@ -13,7 +13,7 @@ Lithops with OpenStack Swift as storage backend. ## Configuration -1. Edit your lithops config file and add the following keys: +1. Edit your Lithops config file and add the following keys: ```yaml lithops: diff --git a/docs/source/supported_clouds.rst b/docs/source/supported_clouds.rst index fc5c21df9..59a44a23b 100644 --- a/docs/source/supported_clouds.rst +++ b/docs/source/supported_clouds.rst @@ -34,7 +34,7 @@ Currently, Lithops for AWS supports these backends: - `AWS S3 `_ * - `AWS Batch `_ - - * - `AWS Elastic Cloud Compute (EC2) `_ + * - `AWS Elastic Compute Cloud (EC2) `_ - Google Cloud @@ -68,7 +68,7 @@ Currently, Lithops for Microsoft Azure supports these backends: - Storage * - `Azure Functions `_ - `Azure Blob Storage `_ - * - `Azure Container APPs `_ + * - `Azure Container Apps `_ - * - `Azure Virtual Machines `_ - diff --git a/lithops/serverless/backends/aws_batch/aws_batch.py b/lithops/serverless/backends/aws_batch/aws_batch.py index ccc80edc2..9c7fa3930 100644 --- a/lithops/serverless/backends/aws_batch/aws_batch.py +++ b/lithops/serverless/backends/aws_batch/aws_batch.py @@ -105,7 +105,7 @@ def _format_jobdef_name(self, runtime_name, runtime_memory, version=__version__) return f'{package}--{self.env_type}--{fmt_runtime_name}--{runtime_memory}mb' def _unformat_jobdef_name(self, jobdef_name): - # Default jobdef name is "lithops_v2-7-2_WU5O--FARGATE_SPOT--batch-default-runtime-v39--latest--1024mb" + # Default jobdef name is "lithops_v2-7-2_WU5O--FARGATE_SPOT--batch-default-runtime-v310--latest--1024mb" prefix, env_type, runtime = jobdef_name.split('--', 2) version = prefix.replace('lithops_v', '').split('_')[0].replace('-', '.') runtime_name, memory = runtime.rsplit("--", 1) diff --git a/lithops/serverless/backends/aws_batch/config.py b/lithops/serverless/backends/aws_batch/config.py index 5d2e94014..02b5a2360 100644 --- a/lithops/serverless/backends/aws_batch/config.py +++ b/lithops/serverless/backends/aws_batch/config.py @@ -67,7 +67,7 @@ psutil # Copy Lithops proxy and lib to the container image. -ENV APP_HOME /lithops +ENV APP_HOME=/lithops WORKDIR $APP_HOME COPY lithops_aws_batch.zip . diff --git a/lithops/serverless/backends/code_engine/config.py b/lithops/serverless/backends/code_engine/config.py index 724b72dba..4d65abefa 100644 --- a/lithops/serverless/backends/code_engine/config.py +++ b/lithops/serverless/backends/code_engine/config.py @@ -65,13 +65,13 @@ tblib \ psutil -ENV PORT 8080 -ENV CONCURRENCY 1 -ENV TIMEOUT 600 -ENV PYTHONUNBUFFERED TRUE +ENV PORT=8080 +ENV CONCURRENCY=1 +ENV TIMEOUT=600 +ENV PYTHONUNBUFFERED=TRUE # Copy Lithops proxy and lib to the container image. -ENV APP_HOME /lithops +ENV APP_HOME=/lithops WORKDIR $APP_HOME COPY lithops_codeengine.zip . diff --git a/lithops/serverless/backends/knative/config.py b/lithops/serverless/backends/knative/config.py index 81c7b4aa0..bafaf4620 100644 --- a/lithops/serverless/backends/knative/config.py +++ b/lithops/serverless/backends/knative/config.py @@ -64,13 +64,13 @@ tblib \ psutil -ENV PORT 8080 -ENV CONCURRENCY 1 -ENV TIMEOUT 600 -ENV PYTHONUNBUFFERED TRUE +ENV PORT=8080 +ENV CONCURRENCY=1 +ENV TIMEOUT=600 +ENV PYTHONUNBUFFERED=TRUE # Copy Lithops proxy and lib to the container image. -ENV APP_HOME /lithops +ENV APP_HOME=/lithops WORKDIR $APP_HOME COPY lithops_knative.zip . @@ -164,9 +164,9 @@ name: lithops-git params: - name: pathToDockerFile - value: lithops/compute/backends/knative/tekton/Dockerfile.python38 + value: lithops/compute/backends/knative/tekton/Dockerfile - name: imageUrl - value: docker.io/jsampe/lithops-knative-v38 + value: docker.io/jsampe/lithops-knative-default-runtime - name: imageTag value: latest """ diff --git a/lithops/serverless/backends/knative/knative.py b/lithops/serverless/backends/knative/knative.py index 1e662817f..7621f7f23 100644 --- a/lithops/serverless/backends/knative/knative.py +++ b/lithops/serverless/backends/knative/knative.py @@ -288,9 +288,8 @@ def _build_default_runtime_from_git(self, runtime_name): task_run = yaml.safe_load(config.task_run) task_run['spec']['inputs']['params'] = [] - python_version = utils.CURRENT_PY_VERSION.replace('.', '') path_to_dockerfile = {'name': 'pathToDockerFile', - 'value': 'lithops/compute/backends/knative/tekton/Dockerfile.python{}'.format(python_version)} + 'value': 'lithops/compute/backends/knative/tekton/Dockerfile'} task_run['spec']['inputs']['params'].append(path_to_dockerfile) image_url = {'name': 'imageUrl', 'value': '/'.join([cr, image_name])} diff --git a/lithops/serverless/backends/knative/tekton/Dockerfile.python38 b/lithops/serverless/backends/knative/tekton/Dockerfile similarity index 91% rename from lithops/serverless/backends/knative/tekton/Dockerfile.python38 rename to lithops/serverless/backends/knative/tekton/Dockerfile index 186ea7d4b..f3fca0c52 100644 --- a/lithops/serverless/backends/knative/tekton/Dockerfile.python38 +++ b/lithops/serverless/backends/knative/tekton/Dockerfile @@ -1,4 +1,4 @@ -FROM python:3.8.2-slim-bookworm +FROM python:3.10-slim-bookworm RUN apt-get update && apt-get install -y \ zip \ @@ -18,7 +18,7 @@ RUN pip install --upgrade setuptools six pip \ numpy # Copy Lithops proxy and lib to the container image. -ENV APP_HOME /lithops +ENV APP_HOME=/lithops WORKDIR $APP_HOME COPY lithops ./lithops diff --git a/runtime/README.md b/runtime/README.md index c7a05a071..0a106b139 100644 --- a/runtime/README.md +++ b/runtime/README.md @@ -6,7 +6,7 @@ Choose your compute backend: 2. [IBM VPC](ibm_vpc/) 3. [AWS Lambda](aws_lambda/) 4. [AWS EC2](aws_ec2/) -5. [Google Cloud Functions](gcp_functions/) +5. [Google Cloud Run functions](gcp_functions/) 6. [Google Cloud Run](gcp_cloudrun/) 7. [Google Compute Engine](gcp_compute_engine/) 8. [Azure Functions](azure_functions/) diff --git a/runtime/aliyun_fc/README.md b/runtime/aliyun_fc/README.md index 882c797e3..05b444c8b 100644 --- a/runtime/aliyun_fc/README.md +++ b/runtime/aliyun_fc/README.md @@ -30,7 +30,7 @@ pw = lithops.FunctionExecutor(runtime_memory=512) If you require additional Python modules not included in the default runtime, you can create your own custom Lithops runtime incorporating them. To create a custom runtime, compile all the necessary modules into a `requirements.txt` file. -For instance, if you wish to integrate the `matplotlib` module into your runtime, which isn't part of the default setup, you need to append it to the existing [requirements.txt](requirements.txt) file. Note that this `requirements.txt` contains the mandatory packages required by lithops, so you don't have to remove any of them from the list, but just add your packages at the end. +For instance, if you wish to integrate the `matplotlib` module into your runtime, which isn't part of the default setup, you need to append it to the existing [requirements.txt](requirements.txt) file. Note that this `requirements.txt` contains the mandatory packages required by Lithops, so you don't have to remove any of them from the list, but just add your packages at the end. After updating the file accordingly, you can proceed to build the custom runtime by specifying the modified `requirements.txt` file along with a chosen runtime name: diff --git a/runtime/aws_batch/Dockerfile b/runtime/aws_batch/Dockerfile index c6923f831..5c263d04a 100644 --- a/runtime/aws_batch/Dockerfile +++ b/runtime/aws_batch/Dockerfile @@ -1,14 +1,11 @@ -# Python 3.8 -#FROM python:3.8-slim-bookworm - -# Python 3.9 -#FROM python:3.9-slim-bookworm - # Python 3.10 FROM python:3.10-slim-bookworm +# Python 3.11 +#FROM python:3.11-slim-bookworm + RUN apt-get update && apt-get install -y \ zip \ && rm -rf /var/lib/apt/lists/* @@ -30,7 +27,7 @@ RUN pip install --upgrade --ignore-installed setuptools six pip \ psutil # Copy Lithops proxy and lib to the container image. -ENV APP_HOME /lithops +ENV APP_HOME=/lithops WORKDIR $APP_HOME COPY lithops_aws_batch.zip . diff --git a/runtime/aws_batch/conda.Dockerfile b/runtime/aws_batch/conda.Dockerfile index f186cced0..7bb60c972 100644 --- a/runtime/aws_batch/conda.Dockerfile +++ b/runtime/aws_batch/conda.Dockerfile @@ -1,13 +1,13 @@ -# Python 3.8 -#FROM continuumio/miniconda3:4.9.2 +# Python 3.10 +#FROM continuumio/miniconda3:23.3.1-0 -# Python 3.9 -FROM continuumio/miniconda3:4.10.3 +# Python 3.11 +FROM continuumio/miniconda3:23.10.0-1 # YOU MUST PIN THE PYTHON VERSION TO PREVENT IT TO BE UPDATED -# For python 3.8 use "python==3.8.5" -# For python 3.9 use "python==3.9.5" -RUN echo "python==3.9.13" >> /opt/conda/conda-meta/pinned +# For python 3.10 use "python==3.10.10" +# For python 3.11 use "python==3.11.7" +RUN echo "python==3.10.10" >> /opt/conda/conda-meta/pinned RUN apt-get --allow-releaseinfo-change update \ # Upgrade installed packages to get latest security fixes if the base image does not contain them already. @@ -48,7 +48,7 @@ RUN pip install --upgrade --no-cache-dir --ignore-installed \ psutil # Put here your pip dependencies... -ENV APP_HOME /lithops +ENV APP_HOME=/lithops WORKDIR $APP_HOME COPY lithops_aws_batch.zip . diff --git a/runtime/aws_lambda/Dockerfile b/runtime/aws_lambda/Dockerfile index 935f6f0fb..88f35e315 100644 --- a/runtime/aws_lambda/Dockerfile +++ b/runtime/aws_lambda/Dockerfile @@ -1,14 +1,11 @@ -# Python 3.8 -#FROM python:3.8-slim-trixie - -# Python 3.9 -#FROM python:3.9-slim-trixie - # Python 3.10 FROM python:3.10-slim-trixie +# Python 3.11 +#FROM python:3.11-slim-trixie + RUN apt-get update \ # Install aws-lambda-cpp build dependencies && apt-get install -y \ diff --git a/runtime/aws_lambda/conda/conda-python3-9.Dockerfile b/runtime/aws_lambda/conda/Dockerfile similarity index 78% rename from runtime/aws_lambda/conda/conda-python3-9.Dockerfile rename to runtime/aws_lambda/conda/Dockerfile index 95714fcae..cccec9b8f 100644 --- a/runtime/aws_lambda/conda/conda-python3-9.Dockerfile +++ b/runtime/aws_lambda/conda/Dockerfile @@ -1,4 +1,4 @@ -FROM public.ecr.aws/lambda/python:3.9 +FROM public.ecr.aws/lambda/python:3.10 ARG FUNCTION_DIR @@ -11,7 +11,7 @@ RUN yum update -y && \ RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o miniconda.sh && \ sh Miniconda3-latest-Linux-x86_64.sh -b -p /opt/miniconda -COPY lithops-env-py39.yml /tmp/lithops-conda.yml +COPY lithops-env.yml /tmp/lithops-conda.yml RUN /opt/miniconda/bin/conda update -n base -c defaults conda && \ /opt/miniconda/bin/conda env create --file /tmp/lithops-conda.yml --prefix /opt/conda-env @@ -37,10 +37,10 @@ RUN /opt/conda-env/bin/pip install --upgrade --no-cache-dir --ignore-installed \ # RUN /opt/conda-env/bin/pip install --upgrade --no-cache-dir --ignore-installed # Replace python intepreter with conda's -RUN mv /var/lang/bin/python3.9 /var/lang/bin/python3.9-clean && \ - ln -sf /opt/conda-env/bin/python /var/lang/bin/python3.9 +RUN mv /var/lang/bin/python3.10 /var/lang/bin/python3.10-clean && \ + ln -sf /opt/conda-env/bin/python /var/lang/bin/python3.10 -ENV PYTHONPATH "/var/lang/lib/python3.9/site-packages:${FUNCTION_DIR}" +ENV PYTHONPATH="/var/lang/lib/python3.10/site-packages:${FUNCTION_DIR}" ENV PATH="${PATH}:/opt/conda-env/bin/" @@ -52,4 +52,4 @@ RUN unzip lithops_lambda.zip \ && touch handler/__init__.py \ && mv entry_point.py handler/ -CMD [ "handler.entry_point.lambda_handler" ] \ No newline at end of file +CMD [ "handler.entry_point.lambda_handler" ] diff --git a/runtime/aws_lambda/conda/lithops-env-py39.yml b/runtime/aws_lambda/conda/lithops-env.yml similarity index 62% rename from runtime/aws_lambda/conda/lithops-env-py39.yml rename to runtime/aws_lambda/conda/lithops-env.yml index 6fcbd28e4..ef2a5df1f 100644 --- a/runtime/aws_lambda/conda/lithops-env-py39.yml +++ b/runtime/aws_lambda/conda/lithops-env.yml @@ -4,6 +4,6 @@ channels: - anaconda - defaults dependencies: - - python=3.9 + - python=3.10 - pip - # Put here your conda dependencies... \ No newline at end of file + # Put here your conda dependencies... diff --git a/runtime/azure_containers/README.md b/runtime/azure_containers/README.md index d86075c05..2cd7eea25 100644 --- a/runtime/azure_containers/README.md +++ b/runtime/azure_containers/README.md @@ -2,7 +2,7 @@ The runtime is the place where the functions are executed. In Azure Container Apps, runtimes are based on Docker images, so you can run functions with modern Python versions supported by your base image. -For running lithops on Azure Container Apps, you need a runtime built on Docker Hub (or any other container registry), or you need a Docker Hub account for placing the runtimes created by lithops. +For running Lithops on Azure Container Apps, you need a runtime built on Docker Hub (or any other container registry), or you need a Docker Hub account for placing the runtimes created by Lithops. If you don't have an already built runtime, the default runtime is built the first time you execute a function. Lithops automatically detects the Python version of your environment and deploys the default runtime based on it. @@ -39,14 +39,14 @@ azure_containers: If you need some Python modules (or other system libraries) which are not included in the default docker images (see table above), it is possible to build your own Lithops runtime with all of them. - This option is based on building a local Docker image, deploy it to the docker hub (you need a [Docker Hub account](https://hub.docker.com)) and use it as a Lithops base runtime. + This option is based on building a local Docker image, deploy it to Docker Hub (you need a [Docker Hub account](https://hub.docker.com)) and use it as a Lithops base runtime. Project provides some skeletons of Docker images, for example: * [Dockerfile](Dockerfile) To build your own runtime, first install the Docker CE version in your client machine. You can find the instructions [here](https://docs.docker.com/get-docker/). If you already have Docker installed omit this step. - Login to your Docker hub account by running in a terminal the next command. + Log in to your Docker Hub account by running in a terminal the next command. $ docker login @@ -55,7 +55,7 @@ azure_containers: $ lithops runtime build -b azure_containers docker_username/runtimename:tag - Note that Docker hub image names look like *"docker_username/runtimename:tag"* and must be all lower case, for example: + Note that Docker Hub image names look like *"docker_username/runtimename:tag"* and must be all lower case, for example: $ lithops runtime build -b azure_containers myaccount/lithops-ca-custom-v312:01 @@ -64,7 +64,7 @@ azure_containers: $ lithops runtime build -b azure_containers -f azure_containers/Dockerfile myaccount/lithops-ca-custom-v312:01 Once you have built your runtime with all of your necessary packages, you can already use it with Lithops. - To do so, you have to specify the full docker image name in the configuration or when you create the **FunctionExecutor** instance, or directly in the config file, for example: + To do so, you have to specify the full Docker image name in the configuration or when you create the **FunctionExecutor** instance, or directly in the config file, for example: ```python import lithops @@ -83,7 +83,7 @@ azure_containers: fexec = lithops.FunctionExecutor(runtime='lithopscloud/ca-conda-v312:01') ``` - Alternatively, you can create a Lithops runtime based on already built Docker image by executing the following command, which will deploy all the necessary information to use the runtime with your Lithops. + Alternatively, you can create a Lithops runtime based on an already built Docker image by executing the following command, which will deploy all the necessary information to use the runtime with your Lithops. $ lithops runtime deploy -b azure_containers -s azure_storage docker_username/runtimename:tag @@ -108,7 +108,7 @@ azure_containers: $ lithops runtime update default -b azure_containers -s azure_storage - You can update any other runtime deployed in your namespace by specifying the docker image that the runtime depends on: + You can update any other runtime deployed in your namespace by specifying the Docker image that the runtime depends on: $ lithops runtime update docker_username/runtimename:tag -b azure_containers -s azure_storage @@ -128,11 +128,11 @@ azure_containers: $ lithops runtime delete default -b azure_containers -s azure_storage - You can delete any other runtime deployed in your namespace by specifying the docker image that the runtime depends on: + You can delete any other runtime deployed in your namespace by specifying the Docker image that the runtime depends on: $ lithops runtime delete docker_username/runtimename:tag -b azure_containers -s azure_storage - For example, you can delete runtime based on the Docker image `lithopscloud/ca-conda-v312:01` by: + For example, you can delete a runtime based on the Docker image `lithopscloud/ca-conda-v312:01` by: $ lithops runtime delete lithopscloud/ca-conda-v312:01 -b azure_containers -s azure_storage diff --git a/runtime/azure_functions/README.md b/runtime/azure_functions/README.md index bcf749e5d..a7e82e4e2 100644 --- a/runtime/azure_functions/README.md +++ b/runtime/azure_functions/README.md @@ -25,7 +25,7 @@ result = lithops.get_result() If you require additional Python modules not included in the default runtime, you can create your own custom Lithops runtime incorporating them. To create a custom runtime, compile all the necessary modules into a `requirements.txt` file. -For instance, if you wish to integrate the `matplotlib` module into your runtime, which isn't part of the default setup, you need to append it to the existing [requirements.txt](requirements.txt) file. Note that this `requirements.txt` contains the mandatory packages required by lithops, so you don't have to remove any of them from the list, but just add your packages at the end. +For instance, if you wish to integrate the `matplotlib` module into your runtime, which isn't part of the default setup, you need to append it to the existing [requirements.txt](requirements.txt) file. Note that this `requirements.txt` contains the mandatory packages required by Lithops, so you don't have to remove any of them from the list, but just add your packages at the end. **IMPORTANT**: Note that the runtime is built using your local machine, and some libraries, like Numpy, compile some *C* code based on the Operating System you are using. Azure functions run on a Linux machine, this means that if you use **macOS** or **Windows** for building the runtime, libraries that compile *C* code may not run correctly inside the function. In this case, you must use a Linux machine for building the runtime. diff --git a/runtime/code_engine/Dockerfile b/runtime/code_engine/Dockerfile index 2a809f0b0..fd2f9295f 100644 --- a/runtime/code_engine/Dockerfile +++ b/runtime/code_engine/Dockerfile @@ -1,16 +1,10 @@ -# Python 3.8 -#FROM python:3.8-slim-bookworm - -# Python 3.9 -#FROM python:3.9-slim-bookworm - # Python 3.10 FROM python:3.10-slim-bookworm # Python 3.11 -# FROM python:3.11-slim-bookworm +#FROM python:3.11-slim-bookworm RUN apt-get update \ # add the required packages @@ -22,13 +16,13 @@ RUN apt-get update \ COPY requirements.txt requirements.txt RUN pip install --upgrade pip setuptools six gevent && pip install --no-cache-dir -r requirements.txt -ENV PORT 8080 -ENV CONCURRENCY 1 -ENV TIMEOUT 600 -ENV PYTHONUNBUFFERED TRUE +ENV PORT=8080 +ENV CONCURRENCY=1 +ENV TIMEOUT=600 +ENV PYTHONUNBUFFERED=TRUE # Copy Lithops proxy and lib to the container image. -ENV APP_HOME /lithops +ENV APP_HOME=/lithops WORKDIR $APP_HOME COPY lithops_codeengine.zip . diff --git a/runtime/code_engine/Dockerfile.conda b/runtime/code_engine/Dockerfile.conda index 2eb41121f..c6ad168aa 100644 --- a/runtime/code_engine/Dockerfile.conda +++ b/runtime/code_engine/Dockerfile.conda @@ -1,9 +1,3 @@ -# Python 3.8 -#FROM docker.io/continuumio/miniconda3:4.9.2 - -# Python 3.9 -#FROM docker.io/continuumio/miniconda3:4.10.3 - # Python 3.10 FROM docker.io/continuumio/miniconda3:23.3.1-0 @@ -11,13 +5,11 @@ FROM docker.io/continuumio/miniconda3:23.3.1-0 #FROM docker.io/continuumio/miniconda3:23.10.0-1 # YOU MUST PIN THE PYTHON VERSION TO PREVENT IT TO BE UPDATED -# For python 3.8 use "python==3.8.5" -# For python 3.9 use "python==3.9.5" # For python 3.10 use "python==3.10.10" # For python 3.11 use "python==3.11.7" RUN echo "python==3.10.10" >> /opt/conda/conda-meta/pinned -ENV FLASK_PROXY_PORT 8080 +ENV FLASK_PROXY_PORT=8080 RUN apt-get --allow-releaseinfo-change update \ # Upgrade installed packages to get latest security fixes if the base image does not contain them already. @@ -47,13 +39,13 @@ RUN pip install --upgrade pip setuptools six gevent \ # Replace python intepreter with conda's RUN ln -sf /opt/conda/bin/python /usr/local/bin/python -ENV PORT 8080 -ENV CONCURRENCY 1 -ENV TIMEOUT 600 -ENV PYTHONUNBUFFERED TRUE +ENV PORT=8080 +ENV CONCURRENCY=1 +ENV TIMEOUT=600 +ENV PYTHONUNBUFFERED=TRUE # Copy Lithops proxy and lib to the container image. -ENV APP_HOME /lithops +ENV APP_HOME=/lithops WORKDIR $APP_HOME COPY lithops_codeengine.zip . diff --git a/runtime/code_engine/Dockerfile.mamba b/runtime/code_engine/Dockerfile.mamba index 012dad98a..540130a36 100644 --- a/runtime/code_engine/Dockerfile.mamba +++ b/runtime/code_engine/Dockerfile.mamba @@ -1,16 +1,10 @@ -# Python 3.8 -#FROM python:3.8-slim-bookworm - -# Python 3.9 -#FROM python:3.9-slim-bookworm - # Python 3.10 FROM python:3.10-slim-bookworm # Python 3.11 -# FROM python:3.11-slim-bookworm +#FROM python:3.11-slim-bookworm RUN apt-get update \ # add the required packages @@ -61,13 +55,13 @@ RUN ln -sf /opt/conda-env/bin/python /usr/local/bin/python ENV PATH="/opt/conda-env/bin/:${PATH}" -ENV PORT 8080 -ENV CONCURRENCY 1 -ENV TIMEOUT 600 -ENV PYTHONUNBUFFERED TRUE +ENV PORT=8080 +ENV CONCURRENCY=1 +ENV TIMEOUT=600 +ENV PYTHONUNBUFFERED=TRUE # Copy Lithops proxy and lib to the container image. -ENV APP_HOME /lithops +ENV APP_HOME=/lithops WORKDIR $APP_HOME COPY lithops_codeengine.zip . diff --git a/runtime/code_engine/Dockerfile.opencv b/runtime/code_engine/Dockerfile.opencv index fc140ed55..338683311 100644 --- a/runtime/code_engine/Dockerfile.opencv +++ b/runtime/code_engine/Dockerfile.opencv @@ -28,13 +28,13 @@ RUN pip install --upgrade pip setuptools six \ opencv-python-headless \ && pip install --no-cache-dir -r requirements.txt -ENV PORT 8080 -ENV CONCURRENCY 1 -ENV TIMEOUT 600 -ENV PYTHONUNBUFFERED TRUE +ENV PORT=8080 +ENV CONCURRENCY=1 +ENV TIMEOUT=600 +ENV PYTHONUNBUFFERED=TRUE # Copy Lithops proxy and lib to the container image. -ENV APP_HOME /lithops +ENV APP_HOME=/lithops WORKDIR $APP_HOME COPY lithops_codeengine.zip . diff --git a/runtime/code_engine/README.md b/runtime/code_engine/README.md index 2cb62ae88..93c0b5437 100644 --- a/runtime/code_engine/README.md +++ b/runtime/code_engine/README.md @@ -1,8 +1,8 @@ # Lithops runtime for IBM Code Engine -The runtime is the place where the functions are executed. In IBM Code Engine, runtimes are based on docker images, in this sense you can run functions using modern Python versions supported by your base image. +The runtime is the place where the functions are executed. In IBM Code Engine, runtimes are based on Docker images, in this sense you can run functions using modern Python versions supported by your base image. -For running lithops on Code Engine, you need a runtime built on Docker Hub (or any other container registry), or you need a Docker Hub account for placing the runtimes automatically created by lithops. +For running Lithops on Code Engine, you need a runtime built on Docker Hub (or any other container registry), or you need a Docker Hub account for placing the runtimes automatically created by Lithops. If you don't have an already built runtime, the default runtime is built the first time you execute a function. Lithops automatically detects the Python version of your environment and deploys the default runtime based on it. @@ -40,14 +40,14 @@ code_engine: If you need some Python modules (or other system libraries) which are not included in the default docker images (see table above), it is possible to build your own Lithops runtime with all of them. - This option is based on building a local Docker image, deploy it to the docker hub (you need a [Docker Hub account](https://hub.docker.com)) and use it as a Lithops base runtime. + This option is based on building a local Docker image, deploy it to Docker Hub (you need a [Docker Hub account](https://hub.docker.com)) and use it as a Lithops base runtime. Project provides some skeletons of Docker images, for example: * [Dockerfile](Dockerfile) To build your own runtime, first install the Docker CE version in your client machine. You can find the instructions [here](https://docs.docker.com/get-docker/). If you already have Docker installed omit this step. - Login to your Docker hub account by running in a terminal the next command. + Log in to your Docker Hub account by running in a terminal the next command. $ docker login @@ -56,7 +56,7 @@ code_engine: $ lithops runtime build docker_username/runtimename:tag -b code_engine - Note that Docker hub image names look like *"docker_username/runtimename:tag"* and must be all lower case, for example: + Note that Docker Hub image names look like *"docker_username/runtimename:tag"* and must be all lower case, for example: $ lithops runtime build -b code_engine myaccount/lithops-ce-custom-v312:01 @@ -65,7 +65,7 @@ code_engine: $ lithops runtime build -b code_engine -f code_engine/Dockerfile.conda myaccount/lithops-ce-custom-v312:01 Once you have built your runtime with all of your necessary packages, you can already use it with Lithops. - To do so, you have to specify the full docker image name in the configuration or when you create the **FunctionExecutor** instance, or directly in the config file, for example: + To do so, you have to specify the full Docker image name in the configuration or when you create the **FunctionExecutor** instance, or directly in the config file, for example: ```python import lithops @@ -84,7 +84,7 @@ code_engine: fexec = lithops.FunctionExecutor(runtime='lithopscloud/ce-conda-v312:01') ``` - Alternatively, you can create a Lithops runtime based on already built Docker image by executing the following command, which will deploy all the necessary information to use the runtime with your Lithops. + Alternatively, you can create a Lithops runtime based on an already built Docker image by executing the following command, which will deploy all the necessary information to use the runtime with your Lithops. $ lithops runtime deploy -b code_engine -s ibm_cos docker_username/runtimename:tag @@ -109,7 +109,7 @@ code_engine: $ lithops runtime update default -b code_engine -s ibm_cos - You can update any other runtime deployed in your namespace by specifying the docker image that the runtime depends on: + You can update any other runtime deployed in your namespace by specifying the Docker image that the runtime depends on: $ lithops runtime update docker_username/runtimename:tag -b code_engine -s ibm_cos @@ -129,11 +129,11 @@ code_engine: $ lithops runtime delete default -b code_engine -s ibm_cos - You can delete any other runtime deployed in your namespace by specifying the docker image that the runtime depends on: + You can delete any other runtime deployed in your namespace by specifying the Docker image that the runtime depends on: $ lithops runtime delete docker_username/runtimename:tag -b code_engine -s ibm_cos - For example, you can delete runtime based on the Docker image `lithopscloud/ce-conda-v312:01` by: + For example, you can delete a runtime based on the Docker image `lithopscloud/ce-conda-v312:01` by: $ lithops runtime delete lithopscloud/ce-conda-v312:01 -b code_engine -s ibm_cos diff --git a/runtime/gcp_cloudrun/Dockerfile b/runtime/gcp_cloudrun/Dockerfile index 4f71a8030..3dfb667ef 100644 --- a/runtime/gcp_cloudrun/Dockerfile +++ b/runtime/gcp_cloudrun/Dockerfile @@ -1,14 +1,11 @@ -# Python 3.8 -#FROM python:3.8-slim-bookworm - -# Python 3.9 -#FROM python:3.9-slim-bookworm - # Python 3.10 FROM python:3.10-slim-bookworm +# Python 3.11 +#FROM python:3.11-slim-bookworm + RUN apt-get update && apt-get install -y \ zip \ && rm -rf /var/lib/apt/lists/* @@ -36,11 +33,11 @@ RUN pip install --upgrade setuptools six pip \ google-auth \ psutil -ENV CONCURRENCY 1 -ENV TIMEOUT 600 +ENV CONCURRENCY=1 +ENV TIMEOUT=600 # Copy Lithops proxy and lib to the container image. -ENV APP_HOME /lithops +ENV APP_HOME=/lithops WORKDIR $APP_HOME COPY lithops_cloudrun.zip . diff --git a/runtime/gcp_cloudrun/Dockerfile.conda b/runtime/gcp_cloudrun/Dockerfile.conda index 79c554cc2..9489f4111 100644 --- a/runtime/gcp_cloudrun/Dockerfile.conda +++ b/runtime/gcp_cloudrun/Dockerfile.conda @@ -1,9 +1,9 @@ -# Python 3.8 -FROM continuumio/miniconda3:4.9.2 +# Python 3.10 +FROM continuumio/miniconda3:23.3.1-0 # YOU MUST PIN THE PYTHON VERSION TO PREVENT IT TO BE UPDATED -# For python 3.8 use "python==3.8.5" -RUN echo "python==3.8.5" >> /opt/conda/conda-meta/pinned +# For python 3.10 use "python==3.10.10" +RUN echo "python==3.10.10" >> /opt/conda/conda-meta/pinned RUN apt-get update && apt-get install -y \ @@ -33,8 +33,8 @@ RUN pip install --upgrade setuptools six pip \ google-auth \ psutil -ENV CONCURRENCY 1 -ENV TIMEOUT 600 +ENV CONCURRENCY=1 +ENV TIMEOUT=600 # Add your Conda required packages here. Ensure "conda clean --all" at # the end to remove temporary data. One "RUN" line is better than multiple @@ -42,7 +42,7 @@ ENV TIMEOUT 600 RUN conda install -c conda-forge zarr && conda clean --all # Copy Lithops proxy and lib to the container image. -ENV APP_HOME /lithops +ENV APP_HOME=/lithops WORKDIR $APP_HOME COPY lithops_cloudrun.zip . diff --git a/runtime/gcp_cloudrun/README.md b/runtime/gcp_cloudrun/README.md index eced23913..517e25cf8 100644 --- a/runtime/gcp_cloudrun/README.md +++ b/runtime/gcp_cloudrun/README.md @@ -86,10 +86,10 @@ gcp_cloudrun: google-api-python-client \ google-auth - ENV PYTHONUNBUFFERED TRUE + ENV PYTHONUNBUFFERED=TRUE # Copy Lithops proxy and lib to the container image. - ENV APP_HOME /lithops + ENV APP_HOME=/lithops WORKDIR $APP_HOME COPY lithops_cloudrun.zip . diff --git a/runtime/gcp_functions/README.md b/runtime/gcp_functions/README.md index f702c0cab..f1f9b04e4 100644 --- a/runtime/gcp_functions/README.md +++ b/runtime/gcp_functions/README.md @@ -36,7 +36,7 @@ pw = lithops.FunctionExecutor(runtime_memory=512) If you require additional Python modules not included in the default runtime, you can create your own custom Lithops runtime incorporating them. To create a custom runtime, compile all the necessary modules into a `requirements.txt` file. -For instance, if you wish to integrate the `matplotlib` module into your runtime, which isn't part of the default setup, you need to append it to the existing [requirements.txt](requirements.txt) file. Note that this `requirements.txt` contains the mandatory packages required by lithops, so you don't have to remove any of them from the list, but just add your packages at the end. +For instance, if you wish to integrate the `matplotlib` module into your runtime, which isn't part of the default setup, you need to append it to the existing [requirements.txt](requirements.txt) file. Note that this `requirements.txt` contains the mandatory packages required by Lithops, so you don't have to remove any of them from the list, but just add your packages at the end. After updating the file accordingly, you can proceed to build the custom runtime by specifying the modified `requirements.txt` file along with a chosen runtime name: diff --git a/runtime/ibm_vpc/Dockerfile.nvidia.cuda b/runtime/ibm_vpc/Dockerfile.nvidia.cuda index 29c886880..f40646441 100644 --- a/runtime/ibm_vpc/Dockerfile.nvidia.cuda +++ b/runtime/ibm_vpc/Dockerfile.nvidia.cuda @@ -1,6 +1,6 @@ FROM nvidia/cuda:11.5.1-runtime-ubuntu20.04 -ENV FLASK_PROXY_PORT 8080 +ENV FLASK_PROXY_PORT=8080 RUN apt-get update \ # add some packages required for the pip install diff --git a/runtime/ibm_vpc/README.md b/runtime/ibm_vpc/README.md index 53806c84f..673f81c5e 100644 --- a/runtime/ibm_vpc/README.md +++ b/runtime/ibm_vpc/README.md @@ -67,14 +67,14 @@ and make it executable with If you plan to run your function within a **docker runtime** in the VM, it is preferable to include the docker image into the VM image. In this way, you will avoid the initial `docker pull ` command, thus reducing the overall execution time. To do so, add the `-d` flag followed by the docker image name you plan to use, for example: ``` - $ ./build_lithops_vm_image.sh -d lithopscloud/ibmcf-python-v38 lithops-ubuntu-20.04.qcow2 + $ ./build_lithops_vm_image.sh -d lithopscloud/ibmcf-python-v312 lithops-ubuntu-20.04.qcow2 ``` **Important** Lithops will include all the local Docker images together with the Lithops runtime. To avoid this and include only Lithops runtime, it's advised to delete all local Docker images or run the script in a vanilla Ubuntu 20.04 VM. To delete all local images and include only Lithops runtime you need to execute ``` - $ ./build_lithops_vm_image.sh -p prune -d lithopscloud/ibmcf-python-v38 lithops-ubuntu-20.04.qcow2 + $ ./build_lithops_vm_image.sh -p prune -d lithopscloud/ibmcf-python-v312 lithops-ubuntu-20.04.qcow2 ``` In this example the script generates a VM image named `lithops-ubuntu-20.04.qcow2` that contains all dependencies required by Lithops. diff --git a/runtime/knative/Dockerfile b/runtime/knative/Dockerfile index be600ca42..772b246dd 100644 --- a/runtime/knative/Dockerfile +++ b/runtime/knative/Dockerfile @@ -1,14 +1,11 @@ -# Python 3.8 -#FROM python:3.8-slim-bookworm - -# Python 3.9 -#FROM python:3.9-slim-bookworm - # Python 3.10 FROM python:3.10-slim-bookworm +# Python 3.11 +#FROM python:3.11-slim-bookworm + RUN apt-get update && apt-get install -y \ zip \ && rm -rf /var/lib/apt/lists/* @@ -34,12 +31,12 @@ RUN pip install --upgrade setuptools six pip \ tblib \ psutil -ENV CONCURRENCY 4 -ENV TIMEOUT 600 -ENV PYTHONUNBUFFERED TRUE +ENV CONCURRENCY=4 +ENV TIMEOUT=600 +ENV PYTHONUNBUFFERED=TRUE # Copy Lithops proxy and lib to the container image. -ENV APP_HOME /lithops +ENV APP_HOME=/lithops WORKDIR $APP_HOME COPY lithops_knative.zip . diff --git a/runtime/knative/Dockerfile.conda b/runtime/knative/Dockerfile.conda index 92797818e..7b82bf047 100644 --- a/runtime/knative/Dockerfile.conda +++ b/runtime/knative/Dockerfile.conda @@ -1,13 +1,13 @@ -# Python 3.8 -FROM continuumio/miniconda3:4.9.2 +# Python 3.10 +FROM continuumio/miniconda3:23.3.1-0 -# Python 3.9 -#FROM continuumio/miniconda3:4.10.3 +# Python 3.11 +#FROM continuumio/miniconda3:23.10.0-1 # YOU MUST PIN THE PYTHON VERSION TO PREVENT IT TO BE UPDATED -# For python 3.8 use "python==3.8.5" -# For python 3.9 use "python==3.9.5" -RUN echo "python==3.8.5" >> /opt/conda/conda-meta/pinned +# For python 3.10 use "python==3.10.10" +# For python 3.11 use "python==3.11.7" +RUN echo "python==3.10.10" >> /opt/conda/conda-meta/pinned RUN apt-get update && apt-get install -y \ zip \ @@ -39,12 +39,12 @@ RUN pip install --upgrade setuptools six pip \ # ones in terms of image size. For example: #RUN conda install -c conda-forge opencv && conda install sortedcontainers gevent-websocket && conda clean --all -ENV CONCURRENCY 4 -ENV TIMEOUT 600 -ENV PYTHONUNBUFFERED TRUE +ENV CONCURRENCY=4 +ENV TIMEOUT=600 +ENV PYTHONUNBUFFERED=TRUE # Copy Lithops proxy and lib to the container image. -ENV APP_HOME /lithops +ENV APP_HOME=/lithops WORKDIR $APP_HOME COPY lithops_knative.zip . diff --git a/runtime/knative/README.md b/runtime/knative/README.md index 5ef6cb074..d1a33053e 100644 --- a/runtime/knative/README.md +++ b/runtime/knative/README.md @@ -1,8 +1,8 @@ # Lithops runtime for Knative -The runtime is the place where the functions are executed. In Knative, runtimes are based on docker images. +The runtime is the place where the functions are executed. In Knative, runtimes are based on Docker images. -For running lithops on knative, you need a runtime built on Docker Hub (or any other container registry), or you need a Docker Hub account for placing the runtimes created by lithops. +For running Lithops on Knative, you need a runtime built on Docker Hub (or any other container registry), or you need a Docker Hub account for placing the runtimes created by Lithops. If you don't have an already built runtime, the default runtime is built the first time you execute a function. Lithops automatically detects the Python version of your environment and deploys the default runtime based on it. @@ -45,14 +45,14 @@ knative: If you need some Python modules (or other system libraries) which are not included in the default docker images (see table above), it is possible to build your own Lithops runtime with all of them. - This option is based on building a local Docker image, deploy it to the docker hub (you need a [Docker Hub account](https://hub.docker.com)) and use it as a Lithops base runtime. + This option is based on building a local Docker image, deploy it to Docker Hub (you need a [Docker Hub account](https://hub.docker.com)) and use it as a Lithops base runtime. Project provides some skeletons of Docker images, for example: * [Dockerfile](Dockerfile) To build your own runtime, first install the Docker CE version in your client machine. You can find the instructions [here](https://docs.docker.com/get-docker/). If you already have Docker installed omit this step. - Login to your Docker hub account by running in a terminal the next command. + Log in to your Docker Hub account by running in a terminal the next command. $ docker login @@ -61,7 +61,7 @@ knative: $ lithops runtime build -b knative docker.io/username/runtimename:tag - Note that Docker hub image names look like *"docker_username/runtimename:tag"* and must be all lower case, and must not include '.' or '_', for example: + Note that Docker Hub image names look like *"docker_username/runtimename:tag"* and must be all lower case, and must not include '.' or '_', for example: $ lithops runtime build -b knative docker.io/username/lithops-kn-custom-v312:01 @@ -70,7 +70,7 @@ knative: $ lithops runtime build -b knative -f knative/Dockerfile.conda docker.io/username/lithops-kn-custom-v312:01 Once you have built your runtime with all of your necessary packages, you can already use it with Lithops. - To do so, you have to specify the full docker image name in the configuration or when you create the **FunctionExecutor** instance, or directly in the config file, for example: + To do so, you have to specify the full Docker image name in the configuration or when you create the **FunctionExecutor** instance, or directly in the config file, for example: ```python import lithops @@ -89,7 +89,7 @@ knative: fexec = lithops.FunctionExecutor(runtime='docker.io/username/lithops-kn-conda-v312:01') ``` - Alternatively, you can create a Lithops runtime based on already built Docker image by executing the following command, which will deploy all the necessary information to use the runtime with your Lithops. + Alternatively, you can create a Lithops runtime based on an already built Docker image by executing the following command, which will deploy all the necessary information to use the runtime with your Lithops. $ lithops runtime deploy -b knative docker.io/username/runtimename:tag @@ -114,7 +114,7 @@ knative: $ lithops runtime update default -b knative - You can update any other runtime deployed in your namespace by specifying the docker image that the runtime depends on: + You can update any other runtime deployed in your namespace by specifying the Docker image that the runtime depends on: $ lithops runtime update docker.io/username/runtimename:tag -b knative @@ -134,11 +134,11 @@ knative: $ lithops runtime delete default -b knative - You can delete any other runtime deployed in your namespace by specifying the docker image that the runtime depends on: + You can delete any other runtime deployed in your namespace by specifying the Docker image that the runtime depends on: $ lithops runtime delete docker.io/username/runtimename:tag -b knative - For example, you can delete runtime based on the Docker image `docker.io/username/lithops-kn-conda-v312:01` by: + For example, you can delete a runtime based on the Docker image `docker.io/username/lithops-kn-conda-v312:01` by: $ lithops runtime delete docker.io/username/lithops-kn-conda-v312:01 -b knative diff --git a/runtime/kubernetes/Dockerfile b/runtime/kubernetes/Dockerfile index f743fd186..9ec79378c 100644 --- a/runtime/kubernetes/Dockerfile +++ b/runtime/kubernetes/Dockerfile @@ -1,9 +1,3 @@ -# Python 3.8 -#FROM python:3.8-slim-bookworm - -# Python 3.9 -#FROM python:3.9-slim-bookworm - # Python 3.10 FROM python:3.10-slim-bookworm diff --git a/runtime/kubernetes/Dockerfile.conda b/runtime/kubernetes/Dockerfile.conda index 98e0d0ca0..f787336f3 100644 --- a/runtime/kubernetes/Dockerfile.conda +++ b/runtime/kubernetes/Dockerfile.conda @@ -1,13 +1,13 @@ -# Python 3.8 -FROM continuumio/miniconda3:4.9.2 +# Python 3.10 +FROM continuumio/miniconda3:23.3.1-0 -# Python 3.9 -#FROM continuumio/miniconda3:4.10.3 +# Python 3.11 +#FROM continuumio/miniconda3:23.10.0-1 # YOU MUST PIN THE PYTHON VERSION TO PREVENT IT TO BE UPDATED -# For python 3.8 use "python==3.8.5" -# For python 3.9 use "python==3.9.5" -RUN echo "python==3.9.5" >> /opt/conda/conda-meta/pinned +# For python 3.10 use "python==3.10.10" +# For python 3.11 use "python==3.11.7" +RUN echo "python==3.10.10" >> /opt/conda/conda-meta/pinned RUN apt-get update && apt-get install -y \ zip redis-server \ @@ -37,10 +37,10 @@ RUN pip install --upgrade setuptools six pip \ # ones in terms of image size. For example: #RUN conda install -c conda-forge opencv && conda install sortedcontainers gevent-websocket && conda clean --all -ENV PYTHONUNBUFFERED TRUE +ENV PYTHONUNBUFFERED=TRUE # Copy Lithops proxy and lib to the container image. -ENV APP_HOME /lithops +ENV APP_HOME=/lithops WORKDIR $APP_HOME COPY lithops_k8s.zip . diff --git a/runtime/kubernetes/README.md b/runtime/kubernetes/README.md index 29c17dfd5..70e7ea0e1 100644 --- a/runtime/kubernetes/README.md +++ b/runtime/kubernetes/README.md @@ -1,13 +1,13 @@ # Lithops runtime for Kubernetes -The runtime is the place where the functions are executed. In Kubernetes, runtimes are based on docker images. +The runtime is the place where the functions are executed. In Kubernetes, runtimes are based on Docker images. -For running lithops on kubernetes, you need a runtime built on Docker Hub (or any other container registry), or you need a Docker Hub account for placing the runtimes created by lithops. +For running Lithops on Kubernetes, you need a runtime built on Docker Hub (or any other container registry), or you need a Docker Hub account for placing the runtimes created by Lithops. If you don't have an already built runtime, the default runtime is built the first time you execute a function. Lithops automatically detects the Python version of your environment and deploys the default runtime based on it. -By default, Lithops uses 256MB as runtime memory size and 0.5vCPU for the kubernetes runtimes. However, you can change it in the `config` by setting the appropriate vCPU size: +By default, Lithops uses 256MB as runtime memory size and 0.5vCPU for the Kubernetes runtimes. However, you can change it in the `config` by setting the appropriate vCPU size: ```yaml k8s: @@ -21,14 +21,14 @@ k8s: If you need some Python modules (or other system libraries) which are not included in the default docker images (see table above), it is possible to build your own Lithops runtime with all of them. - This option is based on building a local Docker image, deploy it to the docker hub (you need a [Docker Hub account](https://hub.docker.com)) and use it as a Lithops base runtime. + This option is based on building a local Docker image, deploy it to Docker Hub (you need a [Docker Hub account](https://hub.docker.com)) and use it as a Lithops base runtime. Project provides some skeletons of Docker images, for example: * [Dockerfile](Dockerfile) To build your own runtime, first install the Docker CE version in your client machine. You can find the instructions [here](https://docs.docker.com/get-docker/). If you already have Docker installed omit this step. - Login to your Docker hub account by running in a terminal the next command. + Log in to your Docker Hub account by running in a terminal the next command. $ docker login @@ -37,7 +37,7 @@ k8s: $ lithops runtime build -b k8s docker_username/runtimename:tag - Note that Docker hub image names look like *"docker_username/runtimename:tag"* and must be all lower case, for example: + Note that Docker Hub image names look like *"docker_username/runtimename:tag"* and must be all lower case, for example: $ lithops runtime build -b k8s myaccount/lithops-k8s-custom-v312:01 @@ -46,7 +46,7 @@ k8s: $ lithops runtime build -b k8s -f kubernetes/Dockerfile.conda myaccount/lithops-k8s-custom-v312:01 Once you have built your runtime with all of your necessary packages, you can already use it with Lithops. - To do so, you have to specify the full docker image name in the configuration or when you create the **FunctionExecutor** instance, or directly in the config file, for example: + To do so, you have to specify the full Docker image name in the configuration or when you create the **FunctionExecutor** instance, or directly in the config file, for example: ```python import lithops diff --git a/runtime/openwhisk/Dockerfile b/runtime/openwhisk/Dockerfile index 751645f96..606d1e26c 100644 --- a/runtime/openwhisk/Dockerfile +++ b/runtime/openwhisk/Dockerfile @@ -1,9 +1,10 @@ -#FROM python:3.8-slim-bookworm -#FROM python:3.9-slim-bookworm +# Python 3.10 #FROM python:3.10-slim-bookworm + +# Python 3.11 FROM python:3.11-slim-bookworm -ENV FLASK_PROXY_PORT 8080 +ENV FLASK_PROXY_PORT=8080 RUN apt-get update \ # add some packages required for the pip install diff --git a/runtime/openwhisk/Dockerfile.conda b/runtime/openwhisk/Dockerfile.conda index 4e52e706e..3c3ca748f 100644 --- a/runtime/openwhisk/Dockerfile.conda +++ b/runtime/openwhisk/Dockerfile.conda @@ -1,9 +1,3 @@ -# Python 3.8 -#FROM continuumio/miniconda3:4.9.2 - -# Python 3.9 -#FROM continuumio/miniconda3:4.10.3 - # Python 3.10 #FROM continuumio/miniconda3:23.3.1-0 @@ -11,13 +5,11 @@ FROM continuumio/miniconda3:23.10.0-1 # YOU MUST PIN THE PYTHON VERSION TO PREVENT IT TO BE UPDATED -# For python 3.8 use "python==3.8.5" -# For python 3.9 use "python==3.9.5" # For python 3.10 use "python==3.10.10" # For python 3.11 use "python==3.11.7" RUN echo "python==3.11.7" >> /opt/conda/conda-meta/pinned -ENV FLASK_PROXY_PORT 8080 +ENV FLASK_PROXY_PORT=8080 RUN apt-get --allow-releaseinfo-change update \ # Upgrade installed packages to get latest security fixes if the base image does not contain them already. diff --git a/runtime/openwhisk/Dockerfile.gdal b/runtime/openwhisk/Dockerfile.gdal index 145a3c748..dab8da46b 100644 --- a/runtime/openwhisk/Dockerfile.gdal +++ b/runtime/openwhisk/Dockerfile.gdal @@ -165,7 +165,7 @@ RUN if test "${GDAL_VERSION}" = "master"; then \ # Build final image FROM ubuntu:18.04 as runner -ENV FLASK_PROXY_PORT 8080 +ENV FLASK_PROXY_PORT=8080 RUN apt-get update \ && apt-get install -y python3-pip python3-dev \ diff --git a/runtime/openwhisk/Dockerfile.slim b/runtime/openwhisk/Dockerfile.slim index 9771dc8a9..3dc406917 100644 --- a/runtime/openwhisk/Dockerfile.slim +++ b/runtime/openwhisk/Dockerfile.slim @@ -1,12 +1,15 @@ # Minimal Docker image for Lithops -#FROM python:3.8-slim-bookworm -#FROM python:3.9-slim-bookworm +# Python 3.10 #FROM python:3.10-slim-bookworm + +# Python 3.11 FROM python:3.11-slim-bookworm + +# Python 3.12 #FROM python:3.12-slim-bookworm -ENV FLASK_PROXY_PORT 8080 +ENV FLASK_PROXY_PORT=8080 RUN pip install --upgrade setuptools six \ && pip install --no-cache-dir \ diff --git a/runtime/openwhisk/README.md b/runtime/openwhisk/README.md index 058a98c96..19e3aa940 100644 --- a/runtime/openwhisk/README.md +++ b/runtime/openwhisk/README.md @@ -1,6 +1,6 @@ # Lithops runtime for OpenWhisk -The runtime is the place where your functions are executed. In Lithops, runtimes are based on docker images, and it includes default runtimes that allow you to run functions with Python >= 3.10 environments. Lithops main runtime is responsible to execute Python functions within OpenWhisk. The strong requirement here is to match Python versions between the client and the runtime. The runtime may also contain additional packages which your code depends on. +The runtime is the place where your functions are executed. In Lithops, runtimes are based on Docker images, and it includes default runtimes that allow you to run functions with Python >= 3.10 environments. Lithops main runtime is responsible to execute Python functions within OpenWhisk. The strong requirement here is to match Python versions between the client and the runtime. The runtime may also contain additional packages which your code depends on. The default runtime is created the first time you execute a function. Lithops automatically detects the Python version of your environment and deploys the default runtime based on it. @@ -37,11 +37,11 @@ fexec = lithops.FunctionExecutor(runtime_memory=512) If you need some Python modules (or other system libraries) which are not included in the default docker images (see table above), it is possible to build your own Lithops runtime with all of them. - This alternative usage is based on to build a local Docker image, deploy it to the docker hub (you need a [Docker Hub account](https://hub.docker.com)) (or another registry) and use it as a Lithops base runtime. + This alternative usage is based on building a local Docker image, deploy it to Docker Hub (you need a [Docker Hub account](https://hub.docker.com)) (or another registry) and use it as a Lithops base runtime. In this folder you will find some Dockerfile skeletons that you can use to build your runtimes. To build your own runtime, first install the Docker CE version in your client machine. You can find the instructions [here](https://docs.docker.com/get-docker/). If you already have Docker installed omit this step. - Login to your Docker hub account by running in a terminal the next command. + Log in to your Docker Hub account by running in a terminal the next command. $ docker login @@ -51,7 +51,7 @@ fexec = lithops.FunctionExecutor(runtime_memory=512) $ lithops runtime build -b openwhisk docker_username/runtimename:tag - Note that Docker hub image names look like *"docker_username/runtimename:tag"* and must be all lower case, for example: + Note that Docker Hub image names look like *"docker_username/runtimename:tag"* and must be all lower case, for example: $ lithops runtime build -b openwhisk jsampe/lithops-custom-runtime-v312:0.1 @@ -79,7 +79,7 @@ fexec = lithops.FunctionExecutor(runtime_memory=512) fexec = lithops.FunctionExecutor(runtime='jsampe/lithops-conda-v312:0.1') ``` - Alternatively, you can create a Lithops runtime based on already built Docker image by executing the following command, which will deploy all the necessary information to use the runtime with your Lithops. + Alternatively, you can create a Lithops runtime based on an already built Docker image by executing the following command, which will deploy all the necessary information to use the runtime with your Lithops. $ lithops runtime deploy docker_username/runtimename:tag -b openwhisk -s minio @@ -104,7 +104,7 @@ fexec = lithops.FunctionExecutor(runtime_memory=512) $ lithops runtime update default -b openwhisk -s minio - You can update any other runtime deployed in your namespace by specifying the docker image that the runtime depends on: + You can update any other runtime deployed in your namespace by specifying the Docker image that the runtime depends on: $ lithops runtime update docker_username/runtimename:tag -b openwhisk -s minio @@ -124,11 +124,11 @@ fexec = lithops.FunctionExecutor(runtime_memory=512) $ lithops runtime delete default -b openwhisk -s minio - You can delete any other runtime deployed in your namespace by specifying the docker image that the runtime depends on: + You can delete any other runtime deployed in your namespace by specifying the Docker image that the runtime depends on: $ lithops runtime delete docker_username/runtimename:tag -b openwhisk -s minio - For example, you can delete runtime based on the Docker image `jsampe/lithops-conda-v312:0.1` by: + For example, you can delete a runtime based on the Docker image `jsampe/lithops-conda-v312:0.1` by: $ lithops runtime delete jsampe/lithops-conda-v312:0.1 -b openwhisk -s minio diff --git a/runtime/oracle_f/Dockerfile b/runtime/oracle_f/Dockerfile index 708a5c614..11435bc4c 100644 --- a/runtime/oracle_f/Dockerfile +++ b/runtime/oracle_f/Dockerfile @@ -44,6 +44,5 @@ RUN unzip lithops_oracle.zip \ && touch handler/__init__.py \ && mv entry_point.py handler/ - ENV PYTHONPATH="${FUNCTION_DIR}" ENTRYPOINT ["/usr/local/bin/fdk", "handler/entry_point.py", "handler"] diff --git a/runtime/singularity/singularity-template.def b/runtime/singularity/singularity-template.def index e16a22642..8eebee250 100644 --- a/runtime/singularity/singularity-template.def +++ b/runtime/singularity/singularity-template.def @@ -1,12 +1,6 @@ Bootstrap: docker -# Python 3.8 -#FROM python:3.8-slim-bookworm - -# Python 3.9 -#FROM python:3.9-slim-bookworm - # Python 3.10 #FROM: python:3.10-slim-bookworm