Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
## [v3.6.5.dev0]

### Added
- [GCP Compute Engine] Added new GCP Compute Engine standalone backend
- [Core] Added support for variable-length parameters in functions passed to the executor.

### Changed
Expand All @@ -15,6 +16,7 @@
### Fixed
- [K8s] Fixed default runtime builds impacted by Debian Buster end-of-life.
- [GCP Cloud Run] Added Artifact Registry (`pkg.dev`) runtime deployment support
- [K8s] Run default runtime image as non-root user (uid 1000) (#1469)


## [v3.6.4]
Expand Down
1 change: 1 addition & 0 deletions config/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ Storage Backends
- [IBM Virtual Private Cloud](../docs/source/compute_config/ibm_vpc.md)
- [AWS Elastic Compute Cloud (EC2)](../docs/source/compute_config/aws_ec2.md)
- [Azure Virtual Machines](../docs/source/compute_config/azure_vms.md)
- [Google Compute Engine](../docs/source/compute_config/gcp_compute_engine.md)

</td>
<td valign="top">
Expand Down
23 changes: 22 additions & 1 deletion config/config_template.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,7 @@
# Google Cloud Platform – credentials shared by all gcp_* sections
# =============================================================================
#gcp:
#project_name: <GCP_PROJECT_ID>
#region: <REGION_NAME> # e.g. us-east1
#credentials_path: <ABSOLUTE_PATH_TO_SERVICE_ACCOUNT_JSON>

Expand Down Expand Up @@ -172,6 +173,26 @@
#region: <REGION_NAME> # Falls back to gcp.region
#storage_bucket: <BUCKET_NAME> # Auto-created if not provided

# Google Compute Engine – standalone compute
#gcp_compute_engine:
#project_name: <GCP_PROJECT_ID> # Falls back to gcp.project_name
#zone: <ZONE_NAME> # e.g. us-east1-b. Mandatory
#region: <REGION_NAME> # Falls back to gcp.region; derived from zone if omitted
#service_account: <SERVICE_ACCOUNT_EMAIL>
#network_name: <VPC_NETWORK_NAME> # Lithops creates a VPC if not provided
#subnet_name: <SUBNET_NAME>
#source_image: projects/ubuntu-os-cloud/global/images/family/ubuntu-2404-lts-amd64
#instance_name: <INSTANCE_NAME> # Mandatory in consume mode
#master_instance_type: e2-small
#worker_instance_type: e2-standard-2
#ssh_username: ubuntu
#ssh_password: <RANDOM_IF_NOT_PROVIDED>
#ssh_key_filename: ~/.ssh/id_rsa
#request_spot_instances: False
#delete_on_dismantle: True
#max_workers: 100
#worker_processes: AUTO # Default: number of CPUs of the worker VM


# =============================================================================
# Microsoft Azure – credentials shared by all azure_* sections
Expand Down Expand Up @@ -460,7 +481,7 @@


# =============================================================================
# Standalone shared settings (applies to ibm_vpc, aws_ec2, azure_vms, vm)
# Standalone shared settings (applies to ibm_vpc, aws_ec2, azure_vms, gcp_compute_engine, vm)
# =============================================================================
#standalone:
#runtime: python3
Expand Down
2 changes: 1 addition & 1 deletion docs/source/compute_backends.rst
Original file line number Diff line number Diff line change
Expand Up @@ -44,4 +44,4 @@ Compute Backends
compute_config/ibm_vpc.md
compute_config/aws_ec2.md
compute_config/azure_vms.md
compute_config/gcp_compute_engie.md
compute_config/gcp_compute_engine.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,27 @@

The GCP Compute Engine backend of Lithops can provide a serverless user experience on top of GCE where Lithops creates new Virtual Machines (VMs) dynamically at runtime and scales Lithops jobs against them (create and reuse modes). Alternatively Lithops can start and stop an existing VM instance (consume mode).

The backend key is `gcp_compute_engie` (matches the Lithops module name).
The backend key is `gcp_compute_engine` (matches the Lithops module name).

## Choose an operating system image for the VM

Any VM needs an operating system image. By default Lithops uses Ubuntu 24.04 (`ubuntu-2404-lts-amd64`). Lithops installs required dependencies on the VM on first use (this can take a few minutes).
Any VM needs an operating system image. By default Lithops uses Ubuntu 24.04 (`projects/ubuntu-os-cloud/global/images/family/ubuntu-2404-lts-amd64`). Lithops installs required dependencies on the VM on first use (this can take a few minutes).

For faster startups, build a pre-configured custom image (see [runtime/gcp_compute_engine](https://github.com/lithops-cloud/lithops/tree/master/runtime/gcp_compute_engine)):

```bash
lithops image build -b gcp_compute_engine
```

This creates `lithops-ubuntu-2404-lts-amd64-server` in your project; Lithops uses it automatically when present.

To list available images:

```bash
lithops image list -b gcp_compute_engie
lithops image list -b gcp_compute_engine
```

Use the **Image ID** column as `source_image` in your config.
Use the **Image ID** column as `source_image` in your config when using a custom image name.

## Installation

Expand Down Expand Up @@ -54,12 +62,12 @@ gcloud projects add-iam-policy-binding <PROJECT_ID> \

```yaml
lithops:
backend: gcp_compute_engie
backend: gcp_compute_engine

gcp:
credentials_path: <FULL_PATH_TO_SERVICE_ACCOUNT_JSON>

gcp_compute_engie:
gcp_compute_engine:
project_name: <GCP_PROJECT_ID>
zone: <ZONE>
exec_mode: reuse
Expand All @@ -78,29 +86,29 @@ Lithops attaches the service account from `credentials_path` to master and worke

|Group|Key|Default|Mandatory|Additional info|
|---|---|---|---|---|
|gcp_compute_engie | project_name | |yes | GCP project ID |
|gcp_compute_engie | zone | |yes | Compute Engine zone, for example `us-east1-b` |
|gcp_compute_engie | region | derived from zone |no | Region used for subnet and NAT |
|gcp_compute_engie | service_account | |no | Service account email attached to VMs. Default: `client_email` from `credentials_path` |
|gcp_compute_engie | network_name | |no | Existing VPC name. If not provided, Lithops creates a new network |
|gcp_compute_engie | subnet_name | |no | Existing subnet name when using a custom VPC |
|gcp_compute_engie | source_image | ubuntu-2404-lts-amd64 |no | Boot image reference |
|gcp_compute_engie | master_instance_type | e2-small |no | Master VM machine type |
|gcp_compute_engie | worker_instance_type | e2-standard-2 |no | Worker VM machine type |
|gcp_compute_engie | ssh_username | ubuntu |no | Username to access the VM |
|gcp_compute_engie | ssh_password | |no | Password for worker VMs. If not provided, it is created randomly |
|gcp_compute_engie | ssh_key_filename | ~/.ssh/id_rsa |no | SSH private key for the master VM. If not provided, Lithops creates one |
|gcp_compute_engie | request_spot_instances | False |no | Use Spot VMs for workers |
|gcp_compute_engie | delete_on_dismantle | True |no | Delete worker VMs when stopped. Master VM is never deleted when stopped |
|gcp_compute_engie | max_workers | 100 |no | Max number of workers per `FunctionExecutor()` |
|gcp_compute_engie | worker_processes | AUTO |no | Parallel Lithops processes per worker. Default: CPUs of `worker_instance_type` |
|gcp_compute_engie | runtime | python3 |no | Runtime name. Default: python3 on the VM |
|gcp_compute_engie | auto_dismantle | True |no | If False, VMs are not stopped automatically |
|gcp_compute_engie | soft_dismantle_timeout | 300 |no | Seconds to stop the VM after a job **completed** |
|gcp_compute_engie | hard_dismantle_timeout | 3600 |no | Seconds to stop the VM after a job **started** |
|gcp_compute_engie | exec_mode | reuse |no | One of: **consume**, **create** or **reuse** |
|gcp_compute_engie | extra_apt_packages | [] |no | Extra apt packages on master/worker VMs during setup |
|gcp_compute_engie | extra_python_packages | [] |no | Extra pip packages on master/worker VMs after Lithops |
|gcp_compute_engine | project_name | |yes | GCP project ID |
|gcp_compute_engine | zone | |yes | Compute Engine zone, for example `us-east1-b` |
|gcp_compute_engine | region | derived from zone |no | Region used for subnet and NAT |
|gcp_compute_engine | service_account | |no | Service account email attached to VMs. Default: `client_email` from `credentials_path` |
|gcp_compute_engine | network_name | |no | Existing VPC name. If not provided, Lithops creates a new network |
|gcp_compute_engine | subnet_name | |no | Existing subnet name when using a custom VPC |
|gcp_compute_engine | source_image | ubuntu-2404-lts-amd64 |no | Boot image reference |
|gcp_compute_engine | master_instance_type | e2-small |no | Master VM machine type |
|gcp_compute_engine | worker_instance_type | e2-standard-2 |no | Worker VM machine type |
|gcp_compute_engine | ssh_username | ubuntu |no | Username to access the VM |
|gcp_compute_engine | ssh_password | |no | Password for worker VMs. If not provided, it is created randomly |
|gcp_compute_engine | ssh_key_filename | ~/.ssh/id_rsa |no | SSH private key for the master VM. If not provided, Lithops creates one |
|gcp_compute_engine | request_spot_instances | False |no | Use Spot VMs for workers |
|gcp_compute_engine | delete_on_dismantle | True |no | Delete worker VMs when stopped. Master VM is never deleted when stopped |
|gcp_compute_engine | max_workers | 100 |no | Max number of workers per `FunctionExecutor()` |
|gcp_compute_engine | worker_processes | AUTO |no | Parallel Lithops processes per worker. Default: CPUs of `worker_instance_type` |
|gcp_compute_engine | runtime | python3 |no | Runtime name. Default: python3 on the VM |
|gcp_compute_engine | auto_dismantle | True |no | If False, VMs are not stopped automatically |
|gcp_compute_engine | soft_dismantle_timeout | 300 |no | Seconds to stop the VM after a job **completed** |
|gcp_compute_engine | hard_dismantle_timeout | 3600 |no | Seconds to stop the VM after a job **started** |
|gcp_compute_engine | exec_mode | reuse |no | One of: **consume**, **create** or **reuse** |
|gcp_compute_engine | extra_apt_packages | [] |no | Extra apt packages on master/worker VMs during setup |
|gcp_compute_engine | extra_python_packages | [] |no | Extra pip packages on master/worker VMs after Lithops |

## Consume mode

Expand All @@ -110,12 +118,12 @@ In this mode, Lithops uses an existing VM. The VM must be reachable by SSH and h

```yaml
lithops:
backend: gcp_compute_engie
backend: gcp_compute_engine

gcp:
credentials_path: <FULL_PATH_TO_SERVICE_ACCOUNT_JSON>

gcp_compute_engie:
gcp_compute_engine:
exec_mode: consume
project_name: <GCP_PROJECT_ID>
zone: <ZONE>
Expand All @@ -126,19 +134,19 @@ gcp_compute_engie:

|Group|Key|Default|Mandatory|Additional info|
|---|---|---|---|---|
|gcp_compute_engie | instance_name | |yes | Existing VM instance name |
|gcp_compute_engie | project_name | |yes | GCP project ID |
|gcp_compute_engie | zone | |yes | Compute Engine zone |
|gcp_compute_engie | ssh_username | ubuntu |no | Username to access the VM |
|gcp_compute_engie | ssh_key_filename | ~/.ssh/id_rsa |no | Path to the SSH private key |
|gcp_compute_engie | worker_processes | AUTO |no | Parallel Lithops processes per worker |
|gcp_compute_engine | instance_name | |yes | Existing VM instance name |
|gcp_compute_engine | project_name | |yes | GCP project ID |
|gcp_compute_engine | zone | |yes | Compute Engine zone |
|gcp_compute_engine | ssh_username | ubuntu |no | Username to access the VM |
|gcp_compute_engine | ssh_key_filename | ~/.ssh/id_rsa |no | Path to the SSH private key |
|gcp_compute_engine | worker_processes | AUTO |no | Parallel Lithops processes per worker |

## Test Lithops

Once you have your compute and storage backends configured, you can run a hello world function with:

```bash
lithops hello -b gcp_compute_engie -s gcp_storage
lithops hello -b gcp_compute_engine -s gcp_storage
```

## Viewing the execution logs
Expand All @@ -158,33 +166,33 @@ All VMs, including the master, are automatically stopped after a configurable ti
You can open an SSH session to the master VM with:

```bash
lithops attach -b gcp_compute_engie
lithops attach -b gcp_compute_engine
```

The master and worker VMs store Lithops service logs in `/tmp/lithops-root/*-service.log`.

To list available workers:

```bash
lithops worker list -b gcp_compute_engie
lithops worker list -b gcp_compute_engine
```

To list submitted jobs:

```bash
lithops job list -b gcp_compute_engie
lithops job list -b gcp_compute_engine
```

To delete workers only:

```bash
lithops clean -b gcp_compute_engie -s gcp_storage
lithops clean -b gcp_compute_engine -s gcp_storage
```

To delete workers, the master VM, and Lithops-created network resources:

```bash
lithops clean -b gcp_compute_engie -s gcp_storage --all
lithops clean -b gcp_compute_engine -s gcp_storage --all
```

## Architecture diagram
Expand Down
2 changes: 1 addition & 1 deletion docs/source/configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Choose your compute and storage engines from the table below:
|| `IBM Virtual Private Cloud <compute_config/ibm_vpc.html>`_ || |
|| `AWS Elastic Compute Cloud (EC2) <compute_config/aws_ec2.html>`_ || |
|| `Azure Virtual Machines <compute_config/azure_vms.html>`_ || |
|| `Google Compute Engine <compute_config/gcp_compute_engie.html>`_ || |
|| `Google Compute Engine <compute_config/gcp_compute_engine.html>`_ || |
+--------------------------------------------------------------------+--------------------------------------------------------------------+

Configuration File
Expand Down
2 changes: 1 addition & 1 deletion docs/source/execution_modes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -83,4 +83,4 @@ underlying infrastructure.

fexec = lithops.StandaloneExecutor()

- Available backends: `IBM Virtual Private Cloud <compute_config/ibm_vpc.html>`_, `AWS Elastic Compute Cloud (EC2) <compute_config/aws_ec2.html>`_, `Azure Virtual Machines <compute_config/azure_vms.html>`_, `Google Compute Engine <compute_config/gcp_compute_engie.html>`_, `Virtual Machine <compute_config/vm.html>`_
- Available backends: `IBM Virtual Private Cloud <compute_config/ibm_vpc.html>`_, `AWS Elastic Compute Cloud (EC2) <compute_config/aws_ec2.html>`_, `Azure Virtual Machines <compute_config/azure_vms.html>`_, `Google Compute Engine <compute_config/gcp_compute_engine.html>`_, `Virtual Machine <compute_config/vm.html>`_
2 changes: 1 addition & 1 deletion lithops/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,6 @@
'ibm_vpc',
'aws_ec2',
'azure_vms',
'gcp_compute_engie',
'gcp_compute_engine',
'vm'
]
3 changes: 0 additions & 3 deletions lithops/standalone/backends/gcp_compute_engie/__init__.py

This file was deleted.

3 changes: 3 additions & 0 deletions lithops/standalone/backends/gcp_compute_engine/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
from .gcp_compute_engine import GCPComputeEngineBackend as StandaloneBackend

__all__ = ['StandaloneBackend']
Original file line number Diff line number Diff line change
Expand Up @@ -43,48 +43,48 @@


def load_config(config_data):
if 'gcp_compute_engie' not in config_data or not config_data['gcp_compute_engie']:
raise Exception("'gcp_compute_engie' section is mandatory in the configuration")
if not config_data['gcp_compute_engine']:
raise Exception("'gcp_compute_engine' section is mandatory in the configuration")

if 'gcp' not in config_data:
config_data['gcp'] = {}

temp = copy.deepcopy(config_data['gcp_compute_engie'])
config_data['gcp_compute_engie'].update(config_data['gcp'])
config_data['gcp_compute_engie'].update(temp)
temp = copy.deepcopy(config_data['gcp_compute_engine'])
config_data['gcp_compute_engine'].update(config_data['gcp'])
config_data['gcp_compute_engine'].update(temp)

if 'credentials_path' in config_data['gcp_compute_engie']:
config_data['gcp_compute_engie']['credentials_path'] = os.path.expanduser(
config_data['gcp_compute_engie']['credentials_path']
if 'credentials_path' in config_data['gcp_compute_engine']:
config_data['gcp_compute_engine']['credentials_path'] = os.path.expanduser(
config_data['gcp_compute_engine']['credentials_path']
)

for key in DEFAULT_CONFIG_KEYS:
if key not in config_data['gcp_compute_engie']:
config_data['gcp_compute_engie'][key] = DEFAULT_CONFIG_KEYS[key]
if key not in config_data['gcp_compute_engine']:
config_data['gcp_compute_engine'][key] = DEFAULT_CONFIG_KEYS[key]

if 'standalone' not in config_data or config_data['standalone'] is None:
config_data['standalone'] = {}

for key in SA_DEFAULT_CONFIG_KEYS:
if key in config_data['gcp_compute_engie']:
config_data['standalone'][key] = config_data['gcp_compute_engie'].pop(key)
if key in config_data['gcp_compute_engine']:
config_data['standalone'][key] = config_data['gcp_compute_engine'].pop(key)
elif key not in config_data['standalone']:
config_data['standalone'][key] = SA_DEFAULT_CONFIG_KEYS[key]

if config_data['standalone']['exec_mode'] == 'consume':
params_to_check = MANDATORY_PARAMETERS_1
config_data['gcp_compute_engie']['max_workers'] = 1
config_data['gcp_compute_engine']['max_workers'] = 1
else:
params_to_check = MANDATORY_PARAMETERS_2

for param in params_to_check:
if param not in config_data['gcp_compute_engie']:
msg = f"'{param}' is mandatory in the 'gcp_compute_engie' section of the configuration"
if param not in config_data['gcp_compute_engine']:
msg = f"'{param}' is mandatory in the 'gcp_compute_engine' or 'gcp' section of the configuration"
raise Exception(msg)

if 'region' not in config_data['gcp_compute_engie']:
zone = config_data['gcp_compute_engie']['zone']
config_data['gcp_compute_engie']['region'] = '-'.join(zone.split('-')[:-1])
if 'region' not in config_data['gcp_compute_engine']:
zone = config_data['gcp_compute_engine']['zone']
config_data['gcp_compute_engine']['region'] = '-'.join(zone.split('-')[:-1])

if 'region' not in config_data['gcp'] and 'region' in config_data['gcp_compute_engie']:
config_data['gcp']['region'] = config_data['gcp_compute_engie']['region']
if 'region' not in config_data['gcp'] and 'region' in config_data['gcp_compute_engine']:
config_data['gcp']['region'] = config_data['gcp_compute_engine']['region']
Loading
Loading