Restoring a Vertex AI Training Cluster: Recovering a Stopped GPU Node from Expired Reservations
Cluster Architecture
Google Cloud’s Vertex AI Training Cluster (VTC) is designed for large-scale AI/ML model training. The environment discussed here is a High-Performance Computing (HPC) cluster provisioned using VTC.
For example, the following screenshots show a cluster named vtctest001.
Click the name vtctest001to see the details.
The above cluster comprised of the following components:
Orchestration
Slurm Workload Manager is used to manage cluster resources and schedule jobs.
Compute Resources
- Login Node is an N2-standard CPU instance acting as the cluster controller and entry point.
- Compute Node is an a4-highgpu-8g instance equipped with 8x NVIDIA B200 GPUs.
Storage
- Filestore is mounted at /home for persistent user directories.
- Managed Lustre (Parallelstore) is mounted at /mnt/lustre for high-throughput training data access.
Networking
VPC (Virtual Private Cloud): a custom VPC vtc-test-net ensures low-latency communication between nodes.
Cloud Shell
Activate the Cloud Shell and run commands in the terminal.
For readability, the full prompt
admin_@cloudshell:~ (vertex-tkim)$
is shown as
$
Scenario
A GPU Virtual Machine (VM) vtctest001-a4-0 was originally provisioned using a reservation via DWS Calendar Mode. This reservation has since expired, causing the VM to enter a TERMINATED state.
$ gcloud compute instances list --project=$(gcloud config get-value project)
(output example)
NAME: vtctest001-a4-0
ZONE: europe-west4-b
MACHINE_TYPE: a4-highgpu-8g
PREEMPTIBLE:
INTERNAL_IP: 192.168.0.9
192.168.64.2
192.168.128.2
192.168.136.2
192.168.144.2
192.168.152.2
192.168.160.2
192.168.168.2
192.168.176.2
192.168.184.2
EXTERNAL_IP:
STATUS: TERMINATED
...
$
In Cloud Console, navigate to the Compute Engine > VM instances to obtain the same information:
Consequently, the cluster’s compute capacity is offline, effectively halting all training operations. Attempts to restart the VM fail because the specific reservation ID associated with the instance is no longer valid. Standard attempts to unbind the reservation also fail due to capacity restrictions on the machine type.
Approach to Resolve This
The node will be recovered using the DWS Flex-Start provisioning model.
Phase 1: Environment Configuration
Environment variables are defined to match the existing cluster configuration. This ensures that all subsequent commands target the specific resources within the project.
# Project and Zone Configuration
PROJECT_ID=$(gcloud config get-value project)
ZONE="europe-west4-b"
# Cluster Resource Names
GPU_NODE="vtctest001-a4-0"
LOGIN_NODE="vtctest001-login-001"
NETWORK="vtc-test-net"
Phase 2: Safe Deletion of the VM Wrapper
The existing VM instance must be deleted to remove the binding to the expired reservation. Crucially, the --keep-disks=boot flag is required. This instruction destroys the compute instance (CPU/RAM allocation) but strictly preserves the boot disk (Storage) containing the operating system, Slurm configuration, and local data.
- Retrieve the boot disk name
BOOT_DISK_NAME=$(gcloud compute instances describe $GPU_NODE \
--zone=$ZONE \
--format="value(disks[0].source.basename())")
Verify the variable
$ echo "BOOT_DISK_NAME=$BOOT_DISK_NAME"
vtctest001-a4-0
$
- Create a snapshot of the boot disk
SNAPSHOT_NAME="snap-$GPU_NODE-recovery"
gcloud compute disks snapshot $BOOT_DISK_NAME \
--snapshot-names=$SNAPSHOT_NAME \
--zone=$ZONE
Note: Detaching the disk from the stopped VM won’t work.
Creating snapshot(s) snap-vtctest001-a4-0-recovery...working
If this succeeds, your data is safe.
(output example)
Creating snapshot(s) snap-vtctest001-a4-0-recovery...done.
$
Phase 2: Force Delete the VM
Now that we have a backup, we can delete the VM without worrying about flags like --keep-disks (which was causing your error). We allow the original disk to be deleted with the VM.
gcloud compute instances delete $GPU_NODE \
--zone=$ZONE
(output example)
The following instances will be deleted. Any attached disks configured to be auto-deleted will be deleted unless they are attached to any other instances or the
`--keep-disks` flag is given and specifies them for keeping. Deleting a disk is irreversible and any data on the disk will be lost.
- [vtctest001-a4-0] in [europe-west4-b]
Do you want to continue (Y/n)?
Enter y
Do you want to continue (Y/n)? y
ERROR: (gcloud.compute.instances.delete) Could not fetch resource:
- Required 'Current principal doesn't have permission to mutate this resource!' permission for 'vtctest001-a4-0'$
$
Delete a faulty node
Node deletion is the default behavior if there’s no action specified. Whether the deleted node will be recreated depends on if the node is a dynamic or static node, controlled by Slurm.
If a static node is deleted (node index less than node pool scaling spec min node count), a new compute node with the same name and spec should be recreated.
If a dynamic node is deleted (node index larger than node pool scaling spec min node count), it won’t be recreated if there’s no workload scheduled on it.
To delete a faulty node:
gcurl -X POST https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/modelDevelopmentClusters/CLUSTER_ID:reportFaultyNodes -d '{"nodeActions": [{"nodeId": "NODE_ID"}]}'
NODE_ID is in a format of --
For example,
ENDPOINT="https://${REGION}-aiplatform.googleapis.com/v1beta1/projects/${PROJECT_ID}/locations/${REGION}/modelDevelopmentClusters"
CLUSTER_URI="${ENDPOINT}/${CLUSTER_ID}"
gcurl -X POST ${CLUSTER_URI}:reportFaultyNodes -d '{"nodeActions": [{"nodeId": "vtctest001-a4-0"}]}'
(output example)
{
"name": "projects/600975808889/locations/europe-west4/operations/6918319164184592384",
"metadata": {
"@type": "type.googleapis.com/google.cloud.aiplatform.v1beta1.ReportFaultyNodesOperationMetadata",
"genericMetadata": {
"createTime": "2025-12-07T17:02:51.863166Z",
"updateTime": "2025-12-07T17:02:51.863166Z"
},
"progressMessage": "Faulty node actions received for cluster vtctest001."
}
}
$
$ gcloud compute instances list
(output example)
NAME: vtctest001-login-001
ZONE: europe-west4-b
MACHINE_TYPE: n2-standard-8
PREEMPTIBLE:
INTERNAL_IP: 192.168.0.8
EXTERNAL_IP: 34.34.124.41
STATUS: RUNNING
NAME: workstations-73e0f942-c2d3-42ee-8878-7436b5fc7e47
ZONE: asia-northeast3-b
MACHINE_TYPE: e2-standard-4
PREEMPTIBLE:
INTERNAL_IP: 10.10.0.13
EXTERNAL_IP: 34.50.39.55
STATUS: RUNNING
$
The faulty node is deleted.
Notice the GPU VM disappeared.
Health is also changed to “Unknown health status”.
To get the result of report faulty node action:
gcurl https://REGION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/REGION/operations/OPERATION_ID
Flex-mode
gcloud compute instances create vtc-test-flex-vm \
--project=vertex-tkim \
--zone=europe-west4-b \
--machine-type=a4-highgpu-8g \
--accelerator="type=nvidia-b200,count=8" \
--maintenance-policy=TERMINATE \
--provisioning-model=FLEX_START \
--max-run-duration=1d \
--instance-termination-action=STOP \
--image-family=common-cu124 \
--image-project=deeplearning-platform-release \
--boot-disk-size=200GB \
--boot-disk-type=hyperdisk-balanced
(output example)
Copy Name res-dws-calendar-20251208–1000kst














Top comments (0)