Kubernetes 1.31: Prevent PersistentVolume Leaks When Deleting out of Order

By Deepak Kinni (Broadcom) |Friday, August 16, 2024

PersistentVolume (or PVs for short) areassociated with Reclaim Policy.The reclaim policy is used to determine the actions that need to be taken by the storagebackend on deletion of the PVC Bound to a PV.When the reclaim policy is Delete, the expectation is that the storage backendreleases the storage resource allocated for the PV. In essence, the reclaimpolicy needs to be honored on PV deletion.

With the recent Kubernetes v1.31 release, a beta feature lets you configure yourcluster to behave that way and honor the configured reclaim policy.

How did reclaim work in previous Kubernetes releases?

PersistentVolumeClaim (or PVC for short) isa user’s request for storage. A PV and PVC are considered Boundif a newly created PV or a matching PV is found. The PVs themselves arebacked by volumes allocated by the storage backend.

Normally, if the volume is to be deleted, then the expectation is to delete thePVC for a bound PV-PVC pair. However, there are no restrictions on deleting a PVbefore deleting a PVC.

First, I’ll demonstrate the behavior for clusters running an older version of Kubernetes.

Retrieve a PVC that is bound to a PV

Retrieve an existing PVC example-vanilla-block-pvc

kubectl get pvc example-vanilla-block-pvc

The following output shows the PVC and its bound PV; the PV is shown under the VOLUME column:

NAME                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS               AGEexample-vanilla-block-pvc   Bound    pvc-6791fdd4-5fad-438e-a7fb-16410363e3da   5Gi        RWO            example-vanilla-block-sc   19s

Delete PV

When I try to delete a bound PV, the kubectl session blocks and the kubectltool does not return back control to the shell; for example:

kubectl delete pv pvc-6791fdd4-5fad-438e-a7fb-16410363e3da

persistentvolume "pvc-6791fdd4-5fad-438e-a7fb-16410363e3da" deleted^C

Retrieving the PV

kubectl get pv pvc-6791fdd4-5fad-438e-a7fb-16410363e3da

It can be observed that the PV is in a Terminating state

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS        CLAIM                               STORAGECLASS               REASON   AGEpvc-6791fdd4-5fad-438e-a7fb-16410363e3da   5Gi        RWO            Delete           Terminating   default/example-vanilla-block-pvc   example-vanilla-block-sc            2m23s

Delete PVC

kubectl delete pvc example-vanilla-block-pvc

The following output is seen if the PVC gets successfully deleted:

persistentvolumeclaim "example-vanilla-block-pvc" deleted

The PV object from the cluster also gets deleted. When attempting to retrieve the PVit will be observed that the PV is no longer found:

kubectl get pv pvc-6791fdd4-5fad-438e-a7fb-16410363e3da

Error from server (NotFound): persistentvolumes "pvc-6791fdd4-5fad-438e-a7fb-16410363e3da" not found

Although the PV is deleted, the underlying storage resource is not deleted andneeds to be removed manually.

To sum up, the reclaim policy associated with the PersistentVolume is currentlyignored under certain circumstances. For a Bound PV-PVC pair, the ordering of PV-PVCdeletion determines whether the PV reclaim policy is honored. The reclaim policyis honored if the PVC is deleted first; however, if the PV is deleted prior todeleting the PVC, then the reclaim policy is not exercised. As a result of this behavior,the associated storage asset in the external infrastructure is not removed.

PV reclaim policy with Kubernetes v1.31

The new behavior ensures that the underlying storage object is deleted from the backend when users attempt to delete a PV manually.

How to enable new behavior?

To take advantage of the new behavior, you must have upgraded your cluster to the v1.31 release of Kubernetesand run the CSI external-provisioner version 5.0.1 or later.

How does it work?

For CSI volumes, the new behavior is achieved by adding a finalizer external-provisioner.volume.kubernetes.io/finalizeron new and existing PVs. The finalizer is only removed after the storage from the backend is deleted.`

An example of a PV with the finalizer, notice the new finalizer in the finalizers list

kubectl get pv pvc-a7b7e3ba-f837-45ba-b243-dec7d8aaed53 -o yaml

apiVersion: v1kind: PersistentVolumemetadata:  annotations:    pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com  creationTimestamp: "2021-11-17T19:28:56Z"  finalizers:  - kubernetes.io/pv-protection  - external-provisioner.volume.kubernetes.io/finalizer  name: pvc-a7b7e3ba-f837-45ba-b243-dec7d8aaed53  resourceVersion: "194711"  uid: 087f14f2-4157-4e95-8a70-8294b039d30espec:  accessModes:  - ReadWriteOnce  capacity:    storage: 1Gi  claimRef:    apiVersion: v1    kind: PersistentVolumeClaim    name: example-vanilla-block-pvc    namespace: default    resourceVersion: "194677"    uid: a7b7e3ba-f837-45ba-b243-dec7d8aaed53  csi:    driver: csi.vsphere.vmware.com    fsType: ext4    volumeAttributes:      storage.kubernetes.io/csiProvisionerIdentity: 1637110610497-8081-csi.vsphere.vmware.com      type: vSphere CNS Block Volume    volumeHandle: 2dacf297-803f-4ccc-afc7-3d3c3f02051e  persistentVolumeReclaimPolicy: Delete  storageClassName: example-vanilla-block-sc  volumeMode: Filesystemstatus:  phase: Bound

The finalizer prevents thisPersistentVolume from being removed from thecluster. As stated previously, the finalizer is only removed from the PV objectafter it is successfully deleted from the storage backend. To learn more aboutfinalizers, please refer to Using Finalizers to Control Deletion.

Similarly, the finalizer kubernetes.io/pv-controller is added to dynamically provisioned in-tree plugin volumes.

What about CSI migrated volumes?

The fix applies to CSI migrated volumes as well.

Some caveats

The fix does not apply to statically provisioned in-tree plugin volumes.

References

How do I get involved?

The Kubernetes Slack channel SIG Storage communication channels are great mediums to reach out to the SIG Storage and migration working group teams.

Special thanks to the following people for the insightful reviews, thorough consideration and valuable contribution:

• Fan Baofa (carlory)
• Jan Šafránek (jsafrane)
• Xing Yang (xing-yang)
• Matthew Wong (wongma7)

Join the Kubernetes Storage Special Interest Group (SIG) if you’re interested in getting involved with the design and development of CSI or any part of the Kubernetes Storage system. We’re rapidly growing and always welcome new contributors.