Plight Of The VolumeClaimTemplate: How To Update Your PVC Once It’s Been Created By A Managed Resource

Live photo of me, each time I find a StatefulSet and the associated VolumeClaimTemplate not matching the actual PersistentVolumeClaim

Recently, I’ve been reminded of an old enemy of mine. The VolumeClaimTemplate. Now you might say to yourself, it’s YAML. It can’t hurt you. Well, that may be true. But it certainly was enough to stress eat a whole box of girl scout cookies whiled I pleaded with the universe that I wasn’t going to accidentally unravel the database that was in charge of something fairly important: making all the money.

Gather ‘round. Let me tell you a story.

There once was a tiny lil’ StatefulSet. And it ran a fairly important database. For you see, people did the things, those things were kept track of , and then this particular database allowed us to collect dollar bills for those things. Without this database, we were not able to keep track of the things and thus, no dollar bills for us.

So, whenever maintenance needed to happen, you wanted it to be real, real fast. So that you know, it didn’t impact the tracking of the things, etc., etc.

Enter: The On-call Engineers

Well, lucky for us, things were going well. Many a thing was happening. Dollars bills were flowing all around. This was great! As you can imagine, everyone was walking around high-fiving, the money was flowing. Life was good.

Except for the cranky on-call engineers. You see.. things were going a little .. too well. The things were being collected and stored, but at a very high rate. So we (yes, I am one of those cranky on-call engineers), would get woken up. Oh, you’re running out of storage? AGAIN!? Well, after more nights than we all probably cared to admit, it was time to do something about this.

Luckily for us, a nice little project called pvc-autoresizeralready existed. I love it when a project already exists! We didn’t even really have to do anything, just deploy the resizer, apply some annotations, and success. The PVC’s would get resized per the configuration we supply.

Oh.

We have to.. apply annotations.

To the PVC.

That is managed by a StatefulSet.

Hello everyone. Welcome. Let me introduce you to the villain of our story.

And why is that? Well that is because once an object like a StatefulSet creates a VolumeClaimTemplate, it doesn’t really let you touch it via the StatefulSet again.

The promise of the VolumeClaimTemplate is great. It allows you to describe the storage you want as part of your initial workload without actually having to manage the PVC itself. This is generally fine if you have a very well defined deployment and understand its storage needs almost completely. However, should you ever find yourself in a spot where you need to modify this at all, you’ll quickly run into a problem. You see, none of the fields under the VolumeClaimTemplate are fields that you are able to modify once a StatefulSet has been created. Need to resize your storage? Not happening. Just need to slap some labels or annotations on this thing? Don’t even think about it.

# statefulsets.apps "web" was not valid:
# * spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden
#

Now, this isn’t to say that there isn’t a way to get the deed done. To accomplish the things you need to do, you can directly patch the managed PVC.

kubectl patch pvc very-fake-pvc \ 
  -p '{"metadata":{"annotations": { "hello": "world"}}}'

Anything you need to add, just patch it directly into the PVC and you’re in great shape. But doesn’t that kind of defeat the purpose? If anything, it almost feels like this is worse. THIS ISN’T THE DECLARATIVE FUTURE THAT I WAS PROMISED!

Following this patch method, I now have a StatefulSet that describes my storage one way, but the reality is that my storage now doesn’t reflect the thing that I originally defined. What happens if I need to go apply this workload to another cluster? What happens if some sort of disaster happens to our cluster and we need to redeploy this workload. Under fire, I certainly don’t want to have to remember:

“oh yes, that patch command”.

When you begin to add any GitOps tooling on top of this, it becomes immediately worse. Because at this point, you’ll forever have some object that is out of sync. You could try to sync it. But then you’ll be cursed with some red nastygram telling you what I mentioned above. These fields can’t be updated. GET LOST!

Out of sync objects and red lettering as far as the eye can see

Ride The Keyboard Lightning Out Of This Nightmare

Now, there is still a way out of this mess should you ever find yourself in this situation. And let me be clear. It’s still super duper gross. You would prefer not to find yourself here. Yet, here is where we are.

The first step is what we mentioned above. Ultimately, your goal and the value you’re looking to find here is updating your PVC. So you’ll need to grab your patch command and go at it alone. The StatefulSet is not going to be here to help you. So whatever you need to patch into your PVC, you can do via this command (or if you’re feeling lazy, feel free to hop right on in viakubectl edit .

DISCLAIMER: Busting out kubectl in production environments is ideally not the first tool you’re leaning into. Humans typing on keyboards are error prone in the best of situations, so make sure you know what you’re doing and double check your work.

At this point, you’ll have accomplished exactly half of the job. The still more problematic bit is that you have a StatefulSet that is describing your PVC one way and reality reflecting another. As the error message that we’ve seen before clearly states: can’t touch this.

But, we have ways of making these particular Kubernetes objects sing. It again requires leaning into kubectl ( read disclaimer above ), but it will get the job done. This step will require us to orphan our running pods from our StatefulSet. What this does is allows the workload itself to continue running (re: NO DOWNTIME!), while allowing us to reapply the object that actually manages the workload. And when we are reapplying that object, we can make sure to update our VolumeClaimTemplate to reflect the new reality of the PVC. To do this, you’ll run the following command:

 kubectl delete statefulset <statefulset-name> --cascade=orphan

Now, if you run kubectl get statefulset , you’ll notice that the StatefulSet itself is gone. However, checking to see that your pods are still running with kubectl get pods will show you that there’s nothing to fear, your application will still be humming along just as expected.

Now, let’s be clear about one thing. While we do want to update the StatefulSet YAML to reflect the reality of the situation, nothing is actually happening to the PVC after we update our StatefulSet YAML. That’s because it’s seeing that the storage already exists so it’s not going to touch it. If the PVC was missing of course, it would create it according to the definition that we’ve provided. The process we’re going through however is purely to save future on-call engineers from having to untangle the mystery of why two definitions of a PVC don’t align.

If you’re living in GitOps world, it also makes all the red go away and everything will believe it’s synced up again.

Looking To A Better Future

I share this story because I’ve needed to walk quite a few folks through situations like this over the past few years. There are long running issues that have been open in the Kubernetes repo asking for updates for these exact issues! But, here we are in 2024, and good things may (hopefully) be coming our way soon. There have been a few KEP’s looking to address this in the past, but this one in particular seems to have some legs.

Hopefully, this workaround will let those of us in this situation continue to run those dollar making workloads that need some additional manual intervention. But If nothing else, hopefully this saves you a few extra minutes and some unnecessary heartburn along the way while we all keep our fingers crossed for a fix to deliver us from the pain.