Cluster Maintenance is required for the below scenarios:
OS Upgrade for worker nodes.
Reboot required for VM's due to hardware change.
Pod Eviction Time:
The controller manager will watch the replicas of the deployment or pod.
If the node is down and not in a running state for 5 minutes then the controller manager will deploy the pods on another node.
POD Eviction Time = 5 mins
Good Practice for Cluster Maintenance:
Drain Node: If we are planning to reboot node1, then we can drain node1. i.e., Workloads will migrate to node2 and no new pods won't be scheduled on node1.
arun@Aruns-MacBook-Air ~ % kg nodes
NAME STATUS ROLES AGE VERSION
master Ready control-plane 97d v1.26.3
worker01 Ready <none> 97d v1.26.3
worker02 Ready <none> 97d v1.26.3
arun@Aruns-MacBook-Air ~ %
arun@Aruns-MacBook-Air ~ % kgp -o wide -A | grep -i worker01
default nginx-748c667d99-498d7 1/1 Running 0 7m4s 10.244.1.85 worker01 <none> <none>
default nginx-748c667d99-9mgb8 1/1 Running 0 7m4s 10.244.1.83 worker01 <none> <none>
default nginx-748c667d99-mhn9g 1/1 Running 0 7m4s 10.244.1.84 worker01 <none> <none>
kube-flannel kube-flannel-ds-gb9bq 1/1 Running 0 89d 172.16.137.138 worker01 <none> <none>
kube-system kube-proxy-227bl 1/1 Running 0 97d 172.16.137.138 worker01 <none> <none>
monitoring prometheus-prometheus-node-exporter-8nrr6 1/1 Running 55 (4d20h ago) 41d 172.16.137.138 worker01 <none> <none>
arun@Aruns-MacBook-Air ~ %
arun@Aruns-MacBook-Air ~ % k drain worker01 --ignore-daemonsets --force --delete-emptydir-data
node/worker01 already cordoned
Warning: ignoring DaemonSet-managed Pods: kube-flannel/kube-flannel-ds-gb9bq, kube-system/kube-proxy-227bl, monitoring/prometheus-prometheus-node-exporter-8nrr6
evicting pod default/nginx-748c667d99-498d7
evicting pod default/nginx-748c667d99-9mgb8
evicting pod default/nginx-748c667d99-mhn9g
pod/nginx-748c667d99-mhn9g evicted
pod/nginx-748c667d99-9mgb8 evicted
pod/nginx-748c667d99-498d7 evicted
node/worker01 drained
arun@Aruns-MacBook-Air ~ %
arun@Aruns-MacBook-Air ~ % kg nodes
NAME STATUS ROLES AGE VERSION
master Ready control-plane 97d v1.26.3
worker01 Ready,SchedulingDisabled <none> 97d v1.26.3
worker02 Ready <none> 97d v1.26.3
arun@Aruns-MacBook-Air ~ %
Here, Pods are evicted from worker01 node and pods are now rescheduled to worker02.
No new pods will schedule on the worker01 node is called the cordon.
Cordon and Uncordon:
Cordon won't schedule new workloads but existing workloads will execute without any problem.
Uncordon is the process of enabling the scheduling so new workloads will start scheduling.
arun@Aruns-MacBook-Air ~ % k cordon worker01
node/worker01 cordoned
arun@Aruns-MacBook-Air ~ % kg nodes
NAME STATUS ROLES AGE VERSION
master Ready control-plane 97d v1.26.3
worker01 Ready,SchedulingDisabled <none> 97d v1.26.3
worker02 Ready <none> 97d v1.26.3
arun@Aruns-MacBook-Air ~ %
arun@Aruns-MacBook-Air ~ % k uncordon worker01
node/worker01 uncordoned
arun@Aruns-MacBook-Air ~ %
arun@Aruns-MacBook-Air ~ % kg nodes
NAME STATUS ROLES AGE VERSION
master Ready control-plane 97d v1.26.3
worker01 Ready <none> 97d v1.26.3
worker02 Ready <none> 97d v1.26.3
arun@Aruns-MacBook-Air ~ %
Cluster Upgrade
As a part of k8s upgrade, We will upgrade only the below components.
API-Server (X version)
Scheduler (X or X-1)
Controller Manager (X or X-1)
Kubelet (X or X-1)
kube-proxy (X or X-1)
kubectl (X or X+1 or X+2)
ETCD and Core-DNS are third-party components that are not a part of k8s upgrade.
Cluster Upgrade Approach:
Upgrade all nodes at a time. (Downtime)
Upgrade one node at a time.
Bring new worker nodes with the latest version and move the workload.
Step 1: SSH into master and execute the kubeadm plan command
master@master:~$ sudo kubeadm upgrade plan
[sudo] password for master:
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.26.3
[upgrade/versions] kubeadm version: v1.26.3
I0707 13:29:11.741403 932860 version.go:256] remote version is much newer: v1.27.3; falling back to: stable-1.26
[upgrade/versions] Target version: v1.26.6
[upgrade/versions] Latest version in the v1.26 series: v1.26.6
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT TARGET
kubelet 3 x v1.26.3 v1.26.6
Upgrade to the latest version in the v1.26 series:
COMPONENT CURRENT TARGET
kube-apiserver v1.26.3 v1.26.6
kube-controller-manager v1.26.3 v1.26.6
kube-scheduler v1.26.3 v1.26.6
kube-proxy v1.26.3 v1.26.6
CoreDNS v1.9.3 v1.9.3
etcd 3.5.6-0 3.5.6-0
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.26.6
Note: Before you can perform this upgrade, you have to update kubeadm to v1.26.6.
_____________________________________________________________________
The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.
API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io v1alpha1 v1alpha1 no
kubelet.config.k8s.io v1beta1 v1beta1 no
_____________________________________________________________________
master@master:~$
Step 2: Upgrade Kubeadm
master@master:~$ sudo apt update
Hit:1 https://baltocdn.com/helm/stable/debian all InRelease
Hit:2 http://in.ports.ubuntu.com/ubuntu-ports focal InRelease
Fetched 2,385 kB in 7s (358 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
36 packages can be upgraded. Run 'apt list --upgradable' to see them.
master@master:~$ sudo apt-cache madison kubeadm
kubeadm | 1.27.3-00 | https://apt.kubernetes.io kubernetes-xenial/main arm64 Packages
kubeadm | 1.27.2-00 | https://apt.kubernetes.io kubernetes-xenial/main arm64 Packages
kubeadm | 1.27.1-00 | https://apt.kubernetes.io kubernetes-xenial/main arm64 Packages
kubeadm | 1.27.0-00 | https://apt.kubernetes.io kubernetes-xenial/main arm64 Packages
kubeadm | 1.26.6-00 | https://apt.kubernetes.io kubernetes-xenial/main arm64 Packages
kubeadm | 1.26.5-00 | https://apt.kubernetes.io kubernetes-xenial/main arm64 Packages
Upgrade kubeadm
root@master:~# apt-mark unhold kubeadm && \apt-get update && apt-get install -y kubeadm=1.26.6-00 && \apt-mark hold kubeadm
Canceled hold on kubeadm.
Hit:2 http://in.ports.ubuntu.com/ubuntu-ports focal InRelease
Hit:3 http://in.ports.ubuntu.com/ubuntu-ports focal-updates InRelease
Hit:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Hit:4 http://in.ports.ubuntu.com/ubuntu-ports focal-backports InRelease
Hit:5 https://baltocdn.com/helm/stable/debian all InRelease
Hit:6 http://in.ports.ubuntu.com/ubuntu-ports focal-security InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
kubeadm
1 upgraded, 0 newly installed, 0 to remove and 35 not upgraded.
Need to get 8,354 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main arm64 kubeadm arm64 1.26.6-00 [8,354 kB]
Fetched 8,354 kB in 3s (2,929 kB/s)
(Reading database ... 112430 files and directories currently installed.)
Preparing to unpack .../kubeadm_1.26.6-00_arm64.deb ...
Unpacking kubeadm (1.26.6-00) over (1.26.3-00) ...
Setting up kubeadm (1.26.6-00) ...
kubeadm set on hold.
root@master:~#
root@master:~# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.6", GitCommit:"11902a838028edef305dfe2f96be929bc4d114d8", GitTreeState:"clean", BuildDate:"2023-06-14T09:55:28Z", GoVersion:"go1.19.10", Compiler:"gc", Platform:"linux/arm64"}
root@master:~#