What happens when a master fails? What happens when a worker fails?

What Happens When a Kubernetes Master Node Fails?

When a Kubernetes master node fails, the impact on the cluster depends on the number of master nodes and the specific components that fail. Here are the key points:

1. Single Master Node Failure:

   • Cluster Management: The cluster loses its ability to manage pods, schedule new pods, and respond to node failures. The API server, controller manager, and scheduler are all unavailable, which means no new pods can be scheduled, and existing pods cannot be managed or moved[2][4].
   • Existing Pods: Existing pods continue to run on worker nodes, and services already running will continue to function as long as they do not require interaction with the API server. However, if any of these pods fail or need to be rescheduled, they will not be replaced until the master node is restored[2][4][8].

2. Multi-Master Node Setup:

   • High Availability: In a multi-master setup, the failure of a single master node does not immediately impact the cluster's ability to manage itself. The remaining master nodes can continue to handle API requests, scheduling, and other control plane functions[1][17].
   • Etcd Quorum: The etcd database, which stores the cluster state, requires a quorum to function. In a typical three-node etcd setup, the cluster can tolerate the failure of one node. If more than one etcd node fails, the cluster will lose its ability to maintain consistency and may become read-only or completely non-functional[10][14][17].

3. Recovery:

   • Rejoining a Master Node: Rejoining a failed master node can be complex. The node must be properly cleaned up and reconfigured before it can rejoin the cluster. This often involves removing the node from the etcd cluster and re-adding it, which can be a manual and error-prone process[1][10][18].
   • Backup and Restore: Regular backups of the etcd database are crucial. In the event of a catastrophic failure, restoring from a backup may be necessary to recover the cluster state[10][17].

What Happens When a Kubernetes Worker Node Fails?

When a worker node fails, the impact on the cluster is generally less severe than a master node failure, but it still affects the availability of applications running on that node. Here are the key points:

1. Pod Rescheduling:

   • Automatic Rescheduling: Kubernetes is designed to handle worker node failures by automatically rescheduling pods that were running on the failed node to other healthy nodes in the cluster. This process is managed by the kube-scheduler[6][11][16].
   • Stateful Applications: For stateful applications using StatefulSets, the rescheduling process can be more complex. Stateful applications often have specific requirements for storage and network identity, which can complicate the rescheduling process[16].

2. Persistent Volumes:

   • Data Persistence: If the pods on the failed node were using Persistent Volumes (PVs) backed by network storage, the data remains intact and can be reattached to the new pods scheduled on other nodes. However, if the PVs were using local storage, data may be lo...