What happens when a master fails? What happens when a worker fails?
Câu trả lời What happens when a master fails? What happens when a worker fails?
What Happens When a Kubernetes Master Node Fails?
When a Kubernetes master node fails, the impact on the cluster depends on the number of master nodes and the specific components that fail. Here are the key points:
Single Master Node Failure:
Cluster Management: The cluster loses its ability to manage pods, schedule new pods, and respond to node failures. The API server, controller manager, and scheduler are all unavailable, which means no new pods can be scheduled, and existing pods cannot be managed or moved[2][4].
Existing Pods: Existing pods continue to run on worker nodes, and services already running will continue to function as long as they do not require interaction with the API server. However, if any of these pods fail or need to be rescheduled, they will not be replaced until the master node is restored[2][4][8].
Multi-Master Node Setup:
High Availability: In a multi-master setup, the failure of a single master node does not immediately impact the cluster's ability to manage itself. The remaining master nodes can continue to handle API requests, scheduling, and other control plane functions[1][17].
Etcd Quorum: The etcd database, which stores the cluster state, requires a quorum to function. In a typical three-node etcd setup, the cluster can tolerate the failure of one node. If more than one etcd node fails, the cluster will lose its ability to maintain consistency and may become read-only or completely non-functional[10][14][17].
Recovery:
Rejoining a Master Node: Rejoining a failed master node can be complex. The node must be properly cleaned up and reconfigured before it can rejoin the cluster. This often involves removing the node from the etcd cluster and re-adding it, which can be a manual and error-prone process[1][10][18].
Backup and Restore: Regular backups of the etcd database are crucial. In the event of a catastrophic failure, restoring from a backup may be necessary to recover the cluster state[10][17].
What Happens When a Kubernetes Worker Node Fails?
When a worker node fails, the impact on the cluster is generally less severe than a master node failure, but it still affects the availability of applications running on that node. Here are the key points:
Pod Rescheduling:
Automatic Rescheduling: Kubernetes is designed to handle worker node failures by automatically rescheduling pods that were running on the failed node to other healthy nodes in the cluster. This process is managed by the kube-scheduler[6][11][16].
Stateful Applications: For stateful applications using StatefulSets, the rescheduling process can be more complex. Stateful applications often have specific requirements for storage and network identity, which can complicate the rescheduling process[16].
Persistent Volumes:
Data Persistence: If the pods on the failed node were using Persistent Volumes (PVs) backed by network storage, the data remains intact and can be reattached to the new pods scheduled on other nodes. However, if the PVs were using local storage, data may be lost unless it is replicated elsewhere[11][15].
Service Availability:
Service Disruption: There may be a brief period of service disruption while Kubernetes detects the node failure and reschedules the pods. The duration of this disruption depends on the cluster's configuration and the time it takes to reschedule and restart the pods[6][16].
High Availability: To minimize disruption, it is recommended to run multiple replicas of critical pods across different nodes. This ensures that if one node fails, other replicas can continue to serve traffic[15][16].
Node Health Checks:
Monitoring and Alerts: Kubernetes continuously monitors the health of nodes using liveness and readiness probes. If a node becomes unhealthy, Kubernetes will mark it as NotReady and begin the process of rescheduling its pods[12][20].
In summary, the failure of a master node in a single-master setup can severely impact the cluster's ability to manage itself, while a multi-master setup provides redundancy and resilience. Worker node failures primarily affect the availability of the applications running on them, but Kubernetes' self-healing capabilities help mitigate this by rescheduling pods to healthy nodes.
