What Should I Do If Pod Scheduling Fails?

Fault Locating

If the pod is in the Pending state and the event contains pod scheduling failure information, locate the cause based on the event information. For details about how to view events, see How Do I Use Events to Fix Abnormal Workloads?

Troubleshooting Process

Determine the cause based on the event information, as listed in Table 1.

Check Item 1: Whether a Node Is Available in the Cluster

Log in to the CCE console and check whether the node status is Available. Alternatively, run the following command to check whether the node status is Ready:

$ kubectl get node
NAME           STATUS   ROLES    AGE   VERSION
192.168.0.37   Ready    <none>   21d   v1.19.10-r1.0.0-source-121-gb9675686c54267
192.168.0.71   Ready    <none>   21d   v1.19.10-r1.0.0-source-121-gb9675686c54267

If the status of all nodes is Not Ready, no node is available in the cluster.

Solution

• Add a node. If an affinity policy is not configured for the workload, the pod will be automatically migrated to the new node to ensure that services are running properly.
• Locate the unavailable node and rectify the fault. For details, see What Should I Do If a Cluster Is Available But Some Nodes Are Unavailable?
• Reset the unavailable node. For details, see Resetting a Node.

Check Item 2: Whether Node Resources (CPU and Memory) Are Sufficient

0/2 nodes are available: 2 Insufficient cpu. This means insufficient CPUs.

0/2 nodes are available: 2 Insufficient memory. This means insufficient memory.

If the resources requested by the pod exceed the allocatable resources of the node where the pod runs, the node cannot provide the resources required to run new pods and pod scheduling onto the node will definitely fail.

If the number of resources that can be allocated to a node is less than the number of resources that a pod requests, the node does not meet the resource requirements of the pod. As a result, the scheduling fails.

Solution

Add nodes to the cluster. Scale-out is the common solution to insufficient resources.

Check Item 3: Affinity and Anti-Affinity Configuration of the Workload

Inappropriate affinity policies will cause pod scheduling to fail.

Example:

An anti-affinity relationship is established between workload 1 and workload 2. Workload 1 is deployed on node 1 while workload 2 is deployed on node 2.

When you try to deploy workload 3 on node 1 and establish an affinity relationship with workload 2, a conflict occurs, resulting in a workload deployment failure.

0/2 nodes are available: 1 node(s) didn't match node selector, 1 node(s) didn't match pod affinity rules, 1 node(s) didn't match pod affinity/anti-affinity.

• node selector indicates that the node affinity is not met.
• pod affinity rules indicate that the pod affinity is not met.
• pod affinity/anti-affinity indicates that the pod affinity/anti-affinity is not met.

Solution

• When adding workload-workload affinity and workload-node affinity policies, ensure that the two types of policies do not conflict each other. Otherwise, workload deployment will fail.
• If the workload has a node affinity policy, make sure that supportContainer in the label of the affinity node is set to true. Otherwise, pods cannot be scheduled onto the affinity node and the following event is generated:

  No nodes are available that match all of the following predicates: MatchNode Selector, NodeNotSupportsContainer

  If the value is false, the scheduling fails.

Check Item 4: Whether the Workload's Volume and Node Reside in the Same AZ

0/2 nodes are available: 2 node(s) had volume node affinity conflict. An affinity conflict occurs between volumes and nodes. As a result, the scheduling fails.

This is because EVS disks cannot be attached to nodes across AZs. For example, if the EVS volume is located in AZ 1 and the node is located in AZ 2, scheduling fails.

The EVS volume created on CCE has affinity settings by default, as shown below.

kind: PersistentVolume
apiVersion: v1
metadata:
  name: pvc-c29bfac7-efa3-40e6-b8d6-229d8a5372ac
spec:
  ...
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: failure-domain.beta.kubernetes.io/zone
              operator: In
              values:
                - ap-southeast-1a

Solution

In the AZ where the workload's node resides, create a volume. Alternatively, create an identical workload and select an automatically assigned cloud storage volume.

Check Item 5: Taint Toleration of Pods

0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate. This means the node is tainted and the pod cannot be scheduled to the node.

Check the taints on the node. If the following information is displayed, taints exist on the node:

$ kubectl describe node 192.168.0.37
Name:               192.168.0.37
...
Taints:             key1=value1:NoSchedule
...

In some cases, the system automatically adds a taint to a node. The current built-in taints include:

• node.kubernetes.io/not-ready: The node is not ready.
• node.kubernetes.io/unreachable: The node controller cannot access the node.
• node.kubernetes.io/memory-pressure: The node has memory pressure.
• node.kubernetes.io/disk-pressure: The node has disk pressure. Follow the instructions described in Check Item 4: Whether the Node Disk Space Is Insufficient to handle it.
• node.kubernetes.io/pid-pressure: The node is under PID pressure. Follow the instructions in Changing Process ID Limits (kernel.pid_max) to handle it.
• node.kubernetes.io/network-unavailable: The node network is unavailable.
• node.kubernetes.io/unschedulable: The node cannot be scheduled.
• node.cloudprovider.kubernetes.io/uninitialized: If an external cloud platform driver is specified when kubelet is started, kubelet adds a taint to the current node and marks it as unavailable. After cloud-controller-manager initializes the node, kubelet deletes the taint.

Solution

To schedule the pod to the node, use either of the following methods:

• If the taint is added by a user, you can delete the taint on the node. If the taint is automatically added by the system, the taint will be automatically deleted after the fault is rectified.
• Specify a toleration for the pod containing the taint. For details, see Taints and Tolerations.

  apiVersion: v1
  kind: Pod
  metadata:
    name: nginx
  spec:
    containers:
    - name: nginx
      image: nginx:alpine
    tolerations:
    - key: "key1"
      operator: "Equal"
      value: "value1"
      effect: "NoSchedule" 

Check Item 6: Ephemeral Volume Usage

0/7 nodes are available: 7 Insufficient ephemeral-storage. This means insufficient ephemeral storage of the node.

Check whether the size of the ephemeral volume in the pod is limited. If the size of the ephemeral volume required by the application exceeds the existing capacity of the node, the application cannot be scheduled. To solve this problem, change the size of the ephemeral volume or expand the disk capacity of the node.

apiVersion: v1
kind: Pod
metadata:
  name: frontend
spec:
  containers:
  - name: app
    image: images.my-company.example/app:v4
    resources:
      requests:
        ephemeral-storage: "2Gi"
      limits:
        ephemeral-storage: "4Gi"
    volumeMounts:
    - name: ephemeral
      mountPath: "/tmp"
  volumes:
    - name: ephemeral
      emptyDir: {}

Check Item 7: Whether everest Works Properly

0/1 nodes are available: 1 everest driver not found at node. This means the everest-csi-driver of everest is not started properly on the node.

Check the daemon named everest-csi-driver in the kube-system namespace and check whether the pod is started properly. If not, delete the pod. The daemon will restart the pod.

Check Item 8: Thin Pool Space

A data disk dedicated for kubelet and the container engine will be attached to a new node. For details, see Data Disk Space Allocation. If the data disk space is insufficient, the pod cannot be created.

Solution 1: Clearing images

Perform the following operations to clear unused images:

• Nodes that use containerd
  1. Obtain local images on the node.

     crictl images -v

  2. Delete the images that are not required by image ID.

     crictl rmi Image ID

• Nodes that use Docker
  1. Obtain local images on the node.

     docker images

  2. Delete the images that are not required by image ID.

     docker rmi Image ID

Do not delete system images such as the cce-pause image. Otherwise, pods may fail to be created.

Solution 2: Expanding the disk capacity

To expand a disk capacity, perform the following steps:

1. Expand the capacity of the data disk on the EVS console.
2. Log in to the CCE console and click the cluster. In the navigation pane, choose Nodes. Click More > Sync Server Data in the row containing the target node.
3. Log in to the target node.
4. Run the lsblk command to check the block device information of the node.

   A data disk is divided depending on the container storage Rootfs:

   • Overlayfs: No independent thin pool is allocated. Image data is stored in the dockersys disk.

     # lsblk
     NAME                MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
     vda                   8:0    0   50G  0 disk 
     └─vda1                8:1    0   50G  0 part /
     vdb                   8:16   0  200G  0 disk 
     ├─vgpaas-dockersys  253:0    0   90G  0 lvm  /var/lib/docker               # Space used by the container engine
     └─vgpaas-kubernetes 253:1    0   10G  0 lvm  /mnt/paas/kubernetes/kubelet  # Space used by Kubernetes

     Run the following commands on the node to add the new disk capacity to the dockersys disk:

     pvresize /dev/vdb 
     lvextend -l+100%FREE -n vgpaas/dockersys
     resize2fs /dev/vgpaas/dockersys

   • Devicemapper: A thin pool is allocated to store image data.

     # lsblk
     NAME                                MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
     vda                                   8:0    0   50G  0 disk 
     └─vda1                                8:1    0   50G  0 part /
     vdb                                   8:16   0  200G  0 disk 
     ├─vgpaas-dockersys                  253:0    0   18G  0 lvm  /var/lib/docker    
     ├─vgpaas-thinpool_tmeta             253:1    0    3G  0 lvm                   
     │ └─vgpaas-thinpool                 253:3    0   67G  0 lvm                   # Space used by thinpool
     │   ...
     ├─vgpaas-thinpool_tdata             253:2    0   67G  0 lvm  
     │ └─vgpaas-thinpool                 253:3    0   67G  0 lvm  
     │   ...
     └─vgpaas-kubernetes                 253:4    0   10G  0 lvm  /mnt/paas/kubernetes/kubelet

     • Run the following commands on the node to add the new disk capacity to the thinpool disk:

       pvresize /dev/vdb 
       lvextend -l+100%FREE -n vgpaas/thinpool

     • Run the following commands on the node to add the new disk capacity to the dockersys disk:

       pvresize /dev/vdb 
       lvextend -l+100%FREE -n vgpaas/dockersys
       resize2fs /dev/vgpaas/dockersys

Check Item 9: Number of Pods Scheduled onto the Node

0/1 nodes are available: 1 Too many pods. indicates excessive number of pods have been scheduled to the node.

When creating a node, configure Max. Pods in Advanced Settings to specify the maximum number of pods that can run properly on the node. The default value varies with the node flavor. You can change the value as needed.

Figure 1 Maximum number of pods

On the Nodes page, obtain the Pods (Allocated/Total) value of the node, and check whether the number of pods scheduled onto the node has reached the upper limit. If so, add nodes or change the maximum number of pods.

To change the maximum number of pods that can run on a node, do as follows:

• For nodes in the default node pool: Change the Max. Pods value when resetting the node.
• For nodes in a customized node pool: Change the value of the node pool parameter max-pods. For details, see Configuring a Node Pool.

Figure 2 Checking the number of pods