node3 kubelet无法启动

Question

nginx-proxy 的容器已经下载下来，但是根据kubelet无法启动，错误提示无法连接6443

nginx-proxy 容器是需要手动启动吗

kubelet报错如下：
Jul 5 15:15:09 node3 kubelet: I0705 15:15:09.434845 12868 csi_plugin.go:1016] Failed to contact API server when waiting for CSINode publishing: Get “https://127.0.0.1:6443/apis/storage.k8s.io/v1/csinodes/node3”: dial tcp 127.0.0.1:6443: connect: connection refused
Jul 5 15:15:09 node3 kubelet: E0705 15:15:09.435235 12868 cri_stats_provider.go:376] Failed to get the info of the filesystem with mountpoint “/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs”: failed to get device for dir “/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs”: could not find device with major: 0, minor: 37 in cached partitions map.
Jul 5 15:15:09 node3 kubelet: E0705 15:15:09.435277 12868 kubelet.go:1274] Image garbage collection failed once. Stats initialization may not have completed yet: invalid capacity 0 on image filesystem
Jul 5 15:15:09 node3 kubelet: E0705 15:15:09.483914 12868 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get “https://127.0.0.1:6443/api/v1/services?limit=500&resourceVersion=0”: dial tcp 127.0.0.1:6443: connect: connection refused
Jul 5 15:15:09 node3 kubelet: E0705 15:15:09.499762 12868 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: Get “https://127.0.0.1:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0”: dial tcp 127.0.0.1:6443: connect: connection refused
Jul 5 15:15:09 node3 kubelet: E0705 15:15:09.499900 12868 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: Get “https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/node3?timeout=10s”: dial tcp 127.0.0.1:6443: connect: connection refused
Jul 5 15:15:09 node3 kubelet: I0705 15:15:09.558885 12868 factory.go:163] Registration of the crio container factory failed: Get “http://%2Fvar%2Frun%2Fcrio%2Fcrio.sock/info”: dial unix /var/run/crio/crio.sock: connect: no such file or directory
Jul 5 15:15:09 node3 kubelet: I0705 15:15:09.559052 12868 factory.go:163] Registration of the docker container factory failed: failed to validate Docker info: failed to detect Docker info: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Jul 5 15:15:09 node3 kubelet: E0705 15:15:09.623246 12868 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: Get “https://127.0.0.1:6443/apis/node.k8s.io/v1/runtimeclasses?limit=500&resourceVersion=0”: dial tcp 127.0.0.1:6443: connect: connection refused
Jul 5 15:15:09 node3 kubelet: E0705 15:15:09.710416 12868 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: Get “https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/node3?timeout=10s”: dial tcp 127.0.0.1:6443: connect: connection refused
Jul 5 15:15:09 node3 systemd: Unit kubelet.service entered failed state.
Jul 5 15:15:09 node3 systemd: kubelet.service failed.
Jul 5 15:15:09 node3 kubelet: F0705 15:15:09.888078 12868 kubelet.go:1350] Failed to start ContainerManager failed to get rootfs info: failed to get device for dir “/var/lib/kubelet”: could not find device with major: 0, minor: 37 in cached partitions map
Jul 5 15:15:16 node3 etcd: WARNING: 2021/07/05 15:15:16 grpc: Server.processUnaryRPC failed to write status: connection error: desc = “transport is closing”

Answer 1

提问者 苏赋 2021-07-05 15:46:24

容器已下载，但是不启动

[root@node3 ~]# crictl images

IMAGE                                                          TAG                 IMAGE ID            SIZE

docker.io/library/nginx                                        1.19                f0b8a9a541369       53.7MB

registry.cn-hangzhou.aliyuncs.com/kubernetes-kubespray/pause   3.2                 80d28bedfe5de       298kB

k8s.gcr.io/pause                                               3.2                 80d28bedfe5de       298kB

[root@node3 ~]# crictl ps -a

CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID

[root@node3 ~]#

问题和：https://coding.imooc.com/learn/questiondetail/GzLgV6kWyw46kWxb.html 这个很相似，但是不知道怎么解决

• 刘果国 #1

  从kubelet的日志入手，要从头看，从开始启动的第一行开始，关注manifest相关的

• 提问者 苏赋 回复 刘果国 #2

  Jul  6 13:02:25 node2 kubelet: I0706 13:02:25.707410   27422 kuberuntime_manager.go:216] Container runtime containerd initialized, version: v1.4.3, apiVersion: v1alpha2
  Jul  6 13:02:31 node2 kubelet: E0706 13:02:31.912433   27422 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
  Jul  6 13:02:31 node2 kubelet: For verbose messaging see aws.Config.CredentialsChainVerboseErrors
  Jul  6 13:02:31 node2 kubelet: I0706 13:02:31.912800   27422 kuberuntime_manager.go:1006] updating runtime config through cri with podcidr 10.200.0.0/16

• 提问者 苏赋 回复 刘果国 #3

  Jul  6 13:02:31 node2 kubelet: I0706 13:02:31.916962   27422 server.go:148] Starting to listen on 192.168.0.34:10250
  Jul  6 13:02:31 node2 kubelet: E0706 13:02:31.919287   27422 cri_stats_provider.go:376] Failed to get the info of the filesystem with mountpoint "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs": failed to get device for dir "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs": could not find device with major: 0, minor: 37 in cached partitions map.
  Jul  6 13:02:31 node2 kubelet: E0706 13:02:31.919339   27422 kubelet.go:1274] Image garbage collection failed once. Stats initialization may not have completed yet: invalid capacity 0 on image filesystem
  Jul  6 13:02:31 node2 kubelet: I0706 13:02:31.920884   27422 server.go:410] Adding debug handlers to kubelet server.
  Jul  6 13:02:31 node2 kubelet: I0706 13:02:31.927089   27422 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer

点开查看后面4条评论