edgecore join failed to get CA certificate 的问题

Question

老师您好，keadm join局域网内执行出现CA问题
控制台输出：

Host has mosquit+ already installed and running. Hence skipping the installation steps !!!
Expected or Default KubeEdge version 1.8.2 is already downloaded and will checksum for it.
kubeedge-v1.8.2-linux-amd64.tar.gz checksum:
checksum_kubeedge-v1.8.2-linux-amd64.tar.gz.txt content:
Expected or Default checksum file checksum_kubeedge-v1.8.2-linux-amd64.tar.gz.txt is already downloaded.
Expected or Default KubeEdge version 1.8.2 is already downloaded
[Run as service] service file already exisits in /etc/kubeedge//edgecore.service, skip download
kubeedge-v1.8.2-linux-amd64/
kubeedge-v1.8.2-linux-amd64/edge/
kubeedge-v1.8.2-linux-amd64/edge/edgecore
kubeedge-v1.8.2-linux-amd64/cloud/
kubeedge-v1.8.2-linux-amd64/cloud/csidriver/
kubeedge-v1.8.2-linux-amd64/cloud/csidriver/csidriver
kubeedge-v1.8.2-linux-amd64/cloud/admission/
kubeedge-v1.8.2-linux-amd64/cloud/admission/admission
kubeedge-v1.8.2-linux-amd64/cloud/cloudcore/
kubeedge-v1.8.2-linux-amd64/cloud/cloudcore/cloudcore
kubeedge-v1.8.2-linux-amd64/version

KubeEdge edgecore is running, For logs visit: journalctl -u edgecore.service -b

但是查看服务状态一直处于activating

root@node1:~# systemctl status edgecore.service
● edgecore.service
     Loaded: loaded (/etc/systemd/system/edgecore.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Tue 2022-05-03 20:15:22 PDT; 2s ago
    Process: 58162 ExecStart=/usr/local/bin/edgecore (code=exited, status=255/EXCEPTION)
   Main PID: 58162 (code=exited, status=255/EXCEPTION)

查看日志发现如下错误信息：

certmanager.go:93] Error: failed to get CA certificate, err: Get "https://192.198.1.17:10002/ca.crt": dial tcp 192.198.1.17:10002: connect: connection

环境中，cloud端和edge端防火墙已关闭

root@node1:~# ufw status
Status: inactive

cloud端服务检查：

edge端ping通cloud端没有问题

PING 192.168.1.17 (192.168.1.17) 56(84) bytes of data.
64 bytes from 192.168.1.17: icmp_seq=1 ttl=64 time=0.303 ms
64 bytes from 192.168.1.17: icmp_seq=2 ttl=64 time=0.167 ms
64 bytes from 192.168.1.17: icmp_seq=3 ttl=64 time=0.164 ms

希望老师提供宝贵的意见，谢谢。
journalctl -u edgecore -f 如下（不停滚动）：

前100行日志如下：

 1 I0504 01:13:37.860841  125401 server.go:70] Version: v1.8.2
   2 I0504 01:13:37.863821  125401 sql.go:21] Begin to register twin db model
   3 I0504 01:13:37.863896  125401 module.go:34] Module twin registered successfully
   4 I0504 01:13:37.870320  125401 client.go:78] "Connecting to docker on the dockerEndpoint" endpoint="unix:///var/run/docker.sock"
   5 I0504 01:13:37.871109  125401 client.go:97] "Start docker client with request timeout" timeout="0s"
   6 I0504 01:13:37.874502  125401 docker_service.go:242] "Hairpin mode is set" hairpinMode=hairpin-veth
   7 I0504 01:13:37.874567  125401 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
   8 I0504 01:13:37.875480  125401 hostport_manager.go:72] "The binary conntrack is not installed, this can cause failures in network connection cleanup."
   9 I0504 01:13:37.875493  125401 hostport_manager.go:72] "The binary conntrack is not installed, this can cause failures in network connection cleanup."
  10 I0504 01:13:37.876369  125401 docker_service.go:257] "Docker cri networking managed by the network plugin" networkPluginName="kubernetes.io/no-op"
  11 I0504 01:13:37.879423  125401 docker_service.go:264] "Docker Info" dockerInfo=&{ID:CKQJ:TFAE:TU6Y:IQXM:2UFY:E2WZ:YFLR:YDT5:Z5ZL:UB5V:6QPR:46OF Containers:0 ContainersRunning:0 ContainersPaused:0 ContainersStopped:0 Images:0 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true] [userxattr false]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay]      Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:24 OomKillDisable:true NGoroutines:33 SystemTime:2022-05-04T01:13:37.876726-07:00 LoggingDriver:json-file      CgroupDriver:cgroupfs CgroupVersion:1 NEventsListener:0 KernelVersion:5.13.0-40-generic OperatingSystem:Ubuntu 20.04.3 LTS OSVersion:20.04 OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc000307dc0 NCPU:2 MemTotal:4078235648 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ubuntu Labels:[] ExperimentalBuild:false ServerVersion:20.10.12 ClusterStore:      ClusterAdvertise: Runtimes:map[io.containerd.runc.v2:{Path:runc Args:[] Shim:<nil>} io.containerd.runtime.v1.linux:{Path:runc Args:[] Shim:<nil>} runc:{Path:runc Args:[] Shim:<nil>}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil> Warnings:[]} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID: Expected:} R     uncCommit:{ID: Expected:} InitCommit:{ID: Expected:} SecurityOptions:[name=apparmor name=seccomp,profile=default] ProductLicense: DefaultAddressPools:[] Warnings:[]}
  12 I0504 01:13:37.879435  125401 docker_service.go:277] "Setting cgroupDriver" cgroupDriver="cgroupfs"
  13 I0504 01:13:37.879475  125401 edged.go:690] RemoteRuntimeEndpoint: "unix:///var/run/dockershim.sock", remoteImageEndpoint: "unix:///var/run/dockershim.sock"
  14 I0504 01:13:37.879480  125401 edged.go:693] Starting the GRPC server for the docker CRI shim.
  15 I0504 01:13:37.883312  125401 remote_runtime.go:62] parsed scheme: ""
  16 I0504 01:13:37.883320  125401 remote_runtime.go:62] scheme "" not registered, fallback to default scheme
  17 I0504 01:13:37.883337  125401 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock  <nil> 0 <nil>}] <nil> <nil>}
  18 I0504 01:13:37.883342  125401 clientconn.go:948] ClientConn switching balancer to "pick_first"
  19 I0504 01:13:37.883361  125401 remote_image.go:50] parsed scheme: ""
  20 I0504 01:13:37.883364  125401 remote_image.go:50] scheme "" not registered, fallback to default scheme
  21 I0504 01:13:37.883368  125401 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock  <nil> 0 <nil>}] <nil> <nil>}
  22 I0504 01:13:37.883377  125401 clientconn.go:948] ClientConn switching balancer to "pick_first"
  23 W0504 01:13:37.883426  125401 edged.go:408] failed to get the edged's cgroup: cpu and memory cgroup hierarchy not unified.  cpu: /user.slice, memory: /user.slice/user-0.slice/session-4.scope. Edged system container metrics may be missing.
  24 I0504 01:13:37.909849  125401 kuberuntime_manager.go:222] "Container runtime initialized" containerRuntime="docker" version="20.10.12" apiVersion="1.41.0"
  25 I0504 01:13:37.909873  125401 edged.go:575] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
  26 I0504 01:13:37.914893  125401 container_manager_linux.go:278] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
  27 I0504 01:13:37.914920  125401 container_manager_linux.go:283] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/edged ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatab     le:map[] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:0s ExperimentalMemoryManagerPolicy: ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:0 EnforceCPULimits:false CPUCFSQuotaPeriod:0s ExperimentalTopologyManagerPolicy:none}
  28 I0504 01:13:37.914932  125401 topology_manager.go:120] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
  29 I0504 01:13:37.914938  125401 container_manager_linux.go:314] "Initializing Topology Manager" policy="none" scope="container"
  30 I0504 01:13:37.914942  125401 container_manager_linux.go:319] "Creating device plugin manager" devicePluginEnabled=false
  31 I0504 01:13:37.914980  125401 module.go:34] Module edged registered successfully
  32 I0504 01:13:37.914984  125401 module.go:34] Module websocket registered successfully
  33 I0504 01:13:37.914987  125401 module.go:34] Module eventbus registered successfully
  34 I0504 01:13:37.915009  125401 metamanager.go:45] Begin to register metaManager db model
  35 I0504 01:13:37.915033  125401 module.go:34] Module metaManager registered successfully
  36 W0504 01:13:37.915036  125401 module.go:37] Module servicebus is disabled, do not register
  37 W0504 01:13:37.915039  125401 module.go:37] Module edgestream is disabled, do not register
  38 W0504 01:13:37.915042  125401 module.go:37] Module testManager is disabled, do not register
  39 I0504 01:13:37.915615  125401 core.go:24] Starting module twin
  40 I0504 01:13:37.915633  125401 core.go:24] Starting module edged
  41 I0504 01:13:37.915643  125401 core.go:24] Starting module websocket
  42 I0504 01:13:37.915655  125401 core.go:24] Starting module eventbus
  43 I0504 01:13:37.915674  125401 core.go:24] Starting module metaManager
  44 I0504 01:13:37.915752  125401 http.go:40] tlsConfig InsecureSkipVerify true
  45 I0504 01:13:37.915883  125401 process.go:113] Begin to sync sqlite
  46 I0504 01:13:37.915916  125401 common.go:96] start connect to mqtt server with client id: hub-client-sub-1651652017
  47 I0504 01:13:37.915926  125401 common.go:98] client hub-client-sub-1651652017 isconnected: false
  48 I0504 01:13:37.915982  125401 edged.go:294] Starting edged...
  49 E0504 01:13:37.916048  125401 csi_plugin.go:224] kubernetes.io/csi: CSIDriverLister not found on KubeletVolumeHost
  50 I0504 01:13:37.916066  125401 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
  51 I0504 01:13:37.916132  125401 client.go:86] parsed scheme: "unix"
  52 I0504 01:13:37.916137  125401 client.go:86] scheme "unix" not registered, fallback to default scheme
  53 I0504 01:13:37.916180  125401 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}
  54 I0504 01:13:37.916186  125401 clientconn.go:948] ClientConn switching balancer to "pick_first"
  55 I0504 01:13:37.916433  125401 client.go:150] finish hub-client sub
  56 I0504 01:13:37.916448  125401 common.go:96] start connect to mqtt server with client id: hub-client-pub-1651652017
  57 I0504 01:13:37.916452  125401 common.go:98] client hub-client-pub-1651652017 isconnected: false
  58 I0504 01:13:37.916645  125401 client.go:91] edge-hub-cli subscribe topic to $hw/events/upload/#
  59 I0504 01:13:37.916689  125401 client.go:166] finish hub-client pub
  60 I0504 01:13:37.916694  125401 eventbus.go:63] Init Sub And Pub Client for externel mqtt broker tcp://127.0.0.1:1883 successfully
  61 I0504 01:13:37.916726  125401 client.go:91] edge-hub-cli subscribe topic to $hw/events/device/+/state/update
  62 I0504 01:13:37.916789  125401 client.go:91] edge-hub-cli subscribe topic to $hw/events/device/+/twin/+
  63 I0504 01:13:37.916860  125401 client.go:91] edge-hub-cli subscribe topic to $hw/events/node/+/membership/get
  64 I0504 01:13:37.916933  125401 client.go:91] edge-hub-cli subscribe topic to SYS/dis/upload_records
  65 I0504 01:13:37.917047  125401 client.go:91] edge-hub-cli subscribe topic to +/user/#
  66 I0504 01:13:37.917102  125401 client.go:99] list edge-hub-cli-topics status, no record, skip sync
  67 I0504 01:13:37.978651  125401 cpu_manager.go:199] "Starting CPU manager" policy="none"
  68 I0504 01:13:37.978668  125401 cpu_manager.go:200] "Reconciling" reconcilePeriod="0s"
  69 I0504 01:13:37.978676  125401 state_mem.go:36] "Initialized new in-memory state store"
  70 I0504 01:13:37.978771  125401 state_mem.go:88] "Updated default CPUSet" cpuSet=""
  71 I0504 01:13:37.978778  125401 state_mem.go:96] "Updated CPUSet assignments" assignments=map[]
  72 I0504 01:13:37.978782  125401 policy_none.go:44] "None policy: Start"
  73 I0504 01:13:37.979190  125401 record.go:19] Normal NodeAllocatableEnforced Updated Node Allocatable limit across pods
  74 E0504 01:13:37.979274  125401 container_manager_linux.go:549] "Failed to find cgroups of kubelet" err="cpu and memory cgroup hierarchy not unified.  cpu: /user.slice, memory: /user.slice/user-0.slice/session-4.scope"
  75 I0504 01:13:37.979295  125401 volume_manager.go:271] "Starting Kubelet Volume Manager"
  76 I0504 01:13:37.979406  125401 desired_state_of_world_populator.go:141] "Desired state populator starts to run"
  77 I0504 01:13:37.983295  125401 edged_status.go:385] Attempting to register node ubuntu
  78 E0504 01:13:37.989575  125401 imitator.go:224] failed to unmarshal message content to unstructured obj: Object 'Kind' is missing in '{"metadata":{"name":"ubuntu","creationTimestamp":null,"labels":{"kubernetes.io/arch":"amd64","kubernetes.io/hostname":"ubuntu","kubernetes.io/os":"linux","node-role.kubernetes.io/agent":"","node-role.kubernetes.io/edge":""}},"spec":{},"status":{"daemonEndpoints":{"kubeletEndpoint":{"Port":0}},"nodeInfo":{     "machineID":"","systemUUID":"","bootID":"","kernelVersion":"","osImage":"","containerRuntimeVersion":"","kubeletVersion":"","kubeProxyVersion":"","operatingSystem":"","architecture":""}}}'
  79 I0504 01:13:37.990756  125401 status_manager.go:53] Starting to sync pod status with apiserver
  80 I0504 01:13:37.990771  125401 edged.go:940] start pod addition queue work 0
  81 I0504 01:13:37.990775  125401 edged.go:940] start pod addition queue work 1
  82 I0504 01:13:37.990779  125401 edged.go:940] start pod addition queue work 2
  83 I0504 01:13:37.990781  125401 edged.go:940] start pod addition queue work 3
  84 I0504 01:13:37.990784  125401 edged.go:940] start pod addition queue work 4
  85 I0504 01:13:37.990797  125401 edged.go:360] starting plugin manager
  86 I0504 01:13:37.990855  125401 server.go:36] starting to listen read-only on 127.0.0.1:10350
  87 I0504 01:13:37.991349  125401 server.go:405] "Adding debug handlers to kubelet server"
  88 I0504 01:13:37.991692  125401 plugin_manager.go:114] "Starting Kubelet Plugin Manager"
  89 I0504 01:13:37.993136  125401 cpu_manager.go:199] "Starting CPU manager" policy="none"
  90 I0504 01:13:37.993145  125401 cpu_manager.go:200] "Reconciling" reconcilePeriod="1s"
  91 I0504 01:13:37.993152  125401 state_mem.go:36] "Initialized new in-memory state store"
  92 I0504 01:13:37.993193  125401 state_mem.go:88] "Updated default CPUSet" cpuSet=""
  93 I0504 01:13:37.993199  125401 state_mem.go:96] "Updated CPUSet assignments" assignments=map[]
  94 I0504 01:13:37.993206  125401 policy_none.go:44] "None policy: Start"
  95 I0504 01:13:37.993217  125401 edged.go:372] starting syncPod
  96 F0504 01:13:38.307675  125401 certmanager.go:93] Error: failed to get CA certificate, err: Get "https://192.198.1.17:10002/ca.crt": dial tcp 192.198.1.17:10002: connect: connection refused
  97 goroutine 132 [running]:
  98 k8s.io/klog/v2.stacks(0xc0000c2001, 0xc00004c000, 0xbd, 0x1df)
  99         /root/codes/src/github.com/kubeedge/kubeedge/vendor/k8s.io/klog/v2/klog.go:1021 +0xb9
 100 k8s.io/klog/v2.(*loggingT).output(0x4656de0, 0xc000000003, 0x0, 0x0, 0xc0002fc2a0, 0x42ee3cc, 0xe, 0x5d, 0x0)
 101         /root/codes/src/github.com/kubeedge/kubeedge/vendor/k8s.io/klog/v2/klog.go:970 +0x19b
 102 k8s.io/klog/v2.(*loggingT).printf(0x4656de0, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x2cc7bd2, 0x9, 0xc000c21ca0, 0x1, ...)
 103         /root/codes/src/github.com/kubeedge/kubeedge/vendor/k8s.io/klog/v2/klog.go:751 +0x191
 104 k8s.io/klog/v2.Fatalf(...)
 105         /root/codes/src/github.com/kubeedge/kubeedge/vendor/k8s.io/klog/v2/klog.go:1509
 106 github.com/kubeedge/kubeedge/edge/pkg/edgehub/certificate.(*CertManager).Start(0xc000bd01a0)
 107         /root/codes/src/github.com/kubeedge/kubeedge/edge/pkg/edgehub/certificate/certmanager.go:93 +0x12b
 108 github.com/kubeedge/kubeedge/edge/pkg/edgehub.(*EdgeHub).Start(0xc000bd01a0)
 109         /root/codes/src/github.com/kubeedge/kubeedge/edge/pkg/edgehub/edgehub.go:65 +0x445
 110 created by github.com/kubeedge/beehive/pkg/core.StartModules
 111         /root/codes/src/github.com/kubeedge/kubeedge/vendor/github.com/kubeedge/beehive/pkg/core/core.go:23 +0x1f5
 112

Answer 1

暮闲 2022-05-04 16:29:45

好的 你加qq群没有呀，在群里找到我，我们私聊，方便的话我远程到你的环境看一下具体情况。

• 提问者 及时的斯托同学 #1

  嗯嗯 感谢老师， 加了的 已私信您。

• 提问者 及时的斯托同学 #2

  问题解决 感谢老师

• qq_慕盖茨2046412 回复 提问者 及时的斯托同学 #3

  请教一下, 我遇到了同样的问题, 请问是怎么解决的?

点开查看后面1条评论

Answer 2

暮闲 2022-05-04 11:37:36

这个报错应该是token没有传正确的问题，检查一下token是否全部复制呢?

• 提问者 及时的斯托同学 #1

  感谢老师的回复 我这边传的应该是keadm gettoken拿到的完整的token
  
  ---cloud---
  root@master0:/etc/kubeedge# keadm gettoken
  4231533fc24eaea259d4aa66bbfcd19853fb048c75fe014a2aed8f4a7996ebf3.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NTE3MTcwODl9.fiX0JFczkR2-U3FH-sjT34NU2hhUvauqJD0ZMrtBOVA
  
  --edge--
  root@node1:~# keadm join --cloudcore-ipport=192.198.1.17:10000 --kubeedge-version=1.8.2 --token 4231533fc24eaea259d4aa66bbfcd19853fb048c75fe014a2aed8f4a7996ebf3.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NTE3MTcwODl9.fiX0JFczkR2-U3FH-sjT34NU2hhUvauqJD0ZMrtBOVA
  
  
  我的环境里面/etc/kubeedge下似乎没有cert的目录，ca是K8s提供的嘛？

• 暮闲 #2

  journalctl -u edgecore -f 我看一下的边缘侧最新的日志呢

• 提问者 及时的斯托同学 回复 暮闲 #3

  老师我将截图贴在describe部分了 回复部分限制了字数，也不能传图片
  --分割线--
  另外提供了前100行的log~