Wait for etcd cluster to be healthy

Question

fatal: [node1-master]: FAILED! => {
“attempts”: 4,
“changed”: false,
“cmd”: “set -o pipefail && /usr/local/bin/etcdctl endpoint --cluster status && /usr/local/bin/etcdctl endpoint --cluster health 2>&1 | grep -v ‘Error: unhealthy cluster’ >/dev/null”,
“delta”: “0:00:10.076238”,
“end”: “2022-01-05 15:42:18.576307”,
“invocation”: {
“module_args”: {
"_raw_params": “set -o pipefail && /usr/local/bin/etcdctl endpoint --cluster status && /usr/local/bin/etcdctl endpoint --cluster health 2>&1 | grep -v ‘Error: unhealthy cluster’ >/dev/null”,
"_uses_shell": true,
“argv”: null,
“chdir”: null,
“creates”: null,
“executable”: “/bin/bash”,
“removes”: null,
“stdin”: null,
“stdin_add_newline”: true,
“strip_empty_ends”: true,
“warn”: true
}
},
“msg”: “non-zero return code”,
“rc”: 1,
“start”: “2022-01-05 15:42:08.500069”,
“stderr”: “{“level”:“warn”,“ts”:“2022-01-05T15:42:13.554+0800”,“caller”:“clientv3/retry_interceptor.go:62”,“msg”:“retrying of unary invoker failed”,“target”:“passthrough:///https://192.168.33.173:2379”,“attempt”:0,“error”:“rpc error: code = DeadlineExceeded desc = context deadline exceeded”}\nFailed to get the status of endpoint https://192.168.33.173:2379 (context deadline exceeded)\n{“level”:“warn”,“ts”:“2022-01-05T15:42:18.555+0800”,“caller”:“clientv3/retry_interceptor.go:62”,“msg”:“retrying of unary invoker failed”,“target”:“passthrough:///https://192.168.33.172:2379”,“attempt”:0,“error”:“rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \“transport: Error while dialing dial tcp 192.168.33.172:2379: connect: no route to host\””}\nFailed to get the status of endpoint https://192.168.33.172:2379 (context deadline exceeded)”,
“stderr_lines”: [
"{“level”:“warn”,“ts”:“2022-01-05T15:42:13.554+0800”,“caller”:“clientv3/retry_interceptor.go:62”,“msg”:“retrying of unary invoker failed”,“target”:“passthrough:///https://192.168.33.173:2379”,“attempt”:0,“error”:“rpc error: code = DeadlineExceeded desc = context deadline exceeded”}",
“Failed to get the status of endpoint https://192.168.33.173:2379 (context deadline exceeded)”,
"{“level”:“warn”,“ts”:“2022-01-05T15:42:18.555+0800”,“caller”:“clientv3/retry_interceptor.go:62”,“msg”:“retrying of unary invoker failed”,“target”:“passthrough:///https://192.168.33.172:2379”,“attempt”:0,“error”:“rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \“transport: Error while dialing dial tcp 192.168.33.172:2379: connect: no route to host\””}",
“Failed to get the status of endpoint https://192.168.33.172:2379 (context deadline exceeded)”
],
“stdout”: "https://192.168.33.171:2379, 4f98cfad1cb4c633, 3.4.13, 20 kB, true, false, 2, 8, 8, ",
“stdout_lines”: [
"https://192.168.33.171:2379, 4f98cfad1cb4c633, 3.4.13, 20 kB, true, false, 2, 8, 8, "
]
}

Answer 1

刘果国 2022-01-06 10:01:42

根据错误日志看：“192.168.33.172:2379” 这个地址无法访问。去这个节点上看看etcd的状态，系统日志、etcd的启动日志，看卡在哪里了？ 是镜像没下载还是启动报错？

• 提问者 weixin_慕圣2119594 #1

  ansible-playbook -i inventory/mycluster/hosts.yaml  -b cluster.yml -vvvv  这一句需要在每个节点都执行吗？

• 提问者 weixin_慕圣2119594 #2

  老师我看你前面的章节没有说哪里启动etcd啊。。。应该是哪一步出了问题呢

• 刘果国 回复 提问者 weixin_慕圣2119594 #3

  只能在一个节点执行。。。我没说要先启动etcd啊，有问题就要从错误入手分析问题，错误是提示etcd有问题我们就要排查为什么有问题，排查的方法就按照我上面的回复操作，查原因，为什么就这个节点部署etcd失败了

Answer 2

泰瑞亚小泰蜜 2023-03-06 10:53:54

我解决这个问题的方式是 ：各个节点的安全组端口全部开放

Answer 3

Mr_Lee_Bin 2022-08-05 09:13:14

我的报错“authentication handshake failed: remote error: tls: bad certificate”,“Failed to get the status of endpoint https://192.168.XXX.xxx:2379 (context deadline exceeded)”

解决： 虚机做了快照，恢复快照后，各节点时间没有调整，时间不对，导致证书不对，etcd那里报错。

Answer 4

提问者 weixin_慕圣2119594 2022-01-05 15:58:20

尝试很多次都不行 麻烦老师给详细解说一下

• 独孤狼子 #1

  我看你的报错：你的 master节点是无法访问192.168.33.172:2379 ，你可以在master节点 #telnet 192.168.33.172 2379 ，如果还不通的话 应该你的192.168.33.172没有关闭防火墙。
  关闭防火墙：#firewall-cmd --state
                      #systemctl stop firewalld.service
                      #systemctl disable firewalld.service

• 独孤狼子 #2

  看你的报错，应该是你的worker节点的防火墙没关导的吧

• Mr_Lee_Bin #3

  我的报错“authentication handshake failed: remote error: tls: bad certificate”“Failed to get the status of endpoint https://192.168.33.172:2379 (context deadline exceeded)”
  
  解决： 虚机做了快照，恢复快照后，各节点时间没有调整，时间不对，导致证书不对，etcd那步报错。