Write by lyc at 2021-7-12
etcd集群添加节点

etcd 集群增加或删除节点

1. 现有集群替换 etcd SSL 证书

etcd 集群服务器信息

名称 etcd版本 系统 IP 备注
etcd-1 3.4.16 CentOS7 192.168.100.190
etcd-2 3.4.16 CentOS7 192.168.100.191
etcd-3 3.4.16 CentOS7 192.168.100.192
etcd-4 3.4.16 CentOS7 192.168.100.193 测试新增和删除

1.1.重新生成 etcd SSL证书

因为在创建旧集群时 server-csr.json 里面写死了证书认证的 hosts，现在要添加新节点的 IP 进去。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$ cd /opt/tls/etcd

# 创建ssl证书所需的配置文件
$ cat > server-csr.json << EOF
{
    "CN": "etcd",
    "hosts": [
    "192.168.100.190",
    "192.168.100.191",
    "192.168.100.192",
    "192.168.100.193"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}
EOF

生成 etcd SSL证书

1
2
3
4
5
6
# 生成etcd https证书
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

# 查看证书
$ ls server*pem
server-key.pem  server.pem

1.2.复制 etcd SSL 证书到所有节点

1
2
3
4
5
6
7
8
9
10
# etcd1 覆盖本地
$ /bin/cp server*pem /opt/etcd/ssl/

# 复制 etcd SSL 新证书到当前另外两个节点
$ cd /opt/etcd/ssl/
$ scp server*pem root@192.168.100.191:/opt/etcd/ssl
$ scp server*pem root@192.168.100.192:/opt/etcd/ssl

# 复制证书到新节点
$ scp server*pem root@192.168.100.193:/opt/etcd/ssl

1.3.现有节点重启 etcd

1
$ systemctl restart etcd

检查状态 etcd 集群健康状态

1
2
3
4
5
6
7
8
9
10
11
12
13
$ ETCDCTL_API=3 /opt/etcd/bin/etcdctl \
  --cacert=/opt/etcd/ssl/ca.pem \
  --cert=/opt/etcd/ssl/server.pem \
  --key=/opt/etcd/ssl/server-key.pem \
  --endpoints="https://192.168.100.190:2379,https://192.168.100.191:2379,https://192.168.100.192:2379" endpoint health --write-out=table

+---------------------------+--------+-------------+-------+
|         ENDPOINT          | HEALTH |    TOOK     | ERROR |
+---------------------------+--------+-------------+-------+
|  https://192.168.100.190:2379 |   true |  13.65141ms |       |
| https://192.168.100.191:2379 |   true |  13.53077ms |       |
| https://192.168.100.192:2379 |   true | 14.625488ms |       |
+---------------------------+--------+-------------+-------+

2.etcd 集群添加新节点

2.1.查看当前集群节点信息

1
2
3
4
5
6
7
8
9
10
11
12
13
$ ETCDCTL_API=3 /opt/etcd/bin/etcdctl \
  --cacert=/opt/etcd/ssl/ca.pem \
  --cert=/opt/etcd/ssl/server.pem \
  --key=/opt/etcd/ssl/server-key.pem \
  --endpoints="https://192.168.100.190:2379" member list --write-out=table

+------------------+---------+--------+---------------------------+---------------------------+------------+
|        ID        | STATUS  |  NAME  |        PEER ADDRS         |       CLIENT ADDRS        | IS LEARNER |
+------------------+---------+--------+---------------------------+---------------------------+------------+
| 45dbb1fb9426c2d2 | started | etcd-1 |  https://192.168.100.190:2380 |  https://192.168.100.190:2379 |      false |
| 4c160a361379b50b | started | etcd-3 | https://192.168.100.192:2380 | https://192.168.100.192:2379 |      false |
| c513ebe4c737b731 | started | etcd-2 | https://192.168.100.191:2380 | https://192.168.100.191:2379 |      false |
+------------------+---------+--------+---------------------------+---------------------------+------------+

2.2.添加新节点到集群

1
2
3
4
5
6
7
8
9
10
11
12
13
$ ETCDCTL_API=3 /opt/etcd/bin/etcdctl \
  --cacert=/opt/etcd/ssl/ca.pem \
  --cert=/opt/etcd/ssl/server.pem \
  --key=/opt/etcd/ssl/server-key.pem \
  --endpoints="https://192.168.100.190:2379" member add etcd-4 --peer-urls="https://192.168.100.193:2380"
Member 1a7b7595a9dac5be added to cluster c9319597a443049f

ETCD_NAME="etcd-4"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.100.190:2380,etcd-3=https://192.168.100.192:2380,etcd-4=https://192.168.100.193:2380,etcd-2=https://192.168.100.191:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.100.193:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

# 新节点的 etcd.conf 配置文件必须包括以上输出内容

再次查看集群节点信息，可以看到 etcd-4 节点还未加入。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ ETCDCTL_API=3 /opt/etcd/bin/etcdctl \
  --cacert=/opt/etcd/ssl/ca.pem \
  --cert=/opt/etcd/ssl/server.pem \
  --key=/opt/etcd/ssl/server-key.pem \
  --endpoints="https://192.168.100.190:2379" member list --write-out=table

+------------------+-----------+--------+---------------------------+---------------------------+------------+
|        ID        |  STATUS   |  NAME  |        PEER ADDRS         |       CLIENT ADDRS        | IS LEARNER |
+------------------+-----------+--------+---------------------------+---------------------------+------------+
| 45dbb1fb9426c2d2 |   started | etcd-1 |  https://192.168.100.190:2380 |  https://192.168.100.190:2379 |      false |
| 4c160a361379b50b |   started | etcd-3 | https://192.168.100.192:2380 | https://192.168.100.192:2379 |      false |
| be74334aab055f5d | unstarted |        | https://192.168.100.193:2380 |                           |      false |
| c513ebe4c737b731 |   started | etcd-2 | https://192.168.100.191:2380 | https://192.168.100.191:2379 |      false |
+------------------+-----------+--------+---------------------------+---------------------------+------------+

2.3.新节点 etcd-4 配置

新节点必须 ETCD_INITIAL_CLUSTER_STATE="existing"

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ cat > /opt/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-4"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.100.193:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.100.193:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.100.193:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.100.193:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.100.190:2380,etcd-3=https://192.168.100.192:2380,etcd-4=https://192.168.100.193:2380,etcd-2=https://192.168.100.191:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="existing"
EOF

启动

1
$ systemctl start etcd

查看集群节点信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ ETCDCTL_API=3 /opt/etcd/bin/etcdctl \
  --cacert=/opt/etcd/ssl/ca.pem \
  --cert=/opt/etcd/ssl/server.pem \
  --key=/opt/etcd/ssl/server-key.pem \
  --endpoints="https://192.168.100.190:2379" member list --write-out=table

+------------------+---------+--------+---------------------------+---------------------------+------------+
|        ID        | STATUS  |  NAME  |        PEER ADDRS         |       CLIENT ADDRS        | IS LEARNER |
+------------------+---------+--------+---------------------------+---------------------------+------------+
| 45dbb1fb9426c2d2 | started | etcd-1 |  https://192.168.100.190:2380 |  https://192.168.100.190:2379 |      false |
| 4c160a361379b50b | started | etcd-3 | https://192.168.100.192:2380 | https://192.168.100.192:2379 |      false |
| 67f69799a6930884 | started | etcd-4 | https://192.168.100.193:2380 | https://192.168.100.193:2379 |      false |
| c513ebe4c737b731 | started | etcd-2 | https://192.168.100.191:2380 | https://192.168.100.191:2379 |      false |
+------------------+---------+--------+---------------------------+---------------------------+------------+

2.4.更新所有集群节点的配置文件

将各节点 etcd.conf 配置文件的变量 ETCD_INITIAL_CLUSTER 添加新节点信息，然后依次重启。

3.etcd 集群删除节点

3.1.查看节点 ID

1
2
3
4
5
6
7
8
9
10
11
12
13
$ ETCDCTL_API=3 /opt/etcd/bin/etcdctl \
  --cacert=/opt/etcd/ssl/ca.pem   \
  --cert=/opt/etcd/ssl/server.pem   \
  --key=/opt/etcd/ssl/server-key.pem   \
  --endpoints="https://192.168.100.190:2379" member list --write-out=table
+------------------+---------+--------+---------------------------+---------------------------+------------+
|        ID        | STATUS  |  NAME  |        PEER ADDRS         |       CLIENT ADDRS        | IS LEARNER |
+------------------+---------+--------+---------------------------+---------------------------+------------+
| 45dbb1fb9426c2d2 | started | etcd-1 |  https://192.168.100.190:2380 |  https://192.168.100.190:2379 |      false |
| 4c160a361379b50b | started | etcd-3 | https://192.168.100.192:2380 | https://192.168.100.192:2379 |      false |
| 67f69799a6930884 | started | etcd-4 |  http://192.168.100.193:2380 | https://192.168.100.193:2379 |      false |
| c513ebe4c737b731 | started | etcd-2 | https://192.168.100.191:2380 | https://192.168.100.191:2379 |      false |
+------------------+---------+--------+---------------------------+---------------------------+------------+

3.2.删除节点

1
2
3
4
5
6
$ ETCDCTL_API=3 /opt/etcd/bin/etcdctl \
  --cacert=/opt/etcd/ssl/ca.pem   \
  --cert=/opt/etcd/ssl/server.pem   \
  --key=/opt/etcd/ssl/server-key.pem   \
  --endpoints="https://192.168.100.190:2379" member remove 67f69799a6930884
Member 67f69799a6930884 removed from cluster c9319597a443049f

3.3.更新配置文件

最后，将各节点 etcd.conf 配置文件的变量 ETCD_INITIAL_CLUSTER 删除剔除的节点信息，然后依次重启。