Kubernetes集群实践（十六）使用keepalived+haproxy创建高可用K8s集群

发表于

本文主要介绍如何使用keepalived+haproxy创建高可用集群，文中主要来源与作者在工作中的一些实际操作，本文仅供作者自己记录和读者参考。

关键词：高可用K8s集群

本文搭建的集群使用3个master节点和4个worker节点，为了节约资源，本文使用嵌入的etcd集群。

集群环境准备

集群拓扑

堆叠ETCD
  • 使用keepalived+haproxy的架构，worker节点通过访问vip和haproxy的后端服务器，实现了K8s集群的高可用；
  • 为了避免堆叠集群出现耦合失败的风险，应该为HA集群运行至少三个堆叠的控制节点；
  • 以上是kubeadm中的默认拓扑。当使用kubeadm init 和kubeadm join --control-plane --upload-certs时，在控制节点平面上自动创建etcd成员。

主机规划

主机IP地址 主机名 主机配置 主机角色 软件列表 其他IP
192.168.100.101 192.168.100.101-master 2C4G master + worker kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、containerd
192.168.100.102 192.168.100.102-master 2C4G master + worker kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、containerd
192.168.100.103 192.168.100.103-master 2C4G master + worker kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、containerd
192.168.100.105 192.168.100.105-worker 2C4G worker+LB1 kubelet、kube-proxy、containerd、keepalived、haproxy 192.168.100.104
192.168.100.106 192.168.100.106-worker 2C4G worker+LB1 kubelet、kube-proxy、containerd、keepalived、haproxy 192.168.100.104
192.168.100.107 192.168.100.107-worker 2C4G worker kubelet、kube-proxy、containerd
192.168.100.108 192.168.100.108-worker 2C4G worker kubelet、kube-proxy、containerd

软件版本

软件名称 版本 备注
CentOS8 kernel版本：4.18.0-477.27.1.el8_8.x86_64
kubernetes v1.28.2
etcd v3.5.9 最新版本
flannel v0.22.3 网络插件
containerd v1.7.6 容器运行时
haproxy v1.8.27
keepalived v2.1.5

网络分配

网络名称 网段 备注
Node网络 192.168.100.101/108 集群节点网络
Service网络 10.96.0.0/16 实现服务发现时所使用的网络
Pod网络 10.244.0.0/16

集群部署

为了方便集群部署，本文使用自动化运维工具ansible配置整个集群。

主要编辑两个文件，一是ansible的配置文件

1
2
3
4
5
6
# vim /etc/ansible/ansible.cfg
[defaults]
# hosts主机文件
inventory = ./hosts
# ssh免指纹
host_key_checking = False

二是ansible的主机文件

1
2
3
4
5
6
7
8
9
10
# vim /etc/ansible/hosts
[master]
192.168.100.101 ansible_ssh_user=root
192.168.100.102 ansible_ssh_user=root
192.168.100.103 ansible_ssh_user=root
[worker]
192.168.100.105 ansible_ssh_user=root
192.168.100.106 ansible_ssh_user=root
192.168.100.107 ansible_ssh_user=root
192.168.100.108 ansible_ssh_user=root

运行时下载

refer：https://github.com/containerd/nerdctl，需要下载full的文件

1
2
3
4
tar Cxzvvf /usr/local nerdctl-full-*-linux-amd64.tar.gz
systemctl enable buildkit.service --now
systemctl enable containerd.service --now
systemctl enable stargz-snapshotter.service --now

主机准备

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
# init-k8s.yml
- name: Set hosts
  hosts: all
  tasks:
    - ansible.builtin.shell: |
        cat > /etc/hosts << EOF
        127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
        ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

        192.168.100.101 192.168.100.101-master
        192.168.100.102 192.168.100.102-master
        192.168.100.103 192.168.100.103-master
        192.168.100.105 192.168.100.105-worker
        192.168.100.106 192.168.100.106-worker
        192.168.100.107 192.168.100.107-worker
        192.168.100.108 192.168.100.108-worker
        EOF
- name: Set master hostname
  hosts: master
  tasks:
    - hostname: name={{ ansible_enp6s18['ipv4']['address'] }}-master 
- name: set worker hostname
  hosts: worker
  tasks:
    - hostname: name={{ ansible_enp6s18['ipv4']['address'] }}-worker
- name: Set k8s env
  hosts: all
  remote_user: root
  tasks:
    - ansible.builtin.shell: |
        systemctl disable firewalld --now

        sed -i 's/enforcing/disabled/' /etc/selinux/config
        setenforce 0

        sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab
        swapoff -a

        yum install ipvsadm ipset sysstat conntrack libseccomp iproute-tc -y

        cat > /etc/modules-load.d/k8s.conf << EOF
        br_netfilter 
        ip_vs 
        ip_vs_lc 
        ip_vs_wlc 
        ip_vs_rr 
        ip_vs_wrr 
        ip_vs_lblc 
        ip_vs_lblcr 
        ip_vs_dh 
        ip_vs_sh 
        ip_vs_fo 
        ip_vs_nq 
        ip_vs_sed 
        ip_vs_ftp 
        ip_vs_sh 
        nf_conntrack 
        ip_tables 
        ip_set 
        xt_set 
        ipt_set 
        ipt_rpfilter 
        ipt_REJECT 
        ipip
        EOF

        systemctl enable --now systemd-modules-load.service

        lsmod | grep -e ip_vs -e nf-conntrack

        cat > /etc/sysctl.d/k8s.conf << EOF
        vm.swappiness = 0
        net.bridge.bridge-nf-call-ip6tables = 1
        net.bridge.bridge-nf-call-iptables = 1
        net.ipv4.ip_forward = 1
        fs.may_detach_mounts = 1
        vm.overcommit_memory = 1
        vm.panic_on.oom = 0
        fs.inotify.max_user_watches = 89100
        fs.file-max = 52706963
        fs.nr_open = 52706963
        net.netfilter.nf_conntrack_max = 2310720
        net.ipv4.tcp_keepalive_time = 600
        net.ipv4.tcp_keepalive_probes = 3
        net.ipv4.tcp_keepalive_intvl = 15
        net.ipv4.tcp_max_tw_buckets = 36000
        net.ipv4.tcp_tw_reuse = 1
        net.ipv4.tcp_max_orphans = 327680
        net.ipv4.tcp_orphan_retries = 3
        net.ipv4.tcp_syncookies = 1
        net.ipv4.tcp_max_syn_backlog = 16384
        net.ipv4.ip_contrack_max = 65536
        net.ipv4.tcp_timestamps = 0
        net.core.somaxconn = 16384
        EOF

        cat >> /etc/security/limits.d/k8s.conf <<EOF
        * soft nofile 1048576
        * hard nofile 1048576
        * soft noproc 1048576
        * hard noproc 1048576
        * soft memlock unlimited
        * hard memlock unlimited
        EOF
        exit 0

    - name: Copy file with owner and permissions
      ansible.builtin.copy:
        src: ./nerdctl-full-1.6.0-linux-amd64.tar.gz
        dest: /tmp/nerdctl-full-1.6.0-linux-amd64.tar.gz
    
    - name: Unzip nerdctl files
      ansible.builtin.unarchive:
        src: /tmp/nerdctl-full-1.6.0-linux-amd64.tar.gz
        dest: /usr/local/
    
    - name: Restart Service
      ansible.builtin.shell: |
        systemctl enable buildkit.service --now
        systemctl enable containerd.service --now
        systemctl enable stargz-snapshotter.service --now
        mkdir -p /etc/containerd
        containerd config default |  tee /etc/containerd/config.toml

    - name: Init containerd Service
      ansible.builtin.shell: |
        systemctl enable buildkit.service --now
        systemctl enable containerd.service --now
        systemctl enable stargz-snapshotter.service --now
        mkdir -p /etc/containerd
        containerd config default |  tee /etc/containerd/config.toml
        sed -ri "s|registry.k8s.io|pkgs.fly97.cn|g" /etc/containerd/config.toml

    - name: Install kubeadm repo
      ansible.builtin.shell: |
        cat > /etc/yum.repos.d/kubernetes.repo << EOF
        [kubernetes]
        name=Kubernetes for EPEL 
        baseurl=https://pkgs.fly97.cn/repository/kubernetes-yum/
        enabled=1
        gpgcheck=0
        EOF
        dnf makecache -y
        dnf install kubelet kubeadm kubectl -y
        systemctl stop kubelet

负载均衡

在105和106机器上分别安装keepalived+HAproxy

1
sudo dnf install keepalived haproxy -y

haproxy

以下是haproxy的配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
cat > /etc/haproxy/haproxy.cfg << EOF
global
    log         127.0.0.1 local2
    pidfile     /var/run/haproxy.pid
    maxconn     65535
    daemon

defaults
    timeout connect 5000
    timeout client 50000
    timeout server 50000
    timeout http-request 15s
    timeout http-keep-alive 15s
    retries    3
    maxconn    65535

frontend  kubernetes-apiserver
    mode tcp
    bind *:6443
    tcp-request inspect-delay 5s
    default_backend  kubernetes-apiserver
    
backend kubernetes-apiserver
    mode        tcp
    balance     roundrobin
    server master01 192.168.100.101:6443 check inter 2000 rise 2 fall 2
    server master02 192.168.100.102:6443 check inter 2000 rise 2 fall 2
    server master03 192.168.100.103:6443 check inter 2000 rise 2 fall 2
EOF

keepalived

以下是106机器keepalived的配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
cat > /etc/keepalived/keepalived.conf << EOF
global_defs {
   max_auto_priority
}
vrrp_script check_apiserver {
   script "/etc/keepalived/check_apiserver_vip.sh"
   interval 2
   weight 5
}
vrrp_instance VI_02 {
   # BACKUP模式
   state BACKUP
   nopreempt
   interface enp6s18
   # 每个集群的router_id要一致
   virtual_router_id 100
   priority 50
   unicast_src_ip 192.168.100.106
   unicast_peer{
      192.168.100.105
   }
   # 虚IP地址
   virtual_ipaddress {
     192.168.100.104/24
   }
   track_script {
     check_apiserver
   }
}
EOF

以下是105机器keepalived的配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
cat > /etc/keepalived/keepalived.conf << EOF
global_defs {
  max_auto_priority
}
vrrp_script check_apiserver {
   script "/etc/keepalived/check_apiserver_vip.sh"
   interval 2
   weight 5
   timeout 1
}
vrrp_instance VI_01 {
   # MASTER模式
   state MASTER
   nopreempt
   interface enp6s18
   # 每个集群的router_id要一致
   virtual_router_id 100
   priority 100
   unicast_src_ip 192.168.100.105
   unicast_peer {
     192.168.100.106
   }
   # 虚IP地址
   virtual_ipaddress {
     192.168.100.104/24
   }
   track_script {
     check_apiserver
   }
}
EOF

以下是检查api_server的脚本文件

1
2
3
4
5
6
7
8
9
10
11
12
13
# vim /etc/keepalived/check_apiserver_vip.sh
#!/bin/sh
killall -0 haproxy
if [ $? == 0 ];then
   http_code=$(curl -k -s -o /dev/null -w "%{http_code}\n" https://127.0.0.1:6443)
   if [ $http_code == 403 ];then
      exit 0
   else
      exit 1
   fi
else
   exit 1
fi

部署master和node

在101主机上执行

1
kubeadm init --config init-k8s.yml --upload-certs

没有问题的话会安装成功

在102、103master主机上执行上面第一行的命令，在剩下的worker节点执行下面的命令

主机 命令
102、103等master节点 kubeadm join 192.168.100.104:6443 --token xxx --discovery-token-ca-cert-hash xxx --control-plane --certificate-key xxx
其他worker节点 kubeadm join 192.168.100.104:6443 --token xxx --discovery-token-ca-cert-hash xxx

执行成功后，在任一master节点下执行kubectl get nodes，会显示下面的节点状态

此时节点状态仍然是NotReady，通过kubelet日志可以看到网络插件尚未安装。

部署网络插件

本文使用flannel作为K8s的网络插件：Flannel 是一种简单易用的方法来配置专为 Kubernetes 设计的第 3 层网络结构。

refer：https://github.com/flannel-io/flannel

资源清单可以参考：https://github.com/flannel-io/flannel/blob/master/Documentation/kube-flannel.yml

主要需要修改image镜像地址，镜像可以参考：https://sci.nju.edu.cn/9e/05/c30384a564741/page.htm

1
kubectl apply -f kube-flannel.yml

等待kube-flannel中pod状态变为running即可

此时集群状态正常，安装完毕

高可用检查

安装etcdctl

refer：https://github.com/etcd-io/etcd/releases/tag/v3.5.9

添加环境变量实现自动别名操作

1
2
3
# vim .bash_profile
export ETCDCTL_API=3
alias etcdctl='etcdctl --endpoints=https://192.168.100.101:2379,https://192.168.100.102:2379,https://192.168.100.103:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key

查看etcd endpoint状态

查看etcd endpoint健康

未完待续。。。