1.k8s集群部署-Kubeadm方式

1.k8s集群部署-Kubeadm方式 | Evergreen Notes

k8s集群部署-Kubeadm方式

在部署高可用k8s之前，我们先来说一说单master架构和多master架构，以及多master架构中各组件工作逻辑 k8s单master架构

提示：这种单master节点的架构，通常只用于测试环境，生产环境绝对不允许；这是因为k8s集群master的节点是单点，一旦master节点宕机，将导致整个集群不可用；其次单master节点apiServer是性能瓶颈；从上图我们就可以看到，master节点所有组件和node节点中的kubelet和客户端kubectl、dashboard都会连接apiserver，同时apiserver还要负责往etcd中更新或读取数据，对客户端的请求做认证、准入控制等；很显然apiserver此时是非常忙碌的，极易成为整个K8S集群的瓶颈；所以不推荐在生产环境中使用单master架构； k8s多master架构

提示：k8s高可用主要是对master节点组件高可用；其中apiserver高可用的逻辑就是通过启用多个实例来对apiserver做高可用；apiserver从某种角度讲它应该是一个有状态服务，但为了降低apiserver的复杂性，apiserver将数据存储到etcd中，从而使得apiserver从有状态服务变成了一个无状态服务；所以高可用apiserver我们只需要启用多个实例通过一个负载均衡器来反向代理多个apiserver，客户端和node的节点的kubelet通过负载均衡器来连接apiserver即可；对于controller-manager、scheduler这两个组件来说，高可用的逻辑也是启用多个实例来实现的，不同与apiserver，这两个组件由于工作逻辑的独特性，一个k8s集群中有且只有一个controller-manager和scheduler在工作，所以启动多个实例它们必须工作在主备模式，即一个active，多个backup的模式；它们通过分布式锁的方式实现内部选举，决定谁来工作，最终抢到分布式锁（k8s集群endpoint）的controller-manager、scheduler成为active状态代表集群controller-manager、scheduler组件工作，抢到锁的controller-manager和scheduler会周期性的向apiserver通告自己的心跳信息，以维护自己active状态，避免其他controller-manager、scheduler进行抢占；其他controller-manager、scheduler收到活动的controller-manager、scheduler的心跳信息后自动切换为backup状态；一旦在规定时间备用controller-manager、scheduler没有收到活动的controller-manager、scheduler的心跳，此时就会触发选举，重复上述过程；

1.基础环境部署

宿主机网段、K8s Service网段、Pod网段不能重复。

VIP（虚拟IP）不要和公司内网IP重复，首先去ping一下，不通才可用VIP需要和你的主机在同一个局域网内。

公有云上搭建VIP是公有云的负载均衡的IP，比如阿里云的内网SLB/NLB的地址，腾讯云内网ELB的地址。不需要再搭建keepalived和haproxy

如果是私有云也需要问一下私有云管理员是否支持VIP！

k8s-master01    192.168.1.104
k8s-master02    192.168.1.105
k8s-master03    192.168.1.106
k8s-master-lb   192.168.1.236
k8s-node01       192.168.1.107
k8s-node02       192.168.1.108
 
Pod网段		172.16.0.0/16
Service网段	10.96.0.0/16

1.1.重新生成machine-id

提示：如果你的环境是通过某一个虚拟机基于快照克隆出来的虚拟机，很有可能对应machine-id一样，可以通过上述命令将对应虚拟机的machine-id修改成不一样；注意上述命令不能再crt中同时对多个虚拟机执行，同时对多个虚拟机执行，生成的machine-id是一样的；(所有节点执行)

BASH

1.2.所有节点更改主机名

BASH

所有节点配置hosts，修改/etc/hosts如下

BASH

1.3.所有节点配置yum源

BASH

1.4.所有节点必备工具安装

BASH

1.5.所有节点关闭防火墙、selinux、dnsmasq、swap

BASH

1.6.所有节点同步时间

BASH

1.7.所有节点配置limit

BASH

1.8.所有节点升级系统并重启，此处升级没有升级内核，下节会单独升级内核

BASH

yum update -y --exclude=kernel* && reboot #CentOS7需要升级，CentOS8可以按需升级系统

1.9.Master01节点免密钥登录其他节点

BASH

1.10.Master01节点下载安装所有的源码文件

BASH

cd /root/ ; git clone https://gitee.com/dukuan/k8s-ha-install.git

2.内核配置

CentOS7 需要升级内核至4.18+，本次升级的版本为4.19

2.1.在Master01节点下载内核

BASH

cd /root
wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm
wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm

2.2.从Master01节点传到其他节点

BASH

2.3.所有节点安装内核

BASH

2.4.所有节点安装ipvsadm

BASH

yum install ipvsadm ipset sysstat conntrack libseccomp -y

2.5.所有节点配置ipvs模块

在内核4.19+版本nf_conntrack_ipv4已经改为nf_conntrack， 4.18以下使用nf_conntrack_ipv4即可：

BASH

2.5.所有节点开启一些k8s集群中必须的内核参数，所有节点配置k8s内核：

BASH

2.6.所有节点配置完内核后，重启服务器，保证重启后内核依旧加载

BASH

reboot
lsmod | grep --color=auto -e ip_vs -e nf_conntrack

然后检查内核是不是4.19

BASH

# uname -a
Linux k8s-master02 4.19.12-1.el7.elrepo.x86_64 #1 SMP Fri Dec 21 11:06:36 EST 2018 x86_64 x86_64 x86_64 GNU/Linux

3.高可用组件安装

（注意：如果不是高可用集群，haproxy和keepalived无需安装）

公有云要用公有云自带的负载均衡，比如阿里云的SLB、NLB，腾讯云的ELB，用来替代haproxy和keepalived，因为公有云大部分都是不支持keepalived的，另外如果用阿里云的话，kubectl控制端不能放在master节点，推荐使用腾讯云，因为阿里云的slb有回环的问题，也就是slb代理的服务器不能反向访问SLB，但是腾讯云修复了这个问题。

3.1.所有Master节点通过yum安装HAProxy和KeepAlived

BASH

yum install keepalived haproxy -y

所有Master节点配置HAProxy（详细配置参考HAProxy文档，所有Master节点的HAProxy配置相同）：

BASH

3.2.所有Master节点配置KeepAlived，配置不一样，注意区分

[root@k8s-master01 pki]# vim /etc/keepalived/keepalived.conf ，注意每个节点的IP和网卡（interface参数）

3.2.1.Master01节点的配置：

BASH

3.2.2.Master02节点的配置：

BASH

3.2.3.Master03节点的配置：

BASH

3.3.所有master节点配置KeepAlived健康检查文件

SHELL

3.4所有master节点启动haproxy和keepalived

BASH

4.K8s组件和Runtime安装

如果安装的版本低于1.24，选择Docker和Containerd均可，高于1.24选择Containerd作为Runtime。

4.1.安装Containerd

所有节点安装docker-ce-20.10

BASH

 yum install docker-ce-20.10.* docker-ce-cli-20.10.* -y （containerd.io）

可以无需启动Docker，只需要配置和启动Containerd即可。首先配置Containerd所需的模块（所有节点）：

BASH

cat

所有节点加载模块:

BASH

 modprobe -- overlay
 modprobe -- br_netfilter

所有节点，配置Containerd所需的内核

BASH

所有节点加载内核

BASH

 sysctl --system

所有节点配置Containerd的配置文件

BASH

 mkdir -p /etc/containerd
 containerd config default |

所有节点将Containerd的Cgroup改为Systemd

BASH

 vim /etc/containerd/config.toml

找到containerd.runtimes.runc.options，添加或修改SystemdCgroup = true****（如果已存在直接修改，否则会报错），如下图所示：

所有节点将sandbox_image的Pause镜像改成符合自己版本的地址registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6

所有节点启动Containerd，并配置开机自启动：

BASH

 systemctl daemon-reload
 systemctl enable --now containerd

所有节点配置crictl客户端连接的运行时位置

BASH

4.2.安装Kubernetes组件

首先在Master01节点查看最新的Kubernetes版本是多少：

BASH

 yum list kubeadm.x86_64 --showduplicates | sort -r

所有节点安装1.28最新版本kubeadm、kubelet和kubectl：

BASH

 yum install kubeadm-1.28* kubelet-1.28* kubectl-1.28* -y

所有节点设置Kubelet开机自启动（由于还未初始化，没有kubelet的配置文件，此时kubelet无法启动，无需管理）：

BASH

 systemctl daemon-reload
 
 systemctl enable --now kubelet

此时kubelet是起不来的，日志会有报错不影响！

5.集群初始化

5.1.master01

YAML

更新kubeadm文件

BASH

kubeadm config migrate --old-config kubeadm-config.yaml --new-config new.yaml

将new.yaml文件复制到其他master节点,之后所有Master节点提前下载镜像，可以节省初始化时间（其他节点不需要更改任何配置，包括IP地址也不需要更改）：

BASH

for i

master01、02、03

BASH

5.2.Master01节点初始化

初始化以后会在/etc/kubernetes目录下生成对应的证书和配置文件，之后其他Master节点加入Master01即可：

BASH

初始化成功以后，会产生Token值，用于其他节点加入时使用，因此要记录下初始化成功生成的token值（令牌值）

Master01节点配置环境变量，用于访问Kubernetes集群：

BASH

Master01节点查看节点状态：（显示NotReady不影响）

BASH

5.3. 初始化失败排查

如果初始化失败，重置后再次初始化，命令如下（没有失败不要执行）：

BASH

kubeadm reset -f ; ipvsadm --clear  ; rm -rf ~/.kube

如果多次尝试都是初始化失败，需要看系统日志，CentOS日志路径:/var/log/messages，Ubuntu日志路径:/var/log/syslog：

BASH

tail -f /var/log/messages | grep -v "not found"

经常出错的原因：

Containerd的配置文件修改的不对，自行参考《安装containerd》小节核对
new.yaml配置问题，比如非高可用集群忘记修改16443端口为6443
new.yaml配置问题，三个网段有交叉，出现IP地址冲突
VIP不通导致无法初始化成功，此时messages日志会有VIP超时的报错

6.高可用Master

其他master加入集群，master02和master03分别执行

BASH

Token过期处理

注意：以下步骤是上述init命令产生的Token过期了才需要执行以下步骤，如果没有过期不需要执行，直接join即可。

Token过期后生成新的token：

BASH

kubeadm token create --print-join-command

Master需要生成--certificate-key

BASH

kubeadm init phase upload-certs  --upload-certs

7.Node节点的配置

Node节点上主要部署公司的一些业务应用，生产环境中不建议Master节点部署系统组件之外的其他Pod，测试环境可以允许Master节点部署Pod以节省系统资源。

BASH

8.Calico组件的安装

BASH

cd /root/k8s-ha-install && git checkout manual-installation-v1.28.x && cd calico/

修改Pod网段

BASH

9.Metrics部署

在新版的Kubernetes中系统资源的采集均使用Metrics-server，可以通过Metrics采集节点和Pod的内存、磁盘、CPU和网络的使用率。

将Master01节点的front-proxy-ca.crt复制到所有Node节点

BASH

以下操作均在master01节点执行

安装metrics server

BASH

查看状态

BASH

10.Dashboard部署

Dashboard用于展示集群中的各类资源，同时也可以通过Dashboard实时查看Pod的日志和在容器中执行一些命令等。

BASH

如果类型不是NodePort，则更改

BASH

kubectl edit svc kubernetes-dashboard -n kubernetes-dashboard

创建登录Token：

BASH

11.一些必须的配置更改

将Kube-proxy改为ipvs模式，因为在初始化集群的时候注释了ipvs配置，所以需要自行修改一下：

在master01节点执行

BASH

kubectl edit cm kube-proxy -n kube-system
 
mode: ipvs

更新Kube-Proxy的Pod：

BASH

验证Kube-Proxy模式

curl 127.0.0.1:10249/proxyMode ipvs

注意事项

kubeadm安装的集群，证书有效期默认是一年。master节点的kube-apiserver、kube-scheduler、kube-controller-manager、etcd都是以容器运行的。可以通过kubectl get po -n kube-system查看。

启动和二进制不同的是，

kubelet的配置文件在/etc/sysconfig/kubelet和/var/lib/kubelet/config.yaml，修改后需要重启kubelet进程

其他组件的配置文件在/etc/kubernetes/manifests目录下，比如kube-apiserver.yaml，该yaml文件更改后，kubelet会自动刷新配置，也就是会重启pod。不能再次创建该文件 kube-proxy的配置在kube-system命名空间下的configmap中，可以通过

BASH

kubectl edit cm kube-proxy -n kube-system

进行更改，更改完成后，可以通过patch重启kube-proxy

BASH

Kubeadm安装后，master节点默认不允许部署pod，可以通过以下方式删除Taint，即可部署Pod：

[root@k8s-master01 ~]# kubectl taint node -l node-role.kubernetes.io/control-plane node-role.kubernetes.io/control-plane:NoSchedule-

12.验证集群是否可用

12.1.节点均正常

BASH

12.2.Pod均正常

BASH

12.3.集群网段无任何冲突

BASH

12.4.能够正常创建资源

BASH

12.5.Pod 必须能够解析 Service(同 namespace 和跨namespace)

BASH

12.6.每个节点都必须要能访问 Kubernetes 的 kubernetes svc 443 和 kube-dns 的 service 53

BASH

12.7.Pod 和 Pod之前要能够正常通讯(同 namespace 和跨 namespace)

BASH

12.8.Pod 和 Pod之前要能够正常通讯(同机器和跨机器)

BASH

13.证书更新：

13.1.更新1年：

BASH

13.2.更新100年

BASH

hostnamectl set-hostname k8s-master01
hostnamectl set-hostname k8s-master02
hostnamectl set-hostname k8s-master03
hostnamectl set-hostname k8s-node01
hostnamectl set-hostname k8s-node02

cat > /etc/hosts<< EOF
192.168.1.104   k8s-master01
192.168.1.105   k8s-master02
192.168.1.106   k8s-master03
192.168.1.107   k8s-node01
192.168.1.108   k8s-node02  
EOF

curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo

systemctl disable --now firewalld 
systemctl disable --now dnsmasq
systemctl disable --now NetworkManager
 
setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
 
swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab

rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm
yum install ntpdate -y
 
ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
echo 'Asia/Shanghai' >/etc/timezone
ntpdate time2.aliyun.com
# 加入到crontab
*/5 * * * * /usr/sbin/ntpdate time2.aliyun.com

ulimit -SHn 65535
cat >> /etc/security/limits.conf <<EOF
# 末尾添加如下内容
* soft nofile 65536
* hard nofile 131072
* soft nproc 65535
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited
EOF

ssh-keygen -t rsa
 
for i in k8s-master01 k8s-master02 k8s-master03 k8s-node01 k8s-node02;do ssh-copy-id -i .ssh/id_rsa.pub $i;done

for i in k8s-master02 k8s-master03 k8s-node01 k8s-node02;do scp kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm $i:/root/ ; done

cd /root && yum localinstall -y kernel-ml*
 
所有节点更改内核启动顺序：
grub2-set-default  0 && grub2-mkconfig -o /etc/grub2.cfg
grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"
 
检查默认内核是不是4.19
[root@k8s-master02 ~]# grubby --default-kernel
/boot/vmlinuz-4.19.12-1.el7.elrepo.x86_64

modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
# 加入以下内容
cat >> /etc/modules-load.d/ipvs.conf <<EOF 
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF
 
然后执行systemctl enable --now systemd-modules-load.service即可（**报错不用管**）

cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
net.ipv4.conf.all.route_localnet = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720
 
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF
sysctl --system

 
[root@k8s-master01 etc]# vim /etc/haproxy/haproxy.cfg 
global
  maxconn  2000
  ulimit-n  16384
  log  127.0.0.1 local0 err
  stats timeout 30s
 
defaults
  log global
  mode  http
  option  httplog
  timeout connect 5000
  timeout client  50000
  timeout server  50000
  timeout http-request 15s
  timeout http-keep-alive 15s
 
frontend monitor-in
  bind *:33305
  mode http
  option httplog
  monitor-uri /monitor
 
frontend k8s-master
  bind 0.0.0.0:16443
  bind 127.0.0.1:16443
  mode tcp
  option tcplog
  tcp-request inspect-delay 5s
  default_backend k8s-master
 
backend k8s-master
  mode tcp
  option tcplog
  option tcp-check
  balance roundrobin
  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
  server k8s-master01	192.168.1.104:6443  check
  server k8s-master02	192.168.1.105:6443  check
  server k8s-master03	192.168.1.106:6443  check

[root@k8s-master01 etc]# mkdir /etc/keepalived
 
[root@k8s-master01 ~]# vim /etc/keepalived/keepalived.conf 
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2  
rise 1
}
vrrp_instance VI_1 {
    state MASTER
    interface ens33
    mcast_src_ip 192.168.1.104
    virtual_router_id 51
    priority 101
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.1.236
    }
    track_script {
       chk_apiserver
    }
}

! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
   interval 5
    weight -5
    fall 2  
rise 1
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    mcast_src_ip 192.168.1.105
    virtual_router_id 51
    priority 100
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.1.236
    }
    track_script {
       chk_apiserver
    }
}

! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
 interval 5
    weight -5
    fall 2  
rise 1
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    mcast_src_ip 192.168.1.106
    virtual_router_id 51
    priority 100
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.1.236
    }
    track_script {
       chk_apiserver
    }
}

[root@k8s-master01 keepalived]# cat /etc/keepalived/check_apiserver.sh 
#!/bin/bash
 
err=0
for k in $(seq 1 3)
do
    check_code=$(pgrep haproxy)
    if [[ $check_code == "" ]]; then
        err=$(expr $err + 1)
        sleep 1
        continue
    else
        err=0
        break
    fi
done
 
if [[ $err != "0" ]]; then
    echo "systemctl stop keepalived"
    /usr/bin/systemctl stop keepalived
    exit 1
else
    exit 0
fi
 
chmod +x /etc/keepalived/check_apiserver.sh

 systemctl daemon-reload
 systemctl enable --now haproxy
 systemctl enable --now keepalived
 
 #所有节点测试VIP
  ping 192.168.1.236 -c 4
  telnet 192.168.1.236 16443

 cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

 cat > /etc/crictl.yaml <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF

vim kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: 7t2weq.bjbawausm0jaxury
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.1.104
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  name: k8s-master01
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/control-plane
---
apiServer:
  certSANs:
  - 192.168.1.236
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: 192.168.1.236:16443
controllerManager: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.28.2 # 更改此处的版本号和kubeadm version一致
networking:
  dnsDomain: cluster.local
  podSubnet: 172.16.0.0/16
  serviceSubnet: 10.96.0.0/16
scheduler: {}

kubeadm config images pull --config /root/new.yaml 
 
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.28.2
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.28.2
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.28.2
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.28.2
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.9-0
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.10.1

[root@k8s-master01 ~]#kubeadm init --config /root/new.yaml  --upload-certs
[init] Using Kubernetes version: v1.28.2
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
W1029 13:36:25.150946    3727 checks.go:835] detected that the sandbox image "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9" as the CRI sandbox image.
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating

[root@k8s-master01 ~]#kubectl get node
NAME           STATUS     ROLES           AGE     VERSION
k8s-master01   NotReady   control-plane   2m57s   v1.28.2
[root@k8s-master01 ~]#kubectl get pod -n kube-system
NAME                                   READY   STATUS    RESTARTS   AGE
coredns-6554b8b87f-8rlsq               0/1     Pending   0          3m29s
coredns-6554b8b87f-jwxrb               0/1     Pending   0          3m29s
etcd-k8s-master01                      1/1     Running   0          3m33s
kube-apiserver-k8s-master01            1/1     Running   0          3m33s
kube-controller-manager-k8s-master01   1/1     Running   0          3m33s
kube-proxy-pj7xx                       1/1     Running   0          3m29s
kube-scheduler-k8s-master01            1/1     Running   0          3m33s
You have new mail in /var/spool/mail/root

kubeadm join 192.168.1.236:16443 --token 7t2weq.bjbawausm0jaxury \
	--discovery-token-ca-cert-hash sha256:979bdd2b4bda65b9504e9347d4bcf55b1e8b3e680e7d3868463f3b97cc784732 \
	--control-plane --certificate-key 1908ce159e6be6b0f0f9acd366f7bb3f7cd7a08e8b22d42447fb958a0bf30a40
 
 
[root@k8s-master01 ~]#kubectl get node
NAME           STATUS     ROLES           AGE     VERSION
k8s-master01   NotReady   control-plane   8m30s   v1.28.2
k8s-master02   NotReady   control-plane   83s     v1.28.2
k8s-master03   NotReady   control-plane   63s     v1.28.2

kubeadm join 192.168.1.236:16443 --token 7t2weq.bjbawausm0jaxury \
	--discovery-token-ca-cert-hash sha256:979bdd2b4bda65b9504e9347d4bcf55b1e8b3e680e7d3868463f3b97cc784732 
	
[root@k8s-master01 ~]#kubectl get node
NAME           STATUS     ROLES           AGE     VERSION
k8s-master01   NotReady   control-plane   11m     v1.28.2
k8s-master02   NotReady   control-plane   4m4s    v1.28.2
k8s-master03   NotReady   control-plane   3m44s   v1.28.2
k8s-node01     NotReady   <none>          19s     v1.28.2
k8s-node02     NotReady   <none>          14s     v1.28.2

POD_SUBNET=`cat /etc/kubernetes/manifests/kube-controller-manager.yaml | grep cluster-cidr= | awk -F= '{print $NF}'`
 
sed -i "s#POD_CIDR#${POD_SUBNET}#g" calico.yaml
kubectl apply -f calico.yaml
 
 
[root@k8s-master01 ~/k8s-ha-install/calico]#kubectl get pod -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-6d48795585-gbr42   1/1     Running   0          76s
calico-node-8n8t7                          1/1     Running   0          76s
calico-node-hmb6z                          1/1     Running   0          76s
calico-node-kl9gj                          1/1     Running   0          76s
calico-node-mzmwc                          1/1     Running   0          76s
calico-node-pqfbp                          1/1     Running   0          76s
coredns-6554b8b87f-8rlsq                   1/1     Running   0          25m
coredns-6554b8b87f-jwxrb                   1/1     Running   0          25m
etcd-k8s-master01                          1/1     Running   0          25m
etcd-k8s-master02                          1/1     Running   0          18m
etcd-k8s-master03                          1/1     Running   0          18m
kube-apiserver-k8s-master01                1/1     Running   0          25m
kube-apiserver-k8s-master02                1/1     Running   0          18m
kube-apiserver-k8s-master03                1/1     Running   0          18m
kube-controller-manager-k8s-master01       1/1     Running   0          25m
kube-controller-manager-k8s-master02       1/1     Running   0          18m
kube-controller-manager-k8s-master03       1/1     Running   0          18m
kube-proxy-ftjgh                           1/1     Running   0          18m
kube-proxy-gl6f5                           1/1     Running   0          15m
kube-proxy-pj7xx                           1/1     Running   0          25m
kube-proxy-ttz22                           1/1     Running   0          18m
kube-proxy-wkcfr                           1/1     Running   0          15m
kube-scheduler-k8s-master01                1/1     Running   0          25m
kube-scheduler-k8s-master02                1/1     Running   0          18m
kube-scheduler-k8s-master03                1/1     Running   0          18m
[root@k8s-master01 ~/k8s-ha-install/calico]#kubectl get node
NAME           STATUS   ROLES           AGE   VERSION
k8s-master01   Ready    control-plane   26m   v1.28.2
k8s-master02   Ready    control-plane   19m   v1.28.2
k8s-master03   Ready    control-plane   18m   v1.28.2
k8s-node01     Ready    <none>          15m   v1.28.2
k8s-node02     Ready    <none>          15m   v1.28.2

scp /etc/kubernetes/pki/front-proxy-ca.crt k8s-node01:/etc/kubernetes/pki/front-proxy-ca.crt
 
scp /etc/kubernetes/pki/front-proxy-ca.crt k8s-node(其他节点自行拷贝):/etc/kubernetes/pki/front-proxy-ca.crt

cd /root/k8s-ha-install/kubeadm-metrics-server
 
kubectl  create -f comp.yaml 
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created

[root@k8s-master01 ~/k8s-ha-install/kubeadm-metrics-server]#kubectl get po -n kube-system -l k8s-app=metrics-server
NAME                             READY   STATUS    RESTARTS   AGE
metrics-server-8df99c47f-rrng7   1/1     Running   0          57s
 
 
[root@k8s-master01 ~/k8s-ha-install/kubeadm-metrics-server]#kubectl top node
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
k8s-master01   164m         8%     1241Mi          32%       
k8s-master02   141m         7%     1063Mi          27%       
k8s-master03   145m         7%     1099Mi          28%       
k8s-node01     64m          3%     721Mi           18%       
k8s-node02

cd /root/k8s-ha-install/dashboard/
 
[root@k8s-master01 ~/k8s-ha-install/dashboard]#kubectl  create -f .
serviceaccount/admin-user created
clusterrolebinding.rbac.authorization.k8s.io/admin-user created
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
 
[root@k8s-master01 ~/k8s-ha-install/dashboard]#kubectl get svc kubernetes-dashboard -n kubernetes-dashboard
NAME                   TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)         AGE
kubernetes-dashboard   NodePort   10.96.83.229   <none>        443:31849/TCP   3m2s

kubectl create token admin-user -n kube-system
eyJhbGciOiJSUzI1NiIsImtpZCI6IlNWZ0Ric1BfTi1wRzhrSU1qUVJiMDE0YzMyQXlTN3ViMmJoUXd4OUZkTUUifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjk4NTYzNzk3LCJpYXQiOjE2OTg1NjAxOTcsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJhZG1pbi11c2VyIiwidWlkIjoiZDg0YWM3YzAtYWY3Yy00ZWVlLTliZjgtNmY2MDVhMDY5NTQ5In19LCJuYmYiOjE2OTg1NjAxOTcsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTphZG1pbi11c2VyIn0.OsEFz4uVUZtsB6wB9qykys6406SfnrQsU47sXa1RoeTz-oEAcGh8BxxtW8UtwMDKhmE0GjJV9mGNpMiGoEAM6d2EAA11n1cHA6OLECMKarEIbK8zZONSGmx82ItCoS8E7TW9fHXf5ODRo-acO8Egl8qzWoE6t5tGIMruyLkMm-4KV1yAJf5UX3ZU1Yx3uMRdgu2ezsHfn2TWMV2goWa6BwalqvZRN4rMa27P1oGgamC1kz1QSKnKL6Jk0G6v6GhQd4Ha0FJ5j09QfO5ufHmejRjDnKNg9jkJVOm2ZTN2Nx-FIw5FUc-JcNRM3bnGMztwlFxrbSyvrV8manUh4LKgBw

kubectl patch daemonset kube-proxy -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}" -n kube-system

kubectl patch daemonset kube-proxy -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}" -n kube-system

[root@k8s-master01 ~]#kubectl get node
NAME           STATUS   ROLES           AGE     VERSION
k8s-master01   Ready    control-plane   6h43m   v1.28.2
k8s-master02   Ready    control-plane   6h36m   v1.28.2
k8s-master03   Ready    control-plane   6h36m   v1.28.2
k8s-node01     Ready    <none>          6h33m   v1.28.2
k8s-node02     Ready    <none>          6h32m   v1.28.2

[root@k8s-master01 ~]#kubectl get pod -A
NAMESPACE              NAME                                         READY   STATUS    RESTARTS      AGE
default                busybox                                      1/1     Running   0             16m
default                net-test1                                    1/1     Running   0             22m
kube-system            calico-kube-controllers-6d48795585-gbr42     1/1     Running   1 (31m ago)   6h19m
kube-system            calico-node-8n8t7                            1/1     Running   1 (31m ago)   6h19m
kube-system            calico-node-hmb6z                            1/1     Running   1 (31m ago)   6h19m
kube-system            calico-node-kl9gj                            1/1     Running   1 (31m ago)   6h19m

[root@k8s-master01 ~]#kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   6h45m
[root@k8s-master01 ~]#kubectl get pod -A -owide
NAMESPACE              NAME                                         READY   STATUS    RESTARTS      AGE     IP              NODE           NOMINATED NODE   READINESS GATES
default                busybox                                      1/1     Running   0             17m     172.16.85.199   k8s-node01     <none>           <none>
default                net-test1                                    1/1     Running   0             23m     172.16.58.199   k8s-node02     <none>           <

kubectl run net-test1 --image=busybox:1.28.3 sleep 360000
kubectl run net-test1 --image=alpine sleep 360000
 
[root@k8s-master01 ~]#kubectl get pod -owide
NAME        READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
busybox     1/1     Running   0          19m   172.16.85.199   k8s-node01   <none>           <none>
net-test1   1/1     Running   0          25m   172.16.58.199   k8s-node02   <none>           <none>

kubectl exec -it busybox sh
 
/ # nslookup kubernetes
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
 
Name:      kubernetes
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local
/ # nslookup kube-dns.kube-system
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
 
Name:      kube-dns.kube-system
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

[root@k8s-master01 ~]#kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   6h42m
[root@k8s-master01 ~]#curl http://10.96.0.10:53
curl: (52) Empty reply from server
[root@k8s-master01 ~]#curl https://10.96.0.1:443
curl: (60) Peer's Certificate issuer is not recognized.
More details here: http://curl.haxx.se/docs/sslcerts.html
 
curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

 
/ # ping 172.16.58.199
PING 172.16.58.199 (172.16.58.199): 56 data bytes
64 bytes from 172.16.58.199: seq=0 ttl=62 time=0.535 ms
64 bytes from 172.16.58.199: seq=1 ttl=62 time=0.350 ms
^C
--- 172.16.58.199 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.350/0.442/0.535 ms

[root@k8s-master01 ~]#kubectl get pod -owide
NAME        READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
busybox     1/1     Running   0          19m   172.16.85.199   k8s-node01   <none>           <none>
net-test1   1/1     Running   0          25m   172.16.58.199   k8s-node02   <none>           <none>
 
/ # ping 172.16.58.199
PING 172.16.58.199 (172.16.58.199): 56 data bytes
64 bytes from 172.16.58.199: seq=0 ttl=62 time=0.535 ms
64 bytes from 172.16.58.199: seq=1 ttl=62 time=0.350 ms
^C
--- 172.16.58.199 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.350/0.442/0.535 ms

查看证书有效期: kubeadm certs check-expiration
备份原有的证书:cp -rp /etc/kubernetes/pki/opt/pki.bak
更新证书: kubeadm certs renew all
systemctl restart kubelet

kubeadm version
下载k8s源码:git clone https://gitee.com/mirrors/kubernetes.git切换分支到自己的k8s版本: 
git checkout v1.25.0根据自己版本切换
启动一个golang环境的容器:docker run -ti --rm -v `pwd`:/go/src/ registry.cn-beijing.aliyuncs.com/dotbalo/golang:kubeadm bash
以下在容器操作
cd /go/src/
go env -w GOPROXY=https://goproxy.cn,directgo env -w GOSUMDB=off
grep "365" cmd/kubeadm/app/constants/constants.gosed -i's#365#365*100#g' cmd/kubeadm/app/constants/constants.go
grep "365" cmd/kubeadm/app/constants/constants.go
mkdir -p _output/
chmod 777 -R_output/
make WHAT=cmd/kubeadm
cp_output/bin/kubeadm ./kubeadm
退出容器
cp kubeadm /opt/
/opt/kubeadm version
/opt/kubeadm certs renew all
kubeadm certs check-expiration
ystemctl restart kubelet