4.ubuntu基于kubeasz部署高可用k8s集群 | Evergreen Notes
返回首页 4.ubuntu基于kubeasz部署高可用k8s集群 1.ubuntu的基础配置 1.1.允许root用户登录 1.2.修改网络地址 1.3.添加清华源镜像 2.地址规划 3.基础环境部署 修改主机名 重新生成machine id 4.基于keepalived及haproxy部署高可用负载均衡 4.1.配置keepalived 在haproxy01上创
1.ubuntu的基础配置
1.1.允许root用户登录
1. 修改 root 密码 sudo passwd root 2. 以其他账户登录，通过vi修改 /etc/ssh/sshd_config : server@ubuntu:~$ su root Password: root@ubuntu:~# vi /etc/ssh/sshd_config 3.修改PermitRootLogin without-passwor 为 PermitRootLogin yes root@docker-server1:~# grep -i permitrootlogin /etc/ssh/sshd_config #PermitRootLogin prohibit-password # the setting of "PermitRootLogin without-password". PermitRootLogin yes 4.重启 ssh 服务 root@ubuntu:~# service ssh restart 1.2.修改网络地址
1.3.添加清华源镜像
2.地址规划
3.基础环境部署
修改主机名
重新生成machine-id
4.基于keepalived及haproxy部署高可用负载均衡
4.1.配置keepalived
在haproxy01上创建/etc/keepalived/keepalived.conf
在haproxy02上创建/etc/keepalived/keepalived.conf
在haproxy02上启动keepalived并设置开机启动
在haproxy01上启动keepalived并设置开机启动 vip地址会漂移至haproxy02
至此keepalived配置好了
4.2.配置haproxy
默认情况下内核不允许监听本机不存在的socket，我们需要修改内核参数允许本机监听不存在的socket
在haproxy01和haproxy02上操作
haproxy01和haproxy02配置一样
systemctl start haproxy systemctl restart haproxy systemctl 至此基于keepalived及haproxy部署高可用负载均衡器就部署完成
4.3.部署https harbor服务提供镜像的分发
在harbor服务器上配置docker-ce的源
在harbor服务器上安装docker和docker-compose
下载docker-compose二进制文件
root@harbor:~# wget https://github.com/docker/compose/releases/download/v2.22.0/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose 给docker-compose二进制文件加上可执行权限，并验证docker-compose的版本
创建存放harbor离线安装包目录，并将离线安装包解压于对应目录，创建存放证书的目录certs上传证书
编辑harbor.yml配置文件
给harbor提供service文件，实现开机自动启动
所有主机/etc/hosts解析
测试基于nerdctl可以登录https harbor并能实现进行分发
5.基于kubeasz部署高可用kubernetes集群
5.1.部署节点免秘钥登录配置
本次我们使用kubeasz项目来部署二进制高可用k8s集群；项目地址：https://github.com/easzlab/kubeasz；该项目使用`ansible-playbook`实现自动化，提供一件安装脚本，也可以分步骤执行安装各组件；所以部署节点首先要安装好ansible,其次该项目使用docker下载部署k8s过程中的各种镜像以及二进制，所以部署节点docker也需要安装好，当然如果你的部署节点没有安装docker，它也会自动帮你安装；
部署节点配置docker源
部署节点安装ansible、docker
部署节点安装sshpass命令⽤于同步公钥到各k8s服务器 编写分发公钥脚本
5.2.下载kubeasz项目安装脚本
编辑ansible hosts配置文件 hosts配置文件主要用来指定etcd节点、master节点、node节点、vip、运行时、网络组件类型、service IP与pod IP范围等配置信息
编辑cluster config.yml文件
准备CA和基础环境初始化
部署etcd集群
部署容器运行时containerd
配置nerdctl客户端
在master节点使用nerdctl 下载上传镜像
部署k8s master node节点
部署网络组件calico
使kubectl命令可以在其他节点执行
5.3.添加、删除master节点和node节点
5.4.升级集群
升级前需提前下载好用于更新的master 组件二进制、node节点组件二进制以及客户端二进制
5.5.部署k8s内部域名解析服务coredns
6.部署官方dashboard
7.部署第三方dashboard kuboard
访问kuboard 默认用户名和密码是admin/Kuboard123
导入集群
添加集群的方式有两种，一种是agent，一种是kubeconfig；agent这种方式需要在k8s上额外运行一个pod来作为agent；
agent方式导入集群
提示：把上述命令在k8s集群上执行下就好；
kubeconfig方式导入集群
：需要将制作好的kubeconfig文件内容直接粘贴进来即可
提示：kuboard它可用管理多个集群；
选择使用集群的身份
第三方dashboard kuboard就部署好了，后面我们管理k8s集群就可用在kuboard上面点点；如果需要卸载kuboard ，只需要在k8s上将部署清单delete一下即可；kubectl delete -f kuboard-all-in-one.yaml；
8.部署第三方dashboard kubesphere
提前准备存储类相关部署清单
授权清单
9.kubectl命令补全
root@docker-server2:~# cat /etc/netplan/00-installer-config.yaml # This is the network config written by 'subiquity' network: ethernets: ens33: addresses: - 192.168.234.202/24 gateway4: 192.168.234.2 nameservers: addresses: - 114.114.114.114 - 8.8.8.8 search: [] version: 2 # 生效 netplan apply
# 添加net.ifnames=0 biosdevname=0 root@docker-server1:~# cat /etc/default/grub | grep GRUB_CMDLINE_LINUX GRUB_CMDLINE_LINUX_DEFAULT = "" GRUB_CMDLINE_LINUX = "net.ifnames=0 biosdevname=0" update-grub
reboot
cp /etc/apt/sources.list /etc/apt/sources.list.bak vim /etc/apt/sources.list # 默认注释了源码镜像以提高 apt update 速度，如有需要可自行取消注释 deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy main restricted universe multiverse # deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy main restricted universe multiverse deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-updates main restricted universe multiverse # deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-updates main restricted universe multiverse deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-backports main restricted universe multiverse # deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-backports main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu/ jammy-security main restricted universe multiverse # deb-src http://security.ubuntu.com/ubuntu/ jammy-security main restricted universe multiverse
# 预发布软件源，不建议启用 # deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-proposed main restricted universe multiverse # # deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-proposed main restricted universe multiverse
sudo apt update
192.168.20.235 k8s-master01 192.168.20.236 k8s-master02 192.168.20.237 k8s-master03 192.168.20.238 k8s-node01 192.168.20.239 k8s-node02 192.168.20.240 k8s-node03 192.168.20.241 k8s-etcd01 192.168.20.242 k8s-etcd02 192.168.20.243 k8s-etcd03 192.168.20.244 haproxy01 192.168.20.245 haproxy02 192.168.20.246 harbor 192.168.20.247 deploy 部署节点
192.168.20.248 vip
hostnamectl set-hostname k8s-master01 hostnamectl set-hostname k8s-master02 hostnamectl set-hostname k8s-master03 hostnamectl set-hostname k8s-node01 hostnamectl set-hostname k8s-node02 hostnamectl set-hostname k8s-node03 hostnamectl set-hostname k8s-etcd01 hostnamectl set-hostname k8s-etcd02 hostnamectl set-hostnamek8s-etcd03 hostnamectl set-hostname haproxy01 hostnamectl set-hostname haproxy02 hostnamectl set-hostname harbor hostnamectl set-hostname deploy
root@k8s-deploy:~# cat /etc/machine-id 1d2aeda997bd417c838377e601fd8e10 root@k8s-deploy:~# rm -rf /etc/machine-id && dbus-uuidgen --ensure=/etc/machine-id && cat /etc/machine-id 8340419bb01397bf654c596f6443cabf root@k8s-deploy:~#
root@haproxy01:~# apt update && apt install keepalived haproxy -y root@haproxy02:~# apt update && apt install keepalived haproxy -y
cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { acassen } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id LVS_DEVEL } vrrp_instance VI_1 { state MASTER interface ens160 garp_master_delay 10 smtp_alert virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.20.248 dev ens160 label ens160:0 } }
cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { acassen } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id LVS_DEVEL } vrrp_instance VI_1 { state BACKUP interface ens160 garp_master_delay 10 smtp_alert virtual_router_id 51 priority 90 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.20.248 dev ens160 label ens160:0 } }
systemctl start keepalived systemctl enable keepalived
root@haproxy02:~# ip a 1: lo: < LOOPBACK,UP,LOWER_U P> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens160: < BROADCAST,MULTICAST,UP,LOWER_U P> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:50:56:ac:25:7c brd ff:ff:ff:ff:ff:ff altname enp3s0 inet 192.168.20.245/24 brd 192.168.20.255 scope global ens160 valid_lft forever preferred_lft forever inet 192.168.20.248/32 scope global ens160:0 valid_lft forever preferred_lft forever inet6 fe80::250:56ff:feac:257c/64 scope link valid_lft forever preferred_lft forever
root@haproxy01:~# ip a 1: lo: < LOOPBACK,UP,LOWER_U P> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens160: < BROADCAST,MULTICAST,UP,LOWER_U P> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:50:56:ac:b7:3b brd ff:ff:ff:ff:ff:ff altname enp3s0 inet 192.168.20.244/24 brd 192.168.20.255 scope global ens160 valid_lft forever preferred_lft forever inet 192.168.20.248/32 scope global ens160:0 valid_lft forever preferred_lft forever inet6 fe80::250:56ff:feac:b73b/64 scope link valid_lft forever preferred_lft forever
root@haproxy02:~# ip a 1: lo: < LOOPBACK,UP,LOWER_U P> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens160: < BROADCAST,MULTICAST,UP,LOWER_U P> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:50:56:ac:25:7c brd ff:ff:ff:ff:ff:ff altname enp3s0 inet 192.168.20.245/24 brd 192.168.20.255 scope global ens160 valid_lft forever preferred_lft forever inet6 fe80::250:56ff:feac:257c/64 scope link valid_lft forever preferred_lft forever
sysctl -a | grep bind
root@haproxy02:~# sysctl -a | grep bind net.ipv4.ip_autobind_reuse = 0 net.ipv4.ip_nonlocal_bind = 0 net.ipv6.bindv6only = 0 net.ipv6.ip_nonlocal_bind = 0
echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf sysctl -p sysctl -a | grep bind
root@haproxy02:~# sysctl -a | grep bind net.ipv4.ip_autobind_reuse = 0 net.ipv4.ip_nonlocal_bind = 1 net.ipv6.bindv6only = 0 net.ipv6.ip_nonlocal_bind = 0
global log /dev/log local0 log /dev/log local1 notice chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners stats timeout 30s user haproxy group haproxy daemon # Default SSL material locations ca-base /etc/ssl/certs crt-base /etc/ssl/private # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384 ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets defaults log global mode http option httplog option dontlognull timeout connect 5000 timeout client 50000 timeout server 50000 errorfile 400 /etc/haproxy/errors/400.http errorfile 403 /etc/haproxy/errors/403.http errorfile 408 /etc/haproxy/errors/408.http errorfile 500 /etc/haproxy/errors/500.http errorfile 502 /etc/haproxy/errors/502.http errorfile 503 /etc/haproxy/errors/503.http errorfile 504 /etc/haproxy/errors/504.http listen k8s_apiserver_6443 bind 192.168.20.248:6443 mode tcp #balance leastconn server k8s-master01 192.168.20.235:6443 check inter 2000 fall 3 rise 5 server k8s-master02 192.168.20.236:6443 check inter 2000 fall 3 rise 5 server k8s-master03 192.168.20.237:6443 check inter 2000 fall 3 rise 5
enable
haproxy
apt-get update apt-get -y install apt-transport-https ca-certificates curl software-properties-common curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | sudo apt-key add - add-apt-repository "deb [arch=amd64] https://mirrors.aliyun.com/docker-ce/linux/ubuntu $( lsb_release -cs ) stable" apt-get -y update
root@harbor:~# apt-cache madison docker-ce docker-ce | 5:24.0.7-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages docker-ce | 5:24.0.6-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages docker-ce | 5:24.0.5-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages docker-ce | 5:24.0.4-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages docker-ce | 5:24.0.3-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages docker-ce | 5:24.0.2-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages chmod +x /usr/local/bin/docker-compose
root@harbor:/usr/local/bin# docker-compose -v Docker Compose version v2.22.0
mkdir /app tar xf harbor-offline-installer-v2.8.0.tgz -C /app/ root@harbor:~# cd /app/ root@harbor:/app# ls harbor
mkdir certs root@harbor:/app/certs# ls harbor.nbbnai.com.key harbor.nbbnai.com.pem #阿里云申请的证书
root@harbor:/app/harbor# grep -v "#" harbor.yml | grep -v "^#" hostname : harbor.nbbnai.com http : port : 80 https : port : 443 certificate : /app/certs/harbor.nbbnai.com.pem private_key : /app/certs/harbor.nbbnai.com.key harbor_admin_password : Harbor12345 database : password : root123 max_idle_conns : 100 max_open_conns : 900 conn_max_lifetime : 5m conn_max_idle_time : 0 data_volume : /data trivy : ignore_unfixed : false skip_update : false cat /usr/lib/systemd/system/harbor.service [Unit] Description = Harbor After = docker.service systemd-networkd.service systemd-resolved.service Requires = docker.service Documentation = http://github.com/vmware/harbor [Service] Type = simple Restart = on-failure RestartSec = 5 ExecStart = /usr/local/bin/docker-compose -f /app/harbor/docker-compose.yml up ExecStop = /usr/local/bin/docker-compose -f /app/harbor/docker-compose.yml down [Install] WantedBy = multi-user.target
systemctl daemon-reload systemctl restart harbor systemctl enable harbor
cat >> /etc/hosts << EOF 192.168.20.235 k8s-master01 192.168.20.236 k8s-master02 192.168.20.237 k8s-master03 192.168.20.238 k8s-node01 192.168.20.239 k8s-node02 192.168.20.240 k8s-node03 192.168.20.241 k8s-etcd01 192.168.20.242 k8s-etcd02 192.168.20.243 k8s-etcd03 192.168.20.244 haproxy01 192.168.20.245 haproxy02 192.168.20.246 harbor harbor.nbbnai.com EOF
docker login harbor.nbbnai.com
root@harbor:/app/harbor# docker login harbor.nbbnai.com Username: admin Password: WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
root@harbor:/app/harbor# docker pull nginx Using default tag: latest latest: Pulling from library/nginx 578acb154839: Pull complete e398db710407: Pull complete 85c41ebe6d66: Pull complete 7170a263b582: Pull complete 8f28d06e2e2e: Pull complete 6f837de2f887: Pull complete c1dfc7e1671e: Pull complete Digest: sha256:86e53c4c16a6a276b204b0fd3a8143d86547c967dc8258b3d47c3a21bb68d3c6 Status: Downloaded newer image for nginx:latest docker.io/library/nginx:latest root@harbor:/app/harbor# docker images REPOSITORY TAG IMAGE ID CREATED SIZE nginx latest c20060033e06 6 days ago 187MB root@harbor:/app/harbor# docker tag nginx harbor.nbbnai.com/images/nginx:v1 root@harbor:/app/harbor# docker push harbor.nbbnai.com/images/nginx:v1 The push refers to repository [harbor.nbbnai.com/images/nginx] 505f49f13fbe: Pushed 9920f1ebf52b: Pushed 768e28a222fd: Pushed 715b32fa0f12: Pushed e503754c9a26: Pushed 609f2a18d224: Pushed ec983b166360: Pushed v1: digest: sha256:d2e65182b5fd330470eca9b8e23e8a1a0d87cc9b820eb1fb3f034bf8248d37ee size: 1778
apt-get update apt-get -y install apt-transport-https ca-certificates curl software-properties-common curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | sudo apt-key add - add-apt-repository "deb [arch=amd64] https://mirrors.aliyun.com/docker-ce/linux/ubuntu $( lsb_release -cs ) stable" apt-get -y update
root@ubuntu:~# apt-cache madison ansible docker-ce ansible | 2.10.7+merged+base+2.10.8+dfsg-1 | https://mirrors.tuna.tsinghua.edu.cn/ubuntu jammy/universe amd64 Packages docker-ce | 5:24.0.7-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages docker-ce | 5:24.0.6-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages docker-ce | 5:24.0.5-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages docker-ce | 5:24.0.4-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages docker-ce | 5:24.0.3-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages apt install sshpass -y
ssh-keygen -t rsa-sha2-512 -b 4096
cat pub-key-scp.sh #!/bin/bash #⽬标主机列表 HOSTS = " 192.168.20.235 192.168.20.236 192.168.20.237 192.168.20.238 192.168.20.239 192.168.20.240 192.168.20.241 192.168.20.242 192.168.20.243 192.168.20.244 192.168.20.245 192.168.20.246 " REMOTE_PORT = "22" REMOTE_USER = "root" REMOTE_PASS = "Rcy2Ka9^YT" for REMOTE_HOST in ${ HOSTS }; do REMOTE_CMD = "echo ${ REMOTE_HOST } is successfully!" #添加目标远程主机的公钥 ssh-keyscan -p "${ REMOTE_PORT }" "${ REMOTE_HOST }" >> ~/.ssh/known_hosts #通过sshpass配置免秘钥登录、并创建python3软连接 sshpass -p "${ REMOTE_PASS }" ssh-copy-id "${ REMOTE_USER }@${ REMOTE_HOST }" ssh ${ REMOTE_HOST } ln -sv /usr/bin/python3 /usr/bin/python echo ${ REMOTE_HOST } 免秘钥配置完成! done
root@ubuntu:~# apt install git -y root@ubuntu:~# export release= 3.6.2 root@ubuntu:~# wget https://github.com/easzlab/kubeasz/releases/download/ ${ release } /ezdown 编辑ezdown 给脚本添加执行权限 chmod a+x ezdown 执行脚本，下载kubeasz项目及组件 执行ezdown脚本它会下载一些镜像和二进制工具等，并将下载的二进制工具和kubeasz项目存放在/etc/kubeasz/目录中 ./ezdown -D
root@deploy:/etc/kubeasz# ll total 140 drwxrwxr-x 12 root root 4096 Nov 7 14:08 ./ drwxr-xr-x 85 root root 4096 Nov 8 00:44 ../ drwxrwxr-x 4 root root 4096 Sep 4 13:54 root@deploy:/etc/kubeasz# cat /etc/kubeasz/clusters/k8s-cluster01/hosts # 'etcd' cluster should have odd member(s) (1,3,5,...) [etcd] #etcd主机 192.168.20.241 192.168.20.242 192.168.20.243
# master node(s), set unique 'k8s_nodename' for each node # CAUTION: 'k8s_nodename' must consist of lower case alphanumeric characters, '-' or '.', # and must start and end with an alphanumeric character [kube_master] #master节点现在指定两个后面扩展 192.168.20.235 k8s_nodename='master-01' 192.168.20.236 k8s_nodename='master-02'
# work node(s), set unique 'k8s_nodename' for each node # CAUTION: 'k8s_nodename' must consist of lower case alphanumeric characters, '-' or '.', # and must start and end with an alphanumeric character [kube_node] #node节点 192.168.20.238 k8s_nodename='node-01' 192.168.20.239 k8s_nodename='node-02'
# [optional] harbor server, a private docker registry # 'NEW_INSTALL': 'true' to install a harbor server; 'false' to integrate with existed one [harbor] #192.168.1.8 NEW_INSTALL=false
# [optional] loadbalance for accessing k8s from outside [ex_lb] #192.168.1.6 LB_ROLE=backup EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443 #192.168.1.7 LB_ROLE=master EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443
# [optional] ntp server for the cluster [chrony] #192.168.1.1
[all:vars] # --------- Main Variables --------------- # Secure port for apiservers SECURE_PORT = "6443"
# Cluster container-runtime supported: docker, containerd # if k8s version >= 1.24, docker is not supported CONTAINER_RUNTIME = "containerd"
# Network plugins supported: calico, flannel, kube-router, cilium, kube-ovn CLUSTER_NETWORK = "calico"
# Service proxy mode of kube-proxy: 'iptables' or 'ipvs' PROXY_MODE = "ipvs"
# K8S Service CIDR, not overlap with node(host) networking SERVICE_CIDR = "10.100.0.0/16" #service网段
# Cluster CIDR (Pod CIDR), not overlap with node(host) networking CLUSTER_CIDR = "10.200.0.0/16" #pod网段
# NodePort Range NODE_PORT_RANGE = "30000-32767"
# Cluster DNS Domain CLUSTER_DNS_DOMAIN = "cluster.local"
# -------- Additional Variables (don't change the default value right now) --- # Binaries Directory bin_dir = "/usr/local/bin" #二进制文件路径
# Deploy Directory (kubeasz workspace) base_dir = "/etc/kubeasz"
# Directory for a specific cluster cluster_dir = "{{ base_dir }}/clusters/k8s-cluster01"
# CA and other components cert/key Directory ca_dir = "/etc/kubernetes/ssl"
# Default 'k8s_nodename' is empty k8s_nodename = ''
# Default python interpreter ansible_python_interpreter = /usr/bin/python3
root@deploy:/etc/kubeasz# cat /etc/kubeasz/clusters/k8s-cluster01/config.yml ############################ # prepare ############################ # 可选离线安装系统软件包 (offline|online) INSTALL_SOURCE: "online"
# 可选进行系统安全加固 github.com/dev-sec/ansible-collection-hardening OS_HARDEN: false
############################ # role:deploy ############################ # default: ca will expire in 100 years # default: certs issued by the ca will expire in 50 years CA_EXPIRY: "876000h" CERT_EXPIRY: "438000h"
# force to recreate CA and other certs, not suggested to set 'true' CHANGE_CA: false
# kubeconfig 配置参数 CLUSTER_NAME: "cluster1" CONTEXT_NAME: "context-{{ CLUSTER_NAME }}"
# k8s version K8S_VER: "1.28.1" root@deploy:/etc/kubeasz# cat ./playbooks/01.prepare.yml # [optional] to synchronize system time of nodes with 'chrony' - hosts: - kube_master - kube_node - etcd # - ex_lb # - chrony roles: - { role: os-harden, when: "OS_HARDEN|bool" } - { role: chrony, when: "groups['chrony']|length > 0" }
# to create CA, kubeconfig, kube-proxy.kubeconfig etc. - hosts: localhost roles: - deploy
# prepare tasks for all nodes - hosts: - kube_master - kube_node - etcd roles: - prepare 注释掉上述ex_lb和chrony表示这两个主机我们自己定义，不需要通过kubeasz来帮我们初始化；即系统初始化，只针对master、node、etcd这三类节点来做
root@deploy:/etc/kubeasz# ./ezctl setup k8s-cluster01 01 ...... PLAY RECAP *************************************************************************************************************************************************************************** 192.168.20.235 : ok=27 changed=21 unreachable=0 failed=0 skipped=115 rescued=0 ignored=0 192.168.20.236 : ok=24 changed=18 unreachable=0 failed=0 skipped=111 rescued=0 ignored=0 192.168.20.238 : ok=24 changed=18 unreachable=0 failed=0 skipped=111 rescued=0 ignored=0 192.168.20.239 : ok=24 changed=19 unreachable=0 failed=0 skipped=111 rescued=0 ignored=0 192.168.20.241 : ok=23 changed=17 unreachable=0 failed=0 skipped=112 rescued=0 ignored=0 192.168.20.242 : ok=23 changed=17 unreachable=0 failed=0 skipped=112 rescued=0 ignored=0 192.168.20.243 : ok=23 changed=17 unreachable=0 failed=0 skipped=112 rescued=0 ignored=0 localhost : ok=33 changed=31 unreachable=0 failed=0 skipped=11 rescued=0 ignored=0
root@deploy:/etc/kubeasz# ./ezctl setup k8s-cluster01 02 ...... PLAY RECAP ******************************************************************************************************************************************************************************************************** 192.168.20.241 : ok= 10 changed= 9 unreachable= 0 failed= 0 skipped= 0 rescued= 0 ignored= 0 192.168.20.242 : ok= 8 changed= 7 unreachable= 0 failed= 0 skipped= 0 rescued= 0 ignored= 0 192.168.20.243 : ok= 8 changed= 7 unreachable= 0 failed= 0 skipped= 0 rescued= 0 ignored= 0
验证etcd集群是否正常 export NODE_IPS = "192.168.20.241 192.168.20.242 192.168.20.243" for ip in ${ NODE_IPS }; do ETCDCTL_API = 3 /usr/local/bin/etcdctl --endpoints=https://${ ip }:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/etcd.pem --key=/etc/kubernetes/ssl/etcd-key.pem endpoint health ; done
root@k8s-etcd01:~# export NODE_IPS="192.168.20.241 192.168.20.242 192.168.20.243" root@k8s-etcd01:~# for ip in ${ NODE_IPS }; do ETCDCTL_API = 3 /usr/local/bin/etcdctl --endpoints=https://${ ip }:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/etcd.pem --key=/etc/kubernetes/ssl/etcd-key.pem endpoint health ; done https://192.168.20.241:2379 is healthy: successfully committed proposal: took = 35.398503ms https://192.168.20.242:2379 is healthy: successfully committed proposal: took = 38.225313ms https://192.168.20.243:2379 is healthy: successfully committed proposal: took = 38.436124ms
grep SANDBOX_IMAGE ./clusters/ * -R root@deploy:/etc/kubeasz# grep SANDBOX_IMAGE ./clusters/ * -R ./clusters/k8s-cluster01/config.yml:SANDBOX_IMAGE: "harbor.nbbnai.com/easzlab/pause:3.9"
下载基础镜像到本地，然后更换标签，上传至harbor之上 root@deploy:/etc/kubeasz# docker images REPOSITORY TAG IMAGE ID CREATED SIZE registry 2 ff1857193a0b 2 weeks ago 25.4MB easzlab/kubeasz-k8s-bin v1.28.1 fc8e624863e7 5 weeks ago 1.22GB easzlab/kubeasz 3.6.2 6ea48e5ea9d3 2 months ago 157MB easzlab/kubeasz-ext-bin 1.8.0 633a9737e967 2 months wget https://github.com/containerd/nerdctl/releases/download/v1.7.0/nerdctl-1.7.0-linux-amd64.tar.gz -C /etc/kubeasz/bin/containerd-bin/ root@deploy:/etc/kubeasz/bin/containerd-bin# ll total 160692 drwxr-xr-x 2 root root 4096 Nov 8 05:28 ./ drwxr-xr-x 4 root root 4096 Nov 7 14:08 ../ -rwxr-xr-x 1 root root 52375288 Aug 14 16:12 containerd * -rwxr-xr-x 1 root root 21618 Nov 4 08:49 containerd-rootless-setuptool.sh * -rwxr-xr-x 1 root root 7187 Nov 4 08:49 containerd-rootless.sh * -rwxr-xr-x 1 root@k8s-master01:~# nerdctl pull ubuntu:22.04 WARN[0000] skipping verifying HTTPS certs for "docker.io" docker.io/library/ubuntu:22.04: resolved | ++++++++++++++++++++++++++++++++++++++ | index-sha256:2b7412e6465c3c7fc5bb21d3e6f1917c167358449fecac8176c6e496e5c1f05f: done | ++++++++++++++++++++++++++++++++++++++ | manifest-sha256:c9cf959fd83770dfdefd8fb42cfef0761432af36a764c077aed54bbc5bb25368: done | ++++++++++++++++++++++++++++++++++++++ | config-sha256:e4c58958181a5925816faa528ce959e487632f4cfd192f8132f71b32df2744b4: done | ++++++++++++++++++++++++++++++++++++++ | layer-sha256:aece8493d3972efa43bfd4ee3cdba659c0f787f8f59c82fb3e48c87cbb22a12e: done | ++++++++++++++++++++++++++++++++++++++ | elapsed: 14.6s total: 28.2 M (1.9 MiB/s )
root@k8s-master01:~# nerdctl login harbor.nbbnai.com Enter Username: admin Enter Password: WARN[0007] skipping verifying HTTPS certs for "harbor.nbbnai.com" WARNING: Your password will be stored unencrypted in /root/.docker/config.json.
root@k8s-master01:~# nerdctl tag ubuntu:22.04 harbor.nbbnai.com/images/ubuntu:22.04 root@k8s-master01:~# nerdctl images REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE ubuntu 22.04 2b7412e6465c 3 minutes ago linux/amd64 83.4 MiB 28.2 MiB harbor.nbbnai.com/images/ubuntu 22.04 2b7412e6465c 9 seconds ago linux/amd64 83.4 MiB 28.2 MiB
root@k8s-master01:~# nerdctl push harbor.nbbnai.com/images/ubuntu:22.04 INFO[0000] pushing as a reduced-platform image (application/vnd.oci.image.index.v1+json, sha256:f96e25a5af70e7f2b4a92961d7ff8d751d52781755d1d22d89744d4b22fcb3f6 ) WARN[0000] skipping verifying HTTPS certs for "harbor.nbbnai.com" index-sha256:f96e25a5af70e7f2b4a92961d7ff8d751d52781755d1d22d89744d4b22fcb3f6: done | ++++++++++++++++++++++++++++++++++++++ | manifest-sha256:c9cf959fd83770dfdefd8fb42cfef0761432af36a764c077aed54bbc5bb25368: done | ++++++++++++++++++++++++++++++++++++++ | config-sha256:e4c58958181a5925816faa528ce959e487632f4cfd192f8132f71b32df2744b4: done | ++++++++++++++++++++++++++++++++++++++ | elapsed: 1.5 s total: 2.9 Ki (2.0 KiB/s )
./ezctl setup k8s-cluster01 04 ./ezctl setup k8s-cluster01 05 root@deploy:/etc/kubeasz# kubectl get nodes NAME STATUS ROLES AGE VERSION master-01 Ready,SchedulingDisabled master 3m19s v1.28.1 master-02 Ready,SchedulingDisabled master 3m19s v1.28.1 node-01 Ready node 44s v1.28.1 node-02 Ready node 44s v1.28.1
查看所需镜像 root@deploy:/etc/kubeasz# grep 'image:' roles/calico/templates/calico-v3.24.yaml.j2 image: easzlab.io.local:5000/calico/cni:{{ calico_ver }} image: easzlab.io.local:5000/calico/node:{{ calico_ver }} image: easzlab.io.local:5000/calico/node:{{ calico_ver }} image: easzlab.io.local:5000/calico/kube-controllers:{{ calico_ver }} 上传本地镜像仓库 docker tag calico/kube-controllers:v3.24.6 harbor.nbbnai.com/images/kube-controllers:v3.24.6 docker tag calico/cni:v3.24.6 harbor.nbbnai.com/images/cni:v3.24.6 docker tag calico/node:v3.24.6 harbor.nbbnai.com/images/node:v3.24.6
docker push harbor.nbbnai.com/images/kube-controllers:v3.24.6 docker push harbor.nbbnai.com/images/cni:v3.24.6 docker push harbor.nbbnai.com/images/node:v3.24.6
root@deploy:/etc/kubeasz# scp /root/.kube/config k8s-master01:/root/.kube
root@k8s-master01:~# kubectl get node NAME STATUS ROLES AGE VERSION master-01 Ready,SchedulingDisabled master 60m v1.28.1 master-02 Ready,SchedulingDisabled master 60m v1.28.1 node-01 Ready node 58m v1.28.1 node-02 Ready node 58m v1.28.1
vim /root/.kube/config apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURtakNDQW9LZ0F3SUJBZ0lVWjEwbTBSN1lyT2MzN2orVXRtRGxBVEdPME0wd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pERUxNQWtHQTFVRUJoTUNRMDR4RVRBUEJnTlZCQWdUQ0VoaGJtZGFhRzkxTVFzd0NRWURWUVFIRXdKWQpVekVNTUFvR0ExVUVDaE1EYXpoek1ROHdEUVlEVlFRTEV3WlRlWE4wWlcweEZqQVVCZ05WQkFNVERXdDFZbVZ5CmJtVjBaWE10WTJFd0lCY05Nak14TVRBNE1ERXhOREF3V2hnUE1qRXlNekV3TVRVd01URTBNREJhTUdReEN6QUoKQmdOVkJBWVRBa05PTVJFd0R3WURWUVFJRXdoSVlXNW5XbWh2ZFRFTE1Ba0dBMVVFQnhNQ1dGTXhEREFLQmdOVgpCQW9UQTJzNGN6RVBNQTBHQTFVRUN4TUdVM2x6ZEdWdE1SWXdGQVlEVlFRREV3MXJkV0psY201bGRHVnpMV05oCk1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBdnAvOC9qU2MwUHdLUHAwRm4wQ1YKNTlFbXdpQ0VpRkJnb0ljUHk0RWVRSFVwY0oyNWNxTVRDVmE5VVFUaURVZC9MTWVXOEdnZU1zeUhnVFBpN1Q3NwpYMXErMkcxNlpQZ0M3QTd4UmxyVXFIOUtDSitXcXJoVlJjQ0VQdmIzeE1DNWsrR2xWSXZnd3ArdHdJeXhzeFFVCmhTM2pWaVNZTEN4SDExR1dreGdnM1QwMmkxQnRyTytMcHFFdStnY0swK0pSaUFwU0ZUR3FLZlVWQXNIVVRLTk4Kak82cHUzL09vOTVVNnFlM1ZoMVNGbXlzdEJmM2dCbXJlOHF3Z2hLUnVDUmo3clcyY0ZGSEJtTGdEVzZVS1EvaQowc2oyZ09aQXBFbXZEckhnWWRDZjdPR2pnckNrVTY2MW12WDZEakoyYXA1cW0vRFNLQzk2WkQyVlg1VU1zSitFCjhRSURBUUFCbzBJd1FEQU9CZ05WSFE4QkFmOEVCQU1DQVFZd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlYKSFE0RUZnUVVVdlgzbDFRejJKU2M4cTJzZ1I4c2xubmkzZjR3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUJYdApkYmZOOUtzNEplSlhxNmJSSG9DUjY4cmREQUc0RW5mdDN2STJhOHZyZ3V3bXpUYmxyL2RmL3lWdjZ1ZE4wK0MyCm9vRDdPb0dlM2RRMVdSNVVzZlYvcHRRVnJwRVEyOHJlS1lva1JaZFh0a1M3dFgwN0xvTmM0VlpqMENMeklSWnYKcmh3MHZlSnAyZnNBZ08yUDVoY1lQR3g2bk1PN3o4b3A4WlF5a3FnNlVreFhLRjhjOXBraUpJYmQxQUJ5UE9zMAo2RGR3VG9vNkVzQldCNHFyMy9zUEdjQVd4RGxxMWJJbHFDSDl0T2JsMHdBaVJHOVc0MTFyMkNicXY4M1BibmJ6Ck5xZ2FHdXg1K3hYLzA0dzhJVVV6ZTRhVmNKTEVrUkxucWhRcksydTYrdzlIVjlJTTNRTU81V1UwTEdQQU5paU4Kc2ZOcjdRcldxUjhjQzk2dllhZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= server: https://192.168.20.248:6443 修改为负载均衡地址
#添加master #k8s_nodename="master-03" 可以加可以不加，不加就会默认是ip 10.0.0.208 ./ezctl add-master k8s-cluster01 192.168.20.237 k8s_nodename="master-03" #删除master ./ezctl del-master k8s-cluster01 192.168.20.237
#添加node ./ezctl add-node k8s-cluster01 192.168.20.240 k8s_nodename="node-03" #删除node ./ezctl del-node k8s-cluster01 192.168.20.240
root@deploy:/etc/kubeasz# kubectl get nodes NAME STATUS ROLES AGE VERSION master-01 Ready,SchedulingDisabled master 84m v1.28.1 master-02 Ready,SchedulingDisabled master 84m v1.28.1 master-03 Ready,SchedulingDisabled master 6m22s v1.28.1 node-01 Ready node 82m v1.28.1 node-02 Ready node 82m v1.28.1 node-03 Ready node 72s v1.28.1
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.28.md#downloads-for-v1282
kubernetes-client-linux-amd64.tar.gz kubernetes-node-linux-amd64.tar.gz kubernetes-server-linux-amd64.tar.gz kubernetes.tar.gz
root@deploy:/usr/local/src# ll total 525380 drwxr-xr-x 2 root root 4096 Nov 8 07:51 ./ drwxr-xr-x 10 root root 4096 Apr 21 2022 ../ -rw-r--r-- 1 root root 30097845 Nov 8 07:42 kubernetes-client-linux-amd64.tar.gz -rw-r--r-- 1 root root 131824978 Nov 8 07:43 kubernetes-node-linux-amd64.tar.gz -rw-r--r-- 1 root root 375513168 Nov 8 07:44 kubernetes-server-linux-amd64.tar.gz -rw-r--r-- 1 root root 529851 Nov 8 07:41 kubernetes.tar.gz
tar xvf kubernetes-server-linux-amd64.tar.gz tar xvf kubernetes.tar.gz tar xvf kubernetes-node-linux-amd64.tar.gz tar xvf kubernetes-client-linux-amd64.tar.gz
cd kubernetes/server/bin cp kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy kubectl /etc/kubeasz/bin/ /etc/kubeasz/bin/kube-apiserver --version Kubernetes v1.28.2
cd /etc/kubeasz/ root@deploy:/etc/kubeasz# ./ezctl upgrade k8s-cluster01
root@deploy:/etc/kubeasz# kubectl get nodes NAME STATUS ROLES AGE VERSION master-01 Ready,SchedulingDisabled master 131m v1.28.2 master-02 Ready,SchedulingDisabled master 131m v1.28.2 master-03 Ready,SchedulingDisabled master 52m v1.28.2 node-01 Ready node 128m v1.28.2 node-02 Ready node 128m v1.28.2 node-03 Ready node 47m v1.28.2 提示：上面我们使用的是kubeasz来更行k8s集群master和node；我们也可以手动更新； 手动更新 方法1：将⼆进制⽂件同步到其它路径，修改service⽂件加载新版本⼆进制，即⽤新版本替换旧版本； 方法2：关闭源服务、替换⼆进制⽂件然后启动服务，即直接替换旧版本；
root@deploy:~# cat coredns-v1.9.4.yaml # __MACHINE_GENERATED_WARNING__ apiVersion : v1 kind : ServiceAccount metadata : name : coredns namespace : kube-system labels : kubernetes.io/cluster-service : "true" addonmanager.kubernetes.io/mode : Reconcile --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : labels : kubernetes.io/bootstrapping : rbac-defaults addonmanager.kubernetes.io/mode : Reconcile name : system:coredns rules : wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml root@k8s-master01:~# mv recommended.yaml dashboard-v2.7.0.yaml root@k8s-master01:~# cat dashboard-v2.7.0.yaml | grep image: image: kubernetesui/dashboard:v2.7.0 image: kubernetesui/metrics-scraper:v1.0.8 root@k8s-master01:~# nerdctl pull kubernetesui/dashboard:v2.7.0 WARN[0000] skipping verifying HTTPS certs for "docker.io" docker.io/kubernetesui/dashboard:v2.7.0: resolved | ++++++++++++++++++++++++++++++++++++++ | index-sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93: done | ++++++++++++++++++++++++++++++++++++++ | manifest-sha256:ca93706ef4e400542202d620b8094a7e4e568ca9b1869c71b053cdf8b5dc3029: done | ++++++++++++++++++++++++++++++++++++++ | config-sha256:07655ddf2eebe5d250f7a72c25f638b27126805d61779741b4e62e69ba080558: root@harbor:~# apt install nfs-server -y root@harbor:~# mkdir -p /data/k8sdata/kuboard root@harbor:~# cat /etc/exports # /etc/exports: the access control list for filesystems which may be exported # to NFS clients. See exports(5). # # Example for NFSv2 and NFSv3: # /srv/homes hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check) # # Example for NFSv4: # /srv/nfs4 gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check) # /srv/nfs4/homes gss/krb5i(rw,sync,no_subtree_check) # /data/k8sdata/kuboard *(rw,no_root_squash) root@harbor:~# systemctl restart nfs-server root@harbor:~# systemctl enable nfs-server root@harbor:~#
root@k8s-master01:~# cat dashboard-v2.7.0.yaml # Copyright 2017 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 root@k8s-master01:~/KubeSphere/1.nfs-stroageclass-cases# cat 1-rbac.yaml apiVersion : v1 kind : Namespace metadata : name : nfs --- apiVersion : v1 kind : ServiceAccount metadata : name : nfs-client-provisioner # replace with namespace where provisioner is deployed namespace : nfs --- kind : ClusterRole apiVersion : rbac.authorization.k8s.io/v1 metadata : name : nfs-client-provisioner-runner rules : - apiGroups : [ "" ] resources : [ "nodes" #设置kubectl命令补全；在 bash 中设置当前 shell 的自动补全，要先安装 bash-completion 包 apt install -y bash-completion kubectl completion bash > /root/kubectl-completion.sh chmod a+x /root/kubectl-completion.sh echo "source /root/kubectl-completion.sh" >> /etc/profile source /etc/profile ## kubectl命令 kubectl命令使用：详情请看官方文档：https://kubernetes.io/zh-cn/docs/reference/kubectl/cheatsheet/
``` powershell kubectl get #查看资源对象 kubectl describe #查看资源对象的详细信息 kubectl apply #创建资源对象 kubectl logs #查看日志 kubectl exec #进入容器 kubectl scale #进行容器的动态伸缩 kubectl explain #查看某些资源对象的某些字段的详细说明 （写yaml文件） kubectl label #给node或pod打标签 kubectl cluster-info #查看集群状态 kubectl top node/pod #查看node或pod的指标数据 kubectl drain #驱逐node上的pod,用于node下线等场景 kubectl taint #给node标记污点，实现反亲pod与node反亲和性 kubectl api-resources/api-versions/version #api资源 kubectl config #客户端kube-config配置 kubectl config view
kubectl cp #复制文件 kubectl cp /tmp/foo_dir my-pod:/tmp/bar_dir # 将 /tmp/foo_dir 本地目录复制到远程当前命名空间中 Pod 中的 /tmp/bar_dir kubectl cp /tmp/foo my-pod:/tmp/bar -c my-container # 将 /tmp/foo 本地文件复制到远程 Pod 中特定容器的 /tmp/bar 下 kubectl cp /tmp/foo my-namespace/my-pod:/tmp/bar # 将 /tmp/foo 本地文件复制到远程 “my-namespace” 命名空间内指定 Pod 中的 /tmp/bar kubectl cp my-namespace/my-pod:/tmp/foo /tmp/bar # 将 /tmp/foo 从远程 Pod 复制到本地 /tmp/bar
kubectl cordon #警戒线，标记node不被调度；即不会在这个node上新建pod呢 kubectl cordon node1 kubectl uncordon node1
docker-ce
|
5:24.0.1-1~ubuntu.22.04~jammy
|
https://mirrors.aliyun.com/docker-ce/linux/ubuntu
jammy/stable
amd64
Packages
docker-ce | 5:24.0.0-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:23.0.6-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:23.0.5-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:23.0.4-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:23.0.3-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:23.0.2-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:23.0.1-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:23.0.0-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.24~3-0~ubuntu-jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.23~3-0~ubuntu-jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.22~3-0~ubuntu-jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.21~3-0~ubuntu-jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.20~3-0~ubuntu-jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.19~3-0~ubuntu-jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.18~3-0~ubuntu-jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.17~3-0~ubuntu-jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.16~3-0~ubuntu-jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.15~3-0~ubuntu-jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.14~3-0~ubuntu-jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
apt install -y docker-ce
root@harbor:~# docker version
Client: Docker Engine - Community
Version: 24.0.7
API version: 1.43
Go version: go1.20.10
Git commit: afdd53b
Built: Thu Oct 26 09:07:41 2023
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 24.0.7
API version: 1.43 (minimum version 1.12 )
Go version: go1.20.10
Git commit: 311b9ff
Built: Thu Oct 26 09:07:41 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.24
GitCommit: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
runc:
Version: 1.1.9
GitCommit: v1.1.9-0-gccaecfc
docker-init:
Version: 0.19.0
GitCommit: de40ad0
offline_scan : false
security_check : vuln
insecure : false
jobservice :
max_job_workers : 10
job_loggers :
- STD_OUTPUT
- FILE
notification :
webhook_job_max_retry : 3
log :
level : info
local :
rotate_count : 50
rotate_size : 200M
location : /var/log/harbor
_version : 2.9.0
proxy :
http_proxy :
https_proxy :
no_proxy :
components :
- core
- jobservice
- trivy
upload_purging :
enabled : true
age : 168h
interval : 24h
dryrun : false
cache :
enabled : false
expire_hours : 24
./install.sh --with-trivy 安装
root@harbor:/app/harbor# ./install.sh --with-notary --with-trivy
Note : Please set hostname and other necessary attributes in harbor.yml first. DO NOT use localhost or 127.0.0.1 for hostname, because Harbor needs to be accessed by external clients.
Please set --with-trivy if needs enable Trivy in Harbor.
Please do NOT set --with-chartmuseum, as chartmusuem has been deprecated and removed.
Please do NOT set --with-notary, as notary has been deprecated and removed.
root@harbor:/app/harbor# ./install.sh --with-trivy
[ Step 0 ]: checking if docker is installed ...
Note : docker version : 24.0.7
[ Step 1 ]: checking docker-compose is installed ...
Note : Docker Compose version v2.21.0
[ Step 2 ]: loading Harbor images ...
88314a2b8588 : Loading layer [==================================================>] 40.07MB/40.07MB
38478029bae1 : Loading layer [==================================================>] 7.869MB/7.869MB
b89fb6bfd54a : Loading layer [==================================================>] 4.096kB/4.096kB
c2e56d8f648b : Loading layer [==================================================>] 17.57MB/17.57MB
922799cd1326 : Loading layer [==================================================>] 3.072kB/3.072kB
b04af0eea10a : Loading layer [==================================================>] 32.65MB/32.65MB
1a3b05cff3f2 : Loading layer [==================================================>] 51.01MB/51.01MB
Loaded image : goharbor/harbor-registryctl:v2.9.0
f40d4969c628 : Loading layer [==================================================>] 115MB/115MB
Loaded image : goharbor/nginx-photon:v2.9.0
6df8272a3071 : Loading layer [==================================================>] 7.869MB/7.869MB
4ff92b4da864 : Loading layer [==================================================>] 4.096kB/4.096kB
a798ae1c0072 : Loading layer [==================================================>] 3.072kB/3.072kB
e3903428fe12 : Loading layer [==================================================>] 17.57MB/17.57MB
99935cebaf06 : Loading layer [==================================================>] 18.36MB/18.36MB
Loaded image : goharbor/registry-photon:v2.9.0
0873522a0d7e : Loading layer [==================================================>] 89.16MB/89.16MB
ca1a2032caab : Loading layer [==================================================>] 63.21MB/63.21MB
4912126f9f08 : Loading layer [==================================================>] 23.92MB/23.92MB
2688a66f7821 : Loading layer [==================================================>] 65.54kB/65.54kB
87879432eb45 : Loading layer [==================================================>] 2.56kB/2.56kB
34b1a60a7677 : Loading layer [==================================================>] 1.536kB/1.536kB
202d78acb319 : Loading layer [==================================================>] 12.29kB/12.29kB
caedfb4a15e2 : Loading layer [==================================================>] 5.315MB/5.315MB
e3d0f9b7e79c : Loading layer [==================================================>] 428.5kB/428.5kB
Loaded image : goharbor/prepare:v2.9.0
02a9bb7f3789 : Loading layer [==================================================>] 115MB/115MB
3aef37314a75 : Loading layer [==================================================>] 6.46MB/6.46MB
644fa5732ead : Loading layer [==================================================>] 245.8kB/245.8kB
5057349879c7 : Loading layer [==================================================>] 1.233MB/1.233MB
Loaded image : goharbor/harbor-portal:v2.9.0
e28c26b320cd : Loading layer [==================================================>] 124.4MB/124.4MB
3e3483c5715d : Loading layer [==================================================>] 3.584kB/3.584kB
de046f3ff0cb : Loading layer [==================================================>] 3.072kB/3.072kB
d69cc21098fe : Loading layer [==================================================>] 2.56kB/2.56kB
e3a8545abf90 : Loading layer [==================================================>] 3.072kB/3.072kB
e7bd1489d57e : Loading layer [==================================================>] 3.584kB/3.584kB
e75e75d849a3 : Loading layer [==================================================>] 20.48kB/20.48kB
Loaded image : goharbor/harbor-log:v2.9.0
b64768d458bb : Loading layer [==================================================>] 10.88MB/10.88MB
0a0628010019 : Loading layer [==================================================>] 27.74MB/27.74MB
4594ca3e621b : Loading layer [==================================================>] 4.608kB/4.608kB
5ab1deca79df : Loading layer [==================================================>] 28.53MB/28.53MB
Loaded image : goharbor/harbor-exporter:v2.9.0
72b47093760e : Loading layer [==================================================>] 25.06MB/25.06MB
2f6efb4b6a6f : Loading layer [==================================================>] 109.6MB/109.6MB
9f8ea7e3fe6f : Loading layer [==================================================>] 3.072kB/3.072kB
74ea56f949e0 : Loading layer [==================================================>] 59.9kB/59.9kB
040146a8c4a9 : Loading layer [==================================================>] 61.95kB/61.95kB
Loaded image : goharbor/redis-photon:v2.9.0
eef0dd714498 : Loading layer [==================================================>] 9.303MB/9.303MB
d00454bf6fc3 : Loading layer [==================================================>] 4.096kB/4.096kB
19349d896266 : Loading layer [==================================================>] 3.072kB/3.072kB
f1edded61c01 : Loading layer [==================================================>] 196MB/196MB
b529682ce476 : Loading layer [==================================================>] 14.1MB/14.1MB
3e9d17a97e88 : Loading layer [==================================================>] 210.9MB/210.9MB
Loaded image : goharbor/trivy-adapter-photon:v2.9.0
f41c53ea21d8 : Loading layer [==================================================>] 10.88MB/10.88MB
97993f208d64 : Loading layer [==================================================>] 3.584kB/3.584kB
7fc3a3307611 : Loading layer [==================================================>] 2.56kB/2.56kB
c4d316593112 : Loading layer [==================================================>] 58.78MB/58.78MB
fe0a88f9d208 : Loading layer [==================================================>] 5.632kB/5.632kB
806aa6f80f8f : Loading layer [==================================================>] 122.4kB/122.4kB
b62f2e569c71 : Loading layer [==================================================>] 80.38kB/80.38kB
1897bc94e827 : Loading layer [==================================================>] 59.77MB/59.77MB
7af95b603e53 : Loading layer [==================================================>] 2.56kB/2.56kB
Loaded image : goharbor/harbor-core:v2.9.0
7fec2b128fb6 : Loading layer [==================================================>] 25.06MB/25.06MB
8cbb6f3937fe : Loading layer [==================================================>] 173.9MB/173.9MB
08caca94de5f : Loading layer [==================================================>] 25.45MB/25.45MB
0e986de68b23 : Loading layer [==================================================>] 27.96MB/27.96MB
364eef2aaddd : Loading layer [==================================================>] 5.12kB/5.12kB
990575d74a02 : Loading layer [==================================================>] 6.144kB/6.144kB
9e4c6c407185 : Loading layer [==================================================>] 3.072kB/3.072kB
5ef89ac26197 : Loading layer [==================================================>] 2.048kB/2.048kB
a7063b9c4dc4 : Loading layer [==================================================>] 2.56kB/2.56kB
9843a1e0e7de : Loading layer [==================================================>] 7.68kB/7.68kB
Loaded image : goharbor/harbor-db:v2.9.0
09779c92d53d : Loading layer [==================================================>] 10.88MB/10.88MB
453b52e507cc : Loading layer [==================================================>] 3.584kB/3.584kB
b3c65a8ecf16 : Loading layer [==================================================>] 2.56kB/2.56kB
7308e2b5a527 : Loading layer [==================================================>] 44.65MB/44.65MB
7b18c7d9db5e : Loading layer [==================================================>] 45.44MB/45.44MB
Loaded image : goharbor/harbor-jobservice:v2.9.0
[ Step 3 ]: preparing environment ...
[ Step 4 ]: preparing harbor configs ...
prepare base dir is set to /app/harbor
Generated configuration file : /config/portal/nginx.conf
Generated configuration file : /config/log/logrotate.conf
Generated configuration file : /config/log/rsyslog_docker.conf
Generated configuration file : /config/nginx/nginx.conf
Generated configuration file : /config/core/env
Generated configuration file : /config/core/app.conf
Generated configuration file : /config/registry/config.yml
Generated configuration file : /config/registryctl/env
Generated configuration file : /config/registryctl/config.yml
Generated configuration file : /config/db/env
Generated configuration file : /config/jobservice/env
Generated configuration file : /config/jobservice/config.yml
Generated and saved secret to file : /data/secret/keys/secretkey
Successfully called func : create_root_cert
Generated configuration file : /config/trivy-adapter/env
Generated configuration file : /compose_location/docker-compose.yml
Clean up the input dir
Note : stopping existing Harbor instance ...
[ Step 5 ]: starting Harbor ...
[ + ] Running 11/11
✔ Network harbor_harbor Created 0.2s
✔ Container harbor-log Started 0.1s
✔ Container redis Started 0.1s
✔ Container registryctl Started 0.1s
✔ Container registry Started 0.1s
✔ Container harbor-db Started 0.1s
✔ Container harbor-portal Started 0.2s
✔ Container trivy-adapter Started 0.1s
✔ Container harbor-core Started 0.1s
✔ Container nginx Started 0.1s
✔ Container harbor-jobservice Started
docker-ce | 5:24.0.2-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:24.0.1-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:24.0.0-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:23.0.6-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:23.0.5-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:23.0.4-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:23.0.3-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:23.0.2-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:23.0.1-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:23.0.0-1~ubuntu.22.04~jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.24~3-0~ubuntu-jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.23~3-0~ubuntu-jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.22~3-0~ubuntu-jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.21~3-0~ubuntu-jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.20~3-0~ubuntu-jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.19~3-0~ubuntu-jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.18~3-0~ubuntu-jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.17~3-0~ubuntu-jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.16~3-0~ubuntu-jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.15~3-0~ubuntu-jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.14~3-0~ubuntu-jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.13~3-0~ubuntu-jammy | https://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
apt install ansible docker-ce -y
.github/
-rw-rw-r-- 1 root root 301 Sep 4 13:35 .gitignore
-rw-rw-r-- 1 root root 6213 Sep 4 13:35 README.md
-rw-rw-r-- 1 root root 20304 Sep 4 13:35 ansible.cfg
drwxr-xr-x 4 root root 4096 Nov 7 14:08 bin/
drwxrwxr-x 8 root root 4096 Sep 4 13:54 docs/
drwxr-xr-x 3 root root 4096 Nov 7 14:15 down/
drwxrwxr-x 2 root root 4096 Sep 4 13:54 example/
-rwxrwxr-x 1 root root 26507 Sep 4 13:35 ezctl *
-rwxrwxr-x 1 root root 32735 Sep 4 13:35 ezdown *
drwxrwxr-x 10 root root 4096 Sep 4 13:54 manifests/
drwxrwxr-x 2 root root 4096 Sep 4 13:54 pics/
drwxrwxr-x 2 root root 4096 Sep 4 13:54 playbooks/
drwxrwxr-x 22 root root 4096 Sep 4 13:54 roles/
drwxrwxr-x 2 root root 4096 Sep 4 13:54 tools/
查看ezctl工具的使用帮助
root@deploy:/etc/kubeasz# ./ezctl --help
Usage: ezctl COMMAND [args]
-------------------------------------------------------------------------------------
Cluster setups:
list to list all of the managed clusters
checkout < cluste r> to switch default kubeconfig of the cluster
new < cluste r> to start a new k8s deploy with name 'cluster'
setup < cluste r> < ste p> to setup a cluster, also supporting a step-by-step way
start < cluste r> to start all of the k8s services stopped by 'ezctl stop'
stop < cluste r> to stop all of the k8s services temporarily
upgrade < cluste r> to upgrade the k8s cluster
destroy < cluste r> to destroy the k8s cluster
backup < cluste r> to backup the cluster state (etcd snapshot )
restore < cluste r> to restore the cluster state from backups
start-aio to quickly setup an all-in-one cluster with default settings
Cluster ops:
add-etcd < cluste r> < i p> to add a etcd-node to the etcd cluster
add-master < cluste r> < i p> to add a master node to the k8s cluster
add-node < cluste r> < i p> to add a work node to the k8s cluster
del-etcd < cluste r> < i p> to delete a etcd-node from the etcd cluster
del-master < cluste r> < i p> to delete a master node from the k8s cluster
del-node < cluste r> < i p> to delete a work node from the k8s cluster
Extra operation:
kca-renew < cluste r> to force renew CA certs and all the other certs (with caution )
kcfg-adm < cluste r> < arg s> to manage client kubeconfig of the k8s cluster
Use "ezctl help <command>" for more information about a given command.
用ezctl工具生成配置文件和hosts文件
root@deploy:/etc/kubeasz# ./ezctl new k8s-cluster01
2023-11-08 00:50:41 DEBUG generate custom cluster files in /etc/kubeasz/clusters/k8s-cluster01
2023-11-08 00:50:41 DEBUG set versions
2023-11-08 00:50:42 DEBUG disable registry mirrors
2023-11-08 00:50:42 DEBUG cluster k8s-cluster01: files successfully created.
2023-11-08 00:50:42 INFO next steps 1: to config '/etc/kubeasz/clusters/k8s-cluster01/hosts'
2023-11-08 00:50:42 INFO next steps 2: to config '/etc/kubeasz/clusters/k8s-cluster01/config.yml'
# set unique 'k8s_nodename' for each node, if not set(default:'') ip add will be used
# CAUTION: 'k8s_nodename' must consist of lower case alphanumeric characters, '-' or '.',
# and must start and end with an alphanumeric character (e.g. 'example.com'),
# regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'
K8S_NODENAME: "{%- if k8s_nodename != '' -%} \
{{ k8s_nodename|replace('_', '-')|lower }} \
{%- else -%} \
{{ inventory_hostname }} \
{%- endif -%}"
############################
# role:etcd
############################
# 设置不同的wal目录，可以避免磁盘io竞争，提高性能
ETCD_DATA_DIR: "/var/lib/etcd"
ETCD_WAL_DIR: ""
############################
# role:runtime [containerd,docker]
############################
# [.]启用拉取加速镜像仓库
ENABLE_MIRROR_REGISTRY: true
# [.]添加信任的私有仓库
INSECURE_REG:
- "http://easzlab.io.local:5000"
- "https://{{ HARBOR_REGISTRY }}"
# [.]基础容器镜像
SANDBOX_IMAGE: "harbor.nbbnai.com/easzlab/pause:3.9"
# [containerd]容器持久化存储目录
CONTAINERD_STORAGE_DIR: "/var/lib/containerd"
# [docker]容器存储目录
DOCKER_STORAGE_DIR: "/var/lib/docker"
# [docker]开启Restful API
DOCKER_ENABLE_REMOTE_API: false
############################
# role:kube-master
############################
# k8s 集群 master 节点证书配置，可以添加多个ip和域名（比如增加公网ip和域名）
MASTER_CERT_HOSTS:
- "192.168.20.248"
- "kubeapi.nbbnai.com"
#- "www.test.com"
# node 节点上 pod 网段掩码长度（决定每个节点最多能分配的pod ip地址）
# 如果flannel 使用 --kube-subnet-mgr 参数，那么它将读取该设置为每个节点分配pod网段
# https://github.com/coreos/flannel/issues/847
NODE_CIDR_LEN: 24
############################
# role:kube-node
############################
# Kubelet 根目录
KUBELET_ROOT_DIR: "/var/lib/kubelet"
# node节点最大pod 数
MAX_PODS: 110
# 配置为kube组件（kubelet,kube-proxy,dockerd等）预留的资源量
# 数值设置详见templates/kubelet-config.yaml.j2
KUBE_RESERVED_ENABLED: "no"
# k8s 官方不建议草率开启 system-reserved, 除非你基于长期监控，了解系统的资源占用状况；
# 并且随着系统运行时间，需要适当增加资源预留，数值设置详见templates/kubelet-config.yaml.j2
# 系统预留设置基于 4c/8g 虚机，最小化安装系统服务，如果使用高性能物理机可以适当增加预留
# 另外，集群安装时候apiserver等资源占用会短时较大，建议至少预留1g内存
SYS_RESERVED_ENABLED: "no"
############################
# role:network [flannel,calico,cilium,kube-ovn,kube-router]
############################
# ------------------------------------------- flannel
# [flannel]设置flannel 后端"host-gw","vxlan"等
FLANNEL_BACKEND: "vxlan"
DIRECT_ROUTING: false
# [flannel]
flannel_ver: "v0.22.2"
# ------------------------------------------- calico
# [calico] IPIP隧道模式可选项有: [Always, CrossSubnet, Never],跨子网可以配置为Always与CrossSubnet(公有云建议使用always比较省事，其他的话需要修改各自公有云的网络配置，具体可以参考各个公有云说明)
# 其次CrossSubnet为隧道+BGP路由混合模式可以提升网络性能，同子网配置为Never即可.
CALICO_IPV4POOL_IPIP: "Always"
# [calico]设置 calico-node使用的host IP，bgp邻居通过该地址建立，可手工指定也可以自动发现
IP_AUTODETECTION_METHOD: "can-reach={{ groups['kube_master'][0] }}"
# [calico]设置calico 网络 backend: bird, vxlan, none
CALICO_NETWORKING_BACKEND: "bird"
# [calico]设置calico 是否使用route reflectors
# 如果集群规模超过50个节点，建议启用该特性
CALICO_RR_ENABLED: false
# CALICO_RR_NODES 配置route reflectors的节点，如果未设置默认使用集群master节点
# CALICO_RR_NODES: ["192.168.1.1", "192.168.1.2"]
CALICO_RR_NODES: []
# [calico]更新支持calico 版本: ["3.19", "3.23"]
calico_ver: "v3.24.6"
# [calico]calico 主版本
calico_ver_main: "{{ calico_ver.split('.')[0] }}.{{ calico_ver.split('.')[1] }}"
# ------------------------------------------- cilium
# [cilium]镜像版本
cilium_ver: "1.13.6"
cilium_connectivity_check: true
cilium_hubble_enabled: false
cilium_hubble_ui_enabled: false
# ------------------------------------------- kube-ovn
# [kube-ovn]离线镜像tar包
kube_ovn_ver: "v1.11.5"
# ------------------------------------------- kube-router
# [kube-router]公有云上存在限制，一般需要始终开启 ipinip；自有环境可以设置为 "subnet"
OVERLAY_TYPE: "full"
# [kube-router]NetworkPolicy 支持开关
FIREWALL_ENABLE: true
# [kube-router]kube-router 镜像版本
kube_router_ver: "v1.5.4"
############################
# role:cluster-addon
############################
# coredns 自动安装
dns_install: "yes"
corednsVer: "1.11.1"
ENABLE_LOCAL_DNS_CACHE: true
dnsNodeCacheVer: "1.22.23"
# 设置 local dns cache 地址
LOCAL_DNS_CACHE: "169.254.20.10"
# metric server 自动安装
metricsserver_install: "yes"
metricsVer: "v0.6.4"
# dashboard 自动安装
dashboard_install: "yes"
dashboardVer: "v2.7.0"
dashboardMetricsScraperVer: "v1.0.8"
# prometheus 自动安装
prom_install: "no"
prom_namespace: "monitor"
prom_chart_ver: "45.23.0"
# kubeapps 自动安装，如果选择安装，默认同时安装local-storage（提供storageClass: "local-path"）
kubeapps_install: "no"
kubeapps_install_namespace: "kubeapps"
kubeapps_working_namespace: "default"
kubeapps_storage_class: "local-path"
kubeapps_chart_ver: "12.4.3"
# local-storage (local-path-provisioner) 自动安装
local_path_provisioner_install: "no"
local_path_provisioner_ver: "v0.0.24"
# 设置默认本地存储路径
local_path_provisioner_dir: "/opt/local-path-provisioner"
# nfs-provisioner 自动安装
nfs_provisioner_install: "no"
nfs_provisioner_namespace: "kube-system"
nfs_provisioner_ver: "v4.0.2"
nfs_storage_class: "managed-nfs-storage"
nfs_server: "192.168.1.10"
nfs_path: "/data/nfs"
# network-check 自动安装
network_check_enabled: false
network_check_schedule: "*/5 * * * *"
############################
# role:harbor
############################
# harbor version，完整版本号
HARBOR_VER: "v2.6.4"
HARBOR_DOMAIN: "harbor.easzlab.io.local"
HARBOR_PATH: /var/data
HARBOR_TLS_PORT: 8443
HARBOR_REGISTRY: "{{ HARBOR_DOMAIN }}:{{ HARBOR_TLS_PORT }}"
# if set 'false', you need to put certs named harbor.pem and harbor-key.pem in directory 'down'
HARBOR_SELF_SIGNED_CERT: true
# install extra component
HARBOR_WITH_NOTARY: false
HARBOR_WITH_TRIVY: false
HARBOR_WITH_CHARTMUSEUM: true
上述配置文件主要定义了CA和证书的过期时长、kubeconfig配置参数、k8s集群版本、etcd数据存放目录、运行时参数、masster证书名称、node节点pod网段子网掩码长度、kubelet根目录、node节点最大pod数量、网络插件相关参数配置以及集群插件安装相关配置
# coredns 自动安装
dns_install: "yes"
corednsVer: "1.11.1"
ENABLE_LOCAL_DNS_CACHE: true
dnsNodeCacheVer: "1.22.23"
# 设置 local dns cache 地址
LOCAL_DNS_CACHE: "169.254.20.10"
提示：这里需要注意一点，虽然我们没有自动安装coredns，但是这两个变量需要设置下，如果ENABLE_LOCAL_DNS_CACHE的值是true，下面的LOCAL_DNS_CACHE就写成对应coredns服务的IP地址；如果ENABLE_LOCAL_DNS_CACHE的值是false，后面的LOCAL_DNS_CACHE是谁的IP地址就无所谓了
ago
636MB
coredns/coredns 1.11.1 cbb01a7bd410 2 months ago 59.8MB
easzlab.io.local:5000/coredns/coredns 1.11.1 cbb01a7bd410 2 months ago 59.8MB
easzlab/metrics-server v0.6.4 c1623971df95 3 months ago 68.9MB
easzlab.io.local:5000/easzlab/metrics-server v0.6.4 c1623971df95 3 months ago 68.9MB
easzlab/k8s-dns-node-cache 1.22.23 81d6450452ae 4 months ago 68.4MB
easzlab.io.local:5000/easzlab/k8s-dns-node-cache 1.22.23 81d6450452ae 4 months ago 68.4MB
calico/kube-controllers v3.24.6 baf4466ddf40 5 months ago 77.5MB
easzlab.io.local:5000/calico/kube-controllers v3.24.6 baf4466ddf40 5 months ago 77.5MB
easzlab.io.local:5000/calico/cni v3.24.6 ca9fea5e07cb 5 months ago 212MB
calico/cni v3.24.6 ca9fea5e07cb 5 months ago 212MB
easzlab.io.local:5000/calico/node v3.24.6 3953a481aa9d 5 months ago 245MB
calico/node v3.24.6 3953a481aa9d 5 months ago 245MB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause 3.9 e6f181688397 13 months ago 744kB
easzlab/pause 3.9 78d53e70b442 13 months ago 744kB
easzlab.io.local:5000/easzlab/pause 3.9 78d53e70b442 13 months ago 744kB
kubernetesui/dashboard v2.7.0 07655ddf2eeb 13 months ago 246MB
easzlab.io.local:5000/kubernetesui/dashboard v2.7.0 07655ddf2eeb 13 months ago 246MB
kubernetesui/metrics-scraper v1.0.8 115053965e86 17 months ago 43.8MB
easzlab.io.local:5000/kubernetesui/metrics-scraper v1.0.8 115053965e86 17 months ago 43.8MB
初始化时候已经下载
docker tag easzlab.io.local:5000/easzlab/pause:3.9 harbor.nbbnai.com/easzlab/pause:3.9
docker login harbor.nbbnai.com
docker push harbor.nbbnai.com/easzlab/pause:3.9
root
root
7356416
Aug
14
16:12
containerd-shim
*
-rwxr-xr-x 1 root root 9478144 Aug 14 16:12 containerd-shim-runc-v1 *
-rwxr-xr-x 1 root root 9510912 Aug 14 16:12 containerd-shim-runc-v2 *
-rwxr-xr-x 1 root root 23133752 Aug 14 16:12 containerd-stress *
-rwxr-xr-x 1 root root 27193112 Aug 14 16:12 ctr *
-rwxr-xr-x 1 root root 24772608 Nov 4 08:49 nerdctl *
-rwxr-xr-x 1 root root 10684992 Aug 11 01:57 runc *
#自己创建nerdctl的配置文件
root@deploy:/etc/kubeasz# cat /etc/kubeasz/roles/containerd/templates/nerdctl.toml.j2
namespace = "k8s.io"
debug = false
debug_full = false
insecure_registry = true
root@deploy:/etc/kubeasz# cat /etc/kubeasz/roles/containerd/tasks/main.yml
- name: 获取是否已经安装containerd
shell: 'systemctl is-active containerd || echo "NoFound"'
register: containerd_svc
- block:
- name: 准备containerd相关目录
file: name={{ item }} state=directory
with_items:
- "{{ bin_dir }}"
- "/etc/containerd"
- "/etc/nerdctl"
- name: 创建 nerdctl 配置文件
template: src=nerdctl.toml.j2 dest=/etc/nerdctl/nerdctl.toml #分发nerdctl配置文件
tags: upgrade
- name: 加载内核模块 overlay
modprobe: name=overlay state=present
- name: 下载 containerd 二进制文件
copy: src={{ item }} dest={{ bin_dir }}/containerd-bin/ mode= 0755
with_fileglob:
- "{{ base_dir }}/bin/containerd-bin/*"
tags: upgrade
- name: 下载 crictl
copy: src={{ base_dir }}/bin/crictl dest={{ bin_dir }}/crictl mode= 0755
- name: 添加 crictl 自动补全
lineinfile:
dest: ~/.bashrc
state: present
regexp: 'crictl completion'
line: 'source <(crictl completion bash) # generated by kubeasz'
- name: 创建 containerd 配置文件
template: src=config.toml.j2 dest=/etc/containerd/config.toml
tags: upgrade
- name: 创建systemd unit文件
template: src=containerd.service.j2 dest=/etc/systemd/system/containerd.service
tags: upgrade
- name: 创建 crictl 配置
template: src=crictl.yaml.j2 dest=/etc/crictl.yaml
- name: 开机启用 containerd 服务
shell: systemctl enable containerd
ignore_errors: true
- name: 开启 containerd 服务
shell: systemctl daemon-reload && systemctl restart containerd
tags: upgrade
- name: 轮询等待containerd服务运行
shell: "systemctl is-active containerd.service"
register: containerd_status
until: '"active" in containerd_status.stdout'
retries: 8
delay: 2
tags: upgrade
when: "'NoFound' in containerd_svc.stdout"
注意：containerd 服务配置文件目录和二进制containerd文件的目录
在node节点或master节点验证containerd的版本信息，以及nerdctl的版本信息
root@k8s-node01:~# containerd -v
containerd github.com/containerd/containerd v1.6.23 b69f1ad231b6d87eeb30504398075a92d615e83e
root@k8s-node01:~# nerdctl -v
nerdctl version 1.7.0
root@k8s-node01:~#
root@deploy:/etc/kubeasz# docker images | grep harbor.nbbnai.com
harbor.nbbnai.com/images/kube-controllers v3.24.6 baf4466ddf40 5 months ago 77.5MB
harbor.nbbnai.com/images/cni v3.24.6 ca9fea5e07cb 5 months ago 212MB
harbor.nbbnai.com/images/node v3.24.6 3953a481aa9d 5 months ago 245MB
harbor.nbbnai.com/easzlab/pause 3.9 78d53e70b442 13 months ago 744kB
修改配置yaml文件中的镜像地址为本地harbor仓库地址
root@deploy:/etc/kubeasz# grep "image:" roles/calico/templates/calico-v3.24.yaml.j2
image: harbor.nbbnai.com/images/cni:v3.24.6
image: harbor.nbbnai.com/images/node:v3.24.6
image: harbor.nbbnai.com/images/node:v3.24.6
image: harbor.nbbnai.com/images/kube-controllers:v3.24.6
./ezctl setup k8s-cluster01 06
root@deploy:/etc/kubeasz# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-6f8ccc88cc-6zzxt 1/1 Running 0 13m
kube-system calico-node-2m424 1/1 Running 0 13m
kube-system calico-node-8vtdt 1/1 Running 0 13m
kube-system calico-node-944jz 1/1 Running 0 13m
kube-system calico-node-lhm8c 1/1 Running 0 13m
root@k8s-master01:~# calicoctl node status
Calico process is running.
IPv4 BGP status
+----------------+-------------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+----------------+-------------------+-------+----------+-------------+
| 192.168.20.238 | node-to-node mesh | up | 06:41:30 | Established |
| 192.168.20.239 | node-to-node mesh | up | 06:41:27 | Established |
| 192.168.20.236 | node-to-node mesh | up | 06:41:39 | Established |
+----------------+-------------------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
root@k8s-node01:~# calicoctl node status
Calico process is running.
IPv4 BGP status
+----------------+-------------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+----------------+-------------------+-------+----------+-------------+
| 192.168.20.235 | node-to-node mesh | up | 06:41:30 | Established |
| 192.168.20.239 | node-to-node mesh | up | 06:41:29 | Established |
| 192.168.20.236 | node-to-node mesh | up | 06:41:37 | Established |
+----------------+-------------------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
root@k8s-node01:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.20.254 0.0.0.0 UG 0 0 0 ens160
10.200.32.128 192.168.20.235 255.255.255.192 UG 0 0 0 tunl0
10.200.58.192 192.168.20.239 255.255.255.192 UG 0 0 0 tunl0
10.200.85.192 0.0.0.0 255.255.255.192 U 0 0 0 *
10.200.122.128 192.168.20.236 255.255.255.192 UG 0 0 0 tunl0
192.168.20.0 0.0.0.0 255.255.255.0 U 0 0 0 ens160
创建Pod验证网络通信正常
kubectl run net-test1 --image=alpine sleep 360000 #创建pod测试夸主机网络通信是否正常
kubectl run net-test2 --image=alpine sleep 360000
kubectl run net-test3 --image=alpine sleep 360000
root@deploy:/etc/kubeasz# kubectl get pods -A -owide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default net-test1 1/1 Running 0 57s 10.200.58.193 node-02 < non e> < non e>
default net-test2 1/1 Running 0 52s 10.200.85.193 node-01 < non e> < non e>
default net-test3 1/1 Running 0 47s 10.200.58.194 node-02 < non e> < non e>
kube-system calico-kube-controllers-6f8ccc88cc-6zzxt 1/1 Running 0 18m 192.168.20.238 node-01 < non e> < non e>
kube-system calico-node-2m424 1/1 Running 0 18m 192.168.20.238 node-01 < non e> < non e>
kube-system calico-node-8vtdt 1/1 Running 0 18m 192.168.20.239 node-02 < non e> < non e>
kube-system calico-node-944jz 1/1 Running 0 18m 192.168.20.235 master-01 < non e> < non e>
kube-system calico-node-lhm8c 1/1 Running 0 18m 192.168.20.236 master-02 < non e> < non e>
root@deploy:/etc/kubeasz# kubectl exec -it net-test1 sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/ # ping 10.200.58.193
PING 10.200.58.193 (10.200.58.193): 56 data bytes
64 bytes from 10.200.58.193: seq= 0 ttl= 64 time= 0.078 ms
64 bytes from 10.200.58.193: seq= 1 ttl= 64 time= 0.134 ms
-
apiGroups
:
- ""
resources :
- endpoints
- services
- pods
- namespaces
verbs :
- list
- watch
- apiGroups :
- ""
resources :
- nodes
verbs :
- get
- apiGroups :
- discovery.k8s.io
resources :
- endpointslices
verbs :
- list
- watch
---
apiVersion : rbac.authorization.k8s.io/v1
kind : ClusterRoleBinding
metadata :
annotations :
rbac.authorization.kubernetes.io/autoupdate : "true"
labels :
kubernetes.io/bootstrapping : rbac-defaults
addonmanager.kubernetes.io/mode : EnsureExists
name : system:coredns
roleRef :
apiGroup : rbac.authorization.k8s.io
kind : ClusterRole
name : system:coredns
subjects :
- kind : ServiceAccount
name : coredns
namespace : kube-system
---
apiVersion : v1
kind : ConfigMap
metadata :
name : coredns
namespace : kube-system
labels :
addonmanager.kubernetes.io/mode : EnsureExists
data :
Corefile : |
.:53 {
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
#forward . /etc/resolv.conf {
forward . 223.6.6.6 {
max_concurrent 1000
}
cache 600
loop
reload
loadbalance
}
myserver.online {
forward . 172.16.16.16:53
}
---
apiVersion : apps/v1
kind : Deployment
metadata :
name : coredns
namespace : kube-system
labels :
k8s-app : kube-dns
kubernetes.io/cluster-service : "true"
addonmanager.kubernetes.io/mode : Reconcile
kubernetes.io/name : "CoreDNS"
spec :
# replicas: not specified here:
# 1. In order to make Addon Manager do not reconcile this replicas parameter.
# 2. Default is 1.
# 3. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
replicas : 2
strategy :
type : RollingUpdate
rollingUpdate :
maxUnavailable : 1
selector :
matchLabels :
k8s-app : kube-dns
template :
metadata :
labels :
k8s-app : kube-dns
spec :
securityContext :
seccompProfile :
type : RuntimeDefault
priorityClassName : system-cluster-critical
serviceAccountName : coredns
affinity :
podAntiAffinity :
preferredDuringSchedulingIgnoredDuringExecution :
- weight : 100
podAffinityTerm :
labelSelector :
matchExpressions :
- key : k8s-app
operator : In
values : [ "kube-dns" ]
topologyKey : kubernetes.io/hostname
tolerations :
- key : "CriticalAddonsOnly"
operator : "Exists"
nodeSelector :
kubernetes.io/os : linux
containers :
- name : coredns
image : harbor.nbbnai.com/images/coredns:1.9.4
imagePullPolicy : IfNotPresent
resources :
limits :
memory : 256Mi
cpu : 200m
requests :
cpu : 100m
memory : 70Mi
args : [ "-conf" , "/etc/coredns/Corefile" ]
volumeMounts :
- name : config-volume
mountPath : /etc/coredns
readOnly : true
ports :
- containerPort : 53
name : dns
protocol : UDP
- containerPort : 53
name : dns-tcp
protocol : TCP
- containerPort : 9153
name : metrics
protocol : TCP
livenessProbe :
httpGet :
path : /health
port : 8080
scheme : HTTP
initialDelaySeconds : 60
timeoutSeconds : 5
successThreshold : 1
failureThreshold : 5
readinessProbe :
httpGet :
path : /ready
port : 8181
scheme : HTTP
securityContext :
allowPrivilegeEscalation : false
capabilities :
add :
- NET_BIND_SERVICE
drop :
- all
readOnlyRootFilesystem : true
dnsPolicy : Default
volumes :
- name : config-volume
configMap :
name : coredns
items :
- key : Corefile
path : Corefile
---
apiVersion : v1
kind : Service
metadata :
name : kube-dns
namespace : kube-system
annotations :
prometheus.io/port : "9153"
prometheus.io/scrape : "true"
labels :
k8s-app : kube-dns
kubernetes.io/cluster-service : "true"
addonmanager.kubernetes.io/mode : Reconcile
kubernetes.io/name : "CoreDNS"
spec :
selector :
k8s-app : kube-dns
clusterIP : 10.100.0.10
ports :
- name : dns
port : 53
protocol : UDP
- name : dns-tcp
port : 53
protocol : TCP
- name : metrics
port : 9153
protocol : TCP
cat coredns-v1.9.4.yaml|grep image : #查看coredns版本
docker pull coredns/coredns:1.9.4
docker tag coredns/coredns:1.9.4 harbor.nbbnai.com/images/coredns:1.9.4
docker push harbor.nbbnai.com/images/coredns:1.9.4
root@deploy:~# cat coredns-v1.9.4.yaml|grep image :
image : harbor.nbbnai.com/images/coredns:1.9.4
root@deploy:~# kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 170m
kube-system kube-dns ClusterIP 10.100.0.10 <none> 53/UDP,53/TCP,9153/TCP 3m15s
root@deploy:~# kubectl exec -it net-test9 -- sh
/ # ping www.baidu.com
PING www.baidu.com (180.101.50.188) : 56 data bytes
64 bytes from 180.101.50.188 : seq=0 ttl=49 time=15.853 ms
/ # cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.100.0.10
--- www.baidu.com ping statistics ---
如果发现不是自己部署的coredns则修改
sed -i 's/10.100.0.10/169.254.20.10/g' /var/lib/kubelet/config.yaml
systemctl daemon-reload && systemctl restart kubelet
cat /var/lib/kubelet/config.yaml
kind : KubeletConfiguration
apiVersion : kubelet.config.k8s.io/v1beta1
address : 0.0.0.0
authentication :
anonymous :
enabled : false
webhook :
cacheTTL : 2m0s
enabled : true
x509 :
clientCAFile : /etc/kubernetes/ssl/ca.pem
authorization :
mode : Webhook
webhook :
cacheAuthorizedTTL : 5m0s
cacheUnauthorizedTTL : 30s
cgroupDriver : systemd
cgroupsPerQOS : true
clusterDNS :
- 10.100.0.10
#- 169.254.20.10
clusterDomain : cluster.local
configMapAndSecretChangeDetectionStrategy : Watch
containerLogMaxFiles : 3
containerLogMaxSize : 10Mi
enforceNodeAllocatable :
- pods
eventBurst : 10
eventRecordQPS : 5
evictionHard :
imagefs.available : 15%
memory.available : 300Mi
nodefs.available : 10%
nodefs.inodesFree : 5%
evictionPressureTransitionPeriod : 5m0s
failSwapOn : true
fileCheckFrequency : 40s
hairpinMode : hairpin-veth
healthzBindAddress : 0.0.0.0
healthzPort : 10248
httpCheckFrequency : 40s
imageGCHighThresholdPercent : 85
imageGCLowThresholdPercent : 80
imageMinimumGCAge : 2m0s
kubeAPIBurst : 100
kubeAPIQPS : 50
makeIPTablesUtilChains : true
maxOpenFiles : 1000000
maxParallelImagePulls : 5
maxPods : 110
nodeLeaseDurationSeconds : 40
nodeStatusReportFrequency : 1m0s
nodeStatusUpdateFrequency : 10s
oomScoreAdj : -999
podPidsLimit : 1024
port : 10250
# disable readOnlyPort
readOnlyPort : 0
resolvConf : /run/systemd/resolve/resolv.conf
runtimeRequestTimeout : 2m0s
serializeImagePulls : false
streamingConnectionIdleTimeout : 4h0m0s
syncFrequency : 1m0s
tlsCertFile : /etc/kubernetes/ssl/kubelet.pem
tlsPrivateKeyFile : /etc/kubernetes/ssl/kubelet-key.pem
done
|
++++++++++++++++++++++++++++++++++++++
|
layer-sha256:ee3247c7e545df975ba3826979c7a8d73f1373cbb3ac47def3b734631cef2965: done | ++++++++++++++++++++++++++++++++++++++ |
layer-sha256:8e052fd7e2d0aec4ef51e4505d006158414775ad5f0ea3e479ac0ba92f90dfff: done | ++++++++++++++++++++++++++++++++++++++ |
elapsed: 28.9s total: 72.3 M (2.5 MiB/s )
root@k8s-master01:~# nerdctl pull kubernetesui/metrics-scraper:v1.0.8
WARN[0000] skipping verifying HTTPS certs for "docker.io"
docker.io/kubernetesui/metrics-scraper:v1.0.8: resolved | ++++++++++++++++++++++++++++++++++++++ |
index-sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c: done | ++++++++++++++++++++++++++++++++++++++ |
manifest-sha256:43227e8286fd379ee0415a5e2156a9439c4056807e3caa38e1dd413b0644807a: done | ++++++++++++++++++++++++++++++++++++++ |
config-sha256:115053965e86b2df4d78af78d7951b8644839d20a03820c6df59a261103315f7: done | ++++++++++++++++++++++++++++++++++++++ |
layer-sha256:5866d2c04d960790300cbd8b18d67be6b930870d044dd75849c8c96191fe7580: done | ++++++++++++++++++++++++++++++++++++++ |
layer-sha256:978be80e3ee3098e11be2b18322822513d692988440ec1e74620e8539b07704d: done | ++++++++++++++++++++++++++++++++++++++ |
elapsed: 11.2s total: 18.8 M (1.7 MiB/s )
root@k8s-master01:~#
root@k8s-master01:~# nerdctl tag kubernetesui/dashboard:v2.7.0 harbor.nbbnai.com/images/dashboard:v2.7.0
root@k8s-master01:~# nerdctl tag kubernetesui/metrics-scraper:v1.0.8 harbor.nbbnai.com/images/metrics-scraper:v1.0.8
root@k8s-master01:~# nerdctl push harbor.nbbnai.com/images/dashboard:v2.7.0
INFO[0000] pushing as a reduced-platform image (application/vnd.docker.distribution.manifest.list.v2+json, sha256:e7b964bcdfcc9dee6b286e58a5ad2d599e837996e8975f7deb96f3707f06ede1 )
WARN[0000] skipping verifying HTTPS certs for "harbor.nbbnai.com"
index-sha256:e7b964bcdfcc9dee6b286e58a5ad2d599e837996e8975f7deb96f3707f06ede1: done | ++++++++++++++++++++++++++++++++++++++ |
manifest-sha256:ca93706ef4e400542202d620b8094a7e4e568ca9b1869c71b053cdf8b5dc3029: done | ++++++++++++++++++++++++++++++++++++++ |
config-sha256:07655ddf2eebe5d250f7a72c25f638b27126805d61779741b4e62e69ba080558: done | ++++++++++++++++++++++++++++++++++++++ |
elapsed: 3.6 s total: 2.5 Ki (724.0 B/s )
root@k8s-master01:~# nerdctl push harbor.nbbnai.com/images/metrics-scraper:v1.0.8
INFO[0000] pushing as a reduced-platform image (application/vnd.docker.distribution.manifest.list.v2+json, sha256:9fdef455b4f9a8ee315a0aa3bd71787cfd929e759da3b4d7e65aaa56510d747b )
WARN[0000] skipping verifying HTTPS certs for "harbor.nbbnai.com"
index-sha256:9fdef455b4f9a8ee315a0aa3bd71787cfd929e759da3b4d7e65aaa56510d747b: done | ++++++++++++++++++++++++++++++++++++++ |
manifest-sha256:43227e8286fd379ee0415a5e2156a9439c4056807e3caa38e1dd413b0644807a: done | ++++++++++++++++++++++++++++++++++++++ |
config-sha256:115053965e86b2df4d78af78d7951b8644839d20a03820c6df59a261103315f7: done | ++++++++++++++++++++++++++++++++++++++ |
elapsed: 1.3 s total: 2.2 Ki (1.7 KiB/s )
#更改镜像地址
root@k8s-master01:~# grep "image:" dashboard-v2.7.0.yaml
image: harbor.nbbnai.com/images/dashboard:v2.7.0
image: harbor.nbbnai.com/images/metrics-scraper:v1.0.8
root@k8s-master01:~# kubectl apply -f dashboard-v2.7.0.yaml
root@k8s-master01:~# vim dashboard-v2.7.0.yaml
root@k8s-master01:~# kubectl get svc -A -owide
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE SELECTOR
default kubernetes ClusterIP 10.100.0.1 < non e> 443/TCP 19h < non e>
kube-system kube-dns ClusterIP 10.100.0.10 < non e> 53/UDP,53/TCP,9153/TCP 16h k8s-app=kube-dns
kubernetes-dashboard dashboard-metrics-scraper ClusterIP 10.100.239.149 < non e> 8000/TCP 2m17s k8s-app=dashboard-metrics-scraper
kubernetes-dashboard kubernetes-dashboard ClusterIP 10.100.184.21 < non e> 443/TCP 2m17s k8s-app=kubernetes-dashboard
更改dashboard的svc为NodePort：
kubectl edit svc kubernetes-dashboard -n kubernetes-dashboard
29 port: 443
30 protocol: TCP
31 targetPort: 8443
32 selector:
33 k8s-app: kubernetes-dashboard
34 sessionAffinity: None
35 type: NodePort
36 status:
37 loadBalancer: {}
root@k8s-master01:~# kubectl get svc -A -owide
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE SELECTOR
default kubernetes ClusterIP 10.100.0.1 < non e> 443/TCP 19h < non e>
kube-system kube-dns ClusterIP 10.100.0.10 < non e> 53/UDP,53/TCP,9153/TCP 16h k8s-app=kube-dns
kubernetes-dashboard dashboard-metrics-scraper ClusterIP 10.100.239.149 < non e> 8000/TCP 3m39s k8s-app=dashboard-metrics-scraper
kubernetes-dashboard kubernetes-dashboard NodePort 10.100.184.21 < non e> 443:31449/TCP 3m39s k8s-app=kubernetes-dashboard
创建admin用户和secret密钥
root@k8s-master01:~# cat admin-user.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard
root@k8s-master01:~# cat admin-secret.yaml
apiVersion: v1
kind: Secret
type : kubernetes.io/service-account-token
metadata:
name: dashboard-admin-user
namespace: kubernetes-dashboard
annotations:
kubernetes.io/service-account.name: "admin-user"
root@k8s-master01:~# kubectl apply -f admin-user.yaml -f admin-secret.yaml
serviceaccount/admin-user created
clusterrolebinding.rbac.authorization.k8s.io/admin-user created
secret/dashboard-admin-user created
root@k8s-master01:~#
获取token
kubernetes-dashboard dashboard-admin-user kubernetes.io/service-account-token 3 16s
root@k8s-master01:~# kubectl describe secret -n kubernetes-dashboard dashboard-admin-user
Name: dashboard-admin-user
Namespace: kubernetes-dashboard
Labels: < non e>
Annotations: kubernetes.io/service-account.name: admin-user
kubernetes.io/service-account.uid: 44b2a8ae-600e-483b-83e4-65fc2cf74496
Type: kubernetes.io/service-account-token
Data
====
ca.crt: 1310 bytes
namespace: 20 bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6ImZtbC13NmNKZVdNOUZWNXhIM3luZmFIT3E2allkV1psb21aVnZWSndDS3MifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdXNlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJhZG1pbi11c2VyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNDRiMmE4YWUtNjAwZS00ODNiLTgzZTQtNjVmYzJjZjc0NDk2Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmVybmV0ZXMtZGFzaGJvYXJkOmFkbWluLXVzZXIifQ.VBZHa0X9rpCgYPOpvX0RCKTGxw04PjxwJ8QCC5rREnKVsoFMLWEO3resL1iG5IXt2oZchwcrFZQiuqi5gF2nPP7TykAD8CVwjuH2ymiqJFLPCmAeeWg6bEPCncq784vih4mKpv9lDjx6t7-e5YiE3tL2wI7KOjs_dGhDsK6Qe0IC5Y4U7c9ugsaJJqZqVhhEuGkEuDPNc4MmfOagf4K8OKewgI5-l92VttOqdm6KnzPe8OlQmF_GiZYnH2k-AICUV65uM94eJxG-t9lufucYJ8VGlFx_I5e-fT7ukaJZ18NmgEUb1ja1kwSwPlSFHAAGZW-LyzkFT7kUkRSC9GeR3w
设置token登录保持时间
root@k8s-master01:~# vim dashboard-v2.7.0.yaml
199 - --auto-generate-certificates
200 - --namespace=kubernetes-dashboard
201 - --token-ttl=3600
基于Kubeconfig⽂件登录 Kubeconfig⽂件
root@k8s-master01:~# cp /root/.kube/config /opt/kubeconfig
root@k8s-master01:~# vim /opt/kubeconfig
在kubeconfig文件中加上token信息即可；如下
root@k8s-master01:~# cat /opt/kubeconfig
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURtakNDQW9LZ0F3SUJBZ0lVWjEwbTBSN1lyT2MzN2orVXRtRGxBVEdPME0wd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pERUxNQWtHQTFVRUJoTUNRMDR4RVRBUEJnTlZCQWdUQ0VoaGJtZGFhRzkxTVFzd0NRWURWUVFIRXdKWQpVekVNTUFvR0ExVUVDaE1EYXpoek1ROHdEUVlEVlFRTEV3WlRlWE4wWlcweEZqQVVCZ05WQkFNVERXdDFZbVZ5CmJtVjBaWE10WTJFd0lCY05Nak14TVRBNE1ERXhOREF3V2hnUE1qRXlNekV3TVRVd01URTBNREJhTUdReEN6QUoKQmdOVkJBWVRBa05PTVJFd0R3WURWUVFJRXdoSVlXNW5XbWh2ZFRFTE1Ba0dBMVVFQnhNQ1dGTXhEREFLQmdOVgpCQW9UQTJzNGN6RVBNQTBHQTFVRUN4TUdVM2x6ZEdWdE1SWXdGQVlEVlFRREV3MXJkV0psY201bGRHVnpMV05oCk1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBdnAvOC9qU2MwUHdLUHAwRm4wQ1YKNTlFbXdpQ0VpRkJnb0ljUHk0RWVRSFVwY0oyNWNxTVRDVmE5VVFUaURVZC9MTWVXOEdnZU1zeUhnVFBpN1Q3NwpYMXErMkcxNlpQZ0M3QTd4UmxyVXFIOUtDSitXcXJoVlJjQ0VQdmIzeE1DNWsrR2xWSXZnd3ArdHdJeXhzeFFVCmhTM2pWaVNZTEN4SDExR1dreGdnM1QwMmkxQnRyTytMcHFFdStnY0swK0pSaUFwU0ZUR3FLZlVWQXNIVVRLTk4Kak82cHUzL09vOTVVNnFlM1ZoMVNGbXlzdEJmM2dCbXJlOHF3Z2hLUnVDUmo3clcyY0ZGSEJtTGdEVzZVS1EvaQowc2oyZ09aQXBFbXZEckhnWWRDZjdPR2pnckNrVTY2MW12WDZEakoyYXA1cW0vRFNLQzk2WkQyVlg1VU1zSitFCjhRSURBUUFCbzBJd1FEQU9CZ05WSFE4QkFmOEVCQU1DQVFZd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlYKSFE0RUZnUVVVdlgzbDFRejJKU2M4cTJzZ1I4c2xubmkzZjR3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUJYdApkYmZOOUtzNEplSlhxNmJSSG9DUjY4cmREQUc0RW5mdDN2STJhOHZyZ3V3bXpUYmxyL2RmL3lWdjZ1ZE4wK0MyCm9vRDdPb0dlM2RRMVdSNVVzZlYvcHRRVnJwRVEyOHJlS1lva1JaZFh0a1M3dFgwN0xvTmM0VlpqMENMeklSWnYKcmh3MHZlSnAyZnNBZ08yUDVoY1lQR3g2bk1PN3o4b3A4WlF5a3FnNlVreFhLRjhjOXBraUpJYmQxQUJ5UE9zMAo2RGR3VG9vNkVzQldCNHFyMy9zUEdjQVd4RGxxMWJJbHFDSDl0T2JsMHdBaVJHOVc0MTFyMkNicXY4M1BibmJ6Ck5xZ2FHdXg1K3hYLzA0dzhJVVV6ZTRhVmNKTEVrUkxucWhRcksydTYrdzlIVjlJTTNRTU81V1UwTEdQQU5paU4Kc2ZOcjdRcldxUjhjQzk2dllhZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
server: https://192.168.20.248:6443
name: cluster1
contexts:
- context:
cluster: cluster1
user: admin
name: context-cluster1
current-context: context-cluster1
kind: Config
preferences: {}
users:
- name: admin
user:
token: eyJhbGciOiJSUzI1NiIsImtpZCI6ImZtbC13NmNKZVdNOUZWNXhIM3luZmFIT3E2allkV1psb21aVnZWSndDS3MifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdXNlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJhZG1pbi11c2VyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNDRiMmE4YWUtNjAwZS00ODNiLTgzZTQtNjVmYzJjZjc0NDk2Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmVybmV0ZXMtZGFzaGJvYXJkOmFkbWluLXVzZXIifQ.VBZHa0X9rpCgYPOpvX0RCKTGxw04PjxwJ8QCC5rREnKVsoFMLWEO3resL1iG5IXt2oZchwcrFZQiuqi5gF2nPP7TykAD8CVwjuH2ymiqJFLPCmAeeWg6bEPCncq784vih4mKpv9lDjx6t7-e5YiE3tL2wI7KOjs_dGhDsK6Qe0IC5Y4U7c9ugsaJJqZqVhhEuGkEuDPNc4MmfOagf4K8OKewgI5-l92VttOqdm6KnzPe8OlQmF_GiZYnH2k-AICUV65uM94eJxG-t9lufucYJ8VGlFx_I5e-fT7ukaJZ18NmgEUb1ja1kwSwPlSFHAAGZW-LyzkFT7kUkRSC9GeR3w
client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQyakNDQXNLZ0F3SUJBZ0lVVG0xMTE4WlI2K0J1azI5UGVhdlVybDluN1lJd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pERUxNQWtHQTFVRUJoTUNRMDR4RVRBUEJnTlZCQWdUQ0VoaGJtZGFhRzkxTVFzd0NRWURWUVFIRXdKWQpVekVNTUFvR0ExVUVDaE1EYXpoek1ROHdEUVlEVlFRTEV3WlRlWE4wWlcweEZqQVVCZ05WQkFNVERXdDFZbVZ5CmJtVjBaWE10WTJFd0lCY05Nak14TVRBNE1ERXhOREF3V2hnUE1qQTNNekV3TWpZd01URTBNREJhTUdjeEN6QUoKQmdOVkJBWVRBa05PTVJFd0R3WURWUVFJRXdoSVlXNW5XbWh2ZFRFTE1Ba0dBMVVFQnhNQ1dGTXhGekFWQmdOVgpCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUTh3RFFZRFZRUUxFd1pUZVhOMFpXMHhEakFNQmdOVkJBTVRCV0ZrCmJXbHVNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXJ4YzJSM1hFcTJxQjNHVisKYVFvY0taY0xPOUY4UE93eXhyRFBSV0NnYkxXRzczRlowODNXa2tEVzBwcGhCZWFiZ21NSVB3bmR3KzIxQ3dEQgpERCtybHRlN1IyUWpvZkdNaEZiYTV4Q3BjUk5uV1lmdlhybHZhOFNjc1loamhIRGJhUkVEaGl4SzVFWXRFRlNCCmVnU2hhV1dVRkx4SFlyWTJsWGppbi9LSlROWFlxUUxyL3E5R2UrNVROeDhIT2l0RnV1SzR3SExvRm01eVIrZVEKeHY2UWd4dWlwL0ZxQVp3S25EaEZDU09wRFJYQklxdlI1ZmZydDBoOFUyaFNFV3dSdXY5MHhPQVkzQ3pQNHUveApLRVdnQWhsajBYdDdwazl1NlJxcXkzTldNd3hFWkFoMGtRcFVMTzlKT2xuMjRNU3JyVFdNaUFFdlk0UEFkRlhNCkQ0Kzlwd0lEQVFBQm8zOHdmVEFPQmdOVkhROEJBZjhFQkFNQ0JhQXdIUVlEVlIwbEJCWXdGQVlJS3dZQkJRVUgKQXdFR0NDc0dBUVVGQndNQ01Bd0dBMVVkRXdFQi93UUNNQUF3SFFZRFZSME9CQllFRk5YdklXOHVtTTFQRXlFZgo4VjNYc0tPOHpkOW5NQjhHQTFVZEl3UVlNQmFBRkZMMTk1ZFVNOWlVblBLdHJJRWZMSlo1NHQzK01BMEdDU3FHClNJYjNEUUVCQ3dVQUE0SUJBUUFyemNZUHVwOFhrNFdGUjllUEFXN2JNck1jVjJEc3NsOHhLRjh4am9zSS9hc1oKUmZsQncxdDc2SmhuM3piUE5hcTZkTDdRY0hRTWl4THA5OUtZbWZGeE1kd0c0a3ZubUtHdk5GUk5vT0l2d3lnVwprRnlOMVBseGpzNGZpYTNmYmdhTVVPZlBBVXBnaVNkZDNSWlRNazhCRjZySVJwVkI1UmZnTFNPcjUweFhtem1zCm1rTGZNUCsyZTFHR0VIWlNRN2RxbXRxR2hJY0VJQ3ROem9SNTlrS0JZMWNMeHFDWmZpTkl2UFZKdTVIdFhVN1EKTENRV01nbllXL0JxVTcyK1JSeUNzRlgxdlVPdjZRVVVJQjFCMURHa2N4RHJCRmNNT0tDRlRJUjRXeVhjbDMwZwpYdGxJTlFtYVBvbDZQajQ1ajM2MnIxTXBvYVdDNDlJUWVVSHhuSE5qCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb2dJQkFBS0NBUUVBcnhjMlIzWEVxMnFCM0dWK2FRb2NLWmNMTzlGOFBPd3l4ckRQUldDZ2JMV0c3M0ZaCjA4M1dra0RXMHBwaEJlYWJnbU1JUHduZHcrMjFDd0RCREQrcmx0ZTdSMlFqb2ZHTWhGYmE1eENwY1JObldZZnYKWHJsdmE4U2NzWWhqaEhEYmFSRURoaXhLNUVZdEVGU0JlZ1NoYVdXVUZMeEhZclkybFhqaW4vS0pUTlhZcVFMcgovcTlHZSs1VE54OEhPaXRGdXVLNHdITG9GbTV5UitlUXh2NlFneHVpcC9GcUFad0tuRGhGQ1NPcERSWEJJcXZSCjVmZnJ0MGg4VTJoU0VXd1J1djkweE9BWTNDelA0dS94S0VXZ0FobGowWHQ3cGs5dTZScXF5M05XTXd4RVpBaDAKa1FwVUxPOUpPbG4yNE1TcnJUV01pQUV2WTRQQWRGWE1ENCs5cHdJREFRQUJBb0lCQUNGM0FFcmNPWURCMUp3NQpVZFhsdHNoRU5FTjBvSmhNVk5BaER4eWxnVVFGMWdxQjBiZG55akVkNzNGRlRISUNSNHErUHQ4OWxZTlROWjBxCndic0RtRm8yVlhjcHJwM1BPeG80TjVLZVhxUmNIdzYvcW9LaVVxZDRacktQN09WdFpLSktQREF5MndXU1hCNlkKNkQ1WUVEbkN6bkRBSTZ5OFF1YUN1bDN6ejBZOTcrRHRvTjdMSDhIaFVkdFdwVElWaFhBRkxkcGVFUFlyMFRNQwpwRWl6VStsd0VLVlRtbTZwQUF3VG9IRkgxbUljeHFVUWRkTERlelloL0VaamRlVWI4L2VQLzBYVVRoY2RqbTE1ClpSdjg3aDY2Z3JQcFhQSk5QOXhBeDJsMVR6ZkVIRFJRTFR0YnJ5aEJYRUErekFOcStGTTBXV2VvNWNURGlwcmIKN3RWVUZvRUNnWUVBd0JxSjJCMmx0VXFzRzdJSkRsMkRjUEY1WjdzWUZmRFlaZnBIVWlic0xrRCtQM0xGTU9HQgpNdW1ock5tSWZzZHRvS3ZlRXlIOXBPRUxLQit0dy9FUjVYZFAvZ1FYQnpTS2xJdzZzVlFoWDlIemNia0hDbU5qCmRFN1BhSWEzdFNTRjR1U2cyanVCZ3V0S0tKNlhXRlJqdUlPUEZnemlZdHZ4QVpoY3VwUXh6Y2NDZ1lFQTZWUUkKSk9wUTdISWE4UDlBT2RhWGFNTjR6NG5LbGpnZ29JbzB0NHdDbC9QTWZvWk44eTdVMmFNNTFxcE9uUGZqaXlhcQppSjgreUo5eFVFVTZoMmxHdkR4cEQySkFEc080MkZHRlFBdXduUHN6eGVhcTg3Mi9RaVRqVWhVTzFqYU9KYy9CCjBzQkgrcW44LzR4em1rci80SVVuMGRZYUViSnNuMEUxa2MyMkVTRUNnWUI3VFUxbXd2akdtV3hJeG9oYmlSaEIKMlN4VXNWanZWRXlCZ0JoaEdZeUV6OENoVnZoTXZLRmgzV1doZldySC9SZFhocGtlN0xWU0xBUXVOODhmc2s4TApXdjZrQzFvQWpPL2FFdnZsTUJ2UDFnR00zL2N0NnNjK013dWJpc1RGWnZEc3hsU2NYbzl0U2JxcXdjaGdzS0hsCkg4UXFvaEI0NWJVV2tnWUsxKzN6cHdLQmdIVzRRTmVuSktiMnR1TUtWVFFaTVpEWXRsWTNWZS9PWjUrM3V2c3oKOTRjRXFSTFkzZkVvbXBrbmdpSHRFUGFmRnIxNUI2Y2dlUU90ZGZVaXlISFRTcTNqNkxtclAxaHpwUW4ycnp6dAo2UEI2eU5jN0Z0c2N5NmNxaDBvOWQ1dmFNSU9oTzFBQ1ZRNkJ2elNMZnpQRVRTN1EzNU05Nnk4c2p5TVdPbG5TCnFONGhBb0dBWUMvRVMzS3RvZXdFYmgwQVpDSzZyR1JJMHRoRnZ5U3JUZXp5QzVjdEpQZnlRemNzRjdRTDJXSlIKN3JQc2JPcURZL0xiWlBoQ1F0bEtYY1JlTjllWDQyNWdYaVhQSjRqVldCbnNYQ1J0U0thai9xaFlqWGpLYWNCUwpEZjJibVNSb2I5UzYxS25KbEQ3Y2VCYWJlM3ZEMFJ5SElpVU5rYTZNZFpCMFZiajdmc0E9Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
将上述文件导出即可使用kubeconfig文件来登录dashboard
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion : v1
kind : Namespace
metadata :
name : kubernetes-dashboard
---
apiVersion : v1
kind : ServiceAccount
metadata :
labels :
k8s-app : kubernetes-dashboard
name : kubernetes-dashboard
namespace : kubernetes-dashboard
---
kind : Service
apiVersion : v1
metadata :
labels :
k8s-app : kubernetes-dashboard
name : kubernetes-dashboard
namespace : kubernetes-dashboard
spec :
ports :
- port : 443
targetPort : 8443
selector :
k8s-app : kubernetes-dashboard
---
apiVersion : v1
kind : Secret
metadata :
labels :
k8s-app : kubernetes-dashboard
name : kubernetes-dashboard-certs
namespace : kubernetes-dashboard
type : Opaque
---
apiVersion : v1
kind : Secret
metadata :
labels :
k8s-app : kubernetes-dashboard
name : kubernetes-dashboard-csrf
namespace : kubernetes-dashboard
type : Opaque
data :
csrf : ""
---
apiVersion : v1
kind : Secret
metadata :
labels :
k8s-app : kubernetes-dashboard
name : kubernetes-dashboard-key-holder
namespace : kubernetes-dashboard
type : Opaque
---
kind : ConfigMap
apiVersion : v1
metadata :
labels :
k8s-app : kubernetes-dashboard
name : kubernetes-dashboard-settings
namespace : kubernetes-dashboard
---
kind : Role
apiVersion : rbac.authorization.k8s.io/v1
metadata :
labels :
k8s-app : kubernetes-dashboard
name : kubernetes-dashboard
namespace : kubernetes-dashboard
rules :
# Allow Dashboard to get, update and delete Dashboard exclusive secrets.
- apiGroups : [ "" ]
resources : [ "secrets" ]
resourceNames : [ "kubernetes-dashboard-key-holder" , "kubernetes-dashboard-certs" , "kubernetes-dashboard-csrf" ]
verbs : [ "get" , "update" , "delete" ]
# Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
- apiGroups : [ "" ]
resources : [ "configmaps" ]
resourceNames : [ "kubernetes-dashboard-settings" ]
verbs : [ "get" , "update" ]
# Allow Dashboard to get metrics.
- apiGroups : [ "" ]
resources : [ "services" ]
resourceNames : [ "heapster" , "dashboard-metrics-scraper" ]
verbs : [ "proxy" ]
- apiGroups : [ "" ]
resources : [ "services/proxy" ]
resourceNames : [ "heapster" , "http:heapster:" , "https:heapster:" , "dashboard-metrics-scraper" , "http:dashboard-metrics-scraper" ]
verbs : [ "get" ]
---
kind : ClusterRole
apiVersion : rbac.authorization.k8s.io/v1
metadata :
labels :
k8s-app : kubernetes-dashboard
name : kubernetes-dashboard
rules :
# Allow Metrics Scraper to get metrics from the Metrics server
- apiGroups : [ "metrics.k8s.io" ]
resources : [ "pods" , "nodes" ]
verbs : [ "get" , "list" , "watch" ]
部署kuboard
---
apiVersion : rbac.authorization.k8s.io/v1
kind : RoleBinding
metadata :
labels :
k8s-app : kubernetes-dashboard
name : kubernetes-dashboard
namespace : kubernetes-dashboard
roleRef :
apiGroup : rbac.authorization.k8s.io
kind : Role
name : kubernetes-dashboard
subjects :
- kind : ServiceAccount
name : kubernetes-dashboard
namespace : kubernetes-dashboard
---
apiVersion : rbac.authorization.k8s.io/v1
kind : ClusterRoleBinding
metadata :
name : kubernetes-dashboard
roleRef :
apiGroup : rbac.authorization.k8s.io
kind : ClusterRole
name : kubernetes-dashboard
subjects :
- kind : ServiceAccount
name : kubernetes-dashboard
namespace : kubernetes-dashboard
---
kind : Deployment
apiVersion : apps/v1
metadata :
labels :
k8s-app : kubernetes-dashboard
name : kubernetes-dashboard
namespace : kubernetes-dashboard
spec :
replicas : 1
revisionHistoryLimit : 10
selector :
matchLabels :
k8s-app : kubernetes-dashboard
template :
metadata :
labels :
k8s-app : kubernetes-dashboard
spec :
securityContext :
seccompProfile :
type : RuntimeDefault
containers :
- name : kubernetes-dashboard
image : harbor.nbbnai.com/images/dashboard:v2.7.0
imagePullPolicy : Always
ports :
- containerPort : 8443
protocol : TCP
args :
- --auto-generate-certificates
- --namespace=kubernetes-dashboard
- --token-ttl=3600
# Uncomment the following line to manually specify Kubernetes API server Host
# If not specified, Dashboard will attempt to auto discover the API server and connect
# to it. Uncomment only if the default does not work.
# - --apiserver-host=http://my-address:port
volumeMounts :
- name : kubernetes-dashboard-certs
mountPath : /certs
# Create on-disk volume to store exec logs
- mountPath : /tmp
name : tmp-volume
livenessProbe :
httpGet :
scheme : HTTPS
path : /
port : 8443
initialDelaySeconds : 30
timeoutSeconds : 30
securityContext :
allowPrivilegeEscalation : false
readOnlyRootFilesystem : true
runAsUser : 1001
runAsGroup : 2001
volumes :
- name : kubernetes-dashboard-certs
secret :
secretName : kubernetes-dashboard-certs
- name : tmp-volume
emptyDir : {}
serviceAccountName : kubernetes-dashboard
nodeSelector :
"kubernetes.io/os" : linux
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations :
- key : node-role.kubernetes.io/master
effect : NoSchedule
---
kind : Service
apiVersion : v1
metadata :
labels :
k8s-app : dashboard-metrics-scraper
name : dashboard-metrics-scraper
namespace : kubernetes-dashboard
spec :
ports :
- port : 8000
targetPort : 8000
selector :
k8s-app : dashboard-metrics-scraper
---
kind : Deployment
apiVersion : apps/v1
metadata :
labels :
k8s-app : dashboard-metrics-scraper
name : dashboard-metrics-scraper
namespace : kubernetes-dashboard
spec :
replicas : 1
revisionHistoryLimit : 10
selector :
matchLabels :
k8s-app : dashboard-metrics-scraper
template :
metadata :
labels :
k8s-app : dashboard-metrics-scraper
spec :
securityContext :
seccompProfile :
type : RuntimeDefault
containers :
- name : dashboard-metrics-scraper
image : harbor.nbbnai.com/images/metrics-scraper:v1.0.8
ports :
- containerPort : 8000
protocol : TCP
livenessProbe :
httpGet :
scheme : HTTP
path : /
port : 8000
initialDelaySeconds : 30
timeoutSeconds : 30
volumeMounts :
- mountPath : /tmp
name : tmp-volume
securityContext :
allowPrivilegeEscalation : false
readOnlyRootFilesystem : true
runAsUser : 1001
runAsGroup : 2001
serviceAccountName : kubernetes-dashboard
nodeSelector :
"kubernetes.io/os" : linux
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations :
- key : node-role.kubernetes.io/master
effect : NoSchedule
volumes :
- name : tmp-volume
emptyDir : {}
root@k8s-master01:~/kuboard# kubectl apply -f kuboard-all-in-one.yaml
namespace/kuboard created
deployment.apps/kuboard-v3 created
service/kuboard-v3 created
root@k8s-master01:~# kubectl get svc -A -owide
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 19h <none>
kube-system kube-dns ClusterIP 10.100.0.10 <none> 53/UDP,53/TCP,9153/TCP 17h k8s-app=kube-dns
kubernetes-dashboard dashboard-metrics-scraper ClusterIP 10.100.239.149 <none> 8000/TCP 36m k8s-app=dashboard-metrics-scraper
kubernetes-dashboard kubernetes-dashboard NodePort 10.100.184.21 <none> 443:31449/TCP 36m k8s-app=kubernetes-dashboard
kuboard kuboard-v3 NodePort 10.100.131.184 <none> 80:30080/TCP,10081:30081/TCP,10081:30081/UDP 8m58s k8s.kuboard.cn/name=kuboard-v3
root@k8s-master01:~#
]
verbs : [ "get" , "list" , "watch" ]
- apiGroups : [ "" ]
resources : [ "persistentvolumes" ]
verbs : [ "get" , "list" , "watch" , "create" , "delete" ]
- apiGroups : [ "" ]
resources : [ "persistentvolumeclaims" ]
verbs : [ "get" , "list" , "watch" , "update" ]
- apiGroups : [ "storage.k8s.io" ]
resources : [ "storageclasses" ]
verbs : [ "get" , "list" , "watch" ]
- apiGroups : [ "" ]
resources : [ "events" ]
verbs : [ "create" , "update" , "patch" ]
---
kind : ClusterRoleBinding
apiVersion : rbac.authorization.k8s.io/v1
metadata :
name : run-nfs-client-provisioner
subjects :
- kind : ServiceAccount
name : nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace : nfs
roleRef :
kind : ClusterRole
name : nfs-client-provisioner-runner
apiGroup : rbac.authorization.k8s.io
---
kind : Role
apiVersion : rbac.authorization.k8s.io/v1
metadata :
name : leader-locking-nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace : nfs
rules :
- apiGroups : [ "" ]
resources : [ "endpoints" ]
verbs : [ "get" , "list" , "watch" , "create" , "update" , "patch" ]
---
kind : RoleBinding
apiVersion : rbac.authorization.k8s.io/v1
metadata :
name : leader-locking-nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace : nfs
subjects :
- kind : ServiceAccount
name : nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace : nfs
roleRef :
kind : Role
name : leader-locking-nfs-client-provisioner
apiGroup : rbac.authorization.k8s.io
root@k8s-master01:~/KubeSphere/1.nfs-stroageclass-cases#
存储类清单
root@k8s-master01:~/KubeSphere/1.nfs-stroageclass-cases# cat 2-storageclass.yaml
apiVersion : storage.k8s.io/v1
kind : StorageClass
metadata :
#name: managed-nfs-storage
name : nfs-csi
annotations :
storageclass.kubernetes.io/is-default-class : "true"
provisioner : k8s-sigs.io/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
reclaimPolicy : Retain #PV的删除策略，默认为delete，删除PV后立即删除NFS server的数据
mountOptions :
#- vers=4.1 #containerd有部分参数异常
#- noresvport #告知NFS客户端在重新建立网络连接时，使用新的传输控制协议源端口
- noatime #访问文件时不更新文件inode中的时间戳，高并发环境可提高性能
parameters :
#mountOptions: "vers=4.1,noresvport,noatime"
archiveOnDelete : "true" #删除pod时保留pod数据，默认为false时为不保留数据
NFS提供者清单
root@k8s-master01:~/KubeSphere/1.nfs-stroageclass-cases# cat 3-nfs-provisioner.yaml
apiVersion : apps/v1
kind : Deployment
metadata :
name : nfs-client-provisioner
labels :
app : nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace : nfs
spec :
replicas : 1
strategy : #部署策略
type : Recreate
selector :
matchLabels :
app : nfs-client-provisioner
template :
metadata :
labels :
app : nfs-client-provisioner
spec :
serviceAccountName : nfs-client-provisioner
containers :
- name : nfs-client-provisioner
#image: k8s.gcr.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
image : registry.cn-qingdao.aliyuncs.com/zhangshijie/nfs-subdir-external-provisioner:v4.0.2
volumeMounts :
- name : nfs-client-root
mountPath : /persistentvolumes
env :
- name : PROVISIONER_NAME
value : k8s-sigs.io/nfs-subdir-external-provisioner
- name : NFS_SERVER
value : 192.168.20.246
- name : NFS_PATH
value : /data/volumes
volumes :
- name : nfs-client-root
nfs :
server:192.168.20.246
path : /data/volumes
在NFS上准备数据目录
root@harbor:~# mkdir -p /data/volumes
root@harbor:~# cat /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
# to NFS clients. See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4 gss/krb5i(rw,sync,fsid=0,crossmnt,应用上述授权、存储类、NFS提供者清单)
# /srv/nfs4/homes gss/krb5i(rw,sync,no_subtree_check)
#
/data/k8sdata/kuboard *(rw,no_root_squash)
/data/volumes *(rw,no_root_squash)
root@harbor:~# exportfs -av
exportfs : /etc/exports [1] : Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/kuboard".
Assuming default behaviour ('no_subtree_check').
NOTE : this default has changed since nfs-utils version 1.0.x
exportfs : /etc/exports [2] : Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/volumes".
Assuming default behaviour ('no_subtree_check').
NOTE : this default has changed since nfs-utils version 1.0.x
exporting *:/data/volumes
exporting *:/data/k8sdata/kuboard
应用上述授权、存储类、NFS提供者清单
root@k8s-master01:~/KubeSphere/1.nfs-stroageclass-cases# kubectl apply -f .
namespace/nfs created
serviceaccount/nfs-client-provisioner created
clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner created
role.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
storageclass.storage.k8s.io/nfs-csi created
deployment.apps/nfs-client-provisioner created
root@k8s-master01:~/KubeSphere/1.nfs-stroageclass-cases#
wget https://github.com/kubesphere/ks-installer/releases/download/v3.4.0/kubesphere-installer.yaml
wget https://github.com/kubesphere/ks-installer/releases/download/v3.4.0/cluster-configuration.yaml
编辑cluster-configuration.yaml，启用插件
启用logging
除了在部署前修改cluster-configuration.yaml文件来启用各种插件，也可以在部署后使用admin用户登录控制台，在平台管理，选集权管理，定制资源定义，在搜索栏里输入clusterconfiguration，点击搜索结果查看详细页面；在自定义资源中，点击kk-installer右侧编辑YAML；
应用kubesphere-installer.yaml安装清单
root@k8s-master01:~/KubeSphere# kubectl apply -f kubesphere-installer.yaml
root@k8s-master01:~/KubeSphere# kubectl apply -f cluster-configuration.yaml
root@k8s-master01:~/KubeSphere# kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f
查看kubesphere 控制台服务端口
root@k8s-master01:~/1.nfs-stroageclass-cases# kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 20h
kube-system kube-dns ClusterIP 10.100.0.10 <none> 53/UDP,53/TCP,9153/TCP 17h
kubernetes-dashboard dashboard-metrics-scraper ClusterIP 10.100.239.149 <none> 8000/TCP 73m
kubernetes-dashboard kubernetes-dashboard NodePort 10.100.184.21 <none> 443:31449/TCP 73m
kuboard kuboard-v3 NodePort 10.100.131.184 <none> 80:30080/TCP,10081:30081/TCP,10081:30081/UDP 45m
访问kubesphere 控制台
默认账户：admin
默认密码：P@88w0rd
