14.Kubernetes总结 | Evergreen Notes

1.k8s各组件的功能

k8s官方组件图

https://kubernetes.io/zh-cn/docs/reference/command-line-tools-reference/ #官方文档

1.1.kube-apiserver

Kubernetes API server 提供了k8s各类资源对象的增删改查及watch等HTTP Rest接口，这些对象包括pods、services、replicationcontrollers等，API Server为REST操作提供服务，并为集群的共享状态提供前端，所有其他组件都通过该前端进行交互。

补充

1，kube-apiserver对外提供了restful API, 完成一系列的功能, 除了通过 kubectl客户端调用, 也可以通过curl、浏览器或者编程调用。
2，REST ful API: 是REST风格的网络接口，REST描述的是在网络中client和server的一种交互形式
3，REST:是一种软件架构风格，或者说是一种规范，其强调HTTP应当以资源为中心，并且规范了URI的风格,规范了HTTP请求动作(GET/PUT/POST/DELETE/HEAD/OPTIONS)的使用,具有对应的语义 https://github.com/Arachni/arachni/wiki/REST-API  #REST API

1.1.1.kube-apiserver的功能

默认端口为6443，可通过启动参数“-- secure-port”的值来修改默认值。
默认监听IP为0.0.0.0及本机所有IP，可以通过启动参数“--bind-address”设置监听指定的内网IP。
该端口用于接收客户端、dashboard等外部HTTPS请求。
实现基于Tocken文件或客户端证书及HTTP Base的认证。
实现基于策略的账户鉴权及准入。
客户端通过API Server实现对kubernetes的API远程以实现对kubernetes内部资源的增删改查
等管理任务的分发

1.1.2.API的版本

Alpha：预览版，可能包含bug或错误，后期版本会修复且不兼容之前的版本，不建议使用。
Beta：测试版,如storage.k8s.io/v1beta1,该版本可能存在不稳定或者潜在的bug，不建议生产使用
v1/v2/vX: 稳定版，如apps/v1，经过验证的stable版本，可以生产环境使用

1.1.3.kubernetes API测试

SHELL

1.2.kube-controler-manager

Controller Manager还包括一些子控制器(副本控制器、节点控制器、命名空间控制器和服务账号控制器等)，控制器作为集群内部的管理控制中心，负责集群内的Node、Pod副本、服务端点(Endpoint)、命名空间(Namespace)，服务账号(ServiceAccount)、资源定额(ResourceQuota)的管理，当某个Node意外宕机时，Controller Manager会及时发现并执行自动化修复流程，确保集群中的pod副本始终处于预期的工作状态

1.2.1.Controller Manager监测node节点机制

controller-manager控制器每间隔5秒检杳一次节点的状态
如果controller-manager控制器没有收到自节点的心跳，则将该node节点被标记为不可达。
controller-manager将在标记为无法访问之前等待40秒
如果该node节点被标记为无法访问后5分钟还没有恢复，controller-manager会驱逐当前node节点的所有pod并在其它可用节点重建这此pod

1.2.2.Pod高可用机制

node monitor period：节点监测周期5s
node monitor grace peroiod：节点监视器宽限期40s
node eviction timeout： pod 驱逐超时时间 5m

1.2.3.补充

kube-controller-manager基于--leader-elect=true 启动参数实现多节点高可用，会自动选举leader，原理是有一把分布式锁，哪个节点先抢到锁谁就是 leader(基于hostname设置为锁的持有者)，leader 需要定期更新自己持有的锁状态，如超时未更新则会触发新的leader选举。

1.3.kube-scheduler

kube-scheduler是k8s的调度器组件，负责集群 Pod 的调度。基本原理是通过监听 kube-apiserver 获取待调度的 Pod，然后基于一系列筛选和评优算法，为待调度Pod列表的每个Pod从可用Node列表中选择一个最适合的Node，并将信息写入etcd中。node节点上的kubelet通过API Server监听到kubernetes Scheduler产生的Pod绑定信息，然后获取对应的Pod清单，下载Image，并启动容器。

1.4.kube-proxy

kube-proxy 是 Kubernetes 集群中的网络代理，负责为 Pod 提供网络代理和负载均衡服务。运行在所有节点上，它反映了 node 上 Kubernetes API 中定义的服务，并可以通过一组后端进行简单的 TCP、UDP和 SCTP 流转发或者在一组后端进行循环 TCP、UDP 和 SCTP 转发，用户必须使用 apiserver API 创建一个服务来配置代理，其实就是kube-proxy通过在主机上维护网络规则并执行连接转发来实现Kubernetes服务访问

1.4.1.Kube-Proxy 不同的版本可支持三种工作模式

UserSpace：k8s v1.1之前使用,k8s 1.2及以后就已经淘汰
IPtables : k8s 1.1版本开始支持，1.2开始为默认
IPVS: k8s 1.9引入到1.11为正式版本，需要安装ipvsadm、ipset 工具包和加载 ip_vs 内核模块

1.4.2.补充

1，IPVS 相对 IPtables 效率会更高一些，使用 IPVS 模式需要在运行 Kube-Proxy 的节点上安装 ipvsadm、ipset 工具包和加载 ip_vs 内 核模块，当 Kube-Proxy 以 IPVS 代理模式启动时，Kube-Proxy 将验证节点上是否安装了 IPVS 模块，如果未安装，则 Kube-Proxy 将回退到 IPtables 代理模式
2，使用IPVS模式，Kube-Proxy会监视Kubernetes Service对象和Endpoints，调用宿主机内核Netlink接口以相应地创建IPVS规则并定 期与Kubernetes Service对象 Endpoints对象同步IPVS规则，以确保IPVS状态与期望一致，访问服务时，流量将被重定向到其中一 个后端 Pod,IPVS使用哈希表作为底层数据结构并在内核空间中工作，这意味着IPVS可以更快地重定向流量，并且在同步代理规则时 具有更好的性能，此外，IPVS 为负载均衡算法提供了更多选项，例如：rr (轮询调度)、lc (最小连接数)、dh (目标哈希)、sh (源哈希)、 sed (最短期望延迟)、nq(不排队调度)等。

1.4.3.配置使用IPVS及指定调度算法

https://kubernetes.io/zh-cn/docs/reference/config-api/kube-proxy-config.v1alpha1/#ClientConnectionConfiguration #官方文档

YAML

1.5.kubelet

kubelet是一个通过命令行对kubernetes集群进行管理的客户端工具。运行在每个worker节点的代理组件，它会监视已分配给节点的pod，具体功能如下：

向master汇报node节点的状态信息
接受指令并在Pod中创建 docker容器
准备Pod所需的数据卷
返回pod的运行状态
在node节点执行容器健康检查

kubectl 在 $HOME/.kube 目录中查找一个名为 config 的配置文件。你可以通过设置 KUBECONFIG 环境变量或设置 --kubeconfig参数来指定其它 kubeconfig 文件

1.6.etcd

etcd是CoreOS公司开发目前是Kubernetes默认使用的key-value数据存储系统，用于保存kubernetes的所有集群数据，etcd支持分布式集群功能，生产环境使用时需要为etcd数据提供定期备份机制。

https://etcd.io/ #官网

https://github.com/etcd-io/etcd #github

1.7.DNS

DNS负责为整个集群提供DNS服务，从而实现服务之间的访问

常见的有以下几种：

coredns #1.18开始目前主要使用coredns
kube-dns #1.3到1.17之间的版本使用kube-dns
sky-dns #k8s 1.3版本之前使用skydns

1.8.dashboard

Dashboard是基于网页的Kubernetes用户界面，可以使用Dashboard获取运行在集群中的应用的概览信息，也可以创建或者修改 Kubernetes资源（如 Deployment，Job，DaemonSet 等等），也可以对Deployment实现弹性伸缩、发起滚动升级、删除 Pod 或者使用向导创建新的应用

第三方dashboard推荐的有如：Kuboard，KubeSphere等

2.etcd

2.1.etcd的集群选举机制

etcd选举简介

节点角色分为三种，集群中每个节点只能处于这三种状态中的一种，分别为：

follower：追随者
candidate：候选节点
leader：主节点

选举过程示意图

选举过程（假设有三个节点）

首次选举

后期选举

1.当一个follower节点在规定时间内未收到leader的消息时，它将转换为candidate状态，向其他节点发送投票请求(自己的termID和日志更新记录)，并等待其他节点的响应，如果该candidate的(日志更新记录最新)，则会获多数投票，它将成为新的leader。
2.新的leader将自己的termID+1并通告至其它节
3.如果旧的leader恢复了，发现已有新的leader，则加入到已有的leader中并 将自己的term ID更新为和leader一致，在同一个任期内所有节点的term ID是 一致的

2.2.etcd进阶

etcd简介

etcd是CoreOS团队于2013年6月发起的开源项目，它的目标是构建一个高可用的分布式键值(key-value)数据库。etcd内部采用 raft协议作为一致性算法，etcd基于Go语言实现
官方网站：https://etcd.io/
github地址：https://github.com/etcd-io/etcd
官方硬件推荐：https://etcd.io/docs/v3.5/op-guide/hardware/
官方文档：https://etcd.io/docs/v3.5/op-guide/maintenance/

etcd具有下面这些属性

SHELL

查看成员信息

配置优化

SHELL

--max-request-bytes=10485760 #request size limit(请求的最大字节数,默认一个key最大1.5Mib,官方推荐最大不要超出10Mib)
--quota-backend-bytes=8589934592 #storage size limit(磁盘存储空间大小限制,默认为2G,此值超过8G启动会有警告信息)

集群碎片整理

SHELL

etcd有多个不同的API访问版本，v1版本已经废弃，etcd v2 和 v3 本质上是共享同一套 raft 协议代码的两个独立的应用，接口不一样，存储不一样，数据互相隔离。也就是说如果从 Etcd v2 升级到 Etcd v3，原来v2 的数据还是只能用 v2 的接口访问，v3 的接口创建的数据也只能访问通过 v3 的接口访问。

WARNING:

Environment variable ETCDCTL_API is not set; defaults to etcdctl v2. #默认使用V2版本
Set environment variable ETCDCTL_API=3 to use v3 API or ETCDCTL_API=2 to use v2 API. #设置API版本

SHELL

验证节点心跳状态

SHELL

详细信息

查看etcd数据

SHELL

查看pod信息

SHELL

namespace信息

SHELL

查看deployment控制器信息

SHELL

查看calico组件信息

etcd增删改查

添加数据

SHELL

root@etcd1:~# ETCDCTL_API=3 /usr/local/bin/etcdctl put /name "tom"
OK

查询数据

SHELL

root@etcd1:~# ETCDCTL_API=3 /usr/local/bin/etcdctl get /name
/name
tom

改动数据，#直接覆盖就是更新数据

SHELL

root@etcd1:~# ETCDCTL_API=3 /usr/local/bin/etcdctl put /name "jack"
OK

验证改动

SHELL

root@etcd1:~# ETCDCTL_API=3 /usr/local/bin/etcdctl get /name
/name
jack

删除数据

SHELL

etcd数据watch机制

基于不断监看数据，发生变化就主动触发通知客户端，Etcd v3 的watch机制支持watch某个固定的key，也支持watch一个范围

在etcd node1上watch一个key，没有此key也可以执行watch，后期可以再创建

SHELL

etcd V3 API版本数据备份与恢复

WAL是write ahead log(预写日志)的缩写，顾名思义，也就是在执行真正的写操作之前先写一个日志，预写日志

wal: 存放预写式日志,最大的作用是记录了整个数据变化的全部历程。在etcd中，所有数据的修改在提交前，都要先写入到WAL中

V3版本备份数据

SHELL

root@k8s-etcd1:~# ETCDCTL_API=3 etcdctl snapshot save snapshot.db

V3版本恢复数据：

SHELL

root@k8s-etcd1:~# ETCDCTL_API=3 etcdctl snapshot restore snapshot.db --data-dir=/opt/etcd-testdir#将数据恢复到一个空 的目录中

自动备份数据

SHELL

集群v3版本数据备份与恢复

SHELL

在恢复数据期间API server不可用，必须在业务低峰期操作或者是在其它紧急场景

SHELL

验证恢复后的集群状态

SHELL

ETCD数据恢复流程

当etcd集群宕机数量超过集群总节点数一半以上的时候(如总数为三台宕机两台)，就会导致整合集群宕机，后期需要重新恢复数据，则恢复流程如下

单机备份

执行etcd备份数据的恢复的机器必须和原先etcd所在机器一致

SHELL

单机数据恢复

SHELL

集群模式下的数据备份与恢复

SHELL

3.K8s资源对象

kubernetes 资源对象操作命令i

Deployments | Kubernetes

3.1.Pod

3.2.Job与cronjob

3.3.RC/RS 副本控制器

3.4.Deployment 副本控制

3.5.Service

Service 示例

3.6.Volume-存储卷简介

3.7.emptyDir

3.8.hostPath

3.9.nfs共享存储

3.10.PV/PVC

PV/PVC总结

PersistentVolume参数

PersistentVolumeClaim创建参数

Volume-存储卷类型

Volume-静态存储卷示例

Volume-动态存储卷示例

存储类 | Kubernetes

3.11.Configmap

3.12.Secret

3.13.pod基于coredns进行域名解析流程

在k8s中pod进行域名解析是基于容器内的/etc/resolv.conf文件，在这个文件中的nameserver就是kube-dns的svc的cluster-ip，所以，pod中所有域名的解析，都要经过kube-dns的cluster-ip进行解析，不论是Kubernetes 内部域名还是外部的域名。Kubernetes中，域名的全称，必须是servicename.namespace.svc.cluster.local这种模式，servicename就是k8s中svc的名称

SHELL

coredns进行域名解析流程

根据/etc/resolve.conf进行解析流程，选择nameserver 10.0.0.2进行解析
将svc名称，namespace，依次带入/etc/resolve.conf 中的search域，进行DNS查找
在/etc/resolve.conf找不到，会通过forword设置的外部DNS进行查找

3.14.RC、RS及Deployment控制器的使用

ReplicationController，简称“RC”

https://kubernetes.io/zh-cn/docs/concepts/workloads/controllers/replicationcontroller/ #RC官方文档

3.14.1.RC的简介

第一代的pod副本控制器，确保在任何时候都有特定数量的 Pod 副本处于运行。前面我们创建的pod都是单个的，无法满足高可用的设计要求。对于无状态的应用，一般采用多副本机制，并将多副本实例打散部署到不同节点上，确保任何一个副本出现问题，整体服务还是可用的。

3.14.2.RC的工作机制

假设我们副本数量设定的3个，一开始就会创建3个pod实例，在运行过程中，其中一个pod由于节点宕机处于不可用，RC就会新开启一个pod，确保副本数为3。同样，如果副本数超过3个，就会删除多余的pod。因此，即使你的应用程序只需要一个pod，你也应该使用RC创建pod。RC类似于进程管理器，但是RC不是监控单个节点上的单个进程，而是监控跨多个节点的多个 pod。

RS和Deployment也大致是这个结构

3.14.3.RC创建

YAML

SHELL

删除pod，测试RC控制器能否保持pod三个副本

SHELL

能看到pod从ng-rc-4hhpm被替换成了ng-rc-2sj4k，从而保证三个副本数

3.14.4.RC的弹性伸缩

通过调整replicas的个数，实现弹性伸缩，从上面的例子，我们把3换成4，修改yaml文件

SHELL

仅修改replicas，其他不变，重新执行yaml文件

SHELL

修改replicas为2，再重新执行文件

SHELL

3.14.5.RC的删除

第一种方式：

采用kubectl delete rc 进行删除

SHELL

第二种方式

采用kubectl delete -f 进行删除

SHELL

补充：你可以删除一个 ReplicationController 而不影响它的任何 Pod，加--cascade=orphan参数

SHELL

3.14.6.RS的简介

ReplicaSet，简称“RS”

https://kubernetes.io/zh-cn/docs/concepts/workloads/controllers/replicaset/ #RS官方文档

RS是第二代的pod副本控制器，是RC的升级版，实现与RC一样的功能，RS对于选择器selector支持更多的表达模式，包括matchLabels和matchExpressions两种

3.14.7.创建RS

提示：matchLabels和matchExpressions只能2选1

YAML

SHELL

将pod从RS中隔离

可以通过改变标签来从 ReplicaSet 中移除 Pod。这种技术可以用来从服务中去除 Pod，以便进行排错、数据恢复等。以这种方式移除的 Pod 将被自动替换（假设副本的数量没有改变）

RC踢pod出集合的算法

首先选择剔除悬决（Pending，且不可调度）的各个 Pod
如果设置了 controller.kubernetes.io/pod-deletion-cost 注解，则注解值较小的优先被裁减掉
所处节点上副本个数较多的 Pod 优先于所处节点上副本较少者
如果 Pod 的创建时间不同，最近创建的 Pod 优先于早前创建的 Pod 被裁减。（当 LogarithmicScaleDown 这一特性门控被启用时，创建时间是按整数幂级来分组的）。

如果以上比较结果都相同，则随机选择。

3.15.Deployment（推荐）

https://kubernetes.io/zh-cn/docs/concepts/workloads/controllers/deployment/ #Deployment官方文档

Deployment的简介

前面讲的RC，RS虽然能解决弹性扩缩容，确保既定副本数的Pod，但是在实际工程中，扩缩容仅是其中一个场景，比如在版本发布过程中如何进行灰度，版本升级，回滚等，那这些问题需要使用Deployment控制器才能解决。K8S也推荐使用Deployment，不推荐直接使用RC或者RS。

创建Deployment

YAML

SHELL

Deployment的更新以及回滚

更新

修改Deployment文件里的image，改成nginx:1.20.1

YAML

SHELL

1，可以看到Deployment是先把新的pod起来了再下线旧的pod的，同时还保留着旧的pod

2，目前起来的是nginx-deployment-6b7fcc8b58的rs，nginx-deployment-577df84c7c处于下线状态

SHELL

此时可以看到nginx-deployment-577df84c7c处于上线状态，而nginx-deployment-6b7fcc8b58则下线

更新以及回滚的工作模式

SHELL

我们并没有在yaml文件中定义RS，创建deployment后RS也创建；，实际上deployment是通过RS来操控pod的。

deployment的结构图

3.16.总结nodeport类型的service访问流程

1，NodePort类型是建立在ClusterIP类型之上的，其在每个node节点的IP地址的某静态端口（NodePort）暴露服务，因此，它依然会为service分配集群IP地址，并将此作为NodePort的路由目标
2，NodePort类型就是在工作节点的IP地址上选择一个端口用于将集群外部的用户请求转发至目标Service的ClusterIP和Port，因此，NodePort类型既可如Cluster一样收到集群内部客户端pod的访问，又可以收到集群外部客户端的请求
3，kube-proxy会监听着k8s-apiserver，一旦service资源发生变化（调k8s-api修改service信息），kube-proxy就会生成对应的负载调度调整，这样就可以保证service的最新状态

3.17.掌握pod挂载nfs的使用

nfs 卷允许将现有的 NFS（网络文件系统）挂载到容器中，且不像 emptyDir会丢失数据，当删除 Pod 时，nfs卷的内容被保留，卷仅仅是被卸载，这意味着 NFS 卷可以预先上传好数据待pod启动后即可直接使用，并且网络存储可以在多 pod 之间共享同一份数据，即NFS 可以被多个pod同时挂载和读写

创建一个pod测试挂载一个NFS

找一台主机搭建NFS

SHELL

编写yaml文件，并创建pod

YAML

SHELL

root@deploy:~/k8s-Resource-N79/case7-nfs# kubectl apply -f 1-deploy_nfs.yml

测试NFS是否正常挂载

SHELL

修改NFS的数据，看容器是否同步变化

SHELL

root@k8s-haproxy1:/data/k8sdata# echo "123,456" > index.html

修改haproxy，测试能否用vip地址访问

1，因为这里我们做了haproxy高可用，所以所有的haproxy都需要改配置文件并且重启

2，在生产环境中不建议直接重启haproxy，这样会导致服务中断

SHELL

删除pod，查看数据是否还在

SHELL

可以看到容器删除了，数据依旧保存在NFS，所以保证了数据的安全

创建一个pod测试挂载多个NFS

1，创建好挂载nfs需要的挂载点pool1，pool2

这里的挂载点需要手动创建，不然创建pod的时候会报错，找不到目录

SHELL

编辑yaml文件，并创建pod

YAML

SHELL

测试NFS是否正常挂载

SHELL

root@nginx-deployment-site2-865cd5c68-nwsrz:~# echo

删除pod，测试nfs数据是否还在

SHELL

重新执行yaml，测试NFS数据是否能用

SHELL

浏览器访问

3.18.基于nfs实现静态pvc的使用

https://v1-24.docs.kubernetes.io/zh-cn/docs/concepts/storage/persistent-volumes/

准备NFS存储

SHELL

创建PV

SHELL

创建PVC

SHELL

创建pod

SHELL

测试是否正常使用

SHELL

查看NFS数据

删除静态存储卷

注意！删除的顺序跟创建相反，先删pod再删pvc最后删pv

SHELL

查看NFS存储数据

SHELL

3.19.基于nfs及storageclass实现动态pvc的使用

https://kubernetes.io/zh-cn/docs/concepts/storage/storage-classes/

https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner

创建账户

YAML

SHELL

创建storageclass

YAML

SHELL

root@deploy:~/k8s-Resource-N79/case9-pv-dynamic-nfs# kubectl apply -f 2-storageclass.yaml 
storageclass.storage.k8s.io/managed-nfs-storage created

创建NFS provisioner

YAML

SHELL

创建PVC

YAML

SHELL

创建web服务

YAML

SHELL

验证NFS存储是否正常使用

SHELL

3.20.pod基于configmap实现配置挂载和环境变量

准备好nfs挂载点

SHELL

创建pod

YAML

SHELL

root@deploy:~/k8s-Resource-N79/case10-configmap# kubectl apply -f 1-deploy_configmap.yml

验证是否成功创建并挂载

SHELL

修改nginx配置

本次实验用的是官方镜像，需要手动改一下，如果是公司自己做的镜像，只需要打镜像的时候在nginx配置文件添加一行incloude就行

SHELL

添加一行：

SHELL

访问测试：

因为我们做了nginx匹配规则，所以访问域名会跳转到对应目录

也可以用lb访问：

SHELL

pod基于configmap实现环境变量设置

创建pod

YAML

SHELL

验证环境变量是否获取

SHELL

3.21.secret简介及常见类型、基于Secret实现Nginx tls认证

1，secret简介

1.1，Secret 的功能类似于 ConfigMap给pod提供额外的配置信息，但是 Secret是一种包含少量敏感信息例如密码、令牌或密钥的对象。

1.2，Secret 的名称必须是合法的 DNS 子域名

1.3，每个Secret的大小最多为1MiB，主要是为了避免用户创建非常大的 Secret进而导致API服务器和kubelet内存耗尽，不过创建很多小的 Secret也可能耗尽内存，可以使用资源配额来约束每个名字空间中 Secret的个数

1.4，在通过yaml文件创建secret时，可以设置data或stringData字段，data和stringData字段都是可选的，data字段中所有键值都必须是base64编码的字符串，如果不希望执行这种 base64字符串的转换操作，也可以选择设置stringData字段，其中可以使用任何非加密的字符串作为其取值。

1.5，Pod 可以用三种方式的任意一种来使用 Secret

作为挂载到一个或多个容器上的卷中的文件(crt文件、key文件)。
作为容器的环境变量。
由 kubelet 在为 Pod 拉取镜像时使用(与镜像仓库的认证)

基于Secret实现Nginx tls认证

创建secret

SHELL

base64的加密解密

SHELL

验证secret

SHELL

用-kubernetes.io/tls类型为nginx提供证书

自签发证书：

在公司里有商业机构签发的证书，这里做实验所以需要自签发证书，否则直接上传证书就可以了

SHELL

传证书到k8s

SHELL

创建web服务nginx并使用证书

YAML

SHELL

修改负载均衡器转发请求到nodeport

SHELL

修改nginx配置文件

本次实验用的是官方镜像，需要手动改一下，如果是公司自己做的镜像，只需要打镜像的时候在nginx配置文件添加一行incloude就行

SHELL

验证证书及配置文件

SHELL

浏览器访问

3.22.基于Secret实现私有镜像仓库的镜像下载认证

存储docker registry的认证信息，在下载镜像的时候使用，这样每一个node节点就可以不登录也可以下载私有级别的镜像了。

1，创建secret

kubectl create secret --help

方式一：通过命令创建

SHELL

#  kubectl create secret docker-registry Name \
   --docker-server=

方式二：通过docker认证文件创建（这里我们用这种方式）：

SHELL

创建pod

YAML

SHELL

通过pod信息也能看到在私有仓库拉的镜像

SHELL

3.23.总结StatefulSet、DaemonSet的特点及使用

StatefulSet的特点及使用

https://kubernetes.io/zh-cn/docs/concepts/workloads/controllers/statefulset/

1.1，StatefulSet简介

Statefulset为了解决有状态服务的集群部署、集群之间的数据同步问题(MySQL主从、Redis Cluster、ES集群等)
Statefulset所管理的Pod拥有唯一且固定的Pod名称
Statefulset按照顺序对pod进行启停、伸缩和回收
Headless Services(无头服务，请求的解析直接解析到pod IP)

部署StatefulSet

YAML

SHELL

StatefulSet的更新

SHELL

可以看到更新操作的顺序是从后往前更新的，先更新1再更新0，跟创建操作相反，跟删除操作一样

StatefulSet的伸缩

修改yaml文件

添加pod：

SHELL

root@deploy:~/k8s-Resource-N79/case12-Statefulset# vi 1-Statefulset.yaml

SHELL

可以看到StatefulSet是按照顺序扩容的

缩减pod

SHELL

root@deploy:~/k8s-Resource-N79/case12-Statefulset# vi 1-Statefulset.yaml

SHELL

可以看到StatefulSet是按照顺序缩减的

DaemonSet的特点及使用

https://kubernetes.io/zh-cn/docs/concepts/workloads/controllers/daemonset/

DaemonSet简介：

aemonSet 在当前集群中每个节点运行同一个pod，当有新的节点加入集群时也会为新的节点配置相同的pod，当节点从集群中移除时其pod也会被kubernetes回收，删除DaemonSet 控制器将删除其创建的所有的pod

DaemonSet的一些典型用法

在每个节点上运行集群守护进程
在每个节点上运行日志收集守护进程
在每个节点上运行监控守护进程

通过DaemonSet在每个节点上运行日志收集守护进程

YAML

SHELL

通过DaemonSet在每个节点上运行监控守护进程

YAML

SHELL

创建的守护进程会在每个节点监听一个自定义的端口，把宿主机的一些指标上传到了守护进程

4.Velero

4.1.Velero简介

Velero 是vmware开源的一个云原生的灾难恢复和迁移工具，它本身也是开源的,采用Go语言编写，可以安全的备份、恢复和迁移Kubernetes集群资源数据，https://velero.io/。
Velero 是西班牙语意思是帆船，非常符合Kubernetes社区的命名风格，Velero的开发公司Heptio，已被VMware收购。
Velero 支持标准的K8S集群，既可以是私有云平台也可以是公有云，除了灾备之外它还能做资源移转，支持把容器应用从一个集群迁移到另一个集群。
Velero 的工作方式就是把kubernetes中的数据备份到对象存储以实现高可用和持久化，默认的备份保存时间为720小时，并在需要的时候进行下载和恢复

4.2.Velero与etcd快照备份的区别

4.3.velero整体架构

4.4.备份流程

SHELL

4.5.部署minio

SHELL

minio web界面

minio 创建bucket:

bucket 名称：velerodata，后期会使用

minio 验证bucket

4.6.部署velero

在master节点部署velero： https://github.com/vmware-tanzu/velero #版本兼容性

SHELL

4.4.对default ns进行备份

minio验证备份数据

删除pod并验证数据恢复

删除pod

SHELL

root@k8s-master1:/data/velero# kubectl delete

恢复pod

SHELL

root@k8s-master1:/data/velero# velero restore create

验证pod

SHELL

4.7.对myserver ns进行备份

SHELL

minio验证备份

删除pod并验证恢复

SHELL

4.8.备份指定资源对象

SHELL

删除pod并验证恢复

删除deployment控制器并验证恢复

SHELL

4.9.批量备份所有namespace

批量备份不同的namespace

SHELL

补充:你想备份一个deployment，比如test命名空间下的nginx-deployment

SHELL

5.基于nerdctl + BuildKit + Containerd构建容器镜像

5.1.部署BuildKit

BuildKit的简介

2017 年由 docker 的创始⼈ ( tonistiigi 托尼斯蒂吉) 提议，将 docker build 的能⼒独⽴成⼀个项⽬，针对构建的底层技术进⾏协作，更好复⽤和定制构建技术，该将项⽬即为 buildkit。

BuildKit 是 Docker 官⽅社区推出的下⼀代镜像构建⼯具，官⽅宣称通过 BuildKit可以更加快速、有效、安全地构建容器镜像。

BuildKit的功能

BuildKit组成部分

buildkitd(服务端)：⽬前⽀持runc和containerd作为镜像构建环境，默认是runc，可以更换为containerd。
buildctl(客户端)：负责解析Dockerfile⽂件，并向服务端buildkitd发出构建请求

安装buildkitd

https://github.com/moby/buildkit #官方文档

SHELL

5.2.测试镜像构建

nerdctl常用命令

SHELL

#配置nerdctl命令补全
echo "source <(nerdctl completion bash)" >> /etc/profile
source /etc/profile

SHELL

提示：http的镜像仓库上传镜像（需要加--insecure-registry）

SHELL

root@k8s-master1:~# nerdctl --insecure-registry push qbnharbor.top/baseimages/centos:7.9.2009

镜像构建

SHELL

6.pod生命周期、pod常见状态、pod调度流程详解

6.1.pod的生命周期

概述

我们一般将pod对象从创建至终的这段时间范围内称为pod生命周期，它主要包含下面过程：

pod创建过程
运行初始化容器过程(init container)
运行主容器过程(main container)
容器启动钩子(post start)，容器终止前钩子(pre stop)
pod终止过程

pod生命周期图

pod的调度流程详解

pod调度流程

6.2.pod的常见状态

https://www.alibabacloud.com/help/zh/ack/ack-managed-and-ack-dedicated/support/troubleshooting-10/?spm=a2c63.p38356.0.0.6b2e1eb1YskwHo #k8s故障排除

6.3.Pod的终止流程

https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-terminating-with-grace

1.8.1，创建pod

用户通过kubectl向API-Server提交创建请求、API-Server完成鉴权和准入并将事件写入etcd
kube-scheduler完成调度流程
kubelet创建并启动pod、然后执行postStart
周期进行livenessProbe和readinessProbe检测，通过后，service关联pod
进入running状态
接受客户端请求

2.删除pod

向API-Server提交删除请求、API-Server完成鉴权和准入并将事件写入etcd
Pod被设置为”Terminating”状态、从service的Endpoints列表中删除并不再接受客户端请求。
pod执行PreStop
kubelet向pod中的容器发送SIGTERM信号(正常终止信号)终止pod里面的主进程，这个信号让容器知道自己很快将会被关闭
如果有设置删除宽限时间，则等待宽限时间到期，否则最多等待30s，Kubernetes等待指定的时间称为优雅终止宽限期,默认情况下是30秒,值得注意的是等待期与preStop Hook和SIGTERM信号并行执行，即Kubernetes可能不会等待preStop Hook完成(最长30秒之后主进程还没有结束就就强制终止pod)。

SIGKILL信号被发送到Pod，并删除Pod

SHELL

root@k8s-master1:-# kubectl explain Deployment.spec.template.spec  #删除宽限期使用说明

7.pause容器、init容器的功能及使用案例

7.1.pause容器的简介

宿主机的namespace

Pause容器，又叫Infra容器，是pod的基础容器，镜像体积只有几百KB 左右，配置在kubelet中，主要的功能是一个pod中多个容器的网络通信。

Infra容器被创建后会初始化Network Namespace，之后其它容器就可以加入到Infra容器中共享Infra容器的网络了，因此如果一个 Pod中的两个容器A和B。关系如下

1.A容器和B容器能够直接使用 localhost 通信。
2.A容器和B容器可以看到网卡、IP与端口监听信息。
3.Pod只有一个IP地址，也就是该Pod的Network Namespace对应的IP 地址(由Infra容器初始化并创建)。
4.k8s环境中的每个Pod有一个独立的IP地址(前提是地址足够用)，并且此IP被当前Pod中所有容器在内部共享使用。
5.pod删除后Infra容器随机被删除，其IP被回收。

Pause容器共享的Namespace

1.NET Namespace：Pod中的多个容器共享同一个网络命名空间，即使用相同的IP和端口信息。
2.IPC Namespace：Pod中的多个容器可以使用System V IPC或POSIX消息队列进行通信。
3.UTS Namespace：pod中的多个容器共享一个主机名。MNT Namespace、PID Namespace、User Namespace未共享

Pause容器的namespace验证

SHELL

pause容器的配置示例

准备nginx配置文件

SHELL

部署pause容器

SHELL

准备测试web页面

SHELL

部署nginx 容器，并使用paush容器网络

SHELL

验证：因为nginx无法直接解析php文件，所以需要转发到php去解析，所以502界面

部署php容器，并使用pause容器网络

SHELL

验证容器共用一个Net namespace

SHELL

7.2.init容器的简介

7.2.1.init容器的作用

可以为业务容器提前准备好业务容器的运行环境，比如将业务容器需要的配置文件提前生成并放在指定位置、检查数据权限或完整性、软件版本等基础运行环境。。
可以在运行业务容器之前准备好需要的业务数据，比如从OSS下载、或者从其它位置copy。
检查依赖的服务是否能够访问。

7.2.2.init容器的特点

一个pod可以有多个业务容器还能同时再有多个init容器，但是每个init容器和业务容器的运行环境都是隔离的。

init容器会比业务容器先启动。
init容器运行成功之后才会继续运行业务容器。
如果一个pod有多个init容器，则需要从上到下逐个运行并且全部成功，最后才会运行业务容器。
init容器不支持探针检测(因为init容器初始化完成后就退出再也不运行了)

init容器示例

YAML

SHELL

可以看到pod创建在node1节点，测试访问

补充：pause容器是必选项，每个pod都必须起的，而init容器是可选项，只有特定需要的pod才会布置init容器

8.Kubernetes实战案例-PV/PVC简介、Kubernetes运行Zookeeper集群并基于PVC实现数据持久化

8.1.基于StatefulSet+storageClassName实现

前提条件

有3个node节点（设置了反亲和）

SHELL

有存储类

SHELL

创建zookeeper集群

YAML

SHELL

验证Zookeeper可用

删除Zookeeper集群

SHELL

8.2.基于Deployment实现

准备好需要用到的yaml文件以及环境

zookeeper-persistentvolume

YAML

zookeeper-persistentvolumeclaim

YAML

zookeeper

YAML

SHELL

创建pv

SHELL

创建pvc

SHELL

创建zookeeper集群

SHELL

验证集群状态

SHELL

验证数据是否持久化到nfs

SHELL

删除zookeeper集群

SHELL

9.自定义镜像实现动静分离的web服务并结合NFS实现数据共享和持久化

先构建基础镜像（baseimages），并且打包到harbor

SHELL

----查看构建目录有那些文件----
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/system/centos# ls
Dockerfile  build-command.sh  filebeat-7.12.1-x86_64.rpm

SHELL

构建后端服务镜像

SHELL

root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/web/pub-images/jdk-1.8.212# ls
Dockerfile  build-command.sh  jdk-8u212-linux-x64.tar.gz  profile

SHELL

构建tomcat基础镜像

SHELL

构建业务镜像

SHELL

验证服务正常

用yaml文件把业务镜像打到k8s上

YAML

SHELL

root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/tomcat-app1# ls
tomcat-app1.yaml

SHELL

创建pod

SHELL

构建前端镜像

构建nginx基础镜像

SHELL

构建nginx业务镜像

SHELL

通过yaml文件把nginx业务镜像打到k8s

YAML

SHELL

配置通过haproxy访问前端服务

SHELL

测试

SHELL

# curl --cacert /etc/kubernetes/ssl/ca.pem -H "Authorization: Bearer ${TOKEN}" https://172.31.7.101:6443
# curl 127.0.0.1:6443/ #返回所有的API列表
# curl 127.0.0.1:6443/apis #分组API
# curl 127.0.0.1:6443/api/v1 #带具体版本号的API
# curl 127.0.0.1:6443/version #API版本信息
# curl 127.0.0.1:6443/healthz/etcd #与etcd的心跳监测
# curl 127.0.0.1:6443/apis/autoscaling/v1 #指定API的详细信息
# curl 127.0.0.1:6443/metrics #指标数据

# cat /var/lib/kube-proxy/kube-proxy-config.yaml
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 172.31.7.111
clientConnection:
  kubeconfig: "/etc/kubernetes/kube-proxy.kubeconfig"
clusterCIDR: "10.100.0.0/16"
conntrack:
  maxPerCore: 32768
  min: 131072
  tcpCloseWaitTimeout: 1h0m0s
  tcpEstablishedTimeout: 24h0m0s
healthzBindAddress: 172.31.7.111:10256
hostnameOverride: "172.31.7.111"
metricsBindAddress: 172.31.7.111:10249
mode: "ipvs" #指定使用ipvs及调度算法
ipvs: scheduler: sh
#cat nginx-service.yaml 开启会话保持
kind: Service apiVersion: v1
metadata:
  labels:
    app: magedu-nginx-service-label
  name: magedu-nginx-service
  namespace: magedu
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
    nodePort: 30004
selector:
  app: nginx
sessionAffinity: ClientIP
sessionAffinityConfig:
  clientIP:
    timeoutSeconds: 1800

1.各etcd节点启动后默认为 follower角色、默认termID为0、如果发现集群 内没有leader，则会变成 candidate角色并进行选举 leader。
2.candidate(候选节点)向其它候选节点发送投票信息(RequestVote)，默认投票给自己。
3.各候选节点相互收到另外的投票信息(如A收到BC的,B收到AC的,C收到AB 的)，然后对比日志是否比自己的更新，如果比自己的更新，则将自己的选票投给目的候选人，并回复一个包含自己最新日志信息的响应消息，如果C的日志更新，那么将会得到A、B、C的投票，则C全票当选，如果B挂了，得到A、 C的投票,则C超过半票当选。
4.C向其它节点发送自己leader心跳信息，以维护自己的身份(heartbeat- interval、默认100毫秒)
5.其它节点将角色切换为Follower并向leader同步数据。
6.如果选举超时(election-timeout )，则重新选举，如果选出来两个leader， 则超过集群总数半票的生效。

root@k8s-etcd1:~# cat /etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/ #数据保存目录
ExecStart=/usr/bin/etcd \ #二进制文件路径
--name=etcd1 \ #当前node 名称
--cert-file=/etc/etcd/ssl/etcd.pem \
--key-file=/etc/etcd/ssl/etcd-key.pem \
--peer-cert-file=/etc/etcd/ssl/etcd.pem \
--peer-key-file=/etc/etcd/ssl/etcd-key.pem \
--trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
--peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
--initial-advertise-peer-urls=https://172.31.7.105:2380 \ #通告自己的集群端口
--listen-peer-urls=https://172.31.7.105:2380 \ #集群之间通讯端口
--listen-client-urls=https://172.31.7.105:2379,http://127.0.0.1:2379 \ #客户端访问地址
--advertise-client-urls=https://172.31.7.105:2379 \ #通告自己的客户端端口
--initial-cluster-token=etcd-cluster-0 \ #创建集群使用的token，一个集群内的节点保持一致
--initial-cluster=etcd1=https://172.31.7.105:2380,etcd2=https://172.31.7.106:2380,etcd3=https://172.31.7.107:2380 \ #集群所有的节点信息
--initial-cluster-state=new \ #新建集群的时候的值为new,如果是已经存在的集群为existing。
--data-dir=/var/lib/etcd #数据目录路径
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target

[root@k8s-etcd01 ~]#ETCDCTL_API=3 /usr/local/bin/etcdctl defrag --cluster --endpoints=https://192.168.20.242:2379 \
> --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem \
> --cert=/etc/kubernetes/pki/etcd/etcd.pem \
> --key=/etc/kubernetes/pki/etcd/etcd-key.pem
Finished defragmenting etcd member[https://192.168.20.242:2379]
Finished defragmenting etcd member[https://192.168.20.243:2379]
Finished defragmenting etcd member[https://192.168.20.244:2379]

[root@k8s-etcd01 ~]#ETCDCTL_API=3 etcdctl member --help
NAME:
        member - Membership related commands
 
USAGE:
        etcdctl member <subcommand> [flags]
 
API VERSION:
        3.5
 
 
COMMANDS:
        add     Adds a member into the cluster
        list    Lists all members in the cluster
        promote Promotes a non-voting member in the cluster
        remove  Removes a member from the cluster
        update  Updates a member in the cluster
 
OPTIONS:
  -h, --help[=false]    help for member
 
GLOBAL OPTIONS:
      --cacert=""                               verify certificates of TLS-enabled secure servers using this CA bundle
      --cert=""                                 identify secure client using this TLS certificate file
      --command-timeout=5s                      timeout for short running command (excluding dial timeout)
      --debug[=false]                           enable client-side debug logging
      --dial-timeout=2s                         dial timeout for client connections
  -d, --discovery-srv=""                        domain name to query for SRV records describing cluster endpoints
      --discovery-srv-name=""                   service name to query when using DNS discovery
      --endpoints=[127.0.0.1:2379]              gRPC endpoints
      --hex[=false]                             print byte strings as hex encoded strings
      --insecure-discovery[=true]               accept insecure SRV records describing cluster endpoints
      --insecure-skip-tls-verify[=false]        skip server certificate verification (CAUTION: this option should be enabled only for testing purposes)
      --insecure-transport[=true]               disable transport security for client connections
      --keepalive-time=2s                       keepalive time for client connections
      --keepalive-timeout=6s                    keepalive timeout for client connections
      --key=""                                  identify secure client using this TLS key file
      --password=""                             password for authentication (if this option is used, --user option shouldn't include password)
      --user=""                                 username[:password] for authentication (prompt if password is not supplied)
  -w, --write-out="simple"                      set the output format (fields, json, protobuf, simple, table)

[root@k8s-etcd01 ~]#ETCDCTL_API=3 etcdctl member list
4a993e7dc41ddeb6, started, k8s-etcd01, https://192.168.20.242:2380, https://192.168.20.242:2379, false
8bbcb26d35527d9f, started, k8s-etcd02, https://192.168.20.243:2380, https://192.168.20.243:2379, false
aac17bc1e2e5cf96, started, k8s-etcd03, https://192.168.20.244:2380, https://192.168.20.244:2379, false
 
[root@k8s-etcd01 ~]#export NODE_IPS="192.168.20.242 192.168.20.243 192.168.20.244"
[root@k8s-etcd01 ~]#ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table member list \
> --endpoints=https://192.168.20.242:2379 \
> --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem \
> --cert=/etc/kubernetes/pki/etcd/etcd.pem \
> --key=/etc/kubernetes/pki/etcd/etcd-key.pem
+------------------+---------+------------+-----------------------------+-----------------------------+------------+
|        ID        | STATUS  |    NAME    |         PEER ADDRS          |        CLIENT ADDRS         | IS LEARNER |
+------------------+---------+------------+-----------------------------+-----------------------------+------------+
| 4a993e7dc41ddeb6 | started | k8s-etcd01 | https://192.168.20.242:2380 | https://192.168.20.242:2379 |      false |
| 8bbcb26d35527d9f | started | k8s-etcd02 | https://192.168.20.243:2380 | https://192.168.20.243:2379 |      false |
| aac17bc1e2e5cf96 | started | k8s-etcd03 | https://192.168.20.244:2380 | https://192.168.20.244:2379 |      false |
+------------------+---------+------------+-----------------------------+-----------------------------+------------+

export NODE_IPS="192.168.20.242 192.168.20.243 192.168.20.244"
for ip in ${NODE_IPS}; do ETCDCTL_API=3 /usr/local/bin/etcdctl --endpoints=https://${ip}:2379 \
--cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem  \
--cert=/etc/kubernetes/pki/etcd/etcd.pem \
--key=/etc/kubernetes/pki/etcd/etcd-key.pem \
endpoint health; done
 
[root@k8s-etcd01 ~]#for ip in ${NODE_IPS}; do ETCDCTL_API=3 /usr/local/bin/etcdctl --endpoints=https://${ip}:2379 \
> --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem  \
> --cert=/etc/kubernetes/pki/etcd/etcd.pem \
> --key=/etc/kubernetes/pki/etcd/etcd-key.pem \
> endpoint health; done
https://192.168.20.242:2379 is healthy: successfully committed proposal: took = 41.90421ms
https://192.168.20.243:2379 is healthy: successfully committed proposal: took = 54.24798ms
https://192.168.20.244:2379 is healthy: successfully committed proposal: took = 39.957666ms

export NODE_IPS="192.168.20.242 192.168.20.243 192.168.20.244"
for ip in ${NODE_IPS}; do ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table endpoint status \
--endpoints=https://${ip}:2379 \
--cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem  \
--cert=/etc/kubernetes/pki/etcd/etcd.pem \
--key=/etc/kubernetes/pki/etcd/etcd-key.pem; \
done

root@k8s-etcd1:~# ETCDCTL_API=3 etcdctl get / --prefix --keys-only #以路径的方式所有key信息
[root@k8s-etcd01 ~]#ETCDCTL_API=3 etcdctl get / --prefix --keys-only
/registry/apiextensions.k8s.io/customresourcedefinitions/bgpconfigurations.crd.projectcalico.org
/registry/apiextensions.k8s.io/customresourcedefinitions/bgpfilters.crd.projectcalico.org
/registry/apiextensions.k8s.io/customresourcedefinitions/bgppeers.crd.projectcalico.org
......

[root@k8s-etcd01 ~]#ETCDCTL_API=3 etcdctl get / --prefix --keys-only | grep pod
/registry/clusterrolebindings/system:controller:horizontal-pod-autoscaler
/registry/clusterrolebindings/system:controller:pod-garbage-collector
/registry/clusterroles/system:controller:horizontal-pod-autoscaler
/registry/clusterroles/system:controller:pod-garbage-collector

[root@k8s-etcd01 ~]#ETCDCTL_API=3 etcdctl get / --prefix --keys-only | grep namespaces
/registry/namespaces/default
/registry/namespaces/kube-node-lease
/registry/namespaces/kube-public
/registry/namespaces/kube-system
/registry/namespaces/kubernetes-dashboard

[root@k8s-etcd01 ~]#ETCDCTL_API=3 etcdctl get / --prefix --keys-only | grep deployment
/registry/clusterrolebindings/system:controller:deployment-controller
/registry/clusterroles/system:controller:deployment-controller
/registry/deployments/kube-system/calico-kube-controllers
/registry/deployments/kube-system/coredns
/registry/deployments/kube-system/metrics-server
/registry/deployments/kubernetes-dashboard/dashboard-metrics-scraper
/registry/deployments/kubernetes-dashboard/kubernetes-dashboard
/registry/serviceaccounts/kube-system/deployment-controller

[root@k8s-etcd01 ~]#ETCDCTL_API=3 etcdctl get / --prefix --keys-only | grep calico
/registry/apiextensions.k8s.io/customresourcedefinitions/bgpconfigurations.crd.projectcalico.org
/registry/apiextensions.k8s.io/customresourcedefinitions/bgpfilters.crd.projectcalico.org
/registry/apiextensions.k8s.io/customresourcedefinitions/bgppeers.crd.projectcalico.org
/registry/apiextensions.k8s.io/customresourcedefinitions/blockaffinities.crd.projectcalico.org
/registry/apiextensions.k8s.io/customresourcedefinitions/caliconodestatuses.crd.projectcalico.org
/registry/apiextensions.k8s.io/customresourcedefinitions/clusterinformations.crd.projectcalico.org
/registry/apiextensions.k8s.io/customresourcedefinitions/felixconfigurations.crd.projectcalico.org

root@etcd1:~# ETCDCTL_API=3 /usr/local/bin/etcdctl del /name
1
root@etcd1:~# ETCDCTL_API=3 /usr/local/bin/etcdctl get /nam

[root@k8s-etcd01 ~]#ETCDCTL_API=3 /usr/local/bin/etcdctl watch /data
root@k8s-etcd2:~# ETCDCTL_API=3 /usr/local/bin/etcdctl put /data "data v1"
OK
root@k8s-etcd2:~# ETCDCTL_API=3 /usr/local/bin/etcdctl put /data "data v1"
OK

root@k8s-etcd1:~# mkdir /data/etcd-backup-dir/ -p
root@k8s-etcd1:~# cat etcd-backup.sh
#!/bin/bash
source /etc/profile
DATE=`date +%Y-%m-%d_%H-%M-%S` ETCDCTL_API=3 /usr/local/bin/etcdctl snapshot save /data/etcd-backup-dir/etcd-snapshot-${DATE}.db

root@k8s-deploy:/etc/kubeasz# ./ezctl backup k8s-cluster1
root@k8s-deploy:/etc/kubeasz# kubectl get deployment -n myserver
root@k8s-deploy:/etc/kubeasz# kubectl delete deployment -n myserver myserver-nginx-deployment

root@k8s-deploy:/etc/kubeasz# grep db_to_restore ./roles/ -R #选择恢复的文件
./roles/cluster-restore/defaults/main.yml:db_to_restore: "snapshot.db"
./roles/cluster-restore/tasks/main.yml: src: "{{ cluster_dir }}/backup/{{ db_to_restore }}
root@k8s-master1:/etc/kubeasz# ./ezctl restore k8s-cluster1 #v3

for ip in ${NODE_IPS}; do ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table endpoint status \
--endpoints=https://${ip}:2379 \
--cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem \
--cert=/etc/kubernetes/pki/etcd/etcd.pem \
--key=/etc/kubernetes/pki/etcd/etcd-key.pem; \
done

1、恢复服务器系统
2、重新部署ETCD集群
3、停止kube-apiserver/controller-manager/scheduler/kubelet/kube-proxy
4、停止ETCD集群
5、各ETCD节点恢复同一份备份数据
6、启动各节点并验证ETCD集群
7、启动kube-apiserver/controller-manager/scheduler/kubelet/kube-proxy
8、验证k8s master状态及pod数据

export ETCDCTL_API=3
 
curl \
--cacert /etc/kubernetes/pki/etcd/etcd-ca.pem \
--cert /etc/kubernetes/pki/etcd/etcd.pem \
--key /etc/kubernetes/pki/etcd/etcd-key.pem \
https://192.168.20.242:2379/metrics -k
写入数据
[root@k8s-etcd01 /etc/kubernetes/pki/etcd]#ETCDCTL_API=3 /usr/local/bin/etcdctl --endpoints="https://192.168.20.242:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem put /name/1 tzh
OK
读取数据
[root@k8s-etcd01 /etc/kubernetes/pki/etcd]#ETCDCTL_API=3 /usr/local/bin/etcdctl --endpoints="https://192.168.20.242:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem get /name/1 
/name/1
tzh
备份数据
ETCDCTL_API=3 /usr/local/bin/etcdctl --endpoints="https://192.168.20.242:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem snapshot save `date +%Y-%m-%d`-etcd_back.db
 
[root@k8s-etcd01 ~]#ll 2024-01-29-etcd_back.db 
-rw------- 1 root root 4952096 Jan 29 16:50 2024-01-29-etcd_back.db

删除数据
[root@k8s-etcd01 ~]#ETCDCTL_API=3 /usr/local/bin/etcdctl --endpoints="https://192.168.20.242:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem del /name/1 
1
 
停止etcd服务
systemctl stop etcd
使用ETCDCTL API 3
export ETCDCTL_API=3
# 修改etcd启动参数--data-dir指向上一步的数据恢复目录，一般在/etc/etcd/etcd文件中
# 注意--data-dir参数，恢复以后修改data-dir,再次启动etcd
[root@k8s-etcd01 ~]#grep data-dir /etc/etcd/etcd.config.yml
data-dir: /var/lib/etcd
 
etcdctl snapshot restore 2024-01-29-etcd_back.db --name=k8s-etcd01 --endpoints="https://192.168.20.242:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem --initial-cluster=k8s-etcd01=https://192.168.20.242:2380 --initial-advertise-peer-urls=https://192.168.20.242:2380 --initial-cluster-token=etcd-k8s-cluster --data-dir=/var/lib/etcd
 
[root@k8s-etcd01 /var/lib/etcd/member]#ls
snap  wal
[root@k8s-etcd01 /var/lib/etcd/member]#mv wal/ ..
[root@k8s-etcd01 /var/lib/etcd]#ls
member  wal
systemctl start etcd
读取数据
[root@k8s-etcd01 ~]#ETCDCTL_API=3 /usr/local/bin/etcdctl --endpoints="https://192.168.20.242:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem get /name/1 
/name/1
tzh

ETCD 不同的版本的 etcdctl 命令不一样，但大致差不多，本文备份使用 napshot save , 每次备份一个节点就行
命令备份
k8s-etcd01 机器上备份
ETCDCTL_API=3 /usr/local/bin/etcdctl --endpoints="https://192.168.20.242:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem snapshot save `date +%Y-%m-%d`-etcd_back.db
备份脚本
 
#!/usr/bin/env bash
date;
CACERT="/etc/kubernetes/pki/etcd/etcd-ca.pem"
CERT="/etc/kubernetes/pki/etcd/etcd.pem"
EKY="/etc/kubernetes/pki/etcd/etcd-key.pem"
ENDPOINTS="https://192.168.20.242:2379"
 
ETCDCTL_API=3 etcdctl \
--cacert="${CACERT}" --cert="${CERT}" --key="${EKY}" \
--endpoints=${ENDPOINTS} \
snapshot save /data/etcd_backup_dir/etcd-snapshot-`date +%Y%m%d`.db
 
# 备份保留30天
find /data/etcd_backup_dir/ -name *.db -mtime +30 -exec rm -f {} \;
 
恢复
systemctl stop etcd
mv /var/lib/etcd /var/lib/etcd.bak
scp /data/etcd_backup_dir/etcd-snapshot-20240130.db root@k8s-etcd02:/data/etcd_backup_dir/ 
scp /data/etcd_backup_dir/etcd-snapshot-20240130.db root@k8s-etcd03:/data/etcd_backup_dir/ 
恢复(备份三台)
etcdctl snapshot restore 2024-01-29-etcd_back.db --name=k8s-etcd01 --endpoints="https://192.168.20.242:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem --initial-cluster=k8s-etcd01=https://192.168.20.242:2380 --initial-advertise-peer-urls=https://192.168.20.242:2380 --initial-cluster-token=etcd-k8s-cluster --data-dir=/var/lib/etcd
[root@k8s-etcd01 /var/lib/etcd/member]#ls
snap  wal
[root@k8s-etcd01 /var/lib/etcd/member]#mv wal/ ..
[root@k8s-etcd01 /var/lib/etcd]#ls
member  wal
systemctl start etcd
读取数据
[root@k8s-etcd01 ~]#ETCDCTL_API=3 /usr/local/bin/etcdctl --endpoints="https://192.168.20.242:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem get /name/1 
/name/1
tzh

root@deploy:/# kubectl get svc -n kube-system
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.100.0.2   <none>        53/UDP,53/TCP,9153/TCP   15d

root@deploy:~#  kubectl run net-test03 --image=centos:7.9.2009 sleep 360000   #创建pod
pod/net-test03 created
root@deploy:~# kubectl exec -it net-test03 sh
[root@net-test03 /]# cat /etc/resolv.conf 
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.100.0.2
options ndots:5
进入pod测试域名解析：
sh-4.2# yum install -y net-tools bind-utils
sh-4.2# nslookup kubernetes
Server:        10.100.0.2
Address:    10.100.0.2#53
Name:    kubernetes.default.svc.cluster.local
Address: 10.100.0.1
sh-4.2# ping www.jd.com
PING wwwv6.jcloudimg.com (119.147.159.135) 56(84) bytes of data.
64 bytes from 119.147.159.135 (119.147.159.135): icmp_seq=1 ttl=55 time=3.79 ms
64 bytes from 119.147.159.135 (119.147.159.135): icmp_seq=2 ttl=55 time=3.43 ms

root@deploy:~/k8s-Resource-N79/case3-controller# cat 1-rc.yml 
apiVersion: v1  
kind: ReplicationController  
metadata:  
  name: ng-rc
spec:
  replicas: 3             #设置副本数
  selector:               #选择带控制的pod标签
    app: ng-rc-80 
    #app1: ng-rc-81
  template:              #pod的模板，其内容是pod的定义
    metadata:  
      labels:            #pod的标签，与selector要保持一致
        app: ng-rc-80
        #app1: ng-rc-81
    spec:  
      containers:             
      - name: ng-rc-80 
        image: nginx  
        ports:  
        - containerPort: 80

root@deploy:~/k8s-Resource-N79/case3-controller# kubectl apply -f 1-rc.yml    #创建RC
replicationcontroller/ng-rc created
root@deploy:~/k8s-Resource-N79/case3-controller# kubectl get pod -o wide      #可以看到三个pod副本在创建中
NAME          READY   STATUS              RESTARTS   AGE   IP       NODE        NOMINATED NODE   READINESS GATES
ng-rc-4hhpm   0/1     ContainerCreating   0          5s    <none>   k8s-node2   <none>           <none>
ng-rc-cvrcx   0/1     ContainerCreating   0          5s    <none>   k8s-node1   <none>           <none>
ng-rc-z24fg   0/1     ContainerCreating   0          5s    <none>   k8s-node3   <none>           <none>
root@deploy:~/k8s-Resource-N79/case3-controller# kubectl get pod -o wide      #创建完成
NAME          READY   STATUS    RESTARTS   AGE     IP               NODE        NOMINATED NODE   READINESS GATES
ng-rc-4hhpm   1/1     Running   0          2m52s   10.200.169.136   k8s-node2   <none>           <none>
ng-rc-cvrcx   1/1     Running   0          2m52s   10.200.36.73     k8s-node1   <none>           <none>
ng-rc-z24fg   1/1     Running   0          2m52s   10.200.219.205   k8s-node3   <none>           <none>

root@deploy:~/k8s-Resource-N79/case3-controller# kubectl delete pod ng-rc-4hhpm  #删除pod "ng-rc-4hhpm"
pod "ng-rc-4hhpm" deleted
root@deploy:~/k8s-Resource-N79/case3-controller# kubectl get pod -o wide     
NAME          READY   STATUS              RESTARTS   AGE     IP               NODE        NOMINATED NODE   READINESS GATES
ng-rc-2sj4k   0/1     ContainerCreating   0          7s      <none>           k8s-node2   <none>           <none>
ng-rc-cvrcx   1/1     Running             0          5m45s   10.200.36.73     k8s-node1   <none>           <none>
ng-rc-z24fg   1/1     Running             0          5m45s   10.200.219.205   k8s-node3   <none>           <none>
root@deploy:~/k8s-Resource-N79/case3-controller# kubectl get pod -o wide 
NAME          READY   STATUS    RESTARTS   AGE     IP               NODE        NOMINATED NODE   READINESS GATES
ng-rc-2sj4k   1/1     Running   0          21s     10.200.169.137   k8s-node2   <none>           <none>
ng-rc-cvrcx   1/1     Running   0          5m59s   10.200.36.73     k8s-node1   <none>           <none>
ng-rc-z24fg   1/1     Running   0          5m59s   10.200.219.205   k8s-node3   <none>           <none>
root@deploy:~/k8s-Resource-N79/case3-controller# kubectl describe rc ng-rc   #从RC的详情也能看到这个过程
Events:
  Type    Reason            Age    From                    Message
  ----    ------            ----   ----                    -------
  Normal  SuccessfulCreate  7m51s  replication-controller  Created pod: ng-rc-4hhpm
  Normal  SuccessfulCreate  7m51s  replication-controller  Created pod: ng-rc-cvrcx
  Normal  SuccessfulCreate  7m51s  replication-controller  Created pod: ng-rc-z24fg
  Normal  SuccessfulCreate  2m13s  replication-controller  Created pod: ng-rc-2sj4k

root@deploy:~/k8s-Resource-N79/case3-controller# cat 1-rc.yml 
....
spec:  
  replicas: 4
   ....

root@deploy:~/k8s-Resource-N79/case3-controller# kubectl apply -f 1-rc.yml 
replicationcontroller/ng-rc configured
root@deploy:~/k8s-Resource-N79/case3-controller# kubectl get pod -o wide  
NAME          READY   STATUS    RESTARTS   AGE     IP               NODE        NOMINATED NODE   READINESS GATES
ng-rc-2sj4k   1/1     Running   0          7m55s   10.200.169.137   k8s-node2   <none>           <none>
ng-rc-cvrcx   1/1     Running   0          13m     10.200.36.73     k8s-node1   <none>           <none>
ng-rc-kwncp   1/1     Running   0          20s     10.200.169.138   k8s-node2   <none>           <none>
ng-rc-z24fg   1/1     Running   0          13m     10.200.219.205   k8s-node3   <none>           <none>

root@deploy:~/k8s-Resource-N79/case3-controller# vi 1-rc.yml 
root@deploy:~/k8s-Resource-N79/case3-controller# kubectl apply -f 1-rc.yml 
replicationcontroller/ng-rc configured
root@deploy:~/k8s-Resource-N79/case3-controller# kubectl get pod -o wide 
NAME          READY   STATUS    RESTARTS   AGE   IP               NODE        NOMINATED NODE   READINESS GATES
ng-rc-2sj4k   1/1     Running   0          10m   10.200.169.137   k8s-node2   <none>           <none>
ng-rc-cvrcx   1/1     Running   0          16m   10.200.36.73     k8s-node1   <none>           <none>

root@deploy:~/k8s-Resource-N79/case3-controller# kubectl delete rc ng-rc
replicationcontroller "ng-rc" deleted
root@deploy:~/k8s-Resource-N79/case3-controller# kubectl get pod -o wide 
No resources found in default namespace.
root@deploy:~/k8s-Resource-N79/case3-controller#

root@deploy:~/k8s-Resource-N79/case3-controller# kubectl get pod -o wide 
NAME          READY   STATUS    RESTARTS   AGE   IP               NODE        NOMINATED NODE   READINESS GATES
ng-rc-l8z56   1/1     Running   0          21s   10.200.169.139   k8s-node2   <none>           <none>
ng-rc-vs2kp   1/1     Running   0          21s   10.200.219.206   k8s-node3   <none>           <none>
root@deploy:~/k8s-Resource-N79/case3-controller# kubectl delete -f 1-rc.yml 
replicationcontroller "ng-rc" deleted
root@deploy:~/k8s-Resource-N79/case3-controller# kubectl get pod -o wide 
No resources found in default namespace.

root@deploy:~/k8s-Resource-N79/case3-controller# kubectl get pod -o wide 
NAME          READY   STATUS    RESTARTS   AGE   IP               NODE        NOMINATED NODE   READINESS GATES
ng-rc-qbdfq   1/1     Running   0          21s   10.200.169.140   k8s-node2   <none>           <none>
ng-rc-s9xt4   1/1     Running   0          21s   10.200.36.74     k8s-node1   <none>           <none>
root@deploy:~/k8s-Resource-N79/case3-controller# kubectl delete rc ng-rc --cascade=orphan
replicationcontroller "ng-rc" deleted
root@deploy:~/k8s-Resource-N79/case3-controller# kubectl get pod -o wide 
NAME          READY   STATUS    RESTARTS   AGE   IP               NODE        NOMINATED NODE   READINESS GATES
ng-rc-qbdfq   1/1     Running   0          43s   10.200.169.140   k8s-node2   <none>           <none>
ng-rc-s9xt4   1/1     Running   0          43s   10.200.36.74     k8s-node1   <none>           <none>
root@deploy:~/k8s-Resource-N79/case3-controller# kubectl get rc 
No resources found in default namespace.

root@deploy:~/k8s-Resource-N79/case3-controller# cat 2-rs.yml 
#apiVersion: extensions/v1beta1
apiVersion: apps/v1 
kind: ReplicaSet
metadata:
  name: frontend
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ng-rs-80
   #matchExpressions:
   #  - {key: app, operator: In, values: [ng-rs-80,ng-rs-81]}
  template:
    metadata:
      labels:
        app: ng-rs-80
    spec:  
      containers:  
      - name: ng-rs-80 
        image: nginx  
        ports:  
        - containerPort: 80

root@deploy:~/k8s-Resource-N79/case3-controller# kubectl apply -f 2-rs.yml 
replicaset.apps/frontend created
root@deploy:~/k8s-Resource-N79/case3-controller# kubectl get pod -o wide 
NAME             READY   STATUS    RESTARTS   AGE   IP               NODE        NOMINATED NODE   READINESS GATES
frontend-8fwd9   1/1     Running   0          16m   10.200.219.207   k8s-node3   <none>           <none>
frontend-mt6l7   1/1     Running   0          16m   10.200.169.141   k8s-node2   <none>           <none>

root@deploy:~/k8s-Resource-N79/case3-controller# cat 3-deployment.yml 
#apiVersion: extensions/v1beta1
apiVersion: apps/v1   #api版本
kind: Deployment    #控制器类型
metadata:
  name: nginx-deployment    #控制器名称
spec:
  replicas: 2        #副本数
  selector:          #选择器
    #app: ng-deploy-80 #rc
    matchLabels: #rs or deployment
      app: ng-deploy-80
    
    #matchExpressions:
    #  - {key: app, operator: In, values: [ng-deploy-80,ng-rs-81]}
  template:       #pod的模板，其内容是pod的定义
    metadata:
      labels:
        app: ng-deploy-80   #定义pod标签，必须与selector的标签一致
    spec:          #pod的信息
      containers:
      - name: ng-deploy-80
        image: nginx:1.20.2
        ports:
        - containerPort: 80

root@deploy:~/k8s-Resource-N79/case3-controller# kubectl apply -f 3-deployment.yml 
deployment.apps/nginx-deployment created
root@deploy:~/k8s-Resource-N79/case3-controller# kubectl get pod -o wide 
NAME                                READY   STATUS    RESTARTS   AGE   IP               NODE        NOMINATED NODE   READINESS GATES
nginx-deployment-577df84c7c-qlwh6   1/1     Running   0          47s   10.200.169.142   k8s-node2   <none>           <none>
nginx-deployment-577df84c7c-tnglq   1/1     Running   0          47s   10.200.36.75     k8s-node1   <none>           <none>

vi 3-deployment.yml
....
      - name: ng-deploy-80
        image: nginx:1.20.1
        ....

root@deploy:~/k8s-Resource-N79/case3-controller# kubectl apply -f 3-deployment.yml  #重新执行文件
deployment.apps/nginx-deployment configured
root@deploy:~/k8s-Resource-N79/case3-controller# kubectl get pod -o wide
NAME                                READY   STATUS              RESTARTS   AGE   IP               NODE        NOMINATED NODE   READINESS GATES
nginx-deployment-577df84c7c-qlwh6   0/1     Terminating         0          16m   10.200.169.142   k8s-node2   <none>           <none>
nginx-deployment-577df84c7c-tnglq   1/1     Running             0          16m   10.200.36.75     k8s-node1   <none>           <none>
nginx-deployment-6b7fcc8b58-27mrd   0/1     ContainerCreating   0          12s   <none>           k8s-node3   <none>           <none>
nginx-deployment-6b7fcc8b58-4w56t   1/1     Running             0          47s   10.200.169.143   k8s-node2   <none>           <none>
root@deploy:~/k8s-Resource-N79/case3-controller# kubectl get pod -o wide 
NAME                                READY   STATUS    RESTARTS   AGE   IP               NODE        NOMINATED NODE   READINESS GATES
nginx-deployment-6b7fcc8b58-27mrd   1/1     Running   0          22s   10.200.219.208   k8s-node3   <none>           <none>
nginx-deployment-6b7fcc8b58-4w56t   1/1     Running   0          57s   10.200.169.143   k8s-node2   <none>           <none>
root@deploy:~/k8s-Resource-N79/case3-controller# kubectl get rs
NAME                          DESIRED   CURRENT   READY   AGE
nginx-deployment-577df84c7c   0         0         0       16m
nginx-deployment-6b7fcc8b58   2         2         2       79s
root@deploy:~/k8s-Resource-N79/case3-controller#

root@deploy:~/k8s-Resource-N79/case3-controller# kubectl rollout undo deployment/nginx-deployment
deployment.apps/nginx-deployment rolled back
root@deploy:~/k8s-Resource-N79/case3-controller# kubectl get pod -o wide 
NAME                                READY   STATUS    RESTARTS   AGE   IP               NODE        NOMINATED NODE   READINESS GATES
nginx-deployment-577df84c7c-4vdrg   1/1     Running   0          6s    10.200.36.76     k8s-node1   <none>           <none>
nginx-deployment-577df84c7c-mlcg4   1/1     Running   0          10s   10.200.169.144   k8s-node2   <none>           <none>
root@deploy:~/k8s-Resource-N79/case3-controller# kubectl get rs
NAME                          DESIRED   CURRENT   READY   AGE
nginx-deployment-577df84c7c   2         2         2       23m
nginx-deployment-6b7fcc8b58   0         0         0       8m1s

root@deploy:~/k8s-Resource-N79/case3-controller# kubectl describe deployments nginx-deployment
.....
  Normal  ScalingReplicaSet  18m   deployment-controller  Scaled up replica set nginx-deployment-577df84c7c to 1 from 0
  Normal  ScalingReplicaSet  18m   deployment-controller  Scaled down replica set nginx-deployment-6b7fcc8b58 to 1 from 2
  Normal  ScalingReplicaSet  18m   deployment-controller  Scaled up replica set nginx-deployment-577df84c7c to 2 from 1
  Normal  ScalingReplicaSet  18m   deployment-controller  Scaled down replica set nginx-deployment-6b7fcc8b58 to 0 from 1
  root@deploy:~/k8s-Resource-N79/case3-controller# kubectl get rs
NAME                          DESIRED   CURRENT   READY   AGE
nginx-deployment-577df84c7c   2         2         2       23m
nginx-deployment-6b7fcc8b58   0         0         0       8m1s

root@k8s-haproxy1:~# apt install nfs-server -y
root@k8s-haproxy1:~# mkdir -p /data/k8sdata/kuboard
root@k8s-haproxy1:~# echo "/data/k8sdata *(rw,no_root_squash)" >> /etc/exports
root@k8s-haproxy1:~# systemctl restart nfs-server.service && systemctl enable nfs-server.service

root@deploy:~/k8s-Resource-N79/case7-nfs# vi 1-deploy_nfs.yml 
apiVersion: apps/v1    #指定api版本
kind: Deployment     #副本控制器类型
metadata:
  name: nginx-deployment
spec:
  replicas: 1      #副本数1个
  selector:
    matchLabels:
      app: ng-deploy-80
  template:
    metadata:
      labels:
        app: ng-deploy-80
    spec:
      containers:
      - name: ng-deploy-80
        image: nginx 
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: /usr/share/nginx/html/mysite    #容器内挂载点
          name: my-nfs-volume
      volumes: 
      - name: my-nfs-volume
        nfs:
          server: 192.168.100.109      #NFS地址
          path: /data/k8sdata          #NFS挂载点
---
apiVersion: v1
kind: Service
metadata:
  name: ng-deploy-80
spec:
  ports:
  - name: http
    port: 81            #service端口（clusterIP端口）
    targetPort: 80     #pod端口
    nodePort: 30016   #NodePort端口
    protocol: TCP
  type: NodePort    #NodePort类型
  selector:
    app: ng-deploy-80

root@deploy:~/k8s-Resource-N79/case7-nfs# kubectl get pod -o wide  #pod创建在node3节点
NAME                                READY   STATUS    RESTARTS   AGE   IP               NODE        NOMINATED NODE   READINESS GATES
nginx-deployment-6df48c4b64-r7p85   1/1     Running   0          19m   10.200.219.211   k8s-node3   <none>           <none>
#在node3节点查看宿主机挂载
root@node3:~# df -h
....
192.168.100.109:/data/k8sdata       14G  8.5G  4.5G  66% /var/lib/kubelet/pods/69fcb833-82c3-4def-89a0-447e57b4b33f/volumes/kubernetes.io~nfs/my-nfs-volume
....
#在容器内查看挂载
root@nginx-deployment-6df48c4b64-r7p85:/usr/share/nginx/html/mysite# df -h
Filesystem                         Size  Used Avail Use% Mounted on
overlay                             14G  9.2G  3.9G  71% /
tmpfs                               64M     0   64M   0% /dev
tmpfs                              2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/mapper/ubuntu--vg-ubuntu--lv   14G  9.2G  3.9G  71% /etc/hosts
shm                                 64M     0   64M   0% /dev/shm
192.168.100.109:/data/k8sdata       14G  8.5G  4.5G  66% /usr/share/nginx/html/mysite
tmpfs                              3.6G   12K  3.6G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                              2.0G     0  2.0G   0% /proc/acpi
tmpfs                              2.0G     0  2.0G   0% /proc/scsi
tmpfs                              2.0G     0  2.0G   0% /sys/firmware
#修改容器内数据
root@nginx-deployment-6df48c4b64-r7p85:/usr/share/nginx/html/mysite# echo "123" > index.html
#在nfs主机看有没有数据生成
root@k8s-haproxy1:/data/k8sdata# ls
index.html  kuboard
root@k8s-haproxy1:/data/k8sdata# cat index.html 
<h1>123</h1>

root@k8s-haproxy1:/data/k8sdata# vi /etc/haproxy/haproxy.cfg
....
listen myserver:80
  bind 192.168.100.201:80
  mode tcp
  server server1 192.168.100.151:30016 check inter 3s fall 3 rise 3
  server server2 192.168.100.152:30016 check inter 3s fall 3 rise 3
  server server3 192.168.100.153:30016 check inter 3s fall 3 rise 3
....
root@k8s-haproxy1:/data/k8sdata# systemctl restart haproxy

root@deploy:~/k8s-Resource-N79/case7-nfs# kubectl delete pod net-test01
pod "net-test01" deleted
root@k8s-haproxy1:/data/k8sdata# ls
index.html  kuboard
root@k8s-haproxy1:/data/k8sdata# cat index.html 
123,456

root@k8s-haproxy1:~# mkdir -p /data/k8sdata/pool1
root@k8s-haproxy1:~# mkdir -p /data/k8sdata/pool2
root@k8s-haproxy1:~# ls /data/k8sdata/
index.html  kuboard  pool1  pool2

root@deploy:~/k8s-Resource-N79/case7-nfs# vi 2-deploy_nfs.yml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-site2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ng-deploy-81
  template:
    metadata:
      labels:
        app: ng-deploy-81
    spec:
      containers:
      - name: ng-deploy-81
        image: nginx 
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: /usr/share/nginx/html/pool1    #pod挂载点1
          name: my-nfs-volume-pool1
        - mountPath: /usr/share/nginx/html/pool2    #pod挂载点1
          name: my-nfs-volume-pool2
        - mountPath: /etc/localtime            #容器时区文件挂载点
          name: timefile
          readOnly: true            #设置只读权限
      volumes:
      - name: my-nfs-volume-pool1
        nfs:
          server: 192.168.100.109
          path: /data/k8sdata/pool1    #nfs挂载点1
      - name: my-nfs-volume-pool2
        nfs:
          server: 192.168.100.109
          path: /data/k8sdata/pool2   #nfs挂载点2
      - name: timefile
        hostPath:
          path: /etc/localtime    #宿主机时区文件挂载点
---
apiVersion: v1
kind: Service
metadata:
  name: ng-deploy-81
spec:
  ports:
  - name: http
    port: 80
    targetPort: 80
    nodePort: 30017
    protocol: TCP
  type: NodePort
  selector:
    app: ng-deploy-81

root@deploy:~/k8s-Resource-N79/case7-nfs# kubectl apply -f 2-deploy_nfs.yml 
deployment.apps/nginx-deployment-site2 created
service/ng-deploy-81 created
root@deploy:~/k8s-Resource-N79/case7-nfs# kubectl get pod 
NAME                                     READY   STATUS    RESTARTS   AGE
nginx-deployment-site2-865cd5c68-nwsrz   1/1     Running   0          6s

root@node3:~# df -h  
....
192.168.100.109:/data/k8sdata/pool1   14G  8.5G  4.5G  66% /var/lib/kubelet/pods/9a1d3b8c-f90a-4c32-b084-76a8df028d03/volumes/kubernetes.io~nfs/my-nfs-volume-pool1
192.168.100.109:/data/k8sdata/pool2   14G  8.5G  4.5G  66% /var/lib/kubelet/pods/9a1d3b8c-f90a-4c32-b084-76a8df028d03/volumes/kubernetes.io~nfs/my-nfs-volume-pool2
#在容器里查看挂载点
root@nginx-deployment-site2-865cd5c68-nwsrz:/# df -h
...
192.168.100.109:/data/k8sdata/pool1   14G  8.5G  4.5G  66% /usr/share/nginx/html/pool1
192.168.100.109:/data/k8sdata/pool2   14G  8.5G  4.5G  66% /usr/share/nginx/html/pool2

root@deploy:~/k8s-Resource-N79/case7-nfs# kubectl delete -f 2-deploy_nfs.yml 
deployment.apps "nginx-deployment-site2" deleted
service "ng-deploy-81" deleted
#查看NFS数据
root@k8s-haproxy1:~# ls /data/k8sdata/ 
index.html  kuboard  pool1  pool2

root@deploy:~/k8s-Resource-N79/case7-nfs# kubectl apply -f 2-deploy_nfs.yml 
deployment.apps/nginx-deployment-site2 created
service/ng-deploy-81 created
root@deploy:~/k8s-Resource-N79/case7-nfs# kubectl get pod
NAME                                     READY   STATUS    RESTARTS   AGE
nginx-deployment-site2-865cd5c68-25xrp   1/1     Running   0          5s

root@k8s-haproxy1:~# apt install nfs-server -y
root@k8s-haproxy1:~# mkdir -p /data/k8sdata/myserver/myappdata
root@k8s-haproxy1:~# echo "/data/k8sdata *(rw,no_root_squash)" >> /etc/exports
root@k8s-haproxy1:~# systemctl restart nfs-server.service && systemctl enable nfs-server.service

#查看yaml文件
root@deploy:~/k8s-Resource-N79/case8-pv-static# cat 1-myapp-persistentvolume.yaml 
apiVersion: v1
kind: PersistentVolume     #定义pv类型
metadata:
  name: myserver-myapp-static-pv   #pv的名字
spec:
  capacity:               #定义PV空间
    storage: 10Gi 
  accessModes:
    - ReadWriteOnce     #访问模式
  nfs:           #定义存储类型
    path: /data/k8sdata/myserver/myappdata
    server: 192.168.100.109

root@deploy:~/k8s-Resource-N79/case8-pv-static# kubectl apply -f 1-myapp-persistentvolume.yaml 
persistentvolume/myserver-myapp-static-pv created
root@deploy:~/k8s-Resource-N79/case8-pv-static# kubectl get pv
NAME                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
myserver-myapp-static-pv   10Gi       RWO            Retain           Available                                   7s

#查看yaml文件
root@deploy:~/k8s-Resource-N79/case8-pv-static# cat 2-myapp-persistentvolumeclaim.yaml 
apiVersion: v1
kind: PersistentVolumeClaim     #指定PVC
metadata:
  name: myserver-myapp-static-pvc
  namespace: myserver      #这里必须跟挂载的pod同个namespace
spec:
  volumeName: myserver-myapp-static-pv
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi   #这个空间不能大于PV的空间

root@deploy:~/k8s-Resource-N79/case8-pv-static# kubectl apply -f 2-myapp-persistentvolumeclaim.yaml 
persistentvolumeclaim/myserver-myapp-static-pvc created
root@deploy:~/k8s-Resource-N79/case8-pv-static# kubectl get pvc -n myserver
NAME                        STATUS   VOLUME                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
myserver-myapp-static-pvc   Bound    myserver-myapp-static-pv   10Gi       RWO                           2m26s

#查看yaml文件
root@deploy:~/k8s-Resource-N79/case8-pv-static# cat 3-myapp-webserver.yaml 
kind: Deployment
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
metadata:
  labels:
    app: myserver-myapp 
  name: myserver-myapp-deployment-name
  namespace: myserver
spec:
  replicas: 1 
  selector:
    matchLabels:
      app: myserver-myapp-frontend
  template:
    metadata:
      labels:
        app: myserver-myapp-frontend
    spec:
      containers:
        - name: myserver-myapp-container
          image: nginx:1.20.0 
          #imagePullPolicy: Always
          volumeMounts:
          - mountPath: "/usr/share/nginx/html/statics"
            name: statics-datadir
      volumes:
        - name: statics-datadir
          persistentVolumeClaim:
            claimName: myserver-myapp-static-pvc 
 
---
kind: Service
apiVersion: v1
metadata:
  labels:
    app: myserver-myapp-service
  name: myserver-myapp-service-name
  namespace: myserver
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    targetPort: 80
    nodePort: 30009
  selector:
    app: myserver-myapp-frontend

root@deploy:~/k8s-Resource-N79/case8-pv-static# kubectl apply -f 3-myapp-webserver.yaml 
deployment.apps/myserver-myapp-deployment-name created
root@deploy:~/k8s-Resource-N79/case8-pv-static# kubectl get pod -n myserver
NAME                                              READY   STATUS    RESTARTS   AGE
myserver-myapp-deployment-name-6c445c87f8-mc9jd   1/1     Running   0          2m7s

root@deploy:~/k8s-Resource-N79/case8-pv-static# kubectl exec -it myserver-myapp-deployment-name-6c445c87f8-mc9jd bash -n myserver
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@myserver-myapp-deployment-name-6c445c87f8-mc9jd:/# df -h
Filesystem                                        Size  Used Avail Use% Mounted on
overlay                                            14G  9.4G  3.7G  72% /
tmpfs                                              64M     0   64M   0% /dev
tmpfs                                             2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/mapper/ubuntu--vg-ubuntu--lv                  14G  9.4G  3.7G  72% /etc/hosts
shm                                                64M     0   64M   0% /dev/shm
192.168.100.109:/data/k8sdata/myserver/myappdata   14G  9.7G  3.4G  75% /usr/share/nginx/html/statics
tmpfs                                             3.6G   12K  3.6G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                             2.0G     0  2.0G   0% /proc/acpi
tmpfs                                             2.0G     0  2.0G   0% /proc/scsi
tmpfs                                             2.0G     0  2.0G   0% /sys/firmware
root@myserver-myapp-deployment-name-6c445c87f8-mc9jd:/# echo "123" > /usr/share/nginx/html/statics/qbn.jss
root@myserver-myapp-deployment-name-6c445c87f8-mc9jd:/# cd /usr/share/nginx/html/statics/
root@myserver-myapp-deployment-name-6c445c87f8-mc9jd:/# wget https://www.magedu.com/wp-content/uploads/2022/01/2022012003114993.jpg

root@deploy:~/k8s-Resource-N79/case8-pv-static# kubectl delete -f 3-myapp-webserver.yaml 
deployment.apps "myserver-myapp-deployment-name" deleted
service "myserver-myapp-service-name" deleted
root@deploy:~/k8s-Resource-N79/case8-pv-static# kubectl delete -f 2-myapp-persistentvolumeclaim.yaml 
persistentvolumeclaim "myserver-myapp-static-pvc" deleted
root@deploy:~/k8s-Resource-N79/case8-pv-static# kubectl delete -f 1-myapp-persistentvolume.yaml 
persistentvolume "myserver-myapp-static-pv" deleted
root@deploy:~/k8s-Resource-N79/case8-pv-static# kubectl get pod -n myserver
No resources found in myserver namespace.
root@deploy:~/k8s-Resource-N79/case8-pv-static# kubectl get pvc -n myserver
No resources found in myserver namespace.
root@deploy:~/k8s-Resource-N79/case8-pv-static# kubectl get pv
No resources found

root@k8s-haproxy1:~# ll /data/k8sdata/myserver/myappdata/
total 152
drwxr-xr-x 2 root root   4096 Aug 22 16:21 ./
drwxr-xr-x 3 root root   4096 Aug 22 15:45 ../
-rw-r--r-- 1 root root 141386 Jan 20  2022 1.jpg
-rw-r--r-- 1 root root      4 Aug 22 15:55 qbn.jss

root@deploy:~/k8s-Resource-N79/case9-pv-dynamic-nfs# cat 1-rbac.yaml 
apiVersion: v1
kind: Namespace
metadata:
  name: nfs
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfs
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: nfs
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfs
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfs
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: nfs
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io

root@deploy:~/k8s-Resource-N79/case9-pv-dynamic-nfs# kubectl apply -f 1-rbac.yaml 
namespace/nfs created
serviceaccount/nfs-client-provisioner created
clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner created
role.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created

root@deploy:~/k8s-Resource-N79/case9-pv-dynamic-nfs# cat 2-storageclass.yaml 
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfs-storage
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
reclaimPolicy: Retain #PV的删除策略，默认为delete，删除PV后立即删除NFS server的数据
mountOptions:
  #- vers=4.1 #containerd有部分参数异常
  #- noresvport #告知NFS客户端在重新建立网络连接时，使用新的传输控制协议源端口
  - noatime #访问文件时不更新文件inode中的时间戳，高并发环境可提高性能
parameters:
  #mountOptions: "vers=4.1,noresvport,noatime"
  archiveOnDelete: "true"  #删除pod时保留pod数据，默认为false时为不保留数据

root@deploy:~/k8s-Resource-N79/case9-pv-dynamic-nfs# cat 3-nfs-provisioner.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfs
spec:
  replicas: 1
  strategy: #部署策略
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          #image: k8s.gcr.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2 
          image: registry.cn-qingdao.aliyuncs.com/zhangshijie/nfs-subdir-external-provisioner:v4.0.2 
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: k8s-sigs.io/nfs-subdir-external-provisioner
            - name: NFS_SERVER
              value: 192.168.100.109
            - name: NFS_PATH
              value: /data/volumes
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.100.109
            path: /data/volumes

root@k8s-haproxy1:~# mkdir -p /data/volumes
root@k8s-haproxy1:/data# echo "/data/volumes *(rw,no_root_squash)" >> /etc/exports     #添加目录共享及权限设置
root@k8s-haproxy1:/data# systemctl restart nfs-server
root@deploy:~/k8s-Resource-N79/case9-pv-dynamic-nfs# kubectl apply -f 3-nfs-provisioner.yaml
deployment.apps/nfs-client-provisioner created
root@deploy:~/k8s-Resource-N79/case9-pv-dynamic-nfs# kubectl get storageclasses.storage.k8s.io 
NAME                  PROVISIONER                                   RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
managed-nfs-storage   k8s-sigs.io/nfs-subdir-external-provisioner   Retain          Immediate           false                  9m22s

root@deploy:~/k8s-Resource-N79/case9-pv-dynamic-nfs# cat 4-create-pvc.yaml 
# Test PVC
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: myserver-myapp-dynamic-pvc
  namespace: myserver
spec:
  storageClassName: managed-nfs-storage #调用的storageclass 名称
  accessModes:
    - ReadWriteMany #访问权限
  resources:
    requests:
      storage: 500Mi #空间大小

root@deploy:~/k8s-Resource-N79/case9-pv-dynamic-nfs# kubectl apply -f 4-create-pvc.yaml 
persistentvolumeclaim/myserver-myapp-dynamic-pvc created
root@deploy:~/k8s-Resource-N79/case9-pv-dynamic-nfs# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                 STORAGECLASS          REASON   AGE
pvc-b3986eb8-3b17-4c59-86ec-d3af40af7b18   500Mi      RWX            Retain           Bound    myserver/myserver-myapp-dynamic-pvc   managed-nfs-storage            4s
root@deploy:~/k8s-Resource-N79/case9-pv-dynamic-nfs# kubectl get pvc -n myserver
NAME                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
myserver-myapp-dynamic-pvc   Bound    pvc-b3986eb8-3b17-4c59-86ec-d3af40af7b18   500Mi      RWX            managed-nfs-storage   5m56s

root@deploy:~/k8s-Resource-N79/case9-pv-dynamic-nfs# cat 5-myapp-webserver.yaml 
kind: Deployment
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
metadata:
  labels:
    app: myserver-myapp 
  name: myserver-myapp-deployment-name
  namespace: myserver
spec:
  replicas: 1 
  selector:
    matchLabels:
      app: myserver-myapp-frontend
  template:
    metadata:
      labels:
        app: myserver-myapp-frontend
    spec:
      containers:
        - name: myserver-myapp-container
          image: nginx:1.20.0 
          #imagePullPolicy: Always
          volumeMounts:
          - mountPath: "/usr/share/nginx/html/statics"
            name: statics-datadir
      volumes:
        - name: statics-datadir
          persistentVolumeClaim:
            claimName: myserver-myapp-dynamic-pvc 
 
---
kind: Service
apiVersion: v1
metadata:
  labels:
    app: myserver-myapp-service
  name: myserver-myapp-service-name
  namespace: myserver
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    targetPort: 80
    nodePort: 30010
  selector:
    app: myserver-myapp-frontend
root@deploy:~/k8s-Resource-N79/case9-pv-dynamic-nfs#

root@deploy:~/k8s-Resource-N79/case9-pv-dynamic-nfs# kubectl apply -f 5-myapp-webserver.yaml 
deployment.apps/myserver-myapp-deployment-name created
service/myserver-myapp-service-name created
root@deploy:~/k8s-Resource-N79/case9-pv-dynamic-nfs# kubectl get pod -n myserver
NAME                                              READY   STATUS    RESTARTS   AGE
myserver-myapp-deployment-name-6bd686c869-xktzl   1/1     Running   0          84s

root@deploy:~/k8s-Resource-N79/case9-pv-dynamic-nfs# kubectl exec -it -n myserver myserver-myapp-deployment-name-6bd686c869-xktzl bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@myserver-myapp-deployment-name-6bd686c869-xktzl:/# df
Filesystem                                                                                                 1K-blocks    Used Available Use% Mounted on
overlay                                                                                                     14339080 9833512   3755388  73% /
tmpfs                                                                                                          65536       0     65536   0% /dev
tmpfs                                                                                                        2006924       0   2006924   0% /sys/fs/cgroup
/dev/mapper/ubuntu--vg-ubuntu--lv                                                                           14339080 9833512   3755388  73% /etc/hosts
shm                                                                                                            65536       0     65536   0% /dev/shm
192.168.100.109:/data/volumes/myserver-myserver-myapp-dynamic-pvc-pvc-b3986eb8-3b17-4c59-86ec-d3af40af7b18  14339584 8906752   4682752  66% /usr/share/nginx/html/statics
tmpfs                                                                                                        3706648      12   3706636   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                                                                                        2006924       0   2006924   0% /proc/acpi
tmpfs                                                                                                        2006924       0   2006924   0% /proc/scsi
tmpfs                                                                                                        2006924       0   2006924   0% /sys/firmware
root@myserver-myapp-deployment-name-6bd686c869-xktzl:/# echo "123" > /usr/share/nginx/html/statics/index.html
#到NFS查看数据
root@k8s-haproxy1:/data# ll volumes/myserver-myserver-myapp-dynamic-pvc-pvc-b3986eb8-3b17-4c59-86ec-d3af40af7b18/
total 12
drwxrwxrwx 2 root root 4096 Aug 22 17:29 ./
drwxr-xr-x 3 root root 4096 Aug 22 17:18 ../
-rw-r--r-- 1 root root    4 Aug 22 17:29 index.html

root@k8s-haproxy1:~# mkdir -p /data/k8sdata/mysite
root@k8s-haproxy1:~# mkdir -p /data/k8sdata/myserver
root@k8s-haproxy1:~# echo "myserver web" > /data/k8sdata/myserver/index.html
root@k8s-haproxy1:~# echo "mysite web" > /data/k8sdata/mysite/index.html

###查看yaml文件####
root@deploy:~/k8s-Resource-N79/case10-configmap# cat 1-deploy_configmap.yml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  mysite: |
    server {
       listen       80;
       server_name  www.mysite.com;
       index        index.html index.php index.htm;
 
       location / {
           root /data/nginx/mysite;
           if (!-e $request_filename) {
               rewrite ^/(.*) /index.html last;
           }
       }
    }
 
  myserver: |
    server {
       listen       80;
       server_name  www.myserver.com;
       index        index.html index.php index.htm;
 
       location / {
           root /data/nginx/myserver;
           if (!-e $request_filename) {
               rewrite ^/(.*) /index.html last;
           }
       }
    }  
---
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ng-deploy-80
  template:
    metadata:
      labels:
        app: ng-deploy-80
    spec:
      containers:
      - name: ng-deploy-80
        image: nginx:1.20.0
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: /data/nginx/mysite
          name: nginx-mysite-statics
        - mountPath: /data/nginx/myserver
          name: nginx-myserver-statics
        - name: nginx-mysite-config
          mountPath:  /etc/nginx/conf.d/mysite/
        - name: nginx-myserver-config
          mountPath:  /etc/nginx/conf.d/myserver/
      volumes:
      - name: nginx-mysite-config
        configMap:
          name: nginx-config
          items:
             - key: mysite
               path: mysite.conf
      - name: nginx-myserver-config
        configMap:
          name: nginx-config
          items:
             - key: myserver
               path: myserver.conf
      - name: nginx-myserver-statics
        nfs:
          server: 192.168.100.109
          path: /data/k8sdata/myserver
      - name: nginx-mysite-statics
        nfs:
          server: 192.168.100.109
          path: /data/k8sdata/mysite
---
apiVersion: v1
kind: Service
metadata:
  name: ng-deploy-80
spec:
  ports:
  - name: http
    port: 81
    targetPort: 80
    nodePort: 30019
    protocol: TCP
  type: NodePort
  selector:
    app: ng-deploy-80

root@deploy:~/k8s-Resource-N79/case10-configmap# kubectl exec -it nginx-deployment-78d749d5f9-hgs97 bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@nginx-deployment-78d749d5f9-hgs97:/# df -h
Filesystem                              Size  Used Avail Use% Mounted on
overlay                                  14G  9.5G  3.6G  73% /
tmpfs                                    64M     0   64M   0% /dev
tmpfs                                   2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/mapper/ubuntu--vg-ubuntu--lv        14G  9.5G  3.6G  73% /etc/hosts
shm                                      64M     0   64M   0% /dev/shm
192.168.100.109:/data/k8sdata/mysite     14G  8.6G  4.5G  66% /data/nginx/mysite
192.168.100.109:/data/k8sdata/myserver   14G  8.6G  4.5G  66% /data/nginx/myserver
tmpfs                                   3.6G   12K  3.6G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                   2.0G     0  2.0G   0% /proc/acpi
tmpfs                                   2.0G     0  2.0G   0% /proc/scsi
tmpfs                                   2.0G     0  2.0G   0% /sys/firmware
root@nginx-deployment-78d749d5f9-hgs97:/# ls /data/nginx/myserver/
index.html
root@nginx-deployment-78d749d5f9-hgs97:/# ls /data/nginx/mysite/  
index.html
root@nginx-deployment-78d749d5f9-hgs97:/# ls /etc/nginx/conf.d/
default.conf  myserver    mysite
root@nginx-deployment-78d749d5f9-hgs97:/# cat /etc/nginx/conf.d/myserver/myserver.conf 
server {
   listen       80;
   server_name  www.myserver.com;
   index        index.html index.php index.htm;
 
   location / {
       root /data/nginx/myserver;
       if (!-e $request_filename) {
           rewrite ^/(.*) /index.html last;
       }
   }
}  
root@nginx-deployment-78d749d5f9-hgs97:/# cat /etc/nginx/conf.d/mysite/mysite.conf     
server {
   listen       80;
   server_name  www.mysite.com;
   index        index.html index.php index.htm;
 
   location / {
       root /data/nginx/mysite;
       if (!-e $request_filename) {
           rewrite ^/(.*) /index.html last;
       }
   }
}

include /etc/nginx/conf.d/*.conf;
include /etc/nginx/conf.d/*/*.conf;
root@nginx-deployment-78d749d5f9-hgs97:/# nginx -s reload  #重载配置文件

root@k8s-haproxy1:~# vi /etc/haproxy/haproxy.cfg
root@k8s-haproxy2:~# vi /etc/haproxy/haproxy.cfg
添加：
listen myserver:80
  bind 192.168.100.201:80
  mode tcp
  server server1 192.168.100.151:30019 check inter 3s fall 3 rise 3
  server server2 192.168.100.152:30019 check inter 3s fall 3 rise 3
  server server3 192.168.100.153:30019 check inter 3s fall 3 rise 3
root@k8s-haproxy1:~# systemctl restart haproxy
root@k8s-haproxy2:~# systemctl restart haproxy

##查看yaml文件##
root@deploy:~/k8s-Resource-N79/case10-configmap# cat 2-deploy_configmap_env.yml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  host: "192.168.100.201"
  username: "user1"
  password: "12345678"
---
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ng-deploy-80
  template:
    metadata:
      labels:
        app: ng-deploy-80
    spec:
      containers:
      - name: ng-deploy-80
        image: nginx 
        env:
        - name: HOST
          valueFrom:
            configMapKeyRef:
              name: nginx-config
              key: host
        - name: USERNAME
          valueFrom:
            configMapKeyRef:
              name: nginx-config
              key: username
        - name: PASSWORD
          valueFrom:
            configMapKeyRef:
              name: nginx-config
              key: password
        ######
        - name: "MySQLPass"
          value: "123456"
        ports:
        - containerPort: 80

root@deploy:~/k8s-Resource-N79/case10-configmap# kubectl apply -f 2-deploy_configmap_env.yml
root@deploy:~/k8s-Resource-N79/case10-configmap# kubectl get pod 
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-5d4f87d989-mt4br   1/1     Running   0          74s

root@deploy:~/k8s-Resource-N79/case10-configmap# kubectl exec -it nginx-deployment-5d4f87d989-mt4br bash 
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@nginx-deployment-5d4f87d989-mt4br:/# env
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_SERVICE_PORT=443
HOSTNAME=nginx-deployment-5d4f87d989-mt4br
MySQLPass=123456
PWD=/
PKG_RELEASE=1~bullseye
HOME=/root
USERNAME=user1
KUBERNETES_PORT_443_TCP=tcp://10.100.0.1:443
PASSWORD=12345678
NJS_VERSION=0.7.1
TERM=xterm
HOST=192.168.100.201
SHLVL=1
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_ADDR=10.100.0.1
KUBERNETES_SERVICE_HOST=10.100.0.1
KUBERNETES_PORT=tcp://10.100.0.1:443
KUBERNETES_PORT_443_TCP_PORT=443
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
NGINX_VERSION=1.21.5
_=/usr/bin/env
root@nginx-deployment-5d4f87d989-mt4br:/# echo $HOST
192.168.100.201

##查看yaml文件##
root@deploy:~/k8s-Resource-N79/case11-secret# cat 1-secret-Opaque-data.yaml 
apiVersion: v1
kind: Secret
metadata:
  name: mysecret-data
  namespace: myserver
type: Opaque
#stringData:             #不加密数据类型，stringdate跟data，二选一      
#  user: 'admin'
#  password: '12345
data:                  #使用base64加密的数据类型
  user: YWRtaW4K
  password: MTIzNDU2Cg==
  #age: 18 #非base64加密的会报错

root@deploy:~/k8s-Resource-N79/case11-secret# echo 123456 | base64     #加密
MTIzNDU2Cg==
root@deploy:~/k8s-Resource-N79/case11-secret# echo  MTIzNDU2Cg== | base64 -d   #解密
123456
root@deploy:~/k8s-Resource-N79/case11-secret# kubectl apply -f 1-secret-Opaque-data.yaml 
secret/mysecret-data created
root@deploy:~/k8s-Resource-N79/case11-secret# kubectl get secret -n myserver
NAME            TYPE     DATA   AGE
mysecret-data   Opaque   2      26s

root@deploy:~/k8s-Resource-N79/case11-secret# kubectl get secrets mysecret-data -n myserver -o yaml
apiVersion: v1
data:
  password: MTIzNDU2Cg==
  user: YWRtaW4K
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"password":"MTIzNDU2Cg==","user":"YWRtaW4K"},"kind":"Secret","metadata":{"annotations":{},"name":"mysecret-data","namespace":"myserver"},"type":"Opaque"}
  creationTimestamp: "2023-08-23T05:27:48Z"
  name: mysecret-data
  namespace: myserver
  resourceVersion: "1772544"
  uid: ebee7194-bb47-435b-8753-31a801671146
type: Opaque

root@deploy:~/k8s-Resource-N79/case11-secret#  mkdir certs   
root@deploy:~/k8s-Resource-N79/case11-secret# cd certs/
root@deploy:~/k8s-Resource-N79/case11-secret/certs# openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 3560 -nodes -subj '/CN=www.ca.com'
root@deploy:~/k8s-Resource-N79/case11-secret/certs# openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj '/CN=www.mysite.com'
root@deploy:~/k8s-Resource-N79/case11-secret/certs# openssl x509 -req -sha256 -days 3650 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt

root@deploy:~/k8s-Resource-N79/case11-secret/certs# kubectl create secret tls myserver-tls-key --cert=./server.crt --key=./server.key -n myserver
secret/myserver-tls-key created
root@deploy:~/k8s-Resource-N79/case11-secret/certs# kubectl get secrets -n myserver
NAME               TYPE                DATA   AGE
mysecret-data      Opaque              2      66m
myserver-tls-key   kubernetes.io/tls   2      4m5s
root@deploy:~/k8s-Resource-N79/case11-secret/certs# kubectl get secrets -n myserver myserver-tls-key -o yaml

##查看yaml文件##
root@deploy:~/k8s-Resource-N79/case11-secret# cat 4-secret-tls.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
  namespace: myserver
data:
 default: |
    server {
       listen       80;
       server_name  www.mysite.com;
       listen 443 ssl;
       ssl_certificate /etc/nginx/conf.d/certs/tls.crt;
       ssl_certificate_key /etc/nginx/conf.d/certs/tls.key;
 
       location / {
           root /usr/share/nginx/html; 
           index index.html;
           if ($scheme = http ){  #未加条件判断，会导致死循环
              rewrite / https://www.mysite.com permanent;
           }  
 
           if (!-e $request_filename) {
               rewrite ^/(.*) /index.html last;
           }
       }
    }
---
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myserver-myapp-frontend-deployment
  namespace: myserver
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myserver-myapp-frontend
  template:
    metadata:
      labels:
        app: myserver-myapp-frontend
    spec:
      containers:
      - name: myserver-myapp-frontend
        image: nginx:1.20.2-alpine 
        ports:
          - containerPort: 80
        volumeMounts:
          - name: nginx-config
            mountPath:  /etc/nginx/conf.d/myserver
          - name: myserver-tls-key
            mountPath:  /etc/nginx/conf.d/certs
      volumes:
      - name: nginx-config
        configMap:
          name: nginx-config
          items:
             - key: default
               path: mysite.conf
      - name: myserver-tls-key
        secret:
          secretName: myserver-tls-key 
---
apiVersion: v1
kind: Service
metadata:
  name: myserver-myapp-frontend
  namespace: myserver
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    targetPort: 80
    nodePort: 30018
    protocol: TCP
  - name: htts
    port: 443
    targetPort: 443
    nodePort: 30029
    protocol: TCP
  selector:
    app: myserver-myapp-frontend

root@deploy:~/k8s-Resource-N79/case11-secret# kubectl apply -f 4-secret-tls.yaml
configmap/nginx-config created
deployment.apps/myserver-myapp-frontend-deployment created
service/myserver-myapp-frontend created
root@deploy:~/k8s-Resource-N79/case11-secret# kubectl get pod -n myserver
NAME                                                  READY   STATUS    RESTARTS   AGE
myserver-myapp-app1-deployment-6fd74bfbb9-fhbw2       1/1     Running   0          81m
myserver-myapp-frontend-deployment-7df66455b9-t2hcw   1/1     Running   0          15m

root@k8s-haproxy1:~# vi /etc/haproxy/haproxy.cfg
root@k8s-haproxy2:~# vi /etc/haproxy/haproxy.cfg
添加：
listen myserver:80
  bind 192.168.100.201:80
  mode tcp
  server server1 192.168.100.151:30018 check inter 3s fall 3 rise 3
  server server2 192.168.100.152:30018 check inter 3s fall 3 rise 3
  server server3 192.168.100.153:30018 check inter 3s fall 3 rise 3
listen myserver:443
  bind 192.168.100.201:443
  mode tcp
  server server1 192.168.100.151:30029 check inter 3s fall 3 rise 3
  server server2 192.168.100.152:30029 check inter 3s fall 3 rise 3
  server server3 192.168.100.153:30029 check inter 3s fall 3 rise 3
root@k8s-haproxy1:~# systemctl restart haproxy
root@k8s-haproxy2:~# systemctl restart haproxy

root@deploy:~/k8s-Resource-N79/case11-secret# kubectl get pod -n myserver
NAME                                                  READY   STATUS    RESTARTS   AGE
myserver-myapp-app1-deployment-6fd74bfbb9-fhbw2       1/1     Running   0          69m
myserver-myapp-frontend-deployment-7df66455b9-t2hcw   1/1     Running   0          4m10s
root@deploy:~/k8s-Resource-N79/case11-secret# kubectl exec -it myserver-myapp-frontend-deployment-7df66455b9-t2hcw -n myserver sh 
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/ # apk update && apk add vim
添加一行：
    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/conf.d/*/*.conf;
}
/ # nginx -s reload   重载配置

/ # ls /etc/nginx/conf.d/*/*.conf
/etc/nginx/conf.d/myserver/mysite.conf
/ # 
/ # ls /etc/nginx/conf.d/myserver/*conf
/etc/nginx/conf.d/myserver/mysite.conf
/ # ls /etc/nginx/conf.d/certs/
tls.crt  tls.key
/ # netstat -tnl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      
tcp        0      0 :::80                   :::*                    LISTEN

root@k8s-master1:~# docker/nerdctl login --username=admin qbnharbor.top    #登录你的私有仓库
root@k8s-master1:~# kubectl create secret generic qbn-image-pull-key --from-file=.dockerconfigjson=/root/.docker/config.json --type=kubernetes.io/dockerconfigjson -n myserver
root@deploy:~/k8s-Resource-N79/case11-secret# kubectl get secrets -n myserver
NAME                 TYPE                             DATA   AGE
mysecret-data        Opaque                           2      3h27m
myserver-tls-key     kubernetes.io/tls                2      145m
qbn-image-pull-key   kubernetes.io/dockerconfigjson   1      21s

##查看yaml文件##
root@deploy:~/k8s-Resource-N79/case11-secret# cat 5-secret-imagePull.yaml 
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myserver-myapp-frontend-deployment
  namespace: myserver
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myserver-myapp-frontend
  template:
    metadata:
      labels:
        app: myserver-myapp-frontend
    spec:
      containers:
      - name: myserver-myapp-frontend
        image: qbnharbor.top/baseimages/nginx:test            #指定你私有仓库的镜像地址
        imagePullPolicy: Always
        ports:
          - containerPort: 80
      imagePullSecrets:                           #指定镜像认证的secret
        - name: qbn-image-pull-key
---
apiVersion: v1
kind: Service
metadata:
  name: myserver-myapp-frontend
  namespace: myserver
spec:
  ports:
  - name: http
    port: 80
    targetPort: 80
    nodePort: 30018
    protocol: TCP
  type: NodePort
  selector:
    app: myserver-myapp-frontend

root@deploy:~/k8s-Resource-N79/case11-secret# kubectl apply -f 5-secret-imagePull.yaml 
deployment.apps/myserver-myapp-frontend-deployment created
service/myserver-myapp-frontend created
root@deploy:~/k8s-Resource-N79/case11-secret# kubectl get pod -n myserver
NAME                                                  READY   STATUS    RESTARTS   AGE
myserver-myapp-app1-deployment-6fd74bfbb9-fhbw2       1/1     Running   0          3h13m
myserver-myapp-frontend-deployment-69d9c54d5d-tvtqw   1/1     Running   0          16s

root@deploy:~/k8s-Resource-N79/case11-secret# kubectl describe pod -n myserver myserver-myapp-frontend-deployment-69d9c54d5d-tvtqw
  Normal  Scheduled  3m18s  default-scheduler  Successfully assigned myserver/myserver-myapp-frontend-deployment-69d9c54d5d-tvtqw to k8s-node3
  Normal  Pulling    3m17s  kubelet            Pulling image "qbnharbor.top/baseimages/nginx:test"
  Normal  Pulled     3m6s   kubelet            Successfully pulled image "qbnharbor.top/baseimages/nginx:test" in 11.899934042s (11.899957462s including waiting)
  Normal  Created    3m5s   kubelet            Created container myserver-myapp-frontend
  Normal  Started    3m5s   kubelet            Started container myserver-myapp-frontend

##查看yaml文件##
root@deploy:~/k8s-Resource-N79/case12-Statefulset# cat 1-Statefulset.yaml 
---
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
kind: StatefulSet 
metadata:
  name: myserver-myapp
  namespace: myserver
spec:
  replicas: 3
  serviceName: "myserver-myapp-service"
  selector:
    matchLabels:
      app: myserver-myapp-frontend
  template:
    metadata:
      labels:
        app: myserver-myapp-frontend
    spec:
      containers:
      - name: myserver-myapp-frontend
        #image: registry.cn-qingdao.aliyuncs.com/zhangshijie/zookeeper:v3.4.14
        image: registry.cn-hangzhou.aliyuncs.com/zhangshijie/tomcat-app1:v1
        ports:
          - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: myserver-myapp-service
  namespace: myserver
spec:
  clusterIP: None
  ports:
  - name: http
    port: 80
  selector:
    app: myserver-myapp-frontend

root@deploy:~/k8s-Resource-N79/case12-Statefulset# kubectl apply -f 1-Statefulset.yaml 
statefulset.apps/myserver-myapp created
service/myserver-myapp-service created
root@deploy:~/k8s-Resource-N79/case12-Statefulset# kubectl get pod -n myserver
NAME               READY   STATUS    RESTARTS   AGE
myserver-myapp-0   1/1     Running   0          4m13s
myserver-myapp-1   1/1     Running   0          4m10s
myserver-myapp-2   1/1     Running   0          4m7s

##修改yaml文件的镜像##
root@deploy:~/k8s-Resource-N79/case13-DaemonSet# vi 1-DaemonSet-webserver.yaml
....
- name: myserver-myapp-frontend
        image: registry.cn-qingdao.aliyuncs.com/zhangshijie/zookeeper:v3.4.14
        #image: registry.cn-hangzhou.aliyuncs.com/zhangshijie/tomcat-app1:v1
        ports:
          - containerPort: 80
....
root@deploy:~/k8s-Resource-N79/case12-Statefulset# kubectl apply -f 1-Statefulset.yaml 
statefulset.apps/myserver-myapp configured
service/myserver-myapp-service unchanged
root@deploy:~/k8s-Resource-N79/case12-Statefulset# kubectl get pod -n myserver
NAME               READY   STATUS              RESTARTS   AGE
myserver-myapp-0   1/1     Running             0          91s
myserver-myapp-1   0/1     ContainerCreating   0          2s
root@deploy:~/k8s-Resource-N79/case12-Statefulset# kubectl get pod -n myserver
NAME               READY   STATUS              RESTARTS   AGE
myserver-myapp-0   0/1     ContainerCreating   0          3s
myserver-myapp-1   1/1     Running             0          101s
root@deploy:~/k8s-Resource-N79/case12-Statefulset# kubectl get pod -n myserver
NAME               READY   STATUS    RESTARTS   AGE
myserver-myapp-0   1/1     Running   0          107s
myserver-myapp-1   1/1     Running   0          3m25s

root@deploy:~/k8s-Resource-N79/case12-Statefulset# kubectl apply -f 1-Statefulset.yaml 
statefulset.apps/myserver-myapp configured
service/myserver-myapp-service unchanged
root@deploy:~/k8s-Resource-N79/case12-Statefulset# kubectl get pod -n myserver
NAME               READY   STATUS              RESTARTS   AGE
myserver-myapp-0   1/1     Running             0          34m
myserver-myapp-1   1/1     Running             0          34m
myserver-myapp-2   1/1     Running             0          34m
myserver-myapp-3   0/1     ContainerCreating   0          7s
root@deploy:~/k8s-Resource-N79/case12-Statefulset# kubectl get pod -n myserver
NAME               READY   STATUS    RESTARTS   AGE
myserver-myapp-0   1/1     Running   0          34m
myserver-myapp-1   1/1     Running   0          34m
myserver-myapp-2   1/1     Running   0          34m
myserver-myapp-3   1/1     Running   0          13s
myserver-myapp-4   1/1     Running   0          6s

root@deploy:~/k8s-Resource-N79/case12-Statefulset# kubectl apply -f 1-Statefulset.yaml
statefulset.apps/myserver-myapp configured
service/myserver-myapp-service unchanged
root@deploy:~/k8s-Resource-N79/case12-Statefulset# kubectl get pod -n myserver
NAME               READY   STATUS        RESTARTS   AGE
myserver-myapp-0   1/1     Running       0          38m
myserver-myapp-1   1/1     Running       0          38m
myserver-myapp-2   1/1     Running       0          38m
myserver-myapp-3   1/1     Running       0          4m26s
myserver-myapp-4   1/1     Terminating   0          4m19s
root@deploy:~/k8s-Resource-N79/case12-Statefulset# kubectl get pod -n myserver
NAME               READY   STATUS        RESTARTS   AGE
myserver-myapp-0   1/1     Running       0          39m
myserver-myapp-1   1/1     Running       0          39m
myserver-myapp-2   1/1     Running       0          38m
myserver-myapp-3   1/1     Terminating   0          5m1s
root@deploy:~/k8s-Resource-N79/case12-Statefulset# kubectl get pod -n myserver
NAME               READY   STATUS        RESTARTS   AGE
myserver-myapp-0   1/1     Running       0          39m
myserver-myapp-1   1/1     Running       0          39m
myserver-myapp-2   1/1     Terminating   0          39m
root@deploy:~/k8s-Resource-N79/case12-Statefulset# kubectl get pod -n myserver
NAME               READY   STATUS    RESTARTS   AGE
myserver-myapp-0   1/1     Running   0          40m
myserver-myapp-1   1/1     Running   0          40m

----查看yaml文件----
root@deploy:~/k8s-Resource-N79/case13-DaemonSet# cat 2-DaemonSet-fluentd.yaml 
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-elasticsearch
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
spec:
  selector:
    matchLabels:
      name: fluentd-elasticsearch
  template:
    metadata:
      labels:
        name: fluentd-elasticsearch
    spec:
      tolerations:
      # this toleration is to have the daemonset runnable on master nodes
      # remove it if your masters can't run pods
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      containers:
      - name: fluentd-elasticsearch
        image: quay.io/fluentd_elasticsearch/fluentd:v2.5.2
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

root@deploy:~/k8s-Resource-N79/case13-DaemonSet# kubectl apply -f 2-DaemonSet-fluentd.yaml 
daemonset.apps/fluentd-elasticsearch created
root@deploy:~/k8s-Resource-N79/case13-DaemonSet# kubectl get pod -A -o wide | grep fluentd
kube-system            fluentd-elasticsearch-6sw5r                  1/1     Running   0               4m6s    10.200.169.152    k8s-node2     <none>           <none>
kube-system            fluentd-elasticsearch-fqh2q                  1/1     Running   0               4m6s    10.200.219.234    k8s-node3     <none>           <none>
kube-system            fluentd-elasticsearch-hmwbt                  1/1     Running   0               4m6s    10.200.135.195    k8s-master3   <none>           <none>
kube-system            fluentd-elasticsearch-mq5bx                  1/1     Running   0               4m6s    10.200.159.130    k8s-master1   <none>           <none>
kube-system            fluentd-elasticsearch-sq4bb                  1/1     Running   0               4m6s    10.200.224.3      k8s-master2   <none>           <none>
kube-system            fluentd-elasticsearch-tp8wd                  1/1     Running   0               4m6s    10.200.36.85      k8s-node1     <none>           <none>

----查看yaml文件----
root@deploy:~/k8s-Resource-N79/case13-DaemonSet# cat 3-DaemonSet-prometheus.yaml 
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring 
  labels:
    k8s-app: node-exporter
spec:
  selector:
    matchLabels:
        k8s-app: node-exporter
  template:
    metadata:
      labels:
        k8s-app: node-exporter
    spec:
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
      containers:
      - image: prom/node-exporter:v1.3.1 
        imagePullPolicy: IfNotPresent
        name: prometheus-node-exporter
        ports:
        - containerPort: 9100
          hostPort: 9100
          protocol: TCP
          name: metrics
        volumeMounts:
        - mountPath: /host/proc
          name: proc
        - mountPath: /host/sys
          name: sys
        - mountPath: /host
          name: rootfs
        args:
        - --path.procfs=/host/proc
        - --path.sysfs=/host/sys
        - --path.rootfs=/host
      volumes:
        - name: proc
          hostPath:
            path: /proc
        - name: sys
          hostPath:
            path: /sys
        - name: rootfs
          hostPath:
            path: /
      hostNetwork: true
      hostPID: true
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/scrape: "true"
  labels:
    k8s-app: node-exporter
  name: node-exporter
  namespace: monitoring 
spec:
  type: NodePort
  ports:
  - name: http
    port: 9100
    nodePort: 39100
    protocol: TCP
  selector:
    k8s-app: node-exporter

root@deploy:~/k8s-Resource-N79/case13-DaemonSet# kubectl apply -f 3-DaemonSet-prometheus.yaml 
daemonset.apps/node-exporter created
service/node-exporter created
root@deploy:~/k8s-Resource-N79/case13-DaemonSet# kubectl get pod -A -o wide | grep node-exporter
monitoring             node-exporter-28rc2                          1/1     Running   0               76s     192.168.100.153   k8s-node3     <none>           <none>
monitoring             node-exporter-cwlcq                          1/1     Running   0               76s     192.168.100.152   k8s-node2     <none>           <none>
monitoring             node-exporter-kg6nw                          1/1     Running   0               76s     192.168.100.102   k8s-master2   <none>           <none>
monitoring             node-exporter-m9xpt                          1/1     Running   0               76s     192.168.100.103   k8s-master3   <none>           <none>
monitoring             node-exporter-q9pns                          1/1     Running   0               76s     192.168.100.151   k8s-node1     <none>           <none>
monitoring             node-exporter-qhpm5                          1/1     Running   0               76s     192.168.100.101   k8s-master1   <none>           <none>

etcd 快照是全局完成备份(类似于MySQL全部备份),即使需要恢复一个资源对象(类似于只恢复MySQL的一个库),但是也需要做全局恢复到备份的状态(类似于MySQL的全库恢复),即会影响其它namespace中pod运行服务(类似于会影响MySQL其它数据库的数据)。
Velero可以有针对性的备份,比如按照namespace单独备份、只备份单独的资源对象等,在恢复的时候可以根据备份只恢复单独的namespace或资源对象，而不影响其它namespace中pod运行服务。
velero支持ceph、oss等对象存储，etcd 快照是一个为本地文件。
velero支持任务计划实现周期备份，但etcd 快照也可以基于cronjob实现。
velero支持对AWS EBS创建快照及还原
https://www.qloudx.com/velero-for-kubernetes-backup-restore-stateful-workloads-with-aws-ebs-snapshots/
https://github.com/vmware-tanzu/velero-plugin-for-aws #Elastic Block Store

~#velero backup create myserver-ns-backup-${DATE} --include-namespaces myserver --kubeconfig=./awsuser.kubeconfig -- namespace velero-syste
1.Velero 客户端调用Kubernetes API Server创建Backup任务。
2.Backup 控制器基于watch 机制通过API Server获取到备份任务。
3.Backup 控制器开始执行备份动作，其会通过请求API Server获取需要备份的数据。
4.Backup 控制器将获取到的数据备份到指定的对象存储server端。

创建数据目录：
root@k8s-deploy:~# docker pull registry.cn-hangzhou.aliyuncs.com/zhangshijie/minio:RELEASE.2022-04-12T06-55-35Z
root@k8s-deploy:/usr/local/src# docker pull minio/minio:RELEASE.2022-04-12T06-55-35Z
root@k8s-deploy:/usr/local/src# mkdir -p /data/minio
#创建minio容器,如果不指定,则默认用户名与密码为 minioadmin/minioadmin,可以通过环境变量自定义,如下:
root@k8s-deploy:/usr/local/src# docker run --name minio \
-p 9000:9000 \
-p 9999:9999 \
-d --restart=always \
-e "MINIO_ROOT_USER=admin" \
-e "MINIO_ROOT_PASSWORD=12345678" \
-v /data/minio/data:/data \
registry.cn-hangzhou.aliyuncs.com/zhangshijie/minio:RELEASE.2022-04-12T06-55-35Z server /data \
--console-address '0.0.0.0:9999

root@k8s-master1:/usr/local/src# wget https://github.com/vmware-tanzu/velero/releases/download/v1.11.1/velero-v1.11.1-linux-amd64.tar.gz
root@k8s-master1:/usr/local/src# tar xvf velero-v1.11.1-linux-amd64.tar.gz 
root@k8s-master1:/usr/local/src# cp velero-v1.11.1-linux-amd64/velero  /usr/local/bin/
root@k8s-master1:/usr/local/src# velero  --help
配置velero认证环境：
#工作目录：
root@k8s-master1:~# mkdir  /data/velero -p
root@k8s-master1:~# cd /data/velero
root@k8s-master1:/data/velero# 
#访问minio的认证文件：
root@k8s-master1:/data/velero# vim velero-auth.txt 
[default]
aws_access_key_id = admin
aws_secret_access_key = 12345678
 
#准备user-csr文件：
root@k8s-master1:/data/velero# vim awsuser-csr.json
{
  "CN"

root@k8s-master1:/data/velero# pwd
/data/velero
root@k8s-master1:/data/velero# DATE=
`date +%Y%m%d%H%M%S`
root@k8s-master1:/data/velero# velero backup create default-backup-${DATE} \
--include-cluster-resources=true \
--include-namespaces default \
--kubeconfig=./awsuser.kubeconfig \
--namespace velero-system
#验证备份
root@k8s-master1:/data/velero# velero backup describe default-backup-20230505160307 --kubeconfig=./awsuser.kubeconfig -- namespace velero-system

root@k8s-master1:/data/velero# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
net-test1 1/1 Running 0 22s 10.200.207.153 172.31.7.111 
root@k8s-master1:/data/velero# kubectl exec -it net-test1 bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead. [root@net-test1 /]# ping www.jd.com
PING wwwv6.jcloudimg.com (111.31.22.3) 56(84) bytes of data. 64 bytes from 111.31.22.3 (111.31.22.3): icmp_seq=1 ttl=127 time=32.4 ms
64 bytes from 111.31.22.3 (111.31.22.3): icmp_seq=2 ttl=127 time=36.4 ms

root@k8s-master1:/data/velero# pwd
/data/velero
root@k8s-master1:/data/velero# DATE=
`date +%Y%m%d%H%M%S`
root@k8s-master1:/data/velero# velero backup create myserver-ns-backup-${DATE} \
--include-cluster-resources=true \
--include-namespaces myserver \
--kubeconfig=/root/.kube/config \
--namespace velero-system

root@k8s-master1:/data/velero# kubectl delete deployment myserver-tomcat-app1-deployment -n myserver
deployment.apps "myserver-tomcat-app1-deployment" deleted
root@k8s-master1:/data/velero# kubectl delete service myserver-tomcat-app1-service -n myserver
service "myserver-tomcat-app1-service" deleted
root@k8s-master1:/data/velero# velero restore create --from-backup myserver-ns-backup-20230505161423 --wait \
--kubeconfig=./awsuser.kubeconfig \
--namespace velero-system

root@k8s-master1:/data/velero# kubectl run net-test1 --image=centos:7.9.2009 sleep 10000000000 -n myserver
pod/net-test1 created
root@k8s-master1:/data/velero# kubectl get pod -n myserver
NAME READY STATUS RESTARTS AGE
myserver-nginx-deployment-76f7df66c8-dq8k8 1/1 Running 0 19m
myserver-tomcat-app1-deployment-6bb596979f-gc2jw 1/1 Running 0 10m
net-test1 1/1 Running 0 24s
root@k8s-master1:/data/velero# kubectl get deployments -n myserver
NAME READY UP-TO-DATE AVAILABLE AGE
myserver-nginx-deployment 1/1 1 1 15m
myserver-tomcat-app1-deployment 1/1 1 1 6m12s
root@k8s-master1:/data/velero# velero backup create pod-backup-202308161135 --include-cluster-resources=true --ordered-resources
'pods=myserver/net-test1,defafut/net-test1' --namespace velero-system

root@k8s-master1:/data/velero# kubectl delete pod -n myserver net-test1
root@k8s-master1:/data/velero# kubectl get pod -n myserver
NAME READY STATUS RESTARTS AGE
myserver-nginx-deployment-76f7df66c8-dq8k8 1/1 Running 0 22m
root@k8s-master1:/data/velero# velero restore create --from-backup pod-backup-202308161135 --wait \
--kubeconfig=./awsuser.kubeconfig \
--namespace velero-system

root@k8s-master1:~# cat /data/velero/ns-back.sh 
#!/bin/bash 
NS_NAME=`kubectl get ns | awk '{if (NR>2){print}}' | awk '{print $1}'` DATE=`date +%Y%m%d%H%M%S` 
cd /data/velero/ 
for i in $NS_NAME;
do velero backup create ${i}-ns-backup-${DATE} \ 
--include-cluster-resources=true \ 
--include-namespaces ${i} \ 
--kubeconfig=/root/.kube/config \ 
--namespace velero-system done

root@k8s-master1:/data/velero# kubectl api-resources #可以查询要备份的数据类型
root@k8s-master1:/data/velero# kubectl get deployments -n test
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   2/2     2            2           2m42s
root@k8s-master1:/data/velero# DATE=`date +%Y%m%d%H%M%S`
root@k8s-master1:/data/velero# velero backup create pod-backup-${DATE} \
--include-cluster-resources=true \
--ordered-resources \
deployment=nginx-deployment \
--namespace velero-system
 
如果你还想备份多一个pod，比如test命名空间下的net-test1，上面的第五行可以这么写
deployment=nginx-deployment;pod=test/net-test1       #同类型资源用逗号隔开，不同资源类型用冒号

⾃动垃圾回收
⽀持并发依赖解析
⾼效的指令缓存
⽀持构建缓存的导⼊/导出
可扩展的前端格式-⽀持dockerfile、yaml等格式进⾏镜像构建
多种输镜像出格式-镜像、本地压缩包等
⽀持嵌套的构建作业调⽤
⽀持多种运⾏时
扩展性强
⽀持⽆root权限

root@k8s-master1:/usr/local/src# pwd
/usr/local/src
root@k8s-master1:/usr/local/src# wget https://github.com/moby/buildkit/releases/download/v0.12.1/buildkit-v0.12.1.linux-amd64.tar.gz
root@k8s-master1:/usr/local/src# tar xvf buildkit-v0.12.1.linux-amd64.tar.gz 
root@k8s-master1:/usr/local/src# mv bin/* /usr/local/bin/
root@k8s-master1:/usr/local/src# buildctl --help

root@k8s-master1:/usr/local/src# vim /lib/systemd/system/buildkit.socket
[Unit]
Description=BuildKit
Documentation=https://github.com/moby/buildkit
[Socket]
ListenStream=%t/buildkit/buildkitd.sock
[Install]
WantedBy=sockets.target

root@k8s-master1:/usr/local/src# vim /lib/systemd/system/buildkitd.service
[Unit]
Description=BuildKit
Requires=buildkit.socket
After=buildkit.socketDocumentation=https://github.com/moby/buildkit
[Service]
ExecStart=/usr/local/bin/buildkitd --oci-worker=false --
containerd-worker=true
[Install]

root@k8s-master1:/usr/local/src# systemctl daemon-reload
root@k8s-master1:/usr/local/src# systemctl enable buildkitd && systemctl restart buildkitd
root@k8s-master1:/usr/local/src# systemctl status buildkitd

root@k8s-master1:/usr/local/src# nerdctl login qbnharbor.top     #登录harbor
WARN[0000] skipping verifying HTTPS certs for "qbnharbor.top" 
WARNING: Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded

root@k8s-master1:~# nerdctl pull registry.cn-hangzhou.aliyuncs.com/zhangshijie/centos:7.9.2009
root@k8s-master1:~# nerdctl tag registry.cn-hangzhou.aliyuncs.com/zhangshijie/centos:7.9.2009 qbnharbor.top/baseimages/centos:7.9.2009
root@k8s-master1:~# nerdctl push qbnharbor.top/baseimages/centos:7.9.2009

root@k8s-master1:/opt/dockerfile/ubuntu2204-nginx-1.22# pwd
/opt/dockerfile/ubuntu2204-nginx-1.22
root@k8s-master1:/opt/dockerfile/ubuntu2204-nginx-1.22# ll
-rw-r--r-- 1 root root     931 Aug 17 12:00 Dockerfile
-rw-r--r-- 1 root root     284 Aug 28 11:12 build-command.sh
-rw-r--r-- 1 root root   38751 Aug  5  2022 frontend.tar.gz
drwxr-xr-x 3 root root    4096 Aug  5  2022 html/
-rw-r--r-- 1 root root    2812 Oct  3  2020 nginx.conf
-rw-r--r-- 1 root root    1139 Aug  5  2022 sources.list

root@k8s-master1:/opt/dockerfile/ubuntu/ubuntu2204-nginx-1.22# cat build-command.sh   
#!/bin/bash
#docker build -t harbor.linuxarchitect.io/baseimages/nginx:v1 .
#docker push harbor.linuxarchitect.io/baseimages/nginx:v1
/usr/local/bin/nerdctl build -t qbnharbor.top/baseimages/nginx-base:1.22.0 .
/usr/local/bin/nerdctl push qbnharbor.top/baseimages/nginx-base:1.22.0
 
root@k8s-master1:/opt/dockerfile/ubuntu/ubuntu2204-nginx-1.22#  bash build-command.sh

1，用户在master节点通过yaml文件描述对容器运行的要求，创建pod的要求
2，使用kubectl命令去创建，应用这个资源描述文件，kubectl会把这个创建pod的请求发给API server
3，API server开始生成pod对象信息，并把信息写入到etcd里，然后返回信息给客户端
4，API server开始反馈etcd中的pod变化，调度器scheduler通过watch机制跟踪到API server的变动，准备pod调度
5，scheduler通过获取的pod信息，开始调用内部算法机制为pod分配合适的节点，并将信息更新到API server
6，API server会将这个信息再次写入到etcd，etcd更新了信息（pod信息+绑定关系，部署到哪个节点），并把信息返回给API server
7，API server会将这个返回的信息远程通知对应node节点上的kubectl
8，node节点上的kubectl发现有pod调度过来，尝试调用运行时启动容器，并将结果反馈给API server
9，API server将收到的信息再次写入到etcd里，etcd更新了数据并返回给API server
10，API server再把信息发送给对应的node节点

Unschedulable：#Pod不能被调度，kube-scheduler没有匹配到合适的node节点
PodScheduled：#pod正处于调度中，在kube-scheduler刚开始调度的时候，还没有将pod分配到指定的node，在筛选出合适的节点后就会更新etcd数据，将pod分配到指定的node
Pending: #正在创建Pod但是Pod中的容器还没有全部被创建完成=[处于此状态的Pod应该检查Pod依赖的存储是否有权限挂载等。
Failed： #Pod中有容器启动失败而导致pod工作异常。
Unknown： #由于某种原因无法获得pod的当前状态，通常是由于与pod所在的node节点通信错误。
Initialized：#所有pod中的初始化容器已经完成了
ImagePullBackOff： #Pod所在的node节点下载镜像失败
Running： #Pod内部的容器已经被创建并且启动。
Ready： #表示pod中的容器已经可以提供访问服务
Error: #pod 启动过程中发生错误
NodeLost: #Pod 所在节点失联
Waiting: #Pod 等待启动
Terminating: #Pod 正在被销毁
CrashLoopBackOff：#pod崩溃，但是kubelet正在将它重启
InvalidImageName：#node节点无法解析镜像名称导致的镜像无法下载
ImageInspectError：#无法校验镜像，镜像不完整导致 
ErrImageNeverPull：#策略禁止拉取镜像，镜像中心权限是私有等 
RegistryUnavailable：#镜像服务器不可用，网络原因或harbor宕机 
ErrImagePull：#镜像拉取出错，超时或下载被强制终止 
CreateContainerConfigError：#不能创建kubelet使用的容器配置 
CreateContainerError：#创建容器失败 
RunContainerError：#pod运行失败，容器中没有初始化PID为1的守护进程等
ContainersNotInitialized：#pod没有初始化完毕 
ContainersNotReady：#pod没有准备完毕 
ContainerCreating：#pod正在创建中 
PodInitializing：#pod正在初始化中 
DockerDaemonNotReady：#node节点decker服务没有启动 
NetworkPluginNotReady：#网络插件没有启动

root@deploy:~# kubectl get pod -n test
NAME                                READY   STATUS    RESTARTS   AGE
net-test1                           1/1     Running   0          2d22h
#进入到容器
root@deploy:~# kubectl exec -it -n test net-test1  bash
[root@net-test1 /]# yum install -y ethtool
[root@net-test1 /]# ethtool -S eth0
NIC statistics:
     peer_ifindex: 44
     rx_queue_0_xdp_packets: 0
     rx_queue_0_xdp_bytes: 0
     rx_queue_0_xdp_drops: 0
#或者
[root@net-test1 /]# cat /sys/class/net/eth0/iflink 
44
#到pod所在宿主机验证网卡
root@k8s-node2:~# ip link show
....
44: cali1843beedbe9@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-fee13063-ad43-4c2e-eefd-c02f2d287e76
    root@k8s-node2:~# ls /run/netns/ -l
total 0
-r--r--r-- 1 root root 0 Aug 25 15:47 cni-31c67025-50d1-aca8-4177-2a149d16a4f1
-r--r--r-- 1 root root 0 Aug 21 09:43 cni-626d4ee8-bcb7-5f62-2fbc-252b6abe9fab
-r--r--r-- 1 root root 0 Aug 25 16:08 cni-8f23dc64-9451-24eb-41d0-64f4da5594f2
-r--r--r-- 1 root root 0 Aug 25 15:54 cni-edeb4bd6-e694-6e1e-f840-a60a6bc4b0d9
-r--r--r-- 1 root root 0 Aug 25 16:58 cni-fee13063-ad43-4c2e-eefd-c02f2d287e76

root@k8s-master1:/home/1.pause-test-case# cat nginx.conf 
error_log stderr;
events { worker_connections  1024; }
http {
    access_log /dev/stdout;
    server {
        listen 80 default_server;
        server_name www.mysite.com;
        location / {
          index index.html index.php;
          root /usr/share/nginx/html;
         }
        location ~ \.php$ {
            root           /usr/share/nginx/html;
            fastcgi_pass   127.0.0.1:9000;
            fastcgi_index  index.php;
            fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
             include        fastcgi_params;
             }
        }
}

root@k8s-master1:~# nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.8
root@k8s-master1:~# nerdctl run -d -p 80:80 --name pause-container-test registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.8
ca97fde43fe69e0bd48f601a55debede17597ed348793f55d3b5d7dec8ed7062

root@k8s-master1:/home/1.pause-test-case# ll html/
-rw-r--r-- 1 root root   34 Aug 28 15:42 index.html
-rw-r--r-- 1 root root   25 Aug 28 15:42 index.php

root@k8s-master1:/home/1.pause-test-case# nerdctl run -d --name nginx-containerd-test \
-v `pwd`/nginx.conf:/etc/nginx/nginx.conf \
-v `pwd`/html:/usr/share/nginx/html \
--net=container:pause-container-test \
registry.cn-hangzhou.aliyuncs.com/zhangshijie/nginx:1.22.0-alpine

root@k8s-master1:/home/1.pause-test-case# nerdctl run -d --name php-container-test \
--net=container:pause-container-test \
-v `pwd`/html:/usr/share/nginx/html \
registry.cn-hangzhou.aliyuncs.com/zhangshijie/php:5.6.40-fpm

root@k8s-master1:/home/1.pause-test-case# nerdctl ps
CONTAINER ID    IMAGE                                                                COMMAND                   CREATED           STATUS    PORTS                 NAMES
026bbec06e36    registry.cn-hangzhou.aliyuncs.com/zhangshijie/php:5.6.40-fpm         "docker-php-entrypoi…"    14 minutes ago    Up                              php-container-test
999d786403b8    registry.cn-hangzhou.aliyuncs.com/zhangshijie/nginx:1.22.0-alpine    "/docker-entrypoint.…"    17 minutes ago    Up                              nginx-containerd-test
ca97fde43fe6    registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.8        "/pause"                  52 minutes ago    Up        0.0.0.0:80->80/tcp    pause-container-test
root@k8s-master1:/home/1.pause-test-case# nerdctl exec -it 026bbec06e36 sh
# hostname -i
10.4.0.3
# exit
root@k8s-master1:/home/1.pause-test-case# nerdctl exec -it 999d786403b8 sh
/ # hostname -i
10.4.0.3

root@k8s-master1:/home/2.Probe-cases/case1-init-container# cat 1-init-container.yaml 
kind: Deployment
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
metadata:
  labels:
    app: myserver-myapp 
  name: myserver-myapp-deployment-name
  namespace: myserver
spec:
  replicas: 1 
  selector:
    matchLabels:
      app: myserver-myapp-frontend
  template:
    metadata:
      labels:
        app: myserver-myapp-frontend
    spec:
      containers:
        - name: myserver-myapp-container
          image: nginx:1.20.0 
          #imagePullPolicy: Always
          volumeMounts:
          - mountPath: "/usr/share/nginx/html/myserver"
            name: myserver-data
          - name: tz-config
            mountPath: /etc/localtime
      initContainers:
        - name: init-web-data
          image: centos:7.9.2009
          command: ['/bin/bash','-c',"for i in `seq 1 10`;do echo '<h1>'$i web page at $(date +%Y%m%d%H%M%S) '<h1>' >> /data/nginx/html/myserver/index.html;sleep 1;done"]
          volumeMounts:
          - mountPath: "/data/nginx/html/myserver"
            name: myserver-data
          - name: tz-config
            mountPath: /etc/localtime
        - name: change-data-owner
          image: busybox:1.28
          command: ['/bin/sh','-c',"/bin/chmod 644 /data/nginx/html/myserver/* -R"]
          volumeMounts:
          - mountPath: "/data/nginx/html/myserver"
            name: myserver-data
          - name: tz-config
            mountPath: /etc/localtime
      volumes:
      - name: myserver-data
        hostPath:
          path: /tmp/data/html
      - name: tz-config
        hostPath:
          path: /etc/localtime 
---
kind: Service
apiVersion: v1
metadata:
  labels:
    app: myserver-myapp-service
  name: myserver-myapp-service-name
  namespace: myserver
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    targetPort: 80
    nodePort: 30088
  selector:
    app: myserver-myapp-frontend

root@k8s-master1:/home/2.Probe-cases/case1-init-container# kubectl apply -f 1-init-container.yaml
root@k8s-master1:/home/2.Probe-cases/case1-init-container# kubectl get pod -A -o wide
NAMESPACE              NAME                                             READY   STATUS    RESTARTS        AGE     IP                NODE          NOMINATED NODE   READINESS GATES
myserver               myserver-myapp-deployment-name-695b5d665-28xfb   1/1     Running   0               4m49s   10.200.36.90      k8s-node1     <none>           <none>

root@deploy:~# kubectl get node | grep node
k8s-node1     Ready                      node     16d   v1.27.4
k8s-node2     Ready                      node     16d   v1.27.4
k8s-node3     Ready                      node     15d   v1.27.4

root@k8s-master1:/home/zookeeper-case-n79# kubectl get storageclasses.storage.k8s.io 
NAME                  PROVISIONER                                   RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
managed-nfs-storage   k8s-sigs.io/nfs-subdir-external-provisioner   Retain          Immediate           false                  31s

apiVersion: v1
kind: Service
metadata:
  name: zookeeper-headless
  labels:
    app: zookeeper
spec:
  ports:
  - port: 2888
    name: server
  - port: 3888
    name: leader-election
  clusterIP: None
  selector:
    app: zookeeper
---
apiVersion: v1
kind: Service
metadata:
  name: zookeeper-service

root@k8s-master1:/home/zookeeper-case-n79/1.StatefulSet# kubectl apply -f zk-cluster-StatefulSet.yaml 
service/zookeeper-headless created
service/zookeeper-service created
poddisruptionbudget.policy/zookeeper-pdb created
statefulset.apps/zookeeper created
----查看pod，此时只有一个pod，在node2节点----
root@k8s-master1:/home/zookeeper-case-n79/1.StatefulSet# kubectl get pod -o wide
NAME          READY   STATUS              RESTARTS   AGE   IP       NODE        NOMINATED NODE   READINESS GATES
zookeeper-0   0/1     ContainerCreating   0          24s   <none>   k8s-node2   <none>           <none>
----创建完第一个pod，才创建第二个pod，在node1节点----
root@k8s-master1:/home/zookeeper-case-n79/1.StatefulSet# kubectl get pod

----用yaml文件删除pod----
root@k8s-master1:/home/zookeeper-case-n79/1.StatefulSet# pwd
/home/zookeeper-case-n79/1.StatefulSet
root@k8s-master1:/home/zookeeper-case-n79/1.StatefulSet# kubectl delete -f zk-cluster-StatefulSet.yaml
service "zookeeper-headless" deleted
service "zookeeper-service" deleted
poddisruptionbudget.policy "zookeeper-pdb" deleted
statefulset.apps "zookeeper" deleted
root@k8s-master1:/home/zookeeper-case-n79/1.StatefulSet# kubectl get pod 
NAME          READY   STATUS        RESTARTS        AGE
zookeeper-0   1/1     Terminating   3 (40m ago)     4h
zookeeper-1   1/1     Terminating   0               3h58m
zookeeper-2   1/1     Terminating   1 (3h22m ago)   3h57m

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: zookeeper-datadir-pv-1
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce 
  nfs:
    server: 172.31.7.109
    path: /data/k8sdata/magedu/zookeeper-datadir-1 
 
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: zookeeper-datadir-pv-2
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    server: 172.31.7.109 
    path: /data/k8sdata/magedu/zookeeper-datadir-2 
 
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: zookeeper-datadir-pv-3
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    server: 172.31.7.109  
    path: /data/k8sdata/magedu/zookeeper-datadir-3

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: zookeeper-datadir-pvc-1
  namespace: magedu
spec:
  accessModes:
    - ReadWriteOnce
  volumeName: zookeeper-datadir-pv-1
  resources:
    requests:
      storage: 10Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: zookeeper-datadir-pvc-2
  namespace: magedu
spec:
  accessModes:
    - ReadWriteOnce
  volumeName: zookeeper-datadir-pv-2
  resources:
    requests:
      storage: 10Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: zookeeper-datadir-pvc-3
  namespace: magedu
spec:
  accessModes:
    - ReadWriteOnce
  volumeName: zookeeper-datadir-pv-3
  resources:
    requests:
      storage: 10Gi

apiVersion: v1
kind: Service
metadata:
  name: zookeeper
  namespace: magedu
spec:
  ports:
    - name: client
      port: 2181
  selector:
    app: zookeeper
---
apiVersion: v1
kind: Service
metadata:
  name: zookeeper1
  namespace: magedu
spec:
  type: NodePort        
  ports:

----修改yaml文件，替换镜像仓库地址为你打的镜像地址，注意一共有三个地方---
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/zookeeper# vi zookeeper.yaml   
....
containers:
     - name: server
       image: qbnharbor.top/magedu/zookeeper:v3.4.14           
       imagePullPolicy: Always
....     
----修改创建pv的yaml文件，修改nfs存储的地址----
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/zookeeper/pv# vi zookeeper-persistentvolume.yaml
....
    server: 192.168.100.109
    path: /data/k8sdata/magedu/zookeeper-datadir-1
....
    server: 192.168.100.109
    path: /data/k8sdata/magedu/zookeeper-datadir-1
....
    server: 192.168.100.109
    path: /data/k8sdata/magedu/zookeeper-datadir-1
....
----在nfs创建对应的目录----
root@k8s-haproxy1:~# mkdir -p /data/k8sdata/magedu/zookeeper-datadir-1
root@k8s-haproxy1:~# mkdir -p /data/k8sdata/magedu/zookeeper-datadir-2
root@k8s-haproxy1:~# mkdir -p /data/k8sdata/magedu/zookeeper-datadir-3
root@k8s-haproxy1:~# ll /data/k8sdata/magedu/
drwxr-xr-x 2 root root 4096 Aug 30 15:26 zookeeper-datadir-1/
drwxr-xr-x 2 root root 4096 Aug 30 15:26 zookeeper-datadir-2/
drwxr-xr-x 2 root root 4096 Aug 30 15:26 zookeeper-datadir-3/

root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/zookeeper/pv# kubectl apply -f zookeeper-persistentvolume.yaml
persistentvolume/zookeeper-datadir-pv-1 created
persistentvolume/zookeeper-datadir-pv-2 created
persistentvolume/zookeeper-datadir-pv-3 created
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/zookeeper/pv# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                                 STORAGECLASS          REASON   AGE
zookeeper-datadir-pv-1                     20Gi       RWO            Retain           Available                                                                        40s
zookeeper-datadir-pv-2                     20Gi       RWO            Retain           Available                                                                        40s
zookeeper-datadir-pv-3                     20Gi       RWO            Retain           Available                                                                        40s

root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/zookeeper/pv# kubectl create namespace magedu
namespace/magedu created
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/zookeeper/pv# kubectl apply -f  zookeeper-persistentvolumeclaim.yaml
persistentvolumeclaim/zookeeper-datadir-pvc-1 created
persistentvolumeclaim/zookeeper-datadir-pvc-2 created
persistentvolumeclaim/zookeeper-datadir-pvc-3 created
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/zookeeper/pv# kubectl get pvc -n magedu
NAME                      STATUS   VOLUME                   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
zookeeper-datadir-pvc-1   Bound    zookeeper-datadir-pv-1   20Gi       RWO                           50s
zookeeper-datadir-pvc-2   Bound    zookeeper-datadir-pv-2   20Gi       RWO                           50s
zookeeper-datadir-pvc-3   Bound    zookeeper-datadir-pv-3   20Gi       RWO                           50s

root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/zookeeper# kubectl apply -f zookeeper.yaml 
service/zookeeper created
service/zookeeper1 created
service/zookeeper2 created
service/zookeeper3 created
deployment.apps/zookeeper1 created
deployment.apps/zookeeper2 created
deployment.apps/zookeeper3 created
----这里的pod由于没有设置反亲和，所以可能在用一个node节点----
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/zookeeper# kubectl get pod -n magedu -o wide
NAME                          READY   STATUS    RESTARTS     AGE   IP               NODE        NOMINATED NODE   READINESS GATES
zookeeper1-85b78848bf-d8h5s   1/1     Running   0            16s   10.200.169.190   k8s-node2   <none>           <none>
zookeeper2-69c994774b-qrkd6   1/1     Running   1 (9s ago)   16s   10.200.169.189   k8s-node2   <none>           <none>
zookeeper3-7cfcfd89dc-5rcmf   1/1     Running   0            16s   10.200.219.243   k8s-node3   <none>           <none>

root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/zookeeper# kubectl exec -it -n magedu zookeeper1-85b78848bf-d8h5s bash
bash-4.3# /zookeeper/bin/zkServer.sh status
Mode: follower
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/zookeeper# kubectl exec -it -n magedu zookeeper2-69c994774b-qrkd6 bash
bash-4.3# /zookeeper/bin/zkServer.sh status
Mode: leader
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/zookeeper# kubectl exec -it -n magedu zookeeper3-7cfcfd89dc-5rcmf bash
bash-4.3# /zookeeper/bin/zkServer.sh status
Mode: follower

root@k8s-haproxy1:/data/k8sdata/magedu# ll
drwxr-xr-x 3 root root 4096 Aug 30 15:37 zookeeper-datadir-1/
drwxr-xr-x 3 root root 4096 Aug 30 15:44 zookeeper-datadir-2/
drwxr-xr-x 3 root root 4096 Aug 30 15:44 zookeeper-datadir-3/

----删除pod----
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/zookeeper# kubectl delete -f zookeeper.yaml 
service "zookeeper" deleted
service "zookeeper1" deleted
service "zookeeper2" deleted
service "zookeeper3" deleted
deployment.apps "zookeeper1" deleted
deployment.apps "zookeeper2" deleted
deployment.apps "zookeeper3" deleted
----删除pvc----
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/zookeeper# cd pv
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/zookeeper/pv# kubectl delete -f zookeeper-persistentvolumeclaim.yaml 
persistentvolumeclaim "zookeeper-datadir-pvc-1" deleted
persistentvolumeclaim "zookeeper-datadir-pvc-2" deleted
persistentvolumeclaim "zookeeper-datadir-pvc-3" deleted
----删除pv----
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/zookeeper/pv# kubectl delete -f zookeeper-persistentvolume.yaml 
persistentvolume "zookeeper-datadir-pv-1" deleted
persistentvolume "zookeeper-datadir-pv-2" deleted
persistentvolume "zookeeper-datadir-pv-3" deleted
---验证是否删除干净---
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/zookeeper/pv# kubectl get pod -n magedu
No resources found in magedu namespace.
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/zookeeper/pv# kubectl get pv -n magedu
No resources found in magedu namespace.
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/zookeeper/pv# kubectl get pvc -n magedu
No resources found in magedu namespace.

----构建基础镜像的Dockerfile----
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/system/centos# cat Dockerfile 
#自定义Centos 基础镜像
FROM centos:7.9.2009 
MAINTAINER Jack.Zhang  2973707860@qq.com
ADD filebeat-7.12.1-x86_64.rpm /tmp
RUN yum install -y /tmp/filebeat-7.12.1-x86_64.rpm vim wget tree  lrzsz gcc gcc-c++ automake pcre pcre-devel zlib zlib-devel openssl openssl-devel iproute net-tools iotop &&  rm -rf /etc/localtime /tmp/filebeat-7.12.1-x86_64.rpm && ln -snf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime  && useradd nginx -u 2088

----构建脚本以及上传到harbor---
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/system/centos# cat build-command.sh 
#!/bin/bash
#docker build -t  harbor.magedu.net/baseimages/magedu-centos-base:7.9.2009 .
#docker push harbor.magedu.net/baseimages/magedu-centos-base:7.9.2009
/usr/local/bin/nerdctl build -t qbnharbor.top/baseimages/magedu-centos-base:7.9.2009 .
/usr/local/bin/nerdctl push qbnharbor.top/baseimages/magedu-centos-base:7.9.2009

----启动脚本，构建镜像----
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/system/centos# cat build-command.sh 
#!/bin/bash
#docker build -t  harbor.magedu.net/baseimages/magedu-centos-base:7.9.2009 .
#docker push harbor.magedu.net/baseimages/magedu-centos-base:7.9.2009
/usr/local/bin/nerdctl build -t qbnharbor.top/baseimages/magedu-centos-base:7.9.2009 .
/usr/local/bin/nerdctl push qbnharbor.top/baseimages/magedu-centos-base:7.9.2009
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/system/centos# bash build-command.sh 
 WARN[0000] skipping verifying HTTPS certs for "qbnharbor.top" 
manifest-sha256:cb42d95fb98de3728dda0ea1573d1b4a08a3185e40c5f424b562fd40ee457cf3: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:67253723ff33a0008d56715c4422c94c6dd1e32c7c0fad08296d7efa9ec26107:   done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 21.6s                                                                    total:  3.3 Ki (155.0 B/s)

----镜像地址改为刚刚构建的基础镜像地址----
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/web/pub-images/jdk-1.8.212# cat Dockerfile 
#JDK Base Image
FROM qbnharbor.top/baseimages/magedu-centos-base:7.9.2009 
#FROM centos:7.9.2009
MAINTAINER zhangshijie "zhangshijie@magedu.net"
ADD jdk-8u212-linux-x64.tar.gz /usr/local/src/
RUN ln -sv /usr/local/src/jdk1.8.0_212 /usr/local/jdk 
ADD profile /etc/profile
ENV JAVA_HOME /usr/local/jdk
ENV JRE_HOME $JAVA_HOME/jre
ENV CLASSPATH $JAVA_HOME/lib/:$JRE_HOME/lib/
ENV PATH $PATH:$JAVA_HOME/bin
----注意harbor创建/pub-images项目----
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/web/pub-images/jdk-1.8.212# cat

root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/web/pub-images/tomcat-base-8.5.43# ls
Dockerfile  apache-tomcat-8.5.43.tar.gz  build-command.sh
----以JDK环境镜像为基础镜像构建tomcat镜像----
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/web/pub-images/tomcat-base-8.5.43# cat Dockerfile 
#Tomcat 8.5.43基础镜像
FROM qbnharbor.top/pub-images/jdk-base:v8.212
MAINTAINER zhangshijie "zhangshijie@magedu.net"
RUN mkdir /apps /data/tomcat/webapps /data/tomcat/logs -pv 
ADD apache-tomcat-8.5.43.tar.gz  /apps
RUN useradd tomcat -u 2050 && ln -sv /apps/apache-tomcat-8.5.43 /apps/tomcat && chown -R tomcat.tomcat /apps /data -R
 
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/web/pub-images/tomcat-base-8.5.43# cat build-command.sh 
#!/bin/bash
#docker build -t qbnharbor.top/pub-images/tomcat-base:v8.5.43  .
#sleep 3
#docker push  qbnharbor.top/pub-images/tomcat-base:v8.5.43
nerdctl  build -t qbnharbor.top/pub-images/tomcat-base:v8.5.43  .
nerdctl  push qbnharbor.top/pub-images/tomcat-base:v8.5.43
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/web/pub-images/tomcat-base-8.5.43# bash build-command.sh
WARN[0000] skipping verifying HTTPS certs for "qbnharbor.top" 
manifest-sha256:7a653ec766f95fcaff9bb6141d011d8d98f5e5282895f77696b7310fabb1b46c: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:cad28b4370c419b109d6fe008b6a330379ac5833cd8973e2604a68149b50a498:   done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 1.7 s                                                                    total:  7.0 Ki (4.1 KiB/s)                                       
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/web/pub-images/tomcat-base-8.5.43#

root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/web/magedu/tomcat-app1# ls
Dockerfile  app1.tar.gz  build-command.sh  catalina.sh  filebeat-7.5.1-x86_64.rpm  filebeat.yml  myapp  run_tomcat.sh  server.xml
 
----以tomcat镜像为基础镜像构建业务镜像----
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/web/magedu/tomcat-app1# cat Dockerfile 
#tomcat web1
FROM qbnharbor.top/pub-images/tomcat-base:v8.5.43 
ADD catalina.sh /apps/tomcat/bin/catalina.sh
ADD server.xml /apps/tomcat/conf/server.xml
#ADD myapp/* /data/tomcat/webapps/myapp/
ADD app1.tar.gz /data/tomcat/webapps/app1/
ADD run_tomcat.sh /apps/tomcat/bin/run_tomcat.sh
#ADD filebeat.yml /etc/filebeat/filebeat.yml 
RUN chown  -R nginx.nginx /data/ /apps/
#ADD filebeat-7.5.1-x86_64.rpm /tmp/
#RUN cd /tmp && yum localinstall -y filebeat-7.5.1-amd64.deb
EXPOSE 8080 8443
CMD ["/apps/tomcat/bin/run_tomcat.sh"]
 
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/web/magedu/tomcat-app1# cat build-command.sh 
#!/bin/bash
TAG=$1
#docker build -t  qbnharbor.top/magedu/tomcat-app1:${TAG} .
#sleep 3
#docker push  qbnharbor.top/magedu/tomcat-app1:${TAG}
nerdctl build -t  qbnharbor.top/magedu/tomcat-app1:${TAG} .
nerdctl push  qbnharbor.top/magedu/tomcat-app1:${TAG}
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/web/magedu/tomcat-app1#
 
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/web/magedu/tomcat-app1# bash build-command.sh v1
WARN[0000] skipping verifying HTTPS certs for "qbnharbor.top" 
manifest-sha256:c377a01ed1760be234e9748cfcb833636ddfcaeea978f25ad6169236e3544d34: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:8e740827f8016a1f5472bf008c57496a38c70ef4ff09d57a0ecac8350e5f3d2b:   done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 2.9 s                                                                    total:  9.5 Ki (3.3 KiB/s)

kind: Deployment
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
metadata:
  labels:
    app: magedu-tomcat-app1-deployment-label
  name: magedu-tomcat-app1-deployment
  namespace: magedu
spec:
  replicas: 1
  selector:
    matchLabels:
      app: magedu-tomcat-app1-selector
  template:
    metadata:
      labels:
        app: magedu-tomcat-app1-selector
    spec:
      containers:
      - name: magedu-tomcat-app1-container
        image: harbor.linuxarchitect.io/magedu/tomcat-app1:v1 
        #command: ["/apps/tomcat/bin/run_tomcat.sh"]
        imagePullPolicy: IfNotPresent
        #imagePullPolicy: Always
        ports:
        - containerPort: 8080
          protocol: TCP
          name: http
        env:
        - name: "password"
          value: "123456"
        - name: "age"
          value: "18"
        #resources:
        #  limits:
        #    cpu: 1
        #    memory: "512Mi"
        #  requests:
        #    cpu: 500m
        #    memory: "512Mi"
        volumeMounts:
        - name: magedu-images
          mountPath: /data/webapp/images
          readOnly: false
        - name: magedu-static
          mountPath: /data/webapp/static
          readOnly: false
      volumes:
      - name: magedu-images
        nfs:
          server: 172.31.7.109
          path: /data/k8sdata/magedu/images
      - name: magedu-static
        nfs:
          server: 172.31.7.109
          path: /data/k8sdata/magedu/static
#      nodeSelector:
#        project: magedu
#        app: tomcat
---
kind: Service
apiVersion: v1
metadata:
  labels:
    app: magedu-tomcat-app1-service-label
  name: magedu-tomcat-app1-service
  namespace: magedu
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
    nodePort: 30092
  selector:
    app: magedu-tomcat-app1-selector

----修改yaml文件的nfs地址以及镜像仓库地址----
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/tomcat-app1# vi tomcat-app1.yaml
....
    spec:
      containers:
      - name: magedu-tomcat-app1-container
        image: qbnharbor.top/magedu/tomcat-app1:v1
....
      volumes:
      - name: magedu-images
        nfs:
          server: 192.168.100.109
          path: /data/k8sdata/magedu/images
      - name: magedu-static
        nfs:
          server: 192.168.100.109
          path: /data/k8sdata/magedu/static
....
spec:
  type: NodePort     
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
    nodePort: 30092
  selector:
    app: magedu-tomcat-app1-selector
--------------------------------------------------------
注意！！！这里的yaml文件开启NodePort是为了测试访问，在公司里的后端服务是不会开启NodePort的

----创建nfs目录----
root@k8s-haproxy1:/data/k8sdata/magedu# mkdir -p /data/k8sdata/magedu/images
root@k8s-haproxy1:/data/k8sdata/magedu# mkdir -p /data/k8sdata/magedu/static
root@k8s-haproxy1:/data/k8sdata/magedu# ll /data/k8sdata/magedu/
drwxr-xr-x 2 root root 4096 Aug 30 17:17 images/
drwxr-xr-x 2 root root 4096 Aug 30 17:18 static/

root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/tomcat-app1# kubectl apply -f tomcat-app1.yaml 
deployment.apps/magedu-tomcat-app1-deployment created
service/magedu-tomcat-app1-service created
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/tomcat-app1# kubectl get pod -n magedu
NAME                                             READY   STATUS              RESTARTS   AGE
magedu-tomcat-app1-deployment-5965b8d944-sr2w5   0/1     ContainerCreating   0          41s
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/tomcat-app1# kubectl get pod -n magedu -o wide
NAME                                             READY   STATUS    RESTARTS   AGE     IP               NODE        NOMINATED NODE   READINESS GATES
magedu-tomcat-app1-deployment-5965b8d944-sr2w5   1/1     Running   0          3m33s   10.200.169.191   k8s-node2   <none>           <none>
----测试业务访问----
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/tomcat-app1# curl 192.168.100.152:30092/app1/index.html
tomcat app1 for magedu

root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/web/pub-images/nginx-base# ls
Dockerfile  build-command.sh  nginx-1.22.0.tar.gz
 
----以baseimages为基础镜像构建nginx基础镜像----
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/web/pub-images/nginx-base# cat Dockerfile 
#Nginx Base Image
FROM qbnharbor.top/baseimages/magedu-centos-base:7.9.2009 
MAINTAINER  zhangshijie@magedu.net
RUN yum install -y vim wget tree  lrzsz gcc gcc-c++ automake pcre pcre-devel zlib zlib-devel openssl openssl-devel iproute net-tools iotop
ADD nginx-1.22.0.tar.gz /usr/local/src/
RUN cd /usr/local/src/nginx-1.22.0 && ./configure  && make && make install && ln -sv  /usr/local/nginx/sbin/nginx /usr/sbin/nginx  &&rm -rf /usr/local/src/nginx-1.22.0.tar.gz 
 
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/web/pub-images/nginx-base# cat build-command.sh 
#!/bin/bash
#docker build -t harbor.magedu.net/pub-images/nginx-base:v1.18.0  .
#sleep 1
#docker push  harbor.magedu.net/pub-images/nginx-base:v1.18.0
nerdctl build -t  qbnharbor.top/pub-images/nginx-base:v1.22.0  .
nerdctl push qbnharbor.top/pub-images/nginx-base:v1.22.0
 
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/web/pub-images/nginx-base# bash build-command.sh 
WARN[0000] skipping verifying HTTPS certs for "qbnharbor.top" 
manifest-sha256:925e83f64e76df2b2ecfbacb8322c52029fe02d1609259490102650acfe860f4: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:0c009f75a8f34d9f8dc7f6609a1798bac28469126340c35fd57e0aef74961bec:   done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 1.4 s                                                                    total:  5.0 Ki (3.5 KiB/s)

root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/web/magedu/nginx# ls
Dockerfile  app1.tar.gz  build-command.sh  index.html  nginx.conf  webapp
 
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/web/magedu/nginx# cat Dockerfile 
#Nginx 1.22.0
FROM qbnharbor.top/pub-images/nginx-base:v1.22.0 
ADD nginx.conf /usr/local/nginx/conf/nginx.conf
ADD app1.tar.gz  /usr/local/nginx/html/webapp/
ADD index.html  /usr/local/nginx/html/index.html
#静态资源挂载路径
RUN mkdir -p /usr/local/nginx/html/webapp/static /usr/local/nginx/html/webapp/images 
EXPOSE 80 443
CMD ["nginx"] 
 
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/web/magedu/nginx# cat build-command.sh 
#!/bin/bash
TAG=$1
#docker build -t qbnharbor.top/magedu/nginx-web1:${TAG} .
#echo "镜像构建完成，即将上传到harbor"
#sleep 1
#docker push qbnharbor.top/magedu/nginx-web1:${TAG}
#echo "镜像上传到harbor完成"
nerdctl build -t qbnharbor.top/magedu/nginx-web1:${TAG} .
nerdctl push qbnharbor.top/magedu/nginx-web1:${TAG}
 
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/web/magedu/nginx# bash build-command.sh v1
WARN[0000] skipping verifying HTTPS certs for "qbnharbor.top" 
manifest-sha256:00bca2c36e2c569e630e8a43641dc8b0b867a95a97d2c62e0885a2beeae9ca86: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:b33a5d55a01cd3ea22214bd6ac1d3f14cb56f66299fc8f8b80821dbd40b988b9:   done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 2.2 s                                                                    total:  7.0 Ki (3.2 KiB/s)

kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    app: magedu-nginx-deployment-label
  name: magedu-nginx-deployment
  namespace: magedu
spec:
  replicas: 1
  selector:
    matchLabels:
      app: magedu-nginx-selector
  template:
    metadata:
      labels:
        app: magedu-nginx-selector
    spec:
      containers:
      - name: magedu-nginx-container
        image: harbor.linuxarchitect.io/magedu/nginx-web1:2023-08-19_18-28-30-eae63c8 
        #command: ["/apps/tomcat/bin/run_tomcat.sh"]
        #imagePullPolicy: IfNotPresent
        imagePullPolicy: Always
        ports:
        - containerPort: 80
          protocol: TCP
          name: http
        - containerPort: 443
          protocol: TCP
          name: https
        env:
        - name: "password"
          value: "123456"
        - name: "age"
          value: "20"
        resources:
          limits:
            cpu: 500m
            memory: 512Mi
          requests:
            cpu: 500m
            memory: 256Mi
 
        volumeMounts:
        - name: magedu-images
          mountPath: /usr/local/nginx/html/webapp/images
          readOnly: false
        - name: magedu-static
          mountPath: /usr/local/nginx/html/webapp/static
          readOnly: false
      volumes:
      - name: magedu-images
        nfs:
          server: 172.31.7.109
          path: /data/k8sdata/magedu/images 
      - name: magedu-static
        nfs:
          server: 172.31.7.109
          path: /data/k8sdata/magedu/static
      #nodeSelector:
      #  group: magedu
 
    
 
---
kind: Service
apiVersion: v1
metadata:
  labels:
    app: magedu-nginx-service-label
  name: magedu-nginx-service
  namespace: magedu
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
    nodePort: 30090
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
    nodePort: 30091
  selector:
    app: magedu-nginx-selector

root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/nginx# ls
nginx.yaml
 
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/nginx# vi nginx.yaml
.....
    spec:
      containers:
      - name: magedu-nginx-container
        image: qbnharbor.top/magedu/nginx-web1:v1
 
....
      volumes:
      - name: magedu-images
        nfs:
          server: 192.168.100.109
          path: /data/k8sdata/magedu/images 
      - name: magedu-static
        nfs:
          server: 192.168.100.109
          path: /data/k8sdata/magedu/static
 
....
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
    nodePort: 30090
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
    nodePort: 30091
  selector:
    app: magedu-nginx-selector
....

root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/nginx# kubectl apply -f nginx.yaml 
deployment.apps/magedu-nginx-deployment created
service/magedu-nginx-service created
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/yaml/magedu/nginx# kubectl get pod -n magedu -o wide
NAME                                             READY   STATUS                   RESTARTS   AGE     IP               NODE        NOMINATED NODE   READINESS GATES
magedu-nginx-deployment-56d8c4bb-2nrlf           1/1     Running                  0          52s     10.200.169.130   k8s-node2   <none>           <none>
magedu-tomcat-app1-deployment-5965b8d944-vtc7c   1/1     Running                  0          11m     10.200.169.129   k8s-node2   <none>           <none>

root@k8s-haproxy1:/data/k8sdata/magedu# vi /etc/haproxy/haproxy.cfg
....
listen myserver:80
  bind 192.168.100.201:80
  mode tcp
  server server1 192.168.100.151:30090 check inter 3s fall 3 rise 3
  server server2 192.168.100.152:30090 check inter 3s fall 3 rise 3
  server server3 192.168.100.153:30090 check inter 3s fall 3 rise 3
....
root@k8s-haproxy2:~# systemctl restart haproxy
----测试访问----
root@k8s-master1:/home/zookeeper-case-n79/2.deployment/k8s-data/dockerfile/web/pub-images/nginx-base# curl 192.168.100.201
nginx web1 magedu v1

root@k8s-haproxy1:/data/k8sdata/magedu/images# ll
-rw-r--r-- 1 root root 452810 Aug 31 11:19 1.jpg
root@k8s-master1:~# kubectl exec -it -n magedu magedu-tomcat-app1-deployment-5965b8d944-vtc7c bash
[root@magedu-tomcat-app1-deployment-5965b8d944-vtc7c /]# df -h
....
192.168.100.109:/data/k8sdata/magedu/images   14G   11G  3.0G  78% /data/webapp/images
192.168.100.109:/data/k8sdata/magedu/static   14G   11G  3.0G  78% /data/webapp/static
....
[root@magedu-tomcat-app1-deployment-5965b8d944-vtc7c ~]# ll /data/webapp/images
-rw-r--r-- 1 root root 452810 Aug 31 11:19 1.jpg