目录
1. 生产环境K8S平台规划
1.1 单Master集群
1.2 多Master集群(HA)
1.3 环境准备
2. 服务器硬件配置推荐
3. 官方提供的三种部署方式
- minikube
Minikube是一个工具,可以在本地快速运行一个单点的Kubernetes,仅用于尝试Kubernetes或日常开发的用户使用。
部署地址:https://kubernetes.io/docs/setup/minikube/ - kubeadm
Kubeadm也是一个工具,提供kubeadm init和kubeadm join,用于快速部署Kubernetes集群。
部署地址:https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/ - 二进制
推荐,从官方下载发行版的二进制包,手动部署每个组件,组成Kubernetes集群。
下载地址:https://github.com/kubernetes/kubernetes/releases
4. 操作系统初始化
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
# 更新系统 yum update -y # 关闭防火墙: systemctl stop firewalld systemctl disable firewalld # 关闭selinux: setenforce 0 # 临时 sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久 # 关闭swap: swapoff -a # 临时 sed -ri 's/.*swap.*/#&/' /etc/fstab # 永久 # 根据规划设置主机名 hostnamectl set-hostname <hostname> # 主机名 # 将桥接的 IPv4 流量传递到 iptables 的链 cat > /etc/sysctl.d/k8s.conf << EOF net.ipv4.ip_forward=1 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF sysctl --system # 生效 # 时间同步 yum install ntpdate -y ntpdate time.windows.com |
1 2 3 4 5 6 7 |
# 在 master添加 hosts cat >> /etc/hosts << EOF 192.168.31.63 k8s-master1 192.168.31.64 k8s-master2 192.168.31.65 k8s-node1 192.168.31.66 k8s-node2 EOF |
5. Etcd集群
Etcd 是一个分布式键值存储系统,Kubernetes 使用 Etcd 进行数据存储,所以先准备一个 Etcd 数据库,为解决 Etcd 单点故障,应采用集群方式部署,这里使用 3 台组建集群,可容忍 1 台机器故障,当然,你也可以使用 5 台组建集群,可容忍 2 台机器故障。
节点名称 | IP |
etcd-1 | 192.168.31.64 |
etcd-2 | 192.168.31.65 |
etcd-3 | 192.168.31.66 |
注:为了节省机器,这里与 K8s 节点机器复用。也可以独立于 k8s 集群之外部署,只要apiserver 能连接到就
5.1 准备 cfssl 证书生成工
生成证书常用的2种工具:openssl、cfssl(推荐)。
cfssl 是一个开源的证书管理工具,使用 json 文件生成证书,相比 openssl 更方便使用。找任意一台服务器操作,这里用 Master 节点
cfssl安装证书
github: https://github.com/cloudflare/cfssl
官网地址: https://pkg.cfssl.org/
1 2 3 4 5 6 7 |
wget https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssl_1.5.0_linux_amd64 -O cfssl_linux-amd64 wget https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssl-certinfo_1.5.0_linux_amd64 -O cfssl-certinfo_linux-amd64 wget https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssljson_1.5.0_linux_amd64 -O cfssljson_linux-amd64 chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64 mv cfssl_linux-amd64 /usr/local/bin/cfssl mv cfssljson_linux-amd64 /usr/local/bin/cfssljson mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo |
5.2 生成 Etcd 证书
(1)自签证书颁发机构(CA)
创建工作目录:
1 2 |
mkdir -p ~/TLS/{etcd,k8s} cd ~/TLS/etcd |
自签 CA
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
cat > ca-config.json<< EOF { "signing": { "default": { "expiry": "87600h" }, "profiles": { "www": { "expiry": "87600h", "usages": [ "signing","key encipherment", "server auth", "client auth" ] } } } } EOF cat > ca-csr.json<< EOF { "CN": "etcd CA", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Beijing", "ST": "Beijing" } ] } EOF |
查看生成文件:
1 2 |
[root@k8s-master etcd]# ls ca-config.json ca-csr.json |
生成证书:
1 |
cfssl gencert -initca ca-csr.json | cfssljson -bare ca - |
查看文件:
1 2 |
[root@k8s-master etcd]# ls *pem ca-key.pem ca.pem |
(2)使用自签 CA 签发 Etcd HTTPS 证书
创建证书申请文件:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
cat > server-csr.json<< EOF { "CN": "etcd", "hosts": [ "127.0.0.1", "192.168.31.63", "192.168.31.64", "192.168.31.65", "192.168.31.66" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing" } ] } EOF |
注:上述文件 hosts 字段中 IP 为所有 etcd 节点的集群内部通信 IP,一个都不能少!为了方便后期扩容可以多写几个预留的 IP。
查看文件:
1 2 |
[root@k8s-master etcd]# ls ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem server-csr.json |
生成证书:
1 |
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server |
查看文件:
1 2 |
[root@k8s-master etcd]# ls server*pem server-key.pem server.pem |
5.3 从 Github 下载二进制文
github:https://github.com/etcd-io/etcd
1 2 |
cd ~ wget https://github.com/etcd-io/etcd/releases/download/v3.2.32/etcd-v3.2.32-linux-amd64.tar.gz |
5.4 部署 Etcd 集群
以下在master1节点 上操作,为简化操作,待会将master1节点 生成的所有文件拷贝到其它节点.。
(1)创建工作目录并解压二进制包
1 2 3 |
mkdir /opt/etcd/{bin,cfg,ssl} -p tar zxvf etcd-v3.* mv etcd-v3.*-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/ |
(2)创建 etcd 配置文件
1 2 3 4 5 6 7 8 9 10 11 12 13 |
cat > /opt/etcd/cfg/etcd.conf << EOF #[Member] ETCD_NAME="etcd-1" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.31.64:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.31.64:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.31.64:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.31.64:2379" ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.31.64:2380,etcd-2=https://192.168.31.65:2380,etcd-3=https://192.168.31.66:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" EOF |
ETCD_NAME:节点名称,集群中唯一
ETCD_DATA_DIR:数据目录
ETCD_LISTEN_PEER_URLS:集群通信监听地址
ETCD_LISTEN_CLIENT_URLS:客户端访问监听地址
ETCD_INITIAL_ADVERTISE_PEER_URLS:集群通告地址
ETCD_ADVERTISE_CLIENT_URLS:客户端通告地址
ETCD_INITIAL_CLUSTER:集群节点地址
ETCD_INITIAL_CLUSTER_TOKEN:集群 Token
ETCD_INITIAL_CLUSTER_STATE:加入集群的当前状态,new 是新集群,existing 表示加入已有集群
(3)systemd 管理 etc
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
cat > /usr/lib/systemd/system/etcd.service << EOF [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify EnvironmentFile=/opt/etcd/cfg/etcd.conf ExecStart=/opt/etcd/bin/etcd \ --cert-file=/opt/etcd/ssl/server.pem \ --key-file=/opt/etcd/ssl/server-key.pem \ --peer-cert-file=/opt/etcd/ssl/server.pem \ --peer-key-file=/opt/etcd/ssl/server-key.pem \ --trusted-ca-file=/opt/etcd/ssl/ca.pem \ --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF |
(4)拷贝刚才生成的证书
把刚才生成的证书拷贝到配置文件中的路径:
1 |
cp ~/TLS/etcd/ca*pem ~/TLS/etcd/server*pem /opt/etcd/ssl/ |
(5)启动并设置开机启动
1 2 3 |
systemctl daemon-reload systemctl start etcd systemctl enable etcd |
(6)将上面节点 1 所有生成的文件拷贝到其他节点
1 2 |
scp -r /opt/etcd/ root@192.168.31.65:/opt/ scp /usr/lib/systemd/system/etcd.service root@192.168.31.65:/usr/lib/systemd/system/ |
然后在各个节点分别修改 etcd.conf 配置文件中的节点名称和当前服务器 IP:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
vi /opt/etcd/cfg/etcd.conf #[Member] ETCD_NAME="etcd-2" # 修改此处,节点 2 改为 etcd-2,节点 3 改为 etcd-3 ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.31.65:2380" # 修改此处为当前服务器 IP ETCD_LISTEN_CLIENT_URLS="https://192.168.31.65:2379" # 修改此处为当前服务器 IP #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.31.65:2380" # 修改此处为当前 服务器 IP ETCD_ADVERTISE_CLIENT_URLS="https://192.168.31.65:2379" # 修改此处为当前服务器 IP ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.31.64:2380,etcd-2=https://192.168.31.65:2380,etcd-3=https://192.168.31.66:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" |
最后启动 etcd 并设置开机启动,同上
最后查看etcd 状态
1 |
systemctl status etcd.service |
1 2 3 4 |
etcd.service - Etcd Server Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled) Active: active (running) since 四 2021-04-29 20:38:00 CST; 1min 40s ago Main PID: 20822 (etcd) |
(7)查看集群状态
1 2 3 4 5 6 |
ETCDCTL_API=3 /opt/etcd/bin/etcdctl \ --cacert=/opt/etcd/ssl/ca.pem \ --cert=/opt/etcd/ssl/server.pem \ --key=/opt/etcd/ssl/server-key.pem \ --endpoints="https://192.168.31.64:2379,https://192.168.31.65:2379,https://192.168.31.66:2379" \ endpoint health |
--endpoints
:需要检测的IP
输出:
1 2 3 |
https://192.168.31.64:2379 is healthy: successfully committed proposal: took = 4.069808ms https://192.168.31.65:2379 is healthy: successfully committed proposal: took = 4.303527ms https://192.168.31.66:2379 is healthy: successfully committed proposal: took = 4.412812ms |
如果输出上面信息,就说明集群部署成功。如果有问题第一步先看日志:
/var/log/message
或 journalctl -u etcd
踩坑
1、status=1/FAILURE 看清配置/opt/etcd/cfg/etcd.conf 是否缺字符/多字符。
2、修改了配置需要删除 /var/lib/etcd 文件夹,然后reboot。
6、安装 Docker
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
# step 1: 移除旧的docker sudo yum remove -y docker \ docker-client \ docker-client-latest \ docker-common \ docker-latest \ docker-latest-logrotate \ docker-logrotate \ docker-engine sudo rm -rf /var/lib/docker # step 1: 安装相关组件和配置yum源 sudo yum install -y yum-utils sudo yum-config-manager \ --add-repo \ http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo # step 2: 执行安装 sudo yum install -y docker-ce # step 3: 配置镜像下载加速 sudo mkdir -p /etc/docker # 注意json的格式,否则启动不了 sudo cat > /etc/docker/daemon.json <<EOF { "registry-mirrors": [ "http://hub-mirror.c.163.com", "https://docker.mirrors.ustc.edu.cn", "https://registry.docker-cn.com" ], "dns": [ "223.5.5.5", "223.6.6.6" ] } EOF # step 4: 启动docker并配置开机启动 sudo systemctl start docker sudo systemctl enable docker |
7. 部署Master组件
7.1 自签 kube-apiserver SSL证书
APIserver是整个集群的入口,只有它启动了,别的组件才可以启动;我们基于https访问,所以需要自签证书。
1 2 |
mkdir -p ~/TLS/k8s/ cd ~/TLS/k8s/ |
(1)自签证书颁发机构(CA)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
cat > ca-config.json<< EOF { "signing": { "default": { "expiry": "87600h" }, "profiles": { "kubernetes": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } EOF cat > ca-csr.json<< EOF { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Beijing", "ST": "Beijing", "O": "k8s", "OU": "System" } ] } EOF |
查看文件:
1 2 |
[root@k8s-master k8s]# ls ca-config.json ca-csr.json |
(2)生成证书:
1 |
cfssl gencert -initca ca-csr.json | cfssljson -bare ca - |
查看:
1 2 |
[root@k8s-master k8s]# ls *pem ca-key.pem ca.pem |
(3)使用自签 CA 签发 kube-apiserver HTTPS 证书
应用程序访问https api(自签证书)可用通过以下2种方式:
1、证书添加IP可信任
2、携带CA证书
注意:在使用网关、转发等情况下,不能实现携带CA证书,那么我们可用添加信任的IP。
创建证书申请文件:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
cat > server-csr.json<< EOF { "CN": "kubernetes", "hosts": [ "10.0.0.1", "127.0.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local", "192.168.31.64", "192.168.31.65", "192.168.31.66" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "k8s", "OU": "System" } ] } EOF |
注意hosts节点,所有需要访问k8s的有关ip都要写进来,包括本机ip,缩主机ip,宿主机默认网关ip。
查看文件:
1 2 |
[root@k8s-master k8s]# ls ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem server-csr.json |
(2)生成证书:
1 |
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server |
查看生成文件:
1 2 |
[root@k8s-master k8s]# ls server*pem server-key.pem server.pem |
7.2 从 Github 下载k8s二进制文件
下载地址:https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#server-binaries
注:打开链接你会发现里面有很多包,下载一个 server 包就够了,包含了 Master 和Worker Node 二进制文件
1 2 |
cd ~ wget https://dl.k8s.io/v1.18.18/kubernetes-server-linux-amd64.tar.gz |
7.3 解压二进制包
1 2 3 4 5 |
mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs} tar zxvf kubernetes-server-linux-amd64.tar.gz cd kubernetes/server/bin cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin cp kubectl /usr/bin |
7.4 部署 kube-apiserver
1. 创建配置文件
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
cat > /opt/kubernetes/cfg/kube-apiserver.conf << EOF KUBE_APISERVER_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/opt/kubernetes/logs \\ --etcd-servers=https://192.168.31.64:2379,https://192.168.31.65:2379,https://192.168.31.66:2379 \\ --bind-address=192.168.31.64 \\ --secure-port=6443 \\ --advertise-address=192.168.31.64 \\ --allow-privileged=true \\ --service-cluster-ip-range=10.0.0.0/24 \\ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\ --authorization-mode=RBAC,Node \\ --enable-bootstrap-token-auth=true \\ --token-auth-file=/opt/kubernetes/cfg/token.csv \\ --service-node-port-range=30000-32767 \\ --kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \\ --kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \\ --tls-cert-file=/opt/kubernetes/ssl/server.pem \\ --tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\ --client-ca-file=/opt/kubernetes/ssl/ca.pem \\ --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\ --etcd-cafile=/opt/etcd/ssl/ca.pem \\ --etcd-certfile=/opt/etcd/ssl/server.pem \\ --etcd-keyfile=/opt/etcd/ssl/server-key.pem \\ --audit-log-maxage=30 \\ --audit-log-maxbackup=3 \\ --audit-log-maxsize=100 \\ --audit-log-path=/opt/kubernetes/logs/k8s-audit.log" EOF |
注:上面两个\\ 第一个是转义符,第二个是换行符,使用转义符是为了使用 EOF 保留换行符。
–logtostderr:启用日志
—v:日志等级
–log-dir:日志目录
–etcd-servers:etcd 集群地址
–bind-address:监听地址
–secure-port:https 安全端口
–advertise-address:集群通告地址
–allow-privileged:启用授权
–service-cluster-ip-range:Service 虚拟 IP 地址段
–enable-admission-plugins:准入控制模块
–authorization-mode:认证授权,启用 RBAC 授权和节点自管理
–enable-bootstrap-token-auth:启用 TLS bootstrap 机制
–token-auth-file:bootstrap token 文件
–service-node-port-range:Service nodeport 类型默认分配端口范围
–kubelet-client-xxx:apiserver 访问 kubelet 客户端证书
–tls-xxx-file:apiserver https 证书
–etcd-xxxfile:连接 Etcd 集群证书
–audit-log-xxx:审计日志
2. 拷贝刚才生成的证书
把刚才生成的证书拷贝到配置文件中的路径:
1 |
cp ~/TLS/k8s/ca*pem ~/TLS/k8s/server*pem /opt/kubernetes/ssl/ |
3. 启用 TLS Bootstrapping 机制
TLS Bootstraping:Master apiserver 启用 TLS 认证后,Node 节点 kubelet 和 kube- proxy 要与 kube-apiserver 进行通信,必须使用 CA 签发的有效证书才可以,当 Node节点很多时,这种客户端证书颁发需要大量工作,同样也会增加集群扩展复杂度。为了简化流程,Kubernetes 引入了 TLS bootstraping 机制来自动颁发客户端证书,kubelet会以一个低权限用户自动向 apiserver 申请证书,kubelet 的证书由 apiserver 动态签署。所以强烈建议在 Node 上使用这种方式,目前主要用于 kubelet,kube-proxy 还是由我们统一颁发一个证书。
TLS bootstraping 工作流程
创建上述配置文件中 token 文件:
1 2 3 |
cat > /opt/kubernetes/cfg/token.csv << EOF c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:node-bootstrapper" EOF |
格式:token,用户名,UID,用户组
token 也可自行生成
1 |
head -c 16 /dev/urandom | od -An -t x | tr -d ' ' |
4. systemd 管理 api
1 2 3 4 5 6 7 8 9 10 11 |
cat > /usr/lib/systemd/system/kube-apiserver.service << EOF [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target EOF |
5. 启动并设置开机启动
1 2 3 |
systemctl daemon-reload systemctl start kube-apiserver systemctl enable kube-apiserver |
查看是否成功启动:
1 2 3 4 5 6 |
[root@k8s-master ~]# systemctl status kube-apiserver ● kube-apiserver.service - Kubernetes API Server Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled) Active: active (running) since 四 2021-04-29 22:50:10 CST; 8s ago Docs: https://github.com/kubernetes/kubernetes Main PID: 18585 (kube-apiserver) |
6. 授权 kubelet-bootstrap
1 2 3 |
kubectl create clusterrolebinding kubelet-bootstrap \ --clusterrole=system:node-bootstrapper \ --user=kubelet-bootstrap |
6.5 部署 kube-controller-manager
1. 创建配置文件
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
cat > /opt/kubernetes/cfg/kube-controller-manager.conf << EOF KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/opt/kubernetes/logs \\ --leader-elect=true \\ --master=127.0.0.1:8080 \\ --bind-address=127.0.0.1 \\ --allocate-node-cidrs=true \\ --cluster-cidr=10.244.0.0/16 \\ --service-cluster-ip-range=10.0.0.0/24 \\ --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\ --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \\ --root-ca-file=/opt/kubernetes/ssl/ca.pem \\ --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\ --experimental-cluster-signing-duration=87600h0m0s" EOF |
--cluster-cidr
:表示pod节点的ip地址段,kube-controller-manager.sh启动文件与kube-proxy.sh启动文件两个要配置成一样。
--service-cluster-ip-range
: 表示apiserver服务器的ip地址段,需要跟kube-apiserver启动文件的 –service-cluster-ip-range 参数配置成一致。
--master
: 表示跟kube-apiserver内部的通讯地址。通过本地非安全本地端口 8080 连接 apiserver。
–leader-elect
:当该组件启动多个时,自动选举(HA)
–cluster-signing-cert-file/–cluster-signing-key-file
:自动为 kubelet 颁发证书的 CA,与 apiserver 保持一致
2. systemd 管理 controller-manager
1 2 3 4 5 6 7 8 9 10 11 |
cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target EOF |
3. 启动并设置开机启动
1 2 3 |
systemctl daemon-reload systemctl start kube-controller-manager systemctl enable kube-controller-manager |
查看状态:
1 2 3 4 5 6 |
[root@k8s-master ~]# systemctl status kube-controller-manager ● kube-controller-manager.service - Kubernetes Controller Manager Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; enabled; vendor preset: disabled) Active: active (running) since 四 2021-04-29 23:05:55 CST; 22s ago Docs: https://github.com/kubernetes/kubernetes Main PID: 31044 (kube-controller) |
7.6 部署 kube-scheduler
1. 创建配置文件
1 2 3 4 5 6 7 8 |
cat > /opt/kubernetes/cfg/kube-scheduler.conf << EOF KUBE_SCHEDULER_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/opt/kubernetes/logs \\ --leader-elect \\ --master=127.0.0.1:8080 \\ --bind-address=127.0.0.1" EOF |
–master
:通过本地非安全本地端口 8080 连接 apiserver。
–leader-elect
:当该组件启动多个时,自动选举(HA)
2. systemd 管理 scheduler
1 2 3 4 5 6 7 8 9 10 11 |
cat > /usr/lib/systemd/system/kube-scheduler.service << EOF [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target EOF |
3. 启动并设置开机启动
1 2 3 |
systemctl daemon-reload systemctl start kube-scheduler systemctl enable kube-scheduler |
查看状态:
1 2 3 4 5 6 |
[root@k8s-master ~]# systemctl status kube-scheduler ● kube-scheduler.service - Kubernetes Scheduler Loaded: loaded (/usr/lib/systemd/system/kube-scheduler.service; enabled; vendor preset: disabled) Active: active (running) since 四 2021-04-29 23:15:18 CST; 25s ago Docs: https://github.com/kubernetes/kubernetes Main PID: 5833 (kube-scheduler) |
4. 查看集群状态
所有组件都已经启动成功,通过 kubectl 工具查看当前集群组件状态
1 2 3 4 5 6 7 |
[root@k8s-master ~]# kubectl get cs NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-2 Healthy {"health": "true"} etcd-0 Healthy {"health": "true"} etcd-1 Healthy {"health": "true"} |
如上输出说明 Master 节点组件运行正常。
8、部署 Worker Node
下面还是在 Master Node 上操作,即同时作为 Worker Node
8.1 创建工作目录并拷贝二进制文件
在所有 worker node 创建工作目录:
1 |
mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs} |
从 master 节点拷贝:
1 2 |
cd ~/kubernetes/server/bin cp kubelet kube-proxy /opt/kubernetes/bin |
8.2 部署 kubelet
1. 创建配置文件
1 2 3 4 5 6 7 8 9 10 11 12 |
cat > /opt/kubernetes/cfg/kubelet.conf << EOF KUBELET_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/opt/kubernetes/logs \\ --hostname-override=k8s-master \\ --network-plugin=cni \\ --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\ --bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\ --config=/opt/kubernetes/cfg/kubelet-config.yml \\ --cert-dir=/opt/kubernetes/ssl \\ --pod-infra-container-image=easzlab/pause-amd64:3.2" EOF |
–hostname-override
:显示名称,集群中唯一
–network-plugin
:启用 CNI –kubeconfig:空路径,会自动生成,后面用于连接 apiserver –bootstrap-kubeconfig:首次启动向 apiserver 申请证书
–config
:配置参数文件
–cert-dir
:kubelet 证书生成目录
–pod-infra-container-image
:管理 Pod 网络容器的镜像
–pod-infra-container-image的原来的值是:
gcr.io/google_containers/pause-amd64:3.0
,由于有国内的原因不能访问,于是在docker hub找到一个镜像作为替换:easzlab/pause-amd64:3.2
。
还有pause-amd64
没有latest标签,标签有3.0
,3.1
,3.2
,最新是3.2
(2021-4-30)
2. 配置参数文件
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
cat > /opt/kubernetes/cfg/kubelet-config.yml << EOF kind: KubeletConfiguration apiVersion: kubelet.config.k8s.io/v1beta1 address: 0.0.0.0 port: 10250 readOnlyPort: 10255 cgroupDriver: cgroupfs clusterDNS: - 10.0.0.2 clusterDomain: cluster.local failSwapOn: false authentication: anonymous: enabled: false webhook: cacheTTL: 2m0s enabled: true x509: clientCAFile: /opt/kubernetes/ssl/ca.pem authorization: mode: Webhook webhook: cacheAuthorizedTTL: 5m0s cacheUnauthorizedTTL: 30s evictionHard: imagefs.available: 15% memory.available: 100Mi nodefs.available: 10% nodefs.inodesFree: 5% maxOpenFiles: 1000000 maxPods: 110 EOF |
3. 生成 bootstrap.kubeconfig 文件
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
KUBE_APISERVER="https://192.168.31.64:6443" # master服务器IP(内网IP) TOKEN="c47ffb939f5ca36231d9e3121a252940" # token要与/opt/kubernetes/cfg/token.csv 里面的token一致 # 配置集群参数 kubectl config set-cluster kubernetes \ --certificate-authority=/opt/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=bootstrap.kubeconfig # 配置客户端认证 kubectl config set-credentials "kubelet-bootstrap" \ --token=${TOKEN} \ --kubeconfig=bootstrap.kubeconfig # 配置关联 kubectl config set-context default \ --cluster=kubernetes \ --user="kubelet-bootstrap" \ --kubeconfig=bootstrap.kubeconfig # 配置默认关联 kubectl config use-context default --kubeconfig=bootstrap.kubeconfig |
拷贝到配置文件路径:
1 |
cp bootstrap.kubeconfig /opt/kubernetes/cfg |
4. systemd 管理 kubelet
1 2 3 4 5 6 7 8 9 10 11 12 |
cat > /usr/lib/systemd/system/kubelet.service << EOF [Unit] Description=Kubernetes Kubelet After=docker.service [Service] EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF |
5. 启动并设置开机启动
1 2 3 |
systemctl daemon-reload systemctl start kubelet systemctl enable kubelet |
查看状态:
1 2 3 4 5 |
[root@k8s-master ~]# systemctl status kubelet ● kubelet.service - Kubernetes Kubelet Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled) Active: active (running) since 五 2021-04-30 00:24:45 CST; 14s ago Main PID: 22689 (kubelet) |
8.3 批准 kubelet 证书申请并加入
1 2 3 4 5 6 7 8 9 10 11 12 |
# 查看 kubelet 证书请求 [root@k8s-master ~]# kubectl get csr NAME AGE SIGNERNAME REQUESTOR CONDITION node-csr-Obs9MKQixjOr8H-5exIFLFimBvzRMRZtdgV0SKCMVuE 86s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending # 批准申请 (请更换node-csr-*) kubectl certificate approve node-csr-Obs9MKQixjOr8H-5exIFLFimBvzRMRZtdgV0SKCMVuE # 查看节点 [root@k8s-master ~]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master NotReady <none> 8s v1.18.18 |
注:由于网络插件还没有部署,节点会没有准备就绪 NotReady
8.4 部署 kube-proxy
1. 创建配置文件
1 2 3 4 5 6 |
cat > /opt/kubernetes/cfg/kube-proxy.conf << EOF KUBE_PROXY_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/opt/kubernetes/logs \\ --config=/opt/kubernetes/cfg/kube-proxy-config.yml" EOF |
2. 配置参数文件
默认ipables,如需要ipvs模式(推荐),请参考这篇文章
1 2 3 4 5 6 7 8 9 10 |
cat > /opt/kubernetes/cfg/kube-proxy-config.yml << EOF kind: KubeProxyConfiguration apiVersion: kubeproxy.config.k8s.io/v1alpha1 bindAddress: 0.0.0.0 metricsBindAddress: 0.0.0.0:10249 clientConnection: kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig hostnameOverride: k8s-master clusterCIDR: 10.244.0.0/16 EOF |
--cluster-cidr
: 表示pod节点的ip地址段,kube-controller-manager启动文件与kube-proxy文件两个要配置成一样。
3. 生成 kube-proxy.kubeconfig 文件
生成 kube-proxy 证书:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
# 切换工作目录 cd ~/TLS/k8s # 创建证书请求文件 cat > kube-proxy-csr.json<< EOF { "CN": "system:kube-proxy", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "k8s", "OU": "System" } ] } EOF # 生成证书 cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy [root@k8s-master k8s]# ls kube-proxy*pem kube-proxy-key.pem kube-proxy.pem |
生成 kubeconfig 文件
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
KUBE_APISERVER="https://192.168.31.64:6443" # masterIP地址(内网) kubectl config set-cluster kubernetes \ --certificate-authority=/opt/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=kube-proxy.kubeconfig kubectl config set-credentials kube-proxy \ --client-certificate=./kube-proxy.pem \ --client-key=./kube-proxy-key.pem \ --embed-certs=true \ --kubeconfig=kube-proxy.kubeconfig kubectl config set-context default \ --cluster=kubernetes \ --user=kube-proxy \ --kubeconfig=kube-proxy.kubeconfig kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig |
拷贝到配置文件指定路径:
1 |
cp kube-proxy.kubeconfig /opt/kubernetes/cfg/ |
4. systemd 管理 kube-proxy
1 2 3 4 5 6 7 8 9 10 11 12 |
cat > /usr/lib/systemd/system/kube-proxy.service << EOF [Unit] Description=Kubernetes Proxy After=network.target [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF |
5. 启动并设置开机启动
1 2 3 |
systemctl daemon-reload systemctl start kube-proxy systemctl enable kube-proxy |
8.5 部署 CNI 网
先准备好 CNI 二进制文件
GitHub:https://github.com/containernetworking/plugins/releases
1 2 |
cd ~ wget https://github.com/containernetworking/plugins/releases/download/v0.9.1/cni-plugins-linux-amd64-v0.9.1.tgz |
解压二进制包并移动到默认工作目录:
1 2 |
mkdir -p /opt/cni/bin tar zxvf cni-plugins-linux-amd64-*.tgz -C /opt/cni/bin |
部署 CNI 网络:
由于https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml国内原因,无法连接,以下是kube-flannel.yml的原文:
注意:kube-flannel.yml文件里的net-conf.json->Network
:这个网络要与kube-controller-manager.conf的cluster-cidr的一致
1 2 3 |
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml # 部署 kubectl apply -f kube-flannel.yml |
查看部署情况:
1 2 3 |
[root@k8s-master ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE kube-flannel-ds-rpfjw 1/1 Running 0 15s |
以上,即为成功
查看节点情况:
1 2 3 |
[root@k8s-master ~]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master Ready <none> 131m v1.18.18 |
部署好网络插件,Node 准备就绪。
8.6 授权 apiserver 访问 kubelet
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
cat > apiserver-to-kubelet-rbac.yaml<< EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults name: system:kube-apiserver-to-kubelet rules: - apiGroups: - "" resources: - nodes/proxy - nodes/stats - nodes/log - nodes/spec - nodes/metrics - pods/log verbs: - "*" --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: system:kube-apiserver namespace: "" roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:kube-apiserver-to-kubelet subjects: - apiGroup: rbac.authorization.k8s.io kind: User name: kubernetes EOF # 部署 kubectl apply -f apiserver-to-kubelet-rbac.yaml |
8.7 新增加 Worker Node
1. 拷贝已部署好的 Node 相关文件到新节点
在 master 节点将 Worker Node 涉及文件拷贝到新节点 192.168.31.65/66
1 2 3 4 |
scp -r /opt/kubernetes root@192.168.31.65:/opt/ scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@192.168.31.65:/usr/lib/systemd/system scp -r /opt/cni/ root@192.168.31.65:/opt/ scp /opt/kubernetes/ssl/ca.pem root@192.168.31.65:/opt/kubernetes/ssl |
2. 在上面复制过的Worker Node,删除 kubelet 证书和 kubeconfig 文件
1 2 |
rm -f /opt/kubernetes/cfg/kubelet.kubeconfig rm -f /opt/kubernetes/ssl/kubelet* |
注:这几个文件是证书申请审批后自动生成的,每个 Node 不同,必须删除重新生成。
3. 在每个Worker Node,修改主机名
1 2 3 4 5 6 7 |
vi /opt/kubernetes/cfg/kubelet.conf # 修改参数 --hostname-override=k8s-node1 vi /opt/kubernetes/cfg/kube-proxy-config.yml # 修改参数 hostnameOverride: k8s-node1 |
4. 启动并设置开机启动
1 2 3 4 5 |
systemctl daemon-reload systemctl start kubelet systemctl enable kubelet systemctl start kube-proxy systemctl enable kube-proxy |
5. 在 Master 上批准新 Node kubele
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# 查看待批准的节点 [root@k8s-master ~]# kubectl get csr NAME AGE SIGNERNAME REQUESTOR CONDITION node-csr-8Layga7vn0ebX97pBfbm63mmn5TIUqvYctentEaFzjg 4m58s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending node-csr-v1le02hgef9D_22pgCbbf4aUHW0UUqDjgC9CYVN26Ms 7m46s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending # 批准节点 (请更换node-csr-*) [root@k8s-master ~]# kubectl certificate approve node-csr-8Layga7vn0ebX97pBfbm63mmn5TIUqvYctentEaFzjg # 查看节点 [root@k8s-master ~]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master Ready <none> 3h52m v1.18.18 k8s-node1 Ready <none> 2m50s v1.18.18 k8s-node2 Ready <none> 3m8s v1.18.18 |
以上,节点以全部加入。
如果,一直是NotReady
状态,看看节点是否下载了这两个image
,有可能下载不来。
1 2 3 4 5 |
$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE quay.io/coreos/flannel v0.14.0-rc1 0a1a2818ce59 3 weeks ago 67.9MB easzlab/pause-amd64 3.2 80d28bedfe5d 14 months ago 683kB |
安装CoreDNS
查看这篇文章
参考
二进制部署k8s集群(3):部署kube-apiserver,签发kube-apiserver证书|kuelete证书|kube-proxy证书
二进制安装部署k8s高可用集群V1.20
2020-07-04 搭建一个完整的Kubernetes集群(上)
2020-07-14 搭建一个完整的Kubernetes集群(下)