介绍

只记录部署过程，不保证ctrl+c和ctrl+v能直接跑！请自行判断参数是否能直接套用！
Kubernetes-v1.15已经在2019-06-19发布，而Kubernetes-v1.14作为上一个release版本已经更新到了1.14.3。
之前写的Kubernetes-v1.11的高可用集群部署文档由于版本变迁部分参数需要改动，部署过程有些地方欠缺考虑，这里以Kubernetes-v1.14.3版本重新做一次文档整理
因此文章整体架构基本与【二进制部署 kubernetes v1.11.x 高可用集群】相同
配置部分参考kubeadm的参数

部署说明

本次部署方式为二进制可执行文件的方式部署

注意请根据自己的实际情况调整
对于生产环境部署，请注意某些参数的选择

如无特殊说明，均在k8s-m1节点上执行

参考博文

感谢两位大佬的文章，这里整合一下两位大佬的内容，结合自己的理解整理本文

漠然大佬的【Kubernetes 1.13.4 搭建】
张馆长的【二进制部署Kubernetes v1.13.5 HA可选】

软件版本

这里参考kubeadm-1.14的镜像版本和yum源依赖

Kubernetes版本v1.14.3
Docker-CE版本18.09.06
CNI-Plugins版本v0.75
etcd版本v3.2.24

网络信息

基于CNI的模式实现容器网络
Cluster IP CIDR: 10.244.0.0/16
Service Cluster IP CIDR: 10.96.0.0/12
Service DNS IP: 10.96.0.10
Kubernetes API VIP: 172.16.80.200

节点信息

操作系统可采用 Ubuntu Server 16.04 LTS 、Ubuntu Server 18.04 LTS和 CentOS 7.6
由keepalived提供VIP
由haproxy提供kube-apiserver四层负载均衡
通过污点的方式防止工作负载被调度到master节点
服务器配置请根据实际情况适当调整

IP地址	主机名	操作系统	内核版本	角色	CPU	内存
172.16.80.201	k8s-m1	CentOS-7.6.1810	3.10.0-957	master+node	4	8G
172.16.80.202	k8s-m2	CentOS-7.6.1810	3.10.0-957	master+node	4	8G
172.16.80.203	k8s-m3	CentOS-7.6.1810	3.10.0-957	master+node	4	8G
172.16.80.204	k8s-n1	CentOS-7.6.1810	3.10.0-957	node	4	8G
172.16.80.205	k8s-n2	CentOS-7.6.1810	3.10.0-957	node	4	8G

目录说明

/usr/local/bin/：存放kubernetes和etcd二进制文件
/opt/cni/bin/：存放cni-plugin二进制文件
/etc/etcd/：存放etcd配置文件和SSL证书
/etc/kubernetes/：存放kubernetes配置和SSL证书
/etc/cni/net.d/：安装CNI插件后会在这里生成配置文件
$HOME/.kube/：kubectl命令会在家目录下建立此目录，用于保存访问kubernetes集群的配置和缓存
$HOME/.helm/：helm命令会建立此目录，用于保存helm缓存和repository信息

事前准备

事情准备在所有服务器上都需要完成
部署过程以root用户完成
所有服务器网络互通，k8s-m1可以通过SSH证书免密登录到其他master节点，用于分发文件

编辑hosts文件

cat > /etc/hosts <<EOF
127.0.0.1 localhost
172.16.80.200 k8s-vip
172.16.80.201 k8s-m1
172.16.80.202 k8s-m2
172.16.80.203 k8s-m3
172.16.80.204 k8s-n1
172.16.80.205 k8s-n2
EOF

时间同步服务

集群系统需要各节点时间同步

参考链接：RHEL7官方文档

这里使用公网对时，如果需要内网对时，请自行配置

1
2
3

yum install -y chrony
systemctl enable chronyd
systemctl start chronyd

关闭firewalld

systemctl stop firewalld
systemctl disable firewalld
# 清空iptables规则
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

关闭无用服务

请根据自己的环境针对性地关闭服务

# 禁用蓝牙
systemctl disable bluetooth.service
# 如果不用NFS可以禁用rpcbind
systemctl disable rpcbind.service
systemctl disable rpcbind.socket
# 禁用cockpit网页端管理工具
systemctl disable cockpit.socket
# 禁用CUPS打印机服务
systemctl disable cups-browsed.service
systemctl disable cups.path           
systemctl disable cups.service        
systemctl disable cups.socket

关闭SELINUX

1 2	setenforce 0 sed -ri '/^[^#]*SELINUX=/s#=.+$#=disabled#' /etc/selinux/config

禁用swap

1 2	swapoff -a && sysctl -w vm.swappiness=0 sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab

配置`sysctl`参数

cat > /etc/sysctl.d/99-centos.conf <<EOF 
# 最大文件句柄数
fs.file-max=5242880
# 最大文件打开数
fs.nr_open=5242880
# 在CentOS7.4引入了一个新的参数来控制内核的行为。 
# /proc/sys/fs/may_detach_mounts 默认设置为0
# 当系统有容器运行的时候，需要将该值设置为1。
fs.may_detach_mounts = 1
# 二层的网桥在转发包时也会被iptables的FORWARD规则所过滤
net.bridge.bridge-nf-call-arptables=1
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
# 关闭严格校验数据包的反向路径
net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.all.rp_filter=0
# 修改动态NAT跟踪记录参数
net.netfilter.nf_conntrack_max = 655350
net.netfilter.nf_conntrack_tcp_timeout_established = 1200
# 加快系统关闭处于 FIN_WAIT2 状态的 TCP 连接
net.ipv4.tcp_fin_timeout = 30
# 系统中处于 SYN_RECV 状态的 TCP 连接数量
net.ipv4.tcp_max_syn_backlog = 8192
# 内核中管理 TIME_WAIT 状态的数量
net.ipv4.tcp_max_tw_buckets = 5120
# 指定重发 SYN/ACK 的次数
net.ipv4.tcp_synack_retries = 2
# 端口最大的监听队列的长度
net.core.somaxconn = 4096
# 打开ipv4数据包转发
net.ipv4.ip_forward = 1
# 允许应用程序能够绑定到不属于本地网卡的地址
net.ipv4.ip_nonlocal_bind=1 
# 内存耗尽才使用swap分区
vm.swappiness = 0
vm.panic_on_oom = 0
# 允许overcommit
vm.overcommit_memory = 1
# 定义了进程能拥有的最多内存区域
vm.max_map_count = 65536
# 设置系统TCP连接keepalive的持续时间，默认7200
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 10
# 禁用ipv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
EOF

# 让sysctl参数生效
sysctl --system

更新软件包

1	yum update -y

安装软件包

1
2
3

yum groups install base -y
yum install epel-release -y
yum install jq nc git redhat-lsb vim ipvsadm tree dstat iotop htop socat ipset conntrack bash-completion-extras -y

配置开机加载ipvs模块

cat > /etc/modules-load.d/ipvs.conf <<EOF
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
EOF

安装docker-ce

# 删除旧版本docker
yum remove docker docker-client docker-client-latest docker-common docker-latest docker-latest-logrotate docker-logrotate docker-selinux docker-engine-selinux docker-engine -y
# 安装docker-ce依赖
yum install -y yum-utils device-mapper-persistent-data lvm2 -y
# 添加docker-ce软件源
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 修改docker-ce软件源为阿里云
sed -e 's,download.docker.com,mirrors.aliyun.com/docker-ce,g' -i /etc/yum.repos.d/docker-ce.repo
# 安装docker-ce 18.09
yum install 1:docker-ce-cli-18.09.6-3.el7.x86_64 3:docker-ce-18.09.6-3.el7.x86_64 containerd.io -y

配置docker-ce

参考【container runtimes】

CentOS/RHEL 7.4+

mkdir -p /etc/docker
cat>/etc/docker/daemon.json<<EOF
{
    "registry-mirrors": ["https://pqbap4ya.mirror.aliyuncs.com"],
    "insecure-registries": [],
    "exec-opts": ["native.cgroupdriver=systemd"],
    "log-driver": "json-file",
    "log-opts": {
        "max-size": "100m"
    },
    "storage-driver": "overlay2",
    "storage-opts": [
        "overlay2.override_kernel_check=true"
    ],
    "data-root": "/var/lib/docker",
    "max-concurrent-downloads": 10
}
EOF

Ubuntu 16.04

mkdir -p /etc/docker
cat>/etc/docker/daemon.json<<EOF
{
  "registry-mirrors": ["https://pqbap4ya.mirror.aliyuncs.com"],
  "insecure-registries": [],
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "data-root": "/var/lib/docker",
  "max-concurrent-downloads": 10
}
EOF

配置docker命令补全

1	cp /usr/share/bash-completion/completions/docker /etc/bash_completion.d/

配置docker服务开机自启动

1 2	systemctl enable docker.service systemctl start docker.service

查看docker信息

1	docker info

禁用docker源

为避免yum update时更新docker，将docker源禁用

1	yum-config-manager --disable docker-ce-stable

确保以最新的内核启动系统

reboot

定义集群变量

注意

这里的变量只对当前会话生效，如果会话断开或者重启服务器，都需要重新定义变量
HostArray定义集群中所有节点的主机名和IP
MasterArray定义master节点的主机名和IP
NodeArray定义node节点的主机名和IP，这里master也运行kubelet所以也需要加入到NodeArray
VIP_IFACE定义keepalived的VIP绑定在哪一个网卡
ETCD_SERVERS以MasterArray的信息生成etcd集群服务器列表
ETCD_INITIAL_CLUSTER以MasterArray信息生成etcd集群初始化列表
POD_DNS_SERVER_IP定义Pod的DNS服务器IP地址

declare -A HostArray MasterArray NodeArray
# 声明所有节点的信息
HostArray=(
   ['k8s-m1']=172.16.80.201
   ['k8s-m2']=172.16.80.202
   ['k8s-m3']=172.16.80.203
   ['k8s-n1']=172.16.80.204
   ['k8s-n2']=172.16.80.205
   )
# 声明master节点信息
MasterArray=(
    ['k8s-m1']=172.16.80.201
    ['k8s-m2']=172.16.80.202
    ['k8s-m3']=172.16.80.203
    )
# 声明node节点信息
NodeArray=(
    ['k8s-n1']=172.16.80.204
    ['k8s-n2']=172.16.80.205
    )
# 配置Kubernetes APIServer的VIP地址
VIP="172.16.80.200"
KUBE_APISERVER="https://172.16.80.200:8443"

# 定义集群名字
CLUSTER_NAME="kubernetes"

# etcd版本号
# kubeadm-v1.14.3里面使用的是v3.2.24
ETCD_VERSION="v3.2.24"
# kubernetes版本号
KUBERNETES_VERSION="v1.14.3"
# cni-plugin版本号
# kubernetes YUM源里用的是v0.7.5
CNI_PLUGIN_VERSION="v0.7.5"

# 声明VIP所在的网卡名称，以ens33为例
VIP_IFACE="ens33"
# 声明etcd_server
ETCD_SERVERS=$( xargs -n1<<<${MasterArray[@]} | sort | sed 's#^#https://#;s#$#:2379#;$s#\n##' | paste -d, -s - )
ETCD_INITIAL_CLUSTER=$( for i in ${!MasterArray[@]};do  echo $i=https://${MasterArray[$i]}:2380; done | sort | paste -d, -s - )

# 定义POD_CLUSTER_CIDR
POD_NET_CIDR="10.244.0.0/16"
# 定义SVC_CLUSTER_CIDR
SVC_CLUSTER_CIDR="10.96.0.0/12"
# 定义POD_DNS_SERVER_IP
POD_DNS_SERVER_IP="10.96.0.10"

下载所需软件包

创建工作目录

1 2	mkdir -p /root/software cd /root/software

下载解压软件包

kubernetes

echo "--- 下载kubernetes ${KUBERNETES_VERSION} 二进制包 ---"
wget https://dl.k8s.io/${KUBERNETES_VERSION}/kubernetes-server-linux-amd64.tar.gz
tar xzf kubernetes-server-linux-amd64.tar.gz \
        kubernetes/server/bin/kube-controller-manager \
        kubernetes/server/bin/hyperkube \
        kubernetes/server/bin/mounter \
        kubernetes/server/bin/kubelet \
        kubernetes/server/bin/kubectl \
        kubernetes/server/bin/kube-scheduler \
        kubernetes/server/bin/kube-proxy \
        kubernetes/server/bin/kube-apiserver \
        kubernetes/server/bin/cloud-controller-manager \
        kubernetes/server/bin/kubeadm \
        kubernetes/server/bin/apiextensions-apiserver

chown -R root:root kubernetes/server/bin/*
chmod 0755 kubernetes/server/bin/*
# 这里需要先拷贝kubectl到/usr/local/bin目录下，用于生成kubeconfig文件
cp kubernetes/server/bin/kubectl /usr/local/bin/kubectl

etcd

# 下载etcd二进制包
echo "--- 下载etcd ${ETCD_VERSION} 二进制包 ---"
wget https://github.com/etcd-io/etcd/releases/download/${ETCD_VERSION}/etcd-${ETCD_VERSION}-linux-amd64.tar.gz
tar xzf etcd-${ETCD_VERSION}-linux-amd64.tar.gz \
        etcd-${ETCD_VERSION}-linux-amd64/etcdctl \
        etcd-${ETCD_VERSION}-linux-amd64/etcd
chown root:root etcd-${ETCD_VERSION}-linux-amd64/etcdctl etcd-${ETCD_VERSION}-linux-amd64/etcd
chmod 0755 etcd-${ETCD_VERSION}-linux-amd64/etcdctl etcd-${ETCD_VERSION}-linux-amd64/etcd

CNI-Plugin

# 下载CNI-plugin
echo "--- 下载cni-plugins ${CNI_PLUGIN_VERSION} 二进制包 ---"
wget https://github.com/containernetworking/plugins/releases/download/${CNI_PLUGIN_VERSION}/cni-plugins-amd64-${CNI_PLUGIN_VERSION}.tgz
mkdir /root/software/cni-plugins
tar xzf cni-plugins-amd64-${CNI_PLUGIN_VERSION}.tgz -C /root/software/cni-plugins/

生成集群Certificates

说明

本次部署，需要为etcd-server、etcd-client、kube-apiserver、kube-controller-manager、kube-scheduler、kube-proxy生成证书。另外还需要生成sa、front-proxy-ca、front-proxy-client证书用于集群的其他功能。

要注意CA JSON文件的CN(Common Name)与O(Organization)等内容是会影响Kubernetes组件认证的。
- CN Common Name，kube-apiserver会从证书中提取该字段作为请求的用户名（User Name）
- O Oragnization，kube-apiserver会从证书中提取该字段作为请求用户的所属组（Group）
CA是自签名根证书，用来给后续各种证书签名
kubernetes集群的所有状态信息都保存在etcd中，kubernetes组件会通过kube-apiserver读写etcd里面的信息
etcd如果暴露在公网且没做SSL/TLS验证，那么任何人都能读写数据，那么很可能会无端端在kubernetes集群里面多了挖坑Pod或者肉鸡Pod
本文使用CFSSL创建证书，证书有效期10年
建立证书过程在k8s-m1上完成
用于生成证书的JSON文件已经打包好在这里pki.zip

下载CFSSL工具

wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -O /usr/local/bin/cfssl-certinfo
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O /usr/local/bin/cfssl
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O /usr/local/bin/cfssljson
chmod 755 /usr/local/bin/cfssl-certinfo \
          /usr/local/bin/cfssl \
          /usr/local/bin/cfssljson

创建工作目录

1 2	mkdir -p /root/pki /root/master /root/node cd /root/pki

创建用于生成证书的json文件

ca-config.json

cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      },
      "etcd-server": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      },
      "etcd-client": {
          "usages": [
              "signing",
              "key encipherment",
              "client auth"
          ],
          "expiry": "87600h"
      }
    }
  }
}
EOF

ca-csr.json

cat > ca-csr.json <<EOF
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Guangdong",
      "L": "Guangzhou",
      "O": "Kubernetes",
      "OU": "System"
    }
  ]
}
EOF

etcd-ca-csr.json

cat > etcd-ca-csr.json <<EOF
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Guangdong",
      "L": "Guangzhou",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ]
}
EOF

etcd-server-csr.json

cat > etcd-server-csr.json <<EOF
{
  "CN": "etcd-server",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Guangdong",
      "L": "Guangzhou",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ]
}
EOF

etcd-client-csr.json

cat > etcd-client-csr.json <<EOF
{
  "CN": "etcd-client",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "hosts": [
    ""
  ],
  "names": [
    {
      "C": "CN",
      "ST": "Guangdong",
      "L": "Guangzhou",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ]
}
EOF

kube-apiserver-csr.json

cat > kube-apiserver-csr.json <<EOF
{
  "CN": "kube-apiserver",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Guangdong",
      "L": "Guangzhou",
      "O": "Kubernetes",
      "OU": "System"
    }
  ]
}
EOF

kube-manager-csr.json

cat > kube-manager-csr.json <<EOF
{
  "CN": "system:kube-controller-manager",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Guangdong",
      "L": "Guangzhou",
      "O": "system:kube-controller-manager",
      "OU": "System"
    }
  ]
}
EOF

kube-scheduler-csr.json

cat > kube-scheduler-csr.json <<EOF
{
  "CN": "system:kube-scheduler",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Guangdong",
      "L": "Guangzhou",
      "O": "system:kube-scheduler",
      "OU": "System"
    }
  ]
}
EOF

kube-proxy-csr.json

cat > kube-proxy-csr.json <<EOF
{
  "CN": "system:kube-proxy",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Guangdong",
      "L": "Guangzhou",
      "O": "system:kube-proxy",
      "OU": "System"
    }
  ]
}
EOF

kube-admin-csr.json

cat > kube-admin-csr.json <<EOF
{
  "CN": "admin",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Guangdong",
      "L": "Guangzhou",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
EOF

front-proxy-ca-csr.json

cat > front-proxy-ca-csr.json <<EOF
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  }
}
EOF

front-proxy-client-csr.json

cat > front-proxy-client-csr.json <<EOF
{
  "CN": "front-proxy-client",
  "key": {
    "algo": "rsa",
    "size": 2048
  }
}
EOF

sa-csr.json

cat > sa-csr.json <<EOF
{
  "CN": "service-accounts",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Guangdong",
      "L": "Guangzhou",
      "O": "Kubernetes",
      "OU": "System"
    }
  ]
}
EOF

创建etcd证书

etcd-ca证书

1 2	echo '--- 创建etcd-ca证书 ---' cfssl gencert -initca etcd-ca-csr.json \| cfssljson -bare etcd-ca

etcd-server证书

echo '--- 创建etcd-server证书 ---'
cfssl gencert \
      -ca=etcd-ca.pem \
      -ca-key=etcd-ca-key.pem \
      -config=ca-config.json \
      -hostname=127.0.0.1,$(xargs -n1<<<${MasterArray[@]} | sort  | paste -d, -s -) \
      -profile=etcd-server etcd-server-csr.json | cfssljson -bare etcd-server

etcd-client证书

echo '--- 创建etcd-client证书 ---'
cfssl gencert \
      -ca=etcd-ca.pem \
      -ca-key=etcd-ca-key.pem \
      -config=ca-config.json \
      -profile=etcd-client etcd-client-csr.json | cfssljson -bare etcd-client

创建kubernetes证书

kubernetes-CA 证书

1
2
3

echo '--- 创建kubernetes-ca证书 ---'
# 创建kubernetes-ca证书
cfssl gencert -initca ca-csr.json | cfssljson -bare kube-ca

kube-apiserver证书

echo '--- 创建kube-apiserver证书 ---'
# 创建kube-apiserver证书
# 这里的hostname字段中的10.96.0.1要跟上文提到的service cluster ip cidr对应
cfssl gencert \
      -ca=kube-ca.pem \
      -ca-key=kube-ca-key.pem \
      -config=ca-config.json \
      -hostname=10.96.0.1,127.0.0.1,localhost,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,${VIP},$(xargs -n1<<<${MasterArray[@]} | sort  | paste -d, -s -) \
      -profile=kubernetes \
      kube-apiserver-csr.json | cfssljson -bare kube-apiserver

kube-controller-manager证书

echo '--- 创建kube-controller-manager证书 ---'
# 创建kube-controller-manager证书
cfssl gencert \
      -ca=kube-ca.pem \
      -ca-key=kube-ca-key.pem \
      -config=ca-config.json \
      -profile=kubernetes \
      kube-manager-csr.json | cfssljson -bare kube-controller-manager

kube-scheduler证书

echo '--- 创建kube-scheduler证书 ---'
# 创建kube-scheduler证书
cfssl gencert \
      -ca=kube-ca.pem \
      -ca-key=kube-ca-key.pem \
      -config=ca-config.json \
      -profile=kubernetes \
      kube-scheduler-csr.json | cfssljson -bare kube-scheduler

kube-proxy证书

echo '--- 创建kube-proxy证书 ---'
# 创建kube-proxy证书
cfssl gencert \
      -ca=kube-ca.pem \
      -ca-key=kube-ca-key.pem \
      -config=ca-config.json \
      -profile=kubernetes \
      kube-proxy-csr.json | cfssljson -bare kube-proxy

kube-admin证书

echo '--- 创建kube-admin证书 ---'
# 创建kube-admin证书
cfssl gencert \
      -ca=kube-ca.pem \
      -ca-key=kube-ca-key.pem \
      -config=ca-config.json \
      -profile=kubernetes \
      kube-admin-csr.json | cfssljson -bare kube-admin

Front Proxy证书

echo '--- 创建Front Proxy Certificate证书 ---'
# 创建Front Proxy Certificate证书
cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare front-proxy-ca
cfssl gencert \
      -ca=front-proxy-ca.pem \
      -ca-key=front-proxy-ca-key.pem \
      -config=ca-config.json \
      -profile=kubernetes \
      front-proxy-client-csr.json | cfssljson -bare front-proxy-client

Service Account证书

echo '--- 创建service account证书 ---'
# 创建创建service account证书
cfssl gencert \
      -ca=kube-ca.pem \
      -ca-key=kube-ca-key.pem \
      -config=ca-config.json \
      -profile=kubernetes \
      sa-csr.json | cfssljson -bare sa

bootstrap-token

# 生成 Bootstrap Token
BOOTSTRAP_TOKEN_ID=$(head -c 6 /dev/urandom | md5sum | head -c 6)
BOOTSTRAP_TOKEN_SECRET=$(head -c 16 /dev/urandom | md5sum | head -c 16)
BOOTSTRAP_TOKEN="${BOOTSTRAP_TOKEN_ID}.${BOOTSTRAP_TOKEN_SECRET}"
echo "Bootstrap Token: ${BOOTSTRAP_TOKEN}"

encryption.yaml

ENCRYPTION_TOKEN=$(head -c 32 /dev/urandom | base64)
echo "ENCRYPTION_TOKEN: ${ENCRYPTION_TOKEN}"

# 创建encryption.yaml文件
cat > encryption.yaml <<EOF
kind: EncryptionConfig
apiVersion: v1
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: ${ENCRYPTION_TOKEN}
      - identity: {}
EOF

audit-policy.yaml

这里使用最低限度的审计策略文件

echo '--- 创建创建高级审计配置 ---'
# 创建高级审计配置
cat >> audit-policy.yaml <<EOF
# Log all requests at the Metadata level.
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
EOF

创建kubeconfig文件

说明

kubeconfig 文件用于组织关于集群、用户、命名空间和认证机制的信息。
命令行工具 kubectl 从 kubeconfig 文件中得到它要选择的集群以及跟集群 API server 交互的信息。
默认情况下，kubectl 会从 $HOME/.kube 目录下查找文件名为 config 的文件。

注意： 用于配置集群访问信息的文件叫作 kubeconfig文件，这是一种引用配置文件的通用方式，并不是说它的文件名就是 kubeconfig。

Components kubeconfig

for TARGET in kube-controller-manager kube-scheduler kube-admin; do
    kubectl config set-cluster kubernetes \
        --certificate-authority=kube-ca.pem \
        --embed-certs=true \
        --server=${KUBE_APISERVER} \
        --kubeconfig=${TARGET}.kubeconfig
    kubectl config set-credentials system:${TARGET} \
        --client-certificate=${TARGET}.pem \
        --client-key=${TARGET}-key.pem \
        --embed-certs=true \
        --kubeconfig=${TARGET}.kubeconfig
    kubectl config set-context default \
        --cluster=kubernetes \
        --user=system:${TARGET} \
        --kubeconfig=${TARGET}.kubeconfig
    kubectl config use-context default --kubeconfig=${TARGET}.kubeconfig
done

Bootstrap kubeconfig

# 设置集群参数
kubectl config set-cluster kubernetes \
  --certificate-authority=kube-ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig

# 设置上下文参数
kubectl config set-context kubelet-bootstrap \
  --cluster=${CLUSTER_NAME} \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig

# 设置客户端认证参数
kubectl config set-credentials kubelet-bootstrap \
  --token=${BOOTSTRAP_TOKEN} \
  --kubeconfig=bootstrap.kubeconfig

# 设置当前使用的上下文
kubectl config use-context kubelet-bootstrap --kubeconfig=bootstrap.kubeconfig

清理证书CSR文件

1 2	echo '--- 删除.csr文件 ---' rm -rf csr

修改文件权限

1
2
3

chown root:root *pem *kubeconfig *yaml
chmod 0444 *pem *kubeconfig *yaml
chmod 0400 *key.pem

Etcd Cluster

说明

Kubernetes集群数据需要存放在etcd中
etcd集群部署在master节点上
部署三节点的etcd cluster
etcd集群启用基于TLS的客户端身份验证+集群节点身份认证

准备环境

查看集群部署的环境变量

这里主要检查上面集群变量定义的时候，是否有遗漏或者不正确

1 2	echo "ETCD_SERVERS=${ETCD_SERVERS}" echo "ETCD_INITIAL_CLUSTER=${ETCD_INITIAL_CLUSTER}"

添加用户

etcd以普通用户运行

echo '--- master节点添加etcd用户 ---'
for NODE in "${!MasterArray[@]}";do
    echo "--- $NODE ---"
    ssh ${NODE} /usr/sbin/useradd -r -s /bin/false etcd
done

创建目录

echo '--- master节点创建目录 ---'
for NODE in "${!MasterArray[@]}";do
    echo "--- $NODE ---"
    echo "--- 创建目录 ---"
    ssh ${NODE} /usr/bin/mkdir -p /etc/etcd/ssl \
                                  /var/lib/etcd
    echo "--- 修改目录权限 ---"
    ssh ${NODE} /usr/bin/chmod 0755 /etc/etcd \
                                    /etc/etcd/ssl \
                                    /var/lib/etcd
    echo "--- 修改目录属组 ---"
    ssh ${NODE} chown -R etcd:etcd /etc/etcd/ /var/lib/etcd
done

创建工作目录

1	mkdir -p /root/master

切换工作目录

1	cd /root/master

部署etcd

创建systemd服务脚本

cat > etcd.service <<EOF
[Unit]
Description=Etcd Service
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target

[Service]
User=etcd
Type=notify
ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yaml
Restart=on-failure
RestartSec=60
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
Alias=etcd3.service
EOF

创建etcd配置模板

cat > etcd.config.yaml.example <<EOF
# This is the configuration file for the etcd server.
# Human-readable name for this member.
name: '{HOSTNAME}'
# Path to the data directory.
data-dir: '/var/lib/etcd/{HOSTNAME}.data/'
# Path to the dedicated wal directory.
wal-dir: '/var/lib/etcd/{HOSTNAME}.wal/'
# Number of committed transactions to trigger a snapshot to disk.
snapshot-count: 5000
# Time (in milliseconds) of a heartbeat interval.
heartbeat-interval: 100
# Time (in milliseconds) for an election to timeout.
election-timeout: 1000
# Raise alarms when backend size exceeds the given quota. 0 means use the
# default quota.
quota-backend-bytes: 0
# List of comma separated URLs to listen on for peer traffic.
listen-peer-urls: 'https://{PUBLIC_IP}:2380'
# List of comma separated URLs to listen on for client traffic.
listen-client-urls: 'https://{PUBLIC_IP}:2379,http://127.0.0.1:2379'
# Maximum number of snapshot files to retain (0 is unlimited).
max-snapshots: 3
# Maximum number of wal files to retain (0 is unlimited).
max-wals: 5
# Comma-separated white list of origins for CORS (cross-origin resource sharing).
cors:
# List of this member's peer URLs to advertise to the rest of the cluster.
# The URLs needed to be a comma-separated list.
initial-advertise-peer-urls: 'https://{PUBLIC_IP}:2380'
# List of this member's client URLs to advertise to the public.
# The URLs needed to be a comma-separated list.
advertise-client-urls: 'https://{PUBLIC_IP}:2379'
# Discovery URL used to bootstrap the cluster.
discovery:
# Valid values include 'exit', 'proxy'
discovery-fallback: 'proxy'
# HTTP proxy to use for traffic to discovery service.
discovery-proxy:
# DNS domain used to bootstrap initial cluster.
discovery-srv:
# Initial cluster configuration for bootstrapping.
initial-cluster: '${ETCD_INITIAL_CLUSTER}'
# Initial cluster token for the etcd cluster during bootstrap.
initial-cluster-token: 'etcd-k8s-cluster'
# Initial cluster state ('new' or 'existing').
initial-cluster-state: 'new'
# Reject reconfiguration requests that would cause quorum loss.
strict-reconfig-check: false
# Accept etcd V2 client requests
enable-v2: true
# Enable runtime profiling data via HTTP server
enable-pprof: true
# Valid values include 'on', 'readonly', 'off'
proxy: 'off'
# Time (in milliseconds) an endpoint will be held in a failed state.
proxy-failure-wait: 5000
# Time (in milliseconds) of the endpoints refresh interval.
proxy-refresh-interval: 30000
# Time (in milliseconds) for a dial to timeout.
proxy-dial-timeout: 1000
# Time (in milliseconds) for a write to timeout.
proxy-write-timeout: 5000
# Time (in milliseconds) for a read to timeout.
proxy-read-timeout: 0
client-transport-security:
  # Path to the client server TLS cert file.
  cert-file: '/etc/etcd/ssl/etcd-server.pem'
  # Path to the client server TLS key file.
  key-file: '/etc/etcd/ssl/etcd-server-key.pem'
  # Enable client cert authentication.
  client-cert-auth: true
  # Path to the client server TLS trusted CA cert file.
  trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  # Client TLS using generated certificates
  auto-tls: true
peer-transport-security:
  # Path to the peer server TLS cert file.
  cert-file: '/etc/etcd/ssl/etcd-server.pem'
  # Path to the peer server TLS key file.
  key-file: '/etc/etcd/ssl/etcd-server-key.pem'
  # Enable peer client cert authentication.
  client-cert-auth: true
  # Path to the peer server TLS trusted CA cert file.
  trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  # Peer TLS using generated certificates.
  auto-tls: true
# Enable debug-level logging for etcd.
debug: false
logger: 'zap'
# Specify 'stdout' or 'stderr' to skip journald logging even when running under systemd.
log-outputs: [default]
# Force to create a new one member cluster.
force-new-cluster: false
auto-compaction-mode: 'periodic'
auto-compaction-retention: 1
# Set level of detail for exported metrics, specify 'extensive' to include histogram metrics.
# default is 'basic'
metrics: 'extensive'
EOF

分发etcd集群文件

for NODE in "${!MasterArray[@]}";do
    echo "---- $NODE ----"
    echo "--- 分发etcd二进制文件 ---"
    rsync -avpt /root/software/etcd-${ETCD_VERSION}-linux-amd64/etcdctl \
                /root/software/etcd-${ETCD_VERSION}-linux-amd64/etcd \
                ${NODE}:/usr/local/bin/
    echo '---- 分发etcd证书 ----'
    rsync -avpt /root/pki/etcd-ca-key.pem \
                /root/pki/etcd-ca.pem \
                /root/pki/etcd-client-key.pem \
                /root/pki/etcd-client.pem \
                /root/pki/etcd-server-key.pem \
                /root/pki/etcd-server.pem \
                ${NODE}:/etc/etcd/ssl/
    echo '---- 创建etcd服务脚本 ----'
    sed -e "s/{HOSTNAME}/${NODE}/g" \
        -e "s/{PUBLIC_IP}/${MasterArray[$NODE]}/g" \
        etcd.config.yaml.example > etcd.config.yaml.${NODE}
    echo '---- 分发etcd服务脚本 ----'
    rsync -avpt etcd.config.yaml.${NODE} ${NODE}:/etc/etcd/etcd.config.yaml
    rsync -avpt etcd.service ${NODE}:/usr/lib/systemd/system/etcd.service
    ssh ${NODE} "chown -R etcd:etcd /etc/etcd && systemctl daemon-reload"
    # 清理配置文件
    rm -rf etcd.config.yaml.${NODE}
done

启动etcd cluster

etcd 进程首次启动时会等待其它节点的 etcd 加入集群，命令 systemctl start etcd 会卡住一段时间，为正常现象
启动之后可以通过etcdctl命令查看集群状态

for NODE in "${!MasterArray[@]}";do
    echo "--- $NODE ${MasterArray[$NODE]} ---"
    ssh ${NODE} systemctl enable etcd.service
    ssh ${NODE} systemctl start etcd.service &
done

为方便维护，可使用alias简化etcdctl命令

cat >> /root/.bashrc <<EOF
alias etcdctl2="export ETCDCTL_API=2;etcdctl --ca-file '/etc/etcd/ssl/etcd-ca.pem' --cert-file '/etc/etcd/ssl/etcd-client.pem' --key-file '/etc/etcd/ssl/etcd-client-key.pem' --endpoints ${ETCD_SERVERS}"
alias etcdctl3="export ETCDCTL_API=3;etcdctl --cacert=/etc/etcd/ssl/etcd-ca.pem --cert=/etc/etcd/ssl/etcd-client.pem --key=/etc/etcd/ssl/etcd-client-key.pem --endpoints=${ETCD_SERVERS}"
EOF

验证etcd集群状态

etcd提供v2和v3两套API
kubernetes使用的是etcd v3 API

应用环境变量

应用上面定义的alias

1	source /root/.bashrc

使用v2 API访问集群

获取集群状态

1	etcdctl2 cluster-health

示例输出

member 222fd3b0bb4a5931 is healthy: got healthy result from https://172.16.80.203:2379
member 8349ef180b115a83 is healthy: got healthy result from https://172.16.80.201:2379
member f525d2d797a7c465 is healthy: got healthy result from https://172.16.80.202:2379
cluster is healthy

获取成员列表

1	etcdctl2 member list

示例输出

1
2
3

222fd3b0bb4a5931: name=k8s-m3 peerURLs=https://172.16.80.203:2380 clientURLs=https://172.16.80.203:2379 isLeader=false
8349ef180b115a83: name=k8s-m1 peerURLs=https://172.16.80.201:2380 clientURLs=https://172.16.80.201:2379 isLeader=false
f525d2d797a7c465: name=k8s-m2 peerURLs=https://172.16.80.202:2380 clientURLs=https://172.16.80.202:2379 isLeader=true

使用v3 API访问集群

获取集群endpoint状态

1	etcdctl3 endpoint health

示例输出

1
2
3

https://172.16.80.201:2379 is healthy: successfully committed proposal: took = 2.879402ms
https://172.16.80.203:2379 is healthy: successfully committed proposal: took = 6.708566ms
https://172.16.80.202:2379 is healthy: successfully committed proposal: took = 7.187607ms

获取集群成员列表

1	etcdctl3 member list --write-out=table

示例输出

+------------------+---------+--------+----------------------------+----------------------------+
|        ID        | STATUS  |  NAME  |         PEER ADDRS         |        CLIENT ADDRS        |
+------------------+---------+--------+----------------------------+----------------------------+
| 222fd3b0bb4a5931 | started | k8s-m3 | https://172.16.80.203:2380 | https://172.16.80.203:2379 |
| 8349ef180b115a83 | started | k8s-m1 | https://172.16.80.201:2380 | https://172.16.80.201:2379 |
| f525d2d797a7c465 | started | k8s-m2 | https://172.16.80.202:2380 | https://172.16.80.202:2379 |
+------------------+---------+--------+----------------------------+----------------------------+

配置定时备份脚本

#!/bin/sh
TIME=$(date +%Y%m%d)
HOUR=$(date +%H)
BASE_DIR="/path/to/etcd_backup"
BACKUP_DIR="${BASE_DIR}/${TIME}"
mkdir -p $BACKUP_DIR
export ETCDCTL_API=3
/usr/local/bin/etcdctl \
    --cacert=/etc/etcd/ssl/etcd-ca.pem \
    --cert=/etc/etcd/ssl/etcd-client.pem \
    --key=/etc/etcd/ssl/etcd-client-key.pem \
    --endpoints=https://172.16.80.201:2379,https://172.16.80.201:2379,https://172.16.80.201:2379 \
    snapshot save $BACKUP_DIR/etcd-snapshot-${HOUR}.db
# 清理15天前的etcd备份
find ${BASE_DIR} -type d -mtime +15 -exec rm -rf {} \;

Kubernetes Masters

Keepalived和HAProxy

说明

HAProxy

提供多个 API Server 的负载均衡(Load Balance)
监听VIP的8443端口负载均衡到三台master节点的6443端口

Keepalived

提供虚拟IP位址(VIP),来让vip落在可用的master主机上供所有组件访问master节点
提供健康检查脚本用于切换VIP

切换工作目录

1	cd /root/master

安装Keepalived和HAProxy

for NODE in "${!MasterArray[@]}";do
    echo "---- $NODE ----"
    echo "---- 安装haproxy和keepalived ----"
    ssh $NODE yum install keepalived haproxy -y
done

Keepalived

创建配置模板

cat > keepalived.conf.example <<EOF
vrrp_script haproxy-check {
    script "/bin/bash /etc/keepalived/check_haproxy.sh"
    interval 3
    weight -2
    fall 10
    rise 2
}

vrrp_instance haproxy-vip {
    state BACKUP
    priority 101
    interface {{ VIP_IFACE }}
    virtual_router_id 47
    advert_int 3

    unicast_peer {
    }

    virtual_ipaddress {
        {{ VIP }}
    }

    track_script {
        haproxy-check
    }
}
EOF

生成配置文件

通过集群变量替换配置模板的字符串，然后重定向到新的配置文件

sed -r -e "s#\{\{ VIP \}\}#${VIP}#" \
       -e "s#\{\{ VIP_IFACE \}\}#${VIP_IFACE}#" \
       -e '/unicast_peer/r '<(xargs -n1<<<${MasterArray[@]} | sort | sed 's#^#\t#') \
       keepalived.conf.example > keepalived.conf

创建keepalived服务检查脚本

cat > check_haproxy.sh <<EOF
#!/bin/bash
VIRTUAL_IP=${VIP}

errorExit() {
    echo "*** $*" 1>&2
    exit 1
}

if ip addr | grep -q \$VIRTUAL_IP ; then
    curl -s --max-time 2 --insecure https://\${VIRTUAL_IP}:8443/ -o /dev/null || errorExit "Error GET https://\${VIRTUAL_IP}:8443/"
fi
EOF

HAproxy

创建配置模板

cat > haproxy.cfg.example <<EOF
global
  maxconn  2000
  ulimit-n  16384
  log  127.0.0.1 local0 err
  stats timeout 30s

defaults
  log global
  mode  http
  option  httplog
  timeout connect 5000
  timeout client  50000
  timeout server  50000
  timeout http-request 15s
  timeout http-keep-alive 15s

frontend monitor-in
  bind ${VIP}:33305
  mode http
  option httplog
  monitor-uri /monitor

listen stats
  bind    ${VIP}:8006
  mode    http
  stats   enable
  stats   hide-version
  stats   uri       /stats
  stats   refresh   30s
  stats   realm     Haproxy\ Statistics
  stats   auth      admin:admin

frontend k8s-api
  bind ${VIP}:8443
  mode tcp
  option tcplog
  tcp-request inspect-delay 5s
  default_backend k8s-api

backend k8s-api
  mode tcp
  option tcplog
  option tcp-check
  balance roundrobin
  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
EOF

生成配置文件

通过集群变量替换配置模板的字符串，然后重定向到新的配置文件

1	sed -e '$r '<(paste <( seq -f' server k8s-api-%g' ${#MasterArray[@]} ) <( xargs -n1<<<${MasterArray[@]} \| sort \| sed 's#$#:6443 check#')) haproxy.cfg.example > haproxy.cfg

分发配置文件

for NODE in "${!MasterArray[@]}";do
    echo "---- $NODE ----"
	rsync -avpt haproxy.cfg $NODE:/etc/haproxy/
	rsync -avpt keepalived.conf \
	            check_haproxy.sh \
	            $NODE:/etc/keepalived/
done

启动keepalived和HAProxy服务

for NODE in "${!MasterArray[@]}";do
    echo "---- $NODE ----"
	ssh $NODE systemctl enable --now keepalived.service haproxy.service
done

验证服务状态

需要大约十秒的时间等待keepalived和haproxy服务起来
这里由于后端的kube-apiserver服务还没启动，只测试是否能ping通VIP
如果VIP没起来，就要去确认一下各master节点的keepalived服务是否正常

1	sleep 15 && ping -c 4 $VIP

Kubernetes Master服务

切换工作目录

1	cd /root/master

添加用户

echo '--- master节点添加用户 ---'
for NODE in "${!MasterArray[@]}";do
    echo "--- $NODE ---"
    ssh ${NODE} /usr/sbin/useradd -r -s /bin/false kube
done

创建程序运行目录

echo '--- master节点创建目录 ---'
for NODE in "${!MasterArray[@]}";do
    echo "--- $NODE ---"
    echo "--- 创建目录 ---"
    ssh ${NODE} /usr/bin/mkdir -p /etc/kubernetes \
                                  /etc/kubernetes/pki \
                                  /etc/kubernetes/manifests \
                                  /var/run/kubernetes \
                                  /var/log/kube-audit
    echo "--- 修改目录权限 ---"
    ssh ${NODE} /usr/bin/chmod 0755 /etc/kubernetes \
                                    /etc/kubernetes/pki \
                                    /etc/kubernetes/manifests \
                                    /var/log/kube-audit \
                                    /var/run/kubernetes
    echo "--- 修改目录属组 ---"
    ssh ${NODE} chown -R kube:kube /etc/kubernetes \
                                   /etc/kubernetes/pki \
                                   /etc/kubernetes/manifests \
                                   /var/run/kubernetes \
                                   /var/log/kube-audit
done

kube-apiserver

说明

以 REST APIs 提供 Kubernetes 资源的 CRUD,如授权、认证、存取控制与 API 注册等机制。
关闭默认非安全端口8080,在安全端口 6443 接收 https 请求
严格的认证和授权策略 (x509、token、RBAC)
开启 bootstrap token 认证，支持 kubelet TLS bootstrapping
使用 https 访问 kubelet、etcd，加密通信

配置参数

--allow-privileged=true启用容器特权模式
--apiserver-count=3指定集群运行模式，其它节点处于阻塞状态
--audit-policy-file=/etc/kubernetes/audit-policy.yaml 基于audit-policy.yaml文件定义的内容启动审计功能
--authorization-mode=Node,RBAC开启 Node 和 RBAC 授权模式，拒绝未授权的请求
--disable-admission-plugins=和--enable-admission-plugins禁用和启用准入控制插件。
准入控制插件会在请求通过认证和授权之后、对象被持久化之前拦截到达apiserver的请求。
准入控制插件依次执行，因此需要注意顺序。
如果插件序列中任何一个拒绝了请求，则整个请求将立刻被拒绝并返回错误给客户端。
关于admission-plugins官方文档里面有推荐配置，这里直接采用官方配置。
注意针对不同kubernetes版本都会有不一样的配置，具体可以看这里。
官方说明： For Kubernetes version 1.10 and later, the recommended admission controllers are enabled by default.
--enable-bootstrap-token-auth=true启用 kubelet bootstrap 的 token 认证
--experimental-encryption-provider-config=/etc/kubernetes/encryption.yaml启用加密特性将Secret数据加密存储到etcd
--insecure-port=0关闭监听非安全端口8080
--runtime-config=api/all=true启用所有版本的 APIs
--service-cluster-ip-range=10.96.0.0/12指定 Service Cluster IP 地址段
--service-node-port-range=30000-32767指定 NodePort 的端口范围
--target-ram-mb配置缓存大小，参考值为节点数*60
--event-ttl配置kubernets events的保留时间，默认1h0m0s

创建配置模板

cat > kube-apiserver.conf.example <<EOF
KUBE_APISERVER_ARGS=" \\
--advertise-address={PUBLIC_IP} \\
--allow-privileged=true \\
--apiserver-count=3 \\
--audit-log-format=json \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=1000 \\
--audit-log-mode=blocking \\
--audit-log-path=/var/log/kube-audit/audit.log \\
--audit-policy-file=/etc/kubernetes/audit-policy.yaml \\
--authorization-mode=Node,RBAC \\
--bind-address=0.0.0.0 \\
--client-ca-file=/etc/kubernetes/pki/kube-ca.pem \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,PersistentVolumeClaimResize,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota \\
--enable-aggregator-routing=true \\
--enable-bootstrap-token-auth=true \\
--enable-garbage-collector=true \\
--etcd-compaction-interval=0s \\
--etcd-cafile=/etc/kubernetes/pki/etcd-ca.pem \\
--etcd-certfile=/etc/kubernetes/pki/etcd-client.pem \\
--etcd-keyfile=/etc/kubernetes/pki/etcd-client-key.pem \\
--etcd-prefix=/registry \\
--etcd-servers=${ETCD_SERVERS} \\
--encryption-provider-config=/etc/kubernetes/encryption.yaml \\
--endpoint-reconciler-type=lease \\
--event-ttl=24h0m0s \\
--feature-gates=PodShareProcessNamespace=true,ExpandPersistentVolumes=true \\
--insecure-port=0 \\
--kubelet-client-certificate=/etc/kubernetes/pki/kube-apiserver.pem \\
--kubelet-client-key=/etc/kubernetes/pki/kube-apiserver-key.pem \\
--kubelet-https=true \\
--kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP \\
--kubelet-timeout=3s \\
--logtostderr=true \\
--max-mutating-requests-inflight=500 \\
--max-requests-inflight=1500 \\
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \\
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \\
--requestheader-allowed-names=aggregator,front-proxy-client \\
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \\
--requestheader-extra-headers-prefix=X-Remote-Extra- \\
--requestheader-group-headers=X-Remote-Group \\
--requestheader-username-headers=X-Remote-User \\
--runtime-config=api/all=true \\
--secure-port=6443 \\
--service-account-key-file=/etc/kubernetes/pki/sa.pem \\
--service-cluster-ip-range=10.96.0.0/12 \\
--service-node-port-range=30000-32767 \\
--storage-backend=etcd3 \\
--storage-media-type=application/vnd.kubernetes.protobuf \\
--target-ram-mb=300 \\
--tls-cert-file=/etc/kubernetes/pki/kube-apiserver.pem \\
--tls-private-key-file=/etc/kubernetes/pki/kube-apiserver-key.pem \\
--v=2 \\
"
EOF

创建systemd服务脚本

cat > kube-apiserver.service <<EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target etcd.service

[Service]
User=kube
Type=simple
EnvironmentFile=-/etc/kubernetes/kube-apiserver.conf
ExecStart=/usr/local/bin/kube-apiserver \$KUBE_APISERVER_ARGS
Restart=on-failure
RestartSec=60
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

kube-controller-manager

说明

通过核心控制循环(Core Control Loop)监听 Kubernetes API
的资源来维护集群的状态，这些资源会被不同的控制器所管理，如 Replication Controller、Namespace
Controller 等等。而这些控制器会处理着自动扩展、滚动更新等等功能。
关闭非安全端口，在安全端口 10252 接收 https 请求
使用 kubeconfig 访问 kube-apiserver 的安全端口

配置参数

--allocate-node-cidrs=true在cloud provider上分配和设置pod的CIDR
--cluster-cidr集群内的pod的CIDR范围，需要 --allocate-node-cidrs设为true
--experimental-cluster-signing-duration=8670h0m0s指定 TLS Bootstrap 证书的有效期
--feature-gates=RotateKubeletServerCertificate=true开启 kublet server 证书的自动更新特性
--horizontal-pod-autoscaler-use-rest-clients=true能够使用自定义资源（Custom Metrics）进行自动水平扩展
--leader-elect=true集群运行模式，启用选举功能，被选为 leader 的节点负责处理工作，其它节点为阻塞状态
--node-cidr-mask-size=24集群中node cidr的掩码
--service-cluster-ip-range=10.96.0.0/16指定 Service Cluster IP 网段，必须和 kube-apiserver 中的同名参数一致
--terminated-pod-gc-thresholdexit状态的pod超过多少会触发gc

创建配置文件

cat > kube-controller-manager.conf <<EOF
KUBE_CONTROLLER_MANAGER_ARGS=" \\
--address=0.0.0.0 \\
--allocate-node-cidrs=true \\
--authentication-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
--authorization-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
--cluster-cidr=$POD_NET_CIDR \\
--cluster-signing-cert-file=/etc/kubernetes/pki/kube-ca.pem \\
--cluster-signing-key-file=/etc/kubernetes/pki/kube-ca-key.pem \\
--concurrent-service-syncs=10 \\
--concurrent-serviceaccount-token-syncs=20 \\
--controllers=*,bootstrapsigner,tokencleaner \\
--enable-garbage-collector=true \\
--experimental-cluster-signing-duration=8670h0m0s \\
--feature-gates=PodShareProcessNamespace=true,ExpandPersistentVolumes=true \\
--horizontal-pod-autoscaler-sync-period=10s \\
--horizontal-pod-autoscaler-use-rest-clients=true \\
--kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
--leader-elect=true \\
--logtostderr=true \\
--node-cidr-mask-size=24 \\
--node-monitor-grace-period=40s \\
--node-monitor-period=5s \\
--pod-eviction-timeout=2m0s \\
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \\
--root-ca-file=/etc/kubernetes/pki/kube-ca.pem \\
--service-account-private-key-file=/etc/kubernetes/pki/sa-key.pem \\
--service-cluster-ip-range=${SVC_CLUSTER_CIDR} \\
--terminated-pod-gc-threshold=12500 \\
--use-service-account-credentials=true \\
--v=2 \\
"
EOF

创建systemd服务脚本

cat > kube-controller-manager.service <<EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target kube-apiserver.service

[Service]
User=kube
Type=simple
EnvironmentFile=-/etc/kubernetes/kube-controller-manager.conf
ExecStart=/usr/local/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_ARGS
Restart=on-failure
RestartSec=60
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

kube-scheduler

说明

负责将一个(或多个)容器依据调度策略分配到对应节点上让容器引擎(如 Docker)执行。
调度受到 QoS 要求、软硬性约束、亲和性(Affinity)等等因素影响。

配置参数

--algorithm-provider=DefaultProvider使用默认调度算法
--leader-elect=true集群运行模式，启用选举功能，被选为 leader 的节点负责处理工作，其它节点为阻塞状态

创建配置文件

cat > kube-scheduler.conf <<EOF
KUBE_SCHEDULER_ARGS="\\
--address=0.0.0.0 \\
--algorithm-provider=DefaultProvider \\
--kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \\
--leader-elect=true \\
--logtostderr=true \\
--v=2 \\
"
EOF

创建systemd服务脚本

cat > kube-scheduler.service <<EOF
[Unit]
Description=Kubernetes Scheduler Plugin
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target kube-apiserver.service

[Service]
User=kube
Type=simple
EnvironmentFile=-/etc/kubernetes/kube-scheduler.conf
ExecStart=/usr/local/bin/kube-scheduler \$KUBE_SCHEDULER_ARGS
Restart=on-failure
RestartSec=60
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

分发配置文件

for NODE in "${!MasterArray[@]}";do
    echo "--- $NODE ${MasterArray[$NODE]} ---"
    echo '---- 分发kubeconfig文件 yaml文件 ----'
    rsync -avpt /root/pki/kube-admin.kubeconfig \
                /root/pki/kube-controller-manager.kubeconfig \
                /root/pki/kube-scheduler.kubeconfig \
                /root/pki/audit-policy.yaml \
                /root/pki/encryption.yaml \
                $NODE:/etc/kubernetes/
    echo '---- 分发sa证书 kube证书 front-proxy证书 ----'
    rsync -avpt /root/pki/etcd-ca.pem \
                /root/pki/etcd-client-key.pem \
                /root/pki/etcd-client.pem \
                /root/pki/front-proxy-ca.pem \
                /root/pki/front-proxy-client-key.pem \
                /root/pki/front-proxy-client.pem \
                /root/pki/kube-apiserver-key.pem \
                /root/pki/kube-apiserver.pem \
                /root/pki/kube-ca.pem \
                /root/pki/kube-ca-key.pem \
                /root/pki/sa-key.pem \
                /root/pki/sa.pem \
                $NODE:/etc/kubernetes/pki/
    echo '---- 分发kubernetes components服务脚本----'
    rsync -avpt kube*service $NODE:/usr/lib/systemd/system/
    echo '---- 分发kubernetes components配置----'
    sed -e "s/{PUBLIC_IP}/${MasterArray[$NODE]}/g" kube-apiserver.conf.example > kube-apiserver.conf.${NODE}
    rsync -avpt kube-apiserver.conf.${NODE} $NODE:/etc/kubernetes/kube-apiserver.conf
    rsync -avpt kube-controller-manager.conf $NODE:/etc/kubernetes/kube-controller-manager.conf
    rsync -avpt kube-scheduler.conf $NODE:/etc/kubernetes/kube-scheduler.conf
    rm -rf kube-apiserver.conf.${NODE}
    ssh $NODE systemctl daemon-reload
    ssh $NODE chown -R kube:kube /etc/kubernetes
done

分发master组件二进制文件

echo '--- 分发kubernetes和etcd二进制文件 ---'
for NODE in "${!MasterArray[@]}";do
    echo "--- $NODE ---"
    rsync -avpt /root/software/kubernetes/server/bin/kube-controller-manager \
                /root/software/kubernetes/server/bin/hyperkube \
                /root/software/kubernetes/server/bin/mounter \
                /root/software/kubernetes/server/bin/kubelet \
                /root/software/kubernetes/server/bin/kubectl \
                /root/software/kubernetes/server/bin/kube-scheduler \
                /root/software/kubernetes/server/bin/kube-proxy \
                /root/software/kubernetes/server/bin/kube-apiserver \
                /root/software/kubernetes/server/bin/cloud-controller-manager \
                /root/software/kubernetes/server/bin/kubeadm \
                /root/software/kubernetes/server/bin/apiextensions-apiserver \
                $NODE:/usr/local/bin/
done

启动Kubernetes master服务

for NODE in "${!MasterArray[@]}";do
    echo "--- $NODE ${MasterArray[$NODE]} ---"
    ssh $NODE "systemctl enable --now kube-apiserver.service"
    # sleep 10秒，让apiserver先初始化完成
    sleep 10
    ssh $NODE "systemctl enable --now kube-controller-manager.service"
    ssh $NODE "systemctl enable --now kube-scheduler.service"
done

验证Kubernetes集群服务

检查集群components状态

1	kubectl --kubeconfig=/etc/kubernetes/kube-admin.kubeconfig get cs

输出示例

NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok                  
scheduler            Healthy   ok                  
etcd-2               Healthy   {"health":"true"}   
etcd-0               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"}

检查集群endpoints状态

1	kubectl --kubeconfig=/etc/kubernetes/kube-admin.kubeconfig get endpoints

输出示例

1 2	NAME ENDPOINTS AGE kubernetes 172.16.80.201:6443,172.16.80.202:6443,172.16.80.203:6443 12m

查看ETCD的数据

Kubernetes集群数据会写入到etcd，这里检查一下是否能正常写入到etcd
检查方式，查看etcd里面的key

export ETCDCTL_API=3
etcdctl --cacert=/etc/etcd/ssl/etcd-ca.pem \
        --cert=/etc/etcd/ssl/etcd-client.pem \
        --key=/etc/etcd/ssl/etcd-client-key.pem \
        --endpoints=${ETCD_SERVERS} \
        get / --prefix --keys-only

输出示例

/registry/apiregistration.k8s.io/apiservices/v1.
/registry/apiregistration.k8s.io/apiservices/v1.apps
/registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.autoscaling
/registry/apiregistration.k8s.io/apiservices/v1.batch
/registry/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.networking.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.storage.k8s.io
...
/registry/serviceaccounts/kube-system/service-account-controller
/registry/serviceaccounts/kube-system/service-controller
/registry/serviceaccounts/kube-system/statefulset-controller
/registry/serviceaccounts/kube-system/token-cleaner
/registry/serviceaccounts/kube-system/ttl-controller
/registry/services/endpoints/default/kubernetes
/registry/services/endpoints/kube-system/kube-controller-manager
/registry/services/endpoints/kube-system/kube-dns
/registry/services/endpoints/kube-system/kube-scheduler
/registry/services/specs/default/kubernetes
/registry/services/specs/kube-system/kube-dns

配置kubectl

集群配置文件

kubectl默认会加载~/.kube/config文件，作为默认连接Kubernetes集群的凭证

for NODE in "${!MasterArray[@]}";do
    echo "--- $NODE ${MasterArray[$NODE]} ---"
    ssh $NODE mkdir -p /root/.kube
    rsync -avpt /root/pki/kube-admin.kubeconfig $NODE:/root/.kube/config
done

kubectl命令补齐

临时生效

1	source <(kubectl completion bash)

开机自动加载

for NODE in "${!MasterArray[@]}";do
    echo "--- $NODE ${MasterArray[$NODE]} ---"
    echo "--- kubectl命令自动补全 ---"
    ssh $NODE "kubectl completion bash > /etc/bash_completion.d/kubectl"
done

配置TLS Bootstrap

当集群开启了 TLS 认证后，每个节点的 kubelet 组件都要使用由 apiserver 使用的 CA 签发的有效证书才能与
apiserver 通讯
如果节点多起来，为每个节点单独签署证书将是一件非常繁琐的事情
TLS bootstrapping 功能就是让 kubelet 先使用一个预定的低权限用户连接到 apiserver，然后向 apiserver 申请证书，kubelet 的证书由 apiserver 动态签署；

创建工作目录

1 2	mkdir -p /root/yaml/ cd /root/yaml/

创建TLS Bootstrap Token对象

kubectl -n kube-system create secret generic bootstrap-token-${BOOTSTRAP_TOKEN_ID} \
        --type 'bootstrap.kubernetes.io/token' \
        --from-literal description="cluster bootstrap token" \
        --from-literal token-id=${BOOTSTRAP_TOKEN_ID} \
        --from-literal token-secret=${BOOTSTRAP_TOKEN_SECRET} \
        --from-literal usage-bootstrap-authentication=true \
        --from-literal usage-bootstrap-signing=true \
        --dry-run -o yaml > /root/yaml/tls-bootstrap-token.yaml
kubectl apply -f /root/yaml/tls-bootstrap-token.yaml

创建YAML文件

cat > /root/yaml/kubelet-tls-bootstrap-rbac.yaml <<EOF
# 允许 kubelet tls bootstrap 创建 csr 请求
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: create-csrs-for-bootstrapping
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node-bootstrapper
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers
---
# 自动批准 system:bootstrappers 组用户 TLS bootstrapping 首次申请证书的 CSR 请求
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: auto-approve-csrs-for-group
subjects:
- kind: Group
  name: system:bootstrappers
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
  apiGroup: rbac.authorization.k8s.io
---
# 自动批准 system:nodes 组用户更新 kubelet 自身与 apiserver 通讯证书的 CSR 请求
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: auto-approve-renewals-for-nodes
subjects:
- kind: Group
  name: system:nodes
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
  apiGroup: rbac.authorization.k8s.io
EOF

创建RBAC规则

1	kubectl apply -f /root/yaml/kubelet-tls-bootstrap-rbac.yaml

kube-apiserver获取node信息的权限

说明

本文部署的kubelet关闭了匿名访问，因此需要额外为kube-apiserver添加权限用于访问kubelet的信息

若没添加此RBAC，则kubectl在执行logs、exec等指令的时候会提示401 Forbidden

1 2	kubectl -n kube-system logs calico-node-pc8lq Error from server (Forbidden): Forbidden (user=kube-apiserver, verb=get, resource=nodes, subresource=proxy) ( pods/log calico-node-pc8lq)

参考文档：Kubelet的认证授权

创建YAML文件

cat > /root/yaml/apiserver-to-kubelet-rbac.yaml <<EOF
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - proxy
- apiGroups:
  - ""
  resources:
  - nodes/log
  - nodes/metrics
  - nodes/proxy
  - nodes/spec
  - nodes/stats
  verbs:
  - '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kube-apiserver
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: systemctl:kubelet-api-admin
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kubelet-api-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: kube-apiserver
---
EOF

创建RBAC规则

1	kubectl apply -f /root/yaml/apiserver-to-kubelet-rbac.yaml

Kubernetes Nodes

说明

节点配置

安装Docker-ce，配置与master节点一致即可
安装cni-plugins、kubelet、kube-proxy
关闭防火墙和SELINUX
kubelet运行需要root权限
这里是以k8s-m1、k8s-m2、k8s-m3、k8s-n1、k8s-n2作为Work节点加入集群

kubelet

管理容器生命周期、节点状态监控
目前 kubelet 支持三种数据源来获取节点Pod信息：
- 本地文件
- 通过 url 从网络上某个地址来获取信息
- API Server：从 kubernetes master 节点获取信息
使用kubeconfig与kube-apiserver通信
这里启用TLS-Bootstrap实现kubelet证书动态签署证书，并自动生成kubeconfig

kube-proxy

Kube-proxy是实现Service的关键插件，kube-proxy会在每台节点上执行，然后监听API Server的Service与Endpoint资源物件的改变，然后来依据变化调用相应的组件来实现网路的转发
kube-proxy可以使用userspace（基本已废弃）、iptables（默认方式）和ipvs来实现数据报文的转发
这里使用的是性能更好、适合大规模使用的ipvs
以DaemonSet方式运行

kubelet配置

说明

kubelet在1.10版本开始动态配置特性进入Beta阶段，因此绝大部分配置被标记为DEPRECATED，官方推荐使用--config指定配置文件，并在配置文件里面指定原来命令行中配置的内容。
因此kubelet的配置实际被分割成两个部分，启动参数和动态配置参数

创建目录

1	mkdir -p /root/node

切换工作目录

1	cd /root/node

配置文件

cat > kubelet.config <<EOF
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    # 这里写kubernetes-ca证书的路径
    clientCAFile: /etc/kubernetes/pki/kube-ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
# cgroups的驱动，可选systemd和cgroupfs
cgroupDriver: systemd
cgroupsPerQOS: true
# 指定Pod的DNS地址，这里的地址需要指向CoreDNS的SVC地址
clusterDNS:
- 10.96.0.10
# 集群域名
clusterDomain: cluster.local
configMapAndSecretChangeDetectionStrategy: Cache
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuCFSQuotaPeriod: 100ms
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
# 达到某些阈值之后，kubelet会驱逐Pod
# A set of eviction thresholds (e.g. memory.available<1Gi) that if met would trigger a pod eviction.
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
# 检测到系统已启用swap分区时kubelet会启动失败
failSwapOn: false
# 定义feature gates
featureGates:
  # 禁用Alpha的功能
  AllAlpha: false 
# 检查kubelet配置文件变更的间隔
fileCheckFrequency: 20s
# 允许endpoint在尝试访问自己的服务时会被负载均衡分发到自身
# 可选值"promiscuous-bridge", "hairpin-veth" and "none"
# 默认值为promiscuous-bridge
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
# 这里定义容器镜像触发回收空间的上限值和下限值
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
# kubelet进程最大能打开的文件数量
maxOpenFiles: 1048576
# 当前节点kubelet所能运行的最大Pod数量
maxPods: 110
nodeLeaseDurationSeconds: 40
# node状态上报间隔
nodeStatusReportFrequency: 1m0s
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
# Pod PID数量限制，默认-1，即无限制
podPidsLimit: -1
# kubelet服务端口
port: 10250
registryBurst: 10
registryPullQPS: 5
# 指定域名解析文件
resolvConf: /etc/resolv.conf
rotateCertificates: true
rotateServerCertificates: true
runtimeRequestTimeout: 2m0s
# 拉镜像时，同一时间只拉取一个镜像，即串行化
# We recommend *not* changing the default value on nodes that run docker daemon with version < 1.9 or an Aufs storage backend. Issue #10959 has more details. (default true)
serializeImagePulls: false
# 静态Pod的manifest目录
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s
EOF

systemd服务脚本

--bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig指定bootstrap启动时使用的kubeconfig
--network-plugin=cni定义网络插件，Pod生命周期使用此网络插件
--node-labels=node-role.kubernetes.io/node=''kubelet注册当前Node时设置的Label，以key=value的格式表示，多个labe以逗号分隔
--pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.1Pod的pause镜像

cat > kubelet.service <<EOF
[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/usr/local/bin/kubelet \\
               --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \\
               --cert-dir=/etc/kubernetes/ssl \\
               --config=/etc/kubernetes/kubelet.config \\
               --cni-conf-dir=/etc/cni/net.d \\
               --cni-bin-dir=/opt/cni/bin \\
               --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\
               --logtostderr=true \\
               --network-plugin=cni \\
               --pod-infra-container-image=gcrxio/pause:3.1 \\
               --v=2
KillMode=process
Restart=on-failure
RestartSec=60
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

创建程序目录

for NODE in "${!HostArray[@]}";do
    echo "--- $NODE ---"
    echo "--- 创建目录 ---"
    ssh $NODE mkdir -p /opt/cni/bin \
                       /etc/cni/net.d \
                       /etc/kubernetes/pki \
                       /etc/kubernetes/ssl \
                       /etc/kubernetes/manifests \
                       /var/lib/kubelet
done

分发文件

for NODE in "${!HostArray[@]}";do
    echo "--- ${NODE} ---"
    echo "--- 分发kubernetes二进制文件 ---"
    rsync -avpt /root/software/kubernetes/server/bin/kubelet \
                ${NODE}:/usr/local/bin/
    echo "--- 分发CNI-Plugins ---"
    rsync -avpt /root/software/cni-plugins/* \
                ${NODE}:/opt/cni/bin/
    echo "--- 分发证书和kubeconfig文件 ---"
    rsync -avpt /root/pki/bootstrap.kubeconfig \
                ${NODE}:/etc/kubernetes/
    rsync -avpt /root/pki/kube-ca.pem \
                /root/pki/front-proxy-ca.pem \
                ${NODE}:/etc/kubernetes/pki/
    echo "--- 分发配置文件 ---"
    rsync -avpt kubelet.config ${NODE}:/etc/kubernetes/
    echo "--- 分发systemd服务脚本 ---"
    rsync -avpt kubelet.service ${NODE}:/usr/lib/systemd/system/
    ssh ${NODE} systemctl daemon-reload
done

启动kubelet服务

for NODE in "${!HostArray[@]}";do
    echo "--- $NODE ---"
    ssh $NODE systemctl enable docker.service kubelet.service
    ssh $NODE systemctl start docker.service kubelet.service
done

获取集群节点信息

此时由于未按照网络插件，所以节点状态为NotReady

1	kubectl get node -o wide

输出示例

NAME      STATUS     ROLES     AGE       VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION              CONTAINER-RUNTIME
k8s-m1    NotReady   node      12s       v1.14.3   172.16.80.201   <none>        CentOS Linux 7 (Core)   3.10.0-957.21.3.el7.x86_64   docker://18.9.6
k8s-m2    NotReady   node      12s       v1.14.3   172.16.80.202   <none>        CentOS Linux 7 (Core)   3.10.0-957.21.3.el7.x86_64   docker://18.9.6
k8s-m3    NotReady   node      12s       v1.14.3   172.16.80.203   <none>        CentOS Linux 7 (Core)   3.10.0-957.21.3.el7.x86_64   docker://18.9.6
k8s-n1    NotReady   node      12s       v1.14.3   172.16.80.204   <none>        CentOS Linux 7 (Core)   3.10.0-957.21.3.el7.x86_64   docker://18.9.6
k8s-n2    NotReady   node      12s       v1.14.3   172.16.80.205   <none>        CentOS Linux 7 (Core)   3.10.0-957.21.3.el7.x86_64   docker://18.9.6

节点标签

master节点

声明污点，避免没有声明容忍该污点的Pod被调度到master节点

1 2	kubectl label node ${!MasterArray[@]} node-role.kubernetes.io/master="" kubectl taint nodes ${!MasterArray[@]} node-role.kubernetes.io/master="":NoSchedule

node节点

1	kubectl label nodes ${!NodeArray[@]} node-role.kubernetes.io/node=""

Kubernetes Core Addons

Kube-Proxy

说明

Kube-proxy是实现Service的关键插件,kube-proxy会在每台节点上执行,然后监听API
Server的Service与Endpoint资源物件的改变,然后来依据变化执行iptables来实现网路的转发。
本文使用DaemonSet来运行Kube-Proxy组件。

创建工作目录

1	mkdir -p /root/yaml/CoreAddons/kube-proxy

切换工作目录

1	cd /root/yaml/CoreAddons/kube-proxy

创建YAML文件

ServicAccount

cat > /root/yaml/CoreAddons/kube-proxy/kube-proxy-sa.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-proxy
  namespace: kube-system
EOF

ConfigMap

cat > /root/yaml/CoreAddons/kube-proxy/kube-proxy-cm.yaml <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: kube-proxy
  name: kube-proxy
  namespace: kube-system
data:
  config.conf: |-
    apiVersion: kubeproxy.config.k8s.io/v1alpha1
    kind: KubeProxyConfiguration
    bindAddress: 0.0.0.0
    clientConnection:
      acceptContentTypes: ""
      burst: 10
      contentType: application/vnd.kubernetes.protobuf
      kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
      qps: 5
    # 集群中pod的CIDR范围，从这个范围以外发送到服务集群IP的流量将被伪装
    # 从POD发送到外部LoadBalanceIP的流量将被定向到各自的集群IP
    clusterCIDR: 10.244.0.0/16
    # 来自apiserver配置的刷新频率
    configSyncPeriod: 15m0s
    conntrack:
      max: null
      # 每个核心最大能跟踪的NAT连接数，默认32768
      maxPerCore: 32768
      min: 131072
      # 处于CLOSE_WAIT状态的TCP连接超时时间
      tcpCloseWaitTimeout: 1h0m0s
      # 已建立的 TCP 连接的空闲超时
      tcpEstablishedTimeout: 24h0m0s
    # 通过Web接口/debug/pprof启用性能分析
    enableProfiling: false
    featureGates:
      SupportIPVSProxyMode: true
    healthzBindAddress: 0.0.0.0:10256
    hostnameOverride: ""
    iptables:
      # SNAT所有通过服务集群ip发送的通信
      masqueradeAll: false
      masqueradeBit: 14
      minSyncPeriod: 0s
      # ipvs 规则刷新的最大时间间隔
      syncPeriod: 30s
    ipvs:
      excludeCIDRs: null
      minSyncPeriod: 0s
      # ipvs调度类型，默认是rr
      scheduler: rr
      strictARP: false
      syncPeriod: 30s
    metricsBindAddress: 127.0.0.1:10249
    # 配置kube-proxy代理模式，可选iptables和ipvs，默认是iptables
    mode: ipvs
    nodePortAddresses: null
    oomScoreAdj: -999
    portRange: ""
    resourceContainer: /kube-proxy
    udpIdleTimeout: 250ms
  kubeconfig.conf: |-
    apiVersion: v1
    kind: Config
    clusters:
    - cluster:
        certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        server: ${KUBE_APISERVER}
      name: default
    contexts:
    - context:
        cluster: default
        namespace: default
        user: default
      name: default
    current-context: default
    users:
    - name: default
      user:
        tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
EOF

DaemonSet

cat > /root/yaml/CoreAddons/kube-proxy/kube-proxy-ds.yaml <<EOF
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    k8s-app: kube-proxy
  name: kube-proxy
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: kube-proxy
  template:
    metadata:
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ""
      labels:
        k8s-app: kube-proxy
    spec:
      containers:
      - command:
        - /usr/local/bin/kube-proxy
        - --config=/var/lib/kube-proxy/config.conf
        - --hostname-override=\$(NODE_NAME)
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        image: gcrxio/kube-proxy:v1.14.3
        imagePullPolicy: IfNotPresent
        name: kube-proxy
        resources: {}
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /var/lib/kube-proxy
          name: kube-proxy
        - mountPath: /run/xtables.lock
          name: xtables-lock
        - mountPath: /lib/modules
          name: lib-modules
          readOnly: true
        - mountPath: /etc/localtime
          name: timezone-volume
          readOnly: true
      hostNetwork: true
      priorityClassName: system-node-critical
      serviceAccountName: kube-proxy
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
      - operator: Exists
      volumes:
      - configMap:
          name: kube-proxy
        name: kube-proxy
      - hostPath:
          path: /run/xtables.lock
          type: FileOrCreate
        name: xtables-lock
      - hostPath:
          path: /lib/modules
        name: lib-modules
      - hostPath:
          path: /usr/share/zoneinfo/Asia/Shanghai
        name: timezone-volume
  updateStrategy:
    type: RollingUpdate
EOF

ClusterRoleBinding

cat > /root/yaml/CoreAddons/kube-proxy/kube-proxy-rbac.yaml <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubeadm:node-proxier
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node-proxier
subjects:
- kind: ServiceAccount
  name: kube-proxy
  namespace: kube-system
EOF

部署Kube-Proxy

1	kubectl apply -f /root/yaml/CoreAddons/kube-proxy/

验证Kube-Proxy

查看Pod

1	kubectl -n kube-system get pod -l k8s-app=kube-proxy

查看IPVS规则

1	ipvsadm -ln

查看Kube-Proxy代理模式

1	curl localhost:10249/proxyMode

网络插件

说明

只要符合CNI规范的网络组件都可以给kubernetes使用
网络组件清单可以在这里看到Network Plugins
这里只列举kube-flannel和calico，flannel和calico的区别可以自己去找资料
网络组件只能选一个来部署
本文使用kube-flannel部署网络组件，calico已测试可用
在k8s-m1上操作

创建工作目录

1	mkdir -p /root/yaml/CoreAddons/network-plugin/{kube-flannel,calico}

kube-flannel

说明

kube-flannel基于VXLAN的方式创建容器二层网络，使用端口8472/UDP通信
flannel 第一次启动时，从 etcd 获取 Pod 网段信息，为本节点分配一个未使用的 /24 段地址，然后创建 flannel.1（也可能是其它名称，如 flannel1 等）接口。
官方提供yaml文件部署为DeamonSet
若需要使用NetworkPolicy功能，可以关注这个项目canal

架构图

切换工作目录

1	cd /root/yaml/CoreAddons/network-plugin/kube-flannel

下载yaml文件

1	wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

官方yaml文件包含多个平台的daemonset，包括amd64、arm64、arm、ppc64le、s390x
这里以amd64作为例子，其他的可以自行根据需要修改或者直接删除不需要的daemonset
官方yaml文件已经配置好容器网络为10.244.0.0/16，这里需要跟kube-controller-manager.conf里面的--cluster-cidr匹配
如果在kube-controller-manager.conf里面把--cluster-cidr改成了其他地址段，例如192.168.0.0/16，用以下命令替换kube-flannel.yaml相应的字段

1	sed -e 's,"Network": "10.244.0.0/16","Network": "192.168.0.0/16," -i kube-flannel.yml

如果服务器有多个网卡，需要指定网卡用于flannel通信，以网卡ens33为例
- 在args下面添加一行- --iface=ens33

containers:
- name: kube-flannel
  command:
  - /opt/bin/flanneld
  args:
  - --ip-masq
  - --kube-subnet-mgr
  - --iface=ens33

修改backend

flannel支持多种后端实现，可选值为VXLAN、host-gw、UDP
从性能上，host-gw是最好的，VXLAN和UDP次之
默认值是VXLAN，这里以修改为host-gw为例，位置大概在75行左右

net-conf.json: |
  {
    "Network": "10.244.0.0/16",
    "Backend": {
      "Type": "host-gw"
    }
  }

部署kube-flannel

1	kubectl apply -f kube-flannel.yml

检查部署情况

1	kubectl -n kube-system get pod -l app=flannel

输出示例

NAME                          READY   STATUS                  RESTARTS   AGE
kube-flannel-ds-amd64-4v8q7   1/1     Running                 0          3m46s
kube-flannel-ds-amd64-6lq5q   1/1     Running                 0          3m47s
kube-flannel-ds-amd64-cczp4   1/1     Running                 0          3m45s
kube-flannel-ds-amd64-wjpkg   1/1     Running                 0          3m44s
kube-flannel-ds-amd64-z2lm8   1/1     Running                 0          3m46s

如果等很久都没Running，可能是你所在的网络环境访问quay.io速度太慢了
可以替换一下镜像，重新apply

1 2	sed -e 's,quay.io/coreos/,zhangguanzhang/quay.io.coreos.,g' -i kube-flannel.yml kubectl apply -f kube-flannel.yaml

Calico

说明

Calico 是一款纯 Layer 3 的网络，节点之间基于BGP协议来通信。
这里以calico-v3.7.0来作为示例
部署文档

架构图

切换工作目录

1	cd /root/yaml/CoreAddons/network-plugin/calico

下载yaml文件

这里选用的是【Installing with the Kubernetes API datastore—50 nodes or less】

1	wget https://docs.projectcalico.org/v3.7/manifests/calico.yaml

修改YAML文件

calico-node服务的主要参数如下

CALICO_IPV4POOL_CIDR配置Calico IPAM的IP地址池，默认是192.168.0.0/16

# The default IPv4 pool to create on startup if none exists. Pod IPs will be
# chosen from this range. Changing this value after installation will have
# no effect. This should fall within `--cluster-cidr`.
- name: CALICO_IPV4POOL_CIDR
  value: "192.168.0.0/16"

CALICO_IPV4POOL_IPIP 配置是否使用IPIP模式，默认是打开的

1
2
3

# Enable IPIP
- name: CALICO_IPV4POOL_IPIP
  value: "Always"

部署Calico

1	kubectl apply -f /root/yaml/CoreAddons/network-plugin/calico/

检查部署情况

检查calico-node

1	kubectl -n kube-system get pod -l k8s-app=calico-node

输出示例

NAME                READY     STATUS    RESTARTS   AGE
calico-node-fjcj4   2/2       Running   0          6m
calico-node-tzppt   2/2       Running   0          6m
calico-node-zdq64   2/2       Running   0          6m
calico-node-2hdqf   2/2       Running   0          6m
calico-node-jh6vd   2/2       Running   0          6m

检查节点状态

网络组件部署完成之后，可以看到node状态已经为Ready

kubectl get node 
NAME      STATUS    ROLES     AGE       VERSION
k8s-m1    Ready     node      1d        v1.14.3
k8s-m2    Ready     node      1d        v1.14.3
k8s-m3    Ready     node      1d        v1.14.3
k8s-n1    Ready     node      1d        v1.14.3
k8s-n2    Ready     node      1d        v1.14.3

服务发现组件部署

kubernetes从v1.11之后，已经使用CoreDNS取代原来的KUBE DNS作为服务发现的组件
CoreDNS 是由 CNCF 维护的开源 DNS 方案，前身是 SkyDNS
配置文件以kubeadm生成的YAML文件作为模板，再添加额外配置

创建工作目录

1	mkdir -p /root/yaml/CoreAddons/coredns

切换工作目录

1	cd /root/yaml/CoreAddons/coredns

CoreDNS

创建yaml文件

ServiceAccount

cat > /root/yaml/CoreAddons/coredns/coredns-sa.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: coredns
  namespace: kube-system
EOF

ConfigMap

cat > /root/yaml/CoreAddons/coredns/coredns-cm.yaml <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        log
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        proxy . /etc/resolv.conf
        cache 30
        reload
        loadbalance
    }
EOF

RBAC

cat > /root/yaml/CoreAddons/coredns/coredns-rbac.yaml <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:coredns
rules:
- apiGroups:
  - ""
  resources:
  - endpoints
  - services
  - pods
  - namespaces
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:coredns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:coredns
subjects:
- kind: ServiceAccount
  name: coredns
  namespace: kube-system
EOF

Deployment

cat > /root/yaml/CoreAddons/coredns/coredns-dp.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: kube-dns
  name: coredns
  namespace: kube-system
spec:
  replicas: 2
  selector:
    matchLabels:
      k8s-app: kube-dns
  strategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        k8s-app: kube-dns
    spec:
      affinity:
        podAntiAffinity: 
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: k8s-app 
                  operator: In 
                  values:
                  - kube-dns
              topologyKey: kubernetes.io/hostname
      containers:
      - args:
        - -conf
        - /etc/coredns/Corefile
        image: gcrxio/coredns:1.2.6
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          successThreshold: 1
          timeoutSeconds: 5
        name: coredns
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9153
          name: metrics
          protocol: TCP
        resources:
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - all
          readOnlyRootFilesystem: true
        volumeMounts:
        - mountPath: /etc/coredns
          name: config-volume
          readOnly: true
        - mountPath: /etc/localtime
          name: timezone-volume
          readOnly: true
      dnsPolicy: Default
      serviceAccountName: coredns
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
      volumes:
      - configMap:
          items:
          - key: Corefile
            path: Corefile
          name: coredns
        name: config-volume
      - hostPath:
          path: /usr/share/zoneinfo/Asia/Shanghai
        name: timezone-volume
EOF

Service

cat > /root/yaml/CoreAddons/coredns/coredns-svc.yaml <<EOF
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/port: "9153"
    prometheus.io/scrape: "true"
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: KubeDNS
  name: kube-dns
  namespace: kube-system
spec:
  clusterIP: 10.96.0.10
  ports:
  - name: dns
    port: 53
    protocol: UDP
    targetPort: 53
  - name: dns-tcp
    port: 53
    protocol: TCP
    targetPort: 53
  - name: metrics
    port: 9153
    protocol: TCP
  selector:
    k8s-app: kube-dns
EOF

修改yaml文件

yaml文件里面定义了clusterIP这里需要与kubelet.config.file里面定义的cluster-dns一致
如果kubelet.conf里面的--cluster-dns改成别的，例如x.x.x.x，这里也要做相应变动，不然Pod找不到DNS，无法正常工作
这里定义静态的hosts解析，这样Pod可以通过hostname来访问到各节点主机
用下面的命令根据HostArray的信息生成静态的hosts解析

sed -e '16r '<(\
    echo '        hosts {'; \
    for NODE in "${!HostArray[@]}";do \
        echo "          ${HostArray[$NODE]} $NODE"; \
    done;\
    echo '          fallthrough'; \
    echo '        }';) \
-i /root/yaml/CoreAddons/coredns/coredns-cm.yaml

上面的命令的作用是，通过HostArray的信息生成hosts解析配置，顺序是打乱的，可以手工调整顺序
也可以手动修改coredns.yaml文件来添加对应字段

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        log
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
        }
        hosts {
          172.16.80.201 k8s-m1
          172.16.80.202 k8s-m2
          172.16.80.203 k8s-m3
          172.16.80.204 k8s-n1
          172.16.80.205 k8s-n2
          fallthrough
        }
        prometheus :9153
        proxy . /etc/resolv.conf
        cache 30
        reload
        loadbalance
    }

部署CoreDNS

1	kubectl apply -f /root/yaml/CoreAddons/coredns/

检查部署状态

1	kubectl -n kube-system get pod -l k8s-app=kube-dns

输出样例

1
2
3

NAME                       READY     STATUS    RESTARTS   AGE
coredns-5566c96697-6gzzc   1/1       Running   0          45s
coredns-5566c96697-q5slk   1/1       Running   0          45s

验证集群DNS服务

创建一个deployment测试DNS解析

cat > /root/yaml/busybox-deployment.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: busybox
  name: busybox
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
      - name: busybox
        imagePullPolicy: IfNotPresent
        image: busybox:1.26
        command:
        - sleep
        - "36000"
EOF

# 基于文件创建deployment
kubectl apply -f /root/yaml/busybox-deployment.yaml

检查deployment部署情况

1
2
3

kubectl get pod
NAME                       READY     STATUS    RESTARTS   AGE
busybox-7b9bfb5658-872gj   1/1       Running   0          6s

验证集群DNS解析
上一个命令获取到pod名字为busybox-7b9bfb5658-872gj
通过kubectl命令连接到Pod运行nslookup命令测试使用域名来访问kube-apiserver和各节点主机

echo "--- 通过CoreDNS访问kubernetes ---"
kubectl exec -it busybox-7b9bfb5658-4cz94 -- nslookup kubernetes
# 示例输出
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name:      kubernetes
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local


echo "--- 通过CoreDNS访问k8s-m1 ---"
# 示例输出
kubectl exec -it busybox-7b9bfb5658-4cz94 -- nslookup k8s-m1
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name:      k8s-m1
Address 1: 172.16.80.201 k8s-m1


echo "--- 通过CoreDNS访问k8s-m2 ---"
kubectl exec -it busybox-7b9bfb5658-4cz94 -- nslookup k8s-m2
# 示例输出
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name:      k8s-n2
Address 1: 172.16.80.202 k8s-m2


echo "--- 通过CoreDNS访问并不存在的k8s-n3 ---"
kubectl exec -it busybox-7b9bfb5658-4cz94 -- nslookup k8s-n3
# 示例输出
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
nslookup: can't resolve 'k8s-n3'

Metrics Server

Metrics Server
是实现了 Metrics API 的元件,其目标是取代 Heapster 作位 Pod 与 Node 提供资源的 Usage
metrics,该元件会从每个 Kubernetes 节点上的 Kubelet 所公开的 Summary API 中收集 Metrics
Horizontal Pod Autoscaler（HPA）控制器用于实现基于CPU使用率进行自动Pod伸缩的功能。
HPA控制器基于Master的kube-controller-manager服务启动参数–horizontal-pod-autoscaler-sync-period定义是时长（默认30秒）,周期性监控目标Pod的CPU使用率,并在满足条件时对ReplicationController或Deployment中的Pod副本数进行调整,以符合用户定义的平均Pod
CPU使用率。
在新版本的kubernetes中 Pod CPU使用率不在来源于heapster,而是来自于metrics-server
官网原话是 The –horizontal-pod-autoscaler-use-rest-clients is true or unset. Setting this to false switches to Heapster-based autoscaling, which is deprecated.

额外参数

设置kube-apiserver参数，这里在配置kube-apiserver阶段已经加进去了
front-proxy证书，在证书生成阶段已经完成且已分发

--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem
--requestheader-allowed-names=aggregator
--requestheader-group-headers=X-Remote-Group
--requestheader-extra-headers-prefix=X-Remote-Extra-
--requestheader-username-headers=X-Remote-User

创建工作目录

1	mkdir -p /root/yaml/CoreAddons/metrics-server

切换工作目录

1	cd /root/yaml/CoreAddons/metrics-server

下载yaml文件

wget https://raw.githubusercontent.com/kubernetes/kubernetes/v1.14.3/cluster/addons/metrics-server/auth-delegator.yaml
wget https://raw.githubusercontent.com/kubernetes/kubernetes/v1.14.3/cluster/addons/metrics-server/auth-reader.yaml
wget https://raw.githubusercontent.com/kubernetes/kubernetes/v1.14.3/cluster/addons/metrics-server/metrics-apiservice.yaml
wget https://raw.githubusercontent.com/kubernetes/kubernetes/v1.14.3/cluster/addons/metrics-server/metrics-server-service.yaml
wget https://raw.githubusercontent.com/kubernetes/kubernetes/v1.14.3/cluster/addons/metrics-server/resource-reader.yaml
wget https://raw.githubusercontent.com/kubernetes/kubernetes/v1.14.3/cluster/addons/metrics-server/metrics-server-deployment.yaml

修改metrics-server-deployment

修改镜像地址
修改metrics-server启动参数

containers:
- name: metrics-server
  image: gcrxio/metrics-server-amd64:v0.3.1
  command:
  - /metrics-server
  - --kubelet-preferred-address-types=Hostname,InternalDNS,InternalIP,ExternalDNS,ExternalIP
  - --kubelet-insecure-tls
  - --requestheader-extra-headers-prefix=x-remote-extra-
  - --requestheader-group-headers=x-remote-group
  - --requestheader-username-headers=x-remote-user
  - -v=2

修改resource-reader

默认配置的权限无法获取node节点信息，会提示403 Forbidden
需要在ClusterRole里面的rules[0].resources增加nodes/stats

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - nodes/stats
  - namespaces
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - "extensions"
  resources:
  - deployments
  verbs:
  - get
  - list
  - update
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:metrics-server
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

部署metrics-server

1	kubectl apply -f .

查看pod状态

1
2
3

kubectl -n kube-system get pod -l k8s-app=metrics-server
NAME                                  READY     STATUS    RESTARTS   AGE
pod/metrics-server-86bd9d7667-5hbn6   1/1       Running   0          1m

验证metrics

完成后,等待一段时间(约 30s - 1m)收集 Metrics

请求metrics api的结果

1	kubectl get --raw /apis/metrics.k8s.io/v1beta1

示例输出

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "nodes",
      "singularName": "",
      "namespaced": false,
      "kind": "NodeMetrics",
      "verbs": [
        "get",
        "list"
      ]
    },
    {
      "name": "pods",
      "singularName": "",
      "namespaced": true,
      "kind": "PodMetrics",
      "verbs": [
        "get",
        "list"
      ]
    }
  ]
}

获取节点性能数据

1	kubectl top node

输出示例

NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
k8s-m1   695m         17%    914Mi           11%       
k8s-m2   360m         9%     553Mi           7%        
k8s-m3   492m         12%    533Mi           6%        
k8s-n1   144m         3%     311Mi           3%        
k8s-n2   149m         3%     321Mi           4%

获取Pod性能数据

1	kubectl -n kube-system top pod

输出示例

NAME                                    CPU(cores)   MEMORY(bytes)   
coredns-56c964478c-g7q8n                14m          14Mi            
coredns-56c964478c-zmt2j                13m          14Mi            
kube-flannel-ds-amd64-8g6f9             6m           15Mi            
kube-flannel-ds-amd64-hslkh             6m           21Mi            
kube-flannel-ds-amd64-ppqkb             8m           14Mi            
kube-flannel-ds-amd64-swj8m             7m           14Mi            
kube-flannel-ds-amd64-zs2sd             8m           17Mi            
kube-proxy-2dq5x                        18m          23Mi            
kube-proxy-9lzmj                        28m          23Mi            
kube-proxy-9xtjc                        18m          23Mi            
kube-proxy-w7mtg                        2m           23Mi            
kube-proxy-zvp8d                        2m           21Mi            
metrics-server-v0.3.1-98bdfb766-s6jf6   8m           22Mi

#############################################################################

Kubernetes集群已基本可用

#############################################################################

Kubernetes ExtraAddons

说明

下面的部署流程，更多的是补充Kubernetes的能力
只记录简单的部署过程，不保证ctrl+c和ctrl+v能直接跑！

Dashboard

说明

Dashboard 是kubernetes社区提供的GUI界面，用于图形化管理kubernetes集群，同时可以看到资源报表。
官方提供yaml文件直接部署，但是需要更改image以便国内部署

创建工作目录

1	mkdir -p /root/yaml/ExtraAddons/kubernetes-dashboard

切换工作目录

1	cd /root/yaml/ExtraAddons/kubernetes-dashboard

获取yaml文件

1	wget https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml

修改镜像地址

1	sed -e 's,k8s.gcr.io/kubernetes-dashboard-amd64,gcrxio/kubernetes-dashboard-amd64,g' -i kubernetes-dashboard.yaml

创建kubernetes-Dashboard

1	kubectl apply -f /root/yaml/ExtraAddons/kubernetes-dashboard/kubernetes-dashboard.yaml

创建ServiceAccount RBAC

官方的yaml文件，ServiceAccount绑定的RBAC权限很低，很多资源无法查看
需要创建一个用于管理全局的ServiceAccount

cat<<EOF | kubectl apply -f -
---
# 在kube-system中创建名为admin-user的ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system
---
# 将admin-user和cluster-admin绑定在一起
# cluster-admin是kubernetes内置的clusterrole，具有集群管理员权限
# 其他内置的clusterrole可以通过kubectl get clusterrole查看
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kube-system
EOF

获取ServiceAccount的Token

1	kubectl -n kube-system describe secret $(kubectl -n kube-system get secret \| grep admin-user \| awk '{print $1}')

查看部署情况

1	kubectl get all -n kube-system --selector k8s-app=kubernetes-dashboard

访问Dashboard

kubernetes-dashborad的svc默认是clusterIP，需要修改为nodePort才能被外部访问
随机分配NodePort，分配范围由kube-apiserver的--service-node-port-range参数指定

1	kubectl patch -n kube-system svc kubernetes-dashboard -p '{"spec":{"type":"NodePort"}}'

修改完之后，通过以下命令获取访问kubernetes-Dashboard的端口

1
2
3

kubectl -n kube-system get svc --selector k8s-app=kubernetes-dashboard
NAME                   TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
kubernetes-dashboard   NodePort   10.106.183.192   <none>        443:30216/TCP   12s

可以看到已经将节点的30216端口暴露出来
IP地址不固定，只要运行了kube-proxy组件，都会在节点上添加30216端口规则用于转发请求到Pod
登录Dashboard，上面已经获取了token，这里只需要把token的值填入输入框，点击SIGN IN即可登录

Dashboard UI预览图

Ingress Controller

说明

Ingress 是 Kubernetes 中的一个抽象资源，其功能是通过 Web Server 的 Virtual Host
概念以域名(Domain Name)方式转发到內部 Service，这避免了使用 Service 中的 NodePort 与
LoadBalancer 类型所带來的限制(如 Port 数量上限)，而实现 Ingress 功能则是通过 Ingress Controller
来达成，它会负责监听 Kubernetes API 中的 Ingress 与 Service 资源，并在发生资源变化时，根据资源预期的结果来设置 Web Server。
Ingress Controller 有许多实现可以选择，这里只是列举一小部分
- Ingress NGINX：Kubernetes 官方维护的方案，本次安装使用此方案
- kubernetes-ingress：由nginx社区维护的方案，使用社区版nginx和nginx-plus
- treafik：一款开源的反向代理与负载均衡工具。它最大的优点是能够与常见的微服务系统直接整合，可以实现自动化动态配置

创建工作目录

1	mkdir -p /root/yaml/ExtraAddons/ingress/ingress-nginx

切换工作目录

1	cd /root/yaml/ExtraAddons/ingress/ingress-nginx

下载yaml文件

1
2

wget https://github.com/kubernetes/ingress-nginx/raw/nginx-0.24.1/deploy/mandatory.yaml
wget https://github.com/kubernetes/ingress-nginx/raw/nginx-0.24.1/deploy/provider/baremetal/service-nodeport.yaml

修改镜像地址

如果访问quay.io比较缓慢的话，可以修改一下镜像源

1 2	sed -e 's,quay.io/kubernetes-ingress-controller/,zhangguanzhang/quay.io.kubernetes-ingress-controller.,g' \ -i mandatory.yaml

创建ingress-nginx

1	kubectl apply -f .

检查部署情况

1	kubectl -n ingress-nginx get pod

访问ingress

获取ingres的nodeport端口

1	kubectl -n ingress-nginx get svc

输出示例

1 2	NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx NodePort 10.96.250.140 <none> 80:32603/TCP,443:30083/TCP 1m

默认的backend会返回404

1 2	curl http://172.16.80.200:32603 curl -k https://172.16.80.200:30083

输出示例

1	default backend - 404

注意

这里部署之后，是deployment，且通过nodePort暴露服务
也可以修改yaml文件，将Ingress-nginx部署为DaemonSet
- 使用labels和nodeSelector来指定运行ingress-nginx的节点
- 使用hostNetwork=true来共享主机网络命名空间，或者使用hostPort指定主机端口映射
- 如果使用hostNetwork共享宿主机网络栈或者hostPort映射宿主机端口，记得要看看有没有端口冲突，否则无法启动
- 修改监听端口可以在ingress-nginx启动命令中添加--http-port=8180和--https-port=8543，还有下面的端口定义也相应变更即可

创建kubernetes-Dashboard的Ingress

kubernetes-Dashboard默认是开启了HTTPS访问的
ingress-nginx需要以HTTPS的方式反向代理kubernetes-Dashboard
以HTTP方式访问kubernetes-Dashboard的时候会被重定向到HTTPS
需要创建HTTPS证书，用于访问ingress-nginx的HTTPS端口

创建HTTPS证书

这里的CN=域名/O=域名需要跟后面的ingress主机名匹配

openssl req -x509 \
            -nodes \
            -days 3650 \
            -newkey rsa:2048 \
            -keyout tls.key \
            -out tls.crt \
            -subj "/CN=dashboard.k8s.local/O=dashboard.k8s.local"

创建secret对象

这里将HTTPS证书创建为kubernetes的secret对象dashboard-tls
ingress创建的时候需要加载这个作为HTTPS证书

1	kubectl -n kube-system create secret tls dashboard-tls --key ./tls.key --cert ./tls.crt

创建dashboard-ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: dashboard-ingress
  namespace: kube-system
  annotations:
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/secure-backends: "true"
spec:
  tls:
  - hosts:
    - dashboard.k8s.local
    secretName: dashboard-tls
  rules:
  - host: dashboard.k8s.local
    http:
      paths:
        - path: /
          backend:
            serviceName: kubernetes-dashboard
            servicePort: 443

创建ingress

1	kubectl apply -f dashboard-ingress.yaml

检查ingress

1
2
3

kubectl -n kube-system get ingress
NAME                HOSTS                 ADDRESS         PORTS     AGE
dashboard-ingress   dashboard.k8s.local                   80, 443   16m

访问kubernetes-Dashboard

修改主机hosts静态域名解析，以本文为例在hosts文件里添加172.16.80.200 dashboard.k8s.local
使用https://dashboard.k8s.local:30083访问kubernetesDashboard了
添加了TLS之后，访问HTTP会被跳转到HTTPS端口，这里比较坑爹，没法自定义跳转HTTPS的端口
此处使用的是自签名证书，浏览器会提示不安全，请忽略
建议搭配external-DNS和LoadBalancer一起食用，效果更佳

效果图

Helm

Helm是一个kubernetes应用的包管理工具，用来管理charts——预先配置好的安装包资源，有点类似于Ubuntu的APT和CentOS中的yum。
Helm chart是用来封装kubernetes原生应用程序的yaml文件，可以在你部署应用的时候自定义应用程序的一些metadata，便与应用程序的分发。
Helm和charts的主要作用：
- 应用程序封装
- 版本管理
- 依赖检查
- 便于应用程序分发

环境要求

kubernetes v1.6及以上的版本，启用RBAC
集群可以访问到chart仓库
helm客户端主机能访问kubernetes集群

安装Helm

安装方式二选一，需要科学上网

直接脚本安装

1	curl -L https://git.io/get_helm.sh \| bash

下载二进制文件安装

1
2
3

wget -O - https://get.helm.sh/helm-v2.14.1-linux-amd64.tar.gz | tar xz linux-amd64/helm
mv linux-amd64/helm /usr/local/bin/helm
rm -rf linux-amd64

创建RBAC规则

cat << EOF | kubectl apply -f -
# 创建名为tiller的ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
# 给tiller绑定cluster-admin权限
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: tiller-cluster-rule
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: tiller
  namespace: kube-system
EOF

安装服务端

helm官方charts仓库 https://kubernetes-charts.storage.googleapis.com/ 需要翻墙访问
这里指定了helm的stable repo国内镜像地址，使用的是微软Azure的源
具体说明请看这里

1
2
3

helm init --tiller-image gcrxio/tiller:v2.14.1 \
          --service-account tiller \
          --stable-repo-url http://mirror.azure.cn/kubernetes/charts/

检查安装情况

查看Pod状态

1	kubectl -n kube-system get pod -l app=helm,name=tiller

输出示例

1 2	NAME READY STATUS RESTARTS AGE tiller-deploy-84fc6cd5f9-nz4m7 1/1 Running 0 1m

查看helm版本

1	helm version

输出示例

1
2

Client: &version.Version{SemVer:"v2.14.1", GitCommit:"d325d2a9c179b33af1a024cdb5a4472b6288016a", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.1", GitCommit:"d325d2a9c179b33af1a024cdb5a4472b6288016a", GitTreeState:"clean"}

本文至此结束

说明

在【二进制部署 kubernetes v1.11.x 高可用集群】里面对于ExtraAddons有额外的一些内容，例如Rook、Prometheus-Operator、ExternalDNS、EFK等等重新做整理适配实在是太麻烦了。后面再考虑另起文章专门来记录这些内容。

介绍

部署说明

参考博文

软件版本

网络信息

节点信息

目录说明

事前准备

编辑hosts文件

时间同步服务

关闭firewalld

关闭无用服务

关闭SELINUX

禁用swap

配置sysctl参数

更新软件包

安装软件包

配置开机加载ipvs模块

安装docker-ce

配置docker-ce

CentOS/RHEL 7.4+

Ubuntu 16.04

配置docker命令补全

配置docker服务开机自启动

查看docker信息

禁用docker源

确保以最新的内核启动系统

定义集群变量

下载所需软件包

创建工作目录

下载解压软件包

kubernetes

etcd

CNI-Plugin

生成集群Certificates

说明

下载CFSSL工具

创建工作目录

创建用于生成证书的json文件

ca-config.json

ca-csr.json

etcd-ca-csr.json

etcd-server-csr.json

etcd-client-csr.json

kube-apiserver-csr.json

kube-manager-csr.json

kube-scheduler-csr.json

kube-proxy-csr.json

kube-admin-csr.json

front-proxy-ca-csr.json

front-proxy-client-csr.json

sa-csr.json

创建etcd证书

etcd-ca证书

etcd-server证书

etcd-client证书

创建kubernetes证书

kubernetes-CA 证书

kube-apiserver证书

kube-controller-manager证书

kube-scheduler证书

kube-proxy证书

kube-admin证书

Front Proxy证书

Service Account证书

bootstrap-token

encryption.yaml

audit-policy.yaml

创建kubeconfig文件

说明

Components kubeconfig

Bootstrap kubeconfig

清理证书CSR文件

修改文件权限

Etcd Cluster

说明

准备环境

查看集群部署的环境变量

添加用户

创建目录

配置`sysctl`参数