kubeadm部署kubernetes-v1.13.x集群

发表于 | 更新于: | 分类于 | 阅读次数:
字数统计: 2.2k

说明

根据kubernetes v1.13的Release Note说明

从Kubernetes v1.13.x开始，kubeadm的kubeadm.k8s.io/v1alpha3被标记为废弃并将于kubernetes v1.14版开始被移除，新的apiVersion为kubeadm.k8s.io/v1beta1

这里是关于kubeadm.k8s.io/v1beta1的说明文档

下面演示环境使用1个master节点+2个node节点部署kubernetes集群

仅记录我的部署流程，不一定满足各种需求，自己看菜吃饭！

服务器配置

主机名IP地址角色操作系统Docker版本kubeadm版本
k8s-master172.16.80.200master+nodeCentOS-7.6.181018.09.6v1.13.7
k8s-node1172.16.80.201nodeCentOS-7.6.181018.09.6v1.13.7
k8s-node2172.16.80.202nodeCentOS-7.6.181018.09.6v1.13.7

服务器初始化

参考CentOS-7.6(1810)虚拟机模板制作

配置/etc/hosts

1
2
3
4
5
127.0.0.1   localhost
::1         localhost
172.16.80.200 k8s-master
172.16.80.201 k8s-node1
172.16.80.202 k8s-node2

禁用swap分区

1
2
swapoff -a
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab

部署kubernetes集群

创建YUM源

使用阿里云的kubernetes YUM源

1
2
3
4
5
6
7
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

刷新YUM缓存

1
yum makecache

安装kubernetes-v1.13.7

1
yum install kubeadm-1.13.7-0.x86_64 kubelet-1.13.7-0.x86_64 kubectl-1.13.7-0.x86_64 cri-tools-1.12.0-0.x86_64 kubernetes-cni-0.7.5-0.x86_64

添加bash自动补全命令

1
2
kubeadm completion bash > /etc/bash_completion.d/kubeadm
kubectl completion bash > /etc/bash_completion.d/kubectl

配置文件说明

kubeadm的配置文件被拆分成了以下几个类型

  • InitConfiguration
  • ClusterConfiguration
  • KubeletConfiguration
  • KubeProxyConfiguration
  • JoinConfiguration

对应到配置文件里面的格式如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
---
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
---
apiVersion: kubeadm.k8s.io/v1beta1
kind: JoinConfiguration

创建配置文件

kubeadm命令可以打印InitConfiguration、ClusterConfiguration和JoinConfiguration的默认配置

init-defaults

1
kubeadm config print init-defaults

输出示例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
---
apiVersion: kubeadm.k8s.io/v1beta1
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 1.2.3.4
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: k8s-master
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta1
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: ""
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.13.0
networking:
  dnsDomain: cluster.local
  podSubnet: ""
  serviceSubnet: 10.96.0.0/12
scheduler: {}
---

join-defaults

1
kubeadm config print join-defaults

输出示例

1
2
3
4
5
6
7
8
9
10
11
12
13
apiVersion: kubeadm.k8s.io/v1beta1
caCertPath: /etc/kubernetes/pki/ca.crt
discovery:
  bootstrapToken:
    apiServerEndpoint: kube-apiserver:6443
    token: abcdef.0123456789abcdef
    unsafeSkipCAVerification: true
  timeout: 5m0s
  tlsBootstrapToken: abcdef.0123456789abcdef
kind: JoinConfiguration
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: k8s-master

修改配置文件

对于没有设置或者配置了一部分的参数，kubeadm会使用默认值

这里精简了很多配置，自己决定是否增加配置！

kubeadm-init.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
---
# InitConfiguration影响master节点初始化时，kubelet的配置
apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
# bootstrapTokens:
# - groups:
#   - system:bootstrappers:kubeadm:default-node-token
#   token: abcdef.0123456789abcdef
#   ttl: 24h0m0s
#   usages:
#   - signing
#   - authentication
nodeRegistration:
  # master节点默认会加taints
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
  kubeletExtraArgs:
    pod-infra-container-image: "k8s.gcr.io/pause:3.1"
    network-plugin: cni
---
# kubeadm生成证书、manifest目录、staticPod的配置
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
# kube-apiserver配置
apiServer:
  timeoutForControlPlane: 4m0s
  # 配置证书的SANs
  certSANs:
  - localhost
  - k8s-master
  - kubernetes
  - kubernetes.default
  - kubernetes.default.svc
  - kubernetes.default.svc.cluster.local
  - 127.0.0.1
  - 10.96.0.1
  - 172.16.80.200
  # 配置apiserver启动参数
  extraArgs:
    authorization-mode: "Node,RBAC"
    runtime-config: "api/all=true"
  # 配置apiserver容器挂载volume
  extraVolumes:
  - name: "timezone-volume"
    hostPath: "/usr/share/zoneinfo/Asia/Shanghai"
    mountPath: "/etc/localtime"
    readOnly: true
    pathType: File
# 配置证书目录
certificatesDir: /etc/kubernetes/pki
# 配置集群名字
clusterName: kubernetes
controlPlaneEndpoint: ""
# kube-controller-manager配置
controllerManager:
  # 配置controller-manager启动参数
  extraArgs:
    address: "0.0.0.0"
  extraVolumes:
  - name: "timezone-volume"
    hostPath: "/usr/share/zoneinfo/Asia/Shanghai"
    mountPath: "/etc/localtime"
    readOnly: true
    pathType: File
# kubernetes集群dns配置
dns:
  # 类型可选 CoreDNS 或者 kube-dns
  type: CoreDNS
  imageRepository: "k8s.gcr.io"
  imageTag: "1.2.6"
# etcd配置
etcd:
  # local和external配置是冲突的，二选一
  local:
    imageRepository: "k8s.gcr.io"
    imageTag: "3.2.24"
    # etcd数据目录
    dataDir: "/var/lib/etcd"
    extraArgs:
      advertise-client-urls: "https://172.16.80.200:2379"
      listen-client-urls: "https://127.0.0.1:2379,https://172.16.80.200:2379"
      listen-peer-urls: "https://172.16.80.200:2380"
    # 配置etcd server证书的SAN
    serverCertSANs:
    - k8s-master
    - localhost
    - ::1
    - 127.0.0.1
    - 172.16.80.200
    # 配置etcd peer证书的SAN
    peerCertSANs:
    - k8s-master
    - localhost
    - ::1
    - 127.0.0.1
    - 172.16.80.200
  # local和external配置是冲突的，二选一
  # external:
  #   endpoints:
  #   - "https://etcd_1:2379"
  #   - "https://etcd_2:2379"
  #   - "https://etcd_3:2379"
  #   caFile: "/path/to/etcd-ca.crt"
  #   certFile: "/path/to/etcd-client.crt"
  #   keyFile: "/path/to/etcd-client.key"
imageRepository: registry.aliyuncs.com/k8sxio
kubernetesVersion: "v1.13.7"
# kubernetes网络配置
networking:
  # DNS域名
  dnsDomain: cluster.local
  # Pod网络
  podSubnet: "10.244.0.0/16"
  # Service网络
  serviceSubnet: "10.96.0.0/12"
# kube-scheduler配置
scheduler:
  extraArgs:
    address: "0.0.0.0"
  extraVolumes:
  - name: "timezone-volume"
    hostPath: "/usr/share/zoneinfo/Asia/Shanghai"
    mountPath: "/etc/localtime"
    readOnly: true
    pathType: File
# 是否使用HyperKube镜像
# hyperkube包含所有kubernetes的二进制文件,实现单二进制文件运行所有kubernetes服务
useHyperKubeImage: false
---
# 配置kubelet
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: cgroupfs
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
# 默认配置下，kubelet检测到系统启用了swap会启动失败
failSwapOn: false
maxOpenFiles: 1048576
maxPods: 110
---
# 配置kube-proxy
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
configSyncPeriod: 15m0s
ipvs:
  scheduler: "rr"
# 可选iptables或者ipvs
# iptables性能好兼容性好
# ipvs大规模场景下性能更好
mode: "ipvs"
---
# 这里的配置影响kubeadm join的节点上运行的kubelet参数
apiVersion: kubeadm.k8s.io/v1beta1
kind: JoinConfiguration
#caCertPath: /etc/kubernetes/pki/ca.crt
#discovery:
#  bootstrapToken:
#    apiServerEndpoint: kube-apiserver:6443
#    token: abcdef.0123456789abcdef
#    unsafeSkipCAVerification: true
#  timeout: 5m0s
#  tlsBootstrapToken: abcdef.0123456789abcdef
nodeRegistration:
  kubeletExtraArgs:
    pod-infra-container-image: "k8s.gcr.io/pause:3.1"
    network-plugin: cni

检查配置文件

1
kubeadm init --config=kubeadm-init.yaml --dry-run

命令执行完毕，可以在/tmp/kubeadm-init-dryrun*里面看到生成的配置文件

可以检查一下是否符合预期！

初始化集群

1
kubeadm init --config=kubeadm-init.yaml

集群初始化完成之后，终端会打印出信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 172.16.80.200:6443 --token uqc4n7.mfs4ep1br9l9ztsi --discovery-token-ca-cert-hash sha256:b30dd4d598baf1a3a53fe794d02302788e71e4314947cf7df5a75e98dd2ed97a

节点加入集群

运行kubeadm init初始化集群之后的提供的join命令，将节点加入集群

1
kubeadm join 172.16.80.200:6443 --token uqc4n7.mfs4ep1br9l9ztsi --discovery-token-ca-cert-hash sha256:b30dd4d598baf1a3a53fe794d02302788e71e4314947cf7df5a75e98dd2ed97a

查看节点信息

还没部署CNI插件，所以节点状态为NotReady

1
kubectl get nodes -o wide
  • 输出示例
1
2
3
4
NAME         STATUS     ROLES    AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
k8s-master   NotReady   master   4m45s   v1.13.7   172.16.80.200   <none>        CentOS Linux 7 (Core)   3.10.0-957.21.2.el7.x86_64   docker://18.9.6
k8s-node1    NotReady   <none>   4m18s   v1.13.7   172.16.80.201   <none>        CentOS Linux 7 (Core)   3.10.0-957.21.2.el7.x86_64   docker://18.9.6
k8s-node2    NotReady   <none>   4m17s   v1.13.7   172.16.80.202   <none>        CentOS Linux 7 (Core)   3.10.0-957.21.2.el7.x86_64   docker://18.9.6

部署CNI插件

这里用flannel

1
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

查看集群状态

  • 查看节点信息
1
kubectl get node

可以看到节点状态为Ready

1
2
3
4
NAME         STATUS   ROLES    AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
k8s-master   Ready    master   7m13s   v1.13.7   172.16.80.200   <none>        CentOS Linux 7 (Core)   3.10.0-957.21.2.el7.x86_64   docker://18.9.6
k8s-node1    Ready    <none>   6m46s   v1.13.7   172.16.80.201   <none>        CentOS Linux 7 (Core)   3.10.0-957.21.2.el7.x86_64   docker://18.9.6
k8s-node2    Ready    <none>   6m45s   v1.13.7   172.16.80.202   <none>        CentOS Linux 7 (Core)   3.10.0-957.21.2.el7.x86_64   docker://18.9.6
  • 查看Pod信息
1
kubectl get pod --all-namespaces -o wide --sort-by=.spec.nodeName

这里根据节点名字作为排序依据来输出结果

1
2
3
4
5
6
7
8
9
10
11
12
13
NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE     IP              NODE         NOMINATED NODE   READINESS GATES
kube-system   etcd-k8s-master                      1/1     Running   0          6m56s   172.16.80.200   k8s-master   <none>           <none>
kube-system   kube-apiserver-k8s-master            1/1     Running   0          7m13s   172.16.80.200   k8s-master   <none>           <none>
kube-system   kube-controller-manager-k8s-master   1/1     Running   0          7m5s    172.16.80.200   k8s-master   <none>           <none>
kube-system   kube-flannel-ds-amd64-kccsf          1/1     Running   0          2m27s   172.16.80.200   k8s-master   <none>           <none>
kube-system   kube-proxy-m68d5                     1/1     Running   0          7m56s   172.16.80.200   k8s-master   <none>           <none>
kube-system   kube-scheduler-k8s-master            1/1     Running   0          7m1s    172.16.80.200   k8s-master   <none>           <none>
kube-system   coredns-86c58d9df4-jv465             1/1     Running   0          7m56s   10.244.1.2      k8s-node1    <none>           <none>
kube-system   coredns-86c58d9df4-td2nl             1/1     Running   0          7m56s   10.244.1.3      k8s-node1    <none>           <none>
kube-system   kube-flannel-ds-amd64-s9pds          1/1     Running   0          2m27s   172.16.80.201   k8s-node1    <none>           <none>
kube-system   kube-proxy-l67jr                     1/1     Running   0          7m48s   172.16.80.201   k8s-node1    <none>           <none>
kube-system   kube-flannel-ds-amd64-r6h8s          1/1     Running   0          2m27s   172.16.80.202   k8s-node2    <none>           <none>
kube-system   kube-proxy-pjz5w                     1/1     Running   0          7m47s   172.16.80.202   k8s-node2    <none>           <none>

master节点参与负载

kubeadm部署的kubernetes集群，出于安全考虑，在初始化集群后会给master节点打上node-role.kubernetes.io/master:NoSchedule污点，避免Pod被调度到master节点。

作为实验环境，可以去除污点，让Pod可以调度到master节点

1
kubectl taint nodes k8s-master node-role.kubernetes.io/master-

至此kubernetes集群已经搭建完成

最简化的部署！！！！