Kubernetes(简称 K8S,8 为中间英文字母ubernete缩写)是用于自动部署、扩展和管理容器化应用程序的开源系统。官方有提供详细的学习文档,包括中文版本,建议新手可以详细阅读和练习一遍。

部署模式

K8S 支持单机集群高可用集群模式部署,其中单机模式适用于个人对 K8S 的使用学习,生产环境部署建议使用高可用群集模式。三者区别如下:

  • 单机,在一台机器上部署 1 个Master和 1 个Worker节点组成最基本 K8S 服务

如只是想简单学习 K8S 环境的使用,建议新手可通过 minikubekind 在本地快速启动一个单机的 K8S 学习环境。

经过我实践验证,在 MacOS 操作系统环境下,通过使用 minikube 简单几个命令就可在本地完成一个 K8S 单机学习环境部署,非常方便。

本次是关于 K8S 高可用集群模式的部署,要部署高可用集群,至少要满足以下要求:

  • 主节点(Master)服务器至少 3 台以上,由于 K8S 数据存储在主节点的 etcd 服务,因此需要确保主节点总数保持奇数,如:3,5,7,…;
  • 工作节点(Worker)服务器至少 2 台以上,所有的工作节点均是无状态,因此只需要多于 2 台即可,可按照实际运行应用所需资源动态扩缩容工作节点数量;
  • 工作节点通过负载均衡连接并注册到主节点服务,负载均衡服务可以使用 HAProxy 或其它同类服务替代。

每个节点是 1 台 Linux 服务器,部署示意图如下:

Master:主要是管理、监控和协调集群资源,一般由数据持久化(etcd)、DNS(coredns)、API(kube-apiserver)、控制器(kube-controller-manager)、调度器(kube-scheduler)和 Pod 网络附加组件等多个重要组件共同构成。

Worker:K8S 集群中的工作负载节点,所有的应用实例均会运行在工作节点,因此工作节点所在的服务器性能决定了整个 K8S 集群的运算能力。

安装部署

准备工作

所需服务器

节点 IP 节点名称 节点角色 节点配置
192.168.0.101 k8s-master-01 Master Debian 10.7.0 x86_64
2Core/4GB
192.168.0.102 k8s-master-02 Master Debian 10.7.0 x86_64
2Core/4GB
192.168.0.103 k8s-master-03 Master Debian 10.7.0 x86_64
2Core/4GB
192.168.0.201 k8s-worker-01 Master Debian 10.7.0 x86_64
2Core/8GB
192.168.0.202 k8s-worker-02 Master Debian 10.7.0 x86_64
2Core/8GB

安装 Docker

K8S 依赖 Docker 服务,需要在所有服务器安装 Docker 服务,安装命令如下:

1
$ curl -sSL https://get.docker.com | sudo bash -

修改 driver:

1
2
3
4
5
6
7
$ sudo cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
$ sudo systemctl restart docker
$ sudo systemctl daemon-reload

网络配置

允许 iptables 检查桥接流量,在所有服务器上执行命令如下:

1
2
3
4
5
6
7
8
9
$ sudo cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

$ sudo cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
$ sudo sysctl --system

禁用交换分区

在所有服务器上打开并编辑文件 /etc/fstab,把系统加载 swap 分区禁用(行前加 # 或直接删除行),然后重启服务器。

APT 源配置

更新索引并安装所需要的包

1
2
$ sudo apt-get update
$ sudo apt-get install -y apt-transport-https ca-certificates curl

添加 APT 安装源(2 选 1)

Google 官方源:

1
2
3
$ sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
$ echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" \
  | sudo tee /etc/apt/sources.list.d/kubernetes.list

国内 Aliyun:

1
2
3
4
$ curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add - 
$ sudo cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF

安装运行时工具

安装kubeletkubeadmkubectl,在每台服务器上执行以下命令安装:

1
2
3
$ sudo apt-get update
$ sudo apt-get -y install kubelet kubeadm kubectl
$ sudo apt-mark hold kubelet kubeadm kubectl

拉取 K8S 镜像

如果能直接访问 Google 服务,可忽略跳到下一段。默认情况下,通过 kubeadm 初始化服务时,需要从 k8s.gcr.io 仓库下载相关的组件镜像运行,但由于国内网络不能访问到 Google 的服务器,导致镜像下载不了,导致初始化失败。解决的办法是可以在执行初始化命令前,先通过国内镜像源把镜像拉到对应的服务器,再执行初始化命令即可。具体操作命令如下:

1
2
3
4
5
6
7
8
kubeadm config images list \
| while read image; do
  image_name=$(echo $image | awk -F '/' '{print $NF}');
  docker pull "registry.aliyuncs.com/google_containers/$image_name";
  docker tag "registry.aliyuncs.com/google_containers/$image_name" "$image";
  docker rmi -f "registry.aliyuncs.com/google_containers/$image_name";
  echo "$image";
done

初始化主节点

初始化第一个主节点

通过 kubeadm 对初始化主节点 K8S 服务,在主节点服务器执行命令如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
$ kubeadm init --control-plane-endpoint="192.168.0.100:6443" --upload-certs --pod-network-cidr=10.10.0.0/16 --v=5
...
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 192.168.0.100:6443 --token 2ivl73.w9gw7qazuooe6tl6 \
        --discovery-token-ca-cert-hash sha256:a4fd3b806c1fc6e8c8d5ac95f8b13dd6a67f90ff888bb7d09c5c603e6ed6612c \
        --control-plane --certificate-key 80741070992e761f8861807055813448d1eda6ed39b354bb58fa562344b30d3c

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.0.100:6443 --token 2ivl73.w9gw7qazuooe6tl6 \
        --discovery-token-ca-cert-hash sha256:a4fd3b806c1fc6e8c8d5ac95f8b13dd6a67f90ff888bb7d09c5c603e6ed6612c

--control-plane-endpoint="192.168.0.100:6443" 为 kube-apiserver 高可用负载均衡服务,由 Keepalied + HAProxy 搭建而成,后面会以其它单独的篇幅进行说明如何搭建,这里只说明一下其中的部署架构:

--pod-network-cidr=10.10.0.0/16 指定集群内服务分配的 IP 网段,可选,不指定会默认。

--v=5 DEBUG 模式日志输出,方便查看出错信息排查问题,可选。

上面的信息我们可以知道几点重要的信息:

  • 主节点初始化成功;
  • 如何配置访问管理 K8S 集群;
  • 如何添加 Master 节点;
  • 如何添加 Worker 节点。

💡 官方推荐生产使用 etcd 独立服务集群,以提高 K8S 服务稳定性,具体可参考:为 Kubernetes 运行 etcd 集群

初始化其它主节点

1
2
3
$ sudo kubeadm join 192.168.0.100:6443 --token 2ivl73.w9gw7qazuooe6tl6 \
    --discovery-token-ca-cert-hash sha256:a4fd3b806c1fc6e8c8d5ac95f8b13dd6a67f90ff888bb7d09c5c603e6ed6612c \
    --control-plane --certificate-key 80741070992e761f8861807055813448d1eda6ed39b354bb58fa562344b30d3c

访问 K8S 集群设置

1
2
3
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

如果是 root 用户,简单直接的办法如下:

1
$ export KUBECONFIG=/etc/kubernetes/admin.conf

通过上面的配置,已经可以通过 kubectl 命令访问 K8S 集群服务了,接下来进行一些必要的检查。

输入命令:

1
2
3
4
5
6
7
8
9
$ kubectl get pods -A
NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
kube-system   coredns-78fcd69978-fh76j             0/1     Pending   0          44m
kube-system   coredns-78fcd69978-wfv96             0/1     Pending   0          44m
kube-system   etcd-svr-lab-05                      1/1     Running   1          44m
kube-system   kube-apiserver-svr-lab-05            1/1     Running   1          44m
kube-system   kube-controller-manager-svr-lab-05   1/1     Running   1          44m
kube-system   kube-proxy-dbbcb                     1/1     Running   0          44m
kube-system   kube-scheduler-svr-lab-05            1/1     Running   1          44m

从上面的结果可以看到,K8S 的核心组件服务已经在主节点上启动,其中 coredns 状态并不是Running,在集群没有安装 Pod 网络附加组件前,coredns 服务都是保持Pending运行状态。

输入命令:

1
2
3
4
5
6
$ kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS      MESSAGE                                                                                       ERROR
scheduler            Unhealthy   Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused   
controller-manager   Healthy     ok                                                                                            
etcd-0               Healthy     {"health":"true","reason":""}

从上面的结果可以看到,K8S 的核心组件 scheduler 当前状态为 Unhealthy。解决办法是打开文件/etc/kubernetes/manifests/kube-scheduler.yaml,找到以下内容:

1
2
3
4
5
6
7
8
9
10
spec:
  containers:
  - command:
    - kube-scheduler
    - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
    - --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
    - --bind-address=127.0.0.1
    - --kubeconfig=/etc/kubernetes/scheduler.conf
    - --leader-elect=true
    - --port=0

删除- --port=0这一行并保存,稍等一会即可发现scheduler服务状态变为Healthy

初始化工作节点

相比于主节点,从节点的初始化相对简单很多,我们只要执行初始化主点系统的提示命令即可,如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ sudo kubeadm join 192.168.0.100:6443 --token 2ivl73.w9gw7qazuooe6tl6 \
    --discovery-token-ca-cert-hash sha256:a4fd3b806c1fc6e8c8d5ac95f8b13dd6a67f90ff888bb7d09c5c603e6ed6612c
[preflight] Running pre-flight checks
        [WARNING SystemVerification]: missing optional cgroups: hugetlb
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

输入以下命令检查新加入的工作节点,命令如下:

1
2
3
4
5
6
$ kubectl get nodes
NAME         STATUS     ROLES                  AGE     VERSION
k8s-worker-01   NotReady   <none>                 1m25s   v1.22.3
k8s-master-03   NotReady   control-plane,master   2m39s   v1.22.3
k8s-master-02   NotReady   control-plane,master   4m20s   v1.22.3
k8s-master-01   NotReady   control-plane,master   6m15s   v1.22.3

从运行结果可以看到集群主节点和工作节点都在上面了,由于集群还没有安装 Pod 网络附加组件,因此两个节点当前仍不能正常工作。

安装 Pod 网络附加组件

K8S 支持的 Pod 网络附加组件有多种,只需要安装其中一种即可,这里以 flannel 为例。

安装命令如下:

1
2
3
$ wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
$ sed -i "s#10.244.0.0/16#10.10.0.0/16#" kube-flannel.yml
$ kubectl apply -f kube-flannel.yml

安装完毕,检查运行情况,输出以下命令:

1
2
3
4
5
6
7
8
9
10
11
12
$ kubectl get pod -A
NAMESPACE     NAME                                 READY   STATUS              RESTARTS        AGE
kube-system   coredns-78fcd69978-fh76j             1/1     Running             2 (6m13s ago)   138m
kube-system   coredns-78fcd69978-wfv96             1/1     Running             2 (6m13s ago)   138m
kube-system   etcd-svr-lab-05                      1/1     Running             3 (6m18s ago)   139m
kube-system   kube-apiserver-svr-lab-05            1/1     Running             3 (6m18s ago)   139m
kube-system   kube-controller-manager-svr-lab-05   1/1     Running             3 (6m18s ago)   139m
kube-system   kube-flannel-ds-8zt5x                1/1     Running             0               5m13s
kube-system   kube-flannel-ds-fhgb6                0/1     Init:0/2            0               4m9s
kube-system   kube-proxy-dbbcb                     1/1     Running             2 (6m18s ago)   138m
kube-system   kube-proxy-lvdl8                     0/1     ContainerCreating   0               4m9s
kube-system   kube-scheduler-svr-lab-05            1/1     Running             2 (6m18s ago)   77m

检查发现 kube-proxy-lvdl8 一直保持在ContainerCreating状态,输入以下命令进一步查看原因:

1
2
3
4
5
6
7
8
$ kubectl -n kube-system describe pod kube-proxy-lvdl8
...
Events:
  Type     Reason                  Age                 From               Message
  ----     ------                  ----                ----               -------
  Normal   Scheduled               2m33s               default-scheduler  Successfully assigned kube-system/kube-proxy-lvdl8 to svr-lab-04
  Warning  FailedCreatePodSandBox  43s                 kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.5": Error response from daemon: Get "https://k8s.gcr.io/v2/": dial tcp 74.125.203.82:443: i/o timeout (Client.Timeout exceeded while awaiting headers)
  Warning  FailedCreatePodSandBox  14s (x4 over 2m7s)  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.5": Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

从输出结果可以看出,由于网络原因,拉取镜像 k8s.gcr.io/pause:3.5 失败了,解决方法如下:

1
2
3
$ docker pull "registry.aliyuncs.com/google_containers/pause:3.5"
$ docker tag "registry.aliyuncs.com/google_containers/pause:3.5" "k8s.gcr.io/pause:3.5"
$ docker rmi -f "registry.aliyuncs.com/google_containers/pause:3.5"

同样的问题解决还有 kube-proxy,解决方法如下:

1
2
3
$ docker pull "registry.aliyuncs.com/google_containers/kube-proxy:v1.22.3"
$ docker tag "registry.aliyuncs.com/google_containers/kube-proxy:v1.22.3" "k8s.gcr.io/kube-proxy:v1.22.3"
$ docker rmi -f "registry.aliyuncs.com/google_containers/kube-proxy:v1.22.3"

最后进行集群检查,所有的组件运行状态应该都是Running状态,运行命令如下(K8S 调度存在延时,有时需要观察几分钟):

1
2
3
4
5
6
7
8
$ watch kubectl get pod -A
NAMESPACE     NAME                                 READY   STATUS    RESTARTS       AGE
kube-system   coredns-78fcd69978-fh76j             1/1     Running   3              176mkube-system   coredns-78fcd69978-wfv96             1/1     Running   3              176m
kube-system   etcd-svr-lab-05                      1/1     Running   4              176m
kube-system   kube-apiserver-svr-lab-05            1/1     Running   4              176mkube-system   kube-controller-manager-svr-lab-05   1/1     Running   4              176m
kube-system   kube-flannel-ds-8zt5x                1/1     Running   1              42mkube-system   kube-flannel-ds-fhgb6                1/1     Running   12 (13m ago)   41m
kube-system   kube-proxy-dbbcb                     1/1     Running   3              176mkube-system   kube-proxy-lvdl8                     1/1     Running   0              41m
kube-system   kube-scheduler-svr-lab-05            1/1     Running   3              115m

部署集群看板界面

安装方法如下:

1
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.2.0/aio/deploy/recommended.yaml

检测安装:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
$ kubectl -n kubernetes-dashboard get pods -o wide
NAME                                         READY   STATUS    RESTARTS   AGE   IP          NODE         NOMINATED NODE   READINESS GATES
dashboard-metrics-scraper-856586f554-t64xk   1/1     Running   0          14m   10.10.1.3   svr-lab-04   <none>           <none>
kubernetes-dashboard-78c79f97b4-l94wm        1/1     Running   0          14m   10.10.1.2   svr-lab-04   <none>           <none>

$ kubectl -n kubernetes-dashboard get service kubernetes-dashboard -o wide
NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE   SELECTOR
kubernetes-dashboard   ClusterIP   10.106.231.67   <none>        443/TCP   16m   k8s-app=kubernetes-dashboard

$ curl -k https://10.106.231.67/
<!--
Copyright 2017 The Kubernetes Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

<!doctype html>
<html lang="en">

<head>
  <meta charset="utf-8">
  <title>Kubernetes Dashboard</title>
  <link rel="icon" type="image/png" href="assets/images/kubernetes-logo.png"/>
  <meta name="viewport" content="width=device-width">
<link rel="stylesheet" href="styles.aa1f928b22a88c391404.css"></head>

<body>
  <kd-root></kd-root>
<script src="runtime.2a456dd93bf6c4890676.js" defer></script><script src="polyfills-es5.354af976502039f97f74.js" nomodule defer></script><script src="polyfills.f4f05ad675be9638106e.js" defer></script><script src="scripts.128068f897fc721c4673.js" defer></script><script src="en.main.09bf52db2dbc808e7279.js" defer></script></body>

</html>

创建登录管理账号 Token

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ kubectl create serviceaccount dashboard-admin -n kube-system
$ kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin 
$ kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')
Name:         dashboard-admin-token-fdk7w
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: dashboard-admin
              kubernetes.io/service-account.uid: ec947e0e-d04c-471b-a02e-815e6a333a4c

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1099 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6ImVkZnVwSkc5RlVRSDRvNlM1Uk0tYTZrajM5YzB2V2JlRzl5UTZ0b2tJOGsifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tZmRrN3ciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiZWM5NDdlMGUtZDA0Yy00NzFiLWEwMmUtODE1ZTZhMzMzYTRjIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.FuJ8PiJLD_-cTr98dLmu90Hle0zor3W7kHw3olw___jgrun62jxkudpp9fChnwoCHaCTmaugPQ2cBHenHALfspBbCl74SjIWcNJUiFzmgWZOyHRkBWkrnvCzSmjzdlH7K0lJFVcNnsCEAZClfZIIfXqIVTMdzci_-E60fqo2Ws7DdPOjue8E-boUIMgSANewzl6t6gCw4qLoYg0XV-3dgyJ1g-QuV4fj7v10mrLKSxwOLsCYRJ5KdVUErCp3FZU-jOLTGLa_zaVv1ytUGJmUy-mX0UYTQt3ZW5ITNoPyEOT0BJV3egjnZdERU1mp2WY6ymqEWFrg7J9ceVKGNe7zBg

上面最后的输出结果表明创建成功,同时生成的 token 是集群后台看板登录的凭证,要妥善保管且不要随意泄露,后面登录到管理界面需要用到。

安装 Ingress

K8S 所有 POD 实例都是临时性的,POD 对应的 IP 也是集群内部动态分配的 IP,默认对外是不可访问的。K8S 如果要对对外提供服务访问,有两种实现方式:

  • Worker 节点对外暴露NodePort端口,由于占用的是节点的端口,但对于 WEB 端的 HTTP/HTTPS 服务来说,一个域名将对应一组 POD 服务,这将会导致需要大量对外的端口,在维护上存在较大的成本。
  • Ingress,通过 Ingress 网关路由对外提供服务,Ingress 自身具备服务路由功能,可以通过 APIServer 获取和动态更新服务 IP,能轻便的管理集群的服务和降低维护成本。可用的 Ingress 有很多,如 Ingress-Nginx,下面将会介绍其安装方法。

网关工作流程示意如下:

安装命令如下

可访问 Google 服务时使用

1
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.4/deploy/static/provider/cloud/deploy.yaml

不可访问 Google 服务时使用

1
2
3
4
5
6
7
8
9
10
11
$ wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.4/deploy/static/provider/cloud/deploy.yaml -O ingress-nginx.yaml
$ sed -i 's#@sha256:.*$##' ingress-nginx.yaml
$ kubectl apply -f ./ingress-nginx.yaml

$ docker pull "registry.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1"
$ docker tag "registry.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1" "k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1"
$ docker rmi -f "registry.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1"

$ docker pull "registry.aliyuncs.com/google_containers/nginx-ingress-controller:v1.0.4"
$ docker tag "registry.aliyuncs.com/google_containers/nginx-ingress-controller:v1.0.4" "k8s.gcr.io/ingress-nginx/controller:v1.0.4"
$ docker rmi -f "registry.aliyuncs.com/google_containers/nginx-ingress-controller:v1.0.4"

检查安装,命令如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
$ kubectl -n ingress-nginx get pods
NAME                                        READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create--1-wzszb     0/1     Completed   0          9m28s
ingress-nginx-admission-patch--1-fktqb      0/1     Completed   2          9m28s
ingress-nginx-controller-69bb54574d-jmggb   1/1     Running     0          9m28s

$ kubectl -n ingress-nginx get service
NAME                                 TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.108.149.112   <pending>     80:31964/TCP,443:31253/TCP   10m
ingress-nginx-controller-admission   ClusterIP      10.109.34.164    <none>        443/TCP                      10m

$ curl http://192.168.0.101:31964/
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

$ curl -k https://192.168.0.101:31253/
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

从上面可以看到,通过Worker 节点 IP + 端口已经可以访问到网关的服务里去了,由于当前网关未配置任何服务路由分发规则,因此请求返回 404。

部署测试

如果你顺利操作到这里,说明整个 K8S 集群已经部署完成,虽然现在只能看到 404 的错误页面。下面演示如何配置通过 Ingress 网关配置路由规则,访问到前面配置的 K8S 看板界面。

创建 Ingress 网关路由规则文件kubernetes-dashboard-ingress.yaml,内容如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kubernetes-dashboard-ingress
  namespace: kubernetes-dashboard
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"

spec:
  tls:
  - hosts:
    - k8s.laijinman.com
    secretName: kubernetes-dashboard-certs
  rules:
    - host: k8s.laijinman.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: kubernetes-dashboard
                port:
                  number: 443

应用规则,命令如下:

1
$ kubectl apply -f kubernetes-dashboard-ingress.yaml 

浏览器访问:

  • hosts 设置192.168.0.101 k8s.laijinman.com
  • 访问https://k8s.laijinman.com:31253/即可看到 K8S 的管理后台,通过前面创建的 token 凭证进行登录

其它

指定安装 K8S 版本

随着时间推移,Google 会不断更新 K8S 版本和功能,新版本对于现有已部署的集群未必兼容,如果没有对新功能有必要的需求,可以不升级,以确保集群服务的稳定性。在集群扩容新加服务器时,可以指定更新版本安装,命令如下:

1
2
$ apt-get update
$ apt-get install -y kubelet=1.22.2-00 kubeadm=1.22.2-00 kubectl=1.22.2-00

相关链接