# 跨VPC网络-使用 kubeadm 创建集群（v1.24） ### 使用 kubeadm 创建集群（v1.24） * 部署单个控制平面的 Kubernetes 集群，使用外部 etcd 集群设置 Kubernetes 集群 * 在集群上安装 Pod 网络，以便你的 Pod 可以相互连通 **前提** * 1. 跨 VPC 网络搭建 k8s 集群-工具安装 * 2. 跨 VPC 网络二进制部署 etcd 集群 **3.1 初始化控制平面节点** > 将 etcd 认证文件从任何一个 etcd 节点复制到控制平面节点（master） ```bash export CONTROL_PLANE="root@112.71.24.110" scp /etc/etcd/ssl/ca.pem "${CONTROL_PLANE}": scp /etc/etcd/ssl/etcd.pem "${CONTROL_PLANE}": scp /etc/etcd/ssl/etcd-key.pem "${CONTROL_PLANE}": ``` **3.2 集群配置参数** 首先生成配置文件 ```bash # 生成默认的 kubeadm 配置文件 kubeadm config print init-defaults > kubeadm.yaml # 编辑修改参数 vim kubeadm.yaml ``` 调整好的 kubeadm.yaml 如下： ```bash apiVersion: kubeadm.k8s.io/v1beta3 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 43.129.25.161 # master 的外网 IP, bindPort: 6443 nodeRegistration: criSocket: unix:///var/run/containerd/containerd.sock imagePullPolicy: IfNotPresent name: master taints: null --- apiServer: certSANs: - 43.129.25.161 # 指定证书认证的域名或者 IP 地址 extraArgs: authorization-mode: Node,RBAC advertise-address: 43.129.25.161 # 对外访问的 IP timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta3 certificatesDir: /etc/kubernetes/pki controlPlaneEndpoint: 43.129.25.161:6443 # 控制平面的外网 IP,或者LB域名 clusterName: kubernetes controllerManager: {} dns: {} etcd: external: endpoints: # 外部 etcd 集群的 IP 地址 - https://43.129.25.161:2379 - https://43.134.80.11:2379 - https://43.135.160.117:2379 caFile: /data/etcd/ssl/ca.pem certFile: /data/etcd/ssl/etcd.pem keyFile: /data/etcd/ssl/etcd-key.pem imageRepository: k8s.gcr.io kind: ClusterConfiguration kubernetesVersion: 1.24.4 networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12 podSubnet: 10.244.0.0/16 # Pod 网络子网，这里是 flannel scheduler: {} --- apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration cgroupDriver: systemd # kubelet 的控制组驱动, kubelet 的 cgroup 选项设置就在此处。 ``` ### 初始化命令 ```bash # 初始化命令格式 kubeadm init # 在控制平面节点（master）上运行： kubeadm init --config kubeadm.yaml --upload-certs ``` > kubeadm init 首先运行一系列预检查。这些预检查会显示警告并在错误时退出。预检查通过后 kubeadm init 下载并安装集群控制平面组件。这可能需要几分钟。 完成之后你应该看到： ```bash Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of the control-plane node running the following command on each as root: kubeadm join 43.129.25.161:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:138fe28c94b751d9b46243391b827b36fe7e1a3b6df086a5c87c3d8113b78fbf \ --control-plane --certificate-key 1801d1376fd21ceccc9dc66b5c9fc782ea5fcb922dfd78e349da4888e55477fc Please note that the certificate-key gives access to cluster sensitive data, keep it secret! As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use "kubeadm init phase upload-certs --upload-certs" to reload certs afterward. Then you can join any number of worker nodes by running the following on each as root: kubeadm join 43.129.25.161:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:138fe28c94b751d9b46243391b827b36fe7e1a3b6df086a5c87c3d8113b78fbf ``` **在控制平面节点运行以下命令，就能够开始使用 kubectl 操作集群** ```bash mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config ``` **加入其它控制平面节点** 在追求高稳定性的生产环境中，集群的冗余是必要的。如果只有一个控制平面节点，当这个节点宕机时，整个集群处于不可用状态。为了降低这类风险，我们需要准备多个控制平面节点。集群数量是1、3、5、7……，奇数。 ```bash kubeadm join : --token --discovery-token-ca-cert-hash sha256: --control-plane --certificate-key ``` **加入工作节点** 在工作节点机器上运行 kubeadm init 输出的 kubeadm join 命令 ```bash kubeadm join --token : --discovery-token-ca-cert-hash sha256: ``` 输出如下： ```bash [preflight] Running pre-flight checks ... (log output of join workflow) ... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. ``` 回到控制平面节点（master）上运行 kubectl get nodes 命令查看节点加入情况，发现 kubectl 命令报错，无法查看节点信息。 因为 kubeadm join 命令将工作节点的内网 IP 上传给控制平面节点，跨 VPC 情况下，在控制平面节点当然无法通过工作节点的内网 IP 与工作节点通讯。 **解决方案：** 在控制平台节点（master）设置各个工作节点内、外网 IP NAT 转发规则（有 n 个工作节点就设置 n 条转发规则），使 kubectl 能够与所有工作节点通讯。命令格式如下： ```bash iptables -t nat -A OUTPUT -d -j DNAT --to-destination ``` 此处有 2 个工作节点，故需要在 master 上执行 2 条命令： ```bash # 在 master 上执行以下命令 iptables -t nat -A OUTPUT -d 172.22.0.10 -j DNAT --to-destination 43.134.80.11 iptables -t nat -A OUTPUT -d 172.26.0.9 -j DNAT --to-destination 43.135.160.117 ``` kubectl 此刻可正常查看各个工作节点信息： ```bash root@VM-200-6-ubuntu:/data# kubectl get nodes NAME STATUS ROLES AGE VERSION node1 NotReady 70s v1.24.4 node2 NotReady 18s v1.24.4 master NotReady control-plane 25m v1.24.4 root@VM-200-6-ubuntu:/data# kubectl get pods --all-namespaces -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system coredns-6d4b75cb6d-fvwdw 0/1 Pending 0 26m kube-system coredns-6d4b75cb6d-zswpf 0/1 Pending 0 26m kube-system kube-apiserver-master 1/1 Running 0 8m4s 172.19.200.6 master kube-system kube-controller-manager-master 1/1 Running 0 8m4s 172.19.200.6 master kube-system kube-proxy-7c2wz 1/1 Running 1 26m 172.19.200.6 master kube-system kube-proxy-jhwcb 1/1 Running 0 2m54s 172.22.0.10 etcd2 kube-system kube-proxy-pktkf 1/1 Running 0 2m2s 172.26.0.9 etcd3 kube-system kube-scheduler-master 1/1 Running 1 27m 172.19.200.6 master ``` ### 安装 Pod 网络插件 flannel > `kubectl get nodes` 查看全部节点状态，都是 Not Ready；再用 `kubectl get pods --all-namespaces -o wide` 查看所有 pod，CoreDNS 都是 pending 状态。 这是还没安装 Pod 网络插件的状态，k8s 集群还没就绪。 第一种方式是在每个节点上安装二进制包，然后以 systemd 服务运行 flanneld，指定-public-ip 参数，绑定外网ip，详见 第二种方式，容器化部署 flannel，一键执行，方便快捷。 **在部署之前，有个关键问题：** > flannel 默认将网卡 IP 设置为 Public IP，跨 VPC 服务器上的 flannel 实例以该服务器的内网 IP 为 Public IP，然后就出现控制平面节点的 flannel 运行成功，而工作节点的 flannel 不断重启、不断失败的状况…… **解决方案：** 在每个节点（控制平台节点、工作节点）上设置节点注释，命令格式如下： ```bash kubectl annotate node flannel.alpha.coreos.com/public-ip-overwrite= ``` 在控制节点执行以下命令： ```bash # 给 master 执行 kubectl annotate node master flannel.alpha.coreos.com/public-ip-overwrite=43.129.25.161 # 给 node1 上执行 kubectl annotate node node1 flannel.alpha.coreos.com/public-ip-overwrite=43.134.80.11 # 给 node2 上执行 kubectl annotate node node2 flannel.alpha.coreos.com/public-ip-overwrite=43.135.160.117 ``` 接着回到 master 上，用以下命令部署 flannel ```bash # flannel 在每个节点启动后，读取该节点的 public-ip-overwrite 值，设为 Public IP。 kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml ``` ```bash root@VM-200-6-ubuntu:/data# kubectl get no NAME STATUS ROLES AGE VERSION node1 Ready 11m v1.24.4 node2 Ready 10m v1.24.4 master Ready control-plane 36m v1.24.4 ``` 每个集群只能安装一个 Pod 网络。 > 安装 Pod 网络后，你可以通过在 kubectl get pods --all-namespaces 输出中检查 CoreDNS Pod 是否 Running 来确认其是否正常运行。 一旦 CoreDNS Pod 和 flannel Pod 启用并运行，集群就完成部署工作。 ```bash root@VM-200-6-ubuntu:/data# kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-flannel kube-flannel-ds-dknqp 1/1 Running 0 2m50s kube-flannel kube-flannel-ds-t2v8z 1/1 Running 0 2m50s kube-flannel kube-flannel-ds-vb4n9 1/1 Running 0 2m50s kube-system coredns-6d4b75cb6d-fvwdw 1/1 Running 0 37m kube-system coredns-6d4b75cb6d-zswpf 1/1 Running 0 37m kube-system kube-apiserver-master 1/1 Running 0 18m kube-system kube-controller-manager-master 1/1 Running 0 18m kube-system kube-proxy-7c2wz 1/1 Running 1 37m kube-system kube-proxy-jhwcb 1/1 Running 0 13m kube-system kube-proxy-pktkf 1/1 Running 0 12m kube-system kube-scheduler-master 1/1 Running 1 37m ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://close.gitbook.io/yun-wei-bi-ji/kubernetes/k8s/kua-vpc-wang-luo-k8s/kua-vpc-wang-luo-shi-yong-kubeadm-chuang-jian-ji-qun-v1.24.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.