这是本节的多页打印视图。 点击此处打印.

返回本页常规视图.

部署指南

1 - 部署环境要求

在进行 All In One 或者多节点部署之前，请按照以下内容确认环境要求

系统版本及硬件要求

操作系统最低要求
Ubuntu 16.04, 18.04CPU：4 核，内存：8 G，磁盘空间：20 G
Debian Buster, StretchCPU：4 核，内存：8 G，磁盘空间：20 G
CentOS 7.xCPU：4 核，内存：8 G，磁盘空间：20 G
Kylin v10CPU：4 核，内存：8 G，磁盘空间：20 G
OS ARCH硬件要求
AMD 64Intel 系列，AMD 系列
ARM 64Phytium 2000（国产化arm64芯片理论上都支持，但未经过充分测试）

以上系统配置要求适用于 KubeCube 默认最小化 All In One 模式安装，如需启动更多可插拔组件和拓展功能，建议机器配置为 8 核 CPU 和 16 G 内存

容器运行时

  • KubeCube 支持的 cri 跟随 Kubernetes 标准

  • KubeCube 部署脚本支持的 cri 如下：

支持的容器运行时版本
Docker19.3.12+
Containerd1.5.x

节点上若无容器运行时，部署脚本将自动安装 containerd 1.5.5 作为容器运行时

网络插件

  • KubeCube 支持的 cni 跟随 Kubernetes 标准

  • KubeCube 部署脚本的 cni 目前支持 calico

Kubernetes 版本

KubeCube 支持的 k8s 版本为 v1.18 ~ v1.23 KubeCube 部署脚本支持的 k8s 版本为 v1.18.20、v1.19.13、v1.20.9、v1.21.2、v1.22.2、v1.23.5

前置准备

在使用部署脚本开始 KubeCube 的安装时，脚本会检测环境，并提示需要安装缺失的依赖

env-check

监控组件说明

KubeCube 会默认安装 Prometheus 等监控组件，如果选择在已有k8s集群中部署 KubeCube，并且集群中已安装 Mertics Server，安装 KubeCube 后 Prometheus 会和 Mertics Server 产生冲突，导致监控功能不可用。需要在安装 KubeCube 后执行以下步骤：

  1. 点击页面右上角【切换到控制台】，点击任意空间，进入到控制台页面；

  2. 在左侧菜单栏点击【自定义资源CRD】，进入到集群级别 CRD 列表，可以点击右上方输入 “hotplug” 进行搜索，找到 “hotplugs.hotplug.kubecube.io” CRD，点击【v1】版本进入 CRD 详情页；

  3. 选择 common 实例，点击【设置YAML】，找到 spec.component. name=kubecube-monitoring，添加环境变量 prometheusAdapter.enabled=false，如：

  - env: |
      grafana:
        enabled: false
      prometheus:
        prometheusSpec:
          externalLabels:
            cluster: "{{.cluster}}"
          remoteWrite:
          - url: http://10.173.32.129:31291/api/v1/receive
      prometheusAdapter:      
  			enabled: false
    name: kubecube-monitoring
    namespace: kubecube-monitoring
    pkgName: kubecube-monitoring-15.4.10.tgz
    status: enabled

2 - All In One 最小化部署

对于想要快速开始、快速体验的用户来说，All In One 是最佳的安装方式

v1.2.x

在 Linux 上部署 KubeCube

开始安装

在 Linux 机器上执行部署脚本

KUBECUBE_VERSION=v1.2

curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

等待部署完成

KubeCube 部署完成后，请根据提示信息登陆 console 管理页面

complete-deploy

使用 admin 账户登陆 console

⚠️请在登陆后修改 admin 用户的密码

login-console

v1.1.x

在 Linux 上部署 KubeCube

开始安装

在 Linux 机器上执行部署脚本

KUBECUBE_VERSION=v1.1

curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

等待部署完成

KubeCube 部署完成后，请根据提示信息登陆 console 管理页面

complete-deploy

使用 admin 账户登陆 console

⚠️请在登陆后修改 admin 用户的密码

login-console

v1.0.x

在 Linux 上部署 KubeCube

开始安装

在 Linux 机器上执行部署脚本

KUBECUBE_VERSION=v1.0

curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

等待部署完成

KubeCube 部署完成后，请根据提示信息登陆 console 管理页面

complete-deploy

使用 admin 账户登陆 console

⚠️请在登陆后修改 admin 用户的密码

login-console

3 - 在已有k8s集群中部署KubeCube

v1.2.x

在 Kubernetes 集群中部署 KubeCube

开始安装

在 Linux 机器上执行部署脚本

KUBECUBE_VERSION=v1.2

export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置安装脚本参数

该安装模式下，需要修改以下参数：

# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="true"

# if install k8s
INSTALL_KUBERNETES="false"

# k8s cni, support now is calico only
CNI="calico"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.19.13, 1.20.9, 1.21.2, 1.22.2, 1.23.5
KUBERNETES_VERSION="1.23.5"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

# +optional
# KUBERNETES_BIND_ADDRESS generally is node_ip
# can be set when NODE_MODE="master" ot "control-plane-master"
# default value is $(hostname -I |awk '{print $1}')
KUBERNETES_BIND_ADDRESS="" #{node_ip}

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP=""

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

#######################################################################
# container runtime config
# if value is docker, then use docker as container runtime
# else if value is containerd, then use containerd as container runtime
#######################################################################
CONTAINER_RUNTIME="containerd"

等待部署完成

KubeCube 部署完成后，请根据提示信息登陆 console 管理页面

complete-deploy

使用 admin 账户登陆 console

⚠️请在登陆后修改 admin 用户的密码

login-console

v1.1.x

在 Kubernetes 集群中部署 KubeCube

开始安装

在 Linux 机器上执行部署脚本

KUBECUBE_VERSION=v1.1

export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置安装脚本参数

该安装模式下，需要修改以下参数：

INSTALL_KUBECUBE_MEMBER=“false”

MASTER_IP="${node ip}"

${node ip} 表示你运行脚本所在 node 机器的 ip，该 node 需要可操作 kubectl

# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="true"

# if install k8s
INSTALL_KUBERNETES="false"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.20.9, 1.19.13, 1.18.20, 1.21.2
KUBERNETES_VERSION="1.20.9"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP=""

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

等待部署完成

KubeCube 部署完成后，请根据提示信息登陆 console 管理页面

complete-deploy

使用 admin 账户登陆 console

⚠️请在登陆后修改 admin 用户的密码

login-console

v1.0.x

在 Kubernetes 集群中部署 KubeCube

⚠️修改 Kubernetes API-Server 配置

必要性

  1. KubeCube 对多集群提供统一的认证和鉴权服务，需要使用 k8s api-server 的 auth-webhook 能力来做拓展。

  2. KubeCube 提供对 k8s-apiserver 日志进行审计的能力，这需要为 k8s api-server 指定审计服务后端。

修改操作

如果您的 k8s api-server 服务是以 deployment 形式运行的，请直接修改 deployment ；如果您的 k8s api-server 服务是以 static pod 形式运行的，您需要修改对应的 manifest 文件，它的文件路径通常为 /etc/kubernetes/manifests/kube-apiserver.yaml ，修改内容如下：

apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
    - command:
        - kube-apiserver
        - --audit-log-format=json
        - --audit-log-maxage=10
        - --audit-log-maxbackup=10
        - --audit-log-maxsize=100
        - --audit-log-path=/var/log/audit
        - --audit-policy-file=/etc/cube/audit/audit-policy.yaml
        - --audit-webhook-config-file=/etc/cube/audit/audit-webhook.config
        - --authentication-token-webhook-config-file=/etc/cube/warden/webhook.config
      name: kube-apiserver
      volumeMounts:
        - mountPath: /var/log/audit
          name: audit-log
        - mountPath: /etc/cube
          name: cube
          readOnly: true
  volumes:
    - hostPath:
        path: /var/log/audit
        type: DirectoryOrCreate
      name: audit-log
    - hostPath:
        path: /etc/cube
        type: DirectoryOrCreate
      name: cube

开始安装

在 Linux 机器上执行部署脚本

KUBECUBE_VERSION=v1.0

export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置安装脚本参数

该安装模式下，需要修改以下参数：

INSTALL_KUBECUBE_MEMBER=“false”

MASTER_IP="${node ip}"

${node ip} 表示你运行脚本所在 node 机器的 ip，该 node 需要可操作 kubectl

# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="true"

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# if install k8s
INSTALL_KUBERNETES="false"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="master"

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

# master ip means master node ip of cluster
MASTER_IP="x.x.x.x"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
KUBERNETES_VERSION="1.20.9"

# +optional
# the user who can access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

等待部署完成

KubeCube 部署完成后，请根据提示信息登陆 console 管理页面

complete-deploy

使用 admin 账户登陆 console

⚠️请在登陆后修改 admin 用户的密码

login-console

4 - 添加计算集群

KubeCube 可以添加其它集群作为计算集群，前提是，计算集群能够访问管控集群的 k8s api-server 和 KubeCube，默认情况下 KubeCube 使用 NodePort 对外暴露服务，用户可自行使用 ingress 进行暴露

v1.2.x

方式一：部署新集群并添加

在 linux 机器上，需要构建 Kubernetes 集群并安装 KubeCube 依赖项

开始安装

KUBECUBE_VERSION=v1.2

export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置安装脚本参数

该安装模式下，需要修改以下参数：

INSTALL_KUBECUBE_PIVOT=“false”

INSTALL_KUBECUBE_MEMBER=“true”

INSTALL_KUBERNETES=“true”

MEMBER_CLUSTER_NAME=“member-1”

MASTER_IP="${node ip}"

KUBECUBE_HOST="${pivot node ip}"

MEMBER_CLUSTER_NAME 表示计算集群的名字，注意，不能与已有的计算集群名称同名, ${node ip} 表示你运行脚本所在 node 机器的 ip，该 node 需要可操作 kubectl, ${pivot node ip} 表示管控集群 node 机器的 ip，用于向 KubeCube 注册集群

# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install k8s
INSTALL_KUBERNETES="true"

# k8s cni, support now is calico only
CNI="calico"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.19.13, 1.20.9, 1.21.2, 1.22.2, 1.23.5
KUBERNETES_VERSION="1.23.5"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

# +optional
# KUBERNETES_BIND_ADDRESS generally is node_ip
# can be set when NODE_MODE="master" ot "control-plane-master"
# default value is $(hostname -I |awk '{print $1}')
KUBERNETES_BIND_ADDRESS="" #{node_ip}

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="true"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST="y.y.y.y"

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME="member-1"

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP=""

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

#######################################################################
# container runtime config
# if value is docker, then use docker as container runtime
# else if value is containerd, then use containerd as container runtime
#######################################################################
CONTAINER_RUNTIME="containerd"

方式二：纳管已有集群

添加已有集群只需从 console 页面导入集群信息即可

在 console 页面中导入集群信息

add-member-cluster

在 console 中确认新集群

check-cluster

v1.1.x

方式一：部署新集群并添加

在 linux 机器上，需要构建 Kubernetes 集群并安装 KubeCube 依赖项

开始安装

KUBECUBE_VERSION=v1.1

export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置安装脚本参数

该安装模式下，需要修改以下参数：

INSTALL_KUBECUBE_PIVOT=“false”

INSTALL_KUBECUBE_MEMBER=“true”

INSTALL_KUBERNETES=“true”

MEMBER_CLUSTER_NAME=“member-1”

MASTER_IP="${node ip}"

KUBECUBE_HOST="${pivot node ip}"

MEMBER_CLUSTER_NAME 表示计算集群的名字，注意，不能与已有的计算集群名称同名, ${node ip} 表示你运行脚本所在 node 机器的 ip，该 node 需要可操作 kubectl, ${pivot node ip} 表示管控集群 node 机器的 ip，用于向 KubeCube 注册集群

# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install k8s
INSTALL_KUBERNETES="true"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.20.9, 1.19.13, 1.18.20, 1.21.2
KUBERNETES_VERSION="1.20.9"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="true"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST="y.y.y.y"

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME="member-1"

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP=""

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

方式二：纳管已有集群

添加已有集群只需从 console 页面导入集群信息即可

在 console 页面中导入集群信息

add-member-cluster

在 console 中确认新集群

check-cluster

v1.0.x

方式一：部署新集群并添加

在 linux 机器上，需要构建 Kubernetes 集群并安装 KubeCube 依赖项

开始安装

KUBECUBE_VERSION=v1.0

export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置安装脚本参数

该安装模式下，需要修改以下参数：

INSTALL_KUBECUBE_PIVOT=“false”

INSTALL_KUBECUBE_MEMBER=“true”

INSTALL_KUBERNETES=“true”

MEMBER_CLUSTER_NAME=“member-1”

MASTER_IP="${node ip}"

KUBECUBE_HOST="${pivot node ip}"

MEMBER_CLUSTER_NAME 表示计算集群的名字，注意，不能与已有的计算集群名称同名 ${node ip} 表示你运行脚本所在 node 机器的 ip，该 node 需要可操作 kubectl ${pivot node ip} 表示管控集群 node 机器的 ip，用于向 KubeCube 注册集群

# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="true"

# if install k8s
INSTALL_KUBERNETES="true"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="master"

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME="member-1"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

# master ip means master node ip of cluster
MASTER_IP="x.x.x.x"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST="y.y.y.y"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
KUBERNETES_VERSION="1.20.9"

# +optional
# the user who can access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

方式二：纳管已有集群

添加已有集群需要从 console 页面获取添加集群的定制脚本

在 console 页面中获取添加集群的脚本

add-member-cluster

使用脚本添加集群

在集群的 node 机器上，使用从 console 中下载的脚本，该机器需要能够执行 kubectl

/bin/bash add_cluster.sh

等待集群添加完成

add-cluster

在 console 中确认新集群

check-cluster

5 - 添加节点

KubeCube 提供为已有集群添加节点的能力，同时也支持使用 kubeadm 的原生方式添加节点

⚠️ 注意通过 KubeCube 的脚本添加节点时，node 机器需要能够通过 ssh 访问 master 机器，支持公钥和密码两种 ssh 方式，执行脚本前可以在 node 上 ssh 到 master 测试连通性

v1.2.x

向集群添加工作节点

在新节点上执行部署脚本

KUBECUBE_VERSION=v1.2

export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待新节点加入集群

  • MASTER_IP 为 master 节点 ip
# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install k8s
INSTALL_KUBERNETES="true"

# k8s cni, support now is calico only
CNI="calico"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="node-join-master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.19.13, 1.20.9, 1.21.2, 1.22.2, 1.23.5
KUBERNETES_VERSION="1.23.5"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

# +optional
# KUBERNETES_BIND_ADDRESS generally is node_ip
# can be set when NODE_MODE="master" ot "control-plane-master"
# default value is $(hostname -I |awk '{print $1}')
KUBERNETES_BIND_ADDRESS="" #{node_ip}

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP=""

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

#######################################################################
# container runtime config
# if value is docker, then use docker as container runtime
# else if value is containerd, then use containerd as container runtime
#######################################################################
CONTAINER_RUNTIME="containerd"

向集群的 control-plane 添加 master 节点

在新节点上执行部署脚本

KUBECUBE_VERSION=v1.2

export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待新节点加入 control-plane

  • MASTER_IP 需要填已有的 master 节点 ip
  • NODE_MODE 当该模式为 node-join-control-plane 时，需要指定 master 节点们的 CONTROL_PLANE_ENDPOINT(vip)
  • CONTROL_PLANE_ENDPOINT 为高可用 vip
  • 可以根据需要选择适合自己的 ssh 方式
# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install k8s
INSTALL_KUBERNETES="true"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="node-join-master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.20.9, 1.19.13, 1.18.20, 1.21.2
KUBERNETES_VERSION="1.20.9"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP="y.y.y.y"

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

#######################################################################
# container runtime config
# if value is docker, then use docker as container runtime
# else if value is containerd, then use containerd as container runtime
#######################################################################
CONTAINER_RUNTIME="containerd"

v1.1.x

向集群添加工作节点

在新节点上执行部署脚本

KUBECUBE_VERSION=v1.1

export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

tip: 你可以通过预先下载离线包和镜像来减少部署时间 export PRE_DOWNLOAD="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待新节点加入集群

  • MASTER_IP 为 master 节点 ip
# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install k8s
INSTALL_KUBERNETES="true"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="node-join-master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.20.9, 1.19.13, 1.18.20, 1.21.2
KUBERNETES_VERSION="1.20.9"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP="y.y.y.y"

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

向集群的 control-plane 添加 master 节点

在新节点上执行部署脚本

KUBECUBE_VERSION=v1.1

export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待新节点加入 control-plane

  • MASTER_IP 需要填已有的 master 节点 ip
  • NODE_MODE 当该模式为 node-join-control-plane 时，需要指定 master 节点们的 CONTROL_PLANE_ENDPOINT(vip)
  • CONTROL_PLANE_ENDPOINT 为高可用 vip
  • 可以根据需要选择适合自己的 ssh 方式
# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install k8s
INSTALL_KUBERNETES="true"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="node-join-master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.20.9, 1.19.13, 1.18.20, 1.21.2
KUBERNETES_VERSION="1.20.9"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP="y.y.y.y"

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

v1.0.x

向集群添加工作节点

在新节点上执行部署脚本

KUBECUBE_VERSION=v1.0

export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

tip: 你可以通过预先下载离线包和镜像来减少部署时间 export PRE_DOWNLOAD="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待新节点加入集群

  • MASTER_IP 为 master 节点 ip
# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# if install k8s
INSTALL_KUBERNETES="true"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="node-join-master"

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

# master ip means master node ip of cluster
MASTER_IP="10.173.32.4"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
KUBERNETES_VERSION="1.20.9"

# +optional
# the user who can access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

向集群的 control-plane 添加 master 节点

在新节点上执行部署脚本

KUBECUBE_VERSION=v1.0

export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待新节点加入 control-plane

  • MASTER_IP 需要填已有的 master 节点 ip
  • CONTROL_PLANE_ENDPOINT 为高可用 vip
# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# if install k8s
INSTALL_KUBERNETES="true"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="node-join-control-plane"

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="10.173.32.10" #{ip}:{port} , dns

# master ip means master node ip of cluster
MASTER_IP="10.173.32.4"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
KUBERNETES_VERSION="1.20.9"

# +optional
# the user who can access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

6 - 多节点高可用部署

本文提供 Kubernetes 的高可用部署和 KubeCube 的高可用部署方案，VIP 的实现需要用户自行提供

v1.2.x

主机规划

IP 地址主机名角色
10.173.32.2lb1Keepalived & HAproxy
10.173.32.3lb2Keepalived & HAproxy
10.173.32.4master1master, etcd
10.173.32.5master2master, etcd
10.173.32.6master3master, etcd
10.173.32.7worker1worker
10.173.32.8worker2worker
10.173.32.9worker3worker
10.173.32.10vip 地址

⚠️master2、master3、worker1、worker2、worker3 需要能够通过密钥或者密码 ssh 访问 master1

部署高可用 Kubernetes

KubeCube 部署脚本提供部署高可用 k8s 的能力，当然，你也可以使用其他工具搭建高可用的 k8s 集群

开始安装

在 master1 上执行部署脚本

KUBECUBE_VERSION=v1.2

export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 Kubernetes 安装完成，master2 和 master3 加入 control-plane 的方式与之相同

  • CONTROL_PLANE_ENDPOINT 为高可用 k8s-apiserver 的 vip，在此我们用任意 master 节点的 ip 代替
# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install k8s
INSTALL_KUBERNETES="true"

# k8s cni, support now is calico only
CNI="calico"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="control-plane-master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.19.13, 1.20.9, 1.21.2, 1.22.2, 1.23.5
KUBERNETES_VERSION="1.23.5"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="10.173.32.4" #{ip}:{port} , dns

# +optional
# KUBERNETES_BIND_ADDRESS generally is node_ip
# can be set when NODE_MODE="master" ot "control-plane-master"
# default value is $(hostname -I |awk '{print $1}')
KUBERNETES_BIND_ADDRESS="" #{node_ip}

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP=""

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

#######################################################################
# container runtime config
# if value is docker, then use docker as container runtime
# else if value is containerd, then use containerd as container runtime
#######################################################################
CONTAINER_RUNTIME="containerd"

worker1 作为工作节点加入集群

在 worker1 上执行部署脚本

KUBECUBE_VERSION=v1.2

export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 worker1 加入集群，worker2 和 worker3 加入集群的方式与之相同

# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install k8s
INSTALL_KUBERNETES="true"

# k8s cni, support now is calico only
CNI="calico"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="node-join-master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.19.13, 1.20.9, 1.21.2, 1.22.2, 1.23.5
KUBERNETES_VERSION="1.23.5"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

# +optional
# KUBERNETES_BIND_ADDRESS generally is node_ip
# can be set when NODE_MODE="master" ot "control-plane-master"
# default value is $(hostname -I |awk '{print $1}')
KUBERNETES_BIND_ADDRESS="" #{node_ip}

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP=""

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

#######################################################################
# container runtime config
# if value is docker, then use docker as container runtime
# else if value is containerd, then use containerd as container runtime
#######################################################################
CONTAINER_RUNTIME="containerd"

部署高可用 KubeCube

在 master1 上执行部署脚本

KUBECUBE_VERSION=v1.2

export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 KubeCube 部署完成

  • install.conf
# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="true"

# if install k8s
INSTALL_KUBERNETES="false"

# k8s cni, support now is calico only
CNI="calico"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.19.13, 1.20.9, 1.21.2, 1.22.2, 1.23.5
KUBERNETES_VERSION="1.23.5"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

# +optional
# KUBERNETES_BIND_ADDRESS generally is node_ip
# can be set when NODE_MODE="master" ot "control-plane-master"
# default value is $(hostname -I |awk '{print $1}')
KUBERNETES_BIND_ADDRESS="" #{node_ip}

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP=""

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

#######################################################################
# container runtime config
# if value is docker, then use docker as container runtime
# else if value is containerd, then use containerd as container runtime
#######################################################################
CONTAINER_RUNTIME="containerd"

v1.1.x

主机规划

IP 地址主机名角色
10.173.32.2lb1Keepalived & HAproxy
10.173.32.3lb2Keepalived & HAproxy
10.173.32.4master1master, etcd
10.173.32.5master2master, etcd
10.173.32.6master3master, etcd
10.173.32.7worker1worker
10.173.32.8worker2worker
10.173.32.9worker3worker
10.173.32.10vip 地址

⚠️master2、master3、worker1、worker2、worker3 需要能够通过密钥或者密码 ssh 访问 master1

部署高可用 Kubernetes

KubeCube 部署脚本提供部署高可用 k8s 的能力，当然，你也可以使用其他工具搭建高可用的 k8s 集群

开始安装

在 master1 上执行部署脚本

KUBECUBE_VERSION=v1.1

export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 Kubernetes 安装完成，master2 和 master3 加入 control-plane 的方式与之相同

  • CONTROL_PLANE_ENDPOINT 为高可用 k8s-apiserver 的 vip，在此我们用任意 master 节点的 ip 代替
# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install k8s
INSTALL_KUBERNETES="true"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="control-plane-master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.20.9, 1.19.13, 1.18.20, 1.21.2
KUBERNETES_VERSION="1.20.9"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="10.173.32.4" #{ip}:{port} , dns

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP=""

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

worker1 作为工作节点加入集群

在 worker1 上执行部署脚本

KUBECUBE_VERSION=v1.1

export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 worker1 加入集群，worker2 和 worker3 加入集群的方式与之相同

# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install k8s
INSTALL_KUBERNETES="true"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="node-join-master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.20.9, 1.19.13, 1.18.20, 1.21.2
KUBERNETES_VERSION="1.20.9"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP="10.173.32.4"

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

部署高可用 KubeCube

在 master1 上执行部署脚本

KUBECUBE_VERSION=v1.1

export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 KubeCube 部署完成

  • install.conf
# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="true"

# if install k8s
INSTALL_KUBERNETES="false"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="control-plane-master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.20.9, 1.19.13, 1.18.20, 1.21.2
KUBERNETES_VERSION="1.20.9"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP="10.173.32.4"

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""
  • cube.conf

将kubecube_replicas设置为3，使得 KubeCube 使用 3 副本部署，并且由于podAntiAffinity

# custom values for kubecube

kubecube_replicas=3
kubecube_args_logLevel="info"

v1.0.x

主机规划

IP 地址主机名角色
10.173.32.2lb1Keepalived & HAproxy
10.173.32.3lb2Keepalived & HAproxy
10.173.32.4master1master, etcd
10.173.32.5master2master, etcd
10.173.32.6master3master, etcd
10.173.32.7worker1worker
10.173.32.8worker2worker
10.173.32.9worker3worker
10.173.32.10vip 地址

⚠️master2、master3、worker1、worker2、worker3 需要能够通过密钥或者密码 ssh 访问 master1

部署高可用 Kubernetes

开始安装

在 master1 上执行部署脚本

KUBECUBE_VERSION=v1.0

export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 Kubernetes 安装完成

# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# if install k8s
INSTALL_KUBERNETES="true"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="control-plane-master"

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="10.173.32.10" #{ip}:{port} , dns

# master ip means master node ip of cluster
MASTER_IP="10.173.32.4"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
KUBERNETES_VERSION="1.20.9"

# +optional
# the user who can access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

master2 节点加入 control-plane

在 master2 上执行部署脚本

KUBECUBE_VERSION=v1.0

export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 master2 加入 control-plane

master3 加入 control-plane 与此类似，仅需修改 LOCAL_IP为10.173.32.6

# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# if install k8s
INSTALL_KUBERNETES="true"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="node-join-control-plane"

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="10.173.32.10" #{ip}:{port} , dns

# master ip means master node ip of cluster
MASTER_IP="10.173.32.4"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
KUBERNETES_VERSION="1.20.9"

# +optional
# the user who can access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

worker1 作为工作节点加入集群

在 worker1 上执行部署脚本

KUBECUBE_VERSION=v1.0

export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 worker1 加入集群

worker2 和 worker3 加入集群的方式与之类似，仅需修改LOCAL_IP为本机 IP 即可

# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# if install k8s
INSTALL_KUBERNETES="true"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="node-join-master"

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

# master ip means master node ip of cluster
MASTER_IP="10.173.32.4"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
KUBERNETES_VERSION="1.20.9"

# +optional
# the user who can access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

部署高可用 KubeCube

在 master1 上执行部署脚本

KUBECUBE_VERSION=v1.0

export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 KubeCube 部署完成

  • install.conf
# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="true"

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# if install k8s
INSTALL_KUBERNETES="false"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="control-plane-master"

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

# master ip means master node ip of cluster
MASTER_IP="10.173.32.4"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
KUBERNETES_VERSION="1.20.9"

# +optional
# the user who can access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"
  • cube.conf

将kubecube_replicas设置为3，使得 KubeCube 使用 3 副本部署，并且由于podAntiAffinity，它们会运行在非controlPlane的节点上，并且每个节点仅运行单个副本

# custom values for kubecube

kubecube_replicas=3
kubecube_args_logLevel="info"

7 - 热插拔

热插拔是通过修改配置文件实现不停机更新监控、日志、审计等组件的启停和配置。

热插拔

热插拔的实现是基于 helm ，因此集群需要预先安装好 helm3 版本。

登录管控 k8s 集群，执行命令可以查看热插拔配置

# kubectl get hotplug
NAME            PHASE     AGE
common          running   23h
pivot-cluster   running   16d

其中：

  • common 表示公共的热插拔配置，pivot-cluster 表示 pivot-cluster 这个 k8s 集群的热插拔配置。

  • 允许自定义各个集群的热插拔配置覆盖 common 热插拔配置实现个性化配置 k8s 集群组件热插拔。

  • 热插拔配置的 .metadata.name 要求与k8s集群名称一致，所有 k8s 集群信息查看命令为 kubectl get cluster。

Common配置说明

目前，默认的 common 配置包括以下几个组件：

audit：操作审计

logseer：日志管理组件，仅在管控集群安装

logagent：日志采集代理组件

kubecube-monitoring：监控 prometheuse 组件

kubecube-thanos：监控 thanos 组件，仅在管控集群安装

示例如下：

apiVersion: hotplug.kubecube.io/v1
kind: Hotplug
metadata:
  annotations:
    kubecube.io/sync: "true"					## 同步信号，kubecube会将这个配置同步到各个集群
  name: common                                  ## 公共配置为cmmon，其余集群特殊配置为集群名字
spec:
  component:
  - name: audit                                 ## 审计日志
    status: disabled
  - name: logseer                               ## 日志管理组件
    namespace: logseer
    pkgName: logseer-v1.0.0.tgz
    status: disabled                            ## 启停标识：这里disabled为禁用
  - name: logagent                              ## 日志采集代理组件
    namespace: logagent
    pkgName: logagent-v1.0.0.tgz
    status: enabled                             ## 启停标识：这里enabled为启用
    env: |                                      ## 环境变量
      clustername: "{{.cluster}}"               ## {{.cluster}} 程序会自动注入集群名字替换
  - name: kubecube-monitoring                   ## 监控组件
    namespace: kubecube-monitoring
    pkgName: kubecube-monitoring-15.4.7.tgz
    status: enabled
    env: |
      grafana:
        enabled: false
      prometheus:
        prometheusSpec:
          externalLabels:
            cluster: "{{.cluster}}"
          remoteWrite:
          - url: http://10.173.32.42:31291/api/v1/receive      
  - name: kubecube-thanos                       ## 监控thanos组件
    namespace: kubecube-monitoring
    pkgName: thanos-3.18.0.tgz
    status: disabled
status:                                         ## message显示各个组件运行状态，phase显示总体运行状态
  message: '{"kubecube-monitoring":"release is running","kubecube-thanos":"release    
    is running","logagent":"release is running","logseer":"release is running"}'
  phase: running

每一个组件基本包含5个要素，在 spec.component 下：

name：组件名称

namespace：指定组件部署的命名空间，若指定的命名空间不存在，会自动以该字段值去创建一个命名空间。

pkgName：安装包名称，安装包默认存放路径为 warden 容器里的 /root/helmchartpkg，以 emptydir 形式存在。

status：组件是否启用

env：环境变量配置

每一个要素都可以使用集群独特配置进行覆盖，未覆盖的要素则依然使用 common 里的配置。

管控集群配置说明

Pivot-cluster 配置是管控集群配置，即 KubeCube 所在的集群。集群独特的配置会与 common 的配置结合，用于个性化配置集群组件，结合时遇到相同字段pivot-cluster 优先。

apiVersion: hotplug.kubecube.io/v1
kind: Hotplug
metadata:
  annotations:
    kubecube.io/sync: "true"            ## 同步信号，kubecube会将这个配置同步到其他集群
  name: pivot-cluster                   ## 与集群名字一致，指明这是pivot-cluster这个集群的热插拔配置
spec:
  component:
  - name: logseer                       ## 日志管理组件
    status: enabled                     ## 结合common设置为disabled，这里设置为enabled，标识其余集群不启用，pivot-cluster集群启用
  - name: kubecube-monitoring
    env: |
      grafana:
        enabled: true
      prometheus:
        prometheusSpec:
          externalLabels:
            cluster: "{{.cluster}}"
          remoteWrite:
          - url: http://thanos-receive:19291/api/v1/receive      
  - name: kubecube-thanos
    status: enabled
    env: |
      receive:
        replicaCount: 1
        replicationFactor: 1      
status:
  message: '{"kubecube-monitoring":"release is running","kubecube-thanos":"release
    is running","logagent":"release is running","logseer":"release is running"}'
  phase: running

8 - KubeCube 卸载

KubeCube 提供一键卸载脚本

⚠️ 注意，卸载脚本仅用于使用 KubeCube 部署脚本安装的环境，不应该用于任何正式环境

下载卸载脚本

curl -o cleanup.sh https://kubecube.nos-eastchina1.126.net/hack/cleanup.sh

一键卸载

一键卸载按顺序卸载 kubecube, kubernetes 和 docker

/bin/bash cleanup.sh all

指定卸载

卸载脚本可以指定卸载对象，但需注意卸载依赖

卸载 KubeCube

删除 kubecube chart release，并清理 kubecube 在本机上的相关文件

/bin/bash cleanup.sh kubecube

卸载 Kubernetes

卸载 kubernetes 集群，并清理 kubernetes 在本机上的相关文件

/bin/bash cleanup.sh k8s

卸载 Docker

卸载 docker，并清理 docker 在本机上的相关文件

⚠️注意：需在卸载 kubernetes 后才能卸载 docker，不然会造成 kubernetes 集群异常

/bin/bash cleanup.sh docker

9 - 已有系统接入

9.1 - Prometheus

KubeCube 在部署时会自动安装 Prometheus 等监控组件，以实现监控功能。本文档介绍了如何在 KubeCube 中接入用户已有的 Prometheus。

准备工作

  1. 在集群部署好 Prometheus，Prometheus 可以正常监控到集群资源的数据；

  2. 在集群中部署好 KubeCube；

    说明：部署好 KubeCube 后，由于集群内已有 Prometheus operator，多个 operator 会导致集群内 Prometheus 相关功能不可用，需要卸载 KubeCube 监控组件或删除本地 operator。

  3. 以平台管理员角色登录 KubeCube 管控集群。

步骤

1、添加 Label

由于 KubeCube 需要实现多集群监控，因此在 KubeCube 查询监控数据时，都会在 query 表达式中添加 cluster={clusterName} 来进行集群过滤。用户需要在 Prometheus 的 exporter 中添加这一 label，前端查询监控数据时才能查询到结果。

2、卸载 KubeCube 监控组件

方式一 页面操作：

  1. 点击页面右上角【切换到控制台】，点击任意空间，进入到控制台页面；

  2. 在左侧菜单栏点击【自定义资源CRD】，进入到集群级别 CRD 列表，可以点击右上方输入 “hotplug” 进行搜索，找到 “hotplugs.hotplug.kubecube.io” CRD，点击【v1】版本进入 CRD 详情页；

  3. 选择 common 实例，点击【设置YAML】，找到 spec.component. name=kubecube-monitoring，将 “status” 改成 “disabled”，即卸载 KubeCube 自带监控组件。

方式二 命令行操作：

  1. kubectl edit hotplug common
  2. 找到 spec.component. name=kubecube-monitoring，将 “status” 改成 “disabled”。

详细配置说明见 热插拔 。

3、部署 ServiceMonitor

查看集群资源

查看控制台内监控数据，需要部署两个 ServiceMonitor：kubelet 和 kube-state-metrics 的 ServiceMonitor，样例如下：

  • 部署 kubecube-monitoring-kubelet

    apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kubecube-monitoring-kubelet
  namespace: kubecube-monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    honorLabels: true
    port: https-metrics
    relabelings:
    - sourceLabels:
      - __metrics_path__
      targetLabel: metrics_path
    scheme: https
    tlsConfig:
      caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecureSkipVerify: true
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    honorLabels: true
    path: /metrics/cadvisor
    port: https-metrics
    relabelings:
    - sourceLabels:
      - __metrics_path__
      targetLabel: metrics_path
    scheme: https
    tlsConfig:
      caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecureSkipVerify: true
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    honorLabels: true
    path: /metrics/probes
    port: https-metrics
    relabelings:
    - sourceLabels:
      - __metrics_path__
      targetLabel: metrics_path
    scheme: https
    tlsConfig:
      caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecureSkipVerify: true
  jobLabel: k8s-app
  namespaceSelector:
    matchNames:
    - kube-system
  selector:
    matchLabels:
      k8s-app: kubelet

  • 部署 kubecube-monitoring-kube-state-metrics

    apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kubecube-monitoring-kube-state-metrics
  namespace: kubecube-monitoring
spec:
  endpoints:
  - honorLabels: true
    port: http
  selector:
    matchLabels:
      app.kubernetes.io/instance: kubecube-monitoring
      app.kubernetes.io/name: kube-state-metrics

查看组件监控

当前 KubeCube 平台支持对组件的监控视图可视化查询，详细说明见 平台组件监控 。接入外部监控后，用户可按需在集群内部署对应组件的 ServiceMonitor。

各个 ServiceMonitor 的 yaml 可参考 https://github.com/kubecube-io/charts/tree/main/kubecube-monitoring/templates/exporters。

4、部署 Dashboard

查看集群资源（控制台内监控数据），需要部署：

查看组件监控，可以部署对应的 Dashboard：

5、修改 Nginx 配置

  1. 使用命令行：kubectl edit configmap nginx-config -n kubecube-system

  2. 找到原有的地址配置，修改为自有 Prometheus 地址

    upstream monitoring {
    server kubecube-thanos-query.kubecube-monitoring:9090;
}

    即 将kubecube-thanos-query.kubecube-monitoring:9090 替换为外部地址。

  3. 重启 pod：kubectl delete pod frontend-xxxxxx-xxxxx -n kubecube-system

9.2 - ElasticSearch

KubeCube 提供了日志服务和操作审计服务，默认关闭。用户在开启后，日志服务和操作审计服务均会将日志发送到 ElasticSearch 进行存储，由 ElasticSearch 对日志进行管理。用户可以在 热插拔 中修改配置，安装内部 ElasticSearch，也可以配置外部的 ElasticSearch 地址，对接已有的 ElasticSearch。下面分别介绍如何在这两个功能中接入外部 ElasticSearch。

日志

方式一 页面操作：

  1. 点击页面右上角【切换到控制台】，点击任意空间，进入到控制台页面；

  2. 在左侧菜单栏点击【自定义资源CRD】，进入到集群级别 CRD 列表，可以点击右上方输入 “hotplug” 进行搜索，找到 “hotplugs.hotplug.kubecube.io” CRD，点击【v1】版本进入 CRD 详情页；

  3. 选择 common 实例，点击【设置YAML】，找到 spec.component. name=logseer，添加环境变量，如：

- name: logseer
  namespace: logseer
  pkgName: logseer-v1.0.0.tgz
  status: disabled
  env: |
    address: elasticsearch-master.elasticsearch.svc

方式二 命令行操作：

  1. kubectl edit hotplug pivot-cluster
  2. 找到 spec.component. name=logseer，添加环境变量，同上。

详细配置说明见 热插拔 。

操作审计

  1. kubectl edit deploy audit -n kubecube-system

  2. 添加环境变量：AUDIT_WEBHOOK_HOST、AUDIT_WEBHOOK_INDEX、AUDIT_WEBHOOK_TYPE，如

    env:
- name: AUDIT_WEBHOOK_HOST
  value: http://elasticsearch-master.elasticsearch:9200
- name: AUDIT_WEBHOOK_INDEX
  value: audit
- name: AUDIT_WEBHOOK_TYPE
  value: logs

注：如果同时配置了内部和外部 ElasticSearch，审计日志将优先发到外部 ElasticSearch。

其他详细说明见：操作审计 。