1 - 部署环境要求

在进行 All In One 或者多节点部署之前,请按照以下内容确认环境要求

系统版本及硬件要求

操作系统最低要求
Ubuntu 16.04, 18.04CPU:4 核,内存:8 G,磁盘空间:20 G
Debian Buster, StretchCPU:4 核,内存:8 G,磁盘空间:20 G
CentOS 7.xCPU:4 核,内存:8 G,磁盘空间:20 G
Kylin v10CPU:4 核,内存:8 G,磁盘空间:20 G
OS ARCH硬件要求
AMD 64Intel 系列,AMD 系列
ARM 64Phytium 2000(国产化arm64芯片理论上都支持,但未经过充分测试)

以上系统配置要求适用于 KubeCube 默认最小化 All In One 模式安装,如需启动更多可插拔组件和拓展功能,建议机器配置为 8 核 CPU 和 16 G 内存

容器运行时

  • KubeCube 支持的 cri 跟随 Kubernetes 标准

  • KubeCube 部署脚本支持的 cri 如下:

支持的容器运行时版本
Docker19.3.12+
Containerd1.5.x

节点上若无容器运行时,部署脚本将自动安装 containerd 1.5.5 作为容器运行时

网络插件

  • KubeCube 支持的 cni 跟随 Kubernetes 标准

  • KubeCube 部署脚本的 cni 目前支持 calico

Kubernetes 版本

KubeCube 支持的 k8s 版本为 v1.18 ~ v1.23 KubeCube 部署脚本支持的 k8s 版本为 v1.18.20、v1.19.13、v1.20.9、v1.21.2、v1.22.2、v1.23.5

前置准备

在使用部署脚本开始 KubeCube 的安装时,脚本会检测环境,并提示需要安装缺失的依赖

env-check

监控组件说明

KubeCube 会默认安装 Prometheus 等监控组件,如果选择在已有k8s集群中部署 KubeCube,并且集群中已安装 Mertics Server,安装 KubeCube 后 Prometheus 会和 Mertics Server 产生冲突,导致监控功能不可用。需要在安装 KubeCube 后执行以下步骤:

  1. 点击页面右上角【切换到控制台】,点击任意空间,进入到控制台页面;

  2. 在左侧菜单栏点击【自定义资源CRD】,进入到集群级别 CRD 列表,可以点击右上方输入 “hotplug” 进行搜索,找到 “hotplugs.hotplug.kubecube.io” CRD,点击【v1】版本进入 CRD 详情页;

  3. 选择 common 实例,点击【设置YAML】,找到 spec.component. name=kubecube-monitoring,添加环境变量 prometheusAdapter.enabled=false,如:

  - env: |
      grafana:
        enabled: false
      prometheus:
        prometheusSpec:
          externalLabels:
            cluster: "{{.cluster}}"
          remoteWrite:
          - url: http://10.173.32.129:31291/api/v1/receive
      prometheusAdapter:      
  			enabled: false
    name: kubecube-monitoring
    namespace: kubecube-monitoring
    pkgName: kubecube-monitoring-15.4.10.tgz
    status: enabled

2 - All In One 最小化部署

对于想要快速开始、快速体验的用户来说,All In One 是最佳的安装方式

v1.2.x

在 Linux 上部署 KubeCube

开始安装

在 Linux 机器上执行部署脚本

KUBECUBE_VERSION=v1.2
curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

等待部署完成

KubeCube 部署完成后,请根据提示信息登陆 console 管理页面

complete-deploy

使用 admin 账户登陆 console

⚠️请在登陆后修改 admin 用户的密码

login-console

v1.1.x

在 Linux 上部署 KubeCube

开始安装

在 Linux 机器上执行部署脚本

KUBECUBE_VERSION=v1.1
curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

等待部署完成

KubeCube 部署完成后,请根据提示信息登陆 console 管理页面

complete-deploy

使用 admin 账户登陆 console

⚠️请在登陆后修改 admin 用户的密码

login-console

v1.0.x

在 Linux 上部署 KubeCube

开始安装

在 Linux 机器上执行部署脚本

KUBECUBE_VERSION=v1.0
curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

等待部署完成

KubeCube 部署完成后,请根据提示信息登陆 console 管理页面

complete-deploy

使用 admin 账户登陆 console

⚠️请在登陆后修改 admin 用户的密码

login-console

3 - 在已有k8s集群中部署KubeCube

v1.2.x

在 Kubernetes 集群中部署 KubeCube

开始安装

在 Linux 机器上执行部署脚本

KUBECUBE_VERSION=v1.2
export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置安装脚本参数

该安装模式下,需要修改以下参数:

# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="true"

# if install k8s
INSTALL_KUBERNETES="false"

# k8s cni, support now is calico only
CNI="calico"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.19.13, 1.20.9, 1.21.2, 1.22.2, 1.23.5
KUBERNETES_VERSION="1.23.5"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

# +optional
# KUBERNETES_BIND_ADDRESS generally is node_ip
# can be set when NODE_MODE="master" ot "control-plane-master"
# default value is $(hostname -I |awk '{print $1}')
KUBERNETES_BIND_ADDRESS="" #{node_ip}

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP=""

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

#######################################################################
# container runtime config
# if value is docker, then use docker as container runtime
# else if value is containerd, then use containerd as container runtime
#######################################################################
CONTAINER_RUNTIME="containerd"

等待部署完成

KubeCube 部署完成后,请根据提示信息登陆 console 管理页面

complete-deploy

使用 admin 账户登陆 console

⚠️请在登陆后修改 admin 用户的密码

login-console

v1.1.x

在 Kubernetes 集群中部署 KubeCube

开始安装

在 Linux 机器上执行部署脚本

KUBECUBE_VERSION=v1.1
export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置安装脚本参数

该安装模式下,需要修改以下参数:

INSTALL_KUBECUBE_MEMBER=“false”

MASTER_IP="${node ip}"

${node ip} 表示你运行脚本所在 node 机器的 ip,该 node 需要可操作 kubectl

# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="true"

# if install k8s
INSTALL_KUBERNETES="false"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.20.9, 1.19.13, 1.18.20, 1.21.2
KUBERNETES_VERSION="1.20.9"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP=""

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

等待部署完成

KubeCube 部署完成后,请根据提示信息登陆 console 管理页面

complete-deploy

使用 admin 账户登陆 console

⚠️请在登陆后修改 admin 用户的密码

login-console

v1.0.x

在 Kubernetes 集群中部署 KubeCube

⚠️修改 Kubernetes API-Server 配置

必要性

  1. KubeCube 对多集群提供统一的认证和鉴权服务,需要使用 k8s api-server 的 auth-webhook 能力来做拓展。

  2. KubeCube 提供对 k8s-apiserver 日志进行审计的能力,这需要为 k8s api-server 指定审计服务后端。

修改操作

如果您的 k8s api-server 服务是以 deployment 形式运行的,请直接修改 deployment ;如果您的 k8s api-server 服务是以 static pod 形式运行的,您需要修改对应的 manifest 文件,它的文件路径通常为 /etc/kubernetes/manifests/kube-apiserver.yaml ,修改内容如下:

apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
    - command:
        - kube-apiserver
        - --audit-log-format=json
        - --audit-log-maxage=10
        - --audit-log-maxbackup=10
        - --audit-log-maxsize=100
        - --audit-log-path=/var/log/audit
        - --audit-policy-file=/etc/cube/audit/audit-policy.yaml
        - --audit-webhook-config-file=/etc/cube/audit/audit-webhook.config
        - --authentication-token-webhook-config-file=/etc/cube/warden/webhook.config
      name: kube-apiserver
      volumeMounts:
        - mountPath: /var/log/audit
          name: audit-log
        - mountPath: /etc/cube
          name: cube
          readOnly: true
  volumes:
    - hostPath:
        path: /var/log/audit
        type: DirectoryOrCreate
      name: audit-log
    - hostPath:
        path: /etc/cube
        type: DirectoryOrCreate
      name: cube

开始安装

在 Linux 机器上执行部署脚本

KUBECUBE_VERSION=v1.0
export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置安装脚本参数

该安装模式下,需要修改以下参数:

INSTALL_KUBECUBE_MEMBER=“false”

MASTER_IP="${node ip}"

${node ip} 表示你运行脚本所在 node 机器的 ip,该 node 需要可操作 kubectl

# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="true"

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# if install k8s
INSTALL_KUBERNETES="false"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="master"

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

# master ip means master node ip of cluster
MASTER_IP="x.x.x.x"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
KUBERNETES_VERSION="1.20.9"

# +optional
# the user who can access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

等待部署完成

KubeCube 部署完成后,请根据提示信息登陆 console 管理页面

complete-deploy

使用 admin 账户登陆 console

⚠️请在登陆后修改 admin 用户的密码

login-console

4 - 添加计算集群

KubeCube 可以添加其它集群作为计算集群,前提是,计算集群能够访问管控集群的 k8s api-server 和 KubeCube,默认情况下 KubeCube 使用 NodePort 对外暴露服务,用户可自行使用 ingress 进行暴露

v1.2.x

方式一:部署新集群并添加

在 linux 机器上,需要构建 Kubernetes 集群并安装 KubeCube 依赖项

开始安装

KUBECUBE_VERSION=v1.2
export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置安装脚本参数

该安装模式下,需要修改以下参数:

INSTALL_KUBECUBE_PIVOT=“false”

INSTALL_KUBECUBE_MEMBER=“true”

INSTALL_KUBERNETES=“true”

MEMBER_CLUSTER_NAME=“member-1”

MASTER_IP="${node ip}"

KUBECUBE_HOST="${pivot node ip}"

MEMBER_CLUSTER_NAME 表示计算集群的名字,注意,不能与已有的计算集群名称同名, ${node ip} 表示你运行脚本所在 node 机器的 ip,该 node 需要可操作 kubectl, ${pivot node ip} 表示管控集群 node 机器的 ip,用于向 KubeCube 注册集群

# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install k8s
INSTALL_KUBERNETES="true"

# k8s cni, support now is calico only
CNI="calico"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.19.13, 1.20.9, 1.21.2, 1.22.2, 1.23.5
KUBERNETES_VERSION="1.23.5"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

# +optional
# KUBERNETES_BIND_ADDRESS generally is node_ip
# can be set when NODE_MODE="master" ot "control-plane-master"
# default value is $(hostname -I |awk '{print $1}')
KUBERNETES_BIND_ADDRESS="" #{node_ip}

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="true"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST="y.y.y.y"

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME="member-1"

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP=""

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

#######################################################################
# container runtime config
# if value is docker, then use docker as container runtime
# else if value is containerd, then use containerd as container runtime
#######################################################################
CONTAINER_RUNTIME="containerd"

方式二:纳管已有集群

添加已有集群只需从 console 页面导入集群信息即可

在 console 页面中导入集群信息

add-member-cluster

在 console 中确认新集群

check-cluster

v1.1.x

方式一:部署新集群并添加

在 linux 机器上,需要构建 Kubernetes 集群并安装 KubeCube 依赖项

开始安装

KUBECUBE_VERSION=v1.1
export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置安装脚本参数

该安装模式下,需要修改以下参数:

INSTALL_KUBECUBE_PIVOT=“false”

INSTALL_KUBECUBE_MEMBER=“true”

INSTALL_KUBERNETES=“true”

MEMBER_CLUSTER_NAME=“member-1”

MASTER_IP="${node ip}"

KUBECUBE_HOST="${pivot node ip}"

MEMBER_CLUSTER_NAME 表示计算集群的名字,注意,不能与已有的计算集群名称同名, ${node ip} 表示你运行脚本所在 node 机器的 ip,该 node 需要可操作 kubectl, ${pivot node ip} 表示管控集群 node 机器的 ip,用于向 KubeCube 注册集群

# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install k8s
INSTALL_KUBERNETES="true"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.20.9, 1.19.13, 1.18.20, 1.21.2
KUBERNETES_VERSION="1.20.9"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="true"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST="y.y.y.y"

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME="member-1"

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP=""

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

方式二:纳管已有集群

添加已有集群只需从 console 页面导入集群信息即可

在 console 页面中导入集群信息

add-member-cluster

在 console 中确认新集群

check-cluster

v1.0.x

方式一:部署新集群并添加

在 linux 机器上,需要构建 Kubernetes 集群并安装 KubeCube 依赖项

开始安装

KUBECUBE_VERSION=v1.0
export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置安装脚本参数

该安装模式下,需要修改以下参数:

INSTALL_KUBECUBE_PIVOT=“false”

INSTALL_KUBECUBE_MEMBER=“true”

INSTALL_KUBERNETES=“true”

MEMBER_CLUSTER_NAME=“member-1”

MASTER_IP="${node ip}"

KUBECUBE_HOST="${pivot node ip}"

MEMBER_CLUSTER_NAME 表示计算集群的名字,注意,不能与已有的计算集群名称同名 ${node ip} 表示你运行脚本所在 node 机器的 ip,该 node 需要可操作 kubectl ${pivot node ip} 表示管控集群 node 机器的 ip,用于向 KubeCube 注册集群

# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="true"

# if install k8s
INSTALL_KUBERNETES="true"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="master"

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME="member-1"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

# master ip means master node ip of cluster
MASTER_IP="x.x.x.x"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST="y.y.y.y"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
KUBERNETES_VERSION="1.20.9"

# +optional
# the user who can access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

方式二:纳管已有集群

添加已有集群需要从 console 页面获取添加集群的定制脚本

在 console 页面中获取添加集群的脚本

add-member-cluster

使用脚本添加集群

在集群的 node 机器上,使用从 console 中下载的脚本,该机器需要能够执行 kubectl

/bin/bash add_cluster.sh

等待集群添加完成

add-cluster

在 console 中确认新集群

check-cluster

5 - 添加节点

KubeCube 提供为已有集群添加节点的能力,同时也支持使用 kubeadm 的原生方式添加节点

⚠️ 注意通过 KubeCube 的脚本添加节点时,node 机器需要能够通过 ssh 访问 master 机器,支持公钥和密码两种 ssh 方式,执行脚本前可以在 node 上 ssh 到 master 测试连通性

v1.2.x

向集群添加工作节点

在新节点上执行部署脚本

KUBECUBE_VERSION=v1.2
export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待新节点加入集群

  • MASTER_IP 为 master 节点 ip
# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install k8s
INSTALL_KUBERNETES="true"

# k8s cni, support now is calico only
CNI="calico"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="node-join-master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.19.13, 1.20.9, 1.21.2, 1.22.2, 1.23.5
KUBERNETES_VERSION="1.23.5"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

# +optional
# KUBERNETES_BIND_ADDRESS generally is node_ip
# can be set when NODE_MODE="master" ot "control-plane-master"
# default value is $(hostname -I |awk '{print $1}')
KUBERNETES_BIND_ADDRESS="" #{node_ip}

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP=""

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

#######################################################################
# container runtime config
# if value is docker, then use docker as container runtime
# else if value is containerd, then use containerd as container runtime
#######################################################################
CONTAINER_RUNTIME="containerd"

向集群的 control-plane 添加 master 节点

在新节点上执行部署脚本

KUBECUBE_VERSION=v1.2
export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待新节点加入 control-plane

  • MASTER_IP 需要填已有的 master 节点 ip
  • NODE_MODE 当该模式为 node-join-control-plane 时,需要指定 master 节点们的 CONTROL_PLANE_ENDPOINT(vip)
  • CONTROL_PLANE_ENDPOINT 为高可用 vip
  • 可以根据需要选择适合自己的 ssh 方式
# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install k8s
INSTALL_KUBERNETES="true"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="node-join-master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.20.9, 1.19.13, 1.18.20, 1.21.2
KUBERNETES_VERSION="1.20.9"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP="y.y.y.y"

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

#######################################################################
# container runtime config
# if value is docker, then use docker as container runtime
# else if value is containerd, then use containerd as container runtime
#######################################################################
CONTAINER_RUNTIME="containerd"

v1.1.x

向集群添加工作节点

在新节点上执行部署脚本

KUBECUBE_VERSION=v1.1
export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

tip: 你可以通过预先下载离线包和镜像来减少部署时间 export PRE_DOWNLOAD="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待新节点加入集群

  • MASTER_IP 为 master 节点 ip
# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install k8s
INSTALL_KUBERNETES="true"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="node-join-master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.20.9, 1.19.13, 1.18.20, 1.21.2
KUBERNETES_VERSION="1.20.9"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP="y.y.y.y"

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

向集群的 control-plane 添加 master 节点

在新节点上执行部署脚本

KUBECUBE_VERSION=v1.1
export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待新节点加入 control-plane

  • MASTER_IP 需要填已有的 master 节点 ip
  • NODE_MODE 当该模式为 node-join-control-plane 时,需要指定 master 节点们的 CONTROL_PLANE_ENDPOINT(vip)
  • CONTROL_PLANE_ENDPOINT 为高可用 vip
  • 可以根据需要选择适合自己的 ssh 方式
# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install k8s
INSTALL_KUBERNETES="true"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="node-join-master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.20.9, 1.19.13, 1.18.20, 1.21.2
KUBERNETES_VERSION="1.20.9"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP="y.y.y.y"

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

v1.0.x

向集群添加工作节点

在新节点上执行部署脚本

KUBECUBE_VERSION=v1.0
export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

tip: 你可以通过预先下载离线包和镜像来减少部署时间 export PRE_DOWNLOAD="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待新节点加入集群

  • MASTER_IP 为 master 节点 ip
# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# if install k8s
INSTALL_KUBERNETES="true"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="node-join-master"

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

# master ip means master node ip of cluster
MASTER_IP="10.173.32.4"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
KUBERNETES_VERSION="1.20.9"

# +optional
# the user who can access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

向集群的 control-plane 添加 master 节点

在新节点上执行部署脚本

KUBECUBE_VERSION=v1.0
export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待新节点加入 control-plane

  • MASTER_IP 需要填已有的 master 节点 ip
  • CONTROL_PLANE_ENDPOINT 为高可用 vip
# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# if install k8s
INSTALL_KUBERNETES="true"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="node-join-control-plane"

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="10.173.32.10" #{ip}:{port} , dns

# master ip means master node ip of cluster
MASTER_IP="10.173.32.4"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
KUBERNETES_VERSION="1.20.9"

# +optional
# the user who can access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

6 - 多节点高可用部署

本文提供 Kubernetes 的高可用部署和 KubeCube 的高可用部署方案,VIP 的实现需要用户自行提供

v1.2.x

主机规划

IP 地址主机名角色
10.173.32.2lb1Keepalived & HAproxy
10.173.32.3lb2Keepalived & HAproxy
10.173.32.4master1master, etcd
10.173.32.5master2master, etcd
10.173.32.6master3master, etcd
10.173.32.7worker1worker
10.173.32.8worker2worker
10.173.32.9worker3worker
10.173.32.10vip 地址

⚠️master2、master3、worker1、worker2、worker3 需要能够通过密钥或者密码 ssh 访问 master1

部署高可用 Kubernetes

KubeCube 部署脚本提供部署高可用 k8s 的能力,当然,你也可以使用其他工具搭建高可用的 k8s 集群

开始安装

在 master1 上执行部署脚本

KUBECUBE_VERSION=v1.2
export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待 Kubernetes 安装完成,master2 和 master3 加入 control-plane 的方式与之相同

  • CONTROL_PLANE_ENDPOINT 为高可用 k8s-apiserver 的 vip,在此我们用任意 master 节点的 ip 代替
# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install k8s
INSTALL_KUBERNETES="true"

# k8s cni, support now is calico only
CNI="calico"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="control-plane-master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.19.13, 1.20.9, 1.21.2, 1.22.2, 1.23.5
KUBERNETES_VERSION="1.23.5"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="10.173.32.4" #{ip}:{port} , dns

# +optional
# KUBERNETES_BIND_ADDRESS generally is node_ip
# can be set when NODE_MODE="master" ot "control-plane-master"
# default value is $(hostname -I |awk '{print $1}')
KUBERNETES_BIND_ADDRESS="" #{node_ip}

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP=""

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

#######################################################################
# container runtime config
# if value is docker, then use docker as container runtime
# else if value is containerd, then use containerd as container runtime
#######################################################################
CONTAINER_RUNTIME="containerd"

worker1 作为工作节点加入集群

在 worker1 上执行部署脚本

KUBECUBE_VERSION=v1.2
export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待 worker1 加入集群,worker2 和 worker3 加入集群的方式与之相同

# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install k8s
INSTALL_KUBERNETES="true"

# k8s cni, support now is calico only
CNI="calico"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="node-join-master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.19.13, 1.20.9, 1.21.2, 1.22.2, 1.23.5
KUBERNETES_VERSION="1.23.5"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

# +optional
# KUBERNETES_BIND_ADDRESS generally is node_ip
# can be set when NODE_MODE="master" ot "control-plane-master"
# default value is $(hostname -I |awk '{print $1}')
KUBERNETES_BIND_ADDRESS="" #{node_ip}

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP=""

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

#######################################################################
# container runtime config
# if value is docker, then use docker as container runtime
# else if value is containerd, then use containerd as container runtime
#######################################################################
CONTAINER_RUNTIME="containerd"

部署高可用 KubeCube

在 master1 上执行部署脚本

KUBECUBE_VERSION=v1.2
export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待 KubeCube 部署完成

  • install.conf
# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="true"

# if install k8s
INSTALL_KUBERNETES="false"

# k8s cni, support now is calico only
CNI="calico"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.19.13, 1.20.9, 1.21.2, 1.22.2, 1.23.5
KUBERNETES_VERSION="1.23.5"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

# +optional
# KUBERNETES_BIND_ADDRESS generally is node_ip
# can be set when NODE_MODE="master" ot "control-plane-master"
# default value is $(hostname -I |awk '{print $1}')
KUBERNETES_BIND_ADDRESS="" #{node_ip}

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP=""

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

#######################################################################
# container runtime config
# if value is docker, then use docker as container runtime
# else if value is containerd, then use containerd as container runtime
#######################################################################
CONTAINER_RUNTIME="containerd"

v1.1.x

主机规划

IP 地址主机名角色
10.173.32.2lb1Keepalived & HAproxy
10.173.32.3lb2Keepalived & HAproxy
10.173.32.4master1master, etcd
10.173.32.5master2master, etcd
10.173.32.6master3master, etcd
10.173.32.7worker1worker
10.173.32.8worker2worker
10.173.32.9worker3worker
10.173.32.10vip 地址

⚠️master2、master3、worker1、worker2、worker3 需要能够通过密钥或者密码 ssh 访问 master1

部署高可用 Kubernetes

KubeCube 部署脚本提供部署高可用 k8s 的能力,当然,你也可以使用其他工具搭建高可用的 k8s 集群

开始安装

在 master1 上执行部署脚本

KUBECUBE_VERSION=v1.1
export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待 Kubernetes 安装完成,master2 和 master3 加入 control-plane 的方式与之相同

  • CONTROL_PLANE_ENDPOINT 为高可用 k8s-apiserver 的 vip,在此我们用任意 master 节点的 ip 代替
# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install k8s
INSTALL_KUBERNETES="true"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="control-plane-master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.20.9, 1.19.13, 1.18.20, 1.21.2
KUBERNETES_VERSION="1.20.9"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="10.173.32.4" #{ip}:{port} , dns

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP=""

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

worker1 作为工作节点加入集群

在 worker1 上执行部署脚本

KUBECUBE_VERSION=v1.1
export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待 worker1 加入集群,worker2 和 worker3 加入集群的方式与之相同

# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install k8s
INSTALL_KUBERNETES="true"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="node-join-master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.20.9, 1.19.13, 1.18.20, 1.21.2
KUBERNETES_VERSION="1.20.9"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP="10.173.32.4"

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""

部署高可用 KubeCube

在 master1 上执行部署脚本

KUBECUBE_VERSION=v1.1
export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待 KubeCube 部署完成

  • install.conf
# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="true"

# if install k8s
INSTALL_KUBERNETES="false"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="control-plane-master"

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
# support now is: 1.20.9, 1.19.13, 1.18.20, 1.21.2
KUBERNETES_VERSION="1.20.9"

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

#######################################################################
# member cluster config
# used when INSTALL_KUBECUBE_MEMBER="true"
#######################################################################

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

#######################################################################
# ssh config
# used when NODE_MODE="node-join-master" or node-join-control-plane
#######################################################################

# +optional
# master ip means master node ip of cluster
MASTER_IP="10.173.32.4"

# +optional
# the user who can access master node, it can be empty
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

#######################################################################
# offline config
# used when offline install choose, must lift offline pkg first
#######################################################################

OFFLINE_INSTALL="false"

OFFLINE_PKG_PATH=""
  • cube.conf

kubecube_replicas设置为3,使得 KubeCube 使用 3 副本部署,并且由于podAntiAffinity

# custom values for kubecube

kubecube_replicas=3
kubecube_args_logLevel="info"

v1.0.x

主机规划

IP 地址主机名角色
10.173.32.2lb1Keepalived & HAproxy
10.173.32.3lb2Keepalived & HAproxy
10.173.32.4master1master, etcd
10.173.32.5master2master, etcd
10.173.32.6master3master, etcd
10.173.32.7worker1worker
10.173.32.8worker2worker
10.173.32.9worker3worker
10.173.32.10vip 地址

⚠️master2、master3、worker1、worker2、worker3 需要能够通过密钥或者密码 ssh 访问 master1

部署高可用 Kubernetes

开始安装

在 master1 上执行部署脚本

KUBECUBE_VERSION=v1.0
export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待 Kubernetes 安装完成

# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# if install k8s
INSTALL_KUBERNETES="true"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="control-plane-master"

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="10.173.32.10" #{ip}:{port} , dns

# master ip means master node ip of cluster
MASTER_IP="10.173.32.4"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
KUBERNETES_VERSION="1.20.9"

# +optional
# the user who can access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

master2 节点加入 control-plane

在 master2 上执行部署脚本

KUBECUBE_VERSION=v1.0
export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待 master2 加入 control-plane

master3 加入 control-plane 与此类似,仅需修改 LOCAL_IP10.173.32.6

# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# if install k8s
INSTALL_KUBERNETES="true"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="node-join-control-plane"

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="10.173.32.10" #{ip}:{port} , dns

# master ip means master node ip of cluster
MASTER_IP="10.173.32.4"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
KUBERNETES_VERSION="1.20.9"

# +optional
# the user who can access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

worker1 作为工作节点加入集群

在 worker1 上执行部署脚本

KUBECUBE_VERSION=v1.0
export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待 worker1 加入集群

worker2 和 worker3 加入集群的方式与之类似,仅需修改LOCAL_IP为本机 IP 即可

# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="false"

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# if install k8s
INSTALL_KUBERNETES="true"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="node-join-master"

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

# master ip means master node ip of cluster
MASTER_IP="10.173.32.4"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
KUBERNETES_VERSION="1.20.9"

# +optional
# the user who can access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

部署高可用 KubeCube

在 master1 上执行部署脚本

KUBECUBE_VERSION=v1.0
export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待 KubeCube 部署完成

  • install.conf
# if install kubecube on pivot cluster
INSTALL_KUBECUBE_PIVOT="true"

# if install kubecube on member cluster
INSTALL_KUBECUBE_MEMBER="false"

# if install k8s
INSTALL_KUBERNETES="false"

# there are four node mode below:
# "master" : node will be installed as a master of cluster
# "node-join-master" : node will be install as a worker of cluster to join master
# "control-plane-master" : node will be installed as a master to control plane of cluster
# "node-join-control-plane" : node will be installed as a master to join control plane
NODE_MODE="control-plane-master"

# +optional
# must be set when INSTALL_KUBECUBE_MEMBER="true"
# this value is the name of member cluster you
# want to take over
MEMBER_CLUSTER_NAME=""

# +optional
# must be set when NODE_MODE="control-plane-master"
# or "node-join-control-plane"
CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns

# master ip means master node ip of cluster
MASTER_IP="10.173.32.4"

# +optional
# KUBECUBE_HOST must be set when as a member cluster to
# join pivot cluster, the value is pivot node ip
KUBECUBE_HOST=""

# zone has two choice
# 1. "cn" : in mainland
# 2. "others" : out of mainland
ZONE="cn"

# k8s version you want to install
KUBERNETES_VERSION="1.20.9"

# +optional
# the user who can access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_USER="root"

# +optional
# the port specified to access master node, it can be empty
# when NODE_MODE="master" or "control-plane-master"
SSH_PORT=22

# +optional
# must be empty when ACCESS_PRIVATE_KEY_PATH set
# password for master user to access master node
ACCESS_PASSWORD=""

# +optional
# must be empty when ACCESS_PASSWORD set
# ACCESS_PRIVATE_KEY for master user to access master node
ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"
  • cube.conf

kubecube_replicas设置为3,使得 KubeCube 使用 3 副本部署,并且由于podAntiAffinity,它们会运行在非controlPlane的节点上,并且每个节点仅运行单个副本

# custom values for kubecube

kubecube_replicas=3
kubecube_args_logLevel="info"

7 - 热插拔

热插拔是通过修改配置文件实现不停机更新监控、日志、审计等组件的启停和配置。

热插拔

热插拔的实现是基于 helm ,因此集群需要预先安装好 helm3 版本。

登录管控 k8s 集群,执行命令可以查看热插拔配置

# kubectl get hotplug
NAME            PHASE     AGE
common          running   23h
pivot-cluster   running   16d

其中:

  • common 表示公共的热插拔配置,pivot-cluster 表示 pivot-cluster 这个 k8s 集群的热插拔配置。

  • 允许自定义各个集群的热插拔配置覆盖 common 热插拔配置实现个性化配置 k8s 集群组件热插拔。

  • 热插拔配置的 .metadata.name 要求与k8s集群名称一致,所有 k8s 集群信息查看命令为 kubectl get cluster

Common配置说明

目前,默认的 common 配置包括以下几个组件:

audit:操作审计

logseer:日志管理组件,仅在管控集群安装

logagent:日志采集代理组件

kubecube-monitoring:监控 prometheuse 组件

kubecube-thanos:监控 thanos 组件,仅在管控集群安装

示例如下:

apiVersion: hotplug.kubecube.io/v1
kind: Hotplug
metadata:
  annotations:
    kubecube.io/sync: "true"					## 同步信号,kubecube会将这个配置同步到各个集群
  name: common                                  ## 公共配置为cmmon,其余集群特殊配置为集群名字
spec:
  component:
  - name: audit                                 ## 审计日志
    status: disabled
  - name: logseer                               ## 日志管理组件
    namespace: logseer
    pkgName: logseer-v1.0.0.tgz
    status: disabled                            ## 启停标识:这里disabled为禁用
  - name: logagent                              ## 日志采集代理组件
    namespace: logagent
    pkgName: logagent-v1.0.0.tgz
    status: enabled                             ## 启停标识:这里enabled为启用
    env: |                                      ## 环境变量
      clustername: "{{.cluster}}"               ## {{.cluster}} 程序会自动注入集群名字替换
  - name: kubecube-monitoring                   ## 监控组件
    namespace: kubecube-monitoring
    pkgName: kubecube-monitoring-15.4.7.tgz
    status: enabled
    env: |
      grafana:
        enabled: false
      prometheus:
        prometheusSpec:
          externalLabels:
            cluster: "{{.cluster}}"
          remoteWrite:
          - url: http://10.173.32.42:31291/api/v1/receive      
  - name: kubecube-thanos                       ## 监控thanos组件
    namespace: kubecube-monitoring
    pkgName: thanos-3.18.0.tgz
    status: disabled
status:                                         ## message显示各个组件运行状态,phase显示总体运行状态
  message: '{"kubecube-monitoring":"release is running","kubecube-thanos":"release    
    is running","logagent":"release is running","logseer":"release is running"}'
  phase: running

每一个组件基本包含5个要素,在 spec.component 下:

name:组件名称

namespace:指定组件部署的命名空间,若指定的命名空间不存在,会自动以该字段值去创建一个命名空间。

pkgName:安装包名称,安装包默认存放路径为 warden 容器里的 /root/helmchartpkg,以 emptydir 形式存在。

status:组件是否启用

env:环境变量配置

每一个要素都可以使用集群独特配置进行覆盖,未覆盖的要素则依然使用 common 里的配置。

管控集群配置说明

Pivot-cluster 配置是管控集群配置,即 KubeCube 所在的集群。集群独特的配置会与 common 的配置结合,用于个性化配置集群组件,结合时遇到相同字段pivot-cluster 优先。

apiVersion: hotplug.kubecube.io/v1
kind: Hotplug
metadata:
  annotations:
    kubecube.io/sync: "true"            ## 同步信号,kubecube会将这个配置同步到其他集群
  name: pivot-cluster                   ## 与集群名字一致,指明这是pivot-cluster这个集群的热插拔配置
spec:
  component:
  - name: logseer                       ## 日志管理组件
    status: enabled                     ## 结合common设置为disabled,这里设置为enabled,标识其余集群不启用,pivot-cluster集群启用
  - name: kubecube-monitoring
    env: |
      grafana:
        enabled: true
      prometheus:
        prometheusSpec:
          externalLabels:
            cluster: "{{.cluster}}"
          remoteWrite:
          - url: http://thanos-receive:19291/api/v1/receive      
  - name: kubecube-thanos
    status: enabled
    env: |
      receive:
        replicaCount: 1
        replicationFactor: 1      
status:
  message: '{"kubecube-monitoring":"release is running","kubecube-thanos":"release
    is running","logagent":"release is running","logseer":"release is running"}'
  phase: running

8 - KubeCube 卸载

KubeCube 提供一键卸载脚本

⚠️ 注意,卸载脚本仅用于使用 KubeCube 部署脚本安装的环境,不应该用于任何正式环境

下载卸载脚本

curl -o cleanup.sh https://kubecube.nos-eastchina1.126.net/hack/cleanup.sh

一键卸载

一键卸载按顺序卸载 kubecube, kubernetes 和 docker

/bin/bash cleanup.sh all

指定卸载

卸载脚本可以指定卸载对象,但需注意卸载依赖

卸载 KubeCube

删除 kubecube chart release,并清理 kubecube 在本机上的相关文件

/bin/bash cleanup.sh kubecube

卸载 Kubernetes

卸载 kubernetes 集群,并清理 kubernetes 在本机上的相关文件

/bin/bash cleanup.sh k8s

卸载 Docker

卸载 docker,并清理 docker 在本机上的相关文件

⚠️注意:需在卸载 kubernetes 后才能卸载 docker,不然会造成 kubernetes 集群异常

/bin/bash cleanup.sh docker

9 - 已有系统接入

9.1 - Prometheus

KubeCube 在部署时会自动安装 Prometheus 等监控组件,以实现监控功能。本文档介绍了如何在 KubeCube 中接入用户已有的 Prometheus。

准备工作

  1. 在集群部署好 Prometheus,Prometheus 可以正常监控到集群资源的数据;

  2. 在集群中部署好 KubeCube;

    说明:部署好 KubeCube 后,由于集群内已有 Prometheus operator,多个 operator 会导致集群内 Prometheus 相关功能不可用,需要卸载 KubeCube 监控组件或删除本地 operator。

  3. 以平台管理员角色登录 KubeCube 管控集群。

步骤

1、添加 Label

由于 KubeCube 需要实现多集群监控,因此在 KubeCube 查询监控数据时,都会在 query 表达式中添加 cluster={clusterName} 来进行集群过滤。用户需要在 Prometheus 的 exporter 中添加这一 label,前端查询监控数据时才能查询到结果。

2、卸载 KubeCube 监控组件

方式一 页面操作:

  1. 点击页面右上角【切换到控制台】,点击任意空间,进入到控制台页面;

  2. 在左侧菜单栏点击【自定义资源CRD】,进入到集群级别 CRD 列表,可以点击右上方输入 “hotplug” 进行搜索,找到 “hotplugs.hotplug.kubecube.io” CRD,点击【v1】版本进入 CRD 详情页;

  3. 选择 common 实例,点击【设置YAML】,找到 spec.component. name=kubecube-monitoring,将 “status” 改成 “disabled”,即卸载 KubeCube 自带监控组件。

方式二 命令行操作:

  1. kubectl edit hotplug common
  2. 找到 spec.component. name=kubecube-monitoring,将 “status” 改成 “disabled”。

详细配置说明见 热插拔

3、部署 ServiceMonitor

查看集群资源

查看控制台内监控数据,需要部署两个 ServiceMonitor:kubelet 和 kube-state-metrics 的 ServiceMonitor,样例如下:

  • 部署 kubecube-monitoring-kubelet

    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: kubecube-monitoring-kubelet
      namespace: kubecube-monitoring
    spec:
      endpoints:
      - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
        honorLabels: true
        port: https-metrics
        relabelings:
        - sourceLabels:
          - __metrics_path__
          targetLabel: metrics_path
        scheme: https
        tlsConfig:
          caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecureSkipVerify: true
      - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
        honorLabels: true
        path: /metrics/cadvisor
        port: https-metrics
        relabelings:
        - sourceLabels:
          - __metrics_path__
          targetLabel: metrics_path
        scheme: https
        tlsConfig:
          caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecureSkipVerify: true
      - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
        honorLabels: true
        path: /metrics/probes
        port: https-metrics
        relabelings:
        - sourceLabels:
          - __metrics_path__
          targetLabel: metrics_path
        scheme: https
        tlsConfig:
          caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecureSkipVerify: true
      jobLabel: k8s-app
      namespaceSelector:
        matchNames:
        - kube-system
      selector:
        matchLabels:
          k8s-app: kubelet
    
  • 部署 kubecube-monitoring-kube-state-metrics

    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: kubecube-monitoring-kube-state-metrics
      namespace: kubecube-monitoring
    spec:
      endpoints:
      - honorLabels: true
        port: http
      selector:
        matchLabels:
          app.kubernetes.io/instance: kubecube-monitoring
          app.kubernetes.io/name: kube-state-metrics
    

查看组件监控

当前 KubeCube 平台支持对组件的监控视图可视化查询,详细说明见 平台组件监控 。接入外部监控后,用户可按需在集群内部署对应组件的 ServiceMonitor。

各个 ServiceMonitor 的 yaml 可参考 https://github.com/kubecube-io/charts/tree/main/kubecube-monitoring/templates/exporters

4、部署 Dashboard

查看集群资源(控制台内监控数据),需要部署:

查看组件监控,可以部署对应的 Dashboard:

5、修改 Nginx 配置

  1. 使用命令行:kubectl edit configmap nginx-config -n kubecube-system

  2. 找到原有的地址配置,修改为自有 Prometheus 地址

    upstream monitoring {
        server kubecube-thanos-query.kubecube-monitoring:9090;
    }
    

    即 将kubecube-thanos-query.kubecube-monitoring:9090 替换为外部地址。

  3. 重启 pod:kubectl delete pod frontend-xxxxxx-xxxxx -n kubecube-system

9.2 - ElasticSearch

KubeCube 提供了日志服务和操作审计服务,默认关闭。用户在开启后,日志服务和操作审计服务均会将日志发送到 ElasticSearch 进行存储,由 ElasticSearch 对日志进行管理。用户可以在 热插拔 中修改配置,安装内部 ElasticSearch,也可以配置外部的 ElasticSearch 地址,对接已有的 ElasticSearch。下面分别介绍如何在这两个功能中接入外部 ElasticSearch。

日志

方式一 页面操作:

  1. 点击页面右上角【切换到控制台】,点击任意空间,进入到控制台页面;

  2. 在左侧菜单栏点击【自定义资源CRD】,进入到集群级别 CRD 列表,可以点击右上方输入 “hotplug” 进行搜索,找到 “hotplugs.hotplug.kubecube.io” CRD,点击【v1】版本进入 CRD 详情页;

  3. 选择 common 实例,点击【设置YAML】,找到 spec.component. name=logseer,添加环境变量,如:

- name: logseer
  namespace: logseer
  pkgName: logseer-v1.0.0.tgz
  status: disabled
  env: |
    address: elasticsearch-master.elasticsearch.svc

方式二 命令行操作:

  1. kubectl edit hotplug pivot-cluster
  2. 找到 spec.component. name=logseer,添加环境变量,同上。

详细配置说明见 热插拔

操作审计

  1. kubectl edit deploy audit -n kubecube-system

  2. 添加环境变量:AUDIT_WEBHOOK_HOST、AUDIT_WEBHOOK_INDEX、AUDIT_WEBHOOK_TYPE,如

    env:
    - name: AUDIT_WEBHOOK_HOST
      value: http://elasticsearch-master.elasticsearch:9200
    - name: AUDIT_WEBHOOK_INDEX
      value: audit
    - name: AUDIT_WEBHOOK_TYPE
      value: logs
    

注:如果同时配置了内部和外部 ElasticSearch,审计日志将优先发到外部 ElasticSearch。

其他详细说明见:操作审计