diff --git a/docs/guide/index.md b/docs/guide/index.md index 22a6519..45bfa1e 100644 --- a/docs/guide/index.md +++ b/docs/guide/index.md @@ -6,7 +6,7 @@ - 安装 [dashboard](dashboard.md) - 安装 [metrics-server](metrics-server.md) - 安装 [prometheus](prometheus.md) -- 安装 [heapster](heapster.md) DEPRECATED WARNNING +- 安装 [kubeapps](kubeapps.md) - 安装 [ingress](ingress.md) - 安装 [helm](helm.md) - 安装 [efk](efk.md) diff --git a/docs/guide/kubeapps.md b/docs/guide/kubeapps.md new file mode 100644 index 0000000..42a90fd --- /dev/null +++ b/docs/guide/kubeapps.md @@ -0,0 +1,67 @@ +# 使用kubeapps管理集群应用 + +Kubeapps 是一个基于 Web 的应用程序，它可以在 Kubernetes 集群上进行一站式安装，并使用户能够部署、管理和升级应用程序。 + +kubeapps_dashboard + +项目地址：https://github.com/vmware-tanzu/kubeapps +部署项目地址：https://github.com/bitnami/charts/tree/main/bitnami/kubeapps + +## 使用kubeasz部署 + +- 1.编辑集群配置文件：clusters/${集群名}/config.yml + +``` bash +kubeapps_install: "yes" # 启用安装 +kubeapps_install_namespace: "kubeapps" # 设置安装命名空间 +kubeapps_working_namespace: "default" # 设置默认应用命名空间 +kubeapps_storage_class: "local-path" # 设置存储storageclass，默认使用local-path-provisioner +kubeapps_chart_ver: "12.4.3" +``` + +- 2.下载相关容器镜像 + +``` bash +# 下载kubeapps镜像 +/etc/kubeasz/ezdown -X kubeapps + +# 下载local-path-provisioner镜像 +/etc/kubeasz/ezdown -X local-path-provisioner +``` + +- 3.安装cluster-addon + +``` bash +$ dk ezctl setup ${集群名} 07 + +# 执行成功后验证 +$ kubectl get pod --all-namespaces |grep kubeapps +``` + +## 验证使用kubeapps + +阅读文档：https://github.com/vmware-tanzu/kubeapps/blob/main/site/content/docs/latest/tutorials/getting-started.md + +正式使用建议配置OAuth2/OIDC用户认证，这里仅验证使用k8s ServiceAccount 方式登陆，项目已预装三个用户权限： + +- 1.kubeapps-admin-token，全局cluster-admin权限，不建议使用 +- 2.kubeapps-edit-token，某个命名空间下的应用可写权限 +- 3.kubeapps-view-token，某个命名空间下的应用只读权限 + +``` bash +# 获取UI访问地址，默认使用NodePort +kubectl get svc -n kubeapps kubeapps +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +kubeapps NodePort 10.68.92.88 80:32490/TCP 117m + +# 获取admin token +kubectl get secrets -n kube-system kubeapps-admin-token -o go-template='{{.data.token | base64decode}}' + +# 获取某命名空间应用部署权限 token +kubectl get secrets -n default kubeapps-edit-token -o go-template='{{.data.token | base64decode}}' + +# 获取某命名空间应用部署权限 token +kubectl get secrets -n default kubeapps-view-token -o go-template='{{.data.token | base64decode}}' +``` + +打开浏览器，访问http://${Node_IP}:32490，输入上面合适权限的token即可。 diff --git a/docs/setup/08-cluster-storage.md b/docs/setup/08-cluster-storage.md index fd87452..54950a5 100644 --- a/docs/setup/08-cluster-storage.md +++ b/docs/setup/08-cluster-storage.md @@ -7,12 +7,16 @@ 如上面所说PV和PVC都只是抽象的概念，在k8s中是通过插件的方式提供具体的存储实现。目前包含有NFS、iSCSI和云提供商指定的存储系统，更多的存储实现[参考官方文档](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes)。 -这里PV又有两种提供方式: 静态或者动态。 -本篇以介绍 **NFS存储** 为例，讲解k8s 众多存储方案中的一个实现。 +以下介绍两种`provisioner`, 可以提供静态或者动态的PV + +- [nfs-provisioner](https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner): NFS存储目录供应者 +- [local-path-provisioner](https://github.com/rancher/local-path-provisioner): 本地存储目录供应者 + +## NFS存储目录供应者 -## 静态 PV 首先我们需要一个NFS服务器，用于提供底层存储。通过文档[nfs-server](../guide/nfs-server.md)，我们可以创建一个NFS服务器。 +### 静态 PV - 创建静态 pv，指定容量，访问模式，回收策略，存储类等 ``` bash @@ -36,7 +40,7 @@ spec: ``` - 创建 pvc即可绑定使用上述 pv了，具体请看后文 test pod例子 -## 创建动态PV +### 创建动态PV 在一个工作k8s 集群中，`PVC`请求会很多，如果每次都需要管理员手动去创建对应的 `PV`资源，那就很不方便；因此 K8S还提供了多种 `provisioner`来动态创建 `PV`，不仅节省了管理员的时间，还可以根据`StorageClasses`封装不同类型的存储供 PVC 选用。 @@ -59,14 +63,14 @@ nfs_path: "/data/nfs" # 修改为实际的nfs共享目录 - 2.创建 nfs provisioner ``` bash -$ ezctl setup ${集群名} 07 +$ dk ezctl setup ${集群名} 07 # 执行成功后验证 $ kubectl get pod --all-namespaces |grep nfs-client kube-system nfs-client-provisioner-84ff87c669-ksw95 1/1 Running 0 21m ``` -## 验证使用动态 PV +- 3.验证使用动态 PV 在目录clusters/${集群名}/yml/nfs-provisioner/ 有个测试例子 @@ -98,5 +102,30 @@ test-claim Bound pvc-44d34a50-e00b-4f6c-8005-40f5cc54af18 2Mi RWX ``` 如上，可以发现挂载的时候，nfs-client根据PVC自动创建了一个目录，我们Pod中挂载的`/mnt`，实际引用的就是该目录，而我们在`/mnt`下创建的`SUCCESS`文件，也自动写入到了这里。 -# 后续 后面当我们需要为上层应用提供持久化存储时，只需要提供`StorageClass`即可。很多应用都会根据`StorageClass`来创建他们的所需的PVC, 最后再把PVC挂载到他们的Deployment或StatefulSet中使用，比如：efk、jenkins等 + +## 本地存储目录供应者 + +当应用对于磁盘I/O性能要求高，比较适合本地文件目录存储，特别地可以本地挂载SSD磁盘（注意本地磁盘需要配置raid冗余策略）。Local Path Provisioner 可以方便地在k8s集群中使用本地文件目录存储。 + +在kubeasz项目中集成安装 + +- 1.编辑集群配置文件：clusters/${集群名}/config.yml + +``` bash +... 省略 +local_path_provisioner_install: "yes" # 修改为yes +# 设置默认本地存储路径 +local_path_provisioner_dir: "/opt/local-path-provisioner" +``` + +- 2.创建 local path provisioner + +``` bash +$ dk ezctl setup ${集群名} 07 + +# 执行成功后验证 +$ kubectl get pod --all-namespaces |grep nfs-client-provisioner +``` + +- 3.验证使用（略） diff --git a/example/config.yml b/example/config.yml index 21a24c8..fac2c2f 100644 --- a/example/config.yml +++ b/example/config.yml @@ -187,6 +187,13 @@ prom_install: "no" prom_namespace: "monitor" prom_chart_ver: "__prom_chart__" +# kubeapps 自动安装，如果选择安装，默认同时安装local-storage（提供storageClass: "local-path"） +kubeapps_install: "no" +kubeapps_install_namespace: "kubeapps" +kubeapps_working_namespace: "default" +kubeapps_storage_class: "local-path" +kubeapps_chart_ver: "__kubeapps_chart__" + # local-storage (local-path-provisioner) 自动安装 local_path_provisioner_install: "no" local_path_provisioner_ver: "__local_path_provisioner__" diff --git a/ezctl b/ezctl index 32d892a..6393b86 100755 --- a/ezctl +++ b/ezctl @@ -162,6 +162,7 @@ function new() { nfsProvisionerVer=$(grep 'nfsProvisionerVer=' ezdown|cut -d'=' -f2) pauseVer=$(grep 'pauseVer=' ezdown|cut -d'=' -f2) promChartVer=$(grep 'promChartVer=' ezdown|cut -d'=' -f2) + kubeappsVer=$(grep 'kubeappsVer=' ezdown|cut -d'=' -f2) harborVer=$(grep 'HARBOR_VER=' ezdown|cut -d'=' -f2) registryMirror=true @@ -181,6 +182,7 @@ function new() { -e "s/__local_path_provisioner__/$localpathProvisionerVer/g" \ -e "s/__nfs_provisioner__/$nfsProvisionerVer/g" \ -e "s/__prom_chart__/$promChartVer/g" \ + -e "s/__kubeapps_chart__/$kubeappsVer/g" \ -e "s/__harbor__/$harborVer/g" \ -e "s/^ENABLE_MIRROR_REGISTRY.*$/ENABLE_MIRROR_REGISTRY: $registryMirror/g" \ -e "s/__metrics__/$metricsVer/g" "clusters/$1/config.yml" diff --git a/ezdown b/ezdown index 78e7b6d..9a3f1e3 100755 --- a/ezdown +++ b/ezdown @@ -38,6 +38,7 @@ kubeOvnVer=v1.11.5 localpathProvisionerVer=v0.0.24 nfsProvisionerVer=v4.0.2 promChartVer=45.23.0 +kubeappsVer=12.4.3 function usage() { echo -e "\033[33mUsage:\033[0m ezdown [options] [args]" @@ -90,6 +91,7 @@ available options: flannel to download images of flannel kube-ovn to download images of kube-ovn kube-router to download images of kube-router + kubeapps to download images of kubeapps local-path-provisioner to download images of local-path-provisioner network-check to download images of network-check nfs-provisioner to download images of nfs-provisioner @@ -469,6 +471,39 @@ function get_extra_images() { docker push "easzlab.io.local:5000/flannel/flannel-cni-plugin:v1.1.2" ;; + # kubeapps images + kubeapps) + if [[ ! -f "$imageDir/kubeapps_$kubeappsVer.tar" ]];then + docker pull "bitnami/kubeapps-apis:2.7.0-debian-11-r10" && \ + docker pull "bitnami/kubeapps-apprepository-controller:2.7.0-scratch-r0" && \ + docker pull "bitnami/kubeapps-asset-syncer:2.7.0-scratch-r0" && \ + docker pull "bitnami/kubeapps-dashboard:2.7.0-debian-11-r12" && \ + docker pull "bitnami/nginx:1.23.4-debian-11-r18" && \ + docker pull "bitnami/postgresql:15.3.0-debian-11-r0" && \ + docker save -o "$imageDir/kubeapps_$kubeappsVer.tar" \ + "bitnami/kubeapps-apis:2.7.0-debian-11-r10" \ + "bitnami/kubeapps-apprepository-controller:2.7.0-scratch-r0" \ + "bitnami/kubeapps-asset-syncer:2.7.0-scratch-r0" \ + "bitnami/kubeapps-dashboard:2.7.0-debian-11-r12" \ + "bitnami/nginx:1.23.4-debian-11-r18" \ + "bitnami/postgresql:15.3.0-debian-11-r0" + else + docker load -i "$imageDir/kubeapps_$kubeappsVer.tar" + fi + docker tag "bitnami/kubeapps-apis:2.7.0-debian-11-r10" "easzlab.io.local:5000/bitnami/kubeapps-apis:2.7.0-debian-11-r10" + docker tag "bitnami/kubeapps-apprepository-controller:2.7.0-scratch-r0" "easzlab.io.local:5000/bitnami/kubeapps-apprepository-controller:2.7.0-scratch-r0" + docker tag "bitnami/kubeapps-asset-syncer:2.7.0-scratch-r0" "easzlab.io.local:5000/bitnami/kubeapps-asset-syncer:2.7.0-scratch-r0" + docker tag "bitnami/kubeapps-dashboard:2.7.0-debian-11-r12" "easzlab.io.local:5000/bitnami/kubeapps-dashboard:2.7.0-debian-11-r12" + docker tag "bitnami/nginx:1.23.4-debian-11-r18" "easzlab.io.local:5000/bitnami/nginx:1.23.4-debian-11-r18" + docker tag "bitnami/postgresql:15.3.0-debian-11-r0" "easzlab.io.local:5000/bitnami/postgresql:15.3.0-debian-11-r0" + docker push "easzlab.io.local:5000/bitnami/kubeapps-apis:2.7.0-debian-11-r10" + docker push "easzlab.io.local:5000/bitnami/kubeapps-apprepository-controller:2.7.0-scratch-r0" + docker push "easzlab.io.local:5000/bitnami/kubeapps-asset-syncer:2.7.0-scratch-r0" + docker push "easzlab.io.local:5000/bitnami/kubeapps-dashboard:2.7.0-debian-11-r12" + docker push "easzlab.io.local:5000/bitnami/nginx:1.23.4-debian-11-r18" + docker push "easzlab.io.local:5000/bitnami/postgresql:15.3.0-debian-11-r0" + ;; + # kube-ovn images kube-ovn) if [[ ! -f "$imageDir/kube-ovn_$kubeOvnVer.tar" ]];then diff --git a/roles/cluster-addon/files/kubeapps-12.4.3.tgz b/roles/cluster-addon/files/kubeapps-12.4.3.tgz new file mode 100644 index 0000000..4609910 Binary files /dev/null and b/roles/cluster-addon/files/kubeapps-12.4.3.tgz differ diff --git a/roles/cluster-addon/tasks/kubeapps.yml b/roles/cluster-addon/tasks/kubeapps.yml new file mode 100644 index 0000000..f41f3c5 --- /dev/null +++ b/roles/cluster-addon/tasks/kubeapps.yml @@ -0,0 +1,24 @@ +# https://github.com/bitnami/charts/tree/main/bitnami/kubeapps + +- block: + - name: prepare some dirs + file: name={{ cluster_dir }}/yml/kubeapps/token state=directory + + - name: 创建 kubeapps chart 个性化设置 + template: src=kubeapps/values.yaml.j2 dest={{ cluster_dir }}/yml/kubeapps/values.yaml + + - name: 准备临时用户tokens + template: src=kubeapps/{{ item }}.j2 dest={{ cluster_dir }}/yml/kubeapps/token/{{ item }} + with_items: + - "kubeapps-admin-token.yaml" + - "single-namespace-edit-token.yaml" + - "single-namespace-view-token.yaml" + + - name: helm 创建 kubeapps + shell: "{{ base_dir }}/bin/helm upgrade kubeapps --install --create-namespace \ + -n {{ kubeapps_install_namespace }} -f {{ cluster_dir }}/yml/kubeapps/values.yaml \ + {{ base_dir }}/roles/cluster-addon/files/kubeapps-{{ kubeapps_chart_ver }}.tgz" + + - name: 创建临时用户tokens + shell: "{{ base_dir }}/bin/kubectl apply -f {{ cluster_dir }}/yml/kubeapps/token/" + when: 'kubeapps_install == "yes"' diff --git a/roles/cluster-addon/tasks/local-storage.yml b/roles/cluster-addon/tasks/local-storage.yml index 7d3e059..4e554bf 100644 --- a/roles/cluster-addon/tasks/local-storage.yml +++ b/roles/cluster-addon/tasks/local-storage.yml @@ -10,4 +10,4 @@ - name: 创建 local-storage部署 shell: "{{ base_dir }}/bin/kubectl apply -f {{ cluster_dir }}/yml/local-storage/local-path-storage.yaml" - when: 'local_path_provisioner_install == "yes"' + when: 'local_path_provisioner_install == "yes" or (kubeapps_install == "yes" and kubeapps_storage_class == "local-path")' diff --git a/roles/cluster-addon/tasks/main.yml b/roles/cluster-addon/tasks/main.yml index 1d1aff2..1078969 100644 --- a/roles/cluster-addon/tasks/main.yml +++ b/roles/cluster-addon/tasks/main.yml @@ -29,3 +29,6 @@ - import_tasks: network_check.yml when: 'network_check_enabled|bool and CLUSTER_NETWORK != "cilium"' + +- import_tasks: kubeapps.yml + when: 'kubeapps_install == "yes"' diff --git a/roles/cluster-addon/templates/kubeapps/kubeapps-admin-token.yaml.j2 b/roles/cluster-addon/templates/kubeapps/kubeapps-admin-token.yaml.j2 new file mode 100644 index 0000000..bb7cb16 --- /dev/null +++ b/roles/cluster-addon/templates/kubeapps/kubeapps-admin-token.yaml.j2 @@ -0,0 +1,30 @@ +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: kubeapps-operator + namespace: kube-system + +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: kubeapps-operator +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: cluster-admin +subjects: +- kind: ServiceAccount + name: kubeapps-operator + namespace: kube-system + +--- +apiVersion: v1 +kind: Secret +metadata: + name: kubeapps-admin-token + namespace: kube-system + annotations: + kubernetes.io/service-account.name: "kubeapps-operator" +type: kubernetes.io/service-account-token diff --git a/roles/cluster-addon/templates/kubeapps/single-namespace-edit-token.yaml.j2 b/roles/cluster-addon/templates/kubeapps/single-namespace-edit-token.yaml.j2 new file mode 100644 index 0000000..ef35313 --- /dev/null +++ b/roles/cluster-addon/templates/kubeapps/single-namespace-edit-token.yaml.j2 @@ -0,0 +1,31 @@ +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: kubeapps-editor + namespace: {{ kubeapps_working_namespace }} + +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: kubeapps-editor + namespace: {{ kubeapps_working_namespace }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: edit +subjects: +- kind: ServiceAccount + name: kubeapps-editor + namespace: {{ kubeapps_working_namespace }} + +--- +apiVersion: v1 +kind: Secret +metadata: + name: kubeapps-edit-token + namespace: {{ kubeapps_working_namespace }} + annotations: + kubernetes.io/service-account.name: "kubeapps-editor" +type: kubernetes.io/service-account-token diff --git a/roles/cluster-addon/templates/kubeapps/single-namespace-view-token.yaml.j2 b/roles/cluster-addon/templates/kubeapps/single-namespace-view-token.yaml.j2 new file mode 100644 index 0000000..679de7c --- /dev/null +++ b/roles/cluster-addon/templates/kubeapps/single-namespace-view-token.yaml.j2 @@ -0,0 +1,31 @@ +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: kubeapps-viewer + namespace: {{ kubeapps_working_namespace }} + +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: kubeapps-viewer + namespace: {{ kubeapps_working_namespace }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: view +subjects: +- kind: ServiceAccount + name: kubeapps-viewer + namespace: {{ kubeapps_working_namespace }} + +--- +apiVersion: v1 +kind: Secret +metadata: + name: kubeapps-view-token + namespace: {{ kubeapps_working_namespace }} + annotations: + kubernetes.io/service-account.name: "kubeapps-viewer" +type: kubernetes.io/service-account-token diff --git a/roles/cluster-addon/templates/kubeapps/values.yaml.j2 b/roles/cluster-addon/templates/kubeapps/values.yaml.j2 new file mode 100644 index 0000000..e07e11d --- /dev/null +++ b/roles/cluster-addon/templates/kubeapps/values.yaml.j2 @@ -0,0 +1,75 @@ +global: + imageRegistry: "easzlab.io.local:5000" + # default to use "local-path-provisioner" + storageClass: "{{ kubeapps_storage_class }}" + +## @section Kubeapps packaging options +packaging: + helm: + enabled: true + carvel: + enabled: false + flux: + enabled: false + +## @section Frontend parameters +frontend: + image: + repository: bitnami/nginx + tag: 1.23.4-debian-11-r18 + replicaCount: 1 + service: + type: NodePort + +## @section Dashboard parameters +dashboard: + enabled: true + image: + repository: bitnami/kubeapps-dashboard + tag: 2.7.0-debian-11-r12 + replicaCount: 1 + +## @section AppRepository Controller parameters +apprepository: + image: + repository: bitnami/kubeapps-apprepository-controller + tag: 2.7.0-scratch-r0 + syncImage: + repository: bitnami/kubeapps-asset-syncer + tag: 2.7.0-scratch-r0 + initialRepos: + - name: bitnami + url: https://charts.bitnami.com/bitnami + ## @param apprepository.crontab Default schedule for syncing App repositories + crontab: "*/10 * * * *" + watchAllNamespaces: true + replicaCount: 1 + +## Auth Proxy configuration for OIDC support +authProxy: + enabled: false + +## @section Other Parameters +clusters: + - name: default + domain: cluster.local + +## @section Database Parameters +postgresql: + enabled: true + auth: + username: "postgres" + postgresPassword: "Postgres1234!" + database: assets + existingSecret: "" + primary: + persistence: + enabled: true + architecture: standalone + +## @section kubeappsapis parameters +kubeappsapis: + image: + repository: bitnami/kubeapps-apis + tag: 2.7.0-debian-11-r10 + replicaCount: 1