kubernetes-handbook/practice/ceph-helm-install-guide-zh.md

323 lines
12 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters!

This file contains ambiguous Unicode characters that may be confused with others in your current locale. If your use case is intentional and legitimate, you can safely ignore this warning. Use the Escape button to highlight these characters.

# 用Helm托管安装Ceph集群并提供后端存储
## 安装
[ceph-helm ](https://github.com/ceph/ceph-helm/)项目可让你在Kubernetes 环境以托管方式部署Ceph . 本文档假定Kubernetes 环境已经可用。
## 当前的限制
* Public网络和Cluster网络必须是同一个网络
* 如果 storage class 用户标识不是admin, 则必须在Ceph集群中手动创建用户并在Kubernetes中创建其secret
* ceph-mgr只能运行1个replica
## 安装并使用Helm
请访问Helm官网的说明安装Helm。
Helm通过从本地读取Kubernetes配置文件来查找Kubernetes集群; 确保文件已下载和且helm客户端可以访问。
Kubernetes群集必须配置并运行Tiller服务器并且须将本地Helm客户端网络可达。要在本地运行Tiller并将Helm连接到它请运行如下命令此命令会在Kubernetes集群部署一个tiller实例
```bash
$ helm init
```
ceph-helm项目默认使用本地的Helm repo来存储charts。要启动本地Helm repo服务器请运行
```bash
$ helm serve &
$ helm repo add local http://localhost:8879/charts
```
## 添加Ceph-Helm charts到本地repo
```bash
$ git clone https://github.com/ceph/ceph-helm
$ cd ceph-helm/ceph
$ make
```
## 配置Ceph集群
创建一个包含Ceph配置的ceph-overrides.yaml文件。这个文件可能存在于任何地方本文档默认此文件在用户的home目录中。
```bash
$ cat ~/ceph-overrides.yaml
```
```yaml
network:
public: 172.21.0.0/20
cluster: 172.21.0.0/20
osd_devices:
- name: dev-sdd
device: /dev/sdd
zap: "1"
- name: dev-sde
device: /dev/sde
zap: "1"
storageclass:
name: ceph-rbd
pool: rbd
user_id: k8s
```
**注意** 如果未设置日志journal设备它将与device设备同位置。另ceph-helm/ceph/ceph/values.yaml文件包含所有可配置的选项。
## 创建Ceph 集群的namespace
默认情况下ceph-helm组件在Kubernetes的ceph namespace中运行。如果要自定义请自定义namespace的名称默认namespace请运行
```bash
$ kubectl create namespace ceph
```
## 配置RBAC权限
Kubernetes> = v1.6使RBAC成为默认的admission controller。ceph-helm要为每个组件提供RBAC角色和权限
```bash
$ kubectl create -f ~/ceph-helm/ceph/rbac.yaml
```
rbac.yaml文件假定Ceph集群将部署在ceph命名空间中。
## 给Kubelet节点打标签
需要设置以下标签才能部署Ceph集群
```
ceph-mon=enabled
ceph-mgr=enabled
ceph-osd=enabled
ceph-osd-device-<name>=enabled
```
ceph-osd-device-标签是基于我们的ceph-overrides.yaml中定义的osd_devices名称值创建的。从我们下面的例子中我们将得到以下两个标签ceph-osd-device-dev-sdb和ceph-osd-device-dev-sdc。
每个 Ceph Monitor节点:
```bash
$ kubectl label node <nodename> ceph-mon=enabled ceph-mgr=enabled
```
每个 OSD node节点:
```bash
$ kubectl label node <nodename> ceph-osd=enabled ceph-osd-device-dev-sdb=enabled ceph-osd-device-dev-sdc=enabled
```
## Ceph 部署
运行helm install命令来部署Ceph
```bash
$ helm install --name=ceph local/ceph --namespace=ceph -f ~/ceph-overrides.yaml
NAME: ceph
LAST DEPLOYED: Wed Oct 18 22:25:06 2017
NAMESPACE: ceph
STATUS: DEPLOYED
RESOURCES:
==> v1/Secret
NAME TYPE DATA AGE
ceph-keystone-user-rgw Opaque 7 1s
==> v1/ConfigMap
NAME DATA AGE
ceph-bin-clients 2 1s
ceph-bin 24 1s
ceph-etc 1 1s
ceph-templates 5 1s
==> v1/Service
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ceph-mon None <none> 6789/TCP 1s
ceph-rgw 10.101.219.239 <none> 8088/TCP 1s
==> v1beta1/DaemonSet
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE-SELECTOR AGE
ceph-mon 3 3 0 3 0 ceph-mon=enabled 1s
ceph-osd-dev-sde 3 3 0 3 0 ceph-osd-device-dev-sde=enabled,ceph-osd=enabled 1s
ceph-osd-dev-sdd 3 3 0 3 0 ceph-osd-device-dev-sdd=enabled,ceph-osd=enabled 1s
==> v1beta1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
ceph-mds 1 1 1 0 1s
ceph-mgr 1 1 1 0 1s
ceph-mon-check 1 1 1 0 1s
ceph-rbd-provisioner 2 2 2 0 1s
ceph-rgw 1 1 1 0 1s
==> v1/Job
NAME DESIRED SUCCESSFUL AGE
ceph-mgr-keyring-generator 1 0 1s
ceph-mds-keyring-generator 1 0 1s
ceph-osd-keyring-generator 1 0 1s
ceph-rgw-keyring-generator 1 0 1s
ceph-mon-keyring-generator 1 0 1s
ceph-namespace-client-key-generator 1 0 1s
ceph-storage-keys-generator 1 0 1s
==> v1/StorageClass
NAME TYPE
ceph-rbd ceph.com/rbd
```
helm install的输出显示了将要部署的不同类型的资源。
将使用ceph-rbd-provisioner Pod创建ceph.com/rbd类型的名为ceph-rbd的StorageClass。这允许创建PVC时自动提供RBD。第一次挂载时RBD设备将被格式化format。所有RBD设备都将使用ext4文件系统。ceph.com/rbd不支持fsType选项。默认情况下RBD将使用镜像格式2和镜像分层特性。可以在values文件中覆盖以下storageclass的默认值
```yaml
storageclass:
name: ceph-rbd
pool: rbd
user_id: k8s
user_secret_name: pvc-ceph-client-key
image_format: "2"
image_features: layering
```
使用下面的命令检查所有Pod是否正常运行。这可能需要几分钟时间
```bash
$ kubectl -n ceph get pods
NAME READY STATUS RESTARTS AGE
ceph-mds-3804776627-976z9 0/1 Pending 0 1m
ceph-mgr-3367933990-b368c 1/1 Running 0 1m
ceph-mon-check-1818208419-0vkb7 1/1 Running 0 1m
ceph-mon-cppdk 3/3 Running 0 1m
ceph-mon-t4stn 3/3 Running 0 1m
ceph-mon-vqzl0 3/3 Running 0 1m
ceph-osd-dev-sdd-6dphp 1/1 Running 0 1m
ceph-osd-dev-sdd-6w7ng 1/1 Running 0 1m
ceph-osd-dev-sdd-l80vv 1/1 Running 0 1m
ceph-osd-dev-sde-6dq6w 1/1 Running 0 1m
ceph-osd-dev-sde-kqt0r 1/1 Running 0 1m
ceph-osd-dev-sde-lp2pf 1/1 Running 0 1m
ceph-rbd-provisioner-2099367036-4prvt 1/1 Running 0 1m
ceph-rbd-provisioner-2099367036-h9kw7 1/1 Running 0 1m
ceph-rgw-3375847861-4wr74 0/1 Pending 0 1m
```
**注意** 因为我们没有用ceph-rgw = enabled或ceph-mds = enabled 给节点打标签ceph对象存储特性需要ceph-rgwcephfs特性需要ceph-mds因此MDS和RGW Pod都处于pending状态一旦其他Pod都在运行状态请用如下命令从某个MON节点检查Ceph的集群状态
```bash
$ kubectl -n ceph exec -ti ceph-mon-cppdk -c ceph-mon -- ceph -s
cluster:
id: e8f9da03-c2d2-4ad3-b807-2a13d0775504
health: HEALTH_OK
services:
mon: 3 daemons, quorum mira115,mira110,mira109
mgr: mira109(active)
osd: 6 osds: 6 up, 6 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 bytes
usage: 644 MB used, 5555 GB / 5556 GB avail
pgs:
```
## 配置一个POD以便从Ceph申请使用一个持久卷
为〜/ ceph-overwrite.yaml中定义的k8s用户创建一个密钥环并将其转换为base64
```bash
$ kubectl -n ceph exec -ti ceph-mon-cppdk -c ceph-mon -- bash
# ceph auth get-or-create-key client.k8s mon 'allow r' osd 'allow rwx pool=rbd' | base64
QVFCLzdPaFoxeUxCRVJBQUVEVGdHcE9YU3BYMVBSdURHUEU0T0E9PQo=
# exit
```
编辑ceph namespace中存在的用户secret
```bash
$ kubectl -n ceph edit secrets/pvc-ceph-client-key
```
将base64值复制到key位置的值并保存:
```yaml
apiVersion: v1
data:
key: QVFCLzdPaFoxeUxCRVJBQUVEVGdHcE9YU3BYMVBSdURHUEU0T0E9PQo=
kind: Secret
metadata:
creationTimestamp: 2017-10-19T17:34:04Z
name: pvc-ceph-client-key
namespace: ceph
resourceVersion: "8665522"
selfLink: /api/v1/namespaces/ceph/secrets/pvc-ceph-client-key
uid: b4085944-b4f3-11e7-add7-002590347682
type: kubernetes.io/rbd
```
我们创建一个在default namespace中使用RBD的Pod。将用户secret从ceph namespace复制到default namespace
```bash
$ kubectl -n ceph get secrets/pvc-ceph-client-key -o json | jq '.metadata.namespace = "default"' | kubectl create -f -
secret "pvc-ceph-client-key" created
$ kubectl get secrets
NAME TYPE DATA AGE
default-token-r43wl kubernetes.io/service-account-token 3 61d
pvc-ceph-client-key kubernetes.io/rbd 1 20s
```
创建并初始化RBD池
```bash
$ kubectl -n ceph exec -ti ceph-mon-cppdk -c ceph-mon -- ceph osd pool create rbd 256
pool 'rbd' created
$ kubectl -n ceph exec -ti ceph-mon-cppdk -c ceph-mon -- rbd pool init rbd
```
**重要** 重要的 Kubernetes使用RBD内核模块将RBD映射到主机。Luminous需要CRUSH_TUNABLES 5Jewel。这些可调参数的最小内核版本是4.5。如果您的内核不支持这些可调参数请运行ceph osd crush tunables hammer。
**重要** 由于RBD映射到主机系统上。主机需要能够解析由kube-dns服务管理的ceph-mon.ceph.svc.cluster.local名称。要获得kube-dns服务的IP地址运行kubectl -n kube-system get svc/kube-dns。
创建一个PVC
```bash
$ cat pvc-rbd.yaml
```
```yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: ceph-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
storageClassName: ceph-rbd
```
```bash
$ kubectl create -f pvc-rbd.yaml
persistentvolumeclaim "ceph-pvc" created
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE
ceph-pvc Bound pvc-1c2ada50-b456-11e7-add7-002590347682 20Gi RWO ceph-rbd 3s
```
检查集群上是否已创建RBD
```bash
$ kubectl -n ceph exec -ti ceph-mon-cppdk -c ceph-mon -- rbd ls
kubernetes-dynamic-pvc-1c2e9442-b456-11e7-9bd2-2a4159ce3915
$ kubectl -n ceph exec -ti ceph-mon-cppdk -c ceph-mon -- rbd info kubernetes-dynamic-pvc-1c2e9442-b456-11e7-9bd2-2a4159ce3915
rbd image 'kubernetes-dynamic-pvc-1c2e9442-b456-11e7-9bd2-2a4159ce3915':
size 20480 MB in 5120 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.10762ae8944a
format: 2
features: layering
flags:
create_timestamp: Wed Oct 18 22:45:59 2017
```
创建一个使用此PVC的Pod
```bash
$ cat pod-with-rbd.yaml
```
```yaml
kind: Pod
apiVersion: v1
metadata:
name: mypod
spec:
containers:
- name: busybox
image: busybox
command:
- sleep
- "3600"
volumeMounts:
- mountPath: "/mnt/rbd"
name: vol1
volumes:
- name: vol1
persistentVolumeClaim:
claimName: ceph-pvc
```
```bash
$ kubectl create -f pod-with-rbd.yaml
pod "mypod" created
```
检查Pod
```bash
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
mypod 1/1 Running 0 17s
$ kubectl exec mypod -- mount | grep rbd
/dev/rbd0 on /mnt/rbd type ext4 (rw,relatime,stripe=1024,data=ordered)
```
## 日志
可以通过kubectl logs [-f]命令访问OSD和Monitor日志。Monitors有多个日志记录流每个流都可以从ceph-mon Pod中的容器访问。
在ceph-mon Pod中有3个容器运行ceph-mon相当于物理机上的ceph-mon.hostname.logcluster-audit-log-tailer相当于物理机上的ceph.audit.logcluster-log-tailer相当于物理机上的ceph.log或ceph -w。每个容器都可以通过--container或-c选项访问。例如要访问cluster-tail-log可以运行
```bash
$ kubectl -n ceph logs ceph-mon-cppdk -c cluster-log-tailer
```