kubernetes-handbook/practice/using-ceph-for-persistent-s...

5155 lines
290 KiB
HTML
Raw Normal View History

<!DOCTYPE HTML>
<html lang="zh-hans" >
<head>
<meta charset="UTF-8">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
<title>使用 Ceph 做持久化存储 · Kubernetes Handbook - Kubernetes中文指南/云原生应用架构实践手册 by Jimmy Song(宋净超)</title>
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="description" content="">
<meta name="generator" content="GitBook 3.2.3">
<meta name="author" content="Jimmy Song宋净超">
<link rel="stylesheet" href="../gitbook/style.css">
<link rel="stylesheet" href="../gitbook/gitbook-plugin-splitter/splitter.css">
<link rel="stylesheet" href="../gitbook/gitbook-plugin-page-toc-button/plugin.css">
<link rel="stylesheet" href="../gitbook/gitbook-plugin-image-captions/image-captions.css">
<link rel="stylesheet" href="../gitbook/gitbook-plugin-back-to-top-button/plugin.css">
<link rel="stylesheet" href="../gitbook/gitbook-plugin-search-plus/search.css">
<link rel="stylesheet" href="../gitbook/gitbook-plugin-tbfed-pagefooter/footer.css">
<link rel="stylesheet" href="../gitbook/gitbook-plugin-prism/prism-ghcolors.css">
<link rel="stylesheet" href="../gitbook/gitbook-plugin-lightbox/css/lightbox.min.css">
<link rel="stylesheet" href="../gitbook/gitbook-plugin-alerts/style.css">
<link rel="stylesheet" href="../gitbook/gitbook-plugin-fontsettings/website.css">
<meta name="HandheldFriendly" content="true"/>
<meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no">
<meta name="apple-mobile-web-app-capable" content="yes">
<meta name="apple-mobile-web-app-status-bar-style" content="black">
<link rel="apple-touch-icon-precomposed" sizes="152x152" href="../gitbook/images/apple-touch-icon-precomposed-152.png">
<link rel="shortcut icon" href="../gitbook/images/favicon.ico" type="image/x-icon">
<link rel="next" href="rbd-provisioner.html" />
<link rel="prev" href="ceph-helm-install-guide-zh.html" />
<link rel="shortcut icon" href='../favicon.ico' type="image/x-icon">
<link rel="bookmark" href='../favicon.ico' type="image/x-icon">
<style>
@media only screen and (max-width: 640px) {
.book-header .hidden-mobile {
display: none;
}
}
</style>
<script>
window["gitbook-plugin-github-buttons"] = {"repo":"rootsongjc/kubernetes-handbook","types":["star"],"size":"small"};
</script>
</head>
<body>
<div class="book">
<div class="book-summary">
<div id="book-search-input" role="search">
<input type="text" placeholder="输入并搜索" />
</div>
<nav role="navigation">
<ul class="summary">
<li>
<a href="https://jimmysong.io" target="_blank" class="custom-link">回到主页</a>
</li>
<li>
<a href="https://jimmysong.io/awesome-cloud-native" target="_blank" class="custom-link">云原生开源项目大全</a>
</li>
<li>
<a href="https://cloudnative.to" target="_blank" class="custom-link">云原生社区</a>
</li>
<li class="divider"></li>
<li class="header">前言</li>
<li class="chapter " data-level="1.1" data-path="../">
<a href="../">
<b>1.1.</b>
序言
</a>
</li>
<li class="header">云原生</li>
<li class="chapter " data-level="2.1" data-path="../cloud-native/cloud-native-definition.html">
<a href="../cloud-native/cloud-native-definition.html">
<b>2.1.</b>
云原生Cloud Native的定义
</a>
</li>
<li class="chapter " data-level="2.2" data-path="../cloud-native/cloud-native-philosophy.html">
<a href="../cloud-native/cloud-native-philosophy.html">
<b>2.2.</b>
云原生的设计哲学
</a>
</li>
<li class="chapter " data-level="2.3" data-path="../cloud-native/quick-start.html">
<a href="../cloud-native/quick-start.html">
<b>2.3.</b>
云原生新手入门指南
</a>
</li>
<li class="chapter " data-level="2.4" data-path="../cloud-native/play-with-kubernetes.html">
<a href="../cloud-native/play-with-kubernetes.html">
<b>2.4.</b>
Play with Kubernetes
</a>
</li>
<li class="chapter " data-level="2.5" data-path="../cloud-native/cloud-native-local-quick-start.html">
<a href="../cloud-native/cloud-native-local-quick-start.html">
<b>2.5.</b>
快速部署一个云原生本地实验环境
</a>
</li>
<li class="chapter " data-level="2.6" data-path="../cloud-native/setup-kubernetes-with-rancher-and-aliyun.html">
<a href="../cloud-native/setup-kubernetes-with-rancher-and-aliyun.html">
<b>2.6.</b>
使用 Rancher 在阿里云上部署 Kubenretes 集群
</a>
</li>
<li class="chapter " data-level="2.7" data-path="../cloud-native/kubernetes-and-cloud-native-app-overview.html">
<a href="../cloud-native/kubernetes-and-cloud-native-app-overview.html">
<b>2.7.</b>
Kubernetes 与云原生应用概览
</a>
</li>
<li class="chapter " data-level="2.8" data-path="../cloud-native/from-kubernetes-to-cloud-native.html">
<a href="../cloud-native/from-kubernetes-to-cloud-native.html">
<b>2.8.</b>
云原生应用之路 —— 从 Kubernetes 到 Cloud Native
</a>
</li>
<li class="chapter " data-level="2.9" data-path="../cloud-native/define-cloud-native-app.html">
<a href="../cloud-native/define-cloud-native-app.html">
<b>2.9.</b>
定义云原生应用
</a>
<ul class="articles">
<li class="chapter " data-level="2.9.1" data-path="../cloud-native/oam.html">
<a href="../cloud-native/oam.html">
<b>2.9.1.</b>
OAM
</a>
<ul class="articles">
<li class="chapter " data-level="2.9.1.1" data-path="../cloud-native/workload.html">
<a href="../cloud-native/workload.html">
<b>2.9.1.1.</b>
Workload
</a>
</li>
<li class="chapter " data-level="2.9.1.2" data-path="../cloud-native/component.html">
<a href="../cloud-native/component.html">
<b>2.9.1.2.</b>
Component
</a>
</li>
<li class="chapter " data-level="2.9.1.3" data-path="../cloud-native/trait.html">
<a href="../cloud-native/trait.html">
<b>2.9.1.3.</b>
Trait
</a>
</li>
<li class="chapter " data-level="2.9.1.4" data-path="../cloud-native/application-scope.html">
<a href="../cloud-native/application-scope.html">
<b>2.9.1.4.</b>
Application Scope
</a>
</li>
<li class="chapter " data-level="2.9.1.5" data-path="../cloud-native/application-configuration.html">
<a href="../cloud-native/application-configuration.html">
<b>2.9.1.5.</b>
Application Configuration
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="2.9.2" data-path="../cloud-native/crossplane.html">
<a href="../cloud-native/crossplane.html">
<b>2.9.2.</b>
Crossplane
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="2.10" data-path="../cloud-native/cloud-native-programming-languages.html">
<a href="../cloud-native/cloud-native-programming-languages.html">
<b>2.10.</b>
云原生编程语言
</a>
<ul class="articles">
<li class="chapter " data-level="2.10.1" data-path="../cloud-native/cloud-native-programming-language-ballerina.html">
<a href="../cloud-native/cloud-native-programming-language-ballerina.html">
<b>2.10.1.</b>
云原生编程语言 Ballerina
</a>
</li>
<li class="chapter " data-level="2.10.2" data-path="../cloud-native/cloud-native-programming-language-pulumi.html">
<a href="../cloud-native/cloud-native-programming-language-pulumi.html">
<b>2.10.2.</b>
云原生编程语言 Pulumi
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="2.11" data-path="../cloud-native/the-future-of-cloud-native.html">
<a href="../cloud-native/the-future-of-cloud-native.html">
<b>2.11.</b>
云原生的未来
</a>
</li>
<li class="header">概念与原理</li>
<li class="chapter " data-level="3.1" data-path="../concepts/">
<a href="../concepts/">
<b>3.1.</b>
Kubernetes 架构
</a>
<ul class="articles">
<li class="chapter " data-level="3.1.1" data-path="../concepts/concepts.html">
<a href="../concepts/concepts.html">
<b>3.1.1.</b>
设计理念
</a>
</li>
<li class="chapter " data-level="3.1.2" data-path="../concepts/etcd.html">
<a href="../concepts/etcd.html">
<b>3.1.2.</b>
Etcd 解析
</a>
</li>
<li class="chapter " data-level="3.1.3" data-path="../concepts/open-interfaces.html">
<a href="../concepts/open-interfaces.html">
<b>3.1.3.</b>
开放接口
</a>
<ul class="articles">
<li class="chapter " data-level="3.1.3.1" data-path="../concepts/cri.html">
<a href="../concepts/cri.html">
<b>3.1.3.1.</b>
CRI - Container Runtime Interface容器运行时接口
</a>
</li>
<li class="chapter " data-level="3.1.3.2" data-path="../concepts/cni.html">
<a href="../concepts/cni.html">
<b>3.1.3.2.</b>
CNI - Container Network Interface容器网络接口
</a>
</li>
<li class="chapter " data-level="3.1.3.3" data-path="../concepts/csi.html">
<a href="../concepts/csi.html">
<b>3.1.3.3.</b>
CSI - Container Storage Interface容器存储接口
</a>
</li>
</ul>
</li>
</ul>
</li>
<li class="chapter " data-level="3.2" data-path="../concepts/networking.html">
<a href="../concepts/networking.html">
<b>3.2.</b>
Kubernetes 中的网络
</a>
<ul class="articles">
<li class="chapter " data-level="3.2.1" data-path="../concepts/flannel.html">
<a href="../concepts/flannel.html">
<b>3.2.1.</b>
Kubernetes 中的网络解析 —— 以 flannel 为例
</a>
</li>
<li class="chapter " data-level="3.2.2" data-path="../concepts/calico.html">
<a href="../concepts/calico.html">
<b>3.2.2.</b>
Kubernetes 中的网络解析 —— 以 calico 为例
</a>
</li>
<li class="chapter " data-level="3.2.3" data-path="../concepts/cilium.html">
<a href="../concepts/cilium.html">
<b>3.2.3.</b>
具备 API 感知的网络和安全性管理开源软件 Cilium
</a>
<ul class="articles">
<li class="chapter " data-level="3.2.3.1" data-path="../concepts/cilium-concepts.html">
<a href="../concepts/cilium-concepts.html">
<b>3.2.3.1.</b>
Cilium 架构设计与概念解析
</a>
</li>
</ul>
</li>
</ul>
</li>
<li class="chapter " data-level="3.3" data-path="../concepts/objects.html">
<a href="../concepts/objects.html">
<b>3.3.</b>
资源对象与基本概念解析
</a>
</li>
<li class="chapter " data-level="3.4" data-path="../concepts/pod-state-and-lifecycle.html">
<a href="../concepts/pod-state-and-lifecycle.html">
<b>3.4.</b>
Pod 状态与生命周期管理
</a>
<ul class="articles">
<li class="chapter " data-level="3.4.1" data-path="../concepts/pod-overview.html">
<a href="../concepts/pod-overview.html">
<b>3.4.1.</b>
Pod 概览
</a>
</li>
<li class="chapter " data-level="3.4.2" data-path="../concepts/pod.html">
<a href="../concepts/pod.html">
<b>3.4.2.</b>
Pod 解析
</a>
</li>
<li class="chapter " data-level="3.4.3" data-path="../concepts/init-containers.html">
<a href="../concepts/init-containers.html">
<b>3.4.3.</b>
Init 容器
</a>
</li>
<li class="chapter " data-level="3.4.4" data-path="../concepts/pause-container.html">
<a href="../concepts/pause-container.html">
<b>3.4.4.</b>
Pause 容器
</a>
</li>
<li class="chapter " data-level="3.4.5" data-path="../concepts/pod-security-policy.html">
<a href="../concepts/pod-security-policy.html">
<b>3.4.5.</b>
Pod 安全策略
</a>
</li>
<li class="chapter " data-level="3.4.6" data-path="../concepts/pod-lifecycle.html">
<a href="../concepts/pod-lifecycle.html">
<b>3.4.6.</b>
Pod 的生命周期
</a>
</li>
<li class="chapter " data-level="3.4.7" data-path="../concepts/pod-hook.html">
<a href="../concepts/pod-hook.html">
<b>3.4.7.</b>
Pod Hook
</a>
</li>
<li class="chapter " data-level="3.4.8" data-path="../concepts/pod-preset.html">
<a href="../concepts/pod-preset.html">
<b>3.4.8.</b>
Pod Preset
</a>
</li>
<li class="chapter " data-level="3.4.9" data-path="../concepts/pod-disruption-budget.html">
<a href="../concepts/pod-disruption-budget.html">
<b>3.4.9.</b>
Pod 中断与 PDBPod 中断预算)
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="3.5" data-path="../concepts/cluster.html">
<a href="../concepts/cluster.html">
<b>3.5.</b>
集群资源管理
</a>
<ul class="articles">
<li class="chapter " data-level="3.5.1" data-path="../concepts/node.html">
<a href="../concepts/node.html">
<b>3.5.1.</b>
Node
</a>
</li>
<li class="chapter " data-level="3.5.2" data-path="../concepts/namespace.html">
<a href="../concepts/namespace.html">
<b>3.5.2.</b>
Namespace
</a>
</li>
<li class="chapter " data-level="3.5.3" data-path="../concepts/label.html">
<a href="../concepts/label.html">
<b>3.5.3.</b>
Label
</a>
</li>
<li class="chapter " data-level="3.5.4" data-path="../concepts/annotation.html">
<a href="../concepts/annotation.html">
<b>3.5.4.</b>
Annotation
</a>
</li>
<li class="chapter " data-level="3.5.5" data-path="../concepts/taint-and-toleration.html">
<a href="../concepts/taint-and-toleration.html">
<b>3.5.5.</b>
Taint 和 Toleration污点和容忍
</a>
</li>
<li class="chapter " data-level="3.5.6" data-path="../concepts/garbage-collection.html">
<a href="../concepts/garbage-collection.html">
<b>3.5.6.</b>
垃圾收集
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="3.6" data-path="../concepts/controllers.html">
<a href="../concepts/controllers.html">
<b>3.6.</b>
控制器
</a>
<ul class="articles">
<li class="chapter " data-level="3.6.1" data-path="../concepts/deployment.html">
<a href="../concepts/deployment.html">
<b>3.6.1.</b>
Deployment
</a>
</li>
<li class="chapter " data-level="3.6.2" data-path="../concepts/statefulset.html">
<a href="../concepts/statefulset.html">
<b>3.6.2.</b>
StatefulSet
</a>
</li>
<li class="chapter " data-level="3.6.3" data-path="../concepts/daemonset.html">
<a href="../concepts/daemonset.html">
<b>3.6.3.</b>
DaemonSet
</a>
</li>
<li class="chapter " data-level="3.6.4" data-path="../concepts/replicaset.html">
<a href="../concepts/replicaset.html">
<b>3.6.4.</b>
ReplicationController 和 ReplicaSet
</a>
</li>
<li class="chapter " data-level="3.6.5" data-path="../concepts/job.html">
<a href="../concepts/job.html">
<b>3.6.5.</b>
Job
</a>
</li>
<li class="chapter " data-level="3.6.6" data-path="../concepts/cronjob.html">
<a href="../concepts/cronjob.html">
<b>3.6.6.</b>
CronJob
</a>
</li>
<li class="chapter " data-level="3.6.7" data-path="../concepts/horizontal-pod-autoscaling.html">
<a href="../concepts/horizontal-pod-autoscaling.html">
<b>3.6.7.</b>
Horizontal Pod Autoscaling
</a>
<ul class="articles">
<li class="chapter " data-level="3.6.7.1" data-path="../concepts/custom-metrics-hpa.html">
<a href="../concepts/custom-metrics-hpa.html">
<b>3.6.7.1.</b>
自定义指标 HPA
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="3.6.8" data-path="../concepts/admission-controller.html">
<a href="../concepts/admission-controller.html">
<b>3.6.8.</b>
准入控制器Admission Controller
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="3.7" data-path="../concepts/service-discovery.html">
<a href="../concepts/service-discovery.html">
<b>3.7.</b>
服务发现
</a>
<ul class="articles">
<li class="chapter " data-level="3.7.1" data-path="../concepts/service.html">
<a href="../concepts/service.html">
<b>3.7.1.</b>
Service
</a>
</li>
<li class="chapter " data-level="3.7.2" data-path="../concepts/ingress.html">
<a href="../concepts/ingress.html">
<b>3.7.2.</b>
Ingress
</a>
<ul class="articles">
<li class="chapter " data-level="3.7.2.1" data-path="../concepts/traefik-ingress-controller.html">
<a href="../concepts/traefik-ingress-controller.html">
<b>3.7.2.1.</b>
Traefik Ingress Controller
</a>
</li>
</ul>
</li>
</ul>
</li>
<li class="chapter " data-level="3.8" data-path="../concepts/authentication-and-permission.html">
<a href="../concepts/authentication-and-permission.html">
<b>3.8.</b>
身份与权限控制
</a>
<ul class="articles">
<li class="chapter " data-level="3.8.1" data-path="../concepts/serviceaccount.html">
<a href="../concepts/serviceaccount.html">
<b>3.8.1.</b>
ServiceAccount
</a>
</li>
<li class="chapter " data-level="3.8.2" data-path="../concepts/rbac.html">
<a href="../concepts/rbac.html">
<b>3.8.2.</b>
RBAC—— 基于角色的访问控制
</a>
</li>
<li class="chapter " data-level="3.8.3" data-path="../concepts/network-policy.html">
<a href="../concepts/network-policy.html">
<b>3.8.3.</b>
NetworkPolicy
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="3.9" data-path="../concepts/storage.html">
<a href="../concepts/storage.html">
<b>3.9.</b>
存储
</a>
<ul class="articles">
<li class="chapter " data-level="3.9.1" data-path="../concepts/secret.html">
<a href="../concepts/secret.html">
<b>3.9.1.</b>
Secret
</a>
</li>
<li class="chapter " data-level="3.9.2" data-path="../concepts/configmap.html">
<a href="../concepts/configmap.html">
<b>3.9.2.</b>
ConfigMap
</a>
<ul class="articles">
<li class="chapter " data-level="3.9.2.1" data-path="../concepts/configmap-hot-update.html">
<a href="../concepts/configmap-hot-update.html">
<b>3.9.2.1.</b>
ConfigMap 的热更新
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="3.9.3" data-path="../concepts/volume.html">
<a href="../concepts/volume.html">
<b>3.9.3.</b>
Volume
</a>
</li>
<li class="chapter " data-level="3.9.4" data-path="../concepts/persistent-volume.html">
<a href="../concepts/persistent-volume.html">
<b>3.9.4.</b>
Persistent Volume持久化卷
</a>
</li>
<li class="chapter " data-level="3.9.5" data-path="../concepts/storageclass.html">
<a href="../concepts/storageclass.html">
<b>3.9.5.</b>
Storage Class
</a>
</li>
<li class="chapter " data-level="3.9.6" data-path="../concepts/local-persistent-storage.html">
<a href="../concepts/local-persistent-storage.html">
<b>3.9.6.</b>
本地持久化存储
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="3.10" data-path="../concepts/extension.html">
<a href="../concepts/extension.html">
<b>3.10.</b>
集群扩展
</a>
<ul class="articles">
<li class="chapter " data-level="3.10.1" data-path="../concepts/custom-resource.html">
<a href="../concepts/custom-resource.html">
<b>3.10.1.</b>
使用自定义资源扩展 API
</a>
</li>
<li class="chapter " data-level="3.10.2" data-path="../concepts/crd.html">
<a href="../concepts/crd.html">
<b>3.10.2.</b>
使用 CRD 扩展 Kubernetes API
</a>
</li>
<li class="chapter " data-level="3.10.3" data-path="../concepts/aggregated-api-server.html">
<a href="../concepts/aggregated-api-server.html">
<b>3.10.3.</b>
Aggregated API Server
</a>
</li>
<li class="chapter " data-level="3.10.4" data-path="../concepts/apiservice.html">
<a href="../concepts/apiservice.html">
<b>3.10.4.</b>
APIService
</a>
</li>
<li class="chapter " data-level="3.10.5" data-path="../concepts/service-catalog.html">
<a href="../concepts/service-catalog.html">
<b>3.10.5.</b>
Service Catalog
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="3.11" data-path="../concepts/scheduling.html">
<a href="../concepts/scheduling.html">
<b>3.11.</b>
资源调度
</a>
<ul class="articles">
<li class="chapter " data-level="3.11.1" data-path="../concepts/qos.html">
<a href="../concepts/qos.html">
<b>3.11.1.</b>
QoS服务质量等级
</a>
</li>
</ul>
</li>
<li class="header">用户指南</li>
<li class="chapter " data-level="4.1" data-path="../guide/">
<a href="../guide/">
<b>4.1.</b>
用户指南
</a>
</li>
<li class="chapter " data-level="4.2" data-path="../guide/resource-configuration.html">
<a href="../guide/resource-configuration.html">
<b>4.2.</b>
资源对象配置
</a>
<ul class="articles">
<li class="chapter " data-level="4.2.1" data-path="../guide/configure-liveness-readiness-probes.html">
<a href="../guide/configure-liveness-readiness-probes.html">
<b>4.2.1.</b>
配置 Pod 的 liveness 和 readiness 探针
</a>
</li>
<li class="chapter " data-level="4.2.2" data-path="../guide/configure-pod-service-account.html">
<a href="../guide/configure-pod-service-account.html">
<b>4.2.2.</b>
配置 Pod 的 Service Account
</a>
</li>
<li class="chapter " data-level="4.2.3" data-path="../guide/secret-configuration.html">
<a href="../guide/secret-configuration.html">
<b>4.2.3.</b>
Secret 配置
</a>
</li>
<li class="chapter " data-level="4.2.4" data-path="../guide/resource-quota-management.html">
<a href="../guide/resource-quota-management.html">
<b>4.2.4.</b>
管理 namespace 中的资源配额
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="4.3" data-path="../guide/command-usage.html">
<a href="../guide/command-usage.html">
<b>4.3.</b>
命令使用
</a>
<ul class="articles">
<li class="chapter " data-level="4.3.1" data-path="../guide/docker-cli-to-kubectl.html">
<a href="../guide/docker-cli-to-kubectl.html">
<b>4.3.1.</b>
Docker 用户过渡到 kubectl 命令行指南
</a>
</li>
<li class="chapter " data-level="4.3.2" data-path="../guide/using-kubectl.html">
<a href="../guide/using-kubectl.html">
<b>4.3.2.</b>
kubectl 命令概览
</a>
</li>
<li class="chapter " data-level="4.3.3" data-path="../guide/kubectl-cheatsheet.html">
<a href="../guide/kubectl-cheatsheet.html">
<b>4.3.3.</b>
kubectl 命令技巧大全
</a>
</li>
<li class="chapter " data-level="4.3.4" data-path="../guide/using-etcdctl-to-access-kubernetes-data.html">
<a href="../guide/using-etcdctl-to-access-kubernetes-data.html">
<b>4.3.4.</b>
使用 etcdctl 访问 kubernetes 数据
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="4.4" data-path="../guide/cluster-security-management.html">
<a href="../guide/cluster-security-management.html">
<b>4.4.</b>
集群安全性管理
</a>
<ul class="articles">
<li class="chapter " data-level="4.4.1" data-path="../guide/managing-tls-in-a-cluster.html">
<a href="../guide/managing-tls-in-a-cluster.html">
<b>4.4.1.</b>
管理集群中的 TLS
</a>
</li>
<li class="chapter " data-level="4.4.2" data-path="../guide/kubelet-authentication-authorization.html">
<a href="../guide/kubelet-authentication-authorization.html">
<b>4.4.2.</b>
kubelet 的认证授权
</a>
</li>
<li class="chapter " data-level="4.4.3" data-path="../guide/tls-bootstrapping.html">
<a href="../guide/tls-bootstrapping.html">
<b>4.4.3.</b>
TLS Bootstrap
</a>
</li>
<li class="chapter " data-level="4.4.4" data-path="../guide/kubectl-user-authentication-authorization.html">
<a href="../guide/kubectl-user-authentication-authorization.html">
<b>4.4.4.</b>
创建用户认证授权的 kubeconfig 文件
</a>
</li>
<li class="chapter " data-level="4.4.5" data-path="../guide/ip-masq-agent.html">
<a href="../guide/ip-masq-agent.html">
<b>4.4.5.</b>
IP 伪装代理
</a>
</li>
<li class="chapter " data-level="4.4.6" data-path="../guide/auth-with-kubeconfig-or-token.html">
<a href="../guide/auth-with-kubeconfig-or-token.html">
<b>4.4.6.</b>
使用 kubeconfig 或 token 进行用户身份认证
</a>
</li>
<li class="chapter " data-level="4.4.7" data-path="../guide/authentication.html">
<a href="../guide/authentication.html">
<b>4.4.7.</b>
Kubernetes 中的用户与身份认证授权
</a>
</li>
<li class="chapter " data-level="4.4.8" data-path="../guide/kubernetes-security-best-practice.html">
<a href="../guide/kubernetes-security-best-practice.html">
<b>4.4.8.</b>
Kubernetes 集群安全性配置最佳实践
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="4.5" data-path="../guide/access-kubernetes-cluster.html">
<a href="../guide/access-kubernetes-cluster.html">
<b>4.5.</b>
访问 Kubernetes 集群
</a>
<ul class="articles">
<li class="chapter " data-level="4.5.1" data-path="../guide/access-cluster.html">
<a href="../guide/access-cluster.html">
<b>4.5.1.</b>
访问集群
</a>
</li>
<li class="chapter " data-level="4.5.2" data-path="../guide/authenticate-across-clusters-kubeconfig.html">
<a href="../guide/authenticate-across-clusters-kubeconfig.html">
<b>4.5.2.</b>
使用 kubeconfig 文件配置跨集群认证
</a>
</li>
<li class="chapter " data-level="4.5.3" data-path="../guide/connecting-to-applications-port-forward.html">
<a href="../guide/connecting-to-applications-port-forward.html">
<b>4.5.3.</b>
通过端口转发访问集群中的应用程序
</a>
</li>
<li class="chapter " data-level="4.5.4" data-path="../guide/service-access-application-cluster.html">
<a href="../guide/service-access-application-cluster.html">
<b>4.5.4.</b>
使用 service 访问群集中的应用程序
</a>
</li>
<li class="chapter " data-level="4.5.5" data-path="../guide/accessing-kubernetes-pods-from-outside-of-the-cluster.html">
<a href="../guide/accessing-kubernetes-pods-from-outside-of-the-cluster.html">
<b>4.5.5.</b>
从外部访问 Kubernetes 中的 Pod
</a>
</li>
<li class="chapter " data-level="4.5.6" data-path="../guide/cabin-mobile-dashboard-for-kubernetes.html">
<a href="../guide/cabin-mobile-dashboard-for-kubernetes.html">
<b>4.5.6.</b>
Cabin - Kubernetes 手机客户端
</a>
</li>
<li class="chapter " data-level="4.5.7" data-path="../guide/kubernetes-desktop-client.html">
<a href="../guide/kubernetes-desktop-client.html">
<b>4.5.7.</b>
Lens - Kubernetes IDE/桌面客户端
</a>
</li>
<li class="chapter " data-level="4.5.8" data-path="../guide/kubernator-kubernetes-ui.html">
<a href="../guide/kubernator-kubernetes-ui.html">
<b>4.5.8.</b>
Kubernator - 更底层的 Kubernetes UI
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="4.6" data-path="../guide/application-development-deployment-flow.html">
<a href="../guide/application-development-deployment-flow.html">
<b>4.6.</b>
在 Kubernetes 中开发部署应用
</a>
<ul class="articles">
<li class="chapter " data-level="4.6.1" data-path="../guide/deploy-applications-in-kubernetes.html">
<a href="../guide/deploy-applications-in-kubernetes.html">
<b>4.6.1.</b>
适用于 kubernetes 的应用开发部署流程
</a>
</li>
<li class="chapter " data-level="4.6.2" data-path="../guide/migrating-hadoop-yarn-to-kubernetes.html">
<a href="../guide/migrating-hadoop-yarn-to-kubernetes.html">
<b>4.6.2.</b>
迁移传统应用到 Kubernetes 中 —— 以 Hadoop YARN 为例
</a>
</li>
<li class="chapter " data-level="4.6.3" data-path="../guide/using-statefulset.html">
<a href="../guide/using-statefulset.html">
<b>4.6.3.</b>
使用 StatefulSet 部署用状态应用
</a>
</li>
</ul>
</li>
<li class="header">最佳实践</li>
<li class="chapter " data-level="5.1" data-path="./">
<a href="./">
<b>5.1.</b>
最佳实践概览
</a>
</li>
<li class="chapter " data-level="5.2" data-path="install-kubernetes-on-centos.html">
<a href="install-kubernetes-on-centos.html">
<b>5.2.</b>
在 CentOS 上部署 Kubernetes 集群
</a>
<ul class="articles">
<li class="chapter " data-level="5.2.1" data-path="create-tls-and-secret-key.html">
<a href="create-tls-and-secret-key.html">
<b>5.2.1.</b>
创建 TLS 证书和秘钥
</a>
</li>
<li class="chapter " data-level="5.2.2" data-path="create-kubeconfig.html">
<a href="create-kubeconfig.html">
<b>5.2.2.</b>
创建 kubeconfig 文件
</a>
</li>
<li class="chapter " data-level="5.2.3" data-path="etcd-cluster-installation.html">
<a href="etcd-cluster-installation.html">
<b>5.2.3.</b>
创建高可用 etcd 集群
</a>
</li>
<li class="chapter " data-level="5.2.4" data-path="kubectl-installation.html">
<a href="kubectl-installation.html">
<b>5.2.4.</b>
安装 kubectl 命令行工具
</a>
</li>
<li class="chapter " data-level="5.2.5" data-path="master-installation.html">
<a href="master-installation.html">
<b>5.2.5.</b>
部署 master 节点
</a>
</li>
<li class="chapter " data-level="5.2.6" data-path="flannel-installation.html">
<a href="flannel-installation.html">
<b>5.2.6.</b>
安装 flannel 网络插件
</a>
</li>
<li class="chapter " data-level="5.2.7" data-path="node-installation.html">
<a href="node-installation.html">
<b>5.2.7.</b>
部署 node 节点
</a>
</li>
<li class="chapter " data-level="5.2.8" data-path="kubedns-addon-installation.html">
<a href="kubedns-addon-installation.html">
<b>5.2.8.</b>
安装 kubedns 插件
</a>
</li>
<li class="chapter " data-level="5.2.9" data-path="dashboard-addon-installation.html">
<a href="dashboard-addon-installation.html">
<b>5.2.9.</b>
安装 dashboard 插件
</a>
</li>
<li class="chapter " data-level="5.2.10" data-path="heapster-addon-installation.html">
<a href="heapster-addon-installation.html">
<b>5.2.10.</b>
安装 heapster 插件
</a>
</li>
<li class="chapter " data-level="5.2.11" data-path="efk-addon-installation.html">
<a href="efk-addon-installation.html">
<b>5.2.11.</b>
安装 EFK 插件
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="5.3" data-path="install-kubernetes-with-kubeadm.html">
<a href="install-kubernetes-with-kubeadm.html">
<b>5.3.</b>
生产级的 Kubernetes 简化管理工具 kubeadm
</a>
<ul class="articles">
<li class="chapter " data-level="5.3.1" data-path="install-kubernetes-on-ubuntu-server-16.04-with-kubeadm.html">
<a href="install-kubernetes-on-ubuntu-server-16.04-with-kubeadm.html">
<b>5.3.1.</b>
使用 kubeadm 在 Ubuntu Server 16.04 上快速构建测试集群
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="5.4" data-path="service-discovery-and-loadbalancing.html">
<a href="service-discovery-and-loadbalancing.html">
<b>5.4.</b>
服务发现与负载均衡
</a>
<ul class="articles">
<li class="chapter " data-level="5.4.1" data-path="traefik-ingress-installation.html">
<a href="traefik-ingress-installation.html">
<b>5.4.1.</b>
安装 Traefik ingress
</a>
</li>
<li class="chapter " data-level="5.4.2" data-path="distributed-load-test.html">
<a href="distributed-load-test.html">
<b>5.4.2.</b>
分布式负载测试
</a>
</li>
<li class="chapter " data-level="5.4.3" data-path="network-and-cluster-perfermance-test.html">
<a href="network-and-cluster-perfermance-test.html">
<b>5.4.3.</b>
网络和集群性能测试
</a>
</li>
<li class="chapter " data-level="5.4.4" data-path="edge-node-configuration.html">
<a href="edge-node-configuration.html">
<b>5.4.4.</b>
边缘节点配置
</a>
</li>
<li class="chapter " data-level="5.4.5" data-path="nginx-ingress-installation.html">
<a href="nginx-ingress-installation.html">
<b>5.4.5.</b>
安装 Nginx ingress
</a>
</li>
<li class="chapter " data-level="5.4.6" data-path="dns-installation.html">
<a href="dns-installation.html">
<b>5.4.6.</b>
安装配置 DNS
</a>
<ul class="articles">
<li class="chapter " data-level="5.4.6.1" data-path="configuring-dns.html">
<a href="configuring-dns.html">
<b>5.4.6.1.</b>
安装配置 Kube-dns
</a>
</li>
<li class="chapter " data-level="5.4.6.2" data-path="coredns.html">
<a href="coredns.html">
<b>5.4.6.2.</b>
安装配置 CoreDNS
</a>
</li>
</ul>
</li>
</ul>
</li>
<li class="chapter " data-level="5.5" data-path="operation.html">
<a href="operation.html">
<b>5.5.</b>
运维管理
</a>
<ul class="articles">
<li class="chapter " data-level="5.5.1" data-path="master-ha.html">
<a href="master-ha.html">
<b>5.5.1.</b>
Master 节点高可用
</a>
</li>
<li class="chapter " data-level="5.5.2" data-path="service-rolling-update.html">
<a href="service-rolling-update.html">
<b>5.5.2.</b>
服务滚动升级
</a>
</li>
<li class="chapter " data-level="5.5.3" data-path="app-log-collection.html">
<a href="app-log-collection.html">
<b>5.5.3.</b>
应用日志收集
</a>
</li>
<li class="chapter " data-level="5.5.4" data-path="configuration-best-practice.html">
<a href="configuration-best-practice.html">
<b>5.5.4.</b>
配置最佳实践
</a>
</li>
<li class="chapter " data-level="5.5.5" data-path="monitor.html">
<a href="monitor.html">
<b>5.5.5.</b>
集群及应用监控
</a>
</li>
<li class="chapter " data-level="5.5.6" data-path="data-persistence-problem.html">
<a href="data-persistence-problem.html">
<b>5.5.6.</b>
数据持久化问题
</a>
</li>
<li class="chapter " data-level="5.5.7" data-path="manage-compute-resources-container.html">
<a href="manage-compute-resources-container.html">
<b>5.5.7.</b>
管理容器的计算资源
</a>
</li>
<li class="chapter " data-level="5.5.8" data-path="federation.html">
<a href="federation.html">
<b>5.5.8.</b>
集群联邦
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="5.6" data-path="storage.html">
<a href="storage.html">
<b>5.6.</b>
存储管理
</a>
<ul class="articles">
<li class="chapter " data-level="5.6.1" data-path="glusterfs.html">
<a href="glusterfs.html">
<b>5.6.1.</b>
GlusterFS
</a>
<ul class="articles">
<li class="chapter " data-level="5.6.1.1" data-path="using-glusterfs-for-persistent-storage.html">
<a href="using-glusterfs-for-persistent-storage.html">
<b>5.6.1.1.</b>
使用 GlusterFS 做持久化存储
</a>
</li>
<li class="chapter " data-level="5.6.1.2" data-path="using-heketi-gluster-for-persistent-storage.html">
<a href="using-heketi-gluster-for-persistent-storage.html">
<b>5.6.1.2.</b>
使用 Heketi 作为 Kubernetes 的持久存储 GlusterFS 的 external provisioner
</a>
</li>
<li class="chapter " data-level="5.6.1.3" data-path="storage-for-containers-using-glusterfs-with-openshift.html">
<a href="storage-for-containers-using-glusterfs-with-openshift.html">
<b>5.6.1.3.</b>
在 OpenShift 中使用 GlusterFS 做持久化存储
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="5.6.2" data-path="glusterd-2.0.html">
<a href="glusterd-2.0.html">
<b>5.6.2.</b>
GlusterD-2.0
</a>
</li>
<li class="chapter " data-level="5.6.3" data-path="ceph.html">
<a href="ceph.html">
<b>5.6.3.</b>
Ceph
</a>
<ul class="articles">
<li class="chapter " data-level="5.6.3.1" data-path="ceph-helm-install-guide-zh.html">
<a href="ceph-helm-install-guide-zh.html">
<b>5.6.3.1.</b>
用 Helm 托管安装 Ceph 集群并提供后端存储
</a>
</li>
<li class="chapter active" data-level="5.6.3.2" data-path="using-ceph-for-persistent-storage.html">
<a href="using-ceph-for-persistent-storage.html">
<b>5.6.3.2.</b>
使用 Ceph 做持久化存储
</a>
</li>
<li class="chapter " data-level="5.6.3.3" data-path="rbd-provisioner.html">
<a href="rbd-provisioner.html">
<b>5.6.3.3.</b>
使用 rbd-provisioner 提供 rbd 持久化存储
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="5.6.4" data-path="openebs.html">
<a href="openebs.html">
<b>5.6.4.</b>
OpenEBS
</a>
<ul class="articles">
<li class="chapter " data-level="5.6.4.1" data-path="using-openebs-for-persistent-storage.html">
<a href="using-openebs-for-persistent-storage.html">
<b>5.6.4.1.</b>
使用 OpenEBS 做持久化存储
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="5.6.5" data-path="rook.html">
<a href="rook.html">
<b>5.6.5.</b>
Rook
</a>
</li>
<li class="chapter " data-level="5.6.6" data-path="nfs.html">
<a href="nfs.html">
<b>5.6.6.</b>
NFS
</a>
<ul class="articles">
<li class="chapter " data-level="5.6.6.1" data-path="using-nfs-for-persistent-storage.html">
<a href="using-nfs-for-persistent-storage.html">
<b>5.6.6.1.</b>
利用 NFS 动态提供 Kubernetes 后端存储卷
</a>
</li>
</ul>
</li>
</ul>
</li>
<li class="chapter " data-level="5.7" data-path="monitoring.html">
<a href="monitoring.html">
<b>5.7.</b>
集群与应用监控
</a>
<ul class="articles">
<li class="chapter " data-level="5.7.1" data-path="heapster.html">
<a href="heapster.html">
<b>5.7.1.</b>
Heapster
</a>
<ul class="articles">
<li class="chapter " data-level="5.7.1.1" data-path="using-heapster-to-get-object-metrics.html">
<a href="using-heapster-to-get-object-metrics.html">
<b>5.7.1.1.</b>
使用 Heapster 获取集群和对象的 metric 数据
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="5.7.2" data-path="prometheus.html">
<a href="prometheus.html">
<b>5.7.2.</b>
Prometheus
</a>
<ul class="articles">
<li class="chapter " data-level="5.7.2.1" data-path="using-prometheus-to-monitor-kuberentes-cluster.html">
<a href="using-prometheus-to-monitor-kuberentes-cluster.html">
<b>5.7.2.1.</b>
使用 Prometheus 监控 kubernetes 集群
</a>
</li>
<li class="chapter " data-level="5.7.2.2" data-path="promql.html">
<a href="promql.html">
<b>5.7.2.2.</b>
Prometheus 查询语言 PromQL 使用说明
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="5.7.3" data-path="vistio-visualize-your-istio-mesh.html">
<a href="vistio-visualize-your-istio-mesh.html">
<b>5.7.3.</b>
使用 Vistio 监控 Istio 服务网格中的流量
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="5.8" data-path="distributed-tracing.html">
<a href="distributed-tracing.html">
<b>5.8.</b>
分布式跟踪
</a>
<ul class="articles">
<li class="chapter " data-level="5.8.1" data-path="opentracing.html">
<a href="opentracing.html">
<b>5.8.1.</b>
OpenTracing
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="5.9" data-path="services-management-tool.html">
<a href="services-management-tool.html">
<b>5.9.</b>
服务编排管理
</a>
<ul class="articles">
<li class="chapter " data-level="5.9.1" data-path="helm.html">
<a href="helm.html">
<b>5.9.1.</b>
使用 Helm 管理 Kubernetes 应用
</a>
</li>
<li class="chapter " data-level="5.9.2" data-path="create-private-charts-repo.html">
<a href="create-private-charts-repo.html">
<b>5.9.2.</b>
构建私有 Chart 仓库
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="5.10" data-path="ci-cd.html">
<a href="ci-cd.html">
<b>5.10.</b>
持续集成与发布
</a>
<ul class="articles">
<li class="chapter " data-level="5.10.1" data-path="jenkins-ci-cd.html">
<a href="jenkins-ci-cd.html">
<b>5.10.1.</b>
使用 Jenkins 进行持续集成与发布
</a>
</li>
<li class="chapter " data-level="5.10.2" data-path="drone-ci-cd.html">
<a href="drone-ci-cd.html">
<b>5.10.2.</b>
使用 Drone 进行持续集成与发布
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="5.11" data-path="update-and-upgrade.html">
<a href="update-and-upgrade.html">
<b>5.11.</b>
更新与升级
</a>
<ul class="articles">
<li class="chapter " data-level="5.11.1" data-path="manually-upgrade.html">
<a href="manually-upgrade.html">
<b>5.11.1.</b>
手动升级 Kubernetes 集群
</a>
</li>
<li class="chapter " data-level="5.11.2" data-path="dashboard-upgrade.html">
<a href="dashboard-upgrade.html">
<b>5.11.2.</b>
升级 dashboard
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="5.12" data-path="controller-extended.html">
<a href="controller-extended.html">
<b>5.12.</b>
扩展控制器
</a>
<ul class="articles">
<li class="chapter " data-level="5.12.1" data-path="openkruise.html">
<a href="openkruise.html">
<b>5.12.1.</b>
OpenKruise
</a>
<ul class="articles">
<li class="chapter " data-level="5.12.1.1" data-path="in-place-update.html">
<a href="in-place-update.html">
<b>5.12.1.1.</b>
原地升级
</a>
</li>
</ul>
</li>
</ul>
</li>
<li class="header">领域应用</li>
<li class="chapter " data-level="6.1" data-path="../usecases/">
<a href="../usecases/">
<b>6.1.</b>
领域应用概览
</a>
</li>
<li class="chapter " data-level="6.2" data-path="../usecases/microservices.html">
<a href="../usecases/microservices.html">
<b>6.2.</b>
微服务架构
</a>
<ul class="articles">
<li class="chapter " data-level="6.2.1" data-path="../usecases/service-discovery-in-microservices.html">
<a href="../usecases/service-discovery-in-microservices.html">
<b>6.2.1.</b>
微服务中的服务发现
</a>
</li>
<li class="chapter " data-level="6.2.2" data-path="../usecases/microservices-for-java-developers.html">
<a href="../usecases/microservices-for-java-developers.html">
<b>6.2.2.</b>
使用 Java 构建微服务并发布到 Kubernetes 平台
</a>
<ul class="articles">
<li class="chapter " data-level="6.2.2.1" data-path="../usecases/spring-boot-quick-start-guide.html">
<a href="../usecases/spring-boot-quick-start-guide.html">
<b>6.2.2.1.</b>
Spring Boot 快速开始指南
</a>
</li>
</ul>
</li>
</ul>
</li>
<li class="chapter " data-level="6.3" data-path="../usecases/observability.html">
<a href="../usecases/observability.html">
<b>6.3.</b>
可观察性
</a>
</li>
<li class="chapter " data-level="6.4" data-path="../usecases/service-mesh.html">
<a href="../usecases/service-mesh.html">
<b>6.4.</b>
Service Mesh 服务网格
</a>
<ul class="articles">
<li class="chapter " data-level="6.4.1" data-path="../usecases/the-enterprise-path-to-service-mesh-architectures.html">
<a href="../usecases/the-enterprise-path-to-service-mesh-architectures.html">
<b>6.4.1.</b>
企业级服务网格架构
</a>
<ul class="articles">
<li class="chapter " data-level="6.4.1.1" data-path="../usecases/service-mesh-fundamental.html">
<a href="../usecases/service-mesh-fundamental.html">
<b>6.4.1.1.</b>
Service Mesh 基础
</a>
</li>
<li class="chapter " data-level="6.4.1.2" data-path="../usecases/comparing-service-mesh-technologies.html">
<a href="../usecases/comparing-service-mesh-technologies.html">
<b>6.4.1.2.</b>
Service Mesh 技术对比
</a>
</li>
<li class="chapter " data-level="6.4.1.3" data-path="../usecases/service-mesh-adoption-and-evolution.html">
<a href="../usecases/service-mesh-adoption-and-evolution.html">
<b>6.4.1.3.</b>
采纳和演进
</a>
</li>
<li class="chapter " data-level="6.4.1.4" data-path="../usecases/service-mesh-customization-and-integration.html">
<a href="../usecases/service-mesh-customization-and-integration.html">
<b>6.4.1.4.</b>
定制和集成
</a>
</li>
<li class="chapter " data-level="6.4.1.5" data-path="../usecases/service-mesh-conclusion.html">
<a href="../usecases/service-mesh-conclusion.html">
<b>6.4.1.5.</b>
总结
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="6.4.2" data-path="../usecases/istio.html">
<a href="../usecases/istio.html">
<b>6.4.2.</b>
Istio
</a>
<ul class="articles">
<li class="chapter " data-level="6.4.2.1" data-path="../usecases/istio-installation.html">
<a href="../usecases/istio-installation.html">
<b>6.4.2.1.</b>
安装并试用 Istio service mesh
</a>
</li>
<li class="chapter " data-level="6.4.2.2" data-path="../usecases/sidecar-spec-in-istio.html">
<a href="../usecases/sidecar-spec-in-istio.html">
<b>6.4.2.2.</b>
Istio 中 sidecar 的注入规范及示例
</a>
</li>
<li class="chapter " data-level="6.4.2.3" data-path="../usecases/istio-community-tips.html">
<a href="../usecases/istio-community-tips.html">
<b>6.4.2.3.</b>
如何参与 Istio 社区及注意事项
</a>
</li>
<li class="chapter " data-level="6.4.2.4" data-path="../usecases/istio-tutorials-collection.html">
<a href="../usecases/istio-tutorials-collection.html">
<b>6.4.2.4.</b>
Istio 免费学习资源汇总
</a>
</li>
<li class="chapter " data-level="6.4.2.5" data-path="../usecases/understand-sidecar-injection-and-traffic-hijack-in-istio-service-mesh.html">
<a href="../usecases/understand-sidecar-injection-and-traffic-hijack-in-istio-service-mesh.html">
<b>6.4.2.5.</b>
Sidecar 的注入与流量劫持
</a>
</li>
<li class="chapter " data-level="6.4.2.6" data-path="../usecases/envoy-sidecar-routing-of-istio-service-mesh-deep-dive.html">
<a href="../usecases/envoy-sidecar-routing-of-istio-service-mesh-deep-dive.html">
<b>6.4.2.6.</b>
Envoy Sidecar 代理的路由转发
</a>
</li>
<li class="chapter " data-level="6.4.2.7" data-path="../usecases/how-to-integrate-istio-with-vm.html">
<a href="../usecases/how-to-integrate-istio-with-vm.html">
<b>6.4.2.7.</b>
Istio 如何支持虚拟机
</a>
</li>
<li class="chapter " data-level="6.4.2.8" data-path="../usecases/istio-vm-support.html">
<a href="../usecases/istio-vm-support.html">
<b>6.4.2.8.</b>
Istio 支持虚拟机的历史
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="6.4.3" data-path="../usecases/linkerd.html">
<a href="../usecases/linkerd.html">
<b>6.4.3.</b>
Linkerd
</a>
<ul class="articles">
<li class="chapter " data-level="6.4.3.1" data-path="../usecases/linkerd-user-guide.html">
<a href="../usecases/linkerd-user-guide.html">
<b>6.4.3.1.</b>
Linkerd 使用指南
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="6.4.4" data-path="../usecases/conduit.html">
<a href="../usecases/conduit.html">
<b>6.4.4.</b>
Conduit
</a>
<ul class="articles">
<li class="chapter " data-level="6.4.4.1" data-path="../usecases/conduit-overview.html">
<a href="../usecases/conduit-overview.html">
<b>6.4.4.1.</b>
Condiut 概览
</a>
</li>
<li class="chapter " data-level="6.4.4.2" data-path="../usecases/conduit-installation.html">
<a href="../usecases/conduit-installation.html">
<b>6.4.4.2.</b>
安装 Conduit
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="6.4.5" data-path="../usecases/envoy.html">
<a href="../usecases/envoy.html">
<b>6.4.5.</b>
Envoy
</a>
<ul class="articles">
<li class="chapter " data-level="6.4.5.1" data-path="../usecases/envoy-terminology.html">
<a href="../usecases/envoy-terminology.html">
<b>6.4.5.1.</b>
Envoy 的架构与基本术语
</a>
</li>
<li class="chapter " data-level="6.4.5.2" data-path="../usecases/envoy-front-proxy.html">
<a href="../usecases/envoy-front-proxy.html">
<b>6.4.5.2.</b>
Envoy 作为前端代理
</a>
</li>
<li class="chapter " data-level="6.4.5.3" data-path="../usecases/envoy-mesh-in-kubernetes-tutorial.html">
<a href="../usecases/envoy-mesh-in-kubernetes-tutorial.html">
<b>6.4.5.3.</b>
Envoy mesh 教程
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="6.4.6" data-path="../usecases/mosn.html">
<a href="../usecases/mosn.html">
<b>6.4.6.</b>
MOSN
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="6.5" data-path="../usecases/big-data.html">
<a href="../usecases/big-data.html">
<b>6.5.</b>
大数据
</a>
<ul class="articles">
<li class="chapter " data-level="6.5.1" data-path="../usecases/spark-standalone-on-kubernetes.html">
<a href="../usecases/spark-standalone-on-kubernetes.html">
<b>6.5.1.</b>
Spark standalone on Kubernetes
</a>
</li>
<li class="chapter " data-level="6.5.2" data-path="../usecases/running-spark-with-kubernetes-native-scheduler.html">
<a href="../usecases/running-spark-with-kubernetes-native-scheduler.html">
<b>6.5.2.</b>
运行支持 Kubernetes 原生调度的 Spark 程序
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="6.6" data-path="../usecases/serverless.html">
<a href="../usecases/serverless.html">
<b>6.6.</b>
Serverless 架构
</a>
<ul class="articles">
<li class="chapter " data-level="6.6.1" data-path="../usecases/understanding-serverless.html">
<a href="../usecases/understanding-serverless.html">
<b>6.6.1.</b>
理解 Serverless
</a>
</li>
<li class="chapter " data-level="6.6.2" data-path="../usecases/faas.html">
<a href="../usecases/faas.html">
<b>6.6.2.</b>
FaaS函数即服务
</a>
<ul class="articles">
<li class="chapter " data-level="6.6.2.1" data-path="../usecases/openfaas-quick-start.html">
<a href="../usecases/openfaas-quick-start.html">
<b>6.6.2.1.</b>
OpenFaaS 快速入门指南
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="6.6.3" data-path="../usecases/knative.html">
<a href="../usecases/knative.html">
<b>6.6.3.</b>
Knative
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="6.7" data-path="../usecases/edge-computing.html">
<a href="../usecases/edge-computing.html">
<b>6.7.</b>
边缘计算
</a>
</li>
<li class="chapter " data-level="6.8" data-path="../usecases/ai.html">
<a href="../usecases/ai.html">
<b>6.8.</b>
人工智能
</a>
</li>
<li class="header">开发指南</li>
<li class="chapter " data-level="7.1" data-path="../develop/">
<a href="../develop/">
<b>7.1.</b>
开发指南概览
</a>
</li>
<li class="chapter " data-level="7.2" data-path="../develop/sigs-and-working-group.html">
<a href="../develop/sigs-and-working-group.html">
<b>7.2.</b>
SIG 和工作组
</a>
</li>
<li class="chapter " data-level="7.3" data-path="../develop/developing-environment.html">
<a href="../develop/developing-environment.html">
<b>7.3.</b>
开发环境搭建
</a>
<ul class="articles">
<li class="chapter " data-level="7.3.1" data-path="../develop/using-vagrant-and-virtualbox-for-development.html">
<a href="../develop/using-vagrant-and-virtualbox-for-development.html">
<b>7.3.1.</b>
本地分布式开发环境搭建(使用 Vagrant 和 Virtualbox
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="7.4" data-path="../develop/testing.html">
<a href="../develop/testing.html">
<b>7.4.</b>
单元测试和集成测试
</a>
</li>
<li class="chapter " data-level="7.5" data-path="../develop/client-go-sample.html">
<a href="../develop/client-go-sample.html">
<b>7.5.</b>
client-go 示例
</a>
</li>
<li class="chapter " data-level="7.6" data-path="../develop/operator.html">
<a href="../develop/operator.html">
<b>7.6.</b>
Operator
</a>
<ul class="articles">
<li class="chapter " data-level="7.6.1" data-path="../develop/operator-sdk.html">
<a href="../develop/operator-sdk.html">
<b>7.6.1.</b>
operator-sdk
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="7.7" data-path="../develop/kubebuilder.html">
<a href="../develop/kubebuilder.html">
<b>7.7.</b>
kubebuilder
</a>
<ul class="articles">
<li class="chapter " data-level="7.7.1" data-path="../develop/kubebuilder-example.html">
<a href="../develop/kubebuilder-example.html">
<b>7.7.1.</b>
使用 kubebuilder 创建 operator 示例
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="7.8" data-path="../develop/advance-developer.html">
<a href="../develop/advance-developer.html">
<b>7.8.</b>
高级开发指南
</a>
</li>
<li class="chapter " data-level="7.9" data-path="../develop/contribute.html">
<a href="../develop/contribute.html">
<b>7.9.</b>
社区贡献
</a>
</li>
<li class="chapter " data-level="7.10" data-path="../develop/minikube.html">
<a href="../develop/minikube.html">
<b>7.10.</b>
Minikube
</a>
</li>
<li class="header">CNCF云原生计算基金会</li>
<li class="chapter " data-level="8.1" data-path="../cloud-native/cncf.html">
<a href="../cloud-native/cncf.html">
<b>8.1.</b>
CNCF - 云原生计算基金会简介
</a>
</li>
<li class="chapter " data-level="8.2" data-path="../cloud-native/cncf-charter.html">
<a href="../cloud-native/cncf-charter.html">
<b>8.2.</b>
CNCF 章程
</a>
</li>
<li class="chapter " data-level="8.3" data-path="../cloud-native/cncf-sig.html">
<a href="../cloud-native/cncf-sig.html">
<b>8.3.</b>
CNCF 特别兴趣小组SIG说明
</a>
</li>
<li class="chapter " data-level="8.4" data-path="../cloud-native/cncf-sandbox-criteria.html">
<a href="../cloud-native/cncf-sandbox-criteria.html">
<b>8.4.</b>
开源项目加入 CNCF Sandbox 的要求
</a>
</li>
<li class="chapter " data-level="8.5" data-path="../cloud-native/cncf-project-governing.html">
<a href="../cloud-native/cncf-project-governing.html">
<b>8.5.</b>
CNCF 中的项目治理
</a>
</li>
<li class="chapter " data-level="8.6" data-path="../cloud-native/cncf-ambassador.html">
<a href="../cloud-native/cncf-ambassador.html">
<b>8.6.</b>
CNCF Ambassador
</a>
</li>
<li class="header">附录</li>
<li class="chapter " data-level="9.1" data-path="../appendix/">
<a href="../appendix/">
<b>9.1.</b>
附录说明
</a>
</li>
<li class="chapter " data-level="9.2" data-path="../appendix/debug-kubernetes-services.html">
<a href="../appendix/debug-kubernetes-services.html">
<b>9.2.</b>
Kubernetes 中的应用故障排查
</a>
</li>
<li class="chapter " data-level="9.3" data-path="../appendix/material-share.html">
<a href="../appendix/material-share.html">
<b>9.3.</b>
Kubernetes 相关资讯和情报链接
</a>
</li>
<li class="chapter " data-level="9.4" data-path="../appendix/docker-best-practice.html">
<a href="../appendix/docker-best-practice.html">
<b>9.4.</b>
Docker 最佳实践
</a>
</li>
<li class="chapter " data-level="9.5" data-path="../appendix/tricks.html">
<a href="../appendix/tricks.html">
<b>9.5.</b>
使用技巧
</a>
</li>
<li class="chapter " data-level="9.6" data-path="../appendix/issues.html">
<a href="../appendix/issues.html">
<b>9.6.</b>
问题记录
</a>
</li>
<li class="chapter " data-level="9.7" data-path="../appendix/kubernetes-changelog.html">
<a href="../appendix/kubernetes-changelog.html">
<b>9.7.</b>
Kubernetes 版本更新日志
</a>
<ul class="articles">
<li class="chapter " data-level="9.7.1" data-path="../appendix/kubernetes-1.7-changelog.html">
<a href="../appendix/kubernetes-1.7-changelog.html">
<b>9.7.1.</b>
Kubernetes1.7 更新日志
</a>
</li>
<li class="chapter " data-level="9.7.2" data-path="../appendix/kubernetes-1.8-changelog.html">
<a href="../appendix/kubernetes-1.8-changelog.html">
<b>9.7.2.</b>
Kubernetes1.8 更新日志
</a>
</li>
<li class="chapter " data-level="9.7.3" data-path="../appendix/kubernetes-1.9-changelog.html">
<a href="../appendix/kubernetes-1.9-changelog.html">
<b>9.7.3.</b>
Kubernetes1.9 更新日志
</a>
</li>
<li class="chapter " data-level="9.7.4" data-path="../appendix/kubernetes-1.10-changelog.html">
<a href="../appendix/kubernetes-1.10-changelog.html">
<b>9.7.4.</b>
Kubernetes1.10 更新日志
</a>
</li>
<li class="chapter " data-level="9.7.5" data-path="../appendix/kubernetes-1.11-changelog.html">
<a href="../appendix/kubernetes-1.11-changelog.html">
<b>9.7.5.</b>
Kubernetes1.11 更新日志
</a>
</li>
<li class="chapter " data-level="9.7.6" data-path="../appendix/kubernetes-1.12-changelog.html">
<a href="../appendix/kubernetes-1.12-changelog.html">
<b>9.7.6.</b>
Kubernetes1.12 更新日志
</a>
</li>
<li class="chapter " data-level="9.7.7" data-path="../appendix/kubernetes-1.13-changelog.html">
<a href="../appendix/kubernetes-1.13-changelog.html">
<b>9.7.7.</b>
Kubernetes1.13 更新日志
</a>
</li>
<li class="chapter " data-level="9.7.8" data-path="../appendix/kubernetes-1.14-changelog.html">
<a href="../appendix/kubernetes-1.14-changelog.html">
<b>9.7.8.</b>
Kubernetes1.14 更新日志
</a>
</li>
<li class="chapter " data-level="9.7.9" data-path="../appendix/kubernetes-1.15-changelog.html">
<a href="../appendix/kubernetes-1.15-changelog.html">
<b>9.7.9.</b>
Kubernetes1.15 更新日志
</a>
</li>
<li class="chapter " data-level="9.7.10" data-path="../appendix/kubernetes-1.16-changelog.html">
<a href="../appendix/kubernetes-1.16-changelog.html">
<b>9.7.10.</b>
Kubernetes1.16 更新日志
</a>
</li>
<li class="chapter " data-level="9.7.11" data-path="../appendix/kubernetes-1.17-changelog.html">
<a href="../appendix/kubernetes-1.17-changelog.html">
<b>9.7.11.</b>
Kubernetes1.17 更新日志
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="9.8" data-path="../appendix/summary-and-outlook.html">
<a href="../appendix/summary-and-outlook.html">
<b>9.8.</b>
Kubernetes 及云原生年度总结及展望
</a>
<ul class="articles">
<li class="chapter " data-level="9.8.1" data-path="../appendix/kubernetes-and-cloud-native-summary-in-2017-and-outlook-for-2018.html">
<a href="../appendix/kubernetes-and-cloud-native-summary-in-2017-and-outlook-for-2018.html">
<b>9.8.1.</b>
Kubernetes 与云原生 2017 年年终总结及 2018 年展望
</a>
</li>
<li class="chapter " data-level="9.8.2" data-path="../appendix/kubernetes-and-cloud-native-summary-in-2018-and-outlook-for-2019.html">
<a href="../appendix/kubernetes-and-cloud-native-summary-in-2018-and-outlook-for-2019.html">
<b>9.8.2.</b>
Kubernetes 与云原生 2018 年年终总结及 2019 年展望
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="9.9" data-path="../appendix/cncf-annual-report.html">
<a href="../appendix/cncf-annual-report.html">
<b>9.9.</b>
CNCF 年度报告解读
</a>
<ul class="articles">
<li class="chapter " data-level="9.9.1" data-path="../appendix/cncf-annual-report-2018.html">
<a href="../appendix/cncf-annual-report-2018.html">
<b>9.9.1.</b>
CNCF 2018 年年度报告解读
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="9.10" data-path="../appendix/about-kcsp.html">
<a href="../appendix/about-kcsp.html">
<b>9.10.</b>
Kubernetes 认证服务提供商KCSP说明
</a>
</li>
<li class="chapter " data-level="9.11" data-path="../appendix/about-cka-candidate.html">
<a href="../appendix/about-cka-candidate.html">
<b>9.11.</b>
认证 Kubernetes 管理员CKA说明
</a>
</li>
<li class="divider"></li>
<li>
<a href="https://www.gitbook.com" target="blank" class="gitbook-link">
本书使用 GitBook 发布
</a>
</li>
</ul>
</nav>
</div>
<div class="book-body">
<div class="body-inner">
<div class="book-header" role="navigation">
<!-- Title -->
<h1>
<i class="fa fa-circle-o-notch fa-spin"></i>
<a href=".." >使用 Ceph 做持久化存储</a>
</h1>
</div>
<div class="page-wrapper" tabindex="-1" role="main">
<div class="page-inner">
<div class="search-plus" id="book-search-results">
<div class="search-noresults">
<section class="normal markdown-section">
<h1 id="&#x4F7F;&#x7528;ceph&#x505A;&#x6301;&#x4E45;&#x5316;&#x5B58;&#x50A8;&#x521B;&#x5EFA;mysql&#x96C6;&#x7FA4;">&#x4F7F;&#x7528;Ceph&#x505A;&#x6301;&#x4E45;&#x5316;&#x5B58;&#x50A8;&#x521B;&#x5EFA;MySQL&#x96C6;&#x7FA4;</h1>
<p>&#x672C;&#x6587;&#x4E2D;&#x7528;&#x5230;&#x7684; yaml &#x6587;&#x4EF6;&#x53EF;&#x4EE5;&#x5728; <a href="https://github.com/rootsongjc/kubernetes-handbook/blob/master/manifests/mariadb-cluster" target="_blank">../manifests/mariadb-cluster</a> &#x76EE;&#x5F55;&#x4E0B;&#x627E;&#x5230;&#x3002;</p>
<p>&#x4E0B;&#x9762;&#x6211;&#x4EEC;&#x4EE5;&#x90E8;&#x7F72;&#x4E00;&#x4E2A;&#x9AD8;&#x53EF;&#x7528;&#x7684; MySQL &#x96C6;&#x7FA4;&#x4E3A;&#x4F8B;&#xFF0C;&#x8BB2;&#x89E3;&#x5982;&#x4F55;&#x4F7F;&#x7528; Ceph &#x505A;&#x6570;&#x636E;&#x6301;&#x4E45;&#x5316;&#xFF0C;&#x5176;&#x4E2D;&#x4F7F;&#x7528; StorageClass &#x52A8;&#x6001;&#x521B;&#x5EFA; PV&#xFF0C;Ceph &#x96C6;&#x7FA4;&#x6211;&#x4EEC;&#x4F7F;&#x7528; kubernetes &#x96C6;&#x7FA4;&#x5916;&#x90E8;&#x7684;&#x5DF2;&#x6709;&#x7684;&#x96C6;&#x7FA4;&#xFF0C;&#x6211;&#x4EEC;&#x6CA1;&#x6709;&#x5FC5;&#x8981;&#x91CD;&#x65B0;&#x90E8;&#x7F72;&#x4E86;&#x3002;</p>
<p>&#x5728; 1.4 &#x4EE5;&#x540E;&#xFF0C;kubernetes &#x63D0;&#x4F9B;&#x4E86;&#x4E00;&#x79CD;&#x66F4;&#x52A0;&#x65B9;&#x4FBF;&#x7684;&#x52A8;&#x6001;&#x521B;&#x5EFA; PV &#x7684;&#x65B9;&#x5F0F;&#xFF1B;&#x4E5F;&#x5C31;&#x662F;&#x8BF4;&#x4F7F;&#x7528; StoragaClass &#x65F6;&#x65E0;&#x9700;&#x9884;&#x5148;&#x521B;&#x5EFA;&#x56FA;&#x5B9A;&#x5927;&#x5C0F;&#x7684; PV&#xFF0C;&#x7B49;&#x5F85;&#x4F7F;&#x7528;&#x8005;&#x521B;&#x5EFA; PVC &#x6765;&#x4F7F;&#x7528;&#xFF1B;&#x800C;&#x662F;&#x76F4;&#x63A5;&#x521B;&#x5EFA; PVC &#x5373;&#x53EF;&#x5206;&#x914D;&#x4F7F;&#x7528;&#x3002;</p>
<h2 id="&#x4F7F;&#x7528;-kubernetes-&#x96C6;&#x7FA4;&#x5916;&#x90E8;&#x7684;-ceph-&#x5B58;&#x50A8;">&#x4F7F;&#x7528; kubernetes &#x96C6;&#x7FA4;&#x5916;&#x90E8;&#x7684; Ceph &#x5B58;&#x50A8;</h2>
<p>&#x5728;&#x90E8;&#x7F72; kubernetes &#x4E4B;&#x524D;&#x6211;&#x4EEC;&#x5C31;&#x5DF2;&#x7ECF;&#x6709;&#x4E86; Ceph &#x96C6;&#x7FA4;&#xFF0C;&#x56E0;&#x6B64;&#x6211;&#x4EEC;&#x53EF;&#x4EE5;&#x76F4;&#x63A5;&#x62FF;&#x6765;&#x7528;&#x3002;&#x4F46;&#x662F; kubernetes &#x7684;&#x6240;&#x6709;&#x8282;&#x70B9;&#xFF08;&#x5C24;&#x5176;&#x662F; master &#x8282;&#x70B9;&#xFF09;&#x4E0A;&#x4F9D;&#x7136;&#x9700;&#x8981;&#x5B89;&#x88C5; ceph &#x5BA2;&#x6237;&#x7AEF;&#x3002;</p>
<pre class="language-"><code class="lang-bash">yum <span class="token function">install</span> -y ceph-common
</code></pre>
<p>&#x8FD8;&#x9700;&#x8981;&#x5C06; ceph &#x7684;&#x914D;&#x7F6E;&#x6587;&#x4EF6; <code>ceph.conf</code> &#x653E;&#x5728;&#x6240;&#x6709;&#x8282;&#x70B9;&#x7684; <code>/etc/ceph</code> &#x76EE;&#x5F55;&#x4E0B;&#x3002;</p>
<p>Kubernetes &#x4F7F;&#x7528; ceph &#x5B58;&#x50A8;&#x9700;&#x8981;&#x7528;&#x5230;&#x5982;&#x4E0B;&#x914D;&#x7F6E;&#xFF1A;</p>
<ul>
<li>Monitors: Ceph montors &#x5217;&#x8868;</li>
<li>Path&#xFF1A;&#x4F5C;&#x4E3A;&#x6302;&#x8F7D;&#x7684;&#x6839;&#x8DEF;&#x5F84;&#xFF0C;&#x9ED8;&#x8BA4;&#x662F; /</li>
<li>User&#xFF1A;RADOS&#x7528;&#x6237;&#x540D;&#xFF0C;&#x9ED8;&#x8BA4;&#x662F; admin</li>
<li>secretFile&#xFF1A;keyring &#x6587;&#x4EF6;&#x8DEF;&#x5F84;&#xFF0C;&#x9ED8;&#x8BA4;&#x662F; /etc/ceph/user.secret&#xFF0C;&#x6211;&#x4EEC; Ceph &#x96C6;&#x7FA4;&#x63D0;&#x4F9B;&#x7684;&#x6587;&#x4EF6;&#x662F; <code>ceph.client.admin.keyring</code>&#xFF0C;&#x5C06;&#x5728;&#x4E0B;&#x9762;&#x7528;&#x5230;</li>
<li>secretRef&#xFF1A;Ceph &#x8BA4;&#x8BC1; secret &#x7684;&#x5F15;&#x7528;&#xFF0C;&#x5982;&#x679C;&#x914D;&#x7F6E;&#x4E86;&#x5C06;&#x4F1A;&#x8986;&#x76D6; secretFile&#x3002;</li>
<li>readOnly&#xFF1A;&#x8BE5;&#x6587;&#x4EF6;&#x7CFB;&#x7EDF;&#x662F;&#x5426;&#x53EA;&#x8BFB;&#x3002;</li>
</ul>
<h2 id="galera-cluster&#x4ECB;&#x7ECD;">Galera Cluster&#x4ECB;&#x7ECD;</h2>
<p>Galera&#x662F;&#x4E00;&#x4E2A;MySQL(&#x4E5F;&#x652F;&#x6301;MariaDB&#xFF0C;Percona)&#x7684;&#x540C;&#x6B65;&#x591A;&#x4E3B;&#x96C6;&#x7FA4;&#x8F6F;&#x4EF6;&#x3002;</p>
<p>&#x4ECE;&#x7528;&#x6237;&#x89C6;&#x89D2;&#x770B;&#xFF0C;&#x4E00;&#x7EC4;Galera&#x96C6;&#x7FA4;&#x53EF;&#x4EE5;&#x770B;&#x4F5C;&#x4E00;&#x4E2A;&#x5177;&#x6709;&#x591A;&#x5165;&#x53E3;&#x7684;MySQL&#x5E93;&#xFF0C;&#x7528;&#x6237;&#x53EF;&#x4EE5;&#x540C;&#x65F6;&#x4ECE;&#x591A;&#x4E2A;IP&#x8BFB;&#x5199;&#x8FD9;&#x4E2A;&#x5E93;&#x3002;&#x76EE;&#x524D;Galera&#x5DF2;&#x7ECF;&#x5F97;&#x5230;&#x5E7F;&#x6CDB;&#x5E94;&#x7528;&#xFF0C;&#x4F8B;&#x5982;Openstack&#x4E2D;&#xFF0C;&#x5728;&#x96C6;&#x7FA4;&#x89C4;&#x6A21;&#x4E0D;&#x5927;&#x7684;&#x60C5;&#x51B5;&#x4E0B;&#xFF0C;&#x7A33;&#x5B9A;&#x6027;&#x5DF2;&#x7ECF;&#x5F97;&#x5230;&#x4E86;&#x5B9E;&#x8DF5;&#x8003;&#x9A8C;&#x3002;&#x771F;&#x6B63;&#x7684;multi-master&#xFF0C;&#x5373;&#x6240;&#x6709;&#x8282;&#x70B9;&#x53EF;&#x4EE5;&#x540C;&#x65F6;&#x8BFB;&#x5199;&#x6570;&#x636E;&#x5E93;&#x3002;</p>
<h2 id="&#x8BE6;&#x7EC6;&#x6B65;&#x9AA4;">&#x8BE6;&#x7EC6;&#x6B65;&#x9AA4;</h2>
<p>&#x4EE5;&#x4E0B;&#x6B65;&#x9AA4;&#x5305;&#x62EC;&#x521B;&#x5EFA; Ceph &#x7684;&#x914D;&#x7F6E; &#x548C; MySQL &#x7684;&#x914D;&#x7F6E;&#x4E24;&#x90E8;&#x5206;&#x3002;</p>
<h3 id="&#x914D;&#x7F6E;-ceph">&#x914D;&#x7F6E; Ceph</h3>
<p>&#x5173;&#x4E8E; Ceph &#x7684; yaml &#x6587;&#x4EF6;&#x53EF;&#x4EE5;&#x5728; <a href="https://github.com/rootsongjc/kubernetes-handbook/tree/master/manifests/mariadb-cluster" target="_blank">../manifest/mariadb-cluster</a> &#x76EE;&#x5F55;&#x4E0B;&#x627E;&#x5230;&#x3002;</p>
<h4 id="1-&#x751F;&#x6210;-ceph-secret">1. &#x751F;&#x6210; Ceph secret</h4>
<p>&#x4F7F;&#x7528; Ceph &#x7BA1;&#x7406;&#x5458;&#x63D0;&#x4F9B;&#x7ED9;&#x4F60;&#x7684; <code>ceph.client.admin.keyring</code> &#x6587;&#x4EF6;&#xFF0C;&#x6211;&#x4EEC;&#x5C06;&#x5B83;&#x653E;&#x5728;&#x4E86; <code>/etc/ceph</code> &#x76EE;&#x5F55;&#x4E0B;&#xFF0C;&#x7528;&#x6765;&#x751F;&#x6210; secret&#x3002;</p>
<pre class="language-"><code class="lang-bash"><span class="token function">grep</span> key /etc/ceph/ceph.client.admin.keyring <span class="token operator">|</span><span class="token function">awk</span> <span class="token string">&apos;{printf &quot;%s&quot;, <span class="token variable">$NF</span>}&apos;</span><span class="token operator">|</span>base64
</code></pre>
<p>&#x5C06;&#x83B7;&#x5F97;&#x52A0;&#x5BC6;&#x540E;&#x7684; key&#xFF1A;<code>QVFDWDA2aFo5TG5TQnhBQVl1b0lUL2V3YlRSaEtwVEhPWkxvUlE9PQ==</code>&#xFF0C;&#x6211;&#x4EEC;&#x5C06;&#x5728;&#x540E;&#x9762;&#x7528;&#x5230;&#x3002;</p>
<h4 id="2-&#x521B;&#x5EFA;&#x79DF;&#x6237;namespace">2. &#x521B;&#x5EFA;&#x79DF;&#x6237;namespace</h4>
<p>&#x521B;&#x5EFA; <code>galera-namespace.yaml</code> &#x6587;&#x4EF6;&#x5185;&#x5BB9;&#x4E3A;&#xFF1A;</p>
<pre class="language-"><code class="lang-yaml"><span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> v1
<span class="token key atrule">kind</span><span class="token punctuation">:</span> Namespace
<span class="token key atrule">metadata</span><span class="token punctuation">:</span>
<span class="token key atrule">name</span><span class="token punctuation">:</span> galera
</code></pre>
<h4 id="3-&#x521B;&#x5EFA;-ceph-secret">3. &#x521B;&#x5EFA; Ceph secret</h4>
<p>&#x521B;&#x5EFA; <code>ceph-secret.yaml</code> &#x6587;&#x4EF6;&#x5185;&#x5BB9;&#x4E3A;&#xFF1A;</p>
<pre class="language-"><code class="lang-yaml"><span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> v1
<span class="token key atrule">kind</span><span class="token punctuation">:</span> Secret
<span class="token key atrule">metadata</span><span class="token punctuation">:</span>
<span class="token key atrule">name</span><span class="token punctuation">:</span> ceph<span class="token punctuation">-</span>secret
<span class="token key atrule">namespace</span><span class="token punctuation">:</span> galera
<span class="token key atrule">type</span><span class="token punctuation">:</span> <span class="token string">&quot;kubernetes.io/rbd&quot;</span>
<span class="token key atrule">data</span><span class="token punctuation">:</span>
<span class="token key atrule">key</span><span class="token punctuation">:</span> QVFDWDA2aFo5TG5TQnhBQVl1b0lUL2V3YlRSaEtwVEhPWkxvUlE9PQ==
</code></pre>
<h4 id="4-&#x521B;&#x5EFA;-storageclass">4. &#x521B;&#x5EFA; StorageClass</h4>
<p>&#x521B;&#x5EFA; <code>ceph-class.yaml</code> &#x6587;&#x4EF6;&#x5185;&#x5BB9;&#x4E3A;&#xFF1A;</p>
<pre class="language-"><code class="lang-yaml"><span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> storage.k8s.io/v1
<span class="token key atrule">kind</span><span class="token punctuation">:</span> StorageClass
<span class="token key atrule">metadata</span><span class="token punctuation">:</span>
<span class="token key atrule">name</span><span class="token punctuation">:</span> ceph<span class="token punctuation">-</span>web
<span class="token key atrule">provisioner</span><span class="token punctuation">:</span> kubernetes.io/rbd
<span class="token key atrule">parameters</span><span class="token punctuation">:</span>
<span class="token key atrule">monitors</span><span class="token punctuation">:</span> 172.28.7.98<span class="token punctuation">,</span>172.28.7.99<span class="token punctuation">,</span>172.28.7.100
<span class="token key atrule">adminId</span><span class="token punctuation">:</span> admin
<span class="token key atrule">adminSecretName</span><span class="token punctuation">:</span> ceph<span class="token punctuation">-</span>secret
<span class="token key atrule">adminSecretNamespace</span><span class="token punctuation">:</span> galera
<span class="token key atrule">pool</span><span class="token punctuation">:</span> rbd <span class="token comment">#&#x6B64;&#x5904;&#x9ED8;&#x8BA4;&#x662F;rbd&#x6C60;&#xFF0C;&#x751F;&#x4EA7;&#x4E0A;&#x5EFA;&#x8BAE;&#x81EA;&#x5DF1;&#x521B;&#x5EFA;&#x5B58;&#x50A8;&#x6C60;&#x9694;&#x79BB;</span>
<span class="token key atrule">userId</span><span class="token punctuation">:</span> admin
<span class="token key atrule">userSecretName</span><span class="token punctuation">:</span> ceph<span class="token punctuation">-</span>secret
</code></pre>
<p>&#x6B64;&#x914D;&#x7F6E;&#x8BF7;&#x53C2;&#x8003; kubernetes &#x5B98;&#x65B9;&#x6587;&#x6863;&#xFF1A;<a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/#ceph-rbd" target="_blank">https://kubernetes.io/docs/concepts/storage/persistent-volumes/#ceph-rbd</a></p>
<h3 id="&#x914D;&#x7F6E;-mysql">&#x914D;&#x7F6E; MySQL</h3>
<h4 id="1-&#x521B;&#x5EFA;-mysql-&#x914D;&#x7F6E;&#x6587;&#x4EF6;">1. &#x521B;&#x5EFA; MySQL &#x914D;&#x7F6E;&#x6587;&#x4EF6;</h4>
<p>&#x521B;&#x5EFA; <code>mysql-config.yaml</code> &#x6587;&#x4EF6;&#x5185;&#x5BB9;&#x4E3A;&#xFF1A;</p>
<pre class="language-"><code class="lang-yaml"><span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> v1
<span class="token key atrule">kind</span><span class="token punctuation">:</span> ConfigMap
<span class="token key atrule">metadata</span><span class="token punctuation">:</span>
<span class="token key atrule">name</span><span class="token punctuation">:</span> mysql<span class="token punctuation">-</span>config<span class="token punctuation">-</span>vol
<span class="token key atrule">namespace</span><span class="token punctuation">:</span> galera
<span class="token key atrule">labels</span><span class="token punctuation">:</span>
<span class="token key atrule">app</span><span class="token punctuation">:</span> mysql
<span class="token key atrule">data</span><span class="token punctuation">:</span>
<span class="token key atrule">mariadb.cnf</span><span class="token punctuation">:</span> <span class="token punctuation">|</span><span class="token scalar string">
[client]
default-character-set = utf8
[mysqld]
character-set-server = utf8
collation-server = utf8_general_ci
# InnoDB optimizations
innodb_log_file_size = 64M</span>
<span class="token key atrule">galera.cnf</span><span class="token punctuation">:</span> <span class="token punctuation">|</span><span class="token scalar string">
[galera]
user = mysql
bind-address = 0.0.0.0
# Optimizations
innodb_flush_log_at_trx_commit = 0
sync_binlog = 0
expire_logs_days = 7
# Required settings
default_storage_engine = InnoDB
binlog_format = ROW
innodb_autoinc_lock_mode = 2
query_cache_size = 0
query_cache_type = 0
# MariaDB Galera settings
#wsrep_debug=ON
wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_sst_method=rsync
# Cluster settings (automatically updated)
wsrep_cluster_address=gcomm://
wsrep_cluster_name=galera
wsrep_node_address=127.0.0.1</span>
</code></pre>
<h4 id="2-&#x521B;&#x5EFA;-mysql-root-&#x7528;&#x6237;&#x548C;&#x5BC6;&#x7801;">2. &#x521B;&#x5EFA; MySQL root &#x7528;&#x6237;&#x548C;&#x5BC6;&#x7801;</h4>
<p><strong>&#x521B;&#x5EFA;&#x52A0;&#x5BC6;&#x5BC6;&#x7801;</strong></p>
<pre class="language-"><code class="lang-bash">$ <span class="token builtin class-name">echo</span> -n jimmysong<span class="token operator">|</span>base64
amltbXlzb25n
</code></pre>
<p>&#x6CE8;&#x610F;&#xFF1A;&#x4E00;&#x5B9A;&#x8981;&#x7528;-n &#x53BB;&#x6389;&#x6362;&#x884C;&#x7B26;&#xFF0C;&#x4E0D;&#x7136;&#x4F1A;&#x62A5;&#x9519;&#x3002;</p>
<p><strong>&#x521B;&#x5EFA; root &#x7528;&#x6237;</strong></p>
<pre class="language-"><code class="lang-bash">$ <span class="token builtin class-name">echo</span> -n root <span class="token operator">|</span>base64
<span class="token assign-left variable">cm9vdA</span><span class="token operator">==</span>
</code></pre>
<p><strong>&#x521B;&#x5EFA; MySQL secret</strong></p>
<p>&#x521B;&#x5EFA; <code>mysql-secret.yaml</code> &#x6587;&#x4EF6;&#x5185;&#x5BB9;&#x4E3A;&#xFF1A;</p>
<pre class="language-"><code class="lang-yaml"><span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> v1
<span class="token key atrule">kind</span><span class="token punctuation">:</span> Secret
<span class="token key atrule">metadata</span><span class="token punctuation">:</span>
<span class="token key atrule">name</span><span class="token punctuation">:</span> mysql<span class="token punctuation">-</span>secrets
<span class="token key atrule">namespace</span><span class="token punctuation">:</span> galera
<span class="token key atrule">labels</span><span class="token punctuation">:</span>
<span class="token key atrule">app</span><span class="token punctuation">:</span> mysql
<span class="token key atrule">data</span><span class="token punctuation">:</span>
<span class="token comment"># Root password: changeit run echo -n jimmysong|base64</span>
<span class="token key atrule">root-password</span><span class="token punctuation">:</span> amltbXlzb25n
<span class="token comment"># Root user: root</span>
<span class="token key atrule">root-user</span><span class="token punctuation">:</span> cm9vdA==
</code></pre>
<h4 id="3-&#x521B;&#x5EFA;-yaml-&#x914D;&#x7F6E;&#x6587;&#x4EF6;">3. &#x521B;&#x5EFA; yaml &#x914D;&#x7F6E;&#x6587;&#x4EF6;</h4>
<p>&#x521B;&#x5EFA; MySQL &#x7684; yaml &#x6587;&#x4EF6; <code>galera-mariadb.yaml</code> &#x5185;&#x5BB9;&#x4E3A;&#xFF1A;</p>
<pre class="language-"><code class="lang-yaml"><span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> v1
<span class="token key atrule">kind</span><span class="token punctuation">:</span> Service
<span class="token key atrule">metadata</span><span class="token punctuation">:</span>
<span class="token key atrule">annotations</span><span class="token punctuation">:</span>
<span class="token key atrule">service.alpha.kubernetes.io/tolerate-unready-endpoints</span><span class="token punctuation">:</span> <span class="token string">&quot;true&quot;</span>
<span class="token key atrule">name</span><span class="token punctuation">:</span> mysql
<span class="token key atrule">namespace</span><span class="token punctuation">:</span> galera
<span class="token key atrule">labels</span><span class="token punctuation">:</span>
<span class="token key atrule">app</span><span class="token punctuation">:</span> mysql
<span class="token key atrule">tier</span><span class="token punctuation">:</span> data
<span class="token key atrule">spec</span><span class="token punctuation">:</span>
<span class="token key atrule">ports</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> <span class="token key atrule">port</span><span class="token punctuation">:</span> <span class="token number">3306</span>
<span class="token key atrule">name</span><span class="token punctuation">:</span> mysql
<span class="token key atrule">clusterIP</span><span class="token punctuation">:</span> None
<span class="token key atrule">selector</span><span class="token punctuation">:</span>
<span class="token key atrule">app</span><span class="token punctuation">:</span> mysql
<span class="token punctuation">---</span>
<span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> apps/v1beta1
<span class="token key atrule">kind</span><span class="token punctuation">:</span> StatefulSet
<span class="token key atrule">metadata</span><span class="token punctuation">:</span>
<span class="token key atrule">name</span><span class="token punctuation">:</span> mysql
<span class="token key atrule">namespace</span><span class="token punctuation">:</span> galera
<span class="token key atrule">spec</span><span class="token punctuation">:</span>
<span class="token key atrule">serviceName</span><span class="token punctuation">:</span> <span class="token string">&quot;mysql&quot;</span>
<span class="token key atrule">replicas</span><span class="token punctuation">:</span> <span class="token number">3</span>
<span class="token key atrule">template</span><span class="token punctuation">:</span>
<span class="token key atrule">metadata</span><span class="token punctuation">:</span>
<span class="token key atrule">labels</span><span class="token punctuation">:</span>
<span class="token key atrule">app</span><span class="token punctuation">:</span> mysql
<span class="token key atrule">tier</span><span class="token punctuation">:</span> data
<span class="token key atrule">annotations</span><span class="token punctuation">:</span>
<span class="token key atrule">pod.beta.kubernetes.io/init-containers</span><span class="token punctuation">:</span> &apos;<span class="token punctuation">[</span>
<span class="token punctuation">{</span>
<span class="token key atrule">&quot;name&quot;</span><span class="token punctuation">:</span> <span class="token string">&quot;galera-init&quot;</span><span class="token punctuation">,</span>
<span class="token key atrule">&quot;image&quot;</span><span class="token punctuation">:</span> <span class="token string">&quot;harbor-001.jimmysong.io/library/k8s-galera-init:latest&quot;</span><span class="token punctuation">,</span>
<span class="token key atrule">&quot;args&quot;</span><span class="token punctuation">:</span> <span class="token punctuation">[</span><span class="token string">&quot;-service=mysql&quot;</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token key atrule">&quot;env&quot;</span><span class="token punctuation">:</span> <span class="token punctuation">[</span>
<span class="token punctuation">{</span>
<span class="token key atrule">&quot;name&quot;</span><span class="token punctuation">:</span> <span class="token string">&quot;POD_NAMESPACE&quot;</span><span class="token punctuation">,</span>
<span class="token key atrule">&quot;valueFrom&quot;</span><span class="token punctuation">:</span> <span class="token punctuation">{</span>
<span class="token key atrule">&quot;fieldRef&quot;</span><span class="token punctuation">:</span> <span class="token punctuation">{</span> <span class="token key atrule">&quot;apiVersion&quot;</span><span class="token punctuation">:</span> <span class="token string">&quot;v1&quot;</span><span class="token punctuation">,</span> <span class="token key atrule">&quot;fieldPath&quot;</span><span class="token punctuation">:</span> <span class="token string">&quot;metadata.namespace&quot;</span> <span class="token punctuation">}</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span>
<span class="token key atrule">&quot;name&quot;</span><span class="token punctuation">:</span> <span class="token string">&quot;SAFE_TO_BOOTSTRAP&quot;</span><span class="token punctuation">,</span>
<span class="token key atrule">&quot;value&quot;</span><span class="token punctuation">:</span> <span class="token string">&quot;1&quot;</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span>
<span class="token key atrule">&quot;name&quot;</span><span class="token punctuation">:</span> <span class="token string">&quot;DEBUG&quot;</span><span class="token punctuation">,</span>
<span class="token key atrule">&quot;value&quot;</span><span class="token punctuation">:</span> <span class="token string">&quot;1&quot;</span>
<span class="token punctuation">}</span>
<span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token key atrule">&quot;volumeMounts&quot;</span><span class="token punctuation">:</span> <span class="token punctuation">[</span>
<span class="token punctuation">{</span>
<span class="token key atrule">&quot;name&quot;</span><span class="token punctuation">:</span> <span class="token string">&quot;config&quot;</span><span class="token punctuation">,</span>
<span class="token key atrule">&quot;mountPath&quot;</span><span class="token punctuation">:</span> <span class="token string">&quot;/etc/mysql/conf.d&quot;</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span>
<span class="token key atrule">&quot;name&quot;</span><span class="token punctuation">:</span> <span class="token string">&quot;data&quot;</span><span class="token punctuation">,</span>
<span class="token key atrule">&quot;mountPath&quot;</span><span class="token punctuation">:</span> <span class="token string">&quot;/var/lib/mysql&quot;</span>
<span class="token punctuation">}</span>
<span class="token punctuation">]</span>
<span class="token punctuation">}</span>
<span class="token punctuation">]</span>&apos;
<span class="token key atrule">spec</span><span class="token punctuation">:</span>
<span class="token key atrule">terminationGracePeriodSeconds</span><span class="token punctuation">:</span> <span class="token number">10</span>
<span class="token key atrule">containers</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> mysql
<span class="token key atrule">image</span><span class="token punctuation">:</span> harbor<span class="token punctuation">-</span>001.jimmysong.io/library/mariadb<span class="token punctuation">:</span><span class="token number">10.1</span>
<span class="token key atrule">imagePullPolicy</span><span class="token punctuation">:</span> IfNotPresent
<span class="token key atrule">ports</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> <span class="token key atrule">containerPort</span><span class="token punctuation">:</span> <span class="token number">3306</span>
<span class="token key atrule">name</span><span class="token punctuation">:</span> mysql
<span class="token punctuation">-</span> <span class="token key atrule">containerPort</span><span class="token punctuation">:</span> <span class="token number">4444</span>
<span class="token key atrule">name</span><span class="token punctuation">:</span> sst
<span class="token punctuation">-</span> <span class="token key atrule">containerPort</span><span class="token punctuation">:</span> <span class="token number">4567</span>
<span class="token key atrule">name</span><span class="token punctuation">:</span> replication
<span class="token punctuation">-</span> <span class="token key atrule">containerPort</span><span class="token punctuation">:</span> <span class="token number">4568</span>
<span class="token key atrule">name</span><span class="token punctuation">:</span> ist
<span class="token key atrule">env</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> MYSQL_ROOT_PASSWORD
<span class="token key atrule">valueFrom</span><span class="token punctuation">:</span>
<span class="token key atrule">secretKeyRef</span><span class="token punctuation">:</span>
<span class="token key atrule">name</span><span class="token punctuation">:</span> mysql<span class="token punctuation">-</span>secrets
<span class="token key atrule">key</span><span class="token punctuation">:</span> root<span class="token punctuation">-</span>password
<span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> MYSQL_ROOT_USER
<span class="token key atrule">valueFrom</span><span class="token punctuation">:</span>
<span class="token key atrule">secretKeyRef</span><span class="token punctuation">:</span>
<span class="token key atrule">name</span><span class="token punctuation">:</span> mysql<span class="token punctuation">-</span>secrets
<span class="token key atrule">key</span><span class="token punctuation">:</span> root<span class="token punctuation">-</span>user
<span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> MYSQL_INITDB_SKIP_TZINFO
<span class="token key atrule">value</span><span class="token punctuation">:</span> <span class="token string">&quot;yes&quot;</span>
<span class="token key atrule">livenessProbe</span><span class="token punctuation">:</span>
<span class="token key atrule">exec</span><span class="token punctuation">:</span>
<span class="token key atrule">command</span><span class="token punctuation">:</span> <span class="token punctuation">[</span><span class="token string">&quot;sh&quot;</span><span class="token punctuation">,</span> <span class="token string">&quot;-c&quot;</span><span class="token punctuation">,</span> <span class="token string">&quot;mysql -u\&quot;${MYSQL_ROOT_USER:-root}\&quot; -p\&quot;${MYSQL_ROOT_PASSWORD}\&quot; -e &apos;show databases;&apos;&quot;</span><span class="token punctuation">]</span>
<span class="token key atrule">initialDelaySeconds</span><span class="token punctuation">:</span> <span class="token number">60</span>
<span class="token key atrule">timeoutSeconds</span><span class="token punctuation">:</span> <span class="token number">5</span>
<span class="token key atrule">readinessProbe</span><span class="token punctuation">:</span>
<span class="token key atrule">exec</span><span class="token punctuation">:</span>
<span class="token key atrule">command</span><span class="token punctuation">:</span> <span class="token punctuation">[</span><span class="token string">&quot;sh&quot;</span><span class="token punctuation">,</span> <span class="token string">&quot;-c&quot;</span><span class="token punctuation">,</span> <span class="token string">&quot;mysql -u\&quot;${MYSQL_ROOT_USER:-root}\&quot; -p\&quot;${MYSQL_ROOT_PASSWORD}\&quot; -e &apos;show databases;&apos;&quot;</span><span class="token punctuation">]</span>
<span class="token key atrule">initialDelaySeconds</span><span class="token punctuation">:</span> <span class="token number">20</span>
<span class="token key atrule">timeoutSeconds</span><span class="token punctuation">:</span> <span class="token number">5</span>
<span class="token key atrule">volumeMounts</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> config
<span class="token key atrule">mountPath</span><span class="token punctuation">:</span> /etc/mysql/conf.d
<span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> data
<span class="token key atrule">mountPath</span><span class="token punctuation">:</span> /var/lib/mysql
<span class="token key atrule">volumes</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> config
<span class="token key atrule">configMap</span><span class="token punctuation">:</span>
<span class="token key atrule">name</span><span class="token punctuation">:</span> mysql<span class="token punctuation">-</span>config<span class="token punctuation">-</span>vol
<span class="token key atrule">imagePullSecrets</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> <span class="token string">&quot;registrykey&quot;</span>
<span class="token key atrule">volumeClaimTemplates</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> <span class="token key atrule">metadata</span><span class="token punctuation">:</span>
<span class="token key atrule">name</span><span class="token punctuation">:</span> data
<span class="token key atrule">annotations</span><span class="token punctuation">:</span>
<span class="token key atrule">volume.beta.kubernetes.io/storage-class</span><span class="token punctuation">:</span> <span class="token string">&quot;ceph-web&quot;</span> <span class="token comment">#&#x5F15;&#x7528;ceph class &#x7684;&#x7C7B;</span>
<span class="token key atrule">spec</span><span class="token punctuation">:</span>
<span class="token key atrule">accessModes</span><span class="token punctuation">:</span> <span class="token punctuation">[</span> <span class="token string">&quot;ReadWriteOnce&quot;</span> <span class="token punctuation">]</span>
<span class="token key atrule">resources</span><span class="token punctuation">:</span>
<span class="token key atrule">requests</span><span class="token punctuation">:</span>
<span class="token key atrule">storage</span><span class="token punctuation">:</span> 3Gi
</code></pre>
<h2 id="&#x90E8;&#x7F72;-mysql-&#x96C6;&#x7FA4;">&#x90E8;&#x7F72; MySQL &#x96C6;&#x7FA4;</h2>
<p>&#x5728; <code>/etc/mariadb-cluster</code> &#x76EE;&#x5F55;&#x4E0B;&#x6267;&#x884C;&#xFF1A;</p>
<pre class="language-"><code class="lang-bash">kubectl create -f <span class="token builtin class-name">.</span>
</code></pre>
<h2 id="&#x9A8C;&#x8BC1;">&#x9A8C;&#x8BC1;</h2>
<p>&#x5B58;&#x5728; issue&#xFF0C;&#x53C2;&#x8003; <a href="https://github.com/kubernetes/kubernetes/issues/38923" target="_blank">Error creating rbd image: executable file not found in $PATH#38923</a></p>
<h2 id="&#x95EE;&#x9898;&#x8BB0;&#x5F55;">&#x95EE;&#x9898;&#x8BB0;&#x5F55;</h2>
<p>&#x5982;&#x679C;&#x6CA1;&#x6709;&#x5B89;&#x88C5; <code>ceph-common</code> &#x7684;&#x8BDD;&#xFF0C;kubernetes &#x5728;&#x521B;&#x5EFA; PVC &#x7684;&#x65F6;&#x5019;&#x4F1A;&#x6709;&#x5982;&#x4E0B;&#x62A5;&#x9519;&#x4FE1;&#x606F;&#xFF1A;</p>
<pre class="language-"><code class="lang-bash">Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
1h 12s <span class="token number">441</span> <span class="token punctuation">{</span>persistentvolume-controller <span class="token punctuation">}</span> Warning ProvisioningFailed Failed to provision volume with StorageClass <span class="token string">&quot;ceph-web&quot;</span><span class="token builtin class-name">:</span> failed to create rbd image: executable <span class="token function">file</span> not found <span class="token keyword">in</span> <span class="token environment constant">$PATH</span>, <span class="token builtin class-name">command</span> output:
</code></pre>
<p>&#x68C0;&#x67E5; <code>kube-controller-manager</code> &#x7684;&#x65E5;&#x5FD7;&#x5C06;&#x770B;&#x5230;&#x5982;&#x4E0B;&#x9519;&#x8BEF;&#x4FE1;&#x606F;&#xFF1A;</p>
<pre class="language-"><code class="lang-bash">journalctl -xe -u kube-controller-manager
<span class="token punctuation">..</span>. rbd_util.go:364<span class="token punctuation">]</span> failed to create rbd image, output
<span class="token punctuation">..</span>. rbd.go:317<span class="token punctuation">]</span> rbd: create volume failed, err: failed to create rbd image: executable <span class="token function">file</span> not found <span class="token keyword">in</span> <span class="token environment constant">$PATH</span>, <span class="token builtin class-name">command</span> output:
</code></pre>
<p>&#x8FD9;&#x662F;&#x56E0;&#x4E3A; <code>kube-controller-manager</code> &#x4E3B;&#x673A;&#x4E0A;&#x6CA1;&#x6709;&#x5B89;&#x88C5; <code>ceph-common</code> &#x7684;&#x7F18;&#x6545;&#x3002;</p>
<p>&#x4F46;&#x662F;&#x5B89;&#x88C5;&#x4E86; <code>ceph-common</code> &#x4E4B;&#x540E;&#x4F9D;&#x7136;&#x6709;&#x95EE;&#x9898;&#xFF1A;</p>
<pre class="language-"><code class="lang-bash">Sep <span class="token number">4</span> <span class="token number">15</span>:25:36 bj-xg-oam-kubernetes-001 kube-controller-manager: W0904 <span class="token number">15</span>:25:36.032128 <span class="token number">13211</span> rbd_util.go:364<span class="token punctuation">]</span> failed to create rbd image, output
Sep <span class="token number">4</span> <span class="token number">15</span>:25:36 bj-xg-oam-kubernetes-001 kube-controller-manager: W0904 <span class="token number">15</span>:25:36.032201 <span class="token number">13211</span> rbd_util.go:364<span class="token punctuation">]</span> failed to create rbd image, output
Sep <span class="token number">4</span> <span class="token number">15</span>:25:36 bj-xg-oam-kubernetes-001 kube-controller-manager: W0904 <span class="token number">15</span>:25:36.032252 <span class="token number">13211</span> rbd_util.go:364<span class="token punctuation">]</span> failed to create rbd image, output
Sep <span class="token number">4</span> <span class="token number">15</span>:25:36 bj-xg-oam-kubernetes-001 kube-controller-manager: E0904 <span class="token number">15</span>:25:36.032276 <span class="token number">13211</span> rbd.go:317<span class="token punctuation">]</span> rbd: create volume failed, err: failed to create rbd image: fork/exec /usr/bin/rbd: invalid argument, <span class="token builtin class-name">command</span> output:
</code></pre>
<p>&#x8BE5;&#x95EE;&#x9898;&#x5C1A;&#x672A;&#x89E3;&#x51B3;&#xFF0C;&#x53C2;&#x8003; <a href="https://github.com/kubernetes/kubernetes/issues/38923" target="_blank">Error creating rbd image: executable file not found in $PATH#38923</a></p>
<p>&#x4ECE;&#x65E5;&#x5FD7;&#x8BB0;&#x5F55;&#x6765;&#x770B;&#x8FFD;&#x67E5;&#x5230; <code>pkg/volume/rbd/rbd.go</code> &#x7684; <code>func (r *rbdVolumeProvisioner) Provision() (*v1.PersistentVolume, error) {</code> &#x65B9;&#x6CD5;&#x5BF9; <code>ceph-class.yaml</code> &#x4E2D;&#x7684;&#x53C2;&#x6570;&#x8FDB;&#x884C;&#x4E86;&#x9A8C;&#x8BC1;&#x548C;&#x5904;&#x7406;&#x540E;&#x8C03;&#x7528;&#x4E86; <code>pkg/volume/rbd/rdb_utils.go</code> &#x6587;&#x4EF6;&#x7B2C; 344 &#x884C; <code>CreateImage</code> &#x65B9;&#x6CD5;&#xFF08;kubernetes v1.6.1&#x7248;&#x672C;&#xFF09;&#xFF1A;</p>
<pre class="language-"><code class="lang-go"><span class="token keyword">func</span> <span class="token punctuation">(</span>util <span class="token operator">*</span>RBDUtil<span class="token punctuation">)</span> <span class="token function">CreateImage</span><span class="token punctuation">(</span>p <span class="token operator">*</span>rbdVolumeProvisioner<span class="token punctuation">)</span> <span class="token punctuation">(</span>r <span class="token operator">*</span>v1<span class="token punctuation">.</span>RBDVolumeSource<span class="token punctuation">,</span> size <span class="token builtin">int</span><span class="token punctuation">,</span> err <span class="token builtin">error</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">var</span> output <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token builtin">byte</span>
capacity <span class="token operator">:=</span> p<span class="token punctuation">.</span>options<span class="token punctuation">.</span>PVC<span class="token punctuation">.</span>Spec<span class="token punctuation">.</span>Resources<span class="token punctuation">.</span>Requests<span class="token punctuation">[</span>v1<span class="token punctuation">.</span><span class="token function">ResourceName</span><span class="token punctuation">(</span>v1<span class="token punctuation">.</span>ResourceStorage<span class="token punctuation">)</span><span class="token punctuation">]</span>
volSizeBytes <span class="token operator">:=</span> capacity<span class="token punctuation">.</span><span class="token function">Value</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token comment">// convert to MB that rbd defaults on</span>
sz <span class="token operator">:=</span> <span class="token function">int</span><span class="token punctuation">(</span>volume<span class="token punctuation">.</span><span class="token function">RoundUpSize</span><span class="token punctuation">(</span>volSizeBytes<span class="token punctuation">,</span> <span class="token number">1024</span><span class="token operator">*</span><span class="token number">1024</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
volSz <span class="token operator">:=</span> fmt<span class="token punctuation">.</span><span class="token function">Sprintf</span><span class="token punctuation">(</span><span class="token string">&quot;%d&quot;</span><span class="token punctuation">,</span> sz<span class="token punctuation">)</span>
<span class="token comment">// rbd create</span>
l <span class="token operator">:=</span> <span class="token function">len</span><span class="token punctuation">(</span>p<span class="token punctuation">.</span>rbdMounter<span class="token punctuation">.</span>Mon<span class="token punctuation">)</span>
<span class="token comment">// pick a mon randomly</span>
start <span class="token operator">:=</span> rand<span class="token punctuation">.</span><span class="token function">Int</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">%</span> l
<span class="token comment">// iterate all monitors until create succeeds.</span>
<span class="token keyword">for</span> i <span class="token operator">:=</span> start<span class="token punctuation">;</span> i <span class="token operator">&lt;</span> start<span class="token operator">+</span>l<span class="token punctuation">;</span> i<span class="token operator">++</span> <span class="token punctuation">{</span>
mon <span class="token operator">:=</span> p<span class="token punctuation">.</span>Mon<span class="token punctuation">[</span>i<span class="token operator">%</span>l<span class="token punctuation">]</span>
glog<span class="token punctuation">.</span><span class="token function">V</span><span class="token punctuation">(</span><span class="token number">4</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">Infof</span><span class="token punctuation">(</span><span class="token string">&quot;rbd: create %s size %s using mon %s, pool %s id %s key %s&quot;</span><span class="token punctuation">,</span> p<span class="token punctuation">.</span>rbdMounter<span class="token punctuation">.</span>Image<span class="token punctuation">,</span> volSz<span class="token punctuation">,</span> mon<span class="token punctuation">,</span> p<span class="token punctuation">.</span>rbdMounter<span class="token punctuation">.</span>Pool<span class="token punctuation">,</span> p<span class="token punctuation">.</span>rbdMounter<span class="token punctuation">.</span>adminId<span class="token punctuation">,</span> p<span class="token punctuation">.</span>rbdMounter<span class="token punctuation">.</span>adminSecret<span class="token punctuation">)</span>
output<span class="token punctuation">,</span> err <span class="token operator">=</span> p<span class="token punctuation">.</span>rbdMounter<span class="token punctuation">.</span>plugin<span class="token punctuation">.</span><span class="token function">execCommand</span><span class="token punctuation">(</span><span class="token string">&quot;rbd&quot;</span><span class="token punctuation">,</span>
<span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token builtin">string</span><span class="token punctuation">{</span><span class="token string">&quot;create&quot;</span><span class="token punctuation">,</span> p<span class="token punctuation">.</span>rbdMounter<span class="token punctuation">.</span>Image<span class="token punctuation">,</span> <span class="token string">&quot;--size&quot;</span><span class="token punctuation">,</span> volSz<span class="token punctuation">,</span> <span class="token string">&quot;--pool&quot;</span><span class="token punctuation">,</span> p<span class="token punctuation">.</span>rbdMounter<span class="token punctuation">.</span>Pool<span class="token punctuation">,</span> <span class="token string">&quot;--id&quot;</span><span class="token punctuation">,</span> p<span class="token punctuation">.</span>rbdMounter<span class="token punctuation">.</span>adminId<span class="token punctuation">,</span> <span class="token string">&quot;-m&quot;</span><span class="token punctuation">,</span> mon<span class="token punctuation">,</span> <span class="token string">&quot;--key=&quot;</span> <span class="token operator">+</span> p<span class="token punctuation">.</span>rbdMounter<span class="token punctuation">.</span>adminSecret<span class="token punctuation">,</span> <span class="token string">&quot;--image-format&quot;</span><span class="token punctuation">,</span> <span class="token string">&quot;1&quot;</span><span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> err <span class="token operator">==</span> <span class="token boolean">nil</span> <span class="token punctuation">{</span>
<span class="token keyword">break</span>
<span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{</span>
glog<span class="token punctuation">.</span><span class="token function">Warningf</span><span class="token punctuation">(</span><span class="token string">&quot;failed to create rbd image, output %v&quot;</span><span class="token punctuation">,</span> <span class="token function">string</span><span class="token punctuation">(</span>output<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span>
<span class="token keyword">if</span> err <span class="token operator">!=</span> <span class="token boolean">nil</span> <span class="token punctuation">{</span>
<span class="token keyword">return</span> <span class="token boolean">nil</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">,</span> fmt<span class="token punctuation">.</span><span class="token function">Errorf</span><span class="token punctuation">(</span><span class="token string">&quot;failed to create rbd image: %v, command output: %s&quot;</span><span class="token punctuation">,</span> err<span class="token punctuation">,</span> <span class="token function">string</span><span class="token punctuation">(</span>output<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token punctuation">}</span>
<span class="token keyword">return</span> <span class="token operator">&amp;</span>v1<span class="token punctuation">.</span>RBDVolumeSource<span class="token punctuation">{</span>
CephMonitors<span class="token punctuation">:</span> p<span class="token punctuation">.</span>rbdMounter<span class="token punctuation">.</span>Mon<span class="token punctuation">,</span>
RBDImage<span class="token punctuation">:</span> p<span class="token punctuation">.</span>rbdMounter<span class="token punctuation">.</span>Image<span class="token punctuation">,</span>
RBDPool<span class="token punctuation">:</span> p<span class="token punctuation">.</span>rbdMounter<span class="token punctuation">.</span>Pool<span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span> sz<span class="token punctuation">,</span> <span class="token boolean">nil</span>
<span class="token punctuation">}</span>
</code></pre>
<p>&#x8BE5;&#x65B9;&#x6CD5;&#x8C03;&#x7528;&#x5931;&#x8D25;&#x3002;</p>
<h2 id="&#x53C2;&#x8003;">&#x53C2;&#x8003;</h2>
<ul>
<li><a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/#ceph-rbd" target="_blank">https://kubernetes.io/docs/concepts/storage/persistent-volumes/#ceph-rbd</a></li>
<li><a href="https://github.com/kubernetes/kubernetes/issues/38923" target="_blank">Error creating rbd image: executable file not found in $PATH#38923</a></li>
</ul>
<footer class="page-footer"><span class="copyright"><a href="https://mp.weixin.qq.com/s/vWlSdzz2MNdXRr0sd2-LFg" target="_blank">&#x52A0;&#x5165;&#x4E91;&#x539F;&#x751F;&#x793E;&#x533A;</a><p></p>Copyright &#xA9; 2017-2020 | Distributed under <a href="https://creativecommons.org/licenses/by-nc-sa/4.0/deed.zh" target="_blank">CC BY 4.0</a> | <a href="https://jimmysong.io" target="_blank">jimmysong.io</a> all right reserved&#xFF0C;powered by Gitbook</span><span class="footer-modification"> Updated at
2020-12-25 13:14:22
</span></footer>
</section>
</div>
<div class="search-results">
<div class="has-results">
<h1 class="search-results-title"><span class='search-results-count'></span> results matching "<span class='search-query'></span>"</h1>
<ul class="search-results-list"></ul>
</div>
<div class="no-results">
<h1 class="search-results-title">No results matching "<span class='search-query'></span>"</h1>
</div>
</div>
</div>
</div>
</div>
</div>
<a href="ceph-helm-install-guide-zh.html" class="navigation navigation-prev " aria-label="Previous page: 用 Helm 托管安装 Ceph 集群并提供后端存储">
<i class="fa fa-angle-left"></i>
</a>
<a href="rbd-provisioner.html" class="navigation navigation-next " aria-label="Next page: 使用 rbd-provisioner 提供 rbd 持久化存储">
<i class="fa fa-angle-right"></i>
</a>
</div>
<script>
var gitbook = gitbook || [];
gitbook.push(function() {
gitbook.page.hasChanged({"page":{"title":"使用 Ceph 做持久化存储","level":"5.6.3.2","depth":3,"next":{"title":"使用 rbd-provisioner 提供 rbd 持久化存储","level":"5.6.3.3","depth":3,"path":"practice/rbd-provisioner.md","ref":"practice/rbd-provisioner.md","articles":[]},"previous":{"title":"用 Helm 托管安装 Ceph 集群并提供后端存储","level":"5.6.3.1","depth":3,"path":"practice/ceph-helm-install-guide-zh.md","ref":"practice/ceph-helm-install-guide-zh.md","articles":[]},"dir":"ltr"},"config":{"plugins":["github","codesnippet","splitter","page-toc-button","image-captions","editlink","back-to-top-button","-lunr","-search","search-plus","github-buttons@2.1.0","favicon@^0.0.2","tbfed-pagefooter@^0.0.1","3-ba","theme-default","-highlight","prism","prism-themes","sitemap-general","lightbox","ga","alerts"],"styles":{"ebook":"styles/ebook.css","epub":"styles/epub.css","mobi":"styles/mobi.css","pdf":"styles/pdf.css","print":"styles/print.css","website":"styles/website.css"},"pluginsConfig":{"tbfed-pagefooter":{"copyright":"<a href=https://mp.weixin.qq.com/s/vWlSdzz2MNdXRr0sd2-LFg>加入云原生社区</a></p>Copyright © 2017-2020 | Distributed under <a href=https://creativecommons.org/licenses/by-nc-sa/4.0/deed.zh>CC BY 4.0</a> | <a href=https://jimmysong.io>jimmysong.io</a>","modify_label":" Updated at ","modify_format":"YYYY-MM-DD HH:mm:ss"},"prism":{"css":["prism-themes/themes/prism-ghcolors.css"]},"github":{"url":"https://github.com/rootsongjc/kubernetes-handbook"},"editlink":{"label":"编辑本页","multilingual":false,"base":"https://github.com/rootsongjc/kubernetes-handbook/blob/master/"},"splitter":{},"codesnippet":{},"sitemap-general":{"prefix":"https://jimmysong.io/kubernetes-handbook/"},"fontsettings":{"theme":"white","family":"sans","size":2},"favicon":{"shortcut":"favicon.ico","bookmark":"favicon.ico"},"lightbox":{"jquery":true,"sameUuid":false},"page-toc-button":{},"back-to-top-button":{},"prism-themes":{},"alerts":{},"github-buttons":{"repo":"rootsongjc/kubernetes-handbook","types":["star"],"size":"small"},"3-ba":{"configuration":"auto","token":"11f7d254cfa4e0ca44b175c66d379ecc"},"ga":{"configuration":"auto","token":"UA-93485976-1"},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"theme-default":{"showLevel":true,"styles":{"ebook":"styles/ebook.css","epub":"styles/epub.css","mobi":"styles/mobi.css","pdf":"styles/pdf.css","print":"styles/print.css","website":"styles/website.css"}},"search-plus":{},"image-captions":{"caption":"图 _PAGE_LEVEL_._PAGE_IMAGE_NUMBER__CAPTION_","variable_name":"_pictures"}},"theme":"default","author":"Jimmy Song宋净超","pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"variables":{"_pictures":[{"backlink":"index.html#fig1.1.1","level":"1.1","list_caption":"Figure: Stargazers over time","alt":"Stargazers over time","nro":1,"url":"https://starcharts.herokuapp.com/rootsongjc/kubernetes-handbook.svg","index":1,"caption_template":"图 _PAGE_LEVEL_._PAGE_IMAGE_NUMBER__CAPTION_","label":"Stargazers over time","attributes":{},"skip":false,"key":"1.1.1"},{"backlink":"cloud-native/play-with-kubernetes.html#fig2.4.1","level":"2.4","list_caption":"Figure: Play with Kubernetes网页截图","alt":"Play with Kubernetes网页截图","nro":2,"url":"../images/play-with-kubernetes.jpg","index":1,"caption_template":"图 _PAGE_LEVEL_._PAGE_IMAGE_NUMBER__CAPTION_","label":"Play with Kubernetes网页截图","attributes":{},"skip":false,"key":"2.4.1"},{"backlink":"cloud-native/cloud-native-local-quick-start.html#fig2.5.1","level":"2.5","list_caption":"Figure: Kubernetes dashboard","alt":"Kubernetes dashboard","nro":3,"url":"https://github.com/rootsongjc/kubernetes-vagrant-centos-cluster/raw/master/images/dashboard-a
});
</script>
</div>
<script src="../gitbook/gitbook.js"></script>
<script src="../gitbook/theme.js"></script>
<script src="../gitbook/gitbook-plugin-github/plugin.js"></script>
<script src="../gitbook/gitbook-plugin-splitter/splitter.js"></script>
<script src="../gitbook/gitbook-plugin-page-toc-button/plugin.js"></script>
<script src="../gitbook/gitbook-plugin-editlink/plugin.js"></script>
<script src="../gitbook/gitbook-plugin-back-to-top-button/plugin.js"></script>
<script src="../gitbook/gitbook-plugin-search-plus/jquery.mark.min.js"></script>
<script src="../gitbook/gitbook-plugin-search-plus/search.js"></script>
<script src="../gitbook/gitbook-plugin-github-buttons/plugin.js"></script>
<script src="../gitbook/gitbook-plugin-3-ba/plugin.js"></script>
<script src="../gitbook/gitbook-plugin-lightbox/js/lightbox.min.js"></script>
<script src="../gitbook/gitbook-plugin-ga/plugin.js"></script>
<script src="../gitbook/gitbook-plugin-alerts/plugin.js"></script>
<script src="../gitbook/gitbook-plugin-sharing/buttons.js"></script>
<script src="../gitbook/gitbook-plugin-fontsettings/fontsettings.js"></script>
</body>
</html>