2017-08-21 18:44:34 +08:00
<!DOCTYPE HTML>
2017-09-19 22:01:07 +08:00
< html lang = "zh-hans" >
2017-08-21 18:44:34 +08:00
< head >
< meta charset = "UTF-8" >
< meta content = "text/html; charset=utf-8" http-equiv = "Content-Type" >
< title > 3.5.2 迁移传统应用到kubernetes中——以Hadoop YARN为例 · Kubernetes Handbook< / title >
< meta http-equiv = "X-UA-Compatible" content = "IE=edge" / >
< meta name = "description" content = "" >
< meta name = "generator" content = "GitBook 3.2.2" >
< meta name = "author" content = "Jimmy Song" >
< link rel = "stylesheet" href = "../gitbook/style.css" >
< link rel = "stylesheet" href = "../gitbook/gitbook-plugin-splitter/splitter.css" >
< link rel = "stylesheet" href = "../gitbook/gitbook-plugin-page-toc-button/plugin.css" >
< link rel = "stylesheet" href = "../gitbook/gitbook-plugin-image-captions/image-captions.css" >
2017-10-09 15:28:05 +08:00
< link rel = "stylesheet" href = "../gitbook/gitbook-plugin-back-to-top-button/plugin.css" >
2017-08-21 18:44:34 +08:00
2017-10-09 15:28:05 +08:00
< link rel = "stylesheet" href = "../gitbook/gitbook-plugin-search-plus/search.css" >
2017-09-19 21:38:03 +08:00
2017-10-09 15:28:05 +08:00
< link rel = "stylesheet" href = "../gitbook/gitbook-plugin-tbfed-pagefooter/footer.css" >
2017-08-21 18:44:34 +08:00
< link rel = "stylesheet" href = "../gitbook/gitbook-plugin-highlight/website.css" >
< link rel = "stylesheet" href = "../gitbook/gitbook-plugin-fontsettings/website.css" >
2017-10-09 15:28:05 +08:00
2017-08-21 18:44:34 +08:00
< meta name = "HandheldFriendly" content = "true" / >
< meta name = "viewport" content = "width=device-width, initial-scale=1, user-scalable=no" >
< meta name = "apple-mobile-web-app-capable" content = "yes" >
< meta name = "apple-mobile-web-app-status-bar-style" content = "black" >
< link rel = "apple-touch-icon-precomposed" sizes = "152x152" href = "../gitbook/images/apple-touch-icon-precomposed-152.png" >
< link rel = "shortcut icon" href = "../gitbook/images/favicon.ico" type = "image/x-icon" >
2017-10-23 13:40:58 +08:00
< link rel = "next" href = "using-statefulset.html" / >
2017-08-21 18:44:34 +08:00
< link rel = "prev" href = "deploy-applications-in-kubernetes.html" / >
2017-10-09 15:28:05 +08:00
< link rel = "shortcut icon" href = '../favicon.ico' type = "image/x-icon" >
< link rel = "bookmark" href = '../favicon.ico' type = "image/x-icon" >
< style >
@media only screen and (max-width: 640px) {
.book-header .hidden-mobile {
display: none;
}
}
< / style >
< script >
window["gitbook-plugin-github-buttons"] = {"repo":"rootsongjc/kubernetes-handbook","types":["star"],"size":"small"};
< / script >
2017-08-21 18:44:34 +08:00
< / head >
< body >
< div class = "book" >
< div class = "book-summary" >
< div id = "book-search-input" role = "search" >
2017-09-19 22:01:07 +08:00
< input type = "text" placeholder = "输入并搜索" / >
2017-08-21 18:44:34 +08:00
< / div >
< nav role = "navigation" >
< ul class = "summary" >
2017-10-09 15:28:05 +08:00
< li >
< a href = "https://jimmysong.io" target = "_blank" class = "custom-link" > Home< / a >
< / li >
2017-08-21 18:44:34 +08:00
2017-10-09 15:28:05 +08:00
< li class = "divider" > < / li >
2017-08-21 18:44:34 +08:00
< li class = "chapter " data-level = "1.1" data-path = "../" >
< a href = "../" >
2017-10-20 10:51:14 +08:00
序言
2017-08-21 18:44:34 +08:00
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.2" data-path = "../cloud-native/kubernetes-and-cloud-native-app-overview.html" >
< a href = "../cloud-native/kubernetes-and-cloud-native-app-overview.html" >
1. Kubernetes与云原生应用概览
< / a >
< / li >
< li class = "chapter " data-level = "1.3" data-path = "../concepts/" >
2017-08-21 18:44:34 +08:00
< a href = "../concepts/" >
2. 概念原理
< / a >
< ul class = "articles" >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.1" data-path = "../concepts/concepts.html" >
2017-08-21 18:44:34 +08:00
< a href = "../concepts/concepts.html" >
2.1 设计理念
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.2" data-path = "../concepts/objects.html" >
2017-08-21 18:44:34 +08:00
< a href = "../concepts/objects.html" >
2017-09-03 13:29:38 +08:00
2.2 Objects
2017-08-21 18:44:34 +08:00
< / a >
< ul class = "articles" >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.2.1" data-path = "../concepts/pod-overview.html" >
2017-08-21 18:44:34 +08:00
< a href = "../concepts/pod-overview.html" >
2.2.1 Pod
< / a >
< ul class = "articles" >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.2.1.1" data-path = "../concepts/pod.html" >
2017-08-21 18:44:34 +08:00
< a href = "../concepts/pod.html" >
2.2.1.1 Pod解析
< / a >
2017-08-31 23:28:33 +08:00
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.2.1.2" data-path = "../concepts/init-containers.html" >
2017-08-31 23:28:33 +08:00
< a href = "../concepts/init-containers.html" >
2.2.1.2 Init容器
< / a >
2017-09-03 15:58:39 +08:00
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.2.1.3" data-path = "../concepts/pod-security-policy.html" >
2017-09-03 15:58:39 +08:00
< a href = "../concepts/pod-security-policy.html" >
2.2.1.3 Pod安全策略
< / a >
2017-09-17 15:39:26 +08:00
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.2.1.4" data-path = "../concepts/pod-lifecycle.html" >
2017-09-17 15:39:26 +08:00
< a href = "../concepts/pod-lifecycle.html" >
2.2.1.4 Pod的生命周期
< / a >
2017-08-21 18:44:34 +08:00
< / li >
< / ul >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.2.2" data-path = "../concepts/node.html" >
2017-08-21 18:44:34 +08:00
< a href = "../concepts/node.html" >
2.2.2 Node
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.2.3" data-path = "../concepts/namespace.html" >
2017-08-21 18:44:34 +08:00
< a href = "../concepts/namespace.html" >
2.2.3 Namespace
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.2.4" data-path = "../concepts/service.html" >
2017-08-21 18:44:34 +08:00
< a href = "../concepts/service.html" >
2.2.4 Service
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.2.5" data-path = "../concepts/volume.html" >
2017-08-21 18:44:34 +08:00
< a href = "../concepts/volume.html" >
2.2.5 Volume和Persistent Volume
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.2.6" data-path = "../concepts/deployment.html" >
2017-08-21 18:44:34 +08:00
< a href = "../concepts/deployment.html" >
2.2.6 Deployment
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.2.7" data-path = "../concepts/secret.html" >
2017-08-21 18:44:34 +08:00
< a href = "../concepts/secret.html" >
2.2.7 Secret
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.2.8" data-path = "../concepts/statefulset.html" >
2017-08-21 18:44:34 +08:00
< a href = "../concepts/statefulset.html" >
2.2.8 StatefulSet
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.2.9" data-path = "../concepts/daemonset.html" >
2017-08-21 18:44:34 +08:00
< a href = "../concepts/daemonset.html" >
2.2.9 DaemonSet
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.2.10" data-path = "../concepts/serviceaccount.html" >
2017-08-21 18:44:34 +08:00
< a href = "../concepts/serviceaccount.html" >
2.2.10 ServiceAccount
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.2.11" data-path = "../concepts/replicaset.html" >
2017-08-21 18:44:34 +08:00
< a href = "../concepts/replicaset.html" >
2.2.11 ReplicationController和ReplicaSet
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.2.12" data-path = "../concepts/job.html" >
2017-08-21 18:44:34 +08:00
< a href = "../concepts/job.html" >
2.2.12 Job
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.2.13" data-path = "../concepts/cronjob.html" >
2017-08-21 18:44:34 +08:00
< a href = "../concepts/cronjob.html" >
2.2.13 CronJob
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.2.14" data-path = "../concepts/ingress.html" >
2017-08-21 18:44:34 +08:00
< a href = "../concepts/ingress.html" >
2.2.14 Ingress
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.2.15" data-path = "../concepts/configmap.html" >
2017-08-21 18:44:34 +08:00
< a href = "../concepts/configmap.html" >
2.2.15 ConfigMap
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.2.16" data-path = "../concepts/horizontal-pod-autoscaling.html" >
2017-08-21 18:44:34 +08:00
< a href = "../concepts/horizontal-pod-autoscaling.html" >
2.2.16 Horizontal Pod Autoscaling
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.2.17" data-path = "../concepts/label.html" >
2017-08-21 18:44:34 +08:00
< a href = "../concepts/label.html" >
2.2.17 Label
< / a >
2017-09-03 15:58:39 +08:00
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.2.18" data-path = "../concepts/garbage-collection.html" >
2017-09-03 15:58:39 +08:00
< a href = "../concepts/garbage-collection.html" >
2.2.18 垃圾收集
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.3.2.19" data-path = "../concepts/network-policy.html" >
2017-09-03 15:58:39 +08:00
< a href = "../concepts/network-policy.html" >
2.2.19 NetworkPolicy
< / a >
2017-08-21 18:44:34 +08:00
< / li >
< / ul >
< / li >
< / ul >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.4" data-path = "./" >
2017-08-21 18:44:34 +08:00
< a href = "./" >
3. 用户指南
< / a >
< ul class = "articles" >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.4.1" data-path = "resource-configuration.html" >
2017-08-21 18:44:34 +08:00
< a href = "resource-configuration.html" >
2017-09-28 21:20:49 +08:00
3.1 资源对象配置
2017-08-21 18:44:34 +08:00
< / a >
< ul class = "articles" >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.4.1.1" data-path = "configure-liveness-readiness-probes.html" >
2017-08-21 18:44:34 +08:00
< a href = "configure-liveness-readiness-probes.html" >
3.1.1 配置Pod的liveness和readiness探针
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.4.1.2" data-path = "configure-pod-service-account.html" >
2017-08-21 18:44:34 +08:00
< a href = "configure-pod-service-account.html" >
3.1.2 配置Pod的Service Account
< / a >
2017-09-28 21:20:49 +08:00
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.4.1.3" data-path = "secret-configuration.html" >
2017-09-28 21:20:49 +08:00
< a href = "secret-configuration.html" >
3.1.3 Secret配置
< / a >
2017-10-10 14:51:45 +08:00
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.4.1.4" data-path = "resource-quota-management.html" >
2017-10-10 14:51:45 +08:00
< a href = "resource-quota-management.html" >
3.2.3 管理namespace中的资源配额
< / a >
2017-08-21 18:44:34 +08:00
< / li >
< / ul >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.4.2" data-path = "command-usage.html" >
2017-08-21 18:44:34 +08:00
< a href = "command-usage.html" >
3.2 命令使用
< / a >
< ul class = "articles" >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.4.2.1" data-path = "using-kubectl.html" >
2017-08-21 18:44:34 +08:00
< a href = "using-kubectl.html" >
3.2.1 使用kubectl
< / a >
2017-09-16 20:56:43 +08:00
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.4.2.2" data-path = "docker-cli-to-kubectl.html" >
2017-09-16 20:56:43 +08:00
< a href = "docker-cli-to-kubectl.html" >
3.2.2 docker用户过度到kubectl命令行指南
< / a >
2017-08-21 18:44:34 +08:00
< / li >
< / ul >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.4.3" data-path = "cluster-security-management.html" >
2017-08-21 18:44:34 +08:00
2017-09-07 12:29:13 +08:00
< a href = "cluster-security-management.html" >
2017-08-21 18:44:34 +08:00
2017-09-07 12:29:13 +08:00
3.3 集群安全性管理
2017-08-21 18:44:34 +08:00
< / a >
< ul class = "articles" >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.4.3.1" data-path = "managing-tls-in-a-cluster.html" >
2017-08-21 18:44:34 +08:00
< a href = "managing-tls-in-a-cluster.html" >
3.3.1 管理集群中的TLS
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.4.3.2" data-path = "kubelet-authentication-authorization.html" >
2017-08-21 18:44:34 +08:00
< a href = "kubelet-authentication-authorization.html" >
3.3.2 kubelet的认证授权
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.4.3.3" data-path = "tls-bootstrapping.html" >
2017-08-21 18:44:34 +08:00
< a href = "tls-bootstrapping.html" >
3.3.3 TLS bootstrap
< / a >
2017-08-31 14:23:44 +08:00
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.4.3.4" data-path = "kubectl-user-authentication-authorization.html" >
2017-08-31 14:23:44 +08:00
< a href = "kubectl-user-authentication-authorization.html" >
3.3.4 kubectl的用户认证授权
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.4.3.5" data-path = "rbac.html" >
2017-08-31 14:23:44 +08:00
< a href = "rbac.html" >
3.3.5 RBAC——基于角色的访问控制
< / a >
2017-09-07 14:13:59 +08:00
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.4.3.6" data-path = "ip-masq-agent.html" >
2017-09-07 14:13:59 +08:00
< a href = "ip-masq-agent.html" >
3.3.6 IP伪装代理
< / a >
2017-08-21 18:44:34 +08:00
< / li >
< / ul >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.4.4" data-path = "access-kubernetes-cluster.html" >
2017-08-21 18:44:34 +08:00
< a href = "access-kubernetes-cluster.html" >
3.4 访问 Kubernetes 集群
< / a >
< ul class = "articles" >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.4.4.1" data-path = "access-cluster.html" >
2017-08-21 18:44:34 +08:00
< a href = "access-cluster.html" >
3.4.1 访问集群
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.4.4.2" data-path = "authenticate-across-clusters-kubeconfig.html" >
2017-08-21 18:44:34 +08:00
< a href = "authenticate-across-clusters-kubeconfig.html" >
3.4.2 使用 kubeconfig 文件配置跨集群认证
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.4.4.3" data-path = "connecting-to-applications-port-forward.html" >
2017-08-21 18:44:34 +08:00
< a href = "connecting-to-applications-port-forward.html" >
3.4.3 通过端口转发访问集群中的应用程序
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.4.4.4" data-path = "service-access-application-cluster.html" >
2017-08-21 18:44:34 +08:00
< a href = "service-access-application-cluster.html" >
3.4.4 使用 service 访问群集中的应用程序
< / a >
< / li >
< / ul >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.4.5" data-path = "application-development-deployment-flow.html" >
2017-08-21 18:44:34 +08:00
< a href = "application-development-deployment-flow.html" >
3.5 在kubernetes中开发部署应用
< / a >
< ul class = "articles" >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.4.5.1" data-path = "deploy-applications-in-kubernetes.html" >
2017-08-21 18:44:34 +08:00
< a href = "deploy-applications-in-kubernetes.html" >
3.5.1 适用于kubernetes的应用开发部署流程
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter active" data-level = "1.4.5.2" data-path = "migrating-hadoop-yarn-to-kubernetes.html" >
2017-08-21 18:44:34 +08:00
< a href = "migrating-hadoop-yarn-to-kubernetes.html" >
3.5.2 迁移传统应用到kubernetes中——以Hadoop YARN为例
< / a >
2017-10-23 13:40:58 +08:00
< / li >
< li class = "chapter " data-level = "1.4.5.3" data-path = "using-statefulset.html" >
< a href = "using-statefulset.html" >
3.5.3 使用StatefulSet部署用状态应用
< / a >
2017-08-21 18:44:34 +08:00
< / li >
< / ul >
< / li >
< / ul >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5" data-path = "../practice/" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/" >
4. 最佳实践
< / a >
< ul class = "articles" >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.1" data-path = "../practice/install-kbernetes1.6-on-centos.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/install-kbernetes1.6-on-centos.html" >
4.1 在CentOS上部署kubernetes1.6集群
< / a >
< ul class = "articles" >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.1.1" data-path = "../practice/create-tls-and-secret-key.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/create-tls-and-secret-key.html" >
4.1.1 创建TLS证书和秘钥
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.1.2" data-path = "../practice/create-kubeconfig.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/create-kubeconfig.html" >
4.1.2 创建kubeconfig文件
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.1.3" data-path = "../practice/etcd-cluster-installation.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/etcd-cluster-installation.html" >
4.1.3 创建高可用etcd集群
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.1.4" data-path = "../practice/kubectl-installation.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/kubectl-installation.html" >
4.1.4 安装kubectl命令行工具
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.1.5" data-path = "../practice/master-installation.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/master-installation.html" >
4.1.5 部署master节点
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.1.6" data-path = "../practice/node-installation.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/node-installation.html" >
4.1.6 部署node节点
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.1.7" data-path = "../practice/kubedns-addon-installation.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/kubedns-addon-installation.html" >
4.1.7 安装kubedns插件
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.1.8" data-path = "../practice/dashboard-addon-installation.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/dashboard-addon-installation.html" >
4.1.8 安装dashboard插件
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.1.9" data-path = "../practice/heapster-addon-installation.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/heapster-addon-installation.html" >
4.1.9 安装heapster插件
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.1.10" data-path = "../practice/efk-addon-installation.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/efk-addon-installation.html" >
4.1.10 安装EFK插件
< / a >
< / li >
< / ul >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.2" data-path = "../practice/service-discovery-and-loadbalancing.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/service-discovery-and-loadbalancing.html" >
4.2 服务发现与负载均衡
< / a >
< ul class = "articles" >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.2.1" data-path = "../practice/traefik-ingress-installation.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/traefik-ingress-installation.html" >
4.2.1 安装Traefik ingress
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.2.2" data-path = "../practice/distributed-load-test.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/distributed-load-test.html" >
4.2.2 分布式负载测试
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.2.3" data-path = "../practice/network-and-cluster-perfermance-test.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/network-and-cluster-perfermance-test.html" >
4.2.3 网络和集群性能测试
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.2.4" data-path = "../practice/edge-node-configuration.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/edge-node-configuration.html" >
4.2.4 边缘节点配置
< / a >
< / li >
< / ul >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.3" data-path = "../practice/operation.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/operation.html" >
4.3 运维管理
< / a >
< ul class = "articles" >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.3.1" data-path = "../practice/service-rolling-update.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/service-rolling-update.html" >
4.3.1 服务滚动升级
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.3.2" data-path = "../practice/app-log-collection.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/app-log-collection.html" >
4.3.2 应用日志收集
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.3.3" data-path = "../practice/configuration-best-practice.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/configuration-best-practice.html" >
4.3.3 配置最佳实践
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.3.4" data-path = "../practice/monitor.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/monitor.html" >
4.3.4 集群及应用监控
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.3.5" data-path = "../practice/jenkins-ci-cd.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/jenkins-ci-cd.html" >
4.3.5 使用Jenkins进行持续构建与发布
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.3.6" data-path = "../practice/data-persistence-problem.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/data-persistence-problem.html" >
4.3.6 数据持久化问题
< / a >
2017-09-03 15:58:39 +08:00
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.3.7" data-path = "../practice/manage-compute-resources-container.html" >
2017-09-03 15:58:39 +08:00
< a href = "../practice/manage-compute-resources-container.html" >
4.3.7 管理容器的计算资源
< / a >
2017-09-25 21:41:08 +08:00
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.3.8" data-path = "../practice/using-prometheus-to-monitor-kuberentes-cluster.html" >
2017-09-25 21:41:08 +08:00
< a href = "../practice/using-prometheus-to-monitor-kuberentes-cluster.html" >
4.3.8 使用Prometheus监控kubernetes集群
< / a >
2017-10-16 17:36:01 +08:00
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.3.9" data-path = "../practice/using-heapster-to-get-object-metrics.html" >
2017-10-16 17:36:01 +08:00
< a href = "../practice/using-heapster-to-get-object-metrics.html" >
4.3.9 使用Heapster获取集群和对象的metric数据
< / a >
2017-10-18 16:53:12 +08:00
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.3.10" data-path = "../practice/manually-upgrade.html" >
2017-10-18 16:53:12 +08:00
< a href = "../practice/manually-upgrade.html" >
4.3.10 手动集群升级
< / a >
2017-08-21 18:44:34 +08:00
< / li >
< / ul >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.4" data-path = "../practice/storage.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/storage.html" >
4.4 存储管理
< / a >
< ul class = "articles" >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.4.1" data-path = "../practice/glusterfs.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/glusterfs.html" >
4.4.1 GlusterFS
< / a >
< ul class = "articles" >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.4.1.1" data-path = "../practice/using-glusterfs-for-persistent-storage.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/using-glusterfs-for-persistent-storage.html" >
4.4.1.1 使用GlusterFS做持久化存储
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.4.1.2" data-path = "../practice/storage-for-containers-using-glusterfs-with-openshift.html" >
2017-08-21 18:44:34 +08:00
< a href = "../practice/storage-for-containers-using-glusterfs-with-openshift.html" >
4.4.1.2 在OpenShift中使用GlusterFS做持久化存储
< / a >
2017-09-01 21:04:51 +08:00
< / li >
< / ul >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.4.2" data-path = "../practice/cephfs.html" >
2017-09-01 21:04:51 +08:00
< a href = "../practice/cephfs.html" >
4.4.2 CephFS
< / a >
< ul class = "articles" >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.4.2.1" data-path = "../practice/using-ceph-for-persistent-storage.html" >
2017-09-01 21:04:51 +08:00
< a href = "../practice/using-ceph-for-persistent-storage.html" >
4.4.2.1 使用Ceph做持久化存储
< / a >
2017-08-21 18:44:34 +08:00
< / li >
< / ul >
< / li >
< / ul >
2017-10-19 15:31:08 +08:00
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.5.5" data-path = "../practice/services-management-tool.html" >
2017-10-19 15:31:08 +08:00
< a href = "../practice/services-management-tool.html" >
4.5 服务编排管理
< / a >
2017-10-20 16:22:43 +08:00
< ul class = "articles" >
< li class = "chapter " data-level = "1.5.5.1" data-path = "../practice/helm.html" >
< a href = "../practice/helm.html" >
4.5.1 Helm
< / a >
< / li >
< / ul >
2017-08-21 18:44:34 +08:00
< / li >
< / ul >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.6" data-path = "../usecases/" >
2017-08-21 18:44:34 +08:00
< a href = "../usecases/" >
5. 领域应用
< / a >
< ul class = "articles" >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.6.1" data-path = "../usecases/microservices.html" >
2017-08-21 18:44:34 +08:00
< a href = "../usecases/microservices.html" >
5.1 微服务架构
< / a >
< ul class = "articles" >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.6.1.1" data-path = "../usecases/service-discovery-in-microservices.html" >
2017-09-20 21:55:19 +08:00
< a href = "../usecases/service-discovery-in-microservices.html" >
5.1.1 微服务中的服务发现
< / a >
< / li >
< / ul >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.6.2" data-path = "../usecases/service-mesh.html" >
2017-09-20 21:55:19 +08:00
< a href = "../usecases/service-mesh.html" >
5.2 Service Mesh 服务网格
< / a >
< ul class = "articles" >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.6.2.1" data-path = "../usecases/istio.html" >
2017-08-21 18:44:34 +08:00
< a href = "../usecases/istio.html" >
5.1.1 Istio
< / a >
< ul class = "articles" >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.6.2.1.1" data-path = "../usecases/istio-installation.html" >
2017-08-21 18:44:34 +08:00
< a href = "../usecases/istio-installation.html" >
5.1.1.1 安装istio
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.6.2.1.2" data-path = "../usecases/configuring-request-routing.html" >
2017-08-21 18:44:34 +08:00
< a href = "../usecases/configuring-request-routing.html" >
5.1.1.2 配置请求的路由规则
< / a >
< / li >
< / ul >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.6.2.2" data-path = "../usecases/linkerd.html" >
2017-08-21 18:44:34 +08:00
< a href = "../usecases/linkerd.html" >
5.1.2 Linkerd
< / a >
< ul class = "articles" >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.6.2.2.1" data-path = "../usecases/linkerd-user-guide.html" >
2017-08-21 18:44:34 +08:00
< a href = "../usecases/linkerd-user-guide.html" >
5.1.2.1 Linkerd 使用指南
< / a >
< / li >
< / ul >
< / li >
< / ul >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.6.3" data-path = "../usecases/big-data.html" >
2017-08-21 18:44:34 +08:00
< a href = "../usecases/big-data.html" >
5.2 大数据
< / a >
< ul class = "articles" >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.6.3.1" data-path = "../usecases/spark-standalone-on-kubernetes.html" >
2017-08-21 18:44:34 +08:00
2017-08-30 14:20:52 +08:00
< a href = "../usecases/spark-standalone-on-kubernetes.html" >
2017-08-21 18:44:34 +08:00
2017-08-30 14:20:52 +08:00
5.2.1 Spark standalone on Kubernetes
2017-08-21 18:44:34 +08:00
< / a >
2017-08-31 14:23:44 +08:00
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.6.3.2" data-path = "../usecases/running-spark-with-kubernetes-native-scheduler.html" >
2017-08-31 14:23:44 +08:00
2017-09-14 15:57:50 +08:00
< a href = "../usecases/running-spark-with-kubernetes-native-scheduler.html" >
2017-08-31 14:23:44 +08:00
5.2.2 运行支持kubernetes原生调度的Spark程序
< / a >
2017-08-21 18:44:34 +08:00
< / li >
< / ul >
2017-08-30 16:52:33 +08:00
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.6.4" data-path = "../usecases/serverless.html" >
2017-08-30 16:52:33 +08:00
< a href = "../usecases/serverless.html" >
5.3 Serverless架构
< / a >
2017-08-21 18:44:34 +08:00
< / li >
< / ul >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.7" data-path = "../develop/" >
2017-08-21 18:44:34 +08:00
< a href = "../develop/" >
6. 开发指南
< / a >
< ul class = "articles" >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.7.1" data-path = "../develop/developing-environment.html" >
2017-08-21 18:44:34 +08:00
< a href = "../develop/developing-environment.html" >
6.1 开发环境搭建
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.7.2" data-path = "../develop/testing.html" >
2017-08-21 18:44:34 +08:00
< a href = "../develop/testing.html" >
6.2 单元测试和集成测试
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.7.3" data-path = "../develop/client-go-sample.html" >
2017-08-21 18:44:34 +08:00
< a href = "../develop/client-go-sample.html" >
6.3 client-go示例
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.7.4" data-path = "../develop/contribute.html" >
2017-08-21 18:44:34 +08:00
< a href = "../develop/contribute.html" >
6.4 社区贡献
< / a >
< / li >
< / ul >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.8" data-path = "../appendix/" >
2017-08-21 18:44:34 +08:00
< a href = "../appendix/" >
7. 附录
< / a >
< ul class = "articles" >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.8.1" data-path = "../appendix/docker-best-practice.html" >
2017-08-21 18:44:34 +08:00
< a href = "../appendix/docker-best-practice.html" >
7.1 Docker最佳实践
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.8.2" data-path = "../appendix/issues.html" >
2017-08-21 18:44:34 +08:00
< a href = "../appendix/issues.html" >
7.2 问题记录
< / a >
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.8.3" data-path = "../appendix/tricks.html" >
2017-08-21 18:44:34 +08:00
< a href = "../appendix/tricks.html" >
7.3 使用技巧
< / a >
2017-09-21 15:00:54 +08:00
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.8.4" data-path = "../appendix/debug-kubernetes-services.html" >
2017-09-21 15:00:54 +08:00
< a href = "../appendix/debug-kubernetes-services.html" >
7.4 kubernetes service中的故障排查
< / a >
2017-10-18 14:16:55 +08:00
< / li >
2017-10-20 10:51:14 +08:00
< li class = "chapter " data-level = "1.8.5" data-path = "../appendix/material-share.html" >
2017-10-18 14:16:55 +08:00
< a href = "../appendix/material-share.html" >
2017-10-18 16:53:12 +08:00
7.5 Kubernetes相关资讯和情报链接
2017-10-18 14:16:55 +08:00
< / a >
2017-08-21 18:44:34 +08:00
< / li >
< / ul >
< / li >
< li class = "divider" > < / li >
< li >
< a href = "https://www.gitbook.com" target = "blank" class = "gitbook-link" >
2017-09-19 22:01:07 +08:00
本书使用 GitBook 发布
2017-08-21 18:44:34 +08:00
< / a >
< / li >
< / ul >
< / nav >
< / div >
< div class = "book-body" >
< div class = "body-inner" >
< div class = "book-header" role = "navigation" >
<!-- Title -->
< h1 >
< i class = "fa fa-circle-o-notch fa-spin" > < / i >
< a href = ".." > 3.5.2 迁移传统应用到kubernetes中——以Hadoop YARN为例< / a >
< / h1 >
< / div >
< div class = "page-wrapper" tabindex = "-1" role = "main" >
< div class = "page-inner" >
< div class = "search-plus" id = "book-search-results" >
< div class = "search-noresults" >
< section class = "normal markdown-section" >
< h1 id = "迁移传统应用到kubernetes步骤详解——以hadoop-yarn为例" > 迁 移 传 统 应 用 到 Kubernetes步 骤 详 解 — — 以 Hadoop YARN为 例 < / h1 >
< p > 本 文 档 不 是 说 明 如 何 在 kubernetes 中 开 发 和 部 署 应 用 程 序 ， 如 果 您 想 要 直 接 开 发 应 用 程 序 在 kubernetes 中 运 行 可 以 参 考 < a href = "deploy-applications-in-kubernetes.html" > 适 用 于 kubernetes的 应 用 开 发 部 署 流 程 < / a > 。 < / p >
< p > 本 文 旨 在 说 明 如 何 将 已 有 的 应 用 程 序 尤 其 是 传 统 的 分 布 式 应 用 程 序 迁 移 到 kubernetes 中 。 如 果 该 类 应 用 程 序 符 合 云 原 生 应 用 规 范 （ 如 12因 素 法 则 ） 的 话 ， 那 么 迁 移 会 比 较 顺 利 ， 否 则 会 遇 到 一 些 麻 烦 甚 至 是 阻 碍 。 具 体 请 参 考 < a href = "https://github.com/rootsongjc/migrating-to-cloud-native-application-architectures" target = "_blank" > 迁 移 至 云 原 生 应 用 架 构 < / a > 。 < / p >
< p > 接 下 来 我 们 将 以 Spark on YARN with kubernetes 为 例 来 说 明 ， 该 例 子 足 够 复 杂 也 很 有 典 型 性 ， 了 解 了 这 个 例 子 可 以 帮 助 大 家 将 自 己 的 应 用 迁 移 到 kubernetes 集 群 上 去 ， 代 码 和 配 置 文 件 可 以 在 < a href = "https://github.com/rootsongjc/kube-yarn" target = "_blank" > 这 里 < / a > 找 到 （ 本 文 中 加 入 Spark 的 配 置 ， 代 码 中 并 没 有 包 含 ， 读 者 可 以 自 己 配 置 ） 。 < / p >
< p > 下 图 即 整 个 架 构 的 示 意 图 ， 代 码 和 详 细 配 置 文 件 请 参 考 < a href = "https://github.com/rootsongjc/kube-yarn" target = "_blank" > kube-yarn< / a > （ 不 包 含 ingress、 spark 配 置 ） ， 所 有 的 进 程 管 理 和 容 器 扩 容 直 接 使 用 Makefile。 < / p >
2017-10-20 10:51:14 +08:00
< figure id = "fig1.4.5.2.1" > < img src = "../images/spark-on-yarn-with-kubernetes.png" alt = "spark on yarn with kubernetes" > < figcaption > 图 片 - spark on yarn with kubernetes< / figcaption > < / figure >
2017-08-21 18:44:34 +08:00
< p > < strong > 注 意 ： 该 例 子 仅 用 来 说 明 具 体 的 步 骤 划 分 和 复 杂 性 ， 在 生 产 环 境 应 用 还 有 待 验 证 ， 请 谨 慎 使 用 。 < / strong > < / p >
< h2 id = "术语" > 术 语 < / h2 >
< p > 对 于 为 曾 接 触 过 kubernetes 或 对 云 平 台 的 技 术 细 节 不 太 了 解 的 人 来 说 ， 如 何 将 应 用 迁 移 到 kubernetes 中 可 能 是 个 头 疼 的 问 题 ， 在 行 动 之 前 有 必 要 先 了 解 整 个 过 程 中 需 要 用 到 哪 些 概 念 和 术 语 ， 有 助 于 大 家 在 行 动 中 达 成 共 识 。 < / p >
< p > 过 程 中 可 能 用 到 的 概 念 和 术 语 初 步 整 理 如 下 ： < / p >
2017-10-20 10:51:14 +08:00
< figure id = "fig1.4.5.2.2" > < img src = "../images/terms-in-kubernetes-app-deployment.png" alt = "Terms" > < figcaption > 图 片 - Terms< / figcaption > < / figure >
2017-08-21 18:44:34 +08:00
< p > 为 了 讲 解 整 改 过 程 和 具 体 细 节 ， 我 们 所 有 操 作 都 是 通 过 命 令 手 动 完 成 ， 不 使 用 自 动 化 工 具 。 当 您 充 分 了 解 到 其 中 的 细 节 后 可 以 通 过 自 动 化 工 具 来 优 化 该 过 程 ， 以 使 其 更 加 自 动 和 高 效 ， 同 时 减 少 因 为 人 为 操 作 失 误 导 致 的 迁 移 失 败 。 < / p >
< h2 id = "迁移应用" > 迁 移 应 用 < / h2 >
2017-10-20 10:51:14 +08:00
< figure id = "fig1.4.5.2.3" > < img src = "../images/migrating-hadoop-yarn-to-kubernetes.png" alt = "分解步骤解析" > < figcaption > 图 片 - 分 解 步 骤 解 析 < / figcaption > < / figure >
2017-08-21 18:44:34 +08:00
< p > 整 个 迁 移 过 程 分 为 如 下 几 个 步 骤 ： < / p >
< ol >
< li > < p > < strong > 将 原 有 应 用 拆 解 为 服 务 < / strong > < / p >
< p > 我 们 不 是 一 上 来 就 开 始 做 镜 像 ， 写 配 置 ， 而 是 应 该 先 梳 理 下 要 迁 移 的 应 用 中 有 哪 些 可 以 作 为 服 务 运 行 ， 哪 些 是 变 的 ， 哪 些 是 不 变 的 部 分 。 < / p >
< p > 服 务 划 分 的 原 则 是 最 小 可 变 原 则 ， 这 个 同 样 适 用 于 镜 像 制 作 ， 将 服 务 中 不 变 的 部 分 编 译 到 同 一 个 镜 像 中 。 < / p >
< p > 对 于 像 Spark on YARN 这 样 复 杂 的 应 用 ， 可 以 将 其 划 分 为 三 大 类 服 务 ： < / p >
< ul >
< li > ResourceManager< / li >
< li > NodeManager< / li >
< li > Spark client< / li >
< / ul >
< / li >
< li > < p > < strong > 制 作 镜 像 < / strong > < / p >
< p > 根 据 拆 解 出 来 的 服 务 ， 我 们 需 要 制 作 两 个 镜 像 ： < / p >
< ul >
< li > Hadoop< / li >
< li > Spark (From hadoop docker image)< / li >
< / ul >
< p > 因 为 我 们 运 行 的 是 Spark on YARN， 因 此 Spark 依 赖 与 Hadoop 镜 像 ， 我 们 在 Spark 的 基 础 上 包 装 了 一 个 web service 作 为 服 务 启 动 。 < / p >
< p > 镜 像 制 作 过 程 中 不 需 要 在 Dockerfile 中 指 定 Entrypoint 和 CMD， 这 些 都 是 在 kubernetes 的 YAML 文 件 中 指 定 的 。 < / p >
< p > Hadoop YARN 的 Dockerfile 参 考 如 下 配 置 。 < / p >
< pre > < code class = "lang-dockerfile" > < span class = "hljs-keyword" > FROM< / span > my-docker-repo/jdk:< span class = "hljs-number" > 7< / span > u8< span class = "hljs-number" > 0
< / span >
< span class = "hljs-comment" > # Add native libs< / span >
ARG HADOOP_VERSION=< span class = "hljs-number" > 2.6< / span > .< span class = "hljs-number" > 0< / span > -cdh5.< span class = "hljs-number" > 5.2< / span >
< span class = "hljs-comment" > ## Prefer to download from server not use local storage< / span >
ADD hadoop-${HADOOP_VERSION}.tar.gz /usr/local
< span class = "hljs-keyword" > ADD< / span > < span class = "bash" > ./lib/* /usr/< span class = "hljs-built_in" > local< / span > /hadoop-< span class = "hljs-variable" > ${HADOOP_VERSION}< / span > /lib/native/
< / span > < span class = "hljs-keyword" > ADD< / span > < span class = "bash" > ./jars/* /usr/< span class = "hljs-built_in" > local< / span > /hadoop-< span class = "hljs-variable" > ${HADOOP_VERSION}< / span > /share/hadoop/yarn/
< / span > < span class = "hljs-keyword" > ENV< / span > HADOOP_PREFIX=/usr/local/hadoop \
HADOOP_COMMON_HOME=/usr/local/hadoop \
HADOOP_HDFS_HOME=/usr/local/hadoop \
HADOOP_MAPRED_HOME=/usr/local/hadoop \
HADOOP_YARN_HOME=/usr/local/hadoop \
HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop \
YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop \
PATH=${PATH}:/usr/local/hadoop/bin
< span class = "hljs-keyword" > RUN< / span > < span class = "bash" > \
< span class = "hljs-built_in" > cd< / span > /usr/< span class = "hljs-built_in" > local< / span > & & ln < span class = "hljs-_" > -s< / span > ./hadoop-< span class = "hljs-variable" > ${HADOOP_VERSION}< / span > hadoop & & \
rm < span class = "hljs-_" > -f< / span > < span class = "hljs-variable" > ${HADOOP_PREFIX}< / span > /logs/*
< / span >
< span class = "hljs-keyword" > WORKDIR< / span > < span class = "bash" > < span class = "hljs-variable" > $HADOOP_PREFIX< / span >
< / span >
< span class = "hljs-comment" > # Hdfs ports< / span >
< span class = "hljs-keyword" > EXPOSE< / span > < span class = "hljs-number" > 50010< / span > < span class = "hljs-number" > 50020< / span > < span class = "hljs-number" > 50070< / span > < span class = "hljs-number" > 50075< / span > < span class = "hljs-number" > 50090< / span > < span class = "hljs-number" > 8020< / span > < span class = "hljs-number" > 9000< / span >
< span class = "hljs-comment" > # Mapred ports< / span >
< span class = "hljs-keyword" > EXPOSE< / span > < span class = "hljs-number" > 19888< / span >
< span class = "hljs-comment" > #Yarn ports< / span >
< span class = "hljs-keyword" > EXPOSE< / span > < span class = "hljs-number" > 8030< / span > < span class = "hljs-number" > 8031< / span > < span class = "hljs-number" > 8032< / span > < span class = "hljs-number" > 8033< / span > < span class = "hljs-number" > 8040< / span > < span class = "hljs-number" > 8042< / span > < span class = "hljs-number" > 8088< / span >
< span class = "hljs-comment" > #Other ports< / span >
< span class = "hljs-keyword" > EXPOSE< / span > < span class = "hljs-number" > 49707< / span > < span class = "hljs-number" > 2122< / span >
< / code > < / pre >
< / li >
< li > < p > < strong > 准 备 应 用 的 配 置 文 件 < / strong > < / p >
< p > 因 为 我 们 只 制 作 了 一 个 Hadoop 的 镜 像 ， 而 需 要 启 动 两 个 服 务 ， 这 就 要 求 在 服 务 启 动 的 时 候 必 须 加 载 不 同 的 配 置 文 件 ， 现 在 我 们 只 需 要 准 备 两 个 服 务 中 需 要 同 时 用 的 的 配 置 的 部 分 。 < / p >
< p > YARN 依 赖 的 配 置 在 < code > artifacts< / code > 目 录 下 ， 包 含 以 下 文 件 ： < / p >
< pre > < code > bootstrap.sh
capacity-scheduler.xml
container-executor.cfg
core-site.xml
hadoop-env.sh
hdfs-site.xml
log4j.properties
mapred-site.xml
nodemanager_exclude.txt
slaves
start-yarn-nm.sh
start-yarn-rm.sh
yarn-env.sh
yarn-site.xml
< / code > < / pre > < p > 其 中 作 为 bootstrap 启 动 脚 本 的 < code > bootstrap.sh< / code > 也 包 含 在 该 目 录 下 ， 该 脚 本 的 如 何 编 写 请 见 下 文 。 < / p >
< / li >
< li > < p > < strong > Kubernetes YAML 文 件 < / strong > < / p >
< p > 根 据 业 务 的 特 性 选 择 最 适 合 的 kubernetes 的 资 源 对 象 来 运 行 ， 因 为 在 YARN 中 NodeManager 需 要 使 用 主 机 名 向 ResourceManger 注 册 ， 因 此 需 要 沿 用 YARN 原 有 的 服 务 发 现 方 式 ， 使 用 headless service 和 StatefulSet 资 源 。 更 多 资 料 请 参 考 < a href = "../concepts/statefulset.html" > StatefulSet< / a > 。 < / p >
< p > 所 有 的 Kubernetes YAML 配 置 文 件 存 储 在 < code > manifest< / code > 目 录 下 ， 包 括 如 下 配 置 ： < / p >
< ul >
< li > yarn-cluster 的 namespace 配 置 < / li >
< li > Spark、 ResourceManager、 NodeManager 的 headless service 和 StatefulSet 配 置 < / li >
< li > 需 要 暴 露 到 kubernetes 集 群 外 部 的 ingress 配 置 （ ResourceManager 的 Web） < / li >
< / ul >
< pre > < code > kube-yarn-ingress.yaml
spark-statefulset.yaml
yarn-cluster-namespace.yaml
yarn-nm-statefulset.yaml
yarn-rm-statefulset.yaml
< / code > < / pre > < / li >
< li > < p > < strong > Bootstrap 脚 本 < / strong > < / p >
< p > Bootstrap 脚 本 的 作 用 是 在 启 动 时 根 据 Pod 的 环 境 变 量 、 主 机 名 或 其 他 可 以 区 分 不 同 Pod 和 将 启 动 角 色 的 变 量 来 修 改 配 置 文 件 和 启 动 服 务 应 用 。 < / p >
< p > 该 脚 本 同 时 将 原 来 YARN 的 日 志 使 用 stdout 输 出 ， 便 于 使 用 < code > kubectl logs< / code > 查 看 日 志 或 其 他 日 志 收 集 工 具 进 行 日 志 收 集 。 < / p >
< p > 启 动 脚 本 < code > bootstrap.sh< / code > 跟 Hadoop 的 配 置 文 件 同 时 保 存 在 < code > artifacts< / code > 目 录 下 。 < / p >
< p > 该 脚 本 根 据 Pod 的 主 机 名 ， 决 定 如 何 修 改 Hadoop 的 配 置 文 件 和 启 动 何 种 服 务 。 < code > bootstrap.sh< / code > 文 件 的 部 分 代 码 如 下 ： < / p >
< pre > < code class = "lang-bash" > < span class = "hljs-keyword" > if< / span > [[ < span class = "hljs-string" > " < span class = "hljs-variable" > ${HOSTNAME}< / span > " < / span > =~ < span class = "hljs-string" > " yarn-nm" < / span > ]]; < span class = "hljs-keyword" > then< / span >
sed -i < span class = "hljs-string" > ' /< \/configuration> /d' < / span > < span class = "hljs-variable" > $HADOOP_PREFIX< / span > /etc/hadoop/yarn-site.xml
cat > > < span class = "hljs-variable" > $HADOOP_PREFIX< / span > /etc/hadoop/yarn-site.xml < < - EOM
< property>
< name> yarn.nodemanager.resource.memory-mb< /name>
< value> < span class = "hljs-variable" > ${MY_MEM_LIMIT:-2048}< / span > < /value>
< /property>
< property>
< name> yarn.nodemanager.resource.cpu-vcores< /name>
< value> < span class = "hljs-variable" > ${MY_CPU_LIMIT:-2}< / span > < /value>
< /property>
EOM
< span class = "hljs-built_in" > echo< / span > < span class = "hljs-string" > ' < /configuration> ' < / span > > > < span class = "hljs-variable" > $HADOOP_PREFIX< / span > /etc/hadoop/yarn-site.xml
cp < span class = "hljs-variable" > ${CONFIG_DIR}< / span > /start-yarn-nm.sh < span class = "hljs-variable" > $HADOOP_PREFIX< / span > /sbin/
< span class = "hljs-built_in" > cd< / span > < span class = "hljs-variable" > $HADOOP_PREFIX< / span > /sbin
chmod +x start-yarn-nm.sh
./start-yarn-nm.sh
< span class = "hljs-keyword" > fi< / span >
< span class = "hljs-keyword" > if< / span > [[ < span class = "hljs-variable" > $1< / span > == < span class = "hljs-string" > " -d" < / span > ]]; < span class = "hljs-keyword" > then< / span >
until find < span class = "hljs-variable" > ${HADOOP_PREFIX}< / span > /logs -mmin -1 | egrep -q < span class = "hljs-string" > ' .*' < / span > ; < span class = "hljs-built_in" > echo< / span > < span class = "hljs-string" > " `date`: Waiting for logs..." < / span > ; < span class = "hljs-keyword" > do< / span > sleep 2 ; < span class = "hljs-keyword" > done< / span >
tail -F < span class = "hljs-variable" > ${HADOOP_PREFIX}< / span > /logs/* &
< span class = "hljs-keyword" > while< / span > < span class = "hljs-literal" > true< / span > ; < span class = "hljs-keyword" > do< / span > sleep 1000; < span class = "hljs-keyword" > done< / span >
< span class = "hljs-keyword" > fi< / span >
< / code > < / pre >
< p > 从 这 部 分 中 代 码 中 可 以 看 到 ， 如 果 Pod 的 主 机 名 中 包 含 < code > yarn-nm< / code > 字 段 则 向 < code > yarn-site.xml< / code > 配 置 文 件 中 增 加 如 下 内 容 ： < / p >
< pre > < code class = "lang-xml" > < span class = "hljs-tag" > < < span class = "hljs-name" > property< / span > > < / span >
< span class = "hljs-tag" > < < span class = "hljs-name" > name< / span > > < / span > yarn.nodemanager.resource.memory-mb< span class = "hljs-tag" > < /< span class = "hljs-name" > name< / span > > < / span >
< span class = "hljs-tag" > < < span class = "hljs-name" > value< / span > > < / span > ${MY_MEM_LIMIT:-2048}< span class = "hljs-tag" > < /< span class = "hljs-name" > value< / span > > < / span >
< span class = "hljs-tag" > < /< span class = "hljs-name" > property< / span > > < / span >
< span class = "hljs-tag" > < < span class = "hljs-name" > property< / span > > < / span >
< span class = "hljs-tag" > < < span class = "hljs-name" > name< / span > > < / span > yarn.nodemanager.resource.cpu-vcores< span class = "hljs-tag" > < /< span class = "hljs-name" > name< / span > > < / span >
< span class = "hljs-tag" > < < span class = "hljs-name" > value< / span > > < / span > ${MY_CPU_LIMIT:-2}< span class = "hljs-tag" > < /< span class = "hljs-name" > value< / span > > < / span >
< span class = "hljs-tag" > < /< span class = "hljs-name" > property< / span > > < / span >
< / code > < / pre >
< p > 其 中 < code > MY_MEM_LIMIT< / code > 和 < code > MY_CPU_LIMIT< / code > 是 kubernetes YAML 中 定 义 的 环 境 变 量 ， 该 环 境 变 量 又 是 引 用 的 Resource limit。 < / p >
< p > 所 有 的 配 置 准 备 完 成 后 ， 执 行 < code > start-yarn-nm.sh< / code > 脚 本 启 动 NodeManager。 < / p >
< p > 如 果 kubernetes YAML 中 的 container CMD args 中 包 含 < code > -d< / code > 则 在 后 台 运 行 NodeManger 并 tail 输 出 NodeManager 的 日 志 到 标 准 输 出 。 < / p >
< / li >
< li > < p > < strong > ConfigMaps< / strong > < / p >
< p > 将 Hadoop 的 配 置 文 件 和 bootstrap 脚 本 作 为 ConfigMap 资 源 保 存 ， 用 作 Pod 启 动 时 挂 载 的 volume。 < / p >
< pre > < code class = "lang-bash" > kubectl create configmap hadoop-config \
--from-file=artifacts/hadoop/bootstrap.sh \
--from-file=artifacts/hadoop/start-yarn-rm.sh \
--from-file=artifacts/hadoop/start-yarn-nm.sh \
--from-file=artifacts/hadoop/slaves \
--from-file=artifacts/hadoop/core-site.xml \
--from-file=artifacts/hadoop/hdfs-site.xml \
--from-file=artifacts/hadoop/mapred-site.xml \
--from-file=artifacts/hadoop/yarn-site.xml \
--from-file=artifacts/hadoop/capacity-scheduler.xml \
--from-file=artifacts/hadoop/container-executor.cfg \
--from-file=artifacts/hadoop/hadoop-env.sh \
--from-file=artifacts/hadoop/< span class = "hljs-built_in" > log< / span > 4j.properties \
--from-file=artifacts/hadoop/nodemanager_exclude.txt \
--from-file=artifacts/hadoop/yarn-env.sh
kubectl create configmap spark-config \
--from-file=artifacts/spark/spark-bootstrap.sh \
--from-file=artifacts/spark/spark-env.sh \
--from-file=artifacts/spark/spark-defaults.conf
< / code > < / pre >
< / li >
< / ol >
2017-10-09 15:28:05 +08:00
< p > 所 有 的 配 置 完 成 后 ， 可 以 可 以 使 用 kubectl 命 令 来 启 动 和 管 理 集 群 了 ， 我 们 编 写 了 Makefile， 您 可 以 直 接 使 用 该 Makefile 封 装 的 命 令 实 现 部 分 的 自 动 化 。 < / p >
< footer class = "page-footer" > < span class = "copyright" > Copyright © jimmysong.io 2017 all right reserved， powered by Gitbook< / span > < span class = "footer-modification" > Updated:
2017-08-21 18:23:34
< / span > < / footer >
2017-08-21 18:44:34 +08:00
< / section >
< / div >
< div class = "search-results" >
< div class = "has-results" >
< h1 class = "search-results-title" > < span class = 'search-results-count' > < / span > results matching "< span class = 'search-query' > < / span > "< / h1 >
< ul class = "search-results-list" > < / ul >
< / div >
< div class = "no-results" >
< h1 class = "search-results-title" > No results matching "< span class = 'search-query' > < / span > "< / h1 >
< / div >
< / div >
< / div >
< / div >
< / div >
< / div >
< a href = "deploy-applications-in-kubernetes.html" class = "navigation navigation-prev " aria-label = "Previous page: 3.5.1 适用于kubernetes的应用开发部署流程" >
< i class = "fa fa-angle-left" > < / i >
< / a >
2017-10-23 13:40:58 +08:00
< a href = "using-statefulset.html" class = "navigation navigation-next " aria-label = "Next page: 3.5.3 使用StatefulSet部署用状态应用" >
2017-08-21 18:44:34 +08:00
< i class = "fa fa-angle-right" > < / i >
< / a >
< / div >
< script >
var gitbook = gitbook || [];
gitbook.push(function() {
2017-10-23 14:24:59 +08:00
gitbook.page.hasChanged({"page":{"title":"3.5.2 迁移传统应用到kubernetes中——以Hadoop YARN为例","level":"1.4.5.2","depth":3,"next":{"title":"3.5.3 使用StatefulSet部署用状态应用","level":"1.4.5.3","depth":3,"path":"guide/using-statefulset.md","ref":"guide/using-statefulset.md","articles":[]},"previous":{"title":"3.5.1 适用于kubernetes的应用开发部署流程","level":"1.4.5.1","depth":3,"path":"guide/deploy-applications-in-kubernetes.md","ref":"guide/deploy-applications-in-kubernetes.md","articles":[]},"dir":"ltr"},"config":{"plugins":["github","codesnippet","splitter","page-toc-button","image-captions","editlink","back-to-top-button","-lunr","-search","search-plus","github-buttons@2.1.0","favicon@^0.0.2","tbfed-pagefooter@^0.0.1","3-ba"],"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"pluginsConfig":{"tbfed-pagefooter":{"copyright":"Copyright © jimmysong.io 2017","modify_label":"Updated:","modify_format":"YYYY-MM-DD HH:mm:ss"},"github":{"url":"https://github.com/rootsongjc/kubernetes-handbook"},"editlink":{"label":"编辑本页","multilingual":false,"base":"https://github.com/rootsongjc/kubernetes-handbook/blob/master/"},"splitter":{},"codesnippet":{},"fontsettings":{"theme":"white","family":"sans","size":2},"highlight":{},"favicon":{"shortcut":"favicon.ico","bookmark":"favicon.ico"},"page-toc-button":{},"back-to-top-button":{},"github-buttons":{"repo":"rootsongjc/kubernetes-handbook","types":["star"],"size":"small"},"3-ba":{"configuration":"auto","token":"11f7d254cfa4e0ca44b175c66d379ecc"},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false},"search-plus":{},"image-captions":{"caption":"图片 - _CAPTION_","variable_name":"_pictures"}},"theme":"default","author":"Jimmy Song","pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"variables":{"_pictures":[{"backlink":"cloud-native/kubernetes-and-cloud-native-app-overview.html#fig1.2.1","level":"1.2","list_caption":"Figure: 云计算演进历程","alt":"云计算演进历程","nro":1,"url":"../images/cloud-computing-evolution-road.jpg","index":1,"caption_template":"图片 - _CAPTION_","label":"云计算演进历程","attributes":{},"skip":false,"key":"1.2.1"},{"backlink":"cloud-native/kubernetes-and-cloud-native-app-overview.html#fig1.2.2","level":"1.2","list_caption":"Figure: Cloud native思维导图","alt":"Cloud native思维导图","nro":2,"url":"../images/cloud-native-architecutre-mindnode.jpg","index":2,"caption_template":"图片 - _CAPTION_","label":"Cloud native思维导图","attributes":{},"skip":false,"key":"1.2.2"},{"backlink":"cloud-native/kubernetes-and-cloud-native-app-overview.html#fig1.2.3","level":"1.2","list_caption":"Figure: 十二因素应用","alt":"十二因素应用","nro":3,"url":"../images/12-factor-app.png","index":3,"caption_template":"图片 - _CAPTION_","label":"十二因素应用","attributes":{},"skip":false,"key":"1.2.3"},{"backlink":"cloud-native/kubernetes-and-cloud-native-app-overview.html#fig1.2.4","level":"1.2","list_caption":"Figure: 使用Jenkins进行持续集成与发布流程图","alt":"使用Jenkins进行持续集成与发布流程图","nro":4,"url":"../images/kubernetes-jenkins-ci-cd.png","index":4,"caption_template":"图片 - _CAPTION_","label":"使用Jenkins进行持续集成与发布流程图","attributes":{},"skip":false,"key":"1.2.4"},{"backlink":"cloud-native/kubernetes-and-cloud-native-app-overview.html#fig1.2.5","level":"1.2","li
2017-08-21 18:44:34 +08:00
});
< / script >
< / div >
< script src = "../gitbook/gitbook.js" > < / script >
< script src = "../gitbook/theme.js" > < / script >
< script src = "../gitbook/gitbook-plugin-github/plugin.js" > < / script >
< script src = "../gitbook/gitbook-plugin-splitter/splitter.js" > < / script >
< script src = "../gitbook/gitbook-plugin-page-toc-button/plugin.js" > < / script >
< script src = "../gitbook/gitbook-plugin-editlink/plugin.js" > < / script >
2017-09-19 21:38:03 +08:00
< script src = "../gitbook/gitbook-plugin-back-to-top-button/plugin.js" > < / script >
2017-08-21 18:44:34 +08:00
< script src = "../gitbook/gitbook-plugin-search-plus/jquery.mark.min.js" > < / script >
< script src = "../gitbook/gitbook-plugin-search-plus/search.js" > < / script >
2017-10-09 15:28:05 +08:00
< script src = "../gitbook/gitbook-plugin-github-buttons/plugin.js" > < / script >
2017-10-11 18:13:13 +08:00
< script src = "../gitbook/gitbook-plugin-3-ba/plugin.js" > < / script >
2017-08-21 18:44:34 +08:00
< script src = "../gitbook/gitbook-plugin-sharing/buttons.js" > < / script >
< script src = "../gitbook/gitbook-plugin-fontsettings/fontsettings.js" > < / script >
< / body >
< / html >
