kubernetes-handbook/guide/migrating-hadoop-yarn-to-ku...

1775 lines
91 KiB
HTML
Raw Normal View History

<!DOCTYPE HTML>
<html lang="zh-cn" >
<head>
<meta charset="UTF-8">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
<title>3.5.2 迁移传统应用到kubernetes中——以Hadoop YARN为例 · Kubernetes Handbook</title>
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="description" content="">
<meta name="generator" content="GitBook 3.2.2">
<meta name="author" content="Jimmy Song">
<link rel="stylesheet" href="../gitbook/style.css">
<link rel="stylesheet" href="../gitbook/gitbook-plugin-splitter/splitter.css">
<link rel="stylesheet" href="../gitbook/gitbook-plugin-page-toc-button/plugin.css">
<link rel="stylesheet" href="../gitbook/gitbook-plugin-image-captions/image-captions.css">
<link rel="stylesheet" href="../gitbook/gitbook-plugin-page-footer-ex/style/plugin.css">
<link rel="stylesheet" href="../gitbook/gitbook-plugin-search-plus/search.css">
<link rel="stylesheet" href="../gitbook/gitbook-plugin-highlight/website.css">
<link rel="stylesheet" href="../gitbook/gitbook-plugin-fontsettings/website.css">
<meta name="HandheldFriendly" content="true"/>
<meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no">
<meta name="apple-mobile-web-app-capable" content="yes">
<meta name="apple-mobile-web-app-status-bar-style" content="black">
<link rel="apple-touch-icon-precomposed" sizes="152x152" href="../gitbook/images/apple-touch-icon-precomposed-152.png">
<link rel="shortcut icon" href="../gitbook/images/favicon.ico" type="image/x-icon">
<link rel="next" href="../practice/" />
<link rel="prev" href="deploy-applications-in-kubernetes.html" />
</head>
<body>
<div class="book">
<div class="book-summary">
<div id="book-search-input" role="search">
<input type="text" placeholder="輸入並搜尋" />
</div>
<nav role="navigation">
<ul class="summary">
<li class="chapter " data-level="1.1" data-path="../">
<a href="../">
1. 前言
</a>
</li>
<li class="chapter " data-level="1.2" data-path="../concepts/">
<a href="../concepts/">
2. 概念原理
</a>
<ul class="articles">
<li class="chapter " data-level="1.2.1" data-path="../concepts/concepts.html">
<a href="../concepts/concepts.html">
2.1 设计理念
</a>
</li>
<li class="chapter " data-level="1.2.2" data-path="../concepts/objects.html">
<a href="../concepts/objects.html">
2017-09-03 13:29:38 +08:00
2.2 Objects
</a>
<ul class="articles">
<li class="chapter " data-level="1.2.2.1" data-path="../concepts/pod-overview.html">
<a href="../concepts/pod-overview.html">
2.2.1 Pod
</a>
<ul class="articles">
<li class="chapter " data-level="1.2.2.1.1" data-path="../concepts/pod.html">
<a href="../concepts/pod.html">
2.2.1.1 Pod解析
</a>
2017-08-31 23:28:33 +08:00
</li>
<li class="chapter " data-level="1.2.2.1.2" data-path="../concepts/init-containers.html">
<a href="../concepts/init-containers.html">
2.2.1.2 Init容器
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="1.2.2.2" data-path="../concepts/node.html">
<a href="../concepts/node.html">
2.2.2 Node
</a>
</li>
<li class="chapter " data-level="1.2.2.3" data-path="../concepts/namespace.html">
<a href="../concepts/namespace.html">
2.2.3 Namespace
</a>
</li>
<li class="chapter " data-level="1.2.2.4" data-path="../concepts/service.html">
<a href="../concepts/service.html">
2.2.4 Service
</a>
</li>
<li class="chapter " data-level="1.2.2.5" data-path="../concepts/volume.html">
<a href="../concepts/volume.html">
2.2.5 Volume和Persistent Volume
</a>
</li>
<li class="chapter " data-level="1.2.2.6" data-path="../concepts/deployment.html">
<a href="../concepts/deployment.html">
2.2.6 Deployment
</a>
</li>
<li class="chapter " data-level="1.2.2.7" data-path="../concepts/secret.html">
<a href="../concepts/secret.html">
2.2.7 Secret
</a>
</li>
<li class="chapter " data-level="1.2.2.8" data-path="../concepts/statefulset.html">
<a href="../concepts/statefulset.html">
2.2.8 StatefulSet
</a>
</li>
<li class="chapter " data-level="1.2.2.9" data-path="../concepts/daemonset.html">
<a href="../concepts/daemonset.html">
2.2.9 DaemonSet
</a>
</li>
<li class="chapter " data-level="1.2.2.10" data-path="../concepts/serviceaccount.html">
<a href="../concepts/serviceaccount.html">
2.2.10 ServiceAccount
</a>
</li>
<li class="chapter " data-level="1.2.2.11" data-path="../concepts/replicaset.html">
<a href="../concepts/replicaset.html">
2.2.11 ReplicationController和ReplicaSet
</a>
</li>
<li class="chapter " data-level="1.2.2.12" data-path="../concepts/job.html">
<a href="../concepts/job.html">
2.2.12 Job
</a>
</li>
<li class="chapter " data-level="1.2.2.13" data-path="../concepts/cronjob.html">
<a href="../concepts/cronjob.html">
2.2.13 CronJob
</a>
</li>
<li class="chapter " data-level="1.2.2.14" data-path="../concepts/ingress.html">
<a href="../concepts/ingress.html">
2.2.14 Ingress
</a>
</li>
<li class="chapter " data-level="1.2.2.15" data-path="../concepts/configmap.html">
<a href="../concepts/configmap.html">
2.2.15 ConfigMap
</a>
</li>
<li class="chapter " data-level="1.2.2.16" data-path="../concepts/horizontal-pod-autoscaling.html">
<a href="../concepts/horizontal-pod-autoscaling.html">
2.2.16 Horizontal Pod Autoscaling
</a>
</li>
<li class="chapter " data-level="1.2.2.17" data-path="../concepts/label.html">
<a href="../concepts/label.html">
2.2.17 Label
</a>
</li>
</ul>
</li>
</ul>
</li>
<li class="chapter " data-level="1.3" data-path="./">
<a href="./">
3. 用户指南
</a>
<ul class="articles">
<li class="chapter " data-level="1.3.1" data-path="resource-configuration.html">
<a href="resource-configuration.html">
3.1 资源配置
</a>
<ul class="articles">
<li class="chapter " data-level="1.3.1.1" data-path="configure-liveness-readiness-probes.html">
<a href="configure-liveness-readiness-probes.html">
3.1.1 配置Pod的liveness和readiness探针
</a>
</li>
<li class="chapter " data-level="1.3.1.2" data-path="configure-pod-service-account.html">
<a href="configure-pod-service-account.html">
3.1.2 配置Pod的Service Account
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="1.3.2" data-path="command-usage.html">
<a href="command-usage.html">
3.2 命令使用
</a>
<ul class="articles">
<li class="chapter " data-level="1.3.2.1" data-path="using-kubectl.html">
<a href="using-kubectl.html">
3.2.1 使用kubectl
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="1.3.3" data-path="cluster-management.html">
<a href="cluster-management.html">
3.3 集群管理
</a>
<ul class="articles">
<li class="chapter " data-level="1.3.3.1" data-path="managing-tls-in-a-cluster.html">
<a href="managing-tls-in-a-cluster.html">
3.3.1 管理集群中的TLS
</a>
</li>
<li class="chapter " data-level="1.3.3.2" data-path="kubelet-authentication-authorization.html">
<a href="kubelet-authentication-authorization.html">
3.3.2 kubelet的认证授权
</a>
</li>
<li class="chapter " data-level="1.3.3.3" data-path="tls-bootstrapping.html">
<a href="tls-bootstrapping.html">
3.3.3 TLS bootstrap
</a>
2017-08-31 14:23:44 +08:00
</li>
<li class="chapter " data-level="1.3.3.4" data-path="kubectl-user-authentication-authorization.html">
<a href="kubectl-user-authentication-authorization.html">
3.3.4 kubectl的用户认证授权
</a>
</li>
<li class="chapter " data-level="1.3.3.5" data-path="rbac.html">
<a href="rbac.html">
3.3.5 RBAC——基于角色的访问控制
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="1.3.4" data-path="access-kubernetes-cluster.html">
<a href="access-kubernetes-cluster.html">
3.4 访问 Kubernetes 集群
</a>
<ul class="articles">
<li class="chapter " data-level="1.3.4.1" data-path="access-cluster.html">
<a href="access-cluster.html">
3.4.1 访问集群
</a>
</li>
<li class="chapter " data-level="1.3.4.2" data-path="authenticate-across-clusters-kubeconfig.html">
<a href="authenticate-across-clusters-kubeconfig.html">
3.4.2 使用 kubeconfig 文件配置跨集群认证
</a>
</li>
<li class="chapter " data-level="1.3.4.3" data-path="connecting-to-applications-port-forward.html">
<a href="connecting-to-applications-port-forward.html">
3.4.3 通过端口转发访问集群中的应用程序
</a>
</li>
<li class="chapter " data-level="1.3.4.4" data-path="service-access-application-cluster.html">
<a href="service-access-application-cluster.html">
3.4.4 使用 service 访问群集中的应用程序
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="1.3.5" data-path="application-development-deployment-flow.html">
<a href="application-development-deployment-flow.html">
3.5 在kubernetes中开发部署应用
</a>
<ul class="articles">
<li class="chapter " data-level="1.3.5.1" data-path="deploy-applications-in-kubernetes.html">
<a href="deploy-applications-in-kubernetes.html">
3.5.1 适用于kubernetes的应用开发部署流程
</a>
</li>
<li class="chapter active" data-level="1.3.5.2" data-path="migrating-hadoop-yarn-to-kubernetes.html">
<a href="migrating-hadoop-yarn-to-kubernetes.html">
3.5.2 迁移传统应用到kubernetes中——以Hadoop YARN为例
</a>
</li>
</ul>
</li>
</ul>
</li>
<li class="chapter " data-level="1.4" data-path="../practice/">
<a href="../practice/">
4. 最佳实践
</a>
<ul class="articles">
<li class="chapter " data-level="1.4.1" data-path="../practice/install-kbernetes1.6-on-centos.html">
<a href="../practice/install-kbernetes1.6-on-centos.html">
4.1 在CentOS上部署kubernetes1.6集群
</a>
<ul class="articles">
<li class="chapter " data-level="1.4.1.1" data-path="../practice/create-tls-and-secret-key.html">
<a href="../practice/create-tls-and-secret-key.html">
4.1.1 创建TLS证书和秘钥
</a>
</li>
<li class="chapter " data-level="1.4.1.2" data-path="../practice/create-kubeconfig.html">
<a href="../practice/create-kubeconfig.html">
4.1.2 创建kubeconfig文件
</a>
</li>
<li class="chapter " data-level="1.4.1.3" data-path="../practice/etcd-cluster-installation.html">
<a href="../practice/etcd-cluster-installation.html">
4.1.3 创建高可用etcd集群
</a>
</li>
<li class="chapter " data-level="1.4.1.4" data-path="../practice/kubectl-installation.html">
<a href="../practice/kubectl-installation.html">
4.1.4 安装kubectl命令行工具
</a>
</li>
<li class="chapter " data-level="1.4.1.5" data-path="../practice/master-installation.html">
<a href="../practice/master-installation.html">
4.1.5 部署master节点
</a>
</li>
<li class="chapter " data-level="1.4.1.6" data-path="../practice/node-installation.html">
<a href="../practice/node-installation.html">
4.1.6 部署node节点
</a>
</li>
<li class="chapter " data-level="1.4.1.7" data-path="../practice/kubedns-addon-installation.html">
<a href="../practice/kubedns-addon-installation.html">
4.1.7 安装kubedns插件
</a>
</li>
<li class="chapter " data-level="1.4.1.8" data-path="../practice/dashboard-addon-installation.html">
<a href="../practice/dashboard-addon-installation.html">
4.1.8 安装dashboard插件
</a>
</li>
<li class="chapter " data-level="1.4.1.9" data-path="../practice/heapster-addon-installation.html">
<a href="../practice/heapster-addon-installation.html">
4.1.9 安装heapster插件
</a>
</li>
<li class="chapter " data-level="1.4.1.10" data-path="../practice/efk-addon-installation.html">
<a href="../practice/efk-addon-installation.html">
4.1.10 安装EFK插件
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="1.4.2" data-path="../practice/service-discovery-and-loadbalancing.html">
<a href="../practice/service-discovery-and-loadbalancing.html">
4.2 服务发现与负载均衡
</a>
<ul class="articles">
<li class="chapter " data-level="1.4.2.1" data-path="../practice/traefik-ingress-installation.html">
<a href="../practice/traefik-ingress-installation.html">
4.2.1 安装Traefik ingress
</a>
</li>
<li class="chapter " data-level="1.4.2.2" data-path="../practice/distributed-load-test.html">
<a href="../practice/distributed-load-test.html">
4.2.2 分布式负载测试
</a>
</li>
<li class="chapter " data-level="1.4.2.3" data-path="../practice/network-and-cluster-perfermance-test.html">
<a href="../practice/network-and-cluster-perfermance-test.html">
4.2.3 网络和集群性能测试
</a>
</li>
<li class="chapter " data-level="1.4.2.4" data-path="../practice/edge-node-configuration.html">
<a href="../practice/edge-node-configuration.html">
4.2.4 边缘节点配置
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="1.4.3" data-path="../practice/operation.html">
<a href="../practice/operation.html">
4.3 运维管理
</a>
<ul class="articles">
<li class="chapter " data-level="1.4.3.1" data-path="../practice/service-rolling-update.html">
<a href="../practice/service-rolling-update.html">
4.3.1 服务滚动升级
</a>
</li>
<li class="chapter " data-level="1.4.3.2" data-path="../practice/app-log-collection.html">
<a href="../practice/app-log-collection.html">
4.3.2 应用日志收集
</a>
</li>
<li class="chapter " data-level="1.4.3.3" data-path="../practice/configuration-best-practice.html">
<a href="../practice/configuration-best-practice.html">
4.3.3 配置最佳实践
</a>
</li>
<li class="chapter " data-level="1.4.3.4" data-path="../practice/monitor.html">
<a href="../practice/monitor.html">
4.3.4 集群及应用监控
</a>
</li>
<li class="chapter " data-level="1.4.3.5" data-path="../practice/jenkins-ci-cd.html">
<a href="../practice/jenkins-ci-cd.html">
4.3.5 使用Jenkins进行持续构建与发布
</a>
</li>
<li class="chapter " data-level="1.4.3.6" data-path="../practice/data-persistence-problem.html">
<a href="../practice/data-persistence-problem.html">
4.3.6 数据持久化问题
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="1.4.4" data-path="../practice/storage.html">
<a href="../practice/storage.html">
4.4 存储管理
</a>
<ul class="articles">
<li class="chapter " data-level="1.4.4.1" data-path="../practice/glusterfs.html">
<a href="../practice/glusterfs.html">
4.4.1 GlusterFS
</a>
<ul class="articles">
<li class="chapter " data-level="1.4.4.1.1" data-path="../practice/using-glusterfs-for-persistent-storage.html">
<a href="../practice/using-glusterfs-for-persistent-storage.html">
4.4.1.1 使用GlusterFS做持久化存储
</a>
</li>
<li class="chapter " data-level="1.4.4.1.2" data-path="../practice/storage-for-containers-using-glusterfs-with-openshift.html">
<a href="../practice/storage-for-containers-using-glusterfs-with-openshift.html">
4.4.1.2 在OpenShift中使用GlusterFS做持久化存储
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="1.4.4.2" data-path="../practice/cephfs.html">
<a href="../practice/cephfs.html">
4.4.2 CephFS
</a>
<ul class="articles">
<li class="chapter " data-level="1.4.4.2.1" data-path="../practice/using-ceph-for-persistent-storage.html">
<a href="../practice/using-ceph-for-persistent-storage.html">
4.4.2.1 使用Ceph做持久化存储
</a>
</li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
<li class="chapter " data-level="1.5" data-path="../usecases/">
<a href="../usecases/">
5. 领域应用
</a>
<ul class="articles">
<li class="chapter " data-level="1.5.1" data-path="../usecases/microservices.html">
<a href="../usecases/microservices.html">
5.1 微服务架构
</a>
<ul class="articles">
<li class="chapter " data-level="1.5.1.1" data-path="../usecases/istio.html">
<a href="../usecases/istio.html">
5.1.1 Istio
</a>
<ul class="articles">
<li class="chapter " data-level="1.5.1.1.1" data-path="../usecases/istio-installation.html">
<a href="../usecases/istio-installation.html">
5.1.1.1 安装istio
</a>
</li>
<li class="chapter " data-level="1.5.1.1.2" data-path="../usecases/configuring-request-routing.html">
<a href="../usecases/configuring-request-routing.html">
5.1.1.2 配置请求的路由规则
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="1.5.1.2" data-path="../usecases/linkerd.html">
<a href="../usecases/linkerd.html">
5.1.2 Linkerd
</a>
<ul class="articles">
<li class="chapter " data-level="1.5.1.2.1" data-path="../usecases/linkerd-user-guide.html">
<a href="../usecases/linkerd-user-guide.html">
5.1.2.1 Linkerd 使用指南
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="1.5.1.3" data-path="../usecases/service-discovery-in-microservices.html">
<a href="../usecases/service-discovery-in-microservices.html">
5.1.3 微服务中的服务发现
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="1.5.2" data-path="../usecases/big-data.html">
<a href="../usecases/big-data.html">
5.2 大数据
</a>
<ul class="articles">
2017-08-30 14:20:52 +08:00
<li class="chapter " data-level="1.5.2.1" data-path="../usecases/spark-standalone-on-kubernetes.html">
2017-08-30 14:20:52 +08:00
<a href="../usecases/spark-standalone-on-kubernetes.html">
2017-08-30 14:20:52 +08:00
5.2.1 Spark standalone on Kubernetes
</a>
2017-08-31 14:23:44 +08:00
</li>
<li class="chapter " data-level="1.5.2.2" data-path="../usecases/support-spark-natively-in-kubernetes.html">
<a href="../usecases/support-spark-natively-in-kubernetes.html">
5.2.2 运行支持kubernetes原生调度的Spark程序
</a>
</li>
</ul>
2017-08-30 16:52:33 +08:00
</li>
<li class="chapter " data-level="1.5.3" data-path="../usecases/serverless.html">
<a href="../usecases/serverless.html">
5.3 Serverless架构
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="1.6" data-path="../develop/">
<a href="../develop/">
6. 开发指南
</a>
<ul class="articles">
<li class="chapter " data-level="1.6.1" data-path="../develop/developing-environment.html">
<a href="../develop/developing-environment.html">
6.1 开发环境搭建
</a>
</li>
<li class="chapter " data-level="1.6.2" data-path="../develop/testing.html">
<a href="../develop/testing.html">
6.2 单元测试和集成测试
</a>
</li>
<li class="chapter " data-level="1.6.3" data-path="../develop/client-go-sample.html">
<a href="../develop/client-go-sample.html">
6.3 client-go示例
</a>
</li>
<li class="chapter " data-level="1.6.4" data-path="../develop/contribute.html">
<a href="../develop/contribute.html">
6.4 社区贡献
</a>
</li>
</ul>
</li>
<li class="chapter " data-level="1.7" data-path="../appendix/">
<a href="../appendix/">
7. 附录
</a>
<ul class="articles">
<li class="chapter " data-level="1.7.1" data-path="../appendix/docker-best-practice.html">
<a href="../appendix/docker-best-practice.html">
7.1 Docker最佳实践
</a>
</li>
<li class="chapter " data-level="1.7.2" data-path="../appendix/issues.html">
<a href="../appendix/issues.html">
7.2 问题记录
</a>
</li>
<li class="chapter " data-level="1.7.3" data-path="../appendix/tricks.html">
<a href="../appendix/tricks.html">
7.3 使用技巧
</a>
</li>
</ul>
</li>
<li class="divider"></li>
<li>
<a href="https://www.gitbook.com" target="blank" class="gitbook-link">
本書使用 GitBook 釋出
</a>
</li>
</ul>
</nav>
</div>
<div class="book-body">
<div class="body-inner">
<div class="book-header" role="navigation">
<!-- Title -->
<h1>
<i class="fa fa-circle-o-notch fa-spin"></i>
<a href=".." >3.5.2 迁移传统应用到kubernetes中——以Hadoop YARN为例</a>
</h1>
</div>
<div class="page-wrapper" tabindex="-1" role="main">
<div class="page-inner">
<div class="search-plus" id="book-search-results">
<div class="search-noresults">
<section class="normal markdown-section">
<h1 id="&#x8FC1;&#x79FB;&#x4F20;&#x7EDF;&#x5E94;&#x7528;&#x5230;kubernetes&#x6B65;&#x9AA4;&#x8BE6;&#x89E3;&#x2014;&#x2014;&#x4EE5;hadoop-yarn&#x4E3A;&#x4F8B;">&#x8FC1;&#x79FB;&#x4F20;&#x7EDF;&#x5E94;&#x7528;&#x5230;Kubernetes&#x6B65;&#x9AA4;&#x8BE6;&#x89E3;&#x2014;&#x2014;&#x4EE5;Hadoop YARN&#x4E3A;&#x4F8B;</h1>
<p>&#x672C;&#x6587;&#x6863;&#x4E0D;&#x662F;&#x8BF4;&#x660E;&#x5982;&#x4F55;&#x5728; kubernetes &#x4E2D;&#x5F00;&#x53D1;&#x548C;&#x90E8;&#x7F72;&#x5E94;&#x7528;&#x7A0B;&#x5E8F;&#xFF0C;&#x5982;&#x679C;&#x60A8;&#x60F3;&#x8981;&#x76F4;&#x63A5;&#x5F00;&#x53D1;&#x5E94;&#x7528;&#x7A0B;&#x5E8F;&#x5728; kubernetes &#x4E2D;&#x8FD0;&#x884C;&#x53EF;&#x4EE5;&#x53C2;&#x8003; <a href="deploy-applications-in-kubernetes.html">&#x9002;&#x7528;&#x4E8E;kubernetes&#x7684;&#x5E94;&#x7528;&#x5F00;&#x53D1;&#x90E8;&#x7F72;&#x6D41;&#x7A0B;</a>&#x3002;</p>
<p>&#x672C;&#x6587;&#x65E8;&#x5728;&#x8BF4;&#x660E;&#x5982;&#x4F55;&#x5C06;&#x5DF2;&#x6709;&#x7684;&#x5E94;&#x7528;&#x7A0B;&#x5E8F;&#x5C24;&#x5176;&#x662F;&#x4F20;&#x7EDF;&#x7684;&#x5206;&#x5E03;&#x5F0F;&#x5E94;&#x7528;&#x7A0B;&#x5E8F;&#x8FC1;&#x79FB;&#x5230; kubernetes &#x4E2D;&#x3002;&#x5982;&#x679C;&#x8BE5;&#x7C7B;&#x5E94;&#x7528;&#x7A0B;&#x5E8F;&#x7B26;&#x5408;&#x4E91;&#x539F;&#x751F;&#x5E94;&#x7528;&#x89C4;&#x8303;&#xFF08;&#x5982;12&#x56E0;&#x7D20;&#x6CD5;&#x5219;&#xFF09;&#x7684;&#x8BDD;&#xFF0C;&#x90A3;&#x4E48;&#x8FC1;&#x79FB;&#x4F1A;&#x6BD4;&#x8F83;&#x987A;&#x5229;&#xFF0C;&#x5426;&#x5219;&#x4F1A;&#x9047;&#x5230;&#x4E00;&#x4E9B;&#x9EBB;&#x70E6;&#x751A;&#x81F3;&#x662F;&#x963B;&#x788D;&#x3002;&#x5177;&#x4F53;&#x8BF7;&#x53C2;&#x8003; <a href="https://github.com/rootsongjc/migrating-to-cloud-native-application-architectures" target="_blank">&#x8FC1;&#x79FB;&#x81F3;&#x4E91;&#x539F;&#x751F;&#x5E94;&#x7528;&#x67B6;&#x6784;</a>&#x3002;</p>
<p>&#x63A5;&#x4E0B;&#x6765;&#x6211;&#x4EEC;&#x5C06;&#x4EE5; Spark on YARN with kubernetes &#x4E3A;&#x4F8B;&#x6765;&#x8BF4;&#x660E;&#xFF0C;&#x8BE5;&#x4F8B;&#x5B50;&#x8DB3;&#x591F;&#x590D;&#x6742;&#x4E5F;&#x5F88;&#x6709;&#x5178;&#x578B;&#x6027;&#xFF0C;&#x4E86;&#x89E3;&#x4E86;&#x8FD9;&#x4E2A;&#x4F8B;&#x5B50;&#x53EF;&#x4EE5;&#x5E2E;&#x52A9;&#x5927;&#x5BB6;&#x5C06;&#x81EA;&#x5DF1;&#x7684;&#x5E94;&#x7528;&#x8FC1;&#x79FB;&#x5230; kubernetes &#x96C6;&#x7FA4;&#x4E0A;&#x53BB;&#xFF0C;&#x4EE3;&#x7801;&#x548C;&#x914D;&#x7F6E;&#x6587;&#x4EF6;&#x53EF;&#x4EE5;&#x5728; <a href="https://github.com/rootsongjc/kube-yarn" target="_blank">&#x8FD9;&#x91CC;</a> &#x627E;&#x5230;&#xFF08;&#x672C;&#x6587;&#x4E2D;&#x52A0;&#x5165; Spark &#x7684;&#x914D;&#x7F6E;&#xFF0C;&#x4EE3;&#x7801;&#x4E2D;&#x5E76;&#x6CA1;&#x6709;&#x5305;&#x542B;&#xFF0C;&#x8BFB;&#x8005;&#x53EF;&#x4EE5;&#x81EA;&#x5DF1;&#x914D;&#x7F6E;&#xFF09;&#x3002;</p>
<p>&#x4E0B;&#x56FE;&#x5373;&#x6574;&#x4E2A;&#x67B6;&#x6784;&#x7684;&#x793A;&#x610F;&#x56FE;&#xFF0C;&#x4EE3;&#x7801;&#x548C;&#x8BE6;&#x7EC6;&#x914D;&#x7F6E;&#x6587;&#x4EF6;&#x8BF7;&#x53C2;&#x8003; <a href="https://github.com/rootsongjc/kube-yarn" target="_blank">kube-yarn</a>&#xFF08;&#x4E0D;&#x5305;&#x542B; ingress&#x3001;spark &#x914D;&#x7F6E;&#xFF09;&#xFF0C;&#x6240;&#x6709;&#x7684;&#x8FDB;&#x7A0B;&#x7BA1;&#x7406;&#x548C;&#x5BB9;&#x5668;&#x6269;&#x5BB9;&#x76F4;&#x63A5;&#x4F7F;&#x7528; Makefile&#x3002;</p>
<figure id="fig1.3.5.2.1"><img src="../images/spark-on-yarn-with-kubernetes.png" alt="spark on yarn with kubernetes"><figcaption>Figure: spark on yarn with kubernetes</figcaption></figure>
<p><strong>&#x6CE8;&#x610F;&#xFF1A; &#x8BE5;&#x4F8B;&#x5B50;&#x4EC5;&#x7528;&#x6765;&#x8BF4;&#x660E;&#x5177;&#x4F53;&#x7684;&#x6B65;&#x9AA4;&#x5212;&#x5206;&#x548C;&#x590D;&#x6742;&#x6027;&#xFF0C;&#x5728;&#x751F;&#x4EA7;&#x73AF;&#x5883;&#x5E94;&#x7528;&#x8FD8;&#x6709;&#x5F85;&#x9A8C;&#x8BC1;&#xFF0C;&#x8BF7;&#x8C28;&#x614E;&#x4F7F;&#x7528;&#x3002;</strong></p>
<h2 id="&#x672F;&#x8BED;">&#x672F;&#x8BED;</h2>
<p>&#x5BF9;&#x4E8E;&#x4E3A;&#x66FE;&#x63A5;&#x89E6;&#x8FC7; kubernetes &#x6216;&#x5BF9;&#x4E91;&#x5E73;&#x53F0;&#x7684;&#x6280;&#x672F;&#x7EC6;&#x8282;&#x4E0D;&#x592A;&#x4E86;&#x89E3;&#x7684;&#x4EBA;&#x6765;&#x8BF4;&#xFF0C;&#x5982;&#x4F55;&#x5C06;&#x5E94;&#x7528;&#x8FC1;&#x79FB;&#x5230; kubernetes &#x4E2D;&#x53EF;&#x80FD;&#x662F;&#x4E2A;&#x5934;&#x75BC;&#x7684;&#x95EE;&#x9898;&#xFF0C;&#x5728;&#x884C;&#x52A8;&#x4E4B;&#x524D;&#x6709;&#x5FC5;&#x8981;&#x5148;&#x4E86;&#x89E3;&#x6574;&#x4E2A;&#x8FC7;&#x7A0B;&#x4E2D;&#x9700;&#x8981;&#x7528;&#x5230;&#x54EA;&#x4E9B;&#x6982;&#x5FF5;&#x548C;&#x672F;&#x8BED;&#xFF0C;&#x6709;&#x52A9;&#x4E8E;&#x5927;&#x5BB6;&#x5728;&#x884C;&#x52A8;&#x4E2D;&#x8FBE;&#x6210;&#x5171;&#x8BC6;&#x3002;</p>
<p>&#x8FC7;&#x7A0B;&#x4E2D;&#x53EF;&#x80FD;&#x7528;&#x5230;&#x7684;&#x6982;&#x5FF5;&#x548C;&#x672F;&#x8BED;&#x521D;&#x6B65;&#x6574;&#x7406;&#x5982;&#x4E0B;&#xFF1A;</p>
<figure id="fig1.3.5.2.2"><img src="../images/terms-in-kubernetes-app-deployment.png" alt="Terms"><figcaption>Figure: Terms</figcaption></figure>
<p>&#x4E3A;&#x4E86;&#x8BB2;&#x89E3;&#x6574;&#x6539;&#x8FC7;&#x7A0B;&#x548C;&#x5177;&#x4F53;&#x7EC6;&#x8282;&#xFF0C;&#x6211;&#x4EEC;&#x6240;&#x6709;&#x64CD;&#x4F5C;&#x90FD;&#x662F;&#x901A;&#x8FC7;&#x547D;&#x4EE4;&#x624B;&#x52A8;&#x5B8C;&#x6210;&#xFF0C;&#x4E0D;&#x4F7F;&#x7528;&#x81EA;&#x52A8;&#x5316;&#x5DE5;&#x5177;&#x3002;&#x5F53;&#x60A8;&#x5145;&#x5206;&#x4E86;&#x89E3;&#x5230;&#x5176;&#x4E2D;&#x7684;&#x7EC6;&#x8282;&#x540E;&#x53EF;&#x4EE5;&#x901A;&#x8FC7;&#x81EA;&#x52A8;&#x5316;&#x5DE5;&#x5177;&#x6765;&#x4F18;&#x5316;&#x8BE5;&#x8FC7;&#x7A0B;&#xFF0C;&#x4EE5;&#x4F7F;&#x5176;&#x66F4;&#x52A0;&#x81EA;&#x52A8;&#x548C;&#x9AD8;&#x6548;&#xFF0C;&#x540C;&#x65F6;&#x51CF;&#x5C11;&#x56E0;&#x4E3A;&#x4EBA;&#x4E3A;&#x64CD;&#x4F5C;&#x5931;&#x8BEF;&#x5BFC;&#x81F4;&#x7684;&#x8FC1;&#x79FB;&#x5931;&#x8D25;&#x3002;</p>
<h2 id="&#x8FC1;&#x79FB;&#x5E94;&#x7528;">&#x8FC1;&#x79FB;&#x5E94;&#x7528;</h2>
<figure id="fig1.3.5.2.3"><img src="../images/migrating-hadoop-yarn-to-kubernetes.png" alt="&#x5206;&#x89E3;&#x6B65;&#x9AA4;&#x89E3;&#x6790;"><figcaption>Figure: &#x5206;&#x89E3;&#x6B65;&#x9AA4;&#x89E3;&#x6790;</figcaption></figure>
<p>&#x6574;&#x4E2A;&#x8FC1;&#x79FB;&#x8FC7;&#x7A0B;&#x5206;&#x4E3A;&#x5982;&#x4E0B;&#x51E0;&#x4E2A;&#x6B65;&#x9AA4;&#xFF1A;</p>
<ol>
<li><p><strong>&#x5C06;&#x539F;&#x6709;&#x5E94;&#x7528;&#x62C6;&#x89E3;&#x4E3A;&#x670D;&#x52A1;</strong></p>
<p>&#x6211;&#x4EEC;&#x4E0D;&#x662F;&#x4E00;&#x4E0A;&#x6765;&#x5C31;&#x5F00;&#x59CB;&#x505A;&#x955C;&#x50CF;&#xFF0C;&#x5199;&#x914D;&#x7F6E;&#xFF0C;&#x800C;&#x662F;&#x5E94;&#x8BE5;&#x5148;&#x68B3;&#x7406;&#x4E0B;&#x8981;&#x8FC1;&#x79FB;&#x7684;&#x5E94;&#x7528;&#x4E2D;&#x6709;&#x54EA;&#x4E9B;&#x53EF;&#x4EE5;&#x4F5C;&#x4E3A;&#x670D;&#x52A1;&#x8FD0;&#x884C;&#xFF0C;&#x54EA;&#x4E9B;&#x662F;&#x53D8;&#x7684;&#xFF0C;&#x54EA;&#x4E9B;&#x662F;&#x4E0D;&#x53D8;&#x7684;&#x90E8;&#x5206;&#x3002;</p>
<p>&#x670D;&#x52A1;&#x5212;&#x5206;&#x7684;&#x539F;&#x5219;&#x662F;&#x6700;&#x5C0F;&#x53EF;&#x53D8;&#x539F;&#x5219;&#xFF0C;&#x8FD9;&#x4E2A;&#x540C;&#x6837;&#x9002;&#x7528;&#x4E8E;&#x955C;&#x50CF;&#x5236;&#x4F5C;&#xFF0C;&#x5C06;&#x670D;&#x52A1;&#x4E2D;&#x4E0D;&#x53D8;&#x7684;&#x90E8;&#x5206;&#x7F16;&#x8BD1;&#x5230;&#x540C;&#x4E00;&#x4E2A;&#x955C;&#x50CF;&#x4E2D;&#x3002;</p>
<p>&#x5BF9;&#x4E8E;&#x50CF; Spark on YARN &#x8FD9;&#x6837;&#x590D;&#x6742;&#x7684;&#x5E94;&#x7528;&#xFF0C;&#x53EF;&#x4EE5;&#x5C06;&#x5176;&#x5212;&#x5206;&#x4E3A;&#x4E09;&#x5927;&#x7C7B;&#x670D;&#x52A1;&#xFF1A;</p>
<ul>
<li>ResourceManager</li>
<li>NodeManager</li>
<li>Spark client</li>
</ul>
</li>
<li><p><strong>&#x5236;&#x4F5C;&#x955C;&#x50CF;</strong></p>
<p>&#x6839;&#x636E;&#x62C6;&#x89E3;&#x51FA;&#x6765;&#x7684;&#x670D;&#x52A1;&#xFF0C;&#x6211;&#x4EEC;&#x9700;&#x8981;&#x5236;&#x4F5C;&#x4E24;&#x4E2A;&#x955C;&#x50CF;&#xFF1A;</p>
<ul>
<li>Hadoop</li>
<li>Spark (From hadoop docker image)</li>
</ul>
<p>&#x56E0;&#x4E3A;&#x6211;&#x4EEC;&#x8FD0;&#x884C;&#x7684;&#x662F; Spark on YARN&#xFF0C;&#x56E0;&#x6B64; Spark &#x4F9D;&#x8D56;&#x4E0E; Hadoop &#x955C;&#x50CF;&#xFF0C;&#x6211;&#x4EEC;&#x5728; Spark &#x7684;&#x57FA;&#x7840;&#x4E0A;&#x5305;&#x88C5;&#x4E86;&#x4E00;&#x4E2A; web service &#x4F5C;&#x4E3A;&#x670D;&#x52A1;&#x542F;&#x52A8;&#x3002;</p>
<p>&#x955C;&#x50CF;&#x5236;&#x4F5C;&#x8FC7;&#x7A0B;&#x4E2D;&#x4E0D;&#x9700;&#x8981;&#x5728; Dockerfile &#x4E2D;&#x6307;&#x5B9A; Entrypoint &#x548C; CMD&#xFF0C;&#x8FD9;&#x4E9B;&#x90FD;&#x662F;&#x5728; kubernetes &#x7684; YAML &#x6587;&#x4EF6;&#x4E2D;&#x6307;&#x5B9A;&#x7684;&#x3002;</p>
<p>Hadoop YARN &#x7684; Dockerfile &#x53C2;&#x8003;&#x5982;&#x4E0B;&#x914D;&#x7F6E;&#x3002;</p>
<pre><code class="lang-dockerfile"><span class="hljs-keyword">FROM</span> my-docker-repo/jdk:<span class="hljs-number">7</span>u8<span class="hljs-number">0
</span>
<span class="hljs-comment"># Add native libs</span>
ARG HADOOP_VERSION=<span class="hljs-number">2.6</span>.<span class="hljs-number">0</span>-cdh5.<span class="hljs-number">5.2</span>
<span class="hljs-comment">## Prefer to download from server not use local storage</span>
ADD hadoop-${HADOOP_VERSION}.tar.gz /usr/local
<span class="hljs-keyword">ADD</span> <span class="bash">./lib/* /usr/<span class="hljs-built_in">local</span>/hadoop-<span class="hljs-variable">${HADOOP_VERSION}</span>/lib/native/
</span><span class="hljs-keyword">ADD</span> <span class="bash">./jars/* /usr/<span class="hljs-built_in">local</span>/hadoop-<span class="hljs-variable">${HADOOP_VERSION}</span>/share/hadoop/yarn/
</span><span class="hljs-keyword">ENV</span> HADOOP_PREFIX=/usr/local/hadoop \
HADOOP_COMMON_HOME=/usr/local/hadoop \
HADOOP_HDFS_HOME=/usr/local/hadoop \
HADOOP_MAPRED_HOME=/usr/local/hadoop \
HADOOP_YARN_HOME=/usr/local/hadoop \
HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop \
YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop \
PATH=${PATH}:/usr/local/hadoop/bin
<span class="hljs-keyword">RUN</span> <span class="bash">\
<span class="hljs-built_in">cd</span> /usr/<span class="hljs-built_in">local</span> &amp;&amp; ln <span class="hljs-_">-s</span> ./hadoop-<span class="hljs-variable">${HADOOP_VERSION}</span> hadoop &amp;&amp; \
rm <span class="hljs-_">-f</span> <span class="hljs-variable">${HADOOP_PREFIX}</span>/logs/*
</span>
<span class="hljs-keyword">WORKDIR</span> <span class="bash"><span class="hljs-variable">$HADOOP_PREFIX</span>
</span>
<span class="hljs-comment"># Hdfs ports</span>
<span class="hljs-keyword">EXPOSE</span> <span class="hljs-number">50010</span> <span class="hljs-number">50020</span> <span class="hljs-number">50070</span> <span class="hljs-number">50075</span> <span class="hljs-number">50090</span> <span class="hljs-number">8020</span> <span class="hljs-number">9000</span>
<span class="hljs-comment"># Mapred ports</span>
<span class="hljs-keyword">EXPOSE</span> <span class="hljs-number">19888</span>
<span class="hljs-comment">#Yarn ports</span>
<span class="hljs-keyword">EXPOSE</span> <span class="hljs-number">8030</span> <span class="hljs-number">8031</span> <span class="hljs-number">8032</span> <span class="hljs-number">8033</span> <span class="hljs-number">8040</span> <span class="hljs-number">8042</span> <span class="hljs-number">8088</span>
<span class="hljs-comment">#Other ports</span>
<span class="hljs-keyword">EXPOSE</span> <span class="hljs-number">49707</span> <span class="hljs-number">2122</span>
</code></pre>
</li>
<li><p><strong>&#x51C6;&#x5907;&#x5E94;&#x7528;&#x7684;&#x914D;&#x7F6E;&#x6587;&#x4EF6;</strong></p>
<p>&#x56E0;&#x4E3A;&#x6211;&#x4EEC;&#x53EA;&#x5236;&#x4F5C;&#x4E86;&#x4E00;&#x4E2A; Hadoop &#x7684;&#x955C;&#x50CF;&#xFF0C;&#x800C;&#x9700;&#x8981;&#x542F;&#x52A8;&#x4E24;&#x4E2A;&#x670D;&#x52A1;&#xFF0C;&#x8FD9;&#x5C31;&#x8981;&#x6C42;&#x5728;&#x670D;&#x52A1;&#x542F;&#x52A8;&#x7684;&#x65F6;&#x5019;&#x5FC5;&#x987B;&#x52A0;&#x8F7D;&#x4E0D;&#x540C;&#x7684;&#x914D;&#x7F6E;&#x6587;&#x4EF6;&#xFF0C;&#x73B0;&#x5728;&#x6211;&#x4EEC;&#x53EA;&#x9700;&#x8981;&#x51C6;&#x5907;&#x4E24;&#x4E2A;&#x670D;&#x52A1;&#x4E2D;&#x9700;&#x8981;&#x540C;&#x65F6;&#x7528;&#x7684;&#x7684;&#x914D;&#x7F6E;&#x7684;&#x90E8;&#x5206;&#x3002;</p>
<p>YARN &#x4F9D;&#x8D56;&#x7684;&#x914D;&#x7F6E;&#x5728; <code>artifacts</code> &#x76EE;&#x5F55;&#x4E0B;&#xFF0C;&#x5305;&#x542B;&#x4EE5;&#x4E0B;&#x6587;&#x4EF6;&#xFF1A;</p>
<pre><code>bootstrap.sh
capacity-scheduler.xml
container-executor.cfg
core-site.xml
hadoop-env.sh
hdfs-site.xml
log4j.properties
mapred-site.xml
nodemanager_exclude.txt
slaves
start-yarn-nm.sh
start-yarn-rm.sh
yarn-env.sh
yarn-site.xml
</code></pre><p>&#x5176;&#x4E2D;&#x4F5C;&#x4E3A; bootstrap &#x542F;&#x52A8;&#x811A;&#x672C;&#x7684; <code>bootstrap.sh</code> &#x4E5F;&#x5305;&#x542B;&#x5728;&#x8BE5;&#x76EE;&#x5F55;&#x4E0B;&#xFF0C;&#x8BE5;&#x811A;&#x672C;&#x7684;&#x5982;&#x4F55;&#x7F16;&#x5199;&#x8BF7;&#x89C1;&#x4E0B;&#x6587;&#x3002;</p>
</li>
<li><p><strong>Kubernetes YAML &#x6587;&#x4EF6;</strong></p>
<p>&#x6839;&#x636E;&#x4E1A;&#x52A1;&#x7684;&#x7279;&#x6027;&#x9009;&#x62E9;&#x6700;&#x9002;&#x5408;&#x7684; kubernetes &#x7684;&#x8D44;&#x6E90;&#x5BF9;&#x8C61;&#x6765;&#x8FD0;&#x884C;&#xFF0C;&#x56E0;&#x4E3A;&#x5728; YARN &#x4E2D; NodeManager &#x9700;&#x8981;&#x4F7F;&#x7528;&#x4E3B;&#x673A;&#x540D;&#x5411; ResourceManger &#x6CE8;&#x518C;&#xFF0C;&#x56E0;&#x6B64;&#x9700;&#x8981;&#x6CBF;&#x7528; YARN &#x539F;&#x6709;&#x7684;&#x670D;&#x52A1;&#x53D1;&#x73B0;&#x65B9;&#x5F0F;&#xFF0C;&#x4F7F;&#x7528; headless service &#x548C; StatefulSet &#x8D44;&#x6E90;&#x3002;&#x66F4;&#x591A;&#x8D44;&#x6599;&#x8BF7;&#x53C2;&#x8003; <a href="../concepts/statefulset.html">StatefulSet</a>&#x3002;</p>
<p>&#x6240;&#x6709;&#x7684; Kubernetes YAML &#x914D;&#x7F6E;&#x6587;&#x4EF6;&#x5B58;&#x50A8;&#x5728; <code>manifest</code> &#x76EE;&#x5F55;&#x4E0B;&#xFF0C;&#x5305;&#x62EC;&#x5982;&#x4E0B;&#x914D;&#x7F6E;&#xFF1A;</p>
<ul>
<li>yarn-cluster &#x7684; namespace &#x914D;&#x7F6E;</li>
<li>Spark&#x3001;ResourceManager&#x3001;NodeManager &#x7684; headless service &#x548C; StatefulSet &#x914D;&#x7F6E;</li>
<li>&#x9700;&#x8981;&#x66B4;&#x9732;&#x5230; kubernetes &#x96C6;&#x7FA4;&#x5916;&#x90E8;&#x7684; ingress &#x914D;&#x7F6E;&#xFF08;ResourceManager &#x7684; Web&#xFF09;</li>
</ul>
<pre><code>kube-yarn-ingress.yaml
spark-statefulset.yaml
yarn-cluster-namespace.yaml
yarn-nm-statefulset.yaml
yarn-rm-statefulset.yaml
</code></pre></li>
<li><p><strong>Bootstrap &#x811A;&#x672C;</strong></p>
<p>Bootstrap &#x811A;&#x672C;&#x7684;&#x4F5C;&#x7528;&#x662F;&#x5728;&#x542F;&#x52A8;&#x65F6;&#x6839;&#x636E; Pod &#x7684;&#x73AF;&#x5883;&#x53D8;&#x91CF;&#x3001;&#x4E3B;&#x673A;&#x540D;&#x6216;&#x5176;&#x4ED6;&#x53EF;&#x4EE5;&#x533A;&#x5206;&#x4E0D;&#x540C; Pod &#x548C;&#x5C06;&#x542F;&#x52A8;&#x89D2;&#x8272;&#x7684;&#x53D8;&#x91CF;&#x6765;&#x4FEE;&#x6539;&#x914D;&#x7F6E;&#x6587;&#x4EF6;&#x548C;&#x542F;&#x52A8;&#x670D;&#x52A1;&#x5E94;&#x7528;&#x3002;</p>
<p>&#x8BE5;&#x811A;&#x672C;&#x540C;&#x65F6;&#x5C06;&#x539F;&#x6765; YARN &#x7684;&#x65E5;&#x5FD7;&#x4F7F;&#x7528; stdout &#x8F93;&#x51FA;&#xFF0C;&#x4FBF;&#x4E8E;&#x4F7F;&#x7528; <code>kubectl logs</code> &#x67E5;&#x770B;&#x65E5;&#x5FD7;&#x6216;&#x5176;&#x4ED6;&#x65E5;&#x5FD7;&#x6536;&#x96C6;&#x5DE5;&#x5177;&#x8FDB;&#x884C;&#x65E5;&#x5FD7;&#x6536;&#x96C6;&#x3002;</p>
<p>&#x542F;&#x52A8;&#x811A;&#x672C; <code>bootstrap.sh</code> &#x8DDF; Hadoop &#x7684;&#x914D;&#x7F6E;&#x6587;&#x4EF6;&#x540C;&#x65F6;&#x4FDD;&#x5B58;&#x5728; <code>artifacts</code> &#x76EE;&#x5F55;&#x4E0B;&#x3002;</p>
<p>&#x8BE5;&#x811A;&#x672C;&#x6839;&#x636E; Pod &#x7684;&#x4E3B;&#x673A;&#x540D;&#xFF0C;&#x51B3;&#x5B9A;&#x5982;&#x4F55;&#x4FEE;&#x6539; Hadoop &#x7684;&#x914D;&#x7F6E;&#x6587;&#x4EF6;&#x548C;&#x542F;&#x52A8;&#x4F55;&#x79CD;&#x670D;&#x52A1;&#x3002;<code>bootstrap.sh</code> &#x6587;&#x4EF6;&#x7684;&#x90E8;&#x5206;&#x4EE3;&#x7801;&#x5982;&#x4E0B;&#xFF1A;</p>
<pre><code class="lang-bash"><span class="hljs-keyword">if</span> [[ <span class="hljs-string">&quot;<span class="hljs-variable">${HOSTNAME}</span>&quot;</span> =~ <span class="hljs-string">&quot;yarn-nm&quot;</span> ]]; <span class="hljs-keyword">then</span>
sed -i <span class="hljs-string">&apos;/&lt;\/configuration&gt;/d&apos;</span> <span class="hljs-variable">$HADOOP_PREFIX</span>/etc/hadoop/yarn-site.xml
cat &gt;&gt; <span class="hljs-variable">$HADOOP_PREFIX</span>/etc/hadoop/yarn-site.xml &lt;&lt;- EOM
&lt;property&gt;
&lt;name&gt;yarn.nodemanager.resource.memory-mb&lt;/name&gt;
&lt;value&gt;<span class="hljs-variable">${MY_MEM_LIMIT:-2048}</span>&lt;/value&gt;
&lt;/property&gt;
&lt;property&gt;
&lt;name&gt;yarn.nodemanager.resource.cpu-vcores&lt;/name&gt;
&lt;value&gt;<span class="hljs-variable">${MY_CPU_LIMIT:-2}</span>&lt;/value&gt;
&lt;/property&gt;
EOM
<span class="hljs-built_in">echo</span> <span class="hljs-string">&apos;&lt;/configuration&gt;&apos;</span> &gt;&gt; <span class="hljs-variable">$HADOOP_PREFIX</span>/etc/hadoop/yarn-site.xml
cp <span class="hljs-variable">${CONFIG_DIR}</span>/start-yarn-nm.sh <span class="hljs-variable">$HADOOP_PREFIX</span>/sbin/
<span class="hljs-built_in">cd</span> <span class="hljs-variable">$HADOOP_PREFIX</span>/sbin
chmod +x start-yarn-nm.sh
./start-yarn-nm.sh
<span class="hljs-keyword">fi</span>
<span class="hljs-keyword">if</span> [[ <span class="hljs-variable">$1</span> == <span class="hljs-string">&quot;-d&quot;</span> ]]; <span class="hljs-keyword">then</span>
until find <span class="hljs-variable">${HADOOP_PREFIX}</span>/logs -mmin -1 | egrep -q <span class="hljs-string">&apos;.*&apos;</span>; <span class="hljs-built_in">echo</span> <span class="hljs-string">&quot;`date`: Waiting for logs...&quot;</span> ; <span class="hljs-keyword">do</span> sleep 2 ; <span class="hljs-keyword">done</span>
tail -F <span class="hljs-variable">${HADOOP_PREFIX}</span>/logs/* &amp;
<span class="hljs-keyword">while</span> <span class="hljs-literal">true</span>; <span class="hljs-keyword">do</span> sleep 1000; <span class="hljs-keyword">done</span>
<span class="hljs-keyword">fi</span>
</code></pre>
<p>&#x4ECE;&#x8FD9;&#x90E8;&#x5206;&#x4E2D;&#x4EE3;&#x7801;&#x4E2D;&#x53EF;&#x4EE5;&#x770B;&#x5230;&#xFF0C;&#x5982;&#x679C; Pod &#x7684;&#x4E3B;&#x673A;&#x540D;&#x4E2D;&#x5305;&#x542B; <code>yarn-nm</code> &#x5B57;&#x6BB5;&#x5219;&#x5411; <code>yarn-site.xml</code> &#x914D;&#x7F6E;&#x6587;&#x4EF6;&#x4E2D;&#x589E;&#x52A0;&#x5982;&#x4E0B;&#x5185;&#x5BB9;&#xFF1A;</p>
<pre><code class="lang-xml"> <span class="hljs-tag">&lt;<span class="hljs-name">property</span>&gt;</span>
<span class="hljs-tag">&lt;<span class="hljs-name">name</span>&gt;</span>yarn.nodemanager.resource.memory-mb<span class="hljs-tag">&lt;/<span class="hljs-name">name</span>&gt;</span>
<span class="hljs-tag">&lt;<span class="hljs-name">value</span>&gt;</span>${MY_MEM_LIMIT:-2048}<span class="hljs-tag">&lt;/<span class="hljs-name">value</span>&gt;</span>
<span class="hljs-tag">&lt;/<span class="hljs-name">property</span>&gt;</span>
<span class="hljs-tag">&lt;<span class="hljs-name">property</span>&gt;</span>
<span class="hljs-tag">&lt;<span class="hljs-name">name</span>&gt;</span>yarn.nodemanager.resource.cpu-vcores<span class="hljs-tag">&lt;/<span class="hljs-name">name</span>&gt;</span>
<span class="hljs-tag">&lt;<span class="hljs-name">value</span>&gt;</span>${MY_CPU_LIMIT:-2}<span class="hljs-tag">&lt;/<span class="hljs-name">value</span>&gt;</span>
<span class="hljs-tag">&lt;/<span class="hljs-name">property</span>&gt;</span>
</code></pre>
<p>&#x5176;&#x4E2D; <code>MY_MEM_LIMIT</code> &#x548C; <code>MY_CPU_LIMIT</code> &#x662F; kubernetes YAML &#x4E2D;&#x5B9A;&#x4E49;&#x7684;&#x73AF;&#x5883;&#x53D8;&#x91CF;&#xFF0C;&#x8BE5;&#x73AF;&#x5883;&#x53D8;&#x91CF;&#x53C8;&#x662F;&#x5F15;&#x7528;&#x7684; Resource limit&#x3002;</p>
<p>&#x6240;&#x6709;&#x7684;&#x914D;&#x7F6E;&#x51C6;&#x5907;&#x5B8C;&#x6210;&#x540E;&#xFF0C;&#x6267;&#x884C; <code>start-yarn-nm.sh</code> &#x811A;&#x672C;&#x542F;&#x52A8; NodeManager&#x3002;</p>
<p>&#x5982;&#x679C; kubernetes YAML &#x4E2D;&#x7684; container CMD args &#x4E2D;&#x5305;&#x542B; <code>-d</code> &#x5219;&#x5728;&#x540E;&#x53F0;&#x8FD0;&#x884C; NodeManger &#x5E76; tail &#x8F93;&#x51FA; NodeManager &#x7684;&#x65E5;&#x5FD7;&#x5230;&#x6807;&#x51C6;&#x8F93;&#x51FA;&#x3002;</p>
</li>
<li><p><strong>ConfigMaps</strong></p>
<p>&#x5C06; Hadoop &#x7684;&#x914D;&#x7F6E;&#x6587;&#x4EF6;&#x548C; bootstrap &#x811A;&#x672C;&#x4F5C;&#x4E3A; ConfigMap &#x8D44;&#x6E90;&#x4FDD;&#x5B58;&#xFF0C;&#x7528;&#x4F5C; Pod &#x542F;&#x52A8;&#x65F6;&#x6302;&#x8F7D;&#x7684; volume&#x3002;</p>
<pre><code class="lang-bash">kubectl create configmap hadoop-config \
--from-file=artifacts/hadoop/bootstrap.sh \
--from-file=artifacts/hadoop/start-yarn-rm.sh \
--from-file=artifacts/hadoop/start-yarn-nm.sh \
--from-file=artifacts/hadoop/slaves \
--from-file=artifacts/hadoop/core-site.xml \
--from-file=artifacts/hadoop/hdfs-site.xml \
--from-file=artifacts/hadoop/mapred-site.xml \
--from-file=artifacts/hadoop/yarn-site.xml \
--from-file=artifacts/hadoop/capacity-scheduler.xml \
--from-file=artifacts/hadoop/container-executor.cfg \
--from-file=artifacts/hadoop/hadoop-env.sh \
--from-file=artifacts/hadoop/<span class="hljs-built_in">log</span>4j.properties \
--from-file=artifacts/hadoop/nodemanager_exclude.txt \
--from-file=artifacts/hadoop/yarn-env.sh
kubectl create configmap spark-config \
--from-file=artifacts/spark/spark-bootstrap.sh \
--from-file=artifacts/spark/spark-env.sh \
--from-file=artifacts/spark/spark-defaults.conf
</code></pre>
</li>
</ol>
<p>&#x6240;&#x6709;&#x7684;&#x914D;&#x7F6E;&#x5B8C;&#x6210;&#x540E;&#xFF0C;&#x53EF;&#x4EE5;&#x53EF;&#x4EE5;&#x4F7F;&#x7528; kubectl &#x547D;&#x4EE4;&#x6765;&#x542F;&#x52A8;&#x548C;&#x7BA1;&#x7406;&#x96C6;&#x7FA4;&#x4E86;&#xFF0C;&#x6211;&#x4EEC;&#x7F16;&#x5199;&#x4E86; Makefile&#xFF0C;&#x60A8;&#x53EF;&#x4EE5;&#x76F4;&#x63A5;&#x4F7F;&#x7528;&#x8BE5; Makefile &#x5C01;&#x88C5;&#x7684;&#x547D;&#x4EE4;&#x5B9E;&#x73B0;&#x90E8;&#x5206;&#x7684;&#x81EA;&#x52A8;&#x5316;&#x3002; </p>
<footer class="page-footer-ex"> <span class="page-footer-ex-copyright">for GitBook</span>&#xA0;&#xA0;&#xA0;&#xA0;&#xA0;&#xA0;&#xA0;&#xA0;&#xA0;&#xA0;&#xA0;<span class="page-footer-ex-footer-update">update
2017-08-21 18:23:34
</span></footer>
</section>
</div>
<div class="search-results">
<div class="has-results">
<h1 class="search-results-title"><span class='search-results-count'></span> results matching "<span class='search-query'></span>"</h1>
<ul class="search-results-list"></ul>
</div>
<div class="no-results">
<h1 class="search-results-title">No results matching "<span class='search-query'></span>"</h1>
</div>
</div>
</div>
</div>
</div>
</div>
<a href="deploy-applications-in-kubernetes.html" class="navigation navigation-prev " aria-label="Previous page: 3.5.1 适用于kubernetes的应用开发部署流程">
<i class="fa fa-angle-left"></i>
</a>
<a href="../practice/" class="navigation navigation-next " aria-label="Next page: 4. 最佳实践">
<i class="fa fa-angle-right"></i>
</a>
</div>
<script>
var gitbook = gitbook || [];
gitbook.push(function() {
2017-09-03 13:29:38 +08:00
gitbook.page.hasChanged({"page":{"title":"3.5.2 迁移传统应用到kubernetes中——以Hadoop YARN为例","level":"1.3.5.2","depth":3,"next":{"title":"4. 最佳实践","level":"1.4","depth":1,"path":"practice/index.md","ref":"practice/index.md","articles":[{"title":" 4.1 在CentOS上部署kubernetes1.6集群","level":"1.4.1","depth":2,"path":"practice/install-kbernetes1.6-on-centos.md","ref":"practice/install-kbernetes1.6-on-centos.md","articles":[{"title":"4.1.1 创建TLS证书和秘钥","level":"1.4.1.1","depth":3,"path":"practice/create-tls-and-secret-key.md","ref":"practice/create-tls-and-secret-key.md","articles":[]},{"title":"4.1.2 创建kubeconfig文件","level":"1.4.1.2","depth":3,"path":"practice/create-kubeconfig.md","ref":"practice/create-kubeconfig.md","articles":[]},{"title":"4.1.3 创建高可用etcd集群","level":"1.4.1.3","depth":3,"path":"practice/etcd-cluster-installation.md","ref":"practice/etcd-cluster-installation.md","articles":[]},{"title":"4.1.4 安装kubectl命令行工具","level":"1.4.1.4","depth":3,"path":"practice/kubectl-installation.md","ref":"practice/kubectl-installation.md","articles":[]},{"title":"4.1.5 部署master节点","level":"1.4.1.5","depth":3,"path":"practice/master-installation.md","ref":"practice/master-installation.md","articles":[]},{"title":"4.1.6 部署node节点","level":"1.4.1.6","depth":3,"path":"practice/node-installation.md","ref":"practice/node-installation.md","articles":[]},{"title":"4.1.7 安装kubedns插件","level":"1.4.1.7","depth":3,"path":"practice/kubedns-addon-installation.md","ref":"practice/kubedns-addon-installation.md","articles":[]},{"title":"4.1.8 安装dashboard插件","level":"1.4.1.8","depth":3,"path":"practice/dashboard-addon-installation.md","ref":"practice/dashboard-addon-installation.md","articles":[]},{"title":"4.1.9 安装heapster插件","level":"1.4.1.9","depth":3,"path":"practice/heapster-addon-installation.md","ref":"practice/heapster-addon-installation.md","articles":[]},{"title":"4.1.10 安装EFK插件","level":"1.4.1.10","depth":3,"path":"practice/efk-addon-installation.md","ref":"practice/efk-addon-installation.md","articles":[]}]},{"title":"4.2 服务发现与负载均衡","level":"1.4.2","depth":2,"path":"practice/service-discovery-and-loadbalancing.md","ref":"practice/service-discovery-and-loadbalancing.md","articles":[{"title":"4.2.1 安装Traefik ingress","level":"1.4.2.1","depth":3,"path":"practice/traefik-ingress-installation.md","ref":"practice/traefik-ingress-installation.md","articles":[]},{"title":"4.2.2 分布式负载测试","level":"1.4.2.2","depth":3,"path":"practice/distributed-load-test.md","ref":"practice/distributed-load-test.md","articles":[]},{"title":"4.2.3 网络和集群性能测试","level":"1.4.2.3","depth":3,"path":"practice/network-and-cluster-perfermance-test.md","ref":"practice/network-and-cluster-perfermance-test.md","articles":[]},{"title":"4.2.4 边缘节点配置","level":"1.4.2.4","depth":3,"path":"practice/edge-node-configuration.md","ref":"practice/edge-node-configuration.md","articles":[]}]},{"title":"4.3 运维管理","level":"1.4.3","depth":2,"path":"practice/operation.md","ref":"practice/operation.md","articles":[{"title":"4.3.1 服务滚动升级","level":"1.4.3.1","depth":3,"path":"practice/service-rolling-update.md","ref":"practice/service-rolling-update.md","articles":[]},{"title":"4.3.2 应用日志收集","level":"1.4.3.2","depth":3,"path":"practice/app-log-collection.md","ref":"practice/app-log-collection.md","articles":[]},{"title":"4.3.3 配置最佳实践","level":"1.4.3.3","depth":3,"path":"practice/configuration-best-practice.md","ref":"practice/configuration-best-practice.md","articles":[]},{"title":"4.3.4 集群及应用监控","level":"1.4.3.4","depth":3,"path":"practice/monitor.md","ref":"practice/monitor.md","articles":[]},{"title":"4.3.5 使用Jenkins进行持续构建与发布","level":"1.4.3.5","depth":3,"path":"practice/jenkins-ci-cd.md","ref":"practice/jenkins-ci-cd.md","articles":[]},{"title":"4.3.6 数据持久化问题","level":"1.4.3.6","depth":3,"path
});
</script>
</div>
<script src="../gitbook/gitbook.js"></script>
<script src="../gitbook/theme.js"></script>
<script src="../gitbook/gitbook-plugin-github/plugin.js"></script>
<script src="../gitbook/gitbook-plugin-splitter/splitter.js"></script>
<script src="../gitbook/gitbook-plugin-page-toc-button/plugin.js"></script>
<script src="../gitbook/gitbook-plugin-editlink/plugin.js"></script>
<script src="../gitbook/gitbook-plugin-search-plus/jquery.mark.min.js"></script>
<script src="../gitbook/gitbook-plugin-search-plus/search.js"></script>
<script src="../gitbook/gitbook-plugin-sharing/buttons.js"></script>
<script src="../gitbook/gitbook-plugin-fontsettings/fontsettings.js"></script>
</body>
</html>