diff --git a/README.md b/README.md index 74c7225..5b5052e 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,7 @@ ## 参考阅读 - 建议阅读 [rootsongjc-Kubernetes指南](https://github.com/rootsongjc/kubernetes-handbook) 原理和实践指南。 -- 建议阅读 [feisky-Kubernetes指南](https://github.com/feiskyer/kubernetes-handbook/blob/master/SUMMARY.md) 原理和部署章节。 +- 建议阅读 [feisky-Kubernetes指南](https://github.com/feiskyer/kubernetes-handbook/blob/master/zh/SUMMARY.md) 原理和部署章节。 - 建议阅读 [opsnull-安装教程](https://github.com/opsnull/follow-me-install-kubernetes-cluster) 二进制手工部署。 ## 版权 diff --git a/docs/guide/efk.md b/docs/guide/efk.md index 5f1cb45..47ad08c 100644 --- a/docs/guide/efk.md +++ b/docs/guide/efk.md @@ -1 +1,54 @@ ## EFK + +`EFK` 插件是`k8s`项目的一个日志解决方案,它包括三个组件:[Elasticsearch](),[Fluentd](),[Kibana]();Elasticsearch 是日志存储和日志搜索引擎,Fluentd 负载把`k8s`集群的日志发送给 Elasticsearch, Kibana 是可视化界面查看和检索存储在 Elasticsearch 的数据。 + +### 部署 + +下载官方最新[release](https://github.com/kubernetes/kubernetes/release),进入目录: `kubernetes/cluster/addons/fluentd-elasticsearch`,参考官方配置的基础上使用本项目`manifests/efk/`部署,以下为几点主要的修改: + ++ 官方提供的`kibana-deployment.yaml`中的参数`SERVER_BASEPATH`在k8s v1.8 版本以后部署需要按照本项目调整 ++ 修改官方docker镜像,方便国内下载加速 + +请使用`kubectl create -f /etc/ansible/manifests/efk/`进行安装 + +注意:Fluentd 是以 DaemonSet 形式运行且只会调度到有`beta.kubernetes.io/fluentd-ds-ready=true`标签的节点,所以对需要收集日志的节点逐个打上标签: + +``` bash +$ kubectl label nodes 192.168.1.2 beta.kubernetes.io/fluentd-ds-ready=true +node "192.168.1.2" labeled +``` + +### 验证 + +``` bash +kubectl get pods -n kube-system|grep -E 'elasticsearch|fluentd|kibana' +elasticsearch-logging-0 1/1 Running 0 19h +elasticsearch-logging-1 1/1 Running 0 19h +fluentd-es-v2.0.2-6c95c 1/1 Running 0 17h +fluentd-es-v2.0.2-f2xh8 1/1 Running 0 8h +fluentd-es-v2.0.2-pv5q5 1/1 Running 0 8h +kibana-logging-d5cffd7c6-9lz2p 1/1 Running 0 1m +``` +kibana Pod 第一次启动时会用较长时间(10-20分钟)来优化和 Cache 状态页面,可以查看 Pod 的日志观察进度,等待 `Ready` 状态 + +``` bash +$ kubectl logs -n kube-system kibana-logging-d5cffd7c6-9lz2p -f +... +{"type":"log","@timestamp":"2018-03-13T07:33:00Z","tags":["listening","info"],"pid":1,"message":"Server running at http://0:5601"} +{"type":"log","@timestamp":"2018-03-13T07:33:00Z","tags":["status","ui settings","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} +``` + +### 访问 Kibana + +这里介绍 `kube-apiserver`方式访问,获取访问 URL + +``` bash +$ kubectl cluster-info | grep Kibana +Kibana is running at https://192.168.1.10:8443/api/v1/namespaces/kube-system/services/kibana-logging/proxy +``` +浏览器访问 URL:`https://192.168.1.10:8443/api/v1/namespaces/kube-system/services/kibana-logging/proxy`,然后使用`basic auth`或者`证书` 的方式认证后即可,关于认证可以参考[dashboard文档](dashboard.md) + +首次登陆需要在`Management` - `Index Patterns` 创建 `index pattern`,可以使用默认的 logstash-* pattern,点击 Create; 创建Index后,稍等几分钟就可以在 Discover 菜单看到 ElasticSearch logging 中汇聚的日志; + + +[前一篇](ingress.md) -- [目录](index.md) -- [后一篇](harbor.md) diff --git a/docs/guide/index.md b/docs/guide/index.md index 29ca723..f5333f3 100644 --- a/docs/guide/index.md +++ b/docs/guide/index.md @@ -6,7 +6,7 @@ - 安装 [dashboard](dashboard.md) - 安装 [heapster](heapster.md) - 安装 [ingress](ingress.md) -- 安装 efk +- 安装 [efk](efk.md) - 安装 [harbor](harbor.md) ### 基础特性演示 diff --git a/manifests/efk/es-service.yaml b/manifests/efk/es-service.yaml new file mode 100644 index 0000000..ede9306 --- /dev/null +++ b/manifests/efk/es-service.yaml @@ -0,0 +1,17 @@ +apiVersion: v1 +kind: Service +metadata: + name: elasticsearch-logging + namespace: kube-system + labels: + k8s-app: elasticsearch-logging + kubernetes.io/cluster-service: "true" + addonmanager.kubernetes.io/mode: Reconcile + kubernetes.io/name: "Elasticsearch" +spec: + ports: + - port: 9200 + protocol: TCP + targetPort: db + selector: + k8s-app: elasticsearch-logging diff --git a/manifests/efk/es-statefulset.yaml b/manifests/efk/es-statefulset.yaml new file mode 100644 index 0000000..4bd6584 --- /dev/null +++ b/manifests/efk/es-statefulset.yaml @@ -0,0 +1,111 @@ +# RBAC authn and authz +apiVersion: v1 +kind: ServiceAccount +metadata: + name: elasticsearch-logging + namespace: kube-system + labels: + k8s-app: elasticsearch-logging + kubernetes.io/cluster-service: "true" + addonmanager.kubernetes.io/mode: Reconcile +--- +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: elasticsearch-logging + labels: + k8s-app: elasticsearch-logging + kubernetes.io/cluster-service: "true" + addonmanager.kubernetes.io/mode: Reconcile +rules: +- apiGroups: + - "" + resources: + - "services" + - "namespaces" + - "endpoints" + verbs: + - "get" +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + namespace: kube-system + name: elasticsearch-logging + labels: + k8s-app: elasticsearch-logging + kubernetes.io/cluster-service: "true" + addonmanager.kubernetes.io/mode: Reconcile +subjects: +- kind: ServiceAccount + name: elasticsearch-logging + namespace: kube-system + apiGroup: "" +roleRef: + kind: ClusterRole + name: elasticsearch-logging + apiGroup: "" +--- +# Elasticsearch deployment itself +apiVersion: apps/v1beta2 +kind: StatefulSet +metadata: + name: elasticsearch-logging + namespace: kube-system + labels: + k8s-app: elasticsearch-logging + version: v5.6.4 + kubernetes.io/cluster-service: "true" + addonmanager.kubernetes.io/mode: Reconcile +spec: + serviceName: elasticsearch-logging + replicas: 2 + selector: + matchLabels: + k8s-app: elasticsearch-logging + version: v5.6.4 + template: + metadata: + labels: + k8s-app: elasticsearch-logging + version: v5.6.4 + kubernetes.io/cluster-service: "true" + spec: + serviceAccountName: elasticsearch-logging + containers: + #- image: gcr.io/google-containers/elasticsearch:v5.6.4 + - image: mirrorgooglecontainers/elasticsearch:v5.6.4 + name: elasticsearch-logging + resources: + # need more cpu upon initialization, therefore burstable class + limits: + cpu: 1000m + requests: + cpu: 100m + ports: + - containerPort: 9200 + name: db + protocol: TCP + - containerPort: 9300 + name: transport + protocol: TCP + volumeMounts: + - name: elasticsearch-logging + mountPath: /data + env: + - name: "NAMESPACE" + valueFrom: + fieldRef: + fieldPath: metadata.namespace + volumes: + - name: elasticsearch-logging + emptyDir: {} + # Elasticsearch requires vm.max_map_count to be at least 262144. + # If your OS already sets up this number to a higher value, feel free + # to remove this init container. + initContainers: + - image: alpine:3.6 + command: ["/sbin/sysctl", "-w", "vm.max_map_count=262144"] + name: elasticsearch-logging-init + securityContext: + privileged: true diff --git a/manifests/efk/fluentd-es-configmap.yaml b/manifests/efk/fluentd-es-configmap.yaml new file mode 100644 index 0000000..09fbad0 --- /dev/null +++ b/manifests/efk/fluentd-es-configmap.yaml @@ -0,0 +1,373 @@ +kind: ConfigMap +apiVersion: v1 +data: + containers.input.conf: |- + # This configuration file for Fluentd / td-agent is used + # to watch changes to Docker log files. The kubelet creates symlinks that + # capture the pod name, namespace, container name & Docker container ID + # to the docker logs for pods in the /var/log/containers directory on the host. + # If running this fluentd configuration in a Docker container, the /var/log + # directory should be mounted in the container. + # + # These logs are then submitted to Elasticsearch which assumes the + # installation of the fluent-plugin-elasticsearch & the + # fluent-plugin-kubernetes_metadata_filter plugins. + # See https://github.com/uken/fluent-plugin-elasticsearch & + # https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter for + # more information about the plugins. + # + # Example + # ======= + # A line in the Docker log file might look like this JSON: + # + # {"log":"2014/09/25 21:15:03 Got request with path wombat\n", + # "stream":"stderr", + # "time":"2014-09-25T21:15:03.499185026Z"} + # + # The time_format specification below makes sure we properly + # parse the time format produced by Docker. This will be + # submitted to Elasticsearch and should appear like: + # $ curl 'http://elasticsearch-logging:9200/_search?pretty' + # ... + # { + # "_index" : "logstash-2014.09.25", + # "_type" : "fluentd", + # "_id" : "VBrbor2QTuGpsQyTCdfzqA", + # "_score" : 1.0, + # "_source":{"log":"2014/09/25 22:45:50 Got request with path wombat\n", + # "stream":"stderr","tag":"docker.container.all", + # "@timestamp":"2014-09-25T22:45:50+00:00"} + # }, + # ... + # + # The Kubernetes fluentd plugin is used to write the Kubernetes metadata to the log + # record & add labels to the log record if properly configured. This enables users + # to filter & search logs on any metadata. + # For example a Docker container's logs might be in the directory: + # + # /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b + # + # and in the file: + # + # 997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log + # + # where 997599971ee6... is the Docker ID of the running container. + # The Kubernetes kubelet makes a symbolic link to this file on the host machine + # in the /var/log/containers directory which includes the pod name and the Kubernetes + # container name: + # + # synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log + # -> + # /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log + # + # The /var/log directory on the host is mapped to the /var/log directory in the container + # running this instance of Fluentd and we end up collecting the file: + # + # /var/log/containers/synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log + # + # This results in the tag: + # + # var.log.containers.synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log + # + # The Kubernetes fluentd plugin is used to extract the namespace, pod name & container name + # which are added to the log message as a kubernetes field object & the Docker container ID + # is also added under the docker field object. + # The final tag is: + # + # kubernetes.var.log.containers.synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log + # + # And the final log record look like: + # + # { + # "log":"2014/09/25 21:15:03 Got request with path wombat\n", + # "stream":"stderr", + # "time":"2014-09-25T21:15:03.499185026Z", + # "kubernetes": { + # "namespace": "default", + # "pod_name": "synthetic-logger-0.25lps-pod", + # "container_name": "synth-lgr" + # }, + # "docker": { + # "container_id": "997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b" + # } + # } + # + # This makes it easier for users to search for logs by pod name or by + # the name of the Kubernetes container regardless of how many times the + # Kubernetes pod has been restarted (resulting in a several Docker container IDs). + + # Json Log Example: + # {"log":"[info:2016-02-16T16:04:05.930-08:00] Some log text here\n","stream":"stdout","time":"2016-02-17T00:04:05.931087621Z"} + # CRI Log Example: + # 2016-02-17T00:04:05.931087621Z stdout F [info:2016-02-16T16:04:05.930-08:00] Some log text here + + type tail + path /var/log/containers/*.log + pos_file /var/log/es-containers.log.pos + time_format %Y-%m-%dT%H:%M:%S.%NZ + tag kubernetes.* + read_from_head true + format multi_format + + format json + time_key time + time_format %Y-%m-%dT%H:%M:%S.%NZ + + + format /^(? + + system.input.conf: |- + # Example: + # 2015-12-21 23:17:22,066 [salt.state ][INFO ] Completed state [net.ipv4.ip_forward] at time 23:17:22.066081 + + type tail + format /^(?