add aggregated api servers

2017-11-27 19:33:18 +08:00 · 2017-11-27 19:33:18 +08:00 · ba130827ff
parent f13804ad0f
commit ba130827ff
7 changed files with 460 additions and 0 deletions
--- a/SUMMARY.md
+++ b/SUMMARY.md
@ -32,10 +32,12 @@
  - [ConfigMap](concepts/configmap.md)
    - [ConfigMap的热更新](concepts/configmap-hot-update.md)
  - [Horizontal Pod Autoscaling](concepts/horizontal-pod-autoscaling.md)
+    - [自定义指标HPA](concepts/custom-metrics-hpa.md)
  - [Label](concepts/label.md)
  - [垃圾收集](concepts/garbage-collection.md)
  - [NetworkPolicy](concepts/network-policy.md)
  - [Annotation](concepts/annotation.md)
+  - [Aggregated API Server](concepts/aggregated-api-server.md)

 ### 用户指南

--- a/concepts/aggregated-api-server.md
+++ b/concepts/aggregated-api-server.md
@ -0,0 +1,33 @@
+## Aggregated API Servers
+
+Aggregated（聚合的）API  server是为了将原来的API server这个巨石（monolithic）应用给拆分成，为了方便用户开发自己的API server集成进来，而不用直接修改kubernetes官方仓库的代码，这样一来也能将API server解耦，方便用户使用实验特性。这些API server可以跟core API server无缝衔接，试用kubectl也可以管理它们。
+
+### 架构
+
+我们需要创建一个新的组件，名为`kube-aggregator`，它需要负责以下几件事：
+
+- 提供用于注册API server的API
+- 汇总所有的API server信息
+- 代理所有的客户端到API server的请求
+
+**注意**：这里说的API server是一组“API Server”，而不是说我们安装集群时候的那个API server，而且这组API server是可以横向扩展的。
+
+关于聚合的API server的更多信息请参考：[Aggregated API Server](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/aggregated-api-servers.md)
+
+### 安装配置聚合的API server
+
+**编译**
+
+下载kubernetes的源码到`$GOPATH/src/k8s.io/`目录，在`$GOPATH/src/k8s.io/kubernetes/staging/src/k8s.io/kube-aggregator`目录下编译生成Linux add64的二进制文件：
+
+```bash
+GOOS=linux GOARCH=amd64 go build
+```
+
+将在当前目录下生成`kube-aggregator`二进制文件。
+
+**配置**
+
+将该二进制文件上传到kubernetes的node节点上。
+
+TBD
--- a/concepts/custom-metrics-hpa.md
+++ b/concepts/custom-metrics-hpa.md
@ -0,0 +1,45 @@
+# 自定义指标HPA
+
+Kubernetes中支持不仅支持CPU、内存为指标的HPA，还支持自定义指标的HPA，例如QPS。
+
+## 设置自定义指标
+
+**kubernetes1.6**
+
+在设置定义指标HPA之前需要先进行如下配置：
+
+- 将heapster的启动参数 `--api-server` 设置为 true
+
+
+- 启用custom metric API
+- 将kube-controller-manager的启动参数中`--horizontal-pod-autoscaler-use-rest-clients`设置为true，并指定`--master`为API server地址，如`--master=http://172.20.0.113:8080`
+
+在kubernetes1.5以前很容易设置，参考[1.6以前版本的kubernetes中开启自定义HPA](https://medium.com/@marko.luksa/kubernetes-autoscaling-based-on-custom-metrics-without-using-a-host-port-b783ed6241ac)，而在1.6中因为取消了原来的annotation方式设置custom metric，只能通过API server和kube-aggregator来获取custom metric，因为只有两种方式来设置了，一是直接通过API server获取heapster的metrics，二是部署[kube-aggragator](https://github.com/kubernetes/kube-aggregator)来实现。
+
+我们将安装聚合的API server来实现自定义指标的HPA。
+
+**kuberentes1.7**
+
+已经内置了`apiregistration.k8s.io/v1beta1` API，可以直接定义APIService，如：
+
+```yaml
+apiVersion: apiregistration.k8s.io/v1beta1
+kind: APIService
+metadata:
+  name: v1alpha1.custom-metrics.metrics.k8s.io
+spec:
+  insecureSkipTLSVerify: true
+  group: custom-metrics.metrics.k8s.io
+  groupPriorityMinimum: 1000
+  versionPriority: 5
+  service:
+    name: api
+    namespace: custom-metrics
+  version: v1alpha1
+```
+
+## 参考
+
+[1.6以前版本的kubernetes中开启自定义HPA](https://medium.com/@marko.luksa/kubernetes-autoscaling-based-on-custom-metrics-without-using-a-host-port-b783ed6241ac)
+
+[1.7版本的kubernetes中启用自定义HPA](https://docs.bitnami.com/kubernetes/how-to/configure-autoscaling-custom-metrics/)
--- a/manifests/HPA/custom-metrics.yaml
+++ b/manifests/HPA/custom-metrics.yaml
@ -0,0 +1,144 @@
+kind: Namespace
+apiVersion: v1
+metadata:
+  name: custom-metrics
+---
+kind: ServiceAccount
+apiVersion: v1
+metadata:
+  name: custom-metrics-apiserver
+  namespace: custom-metrics
+---
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: ClusterRoleBinding
+metadata:
+  name: custom-metrics:system:auth-delegator
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: system:auth-delegator
+subjects:
+- kind: ServiceAccount
+  name: custom-metrics-apiserver
+  namespace: custom-metrics
+---
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: RoleBinding
+metadata:
+  name: custom-metrics-auth-reader
+  namespace: kube-system
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: extension-apiserver-authentication-reader
+subjects:
+- kind: ServiceAccount
+  name: custom-metrics-apiserver
+  namespace: custom-metrics
+---
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: ClusterRole
+metadata:
+  name: custom-metrics-read
+rules:
+- apiGroups:
+  - ""
+  resources:
+  - namespaces
+  - pods
+  - services
+  verbs:
+  - get
+  - list
+---
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: ClusterRoleBinding
+metadata:
+  name: custom-metrics-read
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: custom-metrics-read
+subjects:
+- kind: ServiceAccount
+  name: custom-metrics-apiserver
+  namespace: custom-metrics
+---
+apiVersion: apps/v1beta1
+kind: Deployment
+metadata:
+  name: custom-metrics-apiserver
+  namespace: custom-metrics
+  labels:
+    app: custom-metrics-apiserver
+spec:
+  replicas: 1
+  template:
+    metadata:
+      name: custom-metrics-apiserver
+      labels:
+        app: custom-metrics-apiserver
+    spec:
+      serviceAccountName: custom-metrics-apiserver
+      containers:
+      - name: custom-metrics-server
+        image: sz-pg-oam-docker-hub-001.tendcloud.com/library/k8s-prometheus-adapter
+        args:
+        - --prometheus-url=http://sample-metrics-prom.default.svc:9090
+        - --metrics-relist-interval=30s
+        - --rate-interval=60s
+        - --v=10
+        - --logtostderr=true
+        ports:
+        - containerPort: 443
+        securityContext:
+          runAsUser: 0
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: api
+  namespace: custom-metrics
+spec:
+  ports:
+  - port: 443
+    targetPort: 443
+  selector:
+    app: custom-metrics-apiserver
+---
+apiVersion: apiregistration.k8s.io/v1beta1
+kind: APIService
+metadata:
+  name: v1alpha1.custom-metrics.metrics.k8s.io
+spec:
+  insecureSkipTLSVerify: true
+  group: custom-metrics.metrics.k8s.io
+  groupPriorityMinimum: 1000
+  versionPriority: 5
+  service:
+    name: api
+    namespace: custom-metrics
+  version: v1alpha1
+---
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: ClusterRole
+metadata:
+  name: custom-metrics-server-resources
+rules:
+- apiGroups:
+  - custom-metrics.metrics.k8s.io
+  resources: ["*"]
+  verbs: ["*"]
+---
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: ClusterRoleBinding
+metadata:
+  name: hpa-controller-custom-metrics
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: custom-metrics-server-resources
+subjects:
+- kind: ServiceAccount
+  name: horizontal-pod-autoscaler
+  namespace: kube-system
--- a/manifests/HPA/prometheus-operator.yaml
+++ b/manifests/HPA/prometheus-operator.yaml
@ -0,0 +1,85 @@
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: ClusterRole
+metadata:
+  name: prometheus-operator
+rules:
+- apiGroups:
+  - extensions
+  resources:
+  - thirdpartyresources
+  verbs:
+  - create
+- apiGroups:
+  - monitoring.coreos.com
+  resources:
+  - alertmanagers
+  - prometheuses
+  - servicemonitors
+  verbs:
+  - "*"
+- apiGroups:
+  - apps
+  resources:
+  - statefulsets
+  verbs: ["*"]
+- apiGroups: [""]
+  resources:
+  - configmaps
+  - secrets
+  verbs: ["*"]
+- apiGroups: [""]
+  resources:
+  - pods
+  verbs: ["list", "delete"]
+- apiGroups: [""]
+  resources:
+  - services
+  - endpoints
+  verbs: ["get", "create", "update"]
+- apiGroups: [""]
+  resources:
+  - nodes
+  verbs: ["list", "watch"]
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: prometheus-operator
+---
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: ClusterRoleBinding
+metadata:
+  name: prometheus-operator
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: prometheus-operator
+subjects:
+- kind: ServiceAccount
+  name: prometheus-operator
+  namespace: default
+---
+apiVersion: apps/v1beta1
+kind: Deployment
+metadata:
+  name: prometheus-operator
+  labels:
+    operator: prometheus
+spec:
+  replicas: 1
+  template:
+    metadata:
+      labels:
+        operator: prometheus
+    spec:
+      serviceAccountName: prometheus-operator
+      containers:
+       - name: prometheus-operator
+         image: sz-pg-oam-docker-hub-001.tendcloud.com/library/prometheus-operator:v0.10.1
+         resources:
+           requests:
+             cpu: 100m
+             memory: 50Mi
+           limits:
+             cpu: 200m
+             memory: 100Mi
--- a/manifests/HPA/sample-metrics-app.yaml
+++ b/manifests/HPA/sample-metrics-app.yaml
@ -0,0 +1,77 @@
+apiVersion: apps/v1beta1
+kind: Deployment
+metadata:
+  labels:
+    app: sample-metrics-app
+  name: sample-metrics-app
+spec:
+  replicas: 2
+  template:
+    metadata:
+      labels:
+        app: sample-metrics-app
+    spec:
+      containers:
+      - image: sz-pg-oam-docker-hub-001.tendcloud.com/library/autoscale-demo:v0.1.2
+        name: sample-metrics-app
+        ports:
+        - name: web
+          containerPort: 8080
+        readinessProbe:
+          httpGet:
+            path: /
+            port: 8080
+          initialDelaySeconds: 3
+          periodSeconds: 5
+        livenessProbe:
+          httpGet:
+            path: /
+            port: 8080
+          initialDelaySeconds: 3
+          periodSeconds: 5
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: sample-metrics-app
+  labels:
+    app: sample-metrics-app
+spec:
+  ports:
+  - name: web
+    port: 80
+    targetPort: 8080
+  selector:
+    app: sample-metrics-app
+---
+apiVersion: monitoring.coreos.com/v1alpha1
+kind: ServiceMonitor
+metadata:
+  name: sample-metrics-app
+  labels:
+    service-monitor: function
+spec:
+  selector:
+    matchLabels:
+      app: sample-metrics-app
+  endpoints:
+  - port: web
+---
+kind: HorizontalPodAutoscaler
+apiVersion: autoscaling/v2alpha1
+metadata:
+  name: sample-metrics-app-hpa
+spec:
+  scaleTargetRef:
+    kind: Deployment
+    name: sample-metrics-app
+  minReplicas: 2
+  maxReplicas: 10
+  metrics:
+  - type: Object
+    object:
+      target:
+        kind: Service
+        name: sample-metrics-app
+      metricName: http_requests
+      targetValue: 100
--- a/manifests/HPA/sample-prometheus-instance.yaml
+++ b/manifests/HPA/sample-prometheus-instance.yaml
@ -0,0 +1,74 @@
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: ClusterRole
+metadata:
+  name: prometheus
+rules:
+- apiGroups:
+  - ""
+  resources:
+  - nodes
+  - services
+  - endpoints
+  - pods
+  verbs:
+  - get
+  - list
+  - watch
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: prometheus
+---
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: ClusterRoleBinding
+metadata:
+  name: prometheus
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: prometheus
+subjects:
+- kind: ServiceAccount
+  name: prometheus
+  namespace: default
+---
+apiVersion: monitoring.coreos.com/v1alpha1
+kind: Prometheus
+metadata:
+  name: sample-metrics-prom
+  labels:
+    app: sample-metrics-prom
+    prometheus: sample-metrics-prom
+spec:
+  replicas: 1
+  baseImage: prom/prometheus
+  version: v1.7.1
+  serviceAccountName: prometheus
+  serviceMonitorSelector:
+    matchLabels:
+      service-monitor: function
+  resources:
+    requests:
+      memory: 300Mi
+  #storage:
+  #  resources:
+  #    requests:
+  #      storage: 3Gi
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: sample-metrics-prom
+  labels:
+    app: sample-metrics-prom
+    prometheus: sample-metrics-prom
+spec:
+  type: NodePort
+  ports:
+  - name: web
+    nodePort: 30999
+    port: 9090
+    targetPort: web
+  selector:
+    prometheus: sample-metrics-prom