Kubernetes developers: • The Golang programming patterns of Kubernetes (Controller, codegen etc) • Write your own Controller • gPRC based interface design in Kubernetes (CRI as example) • For Kubernetes desired := getDesiredState() current := getCurrentState() makeChanges(desired, current) } • Write your own controller! Why DIY? Container Pod Deployment I need co-scheduling I have many Pod replicas image: gcr.io/google_containers/testapp:v1 volumeMounts: - name: varlog mountPath: /var/log - name: logging-agent image: gcr.io/google_containers/fluentd:1.30 env: - name: FLUENTD_ARGS

/var/lib/chrony/dri� makestep 1.0 3 rtcsync keyfile /etc/chrony.keys leapsectz right/UTC logdir /var/log/chrony EOF # �medatectl set-�mezone Asia/Shanghai #设置时区 # systemctl restart chronyd # --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key \ endpoint status --write-out=table # 查看集群健康状态 etcdctl -- endpoints=h�ps://10.99.1.51:2379,h�ps://10.99.1.52:2379,h�ps://10 #pod容器要使用的存储卷声明 - name: alog #根据此名称去寻找对应的volume存储卷 mountPath: /var/log/nginx #容器里的挂载点 readOnly: false #非只读（即可读写） volumes:

{ w.WriteHeader(500) w.Write([]byte(fmt.Sprintf("error: %v", duration.Seconds()))) } else { w.WriteHeader(200) w.Write([]byte("ok")) }}) 参考：http://blog.51cto memory: 200Mi volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers volumes: - name: varlog 18-Daemon Set 75 hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers

/etc/docker/daemon.json { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2", "storage-opts": nbiot-udp: address: ":9092/udp" providers: kubernetesCRD: "" log: filePath: "" level: error format: json accessLog: filePath: lvwq Org = iSurpass Bucket = openservice [Log] Filename = /opt/tools/iSurpass/openservice/log/openservice.log MaxSize = 1 MaxBackups = 30 MaxAge = 15

/data/all-log pod-path /data/all-log kafka-cluster elasticsearch-cluster logstash-cluster kafka-groupid=es topic applog logstash kafka-groupid=file file-storage mkdir -p /data/all- log/${namespaces}/${pod-name} app-log-path pod 1 worker node pod * app-log log* daemonsets filebeat 容器启动挂载宿主/data/all-log到容器/data/all-log 容器启动时创建namespace和pod-name 创建软连接到logs目录 daemonset在每一个k8s-node节点启动 挂载/data/all-log目录 实时扫描发现新目录日志

Job Job Job Job Pod Pod Pod Pod ElasticSearch ElasticSearch Logging Service agent to collecting log data ElasticSearch ElasticSearch Monitor/Alert Service CronJob Node Pod Node Pod Unified collecting log data ElasticSearch ElasticSearch Monitor/Alert Service DevOps Operator Cluster AutoScaler k8s API DevOps Service DevOps Service DevOps Manager Restful API realtime log history history log pull metric data ElasticSearch ElasticSearch Prometheus Push Gateway push metric data • Build task can also expose custom metric data • Ephemeral build task can push metric to gateway if

Goundan @coollog @loosebazooka Qingyang “Q” Chen github.com/GoogleContainerTools/jib Containers “Write once, run anywhere” github.com/GoogleContainerTools/jib Building a Java container github.com com/GoogleContainerTools/jib What did we do? 1. Write first Dockerfile 2. Reduce image size 3. Don’t run installs 4. Use better base image 5. Write .dockerignore 6. Improve incremental speed 7. Switch com/GoogleContainerTools/jib What did we do? 1. Write first Dockerfile 2. Reduce image size 3. Don’t run installs 4. Use better base image 5. Write .dockerignore 6. Improve incremental speed 7. Switch

Semi- structured Data SQL�NoSQL Log Service / LogShipper Mobile & Web IoT Mobile Logs Web Text & Logs Services & Languages IoT & Devices Camera �� Log Service / LogHub Real-time Data Stream DataV Log Service / Dashboard ��� Stream Processing SparkStreaming Function Compute Hadoop HIVE Big Data Analytics Batch Processing Pig PAI MaxCompute EMR Interactive Analytics DLA Log Service

ContainerCrashLoopBackOff, FailedPostStartHook, Unhealthy… Trace system Increase of SLO Data Collect Audit log Event The unhealthy node Monitoring Isolation Recover Degrade Data Analysis Failures/Machine problems automatically。 The infrastructure Log Event End User Storage Analysis Platform Trace Report Weakness The trace system Data Collect: Collect Audit log for the whole cluster. Data analysis: Analyze Auto Human experience Improve of strategy …… 1. Collect data from metrics NPD, Trace system and Log. 2. Analyze the problem of the node, such as DiskRO, critical Daemonset is not ready. 3. Processing

logspout K8S DaemonSet stdout stderr K8S Pod LogMinion Kibana Kafka Cluster Log Parser ES Cluster log tail log tail stdout stderr Console 21 2017 Lenovo Internal. All rights reserved. Showcase