SonoFelice——棣琦

cl_303303303

一、heapster简介

什么是Heapster？
　　Heapster是容器集群监控和性能分析工具，天然的支持Kubernetes和CoreOS。
　　Kubernetes有个出名的监控agent---cAdvisor。在每个kubernetes Node上都会运行cAdvisor，它会收集本机以及容器的监控数据(cpu,memory,filesystem,network,uptime)。
　　在较新的版本中，K8S已经将cAdvisor功能集成到kubelet组件中。每个Node节点可以直接进行web访问。

cAdvisor web界面访问： http://< Node-IP >:4194　　cAdvisor也提供Restful API: https://github.com/google/cadv ... pi.md

　　Heapster是一个收集者，将每个Node上的cAdvisor的数据进行汇总，然后导到第三方工具(如InfluxDB)。

二、heapster调用kubelet源码分析

1、整体源码
　　heapster整体的源码分析可以参考文章：https://segmentfault.com/a/1190000008863353#articleHeader2，
　　上面链接中的文章中会对heapster从启动开始进行分析，主要讲述了下面内容：

• main（）
  
• 创建数据源
  
• 创建后端服务
  
• 创建数据processors
  
• 获取源数据并存储
  
• heapster API创建
  

　　给上述文章的作者点个赞，思路真的非常清晰。不过笔者最关注的点是heapster如何进行的kubelet调用。

2、调用kubelet源码分析
　　从上述文章提到的 NewKubeletProvider 函数中，我们找到了创建kubeClient和kubeletClient的地方，见下面代码的飘黄部分：
　　

func NewKubeletProvider(uri *url.URL) (MetricsSourceProvider, error) {// 创建kubernetes master及kubelet client相关的配置　　kubeConfig, kubeletConfig, err := GetKubeConfigs(uri)
　　if err != nil {
　　return nil, err
　　}
　　// 创建kubeClient及kubeletClient
　　kubeClient := kube_client.NewOrDie(kubeConfig)
　　kubeletClient, err := NewKubeletClient(kubeletConfig)
　　if err != nil {
　　return nil, err
　　}
　　

　　// 获取下所有的Nodes，测试下创建的client是否能正常通讯
　　if _, err := kubeClient.Nodes().List(kube_api.ListOptions{
　　LabelSelector: labels.Everything(),
　　FieldSelector: fields.Everything()}); err != nil {
　　glog.Errorf("Failed to load nodes: %v", err)
　　}
　　

　　// 监控k8s的nodes变更
　　// 这里会创建协程进行watch，便于后面调用nodeLister.List()列出所有的nodes。
　　// 该Watch的实现，需要看下apiServer中的实现，后面会进行讲解
　　lw := cache.NewListWatchFromClient(kubeClient, "nodes", kube_api.NamespaceAll, fields.Everything())
　　nodeLister := &cache.StoreToNodeLister{Store: cache.NewStore(cache.MetaNamespaceKeyFunc)}
　　reflector := cache.NewReflector(lw, &kube_api.Node{}, nodeLister.Store, time.Hour)
　　reflector.Run()
　　// 结构在前面介绍过
　　return &kubeletProvider{
　　nodeLister:    nodeLister,
　　reflector:     reflector,
　　kubeletClient: kubeletClient,
　　}, nil
　　
}
　　


1）heapster源码目录
　　我们发现这个函数所在的文件为kubelet.go，所在的包为metrics/sources/kubelet
　　我们可以看到，kubelet调用相关的代码都在这个里面：

2）heapster调用kubelet的API
　　我们进入到NewKubeletClient函数看一下：
　　

// 传入的是KubeletClientConfig，通过读取相关配置，来初始化client　　

func NewKubeletClient(kubeletConfig *kubelet_client.KubeletClientConfig) (*KubeletClient, error) {　　

    transport, err := kubelet_client.MakeTransport(kubeletConfig)if err != nil {return nil, err　　}
　　c :
= &http.Client{ // 此处可以看到，是http调用　　Transport: transport,
　　Timeout:   kubeletConfig.HTTPTimeout,
　　}
return &KubeletClient{　　config: kubeletConfig,
　　client: c,
　　}, nil
　　
}
　　

　　接下来我们看一下kubelet_client.go中KubeletClient有哪些方法：

(i)获取所有的containersInfo
　　第一个，获取所有的container统计信息，如果对cAdvisor比较了解的话，可以进入到该方法的返回值的ContainerInfo里面看下，是非常全面的容器监控信息（后续会有cAdvisor的源码分析，敬请期待）
　　

func (self *KubeletClient) getAllContainers(url string, start, end time.Time) ([]cadvisor.ContainerInfo, error) {// Request data from all subcontainers.此处是构造requestBody　　request := statsRequest{
　　ContainerName: "/",
　　NumStats:      1,
　　Start:         start, // 2017-11-10T06:46:17Z 这种utc格式的时间戳
　　End:           end,
　　Subcontainers: true,
　　}
　　body, err := json.Marshal(request)
　　if err != nil {
　　return nil, err
　　}
　　req, err := http.NewRequest("POST", url, bytes.NewBuffer(body))
　　if err != nil {
　　return nil, err
　　}
　　// 设置请求头
　　req.Header.Set("Content-Type", "application/json")
　　

　　var containers map[string]cadvisor.ContainerInfo
　　client := self.client
　　if client == nil {
　　client = http.DefaultClient
　　}
　　err = self.postRequestAndGetValue(client, req, &containers)
　　if err != nil {
　　return nil, fmt.Errorf("failed to get all container stats from Kubelet URL %q: %v", url, err)
　　}
　　

　　result := make([]cadvisor.ContainerInfo, 0, len(containers))
　　for _, containerInfo := range containers {
　　cont := self.parseStat(&containerInfo)
　　if cont != nil {
　　result = append(result, *cont)
　　}
　　}
　　return result, nil
　　
}
　　

　　我们可以看到上面的源码其实是构造了一个http的post请求。那么我们在本地通过restClient模拟一下：
　　http://ip:10255/stats/container/（为什么用10255端口呢？在上面的源码中初始化NewKubeletClient的地方，我们进入到GetKubeClientConfig看一下就会发现，用的是kubelet的默认开放端口，在configs.go文件中）

　　返回值部分摘录如下（内容太多，折叠一下）：

　　

"/": {"name": "/","subcontainers": [　　{
"name": "/docker"　　},
　　{
"name": "/init.scope"　　},
　　{
"name": "/kube-proxy"　　},
　　{
"name": "/kubepods"　　},
　　{
"name": "/system.slice"　　},
　　{
"name": "/user.slice"　　}
　　],
"spec": {"creation_time": "2017-10-14T17:16:59.086488213+08:00","has_cpu": true,"cpu": {"limit": 1024,"max_limit": 0,"mask": "0-3","period": 100000　　},
"has_memory": true,"memory": {"limit": 33604153344,"reservation": 9223372036854771712　　},
"has_network": true,"has_filesystem": true,"has_diskio": true,"has_custom_metrics": false　　},
"stats": [　　{
"timestamp": "2017-11-10T14:29:25.105117933+08:00","cpu": {"usage": {"total": 916886132160774,"per_cpu_usage": [225713500943530,230866928282148,229896967301966,230408735633130　　],
"user": 443962220000000,"system": 438380440000000　　},
"cfs": {"periods": 0,"throttled_periods": 0,"throttled_time": 0　　},
"load_average": 0　　},
"diskio": {"io_service_bytes": [　　{
"major": 7,"minor": 4,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 15,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 14,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 12,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 0,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 0,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 11,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 4,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 253,"minor": 0,"stats": {"Async": 61989669888,"Read": 23649280,"Sync": 17028616192,"Total": 79018286080,"Write": 78994636800　　}
　　},
　　{
"major": 7,"minor": 7,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 9,"minor": 0,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 13,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 6,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 6,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 5,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 5,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 3,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 7,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 253,"minor": 16,"stats": {"Async": 258048,"Read": 21504,"Sync": 4773888,"Total": 5031936,"Write": 5010432　　}
　　},
　　{
"major": 7,"minor": 3,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 2,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 1,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 10,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 9,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 8,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 2,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 1,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　}
　　],
"io_serviced": [　　{
"major": 7,"minor": 4,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 9,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 8,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 9,"minor": 0,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 1,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 15,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 7,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 1,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 4,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 0,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 253,"minor": 0,"stats": {"Async": 2738092,"Read": 2965,"Sync": 4157380,"Total": 6895472,"Write": 6892507　　}
　　},
　　{
"major": 1,"minor": 13,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 5,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 14,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 5,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 253,"minor": 16,"stats": {"Async": 57,"Read": 7,"Sync": 32,"Total": 89,"Write": 82　　}
　　},
　　{
"major": 1,"minor": 3,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 2,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 6,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 2,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 11,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 10,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 6,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 12,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 7,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 3,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 0,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　}
　　]
　　},
"memory": {"usage": 12189630464,"cache": 288108544,"rss": 3631652864,"swap": 0,"working_set": 7756312576,"failcnt": 0,"container_data": {"pgfault": 8947531,"pgmajfault": 87　　},
"hierarchical_data": {"pgfault": 8947531,"pgmajfault": 87　　}
　　},
"network": {"name": "cbr0","rx_bytes": 1885776219,"rx_packets": 8199780,"rx_errors": 0,"rx_dropped": 0,"tx_bytes": 1854690043,"tx_packets": 8765796,"tx_errors": 0,"tx_dropped": 0,"interfaces": [　　{
"name": "cbr0","rx_bytes": 1885776219,"rx_packets": 8199780,"rx_errors": 0,"rx_dropped": 0,"tx_bytes": 1854690043,"tx_packets": 8765796,"tx_errors": 0,"tx_dropped": 0　　},
　　{
"name": "ens3","rx_bytes": 25108832715,"rx_packets": 74352449,"rx_errors": 0,"rx_dropped": 0,"tx_bytes": 93157559917,"tx_packets": 75983196,"tx_errors": 0,"tx_dropped": 0　　}
　　],
"tcp": {"Established": 0,"SynSent": 0,"SynRecv": 0,"FinWait1": 0,"FinWait2": 0,"TimeWait": 0,"Close": 0,"CloseWait": 0,"LastAck": 0,"Listen": 0,"Closing": 0　　},
"tcp6": {"Established": 0,"SynSent": 0,"SynRecv": 0,"FinWait1": 0,"FinWait2": 0,"TimeWait": 0,"Close": 0,"CloseWait": 0,"LastAck": 0,"Listen": 0,"Closing": 0　　},
"udp": {"Listen": 0,"Dropped": 0,"RxQueued": 0,"TxQueued": 0　　},
"udp6": {"Listen": 0,"Dropped": 0,"RxQueued": 0,"TxQueued": 0　　}
　　},
"filesystem": [　　{
"device": "/dev/root","type": "vfs","capacity": 20749852672,"usage": 9800732672,"base_usage": 0,"available": 10932342784,"has_inodes": true,"inodes": 2560000,"inodes_free": 2340243,"reads_completed": 0,"reads_merged": 0,"sectors_read": 0,"read_time": 0,"writes_completed": 0,"writes_merged": 0,"sectors_written": 0,"write_time": 0,"io_in_progress": 0,"io_time": 0,"weighted_io_time": 0　　},
　　{
"device": "tmpfs","type": "vfs","capacity": 16802074624,"usage": 2224128,"base_usage": 0,"available": 16799850496,"has_inodes": true,"inodes": 4102069,"inodes_free": 4101999,"reads_completed": 0,"reads_merged": 0,"sectors_read": 0,"read_time": 0,"writes_completed": 0,"writes_merged": 0,"sectors_written": 0,"write_time": 0,"io_in_progress": 0,"io_time": 0,"weighted_io_time": 0　　},
　　{
"device": "shm","type": "vfs","capacity": 67108864,"usage": 0,"base_usage": 0,"available": 67108864,"has_inodes": true,"inodes": 4102069,"inodes_free": 4102068,"reads_completed": 0,"reads_merged": 0,"sectors_read": 0,"read_time": 0,"writes_completed": 0,"writes_merged": 0,"sectors_written": 0,"write_time": 0,"io_in_progress": 0,"io_time": 0,"weighted_io_time": 0　　},
　　{
"device": "/dev/vdb","type": "vfs","capacity": 53660876800,"usage": 35852288,"base_usage": 0,"available": 53625024512,"has_inodes": true,"inodes": 26214400,"inodes_free": 26214393,"reads_completed": 457,"reads_merged": 0,"sectors_read": 13162,"read_time": 3428,"writes_completed": 231,"writes_merged": 20,"sectors_written": 72963,"write_time": 1756,"io_in_progress": 0,"io_time": 3584,"weighted_io_time": 5184　　}
　　],
"task_stats": {"nr_sleeping": 0,"nr_running": 0,"nr_stopped": 0,"nr_uninterruptible": 0,"nr_io_wait": 0　　}
　　},
　　{
"timestamp": "2017-11-10T14:29:36.487311961+08:00","cpu": {"usage": {"total": 916888682985895,"per_cpu_usage": [225714036142385,230867670611465,229897633996290,230409342235755　　],
"user": 443963990000000,"system": 438381050000000　　},
"cfs": {"periods": 0,"throttled_periods": 0,"throttled_time": 0　　},
"load_average": 0　　},
"diskio": {"io_service_bytes": [　　{
"major": 9,"minor": 0,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 253,"minor": 0,"stats": {"Async": 61989710848,"Read": 23649280,"Sync": 17028677632,"Total": 79018388480,"Write": 78994739200　　}
　　},
　　{
"major": 7,"minor": 6,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 1,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 10,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 8,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 4,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 2,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 13,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 1,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 7,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 0,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 253,"minor": 16,"stats": {"Async": 258048,"Read": 21504,"Sync": 4773888,"Total": 5031936,"Write": 5010432　　}
　　},
　　{
"major": 1,"minor": 14,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 12,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 9,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 6,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 7,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 15,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 5,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 5,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 4,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 3,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 2,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 3,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 0,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 11,"stats": {"Async": 12288,"Read": 12288,"Sync": 0,"Total": 12288,"Write": 0　　}
　　}
　　],
"io_serviced": [　　{
"major": 1,"minor": 11,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 9,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 4,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 3,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 14,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 10,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 5,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 2,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 1,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 0,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 9,"minor": 0,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 253,"minor": 0,"stats": {"Async": 2738102,"Read": 2965,"Sync": 4157395,"Total": 6895497,"Write": 6892532　　}
　　},
　　{
"major": 7,"minor": 1,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 15,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 3,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 5,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 13,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 12,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 6,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 4,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 253,"minor": 16,"stats": {"Async": 57,"Read": 7,"Sync": 32,"Total": 89,"Write": 82　　}
　　},
　　{
"major": 7,"minor": 6,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 2,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 0,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 7,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　},
　　{
"major": 7,"minor": 7,"stats": {"Async": 0,"Read": 0,"Sync": 0,"Total": 0,"Write": 0　　}
　　},
　　{
"major": 1,"minor": 8,"stats": {"Async": 3,"Read": 3,"Sync": 0,"Total": 3,"Write": 0　　}
　　}
　　]
　　},
"memory": {"usage": 12189343744,"cache": 288108544,"rss": 3631366144,"swap": 0,"working_set": 7756025856,"failcnt": 0,"container_data": {"pgfault": 8947531,"pgmajfault": 87　　},
"hierarchical_data": {"pgfault": 8947531,"pgmajfault": 87　　}
　　},
"network": {"name": "cbr0","rx_bytes": 1885783888,"rx_packets": 8199819,"rx_errors": 0,"rx_dropped": 0,"tx_bytes": 1854698778,"tx_packets": 8765835,"tx_errors": 0,"tx_dropped": 0,"interfaces": [　　{
"name": "cbr0","rx_bytes": 1885783888,"rx_packets": 8199819,"rx_errors": 0,"rx_dropped": 0,"tx_bytes": 1854698778,"tx_packets": 8765835,"tx_errors": 0,"tx_dropped": 0　　},
　　{
"name": "ens3","rx_bytes": 25108971486,"rx_packets": 74353959,"rx_errors": 0,"rx_dropped": 0,"tx_bytes": 93160391886,"tx_packets": 75985129,"tx_errors": 0,"tx_dropped": 0　　}
　　],
"tcp": {"Established": 0,"SynSent": 0,"SynRecv": 0,"FinWait1": 0,"FinWait2": 0,"TimeWait": 0,"Close": 0,"CloseWait": 0,"LastAck": 0,"Listen": 0,"Closing": 0　　},
"tcp6": {"Established": 0,"SynSent": 0,"SynRecv": 0,"FinWait1": 0,"FinWait2": 0,"TimeWait": 0,"Close": 0,"CloseWait": 0,"LastAck": 0,"Listen": 0,"Closing": 0　　},
"udp": {"Listen": 0,"Dropped": 0,"RxQueued": 0,"TxQueued": 0　　},
"udp6": {"Listen": 0,"Dropped": 0,"RxQueued": 0,"TxQueued": 0　　}
　　},
"filesystem": [　　{
"device": "/dev/root","type": "vfs","capacity": 20749852672,"usage": 9800732672,"base_usage": 0,"available": 10932342784,"has_inodes": true,"inodes": 2560000,"inodes_free": 2340243,"reads_completed": 0,"reads_merged": 0,"sectors_read": 0,"read_time": 0,"writes_completed": 0,"writes_merged": 0,"sectors_written": 0,"write_time": 0,"io_in_progress": 0,"io_time": 0,"weighted_io_time": 0　　},
　　{
"device": "tmpfs","type": "vfs","capacity": 16802074624,"usage": 2224128,"base_usage": 0,"available": 16799850496,"has_inodes": true,"inodes": 4102069,"inodes_free": 4101999,"reads_completed": 0,"reads_merged": 0,"sectors_read": 0,"read_time": 0,"writes_completed": 0,"writes_merged": 0,"sectors_written": 0,"write_time": 0,"io_in_progress": 0,"io_time": 0,"weighted_io_time": 0　　},
　　{
"device": "shm","type": "vfs","capacity": 67108864,"usage": 0,"base_usage": 0,"available": 67108864,"has_inodes": true,"inodes": 4102069,"inodes_free": 4102068,"reads_completed": 0,"reads_merged": 0,"sectors_read": 0,"read_time": 0,"writes_completed": 0,"writes_merged": 0,"sectors_written": 0,"write_time": 0,"io_in_progress": 0,"io_time": 0,"weighted_io_time": 0　　},
　　{
"device": "/dev/vdb","type": "vfs","capacity": 53660876800,"usage": 35852288,"base_usage": 0,"available": 53625024512,"has_inodes": true,"inodes": 26214400,"inodes_free": 26214393,"reads_completed": 457,"reads_merged": 0,"sectors_read": 13162,"read_time": 3428,"writes_completed": 231,"writes_merged": 20,"sectors_written": 72963,"write_time": 1756,"io_in_progress": 0,"io_time": 3584,"weighted_io_time": 5184　　}
　　],
"task_stats": {"nr_sleeping": 0,"nr_running": 0,"nr_stopped": 0,"nr_uninterruptible": 0,"nr_io_wait": 0　　}
　　}
　　]
　　}
　　


View Code　　我们可以看到这一段json数据主要分了4部分：

• reference——container本身的引用
  
• subcontainers——子容器的信息
  
• spec——说明信息
  
• stats——统计信息
  

　　与cAdvisor提供的统计数据一致：
　　

type ContainerInfo struct {　　ContainerReference
　　

// The direct subcontainers of the current container.　　Subcontainers []ContainerReference `json:"subcontainers,omitempty"`
　　

　　// The isolation used in the container.
　　Spec ContainerSpec `json:"spec,omitempty"`
　　

　　// Historical statistics gathered from the container.
　　Stats []*ContainerStats `json:"stats,omitempty"`
　　
}
　　

　　太棒了。
　　那么我就可以知道，获取allcontainers的逻辑，其实就是向kubelet发送了一个http的post请求，然后获取到返回的json数据。

(ii) 获取pod相关的summary
　　在kubelet_client.go文件中还有一个获取summary的方法：
　　

func (self *KubeletClient) GetSummary(host Host) (*stats.Summary, error) {　　url :
= url.URL{　　Scheme:
"http",　　Host:   host.String(),
　　Path:
"/stats/summary/",　　}
if self.config != nil && self.config.EnableHttps {　　url.Scheme
= "https"　　}
　　

　　req, err :
= http.NewRequest("GET", url.String(), nil)if err != nil {return nil, err　　}
　　summary :
= &stats.Summary{}　　client :
= self.clientif client == nil {　　client
= http.DefaultClient　　}
　　err
= self.postRequestAndGetValue(client, req, summary)return summary, err　　
}
　　

　　恩，分析一下，很简单，就是一个http  get方法的调用。那我们再来模拟一下：
　　http://ip:10255/stats/summary
　　直接调用这个url即可，返回值太多，就不一一分析了。返回的数据结构如下：
　　

type Summary struct {// Overall node stats.　　Node NodeStats `json:"node"`
　　// Per-pod stats.
　　Pods []PodStats `json:"pods"`
　　
}
　　

　　主要就是当前node的统计信息，还有pod的聚合信息。可以自己追进NodeStats和PodeStats的源码中看一下结构，在此就不列了。
　　值得注意的是，summary里面的stats和ContainerInfo是有些不一样的。比如ContainerInfo里面会有DiskInfo相关的信息。

3）小结
　　heapster调用kubelet，其实就是发送了两个http请求，一个是get方法，获取所有pod的summary信息，另一个是post方法，获取cAdvisor提供的所有ContainerInfo。然后在pod_aggregator.go方法中进行的数据相关的聚合。那我们对kubelet调用的源码分析就到此为止了。接下来是笔者自己做的一个小实验。

三、实验

1、禁掉节点的4194端口
　　禁掉node的4194端口后，就无法通过ip：4194获取到cAdvisor的stats数据了。那我们调用一下kubelet的api：调用kubelet的api：http://ip:10255/stats/container/
　　我们会发现，返回的数据项里缺少了stats这一项，而这一项恰恰是cAdvisor提供的最重要的监控统计数据。

2、重新开启节点的4194端口
　　重新开启node的4194端口，再重新调用一下kubelet的api，这下返回的结果里面又重新出现了stats这一项。

3、总结
　　结合kubelet的源码可以发现，在kubelet启动的时候会去启动cAdvisor的4194端口，获取基础的统计数据。