利用夜莺扩展能力打造全方位监控系统 喻波 滴滴 专家工程师 目 录 运维监控需求来源 01 监控痛点：全面完备、跨云 02 夜莺介绍： 国产开源监控系统 03 夜莺设计实现：Agentd 数据采集 04 夜莺设计实现：Server 数据处理 05 夜莺设计实现：技术难点及细节 06 运维监控需求来源 第一部分 如果贵司的业务强依赖IT技术，IT故障会直接影响营业收入， 过aws大规模宕机的情况，影响不 止是55万美元，直接影响大半个 互联网！ 2018年有美国调研机构指出，如 果服务器宕机1分钟，银行会损失 27万美元，制造业会损失42万美 元 美团故障？滴滴故障？腾讯故障？ 运维监控需求来源 01.监控的原始需求来自业务稳定性 如何减少服务停摆导致的经济损失？尽快发现故障并止损！故障处理过程中，监控是『发现』和『定位』两个环节 的关键工具。故障 第三部分 国产开源监控产品相对比较匮乏，夜莺希望重新定义国产开 源监控，支持云原生监控，经受了滴滴大规模生产检验 Nightingale 夜莺是新一代国产智能监控平台，既可以解决传统物理机虚拟机的场景，也可以解 决容器的场景。衍生自Open-Falcon和滴滴Odin监控，经受了包括小米、美团、滴滴 在内的数百家企业的生产环境验证，简单可依赖，好用到爆！ 3500+ 600+ 500+

network setting for Configure Log Level • Available syslog levels are ERR, WARN, INFO, DEBUG, TRACE • MCM log is save in /var/log/syslog Set to Default MCM Configuration Set all configuration if traffic are drop or accept as expected # nft monitor trace a. Add trace_chain before the existing input chain nft add chain inet filter trace_chain { type filter hook prerouting priority -1\; } inet filter trace_chain meta nftrace set 1 c. Monitor trace (you can use another device with ncat tool to test it)dd nftrace flag moxa@moxa-tbbbb1182816:/# sudo nft monitor trace trace id d51bda11

tracing: Why is a packet being dropped or a request rejected. The policy tracing framework allows to trace the policy decision process for both, running workloads and based on arbitrary label defini�ons. Integrated capability of the Linux kernel to accept compiled bytecode that is run at various hook / trace points within the kernel. Cilium compiles BPF programs and has the kernel run them at key points in way to verify if and what policy rules apply between two endpoints. We can use the cilium policy trace to simulate a policy decision between the source and des�na�on endpoints. We will use the example

tracing: Why is a packet being dropped or a request rejected. The policy tracing framework allows to trace the policy decision process for both, running workloads and based on arbitrary label definitions. Integrated capability of the Linux kernel to accept compiled bytecode that is run at various hook / trace points within the kernel. Cilium compiles BPF programs and has the kernel run them at key points way to verify if and what policy rules apply between two endpoints. We can use the cilium policy trace to simulate a policy decision between the source and destination endpoints. We will use the example

default timeouts. mod_dumpio: DumpIOLogLevel is no longer supported. Data is always logged at LogLevel trace7. On Unix platforms, piped logging commands configured using either ErrorLog or CustomLog were invoked LogLevel configuration The LogLevel can now be configured per module and per directory. New levels trace1 to trace8 have been added above the debug log level. Per-request configuration sections , of the module in your LogLevel directive: LogLevel info rewrite:trace5 This sets the main LogLevel to info, but turns it up to trace5 for mod_rewrite. This replaces the per-module logging directives

default timeouts. mod_dumpio: DumpIOLogLevel is no longer supported. Data is always logged at LogLevel trace7. On Unix platforms, piped logging commands configured using either ErrorLog or CustomLog were invoked LogLevel configuration The LogLevel can now be configured per module and per directory. New levels trace1 to trace8 have been added above the debug log level. Per-request configuration sections , of the module in your LogLevel directive: LogLevel info rewrite:trace5 This sets the main LogLevel to info, but turns it up to trace5 for mod_rewrite. This replaces the per-module logging directives

default timeouts. mod_dumpio: DumpIOLogLevel is no longer supported. Data is always logged at LogLevel trace7. On Unix platforms, piped logging commands configured using either ErrorLog or CustomLog were invoked LogLevel configuration The LogLevel can now be configured per module and per directory. New levels trace1 to trace8 have been added above the debug log level. Per-request configuration sections , of the module in your LogLevel directive: LogLevel info rewrite:trace5 This sets the main LogLevel to info, but turns it up to trace5 for mod_rewrite. This replaces the per-module logging directives

default timeouts. mod_dumpio: DumpIOLogLevel is no longer supported. Data is always logged at LogLevel trace7. On Unix platforms, piped logging commands configured using either ErrorLog or CustomLog were invoked LogLevel configuration The LogLevel can now be configured per module and per directory. New levels trace1 to trace8 have been added above the debug log level. Per-request configuration sections , of the module in your LogLevel directive: LogLevel info rewrite:trace5 This sets the main LogLevel to info, but turns it up to trace5 for mod_rewrite. This replaces the per-module logging directives

default timeouts. mod_dumpio: DumpIOLogLevel is no longer supported. Data is always logged at LogLevel trace7. On Unix platforms, piped logging commands configured using either ErrorLog or CustomLog were invoked LogLevel configuration The LogLevel can now be configured per module and per directory. New levels trace1 to trace8 have been added above the debug log level. Per-request configuration sections , of the module in your LogLevel directive: LogLevel info rewrite:trace5 This sets the main LogLevel to info, but turns it up to trace5 for mod_rewrite. This replaces the per-module logging directives

default timeouts. mod_dumpio: DumpIOLogLevel is no longer supported. Data is always logged at LogLevel trace7. On Unix platforms, piped logging commands configured using either ErrorLog or CustomLog were invoked LogLevel configuration The LogLevel can now be configured per module and per directory. New levels trace1 to trace8 have been added above the debug log level. Per-request configuration sections , of the module in your LogLevel directive: LogLevel info rewrite:trace5 This sets the main LogLevel to info, but turns it up to trace5 for mod_rewrite. This replaces the per-module logging directives