Go runtime related problems in TiDB production environment About me ● Arthur Mao(毛康力), Senior Engineer@PingCAP ● TiDB core developer (top3 contributor) ● GitBook about golang internals (@tiancaiamao) IO is ready => goroutine wake up == 4.3ms ○ Sometime even 10ms+ latency here! ○ The time spend on runtime schedule is not negligible ● When CPU is overload, which goroutine should be given priority? Analysis longer to be scheduled ● The runtime scheduling does not consider priority ● CPU dense workload could affect IO latency Conclusion Part II - Memory control ● Go Runtime ○ Allocated from OS (mmaped)

Go in TiDB 申砾@PingCAP Agenda • About Me • What is TiDB • TiDB Architecture • SQL Layer Internal • Golang in TiDB • Next Step of TiDB About Me • Shen Li (申砾) • Tech Lead of TiDB, VP of Engineering Why Go? • Productivity • Concurrency • Great for network programming • GC • Standard libraries && Tools • Good performance • Quick improvement Go in TiDB • More than 100k lines of Go code and bytes pool Test on a query reads 1 million rows Before # runtime.MemStats # Alloc = 1985803616 # TotalAlloc = 4365147096 After # runtime.MemStats # Alloc = 1303384504 # TotalAlloc = 1676585832

浅谈全链路监控： 从应用到数据库到 Runtime 黄东旭, Co-founder & CTO, PingCAP 关于我 黄东旭，联合创始人 & CTO @ PingCAP 做分布式数据库的程序员 ● 现在能写代码的时间是奢侈品 TiDB 的亲爹之一兼首席客服和新功能的第一个用户 ● 冤有头债有主，SQL 慢了来找我。。。 偶尔玩玩音乐 ● 摇滚乐->实验音乐 Go 的粉丝！！！！ 关于 Trace & Span 例子 Jaeger ● CNCF 毕业项目 ● 实现 OpenTracing 标准 ● UI 友好 ● 多种 Sampling 策略 ● 后端存储分离 ● Go! (图标也好看) Example: Example: Carrier ● Tracing 信息携带者 ○ 解决跨服务调用的 Tracing metadata 序列化和反序列化抽象 ● Network Ready Goroutine Wakeup 4.368ms go tool trace go tool trace ● 优点：好用，好看（UI） ● 缺点：性能损耗太大，不能一直开着 Trace in Go runtime ● go tool trace 的原理是？ Trace 会 Go Runtime 的代码中打桩收集 CPU time，在 Goroutine 开始执行时记录

Go Practices in TiDB 姚维 PingCAP wink@pingcap.com Agenda ● How to build a stable database ○ Schrodinger-test platform ○ Failpoint injection ○ Goroutine-leak detection ● Optimization ○ Chunk vs interface{} errors.New("mock commit error") // } …. } https://github.com/pingcap/tidb/blob/master/store/tikv/txn.go#L223 The generated gofail code func (txn *tikvTxn) Commit(ctx context.Context) error { if vmockCommitError or: __fp_mockCommitError.BadType(vmockCommitError, "bool"); }; …. } $ gofail enable store/tikv/txn.go Why we need a new failpoint ● Generated code is not readable ● Concurrent testing will use the same

No builds required to add new rules. BUSINESS PROBLEM Our Answer Go Language MongoDB Mgo/Beego If we had a system that could do inser*on. Two Go programs perform all the feed management work WHY GO - DYNAMIC FEED MANAGEMENT The Feed System Is Driven By Data Go supports Data from Offer and Internal Feeds DYNAMIC RULE MANAGEMENT Our Answer Go Language MongoDB Mgo/Beego Building rules is a core piece of the

Go � TiDB ����� mongodb lixia@pingcap.com @���_go About me ��� PingCAP �� 15 ���� PingCAP ���� TiDB ��������� review ������� 12 ������� go ���������� 13 ������������������������ ���� ���� ���@���_go Email: lixia@pingcap.com Agenda mongodb introduction gonzo with memory engine gonzo with TiKV engine TiDB and TiKV Transaction Q & A mongodb introduction mongodb introduction

stars 16,000+ commits Marquee adopters 100+ contributors 700+ Slack members History ● Choosing Go ● TCP, Syscalls, GC, Scheduler 2013 2015 2016 First Kubernetes Production launch Last Sunday.

4 40 Request ID CPU Usage 40 30 20 10 1 2 3 4 CPU Profiling By SQL Request ● rust-demo ● go-demo TopSQL: SQL Tag - Digest select id from t where a=1; select id from t where a=2; select id Usage By Tag CPU Usage By Tag Another approach to CPU resource bind in Go Goroutine CPU Stats ● Try to collect goroutine runtime information. begin := GetGoroutineStats()
 
 executeSQL()
 
 end := GetGoroutineStats()
 获取执⾏ SQL 消耗的 cpu stats 信息 Trace ● Use Go trace to collect runtime information. curl http://localhost:10080/debug/pprof/trace\?seconds\=1 --output trace.out go tool trace trace.out ● In Goroutine analysis

............................................................................ 17 Microsoft C/C++ Runtime Libraries ...................................................................................... Mixed usage of the NONE and other character sets is not recommended as it can lead to to unexpected runtime errors. CHECK Constraint Change Formerly, CHECK constraints were not SQL standard-compliant in regard particularly for Windows Embedded deployments. Please refer to the later section entitled Microsoft Runtime Libraries. Installation Choices On Windows, you have three server models to choose from: Superserver

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · 132 4.3.2 Go · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · majority of replicas. This guarantees strong consistency and availabil- ity when a minority of replicas go down. You can configure the geographic location and number of replicas as needed to meet different #48755 @winoros • Fix the issue that queries containing common table expressions (CTEs) re- port runtime error: index out of range [32] with length 32 when tidb_max_chunk_size is set to a small value #48808