Go in TiDB 申砾@PingCAP Agenda • About Me • What is TiDB • TiDB Architecture • SQL Layer Internal • Golang in TiDB • Next Step of TiDB About Me • Shen Li (申砾) • Tech Lead of TiDB, VP of Engineering Why Go? • Productivity • Concurrency • Great for network programming • GC • Standard libraries && Tools • Good performance • Quick improvement Go in TiDB • More than 100k lines of Go code and session is hard • Account for large memory allocation • Account for memory consuming operators New in Go1.8 • Better GC • Pauses 50μs • sort.Slice Next Step of TiDB • Switch to gRPC • Document support

Go Practices in TiDB 姚维 PingCAP wink@pingcap.com Agenda ● How to build a stable database ○ Schrodinger-test platform ○ Failpoint injection ○ Goroutine-leak detection ● Optimization ○ Chunk vs interface{} errors.New("mock commit error") // } …. } https://github.com/pingcap/tidb/blob/master/store/tikv/txn.go#L223 The generated gofail code func (txn *tikvTxn) Commit(ctx context.Context) error { if vmockCommitError or: __fp_mockCommitError.BadType(vmockCommitError, "bool"); }; …. } $ gofail enable store/tikv/txn.go Why we need a new failpoint ● Generated code is not readable ● Concurrent testing will use the same

No builds required to add new rules. BUSINESS PROBLEM Our Answer Go Language MongoDB Mgo/Beego If we had a system that could do inser*on. Two Go programs perform all the feed management work WHY GO - DYNAMIC FEED MANAGEMENT The Feed System Is Driven By Data Go supports Data from Offer and Internal Feeds DYNAMIC RULE MANAGEMENT Our Answer Go Language MongoDB Mgo/Beego Building rules is a core piece of the

Go runtime related problems in TiDB production environment About me ● Arthur Mao(毛康力), Senior Engineer@PingCAP ● TiDB core developer (top3 contributor) ● GitBook about golang internals (@tiancaiamao) consider priority ● CPU dense workload could affect IO latency Conclusion Part II - Memory control ● Go Runtime ○ Allocated from OS (mmaped) ○ Managed Memory ■ Should the memory be returned to the OS? ○ memory footprint is abnormal ● The memory available on this node is not too much Description ● The Go Runtime thinks it does not use much memory ● The OS does not release the memory (RSS is high) Investigate

Go � TiDB ����� mongodb lixia@pingcap.com @���_go About me ��� PingCAP �� 15 ���� PingCAP ���� TiDB ��������� review ������� 12 ������� go ���������� 13 ������������������������ ���� ���� ���@���_go Email: lixia@pingcap.com Agenda mongodb introduction gonzo with memory engine gonzo with TiKV engine TiDB and TiKV Transaction Q & A mongodb introduction mongodb introduction

stars 16,000+ commits Marquee adopters 100+ contributors 700+ Slack members History ● Choosing Go ● TCP, Syscalls, GC, Scheduler 2013 2015 2016 First Kubernetes Production launch Last Sunday.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Learning Go . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table lifecycle aka table garbage collection . . . . . . . . . . . . . . . . . . . . . . . . 48 Transitioning and skipping of states . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 Google I/O 2014 - Scaling with Go: YouTube’s Vitess . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 Vitess Roadmap

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Learning Go . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table lifecycle aka table garbage collection . . . . . . . . . . . . . . . . . . . . . . . . 52 Transitioning and skipping of states . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304 Google I/O 2014 - Scaling with Go: YouTube’s Vitess . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304 Vitess Roadmap

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Learning Go . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table lifecycle aka table garbage collection . . . . . . . . . . . . . . . . . . . . . . . . 50 Transitioning and skipping of states . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 Google I/O 2014 - Scaling with Go: YouTube’s Vitess . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 Vitess Roadmap

for investing in many complex data platforms. Adoption for New Solutions is Long and Costly Transitioning from legacy architectures to a unified view often requires organizational alignment, costly investments