Request-2 Response-1 Request-3 Response2 Response-3 Request-4 Response-4 RPC-1 RPC-2 RPC-3 RPC-4 RPC-1 RPC-2 RPC-3 RPC-4 Blocking Client (Single Thread) Non-Blocking Client(Single Thread) Fault Client RegionServer RegionServer RegionServer Handler-1 Handler-2 Handler-3 Services RegionServer RegionServer RegionServer Handler-1 Handler-1 Handler-1 Services Get Stuck Get Stuck Get Stuck All Availability: 66% Availability: 0% Why Async HBase Client ? ● Region Server / Master STW GC ● Slow RPC to HDFS ● Region Server Crash ● High Load ● Network Failure BTW: HBase may also suffer from fault

HFile ’s Block into ByteBuffer directly. 1. Background For reducing the Java GC impact to p99/p999 RPC latency, HBase 2.x has made an offheap read and write path. The KV are allocated from the JVM offheap Now consider the implementation: Firstly, we need a global ByteBuffAllocator for RPC. when reading a block in RPC, we’ll do: 1. Allocate ByteBuff from ByteBuffAllocator, and read the data from HFile BucketCache asynchronously (don’t block the RPC); In theory, if the RPC finished, we need to free the ByteBuff, but we can’t: ByteBuff can also be referenced by other RPC because it's still in RAMCache and

conventions / style? Linter was run and is happy? • Does this match our current patterns? Example include RPC patterns, Retries / Waits / Timeouts patterns using Context, … Additionally, we recommend that every messages that are older than 86400 seconds (1 day). • vt_batch_size=10: Send up to 10 messages per RPC packet. • vt_cache_size=10000: Store up to 10000 messages in the cache. If the demand is higher, the will come at a higher latency cost and very low throughput. We never use the Topology Service as an RPC or queuing mechanism or as a storage system for logs. We never depend on the Topology Service being

RegionServer-0 RegionServer-1 RegionServer-2 scanResultCache ScannerCallableWithReplicas 1. RPC Request 2. RPC Response 3. Regroup and enqueue 4. Return a result Step.1 + Step.2 + Step.3 = loadCache Result-3 Cell-5 Result-4 Cell-7 Result-5 Cell-6 Cell-8 Cell-2 Result-2 Cell-3 RegionServer Row Data RPC Response Recieved from RS Results get from scanner.next() Cell-9 Size of cell-1 > 1MB scan.setCaching(2) Result-3 Cell-5 Result-4 Cell-7 Result-5 Cell-6 Cell-8 Cell-2 Result-2 Cell-3 RegionServer Row Data RPC Response Recieved from RS Results get from scanner.next() Cell-1 Cell-2 Result-1 Cell-3 Cell-4 Result-2

of bad transient state or bad user queries) * Does this match our current patterns? Example include RPC patterns, Retries / Waits / Timeouts patterns using Context, … We also recommend that every author messages that are older than 86400 seconds (1 day). • vt_batch_size=10: Send up to 10 messages per RPC packet. • vt_cache_size=10000: Store up to 10000 messages in the cache. If the demand is higher, the will come at a higher latency cost and very low throughput. We never use the Topology Service as an RPC or queuing mechanism or as a storage system for logs. We never depend on the Topology Service being

bad transient state or bad user queries) • Does this match our current patterns? Example include RPC patterns, Retries / Waits / Timeouts patterns using Context, … We also recommend that every author While vttablet is up Every flow that needs to change something about a tablet directly issues an rpc request to it. The tablet will immediately execute the request, and will perform a best-effort action messages that are older than 86400 seconds (1 day). • vt_batch_size=10: Send up to 10 messages per RPC packet. • vt_cache_size=10000: Store up to 10000 messages in the cache. If the demand is higher, the

bad transient state or bad user queries) • Does this match our current patterns? Example include RPC patterns, Retries / Waits / Timeouts patterns using Context, … We also recommend that every author While vttablet is up Every flow that needs to change something about a tablet directly issues an rpc request to it. The tablet will immediately execute the request, and will perform a best-effort action messages that are older than 86400 seconds (1 day). • vt_batch_size=10: Send up to 10 messages per RPC packet. • vt_cache_size=10000: Store up to 10000 messages in the cache. If the demand is higher, the

bad transient state or bad user queries) • Does this match our current patterns? Example include RPC patterns, Retries / Waits / Timeouts patterns using Context, … We also recommend that every author While vttablet is up Every flow that needs to change something about a tablet directly issues an rpc request to it. The tablet will immediately execute the request, and will perform a best-effort action messages that are older than 86400 seconds (1 day). • vt_batch_size=10: Send up to 10 messages per RPC packet. • vt_cache_size=10000: Store up to 10000 messages in the cache. If the demand is higher, the

bad transient state or bad user queries) • Does this match our current patterns? Example include RPC patterns, Retries / Waits / Timeouts patterns using Context, … We also recommend that every author While vttablet is up Every flow that needs to change something about a tablet directly issues an rpc request to it. The tablet will immediately execute the request, and will perform a best-effort action pacity • vttablet user flag: -db_dba_user (default ‘vt_dba’) • Used by vttablet ExecuteFetchAsDba RPC. This is used when using vtctlclient ExecuteFetchAsDba Also used implic- itly for various internal

Postgres Conference China 2016 中国用户大会 HBase的GC特点 • 由单个RPC带来的操作类垃圾对象是短期的 • Memstore是相对长期驻留的，按2MB为单位分配 • Blockcache是长期驻留的，按64KB为单位分配 • 如何有效的回收RPC操作带来的临时对象是HBase 的GC重点 • 不建议HBase的堆大小操作操过64GB，否则GC压 力大、执行时间太长 的读性能 Postgres Conference China 2016 中国用户大会 写性能 • HBase理论平均写延时<10ms，时间复杂度O(1) • 没有可用的handler响应 – 考虑增加handler数目或硬件资源 • 更常见的情况是95%－99%的写入都很快，但有些 写入非常慢，甚至慢上万倍，一般问题在服务器端： – 写入Memstore慢 • HLog写入超时——考虑HDFS及硬盘异常