that the total number of disks in Curve system is N, the number of replicas is R, and the data recovery Time in the case of failed disks is T. The Annual Failure Rate of disks is AFR, and the average 50 other disks will restore the data on that disk at the same time. According to this, the data recovery time T can be estimated. 𝐹𝐼𝑇 = 𝐴𝐹𝑅 24 ∗ 365 The probability of failure of disk within 𝑒"#$%∗('"!)∗(,7 ∗ 1 3 ∗ 50 𝑁 3 The probability of the failure of the third disk in tr during the recovery time is as follows𝑃- = 61 − 𝑒"#$%∗(' " ))∗(,7 ∗ 1 3 ∗ 50 ∗ 3 𝑁 The annual probability of

upgrads supported FULL LIFECYCLE Plan to Support app lifecycle, storage lifecycle(backup, failure, recovery) DEEP INSIGHTS Plan to Support metrics, alerts, log processing and workload analysis AUTO PILOT upgrade automation • Supporting Curve Cluster provisioning in helm chart • metadata backup and recovery • MDS / ChunkServer should respect failure domains of Kubernetes • Support for public cloud

Belief in past success makes one invulnerable to failure • It’s a cliff - don’t fall off - recovery is EXPENSIVE • Crisis will often cause the fall • Limit what is moved to Obvious; highly vulnerable

environment. Production failure rate: how often software fails during production. Mean-time to recovery: how long it takes applications in the production stage to recover from failure. Key Principles

Enable working towards shared goals ii. See problems as they occur iii. Enable quick detection & recovery iv. Ensure features operate as intended and achieve organizational goals 2. Ch. 14 – Create Telemetry

with our deployment pipeline iv. Integrate security with our telemetry for better detection and recovery v. Protect our deployment pipeline vi. Integrate our deployment activities with our change approval

1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum Filesystem

code to process ASCII strings, and they will work as expected, both in terms of performance and semantics. If such code encounters non-ASCII text, it will gracefully fail with a clear error message, rather Non-Standard String Literals There are situations when you want to construct a string or use string semantics, but the behavior of the standard string construct is not quite what is needed. For these kinds functions with support for special syntax. (The exceptions are operators with special evaluation semantics like && and ||. These operators cannot be functions since Short-Circuit Evaluation requires that

code to process ASCII strings, and they will work as expected, both in terms of performance and semantics. If such code encounters non-ASCII text, it will gracefully fail with a clear error message, rather Non-Standard String Literals There are situations when you want to construct a string or use string semantics, but the behavior of the standard string construct is not quite what is needed. For these kinds functions with support for special syntax. (The exceptions are operators with special evaluation semantics like && and ||. These operators cannot be functions since Short-Circuit Evaluation requires that

code to process ASCII strings, and they will work as expected, both in terms of performance and semantics. If such code encounters non-ASCII text, it will gracefully fail with a clear error message, rather Non-Standard String Literals There are situations when you want to construct a string or use string semantics, but the behavior of the standard string construct is not quite what is needed. For these kinds functions with support for special syntax. (The exceptions are operators with special evaluation semantics like && and ||. These operators cannot be functions since Short-Circuit Evaluation requires that