and rehearse accidents for practice 1. Schedule the event 2. Give teams time to prepare, make changes, and establish procedures 3. Execute iii. Expose latent defects in the systems and create progressively as possible and incorporate into the pipeline iii. Enable fast feedback on potentially insecure changes g. ENSURE SECURITY OF THE APPLICATION i. Focus on the sad paths or bad paths to effectively address private changes l. CREATE SECURITY TELEMETRY IN OUR ENVIRONMENT i. Create environment telemetry 1. OS changes in production, build infrastructure, etc. 2. Security group changes 3. Changes to configurations

to transform IT acquisition by delivering capabilities faster and responding more effectively to changes in operations, technology, and budgets. This guide provides DoD acquisition professionals with details releases  Valuing working software over comprehensive documentation  Responding rapidly to changes in operations, technology, and budgets  Actively involving users throughout development to ensure effectively measure progress, reduce technical and programmatic risk, and respond to feedback and changes more quickly than traditional methods. Programs can adopt Agile practices within current policy

track what they need to track, at the drop of a hat, without requiring time-sucking configuration changes or complicated processes.” iv. 2015 State of DevOps Report – high performers had MTTR 168x faster ization decisions 2. System and data access 3. System and application changes, especially privileged changes 4. Data changes (CRUD) 5. Invalid input, possible malicious injections or threats 6. Resources or Ops to create telemetry for functions built ii. Generate graphs with overlays of production changes iii. Tools – StatsD, JMX, codahale, New Relic, Prometheus, etc. f. CREATE SELF-SERVICE ACCESS TO

setting. Compiled templates are cached by default; to turn off this caching and reload templates so changes to the underlying files are always visible, use the application settings compiled_template_cache=False flag takes precedence): • autoreload=True: The app will watch for changes to its source files and reload itself when anything changes. This reduces the need to manually restart the server during development to provide extra robustness against syntax errors: set autoreload=True within the app to detect changes while it is running, and start it with python -m tornado.autoreload myserver.py to catch any syntax

— Bridge between asyncio and Tornado Utilities tornado.autoreload — Automatically detect code changes in development tornado.concurrent — Work with Future objects tornado.log — Logging support tornado setting. Compiled templates are cached by default; to turn off this caching and reload templates so changes to the underlying files are always visible, use the application settings compiled_template_cache=False individual flag takes precedence): autoreload=True: The app will watch for changes to its source files and reload itself when anything changes. This reduces the need to manually restart the server during development

discovery: only 50% of the source code in Dev/Test environments matched Prod. They fixed forward, but changes not put back into version control. Focused on version control and automated environment creation environments (Prod, pre-prod(s), dev, & new) iv. Immutable Infrastructure 1. Once created no changes are allowed 2. Changes must be put into version control and then new infrastructure is created 3. To prevent that learning into future work d. Google Web Server (GWS) team was struggling with changes – Hard line: no changes would be accepted into GWS without accompanying automated tests. Spread Testing Grouplet

features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1675 104.2 Language changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1675 104.3 Compiler/Runtime . . . . . . . . . 1676 104.4 Command-line option changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1676 104.5 Build system changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1676 104.8 Standard library changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1677 104.9 Deprecated or removed

features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1675 104.2 Language changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1675 104.3 Compiler/Runtime . . . . . . . . . 1676 104.4 Command-line option changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1676 104.5 Build system changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1676 104.8 Standard library changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1677 104.9 Deprecated or removed

features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2041 109.2 Language changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2041 109.3 Compiler/Runtime . . . . . . . . 2041 109.4 Command-line option changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2041 109.5 Multi-threading changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2041 109.6 Build system changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2042 109.7 New library functions . . . . . . . . . . . . . . . . . . . . . . . . . .

features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1986 108.2 Language changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1986 108.3 Compiler/Runtime . . . . . . . . 1987 108.4 Command-line option changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1987 108.5 Multi-threading changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1987 108.6 Build system changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1987 108.7 New library functions . . . . . . . . . . . . . . . . . . . . . . . . . .