SberBank story: moving Istio from PoC to production Igor Gustomyasov, Sber Maksim Chudnovskii, IBM Sber position across key areas Best client experience Technological leadership In financial services tuning is required 1. Resource consumption 2. Resource Mounts (#15517) 4. Tests on the production-size environment aren’t a waste of time 1. Istio Discovery Restarts (#25495) 2. Proxy Probes (#26792)

way of validating that security expectations in the code were implemented when deployed. Each environment was deployed following Istio Documentation using istioc tl. The assessment included many open customizations to fit it into different envi- ronments, but it’s difficult to say which is a hardened, production-ready approach. Having a secured profile with an opinionated cluster configuration will help guide ing guidelines first as it will give administrators more confidence that they are building an environment following best practices. Pursuing something formal such as CIS benchmarks is not recommended in

docker push kubectl apply docker pull + No local resource utilization + Closely resembles production environments + Can test large scales - Slow, especially without fast upload speeds - Expensive IDE - Very different from production environment, may not be representative - Harder to test actual traffic, especially iptables - May be dependant on local environment - Challenging to have multiple iteration - Very different from production environment, may not be representative - Harder to test actual traffic, especially iptables - May be dependant on local environment - Challenging to have multiple

terms of security. Nevertheless, the operator has not been fully deprecated and is likely used in production by the community which makes some users prone to security issues. Furthermore, successful cyber combination with VerifyConnection or VerifyPeerCertificate.” The issue was found to have no severe production impact due to this happening only in experimental code, test code and in opt-in insecure modes should ever be used in production. If it should, then this vulnerability puts users at risk from untrusted input. If debug mode should never be enabled in a production environment, then this should be clear

safe and secure transactions. Mercari offers a unique customer experience, with a transaction environment that uses the payments Mercari holds in escrow, and simple and affordable shipping options. 1 main production Google Kubernetes Engine (GKE) cluster ● 12k+ pods ● 750+ nodes Istio at Mercari 7 Istio at Mercari Apr 2019 Started Istio PoC Sep 2019 First release in production Feb Feb 2021 ~25% production services ~50% development services migrated to Istio End of 2021 100% services migrated to Istio 8 Features currently used: ● HTTP/2 Load-balancing ● Traffic

service tests from the same data • Key product benefits (#releases, #rollbacks, MTTR, #bugs-in-production, Reduced eng effort for testing, velocity) – Early testing of services components auto-generated REQUEST RESPONSE API MOCKS ASSERTION RULES CONTEXT RULES … … … … … … Test Driver TEST ENVIRONMENT Derive different types of tests Mocks for External APIs Istio enables learning tests from REQUEST RESPONSE API MOCKS ASSERTION RULES CONTEXT RULES … … … … … … Test Driver TEST ENVIRONMENT Derive different types of tests Mocks for External APIs Creating test suites from API traffic

Service Mesh enabled • mutual TLS is enabled to secure the user application traffic end to end in production • Allow platform to use Istio authorization policy to control the access to each Knative service (istio #23029, envoyproxy #13037) o envoy still suffers from overload of XDS pushes in a high churn environment. Istio scalability optimization during Knative Service provisioning • Random missing endpoint churn. This is disabled by default and can be enabled by setting the PILOT_ENABLE_FLOW_CONTROL environment variable in Istiod. o Final solution is envoy delta-XDS push in future Istio release. Istio scalability

Experiences from running Istio in a k8s production environment Line Moseng @linemoseng Johnny Horvi Norwegian Labour and Welfare Administration 5,2 million nais.io github.com/nais CD CD metrics

#IstioCon Creating the .csv Importing the file Generating the Istio configuration Deploy to production 1 2 3 4 ? SEO specialist creates the file manually Matching old URLs with the new ones based configuration Deploy to production 1 3 4 2 How does it work ? #IstioCon Creating the .csv Importing the file Generating the Istio configuration Deploy to production 1 4 2 3 Istio-redirector configuration Deploy to production 1 2 3 4 The files are reviewed, merged and deployed! How does it work ? #IstioCon >26k redirections are now running in production without any impact on performances

meshctl wasm debug workloadSelector 17 | Copyright © 2020 Build Store Deploy Debug Debug in Production Cluster 1 Acco unt User Cluster 2 Istiod Order s User AWS EKS Istiod Order s User Acco Debug in Production 19 | Copyright © 2020 Build Store Deploy Debug Debug in Production Debug Logs Access Logs Metrics 20 | Copyright © 2020 Build Store Deploy Debug Debug in Production 21 |