Kubernetes Chaos Engineering Tip
Chaos Engineering is an approach to test systems by inducing controlled chaos. The motivation is to harden your systems against random failures and monitor the reactions. Examples could be network outages or failed deployments.
Kubernetes Chaos Engineering Tip: Build your staging cluster on preemptible nodes1
Building your staging cluster on preemptibles (or spot instances2 in AWS terminology) creates chaos scenarios continuously. Services have to recover repeatedly from termination and dependency outages.
Explanation
A Kubernetes system is built on one or more nodes. Theses nodes are virtual machines that balance the orchestrated load amongst eachother. These nodes can be regular instances or preemptibles
A preemptible VM is an instance that you can create and run at a much lower price than normal instances. However, Compute Engine might terminate (preempt) these instances if it requires access to those resources for other tasks. Preemptible instances are excess Compute Engine capacity, so their availability varies with usage.1
Having your deployments configured for high availability, you should not notice any outages on the staging services.