Storing state with containers
Kubernetes has become the preferred choice for running not only stateless workloads (e.g., web services) but also for stateful applications (e.g., e-commerce applications). According to the Data on Kubernetes report, over 70% of Kubernetes users run stateful applications in containers. Additionally, there is a rising trend of managed data services like MariaDB and Databricks using Google Kubernetes Engine to power their SaaS businesses to benefit from the portability of Kubernetes, built-in auto-upgrade features such as blue-green deployments, backup for GKE and out-of-the-box cost efficiency for better unit economics.
All of this means that container-native storage on GKE is increasingly important. Specifically, storage that can be seamlessly attached and detached to containers as they churn (because the average container lifetime is much shorter than VMs) and remain portable across zones to stay resilient. That’s where Filestore Enterprise fits in. Customers get a fully managed regional file system with four 9s of availability. Storage is instantaneously attached to containers as they churn and zonal failovers are handled seamlessly.
The rest of this blog explores multiple storage options with containers and how Filestore Enterprise fits in to help guide customers to make decisions of the best storage option that meets their needs.
Storage options
To understand the lay of the land, let’s explore three options for common patterns for attached storage with containers (note: Cloud Storage is accessed via the application code in a container and not covered here).
Local file system over a local SSD device: A local file system (over local ssd block device) is the simplest to set up and can be very cost-effective and provide good performance (over local SSD), but in most cases it lacks enterprise storage capabilities such as snapshots, backups, and asynchronous DR. Also it provides limited reliability and redundancy as the state is host local. This model is well suited for scratch space/ephemeral storage use cases, but much less so for production-level, mission-critical use cases.
Local file system over a remote/shared block device (SAN): The SAN (Storage Area Network) model is powerful and well known. A SAN-backed remote volume can provide good performance, advanced storage services, and good reliability. As the volume is external to the containers’ host, the persistent volume can be reattached (mounted) to a different host in case of container migration or if the original one failed, but is predominantly limited to only one host and Pod at a time. In the cloud world, SAN devices are replaced by networked block services, such as Google Cloud Persistent Disk (PD).
Remote/networked file system (NAS): The NAS (Network Attached Storage) model is semantically a powerful storage model as it also allows read-write sharing of the volume across several containers. In such a model the file system logic is implemented in a remote filer and accessed via a dedicated file system protocol, most commonly Network File System (NFS). In the cloud world, NAS devices are commonly replaced by file system services such as Filestore.
GCP block and file storage backends
In Google Cloud non-local storage can be implemented using either PD or Filestore. PD provides flexible SSD- or HDD-backed block storage, while Filestore provides NFSv3 file volumes. Both models are CSI (Container Storage Interface) managed and fully integrated into the GKE management system. The main advantages and disadvantages of both models (depicted below) are as follows:
- PD provides capacity-optimized storage (HDD) and good price-performance variants (SSD, Balanced). PD provides flexible sizes and zonal volumes. On the other hand, PD based volumes do not support read-write sharing. This means multiple containers can’t read and write to the same volume. Customers can choose Regional support (RePD) but this is limited to active-passive models. PD-backed volumes support container migration and failover (after host failures), but such migration or failover may require time and expertise to implement.
- Filestore provides similar HDD and SSD variants and active-active regional (enterprise) variants. All Filestore variants support the read-write sharing model and almost instantaneous container migration and failover. Because of this increased functionality, Filestore-backed volumes have higher cost compared to the PD-backed volumes and have a minimum size limit of 1TB.
Filestore as fully managed container storage
Both PD and Filestore support container native operations such as migrating containers across hosts for use cases such as upgrades or failover. Customers on PD get best-in-class price/performance with extensive selection of multiple PD types. That’s why PD is popular with many GKE customers, as they benefit from price-performance and capabilities. However, with PD, customers need to have expertise in storage systems. In PD, the file system logic is built into the host. This coupling means during migration the host must cleanly shut down the container, unmount the file system, reattach the PD to the target host, mount the file system and only then boot the container. While GKE manages a lot of these operations automatically, in the case of failover there are potential file system and disk corruption issues. Users will need to run some cleanup processes (“fsck”) on the mounted volume before it can be used.
With Filestore, customers get a fully managed regional file system that is decoupled from the host. Customers don’t need any expertise to operate storage and failovers are handled seamlessly as there are no infrastructure operations to attach/detach volumes. In addition, customers also benefit from storage that can be simultaneously read and written to by multiple containers.
In addition to the general value of the Filestore as a GKE backend, Filestore Enterprise supports mission-critical and medium-to-large stateful deployments as it adds regional (four 9s) availability, active-active zone access, instantaneous snapshots, and smaller SSD entry point for each volume.
Summary and conclusions
Google Cloud offers several fully managed options for GKE persistent volumes. In addition to the PD-based volumes, Filestore Enterprise is a first-class citizen storage backend for GKE and can also serve mission-critical use cases where (active/active) regional redundancy and fast failover/migration are important. Furthermore, Filestore Enterprise is just getting started on delivering better price-performance efficiency for customers. For example, customers can access a private preview to drive higher utilization of Filestore Enterprise instances by bin packing volumes as shares.
Summary table
