Kubernetes on ege.dev

Point-in-time recovery for MySQL on Kubernetes

Fri, 27 Mar 2026 00:00:00 +0000

Since the v1.0.0 release of the new MySQL Operator (K8SPS), point-in-time recovery (PiTR) has been the most anticipated feature. Naturally, we decided to implement it in the upcoming v1.1.0 release.

PiTR relies on two processes:

Reliably collecting binary logs from MySQL servers and storing them somewhere safe
When the recovery is triggered, applying those binary logs up to a specific point

Collecting binary logs

Our Galera replication-based MySQL Operator (K8SPXC) has a binary log collector that was developed by the Cloud Team. It has worked reliably for years but has a certain limitation that is inherent in its design: it depends on flushing the binary logs to collect them. This leads to huge numbers of binary logs on the MySQL server, which becomes a headache after running the collector for a few weeks. We mitigated this for the users by maintaining a cache for binary logs, but it didn’t remove the pain—it just became ours.

At Percona, #FindABetterWay is one of our core values. In the spirit of finding a better way, when we first started to think about PiTR in K8SPS, we decided to improve the process of collecting binary logs. These discussions eventually led to the birth of a new product: Percona Binlog Server (PBS).

PBS works by connecting to the MySQL server as a replica and streaming events. It either uploads these events to S3 or stores them on the filesystem. It supports replication source switchovers and is able to continue from where it left off. On top of these, it provides helper commands to search for a particular GTID or timestamp in the collected binary logs.

Applying binary logs

The official MySQL docs suggest converting each binary log to text using mysqlbinlog and piping them into the mysql client for PiTR. This is already what we do in K8SPXC.

I decided to check if there’s a better way. First, I checked old posts on the Percona Blog to see if our experts had written anything about a different PiTR approach. It’s not surprising that they did. Marcelo wrote about an approach leveraging replication appliers for recovery. It seemed much better than piping mysqlbinlog output into the client, since with replication we can have multithreading and parallel appliers for recovery. My only problem with this approach was that it required two mysqld instances. Of course it’s possible, but I would love to not have to care about the state of two MySQL servers.

Luckily, lefred commented that there’s an “even better” approach that requires only one MySQL server!

At a high level, the process looks like this:

Restore the full backup into the MySQL datadir
Start a temporary mysqld instance
Download binary logs and put them as relay logs in the datadir
Start replication via CHANGE REPLICATION SOURCE TO RELAY_LOG_FILE=..., SOURCE_HOST='dummy'
Start applying binary logs via START REPLICA SQL_THREAD UNTIL <GTID>
Wait for the SQL thread to stop
STOP REPLICA; RESET REPLICA ALL;

I have already created an unpolished but working PoC. There is also an RFC for explaining various decisions and tracking open questions that need answers. There’s still work to do, but I’m confident that this approach is good and we’ll release this as a tech preview in the upcoming v1.1.0 release. If you have thoughts on the RFC or want to try the PoC, we’d love to hear your feedback.

My talk at the Cloud‑Native Databases 2026

Mon, 16 Mar 2026 14:31:03 +0300

My talk at the Cloud‑Native Databases Conference is finally live!

You can watch it on YouTube and find the slide deck here.

gremlins in HAProxy

Thu, 12 Mar 2026 00:00:00 +0000

We are in the process of certifying our operators for <redacted>. We started with PostgreSQL Operator and it worked just fine without any adjustments. Then we moved on to our MySQL Operators and it surfaced a problem in HAProxy.

HAProxy is used by default in our MySQL clusters. They sit in front of MySQL instances as the proxy to have read/write splitting. We have our own external scripts to perform checks for each backend for determining if a MySQL server is good for that particular backend.

After deploying the operator on <redacted>, we realized that our HAProxy pods are failing to get ready because all external checks are failing due to timeouts. But why?

It’s gotta be the DNS. It’s always DNS, isn’t it? Turns out, no. I tested DNS queries from the HAProxy container and it seems they were fast.

Could this be AppArmor? Maybe <redacted> has stricter AppArmor profiles? I configured HAProxy pods to be Unconfined. It didn’t help either.

Then I decided to increase the timeout from 10 seconds to 30 seconds, just to see how much time the script needs to finish. To my surprise, the script was taking 810 milliseconds to finish! How could a check time out in 10 seconds but finish in less than a second with 30 seconds timeout?

In every deep debugging session, there is a moment when engineers start to believe in spiritual beings or gremlins who mess with the software in the system. One needs to resist this temptation of irrationality. In computers there’s always a rational explanation for problems, it’s just very deep and caused by unlucky combinations of design choices and/or bugs.

At this point I decided to attach a debug container into the pod and check what HAProxy is doing with strace. This resulted in the first breakthrough of the problem: the child process of HAProxy that runs the external check command was doing shitloads of poll syscalls until it was killed due to timeout. It wasn’t even running the script, it was simply stuck.

This realization made me shift my focus to HAProxy itself. I started to read the source code to see what’s going on.

src/extcheck.c

int fd;
sa_family_t family;

/* close all FDs. Keep stdin/stdout/stderr in verbose mode */
fd = (global.mode & (MODE_QUIET|MODE_VERBOSE)) == MODE_QUIET ? 0 : 3;

my_closefrom(fd);

/* restore the initial FD limits */
limit.rlim_cur = rlim_fd_cur_at_boot;
limit.rlim_max = rlim_fd_max_at_boot;
if (raise_rlim_nofile(NULL, &limit) != 0) {
	getrlimit(RLIMIT_NOFILE, &limit);
	ha_warning("External check: failed to restore initial FD limits (cur=%u max=%u), using cur=%u max=%u\n",
		   rlim_fd_cur_at_boot, rlim_fd_max_at_boot,
		   (unsigned int)limit.rlim_cur, (unsigned int)limit.rlim_max);
}

HAProxy attempts to close all FDs. What does all mean? And then it restores limits to some value? What’s that value?

All FDs means closing all from FD 0 to FD soft limit. my_closefrom function first polls the FD and then closes it if poll doesn’t fail with POLLNVAL. OK, this explains the excessive polling I see with strace. But it’s the default HAProxy behavior for a long time, why didn’t we see the same problem on some other platform, i.e GKE?

The answer is simple. Turns out, <redacted> has a much higher soft limit than GKE. On GKE ulimit -n returns 1048576, on <redacted> 1073741816! But still there was something that troubled me at this point: I vaguely remembered a configuration option in HAProxy that limits the number of FDs that the process will use. The option is fd-hard-limit and docs say its default value is 1048576. If it had any effect, I wouldn’t have seen any problems. Something was wrong in HAProxy.

Remember that the external check process was doing something to restore limits to some value assigned at boot? Those values are assigned in the main function here:

src/haproxy.c

/* take a copy of initial limits before we possibly change them */
getrlimit(RLIMIT_NOFILE, &limit);

if (limit.rlim_max == RLIM_INFINITY)
	limit.rlim_max = limit.rlim_cur;
rlim_fd_cur_at_boot = limit.rlim_cur;
rlim_fd_max_at_boot = limit.rlim_max;

The code that limits FDs with fd-hard-limit is a few lines below:

if (global.fd_hard_limit && limit.rlim_cur > global.fd_hard_limit)
	limit.rlim_cur = global.fd_hard_limit;

This means external checks restore limits to the value unbounded by fd-hard-limit. Oh, this looks like a bug in HAProxy and explains why we had this issue!

watching it shatter

Fri, 06 Mar 2026 00:00:00 +0000

This week, just like the previous, was filled with PostgreSQL (PG). I’m still working on v2.9.0 release of the PG operator.

Last week I merged the implementation of pg_tde to bring data-at-rest encryption to Kubernetes. Then we started doing QA: oh boy, did it ever surface problems!

One of the goals of v2.9.0 is declaring major version upgrades production-ready. So we decided to test the upgrade with TDE enabled. Boom, data corruption. Point-in-time recovery is problematic, pg_rewind too…

This is one of my favorite things about Kubernetes. You put something fragile on it and watch it shatter. Of course I need to give credit to our QA engineers: they don’t stop until they watch the software crumbling in pain.

Outside of PG stuff, I opened a small PR for adding volume mount information to kubectl-status plugin. I needed to quickly check mounts for a few containers while testing PG major upgrades and did a quick implementation with Claude. Before opening the PR, I was unsure whether I should mention that the code was written by Claude. I still feel embarrassed about using AI code generation. In the end I decided to openly admit it because it’s the right thing to do. I also want our contributors to be open about their usage of AI (I know they use AI).

Speaking of AI code generation, I feel that the vibe has shifted significantly. I see the pessimists have accepted defeat and I am one of them: AI can write code and the code is not crap. It’s not 2023 anymore and it looks like it’s time to update my priors. I am trying to integrate Claude Code into my workflows and I had a bunch of positive interactions with it the last few weeks. I am also coding a big feature using Claude just for the sake of the experience.

Meet me at KubeCon Europe

Wed, 25 Feb 2026 15:10:57 +0300

The Percona team is heading to KubeCon + CloudNativeCon Europe in Amsterdam, and I’d love to meet you in person!

You can find me at Booth 790. This is a great chance to talk with me and other engineers working on Percona Operators.

We will be there to discuss:

Running MySQL, PostgreSQL, and MongoDB on Kubernetes
Production-ready HA setups
Backup and PITR strategies
Multi-cluster and multi-region deployments
Operators roadmap and upcoming features
Real-world troubleshooting stories

If you’re already running Percona Operators in production (or just getting started), I’d love to hear your feedback and learn about the challenges you’re facing. And if you’re just curious—or even a little suspicious—about running databases on Kubernetes, come by! I’d love to answer your questions and share our perspective.

Want to chat?

We have a 20% discount code available for Percona community members. If you’re planning to attend and don’t have a ticket yet, let me know and I’ll send the code your way.

If you’d like to carve out some dedicated time to talk, let’s schedule a meeting or grab a coffee together. You can reach out to me directly at ege.gunes [at] percona [dot] com.

See you in Amsterdam!

Encrypt PostgreSQL Data at Rest on Kubernetes

Fri, 07 Feb 2025 15:10:57 +0300

The upcoming Percona Operator for PostgreSQL v2.6.0 release introduces support for PostgreSQL 17, which opens exciting possibilities for data security. Since pg_tde comes pre-installed in Percona’s official PostgreSQL 17 images, this release presents an excellent opportunity to implement Transparent Data Encryption in your Kubernetes-deployed databases. Let’s look at how to configure and use pg_tde with the Percona PostgreSQL Operator.

Understanding pg_tde

Transparent Data Encryption (TDE) offers encryption at the table level and solves the problem of protecting data at rest. The Percona Distribution for PostgreSQL currently offers the only open source implementation of TDE for PostgreSQL. The encryption is transparent for users, allowing them to access and manipulate the data without requiring application modifications.

While pg_tde has a build available for PostgreSQL Community Server, the extension leverages extended APIs introduced in Percona Server for PostgreSQL to provide more complete and performant encryption. You can read more about the pg_tde extension in the TDE beta announcement blog post.

The new Percona Operator release will include pg_tde pre-installed with PostgreSQL 17, simplifying the implementation of encryption at rest. Note: Currently, pg_tde is in the beta phase. Please try it out on non-production setups and share your feedback!

Initial configuration

To begin, we need to configure PostgreSQL to load pg_tde during startup. This configuration is managed through the deploy/cr.yaml file:

patroni:
  dynamicConfiguration:
    postgresql:
      parameters:
        shared_preload_libraries: pg_tde

After updating the configuration, apply it and restart your PostgreSQL pods:

kubectl apply -f deploy/cr.yaml

for sts in $(kubectl get statefulset -o name | grep cluster1-instance1); do
    kubectl rollout restart $sts
done

Note: Attempting to enable pg_tde without properly configuring shared_preload_libraries will result in an error indicating that pg_tde can only be loaded at server startup.

Secure key management with HashiCorp Vault

While pg_tde supports file-based key storage, it is recommended that production environments use a Key Management Service such as HashiCorp Vault or OpenBao to ensure secure key management.

Deploy Vault in your Kubernetes cluster:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault 
  --disable-openapi-validation 
  --version 0.16.1 
  --namespace vault 
  --set dataStorage.enabled=false 
  --set global.logLevel=trace 
  --set global.platform=kubernetes

After deploying Vault, we need to initialize it and obtain the root token. This process involves creating an initial set of encryption keys and unsealing the Vault instance:

kubectl exec -it vault-0 -- vault operator init 
  -tls-skip-verify 
  -key-shares=1 
  -key-threshold=1 
  -format=json >vault.json

unsealKey=$(jq -r ".unseal_keys_b64[]" <vault.json)
token=$(jq -r ".root_token" <vault.json)

kubectl exec -it vault-0 -- vault operator unseal -tls-skip-verify "$unsealKey"

kubectl exec -it vault-0 -- sh -c 
  "export VAULT_TOKEN=$token && 
   export VAULT_LOG_LEVEL=trace && 
   vault secrets enable --version=1 -tls-skip-verify -path=secret kv"

Now, we can configure pg_tde to use Vault as the key provider:

SELECT pg_tde_add_key_provider_vault_v2(
  'vault-provider',
  '<rootToken>',
  'http://vault.vault.svc.cluster.local:8200',
  'secret',
  NULL
);

SELECT pg_tde_set_principal_key('tde-principal-key','vault-provider');

Creating encrypted tables

Percona Server for PostgreSQL offers two encryption methods:

tde_heap: Available exclusively in Percona Server for PostgreSQL 17, this method provides comprehensive encryption of tuples, Write Ahead Log (WAL), and indexes with optimized performance.
tde_heap_basic: Compatible with Community PostgreSQL 16 and 17, this method encrypts tuples and WAL. Unfortunately this method does not encrypt indexes, which is an important limitation to the production use cases. Considering the performance limitations of this method encrypting/decrypting data whenever it enters/exits shared buffers it is not recommended to use this method in production use cases.

When using Percona Server for PostgreSQL images, we can utilize the more comprehensive tde_heap method:

CREATE TABLE encrypted_data (
  id INTEGER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  t text not null
) USING tde_heap;

Verify the encryption status of your table:

SELECT pg_tde_is_encrypted('encrypted_data');

For more detailed information about pg_tde capabilities and configurations, please refer to the official Percona documentation.

Current implementation status and future plans

In the current release of Percona Operator for PostgreSQL, implementing pg_tde requires manual configuration steps as outlined above. Users need to explicitly configure shared libraries, set up key providers, and manage the encryption infrastructure themselves. This approach, while functional, requires careful attention to detail and a thorough understanding of both pg_tde and Kubernetes operations.

However, future releases of the Percona Operator will introduce seamless integration with pg_tde, significantly simplifying the encryption implementation process. The operator will handle the underlying configuration automatically, allowing users to focus primarily on their database design. In these upcoming versions, enabling encryption will be as straightforward as creating tables with the tde_heap access method while the operator manages all the necessary infrastructure and configuration behind the scenes.

Authenticating Your Clients to Mongodb on Kubernetes Using X509 Certificates

Sun, 13 Feb 2022 20:42:57 +0300

Managing database users and their passwords can be a hassle. Sometimes, they could even wait in various configuration files, hardcoded. Using certificates can help you avoid the toil of managing, rotating, and securing user passwords, so let’s see how to have x509 certificate authentication with the Percona Server for MongoDB Operator and cert-manager.

cert-manager is our recommended way to manage TLS certificates on Kubernetes clusters. The operator is already integrated with it to generate certificates for TLS and cluster member authentication. We’re going to leverage cert-manager APIs to generate valid certificates for MongoDB clients.

There are rules to follow to have a valid certificate for user authentication:

A single Certificate Authority (CA) MUST sign all certificates.
The certificate’s subject MUST be unique.
The certificate MUST not be expired.

For the complete requirements, check the MongoDB docs.

Creating Valid Certificates for Clients

Let’s check our current certificates:

$ kubectl get cert
NAME                      READY   SECRET                    AGE
cluster1-ssl              True    cluster1-ssl              17h
cluster1-ssl-internal     True    cluster1-ssl-internal     17h

The operator configures MongoDB nodes to use “cluster1-ssl-internal” as the certificate authority. We’re going to use it to sign the client certificates to conform to Rule 1.

First, we need to create an Issuer:

$ kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
 name: cluster1-psmdb-x509-ca
spec:
 ca:
   secretName: cluster1-ssl-internal
EOF

Then, our certificate:

$ kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: cluster1-psmdb-egegunes
spec:
 secretName: cluster1-psmdb-egegunes
 isCA: false
 commonName: egegunes
 subject:
   organizations:
     - percona
   organizationalUnits:
     - cloud
 usages:
   - digital signature
   - client auth
 issuerRef:
   name: cluster1-psmdb-x509-ca
   kind: Issuer
   group: cert-manager.io
EOF

The “usages” field is important. You shouldn’t touch its values. You can change the “subject” and “commonName” fields as you wish. They’re going to construct the Distinguished Name (DN) and DN will be the username.

$ kubectl get secret cluster1-psmdb-egegunes -o yaml \
    | yq3 r - 'data."tls.crt"' \
    | base64 -d \
    | openssl x509 -subject -noout

subject=O = percona, OU = cloud, CN = egegunes

Let’s create the user:

rs0:PRIMARY> db.getSiblingDB("$external").runCommand(
 {
   createUser: "CN=egegunes,OU=cloud,O=percona",
   roles: [{ role: 'readWrite', db: 'test' }]
 }
)

{
       "ok" : 1,
       "$clusterTime" : {
               "clusterTime" : Timestamp(1643099623, 3),
               "signature" : {
                       "hash" : BinData(0,"EdPrmPJqfgRpMEZwGMeKNLdCe10="),
                       "keyId" : NumberLong("7056790236952526853")
               }
       },
       "operationTime" : Timestamp(1643099623, 3)
}

We’re creating the user in the “$external” database. You need to use “$external” as your authentication source. Note that we’re reversing the subject fields, this is important.

Authenticating With the Certificate

I have created a simple Go application to show how you can use x509 certificates to authenticate. It’s redacted here for brevity:

// ca.crt is mounted from secret/cluster1-ssl
caFilePath := "/etc/mongodb-ssl/ca.crt"

// tls.pem consists of tls.key and tls.crt, they're mounted from
secret/cluster1-psmdb-egegunes
certKeyFilePath := "/tmp/tls.pem"

endpoint := "cluster1-rs0.psmdb.svc.cluster.local"

uri := fmt.Sprintf(
       "mongodb+srv://%s/?tlsCAFile=%s&tlsCertificateKeyFile=%s",
       endpoint,
       caFilePath,
       certKeyFilePath,
)

credential := options.Credential{
       AuthMechanism: "MONGODB-X509",
       AuthSource:    "$external",
}

opts := options.Client().SetAuth(credential).ApplyURI(uri)

client, _ := mongo.Connect(ctx, opts)

The important part is using “MONGODB-X509” as the authentication mechanism. We also need to pass the CA and client certificate in the MongoDB URI.

$ kubectl logs psmdb-x509-tester-688c989567-rmgxv
2022/01/25 07:50:09 Connecting to database
2022/01/25 07:50:09 URI: mongodb+srv://cluster1-rs0.psmdb.svc.cluster.local/?tlsCAFile=/etc/mongodb-ssl/ca.crt&tlsCertificateKeyFile=/tmp/tls.pem
2022/01/25 07:50:09 Username: O=percona,OU=cloud,CN=egegunes
2022/01/25 07:50:09 Connected to database
2022/01/25 07:50:09 Successful ping

You can see the complete example in this repository. If you have any questions, please add a comment or create a topic in the Percona Forums.

Disaster Recovery for MongoDB on Kubernetes

Fri, 08 Oct 2021 15:10:57 +0300

This is a joint post with Sergey Pronin.

As per the glossary, Disaster Recovery (DR) protocols are an organization’s method of regaining access and functionality to its IT infrastructure in events like a natural disaster, cyber attack, or even business disruptions related to the COVID-19 pandemic. When we talk about data, storing backups on remote servers is enough to pass DR compliance checks for some companies. But for others, Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO) are extremely tight and require more than just a backup/restore procedure.

In this blog post, we are going to show you how to set up MongoDB on two distant Kubernetes clusters with Percona Distribution for MongoDB Operator to meet the toughest DR requirements.

What to Expect

Here is what we are going to do:

Setup two Kubernetes clusters
Deploy Percona Distribution for MongoDB Operator on both of them. The Disaster Recovery site will run a MongoDB cluster in unmanaged mode.
We are going to simulate the failure and perform a failover to DR site

In the 1.10.0 version of the Operator, we have added the Technology Preview of the new feature which enables users to deploy unmanaged MongoDB nodes and connect them to existing Replica Sets.

Set it All Up

We are not going to cover the configuration of the Kubernetes clusters, but in our tests, we relied on two Google Kubernetes Engine (GKE) clusters deployed in different regions.

Prepare Main Site

We have shared all the resources for this blog post in this GitHub repo. As a first step we are going to deploy the operator on the Main site:

$ kubectl apply -f bundle.yaml

Deploy the MongoDB managed cluster with cr-main.yaml:

$ kubectl apply -f cr-main.yaml

It is important to understand that we will need to expose ReplicaSet nodes through a dedicated service. This includes Config Servers. This is required to ensure that ReplicaSet nodes on Main and DR can reach each other. So it is like a full mesh:

To get there, cr-main.yaml has the following changes:

spec:
  replsets:
  - rs0:
    expose:
      enabled: true
      exposeType: LoadBalancer
  sharding:
    configsvrReplSet:
      expose:
        enabled: true
        exposeType: LoadBalancer

We are using the LoadBalancer Kubernetes Service object as it is just simpler for us, but there are other options – ClusterIP, NodePort. It is also possible to utilize 3rd party tools like Submariner to implement a private connection.

If you have an already running MongoDB cluster in Kubernetes, you can expose the ReplicaSets without downtime by changing these variables.

Prepare Disaster Recovery Site

The configuration of the Disaster Recovery site could be broken down into the following steps:

Copy the Secrets from the Main cluster.
1. system users secrets
2. SSL keys – both used for external connections and internal replication traffic
Tune Custom Resource:
1. run nodes in unmanaged mode – Operator does not control replicaset configuration and secrets generation
2. expose ReplicaSets (the same way we do it on the Main cluster)
3. disable backups – backups can be only taken on the cluster managed by the Operator

Copy the Secrets

System user’s credentials are stored by default in my-cluster-name-secrets Secret object and defined in spec.secrets.users. Apply this secret in the DR cluster with kubectl apply -f yaml-with-secrets. If you don’t have it in your source code repository or if you rely on the Operator to generate it, you can get the secret from Kubernetes itself, remove the unnecessary metadata and apply.

On main execute:

$ kubectl get secret my-cluster-name-secrets -o yaml > my-cluster-secrets.yaml

Now remove the following lines from metadata:

annotations
creationTimestamp
resourceVersion
selfLink
uid

Save the file and apply it to the DR cluster.

The procedure to copy SSL keys is almost the same as for users. The difference is the names of the Secret objects – they are usually called <CLUSTER_NAME>-ssl and <CLUSTER_NAME>-ssl-internal. It is also possible to specify them in secrets.ssl and secrets.sslInternal in the Custom Resource. Copy these two keys from Main to DR and reference them in the CR.

Tune Custom Resource

cr-replica.yaml will have the following changes:

  secrets:
    users: my-cluster-name-secrets
    ssl: replica-cluster-ssl
    sslInternal: replica-cluster-ssl-internal
 
  replsets:
  - name: rs0
    size: 3
    expose:
      enabled: true
      exposeType: LoadBalancer
 
  sharding:
    enabled: true
    configsvrReplSet:
      size: 3
      expose:
        enabled: true
        exposeType: LoadBalancer
 
  backup:
    enabled: false

Once the Custom Resource is applied, the services are going to be created. We will need the IP addresses of each ReplicaSet node to configure the DR site.

$ kubectl get services
NAME                  TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)           AGE
replica-cluster-cfg-0    LoadBalancer   10.111.241.213   34.78.119.1       27017:31083/TCP   5m28s
replica-cluster-cfg-1    LoadBalancer   10.111.243.70    35.195.138.253    27017:31957/TCP   4m52s
replica-cluster-cfg-2    LoadBalancer   10.111.246.94    146.148.113.165   27017:30196/TCP   4m6s
...
replica-cluster-rs0-0    LoadBalancer   10.111.241.41    34.79.64.213      27017:31993/TCP   5m28s
replica-cluster-rs0-1    LoadBalancer   10.111.242.158   34.76.238.149     27017:32012/TCP   4m47s
replica-cluster-rs0-2    LoadBalancer   10.111.242.191   35.195.253.107    27017:31209/TCP   4m22s

Add External Nodes to Main

At this step, we are going to add unmanaged nodes to the Replica Set on the Main site. In cr-main.yaml we should add externalNodes under replsets.[] and sharding.configsvrReplSet:

  replsets:
  - name: rs0
    externalNodes:
    - host: 34.79.64.213
      priority: 1
      votes: 1
    - host: 34.76.238.149
      priority: 1
      votes: 1
    - host: 35.195.253.107
      priority: 0
      votes: 0
 
  sharding:
    configsvrReplSet:
      externalNodes:
      - host: 34.78.119.1
        priority: 1
        votes: 1
      - host: 35.195.138.253
        priority: 1
        votes: 1
      - host: 146.148.113.165
        priority: 0
        votes: 0

Please note that we add three nodes, but only two are voters. We do this to avoid split-brain situations and do not start the primary election if the DR site is down or there is a network disruption between the Main and DR sites.

Failover

Once all the configuration above is applied, the situation will look like this:

We have three voters in the main cluster and two voters in the replica cluster. That means replica nodes won’t have the majority in case of main cluster failure and they won’t be able to elect a new primary. Therefore we need to step in and perform a manual failover.

Let’s kill the main cluster:

gcloud compute instances list \
    | grep my-main-gke-demo \
    | awk '{print $1}' \
    | xargs gcloud compute instances delete --zone europe-west3-b

gcloud container node-pools delete \
    --zone europe-west3-b \
    --cluster my-main-gke-demo \
    default-pool

I deleted the nodes and the node pool of the main Kubernetes cluster so now the cluster is in an unhealthy state. Let’s see what mongos on the DR site says when we try to read or write through it:

% ./psmdb-tester
2021/09/03 18:19:19 Successfully connected and pinged 34.141.3.189:27017
2021/09/03 18:19:40 read failed: (FailedToSatisfyReadPreference) Encountered non-retryable error during query :: caused by :: Could not find host matching read preference { mode: "primary" } for set cfg
2021/09/03 18:19:49 write failed: (FailedToSatisfyReadPreference) Could not find host matching read preference { mode: "primary" } for set cfg

Normally, we can only alter the replica set configuration from the primary node but in this kind of situation where you don’t have a primary and only have a few surviving members, MongoDB allows us to force the reconfiguration from any alive member.

Let’s connect to one of the secondary nodes in the replica cluster and perform the failover:

kubectl exec -it psmdb-client-7b9f978649-pjb2k -- mongo 'mongodb://clusterAdmin:<pass>@replica-cluster-rs0-0.replica.svc.cluster.local/admin?ssl=false'
...
rs0:SECONDARY> cfg = rs.config()
rs0:SECONDARY> cfg.members = [cfg.members[3], cfg.members[4], cfg.members[5]]
rs0:SECONDARY> rs.reconfig(cfg, {force: true})

Note that the indexes of surviving members may differ in your environment. You should check rs.status() and rs.config() outputs first. The main idea is to repopulate config members with only surviving members.

After the reconfiguration, the replica set will have just three members and two of them will have votes and a majority. So, they’ll be able to select a new primary. After performing the same process on the cfg replica set, we will be able to read and write through mongos again:

% ./psmdb-tester
2021/09/03 18:41:48 Successfully connected and pinged 34.141.3.189:27017
2021/09/03 18:41:49 read succeed
2021/09/03 18:41:50 read succeed
2021/09/03 18:41:51 read succeed
2021/09/03 18:41:52 read succeed
2021/09/03 18:41:53 read succeed
2021/09/03 18:41:54 read succeed
2021/09/03 18:41:55 read succeed
2021/09/03 18:41:56 read succeed
2021/09/03 18:41:57 read succeed
2021/09/03 18:41:58 read succeed
2021/09/03 18:41:58 write succeed

Once the replica cluster has become the primary, you should reconfigure all clients that connect to the old main cluster and point them to the DR site.

Cluster Statuses in Percona Kubernetes Operators

Wed, 21 Jul 2021 15:10:57 +0300

In Kubernetes, all resources have a status field separated from their spec. The status field is an interface both for humans or applications to read the perceived state of the resource.

When you deploy our Percona Kubernetes Operators – Percona Operator for MongoDB or Percona Operator for MySQL – in your Kubernetes cluster, you’re creating a custom resource (CR for short) and it has its own status, too. Since Kubernetes operators mimic the human operator and aim to have the required expertise to run software in a Kubernetes cluster; the status of the custom resources should be smart.

You can get cluster status with the commands below, or via (Kubernetes API) for Percona Operator for MySQL:

% kubectl get pxc
NAME            ENDPOINT                                   STATUS   PXC   PROXYSQL   HAPROXY   AGE
lisette-18537   lisette-18537-haproxy.subjectivism-22940   ready    3                3         87m

% kubectl get pxc <cluster-name> -o jsonpath='{.status}'
{
  "backup": {
    "version": "8.0.23"
  },
  "conditions": [
    {
      "lastTransitionTime": "2021-07-12T13:13:46Z",
      "status": "True",
      "type": "initializing"
    }
  ],
  "haproxy": {
    "labelSelectorPath": "...",
    "ready": 3,
    "size": 3,
    "status": "ready"
  },
  "host": "lisette-18537-haproxy.subjectivism-22940",
  "logcollector": {
    "version": "1.8.0"
  },
  "observedGeneration": 2,
  "pmm": {
    "version": "2.12.0"
  },
  "proxysql": {},
  "pxc": {
    "image": "percona/percona-xtradb-cluster:8.0.22-13.1",
    "labelSelectorPath": "...",
    "ready": 2,
    "size": 3,
    "status": "initializing",
    "version": "8.0.22-13.1"
  },
  "ready": 5,
  "size": 6,
  "state": "initializing"
}

And for Percona Operator for MongoDB:

% kubectl get psmdb
NAME             ENDPOINT                                                     STATUS   AGE
cynodont-26997   cynodont-26997-mongos.subjectivism-22940.svc.cluster.local   ready    85m


% kubectl get psmdb <cluster-name> -o jsonpath='{.status}'
{
  "conditions": [
    {
      "lastTransitionTime": "2021-07-12T13:13:39Z",
      "status": "True",
      "type": "initializing"
    }
  ],
  "host": "cynodont-26997-mongos.subjectivism-22940.svc.cluster.local",
  "mongoImage": "percona/percona-server-mongodb:4.4.6-8",
  "mongoVersion": "4.4.6-8",
  "mongos": {
    "ready": 1,
    "size": 3,
    "status": "initializing"
  },
  "observedGeneration": 2,
  "ready": 3,
  "replsets": {
    "cfg": {
      "ready": 1,
      "size": 3,
      "status": "initializing"
    },
    "rs0": {
      "initialized": true,
      "ready": 2,
      "size": 3,
      "status": "initializing"
    }
  },
  "size": 6,
  "state": "initializing"
}

As you can see there are several fields in the output: conditions, cluster size, number of ready cluster members, statuses and versions of different components, and the “state”. In the following sections, we’ll take a look at every possible value of the state field.

Initializing

While the cluster is progressing to readiness, CR status is “initializing”. It includes creating the cluster, scaling it up or down, and updating the CR that triggers a rolling restart of pods (for instance updating Percona Operator for MySQL memory limits).

Percona Operator for MongoDB also reconfigures the replica set config if necessary (for instance it adds the new pods as members to replset or removes terminated ones). Replica set in MongoDB is a set of servers that implements replication and automatic failover. Although they have the same name, it’s different from the Kubernetes replica set. While this configuration is happening or if there is an unknown/unpredicted error during it, the status is also “initializing”.

Since version 1.7.0, the Percona Operator for MySQL can handle full crash recovery if necessary. If a pod waits for the recovery, the cluster status is “initializing”.

Ready

The operator keeps track of the status of each component in the cluster. Percona Operator for MongoDB has the following components:

mongod StatefulSet
configsvr StatefulSet if sharding is enabled
mongos Deployment if sharding is enabled

Percona Operator for MySQL components:

PXC StatefulSet
HAProxy StatefulSet if enabled
ProxySQL StatefulSet if enabled

All components need to be in “ready” status for CR to be “ready”. If the number of ready pods controlled by the stateful set reaches the desired number, the operator marks the component as ready. The readiness of the pods is tracked by Kubernetes using readiness probes for each container in the pod. For example, for a Percona XtraDB Cluster container to be ready “wsrep_cluster_status” needs to be “Primary” and “wsrep_local_state” should be “Synced” or “Donor”. For a Percona Server for MongoDB container to be ready, accepting TCP connections on 27017 is enough.

But ready as the CR status means more than that. CR “ready” means the cluster (Percona Server for MongoDB or Percona XtraDB Cluster) is up and running and ready to receive traffic. So, even if all components are ready, the cluster status can be “initializing”. In the Percona Operator for MongoDB, the replica set needs to be initialized and its config up-to-date. Also, with the 1.9.0 release of both operators, the load balancer needs to be ready if the cluster is exposed with exposeType: LoadBalancer.

Stopping

Version 1.9.0 introduced two new statuses:

Stopping
Paused

Stopping means the cluster is paused or deleted and its pods are terminating right now.

If you run kubectl delete psmdb <cluster-name> or `kubectl delete pxc `` the resource can be deleted quickly without a chance to see “stopping” status. If you had finalizers (for example “delete-pxc-pods-in-order” in Percona Operator for MySQL) deletion will be blocked until the finalizer list is exhausted and you can observe “stopping” status.

Paused

Once the cluster is paused and all pods are terminated, the CR status becomes “paused”.

To pause the cluster: kubectl patch <psmdb|pxc> <cluster-name> --type=merge -p '{"spec": {"pause": true}}'

Keep in mind, when the cluster is paused and exposeType is LoadBalancer – Load balancers are still there and you continue to pay for them.

Error

Before 1.9.0, “error” status could mean two different things:

An error occurred in the operator during the reconciliation of the CR
One or more pods in a component are not schedulable

With 1.9.0, the “error” status means only the operator errors. If there is an unschedulable pod, the cluster’s status will be initializing. If the cluster is stuck in initializing for too long, it’s better to check the operator logs to investigate.

% kubectl logs <operator-pod-name>
...
{"level":"info","ts":1626095618.9982307,"logger":"controller_psmdb","msg":"Created a new mongo key","Request.Namespace":"subjectivism-22940","Request.Name":"cynodont-26997","KeyName":"cynodont-26997-mongodb-keyfile"}
{"level":"info","ts":1626095619.0032709,"logger":"controller_psmdb","msg":"Created a new mongo key","Request.Namespace":"subjectivism-22940","Request.Name":"cynodont-26997","KeyName":"cynodont-26997-mongodb-encryption-key"}
{"level":"info","ts":1626095687.3783236,"logger":"controller_psmdb","msg":"initiating replset","replset":"rs0","pod":"cynodont-26997-rs0-1"}
{"level":"info","ts":1626095694.020591,"logger":"controller_psmdb","msg":"replset was initialized","replset":"rs0","pod":"cynodont-26997-rs0-1"}
{"level":"error","ts":1626095694.622869,"logger":"controller_psmdb","msg":"failed to reconcile cluster","Request.Namespace":"subjectivism-22940","Request.Name":"cynodont-26997","replset":"rs0","error":"undefined state of the replset member cynodont-26997-rs0-0.cynodont-26997-rs0.subjectivism-22940.svc.cluster.local:27017: 6","errorVerbose":"undefined state of the replset member cynodont-26997-rs0-0.cynodont-26997-rs0.subjectivism-22940.svc.cluster.local:27017: 6\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileCluster\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/mgo.go:210\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:449\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:451\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
% kubectl logs <operator-pod-name>
 
...
{"level":"info","ts":1626095618.9982307,"logger":"controller_psmdb","msg":"Created a new mongo key","Request.Namespace":"subjectivism-22940","Request.Name":"cynodont-26997","KeyName":"cynodont-26997-mongodb-keyfile"}
{"level":"info","ts":1626095619.0032709,"logger":"controller_psmdb","msg":"Created a new mongo key","Request.Namespace":"subjectivism-22940","Request.Name":"cynodont-26997","KeyName":"cynodont-26997-mongodb-encryption-key"}
{"level":"info","ts":1626095687.3783236,"logger":"controller_psmdb","msg":"initiating replset","replset":"rs0","pod":"cynodont-26997-rs0-1"}
{"level":"info","ts":1626095694.020591,"logger":"controller_psmdb","msg":"replset was initialized","replset":"rs0","pod":"cynodont-26997-rs0-1"}
{"level":"error","ts":1626095694.622869,"logger":"controller_psmdb","msg":"failed to reconcile cluster","Request.Namespace":"subjectivism-22940","Request.Name":"cynodont-26997","replset":"rs0","error":"undefined state of the replset member cynodont-26997-rs0-0.cynodont-26997-rs0.subjectivism-22940.svc.cluster.local:27017: 6","errorVerbose":"undefined state of the replset member cynodont-26997-rs0-0.cynodont-26997-rs0.subjectivism-22940.svc.cluster.local:27017: 6\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileCluster\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/mgo.go:210\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:449\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:451\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}

You can try new statuses in version 1.9.0 of both Percona Operator for MongoDB and Percona Operator for MySQL. Percona Operator for MongoDB was released in June and Percona Operator for MySQL is on the way.

Kubernetes Resource Management

Fri, 13 Nov 2020 00:00:00 +0000

I had the chance to listen to Bekir Doğan’s presentation, a former Kartaca employee, at an event in 2017. I was very impressed when I heard that they set up and distribute all the services they manage with OpenVZ in containers in 2005. Was anyone really into this type of thing?

Apparently, yes. Since the early 2000s, most of the industry and the Linux community have been trying to make containers into what they are today. In particular, Google has been a pioneer in making containers mainstream with its contributions. You might have thought of Kubernetes right away, but this time I’m talking about cgroups technology.

A brief introduction to cgroups

It is easy to mistake a container for the entire system’s sole owner since it isolates a group of processes and runs on the same core with other containers and applications instead of virtualizing the whole system. An aggressively resource-consuming container can also destabilize the fellow containers, making the system unstable.

To prevent this, we can allocate the entire system to a container using virtualization, but this will waste resources most of the time. For example, in Borg’s design documents, maximum utilization of resources is stated as one of the project’s main objectives.

At this point, cgroups comes into play.

Google engineers started developing cgroups in 2006, and it was included in Linux 2.6.24 in 2008. It is a disruptive feature that shaped the ecosystem with the domino effect it creates.

With the inclusion of the code in Linux, system administrators can group the system’s processes/tasks and subject them to common constraints. Process priorities and resource limits can be configured and included in the accounting. Moreover, this kernel capability paved the way for software that radically changed system management such as LXC and later Docker.

Here is a small cgroups demo for you:

To summarize the demo;

We create a control group called fibtest.
Within the group we create, we run a small C application called fibtest. This application generates Fibonacci sequences continuously.
With systemd-cgtop, we monitor resource consumption in all groups in the system.
The real story starts here. Inside the group, we change two values: cpu.cfs_period_us and cpu.cfs_quota_us. While determining how many microseconds each CPU period will last (50000) with cfs_period_us, with cfs_quota_us, we determine the maximum number of microseconds the program can use in each period (1000). Long story short, we choke the program.
We take back the values, and fibtest breathes again.

You can also get the cgcreate and cgexec tools we used in the demo by installing cgroup-tools in Ubuntu 18.04, and libcgroup-tools in Fedora 32.

Kubernetes and cgroups

By default, system programs and containers run competitively on the machine resources. The containers’ resource consumption can make the machine unstable if no resources are allocated for system operations.

Kubernetes also provides resource isolation for system and user processes with cgroups. A cgroup called kubepods is created on every machine (if it doesn’t already exist). For the Kubernetes system and services, the cgroup is not created automatically; system administrators need to reconfigure kubelet.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
systemReserved:
  cpu: 500m
  memory: 500M
kubeReserved:
  cpu: 500m
  memory: 500M

$ kubectl describe node
...
Capacity:
  attachable-volumes-gce-pd:  127
  cpu:                        8
  ephemeral-storage:          47259264Ki
  hugepages-2Mi:              0
  memory:                     30879764Ki
  pods:                       110
Allocatable:
  attachable-volumes-gce-pd:  127
  cpu:                        7
  ephemeral-storage:          18858075679
  hugepages-2Mi:              0
  memory:                     29831188Ki
  pods:                       110
...

Capacity shows the total resources seen by Kubernetes in the machine, and Allocatable shows the total resources allocated to user pods.

Resource requests and limits

Two concepts immediately appear in Kubernetes resource management: resource request and resource limit.

Resource request is taken into account by the scheduler for placing pods on machines. As long as there is enough space in the resource allocated to the user pods, the pod is assigned to a machine. Once the pod is assigned to a machine, kubelet guarantees that the container can always use the requested resource. “Pending” status means that the pod is waiting to be assigned. Every pod’s lifecycle includes a “Pending” status; however, if a pod spends a long time in this situation, it is useful to review its resource request. It consists of the sum of the requests of the containers within the source request of a pod.

It is crucial to define resource requests for each container to efficiently benefit from user pods’ total resources. The scheduler checks whether the machine’s available capacity is higher than the pods’ entire resource request while assigning the pods. In practice, even if the pods in the machine consume fewer resources than their request, if the sum of their requests is equal to the available resources, no new pods are assigned to this machine. Because, as I mentioned above, the source requested by the container is always guaranteed by kubelet.

Resource limit is taken into account by kubelet to prevent a pod from consuming all the system resources.

A container can consume more resources than it requests. However, if it consumes more memory than its request and the machine runs low on memory, it will be evicted.

What happens to a container that consumes more than its limit depends on the respective resource:

If the memory limit is exceeded, the container can be terminated and restarted if possible.
If the CPU limit is exceeded, the container is not terminated; only the CPU usage is throttled.

In practice, the total resource request of all containers on the machine cannot exceed the resource allocated to user pods; however, the sum of resource limits may be well above the available resources.

The total resource limit can exceed the maximum resource limit, just as the aircraft companies sell extra tickets and do overbooking. In this case, it will be more important to allocate resources for the system and Kubernetes processes, as I explained above.

Namespace level resource management

If you use Kubernetes namespaces to separate your services or environments (such as test, qa), you can set predefined resource requests and limits for each namespace. Therefore you can use default values for each container without configuring the resources separately.

For the default resource configuration, it is necessary to define a LimitRange:

apiVersion: v1
kind: LimitRange
metadata:
  name: qa-limitrange
spec:
  limits:
  - default:
      cpu: 1
      memory: 512Mi
    defaultRequest:
      cpu: 500m
      memory: 256Mi
    type: Container


$ kubectl describe limits
Name:       qa-limitrange
Namespace:  qa
Type        Resource  Min  Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---  ---  ---------------  ------------- -----------------------
Container   cpu       -    -    500m             1              -
Container   memory    -    -    256Mi            512Mi          -

Since we are dealing with so many YAML files enough to experience minor crying outbursts during the day, it is desirable to write four lines less in each file. However, LimitRange does more than that. Especially in multi-tenant Kubernetes clusters, we can define LimitRange to approve resource requests and limits.

apiVersion: v1
kind: LimitRange
metadata:
  name: dev-limitrange
spec:
  limits:
  - max:
      cpu: 2
      memory: 1Gi
    min:
      cpu: 1
      memory: 500Mi
    type: Container



$ kubectl describe limits
Name:       dev-limitrange
Namespace:  dev
Type        Resource  Min    Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---    ---  ---------------  ------------- -----------------------
Container   cpu       500m   2    2                2              -
Container   memory    256Mi  1Gi  1Gi              1Gi            -

When we define LimitRange for a namespace, we create an admission controller, enabling us to add any pod to the cluster after being approved in terms of resource configuration before being accepted. Those who do not conform to the LimitRange rules get rejected. This way, third parties, independent of the system administrator, can install new pods without making other services unstable.

Our control over namespace is not limited to this. We can also limit the total resources of the namespace by defining a ResourceQuota.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: qa-resourcequota
  namespace: qa
spec:
  hard:
    requests.cpu: "2"
    requests.memory: 8Gi
    limits.cpu: "2"
    limits.memory: 8Gi
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: test-resourcequota
  namespace: test
spec:
  hard:
    requests.cpu: "1"
    requests.memory: 4Gi
    limits.cpu: "1"
    limits.memory: 4Gi

With the above configuration, we guarantee the following:

The total memory request and limit of pods in Namespace cannot exceed 8 GB for qa and 4 GB for testing.
The total CPU request and limit of pods in Namespace cannot exceed 2 CPUs for qa and 1 CPU for testing.

QoS classes

The QoS (Quality of Service) classes belonging to pods are essential for both scheduling and eviction. There are three classes of QoS we can use:

Guaranteed: If the resource requests and limits of all containers within the pod are equal.
Burstable: Pod does not classify as Guaranteed and requests at least one container resource.
BestEffort: If no container in the pod has any resource requests or limits.

As you can see, these classes are assigned by Kubernetes based on pods’ source configuration.

$ kubectl get pod <pod> -o yaml
apiVersion: v1
kind: Pod
metadata:
  name: pod
spec:
  containers:
  - name: container-1
    image: ...
    resources:
      limits:
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 512Mi
status:
  ...
  phase: Running
  qosClass: Burstable

Eviction

Despite all the configurations I explained above, our machines’ resources may run out, and the cluster may become unstable. In this case, kubelet will try to recover resources quickly. If their efforts are futile, eviction process begins.

For eviction, kubelet puts the pods in a row:

Those pods belonging to BestEffort or Burstable QoS classes, which use more than request sources from pods, are ranked according to their priorities and how much resources they consume out of their requests, and are evicted. Guaranteed pods and Burstable pods that consume fewer resources than their requests are evicted the last. As their name suggests, Guaranteed pods are assured that they will be not be evicted due to other pods’ resource consumption. However, if the resources allocated to the system or Kubernetes start to run out, they can also be evicted, beginning with the lowest priority pod.

Priority is a value that we can set with PriorityClass, again as Kubernetes administrators. In order not to complicate this blog post further, I end it here after sharing relevant documents.

This post is originally published in Kartaca’s blog. Thanks to lovely Gizem for editing and translating the text.