In the previous articles we seen how the replication factor (RF) works, how tablets and their replicas work based on the replication factor, and what happens when a minority fails (the tablet remains serving) and when a majority fails (the tablet goes down).

But what happens when a minority fails and is destroyed 'forever'?

I started up minimalistic YugabyteDB cluster with 3 tablet servers:

➜ yb_stats --print-tablet-servers
yb-1.local:9000      ALIVE Placement: local.local.local1
                     HB time: 0.9s, Uptime: 487, Ram 33.90 MB
                     SST files: nr: 0, size: 0 B, uncompressed: 0 B
                     ops read: 0, write: 0
                     tablets: active: 24, user (leader/total): 0/0, system (leader/total): 8/24
                     Path: /mnt/d0, total: 10724835328, used: 166162432 (1.55%)
yb-2.local:9000      ALIVE Placement: local.local.local2
                     HB time: 0.9s, Uptime: 487, Ram 33.10 MB
                     SST files: nr: 0, size: 0 B, uncompressed: 0 B
                     ops read: 0, write: 0
                     tablets: active: 24, user (leader/total): 0/0, system (leader/total): 8/24
                     Path: /mnt/d0, total: 10724835328, used: 166121472 (1.55%)
yb-3.local:9000      ALIVE Placement: local.local.local3
                     HB time: 0.5s, Uptime: 486, Ram 34.31 MB
                     SST files: nr: 0, size: 0 B, uncompressed: 0 B
                     ops read: 0, write: 0
                     tablets: active: 24, user (leader/total): 0/0, system (leader/total): 8/24
                     Path: /mnt/d0, total: 10724835328, used: 166395904 (1.55%)

For the sake of the demo, we need to have actual user objects.
Let's create a simple table, and fill it with data:

create table test (id int primary key, f1 text) split into 1 tablets;
insert into test select id, repeat('x',1000) from generate_series(1,1000000) id;

Let's see how that looks like:

➜ yb_stats --print-entities
Keyspace:     ysql.postgres id: 000033e6000030008000000000000000
Keyspace:     ysql.yugabyte id: 000033e8000030008000000000000000
Keyspace:     ysql.system_platform id: 000033e9000030008000000000000000
Object:       ysql.yugabyte.test, state: RUNNING, id: 000033e8000030008000000000004000
  Tablet:     ysql.yugabyte.test.acc8f56170dd48598f0ee09a8fa8ba6f state: RUNNING
    Replicas: (yb-2.local:9100(VOTER:LEADER), yb-3.local:9100(VOTER), yb-1.local:9100(VOTER),)

One object (ysql.yugabyte.test), with one tablet, with 3 replicas. All three replicas are 'VOTER', the replica on yb-2.local is LEADER.

Exterminate!

Now let's simulate permanent tablet server 'destruction'. This is a combination of stopping the tablet server:

sudo systemctl stop yb-tserver

Which is what is shown in
Watching a YugabyteDB table: tablets and replicas.

...and the second step is to delete the tablet server data, which is stored at the location indicated by the parameter fs_data_dirs:

fs_data_dirs/pg_data (intermediate postgres files for YSQL)
fs_data_dirs/yb-data/tserver Obviously this makes the data from the tablet server be gone. In my cluster this done in this way:

sudo rm -rf /mnt/d0/pg_data
sudo rm -rf/mnt/d0/yb-data/tserver

Important!

But there is more, which is important to know: if the tablet server now is started again, it will initialise itself because there is no prior data, and start as a completely new tablet server, despite it having the exact same hostname and port number. This is visible using the tablet server screen in the master (master/tablet-servers) with the tablet server UUID: that has changed, making it a totally different server for the cluster.

The situation after termination

I stopped the tablet server on my node yb-3.local (my "third node"). After tserver_unresponsive_timeout_ms time, the tablet server will be considered DEAD, and the thus the tablet under replicated:

➜ yb_stats --print-entities
Keyspace:     ysql.postgres id: 000033e6000030008000000000000000
Keyspace:     ysql.yugabyte id: 000033e8000030008000000000000000
Keyspace:     ysql.system_platform id: 000033e9000030008000000000000000
Object:       ysql.yugabyte.test, state: RUNNING, id: 000033e8000030008000000000004000
  Tablet:     ysql.yugabyte.test.acc8f56170dd48598f0ee09a8fa8ba6f state: RUNNING [UNDER REPLICATED]
    Replicas: (yb-2.local:9100(VOTER:LEADER), yb-3.local:9100(VOTER[DEAD]), yb-1.local:9100(VOTER),)

And because the number of unavailable replicas is a minority, it will be removed from the tablet administration after follower_unavailable_considered_failed_sec.

Adding the tablet server

Because we removed the data directories, the tablet server on the third node will be a completely new tablet server, which for the cluster means it has a different UUID. The cluster does not take the server name or ip address into account for data usage, it uses the UUID exclusively.

Start up the tablet server:

sudo systemctl start yb-tserver

If now the database objects on the YugabyteDB cluster are queried again you will find the tablet having added the 3rd replica:

➜ yb_stats --print-entities
Keyspace:     ysql.postgres id: 000033e6000030008000000000000000
Keyspace:     ysql.yugabyte id: 000033e8000030008000000000000000
Keyspace:     ysql.system_platform id: 000033e9000030008000000000000000
Object:       ysql.yugabyte.test, state: RUNNING, id: 000033e8000030008000000000004000
  Tablet:     ysql.yugabyte.test.acc8f56170dd48598f0ee09a8fa8ba6f state: RUNNING
    Replicas: (yb-2.local:9100(VOTER:LEADER), yb-1.local:9100(VOTER), yb-3.local:9100(VOTER),)

This is great, but you should be aware a lot of things happened automatically, which the YugabyteDB cluster handled for you:

The situation before starting the tablet server that we "exterminated" by removing the data from it was (for the user table) that:

The table is using a single tablet, and there was 1 user tablet in this test scenario.
The tablet is using replication factor 3, so for it to function we need a majority (2 replicas) at least, which allows a leader to be elected, which was the case.

When the tablet server was started:

The cluster (load balancer) found a placement option for the third replica.
The third replica was placed on the tablet server using a process called 'remote bootstrapping'.
After the replica was bootstrapped, the data was transferred.
The newly added replica was synchronised and added to the tablet, making it a fully functioning replica again.

Warning

On my test cluster, this mostly happened. My test cluster is sized smaller than the minimal sizing rules found here: hardware requirements.
As a consequence, the remote bootstrapping put the replica in the status 'PRE_VOTER', and tried allocating a buffer for data transfer, which it couldn't because the memory area was too small, and therefore stopped after bootstrapping in the PRE_VOTER state. To solve this issue, I had to add --remote_bootstrap_max_chunk_size=20000000 to the tablet server configuration file, this will be the subject of another blogpost.

Conclusion

This blogpost shows one of the key strengths of a YugabyteDB database cluster. Completely transparently to the database APIs, we simulated a failure of a node, which didn't interrupt availability of the database, and once we introduced a "replacement" node, the database repaired itself.

Watching a YugabyteDB table: replica destruction and recovery