Jepsen-testing RabbitMQ [with python] – Part 2

This is the second post about my efforts to reproduce the Jepsen RabbitMQ test (using python). The first one failed to reproduce data loss by cutting the network in half the same way every time. Here I’ll try different partitioning schemes.

Random blockade partitions

First, let’s try the blockade’s random partitions:

This is implemented as a nemesis in blockade_random_partitions-rabbitmq-test.py.

Running the test and yet again no messages were lost:

[WARNING] (MainThread) RECEIVED: 25000, unexpected: 19. [15042, 20042, 41, 10041, 5042, 10590, 20609, 604, 15630, 16155, 21134, 1134, 11709, 21709, 22240, 2496, 7498, 12279, 17532], LOST MESSAGES 0, []

Jepsen random halves

Let’s try to cut the network into random halves like in the original Jepsen test:

Meanwhile, the nemesis cuts the network into random halves every sixty seconds, then repairs it for sixty seconds, and so on.

This is implemented in random-majority-rabbitmq-test.py

Running this test and yet again no messages were lost.

Baliant Pato’s partitions

In an excellent post, Baliant Pato explained how he managed to reproduce the data loss. Let’s try Baliant Pato’s partitions (in a 3 node cluster: rabbit1, rabbit2, rabbit3 with pause_minority)

  1. Publisher A connecting to rabbit1
  2. partition rabbit3 away from rabbit2 and rabbit1 (now rabbit3 is stale pretty quickly, he’s gonna be the future “stale node”, >evil laughter<.)
  3. partition rabbit1 away from rabbit2 and rabbit3 (at this point the cluster is stopped, the final two minorities are in sync but dead)
  4. partition rabbit2 away from rabbit1 and rabbit3
  5. heal rabbit3 (cluster is still stopped)
  6. heal rabbit2

For this test I need 3 instead of 5 rabbitmq nodes (make a copy of docker-compose.yml and then sudo docker-compose -f docker-compose-3_nodes.yml up). I’ll be using 3 instead of 2 producers, one per rabbitmq node, and a single queue with a default exchange mirrored to all the nodes (“ha-mode”:”all”).

The partitioning scheme is:

  1. sudo blockade partition n1,n2 n3
  2. sudo blockade partition n1 n2 n3
  3. sudo blockade partition n1 n2,n3
  4. sudo blockade join

Implemented as a nemesis in Baliant_Pato_partitioning-rabbitmq-test.py:

rabbit2&rabbit3 are down, producers cannot finish the test.. let’s run one more time:

This leaves rabbit2 & rabbit3 again down.

hm, rabbit2 synchronized with rabbit3.

So rabbit2 & rabbit3 think rabbit1 is dead and vice versa. If i connect to rabbit2 and drain the queue, then no messages are lost.

But wasn’t rabbitmq supposed to have detected a partition and report it? As in:

 

pause_minority is supposedly an “automatic partition handling” mechanism (excerupt from the rabbitmq.com/partitions.html)

RabbitMQ also offers three ways to deal with network partitions automatically: pause-minority mode, pause-if-all-down mode and autoheal mode.

In pause-minority mode RabbitMQ will automatically pause cluster nodes which determine themselves to be in a minority (i.e. fewer or equal than half the total number of nodes) after seeing other nodes go down. … The minority nodes will pause as soon as a partition starts, and will start again when the partition ends.

If you after reading this assume (like I did) that no intervention is needed then you’re up for a surprise. I ended up here with the rabbit2&rabbit3 nodes reported as being down, and I have to manually restart them.