3.0 configuration: replication administration

andreyaksenov · andreyaksenov · commit 0e7b52a26f62 · 2023-12-04T14:49:59.000+03:00
diff --git a/doc/book/admin/replication/_images/box_info_replication_instance001.png b/doc/book/admin/replication/_images/box_info_replication_instance001.png
diff --git a/doc/book/admin/replication/_images/box_info_replication_instance002.png b/doc/book/admin/replication/_images/box_info_replication_instance002.png
diff --git a/doc/book/admin/replication/repl_monitoring.rst b/doc/book/admin/replication/repl_monitoring.rst
@@ -4,49 +4,26 @@ Monitoring a replica set
 ========================
 
 To learn what instances belong to the replica set and obtain statistics for all
-these instances, issue a :doc:`/reference/reference_lua/box_info/replication` request:
-
-..  code-block:: tarantoolsession
-
-    tarantool> box.info.replication
-    ---
-      replication:
-        1:
-          id: 1
-          uuid: b8a7db60-745f-41b3-bf68-5fcce7a1e019
-          lsn: 88
-        2:
-          id: 2
-          uuid: cd3c7da2-a638-4c5d-ae63-e7767c3a6896
-          lsn: 31
-          upstream:
-            status: follow
-            idle: 43.187747001648
-            peer: replicator@192.168.0.102:3301
-            lag: 0
-          downstream:
-            vclock: {1: 31}
-        3:
-          id: 3
-          uuid: e38ef895-5804-43b9-81ac-9f2cd872b9c4
-          lsn: 54
-          upstream:
-            status: follow
-            idle: 43.187621831894
-            peer: replicator@192.168.0.103:3301
-            lag: 2
-          downstream:
-            vclock: {1: 54}
-    ...
-
-This report is for a master-master replica set of three instances, each having
-its own instance id, UUID and log sequence number.
-
-..  image:: /concepts/replication/images/mm-3m-mesh.svg
+these instances, execute a :ref:`box.info.replication <box_info_replication>` request.
+The output below shows replication status for a replica set containing one :ref:`master and two replicas <replication-master_replica_bootstrap>`:
+
+..  include:: /how-to/replication/repl_bootstrap.rst
+    :start-after: box_info_replication_manual_leader_start
+    :end-before: box_info_replication_manual_leader_end
+
+The following diagram illustrates the ``upstream`` and ``downstream`` connections for the ``box.info.replication`` executed at the master instance (``instance001``):
+
+..  image:: _images/box_info_replication_instance001.png
+    :align: center
+    :alt: replication status on master
+
+If ``box.info.replication`` is executed on ``instance002``, the ``upstream`` and ``downstream`` connections look as follows:
+
+..  image:: _images/box_info_replication_instance002.png
     :align: center
+    :alt: replication status on replica
 
-The request was issued at master #1, and the reply includes statistics for the
-other two masters, given in regard to master #1.
+This means that statistics for replicas is given in regard to the instance on which ``box.info.replication`` is executed.
 
 The primary indicators of replication health are:
 
@@ -74,9 +51,4 @@ The primary indicators of replication health are:
     machines, do not be surprised if it’s negative: a time drift may lead to the
     remote master clock being consistently behind the local instance's clock.
 
-    For multi-master configurations, ``lag`` is the maximal lag.
-
-For better understanding, see the following diagram illustrating the ``upstream`` and ``downstream`` connections within the replica set of three instances:
-
-..  image:: /concepts/replication/images/replication.svg
-    :align: left
+    For a :ref:`master-master <replication-bootstrap-master-master>` configuration, ``lag`` is the maximal lag.
diff --git a/doc/book/admin/replication/repl_recover.rst b/doc/book/admin/replication/repl_recover.rst
@@ -1,107 +1,55 @@
 .. _replication-recover:
 
-================================================================================
 Recovering from a degraded state
-================================================================================
+================================
 
 "Degraded state" is a situation when the master becomes unavailable -- due to
 hardware or network failure, or due to a programming bug.
 
 .. image:: mr-degraded.svg
     :align: center
 
-In a master-replica set, if a master disappears, error messages appear on the
-replicas stating that the connection is lost:
-
-.. code-block:: console
-
-   $ # messages from a replica's log
-   2017-06-14 16:23:10.993 [19153] main/105/applier/replicator@192.168.0. I> can't read row
-   2017-06-14 16:23:10.993 [19153] main/105/applier/replicator@192.168.0. coio.cc:349 !> SystemError
-   unexpected EOF when reading from socket, called on fd 17, aka 192.168.0.101:57815,
-   peer of 192.168.0.101:3301: Broken pipe
-   2017-06-14 16:23:10.993 [19153] main/105/applier/replicator@192.168.0. I> will retry every 1 second
-   2017-06-14 16:23:10.993 [19153] relay/[::ffff:192.168.0.101]:/101/main I> the replica has closed its socket, exiting
-   2017-06-14 16:23:10.993 [19153] relay/[::ffff:192.168.0.101]:/101/main C> exiting the relay loop
-
-... and the master's status is reported as "disconnected":
-
-.. code-block:: tarantoolsession
-
-   # report from replica #1
-   tarantool> box.info.replication
-   ---
-   - 1:
-       id: 1
-       uuid: 70e8e9dc-e38d-4046-99e5-d25419267229
-       lsn: 542
-       upstream:
-         peer: replicator@192.168.0.101:3301
-         lag: 0.00026607513427734
-         status: disconnected
-         idle: 182.36929893494
-         message: connect, called on fd 13, aka 192.168.0.101:58244
-     2:
-       id: 2
-       uuid: fb252ac7-5c34-4459-84d0-54d248b8c87e
-       lsn: 0
-     3:
-       id: 3
-       uuid: fd7681d8-255f-4237-b8bb-c4fb9d99024d
-       lsn: 0
-       downstream:
-         vclock: {1: 542}
-   ...
-
-.. code-block:: tarantoolsession
-
-   # report from replica #2
-   tarantool> box.info.replication
-   ---
-   - 1:
-       id: 1
-       uuid: 70e8e9dc-e38d-4046-99e5-d25419267229
-       lsn: 542
-       upstream:
-         peer: replicator@192.168.0.101:3301
-         lag: 0.00027203559875488
-         status: disconnected
-         idle: 186.76988101006
-         message: connect, called on fd 13, aka 192.168.0.101:58253
-     2:
-       id: 2
-       uuid: fb252ac7-5c34-4459-84d0-54d248b8c87e
-       lsn: 0
-       upstream:
-         status: follow
-         idle: 186.76960110664
-         peer: replicator@192.168.0.102:3301
-         lag: 0.00020599365234375
-     3:
-       id: 3
-       uuid: fd7681d8-255f-4237-b8bb-c4fb9d99024d
-       lsn: 0
-   ...
-
-To declare that one of the replicas must now take over as a new master:
-
-1. Make sure that the old master is gone for good:
-
-   * change network routing rules to avoid any more packets being delivered to
-     the master, or
-   * shut down the master instance, if you have access to the machine, or
-   * power off the container or the machine.
-
-2. Say ``box.cfg{read_only=false, listen=URI}`` on the replica, and
-   ``box.cfg{replication=URI}`` on the other replicas in the set.
-
-.. NOTE::
-
-   If there are updates on the old master that were not propagated before the
-   old master went down,
-   :ref:`re-apply them manually <admin-disaster_recovery-master_replica>` to the
-   new master using ``tt cat`` and ``tt play`` commands.
-
-There is no automatic way for a replica to detect that the master is gone
-forever, since sources of failure and replication environments vary
-significantly. So the detection of degraded state requires an external observer.
+-   In a master-replica set with manual failover, if a master disappears, error messages appear on the
+    replicas stating that the connection is lost:
+
+    .. code-block:: console
+
+        2023-12-04 13:19:04.724 [16755] main/110/applier/replicator@127.0.0.1:3301 I> can't read row
+        2023-12-04 13:19:04.724 [16755] main/110/applier/replicator@127.0.0.1:3301 coio.c:349 E> SocketError: unexpected EOF when reading from socket, called on fd 19, aka 127.0.0.1:55932, peer of 127.0.0.1:3301: Broken pipe
+        2023-12-04 13:19:04.724 [16755] main/110/applier/replicator@127.0.0.1:3301 I> will retry every 1.00 second
+        2023-12-04 13:19:04.724 [16755] relay/127.0.0.1:55940/101/main coio.c:349 E> SocketError: unexpected EOF when reading from socket, called on fd 23, aka 127.0.0.1:3302, peer of 127.0.0.1:55940: Broken pipe
+        2023-12-04 13:19:04.724 [16755] relay/127.0.0.1:55940/101/main I> exiting the relay loop
+
+-   In a master-replica set with automated failover, a log should contain Raft messages showing the process of a new master's election:
+
+    .. code-block:: console
+
+        2023-12-04 13:16:56.340 [16615] main/111/applier/replicator@127.0.0.1:3302 I> can't read row
+        2023-12-04 13:16:56.340 [16615] main/111/applier/replicator@127.0.0.1:3302 coio.c:349 E> SocketError: unexpected EOF when reading from socket, called on fd 24, aka 127.0.0.1:55687, peer of 127.0.0.1:3302: Broken pipe
+        2023-12-04 13:16:56.340 [16615] main/111/applier/replicator@127.0.0.1:3302 I> will retry every 1.00 second
+        2023-12-04 13:16:56.340 [16615] relay/127.0.0.1:55695/101/main coio.c:349 E> SocketError: unexpected EOF when reading from socket, called on fd 25, aka 127.0.0.1:3301, peer of 127.0.0.1:55695: Broken pipe
+        2023-12-04 13:16:56.340 [16615] relay/127.0.0.1:55695/101/main I> exiting the relay loop
+        2023-12-04 13:16:59.690 [16615] main/112/applier/replicator@127.0.0.1:3303 I> RAFT: message {term: 3, vote: 2, state: candidate, vclock: {1: 9}} from 2
+        2023-12-04 13:16:59.690 [16615] main/112/applier/replicator@127.0.0.1:3303 I> RAFT: received a newer term from 2
+        2023-12-04 13:16:59.690 [16615] main/112/applier/replicator@127.0.0.1:3303 I> RAFT: bump term to 3, follow
+        2023-12-04 13:16:59.690 [16615] main/112/applier/replicator@127.0.0.1:3303 I> RAFT: vote for 2, follow
+        2023-12-04 13:16:59.691 [16615] main/119/raft_worker I> RAFT: persisted state {term: 3}
+        2023-12-04 13:16:59.691 [16615] main/119/raft_worker I> RAFT: persisted state {term: 3, vote: 2}
+        2023-12-04 13:16:59.691 [16615] main/112/applier/replicator@127.0.0.1:3303 I> RAFT: message {term: 3, vote: 2, leader: 2, state: leader} from 2
+        2023-12-04 13:16:59.691 [16615] main/112/applier/replicator@127.0.0.1:3303 I> RAFT: vote request is skipped - this is a notification about a vote for a third node, not a request
+        2023-12-04 13:16:59.691 [16615] main/112/applier/replicator@127.0.0.1:3303 I> RAFT: leader is 2, follow
+
+
+
+The master's status is reported as ``disconnected`` when executing ``box.info.replication`` on a replica:
+
+..  include:: /how-to/replication/repl_bootstrap_auto.rst
+    :start-after: box_info_replication_auto_leader_disconnected_start
+    :end-before: box_info_replication_auto_leader_disconnected_end
+
+
+Performing failover:
+
+-   Master-replica: :ref:`Performing manual failover <replication-controlled_failover>`
+-   Master-replica: :ref:`Testing automated failover <replication-automated-failover-testing>`
+
diff --git a/doc/how-to/replication/repl_bootstrap.rst b/doc/how-to/replication/repl_bootstrap.rst
@@ -338,6 +338,8 @@ After adding ``instance003`` to the configuration and starting it, configuration
 3.  Execute ``box.info.replication`` to check a replica set status.
     Make sure that ``upstream.status`` and ``downstream.status`` are ``follow`` for ``instance003``.
 
+    .. box_info_replication_manual_leader_start
+
     .. code-block:: console
 
         manual_leader:instance001> box.info.replication
@@ -379,6 +381,8 @@ After adding ``instance003`` to the configuration and starting it, configuration
               lag: 0
                 ...
 
+    .. box_info_replication_manual_leader_end
+
 
 
 ..  _replication-controlled_failover:
diff --git a/doc/how-to/replication/repl_bootstrap_auto.rst b/doc/how-to/replication/repl_bootstrap_auto.rst
@@ -311,6 +311,8 @@ To test how automated failover works if the current master is stopped, follow th
     -   ``upstream.status`` is ``disconnected``.
     -   ``downstream.status`` is ``stopped``.
 
+    .. box_info_replication_auto_leader_disconnected_start
+
     .. code-block:: console
 
         auto_leader:instance001> box.info.replication
@@ -354,6 +356,8 @@ To test how automated failover works if the current master is stopped, follow th
               lag: 0.00051403045654297
         ...
 
+    .. box_info_replication_auto_leader_disconnected_end
+
 
 4.  Start ``instance002`` back using ``tt start``:
 
diff --git a/doc/reference/reference_lua/box_info/replication.rst b/doc/reference/reference_lua/box_info/replication.rst
@@ -136,9 +136,3 @@ box.info.replication
       from socket'``, and ``system_message = 'Broken pipe'``.
       See also :ref:`degraded state <replication-recover>`.
 
-
-    For better understanding, see the following diagram illustrating the ``upstream`` and ``downstream`` connections within the replica set of three instances:
-
-    ..  image:: /concepts/replication/images/replication.svg
-        :align: left
-
